Test framework

Thunderforce's test framework uses a customized version of XULUnit for both unit and integration testing.

Test Architecture - Deployment View

The Thunderforce test framework exists as a Thunderbird extension in the tests folder in Subversion. Withinthat folder, test suites exist in the content subfolder with a file name that equals the name of the XPCOM component that the test suite tests. Each test suite is included in the test framework by being included as a <script/> line in testrunner.xul.

Running Tests

To run tests, create the Eclipse run configurations for tests and then run those. Those configurations use the "-thunderforceTest Label" command line option in the Thunderforce testing extension to run basic, normal, extra, or all tests. By default, running with the -thunderforceTest option prevents Thunderbird's main user interface from being displayed, but it's possible to force Thunderbird to show with the -thunderforceRunThunderbird command line option. The nsThunderforceTestCommandLine.js file handles the command line options for Thunderforce testing.

Live versus Simulated Testing

By default, Thunderforce's test suite will run against a simulated Salesforce.com API. Note that this simulation is very basic and highly tuned for the tests; it's not a replacement for Salesforce.com and has nowhere near its level of functionality.

When you want to run the tests live against your developer edition organization in Salesforce.com (please don't run the tests using your production organization), follow these steps to configure it:

1. Start Thunderbird normally
2. Open the Thunderbird preferences window
3. Open the "Advanced" tab at the top of the preferences window
4. Ensure that the "General" tab near the middle of the window is selected
5. Click on the "Config Editor..." button
6. Type "thunderforcetest" into the "Filter" input box without the quotes and press the "Enter" key
7. Adjust the following settings to your testing preferences:
   • thunderforcetest.username - Your developer edition organization login username
   • thunderforcetest.password - Your developer edition organization login password. Note that this is saved as clear text in your ThunderforceTest profile's prefs.js file
   • thunderforcetest.email - An email address that you want test email messages to be set to, which applies to tests that send email
   • thunderforcetest.live - When true, the real Salesforce.com is used for testing. When false, a simulated Salesforce.com is used

Creating Tests

To create unit and functionality tests for Thunderforce, please follow the following process:

1. See if the XPCOM component or XUL file that you want to test already has a test in the test extension's content subfolder
   • If a file already exists, then skip to step 3
2. Create a new file in the test extension's content subfolder with a file name that matches the XPCOM component or XUL file that you want to test. The file name must end with .js
   • In this file, add the following contents, renaming nsTheComponentTests to a meaningful name:
     // Tests for your XPCOM component or XUL file (put the name and a light description on this line)
     var nsTheComponentTests = new TestSuite("nsTheComponent");
     
     // Basic tests
     if (runBasicTests) {
     }
     
     // Normal tests
     if (runNormalTests) {
     }
     
     // Extra tests
     if (runExtraTests) {
     }
     
     // Add this test suite to the test runner
     if (nsTheComponentTests.testCase.length > 0) {
         testRunner.add(nsTheComponentTests);
     }
3. Determine what type of test to create
   • Basic test: Simple unit and sanity test cases to ensure that your code change compiles properly and didn't break anything obvious. These are typically run before trying out a new change with the full Thunderforce run configuration
   • Normal test: Normal unit and integration test cases. When tests of this type are written, running this will likely become a check-in requirement
   • Extra test: Integration tests that typically take a long time to run. These tests are important, but do not need to be run with every single change
4. Add a test by creating a new function and adding it to the appropriate code block at the top of the file. As an example, if you are adding a basic test, then you can add the following to the "if (runBasicTests) { }" block: nsTheComponentTests.add(new TestCase("My test case name", testMyNewBasicTestFunction));
   • In the test function's definition, call at least one of the methods in the assertion methods section to assert conditions. Each function requires a name that will appear under the test case in the XULUnit tree
   • When referencing interfaces within Components.interfaces, please check if that interface has a constant declared at the top of baseTest.js. As an example, you can reference nsIProperties instead of having to write out Components.interfaces.nsIProperties. Add other interfaces to this file instead of your test file. When XULUnit is run, all JavaScript files referenced in the XUL file are compiled together. If multiple tests define the same constants, then the JavaScript code will not compile, necessitating the common constants to be defined in one place. The Components.interfaces constants are roughly similar in nature to Java's "import" keyword
5. Try out your test by running it. See the developer startup instructions for Eclipse running and debugging configurations to see how to set up the Eclipse run configurations for tests

Assertion Methods

assert(name, toEval)

• General assertion
• This assertion fails when toEval is false

assertTrue(name, toEval)

• Ensures that toEval is true. The string literal "true" also works
• This assertion fails when toEval is false

assertFalse(name, toEval)

• Ensures that toEval is false. The string literal "false" also works
• This assertion fails when toEval is true

assertEquals(name, value, toEval)

• Ensures that value equals toEval
• The equality matching uses the type-converting == operator instead of ===
• This assertion fails when the expected value does not equal toEval

assertNotEquals(name, value, toEval)

• Ensures that value does not equals toEval
• The equality matching uses the type-converting != operator instead of !==
• This assertion fails when the value equals toEval

assertNull(name, toEval)

• Ensures that toEval is null
• This assertion fails when toEval is not null

assertNotNull(name, toEval)

• Ensures that toEval is not null
• This assertion fails when toEval is null

assertUndefined(name, toEval)

• Ensures that toEval is undefined
• This assertion fails when toEval is not undefined

assertNotUndefined(name, toEval)

• Ensures that toEval is not undefined
• This assertion fails when toEval is undefined

assertNaN(name, toEval)

• Ensures that toEval is NaN (not a number)
• This assertion fails when toEval is a valid number

assertNotNaN(name, toEval)

• Ensures that toEval is not NaN (not a number)
• This assertion fails when toEval is NaN (not a number)

assertRegExp(name, regExp, toEval)

• Ensures that toEval fully matches the regExp regular expression
• This assertion fails when toEval does not fully match the regExp regular expression

assertNotRegExp(name, regExp, toEval)

• Ensures that toEval does not fully match the regExp regular expression
• This assertion fails when toEval fully matches the regExp regular expression

assertTypeOf(name, type, toEval)

• Ensures that the type of toEval is type
• This assertion fails when toEval is not a typeof() type

assertNotTypeOf(name, type, toEval)

• Ensures that the type of toEval is not type
• This assertion fails when toEval is a typeof() type