Continuous Integration and Performance Testing using JUnit Test Cases

Overview
Testing tools, functional such as Quick Test Pro, and load testing such as Load Runner are expensive, and are shared resources typically managed by a dedicated Quality Assurance team making them difficult to use for continuous integration testing. JUnit test cases can be written to cover a wide range of tests which could be parts of the code, and also end to end transactions, and are extensively used during continuous integration testing. Out of box JMeter Graphical User Interface provides JUnitRequest sampler, and several listeners to store/display the test results. While JMeter’s GUI is easy to use, the same GUI becomes a barrier to automate the testing, and rendering it as not a choice of tool for load testing in the continuous integration testing. One option to overcome this barrier is to create a test plan template, derive other test plans from the template, and run test plans from the command prompt. Another issue is that there is no listener to write the results to a database to record the history of performance results. Results from the out of box Sample Data Writer written to a file can certainly be used to store the test results in a database by writing additional code reading the results file, parsing the results and storing them in a database. A simpler approach writing custom classes to store the results in a database is demonstrated in this blog.

CustomSampler
Noting that JMeter’s test plans could be run as standard batch files or shell scripts or ant targets the focus of this blog was to write custom class, reading test parameters from a properties file rather than the JMeter GUI. The class org.apache.jmeter.protocol.java.sampler.JUnitSampler shown in the following figure was followed to develop the custom class:

Of interest are the highlighted methods in the CustomJUnitSampler class:

1. public void threadStarted()
2. public SampleResult sample(Entry entry)
3. public void threadFinished()

Testing of JUnit test starts within the method threadStarted(), and the method sample(Entry entry) is executed for every test finished, and finally executing the method threadFinished() when the test plan is completed. SampleResult provides several methods to retrieve the test results which include JUnit test method, time at which test was started, test ended, and other performance metrics. This information can then be written to the choice of database. Following code changes were used to store the results in the database.

public class CustomJUnitSampler extends AbstractSampler implements ThreadListener {

. . .

. . .

// Add a member variable to store the test results

private List sampleResults = null;
. . .
. . .
}

public CustomJUnitSampler(){
. . .
. . .
// Initialize sampleResults
sampleResults = new ArrayList();
. . .
. . .
}

public void threadStarted() {
. . .
. . .
// read class name, method name from properties file, or command line arguments
methodName = JMeterUtils.getProperty(METHOD);
className = JMeterUtils.getProperty(CLASSNAME);
. . .
. . .
}

public SampleResult sample(Entry entry) {
. . .
. . .
sampleResults.add(sresult);
. . .
. . .
}

public void threadFinished() {
. . .
. . .
for(SampleResult sampleResult:sampleResults ) {
// use sampleResult to store the data in the database
}
. . .
. . .

}

 
GUI for CustomJUnitSampler
JMeter stores test plans in xml format. Following is the xml element used by the CustomJUnitSampler.<com.ngc.test.perf.jmeter.CustomJUnitSampler guiclass="com.ngc.test.perf.jmeter.control.gui.CustomJUnitSamplerGui" testclass="com.ngc.test.perf.jmeter.CustomJUnitSampler" testname="[res_key=junit_request_new]" enabled="true">

          <stringProp name="junitSampler.classname">com.ngc.opensso.unittests.TestHelloWorld</stringProp>
          <stringProp name="junitsampler.constructorstring"></stringProp>
          <stringProp name="junitsampler.method">testSayHello</stringProp>
          <stringProp name="junitsampler.pkg.filter"></stringProp>
          <stringProp name="junitsampler.success">Test successful</stringProp>
          <stringProp name="junitsampler.success.code">1000</stringProp>
          <stringProp name="junitsampler.failure">Test failed</stringProp>
          <stringProp name="junitsampler.failure.code">0001</stringProp>
          <stringProp name="junitsampler.error">An unexpected error occured</stringProp>
          <stringProp name="junitsampler.error.code">9999</stringProp>
          <stringProp name="junitsampler.exec.setup">false</stringProp>
          <stringProp name="junitsampler.append.error">false</stringProp>
          <stringProp name="junitsampler.append.exception">false</stringProp>
          <boolProp name="junitsampler.junit4">true</boolProp>
</com.ngc.test.perf.jmeter.CustomJUnitSampler>

The attribute guiclass="com.ngc.test.perf.jmeter.control.gui.CustomJUnitSamplerGui" is required to open the test plan in JMeter’s user interface and is not required when the test plan is executed using either the command line arguments or ant scripts but is required when viewing the test plan in JMeter GUI. The GUI class was created based on the sample code from jakarta-jmeter-2.4\src\junit\org\apache\jmeter\protocol\java\control\gui
 

public TestElement createTestElement() {
CustomJUnitSampler sampler = new CustomJUnitSampler();
. . .
}

public void modifyTestElement(TestElement el) {
CustomJUnitSampler sampler = (CustomJUnitSampler)el;
. . .
. . .
}

public void configure(TestElement el) {
. . .
CustomJUnitSampler sampler = (CustomJUnitSampler)el;
. . .
. . .
}

 
Test Parameters
Test plan properties can be read either from properties file, or as command line arguments, using JMeter’s utility tool JMeterUtils.

String myProperty = JMeterUtils.getProperty(“property name”);

 
Batch File
Using Properties File

SET JMETER_HOME=path to jakarta-jmeter-2.4
SET SSL_STORE=Trust Store if the URL requires an SSL
SET TEST_PLAN=Try.jmx
SET PROPERTIES=properties file
%JMETER_HOME%\bin\jmeter -Djavax.net.ssl.trustStore=%SSL_STORE% -t %TEST_PLAN% -n -q %PROPERTIES%

Using Command Line Arguments

SET JMETER_HOME=C:\learning\jakarta-jmeter-2.4
SET SSL_STORE=C:/bea10/user_projects/domains/rdbms_domain/keystore/rdbms_trust.jks
SET TEST_PLAN=Try.jmx
SET CLASSNAME=com.ngc.opensso.unittests.TestHelloWorld
SET METHOD=testSayHello
%JMETER_HOME%\bin\jmeter -Djavax.net.ssl.trustStore=%SSL_STORE% -t %TEST_PLAN% -n -DjunitSampler.classname=%CLASSNAME% -Djunitsampler.method=%METHOD%

 
Ant Target
Jar file ant-jmeter-1.0.9.jar containing ANT task for JMeter can be downloaded from http://www.programmerplanet.org/pages/projects/jmeter-ant-task.php. Following ANT snippets show jmeter taskdef, and the usage of specifying JUnit properties from a properties file or as command line arguments:

<taskdef name="jmeter" classname="org.programmerplanet.ant.taskdefs.jmeter.JMeterTask">
    <classpath>
        <pathelement location="${jmeter.home}/extras/ant-jmeter-1.0.9.jar" />
        <pathelement location="${log4j.home}/log4j-1.2.8.jar" />
       
    </classpath>       
</taskdef>

Using Properties File

<target name="run">
    <jmeter jmeterhome="${jmeter.home}"
        testplan="${basedir}/../loadtests/Try.jmx"
        resultlog="${basedir}/../loadtests/JMeterResults.jtl"
        jmeterproperties="${basedir}/../loadtests/testplan.properties">
        <jvmarg value="-Xincgc"/>
        <jvmarg value="-Xmx128m"/>
        <jvmarg value="-Djavax.net.ssl.trustStore=xxx_trust.jks"/>
    </jmeter>   
</target>

Command Line Arguments

<target name="runn">
    <jmeter jmeterhome="${jmeter.home}"
        testplan="${basedir}/../loadtests/Try.jmx"
        resultlog="${basedir}/../loadtests/JMeterResults.jtl">
        <jvmarg value="-Xincgc"/>
        <jvmarg value="-Xmx128m"/>
        <jvmarg value="-DjunitSampler.classname=JUnit Class Name"/>
        <jvmarg value="-Djunitsampler.method=test method"/>
        <jvmarg value="-Djavax.net.ssl.trustStore=xxx_trust.jks"/>
    </jmeter>   
</target>

Summary
Combined with JMeter, the test cases written for continuous integration can also be used for recording the response time during the integration testing period. With the help of batch files or ANT scripts JUnit test cases can be effectively used to automate performance testing during continuous integration testing.