Useful regular expressions in Eclipse

Eclipse understands Java, and Javascript, so searching for things such as function usage or definitions is quite easy. Unfortunately, when you throw in some JSP, and some unwieldy code, there are some things that are slightly harder to find. In particular, two bugbears are:

  • Dangling commas, that cause IE to break
  • Stray instances of console.log, used in debugging that cause IE to break

The easiest way to find these is often with a regular expression. For the first, here is a reasonable regex:

,\s*(]|})

For the second, here is another regex (thanks to http://stackoverflow.com/questions/5374843/regex-to-find-an-uncommented-println):

(?m)^((?!//|/\*).)*console\.log.*

Both are quite naive – they give some false positives. But it’s a start. If anyone wants to suggest anything better, please use the boxes below.

Web applications, continuous integration and unit testing

Not a particularly exciting title, but having spent a week trying to achieve this, I was determined to explain how I did it.  Partly as a reference to myself, for when I need to make some changes 6 months down the line, but also for anyone else who’s been banging their head against this.

I’ve recently started using Jenkins for continuous integration on a couple of java projects. Basically, Jenkins contacts my software repository (svn, in this case), and every time I update the code, it builds the code and runs some basic JUnit tests on it to ensure I’ve not broken anything important.  The standard way to define build scripts is now Maven, and despite only the vaguest notion of how it works, I decided to plough on.

Pleased with the application of Jenkins to a single jar file, I immediately wanted to use it for something more complicated.  I’m currently employed to build and maintain a fairly substantial web application, which is deployed in a domain where code quality is important.  So I decided to try and set up a Jenkins project for this, too.

Selenium is a great tool for testing web-based code, and I’ve recently been impressed by Cobertura as a tool for measuring test-coverage.  So the plan was to incorporate Selenium and Cobertura into my build cycle.  This was more complicated to achieve than I might have imagined.  Part of the trouble was the lack of decent documentation and examples.  As is so often the case, the best sources of information were to be found in message boards, blogs, and bug trackers.  It was clear that other people were having trouble getting the same setup, and in a few cases, some people even claimed this kind of setup was impossible.  My main sticking point was my lack of Maven understanding, and experience, and a particular frustration was that where snippets of Maven were posted, there wasn’t information to use them in my own scripts.

So here I’ll explain the setup of the machine, along with excerpts of Maven and hints as to where I’ve been up many a blind alley.  At the end of the post, I’ll paste the whole Maven pom, which those with similar inexperience with Maven should find invaluable.

System set up

Here’s a quick summary of my build server configuration.  Nothing about the build server is particularly difficult to set up, but a tick-list is always helpful.  I used an Ubuntu machine, and on it I installed:

  • Jenkins  Note that this runs most stably as a stand-alone application (as default), rather than placing in the webapps directory of tomcat.
  • Xvfb Virtual Framebuffer for running a browser without popping up on screen (or for headless servers)
  • ImageMagick (use apt-get to install it – useful for screen-grabbing, to debug or test Xvfb).
  • Google Chrome  For various reasons I’ll go into later, I couldn’t get this to work with Firefox.  But Google Chrome works a treat.
In addition, Google Chrome needs to run on a particular display port (for xvfb).  Here is a script that will achieve this – call it ~/google-chrome.sh
DISPLAY=:20 /usr/bin/google-chrome $@

 Maven

Here are some excerpts from my Maven script.

This is the start of my Maven script.  Nothing important here, other than the fact that we’re building a war, and we’re defining a version of jetty.

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>myWebApp</groupId>
  <artifactId>myWebApp</artifactId>
  <packaging>war</packaging>
  <version>1.0-SNAPSHOT</version>
  <name>My Maven Webapp</name>
  <url>http://maven.apache.org</url>
  <properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <jetty.version>7.2.2.v20101205</jetty.version>
  </properties>

Most of my dependencies are specific to my web app.  But here are some that may be integral to this whole thing working…

<dependencies>
   <dependency>
    <groupId>javax</groupId>
    <artifactId>javaee-api</artifactId>
    <version>6.0</version>
  </dependency>
  <dependency>
    <groupId>junit</groupId>
    <artifactId>junit</artifactId>
    <version>3.8.2</version>
    <scope>test</scope>
  </dependency>
  <dependency>
    <groupId>org.seleniumhq.selenium.client-drivers</groupId>
    <artifactId>selenium-java-client-driver</artifactId>
    <version>1.0.2</version>
    <scope>test</scope>
  </dependency>
  <dependency>
    <groupId>org.seleniumhq.selenium</groupId>
    <artifactId>selenium-firefox-driver</artifactId>
    <version>2.15.0</version>
  </dependency>
  <dependency>
    <groupId>org.codehaus.mojo</groupId>
    <artifactId>cobertura-maven-plugin</artifactId>
    <version>2.5.1</version>
  </dependency>
</dependencies>

Here’s a plugin repository.  I don’t know whether that’s needed or not, to be honest.

<pluginRepositories>
  <pluginRepository>
    <id>codehaus snapshot repository</id>
    <url>http://snapshots.repository.codehaus.org/</url>
    <releases>
      <enabled>true</enabled>
    </releases>
  </pluginRepository>
</pluginRepositories>

Now we get to the meat of what’s going on.  The build cycle is as follows:

  • Before packaging, Maven would normally run some tests.  However, we can’t really test our code until it’s been packaged and deployed, so we do those first, and tell Maven not to run tests as normal
  • First we build a war file – Maven takes care of this without any prompting.
  • Cobertura is our tool for analysing code-coverage of tests.  In order for Cobertura to do its thing, it needs to know the extent of the code, and it needs to instrument class files.  JSPC is a tool that will pre-compile our war file in order to achieve this.  It will also modify the web.xml file.
  • Cobertura can then look through the compiled war file and instrument the files – producing a new set of class files
  • These instrumented class files then need to be included in the war file to be deployed.
  • Selenium RC needs to be started on a particular port for running tests
  • Xvfb needs to be started in order for a browser to run tests
  • Jetty needs to be started (on a port that Jenkins is not using), and the packaged war file deployed
  • Selenium tests (in the form of Selenese – an html-like language for describing web interactions and assertions) need to be run in the ‘integration-test’ phase of the build. It will use the google-chrome script written earlier to run headlessly.
  • After the tests have been run, Selenium, Xvfb and Jetty should close themselves down.
  • Finally, Cobertura should generate a report on the classes used in the tests, and place the results somewhere Jenkins can find them.

To start with, here’s the plugin necessary to run the pre-compiling using JSPC.  It runs in the ‘prepare-package’ phase, and will copy the new web.xml instructions into the existing web.xml file, replacing the comment given.

<build>
  <plugin>
    <groupId>org.mortbay.jetty</groupId>
    <artifactId>maven-jetty-jspc-plugin</artifactId>
    <executions>
      <execution>
        <id>jspc</id>
        <goals>
          <goal>jspc</goal>
        </goals>
        <phase>prepare-package</phase>
        <configuration>
          <verbose>true</verbose>
          <insertionMarker><!-- [INSERT FRAGMENT HERE] --></insertionMarker>
        </configuration>
      </execution>
    </executions>
  </plugin>

Here’s the plugin for running Cobertura.  It is possible to instrument, report and check in one goal, but this relies on the tests being run in the ‘test’ phase.  So we’ll just instrument here.  We want reports to be later generated in xml and html though.

<plugin>
  <groupId>org.codehaus.mojo</groupId>
  <artifactId>cobertura-maven-plugin</artifactId>
  <version>2.5.1</version>
  <configuration>
    <formats>
      <format>xml</format>
      <format>html</format>
    </formats>
  </configuration>
  <executions>
    <execution>
      <id>instrument-code</id>
      <phase>prepare-package</phase>
      <goals>
        <goal>instrument</goal>
      </goals>
      <configuration>
        <attach>true</attach>
      </configuration>
    </execution>
  </executions>
</plugin>

Here is the plugin for generating the war file.  We want the war file to include the instrumented jsp classes, and we want to use the new web.xml file.

<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-war-plugin</artifactId>
<version>2.1.1</version>
<configuration>
<webResources>
<resource>
<!-- this is relative to the pom.xml directory -->
<directory>target/generated-classes/cobertura/jsp</directory>
<targetPath>WEB-INF/classes/jsp</targetPath>
</resource>
</webResources>
<webXml>target/web.xml</webXml>
</configuration>
</plugin>

Next is the plugin for starting and stopping the jetty container.  This uses the Cargo plugin – I tried others but without success.  There are two executions – one to start the container in the pre-integration-test phase, and one to stop in the post-integration-test phase.

<plugin>
  <groupId>org.codehaus.cargo</groupId>
  <artifactId>cargo-maven2-plugin</artifactId>
  <version>1.1.0</version>
  <configuration>
    <wait>false</wait>
    <container>
      <containerId>jetty7x</containerId>
      <artifactInstaller>
        <groupId>org.eclipse.jetty</groupId>
        <artifactId>jetty-distribution</artifactId>
        <version>${jetty.version}</version>
      </artifactInstaller>
    </container>
    <configuration>
      <properties>
        <cargo.servlet.port>8081</cargo.servlet.port>
      </properties>
    </configuration>
  </configuration>
  <executions>
    <execution>
      <id>start-container</id>
      <phase>pre-integration-test</phase>
      <goals>
        <goal>start</goal>
        <goal>deploy</goal>
      </goals>
    </execution>
    <execution>
      <id>stop-container</id>
      <phase>post-integration-test</phase>
      <goals>
        <goal>stop</goal>
      </goals>
    </execution>
  </executions>
</plugin>

Next is the complicated plugin.  This starts the selenium server, and Xvfb before integration tests, and stops it afterwards.  It runs the selenese tests, using Google Chrome, and the script provided.

<plugin>
  <groupId>org.codehaus.mojo</groupId>
  <artifactId>selenium-maven-plugin</artifactId>
  <version>2.1</version>
  <dependencies>
    <dependency>
      <groupId>org.seleniumhq.selenium</groupId>
      <artifactId>selenium</artifactId>
      <version>2.0rc2</version>
      <type>pom</type>
      <exclusions>
        <!-- prevent ant:ant versus org.apache.ant:ant collision -->
        <exclusion>
          <groupId>ant</groupId>
          <artifactId>ant</artifactId>
        </exclusion>
      </exclusions>
    </dependency>
    <dependency>
      <groupId>org.seleniumhq.selenium</groupId>
      <artifactId>selenium-server</artifactId>
      <version>2.3.1</version>
      <exclusions>
        <exclusion>
          <groupId>org.testng</groupId>
          <artifactId>testng</artifactId>
        </exclusion>
      </exclusions>
    </dependency>
    <dependency>
      <groupId>org.seleniumhq.selenium</groupId>
      <artifactId>selenium-chrome-driver</artifactId>
      <version>2.3.1</version>
    </dependency>
  </dependencies>
  <executions>
    <execution>
      <id>xvfb</id>
      <phase>pre-integration-test</phase>
      <goals>
        <goal>xvfb</goal>
      </goals>
    </execution>
    <execution>
      <id>selenium</id>
      <phase>pre-integration-test</phase>
      <goals>
        <goal>start-server</goal>
      </goals>
      <configuration>
        <background>true</background>
      </configuration>
    </execution>
    <execution>
      <id>integration-test</id>
      <phase>integration-test</phase>
      <goals>
        <goal>selenese</goal>
      </goals>
      <configuration>
        <browser>*googlechrome /home/james/google-chrome.sh</browser>
        <suite>selenium/testsuite.html</suite>
        <startURL>http://localhost:8081/myWebApp/</startURL>
        <port>4455</port>
        <results>${project.build.directory}/results/selenium-results.html</results>
      </configuration>
    </execution>
    <execution>
      <id>stop</id>
      <phase>post-integration-test</phase>
      <goals>
        <goal>stop-server</goal>
      </goals>
    </execution>
  </executions>
</plugin>

The next plugin deals with the tests.  In the test phase of the Maven build, we’ll ignore the regular tests.  In the integration-test phase, we’ll run any tests in the selenium class.  These can be java-encoded selenium tests which connect to the same jetty deployable, the same selenium server, and use the same google-chrome browser.

<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-surefire-plugin</artifactId>
  <version>2.8.1</version>
  <configuration>
    <excludes>
      <exclude>**/selenium/*.java</exclude>
    </excludes>
  </configuration>
  <executions>
    <execution>
      <id>integration-tests</id>
      <phase>integration-test</phase>
      <goals>
        <goal>test</goal>
      </goals>
      <configuration>
        <skip>false</skip>
        <excludes>
          <exclude>**/softeng/*.java</exclude>
        </excludes>
        <includes>
          <include>**/selenium/*.java</include>
        </includes>
      </configuration>
    </execution>
  </executions>
</plugin>

The final plugin deals with the cobertura report.  Although there is mention of a patch on the Cobertura forums, there is currently no goal in the Cobertura plugin for simply reporting.  There is an ant task to do just that though, so we’ll use that.

<plugin>
  <artifactId>maven-antrun-plugin</artifactId>
  <dependencies>
    <dependency>
      <groupId>net.sourceforge.cobertura</groupId>
      <artifactId>cobertura</artifactId>
      <version>1.9</version>
    </dependency>
  </dependencies>
  <executions>
    <execution>
      <phase>post-integration-test</phase>
      <id>cobertura-report</id>
      <configuration>
        <tasks>
          <taskdef classpathref="maven.runtime.classpath"
            resource="tasks.properties" />
          <mkdir dir="${project.build.directory}/site/cobertura" />
          <cobertura-report format="xml"
            datafile="${project.build.directory}/cobertura/cobertura.ser"
            destdir="${project.build.directory}/site/cobertura">
          </cobertura-report>
          <cobertura-report format="html"
            datafile="${project.build.directory}/cobertura/cobertura.ser"
            destdir="${project.build.directory}/site/cobertura">
          </cobertura-report>
        </tasks>
      </configuration>
      <goals>
        <goal>run</goal>
      </goals>
    </execution>
  </executions>
</plugin>

That’s pretty much it for the contents of the Maven pom.  A complete copy is pasted at the end for completeness.

Jenkins setup

As for my Jenkins setup, I’ve used the following plugins:

  • Maven Integration plugin
  • Jenkins Subversion Plug-in
  • Jenkins Cobertura Plugin
  • Jenkins Artifactory Plugin
  • Selenium HTML report
Here is a screenshot of my Jenkins project configuration.  To be honest, this was probably the easiest bit of the whole lot!

Pom

And, as promised, here’s the complete (although edited) pom for those who cannot deal with out-of-context snippets of pom (such as myself).

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>myWebApp</groupId>
  <artifactId>myWebApp</artifactId>
  <packaging>war</packaging>
  <version>1.0-SNAPSHOT</version>
  <name>My Maven Webapp</name>
  <url>http://maven.apache.org</url>
  <properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <jetty.version>7.2.2.v20101205</jetty.version>
  </properties>

  <repositories>
    ...
  </repositories>

  <dependencies>
    <dependency>
      <groupId>javax</groupId>
      <artifactId>javaee-api</artifactId>
      <version>6.0</version>
    </dependency>
    <dependency>
      <groupId>junit</groupId>
      <artifactId>junit</artifactId>
      <version>3.8.2</version>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>org.seleniumhq.selenium.client-drivers</groupId>
      <artifactId>selenium-java-client-driver</artifactId>
      <version>1.0.2</version>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>org.seleniumhq.selenium</groupId>
      <artifactId>selenium-firefox-driver</artifactId>
      <version>2.15.0</version>
    </dependency>
    <dependency>
      <groupId>org.codehaus.mojo</groupId>
      <artifactId>cobertura-maven-plugin</artifactId>
      <version>2.5.1</version>
    </dependency>
    ...
  </dependencies>

  <pluginRepositories>
    <pluginRepository>
      <id>codehaus snapshot repository</id>
      <url>http://snapshots.repository.codehaus.org/</url>
      <releases>
        <enabled>true</enabled>
      </releases>
    </pluginRepository>
  </pluginRepositories>

  <build>
    <finalName>myWebApp</finalName>
    <plugins>
      <plugin>
        <groupId>org.mortbay.jetty</groupId>
        <artifactId>maven-jetty-jspc-plugin</artifactId>
        <version>6.1-SNAPSHOT</version>
        <executions>
          <execution>
            <id>jspc</id>
            <goals>
              <goal>jspc</goal>
            </goals>
            <phase>prepare-package</phase>
            <configuration>
              <insertionMarker><!-- [INSERT FRAGMENT HERE] --></insertionMarker>
            </configuration>
          </execution>
        </executions>
      </plugin>
      <plugin>
        <groupId>org.codehaus.mojo</groupId>
        <artifactId>cobertura-maven-plugin</artifactId>
        <version>2.5.1</version>
        <configuration>
          <formats>
            <format>xml</format>
            <format>html</format>
          </formats>
        </configuration>
        <executions>
          <execution>
            <id>instrument-code</id>
            <phase>prepare-package</phase>
            <goals>
              <goal>instrument</goal>
            </goals>
            <configuration>
              <attach>true</attach>
            </configuration>
          </execution>
        </executions>
      </plugin>

      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-war-plugin</artifactId>
        <version>2.1.1</version>
        <configuration>
          <webResources>
            <resource>
              <!-- this is relative to the pom.xml directory -->
              <directory>target/generated-classes/cobertura/jsp</directory>
              <targetPath>WEB-INF/classes/jsp</targetPath>
            </resource>
          </webResources>
          <webXml>target/web.xml</webXml>
        </configuration>
      </plugin>
      <plugin>
        <groupId>org.codehaus.cargo</groupId>
        <artifactId>cargo-maven2-plugin</artifactId>
        <version>1.1.0</version>
        <configuration>
          <wait>false</wait>
          <container>
            <containerId>jetty7x</containerId>
            <artifactInstaller>
              <groupId>org.eclipse.jetty</groupId>
              <artifactId>jetty-distribution</artifactId>
              <version>${jetty.version}</version>
            </artifactInstaller>
          </container>
            <port>8081</port> </connector> </connectors> -->
          <configuration>
            <properties>
              <cargo.servlet.port>8081</cargo.servlet.port>
            </properties>
          </configuration>
        </configuration>

        <executions>
          <execution>
            <id>start-container</id>
            <phase>pre-integration-test</phase>
            <goals>
              <goal>start</goal>
              <goal>deploy</goal>
            </goals>
          </execution>
          <execution>
            <id>stop-container</id>
            <phase>post-integration-test</phase>
            <goals>
              <goal>stop</goal>
            </goals>
          </execution>
        </executions>
        <dependencies></dependencies>
      </plugin>

      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-surefire-plugin</artifactId>
        <version>2.8.1</version>
        <configuration>
          <excludes>
            <exclude>**/selenium/*.java</exclude>
          </excludes>
        </configuration>
        <executions>
          <execution>
            <id>integration-tests</id>
            <phase>integration-test</phase>
            <goals>
              <goal>test</goal>
            </goals>
            <configuration>
              <skip>false</skip>
              <excludes>
                <exclude>**/softeng/*.java</exclude>
              </excludes>
              <includes>
                <include>**/selenium/*.java</include>
              </includes>
            </configuration>
          </execution>
        </executions>
      </plugin>

      <plugin>
        <groupId>org.codehaus.mojo</groupId>
        <artifactId>selenium-maven-plugin</artifactId>
        <version>2.1</version>
        <dependencies>
          <dependency>
            <groupId>org.seleniumhq.selenium</groupId>
            <artifactId>selenium</artifactId>
            <version>2.0rc2</version>
            <type>pom</type>
            <exclusions>
              <!-- prevent ant:ant versus org.apache.ant:ant collision -->
              <exclusion>
                <groupId>ant</groupId>
                <artifactId>ant</artifactId>
              </exclusion>
            </exclusions>
          </dependency>
          <dependency>
            <groupId>org.seleniumhq.selenium</groupId>
            <artifactId>selenium-server</artifactId>
            <version>2.3.1</version>
            <exclusions>
              <exclusion>
                <groupId>org.testng</groupId>
                <artifactId>testng</artifactId>
              </exclusion>
            </exclusions>
          </dependency>
          <dependency>
            <groupId>org.seleniumhq.selenium</groupId>
            <artifactId>selenium-chrome-driver</artifactId>
            <version>2.3.1</version>
          </dependency>

        </dependencies>
        <executions>
          <execution>
            <id>xvfb</id>
            <phase>pre-integration-test</phase>
            <goals>
              <goal>xvfb</goal>
            </goals>
          </execution>

          <execution>
            <id>selenium</id>
            <phase>pre-integration-test</phase>
            <goals>

              <goal>start-server</goal>
            </goals>
            <configuration>
              <background>true</background>
            </configuration>
          </execution>
          <execution>
            <id>integration-test</id>
            <phase>integration-test</phase>
            <goals>
              <goal>selenese</goal>
            </goals>
            <configuration>
              <browser>*googlechrome /home/james/google-chrome.sh</browser>
              <suite>selenium/testsuite.html</suite>
              <startURL>http://localhost:8081/myWebApp/</startURL>
              <port>4455</port>
              <results>${project.build.directory}/results/selenium-results.html</results>
            </configuration>
          </execution>
          <execution>
            <id>stop</id>
            <phase>post-integration-test</phase>
            <goals>
              <goal>stop-server</goal>
            </goals>
          </execution>
        </executions>
      </plugin>

      <plugin>
        <artifactId>maven-antrun-plugin</artifactId>
        <dependencies>
          <dependency>
            <groupId>net.sourceforge.cobertura</groupId>
            <artifactId>cobertura</artifactId>
            <version>1.9</version>
          </dependency>
        </dependencies>
        <executions>
          <execution>
            <phase>post-integration-test</phase>
            <id>cobertura-report</id>
            <configuration>
              <tasks>
                <taskdef classpathref="maven.runtime.classpath"
                  resource="tasks.properties" />
                <mkdir dir="${project.build.directory}/site/cobertura" />
                <cobertura-report format="xml"
                  datafile="${project.build.directory}/cobertura/cobertura.ser"
                  destdir="${project.build.directory}/site/cobertura">
                </cobertura-report>
                <cobertura-report format="html"
                  datafile="${project.build.directory}/cobertura/cobertura.ser"
                  destdir="${project.build.directory}/site/cobertura">
                </cobertura-report>
              </tasks>
            </configuration>
            <goals>
              <goal>run</goal>
            </goals>
          </execution>
        </executions>
      </plugin>
    </plugins>
  </build>
</project>