JUnit 5 - Part I

More than a decade ago I wrote an introduction to JUnit 4, which was – to be quite honest – just a catch-up with the more advanced TestNG. Now JUnit 5 is in the door and it is a complete rewrite, so it’s worth having a fresh look on it. In the first installment of this two part article I will describe what’s new in the basic usage: new asserts, testing exceptions and timing, parameterizing and structuring tests. JUnit 5 comes with a comprehensive user guide, most examples shown in this part are taken from there. So if you already read that, you may skip this and continue directly with the second part, which describes the new extension API that replaces the old runner and rules mechanisms.

One Jar fits all?

Prior to JUnit 5 all parts of the JUnit framework has been packed into one jar: the API to write tests - e.g. assertions - and the API and implementation to actually run tests; for some time, even hamcrest was baked into it. This was a problem since parts of JUnit could not be easily refactored without affecting the tool providers. JUnit 5 is a complete redesign and has been split into a platform and test engines. The platform provides mechanisms for discovering and launching tests, and serves as a foundation for the tool providers (e.g. IDE manufacturers). The platform also defines an SPI for test engines, where those test engines actually define the API to write a test. JUnit 5 provides two engines, the jupiter and the classic engine. The jupiter engine is used to write new JUnit 5 style tests. The classic engine can be used to run your old Junit 4 tests.

So this decouples the tool manufacturers from the test framework providers. This also means, that other test frameworks may be adapted to run on the JUnit platform. You may even write your own test engine, and run your tests in any tool that supports the platform. Currently IntelliJ runs JUnit 5 tests out of the box, other IDEs will follow soon. Also there is support for gradle and maven, and a console runner, so you may start using JUnit right now.

Platform, service providers, engines, a whole bunch of junit jars...holy chihuahua, what do I need to write JUnit tests? Well, to write tests with JUnit 5, you just need the junit-jupiter-api artifact, which defines the JUnit 5 API. It contains the API to write tests and extensions, and this is what we are gonna use in this article. The API in turn is implemented by the junit-jupiter-engine. The current version is 5.0.0-M3, the final release is scheduled for Q3 2017.

Annotations

The first thing to notice is that the new API is in a new namespace org.junit.jupiter.api. Most of the annotation names has been kept, the only one that changed is that Before is now BeforeEach, and BeforeClass is now BeforeAll:

import static org.junit.jupiter.api.Assertions.fail;

import org.junit.jupiter.api.AfterAll; 
import org.junit.jupiter.api.AfterEach; 
import org.junit.jupiter.api.BeforeAll; 
import org.junit.jupiter.api.BeforeEach; 
import org.junit.jupiter.api.Disabled; 
import org.junit.jupiter.api.Test; 

class ClassicTests { 

   @BeforeAll 
   static void setUpAll() { } 

   @BeforeEach 
   void setUp() { } 

   @AfterEach 
   void tearDown() { } 

   @AfterAll 
   static void tearDownAll() { } 

   @Test 
   void succeedingTest() { } 

   @Test 
   void failingTest() { 
      fail("a failing test");
   } 

   @Test 
   @Disabled("for demonstration purposes") 
   void skippedTest() { 
      // not executed 
   } 
}

Asserts

All assertions are still available via static imports, where the enclosing class is now org.junit.jupiter.api.Assertions:

import static org.junit.jupiter.api.Assertions.assertEquals;
...
  assertEquals(2, 2, “the message is now the last argument”);

The message is now the last argument, in order to get the test facts to the pole position. Another improvement is the possibility to create the message lazily using a lambda expression. This avoids unnecessary operations so the test may execute faster:

assertTrue(2 == 2, () -> "This message is created lazily");

Assertions may be grouped using assertAll(). All assertion in the group will  be executed whether they fail or not, and the results are collected.

   @Test 
   void groupedAssertions() { 
      // In a grouped assertion all assertions are executed, and any 
      // failures will be reported together. 
      assertAll("address", 
         () -> assertEquals("John", address.getFirstName()), 
         () -> assertEquals("User", address.getLastName()) 
      ); 
   } 

In JUnit4 exception testing has been done first using the expected property of the @Test annotation. This had the drawback, that you where unable to both inspect the exception and continue after the exception has been thrown. Therefore the ExpectedException rule has been introduced, to come around this flaw. By using the possibilities of lambda expressions, there is now an assertion for testing exceptions which allows to execute a block of code. That code is expected to throw an exception, and also continue the test and inspect that exception. And all that with just some local code:

   @Test 
   void exceptionTesting() { 
      Throwable exception = assertThrows(IllegalArgumentException.class, () -> { 
         throw new IllegalArgumentException("that hurts"); 
      }); 
      assertEquals("that hurts", exception.getMessage()); 
   }

JUnit 5 now also provides some asserts that allows you to time the code under test. This assertion comes in two flavours: the first just measures the time and fails, if the given time has been elapsed. The second is preemptive, means the test fails immediately if the time is up:

   @Test 
   void timeoutExceeded() { 
      // The following assertion fails with an error message similar to: 
      // execution exceeded timeout of 10 ms by 91 ms 
      assertTimeout(ofMillis(10), () -> { 
         // Simulate task that takes more than 10 ms. 
         Thread.sleep(100); 
      });
   } 
   
   @Test 
   void timeoutExceededWithPreemptiveTermination() { 
      // The following assertion fails with an error message similar to: 
      // execution timed out after 10 ms 
      assertTimeoutPreemptively(ofMillis(10), () -> { 
         // Simulate task that takes more than 10 ms. 
         Thread.sleep(100); 
      }); 
   }

Farewell to assertThat()

As already said, JUnit 5 is no longer a one-size-fits-all jar, but has been split up into different responsibilities. The core API now contains just the core API, nothing less, nothing more. Consequently, the baked in hamcrest support has been thrown out. As a replacement just use the bare hamcrest functionality, so all that changes are just some imports:

import static org.hamcrest.CoreMatchers.equalTo; 
import static org.hamcrest.CoreMatchers.is; 
import static org.hamcrest.MatcherAssert.assertThat;

...
   assertThat(2 + 1, is(equalTo(3)));

Assumptions

Some assumptions are gone in JUnit 5, e.g. assumeThat() (see farewell to assertThat()) and assumeNoExceptions(). But you may now use lambda expressions for the condition and the (lazy) message, and there is also a signature that allows you to execute a block of code if the condition matches:

   @Test
   void assumptionWithLambdaCondition() {
      assumeTrue(() -> "CI".equals(System.getenv("ENV")));
      // remainder of test
   }

   @Test
   void assumptionWithLambdaMessage() {
      assumeTrue("DEV".equals(System.getenv("ENV")),
         () -> "Aborting test: not on developer workstation");
      // remainder of test
   }

   @Test
   void assumptionWithCodeBlock() {
      assumingThat("CI".equals(System.getenv("ENV")),
         () -> {
            // perform these assertions only on the CI server
            assertEquals(2, 2);
         });

      // perform these assertions in all environments
      assertEquals("a string", "a string");
   }

Naming, Disabling and Filtering

Test runners use the test method name to visualize the test, so in order to make the result meaningful to us, we used to write XXXL long method names. JUnit 5 allow you to add a @DisplayName annotation that may provide a readable description of the test, which may also contain blanks and all other kind of characters. So you are no longer restricted to the set of characters allowed as java method names:

   @Test 
   @DisplayName("Custom test name containing spaces") 
   void testWithDisplayNameContainingSpaces() { }

The @Ignored annotation has been renamed to @Disabled. You may still provide a reason. Nothing more to say on that

   @Disabled(“This test will be ommited”)
   @Test void testWillBeSkipped() { }

We all use tags to mark content with some metadata in order to find, group or filter the data we are looking for in a certain context. Now tags are supported in JUnit also. You may add one or multiple tags to either a test case, and/ or the complete test class:

   @Test 
   @Tag("acceptance") 
   void testingCalculation() { }

Then you can use these tags to group resp. filter the tests you want to run. Here is a maven example:

<plugin> 
   <artifactId>maven-surefire-plugin</artifactId> 
   <version>2.19</version> 
   <configuration> 
      <properties> 
         <includeTags>acceptance</includeTags> 
         <excludeTags>integration, regression</excludeTags> 
      </properties> 
   </configuration> 
   <dependencies> ... </dependencies> 
</plugin>

Nested Tests

Sometimes it makes sense to organize tests in a hierarchy, e.g. in order to reflect the hierarchy of the structure under test. JUnit 5 allows you to organize tests in nested classes by just marking them with the @Nested annotation. The test discovery will organize these classes in nested test container, which may be visualized reflecting this structure. A core benefit is, that you may also organize the before and after methods hierarchally, means you may define a test setup for a group of nested tests:

...
import org.junit.jupiter.api.Nested;

@DisplayName("A stack")
class TestingAStackDemo {

   Stack<Object> stack;

   @Test
   @DisplayName("is instantiated with new Stack()")
   void isInstantiatedWithNew() {
      new Stack<>();
   }

   @Nested
   @DisplayName("when new")
   class WhenNew {

      @BeforeEach
      void createNewStack() {
         stack = new Stack<>();
      }

      @Test
      @DisplayName("is empty")
      void isEmpty() {
         assertTrue(stack.isEmpty());
      }

      @Test
      @DisplayName("throws EmptyStackException when popped")
      void throwsExceptionWhenPopped() {
         assertThrows(EmptyStackException.class, () -> stack.pop());
      }

      @Test
      @DisplayName("throws EmptyStackException when peeked")
      void throwsExceptionWhenPeeked() {
         assertThrows(EmptyStackException.class, () -> stack.peek());
      }

      @Nested
      @DisplayName("after pushing an element")
      class AfterPushing {

         String anElement = "an element";

         @BeforeEach
         void pushAnElement() {
            stack.push(anElement);
         }

         @Test
         @DisplayName("it is no longer empty")
         void isNotEmpty() {
            assertFalse(stack.isEmpty());
         }

         @Test
         @DisplayName("returns the element when popped and is empty")
         void returnElementWhenPopped() {
            assertEquals(anElement, stack.pop());
            assertTrue(stack.isEmpty());
         }

         @Test
         @DisplayName("returns the element when peeked but remains not empty")
         void returnElementWhenPeeked() {
            assertEquals(anElement, stack.peek());
            assertFalse(stack.isEmpty());
         }
      }
   }
}

You may nest tests arbitrarily deep, but be aware that this works for non-static classes only. Due to this restriction you can not nest @BeforeAll/ @AfterAll annotation since they are static. Let's have a look on how IntelliJ represents those nested test containers:

Dynamic Tests

More than once I had the case that I needed to repeat the same test logic for different contexts, e.g. different parameters. There are certain ways how to manage that, but they all suck. I always wished, I could just generate my tests on the fly for the different context. And finally the JUnit gods must have heard my prayers: create a method that returns either an Iterator or Iterable of DynamicTests, and mark it as @TestFactory:

import static org.junit.jupiter.api.DynamicTest.dynamicTest;
import org.junit.jupiter.api.DynamicTest;
...
class DynamicTestsDemo {

   @TestFactory
   Iterator<DynamicTest> dynamicTestsFromIterator() {
      return Arrays.asList(
         dynamicTest("1st dynamic test", () -> assertTrue(true)),
         dynamicTest("2nd dynamic test", () -> assertEquals(4, 2 * 2))
      ).iterator();
   }

   @TestFactory
   Collection<DynamicTest> dynamicTestsFromCollection() {
      return Arrays.asList(
            dynamicTest("3rd dynamic test", () -> assertTrue(true)),
            dynamicTest("4th dynamic test", () -> assertEquals(4, 2 * 2))
      );
   }

   @TestFactory
   Stream<DynamicTest> dynamicTestsFromIntStream() {
      // Generates tests for the first 10 even integers.
      return IntStream.iterate(0, n -> n + 2).limit(10).mapToObj(
         n -> dynamicTest("test" + n, () -> assertTrue(n % 2 == 0)));
   }
}

Unlike nested tests, the dynamic tests are - at the time of this writing - not organized in nested containers, but run in the context the test factory method. This has some significant flaw, as the before/after lifecycle is not executed for every dynamic test, but only once for the test factory method. Maybe this will change until the final release.

Parameter Resolver

JUnit 5 provides an API to pass parameter to the test constructor and methods, including the before- and after methods. You can define a ParameterResolver, which is responsible for providing parameters of a certain type. Besides passing parameters, this allows also stuff like dependency injection, and builds the foundation for e.g. the Spring- and Mockito-Extensions. Since this is part of the extension mechanism, it will be described in more detail in the second part of this article. By now I will just give you an example using the built-in TestInfo parameter resolver, which will provide you some data on the current test method. Just add TestInfo as a parameter in your test, and JUnit will automatically inject it:

   @Test
   @DisplayName("my wonderful test")
   @Tag("this is my tag")
   void test(TestInfo testInfo) {
      assertEquals("my wonderful test", testInfo.getDisplayName());
      assertTrue(testInfo.getTags().contains("this is my tag"));
   }

Interface Default Methods

Since JUnit 5 has support for composed annotations, you may also use test annotations on interfaces...and also on interface default methods. This allows you to write interface contracts as poor man's mixins instead of abstract base classes. Here is an example for the Comparable interface:

public interface ComparableContract<T extends Comparable<T>> {

   T createValue();
   
   T createSmallerValue();

   @Test
   default void returnsZeroWhenComparedToItself() {
      T value = createValue();
      assertEquals(0, value.compareTo(value));
   }

   @Test
   default void returnsPositiveNumberComparedToSmallerValue() {
      T value = createValue();
      T smallerValue = createSmallerValue();
      assertTrue(value.compareTo(smallerValue) > 0);
   }

   @Test
   default void returnsNegativeNumberComparedToLargerValue() {
      T value = createValue();
      T smallerValue = createSmallerValue();
      assertTrue(smallerValue.compareTo(value) < 0);
   }

}

If we now write a test for a class that implements the Comparable interface, we can inherit all tests provided by ComparableContract. All we have to do is to implement those two methods that provide appropriate values of our class createValue() and createSmallerValue().

public class StringTest implements ComparableContract<String> {

   @Override
   public String createValue() {
      return "foo";
   }

   @Override
   public String createSmallerValue() {
      return "bar"; // "bar" < "foo"
   }

    @Test
    public void someStringSpecificTest() { }

    @Test
    public void anotherStringSpecificTest() { }
}

If we now run the StringTest, both our specific tests and the tests inherited from the ComparableContract.



Ok, that's enough for today. Next week I will explain the JUnit 5 extension mechanism.

Best regards
Ralf

Perhaps a random drawing might be the most impartial way to figure things out.
Jupiter Jones - Jupiter Ascending