Accelerate
Troubleshoot
Observe
When we announced Maven support for Develocity in 2019, we opened with “hell has frozen over”. We’ve come a long way since then. Many of our customers are happily using Develocity to make their Maven builds faster and more reliable. In Develocity 2020.5 we’ve added the missing piece to our testing product for Maven: Test Distribution (available for Gradle builds since Develocity 2020.2).
In this blog post, we’re going to connect the build of the Eclipse Jetty open source project to Develocity. We will demonstrate how Test Distribution and Build Caching – with very little effort – reduce the total build time from over 50 minutes to about 15 minutes for a typical code change. We will also discuss potential options for reducing build time even further by optimizing the build for Test Distribution.
The Jetty project contains 145 modules that contain about 450 000 non-comment lines of Java code. After cloning the project, adding the Develocity Maven extension, and configuring it, we are ready to run our first build. As indicated by the Build Scan, the build takes 51m 35s. In the following, we’ll use this build as a baseline and refer to it as (O). The summary screenshot below shows the total execution time along with the slowest goals – all of them test goals.
The timeline in the Build Scan shows that 92% of the total build time (47m 35s) is spent executing tests. For most software projects it is typical for test execution time to dominate overall build time, especially when there are slow integration tests involved. As a result, it has long been an objective of build engineers and developers to speed up test execution.
Maven provides a means to parallelize test execution by running multiple forks of the test JVM. However, the number of JVMs – and hence tests – that can be run in parallel is limited by the build machine’s CPU and memory resources.
Introducing Test Distribution
Test Distribution lets you scale beyond the limitations of a single machine. It takes your existing test suites and distributes them across remote agents to execute them faster — locally and on CI. The Develocity Maven extension enhances the execution of the test and integration-test goals of the Surefire and Failsafe plugins, respectively. Historical test execution data provided by Develocity is used to optimize distribution across remote agents. The tests along with their supporting files are transferred to each agent. While tests are being executed their logging and results are streamed back to the build in real time. If there are no remote agents available, tests can still be executed locally on the build machine. Test Distribution can be enabled for individual goals, or all.
Test Distribution requires use of the JUnit Platform (which is part of JUnit 5 and supports JUnit 4) and tests that are portable. The Develocity Maven Extension User Manual provides more information about making tests portable.
To assess the effectiveness of Test Distribution, we enabled it for all of Jetty’s test goals and ran a build with 10 remote agents. The build (DT1) takes 22m 35s (18m 37s for test goals) instead of the initial 51m 35s (O). When digging a little deeper into the Build Scan, we can see that for most test goals there are a few test classes dominating the overall execution time.
Test Distribution uses test classes as the unit of execution. Hence, a test goal can never be faster than any of its individual test classes. To demonstrate that splitting slow test classes reduces the overall execution time, we did so for 6 of them. When running the build again, the overall build time (DT2) drops to 20m 12s (16m 14s for test goals).
As the above results show, Test Distribution dramatically reduces test execution time, the degree to which depends on the project. Projects with slow test goals tend to benefit more than those with a larger number of faster test goals. Jetty falls into the latter category but even so, we were able to reduce test execution time from 47m 35s to 16m 14s which is a ~3x speedup.
Combining Test Distribution with the Build Cache
Now that we’ve seen Test Distribution in action on its own, we’re ready to enable the Build Cache to see how both capabilities complement each other. To simulate a typical developer use case, we will make a small change to the implementation of a private method in the jetty-client module. Running another build without Test Distribution yields a total execution time of 39m 37s (C1) of which 36m 33s is spent executing tests. Some modules don’t depend on jetty-client so they are not affected by the change and their goals’ outputs can be loaded from cache. Thanks to compile avoidance, even compile goals of dependent modules can be loaded from cache since the application binary interface (ABI), i.e. the visible API, they are compiled against did not change. However, all of their tests have to be run again, since the code on the test runtime classpath differs.
When tests are slow, developers often don’t run them locally and rely on their CI server to eventually notify them of their status. However, this causes a lot of context switching which gets more expensive in terms of productivity the longer the wait (see the Developer Productivity Engineering ebook for a detailed discussion of this topic). This is where Test Distribution comes in and saves the day (or at least the hour).
After making another small non-ABI change in the same place as before, we’re ready to run the build with Build Cache and Test Distribution enabled. This time, the build (C2) takes 14m 50s (11m 50s for tests) which is a 2.7x speedup compared to the build without Test Distribution (C1) and a 3.5x speedup compared to the initial build (O).
Enabling parallel goal execution
Last but not least, Jetty’s build is a large multi-project build (145 modules), which begs the question of how Maven’s multi-threaded mode affects build times. In the results we’ve seen so far, Maven was executed with a single thread (the default) and hence all test goals were executed sequentially. As a consequence, test classes from different goals were not executed in parallel – even when Test Distribution was enabled.
We ran two additional builds while enabling Maven’s multi-threaded mode (via the -T 1C command line argument, i.e. one thread per CPU core). Without Build Cache and Test Distribution, the build takes 35m 27s (OP); with both enabled, it finishes after 10m 49s (C2P). That’s a 3.3x speedup, i.e. similar to comparing single-threaded builds. We can therefore conclude that Test Distribution and the Build Cache work well in both cases.
Conclusion
Code changes in software projects often require large parts of the test suite to be re-executed – even when using the Build Cache. Thus, test execution usually dominates overall build times. Test Distribution accelerates test execution by extending test parallelism from a single machine to many and increases developer productivity by shortening feedback cycles. Unlike other solutions, such as manually partitioning the set of tests into groups for CI builds, it also works for local builds, requires little setup, and produces a single report and Build Scan for all tests.
Test Distribution for Maven is available now. If you’d like to try it, or other Develocity features such as the Build Cache or Build Scans, talk to us about a free trial where we’ll help you evaluate and quantify the benefits of Develocity for your organization.