Test Distribution

Key Benefits

• Faster test feedback cycles: By distributing text execution to remote agents, test times and overall build times can be reduced by up to 90%.

• Improved DevProd and Developer Experience: Minimizing software testing time makes developers more productive and happier since they spend less idle time waiting for feedback.

• Higher quality software: Fast test feedback cycles mean developers build more often rather than pushing quality checks to later stages of the development cycle.

Key Features

• Uses single-machine parallelism when no agents are available or in combination with remote agents (ensures test distribution will never be slower).

• Requires a simple change to the build configuration to enable.

• Partitions tests sets optimally based on test execution history.

• Works for local and CI builds.

• Show the results of all tests via test reports and in the Build Scan®.

• Makes Information about used agents available in the Build Scan®.

• Transfers and merges outputs back to the build automatically (e.g., JaCoCo).

• Allows running a subset of tests on specific agents based on requirements specified in the build configuration (e.g., certain tests should only run on agents with a specific OS).

• Runs on-premise inside your own network infrastructure.

• Supports auto-scaling remote agents (e.g., via a Kubernetes cluster).

• Resilient against temporary network failures.

• Utilizes remote agents as they become available to perform adaptive scheduling.

Benefits vs. CI fan out and single-machine parallelism

• Test Distribution is available for all local and CI builds regardless of where invoked (e.g., local command line, IDE).

• There is no change to the developer workflow since tests are run automatically in a distributed and transparent manner, instead of run locally.

• The CI user experience is better compared to CI fan out since there is only one build invocation and CI job. If properly implemented, test report aggregation happens as if all tests are run on one agent. There is only one build log.

• Resource consumption is dramatically reduced since the prerequisite build steps for running tests run only once.

• Reliability is increased since efficient fine-grained distribution allows for sequential execution of tests per agent.

• Test Distribution history and dynamic scheduling services can be used to balance the test execution load properly and elastically between test agents and optimize test execution time and compute resource consumption.

• To maximize available resources, Test Distribution supports local test execution fallback if no remote test agents are available or an optional hybrid local/remote execution mode.