Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support tests sharding in dotnet test #4068

Open
Evangelink opened this issue Nov 15, 2024 · 0 comments
Open

Support tests sharding in dotnet test #4068

Evangelink opened this issue Nov 15, 2024 · 0 comments
Labels
Area: Testing Platform Belongs to the Microsoft.Testing.Platform core library

Comments

@Evangelink
Copy link
Member

Is your feature request related to a problem? Please describe.

We are trying to reduce the time of our CI by splitting the execution between multiple runners.

Describe the solution you'd like

Similar to what we can do in Playwright, I think that implementing sharding in the dotnet test CLI would greatly benefit many projects. One possible API could be based on the previous example:

# The first runner could execute this, running the first fifth of the tests in MyTests.dll
dotnet test "MyTests.dll" --shard 1/5
# Run the second fifth of the tests in MyTests.dll
dotnet test "MyTests.dll" --shard 2/5
# ...
dotnet test "MyTests.dll" --shard 3/5
dotnet test "MyTests.dll" --shard 4/5
dotnet test "MyTests.dll" --shard 5/5

Additional context

We already tried these two solutions that does not completely satisfy us:

Work around the issue by splitting our tests in multiple projects

We can have multiple test projects and configure our CI to split the projects to run between multiple runners. This is not ideal as it forces an architecture upon us, multiplying the projects count for no other purpose than the tests execution.

Keep a single test project and split the tests to run manually

We also tried to implement our own "sharding" by:

  1. Listing all of the tests with the dotnet test --list-tests command;
  2. Split the result in shards in a bash script;
  3. Have one CI runner per shard.

The return of the dotnet test --list-tests command forces us to manipulate it multiple times, removing the headers that cannot be muted and manage the parameterized tests that appears multiple times. Those manipulations are tedious and seem hacky, compared to a native solution provided by the SDK.


Opened by @ThomasFerroAgicap in dotnet/sdk#42986 (comment)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Area: Testing Platform Belongs to the Microsoft.Testing.Platform core library
Projects
None yet
Development

No branches or pull requests

1 participant