Stack Overflow Asked by bego on December 7, 2020
I use Azure DevOps Server 2020 with self hosted agents and I created a CI pipeline which executes all my tests in parallel on one agent. The ~5000 tests (no UI tests) need around 7min to complete.Too speed things up I added 3 extra agents, put the task into another job in the same pipeline with parallel: 4. All 4 agents first download the build artifacts and then run a slice of the tests. But unfortunately the test run isn’t faster but instead slower, every agent needs around 8 min.
My vstest yaml for 1 agent
- task: VSTest@2
displayName: 'Run tests'
inputs:
testSelector: 'testAssemblies'
testAssemblyVer2:
***test*.dll
!***TestAdapter.dll
!***TestFramework.dll
!**obj**
searchFolder: '$(System.ArtifactsDirectory)'
runInParallel: true
codeCoverageEnabled: false
rerunFailedTests: false
My vstest yaml for 4 agents
- task: VSTest@2
displayName: 'Run tests'
inputs:
testSelector: 'testAssemblies'
testAssemblyVer2:
***test*.dll
!***TestAdapter.dll
!***TestFramework.dll
!**obj**
searchFolder: '$(System.ArtifactsDirectory)'
runInParallel: true
codeCoverageEnabled: false
distributionBatchType: 'basedOnExecutionTime'
rerunFailedTests: false
I also tried batching by assembly and based on number of tests + number of agents, but nothing seems to work.
What makes it even stranger is that I got an old UI based CI pipeline, with multi-config and multiplier on a variable with 4 TestCategories, which distributes even more tests ~10000 (with my 5000 too) but as mentioned by TestCategory on the same 4 agents (Cat1 on agent1, Cat2 on agent2 and so on) and the slowest one finishes in less than 4 minutes.
The yaml of the UI based one looks like this:
steps:
- task: VSTest@2
displayName: 'Run tests'
inputs:
searchFolder: '$(Build.BinariesDirectory)'
testFiltercriteria: 'TestCategory=$(Tests)'
runInParallel: true
codeCoverageEnabled: false
I think I have to be missing something obvious.
Thanks in advance!
I connected to my agents with RDP and inside the task manager there are multiple instances of testhost.x86 running, up to 8 simultaneously but not constantly. If I start my tests locally the 8+ instances of testhost.x86 are up almost all the time and rarely vanish at all. If that’s any help.
Get help from others!
Recent Questions
Recent Answers
© 2024 TransWikia.com. All rights reserved. Sites we Love: PCI Database, UKBizDB, Menu Kuliner, Sharing RPP