Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Split pipeline build job into x86 and x64 #1852

Merged
merged 19 commits into from Jan 20, 2022
Merged

Conversation

lechacon
Copy link
Member

@lechacon lechacon commented Jan 13, 2022

Change:

This changes the CI build pipeline to build/test x86 and x64 in two separate parallel jobs.

Context:

On #1813 the build failed because the build machine was running out of space. We are using Microsoft Hosted agents for running the pipelines, which have only 10GB free space (according to docs; in practice it seems to be around 12GB). Upon investigation, building the solution was already taking about 9GB since we are building both x86 and x64 at the same time. Out of these, about 4GB were from pre-compiled headers; the biggest offender was the one for CommonCore, with over 800MB for each x86,x64.

I also considered trying to reduce the size of the pre-compiled headers or delete them after compilation (but didn't know how); or setting up a custom agent pool with more space (but setting it up and maintaining it seemed complicated). The idea of splitting the jobs by platform is from @yao-msft :)

By splitting into two jobs, building the solution for each one now takes only about 5GB, which puts us a safe distance from the 10GB limit.

Change details:

  • Added a matrix strategy for the build job to run the same steps for both platforms
  • Renamed all build artifacts that contained the platform in the file name to now be grouped under a single folder per platform
  • Removed duplicated test runs (there was one per platform)
  • Moved some files generated by the pipeline from the MSBuild output directory to the pipeline's artifact staging directory
  • Changed tests to run only if the build succeeded by removing the succeededOrFailed() condition. Added continueOnError: true to the tests so that it still makes it to the publish test results step if they fail.
  • Also added publishing of MSBuild binlogs for diagnosing build errors

Opening as draft first to validate that the pipeline works correctly.

Microsoft Reviewers: Open in CodeFlow

@github-actions

This comment has been hidden.

@lechacon lechacon marked this pull request as ready for review Jan 14, 2022
@lechacon lechacon requested a review from as a code owner Jan 14, 2022
azure-pipelines.yml Outdated Show resolved Hide resolved
@lechacon lechacon merged commit 8650938 into microsoft:master Jan 20, 2022
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

3 participants