Display data for only affected projects on a mono repo PR

Hello folks,

Related to this Feature Request, we are also running a mono repo and our CI runs a subset of the tests for only affected projects in PR builds. We also use flags to separate reports from different projects.

Our problem is that when Codecov posts data back to a PR, information for all defined flags (for 20+ projects in our mono repo) are included, regardless of whether the project associated with the flag is touched in the PR or not. Is there a way we can hide information for those unaffected projects?

If not, can Codecov have some builtin support for this use case? For example:

  • Codecov omits flags (projects) that has no reports uploaded with it?
  • Codecov provides a way for CI to indicate what the affected projects are for a specific PR?
  • Codecov skips post for flags without coverage change? (This can be inaccurate as coverage remaining the same doesn’t imply the project not getting changed. But it would do for us.)

If you need more details:
Our current repo setup is that:

  • We have defined a flag in codecov.yml, for each of the projects in the repo
  • For a push build, all tests will run, and reports of all projects will be generated and uploaded to Codecov with its corresponding flag
  • For a PR build (branched from a commit on master branch), only tests for affected projects will run, and only those reports are uploaded to Codecov

With this setup, the current behavior we observe is that:

  • PR comment has all defined flags including both affected and unaffected projects
  • Two build status entries (absolute coverage (project), and incremental coverage (patch)) are added to the build status list, for each of the flags
  • For those unaffected projects, their coverage numbers are shown as no change

Hey @yifanyang

Thanks for the well written feature request. And really exciting to hear you are using flags in this way.

The great news is that we are building a feature called Carryforward Flags which will only report on flags / coverage areas in which the CI has run. The remainder of the coverage will “carry forward” from the most recent run.

E.g., for your PR build when only tests for affected projects run, only those reports will update coverage in your repo overview.

This feature is set to release this quarter. If implemented to spec, will it help your issue?

Also, is this an open source repo? If so, can we take a look it to use it for testing and development?

Jerrod

Hello @jerrod,

Really impressed by your quick reply! Thank you!

We will definitely need this feature of Carryforward Flags. However, I’m not sure if it solves my problem at hand.

This PR can probably demonstrate my problem. In the PR, only the project :firebase:dynamic:links (associated with the flag FirebaseDynamicLinks) has some code change. However, the flags section of the comment contains many other flags, although others show no coverage change. Same for the check status list, both flags FirebaseDynamicLinks and FirebaseFunctions (only these two are defined in the coverage.status block in codecov.yml) are posted, while FirebaseFunctions shows no coverage change.

What I want to achieve is that the comment and check status list contain entries only for FirebaseDynamicLinks, the affected project in that PR. Is there a way to make it happen?

Hey @yifanyang – the short answer I believe is, it depends:

We will react to the reports that are actually uploaded from your CI build: here is an example from the commit in the same PR of 34 builds https://codecov.io/gh/yifanyang/firebase-android-sdk/commit/adf5e9c768402496b4a3041081725d32d5d744fd/build.

If the builds are sent, we will use them as the most recent source of truth. If they are not sent via CI, we will go backwards in history to find the most recent build of that name.

If you are asking for us to interpolate line by line what code has changed, that’s not yet in scope, but would be a really cool feature for the future. E.g., we are relying on your subjective submission in the CI.

Jerrod

I guess I wasn’t being clear enough, and my previous example was probably a bit confusing as well… Sorry!

Seems like for a PR, in case of Travis CI, 3 builds may be involved in the coverage calculation here

  • Push build for the PR base commit
  • Push build for the PR head commit
  • PR build for the PR (for the tentative merge commit of master head commit and PR head commit)

This PR is probably more illustrative. Codecov received multiple uploads from this PR. Uploads with build # 65.1 are from the Push build for PR head, build # 66.1 are from the PR build. As you can see, there are 17 uploads with 65.1 and 2 uploads with 66.1, which is a reasonable result by our current mono repo setup: we run all tests in Push builds and only affected tests in PR builds. In our case, only one project FirebaseDynamicLinks is changed in the PR (FirebaseDynamicLinks_Ktx depends on FirebaseDynamicLinks), therefore only relevant reports are uploaded from PR build. And the PR comment from Codecov is totally reasonable.

However, I’m trying to see if it is possible to display information for only changed project (FirebaseDynamicLinks in the previous case). Therefore I did another experiment with this PR. Entirely same setup except that the Push build for PR head is disabled. As you can see there is no check status entry for travis push build on that PR, and Codecov received only 2 uploads, as intended. And this explains the ? associated with other flags, and more files appeared in the “Impacted Files” section. All of this sounds reasonable, and is really close to what I want to achieve.

My questions is: In the latter example, is there a way I can hide flags with ? and impacted files with no coverage number? Also in the build checks list in the example, entries for flags other than FirebaseDynamicLinks (the affected project in the PR) are also added, is there a way to avoid them?

Thanks for bearing with me!

Hi @yifanyang, jumping on this a little late. I just wanted to let you know that Carryforward Flags has shipped and you can find more details here: https://blog.codecov.io/2020/04/21/new-test-only-what-you-change-with-carryforward-flags/

I don’t know if this will completely fix your problem, but I’d love for you to try it out and let us know what pieces we are still missing.