Related to this Feature Request, we are also running a mono repo and our CI runs a subset of the tests for only affected projects in PR builds. We also use flags to separate reports from different projects.
Our problem is that when Codecov posts data back to a PR, information for all defined flags (for 20+ projects in our mono repo) are included, regardless of whether the project associated with the flag is touched in the PR or not. Is there a way we can hide information for those unaffected projects?
If not, can Codecov have some builtin support for this use case? For example:
Codecov omits flags (projects) that has no reports uploaded with it?
Codecov provides a way for CI to indicate what the affected projects are for a specific PR?
Codecov skips post for flags without coverage change? (This can be inaccurate as coverage remaining the same doesn’t imply the project not getting changed. But it would do for us.)
If you need more details:
Our current repo setup is that:
We have defined a flag in codecov.yml, for each of the projects in the repo
For a push build, all tests will run, and reports of all projects will be generated and uploaded to Codecov with its corresponding flag
For a PR build (branched from a commit on master branch), only tests for affected projects will run, and only those reports are uploaded to Codecov
With this setup, the current behavior we observe is that:
PR comment has all defined flags including both affected and unaffected projects
Two build status entries (absolute coverage (project), and incremental coverage (patch)) are added to the build status list, for each of the flags
For those unaffected projects, their coverage numbers are shown as no change
Thanks for the well written feature request. And really exciting to hear you are using flags in this way.
The great news is that we are building a feature called Carryforward Flags which will only report on flags / coverage areas in which the CI has run. The remainder of the coverage will “carry forward” from the most recent run.
E.g., for your PR build when only tests for affected projects run, only those reports will update coverage in your repo overview.
This feature is set to release this quarter. If implemented to spec, will it help your issue?
Also, is this an open source repo? If so, can we take a look it to use it for testing and development?
We will definitely need this feature of Carryforward Flags. However, I’m not sure if it solves my problem at hand.
This PR can probably demonstrate my problem. In the PR, only the project :firebase:dynamic:links (associated with the flag FirebaseDynamicLinks) has some code change. However, the flags section of the comment contains many other flags, although others show no coverage change. Same for the check status list, both flags FirebaseDynamicLinks and FirebaseFunctions (only these two are defined in the coverage.status block in codecov.yml) are posted, while FirebaseFunctions shows no coverage change.
What I want to achieve is that the comment and check status list contain entries only for FirebaseDynamicLinks, the affected project in that PR. Is there a way to make it happen?
If the builds are sent, we will use them as the most recent source of truth. If they are not sent via CI, we will go backwards in history to find the most recent build of that name.
If you are asking for us to interpolate line by line what code has changed, that’s not yet in scope, but would be a really cool feature for the future. E.g., we are relying on your subjective submission in the CI.
I guess I wasn’t being clear enough, and my previous example was probably a bit confusing as well… Sorry!
Seems like for a PR, in case of Travis CI, 3 builds may be involved in the coverage calculation here
Push build for the PR base commit
Push build for the PR head commit
PR build for the PR (for the tentative merge commit of master head commit and PR head commit)
This PR is probably more illustrative. Codecov received multiple uploads from this PR. Uploads with build # 65.1 are from the Push build for PR head, build # 66.1 are from the PR build. As you can see, there are 17 uploads with 65.1 and 2 uploads with 66.1, which is a reasonable result by our current mono repo setup: we run all tests in Push builds and only affected tests in PR builds. In our case, only one project FirebaseDynamicLinks is changed in the PR (FirebaseDynamicLinks_Ktx depends on FirebaseDynamicLinks), therefore only relevant reports are uploaded from PR build. And the PR comment from Codecov is totally reasonable.
However, I’m trying to see if it is possible to display information for only changed project (FirebaseDynamicLinks in the previous case). Therefore I did another experiment with this PR. Entirely same setup except that the Push build for PR head is disabled. As you can see there is no check status entry for travis push build on that PR, and Codecov received only 2 uploads, as intended. And this explains the ? associated with other flags, and more files appeared in the “Impacted Files” section. All of this sounds reasonable, and is really close to what I want to achieve.
My questions is: In the latter example, is there a way I can hide flags with ? and impacted files with no coverage number? Also in the build checks list in the example, entries for flags other than FirebaseDynamicLinks (the affected project in the PR) are also added, is there a way to avoid them?
I’d like to chip in to the original request. We are using the Carryforward Flags on our monorepo as well. We don’t have that many sub-repos, so the PR comments generated by Codecov are not too bad. However, it would greatly improve clarity if only affected flags were reported (e.g. not the ones that were carried forward and not the ones that were not used in any uploads in that PR).
Basically, flags that are carried forward will not automatically show in the comment, and you can adjust the settings to exclude carried forward status checks.