Set base commit

Summary

Ability to set a base commit for diffs to compare against.

Description

This will allow you to set any SHA to be the base commit. Currently, in a pull request it uses where you branched off as the base commit. This can cause issues if you have a custom workflow.

Examples

In our case, on pull requests we

git checkout master
git fetch origin [branch]
git merge [SHA of branch head]

This means if there are additional commits on master, their coverage will now be part of the coverage diff for this pull.

master a . . b
pull    \ . . c

In this example, your diff will also include all the changes after a on master.

To fix this, I would want to pass the SHA for b to be the base commit for diff.

bash <(curl -s https://codecov.io/bash) --base [COMMIT SHA]

In, my example I would set this to the SHA of b so that I only get my files included in the diff.

This thread also raised the need for this Flag -N to set the base report does not work

2 Likes

Thank you for the request, @scott!

Wanted to respond here to let you know that we’ve seen this and I’ve shared with product for them to review and prioritize. Thank you for making it.

Hello!

We are also running into this issue which makes codecov really hard to use for our use-case as with a simple merge it seems to always pick the wrong commit (generally a commit on the feature branch rather than master)-- for example:

* 005 2020-02-21 | another commit on featurebranch
*   004 2020-02-21 | Merge branch 'master' into featurebranch
|\
| * 003 2020-02-20 | another new commit on master Heller]
| * 002 2020-02-20 | new commit on master 
* | 001 2020-02-21 | first commit on featurebranch 
|/
* 000 2020-02-20 | Master when branched

Codecov will pick 001, when 003 is the correct best master to compare to 001 isn’t even on master, yet codecov tries to use it as a base… For other purposes in our CI pipeline, we use git merge-base HEAD origin/master as a function to determine the best common base of the current head and master (and in github we use a squash and merge strategy to keep master linear). Anyway, hope this gets addressed soon!

Hey y’all, just clarifying a couple things for @scott @zheller and all future readers.

1.) We have a feature to manually pick parents of commits (docs)

2.) We do not have a feature to manually pick base of a PR.

3.) Codecov will always pick the base of the PR as where the branch happened (or prior). If you want to have the head of master as a comparison:

At that point, the only way to get this to work would be merge master into the feature branch so the head commit of master becomes the base commit of the feature branch.

Codecov should correctly show the diff of the feature branch with master merged into it.

Let me know any questions!

Jerrod

Hi @jerrod, thanks for the clarification.

But still, the codecov configuration to automatically discover the base of the PR is producing misleading coverage reports.

As described in the other thread, we allow outdated PRs, but have the Merging the pull request with the current target branch revision jenkins configuration enabled.

Then, the tree that codecov finds, is a bit different:

Under this scenario, codecov is picking automatically “Base commit of little feature”. While this option might look right in some cases, this is producing wrong information for getting the code coverage of the Pull Request.

Why is wrong? Because it’s counting all the new tests added into the master branch in the meantime. So the coverage won’t measure the difference between “commit head” and the “Base commit of little feature”, but the coverage between the virtual commit and the “Base commit of little feature”. This means that an outdated branch doing nothing is going to increase the coverage in the same proportion as it was increased in master since it was opened.

What we want to do by picking ourselves a different base commit for the report, is compare with the actual head of master. This way, the report we see in the PR will represent actually the changes in the PR, and not the rest.

Cheers!

Hey @rolgalan --> thanks for taking the time and energy to make such a clearly demonstrated use case – it’s really helpful.

Just to clarify:

Question 1.) if you could manually pick the base AND head commit of PR which would you select (a. b. c. d., or e.) ?

Question 2.) And what are you seeing Codecov pick in your use case for base AND head commit of PR (a. b. c. d. or e.) (e.g., the wrong behavior)?

Jerrod

Hi @jerrod!

In the example of the Little Feature PR:

  1. I would manually pick commit E to run the tests. Will use commit D to compare the result and attach the report to commit B (using -C parameter).

  2. In this scenario, codecov correctly picks commit E to run the tests, but uses commit A as base and tries to report to commit E (which does not really exist).

Since commit E is a virtual commit not pushed anywhere, when codecov tries to attach the report to it, this gets “lost” (and, for example, it would produce your github checks to fail).

We’ve solved the report commit in this scenario by using the -C parameter in delayed branches, but still have the base commit to compare issue.

Hey @rolgalan thanks again for the patience and clear explanation.

We now fully understand your behavior and what you are seeking. I also understand why you’d want to overlay the feature branch coverage with the most recent commit on Master.

For now:

  • This is not currently supported on Codecov (sorry!)
  • There is not really a good work around short of ALSO running your CI build on commit B (e.g., most recent non-virtual commit on feature branch)

For future:

  • We are considering a YAML feature to allow you to use the most recent commit on default branch (e.g., Commit D) as the base
  • We are considering a larger feature to allow manual base picking, passed during upload. This is harder than our parent commit picking, but still on the horizon.

Sorry I don’t have a better short term answer for you. Will continue to update this thread as we make progress.

Jerrod

Would it work to use the -N flag to set the parent commit in conjunction with setting the base to “parent” (Status Checks) so that the parent commit is used to compare against?

@jerrod, any thoughts on if this would work?

Thanks for checking in on this @WesleyRosenblum and apologies for delay – reviewing with the team and will get you an answer ASAP.

@WesleyRosenblum – ok discussed with the team.

This was a clever insight you had, and one hadn’t fully considered. So, thank you!

Unfortunately, this particular config is pretty legacy for us and wouldn’t fit together with the problem that this thread is seeking to fix. We also are going to clean up the documentation around it for now.

It’s looking increasingly likely that a manual base of PR setting feature will be on the roadmap. Thanks and sorry this wasn’t more fruitful. We really appreciate the consideration, however.

Jerrod

Did the manual base of PR setting feature make it onto the roadmap?

Is there any update on this?

Edit: To clarify, this question is in regards to the manual PR base.

Hi @loganharbour, unfortunately not. We are working through this internally.