Nearest Pull Request Base/PseudoBase Commit not chosen correctly

Description

Nearest Pull Request Base/PseudoBase Commit is not chosen correctly

(This is started in Bug Fixes because this seems like at least the pseudo base is not chosen correctly)

Repository

CI/CD

Github Actions

Uploader

Github Action codecov/codecov-action@v1 (bash uploader)

Commit SHAs

PR Base: 6279619ba0e56475b13c3c1da41d58fe4bc11c06 (detected as base, but has no coverage uploaded)
Nearest with coverage: 3c7d1027137257913b4648525a51703a95f07ac4
First of PR: 718aed7281c25c7198ea068a87b06924d77de8ba
Latest of PR: 04b0ee9c5685107a1db67a6edc19bb3b2762bbc4

Codecov YAML

default, no file

Steps to Reproduce

  • create an pull request on an commit that dosnt have coverage report, but previous commit has
  • create some commits on the pull request with coverage uploaded

Expected behavior: expected is that commit 3c7d1027137257913b4648525a51703a95f07ac4 is chosen as the nearest commit with coverage

Actual behavior: No Base is assigned / wrong pseudo base is chosen, and history since 869d697ded82c105352b1dc5204f4ba8a8255a3c is shown

Flakiness? Happens on all Pull Requests that the base commit does not have an upload

Additional Information

There might be the question on why the base commit dosnt have an coverage report, but this is because we use automated released (semantic release) which will upload an release commit (base in this case), where tests are not run (thanks to this)

Here the Pull Request to see it happening on (Github)
Here the Pull Request to see it happening on (CodeCov)

Here some screenshots to show the current state (because might change later):
CodeCov PR/Commits site:


VSCode git-log extension:

Commits from beta on github (to show where actions are run):

Update:
since then another Pull Request was made, and when trying to go to the compare an 400 error shows up:
PR: https://github.com/nodkz/mongodb-memory-server/pull/415
/compare: https://app.codecov.io/gh/nodkz/mongodb-memory-server/compare/415
/pull (still working): https://codecov.io/gh/nodkz/mongodb-memory-server/pull/415/commits

the /pull link gets referenced in the comment codecov makes with Continue to review full report at Codecov.
the /compare link gets referenced when going to /pulls and clicking on an PR name

PS: it seems like it dosnt always throw this page and actually load the page (that looks almost exactly like the /pull page, but it is empty in every tab (-1 commits))

PPS: it seems like when going to /pulls it sometimes goes to codecov.io and sometimes to apps.codecov.io
Steps i took: going from the github comment to this page and then clicking on Pulls (near the top) (if it should matter, with middle mouse button), tried 10 times, and 2/10 it was going to app.codecov.io

Hi @hasezoey, there are a couple things going on in this thread. I’m going to try to separate them out so that I’m not missing anything

  1. For this pull request, we are attributing the wrong commit SHA as the base. It is showing 6279619ba0e56475b13c3c1da41d58fe4bc11c06 but it should be a parent of 718aed7281c25c7198ea068a87b06924d77de8ba (or 3c7d1027137257913b4648525a51703a95f07ac4)
  2. You are unable to access this compare page. One note, how are you getting to this particular page?
  3. Sometimes you get the compare page above, but it shows -1 Commits
  4. You are sometimes being routed to app.codecov.io.

Is this an accurate representation of things?

Is this an accurate representation of things?

yes

One note, how are you getting to this particular page?

as i explained in the earlier comment, sometimes i get routed to app.codecov.io and when i there click on Pulls (to get here) and when i then click on any Pull Request listed there, it goes to an /compare/PRNumber and not an /pull/PRNumber

For this pull request, we are attributing the wrong commit SHA as the base.

as for that, i also gave an second example PR that has the same issue (or all PR’s where the base has no report uploaded, but the parent one has)


sorry that there are so many things in this thread, i just didnt know how related they are

Hi @hasezoey, I apologize, this got away from me. Would you mind seeing if you are still getting the wrong base SHA on a recent PR? Are you still seeing the other 3 issues?

as for that, it throws me in an infinite-reload loop if visiting that link

this i didnt see anymore

now when clicking Pulls (from here), it now always brings me to this page

as for the PR’s that were already mentioned, nothing changed (no new coverage upload since then for these PR’s), but there are also new PR’s that were made, which still see this issue (or rather an older PR, but got active again) (which is now this PR), but im still seeing that the wrong base is chosen

new PR’s base is 98bc985e5e851b8741a325530038c7a5ddcf19bf (which dosnt have an coverage report, because of an bot-commit)
the commit before that one is 025df2ea32b0b4233c6d2be57646570311c21139 (which has coverage)
but the chosen base commit is still 0ba071abfac93d4abe016ad0ed7c9e3a5235a28f

Hi @hasezoey, apologies, a lot of work has gone in to make our UI more smooth as well as the base picking based on your feedback.

The /pull endpoint should not be working on app.codecov.io, but should be /compare instead. I believe we have patched entrypoints on our end.

Are you still seeing issues with wrong base/pseduo commit? If so, is there a recent case?

from what i can tell, the old PR’s didnt change the commit, and otherwise there has not been an PR with the exact same setup (like the last mentioned ones), but there has been one after rebasing (onto an structure like the other PR’s), and from what i can tell, it definetly chooses an more up-to-date commit now, but still not the newest that could be chosen
the chosen one is b94c14a65132b3731d120c39269511bfd70ba6bb
the base is 9f11fea63f08081a80b58d3182701938091d19e6
the one that could be chosen as newer one is c541b0e83989d6513b334f0e57f4d05035d37878

as an site note, the last coverage for commit 6cdf4823341446c4b7c2f1ed2b97c03992ca29ea is not shown on this page, but this might just be that this report hasnt been processed yet

the current PR im speaking of is:

as for the site issues:
when going to this page and then clicking on Pulls (the one between Branches and Compare) and then on one PR that is listed there (in my test case this page), it gives me the Oops, something went wrong error with code 400 (more detail error is Changes found in between b94c14a...9f11fea (pseudo...base) which prevent comparing this pull request)

@hasezoey, sorry there is a lot going on here, and we’ll have to tackle it slowly. Could you add this to your codecov.yml?

codecov:
  allow_coverage_offsets: true

can i find somewhere what this option does?

and is any branch / PR ok, or must it be in an specific one (like master)?

i added the option to the pr that is currently open, new coverage is uploaded, but base commit didnt change (does this only affect new Pull Requests?).
aside from that, i cant access the PR page in codecov anymore, because of error Changes found in between b94c14a...9f11fea (pseudo...base) which prevent comparing this pull request (Code: 400) [this error was before i added the config option and is also after]

for reference:

as an small update, in the time since i last wrote here, there were some Pull Requests, which based on an release commit (no ci, no coverage), and all were fixated still on one old commit (not the previous to the release commit), but all Pull Requests that didnt base on an release commit were fine

@hasezoey, I just wanted to make sure I was understanding the problem still. New PRs should have viewable PR pages now due to the yaml change. I see that is true here.

Are you still seeing issues with PRs that are based on commits with coverage?

i dont know if this is an typo, but i said “Pull Requests that are based on an commit with coverage are fine (like before too)”, but “Pull Requestes are based on an commit without coverage (like an release commit, no ci, no coverage), still bases on an very old commit”

ie, PR #455 was based on an commit with coverage, unlike #454, which based on an release commit (ie no coverage)

Hi @hasezoey, this request got away from me due to COVID. Are you able to bring up a more recent instance of a PR based on a commit without coverage? Would you also be able to remind me what your typical merge strategy is and in what cases you do not run CI/coverage on your main branch?

There has not been an PR in some time on that project (due to other reason), but the most recent (where it also still happens) is PR #459

Merge is squash / local rebase if the PR is badly structured
otherwise merge is normal non-fast-forward merge (merge commit)

but from my knowledge this shouldnt matter, because this is about the PR before being merged (maybe you asked about the workflow, there its the default behavior)

Release commits made by an github action (without an dedicated key which would allow running workflows, other reasons on why this is not possible)


pretty much my last comment it the current state still, and the project has not been advanced since then (i will report back when there is an PR again with the same conditions)