In the last few days, since the most recent changes to Codecov, it appears that the final coverage data is not being updated in GItHub PRs.
On the codecov website, the coverage percent shows correctly, but on the codecov comment in the PR it consistently shows a coverage percent that is less than the codecov webpage.
Repository
zlib-ng project on github.
I have a 4 link maximum imposed by this forum, otherwise I would post the link.
CI/CD
GitHub Actions
Uploader
codecov-python
Commit SHAs
GitHub comment shows 75.89
Codecov website shows 79.21
GitHub comment shows 74.42
Codecov website shows 79.28
There are more examples if you just examine my other PR in that same project.
Codecov Output
There are many CI instances which run per PR.
Steps to Reproduce
Fork project repository
Create pull request
GitHub comment by codecov bot will show different coverage percent than Codecov website.
Expected behavior: The coverage percentages to match.
Actual behavior: The coverage percentages do not match.
Flakiness? Every time
Additional Information
This only started happening maybe within this last week.
I have to maintain my own fork because some important PRs are not getting into the main repository especially w/r to GitLab. PRs include #187, #208, #216, Also we had to rework the retry mechanism because it failed too often.
I have rebased to the latest master and reworked all the PRs I had to commit so they conform to your new formatting.
I think I might know what it is from. In the PR it is showing stats from my fork and not from the main repository. In my fork I might not have the same codecov yaml.
But it is curious why it is taking from my fork and not from the main repository PR. I have updated my code that detects GitHub Actions (from bash uploader) to see if that might be it and submitted a PR for it.
No, it seems like that was not the case. Although there is a difference in coverage percentages between my fork (of zlib-ng) and the mainline (zlib-ng), it is not enough to account for the percentage that the codecov commenting bot is reporting. Also the codecov comment on the PR has links to the coverage report for mainline (zlib-ng).
When you look at this PR, the comment show 72% but if you click into the report it shows 78%.
So it still looks like that the comments are not being updated at the end. We have a unique situation with a lot of CI instances.
@tom Could this be codecov commenting system hitting a ratelimiter on updating github comments, causing it to be unable to update it again?
The really weird thing is it will not update it again if you do a force push, but it will if you delete the old comment first. If github has a ratelimiter, that would mean it either blocks them permanently, or codecov gives up on updating it after it fails.
This used to work but suddenly stopped working, we did not change the codecov yaml config in that period.
Also, have a look at the comment it the above linked PR and notice it does not list reports for all the flags, even though the PR viewed at the codecov site clearly has them. The number of āflagsā/reports listed in the comment is usually around 5-20, and it used to be all of them.
A big clue here might be my last PR, it did not get a codecov comment report at all. See: Remove toofar checks by Dead2 Ā· Pull Request #597 Ā· zlib-ng/zlib-ng Ā· GitHub But if you view the codecov entries under āchecksā, it looks perfectly in line with HEAD.
So clearly codecov somehow failed to make the comment, and Iād think it would retry once for each report it gets in, and since we have a lot of CI runs we submit a lot of reports.
This PR was made from the same repo, so no external repo involved at all.
Regarding the pull request that you linked to here, Iām staring at the logs and Iām convinced that itās a GitHub hiccup. It is very unlikely that itās a rate limit issue, but if you run the CI/CD again, I think it will go through fine.
As for this PR, you mentioned a yaml file. Iām actually not able to find one in the repo. I also canāt see us pulling one down either. I am, however, seeing the same issue with GitHub being unreachable. Iād like to see if this is a one-time issue or itās consistent. Have you seen this happening since?
This used to happen before too (we had lots of problems before, making codecov completely useless), but everything was pretty much fixed in early March (most comment reports were complete, just a few were missing a couple of the CI runs).
Ref: Codecov server 500 error and endless "processing" - #8 by Dead2
But after 2-4 weeks, the comment problem came back again in full force.
Oh, and the yaml is using the codecov team yaml.
We have experimented a little the past two days trying to get the comments to work more reliably, but have not succeeded at that.
@Dead2, I opened up a ticket with our product team to investigate why this is happening still. I really donāt have a great answer for you right now or a workaround, but I will update here when they have a fix.
@nmoinvaz, thanks for all the work on the python uploader. I recently pushed a new version to Pypi. Let me know if we are missing anything else.
Iāll take a look into pulling #265 and #267 in the next week. I think weāll want to spend some time working through a retry mechanism, and I donāt have an ETA on that. But thanks for all your work here @nmoinvaz!
@nmoinvaz, thanks again for the two PRs. We will aim to release a new python uploader version later this week. Iāll instrument some changes for retries this week and next.
No problem. Iām happy that the python uploader is being updated. It is essential on Windows machines where bash isnāt available.
The retry mechanism I did simply wraps the requests in retry function. It is very minimal change. It looks like it is more than it is because I got rid of a control block.
@tom We already tested after_n_builds: 30 for a couple of days, but unfortunately that only made it less likely we get any comment at all, with several PRs not getting one. But one PR actually got 3 comments in rapid succession for some reason.
I am having a hard time understanding how this can happen from my end of things unfortunately.
@Dead2 that is extremely strange. You wouldnāt happen to have a link to that PR? Iāll be honest, the reason for after_n_builds is that we are seeing a lot of your reports come quickly to our servers, and weāre wondering if there is some rate limit issue we are hitting back to GitHub when we try to update the comment quickly back. Would you be amenable to trying to add back after_n_builds?