GitHub PR not being updated with final coverage data

Description

In the last few days, since the most recent changes to Codecov, it appears that the final coverage data is not being updated in GItHub PRs.

On the codecov website, the coverage percent shows correctly, but on the codecov comment in the PR it consistently shows a coverage percent that is less than the codecov webpage.

Repository

zlib-ng project on github.

I have a 4 link maximum imposed by this forum, otherwise I would post the link.

CI/CD

GitHub Actions

Uploader

codecov-python

Commit SHAs


GitHub comment shows 75.89

Codecov website shows 79.21
GitHub comment shows 74.42

Codecov website shows 79.28

There are more examples if you just examine my other PR in that same project.

Codecov Output

There are many CI instances which run per PR.

Steps to Reproduce

  1. Fork project repository
  2. Create pull request
  3. GitHub comment by codecov bot will show different coverage percent than Codecov website.

Expected behavior: The coverage percentages to match.

Actual behavior: The coverage percentages do not match.

Flakiness? Every time

Additional Information

This only started happening maybe within this last week.

2 Likes

Hi @nmoinvaz, I noticed you were using a fork of the python uploader, can you pull yours to the latest?

I have to maintain my own fork because some important PRs are not getting into the main repository especially w/r to GitLab. PRs include #187, #208, #216, Also we had to rework the retry mechanism because it failed too often.

I have rebased to the latest master and reworked all the PRs I had to commit so they conform to your new formatting.

I will wait and see what happens.

1 Like

I think I might know what it is from. In the PR it is showing stats from my fork and not from the main repository. In my fork I might not have the same codecov yaml.

But it is curious why it is taking from my fork and not from the main repository PR. I have updated my code that detects GitHub Actions (from bash uploader) to see if that might be it and submitted a PR for it.

1 Like

Hi @nmoinvaz did that end up working for you?

No, it seems like that was not the case. Although there is a difference in coverage percentages between my fork (of zlib-ng) and the mainline (zlib-ng), it is not enough to account for the percentage that the codecov commenting bot is reporting. Also the codecov comment on the PR has links to the coverage report for mainline (zlib-ng).

When you look at this PR, the comment show 72% but if you click into the report it shows 78%.

So it still looks like that the comments are not being updated at the end. We have a unique situation with a lot of CI instances.

@tom Could this be codecov commenting system hitting a ratelimiter on updating github comments, causing it to be unable to update it again?

The really weird thing is it will not update it again if you do a force push, but it will if you delete the old comment first. If github has a ratelimiter, that would mean it either blocks them permanently, or codecov gives up on updating it after it fails.

This used to work but suddenly stopped working, we did not change the codecov yaml config in that period.
Also, have a look at the comment it the above linked PR and notice it does not list reports for all the flags, even though the PR viewed at the codecov site clearly has them. The number of “flags”/reports listed in the comment is usually around 5-20, and it used to be all of them.

A big clue here might be my last PR, it did not get a codecov comment report at all. See: https://github.com/zlib-ng/zlib-ng/pull/597 But if you view the codecov entries under “checks”, it looks perfectly in line with HEAD.
So clearly codecov somehow failed to make the comment, and I’d think it would retry once for each report it gets in, and since we have a lot of CI runs we submit a lot of reports.
This PR was made from the same repo, so no external repo involved at all.

Hi @nmoinvaz and @Dead2,

Regarding the pull request that you linked to here, I’m staring at the logs and I’m convinced that it’s a GitHub hiccup. It is very unlikely that it’s a rate limit issue, but if you run the CI/CD again, I think it will go through fine.

As for this PR, you mentioned a yaml file. I’m actually not able to find one in the repo. I also can’t see us pulling one down either. I am, however, seeing the same issue with GitHub being unreachable. I’d like to see if this is a one-time issue or it’s consistent. Have you seen this happening since?

@tom This happens for every single PR, and has been happening for a month at least.
Sometimes the comment is missing completely, but most of the time it just doesn’t update it.
See https://github.com/zlib-ng/zlib-ng/pull/604 for the most recent example of missing completely.

This used to happen before too (we had lots of problems before, making codecov completely useless), but everything was pretty much fixed in early March (most comment reports were complete, just a few were missing a couple of the CI runs).
Ref: Codecov server 500 error and endless "processing"
But after 2-4 weeks, the comment problem came back again in full force.

Oh, and the yaml is using the codecov team yaml.
We have experimented a little the past two days trying to get the comments to work more reliably, but have not succeeded at that.

@Dead2, I opened up a ticket with our product team to investigate why this is happening still. I really don’t have a great answer for you right now or a workaround, but I will update here when they have a fix.

@nmoinvaz, thanks for all the work on the python uploader. I recently pushed a new version to Pypi. Let me know if we are missing anything else.

Ok sounds good. The only issues that I need are #265, #267, and then perhaps check out my comment on #158 and respond… I have a retry mechanism that are we currently using that works well: https://github.com/codecov/codecov-python/issues/158#issuecomment-636036376

1 Like

I’ll take a look into pulling #265 and #267 in the next week. I think we’ll want to spend some time working through a retry mechanism, and I don’t have an ETA on that. But thanks for all your work here @nmoinvaz!

@nmoinvaz, thanks again for the two PRs. We will aim to release a new python uploader version later this week. I’ll instrument some changes for retries this week and next.

No problem. I’m happy that the python uploader is being updated. It is essential on Windows machines where bash isn’t available.

The retry mechanism I did simply wraps the requests in retry function. It is very minimal change. It looks like it is more than it is because I got rid of a control block.

We have been using it in zlib-ng and rarely do we see an instance where reports aren’t uploaded.

Thanks here @nmoinvaz, I’ll look to make those fixes this week and deploy early next week.

@Dead2, I think this might fix the comment issue not being updated. Can you add this to a codecov.yml file

comment:
  after_n_builds: 32

I’m approximately using 32, but please use the number that seems right for you.

@tom We already tested after_n_builds: 30 for a couple of days, but unfortunately that only made it less likely we get any comment at all, with several PRs not getting one. But one PR actually got 3 comments in rapid succession for some reason.
I am having a hard time understanding how this can happen from my end of things unfortunately.

@Dead2 that is extremely strange. You wouldn’t happen to have a link to that PR? I’ll be honest, the reason for after_n_builds is that we are seeing a lot of your reports come quickly to our servers, and we’re wondering if there is some rate limit issue we are hitting back to GitHub when we try to update the comment quickly back. Would you be amenable to trying to add back after_n_builds?

@tom I will try that again so you can get some up-to-date logs to look at.

1 Like