Codecov showing confusing/nonsensical coverage

Bug Fixes
#1

Description

Starting recently, we’re seeing codecov declare there’s a file with incomplete coverage, however when we look at the codecov UI it shows no missing coverage.

Further, looking at the console output from our builds, python coverage shows that file as having no missed lines/branches.

Commit SHAs


CI/CD or Build URL

Github Actions

Uploader

bash uploader: https://github.com/pyca/cryptography/blob/master/.github/actions/upload-coverage/action.yml#L13

Expected Results

We expect those builds to have 100% coverage.

Actual Results

codecov says src/cryptography/utils.py has a missed line of coverage, but the UI for that file shows no missing lines:

Notably, codecov claims this file has ~600 lines of code, while it actually has 150.

#2

@alex, it looks we like aren’t properly parsing the source field. I’ll bring this to the product team, and thanks for your patience.

#5

@alex, we made a fix here. Would you be able to see if it’s been resolved for you?

#6

I will keep my eyes peeled to see if this occurs again.

#7

Unfortunately I see this recurring: https://codecov.io/gh/pyca/cryptography/src/2428b11ab84bb796bfcc595cd48fc00e3195e6bb/src/cryptography/utils.py currently claims to have 93% coverage on this file, but every line is green.

#8

@alex apologies, I’m wondering if this is a timing issue. It shows 100% coverage right now and given the number of uploads you have, I wonder if we were still processing reports at that time. Are you seeing it on a different file?

#9

It’s definitely possible it’s a timing issue. However I do not understand why at any point it would claim there there’s less than 100% coverage while simultaneously showing me a UI with all lines covered.

#10

@alex, you’re right. It really is unclear to me why you would see a different value. Are you uploading just 1 build?

#11

No, we upload from a couple dozen builds. You can see our CI process
here: cryptography/ci.yml at main · pyca/cryptography · GitHub

#12

@alex, apologies I was dealing with COVID the last few weeks. I’ve lost a little context on this, do you have another example of this happening?