Python erroneous code coverage

Description

In the past week the code coverage reported by codecov for one of my Python projects dropped. The results do not match those observed by using Python’s “coverage” package directly. What’s more, some of the lines reported as uncovered are blank lines or comments!

Repository

An example of a comment line reported as “uncovered”:

And a bunch more here:

(this file has 6 missing lines according to “coverage”, 17 according to codecov)

Steps to Reproduce

  1. Run the test suite
  2. Observe the results

Expected behavior: Blank or comment lines should not appear “uncovered”

Actual behavior: codecov reports spurious uncovered lines

Flakiness? This is fully reproducible.

Versions

The tests are run using Travis CI, and Python 3.6, 3.7 and pypy3.

Additional Information

None

I still don’t understand the root cause, but the problem seems to go away if I only upload Python 3.6 / 3.7 results and not pypy3.

Hi @jlaine

I believe I heard there was a bug in pypy3 where it outputs blank lines that upsets our coverage parser as misses. We are digging in here, but I’m glad you found a workaround.

Thanks for responding @drazisil! I’ll still to my workaround until the pypy3 situation clears up.

In the meantime I seem to have hit another bug which does not occur when I use vanilla “coverage”: files with identical names but different paths don’t seem to show up. Still on the aioquic project I have two filenames which are the same but in different folders:

  • aioquic/connection.py and aioquic/h3/connection.py
  • aioquic/events.py and aioquic/h3/events.py

The files in aioquic/h3 show up, but those immediately in aioquic do not:

See:

Vs:

Hi @jlaine

I believe this is due to a change we made to pathfixing that we are planning to roll back. Can you please let me know if this is working again by Monday?

Hi ! Unfortunately Monday has come and passed, and the files I mentioned are still missing from the coverage report.

Hi @jlaine

Sorry to hear that. If it’s the same report you may need to rerun/re-upload it.

If it’s a new report can you link me on Codecov so I can see what’s up?