Coverage reports missing some covered code after processing

Support Bug Fixes
1

Description

I have several coverage reports from a Python project generated on Travis CI and Appveyor; when uploaded and processed, the Appveyor reports are missing some code that was actually covered. It looks like the missing coverage is the code that’s run in a subprocess in the tests (it’s running the application and collecting coverage for it).

Repository

CI/CD

Appveyor (plus Travis CI, but that appears to work fine).

Example job, with command-line output showing the coverage report for verification: AppVeyor

Uploader

Using codecov-python, invoked with codecov --name "%APPVEYOR_JOB_NAME%"

Commit SHAs

Codecov YAML

comment: false
coverage:
  status:
    project:
      default:
        informational: true
    patch:
      default:
        informational: true

Codecov Output

      _____          _
     / ____|        | |
    | |     ___   __| | ___  ___ _____   __
    | |    / _ \ / _  |/ _ \/ __/ _ \ \ / /
    | |___| (_) | (_| |  __/ (_| (_) \ V /
     \_____\___/ \____|\___|\___\___/ \_/
                                    v2.1.8
==> Detecting CI provider
    AppVeyor Detected
==> Preparing upload
==> Processing gcov (disable by -X gcov)
==> Collecting reports
XX> Searching for reports disabled
    Generating coverage xml reports for Python
    + C:\projects\bfg9000\coverage.xml bytes=368247
==> Uploading
    .url https://codecov.io
    .query commit=b8e2f4523c37a591e8c15d8d8628761e292b8f3d&branch=refactor-cc-msvc&job=jimporter%2Fbfg9000%2Frefactor-cc-msvc-2264&service=appveyor&build=s4st0i2sq0iclndn&slug=jimporter%2Fbfg9000&name=Image%3A%20Visual%20Studio%202013%3B%20Environment%3A%20COMPILER%3Dvs&yaml=codecov.yml&package=py2.1.8
    Gzipping contents..
    Compressed contents to 20136 bytes
    Pinging Codecov...
    Uploading to S3...
    https://codecov.io/github/jimporter/bfg9000/commit/b8e2f4523c37a591e8c15d8d8628761e292b8f3d

Steps to Reproduce

  1. Inject the following sitecustomize.py to collect coverage for Python subprocesses: import coverage; coverage.process_startup()
  2. Run coverage erase
  3. Run coverage run setup.py test
  4. Run coverage combine
  5. Submit coverage

Expected behavior: Full coverage; in particular, note the coverage for the file bfg9000/tools/msvc/__init__.py. In the Appveyor log, it shows 99% coverage.

Actual behavior: In the Codecov report, bfg9000/tools/msvc/__init__.py only shows 86% coverage: https://codecov.io/gh/jimporter/bfg9000/src/b8e2f4523c37a591e8c15d8d8628761e292b8f3d/bfg9000/tools/msvc/__init__.py. I’ve verified that the raw uploaded report looks ok.

Flakiness? This appears to happen all the time, since refactoring this part of the code to be in multiple files; formerly it lived in bfg9000/tools/mscv.py and worked fine.

2

Hi @jimporter, thanks for reaching out. So just a few questions, I looked in your repository, but I can’t seem to find where you are calling the --name argument on codecov. I’m asking because I see this line

Searching for reports disabled

but that should only happen if it’s been disabled.

If you are more in the area of just getting this to work, I would recommend the bash script. You should be able to run something like

bash <(curl -s https://codecov.io/bash) -U "-s" -A "-s" (this is Windows specific)

3

Here’s where I’m adding the --name argument; it’s on the refactor-cc-msvc branch: https://github.com/jimporter/bfg9000/blob/refactor-cc-msvc/appveyor.yml#L151 (I added it because I was having a hard time figuring out which Appveyor job was which; their names were random strings of characters.)

However, on Appveyor, I’ve always seen Searching for reports disabled, and it’s worked fine. It correctly generates the XML version of my coverage (it says Generating coverage xml reports for Python). Searching through your link, it looks like this is an Appveyor-specific thing: https://github.com/codecov/codecov-python/blob/066706eb2fab6c0fbe7c8f35d9b4bf405bf5db9f/codecov/\_\_init\_\_.py#L719-L720

For example, here’s a job from a previous commit on my master branch that says searching for reports is disabled, but I just called codecov with no options (after installing it via pip install codecov): AppVeyor.

Unfortunately, I wasn’t able to get the command bash <(curl -s https://codecov.io/bash) -U "-s" -A "-s" to work successfully on Appveyor (I tried cmd.exe, sh, and powershell syntaxes). I’ll keep poking at it to see if I can get some alternate spelling to work…

4

Hi @jimporter, I’ll take a deeper look at this, but I suggest

curl -s https://codecov.io/bash | bash -s -- -U "-s" -A "-s"
5

Cool, that works, and the coverage appears to be reported as expected! I’ll switch to using the Bash uploader.

I plan to squash the commits and delete the refactor-cc-msvc branch that’s linked above, so some of the links above will 404. If it’s helpful, I’ll set up a branch that uses the codecov-python uploader so you have logs/commits to refer to for testing on your end (I’m not totally sure what info you’d need to fix the codecov-python uploader…)

6

Ok, here’s a branch with the broken behavior for testing: Test out codecov-python for Appveyor · jimporter/bfg9000@da2f21d · GitHub. I’ll keep this branch around for a while so you can refer to it if needed.

On Appveyor, coverage report shows 99% coverage for bfg9000/tools/msvc/__init__.py: AppVeyor

But on Codecov, this shows as 87%: Codecov