Coverage not accurately reported

Description

I have a repository with >40 jobs in the main CI workflow.
I am using GitHub Actions.
I am using the fail_ci_if_error option.

Sometimes I see that there is missing coverage reported by Codecov for code that I know for sure is covered.

On a particular run ( https://github.com/VWS-Python/vws-python-mock/pull/690/commits/143c1ffd35af34b7c14fc3817ddf1e25f31b82f3) the code coverage is reported as 99.25%.

The “missing” coverage is code which I am certain is covered - it even includes tests which I can see were run.

Repository

CI/CD

GitHub Actions.

Uploader

Commit SHAs

143c1ffd35af34b7c14fc3817ddf1e25f31b82f3

Codecov YAML

coverage:
  status:
    patch:
      default:
        # Require 100% test coverage.
        target: 100%

Codecov Output

From a particular job which I believe has not been counted by Codecov:

https://pipelines.actions.githubusercontent.com/24aMLzmFiVrnOAZbDJVsjvzPJ0pTXAUSmss8BIAcQXv2PpCi9Y/_apis/pipelines/1/runs/518/signedlogcontent/18?urlExpires=2020-08-18T20%3A13%3A45.0179966Z&urlSigningMethod=HMACV1&urlSignature=B1ikGK047KcthYnVF4yMpocUrwvIA9I5jsIrhIDZWBU%3D

2020-08-18T08:12:17.4591510Z tests/mock_vws/test_query.py::TestAcceptHeader::test_valid[Real Vuforia-extra_headers0] PASSED
2020-08-18T08:12:17.8542687Z tests/mock_vws/test_query.py::TestAcceptHeader::test_valid[Real Vuforia-extra_headers1] PASSED
2020-08-18T08:12:17.9022981Z tests/mock_vws/test_query.py::TestAcceptHeader::test_valid[In Memory Mock Vuforia-extra_headers0] PASSED
2020-08-18T08:12:17.9163753Z tests/mock_vws/test_query.py::TestAcceptHeader::test_valid[In Memory Mock Vuforia-extra_headers1] PASSED
2020-08-18T08:12:18.5692709Z tests/mock_vws/test_query.py::TestAcceptHeader::test_invalid[Real Vuforia] PASSED
2020-08-18T08:12:19.5470067Z tests/mock_vws/test_query.py::TestAcceptHeader::test_invalid[In Memory Mock Vuforia] PASSED
2020-08-18T08:12:19.5470422Z 
2020-08-18T08:12:19.5471172Z ----------- coverage: platform linux, python 3.8.5-final-0 -----------
2020-08-18T08:12:19.5471334Z Coverage XML written to file coverage.xml
2020-08-18T08:12:19.5471420Z 
2020-08-18T08:12:19.5471537Z ===Flaky Test Report===
2020-08-18T08:12:19.5471601Z 
2020-08-18T08:12:19.5471926Z test_valid[Real Vuforia-extra_headers0] passed 1 out of the required 1 times. Success!
2020-08-18T08:12:19.5472256Z test_valid[Real Vuforia-extra_headers1] passed 1 out of the required 1 times. Success!
2020-08-18T08:12:19.5472592Z test_valid[In Memory Mock Vuforia-extra_headers0] passed 1 out of the required 1 times. Success!
2020-08-18T08:12:19.5472927Z test_valid[In Memory Mock Vuforia-extra_headers1] passed 1 out of the required 1 times. Success!
2020-08-18T08:12:19.5473087Z test_invalid[Real Vuforia] passed 1 out of the required 1 times. Success!
2020-08-18T08:12:19.5473239Z test_invalid[In Memory Mock Vuforia] passed 1 out of the required 1 times. Success!
2020-08-18T08:12:19.5473324Z 
2020-08-18T08:12:19.5473443Z ===End Flaky Test Report===
2020-08-18T08:12:19.5473522Z 
2020-08-18T08:12:19.5473657Z ============================== 6 passed in 3.45s ===============================
2020-08-18T08:12:19.6089999Z ##[group]Run codecov/codecov-action@v1.0.13
2020-08-18T08:12:19.6090200Z with:
2020-08-18T08:12:19.6090326Z   fail_ci_if_error: true
2020-08-18T08:12:19.6090444Z env:
2020-08-18T08:12:19.6090571Z   pythonLocation: /opt/hostedtoolcache/Python/3.8.5/x64
2020-08-18T08:12:19.6090691Z ##[endgroup]
2020-08-18T08:12:20.9660431Z [command]bash codecov.sh -n  -F  -Z
2020-08-18T08:12:20.9755937Z 
2020-08-18T08:12:20.9756627Z   _____          _
2020-08-18T08:12:20.9757017Z  / ____|        | |
2020-08-18T08:12:20.9757474Z | |     ___   __| | ___  ___ _____   __
2020-08-18T08:12:20.9757943Z | |    / _ \ / _` |/ _ \/ __/ _ \ \ / /
2020-08-18T08:12:20.9758152Z | |___| (_) | (_| |  __/ (_| (_) \ V /
2020-08-18T08:12:20.9758286Z  \_____\___/ \__,_|\___|\___\___/ \_/
2020-08-18T08:12:20.9759376Z                               Bash-20200728-9fb7d93
2020-08-18T08:12:20.9759483Z 
2020-08-18T08:12:20.9759549Z 
2020-08-18T08:12:20.9773233Z e[0;90m==>e[0m GitHub Actions detected.
2020-08-18T08:12:20.9875385Z     e[0;90mproject root:e[0m .
2020-08-18T08:12:20.9980662Z     e[0;90mYaml found at:e[0m codecov.yaml
2020-08-18T08:12:21.0473837Z e[0;90m==>e[0m Running gcov in . e[0;90m(disable via -X gcov)e[0m
2020-08-18T08:12:21.0519429Z e[0;90m==>e[0m Searching for coverage reports in:
2020-08-18T08:12:21.0521115Z     e[0;32m+e[0m .
2020-08-18T08:12:21.0647069Z     e[0;90m->e[0m Found 2 reports
2020-08-18T08:12:21.0649805Z e[0;90m==>e[0m Detecting git/mercurial file structure
2020-08-18T08:12:21.0775170Z e[0;90m==>e[0m Reading reports
2020-08-18T08:12:21.0837834Z     e[0;32m+e[0m ./coverage.xml e[0;90mbytes=175783e[0m
2020-08-18T08:12:21.0933147Z     e[0;32m+e[0m ./codecov.yaml e[0;90mbytes=105e[0m
2020-08-18T08:12:21.0964327Z e[0;90m==>e[0m Appending adjustments
2020-08-18T08:12:21.0965193Z     e[0;36mhttps://docs.codecov.io/docs/fixing-reportse[0m
2020-08-18T08:12:21.1157149Z     e[0;90m->e[0m No adjustments found
2020-08-18T08:12:21.1157959Z e[0;90m==>e[0m Gzipping contents
2020-08-18T08:12:21.1355709Z e[0;90m==>e[0m Uploading reports
2020-08-18T08:12:21.1357239Z     e[0;90murl:e[0m https://codecov.io
2020-08-18T08:12:21.1358311Z     e[0;90mquery:e[0m branch=bump-codecov&commit=f285876ef885f1e85d8e897c84a8f9039815aff3&build=213293561&build_url=http%3A%2F%2Fgithub.com%2FVWS-Python%2Fvws-python-mock%2Factions%2Fruns%2F213293561&name=&tag=&slug=VWS-Python%2Fvws-python-mock&service=github-actions&flags=&pr=690&job=&cmd_args=n,F,Z
2020-08-18T08:12:21.1399529Z e[0;90m->e[0m  Pinging Codecov
2020-08-18T08:12:21.1400914Z https://codecov.io/upload/v4?package=bash-20200728-9fb7d93&token=secret&branch=bump-codecov&commit=f285876ef885f1e85d8e897c84a8f9039815aff3&build=213293561&build_url=http%3A%2F%2Fgithub.com%2FVWS-Python%2Fvws-python-mock%2Factions%2Fruns%2F213293561&name=&tag=&slug=VWS-Python%2Fvws-python-mock&service=github-actions&flags=&pr=690&job=&cmd_args=n,F,Z
2020-08-18T08:12:44.4056522Z e[0;90m->e[0m  Uploading to
2020-08-18T08:12:44.4058183Z https://storage.googleapis.com/codecov/v4/raw/2020-08-18/875BAD842DFC1254D7E8EE01DC26615F/f285876ef885f1e85d8e897c84a8f9039815aff3/3237d7b7-3f52-44bc-9e4b-78c21c0b8d21.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=GOOG1EQX6OZVJGHKK3633AAFGLBUCOOATRACRQRQF6HMSMLYUP6EAD6XSWAAY%2F20200818%2FUS%2Fs3%2Faws4_request&X-Amz-Date=20200818T081244Z&X-Amz-Expires=10&X-Amz-SignedHeaders=host&X-Amz-Signature=5c9b4d0317f3f73867751c738f30f0cad27750e3666810e0e075616d94ea9b64
2020-08-18T08:12:44.4186032Z   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
2020-08-18T08:12:44.4186705Z                                  Dload  Upload   Total   Spent    Left  Speed
2020-08-18T08:12:44.4187055Z 
2020-08-18T08:12:44.6083483Z   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
2020-08-18T08:12:44.6084948Z 100 13468    0     0  100 13468      0  69422 --:--:-- --:--:-- --:--:-- 69422
2020-08-18T08:12:44.6162467Z     e[0;32m->e[0m View reports at e[0;36mhttps://codecov.io/github/VWS-Python/vws-python-mock/commit/f285876ef885f1e85d8e897c84a8f9039815aff3e[0m
2020-08-18T08:12:44.6279235Z Post job cleanup.

Steps to Reproduce

I just use the codecov action on a workflow with many jobs and this happens sometimes.

Expected behavior:

I see 100% coverage.

Actual behavior:

I see 99.25% coverage.

Flakiness?

Sometimes.

Additional Information

N/A

@adamtheturtle, I went through all 42 builds here. In particular, I was checking line 25 of mock_vws/_query_validators/accept_header_validators.py, but they all show it not being covered. This implies that the coverage reports being uploaded to Codecov do not run that particular line.

Due to the sheer amount of builds, I wasn’t able to match it up to the build in GitHub. What is the name of the test that runs that line?

Hi @tom,

Wow and thank you for the investigation.
If you need me to dig into that line, I can, but I think that I have a clearer example:

Line 947 of test_query.py is shown as not covered for commit 143c1ffd35af34b7c14fc3817ddf1e25f31b82f3.

This line is the first line of tests/mock_vws/test_query.py::TestAcceptHeader::test_valid.

I believe that if this line is not run, that means the test is not run.

The GitHub Action job which runs that test is at https://github.com/VWS-Python/vws-python-mock/pull/690/checks?check_run_id=997337082.
We can see from the logs that the test is run:

2020-08-18T08:12:17.4591510Z tests/mock_vws/test_query.py::TestAcceptHeader::test_valid[Real Vuforia-extra_headers0] PASSED
2020-08-18T08:12:17.8542687Z tests/mock_vws/test_query.py::TestAcceptHeader::test_valid[Real Vuforia-extra_headers1] PASSED
2020-08-18T08:12:17.9022981Z tests/mock_vws/test_query.py::TestAcceptHeader::test_valid[In Memory Mock Vuforia-extra_headers0] PASSED
2020-08-18T08:12:17.9163753Z tests/mock_vws/test_query.py::TestAcceptHeader::test_valid[In Memory Mock Vuforia-extra_headers1] PASSED
2020-08-18T08:12:18.5692709Z tests/mock_vws/test_query.py::TestAcceptHeader::test_invalid[Real Vuforia] PASSED
2020-08-18T08:12:19.5470067Z tests/mock_vws/test_query.py::TestAcceptHeader::test_invalid[In Memory Mock Vuforia] PASSED
2020-08-18T08:12:19.5470422Z 
2020-08-18T08:12:19.5471172Z ----------- coverage: platform linux, python 3.8.5-final-0 -----------
2020-08-18T08:12:19.5471334Z Coverage XML written to file coverage.xml
2020-08-18T08:12:19.5471420Z 
2020-08-18T08:12:19.5471537Z ===Flaky Test Report===
2020-08-18T08:12:19.5471601Z 
2020-08-18T08:12:19.5471926Z test_valid[Real Vuforia-extra_headers0] passed 1 out of the required 1 times. Success!
2020-08-18T08:12:19.5472256Z test_valid[Real Vuforia-extra_headers1] passed 1 out of the required 1 times. Success!
2020-08-18T08:12:19.5472592Z test_valid[In Memory Mock Vuforia-extra_headers0] passed 1 out of the required 1 times. Success!
2020-08-18T08:12:19.5472927Z test_valid[In Memory Mock Vuforia-extra_headers1] passed 1 out of the required 1 times. Success!
2020-08-18T08:12:19.5473087Z test_invalid[Real Vuforia] passed 1 out of the required 1 times. Success!
2020-08-18T08:12:19.5473239Z test_invalid[In Memory Mock Vuforia] passed 1 out of the required 1 times. Success!
2020-08-18T08:12:19.5473324Z 
2020-08-18T08:12:19.5473443Z ===End Flaky Test Report===
2020-08-18T08:12:19.5473522Z 
2020-08-18T08:12:19.5473657Z ============================== 6 passed in 3.45s ===============================
2020-08-18T08:12:19.6089999Z ##[group]Run codecov/codecov-action@v1.0.13
2020-08-18T08:12:19.6090200Z with:
2020-08-18T08:12:19.6090326Z   fail_ci_if_error: true
2020-08-18T08:12:19.6090444Z env:
2020-08-18T08:12:19.6090571Z   pythonLocation: /opt/hostedtoolcache/Python/3.8.5/x64
2020-08-18T08:12:19.6090691Z ##[endgroup]
2020-08-18T08:12:20.9660431Z [command]bash codecov.sh -n  -F  -Z
2020-08-18T08:12:20.9755937Z 
2020-08-18T08:12:20.9756627Z   _____          _
2020-08-18T08:12:20.9757017Z  / ____|        | |
2020-08-18T08:12:20.9757474Z | |     ___   __| | ___  ___ _____   __
2020-08-18T08:12:20.9757943Z | |    / _ \ / _` |/ _ \/ __/ _ \ \ / /
2020-08-18T08:12:20.9758152Z | |___| (_) | (_| |  __/ (_| (_) \ V /
2020-08-18T08:12:20.9758286Z  \_____\___/ \__,_|\___|\___\___/ \_/
2020-08-18T08:12:20.9759376Z                               Bash-20200728-9fb7d93
2020-08-18T08:12:20.9759483Z 
2020-08-18T08:12:20.9759549Z 
2020-08-18T08:12:20.9773233Z e[0;90m==>e[0m GitHub Actions detected.
2020-08-18T08:12:20.9875385Z     e[0;90mproject root:e[0m .
2020-08-18T08:12:20.9980662Z     e[0;90mYaml found at:e[0m codecov.yaml
2020-08-18T08:12:21.0473837Z e[0;90m==>e[0m Running gcov in . e[0;90m(disable via -X gcov)e[0m
2020-08-18T08:12:21.0519429Z e[0;90m==>e[0m Searching for coverage reports in:
2020-08-18T08:12:21.0521115Z     e[0;32m+e[0m .
2020-08-18T08:12:21.0647069Z     e[0;90m->e[0m Found 2 reports
2020-08-18T08:12:21.0649805Z e[0;90m==>e[0m Detecting git/mercurial file structure
2020-08-18T08:12:21.0775170Z e[0;90m==>e[0m Reading reports
2020-08-18T08:12:21.0837834Z     e[0;32m+e[0m ./coverage.xml e[0;90mbytes=175783e[0m
2020-08-18T08:12:21.0933147Z     e[0;32m+e[0m ./codecov.yaml e[0;90mbytes=105e[0m
2020-08-18T08:12:21.0964327Z e[0;90m==>e[0m Appending adjustments
2020-08-18T08:12:21.0965193Z     e[0;36mhttps://docs.codecov.io/docs/fixing-reportse[0m
2020-08-18T08:12:21.1157149Z     e[0;90m->e[0m No adjustments found
2020-08-18T08:12:21.1157959Z e[0;90m==>e[0m Gzipping contents
2020-08-18T08:12:21.1355709Z e[0;90m==>e[0m Uploading reports
2020-08-18T08:12:21.1357239Z     e[0;90murl:e[0m https://codecov.io
2020-08-18T08:12:21.1358311Z     e[0;90mquery:e[0m branch=bump-codecov&commit=f285876ef885f1e85d8e897c84a8f9039815aff3&build=213293561&build_url=http%3A%2F%2Fgithub.com%2FVWS-Python%2Fvws-python-mock%2Factions%2Fruns%2F213293561&name=&tag=&slug=VWS-Python%2Fvws-python-mock&service=github-actions&flags=&pr=690&job=&cmd_args=n,F,Z
2020-08-18T08:12:21.1399529Z e[0;90m->e[0m  Pinging Codecov
2020-08-18T08:12:21.1400914Z https://codecov.io/upload/v4?package=bash-20200728-9fb7d93&token=secret&branch=bump-codecov&commit=f285876ef885f1e85d8e897c84a8f9039815aff3&build=213293561&build_url=http%3A%2F%2Fgithub.com%2FVWS-Python%2Fvws-python-mock%2Factions%2Fruns%2F213293561&name=&tag=&slug=VWS-Python%2Fvws-python-mock&service=github-actions&flags=&pr=690&job=&cmd_args=n,F,Z
2020-08-18T08:12:44.4056522Z e[0;90m->e[0m  Uploading to
2020-08-18T08:12:44.4058183Z https://storage.googleapis.com/codecov/v4/raw/2020-08-18/875BAD842DFC1254D7E8EE01DC26615F/f285876ef885f1e85d8e897c84a8f9039815aff3/3237d7b7-3f52-44bc-9e4b-78c21c0b8d21.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=GOOG1EQX6OZVJGHKK3633AAFGLBUCOOATRACRQRQF6HMSMLYUP6EAD6XSWAAY%2F20200818%2FUS%2Fs3%2Faws4_request&X-Amz-Date=20200818T081244Z&X-Amz-Expires=10&X-Amz-SignedHeaders=host&X-Amz-Signature=5c9b4d0317f3f73867751c738f30f0cad27750e3666810e0e075616d94ea9b64
2020-08-18T08:12:44.4186032Z   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
2020-08-18T08:12:44.4186705Z                                  Dload  Upload   Total   Spent    Left  Speed
2020-08-18T08:12:44.4187055Z 
2020-08-18T08:12:44.6083483Z   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
2020-08-18T08:12:44.6084948Z 100 13468    0     0  100 13468      0  69422 --:--:-- --:--:-- --:--:-- 69422
2020-08-18T08:12:44.6162467Z     e[0;32m->e[0m View reports at e[0;36mhttps://codecov.io/github/VWS-Python/vws-python-mock/commit/f285876ef885f1e85d8e897c84a8f9039815aff3e[0m

@adamtheturtle, hmmm… I went through all the reports there, and I’m seeing that lines in the method never get covered. However, I see for 16 of the reports, that the function definition gets called.

Are you able to output coverage.xml for those runs before they get to the Codecov step? Probably adding a step that does cat ./coverage.xml.