Reported change in coverage when no code changes made

Description

While debugging an unexplained coverage change in another PR (#803 of the same repo, if curious) I created https://github.com/kensho-technologies/graphql-compiler/pull/806 which adds a Codecov config file but makes no code changes at all. I branched off of the master branch and a few seconds later opened the PR, triggering Travis and Codecov runs.

I changed no code files, and nothing was merged into master between when I branched off of it and when I opened the PR, so I expected no coverage changes. However, Codecov still reports a change in coverage, and the default codecov/project check fails as a result, with default settings.

I found this behavior unexpected and surprising, and couldn’t explain it based on anything I found in the docs or in this forum — apologies if I missed something well-known and obvious.

Repository

Example affected PR: https://github.com/kensho-technologies/graphql-compiler/pull/806

I think my coauthors and I on that project have observed this behavior in the past as well, but this is the most clear-cut example we’ve had.

Steps to Reproduce

Not sure about what the prior conditions necessary to reproduce this issue are, but for the repo above they are:

  1. open a new PR via the web UI, making an unrelated, non-code change (e.g. add a blank line to a text file)
  2. wait for tests and coverage to finish
  3. observe change in coverage reported by Codecov, despite no code changes

Expected behavior: No coverage changes when no code was modified.

Actual behavior: In this particular example, new “missed lines” reported as a change in coverage.

Flakiness? Definitely has happened in the past, but difficult to pin down between “sometimes” / “often” / “always”.

Versions

We get this issue while running tests on Travis CI and submitting coverage via its codecov script.

I also have encountered same kind of problem in my PR for other repo here.
I didn’t change the code but the coverage here is changed from master to my PR.

Hi @obi1kenobi, I dug into that pull request, and it seems like it has to do with the coverage reports you are uploading.

´The change in the PR points to this line being different https://codecov.io/gh/kensho-technologies/graphql-compiler/pull/806/changes

For the base commit https://codecov.io/gh/kensho-technologies/graphql-compiler/tree/0cf541487d8d22776a2c525a5bfa34f113695e9f (the commit that we are comparing to), we see that for build 4273.3 on python 3.7, that this line is covered
Build report: https://codecov.io/codecov/v4/raw/2020-04-28/8781BA0FDA3C43564FE94343C04916E8/0cf541487d8d22776a2c525a5bfa34f113695e9f/c3601297-b62a-4a27-89a6-20f4b0a8f898.txt

However for the commit in this PR, this line is not covered (https://codecov.io/gh/kensho-technologies/graphql-compiler/commit/1ebd37f9f1aeebe97816a0055ddfe8a160df311f/build).

This is why you are seeing the coverage drop. Let me know if that answers your question.

Hi @tom, thanks for looking into this.

Here is the diff between the PR’s commit and the base commit it is comparing to:

$ git checkout codecov_config 
Branch 'codecov_config' set up to track remote branch 'codecov_config' from 'origin'.
Switched to a new branch 'codecov_config'

$ git diff 0cf5414
diff --git a/codecov.yml b/codecov.yml
new file mode 100644
index 0000000..dfc0998
--- /dev/null
+++ b/codecov.yml
@@ -0,0 +1,7 @@
+coverage:
+  status:
+    patch:
+      default:
+        target: auto
+        threshold: 0.03%
+        base: auto

If I understood your message correctly, it would mean that the best explanation given the available evidence (change in coverage but no change in code) is that the test suite is in some way non-deterministic. Would you agree?

If my interpretation above is correct, I am happy to take a look at this with my team — while our tests are do not have any unintentional non-determinism, it’s always possible some has snuck in anyway.

Thanks again for your time, and thank you for working on this awesome tool :slight_smile:

We were able to confirm nondeterminism in our tests. We are still investigating where that nondeterminism comes from. However, we were able to confirm that codecov is not the source of the problem.

Thank you for your help!

1 Like

Amazing! Glad we were able to help!