While debugging an unexplained coverage change in another PR (#803 of the same repo, if curious) I created https://github.com/kensho-technologies/graphql-compiler/pull/806 which adds a Codecov config file but makes no code changes at all. I branched off of the
master branch and a few seconds later opened the PR, triggering Travis and Codecov runs.
I changed no code files, and nothing was merged into
master between when I branched off of it and when I opened the PR, so I expected no coverage changes. However, Codecov still reports a change in coverage, and the default
codecov/project check fails as a result, with default settings.
I found this behavior unexpected and surprising, and couldn’t explain it based on anything I found in the docs or in this forum — apologies if I missed something well-known and obvious.
Example affected PR: https://github.com/kensho-technologies/graphql-compiler/pull/806
I think my coauthors and I on that project have observed this behavior in the past as well, but this is the most clear-cut example we’ve had.
Steps to Reproduce
Not sure about what the prior conditions necessary to reproduce this issue are, but for the repo above they are:
- open a new PR via the web UI, making an unrelated, non-code change (e.g. add a blank line to a text file)
- wait for tests and coverage to finish
- observe change in coverage reported by Codecov, despite no code changes
Expected behavior: No coverage changes when no code was modified.
Actual behavior: In this particular example, new “missed lines” reported as a change in coverage.
Flakiness? Definitely has happened in the past, but difficult to pin down between “sometimes” / “often” / “always”.
We get this issue while running tests on Travis CI and submitting coverage via its