Coverage report ignoring the configuration file on main branch

Description

The coverage stats for pull request and the main branch are different. It looks like the main branch is not using the correct coverage configuration file and probably using an out-dated configuration. For example, elasticdl/python/tests folder has been ignored as specified in the configuration. However, this folder is only ignored in pull request branch but not the main branch. The coverage stat for the pull request branch is correct but not the main branch.

Repository

Steps to Reproduce

  1. Create a pull request
  2. Once the build passes, take a look at the coverage for this pull request branch and you’ll notice that elasticdl/python/tests folder does not appear in the report
  3. Then compare with the coverage report for the main branch once it’s merged, elasticdl/python/tests folder still appears.

Expected behavior: The coverage stats should be the same and elasticdl/python/tests folder should be ignored for both branches.

Actual behavior: The coverage stats are not the same and elasticdl/python/tests folder is not ignored for the main branch.

Flakiness? All the time.

Versions

Please include the browser and OS and what versions you’re running.

Additional Information

Configuration: https://github.com/sql-machine-learning/elasticdl/blob/develop/.codecov.yml

Hey @terrytangyuan – thanks for the well written ticket.

I definitely see what you see regarding ignoring this file path. Strange that it is showing up on default branch.

I noticed tests are not showing up on PRs as well:

I think that your YAML is incorrectly cached here at the https://codecov.io/gh/sql-machine-learning/elasticdl/settings/yaml e.g., your YAML on record doesn’t match on Github

Can you:

1.) Validate your Codecov YAML from Github here:

https://codecov.io/validate

2.) If it validates, try resetting your YAML on Codecov as well?

3.) If you can’t reset on your side, I’ll try clearing cache on my side.

Jerrod

I don’t see anywhere I can delete the cache but I just validated it and it seems successful (see below for the log). Could you help clearing it out? Thanks.

curl -X POST --data-binary @.codecov.yml https://codecov.io/validate
Valid!

{
“ignore”: [
“^elasticdl/python/tests.*”
],
“comment”: false,
“coverage”: {
“status”: {
“project”: true,
“changes”: false,
“patch”: true
},
“range”: [
0.0,
100.0
],
“precision”: 2,
“round”: “down”
},
“codecov”: {
“require_ci_to_pass”: true
}
}

@terrytangyuan – any sense why you added a NOT operator (^) into your ignore files path?

“ignore”: [
“^elasticdl/python/tests.*”

I don’t believe we support that operator.

I believe that’s automatically inferred by codecov? Please check our codecov.yaml file which does not contain that NOT operator and it should be the source of truth instead.

Yep that is weird – I’m seeing the delta now. Investigating.

Hi Codecov team,

Is there any update on this? We’ve been stuck on this and if we cannot fix this I am afraid that we might be switching to other code coverage service.

Thanks,

Yuan

Still under active investigation. Thanks @terrytangyuan. Understood if you need urgent workability and need to switch for now.

We are running pytest elasticdl/python/tests --cov=elasticdl/python --cov-report=xml

The --cov argument might be related and note that the tests folder elasticdl/python/tests is under the same directory elasticdl/python. Hope this helps debugging.