Coverage report ignoring the configuration file on main branch

[terrytangyuan]

Description

The coverage stats for pull request and the main branch are different. It looks like the main branch is not using the correct coverage configuration file and probably using an out-dated configuration. For example, elasticdl/python/tests folder has been ignored as specified in the configuration. However, this folder is only ignored in pull request branch but not the main branch. The coverage stat for the pull request branch is correct but not the main branch.

Repository

GitHub - sql-machine-learning/elasticdl: Kubernetes-native Deep Learning...

Steps to Reproduce

1. Create a pull request
2. Once the build passes, take a look at the coverage for this pull request branch and you’ll notice that elasticdl/python/tests folder does not appear in the report
3. Then compare with the coverage report for the main branch once it’s merged, elasticdl/python/tests folder still appears.

Expected behavior: The coverage stats should be the same and elasticdl/python/tests folder should be ignored for both branches.

Actual behavior: The coverage stats are not the same and elasticdl/python/tests folder is not ignored for the main branch.

Flakiness? All the time.

Versions

Please include the browser and OS and what versions you’re running.

Additional Information

Configuration: elasticdl/.codecov.yml at develop · sql-machine-learning/elasticdl · GitHub

[jerrod]

Hey @terrytangyuan – thanks for the well written ticket.

I definitely see what you see regarding ignoring this file path. Strange that it is showing up on default branch.

I noticed tests are not showing up on PRs as well:

I think that your YAML is incorrectly cached here at the https://codecov.io/gh/sql-machine-learning/elasticdl/settings/yaml e.g., your YAML on record doesn’t match on Github

Can you:

1.) Validate your Codecov YAML from Github here:

https://codecov.io/validate

2.) If it validates, try resetting your YAML on Codecov as well?

3.) If you can’t reset on your side, I’ll try clearing cache on my side.

Jerrod

[terrytangyuan]

I don’t see anywhere I can delete the cache but I just validated it and it seems successful (see below for the log). Could you help clearing it out? Thanks.

curl -X POST --data-binary @.codecov.yml https://codecov.io/validate
Valid!

{
“ignore”: [
“^elasticdl/python/tests.*”
],
“comment”: false,
“coverage”: {
“status”: {
“project”: true,
“changes”: false,
“patch”: true
},
“range”: [
0.0,
100.0
],
“precision”: 2,
“round”: “down”
},
“codecov”: {
“require_ci_to_pass”: true
}
}

[jerrod]

@terrytangyuan – any sense why you added a NOT operator (^) into your ignore files path?

“ignore”: [
“^elasticdl/python/tests.*”

I don’t believe we support that operator.

[terrytangyuan]

I believe that’s automatically inferred by codecov? Please check our codecov.yaml file which does not contain that NOT operator and it should be the source of truth instead.

[jerrod]

Yep that is weird – I’m seeing the delta now. Investigating.

[terrytangyuan]

Hi Codecov team,

Is there any update on this? We’ve been stuck on this and if we cannot fix this I am afraid that we might be switching to other code coverage service.

Thanks,

Yuan

[jerrod]

Still under active investigation. Thanks @terrytangyuan. Understood if you need urgent workability and need to switch for now.

[terrytangyuan]

We are running pytest elasticdl/python/tests --cov=elasticdl/python --cov-report=xml

The --cov argument might be related and note that the tests folder elasticdl/python/tests is under the same directory elasticdl/python. Hope this helps debugging.

[ehnuje]

@jerrod Is there any update regarding the issue?

[Ornias1993]

It seems they just ignore this.
Same issue here:

Codecov.yml Ignore not working

Description As requested here: I hereby report the following bug: Adding “Ignore” to the codecov.yml file in following project, does not work: Other settings from the codecov.yml work fine, making changes to those also can be previewed fine in a Pull Request. As can be seen here: [CANT POST LINK 1] (ignore the wrong ignore syntax in that PR, this has been fixed later) We tested multiple writingstyles, paths and moving codecov.yml to root in 2 PR’s, which did not work out: [CANT POST LI…

@jerrod If you guys are not interested, just remove ignore from the manual and announce to everyone we should all switch to another service.

Ahh, great company… Above bugreport is now hidden, because “community flags”, although this community has no flagging. So they manually hid the thread. Great sign, NOT.

[jerrod]

@Ornias1993

Ahh, great company… Above bugreport is now hidden, because “community flags”, although this community has no flagging. So they manually hid the thread. Great sign, NOT.

Hey Ornias, this definitely makes me sad. We work every day to serve the open source community at large and didn’t do a good enough job. Thanks for bringing this to our attention.

The ignore tool does work at a certain level, but getting it to a working state for a user to work was not well documented.

We’ve made and are making the following changes:

1. About the Codecov YAML → specified that YAML file must code in the root of the project (thank you for helping us locate this)

2. We are adding better logging on from where the YAML file is being called in our logs, so that if the incorrect YAML file gets called we will better to be able a.) identify it and b.) surface that information back to the users.

Appreciate you sticking with us as we continue to improve!

Also, just confirming here, as with Twitter, that your original posts were flagged unexpectedly:

https://twitter.com/jengelberg/status/1205613049874276352

Jerrod