Code lines wrongly detected as executable in parallel jobs for a Julia project

Description

We use Travis to test our Julia project in parallel. That is, different parts of the code are tested in separate jobs to reduce the build time. Merging of the reports generally works fine. However, since we changed from one job per build to multiple parallel jobs per build, the Codecov.io coverage reports indicate a significantly lower coverage than before. We tracked down the issue to the fact that Codecov erroneously reports method declarations as executable lines (which it didn’t before when we did not have parallel builds enabled). To make sure that it’s not just a simple snafu, we compared it to the report by Coveralls, which does not seem to have this issue. Below is an example for a file that shows the weird behavior in action (line 9).

Example from Codecov.io (https://codecov.io/gh/trixi-framework/Trixi.jl/src/6d1950b188673497ae10f8e51211af94433c5c0b/src/amr/amr.jl)

Example from Coveralls.io (https://coveralls.io/builds/33108254/source?filename=src/amr/amr.jl)

image removed due to new user restrictions

Repository

CI/CD

Travis

Here’s the missing example from Coveralls.io:

Hi @sloede, thanks for this. Could you provide a build before you switched to multiple parallel jobs?

As for why you are seeing that line being uncovered, builds 133.3 and 133.7 show explicitly that that line is uncovered. How are you collecting coverage metrics?

Hi! I’m the one who deactivated Codecov integration after enabling parallel tests in https://github.com/trixi-framework/Trixi.jl/pull/140, linked to Codecov build https://codecov.io/gh/trixi-framework/Trixi.jl/commit/24d419ff28837170f816bd2fd84990a6ab518790. I was asked to remove the Codecov integration completely, so we don’t have any old reports and the one linked above will not work anymore. I just activated Codecov again after some time (without adding a badge in the README.md).

We are collecting coverage metrics in Julia using https://github.com/JuliaCI/Coverage.jl.

@tom Any idea what could cause the problem or how we could fix it?

Hi all, this one was tricky, and I still don’t have a great answer.

I’m focusing on this build and the file src/amr/amr.jl. What you’ll notice is that the Codecov step happens before the Coveralls step. But in between those two lines, something else runs. You can see this line in between the two that states

[ Info: CoverageTools.process_file: Detecting coverage for src/amr/amr.jl
┌ Info: CoverageTools.process_cov: Coverage file(s) for src/amr/amr.jl do not exist.
└ Assuming file has no coverage.

My guess is for this build, Coveralls is not receiving line coverage for that file, but we still are. I can’t figure out how to view the raw report sent to Coveralls, but you can find the build here.

We, however, do get coverage for that file here. which shows those two lines being uncovered:

image

My guess as to how to fix this would be to run the two commands at the same time and you should expect the same coverage (dependent on partial coverage). I was unaware of being able to do this in Travis, but we always recommend the bash uploader. In this case, I’m not sure how you would control for timing.