Codecov incorrectly considers implicitly executed Go statements as being not executed

Bug Fixes
#1

Description

See https://github.com/elves/elvish/pull/1190 for details.

Commit SHAs

Repository

CI/CD or Build URL

Uploader

https://codecov.io/bash

Codecov Output

n/a

Expected Results

Coverage of Go language (golang) files implicitly excludes any line not in the coverage data.

Actual Results

The Codecov analysis includes, as not executed, statements in import blocks as well as other constructs that are implicitly executed but not included in the coverage data. This noticably decreases the coverage metric for individual Go (golang) files as well as, obviously, the coverage percentage for the project as a whole.

Additional Information

n/a

#2

@krader1961, I went through all of your raw uploads here, and none of them are showing coverage metrics for histwalk/histwalk.go between lines 4-11. Am I reading this issue incorrectly?

image
image937×346 37.7 KB

#3

This issue might be unique to the Elvish shell and how it communicates coverage data of this project to codecov. My point is that there are no coverage metrics for statements (i.e., lines of code) such as import (...). In the Go language those statements are implicitly (always) executed. Codecov is treating those statements as unexecuted rather than as comments (i.e., implicitly executed).

#4

@krader1961, Codecov does not make assumptions here about what lines of code might be executed if it does not appear on a coverage report (regardless if it is always implicity run).

What is curious to me is that we are receiving a report that is different from go tool cover -func. Are you sending a different report from what is generated?

#5

@tom, I don’t grok your last question. The go tool cover -func output includes, as its last line, the aggregate coverage for the project. For the Elvish shell it, correctly, reports 88% coverage (versus the 82% reported by Codecov.io). Looking at the coverage for individual source files using go tool cover -html shows that it treats blocks such as import(...) as being equivalent to comments because they are a) always executed by the compiler, and b) never executed at runtime. Whereas Codecov.io treats them as unexecuted statements because those lines do not appear in the data file produced by go test -cover and they are not comment lines/blocks.

If there is some way to get Codecov.io to correctly ignore those lines, just as it does for comments, I couldn’t find it in the documentation.

#6

Hi @krader1961 and apologies for the delay here. What I meant is that Codecov is showing coverage as being uploaded. I’d like to see why there’s a difference between how the 88% coverage report is generated and the 82%.

That being said, there is no way to ignore those lines. We will in fact be rolling back the change for go functions as well to more correctly mirror go's output.

#7

The difference is because some statements, such as import and type, are never explicitly executed at run time – they are implicitly “executed” by the Go compiler. So those statements will never appear in the coverage data. I suppose the Go go/parser and go/ast packages could be used to identify those lines and synthesize appropriate lines for inclusion in the coverage data or to otherwise flag that those lines should be ignored by Codecov.