Codecov is reporting inaccurare or incomplete coverage statistics for files that are actually covered by tests, and reported by Jacoco XML reports.
Latest commit affecting coverage:
committed 02:08PM - 01 Mar 21 UTC
https://github.com/apereo/cas/actions/runs/610728238
Github Actions: codecov/codecov-action@v1
Run codecov/codecov-action@v1
/usr/bin/bash codecov.sh -n casconfiguration -F casconfiguration -Q github-action -f ./build/reports/jacoco/jacocoRootReport/jacocoRootReport.xml
_____ _
/ ____| | |
| | ___ __| | ___ ___ _____ __
| | / _ \ / _` |/ _ \/ __/ _ \ \ / /
| |___| (_) | (_| | __/ (_| (_) \ V /
\_____\___/ \__,_|\___|\___\___/ \_/
Bash-20210226-7100762
==> git version 2.30.0 found
==> curl 7.68.0 (x86_64-pc-linux-gnu) libcurl/7.68.0 OpenSSL/1.1.1f zlib/1.2.11 brotli/1.0.7 libidn2/2.2.0 libpsl/0.21.0 (+libidn2/2.2.0) libssh/0.9.3/openssl/zlib nghttp2/1.40.0 librtmp/2.3
Release-Date: 2020-01-08
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: AsynchDNS brotli GSS-API HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM NTLM_WB PSL SPNEGO SSL TLS-SRP UnixSockets
==> GitHub Actions detected.
Env vars used:
-> GITHUB_ACTIONS: true
-> GITHUB_HEAD_REF:
-> GITHUB_REF: refs/heads/master
-> GITHUB_REPOSITORY: apereo/cas
-> GITHUB_RUN_ID: 610728238
-> GITHUB_SHA: f6eed80ad3cb1db5a62c18481c11351c865a7468
-> GITHUB_WORKFLOW: Tests - Ubuntu
project root: .
Yaml found at: .github/codecov.yml
-> Found 1 reports
==> Detecting git/mercurial file structure
==> Reading reports
+ ./build/reports/jacoco/jacocoRootReport/jacocoRootReport.xml bytes=9369233
==> Appending adjustments
docs.codecov.io/docs/fixing-reports
-> No adjustments found
==> Gzipping contents
656K /tmp/codecov.j83LwN.gz
==> Uploading reports
url: codecov.io
query: branch=master&commit=f6eed80ad3cb1db5a62c18481c11351c865a7468&build=610728238&build_url=http%3A%2F%2Fgithub.com%2Fapereo%2Fcas%2Factions%2Fruns%2F610728238&name=casconfiguration&tag=&slug=apereo%2Fcas&service=github-actions&flags=casconfiguration&pr=&job=Tests%20-%20Ubuntu&cmd_args=n,F,Q,f
-> Pinging Codecov
codecov.io/upload/v4?package=github-action-20210226-7100762&token=secret&branch=master&commit=f6eed80ad3cb1db5a62c18481c11351c865a7468&build=610728238&build_url=http%3A%2F%2Fgithub.com%2Fapereo%2Fcas%2Factions%2Fruns%2F610728238&name=casconfiguration&tag=&slug=apereo%2Fcas&service=github-actions&flags=casconfiguration&pr=&job=Tests%20-%20Ubuntu&cmd_args=n,F,Q,f
-> Uploading to
storage.googleapis.com/codecov/v4/raw/2021-03-01/B03E2E4102E93BDBA35B6A551DD6B2B1/f6eed80ad3cb1db5a62c18481c11351c865a7468/998a2d47-a4f8-4d22-aad0-6fddc7053358.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=GOOG1EQX6OZVJGHKK3633AAFGLBUCOOATRACRQRQF6HMSMLYUP6EAD6XSWAAY%2F20210301%2FUS%2Fs3%2Faws4_request&X-Amz-Date=20210301T201325Z&X-Amz-Expires=10&X-Amz-SignedHeaders=host&X-Amz-Signature=79ee9ae8db871efcc3a84555fd6926ec43409a7a6b24ebb3a37316bba1bd6305
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 655k 0 0 100 655k 0 2230k --:--:-- --:--:-- --:--:-- 2237k
-> Reports have been successfully queued for processing at codecov.io/github/apereo/cas/commit/f6eed80ad3cb1db5a62c18481c11351c865a7468
Here is an example file .
Coverage is incorrectly reported as zero.
Here is an example file .
In particular, note that loadYamlProperties is seen as not covered by tests. This is not the case.
This is the Jacoco XML coverage .
Note that it contains:
<class name="org/apereo/cas/configuration/CasCoreConfigurationUtils" sourcefilename="CasCoreConfigurationUtils.java">
<method name="loadYamlProperties" desc="([Lorg/springframework/core/io/Resource;)Ljava/util/Map;" line="47">
<counter type="INSTRUCTION" missed="0" covered="18"/>
<counter type="LINE" missed="0" covered="6"/>
<counter type="COMPLEXITY" missed="0" covered="1"/>
<counter type="METHOD" missed="0" covered="1"/>
</method>
Note the LINE counter type missing 0 lines.
There are many more files like this where Jacoco reports correct coverage, and yet codecov reports missing coverage.
Note that the project runs with JDK 11, Gradle 6.8.3 and Jacoco 0.8.6. Tests are split into test categories and are uploaded individually to codecov after each category is completed.
@tom Sorry to bother you; when you get a chance, is this something you might be able to look into and offer insight? This reported scenario seems to be quite consistent, and as I say, even though lines are covered and reported by jacoco, codecov does not seem to find them.
Could this be an issue with how coverage reports are merged together?
As always, thanks much!
Here is another more recent example.
codecov says this file basically has no coverage:
Coverage is successfully uploaded:https://github.com/apereo/cas/runs/2031544292?check_suite_focus=true
If I look at the jacoco coverage file uploaded on the repo for this test category as the logs suggest, I can see that this file is absolutely covered.
tom
March 9, 2021, 1:14am
4
Hi @mmoayyed , due to the number of builds and the complexity of the project, could you point to which flagged report should be covering the lines in GenerateServiceTicketAction.java?
Certainly. GenerateServiceTicketAction is covereged by the webflowactions tag. This should be the relevant file with the jacoco metrics that is uploaded:
https://github.com/apereo/cas/suites/2178215041/artifacts/44904095
There have been additional commits and changes to the repo, and I hope some of these older reports/commits are still available. If not, I can always dig up a new example. From the outset, something is off here because I can see coverage jumping from 59% to 89% on codecov and then back, with all tests passing, and this is with me changing only a comma in a readme file for example.
tom
March 11, 2021, 6:06pm
6
@mmoayyed , looking at the GitHub Actions file artifact above, I see that we are mirroring the lines covered from the coverage report.
132992 <class name="org/apereo/cas/web/flow/GenerateServiceTicketAction" sourcefilename="GenerateServiceTicketAction.java">
132993 <method name="doExecute" desc="(Lorg/springframework/webflow/execution/RequestContext;)Lorg/springframework/webflow/execution/Event;" line="56">
132994 <counter type="INSTRUCTION" missed="152" covered="0"/>
132995 <counter type="BRANCH" missed="12" covered="0"/>
132996 <counter type="LINE" missed="38" covered="0"/>
132997 <counter type="COMPLEXITY" missed="7" covered="0"/>
132998 <counter type="METHOD" missed="1" covered="0"/>
132999 </method>
133000 <method name="isGatewayPresent" desc="(Lorg/springframework/webflow/execution/RequestContext;)Z" line="118">
133001 <counter type="INSTRUCTION" missed="9" covered="0"/>
133002 <counter type="LINE" missed="2" covered="0"/>
133003 <counter type="COMPLEXITY" missed="1" covered="0"/>
133004 <counter type="METHOD" missed="1" covered="0"/>
133005 </method>
133006 <method name="newEvent" desc="(Ljava/lang/String;Ljava/lang/Exception;)Lorg/springframework/webflow/execution/Event;" line="130">
133007 <counter type="INSTRUCTION" missed="12" covered="0"/>
133008 <counter type="LINE" missed="1" covered="0"/>
133009 <counter type="COMPLEXITY" missed="1" covered="0"/>
133010 <counter type="METHOD" missed="1" covered="0"/>
133011 </method>
133012 <method name="<clinit>" desc="()V" line="32">
133013 <counter type="INSTRUCTION" missed="0" covered="4"/>
133014 <counter type="LINE" missed="0" covered="1"/>
133015 <counter type="COMPLEXITY" missed="0" covered="1"/>
133016 <counter type="METHOD" missed="0" covered="1"/>
133017 </method>
133018 <counter type="INSTRUCTION" missed="173" covered="4"/>
133019 <counter type="BRANCH" missed="12" covered="0"/>
133020 <counter type="LINE" missed="41" covered="1"/>
133021 <counter type="COMPLEXITY" missed="9" covered="1"/>
133022 <counter type="METHOD" missed="3" covered="1"/>
133023 <counter type="CLASS" missed="0" covered="1"/>
133024 </class>
The same is true from opening up the raw upload for that flag. Am I missing something here?
Thanks much for double-checking.
There is something quite odd going on with the tests it seems, because I know for sure that particular component has decent test coverage. Here is subsequent commit where the same component is missing only 3 lines (as opposed to 38):
…and I do know for sure that there have no other changes related to this component to affect its coverage. My suspicion is that either tests somehow do not run at times, or coverage is not calculated correctly. I suspect the Gradle build cache might be involved somehow, but certainly, this does not look like a codecov issue.
Apologies for the noise. I’ll do some digging and if I do find something relevant, I’ll post back.
1 Like
tom
March 12, 2021, 8:30pm
8
If it helps at all, I’m seeing this in the coverage report
2 <sourcefile name="GenerateServiceTicketAction.java">
3 <line nr="32" mi="0" ci="4" mb="0" cb="0"/>
4 <line nr="56" mi="0" ci="3" mb="0" cb="0"/>
5 <line nr="57" mi="0" ci="4" mb="0" cb="0"/>
6 <line nr="59" mi="0" ci="3" mb="0" cb="0"/>
7 <line nr="60" mi="0" ci="4" mb="0" cb="0"/>
8 <line nr="63" mi="0" ci="5" mb="0" cb="0"/>
9 <line nr="64" mi="0" ci="2" mb="0" cb="2"/>
10 <line nr="65" mi="0" ci="6" mb="0" cb="0"/>
11 <line nr="66" mi="0" ci="6" mb="0" cb="0"/>
12 <line nr="69" mi="0" ci="5" mb="0" cb="0"/>
13 <line nr="70" mi="0" ci="5" mb="0" cb="0"/>
14 <line nr="71" mi="0" ci="4" mb="0" cb="0"/>
15 <line nr="72" mi="0" ci="3" mb="0" cb="0"/>
16 <line nr="73" mi="0" ci="3" mb="0" cb="0"/>
17 <line nr="75" mi="0" ci="2" mb="1" cb="1"/>
18 <line nr="76" mi="0" ci="4" mb="0" cb="0"/>
19 <line nr="77" mi="0" ci="2" mb="1" cb="1"/>
20 <line nr="78" mi="4" ci="0" mb="0" cb="0"/>
21 <line nr="80" mi="0" ci="3" mb="0" cb="0"/>
22 <line nr="82" mi="0" ci="3" mb="1" cb="1"/>
23 <line nr="83" mi="4" ci="0" mb="0" cb="0"/>
24 <line nr="84" mi="4" ci="0" mb="0" cb="0"/>
25 <line nr="87" mi="0" ci="3" mb="0" cb="0"/>
26 <line nr="88" mi="0" ci="6" mb="0" cb="0"/>
27 <line nr="89" mi="0" ci="6" mb="0" cb="0"/>
28 <line nr="91" mi="0" ci="5" mb="0" cb="0"/>
29 <line nr="92" mi="0" ci="7" mb="0" cb="0"/>
30 <line nr="93" mi="0" ci="3" mb="0" cb="0"/>
31 <line nr="94" mi="0" ci="4" mb="0" cb="0"/>
32 <line nr="95" mi="0" ci="3" mb="0" cb="0"/>
33 <line nr="97" mi="0" ci="1" mb="0" cb="0"/>
34 <line nr="98" mi="0" ci="3" mb="1" cb="1"/>
35 <line nr="99" mi="0" ci="4" mb="0" cb="0"/>
36 <line nr="100" mi="0" ci="5" mb="0" cb="0"/>
37 <line nr="102" mi="0" ci="4" mb="0" cb="2"/>
38 <line nr="103" mi="0" ci="4" mb="0" cb="0"/>
39 <line nr="104" mi="0" ci="4" mb="0" cb="0"/>
40 <line nr="106" mi="0" ci="6" mb="0" cb="0"/>
41 <line nr="107" mi="0" ci="5" mb="0" cb="0"/>
42 <line nr="118" mi="0" ci="4" mb="0" cb="0"/>
43 <line nr="119" mi="0" ci="5" mb="0" cb="0"/>
44 <line nr="130" mi="0" ci="12" mb="0" cb="0"/>
45 <counter type="INSTRUCTION" missed="12" covered="165"/>
46 <counter type="BRANCH" missed="4" covered="8"/>
47 <counter type="LINE" missed="3" covered="39"/>
48 <counter type="COMPLEXITY" missed="4" covered="6"/>
49 <counter type="METHOD" missed="0" covered="4"/>
50 <counter type="CLASS" missed="0" covered="1"/>
51 </sourcefile>
So it is coming through with a different format…
I do have a theory here; The set of tests that we run on Github Actions with Gradle are set to take advantage of a “Gradle Retry” plugin. The plugin retries failing tests a few times and then backs away. What I am seeing is that if a test class fails to pass a test first, and then passes it against on subsequent attempts using the retry plugin, no coverage data is recorded for the class in the first place. For example, GenerateServiceTicketAction fails on the first try, and then passes on the 2nd try, and yet the jacoco coverage report shows that no lines are covered for this file.
I have disabled the retry plugin with Gradle and am trying a few test rounds on my forked repo to see how this might play out.
@tom Found another example but this time, seems like coverage data does not match what is shown in codecov.
Codecov says this file has 0 coverage:
This is the CI build for that commit:https://github.com/mmoayyed/cas/actions/runs/672780114
This is the coverage report:https://github.com/mmoayyed/cas/suites/2307785070/artifacts/48439601
If you unpack the coverage report and locate that component, this is a summary of what I see:
<class name="org/apereo/cas/audit/DynamoDbAuditTrailManager" sourcefilename="DynamoDbAuditTrailManager.java">
<method name="<init>" desc="(Lorg/apereo/cas/audit/DynamoDbAuditTrailManagerFacilitator;Z)V" line="25"><counter type="INSTRUCTION" missed="0" covered="7"/>
<counter type="LINE" missed="0" covered="3"/>
<counter type="COMPLEXITY" missed="0" covered="1"/>
<counter type="METHOD" missed="0" covered="1"/>
</method>
<method name="saveAuditRecord" desc="(Lorg/apereo/inspektr/audit/AuditActionContext;)V" line="31">
<counter type="INSTRUCTION" missed="0" covered="5"/>
<counter type="LINE" missed="0" covered="2"/>
<counter type="COMPLEXITY" missed="0" covered="1"/>
<counter type="METHOD" missed="0" covered="1"/>
</method>
<method name="getAuditRecordsSince" desc="(Ljava/time/LocalDate;)Ljava/util/Set;" line="36">
<counter type="INSTRUCTION" missed="0" covered="5"/>
<counter type="LINE" missed="0" covered="1"/>
<counter type="COMPLEXITY" missed="0" covered="1"/>
<counter type="METHOD" missed="0" covered="1"/>
</method>
Based on the coverage report, saveAuditRecord and getAuditRecordsSince have no missing lines. Yet codecov says otherwise.
tom
March 25, 2021, 4:48pm
11
@mmoayyed , interesting. What flag does the report get uploaded with?
In my earlier example, the flag should be dynamodb. Here is the full configuration:
- name: "Upload to Codecov"
uses: "codecov/codecov-action@v1"
with:
file: ./build/reports/jacoco/jacocoRootReport/jacocoRootReport.xml
flags: ${{ matrix.category }}
name: ${{ matrix.category }}
verbose: true
tom
March 25, 2021, 5:09pm
13
@mmoayyed , I don’t actually see the coverage report uploaded to Codecov in the build tab . Do you see it uploaded on GitHub?
Bummer. It looks like the workflow run somehow was deleted from Github. I can’t quite locate the coverage report unfortunately, but I am 99% certain it was uploaded on Github. That’s how I pasted coverage data into the original report.
I’ll keep an eye out with a better example. This seems to be hit and miss; so far it’s been more or less OK, and coverage is back to around 90%. Once it hits some unexpected value like 40% then I know something is off.
Thanks for your patience. Will post back.
tom
March 25, 2021, 8:15pm
15
Absolutely @mmoayyed , I’m sorry this has been so hard to track down. Thanks for your patience, and let me know if you hit it again.
Hi @tom
Found another example where coverage data does not match what is shown on CodeCov.
This is the coverage file:https://github.com/mmoayyed/cas/suites/2423438905/artifacts/51726125
CodeCov shows coverage for the following file:
In particular, it shows this method is missing coverage:
However, if I look at the coverage xml report (above link), I see the following:
You can see LINE coverage has 0 for missed and 10 for covered. All lines should be covered.
tom
April 6, 2021, 8:34pm
17
@mmoayyed , sorry, which flag was this coverage report uploaded for? I want to check with what we received.
Hi @tom This was uploaded under the “simple” tag.
tom
April 21, 2021, 4:00pm
19
@mmoayyed apologies I lost track of this response. I looked at a recent commit of this file here and it seems to have coverage metrics for that method. Has this been resolved for you?
Hi @tom No problem at all. I am sure you’re quite busy
Yes this has since been resolved. The scenario appears to be quite random, and generally on subsequent commit the system does bounce back. Cant quite put my finger on what the root cause would be, but last confirmation indicates that everything on our end is uploaded correctly. I’ll continue to monitor this. Thank you very much for the follow-up!
