Inaccurate coverage based on Jacoco XML reports

Description

Codecov is reporting inaccurare or incomplete coverage statistics for files that are actually covered by tests, and reported by Jacoco XML reports.

Commit SHAs

Latest commit affecting coverage:

Repository

CI/CD or Build URL

Uploader

Github Actions: codecov/codecov-action@v1

Codecov Output

Run codecov/codecov-action@v1
/usr/bin/bash codecov.sh -n casconfiguration -F casconfiguration -Q github-action -f ./build/reports/jacoco/jacocoRootReport/jacocoRootReport.xml

  _____          _
 / ____|        | |
| |     ___   __| | ___  ___ _____   __
| |    / _ \ / _` |/ _ \/ __/ _ \ \ / /
| |___| (_) | (_| |  __/ (_| (_) \ V /
 \_____\___/ \__,_|\___|\___\___/ \_/
                              Bash-20210226-7100762


==> git version 2.30.0 found
==> curl 7.68.0 (x86_64-pc-linux-gnu) libcurl/7.68.0 OpenSSL/1.1.1f zlib/1.2.11 brotli/1.0.7 libidn2/2.2.0 libpsl/0.21.0 (+libidn2/2.2.0) libssh/0.9.3/openssl/zlib nghttp2/1.40.0 librtmp/2.3
Release-Date: 2020-01-08
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp 
Features: AsynchDNS brotli GSS-API HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM NTLM_WB PSL SPNEGO SSL TLS-SRP UnixSockets
==> GitHub Actions detected.
    Env vars used:
      -> GITHUB_ACTIONS:    true
      -> GITHUB_HEAD_REF:   
      -> GITHUB_REF:        refs/heads/master
      -> GITHUB_REPOSITORY: apereo/cas
      -> GITHUB_RUN_ID:     610728238
      -> GITHUB_SHA:        f6eed80ad3cb1db5a62c18481c11351c865a7468
      -> GITHUB_WORKFLOW:   Tests - Ubuntu
    project root: .
    Yaml found at: .github/codecov.yml
    -> Found 1 reports
==> Detecting git/mercurial file structure
==> Reading reports
    + ./build/reports/jacoco/jacocoRootReport/jacocoRootReport.xml bytes=9369233
==> Appending adjustments
    docs.codecov.io/docs/fixing-reports
    -> No adjustments found
==> Gzipping contents
        656K	/tmp/codecov.j83LwN.gz
==> Uploading reports
    url: codecov.io
    query: branch=master&commit=f6eed80ad3cb1db5a62c18481c11351c865a7468&build=610728238&build_url=http%3A%2F%2Fgithub.com%2Fapereo%2Fcas%2Factions%2Fruns%2F610728238&name=casconfiguration&tag=&slug=apereo%2Fcas&service=github-actions&flags=casconfiguration&pr=&job=Tests%20-%20Ubuntu&cmd_args=n,F,Q,f
->  Pinging Codecov
codecov.io/upload/v4?package=github-action-20210226-7100762&token=secret&branch=master&commit=f6eed80ad3cb1db5a62c18481c11351c865a7468&build=610728238&build_url=http%3A%2F%2Fgithub.com%2Fapereo%2Fcas%2Factions%2Fruns%2F610728238&name=casconfiguration&tag=&slug=apereo%2Fcas&service=github-actions&flags=casconfiguration&pr=&job=Tests%20-%20Ubuntu&cmd_args=n,F,Q,f
->  Uploading to
storage.googleapis.com/codecov/v4/raw/2021-03-01/B03E2E4102E93BDBA35B6A551DD6B2B1/f6eed80ad3cb1db5a62c18481c11351c865a7468/998a2d47-a4f8-4d22-aad0-6fddc7053358.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=GOOG1EQX6OZVJGHKK3633AAFGLBUCOOATRACRQRQF6HMSMLYUP6EAD6XSWAAY%2F20210301%2FUS%2Fs3%2Faws4_request&X-Amz-Date=20210301T201325Z&X-Amz-Expires=10&X-Amz-SignedHeaders=host&X-Amz-Signature=79ee9ae8db871efcc3a84555fd6926ec43409a7a6b24ebb3a37316bba1bd6305
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100  655k    0     0  100  655k      0  2230k --:--:-- --:--:-- --:--:-- 2237k
    -> Reports have been successfully queued for processing at codecov.io/github/apereo/cas/commit/f6eed80ad3cb1db5a62c18481c11351c865a7468

Expected Results

Here is an example file.

Actual Results

Coverage is incorrectly reported as zero.

Additional Information

Here is an example file.

In particular, note that loadYamlProperties is seen as not covered by tests. This is not the case.

This is the Jacoco XML coverage.

Note that it contains:

<class name="org/apereo/cas/configuration/CasCoreConfigurationUtils" sourcefilename="CasCoreConfigurationUtils.java">

	<method name="loadYamlProperties" desc="([Lorg/springframework/core/io/Resource;)Ljava/util/Map;" line="47">
		<counter type="INSTRUCTION" missed="0" covered="18"/>
		<counter type="LINE" missed="0" covered="6"/>
		<counter type="COMPLEXITY" missed="0" covered="1"/>
		<counter type="METHOD" missed="0" covered="1"/>
	</method>

Note the LINE counter type missing 0 lines.

There are many more files like this where Jacoco reports correct coverage, and yet codecov reports missing coverage.

Note that the project runs with JDK 11, Gradle 6.8.3 and Jacoco 0.8.6. Tests are split into test categories and are uploaded individually to codecov after each category is completed.

@tom Sorry to bother you; when you get a chance, is this something you might be able to look into and offer insight? This reported scenario seems to be quite consistent, and as I say, even though lines are covered and reported by jacoco, codecov does not seem to find them.

Could this be an issue with how coverage reports are merged together?

As always, thanks much!

Here is another more recent example.

codecov says this file basically has no coverage:
https://codecov.io/gh/apereo/cas/src/79ee19d35ce2df5857b1d469a11dc4c1bf39b1e6/support/cas-server-support-actions/src/main/java/org/apereo/cas/web/flow/GenerateServiceTicketAction.java

Coverage is successfully uploaded:

If I look at the jacoco coverage file uploaded on the repo for this test category as the logs suggest, I can see that this file is absolutely covered.

Hi @mmoayyed, due to the number of builds and the complexity of the project, could you point to which flagged report should be covering the lines in GenerateServiceTicketAction.java?

Certainly. GenerateServiceTicketAction is covereged by the webflowactions tag. This should be the relevant file with the jacoco metrics that is uploaded:

https://github.com/apereo/cas/suites/2178215041/artifacts/44904095

There have been additional commits and changes to the repo, and I hope some of these older reports/commits are still available. If not, I can always dig up a new example. From the outset, something is off here because I can see coverage jumping from 59% to 89% on codecov and then back, with all tests passing, and this is with me changing only a comma in a readme file for example.

@mmoayyed, looking at the GitHub Actions file artifact above, I see that we are mirroring the lines covered from the coverage report.

132992     <class name="org/apereo/cas/web/flow/GenerateServiceTicketAction" sourcefilename="GenerateServiceTicketAction.java">
132993       <method name="doExecute" desc="(Lorg/springframework/webflow/execution/RequestContext;)Lorg/springframework/webflow/execution/Event;" line="56">
132994         <counter type="INSTRUCTION" missed="152" covered="0"/>
132995         <counter type="BRANCH" missed="12" covered="0"/>
132996         <counter type="LINE" missed="38" covered="0"/>
132997         <counter type="COMPLEXITY" missed="7" covered="0"/>
132998         <counter type="METHOD" missed="1" covered="0"/>
132999       </method>
133000       <method name="isGatewayPresent" desc="(Lorg/springframework/webflow/execution/RequestContext;)Z" line="118">
133001         <counter type="INSTRUCTION" missed="9" covered="0"/>
133002         <counter type="LINE" missed="2" covered="0"/>
133003         <counter type="COMPLEXITY" missed="1" covered="0"/>
133004         <counter type="METHOD" missed="1" covered="0"/>
133005       </method>
133006       <method name="newEvent" desc="(Ljava/lang/String;Ljava/lang/Exception;)Lorg/springframework/webflow/execution/Event;" line="130">
133007         <counter type="INSTRUCTION" missed="12" covered="0"/>
133008         <counter type="LINE" missed="1" covered="0"/>
133009         <counter type="COMPLEXITY" missed="1" covered="0"/>
133010         <counter type="METHOD" missed="1" covered="0"/>
133011       </method>
133012       <method name="&lt;clinit&gt;" desc="()V" line="32">
133013         <counter type="INSTRUCTION" missed="0" covered="4"/>
133014         <counter type="LINE" missed="0" covered="1"/>
133015         <counter type="COMPLEXITY" missed="0" covered="1"/>
133016         <counter type="METHOD" missed="0" covered="1"/>
133017       </method>
133018       <counter type="INSTRUCTION" missed="173" covered="4"/>
133019       <counter type="BRANCH" missed="12" covered="0"/>
133020       <counter type="LINE" missed="41" covered="1"/>
133021       <counter type="COMPLEXITY" missed="9" covered="1"/>
133022       <counter type="METHOD" missed="3" covered="1"/>
133023       <counter type="CLASS" missed="0" covered="1"/>
133024     </class>

The same is true from opening up the raw upload for that flag. Am I missing something here?

Thanks much for double-checking.

There is something quite odd going on with the tests it seems, because I know for sure that particular component has decent test coverage. Here is subsequent commit where the same component is missing only 3 lines (as opposed to 38):

https://codecov.io/gh/apereo/cas/src/df54538e6054344921976e84285f042c0450056b/support/cas-server-support-actions/src/main/java/org/apereo/cas/web/flow/GenerateServiceTicketAction.java

…and I do know for sure that there have no other changes related to this component to affect its coverage. My suspicion is that either tests somehow do not run at times, or coverage is not calculated correctly. I suspect the Gradle build cache might be involved somehow, but certainly, this does not look like a codecov issue.

Apologies for the noise. I’ll do some digging and if I do find something relevant, I’ll post back.

1 Like

If it helps at all, I’m seeing this in the coverage report

  2 <sourcefile name="GenerateServiceTicketAction.java">
  3   <line nr="32" mi="0" ci="4" mb="0" cb="0"/>
  4   <line nr="56" mi="0" ci="3" mb="0" cb="0"/>
  5   <line nr="57" mi="0" ci="4" mb="0" cb="0"/>
  6   <line nr="59" mi="0" ci="3" mb="0" cb="0"/>
  7   <line nr="60" mi="0" ci="4" mb="0" cb="0"/>
  8   <line nr="63" mi="0" ci="5" mb="0" cb="0"/>
  9   <line nr="64" mi="0" ci="2" mb="0" cb="2"/>
 10   <line nr="65" mi="0" ci="6" mb="0" cb="0"/>
 11   <line nr="66" mi="0" ci="6" mb="0" cb="0"/>
 12   <line nr="69" mi="0" ci="5" mb="0" cb="0"/>
 13   <line nr="70" mi="0" ci="5" mb="0" cb="0"/>
 14   <line nr="71" mi="0" ci="4" mb="0" cb="0"/>
 15   <line nr="72" mi="0" ci="3" mb="0" cb="0"/>
 16   <line nr="73" mi="0" ci="3" mb="0" cb="0"/>
 17   <line nr="75" mi="0" ci="2" mb="1" cb="1"/>
 18   <line nr="76" mi="0" ci="4" mb="0" cb="0"/>
 19   <line nr="77" mi="0" ci="2" mb="1" cb="1"/>
 20   <line nr="78" mi="4" ci="0" mb="0" cb="0"/>
 21   <line nr="80" mi="0" ci="3" mb="0" cb="0"/>
 22   <line nr="82" mi="0" ci="3" mb="1" cb="1"/>
 23   <line nr="83" mi="4" ci="0" mb="0" cb="0"/>
 24   <line nr="84" mi="4" ci="0" mb="0" cb="0"/>
 25   <line nr="87" mi="0" ci="3" mb="0" cb="0"/>
 26   <line nr="88" mi="0" ci="6" mb="0" cb="0"/>
 27   <line nr="89" mi="0" ci="6" mb="0" cb="0"/>
 28   <line nr="91" mi="0" ci="5" mb="0" cb="0"/>
 29   <line nr="92" mi="0" ci="7" mb="0" cb="0"/>
 30   <line nr="93" mi="0" ci="3" mb="0" cb="0"/>
 31   <line nr="94" mi="0" ci="4" mb="0" cb="0"/>
 32   <line nr="95" mi="0" ci="3" mb="0" cb="0"/>
 33   <line nr="97" mi="0" ci="1" mb="0" cb="0"/>
 34   <line nr="98" mi="0" ci="3" mb="1" cb="1"/>
 35   <line nr="99" mi="0" ci="4" mb="0" cb="0"/>
 36   <line nr="100" mi="0" ci="5" mb="0" cb="0"/>
 37   <line nr="102" mi="0" ci="4" mb="0" cb="2"/>
 38   <line nr="103" mi="0" ci="4" mb="0" cb="0"/>
 39   <line nr="104" mi="0" ci="4" mb="0" cb="0"/>
 40   <line nr="106" mi="0" ci="6" mb="0" cb="0"/>
 41   <line nr="107" mi="0" ci="5" mb="0" cb="0"/>
 42   <line nr="118" mi="0" ci="4" mb="0" cb="0"/>
 43   <line nr="119" mi="0" ci="5" mb="0" cb="0"/>
 44   <line nr="130" mi="0" ci="12" mb="0" cb="0"/>
 45   <counter type="INSTRUCTION" missed="12" covered="165"/>
 46   <counter type="BRANCH" missed="4" covered="8"/>
 47   <counter type="LINE" missed="3" covered="39"/>
 48   <counter type="COMPLEXITY" missed="4" covered="6"/>
 49   <counter type="METHOD" missed="0" covered="4"/>
 50   <counter type="CLASS" missed="0" covered="1"/>
 51 </sourcefile>

So it is coming through with a different format…

I do have a theory here; The set of tests that we run on Github Actions with Gradle are set to take advantage of a “Gradle Retry” plugin. The plugin retries failing tests a few times and then backs away. What I am seeing is that if a test class fails to pass a test first, and then passes it against on subsequent attempts using the retry plugin, no coverage data is recorded for the class in the first place. For example, GenerateServiceTicketAction fails on the first try, and then passes on the 2nd try, and yet the jacoco coverage report shows that no lines are covered for this file.

I have disabled the retry plugin with Gradle and am trying a few test rounds on my forked repo to see how this might play out.

@tom Found another example but this time, seems like coverage data does not match what is shown in codecov.

Codecov says this file has 0 coverage:

This is the CI build for that commit:

This is the coverage report:
https://github.com/mmoayyed/cas/suites/2307785070/artifacts/48439601

If you unpack the coverage report and locate that component, this is a summary of what I see:

<class name="org/apereo/cas/audit/DynamoDbAuditTrailManager" sourcefilename="DynamoDbAuditTrailManager.java">
	<method name="&lt;init&gt;" desc="(Lorg/apereo/cas/audit/DynamoDbAuditTrailManagerFacilitator;Z)V" line="25"><counter type="INSTRUCTION" missed="0" covered="7"/>
	 <counter type="LINE" missed="0" covered="3"/>
	 <counter type="COMPLEXITY" missed="0" covered="1"/>
	 <counter type="METHOD" missed="0" covered="1"/>
	</method>
 
    <method name="saveAuditRecord" desc="(Lorg/apereo/inspektr/audit/AuditActionContext;)V" line="31">
        <counter type="INSTRUCTION" missed="0" covered="5"/>
       <counter type="LINE" missed="0" covered="2"/>
       <counter type="COMPLEXITY" missed="0" covered="1"/>
       <counter type="METHOD" missed="0" covered="1"/>
     </method>

     <method name="getAuditRecordsSince" desc="(Ljava/time/LocalDate;)Ljava/util/Set;" line="36">
        <counter type="INSTRUCTION" missed="0" covered="5"/>
        <counter type="LINE" missed="0" covered="1"/>
        <counter type="COMPLEXITY" missed="0" covered="1"/>
        <counter type="METHOD" missed="0" covered="1"/>
    </method>

Based on the coverage report, saveAuditRecord and getAuditRecordsSince have no missing lines. Yet codecov says otherwise.

@mmoayyed, interesting. What flag does the report get uploaded with?

In my earlier example, the flag should be dynamodb. Here is the full configuration:

      - name: "Upload to Codecov"
        uses: "codecov/codecov-action@v1"
        with:
          file: ./build/reports/jacoco/jacocoRootReport/jacocoRootReport.xml
          flags: ${{ matrix.category }}
          name: ${{ matrix.category }}
          verbose: true

@mmoayyed, I don’t actually see the coverage report uploaded to Codecov in the build tab. Do you see it uploaded on GitHub?

Bummer. It looks like the workflow run somehow was deleted from Github. I can’t quite locate the coverage report unfortunately, but I am 99% certain it was uploaded on Github. That’s how I pasted coverage data into the original report.

I’ll keep an eye out with a better example. This seems to be hit and miss; so far it’s been more or less OK, and coverage is back to around 90%. Once it hits some unexpected value like 40% then I know something is off.

Thanks for your patience. Will post back.

Absolutely @mmoayyed, I’m sorry this has been so hard to track down. Thanks for your patience, and let me know if you hit it again.

Hi @tom

Found another example where coverage data does not match what is shown on CodeCov.

This is the coverage file:
https://github.com/mmoayyed/cas/suites/2423438905/artifacts/51726125

CodeCov shows coverage for the following file:

In particular, it shows this method is missing coverage:

However, if I look at the coverage xml report (above link), I see the following:

You can see LINE coverage has 0 for missed and 10 for covered. All lines should be covered.

@mmoayyed, sorry, which flag was this coverage report uploaded for? I want to check with what we received.

Hi @tom This was uploaded under the “simple” tag.