Error Processing Coverage Report

Description

Given the following XML coverage report, produced in the following CI build, for which the codecov bash script reported this as the URL to see the report.

Why is it failing to process the report?

Repository

CI/CD

Jenkins

Uploader

The bash uploader.

Commit SHAs

Codecov YAML

codecov:
  ci:
    - jenkinsci.saltstack.com
    - github.com

  max_report_age: 24           # The age you want coverage reports to expire at, or if you
                               # want to disable this check. Expired reports will not be processed by codecov.
  require_ci_to_pass: yes      # Less spammy. Only notify on passing builds.


ignore:
  - ^*.py$                       # python files at the repo root, ie, setup.py
  - doc/.*                       # ignore any code under doc/
  - salt/ext/.*                  # ignore any code under salt/ext

coverage:
  round: up
  range: 70..100
  precision: 2

  notify: {}
  status: false

flags:
  salt:
    paths:
      - salt/
    carryforward: true           # https://docs.codecov.io/docs/carryforward-flags
  tests:
    paths:
      - tests/
    carryforward: true

# Disable Comments
comment: off

Codecov Output

09:26:35  [Pipeline] {
09:26:36  [Pipeline] sh
09:26:36  + '[' -f artifacts/coverage/salt.xml ']'
09:26:36  + curl -L https://codecov.io/bash
09:26:36  ++ pwd
09:26:36  + /bin/bash -s -- -R /var/jenkins/workspace/pr-amazon2-py3-slow_master -n amazon-2-py3-runtests-zeromq-salt -f artifacts/coverage/salt.xml -F amazon2,py3,runtests,zeromq,salt
09:26:36    % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
09:26:36                                   Dload  Upload   Total   Spent    Left  Speed
09:26:36  
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
09:26:36    _____          _
09:26:36   / ____|        | |
09:26:36  | |     ___   __| | ___  ___ _____   __
09:26:36  | |    / _ \ / _` |/ _ \/ __/ _ \ \ / /
09:26:36  | |___| (_) | (_| |  __/ (_| (_) \ V /
09:26:36   \_____\___/ \__,_|\___|\___\___/ \_/
09:26:36                                Bash-20200728-9fb7d93
09:26:36  
09:26:36  
09:26:36  ==> Jenkins CI detected.
09:26:36      project root: /var/jenkins/workspace/pr-amazon2-py3-slow_master
09:26:36  --> token set from env
09:26:36      Yaml found at: .codecov.yml
09:26:36  
100 51819  100 51819    0     0   104k      0 --:--:-- --:--:-- --:--:--  104k
09:26:36      -> Found 1 reports
09:26:36  ==> Detecting git/mercurial file structure
09:26:36  ==> Reading reports
09:26:36      + artifacts/coverage/salt.xml bytes=14169480
09:26:36  ==> Appending adjustments
09:26:36      https://docs.codecov.io/docs/fixing-reports
09:26:36      + Found adjustments
09:26:36  ==> Gzipping contents
09:26:37  ==> Uploading reports
09:26:37      url: https://codecov.io
09:26:37      query: branch=master&commit=4684a0b585772e05df57c047d673e44229360774&build=380&build_url=https%3A%2F%2Fjenkinsci.saltstack.com%2Fjob%2Fpr-amazon2-py3-slow%2Fjob%2Fmaster%2F380%2F&name=amazon-2-py3-runtests-zeromq-salt&tag=&slug=saltstack%2Fsalt&service=jenkins&flags=amazon2,py3,runtests,zeromq,amazon2,py3,runtests,zeromq,salt&pr=&job=&cmd_args=R,n,f,F
09:26:37  ->  Pinging Codecov
09:26:37  https://codecov.io/upload/v4?package=bash-20200728-9fb7d93&token=secret&branch=master&commit=4684a0b585772e05df57c047d673e44229360774&build=380&build_url=https%3A%2F%2Fjenkinsci.saltstack.com%2Fjob%2Fpr-amazon2-py3-slow%2Fjob%2Fmaster%2F380%2F&name=amazon-2-py3-runtests-zeromq-salt&tag=&slug=saltstack%2Fsalt&service=jenkins&flags=amazon2,py3,runtests,zeromq,amazon2,py3,runtests,zeromq,salt&pr=&job=&cmd_args=R,n,f,F
09:26:38  ->  Uploading to
09:26:38  https://storage.googleapis.com/codecov/v4/raw/2020-08-13/94890F20B2A4896414641D1E2DACE6ED/4684a0b585772e05df57c047d673e44229360774/710d6648-8705-4127-8449-a58b73dc4d91.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=GOOG1EQX6OZVJGHKK3633AAFGLBUCOOATRACRQRQF6HMSMLYUP6EAD6XSWAAY%2F20200813%2FUS%2Fs3%2Faws4_request&X-Amz-Date=20200813T082638Z&X-Amz-Expires=10&X-Amz-SignedHeaders=host&X-Amz-Signature=c0f55b5ae5dbec53eceaf239726ca6c76b79b65d6b6f05c21c3a2b275c7ef853
09:26:38    % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
09:26:38                                   Dload  Upload   Total   Spent    Left  Speed
09:26:38  
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 1060k    0     0  100 1060k      0  3501k --:--:-- --:--:-- --:--:-- 3501k
09:26:38      -> View reports at https://codecov.io/github/saltstack/salt/commit/4684a0b585772e05df57c047d673e44229360774
09:26:38  [Pipeline] }

Additional Information

Let me know what else you need.

Hi @s0undt3ch, the success from the script is just that the coverage report successfully uploaded to our servers. However, we were unable to process that report. I’ve forwarded this to the product team to take a look.

Actually, I’m seeing something really strange here

I’m trying to source out these commits, but I’m having a hard time, mostly because the commit SHAs shown don’t match up to any commits on GitHub. Would you be able to help find a Jenkins build URL that would match one of these commits SHAs?

Since we’ve struggled, and still do, with codecov to report data to us, we disabled uploading coverage reports from pull requests.
These days we only upload coverage data on branch and nightly builds.
Maybe that’s it?
Although, the logic to upload on PRs, if we missed disabling it on PRs somewhere is the same, so they should have been properly identified.

I’ll try to see where that’s coming from…

Is there a size limit on the reports we upload?

About those commits which cannot be found, that was from an internal Jenkins instance which is now disabled from uploading coverage reports.

These days we only upload coverage data on branch and nightly builds.
Maybe that’s it?

That is probably related, but I’m not totally sure. It depends on if coverage is uploaded after a PR is merged into a branch.

Is there a size limit on the reports we upload?

There isn’t strictly, but I don’t think your uploads are overly large (< 10MB). Correct me if I’m wrong here.

About those commits which cannot be found, that was from an internal Jenkins instance which is now disabled from uploading coverage reports.

Ok, that makes sense. As I mentioned, this is a weird scenario, I’ve messaged the product team.

@s0undt3ch, I was taking a look at your repo and noticed you had been getting most of your commits through. Is this still an issue? And if not, I was curious what might have changed.

Something indeed change, but it wasn’t on our part.

On this commit we have uploaded reports, though, almost all are not accounted for, only 4, for the builds which have the fewer tests and are thus way faster. It was like this before we started getting errors.

On this we have way more builds getting reported, but their names have CF, which makes me think they are carry forward information?
So, on this last commit, is all information carry forward information?
Which builds were actually uploaded from the CI build?

So, yeah, we seem to be getting more data, specially more aligned to what the expected coverage is, but now I’m just lost at what actual data gets uploaded and what data was carried forward…

Are there any updates?

@s0undt3ch, sorry this is my mistake. I wasn’t able to pull our logs down from those commits, do you have a more recent instance of this happening?

The CF does denote carryforward, but I’m not sure why there are so many builds. We don’t typically display builds unless there is an upload event from the CI/CD.