CircleCI orb fails to upload quite often

Support Bug Fixes
1

Description

I’m seeing that the uploading of reports fails quite often on our project. Generating the reports works just fine, but when we run the codecov/upload step, we get failures. This causes some wonkiness in our reports because an entire section is missing.

Repository

Private repo

CI/CD

CircleCI

Uploader

CircleCI Orb GitHub - codecov/codecov-circleci-orb

Commit SHAs

We’re using version 1.1.0 of the orb. But this was happening on the previous release as well.

Codecov YAML

coverage:
  range: 70..100
  round: down
  precision: 2

fixes:
  - 'workspace/::'

Codecov Output

#!/bin/bash -eo pipefail
curl -s https://codecov.io/bash | bash -s -- \
  -f "packages/web/coverage/coverage-final.json" \
  -t "${CODECOV_TOKEN}" \
  -n "${CIRCLE_BUILD_NUM}" \
  -y ".codecov.yml" \
  -F "web" \
  -Z || echo 'Codecov upload failed'
Codecov upload failed
CircleCI received exit code 0

Steps to Reproduce

  1. Add orb to CircleCI config. Here’s our CircleCI config.yml (note: all of our packages use the flags option):
- codecov/upload:
    file: packages/web/coverage/coverage-final.json
    flags: web
  1. Wait for CircleCI to build
  2. Observe some uploads fail

Expected behavior: [What you expect to happen]

Not to fail.

Actual behavior: [What actually happens]

Fails.

Flakiness? [Does this happen all the time or only sometimes?]

Only sometimes, hard to give a percentage… maybe 5-10% of the time.

Additional Information

It’s possible this is a timeout because this particular step took 2m9s whereas our other 6 uploads took 2s each and succeeded.

2

Thanks @joelpoloney, could you supply a commit SHA for this so I can dig in?

4

@joelpoloney, we had an incident earlier that has been resolved and might be why you were experiencing this issue. Would you be able to log out of Codecov.io, clear your cookies, and log back in? This should resolve your problem.

5

I am also experiencing upload failures when using CircleCI in a private GitHub repo. The codecov stage takes about 30mins and then it fails without publishing the report. Any idea how this can be fixed?
==> Gzipping contents
==> Uploading reports
url: https://codecov.io
query: branch=pull%2F33&commit=83559ae44b08bb506b158bd7cd90f6f38b1d2278&build=89&…
→ Pinging Codecov
https://codecov.io/upload/v4?…
→ Uploading
X> Failed to upload
→ Sleeping for 30s and trying again…
→ Pinging Codecov
→ Uploading
X> Failed to upload
→ Sleeping for 30s and trying again…
→ Uploading to Codecov
→ Sleeping for 30s and trying again…
→ Sleeping for 30s and trying again…
→ Sleeping for 30s and trying again…
→ Sleeping for 30s and trying again…
X> Failed to upload coverage reports

6

Hi @emdob, what uploader are you using and what version?

7

@tom This issue isn’t with the Codecov website or cookies or anything like that. It’s specifically when the CircleCI orb tries uploading and after 2 minutes it fails. This is still consistently happening a few times a week for us.

8

Here’s a git commit hash for one that failed 2/7 times from this morning: 8df1469538963bddc28210cdbb63bfee5c88578a.

This is what we see:

Screen Shot 2020-07-14 at 11.14.33 AM
Screen Shot 2020-07-14 at 11.14.33 AM3288×1524 290 KB

9

@joelpoloney, is there no other output on that failed bash upload? We typically expect to at least see the bash script running.

10

Nope that’s all there is. Our CircleCI config.yml has steps that look like:

      - codecov/upload:
          file: packages/notifications/coverage/coverage-final.json
          flags: notifications

and this is all we see :man_shrugging:We’re going to try moving off the orb and writing our own script that wraps the codecov npm commands and will see if that’s any better for us.

11

@joelpoloney, before you do that, would you be open to trying the bash script? I’d at least like to see more error logs on why this is happening, and I think directly using the script should give us some output.