Cannot upload to codecov after upgrade to 0.2.8 codecov-ruby gem

Description

Cannot upload to codecov using bash or CircleCI integration. Upload was working prior to upgrade to 0.2.8, forced because the old version of the gem was yanked. Bash upload was not attempted prior to this upgrade. FWIW, when I catch the s3 link before it expires, I get a bad signature error:

<Error>
<Code>SignatureDoesNotMatch</Code>
<Message>The request signature we calculated does not match the signature you provided. Check your Google secret key and signing method.</Message>
<StringToSign>AWS4-HMAC-SHA256 20200825T210637Z 20200825/US/s3/aws4_request 9659cf192fd48f05eea588eec6315f05125b08adf1f3a72b041e794308ac1464</StringToSign>
<CanonicalRequest>GET /codecov/v4/raw/2020-08-25/173FB0B63EE4AA9D7A802BC2222A0FA6/f0cb8dbb48b106864893f7e2181021d40a0b0b76/63cdbd10-8270-4d5f-ac51-ab393ea09b96.txt X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=GOOG1EQX6OZVJGHKK3633AAFGLBUCOOATRACRQRQF6HMSMLYUP6EAD6XSWAAY%2F20200825%2FUS%2Fs3%2Faws4_request&X-Amz-Date=20200825T210637Z&X-Amz-Expires=10&X-Amz-SignedHeaders=host host:storage.googleapis.com host UNSIGNED-PAYLOAD</CanonicalRequest>
</Error>

Repository

Private

CI/CD

CircleCI

Uploader

CircleCI orb and bash uploader

Commit SHAs

Ex a879d0df0ab64feb85df77469d3761b90c32177e

Codecov YAML

Not using

Codecov Output

#!/bin/bash -eo pipefail
curl -s https://codecov.io/bash | bash -s –
-f “coverage/.resultset.json”
-t “${CODECOV_TOKEN}”
-n “${CIRCLE_BUILD_NUM}”
-F “”
-Z || echo ‘Codecov upload failed’

  _____          _
 / ____|        | |
| |     ___   __| | ___  ___ _____   __
| |    / _ \ / _` |/ _ \/ __/ _ \ \ / /
| |___| (_) | (_| |  __/ (_| (_) \ V /
 \_____\___/ \__,_|\___|\___\___/ \_/
                              Bash-20200728-9fb7d93


==> Circle CI detected.
    project root: .
    Yaml not found, that's ok! Learn more at http://docs.codecov.io/docs/codecov-yaml
    -> Found 1 reports
==> Detecting git/mercurial file structure
==> Reading reports
    + coverage/.resultset.json bytes=199649
==> Appending adjustments
    https://docs.codecov.io/docs/fixing-reports
    + Found adjustments
==> Gzipping contents
==> Uploading reports
    url: https://codecov.io
    query: branch=fix%2Fcodecov-experiment&commit=b91ef056db7b5a81078cee9db648b9bc583ba583&build=2921&build_url=&name=2921&tag=&slug=majority-labs%2Ffuture-majority&service=circleci&flags=&pr=&job=0&cmd_args=f,t,n,F,Z
->  Pinging Codecov
https://codecov.io/upload/v4?package=bash-20200728-9fb7d93&token=secret&branch=fix%2Fcodecov-experiment&commit=b91ef056db7b5a81078cee9db648b9bc583ba583&build=2921&build_url=&name=2921&tag=&slug=majority-labs%2Ffuture-majority&service=circleci&flags=&pr=&job=0&cmd_args=f,t,n,F,Z
->  Uploading to
https://storage.googleapis.com/codecov/v4/raw/2020-08-25/173FB0B63EE4AA9D7A802BC2222A0FA6/b91ef056db7b5a81078cee9db648b9bc583ba583/e21f1d4c-4c86-4818-8276-699240433b8a.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=GOOG1EQX6OZVJGHKK3633AAFGLBUCOOATRACRQRQF6HMSMLYUP6EAD6XSWAAY%2F20200825%2FUS%2Fs3%2Faws4_request&X-Amz-Date=20200825T173334Z&X-Amz-Expires=10&X-Amz-SignedHeaders=host&X-Amz-Signature=38c22f9f45fd18618d5367b366cc0c151ceab88e8a76b920014b09ab16bfa5a8
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  102k    0     0  100  102k      0   258k --:--:-- --:--:-- --:--:--  259k
    -> View reports at https://codecov.io/github/majority-labs/future-majority/commit/b91ef056db7b5a81078cee9db648b9bc583ba583
CircleCI received exit code 0

Steps to Reproduce

  1. Run CircleCI build with Codecov orb upload step that uploads .resultset.json
    OR

  2. bash <(curl -s https://codecov.io/bash) -f coverage/.resultset.json -t XXXXXXXXX

Expected behavior: Reports visible on codecov

Actual behavior: There was an error processing coverage reports.

Flakiness? Always, since update

Additional Information

Please let me know what else I can include to help troubleshoot. Thank you!

Hi @sarahstrong, would you mind sharing the Codecov output from using the Ruby gem?

Here’s the output of codecov invoked locally:


/ | | |
| | ___ __| | ___ ___ _____ __
| | / _ \ / `|/ _ / / _ \ \ / /
| |
| (
) | (
| | __/ (
| (
) \ V /
______/ _,|_|____/ _/
Ruby-0.2.8
x> No CI provider detected.
==> Appending file network
==> Gzipping contents
==> Uploading reports
url: https://codecov.io
query: token=secret&flags&package=ruby-0.2.8&branch=fix%2Fcodecov-experiment&commit=b91ef056db7b5a81078cee9db648b9bc583ba583
→ Pinging Codecov
https://codecov.io/upload/v4?token=secret&flags&package=ruby-0.2.8&branch=fix%2Fcodecov-experiment&commit=b91ef056db7b5a81078cee9db648b9bc583ba583
→ Uploading to
https://storage.googleapis.com/codecov/v4/raw/2020-08-26/173FB0B63EE4AA9D7A802BC2222A0FA6/b91ef056db7b5a81078cee9db648b9bc583ba583/b33eac7d-4457-40f7-bfbe-6044d5ab7f95.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=GOOG1EQX6OZVJGHKK3633AAFGLBUCOOATRACRQRQF6HMSMLYUP6EAD6XSWAAY%2F20200826%2FUS%2Fs3%2Faws4_request&X-Amz-Date=20200826T202626Z&X-Amz-Expires=10&X-Amz-SignedHeaders=host&X-Amz-Signature=6f4bc6a9efe1db99f17ecf9b6646acd4f2ad29bf250762a032963cc1c1c289d7
Error uploading coverage reports to Codecov. Sorry
TypeError
no implicit conversion of nil into String
→ Could not upload reports via v4 API, defaulting to v2
→ Uploading to Codecov
https://codecov.io/upload/v2?token=secret&flags&package=ruby-0.2.8&branch=fix%2Fcodecov-experiment&commit=b91ef056db7b5a81078cee9db648b9bc583ba583
Error uploading coverage reports to Codecov. Sorry
TypeError
no implicit conversion of nil into String

Codecov invoked via

if ENV[‘CODECOV_TOKEN’]
require ‘simplecov’
SimpleCov.start ‘rails’ do
enable_coverage :branch
end

require ‘codecov’
SimpleCov.formatter = SimpleCov::Formatter::Codecov
end

I traced the nil into string issue and it appeared to be inside retry_request when invoked on line 404 of codecov.rb. Here’s the backtrace:

[“/home/sarah/.rbenv/versions/2.6.0/lib/ruby/2.6.0/openssl/buffering.rb:383:in `syswrite_nonblock’”, “/home/sarah/.rbenv/versions/2.6.0/lib/ruby/2.6.0/openssl/buffering.rb:383:in `write_nonblock’”, “/home/sarah/.rbenv/versions/2.6.0/lib/ruby/2.6.0/net/protocol.rb:277:in `block in write0’”, “/home/sarah/.rbenv/versions/2.6.0/lib/ruby/2.6.0/net/protocol.rb:275:in `each’”, “/home/sarah/.rbenv/versions/2.6.0/lib/ruby/2.6.0/net/protocol.rb:275:in `each_with_index’”, “/home/sarah/.rbenv/versions/2.6.0/lib/ruby/2.6.0/net/protocol.rb:275:in `write0’”, “/home/sarah/.rbenv/versions/2.6.0/lib/ruby/2.6.0/net/protocol.rb:248:in `block in write’”, “/home/sarah/.rbenv/versions/2.6.0/lib/ruby/2.6.0/net/protocol.rb:265:in `writing’”, “/home/sarah/.rbenv/versions/2.6.0/lib/ruby/2.6.0/net/protocol.rb:247:in `write’”, “/home/sarah/.rbenv/versions/2.6.0/lib/ruby/2.6.0/net/http/generic_request.rb:189:in `send_request_with_body’”, “/home/sarah/.rbenv/versions/2.6.0/lib/ruby/2.6.0/net/http/generic_request.rb:123:in `exec’”, “/home/sarah/.rbenv/versions/2.6.0/lib/ruby/2.6.0/net/http.rb:1507:in `block in transport_request’”, “/home/sarah/.rbenv/versions/2.6.0/lib/ruby/2.6.0/net/http.rb:1506:in `catch’”, “/home/sarah/.rbenv/versions/2.6.0/lib/ruby/2.6.0/net/http.rb:1506:in `transport_request’”, “/home/sarah/.rbenv/versions/2.6.0/lib/ruby/2.6.0/net/http.rb:1479:in `request’”, “/home/sarah/.rbenv/versions/2.6.0/lib/ruby/gems/2.6.0/gems/airbrake-10.0.5/lib/airbrake/rails/net_http.rb:11:in `block in request’”, “/home/sarah/.rbenv/versions/2.6.0/lib/ruby/gems/2.6.0/gems/airbrake-10.0.5/lib/airbrake/rack.rb:23:in `capture_timing’”, “/home/sarah/.rbenv/versions/2.6.0/lib/ruby/gems/2.6.0/gems/airbrake-10.0.5/lib/airbrake/rails/net_http.rb:10:in `request’”, “/home/sarah/.rbenv/versions/2.6.0/lib/ruby/2.6.0/net/http.rb:1472:in `block in request’”, “/home/sarah/.rbenv/versions/2.6.0/lib/ruby/2.6.0/net/http.rb:920:in `start’”, “/home/sarah/.rbenv/versions/2.6.0/lib/ruby/2.6.0/net/http.rb:1470:in `request’”, “/home/sarah/.rbenv/versions/2.6.0/lib/ruby/gems/2.6.0/gems/airbrake-10.0.5/lib/airbrake/rails/net_http.rb:11:in `block in request’”, “/home/sarah/.rbenv/versions/2.6.0/lib/ruby/gems/2.6.0/gems/airbrake-10.0.5/lib/airbrake/rack.rb:23:in `capture_timing’”, “/home/sarah/.rbenv/versions/2.6.0/lib/ruby/gems/2.6.0/gems/airbrake-10.0.5/lib/airbrake/rails/net_http.rb:10:in `request’”, “/home/sarah/.rbenv/versions/2.6.0/lib/ruby/gems/2.6.0/gems/codecov-0.2.8/lib/codecov.rb:293:in `retry_request’”, “/home/sarah/.rbenv/versions/2.6.0/lib/ruby/gems/2.6.0/gems/codecov-0.2.8/lib/codecov.rb:405:in `upload_to_v4’”, “/home/sarah/.rbenv/versions/2.6.0/lib/ruby/gems/2.6.0/gems/codecov-0.2.8/lib/codecov.rb:359:in `upload_to_codecov’”, “/home/sarah/.rbenv/versions/2.6.0/lib/ruby/gems/2.6.0/gems/codecov-0.2.8/lib/codecov.rb:458:in `format’”, “/home/sarah/.rbenv/versions/2.6.0/lib/ruby/gems/2.6.0/gems/simplecov-0.19.0/lib/simplecov/result.rb:51:in `format!'”, “/home/sarah/.rbenv/versions/2.6.0/lib/ruby/gems/2.6.0/gems/simplecov-0.19.0/lib/simplecov/configuration.rb:196:in `block in at_exit’”, “/home/sarah/.rbenv/versions/2.6.0/lib/ruby/gems/2.6.0/gems/simplecov-0.19.0/lib/simplecov.rb:189:in `run_exit_tasks!'”, “/home/sarah/.rbenv/versions/2.6.0/lib/ruby/gems/2.6.0/gems/simplecov-0.19.0/lib/simplecov.rb:179:in `at_exit_behavior’”, “/home/sarah/.rbenv/versions/2.6.0/lib/ruby/gems/2.6.0/gems/simplecov-0.19.0/lib/simplecov/defaults.rb:27:in `block in <top (required)>'”]

I was suspicious about Airbrake (which is an error handler we use) but the issue persists with it disabled.

@sarahstrong, thanks for working with me here. I am not able to reproduce on my end, but would you be able to add a log statement at line 347: puts report['codecov']?

I suspect there might be something wrong there as I can’t figure out what other part of the retry could actually be broken.

The puts just logs the entire report, I’ll copy that at the bottom of my message as [1].

When I retried with a pristine gem aside from the puts my results were slightly different. Looks like a failed upload is not recoverable.

Error uploading coverage reports to Codecov. Sorry
TypeError
no implicit conversion of nil into String
Traceback (most recent call last):
7: from /home/sarah/.rbenv/versions/2.6.0/lib/ruby/gems/2.6.0/gems/simplecov-0.19.0/lib/simplecov/defaults.rb:27:in `block in <top (required)>'
6: from /home/sarah/.rbenv/versions/2.6.0/lib/ruby/gems/2.6.0/gems/simplecov-0.19.0/lib/simplecov.rb:179:in `at_exit_behavior'
5: from /home/sarah/.rbenv/versions/2.6.0/lib/ruby/gems/2.6.0/gems/simplecov-0.19.0/lib/simplecov.rb:189:in `run_exit_tasks!'
4: from /home/sarah/.rbenv/versions/2.6.0/lib/ruby/gems/2.6.0/gems/simplecov-0.19.0/lib/simplecov/configuration.rb:196:in `block in at_exit'
3: from /home/sarah/.rbenv/versions/2.6.0/lib/ruby/gems/2.6.0/gems/simplecov-0.19.0/lib/simplecov/result.rb:51:in `format!'
2: from /home/sarah/.rbenv/versions/2.6.0/lib/ruby/gems/2.6.0/gems/codecov-0.2.8/lib/codecov.rb:458:in `format'
1: from /home/sarah/.rbenv/versions/2.6.0/lib/ruby/gems/2.6.0/gems/codecov-0.2.8/lib/codecov.rb:360:in `upload_to_codecov'
/home/sarah/.rbenv/versions/2.6.0/lib/ruby/gems/2.6.0/gems/codecov-0.2.8/lib/codecov.rb:406:in `upload_to_v4': undefined method `body' for nil:NilClass (NoMethodError)

[1] Report data:
Gemfile
config/boot.rb
config/puma.rb
config/credentials.yml.enc
config/environments/staging.rb
config/environments/development.rb
config/environments/test.rb
config/environments/production.rb
config/spring.rb
config/environment.rb

[snip test files]

public/packs/application-54a9e3f062d355297310.js
public/packs/hello_react-a6c5cbd51f0ea3dc62d9.js.map
public/packs/application-ecd3e76008a791f7597b.js
public/packs/application-04cdae5bec23de54c79a.js.map

[snip hundreds of versions of compiled packs]

[snip actual application files]

tmp/cache/assets/sprockets/v3.0/lQ/lQ2jH4wShAXSal1SaEhkJry_qmGE5C57umTyeZoP9pg.cache
tmp/cache/assets/sprockets/v3.0/7m/7mpiV9awO63AcR9KCYfye9g3fRn4Z_HuCpA_mqUj0To.cache

[snip thousands of tmp files]

[snip actual application and test files]

<<<<<< network
{“coverage”:{“app/admin/accounts.rb”:[null,null,null,1,1,1,1,1,1,null,null,null,1,null,1,3,3,3,3,6,null,3,3,3,3,3,null,null,1,6,6,6,6,6,6,6,6,6,6,6,6,6,6,null,null,null,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,null,0,null,null,1,6,null,null,1,1,1,1,1,null,null,null,1,6,null,null,1,1,1,1,1,null,null,null,1,6,null,null,1,1,1,1,1,null,null,null]

[snip actual results for all files]

Is there a filesize limit or an upload time limit? From this I’m guessing my crazy amount of tempfiles is the issue. I’ll try setting a codecov.yml with an ignore path and get back to you about whether that worked.

The commit that made the codecov.yml change succeeded but all subsequent commits are failing. I’m a little stumped. Example succeeding build: Codecov Example failing build: Codecov

@tom , I’m able to get reports to load successfully in the following circumstances:

#1 I have not yet uploaded through Circle, only locally
#2 I make the following changes to codecov.rb, line 499:

invalid_directories = [
  'node_modules/',
  'tmp/',
  'public/',
  'storage/'
]

puts ['==>'.green, 'Appending file network'].join(' ')
network = []
Dir['**/*'].keep_if do |file|
  if File.file?(file) && !file.end_with?(*invalid_file_types) && !invalid_directories.any? { |dir| file.include?(dir) }
    network.push(file)
  end
end

I’m going to file this as an issue on the codecov-ruby gem as well. My current theory is that the file network component is producing far too big a file, and if a single file that is too big has been uploaded to the commit’s endpoint, then that commit’s whole report cannot be processed.

@sarahstrong, I’m working on a small fix here, but this doesn’t get to the root cause. Thanks for the above, I’ll investigate more.

@sarahstrong, I have incorporated your changes into the PR, thanks!

Awesome, thanks. Maybe if the gem allowed disable file network (I’m not sure what that’s for so maybe it’s necessary?) or configure ignore paths for file network it could fix this issue?

Edit: Oh missed your followup, looks good, thanks!

@sarahstrong, unfortunately file network is critical for the v4 endpoint to work. I’ll hopefully get this in today.

@sarahstrong, I just released 0.2.10, would you be able to see if that works for you?

Wow so fast, thank you! I’ll get back to you after some testing.

@tom thank you for all your help. It’s currently working unmodified locally, but not through CircleCI. Example local output:

                           Ruby-0.2.10

x> No CI provider detected.
==> Appending file network
==> Gzipping contents
==> Uploading reports
url: https://codecov.io
query: token=secret&flags&package=ruby-0.2.10&branch=fix%2Fcodecov-again&commit=a30c1e2acef5693bf30ace34f51b042738d74779
→ Pinging Codecov
https://codecov.io/upload/v4?token=secret&flags&package=ruby-0.2.10&branch=fix%2Fcodecov-again&commit=a30c1e2acef5693bf30ace34f51b042738d74779
→ Uploading to
https://storage.googleapis.com/codecov/v4/raw/2020-09-04/173FB0B63EE4AA9D7A802BC2222A0FA6/a30c1e2acef5693bf30ace34f51b042738d74779/13af621f-c971-4f11-94aa-09cbebae735f.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=GOOG1EQX6OZVJGHKK3633AAFGLBUCOOATRACRQRQF6HMSMLYUP6EAD6XSWAAY%2F20200904%2FUS%2Fs3%2Faws4_request&X-Amz-Date=20200904T201345Z&X-Amz-Expires=10&X-Amz-SignedHeaders=host&X-Amz-Signature=32187332665123982ab842587563296a3b0c3e728f0fde9364c35f8db79068fb
View reports at https://codecov.io/github/majority-labs/future-majority/commit/a30c1e2acef5693bf30ace34f51b042738d74779

Example Circle output:

                           Ruby-0.2.10

==> Circle CI detected
==> Appending file network
==> Gzipping contents
==> Uploading reports
url: https://codecov.io
query: token=secret&flags&package=ruby-0.2.10&service=circleci&build=3050&job=0&slug=majority-labs%2Ffuture-majority&pr&branch=fix%2Fcodecov-again&commit=6c8fd02154398bca74b9b3188073d0abbc970412
→ Pinging Codecov
https://codecov.io/upload/v4?token=secret&flags&package=ruby-0.2.10&service=circleci&build=3050&job=0&slug=majority-labs%2Ffuture-majority&pr&branch=fix%2Fcodecov-again&commit=6c8fd02154398bca74b9b3188073d0abbc970412
→ Uploading to
https://storage.googleapis.com/codecov/v4/raw/2020-09-04/173FB0B63EE4AA9D7A802BC2222A0FA6/6c8fd02154398bca74b9b3188073d0abbc970412/c6c921b1-b338-4cd8-b65a-f6a8d769a7ec.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=GOOG1EQX6OZVJGHKK3633AAFGLBUCOOATRACRQRQF6HMSMLYUP6EAD6XSWAAY%2F20200904%2FUS%2Fs3%2Faws4_request&X-Amz-Date=20200904T193802Z&X-Amz-Expires=10&X-Amz-SignedHeaders=host&X-Amz-Signature=c682364df3eb30a4cd26d8d530bbfa3771329449c35c781753d80ed18a33419a
View reports at https://codecov.io/github/majority-labs/future-majority/commit/6c8fd02154398bca74b9b3188073d0abbc970412
CircleCI received exit code 0

I’ve also tried uploading reports through the Codecov Circle orb, same results: “There was an error processing coverage reports.”

Hi @sarahstrong, thanks for this. I released 0.2.11 which I think will help mitigate the issue. Would you mind checking?

@tom thank you so much, it worked! Let me just get the version bump into master and confirm it works against commits including tracked file changes.

1 Like

@sarahstrong woo hoo!! Glad it worked and thank you so much for all the help :pray:!

1 Like

Confirmed working with new PRs! Thanks so so much @tom, you really went above and beyond.

1 Like