500 unknown error

And just after empty re-commit it’s OK again: https://cirrus-ci.com/task/5345938556321792

And 500 again: https://cirrus-ci.com/task/5479510764158976

So, I’m definitely don’t know what to do. Sorry about multiple messages.

@AlexWayfer, I’m taking a look to see what’s happening here, but would you be able to first try to bash uploader?

We are in the works of upgrading the ruby uploader to a new endpoint, but it is a bit outdated. I’d like to make sure it’s not a different upload issue first.

As for token issues, we’ll be working over the next sprint to get tokenless Cirrus CI uploads.

These errors are not 100% reproducable, but I can try.

But I didn’t understand from documentation how to use bash uploader properly.

Example of fail: https://cirrus-ci.com/task/6181578374119424?command=test#L27

Should I write bash <(curl -s https://codecov.io/bash) before test command or after, and what type of reports I must provide?

You should put it after the test command and after coverage is collected. Can you try doing

bash <(curl -s https://codecov.io/bash) -s /tmp/cirrus-ci-build/coverage

It doesn’t work, even via && (in case if array of commands in YAML removes /tmp between them): https://cirrus-ci.com/task/4720200945762304

Sorry @AlexWayfer I had meant -s not -f to specify a directory

Still no: https://cirrus-ci.com/task/5340234948804608?command=test#L51

Not sure if it’s the same problem, but I had the same “500 Unknown” error for a few days. It got fixed when I signed in to codecov and gave it github permissions again. It’s been a while since I had last signed in. Maybe the token had expired? Don’t know but now it works again.

2 Likes

@AlexWayfer, just an update I’m working on making changes to the Ruby uploader to hopefully make debugging this issue easier. Thanks for your patience here.

OK, thanks, I appreciate it. I’m glad to help, but not sure that uploader can resolve the problem: 5xx are the codes about server errors (4xx about clients, yes). So, maybe you should check server logs.

1 Like

@tom Thanks for looking into this.

(Because I am a new user in this forum, I can only post 4 links in my comment, so some of the links below are intentionally broken; sorry about that.)

I am encountering problems (and have been for a couple of days now) that sound similar to what’s being discussed in this thread.

Here’s a repository where I’m running into the issues: github .com/davidrunger/david_runger/

Here’s a PR created earlier today where Codecov did not comment on the PR with coverage change information: https://github.com/davidrunger/david_runger/pull/2766 . Here’s the Travis build log for that PR: https://travis-ci.org/github/davidrunger/david_runger/builds/696176455 . In the Travis build log, we see this when the SimpleCov::Formatter::Codecov Ruby reporter attempted to upload the coverage data to Codecov:

{"meta": {"status": 500}, "error": {"reason": "Unknown", "context": null}}

In contrast / for reference, here’s a different PR today where Codecov did add a comment to the PR: https://github.com/davidrunger/david_runger/pull/2771 . Here’s the Travis build log for that PR: travis-ci .org/github/davidrunger/david_runger/builds/696221037 . In the Travis log output, instead of a 500 error when uploading the coverage data to Codecov, we see this:

{"uploaded": true, "url": "https://codecov.io/github/davidrunger/david_runger/commit/62df408aa12d025c475f09271113055ad8c59d53", "queued": true, "meta": {"status": 200}, "message": "Coverage reports upload successfully", "id": "49ca4ff5-d8b8-4dd0-ac2b-014a3a5d7bfb"}

It seems that whenever there is a 500 error in the Travis logs, Codecov doesn’t add a comment to the PR. When the logs indicate a 200 response for the upload to Codecov, then Codedov does add a comment to the PR.

Whether a 500 or a 200 occurs seems to be correlated with whether or not I am logged in to Codecov in my browser (Firefox) at the time that the upload is attempted. If I am logged in to Codecov when the coverage data upload is attempted in the Travis build, then we get a 200 and Codecov adds a comment. If I am not logged in to Codecov, then we get a 500 and Codecov does not add a comment.

(I don’t guess that whether or not I am logged in to codecov.io in my browser causes the 200 response vs a 500 response; rather, I suspect that there is some underlying issue that is the cause both of logging me out of codecov.io in my browser and the 500 error when Travis attempts to upload the coverage data?)

Furthermore, it seems that creating a PR in GitHub logs me out of codecov.io .

Therefore, if I create a PR and take no other action, Codecov ultimately does not add a comment to the PR.

If I do this instead, then Codecov will add a comment to the PR:

  1. go to codecov.io and observe that I am logged in
  2. create PR
  3. go to codecov.io and observe that I am no longer logged in
  4. delete my codecov.io cookies
  5. log in to Codecov (via GitHub OAuth) before the Travis build reaches the point where it attempts to upload coverage data to Codecov

As far as I can tell, from looking at recent PRs for that repo ( github .com/davidrunger/david_runger/ ), the issue seems to have emerged some time between 2020-06-06T00:16:49Z (when this PR was created github .com/davidrunger/david_runger/pull/2741 , to which Codecov added a comment) and 2020-06-06T08:56:06Z (when this PR was created github .com/davidrunger/david_runger/pull/2742 , which Codedov did not add a comment to and where we can see {"meta": {"status": 500}, "error": {"reason": "Unknown", "context": null}} in the Travis log output when attempting to upload coverage data to Codecov).

I hope that this info helps to debug the issue! Thanks again!

2 Likes

I have determined that the issue that I am encountering doesn’t seem to have anything to do with the Ruby uploader vs the bash uploader.


Success case

I created this PR https://github.com/davidrunger/david_runger/pull/2781 and followed the 5-step process that I mentioned above (logging in to codecov.io after creating the PR and before the Travis build attempted to upload the coverage data to Codecov).

That PR adds this as the last step of the Travis build flow:

bash <(curl -s https://codecov.io/bash) -f "$(pwd)/coverage/.resultset.json"

As we can see in the Travis build log https://travis-ci.org/github/davidrunger/david_runger/builds/696352922 , both the Ruby coverage formatter/reporter and the bash uploader were able to send the result to Codecov successfully.

Ruby:

{"uploaded": true, "url": "https://codecov.io/github/davidrunger/david_runger/commit/1b17b5652fd18fd4d9798aca8824d2fbf4a2871c", "queued": true, "meta": {"status": 200}, "message": "Coverage reports upload successfully", "id": "02d480e1-82f5-43a6-8cff-c300ec4f6221"}

Bash:

==> Uploading reports
    url: https://codecov.io
        query: branch=codecov-bash-uploader&commit=1b17b5652fd18fd4d9798aca8824d2fbf4a2871c&build=6529.1&build_url=&name=&tag=&slug=davidrunger%2Fdavid_runger&service=travis&flags=&pr=false&job=696352923
        -> Pinging Codecov
        https://codecov.io/upload/v4?package=bash-20200602-f809a24&token=secret&branch=codecov-bash-uploader&commit=1b17b5652fd18fd4d9798aca8824d2fbf4a2871c&build=6529.1&build_url=&name=&tag=&slug=davidrunger%2Fdavid_runger&service=travis&flags=&pr=false&job=696352923
    -> Uploading
        -> View reports at https://codecov.io/github/davidrunger/david_runger/commit/1b17b5652fd18fd4d9798aca8824d2fbf4a2871c

As expected, Codecov also added a comment to that PR.


Failure case

I also created this PR: https://github.com/davidrunger/david_runger/pull/2780 . After creating that PR (and being logged out of codecov.io), I did not log in to codecov.io. As we can see from the Travis build log https://travis-ci.org/github/davidrunger/david_runger/builds/696349662 , both the Ruby and bash attempts to upload code coverage data to Codecov failed.

Ruby:

{"meta": {"status": 500}, "error": {"reason": "Unknown", "context": null}}

bash:

==> Uploading reports
+say '    \033[0;90murl:\033[0m https://codecov.io'
+echo -e '    \033[0;90murl:\033[0m https://codecov.io'
    url: https://codecov.io
    +say '    \033[0;90mquery:\033[0m branch=bash-codecov-uploader&commit=033212e9b75bd033e55a63479f5c724a746dab99&build=6528.1&build_url=&name=&tag=&slug=davidrunger%2Fdavid_runger&service=travis&flags=&pr=false&job=696349663'
    +echo -e '    \033[0;90mquery:\033[0m branch=bash-codecov-uploader&commit=033212e9b75bd033e55a63479f5c724a746dab99&build=6528.1&build_url=&name=&tag=&slug=davidrunger%2Fdavid_runger&service=travis&flags=&pr=false&job=696349663'
    query: branch=bash-codecov-uploader&commit=033212e9b75bd033e55a63479f5c724a746dab99&build=6528.1&build_url=&name=&tag=&slug=davidrunger%2Fdavid_runger&service=travis&flags=&pr=false&job=696349663
    ++tr -d ' '
    ++echo 'package=bash-20200602-f809a24&token=secret&branch=bash-codecov-uploader&commit=033212e9b75bd033e55a63479f5c724a746dab99&build=6528.1&build_url=&name=&tag=&slug=davidrunger%2Fdavid_runger&service=travis&flags=&pr=false&job=696349663'
+queryNoToken='package=bash-20200602-f809a24&token=secret&branch=bash-codecov-uploader&commit=033212e9b75bd033e55a63479f5c724a746dab99&build=6528.1&build_url=&name=&tag=&slug=davidrunger%2Fdavid_runger&service=travis&flags=&pr=false&job=696349663'
++tr -d ' '
++echo 'package=bash-20200602-f809a24&token=[secure]&branch=bash-codecov-uploader&commit=033212e9b75bd033e55a63479f5c724a746dab99&build=6528.1&build_url=&name=&tag=&slug=davidrunger%2Fdavid_runger&service=travis&flags=&pr=false&job=696349663'
+query='package=bash-20200602-f809a24&token=[secure]&branch=bash-codecov-uploader&commit=033212e9b75bd033e55a63479f5c724a746dab99&build=6528.1&build_url=&name=&tag=&slug=davidrunger%2Fdavid_runger&service=travis&flags=&pr=false&job=696349663'
+'[' 1 = 1 ']'
+i=0
+'[' 0 -lt 4 ']'
+i=1
+say '    \033[0;90m->\033[0m Pinging Codecov'
+echo -e '    \033[0;90m->\033[0m Pinging Codecov'
    -> Pinging Codecov
    +say 'https://codecov.io/upload/v4?package=bash-20200602-f809a24&token=secret&branch=bash-codecov-uploader&commit=033212e9b75bd033e55a63479f5c724a746dab99&build=6528.1&build_url=&name=&tag=&slug=davidrunger%2Fdavid_runger&service=travis&flags=&pr=false&job=696349663'
    +echo -e 'https://codecov.io/upload/v4?package=bash-20200602-f809a24&token=secret&branch=bash-codecov-uploader&commit=033212e9b75bd033e55a63479f5c724a746dab99&build=6528.1&build_url=&name=&tag=&slug=davidrunger%2Fdavid_runger&service=travis&flags=&pr=false&job=696349663'
https://codecov.io/upload/v4?package=bash-20200602-f809a24&token=secret&branch=bash-codecov-uploader&commit=033212e9b75bd033e55a63479f5c724a746dab99&build=6528.1&build_url=&name=&tag=&slug=davidrunger%2Fdavid_runger&service=travis&flags=&pr=false&job=696349663
++curl -X POST -H 'X-Reduced-Redundancy: false' -H 'X-Content-Type: application/x-gzip' 'https://codecov.io/upload/v4?package=bash-20200602-f809a24&token=[secure]&branch=bash-codecov-uploader&commit=033212e9b75bd033e55a63479f5c724a746dab99&build=6528.1&build_url=&name=&tag=&slug=davidrunger%2Fdavid_runger&service=travis&flags=&pr=false&job=696349663'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                   Dload  Upload   Total   Spent    Left  Speed
                                   100    17  100    17    0     0     16      0  0:00:01  0:00:01 --:--:--    16
                                 +res='HTTP 500
Unknown'
++head -1
++echo 'HTTP 500
Unknown'
++cut '-d ' -f2
++grep 'HTTP '
+status=500
+'[' 500 = '' ']'
+'[' 500 = 400 ']'
+say '    \033[0;90m->\033[0m Sleeping for 30s and trying again...'
+echo -e '    \033[0;90m->\033[0m Sleeping for 30s and trying again...'
    -> Sleeping for 30s and trying again...

As expected, Codecov did not add a comment to that PR.

Hi @davidrunger, thanks for the incredibly detailed logs here. I’ll take a look shortly to see what went wrong here.

1 Like

Hi all, thanks for the help here. We have identified the root cause and are working on a fix. For now as a quick workaround, logging out, clearing cookies, and logging in should fix these issues.

The incident will be posted here

1 Like

Thank you so much, @tom !

I think that y’all have fixed it? At least, when I created this PR https://github.com/davidrunger/david_runger/pull/2802 , the creation of that PR did not log me out of codecov.io in my browser (as the creation of PRs has been doing for the past several days), and the code coverage data was uploaded successfully to Codecov in the Travis build, and Codecov added a comment to that PR with coverage information.

Thanks again for your work on this issue!

2 Likes

@davidrunger we pushed a fix that we believe should work for this issue. I’ll keep this thread open as we are monitoring any changes. Let me know if you run into this issue again, but :crossed_fingers:it should be resolved.

1 Like

I’m not getting any errors for now, so I’m closing a branch with bash uploader. Thank you for fixes.

1 Like