PR has caused desynchronization of master branch

Description

It appears that our master branch over at Authelia has gone out of sync, with a little investigation it seems that when someone made a PR from their forked master branch this somehow has assumed itself as the main repos master branch.

Repository

CI/CD

Buildkite

Uploader

The following commands are being run:

bash <(curl -s --connect-timeout 10 --retry 10 --retry-max-time 0 https://codecov.io/bash) -Z -c -f "coverage.txt" -F backend
bash <(curl -s --connect-timeout 10 --retry 10 --retry-max-time 0 https://codecov.io/bash) -Z -c -F frontend

Commit SHAs

This is the commit sha of HEAD @ master at the time of this post:
b770d3cb6c572ec313f16e44dcba9d2ef1cc92b6

It also suggests master has not been updated in 3 days.

Codecov YAML

codecov:
  require_ci_to_pass: yes

comment:
  layout: "reach, diff, flags, files"
  behavior: default
  require_changes: no

coverage:
  precision: 2
  round: down
  range: "70...100"
  status:
    project:
      default: off
      backend:
        base: auto
        flags:
          - backend
      frontend:
        base: auto
        flags:
          - frontend

flags:
  backend:
    paths:
      - "cmd/authelia/"
      - "internal/"
      - "!internal/suites/"
  frontend:
    paths:
      - "web/"

parsers:
  gcov:
    branch_detection:
      conditional: yes
      loop: yes
      method: no
      macro: no

Codecov Output

You can see all of the outputs from our publically available CI:

There is also an additional upload on each of the individual suites except the Kubernetes Suite.

Additional Information

The PR in question that seems to have desynced our master branch is Example of authelia lite on docker swarm by xplreitr · Pull Request #1112 · authelia/authelia · GitHub. You can see that I merged in the main repos master branch into the fork 3 days ago, this aligns with all the other times that are suggested in the codecov branches menu and other areas.

Any help would be greatly appreciated and advice on how we can prevent this from happening in future.

Hi @nightah, thanks for the detailed writeup here. I think the master branch is actually a red herring, because what’s more interesting is that we stopped getting coverage reports uploaded for commits to master for the past 15 days (here). Is there any reason that you might be running into this since then?

Hi @tom I honestly have no idea why this would be the case.

We never run/upload coverage to master directly. All of our commits are PR’d from branches or external forks (where coverage is run and uploaded) and eventually squash merged into master.

This has been the case since the introduction of Codecov to our repo. Nothing else has changed since 15 days ago, you should be able to see this in our commit history too.

Hi @nightah, if I had to guess, it might have something to do with this line here. I’ll be honest, my bash isn’t great so I can’t be sure, but it looks like you might no longer by uploading coverage on master branch runs?

I think you may have missed what I mentioned in my previous comment @tom. We have never uploaded coverage for the master branch. That conditional exists because the only time a full set of coverage is run is outside of a deployment (any push that isn’t master or a tagged release).
So all of the runs that are appearing in the master branch right now are PR’d from other branches, never direct runs on master.

Here’s the PR in question where we enabled coverage collection/codecov: [CI] Add Codecov support by nightah · Pull Request #1065 · authelia/authelia · GitHub, you can see that the conditional has existing from 21 days ago and has not changed, so we have never actually run coverage collection/uploading on master and have not once since the inception of codecov to our repo.

Right now it says master hasn’t been updated in several days. I’m 99% certain that if I go into that PR I mentioned initially and update it, our master branch will say it has been updated, as this is what happened last time. I can elaborate further with some screenshots if that might help?

How can we get to the bottom of this?

Actually to further illustrate the problem you can see another commit/PR that was made just today by another user on a forked repos master branch https://github.com/authelia/authelia/pull/1157, 11 hours prior to this post.

You can see our “default” master branch Codecov also now says it was last updated 11 hours ago and has that respective user tagged the last person perform said update Updated 11 hours ago by BankaiNoJutsu.

I suspect this behaviour is not correct, as a forked repo should never be assumed as the default branch set in the codecov settings page. My understanding is that this branch will then be used as the pseudo-commit to compare all future coverage changes.

This is now the second example of what appears to be consistant behaviour where a forked repo with a master branch somehow gets assumed as codecov’s default branch assigned to our repo.

@tom is there any other information we can provide here?

I’m fairly certain that we can wipe our codecov data and start from scratch (while not ideal) and prevent users from submitting PR’s from forked master branches in attempts to prevent this issue.

Should we just go down this path or is there anything else can can be done to understand the root cause of this problem?

Hi @nightah, apologies for the delay here. I’m still digging into why this is happening. If this is blocking you, you are welcome to do the wipe, but if this can wait, I’d like to be able to fix it for you. Let me know what you prefer.

Hi @tom no problems, I understand there’s a significant time delay. I did end up performing the wipe and put some safety mesaures in place which meant coverage will not be run and uplodaed if a PR is submitted from a forked master branch.

The only thing is that now we’re not seeing any commits that are merged showing up in the codecov master branch.
If you have a look at this screenshot you can see that with the same codecov settings, it previously did result in commits being recognised from master and it in turn posted github checks under the master branch.

Could this issue still be on the back of the symptoms we were experiencing before, do you have any advice on how we might resolve this? Should I disconnect it from github and reconnect?

Hi @nightah, my gut instinct is that commits aren’t showing up in Codecov because coverage is not being generated for a commit to the master branch from a fork. When you merge, do you run your CI/CD process again, and if so, do you happen to have a link to a build?

@tom the CI/CD process does run, however, coverage is not run again when merged in to master it’s a simplified deployment process since we don’t need to run tests all over again for the same code.

Essentially tests and coverage uploads are only run in branches and PR’s.

All of our builds have followed this same process, so I’m at a little bit of a loss as to why and how we managed to get commits show up in master before but not now haha.

Hi @nightah, I have escalated this to the product team to work on a fix for this issue. I believe that we will need to fix the master branch issue first.

Hi @tom, thanks for the update.

In the interim I have setup our CI/CD process to run tests and upload coverage for the master branch.

When/if there is a fix I would love to remove those steps cut down our CI/CD process for master deployments back to what they originally were, so I’ll hang tight in the meantime.

1 Like

@nightah absolutely. I’ll continue to make updates to this thread, but thank you for your patience!

@tom so now that we are actually running coverage on every single upload to master it still appears that if a branch is updated within a PR it still causes some odd issues.

You can see a PR that has been open and updated several times here: [MISC] Storage Schema Versioning Model by james-d-elliott · Pull Request #1057 · authelia/authelia · GitHub

You can also see that every single push to master since [CI] Collect and upload coverage on master branch (#1174) · authelia/authelia@0df8f6b · GitHub has also resulted in a corresponding codecov upload: Authelia #2531.

But we aren’t seeing any commits to master: Code coverage done right..

There have been several commits to master where coverage has been both collected and uploaded: https://buildkite.com/authelia/authelia/builds?branch=master.

Hi @nightah, just an update that we are still taking a look at this. Thanks for your patience here.

Hi @tom thanks for the update, since I’m not sure if it’s entirely clear I have worked around the issue when a PR falls out of sync with master and presents the following message:

This branch is out-of-date with the base branch
Merge the latest changes from master into this branch.

By avoiding the “Update branch” button on the GitHub WebUI and instead rebasing all PRs.

I believe you should be able to replicate this issue by updating PR’s with that button and observing the behaviour it will illicit with codecov.

1 Like

Thanks for this @nightah, I’ll make sure the product team is aware of this. I think it’ll greatly help us narrow down the issue.

Hi @nightah, we are still looking into this, thanks for your patience.

Hi @nightah, we have pushed some changes for this, would you be able to see if you are still experiencing this issue?