Flux git repository not reconciling

[devops-newbee]

Describe the bug

When we reconcile the flux-system gitrepository, it gives error.
kubectl get gitrepository -n flux-system
NAME URL AGE READY STATUS
flux-system https://gitlab.xxx.xxxx/xxxxx/xxxxx/flux/flux2.git 461d False failed to checkout and determine revision: unable to list remote for 'https://gitlab.xxx.xxxx/xxxxx/xxxxx/flux/flux2.git ': authorization failed

We can clone the same repository using the gitlab user name and password with git clone command.
The group access token for the gitlab project is val;id as we can clone the repo manually.

Flux reconcile ks flus-system -n flux-system command is working as expected.
kubectl get kustomizations.kustomize.toolkit.fluxcd.io -n flux-system
NAME AGE READY STATUS
flux-system 461d True Applied revision: master@sha1:313444xxxxxxxxxxxxxxx
tenants 425d True Applied revision: master@sha1:3134440xxxxxxxxxxxxxxx

We even tried to bootstap flux but it failed.
flux bootstrap gitlab --hostname=xxxxxxxx --components-extra=image-reflector-controller,image-automation-controller --owner=xxxxxxxxx --repository=flux2 --branch=master --path=xxxxxxx --token-auth --personal
Please enter your GitLab personal access token (PAT):
► connecting to https://xxxxxxxxx
► cloning branch "master" from Git repository "https://xxxxxxxxxxx"
✔ cloned repository
► generating component manifests
✔ generated component manifests
✔ component manifests are up to date
✔ reconciled components
► determining if source secret "flux-system/flux-system" exists
► generating source secret
► applying source secret "flux-system/flux-system"
✔ reconciled source secret
► generating sync manifests
✔ generated sync manifests
✔ committed sync manifests to "master" ("7ed80ce62443e10668e2e985775ae29d74e95d5e")
► pushing sync manifests to "https://xxxxxxxxxx"
► applying sync manifests
✔ reconciled sync configuration
◎ waiting for GitRepository "flux-system/flux-system" to be reconciled
✗ gitrepository 'flux-system/flux-system' not ready: 'failed to checkout and determine revision: unable to list remote for 'https://xxxxxxxxxxxx': authorization failed'
◎ waiting for Kustomization "flux-system/flux-system" to be reconciled
✗ client rate limiter Wait returned an error: context deadline exceeded
► confirming components are healthy
✔ helm-controller: deployment ready
✔ image-automation-controller: deployment ready
✔ image-reflector-controller: deployment ready
✔ kustomize-controller: deployment ready
✔ notification-controller: deployment ready
✔ source-controller: deployment ready
✔ all components are healthy
✗ bootstrap failed with 2 health check failure(s): [error while waiting for GitRepository to be ready: 'gitrepository 'flux-system/flux-system' not ready: 'failed to checkout and determine revision: unable to list remote for 'https://xxxxxxxxxxxxx': authorization failed'', error while waiting for Kustomization to be ready: 'client rate limiter Wait returned an error: context deadline exceeded']

Steps to reproduce

kubectl get gitrepository -n flux-system
NAME URL AGE READY STATUS
flux-system https://xxxxxxxxxxxxxxx 461d False failed to checkout and determine revision: unable to list remote for 'xxxxxxxxxxxxxxxx': authorization failed

Expected behavior

flux should reflect the latest chnages done in gitrepository

Screenshots and recordings

No response

OS / Distro

Red Hat Enterprise Linux 9.4 (Plow)

Flux version

2.3.0

Flux check

flux check
► checking prerequisites
✔ Kubernetes 1.28.9-eks-036c24b >=1.28.0-0
► checking version in cluster
✔ distribution: flux-v2.3.0
✔ bootstrapped: true
► checking controllers
✔ helm-controller: deployment ready
► ghcr.io/fluxcd/helm-controller:v1.0.1
✔ image-automation-controller: deployment ready
► ghcr.io/fluxcd/image-automation-controller:v0.38.0
✔ image-reflector-controller: deployment ready
► ghcr.io/fluxcd/image-reflector-controller:v0.32.0
✔ kustomize-controller: deployment ready
► ghcr.io/fluxcd/kustomize-controller:v1.3.0
✔ notification-controller: deployment ready
► ghcr.io/fluxcd/notification-controller:v1.3.0
✔ source-controller: deployment ready
► ghcr.io/fluxcd/source-controller:v1.3.0
► checking crds
✔ alerts.notification.toolkit.fluxcd.io/v1beta3
✔ buckets.source.toolkit.fluxcd.io/v1beta2
✔ gitrepositories.source.toolkit.fluxcd.io/v1
✔ helmcharts.source.toolkit.fluxcd.io/v1
✔ helmreleases.helm.toolkit.fluxcd.io/v2
✔ helmrepositories.source.toolkit.fluxcd.io/v1
✔ imagepolicies.image.toolkit.fluxcd.io/v1beta2
✔ imagerepositories.image.toolkit.fluxcd.io/v1beta2
✔ imageupdateautomations.image.toolkit.fluxcd.io/v1beta2
✔ kustomizations.kustomize.toolkit.fluxcd.io/v1
✔ ocirepositories.source.toolkit.fluxcd.io/v1beta2
✔ providers.notification.toolkit.fluxcd.io/v1beta3
✔ receivers.notification.toolkit.fluxcd.io/v1
✔ all checks passed

Git provider

No response

Container Registry provider

No response

Additional context

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[stefanprodan]

Delete the flux-system Kubernetes secret from the cluster and rerun bootstrap, it may be that the token has changed since last bootstrap. You can also update the token in the secret directly with flux create secret git or using kubectl.