[dev-v2.9] Update Rancher-monitoring jobs.yaml for error handling

[dfaltum]

Problem

My problem is that the following line gets an error if the RBAC of the system:serviceaccount:cattle-monitoring-system:rancher-monitoring-crd-manager SA is incorrect:

kubectl replace -Rf /etc/crd || kubectl create -Rf /etc/crd;

The reason for this is that kubectl replace fails with an error code, but kubectl create can't succeed either:

Error from server (Forbidden): error when replacing "/etc/crd/charts/crds/crd-alertmanagerconfigs.yaml": customresourcedefinitions.apiextensions.k8s.io "alertmanagerconfigs.monitoring.coreos.com" is forbidden: User "system:serviceaccount:cattle-monitoring-system:rancher-monitoring-crd-manager" cannot update resource "customresourcedefinitions" in API group "apiextensions.k8s.io" at the cluster scope
Error from server (AlreadyExists): error when creating "/etc/crd/charts/crds/crd-alertmanagerconfigs.yaml": customresourcedefinitions.apiextensions.k8s.io "alertmanagerconfigs.monitoring.coreos.com" already exists

In the earlier chart versions, the SA didn't have recreate permission. So in this scenario, the helm upgrade succeeds, but you still have the old CRDs.

Solution

I think the simplest solution is to create another OR condition. With this trick, the pod will fail and the helm upgrade will fail as well.
Another good approach might be something like this for checking permissions:

for verb in 'create' 'get' 'patch' 'delete' 'replace'; do kubectl auth can-i --as system:serviceaccount:cattle-monitoring-system:rancher-monitoring-crd-manager $verb crd || exit 255; done           

[github-actions]

Validation steps

• Ensure all container images have repository and tag on the same level to ensure that all container images are included in rancher-images.txt which are used by airgap customers.

  Ex:-
    longhorn-controller:
      repository: rancher/hardened-sriov-cni
      tag: v2.6.3-build20230913
  

• Add a 👍 (thumbs up) reaction to this comment once done. CI won't pass without this reaction to the github-action bot's latest validation comment.
• Approve the PR to run the CI check.