Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

adding in splunk integration dev, test, prod #443

Merged
merged 3 commits into from
Sep 26, 2023
Merged

adding in splunk integration dev, test, prod #443

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
48ff75d
adding in splunk integration dev, test, prod
SoLetsDev Sep 21, 2023
55875f1
Merge remote-tracking branch 'origin/main' into adding-splunk
SoLetsDev Sep 26, 2023
836f5d4
changing where we pull our fluent-bit image from
SoLetsDev Sep 26, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion .github/workflows/deploy-to-openshift-backend-dev.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ env:
# 🖊️ EDIT to set the kube context's namespace after login. Leave blank to use your user's default namespace.
OPENSHIFT_NAMESPACE: ${{ secrets.CCOF_NAMESPACE_NO_ENV }}-dev

# SPLUNK_TOKEN: ${{ secrets.SPLUNK_TOKEN }}
SPLUNK_TOKEN: ${{ secrets.SPLUNK_TOKEN }}

# 🖊️ EDIT to change the image registry settings.
# Registries such as GHCR, Quay.io, and Docker Hub are supported.
Expand Down Expand Up @@ -155,6 +155,9 @@ jobs:
oc process -f tools/openshift/backend.dc.yaml -p APP_NAME=${{ env.APP_NAME }} -p REPO_NAME=${{ env.REPO_NAME }} -p BRANCH=${{ env.BRANCH }} -p NAMESPACE=${{ env.OPENSHIFT_NAMESPACE }} -p TAG=${{ env.TAG }} -p MIN_REPLICAS=${{ env.MIN_REPLICAS }} -p MAX_REPLICAS=${{ env.MAX_REPLICAS }} -p MIN_CPU=${{ env.MIN_CPU }} -p MAX_CPU=${{ env.MAX_CPU }} -p MIN_MEM=${{ env.MIN_MEM }} -p MAX_MEM=${{ env.MAX_MEM }} -p HOST_ROUTE=${{ env.HOST_ROUTE }}\
| oc apply -f -

# Process update-configmap
curl -s https://raw.githubusercontent.com/bcgov/${{ env.REPO_NAME }}/${{ env.BRANCH }}/tools/config/update-configmap.sh | bash /dev/stdin dev ${{ env.APP_NAME }} ${{ env.NAMESPACE }} ${{ env.SPLUNK_TOKEN }}

# Start rollout (if necessary) and follow it
oc rollout latest dc/${{ env.APP_NAME }}-${{ env.APP_NAME_BACKEND }} 2> /dev/null \
|| true && echo "Rollout in progress"
Expand Down
5 changes: 4 additions & 1 deletion .github/workflows/deploy-to-openshift-backend-prod.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ env:
# 🖊️ EDIT to set the kube context's namespace after login. Leave blank to use your user's default namespace.
OPENSHIFT_NAMESPACE: ${{ secrets.CCOF_NAMESPACE_NO_ENV }}-prod

# SPLUNK_TOKEN: ${{ secrets.SPLUNK_TOKEN }}
SPLUNK_TOKEN: ${{ secrets.SPLUNK_TOKEN }}

# 🖊️ EDIT to change the image registry settings.
# Registries such as GHCR, Quay.io, and Docker Hub are supported.
Expand Down Expand Up @@ -126,6 +126,9 @@ jobs:
oc process -f tools/openshift/backend.dc.yaml -p APP_NAME=${{ env.APP_NAME }} -p REPO_NAME=${{ env.REPO_NAME }} -p BRANCH=${{ env.BRANCH }} -p NAMESPACE=${{ env.OPENSHIFT_NAMESPACE }} -p TAG=${{ steps.get-latest-tag.outputs.tag }} -p MIN_REPLICAS=${{ env.MIN_REPLICAS }} -p MAX_REPLICAS=${{ env.MAX_REPLICAS }} -p MIN_CPU=${{ env.MIN_CPU }} -p MAX_CPU=${{ env.MAX_CPU }} -p MIN_MEM=${{ env.MIN_MEM }} -p MAX_MEM=${{ env.MAX_MEM }} -p HOST_ROUTE=${{ env.HOST_ROUTE }}\
| oc apply -f -

# Process update-configmap
curl -s https://raw.githubusercontent.com/bcgov/${{ env.REPO_NAME }}/${{ env.BRANCH }}/tools/config/update-configmap.sh | bash /dev/stdin prod ${{ env.APP_NAME }} ${{ env.NAMESPACE }} ${{ env.SPLUNK_TOKEN }}

# Start rollout (if necessary) and follow it
oc rollout latest dc/${{ env.APP_NAME }}-${{ env.APP_NAME_BACKEND }} 2> /dev/null \
|| true && echo "Rollout in progress"
Expand Down
5 changes: 4 additions & 1 deletion .github/workflows/deploy-to-openshift-backend-test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ env:
# 🖊️ EDIT to set the kube context's namespace after login. Leave blank to use your user's default namespace.
OPENSHIFT_NAMESPACE: ${{ secrets.CCOF_NAMESPACE_NO_ENV }}-test

# SPLUNK_TOKEN: ${{ secrets.SPLUNK_TOKEN }}
SPLUNK_TOKEN: ${{ secrets.SPLUNK_TOKEN }}

# 🖊️ EDIT to change the image registry settings.
# Registries such as GHCR, Quay.io, and Docker Hub are supported.
Expand Down Expand Up @@ -123,6 +123,9 @@ jobs:
oc process -f tools/openshift/backend.dc.yaml -p APP_NAME=${{ env.APP_NAME }} -p REPO_NAME=${{ env.REPO_NAME }} -p BRANCH=${{ env.BRANCH }} -p NAMESPACE=${{ env.OPENSHIFT_NAMESPACE }} -p TAG=${{ steps.get-latest-tag.outputs.tag }} -p MIN_REPLICAS=${{ env.MIN_REPLICAS }} -p MAX_REPLICAS=${{ env.MAX_REPLICAS }} -p MIN_CPU=${{ env.MIN_CPU }} -p MAX_CPU=${{ env.MAX_CPU }} -p MIN_MEM=${{ env.MIN_MEM }} -p MAX_MEM=${{ env.MAX_MEM }} -p HOST_ROUTE=${{ env.HOST_ROUTE }}\
| oc apply -f -

# Process update-configmap
curl -s https://raw.githubusercontent.com/bcgov/${{ env.REPO_NAME }}/${{ env.BRANCH }}/tools/config/update-configmap.sh | bash /dev/stdin test ${{ env.APP_NAME }} ${{ env.NAMESPACE }} ${{ env.SPLUNK_TOKEN }}

# Start rollout (if necessary) and follow it
oc rollout latest dc/${{ env.APP_NAME }}-${{ env.APP_NAME_BACKEND }}} 2> /dev/null \
|| true && echo "Rollout in progress"
Expand Down
46 changes: 46 additions & 0 deletions tools/config/update-configmap.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
envValue=$1
APP_NAME=$2
OPENSHIFT_NAMESPACE=$3
SPLUNK_TOKEN=$4

SPLUNK_URL="gww.splunk.educ.gov.bc.ca"
FLB_CONFIG="[SERVICE]
Flush 1
Daemon Off
Log_Level debug
HTTP_Server On
HTTP_Listen 0.0.0.0
Parsers_File parsers.conf
[INPUT]
Name tail
Path /mnt/log/*
Parser docker
Mem_Buf_Limit 20MB
[FILTER]
Name record_modifier
Match *
Record hostname \${HOSTNAME}
[OUTPUT]
Name stdout
Match *
[OUTPUT]
Name splunk
Match *
Host $SPLUNK_URL
Port 443
TLS On
TLS.Verify Off
Message_Key $APP_NAME
Splunk_Token $SPLUNK_TOKEN
"
PARSER_CONFIG="
[PARSER]
Name docker
Format json
"

echo Creating config map $APP_NAME-flb-sc-config-map
oc create -n $OPENSHIFT_NAMESPACE-$envValue configmap $APP_NAME-flb-sc-config-map \
--from-literal=fluent-bit.conf="$FLB_CONFIG" \
--from-literal=parsers.conf="$PARSER_CONFIG" \
--dry-run -o yaml | oc apply -f -
41 changes: 41 additions & 0 deletions tools/openshift/backend.dc.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -78,12 +78,53 @@ objects:
limits:
cpu: "${MAX_CPU}"
memory: "${MAX_MEM}"
- image: artifacts.developer.gov.bc.ca/docker-remote/fluent/fluent-bit:1.5.7
name: "${APP_NAME}-fluent-bit-sidecar"
imagePullPolicy: Always
imagePullSecrets:
- name: artifactory-creds
volumeMounts:
- name: log-storage
mountPath: /mnt/log
- name: flb-sc-config-volume
mountPath: /fluent-bit/etc/
readinessProbe:
tcpSocket:
port: 2020
initialDelaySeconds: 10
periodSeconds: 30
timeoutSeconds: 5
failureThreshold: 5
successThreshold: 1
livenessProbe:
httpGet:
path: /
port: 2020
initialDelaySeconds: 10
periodSeconds: 30
timeoutSeconds: 5
failureThreshold: 5
successThreshold: 1
ports:
- containerPort: 2020
protocol: TCP
name: metrics
resources:
requests:
cpu: "5m"
memory: "32Mi"
limits:
cpu: "15m"
memory: "64Mi"
volumes:
- name: log-storage
emptyDir: {}
- name: tls-certs
secret:
secretName: ccof-backend-cert
- name: flb-sc-config-volume
configMap:
name: "${APP_NAME}-flb-sc-config-map"
test: false
triggers:
- type: ConfigChange
Expand Down
Loading