Compare commits

...

28 Commits

Author SHA1 Message Date
Christian Adams
d53c93c864 Document the need for quotes on pg secret port value 2023-03-15 13:35:49 -04:00
aurel333
fe77f1deae Fix awx-manager-role bad match between some apiGroups and resources (#1262)
Co-authored-by: Aurelien Potin <aurelien.potin@michelin.com>
2023-03-08 16:24:03 -05:00
Hao Liu
9156909023 Merge pull request #1263 from rh-dluong/date-format
change date format to avoid :
2023-03-08 15:58:58 -05:00
Christian Adams
03e3b68eeb Helm Release: Need to use shell, not command for multi-line (#1280) 2023-03-08 15:55:30 -05:00
Ravi Teja
6cae8dfa32 fix: use sha1 instead of md5 to make the installer work in FIPS (#1260)
Co-authored-by: RaviTeja Buddabathuni (rbuddaba) <rbuddaba@cisco.com>
2023-03-08 15:10:27 -05:00
Hao Liu
b1ecc89759 Merge pull request #1274 from TheRealHaoLiu/feature-branch-build
Add github workflow for publishing feature branch images
2023-03-07 17:05:08 -05:00
Hao Liu
bb85b31411 Add github workflow to publish feature branch img 2023-03-07 16:50:28 -05:00
Christian Adams
7e7c23e176 Fix helm-release task to work in ansible, not GHA env (#1267) 2023-03-06 15:25:04 -05:00
John Westcott IV
8442cc7ae8 Merge pull request #1268 from john-westcott-iv/ubuntu-change
Changing github actions from ubuntu-18 to ubuntu-latest
2023-03-06 15:06:02 -05:00
John Westcott IV
3faf76d7c9 Changing github actions from ubuntu-18 to ubuntu-latest
See https://github.com/actions/runner-images/issues/6002 for deprication warnings
2023-03-06 14:55:32 -05:00
David Luong
61ef5ef005 change date format to avoid : 2023-03-02 13:54:00 -05:00
Peter Hoffmann
86043361c6 Small fix/adjustment in README (#1257)
* Fix minor typo
2023-02-28 16:20:47 -05:00
Ravi Teja
06194960e9 Bump ansible operator SDK version to v1.26.0 (#1258)
Co-authored-by: RaviTeja Buddabathuni (rbuddaba) <rbuddaba@cisco.com>
2023-02-28 15:45:02 -05:00
Hao Liu
46da413585 Merge pull request #1193 from stanislav-zaprudskiy/add_termination_grace_period_seconds
AWX: Add `termination_grace_period_seconds`
2023-02-28 15:37:51 -05:00
Hao Liu
b5f255cf00 add nodeport_port to instantiate playbook (#1252) 2023-02-22 21:18:12 +00:00
Maxence Button
f328b0adb6 Customization of the init_projects_container_image is now possible (#1248) 2023-02-22 15:05:23 -05:00
Shane McDonald
1bfd3b2028 Merge pull request #1201 from thedoubl3j/debugging_doc_update
Update Installer Debugging Documentation
2023-02-22 14:51:05 -05:00
Christian Adams
edf01f009e Stage and push helm index where gitconfig is set (#1251) 2023-02-22 14:09:35 -05:00
Joel
a3c28be98a [enhancement] restore postgres database if external db (#1211)
* restore postgres fix resolvable_db_host

set the resolvable_db_host ansible variable based on the db deployment
2023-02-22 13:56:38 -05:00
Stanislav Zaprudskiy
49d1f00dbd Do not keep web container running during termination
We could let the web container terminate as usual, as there are no
reasons to keep it running as it doesn't participate in
job control. Additionally, it stops receiving traffic with the beginning
of termination

> At the same time as the kubelet is starting graceful shutdown, the
> control plane removes that shutting-down Pod from EndpointSlice (and
> Endpoints) objects where these represent a Service with a configured
> selector

@ https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-termination
2023-02-22 10:40:40 +01:00
Christian Adams
81939cb521 K8s cpu value example in README.md needs quotes to be valid (#1220) 2023-02-20 17:37:53 -05:00
Stanislav Zaprudskiy
336ea58a0a AWX: Add termination_grace_period_seconds 2023-02-07 16:33:00 +01:00
Stanislav Zaprudskiy
f042cb3d00 Fix lint warnings 2023-02-07 16:31:26 +01:00
Stanislav Zaprudskiy
94d68bf382 Make Deployment to be rolled out on CM and Secrets changes
With the previous approach, not all associated (mounted) CM/Secrets
changes caused the Deployment to be rolled out, but also the Deployment
could have been rolled out unnecessary during e.g. Ingress or Service
changes (which do not require Pod restarts).

Previously existing Pod removal (state: absent) was not complete as
other pods continued to exist, but also is not needed with this commit
change due to added Pods annotations.

The added Deployment Pod annotations now cause the new ReplicaSet
version to be rolled out, effectively causing replacement of the
previously existing Pods in accordance with the deployment `strategy`
(https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.25/#deploymentstrategy-v1-apps,
`RollingUpdate`) whenever there is a change in the associated CMs or
Secrets referenced in annotations. This implementation is quite standard
and widely used for Helm workflows -
https://helm.sh/docs/howto/charts_tips_and_tricks/#automatically-roll-deployments
2023-02-07 11:58:47 +01:00
Stanislav Zaprudskiy
b3a74362af Make AWX Pod variable to be calculated respecting creationTimestamp and deletionTimestamp
Do not consider Pods marked for deletion when calculating tower_pod to
address replicas scale down case - where normally Pods spawned recently
are being taken for removal. As well as the case when operator kicked
off but some old replicas are still terminating.

Respect `creationTimestamp` so to make sure that the newest Pod is taken
after Deployment application, in which case multiple RS Pods (from old
RS and new RS) could be running simultaneously while the rollout is
happening.
2023-02-07 11:47:49 +01:00
Stanislav Zaprudskiy
ad531c8dce Do not wait for a new Pod name after Deployment change
Proper waiting is already performed earlier during Deplyment{apply: yes, wait: yes} -
e6ac874098/plugins/module_utils/k8s/waiter.py (L27).

And also not every Deployment change produces new RS/Pods. For example,
changing Deployment labels won't cause new rollout, but will cause
`until` loop to be invoked unnecessarily (when replicas=1).
2023-02-07 11:43:34 +01:00
Stanislav Zaprudskiy
e589ceb661 When applying Deployment wait up to (timeout * replicas)
There are cases when having a new Deployment may be taking above the
default timeout of 120s.
For instance, when a Deployment has multiple replicas, and each replica
starts on a separate node, and the Deployment specifies new images, then
just pulling these new images for each replica may be taking above the
default timeout of 120s.

Having the default time multiplied by the number of replicas should
provide generally enough time for all replicas to start
2023-02-07 11:41:32 +01:00
thedoubl3j
d0516bcdc1 update vars file entry to include over ride examples 2023-01-25 14:41:18 -05:00
29 changed files with 546 additions and 89 deletions

View File

@@ -11,7 +11,7 @@ on:
jobs:
molecule:
runs-on: ubuntu-18.04
runs-on: ubuntu-latest
name: molecule
env:
DOCKER_API_VERSION: "1.38"
@@ -40,7 +40,7 @@ jobs:
make kustomize
KUSTOMIZE_PATH=$(readlink -f bin/kustomize) molecule test -s kind
helm:
runs-on: ubuntu-18.04
runs-on: ubuntu-latest
name: helm
steps:
- uses: actions/checkout@v2

View File

@@ -8,7 +8,7 @@ on:
jobs:
release:
runs-on: ubuntu-18.04
runs-on: ubuntu-latest
name: Push devel image
steps:
- uses: actions/checkout@v2

75
.github/workflows/feature.yml vendored Normal file
View File

@@ -0,0 +1,75 @@
---
name: Feature Branch Image Build and Push
on:
push:
branches: [feature_*]
jobs:
release:
runs-on: ubuntu-18.04
name: Push devel image
steps:
- uses: actions/checkout@v3
with:
fetch-depth: 0 # needed so that git describe --tag works
- name: Set VERSION
run: |
echo "VERSION=$(git describe --tags)" >>${GITHUB_ENV}
- name: Set lower case owner name
run: |
echo "OWNER_LC=${OWNER,,}" >>${GITHUB_ENV}
env:
OWNER: '${{ github.repository_owner }}'
- name: Set IMAGE_TAG_BASE
run: |
echo "IMAGE_TAG_BASE=ghcr.io/${OWNER_LC}/awx-operator" >>${GITHUB_ENV}
- name: Set ARCH environment variable
run: |
echo "ARCH=$(case $(uname -m) in x86_64) echo -n amd64 ;; aarch64) echo -n arm64 ;; *) echo -n $(uname -m) ;; esac)" >>${GITHUB_ENV}
- name: Set OS environment variable
run: |
echo "OS=$(uname | awk '{print tolower($0)}')" >>${GITHUB_ENV}
- name: Install operator-sdk
run: |
echo "Installing operator-sdk ${OPERATOR_SDK_DL_URL}" && \
curl -LO ${OPERATOR_SDK_DL_URL}/operator-sdk_${OS}_${ARCH} && \
chmod +x operator-sdk_${OS}_${ARCH} && \
sudo mkdir -p /usr/local/bin/ && \
sudo mv operator-sdk_${OS}_${ARCH} /usr/local/bin/operator-sdk && \
operator-sdk version
env:
OPERATOR_SDK_DL_URL: https://github.com/operator-framework/operator-sdk/releases/download/v1.26.0
- name: Log in to registry
run: |
echo "${{ secrets.GITHUB_TOKEN }}" | docker login ghcr.io -u ${{ github.actor }} --password-stdin
- name: Build and Push awx-operator Image
run: |
make docker-build docker-push
docker tag ${IMAGE_TAG_BASE}:${VERSION} ${IMAGE_TAG_BASE}:${GITHUB_REF##*/}
docker push ${IMAGE_TAG_BASE}:${GITHUB_REF##*/}
- name: Build bundle manifests
run: |
make bundle
- name: Build and Push awx-operator Bundle
run: |
make bundle-build bundle-push
docker tag ${IMAGE_TAG_BASE}-bundle:v${VERSION} ${IMAGE_TAG_BASE}-bundle:${GITHUB_REF##*/}
docker push ${IMAGE_TAG_BASE}-bundle:${GITHUB_REF##*/}
- name: Build and Push awx-operator Catalog
run: |
make catalog-build catalog-push
docker tag ${IMAGE_TAG_BASE}-catalog:v${VERSION} ${IMAGE_TAG_BASE}-catalog:${GITHUB_REF##*/}
docker push ${IMAGE_TAG_BASE}-catalog:${GITHUB_REF##*/}

View File

@@ -33,11 +33,6 @@ jobs:
docker push quay.io/${{ github.repository }}:${{ github.event.release.tag_name }}
docker push quay.io/${{ github.repository }}:latest
- name: Configure git
run: |
git config user.name "$GITHUB_ACTOR"
git config user.email "$GITHUB_ACTOR@users.noreply.github.com"
- name: Release Helm chart
run: |
ansible-playbook ansible/helm-release.yml -v \
@@ -45,3 +40,4 @@ jobs:
-e chart_owner=${{ github.repository_owner }} \
-e tag=${{ github.event.release.tag_name }} \
-e gh_token=${{ secrets.GITHUB_TOKEN }}
-e gh_user=${{ github.actor }}

View File

@@ -1,4 +1,4 @@
FROM quay.io/operator-framework/ansible-operator:v1.25.3
FROM quay.io/operator-framework/ansible-operator:v1.26.0
USER 0

View File

@@ -156,7 +156,7 @@ ifeq (,$(shell which ansible-operator 2>/dev/null))
@{ \
set -e ;\
mkdir -p $(dir $(ANSIBLE_OPERATOR)) ;\
curl -sSLo $(ANSIBLE_OPERATOR) https://github.com/operator-framework/operator-sdk/releases/download/v1.25.3/ansible-operator_$(OS)_$(ARCHA) ;\
curl -sSLo $(ANSIBLE_OPERATOR) https://github.com/operator-framework/operator-sdk/releases/download/v1.26.0/ansible-operator_$(OS)_$(ARCHA) ;\
chmod +x $(ANSIBLE_OPERATOR) ;\
}
else
@@ -187,7 +187,7 @@ ifeq (,$(shell which opm 2>/dev/null))
@{ \
set -e ;\
mkdir -p $(dir $(OPM)) ;\
curl -sSLo $(OPM) https://github.com/operator-framework/operator-registry/releases/download/v1.25.3/$(OS)-$(ARCHA)-opm ;\
curl -sSLo $(OPM) https://github.com/operator-framework/operator-registry/releases/download/v1.26.0/$(OS)-$(ARCHA)-opm ;\
chmod +x $(OPM) ;\
}
else
@@ -395,9 +395,3 @@ helm-index:
$(HELM) repo index .cr-release-packages --url https://github.com/$(CHART_OWNER)/$(CHART_REPO)/releases/download/ --merge gh-pages/index.yaml
mv .cr-release-packages/index.yaml gh-pages/index.yaml
@echo "== PUSH INDEX FILE =="
cd gh-pages;\
git add index.yaml;\
git commit -m "Updated index.yaml latest release";\
git push;\

View File

@@ -49,6 +49,7 @@ An [Ansible AWX](https://github.com/ansible/awx) operator for Kubernetes built w
* [Upgrade of instances without auto upgrade](#upgrade-of-instances-without-auto-upgrade)
* [Service Account](#service-account)
* [Labeling operator managed objects](#labeling-operator-managed-objects)
* [Pods termination grace period](#pods-termination-grace-period)
* [Uninstall](#uninstall)
* [Upgrading](#upgrading)
* [Backup](#backup)
@@ -269,7 +270,7 @@ yDL2Cx5Za94g9MvBP6B73nzVLlmfgPjR
You just completed the most basic install of an AWX instance via this operator. Congratulations!!!
For an example using the Nginx Controller in Minukube, don't miss our [demo video](https://asciinema.org/a/416946).
For an example using the Nginx Ingress Controller in Minikube, don't miss our [demo video](https://asciinema.org/a/416946).
### Helm Install on existing cluster
@@ -574,7 +575,7 @@ spec:
cpu: 500m
memory: 2Gi
limits:
cpu: 1
cpu: '1'
memory: 4Gi
postgres_storage_requirements:
requests:
@@ -1246,6 +1247,46 @@ spec:
...
```
#### Pods termination grace period
During deployment restarts or new rollouts, when old ReplicaSet Pods are being
terminated, the corresponding jobs which are managed (executed or controlled)
by old AWX Pods may end up in `Error` state as there is no mechanism to
transfer them to the newly spawned AWX Pods. To work around the problem one
could set `termination_grace_period_seconds` in AWX spec, which does the
following:
* It sets the corresponding
[`terminationGracePeriodSeconds`](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-termination)
Pod spec of the AWX Deployment to the value provided
> The grace period is the duration in seconds after the processes running in
> the pod are sent a termination signal and the time when the processes are
> forcibly halted with a kill signal
* It adds a
[`PreStop`](https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/#hook-handler-execution)
hook script, which will keep AWX Pods in terminating state until it finished,
up to `terminationGracePeriodSeconds`.
> This grace period applies to the total time it takes for both the PreStop
> hook to execute and for the Container to stop normally
While the hook script just waits until the corresponding AWX Pod (instance)
no longer has any managed jobs, in which case it finishes with success and
hands over the overall Pod termination process to normal AWX processes.
One may want to set this value to the maximum duration they accept to wait for
the affected Jobs to finish. Keeping in mind that such finishing jobs may
increase Pods termination time in such situations as `kubectl rollout restart`,
AWX upgrade by the operator, or Kubernetes [API-initiated
evictions](https://kubernetes.io/docs/concepts/scheduling-eviction/api-eviction/).
| Name | Description | Default |
| -------------------------------- | --------------------------------------------------------------- | ------- |
| termination_grace_period_seconds | Optional duration in seconds pods needs to terminate gracefully | not set |
### Uninstall ###
To uninstall an AWX deployment instance, you basically need to remove the AWX kind related to that instance. For example, to delete an AWX instance named awx-demo, you would do:

View File

@@ -37,6 +37,13 @@
register: asset_upload
changed_when: asset_upload.json.state == "uploaded"
- name: Configure git config
shell: |
git config user.name {{ gh_user }}
git config user.email {{ gh_user }}@users.noreply.github.com
args:
chdir: "{{ playbook_dir }}/../gh-pages"
- name: Publish helm index
command: |
make helm-index
@@ -45,3 +52,11 @@
CR_TOKEN: "{{ gh_token }}"
args:
chdir: "{{ playbook_dir }}/../"
- name: Stage and Push commit to gh-pages branch
shell: |
git add index.yaml
git commit -m "Updated index.yaml latest release"
git push
args:
chdir: "{{ playbook_dir }}/../gh-pages"

View File

@@ -26,6 +26,7 @@
image_version: "{{ image_version | default(omit) }}"
development_mode: "{{ development_mode | default(omit) | bool }}"
image_pull_policy: "{{ image_pull_policy | default(omit) }}"
nodeport_port: "{{ nodeport_port | default(omit) }}"
# ee_images:
# - name: test-ee
# image: quay.io/<user>/awx-ee

View File

@@ -525,6 +525,10 @@ spec:
type: array
type: object
type: object
termination_grace_period_seconds:
description: Optional duration in seconds pods needs to terminate gracefully
type: integer
format: int32
service_labels:
description: Additional labels to apply to the service
type: string
@@ -758,6 +762,9 @@ spec:
init_container_extra_volume_mounts:
description: Specify volume mounts to be added to the init container
type: string
init_projects_container_image:
description: Registry path to the init projects container to use
type: string
postgres_image:
description: Registry path to the PostgreSQL container to use
type: string

View File

@@ -622,6 +622,11 @@ spec:
x-descriptors:
- urn:alm:descriptor:com.tectonic.ui:advanced
- urn:alm:descriptor:com.tectonic.ui:hidden
- displayName: Termination Grace Period Seconds
path: termination_grace_period_seconds
x-descriptors:
- urn:alm:descriptor:com.tectonic.ui:advanced
- urn:alm:descriptor:com.tectonic.ui:hidden
- displayName: Service Labels
path: service_labels
x-descriptors:
@@ -687,6 +692,12 @@ spec:
x-descriptors:
- urn:alm:descriptor:com.tectonic.ui:advanced
- urn:alm:descriptor:com.tectonic.ui:hidden
- description: Init Projects Container image to use
displayName: Init Projects Container Image
path: init_projects_container_image
x-descriptors:
- urn:alm:descriptor:com.tectonic.ui:advanced
- urn:alm:descriptor:com.tectonic.ui:hidden
- description: Secret where can be found the trusted Certificate Authority Bundle
path: bundle_cacert_secret
x-descriptors:

View File

@@ -20,7 +20,6 @@ rules:
- watch
- apiGroups:
- ""
- "rbac.authorization.k8s.io"
resources:
- pods
- services
@@ -31,6 +30,17 @@ rules:
- events
- configmaps
- secrets
verbs:
- get
- list
- create
- delete
- patch
- update
- watch
- apiGroups:
- "rbac.authorization.k8s.io"
resources:
- roles
- rolebindings
verbs:
@@ -43,12 +53,22 @@ rules:
- watch
- apiGroups:
- apps
- networking.k8s.io
resources:
- deployments
- daemonsets
- replicasets
- statefulsets
verbs:
- get
- list
- create
- delete
- patch
- update
- watch
- apiGroups:
- networking.k8s.io
resources:
- ingresses
verbs:
- get

View File

@@ -4,7 +4,7 @@
entrypoint:
- scorecard-test
- basic-check-spec
image: quay.io/operator-framework/scorecard-test:v1.25.3
image: quay.io/operator-framework/scorecard-test:v1.26.0
labels:
suite: basic
test: basic-check-spec-test

View File

@@ -4,7 +4,7 @@
entrypoint:
- scorecard-test
- olm-bundle-validation
image: quay.io/operator-framework/scorecard-test:v1.25.3
image: quay.io/operator-framework/scorecard-test:v1.26.0
labels:
suite: olm
test: olm-bundle-validation-test
@@ -14,7 +14,7 @@
entrypoint:
- scorecard-test
- olm-crds-have-validation
image: quay.io/operator-framework/scorecard-test:v1.25.3
image: quay.io/operator-framework/scorecard-test:v1.26.0
labels:
suite: olm
test: olm-crds-have-validation-test
@@ -24,7 +24,7 @@
entrypoint:
- scorecard-test
- olm-crds-have-resources
image: quay.io/operator-framework/scorecard-test:v1.25.3
image: quay.io/operator-framework/scorecard-test:v1.26.0
labels:
suite: olm
test: olm-crds-have-resources-test
@@ -34,7 +34,7 @@
entrypoint:
- scorecard-test
- olm-spec-descriptors
image: quay.io/operator-framework/scorecard-test:v1.25.3
image: quay.io/operator-framework/scorecard-test:v1.26.0
labels:
suite: olm
test: olm-spec-descriptors-test
@@ -44,7 +44,7 @@
entrypoint:
- scorecard-test
- olm-status-descriptors
image: quay.io/operator-framework/scorecard-test:v1.25.3
image: quay.io/operator-framework/scorecard-test:v1.26.0
labels:
suite: olm
test: olm-status-descriptors-test

View File

@@ -35,6 +35,19 @@ ansible_operator_meta:
namespace: awx
service_type: nodeport
```
The vars file will replace the awx resource so any value that you wish to over ride using the awx resource, put in the vars file. For example, if you wish to use your own image, version and pull policy, you can specify it like below:
```yaml
# vars.yml
---
ansible_operator_meta:
name: awx
namespace: awx
service_type: nodeport
image: $DEV_DOCKER_TAG_BASE/awx_kube_devel
image_pull_policy: Always
image_version: $COMPOSE_TAG
```
Run the installer:

View File

@@ -34,7 +34,7 @@ metadata:
namespace: <target namespace>
stringData:
host: <external ip or url resolvable by the cluster>
port: <external port, this usually defaults to 5432>
port: "<external port, this usually defaults to 5432>" # quotes are required
database: <desired database name>
username: <username to connect as>
password: <password to connect with>

View File

@@ -32,22 +32,22 @@
- this_backup['resources'][0]['metadata']['labels']
- block:
- include_tasks: init.yml
- include_tasks: init.yml
- include_tasks: postgres.yml
- include_tasks: postgres.yml
- include_tasks: awx-cro.yml
- include_tasks: awx-cro.yml
- include_tasks: secrets.yml
- include_tasks: secrets.yml
- name: Set flag signifying this backup was successful
set_fact:
backup_complete: true
- name: Set flag signifying this backup was successful
set_fact:
backup_complete: true
- include_tasks: cleanup.yml
- include_tasks: cleanup.yml
when:
- this_backup['resources'][0]['status']['backupDirectory'] is not defined
- this_backup['resources'][0]['status']['backupDirectory'] is not defined
- name: Update status variables
include_tasks: update_status.yml

View File

@@ -50,7 +50,7 @@
- name: Determine the timestamp for the backup once for all nodes
set_fact:
now: '{{ lookup("pipe", "date +%F-%T") }}'
now: '{{ lookup("pipe", "date +%F-%H%M%S") }}'
- name: Set backup directory name
set_fact:

View File

@@ -0,0 +1,66 @@
# file, which when exists, indicates that `master` script has successfully
# completed pre-stop script execution
marker_file="${PRE_STOP_MARKER_FILE:-/var/lib/pre-stop/.termination_marker}"
# file which the running `master` script continuously updates (mtime) to
# indicate it's still running. this file is then read by `watcher`s to
# understand if they still have to wait for `termination_marker`
heartbeat_file="${PRE_STOP_HEARTBEAT_FILE:-/var/lib/pre-stop/.heartbeat}"
# file which:
# * `watcher`s create when they bail out because they didn't see the
# `heartbeat_file` to be updated within `$heartbeat_failed_threshold`;
# * `master` creates when its handler command fails;
# when scripts see such file, they also give up
bailout_file="${PRE_STOP_BAILOUT_FILE:-/var/lib/pre-stop/.bailout}"
heartbeat_threshold="${PRE_STOP_HEARTBEAT_THRESHOLD:-60}"
# where the scripts' stdout/stderr are streamed
stdout="${PRE_STOP_STDOUT:-/proc/1/fd/1}"
stderr="${PRE_STOP_STDERR:-/proc/1/fd/2}"
# command the `master` script executes, which when successfully finishes,
# causes the script to create the `marker_file`
handler="${PRE_STOP_HANDLER:-bash -c \"PYTHONUNBUFFERED=x awx-manage disable_instance --wait --retry=inf\"}"
log_prefix="${PRE_STOP_LOG_PREFIX:-preStop.exec}"
[[ -n ${PRE_STOP_LOG_ROLE} ]] && log_prefix="${log_prefix}] [$PRE_STOP_LOG_ROLE"
# interval at which `watcher`s check for `marker_file` presence
recheck_sleep="${PRE_STOP_RECHECK_SLEEP:-1}"
# interval at which `watcher`s report into $stdout that they are still watching
report_every="${PRE_STOP_REPORT_EVERY:-30}"
function log {
printf "[%s] $1\n" "$log_prefix" "${@:2}"
}
function parameters_string {
for param in "$@"; do
printf "%s=\"%s\"\n" "$param" "${!param}"
done | paste -s -d ' '
}
function check_bailout {
if [[ -f $bailout_file ]]; then
log "\"%s\" file has been detected, accepting bail out signal and failing the hook script" \
"$bailout_file"
exit 1
fi
}
function check_heartbeat {
if [[ -f $heartbeat_file ]]; then
delta=$(( $(date +%s) - $(stat -c %Y "$heartbeat_file") ))
else
delta=$(( $(date +%s) - $1 ))
fi
if [[ $delta -gt $heartbeat_threshold ]]; then
log "The heartbeat file hasn't been updated since %ss, which is above the threshold of %ds, assuming the master is not operating and failing the hook script" \
$delta
$heartbeat_threshold
touch "$bailout_file"
exit 1
fi
}

View File

@@ -0,0 +1,50 @@
#/usr/bin/env bash
PRE_STOP_LOG_ROLE="${PRE_STOP_LOG_ROLE:-master}"
source $(dirname "$0")/termination-env
{
log "The hook has started: %s" \
"$(parameters_string \
"marker_file" \
"heartbeat_file" \
"bailout_file" \
"handler" \
)"
touch "$heartbeat_file"
set -o pipefail
eval "$handler" 2>&1 | while IFS= read -r line; do
# we check the files here and break early, but overall script termination
# happens later - as we need to distinguish between files detection and
# command failure, while bash doesn't offer a simple way to do this here
# inside the loop (`exit` does not terminate the script)
[[ -f $bailout_file ]] && break
[[ -f $marker_file ]] && break
log "[handler] %s" "$line"
touch "$heartbeat_file"
done
ec=$?
set +o pipefail
# process various cases in specific order
check_bailout
if [[ -f $marker_file ]]; then
log "Done! The marker file has been detected, assuming some other instance of the script has run to completion"
exit 0
elif [[ $ec -ne 0 ]]; then
log "The handler has failed with \"%d\" exit code, failing the hook script too" \
$ec
# signal others to bail out
touch "$bailout_file"
exit $ec
else
log "Done! Generating the marker file allowing to proceed to termination"
touch "$marker_file"
fi
} > "$stdout" 2> "$stderr"

View File

@@ -0,0 +1,33 @@
#/usr/bin/env bash
PRE_STOP_LOG_ROLE="${PRE_STOP_LOG_ROLE:-waiter}"
source $(dirname "$0")/termination-env
{
log "The hook has started: %s" \
"$(parameters_string \
"marker_file" \
"heartbeat_file" \
"bailout_file" \
"recheck_sleep" \
"report_every" \
)"
n=0
checks_started=$(date +%s)
while ! [[ -f $marker_file ]]; do
check_bailout
check_heartbeat $checks_started
if [[ $(($n % $report_every)) -eq 0 ]]; then
log "Waiting for the marker file to be accessible..."
fi
n=$(($n + 1))
sleep $recheck_sleep
done
log "The marker file found, exiting to proceed to termination"
} > "$stdout" 2> "$stderr"

View File

@@ -39,17 +39,17 @@
- name: Load LDAP CAcert certificate
include_tasks: load_ldap_cacert_secret.yml
when:
- ldap_cacert_secret != ''
- ldap_cacert_secret != ''
- name: Load ldap bind password
include_tasks: load_ldap_password_secret.yml
when:
- ldap_password_secret != ''
- ldap_password_secret != ''
- name: Load bundle certificate authority certificate
include_tasks: load_bundle_cacert_secret.yml
when:
- bundle_cacert_secret != ''
- bundle_cacert_secret != ''
- name: Include admin password configuration tasks
include_tasks: admin_password_configuration.yml
@@ -66,8 +66,8 @@
- name: Load Route TLS certificate
include_tasks: load_route_tls_secret.yml
when:
- ingress_type | lower == 'route'
- route_tls_secret != ''
- ingress_type | lower == 'route'
- route_tls_secret != ''
- name: Include resources configuration tasks
include_tasks: resources_configuration.yml
@@ -91,8 +91,8 @@
bash -c "awx-manage migrate --noinput"
register: migrate_result
when:
- database_check is defined
- (database_check.stdout|trim) != '0'
- database_check is defined
- (database_check.stdout|trim) != '0'
- name: Initialize Django
include_tasks: initialize_django.yml

View File

@@ -13,9 +13,17 @@
- status.phase=Running
register: tower_pod
- name: Set the resource pod as a variable.
set_fact:
tower_pod: >-
{{ tower_pod['resources']
| rejectattr('metadata.deletionTimestamp', 'defined')
| sort(attribute='metadata.creationTimestamp')
| first | default({}) }}
- name: Set the resource pod name as a variable.
set_fact:
tower_pod_name: "{{ tower_pod['resources'][0]['metadata']['name'] | default('') }}"
tower_pod_name: "{{ tower_pod['metadata']['name'] | default('') }}"
- name: Set user provided control plane ee image
set_fact:
@@ -32,13 +40,13 @@
kind: Secret
namespace: '{{ ansible_operator_meta.namespace }}'
name: '{{ ansible_operator_meta.name }}-receptor-ca'
register: _receptor_ca
register: receptor_ca
no_log: "{{ no_log }}"
- name: Migrate Receptor CA Secret
when:
- _receptor_ca['resources'] | default([]) | length
- _receptor_ca['resources'][0]['type'] != "kubernetes.io/tls"
- receptor_ca['resources'] | default([]) | length
- receptor_ca['resources'][0]['type'] != "kubernetes.io/tls"
block:
- name: Delete old Receptor CA Secret
k8s:
@@ -53,7 +61,7 @@
register: _receptor_ca_key_file
- name: Copy Receptor CA key from old secret to tempfile
copy:
content: "{{ _receptor_ca['resources'][0]['data']['receptor-ca.key'] | b64decode }}"
content: "{{ receptor_ca['resources'][0]['data']['receptor-ca.key'] | b64decode }}"
dest: "{{ _receptor_ca_key_file.path }}"
no_log: "{{ no_log }}"
- name: Create tempfile for receptor-ca.crt
@@ -63,7 +71,7 @@
register: _receptor_ca_crt_file
- name: Copy Receptor CA cert from old secret to tempfile
copy:
content: "{{ _receptor_ca['resources'][0]['data']['receptor-ca.crt'] | b64decode }}"
content: "{{ receptor_ca['resources'][0]['data']['receptor-ca.crt'] | b64decode }}"
dest: "{{ _receptor_ca_crt_file.path }}"
no_log: "{{ no_log }}"
- name: Create New Receptor CA secret
@@ -71,6 +79,17 @@
apply: true
definition: "{{ lookup('template', 'secrets/receptor_ca_secret.yaml.j2') }}"
no_log: "{{ no_log }}"
- name: Read New Receptor CA Secret
k8s_info:
kind: Secret
namespace: '{{ ansible_operator_meta.namespace }}'
name: '{{ ansible_operator_meta.name }}-receptor-ca'
register: _receptor_ca
no_log: "{{ no_log }}"
- name: Set receptor_ca variable
set_fact:
receptor_ca: '{{ _receptor_ca }}'
no_log: "{{ no_log }}"
- name: Remove tempfiles
file:
path: "{{ item }}"
@@ -106,6 +125,17 @@
apply: true
definition: "{{ lookup('template', 'secrets/receptor_ca_secret.yaml.j2') }}"
no_log: "{{ no_log }}"
- name: Read Receptor CA secret
k8s_info:
kind: Secret
namespace: '{{ ansible_operator_meta.namespace }}'
name: '{{ ansible_operator_meta.name }}-receptor-ca'
register: _receptor_ca
no_log: "{{ no_log }}"
- name: Set receptor_ca variable
set_fact:
receptor_ca: '{{ _receptor_ca }}'
no_log: "{{ no_log }}"
- name: Remove tempfiles
file:
path: "{{ item }}"
@@ -113,14 +143,14 @@
loop:
- "{{ _receptor_ca_key_file.path }}"
- "{{ _receptor_ca_crt_file.path }}"
when: not _receptor_ca['resources'] | default([]) | length
when: not receptor_ca['resources'] | default([]) | length
- name: Check for Receptor work signing Secret
k8s_info:
kind: Secret
namespace: '{{ ansible_operator_meta.namespace }}'
name: '{{ ansible_operator_meta.name }}-receptor-work-signing'
register: _receptor_work_signing
register: receptor_work_signing
no_log: "{{ no_log }}"
- name: Generate Receptor work signing RSA key pair
@@ -151,6 +181,17 @@
apply: true
definition: "{{ lookup('template', 'secrets/receptor_work_signing_secret.yaml.j2') }}"
no_log: "{{ no_log }}"
- name: Read Receptor work signing Secret
k8s_info:
kind: Secret
namespace: '{{ ansible_operator_meta.namespace }}'
name: '{{ ansible_operator_meta.name }}-receptor-work-signing'
register: _receptor_work_signing
no_log: "{{ no_log }}"
- name: Set receptor_work_signing variable
set_fact:
receptor_work_signing: '{{ _receptor_work_signing }}'
no_log: "{{ no_log }}"
- name: Remove tempfiles
file:
path: "{{ item }}"
@@ -158,16 +199,16 @@
loop:
- "{{ _receptor_work_signing_private_key_file.path }}"
- "{{ _receptor_work_signing_public_key_file.path }}"
when: not _receptor_work_signing['resources'] | default([]) | length
when: not receptor_work_signing['resources'] | default([]) | length
- name: Apply Resources
k8s:
apply: yes
definition: "{{ lookup('template', item + '.yaml.j2') }}"
wait: yes
register: tower_resources_result
loop:
- 'configmaps/config'
- 'configmaps/pre_stop_scripts'
- 'secrets/app_credentials'
- 'rbac/service_account'
- 'storage/persistent'
@@ -210,21 +251,10 @@
apply: yes
definition: "{{ lookup('template', 'deployments/deployment.yaml.j2') }}"
wait: yes
wait_timeout: "{{ 120 * replicas or 120 }}"
register: this_deployment_result
- block:
- name: Delete pod to reload a resource configuration
k8s:
api_version: v1
state: absent
kind: Pod
namespace: '{{ ansible_operator_meta.namespace }}'
name: '{{ tower_pod_name }}'
wait: yes
when:
- tower_resources_result.changed
- tower_pod_name | length
- name: Get the new resource pod information after updating resource.
k8s_info:
kind: Pod
@@ -236,17 +266,20 @@
field_selectors:
- status.phase=Running
register: _new_pod
until:
- _new_pod['resources'] | length
- _new_pod['resources'][0]['metadata']['name'] != tower_pod_name
delay: 5
retries: 60
- name: Update new resource pod as a variable.
set_fact:
tower_pod: >-
{{ _new_pod['resources']
| rejectattr('metadata.deletionTimestamp', 'defined')
| sort(attribute='metadata.creationTimestamp')
| last | default({}) }}
- name: Update new resource pod name as a variable.
set_fact:
tower_pod_name: '{{ _new_pod["resources"][0]["metadata"]["name"] }}'
tower_pod_name: '{{ tower_pod["metadata"]["name"] | default("")}}'
when:
- tower_resources_result.changed or this_deployment_result.changed
- this_deployment_result.changed
- name: Verify the resource pod name is populated.
assert:

View File

@@ -40,10 +40,10 @@
- name: Set secret key secret
set_fact:
__secret_key_secret: '{{ _generated_secret_key["resources"] | default([]) | length | ternary(_generated_secret_key, _secret_key_secret) }}'
secret_key: '{{ _generated_secret_key["resources"] | default([]) | length | ternary(_generated_secret_key, _secret_key_secret) }}'
no_log: "{{ no_log }}"
- name: Store secret key secret name
set_fact:
secret_key_secret_name: "{{ __secret_key_secret['resources'][0]['metadata']['name'] }}"
secret_key_secret_name: "{{ secret_key['resources'][0]['metadata']['name'] }}"
no_log: "{{ no_log }}"

View File

@@ -17,3 +17,20 @@
{{ _custom_init_container_image |
default(lookup('env', 'RELATED_IMAGE_AWX_INIT_CONTAINER')) |
default(_default_init_container_image, true) }}
- name: Set default awx init projects container image
set_fact:
_default_init_projects_container_image: "{{ _init_projects_container_image }}"
- name: Set user provided awx init projects image
set_fact:
_custom_init_projects_container_image: "{{ init_projects_container_image }}"
when:
- init_projects_container_image | default([]) | length
- name: Set Init projects image URL
set_fact:
_init_projects_container_image: >-
{{ _custom_init_projects_container_image |
default(lookup('env', 'RELATED_IMAGE_AWX_INIT_PROJECTS_CONTAINER')) |
default(_default_init_projects_container_image, true) }}

View File

@@ -0,0 +1,16 @@
{% if termination_grace_period_seconds is defined %}
apiVersion: v1
kind: ConfigMap
metadata:
name: '{{ ansible_operator_meta.name }}-{{ deployment_type }}-pre-stop-scripts'
namespace: '{{ ansible_operator_meta.namespace }}'
labels:
{{ lookup("template", "../common/templates/labels/common.yaml.j2") | indent(width=4) | trim }}
data:
termination-master: |
{{ lookup("file", "files/pre-stop/termination-master") | indent(width=4) | trim }}
termination-waiter: |
{{ lookup("file", "files/pre-stop/termination-waiter") | indent(width=4) | trim }}
termination-env: |
{{ lookup("file", "files/pre-stop/termination-env") | indent(width=4) | trim }}
{% endif %}

View File

@@ -20,8 +20,26 @@ spec:
labels:
{{ lookup("template", "../common/templates/labels/common.yaml.j2") | indent(width=8) | trim }}
{{ lookup("template", "../common/templates/labels/version.yaml.j2") | indent(width=8) | trim }}
{% if annotations %}
annotations:
{% for template in [
"configmaps/config",
"configmaps/pre_stop_scripts",
"secrets/app_credentials",
"storage/persistent",
] %}
checksum-{{ template | replace('/', '-') }}: "{{ lookup('template', template + '.yaml.j2') | sha1 }}"
{% endfor %}
{% for secret in [
"bundle_cacert",
"route_tls",
"ldap_cacert",
"secret_key",
"receptor_ca",
"receptor_work_signing",
] %}
checksum-secret-{{ secret }}: "{{ lookup('ansible.builtin.vars', secret, default='')["resources"][0]["data"] | default('') | sha1 }}"
{% endfor %}
{% if annotations %}
{{ annotations | indent(width=8) }}
{% endif %}
spec:
@@ -122,6 +140,23 @@ spec:
mountPath: "/var/run/redis"
- name: "{{ ansible_operator_meta.name }}-redis-data"
mountPath: "/data"
{% if termination_grace_period_seconds is defined %}
- name: pre-stop-data
mountPath: /var/lib/pre-stop
- name: pre-stop-scripts
mountPath: /var/lib/pre-stop/scripts
lifecycle:
preStop:
exec:
command:
- bash
- -c
# redis image doesn't support writing to `/proc/1/fd/*`
- >
PRE_STOP_STDOUT=/dev/stdout
PRE_STOP_STDERR=/dev/stderr
/var/lib/pre-stop/scripts/termination-waiter
{% endif %}
resources: {{ redis_resource_requirements }}
- image: '{{ _image }}'
name: '{{ ansible_operator_meta.name }}-web'
@@ -291,6 +326,18 @@ spec:
{% endif %}
{% if task_extra_volume_mounts -%}
{{ task_extra_volume_mounts | indent(width=12, first=True) }}
{% endif %}
{% if termination_grace_period_seconds is defined %}
- name: pre-stop-data
mountPath: /var/lib/pre-stop
- name: pre-stop-scripts
mountPath: /var/lib/pre-stop/scripts
lifecycle:
preStop:
exec:
command:
- bash
- /var/lib/pre-stop/scripts/termination-master
{% endif %}
env:
- name: SUPERVISOR_WEB_CONFIG_PATH
@@ -360,6 +407,18 @@ spec:
mountPath: "/var/lib/awx/projects"
{% if ee_extra_volume_mounts -%}
{{ ee_extra_volume_mounts | indent(width=12, first=True) }}
{% endif %}
{% if termination_grace_period_seconds is defined %}
- name: pre-stop-data
mountPath: /var/lib/pre-stop
- name: pre-stop-scripts
mountPath: /var/lib/pre-stop/scripts
lifecycle:
preStop:
exec:
command:
- bash
- /var/lib/pre-stop/scripts/termination-waiter
{% endif %}
env:
{% if development_mode | bool %}
@@ -395,6 +454,9 @@ spec:
{% if security_context_settings|length %}
{{ security_context_settings | to_nice_yaml | indent(8) }}
{% endif %}
{% endif %}
{% if termination_grace_period_seconds is defined %}
terminationGracePeriodSeconds: {{ termination_grace_period_seconds }}
{% endif %}
volumes:
{% if bundle_ca_crt %}
@@ -424,6 +486,14 @@ spec:
items:
- key: ldap-ca.crt
path: 'ldap-ca.crt'
{% endif %}
{% if termination_grace_period_seconds is defined %}
- name: pre-stop-data
emptyDir: {}
- name: pre-stop-scripts
configMap:
name: '{{ ansible_operator_meta.name }}-{{ deployment_type }}-pre-stop-scripts'
defaultMode: 0775
{% endif %}
- name: "{{ ansible_operator_meta.name }}-application-credentials"
secret:

View File

@@ -32,24 +32,24 @@
- this_restore['resources'][0]['metadata']['labels']
- block:
- include_tasks: init.yml
- include_tasks: init.yml
- include_tasks: import_vars.yml
- include_tasks: import_vars.yml
- include_tasks: secrets.yml
- include_tasks: secrets.yml
- include_tasks: deploy_awx.yml
- include_tasks: deploy_awx.yml
- include_tasks: postgres.yml
- include_tasks: postgres.yml
- name: Set flag signifying this restore was successful
set_fact:
tower_restore_complete: True
- name: Set flag signifying this restore was successful
set_fact:
tower_restore_complete: True
- include_tasks: cleanup.yml
- include_tasks: cleanup.yml
when:
- this_restore['resources'][0]['status']['restoreComplete'] is not defined
- this_restore['resources'][0]['status']['restoreComplete'] is not defined
- name: Update status variables
include_tasks: update_status.yml

View File

@@ -66,9 +66,8 @@
- name: Set full resolvable host name for postgres pod
set_fact:
resolvable_db_host: "{{ awx_postgres_host }}.{{ ansible_operator_meta.namespace }}.svc.{{ cluster_name }}"
resolvable_db_host: '{{ (awx_postgres_type == "managed") | ternary(awx_postgres_host + "." + ansible_operator_meta.namespace + ".svc." + cluster_name, awx_postgres_host) }}' # yamllint disable-line rule:line-length
no_log: "{{ no_log }}"
when: awx_postgres_type == 'managed'
- name: Set pg_restore command
set_fact: