internet/awx-operator
3
0
Fork 0
mirror of https://github.com/ansible/awx-operator.git synced 2026-03-26 21:33:14 +00:00
Code Issues Packages Projects Releases Wiki Activity

Upgrading to PostgreSQL 15 and moving to sclorg images (#1486)

Browse Source 
* Upgrading to postgres:15
* Changing image from postgres to sclorg
* Handle scenario where upgrade status is not defined & correct pg tag
* Rework the upgrade logic to be more resiliant for multiple upgrades

---------

Co-authored-by: john-westcott-iv <john-westcott-iv@users.noreply.github.com>
Co-authored-by: Christian M. Adams <chadams@redhat.com>
This commit is contained in:
John Westcott IV
2024-02-29 17:02:11 -05:00
committed by GitHub
parent d11d66e81d
commit 607a7ca58c
15 changed files with 129 additions and 72 deletions
Download Patch File Download Diff File Expand all files Collapse all files

20
CONTRIBUTING.md
View File

@@ -6,13 +6,15 @@ Have questions about this document or anything not covered here? Please file a n
## Table of contents ## Table of contents
* [Things to know prior to submitting code](#things-to-know-prior-to-submitting-code) - [AWX-Operator Contributing Guidelines](#awx-operator-contributing-guidelines)
* [Submmiting your Work](#submitting-your-work) - [Table of contents](#table-of-contents)
* [Testing](#testing) - [Things to know prior to submitting code](#things-to-know-prior-to-submitting-code)
* [Testing in Docker](#testing-in-docker) - [Submmiting your work](#submmiting-your-work)
* [Testing in Minikube](#testing-in-minikube) - [Testing](#testing)
* [Generating a bundle](#generating-a-bundle) - [Testing in Kind](#testing-in-kind)
* [Reporting Issues](#reporting-issues) - [Testing in Minikube](#testing-in-minikube)
- [Generating a bundle](#generating-a-bundle)
- [Reporting Issues](#reporting-issues)
## Things to know prior to submitting code ## Things to know prior to submitting code
@@ -44,12 +46,12 @@ Have questions about this document or anything not covered here? Please file a n
## Testing ## Testing
This Operator includes a [Molecule](https://molecule.readthedocs.io/en/stable/)-based test environment, which can be executed standalone in Docker (e.g. in CI or in a single Docker container anywhere), or inside any kind of Kubernetes cluster (e.g. Minikube). This Operator includes a [Molecule](https://ansible.readthedocs.io/projects/molecule/)-based test environment, which can be executed standalone in Docker (e.g. in CI or in a single Docker container anywhere), or inside any kind of Kubernetes cluster (e.g. Minikube).
You need to make sure you have Molecule installed before running the following commands. You can install Molecule with: You need to make sure you have Molecule installed before running the following commands. You can install Molecule with:
```sh ```sh
#> pip install 'molecule[docker]' #> python -m pip install molecule-plugins[docker]
``` ```
Running `molecule test` sets up a clean environment, builds the operator, runs all configured tests on an example operator instance, then tears down the environment (at least in the case of Docker). Running `molecule test` sets up a clean environment, builds the operator, runs all configured tests on an example operator instance, then tears down the environment (at least in the case of Docker).

4
config/manager/kustomization.yaml
View File

@@ -5,9 +5,9 @@ generatorOptions:
disableNameSuffixHash: true disableNameSuffixHash: true
configMapGenerator: configMapGenerator:
- name: awx-manager-config - files:
files:
- controller_manager_config.yaml - controller_manager_config.yaml
name: awx-manager-config
apiVersion: kustomize.config.k8s.io/v1beta1 apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization kind: Kustomization

2
docs/upgrade/upgrading.md
View File

@@ -14,7 +14,7 @@ The first part of any upgrade should be a backup. Note, there are secrets in the
In the event you need to recover the backup see the [restore role documentation](https://github.com/ansible/awx-operator/tree/devel/roles/restore). *Before Restoring from a backup*, be sure to: In the event you need to recover the backup see the [restore role documentation](https://github.com/ansible/awx-operator/tree/devel/roles/restore). *Before Restoring from a backup*, be sure to:
* delete the old existing AWX CR * delete the old existing AWX CR
* delete the persistent volume claim (PVC) for the database from the old deployment, which has a name like `postgres-13-<deployment-name>-postgres-13-0` * delete the persistent volume claim (PVC) for the database from the old deployment, which has a name like `postgres-15-<deployment-name>-postgres-15-0`
**Note**: Do not delete the namespace/project, as that will delete the backup and the backup's PVC as well. **Note**: Do not delete the namespace/project, as that will delete the backup and the backup's PVC as well.

38
docs/user-guide/advanced-configuration/assigning-awx-pods-to-specific-nodes.md
View File

@@ -9,25 +9,25 @@ If you want to use affinity rules for your AWX pod you can use the `affinity` op
If you want to constrain the web and task pods individually, you can do so by specificying the deployment type before the specific setting. For If you want to constrain the web and task pods individually, you can do so by specificying the deployment type before the specific setting. For
example, specifying `task_tolerations` will allow the AWX task pod to be scheduled onto nodes with matching taints. example, specifying `task_tolerations` will allow the AWX task pod to be scheduled onto nodes with matching taints.
| Name | Description | Default | | Name | Description | Default |
| -------------------------------- | ---------------------------------------- | ------- | | -------------------------------- | ---------------------------------------- | -------------------------------- |
| postgres_image | Path of the image to pull | postgres | | postgres_image | Path of the image to pull | quay.io/sclorg/postgresql-15-c9s |
| postgres_image_version | Image version to pull | 13 | | postgres_image_version | Image version to pull | latest |
| node_selector | AWX pods' nodeSelector | '' | | node_selector | AWX pods' nodeSelector | '' |
| web_node_selector | AWX web pods' nodeSelector | '' | | web_node_selector | AWX web pods' nodeSelector | '' |
| task_node_selector | AWX task pods' nodeSelector | '' | | task_node_selector | AWX task pods' nodeSelector | '' |
| topology_spread_constraints | AWX pods' topologySpreadConstraints | '' | | topology_spread_constraints | AWX pods' topologySpreadConstraints | '' |
| web_topology_spread_constraints | AWX web pods' topologySpreadConstraints | '' | | web_topology_spread_constraints | AWX web pods' topologySpreadConstraints | '' |
| task_topology_spread_constraints | AWX task pods' topologySpreadConstraints | '' | | task_topology_spread_constraints | AWX task pods' topologySpreadConstraints | '' |
| affinity | AWX pods' affinity rules | '' | | affinity | AWX pods' affinity rules | '' |
| web_affinity | AWX web pods' affinity rules | '' | | web_affinity | AWX web pods' affinity rules | '' |
| task_affinity | AWX task pods' affinity rules | '' | | task_affinity | AWX task pods' affinity rules | '' |
| tolerations | AWX pods' tolerations | '' | | tolerations | AWX pods' tolerations | '' |
| web_tolerations | AWX web pods' tolerations | '' | | web_tolerations | AWX web pods' tolerations | '' |
| task_tolerations | AWX task pods' tolerations | '' | | task_tolerations | AWX task pods' tolerations | '' |
| annotations | AWX pods' annotations | '' | | annotations | AWX pods' annotations | '' |
| postgres_selector | Postgres pods' nodeSelector | '' | | postgres_selector | Postgres pods' nodeSelector | '' |
| postgres_tolerations | Postgres pods' tolerations | '' | | postgres_tolerations | Postgres pods' tolerations | '' |
Example of customization could be: Example of customization could be:

20
docs/user-guide/database-configuration.md
View File

@@ -2,7 +2,7 @@
#### Postgres Version #### Postgres Version
The default Postgres version for the version of AWX bundled with the latest version of the awx-operator is Postgres 13. You can find this default for a given version by at the default value for [_postgres_image_version](https://github.com/ansible/awx-operator/blob/devel/roles/installer/defaults/main.yml#L243). The default Postgres version for the version of AWX bundled with the latest version of the awx-operator is Postgres 15. You can find this default for a given version by at the default value for [_postgres_image_version](https://github.com/ansible/awx-operator/blob/devel/roles/installer/defaults/main.yml#L243).
We only have coverage for the default version of Postgres. Newer versions of Postgres (14+) will likely work, but should only be configured as an external database. If your database is managed by the awx-operator (default if you don't specify a `postgres_configuration_secret`), then you should not override the default version as this may cause issues when awx-operator tries to upgrade your postgresql pod. We only have coverage for the default version of Postgres. Newer versions of Postgres (14+) will likely work, but should only be configured as an external database. If your database is managed by the awx-operator (default if you don't specify a `postgres_configuration_secret`), then you should not override the default version as this may cause issues when awx-operator tries to upgrade your postgresql pod.
@@ -56,15 +56,15 @@ If you don't have access to an external PostgreSQL service, the AWX operator can
The following variables are customizable for the managed PostgreSQL service The following variables are customizable for the managed PostgreSQL service
| Name | Description | Default | | Name | Description | Default |
| --------------------------------------------- | --------------------------------------------- | ---------------------------------- | | --------------------------------------------- | --------------------------------------------- | --------------------------------------- |
| postgres_image | Path of the image to pull | postgres:12 | | postgres_image | Path of the image to pull | quay.io/sclorg/postgresql-15-c9s:latest |
| postgres_init_container_resource_requirements | Database init container resource requirements | requests: {cpu: 10m, memory: 64Mi} | | postgres_init_container_resource_requirements | Database init container resource requirements | requests: {cpu: 10m, memory: 64Mi} |
| postgres_resource_requirements | PostgreSQL container resource requirements | requests: {cpu: 10m, memory: 64Mi} | | postgres_resource_requirements | PostgreSQL container resource requirements | requests: {cpu: 10m, memory: 64Mi} |
| postgres_storage_requirements | PostgreSQL container storage requirements | requests: {storage: 8Gi} | | postgres_storage_requirements | PostgreSQL container storage requirements | requests: {storage: 8Gi} |
| postgres_storage_class | PostgreSQL PV storage class | Empty string | | postgres_storage_class | PostgreSQL PV storage class | Empty string |
| postgres_data_path | PostgreSQL data path | `/var/lib/postgresql/data/pgdata` | | postgres_data_path | PostgreSQL data path | `/var/lib/postgresql/data/pgdata` |
| postgres_priority_class | Priority class used for PostgreSQL pod | Empty string | | postgres_priority_class | Priority class used for PostgreSQL pod | Empty string |
Example of customization could be: Example of customization could be:

2
molecule/default/tasks/awx_replicas_test.yml
View File

@@ -61,4 +61,4 @@
expected_web_replicas: 3 expected_web_replicas: 3
expected_task_replicas: 3 expected_task_replicas: 3
tags: tags:
- replicas - replicas

6
roles/backup/vars/main.yml
View File

@@ -1,8 +1,8 @@
--- ---
deployment_type: "awx" deployment_type: "awx"
_postgres_image: postgres _postgres_image: quay.io/sclorg/postgresql-15-c9s
_postgres_image_version: 13 _postgres_image_version: latest
backup_complete: false backup_complete: false
database_type: "unmanaged" database_type: "unmanaged"
supported_pg_version: 13 supported_pg_version: 15
image_pull_policy: IfNotPresent image_pull_policy: IfNotPresent

4
roles/installer/defaults/main.yml
View File

@@ -255,8 +255,8 @@ _image: quay.io/ansible/awx
_image_version: "{{ lookup('env', 'DEFAULT_AWX_VERSION') or 'latest' }}" _image_version: "{{ lookup('env', 'DEFAULT_AWX_VERSION') or 'latest' }}"
_redis_image: docker.io/redis _redis_image: docker.io/redis
_redis_image_version: 7 _redis_image_version: 7
_postgres_image: postgres _postgres_image: quay.io/sclorg/postgresql-15-c9s
_postgres_image_version: 13 _postgres_image_version: latest
image_pull_policy: IfNotPresent image_pull_policy: IfNotPresent
image_pull_secrets: [] image_pull_secrets: []

56
roles/installer/tasks/database_configuration.yml
View File

@@ -106,14 +106,38 @@
set_fact: set_fact:
managed_database: "{{ pg_config['resources'][0]['data']['type'] | default('') | b64decode == 'managed' }}" managed_database: "{{ pg_config['resources'][0]['data']['type'] | default('') | b64decode == 'managed' }}"
- name: Get the old postgres pod information # It is possible that N-2 postgres pods may still be present in the namespace from previous upgrades.
# So we have to take that into account and preferentially set the most recent one.
- name: Get the old postgres pod (N-1)
k8s_info: k8s_info:
kind: Pod kind: Pod
namespace: "{{ ansible_operator_meta.namespace }}" namespace: "{{ ansible_operator_meta.namespace }}"
name: "{{ ansible_operator_meta.name }}-postgres-0"
field_selectors: field_selectors:
- status.phase=Running - status.phase=Running
register: old_postgres_pod register: _running_pods
- block:
- name: Filter pods by name
set_fact:
filtered_old_postgres_pods: "{{ _running_pods.resources |
selectattr('metadata.name', 'match', ansible_operator_meta.name + '-postgres.*-0') |
rejectattr('metadata.name', 'search', '-' + supported_pg_version | string + '-0') |
list }}"
# Sort pods by name in reverse order (most recent PG version first) and set
- name: Set info for previous postgres pod
set_fact:
sorted_old_postgres_pods: "{{ filtered_old_postgres_pods |
sort(attribute='metadata.name') |
reverse }}"
when: filtered_old_postgres_pods | length
- name: Set info for previous postgres pod
set_fact:
old_postgres_pod: "{{ sorted_old_postgres_pods | first }}"
when: filtered_old_postgres_pods | length
when: _running_pods.resources | length
- name: Look up details for this deployment - name: Look up details for this deployment
k8s_info: k8s_info:
@@ -123,7 +147,14 @@
namespace: "{{ ansible_operator_meta.namespace }}" namespace: "{{ ansible_operator_meta.namespace }}"
register: this_awx register: this_awx
- name: Check if postgres pod is running and version 12 # If this deployment has been upgraded before or if upgrade has already been started, set this var
- name: Set previous PG version var
set_fact:
_previous_upgraded_pg_version: "{{ this_awx['resources'][0]['status']['upgradedPostgresVersion'] | default(false) }}"
when:
- "'upgradedPostgresVersion' in this_awx['resources'][0]['status']"
- name: Check if postgres pod is running an older version
block: block:
- name: Set path to PG_VERSION file for given container image - name: Set path to PG_VERSION file for given container image
set_fact: set_fact:
@@ -132,21 +163,24 @@
- name: Get old PostgreSQL version - name: Get old PostgreSQL version
k8s_exec: k8s_exec:
namespace: "{{ ansible_operator_meta.namespace }}" namespace: "{{ ansible_operator_meta.namespace }}"
pod: "{{ ansible_operator_meta.name }}-postgres-0" pod: "{{ old_postgres_pod['metadata']['name'] }}"
command: | command: |
bash -c """ bash -c """
cat {{ path_to_pg_version }} cat {{ path_to_pg_version }}
""" """
register: _old_pg_version register: _old_pg_version
- name: Upgrade data dir from Postgres 12 to 13 if applicable - debug:
msg: "--- Upgrading from {{ old_postgres_pod['metadata']['name'] | default('NONE')}} Pod ---"
- name: Upgrade data dir from old Postgres to {{ supported_pg_version }} if applicable
include_tasks: upgrade_postgres.yml include_tasks: upgrade_postgres.yml
when: when:
- _old_pg_version.stdout | default('0') | trim == '12' - (_old_pg_version.stdout | default(0) | int ) < supported_pg_version
when: when:
- managed_database - managed_database
- this_awx['resources'][0]['status']['upgradedPostgresVersion'] | default('none') != '12' - (_previous_upgraded_pg_version | default(false)) | ternary(_previous_upgraded_pg_version < supported_pg_version, true)
- old_postgres_pod['resources'] | length # upgrade is complete and old pg pod has been removed - old_postgres_pod | length # If empty, then old pg pod has been removed and we can assume the upgrade is complete
- block: - block:
- name: Create Database if no database is specified - name: Create Database if no database is specified
@@ -167,7 +201,7 @@
kubernetes.core.k8s_scale: kubernetes.core.k8s_scale:
api_version: apps/v1 api_version: apps/v1
kind: StatefulSet kind: StatefulSet
name: "{{ ansible_operator_meta.name }}-postgres-13" name: "{{ ansible_operator_meta.name }}-postgres-{{ supported_pg_version }}"
namespace: "{{ ansible_operator_meta.namespace }}" namespace: "{{ ansible_operator_meta.namespace }}"
replicas: 0 replicas: 0
wait: yes wait: yes
@@ -177,7 +211,7 @@
state: absent state: absent
api_version: apps/v1 api_version: apps/v1
kind: StatefulSet kind: StatefulSet
name: "{{ ansible_operator_meta.name }}-postgres-13" name: "{{ ansible_operator_meta.name }}-postgres-{{ supported_pg_version }}"
namespace: "{{ ansible_operator_meta.namespace }}" namespace: "{{ ansible_operator_meta.namespace }}"
wait: yes wait: yes
when: create_statefulset_result.error == 422 when: create_statefulset_result.error == 422

2
roles/installer/tasks/update_status.yml
View File

@@ -111,5 +111,5 @@
name: "{{ ansible_operator_meta.name }}" name: "{{ ansible_operator_meta.name }}"
namespace: "{{ ansible_operator_meta.namespace }}" namespace: "{{ ansible_operator_meta.namespace }}"
status: status:
upgradedPostgresVersion: "{{ upgraded_postgres_version }}" upgradedPostgresVersion: "{{ upgraded_postgres_version | string }}"
when: upgraded_postgres_version is defined when: upgraded_postgres_version is defined

33
roles/installer/tasks/upgrade_postgres.yml
View File

@@ -1,9 +1,9 @@
--- ---
# Upgrade Posgres (Managed Databases only) # Upgrade Posgres (Managed Databases only)
# * If postgres version is not 12, and not an external postgres instance (when managed_database is yes), # * If postgres version is not supported_pg_version, and not an external postgres instance (when managed_database is yes),
# then run this playbook with include_tasks from database_configuration.yml # then run this playbook with include_tasks from database_configuration.yml
# * Data will be streamed via a pg_dump from the postgres 12 pod to the postgres 13 # * Data will be streamed via a pg_dump from the postgres 12/13 pod to the postgres supported_pg_version
# pod via a pg_restore. # pod via a pg_restore.
@@ -62,9 +62,19 @@
set_fact: set_fact:
postgres_pod_name: "{{ postgres_pod['resources'][0]['metadata']['name'] }}" postgres_pod_name: "{{ postgres_pod['resources'][0]['metadata']['name'] }}"
- name: Get the name of the service for the old postgres pod
k8s_info:
kind: Service
namespace: "{{ ansible_operator_meta.namespace }}"
label_selectors:
- "app.kubernetes.io/component=database"
- "app.kubernetes.io/instance={{ old_postgres_pod.metadata.labels['app.kubernetes.io/instance'] }}"
- "app.kubernetes.io/managed-by=awx-operator"
register: old_postgres_svc
- name: Set full resolvable host name for postgres pod - name: Set full resolvable host name for postgres pod
set_fact: set_fact:
resolvable_db_host: "{{ ansible_operator_meta.name }}-postgres.{{ ansible_operator_meta.namespace }}.svc" # yamllint disable-line rule:line-length resolvable_db_host: "{{ old_postgres_svc['resources'][0]['metadata']['name'] }}.{{ ansible_operator_meta.namespace }}.svc" # yamllint disable-line rule:line-length
no_log: "{{ no_log }}" no_log: "{{ no_log }}"
- name: Set pg_dump command - name: Set pg_dump command
@@ -118,7 +128,7 @@
- name: Set flag signifying that this instance has been migrated - name: Set flag signifying that this instance has been migrated
set_fact: set_fact:
upgraded_postgres_version: '13' upgraded_postgres_version: '{{ supported_pg_version }}'
# Cleanup old Postgres resources # Cleanup old Postgres resources
- name: Remove old Postgres StatefulSet - name: Remove old Postgres StatefulSet
@@ -126,23 +136,32 @@
kind: StatefulSet kind: StatefulSet
api_version: v1 api_version: v1
namespace: "{{ ansible_operator_meta.namespace }}" namespace: "{{ ansible_operator_meta.namespace }}"
name: "{{ ansible_operator_meta.name }}-postgres" name: "{{ item }}"
state: absent state: absent
wait: true wait: true
loop:
- "{{ ansible_operator_meta.name }}-postgres"
- "{{ ansible_operator_meta.name }}-postgres-13"
- name: Remove old Postgres Service - name: Remove old Postgres Service
k8s: k8s:
kind: Service kind: Service
api_version: v1 api_version: v1
namespace: "{{ ansible_operator_meta.namespace }}" namespace: "{{ ansible_operator_meta.namespace }}"
name: "{{ ansible_operator_meta.name }}-postgres" name: "{{ item }}"
state: absent state: absent
loop:
- "{{ ansible_operator_meta.name }}-postgres"
- "{{ ansible_operator_meta.name }}-postgres-13"
- name: Remove old persistent volume claim - name: Remove old persistent volume claim
k8s: k8s:
kind: PersistentVolumeClaim kind: PersistentVolumeClaim
api_version: v1 api_version: v1
namespace: "{{ ansible_operator_meta.namespace }}" namespace: "{{ ansible_operator_meta.namespace }}"
name: "postgres-{{ ansible_operator_meta.name }}-postgres-0" name: "{{ item }}"
state: absent state: absent
loop:
- "postgres-{{ ansible_operator_meta.name }}-postgres-0"
- "postgres-{{ ansible_operator_meta.name }}-postgres-13-0"
when: postgres_keep_pvc_after_upgrade when: postgres_keep_pvc_after_upgrade

2
roles/installer/templates/statefulsets/postgres.yaml.j2
View File

@@ -59,7 +59,7 @@ spec:
args: {{ postgres_extra_args }} args: {{ postgres_extra_args }}
{% endif %} {% endif %}
env: env:
# For postgres_image based on rhel8/postgresql-13 # For postgres_image based on rhel8/postgresql-{{ supported_pg_version }}
- name: POSTGRESQL_DATABASE - name: POSTGRESQL_DATABASE
valueFrom: valueFrom:
secretKeyRef: secretKeyRef:

4
roles/installer/vars/main.yml
View File

@@ -4,4 +4,6 @@ postgres_host_auth_method: 'scram-sha-256'
ldap_cacert_ca_crt: '' ldap_cacert_ca_crt: ''
bundle_ca_crt: '' bundle_ca_crt: ''
projects_existing_claim: '' projects_existing_claim: ''
supported_pg_version: 13 supported_pg_version: 15
_previous_upgraded_pg_version: 0
old_postgres_pod: []

2
roles/restore/README.md
View File

@@ -19,7 +19,7 @@ This role assumes you are authenticated with an Openshift or Kubernetes cluster:
*Before Restoring from a backup*, be sure to: *Before Restoring from a backup*, be sure to:
- delete the old existing AWX CR - delete the old existing AWX CR
- delete the persistent volume claim (PVC) for the database from the old deployment, which has a name like `postgres-13-<deployment-name>-postgres-13-0` - delete the persistent volume claim (PVC) for the database from the old deployment, which has a name like `postgres-<postgres version>-<deployment-name>-postgres-<postgres version>-0`
**Note**: Do not delete the namespace/project, as that will delete the backup and the backup's PVC as well. **Note**: Do not delete the namespace/project, as that will delete the backup and the backup's PVC as well.

6
roles/restore/vars/main.yml
View File

@@ -1,8 +1,8 @@
--- ---
deployment_type: "awx" deployment_type: "awx"
_postgres_image: postgres _postgres_image: quay.io/sclorg/postgresql-15-c9s
_postgres_image_version: 13 _postgres_image_version: latest
backup_api_version: '{{ deployment_type }}.ansible.com/v1beta1' backup_api_version: '{{ deployment_type }}.ansible.com/v1beta1'
backup_kind: 'AWXBackup' backup_kind: 'AWXBackup'
@@ -12,7 +12,7 @@ secret_key_secret: '{{ deployment_name }}-secret-key'
admin_password_secret: '{{ deployment_name }}-admin-password' admin_password_secret: '{{ deployment_name }}-admin-password'
broadcast_websocket_secret: '{{ deployment_name }}-broadcast-websocket' broadcast_websocket_secret: '{{ deployment_name }}-broadcast-websocket'
postgres_configuration_secret: '{{ deployment_name }}-postgres-configuration' postgres_configuration_secret: '{{ deployment_name }}-postgres-configuration'
supported_pg_version: 13 supported_pg_version: 15
image_pull_policy: IfNotPresent image_pull_policy: IfNotPresent
# If set to true, the restore process will delete the existing database and create a new one # If set to true, the restore process will delete the existing database and create a new one