Revert change to cast settings values to strings as they could be nested (#1756 )

- We cast the settings value to a string so that it would display properly in the Openshift UI. Unfortunately, the k8s validator will no longer allow arrays for settings values.
Create a new postgres configuration secret when restoring a new instance (#1733 )
2026-03-27 05:43:11 +00:00 · 2024-03-12 11:58:00 -04:00 · 2024-03-11 16:34:43 -04:00 · 2024-03-11 19:01:00 +00:00 · 2024-03-11 14:12:10 -04:00 · 2024-03-08 09:02:31 -05:00
50 changed files with 1066 additions and 218 deletions
--- a/.gitignore
+++ b/.gitignore
@@ -10,3 +10,4 @@ gh-pages/
 .vscode/
 __pycache__
 /site
+venv/*
--- a/.helm/starter/README.md
+++ b/.helm/starter/README.md
@@ -36,6 +36,29 @@ When upgrading to releases with CRD changes use the following command to update
 kubectl apply --server-side -k github.com/ansible/awx-operator/config/crd?ref=<VERSION>
 ```

+If running above command results in an error like below:
+
+```text
+Apply failed with 1 conflict: conflict with "helm" using apiextensions.k8s.io/v1: .spec.versions
+Please review the fields above--they currently have other managers. Here
+are the ways you can resolve this warning:
+* If you intend to manage all of these fields, please re-run the apply
+  command with the `--force-conflicts` flag.
+* If you do not intend to manage all of the fields, please edit your
+  manifest to remove references to the fields that should keep their
+  current managers.
+* You may co-own fields by updating your manifest to match the existing
+  value; in this case, you'll become the manager if the other manager(s)
+  stop managing the field (remove it from their configuration).
+See https://kubernetes.io/docs/reference/using-api/server-side-apply/#conflicts
+```
+
+Use `--force-conflicts` flag to resolve the conflict.
+
+```bash
+kubectl apply --server-side --force-conflicts -k github.com/ansible/awx-operator/config/crd?ref=<VERSION>
+```
+
 ## Configuration
 The goal of adding helm configurations is to abstract out and simplify the creation of multi-resource configs. The `AWX.spec` field maps directly to the spec configs of the `AWX` resource that the operator provides, which are detailed in the [main README](https://github.com/ansible/awx-operator/blob/devel/README.md). Other sub-config can be added with the goal of simplifying more involved setups that require additional resources to be specified.

--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -6,13 +6,15 @@ Have questions about this document or anything not covered here? Please file a n

 ## Table of contents

-* [Things to know prior to submitting code](#things-to-know-prior-to-submitting-code)
-* [Submmiting your Work](#submitting-your-work)
-* [Testing](#testing)
-    * [Testing in Docker](#testing-in-docker)
-    * [Testing in Minikube](#testing-in-minikube)
-* [Generating a bundle](#generating-a-bundle)
-* [Reporting Issues](#reporting-issues)
+- [AWX-Operator Contributing Guidelines](#awx-operator-contributing-guidelines)
+  - [Table of contents](#table-of-contents)
+  - [Things to know prior to submitting code](#things-to-know-prior-to-submitting-code)
+  - [Submmiting your work](#submmiting-your-work)
+  - [Testing](#testing)
+      - [Testing in Kind](#testing-in-kind)
+      - [Testing in Minikube](#testing-in-minikube)
+  - [Generating a bundle](#generating-a-bundle)
+  - [Reporting Issues](#reporting-issues)


 ## Things to know prior to submitting code
@@ -44,12 +46,12 @@ Have questions about this document or anything not covered here? Please file a n

 ## Testing

-This Operator includes a [Molecule](https://molecule.readthedocs.io/en/stable/)-based test environment, which can be executed standalone in Docker (e.g. in CI or in a single Docker container anywhere), or inside any kind of Kubernetes cluster (e.g. Minikube).
+This Operator includes a [Molecule](https://ansible.readthedocs.io/projects/molecule/)-based test environment, which can be executed standalone in Docker (e.g. in CI or in a single Docker container anywhere), or inside any kind of Kubernetes cluster (e.g. Minikube).

 You need to make sure you have Molecule installed before running the following commands. You can install Molecule with:

 ```sh
-#> pip install 'molecule[docker]'
+#> python -m pip install molecule-plugins[docker]
 ```

 Running `molecule test` sets up a clean environment, builds the operator, runs all configured tests on an example operator instance, then tears down the environment (at least in the case of Docker).
--- a/2
+++ b/2
@@ -1,4 +1,4 @@
-FROM quay.io/operator-framework/ansible-operator:v1.32.0
+FROM quay.io/operator-framework/ansible-operator:v1.34.0

 USER root
 RUN dnf update --security --bugfix -y && \
--- a/config/crd/bases/awx.ansible.com_awxmeshingresses.yaml
+++ b/config/crd/bases/awx.ansible.com_awxmeshingresses.yaml
@@ -41,6 +41,11 @@ spec:
              deployment_name:
                description: Name of the AWX deployment to create the Mesh Ingress for.
                type: string
+              image_pull_secrets:
+                description: Image pull secrets for Mesh Ingress containers.
+                type: array
+                items:
+                  type: string
              external_hostname:
                description: External hostname to use for the Mesh Ingress.
                type: string
--- a/config/crd/bases/awx.ansible.com_awxs.yaml
+++ b/config/crd/bases/awx.ansible.com_awxs.yaml
@@ -159,6 +159,10 @@ spec:
                description: Assign LoadBalancer IP address
                type: string
                default: ''
+              loadbalancer_class:
+                description: Class of LoadBalancer to use
+                type: string
+                default: ''
              route_host:
                description: The DNS to use to points to the instance
                type: string
@@ -1571,6 +1575,86 @@ spec:
                description: Number of task instance replicas
                type: integer
                format: int32
+              web_liveness_initial_delay:
+                description: Initial delay before starting liveness checks on web pod
+                type: integer
+                default: 5
+                format: int32
+              task_liveness_initial_delay:
+                description: Initial delay before starting liveness checks on task pod
+                type: integer
+                default: 5
+                format: int32
+              web_liveness_period:
+                description: Time period in seconds between each liveness check for the web pod
+                type: integer
+                default: 0
+                format: int32
+              task_liveness_period:
+                description: Time period in seconds between each liveness check for the task pod
+                type: integer
+                default: 0
+                format: int32
+              web_liveness_failure_threshold:
+                description: Number of consecutive failure events to identify failure of web pod
+                type: integer
+                default: 3
+                format: int32
+              task_liveness_failure_threshold:
+                description: Number of consecutive failure events to identify failure of task pod
+                type: integer
+                default: 3
+                format: int32
+              web_liveness_timeout:
+                description: Number of seconds to wait for a probe response from web pod
+                type: integer
+                default: 1
+                format: int32
+              task_liveness_timeout:
+                description: Number of seconds to wait for a probe response from task pod
+                type: integer
+                default: 1
+                format: int32
+              web_readiness_initial_delay:
+                description: Initial delay before starting readiness checks on web pod
+                type: integer
+                default: 20
+                format: int32
+              task_readiness_initial_delay:
+                description: Initial delay before starting readiness checks on task pod
+                type: integer
+                default: 20
+                format: int32
+              web_readiness_period:
+                description: Time period in seconds between each readiness check for the web pod
+                type: integer
+                default: 0
+                format: int32
+              task_readiness_period:
+                description: Time period in seconds between each readiness check for the task pod
+                type: integer
+                default: 0
+                format: int32
+              web_readiness_failure_threshold:
+                description: Number of consecutive failure events to identify failure of web pod
+                type: integer
+                default: 3
+                format: int32
+              task_readiness_failure_threshold:
+                description: Number of consecutive failure events to identify failure of task pod
+                type: integer
+                default: 3
+                format: int32
+              web_readiness_timeout:
+                description: Number of seconds to wait for a probe response from web pod
+                type: integer
+                default: 1
+                format: int32
+              task_readiness_timeout:
+                description: Number of seconds to wait for a probe response from task pod
+                type: integer
+                default: 1
+                format: int32
              garbage_collect_secrets:
                description: Whether or not to remove secrets upon instance removal
                default: false
@@ -1805,7 +1889,7 @@ spec:
                description: Set log level of receptor service
                type: string
              extra_settings:
-                description: Extra settings to specify for the API
+                description: Extra settings to specify for AWX
                items:
                  properties:
                    setting:
--- a/config/manager/kustomization.yaml
+++ b/config/manager/kustomization.yaml
@@ -5,9 +5,9 @@ generatorOptions:
  disableNameSuffixHash: true

 configMapGenerator:
- name: awx-manager-config
-  files:
+- files:
  - controller_manager_config.yaml
+  name: awx-manager-config

 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
--- a/config/manifests/bases/awx-operator.clusterserviceversion.yaml
+++ b/config/manifests/bases/awx-operator.clusterserviceversion.yaml
@@ -58,6 +58,11 @@ spec:
        path: ingress_controller
        x-descriptors:
        - urn:alm:descriptor:com.tectonic.ui:text
+      - displayName: Image Pull Secrets
+        path: image_pull_secrets
+        x-descriptors:
+        - urn:alm:descriptor:com.tectonic.ui:advanced
+        - urn:alm:descriptor:io.kubernetes:Secret
      version: v1alpha1
    - description: Back up a deployment of the awx, including jobs, inventories, and
        credentials
@@ -91,7 +96,7 @@ spec:
      - displayName: Backup PVC Storage Class
        path: backup_storage_class
        x-descriptors:
-        - urn:alm:descriptor:com.tectonic.ui:text
+        - urn:alm:descriptor:io.kubernetes:StorageClass
        - urn:alm:descriptor:com.tectonic.ui:advanced
      - displayName: Precreate Partition Hours
        path: precreate_partition_hours
@@ -387,6 +392,12 @@ spec:
        - urn:alm:descriptor:com.tectonic.ui:advanced
        - urn:alm:descriptor:com.tectonic.ui:string
        - urn:alm:descriptor:com.tectonic.ui:fieldDependency:service_type:LoadBalancer
+      - displayName: LoadBalancer Class
+        path: loadbalancer_class
+        x-descriptors:
+        - urn:alm:descriptor:com.tectonic.ui:advanced
+        - urn:alm:descriptor:com.tectonic.ui:string
+        - urn:alm:descriptor:com.tectonic.ui:fieldDependency:service_type:LoadBalancer
      - displayName: Route API Version
        path: route_api_version
        x-descriptors:
@@ -713,7 +724,7 @@ spec:
        x-descriptors:
        - urn:alm:descriptor:com.tectonic.ui:advanced
        - urn:alm:descriptor:com.tectonic.ui:fieldDependency:projects_use_existing_claim:_No_
-        - urn:alm:descriptor:com.tectonic.ui:text
+        - urn:alm:descriptor:io.kubernetes:StorageClass
      - description: Projects Storage Size
        displayName: Projects Storage Size
        path: projects_storage_size
@@ -937,7 +948,7 @@ spec:
        x-descriptors:
        - urn:alm:descriptor:com.tectonic.ui:advanced
        - urn:alm:descriptor:com.tectonic.ui:hidden
-      - displayName: API Extra Settings
+      - displayName: Extra Settings
        path: extra_settings
        x-descriptors:
        - urn:alm:descriptor:com.tectonic.ui:advanced
--- a/config/rbac/role.yaml
+++ b/config/rbac/role.yaml
@@ -78,6 +78,17 @@ rules:
      - patch
      - update
      - watch
+  - apiGroups:
+      - batch
+    resources:
+      - jobs
+    verbs:
+      - get
+      - list
+      - create
+      - patch
+      - update
+      - watch
  - apiGroups:
      - monitoring.coreos.com
    resources:
--- a/docs/requirements.in
+++ b/docs/requirements.in
@@ -0,0 +1,3 @@
+# This requirements file is used for AWX Operator latest doc builds.
+
+mkdocs-ansible
--- a/docs/requirements.txt
+++ b/docs/requirements.txt
@@ -1,12 +1,184 @@
+#
+# This file is autogenerated by pip-compile with Python 3.12
+# by the following command:
+#
+#    pip-compile --allow-unsafe --output-file=docs/requirements.txt --strip-extras docs/requirements.in
+#
+babel==2.14.0
+    # via mkdocs-material
+beautifulsoup4==4.12.3
+    # via
+    #   linkchecker
+    #   mkdocs-htmlproofer-plugin
+    #   readtime
+cairocffi==1.6.1
+    # via cairosvg
 cairosvg==2.7.0
-markdown-exec>=1.6.0
-mkdocs-ansible>=0.1.6
-mkdocs-gen-files>=0.4.0
-mkdocs-material-extensions>=1.1.1
-mkdocs-material>=9.1.18
-mkdocs
-mkdocstrings-python>=1.1.0
-mkdocstrings>=0.22.0
+    # via mkdocs-ansible
+certifi==2024.2.2
+    # via requests
+cffi==1.16.0
+    # via cairocffi
+charset-normalizer==3.3.2
+    # via requests
+click==8.1.7
+    # via
+    #   mkdocs
+    #   mkdocstrings
+colorama==0.4.6
+    # via
+    #   griffe
+    #   mkdocs-material
+csscompressor==0.9.5
+    # via mkdocs-minify-plugin
+cssselect==1.2.0
+    # via pyquery
+cssselect2==0.7.0
+    # via cairosvg
+defusedxml==0.7.1
+    # via cairosvg
+dnspython==2.6.1
+    # via linkchecker
+ghp-import==2.1.0
+    # via mkdocs
+griffe==0.40.1
+    # via mkdocstrings-python
+htmlmin2==0.1.13
+    # via mkdocs-minify-plugin
+idna==3.6
+    # via requests
+jinja2==3.1.3
+    # via
+    #   mkdocs
+    #   mkdocs-material
+    #   mkdocstrings
+jsmin==3.0.1
+    # via mkdocs-minify-plugin
+linkchecker==10.4.0
+    # via mkdocs-ansible
+lxml==5.1.0
+    # via
+    #   mkdocs-material
+    #   pyquery
+markdown==3.5.2
+    # via
+    #   markdown-include
+    #   mkdocs
+    #   mkdocs-autorefs
+    #   mkdocs-htmlproofer-plugin
+    #   mkdocs-material
+    #   mkdocstrings
+    #   pymdown-extensions
+markdown-exec==1.8.0
+    # via mkdocs-ansible
+markdown-include==0.8.1
+    # via mkdocs-ansible
+markdown2==2.4.12
+    # via readtime
+markupsafe==2.1.5
+    # via
+    #   jinja2
+    #   mkdocs
+    #   mkdocstrings
+mergedeep==1.3.4
+    # via mkdocs
+mkdocs==1.5.3
+    # via
+    #   mkdocs-ansible
+    #   mkdocs-autorefs
+    #   mkdocs-gen-files
+    #   mkdocs-htmlproofer-plugin
+    #   mkdocs-material
+    #   mkdocs-minify-plugin
+    #   mkdocs-monorepo-plugin
+    #   mkdocstrings
+mkdocs-ansible==24.2.1
+    # via -r docs/requirements.in
+mkdocs-autorefs==0.5.0
+    # via mkdocstrings
+mkdocs-gen-files==0.5.0
+    # via mkdocs-ansible
+mkdocs-htmlproofer-plugin==1.0.0
+    # via mkdocs-ansible
+mkdocs-material==9.2.6
+    # via mkdocs-ansible
+mkdocs-material-extensions==1.3.1
+    # via
+    #   mkdocs-ansible
+    #   mkdocs-material
+mkdocs-minify-plugin==0.8.0
+    # via mkdocs-ansible
+mkdocs-monorepo-plugin==1.1.0
+    # via mkdocs-ansible
+mkdocstrings==0.24.0
+    # via
+    #   mkdocs-ansible
+    #   mkdocstrings-python
+mkdocstrings-python==1.8.0
+    # via mkdocs-ansible
+packaging==23.2
+    # via mkdocs
+paginate==0.5.6
+    # via mkdocs-material
+pathspec==0.12.1
+    # via mkdocs
 pillow==10.0.1
+    # via
+    #   cairosvg
+    #   mkdocs-ansible
 pipdeptree==2.7.1
+    # via mkdocs-ansible
+platformdirs==4.2.0
+    # via
+    #   mkdocs
+    #   mkdocstrings
+pycparser==2.21
+    # via cffi
+pygments==2.17.2
+    # via mkdocs-material
 pymdown-extensions==10.0.1
+    # via
+    #   markdown-exec
+    #   mkdocs-ansible
+    #   mkdocs-material
+    #   mkdocstrings
+pyquery==2.0.0
+    # via readtime
+python-dateutil==2.8.2
+    # via ghp-import
+python-slugify==8.0.4
+    # via mkdocs-monorepo-plugin
+pyyaml==6.0.1
+    # via
+    #   mkdocs
+    #   pymdown-extensions
+    #   pyyaml-env-tag
+pyyaml-env-tag==0.1
+    # via mkdocs
+readtime==3.0.0
+    # via mkdocs-material
+regex==2023.12.25
+    # via mkdocs-material
+requests==2.31.0
+    # via
+    #   linkchecker
+    #   mkdocs-htmlproofer-plugin
+    #   mkdocs-material
+six==1.16.0
+    # via python-dateutil
+soupsieve==2.5
+    # via beautifulsoup4
+text-unidecode==1.3
+    # via python-slugify
+tinycss2==1.2.1
+    # via
+    #   cairosvg
+    #   cssselect2
+urllib3==2.2.1
+    # via requests
+watchdog==4.0.0
+    # via mkdocs
+webencodings==0.5.1
+    # via
+    #   cssselect2
+    #   tinycss2
--- a/docs/upgrade/upgrading.md
+++ b/docs/upgrade/upgrading.md
@@ -14,7 +14,7 @@ The first part of any upgrade should be a backup. Note, there are secrets in the

 In the event you need to recover the backup see the [restore role documentation](https://github.com/ansible/awx-operator/tree/devel/roles/restore). *Before Restoring from a backup*, be sure to:
 * delete the old existing AWX CR
-* delete the persistent volume claim (PVC) for the database from the old deployment, which has a name like `postgres-13-<deployment-name>-postgres-13-0`
+* delete the persistent volume claim (PVC) for the database from the old deployment, which has a name like `postgres-15-<deployment-name>-postgres-15-0`

 **Note**: Do not delete the namespace/project, as that will delete the backup and the backup's PVC as well.

--- a/docs/user-guide/advanced-configuration/adding-execution-nodes.md
+++ b/docs/user-guide/advanced-configuration/adding-execution-nodes.md
@@ -1,26 +1,4 @@
 ### Adding Execution Nodes
 Starting with AWX Operator v0.30.0 and AWX v21.7.0, standalone execution nodes can be added to your deployments.
-See [AWX execution nodes docs](https://github.com/ansible/awx/blob/devel/docs/execution_nodes.md) for information about this feature.
+See [Managing Capacity With Instances](https://ansible.readthedocs.io/projects/awx/en/latest/administration/instances.html) chapter of the AWX Administration Guide for information about this feature.

-#### Custom Receptor CA
-The control nodes on the K8S cluster will communicate with execution nodes via mutual TLS TCP connections, running via Receptor.
-Execution nodes will verify incoming connections by ensuring the x509 certificate was issued by a trusted Certificate Authority (CA).
-
-A user may wish to provide their own CA for this validation. If no CA is provided, AWX Operator will automatically generate one using OpenSSL.
-
-Given custom `ca.crt` and `ca.key` stored locally, run the following,
-
-```bash
-kubectl create secret tls awx-demo-receptor-ca \
-   --cert=/path/to/ca.crt --key=/path/to/ca.key
-```
-
-The secret should be named `{AWX Custom Resource name}-receptor-ca`. In the above the AWX CR name is "awx-demo". Please replace "awx-demo" with your AWX Custom Resource name.
-
-If this secret is created after AWX is deployed, run the following to restart the deployment,
-
-```bash
-kubectl rollout restart deployment awx-demo
-```
-
-**Important Note**, changing the receptor CA will break connections to any existing execution nodes. These nodes will enter an `unavailable` state, and jobs will not be able to run on them. Users will need to download and re-run the install bundle for each execution node. This will replace the TLS certificate files with those signed by the new CA. The execution nodes should then appear in a `ready` state after a few minutes.
--- a/docs/user-guide/advanced-configuration/assigning-awx-pods-to-specific-nodes.md
+++ b/docs/user-guide/advanced-configuration/assigning-awx-pods-to-specific-nodes.md
@@ -9,25 +9,25 @@ If you want to use affinity rules for your AWX pod you can use the `affinity` op
 If you want to constrain the web and task pods individually, you can do so by specificying the deployment type before the specific setting. For
 example, specifying `task_tolerations` will allow the AWX task pod to be scheduled onto nodes with matching taints. 

-| Name                             | Description                              | Default  |
-| -------------------------------- | ---------------------------------------- | -------  |
-| postgres_image                   | Path of the image to pull                | postgres |
-| postgres_image_version           | Image version to pull                    | 13       |
-| node_selector                    | AWX pods' nodeSelector                   | ''       |
-| web_node_selector                | AWX web pods' nodeSelector               | ''       |
-| task_node_selector               | AWX task pods' nodeSelector              | ''       |
-| topology_spread_constraints      | AWX pods' topologySpreadConstraints      | ''       |
-| web_topology_spread_constraints  | AWX web pods' topologySpreadConstraints  | ''       |
-| task_topology_spread_constraints | AWX task pods' topologySpreadConstraints | ''       |
-| affinity                         | AWX pods' affinity rules                 | ''       |
-| web_affinity                     | AWX web pods' affinity rules             | ''       |
-| task_affinity                    | AWX task pods' affinity rules            | ''       |
-| tolerations                      | AWX pods' tolerations                    | ''       |
-| web_tolerations                  | AWX web pods' tolerations                | ''       |
-| task_tolerations                 | AWX task pods' tolerations               | ''       |
-| annotations                      | AWX pods' annotations                    | ''       |
-| postgres_selector                | Postgres pods' nodeSelector              | ''       |
-| postgres_tolerations             | Postgres pods' tolerations               | ''       |
+| Name                             | Description                              | Default                          |
+| -------------------------------- | ---------------------------------------- | -------------------------------- |
+| postgres_image                   | Path of the image to pull                | quay.io/sclorg/postgresql-15-c9s |
+| postgres_image_version           | Image version to pull                    | latest                           |
+| node_selector                    | AWX pods' nodeSelector                   | ''                               |
+| web_node_selector                | AWX web pods' nodeSelector               | ''                               |
+| task_node_selector               | AWX task pods' nodeSelector              | ''                               |
+| topology_spread_constraints      | AWX pods' topologySpreadConstraints      | ''                               |
+| web_topology_spread_constraints  | AWX web pods' topologySpreadConstraints  | ''                               |
+| task_topology_spread_constraints | AWX task pods' topologySpreadConstraints | ''                               |
+| affinity                         | AWX pods' affinity rules                 | ''                               |
+| web_affinity                     | AWX web pods' affinity rules             | ''                               |
+| task_affinity                    | AWX task pods' affinity rules            | ''                               |
+| tolerations                      | AWX pods' tolerations                    | ''                               |
+| web_tolerations                  | AWX web pods' tolerations                | ''                               |
+| task_tolerations                 | AWX task pods' tolerations               | ''                               |
+| annotations                      | AWX pods' annotations                    | ''                               |
+| postgres_selector                | Postgres pods' nodeSelector              | ''                               |
+| postgres_tolerations             | Postgres pods' tolerations               | ''                               |

 Example of customization could be:

--- a/docs/user-guide/advanced-configuration/container-probes.md
+++ b/docs/user-guide/advanced-configuration/container-probes.md
@@ -0,0 +1,41 @@
+#### Container Probes
+These parameters control the usage of liveness and readiness container probes for
+the web and task containers.
+
+#### Web / Task Container Liveness Check
+
+The liveness probe queries the status of the supervisor daemon of the container.  The probe will fail if it
+detects one of the services in a state other than "RUNNING".
+
+| Name         | Description                        | Default |
+| -------------| -----------------------------------|---------|
+| web_liveness_period | Time period in seconds between each probe check.  The value of 0 disables the probe. | 0    |
+| web_liveness_initial_delay | Initial delay before starting probes in seconds | 5   |
+| web_liveness_failure_threshold| Number of consecutive failure events to identify failure of container | 3    |
+| web_liveness_timeout | Number of seconds to wait for a probe response from container | 1    |
+| task_liveness_period | Time period in seconds between each probe check.  The value of 0 disables the probe. | 0    |
+| task_liveness_initial_delay | Initial delay before starting probes in seconds | 5   |
+| task_liveness_failure_threshold| Number of consecutive failure events to identify failure of container | 3    |
+| task_liveness_timeout | Number of seconds to wait for a probe response from container | 1    |
+
+#### Web Container Readiness Check
+
+This is an HTTP check against the status endpoint to confirm the system is still able to respond to web requests.
+
+| Name         | Description                        | Default |
+| -------------| ---------------------------------- | ------- |
+| web_readiness_period | Time period in seconds between each probe check.  The value of 0 disables the probe. | 0    |
+| web_readiness_initial_delay | Initial delay before starting probes in seconds | 5   |
+| web_readiness_failure_threshold| Number of consecutive failure events to identify failure of container | 3    |
+| web_readiness_timeout | Number of seconds to wait for a probe response from container | 1    |
+
+#### Task Container Readiness Check
+
+This is a command probe using the builtin check command of the awx-manage utility.
+
+| Name         | Description                        | Default |
+| -------------| ---------------------------------- | ------- |
+| task_readiness_period | Time period in seconds between each probe check.  The value of 0 disables the probe. | 0    |
+| task_readiness_initial_delay | Initial delay before starting probes in seconds | 5   |
+| task_readiness_failure_threshold| Number of consecutive failure events to identify failure of container | 3    |
+| task_readiness_timeout | Number of seconds to wait for a probe response from container | 1    |
--- a/docs/user-guide/advanced-configuration/custom-receptor-certs.md
+++ b/docs/user-guide/advanced-configuration/custom-receptor-certs.md
@@ -0,0 +1,24 @@
+
+### Custom Receptor CA
+
+The control nodes on the K8S cluster will communicate with execution nodes via mutual TLS TCP connections, running via Receptor.
+Execution nodes will verify incoming connections by ensuring the x509 certificate was issued by a trusted Certificate Authority (CA).
+
+A user may wish to provide their own CA for this validation. If no CA is provided, AWX Operator will automatically generate one using OpenSSL.
+
+Given custom `ca.crt` and `ca.key` stored locally, run the following,
+
+```bash
+kubectl create secret tls awx-demo-receptor-ca \
+   --cert=/path/to/ca.crt --key=/path/to/ca.key
+```
+
+The secret should be named `{AWX Custom Resource name}-receptor-ca`. In the above the AWX CR name is "awx-demo". Please replace "awx-demo" with your AWX Custom Resource name.
+
+If this secret is created after AWX is deployed, run the following to restart the deployment,
+
+```bash
+kubectl rollout restart deployment awx-demo
+```
+
+**Important Note**, changing the receptor CA will break connections to any existing execution nodes. These nodes will enter an `unavailable` state, and jobs will not be able to run on them. Users will need to download and re-run the install bundle for each execution node. This will replace the TLS certificate files with those signed by the new CA. The execution nodes should then appear in a `ready` state after a few minutes.
--- a/docs/user-guide/advanced-configuration/deploying-a-specific-version-of-awx.md
+++ b/docs/user-guide/advanced-configuration/deploying-a-specific-version-of-awx.md
@@ -2,15 +2,15 @@

 There are a few variables that are customizable for awx the image management.

-| Name                | Description               | Default                                 |
-| ------------------- | ------------------------- | --------------------------------------  |
-| image               | Path of the image to pull | quay.io/ansible/awx                     |
-| image_version       | Image version to pull     | value of DEFAULT_AWX_VERSION or latest  |
-| image_pull_policy   | The pull policy to adopt  | IfNotPresent                            |
-| image_pull_secrets  | The pull secrets to use   | None                                    |
-| ee_images           | A list of EEs to register | quay.io/ansible/awx-ee:latest           |
-| redis_image         | Path of the image to pull | docker.io/redis                         |
-| redis_image_version | Image version to pull     | latest                                  |
+| Name                | Description               | Default                                    |
+| ------------------- | ------------------------- | -----------------------------------------  |
+| image               | Path of the image to pull | quay.io/ansible/awx                        |
+| image_version       | Image version to pull     | value of DEFAULT_AWX_VERSION or latest     |
+| image_pull_policy   | The pull policy to adopt  | IfNotPresent                               |
+| image_pull_secrets  | The pull secrets to use   | None                                       |
+| ee_images           | A list of EEs to register | quay.io/ansible/awx-ee:DEFAULT_AWX_VERSION |
+| redis_image         | Path of the image to pull | docker.io/redis                            |
+| redis_image_version | Image version to pull     | latest                                     |

 Example of customization could be:

--- a/docs/user-guide/advanced-configuration/mesh-ingress-instance-listener-address-on-awx-ui.png
+++ b/docs/user-guide/advanced-configuration/mesh-ingress-instance-listener-address-on-awx-ui.png
--- a/docs/user-guide/advanced-configuration/mesh-ingress-instance-on-awx-ui.png
+++ b/docs/user-guide/advanced-configuration/mesh-ingress-instance-on-awx-ui.png
--- a/docs/user-guide/advanced-configuration/mesh-ingress.md
+++ b/docs/user-guide/advanced-configuration/mesh-ingress.md
@@ -0,0 +1,224 @@
+# Mesh Ingress
+
+The mesh ingress allows users to peer external execution and hop nodes into the AWX control plane.
+This guide focuses on how to enable and configure the mesh ingress.
+For more information about remote execution and hop nodes and how to create them, refer to the [Managing Capacity With Instances](https://ansible.readthedocs.io/projects/awx/en/latest/administration/instances.html) chapter of the AWX Administration Guide.
+
+## Prerequisites
+
+- AWX operator version > 2.11.0
+- AWX > 23.8.0
+
+## Deploy and configure AWXMeshIngress
+
+!!! note
+    The mesh ingress uses the `control_plane_ee_image` and `image_pull_policy` fields of the AWX instance to determine image and policy to be adopted.
+    Defaulted to `quay.io/ansible/awx-ee:latest` and `Always`.
+    Currently there are no dedicated parameters to specify the image and policy.
+
+### On Red Hat OpenShift with Operator managed Route
+
+To deploy a mesh ingress on OpenShift, create the AWXMeshIngress resource on the namespace where your AWX instance is running on.
+
+Example:
+
+```yaml
+---
+apiVersion: awx.ansible.com/v1alpha1
+kind: AWXMeshIngress
+metadata:
+  name: <mesh ingress name>
+spec:
+  deployment_name: <awx instance name>
+```
+
+### On Kubernetes with Operator managed Ingress (NGINX)
+
+To deploy a mesh ingress on Kubernetes cluster which has [NGINX Ingress Controller](https://www.nginx.com/products/nginx-ingress-controller/), create the AWXMeshIngress resource on the namespace where your AWX instance is running on.
+
+Note that AWXMeshIngress requires [SSL Passthrough](https://kubernetes.github.io/ingress-nginx/user-guide/tls/#ssl-passthrough) enabled which is disabled by default. Ensure it is enabled on your NGINX Ingress Controller.
+
+By specifying `ingress_controller` as `nginx`, AWX Operator will generate Ingress resource that has `nginx.ingress.kubernetes.io/ssl-passthrough` annotation set to `"true"`.
+
+Example:
+
+```yaml
+---
+apiVersion: awx.ansible.com/v1alpha1
+kind: AWXMeshIngress
+metadata:
+  name: <mesh ingress name>
+spec:
+  deployment_name: <awx instance name>
+
+  ingress_type: Ingress
+  ingress_controller: nginx
+  ingress_class_name: nginx
+
+  external_hostname: <fqdn for mesh ingress>
+```
+
+### On Kubernetes with Operator managed Ingress (Traefik)
+
+To deploy a mesh ingress on Kubernetes cluster which has [Traefik Kubernetes Ingress provider](https://doc.traefik.io/traefik/providers/kubernetes-ingress/), create the AWXMeshIngress resource on the namespace where your AWX instance is running on.
+
+Note that by deploying following AWXMeshIngress, AWX Operator will generate IngressRouteTCP resource that has `websecure` as an `entryPoints`. If this does not satisfy your requirement, refer to [User managed Ingress section](#on-kubernetes-with-user-managed-ingress) and  create an IngressRouteTCP resource manually.
+
+Example:
+
+```yaml
+---
+apiVersion: awx.ansible.com/v1alpha1
+kind: AWXMeshIngress
+metadata:
+  name: <mesh ingress name>
+spec:
+  deployment_name: <awx instance name>
+
+  ingress_type: IngressRouteTCP
+  ingress_controller: traefik
+  ingress_class_name: traefik
+  ingress_api_version: traefik.io/v1alpha1
+
+  external_hostname: <fqdn for mesh ingress>
+```
+
+### On Kubernetes with User managed Ingress
+
+To deploy a mesh ingress on Kubernetes cluster, create the AWXMeshIngress resource on the namespace where your AWX instance is running on.
+
+Alternatively, if you wish to create your own Ingress resource, you can deploy a mesh ingress with `ingress_type` set to `none` and then manually create an Ingress resource with any configuration.
+
+In this case, the `external_hostname` is still required as it is used to generate the certificate that will be used by Receptor.
+
+Example:
+
+```yaml
+---
+apiVersion: awx.ansible.com/v1alpha1
+kind: AWXMeshIngress
+metadata:
+  name: <mesh ingress name>
+spec:
+  deployment_name: <awx instance name>
+
+  ingress_type: none  # This line can be omitted since this is the default value
+  external_hostname: <fqdn for mesh ingress>
+```
+
+The requirements for user managed Ingress resource are as follows:
+
+- Supports WebSocket
+- SSL/TLS Passthrough enabled
+- Accessible over port `443`
+- Having the same hostname as `external_hostname` in the AWXMeshIngress resource
+- Routing the traffic to port `27199` of the Service of the same name as the AWXMeshIngress resource
+
+These are example Ingress resources for NGINX and Traefik.
+
+```yaml
+# Ingress for NGINX Ingress Controller
+---
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: <mesh ingress name>
+  annotations:
+    nginx.ingress.kubernetes.io/ssl-passthrough: "true"
+spec:
+  ingressClassName: nginx
+  rules:
+    - host: <fqdn for mesh ingress>
+      http:
+        paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: <mesh ingress name>
+                port:
+                  number: 27199
+```
+
+```yaml
+# Ingress for Traefik Kubernetes Ingress provider
+---
+apiVersion: traefik.io/v1alpha1
+kind: IngressRouteTCP
+metadata:
+  name: <mesh ingress name>
+spec:
+  entryPoints:
+    - websecure
+  routes:
+    - match: HostSNI(`<fqdn for mesh ingress>`)
+      services:
+        - name: <mesh ingress name>
+          port: 27199
+  tls:
+    passthrough: true
+```
+
+## Validating setup of Mesh Ingress
+
+After AWXMeshIngress has been successfully created, a new Instance with the same name will be registered to AWX and will be visible on the Instance UI page
+
+![mesh ingress instance on AWX UI](mesh-ingress-instance-on-awx-ui.png)
+
+The Instance should have at least 2 listener addresses.
+
+In this example, the mesh ingress has two listener addresses:
+
+- one for internal, that is used for peering to by all control nodes (top)
+- one for external, that is exposed to a route so external execution nodes can peer into it (bottom))
+
+![mesh ingress instance listener address on awx ui](mesh-ingress-instance-listener-address-on-awx-ui.png)
+
+When selecting peer for new instance the mesh ingress instance should now be present as a option.
+![peering to mesh ingress on awx ui](peering-to-mesh-ingress-on-awx-ui.png)
+
+For more information about how to create external remote execution and hop nodes and configuring the mesh, see AWX Documentation on [Add a instance](https://ansible.readthedocs.io/projects/awx/en/latest/administration/instances.html#add-an-instance).
+
+## Custom Resource Definitions
+
+### AWXMeshIngress
+
+AWXMeshIngress controls the deployment and configuration of mesh ingress on AWX
+
+| Name                                                                                                                          | Description                                                                                                                                                         |
+| ----------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| **`apiVersion`**                                                                                                              | awx.ansible.com/v1alpha1                                                                                                                                            |
+| **`kind`**                                                                                                                    | AWXMeshIngress                                                                                                                                                      |
+| **`metadata`** ([ObjectMeta](https://kubernetes.io/docs/reference/kubernetes-api/common-definitions/object-meta/#ObjectMeta)) | Standard object's metadata. [More info](https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata)                               |
+| **`spec`** ([AWXMeshIngressSpec](#awxmeshingressspec))                                                                        | Spec is the desired state of the AWXMeshIngress. [More info](https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-status)   |
+| **`status`** ([AWXMeshIngressStatus](#awxmeshingressstatus))                                                                  | Status is the current state of the AWXMeshIngress. [More info](https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-status) |
+
+#### AWXMeshIngressSpec
+
+AWXMeshIngressSpec is the description of the configuration for AWXMeshIngress.
+
+| Name                                     | Description                                                                                                                                                                                                                                 | Default                                        |
+| ---------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------- |
+| **`deployment_name`** (string), required | Name of the AWX deployment to create the Mesh Ingress for.                                                                                                                                                                                  | `awx`                                          |
+| **`ingress_type`** (string)              | Ingress type for ingress managed by the operator. Options: `none`, `Ingress`, `IngressRouteTCP`, `Route`                                                                                                                                    | `Route` (on OpenShift), `none` (on Kubernetes) |
+| **`external_hostname`** (string)         | External hostname is an optional field used for specifying the external hostname defined in an [Ingress](https://kubernetes.io/docs/concepts/services-networking/ingress/). This parameter is automatically generated on OpenShift          | N/A                                            |
+| **`external_ipaddress`** (string)        | External IP Address is an optional field used for specifying the external IP address defined in an [Ingress](https://kubernetes.io/docs/concepts/services-networking/ingress/)                                                              | N/A                                            |
+| **`ingress_api_version`** (string)       | API Version for ingress managed by the operator. This parameter is ignored when `ingress_type` is `Route`                                                                                                                                   | `networking.k8s.io/v1`                         |
+| **`ingress_annotations`** (string)       | Additional annotation on the ingress managed by the operator. This parameter is ignored when `ingress_type` is `Route`                                                                                                                      | `""`                                           |
+| **`ingress_controller`** (string)        | Special configuration for specific Ingress Controllers. This parameter is ignored when `ingress_type` is `Route`                                                                                                                            | `""`                                           |
+| **`ingress_class_name`** (string)        | The name of ingress class to use instead of the cluster default. see [IngressSpec](https://kubernetes.io/docs/reference/kubernetes-api/service-resources/ingress-v1/#IngressSpec). This parameter is ignored when `ingress_type` is `Route` | `""`                                           |
+
+#### AWXMeshIngressStatus
+
+AWXMeshIngressStatus describe the current state of the AWXMeshIngress.
+
+### AWXMeshIngressList
+
+AWXMeshIngressList is a collection of AWXMeshIngress.
+
+| Name                                                                                                                    | Description                                                                                                                                                                                                                                                                                          |
+| ----------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| **`items`** ([AWXMeshIngress](#awxmeshingress))                                                                         | items is the list of Ingress.                                                                                                                                                                                                                                                                        |
+| **`apiVersion`** (string)                                                                                               | APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. [More info](https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources)  |
+| **`kind`** (string)                                                                                                     | Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. [More info](https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds) |
+| **`metadata`** ([ListMeta](https://kubernetes.io/docs/reference/kubernetes-api/common-definitions/list-meta/#ListMeta)) | Standard object's metadata. [More info](https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata)                                                                                                                                                                |
--- a/docs/user-guide/advanced-configuration/peering-to-mesh-ingress-on-awx-ui.png
+++ b/docs/user-guide/advanced-configuration/peering-to-mesh-ingress-on-awx-ui.png
--- a/docs/user-guide/database-configuration.md
+++ b/docs/user-guide/database-configuration.md
@@ -1,10 +1,10 @@
 ### Database Configuration

-#### Postgres Version
+#### PostgreSQL Version

-The default Postgres version for the version of AWX bundled with the latest version of the awx-operator is Postgres 13. You can find this default for a given version by at the default value for [_postgres_image_version](https://github.com/ansible/awx-operator/blob/devel/roles/installer/defaults/main.yml#L243).
+The default PostgreSQL version for the version of AWX bundled with the latest version of the awx-operator is PostgreSQL 15. You can find this default for a given version by at the default value for [supported_pg_version](https://github.com/ansible/awx-operator/blob/ffba1b4712a0b03f1faedfa70e3a9ef0d443e4a6/roles/installer/vars/main.yml#L7).

-We only have coverage for the default version of Postgres. Newer versions of Postgres (14+) will likely work, but should only be configured as an external database. If your database is managed by the awx-operator (default if you don't specify a `postgres_configuration_secret`), then you should not override the default version as this may cause issues when awx-operator tries to upgrade your postgresql pod.
+We only have coverage for the default version of PostgreSQL. Newer versions of PostgreSQL will likely work, but should only be configured as an external database. If your database is managed by the awx-operator (default if you don't specify a `postgres_configuration_secret`), then you should not override the default version as this may cause issues when awx-operator tries to upgrade your postgresql pod.

 #### External PostgreSQL Service

@@ -56,15 +56,15 @@ If you don't have access to an external PostgreSQL service, the AWX operator can

 The following variables are customizable for the managed PostgreSQL service

-| Name                                          | Description                                   | Default                            |
-| --------------------------------------------- | --------------------------------------------- | ---------------------------------- |
-| postgres_image                                | Path of the image to pull                     | postgres:12                        |
-| postgres_init_container_resource_requirements | Database init container resource requirements | requests: {cpu: 10m, memory: 64Mi} |
-| postgres_resource_requirements                | PostgreSQL container resource requirements    | requests: {cpu: 10m, memory: 64Mi} |
-| postgres_storage_requirements                 | PostgreSQL container storage requirements     | requests: {storage: 8Gi}           |
-| postgres_storage_class                        | PostgreSQL PV storage class                   | Empty string                       |
-| postgres_data_path                            | PostgreSQL data path                          | `/var/lib/postgresql/data/pgdata`  |
-| postgres_priority_class                       | Priority class used for PostgreSQL pod        | Empty string                       |
+| Name                                          | Description                                   | Default                                 |
+| --------------------------------------------- | --------------------------------------------- | --------------------------------------- |
+| postgres_image                                | Path of the image to pull                     | quay.io/sclorg/postgresql-15-c9s:latest |
+| postgres_init_container_resource_requirements | Database init container resource requirements | requests: {cpu: 10m, memory: 64Mi}      |
+| postgres_resource_requirements                | PostgreSQL container resource requirements    | requests: {cpu: 10m, memory: 64Mi}      |
+| postgres_storage_requirements                 | PostgreSQL container storage requirements     | requests: {storage: 8Gi}                |
+| postgres_storage_class                        | PostgreSQL PV storage class                   | Empty string                            |
+| postgres_data_path                            | PostgreSQL data path                          | `/var/lib/postgresql/data/pgdata`       |
+| postgres_priority_class                       | Priority class used for PostgreSQL pod        | Empty string                            |

 Example of customization could be:

@@ -90,4 +90,4 @@ spec:
    - 'max_connections=1000'
 ```

-**Note**: If `postgres_storage_class` is not defined, Postgres will store it's data on a volume using the default storage class for your cluster.
+**Note**: If `postgres_storage_class` is not defined, PostgreSQL will store it's data on a volume using the default storage class for your cluster.
--- a/docs/user-guide/network-and-tls-configuration.md
+++ b/docs/user-guide/network-and-tls-configuration.md
@@ -33,6 +33,7 @@ The following variables are customizable only when `service_type=LoadBalancer`
 | loadbalancer_protocol | Protocol to use for Loadbalancer ingress | http    |
 | loadbalancer_port     | Port used for Loadbalancer ingress       | 80      |
 | loadbalancer_ip       | Assign Loadbalancer IP                   | ''      |
+| loadbalancer_class    | LoadBalancer class to use                | ''      |

 ```yaml
 ---
@@ -42,6 +43,7 @@ spec:
  loadbalancer_ip: '192.168.10.25'
  loadbalancer_protocol: https
  loadbalancer_port: 443
+  loadbalancer_class: service.k8s.aws/nlb
  service_annotations: |
    environment: testing
  service_labels: |
--- a/mkdocs.yml
+++ b/mkdocs.yml
@@ -18,6 +18,9 @@ theme:
    - navigation.indexes
    - navigation.tracking
    - toc.integrate
+    - search.highlight
+    - search.share
+    - search.suggest
  palette:
    - media: "(prefers-color-scheme: light)"
      primary: teal
@@ -67,6 +70,7 @@ nav:
      - user-guide/advanced-configuration/scaling-the-web-and-task-pods-independently.md
      - user-guide/advanced-configuration/assigning-awx-pods-to-specific-nodes.md
      - user-guide/advanced-configuration/trusting-a-custom-certificate-authority.md
+      - user-guide/advanced-configuration/custom-receptor-certs.md
      - user-guide/advanced-configuration/enabling-ldap-integration-at-awx-bootstrap.md
      - user-guide/advanced-configuration/persisting-projects-directory.md
      - user-guide/advanced-configuration/custom-volume-and-volume-mount-options.md
@@ -81,6 +85,7 @@ nav:
      - user-guide/advanced-configuration/labeling-operator-managed-objects.md
      - user-guide/advanced-configuration/pods-termination-grace-period.md
      - user-guide/advanced-configuration/disable-ipv6.md
+      - user-guide/advanced-configuration/mesh-ingress.md
  - Troubleshooting:
      - troubleshooting/debugging.md

@@ -90,7 +95,8 @@ exclude_docs:
 plugins:
  - autorefs
  - markdown-exec
-  - search
+  - material/search:
+      separator: '[\s\-,:!=\[\]()"`/]+|\.(?!\d)|&[lg]t;|(?!\b)(?=[A-Z][a-z])'
  - mkdocstrings:
      handlers:
        python:
--- a/roles/backup/tasks/secrets.yml
+++ b/roles/backup/tasks/secrets.yml
@@ -22,7 +22,7 @@
  include_tasks: dump_ingress_tls_secrets.yml
  with_items:
    - "{{ awx_spec.spec['ingress_hosts'] | default('') | map(attribute='tls_secret', default='') | select() | list }}"
-  when: "{{ awx_spec.spec['ingress_hosts'] | default('') | map(attribute='tls_secret', default='') | select() | list | length }}"
+  when: awx_spec.spec['ingress_hosts'] | default('') | map(attribute='tls_secret', default='') | select() | list | length

 - name: Dump receptor secret names and data into file
  include_tasks: dump_receptor_secrets.yml
--- a/roles/backup/vars/main.yml
+++ b/roles/backup/vars/main.yml
@@ -1,8 +1,8 @@
 ---
 deployment_type: "awx"
-_postgres_image: postgres
-_postgres_image_version: 13
+_postgres_image: quay.io/sclorg/postgresql-15-c9s
+_postgres_image_version: latest
 backup_complete: false
 database_type: "unmanaged"
-supported_pg_version: 13
+supported_pg_version: 15
 image_pull_policy: IfNotPresent
--- a/roles/installer/defaults/main.yml
+++ b/roles/installer/defaults/main.yml
@@ -52,6 +52,7 @@ ingress_hosts: ''

 loadbalancer_protocol: 'http'
 loadbalancer_port: '80'
+loadbalancer_class: ''
 service_annotations: ''

 # Port to be used for NodePort configuration, default is to auto-assign a port between 30000-32768
@@ -255,8 +256,8 @@ _image: quay.io/ansible/awx
 _image_version: "{{ lookup('env', 'DEFAULT_AWX_VERSION') or 'latest' }}"
 _redis_image: docker.io/redis
 _redis_image_version: 7
-_postgres_image: postgres
-_postgres_image_version: 13
+_postgres_image: quay.io/sclorg/postgresql-15-c9s
+_postgres_image_version: latest
 image_pull_policy: IfNotPresent
 image_pull_secrets: []

@@ -277,10 +278,12 @@ init_container_extra_commands: ''
 init_container_extra_volume_mounts: ''

 ee_images:
-  - name: AWX EE (latest)
-    image: quay.io/ansible/awx-ee:latest
+  - name: "AWX EE (latest)"
+    image: "quay.io/ansible/awx-ee:latest"
+  - name: "AWX EE ({{ _image_version }})"
+    image: "quay.io/ansible/awx-ee:{{ _image_version }}"

-_control_plane_ee_image: quay.io/ansible/awx-ee:latest
+_control_plane_ee_image: "quay.io/ansible/awx-ee:{{ _image_version }}"

 _init_container_image: "{{ _control_plane_ee_image.split(':')[0] }}"
 _init_container_image_version: "{{ _control_plane_ee_image.split(':')[1] }}"
--- a/roles/installer/tasks/database_configuration.yml
+++ b/roles/installer/tasks/database_configuration.yml
@@ -106,14 +106,38 @@
  set_fact:
    managed_database: "{{ pg_config['resources'][0]['data']['type'] | default('') | b64decode == 'managed' }}"

- name: Get the old postgres pod information
+# It is possible that N-2 postgres pods may still be present in the namespace from previous upgrades.
+# So we have to take that into account and preferentially set the most recent one.
+- name: Get the old postgres pod (N-1)
  k8s_info:
    kind: Pod
    namespace: "{{ ansible_operator_meta.namespace }}"
-    name: "{{ ansible_operator_meta.name }}-postgres-0"
    field_selectors:
      - status.phase=Running
-  register: old_postgres_pod
+  register: _running_pods
+
+- block:
+    - name: Filter pods by name
+      set_fact:
+        filtered_old_postgres_pods: "{{ _running_pods.resources |
+          selectattr('metadata.name', 'match', ansible_operator_meta.name + '-postgres.*-0') |
+          rejectattr('metadata.name', 'search', '-' + supported_pg_version | string + '-0') |
+          list }}"
+
+  # Sort pods by name in reverse order (most recent PG version first) and set
+    - name: Set info for previous postgres pod
+      set_fact:
+        sorted_old_postgres_pods: "{{ filtered_old_postgres_pods |
+          sort(attribute='metadata.name') |
+          reverse }}"
+      when: filtered_old_postgres_pods | length
+
+
+    - name: Set info for previous postgres pod
+      set_fact:
+        old_postgres_pod: "{{ sorted_old_postgres_pods | first }}"
+      when: filtered_old_postgres_pods | length
+  when: _running_pods.resources | length

 - name: Look up details for this deployment
  k8s_info:
@@ -123,7 +147,14 @@
    namespace: "{{ ansible_operator_meta.namespace }}"
  register: this_awx

- name: Check if postgres pod is running and version 12
+# If this deployment has been upgraded before or if upgrade has already been started, set this var
+- name: Set previous PG version var
+  set_fact:
+    _previous_upgraded_pg_version: "{{ this_awx['resources'][0]['status']['upgradedPostgresVersion'] | default(false) }}"
+  when:
+    - "'upgradedPostgresVersion' in this_awx['resources'][0]['status']"
+
+- name: Check if postgres pod is running an older version
  block:
    - name: Set path to PG_VERSION file for given container image
      set_fact:
@@ -132,21 +163,24 @@
    - name: Get old PostgreSQL version
      k8s_exec:
        namespace: "{{ ansible_operator_meta.namespace }}"
-        pod: "{{ ansible_operator_meta.name }}-postgres-0"
+        pod: "{{ old_postgres_pod['metadata']['name'] }}"
        command: |
          bash -c """
          cat {{ path_to_pg_version }}
          """
      register: _old_pg_version

-    - name: Upgrade data dir from Postgres 12 to 13 if applicable
+    - debug:
+        msg: "--- Upgrading from {{ old_postgres_pod['metadata']['name'] | default('NONE')}} Pod ---"
+
+    - name: Upgrade data dir from old Postgres to {{ supported_pg_version }} if applicable
      include_tasks: upgrade_postgres.yml
      when:
-        - _old_pg_version.stdout | default('0') | trim == '12'
+        - (_old_pg_version.stdout | default(0) | int ) < supported_pg_version
  when:
    - managed_database
-    - this_awx['resources'][0]['status']['upgradedPostgresVersion'] | default('none') != '12'
-    - old_postgres_pod['resources'] | length  # upgrade is complete and old pg pod has been removed
+    - (_previous_upgraded_pg_version | default(false)) | ternary(_previous_upgraded_pg_version | int < supported_pg_version, true)
+    - old_postgres_pod | length  # If empty, then old pg pod has been removed and we can assume the upgrade is complete

 - block:
    - name: Create Database if no database is specified
@@ -167,7 +201,7 @@
      kubernetes.core.k8s_scale:
        api_version: apps/v1
        kind: StatefulSet
-        name: "{{ ansible_operator_meta.name }}-postgres-13"
+        name: "{{ ansible_operator_meta.name }}-postgres-{{ supported_pg_version }}"
        namespace: "{{ ansible_operator_meta.namespace }}"
        replicas: 0
        wait: yes
@@ -177,7 +211,7 @@
        state: absent
        api_version: apps/v1
        kind: StatefulSet
-        name: "{{ ansible_operator_meta.name }}-postgres-13"
+        name: "{{ ansible_operator_meta.name }}-postgres-{{ supported_pg_version }}"
        namespace: "{{ ansible_operator_meta.namespace }}"
        wait: yes
      when: create_statefulset_result.error == 422
--- a/roles/installer/tasks/initialize_django.yml
+++ b/roles/installer/tasks/initialize_django.yml
@@ -2,8 +2,8 @@
 - name: Check if there are any super users defined.
  k8s_exec:
    namespace: "{{ ansible_operator_meta.namespace }}"
-    pod: "{{ awx_task_pod_name }}"
-    container: "{{ ansible_operator_meta.name }}-task"
+    pod: "{{ awx_web_pod_name }}"
+    container: "{{ ansible_operator_meta.name }}-web"
    command: >-
      bash -c "echo 'from django.contrib.auth.models import User;
      nsu = User.objects.filter(is_superuser=True, username=\"{{ admin_user }}\").count();
@@ -16,8 +16,8 @@
 - name: Create super user via Django if it doesn't exist.
  k8s_exec:
    namespace: "{{ ansible_operator_meta.namespace }}"
-    pod: "{{ awx_task_pod_name }}"
-    container: "{{ ansible_operator_meta.name }}-task"
+    pod: "{{ awx_web_pod_name }}"
+    container: "{{ ansible_operator_meta.name }}-web"
    command: awx-manage createsuperuser --username={{ admin_user | quote }} --email={{ admin_email | quote }} --noinput
  register: result
  changed_when: "'That username is already taken' not in result.stderr"
@@ -28,8 +28,8 @@
 - name: Update Django super user password
  k8s_exec:
    namespace: "{{ ansible_operator_meta.namespace }}"
-    pod: "{{ awx_task_pod_name }}"
-    container: "{{ ansible_operator_meta.name }}-task"
+    pod: "{{ awx_web_pod_name }}"
+    container: "{{ ansible_operator_meta.name }}-web"
    command: awx-manage update_password --username='{{ admin_user }}' --password='{{ admin_password }}'
  register: result
  changed_when: "'Password updated' in result.stdout"
@@ -39,8 +39,8 @@
 - name: Check if legacy queue is present
  k8s_exec:
    namespace: "{{ ansible_operator_meta.namespace }}"
-    pod: "{{ awx_task_pod_name }}"
-    container: "{{ ansible_operator_meta.name }}-task"
+    pod: "{{ awx_web_pod_name }}"
+    container: "{{ ansible_operator_meta.name }}-web"
    command: >-
      bash -c "awx-manage list_instances | grep '^\[tower capacity=[0-9]*\]'"
  register: legacy_queue
@@ -50,8 +50,8 @@
 - name: Unregister legacy queue
  k8s_exec:
    namespace: "{{ ansible_operator_meta.namespace }}"
-    pod: "{{ awx_task_pod_name }}"
-    container: "{{ ansible_operator_meta.name }}-task"
+    pod: "{{ awx_web_pod_name }}"
+    container: "{{ ansible_operator_meta.name }}-web"
    command: >-
      bash -c "awx-manage unregister_queue --queuename=tower"
  when: "'[tower capacity=' in legacy_queue.stdout"
@@ -74,8 +74,8 @@
 - name: Register default execution environments (without authentication)
  k8s_exec:
    namespace: "{{ ansible_operator_meta.namespace }}"
-    pod: "{{ awx_task_pod_name }}"
-    container: "{{ ansible_operator_meta.name }}-task"
+    pod: "{{ awx_web_pod_name }}"
+    container: "{{ ansible_operator_meta.name }}-web"
    command: >-
      bash -c "awx-manage register_default_execution_environments"
  register: ree
@@ -95,8 +95,8 @@
    - name: Register default execution environments (with authentication)
      k8s_exec:
        namespace: "{{ ansible_operator_meta.namespace }}"
-        pod: "{{ awx_task_pod_name }}"
-        container: "{{ ansible_operator_meta.name }}-task"
+        pod: "{{ awx_web_pod_name }}"
+        container: "{{ ansible_operator_meta.name }}-web"
        command: >-
          bash -c "awx-manage register_default_execution_environments
          --registry-username='{{ default_execution_environment_pull_credentials_user }}'
@@ -111,8 +111,8 @@
 - name: Create preload data if necessary.  # noqa 305
  k8s_exec:
    namespace: "{{ ansible_operator_meta.namespace }}"
-    pod: "{{ awx_task_pod_name }}"
-    container: "{{ ansible_operator_meta.name }}-task"
+    pod: "{{ awx_web_pod_name }}"
+    container: "{{ ansible_operator_meta.name }}-web"
    command: >-
      bash -c "awx-manage create_preload_data"
  register: cdo
--- a/roles/installer/tasks/install.yml
+++ b/roles/installer/tasks/install.yml
@@ -94,51 +94,13 @@
 - name: Include resources configuration tasks
  include_tasks: resources_configuration.yml

- name: Check for pending migrations
-  k8s_exec:
-    namespace: "{{ ansible_operator_meta.namespace }}"
-    pod: "{{ awx_task_pod_name }}"
-    container: "{{ ansible_operator_meta.name }}-task"
-    command: >-
-      bash -c "awx-manage showmigrations | grep -v '[X]' | grep '[ ]' | wc -l"
-  changed_when: false
-  when: awx_task_pod_name != ''
-  register: database_check
-
- name: Migrate the database if the K8s resources were updated  # noqa 305
-  k8s_exec:
-    namespace: "{{ ansible_operator_meta.namespace }}"
-    pod: "{{ awx_task_pod_name }}"
-    container: "{{ ansible_operator_meta.name }}-task"
-    command: |
-      bash -c "
-      function end_keepalive {
-        rc=$?
-        rm -f \"$1\"
-        kill $(cat /proc/$2/task/$2/children 2>/dev/null) 2>/dev/null || true
-        wait $2 || true
-        exit $rc
-      }
-      keepalive_file=\"$(mktemp)\"
-      while [[ -f \"$keepalive_file\" ]]; do
-        echo 'Database schema migration in progress...'
-        sleep 60
-      done &
-      keepalive_pid=$!
-      trap 'end_keepalive \"$keepalive_file\" \"$keepalive_pid\"' EXIT SIGINT SIGTERM
-      echo keepalive_pid: $keepalive_pid
-      awx-manage migrate --noinput
-      echo 'Successful'
-      "
-  register: migrate_result
-  when:
-  - awx_task_pod_name != ''
-  - database_check is defined
-  - (database_check.stdout|trim) != '0'
+- name: Migrate database to the latest schema
+  include_tasks: migrate_schema.yml
+  when: awx_web_pod_name != ''

 - name: Initialize Django
  include_tasks: initialize_django.yml
-  when: awx_task_pod_name != ''
+  when: awx_web_pod_name != ''

 - name: Update status variables
  include_tasks: update_status.yml
--- a/roles/installer/tasks/migrate_schema.yml
+++ b/roles/installer/tasks/migrate_schema.yml
@@ -0,0 +1,57 @@
+---
+
+- name: Check for pending migrations
+  k8s_exec:
+    namespace: "{{ ansible_operator_meta.namespace }}"
+    pod: "{{ awx_web_pod_name }}"
+    container: "{{ ansible_operator_meta.name }}-web"
+    command: >-
+      bash -c "awx-manage showmigrations | grep -v '[X]' | grep '[ ]' | wc -l"
+  changed_when: false
+  when: awx_web_pod_name != ''
+  register: database_check
+
+- block:
+    - name: Get version of controller for tracking
+      k8s_exec:
+        namespace: "{{ ansible_operator_meta.namespace }}"
+        pod: "{{ awx_web_pod_name }}"
+        container: "{{ ansible_operator_meta.name }}-web"
+        command: >-
+          bash -c "awx-manage --version"
+      changed_when: false
+      register: version_check
+
+    - name: Sanitize instance version
+      set_fact:
+        version: "{{ version_check.stdout.split('+')[0] | trim }}"
+
+    # It is possible to do a wait on this task to create the job and wait
+    # until it completes. Unfortunately, if the job doesn't wait finish within
+    # the timeout period that is considered an error.  We only want this to
+    # error if there is an issue with creating the job.
+    - name: Create kubernetes job to perform the migration
+      k8s:
+        apply: yes
+        definition: "{{ lookup('template', 'jobs/migration.yaml.j2') }}"
+      register: migrate_result
+
+    # This task is really only necessary for new installations. We need to
+    # ensure the database has a schema loaded before continuing with the
+    # initialization of admin user, etc.
+    - name: Watch for the migration job to finish
+      k8s_info:
+        kind: Job
+        namespace: "{{ ansible_operator_meta.namespace }}"
+        name: "{{ ansible_operator_meta.name }}-migration-{{ version }}"
+      register: result
+      until:
+        - result.resources[0].status.succeeded is defined
+        - result.resources[0].status.succeeded == 1
+      retries: 180
+      delay: 5
+      ignore_errors: true
+
+  when:
+    - database_check is defined
+    - (database_check.stdout|trim) != '0'
--- a/roles/installer/tasks/resources_configuration.yml
+++ b/roles/installer/tasks/resources_configuration.yml
@@ -1,28 +1,28 @@
 ---
- name: Get the current resource task pod information.
+- name: Get the current resource web pod information.
  k8s_info:
    api_version: v1
    kind: Pod
    namespace: '{{ ansible_operator_meta.namespace }}'
    label_selectors:
-      - "app.kubernetes.io/name={{ ansible_operator_meta.name }}-task"
+      - "app.kubernetes.io/name={{ ansible_operator_meta.name }}-web"
      - "app.kubernetes.io/managed-by={{ deployment_type }}-operator"
      - "app.kubernetes.io/component={{ deployment_type }}"
    field_selectors:
      - status.phase=Running
-  register: awx_task_pod
+  register: awx_web_pod

 - name: Set the resource pod as a variable.
  set_fact:
-    awx_task_pod: >-
-      {{ awx_task_pod['resources']
+    awx_web_pod: >-
+      {{ awx_web_pod['resources']
         | rejectattr('metadata.deletionTimestamp', 'defined')
         | sort(attribute='metadata.creationTimestamp')
         | first | default({}) }}

 - name: Set the resource pod name as a variable.
  set_fact:
-    awx_task_pod_name: "{{ awx_task_pod['metadata']['name'] | default('') }}"
+    awx_web_pod_name: "{{ awx_web_pod['metadata']['name'] | default('') }}"

 - name: Set user provided control plane ee image
  set_fact:
@@ -249,11 +249,9 @@
  k8s:
    apply: yes
    definition: "{{ lookup('template', 'deployments/{{ item }}.yaml.j2') }}"
-    wait: yes
-    wait_timeout: "{{ (120 * replicas) or 120 }}"
  loop:
-    - task
    - web
+    - task
  register: this_deployment_result

 - block:
@@ -262,7 +260,7 @@
        kind: Pod
        namespace: '{{ ansible_operator_meta.namespace }}'
        label_selectors:
-          - "app.kubernetes.io/name={{ ansible_operator_meta.name }}-task"
+          - "app.kubernetes.io/name={{ ansible_operator_meta.name }}-web"
          - "app.kubernetes.io/managed-by={{ deployment_type }}-operator"
          - "app.kubernetes.io/component={{ deployment_type }}"
        field_selectors:
@@ -271,7 +269,7 @@

    - name: Update new resource pod as a variable.
      set_fact:
-        awx_task_pod: >-
+        awx_web_pod: >-
          {{ _new_pod['resources']
             | rejectattr('metadata.deletionTimestamp', 'defined')
             | sort(attribute='metadata.creationTimestamp')
@@ -279,13 +277,13 @@

    - name: Update new resource pod name as a variable.
      set_fact:
-        awx_task_pod_name: '{{ awx_task_pod["metadata"]["name"] | default("")}}'
+        awx_web_pod_name: '{{ awx_web_pod["metadata"]["name"] | default("")}}'
  when:
    - this_deployment_result.changed

 - name: Verify the resource pod name is populated.
  assert:
    that:
-      - awx_task_pod_name != ''
+      - awx_web_pod_name != ''
    fail_msg: "Could not find the tower pod's name."
-  when: task_replicas | int > 0 or (task_replicas == '' and replicas > 0)
+  when: web_replicas | int > 0 or (web_replicas == '' and replicas > 0)
--- a/roles/installer/tasks/set_images.yml
+++ b/roles/installer/tasks/set_images.yml
@@ -8,8 +8,8 @@
  set_fact:
    _custom_init_container_image: "{{ init_container_image }}:{{ init_container_image_version }}"
  when:
-    - init_container_image | default([]) | length
-    - init_container_image_version is defined or init_container_image_version != ''
+    - init_container_image | default('_undefined',true) != '_undefined'
+    - init_container_image_version | default('_undefined',true) != '_undefined'

 - name: Set Init image URL
  set_fact:
--- a/roles/installer/tasks/update_status.yml
+++ b/roles/installer/tasks/update_status.yml
@@ -47,13 +47,13 @@
 - name: Retrieve instance version
  k8s_exec:
    namespace: "{{ ansible_operator_meta.namespace }}"
-    pod: "{{ awx_task_pod_name }}"
-    container: "{{ ansible_operator_meta.name }}-task"
+    pod: "{{ awx_web_pod_name }}"
+    container: "{{ ansible_operator_meta.name }}-web"
    command: >-
      bash -c "awx-manage --version"
  register: instance_version
  changed_when: false
-  when: awx_task_pod_name != ''
+  when: awx_web_pod_name != ''

 - name: Update version status
  operator_sdk.util.k8s_status:
@@ -111,5 +111,5 @@
    name: "{{ ansible_operator_meta.name }}"
    namespace: "{{ ansible_operator_meta.namespace }}"
    status:
-      upgradedPostgresVersion: "{{ upgraded_postgres_version }}"
+      upgradedPostgresVersion: "{{ upgraded_postgres_version | string }}"
  when: upgraded_postgres_version is defined
--- a/roles/installer/tasks/upgrade_postgres.yml
+++ b/roles/installer/tasks/upgrade_postgres.yml
@@ -1,9 +1,9 @@
 ---

 # Upgrade Posgres (Managed Databases only)
-#  * If postgres version is not 12, and not an external postgres instance (when managed_database is yes),
+#  * If postgres version is not supported_pg_version, and not an external postgres instance (when managed_database is yes),
 #    then run this playbook with include_tasks from database_configuration.yml
-#  * Data will be streamed via a pg_dump from the postgres 12 pod to the postgres 13
+#  * Data will be streamed via a pg_dump from the postgres 12/13 pod to the postgres supported_pg_version
 #    pod via a pg_restore.


@@ -62,9 +62,19 @@
  set_fact:
    postgres_pod_name: "{{ postgres_pod['resources'][0]['metadata']['name'] }}"

+- name: Get the name of the service for the old postgres pod
+  k8s_info:
+    kind: Service
+    namespace: "{{ ansible_operator_meta.namespace }}"
+    label_selectors:
+      - "app.kubernetes.io/component=database"
+      - "app.kubernetes.io/instance={{ old_postgres_pod.metadata.labels['app.kubernetes.io/instance'] }}"
+      - "app.kubernetes.io/managed-by={{ deployment_type }}-operator"
+  register: old_postgres_svc
+
 - name: Set full resolvable host name for postgres pod
  set_fact:
-    resolvable_db_host: "{{ ansible_operator_meta.name }}-postgres.{{ ansible_operator_meta.namespace }}.svc"  # yamllint disable-line rule:line-length
+    resolvable_db_host: "{{ old_postgres_svc['resources'][0]['metadata']['name'] }}.{{ ansible_operator_meta.namespace }}.svc"  # yamllint disable-line rule:line-length
  no_log: "{{ no_log }}"

 - name: Set pg_dump command
@@ -118,31 +128,41 @@

 - name: Set flag signifying that this instance has been migrated
  set_fact:
-    upgraded_postgres_version: '13'
+    upgraded_postgres_version: '{{ supported_pg_version }}'

-# Cleanup old Postgres resources
- name: Remove old Postgres StatefulSet
+# Cleanup old PostgreSQL resources
+- name: Remove old PostgreSQL StatefulSet
  k8s:
    kind: StatefulSet
    api_version: v1
    namespace: "{{ ansible_operator_meta.namespace }}"
-    name: "{{ ansible_operator_meta.name }}-postgres"
+    name: "{{ item }}"
    state: absent
    wait: true
+  loop:
+    - "{{ ansible_operator_meta.name }}-postgres"
+    - "{{ ansible_operator_meta.name }}-postgres-13"

- name: Remove old Postgres Service
+- name: Remove old PostgreSQL Service
  k8s:
    kind: Service
    api_version: v1
    namespace: "{{ ansible_operator_meta.namespace }}"
-    name: "{{ ansible_operator_meta.name }}-postgres"
+    name: "{{ item }}"
    state: absent
+  loop:
+    - "{{ ansible_operator_meta.name }}-postgres"
+    - "{{ ansible_operator_meta.name }}-postgres-13"

 - name: Remove old persistent volume claim
  k8s:
    kind: PersistentVolumeClaim
    api_version: v1
    namespace: "{{ ansible_operator_meta.namespace }}"
-    name: "postgres-{{ ansible_operator_meta.name }}-postgres-0"
+    name: "{{ item }}"
    state: absent
+  loop:
+    - "postgres-{{ ansible_operator_meta.name }}-postgres-0"
+    - "postgres-{{ ansible_operator_meta.name }}-postgres-13-0"
+    - "postgres-13-{{ ansible_operator_meta.name }}-postgres-13-0"
  when: postgres_keep_pvc_after_upgrade
--- a/roles/installer/templates/configmaps/config.yaml.j2
+++ b/roles/installer/templates/configmaps/config.yaml.j2
@@ -188,7 +188,7 @@ data:
                alias /var/lib/awx/public/static/media/favicon.ico;
            }

-            location ~ ({{ (ingress_path + '/websocket').replace('//', '/') }}|{{ (ingress_path + '/api/websocket').replace('//', '/') }}) {
+            location ~ ^({{ (ingress_path + '/websocket/').replace('//', '/') }}|{{ (ingress_path + '/api/websocket/').replace('//', '/') }}) {
                # Pass request to the upstream alias
                proxy_pass http://daphne;
                # Require http version 1.1 to allow for upgrade requests
--- a/roles/installer/templates/deployments/task.yaml.j2
+++ b/roles/installer/templates/deployments/task.yaml.j2
@@ -74,7 +74,32 @@ spec:
      priorityClassName: '{{ control_plane_priority_class }}'
 {% endif %}
      initContainers:
-        - name: init
+        - name: init-database
+          image: '{{ _image }}'
+          imagePullPolicy: '{{ image_pull_policy }}'
+          resources: {{ init_container_resource_requirements }}
+          command:
+            - /bin/sh
+            - -c
+            - wait-for-migrations
+          volumeMounts:
+            - name: {{ ansible_operator_meta.name }}-application-credentials
+              mountPath: "/etc/tower/conf.d/credentials.py"
+              subPath: credentials.py
+              readOnly: true
+            - name: "{{ secret_key_secret_name }}"
+              mountPath: /etc/tower/SECRET_KEY
+              subPath: SECRET_KEY
+              readOnly: true
+            - name: {{ ansible_operator_meta.name }}-settings
+              mountPath: "/etc/tower/settings.py"
+              subPath: settings.py
+              readOnly: true
+{% if development_mode | bool %}
+            - name: awx-devel
+              mountPath: "/awx_devel"
+{% endif %}
+        - name: init-receptor
          image: '{{ _init_container_image }}'
          imagePullPolicy: '{{ image_pull_policy }}'
          resources: {{ init_container_resource_requirements }}
@@ -83,8 +108,20 @@ spec:
            - -c
            - |
              hostname=$MY_POD_NAME
-              receptor --cert-makereq bits=2048 commonname=$hostname dnsname=$hostname nodeid=$hostname outreq=/etc/receptor/tls/receptor.req outkey=/etc/receptor/tls/receptor.key
-              receptor --cert-signreq req=/etc/receptor/tls/receptor.req cacert=/etc/receptor/tls/ca/mesh-CA.crt cakey=/etc/receptor/tls/ca/mesh-CA.key outcert=/etc/receptor/tls/receptor.crt verify=yes
+              receptor --cert-makereq \
+                bits=2048 \
+                commonname=$hostname \
+                dnsname=$hostname \
+                nodeid=$hostname \
+                outreq=/etc/receptor/tls/receptor.req \
+                outkey=/etc/receptor/tls/receptor.key
+              receptor --cert-signreq \
+                req=/etc/receptor/tls/receptor.req \
+                cacert=/etc/receptor/tls/ca/mesh-CA.crt \
+                cakey=/etc/receptor/tls/ca/mesh-CA.key \
+                outcert=/etc/receptor/tls/receptor.crt \
+                notafter=$(date --iso-8601=seconds --utc --date "10 years") \
+                verify=yes
 {% if bundle_ca_crt  %}
              mkdir -p /etc/pki/ca-trust/extracted/{java,pem,openssl,edk2}
              update-ca-trust
@@ -188,6 +225,30 @@ spec:
 {% endif %}
 {% if task_args %}
          args: {{ task_args }}
+{% endif %}
+{% if task_liveness_period|int > 0 %}
+          livenessProbe:
+            exec:
+              command:
+                - sh
+                - -c
+                - |
+                  (exit $(/usr/bin/supervisorctl -c /etc/supervisord_task.conf status | grep -vc RUNNING))
+            initialDelaySeconds: {{ task_liveness_initial_delay }}
+            periodSeconds: {{ task_liveness_period }}
+            failureThreshold: {{ task_liveness_failure_threshold }}
+            timeoutSeconds: {{ task_liveness_timeout }}
+{% endif %}
+{% if task_readiness_period|int > 0 %}
+          readinessProbe:
+            exec:
+              command:
+                - /usr/bin/awx-manage
+                - check
+            initialDelaySeconds: {{ task_readiness_initial_delay }}
+            periodSeconds: {{ task_readiness_period }}
+            failureThreshold: {{ task_readiness_failure_threshold }}
+            timeoutSeconds: {{ task_readiness_timeout }}
 {% endif %}
          volumeMounts:
 {% if bundle_ca_crt %}
--- a/roles/installer/templates/deployments/web.yaml.j2
+++ b/roles/installer/templates/deployments/web.yaml.j2
@@ -162,6 +162,31 @@ spec:
            - containerPort: 8052
 {% if ingress_type | lower == 'route' and route_tls_termination_mechanism | lower == 'passthrough' %}
            - containerPort: 8053
+{% endif %}
+{% if web_liveness_period|int > 0 %}
+          livenessProbe:
+            exec:
+              command:
+                - sh
+                - -c
+                - |
+                  (exit $(/usr/bin/supervisorctl -c /etc/supervisord_task.conf status | grep -vc RUNNING))
+            initialDelaySeconds: {{ web_liveness_initial_delay }}
+            periodSeconds: {{ web_liveness_period }}
+            failureThreshold: {{ web_liveness_failure_threshold }}
+            timeoutSeconds: {{ web_liveness_timeout }}
+{% endif %}
+{% if web_readiness_period|int > 0 %}
+          readinessProbe:
+            exec:
+              httpGet:
+                path: /api/v2/ping/
+                scheme: HTTP
+                port: 8052
+            initialDelaySeconds: {{ web_readiness_initial_delay }}
+            periodSeconds: {{ web_readiness_period }}
+            failureThreshold: {{ web_readiness_failure_threshold }}
+            timeoutSeconds: {{ web_readiness_timeout }}
 {% endif %}
          volumeMounts:
 {% if bundle_ca_crt %}
--- a/roles/installer/templates/jobs/migration.yaml.j2
+++ b/roles/installer/templates/jobs/migration.yaml.j2
@@ -0,0 +1,66 @@
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: '{{ ansible_operator_meta.name }}-migration-{{ version }}'
+  namespace: '{{ ansible_operator_meta.namespace }}'
+  labels:
+    {{ lookup("template", "../common/templates/labels/common.yaml.j2")  | indent(width=4) | trim }}
+    {{ lookup("template", "../common/templates/labels/version.yaml.j2") | indent(width=4) | trim }}
+spec:
+  template:
+    spec:
+      containers:
+        - name: "migration-job"
+          image: '{{ _image }}'
+          command:
+            - awx-manage
+            - migrate
+            - --noinput
+          volumeMounts:
+            - name: {{ ansible_operator_meta.name }}-application-credentials
+              mountPath: "/etc/tower/conf.d/credentials.py"
+              subPath: credentials.py
+              readOnly: true
+            - name: "{{ secret_key_secret_name }}"
+              mountPath: /etc/tower/SECRET_KEY
+              subPath: SECRET_KEY
+              readOnly: true
+            - name: {{ ansible_operator_meta.name }}-settings
+              mountPath: "/etc/tower/settings.py"
+              subPath: settings.py
+              readOnly: true
+{% if development_mode | bool %}
+            - name: awx-devel
+              mountPath: "/awx_devel"
+{% endif %}
+      volumes:
+        - name: "{{ ansible_operator_meta.name }}-application-credentials"
+          secret:
+            secretName: "{{ ansible_operator_meta.name }}-app-credentials"
+            items:
+              - key: credentials.py
+                path: 'credentials.py'
+              - key: ldap.py
+                path: 'ldap.py'
+              - key: execution_environments.py
+                path: 'execution_environments.py'
+        - name: "{{ secret_key_secret_name }}"
+          secret:
+            secretName: '{{ secret_key_secret_name }}'
+            items:
+              - key: secret_key
+                path: SECRET_KEY
+        - name: {{ ansible_operator_meta.name }}-settings
+          configMap:
+            name: '{{ ansible_operator_meta.name }}-{{ deployment_type }}-configmap'
+            items:
+              - key: settings
+                path: settings.py
+{% if development_mode | bool %}
+        - name: awx-devel
+          hostPath:
+            path: /awx_devel
+{% endif %}
+      dnsPolicy: ClusterFirst
+      restartPolicy: Never
+      terminationGracePeriodSeconds: 30
--- a/roles/installer/templates/networking/ingress.yaml.j2
+++ b/roles/installer/templates/networking/ingress.yaml.j2
@@ -13,7 +13,7 @@ metadata:
  annotations:
 {% if ingress_annotations %}
    {{ ingress_annotations | indent(width=4) }}
-{%- endif %}
+{% endif %}
 {% if ingress_controller|lower == "contour" %}
    projectcontour.io/websocket-routes: "/websocket"
    kubernetes.io/ingress.class: contour
--- a/roles/installer/templates/networking/service.yaml.j2
+++ b/roles/installer/templates/networking/service.yaml.j2
@@ -55,6 +55,9 @@ spec:
 {% if loadbalancer_ip is defined and loadbalancer_ip | length %}
  loadbalancerip: '{{ loadbalancer_ip }}'
 {% endif %}
+{% if loadbalancer_class is defined and loadbalancer_class | length %}
+  loadBalancerClass: {{ loadbalancer_class }}
+{% endif %}
 {% else %}
  type: ClusterIP
 {% endif %}
--- a/roles/installer/templates/statefulsets/postgres.yaml.j2
+++ b/roles/installer/templates/statefulsets/postgres.yaml.j2
@@ -59,7 +59,7 @@ spec:
          args: {{ postgres_extra_args }}
 {% endif %}
          env:
-            # For postgres_image based on rhel8/postgresql-13
+            # For postgres_image based on rhel8/postgresql-{{ supported_pg_version }}
            - name: POSTGRESQL_DATABASE
              valueFrom:
                secretKeyRef:
--- a/roles/installer/vars/main.yml
+++ b/roles/installer/vars/main.yml
@@ -4,4 +4,6 @@ postgres_host_auth_method: 'scram-sha-256'
 ldap_cacert_ca_crt: ''
 bundle_ca_crt: ''
 projects_existing_claim: ''
-supported_pg_version: 13
+supported_pg_version: 15
+_previous_upgraded_pg_version: 0
+old_postgres_pod: []
--- a/roles/mesh_ingress/defaults/main.yml
+++ b/roles/mesh_ingress/defaults/main.yml
@@ -9,7 +9,8 @@ ingress_controller: ''

 set_self_owneref: true

-_control_plane_ee_image: quay.io/ansible/awx-ee:latest
+_control_plane_ee_image: "quay.io/ansible/awx-ee:{{ lookup('env', 'DEFAULT_AWX_VERSION') or 'latest' }}"
 _image_pull_policy: Always
+image_pull_secrets: []

 finalizer_run: false
--- a/roles/mesh_ingress/tasks/creation.yml
+++ b/roles/mesh_ingress/tasks/creation.yml
@@ -61,7 +61,7 @@
 - name: Default ingress_type to Route if OpenShift
  set_fact:
    ingress_type: route
-  when: is_openshift | bool and ingress_type == 'none'
+  when: is_openshift | bool and ingress_type | lower == 'none'

 - name: Apply Ingress resource
  k8s:
@@ -77,7 +77,7 @@
 - name: Set external_hostname
  set_fact:
    external_hostname: "{{ ingress.result.status.ingress[0].host }}"
-  when: ingress_type == 'route'
+  when: ingress_type | lower == 'route'

 - name: Create other resources
  k8s:
--- a/roles/mesh_ingress/templates/deployment.yml.j2
+++ b/roles/mesh_ingress/templates/deployment.yml.j2
@@ -12,6 +12,12 @@ spec:
      labels:
        app.kubernetes.io/name: {{ ansible_operator_meta.name }}
    spec:
+{% if image_pull_secrets | length > 0 %}
+      imagePullSecrets:
+{% for secret in image_pull_secrets %}
+        - name: {{ secret }}
+{% endfor %}
+{% endif %}
      containers:
      - args:
        - /bin/sh
@@ -24,7 +30,8 @@ spec:
 {% if external_ipaddress is defined %}
          external_ipaddress={{ external_ipaddress }}
 {% endif %}
-          receptor --cert-makereq bits=2048 \
+          receptor --cert-makereq \
+            bits=2048 \
            commonname=$internal_hostname \
            dnsname=$internal_hostname \
            nodeid=$internal_hostname \
@@ -41,6 +48,7 @@ spec:
            cacert=/etc/receptor/tls/ca/mesh-CA.crt \
            cakey=/etc/receptor/tls/ca/mesh-CA.key \
            outcert=/etc/receptor/tls/receptor.crt \
+            notafter=$(date --iso-8601=seconds --utc --date "10 years") \
            verify=yes
          exec receptor --config /etc/receptor/receptor.conf
        image: '{{ _control_plane_ee_image }}'
--- a/roles/mesh_ingress/templates/ingress.yml.j2
+++ b/roles/mesh_ingress/templates/ingress.yml.j2
@@ -12,7 +12,7 @@ metadata:
    {{ ingress_annotations | indent(width=4) }}
 {% endif %}
 {% if ingress_controller|lower == "nginx" %}
-      nginx.ingress.kubernetes.io/ssl-passthrough: "true"
+    nginx.ingress.kubernetes.io/ssl-passthrough: "true"
 {% endif %}
 spec:
 {% if ingress_class_name %}
@@ -28,7 +28,7 @@ spec:
                name: {{ ansible_operator_meta.name }}
                port:
                  number: 27199
-{% if external_hostname %}
+{% if external_hostname is defined %}
      host: {{ external_hostname }}
 {% endif %}
 {% endif %}
@@ -53,7 +53,7 @@ spec:
    - services:
      - name: {{ ansible_operator_meta.name }}
        port: 27199
-{% if external_hostname %}
+{% if external_hostname is defined %}
      match: HostSNI(`{{ external_hostname }}`)
 {% endif %}
  tls:
@@ -70,6 +70,9 @@ metadata:
  name: {{ ansible_operator_meta.name }}
  namespace: {{ ansible_operator_meta.namespace }}
 spec:
+{% if external_hostname %}
+  host: {{ external_hostname }}
+{% endif %}
  port:
    targetPort: ws
  tls:
--- a/roles/restore/README.md
+++ b/roles/restore/README.md
@@ -19,7 +19,7 @@ This role assumes you are authenticated with an Openshift or Kubernetes cluster:

 *Before Restoring from a backup*, be sure to:
  - delete the old existing AWX CR
-  - delete the persistent volume claim (PVC) for the database from the old deployment, which has a name like `postgres-13-<deployment-name>-postgres-13-0`
+  - delete the persistent volume claim (PVC) for the database from the old deployment, which has a name like `postgres-<postgres version>-<deployment-name>-postgres-<postgres version>-0`

 **Note**: Do not delete the namespace/project, as that will delete the backup and the backup's PVC as well.

--- a/roles/restore/tasks/secrets.yml
+++ b/roles/restore/tasks/secrets.yml
@@ -19,6 +19,18 @@
  include_vars: "{{ tmp_secrets.path }}"
  no_log: "{{ no_log }}"

+- name: If deployment is managed, set the new postgres_configuration_secret name
+  block:
+  - name: Set new postgres_configuration_secret name
+    set_fact:
+      _generated_pg_secret_name: "{{ deployment_name }}-postgres-configuration"
+
+  - name: Override postgres_configuration_secret
+    set_fact:
+      spec:
+        "{{ spec | combine({'postgres_configuration_secret': _generated_pg_secret_name}, recursive=True) }}"
+  when: secrets['postgresConfigurationSecret']['data']['type'] | b64decode == 'managed'
+
 - name: If deployment is managed, set the database_host in the pg config secret
  block:
    - name: Set new database host
@@ -31,12 +43,18 @@
        _pg_secret: "{{ secrets['postgresConfigurationSecret'] }}"
      no_log: "{{ no_log }}"

-    - name: Change postgres host value
+    - name: Change postgres host and name value
      set_fact:
        _pg_data: "{{ _pg_secret['data'] | combine({'host': database_host | b64encode }) }}"
+        _pg_secret_name: "{{ deployment_name }}-postgres-configuration"
      no_log: "{{ no_log }}"

-    - name: Create a postgres secret with the new host value
+    - name: Override postgres secret name
+      set_fact:
+        _pg_secret: "{{ _pg_secret | combine({'name': _pg_secret_name}) }}"
+      no_log: "{{ no_log }}"
+
+    - name: Override postgres secret host with new Postgres service
      set_fact:
        _pg_secret: "{{ _pg_secret | combine({'data': _pg_data}) }}"
      no_log: "{{ no_log }}"
--- a/roles/restore/vars/main.yml
+++ b/roles/restore/vars/main.yml
@@ -1,8 +1,8 @@
 ---

 deployment_type: "awx"
-_postgres_image: postgres
-_postgres_image_version: 13
+_postgres_image: quay.io/sclorg/postgresql-15-c9s
+_postgres_image_version: latest

 backup_api_version: '{{ deployment_type }}.ansible.com/v1beta1'
 backup_kind: 'AWXBackup'
@@ -12,7 +12,7 @@ secret_key_secret: '{{ deployment_name }}-secret-key'
 admin_password_secret: '{{ deployment_name }}-admin-password'
 broadcast_websocket_secret: '{{ deployment_name }}-broadcast-websocket'
 postgres_configuration_secret: '{{ deployment_name }}-postgres-configuration'
-supported_pg_version: 13
+supported_pg_version: 15
 image_pull_policy: IfNotPresent

 # If set to true, the restore process will delete the existing database and create a new one
Author	SHA1	Message	Date
Christian Adams	d5a3cb7519	Revert change to cast settings values to strings as they could be nested (#1756 ) - We cast the settings value to a string so that it would display properly in the Openshift UI. Unfortunately, the k8s validator will no longer allow arrays for settings values.	2024-03-12 11:58:00 -04:00
Christian Adams	ed6ac1a11a	Create a new postgres configuration secret when restoring a new instance (#1733 ) - This will avoid the operator changing the host value of the original deployments postgres_configuration_secret. As it was, restores broke the original deployment if it was still around before this change. Signed-off-by: Christian M. Adams <chadams@redhat.com>	2024-03-11 16:34:43 -04:00
Hao Liu	b5d81b8e5d	Fix awx_kube_devel (#1759 ) * Fix awx_kube_devel * Sanitize version name for kube_dev When in development mode, awx version may look like 23.9.1.dev18+gee9eac15dc.d20240311 k8s job to the migration can only have a name with alphanumeric, and '.', '-' so we can just drop off the + Signed-off-by: Seth Foster <fosterbseth@gmail.com> --------- Signed-off-by: Seth Foster <fosterbseth@gmail.com> Co-authored-by: Seth Foster <fosterbseth@gmail.com>	2024-03-11 19:01:00 +00:00
bartowl	3abeec518a	Bind EE images version with DEFAULT_AWX_VERSION (#1740 ) * bind ee_images, control_plane_ee_image and init_container_image with DEFAULT_AWX_VERSION instead of "latest" * fix when condition on init_container_image_version check * Use DEFAULT_AWX_VERSION for AWXMeshIngress * Add back AWX EE latest for backward compatibility --------- Co-authored-by: Hao Liu <44379968+TheRealHaoLiu@users.noreply.github.com>	2024-03-11 14:12:10 -04:00
Christian Adams	d2c4b9c8a4	The pg service label_selector now uses the deployment_type variable (#1755 )	2024-03-08 09:02:31 -05:00
Christian Adams	2ad1d25120	Update PostgreSQL docs about finding default version (#1747 )	2024-03-07 21:47:18 -05:00
Hao Liu	26e72b4e1d	Fix undefined external_hostname when using Route (#1753 )	2024-03-07 22:53:48 +00:00
Hao Liu	3434cbef96	AWXMeshIngress route respect external_hostname (#1752 ) * AwxMeshIngress route respect external_hostname * Set host in AWXMeshIngress route if external_hostname is defined	2024-03-07 15:43:06 -05:00
David Hageman	256d84a42a	Add imagePullSecrets option to Mesh Ingress (#1750 )	2024-03-07 13:17:39 -05:00
kurokobo	03cfe14c07	fix: extend expiration date for the certs for receptor nodes to 10 years (#1744 )	2024-03-06 19:52:04 +00:00
Tyler Muir	82c7dd2f44	add support for defining loadbalancer class (#1746 )	2024-03-06 14:35:16 -05:00
Erez Samimi	818b3682fa	Fix table format in container-probes.md (#1748 )	2024-03-06 19:28:46 +00:00
David Hageman	ffba1b4712	Add -ness checks and refactor migrations (#1674 )	2024-03-05 19:54:22 -05:00
kurokobo	dba934daa0	fix: revert type of status.upgradedPostgresVersion to string (#1745 )	2024-03-04 15:55:16 -05:00
aknochow	d0827ba426	Fixing postgres upgrade conditional (#1741 )	2024-03-01 17:09:15 -05:00
kurokobo	16b2f2a34f	fix: correct unsafe conditional (#1737 )	2024-03-01 20:54:40 +00:00
John Westcott IV	607a7ca58c	Upgrading to PostgreSQL 15 and moving to sclorg images (#1486 ) * Upgrading to postgres:15 * Changing image from postgres to sclorg * Handle scenario where upgrade status is not defined & correct pg tag * Rework the upgrade logic to be more resiliant for multiple upgrades --------- Co-authored-by: john-westcott-iv <john-westcott-iv@users.noreply.github.com> Co-authored-by: Christian M. Adams <chadams@redhat.com>	2024-02-29 17:02:11 -05:00
kurokobo	d11d66e81d	docs: enable search feature (#1725 )	2024-02-28 19:46:04 +00:00
Christian Adams	7a937b0932	Expose extra_settings in OLM UI form (#1732 ) Expose extra_settings in olm UI form Signed-off-by: Christian M. Adams <chadams@redhat.com>	2024-02-28 14:35:05 -05:00
Don Naro	e1c0e33b4f	Use pip tools to generate the full dependency tree (#1727 ) * add venv to gitignore * use pip compiled docs requirements	2024-02-28 14:33:29 -05:00
Neev Geffen	cf905ca5d0	CSV Change Descriptor of StorageClass From Text to StorageClass (#1729 ) Update awx-operator.clusterserviceversion.yaml: Some Descriptors for StorageClasses were set as text and not Storage Class	2024-02-28 19:19:24 +00:00
Hao Liu	630a5ee1f3	Fix bug where uppercase Route fails (#1731 )	2024-02-26 17:17:37 +00:00
Matt Miller	3d78e90ab1	Update Dockerfile operator-framework/ansible-operator to v1.34.0 (#1714 ) Update Dockerfile base image * Vulnerability scans against this image when deployed shows: CVE-2023-4911 * https://quay.io/repository/operator-framework/ansible-operator/manifest/sha256:f08f675976f42dc3a8ebbb8482acea153a8f57232e2ee48940e3d40ca40d24d9?tab=vulnerabilities * It appears if `5f3d9ed96f/Dockerfile (L1C14-L1C49)` is updated to `v1.34.0` this vulnerability is mitigated.	2024-02-21 14:50:08 -05:00
kurokobo	3981e6ba5e	fix: correct indentation for annotations for awxmeshingress (#1723 ) fix: correct indentation for annotations	2024-02-21 19:46:34 +00:00
kurokobo	ac682a9c05	docs: improve documentation for awxmeshingress (#1724 ) * add descriptions and examples for awxmeshingress * convert list to table * add note for image * correct minor wording issues * apply suggested changes from code review for docs/user-guide/advanced-configuration/mesh-ingress.md Co-authored-by: Seth Foster <fosterseth@users.noreply.github.com>	2024-02-21 14:36:50 -05:00
kurokobo	7bdf48ffc0	docs: add description for --force-conflicts option to upgrade crds (#1717 )	2024-02-21 14:31:32 -05:00
Seth Foster	fc11db4ece	Fix syntax error in mesh ingress docs (#1720 ) Signed-off-by: Seth Foster <fosterbseth@gmail.com>	2024-02-16 17:16:28 +00:00
TVo	148309325e	Separate out the custom receptor CA section to its own section. (#1707 )	2024-02-16 10:06:05 -07:00
Hao Liu	82756ebfe7	Add new doc for AWXMeshIngress (#1706 ) * Add new doc for AWXMeshIngress * Update docs/user-guide/advanced-configuration/mesh-ingress.md Co-authored-by: TVo <thavo@redhat.com> * Update docs/user-guide/advanced-configuration/mesh-ingress.md Co-authored-by: Seth Foster <fosterseth@users.noreply.github.com> * Update docs/user-guide/advanced-configuration/mesh-ingress.md Co-authored-by: TVo <thavo@redhat.com> * Update docs/user-guide/advanced-configuration/mesh-ingress.md Co-authored-by: TVo <thavo@redhat.com> * Update mesh-ingress.md * Update mesh-ingress.md * Grammar on line 48 --------- Co-authored-by: TVo <thavo@redhat.com> Co-authored-by: Seth Foster <fosterseth@users.noreply.github.com>	2024-02-15 13:07:35 -07:00
kurokobo	a9cee5f4da	fix: revert removal trim symbol before endif (#1715 )	2024-02-15 17:02:48 +00:00
Chris Meyers	5f3d9ed96f	More locked down websocket path * Previously, the nginx location would match on /foo/websocket... or /foo/api/websocket... Now, we require these two paths to start at the root i.e. <host>/websocket/... /api/websocket/... * Note: We now also require an ending / and do NOT support <host>/websocket_foobar but DO support <host>/websocket/foobar. This was always the intended behavior. We want to keep <host>/api/websocket/... "open" and routing to daphne in case we want to add more websocket urls in the future.	2024-02-13 15:53:34 -05:00