AWX Operator
An Ansible AWX operator for Kubernetes built with Operator SDK and Ansible.
Table of Contents
NOTE: we are in the process of moving this readme into official docs in the /docs folder. Please go there to find additional sections during this interim move phase.
Uninstall
To uninstall an AWX deployment instance, you basically need to remove the AWX kind related to that instance. For example, to delete an AWX instance named awx-demo, you would do:
$ kubectl delete awx awx-demo
awx.awx.ansible.com "awx-demo" deleted
Deleting an AWX instance will remove all related deployments and statefulsets, however, persistent volumes and secrets will remain. To enforce secrets also getting removed, you can use garbage_collect_secrets: true.
Note: If you ever intend to recover an AWX from an existing database you will need a copy of the secrets in order to perform a successful recovery.
Upgrading
To upgrade AWX, it is recommended to upgrade the awx-operator to the version that maps to the desired version of AWX. To find the version of AWX that will be installed by the awx-operator by default, check the version specified in the image_version variable in roles/installer/defaults/main.yml for that particular release.
Apply the awx-operator.yml for that release to upgrade the operator, and in turn also upgrade your AWX deployment.
Backup
The first part of any upgrade should be a backup. Note, there are secrets in the pod which work in conjunction with the database. Having just a database backup without the required secrets will not be sufficient for recovering from an issue when upgrading to a new version. See the backup role documentation for information on how to backup your database and secrets.
In the event you need to recover the backup see the restore role documentation. Before Restoring from a backup, be sure to:
- delete the old existing AWX CR
- delete the persistent volume claim (PVC) for the database from the old deployment, which has a name like
postgres-13-<deployment-name>-postgres-13-0
Note: Do not delete the namespace/project, as that will delete the backup and the backup's PVC as well.
PostgreSQL Upgrade Considerations
If there is a PostgreSQL major version upgrade, after the data directory on the PVC is migrated to the new version, the old PVC is kept by default. This provides the ability to roll back if needed, but can take up extra storage space in your cluster unnecessarily. You can configure it to be deleted automatically after a successful upgrade by setting the following variable on the AWX spec.
spec:
postgres_keep_pvc_after_upgrade: False
v0.14.0
Cluster-scope to Namespace-scope considerations
Starting with awx-operator 0.14.0, AWX can only be deployed in the namespace that the operator exists in. This is called a namespace-scoped operator. If you are upgrading from an earlier version, you will want to
delete your existing awx-operator service account, role and role binding.
Project is now based on v1.x of the operator-sdk project
Starting with awx-operator 0.14.0, the project is now based on operator-sdk 1.x. You may need to manually delete your old operator Deployment to avoid issues.
Steps to upgrade
Delete your old AWX Operator and existing awx-operator service account, role and role binding in default namespace first:
$ kubectl -n default delete deployment awx-operator
$ kubectl -n default delete serviceaccount awx-operator
$ kubectl -n default delete clusterrolebinding awx-operator
$ kubectl -n default delete clusterrole awx-operator
Then install the new AWX Operator by following the instructions in Basic Install. The NAMESPACE environment variable have to be the name of the namespace in which your old AWX instance resides.
Once the new AWX Operator is up and running, your AWX deployment will also be upgraded.
Disable IPV6
Starting with AWX Operator release 0.24.0,IPV6 was enabled in ngnix configuration which causes
upgrades and installs to fail in environments where IPv6 is not allowed. Starting in 1.1.1 release, you can set the ipv6_disabled flag on the AWX
spec. If you need to use an AWX operator version between 0.24.0 and 1.1.1 in an IPv6 disabled environment, it is suggested to enabled ipv6 on worker
nodes.
In order to disable ipv6 on ngnix configuration (awx-web container), add following to the AWX spec.
The following variables are customizable
| Name | Description | Default |
|---|---|---|
| ipv6_disabled | Flag to disable ipv6 | false |
spec:
ipv6_disabled: true
Adding Execution Nodes
Starting with AWX Operator v0.30.0 and AWX v21.7.0, standalone execution nodes can be added to your deployments. See AWX execution nodes docs for information about this feature.
Custom Receptor CA
The control nodes on the K8S cluster will communicate with execution nodes via mutual TLS TCP connections, running via Receptor. Execution nodes will verify incoming connections by ensuring the x509 certificate was issued by a trusted Certificate Authority (CA).
A user may wish to provide their own CA for this validation. If no CA is provided, AWX Operator will automatically generate one using OpenSSL.
Given custom ca.crt and ca.key stored locally, run the following,
kubectl create secret tls awx-demo-receptor-ca \
--cert=/path/to/ca.crt --key=/path/to/ca.key
The secret should be named {AWX Custom Resource name}-receptor-ca. In the above the AWX CR name is "awx-demo". Please replace "awx-demo" with your AWX Custom Resource name.
If this secret is created after AWX is deployed, run the following to restart the deployment,
kubectl rollout restart deployment awx-demo
Important Note, changing the receptor CA will break connections to any existing execution nodes. These nodes will enter an unavailable state, and jobs will not be able to run on them. Users will need to download and re-run the install bundle for each execution node. This will replace the TLS certificate files with those signed by the new CA. The execution nodes should then appear in a ready state after a few minutes.
Contributing
Please visit our contributing guidelines.
Release Process
The first step is to create a draft release. Typically this will happen in the Stage Release workflow for AWX and you don't need to do it as a separate step.
If you need to do an independent release of the operator, you can run the Stage Release in the awx-operator repo. Both of these workflows will run smoke tests, so there is no need to do this manually.
After the draft release is created, publish it and the Promote AWX Operator image will run, which will:
- Publish image to Quay
- Release Helm chart
Author
This operator was originally built in 2019 by Jeff Geerling and is now maintained by the Ansible Team
Code of Conduct
We ask all of our community members and contributors to adhere to the Ansible code of conduct. If you have questions or need assistance, please reach out to our community team at codeofconduct@ansible.com
Get Involved
We welcome your feedback and ideas. The AWX operator uses the same mailing list and IRC channel as AWX itself. Here's how to reach us with feedback and questions:
- Join the
#ansible-awxchannel on irc.libera.chat - Join the mailing list