awx-operator/docs/troubleshooting/debugging.md

Debugging the AWX Operator

General Debugging

When the operator is deploying AWX, it is running the installer role inside the operator container. If the AWX CR's status is Failed, it is often useful to look at the awx-operator container logs, which shows the output of the installer role. To see these logs, run:

kubectl logs deployments/awx-operator-controller-manager -c awx-manager -f

Inspect k8s Resources

Past that, it is often useful to inspect various resources the AWX Operator manages like:

• awx
• awxbackup
• awxrestore
• pod
• deployment
• pvc
• service
• ingress
• route
• secrets
• serviceaccount

To inspect these resources you can use these commands

# Inspecting k8s resources
kubectl describe -n <namespace> <resource> <resource-name>
kubectl get -n <namespace> <resource> <resource-name> -o yaml
kubectl logs -n <namespace> <resource> <resource-name>

# Inspecting Pods
kubectl exec -it -n <namespace> <pod> <pod-name>

Configure No Log

It is possible to show task output for debugging by setting no_log to false on the AWX CR spec. This will show output in the awx-operator logs for any failed tasks where no_log was set to true.

For example:

---
apiVersion: awx.ansible.com/v1beta1
kind: AWX
metadata:
  name: awx-demo
spec:
  service_type: nodeport
  no_log: false                  # <------------

Iterating on the installer without deploying the operator

Go through the normal basic install steps.

Install some dependencies:

ansible-galaxy collection install -r molecule/requirements.yml
pip install -r molecule/requirements.txt

To prevent the changes we're about to make from being overwritten, scale down any running instance of the operator:

kubectl scale deployment awx-operator-controller-manager --replicas=0

Create a playbook that invokes the installer role (the operator uses ansible-runner's role execution feature):

# run.yml
---
- hosts: localhost
  roles:
    - installer

Create a vars file:

# vars.yml
---
ansible_operator_meta:
  name: awx
  namespace: awx
set_self_labels: false
update_status: false
service_type: nodeport

The vars file will replace the awx resource so any value that you wish to over ride using the awx resource, put in the vars file. For example, if you wish to use your own image, version and pull policy, you can specify it like below:

# vars.yml
---
ansible_operator_meta:
  name: awx
  namespace: awx
set_self_labels: false
update_status: false
service_type: nodeport
image: $DEV_DOCKER_TAG_BASE/awx_kube_devel
image_pull_policy: Always
image_version: $COMPOSE_TAG

Run the installer:

ansible-playbook run.yml -e @vars.yml -v

Grab the URL and admin password:

$ kubectl get secret awx-admin-password -- -o jsonpath="{.data.password}" | base64 --decode ; echo
LU6lTfvnkjUvDwL240kXKy1sNhjakZmT