This patch removes 'molecule' as a dependency for tests, by using the
scripts under `utils` to setup the environment.
By not using molecule, we have more flexibility on using either docker
or podman as the container engine, and makes it easy to reproduce the
environment on different distros, allowing for a more consistent error
reproduction off Azure.
The usual scenario for PR checks is to execute only a few tests, and
searching for the results in several jobs makes it harder to find
issues.
By using a single job run the tests would take some more time to
complete, although not much, as only a small subset is executed, and
test verification would be easier and less error prone.
Azure Ubuntu images have Python 3.12 available, and as we did not pin
the requested Python version, the latest available one was used, causing
image preparation and tests to fail.
This patch pins Python version to 3.11 until test can be executed with
Python 3.12 and later.
The current workflow for bug fixing or new enhancements in
ansible-freeipa includes running Ansible playbooks tests for all the
available plugins for every pull request, even for contained
modifications.
This patch creates a new workflow for pull requests where only the
affected plugins are tested in the PR. Changes that might affect other
parts of the code will trigger tests for the parts affected.
A utility script, utils/filter_tests, is used to set the variables
IPA_ENABLED_MODULES and IPA_ENABLED_TESTS before executing the tests,
effectively limiting which tests are executed. The script uses the
python plugin 'utils/filter_plugins.py' which lists all test modules
that should be executed for a list of modified source files.
Tests are selected for execution based on the plugin name. For example,
a change to 'plugins/modules/ipalocation.py' would trigger all playbook
tests under 'tests/location'. If a test playbook is modified, it is
scheduled to be executed. Changes to any file under
'plugins/module_utils' will force the execution of all tests, since any
module might be affected by that change.
The nature of the change is not evaluated, so a simple typo fix of a
comment in a file under 'plugins/module_utils' would still schedule all
test playbooks to be executed.
For roles, any file changed under the role directory would set the role
to be included in the tests. Playbook tests for roles must be created
under 'tests/<rolename>_role', where role name in the name of the role
without 'ipa', for example, the 'ipabackup' role test playbooks would
be stored under 'tests/backup_role'.
Since there is the possibility that the list of tests to be executed
might be less than the number of tests groups used (3), a new pytest
dependency was added, pytest-custom_exit_code, so that having no tests
to run isn't a test failure.
A new pipeline on Azure needs to be created to use the new test script.
