Some test failures requires more information than just the playbook
simple output. By increasing verbosity, the used parameters and the
failed line will be visible in the test error report, making it easier
to identify, reproduce and fix the issue.
Due to DNS issues and the increase number of tests, the timeout setting
used for upstream tests was being reached. As we still have room for
running the tests using Azure infrastructure, this patch increases the
timeout to 240 minutes (4h), per worker.
Some tasks used to setup Azure environment might fail to temporary
errors like timeouts and connection failures. Allowing the tasks to
retry a few times will allow the test to be correctly executed rather
than returning an error that is not related to the feature tested.
The plugin pytest-split-tests is used to split the tests in several
groups so that the tests can be executed in parallel is upstream CI.
The issue is thet pytest-split-tests last release was more that a
year ago, upstream developers have not been responsive, and there is
a bug when the number of tests to be executed by a group is zero.
The patch to fix this issue has been open for a year.
This patch deprecates the use of pytest-split-tests, changing the
plugin used to split the tests to pytest-split, which is actively
mainatined.
Since test configuration can vary in different scenarios (test images)
this patch adds a script to list the scenarios configuration, and a
step to the playbook test jobs to display the scenario configuration.
Sometimes, mostly due do differences in package versions, there are
some tests that fail on a single distribution which cannot be fixed
by ansible-freeipa, requiring that the offending package is fixed.
To keep tests running succesfully we have options to disable the
failing tests, but this changes are globally applied, meaning that, by
disabling a test, it is disable in all tested distributions.
This patch allows tests to be enabled or disabled for a specific
distribution, by setting the configuration on the 'variable' template
for the specific testing scenario.
pytest provide the means to skip tests based on patterns, but writing
these patterns for ansible-freeipa might not be feasible.
This PR allows the selection of playbook tests and modules that will
be executed with pytest using the environmentt variables IPA_ENABLED_TESTS
IPA_ENABLED_MODULES, IPA_DISABLED_TESTS or IPA_DISABLED_MODULES.
When using IPA_ENABLED_MODULES, all modules will be disabled, and only
the modules in the enabled list will be tested. If using the test
filter, IPA_ENABLED_TESTS, all tests are disabled, unless they are in
the enabled test lists.
If the IPA_DISABLED_* version is used, tests and modules are enabled by
default, and the list is used to disable the module or specific test.
To disable a test or module in Azure CI, edit the file
`tests/azure/variables` and add the desired tests or modules to the
parameter variables `enabled_modules`, 'enabled_tests`, `disabled_tests`
or `disable_modules`.
Note that, if added to the `master` branch, this will affect the tests
for every pipeline that it is include (including 'nightly'), so it should
be used with care.
It can be used with TEMP commits to enable only the desired tests,
speeding up upstream tests.
This patch modifies the Python version used to be the latest available,
and add stages to execute the tests using ansible-core 2.12. As we
use Ubuntu 20.04, Python version 3.8 is avaiable.
Previously, ansible-core 2.12 was not available as it cannot be
installed with Python 3.6, which was the version used.
Currently, upstream CI test documentation against different Ansible
versions, but playbook tests are only executed with Ansible 2.9 series.
This patch add support for running playbook tests against Ansible 2.9,
ansible-core 2.11, and against latest version of Ansible.
As running all the tests for every PR would take too long, the tests
for every PR use only Anisble 2.9, and are executed on Fedora-latest
and CentOS 7 and 8.
A new pipeline for nightly tests was added, which runs the tests in the
same distros, using Ansible 2.9, latest and Ansible-core 2.11.
The test preparation failed with "the connection plugin
'community.docker.docker' was not found" in "Setup test container".
"ansible-galaxy collection install community.docker" has been added
to
tests/azure/templates/playbook_tests.yml and
tests/azure/templates/pytest_tests.yml
Until now ansible-freeipa repository only had playbook tests. This
commit introduces the ability of creating TestCase classes connected to
the master host. This connection can be used to run commands in the
managed host after the ansible playbook execution is the allowing the
verification of the machine state.
test_users_absent.yml was using users_absent.json. It has been adapted to
use users.json instead with an additional json_query to get only the names
from users_present.json.
create_users_json.yml has been added to create users.json if it is missing
containing 500 users. It is included by test_users_present.yml and
test_users_absent.yml.
users_present.sh has been renamed to users.sh and modified to create by
default users.json with 1000 users and additional with password and
passwordexpiration in two years.
jmespath has been added to pip install list in
tests/azure/templates/playbook_tests.yml to emable the use of json_query.
The requirement for jmespath has been added to tests/README.md.
* Moved azure CI definitions from azure-pipelines.yml to
tests/azure/azure-pipelines.yml.
* Updated azure CI definitions to run playbook tests using docker
containers.
