Some test failures requires more information than just the playbook
simple output. By increasing verbosity, the used parameters and the
failed line will be visible in the test error report, making it easier
to identify, reproduce and fix the issue.
Due to DNS issues and the increase number of tests, the timeout setting
used for upstream tests was being reached. As we still have room for
running the tests using Azure infrastructure, this patch increases the
timeout to 240 minutes (4h), per worker.
Some tasks used to setup Azure environment might fail to temporary
errors like timeouts and connection failures. Allowing the tasks to
retry a few times will allow the test to be correctly executed rather
than returning an error that is not related to the feature tested.
The plugin pytest-split-tests is used to split the tests in several
groups so that the tests can be executed in parallel is upstream CI.
The issue is thet pytest-split-tests last release was more that a
year ago, upstream developers have not been responsive, and there is
a bug when the number of tests to be executed by a group is zero.
The patch to fix this issue has been open for a year.
This patch deprecates the use of pytest-split-tests, changing the
plugin used to split the tests to pytest-split, which is actively
mainatined.
Since test configuration can vary in different scenarios (test images)
this patch adds a script to list the scenarios configuration, and a
step to the playbook test jobs to display the scenario configuration.
Sometimes, mostly due do differences in package versions, there are
some tests that fail on a single distribution which cannot be fixed
by ansible-freeipa, requiring that the offending package is fixed.
To keep tests running succesfully we have options to disable the
failing tests, but this changes are globally applied, meaning that, by
disabling a test, it is disable in all tested distributions.
This patch allows tests to be enabled or disabled for a specific
distribution, by setting the configuration on the 'variable' template
for the specific testing scenario.
Provide a pipeline to test ansible-freeipa as an Ansible Galaxy
collection. The tests will use 'utils/build-galaxy-release.sh' to
create the galaxy release file, install it as a collection, and run
the tests in it, which were modified to use FQCN.
The tests will run only on 'fedora-latest' for each PR, and on all
platforms for nightly and weekly tests.
