After the change for a single job to run PR tests, and if there is any
change to ansible_module_utils, all the playbook tests are executed,
and the result is a failure due to timeout.
This PR increases the timeout so that a PR with changes to
ansible_module_utils can have the tests executed.
In the current build container pipeline, all steps are serialized in a
single job, and if one of the jobs fail to build, due to broken
dependent image, or some Azure glitch, like slow connection, the only
way to rebuild the failed container is to rebuild all containers.
By building containers in parallel jobs, if a container fails to build
it is possible to restart only the failed job.
The usual scenario for PR checks is to execute only a few tests, and
searching for the results in several jobs makes it harder to find
issues.
By using a single job run the tests would take some more time to
complete, although not much, as only a small subset is executed, and
test verification would be easier and less error prone.
Dependind on how long it took for the jobs to start, a different seed
would be used to group tests and tests could either repeat or not be
selected at all.
By using a seed based on the day the test run reduces the chance of
using different random seeds, and still allow for the tests to be
executed in a different order.
The execution in different order is important to identify tests that
work or fail only if executed after other tests.
Azure Ubuntu images have Python 3.12 available, and as we did not pin
the requested Python version, the latest available one was used, causing
image preparation and tests to fail.
This patch pins Python version to 3.11 until test can be executed with
Python 3.12 and later.
Recently it was announced that Ansible 2.9 will be supported for some
time, and this patch ensures that we run the nightly tests against this
version of Ansible.
Recently, a change in the deployment roles forced the change to the
minimum version of ansible-core, and the change was unnoticed until
reported.
With this patch, we ensure all PRs checks are executed against the
minimun supported ansible-core version, so we can ensure that both
documentation and role metadata are correct and still valid.
Some test failures requires more information than just the playbook
simple output. By increasing verbosity, the used parameters and the
failed line will be visible in the test error report, making it easier
to identify, reproduce and fix the issue.
ansible-core 2.15 has been released on May 15th, 2023, and version 2.12
has reached EOL on May 22nd, 2023.
This patch updates the ansible-core versions used on upstream CI tests
to reflect Ansible's new releases.
The tests under 'tests/pytests' were a POC to bring tests that evaluate
the result of playbook execution on the IPA environment. This is
currently only implemented for dnszone tests, and similar test coverage
is obtained with other tests.
As there is an ongoing issue with Ansible's docker pluging
("the connection plugin 'docker' was not found"), which is stil under
investigation, by removing the pytest tests we'll remove the consistent
failures currently seen on upstream CI, and will not loose test
coverage, specially if we take into account downstream tests.
Also, a new version for the pytests will be available once multihost
testing is implemented for upstream.
The version requirement for requests need to be quoted not to lead into
a pip install command issue.
This is related to PR #1089 (Pin requests to < 2.29 temporarily)
Due to DNS issues and the increase number of tests, the timeout setting
used for upstream tests was being reached. As we still have room for
running the tests using Azure infrastructure, this patch increases the
timeout to 240 minutes (4h), per worker.
As we now have ansible-core 2.14 available through 'pip', the versions
used for testing on Azure should be 2.12, 2.13 and 2.14, as Ansible
keeps upstream support for the latest version plus the two previous
ones.
This patch update the version used in tests by increasing the version
used by 1 (MINOR).
Some tasks used to setup Azure environment might fail to temporary
errors like timeouts and connection failures. Allowing the tasks to
retry a few times will allow the test to be correctly executed rather
than returning an error that is not related to the feature tested.
Github has been migrating ubuntu-latest from 20.04 to 22.04. 22.04 comes
with cgroups version 2.
No tests are run at the momens as the setup of the test container always
fails with "Failed to create temporary directory" for gathering facts.
See also:
https://github.com/ansible-collections/news-for-maintainers/issues/28
Since FreeIPA 4.9.8 the 'config_mod' command has parameters to enable
and configure SIDs, and set the Netbios name.
This patch adds the following parameters to ipaconfig plugin:
enable_sids: New users and groups automatically get a SID assigned
add_sids: Add SIDs for existing users and groups
netbios_name: NetBIOS name of the IPA domain
Both add_sids and netbios_name requires 'enable_sid: yes'.
'enable_sid' and 'netbios_name' are returned when querying IPA
configuration.
'add_sids' always generate SIDs for users and groups, so, muiltiple
executions of the playbook with 'add_sids: yes' will return 'changed',
even if users and groups SIDs are not modified.
A new test playbook is available:
tests/config/test_config_sid.yml
New examples playbooks are available:
playbooks/config/change-ipa-domain-netbios-name.yml
playbooks/config/generate-users-groups-sids.yml
Fixes: #781
Related: https://bugzilla.redhat.com/show_bug.cgi?id=2069174
Related: https://bugzilla.redhat.com/show_bug.cgi?id=2069184
The current workflow for bug fixing or new enhancements in
ansible-freeipa includes running Ansible playbooks tests for all the
available plugins for every pull request, even for contained
modifications.
This patch creates a new workflow for pull requests where only the
affected plugins are tested in the PR. Changes that might affect other
parts of the code will trigger tests for the parts affected.
A utility script, utils/filter_tests, is used to set the variables
IPA_ENABLED_MODULES and IPA_ENABLED_TESTS before executing the tests,
effectively limiting which tests are executed. The script uses the
python plugin 'utils/filter_plugins.py' which lists all test modules
that should be executed for a list of modified source files.
Tests are selected for execution based on the plugin name. For example,
a change to 'plugins/modules/ipalocation.py' would trigger all playbook
tests under 'tests/location'. If a test playbook is modified, it is
scheduled to be executed. Changes to any file under
'plugins/module_utils' will force the execution of all tests, since any
module might be affected by that change.
The nature of the change is not evaluated, so a simple typo fix of a
comment in a file under 'plugins/module_utils' would still schedule all
test playbooks to be executed.
For roles, any file changed under the role directory would set the role
to be included in the tests. Playbook tests for roles must be created
under 'tests/<rolename>_role', where role name in the name of the role
without 'ipa', for example, the 'ipabackup' role test playbooks would
be stored under 'tests/backup_role'.
Since there is the possibility that the list of tests to be executed
might be less than the number of tests groups used (3), a new pytest
dependency was added, pytest-custom_exit_code, so that having no tests
to run isn't a test failure.
A new pipeline on Azure needs to be created to use the new test script.
The plugin pytest-split-tests is used to split the tests in several
groups so that the tests can be executed in parallel is upstream CI.
The issue is thet pytest-split-tests last release was more that a
year ago, upstream developers have not been responsive, and there is
a bug when the number of tests to be executed by a group is zero.
The patch to fix this issue has been open for a year.
This patch deprecates the use of pytest-split-tests, changing the
plugin used to split the tests to pytest-split, which is actively
mainatined.
Some DNS tests have been disabled for Fedora latest, but not for Fedora
Rawhide. Therefore these tests are filin still in nighty:
- dnsforwardzone
- test_dnsconfig_forwarders_ports
Since test configuration can vary in different scenarios (test images)
this patch adds a script to list the scenarios configuration, and a
step to the playbook test jobs to display the scenario configuration.
Sometimes, mostly due do differences in package versions, there are
some tests that fail on a single distribution which cannot be fixed
by ansible-freeipa, requiring that the offending package is fixed.
To keep tests running succesfully we have options to disable the
failing tests, but this changes are globally applied, meaning that, by
disabling a test, it is disable in all tested distributions.
This patch allows tests to be enabled or disabled for a specific
distribution, by setting the configuration on the 'variable' template
for the specific testing scenario.
As the current latest upstream version of ansible-core is 2.13.0, to
test against ansible-core 2.12 series we need to pin the version used
on the test.
This patch enables the already defined tests for ansible-core 2.12 that
were available but commented out.
Provide a pipeline to test ansible-freeipa as an Ansible Galaxy
collection. The tests will use 'utils/build-galaxy-release.sh' to
create the galaxy release file, install it as a collection, and run
the tests in it, which were modified to use FQCN.
The tests will run only on 'fedora-latest' for each PR, and on all
platforms for nightly and weekly tests.
Due to an issue with IPA in Fedora 36, dnsconfig and dnsforwardzone
plugin tests must be disabled.
See FreeIPA issue: https://pagure.io/freeipa/issue/9158
pytest provide the means to skip tests based on patterns, but writing
these patterns for ansible-freeipa might not be feasible.
This PR allows the selection of playbook tests and modules that will
be executed with pytest using the environmentt variables IPA_ENABLED_TESTS
IPA_ENABLED_MODULES, IPA_DISABLED_TESTS or IPA_DISABLED_MODULES.
When using IPA_ENABLED_MODULES, all modules will be disabled, and only
the modules in the enabled list will be tested. If using the test
filter, IPA_ENABLED_TESTS, all tests are disabled, unless they are in
the enabled test lists.
If the IPA_DISABLED_* version is used, tests and modules are enabled by
default, and the list is used to disable the module or specific test.
To disable a test or module in Azure CI, edit the file
`tests/azure/variables` and add the desired tests or modules to the
parameter variables `enabled_modules`, 'enabled_tests`, `disabled_tests`
or `disable_modules`.
Note that, if added to the `master` branch, this will affect the tests
for every pipeline that it is include (including 'nightly'), so it should
be used with care.
It can be used with TEMP commits to enable only the desired tests,
speeding up upstream tests.
The current ansible-core available in Fedora and RHEL is 2.12 series.
This patch sets the version used for every PR CI to match this series.
Other versions should be used only in the nightly/weekly tests.
This patch adds the latest ansible-core as a test target in upstream
nightl/weekly CI.
As, currently, the latest available ansible-core is still 2.12.z, the
current ansible-core 2.12 targets were disabled. They should be enabled
when ansible-core 2.13 is available.
CentOS 8 images are not supported anymore, and we are using CentOS 8
Stream images.
This patch removes all configuration for CentOS 8 and updates test
README to point to the available container images.
As Ansible versions might change, and as we don't need to report which
version is used on every test, as the information is avaiable in case
it is needed, the jobs labels are changed to easier display which image
was used for testing.
The correct name for upcoming release of CentOS is CentOS 9 Stream,
usually abbreviated to 'c9s'. As we need to differentiate from the
stream and the standard versions, this patch modifies the Azure
piipelines to use 'c9s' instead of 'CentOS 9'.
Newer Ansible versions will require at least Python 3.8 to be used,
and the build containers pipeline was requiring Python 3.6, which is
EOL.
This patch requests the latest Python version available for the
controller, and allows it to be configured to a specific version if,
and when, needed.
Add configuration to build a testing CentOS 8 stream image and to
execute upstream tests using that image in pull requests (Ansible
2.9) and on the nightly tests (all supported Ansible versions).
