On some systems it is required or desired to run Ansible with a specific
Python interpreter. This patch allows the selection of the Python binary
to use for the pytest playbook tests by setting the environment variable
IPA_PYTHON_PATH. Set it the the full path of the Python interpreter.
Truncate stdout and stderr in the way that it hopefully shows all
important information. At least 15 lines of stdout (Ansible tasks) and
remaining from stderr to fill up to maxlen size.
The test log mostly only showed lines from stderr as the log was a
concatenation of stdout and stderr and only the last 2500 chars of this
have been used. This was hiding the test result from stdout and failures
have not beeen visible.
When enabling or disabling multiple tests, a comma separated list must
be used, but current code is using a colon to split the list.
This patch fixes this behavior.
Currently, all tests are scheduled to execution, even those that are
not executed due to being absent from the list of enabled tests
configured in the IPA_ENABLED_* variables. The tests that are not
executed are marked 'skipped'.
This patch change this behavior by not scheduling tests that are not
configured to be executed. It means that tests not the IPA_DISABLED_*
lists are not skipped anymore, but not scheduled to be executed. If
any test is in IPA_ENABLED_* lists, only those tests are marked for
execution. A side effect is that there is no visual feedback on which
tests were not executed, as disabled tests are not evaluated anymore.
Also, when IPA_SERVER_HOST was not set, all tests were skipped, but
an error should raised in this case, as there are no hosts to run the
tests against.
This patch modifies this behavior to fail the test with an exception if
IPA_SERVER_HOST is not set.
Checking if some output is present or absent from standard streams was
done by simple string searching. Due to recent changes in FreeIPA, this
search is not effective due to capitalization differences in boolean
values output. Changing the string searching to regular expression
searches fixes this behavior for current and previous versions of
FreeIPA.
This patch also adds more information on the assert tests in case of an
error, so that it is easier to understand why the test failed.
pytest provide the means to skip tests based on patterns, but writing
these patterns for ansible-freeipa might not be feasible.
This PR allows the selection of playbook tests and modules that will
be executed with pytest using the environmentt variables IPA_ENABLED_TESTS
IPA_ENABLED_MODULES, IPA_DISABLED_TESTS or IPA_DISABLED_MODULES.
When using IPA_ENABLED_MODULES, all modules will be disabled, and only
the modules in the enabled list will be tested. If using the test
filter, IPA_ENABLED_TESTS, all tests are disabled, unless they are in
the enabled test lists.
If the IPA_DISABLED_* version is used, tests and modules are enabled by
default, and the list is used to disable the module or specific test.
To disable a test or module in Azure CI, edit the file
`tests/azure/variables` and add the desired tests or modules to the
parameter variables `enabled_modules`, 'enabled_tests`, `disabled_tests`
or `disable_modules`.
Note that, if added to the `master` branch, this will affect the tests
for every pipeline that it is include (including 'nightly'), so it should
be used with care.
It can be used with TEMP commits to enable only the desired tests,
speeding up upstream tests.
In order to run the tests in idm-ci we need to configure the our pytest
tests environment variables. This PR configures that automatically if an
environment variable TWD is available and $TWD/config exists.
Until now ansible-freeipa repository only had playbook tests. This
commit introduces the ability of creating TestCase classes connected to
the master host. This connection can be used to run commands in the
managed host after the ansible playbook execution is the allowing the
verification of the machine state.
