Release 4.5.0.

* keycloak: fix creating a user federation w/ idempotent id

Creating a user federation while specifying an id (that doesn't exist
yet) will fail with a 404. This commits fix this behavior.

* keycloak: fix user federation mapper duplication

This commit fixes a bug where mappers are duplicated instead of
configured when creating a user federation.

When creating a user federation, some mappers are autogenerated by
keycloak. This commit lets the keycloak_user_federation module recompute
mappers final values after the user federation is created so that the
module can try to merge them by their name.

* add missing fragment for pr #4212

(cherry picked from commit c1485b885d)

Co-authored-by: Jules Lamur <jlamur@users.noreply.github.com>

.azure-pipelines/azure-pipelines.yml

@@ -310,6 +310,8 @@ stages:
               test: ubuntu1804
             - name: Ubuntu 20.04
               test: ubuntu2004
+            - name: Alpine 3
+              test: alpine3
           groups:
             - 1
             - 2
@@ -324,8 +326,6 @@ stages:
           targets:
             - name: CentOS 6
               test: centos6
-            - name: CentOS 8
-              test: centos8
             - name: Fedora 34
               test: fedora34
             - name: openSUSE 15 py3
@@ -350,6 +350,8 @@ stages:
               test: fedora33
             - name: openSUSE 15 py2
               test: opensuse15py2
+            - name: Alpine 3
+              test: alpine3
           groups:
             - 2
             - 3
@@ -384,6 +386,26 @@ stages:
             - 2
             - 3
 
+### Community Docker
+  - stage: Docker_community_devel
+    displayName: Docker (community images) devel
+    dependsOn: []
+    jobs:
+      - template: templates/matrix.yml
+        parameters:
+          testFormat: devel/linux-community/{0}
+          targets:
+            - name: Debian Bullseye
+              test: debian-bullseye/3.9
+            - name: ArchLinux
+              test: archlinux/3.10
+            - name: CentOS Stream 8
+              test: centos-stream8/3.8
+          groups:
+            - 1
+            - 2
+            - 3
+
 ### Cloud
   - stage: Cloud_devel
     displayName: Cloud devel
@@ -459,6 +481,7 @@ stages:
       - Docker_2_10
       - Docker_2_11
       - Docker_2_12
+      - Docker_community_devel
       - Cloud_devel
       - Cloud_2_9
       - Cloud_2_10

.github/BOTMETA.yml

@@ -418,6 +418,8 @@ files:
     maintainers: Spredzy
   $modules/cloud/scaleway/scaleway_organization_info.py:
     maintainers: Spredzy
+  $modules/cloud/scaleway/scaleway_private_network.py:
+    maintainers: pastral
   $modules/cloud/scaleway/scaleway_security_group.py:
     maintainers: DenBeke
   $modules/cloud/scaleway/scaleway_security_group_info.py:
@@ -819,7 +821,7 @@ files:
   $modules/packaging/os/opkg.py:
     maintainers: skinp
   $modules/packaging/os/pacman.py:
-    maintainers: elasticdog indrajitr tchernomax
+    maintainers: elasticdog indrajitr tchernomax jraby
     labels: pacman
     ignore: elasticdog
   $modules/packaging/os/pacman_key.py:
@@ -977,6 +979,8 @@ files:
     maintainers: farhan7500 gautamphegde
   $modules/storage/ibm/:
     maintainers: tzure
+  $modules/storage/pmem/pmem.py:
+    maintainers: mizumm
   $modules/storage/vexata/:
     maintainers: vexata
   $modules/storage/zfs/:
@@ -1233,7 +1237,7 @@ macros:
   team_cyberark_conjur: jvanderhoof ryanprior
   team_e_spirit: MatrixCrawler getjack
   team_flatpak: JayKayy oolongbrothers
-  team_gitlab: Lunik Shaps dj-wasabi marwatk waheedi zanssa scodeman metanovii sh0shin
+  team_gitlab: Lunik Shaps dj-wasabi marwatk waheedi zanssa scodeman metanovii sh0shin nejch lgatellier
   team_hpux: bcoca davx8342
   team_huawei: QijunPan TommyLike edisonxiang freesky-edward hwDCN niuzhenguo xuxiaowei0512 yanzhangi zengchen1024 zhongjun2
   team_ipa: Akasurde Nosmoht fxfitz justchris1

CHANGELOG.rst

@@ -6,6 +6,68 @@ Community General Release Notes
 
 This changelog describes changes after version 3.0.0.
 
+v4.5.0
+======
+
+Release Summary
+---------------
+
+Regular feature and bugfix release.
+
+Minor Changes
+-------------
+
+- Avoid internal ansible-core module_utils in favor of equivalent public API available since at least Ansible 2.9. This fixes some instances added since the last time this was fixed (https://github.com/ansible-collections/community.general/pull/4232).
+- ansible_galaxy_install - added option ``no_deps`` to the module (https://github.com/ansible-collections/community.general/issues/4174).
+- gitlab_group_variable - new ``variables`` parameter (https://github.com/ansible-collections/community.general/pull/4038 and https://github.com/ansible-collections/community.general/issues/4074).
+- keycloak_* modules - added connection timeout parameter when calling server (https://github.com/ansible-collections/community.general/pull/4168).
+- linode inventory plugin - add support for caching inventory results (https://github.com/ansible-collections/community.general/pull/4179).
+- opentelemetry_plugin - enrich service when using the ``jenkins``, ``hetzner`` or ``jira`` modules (https://github.com/ansible-collections/community.general/pull/4105).
+- pacman - the module has been rewritten and is now much faster when using ``state=latest``. Operations are now done all packages at once instead of package per package and the configured output format of ``pacman`` no longer affect the module's operation. (https://github.com/ansible-collections/community.general/pull/3907, https://github.com/ansible-collections/community.general/issues/3783, https://github.com/ansible-collections/community.general/issues/4079)
+- passwordstore lookup plugin - add configurable ``lock`` and ``locktimeout`` options to avoid race conditions in itself and in the ``pass`` utility it calls. By default, the plugin now locks on write operations (https://github.com/ansible-collections/community.general/pull/4194).
+- proxmox modules - move common code into ``module_utils`` (https://github.com/ansible-collections/community.general/pull/4029).
+- proxmox_kvm - added EFI disk support when creating VM with OVMF UEFI BIOS with new ``efidisk0`` option (https://github.com/ansible-collections/community.general/pull/4106, https://github.com/ansible-collections/community.general/issues/1638).
+- proxmox_kwm - add ``win11`` to ``ostype`` parameter for Windows 11 and Windows Server 2022 support (https://github.com/ansible-collections/community.general/issues/4023, https://github.com/ansible-collections/community.general/pull/4191).
+
+Bugfixes
+--------
+
+- dconf - skip processes that disappeared while we inspected them (https://github.com/ansible-collections/community.general/issues/4151).
+- gitlab_group_variable - add missing documentation about GitLab versions that support ``environment_scope`` and ``variable_type`` (https://github.com/ansible-collections/community.general/pull/4038).
+- gitlab_group_variable - allow to set same variable name under different environment scopes. Due this change, the return value ``group_variable`` differs from previous version in check mode. It was counting ``updated`` values, because it was accidentally overwriting environment scopes (https://github.com/ansible-collections/community.general/pull/4038).
+- gitlab_group_variable - fix idempotent change behaviour for float and integer variables (https://github.com/ansible-collections/community.general/pull/4038).
+- gitlab_project_variable - ``value`` is not necessary when deleting variables (https://github.com/ansible-collections/community.general/pull/4150).
+- gitlab_runner - make ``project`` and ``owned`` mutually exclusive (https://github.com/ansible-collections/community.general/pull/4136).
+- homebrew_cask - fix force install operation (https://github.com/ansible-collections/community.general/issues/3703).
+- imc_rest - fixes the module failure due to the usage of ``itertools.izip_longest`` which is not available in Python 3 (https://github.com/ansible-collections/community.general/issues/4206).
+- ini_file - when removing nothing do not report changed (https://github.com/ansible-collections/community.general/issues/4154).
+- keycloak_user_federation - creating a user federation while specifying an ID (that does not exist yet) no longer fail with a 404 Not Found (https://github.com/ansible-collections/community.general/pull/4212).
+- keycloak_user_federation - mappers auto-created by keycloak are matched and merged by their name and no longer create duplicated entries (https://github.com/ansible-collections/community.general/pull/4212).
+- mail callback plugin - fix encoding of the name of sender and recipient (https://github.com/ansible-collections/community.general/issues/4060, https://github.com/ansible-collections/community.general/pull/4061).
+- passwordstore lookup plugin - fix error detection for non-English locales (https://github.com/ansible-collections/community.general/pull/4219).
+- passwordstore lookup plugin - prevent returning path names as passwords by accident (https://github.com/ansible-collections/community.general/issues/4185, https://github.com/ansible-collections/community.general/pull/4192).
+- vdo - fix options error (https://github.com/ansible-collections/community.general/pull/4163).
+- yum_versionlock - fix matching of existing entries with names passed to the module. Match yum and dnf lock format (https://github.com/ansible-collections/community.general/pull/4183).
+
+New Modules
+-----------
+
+Cloud
+~~~~~
+
+scaleway
+^^^^^^^^
+
+- scaleway_private_network - Scaleway private network management
+
+Storage
+~~~~~~~
+
+pmem
+^^^^
+
+- pmem - Configure Intel Optane Persistent Memory modules
+
 v4.4.0
 ======
 

changelogs/changelog.yaml

@@ -1399,3 +1399,99 @@ releases:
       name: homectl
       namespace: system
     release_date: '2022-02-01'
+  4.5.0:
+    changes:
+      bugfixes:
+      - dconf - skip processes that disappeared while we inspected them (https://github.com/ansible-collections/community.general/issues/4151).
+      - gitlab_group_variable - add missing documentation about GitLab versions that
+        support ``environment_scope`` and ``variable_type`` (https://github.com/ansible-collections/community.general/pull/4038).
+      - 'gitlab_group_variable - allow to set same variable name under different environment
+        scopes. Due this change, the return value ``group_variable`` differs from
+        previous version in check mode. It was counting ``updated`` values, because
+        it was accidentally overwriting environment scopes (https://github.com/ansible-collections/community.general/pull/4038).
+
+        '
+      - gitlab_group_variable - fix idempotent change behaviour for float and integer
+        variables (https://github.com/ansible-collections/community.general/pull/4038).
+      - gitlab_project_variable - ``value`` is not necessary when deleting variables
+        (https://github.com/ansible-collections/community.general/pull/4150).
+      - gitlab_runner - make ``project`` and ``owned`` mutually exclusive (https://github.com/ansible-collections/community.general/pull/4136).
+      - homebrew_cask - fix force install operation (https://github.com/ansible-collections/community.general/issues/3703).
+      - imc_rest - fixes the module failure due to the usage of ``itertools.izip_longest``
+        which is not available in Python 3 (https://github.com/ansible-collections/community.general/issues/4206).
+      - ini_file - when removing nothing do not report changed (https://github.com/ansible-collections/community.general/issues/4154).
+      - keycloak_user_federation - creating a user federation while specifying an
+        ID (that does not exist yet) no longer fail with a 404 Not Found (https://github.com/ansible-collections/community.general/pull/4212).
+      - keycloak_user_federation - mappers auto-created by keycloak are matched and
+        merged by their name and no longer create duplicated entries (https://github.com/ansible-collections/community.general/pull/4212).
+      - mail callback plugin - fix encoding of the name of sender and recipient (https://github.com/ansible-collections/community.general/issues/4060,
+        https://github.com/ansible-collections/community.general/pull/4061).
+      - passwordstore lookup plugin - fix error detection for non-English locales
+        (https://github.com/ansible-collections/community.general/pull/4219).
+      - passwordstore lookup plugin - prevent returning path names as passwords by
+        accident (https://github.com/ansible-collections/community.general/issues/4185,
+        https://github.com/ansible-collections/community.general/pull/4192).
+      - vdo - fix options error (https://github.com/ansible-collections/community.general/pull/4163).
+      - yum_versionlock - fix matching of existing entries with names passed to the
+        module. Match yum and dnf lock format (https://github.com/ansible-collections/community.general/pull/4183).
+      minor_changes:
+      - Avoid internal ansible-core module_utils in favor of equivalent public API
+        available since at least Ansible 2.9. This fixes some instances added since
+        the last time this was fixed (https://github.com/ansible-collections/community.general/pull/4232).
+      - ansible_galaxy_install - added option ``no_deps`` to the module (https://github.com/ansible-collections/community.general/issues/4174).
+      - gitlab_group_variable - new ``variables`` parameter (https://github.com/ansible-collections/community.general/pull/4038
+        and https://github.com/ansible-collections/community.general/issues/4074).
+      - keycloak_* modules - added connection timeout parameter when calling server
+        (https://github.com/ansible-collections/community.general/pull/4168).
+      - linode inventory plugin - add support for caching inventory results (https://github.com/ansible-collections/community.general/pull/4179).
+      - opentelemetry_plugin - enrich service when using the ``jenkins``, ``hetzner``
+        or ``jira`` modules (https://github.com/ansible-collections/community.general/pull/4105).
+      - pacman - the module has been rewritten and is now much faster when using ``state=latest``.
+        Operations are now done all packages at once instead of package per package
+        and the configured output format of ``pacman`` no longer affect the module's
+        operation. (https://github.com/ansible-collections/community.general/pull/3907,
+        https://github.com/ansible-collections/community.general/issues/3783, https://github.com/ansible-collections/community.general/issues/4079)
+      - passwordstore lookup plugin - add configurable ``lock`` and ``locktimeout``
+        options to avoid race conditions in itself and in the ``pass`` utility it
+        calls. By default, the plugin now locks on write operations (https://github.com/ansible-collections/community.general/pull/4194).
+      - proxmox modules - move common code into ``module_utils`` (https://github.com/ansible-collections/community.general/pull/4029).
+      - proxmox_kvm - added EFI disk support when creating VM with OVMF UEFI BIOS
+        with new ``efidisk0`` option (https://github.com/ansible-collections/community.general/pull/4106,
+        https://github.com/ansible-collections/community.general/issues/1638).
+      - proxmox_kwm - add ``win11`` to ``ostype`` parameter for Windows 11 and Windows
+        Server 2022 support (https://github.com/ansible-collections/community.general/issues/4023,
+        https://github.com/ansible-collections/community.general/pull/4191).
+      release_summary: Regular feature and bugfix release.
+    fragments:
+    - 3703-force-install-homebrew-cask.yml
+    - 3907-pacman-speedup.yml
+    - 3916-fix-vdo-options-type.yml
+    - 4.5.0.yml
+    - 4029-proxmox-refactor.yml
+    - 4061-fix-mail-recipient-encoding.yml
+    - 4086-rework_of_gitlab_proyect_variable_over_gitlab_group_variable.yml
+    - 4105-opentelemetry_plugin-enrich_jira_hetzner_jenkins_services.yaml
+    - 4106-proxmox-efidisk0-support.yaml
+    - 4136-gitlab_runner-make-project-owned-mutually-exclusive.yml
+    - 4150-gitlab-project-variable-absent-fix.yml
+    - 4151-dconf-catch-psutil-nosuchprocess.yaml
+    - 4154-ini_file_changed.yml
+    - 4168-add-keycloak-url-timeout.yml
+    - 4179-linode-inventory-cache.yaml
+    - 4183-fix-yum_versionlock.yaml
+    - 4191-proxmox-add-win11.yml
+    - 4192-improve-passwordstore-consistency.yml
+    - 4194-configurable-passwordstore-locking.yml
+    - 4206-imc-rest-module.yaml
+    - 4212-fixes-for-keycloak-user-federation.yml
+    - 4219-passwordstore-locale-fix.yml
+    - 4232-text-converter-import.yml
+    - 4240-ansible_galaxy_install-no_deps.yml
+    modules:
+    - description: Configure Intel Optane Persistent Memory modules
+      name: pmem
+      namespace: storage.pmem
+    - description: Scaleway private network management
+      name: scaleway_private_network
+      namespace: cloud.scaleway
+    release_date: '2022-02-22'

docs/docsite/helper/lists_mergeby/default-common.yml

@@ -0,0 +1,13 @@
+list1:
+  - name: foo
+    extra: true
+  - name: bar
+    extra: false
+  - name: meh
+    extra: true
+
+list2:
+  - name: foo
+    path: /foo
+  - name: baz
+    path: /baz

docs/docsite/helper/lists_mergeby/default-recursive-true.yml

@@ -0,0 +1,19 @@
+list1:
+  - name: myname01
+    param01:
+      x: default_value
+      y: default_value
+      list:
+        - default_value
+  - name: myname02
+    param01: [1, 1, 2, 3]
+
+list2:
+  - name: myname01
+    param01:
+      y: patch_value
+      z: patch_value
+      list:
+        - patch_value
+  - name: myname02
+    param01: [3, 4, 4, {key: value}]

docs/docsite/helper/lists_mergeby/example-001.yml

@@ -0,0 +1,10 @@
+---
+- name: 1. Merge two lists by common attribute 'name'
+  include_vars:
+    dir: example-001_vars
+- debug:
+    var: list3
+  when: debug|d(false)|bool
+- template:
+    src: list3.out.j2
+    dest: example-001.out

docs/docsite/helper/lists_mergeby/example-001_vars/default-common.yml

@@ -0,0 +1 @@
+../default-common.yml

docs/docsite/helper/lists_mergeby/example-001_vars/list3.yml

@@ -0,0 +1,2 @@
+list3: "{{ list1|
+           community.general.lists_mergeby(list2, 'name') }}"

docs/docsite/helper/lists_mergeby/example-002.yml

@@ -0,0 +1,10 @@
+---
+- name: 2. Merge two lists by common attribute 'name'
+  include_vars:
+    dir: example-002_vars
+- debug:
+    var: list3
+  when: debug|d(false)|bool
+- template:
+    src: list3.out.j2
+    dest: example-002.out

docs/docsite/helper/lists_mergeby/example-002_vars/default-common.yml

@@ -0,0 +1 @@
+../default-common.yml

docs/docsite/helper/lists_mergeby/example-002_vars/list3.yml

@@ -0,0 +1,2 @@
+list3: "{{ [list1, list2]|
+           community.general.lists_mergeby('name') }}"

docs/docsite/helper/lists_mergeby/example-003.yml

@@ -0,0 +1,10 @@
+---
+- name: 3. Merge recursive by 'name', replace lists (default)
+  include_vars:
+    dir: example-003_vars
+- debug:
+    var: list3
+  when: debug|d(false)|bool
+- template:
+    src: list3.out.j2
+    dest: example-003.out

docs/docsite/helper/lists_mergeby/example-003_vars/default-recursive-true.yml

@@ -0,0 +1 @@
+../default-recursive-true.yml

docs/docsite/helper/lists_mergeby/example-003_vars/list3.yml

@@ -0,0 +1,3 @@
+list3: "{{ [list1, list2]|
+           community.general.lists_mergeby('name',
+                                           recursive=true) }}"

docs/docsite/helper/lists_mergeby/example-004.yml

@@ -0,0 +1,10 @@
+---
+- name: 4. Merge recursive by 'name', keep lists
+  include_vars:
+    dir: example-004_vars
+- debug:
+    var: list3
+  when: debug|d(false)|bool
+- template:
+    src: list3.out.j2
+    dest: example-004.out

docs/docsite/helper/lists_mergeby/example-004_vars/default-recursive-true.yml

@@ -0,0 +1 @@
+../default-recursive-true.yml

docs/docsite/helper/lists_mergeby/example-004_vars/list3.yml

@@ -0,0 +1,4 @@
+list3: "{{ [list1, list2]|
+           community.general.lists_mergeby('name',
+                                           recursive=true,
+                                           list_merge='keep') }}"

docs/docsite/helper/lists_mergeby/example-005.yml

@@ -0,0 +1,10 @@
+---
+- name: 5. Merge recursive by 'name', append lists
+  include_vars:
+    dir: example-005_vars
+- debug:
+    var: list3
+  when: debug|d(false)|bool
+- template:
+    src: list3.out.j2
+    dest: example-005.out

docs/docsite/helper/lists_mergeby/example-005_vars/default-recursive-true.yml

@@ -0,0 +1 @@
+../default-recursive-true.yml

docs/docsite/helper/lists_mergeby/example-005_vars/list3.yml

@@ -0,0 +1,4 @@
+list3: "{{ [list1, list2]|
+           community.general.lists_mergeby('name',
+                                           recursive=true,
+                                           list_merge='append') }}"

docs/docsite/helper/lists_mergeby/example-006.yml

@@ -0,0 +1,10 @@
+---
+- name: 6. Merge recursive by 'name', prepend lists
+  include_vars:
+    dir: example-006_vars
+- debug:
+    var: list3
+  when: debug|d(false)|bool
+- template:
+    src: list3.out.j2
+    dest: example-006.out

docs/docsite/helper/lists_mergeby/example-006_vars/default-recursive-true.yml

@@ -0,0 +1 @@
+../default-recursive-true.yml

docs/docsite/helper/lists_mergeby/example-006_vars/list3.yml

@@ -0,0 +1,4 @@
+list3: "{{ [list1, list2]|
+           community.general.lists_mergeby('name',
+                                           recursive=true,
+                                           list_merge='prepend') }}"

docs/docsite/helper/lists_mergeby/example-007.yml

@@ -0,0 +1,10 @@
+---
+- name: 7. Merge recursive by 'name', append lists 'remove present'
+  include_vars:
+    dir: example-007_vars
+- debug:
+    var: list3
+  when: debug|d(false)|bool
+- template:
+    src: list3.out.j2
+    dest: example-007.out

docs/docsite/helper/lists_mergeby/example-007_vars/default-recursive-true.yml

@@ -0,0 +1 @@
+../default-recursive-true.yml

docs/docsite/helper/lists_mergeby/example-007_vars/list3.yml

@@ -0,0 +1,4 @@
+list3: "{{ [list1, list2]|
+           community.general.lists_mergeby('name',
+                                           recursive=true,
+                                           list_merge='append_rp') }}"

docs/docsite/helper/lists_mergeby/example-008.yml

@@ -0,0 +1,10 @@
+---
+- name: 8. Merge recursive by 'name', prepend lists 'remove present'
+  include_vars:
+    dir: example-008_vars
+- debug:
+    var: list3
+  when: debug|d(false)|bool
+- template:
+    src: list3.out.j2
+    dest: example-008.out

docs/docsite/helper/lists_mergeby/example-008_vars/default-recursive-true.yml

@@ -0,0 +1 @@
+../default-recursive-true.yml

docs/docsite/helper/lists_mergeby/example-008_vars/list3.yml

@@ -0,0 +1,4 @@
+list3: "{{ [list1, list2]|
+           community.general.lists_mergeby('name',
+                                           recursive=true,
+                                           list_merge='prepend_rp') }}"

docs/docsite/helper/lists_mergeby/examples.yml

@@ -1,37 +1,49 @@
 ---
 examples:
+  - label: 'In the example below the lists are merged by the attribute ``name``:'
+    file: example-001_vars/list3.yml
+    lang: 'yaml+jinja'
+  - label: 'This produces:'
+    file: example-001.out
+    lang: 'yaml'
+  - label: 'It is possible to use a list of lists as an input of the filter:'
+    file: example-002_vars/list3.yml
+    lang: 'yaml+jinja'
+  - label: 'This produces the same result as in the previous example:'
+    file: example-002.out
+    lang: 'yaml'
   - label: 'Example ``list_merge=replace`` (default):'
-    file: example-003.yml
+    file: example-003_vars/list3.yml
     lang: 'yaml+jinja'
   - label: 'This produces:'
     file: example-003.out
     lang: 'yaml'
   - label: 'Example ``list_merge=keep``:'
-    file: example-004.yml
+    file: example-004_vars/list3.yml
     lang: 'yaml+jinja'
   - label: 'This produces:'
     file: example-004.out
     lang: 'yaml'
   - label: 'Example ``list_merge=append``:'
-    file: example-005.yml
+    file: example-005_vars/list3.yml
     lang: 'yaml+jinja'
   - label: 'This produces:'
     file: example-005.out
     lang: 'yaml'
   - label: 'Example ``list_merge=prepend``:'
-    file: example-006.yml
+    file: example-006_vars/list3.yml
     lang: 'yaml+jinja'
   - label: 'This produces:'
     file: example-006.out
     lang: 'yaml'
   - label: 'Example ``list_merge=append_rp``:'
-    file: example-007.yml
+    file: example-007_vars/list3.yml
     lang: 'yaml+jinja'
   - label: 'This produces:'
     file: example-007.out
     lang: 'yaml'
   - label: 'Example ``list_merge=prepend_rp``:'
-    file: example-008.yml
+    file: example-008_vars/list3.yml
     lang: 'yaml+jinja'
   - label: 'This produces:'
     file: example-008.out

docs/docsite/helper/lists_mergeby/examples_all.rst.j2

@@ -3,6 +3,6 @@
 
 .. code-block:: {{ i.lang }}
 
-  {{ lookup('file', source_path ~ i.file)|indent(2) }}
+  {{ lookup('file', i.file)|indent(2) }}
 
 {% endfor %}

docs/docsite/helper/lists_mergeby/filter_guide_abstract_informations_merging_lists_of_dictionaries.rst.j2

@@ -0,0 +1,57 @@
+Merging lists of dictionaries
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+If you have two or more lists of dictionaries and want to combine them into a list of merged dictionaries, where the dictionaries are merged by an attribute, you can use the ``lists_mergeby`` filter.
+
+.. note:: The output of the examples in this section use the YAML callback plugin. Quoting: "Ansible output that can be quite a bit easier to read than the default JSON formatting." See :ref:`the documentation for the community.general.yaml callback plugin <ansible_collections.community.general.yaml_callback>`.
+
+Let us use the lists below in the following examples:
+
+.. code-block:: yaml
+
+  {{ lookup('file', 'default-common.yml')|indent(2) }}
+
+{% for i in examples[0:2] %}
+{{ i.label }}
+
+.. code-block:: {{ i.lang }}
+
+  {{ lookup('file', i.file)|indent(2) }}
+
+{% endfor %}
+
+.. versionadded:: 2.0.0
+
+{% for i in examples[2:4] %}
+{{ i.label }}
+
+.. code-block:: {{ i.lang }}
+
+  {{ lookup('file', i.file)|indent(2) }}
+
+{% endfor %}
+
+The filter also accepts two optional parameters: ``recursive`` and ``list_merge``. These parameters are only supported when used with ansible-base 2.10 or ansible-core, but not with Ansible 2.9. This is available since community.general 4.4.0.
+
+**recursive**
+    Is a boolean, default to ``False``. Should the ``community.general.lists_mergeby`` recursively merge nested hashes. Note: It does not depend on the value of the ``hash_behaviour`` setting in ``ansible.cfg``.
+
+**list_merge**
+    Is a string, its possible values are ``replace`` (default), ``keep``, ``append``, ``prepend``, ``append_rp`` or ``prepend_rp``. It modifies the behaviour of ``community.general.lists_mergeby`` when the hashes to merge contain arrays/lists.
+
+The examples below set ``recursive=true`` and display the differences among all six options of ``list_merge``. Functionality of the parameters is exactly the same as in the filter ``combine``. See :ref:`Combining hashes/dictionaries <combine_filter>` to learn details about these options.
+
+Let us use the lists below in the following examples
+
+.. code-block:: yaml
+
+  {{ lookup('file', 'default-recursive-true.yml')|indent(2) }}
+
+{% for i in examples[4:16] %}
+{{ i.label }}
+
+.. code-block:: {{ i.lang }}
+
+  {{ lookup('file', i.file)|indent(2) }}
+
+{% endfor %}

docs/docsite/helper/lists_mergeby/list3.out.j2

@@ -0,0 +1,2 @@
+list3:
+{{ list3|to_nice_yaml(indent=0) }}

docs/docsite/helper/lists_mergeby/playbook.yml

@@ -1,41 +1,59 @@
 ---
-# The following runs all examples:
+
+# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
+# 1) Run all examples and create example-XXX.out
+# shell> ansible-playbook playbook.yml -e examples=true
 #
-#    ANSIBLE_STDOUT_CALLBACK=community.general.yaml ansible-playbook playbook.yml -e examples=true
+# 2) Optionally, for testing, create examples_all.rst
+# shell> ansible-playbook playbook.yml -e examples_all=true
 #
-# You need to copy the YAML output of example-XXX.yml into example-XXX.out.
+# 3) Create docs REST files
+# shell> ansible-playbook playbook.yml -e merging_lists_of_dictionaries=true
 #
-# The following generates examples.rst out of the .out files:
+# Notes:
+# * Use YAML callback, e.g. set ANSIBLE_STDOUT_CALLBACK=community.general.yaml
+# * Use sphinx-view to render and review the REST files
+#   shell> sphinx-view <path_to_helper>/examples_all.rst
+# * Proofread and copy completed docs *.rst files into the directory rst.
+# * Then delete the *.rst and *.out files from this directory. Do not
+#   add *.rst and *.out in this directory to the version control.
 #
-#    ansible-playbook playbook.yml -e template=true
+# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
+# community.general/docs/docsite/helper/lists_mergeby/playbook.yml
+
 - hosts: localhost
   gather_facts: false
-  vars:
-    source_path: ../../rst/examples/lists_mergeby/
   tasks:
 
     - block:
-        - import_tasks: '{{ source_path }}example-001.yml'
+        - import_tasks: example-001.yml
           tags: t001
-        - import_tasks: '{{ source_path }}example-002.yml'
+        - import_tasks: example-002.yml
           tags: t002
-        - import_tasks: '{{ source_path }}example-003.yml'
+        - import_tasks: example-003.yml
           tags: t003
-        - import_tasks: '{{ source_path }}example-004.yml'
+        - import_tasks: example-004.yml
           tags: t004
-        - import_tasks: '{{ source_path }}example-005.yml'
+        - import_tasks: example-005.yml
           tags: t005
-        - import_tasks: '{{ source_path }}example-006.yml'
+        - import_tasks: example-006.yml
           tags: t006
-        - import_tasks: '{{ source_path }}example-007.yml'
+        - import_tasks: example-007.yml
           tags: t007
-        - import_tasks: '{{ source_path }}example-008.yml'
+        - import_tasks: example-008.yml
           tags: t008
       when: examples|d(false)|bool
 
     - block:
         - include_vars: examples.yml
         - template:
-            src: examples.rst.j2
-            dest: examples.rst
-      when: template|d(false)|bool
+            src: examples_all.rst.j2
+            dest: examples_all.rst
+      when: examples_all|d(false)|bool
+
+    - block:
+        - include_vars: examples.yml
+        - template:
+            src: filter_guide_abstract_informations_merging_lists_of_dictionaries.rst.j2
+            dest: filter_guide_abstract_informations_merging_lists_of_dictionaries.rst
+      when: merging_lists_of_dictionaries|d(false)|bool

docs/docsite/rst/examples/lists_mergeby/example-001.out

@@ -1,10 +0,0 @@
-  list3:
-  - extra: false
-    name: bar
-  - name: baz
-    path: /baz
-  - extra: true
-    name: foo
-    path: /foo
-  - extra: true
-    name: meh

docs/docsite/rst/examples/lists_mergeby/example-001.yml

@@ -1,20 +0,0 @@
----
-- name: Merge two lists by common attribute 'name'
-  set_fact:
-    list3: "{{ list1|
-               community.general.lists_mergeby(list2, 'name') }}"
-  vars:
-    list1:
-      - name: foo
-        extra: true
-      - name: bar
-        extra: false
-      - name: meh
-        extra: true
-    list2:
-      - name: foo
-        path: /foo
-      - name: baz
-        path: /baz
-- debug:
-    var: list3

docs/docsite/rst/examples/lists_mergeby/example-002.out

@@ -1,10 +0,0 @@
-  list3:
-  - extra: false
-    name: bar
-  - name: baz
-    path: /baz
-  - extra: true
-    name: foo
-    path: /foo
-  - extra: true
-    name: meh

docs/docsite/rst/examples/lists_mergeby/example-002.yml

@@ -1,20 +0,0 @@
----
-- name: Merge two lists by common attribute 'name'
-  set_fact:
-    list3: "{{ [list1, list2]|
-               community.general.lists_mergeby('name') }}"
-  vars:
-    list1:
-      - name: foo
-        extra: true
-      - name: bar
-        extra: false
-      - name: meh
-        extra: true
-    list2:
-      - name: foo
-        path: /foo
-      - name: baz
-        path: /baz
-- debug:
-    var: list3

docs/docsite/rst/examples/lists_mergeby/example-003.out

@@ -1,14 +0,0 @@
-  list3:
-  - name: myname01
-    param01:
-      list:
-      - patch_value
-      x: default_value
-      y: patch_value
-      z: patch_value
-  - name: myname02
-    param01:
-    - 3
-    - 4
-    - 4
-    - key: value

docs/docsite/rst/examples/lists_mergeby/example-003.yml

@@ -1,28 +0,0 @@
----
-- name: Merge recursive by 'name', replace lists (default)
-  set_fact:
-    list3: "{{ [list1, list2]|
-               community.general.lists_mergeby('name',
-                                               recursive=true) }}"
-  vars:
-    list1:
-      - name: myname01
-        param01:
-          x: default_value
-          y: default_value
-          list:
-            - default_value
-      - name: myname02
-        param01: [1, 1, 2, 3]
-
-    list2:
-      - name: myname01
-        param01:
-          y: patch_value
-          z: patch_value
-          list:
-            - patch_value
-      - name: myname02
-        param01: [3, 4, 4, {key: value}]
-- debug:
-    var: list3

docs/docsite/rst/examples/lists_mergeby/example-004.out

@@ -1,14 +0,0 @@
-  list3:
-  - name: myname01
-    param01:
-      list:
-      - default_value
-      x: default_value
-      y: patch_value
-      z: patch_value
-  - name: myname02
-    param01:
-    - 1
-    - 1
-    - 2
-    - 3

docs/docsite/rst/examples/lists_mergeby/example-004.yml

@@ -1,29 +0,0 @@
----
-- name: Merge recursive by 'name', keep lists
-  set_fact:
-    list3: "{{ [list1, list2]|
-               community.general.lists_mergeby('name',
-                                               recursive=true,
-                                               list_merge='keep') }}"
-  vars:
-    list1:
-      - name: myname01
-        param01:
-          x: default_value
-          y: default_value
-          list:
-            - default_value
-      - name: myname02
-        param01: [1, 1, 2, 3]
-
-    list2:
-      - name: myname01
-        param01:
-          y: patch_value
-          z: patch_value
-          list:
-            - patch_value
-      - name: myname02
-        param01: [3, 4, 4, {key: value}]
-- debug:
-    var: list3

docs/docsite/rst/examples/lists_mergeby/example-005.out

@@ -1,19 +0,0 @@
-  list3:
-  - name: myname01
-    param01:
-      list:
-      - default_value
-      - patch_value
-      x: default_value
-      y: patch_value
-      z: patch_value
-  - name: myname02
-    param01:
-    - 1
-    - 1
-    - 2
-    - 3
-    - 3
-    - 4
-    - 4
-    - key: value

docs/docsite/rst/examples/lists_mergeby/example-005.yml

@@ -1,29 +0,0 @@
----
-- name: Merge recursive by 'name', append lists
-  set_fact:
-    list3: "{{ [list1, list2]|
-               community.general.lists_mergeby('name',
-                                               recursive=true,
-                                               list_merge='append') }}"
-  vars:
-    list1:
-      - name: myname01
-        param01:
-          x: default_value
-          y: default_value
-          list:
-            - default_value
-      - name: myname02
-        param01: [1, 1, 2, 3]
-
-    list2:
-      - name: myname01
-        param01:
-          y: patch_value
-          z: patch_value
-          list:
-            - patch_value
-      - name: myname02
-        param01: [3, 4, 4, {key: value}]
-- debug:
-    var: list3

docs/docsite/rst/examples/lists_mergeby/example-006.out

@@ -1,19 +0,0 @@
-  list3:
-  - name: myname01
-    param01:
-      list:
-      - patch_value
-      - default_value
-      x: default_value
-      y: patch_value
-      z: patch_value
-  - name: myname02
-    param01:
-    - 3
-    - 4
-    - 4
-    - key: value
-    - 1
-    - 1
-    - 2
-    - 3

docs/docsite/rst/examples/lists_mergeby/example-006.yml

@@ -1,29 +0,0 @@
----
-- name: Merge recursive by 'name', prepend lists
-  set_fact:
-    list3: "{{ [list1, list2]|
-               community.general.lists_mergeby('name',
-                                               recursive=true,
-                                               list_merge='prepend') }}"
-  vars:
-    list1:
-      - name: myname01
-        param01:
-          x: default_value
-          y: default_value
-          list:
-            - default_value
-      - name: myname02
-        param01: [1, 1, 2, 3]
-
-    list2:
-      - name: myname01
-        param01:
-          y: patch_value
-          z: patch_value
-          list:
-            - patch_value
-      - name: myname02
-        param01: [3, 4, 4, {key: value}]
-- debug:
-    var: list3

docs/docsite/rst/examples/lists_mergeby/example-007.out

@@ -1,18 +0,0 @@
-  list3:
-  - name: myname01
-    param01:
-      list:
-      - default_value
-      - patch_value
-      x: default_value
-      y: patch_value
-      z: patch_value
-  - name: myname02
-    param01:
-    - 1
-    - 1
-    - 2
-    - 3
-    - 4
-    - 4
-    - key: value

docs/docsite/rst/examples/lists_mergeby/example-007.yml

@@ -1,29 +0,0 @@
----
-- name: Merge recursive by 'name', append lists 'remove present'
-  set_fact:
-    list3: "{{ [list1, list2]|
-               community.general.lists_mergeby('name',
-                                               recursive=true,
-                                               list_merge='append_rp') }}"
-  vars:
-    list1:
-      - name: myname01
-        param01:
-          x: default_value
-          y: default_value
-          list:
-            - default_value
-      - name: myname02
-        param01: [1, 1, 2, 3]
-
-    list2:
-      - name: myname01
-        param01:
-          y: patch_value
-          z: patch_value
-          list:
-            - patch_value
-      - name: myname02
-        param01: [3, 4, 4, {key: value}]
-- debug:
-    var: list3

docs/docsite/rst/examples/lists_mergeby/example-008.out

@@ -1,18 +0,0 @@
-  list3:
-  - name: myname01
-    param01:
-      list:
-      - patch_value
-      - default_value
-      x: default_value
-      y: patch_value
-      z: patch_value
-  - name: myname02
-    param01:
-    - 3
-    - 4
-    - 4
-    - key: value
-    - 1
-    - 1
-    - 2

docs/docsite/rst/examples/lists_mergeby/example-008.yml

@@ -1,29 +0,0 @@
----
-- name: Merge recursive by 'name', prepend lists 'remove present'
-  set_fact:
-    list3: "{{ [list1, list2]|
-               community.general.lists_mergeby('name',
-                                               recursive=true,
-                                               list_merge='prepend_rp') }}"
-  vars:
-    list1:
-      - name: myname01
-        param01:
-          x: default_value
-          y: default_value
-          list:
-            - default_value
-      - name: myname02
-        param01: [1, 1, 2, 3]
-
-    list2:
-      - name: myname01
-        param01:
-          y: patch_value
-          z: patch_value
-          list:
-            - patch_value
-      - name: myname02
-        param01: [3, 4, 4, {key: value}]
-- debug:
-    var: list3

docs/docsite/rst/filter_guide_abstract_informations_merging_lists_of_dictionaries.rst

@@ -5,30 +5,30 @@ If you have two or more lists of dictionaries and want to combine them into a li
 
 .. note:: The output of the examples in this section use the YAML callback plugin. Quoting: "Ansible output that can be quite a bit easier to read than the default JSON formatting." See :ref:`the documentation for the community.general.yaml callback plugin <ansible_collections.community.general.yaml_callback>`.
 
+Let us use the lists below in the following examples:
+
+.. code-block:: yaml
+
+  list1:
+    - name: foo
+      extra: true
+    - name: bar
+      extra: false
+    - name: meh
+      extra: true
+
+  list2:
+    - name: foo
+      path: /foo
+    - name: baz
+      path: /baz
+
 In the example below the lists are merged by the attribute ``name``:
 
 .. code-block:: yaml+jinja
 
-  ---
-  - name: Merge two lists by common attribute 'name'
-    set_fact:
-      list3: "{{ list1|
-                 community.general.lists_mergeby(list2, 'name') }}"
-    vars:
-      list1:
-        - name: foo
-          extra: true
-        - name: bar
-          extra: false
-        - name: meh
-          extra: true
-      list2:
-        - name: foo
-          path: /foo
-        - name: baz
-          path: /baz
-  - debug:
-      var: list3
+  list3: "{{ list1|
+             community.general.lists_mergeby(list2, 'name') }}"
 
 This produces:
 
@@ -45,32 +45,15 @@ This produces:
   - extra: true
     name: meh
 
+
 .. versionadded:: 2.0.0
 
 It is possible to use a list of lists as an input of the filter:
 
 .. code-block:: yaml+jinja
 
-  ---
-  - name: Merge two lists by common attribute 'name'
-    set_fact:
-      list3: "{{ [list1, list2]|
-                 community.general.lists_mergeby('name') }}"
-    vars:
-      list1:
-        - name: foo
-          extra: true
-        - name: bar
-          extra: false
-        - name: meh
-          extra: true
-      list2:
-        - name: foo
-          path: /foo
-        - name: baz
-          path: /baz
-  - debug:
-      var: list3
+  list3: "{{ [list1, list2]|
+             community.general.lists_mergeby('name') }}"
 
 This produces the same result as in the previous example:
 
@@ -87,6 +70,7 @@ This produces the same result as in the previous example:
   - extra: true
     name: meh
 
+
 The filter also accepts two optional parameters: ``recursive`` and ``list_merge``. These parameters are only supported when used with ansible-base 2.10 or ansible-core, but not with Ansible 2.9. This is available since community.general 4.4.0.
 
 **recursive**
@@ -97,337 +81,212 @@ The filter also accepts two optional parameters: ``recursive`` and ``list_merge`
 
 The examples below set ``recursive=true`` and display the differences among all six options of ``list_merge``. Functionality of the parameters is exactly the same as in the filter ``combine``. See :ref:`Combining hashes/dictionaries <combine_filter>` to learn details about these options.
 
+Let us use the lists below in the following examples
+
+.. code-block:: yaml
+
+  list1:
+    - name: myname01
+      param01:
+        x: default_value
+        y: default_value
+        list:
+          - default_value
+    - name: myname02
+      param01: [1, 1, 2, 3]
+
+  list2:
+    - name: myname01
+      param01:
+        y: patch_value
+        z: patch_value
+        list:
+          - patch_value
+    - name: myname02
+      param01: [3, 4, 4, {key: value}]
+
 Example ``list_merge=replace`` (default):
 
 .. code-block:: yaml+jinja
 
-  ---
-  - name: Merge recursive by 'name', replace lists (default)
-    set_fact:
-      list3: "{{ [list1, list2]|
-                 community.general.lists_mergeby('name',
-                                                 recursive=true) }}"
-    vars:
-      list1:
-        - name: myname01
-          param01:
-            x: default_value
-            y: default_value
-            list:
-              - default_value
-        - name: myname02
-          param01: [1, 1, 2, 3]
-
-      list2:
-        - name: myname01
-          param01:
-            y: patch_value
-            z: patch_value
-            list:
-              - patch_value
-        - name: myname02
-          param01: [3, 4, 4, {key: value}]
-  - debug:
-      var: list3
+  list3: "{{ [list1, list2]|
+             community.general.lists_mergeby('name',
+                                             recursive=true) }}"
 
 This produces:
 
 .. code-block:: yaml
 
-    list3:
-    - name: myname01
-      param01:
-        list:
-        - patch_value
-        x: default_value
-        y: patch_value
-        z: patch_value
-    - name: myname02
-      param01:
-      - 3
-      - 4
-      - 4
-      - key: value
+  list3:
+  - name: myname01
+    param01:
+      list:
+      - patch_value
+      x: default_value
+      y: patch_value
+      z: patch_value
+  - name: myname02
+    param01:
+    - 3
+    - 4
+    - 4
+    - key: value
 
 Example ``list_merge=keep``:
 
 .. code-block:: yaml+jinja
 
-  ---
-  - name: Merge recursive by 'name', keep lists
-    set_fact:
-      list3: "{{ [list1, list2]|
-                 community.general.lists_mergeby('name',
-                                                 recursive=true,
-                                                 list_merge='keep') }}"
-    vars:
-      list1:
-        - name: myname01
-          param01:
-            x: default_value
-            y: default_value
-            list:
-              - default_value
-        - name: myname02
-          param01: [1, 1, 2, 3]
-
-      list2:
-        - name: myname01
-          param01:
-            y: patch_value
-            z: patch_value
-            list:
-              - patch_value
-        - name: myname02
-          param01: [3, 4, 4, {key: value}]
-  - debug:
-      var: list3
+  list3: "{{ [list1, list2]|
+             community.general.lists_mergeby('name',
+                                             recursive=true,
+                                             list_merge='keep') }}"
 
 This produces:
 
 .. code-block:: yaml
 
-    list3:
-    - name: myname01
-      param01:
-        list:
-        - default_value
-        x: default_value
-        y: patch_value
-        z: patch_value
-    - name: myname02
-      param01:
-      - 1
-      - 1
-      - 2
-      - 3
+  list3:
+  - name: myname01
+    param01:
+      list:
+      - default_value
+      x: default_value
+      y: patch_value
+      z: patch_value
+  - name: myname02
+    param01:
+    - 1
+    - 1
+    - 2
+    - 3
 
 Example ``list_merge=append``:
 
 .. code-block:: yaml+jinja
 
-  ---
-  - name: Merge recursive by 'name', append lists
-    set_fact:
-      list3: "{{ [list1, list2]|
-                 community.general.lists_mergeby('name',
-                                                 recursive=true,
-                                                 list_merge='append') }}"
-    vars:
-      list1:
-        - name: myname01
-          param01:
-            x: default_value
-            y: default_value
-            list:
-              - default_value
-        - name: myname02
-          param01: [1, 1, 2, 3]
-
-      list2:
-        - name: myname01
-          param01:
-            y: patch_value
-            z: patch_value
-            list:
-              - patch_value
-        - name: myname02
-          param01: [3, 4, 4, {key: value}]
-  - debug:
-      var: list3
+  list3: "{{ [list1, list2]|
+             community.general.lists_mergeby('name',
+                                             recursive=true,
+                                             list_merge='append') }}"
 
 This produces:
 
 .. code-block:: yaml
 
-    list3:
-    - name: myname01
-      param01:
-        list:
-        - default_value
-        - patch_value
-        x: default_value
-        y: patch_value
-        z: patch_value
-    - name: myname02
-      param01:
-      - 1
-      - 1
-      - 2
-      - 3
-      - 3
-      - 4
-      - 4
-      - key: value
+  list3:
+  - name: myname01
+    param01:
+      list:
+      - default_value
+      - patch_value
+      x: default_value
+      y: patch_value
+      z: patch_value
+  - name: myname02
+    param01:
+    - 1
+    - 1
+    - 2
+    - 3
+    - 3
+    - 4
+    - 4
+    - key: value
 
 Example ``list_merge=prepend``:
 
 .. code-block:: yaml+jinja
 
-  ---
-  - name: Merge recursive by 'name', prepend lists
-    set_fact:
-      list3: "{{ [list1, list2]|
-                 community.general.lists_mergeby('name',
-                                                 recursive=true,
-                                                 list_merge='prepend') }}"
-    vars:
-      list1:
-        - name: myname01
-          param01:
-            x: default_value
-            y: default_value
-            list:
-              - default_value
-        - name: myname02
-          param01: [1, 1, 2, 3]
-
-      list2:
-        - name: myname01
-          param01:
-            y: patch_value
-            z: patch_value
-            list:
-              - patch_value
-        - name: myname02
-          param01: [3, 4, 4, {key: value}]
-  - debug:
-      var: list3
+  list3: "{{ [list1, list2]|
+             community.general.lists_mergeby('name',
+                                             recursive=true,
+                                             list_merge='prepend') }}"
 
 This produces:
 
 .. code-block:: yaml
 
-    list3:
-    - name: myname01
-      param01:
-        list:
-        - patch_value
-        - default_value
-        x: default_value
-        y: patch_value
-        z: patch_value
-    - name: myname02
-      param01:
-      - 3
-      - 4
-      - 4
-      - key: value
-      - 1
-      - 1
-      - 2
-      - 3
+  list3:
+  - name: myname01
+    param01:
+      list:
+      - patch_value
+      - default_value
+      x: default_value
+      y: patch_value
+      z: patch_value
+  - name: myname02
+    param01:
+    - 3
+    - 4
+    - 4
+    - key: value
+    - 1
+    - 1
+    - 2
+    - 3
 
 Example ``list_merge=append_rp``:
 
 .. code-block:: yaml+jinja
 
-  ---
-  - name: Merge recursive by 'name', append lists 'remove present'
-    set_fact:
-      list3: "{{ [list1, list2]|
-                 community.general.lists_mergeby('name',
-                                                 recursive=true,
-                                                 list_merge='append_rp') }}"
-    vars:
-      list1:
-        - name: myname01
-          param01:
-            x: default_value
-            y: default_value
-            list:
-              - default_value
-        - name: myname02
-          param01: [1, 1, 2, 3]
-
-      list2:
-        - name: myname01
-          param01:
-            y: patch_value
-            z: patch_value
-            list:
-              - patch_value
-        - name: myname02
-          param01: [3, 4, 4, {key: value}]
-  - debug:
-      var: list3
+  list3: "{{ [list1, list2]|
+             community.general.lists_mergeby('name',
+                                             recursive=true,
+                                             list_merge='append_rp') }}"
 
 This produces:
 
 .. code-block:: yaml
 
-    list3:
-    - name: myname01
-      param01:
-        list:
-        - default_value
-        - patch_value
-        x: default_value
-        y: patch_value
-        z: patch_value
-    - name: myname02
-      param01:
-      - 1
-      - 1
-      - 2
-      - 3
-      - 4
-      - 4
-      - key: value
+  list3:
+  - name: myname01
+    param01:
+      list:
+      - default_value
+      - patch_value
+      x: default_value
+      y: patch_value
+      z: patch_value
+  - name: myname02
+    param01:
+    - 1
+    - 1
+    - 2
+    - 3
+    - 4
+    - 4
+    - key: value
 
 Example ``list_merge=prepend_rp``:
 
 .. code-block:: yaml+jinja
 
-  ---
-  - name: Merge recursive by 'name', prepend lists 'remove present'
-    set_fact:
-      list3: "{{ [list1, list2]|
-                 community.general.lists_mergeby('name',
-                                                 recursive=true,
-                                                 list_merge='prepend_rp') }}"
-    vars:
-      list1:
-        - name: myname01
-          param01:
-            x: default_value
-            y: default_value
-            list:
-              - default_value
-        - name: myname02
-          param01: [1, 1, 2, 3]
-
-      list2:
-        - name: myname01
-          param01:
-            y: patch_value
-            z: patch_value
-            list:
-              - patch_value
-        - name: myname02
-          param01: [3, 4, 4, {key: value}]
-  - debug:
-      var: list3
+  list3: "{{ [list1, list2]|
+             community.general.lists_mergeby('name',
+                                             recursive=true,
+                                             list_merge='prepend_rp') }}"
 
 This produces:
 
 .. code-block:: yaml
 
-    list3:
-    - name: myname01
-      param01:
-        list:
-        - patch_value
-        - default_value
-        x: default_value
-        y: patch_value
-        z: patch_value
-    - name: myname02
-      param01:
-      - 3
-      - 4
-      - 4
-      - key: value
-      - 1
-      - 1
-      - 2
+  list3:
+  - name: myname01
+    param01:
+      list:
+      - patch_value
+      - default_value
+      x: default_value
+      y: patch_value
+      z: patch_value
+  - name: myname02
+    param01:
+    - 3
+    - 4
+    - 4
+    - key: value
+    - 1
+    - 1
+    - 2
+

galaxy.yml

@@ -1,6 +1,6 @@
 namespace: community
 name: general
-version: 4.4.0
+version: 4.5.0
 readme: README.md
 authors:
   - Ansible (https://github.com/ansible)

plugins/callback/mail.py

@@ -120,27 +120,34 @@ class CallbackModule(CallbackBase):
 
         smtp = smtplib.SMTP(self.smtphost, port=self.smtpport)
 
-        content = 'Date: %s\n' % email.utils.formatdate()
-        content += 'From: %s\n' % self.sender
+        sender_address = email.utils.parseaddr(self.sender)
         if self.to:
-            content += 'To: %s\n' % ','.join(self.to)
+            to_addresses = email.utils.getaddresses(self.to)
         if self.cc:
-            content += 'Cc: %s\n' % ','.join(self.cc)
+            cc_addresses = email.utils.getaddresses(self.cc)
+        if self.bcc:
+            bcc_addresses = email.utils.getaddresses(self.bcc)
+
+        content = 'Date: %s\n' % email.utils.formatdate()
+        content += 'From: %s\n' % email.utils.formataddr(sender_address)
+        if self.to:
+            content += 'To: %s\n' % ', '.join([email.utils.formataddr(pair) for pair in to_addresses])
+        if self.cc:
+            content += 'Cc: %s\n' % ', '.join([email.utils.formataddr(pair) for pair in cc_addresses])
         content += 'Message-ID: %s\n' % email.utils.make_msgid()
         content += 'Subject: %s\n\n' % subject.strip()
         content += body
 
-        addresses = self.to
+        addresses = to_addresses
         if self.cc:
-            addresses += self.cc
+            addresses += cc_addresses
         if self.bcc:
-            addresses += self.bcc
+            addresses += bcc_addresses
 
         if not addresses:
             self._display.warning('No receiver has been specified for the mail callback plugin.')
 
-        for address in addresses:
-            smtp.sendmail(self.sender, address, to_bytes(content))
+        smtp.sendmail(self.sender, [address for name, address in addresses], to_bytes(content))
 
         smtp.quit()
 

plugins/callback/opentelemetry.py

@@ -319,7 +319,7 @@ class OpenTelemetrySource(object):
     @staticmethod
     def url_from_args(args):
         # the order matters
-        url_args = ("url", "api_url", "baseurl", "repo", "server_url", "chart_repo_url", "registry_url")
+        url_args = ("url", "api_url", "baseurl", "repo", "server_url", "chart_repo_url", "registry_url", "endpoint", "uri", "updates_url")
         for arg in url_args:
             if args is not None and args.get(arg):
                 return args.get(arg)

plugins/doc_fragments/keycloak.py

@@ -61,4 +61,11 @@ options:
             - Verify TLS certificates (do not disable this in production).
         type: bool
         default: yes
+
+    connection_timeout:
+        description:
+            - Controls the HTTP connections timeout period (in seconds) to Keycloak API.
+        type: int
+        default: 10
+        version_added: 4.5.0
 '''

plugins/inventory/linode.py

@@ -21,7 +21,18 @@ DOCUMENTATION = r'''
           Linode) and not tags.
     extends_documentation_fragment:
         - constructed
+        - inventory_cache
     options:
+        cache:
+            version_added: 4.5.0
+        cache_plugin:
+            version_added: 4.5.0
+        cache_timeout:
+            version_added: 4.5.0
+        cache_connection:
+            version_added: 4.5.0
+        cache_prefix:
+            version_added: 4.5.0
         plugin:
             description: Marks this as an instance of the 'linode' plugin.
             required: true
@@ -110,19 +121,20 @@ import os
 
 from ansible.errors import AnsibleError, AnsibleParserError
 from ansible.module_utils.six import string_types
-from ansible.plugins.inventory import BaseInventoryPlugin, Constructable
+from ansible.plugins.inventory import BaseInventoryPlugin, Constructable, Cacheable
 from ansible.template import Templar
 
 
 try:
     from linode_api4 import LinodeClient
+    from linode_api4.objects.linode import Instance
     from linode_api4.errors import ApiError as LinodeApiError
     HAS_LINODE = True
 except ImportError:
     HAS_LINODE = False
 
 
-class InventoryModule(BaseInventoryPlugin, Constructable):
+class InventoryModule(BaseInventoryPlugin, Constructable, Cacheable):
 
     NAME = 'community.general.linode'
 
@@ -282,26 +294,10 @@ class InventoryModule(BaseInventoryPlugin, Constructable):
 
         return regions, types, tags
 
-    def verify_file(self, path):
-        """Verify the Linode configuration file."""
-        if super(InventoryModule, self).verify_file(path):
-            endings = ('linode.yaml', 'linode.yml')
-            if any((path.endswith(ending) for ending in endings)):
-                return True
-        return False
-
-    def parse(self, inventory, loader, path, cache=True):
-        """Dynamically parse Linode the cloud inventory."""
-        super(InventoryModule, self).parse(inventory, loader, path)
-
-        if not HAS_LINODE:
-            raise AnsibleError('the Linode dynamic inventory plugin requires linode_api4.')
-
-        config_data = self._read_config_data(path)
-        self._build_client(loader)
-
-        self._get_instances_inventory()
+    def _cacheable_inventory(self):
+        return [i._raw_json for i in self.instances]
 
+    def populate(self, config_data):
         strict = self.get_option('strict')
         regions, types, tags = self._get_query_options(config_data)
         self._filter_by_config(regions, types, tags)
@@ -326,3 +322,45 @@ class InventoryModule(BaseInventoryPlugin, Constructable):
                 variables,
                 instance.label,
                 strict=strict)
+
+    def verify_file(self, path):
+        """Verify the Linode configuration file."""
+        if super(InventoryModule, self).verify_file(path):
+            endings = ('linode.yaml', 'linode.yml')
+            if any((path.endswith(ending) for ending in endings)):
+                return True
+        return False
+
+    def parse(self, inventory, loader, path, cache=True):
+        """Dynamically parse Linode the cloud inventory."""
+        super(InventoryModule, self).parse(inventory, loader, path)
+        self.instances = None
+
+        if not HAS_LINODE:
+            raise AnsibleError('the Linode dynamic inventory plugin requires linode_api4.')
+
+        config_data = self._read_config_data(path)
+        self._consume_options(config_data)
+
+        cache_key = self.get_cache_key(path)
+
+        if cache:
+            cache = self.get_option('cache')
+
+        update_cache = False
+        if cache:
+            try:
+                self.instances = [Instance(None, i["id"], i) for i in self._cache[cache_key]]
+            except KeyError:
+                update_cache = True
+
+        # Check for None rather than False in order to allow
+        # for empty sets of cached instances
+        if self.instances is None:
+            self._build_client(loader)
+            self._get_instances_inventory()
+
+        if update_cache:
+            self._cache[cache_key] = self._cacheable_inventory()
+
+        self.populate(config_data)

plugins/inventory/opennebula.py

@@ -95,7 +95,7 @@ except ImportError:
 
 from ansible.errors import AnsibleError
 from ansible.plugins.inventory import BaseInventoryPlugin, Constructable
-from ansible.module_utils._text import to_native
+from ansible.module_utils.common.text.converters import to_native
 
 from collections import namedtuple
 import os

plugins/inventory/proxmox.py

@@ -114,6 +114,22 @@ groups:
   mailservers: "'mail' in (proxmox_tags_parsed|list)"
 compose:
   ansible_port: 2222
+
+# Using the inventory to allow ansible to connect via the first IP address of the VM / Container
+# (Default is connection by name of QEMU/LXC guests)
+# Note: my_inv_var demonstrates how to add a string variable to every host used by the inventory.
+# my.proxmox.yml
+plugin: community.general.proxmox
+url: http://pve.domain.com:8006
+user: ansible@pve
+password: secure
+validate_certs: false
+want_facts: true
+compose:
+  ansible_host: proxmox_ipconfig0.ip | default(proxmox_net0.ip) | ipaddr('address')
+  my_inv_var_1: "'my_var1_value'"
+  my_inv_var_2: >
+    "my_var_2_value"
 '''
 
 import re

plugins/lookup/passwordstore.py

@@ -14,6 +14,8 @@ DOCUMENTATION = '''
     description:
       - Enables Ansible to retrieve, create or update passwords from the passwordstore.org pass utility.
         It also retrieves YAML style keys stored as multilines in the passwordfile.
+      - To avoid problems when accessing multiple secrets at once, add C(auto-expand-secmem) to
+        C(~/.gnupg/gpg-agent.conf). Where this is not possible, consider using I(lock=readwrite) instead.
     options:
       _terms:
         description: query key.
@@ -77,54 +79,89 @@ DOCUMENTATION = '''
           - warn
           - empty
           - create
+      lock:
+        description:
+          - How to synchronize operations.
+          - The default of C(write) only synchronizes write operations.
+          - C(readwrite) synchronizes all operations (including read). This makes sure that gpg-agent is never called in parallel.
+          - C(none) does not do any synchronization.
+        ini:
+          - section: passwordstore_lookup
+            key: lock
+        type: str
+        default: write
+        choices:
+          - readwrite
+          - write
+          - none
+        version_added: 4.5.0
+      locktimeout:
+        description:
+          - Lock timeout applied when I(lock) is not C(none).
+          - Time with a unit suffix, C(s), C(m), C(h) for seconds, minutes, and hours, respectively. For example, C(900s) equals C(15m).
+          - Correlates with C(pinentry-timeout) in C(~/.gnupg/gpg-agent.conf), see C(man gpg-agent) for details.
+        ini:
+          - section: passwordstore_lookup
+            key: locktimeout
+        type: str
+        default: 15m
+        version_added: 4.5.0
 '''
 EXAMPLES = """
-# Debug is used for examples, BAD IDEA to show passwords on screen
-- name: Basic lookup. Fails if example/test doesn't exist
-  ansible.builtin.debug:
-    msg: "{{ lookup('community.general.passwordstore', 'example/test')}}"
+ansible.cfg: |
+  [passwordstore_lookup]
+  lock=readwrite
+  locktimeout=45s
 
-- name: Basic lookup. Warns if example/test does not exist and returns empty string
-  ansible.builtin.debug:
-    msg: "{{ lookup('community.general.passwordstore', 'example/test missing=warn')}}"
+playbook.yml: |
+  ---
 
-- name: Create pass with random 16 character password. If password exists just give the password
-  ansible.builtin.debug:
-    var: mypassword
-  vars:
-    mypassword: "{{ lookup('community.general.passwordstore', 'example/test create=true')}}"
+  # Debug is used for examples, BAD IDEA to show passwords on screen
+  - name: Basic lookup. Fails if example/test does not exist
+    ansible.builtin.debug:
+      msg: "{{ lookup('community.general.passwordstore', 'example/test')}}"
 
-- name: Create pass with random 16 character password. If password exists just give the password
-  ansible.builtin.debug:
-    var: mypassword
-  vars:
-    mypassword: "{{ lookup('community.general.passwordstore', 'example/test missing=create')}}"
+  - name: Basic lookup. Warns if example/test does not exist and returns empty string
+    ansible.builtin.debug:
+      msg: "{{ lookup('community.general.passwordstore', 'example/test missing=warn')}}"
 
-- name: Prints 'abc' if example/test does not exist, just give the password otherwise
-  ansible.builtin.debug:
-    var: mypassword
-  vars:
-    mypassword: "{{ lookup('community.general.passwordstore', 'example/test missing=empty') | default('abc', true) }}"
+  - name: Create pass with random 16 character password. If password exists just give the password
+    ansible.builtin.debug:
+      var: mypassword
+    vars:
+      mypassword: "{{ lookup('community.general.passwordstore', 'example/test create=true')}}"
 
-- name: Different size password
-  ansible.builtin.debug:
-    msg: "{{ lookup('community.general.passwordstore', 'example/test create=true length=42')}}"
+  - name: Create pass with random 16 character password. If password exists just give the password
+    ansible.builtin.debug:
+      var: mypassword
+    vars:
+      mypassword: "{{ lookup('community.general.passwordstore', 'example/test missing=create')}}"
 
-- name: Create password and overwrite the password if it exists. As a bonus, this module includes the old password inside the pass file
-  ansible.builtin.debug:
-    msg: "{{ lookup('community.general.passwordstore', 'example/test create=true overwrite=true')}}"
+  - name: Prints 'abc' if example/test does not exist, just give the password otherwise
+    ansible.builtin.debug:
+      var: mypassword
+    vars:
+      mypassword: "{{ lookup('community.general.passwordstore', 'example/test missing=empty') | default('abc', true) }}"
 
-- name: Create an alphanumeric password
-  ansible.builtin.debug:
-    msg: "{{ lookup('community.general.passwordstore', 'example/test create=true nosymbols=true') }}"
+  - name: Different size password
+    ansible.builtin.debug:
+      msg: "{{ lookup('community.general.passwordstore', 'example/test create=true length=42')}}"
 
-- name: Return the value for user in the KV pair user, username
-  ansible.builtin.debug:
-    msg: "{{ lookup('community.general.passwordstore', 'example/test subkey=user')}}"
+  - name: Create password and overwrite the password if it exists. As a bonus, this module includes the old password inside the pass file
+    ansible.builtin.debug:
+      msg: "{{ lookup('community.general.passwordstore', 'example/test create=true overwrite=true')}}"
 
-- name: Return the entire password file content
-  ansible.builtin.set_fact:
-    passfilecontent: "{{ lookup('community.general.passwordstore', 'example/test returnall=true')}}"
+  - name: Create an alphanumeric password
+    ansible.builtin.debug:
+      msg: "{{ lookup('community.general.passwordstore', 'example/test create=true nosymbols=true') }}"
+
+  - name: Return the value for user in the KV pair user, username
+    ansible.builtin.debug:
+      msg: "{{ lookup('community.general.passwordstore', 'example/test subkey=user')}}"
+
+  - name: Return the entire password file content
+    ansible.builtin.set_fact:
+      passfilecontent: "{{ lookup('community.general.passwordstore', 'example/test returnall=true')}}"
 """
 
 RETURN = """
@@ -135,13 +172,15 @@ _raw:
   elements: str
 """
 
+from contextlib import contextmanager
 import os
+import re
 import subprocess
 import time
 import yaml
 
-
 from ansible.errors import AnsibleError, AnsibleAssertionError
+from ansible.module_utils.common.file import FileLock
 from ansible.module_utils.common.text.converters import to_bytes, to_native, to_text
 from ansible.module_utils.parsing.convert_bool import boolean
 from ansible.utils.display import Display
@@ -154,6 +193,7 @@ display = Display()
 
 # backhacked check_output with input for python 2.7
 # http://stackoverflow.com/questions/10103551/passing-data-to-subprocess-check-output
+# note: contains special logic for calling 'pass', so not a drop-in replacement for check_output
 def check_output2(*popenargs, **kwargs):
     if 'stdout' in kwargs:
         raise ValueError('stdout argument not allowed, it will be overridden.')
@@ -175,9 +215,10 @@ def check_output2(*popenargs, **kwargs):
         process.wait()
         raise
     retcode = process.poll()
-    if retcode != 0 or \
-            b'encryption failed: Unusable public key' in b_out or \
-            b'encryption failed: Unusable public key' in b_err:
+    if retcode == 0 and (b'encryption failed: Unusable public key' in b_out or
+                         b'encryption failed: Unusable public key' in b_err):
+        retcode = 78  # os.EX_CONFIG
+    if retcode != 0:
         cmd = kwargs.get("args")
         if cmd is None:
             cmd = popenargs[0]
@@ -227,13 +268,13 @@ class LookupModule(LookupBase):
 
             # Collect pass environment variables from the plugin's parameters.
             self.env = os.environ.copy()
+            self.env['LANGUAGE'] = 'C'  # make sure to get errors in English as required by check_output2
 
-            # Set PASSWORD_STORE_DIR if directory is set
-            if self.paramvals['directory']:
-                if os.path.isdir(self.paramvals['directory']):
-                    self.env['PASSWORD_STORE_DIR'] = self.paramvals['directory']
-                else:
-                    raise AnsibleError('Passwordstore directory \'{0}\' does not exist'.format(self.paramvals['directory']))
+            # Set PASSWORD_STORE_DIR
+            if os.path.isdir(self.paramvals['directory']):
+                self.env['PASSWORD_STORE_DIR'] = self.paramvals['directory']
+            else:
+                raise AnsibleError('Passwordstore directory \'{0}\' does not exist'.format(self.paramvals['directory']))
 
             # Set PASSWORD_STORE_UMASK if umask is set
             if 'umask' in self.paramvals:
@@ -261,19 +302,20 @@ class LookupModule(LookupBase):
                     if ':' in line:
                         name, value = line.split(':', 1)
                         self.passdict[name.strip()] = value.strip()
+            if os.path.isfile(os.path.join(self.paramvals['directory'], self.passname + ".gpg")):
+                # Only accept password as found, if there a .gpg file for it (might be a tree node otherwise)
+                return True
         except (subprocess.CalledProcessError) as e:
-            if e.returncode != 0 and 'not in the password store' in e.output:
-                # if pass returns 1 and return string contains 'is not in the password store.'
-                # We need to determine if this is valid or Error.
-                if self.paramvals['missing'] == 'error':
-                    raise AnsibleError('passwordstore: passname {0} not found and missing=error is set'.format(self.passname))
-                else:
-                    if self.paramvals['missing'] == 'warn':
-                        display.warning('passwordstore: passname {0} not found'.format(self.passname))
-                    return False
-            else:
+            # 'not in password store' is the expected error if a password wasn't found
+            if 'not in the password store' not in e.output:
                 raise AnsibleError(e)
-        return True
+
+        if self.paramvals['missing'] == 'error':
+            raise AnsibleError('passwordstore: passname {0} not found and missing=error is set'.format(self.passname))
+        elif self.paramvals['missing'] == 'warn':
+            display.warning('passwordstore: passname {0} not found'.format(self.passname))
+
+        return False
 
     def get_newpass(self):
         if self.paramvals['nosymbols']:
@@ -325,11 +367,30 @@ class LookupModule(LookupBase):
             else:
                 return None
 
-    def run(self, terms, variables, **kwargs):
-        result = []
+    @contextmanager
+    def opt_lock(self, type):
+        if self.get_option('lock') == type:
+            tmpdir = os.environ.get('TMPDIR', '/tmp')
+            lockfile = os.path.join(tmpdir, '.passwordstore.lock')
+            with FileLock().lock_file(lockfile, tmpdir, self.lock_timeout):
+                self.locked = type
+                yield
+            self.locked = None
+        else:
+            yield
+
+    def setup(self, variables):
+        self.locked = None
+        timeout = self.get_option('locktimeout')
+        if not re.match('^[0-9]+[smh]$', timeout):
+            raise AnsibleError("{0} is not a correct value for locktimeout".format(timeout))
+        unit_to_seconds = {"s": 1, "m": 60, "h": 3600}
+        self.lock_timeout = int(timeout[:-1]) * unit_to_seconds[timeout[-1]]
         self.paramvals = {
             'subkey': 'password',
-            'directory': variables.get('passwordstore'),
+            'directory': variables.get('passwordstore', os.environ.get(
+                                       'PASSWORD_STORE_DIR',
+                                       os.path.expanduser('~/.password-store'))),
             'create': False,
             'returnall': False,
             'overwrite': False,
@@ -340,17 +401,27 @@ class LookupModule(LookupBase):
             'missing': 'error',
         }
 
+    def run(self, terms, variables, **kwargs):
+        self.setup(variables)
+        result = []
+
         for term in terms:
             self.parse_params(term)   # parse the input into paramvals
-            if self.check_pass():     # password exists
-                if self.paramvals['overwrite'] and self.paramvals['subkey'] == 'password':
-                    result.append(self.update_password())
-                else:
-                    result.append(self.get_passresult())
-            else:                     # password does not exist
-                if self.paramvals['missing'] == 'create':
-                    result.append(self.generate_password())
-                else:
-                    result.append(None)
+            with self.opt_lock('readwrite'):
+                if self.check_pass():     # password exists
+                    if self.paramvals['overwrite'] and self.paramvals['subkey'] == 'password':
+                        with self.opt_lock('write'):
+                            result.append(self.update_password())
+                    else:
+                        result.append(self.get_passresult())
+                else:                     # password does not exist
+                    if self.paramvals['missing'] == 'create':
+                        with self.opt_lock('write'):
+                            if self.locked == 'write' and self.check_pass():  # lookup password again if under write lock
+                                result.append(self.get_passresult())
+                            else:
+                                result.append(self.generate_password())
+                    else:
+                        result.append(None)
 
         return result

plugins/module_utils/identity/keycloak/keycloak.py

@@ -102,6 +102,7 @@ def keycloak_argument_spec():
         auth_username=dict(type='str', aliases=['username']),
         auth_password=dict(type='str', aliases=['password'], no_log=True),
         validate_certs=dict(type='bool', default=True),
+        connection_timeout=dict(type='int', default=10),
         token=dict(type='str', no_log=True),
     )
 
@@ -134,6 +135,7 @@ def get_token(module_params):
         auth_username = module_params.get('auth_username')
         auth_password = module_params.get('auth_password')
         client_secret = module_params.get('auth_client_secret')
+        connection_timeout = module_params.get('connection_timeout')
         auth_url = URL_TOKEN.format(url=base_url, realm=auth_realm)
         temp_payload = {
             'grant_type': 'password',
@@ -147,7 +149,7 @@ def get_token(module_params):
             (k, v) for k, v in temp_payload.items() if v is not None)
         try:
             r = json.loads(to_native(open_url(auth_url, method='POST',
-                                              validate_certs=validate_certs,
+                                              validate_certs=validate_certs, timeout=connection_timeout,
                                               data=urlencode(payload)).read()))
         except ValueError as e:
             raise KeycloakError(
@@ -229,6 +231,7 @@ class KeycloakAPI(object):
         self.module = module
         self.baseurl = self.module.params.get('auth_keycloak_url')
         self.validate_certs = self.module.params.get('validate_certs')
+        self.connection_timeout = self.module.params.get('connection_timeout')
         self.restheaders = connection_header
 
     def get_realm_info_by_id(self, realm='master'):
@@ -240,7 +243,7 @@ class KeycloakAPI(object):
         realm_info_url = URL_REALM_INFO.format(url=self.baseurl, realm=realm)
 
         try:
-            return json.loads(to_native(open_url(realm_info_url, method='GET', headers=self.restheaders,
+            return json.loads(to_native(open_url(realm_info_url, method='GET', headers=self.restheaders, timeout=self.connection_timeout,
                                                  validate_certs=self.validate_certs).read()))
 
         except HTTPError as e:
@@ -265,7 +268,7 @@ class KeycloakAPI(object):
         realm_url = URL_REALM.format(url=self.baseurl, realm=realm)
 
         try:
-            return json.loads(to_native(open_url(realm_url, method='GET', headers=self.restheaders,
+            return json.loads(to_native(open_url(realm_url, method='GET', headers=self.restheaders, timeout=self.connection_timeout,
                                                  validate_certs=self.validate_certs).read()))
 
         except HTTPError as e:
@@ -290,7 +293,7 @@ class KeycloakAPI(object):
         realm_url = URL_REALM.format(url=self.baseurl, realm=realm)
 
         try:
-            return open_url(realm_url, method='PUT', headers=self.restheaders,
+            return open_url(realm_url, method='PUT', headers=self.restheaders, timeout=self.connection_timeout,
                             data=json.dumps(realmrep), validate_certs=self.validate_certs)
         except Exception as e:
             self.module.fail_json(msg='Could not update realm %s: %s' % (realm, str(e)),
@@ -304,7 +307,7 @@ class KeycloakAPI(object):
         realm_url = URL_REALMS.format(url=self.baseurl)
 
         try:
-            return open_url(realm_url, method='POST', headers=self.restheaders,
+            return open_url(realm_url, method='POST', headers=self.restheaders, timeout=self.connection_timeout,
                             data=json.dumps(realmrep), validate_certs=self.validate_certs)
         except Exception as e:
             self.module.fail_json(msg='Could not create realm %s: %s' % (realmrep['id'], str(e)),
@@ -319,7 +322,7 @@ class KeycloakAPI(object):
         realm_url = URL_REALM.format(url=self.baseurl, realm=realm)
 
         try:
-            return open_url(realm_url, method='DELETE', headers=self.restheaders,
+            return open_url(realm_url, method='DELETE', headers=self.restheaders, timeout=self.connection_timeout,
                             validate_certs=self.validate_certs)
         except Exception as e:
             self.module.fail_json(msg='Could not delete realm %s: %s' % (realm, str(e)),
@@ -337,7 +340,7 @@ class KeycloakAPI(object):
             clientlist_url += '?clientId=%s' % filter
 
         try:
-            return json.loads(to_native(open_url(clientlist_url, method='GET', headers=self.restheaders,
+            return json.loads(to_native(open_url(clientlist_url, method='GET', headers=self.restheaders, timeout=self.connection_timeout,
                                                  validate_certs=self.validate_certs).read()))
         except ValueError as e:
             self.module.fail_json(msg='API returned incorrect JSON when trying to obtain list of clients for realm %s: %s'
@@ -368,7 +371,7 @@ class KeycloakAPI(object):
         client_url = URL_CLIENT.format(url=self.baseurl, realm=realm, id=id)
 
         try:
-            return json.loads(to_native(open_url(client_url, method='GET', headers=self.restheaders,
+            return json.loads(to_native(open_url(client_url, method='GET', headers=self.restheaders, timeout=self.connection_timeout,
                                                  validate_certs=self.validate_certs).read()))
 
         except HTTPError as e:
@@ -407,7 +410,7 @@ class KeycloakAPI(object):
         client_url = URL_CLIENT.format(url=self.baseurl, realm=realm, id=id)
 
         try:
-            return open_url(client_url, method='PUT', headers=self.restheaders,
+            return open_url(client_url, method='PUT', headers=self.restheaders, timeout=self.connection_timeout,
                             data=json.dumps(clientrep), validate_certs=self.validate_certs)
         except Exception as e:
             self.module.fail_json(msg='Could not update client %s in realm %s: %s'
@@ -422,7 +425,7 @@ class KeycloakAPI(object):
         client_url = URL_CLIENTS.format(url=self.baseurl, realm=realm)
 
         try:
-            return open_url(client_url, method='POST', headers=self.restheaders,
+            return open_url(client_url, method='POST', headers=self.restheaders, timeout=self.connection_timeout,
                             data=json.dumps(clientrep), validate_certs=self.validate_certs)
         except Exception as e:
             self.module.fail_json(msg='Could not create client %s in realm %s: %s'
@@ -438,7 +441,7 @@ class KeycloakAPI(object):
         client_url = URL_CLIENT.format(url=self.baseurl, realm=realm, id=id)
 
         try:
-            return open_url(client_url, method='DELETE', headers=self.restheaders,
+            return open_url(client_url, method='DELETE', headers=self.restheaders, timeout=self.connection_timeout,
                             validate_certs=self.validate_certs)
         except Exception as e:
             self.module.fail_json(msg='Could not delete client %s in realm %s: %s'
@@ -453,7 +456,7 @@ class KeycloakAPI(object):
         """
         client_roles_url = URL_CLIENT_ROLES.format(url=self.baseurl, realm=realm, id=cid)
         try:
-            return json.loads(to_native(open_url(client_roles_url, method="GET", headers=self.restheaders,
+            return json.loads(to_native(open_url(client_roles_url, method="GET", headers=self.restheaders, timeout=self.connection_timeout,
                                                  validate_certs=self.validate_certs).read()))
         except Exception as e:
             self.module.fail_json(msg="Could not fetch rolemappings for client %s in realm %s: %s"
@@ -485,7 +488,7 @@ class KeycloakAPI(object):
         """
         rolemappings_url = URL_CLIENT_ROLEMAPPINGS.format(url=self.baseurl, realm=realm, id=gid, client=cid)
         try:
-            rolemappings = json.loads(to_native(open_url(rolemappings_url, method="GET", headers=self.restheaders,
+            rolemappings = json.loads(to_native(open_url(rolemappings_url, method="GET", headers=self.restheaders, timeout=self.connection_timeout,
                                                          validate_certs=self.validate_certs).read()))
             for role in rolemappings:
                 if rid == role['id']:
@@ -505,7 +508,7 @@ class KeycloakAPI(object):
         """
         available_rolemappings_url = URL_CLIENT_ROLEMAPPINGS_AVAILABLE.format(url=self.baseurl, realm=realm, id=gid, client=cid)
         try:
-            return json.loads(to_native(open_url(available_rolemappings_url, method="GET", headers=self.restheaders,
+            return json.loads(to_native(open_url(available_rolemappings_url, method="GET", headers=self.restheaders, timeout=self.connection_timeout,
                                                  validate_certs=self.validate_certs).read()))
         except Exception as e:
             self.module.fail_json(msg="Could not fetch available rolemappings for client %s in group %s, realm %s: %s"
@@ -521,7 +524,7 @@ class KeycloakAPI(object):
         """
         available_rolemappings_url = URL_CLIENT_ROLEMAPPINGS_COMPOSITE.format(url=self.baseurl, realm=realm, id=gid, client=cid)
         try:
-            return json.loads(to_native(open_url(available_rolemappings_url, method="GET", headers=self.restheaders,
+            return json.loads(to_native(open_url(available_rolemappings_url, method="GET", headers=self.restheaders, timeout=self.connection_timeout,
                                                  validate_certs=self.validate_certs).read()))
         except Exception as e:
             self.module.fail_json(msg="Could not fetch available rolemappings for client %s in group %s, realm %s: %s"
@@ -538,7 +541,8 @@ class KeycloakAPI(object):
         """
         available_rolemappings_url = URL_CLIENT_ROLEMAPPINGS.format(url=self.baseurl, realm=realm, id=gid, client=cid)
         try:
-            open_url(available_rolemappings_url, method="POST", headers=self.restheaders, data=json.dumps(role_rep), validate_certs=self.validate_certs)
+            open_url(available_rolemappings_url, method="POST", headers=self.restheaders, data=json.dumps(role_rep),
+                     validate_certs=self.validate_certs, timeout=self.connection_timeout)
         except Exception as e:
             self.module.fail_json(msg="Could not fetch available rolemappings for client %s in group %s, realm %s: %s"
                                       % (cid, gid, realm, str(e)))
@@ -554,7 +558,8 @@ class KeycloakAPI(object):
         """
         available_rolemappings_url = URL_CLIENT_ROLEMAPPINGS.format(url=self.baseurl, realm=realm, id=gid, client=cid)
         try:
-            open_url(available_rolemappings_url, method="DELETE", headers=self.restheaders, validate_certs=self.validate_certs)
+            open_url(available_rolemappings_url, method="DELETE", headers=self.restheaders,
+                     validate_certs=self.validate_certs, timeout=self.connection_timeout)
         except Exception as e:
             self.module.fail_json(msg="Could not delete available rolemappings for client %s in group %s, realm %s: %s"
                                       % (cid, gid, realm, str(e)))
@@ -568,7 +573,7 @@ class KeycloakAPI(object):
         url = URL_CLIENTTEMPLATES.format(url=self.baseurl, realm=realm)
 
         try:
-            return json.loads(to_native(open_url(url, method='GET', headers=self.restheaders,
+            return json.loads(to_native(open_url(url, method='GET', headers=self.restheaders, timeout=self.connection_timeout,
                                                  validate_certs=self.validate_certs).read()))
         except ValueError as e:
             self.module.fail_json(msg='API returned incorrect JSON when trying to obtain list of client templates for realm %s: %s'
@@ -587,7 +592,7 @@ class KeycloakAPI(object):
         url = URL_CLIENTTEMPLATE.format(url=self.baseurl, id=id, realm=realm)
 
         try:
-            return json.loads(to_native(open_url(url, method='GET', headers=self.restheaders,
+            return json.loads(to_native(open_url(url, method='GET', headers=self.restheaders, timeout=self.connection_timeout,
                                                  validate_certs=self.validate_certs).read()))
         except ValueError as e:
             self.module.fail_json(msg='API returned incorrect JSON when trying to obtain client templates %s for realm %s: %s'
@@ -633,7 +638,7 @@ class KeycloakAPI(object):
         url = URL_CLIENTTEMPLATE.format(url=self.baseurl, realm=realm, id=id)
 
         try:
-            return open_url(url, method='PUT', headers=self.restheaders,
+            return open_url(url, method='PUT', headers=self.restheaders, timeout=self.connection_timeout,
                             data=json.dumps(clienttrep), validate_certs=self.validate_certs)
         except Exception as e:
             self.module.fail_json(msg='Could not update client template %s in realm %s: %s'
@@ -648,7 +653,7 @@ class KeycloakAPI(object):
         url = URL_CLIENTTEMPLATES.format(url=self.baseurl, realm=realm)
 
         try:
-            return open_url(url, method='POST', headers=self.restheaders,
+            return open_url(url, method='POST', headers=self.restheaders, timeout=self.connection_timeout,
                             data=json.dumps(clienttrep), validate_certs=self.validate_certs)
         except Exception as e:
             self.module.fail_json(msg='Could not create client template %s in realm %s: %s'
@@ -664,7 +669,7 @@ class KeycloakAPI(object):
         url = URL_CLIENTTEMPLATE.format(url=self.baseurl, realm=realm, id=id)
 
         try:
-            return open_url(url, method='DELETE', headers=self.restheaders,
+            return open_url(url, method='DELETE', headers=self.restheaders, timeout=self.connection_timeout,
                             validate_certs=self.validate_certs)
         except Exception as e:
             self.module.fail_json(msg='Could not delete client template %s in realm %s: %s'
@@ -681,7 +686,7 @@ class KeycloakAPI(object):
         """
         clientscopes_url = URL_CLIENTSCOPES.format(url=self.baseurl, realm=realm)
         try:
-            return json.loads(to_native(open_url(clientscopes_url, method="GET", headers=self.restheaders,
+            return json.loads(to_native(open_url(clientscopes_url, method="GET", headers=self.restheaders, timeout=self.connection_timeout,
                                                  validate_certs=self.validate_certs).read()))
         except Exception as e:
             self.module.fail_json(msg="Could not fetch list of clientscopes in realm %s: %s"
@@ -698,7 +703,7 @@ class KeycloakAPI(object):
         """
         clientscope_url = URL_CLIENTSCOPE.format(url=self.baseurl, realm=realm, id=cid)
         try:
-            return json.loads(to_native(open_url(clientscope_url, method="GET", headers=self.restheaders,
+            return json.loads(to_native(open_url(clientscope_url, method="GET", headers=self.restheaders, timeout=self.connection_timeout,
                                                  validate_certs=self.validate_certs).read()))
 
         except HTTPError as e:
@@ -743,7 +748,7 @@ class KeycloakAPI(object):
         """
         clientscopes_url = URL_CLIENTSCOPES.format(url=self.baseurl, realm=realm)
         try:
-            return open_url(clientscopes_url, method='POST', headers=self.restheaders,
+            return open_url(clientscopes_url, method='POST', headers=self.restheaders, timeout=self.connection_timeout,
                             data=json.dumps(clientscoperep), validate_certs=self.validate_certs)
         except Exception as e:
             self.module.fail_json(msg="Could not create clientscope %s in realm %s: %s"
@@ -758,7 +763,7 @@ class KeycloakAPI(object):
         clientscope_url = URL_CLIENTSCOPE.format(url=self.baseurl, realm=realm, id=clientscoperep['id'])
 
         try:
-            return open_url(clientscope_url, method='PUT', headers=self.restheaders,
+            return open_url(clientscope_url, method='PUT', headers=self.restheaders, timeout=self.connection_timeout,
                             data=json.dumps(clientscoperep), validate_certs=self.validate_certs)
 
         except Exception as e:
@@ -796,7 +801,7 @@ class KeycloakAPI(object):
         # should have a good cid by here.
         clientscope_url = URL_CLIENTSCOPE.format(realm=realm, id=cid, url=self.baseurl)
         try:
-            return open_url(clientscope_url, method='DELETE', headers=self.restheaders,
+            return open_url(clientscope_url, method='DELETE', headers=self.restheaders, timeout=self.connection_timeout,
                             validate_certs=self.validate_certs)
 
         except Exception as e:
@@ -814,7 +819,7 @@ class KeycloakAPI(object):
         """
         protocolmappers_url = URL_CLIENTSCOPE_PROTOCOLMAPPERS.format(id=cid, url=self.baseurl, realm=realm)
         try:
-            return json.loads(to_native(open_url(protocolmappers_url, method="GET", headers=self.restheaders,
+            return json.loads(to_native(open_url(protocolmappers_url, method="GET", headers=self.restheaders, timeout=self.connection_timeout,
                                                  validate_certs=self.validate_certs).read()))
         except Exception as e:
             self.module.fail_json(msg="Could not fetch list of protocolmappers in realm %s: %s"
@@ -833,7 +838,7 @@ class KeycloakAPI(object):
         """
         protocolmapper_url = URL_CLIENTSCOPE_PROTOCOLMAPPER.format(url=self.baseurl, realm=realm, id=cid, mapper_id=pid)
         try:
-            return json.loads(to_native(open_url(protocolmapper_url, method="GET", headers=self.restheaders,
+            return json.loads(to_native(open_url(protocolmapper_url, method="GET", headers=self.restheaders, timeout=self.connection_timeout,
                                                  validate_certs=self.validate_certs).read()))
 
         except HTTPError as e:
@@ -880,7 +885,7 @@ class KeycloakAPI(object):
         """
         protocolmappers_url = URL_CLIENTSCOPE_PROTOCOLMAPPERS.format(url=self.baseurl, id=cid, realm=realm)
         try:
-            return open_url(protocolmappers_url, method='POST', headers=self.restheaders,
+            return open_url(protocolmappers_url, method='POST', headers=self.restheaders, timeout=self.connection_timeout,
                             data=json.dumps(mapper_rep), validate_certs=self.validate_certs)
         except Exception as e:
             self.module.fail_json(msg="Could not create protocolmapper %s in realm %s: %s"
@@ -896,7 +901,7 @@ class KeycloakAPI(object):
         protocolmapper_url = URL_CLIENTSCOPE_PROTOCOLMAPPER.format(url=self.baseurl, realm=realm, id=cid, mapper_id=mapper_rep['id'])
 
         try:
-            return open_url(protocolmapper_url, method='PUT', headers=self.restheaders,
+            return open_url(protocolmapper_url, method='PUT', headers=self.restheaders, timeout=self.connection_timeout,
                             data=json.dumps(mapper_rep), validate_certs=self.validate_certs)
 
         except Exception as e:
@@ -913,7 +918,7 @@ class KeycloakAPI(object):
         """
         groups_url = URL_GROUPS.format(url=self.baseurl, realm=realm)
         try:
-            return json.loads(to_native(open_url(groups_url, method="GET", headers=self.restheaders,
+            return json.loads(to_native(open_url(groups_url, method="GET", headers=self.restheaders, timeout=self.connection_timeout,
                                                  validate_certs=self.validate_certs).read()))
         except Exception as e:
             self.module.fail_json(msg="Could not fetch list of groups in realm %s: %s"
@@ -930,7 +935,7 @@ class KeycloakAPI(object):
         """
         groups_url = URL_GROUP.format(url=self.baseurl, realm=realm, groupid=gid)
         try:
-            return json.loads(to_native(open_url(groups_url, method="GET", headers=self.restheaders,
+            return json.loads(to_native(open_url(groups_url, method="GET", headers=self.restheaders, timeout=self.connection_timeout,
                                                  validate_certs=self.validate_certs).read()))
 
         except HTTPError as e:
@@ -976,7 +981,7 @@ class KeycloakAPI(object):
         """
         groups_url = URL_GROUPS.format(url=self.baseurl, realm=realm)
         try:
-            return open_url(groups_url, method='POST', headers=self.restheaders,
+            return open_url(groups_url, method='POST', headers=self.restheaders, timeout=self.connection_timeout,
                             data=json.dumps(grouprep), validate_certs=self.validate_certs)
         except Exception as e:
             self.module.fail_json(msg="Could not create group %s in realm %s: %s"
@@ -991,7 +996,7 @@ class KeycloakAPI(object):
         group_url = URL_GROUP.format(url=self.baseurl, realm=realm, groupid=grouprep['id'])
 
         try:
-            return open_url(group_url, method='PUT', headers=self.restheaders,
+            return open_url(group_url, method='PUT', headers=self.restheaders, timeout=self.connection_timeout,
                             data=json.dumps(grouprep), validate_certs=self.validate_certs)
         except Exception as e:
             self.module.fail_json(msg='Could not update group %s in realm %s: %s'
@@ -1028,7 +1033,7 @@ class KeycloakAPI(object):
         # should have a good groupid by here.
         group_url = URL_GROUP.format(realm=realm, groupid=groupid, url=self.baseurl)
         try:
-            return open_url(group_url, method='DELETE', headers=self.restheaders,
+            return open_url(group_url, method='DELETE', headers=self.restheaders, timeout=self.connection_timeout,
                             validate_certs=self.validate_certs)
         except Exception as e:
             self.module.fail_json(msg="Unable to delete group %s: %s" % (groupid, str(e)))
@@ -1041,7 +1046,7 @@ class KeycloakAPI(object):
         """
         rolelist_url = URL_REALM_ROLES.format(url=self.baseurl, realm=realm)
         try:
-            return json.loads(to_native(open_url(rolelist_url, method='GET', headers=self.restheaders,
+            return json.loads(to_native(open_url(rolelist_url, method='GET', headers=self.restheaders, timeout=self.connection_timeout,
                                                  validate_certs=self.validate_certs).read()))
         except ValueError as e:
             self.module.fail_json(msg='API returned incorrect JSON when trying to obtain list of roles for realm %s: %s'
@@ -1059,7 +1064,7 @@ class KeycloakAPI(object):
         """
         role_url = URL_REALM_ROLE.format(url=self.baseurl, realm=realm, name=quote(name))
         try:
-            return json.loads(to_native(open_url(role_url, method="GET", headers=self.restheaders,
+            return json.loads(to_native(open_url(role_url, method="GET", headers=self.restheaders, timeout=self.connection_timeout,
                                                  validate_certs=self.validate_certs).read()))
         except HTTPError as e:
             if e.code == 404:
@@ -1079,7 +1084,7 @@ class KeycloakAPI(object):
         """
         roles_url = URL_REALM_ROLES.format(url=self.baseurl, realm=realm)
         try:
-            return open_url(roles_url, method='POST', headers=self.restheaders,
+            return open_url(roles_url, method='POST', headers=self.restheaders, timeout=self.connection_timeout,
                             data=json.dumps(rolerep), validate_certs=self.validate_certs)
         except Exception as e:
             self.module.fail_json(msg='Could not create role %s in realm %s: %s'
@@ -1093,7 +1098,7 @@ class KeycloakAPI(object):
         """
         role_url = URL_REALM_ROLE.format(url=self.baseurl, realm=realm, name=quote(rolerep['name']))
         try:
-            return open_url(role_url, method='PUT', headers=self.restheaders,
+            return open_url(role_url, method='PUT', headers=self.restheaders, timeout=self.connection_timeout,
                             data=json.dumps(rolerep), validate_certs=self.validate_certs)
         except Exception as e:
             self.module.fail_json(msg='Could not update role %s in realm %s: %s'
@@ -1107,7 +1112,7 @@ class KeycloakAPI(object):
         """
         role_url = URL_REALM_ROLE.format(url=self.baseurl, realm=realm, name=quote(name))
         try:
-            return open_url(role_url, method='DELETE', headers=self.restheaders,
+            return open_url(role_url, method='DELETE', headers=self.restheaders, timeout=self.connection_timeout,
                             validate_certs=self.validate_certs)
         except Exception as e:
             self.module.fail_json(msg='Unable to delete role %s in realm %s: %s'
@@ -1126,7 +1131,7 @@ class KeycloakAPI(object):
                                       % (clientid, realm))
         rolelist_url = URL_CLIENT_ROLES.format(url=self.baseurl, realm=realm, id=cid)
         try:
-            return json.loads(to_native(open_url(rolelist_url, method='GET', headers=self.restheaders,
+            return json.loads(to_native(open_url(rolelist_url, method='GET', headers=self.restheaders, timeout=self.connection_timeout,
                                                  validate_certs=self.validate_certs).read()))
         except ValueError as e:
             self.module.fail_json(msg='API returned incorrect JSON when trying to obtain list of roles for client %s in realm %s: %s'
@@ -1150,7 +1155,7 @@ class KeycloakAPI(object):
                                       % (clientid, realm))
         role_url = URL_CLIENT_ROLE.format(url=self.baseurl, realm=realm, id=cid, name=quote(name))
         try:
-            return json.loads(to_native(open_url(role_url, method="GET", headers=self.restheaders,
+            return json.loads(to_native(open_url(role_url, method="GET", headers=self.restheaders, timeout=self.connection_timeout,
                                                  validate_certs=self.validate_certs).read()))
         except HTTPError as e:
             if e.code == 404:
@@ -1176,7 +1181,7 @@ class KeycloakAPI(object):
                                       % (clientid, realm))
         roles_url = URL_CLIENT_ROLES.format(url=self.baseurl, realm=realm, id=cid)
         try:
-            return open_url(roles_url, method='POST', headers=self.restheaders,
+            return open_url(roles_url, method='POST', headers=self.restheaders, timeout=self.connection_timeout,
                             data=json.dumps(rolerep), validate_certs=self.validate_certs)
         except Exception as e:
             self.module.fail_json(msg='Could not create role %s for client %s in realm %s: %s'
@@ -1196,7 +1201,7 @@ class KeycloakAPI(object):
                                       % (clientid, realm))
         role_url = URL_CLIENT_ROLE.format(url=self.baseurl, realm=realm, id=cid, name=quote(rolerep['name']))
         try:
-            return open_url(role_url, method='PUT', headers=self.restheaders,
+            return open_url(role_url, method='PUT', headers=self.restheaders, timeout=self.connection_timeout,
                             data=json.dumps(rolerep), validate_certs=self.validate_certs)
         except Exception as e:
             self.module.fail_json(msg='Could not update role %s for client %s in realm %s: %s'
@@ -1215,7 +1220,7 @@ class KeycloakAPI(object):
                                       % (clientid, realm))
         role_url = URL_CLIENT_ROLE.format(url=self.baseurl, realm=realm, id=cid, name=quote(name))
         try:
-            return open_url(role_url, method='DELETE', headers=self.restheaders,
+            return open_url(role_url, method='DELETE', headers=self.restheaders, timeout=self.connection_timeout,
                             validate_certs=self.validate_certs)
         except Exception as e:
             self.module.fail_json(msg='Unable to delete role %s for client %s in realm %s: %s'
@@ -1231,7 +1236,8 @@ class KeycloakAPI(object):
         try:
             authentication_flow = {}
             # Check if the authentication flow exists on the Keycloak serveraders
-            authentications = json.load(open_url(URL_AUTHENTICATION_FLOWS.format(url=self.baseurl, realm=realm), method='GET', headers=self.restheaders))
+            authentications = json.load(open_url(URL_AUTHENTICATION_FLOWS.format(url=self.baseurl, realm=realm), method='GET',
+                                                 headers=self.restheaders, timeout=self.connection_timeout))
             for authentication in authentications:
                 if authentication["alias"] == alias:
                     authentication_flow = authentication
@@ -1250,7 +1256,7 @@ class KeycloakAPI(object):
         flow_url = URL_AUTHENTICATION_FLOW.format(url=self.baseurl, realm=realm, id=id)
 
         try:
-            return open_url(flow_url, method='DELETE', headers=self.restheaders,
+            return open_url(flow_url, method='DELETE', headers=self.restheaders, timeout=self.connection_timeout,
                             validate_certs=self.validate_certs)
         except Exception as e:
             self.module.fail_json(msg='Could not delete authentication flow %s in realm %s: %s'
@@ -1274,13 +1280,15 @@ class KeycloakAPI(object):
                     copyfrom=quote(config["copyFrom"])),
                 method='POST',
                 headers=self.restheaders,
-                data=json.dumps(new_name))
+                data=json.dumps(new_name),
+                timeout=self.connection_timeout)
             flow_list = json.load(
                 open_url(
                     URL_AUTHENTICATION_FLOWS.format(url=self.baseurl,
                                                     realm=realm),
                     method='GET',
-                    headers=self.restheaders))
+                    headers=self.restheaders,
+                    timeout=self.connection_timeout))
             for flow in flow_list:
                 if flow["alias"] == config["alias"]:
                     return flow
@@ -1309,14 +1317,16 @@ class KeycloakAPI(object):
                     realm=realm),
                 method='POST',
                 headers=self.restheaders,
-                data=json.dumps(new_flow))
+                data=json.dumps(new_flow),
+                timeout=self.connection_timeout)
             flow_list = json.load(
                 open_url(
                     URL_AUTHENTICATION_FLOWS.format(
                         url=self.baseurl,
                         realm=realm),
                     method='GET',
-                    headers=self.restheaders))
+                    headers=self.restheaders,
+                    timeout=self.connection_timeout))
             for flow in flow_list:
                 if flow["alias"] == config["alias"]:
                     return flow
@@ -1340,7 +1350,8 @@ class KeycloakAPI(object):
                     flowalias=quote(flowAlias)),
                 method='PUT',
                 headers=self.restheaders,
-                data=json.dumps(updatedExec))
+                data=json.dumps(updatedExec),
+                timeout=self.connection_timeout)
         except Exception as e:
             self.module.fail_json(msg="Unable to update executions %s: %s" % (updatedExec, str(e)))
 
@@ -1359,7 +1370,8 @@ class KeycloakAPI(object):
                     id=executionId),
                 method='POST',
                 headers=self.restheaders,
-                data=json.dumps(authenticationConfig))
+                data=json.dumps(authenticationConfig),
+                timeout=self.connection_timeout)
         except Exception as e:
             self.module.fail_json(msg="Unable to add authenticationConfig %s: %s" % (executionId, str(e)))
 
@@ -1382,7 +1394,8 @@ class KeycloakAPI(object):
                     flowalias=quote(flowAlias)),
                 method='POST',
                 headers=self.restheaders,
-                data=json.dumps(newSubFlow))
+                data=json.dumps(newSubFlow),
+                timeout=self.connection_timeout)
         except Exception as e:
             self.module.fail_json(msg="Unable to create new subflow %s: %s" % (subflowName, str(e)))
 
@@ -1404,7 +1417,8 @@ class KeycloakAPI(object):
                     flowalias=quote(flowAlias)),
                 method='POST',
                 headers=self.restheaders,
-                data=json.dumps(newExec))
+                data=json.dumps(newExec),
+                timeout=self.connection_timeout)
         except Exception as e:
             self.module.fail_json(msg="Unable to create new execution %s: %s" % (execution["provider"], str(e)))
 
@@ -1425,7 +1439,8 @@ class KeycloakAPI(object):
                             realm=realm,
                             id=executionId),
                         method='POST',
-                        headers=self.restheaders)
+                        headers=self.restheaders,
+                        timeout=self.connection_timeout)
             elif diff < 0:
                 for i in range(-diff):
                     open_url(
@@ -1434,7 +1449,8 @@ class KeycloakAPI(object):
                             realm=realm,
                             id=executionId),
                         method='POST',
-                        headers=self.restheaders)
+                        headers=self.restheaders,
+                        timeout=self.connection_timeout)
         except Exception as e:
             self.module.fail_json(msg="Unable to change execution priority %s: %s" % (executionId, str(e)))
 
@@ -1454,7 +1470,8 @@ class KeycloakAPI(object):
                         realm=realm,
                         flowalias=quote(config["alias"])),
                     method='GET',
-                    headers=self.restheaders))
+                    headers=self.restheaders,
+                    timeout=self.connection_timeout))
             for execution in executions:
                 if "authenticationConfig" in execution:
                     execConfigId = execution["authenticationConfig"]
@@ -1465,7 +1482,8 @@ class KeycloakAPI(object):
                                 realm=realm,
                                 id=execConfigId),
                             method='GET',
-                            headers=self.restheaders))
+                            headers=self.restheaders,
+                            timeout=self.connection_timeout))
                     execution["authenticationConfig"] = execConfig
             return executions
         except Exception as e:
@@ -1479,7 +1497,7 @@ class KeycloakAPI(object):
         """
         idps_url = URL_IDENTITY_PROVIDERS.format(url=self.baseurl, realm=realm)
         try:
-            return json.loads(to_native(open_url(idps_url, method='GET', headers=self.restheaders,
+            return json.loads(to_native(open_url(idps_url, method='GET', headers=self.restheaders, timeout=self.connection_timeout,
                                                  validate_certs=self.validate_certs).read()))
         except ValueError as e:
             self.module.fail_json(msg='API returned incorrect JSON when trying to obtain list of identity providers for realm %s: %s'
@@ -1496,7 +1514,7 @@ class KeycloakAPI(object):
         """
         idp_url = URL_IDENTITY_PROVIDER.format(url=self.baseurl, realm=realm, alias=alias)
         try:
-            return json.loads(to_native(open_url(idp_url, method="GET", headers=self.restheaders,
+            return json.loads(to_native(open_url(idp_url, method="GET", headers=self.restheaders, timeout=self.connection_timeout,
                                                  validate_certs=self.validate_certs).read()))
         except HTTPError as e:
             if e.code == 404:
@@ -1516,7 +1534,7 @@ class KeycloakAPI(object):
         """
         idps_url = URL_IDENTITY_PROVIDERS.format(url=self.baseurl, realm=realm)
         try:
-            return open_url(idps_url, method='POST', headers=self.restheaders,
+            return open_url(idps_url, method='POST', headers=self.restheaders, timeout=self.connection_timeout,
                             data=json.dumps(idprep), validate_certs=self.validate_certs)
         except Exception as e:
             self.module.fail_json(msg='Could not create identity provider %s in realm %s: %s'
@@ -1530,7 +1548,7 @@ class KeycloakAPI(object):
         """
         idp_url = URL_IDENTITY_PROVIDER.format(url=self.baseurl, realm=realm, alias=idprep['alias'])
         try:
-            return open_url(idp_url, method='PUT', headers=self.restheaders,
+            return open_url(idp_url, method='PUT', headers=self.restheaders, timeout=self.connection_timeout,
                             data=json.dumps(idprep), validate_certs=self.validate_certs)
         except Exception as e:
             self.module.fail_json(msg='Could not update identity provider %s in realm %s: %s'
@@ -1543,7 +1561,7 @@ class KeycloakAPI(object):
         """
         idp_url = URL_IDENTITY_PROVIDER.format(url=self.baseurl, realm=realm, alias=alias)
         try:
-            return open_url(idp_url, method='DELETE', headers=self.restheaders,
+            return open_url(idp_url, method='DELETE', headers=self.restheaders, timeout=self.connection_timeout,
                             validate_certs=self.validate_certs)
         except Exception as e:
             self.module.fail_json(msg='Unable to delete identity provider %s in realm %s: %s'
@@ -1557,7 +1575,7 @@ class KeycloakAPI(object):
         """
         mappers_url = URL_IDENTITY_PROVIDER_MAPPERS.format(url=self.baseurl, realm=realm, alias=alias)
         try:
-            return json.loads(to_native(open_url(mappers_url, method='GET', headers=self.restheaders,
+            return json.loads(to_native(open_url(mappers_url, method='GET', headers=self.restheaders, timeout=self.connection_timeout,
                                                  validate_certs=self.validate_certs).read()))
         except ValueError as e:
             self.module.fail_json(msg='API returned incorrect JSON when trying to obtain list of identity provider mappers for idp %s in realm %s: %s'
@@ -1575,7 +1593,7 @@ class KeycloakAPI(object):
         """
         mapper_url = URL_IDENTITY_PROVIDER_MAPPER.format(url=self.baseurl, realm=realm, alias=alias, id=mid)
         try:
-            return json.loads(to_native(open_url(mapper_url, method="GET", headers=self.restheaders,
+            return json.loads(to_native(open_url(mapper_url, method="GET", headers=self.restheaders, timeout=self.connection_timeout,
                                                  validate_certs=self.validate_certs).read()))
         except HTTPError as e:
             if e.code == 404:
@@ -1596,7 +1614,7 @@ class KeycloakAPI(object):
         """
         mappers_url = URL_IDENTITY_PROVIDER_MAPPERS.format(url=self.baseurl, realm=realm, alias=alias)
         try:
-            return open_url(mappers_url, method='POST', headers=self.restheaders,
+            return open_url(mappers_url, method='POST', headers=self.restheaders, timeout=self.connection_timeout,
                             data=json.dumps(mapper), validate_certs=self.validate_certs)
         except Exception as e:
             self.module.fail_json(msg='Could not create identity provider mapper %s for idp %s in realm %s: %s'
@@ -1611,7 +1629,7 @@ class KeycloakAPI(object):
         """
         mapper_url = URL_IDENTITY_PROVIDER_MAPPER.format(url=self.baseurl, realm=realm, alias=alias, id=mapper['id'])
         try:
-            return open_url(mapper_url, method='PUT', headers=self.restheaders,
+            return open_url(mapper_url, method='PUT', headers=self.restheaders, timeout=self.connection_timeout,
                             data=json.dumps(mapper), validate_certs=self.validate_certs)
         except Exception as e:
             self.module.fail_json(msg='Could not update mapper %s for identity provider %s in realm %s: %s'
@@ -1625,7 +1643,7 @@ class KeycloakAPI(object):
         """
         mapper_url = URL_IDENTITY_PROVIDER_MAPPER.format(url=self.baseurl, realm=realm, alias=alias, id=mid)
         try:
-            return open_url(mapper_url, method='DELETE', headers=self.restheaders,
+            return open_url(mapper_url, method='DELETE', headers=self.restheaders, timeout=self.connection_timeout,
                             validate_certs=self.validate_certs)
         except Exception as e:
             self.module.fail_json(msg='Unable to delete mapper %s for identity provider %s in realm %s: %s'
@@ -1642,7 +1660,7 @@ class KeycloakAPI(object):
             comps_url += '?%s' % filter
 
         try:
-            return json.loads(to_native(open_url(comps_url, method='GET', headers=self.restheaders,
+            return json.loads(to_native(open_url(comps_url, method='GET', headers=self.restheaders, timeout=self.connection_timeout,
                                                  validate_certs=self.validate_certs).read()))
         except ValueError as e:
             self.module.fail_json(msg='API returned incorrect JSON when trying to obtain list of components for realm %s: %s'
@@ -1659,7 +1677,7 @@ class KeycloakAPI(object):
         """
         comp_url = URL_COMPONENT.format(url=self.baseurl, realm=realm, id=cid)
         try:
-            return json.loads(to_native(open_url(comp_url, method="GET", headers=self.restheaders,
+            return json.loads(to_native(open_url(comp_url, method="GET", headers=self.restheaders, timeout=self.connection_timeout,
                                                  validate_certs=self.validate_certs).read()))
         except HTTPError as e:
             if e.code == 404:
@@ -1679,13 +1697,13 @@ class KeycloakAPI(object):
         """
         comps_url = URL_COMPONENTS.format(url=self.baseurl, realm=realm)
         try:
-            resp = open_url(comps_url, method='POST', headers=self.restheaders,
+            resp = open_url(comps_url, method='POST', headers=self.restheaders, timeout=self.connection_timeout,
                             data=json.dumps(comprep), validate_certs=self.validate_certs)
             comp_url = resp.getheader('Location')
             if comp_url is None:
                 self.module.fail_json(msg='Could not create component in realm %s: %s'
                                           % (realm, 'unexpected response'))
-            return json.loads(to_native(open_url(comp_url, method="GET", headers=self.restheaders,
+            return json.loads(to_native(open_url(comp_url, method="GET", headers=self.restheaders, timeout=self.connection_timeout,
                                                  validate_certs=self.validate_certs).read()))
         except Exception as e:
             self.module.fail_json(msg='Could not create component in realm %s: %s'
@@ -1702,7 +1720,7 @@ class KeycloakAPI(object):
             self.module.fail_json(msg='Cannot update component without id')
         comp_url = URL_COMPONENT.format(url=self.baseurl, realm=realm, id=cid)
         try:
-            return open_url(comp_url, method='PUT', headers=self.restheaders,
+            return open_url(comp_url, method='PUT', headers=self.restheaders, timeout=self.connection_timeout,
                             data=json.dumps(comprep), validate_certs=self.validate_certs)
         except Exception as e:
             self.module.fail_json(msg='Could not update component %s in realm %s: %s'
@@ -1715,7 +1733,7 @@ class KeycloakAPI(object):
         """
         comp_url = URL_COMPONENT.format(url=self.baseurl, realm=realm, id=cid)
         try:
-            return open_url(comp_url, method='DELETE', headers=self.restheaders,
+            return open_url(comp_url, method='DELETE', headers=self.restheaders, timeout=self.connection_timeout,
                             validate_certs=self.validate_certs)
         except Exception as e:
             self.module.fail_json(msg='Unable to delete component %s in realm %s: %s'

plugins/module_utils/proxmox.py

@@ -21,6 +21,8 @@ except ImportError:
 
 
 from ansible.module_utils.basic import env_fallback, missing_required_lib
+from ansible.module_utils.common.text.converters import to_native
+from ansible_collections.community.general.plugins.module_utils.version import LooseVersion
 
 
 def proxmox_auth_argument_spec():
@@ -98,3 +100,46 @@ class ProxmoxAnsible(object):
             return ProxmoxAPI(api_host, verify_ssl=validate_certs, **auth_args)
         except Exception as e:
             self.module.fail_json(msg='%s' % e, exception=traceback.format_exc())
+
+    def version(self):
+        apireturn = self.proxmox_api.version.get()
+        return LooseVersion(apireturn['version'])
+
+    def get_node(self, node):
+        nodes = [n for n in self.proxmox_api.nodes.get() if n['node'] == node]
+        return nodes[0] if nodes else None
+
+    def get_nextvmid(self):
+        vmid = self.proxmox_api.cluster.nextid.get()
+        return vmid
+
+    def get_vmid(self, name, ignore_missing=False, choose_first_if_multiple=False):
+        vms = [vm['vmid'] for vm in self.proxmox_api.cluster.resources.get(type='vm') if vm.get('name') == name]
+
+        if not vms:
+            if ignore_missing:
+                return None
+
+            self.module.fail_json(msg='No VM with name %s found' % name)
+        elif len(vms) > 1:
+            if choose_first_if_multiple:
+                self.module.deprecate(
+                    'Multiple VMs with name %s found, choosing the first one. ' % name +
+                    'This will be an error in the future. To ensure the correct VM is used, ' +
+                    'also pass the vmid parameter.',
+                    version='5.0.0', collection_name='community.general')
+            else:
+                self.module.fail_json(msg='Multiple VMs with name %s found, provide vmid instead' % name)
+
+        return vms[0]
+
+    def get_vm(self, vmid, ignore_missing=False):
+        vms = [vm for vm in self.proxmox_api.cluster.resources.get(type='vm') if vm['vmid'] == int(vmid)]
+
+        if vms:
+            return vms[0]
+        else:
+            if ignore_missing:
+                return None
+
+            self.module.fail_json(msg='VM with vmid %s does not exist in cluster' % vmid)

plugins/module_utils/scaleway.py

@@ -167,17 +167,61 @@ class Scaleway(object):
 
 
 SCALEWAY_LOCATION = {
-    'par1': {'name': 'Paris 1', 'country': 'FR', "api_endpoint": 'https://api.scaleway.com/instance/v1/zones/fr-par-1'},
-    'EMEA-FR-PAR1': {'name': 'Paris 1', 'country': 'FR', "api_endpoint": 'https://api.scaleway.com/instance/v1/zones/fr-par-1'},
+    'par1': {
+        'name': 'Paris 1',
+        'country': 'FR',
+        'api_endpoint': 'https://api.scaleway.com/instance/v1/zones/fr-par-1',
+        'api_endpoint_vpc': 'https://api.scaleway.com/vpc/v1/zones/fr-par-1'
+    },
 
-    'par2': {'name': 'Paris 2', 'country': 'FR', "api_endpoint": 'https://api.scaleway.com/instance/v1/zones/fr-par-2'},
-    'EMEA-FR-PAR2': {'name': 'Paris 2', 'country': 'FR', "api_endpoint": 'https://api.scaleway.com/instance/v1/zones/fr-par-2'},
+    'EMEA-FR-PAR1': {
+        'name': 'Paris 1',
+        'country': 'FR',
+        'api_endpoint': 'https://api.scaleway.com/instance/v1/zones/fr-par-1',
+        'api_endpoint_vpc': 'https://api.scaleway.com/vpc/v1/zones/fr-par-1'
+    },
 
-    'ams1': {'name': 'Amsterdam 1', 'country': 'NL', "api_endpoint": 'https://api.scaleway.com/instance/v1/zones/nl-ams-1'},
-    'EMEA-NL-EVS': {'name': 'Amsterdam 1', 'country': 'NL', "api_endpoint": 'https://api.scaleway.com/instance/v1/zones/nl-ams-1'},
+    'par2': {
+        'name': 'Paris 2',
+        'country': 'FR',
+        'api_endpoint': 'https://api.scaleway.com/instance/v1/zones/fr-par-2',
+        'api_endpoint_vpc': 'https://api.scaleway.com/vpc/v1/zones/fr-par-2'
+    },
 
-    'waw1': {'name': 'Warsaw 1', 'country': 'PL', "api_endpoint": 'https://api.scaleway.com/instance/v1/zones/pl-waw-1'},
-    'EMEA-PL-WAW1': {'name': 'Warsaw 1', 'country': 'PL', "api_endpoint": 'https://api.scaleway.com/instance/v1/zones/pl-waw-1'},
+    'EMEA-FR-PAR2': {
+        'name': 'Paris 2',
+        'country': 'FR',
+        'api_endpoint': 'https://api.scaleway.com/instance/v1/zones/fr-par-2',
+        'api_endpoint_vpc': 'https://api.scaleway.com/vpc/v1/zones/fr-par-2'
+    },
+
+    'ams1': {
+        'name': 'Amsterdam 1',
+        'country': 'NL',
+        'api_endpoint': 'https://api.scaleway.com/instance/v1/zones/nl-ams-1',
+        'api_endpoint_vpc': 'https://api.scaleway.com/vpc/v1/zones/nl-ams-10'
+    },
+
+    'EMEA-NL-EVS': {
+        'name': 'Amsterdam 1',
+        'country': 'NL',
+        'api_endpoint': 'https://api.scaleway.com/instance/v1/zones/nl-ams-1',
+        'api_endpoint_vpc': 'https://api.scaleway.com/vpc/v1/zones/nl-ams-1'
+    },
+
+    'waw1': {
+        'name': 'Warsaw 1',
+        'country': 'PL',
+        'api_endpoint': 'https://api.scaleway.com/instance/v1/zones/pl-waw-1',
+        'api_endpoint_vpc': 'https://api.scaleway.com/vpc/v1/zones/pl-waw-1'
+    },
+
+    'EMEA-PL-WAW1': {
+        'name': 'Warsaw 1',
+        'country': 'PL',
+        'api_endpoint': 'https://api.scaleway.com/instance/v1/zones/pl-waw-1',
+        'api_endpoint_vpc': 'https://api.scaleway.com/vpc/v1/zones/pl-waw-1'
+    },
 }
 
 SCALEWAY_ENDPOINT = "https://api.scaleway.com"

plugins/modules/cloud/misc/proxmox.py

@@ -1,5 +1,6 @@
 #!/usr/bin/python
 # -*- coding: utf-8 -*-
+#
 # Copyright: Ansible Project
 # GNU General Public License v3.0+ (see COPYING or https://www.gnu.org/licenses/gpl-3.0.txt)
 
@@ -392,229 +393,189 @@ import traceback
 
 from ansible_collections.community.general.plugins.module_utils.version import LooseVersion
 
-try:
-    from proxmoxer import ProxmoxAPI
-    HAS_PROXMOXER = True
-except ImportError:
-    HAS_PROXMOXER = False
-
 from ansible.module_utils.basic import AnsibleModule, env_fallback
 from ansible.module_utils.common.text.converters import to_native
 
 from ansible_collections.community.general.plugins.module_utils.proxmox import (
-    ansible_to_proxmox_bool
-)
-
+    ansible_to_proxmox_bool, proxmox_auth_argument_spec, ProxmoxAnsible)
 
 VZ_TYPE = None
 
 
-def get_nextvmid(module, proxmox):
-    try:
-        vmid = proxmox.cluster.nextid.get()
-        return vmid
-    except Exception as e:
-        module.fail_json(msg="Unable to get next vmid. Failed with exception: %s" % to_native(e),
-                         exception=traceback.format_exc())
+class ProxmoxLxcAnsible(ProxmoxAnsible):
+    def content_check(self, node, ostemplate, template_store):
+        return [True for cnt in self.proxmox_api.nodes(node).storage(template_store).content.get() if cnt['volid'] == ostemplate]
 
+    def is_template_container(self, node, vmid):
+        """Check if the specified container is a template."""
+        proxmox_node = self.proxmox_api.nodes(node)
+        config = getattr(proxmox_node, VZ_TYPE)(vmid).config.get()
+        return config['template']
 
-def get_vmid(proxmox, hostname):
-    return [vm['vmid'] for vm in proxmox.cluster.resources.get(type='vm') if 'name' in vm and vm['name'] == hostname]
+    def create_instance(self, vmid, node, disk, storage, cpus, memory, swap, timeout, clone, **kwargs):
+        proxmox_node = self.proxmox_api.nodes(node)
 
+        # Remove all empty kwarg entries
+        kwargs = dict((k, v) for k, v in kwargs.items() if v is not None)
 
-def get_instance(proxmox, vmid):
-    return [vm for vm in proxmox.cluster.resources.get(type='vm') if vm['vmid'] == int(vmid)]
-
-
-def content_check(proxmox, node, ostemplate, template_store):
-    return [True for cnt in proxmox.nodes(node).storage(template_store).content.get() if cnt['volid'] == ostemplate]
-
-
-def is_template_container(proxmox, node, vmid):
-    """Check if the specified container is a template."""
-    proxmox_node = proxmox.nodes(node)
-    config = getattr(proxmox_node, VZ_TYPE)(vmid).config.get()
-    return config['template']
-
-
-def node_check(proxmox, node):
-    return [True for nd in proxmox.nodes.get() if nd['node'] == node]
-
-
-def proxmox_version(proxmox):
-    apireturn = proxmox.version.get()
-    return LooseVersion(apireturn['version'])
-
-
-def create_instance(module, proxmox, vmid, node, disk, storage, cpus, memory, swap, timeout, clone, **kwargs):
-    proxmox_node = proxmox.nodes(node)
-
-    # Remove all empty kwarg entries
-    kwargs = dict((k, v) for k, v in kwargs.items() if v is not None)
-
-    if VZ_TYPE == 'lxc':
-        kwargs['cpulimit'] = cpus
-        kwargs['rootfs'] = disk
-        if 'netif' in kwargs:
-            kwargs.update(kwargs['netif'])
-            del kwargs['netif']
-        if 'mounts' in kwargs:
-            kwargs.update(kwargs['mounts'])
-            del kwargs['mounts']
-        if 'pubkey' in kwargs:
-            if proxmox_version(proxmox) >= LooseVersion('4.2'):
-                kwargs['ssh-public-keys'] = kwargs['pubkey']
-            del kwargs['pubkey']
-    else:
-        kwargs['cpus'] = cpus
-        kwargs['disk'] = disk
-
-    if clone is not None:
-        if VZ_TYPE != 'lxc':
-            module.fail_json(changed=False, msg="Clone operator is only supported for LXC enabled proxmox clusters.")
-
-        clone_is_template = is_template_container(proxmox, node, clone)
-
-        # By default, create a full copy only when the cloned container is not a template.
-        create_full_copy = not clone_is_template
-
-        # Only accept parameters that are compatible with the clone endpoint.
-        valid_clone_parameters = ['hostname', 'pool', 'description']
-        if module.params['storage'] is not None and clone_is_template:
-            # Cloning a template, so create a full copy instead of a linked copy
-            create_full_copy = True
-        elif module.params['storage'] is None and not clone_is_template:
-            # Not cloning a template, but also no defined storage. This isn't possible.
-            module.fail_json(changed=False, msg="Cloned container is not a template, storage needs to be specified.")
-
-        if module.params['clone_type'] == 'linked':
-            if not clone_is_template:
-                module.fail_json(changed=False, msg="'linked' clone type is specified, but cloned container is not a template container.")
-            # Don't need to do more, by default create_full_copy is set to false already
-        elif module.params['clone_type'] == 'opportunistic':
-            if not clone_is_template:
-                # Cloned container is not a template, so we need our 'storage' parameter
-                valid_clone_parameters.append('storage')
-        elif module.params['clone_type'] == 'full':
-            create_full_copy = True
-            valid_clone_parameters.append('storage')
-
-        clone_parameters = {}
-
-        if create_full_copy:
-            clone_parameters['full'] = '1'
+        if VZ_TYPE == 'lxc':
+            kwargs['cpulimit'] = cpus
+            kwargs['rootfs'] = disk
+            if 'netif' in kwargs:
+                kwargs.update(kwargs['netif'])
+                del kwargs['netif']
+            if 'mounts' in kwargs:
+                kwargs.update(kwargs['mounts'])
+                del kwargs['mounts']
+            if 'pubkey' in kwargs:
+                if self.version() >= LooseVersion('4.2'):
+                    kwargs['ssh-public-keys'] = kwargs['pubkey']
+                del kwargs['pubkey']
         else:
-            clone_parameters['full'] = '0'
-        for param in valid_clone_parameters:
-            if module.params[param] is not None:
-                clone_parameters[param] = module.params[param]
+            kwargs['cpus'] = cpus
+            kwargs['disk'] = disk
 
-        taskid = getattr(proxmox_node, VZ_TYPE)(clone).clone.post(newid=vmid, **clone_parameters)
-    else:
-        taskid = getattr(proxmox_node, VZ_TYPE).create(vmid=vmid, storage=storage, memory=memory, swap=swap, **kwargs)
+        if clone is not None:
+            if VZ_TYPE != 'lxc':
+                self.module.fail_json(changed=False, msg="Clone operator is only supported for LXC enabled proxmox clusters.")
 
-    while timeout:
-        if (proxmox_node.tasks(taskid).status.get()['status'] == 'stopped' and
-                proxmox_node.tasks(taskid).status.get()['exitstatus'] == 'OK'):
-            return True
-        timeout -= 1
-        if timeout == 0:
-            module.fail_json(msg='Reached timeout while waiting for creating VM. Last line in task before timeout: %s' %
-                                 proxmox_node.tasks(taskid).log.get()[:1])
+            clone_is_template = self.is_template_container(node, clone)
 
-        time.sleep(1)
-    return False
+            # By default, create a full copy only when the cloned container is not a template.
+            create_full_copy = not clone_is_template
 
+            # Only accept parameters that are compatible with the clone endpoint.
+            valid_clone_parameters = ['hostname', 'pool', 'description']
+            if self.module.params['storage'] is not None and clone_is_template:
+                # Cloning a template, so create a full copy instead of a linked copy
+                create_full_copy = True
+            elif self.module.params['storage'] is None and not clone_is_template:
+                # Not cloning a template, but also no defined storage. This isn't possible.
+                self.module.fail_json(changed=False, msg="Cloned container is not a template, storage needs to be specified.")
 
-def start_instance(module, proxmox, vm, vmid, timeout):
-    taskid = getattr(proxmox.nodes(vm[0]['node']), VZ_TYPE)(vmid).status.start.post()
-    while timeout:
-        if (proxmox.nodes(vm[0]['node']).tasks(taskid).status.get()['status'] == 'stopped' and
-                proxmox.nodes(vm[0]['node']).tasks(taskid).status.get()['exitstatus'] == 'OK'):
-            return True
-        timeout -= 1
-        if timeout == 0:
-            module.fail_json(msg='Reached timeout while waiting for starting VM. Last line in task before timeout: %s' %
-                                 proxmox.nodes(vm[0]['node']).tasks(taskid).log.get()[:1])
+            if self.module.params['clone_type'] == 'linked':
+                if not clone_is_template:
+                    self.module.fail_json(changed=False, msg="'linked' clone type is specified, but cloned container is not a template container.")
+                # Don't need to do more, by default create_full_copy is set to false already
+            elif self.module.params['clone_type'] == 'opportunistic':
+                if not clone_is_template:
+                    # Cloned container is not a template, so we need our 'storage' parameter
+                    valid_clone_parameters.append('storage')
+            elif self.module.params['clone_type'] == 'full':
+                create_full_copy = True
+                valid_clone_parameters.append('storage')
 
-        time.sleep(1)
-    return False
+            clone_parameters = {}
 
+            if create_full_copy:
+                clone_parameters['full'] = '1'
+            else:
+                clone_parameters['full'] = '0'
+            for param in valid_clone_parameters:
+                if self.module.params[param] is not None:
+                    clone_parameters[param] = self.module.params[param]
 
-def stop_instance(module, proxmox, vm, vmid, timeout, force):
-    if force:
-        taskid = getattr(proxmox.nodes(vm[0]['node']), VZ_TYPE)(vmid).status.shutdown.post(forceStop=1)
-    else:
-        taskid = getattr(proxmox.nodes(vm[0]['node']), VZ_TYPE)(vmid).status.shutdown.post()
-    while timeout:
-        if (proxmox.nodes(vm[0]['node']).tasks(taskid).status.get()['status'] == 'stopped' and
-                proxmox.nodes(vm[0]['node']).tasks(taskid).status.get()['exitstatus'] == 'OK'):
-            return True
-        timeout -= 1
-        if timeout == 0:
-            module.fail_json(msg='Reached timeout while waiting for stopping VM. Last line in task before timeout: %s' %
-                                 proxmox.nodes(vm[0]['node']).tasks(taskid).log.get()[:1])
+            taskid = getattr(proxmox_node, VZ_TYPE)(clone).clone.post(newid=vmid, **clone_parameters)
+        else:
+            taskid = getattr(proxmox_node, VZ_TYPE).create(vmid=vmid, storage=storage, memory=memory, swap=swap, **kwargs)
 
-        time.sleep(1)
-    return False
+        while timeout:
+            if (proxmox_node.tasks(taskid).status.get()['status'] == 'stopped' and
+                    proxmox_node.tasks(taskid).status.get()['exitstatus'] == 'OK'):
+                return True
+            timeout -= 1
+            if timeout == 0:
+                self.module.fail_json(msg='Reached timeout while waiting for creating VM. Last line in task before timeout: %s' %
+                                      proxmox_node.tasks(taskid).log.get()[:1])
 
+            time.sleep(1)
+        return False
 
-def umount_instance(module, proxmox, vm, vmid, timeout):
-    taskid = getattr(proxmox.nodes(vm[0]['node']), VZ_TYPE)(vmid).status.umount.post()
-    while timeout:
-        if (proxmox.nodes(vm[0]['node']).tasks(taskid).status.get()['status'] == 'stopped' and
-                proxmox.nodes(vm[0]['node']).tasks(taskid).status.get()['exitstatus'] == 'OK'):
-            return True
-        timeout -= 1
-        if timeout == 0:
-            module.fail_json(msg='Reached timeout while waiting for unmounting VM. Last line in task before timeout: %s' %
-                                 proxmox.nodes(vm[0]['node']).tasks(taskid).log.get()[:1])
+    def start_instance(self, vm, vmid, timeout):
+        taskid = getattr(self.proxmox_api.nodes(vm['node']), VZ_TYPE)(vmid).status.start.post()
+        while timeout:
+            if (self.proxmox_api.nodes(vm['node']).tasks(taskid).status.get()['status'] == 'stopped' and
+                    self.proxmox_api.nodes(vm['node']).tasks(taskid).status.get()['exitstatus'] == 'OK'):
+                return True
+            timeout -= 1
+            if timeout == 0:
+                self.module.fail_json(msg='Reached timeout while waiting for starting VM. Last line in task before timeout: %s' %
+                                      self.proxmox_api.nodes(vm['node']).tasks(taskid).log.get()[:1])
 
-        time.sleep(1)
-    return False
+            time.sleep(1)
+        return False
+
+    def stop_instance(self, vm, vmid, timeout, force):
+        if force:
+            taskid = getattr(self.proxmox_api.nodes(vm['node']), VZ_TYPE)(vmid).status.shutdown.post(forceStop=1)
+        else:
+            taskid = getattr(self.proxmox_api.nodes(vm['node']), VZ_TYPE)(vmid).status.shutdown.post()
+        while timeout:
+            if (self.proxmox_api.nodes(vm['node']).tasks(taskid).status.get()['status'] == 'stopped' and
+                    self.proxmox_api.nodes(vm['node']).tasks(taskid).status.get()['exitstatus'] == 'OK'):
+                return True
+            timeout -= 1
+            if timeout == 0:
+                self.module.fail_json(msg='Reached timeout while waiting for stopping VM. Last line in task before timeout: %s' %
+                                      self.proxmox_api.nodes(vm['node']).tasks(taskid).log.get()[:1])
+
+            time.sleep(1)
+        return False
+
+    def umount_instance(self, vm, vmid, timeout):
+        taskid = getattr(self.proxmox_api.nodes(vm['node']), VZ_TYPE)(vmid).status.umount.post()
+        while timeout:
+            if (self.proxmox_api.nodes(vm['node']).tasks(taskid).status.get()['status'] == 'stopped' and
+                    self.proxmox_api.nodes(vm['node']).tasks(taskid).status.get()['exitstatus'] == 'OK'):
+                return True
+            timeout -= 1
+            if timeout == 0:
+                self.module.fail_json(msg='Reached timeout while waiting for unmounting VM. Last line in task before timeout: %s' %
+                                      self.proxmox_api.nodes(vm['node']).tasks(taskid).log.get()[:1])
+
+            time.sleep(1)
+        return False
 
 
 def main():
+    module_args = proxmox_auth_argument_spec()
+    proxmox_args = dict(
+        vmid=dict(type='int', required=False),
+        node=dict(),
+        pool=dict(),
+        password=dict(no_log=True),
+        hostname=dict(),
+        ostemplate=dict(),
+        disk=dict(type='str'),
+        cores=dict(type='int'),
+        cpus=dict(type='int'),
+        memory=dict(type='int'),
+        swap=dict(type='int'),
+        netif=dict(type='dict'),
+        mounts=dict(type='dict'),
+        ip_address=dict(),
+        onboot=dict(type='bool'),
+        features=dict(type='list', elements='str'),
+        storage=dict(default='local'),
+        cpuunits=dict(type='int'),
+        nameserver=dict(),
+        searchdomain=dict(),
+        timeout=dict(type='int', default=30),
+        force=dict(type='bool', default=False),
+        purge=dict(type='bool', default=False),
+        state=dict(default='present', choices=['present', 'absent', 'stopped', 'started', 'restarted']),
+        pubkey=dict(type='str', default=None),
+        unprivileged=dict(type='bool', default=False),
+        description=dict(type='str'),
+        hookscript=dict(type='str'),
+        proxmox_default_behavior=dict(type='str', default='no_defaults', choices=['compatibility', 'no_defaults']),
+        clone=dict(type='int'),
+        clone_type=dict(default='opportunistic', choices=['full', 'linked', 'opportunistic']),
+    )
+    module_args.update(proxmox_args)
+
     module = AnsibleModule(
-        argument_spec=dict(
-            api_host=dict(required=True),
-            api_password=dict(no_log=True, fallback=(env_fallback, ['PROXMOX_PASSWORD'])),
-            api_token_id=dict(no_log=True),
-            api_token_secret=dict(no_log=True),
-            api_user=dict(required=True),
-            vmid=dict(type='int', required=False),
-            validate_certs=dict(type='bool', default=False),
-            node=dict(),
-            pool=dict(),
-            password=dict(no_log=True),
-            hostname=dict(),
-            ostemplate=dict(),
-            disk=dict(type='str'),
-            cores=dict(type='int'),
-            cpus=dict(type='int'),
-            memory=dict(type='int'),
-            swap=dict(type='int'),
-            netif=dict(type='dict'),
-            mounts=dict(type='dict'),
-            ip_address=dict(),
-            onboot=dict(type='bool'),
-            features=dict(type='list', elements='str'),
-            storage=dict(default='local'),
-            cpuunits=dict(type='int'),
-            nameserver=dict(),
-            searchdomain=dict(),
-            timeout=dict(type='int', default=30),
-            force=dict(type='bool', default=False),
-            purge=dict(type='bool', default=False),
-            state=dict(default='present', choices=['present', 'absent', 'stopped', 'started', 'restarted']),
-            pubkey=dict(type='str', default=None),
-            unprivileged=dict(type='bool', default=False),
-            description=dict(type='str'),
-            hookscript=dict(type='str'),
-            proxmox_default_behavior=dict(type='str', default='no_defaults', choices=['compatibility', 'no_defaults']),
-            clone=dict(type='int'),
-            clone_type=dict(default='opportunistic', choices=['full', 'linked', 'opportunistic']),
-        ),
+        argument_spec=module_args,
         required_if=[
             ('state', 'present', ['node', 'hostname']),
             ('state', 'present', ('clone', 'ostemplate'), True),  # Require one of clone and ostemplate. Together with mutually_exclusive this ensures that we
@@ -627,17 +588,13 @@ def main():
         mutually_exclusive=[('clone', 'ostemplate')],  # Creating a new container is done either by cloning an existing one, or based on a template.
     )
 
-    if not HAS_PROXMOXER:
-        module.fail_json(msg='proxmoxer required for this module')
+    proxmox = ProxmoxLxcAnsible(module)
+
+    global VZ_TYPE
+    VZ_TYPE = 'openvz' if proxmox.version() < LooseVersion('4.0') else 'lxc'
 
     state = module.params['state']
-    api_host = module.params['api_host']
-    api_password = module.params['api_password']
-    api_token_id = module.params['api_token_id']
-    api_token_secret = module.params['api_token_secret']
-    api_user = module.params['api_user']
     vmid = module.params['vmid']
-    validate_certs = module.params['validate_certs']
     node = module.params['node']
     disk = module.params['disk']
     cpus = module.params['cpus']
@@ -664,68 +621,54 @@ def main():
             if module.params[param] is None:
                 module.params[param] = value
 
-    auth_args = {'user': api_user}
-    if not api_token_id:
-        auth_args['password'] = api_password
-    else:
-        auth_args['token_name'] = api_token_id
-        auth_args['token_value'] = api_token_secret
-
-    try:
-        proxmox = ProxmoxAPI(api_host, verify_ssl=validate_certs, **auth_args)
-        global VZ_TYPE
-        VZ_TYPE = 'openvz' if proxmox_version(proxmox) < LooseVersion('4.0') else 'lxc'
-    except Exception as e:
-        module.fail_json(msg='authorization on proxmox cluster failed with exception: %s' % e)
-
     # If vmid not set get the Next VM id from ProxmoxAPI
     # If hostname is set get the VM id from ProxmoxAPI
     if not vmid and state == 'present':
-        vmid = get_nextvmid(module, proxmox)
+        vmid = proxmox.get_nextvmid()
     elif not vmid and hostname:
-        hosts = get_vmid(proxmox, hostname)
-        if len(hosts) == 0:
-            module.fail_json(msg="Vmid could not be fetched => Hostname doesn't exist (action: %s)" % state)
-        vmid = hosts[0]
+        vmid = proxmox.get_vmid(hostname, choose_first_if_multiple=True)
     elif not vmid:
         module.exit_json(changed=False, msg="Vmid could not be fetched for the following action: %s" % state)
 
     # Create a new container
     if state == 'present' and clone is None:
         try:
-            if get_instance(proxmox, vmid) and not module.params['force']:
+            if proxmox.get_vm(vmid, ignore_missing=True) and not module.params['force']:
                 module.exit_json(changed=False, msg="VM with vmid = %s is already exists" % vmid)
             # If no vmid was passed, there cannot be another VM named 'hostname'
-            if not module.params['vmid'] and get_vmid(proxmox, hostname) and not module.params['force']:
-                module.exit_json(changed=False, msg="VM with hostname %s already exists and has ID number %s" % (hostname, get_vmid(proxmox, hostname)[0]))
-            elif not node_check(proxmox, node):
+            if (not module.params['vmid'] and
+                    proxmox.get_vmid(hostname, ignore_missing=True, choose_first_if_multiple=True) and
+                    not module.params['force']):
+                vmid = proxmox.get_vmid(hostname, choose_first_if_multiple=True)
+                module.exit_json(changed=False, msg="VM with hostname %s already exists and has ID number %s" % (hostname, vmid))
+            elif not proxmox.get_node(node):
                 module.fail_json(msg="node '%s' not exists in cluster" % node)
-            elif not content_check(proxmox, node, module.params['ostemplate'], template_store):
+            elif not proxmox.content_check(node, module.params['ostemplate'], template_store):
                 module.fail_json(msg="ostemplate '%s' not exists on node %s and storage %s"
                                  % (module.params['ostemplate'], node, template_store))
         except Exception as e:
             module.fail_json(msg="Pre-creation checks of {VZ_TYPE} VM {vmid} failed with exception: {e}".format(VZ_TYPE=VZ_TYPE, vmid=vmid, e=e))
 
         try:
-            create_instance(module, proxmox, vmid, node, disk, storage, cpus, memory, swap, timeout, clone,
-                            cores=module.params['cores'],
-                            pool=module.params['pool'],
-                            password=module.params['password'],
-                            hostname=module.params['hostname'],
-                            ostemplate=module.params['ostemplate'],
-                            netif=module.params['netif'],
-                            mounts=module.params['mounts'],
-                            ip_address=module.params['ip_address'],
-                            onboot=ansible_to_proxmox_bool(module.params['onboot']),
-                            cpuunits=module.params['cpuunits'],
-                            nameserver=module.params['nameserver'],
-                            searchdomain=module.params['searchdomain'],
-                            force=ansible_to_proxmox_bool(module.params['force']),
-                            pubkey=module.params['pubkey'],
-                            features=",".join(module.params['features']) if module.params['features'] is not None else None,
-                            unprivileged=ansible_to_proxmox_bool(module.params['unprivileged']),
-                            description=module.params['description'],
-                            hookscript=module.params['hookscript'])
+            proxmox.create_instance(vmid, node, disk, storage, cpus, memory, swap, timeout, clone,
+                                    cores=module.params['cores'],
+                                    pool=module.params['pool'],
+                                    password=module.params['password'],
+                                    hostname=module.params['hostname'],
+                                    ostemplate=module.params['ostemplate'],
+                                    netif=module.params['netif'],
+                                    mounts=module.params['mounts'],
+                                    ip_address=module.params['ip_address'],
+                                    onboot=ansible_to_proxmox_bool(module.params['onboot']),
+                                    cpuunits=module.params['cpuunits'],
+                                    nameserver=module.params['nameserver'],
+                                    searchdomain=module.params['searchdomain'],
+                                    force=ansible_to_proxmox_bool(module.params['force']),
+                                    pubkey=module.params['pubkey'],
+                                    features=",".join(module.params['features']) if module.params['features'] is not None else None,
+                                    unprivileged=ansible_to_proxmox_bool(module.params['unprivileged']),
+                                    description=module.params['description'],
+                                    hookscript=module.params['hookscript'])
 
             module.exit_json(changed=True, msg="Deployed VM %s from template %s" % (vmid, module.params['ostemplate']))
         except Exception as e:
@@ -734,18 +677,21 @@ def main():
     # Clone a container
     elif state == 'present' and clone is not None:
         try:
-            if get_instance(proxmox, vmid) and not module.params['force']:
+            if proxmox.get_vm(vmid, ignore_missing=True) and not module.params['force']:
                 module.exit_json(changed=False, msg="VM with vmid = %s is already exists" % vmid)
             # If no vmid was passed, there cannot be another VM named 'hostname'
-            if not module.params['vmid'] and get_vmid(proxmox, hostname) and not module.params['force']:
-                module.exit_json(changed=False, msg="VM with hostname %s already exists and has ID number %s" % (hostname, get_vmid(proxmox, hostname)[0]))
-            if not get_instance(proxmox, clone):
+            if (not module.params['vmid'] and
+                    proxmox.get_vmid(hostname, ignore_missing=True, choose_first_if_multiple=True) and
+                    not module.params['force']):
+                vmid = proxmox.get_vmid(hostname, choose_first_if_multiple=True)
+                module.exit_json(changed=False, msg="VM with hostname %s already exists and has ID number %s" % (hostname, vmid))
+            if not proxmox.get_vm(clone, ignore_missing=True):
                 module.exit_json(changed=False, msg="Container to be cloned does not exist")
         except Exception as e:
             module.fail_json(msg="Pre-clone checks of {VZ_TYPE} VM {vmid} failed with exception: {e}".format(VZ_TYPE=VZ_TYPE, vmid=vmid, e=e))
 
         try:
-            create_instance(module, proxmox, vmid, node, disk, storage, cpus, memory, swap, timeout, clone)
+            proxmox.create_instance(vmid, node, disk, storage, cpus, memory, swap, timeout, clone)
 
             module.exit_json(changed=True, msg="Cloned VM %s from %s" % (vmid, clone))
         except Exception as e:
@@ -753,64 +699,60 @@ def main():
 
     elif state == 'started':
         try:
-            vm = get_instance(proxmox, vmid)
-            if not vm:
-                module.fail_json(msg='VM with vmid = %s not exists in cluster' % vmid)
-            if getattr(proxmox.nodes(vm[0]['node']), VZ_TYPE)(vmid).status.current.get()['status'] == 'running':
+            vm = proxmox.get_vm(vmid)
+            if getattr(proxmox.proxmox_api.nodes(vm['node']), VZ_TYPE)(vmid).status.current.get()['status'] == 'running':
                 module.exit_json(changed=False, msg="VM %s is already running" % vmid)
 
-            if start_instance(module, proxmox, vm, vmid, timeout):
+            if proxmox.start_instance(vm, vmid, timeout):
                 module.exit_json(changed=True, msg="VM %s started" % vmid)
         except Exception as e:
             module.fail_json(msg="starting of VM %s failed with exception: %s" % (vmid, e))
 
     elif state == 'stopped':
         try:
-            vm = get_instance(proxmox, vmid)
-            if not vm:
-                module.fail_json(msg='VM with vmid = %s not exists in cluster' % vmid)
+            vm = proxmox.get_vm(vmid)
 
-            if getattr(proxmox.nodes(vm[0]['node']), VZ_TYPE)(vmid).status.current.get()['status'] == 'mounted':
+            if getattr(proxmox.proxmox_api.nodes(vm['node']), VZ_TYPE)(vmid).status.current.get()['status'] == 'mounted':
                 if module.params['force']:
-                    if umount_instance(module, proxmox, vm, vmid, timeout):
+                    if proxmox.umount_instance(vm, vmid, timeout):
                         module.exit_json(changed=True, msg="VM %s is shutting down" % vmid)
                 else:
                     module.exit_json(changed=False, msg=("VM %s is already shutdown, but mounted. "
                                                          "You can use force option to umount it.") % vmid)
 
-            if getattr(proxmox.nodes(vm[0]['node']), VZ_TYPE)(vmid).status.current.get()['status'] == 'stopped':
+            if getattr(proxmox.proxmox_api.nodes(vm['node']), VZ_TYPE)(vmid).status.current.get()['status'] == 'stopped':
                 module.exit_json(changed=False, msg="VM %s is already shutdown" % vmid)
 
-            if stop_instance(module, proxmox, vm, vmid, timeout, force=module.params['force']):
+            if proxmox.stop_instance(vm, vmid, timeout, force=module.params['force']):
                 module.exit_json(changed=True, msg="VM %s is shutting down" % vmid)
         except Exception as e:
             module.fail_json(msg="stopping of VM %s failed with exception: %s" % (vmid, e))
 
     elif state == 'restarted':
         try:
-            vm = get_instance(proxmox, vmid)
-            if not vm:
-                module.fail_json(msg='VM with vmid = %s not exists in cluster' % vmid)
-            if (getattr(proxmox.nodes(vm[0]['node']), VZ_TYPE)(vmid).status.current.get()['status'] == 'stopped' or
-                    getattr(proxmox.nodes(vm[0]['node']), VZ_TYPE)(vmid).status.current.get()['status'] == 'mounted'):
+            vm = proxmox.get_vm(vmid)
+
+            vm_status = getattr(proxmox.proxmox_api.nodes(vm['node']), VZ_TYPE)(vmid).status.current.get()['status']
+            if vm_status in ['stopped', 'mounted']:
                 module.exit_json(changed=False, msg="VM %s is not running" % vmid)
 
-            if (stop_instance(module, proxmox, vm, vmid, timeout, force=module.params['force']) and
-                    start_instance(module, proxmox, vm, vmid, timeout)):
+            if (proxmox.stop_instance(vm, vmid, timeout, force=module.params['force']) and
+                    proxmox.start_instance(vm, vmid, timeout)):
                 module.exit_json(changed=True, msg="VM %s is restarted" % vmid)
         except Exception as e:
             module.fail_json(msg="restarting of VM %s failed with exception: %s" % (vmid, e))
 
     elif state == 'absent':
         try:
-            vm = get_instance(proxmox, vmid)
+            vm = proxmox.get_vm(vmid, ignore_missing=True)
             if not vm:
                 module.exit_json(changed=False, msg="VM %s does not exist" % vmid)
 
-            if getattr(proxmox.nodes(vm[0]['node']), VZ_TYPE)(vmid).status.current.get()['status'] == 'running':
+            vm_status = getattr(proxmox.proxmox_api.nodes(vm['node']), VZ_TYPE)(vmid).status.current.get()['status']
+            if vm_status == 'running':
                 module.exit_json(changed=False, msg="VM %s is running. Stop it before deletion." % vmid)
 
-            if getattr(proxmox.nodes(vm[0]['node']), VZ_TYPE)(vmid).status.current.get()['status'] == 'mounted':
+            if vm_status == 'mounted':
                 module.exit_json(changed=False, msg="VM %s is mounted. Stop it with force option before deletion." % vmid)
 
             delete_params = {}
@@ -818,16 +760,16 @@ def main():
             if module.params['purge']:
                 delete_params['purge'] = 1
 
-            taskid = getattr(proxmox.nodes(vm[0]['node']), VZ_TYPE).delete(vmid, **delete_params)
+            taskid = getattr(proxmox.proxmox_api.nodes(vm['node']), VZ_TYPE).delete(vmid, **delete_params)
 
             while timeout:
-                if (proxmox.nodes(vm[0]['node']).tasks(taskid).status.get()['status'] == 'stopped' and
-                        proxmox.nodes(vm[0]['node']).tasks(taskid).status.get()['exitstatus'] == 'OK'):
+                task_status = proxmox.proxmox_api.nodes(vm['node']).tasks(taskid).status.get()
+                if (task_status['status'] == 'stopped' and task_status['exitstatus'] == 'OK'):
                     module.exit_json(changed=True, msg="VM %s removed" % vmid)
                 timeout -= 1
                 if timeout == 0:
                     module.fail_json(msg='Reached timeout while waiting for removing VM. Last line in task before timeout: %s'
-                                     % proxmox.nodes(vm[0]['node']).tasks(taskid).log.get()[:1])
+                                     % proxmox.proxmox_api.nodes(vm['node']).tasks(taskid).log.get()[:1])
 
                 time.sleep(1)
         except Exception as e:

plugins/modules/cloud/misc/proxmox_nic.py

@@ -1,6 +1,6 @@
 #!/usr/bin/python
 # -*- coding: utf-8 -*-
-
+#
 # Copyright: (c) 2021, Lammert Hellinga (@Kogelvis) <lammert@hellinga.it>
 # GNU General Public License v3.0+ (see COPYING or https://www.gnu.org/licenses/gpl-3.0.txt)
 
@@ -136,120 +136,96 @@ msg:
   sample: "Nic net0 unchanged on VM with vmid 103"
 '''
 
-try:
-    from proxmoxer import ProxmoxAPI
-    HAS_PROXMOXER = True
-except ImportError:
-    HAS_PROXMOXER = False
-
 from ansible.module_utils.basic import AnsibleModule, env_fallback
-from ansible_collections.community.general.plugins.module_utils.proxmox import proxmox_auth_argument_spec
+from ansible_collections.community.general.plugins.module_utils.proxmox import (proxmox_auth_argument_spec, ProxmoxAnsible)
 
 
-def get_vmid(module, proxmox, name):
-    try:
-        vms = [vm['vmid'] for vm in proxmox.cluster.resources.get(type='vm') if vm.get('name') == name]
-    except Exception as e:
-        module.fail_json(msg='Error: %s occurred while retrieving VM with name = %s' % (e, name))
+class ProxmoxNicAnsible(ProxmoxAnsible):
+    def update_nic(self, vmid, interface, model, **kwargs):
+        vm = self.get_vm(vmid)
 
-    if not vms:
-        module.fail_json(msg='No VM found with name: %s' % name)
-    elif len(vms) > 1:
-        module.fail_json(msg='Multiple VMs found with name: %s, provide vmid instead' % name)
+        try:
+            vminfo = self.proxmox_api.nodes(vm['node']).qemu(vmid).config.get()
+        except Exception as e:
+            self.module.fail_json(msg='Getting information for VM with vmid = %s failed with exception: %s' % (vmid, e))
 
-    return vms[0]
+        if interface in vminfo:
+            # Convert the current config to a dictionary
+            config = vminfo[interface].split(',')
+            config.sort()
 
+            config_current = {}
 
-def get_vm(proxmox, vmid):
-    return [vm for vm in proxmox.cluster.resources.get(type='vm') if vm['vmid'] == int(vmid)]
+            for i in config:
+                kv = i.split('=')
+                try:
+                    config_current[kv[0]] = kv[1]
+                except IndexError:
+                    config_current[kv[0]] = ''
 
+            # determine the current model nic and mac-address
+            models = ['e1000', 'e1000-82540em', 'e1000-82544gc', 'e1000-82545em', 'i82551', 'i82557b',
+                      'i82559er', 'ne2k_isa', 'ne2k_pci', 'pcnet', 'rtl8139', 'virtio', 'vmxnet3']
+            current_model = set(models) & set(config_current.keys())
+            current_model = current_model.pop()
+            current_mac = config_current[current_model]
 
-def update_nic(module, proxmox, vmid, interface, model, **kwargs):
-    vm = get_vm(proxmox, vmid)
+            # build nic config string
+            config_provided = "{0}={1}".format(model, current_mac)
+        else:
+            config_provided = model
 
-    try:
-        vminfo = proxmox.nodes(vm[0]['node']).qemu(vmid).config.get()
-    except Exception as e:
-        module.fail_json(msg='Getting information for VM with vmid = %s failed with exception: %s' % (vmid, e))
+        if kwargs['mac']:
+            config_provided = "{0}={1}".format(model, kwargs['mac'])
 
-    if interface in vminfo:
-        # Convert the current config to a dictionary
-        config = vminfo[interface].split(',')
-        config.sort()
+        if kwargs['bridge']:
+            config_provided += ",bridge={0}".format(kwargs['bridge'])
 
-        config_current = {}
+        if kwargs['firewall']:
+            config_provided += ",firewall=1"
 
-        for i in config:
-            kv = i.split('=')
-            try:
-                config_current[kv[0]] = kv[1]
-            except IndexError:
-                config_current[kv[0]] = ''
+        if kwargs['link_down']:
+            config_provided += ',link_down=1'
 
-        # determine the current model nic and mac-address
-        models = ['e1000', 'e1000-82540em', 'e1000-82544gc', 'e1000-82545em', 'i82551', 'i82557b',
-                  'i82559er', 'ne2k_isa', 'ne2k_pci', 'pcnet', 'rtl8139', 'virtio', 'vmxnet3']
-        current_model = set(models) & set(config_current.keys())
-        current_model = current_model.pop()
-        current_mac = config_current[current_model]
+        if kwargs['mtu']:
+            config_provided += ",mtu={0}".format(kwargs['mtu'])
+            if model != 'virtio':
+                self.module.warn(
+                    'Ignoring MTU for nic {0} on VM with vmid {1}, '
+                    'model should be set to \'virtio\': '.format(interface, vmid))
 
-        # build nic config string
-        config_provided = "{0}={1}".format(model, current_mac)
-    else:
-        config_provided = model
+        if kwargs['queues']:
+            config_provided += ",queues={0}".format(kwargs['queues'])
 
-    if kwargs['mac']:
-        config_provided = "{0}={1}".format(model, kwargs['mac'])
+        if kwargs['rate']:
+            config_provided += ",rate={0}".format(kwargs['rate'])
 
-    if kwargs['bridge']:
-        config_provided += ",bridge={0}".format(kwargs['bridge'])
+        if kwargs['tag']:
+            config_provided += ",tag={0}".format(kwargs['tag'])
 
-    if kwargs['firewall']:
-        config_provided += ",firewall=1"
+        if kwargs['trunks']:
+            config_provided += ",trunks={0}".format(';'.join(str(x) for x in kwargs['trunks']))
 
-    if kwargs['link_down']:
-        config_provided += ',link_down=1'
+        net = {interface: config_provided}
+        vm = self.get_vm(vmid)
 
-    if kwargs['mtu']:
-        config_provided += ",mtu={0}".format(kwargs['mtu'])
-        if model != 'virtio':
-            module.warn(
-                'Ignoring MTU for nic {0} on VM with vmid {1}, '
-                'model should be set to \'virtio\': '.format(interface, vmid))
+        if ((interface not in vminfo) or (vminfo[interface] != config_provided)):
+            if not self.module.check_mode:
+                self.proxmox_api.nodes(vm['node']).qemu(vmid).config.set(**net)
+            return True
 
-    if kwargs['queues']:
-        config_provided += ",queues={0}".format(kwargs['queues'])
+        return False
 
-    if kwargs['rate']:
-        config_provided += ",rate={0}".format(kwargs['rate'])
+    def delete_nic(self, vmid, interface):
+        vm = self.get_vm(vmid)
+        vminfo = self.proxmox_api.nodes(vm['node']).qemu(vmid).config.get()
 
-    if kwargs['tag']:
-        config_provided += ",tag={0}".format(kwargs['tag'])
+        if interface in vminfo:
+            if not self.module.check_mode:
+                self.proxmox_api.nodes(vm['node']).qemu(vmid).config.set(vmid=vmid, delete=interface)
+            return True
 
-    if kwargs['trunks']:
-        config_provided += ",trunks={0}".format(';'.join(str(x) for x in kwargs['trunks']))
-
-    net = {interface: config_provided}
-    vm = get_vm(proxmox, vmid)
-
-    if ((interface not in vminfo) or (vminfo[interface] != config_provided)):
-        if not module.check_mode:
-            proxmox.nodes(vm[0]['node']).qemu(vmid).config.set(**net)
-        return True
-
-    return False
-
-
-def delete_nic(module, proxmox, vmid, interface):
-    vm = get_vm(proxmox, vmid)
-    vminfo = proxmox.nodes(vm[0]['node']).qemu(vmid).config.get()
-
-    if interface in vminfo:
-        if not module.check_mode:
-            proxmox.nodes(vm[0]['node']).qemu(vmid).config.set(vmid=vmid, delete=interface)
-        return True
-
-    return False
+        return False
 
 
 def main():
@@ -281,53 +257,33 @@ def main():
         supports_check_mode=True,
     )
 
-    if not HAS_PROXMOXER:
-        module.fail_json(msg='proxmoxer required for this module')
+    proxmox = ProxmoxNicAnsible(module)
 
-    api_host = module.params['api_host']
-    api_password = module.params['api_password']
-    api_token_id = module.params['api_token_id']
-    api_token_secret = module.params['api_token_secret']
-    api_user = module.params['api_user']
     interface = module.params['interface']
     model = module.params['model']
     name = module.params['name']
     state = module.params['state']
-    validate_certs = module.params['validate_certs']
     vmid = module.params['vmid']
 
-    auth_args = {'user': api_user}
-    if not (api_token_id and api_token_secret):
-        auth_args['password'] = api_password
-    else:
-        auth_args['token_name'] = api_token_id
-        auth_args['token_value'] = api_token_secret
-
-    try:
-        proxmox = ProxmoxAPI(api_host, verify_ssl=validate_certs, **auth_args)
-    except Exception as e:
-        module.fail_json(msg='authorization on proxmox cluster failed with exception: %s' % e)
-
     # If vmid is not defined then retrieve its value from the vm name,
     if not vmid:
-        vmid = get_vmid(module, proxmox, name)
+        vmid = proxmox.get_vmid(name)
 
     # Ensure VM id exists
-    if not get_vm(proxmox, vmid):
-        module.fail_json(vmid=vmid, msg='VM with vmid = %s does not exist in cluster' % vmid)
+    proxmox.get_vm(vmid)
 
     if state == 'present':
         try:
-            if update_nic(module, proxmox, vmid, interface, model,
-                          bridge=module.params['bridge'],
-                          firewall=module.params['firewall'],
-                          link_down=module.params['link_down'],
-                          mac=module.params['mac'],
-                          mtu=module.params['mtu'],
-                          queues=module.params['queues'],
-                          rate=module.params['rate'],
-                          tag=module.params['tag'],
-                          trunks=module.params['trunks']):
+            if proxmox.update_nic(vmid, interface, model,
+                                  bridge=module.params['bridge'],
+                                  firewall=module.params['firewall'],
+                                  link_down=module.params['link_down'],
+                                  mac=module.params['mac'],
+                                  mtu=module.params['mtu'],
+                                  queues=module.params['queues'],
+                                  rate=module.params['rate'],
+                                  tag=module.params['tag'],
+                                  trunks=module.params['trunks']):
                 module.exit_json(changed=True, vmid=vmid, msg="Nic {0} updated on VM with vmid {1}".format(interface, vmid))
             else:
                 module.exit_json(vmid=vmid, msg="Nic {0} unchanged on VM with vmid {1}".format(interface, vmid))
@@ -336,7 +292,7 @@ def main():
 
     elif state == 'absent':
         try:
-            if delete_nic(module, proxmox, vmid, interface):
+            if proxmox.delete_nic(vmid, interface):
                 module.exit_json(changed=True, vmid=vmid, msg="Nic {0} deleted on VM with vmid {1}".format(interface, vmid))
             else:
                 module.exit_json(vmid=vmid, msg="Nic {0} does not exist on VM with vmid {1}".format(interface, vmid))

plugins/modules/cloud/misc/proxmox_snap.py

@@ -1,6 +1,6 @@
 #!/usr/bin/python
 # -*- coding: utf-8 -*-
-
+#
 # Copyright: (c) 2020, Jeffrey van Pelt (@Thulium-Drake) <jeff@vanpelt.one>
 # GNU General Public License v3.0+ (see COPYING or https://www.gnu.org/licenses/gpl-3.0.txt)
 
@@ -16,22 +16,6 @@ description:
   - Allows you to create/delete snapshots from instances in Proxmox VE cluster.
   - Supports both KVM and LXC, OpenVZ has not been tested, as it is no longer supported on Proxmox VE.
 options:
-  api_host:
-    description:
-      - The host of the Proxmox VE cluster.
-    required: true
-    type: str
-  api_user:
-    description:
-      - The user to authenticate with.
-    required: true
-    type: str
-  api_password:
-    description:
-      - The password to authenticate with.
-      - You can use PROXMOX_PASSWORD environment variable.
-    type: str
-    required: yes
   hostname:
     description:
       - The instance name.
@@ -41,11 +25,6 @@ options:
       - The instance id.
       - If not set, will be fetched from PromoxAPI based on the hostname.
     type: str
-  validate_certs:
-    description:
-      - Enable / disable https certificate verification.
-    type: bool
-    default: no
   state:
     description:
      - Indicate desired state of the instance snapshot.
@@ -83,6 +62,8 @@ notes:
   - Supports C(check_mode).
 requirements: [ "proxmoxer", "python >= 2.7", "requests" ]
 author: Jeffrey van Pelt (@Thulium-Drake)
+extends_documentation_fragment:
+    - community.general.proxmox.documentation
 '''
 
 EXAMPLES = r'''
@@ -110,102 +91,76 @@ RETURN = r'''#'''
 import time
 import traceback
 
-PROXMOXER_IMP_ERR = None
-try:
-    from proxmoxer import ProxmoxAPI
-    HAS_PROXMOXER = True
-except ImportError:
-    PROXMOXER_IMP_ERR = traceback.format_exc()
-    HAS_PROXMOXER = False
-
 from ansible.module_utils.basic import AnsibleModule, missing_required_lib, env_fallback
 from ansible.module_utils.common.text.converters import to_native
+from ansible_collections.community.general.plugins.module_utils.proxmox import (proxmox_auth_argument_spec, ProxmoxAnsible, HAS_PROXMOXER, PROXMOXER_IMP_ERR)
 
 
-VZ_TYPE = None
+class ProxmoxSnapAnsible(ProxmoxAnsible):
+    def snapshot(self, vm, vmid):
+        return getattr(self.proxmox_api.nodes(vm['node']), vm['type'])(vmid).snapshot
 
-
-def get_vmid(proxmox, hostname):
-    return [vm['vmid'] for vm in proxmox.cluster.resources.get(type='vm') if 'name' in vm and vm['name'] == hostname]
-
-
-def get_instance(proxmox, vmid):
-    return [vm for vm in proxmox.cluster.resources.get(type='vm') if int(vm['vmid']) == int(vmid)]
-
-
-def snapshot_create(module, proxmox, vm, vmid, timeout, snapname, description, vmstate):
-    if module.check_mode:
-        return True
-
-    if VZ_TYPE == 'lxc':
-        taskid = getattr(proxmox.nodes(vm[0]['node']), VZ_TYPE)(vmid).snapshot.post(snapname=snapname, description=description)
-    else:
-        taskid = getattr(proxmox.nodes(vm[0]['node']), VZ_TYPE)(vmid).snapshot.post(snapname=snapname, description=description, vmstate=int(vmstate))
-    while timeout:
-        if (proxmox.nodes(vm[0]['node']).tasks(taskid).status.get()['status'] == 'stopped' and
-                proxmox.nodes(vm[0]['node']).tasks(taskid).status.get()['exitstatus'] == 'OK'):
+    def snapshot_create(self, vm, vmid, timeout, snapname, description, vmstate):
+        if self.module.check_mode:
             return True
-        timeout -= 1
-        if timeout == 0:
-            module.fail_json(msg='Reached timeout while waiting for creating VM snapshot. Last line in task before timeout: %s' %
-                                 proxmox.nodes(vm[0]['node']).tasks(taskid).log.get()[:1])
 
-        time.sleep(1)
-    return False
+        if vm['type'] == 'lxc':
+            taskid = self.snapshot(vm, vmid).post(snapname=snapname, description=description)
+        else:
+            taskid = self.snapshot(vm, vmid).post(snapname=snapname, description=description, vmstate=int(vmstate))
+        while timeout:
+            if (self.proxmox_api.nodes(vm['node']).tasks(taskid).status.get()['status'] == 'stopped' and
+                    self.proxmox_api.nodes(vm['node']).tasks(taskid).status.get()['exitstatus'] == 'OK'):
+                return True
+            timeout -= 1
+            if timeout == 0:
+                self.module.fail_json(msg='Reached timeout while waiting for creating VM snapshot. Last line in task before timeout: %s' %
+                                      self.proxmox_api.nodes(vm['node']).tasks(taskid).log.get()[:1])
 
+            time.sleep(1)
+        return False
 
-def snapshot_remove(module, proxmox, vm, vmid, timeout, snapname, force):
-    if module.check_mode:
-        return True
-
-    taskid = getattr(proxmox.nodes(vm[0]['node']), VZ_TYPE)(vmid).snapshot.delete(snapname, force=int(force))
-    while timeout:
-        if (proxmox.nodes(vm[0]['node']).tasks(taskid).status.get()['status'] == 'stopped' and
-                proxmox.nodes(vm[0]['node']).tasks(taskid).status.get()['exitstatus'] == 'OK'):
+    def snapshot_remove(self, vm, vmid, timeout, snapname, force):
+        if self.module.check_mode:
             return True
-        timeout -= 1
-        if timeout == 0:
-            module.fail_json(msg='Reached timeout while waiting for removing VM snapshot. Last line in task before timeout: %s' %
-                                 proxmox.nodes(vm[0]['node']).tasks(taskid).log.get()[:1])
 
-        time.sleep(1)
-    return False
+        taskid = self.snapshot(vm, vmid).delete(snapname, force=int(force))
+        while timeout:
+            if (self.proxmox_api.nodes(vm['node']).tasks(taskid).status.get()['status'] == 'stopped' and
+                    self.proxmox_api.nodes(vm['node']).tasks(taskid).status.get()['exitstatus'] == 'OK'):
+                return True
+            timeout -= 1
+            if timeout == 0:
+                self.module.fail_json(msg='Reached timeout while waiting for removing VM snapshot. Last line in task before timeout: %s' %
+                                      self.proxmox_api.nodes(vm['node']).tasks(taskid).log.get()[:1])
 
-
-def setup_api(api_host, api_user, api_password, validate_certs):
-    api = ProxmoxAPI(api_host, user=api_user, password=api_password, verify_ssl=validate_certs)
-    return api
+            time.sleep(1)
+        return False
 
 
 def main():
+    module_args = proxmox_auth_argument_spec()
+    snap_args = dict(
+        vmid=dict(required=False),
+        hostname=dict(),
+        timeout=dict(type='int', default=30),
+        state=dict(default='present', choices=['present', 'absent']),
+        description=dict(type='str'),
+        snapname=dict(type='str', default='ansible_snap'),
+        force=dict(type='bool', default='no'),
+        vmstate=dict(type='bool', default='no'),
+    )
+    module_args.update(snap_args)
+
     module = AnsibleModule(
-        argument_spec=dict(
-            api_host=dict(required=True),
-            api_user=dict(required=True),
-            api_password=dict(no_log=True, required=True, fallback=(env_fallback, ['PROXMOX_PASSWORD'])),
-            vmid=dict(required=False),
-            validate_certs=dict(type='bool', default='no'),
-            hostname=dict(),
-            timeout=dict(type='int', default=30),
-            state=dict(default='present', choices=['present', 'absent']),
-            description=dict(type='str'),
-            snapname=dict(type='str', default='ansible_snap'),
-            force=dict(type='bool', default='no'),
-            vmstate=dict(type='bool', default='no'),
-        ),
+        argument_spec=module_args,
         supports_check_mode=True
     )
 
-    if not HAS_PROXMOXER:
-        module.fail_json(msg=missing_required_lib('proxmoxer'),
-                         exception=PROXMOXER_IMP_ERR)
+    proxmox = ProxmoxSnapAnsible(module)
 
     state = module.params['state']
-    api_user = module.params['api_user']
-    api_host = module.params['api_host']
-    api_password = module.params['api_password']
     vmid = module.params['vmid']
-    validate_certs = module.params['validate_certs']
     hostname = module.params['hostname']
     description = module.params['description']
     snapname = module.params['snapname']
@@ -213,37 +168,21 @@ def main():
     force = module.params['force']
     vmstate = module.params['vmstate']
 
-    try:
-        proxmox = setup_api(api_host, api_user, api_password, validate_certs)
-
-    except Exception as e:
-        module.fail_json(msg='authorization on proxmox cluster failed with exception: %s' % to_native(e))
-
     # If hostname is set get the VM id from ProxmoxAPI
     if not vmid and hostname:
-        hosts = get_vmid(proxmox, hostname)
-        if len(hosts) == 0:
-            module.fail_json(msg="Vmid could not be fetched => Hostname does not exist (action: %s)" % state)
-        vmid = hosts[0]
+        vmid = proxmox.get_vmid(hostname, choose_first_if_multiple=True)
     elif not vmid:
         module.exit_json(changed=False, msg="Vmid could not be fetched for the following action: %s" % state)
 
-    vm = get_instance(proxmox, vmid)
-
-    global VZ_TYPE
-    VZ_TYPE = vm[0]['type']
+    vm = proxmox.get_vm(vmid)
 
     if state == 'present':
         try:
-            vm = get_instance(proxmox, vmid)
-            if not vm:
-                module.fail_json(msg='VM with vmid = %s not exists in cluster' % vmid)
-
-            for i in getattr(proxmox.nodes(vm[0]['node']), VZ_TYPE)(vmid).snapshot.get():
+            for i in proxmox.snapshot(vm, vmid).get():
                 if i['name'] == snapname:
                     module.exit_json(changed=False, msg="Snapshot %s is already present" % snapname)
 
-            if snapshot_create(module, proxmox, vm, vmid, timeout, snapname, description, vmstate):
+            if proxmox.snapshot_create(vm, vmid, timeout, snapname, description, vmstate):
                 if module.check_mode:
                     module.exit_json(changed=False, msg="Snapshot %s would be created" % snapname)
                 else:
@@ -254,13 +193,9 @@ def main():
 
     elif state == 'absent':
         try:
-            vm = get_instance(proxmox, vmid)
-            if not vm:
-                module.fail_json(msg='VM with vmid = %s not exists in cluster' % vmid)
-
             snap_exist = False
 
-            for i in getattr(proxmox.nodes(vm[0]['node']), VZ_TYPE)(vmid).snapshot.get():
+            for i in proxmox.snapshot(vm, vmid).get():
                 if i['name'] == snapname:
                     snap_exist = True
                     continue
@@ -268,7 +203,7 @@ def main():
             if not snap_exist:
                 module.exit_json(changed=False, msg="Snapshot %s does not exist" % snapname)
             else:
-                if snapshot_remove(module, proxmox, vm, vmid, timeout, snapname, force):
+                if proxmox.snapshot_remove(vm, vmid, timeout, snapname, force):
                     if module.check_mode:
                         module.exit_json(changed=False, msg="Snapshot %s would be removed" % snapname)
                     else:

plugins/modules/cloud/misc/proxmox_template.py

@@ -2,7 +2,6 @@
 # -*- coding: utf-8 -*-
 #
 # Copyright: Ansible Project
-#
 # GNU General Public License v3.0+ (see COPYING or https://www.gnu.org/licenses/gpl-3.0.txt)
 
 from __future__ import absolute_import, division, print_function
@@ -117,112 +116,81 @@ EXAMPLES = '''
 import os
 import time
 
-try:
-    from proxmoxer import ProxmoxAPI
-    HAS_PROXMOXER = True
-except ImportError:
-    HAS_PROXMOXER = False
-
 from ansible.module_utils.basic import AnsibleModule, env_fallback
+from ansible_collections.community.general.plugins.module_utils.proxmox import (proxmox_auth_argument_spec, ProxmoxAnsible)
 
 
-def get_template(proxmox, node, storage, content_type, template):
-    return [True for tmpl in proxmox.nodes(node).storage(storage).content.get()
-            if tmpl['volid'] == '%s:%s/%s' % (storage, content_type, template)]
+class ProxmoxTemplateAnsible(ProxmoxAnsible):
+    def get_template(self, node, storage, content_type, template):
+        return [True for tmpl in self.proxmox_api.nodes(node).storage(storage).content.get()
+                if tmpl['volid'] == '%s:%s/%s' % (storage, content_type, template)]
 
+    def task_status(self, node, taskid, timeout):
+        """
+        Check the task status and wait until the task is completed or the timeout is reached.
+        """
+        while timeout:
+            task_status = self.proxmox_api.nodes(node).tasks(taskid).status.get()
+            if task_status['status'] == 'stopped' and task_status['exitstatus'] == 'OK':
+                return True
+            timeout = timeout - 1
+            if timeout == 0:
+                self.module.fail_json(msg='Reached timeout while waiting for uploading/downloading template. Last line in task before timeout: %s' %
+                                      self.proxmox_api.node(node).tasks(taskid).log.get()[:1])
 
-def task_status(module, proxmox, node, taskid, timeout):
-    """
-    Check the task status and wait until the task is completed or the timeout is reached.
-    """
-    while timeout:
-        task_status = proxmox.nodes(node).tasks(taskid).status.get()
-        if task_status['status'] == 'stopped' and task_status['exitstatus'] == 'OK':
-            return True
-        timeout = timeout - 1
-        if timeout == 0:
-            module.fail_json(msg='Reached timeout while waiting for uploading/downloading template. Last line in task before timeout: %s'
-                                 % proxmox.node(node).tasks(taskid).log.get()[:1])
+            time.sleep(1)
+        return False
 
-        time.sleep(1)
-    return False
+    def upload_template(self, node, storage, content_type, realpath, timeout):
+        taskid = self.proxmox_api.nodes(node).storage(storage).upload.post(content=content_type, filename=open(realpath, 'rb'))
+        return self.task_status(node, taskid, timeout)
 
+    def download_template(self, node, storage, template, timeout):
+        taskid = self.proxmox_api.nodes(node).aplinfo.post(storage=storage, template=template)
+        return self.task_status(node, taskid, timeout)
 
-def upload_template(module, proxmox, node, storage, content_type, realpath, timeout):
-    taskid = proxmox.nodes(node).storage(storage).upload.post(content=content_type, filename=open(realpath, 'rb'))
-    return task_status(module, proxmox, node, taskid, timeout)
+    def delete_template(self, node, storage, content_type, template, timeout):
+        volid = '%s:%s/%s' % (storage, content_type, template)
+        self.proxmox_api.nodes(node).storage(storage).content.delete(volid)
+        while timeout:
+            if not self.get_template(node, storage, content_type, template):
+                return True
+            timeout = timeout - 1
+            if timeout == 0:
+                self.module.fail_json(msg='Reached timeout while waiting for deleting template.')
 
-
-def download_template(module, proxmox, node, storage, template, timeout):
-    taskid = proxmox.nodes(node).aplinfo.post(storage=storage, template=template)
-    return task_status(module, proxmox, node, taskid, timeout)
-
-
-def delete_template(module, proxmox, node, storage, content_type, template, timeout):
-    volid = '%s:%s/%s' % (storage, content_type, template)
-    proxmox.nodes(node).storage(storage).content.delete(volid)
-    while timeout:
-        if not get_template(proxmox, node, storage, content_type, template):
-            return True
-        timeout = timeout - 1
-        if timeout == 0:
-            module.fail_json(msg='Reached timeout while waiting for deleting template.')
-
-        time.sleep(1)
-    return False
+            time.sleep(1)
+        return False
 
 
 def main():
+    module_args = proxmox_auth_argument_spec()
+    template_args = dict(
+        node=dict(),
+        src=dict(type='path'),
+        template=dict(),
+        content_type=dict(default='vztmpl', choices=['vztmpl', 'iso']),
+        storage=dict(default='local'),
+        timeout=dict(type='int', default=30),
+        force=dict(type='bool', default=False),
+        state=dict(default='present', choices=['present', 'absent']),
+    )
+    module_args.update(template_args)
+
     module = AnsibleModule(
-        argument_spec=dict(
-            api_host=dict(required=True),
-            api_password=dict(no_log=True, fallback=(env_fallback, ['PROXMOX_PASSWORD'])),
-            api_token_id=dict(no_log=True),
-            api_token_secret=dict(no_log=True),
-            api_user=dict(required=True),
-            validate_certs=dict(type='bool', default=False),
-            node=dict(),
-            src=dict(type='path'),
-            template=dict(),
-            content_type=dict(default='vztmpl', choices=['vztmpl', 'iso']),
-            storage=dict(default='local'),
-            timeout=dict(type='int', default=30),
-            force=dict(type='bool', default=False),
-            state=dict(default='present', choices=['present', 'absent']),
-        ),
+        argument_spec=module_args,
         required_together=[('api_token_id', 'api_token_secret')],
         required_one_of=[('api_password', 'api_token_id')],
         required_if=[('state', 'absent', ['template'])]
     )
 
-    if not HAS_PROXMOXER:
-        module.fail_json(msg='proxmoxer required for this module')
+    proxmox = ProxmoxTemplateAnsible(module)
 
     state = module.params['state']
-    api_host = module.params['api_host']
-    api_password = module.params['api_password']
-    api_token_id = module.params['api_token_id']
-    api_token_secret = module.params['api_token_secret']
-    api_user = module.params['api_user']
-    validate_certs = module.params['validate_certs']
     node = module.params['node']
     storage = module.params['storage']
     timeout = module.params['timeout']
 
-    auth_args = {'user': api_user}
-    if not (api_token_id and api_token_secret):
-        auth_args['password'] = api_password
-    else:
-        auth_args['token_name'] = api_token_id
-        auth_args['token_value'] = api_token_secret
-
-    try:
-        proxmox = ProxmoxAPI(api_host, verify_ssl=validate_certs, **auth_args)
-        # Used to test the validity of the token if given
-        proxmox.version.get()
-    except Exception as e:
-        module.fail_json(msg='authorization on proxmox cluster failed with exception: %s' % e)
-
     if state == 'present':
         try:
             content_type = module.params['content_type']
@@ -235,21 +203,21 @@ def main():
                 if not template:
                     module.fail_json(msg='template param for downloading appliance template is mandatory')
 
-                if get_template(proxmox, node, storage, content_type, template) and not module.params['force']:
+                if proxmox.get_template(node, storage, content_type, template) and not module.params['force']:
                     module.exit_json(changed=False, msg='template with volid=%s:%s/%s already exists' % (storage, content_type, template))
 
-                if download_template(module, proxmox, node, storage, template, timeout):
+                if proxmox.download_template(node, storage, template, timeout):
                     module.exit_json(changed=True, msg='template with volid=%s:%s/%s downloaded' % (storage, content_type, template))
 
             template = os.path.basename(src)
-            if get_template(proxmox, node, storage, content_type, template) and not module.params['force']:
+            if proxmox.get_template(node, storage, content_type, template) and not module.params['force']:
                 module.exit_json(changed=False, msg='template with volid=%s:%s/%s is already exists' % (storage, content_type, template))
             elif not src:
                 module.fail_json(msg='src param to uploading template file is mandatory')
             elif not (os.path.exists(src) and os.path.isfile(src)):
                 module.fail_json(msg='template file on path %s not exists' % src)
 
-            if upload_template(module, proxmox, node, storage, content_type, src, timeout):
+            if proxmox.upload_template(node, storage, content_type, src, timeout):
                 module.exit_json(changed=True, msg='template with volid=%s:%s/%s uploaded' % (storage, content_type, template))
         except Exception as e:
             module.fail_json(msg="uploading/downloading of template %s failed with exception: %s" % (template, e))
@@ -259,10 +227,10 @@ def main():
             content_type = module.params['content_type']
             template = module.params['template']
 
-            if not get_template(proxmox, node, storage, content_type, template):
+            if not proxmox.get_template(node, storage, content_type, template):
                 module.exit_json(changed=False, msg='template with volid=%s:%s/%s is already deleted' % (storage, content_type, template))
 
-            if delete_template(module, proxmox, node, storage, content_type, template, timeout):
+            if proxmox.delete_template(node, storage, content_type, template, timeout):
                 module.exit_json(changed=True, msg='template with volid=%s:%s/%s deleted' % (storage, content_type, template))
         except Exception as e:
             module.fail_json(msg="deleting of template %s failed with exception: %s" % (template, e))

plugins/modules/cloud/scaleway/scaleway_private_network.py

@@ -0,0 +1,234 @@
+#!/usr/bin/python
+# -*- coding: utf-8 -*-
+#
+# Scaleway VPC management module
+#
+# GNU General Public License v3.0+ (see COPYING or https://www.gnu.org/licenses/gpl-3.0.txt)
+
+from __future__ import absolute_import, division, print_function
+
+__metaclass__ = type
+
+DOCUMENTATION = '''
+---
+module: scaleway_private_network
+short_description: Scaleway private network management
+version_added: 4.5.0
+author: Pascal MANGIN (@pastral)
+description:
+    - This module manages private network on Scaleway account
+      (U(https://developer.scaleway.com)).
+extends_documentation_fragment:
+- community.general.scaleway
+
+
+options:
+  state:
+    type: str
+    description:
+     - Indicate desired state of the VPC.
+    default: present
+    choices:
+      - present
+      - absent
+
+  project:
+    type: str
+    description:
+      - Project identifier.
+    required: true
+
+  region:
+    type: str
+    description:
+     - Scaleway region to use (for example C(par1)).
+    required: true
+    choices:
+      - ams1
+      - EMEA-NL-EVS
+      - par1
+      - EMEA-FR-PAR1
+      - par2
+      - EMEA-FR-PAR2
+      - waw1
+      - EMEA-PL-WAW1
+
+  name:
+    type: str
+    description:
+    - Name of the VPC.
+
+  tags:
+    type: list
+    elements: str
+    description:
+    - List of tags to apply to the instance.
+    default: []
+
+'''
+
+EXAMPLES = '''
+- name: Create an private network
+  community.general.scaleway_vpc:
+    project: '{{ scw_project }}'
+    name: 'vpc_one'
+    state: present
+    region: par1
+  register: vpc_creation_task
+
+- name: Make sure private network with name 'foo' is deleted in region par1
+  community.general.scaleway_vpc:
+    name: 'foo'
+    state: absent
+    region: par1
+'''
+
+RETURN = '''
+scaleway_private_network:
+    description: Information on the VPC.
+    returned: success when C(state=present)
+    type: dict
+    sample:
+        {
+            "created_at": "2022-01-15T11:11:12.676445Z",
+            "id": "12345678-f1e6-40ec-83e5-12345d67ed89",
+            "name": "network",
+            "organization_id": "a123b4cd-ef5g-678h-90i1-jk2345678l90",
+            "project_id": "a123b4cd-ef5g-678h-90i1-jk2345678l90",
+            "tags": [
+                "tag1",
+                "tag2",
+                "tag3",
+                "tag4",
+                "tag5"
+            ],
+            "updated_at": "2022-01-15T11:12:04.624837Z",
+            "zone": "fr-par-2"
+        }
+'''
+
+from ansible_collections.community.general.plugins.module_utils.scaleway import SCALEWAY_LOCATION, scaleway_argument_spec, Scaleway
+from ansible.module_utils.basic import AnsibleModule
+
+
+def get_private_network(api, name, page=1):
+    page_size = 10
+    response = api.get('private-networks', params={'name': name, 'order_by': 'name_asc', 'page': page, 'page_size': page_size})
+    if not response.ok:
+        msg = "Error during get private network creation: %s: '%s' (%s)" % (response.info['msg'], response.json['message'], response.json)
+        api.module.fail_json(msg=msg)
+
+    if response.json['total_count'] == 0:
+        return None
+
+    i = 0
+    while i < len(response.json['private_networks']):
+        if response.json['private_networks'][i]['name'] == name:
+            return response.json['private_networks'][i]
+        i += 1
+
+    # search on next page if needed
+    if (page * page_size) < response.json['total_count']:
+        return get_private_network(api, name, page + 1)
+
+    return None
+
+
+def present_strategy(api, wished_private_network):
+
+    changed = False
+    private_network = get_private_network(api, wished_private_network['name'])
+    if private_network is not None:
+        if set(wished_private_network['tags']) == set(private_network['tags']):
+            return changed, private_network
+        else:
+            # private network need to be updated
+            data = {'name': wished_private_network['name'],
+                    'tags': wished_private_network['tags']
+                    }
+            changed = True
+            if api.module.check_mode:
+                return changed, {"status": "private network would be updated"}
+
+            response = api.patch(path='private-networks/' + private_network['id'], data=data)
+            if not response.ok:
+                api.module.fail_json(msg='Error updating private network [{0}: {1}]'.format(response.status_code, response.json))
+
+            return changed, response.json
+
+    # private network need to be create
+    changed = True
+    if api.module.check_mode:
+        return changed, {"status": "private network would be created"}
+
+    data = {'name': wished_private_network['name'],
+            'project_id': wished_private_network['project'],
+            'tags': wished_private_network['tags']
+            }
+
+    response = api.post(path='private-networks/', data=data)
+
+    if not response.ok:
+        api.module.fail_json(msg='Error creating private network [{0}: {1}]'.format(response.status_code, response.json))
+
+    return changed, response.json
+
+
+def absent_strategy(api, wished_private_network):
+
+    changed = False
+    private_network = get_private_network(api, wished_private_network['name'])
+    if private_network is None:
+        return changed, {}
+
+    changed = True
+    if api.module.check_mode:
+        return changed, {"status": "private network would be destroyed"}
+
+    response = api.delete('private-networks/' + private_network['id'])
+
+    if not response.ok:
+        api.module.fail_json(msg='Error deleting private network [{0}: {1}]'.format(
+            response.status_code, response.json))
+
+    return changed, response.json
+
+
+def core(module):
+
+    wished_private_network = {
+        "project": module.params['project'],
+        "tags": module.params['tags'],
+        "name": module.params['name']
+    }
+
+    region = module.params["region"]
+    module.params['api_url'] = SCALEWAY_LOCATION[region]["api_endpoint_vpc"]
+
+    api = Scaleway(module=module)
+    if module.params["state"] == "absent":
+        changed, summary = absent_strategy(api=api, wished_private_network=wished_private_network)
+    else:
+        changed, summary = present_strategy(api=api, wished_private_network=wished_private_network)
+    module.exit_json(changed=changed, scaleway_private_network=summary)
+
+
+def main():
+    argument_spec = scaleway_argument_spec()
+    argument_spec.update(dict(
+        state=dict(default='present', choices=['absent', 'present']),
+        project=dict(required=True),
+        region=dict(required=True, choices=list(SCALEWAY_LOCATION.keys())),
+        tags=dict(type="list", elements="str", default=[]),
+        name=dict()
+    ))
+    module = AnsibleModule(
+        argument_spec=argument_spec,
+        supports_check_mode=True,
+    )
+
+    core(module)
+
+
+if __name__ == '__main__':
+    main()

plugins/modules/files/ini_file.py

@@ -367,9 +367,10 @@ def do_ini(module, filename, section=None, option=None, values=None,
                     section_lines = new_section_lines
         else:
             # drop the entire section
-            section_lines = []
-            msg = 'section removed'
-            changed = True
+            if section_lines:
+                section_lines = []
+                msg = 'section removed'
+                changed = True
 
     # reassemble the ini_lines after manipulation
     ini_lines = before + section_lines + after

plugins/modules/identity/keycloak/keycloak_user_federation.py

@@ -845,7 +845,7 @@ def main():
         before_comp = {}
 
     # if user federation exists, get associated mappers
-    if cid is not None:
+    if cid is not None and before_comp:
         before_comp['mappers'] = sorted(kc.get_components(urlencode(dict(parent=cid)), realm), key=lambda x: x.get('name'))
 
     # Build a proposed changeset from parameters given to this module
@@ -921,12 +921,23 @@ def main():
         after_comp = kc.create_component(desired_comp, realm)
 
         for mapper in updated_mappers:
-            if mapper.get('id') is not None:
-                kc.update_component(mapper, realm)
+            found = kc.get_components(urlencode(dict(parent=cid, name=mapper['name'])), realm)
+            if len(found) > 1:
+                module.fail_json(msg='Found multiple mappers with name `{name}`. Cannot continue.'.format(name=mapper['name']))
+            if len(found) == 1:
+                old_mapper = found[0]
             else:
-                if mapper.get('parentId') is None:
-                    mapper['parentId'] = after_comp['id']
-                mapper = kc.create_component(mapper, realm)
+                old_mapper = {}
+
+            new_mapper = old_mapper.copy()
+            new_mapper.update(mapper)
+
+            if new_mapper.get('id') is not None:
+                kc.update_component(new_mapper, realm)
+            else:
+                if new_mapper.get('parentId') is None:
+                    new_mapper['parentId'] = after_comp['id']
+                mapper = kc.create_component(new_mapper, realm)
 
         after_comp['mappers'] = updated_mappers
         result['end_state'] = sanitize(after_comp)

plugins/modules/packaging/language/ansible_galaxy_install.py

@@ -53,6 +53,12 @@ options:
         Please notice that C(ansible-galaxy) will not install collections with I(type=both), when I(requirements_file)
         contains both roles and collections and I(dest) is specified.
     type: path
+  no_deps:
+    description:
+      - Refrain from installing dependencies.
+    version_added: 4.5.0
+    type: bool
+    default: false
   force:
     description:
       - Force overwriting an existing role or collection.
@@ -178,7 +184,7 @@ class AnsibleGalaxyInstall(CmdModuleHelper):
     ansible_version = None
     is_ansible29 = None
 
-    output_params = ('type', 'name', 'dest', 'requirements_file', 'force')
+    output_params = ('type', 'name', 'dest', 'requirements_file', 'force', 'no_deps')
     module = dict(
         argument_spec=dict(
             type=dict(type='str', choices=('collection', 'role', 'both'), required=True),
@@ -186,6 +192,7 @@ class AnsibleGalaxyInstall(CmdModuleHelper):
             requirements_file=dict(type='path'),
             dest=dict(type='path'),
             force=dict(type='bool', default=False),
+            no_deps=dict(type='bool', default=False),
             ack_ansible29=dict(type='bool', default=False),
         ),
         mutually_exclusive=[('name', 'requirements_file')],
@@ -201,6 +208,7 @@ class AnsibleGalaxyInstall(CmdModuleHelper):
         requirements_file=dict(fmt=('-r', '{0}'),),
         dest=dict(fmt=('-p', '{0}'),),
         force=dict(fmt="--force", style=ArgFormat.BOOLEAN),
+        no_deps=dict(fmt="--no-deps", style=ArgFormat.BOOLEAN),
     )
     force_lang = "en_US.UTF-8"
     check_rc = True
@@ -293,7 +301,7 @@ class AnsibleGalaxyInstall(CmdModuleHelper):
             self._setup29()
         else:
             self._setup210plus()
-        params = ('type', {'galaxy_cmd': 'install'}, 'force', 'dest', 'requirements_file', 'name')
+        params = ('type', {'galaxy_cmd': 'install'}, 'force', 'no_deps', 'dest', 'requirements_file', 'name')
         self.run_command(params=params)
 
     def process_command_output(self, rc, out, err):

plugins/modules/packaging/os/homebrew_cask.py

@@ -102,6 +102,12 @@ EXAMPLES = '''
     state: present
     install_options: 'debug,appdir=/Applications'
 
+- name: Install cask with force option
+  community.general.homebrew_cask:
+    name: alfred
+    state: present
+    install_options: force
+
 - name: Allow external app
   community.general.homebrew_cask:
     name: alfred
@@ -600,7 +606,7 @@ class HomebrewCask(object):
             self.message = 'Invalid cask: {0}.'.format(self.current_cask)
             raise HomebrewCaskException(self.message)
 
-        if self._current_cask_is_installed():
+        if '--force' not in self.install_options and self._current_cask_is_installed():
             self.unchanged_count += 1
             self.message = 'Cask already installed: {0}'.format(
                 self.current_cask,

plugins/modules/packaging/os/pacman.py

@@ -4,12 +4,14 @@
 # Copyright: (c) 2012, Afterburn <https://github.com/afterburn>
 # Copyright: (c) 2013, Aaron Bull Schaefer <aaron@elasticdog.com>
 # Copyright: (c) 2015, Indrajit Raychaudhuri <irc+code@indrajit.com>
+# Copyright: (c) 2022, Jean Raby <jean@raby.sh>
 # GNU General Public License v3.0+ (see COPYING or https://www.gnu.org/licenses/gpl-3.0.txt)
 
 from __future__ import absolute_import, division, print_function
+
 __metaclass__ = type
 
-DOCUMENTATION = '''
+DOCUMENTATION = """
 ---
 module: pacman
 short_description: Manage packages with I(pacman)
@@ -19,6 +21,7 @@ author:
     - Indrajit Raychaudhuri (@indrajitr)
     - Aaron Bull Schaefer (@elasticdog) <aaron@elasticdog.com>
     - Maxime de Roucy (@tchernomax)
+    - Jean Raby (@jraby)
 options:
     name:
         description:
@@ -66,7 +69,7 @@ options:
             - Whether or not to refresh the master package lists.
             - This can be run as part of a package installation or as a separate step.
             - Alias C(update-cache) has been deprecated and will be removed in community.general 5.0.0.
-        default: no
+            - If not specified, it defaults to C(false).
         type: bool
         aliases: [ update-cache ]
 
@@ -80,7 +83,7 @@ options:
         description:
             - Whether or not to upgrade the whole system.
               Can't be used in combination with C(name).
-        default: no
+            - If not specified, it defaults to C(false).
         type: bool
 
     upgrade_extra_args:
@@ -94,9 +97,9 @@ notes:
     it is much more efficient to pass the list directly to the I(name) option.
   - To use an AUR helper (I(executable) option), a few extra setup steps might be required beforehand.
     For example, a dedicated build user with permissions to install packages could be necessary.
-'''
+"""
 
-RETURN = '''
+RETURN = """
 packages:
     description: a list of packages that have been changed
     returned: when upgrade is set to yes
@@ -116,9 +119,9 @@ stderr:
     type: str
     sample: "warning: libtool: local (2.4.6+44+gb9b44533-14) is newer than core (2.4.6+42+gb88cebd5-15)\nwarning ..."
     version_added: 4.1.0
-'''
+"""
 
-EXAMPLES = '''
+EXAMPLES = """
 - name: Install package foo from repo
   community.general.pacman:
     name: foo
@@ -180,357 +183,468 @@ EXAMPLES = '''
     name: baz
     state: absent
     force: yes
-'''
-
-import re
+"""
 
+import shlex
 from ansible.module_utils.basic import AnsibleModule
+from collections import defaultdict, namedtuple
 
 
-def get_version(pacman_output):
-    """Take pacman -Q or pacman -S output and get the Version"""
-    fields = pacman_output.split()
-    if len(fields) == 2:
-        return fields[1]
-    return None
+Package = namedtuple("Package", ["name", "source"])
+VersionTuple = namedtuple("VersionTuple", ["current", "latest"])
 
 
-def get_name(module, pacman_output):
-    """Take pacman -Q or pacman -S output and get the package name"""
-    fields = pacman_output.split()
-    if len(fields) == 2:
-        return fields[0]
-    module.fail_json(msg="get_name: fail to retrieve package name from pacman output")
+class Pacman(object):
+    def __init__(self, module):
+        self.m = module
 
+        self.m.run_command_environ_update = dict(LC_ALL="C")
+        p = self.m.params
 
-def query_package(module, pacman_path, name, state):
-    """Query the package status in both the local system and the repository. Returns a boolean to indicate if the package is installed, a second
-    boolean to indicate if the package is up-to-date and a third boolean to indicate whether online information were available
-    """
+        self._msgs = []
+        self._stdouts = []
+        self._stderrs = []
+        self.changed = False
+        self.exit_params = {}
 
-    lcmd = "%s --query %s" % (pacman_path, name)
-    lrc, lstdout, lstderr = module.run_command(lcmd, check_rc=False)
-    if lrc != 0:
-        # package is not installed locally
-        return False, False, False
-    else:
-        # a non-zero exit code doesn't always mean the package is installed
-        # for example, if the package name queried is "provided" by another package
-        installed_name = get_name(module, lstdout)
-        if installed_name != name:
-            return False, False, False
+        self.pacman_path = self.m.get_bin_path(p["executable"], True)
 
-    # no need to check the repository if state is present or absent
-    # return False for package version check, because we didn't check it
-    if state == 'present' or state == 'absent':
-        return True, False, False
+        # Normalize for old configs
+        if p["state"] == "installed":
+            self.target_state = "present"
+        elif p["state"] == "removed":
+            self.target_state = "absent"
+        else:
+            self.target_state = p["state"]
 
-    # get the version installed locally (if any)
-    lversion = get_version(lstdout)
+    def add_exit_infos(self, msg=None, stdout=None, stderr=None):
+        if msg:
+            self._msgs.append(msg)
+        if stdout:
+            self._stdouts.append(stdout)
+        if stderr:
+            self._stderrs.append(stderr)
 
-    rcmd = "%s --sync --print-format \"%%n %%v\" %s" % (pacman_path, name)
-    rrc, rstdout, rstderr = module.run_command(rcmd, check_rc=False)
-    # get the version in the repository
-    rversion = get_version(rstdout)
+    def _set_mandatory_exit_params(self):
+        msg = "\n".join(self._msgs)
+        stdouts = "\n".join(self._stdouts)
+        stderrs = "\n".join(self._stderrs)
+        if stdouts:
+            self.exit_params["stdout"] = stdouts
+        if stderrs:
+            self.exit_params["stderr"] = stderrs
+        self.exit_params["msg"] = msg  # mandatory, but might be empty
 
-    if rrc == 0:
-        # Return True to indicate that the package is installed locally, and the result of the version number comparison
-        # to determine if the package is up-to-date.
-        return True, (lversion == rversion), False
+    def fail(self, msg=None, stdout=None, stderr=None, **kwargs):
+        self.add_exit_infos(msg, stdout, stderr)
+        self._set_mandatory_exit_params()
+        if kwargs:
+            self.exit_params.update(**kwargs)
+        self.m.fail_json(**self.exit_params)
 
-    # package is installed but cannot fetch remote Version. Last True stands for the error
-    return True, True, True
+    def success(self):
+        self._set_mandatory_exit_params()
+        self.m.exit_json(changed=self.changed, **self.exit_params)
 
+    def run(self):
+        if self.m.params["update_cache"]:
+            self.update_package_db()
 
-def update_package_db(module, pacman_path):
-    if module.params['force']:
-        module.params["update_cache_extra_args"] += " --refresh --refresh"
+            if not (self.m.params["name"] or self.m.params["upgrade"]):
+                self.success()
 
-    cmd = "%s --sync --refresh %s" % (pacman_path, module.params["update_cache_extra_args"])
-    rc, stdout, stderr = module.run_command(cmd, check_rc=False)
+        self.inventory = self._build_inventory()
+        if self.m.params["upgrade"]:
+            self.upgrade()
+            self.success()
 
-    if rc == 0:
-        return stdout, stderr
-    else:
-        module.fail_json(msg="could not update package db", stdout=stdout, stderr=stderr)
+        if self.m.params["name"]:
+            pkgs = self.package_list()
 
-
-def upgrade(module, pacman_path):
-    cmdupgrade = "%s --sync --sysupgrade --quiet --noconfirm %s" % (pacman_path, module.params["upgrade_extra_args"])
-    cmdneedrefresh = "%s --query --upgrades" % (pacman_path)
-    rc, stdout, stderr = module.run_command(cmdneedrefresh, check_rc=False)
-    data = stdout.split('\n')
-    data.remove('')
-    packages = []
-    diff = {
-        'before': '',
-        'after': '',
-    }
-
-    if rc == 0:
-        # Match lines of `pacman -Qu` output of the form:
-        #   (package name) (before version-release) -> (after version-release)
-        # e.g., "ansible 2.7.1-1 -> 2.7.2-1"
-        regex = re.compile(r'([\w+\-.@]+) (\S+-\S+) -> (\S+-\S+)')
-        for p in data:
-            if '[ignored]' not in p:
-                m = regex.search(p)
-                packages.append(m.group(1))
-                if module._diff:
-                    diff['before'] += "%s-%s\n" % (m.group(1), m.group(2))
-                    diff['after'] += "%s-%s\n" % (m.group(1), m.group(3))
-        if module.check_mode:
-            if packages:
-                module.exit_json(changed=True, msg="%s package(s) would be upgraded" % (len(data)), packages=packages, diff=diff)
+            if self.target_state == "absent":
+                self.remove_packages(pkgs)
+                self.success()
             else:
-                module.exit_json(changed=False, msg='Nothing to upgrade', packages=packages)
-        rc, stdout, stderr = module.run_command(cmdupgrade, check_rc=False)
+                self.install_packages(pkgs)
+                self.success()
+
+        # This shouldn't happen...
+        self.fail("This is a bug")
+
+    def install_packages(self, pkgs):
+        pkgs_to_install = []
+        for p in pkgs:
+            if (
+                p.name not in self.inventory["installed_pkgs"]
+                or self.target_state == "latest"
+                and p.name in self.inventory["upgradable_pkgs"]
+            ):
+                pkgs_to_install.append(p)
+
+        if len(pkgs_to_install) == 0:
+            self.add_exit_infos("package(s) already installed")
+            return
+
+        self.changed = True
+        cmd_base = [
+            self.pacman_path,
+            "--sync",
+            "--noconfirm",
+            "--noprogressbar",
+            "--needed",
+        ]
+        if self.m.params["extra_args"]:
+            cmd_base.extend(self.m.params["extra_args"])
+
+        # Dry run first to gather what will be done
+        cmd = cmd_base + ["--print-format", "%n %v"] + [p.source for p in pkgs_to_install]
+        rc, stdout, stderr = self.m.run_command(cmd, check_rc=False)
+        if rc != 0:
+            self.fail("Failed to list package(s) to install", stdout=stdout, stderr=stderr)
+
+        name_ver = [l.strip() for l in stdout.splitlines()]
+        before = []
+        after = []
+        installed_pkgs = []
+        self.exit_params["packages"] = []
+        for p in name_ver:
+            name, version = p.split()
+            if name in self.inventory["installed_pkgs"]:
+                before.append("%s-%s" % (name, self.inventory["installed_pkgs"][name]))
+            after.append("%s-%s" % (name, version))
+            installed_pkgs.append(name)
+
+        self.exit_params["diff"] = {
+            "before": "\n".join(before) + "\n" if before else "",
+            "after": "\n".join(after) + "\n" if after else "",
+        }
+
+        if self.m.check_mode:
+            self.add_exit_infos("Would have installed %d packages" % len(installed_pkgs))
+            self.exit_params["packages"] = installed_pkgs
+            return
+
+        # actually do it
+        cmd = cmd_base + [p.source for p in pkgs_to_install]
+
+        rc, stdout, stderr = self.m.run_command(cmd, check_rc=False)
+        if rc != 0:
+            self.fail("Failed to install package(s)", stdout=stdout, stderr=stderr)
+
+        self.exit_params["packages"] = installed_pkgs
+        self.add_exit_infos(
+            "Installed %d package(s)" % len(installed_pkgs), stdout=stdout, stderr=stderr
+        )
+
+    def remove_packages(self, pkgs):
+        force_args = ["--nodeps", "--nodeps"] if self.m.params["force"] else []
+
+        # filter out pkgs that are already absent
+        pkg_names_to_remove = [p.name for p in pkgs if p.name in self.inventory["installed_pkgs"]]
+
+        if len(pkg_names_to_remove) == 0:
+            self.add_exit_infos("package(s) already absent")
+            return
+
+        # There's something to do, set this in advance
+        self.changed = True
+
+        cmd_base = [self.pacman_path, "--remove", "--noconfirm", "--noprogressbar"]
+        if self.m.params["extra_args"]:
+            cmd_base.extend(self.m.params["extra_args"])
+        if force_args:
+            cmd_base.extend(force_args)
+
+        # This is a bit of a TOCTOU but it is better than parsing the output of
+        # pacman -R, which is different depending on the user config (VerbosePkgLists)
+        # Start by gathering what would be removed
+        cmd = cmd_base + ["--print-format", "%n-%v"] + pkg_names_to_remove
+
+        rc, stdout, stderr = self.m.run_command(cmd, check_rc=False)
+        if rc != 0:
+            self.fail("failed to list package(s) to remove", stdout=stdout, stderr=stderr)
+
+        removed_pkgs = stdout.split()
+        self.exit_params["packages"] = removed_pkgs
+        self.exit_params["diff"] = {
+            "before": "\n".join(removed_pkgs) + "\n",  # trailing \n to avoid diff complaints
+            "after": "",
+        }
+
+        if self.m.check_mode:
+            self.exit_params["packages"] = removed_pkgs
+            self.add_exit_infos("Would have removed %d packages" % len(removed_pkgs))
+            return
+
+        # actually do it
+        cmd = cmd_base + pkg_names_to_remove
+
+        rc, stdout, stderr = self.m.run_command(cmd, check_rc=False)
+        if rc != 0:
+            self.fail("failed to remove package(s)", stdout=stdout, stderr=stderr)
+        self.exit_params["packages"] = removed_pkgs
+        self.add_exit_infos("Removed %d package(s)" % len(removed_pkgs), stdout=stdout, stderr=stderr)
+
+    def upgrade(self):
+        """Runs pacman --sync --sysupgrade if there are upgradable packages"""
+
+        if len(self.inventory["upgradable_pkgs"]) == 0:
+            self.add_exit_infos("Nothing to upgrade")
+            return
+
+        self.changed = True  # there are upgrades, so there will be changes
+
+        # Build diff based on inventory first.
+        diff = {"before": "", "after": ""}
+        for pkg, versions in self.inventory["upgradable_pkgs"].items():
+            diff["before"] += "%s-%s\n" % (pkg, versions.current)
+            diff["after"] += "%s-%s\n" % (pkg, versions.latest)
+        self.exit_params["diff"] = diff
+        self.exit_params["packages"] = self.inventory["upgradable_pkgs"].keys()
+
+        if self.m.check_mode:
+            self.add_exit_infos(
+                "%d packages would have been upgraded" % (len(self.inventory["upgradable_pkgs"]))
+            )
+        else:
+            cmd = [
+                self.pacman_path,
+                "--sync",
+                "--sys-upgrade",
+                "--quiet",
+                "--noconfirm",
+            ]
+            if self.m.params["upgrade_extra_args"]:
+                cmd += self.m.params["upgrade_extra_args"]
+            rc, stdout, stderr = self.m.run_command(cmd, check_rc=False)
+            if rc == 0:
+                self.add_exit_infos("System upgraded", stdout=stdout, stderr=stderr)
+            else:
+                self.fail("Could not upgrade", stdout=stdout, stderr=stderr)
+
+    def update_package_db(self):
+        """runs pacman --sync --refresh"""
+        if self.m.check_mode:
+            self.add_exit_infos("Would have updated the package db")
+            self.changed = True
+            return
+
+        cmd = [
+            self.pacman_path,
+            "--sync",
+            "--refresh",
+        ]
+        if self.m.params["update_cache_extra_args"]:
+            cmd += self.m.params["update_cache_extra_args"]
+        if self.m.params["force"]:
+            cmd += ["--refresh"]
+
+        rc, stdout, stderr = self.m.run_command(cmd, check_rc=False)
+
+        self.changed = True
+
         if rc == 0:
-            if packages:
-                module.exit_json(changed=True, msg='System upgraded', packages=packages, diff=diff, stdout=stdout, stderr=stderr)
-            else:
-                module.exit_json(changed=False, msg='Nothing to upgrade', packages=packages)
+            self.add_exit_infos("Updated package db", stdout=stdout, stderr=stderr)
         else:
-            module.fail_json(msg="Could not upgrade", stdout=stdout, stderr=stderr)
-    else:
-        module.exit_json(changed=False, msg='Nothing to upgrade', packages=packages)
+            self.fail("could not update package db", stdout=stdout, stderr=stderr)
 
+    def package_list(self):
+        """Takes the input package list and resolves packages groups to their package list using the inventory,
+        extracts package names from packages given as files or URLs using calls to pacman
 
-def remove_packages(module, pacman_path, packages):
-    data = []
-    diff = {
-        'before': '',
-        'after': '',
-    }
+        Returns the expanded/resolved list as a list of Package
+        """
+        pkg_list = []
+        for pkg in self.m.params["name"]:
+            if not pkg:
+                continue
 
-    if module.params["force"]:
-        module.params["extra_args"] += " --nodeps --nodeps"
-
-    remove_c = 0
-    stdout_total = ""
-    stderr_total = ""
-    # Using a for loop in case of error, we can report the package that failed
-    for package in packages:
-        # Query the package first, to see if we even need to remove
-        installed, updated, unknown = query_package(module, pacman_path, package, 'absent')
-        if not installed:
-            continue
-
-        cmd = "%s --remove --noconfirm --noprogressbar %s %s" % (pacman_path, module.params["extra_args"], package)
-        rc, stdout, stderr = module.run_command(cmd, check_rc=False)
-
-        if rc != 0:
-            module.fail_json(msg="failed to remove %s" % (package), stdout=stdout, stderr=stderr)
-
-        stdout_total += stdout
-        stderr_total += stderr
-        if module._diff:
-            d = stdout.split('\n')[2].split(' ')[2:]
-            for i, pkg in enumerate(d):
-                d[i] = re.sub('-[0-9].*$', '', d[i].split('/')[-1])
-                diff['before'] += "%s\n" % pkg
-            data.append('\n'.join(d))
-
-        remove_c += 1
-
-    if remove_c > 0:
-        module.exit_json(changed=True, msg="removed %s package(s)" % remove_c, diff=diff, stdout=stdout_total, stderr=stderr_total)
-
-    module.exit_json(changed=False, msg="package(s) already absent")
-
-
-def install_packages(module, pacman_path, state, packages, package_files):
-    install_c = 0
-    package_err = []
-    message = ""
-    data = []
-    diff = {
-        'before': '',
-        'after': '',
-    }
-
-    to_install_repos = []
-    to_install_files = []
-    for i, package in enumerate(packages):
-        # if the package is installed and state == present or state == latest and is up-to-date then skip
-        installed, updated, latestError = query_package(module, pacman_path, package, state)
-        if latestError and state == 'latest':
-            package_err.append(package)
-
-        if installed and (state == 'present' or (state == 'latest' and updated)):
-            continue
-
-        if package_files[i]:
-            to_install_files.append(package_files[i])
-        else:
-            to_install_repos.append(package)
-
-    if to_install_repos:
-        cmd = "%s --sync --noconfirm --noprogressbar --needed %s %s" % (pacman_path, module.params["extra_args"], " ".join(to_install_repos))
-        rc, stdout, stderr = module.run_command(cmd, check_rc=False)
-
-        if rc != 0:
-            module.fail_json(msg="failed to install %s: %s" % (" ".join(to_install_repos), stderr), stdout=stdout, stderr=stderr)
-
-        # As we pass `--needed` to pacman returns a single line of ` there is nothing to do` if no change is performed.
-        # The check for > 3 is here because we pick the 4th line in normal operation.
-        if len(stdout.split('\n')) > 3:
-            data = stdout.split('\n')[3].split(' ')[2:]
-            data = [i for i in data if i != '']
-            for i, pkg in enumerate(data):
-                data[i] = re.sub('-[0-9].*$', '', data[i].split('/')[-1])
-                if module._diff:
-                    diff['after'] += "%s\n" % pkg
-
-            install_c += len(to_install_repos)
-
-    if to_install_files:
-        cmd = "%s --upgrade --noconfirm --noprogressbar --needed %s %s" % (pacman_path, module.params["extra_args"], " ".join(to_install_files))
-        rc, stdout, stderr = module.run_command(cmd, check_rc=False)
-
-        if rc != 0:
-            module.fail_json(msg="failed to install %s: %s" % (" ".join(to_install_files), stderr), stdout=stdout, stderr=stderr)
-
-        # As we pass `--needed` to pacman returns a single line of ` there is nothing to do` if no change is performed.
-        # The check for > 3 is here because we pick the 4th line in normal operation.
-        if len(stdout.split('\n')) > 3:
-            data = stdout.split('\n')[3].split(' ')[2:]
-            data = [i for i in data if i != '']
-            for i, pkg in enumerate(data):
-                data[i] = re.sub('-[0-9].*$', '', data[i].split('/')[-1])
-                if module._diff:
-                    diff['after'] += "%s\n" % pkg
-
-            install_c += len(to_install_files)
-
-    if state == 'latest' and len(package_err) > 0:
-        message = "But could not ensure 'latest' state for %s package(s) as remote version could not be fetched." % (package_err)
-
-    if install_c > 0:
-        module.exit_json(changed=True, msg="installed %s package(s). %s" % (install_c, message), diff=diff, stdout=stdout, stderr=stderr)
-
-    module.exit_json(changed=False, msg="package(s) already installed. %s" % (message), diff=diff)
-
-
-def check_packages(module, pacman_path, packages, state):
-    would_be_changed = []
-    diff = {
-        'before': '',
-        'after': '',
-        'before_header': '',
-        'after_header': ''
-    }
-
-    for package in packages:
-        installed, updated, unknown = query_package(module, pacman_path, package, state)
-        if ((state in ["present", "latest"] and not installed) or
-                (state == "absent" and installed) or
-                (state == "latest" and not updated)):
-            would_be_changed.append(package)
-    if would_be_changed:
-        if state == "absent":
-            state = "removed"
-
-        if module._diff and (state == 'removed'):
-            diff['before_header'] = 'removed'
-            diff['before'] = '\n'.join(would_be_changed) + '\n'
-        elif module._diff and ((state == 'present') or (state == 'latest')):
-            diff['after_header'] = 'installed'
-            diff['after'] = '\n'.join(would_be_changed) + '\n'
-
-        module.exit_json(changed=True, msg="%s package(s) would be %s" % (
-            len(would_be_changed), state), diff=diff)
-    else:
-        module.exit_json(changed=False, msg="package(s) already %s" % state, diff=diff)
-
-
-def expand_package_groups(module, pacman_path, pkgs):
-    expanded = []
-
-    __, stdout, __ = module.run_command([pacman_path, "--sync", "--groups", "--quiet"], check_rc=True)
-    available_groups = stdout.splitlines()
-
-    for pkg in pkgs:
-        if pkg:  # avoid empty strings
-            if pkg in available_groups:
-                # A group was found matching the package name: expand it
-                cmd = [pacman_path, "--sync", "--groups", "--quiet", pkg]
-                rc, stdout, stderr = module.run_command(cmd, check_rc=True)
-                expanded.extend([name.strip() for name in stdout.splitlines()])
+            if pkg in self.inventory["available_groups"]:
+                # Expand group members
+                for group_member in self.inventory["available_groups"][pkg]:
+                    pkg_list.append(Package(name=group_member, source=group_member))
+            elif pkg in self.inventory["available_pkgs"]:
+                # just a regular pkg
+                pkg_list.append(Package(name=pkg, source=pkg))
             else:
-                expanded.append(pkg)
+                # Last resort, call out to pacman to extract the info,
+                # pkg is possibly in the <repo>/<pkgname> format, or a filename or a URL
 
-    return expanded
+                # Start with <repo>/<pkgname> case
+                cmd = [self.pacman_path, "--sync", "--print-format", "%n", pkg]
+                rc, stdout, stderr = self.m.run_command(cmd, check_rc=False)
+                if rc != 0:
+                    # fallback to filename / URL
+                    cmd = [self.pacman_path, "--upgrade", "--print-format", "%n", pkg]
+                    rc, stdout, stderr = self.m.run_command(cmd, check_rc=False)
+                    if rc != 0:
+                        if self.target_state == "absent":
+                            continue  # Don't bark for unavailable packages when trying to remove them
+                        else:
+                            self.fail(
+                                msg="Failed to list package %s" % (pkg),
+                                cmd=cmd,
+                                stdout=stdout,
+                                stderr=stderr,
+                                rc=rc,
+                            )
+                pkg_name = stdout.strip()
+                pkg_list.append(Package(name=pkg_name, source=pkg))
+
+        return pkg_list
+
+    def _build_inventory(self):
+        """Build a cache datastructure used for all pkg lookups
+        Returns a dict:
+        {
+            "installed_pkgs": {pkgname: version},
+            "installed_groups": {groupname: set(pkgnames)},
+            "available_pkgs": {pkgname: version},
+            "available_groups": {groupname: set(pkgnames)},
+            "upgradable_pkgs": {pkgname: (current_version,latest_version)},
+        }
+
+        Fails the module if a package requested for install cannot be found
+        """
+
+        installed_pkgs = {}
+        dummy, stdout, dummy = self.m.run_command([self.pacman_path, "--query"], check_rc=True)
+        # Format of a line: "pacman 6.0.1-2"
+        for l in stdout.splitlines():
+            l = l.strip()
+            if not l:
+                continue
+            pkg, ver = l.split()
+            installed_pkgs[pkg] = ver
+
+        installed_groups = defaultdict(set)
+        dummy, stdout, dummy = self.m.run_command(
+            [self.pacman_path, "--query", "--group"], check_rc=True
+        )
+        # Format of lines:
+        #     base-devel file
+        #     base-devel findutils
+        #     ...
+        for l in stdout.splitlines():
+            l = l.strip()
+            if not l:
+                continue
+            group, pkgname = l.split()
+            installed_groups[group].add(pkgname)
+
+        available_pkgs = {}
+        dummy, stdout, dummy = self.m.run_command([self.pacman_path, "--sync", "--list"], check_rc=True)
+        # Format of a line: "core pacman 6.0.1-2"
+        for l in stdout.splitlines():
+            l = l.strip()
+            if not l:
+                continue
+            repo, pkg, ver = l.split()[:3]
+            available_pkgs[pkg] = ver
+
+        available_groups = defaultdict(set)
+        dummy, stdout, dummy = self.m.run_command(
+            [self.pacman_path, "--sync", "--group", "--group"], check_rc=True
+        )
+        # Format of lines:
+        #     vim-plugins vim-airline
+        #     vim-plugins vim-airline-themes
+        #     vim-plugins vim-ale
+        #     ...
+        for l in stdout.splitlines():
+            l = l.strip()
+            if not l:
+                continue
+            group, pkg = l.split()
+            available_groups[group].add(pkg)
+
+        upgradable_pkgs = {}
+        rc, stdout, stderr = self.m.run_command(
+            [self.pacman_path, "--query", "--upgrades"], check_rc=False
+        )
+
+        # non-zero exit with nothing in stdout -> nothing to upgrade, all good
+        # stderr can have warnings, so not checked here
+        if rc == 1 and stdout == "":
+            pass  # nothing to upgrade
+        elif rc == 0:
+            # Format of lines:
+            #     strace 5.14-1 -> 5.15-1
+            #     systemd 249.7-1 -> 249.7-2 [ignored]
+            for l in stdout.splitlines():
+                l = l.strip()
+                if not l:
+                    continue
+                if "[ignored]" in l:
+                    continue
+                s = l.split()
+                if len(s) != 4:
+                    self.fail(msg="Invalid line: %s" % l)
+
+                pkg = s[0]
+                current = s[1]
+                latest = s[3]
+                upgradable_pkgs[pkg] = VersionTuple(current=current, latest=latest)
+        else:
+            # stuff in stdout but rc!=0, abort
+            self.fail(
+                "Couldn't get list of packages available for upgrade",
+                stdout=stdout,
+                stderr=stderr,
+                rc=rc,
+            )
+
+        return dict(
+            installed_pkgs=installed_pkgs,
+            installed_groups=installed_groups,
+            available_pkgs=available_pkgs,
+            available_groups=available_groups,
+            upgradable_pkgs=upgradable_pkgs,
+        )
 
 
-def main():
+def setup_module():
     module = AnsibleModule(
         argument_spec=dict(
-            name=dict(type='list', elements='str', aliases=['pkg', 'package']),
-            state=dict(type='str', default='present', choices=['present', 'installed', 'latest', 'absent', 'removed']),
-            force=dict(type='bool', default=False),
-            executable=dict(type='str', default='pacman'),
-            extra_args=dict(type='str', default=''),
-            upgrade=dict(type='bool', default=False),
-            upgrade_extra_args=dict(type='str', default=''),
+            name=dict(type="list", elements="str", aliases=["pkg", "package"]),
+            state=dict(
+                type="str",
+                default="present",
+                choices=["present", "installed", "latest", "absent", "removed"],
+            ),
+            force=dict(type="bool", default=False),
+            executable=dict(type="str", default="pacman"),
+            extra_args=dict(type="str", default=""),
+            upgrade=dict(type="bool"),
+            upgrade_extra_args=dict(type="str", default=""),
             update_cache=dict(
-                type='bool', default=False, aliases=['update-cache'],
-                deprecated_aliases=[dict(name='update-cache', version='5.0.0', collection_name='community.general')]),
-            update_cache_extra_args=dict(type='str', default=''),
+                type="bool",
+                aliases=["update-cache"],
+                deprecated_aliases=[
+                    dict(
+                        name="update-cache",
+                        version="5.0.0",
+                        collection_name="community.general",
+                    )
+                ],
+            ),
+            update_cache_extra_args=dict(type="str", default=""),
         ),
-        required_one_of=[['name', 'update_cache', 'upgrade']],
-        mutually_exclusive=[['name', 'upgrade']],
+        required_one_of=[["name", "update_cache", "upgrade"]],
+        mutually_exclusive=[["name", "upgrade"]],
         supports_check_mode=True,
     )
 
-    module.run_command_environ_update = dict(LC_ALL='C')
+    # Split extra_args as the shell would for easier handling later
+    for str_args in ["extra_args", "upgrade_extra_args", "update_cache_extra_args"]:
+        module.params[str_args] = shlex.split(module.params[str_args])
 
-    p = module.params
+    return module
 
-    # find pacman binary
-    pacman_path = module.get_bin_path(p['executable'], True)
 
-    # normalize the state parameter
-    if p['state'] in ['present', 'installed']:
-        p['state'] = 'present'
-    elif p['state'] in ['absent', 'removed']:
-        p['state'] = 'absent'
+def main():
 
-    if p["update_cache"] and not module.check_mode:
-        stdout, stderr = update_package_db(module, pacman_path)
-        if not (p['name'] or p['upgrade']):
-            module.exit_json(changed=True, msg='Updated the package master lists', stdout=stdout, stderr=stderr)
-
-    if p['update_cache'] and module.check_mode and not (p['name'] or p['upgrade']):
-        module.exit_json(changed=True, msg='Would have updated the package cache')
-
-    if p['upgrade']:
-        upgrade(module, pacman_path)
-
-    if p['name']:
-        pkgs = expand_package_groups(module, pacman_path, p['name'])
-
-        pkg_files = []
-        for i, pkg in enumerate(pkgs):
-            if not pkg:  # avoid empty strings
-                continue
-            elif re.match(r".*\.pkg\.tar(\.(gz|bz2|xz|lrz|lzo|Z|zst))?$", pkg):
-                # The package given is a filename, extract the raw pkg name from
-                # it and store the filename
-                pkg_files.append(pkg)
-                pkgs[i] = re.sub(r'-[0-9].*$', '', pkgs[i].split('/')[-1])
-            else:
-                pkg_files.append(None)
-
-        if module.check_mode:
-            check_packages(module, pacman_path, pkgs, p['state'])
-
-        if p['state'] in ['present', 'latest']:
-            install_packages(module, pacman_path, p['state'], pkgs, pkg_files)
-        elif p['state'] == 'absent':
-            remove_packages(module, pacman_path, pkgs)
-    else:
-        module.exit_json(changed=False, msg="No package specified to work on.")
+    Pacman(setup_module()).run()
 
 
 if __name__ == "__main__":

plugins/modules/packaging/os/yum_versionlock.py

@@ -17,7 +17,7 @@ description:
 options:
   name:
     description:
-      - Package name or a list of packages.
+      - Package name or a list of package names with optional wildcards.
     type: list
     required: true
     elements: str
@@ -74,10 +74,17 @@ state:
     sample: present
 '''
 
+import re
 from ansible.module_utils.basic import AnsibleModule
 from ansible.module_utils.common.text.converters import to_native
 from fnmatch import fnmatch
 
+# on DNF-based distros, yum is a symlink to dnf, so we try to handle their different entry formats.
+NEVRA_RE_YUM = re.compile(r'^(?P<exclude>!)?(?P<epoch>\d+):(?P<name>.+)-'
+                          r'(?P<version>.+)-(?P<release>.+)\.(?P<arch>.+)$')
+NEVRA_RE_DNF = re.compile(r"^(?P<exclude>!)?(?P<name>.+)-(?P<epoch>\d+):(?P<version>.+)-"
+                          r"(?P<release>.+)\.(?P<arch>.+)$")
+
 
 class YumVersionLock:
     def __init__(self, module):
@@ -102,6 +109,15 @@ class YumVersionLock:
         self.module.fail_json(msg="Error: " + to_native(err) + to_native(out))
 
 
+def match(entry, name):
+    m = NEVRA_RE_YUM.match(entry)
+    if not m:
+        m = NEVRA_RE_DNF.match(entry)
+    if not m:
+        return False
+    return fnmatch(m.group("name"), name)
+
+
 def main():
     """ start main program to add/remove a package to yum versionlock"""
     module = AnsibleModule(
@@ -123,20 +139,20 @@ def main():
 
     # Ensure versionlock state of packages
     packages_list = []
-    if state in ('present'):
+    if state in ('present', ):
         command = 'add'
         for single_pkg in packages:
-            if not any(fnmatch(pkg.split(":", 1)[-1], single_pkg) for pkg in versionlock_packages.split()):
+            if not any(match(pkg, single_pkg) for pkg in versionlock_packages.split()):
                 packages_list.append(single_pkg)
         if packages_list:
             if module.check_mode:
                 changed = True
             else:
                 changed = yum_v.ensure_state(packages_list, command)
-    elif state in ('absent'):
+    elif state in ('absent', ):
         command = 'delete'
         for single_pkg in packages:
-            if any(fnmatch(pkg, single_pkg) for pkg in versionlock_packages.split()):
+            if any(match(pkg, single_pkg) for pkg in versionlock_packages.split()):
                 packages_list.append(single_pkg)
         if packages_list:
             if module.check_mode:

plugins/modules/pmem.py

@@ -0,0 +1 @@
+storage/pmem/pmem.py

plugins/modules/remote_management/imc/imc_rest.py

@@ -261,7 +261,6 @@ output:
 '''
 
 import datetime
-import itertools
 import os
 import traceback
 from functools import partial
@@ -283,6 +282,7 @@ except ImportError:
     HAS_XMLJSON_COBRA = False
 
 from ansible.module_utils.basic import AnsibleModule, missing_required_lib
+from ansible.module_utils.six.moves import zip_longest
 from ansible.module_utils.urls import fetch_url
 
 
@@ -318,7 +318,7 @@ def merge(one, two):
         return copy
 
     elif isinstance(one, list) and isinstance(two, list):
-        return [merge(alpha, beta) for (alpha, beta) in itertools.izip_longest(one, two)]
+        return [merge(alpha, beta) for (alpha, beta) in zip_longest(one, two)]
 
     return one if two is None else two
 

plugins/modules/remote_management/redfish/ilo_redfish_config.py

@@ -98,7 +98,7 @@ CATEGORY_COMMANDS_ALL = {
 
 from ansible_collections.community.general.plugins.module_utils.ilo_redfish_utils import iLORedfishUtils
 from ansible.module_utils.basic import AnsibleModule
-from ansible.module_utils._text import to_native
+from ansible.module_utils.common.text.converters import to_native
 
 
 def main():

plugins/modules/remote_management/redfish/ilo_redfish_info.py

@@ -105,7 +105,7 @@ CATEGORY_COMMANDS_DEFAULT = {
 }
 
 from ansible.module_utils.basic import AnsibleModule
-from ansible.module_utils._text import to_native
+from ansible.module_utils.common.text.converters import to_native
 from ansible_collections.community.general.plugins.module_utils.ilo_redfish_utils import iLORedfishUtils
 
 

plugins/modules/scaleway_private_network.py

@@ -0,0 +1 @@
+cloud/scaleway/scaleway_private_network.py

plugins/modules/source_control/gitlab/gitlab_group_variable.py

@@ -48,6 +48,8 @@ options:
       - When the list element is a simple key-value pair, set masked and protected to false.
       - When the list element is a dict with the keys I(value), I(masked) and I(protected), the user can
         have full control about whether a value should be masked, protected or both.
+      - Support for group variables requires GitLab >= 9.5.
+      - Support for environment_scope requires GitLab Premium >= 13.11.
       - Support for protected values requires GitLab >= 9.3.
       - Support for masked values requires GitLab >= 11.10.
       - A I(value) must be a string or a number.
@@ -56,6 +58,46 @@ options:
         See GitLab documentation on acceptable values for a masked variable (U(https://docs.gitlab.com/ce/ci/variables/#masked-variables)).
     default: {}
     type: dict
+  variables:
+    version_added: 4.5.0
+    description:
+      - A list of dictionaries that represents CI/CD variables.
+      - This modules works internal with this sructure, even if the older I(vars) parameter is used.
+    default: []
+    type: list
+    elements: dict
+    suboptions:
+      name:
+        description:
+          - The name of the variable.
+        type: str
+        required: true
+      value:
+        description:
+          - The variable value.
+          - Required when I(state=present).
+        type: str
+      masked:
+        description:
+          - Wether variable value is masked or not.
+        type: bool
+        default: false
+      protected:
+        description:
+          - Wether variable value is protected or not.
+        type: bool
+        default: false
+      variable_type:
+        description:
+          - Wether a variable is an environment variable (C(env_var)) or a file (C(file)).
+        type: str
+        choices: [ "env_var", "file" ]
+        default: env_var
+      environment_scope:
+        description:
+          - The scope for the variable.
+        type: str
+        default: '*'
 notes:
 - Supports I(check_mode).
 '''
@@ -68,23 +110,15 @@ EXAMPLES = r'''
     api_token: secret_access_token
     group: scodeman/testgroup/
     purge: false
-    vars:
-      ACCESS_KEY_ID: abc123
-      SECRET_ACCESS_KEY: 321cba
-
-- name: Set or update some CI/CD variables
-  community.general.gitlab_group_variable:
-    api_url: https://gitlab.com
-    api_token: secret_access_token
-    group: scodeman/testgroup/
-    purge: false
-    vars:
-      ACCESS_KEY_ID: abc123
-      SECRET_ACCESS_KEY:
+    variables:
+      - name: ACCESS_KEY_ID
+        value: abc123
+      - name: SECRET_ACCESS_KEY
         value: 3214cbad
         masked: true
         protected: true
         variable_type: env_var
+        environment_scope: production
 
 - name: Delete one variable
   community.general.gitlab_group_variable:
@@ -125,13 +159,11 @@ group_variable:
 '''
 
 import traceback
-
 from ansible.module_utils.basic import AnsibleModule, missing_required_lib
 from ansible.module_utils.api import basic_auth_argument_spec
 from ansible.module_utils.six import string_types
 from ansible.module_utils.six import integer_types
 
-
 GITLAB_IMP_ERR = None
 try:
     import gitlab
@@ -143,6 +175,44 @@ except Exception:
 from ansible_collections.community.general.plugins.module_utils.gitlab import auth_argument_spec, gitlab_authentication
 
 
+def vars_to_variables(vars, module):
+    # transform old vars to new variables structure
+    variables = list()
+    for item, value in vars.items():
+        if (isinstance(value, string_types) or
+           isinstance(value, (integer_types, float))):
+            variables.append(
+                {
+                    "name": item,
+                    "value": str(value),
+                    "masked": False,
+                    "protected": False,
+                    "variable_type": "env_var",
+                }
+            )
+
+        elif isinstance(value, dict):
+            new_item = {"name": item, "value": value.get('value')}
+
+            new_item = {
+                "name": item,
+                "value": value.get('value'),
+                "masked": value.get('masked'),
+                "protected": value.get('protected'),
+                "variable_type": value.get('variable_type'),
+            }
+
+            if value.get('environment_scope'):
+                new_item['environment_scope'] = value.get('environment_scope')
+
+            variables.append(new_item)
+
+        else:
+            module.fail_json(msg="value must be of type string, integer, float or dict")
+
+    return variables
+
+
 class GitlabGroupVariables(object):
 
     def __init__(self, module, gitlab_instance):
@@ -163,103 +233,150 @@ class GitlabGroupVariables(object):
             vars_page = self.group.variables.list(page=page_nb)
         return variables
 
-    def create_variable(self, key, value, masked, protected, variable_type):
-        if self._module.check_mode:
-            return
-        return self.group.variables.create({
-            "key": key,
-            "value": value,
-            "masked": masked,
-            "protected": protected,
-            "variable_type": variable_type,
-        })
-
-    def update_variable(self, key, var, value, masked, protected, variable_type):
-        if var.value == value and var.protected == protected and var.masked == masked and var.variable_type == variable_type:
-            return False
-
+    def create_variable(self, var_obj):
         if self._module.check_mode:
             return True
+        var = {
+            "key": var_obj.get('key'),
+            "value": var_obj.get('value'),
+            "masked": var_obj.get('masked'),
+            "protected": var_obj.get('protected'),
+            "variable_type": var_obj.get('variable_type'),
+        }
+        if var_obj.get('environment_scope') is not None:
+            var["environment_scope"] = var_obj.get('environment_scope')
 
-        if var.protected == protected and var.masked == masked and var.variable_type == variable_type:
-            var.value = value
-            var.save()
-            return True
-
-        self.delete_variable(key)
-        self.create_variable(key, value, masked, protected, variable_type)
+        self.group.variables.create(var)
         return True
 
-    def delete_variable(self, key):
+    def update_variable(self, var_obj):
         if self._module.check_mode:
-            return
-        return self.group.variables.delete(key)
+            return True
+        self.delete_variable(var_obj)
+        self.create_variable(var_obj)
+        return True
+
+    def delete_variable(self, var_obj):
+        if self._module.check_mode:
+            return True
+        self.group.variables.delete(var_obj.get('key'), filter={'environment_scope': var_obj.get('environment_scope')})
+        return True
 
 
-def native_python_main(this_gitlab, purge, var_list, state, module):
+def compare(requested_variables, existing_variables, state):
+    # we need to do this, because it was determined in a previous version - more or less buggy
+    # basically it is not necessary and might results in more/other bugs!
+    # but it is required  and only relevant for check mode!!
+    # logic represents state 'present' when not purge. all other can be derived from that
+    # untouched => equal in both
+    # updated => name and scope are equal
+    # added => name and scope does not exist
+    untouched = list()
+    updated = list()
+    added = list()
+
+    if state == 'present':
+        existing_key_scope_vars = list()
+        for item in existing_variables:
+            existing_key_scope_vars.append({'key': item.get('key'), 'environment_scope': item.get('environment_scope')})
+
+        for var in requested_variables:
+            if var in existing_variables:
+                untouched.append(var)
+            else:
+                compare_item = {'key': var.get('name'), 'environment_scope': var.get('environment_scope')}
+                if compare_item in existing_key_scope_vars:
+                    updated.append(var)
+                else:
+                    added.append(var)
+
+    return untouched, updated, added
+
+
+def native_python_main(this_gitlab, purge, requested_variables, state, module):
 
     change = False
     return_value = dict(added=list(), updated=list(), removed=list(), untouched=list())
 
     gitlab_keys = this_gitlab.list_all_group_variables()
-    existing_variables = [x.get_id() for x in gitlab_keys]
+    before = [x.attributes for x in gitlab_keys]
 
-    for key in var_list:
-        if not isinstance(var_list[key], (string_types, integer_types, float, dict)):
-            module.fail_json(msg="Value of %s variable must be of type string, integer, float or dict, passed %s" % (key, var_list[key].__class__.__name__))
+    gitlab_keys = this_gitlab.list_all_group_variables()
+    existing_variables = [x.attributes for x in gitlab_keys]
 
-    for key in var_list:
+    # preprocessing:filter out and enrich before compare
+    for item in existing_variables:
+        item.pop('group_id')
 
-        if isinstance(var_list[key], (string_types, integer_types, float)):
-            value = var_list[key]
-            masked = False
-            protected = False
-            variable_type = 'env_var'
-        elif isinstance(var_list[key], dict):
-            value = var_list[key].get('value')
-            masked = var_list[key].get('masked', False)
-            protected = var_list[key].get('protected', False)
-            variable_type = var_list[key].get('variable_type', 'env_var')
+    for item in requested_variables:
+        item['key'] = item.pop('name')
+        item['value'] = str(item.get('value'))
+        if item.get('protected') is None:
+            item['protected'] = False
+        if item.get('masked') is None:
+            item['masked'] = False
+        if item.get('environment_scope') is None:
+            item['environment_scope'] = '*'
+        if item.get('variable_type') is None:
+            item['variable_type'] = 'env_var'
 
-        if key in existing_variables:
-            index = existing_variables.index(key)
-            existing_variables[index] = None
+    if module.check_mode:
+        untouched, updated, added = compare(requested_variables, existing_variables, state)
 
-            if state == 'present':
-                single_change = this_gitlab.update_variable(
-                    key,
-                    gitlab_keys[index],
-                    value,
-                    masked,
-                    protected,
-                    variable_type,
-                )
-                change = single_change or change
-                if single_change:
-                    return_value['updated'].append(key)
-                else:
-                    return_value['untouched'].append(key)
+    if state == 'present':
+        add_or_update = [x for x in requested_variables if x not in existing_variables]
+        for item in add_or_update:
+            try:
+                if this_gitlab.create_variable(item):
+                    return_value['added'].append(item)
 
-            elif state == 'absent':
-                this_gitlab.delete_variable(key)
-                change = True
-                return_value['removed'].append(key)
+            except Exception:
+                if this_gitlab.update_variable(item):
+                    return_value['updated'].append(item)
 
-        elif key not in existing_variables and state == 'present':
-            this_gitlab.create_variable(key, value, masked, protected, variable_type)
-            change = True
-            return_value['added'].append(key)
+        if purge:
+            # refetch and filter
+            gitlab_keys = this_gitlab.list_all_group_variables()
+            existing_variables = [x.attributes for x in gitlab_keys]
+            for item in existing_variables:
+                item.pop('group_id')
 
-    existing_variables = list(filter(None, existing_variables))
-    if purge:
+            remove = [x for x in existing_variables if x not in requested_variables]
+            for item in remove:
+                if this_gitlab.delete_variable(item):
+                    return_value['removed'].append(item)
+
+    elif state == 'absent':
+        # value does not matter on removing variables.
+        # key and environment scope are sufficient
         for item in existing_variables:
-            this_gitlab.delete_variable(item)
-            change = True
-            return_value['removed'].append(item)
-    else:
-        return_value['untouched'].extend(existing_variables)
+            item.pop('value')
+            item.pop('variable_type')
+        for item in requested_variables:
+            item.pop('value')
+            item.pop('variable_type')
 
-    return change, return_value
+        if not purge:
+            remove_requested = [x for x in requested_variables if x in existing_variables]
+            for item in remove_requested:
+                if this_gitlab.delete_variable(item):
+                    return_value['removed'].append(item)
+
+        else:
+            for item in existing_variables:
+                if this_gitlab.delete_variable(item):
+                    return_value['removed'].append(item)
+
+    if module.check_mode:
+        return_value = dict(added=added, updated=updated, removed=return_value['removed'], untouched=untouched)
+
+    if len(return_value['added'] + return_value['removed'] + return_value['updated']) > 0:
+        change = True
+
+    gitlab_keys = this_gitlab.list_all_group_variables()
+    after = [x.attributes for x in gitlab_keys]
+
+    return change, return_value, before, after
 
 
 def main():
@@ -269,7 +386,15 @@ def main():
         group=dict(type='str', required=True),
         purge=dict(type='bool', required=False, default=False),
         vars=dict(type='dict', required=False, default=dict(), no_log=True),
-        state=dict(type='str', default="present", choices=["absent", "present"])
+        variables=dict(type='list', elements='dict', required=False, default=list(), options=dict(
+            name=dict(type='str', required=True),
+            value=dict(type='str', no_log=True),
+            masked=dict(type='bool', default=False),
+            protected=dict(type='bool', default=False),
+            environment_scope=dict(type='str', default='*'),
+            variable_type=dict(type='str', default='env_var', choices=["env_var", "file"])
+        )),
+        state=dict(type='str', default="present", choices=["absent", "present"]),
     )
 
     module = AnsibleModule(
@@ -280,6 +405,7 @@ def main():
             ['api_username', 'api_job_token'],
             ['api_token', 'api_oauth_token'],
             ['api_token', 'api_job_token'],
+            ['vars', 'variables'],
         ],
         required_together=[
             ['api_username', 'api_password'],
@@ -290,18 +416,46 @@ def main():
         supports_check_mode=True
     )
 
+    if not HAS_GITLAB_PACKAGE:
+        module.fail_json(msg=missing_required_lib("python-gitlab"), exception=GITLAB_IMP_ERR)
+
     purge = module.params['purge']
     var_list = module.params['vars']
     state = module.params['state']
 
-    if not HAS_GITLAB_PACKAGE:
-        module.fail_json(msg=missing_required_lib("python-gitlab"), exception=GITLAB_IMP_ERR)
+    if var_list:
+        variables = vars_to_variables(var_list, module)
+    else:
+        variables = module.params['variables']
+
+    if state == 'present':
+        if any(x['value'] is None for x in variables):
+            module.fail_json(msg='value parameter is required in state present')
 
     gitlab_instance = gitlab_authentication(module)
 
     this_gitlab = GitlabGroupVariables(module=module, gitlab_instance=gitlab_instance)
 
-    changed, return_value = native_python_main(this_gitlab, purge, var_list, state, module)
+    changed, raw_return_value, before, after = native_python_main(this_gitlab, purge, variables, state, module)
+
+    # postprocessing
+    for item in after:
+        item.pop('group_id')
+        item['name'] = item.pop('key')
+    for item in before:
+        item.pop('group_id')
+        item['name'] = item.pop('key')
+
+    untouched_key_name = 'key'
+    if not module.check_mode:
+        untouched_key_name = 'name'
+        raw_return_value['untouched'] = [x for x in before if x in after]
+
+    added = [x.get('key') for x in raw_return_value['added']]
+    updated = [x.get('key') for x in raw_return_value['updated']]
+    removed = [x.get('key') for x in raw_return_value['removed']]
+    untouched = [x.get(untouched_key_name) for x in raw_return_value['untouched']]
+    return_value = dict(added=added, updated=updated, removed=removed, untouched=untouched)
 
     module.exit_json(changed=changed, group_variable=return_value)
 

plugins/modules/source_control/gitlab/gitlab_project_variable.py

@@ -74,8 +74,8 @@ options:
       value:
         description:
           - The variable value.
+          - Required when I(state=present).
         type: str
-        required: true
       masked:
         description:
           - Wether variable value is masked or not.
@@ -403,7 +403,7 @@ def main():
         vars=dict(type='dict', required=False, default=dict(), no_log=True),
         variables=dict(type='list', elements='dict', required=False, default=list(), options=dict(
             name=dict(type='str', required=True),
-            value=dict(type='str', required=True, no_log=True),
+            value=dict(type='str', no_log=True),
             masked=dict(type='bool', default=False),
             protected=dict(type='bool', default=False),
             environment_scope=dict(type='str', default='*'),
@@ -431,18 +431,21 @@ def main():
         supports_check_mode=True
     )
 
+    if not HAS_GITLAB_PACKAGE:
+        module.fail_json(msg=missing_required_lib("python-gitlab"), exception=GITLAB_IMP_ERR)
+
     purge = module.params['purge']
     var_list = module.params['vars']
+    state = module.params['state']
 
     if var_list:
         variables = vars_to_variables(var_list, module)
     else:
         variables = module.params['variables']
 
-    state = module.params['state']
-
-    if not HAS_GITLAB_PACKAGE:
-        module.fail_json(msg=missing_required_lib("python-gitlab"), exception=GITLAB_IMP_ERR)
+    if state == 'present':
+        if any(x['value'] is None for x in variables):
+            module.fail_json(msg='value parameter is required in state present')
 
     gitlab_instance = gitlab_authentication(module)
 

plugins/modules/source_control/gitlab/gitlab_runner.py

@@ -39,6 +39,7 @@ options:
   project:
     description:
       - ID or full path of the project in the form of group/name.
+      - Mutually exclusive with I(owned) since community.general 4.5.0.
     type: str
     version_added: '3.7.0'
   description:
@@ -63,6 +64,7 @@ options:
   owned:
     description:
       - Searches only runners available to the user when searching for existing, when false admin token required.
+      - Mutually exclusive with I(project) since community.general 4.5.0.
     default: no
     type: bool
     version_added: 2.0.0
@@ -199,7 +201,13 @@ class GitLabRunner(object):
         # Whether to operate on GitLab-instance-wide or project-wide runners
         # See https://gitlab.com/gitlab-org/gitlab-ce/issues/60774
         # for group runner token access
-        self._runners_endpoint = project.runners if project else gitlab_instance.runners
+        if project:
+            self._runners_endpoint = project.runners.list
+        elif module.params['owned']:
+            self._runners_endpoint = gitlab_instance.runners.list
+        else:
+            self._runners_endpoint = gitlab_instance.runners.all
+
         self.runner_object = None
 
     def create_or_update_runner(self, description, options):
@@ -281,11 +289,8 @@ class GitLabRunner(object):
     '''
     @param description Description of the runner
     '''
-    def find_runner(self, description, owned=False):
-        if owned:
-            runners = self._runners_endpoint.list(as_list=False)
-        else:
-            runners = self._runners_endpoint.all(as_list=False)
+    def find_runner(self, description):
+        runners = self._runners_endpoint(as_list=False)
 
         for runner in runners:
             # python-gitlab 2.2 through at least 2.5 returns a list of dicts for list() instead of a Runner
@@ -300,9 +305,9 @@ class GitLabRunner(object):
     '''
     @param description Description of the runner
     '''
-    def exists_runner(self, description, owned=False):
+    def exists_runner(self, description):
         # When runner exists, object will be stored in self.runner_object.
-        runner = self.find_runner(description, owned)
+        runner = self.find_runner(description)
 
         if runner:
             self.runner_object = runner
@@ -343,6 +348,7 @@ def main():
             ['api_username', 'api_job_token'],
             ['api_token', 'api_oauth_token'],
             ['api_token', 'api_job_token'],
+            ['project', 'owned'],
         ],
         required_together=[
             ['api_username', 'api_password'],
@@ -357,7 +363,6 @@ def main():
     )
 
     state = module.params['state']
-    owned = module.params['owned']
     runner_description = module.params['description']
     runner_active = module.params['active']
     tag_list = module.params['tag_list']
@@ -380,7 +385,7 @@ def main():
             module.fail_json(msg='No such a project %s' % project, exception=to_native(e))
 
     gitlab_runner = GitLabRunner(module, gitlab_instance, gitlab_project)
-    runner_exists = gitlab_runner.exists_runner(runner_description, owned)
+    runner_exists = gitlab_runner.exists_runner(runner_description)
 
     if state == 'absent':
         if runner_exists:

plugins/modules/storage/pmem/pmem.py

@@ -0,0 +1,628 @@
+#!/usr/bin/python
+# -*- coding: utf-8 -*-
+# Copyright (c) 2022, Masayoshi Mizuma <msys.mizuma@gmail.com>
+# GNU General Public License v3.0+ (see COPYING or https://www.gnu.org/licenses/gpl-3.0.txt)
+
+from __future__ import (absolute_import, division, print_function)
+__metaclass__ = type
+
+DOCUMENTATION = r'''
+---
+author:
+ - Masayoshi Mizuma (@mizumm)
+module: pmem
+short_description: Configure Intel Optane Persistent Memory modules
+version_added: 4.5.0
+description:
+ - This module allows Configuring Intel Optane Persistent Memory modules
+   (PMem) using ipmctl and ndctl command line tools.
+requirements:
+ - ipmctl and ndctl command line tools
+ - xmltodict
+options:
+  appdirect:
+    description:
+     - Percentage of the total capacity to use in AppDirect Mode (C(0)-C(100)).
+     - Create AppDirect capacity utilizing hardware interleaving across the
+       requested PMem modules if applicable given the specified target.
+     - Total of I(appdirect), I(memorymode) and I(reserved) must be C(100)
+    type: int
+  appdirect_interleaved:
+    description:
+     - Create AppDirect capacity that is interleaved any other PMem modules.
+    type: bool
+    required: false
+    default: true
+  memorymode:
+    description:
+     - Percentage of the total capacity to use in Memory Mode (C(0)-C(100)).
+    type: int
+  reserved:
+    description:
+     - Percentage of the capacity to reserve (C(0)-C(100)). I(reserved) will not be mapped
+       into the system physical address space and will be presented as reserved
+       capacity with Show Device and Show Memory Resources Commands.
+     - I(reserved) will be set automatically if this is not configured.
+    type: int
+    required: false
+  socket:
+    description:
+     - This enables to set the configuration for each socket by using the socket ID.
+     - Total of I(appdirect), I(memorymode) and I(reserved) must be C(100) within one socket.
+    type: list
+    elements: dict
+    suboptions:
+      id:
+        description: The socket ID of the PMem module.
+        type: int
+        required: true
+      appdirect:
+        description:
+         - Percentage of the total capacity to use in AppDirect Mode (C(0)-C(100)) within the socket ID.
+        type: int
+        required: true
+      appdirect_interleaved:
+        description:
+         - Create AppDirect capacity that is interleaved any other PMem modules within the socket ID.
+        type: bool
+        required: false
+        default: true
+      memorymode:
+        description:
+         - Percentage of the total capacity to use in Memory Mode (C(0)-C(100)) within the socket ID.
+        type: int
+        required: true
+      reserved:
+        description:
+          - Percentage of the capacity to reserve (C(0)-C(100)) within the socket ID.
+        type: int
+  namespace:
+    description:
+     - This enables to set the configuration for the namespace of the PMem.
+    type: list
+    elements: dict
+    suboptions:
+      mode:
+        description:
+         - The mode of namespace. The detail of the mode is in the man page of ndctl-create-namespace.
+        type: str
+        required: true
+        choices: ['raw', 'sector', 'fsdax', 'devdax']
+      type:
+        description:
+         - The type of namespace. The detail of the type is in the man page of ndctl-create-namespace.
+        type: str
+        required: false
+        choices: ['pmem', 'blk']
+      size:
+        description:
+          - The size of namespace. This option supports the suffixes C(k) or C(K) or C(KB) for KiB,
+            C(m) or C(M) or C(MB) for MiB, C(g) or C(G) or C(GB) for GiB and C(t) or C(T) or C(TB) for TiB.
+          - This option is required if multiple namespaces are configured.
+          - If this option is not set, all of the avaiable space of a region is configured.
+        type: str
+        required: false
+  namespace_append:
+    description:
+     - Enable to append the new namespaces to the system.
+     - The default is C(false) so the all existing namespaces not listed in I(namespace) are removed.
+    type: bool
+    default: false
+    required: false
+'''
+
+RETURN = r'''
+reboot_required:
+    description: Indicates that the system reboot is required to complete the PMem configuration.
+    returned: success
+    type: bool
+    sample: True
+result:
+    description:
+     - Shows the value of AppDirect, Memory Mode and Reserved size in bytes.
+     - If I(socket) argument is provided, shows the values in each socket with C(socket) which contains the socket ID.
+     - If I(namespace) argument is provided, shows the detail of each namespace.
+    returned: success
+    type: list
+    elements: dict
+    contains:
+        appdirect:
+          description: AppDirect size in bytes.
+          type: int
+        memorymode:
+          description: Memory Mode size in bytes.
+          type: int
+        reserved:
+          description: Reserved size in bytes.
+          type: int
+        socket:
+          description: The socket ID to be configured.
+          type: int
+        namespace:
+          description: The list of the detail of namespace.
+          type: list
+    sample: [
+                {
+                    "appdirect": 111669149696,
+                    "memorymode": 970662608896,
+                    "reserved": 3626500096,
+                    "socket": 0
+                },
+                {
+                    "appdirect": 111669149696,
+                    "memorymode": 970662608896,
+                    "reserved": 3626500096,
+                    "socket": 1
+                }
+            ]
+'''
+
+EXAMPLES = r'''
+- name: Configure the Pmem as AppDirect 10, Memory Mode 70, and the Reserved 20 percent.
+  community.general.pmem:
+    appdirect: 10
+    memorymode: 70
+
+- name: Configure the Pmem as AppDirect 10, Memory Mode 80, and the Reserved 10 percent.
+  community.general.pmem:
+    appdirect: 10
+    memorymode: 80
+    reserved: 10
+
+- name: Configure the Pmem as AppDirect with not interleaved 10, Memory Mode 70, and the Reserved 20 percent.
+  community.general.pmem:
+    appdirect: 10
+    appdirect_interleaved: False
+    memorymode: 70
+
+- name: Configure the Pmem each socket.
+  community.general.pmem:
+    socket:
+      - id: 0
+        appdirect: 10
+        appdirect_interleaved: False
+        memorymode: 70
+        reserved: 20
+      - id: 1
+        appdirect: 10
+        memorymode: 80
+        reserved: 10
+
+- name: Configure the two namespaces.
+  community.general.pmem:
+    namespace:
+      - size: 1GB
+        type: pmem
+        mode: raw
+      - size: 320MB
+        type: pmem
+        mode: sector
+'''
+
+import json
+import re
+import traceback
+from ansible.module_utils.basic import AnsibleModule, missing_required_lib, human_to_bytes
+
+try:
+    import xmltodict
+except ImportError:
+    HAS_XMLTODICT_LIBRARY = False
+    XMLTODICT_LIBRARY_IMPORT_ERROR = traceback.format_exc()
+else:
+    HAS_XMLTODICT_LIBRARY = True
+
+
+class PersistentMemory(object):
+    def __init__(self):
+        module = AnsibleModule(
+            argument_spec=dict(
+                appdirect=dict(type='int'),
+                appdirect_interleaved=dict(type='bool', default=True),
+                memorymode=dict(type='int'),
+                reserved=dict(type='int'),
+                socket=dict(
+                    type='list', elements='dict',
+                    options=dict(
+                        id=dict(required=True, type='int'),
+                        appdirect=dict(required=True, type='int'),
+                        appdirect_interleaved=dict(type='bool', default=True),
+                        memorymode=dict(required=True, type='int'),
+                        reserved=dict(type='int'),
+                    ),
+                ),
+                namespace=dict(
+                    type='list', elements='dict',
+                    options=dict(
+                        mode=dict(required=True, type='str', choices=['raw', 'sector', 'fsdax', 'devdax']),
+                        type=dict(type='str', choices=['pmem', 'blk']),
+                        size=dict(type='str'),
+                    ),
+                ),
+                namespace_append=dict(type='bool', default=False),
+            ),
+            required_together=(
+                ['appdirect', 'memorymode'],
+            ),
+            required_one_of=(
+                ['appdirect', 'memorymode', 'socket', 'namespace'],
+            ),
+            mutually_exclusive=(
+                ['appdirect', 'socket'],
+                ['memorymode', 'socket'],
+                ['appdirect', 'namespace'],
+                ['memorymode', 'namespace'],
+                ['socket', 'namespace'],
+                ['appdirect', 'namespace_append'],
+                ['memorymode', 'namespace_append'],
+                ['socket', 'namespace_append'],
+            ),
+        )
+
+        if not HAS_XMLTODICT_LIBRARY:
+            module.fail_json(
+                msg=missing_required_lib('xmltodict'),
+                exception=XMLTODICT_LIBRARY_IMPORT_ERROR)
+
+        self.ipmctl_exec = module.get_bin_path('ipmctl', True)
+        self.ndctl_exec = module.get_bin_path('ndctl', True)
+
+        self.appdirect = module.params['appdirect']
+        self.interleaved = module.params['appdirect_interleaved']
+        self.memmode = module.params['memorymode']
+        self.reserved = module.params['reserved']
+        self.socket = module.params['socket']
+        self.namespace = module.params['namespace']
+        self.namespace_append = module.params['namespace_append']
+
+        self.module = module
+        self.changed = False
+        self.result = []
+
+    def pmem_run_command(self, command, returnCheck=True):
+        # in case command[] has number
+        cmd = [str(part) for part in command]
+
+        self.module.log(msg='pmem_run_command: execute: %s' % cmd)
+
+        rc, out, err = self.module.run_command(cmd)
+
+        self.module.log(msg='pmem_run_command: result: %s' % out)
+
+        if returnCheck and rc != 0:
+            self.module.fail_json(msg='Error while running: %s' %
+                                  cmd, rc=rc, out=out, err=err)
+
+        return out
+
+    def pmem_run_ipmctl(self, command, returnCheck=True):
+
+        command = [self.ipmctl_exec] + command
+
+        return self.pmem_run_command(command, returnCheck)
+
+    def pmem_run_ndctl(self, command, returnCheck=True):
+
+        command = [self.ndctl_exec] + command
+
+        return self.pmem_run_command(command, returnCheck)
+
+    def pmem_is_dcpmm_installed(self):
+        # To check this system has dcpmm
+        command = ['show', '-system', '-capabilities']
+        return self.pmem_run_ipmctl(command)
+
+    def pmem_get_region_align_size(self, region):
+        aligns = []
+        for rg in region:
+            if rg['align'] not in aligns:
+                aligns.append(rg['align'])
+
+        return aligns
+
+    def pmem_get_available_region_size(self, region):
+        available_size = []
+        for rg in region:
+            available_size.append(rg['available_size'])
+
+        return available_size
+
+    def pmem_get_available_region_type(self, region):
+        types = []
+        for rg in region:
+            if rg['type'] not in types:
+                types.append(rg['type'])
+
+        return types
+
+    def pmem_argument_check(self):
+        def namespace_check(self):
+            command = ['list', '-R']
+            out = self.pmem_run_ndctl(command)
+            if not out:
+                return 'Available region(s) is not in this system.'
+            region = json.loads(out)
+
+            aligns = self.pmem_get_region_align_size(region)
+            if len(aligns) != 1:
+                return 'Not supported the regions whose alignment size is different.'
+
+            available_size = self.pmem_get_available_region_size(region)
+            types = self.pmem_get_available_region_type(region)
+            for ns in self.namespace:
+                if ns['size']:
+                    try:
+                        size_byte = human_to_bytes(ns['size'])
+                    except ValueError:
+                        return 'The format of size: NNN TB|GB|MB|KB|T|G|M|K|B'
+
+                    if size_byte % aligns[0] != 0:
+                        return 'size: %s should be align with %d' % (ns['size'], aligns[0])
+
+                    is_space_enough = False
+                    for i, avail in enumerate(available_size):
+                        if avail > size_byte:
+                            available_size[i] -= size_byte
+                            is_space_enough = True
+                            break
+
+                    if is_space_enough is False:
+                        return 'There is not available region for size: %s' % ns['size']
+
+                    ns['size_byte'] = size_byte
+
+                elif len(self.namespace) != 1:
+                    return 'size option is required to configure multiple namespaces'
+
+                if ns['type'] not in types:
+                    return 'type %s is not supported in this system. Supported type: %s' % (ns['type'], types)
+
+            return None
+
+        def percent_check(self, appdirect, memmode, reserved=None):
+            if appdirect is None or (appdirect < 0 or appdirect > 100):
+                return 'appdirect percent should be from 0 to 100.'
+            if memmode is None or (memmode < 0 or memmode > 100):
+                return 'memorymode percent should be from 0 to 100.'
+
+            if reserved is None:
+                if appdirect + memmode > 100:
+                    return 'Total percent should be less equal 100.'
+            else:
+                if reserved < 0 or reserved > 100:
+                    return 'reserved percent should be from 0 to 100.'
+                if appdirect + memmode + reserved != 100:
+                    return 'Total percent should be 100.'
+
+        def socket_id_check(self):
+            command = ['show', '-o', 'nvmxml', '-socket']
+            out = self.pmem_run_ipmctl(command)
+            sockets_dict = xmltodict.parse(out, dict_constructor=dict)['SocketList']['Socket']
+            socket_ids = []
+            for sl in sockets_dict:
+                socket_ids.append(int(sl['SocketID'], 16))
+
+            for skt in self.socket:
+                if skt['id'] not in socket_ids:
+                    return 'Invalid socket number: %d' % skt['id']
+
+            return None
+
+        if self.namespace:
+            return namespace_check(self)
+        elif self.socket is None:
+            return percent_check(self, self.appdirect, self.memmode, self.reserved)
+        else:
+            ret = socket_id_check(self)
+            if ret is not None:
+                return ret
+
+            for skt in self.socket:
+                ret = percent_check(
+                    self, skt['appdirect'], skt['memorymode'], skt['reserved'])
+                if ret is not None:
+                    return ret
+
+            return None
+
+    def pmem_remove_namespaces(self):
+        command = ['list', '-N']
+        out = self.pmem_run_ndctl(command)
+
+        # There's nothing namespaces in this system. Nothing to do.
+        if not out:
+            return
+
+        namespaces = json.loads(out)
+
+        # Disable and destroy all namespaces
+        for ns in namespaces:
+            command = ['disable-namespace', ns['dev']]
+            self.pmem_run_ndctl(command)
+
+            command = ['destroy-namespace', ns['dev']]
+            self.pmem_run_ndctl(command)
+
+        return
+
+    def pmem_delete_goal(self):
+        # delete the goal request
+        command = ['delete', '-goal']
+        self.pmem_run_ipmctl(command)
+
+    def pmem_init_env(self):
+        if self.namespace is None or (self.namespace and self.namespace_append is False):
+            self.pmem_remove_namespaces()
+        if self.namespace is None:
+            self.pmem_delete_goal()
+
+    def pmem_get_capacity(self, skt=None):
+        command = ['show', '-d', 'Capacity', '-u', 'B', '-o', 'nvmxml', '-dimm']
+        if skt:
+            command += ['-socket', skt['id']]
+        out = self.pmem_run_ipmctl(command)
+
+        dimm_list = xmltodict.parse(out, dict_constructor=dict)['DimmList']['Dimm']
+        capacity = 0
+        for entry in dimm_list:
+            for key, v in entry.items():
+                if key == 'Capacity':
+                    capacity += int(v.split()[0])
+
+        return capacity
+
+    def pmem_create_memory_allocation(self, skt=None):
+        def build_ipmctl_creation_opts(self, skt=None):
+            ipmctl_opts = []
+
+            if skt:
+                appdirect = skt['appdirect']
+                memmode = skt['memorymode']
+                reserved = skt['reserved']
+                socket_id = skt['id']
+                ipmctl_opts += ['-socket', socket_id]
+            else:
+                appdirect = self.appdirect
+                memmode = self.memmode
+                reserved = self.reserved
+
+            if reserved is None:
+                res = 100 - memmode - appdirect
+                ipmctl_opts += ['memorymode=%d' % memmode, 'reserved=%d' % res]
+            else:
+                ipmctl_opts += ['memorymode=%d' % memmode, 'reserved=%d' % reserved]
+
+            if self.interleaved:
+                ipmctl_opts += ['PersistentMemoryType=AppDirect']
+            else:
+                ipmctl_opts += ['PersistentMemoryType=AppDirectNotInterleaved']
+
+            return ipmctl_opts
+
+        def is_allocation_good(self, ipmctl_out, command):
+            warning = re.compile('WARNING')
+            error = re.compile('.*Error.*')
+            ignore_error = re.compile(
+                'Do you want to continue? [y/n] Error: Invalid data input.')
+
+            errmsg = ''
+            rc = True
+            for line in ipmctl_out.splitlines():
+                if warning.match(line):
+                    errmsg = '%s (command: %s)' % (line, command)
+                    rc = False
+                    break
+                elif error.match(line):
+                    if not ignore_error:
+                        errmsg = '%s (command: %s)' % (line, command)
+                        rc = False
+                        break
+
+            return rc, errmsg
+
+        def get_allocation_result(self, goal, skt=None):
+            ret = {'appdirect': 0, 'memorymode': 0}
+
+            if skt:
+                ret['socket'] = skt['id']
+
+            out = xmltodict.parse(goal, dict_constructor=dict)['ConfigGoalList']['ConfigGoal']
+            for entry in out:
+
+                # Probably it's a bug of ipmctl to show the socket goal
+                # which isn't specified by the -socket option.
+                # Anyway, filter the noise out here:
+                if skt and skt['id'] != int(entry['SocketID'], 16):
+                    continue
+
+                for key, v in entry.items():
+                    if key == 'MemorySize':
+                        ret['memorymode'] += int(v.split()[0])
+                    elif key == 'AppDirect1Size' or key == 'AapDirect2Size':
+                        ret['appdirect'] += int(v.split()[0])
+
+            capacity = self.pmem_get_capacity(skt)
+            ret['reserved'] = capacity - ret['appdirect'] - ret['memorymode']
+
+            return ret
+
+        reboot_required = False
+
+        ipmctl_opts = build_ipmctl_creation_opts(self, skt)
+
+        # First, do dry run ipmctl create command to check the error and warning.
+        command = ['create', '-goal'] + ipmctl_opts
+        out = self.pmem_run_ipmctl(command, returnCheck=False)
+        rc, errmsg = is_allocation_good(self, out, command)
+        if rc is False:
+            return reboot_required, {}, errmsg
+
+        # Run actual creation here
+        command = ['create', '-u', 'B', '-o', 'nvmxml', '-force', '-goal'] + ipmctl_opts
+        goal = self.pmem_run_ipmctl(command)
+        ret = get_allocation_result(self, goal, skt)
+        reboot_required = True
+
+        return reboot_required, ret, ''
+
+    def pmem_config_namespaces(self, namespace):
+        command = ['create-namespace', '-m', namespace['mode']]
+        if namespace['type']:
+            command += ['-t', namespace['type']]
+        if 'size_byte' in namespace:
+            command += ['-s', namespace['size_byte']]
+
+        self.pmem_run_ndctl(command)
+
+        return None
+
+
+def main():
+
+    pmem = PersistentMemory()
+
+    pmem.pmem_is_dcpmm_installed()
+
+    error = pmem.pmem_argument_check()
+    if error:
+        pmem.module.fail_json(msg=error)
+
+    pmem.pmem_init_env()
+    pmem.changed = True
+
+    if pmem.namespace:
+        for ns in pmem.namespace:
+            pmem.pmem_config_namespaces(ns)
+
+        command = ['list', '-N']
+        out = pmem.pmem_run_ndctl(command)
+        all_ns = json.loads(out)
+
+        pmem.result = all_ns
+        reboot_required = False
+    elif pmem.socket is None:
+        reboot_required, ret, errmsg = pmem.pmem_create_memory_allocation()
+        if errmsg:
+            pmem.module.fail_json(msg=errmsg)
+        pmem.result.append(ret)
+    else:
+        for skt in pmem.socket:
+            skt_reboot_required, skt_ret, skt_errmsg = pmem.pmem_create_memory_allocation(skt)
+
+            if skt_errmsg:
+                pmem.module.fail_json(msg=skt_errmsg)
+
+            if skt_reboot_required:
+                reboot_required = True
+
+            pmem.result.append(skt_ret)
+
+    pmem.module.exit_json(
+        changed=pmem.changed,
+        reboot_required=reboot_required,
+        result=pmem.result
+    )
+
+
+if __name__ == '__main__':
+    main()

plugins/modules/system/dconf.py

@@ -180,9 +180,9 @@ class DBusWrapper(object):
         self.module.debug("Trying to detect existing D-Bus user session for user: %d" % uid)
 
         for pid in psutil.pids():
-            process = psutil.Process(pid)
-            process_real_uid, dummy, dummy = process.uids()
             try:
+                process = psutil.Process(pid)
+                process_real_uid, dummy, dummy = process.uids()
                 if process_real_uid == uid and 'DBUS_SESSION_BUS_ADDRESS' in process.environ():
                     dbus_session_bus_address_candidate = process.environ()['DBUS_SESSION_BUS_ADDRESS']
                     self.module.debug("Found D-Bus user session candidate at address: %s" % dbus_session_bus_address_candidate)
@@ -198,6 +198,9 @@ class DBusWrapper(object):
             # This can happen with things like SSH sessions etc.
             except psutil.AccessDenied:
                 pass
+            # Process has disappeared while inspecting it
+            except psutil.NoSuchProcess:
+                pass
 
         self.module.debug("Failed to find running D-Bus user session, will use dbus-run-session")
 

plugins/modules/system/vdo.py

@@ -386,7 +386,6 @@ def deactivate_vdo(module, vdoname, vdocmd):
 
 
 def add_vdooptions(params):
-    vdocmdoptions = ""
     options = []
 
     if params.get('logicalsize') is not None:
@@ -437,7 +436,7 @@ def add_vdooptions(params):
     if params.get('physicalthreads') is not None:
         options.append("--vdoPhysicalThreads=" + params['physicalthreads'])
 
-    return vdocmdoptions
+    return options
 
 
 def run_module():

plugins/modules/system/xfconf.py

@@ -107,23 +107,26 @@ RETURN = '''
     - The type of the value that was changed (C(none) for C(get) and C(reset)
       state). Either a single string value or a list of strings for array
       types.
+    - This is a string or a list of strings.
     returned: success
-    type: string or list of strings
+    type: any
     sample: '"int" or ["str", "str", "str"]'
   value:
     description:
     - The value of the preference key after executing the module. Either a
       single string value or a list of strings for array types.
+    - This is a string or a list of strings.
     returned: success
-    type: string or list of strings
+    type: any
     sample: '"192" or ["orange", "yellow", "violet"]'
   previous_value:
     description:
     - The value of the preference key before executing the module (C(none) for
       C(get) state). Either a single string value or a list of strings for array
       types.
+    - This is a string or a list of strings.
     returned: success
-    type: string or list of strings
+    type: any
     sample: '"96" or ["red", "blue", "green"]'
 '''
 

tests/integration/targets/alternatives/tasks/main.yml

@@ -68,4 +68,13 @@
   # in chkconfig-1.7-2 fails when /etc/alternatives/dummy link is missing,
   # error is: 'failed to read link /usr/bin/dummy: No such file or directory'.
   # Moreover Fedora 24 is no longer maintained.
-  when: ansible_distribution != 'Fedora' or ansible_distribution_major_version|int > 24
+  #
+  # *Disable tests on Arch Linux*
+  # TODO: figure out whether there is an alternatives tool for Arch Linux
+  #
+  # *Disable tests on Alpine*
+  # TODO: figure out whether there is an alternatives tool for Alpine
+  when:
+    - ansible_distribution != 'Fedora' or ansible_distribution_major_version|int > 24
+    - ansible_distribution != 'Archlinux'
+    - ansible_distribution != 'Alpine'

tests/integration/targets/ansible_galaxy_install/aliases

@@ -2,3 +2,4 @@ destructive
 shippable/posix/group3
 skip/python2.6
 context/controller  # While this is not really true, this module mainly is run on the controller, *and* needs access to the ansible-galaxy CLI tool
+disabled  # FIXME

tests/integration/targets/cloud_init_data_facts/tasks/main.yml

@@ -15,6 +15,8 @@
   when:
   - not (ansible_distribution == "Ubuntu" and ansible_distribution_major_version|int == 14)
   - not (ansible_os_family == "Suse" and ansible_distribution_major_version|int != 42 and ansible_python.version.major != 3)
+  - not (ansible_distribution == 'Archlinux')  # TODO: package seems to be broken, cannot be downloaded from mirrors?
+  - not (ansible_distribution == 'Alpine')  # TODO: not sure what's wrong here, the module doesn't return what the tests expect
   block:
   - name: setup install cloud-init
     package:

tests/integration/targets/consul/tasks/consul_session.yml

@@ -82,50 +82,48 @@
     that:
       - result is failed
 
-- when: pyopenssl_version.stdout is version('0.15', '>=')
-  block:
-    - name: ensure SSL certificate is checked
-      consul_session:
-        state: info
-        id: '{{ session_id }}'
-        port: 8501
-        scheme: https
-      register: result
-      ignore_errors: True
+- name: ensure SSL certificate is checked
+  consul_session:
+    state: info
+    id: '{{ session_id }}'
+    port: 8501
+    scheme: https
+  register: result
+  ignore_errors: True
 
-    - name: previous task should fail since certificate is not known
-      assert:
-        that:
-          - result is failed
-          - "'certificate verify failed' in result.msg"
+- name: previous task should fail since certificate is not known
+  assert:
+    that:
+      - result is failed
+      - "'certificate verify failed' in result.msg"
 
-    - name: ensure SSL certificate isn't checked when validate_certs is disabled
-      consul_session:
-        state: info
-        id: '{{ session_id }}'
-        port: 8501
-        scheme: https
-        validate_certs: False
-      register: result
+- name: ensure SSL certificate isn't checked when validate_certs is disabled
+  consul_session:
+    state: info
+    id: '{{ session_id }}'
+    port: 8501
+    scheme: https
+    validate_certs: False
+  register: result
 
-    - name: previous task should succeed since certificate isn't checked
-      assert:
-        that:
-          - result is changed
+- name: previous task should succeed since certificate isn't checked
+  assert:
+    that:
+      - result is changed
 
-    - name: ensure a secure connection is possible
-      consul_session:
-        state: info
-        id: '{{ session_id }}'
-        port: 8501
-        scheme: https
-      environment:
-        REQUESTS_CA_BUNDLE: '{{ remote_dir }}/cert.pem'
-      register: result
+- name: ensure a secure connection is possible
+  consul_session:
+    state: info
+    id: '{{ session_id }}'
+    port: 8501
+    scheme: https
+  environment:
+    REQUESTS_CA_BUNDLE: '{{ remote_dir }}/cert.pem'
+  register: result
 
-    - assert:
-        that:
-          - result is changed
+- assert:
+    that:
+      - result is changed
 
 - name: delete a session
   consul_session:

tests/integration/targets/consul/tasks/main.yml

@@ -9,9 +9,6 @@
     consul_uri: https://s3.amazonaws.com/ansible-ci-files/test/integration/targets/consul/consul_{{ consul_version }}_{{ ansible_system | lower }}_{{ consul_arch }}.zip
     consul_cmd: '{{ remote_tmp_dir }}/consul'
   block:
-  - name: register pyOpenSSL version
-    command: '{{ ansible_python_interpreter }} -c ''import OpenSSL; print(OpenSSL.__version__)'''
-    register: pyopenssl_version
   - name: Install requests<2.20 (CentOS/RHEL 6)
     pip:
       name: requests<2.20
@@ -23,25 +20,23 @@
       name: python-consul
     register: result
     until: result is success
-  - when: pyopenssl_version.stdout is version('0.15', '>=')
-    block:
-    - name: Generate privatekey
-      community.crypto.openssl_privatekey:
-        path: '{{ remote_tmp_dir }}/privatekey.pem'
-    - name: Generate CSR
-      community.crypto.openssl_csr:
-        path: '{{ remote_tmp_dir }}/csr.csr'
-        privatekey_path: '{{ remote_tmp_dir }}/privatekey.pem'
-        subject:
-          commonName: localhost
-    - name: Generate selfsigned certificate
-      register: selfsigned_certificate
-      community.crypto.x509_certificate:
-        path: '{{ remote_tmp_dir }}/cert.pem'
-        csr_path: '{{ remote_tmp_dir }}/csr.csr'
-        privatekey_path: '{{ remote_tmp_dir }}/privatekey.pem'
-        provider: selfsigned
-        selfsigned_digest: sha256
+  - name: Generate privatekey
+    community.crypto.openssl_privatekey:
+      path: '{{ remote_tmp_dir }}/privatekey.pem'
+  - name: Generate CSR
+    community.crypto.openssl_csr:
+      path: '{{ remote_tmp_dir }}/csr.csr'
+      privatekey_path: '{{ remote_tmp_dir }}/privatekey.pem'
+      subject:
+        commonName: localhost
+  - name: Generate selfsigned certificate
+    register: selfsigned_certificate
+    community.crypto.x509_certificate:
+      path: '{{ remote_tmp_dir }}/cert.pem'
+      csr_path: '{{ remote_tmp_dir }}/csr.csr'
+      privatekey_path: '{{ remote_tmp_dir }}/privatekey.pem'
+      provider: selfsigned
+      selfsigned_digest: sha256
   - name: Install unzip
     package:
       name: unzip

tests/integration/targets/consul/templates/consul_config.hcl.j2

@@ -3,11 +3,7 @@ server = true
 pid_file = "{{ remote_dir }}/consul.pid"
 ports {
   http = 8500
-  {% if  pyopenssl_version.stdout is version('0.15', '>=') %}
   https = 8501
-  {% endif %}
 }
-{% if  pyopenssl_version.stdout is version('0.15', '>=') %}
 key_file = "{{ remote_dir }}/privatekey.pem"
 cert_file = "{{ remote_dir }}/cert.pem"
-{% endif %}

tests/integration/targets/cronvar/tasks/main.yml

@@ -3,112 +3,115 @@
 # and should not be used as examples of how to write Ansible roles #
 ####################################################################
 
-- name: Create EMAIL cron var
-  cronvar:
-    name: EMAIL
-    value: doug@ansibmod.con.com
-  register: create_cronvar1
+- when:
+    - not (ansible_os_family == 'Alpine' and ansible_distribution_version is version('3.15', '<'))  # TODO
+  block:
+  - name: Create EMAIL cron var
+    cronvar:
+      name: EMAIL
+      value: doug@ansibmod.con.com
+    register: create_cronvar1
 
-- name: Create EMAIL cron var again
-  cronvar:
-    name: EMAIL
-    value: doug@ansibmod.con.com
-  register: create_cronvar2
+  - name: Create EMAIL cron var again
+    cronvar:
+      name: EMAIL
+      value: doug@ansibmod.con.com
+    register: create_cronvar2
 
-- name: Check cron var value
-  shell: crontab -l -u root | grep -c EMAIL=doug@ansibmod.con.com
-  register: varcheck1
+  - name: Check cron var value
+    shell: crontab -l -u root | grep -c EMAIL=doug@ansibmod.con.com
+    register: varcheck1
 
-- name: Modify EMAIL cron var
-  cronvar:
-    name: EMAIL
-    value: jane@ansibmod.con.com
-  register: create_cronvar3
+  - name: Modify EMAIL cron var
+    cronvar:
+      name: EMAIL
+      value: jane@ansibmod.con.com
+    register: create_cronvar3
 
-- name: Check cron var value again
-  shell: crontab -l -u root | grep -c EMAIL=jane@ansibmod.con.com
-  register: varcheck2
+  - name: Check cron var value again
+    shell: crontab -l -u root | grep -c EMAIL=jane@ansibmod.con.com
+    register: varcheck2
 
-- name: Remove EMAIL cron var
-  cronvar:
-    name: EMAIL
-    state: absent
-  register: remove_cronvar1
+  - name: Remove EMAIL cron var
+    cronvar:
+      name: EMAIL
+      state: absent
+    register: remove_cronvar1
 
-- name: Remove EMAIL cron var again
-  cronvar:
-    name: EMAIL
-    state: absent
-  register: remove_cronvar2
+  - name: Remove EMAIL cron var again
+    cronvar:
+      name: EMAIL
+      state: absent
+    register: remove_cronvar2
 
-- name: Check cron var value again
-  shell: crontab -l -u root | grep -c EMAIL
-  register: varcheck3
-  failed_when: varcheck3.rc == 0
+  - name: Check cron var value again
+    shell: crontab -l -u root | grep -c EMAIL
+    register: varcheck3
+    failed_when: varcheck3.rc == 0
 
-- name: Add cron var to custom file
-  cronvar:
-    name: TESTVAR
-    value: somevalue
-    cron_file: cronvar_test
-  register: custom_cronfile1
+  - name: Add cron var to custom file
+    cronvar:
+      name: TESTVAR
+      value: somevalue
+      cron_file: cronvar_test
+    register: custom_cronfile1
 
-- name: Add cron var to custom file again
-  cronvar:
-    name: TESTVAR
-    value: somevalue
-    cron_file: cronvar_test
-  register: custom_cronfile2
+  - name: Add cron var to custom file again
+    cronvar:
+      name: TESTVAR
+      value: somevalue
+      cron_file: cronvar_test
+    register: custom_cronfile2
 
-- name: Check cron var value in custom file
-  command: grep -c TESTVAR=somevalue {{ cron_config_path }}/cronvar_test
-  register: custom_varcheck1
+  - name: Check cron var value in custom file
+    command: grep -c TESTVAR=somevalue {{ cron_config_path }}/cronvar_test
+    register: custom_varcheck1
 
-- name: Change cron var in custom file
-  cronvar:
-    name: TESTVAR
-    value: newvalue
-    cron_file: cronvar_test
-  register: custom_cronfile3
+  - name: Change cron var in custom file
+    cronvar:
+      name: TESTVAR
+      value: newvalue
+      cron_file: cronvar_test
+    register: custom_cronfile3
 
-- name: Check cron var value in custom file
-  command: grep -c TESTVAR=newvalue {{ cron_config_path }}/cronvar_test
-  register: custom_varcheck2
+  - name: Check cron var value in custom file
+    command: grep -c TESTVAR=newvalue {{ cron_config_path }}/cronvar_test
+    register: custom_varcheck2
 
-- name: Remove cron var from custom file
-  cronvar:
-    name: TESTVAR
-    value: newvalue
-    cron_file: cronvar_test
-    state: absent
-  register: custom_remove_cronvar1
+  - name: Remove cron var from custom file
+    cronvar:
+      name: TESTVAR
+      value: newvalue
+      cron_file: cronvar_test
+      state: absent
+    register: custom_remove_cronvar1
 
-- name: Remove cron var from custom file again
-  cronvar:
-    name: TESTVAR
-    value: newvalue
-    cron_file: cronvar_test
-    state: absent
-  register: custom_remove_cronvar2
+  - name: Remove cron var from custom file again
+    cronvar:
+      name: TESTVAR
+      value: newvalue
+      cron_file: cronvar_test
+      state: absent
+    register: custom_remove_cronvar2
 
-- name: Check cron var value
-  command: grep -c TESTVAR=newvalue {{ cron_config_path }}/cronvar_test
-  register: custom_varcheck3
-  failed_when: custom_varcheck3.rc == 0
+  - name: Check cron var value
+    command: grep -c TESTVAR=newvalue {{ cron_config_path }}/cronvar_test
+    register: custom_varcheck3
+    failed_when: custom_varcheck3.rc == 0
 
-- name: Esure cronvar tasks did the right thing
-  assert:
-    that:
-      - create_cronvar1 is changed
-      - create_cronvar2 is not changed
-      - create_cronvar3 is changed
-      - remove_cronvar1 is changed
-      - remove_cronvar2 is not changed
-      - varcheck1.stdout == '1'
-      - varcheck2.stdout == '1'
-      - varcheck3.stdout == '0'
-      - custom_remove_cronvar1 is changed
-      - custom_remove_cronvar2 is not changed
-      - custom_varcheck1.stdout == '1'
-      - custom_varcheck2.stdout == '1'
-      - custom_varcheck3.stdout == '0'
+  - name: Esure cronvar tasks did the right thing
+    assert:
+      that:
+        - create_cronvar1 is changed
+        - create_cronvar2 is not changed
+        - create_cronvar3 is changed
+        - remove_cronvar1 is changed
+        - remove_cronvar2 is not changed
+        - varcheck1.stdout == '1'
+        - varcheck2.stdout == '1'
+        - varcheck3.stdout == '0'
+        - custom_remove_cronvar1 is changed
+        - custom_remove_cronvar2 is not changed
+        - custom_varcheck1.stdout == '1'
+        - custom_varcheck2.stdout == '1'
+        - custom_varcheck3.stdout == '0'

tests/integration/targets/django_manage/tasks/main.yaml

@@ -8,6 +8,18 @@
     suffix: .django_manage
   register: tmp_django_root
 
+- name: Install virtualenv
+  package:
+    name: virtualenv
+    state: present
+  when: ansible_distribution == 'CentOS' and ansible_distribution_major_version == '8'
+
+- name: Install virtualenv
+  package:
+    name: python-virtualenv
+    state: present
+  when: ansible_os_family == 'Archlinux'
+
 - name: Install required library
   pip:
     name: django

tests/integration/targets/filesize/tasks/main.yml

@@ -29,6 +29,7 @@
       include_tasks: sparse.yml
       when:
         - not (ansible_os_family == 'Darwin' and ansible_distribution_version is version('11', '<'))
+        - not (ansible_os_family == 'Alpine')  # TODO figure out why it fails
 
     - name: Include tasks to test playing with symlinks
       include_tasks: symlinks.yml

tests/integration/targets/filesystem/tasks/create_fs.yml

@@ -1,5 +1,5 @@
 ---
-- name: "Create filesystem"
+- name: "Create filesystem ({{ fstype }})"
   community.general.filesystem:
     dev: '{{ dev }}'
     fstype: '{{ fstype }}'

tests/integration/targets/filesystem/tasks/main.yml

@@ -48,10 +48,18 @@
     # reiserfs-utils package not available with Fedora 35 on CI
     - 'not (ansible_distribution == "Fedora" and (ansible_facts.distribution_major_version | int >= 35) and
             item.0.key == "reiserfs")'
+    # reiserfs packages apparently not available with Alpine
+    - 'not (ansible_distribution == "Alpine" and item.0.key == "reiserfs")'
     # ocfs2 only available on Debian based distributions
     - 'not (item.0.key == "ocfs2" and ansible_os_family != "Debian")'
     # Tests use losetup which can not be used inside unprivileged container
     - 'not (item.0.key == "lvm" and ansible_virtualization_type in ["docker", "container", "containerd"])'
+    # vfat resizing fails on Debian (but not Ubuntu)
+    - 'not (item.0.key == "vfat" and ansible_distribution == "Debian")'  # TODO: figure out why it fails, fix it!
+    # vfat resizing fails on ArchLinux
+    - 'not (item.0.key == "vfat" and ansible_distribution == "Archlinux")'  # TODO: figure out why it fails, fix it!
+    # btrfs-progs cannot be installed on ArchLinux
+    - 'not (item.0.key == "btrfs" and ansible_distribution == "Archlinux")'  # TODO: figure out why it fails, fix it!
 
     # On CentOS 6 shippable containers, wipefs seems unable to remove vfat signatures
     - 'not (ansible_distribution == "CentOS" and ansible_distribution_version is version("7.0", "<") and
@@ -65,6 +73,9 @@
     - 'not (ansible_os_family == "Suse" and ansible_distribution_major_version|int != 42 and
             item.0.key == "xfs" and ansible_python.version.major == 2)'
 
+    # TODO: something seems to be broken on Alpine
+    - 'not (ansible_distribution == "Alpine")'
+
   loop: "{{ query('dict', tested_filesystems)|product(['create_fs', 'overwrite_another_fs', 'remove_fs'])|list }}"
 
 

tests/integration/targets/filesystem/tasks/setup.yml

@@ -21,6 +21,7 @@
     - not (ansible_distribution == 'Ubuntu' and ansible_distribution_version is version('16.04', '<='))
     - ansible_system != "FreeBSD"
     - not (ansible_facts.os_family == "RedHat" and ansible_facts.distribution_major_version is version('8', '>='))
+    - ansible_os_family != 'Archlinux'  # TODO
 
 - name: "Install btrfs tools (Ubuntu <= 16.04)"
   ansible.builtin.package:
@@ -60,7 +61,7 @@
     state: present
   when:
     - ansible_system == 'Linux'
-    - ansible_os_family not in ['Suse', 'RedHat']
+    - ansible_os_family not in ['Suse', 'RedHat', 'Alpine']
 
 - name: "Install reiserfs progs (FreeBSD)"
   ansible.builtin.package:
@@ -111,6 +112,7 @@
     - ansible_system == 'Linux'
     - ansible_os_family != 'Suse'
     - ansible_os_family != 'RedHat' or (ansible_distribution == 'CentOS' and ansible_distribution_version is version('7.0', '=='))
+    - ansible_os_family != 'Alpine'
   block:
     - name: "Install fatresize"
       ansible.builtin.package:

tests/integration/targets/gem/tasks/main.yml

@@ -21,201 +21,205 @@
 # You should have received a copy of the GNU General Public License
 # along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
 
-- include_vars: '{{ item }}'
-  with_first_found:
-    - files:
-        - '{{ ansible_distribution }}-{{ ansible_distribution_major_version }}.yml'
-        - '{{ ansible_distribution }}-{{ ansible_distribution_version }}.yml'
-        - '{{ ansible_os_family }}.yml'
-        - 'default.yml'
-      paths: '../vars'
+- when:
+    - not (ansible_os_family == 'Alpine')  # TODO
+  block:
 
-- name: Install dependencies for test
-  package:
-    name: "{{ item }}"
-    state: present
-  loop: "{{ test_packages }}"
-  when: ansible_distribution != "MacOSX"
+  - include_vars: '{{ item }}'
+    with_first_found:
+      - files:
+          - '{{ ansible_distribution }}-{{ ansible_distribution_major_version }}.yml'
+          - '{{ ansible_distribution }}-{{ ansible_distribution_version }}.yml'
+          - '{{ ansible_os_family }}.yml'
+          - 'default.yml'
+        paths: '../vars'
 
-- name: Install a gem
-  gem:
-    name: gist
-    state: present
-  register: install_gem_result
-  ignore_errors: yes
+  - name: Install dependencies for test
+    package:
+      name: "{{ item }}"
+      state: present
+    loop: "{{ test_packages }}"
+    when: ansible_distribution != "MacOSX"
 
-# when running as root on Fedora, '--install-dir' is set in the os defaults which is
-# incompatible with '--user-install', we ignore this error for this case only
-- name: fail if failed to install gem
-  fail:
-    msg: "failed to install gem: {{ install_gem_result.msg }}"
-  when:
-  - install_gem_result is failed
-  - not (ansible_user_uid == 0 and "User --install-dir or --user-install but not both" not in install_gem_result.msg)
-
-- block:
-  - name: List gems
-    command: gem list
-    register: current_gems
-
-  - name: Ensure gem was installed
-    assert:
-      that:
-        - install_gem_result is changed
-        - current_gems.stdout is search('gist\s+\([0-9.]+\)')
-
-  - name: Remove a gem
+  - name: Install a gem
     gem:
       name: gist
-      state: absent
-    register: remove_gem_results
+      state: present
+    register: install_gem_result
+    ignore_errors: yes
 
-  - name: List gems
-    command: gem list
-    register: current_gems
+  # when running as root on Fedora, '--install-dir' is set in the os defaults which is
+  # incompatible with '--user-install', we ignore this error for this case only
+  - name: fail if failed to install gem
+    fail:
+      msg: "failed to install gem: {{ install_gem_result.msg }}"
+    when:
+    - install_gem_result is failed
+    - not (ansible_user_uid == 0 and "User --install-dir or --user-install but not both" not in install_gem_result.msg)
 
-  - name: Verify gem is not installed
+  - block:
+    - name: List gems
+      command: gem list
+      register: current_gems
+
+    - name: Ensure gem was installed
+      assert:
+        that:
+          - install_gem_result is changed
+          - current_gems.stdout is search('gist\s+\([0-9.]+\)')
+
+    - name: Remove a gem
+      gem:
+        name: gist
+        state: absent
+      register: remove_gem_results
+
+    - name: List gems
+      command: gem list
+      register: current_gems
+
+    - name: Verify gem is not installed
+      assert:
+        that:
+          - remove_gem_results is changed
+          - current_gems.stdout is not search('gist\s+\([0-9.]+\)')
+    when: not install_gem_result is failed
+
+  # install gem in --no-user-install
+  - block:
+    - name: Install a gem with --no-user-install
+      gem:
+        name: gist
+        state: present
+        user_install: no
+      register: install_gem_result
+
+    - name: List gems
+      command: gem list
+      register: current_gems
+
+    - name: Ensure gem was installed
+      assert:
+        that:
+          - install_gem_result is changed
+          - current_gems.stdout is search('gist\s+\([0-9.]+\)')
+
+    - name: Remove a gem
+      gem:
+        name: gist
+        state: absent
+      register: remove_gem_results
+
+    - name: List gems
+      command: gem list
+      register: current_gems
+
+    - name: Verify gem is not installed
+      assert:
+        that:
+          - remove_gem_results is changed
+          - current_gems.stdout is not search('gist\s+\([0-9.]+\)')
+    when: ansible_user_uid == 0
+
+  # Check cutom gem directory
+  - name: Install gem in a custom directory with incorrect options
+    gem:
+      name: gist
+      state: present
+      install_dir: "{{ remote_tmp_dir }}/gems"
+    ignore_errors: yes
+    register: install_gem_fail_result
+
+  - debug:
+      var: install_gem_fail_result
+    tags: debug
+
+  - name: Ensure previous task failed
     assert:
       that:
-        - remove_gem_results is changed
-        - current_gems.stdout is not search('gist\s+\([0-9.]+\)')
-  when: not install_gem_result is failed
+        - install_gem_fail_result is failed
+        - install_gem_fail_result.msg == 'install_dir requires user_install=false'
 
-# install gem in --no-user-install
-- block:
-  - name: Install a gem with --no-user-install
+  - name: Install a gem in a custom directory
     gem:
       name: gist
       state: present
       user_install: no
+      install_dir: "{{ remote_tmp_dir }}/gems"
     register: install_gem_result
 
-  - name: List gems
-    command: gem list
-    register: current_gems
+  - name: Find gems in custom directory
+    find:
+      paths: "{{ remote_tmp_dir }}/gems/gems"
+      file_type: directory
+      contains: gist
+    register: gem_search
 
-  - name: Ensure gem was installed
+  - name: Ensure gem was installed in custom directory
     assert:
       that:
         - install_gem_result is changed
-        - current_gems.stdout is search('gist\s+\([0-9.]+\)')
+        - gem_search.files[0].path is search('gist-[0-9.]+')
+    ignore_errors: yes
 
-  - name: Remove a gem
+  - name: Remove a gem in a custom directory
     gem:
       name: gist
       state: absent
-    register: remove_gem_results
+      user_install: no
+      install_dir: "{{ remote_tmp_dir }}/gems"
+    register: install_gem_result
 
-  - name: List gems
-    command: gem list
-    register: current_gems
+  - name: Find gems in custom directory
+    find:
+      paths: "{{ remote_tmp_dir }}/gems/gems"
+      file_type: directory
+      contains: gist
+    register: gem_search
 
-  - name: Verify gem is not installed
+  - name: Ensure gem was removed in custom directory
     assert:
       that:
-        - remove_gem_results is changed
-        - current_gems.stdout is not search('gist\s+\([0-9.]+\)')
-  when: ansible_user_uid == 0
+        - install_gem_result is changed
+        - gem_search.files | length == 0
 
-# Check cutom gem directory
-- name: Install gem in a custom directory with incorrect options
-  gem:
-    name: gist
-    state: present
-    install_dir: "{{ remote_tmp_dir }}/gems"
-  ignore_errors: yes
-  register: install_gem_fail_result
+  # Custom directory for executables (--bindir)
+  - name: Install gem with custom bindir
+    gem:
+      name: gist
+      state: present
+      bindir: "{{ remote_tmp_dir }}/custom_bindir"
+      norc: yes
+      user_install: no  # Avoid conflicts between --install-dir and --user-install when running as root on CentOS / Fedora / RHEL
+    register: install_gem_result
 
-- debug:
-    var: install_gem_fail_result
-  tags: debug
+  - name: Get stats of gem executable
+    stat:
+      path: "{{ remote_tmp_dir }}/custom_bindir/gist"
+    register: gem_bindir_stat
 
-- name: Ensure previous task failed
-  assert:
-    that:
-      - install_gem_fail_result is failed
-      - install_gem_fail_result.msg == 'install_dir requires user_install=false'
+  - name: Ensure gem executable was installed in custom directory
+    assert:
+      that:
+        - install_gem_result is changed
+        - gem_bindir_stat.stat.exists and gem_bindir_stat.stat.isreg
 
-- name: Install a gem in a custom directory
-  gem:
-    name: gist
-    state: present
-    user_install: no
-    install_dir: "{{ remote_tmp_dir }}/gems"
-  register: install_gem_result
+  - name: Remove gem with custom bindir
+    gem:
+      name: gist
+      state: absent
+      bindir: "{{ remote_tmp_dir }}/custom_bindir"
+      norc: yes
+      user_install: no  # Avoid conflicts between --install-dir and --user-install when running as root on CentOS / Fedora / RHEL
+    register: install_gem_result
 
-- name: Find gems in custom directory
-  find:
-    paths: "{{ remote_tmp_dir }}/gems/gems"
-    file_type: directory
-    contains: gist
-  register: gem_search
+  - name: Get stats of gem executable
+    stat:
+      path: "{{ remote_tmp_dir }}/custom_bindir/gist"
+    register: gem_bindir_stat
 
-- name: Ensure gem was installed in custom directory
-  assert:
-    that:
-      - install_gem_result is changed
-      - gem_search.files[0].path is search('gist-[0-9.]+')
-  ignore_errors: yes
-
-- name: Remove a gem in a custom directory
-  gem:
-    name: gist
-    state: absent
-    user_install: no
-    install_dir: "{{ remote_tmp_dir }}/gems"
-  register: install_gem_result
-
-- name: Find gems in custom directory
-  find:
-    paths: "{{ remote_tmp_dir }}/gems/gems"
-    file_type: directory
-    contains: gist
-  register: gem_search
-
-- name: Ensure gem was removed in custom directory
-  assert:
-    that:
-      - install_gem_result is changed
-      - gem_search.files | length == 0
-
-# Custom directory for executables (--bindir)
-- name: Install gem with custom bindir
-  gem:
-    name: gist
-    state: present
-    bindir: "{{ remote_tmp_dir }}/custom_bindir"
-    norc: yes
-    user_install: no  # Avoid conflicts between --install-dir and --user-install when running as root on CentOS / Fedora / RHEL
-  register: install_gem_result
-
-- name: Get stats of gem executable
-  stat:
-    path: "{{ remote_tmp_dir }}/custom_bindir/gist"
-  register: gem_bindir_stat
-
-- name: Ensure gem executable was installed in custom directory
-  assert:
-    that:
-      - install_gem_result is changed
-      - gem_bindir_stat.stat.exists and gem_bindir_stat.stat.isreg
-
-- name: Remove gem with custom bindir
-  gem:
-    name: gist
-    state: absent
-    bindir: "{{ remote_tmp_dir }}/custom_bindir"
-    norc: yes
-    user_install: no  # Avoid conflicts between --install-dir and --user-install when running as root on CentOS / Fedora / RHEL
-  register: install_gem_result
-
-- name: Get stats of gem executable
-  stat:
-    path: "{{ remote_tmp_dir }}/custom_bindir/gist"
-  register: gem_bindir_stat
-
-- name: Ensure gem executable was removed from custom directory
-  assert:
-    that:
-      - install_gem_result is changed
-      - not gem_bindir_stat.stat.exists
+  - name: Ensure gem executable was removed from custom directory
+    assert:
+      that:
+        - install_gem_result is changed
+        - not gem_bindir_stat.stat.exists