Release 2.5.4.

* [nmcli] add connection.slave-type for teamed devices

* [nmcli] add fragment with changes for #2827

* [nmcli] add tests for network team

* [nmcli] fix testing

Co-authored-by: Oriol MULA VALLS <oriol.mula@lxp.lu>
(cherry picked from commit 2d1527a564)

Co-authored-by: omula <joriol.mula@gmail.com>

.github/BOTMETA.yml

@@ -1,5 +1,7 @@
 automerge: true
 files:
+  plugins/:
+    supershipit: quidame Ajpantuso
   changelogs/fragments/:
     support: community
   $actions:

CHANGELOG.rst

@@ -6,6 +6,26 @@ Community General Release Notes
 
 This changelog describes changes after version 1.0.0.
 
+v2.5.4
+======
+
+Release Summary
+---------------
+
+Regular bugfix release.
+
+Bugfixes
+--------
+
+- _mount module utils - fixed the sanity checks (https://github.com/ansible-collections/community.general/pull/2883).
+- gitlab_project - user projects are created using namespace ID now, instead of user ID (https://github.com/ansible-collections/community.general/pull/2881).
+- ipa_sudorule - call ``sudorule_add_allow_command`` method instead of  ``sudorule_add_allow_command_group`` (https://github.com/ansible-collections/community.general/issues/2442).
+- modprobe - added additional checks to ensure module load/unload is effective (https://github.com/ansible-collections/community.general/issues/1608).
+- nmcli - fixes team-slave configuration by adding connection.slave-type (https://github.com/ansible-collections/community.general/issues/766).
+- npm - when the ``version`` option is used the comparison of installed vs missing will use name@version instead of just name, allowing version specific updates (https://github.com/ansible-collections/community.general/issues/2021).
+- proxmox_kvm - fix parsing of Proxmox VM information with device info not containing a comma, like disks backed by ZFS zvols (https://github.com/ansible-collections/community.general/issues/2840).
+- scaleway plugin inventory - fix ``JSON object must be str, not 'bytes'`` with Python 3.5 (https://github.com/ansible-collections/community.general/issues/2769).
+
 v2.5.3
 ======
 

CONTRIBUTING.md

@@ -29,4 +29,8 @@ Also, consider taking up a valuable, reviewed, but abandoned pull request which
 
 You can also read [our Quick-start development guide](https://github.com/ansible/community-docs/blob/main/create_pr_quick_start_guide.rst).
 
+## Test pull requests
+
+If you want to test a PR locally, refer to [our testing guide](https://github.com/ansible/community-docs/blob/main/test_pr_locally_guide.rst) for instructions on how do it quickly.
+
 If you find any inconsistencies or places in this document which can be improved, feel free to raise an issue or pull request to fix it.

README.md

@@ -3,12 +3,18 @@
 [![Build Status](https://dev.azure.com/ansible/community.general/_apis/build/status/CI?branchName=stable-2)](https://dev.azure.com/ansible/community.general/_build?definitionId=31)
 [![Codecov](https://img.shields.io/codecov/c/github/ansible-collections/community.general)](https://codecov.io/gh/ansible-collections/community.general)
 
-This repo contains the `community.general` Ansible Collection. The collection includes many modules and plugins supported by Ansible community which are not part of more specialized community collections.
+This repository contains the `community.general` Ansible Collection. The collection is a part of the Ansible package and includes many modules and plugins supported by Ansible community which are not part of more specialized community collections.
 
 You can find [documentation for this collection on the Ansible docs site](https://docs.ansible.com/ansible/latest/collections/community/general/).
 
 Please note that this collection does **not** support Windows targets. Only connection plugins included in this collection might support Windows targets, and will explicitly mention that in their documentation if they do so.
 
+## Code of Conduct
+
+We follow [Ansible Code of Conduct](https://docs.ansible.com/ansible/latest/community/code_of_conduct.html) in all our interactions within this project.
+
+If you encounter abusive behavior violating the [Ansible Code of Conduct](https://docs.ansible.com/ansible/latest/community/code_of_conduct.html), please refer to the [policy violations](https://docs.ansible.com/ansible/latest/community/code_of_conduct.html#policy-violations) section of the Code of Conduct for information on how to raise a complaint.
+
 ## Tested with Ansible
 
 Tested with the current Ansible 2.9, ansible-base 2.10 and ansible-core 2.11 releases and the current development version of ansible-core. Ansible versions before 2.9.10 are not supported.
@@ -23,7 +29,9 @@ Please check the included content on the [Ansible Galaxy page for this collectio
 
 ## Using this collection
 
-Before using the General community collection, you need to install the collection with the `ansible-galaxy` CLI:
+This collection is shipped with the Ansible package. So if you have it installed, no more action is required.
+
+If you have a minimal installation (only Ansible Core installed) or you want to use the latest version of the collection along with the whole Ansible package, you need to install the collection from [Ansible Galaxy](https://galaxy.ansible.com/community/general) manually with the `ansible-galaxy` command-line tool:
 
     ansible-galaxy collection install community.general
 
@@ -34,19 +42,29 @@ collections:
 - name: community.general
 ```
 
+Note that if you install the collection manually, it will not be upgraded automatically when you upgrade the Ansible package. To upgrade the collection to the latest available version, run the following command:
+
+```bash
+ansible-galaxy collection install community.general --upgrade
+```
+
+You can also install a specific version of the collection, for example, if you need to downgrade when something is broken in the latest version (please report an issue in this repository). Use the following syntax where `X.Y.Z` can be any [available version](https://galaxy.ansible.com/community/general):
+
+```bash
+ansible-galaxy collection install community.general:==X.Y.Z
+```
+
 See [Ansible Using collections](https://docs.ansible.com/ansible/latest/user_guide/collections_using.html) for more details.
 
 ## Contributing to this collection
 
-If you want to develop new content for this collection or improve what is already here, the easiest way to work on the collection is to clone it into one of the configured [`COLLECTIONS_PATH`](https://docs.ansible.com/ansible/latest/reference_appendices/config.html#collections-paths), and work on it there.
+The content of this collection is made by good people like you, a community of individuals collaborating on making the world better through developing automation software.
 
-For example, if you are working in the `~/dev` directory:
+All types of contributions are very welcome.
 
-```
-cd ~/dev
-git clone git@github.com:ansible-collections/community.general.git collections/ansible_collections/community/general
-export COLLECTIONS_PATH=$(pwd)/collections:$COLLECTIONS_PATH
-```
+You don't know how to start? Refer to our [contribution guide](https://github.com/ansible-collections/community.general/blob/main/CONTRIBUTING.md)!
+
+The current maintainers are listed in the [commit-rights.md](https://github.com/ansible-collections/community.general/blob/main/commit-rights.md#people) file. If you have questions or need help, feel free to mention them in the proposals.
 
 You can find more information in the [developer guide for collections](https://docs.ansible.com/ansible/devel/dev_guide/developing_collections.html#contributing-to-collections), and in the [Ansible Community Guide](https://docs.ansible.com/ansible/latest/community/index.html).
 
@@ -58,16 +76,15 @@ See [here](https://docs.ansible.com/ansible/devel/dev_guide/developing_collectio
 
 ### Communication
 
-We have a dedicated Working Group for Ansible development.
+We announce important development changes and releases through Ansible's [The Bullhorn newsletter](https://eepurl.com/gZmiEP). If you are a collection developer, be sure you are subscribed.
 
-You can find other people interested on the following [Libera.chat](https://libera.chat/) IRC channels -
-- `#ansible` - For general use questions and support.
-- `#ansible-devel` - For discussions on developer topics and code related to features or bugs in ansible-core.
-- `#ansible-community` - For discussions on community topics and community meetings, and for general development questions for community collections.
+Join us in the `#ansible` (general use questions and support), `#ansible-community` (community and collection development questions), and other [IRC channels](https://docs.ansible.com/ansible/devel/community/communication.html#irc-channels) on [Libera.chat](https://libera.chat).
+
+We take part in the global quarterly [Ansible Contributor Summit](https://github.com/ansible/community/wiki/Contributor-Summit) virtually or in-person. Track [The Bullhorn newsletter](https://eepurl.com/gZmiEP) and join us.
 
 For more information about communities, meetings and agendas see [Community Wiki](https://github.com/ansible/community/wiki/Community).
 
-For more information about [communication](https://docs.ansible.com/ansible/latest/community/communication.html)
+For more information about communication, refer to the [Ansible communication guide](https://docs.ansible.com/ansible/devel/community/communication.html).
 
 ### Publishing New Version
 

changelogs/changelog.yaml

@@ -1955,3 +1955,33 @@ releases:
     - 2731-mh-cmd-locale.yml
     - json_query_more_types.yml
     release_date: '2021-06-08'
+  2.5.4:
+    changes:
+      bugfixes:
+      - _mount module utils - fixed the sanity checks (https://github.com/ansible-collections/community.general/pull/2883).
+      - gitlab_project - user projects are created using namespace ID now, instead
+        of user ID (https://github.com/ansible-collections/community.general/pull/2881).
+      - ipa_sudorule - call ``sudorule_add_allow_command`` method instead of  ``sudorule_add_allow_command_group``
+        (https://github.com/ansible-collections/community.general/issues/2442).
+      - modprobe - added additional checks to ensure module load/unload is effective
+        (https://github.com/ansible-collections/community.general/issues/1608).
+      - nmcli - fixes team-slave configuration by adding connection.slave-type (https://github.com/ansible-collections/community.general/issues/766).
+      - npm - when the ``version`` option is used the comparison of installed vs missing
+        will use name@version instead of just name, allowing version specific updates
+        (https://github.com/ansible-collections/community.general/issues/2021).
+      - proxmox_kvm - fix parsing of Proxmox VM information with device info not containing
+        a comma, like disks backed by ZFS zvols (https://github.com/ansible-collections/community.general/issues/2840).
+      - scaleway plugin inventory - fix ``JSON object must be str, not 'bytes'`` with
+        Python 3.5 (https://github.com/ansible-collections/community.general/issues/2769).
+      release_summary: Regular bugfix release.
+    fragments:
+    - 2.5.4.yml
+    - 2771-scaleway_inventory_json_accept_byte_array.yml
+    - 2821-ipa_sudorule.yml
+    - 2827-nmcli_fix_team_slave.yml
+    - 2830-npm-version-update.yml
+    - 2841-proxmox_kvm_zfs_devstr.yml
+    - 2843-modprobe-failure-conditions.yml
+    - 2881-gitlab_project-fix_workspace_user.yaml
+    - 2883-_mount-fixed-sanity-checks.yml
+    release_date: '2021-06-29'

galaxy.yml

@@ -1,6 +1,6 @@
 namespace: community
 name: general
-version: 2.5.3
+version: 2.5.4
 readme: README.md
 authors:
   - Ansible (https://github.com/ansible)

plugins/inventory/scaleway.py

@@ -1,24 +1,24 @@
-# Copyright (c) 2017 Ansible Project
+# Copyright: (c) 2017 Ansible Project
 # GNU General Public License v3.0+ (see COPYING or https://www.gnu.org/licenses/gpl-3.0.txt)
 
 from __future__ import (absolute_import, division, print_function)
 
 __metaclass__ = type
 
-DOCUMENTATION = '''
+DOCUMENTATION = r'''
     name: scaleway
     author:
       - Remy Leone (@sieben)
     short_description: Scaleway inventory source
     description:
-        - Get inventory hosts from Scaleway
+        - Get inventory hosts from Scaleway.
     options:
         plugin:
-            description: token that ensures this is a source file for the 'scaleway' plugin.
+            description: Token that ensures this is a source file for the 'scaleway' plugin.
             required: True
             choices: ['scaleway', 'community.general.scaleway']
         regions:
-            description: Filter results on a specific Scaleway region
+            description: Filter results on a specific Scaleway region.
             type: list
             default:
                 - ams1
@@ -26,11 +26,13 @@ DOCUMENTATION = '''
                 - par2
                 - waw1
         tags:
-            description: Filter results on a specific tag
+            description: Filter results on a specific tag.
             type: list
         oauth_token:
             required: True
-            description: Scaleway OAuth token.
+            description:
+            - Scaleway OAuth token.
+            - More details on L(how to generate token, https://www.scaleway.com/en/docs/generate-api-keys/).
             env:
                 # in order of precedence
                 - name: SCW_TOKEN
@@ -48,14 +50,14 @@ DOCUMENTATION = '''
                 - hostname
                 - id
         variables:
-            description: 'set individual variables: keys are variable names and
+            description: 'Set individual variables: keys are variable names and
                           values are templates. Any value returned by the
                           L(Scaleway API, https://developer.scaleway.com/#servers-server-get)
                           can be used.'
             type: dict
 '''
 
-EXAMPLES = '''
+EXAMPLES = r'''
 # scaleway_inventory.yml file in YAML format
 # Example command line: ansible-inventory --list -i scaleway_inventory.yml
 
@@ -81,6 +83,15 @@ regions:
   - par1
 variables:
   ansible_host: public_ip.address
+
+# Using static strings as variables
+plugin: community.general.scaleway
+hostnames:
+  - hostname
+variables:
+  ansible_host: public_ip.address
+  ansible_connection: "'ssh'"
+  ansible_user: "'admin'"
 '''
 
 import json
@@ -89,7 +100,7 @@ from ansible.errors import AnsibleError
 from ansible.plugins.inventory import BaseInventoryPlugin, Constructable
 from ansible_collections.community.general.plugins.module_utils.scaleway import SCALEWAY_LOCATION, parse_pagination_link
 from ansible.module_utils.urls import open_url
-from ansible.module_utils._text import to_native
+from ansible.module_utils._text import to_native, to_text
 
 import ansible.module_utils.six.moves.urllib.parse as urllib_parse
 
@@ -105,7 +116,7 @@ def _fetch_information(token, url):
         except Exception as e:
             raise AnsibleError("Error while fetching %s: %s" % (url, to_native(e)))
         try:
-            raw_json = json.loads(response.read())
+            raw_json = json.loads(to_text(response.read()))
         except ValueError:
             raise AnsibleError("Incorrect JSON payload")
 
@@ -230,8 +241,7 @@ class InventoryModule(BaseInventoryPlugin, Constructable):
 
         if not matching_tags:
             return set()
-        else:
-            return matching_tags.union((server_zone,))
+        return matching_tags.union((server_zone,))
 
     def _filter_host(self, host_infos, hostname_preferences):
 

plugins/module_utils/_mount.py

@@ -48,6 +48,10 @@
 # agrees to be bound by the terms and conditions of this License
 # Agreement.
 
+from __future__ import absolute_import, division, print_function
+
+__metaclass__ = type
+
 import os
 
 

plugins/modules/cloud/misc/proxmox_kvm.py

@@ -815,26 +815,27 @@ def get_vminfo(module, proxmox, node, vmid, **kwargs):
             del kwargs[k]
 
     # Split information by type
-    for k, v in kwargs.items():
-        if re.match(r'net[0-9]', k) is not None:
-            interface = k
-            k = vm[k]
-            k = re.search('=(.*?),', k).group(1)
-            mac[interface] = k
-        if (re.match(r'virtio[0-9]', k) is not None or
-                re.match(r'ide[0-9]', k) is not None or
-                re.match(r'scsi[0-9]', k) is not None or
-                re.match(r'sata[0-9]', k) is not None):
-            device = k
-            k = vm[k]
-            k = re.search('(.*?),', k).group(1)
-            devices[device] = k
+    re_net = re.compile(r'net[0-9]')
+    re_dev = re.compile(r'(virtio|ide|scsi|sata)[0-9]')
+    for k in kwargs.keys():
+        if re_net.match(k):
+            mac[k] = parse_mac(vm[k])
+        elif re_dev.match(k):
+            devices[k] = parse_dev(vm[k])
 
     results['mac'] = mac
     results['devices'] = devices
     results['vmid'] = int(vmid)
 
 
+def parse_mac(netstr):
+    return re.search('=(.*?),', netstr).group(1)
+
+
+def parse_dev(devstr):
+    return re.search('(.*?)(,|$)', devstr).group(1)
+
+
 def settings(module, proxmox, vmid, node, name, **kwargs):
     proxmox_node = proxmox.nodes(node)
 

plugins/modules/cloud/softlayer/sl_vm.py

@@ -217,7 +217,7 @@ EXAMPLES = '''
         datacenter: dal09
         tags:
           - ansible-module-test
-          - ansible-module-test-slaves
+          - ansible-module-test-replicas
         hourly: yes
         private: no
         dedicated: no
@@ -235,7 +235,7 @@ EXAMPLES = '''
         datacenter: dal09
         tags:
           - ansible-module-test
-          - ansible-module-test-slaves
+          - ansible-module-test-replicas
         hourly: yes
         private: no
         dedicated: no

plugins/modules/identity/ipa/ipa_sudorule.py

@@ -237,7 +237,7 @@ class SudoRuleIPAClient(IPAClient):
         return self._post_json(method='sudorule_add_allow_command', name=name, item={'sudocmd': item})
 
     def sudorule_add_allow_command_group(self, name, item):
-        return self._post_json(method='sudorule_add_allow_command_group', name=name, item={'sudocmdgroup': item})
+        return self._post_json(method='sudorule_add_allow_command', name=name, item={'sudocmdgroup': item})
 
     def sudorule_remove_allow_command(self, name, item):
         return self._post_json(method='sudorule_remove_allow_command', name=name, item=item)

plugins/modules/net_tools/nmcli.py

@@ -757,6 +757,10 @@ class Nmcli(object):
                 'bridge-port.hairpin-mode': self.hairpin,
                 'bridge-port.priority': self.slavepriority,
             })
+        elif self.type == 'team-slave':
+            options.update({
+                'connection.slave-type': 'team',
+            })
         elif self.tunnel_conn_type:
             options.update({
                 'ip-tunnel.local': self.ip_tunnel_local,

plugins/modules/packaging/language/npm.py

@@ -181,7 +181,7 @@ class Npm(object):
                 cmd.append('--ignore-scripts')
             if self.unsafe_perm:
                 cmd.append('--unsafe-perm')
-            if self.name and add_package_name:
+            if self.name_version and add_package_name:
                 cmd.append(self.name_version)
             if self.registry:
                 cmd.append('--registry')
@@ -215,14 +215,17 @@ class Npm(object):
         except (getattr(json, 'JSONDecodeError', ValueError)) as e:
             self.module.fail_json(msg="Failed to parse NPM output with error %s" % to_native(e))
         if 'dependencies' in data:
-            for dep in data['dependencies']:
-                if 'missing' in data['dependencies'][dep] and data['dependencies'][dep]['missing']:
+            for dep, props in data['dependencies'].items():
+                dep_version = dep + '@' + str(props['version'])
+
+                if 'missing' in props and props['missing']:
                     missing.append(dep)
-                elif 'invalid' in data['dependencies'][dep] and data['dependencies'][dep]['invalid']:
+                elif 'invalid' in props and props['invalid']:
                     missing.append(dep)
                 else:
                     installed.append(dep)
-            if self.name and self.name not in installed:
+                    installed.append(dep_version)
+            if self.name_version and self.name_version not in installed:
                 missing.append(self.name)
         # Named dependency not installed
         else:

plugins/modules/packaging/os/pacman.py

@@ -30,9 +30,12 @@ options:
 
     state:
         description:
-            - Desired state of the package.
+          - Whether to install (C(present) or C(installed), C(latest)), or remove (C(absent) or C(removed)) a package.
+          - C(present) and C(installed) will simply ensure that a desired package is installed.
+          - C(latest) will update the specified package if it is not of the latest available version.
+          - C(absent) and C(removed) will remove the specified package.
         default: present
-        choices: [ absent, latest, present, installed, removed ]
+        choices: [ absent, installed, latest, present, removed ]
         type: str
 
     force:

plugins/modules/source_control/gitlab/gitlab_project.py

@@ -345,22 +345,22 @@ def main():
     gitlab_project = GitLabProject(module, gitlab_instance)
 
     namespace = None
-    user_group_id = None
+    namespace_id = None
     if group_identifier:
         group = findGroup(gitlab_instance, group_identifier)
         if group is None:
             module.fail_json(msg="Failed to create project: group %s doesn't exists" % group_identifier)
 
-        user_group_id = group.id
+        namespace_id = group.id
     else:
-        user = gitlab_instance.users.list(username=gitlab_instance.user.username)[0]
-        user_group_id = user.id
+        namespace = gitlab_instance.namespaces.list(search=gitlab_instance.user.username)[0]
+        namespace_id = namespace.id
 
-    if not user_group_id:
-        module.fail_json(msg="Failed to find the user/group id which required to find namespace")
+    if not namespace_id:
+        module.fail_json(msg="Failed to find the namespace or group ID which is required to look up the namespace")
 
     try:
-        namespace = gitlab_instance.namespaces.get(user_group_id)
+        namespace = gitlab_instance.namespaces.get(namespace_id)
     except gitlab.exceptions.GitlabGetError as e:
         module.fail_json(msg="Failed to find the namespace for the given user: %s" % to_native(e))
 

plugins/modules/source_control/gitlab/gitlab_runner.py

@@ -77,7 +77,9 @@ options:
     type: bool
   access_level:
     description:
-      - Determines if a runner can pick up jobs from protected branches.
+      - Determines if a runner can pick up jobs only from protected branches.
+      - If set to C(ref_protected), runner can pick up jobs only from protected branches.
+      - If set to C(not_protected), runner can pick up jobs from both protected and unprotected branches.
     required: False
     default: ref_protected
     choices: ["ref_protected", "not_protected"]

plugins/modules/system/modprobe.py

@@ -50,11 +50,90 @@ EXAMPLES = '''
 '''
 
 import os.path
+import platform
 import shlex
 import traceback
 
 from ansible.module_utils.basic import AnsibleModule
-from ansible.module_utils._text import to_native
+from ansible.module_utils.common.text.converters import to_native
+
+RELEASE_VER = platform.release()
+
+
+class Modprobe(object):
+    def __init__(self, module):
+        self.module = module
+        self.modprobe_bin = module.get_bin_path('modprobe', True)
+
+        self.check_mode = module.check_mode
+        self.desired_state = module.params['state']
+        self.name = module.params['name']
+        self.params = module.params['params']
+
+        self.changed = False
+
+    def load_module(self):
+        command = [self.modprobe_bin]
+        if self.check_mode:
+            command.append('-n')
+        command.extend([self.name] + shlex.split(self.params))
+
+        rc, out, err = self.module.run_command(command)
+
+        if rc != 0:
+            return self.module.fail_json(msg=err, rc=rc, stdout=out, stderr=err, **self.result)
+
+        if self.check_mode or self.module_loaded():
+            self.changed = True
+        else:
+            rc, stdout, stderr = self.module.run_command(
+                [self.modprobe_bin, '-n', '--first-time', self.name] + shlex.split(self.params)
+            )
+            if rc != 0:
+                self.module.warn(stderr)
+
+    def module_loaded(self):
+        is_loaded = False
+        try:
+            with open('/proc/modules') as modules:
+                module_name = self.name.replace('-', '_') + ' '
+                for line in modules:
+                    if line.startswith(module_name):
+                        is_loaded = True
+                        break
+
+            if not is_loaded:
+                module_file = '/' + self.name + '.ko'
+                builtin_path = os.path.join('/lib/modules/', RELEASE_VER, 'modules.builtin')
+                with open(builtin_path) as builtins:
+                    for line in builtins:
+                        if line.rstrip().endswith(module_file):
+                            is_loaded = True
+                            break
+        except (IOError, OSError) as e:
+            self.module.fail_json(msg=to_native(e), exception=traceback.format_exc(), **self.result)
+
+        return is_loaded
+
+    def unload_module(self):
+        command = [self.modprobe_bin, '-r', self.name]
+        if self.check_mode:
+            command.append('-n')
+
+        rc, out, err = self.module.run_command(command)
+        if rc != 0:
+            return self.module.fail_json(msg=err, rc=rc, stdout=out, stderr=err, **self.result)
+
+        self.changed = True
+
+    @property
+    def result(self):
+        return {
+            'changed': self.changed,
+            'name': self.name,
+            'params': self.params,
+            'state': self.desired_state,
+        }
 
 
 def main():
@@ -67,60 +146,14 @@ def main():
         supports_check_mode=True,
     )
 
-    name = module.params['name']
-    params = module.params['params']
-    state = module.params['state']
+    modprobe = Modprobe(module)
 
-    # FIXME: Adding all parameters as result values is useless
-    result = dict(
-        changed=False,
-        name=name,
-        params=params,
-        state=state,
-    )
+    if modprobe.desired_state == 'present' and not modprobe.module_loaded():
+        modprobe.load_module()
+    elif modprobe.desired_state == 'absent' and modprobe.module_loaded():
+        modprobe.unload_module()
 
-    # Check if module is present
-    try:
-        present = False
-        with open('/proc/modules') as modules:
-            module_name = name.replace('-', '_') + ' '
-            for line in modules:
-                if line.startswith(module_name):
-                    present = True
-                    break
-        if not present:
-            command = [module.get_bin_path('uname', True), '-r']
-            rc, uname_kernel_release, err = module.run_command(command)
-            module_file = '/' + name + '.ko'
-            builtin_path = os.path.join('/lib/modules/', uname_kernel_release.strip(),
-                                        'modules.builtin')
-            with open(builtin_path) as builtins:
-                for line in builtins:
-                    if line.endswith(module_file):
-                        present = True
-                        break
-    except IOError as e:
-        module.fail_json(msg=to_native(e), exception=traceback.format_exc(), **result)
-
-    # Add/remove module as needed
-    if state == 'present':
-        if not present:
-            if not module.check_mode:
-                command = [module.get_bin_path('modprobe', True), name]
-                command.extend(shlex.split(params))
-                rc, out, err = module.run_command(command)
-                if rc != 0:
-                    module.fail_json(msg=err, rc=rc, stdout=out, stderr=err, **result)
-            result['changed'] = True
-    elif state == 'absent':
-        if present:
-            if not module.check_mode:
-                rc, out, err = module.run_command([module.get_bin_path('modprobe', True), '-r', name])
-                if rc != 0:
-                    module.fail_json(msg=err, rc=rc, stdout=out, stderr=err, **result)
-            result['changed'] = True
-
-    module.exit_json(**result)
+    module.exit_json(**modprobe.result)
 
 
 if __name__ == '__main__':

plugins/modules/system/pamd.py

@@ -16,7 +16,9 @@ short_description: Manage PAM Modules
 description:
   - Edit PAM service's type, control, module path and module arguments.
   - In order for a PAM rule to be modified, the type, control and
-    module_path must match an existing rule.  See man(5) pam.d for details.
+    module_path must match an existing rule. See man(5) pam.d for details.
+notes:
+  - This module does not handle authselect profiles.
 options:
   name:
     description:

tests/sanity/ignore-2.10.txt

@@ -1,8 +1,6 @@
 plugins/module_utils/cloud.py pylint:bad-option-value  # a pylint test that is disabled was modified over time
 plugins/module_utils/compat/ipaddress.py no-assert
 plugins/module_utils/compat/ipaddress.py no-unicode-literals
-plugins/module_utils/_mount.py future-import-boilerplate
-plugins/module_utils/_mount.py metaclass-boilerplate
 plugins/modules/cloud/linode/linode.py validate-modules:parameter-list-no-elements
 plugins/modules/cloud/linode/linode.py validate-modules:parameter-type-not-in-doc
 plugins/modules/cloud/linode/linode.py validate-modules:undocumented-parameter
@@ -158,5 +156,3 @@ plugins/modules/system/xfconf.py validate-modules:return-syntax-error
 plugins/modules/web_infrastructure/jenkins_plugin.py use-argspec-type-path
 tests/integration/targets/django_manage/files/base_test/simple_project/p1/manage.py compile-2.6   # django generated code
 tests/integration/targets/django_manage/files/base_test/simple_project/p1/manage.py compile-2.7   # django generated code
-tests/utils/shippable/check_matrix.py replace-urlopen
-tests/utils/shippable/timing.py shebang

tests/sanity/ignore-2.11.txt

@@ -1,7 +1,5 @@
 plugins/module_utils/compat/ipaddress.py no-assert
 plugins/module_utils/compat/ipaddress.py no-unicode-literals
-plugins/module_utils/_mount.py future-import-boilerplate
-plugins/module_utils/_mount.py metaclass-boilerplate
 plugins/modules/cloud/linode/linode.py validate-modules:parameter-list-no-elements
 plugins/modules/cloud/linode/linode.py validate-modules:parameter-type-not-in-doc
 plugins/modules/cloud/linode/linode.py validate-modules:undocumented-parameter
@@ -157,5 +155,3 @@ plugins/modules/system/xfconf.py validate-modules:return-syntax-error
 plugins/modules/web_infrastructure/jenkins_plugin.py use-argspec-type-path
 tests/integration/targets/django_manage/files/base_test/simple_project/p1/manage.py compile-2.6   # django generated code
 tests/integration/targets/django_manage/files/base_test/simple_project/p1/manage.py compile-2.7   # django generated code
-tests/utils/shippable/check_matrix.py replace-urlopen
-tests/utils/shippable/timing.py shebang

tests/sanity/ignore-2.12.txt

@@ -1,7 +1,5 @@
 plugins/module_utils/compat/ipaddress.py no-assert
 plugins/module_utils/compat/ipaddress.py no-unicode-literals
-plugins/module_utils/_mount.py future-import-boilerplate
-plugins/module_utils/_mount.py metaclass-boilerplate
 plugins/modules/cloud/linode/linode.py validate-modules:parameter-list-no-elements
 plugins/modules/cloud/linode/linode.py validate-modules:parameter-type-not-in-doc
 plugins/modules/cloud/linode/linode.py validate-modules:undocumented-parameter
@@ -155,5 +153,3 @@ plugins/modules/system/ssh_config.py use-argspec-type-path # Required since modu
 plugins/modules/system/xfconf.py validate-modules:parameter-state-invalid-choice
 plugins/modules/system/xfconf.py validate-modules:return-syntax-error
 plugins/modules/web_infrastructure/jenkins_plugin.py use-argspec-type-path
-tests/utils/shippable/check_matrix.py replace-urlopen
-tests/utils/shippable/timing.py shebang

tests/sanity/ignore-2.9.txt

@@ -1,8 +1,6 @@
 plugins/module_utils/cloud.py pylint:bad-option-value  # a pylint test that is disabled was modified over time
 plugins/module_utils/compat/ipaddress.py no-assert
 plugins/module_utils/compat/ipaddress.py no-unicode-literals
-plugins/module_utils/_mount.py future-import-boilerplate
-plugins/module_utils/_mount.py metaclass-boilerplate
 plugins/modules/cloud/linode/linode.py validate-modules:parameter-type-not-in-doc
 plugins/modules/cloud/linode/linode.py validate-modules:undocumented-parameter
 plugins/modules/cloud/lxc/lxc_container.py use-argspec-type-path
@@ -193,5 +191,3 @@ plugins/modules/web_infrastructure/nginx_status_facts.py validate-modules:deprec
 plugins/modules/web_infrastructure/nginx_status_facts.py validate-modules:invalid-documentation
 tests/integration/targets/django_manage/files/base_test/simple_project/p1/manage.py compile-2.6   # django generated code
 tests/integration/targets/django_manage/files/base_test/simple_project/p1/manage.py compile-2.7   # django generated code
-tests/utils/shippable/check_matrix.py replace-urlopen
-tests/utils/shippable/timing.py shebang

tests/unit/plugins/modules/cloud/misc/test_proxmox_kvm.py

@@ -0,0 +1,17 @@
+# Copyright: (c) 2021, Ansible Project
+# GNU General Public License v3.0+ (see COPYING or https://www.gnu.org/licenses/gpl-3.0.txt)
+
+from __future__ import (absolute_import, division, print_function)
+__metaclass__ = type
+
+from ansible_collections.community.general.plugins.modules.cloud.misc.proxmox_kvm import parse_dev, parse_mac
+
+
+def test_parse_mac():
+    assert parse_mac('virtio=00:11:22:AA:BB:CC,bridge=vmbr0,firewall=1') == '00:11:22:AA:BB:CC'
+
+
+def test_parse_dev():
+    assert parse_dev('local-lvm:vm-1000-disk-0,format=qcow2') == 'local-lvm:vm-1000-disk-0'
+    assert parse_dev('local-lvm:vm-101-disk-1,size=8G') == 'local-lvm:vm-101-disk-1'
+    assert parse_dev('local-zfs:vm-1001-disk-0') == 'local-zfs:vm-1001-disk-0'

tests/unit/plugins/modules/net_tools/test_nmcli.py

@@ -232,6 +232,50 @@ bridge-port.hairpin-mode:               yes
 bridge-port.priority:                   32
 """
 
+TESTCASE_TEAM = [
+    {
+        'type': 'team',
+        'conn_name': 'non_existent_nw_device',
+        'ifname': 'team0_non_existant',
+        'state': 'present',
+        '_ansible_check_mode': False,
+    }
+]
+
+TESTCASE_TEAM_SHOW_OUTPUT = """\
+connection.id:                          non_existent_nw_device
+connection.interface-name:              team0_non_existant
+connection.autoconnect:                 yes
+connection.type:                        team
+ipv4.ignore-auto-dns:                   no
+ipv4.ignore-auto-routes:                no
+ipv4.never-default:                     no
+ipv4.may-fail:                          yes
+ipv6.method:                            auto
+ipv6.ignore-auto-dns:                   no
+ipv6.ignore-auto-routes:                no
+"""
+
+TESTCASE_TEAM_SLAVE = [
+    {
+        'type': 'team-slave',
+        'conn_name': 'non_existent_nw_slaved_device',
+        'ifname': 'generic_slaved_non_existant',
+        'master': 'team0_non_existant',
+        'state': 'present',
+        '_ansible_check_mode': False,
+    }
+]
+
+TESTCASE_TEAM_SLAVE_SHOW_OUTPUT = """\
+connection.id:                          non_existent_nw_slaved_device
+connection.interface-name:              generic_slaved_non_existant
+connection.autoconnect:                 yes
+connection.master:                      team0_non_existant
+connection.slave-type:                  team
+802-3-ethernet.mtu:                     auto
+"""
+
 TESTCASE_VLAN = [
     {
         'type': 'vlan',
@@ -455,6 +499,20 @@ def mocked_bridge_slave_unchanged(mocker):
                execute_return=(0, TESTCASE_BRIDGE_SLAVE_SHOW_OUTPUT, ""))
 
 
+@pytest.fixture
+def mocked_team_connection_unchanged(mocker):
+    mocker_set(mocker,
+               connection_exists=True,
+               execute_return=(0, TESTCASE_TEAM_SHOW_OUTPUT, ""))
+
+
+@pytest.fixture
+def mocked_team_slave_connection_unchanged(mocker):
+    mocker_set(mocker,
+               connection_exists=True,
+               execute_return=(0, TESTCASE_TEAM_SLAVE_SHOW_OUTPUT, ""))
+
+
 @pytest.fixture
 def mocked_vlan_connection_unchanged(mocker):
     mocker_set(mocker,
@@ -912,6 +970,93 @@ def test_bridge_slave_unchanged(mocked_bridge_slave_unchanged, capfd):
     assert not results['changed']
 
 
+@pytest.mark.parametrize('patch_ansible_module', TESTCASE_TEAM, indirect=['patch_ansible_module'])
+def test_team_connection_create(mocked_generic_connection_create, capfd):
+    """
+    Test : Team connection created
+    """
+    with pytest.raises(SystemExit):
+        nmcli.main()
+
+    assert nmcli.Nmcli.execute_command.call_count == 1
+    arg_list = nmcli.Nmcli.execute_command.call_args_list
+    args, kwargs = arg_list[0]
+
+    assert args[0][0] == '/usr/bin/nmcli'
+    assert args[0][1] == 'con'
+    assert args[0][2] == 'add'
+    assert args[0][3] == 'type'
+    assert args[0][4] == 'team'
+    assert args[0][5] == 'con-name'
+    assert args[0][6] == 'non_existent_nw_device'
+
+    for param in ['connection.autoconnect', 'connection.interface-name', 'team0_non_existant']:
+        assert param in args[0]
+
+    out, err = capfd.readouterr()
+    results = json.loads(out)
+    assert not results.get('failed')
+    assert results['changed']
+
+
+@pytest.mark.parametrize('patch_ansible_module', TESTCASE_TEAM, indirect=['patch_ansible_module'])
+def test_team_connection_unchanged(mocked_team_connection_unchanged, capfd):
+    """
+    Test : Team connection unchanged
+    """
+    with pytest.raises(SystemExit):
+        nmcli.main()
+
+    out, err = capfd.readouterr()
+    results = json.loads(out)
+    assert not results.get('failed')
+    assert not results['changed']
+
+
+@pytest.mark.parametrize('patch_ansible_module', TESTCASE_TEAM_SLAVE, indirect=['patch_ansible_module'])
+def test_create_team_slave(mocked_generic_connection_create, capfd):
+    """
+    Test if Team_slave created
+    """
+
+    with pytest.raises(SystemExit):
+        nmcli.main()
+
+    assert nmcli.Nmcli.execute_command.call_count == 1
+    arg_list = nmcli.Nmcli.execute_command.call_args_list
+    args, kwargs = arg_list[0]
+
+    assert args[0][0] == '/usr/bin/nmcli'
+    assert args[0][1] == 'con'
+    assert args[0][2] == 'add'
+    assert args[0][3] == 'type'
+    assert args[0][4] == 'team-slave'
+    assert args[0][5] == 'con-name'
+    assert args[0][6] == 'non_existent_nw_slaved_device'
+
+    for param in ['connection.autoconnect', 'connection.interface-name', 'connection.master', 'team0_non_existant', 'connection.slave-type']:
+        assert param in args[0]
+
+    out, err = capfd.readouterr()
+    results = json.loads(out)
+    assert not results.get('failed')
+    assert results['changed']
+
+
+@pytest.mark.parametrize('patch_ansible_module', TESTCASE_TEAM_SLAVE, indirect=['patch_ansible_module'])
+def test_team_slave_connection_unchanged(mocked_team_slave_connection_unchanged, capfd):
+    """
+    Test : Team slave connection unchanged
+    """
+    with pytest.raises(SystemExit):
+        nmcli.main()
+
+    out, err = capfd.readouterr()
+    results = json.loads(out)
+    assert not results.get('failed')
+    assert not results['changed']
+
+
 @pytest.mark.parametrize('patch_ansible_module', TESTCASE_VLAN, indirect=['patch_ansible_module'])
 def test_create_vlan_con(mocked_generic_connection_create, capfd):
     """

tests/unit/plugins/modules/packaging/language/test_npm.py

@@ -47,6 +47,66 @@ class NPMModuleTestCase(ModuleTestCase):
         result = self.module_main(AnsibleExitJson)
 
         self.assertTrue(result['changed'])
+        self.module_main_command.assert_has_calls([
+            call(['/testbin/npm', 'list', '--json', '--long', '--global'], check_rc=False, cwd=None),
+            call(['/testbin/npm', 'install', '--global', 'coffee-script'], check_rc=True, cwd=None),
+        ])
+
+    def test_present_version(self):
+        set_module_args({
+            'name': 'coffee-script',
+            'global': 'true',
+            'state': 'present',
+            'version': '2.5.1'
+        })
+        self.module_main_command.side_effect = [
+            (0, '{}', ''),
+            (0, '{}', ''),
+        ]
+
+        result = self.module_main(AnsibleExitJson)
+
+        self.assertTrue(result['changed'])
+        self.module_main_command.assert_has_calls([
+            call(['/testbin/npm', 'list', '--json', '--long', '--global'], check_rc=False, cwd=None),
+            call(['/testbin/npm', 'install', '--global', 'coffee-script@2.5.1'], check_rc=True, cwd=None),
+        ])
+
+    def test_present_version_update(self):
+        set_module_args({
+            'name': 'coffee-script',
+            'global': 'true',
+            'state': 'present',
+            'version': '2.5.1'
+        })
+        self.module_main_command.side_effect = [
+            (0, '{"dependencies": {"coffee-script": {"version" : "2.5.0"}}}', ''),
+            (0, '{}', ''),
+        ]
+
+        result = self.module_main(AnsibleExitJson)
+
+        self.assertTrue(result['changed'])
+        self.module_main_command.assert_has_calls([
+            call(['/testbin/npm', 'list', '--json', '--long', '--global'], check_rc=False, cwd=None),
+            call(['/testbin/npm', 'install', '--global', 'coffee-script@2.5.1'], check_rc=True, cwd=None),
+        ])
+
+    def test_present_version_exists(self):
+        set_module_args({
+            'name': 'coffee-script',
+            'global': 'true',
+            'state': 'present',
+            'version': '2.5.1'
+        })
+        self.module_main_command.side_effect = [
+            (0, '{"dependencies": {"coffee-script": {"version" : "2.5.1"}}}', ''),
+            (0, '{}', ''),
+        ]
+
+        result = self.module_main(AnsibleExitJson)
+
+        self.assertFalse(result['changed'])
         self.module_main_command.assert_has_calls([
             call(['/testbin/npm', 'list', '--json', '--long', '--global'], check_rc=False, cwd=None),
         ])
@@ -58,7 +118,7 @@ class NPMModuleTestCase(ModuleTestCase):
             'state': 'absent'
         })
         self.module_main_command.side_effect = [
-            (0, '{"dependencies": {"coffee-script": {}}}', ''),
+            (0, '{"dependencies": {"coffee-script": {"version" : "2.5.1"}}}', ''),
             (0, '{}', ''),
         ]
 
@@ -66,5 +126,46 @@ class NPMModuleTestCase(ModuleTestCase):
 
         self.assertTrue(result['changed'])
         self.module_main_command.assert_has_calls([
+            call(['/testbin/npm', 'list', '--json', '--long', '--global'], check_rc=False, cwd=None),
+            call(['/testbin/npm', 'uninstall', '--global', 'coffee-script'], check_rc=True, cwd=None),
+        ])
+
+    def test_absent_version(self):
+        set_module_args({
+            'name': 'coffee-script',
+            'global': 'true',
+            'state': 'absent',
+            'version': '2.5.1'
+        })
+        self.module_main_command.side_effect = [
+            (0, '{"dependencies": {"coffee-script": {"version" : "2.5.1"}}}', ''),
+            (0, '{}', ''),
+        ]
+
+        result = self.module_main(AnsibleExitJson)
+
+        self.assertTrue(result['changed'])
+        self.module_main_command.assert_has_calls([
+            call(['/testbin/npm', 'list', '--json', '--long', '--global'], check_rc=False, cwd=None),
+            call(['/testbin/npm', 'uninstall', '--global', 'coffee-script'], check_rc=True, cwd=None),
+        ])
+
+    def test_absent_version_different(self):
+        set_module_args({
+            'name': 'coffee-script',
+            'global': 'true',
+            'state': 'absent',
+            'version': '2.5.1'
+        })
+        self.module_main_command.side_effect = [
+            (0, '{"dependencies": {"coffee-script": {"version" : "2.5.0"}}}', ''),
+            (0, '{}', ''),
+        ]
+
+        result = self.module_main(AnsibleExitJson)
+
+        self.assertTrue(result['changed'])
+        self.module_main_command.assert_has_calls([
+            call(['/testbin/npm', 'list', '--json', '--long', '--global'], check_rc=False, cwd=None),
             call(['/testbin/npm', 'uninstall', '--global', 'coffee-script'], check_rc=True, cwd=None),
         ])

tests/unit/plugins/modules/system/test_modprobe.py

@@ -0,0 +1,174 @@
+# -*- coding: utf-8 -*-
+# GNU General Public License v3.0+ (see COPYING or https://www.gnu.org/licenses/gpl-3.0.txt)
+
+from __future__ import (absolute_import, division, print_function)
+__metaclass__ = type
+
+from ansible_collections.community.general.tests.unit.plugins.modules.utils import ModuleTestCase, set_module_args
+from ansible_collections.community.general.tests.unit.compat.mock import patch
+from ansible_collections.community.general.tests.unit.compat.mock import Mock
+from ansible.module_utils.basic import AnsibleModule
+from ansible_collections.community.general.plugins.modules.system.modprobe import Modprobe
+
+
+class TestLoadModule(ModuleTestCase):
+    def setUp(self):
+        super(TestLoadModule, self).setUp()
+
+        self.mock_module_loaded = patch(
+            'ansible_collections.community.general.plugins.modules.system.modprobe.Modprobe.module_loaded'
+        )
+        self.mock_run_command = patch('ansible.module_utils.basic.AnsibleModule.run_command')
+        self.mock_get_bin_path = patch('ansible.module_utils.basic.AnsibleModule.get_bin_path')
+
+        self.module_loaded = self.mock_module_loaded.start()
+        self.run_command = self.mock_run_command.start()
+        self.get_bin_path = self.mock_get_bin_path.start()
+
+    def tearDown(self):
+        """Teardown."""
+        super(TestLoadModule, self).tearDown()
+        self.mock_module_loaded.stop()
+        self.mock_run_command.stop()
+        self.mock_get_bin_path.stop()
+
+    def test_load_module_success(self):
+        set_module_args(dict(
+            name='test',
+            state='present',
+        ))
+
+        module = AnsibleModule(
+            argument_spec=dict(
+                name=dict(type='str', required=True),
+                state=dict(type='str', default='present', choices=['absent', 'present']),
+                params=dict(type='str', default=''),
+            ),
+            supports_check_mode=True,
+        )
+
+        self.get_bin_path.side_effect = ['modprobe']
+        self.module_loaded.side_effect = [True]
+        self.run_command.side_effect = [(0, '', '')]
+
+        modprobe = Modprobe(module)
+        modprobe.load_module()
+
+        assert modprobe.result == {
+            'changed': True,
+            'name': 'test',
+            'params': '',
+            'state': 'present',
+        }
+
+    def test_load_module_unchanged(self):
+        set_module_args(dict(
+            name='test',
+            state='present',
+        ))
+
+        module = AnsibleModule(
+            argument_spec=dict(
+                name=dict(type='str', required=True),
+                state=dict(type='str', default='present', choices=['absent', 'present']),
+                params=dict(type='str', default=''),
+            ),
+            supports_check_mode=True,
+        )
+
+        module.warn = Mock()
+
+        self.get_bin_path.side_effect = ['modprobe']
+        self.module_loaded.side_effect = [False]
+        self.run_command.side_effect = [(0, '', ''), (1, '', '')]
+
+        modprobe = Modprobe(module)
+        modprobe.load_module()
+
+        module.warn.assert_called_once_with('')
+
+
+class TestUnloadModule(ModuleTestCase):
+    def setUp(self):
+        super(TestUnloadModule, self).setUp()
+
+        self.mock_module_loaded = patch(
+            'ansible_collections.community.general.plugins.modules.system.modprobe.Modprobe.module_loaded'
+        )
+        self.mock_run_command = patch('ansible.module_utils.basic.AnsibleModule.run_command')
+        self.mock_get_bin_path = patch('ansible.module_utils.basic.AnsibleModule.get_bin_path')
+
+        self.module_loaded = self.mock_module_loaded.start()
+        self.run_command = self.mock_run_command.start()
+        self.get_bin_path = self.mock_get_bin_path.start()
+
+    def tearDown(self):
+        """Teardown."""
+        super(TestUnloadModule, self).tearDown()
+        self.mock_module_loaded.stop()
+        self.mock_run_command.stop()
+        self.mock_get_bin_path.stop()
+
+    def test_unload_module_success(self):
+        set_module_args(dict(
+            name='test',
+            state='absent',
+        ))
+
+        module = AnsibleModule(
+            argument_spec=dict(
+                name=dict(type='str', required=True),
+                state=dict(type='str', default='present', choices=['absent', 'present']),
+                params=dict(type='str', default=''),
+            ),
+            supports_check_mode=True,
+        )
+
+        self.get_bin_path.side_effect = ['modprobe']
+        self.module_loaded.side_effect = [False]
+        self.run_command.side_effect = [(0, '', '')]
+
+        modprobe = Modprobe(module)
+        modprobe.unload_module()
+
+        assert modprobe.result == {
+            'changed': True,
+            'name': 'test',
+            'params': '',
+            'state': 'absent',
+        }
+
+    def test_unload_module_failure(self):
+        set_module_args(dict(
+            name='test',
+            state='absent',
+        ))
+
+        module = AnsibleModule(
+            argument_spec=dict(
+                name=dict(type='str', required=True),
+                state=dict(type='str', default='present', choices=['absent', 'present']),
+                params=dict(type='str', default=''),
+            ),
+            supports_check_mode=True,
+        )
+
+        module.fail_json = Mock()
+
+        self.get_bin_path.side_effect = ['modprobe']
+        self.module_loaded.side_effect = [True]
+        self.run_command.side_effect = [(1, '', '')]
+
+        modprobe = Modprobe(module)
+        modprobe.unload_module()
+
+        dummy_result = {
+            'changed': False,
+            'name': 'test',
+            'state': 'absent',
+            'params': '',
+        }
+
+        module.fail_json.assert_called_once_with(
+            msg='', rc=1, stdout='', stderr='', **dummy_result
+        )

tests/utils/shippable/check_matrix.py

@@ -1,120 +0,0 @@
-#!/usr/bin/env python
-"""Verify the currently executing Shippable test matrix matches the one defined in the "shippable.yml" file."""
-from __future__ import (absolute_import, division, print_function)
-__metaclass__ = type
-
-import datetime
-import json
-import os
-import re
-import sys
-import time
-
-try:
-    from typing import NoReturn
-except ImportError:
-    NoReturn = None
-
-try:
-    # noinspection PyCompatibility
-    from urllib2 import urlopen  # pylint: disable=ansible-bad-import-from
-except ImportError:
-    # noinspection PyCompatibility
-    from urllib.request import urlopen
-
-
-def main():  # type: () -> None
-    """Main entry point."""
-    repo_full_name = os.environ['REPO_FULL_NAME']
-    required_repo_full_name = 'ansible-collections/community.general'
-
-    if repo_full_name != required_repo_full_name:
-        sys.stderr.write('Skipping matrix check on repo "%s" which is not "%s".\n' % (repo_full_name, required_repo_full_name))
-        return
-
-    with open('shippable.yml', 'rb') as yaml_file:
-        yaml = yaml_file.read().decode('utf-8').splitlines()
-
-    defined_matrix = [match.group(1) for match in [re.search(r'^ *- env: T=(.*)$', line) for line in yaml] if match and match.group(1) != 'none']
-
-    if not defined_matrix:
-        fail('No matrix entries found in the "shippable.yml" file.',
-             'Did you modify the "shippable.yml" file?')
-
-    run_id = os.environ['SHIPPABLE_BUILD_ID']
-    sleep = 1
-    jobs = []
-
-    for attempts_remaining in range(4, -1, -1):
-        try:
-            jobs = json.loads(urlopen('https://api.shippable.com/jobs?runIds=%s' % run_id).read())
-
-            if not isinstance(jobs, list):
-                raise Exception('Shippable run %s data is not a list.' % run_id)
-
-            break
-        except Exception as ex:
-            if not attempts_remaining:
-                fail('Unable to retrieve Shippable run %s matrix.' % run_id,
-                     str(ex))
-
-            sys.stderr.write('Unable to retrieve Shippable run %s matrix: %s\n' % (run_id, ex))
-            sys.stderr.write('Trying again in %d seconds...\n' % sleep)
-            time.sleep(sleep)
-            sleep *= 2
-
-    if len(jobs) != len(defined_matrix):
-        if len(jobs) == 1:
-            hint = '\n\nMake sure you do not use the "Rebuild with SSH" option.'
-        else:
-            hint = ''
-
-        fail('Shippable run %s has %d jobs instead of the expected %d jobs.' % (run_id, len(jobs), len(defined_matrix)),
-             'Try re-running the entire matrix.%s' % hint)
-
-    actual_matrix = dict((job.get('jobNumber'), dict(tuple(line.split('=', 1)) for line in job.get('env', [])).get('T', '')) for job in jobs)
-    errors = [(job_number, test, actual_matrix.get(job_number)) for job_number, test in enumerate(defined_matrix, 1) if actual_matrix.get(job_number) != test]
-
-    if len(errors):
-        error_summary = '\n'.join('Job %s expected "%s" but found "%s" instead.' % (job_number, expected, actual) for job_number, expected, actual in errors)
-
-        fail('Shippable run %s has a job matrix mismatch.' % run_id,
-             'Try re-running the entire matrix.\n\n%s' % error_summary)
-
-
-def fail(message, output):  # type: (str, str) -> NoReturn
-    # Include a leading newline to improve readability on Shippable "Tests" tab.
-    # Without this, the first line becomes indented.
-    output = '\n' + output.strip()
-
-    timestamp = datetime.datetime.utcnow().replace(microsecond=0).isoformat()
-
-    # hack to avoid requiring junit-xml, which isn't pre-installed on Shippable outside our test containers
-    xml = '''
-<?xml version="1.0" encoding="utf-8"?>
-<testsuites disabled="0" errors="1" failures="0" tests="1" time="0.0">
-\t<testsuite disabled="0" errors="1" failures="0" file="None" log="None" name="ansible-test" skipped="0" tests="1" time="0" timestamp="%s" url="None">
-\t\t<testcase classname="timeout" name="timeout">
-\t\t\t<error message="%s" type="error">%s</error>
-\t\t</testcase>
-\t</testsuite>
-</testsuites>
-''' % (timestamp, message, output)
-
-    path = 'shippable/testresults/check-matrix.xml'
-    dir_path = os.path.dirname(path)
-
-    if not os.path.exists(dir_path):
-        os.makedirs(dir_path)
-
-    with open(path, 'w') as junit_fd:
-        junit_fd.write(xml.lstrip())
-
-    sys.stderr.write(message + '\n')
-    sys.stderr.write(output + '\n')
-
-    sys.exit(1)
-
-
-if __name__ == '__main__':
-    main()

tests/utils/shippable/sanity.sh

@@ -14,7 +14,7 @@ else
 fi
 
 if [ "${group}" == "extra" ]; then
-    ../internal_test_tools/tools/run.py --color
+    ../internal_test_tools/tools/run.py --color --bot --junit
     exit
 fi
 

tests/utils/shippable/shippable.sh

@@ -73,6 +73,10 @@ else
     export ANSIBLE_COLLECTIONS_PATHS="${PWD}/../../../"
 fi
 
+if [ "${test}" == "sanity/extra" ]; then
+    retry pip install junit-xml --disable-pip-version-check
+fi
+
 # START: HACK install dependencies
 if [ "${script}" != "sanity" ] || [ "${test}" == "sanity/extra" ]; then
     # Nothing further should be added to this list.
@@ -93,6 +97,13 @@ fi
 
 # END: HACK
 
+if [ "${script}" != "sanity" ] && [ "${script}" != "units" ]; then
+    # Adds meta/runtime.yml redirects for all modules before running integration tests.
+    # This ensures that ansible-base and ansible-core will use the "real" modules instead of the
+    # symbolic links, which results in coverage to be reported correctly.
+    "${ANSIBLE_COLLECTIONS_PATHS}/ansible_collections/community/internal_test_tools/tools/meta_runtime.py" redirect --target both --flatmap
+fi
+
 export PYTHONIOENCODING='utf-8'
 
 if [ "${JOB_TRIGGERED_BY_NAME:-}" == "nightly-trigger" ]; then

tests/utils/shippable/timing.py

@@ -1,16 +0,0 @@
-#!/usr/bin/env python3.7
-from __future__ import (absolute_import, division, print_function)
-__metaclass__ = type
-
-import sys
-import time
-
-start = time.time()
-
-sys.stdin.reconfigure(errors='surrogateescape')
-sys.stdout.reconfigure(errors='surrogateescape')
-
-for line in sys.stdin:
-    seconds = time.time() - start
-    sys.stdout.write('%02d:%02d %s' % (seconds // 60, seconds % 60, line))
-    sys.stdout.flush()

tests/utils/shippable/timing.sh

@@ -1,5 +0,0 @@
-#!/usr/bin/env bash
-
-set -o pipefail -eu
-
-"$@" 2>&1 | "$(dirname "$0")/timing.py"