Release 6.2.0.

* rax modules: deprecation notice for branch stable-6

* add changelog fragment

* adjust changelog message

* adjust changelog message

.azure-pipelines/azure-pipelines.yml

@@ -29,6 +29,7 @@ schedules:
     always: true
     branches:
       include:
+        - stable-6
         - stable-5
   - cron: 0 11 * * 0
     displayName: Weekly (old stable branches)
@@ -188,6 +189,24 @@ stages:
             - test: 3.5
 
 ## Remote
+  - stage: Remote_devel_extra_vms
+    displayName: Remote devel extra VMs
+    dependsOn: []
+    jobs:
+      - template: templates/matrix.yml
+        parameters:
+          testFormat: devel/{0}
+          targets:
+            - name: Alpine 3.16
+              test: alpine/3.16
+            # - name: Fedora 36
+            #   test: fedora/36
+            # - name: Ubuntu 20.04
+            #   test: ubuntu/20.04
+            - name: Ubuntu 22.04
+              test: ubuntu/22.04
+          groups:
+            - vm
   - stage: Remote_devel
     displayName: Remote devel
     dependsOn: []
@@ -202,8 +221,6 @@ stages:
               test: rhel/7.9
             - name: RHEL 9.0
               test: rhel/9.0
-            - name: FreeBSD 12.3
-              test: freebsd/12.3
             - name: FreeBSD 13.1
               test: freebsd/13.1
           groups:
@@ -220,8 +237,8 @@ stages:
           targets:
             - name: RHEL 9.0
               test: rhel/9.0
-            - name: FreeBSD 13.1
-              test: freebsd/13.1
+            - name: FreeBSD 12.3
+              test: freebsd/12.3
           groups:
             - 1
             - 2
@@ -385,7 +402,7 @@ stages:
             - name: ArchLinux
               test: archlinux/3.10
             - name: CentOS Stream 8
-              test: centos-stream8/3.8
+              test: centos-stream8/3.9
           groups:
             - 1
             - 2
@@ -458,6 +475,7 @@ stages:
       - Units_2_12
       - Units_2_13
       - Units_2_14
+      - Remote_devel_extra_vms
       - Remote_devel
       - Remote_2_11
       - Remote_2_12

.github/workflows/reuse.yml

@@ -8,7 +8,8 @@ name: Verify REUSE
 on:
   push:
     branches: [main]
-  pull_request:
+  pull_request_target:
+    types: [opened, synchronize, reopened]
     branches: [main]
   # Run CI once per day (at 07:30 UTC)
   schedule:
@@ -21,7 +22,9 @@ jobs:
     runs-on: ubuntu-latest
 
     steps:
-      - uses: actions/checkout@v2
+      - uses: actions/checkout@v3
+        with:
+          ref: ${{ github.event.pull_request.head.sha || '' }}
 
       - name: Install dependencies
         run: |

CONTRIBUTING.md

@@ -112,19 +112,9 @@ Creating new modules and plugins requires a bit more work than other Pull Reques
    - Make sure that new plugins and modules have tests (unit tests, integration tests, or both); it is preferable to have some tests
      which run in CI.
 
-4. For modules and action plugins, make sure to create your module/plugin in the correct subdirectory, and add a redirect entry
-   in `meta/runtime.yml`. For example, for the `aerospike_migrations` module located in
-   `plugins/modules/database/aerospike/aerospike_migrations.py`, you need to create the following entry:
-   ```.yaml
-       aerospike_migrations:
-         redirect: community.general.database.aerospike.aerospike_migrations
-   ```
-   Here, the relative path `database/aerospike/` is inserted into the module's FQCN (Fully Qualified Collection Name) after the
-   collection's name and before the module's name. This must not be done for other plugin types but modules and action plugins!
-
-   - Action plugins need to be accompanied by a module, even if the module file only contains documentation
-     (`DOCUMENTATION`, `EXAMPLES` and `RETURN`). The module must have the same name and directory path in `plugins/modules/`
-     than the action plugin has in `plugins/action/`.
+4. Action plugins need to be accompanied by a module, even if the module file only contains documentation
+   (`DOCUMENTATION`, `EXAMPLES` and `RETURN`). The module must have the same name and directory path in `plugins/modules/`
+   than the action plugin has in `plugins/action/`.
 
 5. Make sure to add a BOTMETA entry for your new module/plugin in `.github/BOTMETA.yml`. Search for other plugins/modules in the
    same directory to see how entries could look. You should list all authors either as `maintainers` or under `ignore`. People

README.md

@@ -6,7 +6,7 @@ SPDX-License-Identifier: GPL-3.0-or-later
 
 # Community General Collection
 
-[![Build Status](https://dev.azure.com/ansible/community.general/_apis/build/status/CI?branchName=stable-5)](https://dev.azure.com/ansible/community.general/_build?definitionId=31)
+[![Build Status](https://dev.azure.com/ansible/community.general/_apis/build/status/CI?branchName=stable-6)](https://dev.azure.com/ansible/community.general/_build?definitionId=31)
 [![Codecov](https://img.shields.io/codecov/c/github/ansible-collections/community.general)](https://codecov.io/gh/ansible-collections/community.general)
 
 This repository contains the `community.general` Ansible Collection. The collection is a part of the Ansible package and includes many modules and plugins supported by Ansible community which are not part of more specialized community collections.
@@ -64,10 +64,6 @@ ansible-galaxy collection install community.general:==X.Y.Z
 
 See [Ansible Using collections](https://docs.ansible.com/ansible/latest/user_guide/collections_using.html) for more details.
 
-### FQCNs for modules and actions
-
-⚠️ The collection uses a similar directory structure for modules as the Ansible repository used for Ansible 2.9 and before. This directory structure was never exposed to the user. Due to changes in community.general 5.0.0 (using `meta/runtime.yml` redirects instead of symbolic links) some tooling started exposing the internal module names to end-users. These **internal names**, like `community.general.system.ufw` for the UFW firewall managing module, do work, but should be avoided since they are treated as an implementation detail that can change at any time, even in bugfix releases. Always use the three-component FQCN form, for example `community.general.ufw` for the UFW module. ⚠️
-
 ## Contributing to this collection
 
 The content of this collection is made by good people just like you, a community of individuals collaborating on making the world better through developing automation software.
@@ -76,13 +72,13 @@ We are actively accepting new contributors.
 
 All types of contributions are very welcome.
 
-You don't know how to start? Refer to our [contribution guide](https://github.com/ansible-collections/community.general/blob/main/CONTRIBUTING.md)!
+You don't know how to start? Refer to our [contribution guide](https://github.com/ansible-collections/community.general/blob/stable-6/CONTRIBUTING.md)!
 
-The current maintainers are listed in the [commit-rights.md](https://github.com/ansible-collections/community.general/blob/main/commit-rights.md#people) file. If you have questions or need help, feel free to mention them in the proposals.
+The current maintainers are listed in the [commit-rights.md](https://github.com/ansible-collections/community.general/blob/stable-6/commit-rights.md#people) file. If you have questions or need help, feel free to mention them in the proposals.
 
 You can find more information in the [developer guide for collections](https://docs.ansible.com/ansible/devel/dev_guide/developing_collections.html#contributing-to-collections), and in the [Ansible Community Guide](https://docs.ansible.com/ansible/latest/community/index.html).
 
-Also for some notes specific to this collection see [our CONTRIBUTING documentation](https://github.com/ansible-collections/community.general/blob/main/CONTRIBUTING.md).
+Also for some notes specific to this collection see [our CONTRIBUTING documentation](https://github.com/ansible-collections/community.general/blob/stable-6/CONTRIBUTING.md).
 
 ### Running tests
 
@@ -92,7 +88,7 @@ See [here](https://docs.ansible.com/ansible/devel/dev_guide/developing_collectio
 
 To learn how to maintain / become a maintainer of this collection, refer to:
 
-* [Committer guidelines](https://github.com/ansible-collections/community.general/blob/main/commit-rights.md).
+* [Committer guidelines](https://github.com/ansible-collections/community.general/blob/stable-6/commit-rights.md).
 * [Maintainer guidelines](https://github.com/ansible/community-docs/blob/main/maintaining.rst).
 
 It is necessary for maintainers of this collection to be subscribed to:
@@ -120,7 +116,7 @@ See the [Releasing guidelines](https://github.com/ansible/community-docs/blob/ma
 
 ## Release notes
 
-See the [changelog](https://github.com/ansible-collections/community.general/blob/stable-5/CHANGELOG.rst).
+See the [changelog](https://github.com/ansible-collections/community.general/blob/stable-6/CHANGELOG.rst).
 
 ## Roadmap
 
@@ -139,8 +135,8 @@ See [this issue](https://github.com/ansible-collections/community.general/issues
 
 This collection is primarily licensed and distributed as a whole under the GNU General Public License v3.0 or later.
 
-See [LICENSES/GPL-3.0-or-later.txt](https://github.com/ansible-collections/community.general/blob/main/COPYING) for the full text.
+See [LICENSES/GPL-3.0-or-later.txt](https://github.com/ansible-collections/community.general/blob/stable-6/COPYING) for the full text.
 
-Parts of the collection are licensed under the [BSD 2-Clause license](https://github.com/ansible-collections/community.general/blob/main/LICENSES/BSD-2-Clause.txt), the [MIT license](https://github.com/ansible-collections/community.general/blob/main/LICENSES/MIT.txt), and the [PSF 2.0 license](https://github.com/ansible-collections/community.general/blob/main/LICENSES/PSF-2.0.txt).
+Parts of the collection are licensed under the [BSD 2-Clause license](https://github.com/ansible-collections/community.general/blob/stable-6/LICENSES/BSD-2-Clause.txt), the [MIT license](https://github.com/ansible-collections/community.general/blob/stable-6/LICENSES/MIT.txt), and the [PSF 2.0 license](https://github.com/ansible-collections/community.general/blob/stable-6/LICENSES/PSF-2.0.txt).
 
 All files have a machine readable `SDPX-License-Identifier:` comment denoting its respective license(s) or an equivalent entry in an accompanying `.license` file. Only changelog fragments (which will not be part of a release) are covered by a blanket statement in `.reuse/dep5`. This conforms to the [REUSE specification](https://reuse.software/spec/).

galaxy.yml

@@ -5,7 +5,7 @@
 
 namespace: community
 name: general
-version: 5.8.2
+version: 6.2.0
 readme: README.md
 authors:
   - Ansible (https://github.com/ansible)

plugins/callback/mail.py

@@ -49,8 +49,9 @@ options:
   sender:
     description:
         - Mail sender.
-        - Note that this will be required from community.general 6.0.0 on.
+        - This is required since community.general 6.0.0.
     type: str
+    required: true
     ini:
         - section: callback_mail
           key: sender
@@ -105,10 +106,6 @@ class CallbackModule(CallbackBase):
         super(CallbackModule, self).set_options(task_keys=task_keys, var_options=var_options, direct=direct)
 
         self.sender = self.get_option('sender')
-        if self.sender is None:
-            self._display.deprecated(
-                'The sender for the mail callback has not been specified. This will be an error in the future',
-                version='6.0.0', collection_name='community.general')
         self.to = self.get_option('to')
         self.smtphost = self.get_option('mta')
         self.smtpport = self.get_option('mtaport')

plugins/callback/unixy.py

@@ -142,7 +142,7 @@ class CallbackModule(CallbackModule_default):
             display_color = C.COLOR_CHANGED
             task_result = self._process_result_output(result, msg)
             self._display.display("  " + task_result, display_color)
-        elif self.get('display_ok_hosts'):
+        elif self.get_option('display_ok_hosts'):
             task_result = self._process_result_output(result, msg)
             self._display.display("  " + task_result, display_color)
 

plugins/doc_fragments/attributes.py

@@ -0,0 +1,89 @@
+# -*- coding: utf-8 -*-
+
+# Copyright (c) Ansible Project
+# GNU General Public License v3.0+ (see COPYING or https://www.gnu.org/licenses/gpl-3.0.txt)
+# SPDX-License-Identifier: GPL-3.0-or-later
+
+from __future__ import (absolute_import, division, print_function)
+__metaclass__ = type
+
+
+class ModuleDocFragment(object):
+
+    # Standard documentation fragment
+    DOCUMENTATION = r'''
+options: {}
+attributes:
+    check_mode:
+      description: Can run in C(check_mode) and return changed status prediction without modifying target.
+    diff_mode:
+      description: Will return details on what has changed (or possibly needs changing in C(check_mode)), when in diff mode.
+'''
+
+#    platform:
+#      description: Target OS/families that can be operated against.
+#      support: N/A
+
+    # Should be used together with the standard fragment
+    INFO_MODULE = r'''
+options: {}
+attributes:
+    check_mode:
+      support: full
+      details:
+        - This action does not modify state.
+    diff_mode:
+      support: N/A
+      details:
+        - This action does not modify state.
+'''
+
+    CONN = r'''
+options: {}
+attributes:
+    become:
+      description: Is usable alongside C(become) keywords.
+    connection:
+      description: Uses the target's configured connection information to execute code on it.
+    delegation:
+      description: Can be used in conjunction with C(delegate_to) and related keywords.
+'''
+
+    FACTS = r'''
+options: {}
+attributes:
+    facts:
+      description: Action returns an C(ansible_facts) dictionary that will update existing host facts.
+'''
+
+    # Should be used together with the standard fragment and the FACTS fragment
+    FACTS_MODULE = r'''
+options: {}
+attributes:
+    check_mode:
+      support: full
+      details:
+        - This action does not modify state.
+    diff_mode:
+      support: N/A
+      details:
+        - This action does not modify state.
+    facts:
+      support: full
+'''
+
+    FILES = r'''
+options: {}
+attributes:
+    safe_file_operations:
+      description: Uses Ansible's strict file operation functions to ensure proper permissions and avoid data corruption.
+'''
+
+    FLOW = r'''
+options: {}
+attributes:
+    action:
+      description: Indicates this has a corresponding action plugin so some parts of the options can be executed on the controller.
+    async:
+      description: Supports being used with the C(async) keyword.
+'''

plugins/doc_fragments/bitbucket.py

@@ -27,8 +27,10 @@ options:
     description:
       - The username.
       - If not set the environment variable C(BITBUCKET_USERNAME) will be used.
+      - I(username) is an alias of I(user) since community.genreal 6.0.0. It was an alias of I(workspace) before.
     type: str
     version_added: 4.0.0
+    aliases: [ username ]
   password:
     description:
       - The App password.

plugins/inventory/linode.py

@@ -126,7 +126,6 @@ import os
 from ansible.errors import AnsibleError, AnsibleParserError
 from ansible.module_utils.six import string_types
 from ansible.plugins.inventory import BaseInventoryPlugin, Constructable, Cacheable
-from ansible.template import Templar
 
 
 try:
@@ -145,22 +144,14 @@ class InventoryModule(BaseInventoryPlugin, Constructable, Cacheable):
     def _build_client(self, loader):
         """Build the Linode client."""
 
-        t = Templar(loader=loader)
-
         access_token = self.get_option('access_token')
-        if t.is_template(access_token):
-            access_token = t.template(variable=access_token, disable_lookups=False)
-
-        if access_token is None:
-            try:
-                access_token = os.environ['LINODE_ACCESS_TOKEN']
-            except KeyError:
-                pass
+        if self.templar.is_template(access_token):
+            access_token = self.templar.template(variable=access_token, disable_lookups=False)
 
         if access_token is None:
             raise AnsibleError((
                 'Could not retrieve Linode access token '
-                'from plugin configuration or environment'
+                'from plugin configuration sources'
             ))
 
         self.client = LinodeClient(access_token)

plugins/inventory/lxd.py

@@ -55,6 +55,11 @@ DOCUMENTATION = r'''
             type: str
             default: none
             choices: [ 'STOPPED', 'STARTING', 'RUNNING', 'none' ]
+        project:
+            description: Filter the instance according to the given project.
+            type: str
+            default: default
+            version_added: 6.2.0
         type_filter:
             description:
             - Filter the instances by type C(virtual-machine), C(container) or C(both).
@@ -140,6 +145,9 @@ groupby:
   vlan666:
     type: vlanid
     attribute: 666
+  projectInternals:
+    type: project
+    attribute: internals
 '''
 
 import binascii
@@ -153,6 +161,7 @@ from ansible.module_utils.common.text.converters import to_native, to_text
 from ansible.module_utils.common.dict_transformations import dict_merge
 from ansible.module_utils.six import raise_from
 from ansible.errors import AnsibleError, AnsibleParserError
+from ansible.module_utils.six.moves.urllib.parse import urlencode
 from ansible_collections.community.general.plugins.module_utils.lxd import LXDClient, LXDClientException
 
 try:
@@ -330,7 +339,15 @@ class InventoryModule(BaseInventoryPlugin):
         #        "status_code": 200,
         #        "type": "sync"
         #      }
-        instances = self.socket.do('GET', '/1.0/instances')
+        url = '/1.0/instances'
+        if self.project:
+            url = url + '?{0}'.format(urlencode(dict(project=self.project)))
+
+        instances = self.socket.do('GET', url)
+
+        if self.project:
+            return [m.split('/')[3].split('?')[0] for m in instances['metadata']]
+
         return [m.split('/')[3] for m in instances['metadata']]
 
     def _get_config(self, branch, name):
@@ -351,9 +368,11 @@ class InventoryModule(BaseInventoryPlugin):
             dict(config): Config of the instance"""
         config = {}
         if isinstance(branch, (tuple, list)):
-            config[name] = {branch[1]: self.socket.do('GET', '/1.0/{0}/{1}/{2}'.format(to_native(branch[0]), to_native(name), to_native(branch[1])))}
+            config[name] = {branch[1]: self.socket.do(
+                'GET', '/1.0/{0}/{1}/{2}?{3}'.format(to_native(branch[0]), to_native(name), to_native(branch[1]), urlencode(dict(project=self.project))))}
         else:
-            config[name] = {branch: self.socket.do('GET', '/1.0/{0}/{1}'.format(to_native(branch), to_native(name)))}
+            config[name] = {branch: self.socket.do(
+                'GET', '/1.0/{0}/{1}?{2}'.format(to_native(branch), to_native(name), urlencode(dict(project=self.project))))}
         return config
 
     def get_instance_data(self, names):
@@ -583,6 +602,8 @@ class InventoryModule(BaseInventoryPlugin):
             self._set_data_entry(instance_name, 'network_interfaces', self.extract_network_information_from_instance_config(instance_name))
             self._set_data_entry(instance_name, 'preferred_interface', self.get_prefered_instance_network_interface(instance_name))
             self._set_data_entry(instance_name, 'vlan_ids', self.get_instance_vlans(instance_name))
+            self._set_data_entry(instance_name, 'project', self._get_data_entry(
+                'instances/{0}/instances/metadata/project'.format(instance_name)))
 
     def build_inventory_network(self, instance_name):
         """Add the network interfaces of the instance to the inventory
@@ -686,6 +707,8 @@ class InventoryModule(BaseInventoryPlugin):
             # add VLAN_ID information
             if self._get_data_entry('inventory/{0}/vlan_ids'.format(instance_name)):
                 self.inventory.set_variable(instance_name, 'ansible_lxd_vlan_ids', self._get_data_entry('inventory/{0}/vlan_ids'.format(instance_name)))
+            # add project
+            self.inventory.set_variable(instance_name, 'ansible_lxd_project', self._get_data_entry('inventory/{0}/project'.format(instance_name)))
 
     def build_inventory_groups_location(self, group_name):
         """create group by attribute: location
@@ -761,6 +784,28 @@ class InventoryModule(BaseInventoryPlugin):
                             # Ignore invalid IP addresses returned by lxd
                             pass
 
+    def build_inventory_groups_project(self, group_name):
+        """create group by attribute: project
+
+        Args:
+            str(group_name): Group name
+        Kwargs:
+            None
+        Raises:
+            None
+        Returns:
+            None"""
+        # maybe we just want to expand one group
+        if group_name not in self.inventory.groups:
+            self.inventory.add_group(group_name)
+
+        gen_instances = [
+            instance_name for instance_name in self.inventory.hosts
+            if 'ansible_lxd_project' in self.inventory.get_host(instance_name).get_vars()]
+        for instance_name in gen_instances:
+            if self.groupby[group_name].get('attribute').lower() == self.inventory.get_host(instance_name).get_vars().get('ansible_lxd_project'):
+                self.inventory.add_child(group_name, instance_name)
+
     def build_inventory_groups_os(self, group_name):
         """create group by attribute: os
 
@@ -899,6 +944,7 @@ class InventoryModule(BaseInventoryPlugin):
                 * 'profile'
                 * 'vlanid'
                 * 'type'
+                * 'project'
 
             Args:
                 str(group_name): Group name
@@ -926,6 +972,8 @@ class InventoryModule(BaseInventoryPlugin):
                 self.build_inventory_groups_vlanid(group_name)
             elif self.groupby[group_name].get('type') == 'type':
                 self.build_inventory_groups_type(group_name)
+            elif self.groupby[group_name].get('type') == 'project':
+                self.build_inventory_groups_project(group_name)
             else:
                 raise AnsibleParserError('Unknown group type: {0}'.format(to_native(group_name)))
 
@@ -1032,6 +1080,7 @@ class InventoryModule(BaseInventoryPlugin):
         try:
             self.client_key = self.get_option('client_key')
             self.client_cert = self.get_option('client_cert')
+            self.project = self.get_option('project')
             self.debug = self.DEBUG
             self.data = {}  # store for inventory-data
             self.groupby = self.get_option('groupby')

plugins/inventory/nmap.py

@@ -46,6 +46,25 @@ DOCUMENTATION = '''
             description: use IPv6 type addresses
             type: boolean
             default: true
+        udp_scan:
+            description:
+                - Scan via UDP.
+                - Depending on your system you might need I(sudo=true) for this to work.
+            type: boolean
+            default: false
+            version_added: 6.1.0
+        icmp_timestamp:
+            description:
+                - Scan via ICMP Timestamp (C(-PP)).
+                - Depending on your system you might need I(sudo=true) for this to work.
+            type: boolean
+            default: false
+            version_added: 6.1.0
+        dns_resolve:
+            description: Whether to always (C(true)) or never (C(false)) do DNS resolution.
+            type: boolean
+            default: false
+            version_added: 6.1.0
     notes:
         - At least one of ipv4 or ipv6 is required to be True, both can be True, but they cannot both be False.
         - 'TODO: add OS fingerprinting'
@@ -166,6 +185,15 @@ class InventoryModule(BaseInventoryPlugin, Constructable, Cacheable):
                 cmd.append('--exclude')
                 cmd.append(','.join(self._options['exclude']))
 
+            if self._options['dns_resolve']:
+                cmd.append('-n')
+
+            if self._options['udp_scan']:
+                cmd.append('-sU')
+
+            if self._options['icmp_timestamp']:
+                cmd.append('-PP')
+
             cmd.append(self._options['address'])
             try:
                 # execute

plugins/inventory/proxmox.py

@@ -113,10 +113,9 @@ DOCUMENTATION = '''
         description:
           - Whether to set C(ansbile_host) for proxmox nodes.
           - When set to C(true) (default), will use the first available interface. This can be different from what you expect.
-          - This currently defaults to C(true), but the default is deprecated since community.general 4.8.0.
-            The default will change to C(false) in community.general 6.0.0. To avoid a deprecation warning, please
-            set this parameter explicitly.
+          - The default of this option changed from C(true) to C(false) in community.general 6.0.0.
         type: bool
+        default: false
       filters:
         version_added: 4.6.0
         description: A list of Jinja templates that allow filtering hosts.
@@ -223,7 +222,6 @@ from ansible.module_utils.common.text.converters import to_native
 from ansible.module_utils.six import string_types
 from ansible.module_utils.six.moves.urllib.parse import urlencode
 from ansible.utils.display import Display
-from ansible.template import Templar
 
 from ansible_collections.community.general.plugins.module_utils.version import LooseVersion
 
@@ -410,7 +408,7 @@ class InventoryModule(BaseInventoryPlugin, Constructable, Cacheable):
                     stripped_value = value.strip()
                     if stripped_value:
                         parsed_key = key + "_parsed"
-                        properties[parsed_key] = [tag.strip() for tag in stripped_value.split(",")]
+                        properties[parsed_key] = [tag.strip() for tag in stripped_value.replace(',', ';').split(";")]
 
                 # The first field in the agent string tells you whether the agent is enabled
                 # the rest of the comma separated string is extra config for the agent.
@@ -567,14 +565,6 @@ class InventoryModule(BaseInventoryPlugin, Constructable, Cacheable):
         self.inventory.add_group(nodes_group)
 
         want_proxmox_nodes_ansible_host = self.get_option("want_proxmox_nodes_ansible_host")
-        if want_proxmox_nodes_ansible_host is None:
-            display.deprecated(
-                'The want_proxmox_nodes_ansible_host option of the community.general.proxmox inventory plugin'
-                ' currently defaults to `true`, but this default has been deprecated and will change to `false`'
-                ' in community.general 6.0.0. To keep the current behavior and remove this deprecation warning,'
-                ' explicitly set `want_proxmox_nodes_ansible_host` to `true` in your inventory configuration',
-                version='6.0.0', collection_name='community.general')
-            want_proxmox_nodes_ansible_host = True
 
         # gather vm's on nodes
         self._get_auth()
@@ -621,40 +611,23 @@ class InventoryModule(BaseInventoryPlugin, Constructable, Cacheable):
         # read config from file, this sets 'options'
         self._read_config_data(path)
 
-        t = Templar(loader=loader)
+        # read and template auth options
+        for o in ('url', 'user', 'password', 'token_id', 'token_secret'):
+            v = self.get_option(o)
+            if self.templar.is_template(v):
+                v = self.templar.template(v, disable_lookups=False)
+            setattr(self, 'proxmox_%s' % o, v)
 
-        # read options
-        proxmox_url = self.get_option('url')
-        if t.is_template(proxmox_url):
-            proxmox_url = t.template(variable=proxmox_url, disable_lookups=False)
-        self.proxmox_url = proxmox_url.rstrip('/')
+        # some more cleanup and validation
+        self.proxmox_url = self.proxmox_url.rstrip('/')
 
-        proxmox_user = self.get_option('user')
-        if t.is_template(proxmox_user):
-            proxmox_user = t.template(variable=proxmox_user, disable_lookups=False)
-        self.proxmox_user = proxmox_user
-
-        proxmox_password = self.get_option('password')
-        if t.is_template(proxmox_password):
-            proxmox_password = t.template(variable=proxmox_password, disable_lookups=False)
-        self.proxmox_password = proxmox_password
-
-        proxmox_token_id = self.get_option('token_id')
-        if t.is_template(proxmox_token_id):
-            proxmox_token_id = t.template(variable=proxmox_token_id, disable_lookups=False)
-        self.proxmox_token_id = proxmox_token_id
-
-        proxmox_token_secret = self.get_option('token_secret')
-        if t.is_template(proxmox_token_secret):
-            proxmox_token_secret = t.template(variable=proxmox_token_secret, disable_lookups=False)
-        self.proxmox_token_secret = proxmox_token_secret
-
-        if proxmox_password is None and (proxmox_token_id is None or proxmox_token_secret is None):
+        if self.proxmox_password is None and (self.proxmox_token_id is None or self.proxmox_token_secret is None):
             raise AnsibleError('You must specify either a password or both token_id and token_secret.')
 
         if self.get_option('qemu_extended_statuses') and not self.get_option('want_facts'):
             raise AnsibleError('You must set want_facts to True if you want to use qemu_extended_statuses.')
 
+        # read rest of options
         self.cache_key = self.get_cache_key(path)
         self.use_cache = cache and self.get_option('cache')
         self.host_filters = self.get_option('filters')

plugins/lookup/cartesian.py

@@ -15,9 +15,11 @@ DOCUMENTATION = '''
         - It is clearer with an example, it turns [1, 2, 3], [a, b] into [1, a], [1, b], [2, a], [2, b], [3, a], [3, b].
          You can see the exact syntax in the examples section.
     options:
-      _raw:
+      _terms:
         description:
           - a set of lists
+        type: list
+        elements: list
         required: true
 '''
 
@@ -69,6 +71,7 @@ class LookupModule(LookupBase):
         return results
 
     def run(self, terms, variables=None, **kwargs):
+        self.set_options(var_options=variables, direct=kwargs)
 
         terms = self._lookup_variables(terms)
 

plugins/lookup/credstash.py

@@ -22,25 +22,33 @@ DOCUMENTATION = '''
         required: true
       table:
         description: name of the credstash table to query
+        type: str
         default: 'credential-store'
       version:
         description: Credstash version
+        type: str
+        default: ''
       region:
         description: AWS region
+        type: str
       profile_name:
         description: AWS profile to use for authentication
+        type: str
         env:
           - name: AWS_PROFILE
       aws_access_key_id:
         description: AWS access key ID
+        type: str
         env:
           - name: AWS_ACCESS_KEY_ID
       aws_secret_access_key:
         description: AWS access key
+        type: str
         env:
           - name: AWS_SECRET_ACCESS_KEY
       aws_session_token:
         description: AWS session token
+        type: str
         env:
           - name: AWS_SESSION_TOKEN
 '''
@@ -100,28 +108,39 @@ except ImportError:
 
 
 class LookupModule(LookupBase):
-    def run(self, terms, variables, **kwargs):
-
+    def run(self, terms, variables=None, **kwargs):
         if not CREDSTASH_INSTALLED:
             raise AnsibleError('The credstash lookup plugin requires credstash to be installed.')
 
+        self.set_options(var_options=variables, direct=kwargs)
+
+        version = self.get_option('version')
+        region = self.get_option('region')
+        table = self.get_option('table')
+        profile_name = self.get_option('profile_name')
+        aws_access_key_id = self.get_option('aws_access_key_id')
+        aws_secret_access_key = self.get_option('aws_secret_access_key')
+        aws_session_token = self.get_option('aws_session_token')
+
+        context = dict(
+            (k, v) for k, v in kwargs.items()
+            if k not in ('version', 'region', 'table', 'profile_name', 'aws_access_key_id', 'aws_secret_access_key', 'aws_session_token')
+        )
+
+        kwargs_pass = {
+            'profile_name': profile_name,
+            'aws_access_key_id': aws_access_key_id,
+            'aws_secret_access_key': aws_secret_access_key,
+            'aws_session_token': aws_session_token,
+        }
+
         ret = []
         for term in terms:
             try:
-                version = kwargs.pop('version', '')
-                region = kwargs.pop('region', None)
-                table = kwargs.pop('table', 'credential-store')
-                profile_name = kwargs.pop('profile_name', os.getenv('AWS_PROFILE', None))
-                aws_access_key_id = kwargs.pop('aws_access_key_id', os.getenv('AWS_ACCESS_KEY_ID', None))
-                aws_secret_access_key = kwargs.pop('aws_secret_access_key', os.getenv('AWS_SECRET_ACCESS_KEY', None))
-                aws_session_token = kwargs.pop('aws_session_token', os.getenv('AWS_SESSION_TOKEN', None))
-                kwargs_pass = {'profile_name': profile_name, 'aws_access_key_id': aws_access_key_id,
-                               'aws_secret_access_key': aws_secret_access_key, 'aws_session_token': aws_session_token}
-                val = credstash.getSecret(term, version, region, table, context=kwargs, **kwargs_pass)
+                ret.append(credstash.getSecret(term, version, region, table, context=context, **kwargs_pass))
             except credstash.ItemNotFound:
                 raise AnsibleError('Key {0} not found'.format(term))
             except Exception as e:
                 raise AnsibleError('Encountered exception while fetching {0}: {1}'.format(term, e))
-            ret.append(val)
 
         return ret

plugins/lookup/cyberarkpassword.py

@@ -174,7 +174,6 @@ class LookupModule(LookupBase):
     """
 
     def run(self, terms, variables=None, **kwargs):
-
         display.vvvv("%s" % terms)
         if isinstance(terms, list):
             return_values = []

plugins/lookup/dependent.py

@@ -16,7 +16,7 @@ description:
      or template expressions which evaluate to lists or dicts, composed of the elements of
      the input evaluated lists and dictionaries."
 options:
-  _raw:
+  _terms:
     description:
       - A list where the elements are one-element dictionaries, mapping a name to a string, list, or dictionary.
         The name is the index that is used in the result object. The value is iterated over as described below.
@@ -191,6 +191,8 @@ class LookupModule(LookupBase):
 
     def run(self, terms, variables=None, **kwargs):
         """Generate list."""
+        self.set_options(var_options=variables, direct=kwargs)
+
         result = []
         if len(terms) > 0:
             templar = Templar(loader=self._templar._loader)

plugins/lookup/dig.py

@@ -21,22 +21,26 @@ DOCUMENTATION = '''
       - In addition to (default) A record, it is also possible to specify a different record type that should be queried.
         This can be done by either passing-in additional parameter of format qtype=TYPE to the dig lookup, or by appending /TYPE to the FQDN being queried.
       - If multiple values are associated with the requested record, the results will be returned as a comma-separated list.
-        In such cases you may want to pass option wantlist=True to the plugin, which will result in the record values being returned as a list
-        over which you can iterate later on.
+        In such cases you may want to pass option I(wantlist=true) to the lookup call, or alternatively use C(query) instead of C(lookup),
+        which will result in the record values being returned as a list over which you can iterate later on.
       - By default, the lookup will rely on system-wide configured DNS servers for performing the query.
         It is also possible to explicitly specify DNS servers to query using the @DNS_SERVER_1,DNS_SERVER_2,...,DNS_SERVER_N notation.
         This needs to be passed-in as an additional parameter to the lookup
     options:
       _terms:
         description: Domain(s) to query.
+        type: list
+        elements: str
       qtype:
         description:
             - Record type to query.
-            - C(DLV) is deprecated and will be removed in community.general 6.0.0.
+            - C(DLV) has been removed in community.general 6.0.0.
+        type: str
         default: 'A'
-        choices: [A, ALL, AAAA, CNAME, DNAME, DLV, DNSKEY, DS, HINFO, LOC, MX, NAPTR, NS, NSEC3PARAM, PTR, RP, RRSIG, SOA, SPF, SRV, SSHFP, TLSA, TXT]
+        choices: [A, ALL, AAAA, CNAME, DNAME, DNSKEY, DS, HINFO, LOC, MX, NAPTR, NS, NSEC3PARAM, PTR, RP, RRSIG, SOA, SPF, SRV, SSHFP, TLSA, TXT]
       flat:
         description: If 0 each record is returned as a dictionary, otherwise a string.
+        type: int
         default: 1
       retry_servfail:
         description: Retry a nameserver if it returns SERVFAIL.
@@ -52,6 +56,18 @@ DOCUMENTATION = '''
         default: false
         type: bool
         version_added: 5.4.0
+      real_empty:
+        description:
+          - Return empty result without empty strings, and return empty list instead of C(NXDOMAIN).
+          - The default for this option will likely change to C(true) in the future.
+        default: false
+        type: bool
+        version_added: 6.0.0
+      class:
+        description:
+          - "Class."
+        type: str
+        default: 'IN'
     notes:
       - ALL is not a record per-se, merely the listed fields are available for any record results you retrieve in the form of a dictionary.
       - While the 'dig' lookup plugin supports anything which dnspython supports out of the box, only a subset can be converted into a dictionary.
@@ -67,7 +83,7 @@ EXAMPLES = """
 
 - name: "The TXT record for example.org."
   ansible.builtin.debug:
-    msg: "{{ lookup('community.general.dig', 'example.org.', 'qtype=TXT') }}"
+    msg: "{{ lookup('community.general.dig', 'example.org.', qtype='TXT') }}"
 
 - name: "The TXT record for example.org, alternative syntax."
   ansible.builtin.debug:
@@ -76,24 +92,24 @@ EXAMPLES = """
 - name: use in a loop
   ansible.builtin.debug:
     msg: "MX record for gmail.com {{ item }}"
-  with_items: "{{ lookup('community.general.dig', 'gmail.com./MX', wantlist=True) }}"
+  with_items: "{{ lookup('community.general.dig', 'gmail.com./MX', wantlist=true) }}"
 
 - ansible.builtin.debug:
     msg: "Reverse DNS for 192.0.2.5 is {{ lookup('community.general.dig', '192.0.2.5/PTR') }}"
 - ansible.builtin.debug:
     msg: "Reverse DNS for 192.0.2.5 is {{ lookup('community.general.dig', '5.2.0.192.in-addr.arpa./PTR') }}"
 - ansible.builtin.debug:
-    msg: "Reverse DNS for 192.0.2.5 is {{ lookup('community.general.dig', '5.2.0.192.in-addr.arpa.', 'qtype=PTR') }}"
+    msg: "Reverse DNS for 192.0.2.5 is {{ lookup('community.general.dig', '5.2.0.192.in-addr.arpa.', qtype='PTR') }}"
 - ansible.builtin.debug:
     msg: "Querying 198.51.100.23 for IPv4 address for example.com. produces {{ lookup('dig', 'example.com', '@198.51.100.23') }}"
 
 - ansible.builtin.debug:
     msg: "XMPP service for gmail.com. is available at {{ item.target }} on port {{ item.port }}"
-  with_items: "{{ lookup('community.general.dig', '_xmpp-server._tcp.gmail.com./SRV', 'flat=0', wantlist=True) }}"
+  with_items: "{{ lookup('community.general.dig', '_xmpp-server._tcp.gmail.com./SRV', flat=0, wantlist=true) }}"
 
 - name: Retry nameservers that return SERVFAIL
   ansible.builtin.debug:
-    msg: "{{ lookup('community.general.dig', 'example.org./A', 'retry_servfail=True') }}"
+    msg: "{{ lookup('community.general.dig', 'example.org./A', retry_servfail=true) }}"
 """
 
 RETURN = """
@@ -119,9 +135,6 @@ RETURN = """
        DNAME:
            description:
                - target
-       DLV:
-            description:
-                - algorithm, digest_type, key_tag, digest
        DNSKEY:
             description:
                 - flags, algorithm, protocol, key
@@ -185,7 +198,7 @@ try:
     import dns.resolver
     import dns.reversename
     import dns.rdataclass
-    from dns.rdatatype import (A, AAAA, CNAME, DLV, DNAME, DNSKEY, DS, HINFO, LOC,
+    from dns.rdatatype import (A, AAAA, CNAME, DNAME, DNSKEY, DS, HINFO, LOC,
                                MX, NAPTR, NS, NSEC3PARAM, PTR, RP, SOA, SPF, SRV, SSHFP, TLSA, TXT)
     HAVE_DNS = True
 except ImportError:
@@ -207,7 +220,6 @@ def make_rdata_dict(rdata):
         AAAA: ['address'],
         CNAME: ['target'],
         DNAME: ['target'],
-        DLV: ['algorithm', 'digest_type', 'key_tag', 'digest'],
         DNSKEY: ['flags', 'algorithm', 'protocol', 'key'],
         DS: ['algorithm', 'digest_type', 'key_tag', 'digest'],
         HINFO: ['cpu', 'os'],
@@ -237,8 +249,6 @@ def make_rdata_dict(rdata):
             if isinstance(val, dns.name.Name):
                 val = dns.name.Name.to_text(val)
 
-            if rdata.rdtype == DLV and f == 'digest':
-                val = dns.rdata._hexify(rdata.digest).replace(' ', '')
             if rdata.rdtype == DS and f == 'digest':
                 val = dns.rdata._hexify(rdata.digest).replace(' ', '')
             if rdata.rdtype == DNSKEY and f == 'key':
@@ -278,20 +288,26 @@ class LookupModule(LookupBase):
 
             ... flat=0                                      # returns a dict; default is 1 == string
         '''
-
         if HAVE_DNS is False:
             raise AnsibleError("The dig lookup requires the python 'dnspython' library and it is not installed")
 
+        self.set_options(var_options=variables, direct=kwargs)
+
         # Create Resolver object so that we can set NS if necessary
         myres = dns.resolver.Resolver(configure=True)
         edns_size = 4096
         myres.use_edns(0, ednsflags=dns.flags.DO, payload=edns_size)
 
         domain = None
-        qtype = 'A'
-        flat = True
-        fail_on_error = False
-        rdclass = dns.rdataclass.from_text('IN')
+        qtype = self.get_option('qtype')
+        flat = self.get_option('flat')
+        fail_on_error = self.get_option('fail_on_error')
+        real_empty = self.get_option('real_empty')
+        try:
+            rdclass = dns.rdataclass.from_text(self.get_option('class'))
+        except Exception as e:
+            raise AnsibleError("dns lookup illegal CLASS: %s" % to_native(e))
+        myres.retry_servfail = self.get_option('retry_servfail')
 
         for t in terms:
             if t.startswith('@'):       # e.g. "@10.0.1.2,192.0.2.1" is ok.
@@ -314,7 +330,7 @@ class LookupModule(LookupBase):
                 continue
             if '=' in t:
                 try:
-                    opt, arg = t.split('=')
+                    opt, arg = t.split('=', 1)
                 except Exception:
                     pass
 
@@ -331,6 +347,8 @@ class LookupModule(LookupBase):
                     myres.retry_servfail = boolean(arg)
                 elif opt == 'fail_on_error':
                     fail_on_error = boolean(arg)
+                elif opt == 'real_empty':
+                    real_empty = boolean(arg)
 
                 continue
 
@@ -346,11 +364,6 @@ class LookupModule(LookupBase):
 
         ret = []
 
-        if qtype.upper() == 'DLV':
-            display.deprecated('The DLV record type has been decommissioned in 2017 and support for'
-                               ' it will be removed from community.general 6.0.0',
-                               version='6.0.0', collection_name='community.general')
-
         if qtype.upper() == 'PTR':
             try:
                 n = dns.reversename.from_address(domain)
@@ -386,15 +399,18 @@ class LookupModule(LookupBase):
         except dns.resolver.NXDOMAIN as err:
             if fail_on_error:
                 raise AnsibleError("Lookup failed: %s" % str(err))
-            ret.append('NXDOMAIN')
+            if not real_empty:
+                ret.append('NXDOMAIN')
         except dns.resolver.NoAnswer as err:
             if fail_on_error:
                 raise AnsibleError("Lookup failed: %s" % str(err))
-            ret.append("")
+            if not real_empty:
+                ret.append("")
         except dns.resolver.Timeout as err:
             if fail_on_error:
                 raise AnsibleError("Lookup failed: %s" % str(err))
-            ret.append('')
+            if not real_empty:
+                ret.append("")
         except dns.exception.DNSException as err:
             raise AnsibleError("dns.resolver unhandled exception %s" % to_native(err))
 

plugins/lookup/dnstxt.py

@@ -20,6 +20,13 @@ DOCUMENTATION = '''
         required: true
         type: list
         elements: string
+      real_empty:
+        description:
+          - Return empty result without empty strings, and return empty list instead of C(NXDOMAIN).
+          - The default for this option will likely change to C(true) in the future.
+        default: false
+        type: bool
+        version_added: 6.0.0
 '''
 
 EXAMPLES = """
@@ -71,10 +78,13 @@ from ansible.plugins.lookup import LookupBase
 class LookupModule(LookupBase):
 
     def run(self, terms, variables=None, **kwargs):
+        self.set_options(var_options=variables, direct=kwargs)
 
         if HAVE_DNS is False:
             raise AnsibleError("Can't LOOKUP(dnstxt): module dns.resolver is not installed")
 
+        real_empty = self.get_option('real_empty')
+
         ret = []
         for term in terms:
             domain = term.split()[0]
@@ -86,10 +96,16 @@ class LookupModule(LookupBase):
                     string.append(s[1:-1])  # Strip outside quotes on TXT rdata
 
             except dns.resolver.NXDOMAIN:
+                if real_empty:
+                    continue
                 string = 'NXDOMAIN'
             except dns.resolver.Timeout:
+                if real_empty:
+                    continue
                 string = ''
             except dns.resolver.NoAnswer:
+                if real_empty:
+                    continue
                 string = ''
             except DNSException as e:
                 raise AnsibleError("dns.resolver unhandled exception %s" % to_native(e))

plugins/lookup/filetree.py

@@ -201,6 +201,8 @@ def file_props(root, path):
 class LookupModule(LookupBase):
 
     def run(self, terms, variables=None, **kwargs):
+        self.set_options(var_options=variables, direct=kwargs)
+
         basedir = self.get_basedir(variables)
 
         ret = []

plugins/lookup/flattened.py

@@ -11,14 +11,17 @@ DOCUMENTATION = '''
     author: Serge van Ginderachter (!UNKNOWN) <serge@vanginderachter.be>
     short_description: return single list completely flattened
     description:
-      - given one or more lists, this lookup will flatten any list elements found recursively until only 1 list is left.
+      - Given one or more lists, this lookup will flatten any list elements found recursively until only 1 list is left.
     options:
       _terms:
         description: lists to flatten
+        type: list
+        elements: raw
         required: true
     notes:
-      - unlike 'items' which only flattens 1 level, this plugin will continue to flatten until it cannot find lists anymore.
-      - aka highlander plugin, there can only be one (list).
+      - Unlike the R(items lookup,ansible_collections.ansible.builtin.items_lookup) which only flattens 1 level,
+        this plugin will continue to flatten until it cannot find lists anymore.
+      - Aka highlander plugin, there can only be one (list).
 '''
 
 EXAMPLES = """
@@ -78,9 +81,10 @@ class LookupModule(LookupBase):
 
         return ret
 
-    def run(self, terms, variables, **kwargs):
-
+    def run(self, terms, variables=None, **kwargs):
         if not isinstance(terms, list):
             raise AnsibleError("with_flattened expects a list")
 
+        self.set_options(var_options=variables, direct=kwargs)
+
         return self._do_flatten(terms, variables)

plugins/lookup/hiera.py

@@ -14,23 +14,23 @@ DOCUMENTATION = '''
     requirements:
       - hiera (command line utility)
     description:
-        - Retrieves data from an Puppetmaster node using Hiera as ENC
+        - Retrieves data from an Puppetmaster node using Hiera as ENC.
     options:
-      _hiera_key:
+      _terms:
             description:
-                - The list of keys to lookup on the Puppetmaster
+                - The list of keys to lookup on the Puppetmaster.
             type: list
             elements: string
             required: true
-      _bin_file:
+      executable:
             description:
-                - Binary file to execute Hiera
+                - Binary file to execute Hiera.
             default: '/usr/bin/hiera'
             env:
                 - name: ANSIBLE_HIERA_BIN
-      _hierarchy_file:
+      config_file:
             description:
-                - File that describes the hierarchy of Hiera
+                - File that describes the hierarchy of Hiera.
             default: '/etc/hiera.yaml'
             env:
                 - name: ANSIBLE_HIERA_CFG
@@ -67,25 +67,28 @@ from ansible.plugins.lookup import LookupBase
 from ansible.utils.cmd_functions import run_cmd
 from ansible.module_utils.common.text.converters import to_text
 
-ANSIBLE_HIERA_CFG = os.getenv('ANSIBLE_HIERA_CFG', '/etc/hiera.yaml')
-ANSIBLE_HIERA_BIN = os.getenv('ANSIBLE_HIERA_BIN', '/usr/bin/hiera')
-
 
 class Hiera(object):
+    def __init__(self, hiera_cfg, hiera_bin):
+        self.hiera_cfg = hiera_cfg
+        self.hiera_bin = hiera_bin
+
     def get(self, hiera_key):
-        pargs = [ANSIBLE_HIERA_BIN]
-        pargs.extend(['-c', ANSIBLE_HIERA_CFG])
+        pargs = [self.hiera_bin]
+        pargs.extend(['-c', self.hiera_cfg])
 
         pargs.extend(hiera_key)
 
         rc, output, err = run_cmd("{0} -c {1} {2}".format(
-            ANSIBLE_HIERA_BIN, ANSIBLE_HIERA_CFG, hiera_key[0]))
+            self.hiera_bin, self.hiera_cfg, hiera_key[0]))
 
         return to_text(output.strip())
 
 
 class LookupModule(LookupBase):
-    def run(self, terms, variables=''):
-        hiera = Hiera()
+    def run(self, terms, variables=None, **kwargs):
+        self.set_options(var_options=variables, direct=kwargs)
+
+        hiera = Hiera(self.get_option('config_file'), self.get_option('executable'))
         ret = [hiera.get(terms)]
         return ret

plugins/lookup/keyring.py

@@ -26,7 +26,9 @@ EXAMPLES = """
     - 'servicename username'
 
 - name: access mysql with password from keyring
-  mysql_db: login_password={{lookup('community.general.keyring','mysql joe')}} login_user=joe
+  community.mysql.mysql_db:
+    login_password: "{{ lookup('community.general.keyring', 'mysql joe') }}"
+    login_user: joe
 """
 
 RETURN = """
@@ -53,10 +55,12 @@ display = Display()
 
 class LookupModule(LookupBase):
 
-    def run(self, terms, **kwargs):
+    def run(self, terms, variables=None, **kwargs):
         if not HAS_KEYRING:
             raise AnsibleError(u"Can't LOOKUP(keyring): missing required python library 'keyring'")
 
+        self.set_options(var_options=variables, direct=kwargs)
+
         display.vvvv(u"keyring: %s" % keyring.get_keyring())
         ret = []
         for term in terms:

plugins/lookup/lmdb_kv.py

@@ -13,15 +13,20 @@ DOCUMENTATION = '''
     version_added: '0.2.0'
     short_description: fetch data from LMDB
     description:
-      - This lookup returns a list of results from an LMDB DB corresponding to a list of items given to it
+      - This lookup returns a list of results from an LMDB DB corresponding to a list of items given to it.
     requirements:
       - lmdb (python library https://lmdb.readthedocs.io/en/release/)
     options:
       _terms:
-        description: list of keys to query
+        description: List of keys to query.
+        type: list
+        elements: str
       db:
-        description: path to LMDB database
+        description: Path to LMDB database.
+        type: str
         default: 'ansible.mdb'
+        vars:
+          - name: lmdb_kv_db
 '''
 
 EXAMPLES = """
@@ -43,8 +48,8 @@ EXAMPLES = """
       - item == 'Belgium'
     vars:
       - lmdb_kv_db: jp.mdb
-    with_community.general.lmdb_kv:
-      - be
+  with_community.general.lmdb_kv:
+    - be
 """
 
 RETURN = """
@@ -58,6 +63,7 @@ _raw:
 from ansible.errors import AnsibleError
 from ansible.plugins.lookup import LookupBase
 from ansible.module_utils.common.text.converters import to_native, to_text
+
 HAVE_LMDB = True
 try:
     import lmdb
@@ -67,8 +73,7 @@ except ImportError:
 
 class LookupModule(LookupBase):
 
-    def run(self, terms, variables, **kwargs):
-
+    def run(self, terms, variables=None, **kwargs):
         '''
         terms contain any number of keys to be retrieved.
         If terms is None, all keys from the database are returned
@@ -81,17 +86,15 @@ class LookupModule(LookupBase):
               vars:
                 - lmdb_kv_db: "jp.mdb"
         '''
-
         if HAVE_LMDB is False:
             raise AnsibleError("Can't LOOKUP(lmdb_kv): this module requires lmdb to be installed")
 
-        db = variables.get('lmdb_kv_db', None)
-        if db is None:
-            db = kwargs.get('db', 'ansible.mdb')
-        db = str(db)
+        self.set_options(var_options=variables, direct=kwargs)
+
+        db = self.get_option('db')
 
         try:
-            env = lmdb.open(db, readonly=True)
+            env = lmdb.open(str(db), readonly=True)
         except Exception as e:
             raise AnsibleError("LMDB can't open database %s: %s" % (db, to_native(e)))
 

plugins/lookup/manifold.py

@@ -207,7 +207,7 @@ class ManifoldApiClient(object):
 
 class LookupModule(LookupBase):
 
-    def run(self, terms, variables=None, api_token=None, project=None, team=None):
+    def run(self, terms, variables=None, **kwargs):
         """
         :param terms: a list of resources lookups to run.
         :param variables: ansible variables active at the time of the lookup
@@ -217,10 +217,11 @@ class LookupModule(LookupBase):
         :return: a dictionary of resources credentials
         """
 
-        if not api_token:
-            api_token = os.getenv('MANIFOLD_API_TOKEN')
-        if not api_token:
-            raise AnsibleError('API token is required. Please set api_token parameter or MANIFOLD_API_TOKEN env var')
+        self.set_options(var_options=variables, direct=kwargs)
+
+        api_token = self.get_option('api_token')
+        project = self.get_option('project')
+        team = self.get_option('team')
 
         try:
             labels = terms

plugins/lookup/onepassword.py

@@ -32,7 +32,7 @@ DOCUMENTATION = '''
       section:
         description: Item section containing the field to retrieve (case-insensitive). If absent will return first match from any section.
       domain:
-        description: Domain of 1Password. Default is U(1password.com).
+        description: Domain of 1Password.
         version_added: 3.2.0
         default: '1password.com'
         type: str
@@ -55,7 +55,7 @@ DOCUMENTATION = '''
       - This lookup stores potentially sensitive data from 1Password as Ansible facts.
         Facts are subject to caching if enabled, which means this data could be stored in clear text
         on disk or in a database.
-      - Tested with C(op) version 0.5.3
+      - Tested with C(op) version 2.7.2
 '''
 
 EXAMPLES = """
@@ -96,106 +96,123 @@ RETURN = """
     elements: str
 """
 
-import errno
-import json
+import abc
 import os
-
-from subprocess import Popen, PIPE
+import json
+import subprocess
 
 from ansible.plugins.lookup import LookupBase
 from ansible.errors import AnsibleLookupError
+from ansible.module_utils.common.process import get_bin_path
 from ansible.module_utils.common.text.converters import to_bytes, to_text
+from ansible.module_utils.six import with_metaclass
 
 from ansible_collections.community.general.plugins.module_utils.onepassword import OnePasswordConfig
 
 
-class OnePass(object):
-    def __init__(self, path='op'):
-        self.cli_path = path
-        self.logged_in = False
-        self.token = None
-        self.subdomain = None
-        self.domain = None
-        self.username = None
-        self.secret_key = None
-        self.master_password = None
+class OnePassCLIBase(with_metaclass(abc.ABCMeta, object)):
+    bin = "op"
 
-        self._config = OnePasswordConfig()
+    def __init__(self, subdomain=None, domain="1password.com", username=None, secret_key=None, master_password=None):
+        self.subdomain = subdomain
+        self.domain = domain
+        self.username = username
+        self.master_password = master_password
+        self.secret_key = secret_key
 
-    def get_token(self):
-        # If the config file exists, assume an initial signin has taken place and try basic sign in
-        if os.path.isfile(self._config.config_file_path):
+        self._path = None
+        self._version = None
 
-            if not self.master_password:
-                raise AnsibleLookupError('Unable to sign in to 1Password. master_password is required.')
+    def _check_required_params(self, required_params):
+        non_empty_attrs = dict((param, getattr(self, param, None)) for param in required_params if getattr(self, param, None))
+        missing = set(required_params).difference(non_empty_attrs)
+        if missing:
+            prefix = "Unable to sign in to 1Password. Missing required parameter"
+            plural = ""
+            suffix = ": {params}.".format(params=", ".join(missing))
+            if len(missing) > 1:
+                plural = "s"
 
-            try:
-                args = ['signin', '--output=raw']
+            msg = "{prefix}{plural}{suffix}".format(prefix=prefix, plural=plural, suffix=suffix)
+            raise AnsibleLookupError(msg)
 
-                if self.subdomain:
-                    args = ['signin', self.subdomain, '--output=raw']
+    @abc.abstractmethod
+    def _parse_field(self, data_json, field_name, section_title):
+        """Main method for parsing data returned from the op command line tool"""
 
-                rc, out, err = self._run(args, command_input=to_bytes(self.master_password))
-                self.token = out.strip()
+    def _run(self, args, expected_rc=0, command_input=None, ignore_errors=False, environment_update=None):
+        command = [self.path] + args
+        call_kwargs = {
+            "stdout": subprocess.PIPE,
+            "stderr": subprocess.PIPE,
+            "stdin": subprocess.PIPE,
+        }
 
-            except AnsibleLookupError:
-                self.full_login()
+        if environment_update:
+            env = os.environ.copy()
+            env.update(environment_update)
+            call_kwargs["env"] = env
 
-        else:
-            # Attempt a full sign in since there appears to be no existing sign in
-            self.full_login()
-
-    def assert_logged_in(self):
-        try:
-            rc, out, err = self._run(['get', 'account'], ignore_errors=True)
-            if rc == 0:
-                self.logged_in = True
-            if not self.logged_in:
-                self.get_token()
-        except OSError as e:
-            if e.errno == errno.ENOENT:
-                raise AnsibleLookupError("1Password CLI tool '%s' not installed in path on control machine" % self.cli_path)
-            raise e
-
-    def get_raw(self, item_id, vault=None):
-        args = ["get", "item", item_id]
-        if vault is not None:
-            args += ['--vault={0}'.format(vault)]
-        if not self.logged_in:
-            args += [to_bytes('--session=') + self.token]
-        rc, output, dummy = self._run(args)
-        return output
-
-    def get_field(self, item_id, field, section=None, vault=None):
-        output = self.get_raw(item_id, vault)
-        return self._parse_field(output, field, section) if output != '' else ''
-
-    def full_login(self):
-        if None in [self.subdomain, self.username, self.secret_key, self.master_password]:
-            raise AnsibleLookupError('Unable to perform initial sign in to 1Password. '
-                                     'subdomain, username, secret_key, and master_password are required to perform initial sign in.')
-
-        args = [
-            'signin',
-            '{0}.{1}'.format(self.subdomain, self.domain),
-            to_bytes(self.username),
-            to_bytes(self.secret_key),
-            '--output=raw',
-        ]
-
-        rc, out, err = self._run(args, command_input=to_bytes(self.master_password))
-        self.token = out.strip()
-
-    def _run(self, args, expected_rc=0, command_input=None, ignore_errors=False):
-        command = [self.cli_path] + args
-        p = Popen(command, stdout=PIPE, stderr=PIPE, stdin=PIPE)
+        p = subprocess.Popen(command, **call_kwargs)
         out, err = p.communicate(input=command_input)
         rc = p.wait()
+
         if not ignore_errors and rc != expected_rc:
             raise AnsibleLookupError(to_text(err))
+
         return rc, out, err
 
-    def _parse_field(self, data_json, field_name, section_title=None):
+    @abc.abstractmethod
+    def assert_logged_in(self):
+        """Check whether a login session exists"""
+
+    @abc.abstractmethod
+    def full_signin(self):
+        """Performa full login"""
+
+    @abc.abstractmethod
+    def get_raw(self, item_id, vault=None, token=None):
+        """Gets the specified item from the vault"""
+
+    @abc.abstractmethod
+    def signin(self):
+        """Sign in using the master password"""
+
+    @property
+    def path(self):
+        if self._path is None:
+            self._path = get_bin_path(self.bin)
+
+        return self._path
+
+    @property
+    def version(self):
+        if self._version is None:
+            self._version = self.get_current_version()
+
+        return self._version
+
+    @classmethod
+    def get_current_version(cls):
+        """Standalone method to get the op CLI version. Useful when determining which class to load
+        based on the current version."""
+        try:
+            bin_path = get_bin_path(cls.bin)
+        except ValueError:
+            raise AnsibleLookupError("Unable to locate '%s' command line tool" % cls.bin)
+
+        try:
+            b_out = subprocess.check_output([bin_path, "--version"], stderr=subprocess.PIPE)
+        except subprocess.CalledProcessError as cpe:
+            raise AnsibleLookupError("Unable to get the op version: %s" % cpe)
+
+        return to_text(b_out).strip()
+
+
+class OnePassCLIv1(OnePassCLIBase):
+    supports_version = "1"
+
+    def _parse_field(self, data_json, field_name, section_title):
         """
         Retrieves the desired field from the `op` response payload
 
@@ -249,36 +266,356 @@ class OnePass(object):
             # check the details dictionary for `field_name` and return it immediately if it exists
             # when the entry is a "password" instead of a "login" item, the password field is a key
             # in the `details` dictionary:
-            if field_name in data['details']:
-                return data['details'][field_name]
+            if field_name in data["details"]:
+                return data["details"][field_name]
 
             # when the field is not found above, iterate through the fields list in the object details
-            for field_data in data['details'].get('fields', []):
-                if field_data.get('name', '').lower() == field_name.lower():
-                    return field_data.get('value', '')
-        for section_data in data['details'].get('sections', []):
-            if section_title is not None and section_title.lower() != section_data['title'].lower():
+            for field_data in data["details"].get("fields", []):
+                if field_data.get("name", "").lower() == field_name.lower():
+                    return field_data.get("value", "")
+
+        for section_data in data["details"].get("sections", []):
+            if section_title is not None and section_title.lower() != section_data["title"].lower():
                 continue
-            for field_data in section_data.get('fields', []):
-                if field_data.get('t', '').lower() == field_name.lower():
-                    return field_data.get('v', '')
-        return ''
+
+            for field_data in section_data.get("fields", []):
+                if field_data.get("t", "").lower() == field_name.lower():
+                    return field_data.get("v", "")
+
+        return ""
+
+    def assert_logged_in(self):
+        args = ["get", "account"]
+        if self.subdomain:
+            account = "{subdomain}.{domain}".format(subdomain=self.subdomain, domain=self.domain)
+            args.extend(["--account", account])
+
+        rc, out, err = self._run(args, ignore_errors=True)
+
+        return not bool(rc)
+
+    def full_signin(self):
+        required_params = [
+            "subdomain",
+            "username",
+            "secret_key",
+            "master_password",
+        ]
+        self._check_required_params(required_params)
+
+        args = [
+            "signin",
+            "{0}.{1}".format(self.subdomain, self.domain),
+            to_bytes(self.username),
+            to_bytes(self.secret_key),
+            "--raw",
+        ]
+
+        return self._run(args, command_input=to_bytes(self.master_password))
+
+    def get_raw(self, item_id, vault=None, token=None):
+        args = ["get", "item", item_id]
+        if vault is not None:
+            args += ["--vault={0}".format(vault)]
+
+        if token is not None:
+            args += [to_bytes("--session=") + token]
+
+        return self._run(args)
+
+    def signin(self):
+        self._check_required_params(['master_password'])
+
+        args = ["signin", "--raw"]
+        if self.subdomain:
+            args.append(self.subdomain)
+
+        return self._run(args, command_input=to_bytes(self.master_password))
+
+
+class OnePassCLIv2(OnePassCLIBase):
+    """
+    CLIv2 Syntax Reference: https://developer.1password.com/docs/cli/upgrade#step-2-update-your-scripts
+    """
+    supports_version = "2"
+
+    def _parse_field(self, data_json, field_name, section_title=None):
+        """
+        Schema reference: https://developer.1password.com/docs/cli/item-template-json
+
+        Example Data:
+
+            # Password item
+            {
+              "id": "ywvdbojsguzgrgnokmcxtydgdv",
+              "title": "Authy Backup",
+              "version": 1,
+              "vault": {
+                "id": "bcqxysvcnejjrwzoqrwzcqjqxc",
+                "name": "Personal"
+              },
+              "category": "PASSWORD",
+              "last_edited_by": "7FUPZ8ZNE02KSHMAIMKHIVUE17",
+              "created_at": "2015-01-18T13:13:38Z",
+              "updated_at": "2016-02-20T16:23:54Z",
+              "additional_information": "Jan 18, 2015, 08:13:38",
+              "fields": [
+                {
+                  "id": "password",
+                  "type": "CONCEALED",
+                  "purpose": "PASSWORD",
+                  "label": "password",
+                  "value": "OctoberPoppyNuttyDraperySabbath",
+                  "reference": "op://Personal/Authy Backup/password",
+                  "password_details": {
+                    "strength": "FANTASTIC"
+                  }
+                },
+                {
+                  "id": "notesPlain",
+                  "type": "STRING",
+                  "purpose": "NOTES",
+                  "label": "notesPlain",
+                  "value": "Backup password to restore Authy",
+                  "reference": "op://Personal/Authy Backup/notesPlain"
+                }
+              ]
+            }
+
+            # Login item
+            {
+              "id": "awk4s2u44fhnrgppszcsvc663i",
+              "title": "Dummy Login",
+              "version": 2,
+              "vault": {
+                "id": "stpebbaccrq72xulgouxsk4p7y",
+                "name": "Personal"
+              },
+              "category": "LOGIN",
+              "last_edited_by": "LSGPJERUYBH7BFPHMZ2KKGL6AU",
+              "created_at": "2018-04-25T21:55:19Z",
+              "updated_at": "2018-04-25T21:56:06Z",
+              "additional_information": "agent.smith",
+              "urls": [
+                {
+                  "primary": true,
+                  "href": "https://acme.com"
+                }
+              ],
+              "sections": [
+                {
+                  "id": "linked items",
+                  "label": "Related Items"
+                }
+              ],
+              "fields": [
+                {
+                  "id": "username",
+                  "type": "STRING",
+                  "purpose": "USERNAME",
+                  "label": "username",
+                  "value": "agent.smith",
+                  "reference": "op://Personal/Dummy Login/username"
+                },
+                {
+                  "id": "password",
+                  "type": "CONCEALED",
+                  "purpose": "PASSWORD",
+                  "label": "password",
+                  "value": "Q7vFwTJcqwxKmTU]Dzx7NW*wrNPXmj",
+                  "entropy": 159.6083697084228,
+                  "reference": "op://Personal/Dummy Login/password",
+                  "password_details": {
+                    "entropy": 159,
+                    "generated": true,
+                    "strength": "FANTASTIC"
+                  }
+                },
+                {
+                  "id": "notesPlain",
+                  "type": "STRING",
+                  "purpose": "NOTES",
+                  "label": "notesPlain",
+                  "reference": "op://Personal/Dummy Login/notesPlain"
+                }
+              ]
+            }
+        """
+        data = json.loads(data_json)
+        for field in data.get("fields", []):
+            if section_title is None:
+                # If the field name exists in the section, return that value
+                if field.get(field_name):
+                    return field.get(field_name)
+
+                # If the field name doesn't exist in the section, match on the value of "label"
+                # then "id" and return "value"
+                if field.get("label") == field_name:
+                    return field["value"]
+
+                if field.get("id") == field_name:
+                    return field["value"]
+
+            # Look at the section data and get an indentifier. The value of 'id' is either a unique ID
+            # or a human-readable string. If a 'label' field exists, prefer that since
+            # it is the value visible in the 1Password UI when both 'id' and 'label' exist.
+            section = field.get("section", {})
+            current_section_title = section.get("label", section.get("id"))
+            if section_title == current_section_title:
+                # In the correct section. Check "label" then "id" for the desired field_name
+                if field.get("label") == field_name:
+                    return field["value"]
+
+                if field.get("id") == field_name:
+                    return field["value"]
+
+        return ""
+
+    def assert_logged_in(self):
+        args = ["account", "list"]
+        if self.subdomain:
+            account = "{subdomain}.{domain}".format(subdomain=self.subdomain, domain=self.domain)
+            args.extend(["--account", account])
+
+        rc, out, err = self._run(args)
+
+        if out:
+            # Running 'op account get' if there are no accounts configured on the system drops into
+            # an interactive prompt. Only run 'op account get' after first listing accounts to see
+            # if there are any previously configured accounts.
+            args = ["account", "get"]
+            if self.subdomain:
+                account = "{subdomain}.{domain}".format(subdomain=self.subdomain, domain=self.domain)
+                args.extend(["--account", account])
+
+            rc, out, err = self._run(args)
+
+            return not bool(rc)
+
+        return False
+
+    def full_signin(self):
+        required_params = [
+            "subdomain",
+            "username",
+            "secret_key",
+            "master_password",
+        ]
+        self._check_required_params(required_params)
+
+        args = [
+            "account", "add", "--raw",
+            "--address", "{0}.{1}".format(self.subdomain, self.domain),
+            "--email", to_bytes(self.username),
+            "--signin",
+        ]
+
+        environment_update = {"OP_SECRET_KEY": self.secret_key}
+        return self._run(args, command_input=to_bytes(self.master_password), environment_update=environment_update)
+
+    def get_raw(self, item_id, vault=None, token=None):
+        args = ["item", "get", item_id, "--format", "json"]
+        if vault is not None:
+            args += ["--vault={0}".format(vault)]
+        if token is not None:
+            args += [to_bytes("--session=") + token]
+
+        return self._run(args)
+
+    def signin(self):
+        self._check_required_params(['master_password'])
+
+        args = ["signin", "--raw"]
+        if self.subdomain:
+            args.extend(["--account", self.subdomain])
+
+        return self._run(args, command_input=to_bytes(self.master_password))
+
+
+class OnePass(object):
+    def __init__(self, subdomain=None, domain="1password.com", username=None, secret_key=None, master_password=None):
+        self.subdomain = subdomain
+        self.domain = domain
+        self.username = username
+        self.secret_key = secret_key
+        self.master_password = master_password
+
+        self.logged_in = False
+        self.token = None
+
+        self._config = OnePasswordConfig()
+        self._cli = self._get_cli_class()
+
+    def _get_cli_class(self):
+        version = OnePassCLIBase.get_current_version()
+        for cls in OnePassCLIBase.__subclasses__():
+            if cls.supports_version == version.split(".")[0]:
+                try:
+                    return cls(self.subdomain, self.domain, self.username, self.secret_key, self.master_password)
+                except TypeError as e:
+                    raise AnsibleLookupError(e)
+
+        raise AnsibleLookupError("op version %s is unsupported" % version)
+
+    def set_token(self):
+        if self._config.config_file_path and os.path.isfile(self._config.config_file_path):
+            # If the config file exists, assume an initial sign in has taken place and try basic sign in
+            try:
+                rc, out, err = self._cli.signin()
+            except AnsibleLookupError as exc:
+                test_strings = (
+                    "missing required parameters",
+                    "unauthorized",
+                )
+                if any(string in exc.message.lower() for string in test_strings):
+                    # A required parameter is missing, or a bad master password was supplied
+                    # so don't bother attempting a full signin
+                    raise
+
+                rc, out, err = self._cli.full_signin()
+
+            self.token = out.strip()
+
+        else:
+            # Attempt a full signin since there appears to be no existing signin
+            rc, out, err = self._cli.full_signin()
+            self.token = out.strip()
+
+    def assert_logged_in(self):
+        logged_in = self._cli.assert_logged_in()
+        if logged_in:
+            self.logged_in = logged_in
+            pass
+        else:
+            self.set_token()
+
+    def get_raw(self, item_id, vault=None):
+        rc, out, err = self._cli.get_raw(item_id, vault, self.token)
+        return out
+
+    def get_field(self, item_id, field, section=None, vault=None):
+        output = self.get_raw(item_id, vault)
+        if output:
+            return self._cli._parse_field(output, field, section)
+
+        return ""
 
 
 class LookupModule(LookupBase):
 
     def run(self, terms, variables=None, **kwargs):
-        op = OnePass()
+        self.set_options(var_options=variables, direct=kwargs)
 
-        field = kwargs.get('field', 'password')
-        section = kwargs.get('section')
-        vault = kwargs.get('vault')
-        op.subdomain = kwargs.get('subdomain')
-        op.domain = kwargs.get('domain', '1password.com')
-        op.username = kwargs.get('username')
-        op.secret_key = kwargs.get('secret_key')
-        op.master_password = kwargs.get('master_password', kwargs.get('vault_password'))
+        field = self.get_option("field")
+        section = self.get_option("section")
+        vault = self.get_option("vault")
+        subdomain = self.get_option("subdomain")
+        domain = self.get_option("domain")
+        username = self.get_option("username")
+        secret_key = self.get_option("secret_key")
+        master_password = self.get_option("master_password")
 
+        op = OnePass(subdomain, domain, username, secret_key, master_password)
         op.assert_logged_in()
 
         values = []

plugins/lookup/onepassword_raw.py

@@ -30,6 +30,11 @@ DOCUMENTATION = '''
         description: Item section containing the field to retrieve (case-insensitive). If absent will return first match from any section.
       subdomain:
         description: The 1Password subdomain to authenticate against.
+      domain:
+        description: Domain of 1Password.
+        version_added: 6.0.0
+        default: '1password.com'
+        type: str
       username:
         description: The username used to sign in.
       secret_key:
@@ -47,7 +52,7 @@ DOCUMENTATION = '''
       - This lookup stores potentially sensitive data from 1Password as Ansible facts.
         Facts are subject to caching if enabled, which means this data could be stored in clear text
         on disk or in a database.
-      - Tested with C(op) version 0.5.3
+      - Tested with C(op) version 2.7.0
 '''
 
 EXAMPLES = """
@@ -76,18 +81,21 @@ from ansible.plugins.lookup import LookupBase
 class LookupModule(LookupBase):
 
     def run(self, terms, variables=None, **kwargs):
-        op = OnePass()
+        self.set_options(var_options=variables, direct=kwargs)
 
-        vault = kwargs.get('vault')
-        op.subdomain = kwargs.get('subdomain')
-        op.username = kwargs.get('username')
-        op.secret_key = kwargs.get('secret_key')
-        op.master_password = kwargs.get('master_password', kwargs.get('vault_password'))
+        vault = self.get_option("vault")
+        subdomain = self.get_option("subdomain")
+        domain = self.get_option("domain", "1password.com")
+        username = self.get_option("username")
+        secret_key = self.get_option("secret_key")
+        master_password = self.get_option("master_password")
 
+        op = OnePass(subdomain, domain, username, secret_key, master_password)
         op.assert_logged_in()
 
         values = []
         for term in terms:
             data = json.loads(op.get_raw(term, vault))
             values.append(data)
+
         return values

plugins/lookup/passwordstore.py

@@ -21,17 +21,15 @@ DOCUMENTATION = '''
       _terms:
         description: query key.
         required: true
-      passwordstore:
-        description:
-          - Location of the password store.
-          - 'The value is decided by checking the following in order:'
-          - If set, this value is used.
-          - If C(directory) is set, that value will be used.
-          - If I(backend=pass), then C(~/.password-store) is used.
-          - If I(backend=gopass), then the C(path) field in C(~/.config/gopass/config.yml) is used,
-            falling back to C(~/.local/share/gopass/stores/root) if not defined.
       directory:
-        description: The directory of the password store.
+        description:
+          - The directory of the password store.
+          - If I(backend=pass), the default is C(~/.password-store) is used.
+          - If I(backend=gopass), then the default is the C(path) field in C(~/.config/gopass/config.yml),
+            falling back to C(~/.local/share/gopass/stores/root) if C(path) is not defined in the gopass config.
+        type: path
+        vars:
+          - name: passwordstore
         env:
           - name: PASSWORD_STORE_DIR
       create:
@@ -55,9 +53,11 @@ DOCUMENTATION = '''
         default: false
       subkey:
         description: Return a specific subkey of the password. When set to C(password), always returns the first line.
+        type: str
         default: password
       userpass:
         description: Specify a password to save, instead of a generated one.
+        type: str
       length:
         description: The length of the generated password.
         type: integer
@@ -67,7 +67,7 @@ DOCUMENTATION = '''
         type: bool
         default: false
       nosymbols:
-        description: use alphanumeric characters.
+        description: Use alphanumeric characters.
         type: bool
         default: false
       missing:
@@ -129,6 +129,8 @@ DOCUMENTATION = '''
           - pass
           - gopass
         version_added: 5.2.0
+    notes:
+      - The lookup supports passing all options as lookup parameters since community.general 6.0.0.
 '''
 EXAMPLES = """
 ansible.cfg: |
@@ -136,7 +138,7 @@ ansible.cfg: |
   lock=readwrite
   locktimeout=45s
 
-playbook.yml: |
+tasks.yml: |
   ---
 
   # Debug is used for examples, BAD IDEA to show passwords on screen
@@ -146,45 +148,49 @@ playbook.yml: |
 
   - name: Basic lookup. Warns if example/test does not exist and returns empty string
     ansible.builtin.debug:
-      msg: "{{ lookup('community.general.passwordstore', 'example/test missing=warn')}}"
+      msg: "{{ lookup('community.general.passwordstore', 'example/test', missing='warn')}}"
 
   - name: Create pass with random 16 character password. If password exists just give the password
     ansible.builtin.debug:
       var: mypassword
     vars:
-      mypassword: "{{ lookup('community.general.passwordstore', 'example/test create=true')}}"
+      mypassword: "{{ lookup('community.general.passwordstore', 'example/test', create=true)}}"
 
   - name: Create pass with random 16 character password. If password exists just give the password
     ansible.builtin.debug:
       var: mypassword
     vars:
-      mypassword: "{{ lookup('community.general.passwordstore', 'example/test missing=create')}}"
+      mypassword: "{{ lookup('community.general.passwordstore', 'example/test', missing='create')}}"
 
   - name: Prints 'abc' if example/test does not exist, just give the password otherwise
     ansible.builtin.debug:
       var: mypassword
     vars:
-      mypassword: "{{ lookup('community.general.passwordstore', 'example/test missing=empty') | default('abc', true) }}"
+      mypassword: >-
+        {{ lookup('community.general.passwordstore', 'example/test', missing='empty')
+           | default('abc', true) }}
 
   - name: Different size password
     ansible.builtin.debug:
-      msg: "{{ lookup('community.general.passwordstore', 'example/test create=true length=42')}}"
+      msg: "{{ lookup('community.general.passwordstore', 'example/test', create=true, length=42)}}"
 
-  - name: Create password and overwrite the password if it exists. As a bonus, this module includes the old password inside the pass file
+  - name: >-
+      Create password and overwrite the password if it exists.
+      As a bonus, this module includes the old password inside the pass file
     ansible.builtin.debug:
-      msg: "{{ lookup('community.general.passwordstore', 'example/test create=true overwrite=true')}}"
+      msg: "{{ lookup('community.general.passwordstore', 'example/test', create=true, overwrite=true)}}"
 
   - name: Create an alphanumeric password
     ansible.builtin.debug:
-      msg: "{{ lookup('community.general.passwordstore', 'example/test create=true nosymbols=true') }}"
+      msg: "{{ lookup('community.general.passwordstore', 'example/test', create=true, nosymbols=true) }}"
 
   - name: Return the value for user in the KV pair user, username
     ansible.builtin.debug:
-      msg: "{{ lookup('community.general.passwordstore', 'example/test subkey=user')}}"
+      msg: "{{ lookup('community.general.passwordstore', 'example/test', subkey='user')}}"
 
   - name: Return the entire password file content
     ansible.builtin.set_fact:
-      passfilecontent: "{{ lookup('community.general.passwordstore', 'example/test returnall=true')}}"
+      passfilecontent: "{{ lookup('community.general.passwordstore', 'example/test', returnall=true)}}"
 """
 
 RETURN = """
@@ -320,7 +326,7 @@ class LookupModule(LookupBase):
                 raise AnsibleError('Passwordstore directory \'{0}\' does not exist'.format(self.paramvals['directory']))
 
             # Set PASSWORD_STORE_UMASK if umask is set
-            if 'umask' in self.paramvals:
+            if self.paramvals.get('umask') is not None:
                 if len(self.paramvals['umask']) != 3:
                     raise AnsibleError('Passwordstore umask must have a length of 3.')
                 elif int(self.paramvals['umask'][0]) > 3:
@@ -435,8 +441,7 @@ class LookupModule(LookupBase):
         unit_to_seconds = {"s": 1, "m": 60, "h": 3600}
         self.lock_timeout = int(timeout[:-1]) * unit_to_seconds[timeout[-1]]
 
-        directory = variables.get('passwordstore', os.environ.get('PASSWORD_STORE_DIR', None))
-
+        directory = self.get_option('directory')
         if directory is None:
             if self.backend == 'gopass':
                 try:
@@ -448,16 +453,17 @@ class LookupModule(LookupBase):
                 directory = os.path.expanduser('~/.password-store')
 
         self.paramvals = {
-            'subkey': 'password',
+            'subkey': self.get_option('subkey'),
             'directory': directory,
-            'create': False,
-            'returnall': False,
-            'overwrite': False,
-            'nosymbols': False,
-            'userpass': '',
-            'length': 16,
-            'backup': False,
-            'missing': 'error',
+            'create': self.get_option('create'),
+            'returnall': self.get_option('returnall'),
+            'overwrite': self.get_option('overwrite'),
+            'nosymbols': self.get_option('nosymbols'),
+            'userpass': self.get_option('userpass') or '',
+            'length': self.get_option('length'),
+            'backup': self.get_option('backup'),
+            'missing': self.get_option('missing'),
+            'umask': self.get_option('umask'),
         }
 
     def run(self, terms, variables, **kwargs):

plugins/lookup/shelvefile.py

@@ -14,23 +14,24 @@ DOCUMENTATION = '''
       - Read keys from Python shelve file.
     options:
       _terms:
-        description: sets of key value pairs of parameters
+        description: Sets of key value pairs of parameters.
       key:
-        description: key to query
+        description: Key to query.
         required: true
       file:
-        description: path to shelve file
+        description: Path to shelve file.
         required: true
 '''
 
 EXAMPLES = """
-- name: retrieve a string value corresponding to a key inside a Python shelve file
-  ansible.builtin.debug: msg="{{ lookup('community.general.shelvefile', 'file=path_to_some_shelve_file.db key=key_to_retrieve') }}
+- name: Retrieve a string value corresponding to a key inside a Python shelve file
+  ansible.builtin.debug:
+    msg: "{{ lookup('community.general.shelvefile', 'file=path_to_some_shelve_file.db key=key_to_retrieve') }}"
 """
 
 RETURN = """
 _list:
-  description: value(s) of key(s) in shelve file(s)
+  description: Value(s) of key(s) in shelve file(s).
   type: list
   elements: str
 """
@@ -53,7 +54,6 @@ class LookupModule(LookupBase):
         return res
 
     def run(self, terms, variables=None, **kwargs):
-
         if not isinstance(terms, list):
             terms = [terms]
 

plugins/module_utils/cmd_runner.py

@@ -103,8 +103,13 @@ class _ArgFormat(object):
 
 class _Format(object):
     @staticmethod
-    def as_bool(args):
-        return _ArgFormat(lambda value: _ensure_list(args) if value else [])
+    def as_bool(args_true, args_false=None, ignore_none=None):
+        if args_false is not None:
+            if ignore_none is None:
+                ignore_none = False
+        else:
+            args_false = []
+        return _ArgFormat(lambda value: _ensure_list(args_true) if value else _ensure_list(args_false), ignore_none=ignore_none)
 
     @staticmethod
     def as_bool_not(args):

plugins/module_utils/deps.py

@@ -0,0 +1,90 @@
+# -*- coding: utf-8 -*-
+# (c) 2022, Alexei Znamensky <russoz@gmail.com>
+# Copyright (c) 2022, Ansible Project
+# Simplified BSD License (see LICENSES/BSD-2-Clause.txt or https://opensource.org/licenses/BSD-2-Clause)
+# SPDX-License-Identifier: BSD-2-Clause
+
+from __future__ import absolute_import, division, print_function
+__metaclass__ = type
+
+
+import traceback
+from contextlib import contextmanager
+
+from ansible.module_utils.common.text.converters import to_native
+from ansible.module_utils.basic import missing_required_lib
+
+
+_deps = dict()
+
+
+class _Dependency(object):
+    _states = ["pending", "failure", "success"]
+
+    def __init__(self, name, reason=None, url=None, msg=None):
+        self.name = name
+        self.reason = reason
+        self.url = url
+        self.msg = msg
+
+        self.state = 0
+        self.trace = None
+        self.exc = None
+
+    def succeed(self):
+        self.state = 2
+
+    def fail(self, exc, trace):
+        self.state = 1
+        self.exc = exc
+        self.trace = trace
+
+    @property
+    def message(self):
+        if self.msg:
+            return to_native(self.msg)
+        else:
+            return missing_required_lib(self.name, reason=self.reason, url=self.url)
+
+    @property
+    def failed(self):
+        return self.state == 1
+
+    def verify(self, module):
+        if self.failed:
+            module.fail_json(msg=self.message, exception=self.trace)
+
+    def __str__(self):
+        return "<dependency: {0} [{1}]>".format(self.name, self._states[self.state])
+
+
+@contextmanager
+def declare(name, *args, **kwargs):
+    dep = _Dependency(name, *args, **kwargs)
+    try:
+        yield dep
+    except Exception as e:
+        dep.fail(e, traceback.format_exc())
+    else:
+        dep.succeed()
+    finally:
+        _deps[name] = dep
+
+
+def validate(module, spec=None):
+    dep_names = sorted(_deps)
+
+    if spec is not None:
+        if spec.startswith("-"):
+            spec_split = spec[1:].split(":")
+            for d in spec_split:
+                dep_names.remove(d)
+        else:
+            spec_split = spec[1:].split(":")
+            dep_names = []
+            for d in spec_split:
+                _deps[d]  # ensure it exists
+                dep_names.append(d)
+
+    for dep in dep_names:
+        _deps[dep].verify(module)

plugins/module_utils/gconftool2.py

@@ -6,7 +6,14 @@
 from __future__ import absolute_import, division, print_function
 __metaclass__ = type
 
-from ansible_collections.community.general.plugins.module_utils.cmd_runner import CmdRunner, cmd_runner_fmt as fmt
+from ansible_collections.community.general.plugins.module_utils.cmd_runner import CmdRunner, cmd_runner_fmt
+
+
+_state_map = {
+    "present": "--set",
+    "absent": "--unset",
+    "get": "--get",
+}
 
 
 def gconftool2_runner(module, **kwargs):
@@ -14,14 +21,12 @@ def gconftool2_runner(module, **kwargs):
         module,
         command='gconftool-2',
         arg_formats=dict(
-            key=fmt.as_list(),
-            value_type=fmt.as_opt_val("--type"),
-            value=fmt.as_list(),
-            direct=fmt.as_bool("--direct"),
-            config_source=fmt.as_opt_val("--config-source"),
-            get=fmt.as_bool("--get"),
-            set_arg=fmt.as_bool("--set"),
-            unset=fmt.as_bool("--unset"),
+            state=cmd_runner_fmt.as_map(_state_map),
+            key=cmd_runner_fmt.as_list(),
+            value_type=cmd_runner_fmt.as_opt_val("--type"),
+            value=cmd_runner_fmt.as_list(),
+            direct=cmd_runner_fmt.as_bool("--direct"),
+            config_source=cmd_runner_fmt.as_opt_val("--config-source"),
         ),
         **kwargs
     )

plugins/module_utils/gitlab.py

@@ -110,3 +110,14 @@ def gitlab_authentication(module):
             GitLab remove Session API now that private tokens are removed from user API endpoints since version 10.2." % to_native(e))
 
     return gitlab_instance
+
+
+def filter_returned_variables(gitlab_variables):
+    # pop properties we don't know
+    existing_variables = [dict(x.attributes) for x in gitlab_variables]
+    KNOWN = ['key', 'value', 'masked', 'protected', 'variable_type', 'environment_scope']
+    for item in existing_variables:
+        for key in list(item.keys()):
+            if key not in KNOWN:
+                item.pop(key)
+    return existing_variables

plugins/module_utils/identity/keycloak/keycloak.py

@@ -58,6 +58,8 @@ URL_CLIENT_USER_ROLEMAPPINGS = "{url}/admin/realms/{realm}/users/{id}/role-mappi
 URL_CLIENT_USER_ROLEMAPPINGS_AVAILABLE = "{url}/admin/realms/{realm}/users/{id}/role-mappings/clients/{client}/available"
 URL_CLIENT_USER_ROLEMAPPINGS_COMPOSITE = "{url}/admin/realms/{realm}/users/{id}/role-mappings/clients/{client}/composite"
 
+URL_CLIENTSECRET = "{url}/admin/realms/{realm}/clients/{id}/client-secret"
+
 URL_AUTHENTICATION_FLOWS = "{url}/admin/realms/{realm}/authentication/flows"
 URL_AUTHENTICATION_FLOW = "{url}/admin/realms/{realm}/authentication/flows/{id}"
 URL_AUTHENTICATION_FLOW_COPY = "{url}/admin/realms/{realm}/authentication/flows/{copyfrom}/copy"
@@ -606,7 +608,7 @@ class KeycloakAPI(object):
         """
         available_rolemappings_url = URL_CLIENT_GROUP_ROLEMAPPINGS.format(url=self.baseurl, realm=realm, id=gid, client=cid)
         try:
-            open_url(available_rolemappings_url, method="DELETE", http_agent=self.http_agent, headers=self.restheaders,
+            open_url(available_rolemappings_url, method="DELETE", http_agent=self.http_agent, headers=self.restheaders, data=json.dumps(role_rep),
                      validate_certs=self.validate_certs, timeout=self.connection_timeout)
         except Exception as e:
             self.module.fail_json(msg="Could not delete available rolemappings for client %s in group %s, realm %s: %s"
@@ -1160,6 +1162,52 @@ class KeycloakAPI(object):
             self.module.fail_json(msg='Could not update protocolmappers for clientscope %s in realm %s: %s'
                                       % (mapper_rep, realm, str(e)))
 
+    def create_clientsecret(self, id, realm="master"):
+        """ Generate a new client secret by id
+
+        :param id: id (not clientId) of client to be queried
+        :param realm: client from this realm
+        :return: dict of credential representation
+        """
+        clientsecret_url = URL_CLIENTSECRET.format(url=self.baseurl, realm=realm, id=id)
+
+        try:
+            return json.loads(to_native(open_url(clientsecret_url, method='POST', headers=self.restheaders, timeout=self.connection_timeout,
+                                                 validate_certs=self.validate_certs).read()))
+
+        except HTTPError as e:
+            if e.code == 404:
+                return None
+            else:
+                self.module.fail_json(msg='Could not obtain clientsecret of client %s for realm %s: %s'
+                                          % (id, realm, str(e)))
+        except Exception as e:
+            self.module.fail_json(msg='Could not obtain clientsecret of client %s for realm %s: %s'
+                                      % (id, realm, str(e)))
+
+    def get_clientsecret(self, id, realm="master"):
+        """ Obtain client secret by id
+
+        :param id: id (not clientId) of client to be queried
+        :param realm: client from this realm
+        :return: dict of credential representation
+        """
+        clientsecret_url = URL_CLIENTSECRET.format(url=self.baseurl, realm=realm, id=id)
+
+        try:
+            return json.loads(to_native(open_url(clientsecret_url, method='GET', headers=self.restheaders, timeout=self.connection_timeout,
+                                                 validate_certs=self.validate_certs).read()))
+
+        except HTTPError as e:
+            if e.code == 404:
+                return None
+            else:
+                self.module.fail_json(msg='Could not obtain clientsecret of client %s for realm %s: %s'
+                                          % (id, realm, str(e)))
+        except Exception as e:
+            self.module.fail_json(msg='Could not obtain clientsecret of client %s for realm %s: %s'
+                                      % (id, realm, str(e)))
+
     def get_groups(self, realm="master"):
         """ Fetch the name and ID of all groups on the Keycloak server.
 

plugins/module_utils/identity/keycloak/keycloak_clientsecret.py

@@ -0,0 +1,77 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+# Copyright (c) 2022, John Cant <a.johncant@gmail.com>
+# GNU General Public License v3.0+ (see COPYING or https://www.gnu.org/licenses/gpl-3.0.txt)
+# SPDX-License-Identifier: GPL-3.0-or-later
+
+from __future__ import absolute_import, division, print_function
+
+__metaclass__ = type
+
+from ansible.module_utils.basic import AnsibleModule
+
+from ansible_collections.community.general.plugins.module_utils.identity.keycloak.keycloak import \
+    keycloak_argument_spec
+
+
+def keycloak_clientsecret_module():
+    """
+    Returns an AnsibleModule definition for modules that interact with a client
+    secret.
+
+    :return: argument_spec dict
+    """
+    argument_spec = keycloak_argument_spec()
+
+    meta_args = dict(
+        realm=dict(default='master'),
+        id=dict(type='str'),
+        client_id=dict(type='str', aliases=['clientId']),
+    )
+
+    argument_spec.update(meta_args)
+
+    module = AnsibleModule(
+        argument_spec=argument_spec,
+        supports_check_mode=True,
+        required_one_of=([['id', 'client_id'],
+                         ['token', 'auth_realm', 'auth_username', 'auth_password']]),
+        required_together=([['auth_realm', 'auth_username', 'auth_password']]),
+        mutually_exclusive=[
+            ['token', 'auth_realm'],
+            ['token', 'auth_username'],
+            ['token', 'auth_password']
+        ])
+
+    return module
+
+
+def keycloak_clientsecret_module_resolve_params(module, kc):
+    """
+    Given an AnsibleModule definition for keycloak_clientsecret_*, and a
+    KeycloakAPI client, resolve the params needed to interact with the Keycloak
+    client secret, looking up the client by clientId if necessary via an API
+    call.
+
+    :return: tuple of id, realm
+    """
+
+    realm = module.params.get('realm')
+    id = module.params.get('id')
+    client_id = module.params.get('client_id')
+
+    # only lookup the client_id if id isn't provided.
+    # in the case that both are provided, prefer the ID, since it's one
+    # less lookup.
+    if id is None:
+        # Due to the required_one_of spec, client_id is guaranteed to not be None
+        client = kc.get_client_by_clientid(client_id, realm=realm)
+
+        if client is None:
+            module.fail_json(
+                msg='Client does not exist {client_id}'.format(client_id=client_id)
+            )
+
+        id = client['id']
+
+    return id, realm

plugins/module_utils/jenkins.py

@@ -0,0 +1,35 @@
+# -*- coding: utf-8 -*-
+
+# Copyright (c) 2022, Alexei Znamensky <russoz@gmail.com>
+#
+# GNU General Public License v3.0+ (see LICENSES/GPL-3.0-or-later.txt or https://www.gnu.org/licenses/gpl-3.0.txt)
+# SPDX-License-Identifier: GPL-3.0-or-later
+
+from __future__ import absolute_import, division, print_function
+__metaclass__ = type
+
+
+import os
+import time
+
+
+def download_updates_file(updates_expiration):
+    updates_filename = 'jenkins-plugin-cache.json'
+    updates_dir = os.path.expanduser('~/.ansible/tmp')
+    updates_file = os.path.join(updates_dir, updates_filename)
+    download_updates = True
+
+    # Make sure the destination directory exists
+    if not os.path.isdir(updates_dir):
+        os.makedirs(updates_dir, 0o700)
+
+    # Check if we need to download new updates file
+    if os.path.isfile(updates_file):
+        # Get timestamp when the file was changed last time
+        ts_file = os.stat(updates_file).st_mtime
+        ts_now = time.time()
+
+        if ts_now - ts_file < updates_expiration:
+            download_updates = False
+
+    return updates_file, download_updates

plugins/module_utils/ldap.py

@@ -15,6 +15,8 @@ from ansible.module_utils.common.text.converters import to_native
 
 try:
     import ldap
+    import ldap.dn
+    import ldap.filter
     import ldap.sasl
 
     HAS_LDAP = True
@@ -48,7 +50,6 @@ class LdapGeneric(object):
         self.module = module
         self.bind_dn = self.module.params['bind_dn']
         self.bind_pw = self.module.params['bind_pw']
-        self.dn = self.module.params['dn']
         self.referrals_chasing = self.module.params['referrals_chasing']
         self.server_uri = self.module.params['server_uri']
         self.start_tls = self.module.params['start_tls']
@@ -58,6 +59,9 @@ class LdapGeneric(object):
         # Establish connection
         self.connection = self._connect_to_ldap()
 
+        # Try to find the X_ORDERed version of the DN
+        self.dn = self._find_dn()
+
     def fail(self, msg, exn):
         self.module.fail_json(
             msg=msg,
@@ -65,6 +69,24 @@ class LdapGeneric(object):
             exception=traceback.format_exc()
         )
 
+    def _find_dn(self):
+        dn = self.module.params['dn']
+
+        explode_dn = ldap.dn.explode_dn(dn)
+
+        if len(explode_dn) > 1:
+            try:
+                escaped_value = ldap.filter.escape_filter_chars(explode_dn[0])
+                filterstr = "(%s)" % escaped_value
+                dns = self.connection.search_s(','.join(explode_dn[1:]),
+                                               ldap.SCOPE_ONELEVEL, filterstr)
+                if len(dns) == 1:
+                    dn, dummy = dns[0]
+            except Exception:
+                pass
+
+        return dn
+
     def _connect_to_ldap(self):
         if not self.verify_cert:
             ldap.set_option(ldap.OPT_X_TLS_REQUIRE_CERT, ldap.OPT_X_TLS_NEVER)

plugins/module_utils/lxd.py

@@ -8,8 +8,10 @@ from __future__ import (absolute_import, division, print_function)
 __metaclass__ = type
 
 
+import os
 import socket
 import ssl
+import json
 
 from ansible.module_utils.urls import generic_urlparse
 from ansible.module_utils.six.moves.urllib.parse import urlparse
@@ -20,8 +22,6 @@ from ansible.module_utils.common.text.converters import to_text
 HTTPConnection = http_client.HTTPConnection
 HTTPSConnection = http_client.HTTPSConnection
 
-import json
-
 
 class UnixHTTPConnection(HTTPConnection):
     def __init__(self, path):
@@ -124,3 +124,11 @@ class LXDClient(object):
         if err is None:
             err = resp_json.get('error', None)
         return err
+
+
+def default_key_file():
+    return os.path.expanduser('~/.config/lxc/client.key')
+
+
+def default_cert_file():
+    return os.path.expanduser('~/.config/lxc/client.crt')

plugins/module_utils/mh/module_helper.py

@@ -13,7 +13,7 @@ from ansible_collections.community.general.plugins.module_utils.mh.base import M
 from ansible_collections.community.general.plugins.module_utils.mh.mixins.cmd import CmdMixin
 from ansible_collections.community.general.plugins.module_utils.mh.mixins.state import StateMixin
 from ansible_collections.community.general.plugins.module_utils.mh.mixins.deps import DependencyMixin
-from ansible_collections.community.general.plugins.module_utils.mh.mixins.vars import VarsMixin, VarDict as _VD
+from ansible_collections.community.general.plugins.module_utils.mh.mixins.vars import VarsMixin
 from ansible_collections.community.general.plugins.module_utils.mh.mixins.deprecate_attrs import DeprecateAttrsMixin
 
 
@@ -25,8 +25,6 @@ class ModuleHelper(DeprecateAttrsMixin, VarsMixin, DependencyMixin, ModuleHelper
     change_params = ()
     facts_params = ()
 
-    VarDict = _VD  # for backward compatibility, will be deprecated at some point
-
     def __init__(self, module=None):
         super(ModuleHelper, self).__init__(module)
         for name, value in self.module.params.items():
@@ -38,16 +36,6 @@ class ModuleHelper(DeprecateAttrsMixin, VarsMixin, DependencyMixin, ModuleHelper
                 fact=name in self.facts_params,
             )
 
-        self._deprecate_attr(
-            attr="VarDict",
-            msg="ModuleHelper.VarDict attribute is deprecated, use VarDict from "
-                "the ansible_collections.community.general.plugins.module_utils.mh.mixins.vars module instead",
-            version="6.0.0",
-            collection_name="community.general",
-            target=ModuleHelper,
-            module=self.module,
-        )
-
     def update_output(self, **kwargs):
         self.update_vars(meta={"output": True}, **kwargs)
 

plugins/module_utils/puppet.py

@@ -0,0 +1,114 @@
+# -*- coding: utf-8 -*-
+# Copyright (c) 2022, Alexei Znamensky <russoz@gmail.com>
+# GNU General Public License v3.0+ (see LICENSES/GPL-3.0-or-later.txt or https://www.gnu.org/licenses/gpl-3.0.txt)
+# SPDX-License-Identifier: GPL-3.0-or-later
+
+from __future__ import absolute_import, division, print_function
+__metaclass__ = type
+
+
+import os
+
+from ansible_collections.community.general.plugins.module_utils.cmd_runner import CmdRunner, cmd_runner_fmt
+
+
+_PUPPET_PATH_PREFIX = ["/opt/puppetlabs/bin"]
+
+
+def get_facter_dir():
+    if os.getuid() == 0:
+        return '/etc/facter/facts.d'
+    else:
+        return os.path.expanduser('~/.facter/facts.d')
+
+
+def _puppet_cmd(module):
+    return module.get_bin_path("puppet", False, _PUPPET_PATH_PREFIX)
+
+
+# If the `timeout` CLI command feature is removed,
+# Then we could add this as a fixed param to `puppet_runner`
+def ensure_agent_enabled(module):
+    runner = CmdRunner(
+        module,
+        command="puppet",
+        path_prefix=_PUPPET_PATH_PREFIX,
+        arg_formats=dict(
+            _agent_disabled=cmd_runner_fmt.as_fixed(['config', 'print', 'agent_disabled_lockfile']),
+        ),
+        check_rc=False,
+    )
+
+    rc, stdout, stderr = runner("_agent_disabled").run()
+    if os.path.exists(stdout.strip()):
+        module.fail_json(
+            msg="Puppet agent is administratively disabled.",
+            disabled=True)
+    elif rc != 0:
+        module.fail_json(
+            msg="Puppet agent state could not be determined.")
+
+
+def puppet_runner(module):
+
+    # Keeping backward compatibility, allow for running with the `timeout` CLI command.
+    # If this can be replaced with ansible `timeout` parameter in playbook,
+    # then this function could be removed.
+    def _prepare_base_cmd():
+        _tout_cmd = module.get_bin_path("timeout", False)
+        if _tout_cmd:
+            cmd = ["timeout", "-s", "9", module.params["timeout"], _puppet_cmd(module)]
+        else:
+            cmd = ["puppet"]
+        return cmd
+
+    def noop_func(v):
+        _noop = cmd_runner_fmt.as_map({
+            True: "--noop",
+            False: "--no-noop",
+        })
+        return _noop(module.check_mode or v)
+
+    _logdest_map = {
+        "syslog": ["--logdest", "syslog"],
+        "all": ["--logdest", "syslog", "--logdest", "console"],
+    }
+
+    @cmd_runner_fmt.unpack_args
+    def execute_func(execute, manifest):
+        if execute:
+            return ["--execute", execute]
+        else:
+            return [manifest]
+
+    runner = CmdRunner(
+        module,
+        command=_prepare_base_cmd(),
+        path_prefix=_PUPPET_PATH_PREFIX,
+        arg_formats=dict(
+            _agent_fixed=cmd_runner_fmt.as_fixed([
+                "agent", "--onetime", "--no-daemonize", "--no-usecacheonfailure",
+                "--no-splay", "--detailed-exitcodes", "--verbose", "--color", "0",
+            ]),
+            _apply_fixed=cmd_runner_fmt.as_fixed(["apply", "--detailed-exitcodes"]),
+            puppetmaster=cmd_runner_fmt.as_opt_val("--server"),
+            show_diff=cmd_runner_fmt.as_bool("--show-diff"),
+            confdir=cmd_runner_fmt.as_opt_val("--confdir"),
+            environment=cmd_runner_fmt.as_opt_val("--environment"),
+            tags=cmd_runner_fmt.as_func(lambda v: ["--tags", ",".join(v)]),
+            certname=cmd_runner_fmt.as_opt_eq_val("--certname"),
+            noop=cmd_runner_fmt.as_func(noop_func),
+            use_srv_records=cmd_runner_fmt.as_map({
+                True: "--usr_srv_records",
+                False: "--no-usr_srv_records",
+            }),
+            logdest=cmd_runner_fmt.as_map(_logdest_map, default=[]),
+            modulepath=cmd_runner_fmt.as_opt_eq_val("--modulepath"),
+            _execute=cmd_runner_fmt.as_func(execute_func),
+            summarize=cmd_runner_fmt.as_bool("--summarize"),
+            debug=cmd_runner_fmt.as_bool("--debug"),
+            verbose=cmd_runner_fmt.as_bool("--verbose"),
+        ),
+        check_rc=False,
+    )
+    return runner

plugins/module_utils/rax.py

@@ -314,3 +314,21 @@ def setup_rax_module(module, rax_module, region_required=True):
                          (region, ','.join(rax_module.regions)))
 
     return rax_module
+
+
+def rax_scaling_group_personality_file(module, files):
+    if not files:
+        return []
+
+    results = []
+    for rpath, lpath in files.items():
+        lpath = os.path.expanduser(lpath)
+        try:
+            with open(lpath, 'r') as f:
+                results.append({
+                    'path': rpath,
+                    'contents': f.read(),
+                })
+        except Exception as e:
+            module.fail_json(msg='Failed to load %s: %s' % (lpath, str(e)))
+    return results

plugins/module_utils/scaleway.py

@@ -11,11 +11,21 @@ import re
 import sys
 import datetime
 import time
+import traceback
 
-from ansible.module_utils.basic import env_fallback
+from ansible.module_utils.basic import env_fallback, missing_required_lib
 from ansible.module_utils.urls import fetch_url
 from ansible.module_utils.six.moves.urllib.parse import urlencode
 
+SCALEWAY_SECRET_IMP_ERR = None
+try:
+    from passlib.hash import argon2
+    HAS_SCALEWAY_SECRET_PACKAGE = True
+except Exception:
+    argon2 = None
+    SCALEWAY_SECRET_IMP_ERR = traceback.format_exc()
+    HAS_SCALEWAY_SECRET_PACKAGE = False
+
 
 def scaleway_argument_spec():
     return dict(
@@ -80,6 +90,44 @@ def filter_sensitive_attributes(container, attributes):
     return container
 
 
+class SecretVariables(object):
+    @staticmethod
+    def ensure_scaleway_secret_package(module):
+        if not HAS_SCALEWAY_SECRET_PACKAGE:
+            module.fail_json(
+                msg=missing_required_lib("passlib[argon2]", url='https://passlib.readthedocs.io/en/stable/'),
+                exception=SCALEWAY_SECRET_IMP_ERR
+            )
+
+    @staticmethod
+    def dict_to_list(source_dict):
+        return [
+            dict(key=var[0], value=var[1])
+            for var in source_dict.items()
+        ]
+
+    @staticmethod
+    def list_to_dict(source_list, hashed=False):
+        key_value = 'hashed_value' if hashed else 'value'
+        return dict(
+            (var['key'], var[key_value])
+            for var in source_list
+        )
+
+    @classmethod
+    def decode(cls, secrets_list, values_list):
+        secrets_dict = cls.list_to_dict(secrets_list, hashed=True)
+        values_dict = cls.list_to_dict(values_list, hashed=False)
+        for key in values_dict:
+            if key in secrets_dict:
+                if argon2.verify(values_dict[key], secrets_dict[key]):
+                    secrets_dict[key] = values_dict[key]
+                else:
+                    secrets_dict[key] = secrets_dict[key]
+
+        return cls.dict_to_list(secrets_dict)
+
+
 def resource_attributes_should_be_changed(target, wished, verifiable_mutable_attributes, mutable_attributes):
     diff = dict()
     for attr in verifiable_mutable_attributes:

plugins/module_utils/source_control/bitbucket.py

@@ -28,7 +28,7 @@ class BitbucketHelper:
             # TODO:
             # - Rename user to username once current usage of username is removed
             # - Alias user to username and deprecate it
-            user=dict(type='str', fallback=(env_fallback, ['BITBUCKET_USERNAME'])),
+            user=dict(type='str', aliases=['username'], fallback=(env_fallback, ['BITBUCKET_USERNAME'])),
             password=dict(type='str', no_log=True, fallback=(env_fallback, ['BITBUCKET_PASSWORD'])),
         )
 

plugins/module_utils/ssh.py

@@ -0,0 +1,21 @@
+# -*- coding: utf-8 -*-
+# Copyright (c) 2015, Björn Andersson
+# Copyright (c) 2021, Ansible Project
+# Copyright (c) 2021, Abhijeet Kasurde <akasurde@redhat.com>
+# Copyright (c) 2022, Alexei Znamensky <russoz@gmail.com>
+# GNU General Public License v3.0+ (see LICENSES/GPL-3.0-or-later.txt or https://www.gnu.org/licenses/gpl-3.0.txt)
+# SPDX-License-Identifier: GPL-3.0-or-later
+
+from __future__ import absolute_import, division, print_function
+__metaclass__ = type
+
+
+import os
+
+
+def determine_config_file(user, config_file):
+    if user:
+        config_file = os.path.join(os.path.expanduser('~%s' % user), '.ssh', 'config')
+    elif config_file is None:
+        config_file = '/etc/ssh/ssh_config'
+    return config_file

plugins/modules/ali_instance_info.py

@@ -60,6 +60,8 @@ requirements:
     - "footmark >= 1.13.0"
 extends_documentation_fragment:
     - community.general.alicloud
+    - community.general.attributes
+    - community.general.attributes.info_module
 '''
 
 EXAMPLES = '''

plugins/modules/ansible_galaxy_install.py

@@ -20,6 +20,10 @@ notes:
   - >
     B(Ansible 2.9/2.10): The C(ansible-galaxy) command changed significantly between Ansible 2.9 and
     ansible-base 2.10 (later ansible-core 2.11). See comments in the parameters.
+  - >
+    The module will try and run using the C(C.UTF-8) locale.
+    If that fails, it will try C(en_US.UTF-8).
+    If that one also fails, the module will fail.
 requirements:
   - Ansible 2.9, ansible-base 2.10, or ansible-core 2.11 or newer
 options:
@@ -184,10 +188,11 @@ RETURN = """
 
 import re
 
-from ansible_collections.community.general.plugins.module_utils.module_helper import CmdModuleHelper, ArgFormat
+from ansible_collections.community.general.plugins.module_utils.cmd_runner import CmdRunner, cmd_runner_fmt as fmt
+from ansible_collections.community.general.plugins.module_utils.module_helper import ModuleHelper, ModuleHelperException
 
 
-class AnsibleGalaxyInstall(CmdModuleHelper):
+class AnsibleGalaxyInstall(ModuleHelper):
     _RE_GALAXY_VERSION = re.compile(r'^ansible-galaxy(?: \[core)? (?P<version>\d+\.\d+\.\d+)(?:\.\w+)?(?:\])?')
     _RE_LIST_PATH = re.compile(r'^# (?P<path>.*)$')
     _RE_LIST_COLL = re.compile(r'^(?P<elem>\w+\.\w+)\s+(?P<version>[\d\.]+)\s*$')
@@ -216,29 +221,46 @@ class AnsibleGalaxyInstall(CmdModuleHelper):
 
     command = 'ansible-galaxy'
     command_args_formats = dict(
-        type=dict(fmt=lambda v: [] if v == 'both' else [v]),
-        galaxy_cmd=dict(),
-        requirements_file=dict(fmt=('-r', '{0}'),),
-        dest=dict(fmt=('-p', '{0}'),),
-        force=dict(fmt="--force", style=ArgFormat.BOOLEAN),
-        no_deps=dict(fmt="--no-deps", style=ArgFormat.BOOLEAN),
+        type=fmt.as_func(lambda v: [] if v == 'both' else [v]),
+        galaxy_cmd=fmt.as_list(),
+        requirements_file=fmt.as_opt_val('-r'),
+        dest=fmt.as_opt_val('-p'),
+        force=fmt.as_bool("--force"),
+        no_deps=fmt.as_bool("--no-deps"),
+        version=fmt.as_bool("--version"),
+        name=fmt.as_list(),
     )
-    force_lang = "en_US.UTF-8"
-    check_rc = True
+
+    def _make_runner(self, lang):
+        return CmdRunner(self.module, command=self.command, arg_formats=self.command_args_formats, force_lang=lang, check_rc=True)
 
     def _get_ansible_galaxy_version(self):
-        ansible_galaxy = self.get_bin_path("ansible-galaxy", required=True)
-        dummy, out, dummy = self.module.run_command([ansible_galaxy, "--version"], check_rc=True)
-        line = out.splitlines()[0]
-        match = self._RE_GALAXY_VERSION.match(line)
-        if not match:
-            raise RuntimeError("Unable to determine ansible-galaxy version from: {0}".format(line))
-        version = match.group("version")
-        version = tuple(int(x) for x in version.split('.')[:3])
-        return version
+        class UnsupportedLocale(ModuleHelperException):
+            pass
+
+        def process(rc, out, err):
+            if (rc != 0 and "unsupported locale setting" in err) or (rc == 0 and "cannot change locale" in err):
+                raise UnsupportedLocale(msg=err)
+            line = out.splitlines()[0]
+            match = self._RE_GALAXY_VERSION.match(line)
+            if not match:
+                self.do_raise("Unable to determine ansible-galaxy version from: {0}".format(line))
+            version = match.group("version")
+            version = tuple(int(x) for x in version.split('.')[:3])
+            return version
+
+        try:
+            runner = self._make_runner("C.UTF-8")
+            with runner("version", check_rc=False, output_process=process) as ctx:
+                return runner, ctx.run(version=True)
+        except UnsupportedLocale as e:
+            runner = self._make_runner("en_US.UTF-8")
+            with runner("version", check_rc=True, output_process=process) as ctx:
+                return runner, ctx.run(version=True)
 
     def __init_module__(self):
-        self.ansible_version = self._get_ansible_galaxy_version()
+        # self.runner = CmdRunner(self.module, command=self.command, arg_formats=self.command_args_formats, force_lang=self.force_lang)
+        self.runner, self.ansible_version = self._get_ansible_galaxy_version()
         if self.ansible_version < (2, 11) and not self.vars.ack_min_ansiblecore211:
             self.module.deprecate(
                 "Support for Ansible 2.9 and ansible-base 2.10 is being deprecated. "
@@ -260,18 +282,13 @@ class AnsibleGalaxyInstall(CmdModuleHelper):
                                                  r'|- (?P<role>\w+\.\w+) \((?P<rversion>[\d\.]+)\))'
                                                  r' was installed successfully$')
 
-    @staticmethod
-    def _process_output_list(*args):
-        if "None of the provided paths were usable" in args[1]:
-            return []
-        return args[1].splitlines()
-
     def _list_element(self, _type, path_re, elem_re):
-        params = ({'type': _type}, {'galaxy_cmd': 'list'}, 'dest')
-        elems = self.run_command(params=params,
-                                 publish_rc=False, publish_out=False, publish_err=False, publish_cmd=False,
-                                 process_output=self._process_output_list,
-                                 check_rc=False)
+        def process(rc, out, err):
+            return [] if "None of the provided paths were usable" in out else out.splitlines()
+
+        with self.runner('type galaxy_cmd dest', output_process=process, check_rc=False) as ctx:
+            elems = ctx.run(type=_type, galaxy_cmd='list')
+
         elems_dict = {}
         current_path = None
         for line in elems:
@@ -316,33 +333,34 @@ class AnsibleGalaxyInstall(CmdModuleHelper):
             self.vars.installed_collections = self._list_collections()
 
     def __run__(self):
+        def process(rc, out, err):
+            for line in out.splitlines():
+                match = self._RE_INSTALL_OUTPUT.match(line)
+                if not match:
+                    continue
+                if match.group("collection"):
+                    self.vars.new_collections[match.group("collection")] = match.group("cversion")
+                    if self.is_ansible29:
+                        self.vars.ansible29_change = True
+                elif match.group("role"):
+                    self.vars.new_roles[match.group("role")] = match.group("rversion")
+                    if self.is_ansible29:
+                        self.vars.ansible29_change = True
+
         if self.is_ansible29:
             if self.vars.type == 'both':
                 raise ValueError("Type 'both' not supported in Ansible 2.9")
             self._setup29()
         else:
             self._setup210plus()
-        params = ('type', {'galaxy_cmd': 'install'}, 'force', 'no_deps', 'dest', 'requirements_file', 'name')
-        self.run_command(params=params)
-
-    def process_command_output(self, rc, out, err):
-        for line in out.splitlines():
-            match = self._RE_INSTALL_OUTPUT.match(line)
-            if not match:
-                continue
-            if match.group("collection"):
-                self.vars.new_collections[match.group("collection")] = match.group("cversion")
-                if self.is_ansible29:
-                    self.vars.ansible29_change = True
-            elif match.group("role"):
-                self.vars.new_roles[match.group("role")] = match.group("rversion")
-                if self.is_ansible29:
-                    self.vars.ansible29_change = True
+        with self.runner("type galaxy_cmd force no_deps dest requirements_file name", output_process=process) as ctx:
+            ctx.run(galaxy_cmd="install")
+            if self.verbosity > 2:
+                self.vars.set("run_info", ctx.run_info)
 
 
 def main():
-    galaxy = AnsibleGalaxyInstall()
-    galaxy.run()
+    AnsibleGalaxyInstall.execute()
 
 
 if __name__ == '__main__':

plugins/modules/bitbucket_access_key.py

@@ -27,10 +27,9 @@ options:
   workspace:
     description:
       - The repository owner.
-      - Alias I(username) has been deprecated and will become an alias of I(user) in community.general 6.0.0.
+      - I(username) used to be an alias of this option. Since community.general 6.0.0 it is an alias of I(user).
     type: str
     required: true
-    aliases: [ username ]
   key:
     description:
       - The SSH public key.
@@ -218,8 +217,7 @@ def main():
     argument_spec.update(
         repository=dict(type='str', required=True),
         workspace=dict(
-            type='str', aliases=['username'], required=True,
-            deprecated_aliases=[dict(name='username', version='6.0.0', collection_name='community.general')],
+            type='str', required=True,
         ),
         key=dict(type='str', no_log=False),
         label=dict(type='str', required=True),

plugins/modules/bitbucket_pipeline_key_pair.py

@@ -27,10 +27,9 @@ options:
   workspace:
     description:
       - The repository owner.
-      - Alias I(username) has been deprecated and will become an alias of I(user) in community.general 6.0.0.
+      - I(username) used to be an alias of this option. Since community.general 6.0.0 it is an alias of I(user).
     type: str
     required: true
-    aliases: [ username ]
   public_key:
     description:
       - The public key.
@@ -154,10 +153,7 @@ def main():
     argument_spec = BitbucketHelper.bitbucket_argument_spec()
     argument_spec.update(
         repository=dict(type='str', required=True),
-        workspace=dict(
-            type='str', aliases=['username'], required=True,
-            deprecated_aliases=[dict(name='username', version='6.0.0', collection_name='community.general')],
-        ),
+        workspace=dict(type='str', required=True),
         public_key=dict(type='str'),
         private_key=dict(type='str', no_log=True),
         state=dict(type='str', choices=['present', 'absent'], required=True),

plugins/modules/bitbucket_pipeline_known_host.py

@@ -30,10 +30,9 @@ options:
   workspace:
     description:
       - The repository owner.
-      - Alias I(username) has been deprecated and will become an alias of I(user) in community.general 6.0.0.
+      - I(username) used to be an alias of this option. Since community.general 6.0.0 it is an alias of I(user).
     type: str
     required: true
-    aliases: [ username ]
   name:
     description:
       - The FQDN of the known host.
@@ -255,10 +254,7 @@ def main():
     argument_spec = BitbucketHelper.bitbucket_argument_spec()
     argument_spec.update(
         repository=dict(type='str', required=True),
-        workspace=dict(
-            type='str', aliases=['username'], required=True,
-            deprecated_aliases=[dict(name='username', version='6.0.0', collection_name='community.general')],
-        ),
+        workspace=dict(type='str', required=True),
         name=dict(type='str', required=True),
         key=dict(type='str', no_log=False),
         state=dict(type='str', choices=['present', 'absent'], required=True),

plugins/modules/bitbucket_pipeline_variable.py

@@ -27,10 +27,9 @@ options:
   workspace:
     description:
       - The repository owner.
-      - Alias I(username) has been deprecated and will become an alias of I(user) in community.general 6.0.0.
+      - I(username) used to be an alias of this option. Since community.general 6.0.0 it is an alias of I(user).
     type: str
     required: true
-    aliases: [ username ]
   name:
     description:
       - The pipeline variable name.
@@ -215,10 +214,7 @@ def main():
     argument_spec = BitbucketHelper.bitbucket_argument_spec()
     argument_spec.update(
         repository=dict(type='str', required=True),
-        workspace=dict(
-            type='str', aliases=['username'], required=True,
-            deprecated_aliases=[dict(name='username', version='6.0.0', collection_name='community.general')],
-        ),
+        workspace=dict(type='str', required=True),
         name=dict(type='str', required=True),
         value=dict(type='str'),
         secured=dict(type='bool', default=False),

plugins/modules/cloud_init_data_facts.py

@@ -15,6 +15,10 @@ short_description: Retrieve facts of cloud-init
 description:
   - Gathers facts by reading the status.json and result.json of cloud-init.
 author: René Moser (@resmo)
+extends_documentation_fragment:
+  - community.general.attributes
+  - community.general.attributes.facts
+  - community.general.attributes.facts_module
 options:
   filter:
     description: