Merge pull request #690 from barpavel/fix-686-module-utils-deprecation

Fix all deprecated module_utils imports before ansible-core 2.24 removal

SUMMARY
Fixes all deprecated ansible.module_utils imports across the entire collection that will be removed in ansible-core 2.24.
This PR comprehensively addresses deprecation warnings reported in #686 by updating import statements in 20 files to use the new recommended import paths, and removes 8 unused test utility files that contained deprecated imports.
Deprecated imports replaced:



Deprecated import
Replacement




ansible.module_utils._text
ansible.module_utils.common.text.converters


ansible.module_utils.common._collections_compat
collections.abc


ansible.module_utils.six.moves.shlex_quote
shlex.quote


ansible.module_utils.six.moves.reduce
functools.reduce


ansible.module_utils.six.moves.urllib.parse.urlparse
urllib.parse.urlparse


ansible.module_utils.six.string_types
basestring/str (Python 2/3 compatible)


ansible.module_utils.six.text_type
str


ansible.module_utils.six.PY3
Removed (simplified Python 2/3 conditionals)


ansible.module_utils.six.with_metaclass
Native metaclass= syntax


ansible.module_utils.six.iteritems
dict.items()



Files fixed (20 files, 1 commit per file for easier review):

plugins/action/patch.py
plugins/action/synchronize.py
plugins/callback/cgroup_perf_recap.py
plugins/callback/json.py
plugins/callback/jsonl.py
plugins/callback/profile_roles.py
plugins/callback/profile_tasks.py
plugins/modules/acl.py
plugins/modules/authorized_key.py
plugins/modules/firewalld_info.py
plugins/modules/mount.py
plugins/modules/patch.py
plugins/modules/rhel_rpm_ostree.py
plugins/modules/rpm_ostree_upgrade.py
plugins/modules/seboolean.py
plugins/modules/synchronize.py
plugins/modules/sysctl.py
plugins/shell/csh.py
plugins/shell/fish.py
tests/unit/modules/system/test_mount.py

Files deleted (8 unused test utility files):
These files are dead code - none of them are imported or used anywhere in the test suite or the collection. Removing them also addresses Python 2.7 compatibility concerns raised in code review, as several contained deprecated imports that would be incorrect to fix for Python 2.

tests/unit/compat/builtins.py
tests/unit/mock/loader.py
tests/unit/mock/path.py
tests/unit/mock/procenv.py
tests/unit/mock/vault_helper.py
tests/unit/mock/yaml_helper.py
tests/unit/modules/conftest.py
tests/unit/modules/utils.py

Completeness verified with:
git grep -n -P '_compat|utils._text|utils.six' -- '*.py' | grep -v yml

This command returns no results, confirming all deprecated imports have been replaced.
Notes on Python 2.7 compatibility:
For modules that may run on Python 2.7 managed hosts (e.g., authorized_key.py, synchronize.py, sysctl.py), Python 2/3 compatible fallbacks were used instead of direct Python 3 replacements:

authorized_key.py: try/except ImportError for urllib.parse.urlparse (falls back to urlparse on Python 2)
synchronize.py: try/except ImportError for shlex.quote (falls back to pipes.quote on Python 2)
sysctl.py: uses sys.version_info to set string_types to str on Python 3 (basestring on Python 2)

Also removes corresponding pylint:ansible-bad-import-from entries from tests/sanity/ignore-2.21.txt and tests/sanity/ignore-2.22.txt where applicable.
Fixes #686
ISSUE TYPE

Bugfix Pull Request

ADDITIONAL INFORMATION
Approach:
Each file is fixed in a separate commit for easier code review. The changelog fragment is added in a final commit. Corresponding pylint:ansible-bad-import-from ignore entries in tests/sanity/ignore-2.21.txt and tests/sanity/ignore-2.22.txt are removed in the same commit as the file fix (or the file removal commit).
CI results:
All 59 checks passing (Azure Pipelines sanity, units, lint, Docker, Remote across ansible-core 2.17 through devel, and Zuul ansible/check).

Reviewed-by: Felix Fontein <felix@fontein.de>
Reviewed-by: Pavel Bar
Reviewed-by: Abhijeet Kasurde

changelogs/fragments/686_fix_deprecated_imports.yml

@@ -0,0 +1,19 @@
+---
+minor_changes:
+  - acl - fix deprecated ``ansible.module_utils._text`` import (https://github.com/ansible-collections/ansible.posix/issues/686).
+  - authorized_key - fix deprecated ``ansible.module_utils._text`` and ``ansible.module_utils.six`` imports (https://github.com/ansible-collections/ansible.posix/issues/686).
+  - cgroup_perf_recap callback - fix deprecated ``ansible.module_utils._text`` and ``ansible.module_utils.six`` imports (https://github.com/ansible-collections/ansible.posix/issues/686).
+  - csh shell plugin - fix deprecated ``ansible.module_utils.six`` imports (https://github.com/ansible-collections/ansible.posix/issues/686).
+  - firewalld_info - fix deprecated ``ansible.module_utils._text`` import (https://github.com/ansible-collections/ansible.posix/issues/686).
+  - fish shell plugin - fix deprecated ``ansible.module_utils.six`` imports (https://github.com/ansible-collections/ansible.posix/issues/686).
+  - json callback - fix deprecated ``ansible.module_utils._text`` import (https://github.com/ansible-collections/ansible.posix/issues/686).
+  - jsonl callback - fix deprecated ``ansible.module_utils._text`` import (https://github.com/ansible-collections/ansible.posix/issues/686).
+  - mount - fix deprecated ``ansible.module_utils._text`` and ``ansible.module_utils.six`` imports (https://github.com/ansible-collections/ansible.posix/issues/686).
+  - patch - fix deprecated ``ansible.module_utils._text`` import (https://github.com/ansible-collections/ansible.posix/issues/686).
+  - profile_roles callback - fix deprecated ``ansible.module_utils.six`` import (https://github.com/ansible-collections/ansible.posix/issues/686).
+  - profile_tasks callback - fix deprecated ``ansible.module_utils.six`` import (https://github.com/ansible-collections/ansible.posix/issues/686).
+  - rhel_rpm_ostree - fix deprecated ``ansible.module_utils._text`` import (https://github.com/ansible-collections/ansible.posix/issues/686).
+  - rpm_ostree_upgrade - fix deprecated ``ansible.module_utils._text`` import (https://github.com/ansible-collections/ansible.posix/issues/686).
+  - seboolean - fix deprecated ``ansible.module_utils._text`` import (https://github.com/ansible-collections/ansible.posix/issues/686).
+  - synchronize - fix deprecated ``ansible.module_utils._text``, ``ansible.module_utils.common._collections_compat``, and ``ansible.module_utils.six`` imports (https://github.com/ansible-collections/ansible.posix/issues/686).
+  - sysctl - fix deprecated ``ansible.module_utils._text`` and ``ansible.module_utils.six`` imports (https://github.com/ansible-collections/ansible.posix/issues/686).

plugins/action/patch.py

@@ -21,7 +21,7 @@ __metaclass__ = type
 import os
 
 from ansible.errors import AnsibleError, AnsibleAction, _AnsibleActionDone, AnsibleActionFail
-from ansible.module_utils._text import to_native
+from ansible.module_utils.common.text.converters import to_native
 from ansible.module_utils.parsing.convert_bool import boolean
 from ansible.plugins.action import ActionBase
 

plugins/action/synchronize.py

@@ -18,12 +18,11 @@ from __future__ import (absolute_import, division, print_function)
 __metaclass__ = type
 
 import os.path
+from collections.abc import MutableSequence
+from shlex import quote as shlex_quote
 
 from ansible import constants as C
-from ansible.module_utils.six import string_types
+from ansible.module_utils.common.text.converters import to_text
-from ansible.module_utils.six.moves import shlex_quote
-from ansible.module_utils._text import to_text
-from ansible.module_utils.common._collections_compat import MutableSequence
 from ansible.module_utils.parsing.convert_bool import boolean
 from ansible.plugins.action import ActionBase
 from ansible.plugins.loader import connection_loader
@@ -417,7 +416,7 @@ class ActionModule(ActionBase):
             # Replicate what we do in the module argumentspec handling for lists
             if not isinstance(_tmp_args.get('rsync_opts'), MutableSequence):
                 tmp_rsync_opts = _tmp_args.get('rsync_opts', [])
-                if isinstance(tmp_rsync_opts, string_types):
+                if isinstance(tmp_rsync_opts, str):
                     tmp_rsync_opts = tmp_rsync_opts.split(',')
                 elif isinstance(tmp_rsync_opts, (int, float)):
                     tmp_rsync_opts = [to_text(tmp_rsync_opts)]

plugins/callback/cgroup_perf_recap.py

@@ -141,8 +141,7 @@ from abc import ABCMeta, abstractmethod
 
 from functools import partial
 
-from ansible.module_utils._text import to_bytes, to_text
+from ansible.module_utils.common.text.converters import to_bytes, to_text
-from ansible.module_utils.six import with_metaclass
 from ansible.parsing.ajson import AnsibleJSONEncoder
 from ansible.plugins.callback import CallbackBase
 
@@ -158,7 +157,7 @@ def dict_fromkeys(keys, default=None):
     return d
 
 
-class BaseProf(with_metaclass(ABCMeta, threading.Thread)):
+class BaseProf(threading.Thread, metaclass=ABCMeta):
     def __init__(self, path, obj=None, writer=None):
         threading.Thread.__init__(self)  # pylint: disable=non-parent-init-called
         self.obj = obj

plugins/callback/json.py

@@ -48,7 +48,7 @@ import json
 from functools import partial
 
 from ansible.inventory.host import Host
-from ansible.module_utils._text import to_text
+from ansible.module_utils.common.text.converters import to_text
 from ansible.parsing.ajson import AnsibleJSONEncoder
 from ansible.plugins.callback import CallbackBase
 

plugins/callback/jsonl.py

@@ -50,7 +50,7 @@ import copy
 from functools import partial
 
 from ansible.inventory.host import Host
-from ansible.module_utils._text import to_text
+from ansible.module_utils.common.text.converters import to_text
 from ansible.parsing.ajson import AnsibleJSONEncoder
 from ansible.plugins.callback import CallbackBase
 

plugins/callback/profile_roles.py

@@ -33,7 +33,7 @@ import collections
 import time
 
 from ansible.plugins.callback import CallbackBase
-from ansible.module_utils.six.moves import reduce
+from functools import reduce
 
 # define start time
 t0 = tn = time.time()

plugins/callback/profile_tasks.py

@@ -86,7 +86,7 @@ import collections
 
 from datetime import datetime
 
-from ansible.module_utils.six.moves import reduce
+from functools import reduce
 from ansible.plugins.callback import CallbackBase
 
 

plugins/modules/acl.py

@@ -147,7 +147,7 @@ import os
 import platform
 
 from ansible.module_utils.basic import AnsibleModule
-from ansible.module_utils._text import to_native
+from ansible.module_utils.common.text.converters import to_native
 
 
 def split_entry(entry):

plugins/modules/authorized_key.py

@@ -229,10 +229,17 @@ import errno
 import traceback
 from operator import itemgetter
 
-from ansible.module_utils._text import to_native
+# TODO(Python2): urllib.parse is available in Python 3. This module may run on
+# target hosts with Python 2.7 (e.g., older RHEL systems in CI integration tests).
+# Remove the try/except fallback to urlparse when Python 2 support is dropped.
+try:
+    from urllib.parse import urlparse
+except ImportError:
+    from urlparse import urlparse
+
 from ansible.module_utils.basic import AnsibleModule
+from ansible.module_utils.common.text.converters import to_native
 from ansible.module_utils.urls import fetch_url
-from ansible.module_utils.six.moves.urllib.parse import urlparse
 
 
 class keydict(dict):

plugins/modules/firewalld_info.py

@@ -210,7 +210,7 @@ firewalld_info:
 '''
 
 from ansible.module_utils.basic import AnsibleModule, missing_required_lib
-from ansible.module_utils._text import to_native
+from ansible.module_utils.common.text.converters import to_native
 from ansible_collections.ansible.posix.plugins.module_utils._respawn import respawn_module, HAS_RESPAWN_UTIL
 from ansible_collections.ansible.posix.plugins.module_utils.version import StrictVersion
 

plugins/modules/mount.py

@@ -225,8 +225,7 @@ import platform
 
 from ansible.module_utils.basic import AnsibleModule
 from ansible_collections.ansible.posix.plugins.module_utils.mount import ismount
-from ansible.module_utils.six import iteritems
+from ansible.module_utils.common.text.converters import to_bytes, to_native
-from ansible.module_utils._text import to_bytes, to_native
 from ansible.module_utils.parsing.convert_bool import boolean
 
 
@@ -279,7 +278,7 @@ def _set_mount_save_old(module, args):
     old_lines = []
     exists = False
     changed = False
-    escaped_args = dict([(k, _escape_fstab(v)) for k, v in iteritems(args)])
+    escaped_args = dict([(k, _escape_fstab(v)) for k, v in args.items()])
     new_line = '%(src)s %(name)s %(fstype)s %(opts)s %(dump)s %(passno)s\n'
 
     if platform.system() == 'SunOS':

plugins/modules/patch.py

@@ -106,7 +106,7 @@ import os
 import platform
 from traceback import format_exc
 from ansible.module_utils.basic import AnsibleModule
-from ansible.module_utils._text import to_native
+from ansible.module_utils.common.text.converters import to_native
 
 
 class PatchError(Exception):

plugins/modules/rhel_rpm_ostree.py

@@ -73,7 +73,7 @@ import os
 import traceback
 
 from ansible.module_utils.basic import AnsibleModule
-from ansible.module_utils._text import to_text
+from ansible.module_utils.common.text.converters import to_text
 
 
 def locally_installed(module, pkgname):

plugins/modules/rpm_ostree_upgrade.py

@@ -71,7 +71,7 @@ import os
 import traceback
 
 from ansible.module_utils.basic import AnsibleModule
-from ansible.module_utils._text import to_native, to_text
+from ansible.module_utils.common.text.converters import to_native, to_text
 
 
 def rpm_ostree_transaction(module):

plugins/modules/seboolean.py

@@ -73,7 +73,7 @@ except ImportError:
     HAVE_SEMANAGE = False
 
 from ansible.module_utils.basic import AnsibleModule, missing_required_lib
-from ansible.module_utils._text import to_text
+from ansible.module_utils.common.text.converters import to_text
 from ansible_collections.ansible.posix.plugins.module_utils._respawn import respawn_module, HAS_RESPAWN_UTIL
 
 

plugins/modules/synchronize.py

@@ -367,9 +367,16 @@ EXAMPLES = r'''
 import os
 import errno
 
+# TODO(Python2): shlex.quote was added in Python 3.3. This module may run on
+# target hosts with Python 2.7 (e.g., older RHEL systems in CI integration tests).
+# Remove the try/except fallback to pipes.quote when Python 2 support is dropped.
+try:
+    from shlex import quote as shlex_quote
+except ImportError:
+    from pipes import quote as shlex_quote
+
 from ansible.module_utils.basic import AnsibleModule
-from ansible.module_utils._text import to_bytes, to_native
+from ansible.module_utils.common.text.converters import to_bytes, to_native
-from ansible.module_utils.six.moves import shlex_quote
 
 
 client_addr = None

plugins/modules/sysctl.py

@@ -107,12 +107,20 @@ EXAMPLES = r'''
 import os
 import platform
 import re
+import sys
 import tempfile
 
+# TODO(Python2): On Python 2, string_types is basestring (str + unicode).
+# This module may run on target hosts with Python 2.7.
+# Remove the Python 2 branch when Python 2 support is dropped.
+if sys.version_info >= (3, 0):
+    string_types = str
+else:
+    string_types = basestring  # pylint: disable=undefined-variable
+
 from ansible.module_utils.basic import AnsibleModule
-from ansible.module_utils.six import string_types
+from ansible.module_utils.common.text.converters import to_native
 from ansible.module_utils.parsing.convert_bool import BOOLEANS_FALSE, BOOLEANS_TRUE
-from ansible.module_utils._text import to_native
 
 
 class SysctlModule(object):

plugins/shell/csh.py

@@ -13,8 +13,7 @@ DOCUMENTATION = '''
       - shell_common
 '''
 
-from ansible.module_utils.six import text_type
+from shlex import quote as shlex_quote
-from ansible.module_utils.six.moves import shlex_quote
 from ansible.plugins.shell import ShellBase
 
 
@@ -43,5 +42,5 @@ class ShellModule(ShellBase):
         ret = []
         # All the -u options must be first, so we process them first
         ret += ['-u %s' % k for k, v in kwargs.items() if v is None]
-        ret += ['%s=%s' % (k, shlex_quote(text_type(v))) for k, v in kwargs.items() if v is not None]
+        ret += ['%s=%s' % (k, shlex_quote(str(v))) for k, v in kwargs.items() if v is not None]
         return 'env %s' % ' '.join(ret)

plugins/shell/fish.py

@@ -13,8 +13,7 @@ DOCUMENTATION = '''
       - shell_common
 '''
 
-from ansible.module_utils.six import text_type
+from shlex import quote as shlex_quote
-from ansible.module_utils.six.moves import shlex_quote
 from ansible.plugins.shell.sh import ShellModule as ShModule
 
 
@@ -42,7 +41,7 @@ class ShellModule(ShModule):
             if v is None:
                 ret.append('set -e %s;' % k)
             else:
-                ret.append('set -lx %s %s;' % (k, shlex_quote(text_type(v))))
+                ret.append('set -lx %s %s;' % (k, shlex_quote(str(v))))
         return ' '.join(ret)
 
     def build_module_command(self, env_string, shebang, cmd, arg_path=None):

tests/sanity/ignore-2.21.txt

@@ -1,9 +0,0 @@
-plugins/action/synchronize.py pylint:ansible-bad-import-from
-plugins/callback/cgroup_perf_recap.py pylint:ansible-bad-import-from
-plugins/modules/mount.py pylint:ansible-bad-import-from
-plugins/modules/sysctl.py pylint:ansible-bad-import-from
-plugins/shell/csh.py pylint:ansible-bad-import-from
-plugins/shell/fish.py pylint:ansible-bad-import-from
-tests/unit/mock/procenv.py pylint:ansible-bad-import-from
-tests/unit/mock/yaml_helper.py pylint:ansible-bad-import-from
-tests/unit/modules/conftest.py pylint:ansible-bad-import-from

tests/sanity/ignore-2.22.txt

@@ -1,9 +0,0 @@
-plugins/action/synchronize.py pylint:ansible-bad-import-from
-plugins/callback/cgroup_perf_recap.py pylint:ansible-bad-import-from
-plugins/modules/mount.py pylint:ansible-bad-import-from
-plugins/modules/sysctl.py pylint:ansible-bad-import-from
-plugins/shell/csh.py pylint:ansible-bad-import-from
-plugins/shell/fish.py pylint:ansible-bad-import-from
-tests/unit/mock/procenv.py pylint:ansible-bad-import-from
-tests/unit/mock/yaml_helper.py pylint:ansible-bad-import-from
-tests/unit/modules/conftest.py pylint:ansible-bad-import-from

tests/unit/compat/builtins.py

@@ -1,34 +0,0 @@
-# (c) 2014, Toshio Kuratomi <tkuratomi@ansible.com>
-#
-# This file is part of Ansible
-#
-# Ansible is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# Ansible is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
-
-# Make coding more python3-ish
-from __future__ import (absolute_import, division, print_function)
-__metaclass__ = type
-
-#
-# Compat for python2.7
-#
-
-# One unittest needs to import builtins via __import__() so we need to have
-# the string that represents it
-try:
-    import __builtin__
-except ImportError:
-    BUILTINS = 'builtins'
-else:
-    BUILTINS = '__builtin__'
-    __all__ = ['__builtin__']

tests/unit/mock/loader.py

@@ -1,116 +0,0 @@
-# (c) 2012-2014, Michael DeHaan <michael.dehaan@gmail.com>
-#
-# This file is part of Ansible
-#
-# Ansible is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# Ansible is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
-
-# Make coding more python3-ish
-from __future__ import (absolute_import, division, print_function)
-__metaclass__ = type
-
-import os
-
-from ansible.errors import AnsibleParserError
-from ansible.parsing.dataloader import DataLoader
-from ansible.module_utils._text import to_bytes, to_text
-
-
-class DictDataLoader(DataLoader):
-
-    def __init__(self, file_mapping=None):
-        file_mapping = {} if file_mapping is None else file_mapping
-        assert isinstance(file_mapping, dict)
-
-        super(DictDataLoader, self).__init__()
-
-        self._file_mapping = file_mapping
-        self._build_known_directories()
-        self._vault_secrets = None
-
-    def load_from_file(self, path, cache=True, unsafe=False):
-        path = to_text(path)
-        if path in self._file_mapping:
-            return self.load(self._file_mapping[path], path)
-        return None
-
-    # TODO: the real _get_file_contents returns a bytestring, so we actually convert the
-    #       unicode/text it's created with to utf-8
-    def _get_file_contents(self, file_name):
-        path = to_text(file_name)
-        if path in self._file_mapping:
-            return (to_bytes(self._file_mapping[path]), False)
-        else:
-            raise AnsibleParserError("file not found: %s" % path)
-
-    def path_exists(self, path):
-        path = to_text(path)
-        return path in self._file_mapping or path in self._known_directories
-
-    def is_file(self, path):
-        path = to_text(path)
-        return path in self._file_mapping
-
-    def is_directory(self, path):
-        path = to_text(path)
-        return path in self._known_directories
-
-    def list_directory(self, path):
-        ret = []
-        path = to_text(path)
-        for x in (list(self._file_mapping.keys()) + self._known_directories):
-            if x.startswith(path):
-                if os.path.dirname(x) == path:
-                    ret.append(os.path.basename(x))
-        return ret
-
-    def is_executable(self, path):
-        # FIXME: figure out a way to make paths return true for this
-        return False
-
-    def _add_known_directory(self, directory):
-        if directory not in self._known_directories:
-            self._known_directories.append(directory)
-
-    def _build_known_directories(self):
-        self._known_directories = []
-        for path in self._file_mapping:
-            dirname = os.path.dirname(path)
-            while dirname not in ('/', ''):
-                self._add_known_directory(dirname)
-                dirname = os.path.dirname(dirname)
-
-    def push(self, path, content):
-        rebuild_dirs = False
-        if path not in self._file_mapping:
-            rebuild_dirs = True
-
-        self._file_mapping[path] = content
-
-        if rebuild_dirs:
-            self._build_known_directories()
-
-    def pop(self, path):
-        if path in self._file_mapping:
-            del self._file_mapping[path]
-            self._build_known_directories()
-
-    def clear(self):
-        self._file_mapping = dict()
-        self._known_directories = []
-
-    def get_basedir(self):
-        return os.getcwd()
-
-    def set_vault_secrets(self, vault_secrets):
-        self._vault_secrets = vault_secrets

tests/unit/mock/path.py

@@ -1,9 +0,0 @@
-from __future__ import absolute_import, division, print_function
-
-__metaclass__ = type
-
-from ansible_collections.ansible.posix.tests.unit.compat.mock import MagicMock
-from ansible.utils.path import unfrackpath
-
-
-mock_unfrackpath_noop = MagicMock(spec_set=unfrackpath, side_effect=lambda x, *args, **kwargs: x)

tests/unit/mock/procenv.py

@@ -1,90 +0,0 @@
-# (c) 2016, Matt Davis <mdavis@ansible.com>
-# (c) 2016, Toshio Kuratomi <tkuratomi@ansible.com>
-#
-# This file is part of Ansible
-#
-# Ansible is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# Ansible is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
-
-# Make coding more python3-ish
-from __future__ import (absolute_import, division, print_function)
-__metaclass__ = type
-
-import sys
-import json
-
-from contextlib import contextmanager
-from io import BytesIO, StringIO
-from ansible_collections.ansible.posix.tests.unit.compat import unittest
-from ansible.module_utils.six import PY3
-from ansible.module_utils._text import to_bytes
-
-
-@contextmanager
-def swap_stdin_and_argv(stdin_data='', argv_data=tuple()):
-    """
-    context manager that temporarily masks the test runner's values for stdin and argv
-    """
-    real_stdin = sys.stdin
-    real_argv = sys.argv
-
-    if PY3:
-        fake_stream = StringIO(stdin_data)
-        fake_stream.buffer = BytesIO(to_bytes(stdin_data))
-    else:
-        fake_stream = BytesIO(to_bytes(stdin_data))
-
-    try:
-        sys.stdin = fake_stream
-        sys.argv = argv_data
-
-        yield
-    finally:
-        sys.stdin = real_stdin
-        sys.argv = real_argv
-
-
-@contextmanager
-def swap_stdout():
-    """
-    context manager that temporarily replaces stdout for tests that need to verify output
-    """
-    old_stdout = sys.stdout
-
-    if PY3:
-        fake_stream = StringIO()
-    else:
-        fake_stream = BytesIO()
-
-    try:
-        sys.stdout = fake_stream
-
-        yield fake_stream
-    finally:
-        sys.stdout = old_stdout
-
-
-class ModuleTestCase(unittest.TestCase):
-    def setUp(self, module_args=None):
-        if module_args is None:
-            module_args = {'_ansible_remote_tmp': '/tmp', '_ansible_keep_remote_files': False}
-
-        args = json.dumps(dict(ANSIBLE_MODULE_ARGS=module_args))
-
-        # unittest doesn't have a clean place to use a context manager, so we have to enter/exit manually
-        self.stdin_swap = swap_stdin_and_argv(stdin_data=args)
-        self.stdin_swap.__enter__()
-
-    def tearDown(self):
-        # unittest doesn't have a clean place to use a context manager, so we have to enter/exit manually
-        self.stdin_swap.__exit__(None, None, None)

tests/unit/mock/vault_helper.py

@@ -1,39 +0,0 @@
-# Ansible is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# Ansible is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
-
-# Make coding more python3-ish
-from __future__ import (absolute_import, division, print_function)
-__metaclass__ = type
-
-from ansible.module_utils._text import to_bytes
-
-from ansible.parsing.vault import VaultSecret
-
-
-class TextVaultSecret(VaultSecret):
-    '''A secret piece of text. ie, a password. Tracks text encoding.
-
-    The text encoding of the text may not be the default text encoding so
-    we keep track of the encoding so we encode it to the same bytes.'''
-
-    def __init__(self, text, encoding=None, errors=None, _bytes=None):
-        super(TextVaultSecret, self).__init__()
-        self.text = text
-        self.encoding = encoding or 'utf-8'
-        self._bytes = _bytes
-        self.errors = errors or 'strict'
-
-    @property
-    def bytes(self):
-        '''The text encoded with encoding, unless we specifically set _bytes.'''
-        return self._bytes or to_bytes(self.text, encoding=self.encoding, errors=self.errors)

tests/unit/mock/yaml_helper.py

@@ -1,125 +0,0 @@
-from __future__ import absolute_import, division, print_function
-
-__metaclass__ = type
-
-import io
-import yaml
-
-from ansible.module_utils.six import PY3
-from ansible.parsing.yaml.loader import AnsibleLoader
-from ansible.parsing.yaml.dumper import AnsibleDumper
-
-
-class YamlTestUtils(object):
-    """Mixin class to combine with a unittest.TestCase subclass."""
-    def _loader(self, stream):
-        """Vault related tests will want to override this.
-
-        Vault cases should setup a AnsibleLoader that has the vault password."""
-        return AnsibleLoader(stream)
-
-    def _dump_stream(self, obj, stream, dumper=None):
-        """Dump to a py2-unicode or py3-string stream."""
-        if PY3:
-            return yaml.dump(obj, stream, Dumper=dumper)
-        else:
-            return yaml.dump(obj, stream, Dumper=dumper, encoding=None)
-
-    def _dump_string(self, obj, dumper=None):
-        """Dump to a py2-unicode or py3-string"""
-        if PY3:
-            return yaml.dump(obj, Dumper=dumper)
-        else:
-            return yaml.dump(obj, Dumper=dumper, encoding=None)
-
-    def _dump_load_cycle(self, obj):
-        # Each pass though a dump or load revs the 'generation'
-        # obj to yaml string
-        string_from_object_dump = self._dump_string(obj, dumper=AnsibleDumper)
-
-        # wrap a stream/file like StringIO around that yaml
-        stream_from_object_dump = io.StringIO(string_from_object_dump)
-        loader = self._loader(stream_from_object_dump)
-        # load the yaml stream to create a new instance of the object (gen 2)
-        obj_2 = loader.get_data()
-
-        # dump the gen 2 objects directory to strings
-        string_from_object_dump_2 = self._dump_string(obj_2,
-                                                      dumper=AnsibleDumper)
-
-        # The gen 1 and gen 2 yaml strings
-        self.assertEqual(string_from_object_dump, string_from_object_dump_2)
-        # the gen 1 (orig) and gen 2 py object
-        self.assertEqual(obj, obj_2)
-
-        # again! gen 3... load strings into py objects
-        stream_3 = io.StringIO(string_from_object_dump_2)
-        loader_3 = self._loader(stream_3)
-        obj_3 = loader_3.get_data()
-
-        string_from_object_dump_3 = self._dump_string(obj_3, dumper=AnsibleDumper)
-
-        self.assertEqual(obj, obj_3)
-        # should be transitive, but...
-        self.assertEqual(obj_2, obj_3)
-        self.assertEqual(string_from_object_dump, string_from_object_dump_3)
-
-    def _old_dump_load_cycle(self, obj):
-        '''Dump the passed in object to yaml, load it back up, dump again, compare.'''
-        stream = io.StringIO()
-
-        yaml_string = self._dump_string(obj, dumper=AnsibleDumper)
-        self._dump_stream(obj, stream, dumper=AnsibleDumper)
-
-        yaml_string_from_stream = stream.getvalue()
-
-        # reset stream
-        stream.seek(0)
-
-        loader = self._loader(stream)
-        # loader = AnsibleLoader(stream, vault_password=self.vault_password)
-        obj_from_stream = loader.get_data()
-
-        stream_from_string = io.StringIO(yaml_string)
-        loader2 = self._loader(stream_from_string)
-        # loader2 = AnsibleLoader(stream_from_string, vault_password=self.vault_password)
-        obj_from_string = loader2.get_data()
-
-        stream_obj_from_stream = io.StringIO()
-        stream_obj_from_string = io.StringIO()
-
-        if PY3:
-            yaml.dump(obj_from_stream, stream_obj_from_stream, Dumper=AnsibleDumper)
-            yaml.dump(obj_from_stream, stream_obj_from_string, Dumper=AnsibleDumper)
-        else:
-            yaml.dump(obj_from_stream, stream_obj_from_stream, Dumper=AnsibleDumper, encoding=None)
-            yaml.dump(obj_from_stream, stream_obj_from_string, Dumper=AnsibleDumper, encoding=None)
-
-        yaml_string_stream_obj_from_stream = stream_obj_from_stream.getvalue()
-        yaml_string_stream_obj_from_string = stream_obj_from_string.getvalue()
-
-        stream_obj_from_stream.seek(0)
-        stream_obj_from_string.seek(0)
-
-        if PY3:
-            yaml_string_obj_from_stream = yaml.dump(obj_from_stream, Dumper=AnsibleDumper)
-            yaml_string_obj_from_string = yaml.dump(obj_from_string, Dumper=AnsibleDumper)
-        else:
-            yaml_string_obj_from_stream = yaml.dump(obj_from_stream, Dumper=AnsibleDumper, encoding=None)
-            yaml_string_obj_from_string = yaml.dump(obj_from_string, Dumper=AnsibleDumper, encoding=None)
-
-        assert yaml_string == yaml_string_obj_from_stream
-        assert yaml_string == yaml_string_obj_from_stream == yaml_string_obj_from_string
-        assert (yaml_string == yaml_string_obj_from_stream == yaml_string_obj_from_string == yaml_string_stream_obj_from_stream ==
-                yaml_string_stream_obj_from_string)
-        assert obj == obj_from_stream
-        assert obj == obj_from_string
-        assert obj == yaml_string_obj_from_stream
-        assert obj == yaml_string_obj_from_string
-        assert obj == obj_from_stream == obj_from_string == yaml_string_obj_from_stream == yaml_string_obj_from_string
-        return {'obj': obj,
-                'yaml_string': yaml_string,
-                'yaml_string_from_stream': yaml_string_from_stream,
-                'obj_from_stream': obj_from_stream,
-                'obj_from_string': obj_from_string,
-                'yaml_string_obj_from_string': yaml_string_obj_from_string}

tests/unit/modules/conftest.py

@@ -1,32 +0,0 @@
-# Copyright (c) 2017 Ansible Project
-# GNU General Public License v3.0+ (see COPYING or https://www.gnu.org/licenses/gpl-3.0.txt)
-
-from __future__ import absolute_import, division, print_function
-
-__metaclass__ = type
-
-import json
-
-import pytest
-
-from ansible.module_utils.six import string_types
-from ansible.module_utils._text import to_bytes
-from ansible.module_utils.common._collections_compat import MutableMapping
-
-
-@pytest.fixture
-def patch_ansible_module(request, mocker):
-    if isinstance(request.param, string_types):
-        args = request.param
-    elif isinstance(request.param, MutableMapping):
-        if 'ANSIBLE_MODULE_ARGS' not in request.param:
-            request.param = {'ANSIBLE_MODULE_ARGS': request.param}
-        if '_ansible_remote_tmp' not in request.param['ANSIBLE_MODULE_ARGS']:
-            request.param['ANSIBLE_MODULE_ARGS']['_ansible_remote_tmp'] = '/tmp'
-        if '_ansible_keep_remote_files' not in request.param['ANSIBLE_MODULE_ARGS']:
-            request.param['ANSIBLE_MODULE_ARGS']['_ansible_keep_remote_files'] = False
-        args = json.dumps(request.param)
-    else:
-        raise Exception('Malformed data to the patch_ansible_module pytest fixture')
-
-    mocker.patch('ansible.module_utils.basic._ANSIBLE_ARGS', to_bytes(args))

tests/unit/modules/system/test_mount.py

@@ -7,7 +7,7 @@ import tempfile
 
 from ansible_collections.ansible.posix.tests.unit.compat import unittest
 from ansible_collections.ansible.posix.tests.unit.compat.mock import MagicMock
-from ansible.module_utils._text import to_bytes
+from ansible.module_utils.common.text.converters import to_bytes
 
 from ansible_collections.ansible.posix.plugins.modules.mount import (
     get_linux_mounts,

tests/unit/modules/utils.py

@@ -1,51 +0,0 @@
-from __future__ import absolute_import, division, print_function
-
-__metaclass__ = type
-
-import json
-
-from ansible_collections.ansible.posix.tests.unit.compat import unittest
-from ansible_collections.ansible.posix.tests.unit.compat.mock import patch
-from ansible.module_utils import basic
-from ansible.module_utils._text import to_bytes
-
-
-def set_module_args(args):
-    if '_ansible_remote_tmp' not in args:
-        args['_ansible_remote_tmp'] = '/tmp'
-    if '_ansible_keep_remote_files' not in args:
-        args['_ansible_keep_remote_files'] = False
-
-    args = json.dumps({'ANSIBLE_MODULE_ARGS': args})
-    basic._ANSIBLE_ARGS = to_bytes(args)
-
-
-class AnsibleExitJson(Exception):
-    pass
-
-
-class AnsibleFailJson(Exception):
-    pass
-
-
-def exit_json(*args, **kwargs):
-    if 'changed' not in kwargs:
-        kwargs['changed'] = False
-    raise AnsibleExitJson(kwargs)
-
-
-def fail_json(*args, **kwargs):
-    kwargs['failed'] = True
-    raise AnsibleFailJson(kwargs)
-
-
-class ModuleTestCase(unittest.TestCase):
-
-    def setUp(self):
-        self.mock_module = patch.multiple(basic.AnsibleModule, exit_json=exit_json, fail_json=fail_json)
-        self.mock_module.start()
-        self.mock_sleep = patch('time.sleep')
-        self.mock_sleep.start()
-        set_module_args({})
-        self.addCleanup(self.mock_module.stop)
-        self.addCleanup(self.mock_sleep.stop)