Become plugins (#50991)

* [WIP] become plugins

Move from hardcoded method to plugins for ease of use, expansion and overrides
  - load into connection as it is going to be the main consumer
  - play_context will also use to keep backwards compat API
  - ensure shell is used to construct commands when needed
  - migrate settings remove from base config in favor of plugin specific configs
  - cleanup ansible-doc
  - add become plugin docs
  - remove deprecated sudo/su code and keywords
  - adjust become options for cli
  - set plugin options from context
  - ensure config defs are avaialbe before instance
  - refactored getting the shell plugin, fixed tests
     - changed into regex as they were string matching, which does not work with random string generation
     - explicitly set flags for play context tests
 - moved plugin loading up front
 - now loads for basedir also
 - allow pyc/o for non m modules
 - fixes to tests and some plugins
 - migrate to play objects fro play_context
 - simiplify gathering
 -  added utf8 headers
 - moved option setting
 - add fail msg to dzdo
 - use tuple for multiple options on fail/missing
 - fix relative plugin paths
 - shift from play context to play
 - all tasks already inherit this from play directly
 - remove obsolete 'set play'
 - correct environment handling
 - add wrap_exe option to pfexec
 - fix runas to noop
 - fixed setting play context
 - added password configs
 - removed required false
 - remove from doc building till they are ready

future development:
  - deal with 'enable' and 'runas' which are not 'command wrappers' but 'state flags' and currently hardcoded in diff subsystems

* cleanup

  remove callers to removed func
  removed --sudo cli doc refs
  remove runas become_exe
  ensure keyerorr on plugin
  also fix backwards compat, missing method is attributeerror, not ansible error
  get remote_user consistently
  ignore missing system_tmpdirs on plugin load
  correct config precedence
  add deprecation
  fix networking imports
  backwards compat for plugins using BECOME_METHODS

* Port become_plugins to context.CLIARGS

This is a work in progress:
* Stop passing options around everywhere as we can use context.CLIARGS
  instead

* Refactor make_become_commands as asked for by alikins

* Typo in comment fix

* Stop loading values from the cli in more than one place

Both play and play_context were saving default values from the cli
arguments directly.  This changes things so that the default values are
loaded into the play and then play_context takes them from there.

* Rename BECOME_PLUGIN_PATH to DEFAULT_BECOME_PLUGIN_PATH

As alikins said, all other plugin paths are named
DEFAULT_plugintype_PLUGIN_PATH.  If we're going to rename these, that
should be done all at one time rather than piecemeal.

* One to throw away

This is a set of hacks to get setting FieldAttribute defaults to command
line args to work.  It's not fully done yet.

After talking it over with sivel and jimi-c this should be done by
fixing FieldAttributeBase and _get_parent_attribute() calls to do the
right thing when there is a non-None default.

What we want to be able to do ideally is something like this:

class Base(FieldAttributeBase):
    _check_mode = FieldAttribute([..] default=lambda: context.CLIARGS['check'])

class Play(Base):
    # lambda so that we have a chance to parse the command line args
    # before we get here.  In the future we might be able to restructure
    # this so that the cli parsing code runs before these classes are
    # defined.

class Task(Base):
    pass

And still have a playbook like this function:

---
- hosts:
  tasks:
  - command: whoami
    check_mode: True

(The check_mode test that is added as a separate commit in this PR will
let you test variations on this case).

There's a few separate reasons that the code doesn't let us do this or
a non-ugly workaround for this as written right now.  The fix that
jimi-c, sivel, and I talked about may let us do this or it may still
require a workaround (but less ugly) (having one class that has the
FieldAttributes with default values and one class that inherits from
that but just overrides the FieldAttributes which now have defaults)

* Revert "One to throw away"

This reverts commit 23aa883cbed11429ef1be2a2d0ed18f83a3b8064.

* Set FieldAttr defaults directly from CLIARGS

* Remove dead code

* Move timeout directly to PlayContext, it's never needed on Play

* just for backwards compat, add a static version of BECOME_METHODS to constants

* Make the become attr on the connection public, since it's used outside of the connection

* Logic fix

* Nuke connection testing if it supports specific become methods

* Remove unused vars

* Address rebase issues

* Fix path encoding issue

* Remove unused import

* Various cleanups

* Restore network_cli check in _low_level_execute_command

* type improvements for cliargs_deferred_get and swap shallowcopy to default to False

* minor cleanups

* Allow the su plugin to work, since it doesn't define a prompt the same way

* Fix up ksu become plugin

* Only set prompt if build_become_command was called

* Add helper to assist connection plugins in knowing they need to wait for a prompt

* Fix tests and code expectations

* Doc updates

* Various additional minor cleanups

* Make doas functional

* Don't change connection signature, load become plugin from TaskExecutor

* Remove unused imports

* Add comment about setting the become plugin on the playcontext

* Fix up tests for recent changes

* Support 'Password:' natively for the doas plugin

* Make default prompts raw

* wording cleanups. ci_complete

* Remove unrelated changes

* Address spelling mistake

* Restore removed test, and udpate to use new functionality

* Add changelog fragment

* Don't hard fail in set_attributes_from_cli on missing CLI keys

* Remove unrelated change to loader

* Remove internal deprecated FieldAttributes now

* Emit deprecation warnings now

test/units/plugins/action/test_action.py

@@ -500,31 +500,31 @@ class TestActionBase(unittest.TestCase):
         fake_loader = MagicMock()
         fake_loader.get_basedir.return_value = os.getcwd()
         play_context = PlayContext()
-        action_base = DerivedActionBase(None, None, play_context, fake_loader, None, None)
-        action_base._connection = MagicMock(exec_command=MagicMock(return_value=(0, '', '')))
-        action_base._connection._shell = MagicMock(append_command=MagicMock(return_value=('JOINED CMD')))
 
-        play_context.become = True
-        play_context.become_user = play_context.remote_user = 'root'
-        play_context.make_become_cmd = MagicMock(return_value='CMD')
+        action_base = DerivedActionBase(None, None, play_context, fake_loader, None, None)
+        action_base.get_become_option = MagicMock(return_value='root')
+        action_base._get_remote_user = MagicMock(return_value='root')
+
+        action_base._connection = MagicMock(exec_command=MagicMock(return_value=(0, '', '')))
+
+        action_base._connection._shell = shell = MagicMock(append_command=MagicMock(return_value=('JOINED CMD')))
+
+        action_base._connection.become = become = MagicMock()
+        become.build_become_command.return_value = 'foo'
 
         action_base._low_level_execute_command('ECHO', sudoable=True)
-        play_context.make_become_cmd.assert_not_called()
+        become.build_become_command.assert_not_called()
 
-        play_context.remote_user = 'apo'
+        action_base._get_remote_user.return_value = 'apo'
         action_base._low_level_execute_command('ECHO', sudoable=True, executable='/bin/csh')
-        play_context.make_become_cmd.assert_called_once_with("ECHO", executable='/bin/csh')
+        become.build_become_command.assert_called_once_with("ECHO", shell)
 
-        play_context.make_become_cmd.reset_mock()
+        become.build_become_command.reset_mock()
 
-        become_allow_same_user = C.BECOME_ALLOW_SAME_USER
-        C.BECOME_ALLOW_SAME_USER = True
-        try:
-            play_context.remote_user = 'root'
+        with patch.object(C, 'BECOME_ALLOW_SAME_USER', new=True):
+            action_base._get_remote_user.return_value = 'root'
             action_base._low_level_execute_command('ECHO SAME', sudoable=True)
-            play_context.make_become_cmd.assert_called_once_with("ECHO SAME", executable=None)
-        finally:
-            C.BECOME_ALLOW_SAME_USER = become_allow_same_user
+            become.build_become_command.assert_called_once_with("ECHO SAME", shell)
 
 
 class TestActionBaseCleanReturnedData(unittest.TestCase):

test/units/plugins/connection/test_connection.py

@@ -30,6 +30,7 @@ from units.compat.mock import patch
 from ansible.errors import AnsibleError
 from ansible.playbook.play_context import PlayContext
 from ansible.plugins.connection import ConnectionBase
+from ansible.plugins.loader import become_loader
 # from ansible.plugins.connection.accelerate import Connection as AccelerateConnection
 # from ansible.plugins.connection.chroot import Connection as ChrootConnection
 # from ansible.plugins.connection.funcd import Connection as FuncdConnection
@@ -250,6 +251,8 @@ debug1: Sending command: /bin/sh -c 'sudo -H -S  -p "[sudo via ansible, key=ouzm
                 pass
 
         c = ConnectionFoo(self.play_context, self.in_stream)
+        c.set_become_plugin(become_loader.get('sudo'))
+        c.become.prompt = '[sudo via ansible, key=ouzmdnewuhucvuaabtjmweasarviygqq] password: '
 
         self.assertTrue(c.check_password_prompt(local))
         self.assertTrue(c.check_password_prompt(ssh_pipelining_vvvv))

test/units/plugins/connection/test_ssh.py

@@ -34,7 +34,7 @@ from ansible.module_utils.six.moves import shlex_quote
 from ansible.module_utils._text import to_bytes
 from ansible.playbook.play_context import PlayContext
 from ansible.plugins.connection import ssh
-from ansible.plugins.loader import connection_loader
+from ansible.plugins.loader import connection_loader, become_loader
 
 
 class TestConnectionBaseClass(unittest.TestCase):
@@ -93,6 +93,7 @@ class TestConnectionBaseClass(unittest.TestCase):
         new_stdin = StringIO()
 
         conn = connection_loader.get('ssh', pc, new_stdin)
+        conn.set_become_plugin(become_loader.get('sudo'))
 
         conn.check_password_prompt = MagicMock()
         conn.check_become_success = MagicMock()
@@ -133,6 +134,14 @@ class TestConnectionBaseClass(unittest.TestCase):
         )
 
         pc.prompt = True
+        conn.become.prompt = True
+
+        def get_option(option):
+            if option == 'become_pass':
+                return 'password'
+            return None
+
+        conn.become.get_option = get_option
         output, unprocessed = conn._examine_output(u'source', u'state', b'line 1\nline 2\nfoo\nline 3\nthis should be the remainder', False)
         self.assertEqual(output, b'line 1\nline 2\nline 3\n')
         self.assertEqual(unprocessed, b'this should be the remainder')
@@ -150,7 +159,9 @@ class TestConnectionBaseClass(unittest.TestCase):
         )
 
         pc.prompt = False
+        conn.become.prompt = False
         pc.success_key = u'BECOME-SUCCESS-abcdefghijklmnopqrstuvxyz'
+        conn.become.success = u'BECOME-SUCCESS-abcdefghijklmnopqrstuvxyz'
         output, unprocessed = conn._examine_output(u'source', u'state', b'line 1\nline 2\nBECOME-SUCCESS-abcdefghijklmnopqrstuvxyz\nline 3\n', False)
         self.assertEqual(output, b'line 1\nline 2\nline 3\n')
         self.assertEqual(unprocessed, b'')
@@ -168,6 +179,7 @@ class TestConnectionBaseClass(unittest.TestCase):
         )
 
         pc.prompt = False
+        conn.become.prompt = False
         pc.success_key = None
         output, unprocessed = conn._examine_output(u'source', u'state', b'line 1\nline 2\nincorrect password\n', True)
         self.assertEqual(output, b'line 1\nline 2\nincorrect password\n')
@@ -186,6 +198,7 @@ class TestConnectionBaseClass(unittest.TestCase):
         )
 
         pc.prompt = False
+        conn.become.prompt = False
         pc.success_key = None
         output, unprocessed = conn._examine_output(u'source', u'state', b'line 1\nbad password\n', True)
         self.assertEqual(output, b'line 1\nbad password\n')
@@ -332,6 +345,7 @@ def mock_run_env(request, mocker):
     new_stdin = StringIO()
 
     conn = connection_loader.get('ssh', pc, new_stdin)
+    conn.set_become_plugin(become_loader.get('sudo'))
     conn._send_initial_data = MagicMock()
     conn._examine_output = MagicMock()
     conn._terminate_process = MagicMock()
@@ -425,7 +439,7 @@ class TestSSHConnectionRun(object):
     def test_password_with_prompt(self):
         # test with password prompting enabled
         self.pc.password = None
-        self.pc.prompt = b'Password:'
+        self.conn.become.prompt = b'Password:'
         self.conn._examine_output.side_effect = self._password_with_prompt_examine_output
         self.mock_popen_res.stdout.read.side_effect = [b"Password:", b"Success", b""]
         self.mock_popen_res.stderr.read.side_effect = [b""]
@@ -450,8 +464,10 @@ class TestSSHConnectionRun(object):
     def test_password_with_become(self):
         # test with some become settings
         self.pc.prompt = b'Password:'
+        self.conn.become.prompt = b'Password:'
         self.pc.become = True
         self.pc.success_key = 'BECOME-SUCCESS-abcdefg'
+        self.conn.become._id = 'abcdefg'
         self.conn._examine_output.side_effect = self._password_with_prompt_examine_output
         self.mock_popen_res.stdout.read.side_effect = [b"Password:", b"BECOME-SUCCESS-abcdefg", b"abc"]
         self.mock_popen_res.stderr.read.side_effect = [b"123"]