Fix regression in jinja2 include search path (#18617)

* Fix regression in jinja2 include search path

Since commit 3c39bb5, the 'ansible_search_path' variable is used to set
jinja2's search path for {% include %} directives. However, this path is
the the proper one because our templates live in 'templates' subdirs in
our search path.

This is a regression because previously, our include search path would
include the dirname of the currently interpreted file, which worked most
of the time.

fixes #18526

* Fix template lookup search path

Improve fix in commit c96c853 so that the search path contain both
template-suffixed paths as well as original paths.

ref PR #18617

* Add integration test for template lookups

Tests regression at #18526

This test fails on current devel branch and succeeds on PR #18617

lib/ansible/plugins/lookup/template.py

@@ -49,6 +49,14 @@ class LookupModule(LookupBase):
                     # set jinja2 internal search path for includes
                     if 'ansible_search_path' in variables:
                         searchpath = variables['ansible_search_path']
+                        # our search paths aren't actually the proper ones for jinja includes.
+                        # We want to search into the 'templates' subdir of each search path in
+                        # addition to our original search paths.
+                        newsearchpath = []
+                        for p in searchpath:
+                            newsearchpath.append(os.path.join(p, 'templates'))
+                            newsearchpath.append(p)
+                        searchpath = newsearchpath
                     else:
                         searchpath = [self._loader._basedir, os.path.dirname(lookupfile)]
                     self._templar.environment.loader.searchpath = searchpath

test/integration/targets/lookups/tasks/main.yml

@@ -268,3 +268,14 @@
         - product.results[6]['item'] == ["C", "1"]
         - product.results[7]['item'] == ["C", "2"]
         - product.results[8]['item'] == ["C", "3"]
+
+# Template lookups
+
+# ref #18526
+- name: Test that we have a proper jinja search path in template lookup
+  set_fact:
+    hello_world: "{{ lookup('template', 'hello.txt') }}"
+
+- assert:
+    that:
+      - "hello_world|trim == 'Hello world!'"

test/integration/targets/lookups/templates/hello.txt

@@ -0,0 +1 @@
+Hello {% include 'world.txt' %}!

test/integration/targets/lookups/templates/world.txt

@@ -0,0 +1 @@
+world