k8s_cp - wait for remote tar to finish before closing the stream (#1217)

* fix(k8s_cp): wait for remote tar to finish before closing the stream

The module closed the exec connection right after writing the last chunk,
killing tar mid-extract, and never checked its exit status, so a partial
or failed copy was reported as success (#776).

The v4 exec protocol cannot half-close stdin, and busybox tar blocks for
EOF rather than stopping at the end-of-archive marker, so bound the stream
with `head -c N | tar ...`. tar then gets a clean EOF, exits on its own,
and its status is read from the error channel. Containers without /bin/sh
and head keep the previous behaviour and get a warning.

Also add copy_timeout (default 300s, #621) and stream the archive from the
temp file instead of buffering every chunk in memory.

* test(k8s_cp): split combined assert into separate statements

Addresses review feedback on #1217.

* chore(changelog): link the PR and use single-line fragment entries

* fix(linter): reformat with black

* fix(linter): sort imports in test_copy.py

isort --profile black puts ansible_collections in the same third-party
section as pytest, so the blank line between them is a failure.

* fix(k8s_cp): report what tar said when the copy fails early

An early close or a stall is the symptom; tar has usually already
written the cause (a read-only mount, a missing directory) to stderr,
and drain() has it. Discarding it left only the generic truncation or
timeout message.

* refactor(k8s_cp): move the stderr suffix into _fail

The nested fail() helper added a branch inside a closure, which counts
double for nesting and took _stream_tar_to_pod over Sonar's cognitive
complexity limit (14 -> 16). _fail already had the response; give it
the stderr too.

* test(k8s_cp): tempfile for the repeated-copy test directory

Co-authored-by: Bikouo Aubin <79859644+abikouo@users.noreply.github.com>

---------

Co-authored-by: Bikouo Aubin <79859644+abikouo@users.noreply.github.com>
This commit is contained in:
Yuriy Novostavskiy
2026-09-15 09:20:10 +01:00
committed by GitHub
co-authored by Bikouo Aubin
parent 031bc475fa
commit 8a9dd54fe7
8 changed files with 568 additions and 56 deletions
@@ -0,0 +1,8 @@
---
bugfixes:
- k8s_cp - Fix silent truncation when copying to a pod. The module closed the exec connection immediately after writing the last chunk of the tar archive, which killed the remote ``tar`` before it had finished extracting, and never checked the process exit status, so a partial copy was reported as a success. The archive is now bounded with ``head -c`` so ``tar`` gets a clean EOF and exits on its own, and the module waits for that exit and fails on a non-zero status (https://github.com/ansible-collections/kubernetes.core/pull/1217).
minor_changes:
- k8s_cp - Add the ``copy_timeout`` option (default 300 seconds) bounding how long the module will spend streaming an archive to a pod and waiting for the remote ``tar`` to finish, so that a stalled copy fails instead of hanging indefinitely (https://github.com/ansible-collections/kubernetes.core/pull/1217).
- k8s_cp - Stream the tar archive to the pod chunk by chunk instead of building the whole archive in memory first, which cuts peak memory use when copying large files (https://github.com/ansible-collections/kubernetes.core/pull/1217).
- k8s_cp - When copying to a pod, the module now also uses ``/bin/sh`` and ``head`` in the container, when present, to confirm the copy completed. Containers without them keep the previous behaviour and get a warning that completion could not be verified (https://github.com/ansible-collections/kubernetes.core/pull/1217).
+178 -53
View File
@@ -18,7 +18,9 @@ from __future__ import absolute_import, division, print_function
__metaclass__ = type
import os
import shlex
import tarfile
import time
from abc import ABCMeta, abstractmethod
from select import select
from tempfile import NamedTemporaryFile, TemporaryFile
@@ -296,14 +298,156 @@ class K8SCopyToPod(K8SCopy):
Copy files/directory from local filesystem into remote Pod
"""
# Size of the chunks the tar archive is written to stdin with.
CHUNK_SIZE = 1024 * 1024
SHELL = "/bin/sh"
def __init__(self, module, client):
super(K8SCopyToPod, self).__init__(module, client)
self.files_to_copy = list()
self.named_temp_file = None
self.copy_timeout = module.params.get("copy_timeout")
def close_temp_file(self):
if self.named_temp_file:
self.named_temp_file.close()
def _fail(self, response, msg, stderr=(), **kwargs):
"""Fail, keeping whatever the remote already wrote to stderr.
An early close or a stall is usually the symptom, not the cause: tar
has typically said why on stderr first.
"""
if stderr:
msg = "{0}: {1}".format(msg, "".join(stderr))
response.close()
self.close_temp_file()
self.module.fail_json(
msg="Failed to copy local file/directory into Pod: {0}".format(msg),
**kwargs,
)
def tar_command(self, dest_file):
if self.no_preserve:
command = [
"tar",
"--no-same-permissions",
"--no-same-owner",
"-xmf",
"-",
]
else:
command = ["tar", "-xmf", "-"]
if dest_file.startswith("/"):
command.extend(["-C", "/"])
return command
def can_bound_stream(self):
"""Whether the container can run ``sh -c 'head -c ...'``."""
error, _out, _err = self._run_from_pod(
cmd=[self.SHELL, "-c", "command -v head"]
)
return (error or {}).get("status") == "Success"
def remote_command(self, dest_file, archive_size):
"""Build the command that extracts the archive inside the container.
The websocket exec protocol the kubernetes client speaks (v4) cannot
half-close stdin, so the only EOF the remote process can get is the
connection going away. Relying on that races with tar finishing and
truncates the file (issue #776), and tar cannot be relied on to stop at
the end-of-archive marker either -- busybox tar blocks for EOF.
Piping through ``head -c`` bounds the stream at the exact archive size,
so tar sees a clean EOF, exits on its own, and the API server reports
its exit status on the error channel. Only where the container has no
shell do we fall back to feeding tar directly, which cannot confirm
that the copy completed.
"""
command = self.tar_command(dest_file)
if not self.can_bound_stream():
self.module.warn(
"Container has no '{0}' with 'head', falling back to writing the"
" archive straight to tar. The module cannot confirm the copy"
" completed; verify the file after copying.".format(self.SHELL)
)
return command, False
return [
self.SHELL,
"-c",
"head -c {0} | {1}".format(
archive_size, " ".join(shlex.quote(arg) for arg in command)
),
], True
def _stream_tar_to_pod(self, response, tar_buffer, wait_for_exit):
"""Feed the archive to the remote extractor and wait for it to finish.
Writing the last chunk only hands the bytes to the local socket; they
still have to cross the API server and reach the container. Closing the
connection at that point kills the extractor mid-write and silently
truncates the file, which is what issue #776 reports.
"""
stdout, stderr = [], []
def drain(timeout):
response.update(timeout=timeout)
if response.peek_stdout():
stdout.append(response.read_stdout().rstrip("\n"))
if response.peek_stderr():
stderr.append(response.read_stderr().rstrip("\n"))
deadline = time.monotonic() + self.copy_timeout
chunk = tar_buffer.read(self.CHUNK_SIZE)
while chunk:
if not response.is_open():
self._fail(
response,
"connection to Pod {0}/{1} closed before the whole archive was"
" sent, the remote file is incomplete".format(
self.namespace, self.name
),
stderr,
)
if time.monotonic() > deadline:
self._fail(
response,
"timed out after {0}s sending the archive to Pod {1}/{2}".format(
self.copy_timeout, self.namespace, self.name
),
stderr,
)
# Non-blocking, keeps the receive buffer clear while we write.
drain(0)
response.write_stdin(chunk)
chunk = tar_buffer.read(self.CHUNK_SIZE)
if not wait_for_exit:
# No way to make the extractor exit, so no status to collect: read
# whatever it has already said and close, as the module always did.
drain(0)
response.close()
return None, stdout, stderr
# The archive is on the wire; wait for the extractor to consume it,
# exit, and have its status reported on the error channel.
while response.is_open():
if time.monotonic() > deadline:
self._fail(
response,
"timed out after {0}s waiting for tar to finish extracting on Pod"
" {1}/{2}".format(self.copy_timeout, self.namespace, self.name),
stderr,
)
drain(1)
error = yaml.safe_load(response.read_channel(ERROR_CHANNEL)) or {}
response.close()
return error, stdout, stderr
def run(self):
# remove trailing slash from destination path
dest_file = self.remote_path.rstrip("/")
@@ -336,65 +480,46 @@ class K8SCopyToPod(K8SCopy):
dest_file = os.path.join(dest_file, os.path.basename(src_file))
if not self.check_mode:
if self.no_preserve:
tar_command = [
"tar",
"--no-same-permissions",
"--no-same-owner",
"-xmf",
"-",
]
else:
tar_command = ["tar", "-xmf", "-"]
if dest_file.startswith("/"):
tar_command.extend(["-C", "/"])
response = stream(
self.api_instance.connect_get_namespaced_pod_exec,
self.name,
self.namespace,
command=tar_command,
stderr=True,
stdin=True,
stdout=True,
tty=False,
_preload_content=False,
**self.container_arg,
)
with TemporaryFile() as tar_buffer:
# Build the archive first: bounding the remote read needs its
# exact size.
with tarfile.open(fileobj=tar_buffer, mode="w") as tar:
tar.add(src_file, dest_file)
archive_size = tar_buffer.tell()
tar_buffer.seek(0)
commands = []
# push command in chunk mode
size = 1024 * 1024
while True:
data = tar_buffer.read(size)
if not data:
break
commands.append(data)
stderr, stdout = [], []
while response.is_open():
if response.peek_stdout():
stdout.append(response.read_stdout().rstrip("\n"))
if response.peek_stderr():
stderr.append(response.read_stderr().rstrip("\n"))
if commands:
cmd = commands.pop(0)
response.write_stdin(cmd)
else:
break
response.close()
if stderr:
self.close_temp_file()
self.module.fail_json(
command=tar_command,
msg="Failed to copy local file/directory into Pod due to: {0}".format(
"".join(stderr)
),
command, wait_for_exit = self.remote_command(dest_file, archive_size)
response = stream(
self.api_instance.connect_get_namespaced_pod_exec,
self.name,
self.namespace,
command=command,
stderr=True,
stdin=True,
stdout=True,
tty=False,
_preload_content=False,
**self.container_arg,
)
error, stdout, stderr = self._stream_tar_to_pod(
response, tar_buffer, wait_for_exit
)
if error is not None and error.get("status") != "Success":
self._fail(
response,
"".join(stderr) or error.get("message", "unknown error"),
command=command,
)
if error is None and stderr:
# Legacy path: no exit status to go on, so any output is fatal.
self._fail(response, "".join(stderr), command=command)
if stderr:
self.module.warn(
"tar wrote to stderr while extracting into Pod {0}/{1}: {2}".format(
self.namespace, self.name, "".join(stderr)
)
)
self.close_temp_file()
if self.content:
self.module.exit_json(
+13
View File
@@ -76,9 +76,21 @@ options:
- This option is ignored when I(content) is set or when I(state) is set to C(from_pod).
type: bool
default: False
copy_timeout:
description:
- Maximum time in seconds to spend streaming the archive to the pod and waiting for the remote
C(tar) process to finish extracting it.
- Raise this when copying large files over a slow connection.
- This option is ignored when I(state) is set to C(from_pod).
type: int
default: 300
version_added: 6.6.0
notes:
- the tar binary is required on the container when copying from local filesystem to pod.
- a POSIX shell at C(/bin/sh) and C(head) are also used, when present, to bound the archive
sent to the container so that the module can confirm the copy finished. Without them the
copy still runs but completion cannot be verified, and a warning is emitted.
- the (init) container has to be started before you copy files or directories to it.
"""
@@ -180,6 +192,7 @@ def argspec():
"choices": ["to_pod", "from_pod"],
}
argument_spec["no_preserve"] = {"type": "bool", "default": False}
argument_spec["copy_timeout"] = {"type": "int", "default": 300}
return argument_spec
@@ -143,9 +143,7 @@ def compare_directories(dir1, dir2):
[len(test.left_only) > 0, len(test.right_only) > 0, len(test.funny_files) > 0]
):
return False
(t, mismatch, errors) = filecmp.cmpfiles(
dir1, dir2, test.common_files, shallow=False
)
t, mismatch, errors = filecmp.cmpfiles(dir1, dir2, test.common_files, shallow=False)
if len(mismatch) > 0 or len(errors) > 0:
return False
for common_dir in test.common_dirs:
@@ -41,6 +41,7 @@
- include_tasks: test_multi_container_pod.yml
- include_tasks: test_copy_directory.yml
- include_tasks: test_copy_large_file.yml
- include_tasks: test_copy_repeated.yml
- include_tasks: test_copy_item_with_space_in_its_name.yml
- include_tasks: test_init_container_pod.yml
@@ -0,0 +1,46 @@
---
# Regression coverage for
# https://github.com/ansible-collections/kubernetes.core/issues/776
#
# The module used to close the exec connection as soon as the last chunk had
# been written, which killed the remote tar mid-extract and truncated the file.
# The failure was timing dependent, so a single copy rarely caught it: repeat
# the copy and compare every time.
- name: create temporary directory for local files
ansible.builtin.tempfile:
state: directory
suffix: k8scp_repeated
register: test_dir
- name: test repeated copies are never truncated
block:
- set_fact:
test_directory: "{{ test_dir.path }}"
copy_iterations: 10
no_log: true
# Large enough to span several 1MiB write chunks, so the last chunk is not
# also the first and the drain actually has something to wait for.
- name: create a multi-chunk file
ansible.builtin.command:
cmd: dd if=/dev/urandom of={{ test_directory }}/repeated.bin bs=1M count=12
- name: copy the same file repeatedly and verify each time
ansible.builtin.include_tasks: verify_one_copy.yml
loop: "{{ range(0, copy_iterations) | list }}"
loop_control:
loop_var: iteration
always:
- name: delete temporary directory created for the test
ansible.builtin.file:
path: "{{ test_dir.path }}"
state: absent
ignore_errors: true
- name: delete file created on Pod
k8s_exec:
namespace: '{{ copy_namespace }}'
pod: '{{ pod_with_one_container.name }}'
command: 'rm -f /repeated.bin'
ignore_errors: true
@@ -0,0 +1,23 @@
---
# One iteration of the repeated-copy regression check, see test_copy_repeated.yml
- name: remove any file left by the previous iteration
k8s_exec:
namespace: '{{ copy_namespace }}'
pod: '{{ pod_with_one_container.name }}'
command: 'rm -f /repeated.bin'
ignore_errors: true
- name: "copy file into remote Pod (iteration {{ iteration }})"
k8s_cp:
namespace: '{{ copy_namespace }}'
pod: '{{ pod_with_one_container.name }}'
remote_path: /repeated.bin
local_path: "{{ test_directory }}/repeated.bin"
state: to_pod
- name: "compare local and remote file (iteration {{ iteration }})"
kubectl_file_compare:
namespace: '{{ copy_namespace }}'
pod: '{{ pod_with_one_container.name }}'
remote_path: /repeated.bin
local_path: "{{ test_directory }}/repeated.bin"
+298
View File
@@ -0,0 +1,298 @@
# Copyright [2026] [Red Hat, Inc.]
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Tests for the to_pod streaming path of K8SCopyToPod. The v4 exec protocol
# cannot half-close stdin, so the archive is bounded with `head -c` to give the
# remote tar a clean EOF; the module must then keep reading until the API server
# closes the connection and check the status reported on the error channel. See
# https://github.com/ansible-collections/kubernetes.core/issues/776
from __future__ import absolute_import, division, print_function
__metaclass__ = type
import io
import pytest
from ansible_collections.kubernetes.core.plugins.module_utils.copy import K8SCopyToPod
ERROR_CHANNEL = 3
SUCCESS = '{"metadata":{},"status":"Success"}'
FAILURE = (
'{"metadata":{},"status":"Failure","message":"command terminated with'
' non-zero exit code","reason":"NonZeroExitCode"}'
)
class FakeModuleFailure(Exception):
"""Stands in for the SystemExit that AnsibleModule.fail_json raises."""
def __init__(self, **kwargs):
self.kwargs = kwargs
super(FakeModuleFailure, self).__init__(kwargs.get("msg"))
class FakeModule:
def __init__(self, **params):
params.setdefault("copy_timeout", 300)
self.params = params
self.check_mode = False
self.warnings = []
def fail_json(self, **kwargs):
raise FakeModuleFailure(**kwargs)
def warn(self, warning):
self.warnings.append(warning)
class FakeWSClient:
"""Minimal stand-in for kubernetes.stream.ws_client.WSClient.
``close_after`` is the number of ``update()`` calls the connection stays
open for once the last chunk has been written, modelling the remote tar
consuming the archive and exiting.
"""
def __init__(
self,
close_after=2,
error=SUCCESS,
stderr=None,
drop_after=None,
stderr_early=False,
):
self.written = []
self.closed = False
self._open = True
self._updates = 0
self._close_after = close_after
self._error = error
self._pending_stderr = stderr
self._drop_after = drop_after
self._stderr_early = stderr_early
self.writes_done = False
def is_open(self):
return self._open
def update(self, timeout=0):
self._updates += 1
if self.writes_done and self._updates > self._close_after:
self._open = False
def write_stdin(self, data):
if self._drop_after is not None and len(self.written) >= self._drop_after:
self._open = False
return
self.written.append(data)
def peek_stdout(self):
return ""
def read_stdout(self):
return ""
def peek_stderr(self):
# tar only complains once it has started extracting, i.e. after the
# whole archive has been handed over, unless it failed outright.
if not (self.writes_done or self._stderr_early):
return ""
return self._pending_stderr or ""
def read_stderr(self):
out, self._pending_stderr = self._pending_stderr, None
return out
def read_channel(self, channel):
assert channel == ERROR_CHANNEL
return self._error
def close(self):
self.closed = True
self._open = False
def make_copier(has_shell=True, no_preserve=False, **params):
module = FakeModule(**params)
copier = K8SCopyToPod.__new__(K8SCopyToPod)
copier.module = module
copier.namespace = "testing"
copier.name = "some-pod"
copier.named_temp_file = None
copier.no_preserve = no_preserve
copier.copy_timeout = module.params["copy_timeout"]
copier.can_bound_stream = lambda: has_shell
return copier
def stream_archive(copier, response, size, wait_for_exit=True):
"""Drive _stream_tar_to_pod, marking the writes finished for the fake."""
payload = io.BytesIO(b"x" * size)
original_read = payload.read
def read(n):
data = original_read(n)
if not data:
response.writes_done = True
return data
payload.read = read
return copier._stream_tar_to_pod(response, payload, wait_for_exit)
def test_waits_for_tar_to_exit_before_closing():
"""The connection must not be closed until the remote process is done."""
copier = make_copier()
response = FakeWSClient(close_after=5)
error, stdout, stderr = stream_archive(
copier, response, 3 * K8SCopyToPod.CHUNK_SIZE
)
assert error["status"] == "Success"
assert stdout == []
assert stderr == []
# Whole archive sent, in CHUNK_SIZE pieces.
assert len(response.written) == 3
assert sum(len(c) for c in response.written) == 3 * K8SCopyToPod.CHUNK_SIZE
# Closed only after the server hung up, not right after the last write.
assert response.closed
assert response._updates > 3
def test_partial_write_is_not_reported_as_success():
"""A connection dropped mid-archive must fail, not silently truncate."""
copier = make_copier()
response = FakeWSClient(drop_after=2)
with pytest.raises(FakeModuleFailure) as exc:
stream_archive(copier, response, 5 * K8SCopyToPod.CHUNK_SIZE)
assert "the remote file is incomplete" in exc.value.kwargs["msg"]
assert len(response.written) == 2
assert response.closed
def test_early_close_reports_what_tar_said():
"""The remote error must not be masked by the generic truncation message."""
copier = make_copier()
response = FakeWSClient(
drop_after=2,
stderr="tar: /etc/foo: Cannot open: Read-only file system",
stderr_early=True,
)
with pytest.raises(FakeModuleFailure) as exc:
stream_archive(copier, response, 5 * K8SCopyToPod.CHUNK_SIZE)
assert "the remote file is incomplete" in exc.value.kwargs["msg"]
assert "Read-only file system" in exc.value.kwargs["msg"]
def test_non_zero_tar_exit_is_surfaced():
"""A Failure status on the error channel must reach the caller."""
copier = make_copier()
response = FakeWSClient(error=FAILURE)
error, _stdout, _stderr = stream_archive(copier, response, K8SCopyToPod.CHUNK_SIZE)
assert error["status"] == "Failure"
assert error["reason"] == "NonZeroExitCode"
def test_stderr_is_collected_while_draining():
"""Output tar writes after the last chunk must still be captured."""
copier = make_copier()
response = FakeWSClient(stderr="tar: /foo: Cannot utime: Operation not permitted")
_error, _stdout, stderr = stream_archive(copier, response, K8SCopyToPod.CHUNK_SIZE)
assert stderr == ["tar: /foo: Cannot utime: Operation not permitted"]
def test_drain_honours_copy_timeout(monkeypatch):
"""A remote tar that never exits must time out instead of hanging."""
copier = make_copier(copy_timeout=30)
response = FakeWSClient(close_after=10**6)
# Time only moves past the deadline once the archive has been sent, so the
# timeout is exercised in the drain loop rather than the write loop.
monkeypatch.setattr(
"time.monotonic", lambda: 10**6 if response.writes_done else 0
)
with pytest.raises(FakeModuleFailure) as exc:
stream_archive(copier, response, K8SCopyToPod.CHUNK_SIZE)
assert "waiting for tar to finish extracting" in exc.value.kwargs["msg"]
assert response.closed
def test_archive_is_bounded_with_head():
"""head -c gives the remote tar a clean EOF the v4 protocol cannot."""
copier = make_copier()
command, wait_for_exit = copier.remote_command("/tmp/foo", 1003520)
assert wait_for_exit is True
assert command == ["/bin/sh", "-c", "head -c 1003520 | tar -xmf - -C /"]
assert copier.module.warnings == []
def test_bounded_command_carries_no_preserve():
copier = make_copier(no_preserve=True)
command, _wait = copier.remote_command("/tmp/foo", 512)
assert command[2] == (
"head -c 512 | tar --no-same-permissions --no-same-owner -xmf - -C /"
)
def test_relative_destination_is_not_anchored():
"""Only absolute destinations get -C /, as before."""
copier = make_copier()
command, _wait = copier.remote_command("tmp/foo", 512)
assert command[2] == "head -c 512 | tar -xmf -"
def test_shell_less_container_falls_back_and_warns():
"""Without a shell the copy still runs, but completion is unverifiable."""
copier = make_copier(has_shell=False)
command, wait_for_exit = copier.remote_command("/tmp/foo", 512)
assert wait_for_exit is False
assert command == ["tar", "-xmf", "-", "-C", "/"]
assert len(copier.module.warnings) == 1
assert "cannot confirm the copy" in copier.module.warnings[0]
def test_fallback_path_does_not_wait_for_exit():
"""The legacy path must not block on a tar that will never exit."""
copier = make_copier(has_shell=False)
response = FakeWSClient(close_after=10**6)
error, _stdout, _stderr = stream_archive(
copier, response, K8SCopyToPod.CHUNK_SIZE, wait_for_exit=False
)
# No exit status is available on this path.
assert error is None
assert response.closed
assert sum(len(c) for c in response.written) == K8SCopyToPod.CHUNK_SIZE