diff --git a/.github/workflows/lint.yml b/.github/workflows/lint.yml index 1d6cfa1..06aa091 100644 --- a/.github/workflows/lint.yml +++ b/.github/workflows/lint.yml @@ -9,7 +9,7 @@ jobs: runs-on: ubuntu-latest strategy: matrix: - python-version: ['3.8', '3.9', '3.10', '3.11', '3.12', '3.13', '3.14', '3.14t', '3.15.0-beta.1'] + python-version: ['3.8', '3.9', '3.10', '3.11', '3.12', '3.13', '3.14', '3.14t', '3.15'] steps: - uses: actions/checkout@v4 @@ -18,6 +18,9 @@ jobs: uses: actions/setup-python@v5 with: python-version: ${{ matrix.python-version }} + # Prerelease CPython ABIs can change; keep 3.15 aligned with current wheels. + allow-prereleases: true + check-latest: ${{ matrix.python-version == '3.15' }} - name: Set up uv uses: astral-sh/setup-uv@v7 diff --git a/.github/workflows/tests_and_coverage.yml b/.github/workflows/tests_and_coverage.yml index 681f11a..3f3a0e2 100644 --- a/.github/workflows/tests_and_coverage.yml +++ b/.github/workflows/tests_and_coverage.yml @@ -10,7 +10,7 @@ jobs: strategy: matrix: os: [macos-latest, ubuntu-latest, windows-latest] - python-version: ['3.8', '3.9', '3.10', '3.11', '3.12', '3.13', '3.14', '3.14t', '3.15.0-beta.1'] + python-version: ['3.8', '3.9', '3.10', '3.11', '3.12', '3.13', '3.14', '3.14t', '3.15'] steps: - uses: actions/checkout@v4 @@ -18,6 +18,9 @@ jobs: uses: actions/setup-python@v5 with: python-version: ${{ matrix.python-version }} + # Prerelease CPython ABIs can change; keep 3.15 aligned with current wheels. + allow-prereleases: true + check-latest: ${{ matrix.python-version == '3.15' }} - name: Set up uv uses: astral-sh/setup-uv@v7 diff --git a/README.md b/README.md index 89f51d6..96c3715 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ -![logo](https://raw.githubusercontent.com/mutating/throng/develop/docs/assets/logo_1.svg) +![throng logo](https://raw.githubusercontent.com/mutating/throng/develop/docs/assets/logo_1.svg) -Sometimes our programs need to execute console commands. In some cases, a command may be executed locally, while in others it may be executed in parallel across thousands of machines in the cloud. This library serves as an abstraction layer over various command execution environments, allowing you to write code once that will run anywhere. Specific execution environments are connected here as plugins (and you can even write your own!) with a unified API, and your code doesn’t need to know the internal workings of a specific plugin to run commands within it. +Sometimes our programs need to execute console commands. In some cases, a command may be executed locally, while in others it may be executed in parallel across thousands of machines in the cloud. This library serves as an abstraction layer over various command execution environments, allowing you to write code once that will run anywhere. Specific execution environments are connected here as plugins with a unified API, and your code doesn’t need to know the internal workings of a specific plugin to run commands within it. This library provides: @@ -50,8 +50,8 @@ SubprocessResult(id='fc1d7602b68211f196d7f6817fdcabf4', stdout='LICENSE\nREADME. Let’s understand this code: -- The `with ... isolate` construct creates an environment for executing commands, while ensuring that this environment is cleaned up or destroyed as needed after exiting the code block. -- `scope.run()` executes the commands in the created temporary environment. +- The `with ... as isolate` construct creates an environment for executing commands, while ensuring that this environment is cleaned up or destroyed as needed after exiting the code block. +- `isolate.run()` executes the commands in the created temporary environment. - Where exactly will the commands be executed? In this case, it’s determined by the `'temporary_directory'` string, which is actually the name of a built-in plugin. If needed, it can be replaced with the name of any other command-execution plugin you can connect, and your main code will work with it as if nothing had changed. This particular plugin creates a temporary directory on your computer, copies all files from the current directory into it, and deletes the entire directory once it’s finished. - `'.'` means that the state of the current directory is transferred to the execution environment. You can use a different directory. @@ -62,14 +62,14 @@ There are many places and ways to run console commands. You can almost certainly There are two independent axes along which we can evaluate different methods of executing commands. First, different methods vary in their degree of parallelism. For example, your computer almost certainly has limited memory and processing power, so we cannot increase parallelism indefinitely. Second, different methods have different levels of isolation for command execution. For instance, if you run commands in different directories on your computer, such isolation can only be described as very limited—processes can easily interact with one another and interfere with each other. If, on the other hand, you run commands via serverless environments, they are usually fairly well isolated from one another, and it is much more difficult for them to interact with each other. -![logo](https://raw.githubusercontent.com/mutating/throng/develop/docs/assets/illustration_1.png) +![Comparison of command execution environments by parallelism and isolation](https://raw.githubusercontent.com/mutating/throng/develop/docs/assets/illustration_1.png) > ⓘ This is roughly what the breakdown by axis might look like for various popular command-launching technologies. The axes here aren't strict, so don't take the image too seriously. Importantly, both parallelism and isolation come at a cost. For example, if you decide to thoroughly isolate a command when running it on your computer by using virtual machines, you’ll need much more memory and time to start executing the command than you would with a “naive” execution of the command. That’s why it’s good to have a choice of different options, so you can select the one that best fits your specific needs and the price you’re willing to pay for it. Most systems that offer you various ways to execute commands are typically tied to a specific infrastructure—whether physical or software-based—whose position on the two axes described above is fixed. If you want to write your own logic on top of such systems, your code will generally contain duplicate components. And with every new way of executing commands you add, your codebase will become bloated. -`throng` solves this problem. Various command execution systems are abstracted here into a separate layer that can be easily swapped out for another using the modern [`pristan`](https://github.com/mutating/pristan) plugin system. You can enable the plugin that is most optimal for you in terms of operation cost, while also offering the right balance on the axes of parallelism and execution isolation. Now you don’t need to rewrite or bloat your program when you want to add another way to execute code—just select the appropriate plugin and connect it. +`throng` solves this problem. Various command execution systems are abstracted here into a separate layer that can be easily swapped out for another using the modern [`pristan`](https://github.com/mutating/pristan) plugin system. You can enable the plugin that is best suited to your needs in terms of execution cost, while also offering the right balance on the axes of parallelism and execution isolation. Now you don’t need to rewrite or bloat your program when you want to add another way to execute code—just select the appropriate plugin and connect it. Now it is the plugin's responsibility to determine how much of your code can be executed in parallel and to what extent different executions will be isolated from one another. Your code knows nothing about this—it simply executes commands and receives results. You write compact and powerful programs, isolated from the specifics of execution on distributed systems or virtual machines, debug them locally, and then run them anywhere. @@ -85,7 +85,7 @@ When working with the `throng` system, you need to understand four key concepts: The main idea behind `throng` is that your main program doesn't “know” exactly how its command will be executed. It simply issues the command and receives the result. Different execution methods are connected as plugins that can be easily installed and removed, and from which you can choose. -Throng provides a special object that you can import and call like a regular function; in `pristan` terminology, such an object is called a slot. When called, it returns a dictionary whose keys are the names of all available plugins, and whose values are special managers—whose capabilities we’ll explore later. Immediately after installing `throng`, while no additional plugins have been installed yet, calling the slot will look something like this: +`throng` provides a special object that you can import and call like a regular function; in `pristan` terminology, such an object is called a slot. When called, it returns a dictionary whose keys are the names of all available plugins, and whose values are special managers. We’ll explore their capabilities later. Immediately after installing `throng`, while no additional plugins have been installed yet, calling the slot will look something like this: ```python from throng import throng @@ -95,7 +95,7 @@ print(managers) #> {'local': LocalManager('.'), 'temporary_directory': TemporaryDirectoryManager('.')} ``` -> ↑ We pass a point as a reference to the current directory, the state of which will serve as the basis for all isolates that are created (you'll learn what these are later). +> ↑ We pass a dot (`'.'`) as a reference to the current directory, the state of which will serve as the basis for all isolates that are created (you'll learn what these are later). As you can see, by default, `throng` comes with two built-in plugins. We’ll take a closer look at them a little later. Let’s retrieve the manager object returned by one of the plugins and explore the features it offers. @@ -107,7 +107,7 @@ All manager objects have a uniform API, which allows them to be used in the same An isolate is a special object responsible for executing commands. Operations related to its lifecycle come from outside; that is, it is always created and destroyed by someone (usually a manager). Thus, responsibility is clearly divided between them: the isolate is responsible only for executing commands, while the manager handles lifecycle issues. -To create an instance of an isolate, use the manager to read the initial state, and then use it to create the isolate:: +To create an instance of an isolate, use the manager to read the initial state, and then use it to create the isolate: ```python state = manager.read() @@ -131,17 +131,17 @@ print(result.stdout) #> venv ``` -> ↑ Just in case: The author ran this command in the throng project directory; the output of the `ls` command may be different for you. +> ↑ Just in case: The author ran this command in the `throng` project directory; the output of the `ls` command may be different for you. -When we no longer need a specific isolate, you must destroy it by calling its `kill()` method: +When you no longer need a specific isolate, you must destroy it by calling its `kill()` method: ```python isolate.kill() ``` -It may be necessary to destroy isolates to conserve resources if those resources were specifically allocated for that isolate. For example, if an isolate abstracts a virtual machine from you, the memory and other resources allocated to it will remain occupied until you destroy the isolate. The specific details of what needs to be done to free up the occupied resources are abstracted from your code and are entirely determined by the internal workings of the connected plugin. +It may be necessary to destroy isolates to conserve resources if those resources were specifically allocated for that isolate. For example, if an isolate represents a virtual machine, the memory and other resources allocated to it will remain occupied until you destroy the isolate. The specific details of what needs to be done to free up the occupied resources are abstracted from your code and are entirely determined by the internal workings of the connected plugin. -However, determining the lifecycle of isolates “manually” can be too tedious, so you might find it more convenient to use a context manager for this: +However, managing the lifecycle of isolates “manually” can be too tedious, so you might find it more convenient to use a context manager for this: ```python with manager.scope as isolate: @@ -164,7 +164,7 @@ with manager.scope as isolate: As you can see, in the example above, there was no need to destroy the isolate; it was destroyed automatically after exiting the code block where its commands were executed. -However, in some cases, even creating a context is an unnecessary complication. You may need an isolate simply to execute a command within it and get the result. In this case, instead of creating an isolate, you can pass the command directly to the manager, which will create an instance of the isolate “behind the scenes” specifically for that command, pass the command to it, destroy the isolate, and return the command to you: +However, in some cases, even creating a context is an unnecessary complication. You may need an isolate simply to execute a command within it and get the result. In this case, instead of creating an isolate, you can pass the command directly to the manager, which will create an instance of the isolate “behind the scenes” specifically for that command, pass the command to it, destroy the isolate, and return the result to you: ```python print(manager.run('ls').stdout) @@ -183,7 +183,7 @@ print(manager.run('ls').stdout) Generally, creating a new isolate for each command is costly, since the operations involved in reading the state and creating isolates can be expensive. Do this only if you are certain that you do not plan to reuse this environment. -Now that you know all the necessary basic concepts, read on to learn the details of working with `throng` — such as how the built-in plugins work or how to create your own. +Now that you know all the necessary basic concepts, read on to learn the details of working with `throng` — such as how the built-in plugins work or how to install additional ones. ## Isolates and command execution @@ -201,7 +201,7 @@ state = manager.read() isolate = manager.get(state) ``` -A command that can be passed to an isolate is a string, usually containing Bash code; however, the specific string format accepted and its interpretation are the responsibility of the particular plugin. You must understand and expect that plugins may represent completely different internal structures of isolation environments—in some cases, your commands may be executed locally, while in others they may be executed on remote server farms running an unknown operating system designed for cluster computing. Your code cannot expect a precisely guaranteed output from commands, and it is recommended that it double-check the results of command execution. +A command that can be passed to an isolate is a string containing the command to execute; however, the specific string format accepted and its interpretation are the responsibility of the particular plugin. You must understand and expect that plugins may represent completely different internal structures of isolation environments—in some cases, your commands may be executed locally, while in others they may be executed on remote server farms running an unknown operating system designed for cluster computing. Your code cannot expect a precisely guaranteed output from commands, and it is recommended that it double-check the results of command execution. As a result of executing any command, you will receive a special object that must contain the following fields: @@ -212,7 +212,7 @@ As a result of executing any command, you will receive a special object that mus The availability of these fields is guaranteed, and you can base your code on them. Individual plugin implementations may add their own fields to this list, but you should not expect anything else in your programs. -In addition to the command, you can pass one more thing to the isolator—a cancellation token from the cantok library. A token is a special object that allows the isolator to know when to stop executing the command. It might look something like this: +In addition to the command, you can pass one more thing to the isolate—a cancellation token from the `cantok` library. A token is a special object that allows the isolate to know when to stop executing the command. It might look something like this: ```python from cantok import TimeoutToken @@ -221,7 +221,7 @@ print(isolate.run('python -c "import time; time.sleep(1000)"', token=TimeoutToke #> SubprocessResult(id='01fcdbacb6d911f1808df6817fdcabf4', stdout='', stderr='', returncode=-9, killed_by_token=True) ``` -Cancelling a token does not guarantee that the team in the isolate will stop working early; it simply requests that they do so. Whether or not to respond to such a request is the plugin’s responsibility. Do not base your code on the expectation that isolates will always read the token’s status. +Cancelling a token does not guarantee that the command in the isolate will stop running early; it simply requests that it do so. Whether or not to respond to such a request is the plugin’s responsibility. Do not base your code on the expectation that isolates will always read the token’s status. If you need to execute not just one command but a whole series of them, it is recommended that you use the `chain()` method: @@ -242,7 +242,7 @@ When you no longer need a particular isolate, call its `kill()` method: isolate.kill() ``` -Do not attempt to call a command in an isolate that has been destroyed — this may cause an exception. The execution time of the method when it is called is not guaranteed—there may be a network call or some other resource-intensive operation happening behind the scenes. However, plugin authors are advised to make this operation fast. +Do not attempt to run a command in an isolate that has been destroyed — this may cause an exception. The execution time of `kill()` is not guaranteed—there may be a network call or some other resource-intensive operation happening behind the scenes. However, plugin authors are advised to make this operation fast. With some "expensive" isolates, it may be important to you that they do not remain in a suspended state if, for example, your code "forgot" to destroy the isolate, or if it terminated abnormally without having had time to release resources. `throng` does not provide such guarantees, as they depend on the specific infrastructure used to run the commands. Check the documentation for the specific plugin to see if this kind of problem could arise in its infrastructure and how it is recommended to resolve it. @@ -251,14 +251,14 @@ With some "expensive" isolates, it may be important to you that they do not rema The primary task of managers is to create isolates and, in some cases, to manage their subsequent lifecycle. By regulating the creation of isolates, a manager effectively regulates parallelism as well. In other words, the isolate is responsible for isolation, and the manager is responsible for parallelism. -How does this work? For example, a manager might maintain a pool of executables behind the scenes, and a new isolate will be created only when space becomes available in that pool. When your code requests a new isolate, the manager may “hang” until the necessary resources become available. The manager may also maintain a mutex or semaphore internally to limit local concurrency. In some cases, it may take into account feedback signals from the execution system and adjust its resource requests accordingly. All these details are internal aspects of the manager’s implementation, and that’s where the magic lies: you simply request an isolate from the manager and wait, and it handles everything else. +How does this work? For example, a manager might maintain a pool of workers behind the scenes, and a new isolate will be created only when space becomes available in that pool. When your code requests a new isolate, the call may block until the necessary resources become available. The manager may also maintain a mutex or semaphore internally to limit local concurrency. In some cases, it may take into account feedback signals from the execution system and adjust its resource requests accordingly. All these details are internal aspects of the manager’s implementation, and that’s where the magic lies: you simply request an isolate from the manager and wait, and it handles everything else. Unfortunately, execution abstraction comes at a cost. For you as a user, the main drawback of `throng` may be the unpredictability of wait times for basic operations in your software, since you can’t tell for sure whether a command is executed immediately via a local subprocess or is sent to a data center on the other side of the globe. Although we’ve already shown above how to obtain a manager and how to use it, we’ll briefly review this in this section so that everything related to managers is covered here. Essentially, there are two ways to use `throng`: -- Query individual isolate objects and work with them — let’s call this the "open" method. -- Passing commands directly to the manager without retrieving the isolates for them — let’s call this the "closed" method, since the isolates remain hidden from you. +- Obtain individual isolate objects and work with them — let’s call this the "open" method. +- Pass commands directly to the manager without retrieving the isolates for them — let’s call this the "closed" method, since the isolates remain hidden from you. Let's get a manager object for further demonstrations: @@ -283,7 +283,7 @@ And here's the closed one: manager.run('ls') ``` -The `chain()` method, which executes a series of commands at once, can also be used in both open and closed modes. The open mode is called from an isolate: +The `chain()` method, which executes a series of commands sequentially, can also be used in both open and closed modes. In open mode, it is called on an isolate: ```python isolate.chain( @@ -293,7 +293,7 @@ isolate.chain( ) ``` -And the closed way is called directly from the manager: +In closed mode, it is called directly on the manager: ```python manager.chain( @@ -303,14 +303,14 @@ manager.chain( ) ``` -As you can see, the open and closed approaches aren’t all that different. So which one should you choose? For simple scenarios, the closed approach is almost always preferable: it allows you to focus on your logic without having to worry about isolator management. The open approach is intended for exceptional situations, such as when, for some reason, the lifecycle of an isolate becomes long and the logic for working with it becomes complex and nonlinear. For example, if your code frequently reloads a saved state instead of reading the current state each time. +As you can see, the open and closed approaches aren’t all that different. So which one should you choose? For simple scenarios, the closed approach is almost always preferable: it allows you to focus on your logic without having to worry about isolate management. The open approach is intended for exceptional situations, such as when, for some reason, the lifecycle of an isolate becomes long and the logic for working with it becomes complex and nonlinear. For example, if your code frequently reloads a saved state instead of reading the current state each time. ## Plugins -The main feature of `throng` is that it doesn’t force you to use any specific method for executing commands, with all its advantages and limitations. You have the freedom to choose. This is made possible by the powerful pristan plugin system. +The main feature of `throng` is that it doesn’t force you to use any specific method for executing commands, with all its advantages and limitations. You have the freedom to choose. This is made possible by the powerful `pristan` plugin system. -Throng includes two basic plugins: +`throng` includes two basic plugins: - A plugin for running commands **locally**. - A plugin for running commands in **temporary directories**. @@ -336,9 +336,9 @@ print(manager.run('ls').stdout) #> venv ``` -With the open method of accessing isolates, both reading the current directory and “recreating” an isolate from it are fake operations. These operations simply transfer an empty set of bytes. This is fast, but it’s not secure, because any changes you make alter your main set of files in the specified directory, rather than the copy you specifically created for experiments. +With the open method of accessing isolates, both reading the current directory and “recreating” an isolate from it are no-ops. These operations simply pass an empty byte string. This is fast, but it’s not secure, because any changes you make alter your main set of files in the specified directory, rather than the copy you specifically created for experiments. -The plugin that uses temporary directories does almost the same thing, but when each isolate is created, the following actually happens: all files in the current directory (and this is important: only files, not empty directories or symlinks) are packed into a tar archive; then a temporary directory with a random name is created on your computer; and finally, the archive is extracted there. When the isolate is destroyed, the entire directory that was created is deleted. Commands are executed via subprocesses, which are passed the path to the temporary directory for command execution. +The plugin that uses temporary directories does almost the same thing, but creating an isolate involves the following steps: `manager.read()` packs all files in the specified directory (and this is important: only files, not empty directories) into a tar archive with paths relative to that directory; then `manager.get(state)` creates a temporary directory with a random name on your computer; and finally, the supplied archive is extracted there. When the isolate is destroyed, the entire directory that was created is deleted. Commands are executed via subprocesses, which are passed the path to the temporary directory for command execution. You can retrieve the manager from the plugin that handles temporary directories using the `'temporary_directory'` key: @@ -347,7 +347,7 @@ managers = throng('.') manager = managers['temporary_directory'] ``` -As you can see, the built-in plugins are very limited, and they will almost certainly not be enough for you, so you can install additional ones. Any Throng plugin is simply a Python package that can be installed via pip or uv and is packaged in a specific way using Pristan. In other words, all you need to install a plugin is a command like this: +As you can see, the built-in plugins are very limited, and they will almost certainly not be enough for you, so you can install additional ones. Any `throng` plugin is simply a Python package that can be installed via `pip` or `uv` and is packaged in a specific way using `pristan`. In other words, all you need to install a plugin is a command like this: ```bash pip install plugin-name diff --git a/pyproject.toml b/pyproject.toml index 0730f41..0e74f5e 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta" [project] name = "throng" -version = "0.0.3" +version = "0.0.4" authors = [ { name="Evgeniy Blinov", email="zheni-b@yandex.ru" }, ] @@ -20,7 +20,6 @@ dependencies = [ 'dirstree>=0.0.12', 'locklib>=0.0.25', 'pathspec>=0.12.0', - 'cython==3.2.4', ] classifiers = [ "Operating System :: OS Independent", diff --git a/tests/extensions/local/test_manager.py b/tests/extensions/local/test_manager.py deleted file mode 100644 index fdedf59..0000000 --- a/tests/extensions/local/test_manager.py +++ /dev/null @@ -1,10 +0,0 @@ -from throng import throng - - -def test_simple_print(): - result = throng('.')['local'].run('echo lol') - - assert result.success - assert result.returncode == 0 - assert result.stderr == '' - assert result.stdout == 'lol\n' diff --git a/tests/extensions/__init__.py b/tests/units/__init__.py similarity index 100% rename from tests/extensions/__init__.py rename to tests/units/__init__.py diff --git a/tests/extensions/local/__init__.py b/tests/units/abstracts/__init__.py similarity index 100% rename from tests/extensions/local/__init__.py rename to tests/units/abstracts/__init__.py diff --git a/tests/units/abstracts/test_abstract_isolate.py b/tests/units/abstracts/test_abstract_isolate.py new file mode 100644 index 0000000..4e73783 --- /dev/null +++ b/tests/units/abstracts/test_abstract_isolate.py @@ -0,0 +1,228 @@ +from unittest.mock import MagicMock, Mock, call + +import pytest +from cantok import DefaultToken, SimpleToken + +from throng.abstracts.abstract_isolate import AbstractIsolate +from throng.abstracts.results import SimpleRunResult +from throng.errors import NotSupportedCommandError + + +@pytest.mark.parametrize('token_kind', ['default', 'active', 'cancelled', 'unreadable']) +def test_empty_chain_does_not_inspect_token(token_kind): + """Return a fresh empty list without execution or cancellation checks.""" + isolate = Mock(spec=AbstractIsolate) + token = MagicMock() + token.__bool__.side_effect = AssertionError('The token must not be inspected.') + options = { + 'default': {}, + 'active': {'token': SimpleToken()}, + 'cancelled': {'token': SimpleToken(cancelled=True)}, + 'unreadable': {'token': token}, + } + + first = AbstractIsolate.chain(isolate, **options[token_kind]) + second = AbstractIsolate.chain(isolate, **options[token_kind]) + + assert first == second == [] + assert first is not second + first.append(SimpleRunResult(True)) + assert second == [] + assert isolate.mock_calls == [] + token.__bool__.assert_not_called() + + +@pytest.mark.parametrize( + 'commands', + [('one',), ('one', 'two', 'one'), ('', ' ', 'Привет', 'a\nb', '"a b"; x')], +) +@pytest.mark.parametrize('explicit_token', [False, True]) +def test_chain_preserves_commands_results_and_token(commands, explicit_token): + """Keep command order, plugin results and the same token throughout a chain.""" + isolate = Mock(spec=AbstractIsolate) + fields = [ + { + 'success': True, + 'returncode': 0, + 'stdout': command, + 'stderr': ' diagnostic\n', + 'extra': object(), + } + for command in commands + ] + expected = [Mock(**values) for values in fields] + isolate.run.side_effect = expected + token = SimpleToken() + + results = AbstractIsolate.chain( + isolate, + *commands, + **({'token': token} if explicit_token else {}), + ) + + passed_token = isolate.run.call_args.kwargs['token'] + if explicit_token: + assert passed_token is token + else: + assert isinstance(passed_token, DefaultToken) + assert isolate.mock_calls == [ + call.run(command, token=passed_token) for command in commands + ] + assert len(results) == len(expected) + assert all(actual is original for actual, original in zip(results, expected)) + for result, values in zip(results, fields): + assert {name: getattr(result, name) for name in values} == values + + +@pytest.mark.parametrize('failed_at', [0, 1, 2, 'all']) +@pytest.mark.parametrize('returncode', [0, 1, -9, None]) +def test_chain_continues_after_unsuccessful_results(failed_at, returncode): + """Leave stopping decisions to cancellation rather than command success.""" + isolate = Mock(spec=AbstractIsolate) + expected = [ + SimpleRunResult(success=failed_at not in (index, 'all'), returncode=returncode) + for index in range(3) + ] + isolate.run.side_effect = expected + + results = AbstractIsolate.chain(isolate, 'first', 'second', 'third') + + assert isolate.run.call_count == 3 + assert all(actual is original for actual, original in zip(results, expected)) + assert [ + (result.success, result.returncode, result.stdout, result.stderr) + for result in results + ] == [ + (failed_at not in (index, 'all'), returncode, None, None) for index in range(3) + ] + + +@pytest.mark.parametrize( + ('command_count', 'completed'), + [(1, 0), (4, 0), (4, 1), (4, 2), (4, 4)], +) +@pytest.mark.parametrize('success', [False, True]) +def test_chain_keeps_completed_results_when_cancelled( + command_count, + completed, + success, +): + """Keep completed results and mark every command skipped after cancellation.""" + commands = ('first', 'second', 'third', 'fourth')[:command_count] + token = SimpleToken(cancelled=completed == 0) + isolate = Mock(spec=AbstractIsolate) + executed = [] + + def execute(_command, *, token): + result = SimpleRunResult(success, 0 if success else 1) + executed.append(result) + if len(executed) == completed: + token.cancel() + return result + + isolate.run.side_effect = execute + + results = AbstractIsolate.chain(isolate, *commands, token=token) + + assert isolate.mock_calls == [ + call.run(command, token=token) for command in commands[:completed] + ] + assert len(results) == len(commands) + assert all( + result is executed[index] for index, result in enumerate(results[:completed]) + ) + assert results[completed:] == [SimpleRunResult(False) for _ in commands[completed:]] + assert ( + len({id(result) for result in results[completed:]}) == len(commands) - completed + ) + + +def test_skipped_results_are_independent(): + """Allow callers to update one skipped result without changing its neighbors.""" + isolate = Mock(spec=AbstractIsolate) + + results = AbstractIsolate.chain( + isolate, + 'one', + 'two', + token=SimpleToken(cancelled=True), + ) + results[0].stdout = 'annotated' + + assert results[1].stdout is None + assert isolate.mock_calls == [] + + +@pytest.mark.parametrize('cancel_first', [False, True]) +def test_chains_do_not_share_results_or_cancellation(cancel_first): + """Start each chain independently while preserving repeated plugin objects.""" + isolate = Mock(spec=AbstractIsolate) + expected = SimpleRunResult(True) + isolate.run.return_value = expected + + first = AbstractIsolate.chain( + isolate, + 'one', + 'two', + token=SimpleToken(cancelled=cancel_first), + ) + second = AbstractIsolate.chain(isolate, 'one', 'two') + + assert first is not second + assert second[0] is expected + assert second[1] is expected + assert isolate.run.call_count == (2 if cancel_first else 4) + + +@pytest.mark.parametrize('position', [0, 1, 2]) +@pytest.mark.parametrize( + 'error_type', + [RuntimeError, NotSupportedCommandError, KeyboardInterrupt], +) +def test_chain_stops_on_execution_exception(position, error_type): + """Stop at an execution exception and propagate the original cause.""" + isolate = Mock(spec=AbstractIsolate) + error = error_type('execution failed') + isolate.run.side_effect = [SimpleRunResult(True)] * position + [error] + token = SimpleToken() + commands = ('one', 'two', 'three') + + with pytest.raises(error_type) as caught: + AbstractIsolate.chain(isolate, *commands, token=token) + + assert caught.value is error + assert isolate.mock_calls == [ + call.run(command, token=token) for command in commands[: position + 1] + ] + + +@pytest.mark.parametrize('completed', [0, 1]) +def test_chain_stops_on_token_exception(completed): + """Do not execute a command whose cancellation check failed.""" + isolate = Mock(spec=AbstractIsolate) + token = MagicMock() + error = RuntimeError('token failed') + token.__bool__.side_effect = [True] * completed + [error] + + with pytest.raises(RuntimeError) as caught: + AbstractIsolate.chain(isolate, 'one', 'two', token=token) + + assert caught.value is error + assert isolate.run.call_count == completed + isolate.kill.assert_not_called() + + +def test_destructor_delegates_cleanup(): + """Delegate final resource cleanup to the concrete isolate implementation.""" + isolate = Mock(spec=AbstractIsolate) + + AbstractIsolate.__del__(isolate) + + assert isolate.mock_calls == [call.kill()] + + +def test_isolate_requires_concrete_operations(): + """Require plugins to implement execution, snapshots, cleanup and installation.""" + assert AbstractIsolate.__abstractmethods__ == {'run', 'read', 'kill', 'install'} + with pytest.raises(TypeError, match='abstract'): + AbstractIsolate() diff --git a/tests/units/abstracts/test_abstract_manager.py b/tests/units/abstracts/test_abstract_manager.py new file mode 100644 index 0000000..72dc9be --- /dev/null +++ b/tests/units/abstracts/test_abstract_manager.py @@ -0,0 +1,383 @@ +from contextlib import nullcontext +from pathlib import Path +from unittest.mock import MagicMock, Mock, call + +import pytest +from cantok import DefaultToken, SimpleToken + +from throng.abstracts.abstract_manager import AbstractManager, ContextIsolateManager +from throng.errors import CannotCancelNonExistingIsolateError +from throng.extensions.local.manager import LocalManager +from throng.extensions.temporary_directory.manager import TemporaryDirectoryManager + + +@pytest.mark.parametrize('path_kind', ['string', 'path', 'absolute']) +@pytest.mark.parametrize('name', ['.', 'missing', 'space here/каталог']) +@pytest.mark.parametrize('exclude', [None, [], ['cache/', '*.tmp', '!keep.tmp']]) +def test_initialization_is_lazy(tmp_path, monkeypatch, path_kind, name, exclude): + """Store source settings without reading or creating the source directory.""" + monkeypatch.chdir(tmp_path) + path = {'string': name, 'path': Path(name), 'absolute': tmp_path / name}[path_kind] + expected_exclude = None if exclude is None else exclude.copy() + + manager = TemporaryDirectoryManager(path, exclude) + + assert manager.path == Path(path) + assert manager.exclude == expected_exclude + assert list(tmp_path.iterdir()) == [] + + +def test_default_exclusions_are_absent(): + """Leave file exclusions unset when callers omit them.""" + assert TemporaryDirectoryManager('.').exclude is None + + +@pytest.mark.parametrize( + ('manager_type', 'name'), + [ + (LocalManager, 'LocalManager'), + (TemporaryDirectoryManager, 'TemporaryDirectoryManager'), + ], +) +@pytest.mark.parametrize( + ('path', 'exclude', 'expected'), + [ + ('.', None, "('.')"), + ('.', [], "('.', exclude=[])"), + ( + 'каталог', + ['*.tmp'], + "('каталог', exclude=['*.tmp'])", + ), + ], +) +def test_representation_includes_explicit_settings( + manager_type, + name, + path, + exclude, + expected, +): + """Show the concrete manager, source path and explicitly supplied exclusions.""" + assert repr(manager_type(path, exclude)) == name + expected + + +@pytest.mark.parametrize('implemented', [(), ('read',), ('get',), ('read', 'get')]) +def test_manager_requires_read_and_get(implemented): + """Reject incomplete plugins while allowing managers implementing both operations.""" + manager_type = type( + 'Manager', + (AbstractManager,), + {name: Mock() for name in implemented}, + ) + + if len(implemented) == 2: + assert isinstance(manager_type('.'), AbstractManager) + else: + with pytest.raises(TypeError, match='abstract'): + manager_type('.') + + +def test_scope_is_lazy_and_independent(monkeypatch): + """Allocate a fresh context on demand without creating an isolate early.""" + manager = TemporaryDirectoryManager('.') + read, get = Mock(), Mock() + monkeypatch.setattr(manager, 'read', read) + monkeypatch.setattr(manager, 'get', get) + + first, second = manager.scope, manager.scope + + assert first is not second + assert first.manager is second.manager is manager + assert first.isolate is second.isolate is None + read.assert_not_called() + get.assert_not_called() + + +@pytest.mark.parametrize('state', [b'', b'snapshot', b'\x00\xff']) +def test_context_passes_opaque_state(state): + """Create the isolate from the exact snapshot returned by the manager.""" + manager = Mock() + manager.read.return_value = state + context = ContextIsolateManager(manager) + + isolate = context.__enter__() + + assert manager.mock_calls == [call.read(), call.get(state)] + assert isolate is context.isolate is manager.get.return_value + context.__exit__(None, None, None) + isolate.kill.assert_called_once_with() + + +@pytest.mark.parametrize('stage', ['read', 'get']) +def test_failed_context_entry_preserves_error(stage): + """Stop construction immediately when reading or creating an isolate fails.""" + manager = Mock() + error = OSError('creation failed') + getattr(manager, stage).side_effect = error + context = ContextIsolateManager(manager) + + with pytest.raises(OSError, match='creation failed') as caught, context: + pytest.fail('The context body must not be entered.') + + assert caught.value is error + assert context.isolate is None + assert manager.mock_calls == [call.read()] + ( + [call.get(manager.read.return_value)] if stage == 'get' else [] + ) + + +@pytest.mark.parametrize( + 'error_type', + [None, ValueError, KeyboardInterrupt, SystemExit], +) +@pytest.mark.parametrize('truthy_isolate', [False, True]) +def test_context_cleans_up_on_every_exit(error_type, truthy_isolate): + """Clean up even a false-valued isolate without suppressing body exceptions.""" + manager = Mock() + isolate = MagicMock() + isolate.__bool__.return_value = truthy_isolate + manager.get.return_value = isolate + error = error_type('body failed') if error_type else None + + expectation = pytest.raises(error_type) if error_type else nullcontext() + with expectation as caught, ContextIsolateManager(manager) as actual: + assert actual is isolate + if error is not None: + raise error + + if error is not None: + assert caught.value is error + isolate.kill.assert_called_once_with() + + +@pytest.mark.parametrize('failed_stage', [None, 'read', 'get']) +def test_exit_without_isolate_is_rejected(failed_stage): + """Explain why a context without a successfully created isolate cannot close.""" + manager = Mock() + context = ContextIsolateManager(manager) + if failed_stage: + getattr(manager, failed_stage).side_effect = OSError('entry failed') + with pytest.raises(OSError, match='entry failed'): + context.__enter__() + + with pytest.raises(CannotCancelNonExistingIsolateError, match="haven't entered"): + context.__exit__(None, None, None) + + manager.get.return_value.kill.assert_not_called() + + +@pytest.mark.parametrize('nested', [False, True]) +def test_contexts_manage_separate_isolates(monkeypatch, nested): + """Keep separate scopes independent and close nested isolates in reverse order.""" + manager = TemporaryDirectoryManager('.') + events = Mock() + first, second = Mock(), Mock() + events.attach_mock(first, 'first') + events.attach_mock(second, 'second') + read = Mock(side_effect=[b'first', b'second']) + get = Mock(side_effect=[first, second]) + monkeypatch.setattr(manager, 'read', read) + monkeypatch.setattr(manager, 'get', get) + + with manager.scope as outer: + assert outer is first + if nested: + with manager.scope as inner: + assert inner is second + first.kill.assert_not_called() + if not nested: + with manager.scope as later: + assert later is second + + assert read.call_count == 2 + assert get.call_args_list == [call(b'first'), call(b'second')] + assert events.mock_calls == ( + [call.second.kill(), call.first.kill()] + if nested + else [call.first.kill(), call.second.kill()] + ) + + +@pytest.mark.parametrize( + ('method', 'commands'), + [ + ('run', ('',)), + ('run', (' \t世界\n"a b"; x\n',)), + ('chain', ()), + ('chain', ('one',)), + ('chain', ('one', '', ' two\n', 'one')), + ], +) +@pytest.mark.parametrize('token_kind', ['default', 'active', 'cancelled']) +@pytest.mark.parametrize( + 'outcome', + [ + (False, None, None, None), + (True, 0, '\n世界\n', ' warning\n'), + (False, 0, '', ''), + (True, 1, '', 'diagnostic'), + ], +) +def test_closed_execution_delegates_and_cleans_up( + monkeypatch, + method, + commands, + token_kind, + outcome, +): + """Delegate unchanged commands and tokens, returning the result only after cleanup.""" + manager = TemporaryDirectoryManager('.') + events = Mock() + isolate = events.isolate + events.read.return_value = b'\xffstate' + events.get.return_value = isolate + success, returncode, stdout, stderr = outcome + fields = { + 'success': success, + 'returncode': returncode, + 'stdout': stdout, + 'stderr': stderr, + 'extra': object(), + } + result = Mock(**fields) + expected = result if method == 'run' else [result for _ in commands] + getattr(isolate, method).return_value = expected + monkeypatch.setattr(manager, 'read', events.read) + monkeypatch.setattr(manager, 'get', events.get) + token = SimpleToken(cancelled=token_kind == 'cancelled') + + actual = getattr(manager, method)( + *commands, + **({} if token_kind == 'default' else {'token': token}), + ) + + passed_token = getattr(isolate, method).call_args.kwargs['token'] + if token_kind == 'default': + assert isinstance(passed_token, DefaultToken) + else: + assert passed_token is token + assert actual is expected + for result in [actual] if method == 'run' else actual: + assert {name: getattr(result, name) for name in fields} == fields + assert events.mock_calls == [ + call.read(), + call.get(b'\xffstate'), + getattr(call.isolate, method)(*commands, token=passed_token), + call.isolate.kill(), + ] + + +@pytest.mark.parametrize( + 'outcomes', + [ + ((True, 0), (False, 1), (False, None)), + ((False, None), (True, 0), (False, 1)), + ], +) +def test_chain_preserves_mixed_plugin_results(monkeypatch, outcomes): + """Return successful, failed and skipped plugin results unchanged and in order. + + Save the original order separately to detect mutations of the returned list. + """ + manager = TemporaryDirectoryManager('.') + isolate = Mock() + originals = tuple( + Mock(success=success, returncode=code, extra=object()) + for success, code in outcomes + ) + extras = [result.extra for result in originals] + expected = list(originals) + isolate.chain.return_value = expected + monkeypatch.setattr(manager, 'read', Mock(return_value=b'state')) + monkeypatch.setattr(manager, 'get', Mock(return_value=isolate)) + + actual = manager.chain('first', 'second', 'third') + + assert actual is expected + assert len(actual) == len(originals) + assert all(result is original for result, original in zip(actual, originals)) + assert [(result.success, result.returncode) for result in actual] == list(outcomes) + assert [result.extra for result in actual] == extras + isolate.kill.assert_called_once_with() + + +@pytest.mark.parametrize('method', ['run', 'chain']) +@pytest.mark.parametrize('stage', ['read', 'get', 'execute', 'kill']) +def test_closed_execution_errors_and_retry(monkeypatch, method, stage): + """Propagate stage errors, clean up created isolates and allow a fresh retry.""" + manager = TemporaryDirectoryManager('.') + events = Mock() + isolate = events.isolate + events.read.return_value = b'state' + events.get.return_value = isolate + monkeypatch.setattr(manager, 'read', events.read) + monkeypatch.setattr(manager, 'get', events.get) + target = { + 'read': events.read, + 'get': events.get, + 'execute': getattr(isolate, method), + 'kill': isolate.kill, + }[stage] + error = OSError('stage failed') + target.side_effect = error + + with pytest.raises(OSError, match='stage failed') as caught: + getattr(manager, method)('command') + + assert caught.value is error + expected_calls = [call.read()] + if stage != 'read': + expected_calls.append(call.get(b'state')) + if stage in ('execute', 'kill'): + passed_token = getattr(isolate, method).call_args.kwargs['token'] + expected_calls.extend( + [ + getattr(call.isolate, method)('command', token=passed_token), + call.isolate.kill(), + ], + ) + assert events.mock_calls == expected_calls + events.reset_mock() + target.side_effect = None + replacement = Mock() + events.attach_mock(replacement, 'replacement') + events.get.return_value = replacement + + result = getattr(manager, method)('retry') + + assert result is getattr(replacement, method).return_value + passed_token = getattr(replacement, method).call_args.kwargs['token'] + assert events.mock_calls == [ + call.read(), + call.get(b'state'), + getattr(call.replacement, method)('retry', token=passed_token), + call.replacement.kill(), + ] + + +@pytest.mark.parametrize( + ('first_method', 'second_method'), + [('run', 'run'), ('chain', 'chain'), ('run', 'chain'), ('chain', 'run')], +) +def test_closed_calls_read_fresh_state(monkeypatch, first_method, second_method): + """Create a new isolate from fresh state for each independent manager call.""" + manager = TemporaryDirectoryManager('.') + first, second = Mock(), Mock() + read = Mock(side_effect=[b'old', b'new']) + get = Mock(side_effect=[first, second]) + monkeypatch.setattr(manager, 'read', read) + monkeypatch.setattr(manager, 'get', get) + + assert ( + getattr(manager, first_method)('one') + is getattr(first, first_method).return_value + ) + assert ( + getattr(manager, second_method)('two') + is getattr(second, second_method).return_value + ) + assert read.call_count == 2 + assert get.call_args_list == [call(b'old'), call(b'new')] + first.kill.assert_called_once_with() + second.kill.assert_called_once_with() diff --git a/tests/units/abstracts/test_results.py b/tests/units/abstracts/test_results.py new file mode 100644 index 0000000..5d0cc7d --- /dev/null +++ b/tests/units/abstracts/test_results.py @@ -0,0 +1,37 @@ +import pytest + +from throng.abstracts.results import SimpleRunResult + + +@pytest.mark.parametrize('success', [False, True]) +def test_result_without_execution_details(success): + """Expose all result fields even when no execution details are available.""" + result = SimpleRunResult(success) + + assert result.success is success + assert result.returncode is None + assert result.stdout is None + assert result.stderr is None + + +@pytest.mark.parametrize( + ('success', 'returncode', 'stdout', 'stderr'), + [ + (True, 0, 'output', ''), + (False, 2, '', 'error'), + (False, -9, 'partial', 'interrupted'), + (False, None, None, None), + (True, 0, '', ''), + (False, 1, 'Привет\n世界\n', 'ошибка\n'), + ], +) +def test_result_preserves_execution_details(success, returncode, stdout, stderr): + """Preserve the supplied status and output without interpreting them.""" + result = SimpleRunResult(success, returncode, stdout, stderr) + + assert (result.success, result.returncode, result.stdout, result.stderr) == ( + success, + returncode, + stdout, + stderr, + ) diff --git a/tests/units/extensions/__init__.py b/tests/units/extensions/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/tests/units/extensions/local/__init__.py b/tests/units/extensions/local/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/tests/units/extensions/local/test_isolate.py b/tests/units/extensions/local/test_isolate.py new file mode 100644 index 0000000..3f3c2f4 --- /dev/null +++ b/tests/units/extensions/local/test_isolate.py @@ -0,0 +1,248 @@ +from pathlib import Path +from threading import Lock +from unittest.mock import MagicMock, Mock, call + +import pytest +from cantok import DefaultToken, SimpleToken + +from throng.abstracts.results import SimpleRunResult +from throng.errors import CannotInstallDependencyError +from throng.extensions.local.isolate import LocalIsolate + + +@pytest.mark.parametrize('path_kind', ['default', 'existing', 'missing']) +def test_constructor_preserves_resources(tmp_path, path_kind): + """Store the supplied lock and path without creating or validating directories.""" + lock = Lock() + options = { + 'default': {}, + 'existing': {'path': tmp_path}, + 'missing': {'path': tmp_path / 'missing'}, + } + + isolate = LocalIsolate(lock, **options[path_kind]) + + assert isolate.lock is lock + assert isolate.path == options[path_kind].get('path', Path()) + assert list(tmp_path.iterdir()) == [] + + +@pytest.mark.parametrize( + 'command', + ['', ' ', 'command', 'Привет', 'one\ntwo', '"a b"; x'], +) +@pytest.mark.parametrize('token_kind', ['default', 'active', 'cancelled']) +def test_run_forwards_command_and_token(tmp_path, monkeypatch, command, token_kind): + """Forward commands with the source directory, capture options and cancellation.""" + isolate = LocalIsolate(Lock(), tmp_path) + run = Mock(return_value=SimpleRunResult(True)) + monkeypatch.setattr('throng.extensions.local.isolate.run', run) + token = SimpleToken(cancelled=token_kind == 'cancelled') + + result = isolate.run( + command, + **({} if token_kind == 'default' else {'token': token}), + ) + + passed_token = run.call_args.kwargs['token'] + if token_kind == 'default': + assert isinstance(passed_token, DefaultToken) + else: + assert passed_token is token + run.assert_called_once_with( + command, + token=passed_token, + catch_output=True, + catch_exceptions=True, + directory=tmp_path, + ) + assert result is run.return_value + + +@pytest.mark.parametrize( + 'outcome', + [ + (True, 0, '\n世界\n', ' warning\n', False), + (False, 1, '', 'error', False), + (False, None, None, None, False), + (False, -9, 'partial', '', True), + (False, 0, '', '', True), + (True, 1, 'output', 'diagnostic', False), + ], +) +def test_run_preserves_result_and_holds_lock(tmp_path, monkeypatch, outcome): + """Protect execution with the lock while retaining all plugin result data.""" + lock = Lock() + isolate = LocalIsolate(lock, tmp_path) + success, code, stdout, stderr, killed = outcome + fields = { + 'success': success, + 'returncode': code, + 'stdout': stdout, + 'stderr': stderr, + 'killed_by_token': killed, + } + expected = Mock(**fields) + + def execute(*_args, **_kwargs): + assert lock.locked() + return expected + + monkeypatch.setattr('throng.extensions.local.isolate.run', execute) + + assert isolate.run('command') is expected + assert {name: getattr(expected, name) for name in fields} == fields + assert not lock.locked() + + +@pytest.mark.parametrize('error_type', [RuntimeError, OSError, KeyboardInterrupt]) +def test_execution_error_releases_lock(tmp_path, monkeypatch, error_type): + """Propagate executor errors without blocking subsequent commands.""" + lock = Lock() + isolate = LocalIsolate(lock, tmp_path) + error = error_type('execution failed') + expected = SimpleRunResult(True) + run = Mock(side_effect=[error, expected]) + monkeypatch.setattr('throng.extensions.local.isolate.run', run) + + with pytest.raises(error_type) as caught: + isolate.run('failing') + + assert caught.value is error + assert not lock.locked() + assert isolate.run('retry') is expected + assert not lock.locked() + + +@pytest.mark.parametrize('error_type', [RuntimeError, KeyboardInterrupt]) +def test_lock_failure_prevents_execution(tmp_path, monkeypatch, error_type): + """Do not reach the executor if entering the command lock fails.""" + lock = MagicMock() + error = error_type('lock failed') + lock.__enter__.side_effect = error + isolate = LocalIsolate(lock, tmp_path) + run = Mock() + monkeypatch.setattr('throng.extensions.local.isolate.run', run) + + with pytest.raises(error_type) as caught: + isolate.run('command') + + assert caught.value is error + run.assert_not_called() + lock.__exit__.assert_not_called() + + +@pytest.mark.parametrize('populated', [False, True]) +def test_read_returns_empty_state_without_changes(tmp_path, monkeypatch, populated): + """Return empty local state without reading or modifying the user's files.""" + isolate = LocalIsolate(Lock(), tmp_path) + + for content in (b'original', b'changed'): + if populated: + (tmp_path / 'file').write_bytes(content) + operations = { + name: Mock( + side_effect=AssertionError(f'Local state must not read files: {name}'), + ) + for name in ('builtins.open', 'io.open', 'os.scandir', 'os.listdir') + } + with monkeypatch.context() as patcher: + for name, operation in operations.items(): + patcher.setattr(name, operation) + assert isolate.read() == b'' + for operation in operations.values(): + operation.assert_not_called() + if populated: + assert (tmp_path / 'file').read_bytes() == content + assert list(tmp_path.iterdir()) == ([tmp_path / 'file'] if populated else []) + + +@pytest.mark.parametrize('repetitions', [1, 3]) +@pytest.mark.parametrize('populated', [False, True]) +def test_kill_preserves_source_files(tmp_path, repetitions, populated): + """Leave the original local directory intact when releasing an isolate.""" + if populated: + (tmp_path / 'nested').mkdir() + (tmp_path / 'nested' / 'file').write_bytes(b'keep') + isolate = LocalIsolate(Lock(), tmp_path) + + for _ in range(repetitions): + assert isolate.kill() is None + + assert tmp_path.is_dir() + if populated: + assert (tmp_path / 'nested' / 'file').read_bytes() == b'keep' + else: + assert list(tmp_path.iterdir()) == [] + + +@pytest.mark.parametrize( + 'packages', + [(), ('package',), ('one', 'two', 'one'), ('pkg==1.2', 'pkg[extra]')], +) +@pytest.mark.parametrize( + ('returncode', 'stderr'), + [(None, None), (0, ''), (0, 'installer warning'), (1, ''), (-9, '')], +) +def test_install_preserves_package_order( + tmp_path, + monkeypatch, + packages, + returncode, + stderr, +): + """Install in order, trusting success regardless of the exit code or diagnostics.""" + isolate = LocalIsolate(Lock(), tmp_path) + run = Mock(return_value=SimpleRunResult(True, returncode, 'installed', stderr)) + monkeypatch.setattr(isolate, 'run', run) + + assert isolate.install(*packages) is None + assert run.call_args_list == [ + call(f'pip install {package}') for package in packages + ] + + +@pytest.mark.parametrize('position', [0, 1, 2]) +@pytest.mark.parametrize('returncode', [0, 1, None]) +def test_install_stops_at_unsuccessful_result( + tmp_path, + monkeypatch, + position, + returncode, +): + """Stop on the first unsuccessful install using success rather than the exit code.""" + isolate = LocalIsolate(Lock(), tmp_path) + packages = ('one', 'two', 'three') + run = Mock( + side_effect=[SimpleRunResult(True)] * position + + [SimpleRunResult(False, returncode)], + ) + monkeypatch.setattr(isolate, 'run', run) + + with pytest.raises(CannotInstallDependencyError): + isolate.install(*packages) + + assert run.call_args_list == [ + call(f'pip install {package}') for package in packages[: position + 1] + ] + + +@pytest.mark.parametrize('position', [0, 1, 2]) +@pytest.mark.parametrize('error_type', [RuntimeError, OSError, KeyboardInterrupt]) +def test_install_preserves_execution_exception( + tmp_path, + monkeypatch, + position, + error_type, +): + """Propagate installation execution errors without trying later packages.""" + isolate = LocalIsolate(Lock(), tmp_path) + error = error_type('installer failed') + run = Mock(side_effect=[SimpleRunResult(True)] * position + [error]) + monkeypatch.setattr(isolate, 'run', run) + + with pytest.raises(error_type) as caught: + isolate.install('one', 'two', 'three') + + assert caught.value is error + assert run.call_count == position + 1 diff --git a/tests/units/extensions/local/test_manager.py b/tests/units/extensions/local/test_manager.py new file mode 100644 index 0000000..f47ea2e --- /dev/null +++ b/tests/units/extensions/local/test_manager.py @@ -0,0 +1,197 @@ +from concurrent.futures import ThreadPoolExecutor +from pathlib import Path +from threading import Event +from unittest.mock import MagicMock, Mock + +import pytest + +from throng.abstracts.results import SimpleRunResult +from throng.extensions.local.isolate import LocalIsolate +from throng.extensions.local.manager import LocalManager + + +@pytest.mark.parametrize('as_string', [False, True]) +@pytest.mark.parametrize('exclude', [None, [], ['cache/', '*.tmp', '!keep.tmp']]) +def test_manager_keeps_source_settings(tmp_path, as_string, exclude): + """Use shared manager initialization for paths and file exclusions.""" + expected_exclude = None if exclude is None else exclude.copy() + manager = LocalManager(str(tmp_path) if as_string else tmp_path, exclude) + + assert manager.path == tmp_path + assert isinstance(manager.path, Path) + assert manager.exclude == expected_exclude + assert not manager.lock.locked() + + +@pytest.mark.parametrize('same_path', [False, True]) +def test_managers_have_independent_locks(tmp_path, same_path): + """Keep concurrency limits independent even for managers using the same source.""" + first = LocalManager(tmp_path, None) + second = LocalManager(tmp_path if same_path else tmp_path / 'other', None) + + assert first.lock is not second.lock + with first.lock: + assert not second.lock.locked() + + +@pytest.mark.parametrize( + 'state', + [b'', b'state', b'\x00\xff', b'not really a tar archive'], +) +@pytest.mark.parametrize('same_state', [False, True]) +def test_get_ignores_state_and_shares_manager_lock(tmp_path, state, same_state): + """Create distinct isolates using the original directory and one manager lock.""" + manager = LocalManager(tmp_path, None) + + first = manager.get(state) + second = manager.get(state if same_state else b'different state') + + assert isinstance(first, LocalIsolate) + assert isinstance(second, LocalIsolate) + assert first is not second + assert first.path is second.path is manager.path + assert first.lock is second.lock is manager.lock + assert list(tmp_path.iterdir()) == [] + + +@pytest.mark.parametrize('exclude', [None, [], ['*']]) +@pytest.mark.parametrize('populated', [False, True]) +@pytest.mark.parametrize('operation', ['read', 'get']) +def test_state_operations_do_not_snapshot_or_modify_files( + tmp_path, + monkeypatch, + exclude, + populated, + operation, +): + """Read state and recreate local isolates without snapshotting or changing files.""" + manager = LocalManager(tmp_path, exclude) + if populated: + (tmp_path / 'nested').mkdir() + for content in (b'original', b'changed'): + if populated: + (tmp_path / 'nested' / 'file').write_bytes(content) + operations = { + name: Mock( + side_effect=AssertionError(f'Local state must not read files: {name}'), + ) + for name in ('builtins.open', 'io.open', 'os.scandir', 'os.listdir') + } + with monkeypatch.context() as patcher: + for name, spy in operations.items(): + patcher.setattr(name, spy) + if operation == 'read': + assert manager.read() == b'' + else: + assert isinstance(manager.get(b'ignored'), LocalIsolate) + for spy in operations.values(): + spy.assert_not_called() + if populated: + assert (tmp_path / 'nested' / 'file').read_bytes() == content + assert sorted( + path.relative_to(tmp_path).as_posix() for path in tmp_path.rglob('*') + ) == (['nested', 'nested/file'] if populated else []) + + +@pytest.mark.parametrize('change', ['modify', 'delete']) +def test_get_keeps_current_source_contents(tmp_path, change): + """Keep current local files rather than restoring an earlier snapshot.""" + source = tmp_path / 'file' + source.write_bytes(b'old') + manager = LocalManager(tmp_path, None) + state = manager.read() + if change == 'modify': + source.write_bytes(b'new') + else: + source.unlink() + + isolate = manager.get(state) + + assert isolate.path == tmp_path + if change == 'modify': + assert (isolate.path / 'file').read_bytes() == b'new' + else: + assert not (isolate.path / 'file').exists() + + +@pytest.mark.parametrize('fail_first', [False, True]) +def test_isolates_of_one_manager_serialize_commands(tmp_path, monkeypatch, fail_first): + """Release the shared lock before another isolate executes, including on failure. + + Events confirm the second worker attempts acquisition while the first owns it. + Acquisition times out so a leaked lock fails instead of blocking pool shutdown. + """ + manager = LocalManager(tmp_path, None) + lock = manager.lock + first_entered, second_attempted, release_first, second_entered = ( + Event() for _ in range(4) + ) + observed_lock = MagicMock() + + def acquire(): + if first_entered.is_set(): + second_attempted.set() + assert lock.acquire(timeout=5), 'The shared lock was not released.' + + observed_lock.__enter__.side_effect = acquire + observed_lock.__exit__.side_effect = lambda *_args: lock.release() + monkeypatch.setattr(manager, 'lock', observed_lock) + first, second = manager.get(b''), manager.get(b'') + + def execute(command, **_kwargs): + if command == 'first': + first_entered.set() + assert release_first.wait(5) + if fail_first: + raise RuntimeError('first failed') + else: + second_entered.set() + return SimpleRunResult(True) + + monkeypatch.setattr('throng.extensions.local.isolate.run', execute) + with ThreadPoolExecutor(max_workers=2) as pool: + first_future = pool.submit(first.run, 'first') + try: + assert first_entered.wait(5) + second_future = pool.submit(second.run, 'second') + assert second_attempted.wait(5) + assert not second_entered.is_set() + finally: + release_first.set() + if fail_first: + with pytest.raises(RuntimeError, match='first failed'): + first_future.result(timeout=5) + else: + assert first_future.result(timeout=5).success + assert second_future.result(timeout=5).success + assert not lock.locked() + + +@pytest.mark.parametrize('same_path', [False, True]) +def test_different_managers_can_execute_together(tmp_path, monkeypatch, same_path): + """Let independent managers enter the executor before either command finishes.""" + first = LocalManager(tmp_path, None).get(b'') + second = LocalManager(tmp_path if same_path else tmp_path / 'other', None).get(b'') + entered = {'first': Event(), 'second': Event()} + release = Event() + + def execute(command, **_kwargs): + entered[command].set() + assert release.wait(5) + return SimpleRunResult(True) + + monkeypatch.setattr( + 'throng.extensions.local.isolate.run', + Mock(side_effect=execute), + ) + with ThreadPoolExecutor(max_workers=2) as pool: + futures = [ + pool.submit(isolate.run, command) + for isolate, command in ((first, 'first'), (second, 'second')) + ] + try: + assert entered['first'].wait(5) + assert entered['second'].wait(5) + finally: + release.set() + assert all(future.result(timeout=5).success for future in futures) diff --git a/tests/units/extensions/temporary_directory/__init__.py b/tests/units/extensions/temporary_directory/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/tests/units/extensions/temporary_directory/test_errors.py b/tests/units/extensions/temporary_directory/test_errors.py new file mode 100644 index 0000000..fea5fc2 --- /dev/null +++ b/tests/units/extensions/temporary_directory/test_errors.py @@ -0,0 +1,11 @@ +from throng.extensions.temporary_directory.errors import DirectoryDoesNotExistError + + +def test_destroyed_directory_has_specific_error_type(): + """Let callers distinguish a destroyed directory from other execution failures.""" + assert issubclass(DirectoryDoesNotExistError, Exception) + assert DirectoryDoesNotExistError is not Exception + assert ( + DirectoryDoesNotExistError.__module__ + == 'throng.extensions.temporary_directory.errors' + ) diff --git a/tests/units/extensions/temporary_directory/test_isolate.py b/tests/units/extensions/temporary_directory/test_isolate.py new file mode 100644 index 0000000..852a7b7 --- /dev/null +++ b/tests/units/extensions/temporary_directory/test_isolate.py @@ -0,0 +1,699 @@ +import tarfile +from concurrent.futures import ThreadPoolExecutor +from contextlib import nullcontext +from io import BytesIO +from pathlib import Path +from shutil import rmtree +from threading import Event +from unittest.mock import DEFAULT, MagicMock, Mock, call + +import pytest +from cantok import DefaultToken, SimpleToken + +from throng.abstracts.results import SimpleRunResult +from throng.errors import CannotInstallDependencyError +from throng.extensions.temporary_directory.errors import DirectoryDoesNotExistError +from throng.extensions.temporary_directory.isolate import TemporaryDirectoryIsolate +from throng.extensions.temporary_directory.read import read_directory + + +@pytest.mark.parametrize( + 'files', + [(), (('file', b''),), (('nested/файл', b'\x00\xff'), ('with spaces', b'text'))], +) +@pytest.mark.parametrize('exclude', [None, [], ['cache/', '*.tmp', '!keep.tmp']]) +def test_constructor_restores_snapshot(files, exclude): + """Create a live temporary directory containing exactly the supplied files.""" + expected_exclude = None if exclude is None else exclude.copy() + buffer = BytesIO() + with tarfile.open(fileobj=buffer, mode='w') as archive: + for name, data in files: + member = tarfile.TarInfo(name) + member.size = len(data) + archive.addfile(member, BytesIO(data)) + + isolate = TemporaryDirectoryIsolate(buffer.getvalue(), exclude) + try: + assert isinstance(isolate.path, Path) + assert isolate.path == Path(isolate.directory.name) + assert isolate.path.is_dir() + assert isolate.used is False + assert not isolate.lock.locked() + assert isolate.exclude == expected_exclude + assert { + path.relative_to(isolate.path).as_posix(): path.read_bytes() + for path in isolate.path.rglob('*') + if path.is_file() + } == dict(files) + finally: + isolate.directory.cleanup() + if isolate.lock.locked(): + isolate.lock.release() + + +@pytest.mark.parametrize( + ('mode', 'module'), + [('w', None), ('w:gz', 'gzip'), ('w:bz2', 'bz2'), ('w:xz', 'lzma')], +) +def test_constructor_accepts_supported_tar_formats(mode, module): + """Autodetect supported TAR compression when restoring a state.""" + if module: + pytest.importorskip(module) + buffer = BytesIO() + with tarfile.open(fileobj=buffer, mode=mode) as archive: + member = tarfile.TarInfo('file') + member.size = 4 + archive.addfile(member, BytesIO(b'data')) + + isolate = TemporaryDirectoryIsolate(buffer.getvalue(), None) + try: + assert (isolate.path / 'file').read_bytes() == b'data' + finally: + isolate.directory.cleanup() + if isolate.lock.locked(): + isolate.lock.release() + + +@pytest.mark.parametrize('change', ['add', 'replace', 'nested']) +def test_set_state_updates_live_files(tmp_path, change): + """Apply new snapshot contents to a live isolate without changing the source.""" + (tmp_path / 'file').write_bytes(b'original') + isolate = TemporaryDirectoryIsolate(read_directory(tmp_path, None), None) + name = {'add': 'new', 'replace': 'file', 'nested': 'nested/new'}[change] + buffer = BytesIO() + with tarfile.open(fileobj=buffer, mode='w') as archive: + member = tarfile.TarInfo(name) + member.size = 7 + archive.addfile(member, BytesIO(b'changed')) + try: + assert isolate.set_state(buffer.getvalue()) is None + assert (isolate.path / name).read_bytes() == b'changed' + assert (tmp_path / 'file').read_bytes() == b'original' + finally: + isolate.directory.cleanup() + if isolate.lock.locked(): + isolate.lock.release() + + +@pytest.mark.parametrize('exclude', [None, [], ['cache/', '*.tmp', '!keep.tmp']]) +def test_read_uses_own_directory_and_exclusions(tmp_path, monkeypatch, exclude): + """Read the isolate directory with its exclusions and preserve the snapshot bytes.""" + expected_exclude = None if exclude is None else exclude.copy() + isolate = TemporaryDirectoryIsolate(read_directory(tmp_path, None), exclude) + read = Mock(return_value=b'\x00\xffsnapshot') + monkeypatch.setattr( + 'throng.extensions.temporary_directory.isolate.read_directory', + read, + ) + try: + assert isolate.read() is read.return_value + read.assert_called_once_with(isolate.path, expected_exclude) + finally: + isolate.directory.cleanup() + if isolate.lock.locked(): + isolate.lock.release() + + +@pytest.mark.parametrize('change', ['add', 'modify', 'delete']) +def test_read_captures_changes_for_an_independent_isolate(tmp_path, change): + """Transfer current isolate state without sharing future modifications or cleanup.""" + (tmp_path / 'file').write_bytes(b'old') + first = TemporaryDirectoryIsolate(read_directory(tmp_path, None), None) + try: + if change == 'add': + (first.path / 'new').write_bytes(b'new') + elif change == 'modify': + (first.path / 'file').write_bytes(b'new') + else: + (first.path / 'file').unlink() + state = first.read() + assert not first.lock.locked() + expected = { + 'add': {'file': b'old', 'new': b'new'}, + 'modify': {'file': b'new'}, + 'delete': {}, + }[change] + with tarfile.open(fileobj=BytesIO(state)) as archive: + assert { + member.name: archive.extractfile(member).read() + for member in archive.getmembers() + } == expected + second = TemporaryDirectoryIsolate(state, None) + try: + assert { + path.name: path.read_bytes() for path in second.path.iterdir() + } == expected + (first.path / 'file').write_bytes(b'changed again') + first.kill() + assert { + path.name: path.read_bytes() for path in second.path.iterdir() + } == expected + assert (tmp_path / 'file').read_bytes() == b'old' + finally: + second.directory.cleanup() + if second.lock.locked(): + second.lock.release() + finally: + first.directory.cleanup() + if first.lock.locked(): + first.lock.release() + + +@pytest.mark.parametrize('created_later', [False, True]) +def test_read_excludes_existing_and_new_files(tmp_path, created_later): + """Apply exclusions to snapshots without filtering the initial extraction.""" + (tmp_path / 'keep').write_bytes(b'keep') + if not created_later: + (tmp_path / 'drop.tmp').write_bytes(b'drop') + isolate = TemporaryDirectoryIsolate(read_directory(tmp_path, None), ['*.tmp']) + try: + if created_later: + (isolate.path / 'drop.tmp').write_bytes(b'drop') + assert (isolate.path / 'drop.tmp').is_file() + with tarfile.open(fileobj=BytesIO(isolate.read())) as archive: + assert archive.getnames() == ['keep'] + assert archive.extractfile('keep').read() == b'keep' + finally: + isolate.directory.cleanup() + if isolate.lock.locked(): + isolate.lock.release() + + +@pytest.mark.parametrize( + 'state', + [b'', b'not an archive', b'file' + bytes(508), b'\x1f\x8bcorrupt'], +) +def test_invalid_state_releases_lock(tmp_path, state): + """Reject unreadable archives without retaining the isolate lock.""" + valid_state = read_directory(tmp_path, None) + isolate = TemporaryDirectoryIsolate(valid_state, None) + try: + with pytest.raises((tarfile.ReadError, EOFError)): + isolate.set_state(state) + assert not isolate.lock.locked() + assert isolate.used is False + isolate.set_state(valid_state) + finally: + isolate.directory.cleanup() + if isolate.lock.locked(): + isolate.lock.release() + + +@pytest.mark.parametrize('error_type', [OSError, PermissionError]) +def test_extraction_error_closes_archive(tmp_path, monkeypatch, error_type): + """Close the archive and release the lock when restoring files fails.""" + isolate = TemporaryDirectoryIsolate(read_directory(tmp_path, None), None) + error = error_type('extraction failed') + context = MagicMock() + archive = context.__enter__.return_value + archive.extractall.side_effect = error + open_archive = Mock(return_value=context) + try: + with monkeypatch.context() as patcher: + patcher.setattr( + 'throng.extensions.temporary_directory.isolate.tarfile.open', + open_archive, + ) + with pytest.raises(error_type) as caught: + isolate.set_state(b'opaque state') + assert caught.value is error + assert open_archive.call_args.kwargs['mode'] == 'r:*' + assert ( + open_archive.call_args.kwargs['fileobj'].getvalue() == b'opaque state' + ) + archive.extractall.assert_called_once_with(path=isolate.path) + context.__exit__.assert_called_once() + assert context.__exit__.call_args.args[1] is error + assert not isolate.lock.locked() + finally: + isolate.directory.cleanup() + if isolate.lock.locked(): + isolate.lock.release() + + +@pytest.mark.parametrize( + 'command', + ['', ' ', 'command', 'Привет', 'one\ntwo', '"a b"; x'], +) +@pytest.mark.parametrize('token_kind', ['default', 'active', 'cancelled']) +def test_run_forwards_command_and_token(tmp_path, monkeypatch, command, token_kind): + """Execute unchanged commands in the temporary directory with cancellation.""" + isolate = TemporaryDirectoryIsolate(read_directory(tmp_path, None), None) + run = Mock(return_value=SimpleRunResult(True)) + monkeypatch.setattr('throng.extensions.temporary_directory.isolate.run', run) + token = SimpleToken(cancelled=token_kind == 'cancelled') + try: + result = isolate.run( + command, + **({} if token_kind == 'default' else {'token': token}), + ) + passed_token = run.call_args.kwargs['token'] + if token_kind == 'default': + assert isinstance(passed_token, DefaultToken) + else: + assert passed_token is token + run.assert_called_once_with( + command, + token=passed_token, + catch_output=True, + catch_exceptions=True, + directory=isolate.path, + ) + assert result is run.return_value + finally: + isolate.directory.cleanup() + if isolate.lock.locked(): + isolate.lock.release() + + +@pytest.mark.parametrize( + 'outcome', + [ + (True, 0, '\n世界\n', ' warning\n', False), + (False, 1, '', 'error', False), + (False, None, None, None, False), + (False, -9, 'partial', '', True), + (False, 0, '', '', True), + (True, 1, 'output', 'diagnostic', False), + ], +) +def test_run_preserves_plugin_result(tmp_path, monkeypatch, outcome): + """Preserve unsuccessful and cancelled results, including plugin-specific fields.""" + isolate = TemporaryDirectoryIsolate(read_directory(tmp_path, None), None) + success, code, stdout, stderr, killed = outcome + fields = { + 'success': success, + 'returncode': code, + 'stdout': stdout, + 'stderr': stderr, + 'killed_by_token': killed, + } + expected = Mock(**fields) + monkeypatch.setattr( + 'throng.extensions.temporary_directory.isolate.run', + Mock(return_value=expected), + ) + try: + assert isolate.run('command') is expected + assert {name: getattr(expected, name) for name in fields} == fields + finally: + isolate.directory.cleanup() + if isolate.lock.locked(): + isolate.lock.release() + + +@pytest.mark.parametrize('error_type', [RuntimeError, OSError, KeyboardInterrupt]) +def test_execution_error_allows_retry(tmp_path, monkeypatch, error_type): + """Keep the isolate usable and its lock free after an executor exception.""" + isolate = TemporaryDirectoryIsolate(read_directory(tmp_path, None), None) + error = error_type('execution failed') + expected = SimpleRunResult(True) + run = Mock(side_effect=[error, expected]) + monkeypatch.setattr('throng.extensions.temporary_directory.isolate.run', run) + try: + with pytest.raises(error_type) as caught: + isolate.run('failing') + assert caught.value is error + assert not isolate.lock.locked() + assert isolate.used is False + assert isolate.run('retry') is expected + finally: + isolate.directory.cleanup() + if isolate.lock.locked(): + isolate.lock.release() + + +@pytest.mark.parametrize( + 'packages', + [(), ('package',), ('one', 'two', 'one'), ('pkg==1.2', 'pkg[extra]')], +) +@pytest.mark.parametrize( + ('returncode', 'stderr'), + [(None, None), (0, ''), (0, 'installer warning'), (1, ''), (-9, '')], +) +def test_install_preserves_package_order( + tmp_path, + monkeypatch, + packages, + returncode, + stderr, +): + """Install in order, trusting success regardless of the exit code or diagnostics.""" + isolate = TemporaryDirectoryIsolate(read_directory(tmp_path, None), None) + run = Mock(return_value=SimpleRunResult(True, returncode, 'installed', stderr)) + monkeypatch.setattr(isolate, 'run', run) + try: + assert isolate.install(*packages) is None + assert run.call_args_list == [ + call(f'pip install {package}') for package in packages + ] + finally: + isolate.directory.cleanup() + if isolate.lock.locked(): + isolate.lock.release() + + +@pytest.mark.parametrize('position', [0, 1, 2]) +@pytest.mark.parametrize('returncode', [0, 1, None]) +def test_install_stops_at_unsuccessful_result( + tmp_path, + monkeypatch, + position, + returncode, +): + """Stop installing at the first false success flag regardless of the exit code.""" + isolate = TemporaryDirectoryIsolate(read_directory(tmp_path, None), None) + packages = ('one', 'two', 'three') + run = Mock( + side_effect=[SimpleRunResult(True)] * position + + [SimpleRunResult(False, returncode)], + ) + monkeypatch.setattr(isolate, 'run', run) + try: + with pytest.raises(CannotInstallDependencyError): + isolate.install(*packages) + assert run.call_args_list == [ + call(f'pip install {package}') for package in packages[: position + 1] + ] + finally: + isolate.directory.cleanup() + if isolate.lock.locked(): + isolate.lock.release() + + +@pytest.mark.parametrize('position', [0, 1, 2]) +@pytest.mark.parametrize('error_type', [RuntimeError, OSError, KeyboardInterrupt]) +def test_install_preserves_execution_exception( + tmp_path, + monkeypatch, + position, + error_type, +): + """Keep installer exceptions intact and avoid executing later packages.""" + isolate = TemporaryDirectoryIsolate(read_directory(tmp_path, None), None) + error = error_type('installer failed') + run = Mock(side_effect=[SimpleRunResult(True)] * position + [error]) + monkeypatch.setattr(isolate, 'run', run) + try: + with pytest.raises(error_type) as caught: + isolate.install('one', 'two', 'three') + assert caught.value is error + assert run.call_count == position + 1 + finally: + isolate.directory.cleanup() + if isolate.lock.locked(): + isolate.lock.release() + + +@pytest.mark.parametrize('tree', ['empty', 'populated', 'already_removed']) +@pytest.mark.parametrize('repetitions', [1, 3]) +def test_kill_removes_only_own_directory(tmp_path, tree, repetitions): + """Safely repeat cleanup without deleting source files or another isolate.""" + (tmp_path / 'source').write_bytes(b'keep') + state = read_directory(tmp_path, None) + first = TemporaryDirectoryIsolate(state, None) + try: + second = TemporaryDirectoryIsolate(state, None) + try: + (first.path / 'source').unlink() + if tree == 'populated': + (first.path / 'nested').mkdir() + (first.path / 'nested' / 'file').write_bytes(b'data') + elif tree == 'already_removed': + rmtree(first.path) + for _ in range(repetitions): + assert first.kill() is None + assert not first.lock.locked() + first.__del__() + assert first.used is True + assert not first.path.exists() + assert (second.path / 'source').read_bytes() == b'keep' + assert (tmp_path / 'source').read_bytes() == b'keep' + finally: + second.directory.cleanup() + if second.lock.locked(): + second.lock.release() + finally: + first.directory.cleanup() + if first.lock.locked(): + first.lock.release() + + +@pytest.mark.parametrize( + ('operation', 'argument_kind', 'message'), + [ + ('run', 'command', 'reuse'), + ('read', 'none', 're-read'), + ('set_state', 'invalid_state', 'reuse'), + ('set_state', 'valid_state', 'reuse'), + ('install', 'one_package', 'reuse'), + ('install', 'several_packages', 'reuse'), + ], +) +def test_destroyed_isolate_rejects_work_before_external_calls( + tmp_path, + monkeypatch, + operation, + argument_kind, + message, +): + """Reject destroyed isolates before running, reading, restoring or installing.""" + state = read_directory(tmp_path, None) + arguments = { + 'command': ('command',), + 'none': (), + 'invalid_state': (b'invalid',), + 'valid_state': (state,), + 'one_package': ('one',), + 'several_packages': ('one', 'two'), + }[argument_kind] + isolate = TemporaryDirectoryIsolate(state, None) + try: + isolate.kill() + assert not isolate.lock.locked() + run, read, open_archive = Mock(), Mock(), Mock() + monkeypatch.setattr('throng.extensions.temporary_directory.isolate.run', run) + monkeypatch.setattr( + 'throng.extensions.temporary_directory.isolate.read_directory', + read, + ) + monkeypatch.setattr( + 'throng.extensions.temporary_directory.isolate.tarfile.open', + open_archive, + ) + + with pytest.raises(DirectoryDoesNotExistError, match=message): + getattr(isolate, operation)(*arguments) + + run.assert_not_called() + read.assert_not_called() + open_archive.assert_not_called() + assert not isolate.path.exists() + assert not isolate.lock.locked() + finally: + isolate.directory.cleanup() + if isolate.lock.locked(): + isolate.lock.release() + + +def test_cleanup_can_be_retried_after_failure(tmp_path, monkeypatch): + """Report cleanup failure without retaining the lock or preventing a retry.""" + isolate = TemporaryDirectoryIsolate(read_directory(tmp_path, None), None) + cleanup = isolate.directory.cleanup + error = OSError('cleanup failed') + attempts = Mock(wraps=cleanup, side_effect=[error, DEFAULT]) + try: + with monkeypatch.context() as patcher: + patcher.setattr(isolate.directory, 'cleanup', attempts) + with pytest.raises(OSError, match='cleanup failed') as caught: + isolate.kill() + assert caught.value is error + attempts.assert_called_once_with() + assert not isolate.lock.locked() + assert isolate.path.is_dir() + isolate.kill() + assert attempts.call_args_list == [call(), call()] + assert not isolate.path.exists() + assert isolate.used is True + finally: + cleanup() + if isolate.lock.locked(): + isolate.lock.release() + + +@pytest.mark.parametrize('operation', ['run', 'read', 'set_state', 'kill']) +@pytest.mark.parametrize('fail', [False, True]) +def test_operations_hold_and_release_lock(tmp_path, monkeypatch, operation, fail): + """Protect each directory operation and release the lock on success or failure. + + Successful cleanup must mark the isolate destroyed before another worker enters. + Cleanup bypasses kill and releases leaked locks so a failed assertion cannot hang. + """ + state = read_directory(tmp_path, None) + isolate = TemporaryDirectoryIsolate(state, None) + error = OSError('operation failed') + lock = isolate.lock + observed_lock = MagicMock(wraps=lock) + + def acquire(): + assert lock.acquire(timeout=5), 'The isolate lock was not released.' + + def release(*_args): + try: + if operation == 'kill' and not fail: + assert isolate.used is True + finally: + lock.release() + + observed_lock.__enter__.side_effect = acquire + observed_lock.__exit__.side_effect = release + monkeypatch.setattr(isolate, 'lock', observed_lock) + + def dependency(*_args, **_kwargs): + assert isolate.lock.locked() + if fail: + raise error + return b'state' if operation == 'read' else SimpleRunResult(True) + + try: + assert not isolate.lock.locked() + with monkeypatch.context() as patcher: + if operation == 'kill': + patcher.setattr(isolate.directory, 'cleanup', dependency) + else: + target = { + 'run': 'throng.extensions.temporary_directory.isolate.run', + 'read': 'throng.extensions.temporary_directory.isolate.read_directory', + 'set_state': 'tarfile.TarFile.extractall', + }[operation] + patcher.setattr(target, dependency) + arguments = { + 'run': ('command',), + 'read': (), + 'set_state': (state,), + 'kill': (), + }[operation] + expectation = ( + pytest.raises(OSError, match='operation failed') + if fail + else nullcontext() + ) + with expectation as caught: + getattr(isolate, operation)(*arguments) + if fail: + assert caught.value is error + assert not isolate.lock.locked() + finally: + isolate.directory.cleanup() + if isolate.lock.locked(): + isolate.lock.release() + + +@pytest.mark.parametrize('same_state', [False, True]) +def test_isolates_have_independent_locks(tmp_path, same_state): + """Keep separate isolates independently lockable regardless of snapshot contents.""" + state = read_directory(tmp_path, None) + first = TemporaryDirectoryIsolate(state, None) + try: + if not same_state: + (tmp_path / 'new').write_bytes(b'different snapshot') + state = read_directory(tmp_path, None) + second = TemporaryDirectoryIsolate(state, None) + try: + assert first.lock is not second.lock + assert not first.lock.locked() + assert not second.lock.locked() + with first.lock: + assert not second.lock.locked() + finally: + second.directory.cleanup() + if second.lock.locked(): + second.lock.release() + finally: + first.directory.cleanup() + if first.lock.locked(): + first.lock.release() + + +@pytest.mark.parametrize('operation', ['run', 'read', 'set_state']) +@pytest.mark.parametrize('kill_first', [False, True]) +def test_operations_and_kill_share_lock(tmp_path, monkeypatch, operation, kill_first): + """Serialize work and cleanup, rejecting operations queued behind kill. + + Events confirm the second operation attempts entry while the first holds the lock. + Acquisition times out so a leaked lock fails instead of blocking pool shutdown. + Final cleanup bypasses tested methods and releases any leaked lock after workers stop. + """ + state = read_directory(tmp_path, None) + isolate = TemporaryDirectoryIsolate(state, None) + lock, cleanup = isolate.lock, isolate.directory.cleanup + entered, second_attempted, release = (Event() for _ in range(3)) + observed_lock = MagicMock() + + def acquire(): + if entered.is_set(): + second_attempted.set() + assert lock.acquire(timeout=5), 'The isolate lock was not released.' + + observed_lock.__enter__.side_effect = acquire + observed_lock.__exit__.side_effect = lambda *_args: lock.release() + monkeypatch.setattr(isolate, 'lock', observed_lock) + + dependency = Mock( + return_value=b'state' if operation == 'read' else SimpleRunResult(True), + ) + + def hold_first(*_args, **_kwargs): + entered.set() + assert release.wait(5) + assert isolate.path.is_dir() + return cleanup() if kill_first else dependency.return_value + + try: + with monkeypatch.context() as patcher: + if kill_first: + patcher.setattr(isolate.directory, 'cleanup', hold_first) + else: + dependency.side_effect = hold_first + target = { + 'run': 'throng.extensions.temporary_directory.isolate.run', + 'read': 'throng.extensions.temporary_directory.isolate.read_directory', + 'set_state': 'tarfile.TarFile.extractall', + }[operation] + patcher.setattr(target, dependency) + arguments = {'run': ('command',), 'read': (), 'set_state': (state,)}[ + operation + ] + with ThreadPoolExecutor(max_workers=2) as pool: + work = getattr(isolate, operation) + first = ( + pool.submit(isolate.kill) + if kill_first + else pool.submit(work, *arguments) + ) + try: + assert entered.wait(5) + second = ( + pool.submit(work, *arguments) + if kill_first + else pool.submit(isolate.kill) + ) + assert second_attempted.wait(5) + assert not second.done() + finally: + release.set() + first.result(timeout=5) + with ( + pytest.raises(DirectoryDoesNotExistError) + if kill_first + else nullcontext() + ): + second.result(timeout=5) + assert dependency.call_count == (0 if kill_first else 1) + assert not isolate.path.exists() + assert isolate.used is True + assert not lock.locked() + finally: + isolate.directory.cleanup() + if lock.locked(): + lock.release() diff --git a/tests/units/extensions/temporary_directory/test_manager.py b/tests/units/extensions/temporary_directory/test_manager.py new file mode 100644 index 0000000..efe3413 --- /dev/null +++ b/tests/units/extensions/temporary_directory/test_manager.py @@ -0,0 +1,147 @@ +from contextlib import nullcontext +from shutil import rmtree +from unittest.mock import Mock + +import pytest + +from throng.extensions.temporary_directory.manager import TemporaryDirectoryManager + + +@pytest.mark.parametrize('exclude', [None, [], ['cache/', '*.tmp', '!keep.tmp']]) +def test_read_delegates_source_settings(tmp_path, monkeypatch, exclude): + """Read the configured source directory with the manager's exclusions.""" + expected_exclude = None if exclude is None else exclude.copy() + manager = TemporaryDirectoryManager(tmp_path, exclude) + read = Mock(return_value=b'\x00\xffstate') + monkeypatch.setattr( + 'throng.extensions.temporary_directory.manager.read_directory', + read, + ) + + assert manager.read() is read.return_value + read.assert_called_once_with(tmp_path, expected_exclude) + + +@pytest.mark.parametrize('state', [b'', b'state', b'\x00\xff']) +@pytest.mark.parametrize('exclude', [None, [], ['cache/', '*.tmp', '!keep.tmp']]) +def test_get_delegates_snapshot_without_reading_source( + tmp_path, + monkeypatch, + state, + exclude, +): + """Restore the supplied opaque snapshot without rereading the source directory.""" + expected_exclude = None if exclude is None else exclude.copy() + manager = TemporaryDirectoryManager(tmp_path, exclude) + constructor, read, read_source = Mock(), Mock(), Mock() + monkeypatch.setattr( + 'throng.extensions.temporary_directory.manager.TemporaryDirectoryIsolate', + constructor, + ) + monkeypatch.setattr(manager, 'read', read) + monkeypatch.setattr( + 'throng.extensions.temporary_directory.manager.read_directory', + read_source, + ) + + assert manager.get(state) is constructor.return_value + constructor.assert_called_once_with(state, expected_exclude) + read.assert_not_called() + read_source.assert_not_called() + + +@pytest.mark.parametrize('operation', ['read', 'get']) +def test_delegate_error_is_preserved(tmp_path, monkeypatch, operation): + """Keep the original cause when snapshot reading or isolate creation fails.""" + error = OSError('delegate failed') + dependency = Mock(side_effect=error) + name = 'read_directory' if operation == 'read' else 'TemporaryDirectoryIsolate' + monkeypatch.setattr( + f'throng.extensions.temporary_directory.manager.{name}', + dependency, + ) + manager = TemporaryDirectoryManager(tmp_path) + + with pytest.raises(OSError, match='delegate failed') as caught: + getattr(manager, operation)(*(() if operation == 'read' else (b'state',))) + + assert caught.value is error + assert dependency.call_count == 1 + + +@pytest.mark.parametrize('populated', [False, True]) +def test_same_snapshot_produces_independent_isolates(tmp_path, populated): + """Keep directories, modifications and cleanup independent for each isolate.""" + if populated: + (tmp_path / 'file').write_bytes(b'original') + manager = TemporaryDirectoryManager(tmp_path) + state = manager.read() + first = manager.get(state) + try: + second = manager.get(state) + try: + assert first is not second + assert first.path != second.path + (first.path / 'file').write_bytes(b'changed') + if populated: + assert (second.path / 'file').read_bytes() == b'original' + else: + assert not (second.path / 'file').exists() + first.kill() + assert not first.lock.locked() + assert second.path.is_dir() + assert tmp_path.is_dir() + finally: + second.directory.cleanup() + if second.lock.locked(): + second.lock.release() + finally: + first.directory.cleanup() + if first.lock.locked(): + first.lock.release() + + +@pytest.mark.parametrize('change', ['modify_file', 'delete_file', 'delete_directory']) +def test_snapshot_survives_source_changes(tmp_path, change): + """Restore saved contents even after the source file or whole directory disappears.""" + directory = tmp_path / 'source' + directory.mkdir() + source = directory / 'file' + source.write_bytes(b'saved') + manager = TemporaryDirectoryManager(directory) + state = manager.read() + if change == 'modify_file': + source.write_bytes(b'new') + elif change == 'delete_file': + source.unlink() + else: + rmtree(directory) + isolate = manager.get(state) + try: + assert (isolate.path / 'file').read_bytes() == b'saved' + finally: + isolate.directory.cleanup() + if isolate.lock.locked(): + isolate.lock.release() + + +@pytest.mark.parametrize('fail_body', [False, True]) +def test_scope_discards_copy_and_keeps_source(tmp_path, fail_body): + """Remove the temporary copy on every exit without changing source files.""" + (tmp_path / 'file').write_bytes(b'original') + manager = TemporaryDirectoryManager(tmp_path) + expectation = ( + pytest.raises(ValueError, match='body failed') if fail_body else nullcontext() + ) + + with expectation, manager.scope as isolate: + assert isolate.path != tmp_path + assert (isolate.path / 'file').read_bytes() == b'original' + (isolate.path / 'file').write_bytes(b'changed') + (isolate.path / 'new').write_bytes(b'new') + if fail_body: + raise ValueError('body failed') + + assert not isolate.path.exists() + assert (tmp_path / 'file').read_bytes() == b'original' + assert not (tmp_path / 'new').exists() diff --git a/tests/units/extensions/temporary_directory/test_read.py b/tests/units/extensions/temporary_directory/test_read.py new file mode 100644 index 0000000..5e60268 --- /dev/null +++ b/tests/units/extensions/temporary_directory/test_read.py @@ -0,0 +1,242 @@ +import tarfile +from io import BytesIO +from pathlib import Path +from unittest.mock import MagicMock, Mock, call + +import pytest + +from throng.extensions.temporary_directory.read import read_directory + + +@pytest.mark.parametrize( + 'content', + [b'', b'plain text', 'Привет\n世界'.encode(), bytes(range(256))], +) +@pytest.mark.parametrize('name', ['file', 'nested/file', 'one/two/file']) +def test_archive_preserves_file_data_and_relative_paths(tmp_path, name, content): + """Preserve bytes and relative paths so snapshots can move to another directory.""" + source = tmp_path / name + source.parent.mkdir(parents=True, exist_ok=True) + source.write_bytes(content) + + state = read_directory(tmp_path, None) + + assert isinstance(state, bytes) + with tarfile.open(fileobj=BytesIO(state)) as archive: + assert archive.getnames() == [name] + assert archive.getmember(name).isfile() + assert archive.extractfile(name).read() == content + assert source.read_bytes() == content + + +@pytest.mark.parametrize( + 'name', + [ + 'with spaces.txt', + 'каталог/файл', + 'emoji-🌍', + 'many.dots.txt', + '.hidden', + '.hidden/file', + 'x' * 120, + ], +) +def test_archive_preserves_supported_names(tmp_path, name): + """Keep valid unusual names without losing or renaming their files.""" + source = tmp_path / name + source.parent.mkdir(parents=True, exist_ok=True) + source.write_bytes(b'data') + + with tarfile.open(fileobj=BytesIO(read_directory(tmp_path, None))) as archive: + assert archive.getnames() == [name] + assert archive.extractfile(name).read() == b'data' + + +@pytest.mark.parametrize('tree', ['empty', 'directories', 'mixed']) +def test_empty_directories_are_not_archived(tmp_path, tree): + """Transfer files only, including a valid empty archive when there are none.""" + if tree != 'empty': + (tmp_path / 'empty' / 'nested').mkdir(parents=True) + if tree == 'mixed': + (tmp_path / 'file').write_bytes(b'data') + + with tarfile.open(fileobj=BytesIO(read_directory(tmp_path, None))) as archive: + assert archive.getnames() == (['file'] if tree == 'mixed' else []) + assert all(member.isfile() for member in archive.getmembers()) + + +@pytest.mark.parametrize('path_kind', ['absolute', 'relative', 'current']) +def test_path_form_does_not_change_snapshot(tmp_path, monkeypatch, path_kind): + """Produce the same relative file names for equivalent source paths.""" + source = tmp_path / 'source' + (source / 'nested').mkdir(parents=True) + (source / 'file').write_bytes(b'root') + (source / 'nested' / 'file').write_bytes(b'nested') + monkeypatch.chdir(source if path_kind == 'current' else tmp_path) + path = {'absolute': source, 'relative': Path('source'), 'current': Path()}[ + path_kind + ] + + with tarfile.open(fileobj=BytesIO(read_directory(path, None))) as archive: + assert { + member.name: archive.extractfile(member).read() + for member in archive.getmembers() + } == { + 'file': b'root', + 'nested/file': b'nested', + } + + +@pytest.mark.parametrize( + ('patterns', 'expected'), + [ + (None, ('keep.txt', 'drop.tmp', 'cache/file', 'nested/keep.txt')), + ((), ('keep.txt', 'drop.tmp', 'cache/file', 'nested/keep.txt')), + (('missing',), ('keep.txt', 'drop.tmp', 'cache/file', 'nested/keep.txt')), + (('drop.tmp',), ('keep.txt', 'cache/file', 'nested/keep.txt')), + (('*.tmp',), ('keep.txt', 'cache/file', 'nested/keep.txt')), + (('cache/',), ('keep.txt', 'drop.tmp', 'nested/keep.txt')), + (('*.tmp', 'cache/'), ('keep.txt', 'nested/keep.txt')), + (('*',), ()), + ( + ('*.txt', '!keep.txt'), + ('keep.txt', 'drop.tmp', 'cache/file', 'nested/keep.txt'), + ), + (('!keep.txt', '*.txt'), ('drop.tmp', 'cache/file')), + (('*.tmp', '*.tmp'), ('keep.txt', 'cache/file', 'nested/keep.txt')), + ], +) +def test_exclusions_select_files_without_changing_source(tmp_path, patterns, expected): + """Apply supported exclusion rules while leaving the original tree untouched.""" + names = ('keep.txt', 'drop.tmp', 'cache/file', 'nested/keep.txt') + for name in names: + source = tmp_path / name + source.parent.mkdir(parents=True, exist_ok=True) + source.write_bytes(name.encode()) + exclude = None if patterns is None else list(patterns) + + with tarfile.open(fileobj=BytesIO(read_directory(tmp_path, exclude))) as archive: + assert set(archive.getnames()) == set(expected) + assert all( + archive.extractfile(name).read() == name.encode() for name in expected + ) + assert { + path.relative_to(tmp_path).as_posix(): path.read_bytes() + for path in tmp_path.rglob('*') + if path.is_file() + } == {name: name.encode() for name in names} + + +@pytest.mark.parametrize('exclude', [None, [], ['*.tmp']]) +def test_crawler_receives_source_and_exclusions(tmp_path, monkeypatch, exclude): + """Delegate selection to the crawler and archive each returned relative path.""" + paths = [tmp_path / 'first', tmp_path / 'second'] + for path in paths: + path.write_bytes(path.name.encode()) + crawler = Mock(return_value=iter(paths)) + monkeypatch.setattr('throng.extensions.temporary_directory.read.Crawler', crawler) + + with tarfile.open(fileobj=BytesIO(read_directory(tmp_path, exclude))) as archive: + assert set(archive.getnames()) == {'first', 'second'} + crawler.assert_called_once_with(tmp_path, exclude=exclude) + + +@pytest.mark.parametrize('change', ['add', 'modify', 'delete']) +def test_snapshots_are_independent_of_later_changes(tmp_path, change): + """Keep previous snapshots immutable while later reads reflect source changes.""" + source = tmp_path / 'file' + source.write_bytes(b'old') + first = read_directory(tmp_path, None) + if change == 'add': + (tmp_path / 'new').write_bytes(b'new') + elif change == 'modify': + source.write_bytes(b'new') + else: + source.unlink() + second = read_directory(tmp_path, None) + + with tarfile.open(fileobj=BytesIO(first)) as archive: + assert archive.getnames() == ['file'] + assert archive.extractfile('file').read() == b'old' + with tarfile.open(fileobj=BytesIO(second)) as archive: + expected = { + 'add': {'file': b'old', 'new': b'new'}, + 'modify': {'file': b'new'}, + 'delete': {}, + }[change] + assert { + member.name: archive.extractfile(member).read() + for member in archive.getmembers() + } == expected + + +@pytest.mark.parametrize( + 'stage', + [ + 'crawler', + 'open', + 'iterate', + 'first_next', + 'second_next', + 'first_file', + 'second_file', + ], +) +@pytest.mark.parametrize('error_type', [FileNotFoundError, PermissionError]) +def test_read_errors_propagate_and_close_open_archives( + tmp_path, + monkeypatch, + stage, + error_type, +): + """Reject incomplete snapshots and close any archive already opened on failure.""" + error = error_type('snapshot failed') + crawler = Mock(return_value=iter([tmp_path / 'one', tmp_path / 'two'])) + archive_context = MagicMock() + archive = archive_context.__enter__.return_value + open_archive = Mock(return_value=archive_context) + if stage == 'crawler': + crawler.side_effect = error + elif stage == 'open': + open_archive.side_effect = error + elif stage == 'iterate': + iterator = MagicMock() + iterator.__iter__.side_effect = error + crawler.return_value = iterator + elif stage in ('first_next', 'second_next'): + iterator = MagicMock() + iterator.__iter__.side_effect = lambda: iterator + iterator.__next__.side_effect = ( + [tmp_path / 'one'] if stage == 'second_next' else [] + ) + [error] + crawler.return_value = iterator + else: + archive.add.side_effect = [None] * (stage == 'second_file') + [error] + monkeypatch.setattr('throng.extensions.temporary_directory.read.Crawler', crawler) + monkeypatch.setattr( + 'throng.extensions.temporary_directory.read.tarfile.open', + open_archive, + ) + + with pytest.raises(error_type) as caught: + read_directory(tmp_path, None) + + assert caught.value is error + if stage not in ('crawler', 'open'): + archive_context.__exit__.assert_called_once() + assert archive_context.__exit__.call_args.args[1] is error + else: + archive_context.__exit__.assert_not_called() + if stage == 'crawler': + open_archive.assert_not_called() + if stage in ('first_next', 'second_next'): + assert archive.add.call_args_list == ( + [call(tmp_path / 'one', arcname=Path('one'))] + if stage == 'second_next' + else [] + ) + if stage == 'second_file': + assert archive.add.call_args_list == [ + call(tmp_path / 'one', arcname=Path('one')), + call(tmp_path / 'two', arcname=Path('two')), + ] diff --git a/tests/units/extensions/test_plugins.py b/tests/units/extensions/test_plugins.py new file mode 100644 index 0000000..f6598fe --- /dev/null +++ b/tests/units/extensions/test_plugins.py @@ -0,0 +1,145 @@ +from importlib import import_module +from pathlib import Path +from unittest.mock import Mock, call + +import pytest + + +@pytest.mark.parametrize( + ('name', 'constructor_name'), + [('local', 'LocalManager'), ('temporary_directory', 'TemporaryDirectoryManager')], +) +@pytest.mark.parametrize( + 'form', + [ + 'default', + 'positional_path', + 'keyword_path', + 'positional_both', + 'mixed', + 'keyword_both', + 'exclude_only', + ], +) +@pytest.mark.parametrize('as_string', [False, True]) +def test_factory_forwards_settings_and_result( + monkeypatch, + name, + constructor_name, + form, + as_string, +): + """Forward supported call forms unchanged and return the constructor's manager.""" + plugins = import_module('throng.extensions.plugins') + constructor = Mock() + monkeypatch.setattr(plugins, constructor_name, constructor) + source = Path('space here') / 'каталог' + path = str(source) if as_string else source + exclude = ['cache/', '*.tmp', '!keep.tmp'] + forms = { + 'default': ((), {}, '.', None), + 'positional_path': ((path,), {}, path, None), + 'keyword_path': ((), {'path': path}, path, None), + 'positional_both': ((path, exclude), {}, path, exclude), + 'mixed': ((path,), {'exclude': exclude}, path, exclude), + 'keyword_both': ((), {'path': path, 'exclude': exclude}, path, exclude), + 'exclude_only': ((), {'exclude': exclude}, '.', exclude), + } + args, kwargs, expected_path, expected_exclude = forms[form] + expected_exclude = None if expected_exclude is None else expected_exclude.copy() + + result = getattr(plugins, name)(*args, **kwargs) + + assert result is constructor.return_value + constructor.assert_called_once_with(expected_path, expected_exclude) + + +@pytest.mark.parametrize( + ('name', 'constructor_name'), + [('local', 'LocalManager'), ('temporary_directory', 'TemporaryDirectoryManager')], +) +@pytest.mark.parametrize('exclude', [None, [], ['*.tmp']]) +def test_factory_does_not_cache_managers(monkeypatch, name, constructor_name, exclude): + """Preserve exclusion values while constructing a fresh manager on every call.""" + plugins = import_module('throng.extensions.plugins') + first, second = Mock(), Mock() + constructor = Mock(side_effect=[first, second]) + monkeypatch.setattr(plugins, constructor_name, constructor) + + assert getattr(plugins, name)('.', exclude) is first + assert getattr(plugins, name)('other') is second + assert constructor.call_args_list == [call('.', exclude), call('other', None)] + + +@pytest.mark.parametrize( + ('name', 'constructor_name'), + [('local', 'LocalManager'), ('temporary_directory', 'TemporaryDirectoryManager')], +) +@pytest.mark.parametrize('error_type', [RuntimeError, OSError]) +def test_factory_preserves_constructor_error( + monkeypatch, + name, + constructor_name, + error_type, +): + """Expose the original construction failure without retrying or wrapping it.""" + plugins = import_module('throng.extensions.plugins') + error = error_type('constructor failed') + constructor = Mock(side_effect=error) + monkeypatch.setattr(plugins, constructor_name, constructor) + + with pytest.raises(error_type) as caught: + getattr(plugins, name)() + + assert caught.value is error + constructor.assert_called_once_with('.', None) + + +@pytest.mark.parametrize( + ('name', 'constructor_name'), + [('local', 'LocalManager'), ('temporary_directory', 'TemporaryDirectoryManager')], +) +@pytest.mark.parametrize('existing', [False, True]) +def test_factory_creation_is_lazy( + tmp_path, + monkeypatch, + name, + constructor_name, + existing, +): + """Create builtin managers without reading files or allocating an isolate. + + Call each builtin factory directly so unrelated plugins may perform their own I/O. + Restore filesystem operations before checking the source or cleaning up the test. + """ + plugins = import_module('throng.extensions.plugins') + manager_type = getattr(plugins, constructor_name) + path = tmp_path if existing else tmp_path / 'missing' + read, get = Mock(), Mock() + monkeypatch.setattr(manager_type, 'read', read) + monkeypatch.setattr(manager_type, 'get', get) + operations = { + operation: Mock( + side_effect=AssertionError(f'Unexpected initialization I/O: {operation}'), + ) + for operation in ( + 'builtins.open', + 'io.open', + 'os.scandir', + 'os.listdir', + 'pathlib.Path.stat', + 'tempfile.mkdtemp', + ) + } + with monkeypatch.context() as patcher: + for operation, mock in operations.items(): + patcher.setattr(operation, mock) + manager = getattr(plugins, name)(path) + + assert isinstance(manager, manager_type) + assert manager.path == path + read.assert_not_called() + get.assert_not_called() + for mock in operations.values(): + mock.assert_not_called() + assert list(tmp_path.iterdir()) == [] diff --git a/tests/units/test_errors.py b/tests/units/test_errors.py new file mode 100644 index 0000000..04f1ec1 --- /dev/null +++ b/tests/units/test_errors.py @@ -0,0 +1,34 @@ +import pytest + +from throng.errors import ( + CannotCancelNonExistingIsolateError, + CannotInstallDependencyError, + NotSupportedCommandError, +) + + +@pytest.mark.parametrize( + 'error_type', + [ + CannotCancelNonExistingIsolateError, + CannotInstallDependencyError, + NotSupportedCommandError, + ], +) +def test_library_errors_specialize_runtime_error(error_type): + """Let callers handle throng failures specifically or as general runtime errors.""" + assert issubclass(error_type, RuntimeError) + assert error_type is not RuntimeError + + +@pytest.mark.parametrize( + ('first', 'second'), + [ + (CannotCancelNonExistingIsolateError, CannotInstallDependencyError), + (CannotCancelNonExistingIsolateError, NotSupportedCommandError), + (CannotInstallDependencyError, NotSupportedCommandError), + ], +) +def test_library_errors_have_distinct_types(first, second): + """Keep failure types distinct so callers can handle each cause separately.""" + assert first is not second diff --git a/tests/units/test_init.py b/tests/units/test_init.py new file mode 100644 index 0000000..02949bb --- /dev/null +++ b/tests/units/test_init.py @@ -0,0 +1,22 @@ +from importlib import import_module + +import pytest + + +@pytest.mark.parametrize( + ('name', 'module'), + [ + ('AbstractIsolate', 'throng.abstracts.abstract_isolate'), + ('AbstractManager', 'throng.abstracts.abstract_manager'), + ('RunResultProtocol', 'throng.abstracts.results'), + ('throng', 'throng.slots'), + ('local', 'throng.extensions.plugins'), + ('temporary_directory', 'throng.extensions.plugins'), + ], +) +def test_public_exports_match_implementation(name, module): + """Provide the canonical implementation through each public package export.""" + assert getattr(import_module('throng'), name) is getattr( + import_module(module), + name, + ) diff --git a/tests/units/test_slots.py b/tests/units/test_slots.py new file mode 100644 index 0000000..744f4b6 --- /dev/null +++ b/tests/units/test_slots.py @@ -0,0 +1,198 @@ +from pathlib import Path + +import pytest +from cantok import SimpleToken + +from throng import local, temporary_directory, throng +from throng.abstracts.results import SimpleRunResult +from throng.extensions.local.manager import LocalManager +from throng.extensions.temporary_directory.manager import TemporaryDirectoryManager + + +@pytest.mark.parametrize( + 'form', + [ + 'default', + 'positional_path', + 'keyword_path', + 'positional_both', + 'mixed', + 'keyword_both', + 'exclude_only', + ], +) +def test_slot_provides_builtin_managers(tmp_path, monkeypatch, form): + """Provide both documented managers with the requested source settings.""" + monkeypatch.chdir(tmp_path) + path = tmp_path / 'source' + exclude = ['*.tmp'] + forms = { + 'default': ((), {}, Path(), None), + 'positional_path': ((path,), {}, path, None), + 'keyword_path': ((), {'path': path}, path, None), + 'positional_both': ((path, exclude), {}, path, exclude), + 'mixed': ((path,), {'exclude': exclude}, path, exclude), + 'keyword_both': ((), {'path': path, 'exclude': exclude}, path, exclude), + 'exclude_only': ((), {'exclude': exclude}, Path(), exclude), + } + args, kwargs, expected_path, expected_exclude = forms[form] + + managers = throng(*args, **kwargs) + + assert isinstance(managers, dict) + assert isinstance(managers['local'], LocalManager) + assert isinstance(managers['temporary_directory'], TemporaryDirectoryManager) + for name in ('local', 'temporary_directory'): + assert managers[name].path == expected_path + assert managers[name].exclude == expected_exclude + + +def test_slot_uses_throng_entrypoint_group(): + """Discover third-party implementations in the package's own entrypoint group.""" + assert throng.entrypoint_group == 'throng' + + +@pytest.mark.parametrize( + ('name', 'factory'), + [('local', local), ('temporary_directory', temporary_directory)], +) +def test_builtin_registration_is_unique(name, factory): + """Register each builtin factory once under its documented unique name.""" + plugins = [plugin for plugin in throng if plugin.name == name] + + assert len(plugins) == 1 + assert plugins[0].unique is True + assert plugins[0].plugin_function is factory + + +@pytest.mark.parametrize( + 'path_kind', + ['relative_string', 'relative_path', 'absolute_string', 'absolute_path'], +) +@pytest.mark.parametrize('exclude', [None, [], ['cache/', '*.tmp', '!keep.tmp']]) +def test_slot_preserves_path_and_exclusion_values( + tmp_path, + monkeypatch, + path_kind, + exclude, +): + """Keep meaningful path and exclusion values consistent across builtin plugins.""" + monkeypatch.chdir(tmp_path) + relative = Path('space here') / 'каталог' + path = { + 'relative_string': str(relative), + 'relative_path': relative, + 'absolute_string': str(tmp_path / relative), + 'absolute_path': tmp_path / relative, + }[path_kind] + expected_exclude = None if exclude is None else exclude.copy() + + managers = throng(path, exclude) + + for name in ('local', 'temporary_directory'): + assert managers[name].path == Path(path) + assert managers[name].exclude == expected_exclude + + +def test_slot_calls_do_not_share_managers_or_settings(tmp_path): + """Create independent managers without leaking earlier settings into defaults.""" + first = throng(tmp_path, ['*.tmp']) + second = throng(tmp_path, ['*.tmp']) + for name in ('local', 'temporary_directory'): + first[name].path = tmp_path / 'changed' + first[name].exclude.append('extra') + defaults = throng() + + for name in ('local', 'temporary_directory'): + assert first[name] is not second[name] + assert second[name].path == tmp_path + assert second[name].exclude == ['*.tmp'] + assert defaults[name].path == Path() + assert defaults[name].exclude is None + + +@pytest.mark.parametrize('plugin', ['local', 'temporary_directory']) +@pytest.mark.parametrize('mode', ['open', 'scope', 'run', 'chain']) +@pytest.mark.parametrize('success', [False, True]) +def test_builtin_lifecycle_through_public_api( + tmp_path, + monkeypatch, + plugin, + mode, + success, +): + """Use the same lifecycle API while preserving each plugin's directory behavior.""" + (tmp_path / 'source').write_bytes(b'data') + manager = throng(tmp_path)[plugin] + token = SimpleToken() + expected = SimpleRunResult(success, 0 if success else 1, 'output', 'diagnostic') + directories = [] + + def execute(command, **kwargs): + assert command == 'command' + assert kwargs['token'] is token + assert kwargs['catch_output'] is True + assert kwargs['catch_exceptions'] is True + directory = kwargs['directory'] + assert (directory / 'source').read_bytes() == b'data' + directories.append(directory) + return expected + + monkeypatch.setattr(f'throng.extensions.{plugin}.isolate.run', execute) + if mode == 'open': + isolate = manager.get(manager.read()) + try: + result = isolate.run('command', token=token) + finally: + isolate.kill() + elif mode == 'scope': + with manager.scope as isolate: + result = isolate.run('command', token=token) + else: + result = getattr(manager, mode)('command', token=token) + + if mode == 'chain': + assert len(result) == 1 + assert result[0] is expected + else: + assert result is expected + assert len(directories) == 1 + assert (directories[0] == tmp_path) is (plugin == 'local') + assert directories[0].exists() is (plugin == 'local') + assert (tmp_path / 'source').read_bytes() == b'data' + + +@pytest.mark.parametrize('plugin', ['local', 'temporary_directory']) +@pytest.mark.parametrize('closed', [False, True]) +def test_chained_commands_share_the_same_environment( + tmp_path, + monkeypatch, + plugin, + closed, +): + """Keep command changes visible within a chain and isolated according to the plugin.""" + manager = throng(tmp_path)[plugin] + directories = [] + + def execute(command, *, directory, **_kwargs): + directories.append(directory) + if command == 'write': + (directory / 'created').write_bytes(b'shared') + return SimpleRunResult(True, 0, '', '') + return SimpleRunResult(True, 0, (directory / 'created').read_text(), '') + + monkeypatch.setattr(f'throng.extensions.{plugin}.isolate.run', execute) + if closed: + results = manager.chain('write', 'read') + else: + with manager.scope as isolate: + results = isolate.chain('write', 'read') + + assert results == [ + SimpleRunResult(True, 0, '', ''), + SimpleRunResult(True, 0, 'shared', ''), + ] + assert len(directories) == 2 + assert directories[0] == directories[1] + assert (tmp_path / 'created').exists() is (plugin == 'local') + assert directories[0].exists() is (plugin == 'local') diff --git a/throng/abstracts/abstract_isolate.py b/throng/abstracts/abstract_isolate.py index 1ec2c2a..bc1fdff 100644 --- a/throng/abstracts/abstract_isolate.py +++ b/throng/abstracts/abstract_isolate.py @@ -12,19 +12,19 @@ def __del__(self) -> None: @abstractmethod def run(self, command: str, token: AbstractToken = DefaultToken()) -> RunResultProtocol: # noqa: B008 - ... + ... # pragma: no cover @abstractmethod def read(self) -> bytes: - ... + ... # pragma: no cover @abstractmethod def kill(self) -> None: - ... + ... # pragma: no cover @abstractmethod def install(self, *packages: str) -> None: - ... + ... # pragma: no cover def chain(self, *commands: str, token: AbstractToken = DefaultToken()) -> List[RunResultProtocol]: # noqa: B008 results = [] diff --git a/throng/abstracts/abstract_manager.py b/throng/abstracts/abstract_manager.py index cd86edf..5f4494f 100644 --- a/throng/abstracts/abstract_manager.py +++ b/throng/abstracts/abstract_manager.py @@ -54,8 +54,8 @@ def chain(self, *commands: str, token: AbstractToken = DefaultToken()) -> List[R @abstractmethod def get(self, state: bytes) -> AbstractIsolate: - ... + ... # pragma: no cover @abstractmethod def read(self) -> bytes: - ... + ... # pragma: no cover diff --git a/throng/errors.py b/throng/errors.py index a466f6e..1795130 100644 --- a/throng/errors.py +++ b/throng/errors.py @@ -1,8 +1,8 @@ class CannotCancelNonExistingIsolateError(RuntimeError): - ... + ... # pragma: no cover class NotSupportedCommandError(RuntimeError): - ... + ... # pragma: no cover class CannotInstallDependencyError(RuntimeError): - ... + ... # pragma: no cover diff --git a/throng/extensions/local/isolate.py b/throng/extensions/local/isolate.py index e5dfd09..80589c1 100644 --- a/throng/extensions/local/isolate.py +++ b/throng/extensions/local/isolate.py @@ -1,3 +1,5 @@ +from pathlib import Path + from cantok import AbstractToken, DefaultToken from locklib import ContextLockProtocol from suby import SubprocessResult, run @@ -7,12 +9,13 @@ class LocalIsolate(AbstractIsolate): - def __init__(self, lock: ContextLockProtocol) -> None: + def __init__(self, lock: ContextLockProtocol, path: Path = Path()) -> None: self.lock = lock + self.path = path def run(self, command: str, token: AbstractToken = DefaultToken()) -> SubprocessResult: # noqa: B008 with self.lock: - return run(command, token=token, catch_output=True, catch_exceptions=True) + return run(command, token=token, catch_output=True, catch_exceptions=True, directory=self.path) def read(self) -> bytes: return b'' diff --git a/throng/extensions/local/manager.py b/throng/extensions/local/manager.py index 9664b05..ac6eede 100644 --- a/throng/extensions/local/manager.py +++ b/throng/extensions/local/manager.py @@ -7,12 +7,12 @@ class LocalManager(AbstractManager): - def __init__(self, path: Optional[Union[str, Path]], exclude: Optional[List[str]]) -> None: + def __init__(self, path: Union[str, Path], exclude: Optional[List[str]]) -> None: self.lock = Lock() super().__init__(path, exclude) def get(self, state: bytes) -> LocalIsolate: # noqa: ARG002 - return LocalIsolate(self.lock) + return LocalIsolate(self.lock, self.path) def read(self) -> bytes: return b'' diff --git a/throng/extensions/temporary_directory/errors.py b/throng/extensions/temporary_directory/errors.py index 78249ae..5cee16f 100644 --- a/throng/extensions/temporary_directory/errors.py +++ b/throng/extensions/temporary_directory/errors.py @@ -1,2 +1,2 @@ class DirectoryDoesNotExistError(Exception): - ... + ... # pragma: no cover diff --git a/throng/extensions/temporary_directory/isolate.py b/throng/extensions/temporary_directory/isolate.py index 79789f0..9a2b558 100644 --- a/throng/extensions/temporary_directory/isolate.py +++ b/throng/extensions/temporary_directory/isolate.py @@ -17,6 +17,7 @@ class TemporaryDirectoryIsolate(AbstractIsolate): def __init__(self, state: bytes, exclude: Optional[List[str]]) -> None: self.lock = Lock() + self.exclude = exclude self.used = False self.directory = TemporaryDirectory() self.path = Path(self.directory.name) diff --git a/throng/extensions/temporary_directory/manager.py b/throng/extensions/temporary_directory/manager.py index b470e43..8044814 100644 --- a/throng/extensions/temporary_directory/manager.py +++ b/throng/extensions/temporary_directory/manager.py @@ -4,8 +4,8 @@ class TemporaryDirectoryManager(AbstractManager): - def get(self, state: bytes) -> TemporaryDirectoryIsolate: # noqa: ARG002 - return TemporaryDirectoryIsolate(self.read(), self.exclude) + def get(self, state: bytes) -> TemporaryDirectoryIsolate: + return TemporaryDirectoryIsolate(state, self.exclude) def read(self) -> bytes: return read_directory(self.path, self.exclude) diff --git a/throng/extensions/temporary_directory/read.py b/throng/extensions/temporary_directory/read.py index c0d7778..fa4f3eb 100644 --- a/throng/extensions/temporary_directory/read.py +++ b/throng/extensions/temporary_directory/read.py @@ -12,6 +12,6 @@ def read_directory(path: Path, exclude: Optional[List[str]]) -> bytes: with tarfile.open(fileobj=buffer, mode='w') as tar: for file in crawler: - tar.add(file) + tar.add(file, arcname=file.relative_to(path)) return buffer.getvalue() diff --git a/throng/slots.py b/throng/slots.py index a01c5d3..ceab5d3 100644 --- a/throng/slots.py +++ b/throng/slots.py @@ -8,4 +8,4 @@ @slot(entrypoint_group='throng') def throng(path: Union[str, Path] = '.', exclude: Optional[List[str]] = None) -> Dict[str, AbstractManager]: # type: ignore[empty-body] - ... + ... # pragma: no cover