From 8b9899e9f52994e83e6ecb91fca0054a51d3d164 Mon Sep 17 00:00:00 2001 From: mrburke Date: Wed, 12 Jan 2022 11:37:47 -0800 Subject: [PATCH 01/21] Release (#418) * DOC-1074 Document password vault support for VSDK plugins (#347) * Correct Typo in Example, it should be UNIX not WINDOWS (#360) * Add documentation for gRPC message and memory limits #390 (#391) * Fix version number in doc header (#358) (#397) * Update Delphix Engine version required to 6.0.7.0. (#386) (#398) * Update SDK Docs deployment to use a deploy_key (#392) (#399) * CP-6005 Ensure vSDK test coverage (#403) * CP-6006 Futurize vSDK - Stage 1 (#404) * Add support for virtualCleanup decorator (#400) (#405) Co-authored-by: Kurt Hutchison * Fixes #401 Networking capabilities should be documented (#402) Fixes #401 Networking capabilities should be documented * CP-6009 Add to_str and to_byte helpers (#407) * CP-6621 bump sdk version to 4.0.0 (#409) * CP-6694 Update docs for SDK 4.0.0 (#410) * CP-7022 vSDK 4.0.0 release tasks * Update VirtualDelete flow diagram to include the virtual.cleanup call. (#414) Co-authored-by: Kurt Hutchison * Update documentation with new gRPC message limits (#415) Co-authored-by: Raul Santelices Co-authored-by: Ranzo J Taylor, III <36489926+Ranzo3@users.noreply.github.com> Co-authored-by: Jeff Ngo <35587243+jeffngo@users.noreply.github.com> Co-authored-by: Ankur Co-authored-by: kurthdelphix <89274759+kurthdelphix@users.noreply.github.com> Co-authored-by: Kurt Hutchison Co-authored-by: Tom Walsh <61519496+mothslaw@users.noreply.github.com> --- .bumpversion.cfg | 3 +- .github/workflows/pre-commit.yml | 86 ++- .github/workflows/publish-docs.yml | 2 +- .gitignore | 7 +- README-dev.md | 6 +- README.md | 5 +- common/.python-version | 1 - common/setup.cfg | 19 +- common/setup.py | 3 +- .../python/dlpx/virtualization/common/VERSION | 2 +- .../dlpx/virtualization/common/__init__.py | 4 +- .../virtualization/common/_common_classes.py | 67 +-- .../dlpx/virtualization/common/exceptions.py | 31 +- .../python/dlpx/virtualization/common/util.py | 125 +++++ .../dlpx/virtualization/common/__init__.py | 4 +- .../common/test_common_classes.py | 315 +++++++++-- docs/CONTRIBUTING.md | 10 +- docs/docs/Best_Practices/.pages | 4 +- docs/docs/Best_Practices/Code_Sharing.md | 27 +- .../Managing_Scripts_For_Remote_Execution.md | 16 +- docs/docs/Best_Practices/Message_Limits.md | 44 ++ .../Best_Practices/Runtime_Environment.md | 63 +++ docs/docs/Best_Practices/Strings.md | 90 ++++ docs/docs/Best_Practices/Unicode_Data.md | 29 -- .../Best_Practices/User_Visible_Errors.md | 11 +- .../Initial_Setup.md | 4 +- docs/docs/Getting_Started.md | 13 +- docs/docs/References/Decorators.md | 3 +- docs/docs/References/Logging.md | 35 +- docs/docs/References/Platform_Libraries.md | 28 +- docs/docs/References/Plugin_Config.md | 7 +- docs/docs/References/Plugin_Operations.md | 46 +- docs/docs/References/Schemas.md | 3 +- docs/docs/References/Version_Compatibility.md | 26 + .../References/html/VirtualSourceDelete.html | 10 +- .../References/images/VirtualSourceDelete.png | Bin 23291 -> 17644 bytes docs/docs/Release_Notes/4.0.0/.pages | 3 + docs/docs/Release_Notes/4.0.0/4.0.0.md | 11 + .../4.0.0/4.0.0_Breaking_Changes.md | 4 + docs/docs/images/.DS_Store | Bin 6148 -> 0 bytes docs/material/.DS_Store | Bin 8196 -> 0 bytes docs/mkdocs.yml | 2 +- docs/readme.md | 17 +- dvp/.python-version | 2 +- dvp/setup.cfg | 24 +- dvp/setup.py | 1 + .../main/python/dlpx/virtualization/VERSION | 2 +- libs/.python-version | 1 - libs/setup.cfg | 21 +- libs/setup.py | 3 +- .../python/dlpx/virtualization/libs/VERSION | 2 +- .../dlpx/virtualization/libs/__init__.py | 6 +- .../dlpx/virtualization/libs/_logging.py | 4 +- .../dlpx/virtualization/libs/exceptions.py | 24 +- .../python/dlpx/virtualization/libs/libs.py | 160 +++--- .../dlpx/virtualization/_engine/libs.py | 4 +- .../python/dlpx/virtualization/conftest.py | 5 +- .../python/dlpx/virtualization/test_libs.py | 489 +++++++++++------- .../dlpx/virtualization/test_logging.py | 4 +- platform/.python-version | 1 - platform/setup.cfg | 21 +- platform/setup.py | 3 +- .../dlpx/virtualization/platform/VERSION | 2 +- .../dlpx/virtualization/platform/__init__.py | 24 +- .../platform/_plugin_classes.py | 10 +- .../dlpx/virtualization/platform/_virtual.py | 69 ++- .../platform/import_validations.py | 9 +- .../platform/migration_helper.py | 8 +- .../dlpx/virtualization/platform/operation.py | 1 + .../dlpx/virtualization/platform/util.py | 5 +- .../fake_generated_definitions.py | 13 +- .../virtualization/test_migration_helper.py | 3 +- .../python/dlpx/virtualization/test_plugin.py | 226 +++++--- .../virtualization/test_plugin_classes.py | 109 ++-- .../dlpx/virtualization/test_upgrade.py | 14 +- tools/.python-version | 1 - tools/setup.cfg | 22 +- tools/setup.py | 20 +- .../dlpx/virtualization/_internal/VERSION | 2 +- .../dlpx/virtualization/_internal/cli.py | 11 +- .../dlpx/virtualization/_internal/codegen.py | 19 +- .../codegen/templates/base_model_.mustache | 61 ++- .../_internal/codegen/templates/util.mustache | 28 +- .../_internal/commands/build.py | 34 +- .../_internal/commands/download_logs.py | 14 +- .../_internal/commands/initialize.py | 51 +- .../_internal/commands/upload.py | 15 +- .../_internal/delphix_client.py | 27 +- .../virtualization/_internal/exceptions.py | 13 +- .../virtualization/_internal/package_util.py | 4 +- .../_internal/plugin_dependency_util.py | 27 +- .../_internal/plugin_importer.py | 56 +- .../_internal/plugin_validator.py | 17 +- .../_internal/schema_validator.py | 21 +- .../virtualization/_internal/settings.cfg | 2 +- .../plugin_config_schema.json | 2 +- .../validation_schemas/plugin_importer.yaml | 8 + .../validation_schemas/plugin_schema.json | 4 - .../_internal/commands/test_build.py | 71 +-- .../_internal/commands/test_codegen.py | 8 +- .../_internal/commands/test_delphix_client.py | 47 +- .../_internal/commands/test_initialize.py | 36 +- .../_internal/commands/test_templates.py | 320 +++++++----- .../dlpx/virtualization/_internal/conftest.py | 22 +- .../fake_plugin/direct/multiple_warnings.py | 5 + .../fake_plugin/direct/successful.py | 5 + .../fake_plugin/direct/upgrade_warnings.py | 5 + .../dlpx/virtualization/_internal/test_cli.py | 44 +- .../_internal/test_package_util.py | 12 +- .../_internal/test_plugin_importer.py | 27 +- .../_internal/test_plugin_validator.py | 6 +- .../_internal/test_schema_validator.py | 6 +- 112 files changed, 2451 insertions(+), 1078 deletions(-) delete mode 100644 common/.python-version create mode 100644 common/src/main/python/dlpx/virtualization/common/util.py create mode 100644 docs/docs/Best_Practices/Message_Limits.md create mode 100644 docs/docs/Best_Practices/Runtime_Environment.md create mode 100644 docs/docs/Best_Practices/Strings.md delete mode 100644 docs/docs/Best_Practices/Unicode_Data.md create mode 100644 docs/docs/References/Version_Compatibility.md create mode 100644 docs/docs/Release_Notes/4.0.0/.pages create mode 100644 docs/docs/Release_Notes/4.0.0/4.0.0.md create mode 100644 docs/docs/Release_Notes/4.0.0/4.0.0_Breaking_Changes.md delete mode 100644 docs/docs/images/.DS_Store delete mode 100644 docs/material/.DS_Store delete mode 100644 libs/.python-version delete mode 100644 platform/.python-version delete mode 100644 tools/.python-version diff --git a/.bumpversion.cfg b/.bumpversion.cfg index 8213c309..afb68a8c 100644 --- a/.bumpversion.cfg +++ b/.bumpversion.cfg @@ -1,5 +1,5 @@ [bumpversion] -current_version = 3.1.0 +current_version = 4.0.0 commit = False tag = False parse = (?P\d+)\.(?P\d+)\.(?P\d+)(\.(?P[a-z]+)(?P\d+))? @@ -25,4 +25,3 @@ values = [bumpversion:file:./libs/src/main/python/dlpx/virtualization/libs/VERSION] [bumpversion:file:./tools/src/main/python/dlpx/virtualization/_internal/VERSION] - diff --git a/.github/workflows/pre-commit.yml b/.github/workflows/pre-commit.yml index ab5d1efe..98614524 100644 --- a/.github/workflows/pre-commit.yml +++ b/.github/workflows/pre-commit.yml @@ -3,13 +3,48 @@ name: Pre-commit actions for Delphix Virtualization SDK on: [pull_request] jobs: - pytest: - name: Test ${{ matrix.package }} on ${{ matrix.os }} using pytest + pytest27: + name: Test ${{ matrix.package }} on ${{ matrix.os }} using pytest (Python 2.7) runs-on: ${{ matrix.os }} strategy: max-parallel: 4 matrix: python-version: [2.7] + os: [ubuntu-latest, macos-10.15, windows-latest] + package: [common, libs, platform] + + steps: + - name: Checkout ${{ matrix.package }} project + uses: actions/checkout@v1 + + - name: Set up Python ${{ matrix.python-version }} + uses: actions/setup-python@v1 + with: + python-version: ${{ matrix.python-version }} + + - name: Install ${{ matrix.package }} dependencies + working-directory: ${{ matrix.package }} + run: | + python -m pip install --upgrade pip + pip install -r requirements.txt --find-links https://test.pypi.org/simple/dvp-api/ + + - name: Install ${{ matrix.package }} project + working-directory: ${{ matrix.package }} + run: | + pip install . --find-links https://test.pypi.org/simple/dvp-api/ + + - name: Test ${{ matrix.package }} project with pytest + working-directory: ${{ matrix.package }} + run: | + python -m pytest src/test/python + + pytest38: + name: Test ${{ matrix.package }} on ${{ matrix.os }} using pytest (Python 3.8) + runs-on: ${{ matrix.os }} + strategy: + max-parallel: 4 + matrix: + python-version: [3.8] os: [ubuntu-latest, macos-latest, windows-latest] package: [common, libs, platform, tools] @@ -38,33 +73,64 @@ jobs: run: | python -m pytest src/test/python - lint: - name: Lint ${{ matrix.package }} + lintpython27: + name: Lint ${{ matrix.package }} - Python27 runs-on: ubuntu-latest strategy: max-parallel: 4 matrix: - package: [tools] - + package: [common, libs, platform] + steps: - name: Checkout ${{ matrix.package }} uses: actions/checkout@v1 - + - name: Set up Python 2.7 uses: actions/setup-python@v1 with: python-version: 2.7 - + - name: Install flake8 run: | python -m pip install --upgrade pip pip install flake8 - + - name: Run flake8 on src directory working-directory: ${{ matrix.package }} run: python -m flake8 src/main/python --max-line-length 88 - + + - name: Run flake8 on test directory + working-directory: ${{ matrix.package }} + run: python -m flake8 src/test/python --max-line-length 88 + + lintpython38: + name: Lint ${{ matrix.package }} - Python38 + + runs-on: ubuntu-latest + strategy: + max-parallel: 4 + matrix: + package: [common, libs, platform, tools] + + steps: + - name: Checkout ${{ matrix.package }} + uses: actions/checkout@v1 + + - name: Set up Python 3.8 + uses: actions/setup-python@v1 + with: + python-version: 3.8 + + - name: Install flake8 + run: | + python -m pip install --upgrade pip + pip install flake8 + + - name: Run flake8 on src directory + working-directory: ${{ matrix.package }} + run: python -m flake8 src/main/python --max-line-length 88 + - name: Run flake8 on test directory working-directory: ${{ matrix.package }} run: python -m flake8 src/test/python --max-line-length 88 diff --git a/.github/workflows/publish-docs.yml b/.github/workflows/publish-docs.yml index a7165fef..12938703 100644 --- a/.github/workflows/publish-docs.yml +++ b/.github/workflows/publish-docs.yml @@ -75,7 +75,7 @@ jobs: if: ${{ github.repository == matrix.repository }} uses: peaceiris/actions-gh-pages@v3 with: - github_token: ${{ secrets.GITHUB_TOKEN }} + deploy_key: ${{ secrets.SDK_DEPLOY_KEY }} publish_dir: docs/site commit_message: Deploy to gh-pages 🚀 user_name: "github-actions[bot]" diff --git a/.gitignore b/.gitignore index c73fd091..b1fe796c 100644 --- a/.gitignore +++ b/.gitignore @@ -1,5 +1,5 @@ # -# Copyright (c) 2019 by Delphix. All rights reserved. +# Copyright (c) 2019, 2021 by Delphix. All rights reserved. # # IntelliJ config files @@ -35,3 +35,8 @@ venv/ # Python cache __pycache__ +# .python-version files +*.python-version + +# OSX DS Store files +*.DS_Store diff --git a/README-dev.md b/README-dev.md index 34540f17..65f3ce74 100644 --- a/README-dev.md +++ b/README-dev.md @@ -99,6 +99,8 @@ If you want to bump the major/minor/patch version, run `bumpversion [major|minor If you want to get rid of the dev label (bump from `1.1.0.dev7` to `1.1.0`), run `bumpversion release`. +Note: After bumpversion the tools unit test will need to be manually updated to test for the new version. + ## Testing Currently, there are three types of SDK testing: unit, manual, and functional (blackbox). @@ -122,7 +124,9 @@ all the standard workflows. The same workflows will be exercised by functional ( ### Functional (blackbox) testing To run blackbox tests, follow these steps: 1. Push your code to a branch in the forked repository on Github. Let's say the branch is called `my-feature` in repository called `/virtualization-sdk`. -2. Navigate to the app-gate directory and start tests using `git blackbox`. For the guide on which test suite to use, +2. If you bumped the version (one of major, minor, or micro, not the dev version part), then QA will have to createa a new branch (qa-appdata-toolkits branch sdk-3-2-0 for example with version 3.2.0) and update their map before you can run the blackbox tests: +* automation/regression/BlackBox/blackbox/appdata/virtualization_sdk/dvp_settings.py +3. Navigate to the app-gate directory and start tests using `git blackbox`. For the guide on which test suite to use, see the next sections. At a minimum, each pull request should pass `appdata_python_samples` and `appdata_basic` tests with a direct or staged plugin. diff --git a/README.md b/README.md index 9299bf8b..9e1229f0 100644 --- a/README.md +++ b/README.md @@ -10,9 +10,10 @@ The latest user documentation can be found [here](https://developer.delphix.com) ### Prerequisites - macOS 10.14+, Ubuntu 16.04+, or Windows 10 -- Python 2.7 (Python 3 is not supported) +- Python 2.7 (vSDK 3.1.0 and earlier) +- Python 3.8 (vSDK 4.0.0 and later) - Java 7+ -- Delphix Engine 5.3.5.0 or above +- A Delphix Engine of an [appropriate version](/References/Version_Compatibility.md) ### Installing diff --git a/common/.python-version b/common/.python-version deleted file mode 100644 index 43c4dbe6..00000000 --- a/common/.python-version +++ /dev/null @@ -1 +0,0 @@ -2.7.17 diff --git a/common/setup.cfg b/common/setup.cfg index 8199f265..a482f5f1 100644 --- a/common/setup.cfg +++ b/common/setup.cfg @@ -1,20 +1,21 @@ # -# Copyright (c) 2019 by Delphix. All rights reserved. +# Copyright (c) 2019, 2021 by Delphix. All rights reserved. # [metadata] -Metadata-Version: 1.2 -Author: Delphix -Author-email: virtualization-plugins@delphix.com -Home-page: https://developer.delphix.com -Long-description: file: README.md -Long-description-content-type: text/markdown -Classifiers: +metadata_version: 1.2 +author: Delphix +author_email: virtualization-plugins@delphix.com +home_page: https://developer.delphix.com +long_description: file: README.md +long_description_content_type: text/markdown +classifiers: Development Status :: 5 - Production/Stable Programming Language :: Python Programming Language :: Python :: 2.7 + Programming Language :: Python :: 3.8 License :: OSI Approved :: Apache Software License Operating System :: OS Independent [options] -Requires-Python: 2.7 +requires_python: >=2.7, <=3.8, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*, !=3.6.*, !=3.7.* diff --git a/common/setup.py b/common/setup.py index 56d61c75..68fc5cdd 100644 --- a/common/setup.py +++ b/common/setup.py @@ -4,7 +4,7 @@ PYTHON_SRC = 'src/main/python' install_requires = [ - "dvp-api == 1.5.0", + "dvp-api == 1.6.1", ] with open(os.path.join(PYTHON_SRC, 'dlpx/virtualization/common/VERSION')) as version_file: @@ -15,4 +15,5 @@ install_requires=install_requires, package_dir={'': PYTHON_SRC}, packages=setuptools.find_packages(PYTHON_SRC), + python_requires='>=2.7, <3.9, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*, !=3.6.*, !=3.7.*', ) diff --git a/common/src/main/python/dlpx/virtualization/common/VERSION b/common/src/main/python/dlpx/virtualization/common/VERSION index fd2a0186..fcdb2e10 100644 --- a/common/src/main/python/dlpx/virtualization/common/VERSION +++ b/common/src/main/python/dlpx/virtualization/common/VERSION @@ -1 +1 @@ -3.1.0 +4.0.0 diff --git a/common/src/main/python/dlpx/virtualization/common/__init__.py b/common/src/main/python/dlpx/virtualization/common/__init__.py index b4692226..c4c8dcd6 100644 --- a/common/src/main/python/dlpx/virtualization/common/__init__.py +++ b/common/src/main/python/dlpx/virtualization/common/__init__.py @@ -1,7 +1,7 @@ # -# Copyright (c) 2019 by Delphix. All rights reserved. +# Copyright (c) 2019, 2021 by Delphix. All rights reserved. # __path__ = __import__('pkgutil').extend_path(__path__, __name__) -from dlpx.virtualization.common._common_classes import * +from dlpx.virtualization.common._common_classes import * # noqa diff --git a/common/src/main/python/dlpx/virtualization/common/_common_classes.py b/common/src/main/python/dlpx/virtualization/common/_common_classes.py index 2ad05804..57a698f4 100644 --- a/common/src/main/python/dlpx/virtualization/common/_common_classes.py +++ b/common/src/main/python/dlpx/virtualization/common/_common_classes.py @@ -1,11 +1,11 @@ # -# Copyright (c) 2019 by Delphix. All rights reserved. +# Copyright (c) 2019, 2021 by Delphix. All rights reserved. # from abc import ABCMeta +import six from dlpx.virtualization.api import common_pb2, libs_pb2 from dlpx.virtualization.common.exceptions import IncorrectTypeError -from enum import IntEnum """Classes used for Plugin Operations @@ -101,19 +101,19 @@ class RemoteEnvironment(object): """ def __init__(self, name, reference, host): - if not isinstance(name, basestring): + if not isinstance(name, six.string_types): raise IncorrectTypeError( RemoteEnvironment, 'name', type(name), - basestring) + six.string_types[0]) self.__name = name - if not isinstance(reference, basestring): + if not isinstance(reference, six.string_types): raise IncorrectTypeError( RemoteEnvironment, 'reference', type(reference), - basestring) + six.string_types[0]) self.__reference = reference if isinstance(host, RemoteHost): @@ -134,7 +134,9 @@ def reference(self): return self.__reference def to_proto(self): - """Converts plugin class RemoteEnvironment to protobuf class common_pb2.RemoteEnvironment + """ + Converts plugin class RemoteEnvironment to protobuf + class common_pb2.RemoteEnvironment """ remote_environment = common_pb2.RemoteEnvironment() remote_environment.name = self.name @@ -144,7 +146,9 @@ def to_proto(self): @staticmethod def from_proto(environment): - """Converts protobuf class common_pb2.RemoteEnvironment to plugin class RemoteEnvironment + """ + Converts protobuf class common_pb2.RemoteEnvironment to plugin + class RemoteEnvironment """ if not isinstance(environment, common_pb2.RemoteEnvironment): raise IncorrectTypeError( @@ -172,33 +176,33 @@ class RemoteHost(object): """ def __init__(self, name, reference, binary_path, scratch_path): - if not isinstance(name, basestring): + if not isinstance(name, six.string_types): raise IncorrectTypeError( RemoteHost, 'name', type(name), - basestring) + six.string_types[0]) self.__name = name - if not isinstance(reference, basestring): + if not isinstance(reference, six.string_types): raise IncorrectTypeError( RemoteHost, 'reference', type(reference), - basestring) + six.string_types[0]) self.__reference = reference - if not isinstance(binary_path, basestring): + if not isinstance(binary_path, six.string_types): raise IncorrectTypeError( RemoteHost, 'binary_path', type(binary_path), - basestring) + six.string_types[0]) self.__binary_path = binary_path - if not isinstance(scratch_path, basestring): + if not isinstance(scratch_path, six.string_types): raise IncorrectTypeError( RemoteHost, 'scratch_path', type(scratch_path), - basestring) + six.string_types[0]) self.__scratch_path = scratch_path @property @@ -255,19 +259,19 @@ class RemoteUser(object): reference: Reference of the RemoteUser. """ def __init__(self, name, reference): - if not isinstance(name, basestring): + if not isinstance(name, six.string_types): raise IncorrectTypeError( RemoteUser, 'name', type(name), - basestring) + six.string_types[0]) self.__name = name - if not isinstance(reference, basestring): + if not isinstance(reference, six.string_types): raise IncorrectTypeError( RemoteUser, 'reference', type(reference), - basestring) + six.string_types[0]) self.__reference = reference @property @@ -311,12 +315,12 @@ class Credentials(object): username: User name. """ def __init__(self, username): - if not isinstance(username, basestring): + if not isinstance(username, six.string_types): raise IncorrectTypeError( Credentials, 'username', type(username), - basestring) + six.string_types[0]) self.__username = username __metaclass__ = ABCMeta @@ -339,12 +343,12 @@ class PasswordCredentials(Credentials): """ def __init__(self, username, password): super(PasswordCredentials, self).__init__(username) - if not isinstance(password, basestring): + if not isinstance(password, six.string_types): raise IncorrectTypeError( PasswordCredentials, 'password', type(password), - basestring) + six.string_types[0]) self.__password = password @property @@ -353,7 +357,9 @@ def password(self): @staticmethod def from_proto(credentials_result): - """Converts protobuf class libs_pb2.CredentialsResult to plugin class PasswordCredentials + """ + Converts protobuf class libs_pb2.CredentialsResult to plugin + class PasswordCredentials """ if not isinstance(credentials_result, libs_pb2.CredentialsResult): raise IncorrectTypeError( @@ -375,23 +381,24 @@ class KeyPairCredentials(Credentials): Args: username (str): User name. private_key (str): Private key. - public_key (str): Public key corresponding to private key. Empty string if not present. + public_key (str): Public key corresponding to private key. Empty string if + not present. """ def __init__(self, username, private_key, public_key): super(KeyPairCredentials, self).__init__(username) - if not isinstance(private_key, basestring): + if not isinstance(private_key, six.string_types): raise IncorrectTypeError( KeyPairCredentials, 'private_key', type(private_key), - basestring) + six.string_types[0]) self.__private_key = private_key - if not isinstance(public_key, basestring): + if not isinstance(public_key, six.string_types): raise IncorrectTypeError( KeyPairCredentials, 'public_key', type(public_key), - basestring) + six.string_types[0]) self.__public_key = public_key @property diff --git a/common/src/main/python/dlpx/virtualization/common/exceptions.py b/common/src/main/python/dlpx/virtualization/common/exceptions.py index d1032a81..3ecef88e 100644 --- a/common/src/main/python/dlpx/virtualization/common/exceptions.py +++ b/common/src/main/python/dlpx/virtualization/common/exceptions.py @@ -1,5 +1,5 @@ # -# Copyright (c) 2019 by Delphix. All rights reserved. +# Copyright (c) 2019, 2021 by Delphix. All rights reserved. # @@ -77,7 +77,12 @@ def _remove_angle_brackets(type_string): return type_string.replace('<', '').replace('>', '') def _get_type_name(type_object): - if type_object.__module__ != '__builtin__': + # + # In py3 the builtins module will be named 'builtins', in py2 it will + # be '__builtin__'. + # + builtins = ['__builtin__', 'builtins'] + if type_object.__module__ not in builtins: type_name = '{}.{}'.format( type_object.__module__, type_object.__name__) else: @@ -99,9 +104,13 @@ def _get_type_name(type_object): if len(expected_type) > 1: for index in range(0, len(expected_type)): expected_type[index] = _get_type_name(expected_type[index]) - expected_type[index] = _remove_angle_brackets(str(expected_type[index])) + expected_type[index] = _remove_angle_brackets( + str(expected_type[index])) expected = "any one of the following types: '{}'".format(expected_type) + elif len(expected_type) == 0: + raise PlatformError('The thrown TypeError should have had a list' + ' of size >= 1 as the expected_type') else: single_type = expected_type[0] type_name = _get_type_name(single_type) @@ -112,8 +121,8 @@ def _get_type_name(type_object): raise PlatformError('The thrown TypeError should have had a' ' dict of size 1 as the expected_type') - key_type = expected_type.keys()[0] - value_type = expected_type.values()[0] + key_type = list(expected_type.keys())[0] + value_type = list(expected_type.values())[0] key_type_name = _get_type_name(key_type) value_type_name = _get_type_name(value_type) @@ -139,7 +148,7 @@ def _get_type_name(type_object): raise PlatformError('The thrown TypeError should have had a' ' set of tuples to represent a dict') actual = 'a dict of {{{}}}'.format(', '.join(['{0}:{1}'.format( - _remove_angle_brackets(str(k)), + _remove_angle_brackets(str(k)), _remove_angle_brackets(str(v))) for k, v in actual_type])) else: actual = _remove_angle_brackets(str(actual_type)) @@ -164,12 +173,8 @@ class IncorrectTypeError(PluginRuntimeError): """ def __init__( - self, - object_type, - parameter_name, - actual_type, - expected_type, - required=True): + self, object_type, parameter_name, actual_type, expected_type, + required=True): actual, expected = self.get_actual_and_expected_type( actual_type, expected_type) @@ -179,4 +184,4 @@ def __init__( actual, expected, (' if defined', '')[required])) - super(IncorrectTypeError, self).__init__(message) \ No newline at end of file + super(IncorrectTypeError, self).__init__(message) diff --git a/common/src/main/python/dlpx/virtualization/common/util.py b/common/src/main/python/dlpx/virtualization/common/util.py new file mode 100644 index 00000000..2ff42f6f --- /dev/null +++ b/common/src/main/python/dlpx/virtualization/common/util.py @@ -0,0 +1,125 @@ +# +# Copyright (c) 2021 by Delphix. All rights reserved. +# + + +""" +Utility functions to convert between unicode and bytes. +""" + +import six + + +def to_bytes(string, encoding="utf-8"): + """ + Converts the given object to binary object, bytes (Py3) or str (Py2). + + :param string: The string like object to convert to bytes + :type string: ``object`` + :param encoding: The encoding to encode the string with. + :type encoding: ``str`` + :returns: The encoded string. + :rtype: ``bytes`` + """ + if string is None: + return + + if isinstance(string, dict): + for k, v in string.items(): + string[k] = to_bytes(v, encoding=encoding) + return string + + if isinstance(string, list): + return [to_bytes(i, encoding=encoding) for i in string] + + if isinstance(string, set): + return {to_bytes(i, encoding=encoding) for i in string} + + if isinstance(string, str): + return _to_bytes(string, encoding) + + return string + + +def _to_bytes(string, encoding): + if six.PY3: + if isinstance(string, str): + return string.encode(encoding) + else: + return bytes(string) + else: + if isinstance(string, unicode): # noqa + return string.encode(encoding) + else: + return str(string) + + +def to_str(b, encoding="utf-8"): + """ + Converts the given object to a text object, unicode (Py2) or str (Py3). + + :param b: The object to convert + :type b: ``object`` + :param encoding: The encoding to encode the string with. + :type encoding: ``str`` + :returns: The decoded string. + :rtype: ``str`` + """ + if b is None: + return + + if isinstance(b, dict): + for k, v in b.items(): + b[k] = to_str(v, encoding=encoding) + return b + + if isinstance(b, list): + return [to_str(i, encoding=encoding) for i in b] + + if isinstance(b, set): + return {to_str(i, encoding=encoding) for i in b} + + if isinstance(b, bytes): + return _to_str(b, encoding=encoding) + return b + + +def _to_str(b, encoding): + if six.PY3: + if isinstance(b, bytes): + try: + return str(b, encoding) + except UnicodeDecodeError: + pass + raise UnicodeError( + "Could not decode value with encoding {}".format(encoding) + ) + else: + return b + else: + if isinstance(b, str): + try: + return b.decode(encoding) + except UnicodeDecodeError: + pass + raise UnicodeError( + "Could not decode value with encoding {}".format(encoding) + ) + return b + + +def response_to_str(response): + """ + The response_to_str function ensures all relevant properties of the given + response are unicode (py2) / str (py3). Should be called on a response as + soon as it is received. + + Args: + response (RunPowerShellResponse or RunBashResponse or RunExpectResponse): + Response received by run_bash or run_powershell or run_expect + """ + if response.HasField("return_value"): + if hasattr(response.return_value, "stdout"): + response.return_value.stdout = to_str(response.return_value.stdout) + if hasattr(response.return_value, "stderr"): + response.return_value.stderr = to_str(response.return_value.stderr) diff --git a/common/src/test/python/dlpx/virtualization/common/__init__.py b/common/src/test/python/dlpx/virtualization/common/__init__.py index c7fd3fc1..e9e178ce 100644 --- a/common/src/test/python/dlpx/virtualization/common/__init__.py +++ b/common/src/test/python/dlpx/virtualization/common/__init__.py @@ -1,5 +1,5 @@ # -# Copyright (c) 2019 by Delphix. All rights reserved. +# Copyright (c) 2019, 2021 by Delphix. All rights reserved. # -__path__ = __import__('pkgutil').extend_path(__path__, __name__) \ No newline at end of file +__path__ = __import__('pkgutil').extend_path(__path__, __name__) diff --git a/common/src/test/python/dlpx/virtualization/common/test_common_classes.py b/common/src/test/python/dlpx/virtualization/common/test_common_classes.py index d0ede0f7..16d6373d 100644 --- a/common/src/test/python/dlpx/virtualization/common/test_common_classes.py +++ b/common/src/test/python/dlpx/virtualization/common/test_common_classes.py @@ -1,11 +1,16 @@ # -# Copyright (c) 2019 by Delphix. All rights reserved. +# Copyright (c) 2019, 2021 by Delphix. All rights reserved. # import pytest +import six from dlpx.virtualization.api import common_pb2 -from dlpx.virtualization.common._common_classes import (RemoteConnection, RemoteEnvironment, RemoteHost, RemoteUser) -from dlpx.virtualization.common.exceptions import IncorrectTypeError +from dlpx.virtualization.common._common_classes import ( + KeyPairCredentials, PasswordCredentials, RemoteConnection, RemoteEnvironment, + RemoteHost, RemoteUser) +from dlpx.virtualization.common.exceptions import ( + IncorrectTypeError, PluginRuntimeError, PlatformError) + @pytest.fixture def remote_user(): @@ -31,19 +36,31 @@ def test_init_remote_connection_success(remote_user, remote_environment): def test_init_remote_connection_incorrect_environment(remote_user): with pytest.raises(IncorrectTypeError) as err_info: RemoteConnection('', remote_user) - assert err_info.value.message == ( - "RemoteConnection's parameter 'environment' was" - " type 'str' but should be of class 'dlpx.virtualization" - ".common._common_classes.RemoteEnvironment'.") + if six.PY2: + assert err_info.value.message == ( + "RemoteConnection's parameter 'environment' was" + " type 'str' but should be of class 'dlpx.virtualization" + ".common._common_classes.RemoteEnvironment'.") + else: + assert err_info.value.message == ( + "RemoteConnection's parameter 'environment' was" + " class 'str' but should be of class 'dlpx.virtualization" + ".common._common_classes.RemoteEnvironment'.") @staticmethod def test_init_remote_connection_incorrect_user(remote_environment): with pytest.raises(IncorrectTypeError) as err_info: RemoteConnection(remote_environment, '') - assert err_info.value.message == ( - "RemoteConnection's parameter 'user' was" - " type 'str' but should be of class 'dlpx.virtualization" - ".common._common_classes.RemoteUser'.") + if six.PY2: + assert err_info.value.message == ( + "RemoteConnection's parameter 'user' was" + " type 'str' but should be of class 'dlpx.virtualization" + ".common._common_classes.RemoteUser'.") + else: + assert err_info.value.message == ( + "RemoteConnection's parameter 'user' was" + " class 'str' but should be of class 'dlpx.virtualization" + ".common._common_classes.RemoteUser'.") @staticmethod def test_remote_connection_to_proto(remote_user, remote_environment): @@ -61,10 +78,16 @@ def test_remote_connection_from_proto_success(): def test_remote_connection_from_proto_fail(): with pytest.raises(IncorrectTypeError) as err_info: RemoteConnection.from_proto('') - assert err_info.value.message == ( - "RemoteConnection's parameter 'connection' was" - " type 'str' but should be of class 'dlpx.virtualization.api" - ".common_pb2.RemoteConnection'.") + if six.PY2: + assert err_info.value.message == ( + "RemoteConnection's parameter 'connection' was" + " type 'str' but should be of class 'dlpx.virtualization.api" + ".common_pb2.RemoteConnection'.") + else: + assert err_info.value.message == ( + "RemoteConnection's parameter 'connection' was" + " class 'str' but should be of class 'dlpx.virtualization.api" + ".common_pb2.RemoteConnection'.") class TestRemoteEnvironment: @@ -76,26 +99,42 @@ def test_init_remote_environment_success(remote_host): def test_init_remote_environment_incorrect_name(remote_host): with pytest.raises(IncorrectTypeError) as err_info: RemoteEnvironment(1, '', remote_host) - assert err_info.value.message == ( - "RemoteEnvironment's parameter 'name' was" - " type 'int' but should be of type 'basestring'.") + if six.PY2: + assert err_info.value.message == ( + "RemoteEnvironment's parameter 'name' was" + " type 'int' but should be of type 'basestring'.") + else: + assert err_info.value.message == ( + "RemoteEnvironment's parameter 'name' was" + " class 'int' but should be of class 'str'.") @staticmethod def test_init_remote_environment_incorrect_reference(remote_host): with pytest.raises(IncorrectTypeError) as err_info: RemoteEnvironment('', 1, remote_host) - assert err_info.value.message == ( - "RemoteEnvironment's parameter 'reference' was" - " type 'int' but should be of type 'basestring'.") + if six.PY2: + assert err_info.value.message == ( + "RemoteEnvironment's parameter 'reference' was" + " type 'int' but should be of type 'basestring'.") + else: + assert err_info.value.message == ( + "RemoteEnvironment's parameter 'reference' was" + " class 'int' but should be of class 'str'.") @staticmethod def test_init_remote_environment_incorrect_host(): with pytest.raises(IncorrectTypeError) as err_info: RemoteEnvironment('', '', '') - assert err_info.value.message == ( - "RemoteEnvironment's parameter 'host' was" - " type 'str' but should be of class 'dlpx.virtualization" - ".common._common_classes.RemoteHost'.") + if six.PY2: + assert err_info.value.message == ( + "RemoteEnvironment's parameter 'host' was" + " type 'str' but should be of class 'dlpx.virtualization" + ".common._common_classes.RemoteHost'.") + else: + assert err_info.value.message == ( + "RemoteEnvironment's parameter 'host' was" + " class 'str' but should be of class 'dlpx.virtualization" + ".common._common_classes.RemoteHost'.") @staticmethod def test_remote_environment_to_proto(remote_host): @@ -113,10 +152,16 @@ def test_remote_environment_from_proto_success(): def test_remote_environment_from_proto_fail(): with pytest.raises(IncorrectTypeError) as err_info: RemoteEnvironment.from_proto('') - assert err_info.value.message == ( - "RemoteEnvironment's parameter 'environment' was" - " type 'str' but should be of class 'dlpx.virtualization.api" - ".common_pb2.RemoteEnvironment'.") + if six.PY2: + assert err_info.value.message == ( + "RemoteEnvironment's parameter 'environment' was" + " type 'str' but should be of class 'dlpx.virtualization.api" + ".common_pb2.RemoteEnvironment'.") + else: + assert err_info.value.message == ( + "RemoteEnvironment's parameter 'environment' was" + " class 'str' but should be of class 'dlpx.virtualization.api" + ".common_pb2.RemoteEnvironment'.") class TestRemoteHost: @@ -131,33 +176,53 @@ def test_init_remote_host_success(): def test_init_remote_host_incorrect_name(): with pytest.raises(IncorrectTypeError) as err_info: RemoteHost(1, '', '', '') - assert err_info.value.message == ( - "RemoteHost's parameter 'name' was" - " type 'int' but should be of type 'basestring'.") + if six.PY2: + assert err_info.value.message == ( + "RemoteHost's parameter 'name' was" + " type 'int' but should be of type 'basestring'.") + else: + assert err_info.value.message == ( + "RemoteHost's parameter 'name' was" + " class 'int' but should be of class 'str'.") @staticmethod def test_init_remote_host_incorrect_reference(): with pytest.raises(IncorrectTypeError) as err_info: RemoteHost('', 1, '', '') - assert err_info.value.message == ( - "RemoteHost's parameter 'reference' was" - " type 'int' but should be of type 'basestring'.") + if six.PY2: + assert err_info.value.message == ( + "RemoteHost's parameter 'reference' was" + " type 'int' but should be of type 'basestring'.") + else: + assert err_info.value.message == ( + "RemoteHost's parameter 'reference' was" + " class 'int' but should be of class 'str'.") @staticmethod def test_init_remote_host_incorrect_binary_path(): with pytest.raises(IncorrectTypeError) as err_info: RemoteHost('', '', 1, '') - assert err_info.value.message == ( - "RemoteHost's parameter 'binary_path' was" - " type 'int' but should be of type 'basestring'.") + if six.PY2: + assert err_info.value.message == ( + "RemoteHost's parameter 'binary_path' was" + " type 'int' but should be of type 'basestring'.") + else: + assert err_info.value.message == ( + "RemoteHost's parameter 'binary_path' was" + " class 'int' but should be of class 'str'.") @staticmethod def test_init_remote_host_incorrect_scratch_path(): with pytest.raises(IncorrectTypeError) as err_info: RemoteHost('', '', '', 1) - assert err_info.value.message == ( - "RemoteHost's parameter 'scratch_path' was" - " type 'int' but should be of type 'basestring'.") + if six.PY2: + assert err_info.value.message == ( + "RemoteHost's parameter 'scratch_path' was" + " type 'int' but should be of type 'basestring'.") + else: + assert err_info.value.message == ( + "RemoteHost's parameter 'scratch_path' was" + " class 'int' but should be of class 'str'.") @staticmethod def test_remote_host_to_proto_non_default(): @@ -175,10 +240,16 @@ def test_remote_host_from_proto_success(): def test_remote_host_from_proto_fail(): with pytest.raises(IncorrectTypeError) as err_info: RemoteHost.from_proto('') - assert err_info.value.message == ( - "RemoteHost's parameter 'host' was" - " type 'str' but should be of class 'dlpx.virtualization.api" - ".common_pb2.RemoteHost'.") + if six.PY2: + assert err_info.value.message == ( + "RemoteHost's parameter 'host' was" + " type 'str' but should be of class 'dlpx.virtualization.api" + ".common_pb2.RemoteHost'.") + else: + assert err_info.value.message == ( + "RemoteHost's parameter 'host' was" + " class 'str' but should be of class 'dlpx.virtualization.api" + ".common_pb2.RemoteHost'.") class TestRemoteUser: @@ -190,17 +261,27 @@ def test_init_remote_user_success(): def test_init_remote_user_incorrect_name(): with pytest.raises(IncorrectTypeError) as err_info: RemoteUser(1, '') - assert err_info.value.message == ( - "RemoteUser's parameter 'name' was" - " type 'int' but should be of type 'basestring'.") + if six.PY2: + assert err_info.value.message == ( + "RemoteUser's parameter 'name' was" + " type 'int' but should be of type 'basestring'.") + else: + assert err_info.value.message == ( + "RemoteUser's parameter 'name' was" + " class 'int' but should be of class 'str'.") @staticmethod def test_init_remote_user_incorrect_reference(): with pytest.raises(IncorrectTypeError) as err_info: RemoteUser('', 1) - assert err_info.value.message == ( - "RemoteUser's parameter 'reference' was" - " type 'int' but should be of type 'basestring'.") + if six.PY2: + assert err_info.value.message == ( + "RemoteUser's parameter 'reference' was" + " type 'int' but should be of type 'basestring'.") + else: + assert err_info.value.message == ( + "RemoteUser's parameter 'reference' was" + " class 'int' but should be of class 'str'.") @staticmethod def test_remote_user_to_proto_non_default(): @@ -218,7 +299,133 @@ def test_remote_user_from_proto_success(): def test_remote_user_from_proto_fail(): with pytest.raises(IncorrectTypeError) as err_info: RemoteUser.from_proto('') + if six.PY2: + assert err_info.value.message == ( + "RemoteUser's parameter 'user' was" + " type 'str' but should be of class 'dlpx.virtualization.api" + ".common_pb2.RemoteUser'.") + else: + assert err_info.value.message == ( + "RemoteUser's parameter 'user' was" + " class 'str' but should be of class 'dlpx.virtualization.api" + ".common_pb2.RemoteUser'.") + + +class TestCredentials: + @staticmethod + def test_init_credentials_incorrect_username_type(): + with pytest.raises(IncorrectTypeError) as err_info: + KeyPairCredentials(RemoteUser("user1", "reference1"), "1234", "5678") + if six.PY2: + assert err_info.value.message == ( + "Credentials's parameter 'username' was class " + "'dlpx.virtualization.common._common_classes.RemoteUser' but should be " + "of type 'basestring'.") + else: + assert err_info.value.message == ( + "Credentials's parameter 'username' was class " + "'dlpx.virtualization.common._common_classes.RemoteUser' but should be " + "of class 'str'.") + + +class TestKeyPairCredentials: + @staticmethod + def test_init_key_pair_credentials_incorrect_public_key_type(): + with pytest.raises(IncorrectTypeError) as err_info: + KeyPairCredentials("user1", "1234", 1) + if six.PY2: + assert err_info.value.message == ( + "KeyPairCredentials's parameter 'public_key' was type 'int' " + "but should be of type 'basestring'.") + else: + assert err_info.value.message == ( + "KeyPairCredentials's parameter 'public_key' was class 'int' " + "but should be of class 'str'.") + + @staticmethod + def test_init_key_pair_credentials_incorrect_private_key_type(): + with pytest.raises(IncorrectTypeError) as err_info: + KeyPairCredentials("user1", 1, "1234") + if six.PY2: + assert err_info.value.message == ( + "KeyPairCredentials's parameter 'private_key' was type 'int' " + "but should be of type 'basestring'.") + else: + assert err_info.value.message == ( + "KeyPairCredentials's parameter 'private_key' was class 'int' " + "but should be of class 'str'.") + + +class TestPasswordCredentials: + @staticmethod + def test_init_password_credentials_incorrect_password_type(): + with pytest.raises(IncorrectTypeError) as err_info: + PasswordCredentials("user1", 12345) + if six.PY2: + assert err_info.value.message == ( + "PasswordCredentials's parameter 'password' was type 'int' but " + "should be of type 'basestring'.") + else: + assert err_info.value.message == ( + "PasswordCredentials's parameter 'password' was class 'int' but " + "should be of class 'str'.") + + +class TestPluginRuntimeError: + @staticmethod + def test_plugin_runtime_error_get_actual_and_expected_type(): + actual, expected = PluginRuntimeError.get_actual_and_expected_type( + list([str]), dict({str: dict})) + if six.PY2: + assert actual == "a list of [type 'str']" + else: + assert actual == "a list of [class 'str']" + assert expected == "type 'dict of str:dict'" + + @staticmethod + def test_plugin_runtime_error_get_actual_and_expected_type_multi_expected_types(): + actual, expected = PluginRuntimeError.get_actual_and_expected_type( + list([str]), list([str, int, dict, bool])) + if six.PY2: + assert actual == "a list of [type 'str']" + else: + assert actual == "a list of [class 'str']" + assert expected == ( + "any one of the following types: '['str', 'int', 'dict', 'bool']'") + + @staticmethod + def test_plugin_runtime_error_get_actual_and_expected_type_single_item_list_expected_types(): # noqa + actual, expected = PluginRuntimeError.get_actual_and_expected_type( + list([str]), list([str])) + if six.PY2: + assert actual == "a list of [type 'str']" + else: + assert actual == "a list of [class 'str']" + assert expected == "type 'list of str'" + + @staticmethod + def test_plugin_runtime_error_get_actual_and_expected_type_empty_dict_expected_type(): # noqa + with pytest.raises(PlatformError) as err_info: + PluginRuntimeError.get_actual_and_expected_type(list(), dict()) + + assert err_info.value.message == ( + "The thrown TypeError should have had a dict of size 1 as the " + "expected_type") + + @staticmethod + def test_plugin_runtime_error_get_actual_and_expected_type_empty_list_expected_type(): # noqa + with pytest.raises(PlatformError) as err_info: + PluginRuntimeError.get_actual_and_expected_type(list(), list()) + + assert err_info.value.message == ( + "The thrown TypeError should have had a list of size >= 1 as the " + "expected_type") + + @staticmethod + def test_plugin_runtime_error_get_actual_and_expected_type_list_with_duplicate_expected_type(): # noqa + with pytest.raises(PlatformError) as err_info: + PluginRuntimeError.get_actual_and_expected_type(list(), list([str, str])) + assert err_info.value.message == ( - "RemoteUser's parameter 'user' was" - " type 'str' but should be of class 'dlpx.virtualization.api" - ".common_pb2.RemoteUser'.") + "The thrown TypeError should have had a list of size 1 as the " + "expected_type") diff --git a/docs/CONTRIBUTING.md b/docs/CONTRIBUTING.md index 9db47fd6..1e44ce46 100644 --- a/docs/CONTRIBUTING.md +++ b/docs/CONTRIBUTING.md @@ -70,7 +70,7 @@ We have two goals: Provide the best documentation we can for our customers, and 1. Learn Markdown or use a really good IDE. It's easy to use, but there are complex topics like tables, admonishments, links, and images that you may need some practice with. Look at the other docs in the repo for inspiration and tutelage. 2. Test everything in mkdocs locally. Best practice is to always have mkdocs running in one terminal tab. It auto-refreshes when you make changes, so you can make sure that nothing breaks, and that your content looks good. -3. Do not create new directories (nav categories) in /docs without working with Jaspal Sumal (jaspal.sumal@delphix.com) +3. Do not create new directories (nav categories) in /docs without working with Ryan Fowler (ryan.fowler@delphix.com) 4. Place all screenshots in the local media/ directory of the category you're editing in. For example, if you're editing a page in docs/Getting_Started, put any screenshots you're going to use in docs/Getting_Started/media 5. Use relative links to reference screenshots (./media/image.png) and other pages (../Getting_Started/pagename/) 6. Beware the .pages file. .pages is a hidden file in every folder that provides page order. Any pages not listed in .pages will be alphabetically ordered _after_ the pages that have been listed. If you have a typo in this file or specify a renamed/deleted page, it will break mkdocs. @@ -84,16 +84,16 @@ We have two goals: Provide the best documentation we can for our customers, and 1. The diff can usually provide what you need for reviewing changes. However, use mkdocs to review locally whenever possible to ensure good formatting and no breaks to mkdocs. 2. For minor corrections, leave a general comment on the review and vote to ship it so the author can fix it and push. -3. For major docs projects (e.g. whole new sections of docs or large batches of changes), coordinate with Jas. It is possible we'd be better off using another approach to review (e.g. track notes via Google Sheets) -4. If you're a reviewer that is not hooked into reviewboard, and unable to get set up to use it, work with Jas on an alternative approach (e.g. track notes via Google Sheets) +3. For major docs projects (e.g. whole new sections of docs or large batches of changes), coordinate with Ryan. It is possible we'd be better off using another approach to review (e.g. track notes via Google Sheets) +4. If you're a reviewer that is not hooked into reviewboard, and unable to get set up to use it, work with Ryan on an alternative approach (e.g. track notes via Google Sheets) 5. If there are issues in production docs, the current procedure is to post the issue in the #docs channel. ## Publishing Publishing is currently a manual process that will be automated into the release process at a future point in time. The publishing workflow follows these steps: -1. After the git repo is frozen, Jas begins review and adjustments. -2. If there are technical questions or issues, Jas will take back to engineering for review. +1. After the git repo is frozen, Ryan begins review and adjustments. +2. If there are technical questions or issues, Ryan will take back to engineering for review. 3. The publish process will run. This process will: * Pull the appropriate branch to a build machine * Run "mkdocs build clean" to compile documentation to HTML diff --git a/docs/docs/Best_Practices/.pages b/docs/docs/Best_Practices/.pages index 9a05e0d3..ceea0305 100644 --- a/docs/docs/Best_Practices/.pages +++ b/docs/docs/Best_Practices/.pages @@ -4,6 +4,8 @@ arrange: - Managing_Scripts_For_Remote_Execution.md - User_Visible_Errors.md - Sensitive_Data.md - - Unicode_Data.md + - Strings.md + - Runtime_Environment.md - Working_with_Powershell.md - Scratch_Paths.md + - Message_Limits.md diff --git a/docs/docs/Best_Practices/Code_Sharing.md b/docs/docs/Best_Practices/Code_Sharing.md index 5640e6a2..1dafe9bb 100644 --- a/docs/docs/Best_Practices/Code_Sharing.md +++ b/docs/docs/Best_Practices/Code_Sharing.md @@ -2,7 +2,7 @@ All Python modules inside of `srcDir` can be imported just as they would be if the plugin was executing locally. When a plugin operation is executed `srcDir` is the current working directory so all imports need to be relative to `srcDir` regardless of the path of the module doing the import. -Please refer to Python's [documentation on modules](https://docs.python.org/2/tutorial/modules.html#modules) to learn more about modules and imports. +Please refer to Python's [documentation on modules](https://docs.python.org/3.8/tutorial/modules.html#modules) to learn more about modules and imports. ## Example @@ -30,10 +30,12 @@ postgres Any module in the plugin could import `execution_util.py` with `from utils import execution_util`. !!! warning "Gotcha" - Since the platform uses Python 2.7, every directory needs to have an `__init__.py` file in it otherwise the modules and resources in the folder will not be found at runtime. For more information on `__init__.py` files refer to Python's [documentation on packages](https://docs.python.org/2/tutorial/modules.html#packages). - + When using a vSDK version that was built on Python 2.7, every directory needs to have an `__init__.py` file in it otherwise the modules and resources in the folder will not be found at runtime. For more information on `__init__.py` files refer to Python's [documentation on packages](https://docs.python.org/2/tutorial/modules.html#packages). + Note that the `srcDir` in the plugin config file (`src` in this example) does _not_ need an `__init__.py` file. + For information on which vSDK versions run on Python 2.7, visit the [Version Compatibility Page](/References/Version_Compatibility/#virtualization-sdk-and-python-compatibility-map). + Assume `schema.json` contains: ``` @@ -117,21 +119,21 @@ def find_schemas(source_connection, repository): return [SourceConfigDefinition(name=name) for name in schema_names] ``` !!! note - Even though `discovery.py` is in the `operations` package, the import for `execution_util` is still relative to the `srcDir` specified in the plugin config file. `execution_util` is in the `utils` package so it is imported with `from utils import execution_util`. - + Even though `discovery.py` is in the `operations` package, the import for `execution_util` is still relative to the `srcDir` specified in the plugin config file. `execution_util` is in the `utils` package so it is imported with `from utils import execution_util`. + ### execution_util.py `execution_util.py ` has two methods `execute_sql` and `execute_shell`. `execute_sql` takes the name of a SQL script in `resources/` and executes it with `resources/execute_sql.sh`. `execute_shell` takes the name of a shell script in `resources/` and executes it. ```python -import pkgutil +from importlib import resources from dlpx.virtualization import libs def execute_sql(source_connection, install_name, script_name): - psql_script = pkgutil.get_data("resources", "execute_sql.sh") - sql_script = pkgutil.get_data("resources", script_name) + psql_script = resources.read_text("resources", "execute_sql.sh") + sql_script = resources.read_text("resources", script_name) result = libs.run_bash( source_connection, psql_script, variables={"SCRIPT": sql_script}, check=True @@ -140,11 +142,16 @@ def execute_sql(source_connection, install_name, script_name): def execute_shell(source_connection, script_name): - script = pkgutil.get_data("resources", script_name) + script = resources.read_text("resources", script_name) result = libs.run_bash(source_connection, script, check=True) return result.stdout ``` +!!! warning + If developing a plugin in Python 2.7, you will need to use `pkgutil.get_data` rather than `importlib.resources.read_text`. + + See [Managing Scripts For Remote Execution](/Best_Practices/Managing_Scripts_For_Remote_Execution.md) for more info. + !!! note - Both `execute_sql` and `execute_shell` use the `check` parameter which will cause an error to be raised if the exit code is non-zero. For more information refer to the `run_bash` [documentation](/References/Platform_Libraries.md#run_bash). \ No newline at end of file + Both `execute_sql` and `execute_shell` use the `check` parameter which will cause an error to be raised if the exit code is non-zero. For more information refer to the `run_bash` [documentation](/References/Platform_Libraries.md#run_bash). diff --git a/docs/docs/Best_Practices/Managing_Scripts_For_Remote_Execution.md b/docs/docs/Best_Practices/Managing_Scripts_For_Remote_Execution.md index ef63e26c..853f0f56 100644 --- a/docs/docs/Best_Practices/Managing_Scripts_For_Remote_Execution.md +++ b/docs/docs/Best_Practices/Managing_Scripts_For_Remote_Execution.md @@ -1,9 +1,11 @@ # Managing Scripts for Remote Execution -To execute a PowerShell or Bash script or Expect script on a remote host, you must provide the script as a string to `run_powershell` or `run_bash` or `run_expect`. While you can keep these strings as literals in your Python code, best practice is to keep them as resource files in your source directory and access them with `pkgutil`. +To execute a PowerShell or Bash script or Expect script on a remote host, you must provide the script as a string to `run_powershell` or `run_bash` or `run_expect`. While you can keep these strings as literals in your Python code, best practice is to keep them as resource files in your source directory and access them with `pkgutil` or `importlib`, depending on your plugin language. [pkgutil](https://docs.python.org/2/library/pkgutil.html) is part of the standard Python library. The method that is applicable to resources is [pkgutil.get_data](https://docs.python.org/2/library/pkgutil.html#pkgutil.get_data). +When developing a plugin in Python3, it is instead suggested to use the newer `importlib.resources`. This package is part of the standard Python 3 library. The method that is applicable to resources is [resources.read_text](https://docs.python.org/3.8/library/importlib.html#importlib.resources.read_text), which accepts the same arguments as `pkgutil.get_data`. + ### Basic Usage Given the following plugin structure: @@ -66,7 +68,7 @@ def post_snapshot(direct_source, repository, source_config): raise UserError( 'Failed to get date', 'Make sure the user has the required permissions', - '{}\n{}'.format(response.stdout, rsponse.stderr)) + '{}\n{}'.format(response.stdout, response.stderr)) return SnapshotDefinition(name='Snapshot', date=response.stdout) ``` @@ -75,7 +77,9 @@ def post_snapshot(direct_source, repository, source_config): This assumes that `src/` is Python's current working directory. This is the behavior of the Virtualization Platform. !!! warning "Resources need to be in a Python module" - `pkgutil.get_data` cannot retrieve the contents of a resource that is not in a Python package. This means that a resource that is in the first level of your source directory will not be retrievable with `pkgutil`. Resources must be in a subdirectory of your source directory, and that subdirectory must contain an `__init__.py` file. + `pkgutil.get_data` cannot retrieve the contents of a resource that is not in a Python package. When developing with Python 2.7, this means that a resource that is in the first level of your source directory will not be retrievable with `pkgutil`. Resources must be in a subdirectory of your source directory, and that subdirectory must contain an `__init__.py` file. + + Python 3 does not have this requirement. ### Multi-level Packages @@ -101,3 +105,9 @@ The contents of `src/resources/platform/get_date.sh` can be retrieved with: ```python script_content = pkgutil.get_data('resources.platform', 'get_date.sh') ``` + +In a Python 3.8 plugin, the suggested approach is: + +```python +script_content = resources.read_text('resources.platform', 'get_date.sh') +``` diff --git a/docs/docs/Best_Practices/Message_Limits.md b/docs/docs/Best_Practices/Message_Limits.md new file mode 100644 index 00000000..a17d352e --- /dev/null +++ b/docs/docs/Best_Practices/Message_Limits.md @@ -0,0 +1,44 @@ +# Message Limits + +There are limits on how much data can be sent back and forth between the plugin and engine at a time. There are five scenarios where this comes into play: + +1. Inputs sent from the engine to the plugin, as arguments to a [Plugin Operation](/References/Plugin_Operations.md). For example, the schema-defined `Repository` object that is provided as input to plugin operations. + +2. Outputs sent back from the plugin to the engine, as the return values from plugin operations. + +3. Exception messages and call stacks thrown by plugin code. For example, the `message` field within [User Visible Errors](/Best_Practices/User_Visible_Errors.md). + +4. Inputs sent from the plugin to the engine, as arguments to a [Platform library](/References/Platform_Libraries.md) function. For example, the `message` field that is passed to `logger.debug`. + +5. Outputs sent back from the engine to the plugin, as the return values from Platform Library functions. For example, the `stdout` resulting from a call to `libs.run_bash`. + +For case 1 and 2, the total size of data must be less than 4 mebibytes (4 MiB). + +For case 3, the total size of data must be less than 128 kibibytes (128 KiB). + +For case 4 and 5, the total size of data must be less than 192 mebibytes (192 MiB). + +The actual size of this information at runtime is dependent on how the Python interpreter chooses to represent the information, so it's not always possible to know ahead of time what the exact size will be. + +Here are some examples of where problems may occur: + +1. Using `libs.run_bash` to print the entire contents of a large file to stdout. + +2. Using a single `logger` command with many pages of output. + +3. Throwing an exception with a large message or stack trace. + +4. Large amount of metadata in a plugin defined schema like `Repository` or `Virtual Source`. + +## How to tell if the message size was exceeded + +The plugin operation or platform library callback will fail with a RPC error. The exception will look like: + +``` +Error +Discovery of "my_plugin" failed: Plugin operation "Repository Discovery" got a RPC error for plugin "my_plugin". UNAVAILABLE: Network closed for unknown reason +``` + +## What to do if the maximum metadata or message size is exceeded + +Reach out to us via the [Virtualization SDK GitHub repository](https://github.com/delphix/virtualization-sdk/) for guidance. \ No newline at end of file diff --git a/docs/docs/Best_Practices/Runtime_Environment.md b/docs/docs/Best_Practices/Runtime_Environment.md new file mode 100644 index 00000000..aadaf9cf --- /dev/null +++ b/docs/docs/Best_Practices/Runtime_Environment.md @@ -0,0 +1,63 @@ +# Plugin Runtime Environment + +## Process Lifetime +Plugin code runs inside of a Python interpreter process on the Delphix Engine. + +A fair question to ask is "What is the lifetime of this interpreter process?" After all, if the interpreter +process runs for a long time, then the plugin might be able to store things in memory for later access. + +Unfortunately, **there are no guarantees about process lifetime**. Your interpreter process could last two years, or it could last 400 microseconds. There is no way to know or predict this ahead of time. + +So, do not make any assumptions about interpreter process lifetime in your plugin code. + + +## Available Modules +Our Python 2.7 runtime environment only contains the [Python Standard Library](https://docs.python.org/2/library/). No additional Python modules/libraries are available. + +If you want to use some Python module that is not part of the standard library, you might be able to do so. +You would need to include that library as part of your plugin. That would involve downloading the source +code for that module, and copying it into your source directory. For more information on how to lay out code in your source directory, see [Code Sharing](/Best_Practices/Code_Sharing.md). + +### Warnings +There are two major things to watch out for if you decide to incorporate a 3rd-party library. + +1) Make sure you're legally allowed to do so! The licensing agreement on the module will decide if, and +under what circumstances, you're allowed to make copies of, and redistribute the module. Some modules will +allow this, some will disallow this, and some will allow this for a fee. + +2) Some Python libraries include native code (often written in C or C++). There is no support for using +such libraries with plugin code. The reason for this is that native code needs to be +specially compiled and built for the machine that it the library will be running on. And, unfortunately, +the machine your plugin runs on (the Delphix Engine) is likely very different from the machine you use +to develop and build your plugin. + +## Network Access +As of Delphix Engine version 6.0.11.0, plugin code is able to use the network directly. No network access is +possible in earlier versions. + +For example, suppose your plugin wants to talk to some DBMS running on some remote host. +If the DBMS supports it, your plugin code might be able to connect to the DBMS server and talk to the +DBMS directly. This can avoid the need to do DBMS operations via running Bash/Powershell code on the remote host. + + +### Example +```python +import httplib +import json + +dbms_port = 5432 + +# Directly contact our DBMS's REST server to get a list of databases +def list_databases(remote_ip): + cx = httplib.HTTPConnection(remote_ip, dbms_port) + cx.request("GET", "/databases") + response = cx.getresponse() + return json.loads(response.read()) +``` + +What your plugin can access depends entirely on the customer. Some customers will set up their Delphix +Engines such that plugins have full access to the entire internet. Some will completely restrict the network +so that the plugin can only access a small handful of remote hosts. + +If your plugin has any specific network requirements, it's recommended to try, in your code, to confirm that these requirements are met. For example, the plugin could make such a check in the +`discovery.repository()` operation, and throw an error if the check fails. Like any other requirement, this should of course be documented. diff --git a/docs/docs/Best_Practices/Strings.md b/docs/docs/Best_Practices/Strings.md new file mode 100644 index 00000000..1dfb5f5c --- /dev/null +++ b/docs/docs/Best_Practices/Strings.md @@ -0,0 +1,90 @@ +# Working With Strings + +Unfortunately, Python 2.7 makes it very easy to accidentally write string-related code that will sometimes work, but sometimes fail (especially for people who are not using English). Read on for some tips for how to avoid this. + +## The Two String Types +Python 2.7 has two different types that are both called "strings". One represents +a sequence of **bytes**, and the other represents a sequence of **characters**. + +```python +# The default string (aka 'str object') represents bytes +my_bytes = "This string is a sequence of bytes" + +# A 'Unicode object' represents characters (note the u just before the quote) +my_characters = u"This string is a sequence of characters" +``` + +## Unicode Strings Are Preferred + +There are a couple of reasons to prefer the "unicode object" over the "str object". + +First, in most cases, we care about characters, and we're not particularly interested in which bytes +are used to represent those characters. That is, we might care that we have a "letter H" followed by a "letter I", but it's usually irrelevant to us what byte values happen to be used. + +Second, there are lots of different schemes available which give rules for how to represent characters as bytes. These schemes are called "encodings"; some examples include "ASCII", "UTF-8", "Shift-JIS", and "UCS-2". Each encoding uses different rules about which characters are represented by which bytes. + +A "str object" doesn't know anything about encodings... it is just a sequence of bytes. So, when a programmer is working with one of these byte strings, they have to know which encoding rules are in play. + +In order to avoid problems, **we recommend using Unicode strings everywhere** in your plugin code. + +## Delphix I/O + +Your plugin will sometimes need to send strings back and forth to Delphix code. There are two supported formats for doing this. Any time you receive a string from Delphix, it will be in one of the two following forms. This includes arguments to your plugin operations, and return values from "Delphix Libs" functions. Likewise, any time you send a string to Delphix, it must be in one of these two forms. + +Acceptable forms: + +1. A Unicode string (recommended) +2. A "str object" (byte string) that uses the UTF-8 encoding + +## Converting Between Types + +Sometimes (hopefully rarely!), you might find yourself needing to convert back and forth between byte strings and character strings. For example, you might need to read or write a file on a remote system that is required to use some specific encoding. Here's how to do that: + +```python +# Converting from a character string ("unicode") to a byte string ("str") +my_utf8_byte_string = my_character_string.encode("utf-8") +my_utf16_byte_string = my_character_string.encode("utf-16") + +# Converting from a byte string to a character string +my_character_string1 = my_utf8_byte_string.decode("utf-8") +my_character_string2 = my_utf16_byte_string.decode("utf-16") +my_character_string3 = my_ascii_byte_string.decode("ascii") +``` + +Things to note: + +- `encode` goes from characters to bytes. `decode` goes from bytes to characters. +- If you try to `encode` a character string using the `ascii` encoding, but your character string contains non-ascii characters, you'll get an error. More generally: some encodings will error out with some characters. +- If you don't specify an encoding, Python will supply a default. But, there's a good chance the default will be wrong for your use case. So, always specify the encoding! +- Don't try to `encode` a byte string. If you do this, Python will "helpfully" insert an implicit `decode` first, which tends to cause very confusing error messages. Likewise, don't try to `decode` a character string. +- `utf-8` is likely the best encoding to use for most situations. It accepts all characters, does not have issues with byte ordering, and is understood by most systems. This is not true of most other encodings. + +## Using Non-ASCII characters in Python files + +Python 2.7 source code files are assumed to use the "ASCII" encoding, unless told otherwise. Unfortunately, ASCII is an obsolete encoding that only knows how to deal with a small number of characters, and only really supports American English. + +In order to include non-ASCII characters in your source code, you need to use a different encoding than ASCII, and you need to tell the Python interpreter which encoding you're using. In Python 2.7, this is done with a "magic" comment at the very top of each file. + +Here is an example of the first line of a Python file that uses the UTF-8 encoding: +```python +# -*- coding: utf-8 -*- +``` + +If you do not specify an encoding, and the source code contains any non-ASCII characters, you will get errors + when building the plugin using [dvp build](/References/CLI.md#build) or during the execution of a plugin operation. + +### Example + +```python +# -*- coding: utf-8 -*- +from dlpx.virtualization.platform import Plugin +from dlpx.virtualization import libs +from generated.definitions import RepositoryDefinition + +plugin = Plugin() + +@plugin.discovery.repository() +def repository_discovery(source_connection): + # Create a repository with name that uses non-ASCII characters + return [RepositoryDefinition(name=u"Théâtre")] +``` diff --git a/docs/docs/Best_Practices/Unicode_Data.md b/docs/docs/Best_Practices/Unicode_Data.md deleted file mode 100644 index 06b6f88a..00000000 --- a/docs/docs/Best_Practices/Unicode_Data.md +++ /dev/null @@ -1,29 +0,0 @@ -# Working with Unicode Data - -To use unicode characters in the plugin code, the following lines should be included at top of the plugin code: - -```python -#!/usr/bin/env python -# -*- coding: utf-8 -*- -``` - -Otherwise, there may be errors when building the plugin using [dvp build](/References/CLI.md#build) or during the execution of a plugin operation. - -## Example - -```python -#!/usr/bin/env python -# -*- coding: utf-8 -*- -from dlpx.virtualization.platform import Plugin -from dlpx.virtualization import libs -from generated.definitions import RepositoryDefinition - -plugin = Plugin() - -@plugin.discovery.repository() -def repository_discovery(source_connection): - # Create a repository with name ☃ - command = 'echo ☃' - result = libs.run_bash(source_connection, command) - return [RepositoryDefinition(name=result.stdout)] -``` \ No newline at end of file diff --git a/docs/docs/Best_Practices/User_Visible_Errors.md b/docs/docs/Best_Practices/User_Visible_Errors.md index db3476b5..e8ce3a41 100644 --- a/docs/docs/Best_Practices/User_Visible_Errors.md +++ b/docs/docs/Best_Practices/User_Visible_Errors.md @@ -10,11 +10,13 @@ message | String | Description of the failure to show the end user. action | String | **Optional**. List of actions that the end user could take to fix the problem. If not provided, it defaults to `Contact the plugin author to correct the error.` output | String | **Optional**. Output or stack trace from the failure to give the end user more information so that they can self diagnose. If not provided, it defaults to the stack trace of the failure. +!!! warning + There is a limit to how much data can be stored within the fields of a `UserError`. See [Message Limits](/Best_Practices/Message_Limits.md) for details. ## Example ```python -import pkgutil +from importlib import resources from dlpx.virtualization.platform import Plugin from generated.definitions import SourceConfigDefinition from dlpx.virtualization.platform.exceptions import UserError @@ -23,7 +25,7 @@ plugin = Plugin() @plugin.virtual.start() def start(virtual_source, repository, source_config): - script_content = pkgutil.get_data('resources', 'start_database.sh') + script_content = resources.read_text('resources', 'start_database.sh') response = libs.run_bash(virtual_source.connection, script_content) @@ -35,6 +37,11 @@ def start(virtual_source, repository, source_config): '{}\n{}'.format(response.stdout, response.stderr)) ``` +!!! warning + If developing a plugin in Python 2.7, you will need to use `pkgutil.get_data` rather than `importlib.resources.read_text`. + + See [Managing Scripts For Remote Execution](/Best_Practices/Managing_Scripts_For_Remote_Execution.md) for more info. + The UI would show the end user if the plugin operation above fails: ![Screenshot](images/UserError_Start.png) \ No newline at end of file diff --git a/docs/docs/Building_Your_First_Plugin/Initial_Setup.md b/docs/docs/Building_Your_First_Plugin/Initial_Setup.md index d2b0424e..f3bcaa19 100644 --- a/docs/docs/Building_Your_First_Plugin/Initial_Setup.md +++ b/docs/docs/Building_Your_First_Plugin/Initial_Setup.md @@ -53,7 +53,7 @@ To start, we will create a new directory where our new plugin code will live. Now that we are in our new plugin directory, we can use the `dvp` tool to create a plugin for us. This plugin will be a mere skeleton -- it will not do anything useful until we modify it in the subsequent pages. ``` -(venv) first_plugin$ dvp init -n first_plugin -s STAGED -t WINDOWS +(venv) first_plugin$ dvp init -n first_plugin -s STAGED -t UNIX ``` The `-n` argument here means "plugin name." We are using the name `first_plugin`. @@ -111,4 +111,4 @@ You will be prompted for a password. Once the upload is finished, you can verify the installation from the Manage > Toolkits screen in the Delphix Engine UI. -![Screenshot](images/PostUpload.png) \ No newline at end of file +![Screenshot](images/PostUpload.png) diff --git a/docs/docs/Getting_Started.md b/docs/docs/Getting_Started.md index 83d08838..32b9ef8b 100644 --- a/docs/docs/Getting_Started.md +++ b/docs/docs/Getting_Started.md @@ -12,9 +12,10 @@ The platform and libs modules expose objects and methods needed to develop a plu ## Requirements - macOS 10.14+, Ubuntu 16.04+, or Windows 10 -- Python 2.7 (Python 3 is not supported) +- Python 2.7 (vSDK 3.1.0 and earlier) +- Python 3.8 (vSDK 4.0.0 and later) - Java 7+ -- Delphix Engine 6.0.3.0 or above +- A Delphix Engine of an [appropriate version](/References/Version_Compatibility.md) ## Installation To install the latest version of the SDK run: @@ -26,9 +27,13 @@ $ pip install dvp !!! tip "Use a Virtual Environment" We highly recommended that you develop plugins inside of a virtual environment. To learn more about virtual environments, refer to [Virtualenv's documentation](https://virtualenv.pypa.io/en/latest/). - The virtual environment needs to use Python 2.7. This is configured when creating the virtualenv: + If using vSDK 3.1.0 or earlier, the virtual environment needs to use Python 2.7. - ```$ virtualenv -p /path/to/python2.7/binary ENV``` + If using vSDK 4.0.0 or earlier, the virtual environment needs to use Python 3.8. + + This is configured when creating the virtualenv: + + ```$ virtualenv -p /path/to/python2.7/binary ENV``` or ```$ virtualenv -p /path/to/python3.8/binary ENV``` To install a specific version of the SDK run: diff --git a/docs/docs/References/Decorators.md b/docs/docs/References/Decorators.md index 038cf01c..4abb7223 100644 --- a/docs/docs/References/Decorators.md +++ b/docs/docs/References/Decorators.md @@ -13,7 +13,7 @@ plugin = Plugin() # Use the decorator to annotate the function that corresponds to the "Virtual Source Start" Plugin Operation @plugin.virtual_source.start() def my_start(virtual_source, repository, source_config): - print "running start" + print("running start") ``` !!! info @@ -38,6 +38,7 @@ Plugin Operation | Decorator [Virtual Source Initialize](Plugin_Operations.md#virtual-source-initialize) | `@plugin.virtual.initialize()` [Virtual Source Unconfigure](Plugin_Operations.md#virtual-source-unconfigure) | `@plugin.virtual.unconfigure()` [Virtual Source Reconfigure](Plugin_Operations.md#virtual-source-reconfigure) | `@plugin.virtual.reconfigure()` +[Virtual Source Cleanup](Plugin_Operations.md#virtual-source-cleanup) | `@plugin.virtual.cleanup()` [Virtual Source Start](Plugin_Operations.md#virtual-source-start) | `@plugin.virtual.start()` [Virtual Source Stop](Plugin_Operations.md#virtual-source-stop) | `@plugin.virtual.stop()` [VirtualSource Pre-Snapshot](Plugin_Operations.md#virtualsource-pre-snapshot) | `@plugin.virtual.pre_snapshot()` diff --git a/docs/docs/References/Logging.md b/docs/docs/References/Logging.md index d4e09439..180833b7 100644 --- a/docs/docs/References/Logging.md +++ b/docs/docs/References/Logging.md @@ -6,10 +6,10 @@ The Virtualization Platform keeps plugin-specific log files. A plugin can, at an ## Overview -The Virtualization Platform integrates with Python's built-in [logging framework](https://docs.python.org/2/library/logging.html). A special [Handler](https://docs.python.org/2/library/logging.html#handler-objects) is exposed by the platform at `dlpx.virtualization.libs.PlatformHandler`. This handler needs to be added to the Python logger your plugin creates. Logging statements made through Python's logging framework will then be routed to the platform. +The Virtualization Platform integrates with Python's built-in [logging framework](https://docs.python.org/3.8/library/logging.html). A special [Handler](https://docs.python.org/3.8/library/logging.html#handler-objects) is exposed by the platform at `dlpx.virtualization.libs.PlatformHandler`. This handler needs to be added to the Python logger your plugin creates. Logging statements made through Python's logging framework will then be routed to the platform. ## Basic Setup - Below is the absolute minimum needed to setup logging for the platform. Please refer to Python's [logging documentation](https://docs.python.org/2/library/logging.html) and the [example below](#customized-example) to better understand how it can be customized. + Below is the absolute minimum needed to setup logging for the platform. Please refer to Python's [logging documentation](https://docs.python.org/3.8/library/logging.html) and the [example below](#customized-example) to better understand how it can be customized. ```python import logging @@ -36,9 +36,11 @@ logger.setLevel(logging.DEBUG) To avoid this complexity, add the `PlatformHandler` to the root logger. The root logger can be retrieved with `logging.getLogger()`. +!!! warning + There is a limit to how much data can be stored within a log message. See [Message Limits](/Best_Practices/Message_Limits.md) for details. ## Usage -Once the `PlatformHandler` has been added to the logger, logging is done with Python's [Logger](https://docs.python.org/2/library/logging.html#logger-objects) object. Below is a simple example including the basic setup code used above: +Once the `PlatformHandler` has been added to the logger, logging is done with Python's [Logger](https://docs.python.org/3.8/library/logging.html#logger-objects) object. Below is a simple example including the basic setup code used above: ```python import logging @@ -66,7 +68,7 @@ Imagine you notice that your plugin is taking a very long time to do discovery. Suppose your plugin has a source config discovery operation that looks like this (code is abbreviated to be easier to follow): ```python -import pkgutil +from importlib import resources from dlpx.virtualization import libs from dlpx.virtualization.platform import Plugin @@ -81,10 +83,10 @@ def repository_discovery(source_connection): @plugin.discovery.source_config() def source_config_discovery(source_connection, repository): - version_result = libs.run_bash(source_connection, pkgutil.get_data('resources', 'get_db_version.sh')) - users_result = libs.run_bash(source_connection, pkgutil.get_data('resources', 'get_db_users.sh')) - db_results = libs.run_bash(source_connection, pkgutil.get_data('resources', 'get_databases.sh')) - status_result = libs.run_bash(source_connection, pkgutil.get_data('resources', 'get_database_statuses.sh')) + version_result = libs.run_bash(source_connection, resources.read_text('resources', 'get_db_version.sh')) + users_result = libs.run_bash(source_connection, resources.read_text('resources', 'get_db_users.sh')) + db_results = libs.run_bash(source_connection, resources.read_text('resources', 'get_databases.sh')) + status_result = libs.run_bash(source_connection, resources.read_text('resources', 'get_database_statuses.sh')) # Return an empty list for simplicity. In reality # something would be done with the results above. @@ -92,10 +94,15 @@ def source_config_discovery(source_connection, repository): ``` +!!! warning + If developing a plugin in Python 2.7, you will need to use `pkgutil.get_data` rather than `importlib.resources.read_text`. + + See [Managing Scripts For Remote Execution](/Best_Practices/Managing_Scripts_For_Remote_Execution.md) for more info. + Now, imagine that you notice that it's taking a long time to do discovery, and you'd like to try to figure out why. One thing that might help is to add logging, like this: ```python import logging -import pkgutil +from importlib import resources from dlpx.virtualization import libs from dlpx.virtualization.platform import Plugin @@ -141,13 +148,13 @@ def repository_discovery(source_connection): @plugin.discovery.source_config() def source_config_discovery(source_connection, repository): logger.debug('About to get DB version') - version_result = libs.run_bash(source_connection, pkgutil.get_data('resources', 'get_db_version.sh')) + version_result = libs.run_bash(source_connection, resources.read_text('resources', 'get_db_version.sh')) logger.debug('About to get DB users') - users_result = libs.run_bash(source_connection, pkgutil.get_data('resources', 'get_db_users.sh')) + users_result = libs.run_bash(source_connection, resources.read_text('resources', 'get_db_users.sh')) logger.debug('About to get databases') - db_results = libs.run_bash(source_connection, pkgutil.get_data('resources', 'get_databases.sh')) + db_results = libs.run_bash(source_connection, resources.read_text('resources', 'get_databases.sh')) logger.debug('About to get DB statuses') - status_result = libs.run_bash(source_connection, pkgutil.get_data('resources', 'get_database_statuses.sh')) + status_result = libs.run_bash(source_connection, resources.read_text('resources', 'get_database_statuses.sh')) logger.debug('Done collecting data') # Return an empty list for simplicity. In reality @@ -174,7 +181,7 @@ Download a support bundle by going to **Help** > **Support Logs** and select ** ## Logging Levels -Python has a number of [preset logging levels](https://docs.python.org/2/library/logging.html#logging-levels) and allows for custom ones as well. Since logging on the Virtualization Platform uses the `logging` framework, log statements of all levels are supported. +Python has a number of [preset logging levels](https://docs.python.org/3.8/library/logging.html#logging-levels) and allows for custom ones as well. Since logging on the Virtualization Platform uses the `logging` framework, log statements of all levels are supported. However, the Virtualization Platform will map all logging levels into three files: `debug.log`, `info.log`, and `error.log` in the following way: diff --git a/docs/docs/References/Platform_Libraries.md b/docs/docs/References/Platform_Libraries.md index 39301549..82392d48 100644 --- a/docs/docs/References/Platform_Libraries.md +++ b/docs/docs/References/Platform_Libraries.md @@ -1,5 +1,5 @@ # Platform Libraries -Set of functions that plugins can use these for executing remote commands, etc. +Delphix provides a set of functions that plugins can use for executing remote commands, etc. ## retrieve_credentials @@ -56,7 +56,7 @@ Argument | Type | Description -------- | ---- | ----------- remote_connection | [RemoteConnection](Classes.md#remoteconnection) | Connection associated with the remote host to run the command on. command | String | Command to run on the host. -variables | dict[String, String] | **Optional**. Environement variables to set when running the command. +variables | dict[String, String] | **Optional**. Environment variables to set when running the command. use_login_shell | boolean | **Optional**. Whether to use a login shell. check | boolean | **Optional**. Whether or not to raise an exception if the `exit_code` in the `RunBashResponse` is non-zero. @@ -71,7 +71,7 @@ stderr | String | Stderr from the command. ### Examples -Calling bash with an inline command. +##### Calling bash with an inline command. ```python from dlpx.virtualization import libs @@ -86,7 +86,7 @@ print response.stdout print response.stderr ``` -Using parameters to construct a bash command. +##### Using parameters to construct a bash command. ```python from dlpx.virtualization import libs @@ -98,10 +98,11 @@ command = "mysqldump -u {} -p {}".format(name,port) response = libs.run_bash(connection, command) ``` -Running a bash script that is saved in a directory. +##### Running a bash script that is saved in a directory. +###### Python 2.7 recommended approach ```python - + import pkgutil from dlpx.virtualization import libs @@ -110,6 +111,17 @@ Running a bash script that is saved in a directory. # Execute script on remote host response = libs.run_bash(direct_source.connection, script_content) ``` +###### Python 3.8 recommended approach +```python + + from importlib import resources + from dlpx.virtualization import libs + + script_content = resources.read_text('resources', 'get_date.sh') + + # Execute script on remote host + response = libs.run_bash(direct_source.connection, script_content) +``` For more information please go to [Managing Scripts for Remote Execution](/Best_Practices/Managing_Scripts_For_Remote_Execution.md) section. ## run_expect @@ -126,7 +138,7 @@ Argument | Type | Description -------- | ---- | ----------- remote_connection | [RemoteConnection](Classes.md#remoteconnection) | Connection associated with the remote host to run the command on. command | String | Expect(Tcl) command to run. -variables | dict[String, String] | **Optional**. Environement variables to set when running the command. +variables | dict[String, String] | **Optional**. Environment variables to set when running the command. ### Returns An object of `RunExpectResponse` @@ -168,7 +180,7 @@ Argument | Type | Description -------- | ---- | ----------- remote_connection | [RemoteConnection](Classes.md#remoteconnection) | Connection associated with the remote host to run the command on. command | String | Command to run to the remote host. -variables | dict[String, String] | **Optional**. Environement variables to set when running the command. +variables | dict[String, String] | **Optional**. Environment variables to set when running the command. check | boolean | **Optional**. Whether or not to raise an exception if the `exit_code` in the `RunPowershellResponse` is non-zero. ### Returns diff --git a/docs/docs/References/Plugin_Config.md b/docs/docs/References/Plugin_Config.md index 1626d6f8..0351ff78 100644 --- a/docs/docs/References/Plugin_Config.md +++ b/docs/docs/References/Plugin_Config.md @@ -17,7 +17,7 @@ The name of the file can be specified during the build. By default, the build lo |entryPoint|Y|string|A fully qualified Python symbol that points to the `dlpx.virtualization.platform.Plugin` object that defines the plugin.

It must be in the form `importable.module:object_name` where `importable.module` is in `srcDir`.| |manualDiscovery|N|boolean|True if the plugin supports manual discovery of source config objects. The default value is `true`.| |pluginType|Y|enum|The ingestion strategy of the plugin. Can be either `STAGED` or `DIRECT`.| -|language|Y|enum|Must be `PYTHON27`.| +|language|Y|enum|Must be `PYTHON38`.| |defaultLocale|N|enum|The locale to be used by the plugin if the Delphix user does not specify one. Plugin messages will be displayed in this locale by default. The default value is `en-us`.| |rootSquashEnabled|N|boolean|This dictates whether "root squash" is enabled on NFS mounts for the plugin (i.e. whether the `root` user on remote hosts has access to the NFS mounts). Setting this to `false` allows processes usually run as `root`, like Docker daemons, access to the NFS mounts. The default value is `true`. This field only applies to Unix hosts.| |extendedStartStopHooks|N|boolean|This controls whether the user's pre-start and post-start hooks will run during enable operations (and, likewise, whether pre-stop and post-stop hooks will run during disable operations). The default value is `false`.| @@ -46,14 +46,13 @@ This is a valid plugin config for the plugin: ```yaml id: 7cf830f2-82f3-4d5d-a63c-7bbe50c22b32 name: MongoDB -version: 2.0.0 hostTypes: - UNIX entryPoint: mongo_runner:mongodb srcDir: src/ schemaFile: schema.json pluginType: DIRECT -language: PYTHON27 +language: PYTHON38 buildNumber: 0.1.0 ``` This is a valid plugin config for the plugin with `manualDiscovery` set to `false` and an `externalVersion` set: @@ -68,7 +67,7 @@ srcDir: src/ schemaFile: schema.json manualDiscovery: false pluginType: DIRECT -language: PYTHON27 +language: PYTHON38 externalVersion: "MongoDB 1.0" buildNumber: "1" ``` diff --git a/docs/docs/References/Plugin_Operations.md b/docs/docs/References/Plugin_Operations.md index e02e3773..a66e1590 100644 --- a/docs/docs/References/Plugin_Operations.md +++ b/docs/docs/References/Plugin_Operations.md @@ -8,7 +8,6 @@ For each operation, the argument names must match exactly. For example, the Repository Discovery operation must have a single argument named `source_connection`. - Plugin Operation | **Required** | Decorator | Delphix Engine Operations ---------------- | -------- | --------- | ------------------------- [Repository
Discovery](#repository-discovery) | **Yes** |`discovery.repository()` | [Environment Discovery](Workflows.md#environment-discovery-refresh)
[Environment Refresh](Workflows.md#environment-discovery-refresh) @@ -26,6 +25,7 @@ Plugin Operation | **Required** | Decorator | Delphix Engine Operations [Virtual Source
Configure](#virtual-source-configure) | **Yes** | `virtual.configure()` | [Virtual Source Provision](Workflows.md#virtual-source-provision)
[Virtual Source Refresh](Workflows.md#virtual-source-refresh) [Virtual Source
Unconfigure](#virtual-source-unconfigure) | **No** | `virtual.unconfigure()` | [Virtual Source Refresh](Workflows.md#virtual-source-refresh)
[Virtual Source Delete](Workflows.md#virtual-source-delete) [Virtual Source
Reconfigure](#virtual-source-reconfigure) | **Yes** | `virtual.reconfigure()` | [Virtual Source Rollback](Workflows.md#virtual-source-rollback)
[Virtual Source Enable](Workflows.md#virtual-source-enable) +[Virtual Source
Cleanup](#virtual-source-cleanup) | **No** | `virtual.cleanup()` | [Virtual Source Delete](Workflows.md#virtual-source-delete) [Virtual Source
Start](#virtual-source-start) | **No** | `virtual.start()` | [Virtual Source Start](Workflows.md#virtual-source-start) [Virtual Source
Stop](#virtual-source-stop) | **No** | `virtual.stop()` | [Virtual Source Stop](Workflows.md#virtual-source-stop) [Virtual Source
Pre-Snapshot](#virtual-source-pre-snapshot) | **No** | `virtual.pre_snapshot()` | [Virtual Source Snapshot](Workflows.md#virtual-source-snapshot) @@ -824,6 +824,50 @@ def reconfigure(virtual_source, repository, source_config, snapshot): } ``` +## Virtual Source Cleanup + +Intended to allow a final cleanup during a delete operation, unlike unconfigure which can be used to signal a temporary dissassociation with a database. + +Cleanup is called during the delete flow after unconfigure. + +### Required / Optional +**Optional.** + +### Delphix Engine Operations + +* [Virtual Source Delete](Workflows.md#virtual-source-delete) + +### Signature + +`def cleanup(virtual_source, repository, source_config)` + +### Decorator + +`virtual.cleanup()` + +### Arguments + +Argument | Type | Description +-------- | ---- | ----------- +virtual_source | [VirtualSource](Classes.md#virtualsource) | The source associated with this operation. +repository | [RepositoryDefinition](Schemas_and_Autogenerated_Classes.md#repositorydefinition-class) | The repository associated with this source. +source_config | [SourceConfigDefinition](Schemas_and_Autogenerated_Classes.md#sourceconfigdefinition-class) | The source config associated with this source. + +### Returns +None + +### Example + +```python +from dlpx.virtualization.platform import Plugin + +plugin = Plugin() + +@plugin.virtual.cleanup() +def cleanup(virtual_source, repository, source_config): + pass +``` + ## Virtual Source Start Executed whenever the data should be placed in a "running" state. diff --git a/docs/docs/References/Schemas.md b/docs/docs/References/Schemas.md index 2f2a11a5..88c765b1 100644 --- a/docs/docs/References/Schemas.md +++ b/docs/docs/References/Schemas.md @@ -24,6 +24,8 @@ Delphix Object | Schema | Autogenerated Class [Snapshot](Glossary.md#linked-source) | [SnapshotDefinition](Schemas_and_Autogenerated_Classes.md#snapshotdefinition-schema) | [SnapshotDefinition](Schemas_and_Autogenerated_Classes.md#snapshotdefinition-class) [Snapshot Parameters](Glossary.md#snapshot-parameters) | [SnapshotParametersDefinition](Schemas_and_Autogenerated_Classes.md#snapshotparametersdefinition-schema) | [SnapshotParametersDefinition](Schemas_and_Autogenerated_Classes.md#snapshotparametersdefinition-class) +!!! warning + There are limits to how much data can be stored within a plugin defined schema. See [Message Limits](/Best_Practices/Message_Limits.md) for details. ## JSON Schemas @@ -116,7 +118,6 @@ For much more detail on JSON schemas, including which keywords are available, wh !!! info Be careful when using the JSON schema keyword `default`. This keyword is commonly misunderstood. Specifically, it does not mean "If the user does not provide a value, then this default value will be auto-substituted instead", as you might expect. In fact, in JSON schemas, `default` has no semantic meaning at all! Currently, the only thing the Delphix Engine will use this for is to pre-fill widgets on the UI. - ### Delphix-specific Extensions to JSON Schema The JSON schema vocabulary is designed to be extensible for special uses, and Delphix has taken advantage of this to add some new Delphix-specific keywords. diff --git a/docs/docs/References/Version_Compatibility.md b/docs/docs/References/Version_Compatibility.md new file mode 100644 index 00000000..4260bfe7 --- /dev/null +++ b/docs/docs/References/Version_Compatibility.md @@ -0,0 +1,26 @@ +# Virtualization SDK Version Compatibility + +## Virtualization SDK and Delphix Engine (DE) Compatibility Map + +|vSDK Version|Earliest Supported DE Version|Latest Supported DE Version| +|------------|:---------------------------:|:-------------------------:| +|4.0.0|6.0.12.0|[Latest Release](https://docs.delphix.com/docs/release-notes/)| +|3.1.0|6.0.7.0|[Latest Release](https://docs.delphix.com/docs/release-notes/)| +|3.0.0|6.0.6.0|[Latest Release](https://docs.delphix.com/docs/release-notes/)| +|2.1.0|6.0.3.0|[Latest Release](https://docs.delphix.com/docs/release-notes/)| +|2.0.0|6.0.2.0|[Latest Release](https://docs.delphix.com/docs/release-notes/)| +|1.0.0|6.0.2.0|[Latest Release](https://docs.delphix.com/docs/release-notes/)| +|0.4.0|5.3.5.0|6.0.1.0| + +## Virtualization SDK and Python Compatibility Map + +|vSDK Version|Python Version| +|------------|:------------:| +|4.0.0|3.8| +|3.1.0|2.7| +|3.0.0|2.7| +|2.1.0|2.7| +|2.0.0|2.7| +|1.0.0|2.7| +|0.4.0|2.7| + diff --git a/docs/docs/References/html/VirtualSourceDelete.html b/docs/docs/References/html/VirtualSourceDelete.html index 9e42c12e..7953f998 100644 --- a/docs/docs/References/html/VirtualSourceDelete.html +++ b/docs/docs/References/html/VirtualSourceDelete.html @@ -1,12 +1,12 @@ + - + -Draw.io Diagram - +VirtualSourceDelete.html -
- +
+ diff --git a/docs/docs/References/images/VirtualSourceDelete.png b/docs/docs/References/images/VirtualSourceDelete.png index ab8f2cffaa9b812d0f7022457d499aeb21925655..9e1c005ce4451830b92287874c602081402dbca1 100644 GIT binary patch literal 17644 zcmeIaWmsHI(=IxL1_&es4GszJK4{PZ24@DhVQ{y>gOfmz1QOhXyGw9Lf&_OB?h>33 zd{5rz+2{NAwa@v#uXE0iwYpcY>R#Q|T~*yx_slyDH3fWJ3S0mHfUl$|r-e$B0RUjv zBP>)dLBOp9l>ptf6l4Gu!&JKf0JaKJUJvQ$ZDs3V31H-x{U^uB$K&X1VQXb;X~D=Z z$H>RW$HT|V$;-#dD-7cmV&s?R75;k@rcAt3=SUf#dog_>EJx!L}=X8!ix5@}}rPw(B86ji<4b&={S3T|$S`j(ms5WRm! z;A!dRZtLvy4;k;@R#DCUt&H?>vHYjh!qU^$9Q9E?Mt*q|GymlRip2I`K5`mZDI*c? zf_7^B0vZq*KBT3F%Rh}cnW1|7A6{#qSjEZvPb0!QP6!0jT|i!g$4pw-T|-q5>J727 z_qKC!S5TIQ*bC_LTZky}>1x@SyF;MP+F%P$5k@{4TWtp(S0xXmmWCY6nI8Uka5-3)65sxl7Njv71)vNrZG6^N3ot%13;&oTjbkT*u)U4eh2-Jki zm^pjd2zjYN6g`wsPagKpa$s`@B+MBkZ>6NDW; zI~1*e0MyIF1?u6(Bd4P1MHX)+iCGTfIT3Z2t@~bbwLjeHCu0Q2TvIv2*}M+-rYtS;$>iF?;jCeFddG5poDC zdtHc)wUxZ3mzla1j89r#7PWA}FtELv4oKJ8)6P;@T3v+46Q*e91Cr76QPK6{gUWe= zP!pu1U?JiykF*z5bwzza$VW#Q1d?+>DnNx0P;FOV6es2N?LC#%MC?`hzou#+JU-6Ye@Tdu^Aw@)N zPGRvm>gZ}95E>4s(SU_*oOsNIlyqT8 zB?~uSFBzDXEuXcPFic04Uj~GLIl5{fR9!`sE!AKeW*}V+sI8`lli6R1@izF&JXZ}} zcV{<*w=D?43o%nNmv@o&L~6^T^qY*lt0J$6I+)MK$<~cW*&CuOte`H)%d2drWM%$$ zeLA_@nyaYUyC{HMd{M*qfq_s`BO{;-;gxsPQneHW+gYe7qe>J}dcxjS#!}BrQ^r~j zV&?^e!5l5kcrA2XdH7Hwf$(*4GH`T(>OgpPy?ObxY!FJW{N7Ngv$~dPYYij7e{MfC0;o}ISnUwXB`naL5Q`U@LwU8R*^>t z{#^o|Hb@%{b5u@WO~48&C*UP3Bqo(3wuB+)RA|zy^WTj<>qEr`l z)pyeKRq_Pe@bJO@3c8~PN(7KrD8mFsSP42Q3s?!G^reiFuqFtV*g5e@qY`NZL_?NG z1Bqfc%BaX#C>!wdqf~|rSlU8FSzX=IUe8#cMV zYmA&*#eO{<5})^SyMyY;@7+_;uB1#5)@S*#?_z7f@!_;}$1kaR>0{CR)?NCjR-=bp z`&-u)i{d|}{M<&4;^Ml28TzggL|ZhuIrjDRA`S^6oSxN)(HQ{w2&IgQ`O87fTj(Jx z3v6^X>9M8eh*&{$WEOs0A{|2zqpIoH@G88F_cIuqy)!qjE~ zltLF1Itd7K=cK2N=sRorKPa?`G#YiZC(E^muJv2CFiBy&@2lRBt1X5a=zNA698*i& zQxZ^%#mM%m>AzA>A3oC=HUKv57)f{K%&k`}P)tg-T9A%U`?>Tad8A*xK@S5+PZ=hd zGJbwWF z&G5l4n)G&uDr7(K=^7IWQY9L_Jk=yZM|D5jC)eO}5#F|~Y%sOo2ebW(Z?nz&k8tra z9>iq(yT8(twu1^r#3|=y)tk z83sDHS;jD)7s7qXBq!rhzxVU&o2wt+wod&evc2vOTJK97r=6u#A7!Q!NW5} z|DFIN_jrRbjhVonk~p-RYZSaZoL)8-EQ@f`b?f16^x_*0rUk zON?`IuT7>YF6?xp+)_ZTQY0YnbK=B+4on604Xr{Sm9IX5;Qpzjg Q@E^9O(PT0z z1uC{WiY{Z?o|0^D(U6Tnbh-M4o0eK?eTr?qksb3lm=5E4memTS1GlwnAyDQl@p|7jZ@&JpMuLI}6 z+q?YM)LINLUeu2$I`{uUR&DGKK*Qo~ky;P!N!glqJmy0Ju~&wx36pYk~zd6qAw4HpK)Usw{V0!UYm}NFTGcvSrPBt z*=vnF%~!K>wS-}xli*WKTn9W%sFlsb@oxj&anHvqud}dc`_4B@fphhaXU$&@ zPldpi)FUF1lX)$%;#<7PmUG;hroA)zA5T`lbw{}jIC?eTWjs5az2~V;OZt-)*?Pf!=&)GeiP#yh(3R87_5|P3d1?g=6MOY#{+RB7q{qN7)2I${ zb@q|9)6-%2DN(VB$JE3}tQN2Pkd64>kf+stq*(!%{54)wIr|p&Rs}{=s&`ZPRU79d z=osk_75e3Thxcbk5E2YQK5Ab^oyu#{=M!1d23v-bt2m7Cmj^cwJ`39Ic7tioE&O(L zXTLXJs)CK` zU!=7wFWZlfb?QPjx7E7fRm_X_Avf^x4hM~;Xb-!=^VSXZ)}}O z9}CXRMTE7JI37vcIKjF+sIogTfM11O7&xX5IWcW6{L^Xbo4T=|I1OcK)1(lv+ryg8 zJ9%O3vO|!l(G;1sx$m<MBpiE{C4NCg{okLxHo*+$Mp}I%FN~fx1_5->+hq*O~K(P!}=5br9$QVktuoK z#BFvi=u?xcj)ZK#WSPEKZJ@0^MlNhh5&yZ~*e~~yTx6?WztWPo_7hEWN>XIykX@ux%==Mz2;5?gXO{E* z#(Fu{f^-hr{q)RqU1c-V_sX}iLh=$bxnCEb^4zzy*^fX1rdxEXvsrGMwuzNoiOe)b z5cTb3GB(XN-O4=bE9K|}7jTaNEVMqnS9SGy%BB32LqPI5^5_wV&(U&4ikU4CdKG6F zt9U)Dl=a*PJ-R=g_d@BXg)S35xo8}MtgAG+pOl%e@* z-dMJuXwO8U97?*TPL(rR?JHg4?Kif}0aaXv5rYV9!v+}I%+J@_3Ig_uBh6TzJBx7? z8d~oQf~HEBM=7G8$#Gk&)%Mtn!hNjW%1^+JG`f1buSJr#rS!W$Z<~&j4;oTs2Kk`n z^s7*uvCybKUu#CLi{x_t=n}V%^Ob-t+A@`S#buiFw28Mxos(8dP7s6zW!$(O-=E?O zgx>x{bC^8eyh+y*`m$X>lf6w8>RObN89-h5vJ>{E*df)wxJziLwda)x{;TqE=<#0-@u~)jy|EAIf<%3yFLJ{4Vz4qJQFt7dSpj&R84x{_Rk~g<#bAw8VImXJzQF zYl*108qaz1#VO3|mm}#ed&PK_)QEX|^&f15-Bwd^(E80@&B`E-ZyCoGfqfhsZJAUj zuEp&Q<$g7Y_d099k63iG1kc_iNLsj=s}suzV69P2lKil0M%V8# z*nG~D#cse!hW9n~79K)a)_U$?WT%?k9xd<1L(Nll1txbckeCTDisH=c@PvAe2D zzIz&77#V^oUv+lUN>DIzdY&(H-LJW~29^-`QE&5cWx}{557nCi)ANLK^;fU(#Jn7O zi(HAyvDNwqRxgWl7|7V~+~YRoSt`Pt&>AwM@*J*yIJvvBUq1({dy;u1+&V=@62vTC zQ+A}uSPnYRRI|HwtLpAG?th_MzYNtj^1YoDU%3n|zP-W^A51PpllJ>YZ#j5mW7&zs z47qLC?vicX zl5^fsH99G_b-_SnmDA4C(A~XuH_H@%m;6aKxBu{sd?M$3j%SCW`g5P&g&K67l=+is zsLoQYn6EVZgr8>?_Ixz=KIjY&`%&-$@LVUb@}mY0{vJ$W(Lmeyfvxv!dZ4T4ba3u9 zm8j?Zh4aJer@kSvLb6&J2#4GEZjRTkle7fbEwB4Wj;!>$*pRc6>W12pF>S^&PaTZK z&+;!nk^pswu9a>Wx^*MvqhC1CW0jVejTDZuy8bo_t`F28)F>>=I)9y$WL_R*gsjMh zj~%$q%q{8F)g?qK?*I2g!Y*?&`qzCm!}E4MG}ZrkLKxR+Ez3{<=s%p|#TMS1-hQg6 z@q4x59Q&T*k%e^slv?-;c6XJdq17@!Il!4t*MC2zdHFJY`+P=tHu1b`r8(xN3Jr8aWBFKS&`p2Kxbzk@Dz*)o~4VV{P0%` zm!L_*1WUk$_6dhA_|na1Ty!{yEqUQvC46wKR9-k1f`r+!^Ra+==i~2)&cSZfC1{S# zKN=0>e;P$SXvBdvXvFWygFZyggI+z`^?w?{8Mh8wkTpOYZ3Pp6TQvr_2d?k}Z0~D} zlUd1m9)6kNcWMGK0dBxaESJZCU*E_KZ*N#1=3s-wkCWRxv(V@;r5HIy+HBaV0d1o3 zPiS#@;J-)d$a2Z*y6u0ApcSD9EtX>7Kj-la(qvogo^5m3>j|18#tk||9GD1sn0Gqh zYNtmVMO!5Z*l$!P;s5H#fV=m$E&eH+82i2&AG6xoJ#IU+@--k`%&TF%8{5w-V%Dgk^G=Uh;wrgxXc`EZ0D_i#E#YT-k zTh&#pPEG@&jlX+kMewp%?0tlR|i{VciD*(3H? zKRzyfO(K`o?Pi(eMyeT%vRCNfsE52i}uRsVeX&R+AN1I`F9MWSOer|zdZ=T*Hv*Q z?I&|+M2(i_DV$FTGjId}lzPPL`@Jrh{~(Nru#-<~Ymq(7I|i-w3lv(nKlW9vES*UP z*TJ*{Z|;=?cZhQ24LI+~>qRFPGi-N?u(pMmv&15{#x;xBC)B7RF4=JE*lYE=~OegXTn9$iQy|##v9M)$P(yT?Y*XW zx+Dwy{pIl4q@xP^`>O~YOr3yMQib1}8r~u&ygBRWQ0zmzm(#Imi7Cx1)KW_jpvh7t zrQK91p<>x!7$Jm zlCRk`%zQcd9Iv9)$RLlsT^&N{?Zd-Ph3w>mZchlcQ-L~yN#`V;P&bpt<#TG2}LMZtpWKvr239?P;98F$S}el7>V(`H2T_Sp6$ z^3h66UY#`k!lop=)2eXT+3-;{O>A`?sOm;gnvyoPp@Y3mxHG`$EEMzWmk#yL$w0N9 zA5i9#E`_7@j7G9t^A;eFlouHt;vrR`3P!0dw0{xjX8Zn_AA z|4L{+>3q*SAW+EuXREU1XFa7&0Vi$RjKJk#aNKA0HGH0nxlum_y%>`Y6ZE9C-CdmT zy_q!5z_qy*A0Hnsc&qfzoV->$Ho;*p(UVti@Y7~mP8l&jYK5=%>|g8i#^P1)Vzn}8FU+RH} zV*j)hq@%`|&0plMZ5G*-$_3}?RpisGMY>eIK-diE-DJhM4I`8c0qtVcE1pvT$*YPiU}BRY|pIskS|C3%t_ay{k7e=ed5Ribeb-p1XUh&XpgJXOAJv z7g-~?`$!OOICCG#S?Wbx%}Dj+ee&;_6eeG+kJIE;`mA_|2{H!uP4mBPf=M${v9tmd zhXpYQ4?EM-_ZJdOTY?MF7jk@*$>2-`TaFH79A-ifRn#42%a)eR`K{a{{#C!~jk+Sb z9#!d^lSO~j8W*b>J&6>unQk!SK|J!MVG3fk{+uO1lhadhGRFTwU8FMV{6J0HtGLIZ zIdtsL&@#EhpF1TTVd@gz*=y&(o{9s1emj#jdP*ZTUqR# zJx2|CBb>Mqp=#Evi$5lgZYLGL(=j$+>RI)-F3^9MVXN(}Ps(;8G9e_4xZmS@LlKJk zF=@LA+l!_vQBX`ucNbVa#JaKgq7eU@E7>x6W0Q=BxM%YZWBA{8F2RJ)2e=iJ>2=uG@YQDd1>{hgvDM)g|u!S zg00xdFi75fg)sV^FoNf47HU6L3SvdK%6^V+MMsKm#f7>mo{J?1TRlz=);oZ2AyAvL z)auaZHa$VlBf1Ixe}ezw!aNgg=KRB>i;bIM%v`pMjvjW&`e~gSVu>g7&267Muh|Yt0 zv8Guwz=^?WEnp7-zhsk^#Tul6$3wh$J-O4GdEr8D(6U>|Tar{4$$8RbNSI%cpY;#UO{;Gyms(ar@cO#@2VBuQn?>BW zMh|hM?cFx5WpV?6-dX#^W%l1wE$GI_^pRug=*`K0@4I*wGNir{2pF}eMt$i zAI%0D)Y$}=XqRHK>kKo?;igOaQ!F$(_uk*#lt0UhKN7Z^+-Xqy#AJITm{f6`4HUBI zi@khQu{Tw^K5ghMzdbxDcB+|3tk>v7z2tvsoG$J&9^!ELv#Ak-B2WXsen?tKWc1_5 zk0QfHr)<}gD?+UmRQLM&`l{%1KdWUgT2=uTl*}vqj@ptk*k#}twR&zfGp2zVH9rNd zhtsWsH>i2w-L>7^HiLprKfg#JR^4+IgNvbsCA@P}!Gm0(K=L7q!pprWyt|vr4-D@> z@lwRz+ZkTR7c-{VKUJ~NtKI>a7G1vKf4RTjM=vJIqLjydefop>4~mDZIZEzZgFZEN zpLyZ=+7;cNRQwRBLhb(a5h^O+QOx9p{A5|^>vouqlsyG2Ec;phk=RR$Z zx{a8qRIp`m)I*q%VI8I&$hN5zFW9WKnO6{@8z6 zR%r4*B)%u}nk=$?bg9RP(c_aP8c*9tzstk=W`%3d-KT``<}aQ===-8(s*#Gm$*yyy z@TJbIzP|&U32vg!bGJuTs%b(^F~vKz;24S~ z+p?M9`y)px9VS5T%9QZ?`^p|bKBkH@Y~-!52ao9?WYLj_Y7m2m2>VQscvSj#x|BD; z@c;?jF36}wAi{ouDz$yY^z^7S9VYET}mt+G&jNQdLiY@8vHX@|f=P zL#QB+DhxsqngZBzeq8V&A5u~%i3M$*I1W3#6w2m4Gv~1bO{wmMj-eRq%Rd6>!&Mr`}H2Xy^b8Bq^mZSwFDv;BU=29=;XiRQ0x0_y$| zc`!!kC^3chh&;-EdVIRI@Uo#0Pj3Qmep-A9=tyHoCl4zqEAkh{)J|HiAW8*4QK+z7 z`y~pAIK2fm1=N_A*TRCiE{eGkFE_Fe4% zSc0n=4w6oO01faM4X_E%1%|Si>jMDPGJmlEx9FYbfbm;vC@+);e!tL05tIwS1#F-d z;jfSf^#c%5fv+}SrA4NM0BdIsp@2LJW8gk*%&r?LLDn>@#=QE*D)_dh0y z+nP`J=bqd|Kpc^6X%3~+7a+>G`FA_Ux#^tQ%%3Cz*{xkPJ;kqKEA zSDCmO;!(u08p({wl<+fXDmQ9+_U$b`9%vgEsHeon3zy$}dM*mX6&m5rV|Crce=P4Kbqm!8z87A`&B)?}%6_O|bLHvQNS1T{ z!~M-s`7BE5`IK@sL-$Xh2IDy>bCE3ORk2Uy=IHika8PCcG}g^EGL4J(B_rdUo2BVW z0(+-;uwIP?4$8WQ`W-g)GKvGi_>yK{G4KfPr~>ij;Nw1%kK%=Y8SLXkFUkA%$X1Mg zaWxmPo9x4!rF3pJtH<|Hz+g*|t{T z`+Q7+ke2pn3yx}qil|fZ*yn7ksl9KhS)7_@k+C(n5U@~Z`zcN2uL0YcsiZwfv?k** zy;!O;3&kwj&dvm$tHJhrGiYvO~EO$z!Z1L2r5#1(?QBc1f{2NdRktGOk(A zjz@GyE2z2b^M|g=MTHQDut{YZrQKB&6iMKnGHl$|G^#;&)SXC^Pp0kJ(FKB6 z{s5G2-;Ao<@hOQwZf-LSkQr-U7+LVbb%gcXd_DH}d={HM29nlDii9+;EQEJQjtW?A zh{3;FSQQNdTGZw+%T!v86@CmHVY9bn10$9$Lw))UXqH_RJqG*xQzzT}E|n}Pa?e7! z|5uex@25~I{n(Ndbq*Uv{5^MdD?R_$w@uFDUX8kjX1o4%hGX-SMFII|qNxX^_)*%Q z!}I^k1ZS1{|CgV=zYC212RrD0JADxY`Y<&pJ9z;K!0)6F0sVnOB>b4@R+Wo|07amN zl5zlnlxWb&@0z0IVsak8Uus;Q>HqIC^oQ!0BSw^t ze_4(0AAQ|6I!d)*8o#;G3Kf8m*UyER5m9PYDz!Os0wa!$>Bn3Ly-n`c?B)IUTZzgG zCRZl;kdsMOok7_ZX8P|9tzT!$Stqd5S(UulXQZc`Yrio~FjsnSkDN}`MYq)ULi`WE8o zcPMOVQ^_{;Iv%IfDt|aoj?`H2gX?$D=t_=g(Q+BVo~fKJ`kx3U%-?rzkL(kjIVgjM zC~mA2Sk0Ny|M1KIY?U=Pnrz1@fhk8Ysfr{bFbR$%-fCx+v0tn91C@3|$>hsJhp5CJ zPRUpBS;tCz%S&t)>qYhiTVg5sAta!>8g1$gz zcy6HEyy>za&Wo}j%l8?tXr2ie%H3pgjYhj-zU3DA<)-)sDMY! ziahIvB5svo7J zJ$k7lDIA7NOe%VE7jExzd#u#_xOzYDMdSK6DD5NfdZ)hsgkPXIMwCQkq3qWl=*Icm zFGNW|NnQlYocf^?%sVNh4k{4JNnX9Uw-{A}lbFlE)0!~0lP|5Ny0Bu>xJ|>Z@gk$` zPkwvRn7SrhoZ}?RsJKq6crporD9XooCOmpwULshc=b{Z6rTvbBI)UkC@n z$9%n&X}10|7f~lYgPB^meI<>4(BxVPD?Kc$k%r9GS!L{fIq0c~c;|{CW|DPvS7R=Q z^6Z+OjZ8m#HgkgVmVVh(pXM3;lYBCc6ma}GxP6srNkHR5mg)V6_2tW#LH9xu?i886 zNWN$GOcY7fdqnT4#eRe>!1{|~PA#AIf{oqyfiYinEbjhSxmw^!a- zGs9KJ^=7P@f)g;3L-td83s=01{q*LvuFTbO_r4#pJ*( zH(hi*AMrEwAu&l#eMe%-{cR`wQ+eHzQS5LEFZ&{-lnT-vd%Xh68+lE`-mXuOV5&CE zoAj+@WJkPzd$Bu8qux$%FhFo3K@wKc7}!f+W}>rkDifK45(P<{mvv@qL4Y(tr=|QF zjj}vG%Epop!yV4ndasW^`k|{lL_>18;;9Nzet%N$i+#aPK6I6R73m9 z-}vO0!!_YFFnokL<&?;>g-x3a)hNLYF-!|K49=hRHwr@uaS3F%$eJ>iA>}T z#S2#K9A2&Y!D5pZ%|LS=+=w8U+DwHiHkpGce8c40TLt@J_Qc#tjl86K;eAX?we4mD z13o6xOI1p_4=ZnOSxDf|iyxY`JFf`jXKLK}>vjWN`_M0RgM_gsM(CgFSo$A%Jgx9D z^_?jOw0u5_Nn9T?{F9iFU->R{$#GA^ak+X3pTXZjF=%%s9IdL{KCnA+N-T2r*?VjS zm79H)Kwqr=qn@B4h`viD@wmajRD%Lu-Oi+dUNuJihgY4Gdc6}}=e+T5dB%R82F5TY zf&#C6HWzBDYn7(NccDI%3z!UpTeJ=9Nf#|;9C`7VjD=4c+7j7g-KLtrC|%qFkB|Sj z6!+%6Zk=>(1AZu>tj6gvcCzIDL@zwB`JSoqaNrOHI`TnsH4#de%F2_R3u8fzeRz7| z`yF@BTWT)H%Cn3>n~SI!*Kbn;Jy{5{Ea38f9IFm>G4^Xz_(pp1^?BiP4QiA0{7bcb|zvjF@mS~(SUxwT$#s*dZC_5Gq z5{)c$1fm&lyqMKH&L+mU$ohO500Sy1USeT-ikKH)exo9SU)dSZ2#sIO)b0}qEQgAH zfDfcnXP2G_&2AB&9>z^z`IFhS=SwVe!$pmfOqQQ}P)xBo>c7q;!wkXHJ1iDQX-L?OACtY5!bQ)q`9fD#KSf zkXj`&Kye`k$ER(tgx10xJ9nD8*``Y6#8)C|f@-s$7OHQp4Hk>1i)3I^I5!c{lZxcK zZ}ru6nS!I#0)V-VKS7#O>)Zj~@Pina+d~6%!#3JoY*YT56COH0h%Pb04TnixC%;J` zLf{%Gu^P&(eU!rE9N1?45gX%&h?r4q+H1EH$q@e&3e!sxZ7HYukqbT?HCJVsMd5eT6M}IL8 zeJBhBwdi{NMx7sw+OzT_rbXug5M2CUM|T!Lb5yn8+++-}mKEm8=f<0=Jzf^qYa&z^ z<=ZJ_m0d!lD98A+2X7Jyc?r3A@@SuCC|r$B(uc`unEc)4{xqOTZ}vSo_Iu@Es>1KC z^sz4xvC!D4bppCVS-zcwaZMi-1SZV{tN=ukJzIQvhhdjeq>4Tiwt|Y0;N?T_ zcvTUCYB&VSGRS?sJ@`uh_TiJu_|Y5d)=0YGq8*(WP;7)l+OiX1=|GOmOZ!ip)O8(A* zCrHEfg@3?m6BYiKA<(BYBHZC|Rh*gX5O`En*H@iO!_%Kkm}8Re4S8Y687B+U87vu^n1_Q5_l{){)h#4n5Gs!FxT{STkXM)G5(oyerG&^ z?6ExE2{y)TqR5UG9gm5I*CW04qIu3E+48MWw0G}DVzSYCJqLTo94m8=sW{22x%fGd ziYm4e#P-TWI>o>R0kT>Qo}aif-}T^APOh|a(E+c%2+*I1#OX+pW_FMLs$Ab`(51ID zi^~Oa*VBd&7F0h+Z@3ez6V(OGYh6LuI8JhaqKkzL z*V7=?Un;x(e{(yN0~004gq4FDtjkGE%P}dCnGvY@9r#pWM8=J~kue)~x1afa4!W>g z*^A*4xIpi{{9do&F9+A@c~bcDyTrQWb?2^%$Z zIMw(R>)NcYWSdB{($8P2E;RlH@+&k9cEy#um(Ak@H~v2Ay7On!ANbLRs)HK1GKNd; z3nr84BXK2|Gu-&yN-i3^i`#=WsLJUhk~EX#*`KPrr{^drf7%wJ0PxYAXiqfF=sG8~ z$M`=;{+M3RO5aiaSg$_%;OO77JmYD6lGF}o?25s)JFaB~Nh8t;28(+7I-RP)Kg~<4 z_gmIhKcW}^qN$<@aTRMtYp^A_XTVs>j~7U;9Q!lE!O6a&9ZcjWS%vk0RD5T+@TB`; z7iR?BVVw%;`sg)2D$Zqy-}RK!iC93Q1Z!*ISphX|vu$7sc04F6V>9)7lc z3|KfvI;k)^Pa@SFE%X!Zw{`RTx9omMPZTDOHtDr6DcAjDuf^$+Uq|uB+rPwkZRZ^Z z({Fj~m=m2C&nG5Kd%A}x%#O9&PR_T~@DEw<~#Ttz| z0v;01>P|U+t1Rh9rzCG_>q>Ci9uEw+>D*x4%y#_7NuRx;Uo3ux*RqxK7Hm@M~%CG+JVv^+Xlw#2^XT`Vc(+;tMC&t zM=4cso2y?Oy7^D+q!*5FudS?_?~GRq{-yy%=bKi1ztA;)uRBiDJMr*(?jq*piGrFp z!KWSq*)-Cj>=w=K-{mJqbI`I8f01P29o7$(Iu%{#%ur1c`X$lJciW~lB{@HbYFE!e zEQ>|Gng1Jl-*T~l6NTPyzWNuvf8f}WGc)u=PhRq3)MWs0@ZAzkdbskTrz<9rX}OLv z#KbAcF*PROYw$x)k7-1r|0|&%Go0mx8os##j<99a&mK-jnpQ@s%OIN=!UHpBAh&}r zX_R}%f6$_{nO(ydJxcLdMk-`Qs5UE&at#X`B-gXgJiJcAG(8lQE|(649J0wTX+TI5 zs&>GH$CdY7V+D72MH_Z&&s@W%yySr2uVcPZk#4T;j!3Wwb#ki+aTQTj*h~*X|Nvf|;?sC#7HF;`GAF*#8iZ2eqn%b}n>0E~$Kr zDl|6B`)^zqnS?uE=bzm^OOpa$a{TB{$Ls*|e(6W!xUsZQe3v)RNh_5h)T10Q%$^1R ze1hZgFsmIF2 z7&}XG)?XK-uKLOEGwmJv^Ue4b3Je`w6Dd2=k;j|d2oeXLVaO$1{aJo^i>;OAY2190 zK9;xv&#YB!qbWHQAewGbL}8~*GEvUrDBUbuY_u-t@Q&_7?c7IIBT+#2gHPjg+45PJ zgu3~fbUrP^+WZoCqH0P(#kf!`fg7z*$5fwubx2oorRI z=~1(PRS2{4=4bypfBPM4E7Bc&4rY5JQ{C2V}+KKHN9GgdmgXv4mBd_>{3< zn*4QSsQ}N$_F?yLOJUhZy#du1jxcUeDADJh>p?IqmS(4`7@ zvN-vByxqok^BB`xBrVZwW1DP~z6+@dco>_(eP5HUX={G&#`x;XC4(}h9%$qYGkIGA zQeS6vaqW=QqM2R?+y0!_QkH;!twm-4mc2CFZ7*)-t#lH6QyUP+vES+!*JYwg_@jT0 zbAc9~DV&s$HdYd~?BR42649(n4KZ-fUygpVscPBcfp0BZS<1>11h2rxvwj0HOyur- zy_l!7$XtHK+$o;4S42XDksL7cX?WsDIZ#E>#Q?7^UmX&^%r6U>G;CEE(16FjkoQxco$? zU|y`O3+pz32cIwCo0(o3*EJsAJS9PK$?f~;9p_^gF!{MxU2~5GDu(6KU_O~ZQq!oi zFSvQRpp zT(DnVE>--HjsoK2MmGx4)_WS3zCmrVHW#PS?3Vg6jp`QW?Xwr%b|Yq;?3*7`Zm-$V z^O`(5&mZyW=pqJCAP z<(o>Bw5z4>h|`F#lRt^aDITvB0QIBS@_QOeQ_2J)ZfThq&+dnLgO<%xLL*EYsst|v zN*`t+!-wIG+pv&Yg$o%SMYyKXW739Z+VD_AUrna~;cvJ)pa0!n;|Sp(PX>km*t8(Y&(v7}sCPANppiMg0 zyt!g#Fa?M_n}tXz1y4QevwIMBI1aS4f!@Aq^?Kv@75i-<>hDVcO7d!Q6*BOU{{aYm BqiFyD literal 23291 zcmd?RRa9J26Yq(;yF+kycY-?sf=lD>?gWBcAi>?;-D%vRaS84g+->^1ckUg#YhLDI z?5WRMeb(;1_o=E~|N7PGC^Z#XR3suK2nYyNc{wR{@MjtX1Y`gL9Qf)`wMp? z#qPYiGm_K_gP|RWAxpmkK_Q5O6ZJbOUEknRXoz5ngkFTFn0@~cVey>cm0yGCL%+OG z%mP7i^QoORXR3mOXXvLrYFSnn~*wz$#62hWJU=OX<%|K^cPWy;}> z`Y_7;^o5<2U5uK7V|!@ETl^FGcFc1u9Mj{1Z%Z{wY=rHXvdhQ)%k-rL)jP$ae9`{7 z^i{y;MEYjH!@<5`Y7i?HNuGPKY4omtB(ai2P|cm-pE+8PDe^uYlZpW>=e?rUfVyId zOERm}BzZ7IbU_z-ix+%eiTjSbYZYT$vTf3x+L(tyL#zgl@8Z1V1V*X7WTnt`yYaWP z*A!_2v3xtu%!9-jqvCd@t~=f>eIBF9^-yo?D$m{ih&G3@(=L92V^d?YrU^!rr?S3? z`X@@ho*qNS)j5Ym3#t=`tGLAKOtlg`ct{&>KHfif$5W-C8btxS){DUQRwH-6HR zEes{x=2|hDzo(tLT2vgYgtd_SGRUW*g!}8llAUg8b0JVH|DD;~mx6Ywo<0+_ZU9{e zNUnZktoJ>rh;>ob9+Uqiy%oSjY<90?!PK^aEiS(huOYA{P^q}rN05C>V>Q%Y^}RBn z`fb+dZ(-0s*_i|h!^AkNO{eIplR(Z#(W^VtrH7K7ZA)y`e}9&r@swr`zxo-b(-yjd z%GSm+ySOc99q2rjxFeFr{HP!ZlsP$`iN14&GCOPvuk^+xit_@F z7p@MKe;lu3$1JETCQdlL>HD$1v>{=(g~|6=nu}dNqDzu|7i1SUo2NQ#ie|DrnX^@I zb>*vlT2BFmOyYR%SJ~9bl{JW-w2++O@yPvE3=1}HEy5!oBhD>K@i1(f)%9Byz~}K2 z(RWD~qM=^KjL5Lm+qf;)_Kj*9__fhrUpQ}tD*X+a|!`8pJ?`rVnR!;HQktcb#iJMW!?YIAae@Xv_xv)g{n?2g(h~k(+`DDT06(Y(ewj(H+PVxU9S zR|4dN(xY|1LZ`@d=J7T1?-8Nl+4t!NhhK|3ll2K8Z->J{!eI*k%9MtAvhaH}(Kwn; z^tIDe#Vw8k5{k|*3nmqE!ZW(Xk3-Z>5=_*Vihi&!I!}K>n;ijNAh+=~WeBHdNNtIK zWk3`O1Np5&t)7==oh>KC&A(c}z1`6un5QKsLdM3Sn3$R)n7#DYL&l;O_7rX%zmk7( zasDYQMM+7CwS#Hk2%^3n6!eF}()_F1gxyEBUnBzrMT0I)m zK62z5iG;{6cs;bubIOCHGiDF|iQ?o+($ z0UayoT?_^}JRmXWhgr7_Z0SXzT}E0{Lb*f=-e5Lw=%p>YyxTvLcs7T z!7M}4@>3`{KR7LAJ8or{up3JWkiTvsTWPaq={~^}%qexU-_0rXKM|kM2-U6O94&!q zTLS+DqN+zSn@26~hfdSj=OHJ$lZIBxyOOjx$AwizwIMe%pq};PKofq+dode~0EctD&%%T8cY3x1Rg;%lH20$>}hcarzX3UJw zh%-Zl%fwOmYk2%;Sf$IsKR1J(TVVwd{{qR?74oE@%WuCEXPQGs+O*rrJsM4eMqF_h zGc!q}dT&Rq9vw${W}31fI-e+O?aRahkv&Yz#KB}H#$=OBfdM-)P&0Bg0OawH-i#9jisU|ZDt#()kAdB^8*5)`$5#9NIbE53Sd2$phlH=b$&(kNtGz#F$ zhtK%jB)dPKSylXx{Aj)!qgdRrOd7*rTA_TBAFrR?K%m>Bcah#eU;VqsCLigZka) z?FjR})JrWTIbycGuAsxK*zvuzho;)SH?K_(Q>M+MY%$BAv=U0D=;0z2m_`Yop#4c8 z-;Al864W?!Uv6DEpMB0$bvZmW3%33f-X1vkzm8A`+w2XOn)oH#VOPZ_9ZJx}A6k&?>OG~zPo_+*vnr!8?p&*>fldxZ@?FAC;xSXW@pJ*6Jd zKsdmPy)W*eokUMd<*r(AA*!bOv@d@~yy&IfdC#HqzZ#BclSon%k+C0 zgiGQj;{;K!&sOrSXNz#Pyl3#n8%*aRyL_)nBqSiMK_!i4dryh)pxjTtJGPIE8)USL zjd;$3O2AyxD!qO_V~;fZ_*R+kwT}6mHFuh@CBh3D$&WYKp#CYlRQ84XO0~B5jTgYe zPqAMNjmlM?y-lI#Q`xN$ywA+i*>K9B)gE#=;E_@;^9zUBCgVcYx|-71I$zE3%)m?Z zW$Bb&h%ahp5Kp*NQZo}CU@>I?kUKCG3I#n|Vqy`Sa|NH#WS>x~O16Xk_^seltuD9A3!U+iy`KF~? z6H|_VC{~UN^YHimc*?(#3?Z2>Hc#{4R_nJTbhcEFR7^z-Ci8m@*P31-UK#2y11}em zA<@dmdmkN5ZgOwc3N!_j2*sc-I_|_PJvW@+Mnw~Adne>T!_zEwLq|QK2B2~uWzClA zu+4Z$m&1i`lP_*bX?R;N-O(Dr$UD*uP3+J;K}#e?t@0mT&&WW=!sfEf9w)VCguur; znYcAn_OqF&-GUXjH=>7sD+Hya=F9uW;eA#^P;G7T`1t)D-#7U^z1g~!&j%M3kJv5# zI+FUx|SumuS?*R|bUHu)Y znmR+ z=DP;mVeA&b&%_eL>Rs^o=%$+^yNyHw^aV~mm9ACJ0a-#U9JkT8My z35`#IH@QY2@dScn2-(y`dfk$*?xK``QUIDYrF&{wVt!Z_aSYd16BQwP4Th zf>#0#i^)Nc52(D=`r)`SmII&Fxn;lQL@jSn&R4qNN>xcPKagqEQPsP?>!a;F#8vih zvR&lO>=|VL;tP^MA>oy2gWM z0TxuEI1dtO6g?;3%t9zc4cnw@)(&oWM#Uao_Qs_p*e`iHXYu z641d1st>K+j1L_(F>G!D3^2k6yc; z2uid|Hpcg0mxe%jqKIgX$rzbHSp*gt7xPq-$;d(fF12?(q; zF3@W4dU{>sGM`G3q+cm%o>0qI@IW~S;c*>SOO-byBe)Cuyoo=_8?(Ca-T;ky{o~hg zZ(<36#yLtvw8STs=rudo?iZffX0l^Lgr77YAe77}dm@I6pNB z2@r-BhggeY3eAd!^rzLejXCgZ2Y8wPZnG%INOml0=`Hi!_TB~|ud53TUaq5kgsNK< zwutOf5tOSBlvIPdl&y9ur#>b~Gaqt>{XY4uwMtP+l#GMNTbe;to;DmT<}*3Sw^qtS z;fpc#x5qRv@UX)I(zfs?_>U)bmpp1H|C%R0`|=F~s#e@F!r|ZuW*b>}FGX13vRP4b z50GJO9xwjll#&|y@<%f?c$YVrRuoHzO6m*c|J?VU=iY3Tl&r5KIGDp8QcR6lU)jRM zxx(C7=MxXYZ#|a&g(r2v=(iwP$36X!(Q-;J$K&*DiNo|wW|FgGC7D2d($Gu6z}!CasD6#) zO-cq?X(!Dg#Y(Y#lya%dszik}lH9zXRYYI`2@BpR>FmGe zDT5|0{?8)EI}sLZdYb^Tn?lo)wg+GqT*7d0zL>1)C2>GXbTzWV&xwp`*})-23opm( zAIl9>A!#nnvKCo>t@!(;4n=i!YT4%sC_t@1BW2E`^d8dhB$+pdbuyhV7FZgoMiW0H z!hkD-QVWEHgi(FL38piL6TmzXN-8t6v`l>pVPS)FhxD}JAPUuI>o4N-`N_#1?m}y5 z80lXnMS-+-4>}h^RwJp|%ZF3HfNzyAHZCUi_E^d(gHZT5IFXT8&fjKI+Xky{^)H47 z({9ctYSJ8>L|jQ{jDjHCuCF{yP>g7hEF`JcO`|RvndvYi!}@j_y~7rl+XwTFf9ouU7mM!>(QlN0q#GOSLEUc{}eWUTSIwwn{GJPh}|c@`D6Ek zjJO%FX9EI@L26JmH$k^#sZ7CrV~y3#XqPst7dI*(r5DzDf@!qR~*U$fX*A z5=j*Cy+fvj7+7Pbzt|iF%v@KFruI5_VNz())*S!DDxQe>mB|Q>2;H@f{4k|uG1c(X zQH4!+DbogivA29VAD_&wbw~B05XLg-94A{nL?}I&n9m~65~q!RufA2r%>>%4Ik<-T zZDr3)Efk7N=et)1#A6cqyt<7CII`SLOpaS6IsAX)sjJW?3SB#df)yqXNh#*@pX_l z-PXnpltT9Soe>SUqhVM@V3$tECOE0 z{_@7`AlPP=8QHMnc`W=F&IA7_LkdT|d=ol6UwWODooj+(ns4Oe6q5}Y$f$Lb2|`rO z6&`MqcLae*P8P>XBbSneJ|x5!rrPF@+Ly>lLjiO#A<};f(%yLQ6{&*a2RFPE@#N+C zgxqa(iG6MkM+R-&05%qJN2=szX=d7(A6)iLooSQ|Nrgs*c-Dr!l*bfZD(|!uSZM#2 z5$#%Eqp!}{aFz7qP2XSL9kHz3p~9Fn=x4etGC_5^#F*bWZJle~kM_mj`E`u3` z$u$(qnMWFQliKvHZod^1b5Ef|hR(W>;+Qt>E~UWiTEMYu`n9C%Axg-2V^zev#7V^Y z2@AsEVmwkdeu5e^5*-q0BL9lSw=q&?ppuzD_p^XEC<|Q-B%0Yj8=UX)YjRS&5RVCcA-bjl7__J_eu{ueR9A+lcYh;BNknAXri*%}x`8Nz~` zwt>faGwu_&I9J$j?V2xsr_%{4M24^A?>MB_B!EEH{=XH0{<|HhfZl82MJ=!ezHSmu_Z;PYo?Kt8S8E?5$t{f4i z#~jt0#9oU*SSBs5kvN>DO2BEni@_cHk;mRyiV55bC2aH&Uw_D}u#KK_%`|_=7`~JX z0Y#S&cYT_0f>BYbaJo+lZBP8G3_u3#zh)+vVJ|Nk(aV>bU)io_;^@F9%T(XLarHqb_$`zTKz}G0uD9W@J$W%hvo6z#W1m9=+v^WpAg&H-4XV;S4 zP*)m2NkmuES7D!%FPrc`BeQ*mcM5;lp8L0UsAA9eQe!Hu=iMr8-N2D>@y`bLOk__X zc`$y$+{~N5h+{J5V(a3MHtG{ z--*T>4KH|Zpa@NOQ9u7<@f#A-b@NgzAjU7@U9|UDq3ik~8p;33VEJX_9#mw6NQ<(Y z)$W~DHl>-^4b(L3^MSLa)_#cHT^tX)wM=AE4Tv?MJVQW?>wBkja``+V77lv9Zq)gr zGw=+DJ5bS1b*?gaKO8T`o7yG%j4(5AX~JC94Jk`L-A<~r@VZ*XWhwqJi8tzK7In^x zH)`sDD5#Sv--}STNYwRCKB5LCI*L0+5LbH z^Q`%C8JZvE3oYxuyUj;9z}UQIKGgq>r!a>mBbMR2%rC9n*m8RTzyyRo`>ZlVJz0u$ zI5-1|#GjmhO;S3p{qObJ&i~YBuSN4BBjsFLRoVoxc6JgG3P0_73$@9E5(&KdkIu2+ zr#ZKok1`tnOA(ZYLlHM5U{%5w-W#pb>ww3ARGOci(lEn+VMYvItHAbm*+^q)^B|C1 z1{3i}bMneQ%_n?9;j#K|qm)~su1!$d$^X^b!W6f^zY3*&9E*7lNljN`pqd~p8IDM? z#584=x|aO8*ulur^EaH|lgiSg_@Fkq)m@M7|5-WvOAmf%Nl`b)xu2J|n!r3p|2-HE+?HCr<_LlG#DA$(I>86bQsa{NbB zgb5TvPGstgA!cvsO+`yM&TqZu-79xTCTS2$UfX{ACfeFyV=^CFZP>m1a0TBgy29~Y zsUkJLO+MrVVd2PL5BK+JF_mKyKoII)AX97TFQnX-BK-NBFp$@C&3ZeRt{_P!OjxOE zTXU6vxoEtRi-#!g^l$a=>WY-I=Jme5?K3N_Q;4U>oG^}`Ze{-nL$*fm2Co8-l(YAS zz9?hGhBm@Fp81B{#*(8zTbJxs~f+sg_d@ z|5L`sWDgIaxy_RcgJS1Y!PQoDWrT!z%jp>^F68quFAA3!Ajz`w)X%g;1}oU3a{WYyNQc)o?NbvYPhaiN?LFy7fcnWYoe*I0zVbv0ZL{&8uFV!3*| zmS_4tr#80CUHaW|=X%Pj&F9~T)$5a{6?%jVB4MU2%q`K*yv$G?Vl3eS^Do?k3xzT5 z5o&I|@~7ALi3FF+n?ibIei?qhwVAo0EX^+~!hGcL5S0XJjK=i_^csa}{&^j&g~>!d z{Lb%phQjTyA(2CbI-e3b8!0EY3enj1KPqP^(0I{Ns+?E->5ba!YFftq8Db%h44(Aa z$et1>iY>F&h5Q8V)lFrhvFv-+*P?PI4b&?1FmdRUWuvNX7@g!s>3)Vq09Qs~2szE= zG+Isu2x>sv{UG8izg_fc%;B_>>_*wg0EIIZj*uxYLm_OGfw4ZE`KnG2QeqpZt2i2_`?kZ+|CS^Qwh<@HWC91P$Wc;A?gVG9lK& zpZW-h1pW98l&X}+6zZf^VqzEY@G=R^9F7fJzZ0cMED7^QHGe;Hm8y#uS=(OOoUdz& z6oie8jg|WCd4Ii(qdI%{aCI&D)aw^>##d`{{r8~F`I)>ej+hmJ>m=0}G||I`f`Zax z>o&J=UDwcrWj#;-mFNkT_W7^a6;enkWr6OozV>Jk@Vwm_7=M)U0iR}Cnq2x zGF|MzW#`}k)&vMS-`u$R2_wW#?id8W?16cRo59wB4yT zo!(u5$cKk%t>>Es@TQESh1+kVw>p|G@k)_MMOUvkWmxH$L$EWrQ7c^ zp}4Z^dhL=Y9L;rTMkSxT|WVVhj336N!PwXrgHAY zEokpZ-qBPr;QFSWf}XSvf6^F%F&Sw@_1SbP{CnO>Qc_qu>=Nq+HIDt!=7sm?PPSSF zkP5ya(a@~PVO3(k)9sSi7yUGTKqz5vS&S~;<)jJWBTBR;S!m$O>nwgHFa-NIqhK`P&ZZS zMd4O5xv$vQ*N36oXpZY~E>MSDxm=4;lm{%baSEH6)488706Y#u*Ey&$yu4PHI)VNn zRaJ2!^@qv++&gKc>pv0mx@JamcpUl|@Y?}bT06Y{MbTcInSrW`>2vv>5Ia~rabb2hK)H>TrEVG4b+0AAn09mVkx>7O_Z6%0tWn$h&Y(Zg* z{#>0#h`e5R9R_fTZv>=z8am4NDY%J1%7e?8$O-tEsJ>JfVoqBO#+1sRWQKlV`Z+w3 zKz?=ptf5(<2AeAy5Y4JEq7%2aFmT7fWujl~d-phFBrN&6#gjkaSlC~c3>c9>KHIIx z*5qX!c3nu9Z%d<}__*2Y1#%1SR%2J6+I$F*Cf1uH=Wify{Jc}(5IZjnqL}{Ijl$D? znYmXa_;ud(`0qc+7fSraLcWkG()RkiA2Ibsi~q<@x3)^3W*$r z_uA=EfFd1YEqh!fnfdvGgWLKkD?VVGKJdTw0x+|%a5&z~JP}-BZn~ue@T_nhP4#7< zGSvq@YvD{9tNpV1ND{abanV#N?|i)ME3Qn1sh3-g&lZkOj#=s-oy7>q^2@V7xgj{? zl_)!GCe|Kw0s(a|4QBJyOCn%zuz zFBFB}W%2=beA3Ob<mP zeCfpv%v*vlx-|F;*SG3|fP_yUI}5JFttrjYlX7J=*Nxw8M-;T1&HR4sKR;2q^zgP$)~!{QQYza?2#-HNFQ zGfQbKeUGHpN(Wp^VS|sl49l#_77jRUq}Fn^T5266wp#3SL!xPMKNp|deaQ#=F)wqx z##i(j5nwhldO_RJvlPB+1Oo%4Y5@Z8)?z8Pl|2hI$}P^=mubBFRs*>x$4?!Hzrudx z2~HxfB#r}ya6W>}attsy;6wPVqld&F(E$OZOjv9mQ}n00fwUOGdHb|{-P`jMH+y8x zwZ1eaA~a#nHYv#%6L{EJLy8ujz5oCr!4Aa{auzGYaN|rI_+hh=_vQ<5d>sJUTq#A6 zGB{LE!(<+HJn#)unAS}!&`0|ZVUs8Rn#+Czt<6ih8VLK62C7OkYa_th@vJniCoc3! z(1FkBTSDZzt+5$K<5Fr@{OKH`mT_JWaf*Enf@$)$=)Onb{v}Z2w{8Qq z96OtK?jhsw!z`fzY-iFK*+epP_ZuYRqBXyfs6WfUmnZ~{-!u10Lns-j8K8R^TTd~G z|D*zbQElIAJYDt~lDK#rk{w2VhzAE@7>Pa%{PttRB*P+04!v#6VE0fAAk(A6dxJsb zomo5*H9M!fjU@zR4`ErRlu#Ckmn=vGyPm#2nK&L78O)C?4A<}cLiN~E}?{6_kp z#R^zspruf!xlV#LMxPRv5~Pkg@J}c>Nhjed2OzQkhS7jX(ft4QFgpEdJWzFneQFR) zg+^sx$UZlb`|KgIAxV_lGRV0isM8$x$dJTQDPY9$t|?*qaDq7?us156INZQWhWB4D zL|&^OMt&2+|4{%9QR__3bS&RNAN+xt52!P%NYi8^b@>f5k6AXe+QcXW16DxLo?rzO zj6)Uk-xW|&7$95FgR`Zffw4lpT!C{q8iIWen8KYaU9DB=dAYgKq1$NLsidStIo}2B zAS3_^^wxr`9bsW${;oIK%zthU|$ALdT5`fRfW9U2|@OKNNyB zH&GV93zXl(Ib>oyM~fM#K-%FqQkD_H_VMv?AA!CZ01ylwnM%LKUcOl|MU@7|s5dZ< zkjoMw$z{5e#i+|WKwSxh41_}=aE@{_%9o5lEBjulrGB1hF%JP%WWL_wK&8m)i$^yQ_Wpb9>v7)*I+j^FE1|`7#Qo#-XN!Xn|a!`Pen}y z&i_oF2%2B5M2Yb3Vx8}s%;;Ae^dB^eDcj(+a4Ylbb$MU!4C+0_Ql&kW2p&+=hRkfWfLd@WG$j}ioZ|4s0`*?qK+3tgE zux)5<<%@GENTSv$04RQjQ3K~b2}*6zX@KguAA@{kV5$=tiIA32;?FK4Cl_g*`e8Zx z>&vE!e=AHOOB1oXQ<|SGIR{AlIIh6;HN; zK5>~G1ixnTyHLu+lSEbPx0Xa^e|&t@wYH9wC}++V%Azytx1jLy@^1Y+>ZBpKKVECL z+l{@MK(ie?+T9I7!eyk6CT@DsZFMBos@6-Ugf1PFt9D@0uEAma`jg4H$G?HR9J8iL zRz9u8W?qK>awMK~;Ij7+A22{e};EzsI*-s){<9FOAWFIPI#h=Qx~9hc-4g=C)lxkdDDCekYjB6%%{89f&}e zCS0o=%MlS8jwfY;D3bVFX9BHNrGps*kWCh8ad&fV>_b9((2 z=D^nrJ{UN-0@Ykm1ax%STBGjSDqXJgjZSTPBIWcLJeGd2Nen&y)NZz2R4Z4f$#=)5 zRSFIZ3oFxjO}`3wd*YBw2OEDFEHd8q?of7J0}8fBHRRjKdb1s!pizY;1ICw^`!l))i?^q10%7ku0t}+b$IGpOu`xuj zJr081T0C^K!^2KOjtHV-JPH6dM(kXh*)oH6*Mq^lY|#L{U*;1jEW6{eF<`YNVg4nL z!A1<7fL**v^1yn$05VVF=H>=#u;fRcvUx+c=5>D%&}=GECu*QmG5jA8DAkquijuAsIi_1$5hl@n5@Y-QW$z z8k`L*5!Dbfmz>EFk&u7`KQ(-OqA4`7&$q^v)S}hk7X^IuX4PnEop=I{uLiTELY{A; zh%(tv2dT!t*o;L%V?1_Ckxa2@>g8@yI4nT0^HSj`uU~^*2U*<~@ty{#kjfg@Xf=Zr z@=XV$9GSFX9`OGDxV1&SlF8=;lh_PXIJ1vF+@xX0m)K^zSV6#Tt#+6ExibJinIoc) z5kCHlQteeU{6!L4A-B7`yD)%BeIZ-OD;ha&VRdFk6(d&DVxjC?-bq8i>qCK3{@1Vq zHfnV?N}W=bY?%6HnE623kZ**PROL8^P3c@#;gyw@X|5+;l&sI*MX^Om?i{w*hCj*0 z0%M2w6jO?y8In*$-40{ZC~daO-N5k*S85I`0zqc`<9dYrYqDttPU4Yb2quS(^Kq+T zrzc{AEw*wc%zn8>sXHY>VklfDLP@o@qB$jcd3ka&?!(18-6*CTP&^vr)~_KGF0DjG zFKdw6b&?@7vgADuXJcwg`OdJCGbJSj3c4a@flD-GsOy(p^5{r`jhH2+5`tg#r}TUY zZ@@(I?8k>8M(i6JR1~%U^UcAoEf*i(XEcDD1^1nap)i zlK)NTu|kn2xb*<@bhXjS)V8F_dFLyKAqKFAx*-KnM9=MjnNM$>1m0Jf3SqB*f31_} z*1$`0NK{1xjy_VZBND4qI2dx^#=gnxVI+pk{dp)>*APQs>X1XUyYWH<`(F-%*uQ1t zAZcI-EUe{PQFmWc^oJoM!4I%KFms5*P#TtV$QOODOM{`xJ3l}FNiJi~$-#kHN!n3h zT{p&8AVQLnWW7cMp9z;4NVgT~Pk6zYRHCx>01jzPp64q+hGOtz!bUp1E`KISeqgg~ z>`@Y|%4mFfPTV)=vz`@y@+r!aQlS(LP>sP*x4rQxGqg1Z&%Bp1D4vzF|LEy@@9X`T zC?&RX)PnZSiS;;D*piVPhxWuz*!$cP4fU_;5*-xQCQ{?_yjatEk?Z$z~hQM zTq;P}&_Y90+fwRkbeZ6!8R>of4)=11C@uaXPd$iZq>?ab%qpFFQ;Oa=5?cj#GzDly zG{MBEtbabYhfy|4LI^Ot<+uVb+#89tMc$d4r{nDg$(NY zZ1)jbX-)k3mkgLbkp`ON&wdNC)mZcP+uPd|tjw;uy9!%FIE1AKg8uRV5Fn}X>n}c5fT!zAmAl6O^O%f ziK6UOH0++3m@EI2BKH^8s?ruV93qWgq6Ve0T1@Ivx3j0G%>C)2gg!~rMn277blJL} z;!3K$c7||k69o?8;3%@U+3jcAp?Gd1WAOC1H^{h(y;+}P0q@yrOW{NhIEVei{RH{q z^tY2VannAY+HpIpZL#78$E`t!JEC+n4kk>=5UADjx;&OTP2JrhP#U3#FhoF5Fy16q zx%fL?^_#f@Il^G1HR;jCg$Pf5w}NY*$>+!WdYNX$1t+aoz2q<6iJ{0br$+DV-R!7l z&M}y?wS%_VL3_lv!DnoS9bedmAhDS5F-dxIPG!e!{wxzs2MY3W_A0+S6HmH+6A+(w1}CN;oHq6kEpXS~Ta}-(n3{?_`v$yFNwoa^ z944PUJL!*?Eagg5DfV(BK>!_M3A7wXgZ1}G(%`#p_deDsY zHc0@IEEZ?|W$Y}keZW0I=#W?NJGvF+k@o=i-%Xzs=(eonxYB~_xvd`1C0Tv%UAH>{{@%pFSF;Ff9zA=zCdQOg*pqI9GF%#X zJCMV0iS^+!aP(@N=&m@U(XqTKVv`VoY@`9NyCTyUydeV-%GDrbhRb(t1K7;6Y&~B@0%8rc&EUS;*3Zi%`=$8 z(!$XqJleC^#DdDOM$=3c3RNuVibz}KFBV-x)-5h;4qCu2Q&PorZ0cDD9Ev}_8T&-e zG92z%YWTQ#vR3{3&V{f4Eumvh=O-b&=;G>tiNV*rA|hay%u~}h4tppw83AN-X`fDY zHTffv{A-2gtxMy;Sie+q>E;=Z{?+-w6xQJ)B;aBDGe3mRP@m>=^ugD3qM=~TdMW(K zz$;;rw!jzGoWB7QgS#kIpA<8W6YkP9LlGYTSBVVgCAfTg+#ep+PJ@CvKU0xv$KB?9 z2ltlq6zcYqqr~Ru92Af{Cn5JgqA_uNpPC5_I!*S0Rl?Kn5rDQBVx`#R5pd28yW3lu z@U27YufFid<=YOgCq20G8t3Tt;=~0icfha5^gr5k*q_Y7CgjpPYtCi(rfXgVcZ6;f z_NnM{4Z{^zk$yYrNQXV|*?>K{v0<3ZlXq-7Je&Gf6@9Pp#R_O2RnN1@SxDCb9Ph{K zCVBb7sg4@-+4|%&C&z8K0Ocqt!{m6`OP2rwa{!Vx7v|~Tq4F9dl7&%&&PR!(0LySi zZMOa%W2-8|4on^gyTSxFbQJRaa94|QCPyJ2qXbQ5MPm&qm}DH`d_!+GF8`Wr4y4alI5|E#?_omlCzn!` z7Q=1Z5Ks8mip7Y(bvY)(F8IP>g4s!6hN>CPsA*?FmhNV*6kUyobzT}G2CRP+|0(;E zik%%JR2EGmm9~Svl)E(oue)n`GI65ZuM5=KSalVwFi^W(RgGf5!?Ct1{Q^o z2f-@ApnuCs)%4-3>SSg;f`wx0YsH0_aL`Cdu@S+)^YDz8W;KeSFXj_i#gpjdVtrS+ z@P(3@)7I++qH#EdrF5gLB7)Y!hjlMdIBUIz%%2c47p{0YKZ+oP6mF4k5rU)B`$7dr+A#|j>chSym9#3LPn=bK61y9W4*A1#)SWp zr}m@2pEbiWlh;1p=;7M&7hT}s9{uzG82l7s{&(;bDZKT+0Y7#jNGXf7$S;pX!F6@) zQnr+dEQ+24goGxq|5#Rez}G`V=O$7 zhZ(=p`HVp(>>C_jeHOVDMa_Kx_VPYKGurEoQq^4N*G-l;VGt@$#bST$USLGbIt2ec z8H}5ooBPQI*~mjqc(L?C{`+QVXTARvmP8d^b_U=mJOI>db8&hx%eKl<2p1v>vp2ap zLCFc#$D?d5^$xuCeLkyn>Bh4EckI!QLix-=1dmi%zS%&@sZ*q6&LQwz9Gx0}J1*vN z2=lj88Do$gY?=xj-uvPUm4@VtjM}m-%(}1)Y=)iGCAHaSkHGP9q(Ztj(}Cz}O0jp9 zsQ3n>?!c&yDckMcsI&f=xsrg02h#^A^z&7B3`tobBbx1E})| zs;iub2LrUDVEAb+y783Fe}|%cF}+(%oPbjveKsG zZm^@&s+Pie35ubuo^99mTPV}gc(SW~H_HE>T2c_F$8(B69;?+sWgxRVmyCQoTZroa zn1!Wb>00tX;zYM)m2wI4Y%u!AEhR;8llXvDW-i^n7G0m&yR@XJk`&J3@%|o-1pZ7B z*K>@Qj=R_o2he$O|8l$kMUoDLwZ=qA^$v#zFBJGaoT>n5^M+}XYJU!%fE~A10nl5n!HVM5v zx+^T@jq}aCxqMuCIL9GGt?XMRPi$^%#5*!C`GISIhn>%wyCe~WW=l)D)oRu90t6)> zepYpQ%J{G_nn|v=EjH>15ef>3@Nl>gwKW_GrMQAoZDB8kb;k91Oh{$wk)gunokz_<*D&Wgtx%CpQc!)HQR9hJ-Ls>p8RUb| ziQ%&#O=73BXE)Iorg?IxFS-5BAAR4!Z3eVfqQO7-TprE?wSrjaD;Xq0eA-{QwK-ZM ze&mN3j+sXts9QEnlHx-Vp^i*M)D9VnVVc<9wC9wkNi<4ub!^Hh4`NZT_%Ao7M(Q8f z;>+ex3TVu0t$LCmz&ye49GHm-w?AU-eA)RXz%N#bsL{G`l<#2*RQ(0GH3;~}H5|w} zlLon@%WPt4qkud**g`i!g9$m!gMb=^VgB+&i(!(VE025czN?Lrc50RGCgDo@PR}DI zQNrx5zfPYt5W!*oUC+mNviIl?uc-|5Wy+mpe5M}}ZBb<)FBliP3qGUj zn6VfID7OI++3WY5NCNZZz%6QLZd%~Jw*T6#25x5i{{}#!iPfco$b>|Jt>Zk!#Mli# zFoLjz&f}x~Ut=AGOJfKOs9>WD3G}W~?X=lU+CROq>aD!<(yU;W6n>pL<8&JLe2V4E z1Se8xn&J5UEv+5g5=ZW(UYo{dL>W>%8I$JT$}#m{0FRvM{|@lbE-K3Y1>X*H>c(3g z-LAN$$hSSm3~XaVaS{nmW`xpG111snMcV*y#{jT!iZ5dC%41epVLv4om!Ydu+e}m2 z5h*=Fbf35V%YK`CkAiAO;{$0z&)8<^{oM9H!b|6mp#-oF@4Z z$6~|~5k0iCUkI9(D9ONY>0FwY;(?^E6pime@K0iJSw|%JD4K0~oI}r)8?cBPEXDjl ze)!H#93K7%2NcftJ02*;G88Ux3u`oJD&#VhlidF)dliK0dDQ36e^o}w|ElIx8=065 za}JIA6Ex~zM5=?u->82Qq|{v5Rj94m1SJYXCbz;FQ^XNo)G3}atsOs;;W7{P8X zK?Rit_F%9FkSkV)m;p>I3c}R76wItk!8Bt@m|7J9m_c&&JN?zm)d&lCfRK+j5%S?C zLO{njj%krm9dRD&52 zDN`x&ml?<{C4CPzj|@S`$J+>fPo%q!;J5z>)r*nta1I}+u)1V_{HO2ssXo0E?BQZ$ z4+pwvIP4iiHLcM!#aZU42dkWoV4l@hs#9`|WJ2z=!$rdDm@z*(k& zj|x~{Y=Afl(@WAD!EVfhg~|vM0Jt&+L&08y46w{z2UfWn!y;P^fQheVN&^URO*n$z zoJGKkg9vZP<1LI7( zwk7j@rwFLS-s0JXQ3(BT9pEfL1>Uuo6ZYxS9i0fq7s2Qb!*Vb}Eh6W?;f|mH(Al=4g;! zV3Yk&hyd-Oy}atqd1M&l9NSA+qhoxB?C}!p#Uf<*sKN;Vm5egv;(4pgX2uo;6`CZo zr#A%X2Xe(g#0zUVP;(>y<#N$l|%&S5dFWu z9i#5*$fA|FTI}gIGV;f;<5&Mw+^7T9D==3ul z{`XvD&z2=KHZGcmy45Qss;%TGw0ZXtbaMJRR@I}Odt%(MzOb^4{|JuCZrO7LKJ$Kq zN~Ma^JhfVlD>v_26oy2SvJJZ8zw^71FLw@6NARZ*RBhf1GI zO8HWWYMTgFt8IVWIk7nwNkc=!P^oDTxDIeY=T;4&(QvHUvKNEL{e>5o_akR^dKy6L zCnCwuzt6$m)w5w}XozzfSK91gEBYmI*hr?Ho$lD!dMq}%w9!A zs9~Jn79Z~(OOUXML4#t!%wM?)M>j15V^KpB5Ez6C zjU9wN9a=QNqbJYdHhGrl`;fj}#05!qYRewkxImQsvb`qtY|&@Luj0b8tyBj4*33ah zD+`_M(w%G^4}Ciz{L>R6*^ClS7$th~`6%cG`ChpyP7_h5Nl6P=ZNb`Y`*C#B0&yj} zdw7e%C&20Z7q8w3a7a~d-Ft`ye~dx-QbjRz!VL84)B^VHnqu$KzeT=L&QX4#&(?k7 zv)jP#ux;4~wX2oK@c!L!h-mi2mr>rZK4B zdybsKp4GE)_TqKXp9g2QgOrId*u82dUcP>dHhtW1e5*I=*j9`Mb4-%NoaP3Mo{G}N z3!{7cW++~#BZl?wijFNB2qHg!^+w>lZQn6LP9oT%)m!6ikBDdQG9C;67=y-jZN-J5 zi(N{%5b%5-UR;?#dt*;ZIw8Lx*zLt=4`7RtSsoO&+X}F_=Wtycdi?YS3RY{4C+Bw~ zt4$_Bq&ZVYqCxGdf+$1;m)@Pk_Y3~`B^u~%gT{h{)Fv?$7yx_;lgY1(&;GGdw)I0~fB|76jX~Y8I+A z?-gfzzyIuv()GLG{N9x?HZqDeF7er=>$r=2T!RO`Z4nXjf>f-gmS8LBc~?2h{fdlv znk%X=uFM2(!!FL~*R2i86#q&5|KHQwgyS8PV5RIjW5dE}s9dh3sHV%nF&H_pyO{R0 z`LqMOZu@~~1zW%U0J=Gki*A~xo%jg$DI^r6Vl*XCqLt*)zt5wirF&N~lkw0~6`w~>ESX%gVrR}=JOE8+2CwnfAzVfwz(bQA4P(GnP1_r9= zw&*iWnWSLXZvPvn9a;q=Wg&vm)R(J5Q@p&gA33sR5ei0%S)+1UwCXhiYvxWtrLrZ& z_YeMiCj6u?7dmnIV9#IrAy1{|VoH7b;X^A== z#k7zsdsh55dnvrvY{S*VYY`aqDb6WUJ4harMqp6S4q|Z0$0Eg{!7?;7K&xJ^IJako z7alq5F`D z!pEF8W-vYm2BBVu0T?x?NAw~{3!-)vNhBcx(A3<|?N{W^kxh`DR_Jv*3=kJr*^))D zdh;HP{B<@SpWBJuhfhVXA{%!b9A|r-S~V1;+O~8C%9MzjS#+^0F{HS&w;I4B!R{_b zZ4Ge3I72=ZBL7QIiLTUP6K3H2)mu2Rb#e5*i*}09`e1A4{U!#9(oAn3xVq28nO)0K ztZ;rbu^%E~P(bAn(LzGBK_8mP9n@kz8c*Z!?Y4`OsMfnaMKZ_>a z2FL2%ys1B9;Ag^3owH0h_ms3NSsDbIPUEtJRErd_O8Jt)CQ7teph45_wXI(;Ra`8j zZuC7ZXeeoYKk}nurnvUOes<}FF+=)9Pw(WIJC2O<6$zJPTNa6nm&y_*TbEA9>o@P> zY>&#c?K@JC|G$5BAv7#pC%Y0uiHE-J#cGn>S&WVx*p$JhD)QORPfT5UEfW?N4il58 z2MN*?@G&3=R+i?%sg0T5z5fVfV^GskK#K0Chbyf&|S2D7i~r^SVdv2iq65(`{lP_XDT`IKMc60^>4wHFg(%C9fG zVz&L{>>zydKM!_mFS;|#N`ayLV+@>3FBp)4p7h>5_9%M%drVjtM;CFli? z6{gl@P@v{=iIPM>F9njkCJE~i*~fNrvWJ#1j*7hTh7P(5)LH}6j8Pk+KSGLm)T8b1l^b_O^uoPU zTT!G?K5X546dn4H7C+DW)lHnC($)ZN#!~FsKHt+=v0)dcjU6mDPicdcwmPQFT!KHm zR-j3}>hPE}9HojC{t+M*eGL6@3y=TqNjv$kwMRwU8PFz65l3VxI^c%^A})f(Jcraq zKKO24BsO~KP`yu=)*`M=8%aUuiL@U@%138{bR0yRs-%oFCJx80gC{U$)>09wL`O(J zLVA05A>RJGgt{7a7I|ke`fy+wJ;bC)!~I~Ym~Tu;zmOzEuo(;2pmpOqaA@C5fIvqz z&3g%)e5CxSEDkD)@m;zN%H2-0g57}qaoUnZ`C+z+DOld^9bBO z*aRAN!2hEuUaf-pyD%YV<~hnvfOXzB=^2CQzQsi_ZP?B7nUm0G#N_Dk8!8ktpwW5E zA0k3l8-1&-i{h+^U|*)iGsZ_BDWzv{Km4~9uWn4G>G5evN96l|M60)W_C)Wfh}KQ% z8G~2^qlgS`m=67L_{2H1?>}0o6#XUtsdlvrqRh>^575+MsQ8V}L$#qrS_PxX3W|B7 zLv%WPr#N~QVU-% zQXM+?9N}mWb!4QN0y_`*MMS+)5FrHyk`uq*XRbJ`Z_#6zP%H{PiW$46B^VvrYoksn zz>*HSFJ8Ni%1wLV=CO?^S+r2PMl{IpB%YlgK~rOhAw$t3j(8U9wzXT{n8z5L&9hcV z-l}ucH5k;nq2s5E5EpHt=!995UD2y^OR*ZHc=)b^#)@bSO4?X<3bm%tqW`sbcCk%X zaTx!dckSA3Z8wJ#HWi1m>??*%BV4F4!H*?gcteaAnh0SZnIw}$6R%K1G`xY$j-qddN?49BukPHC{%Q^Ss4l#WYtC5GQoA|}9pH!PdSzgK?`4o@; ziid)Ow1dD^2uPys+2b#wqhZeZu z8{5h86g&8@a0FyCKiIXCsj**$2NM=-t7=9RauzHp6U=U@cR#fX4l6~i<@W2_E4HiK z%AZ3i_kx`{6>cpAWYxO##}WPx9@Nq4aq-DL?b2+iXu*hprBZH~VA5#teLRso-qxBn zrEt5vYbdE~jq6(6RFkQU)v2>o2EvG}uKZ>46JYB6?yDVMa^V8X4Z z*@NO4=}{iY-cj9#oMk%Y)b;jw-uNBZJ>s~D-!BHP7hZpBCI8#$Bd{ugBhyWSw< z8n9x!++bb`i9wzvw{qd}-)G3?Evg}}u*Q5w&B~xJ#^hI2n>vD+R85ZdnY1zU3I&Zu zS3(hHoka6B&{0)E=R$UT>0-2~E7*L43w}b$OoXSj;%N;V6-d(HoUvF~u6$!EFE-6N zZeRR;0XgP>1z7GZF`U;LNoEyK?U~5G=C64_J#H+I?a%h*E|E|gep7G)MMo5l);1Co zs9mD+zYuKh+WkL^!O3Zsf?OM?rTKx-CU_V1`r4&Iy-d a(*FUsfqLB-RR3Q90000H1@V-^m;4Wg<&0T*E43hX&L&p$$qDprKhvt+--jT7}7np#A3 zem<@ulZcFPQ@L2!n>{z**++&mCkOWA81W14cNZlEfg7;MkzE(HCqgga^y>{tEnwC%0;vJ&^%eQ zLs35+`xjp>T0W7%LH>+6%r*|hl4&cGEwK&; z0t5mC0t5mC0t5mCCIbR=XY(RVaPNECpbiiS5cpprz~2v1dbo`Ha?q3RtAi>x1t3{Q z!)~EA7(47WZaj7p7h)qJa-0vW%wBi+||i1 z>Y6jeJsH#i0s#W02=Lr}8dI1}KUzutUJ*)+)n}}h7B%mr89V>{b*Ij~SgKbqtIzh1 z_&y-V$(N-h-*{MD|n=-5v>CU9GhHg6z4d)S>QZ{v3xgZFVFhkgz?6vxm zj^gC)W+k_q_qHicZ_00XSe7x|R#H23t*=L(m9R`l(M>JqI_s)NCVRScQ?>e!Pky#Z0Eu8ZB;x<7OHNdH{}cOU0x zp*N5JhGG?7f4F{zS|vvv{ki0^i{skZg@gx3=;IomV-RmIn6&vIvbhnBNRisH*Nt0M~+-?VJy znx+jy6>?>jG)tP@RnTocWoVs!y6W`qRP3~7s=9e#7vXO@`W?EK6Dx5{>@by#HdIwJ zXYRaf!{M>k#aX)ZDg9txcMi+7f-qDiS5`#lhowC#oo>{gy7}SoZmvn> zG^$l{a5uFiTrbLs&|%Ygy^*C-Oq6Z0<4z)pI}%!o6hDXA!|ZW(f}LWg*<0)bc7}byF0k*}FYH(L8~dGI z0#J@h%)tUIL>;cjQmnyRtV0~F=)?|mVLue4p+ZLvHu5N-ANS#YJb(vr6i?tuJdNXc z0k7ayyn#3IE^t@| z@%0k>6H_rA)rb;P>(NL|jU#~;tjAWg<92LE2lis0&shT|vN%Y*?Z*J_!65F%LwJN( z`zVg#DPrxjcn&A<5>66#U&HHo8}AT*-^2U(2%n8%YujkH;;Zr4I`SU;cG<}_OKs8&X+EUj6yPJ8kTv0bj66`c2`=acoWp{{D1w#-~X?4nZQ3lAV6ReBY?7& zWJ@z`aH5wB&$XlU?5BqpZa3&j*M%xK9Vh9g<0OawVMy&LnR1`FF9$tIBUJwL4*~z; UpSi*PAKd>V^Vp0vdGG(f0m+@jLjV8( diff --git a/docs/mkdocs.yml b/docs/mkdocs.yml index d1799bc9..258d6be2 100644 --- a/docs/mkdocs.yml +++ b/docs/mkdocs.yml @@ -1,4 +1,4 @@ -site_name: Delphix Virtualization SDK 3.0.0 +site_name: Delphix Virtualization SDK 4.0.0 theme: name: material custom_dir: 'material/' diff --git a/docs/readme.md b/docs/readme.md index ece74f2f..89fa73bf 100644 --- a/docs/readme.md +++ b/docs/readme.md @@ -2,6 +2,13 @@ This is the Markdown-based documentation for the Virtualization SDK. +## Important Note On Building Docs + +As of this writing, the rest of the Virtualization SDK codebase is based on Python 2. +However, our docs infrastructure is based on Python 3! So, **all of the below commands +must be run in a Python 3 environment**. It's recommended to use a totally separate +virtual environment for docs work than the one you use in the rest of the SDK codebase. + ## Local Testing Install dependencies for building documentation and run `pipenv run mkdocs serve` @@ -13,8 +20,8 @@ To activate this project's virtualenv, run pipenv shell. Alternatively, run a command inside the virtualenv with pipenv run. $ pipenv run mkdocs serve -INFO - Building documentation... -INFO - Cleaning site directory +INFO - Building documentation... +INFO - Cleaning site directory [I 200424 15:54:06 server:292] Serving on http://127.0.0.1:8000 [I 200424 15:54:06 handlers:59] Start watching changes [I 200424 15:54:06 handlers:61] Start detecting changes @@ -59,7 +66,7 @@ Install `setuptools==45` to get around a deprecated API in version 46. $ pip install setuptools==45 Collecting setuptools==45 Downloading setuptools-45.0.0-py2.py3-none-any.whl (583 kB) - |████████████████████████████████| 583 kB 2.7 MB/s + |████████████████████████████████| 583 kB 2.7 MB/s Installing collected packages: setuptools Attempting uninstall: setuptools Found existing installation: setuptools 46.1.3 @@ -85,13 +92,13 @@ This will generate the `site` directory which will contain all the gererated doc 5. Go to your individual virtualization-sdk repo's settings, scroll to the bottom and verify under the GitHub Pages section the `Source` is set to `gh-pages branch`. 6. Right above this will be a link explaining where your docs are published. -You can also utilize the GitHub workflow for publishing docs (`.github/workflows/publish-docs.yml`) associated with a pull request. +You can also utilize the GitHub workflow for publishing docs (`.github/workflows/publish-docs.yml`) associated with a pull request. The workflow is present on the `develop` branch. Create a branch called `docs/x.y.z` off `develop` on your fork of the repository to ensure that your docs branch triggers the workflow. If you have more than one `docs/x.y.z` branch in your fork, you have to push your doc changes to the docs branch with the latest `x.y.z` version. Otherwise, the workflow won't run. You also have to make sure to choose `gh-pages` branch on your fork as the [publishing source](https://help.github.com/en/github/working-with-github-pages/configuring-a-publishing-source-for-your-github-pages-site#choosing-a-publishing-source). Once you push doc changes to the `docs/.x.y.z` branch, the docs site should be available under -`.github.io/virtualization-sdk` shortly after. You can see the status of publishing under +`.github.io/virtualization-sdk` shortly after. You can see the status of publishing under `https://github.com//virtualization-sdk/actions`. This is a fast way to give a preview of your changes in a pull request. diff --git a/dvp/.python-version b/dvp/.python-version index 43c4dbe6..d20cc2bf 100644 --- a/dvp/.python-version +++ b/dvp/.python-version @@ -1 +1 @@ -2.7.17 +3.8.10 diff --git a/dvp/setup.cfg b/dvp/setup.cfg index 776cfe6a..48fb5749 100644 --- a/dvp/setup.cfg +++ b/dvp/setup.cfg @@ -1,22 +1,22 @@ # -# Copyright (c) 2019 by Delphix. All rights reserved. +# Copyright (c) 2019, 2021 by Delphix. All rights reserved. # [metadata] -Metadata-Version: 1.2 -Author: Delphix -Author-email: virtualization-plugins@delphix.com -Home-page: https://developer.delphix.com -Summary: Delphix Virtualization Platform SDK -Long-description: file: README.md -Long-description-content-type: text/markdown -Keywords: virtualization plugin -Classifiers: +metadata_version: 1.2 +author: Delphix +author_email: virtualization-plugins@delphix.com +home_page: https://developer.delphix.com +summary: Delphix Virtualization Platform SDK +long_description: file: README.md +long_description_content_type: text/markdown +keywords: virtualization plugin +classifiers: Development Status :: 5 - Production/Stable Programming Language :: Python - Programming Language :: Python :: 2.7 + Programming Language :: Python :: 3.8 License :: OSI Approved :: Apache Software License Operating System :: OS Independent [options] -Requires-Python: 2.7 +requires_python: 3.8 diff --git a/dvp/setup.py b/dvp/setup.py index b530d493..e7f78c7b 100644 --- a/dvp/setup.py +++ b/dvp/setup.py @@ -18,4 +18,5 @@ install_requires=install_requires, package_dir={'': PYTHON_SRC}, packages=setuptools.find_packages(PYTHON_SRC), + python_requires='>=3.8, <3.9', ) diff --git a/dvp/src/main/python/dlpx/virtualization/VERSION b/dvp/src/main/python/dlpx/virtualization/VERSION index fd2a0186..fcdb2e10 100644 --- a/dvp/src/main/python/dlpx/virtualization/VERSION +++ b/dvp/src/main/python/dlpx/virtualization/VERSION @@ -1 +1 @@ -3.1.0 +4.0.0 diff --git a/libs/.python-version b/libs/.python-version deleted file mode 100644 index 43c4dbe6..00000000 --- a/libs/.python-version +++ /dev/null @@ -1 +0,0 @@ -2.7.17 diff --git a/libs/setup.cfg b/libs/setup.cfg index 3e46329a..09cd385a 100644 --- a/libs/setup.cfg +++ b/libs/setup.cfg @@ -1,21 +1,22 @@ # -# Copyright (c) 2019 by Delphix. All rights reserved. +# Copyright (c) 2019, 2021 by Delphix. All rights reserved. # [metadata] -Metadata-Version: 1.2 -Author: Delphix -Author-email: virtualization-plugins@delphix.com -Home-page: https://developer.delphix.com -Summary: Delphix Virtualization Platform Libraries -Long-description: file: README.md -Long-description-content-type: text/markdown -Classifiers: +metadata_version: 1.2 +author: Delphix +author_email: virtualization-plugins@delphix.com +home_page: https://developer.delphix.com +summary: Delphix Virtualization Platform Libraries +long_description: file: README.md +long_description_content_type: text/markdown +classifiers: Development Status :: 5 - Production/Stable Programming Language :: Python Programming Language :: Python :: 2.7 + Programming Language :: Python :: 3.8 License :: OSI Approved :: Apache Software License Operating System :: OS Independent [options] -Requires-Python: 2.7 +requires_python: >=2.7, <=3.8, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*, !=3.6.*, !=3.7.* diff --git a/libs/setup.py b/libs/setup.py index 907b103d..7ab8eff5 100644 --- a/libs/setup.py +++ b/libs/setup.py @@ -7,7 +7,7 @@ version = version_file.read().strip() install_requires = [ - "dvp-api == 1.5.0", + "dvp-api == 1.6.1", "dvp-common == {}".format(version) ] @@ -16,4 +16,5 @@ install_requires=install_requires, package_dir={'': PYTHON_SRC}, packages=setuptools.find_packages(PYTHON_SRC), + python_requires='>=2.7, <3.9, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*, !=3.6.*, !=3.7.*', ) diff --git a/libs/src/main/python/dlpx/virtualization/libs/VERSION b/libs/src/main/python/dlpx/virtualization/libs/VERSION index fd2a0186..fcdb2e10 100644 --- a/libs/src/main/python/dlpx/virtualization/libs/VERSION +++ b/libs/src/main/python/dlpx/virtualization/libs/VERSION @@ -1 +1 @@ -3.1.0 +4.0.0 diff --git a/libs/src/main/python/dlpx/virtualization/libs/__init__.py b/libs/src/main/python/dlpx/virtualization/libs/__init__.py index 6af6f0e5..040a606b 100644 --- a/libs/src/main/python/dlpx/virtualization/libs/__init__.py +++ b/libs/src/main/python/dlpx/virtualization/libs/__init__.py @@ -1,8 +1,8 @@ # -# Copyright (c) 2019 by Delphix. All rights reserved. +# Copyright (c) 2019, 2021 by Delphix. All rights reserved. # __path__ = __import__('pkgutil').extend_path(__path__, __name__) -from dlpx.virtualization.libs.libs import * -from dlpx.virtualization.libs._logging import * +from dlpx.virtualization.libs.libs import * # noqa +from dlpx.virtualization.libs._logging import * # noqa diff --git a/libs/src/main/python/dlpx/virtualization/libs/_logging.py b/libs/src/main/python/dlpx/virtualization/libs/_logging.py index ec377932..0e97eb7e 100644 --- a/libs/src/main/python/dlpx/virtualization/libs/_logging.py +++ b/libs/src/main/python/dlpx/virtualization/libs/_logging.py @@ -1,10 +1,11 @@ # -# Copyright (c) 2019 by Delphix. All rights reserved. +# Copyright (c) 2019, 2021 by Delphix. All rights reserved. # from logging import Handler from dlpx.virtualization.libs import libs +from dlpx.virtualization.common.util import to_str __all__ = [ "PlatformHandler" @@ -17,4 +18,5 @@ class PlatformHandler(Handler): """ def emit(self, record): msg = self.format(record) + msg = to_str(msg) libs._log_request(msg, record.levelno) diff --git a/libs/src/main/python/dlpx/virtualization/libs/exceptions.py b/libs/src/main/python/dlpx/virtualization/libs/exceptions.py index fc4a7c11..f2d4c3d9 100644 --- a/libs/src/main/python/dlpx/virtualization/libs/exceptions.py +++ b/libs/src/main/python/dlpx/virtualization/libs/exceptions.py @@ -1,5 +1,5 @@ # -# Copyright (c) 2019 by Delphix. All rights reserved. +# Copyright (c) 2019, 2021 by Delphix. All rights reserved. # import sys @@ -66,22 +66,16 @@ class IncorrectArgumentTypeError(PluginRuntimeError): message (str): A user-readable message describing the exception. """ - def __init__( - self, - parameter_name, - actual_type, - expected_type, - required=True): - actual, expected = self.get_actual_and_expected_type( - actual_type, expected_type) + def __init__(self, parameter_name, actual_type, expected_type, required=True): + actual, expected = self.get_actual_and_expected_type(actual_type, expected_type) - # Get the name of the function that is throwning this error. + # Get the name of the function that is throwing this error. func_name = sys._getframe(1).f_code.co_name message = ("The function {}'s argument '{}' was {} but should" " be of {}{}.".format( - func_name, - parameter_name, - actual, - expected, - (' if defined', '')[required])) + func_name, + parameter_name, + actual, + expected, + (' if defined', '')[required])) super(IncorrectArgumentTypeError, self).__init__(message) diff --git a/libs/src/main/python/dlpx/virtualization/libs/libs.py b/libs/src/main/python/dlpx/virtualization/libs/libs.py index 69355a4b..50b57d5c 100644 --- a/libs/src/main/python/dlpx/virtualization/libs/libs.py +++ b/libs/src/main/python/dlpx/virtualization/libs/libs.py @@ -1,4 +1,4 @@ -# Copyright (c) 2019 by Delphix. All rights reserved. +# Copyright (c) 2019, 2021 by Delphix. All rights reserved. # # -*- coding: utf-8 -*- @@ -33,10 +33,12 @@ from dlpx.virtualization.common._common_classes import (RemoteConnection, PasswordCredentials, KeyPairCredentials) +from dlpx.virtualization.common.util import response_to_str, to_str from google.protobuf import json_format from google.protobuf.struct_pb2 import Struct import logging +import six __all__ = [ @@ -74,23 +76,24 @@ def _handle_response(response): def _check_exit_code(response, check): - """ - This functions checks the exitcode received in response and throws PluginScriptError - if check is True. - - Args: - response (RunPowerShellResponse or RunBashResponse or RunExpectResponse): Response received by run_bash or - run_powershell or run_expect - check (bool): if True and non-zero exitcode is received in response, raise PluginScriptError - """ - if (check and response.HasField('return_value') - and response.return_value.exit_code != 0): - raise PluginScriptError('The script failed with exit code {}.' - ' stdout : {} and ' - ' stderr : {}'.format( - response.return_value.exit_code, - response.return_value.stdout, - response.return_value.stderr)) + """ + This functions checks the exitcode received in response and throws + PluginScriptError if check is True. + + Args: + response (RunPowerShellResponse or RunBashResponse or RunExpectResponse): Response + received by run_bash or run_powershell or run_expect + check (bool): if True and non-zero exitcode is received in response, raise + PluginScriptError + """ + if (check and response.HasField('return_value') + and response.return_value.exit_code != 0): + raise PluginScriptError('The script failed with exit code {}.' + ' stdout : {} and ' + ' stderr : {}'.format( + response.return_value.exit_code, + response.return_value.stdout, + response.return_value.stderr)) def run_bash(remote_connection, command, variables=None, use_login_shell=False, @@ -128,6 +131,8 @@ def run_bash(remote_connection, command, variables=None, use_login_shell=False, if variables is None: variables = {} + command = to_str(command) + variables = to_str(variables) # Validate all the arguments passed in are the right types based on docs. if not isinstance(remote_connection, RemoteConnection): @@ -135,23 +140,23 @@ def run_bash(remote_connection, command, variables=None, use_login_shell=False, 'remote_connection', type(remote_connection), RemoteConnection) - if not isinstance(command, basestring): - raise IncorrectArgumentTypeError('command', type(command), basestring) + if not isinstance(command, six.string_types): + raise IncorrectArgumentTypeError('command', type(command), six.string_types[0]) if variables and not isinstance(variables, dict): raise IncorrectArgumentTypeError( 'variables', type(variables), - {basestring: basestring}, + {six.string_types[0]: six.string_types[0]}, False) - if (variables and (not all(isinstance(variable, basestring) + if (variables and (not all(isinstance(variable, six.string_types) for variable in variables.keys()) or - not all(isinstance(value, basestring) + not all(isinstance(value, six.string_types) for value in variables.values()))): raise IncorrectArgumentTypeError( 'variables', {(type(variable), type(value)) for variable, value in variables.items()}, - {basestring: basestring}, + {six.string_types[0]: six.string_types[0]}, False) if use_login_shell and not isinstance(use_login_shell, bool): raise IncorrectArgumentTypeError( @@ -165,6 +170,7 @@ def run_bash(remote_connection, command, variables=None, use_login_shell=False, run_bash_request.variables[variable] = value run_bash_response = internal_libs.run_bash(run_bash_request) + response_to_str(run_bash_response) _check_exit_code(run_bash_response, check) return _handle_response(run_bash_response) @@ -192,46 +198,54 @@ def run_sync(remote_connection, source_directory, rsync_user=None, from dlpx.virtualization._engine import libs as internal_libs + source_directory = to_str(source_directory) + if rsync_user is not None: + rsync_user = to_str(rsync_user) + if exclude_paths is not None: + exclude_paths = to_str(exclude_paths) + if sym_links_to_follow is not None: + sym_links_to_follow = to_str(sym_links_to_follow) + # Validate all the arguments passed in are the right types based on docs. if not isinstance(remote_connection, RemoteConnection): raise IncorrectArgumentTypeError( 'remote_connection', type(remote_connection), RemoteConnection) - if not isinstance(source_directory, basestring): + if not isinstance(source_directory, six.string_types): raise IncorrectArgumentTypeError( - 'source_directory', type(source_directory), basestring) - if rsync_user and not isinstance(rsync_user, basestring): + 'source_directory', type(source_directory), six.string_types[0]) + if rsync_user and not isinstance(rsync_user, six.string_types): raise IncorrectArgumentTypeError( 'rsync_user', type(rsync_user), - basestring, + six.string_types[0], False) if exclude_paths and not isinstance(exclude_paths, list): raise IncorrectArgumentTypeError( 'exclude_paths', type(exclude_paths), - [basestring], + [six.string_types[0]], False) if (exclude_paths and not all(isinstance( - path, basestring) for path in exclude_paths)): + path, six.string_types) for path in exclude_paths)): raise IncorrectArgumentTypeError( 'exclude_paths', [type(path) for path in exclude_paths], - [basestring], + [six.string_types[0]], False) if sym_links_to_follow and not isinstance(sym_links_to_follow, list): raise IncorrectArgumentTypeError( 'sym_links_to_follow', type(sym_links_to_follow), - [basestring], + [six.string_types[0]], False) - if (sym_links_to_follow and not all(isinstance(link, basestring) + if (sym_links_to_follow and not all(isinstance(link, six.string_types) for link in sym_links_to_follow)): raise IncorrectArgumentTypeError( 'sym_links_to_follow', [type(link) for link in sym_links_to_follow], - [basestring], + [six.string_types[0]], False) run_sync_request = libs_pb2.RunSyncRequest() @@ -245,6 +259,7 @@ def run_sync(remote_connection, source_directory, rsync_user=None, run_sync_request.sym_links_to_follow.extend(sym_links_to_follow) response = internal_libs.run_sync(run_sync_request) + response_to_str(response) _handle_response(response) @@ -282,29 +297,32 @@ def run_powershell(remote_connection, command, variables=None, check=False): if variables is None: variables = {} + command = to_str(command) + variables = to_str(variables) + # Validate all the arguments passed in are the right types based on docs. if not isinstance(remote_connection, RemoteConnection): raise IncorrectArgumentTypeError( 'remote_connection', type(remote_connection), RemoteConnection) - if not isinstance(command, basestring): - raise IncorrectArgumentTypeError('command', type(command), basestring) + if not isinstance(command, six.string_types): + raise IncorrectArgumentTypeError('command', type(command), six.string_types[0]) if variables and not isinstance(variables, dict): raise IncorrectArgumentTypeError( 'variables', type(variables), - {basestring: basestring}, + {six.string_types[0]: six.string_types[0]}, False) - if (variables and (not all(isinstance(variable, basestring) + if (variables and (not all(isinstance(variable, six.string_types) for variable in variables.keys()) or - not all(isinstance(value, basestring) + not all(isinstance(value, six.string_types) for value in variables.values()))): raise IncorrectArgumentTypeError( 'variables', {(type(variable), type(value)) for variable, value in variables.items()}, - {basestring: basestring}, + {six.string_types[0]: six.string_types[0]}, False) run_powershell_request = libs_pb2.RunPowerShellRequest() @@ -314,6 +332,7 @@ def run_powershell(remote_connection, command, variables=None, check=False): run_powershell_request.variables[variable] = value run_powershell_response = internal_libs.run_powershell( run_powershell_request) + response_to_str(run_powershell_response) _check_exit_code(run_powershell_response, check) return _handle_response(run_powershell_response) @@ -343,33 +362,35 @@ def run_expect(remote_connection, command, variables=None, check=False): # scope to allow unit testing of this module. # from dlpx.virtualization._engine import libs as internal_libs - if variables is None: variables = {} + command = to_str(command) + variables = to_str(variables) + # Validate all the arguments passed in are the right types based on docs. if not isinstance(remote_connection, RemoteConnection): raise IncorrectArgumentTypeError( 'remote_connection', type(remote_connection), RemoteConnection) - if not isinstance(command, basestring): - raise IncorrectArgumentTypeError('command', type(command), basestring) + if not isinstance(command, six.string_types): + raise IncorrectArgumentTypeError('command', type(command), six.string_types[0]) if variables and not isinstance(variables, dict): raise IncorrectArgumentTypeError( 'variables', type(variables), - {basestring: basestring}, + {six.string_types[0]: six.string_types[0]}, False) - if (variables and (not all(isinstance(variable, basestring) + if (variables and (not all(isinstance(variable, six.string_types) for variable in variables.keys()) or - not all(isinstance(value, basestring) + not all(isinstance(value, six.string_types) for value in variables.values()))): raise IncorrectArgumentTypeError( 'variables', {(type(variable), type(value)) for variable, value in variables.items()}, - {basestring: basestring}, + {six.string_types[0]: six.string_types[0]}, False) run_expect_request = libs_pb2.RunExpectRequest() @@ -379,6 +400,7 @@ def run_expect(remote_connection, command, variables=None, check=False): run_expect_request.variables[variable] = value run_expect_response = internal_libs.run_expect(run_expect_request) + response_to_str(run_expect_response) _check_exit_code(run_expect_response, check) return _handle_response(run_expect_response) @@ -400,6 +422,7 @@ def _log_request(message, log_level): """ from dlpx.virtualization._engine import libs as internal_libs + message = to_str(message) log_request = libs_pb2.LogRequest() log_request.message = message @@ -415,23 +438,27 @@ def _log_request(message, log_level): log_request.level = libs_pb2.LogRequest.ERROR response = internal_libs.log(log_request) + response_to_str(response) _handle_response(response) def retrieve_credentials(credentials_supplier): - """This is an internal wrapper around the Virtualization library's credentials retrieval API. - Given a supplier provided by Virtualization, retrieves the credentials from that supplier. + """ + This is an internal wrapper around the Virtualization library's credentials + retrieval API. Given a supplier provided by Virtualization, retrieves the + credentials from that supplier. Args: credentials_supplier (dict): Properties that make up a supplier of credentials. Return: - Subclass of Credentials retrieved from supplier. Either a PasswordCredentials or a KeyPairCredentials - from dlpx.virtualization.common._common_classes. + Subclass of Credentials retrieved from supplier. Either a PasswordCredentials + or a KeyPairCredentials from dlpx.virtualization.common._common_classes. """ from dlpx.virtualization._engine import libs as internal_libs if not isinstance(credentials_supplier, dict): - raise IncorrectArgumentTypeError('credentials_supplier', type(credentials_supplier), dict) + raise IncorrectArgumentTypeError( + 'credentials_supplier', type(credentials_supplier), dict) credentials_request = libs_pb2.CredentialsRequest() credentials_struct = Struct() @@ -439,10 +466,11 @@ def retrieve_credentials(credentials_supplier): credentials_request.credentials_supplier.CopyFrom(credentials_struct) response = internal_libs.retrieve_credentials(credentials_request) - + response_to_str(response) credentials_result = _handle_response(response) if credentials_result.password != "": - return PasswordCredentials(credentials_result.username, credentials_result.password) + return PasswordCredentials( + credentials_result.username, credentials_result.password) return KeyPairCredentials( credentials_result.username, credentials_result.key_pair.private_key, @@ -450,22 +478,26 @@ def retrieve_credentials(credentials_supplier): def upgrade_password(password, username=None): - """This is an internal wrapper around Virtualization's credentials-supplier conversion API. - It is intended for use during plugin upgrade when a plugin needs to transform a password - value into a more generic credentials supplier object. + """ + This is an internal wrapper around Virtualization's credentials-supplier conversion + API. It is intended for use during plugin upgrade when a plugin needs to transform + a password value into a more generic credentials supplier object. Args: - password (basestring): Plain password string. - username (basestring, defaults to None): User name contained in the password credential supplier to return. + password (str): Plain password string. + username (str, defaults to None): User name contained in the password + credential supplier to return. Return: Credentials supplier (dict) that supplies the given password and username. """ from dlpx.virtualization._engine import libs as internal_libs - if not isinstance(password, basestring): - raise IncorrectArgumentTypeError('password', type(password), basestring) - if username and not isinstance(username, basestring): - raise IncorrectArgumentTypeError('username', type(username), basestring, required=False) + if not isinstance(password, six.string_types): + raise IncorrectArgumentTypeError( + 'password', type(password), six.string_types[0]) + if username and not isinstance(username, six.string_types): + raise IncorrectArgumentTypeError( + 'username', type(username), six.string_types[0], required=False) upgrade_password_request = libs_pb2.UpgradePasswordRequest() upgrade_password_request.password = password @@ -473,6 +505,6 @@ def upgrade_password(password, username=None): upgrade_password_request.username = username response = internal_libs.upgrade_password(upgrade_password_request) - + response_to_str(response) upgrade_password_result = _handle_response(response) return json_format.MessageToDict(upgrade_password_result.credentials_supplier) diff --git a/libs/src/test/python/dlpx/virtualization/_engine/libs.py b/libs/src/test/python/dlpx/virtualization/_engine/libs.py index 99356f49..bb0d58dc 100644 --- a/libs/src/test/python/dlpx/virtualization/_engine/libs.py +++ b/libs/src/test/python/dlpx/virtualization/_engine/libs.py @@ -1,5 +1,5 @@ # -# Copyright (c) 2019 by Delphix. All rights reserved. +# Copyright (c) 2019, 2021 by Delphix. All rights reserved. # """ @@ -25,4 +25,4 @@ def run_expect(run_expect_request): def log(log_debug_request): - pass \ No newline at end of file + pass diff --git a/libs/src/test/python/dlpx/virtualization/conftest.py b/libs/src/test/python/dlpx/virtualization/conftest.py index 4c805eb0..52e16d2b 100644 --- a/libs/src/test/python/dlpx/virtualization/conftest.py +++ b/libs/src/test/python/dlpx/virtualization/conftest.py @@ -1,9 +1,10 @@ # -# Copyright (c) 2019 by Delphix. All rights reserved. +# Copyright (c) 2019, 2021 by Delphix. All rights reserved. # import pytest -from dlpx.virtualization.common._common_classes import RemoteUser, RemoteHost, RemoteEnvironment, RemoteConnection +from dlpx.virtualization.common._common_classes import ( + RemoteUser, RemoteHost, RemoteEnvironment, RemoteConnection) @pytest.fixture diff --git a/libs/src/test/python/dlpx/virtualization/test_libs.py b/libs/src/test/python/dlpx/virtualization/test_libs.py index 4e94d375..abf8eba0 100644 --- a/libs/src/test/python/dlpx/virtualization/test_libs.py +++ b/libs/src/test/python/dlpx/virtualization/test_libs.py @@ -1,16 +1,16 @@ # -# Copyright (c) 2019, 2020 by Delphix. All rights reserved. +# Copyright (c) 2019, 2021 by Delphix. All rights reserved. # import mock import pytest +import six from dlpx.virtualization.api import libs_pb2 from dlpx.virtualization import libs from dlpx.virtualization.libs.exceptions import ( IncorrectArgumentTypeError, LibraryError, PluginScriptError) from google.protobuf import json_format -from dlpx.virtualization.common._common_classes import (PasswordCredentials) class TestLibsRunBash: @@ -53,10 +53,10 @@ def mock_run_bash(actual_run_bash_request): @staticmethod def test_run_bash_check_true_success_exitcode(remote_connection): - expected_run_bash_response = libs_pb2.RunBashResponse() - expected_run_bash_response.return_value.exit_code = 0 - expected_run_bash_response.return_value.stdout = "stdout" - expected_run_bash_response.return_value.stderr = "stderr" + expected_response = libs_pb2.RunBashResponse() + expected_response.return_value.exit_code = 0 + expected_response.return_value.stdout = "stdout" + expected_response.return_value.stderr = "stderr" expected_command = "command" expected_variables = None @@ -73,19 +73,17 @@ def mock_run_bash(actual_run_bash_request): actual_run_bash_request.remote_connection.environment.reference == remote_connection.environment.reference ) - return expected_run_bash_response + return expected_response with mock.patch("dlpx.virtualization._engine.libs.run_bash", side_effect=mock_run_bash, create=True): - actual_run_bash_result = libs.run_bash(remote_connection, - expected_command, - expected_variables, - expected_use_login_shell, - check=True) + actual_result = libs.run_bash( + remote_connection, expected_command, expected_variables, + expected_use_login_shell, check=True) - assert actual_run_bash_result.exit_code == expected_run_bash_response.return_value.exit_code - assert actual_run_bash_result.stdout == expected_run_bash_response.return_value.stdout - assert actual_run_bash_result.stderr == expected_run_bash_response.return_value.stderr + assert actual_result.exit_code == expected_response.return_value.exit_code + assert actual_result.stdout == expected_response.return_value.stdout + assert actual_result.stderr == expected_response.return_value.stderr @staticmethod def test_run_bash_with_check_true_failed_exitcode(remote_connection): @@ -143,11 +141,16 @@ def test_run_bash_bad_remote_connection(): with pytest.raises(IncorrectArgumentTypeError) as err_info: libs.run_bash(connection, command, variables, use_login_shell) - - assert err_info.value.message == ( - "The function run_bash's argument 'remote_connection' was" - " type 'str' but should be of" - " class 'dlpx.virtualization.common._common_classes.RemoteConnection'.") + if six.PY2: + assert err_info.value.message == ( + "The function run_bash's argument 'remote_connection' was" + " type 'str' but should be of" + " class 'dlpx.virtualization.common._common_classes.RemoteConnection'.") + else: + assert err_info.value.message == ( + "The function run_bash's argument 'remote_connection' was" + " class 'str' but should be of" + " class 'dlpx.virtualization.common._common_classes.RemoteConnection'.") @staticmethod def test_run_bash_bad_command(remote_connection): @@ -158,10 +161,14 @@ def test_run_bash_bad_command(remote_connection): with pytest.raises(IncorrectArgumentTypeError) as err_info: libs.run_bash(remote_connection, command, variables, use_login_shell) - - assert err_info.value.message == ( - "The function run_bash's argument 'command' was" - " type 'int' but should be of type 'basestring'.") + if six.PY2: + assert err_info.value.message == ( + "The function run_bash's argument 'command' was" + " type 'int' but should be of type 'basestring'.") + else: + assert err_info.value.message == ( + "The function run_bash's argument 'command' was" + " class 'int' but should be of class 'str'.") @staticmethod def test_run_bash_variables_not_dict(remote_connection): @@ -172,11 +179,16 @@ def test_run_bash_variables_not_dict(remote_connection): with pytest.raises(IncorrectArgumentTypeError) as err_info: libs.run_bash(remote_connection, command, variables, use_login_shell) - - assert err_info.value.message == ( - "The function run_bash's argument 'variables' was" - " type 'str' but should be of" - " type 'dict of basestring:basestring' if defined.") + if six.PY2: + assert err_info.value.message == ( + "The function run_bash's argument 'variables' was" + " type 'unicode' but should be of" + " type 'dict of basestring:basestring' if defined.") + else: + assert err_info.value.message == ( + "The function run_bash's argument 'variables' was" + " class 'str' but should be of" + " type 'dict of str:str' if defined.") @staticmethod def test_run_bash_bad_variables(remote_connection): @@ -190,13 +202,20 @@ def test_run_bash_bad_variables(remote_connection): with pytest.raises(IncorrectArgumentTypeError) as err_info: libs.run_bash(remote_connection, command, variables, use_login_shell) - - message = ("The function run_bash's argument 'variables' was" - " a dict of {{type 'str':type '{}', type 'str':type '{}'}}" - " but should be of" - " type 'dict of basestring:basestring' if defined.") - assert (err_info.value.message == message.format('int', 'str') or - err_info.value.message == message.format('str', 'int')) + if six.PY2: + message = ("The function run_bash's argument 'variables' was" + " a dict of {{type 'str':type '{}', type 'str':type '{}'}}" + " but should be of" + " type 'dict of basestring:basestring' if defined.") + assert (err_info.value.message == message.format('int', 'unicode') or + err_info.value.message == message.format('unicode', 'int')) + else: + message = ("The function run_bash's argument 'variables' was" + " a dict of {{class 'str':class '{}', class 'str':class '{}'}}" + " but should be of" + " type 'dict of str:str' if defined.") + assert (err_info.value.message == message.format('int', 'str') or + err_info.value.message == message.format('str', 'int')) @staticmethod def test_run_bash_bad_use_login_shell(remote_connection): @@ -207,10 +226,14 @@ def test_run_bash_bad_use_login_shell(remote_connection): with pytest.raises(IncorrectArgumentTypeError) as err_info: libs.run_bash(remote_connection, command, variables, use_login_shell) - - assert err_info.value.message == ( - "The function run_bash's argument 'use_login_shell' was" - " type 'str' but should be of type 'bool' if defined.") + if six.PY2: + assert err_info.value.message == ( + "The function run_bash's argument 'use_login_shell' was" + " type 'str' but should be of type 'bool' if defined.") + else: + assert err_info.value.message == ( + "The function run_bash's argument 'use_login_shell' was" + " class 'str' but should be of class 'bool' if defined.") class TestLibsRunSync: @@ -296,11 +319,16 @@ def test_run_sync_bad_remote_connection(): rsync_user, exclude_paths, sym_links_to_follow) - - assert err_info.value.message == ( - "The function run_sync's argument 'remote_connection' was" - " type 'str' but should be of" - " class 'dlpx.virtualization.common._common_classes.RemoteConnection'.") + if six.PY2: + assert err_info.value.message == ( + "The function run_sync's argument 'remote_connection' was" + " type 'str' but should be of" + " class 'dlpx.virtualization.common._common_classes.RemoteConnection'.") + else: + assert err_info.value.message == ( + "The function run_sync's argument 'remote_connection' was" + " class 'str' but should be of" + " class 'dlpx.virtualization.common._common_classes.RemoteConnection'.") @staticmethod def test_run_sync_bad_source_directory(remote_connection): @@ -317,10 +345,14 @@ def test_run_sync_bad_source_directory(remote_connection): rsync_user, exclude_paths, sym_links_to_follow) - - assert err_info.value.message == ( - "The function run_sync's argument 'source_directory' was" - " type 'int' but should be of type 'basestring'.") + if six.PY2: + assert err_info.value.message == ( + "The function run_sync's argument 'source_directory' was" + " type 'int' but should be of type 'basestring'.") + else: + assert err_info.value.message == ( + "The function run_sync's argument 'source_directory' was" + " class 'int' but should be of class 'str'.") @staticmethod def test_run_sync_bad_rsync_user(remote_connection): @@ -337,10 +369,14 @@ def test_run_sync_bad_rsync_user(remote_connection): rsync_user, exclude_paths, sym_links_to_follow) - - assert err_info.value.message == ( - "The function run_sync's argument 'rsync_user' was" - " type 'int' but should be of type 'basestring' if defined.") + if six.PY2: + assert err_info.value.message == ( + "The function run_sync's argument 'rsync_user' was" + " type 'int' but should be of type 'basestring' if defined.") + else: + assert err_info.value.message == ( + "The function run_sync's argument 'rsync_user' was" + " class 'int' but should be of class 'str' if defined.") @staticmethod def test_run_sync_exclude_paths_not_list(remote_connection): @@ -357,11 +393,16 @@ def test_run_sync_exclude_paths_not_list(remote_connection): rsync_user, exclude_paths, sym_links_to_follow) - - assert err_info.value.message == ( - "The function run_sync's argument 'exclude_paths' was" - " type 'str' but should be of" - " type 'list of basestring' if defined.") + if six.PY2: + assert err_info.value.message == ( + "The function run_sync's argument 'exclude_paths' was" + " type 'unicode' but should be of" + " type 'list of basestring' if defined.") + else: + assert err_info.value.message == ( + "The function run_sync's argument 'exclude_paths' was" + " class 'str' but should be of" + " type 'list of str' if defined.") @staticmethod def test_run_sync_bad_exclude_paths(remote_connection): @@ -378,11 +419,16 @@ def test_run_sync_bad_exclude_paths(remote_connection): rsync_user, exclude_paths, sym_links_to_follow) - - assert err_info.value.message == ( - "The function run_sync's argument 'exclude_paths' was a list of" - " [type 'str', type 'int'] but should be of" - " type 'list of basestring' if defined.") + if six.PY2: + assert err_info.value.message == ( + "The function run_sync's argument 'exclude_paths' was a list of" + " [type 'unicode', type 'int'] but should be of" + " type 'list of basestring' if defined.") + else: + assert err_info.value.message == ( + "The function run_sync's argument 'exclude_paths' was a list of" + " [class 'str', class 'int'] but should be of" + " type 'list of str' if defined.") @staticmethod def test_run_sync_sym_links_to_follow_not_list(remote_connection): @@ -399,11 +445,16 @@ def test_run_sync_sym_links_to_follow_not_list(remote_connection): rsync_user, exclude_paths, sym_links_to_follow) - - assert err_info.value.message == ( - "The function run_sync's argument 'sym_links_to_follow' was" - " type 'str' but should be of" - " type 'list of basestring' if defined.") + if six.PY2: + assert err_info.value.message == ( + "The function run_sync's argument 'sym_links_to_follow' was" + " type 'unicode' but should be of" + " type 'list of basestring' if defined.") + else: + assert err_info.value.message == ( + "The function run_sync's argument 'sym_links_to_follow' was" + " class 'str' but should be of" + " type 'list of str' if defined.") @staticmethod def test_run_sync_bad_sym_links_to_follow(remote_connection): @@ -420,11 +471,18 @@ def test_run_sync_bad_sym_links_to_follow(remote_connection): rsync_user, exclude_paths, sym_links_to_follow) - - assert err_info.value.message == ( - "The function run_sync's argument 'sym_links_to_follow' was" - " a list of [type 'str', type 'int'] but should be of" - " type 'list of basestring' if defined.") + if six.PY2: + assert err_info.value.message == ( + "The function run_sync's argument 'sym_links_to_follow' was" + " a list of [type 'unicode', type 'int'] but should be of" + " type 'list of " + "basestring' if defined.") + else: + assert err_info.value.message == ( + "The function run_sync's argument 'sym_links_to_follow' was" + " a list of [class 'str', class 'int'] but should be of" + " type 'list of " + "str' if defined.") class TestLibsRunPowershell: @@ -463,35 +521,35 @@ def mock_run_powershell(actual_run_powershell_request): @staticmethod def test_run_powershell_check_true_exitcode_success(remote_connection): - expected_run_powershell_response = libs_pb2.RunPowerShellResponse() - expected_run_powershell_response.return_value.exit_code = 0 - expected_run_powershell_response.return_value.stdout = "stdout" - expected_run_powershell_response.return_value.stderr = "stderr" + expected_response = libs_pb2.RunPowerShellResponse() + expected_response.return_value.exit_code = 0 + expected_response.return_value.stdout = "stdout" + expected_response.return_value.stderr = "stderr" expected_command = "command" expected_variables = None - def mock_run_powershell(actual_run_powershell_request): - assert actual_run_powershell_request.command == expected_command + def mock_run_powershell(actual_request): + assert actual_request.command == expected_command assert ( - actual_run_powershell_request.remote_connection.environment.name + actual_request.remote_connection.environment.name == remote_connection.environment.name ) assert ( - actual_run_powershell_request.remote_connection.environment.reference + actual_request.remote_connection.environment.reference == remote_connection.environment.reference ) - return expected_run_powershell_response + return expected_response with mock.patch("dlpx.virtualization._engine.libs.run_powershell", side_effect=mock_run_powershell, create=True): - actual_run_powershell_result = libs.run_powershell( + actual_result = libs.run_powershell( remote_connection, expected_command, expected_variables, check=True) - assert actual_run_powershell_result.exit_code == expected_run_powershell_response.return_value.exit_code - assert actual_run_powershell_result.stdout == expected_run_powershell_response.return_value.stdout - assert actual_run_powershell_result.stderr == expected_run_powershell_response.return_value.stderr + assert actual_result.exit_code == expected_response.return_value.exit_code + assert actual_result.stdout == expected_response.return_value.stdout + assert actual_result.stderr == expected_response.return_value.stderr @staticmethod def test_run_powershell_check_true_exitcode_failed(remote_connection): @@ -508,8 +566,8 @@ def test_run_powershell_check_true_exitcode_failed(remote_connection): with mock.patch("dlpx.virtualization._engine.libs.run_powershell", return_value=response, create=True): with pytest.raises(PluginScriptError) as info: - response = libs.run_powershell(remote_connection, "test_command", - check=True) + libs.run_powershell(remote_connection, "test_command", + check=True) assert info.value.message == expected_message @staticmethod @@ -549,11 +607,16 @@ def test_run_powershell_bad_remote_connection(): with pytest.raises(IncorrectArgumentTypeError) as err_info: libs.run_powershell(connection, command, variables) - - assert err_info.value.message == ( - "The function run_powershell's argument 'remote_connection' was" - " type 'str' but should be of" - " class 'dlpx.virtualization.common._common_classes.RemoteConnection'.") + if six.PY2: + assert err_info.value.message == ( + "The function run_powershell's argument 'remote_connection' was" + " type 'str' but should be of" + " class 'dlpx.virtualization.common._common_classes.RemoteConnection'.") + else: + assert err_info.value.message == ( + "The function run_powershell's argument 'remote_connection' was" + " class 'str' but should be of" + " class 'dlpx.virtualization.common._common_classes.RemoteConnection'.") @staticmethod def test_run_powershell_bad_command(remote_connection): @@ -563,10 +626,14 @@ def test_run_powershell_bad_command(remote_connection): with pytest.raises(IncorrectArgumentTypeError) as err_info: libs.run_powershell(remote_connection, command, variables) - - assert err_info.value.message == ( - "The function run_powershell's argument 'command' was" - " type 'int' but should be of type 'basestring'.") + if six.PY2: + assert err_info.value.message == ( + "The function run_powershell's argument 'command' was" + " type 'int' but should be of type 'basestring'.") + else: + assert err_info.value.message == ( + "The function run_powershell's argument 'command' was" + " class 'int' but should be of class 'str'.") @staticmethod def test_run_powershell_variables_not_dict(remote_connection): @@ -576,11 +643,16 @@ def test_run_powershell_variables_not_dict(remote_connection): with pytest.raises(IncorrectArgumentTypeError) as err_info: libs.run_powershell(remote_connection, command, variables) - - assert err_info.value.message == ( - "The function run_powershell's argument 'variables' was" - " type 'str' but should be of" - " type 'dict of basestring:basestring' if defined.") + if six.PY2: + assert err_info.value.message == ( + "The function run_powershell's argument 'variables' was" + " type 'unicode' but should be of" + " type 'dict of basestring:basestring' if defined.") + else: + assert err_info.value.message == ( + "The function run_powershell's argument 'variables' was" + " class 'str' but should be of" + " type 'dict of str:str' if defined.") @staticmethod def test_run_powershell_bad_variables(remote_connection): @@ -593,13 +665,20 @@ def test_run_powershell_bad_variables(remote_connection): with pytest.raises(IncorrectArgumentTypeError) as err_info: libs.run_powershell(remote_connection, command, variables) - - message = ("The function run_powershell's argument 'variables' was" - " a dict of {{type 'str':type '{}', type 'str':type '{}'}}" - " but should be of" - " type 'dict of basestring:basestring' if defined.") - assert (err_info.value.message == message.format('int', 'str') or - err_info.value.message == message.format('str', 'int')) + if six.PY2: + message = ("The function run_powershell's argument 'variables' was" + " a dict of {{type 'str':type '{}', type 'str':type '{}'}}" + " but should be of" + " type 'dict of basestring:basestring' if defined.") + assert (err_info.value.message == message.format('int', 'unicode') or + err_info.value.message == message.format('unicode', 'int')) + else: + message = ("The function run_powershell's argument 'variables' was" + " a dict of {{class 'str':class '{}', class 'str':class '{}'}}" + " but should be of" + " type 'dict of str:str' if defined.") + assert (err_info.value.message == message.format('int', 'str') or + err_info.value.message == message.format('str', 'int')) class TestLibsRunExpect: @@ -638,10 +717,10 @@ def mock_run_expect(actual_run_expect_request): @staticmethod def test_run_expect_check_true_exitcode_success(remote_connection): - expected_run_expect_response = libs_pb2.RunPowerShellResponse() - expected_run_expect_response.return_value.exit_code = 0 - expected_run_expect_response.return_value.stdout = "stdout" - expected_run_expect_response.return_value.stderr = "stderr" + expected_response = libs_pb2.RunPowerShellResponse() + expected_response.return_value.exit_code = 0 + expected_response.return_value.stdout = "stdout" + expected_response.return_value.stderr = "stderr" expected_command = "command" expected_variables = None @@ -656,17 +735,17 @@ def mock_run_expect(actual_run_expect_request): actual_run_expect_request.remote_connection.environment.reference == remote_connection.environment.reference ) - return expected_run_expect_response + return expected_response with mock.patch("dlpx.virtualization._engine.libs.run_expect", side_effect=mock_run_expect, create=True): - actual_run_expect_result = libs.run_expect( + actual_result = libs.run_expect( remote_connection, expected_command, expected_variables, check=True) - assert actual_run_expect_result.exit_code == expected_run_expect_response.return_value.exit_code - assert actual_run_expect_result.stdout == expected_run_expect_response.return_value.stdout - assert actual_run_expect_result.stderr == expected_run_expect_response.return_value.stderr + assert actual_result.exit_code == expected_response.return_value.exit_code + assert actual_result.stdout == expected_response.return_value.stdout + assert actual_result.stderr == expected_response.return_value.stderr @staticmethod def test_run_expect_check_true_exitcode_failed(remote_connection): @@ -683,8 +762,7 @@ def test_run_expect_check_true_exitcode_failed(remote_connection): with mock.patch("dlpx.virtualization._engine.libs.run_expect", return_value=response, create=True): with pytest.raises(PluginScriptError) as info: - response = libs.run_expect(remote_connection, "test_command", - check=True) + libs.run_expect(remote_connection, "test_command", check=True) assert info.value.message == expected_message @staticmethod @@ -724,11 +802,16 @@ def test_run_expect_bad_remote_connection(): with pytest.raises(IncorrectArgumentTypeError) as err_info: libs.run_expect(connection, command, variables) - - assert err_info.value.message == ( - "The function run_expect's argument 'remote_connection' was" - " type 'str' but should be of" - " class 'dlpx.virtualization.common._common_classes.RemoteConnection'.") + if six.PY2: + assert err_info.value.message == ( + "The function run_expect's argument 'remote_connection' was" + " type 'str' but should be of" + " class 'dlpx.virtualization.common._common_classes.RemoteConnection'.") + else: + assert err_info.value.message == ( + "The function run_expect's argument 'remote_connection' was" + " class 'str' but should be of" + " class 'dlpx.virtualization.common._common_classes.RemoteConnection'.") @staticmethod def test_run_expect_bad_command(remote_connection): @@ -739,9 +822,14 @@ def test_run_expect_bad_command(remote_connection): with pytest.raises(IncorrectArgumentTypeError) as err_info: libs.run_expect(remote_connection, command, variables) - assert err_info.value.message == ( - "The function run_expect's argument 'command' was" - " type 'int' but should be of type 'basestring'.") + if six.PY2: + assert err_info.value.message == ( + "The function run_expect's argument 'command' was" + " type 'int' but should be of type 'basestring'.") + else: + assert err_info.value.message == ( + "The function run_expect's argument 'command' was" + " class 'int' but should be of class 'str'.") @staticmethod def test_run_expect_variables_not_dict(remote_connection): @@ -751,11 +839,16 @@ def test_run_expect_variables_not_dict(remote_connection): with pytest.raises(IncorrectArgumentTypeError) as err_info: libs.run_expect(remote_connection, command, variables) - - assert err_info.value.message == ( - "The function run_expect's argument 'variables' was" - " type 'str' but should be of" - " type 'dict of basestring:basestring' if defined.") + if six.PY2: + assert err_info.value.message == ( + "The function run_expect's argument 'variables' was" + " type 'unicode' but should be of" + " type 'dict of basestring:basestring' if defined.") + else: + assert err_info.value.message == ( + "The function run_expect's argument 'variables' was" + " class 'str' but should be of" + " type 'dict of str:str' if defined.") @staticmethod def test_run_expect_bad_variables(remote_connection): @@ -769,60 +862,72 @@ def test_run_expect_bad_variables(remote_connection): with pytest.raises(IncorrectArgumentTypeError) as err_info: libs.run_expect(remote_connection, command, variables) - message = ("The function run_expect's argument 'variables' was" - " a dict of {{type 'str':type '{}', type 'str':type '{}'}}" - " but should be of" - " type 'dict of basestring:basestring' if defined.") - assert (err_info.value.message == message.format('int', 'str') or - err_info.value.message == message.format('str', 'int')) + if six.PY2: + message = ("The function run_expect's argument 'variables' was" + " a dict of {{type 'str':type '{}', type 'str':type '{}'}}" + " but should be of" + " type 'dict of basestring:basestring' if defined.") + assert (err_info.value.message == message.format('int', 'unicode') or + err_info.value.message == message.format('unicode', 'int')) + else: + message = ("The function run_expect's argument 'variables' was" + " a dict of {{class 'str':class '{}', class 'str':class '{}'}}" + " but should be of" + " type 'dict of str:str' if defined.") + assert (err_info.value.message == message.format('int', 'str') or + err_info.value.message == message.format('str', 'int')) class TestLibsRetrieveCredentials: @staticmethod def test_retrieve_password_credentials(): - expected_retrieve_credentials_response = libs_pb2.CredentialsResponse() - expected_retrieve_credentials_response.return_value.username = 'some user' - expected_retrieve_credentials_response.return_value.password = 'some password' + expected_response = libs_pb2.CredentialsResponse() + expected_response.return_value.username = 'some user' + expected_response.return_value.password = 'some password' - expected_credentials_supplier = {'some supplier property': 'some supplier value'} + expected_credentials_supplier = { + 'some supplier property': 'some supplier value'} def mock_retrieve_credentials(actual_retrieve_credentials_request): - assert json_format.MessageToDict(actual_retrieve_credentials_request.credentials_supplier) == \ - expected_credentials_supplier + assert json_format.MessageToDict( + actual_retrieve_credentials_request.credentials_supplier + ) == expected_credentials_supplier - return expected_retrieve_credentials_response + return expected_response with mock.patch('dlpx.virtualization._engine.libs.retrieve_credentials', side_effect=mock_retrieve_credentials, create=True): - actual_retrieve_credentials_result = libs.retrieve_credentials(expected_credentials_supplier) + actual_result = libs.retrieve_credentials(expected_credentials_supplier) - expected = expected_retrieve_credentials_response.return_value - assert actual_retrieve_credentials_result.username == expected.username - assert actual_retrieve_credentials_result.password == expected.password + expected = expected_response.return_value + assert actual_result.username == expected.username + assert actual_result.password == expected.password @staticmethod def test_retrieve_keypair_credentials(): - expected_retrieve_credentials_response = libs_pb2.CredentialsResponse() - expected_retrieve_credentials_response.return_value.username = 'some user' - expected_retrieve_credentials_response.return_value.key_pair.private_key = 'some private key' - expected_retrieve_credentials_response.return_value.key_pair.public_key = 'some public key' + expected_response = libs_pb2.CredentialsResponse() + expected_response.return_value.username = 'some user' + expected_response.return_value.key_pair.private_key = 'some private key' + expected_response.return_value.key_pair.public_key = 'some public key' - expected_credentials_supplier = {'some supplier property': 'some supplier value'} + expected_credentials_supplier = { + 'some supplier property': 'some supplier value'} def mock_retrieve_credentials(actual_retrieve_credentials_request): - assert json_format.MessageToDict(actual_retrieve_credentials_request.credentials_supplier) == \ - expected_credentials_supplier + assert json_format.MessageToDict( + actual_retrieve_credentials_request.credentials_supplier + ) == expected_credentials_supplier - return expected_retrieve_credentials_response + return expected_response with mock.patch('dlpx.virtualization._engine.libs.retrieve_credentials', side_effect=mock_retrieve_credentials, create=True): - actual_retrieve_credentials_result = libs.retrieve_credentials(expected_credentials_supplier) + actual_result = libs.retrieve_credentials(expected_credentials_supplier) - expected = expected_retrieve_credentials_response.return_value - assert actual_retrieve_credentials_result.username == expected.username - assert actual_retrieve_credentials_result.private_key == expected.key_pair.private_key - assert actual_retrieve_credentials_result.public_key == expected.key_pair.public_key + expected = expected_response.return_value + assert actual_result.username == expected.username + assert actual_result.private_key == expected.key_pair.private_key + assert actual_result.public_key == expected.key_pair.public_key @staticmethod def test_retrieve_credentials_with_actionable_error(): @@ -836,7 +941,8 @@ def test_retrieve_credentials_with_actionable_error(): with mock.patch('dlpx.virtualization._engine.libs.retrieve_credentials', return_value=response, create=True): with pytest.raises(LibraryError) as err_info: - libs.retrieve_credentials({'some supplier property': 'some supplier value'}) + libs.retrieve_credentials( + {'some supplier property': 'some supplier value'}) assert err_info.value._id == expected_id assert err_info.value.message == expected_message @@ -850,7 +956,8 @@ def test_retrieve_credentials_with_nonactionable_error(): with mock.patch('dlpx.virtualization._engine.libs.retrieve_credentials', return_value=response, create=True): with pytest.raises(SystemExit): - libs.retrieve_credentials({'some supplier property': 'some supplier value'}) + libs.retrieve_credentials( + {'some supplier property': 'some supplier value'}) @staticmethod def test_retrieve_credentials_bad_supplier(): @@ -859,10 +966,14 @@ def test_retrieve_credentials_bad_supplier(): with pytest.raises(IncorrectArgumentTypeError) as err_info: libs.retrieve_credentials(credentials_supplier) - - assert err_info.value.message == ( - "The function retrieve_credentials's argument 'credentials_supplier' was" - " type 'int' but should be of type 'dict'.") + if six.PY2: + assert err_info.value.message == ( + "The function retrieve_credentials's argument 'credentials_supplier' " + "was type 'int' but should be of type 'dict'.") + else: + assert err_info.value.message == ( + "The function retrieve_credentials's argument 'credentials_supplier' " + "was class 'int' but should be of class 'dict'.") class TestLibsUpgradePassword: @@ -870,9 +981,11 @@ class TestLibsUpgradePassword: def test_upgrade_password(): expected_password = 'some password' - expected_credentials_supplier = {'type': 'NamedPasswordCredential', 'password': expected_password} + expected_credentials_supplier = { + 'type': 'NamedPasswordCredential', 'password': expected_password} expected_upgrade_password_response = libs_pb2.UpgradePasswordResponse() - expected_upgrade_password_response.return_value.credentials_supplier.update(expected_credentials_supplier) + expected_upgrade_password_response.return_value.credentials_supplier.update( + expected_credentials_supplier) def mock_upgrade_password(actual_upgrade_password_request): assert actual_upgrade_password_request.password == expected_password @@ -890,9 +1003,12 @@ def test_upgrade_password_with_username(): expected_password = 'some password' expected_username = 'some user name' - expected_credentials_supplier = {'type': 'NamedPasswordCredential', 'password': expected_password, 'username': expected_username} + expected_credentials_supplier = { + 'type': 'NamedPasswordCredential', 'password': expected_password, + 'username': expected_username} expected_upgrade_password_response = libs_pb2.UpgradePasswordResponse() - expected_upgrade_password_response.return_value.credentials_supplier.update(expected_credentials_supplier) + expected_upgrade_password_response.return_value.credentials_supplier.update( + expected_credentials_supplier) def mock_upgrade_password(actual_upgrade_password_request): assert actual_upgrade_password_request.password == expected_password @@ -902,7 +1018,8 @@ def mock_upgrade_password(actual_upgrade_password_request): with mock.patch('dlpx.virtualization._engine.libs.upgrade_password', side_effect=mock_upgrade_password, create=True): - actual_upgrade_password_result = libs.upgrade_password(expected_password, username=expected_username) + actual_upgrade_password_result = libs.upgrade_password( + expected_password, username=expected_username) assert actual_upgrade_password_result == expected_credentials_supplier @@ -913,10 +1030,14 @@ def test_upgrade_password_invalid_password(): with pytest.raises(IncorrectArgumentTypeError) as err_info: libs.upgrade_password(expected_password, username=expected_username) - - assert err_info.value.message == ( - "The function upgrade_password's argument 'password' was" - " type 'int' but should be of type 'basestring'.") + if six.PY2: + assert err_info.value.message == ( + "The function upgrade_password's argument 'password' was" + " type 'int' but should be of type 'basestring'.") + else: + assert err_info.value.message == ( + "The function upgrade_password's argument 'password' was" + " class 'int' but should be of class 'str'.") @staticmethod def test_upgrade_password_invalid_username(): @@ -925,7 +1046,11 @@ def test_upgrade_password_invalid_username(): with pytest.raises(IncorrectArgumentTypeError) as err_info: libs.upgrade_password(expected_password, username=expected_username) - - assert err_info.value.message == ( - "The function upgrade_password's argument 'username' was" - " type 'int' but should be of type 'basestring' if defined.") + if six.PY2: + assert err_info.value.message == ( + "The function upgrade_password's argument 'username' was" + " type 'int' but should be of type 'basestring' if defined.") + else: + assert err_info.value.message == ( + "The function upgrade_password's argument 'username' was" + " class 'int' but should be of class 'str' if defined.") diff --git a/libs/src/test/python/dlpx/virtualization/test_logging.py b/libs/src/test/python/dlpx/virtualization/test_logging.py index 022cffa8..45a4ddc2 100644 --- a/libs/src/test/python/dlpx/virtualization/test_logging.py +++ b/libs/src/test/python/dlpx/virtualization/test_logging.py @@ -1,5 +1,5 @@ # -# Copyright (c) 2019 by Delphix. All rights reserved. +# Copyright (c) 2019, 2021 by Delphix. All rights reserved. # import logging @@ -30,7 +30,6 @@ def successful_response(): (logging.ERROR, LogRequest.ERROR), (logging.CRITICAL, LogRequest.ERROR) ]) - @mock.patch("dlpx.virtualization._engine.libs", create=True) def test_levels(mock_internal_libs, py_level, expected_level, successful_response): mock_internal_libs.log.return_value = successful_response @@ -67,7 +66,6 @@ def test_format(mock_internal_libs, successful_response): mock_internal_libs.log.assert_called_with(log_request) - @staticmethod @mock.patch("dlpx.virtualization._engine.libs", create=True) def test_log_non_string(mock_internal_libs, successful_response): diff --git a/platform/.python-version b/platform/.python-version deleted file mode 100644 index 43c4dbe6..00000000 --- a/platform/.python-version +++ /dev/null @@ -1 +0,0 @@ -2.7.17 diff --git a/platform/setup.cfg b/platform/setup.cfg index 1e4a607a..5e67bf3e 100644 --- a/platform/setup.cfg +++ b/platform/setup.cfg @@ -1,21 +1,22 @@ # -# Copyright (c) 2019 by Delphix. All rights reserved. +# Copyright (c) 2019, 2021 by Delphix. All rights reserved. # [metadata] -Metadata-Version: 1.2 -Author: Delphix -Author-email: virtualization-plugins@delphix.com -Home-page: https://developer.delphix.com -Summary: Delphix Virtualization Platform APIs -Long-description: file: README.md -Long-description-content-type: text/markdown -Classifiers: +metadata_version: 1.2 +author: Delphix +author_email: virtualization-plugins@delphix.com +home_page: https://developer.delphix.com +summary: Delphix Virtualization Platform APIs +long_description: file: README.md +long_description_content_type: text/markdown +classifiers: Development Status :: 5 - Production/Stable Programming Language :: Python Programming Language :: Python :: 2.7 + Programming Language :: Python :: 3.8 License :: OSI Approved :: Apache Software License Operating System :: OS Independent [options] -Requires-Python: 2.7 +requires_python: >=2.7, <=3.8, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*, !=3.6.*, !=3.7.* diff --git a/platform/setup.py b/platform/setup.py index c9ec3a5d..5a696bb3 100644 --- a/platform/setup.py +++ b/platform/setup.py @@ -7,7 +7,7 @@ version = version_file.read().strip() install_requires = [ - "dvp-api == 1.5.0", + "dvp-api == 1.6.1", "dvp-common == {}".format(version), "enum34;python_version < '3.4'", ] @@ -17,4 +17,5 @@ install_requires=install_requires, package_dir={'': PYTHON_SRC}, packages=setuptools.find_packages(PYTHON_SRC), + python_requires='>=2.7, <3.9, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*, !=3.6.*, !=3.7.*', ) diff --git a/platform/src/main/python/dlpx/virtualization/platform/VERSION b/platform/src/main/python/dlpx/virtualization/platform/VERSION index fd2a0186..fcdb2e10 100644 --- a/platform/src/main/python/dlpx/virtualization/platform/VERSION +++ b/platform/src/main/python/dlpx/virtualization/platform/VERSION @@ -1 +1 @@ -3.1.0 +4.0.0 diff --git a/platform/src/main/python/dlpx/virtualization/platform/__init__.py b/platform/src/main/python/dlpx/virtualization/platform/__init__.py index 589f76e4..f22e502c 100644 --- a/platform/src/main/python/dlpx/virtualization/platform/__init__.py +++ b/platform/src/main/python/dlpx/virtualization/platform/__init__.py @@ -1,17 +1,17 @@ # -# Copyright (c) 2019 by Delphix. All rights reserved. +# Copyright (c) 2019, 2021 by Delphix. All rights reserved. # __path__ = __import__('pkgutil').extend_path(__path__, __name__) -from dlpx.virtualization.platform.validation_util import * -from dlpx.virtualization.platform.migration_helper import * -from dlpx.virtualization.platform._plugin_classes import * -from dlpx.virtualization.platform._discovery import * -from dlpx.virtualization.platform._linked import * -from dlpx.virtualization.platform._upgrade import * -from dlpx.virtualization.platform._virtual import * -from dlpx.virtualization.platform._plugin import * -from dlpx.virtualization.platform.util import * -from dlpx.virtualization.platform.import_util import * -from dlpx.virtualization.platform.import_validations import * +from dlpx.virtualization.platform.validation_util import * # noqa +from dlpx.virtualization.platform.migration_helper import * # noqa +from dlpx.virtualization.platform._plugin_classes import * # noqa +from dlpx.virtualization.platform._discovery import * # noqa +from dlpx.virtualization.platform._linked import * # noqa +from dlpx.virtualization.platform._upgrade import * # noqa +from dlpx.virtualization.platform._virtual import * # noqa +from dlpx.virtualization.platform._plugin import * # noqa +from dlpx.virtualization.platform.util import * # noqa +from dlpx.virtualization.platform.import_util import * # noqa +from dlpx.virtualization.platform.import_validations import * # noqa diff --git a/platform/src/main/python/dlpx/virtualization/platform/_plugin_classes.py b/platform/src/main/python/dlpx/virtualization/platform/_plugin_classes.py index 4d2da24f..97ff49ca 100644 --- a/platform/src/main/python/dlpx/virtualization/platform/_plugin_classes.py +++ b/platform/src/main/python/dlpx/virtualization/platform/_plugin_classes.py @@ -1,10 +1,10 @@ # -# Copyright (c) 2019 by Delphix. All rights reserved. +# Copyright (c) 2019, 2021 by Delphix. All rights reserved. # import re +import six from enum import Enum -import six from dlpx.virtualization.common import (RemoteConnection, RemoteEnvironment, RemoteHost) from dlpx.virtualization.common.exceptions import IncorrectTypeError @@ -160,8 +160,8 @@ def __init__(self, remote_environment, mount_path, shared_path=None): plugin writer from attempting to provide parameter values that they won't have access to.""" def __is_correct_reference_format(reference): - unix_format = re.compile("^UNIX_HOST_ENVIRONMENT-\d+$") - win_format = re.compile("^WINDOWS_HOST_ENVIRONMENT-\d+$") + unix_format = re.compile(r"^UNIX_HOST_ENVIRONMENT-\d+$") + win_format = re.compile(r"^WINDOWS_HOST_ENVIRONMENT-\d+$") return (bool(unix_format.match(reference)) or bool(win_format.match(reference))) @@ -199,7 +199,7 @@ def __make_remote_environment_from_reference(reference): six.string_types[0]) self._mount_path = mount_path - if shared_path and not isinstance(shared_path, six.string_types[0]): + if shared_path and not isinstance(shared_path, six.string_types): raise IncorrectTypeError(Mount, 'shared_path', type(shared_path), six.string_types[0], False) self._shared_path = shared_path diff --git a/platform/src/main/python/dlpx/virtualization/platform/_virtual.py b/platform/src/main/python/dlpx/virtualization/platform/_virtual.py index 2b2318a9..02d8826e 100644 --- a/platform/src/main/python/dlpx/virtualization/platform/_virtual.py +++ b/platform/src/main/python/dlpx/virtualization/platform/_virtual.py @@ -1,5 +1,5 @@ # -# Copyright (c) 2019 by Delphix. All rights reserved. +# Copyright (c) 2019, 2021 by Delphix. All rights reserved. # # -*- coding: utf-8 -*- @@ -25,6 +25,7 @@ class VirtualOperations(object): def __init__(self): self.configure_impl = None self.unconfigure_impl = None + self.cleanup_impl = None self.reconfigure_impl = None self.start_impl = None self.stop_impl = None @@ -54,6 +55,16 @@ def unconfigure_decorator(unconfigure_impl): return unconfigure_decorator + def cleanup(self): + def cleanup_decorator(cleanup_impl): + if self.cleanup_impl: + raise OperationAlreadyDefinedError(Op.VIRTUAL_CLEANUP) + self.cleanup_impl = v.check_function( + cleanup_impl, Op.VIRTUAL_CLEANUP) + return cleanup_impl + + return cleanup_decorator + def reconfigure(self): def reconfigure_decorator(reconfigure_impl): if self.reconfigure_impl: @@ -255,6 +266,58 @@ def _internal_unconfigure(self, request): platform_pb2.UnconfigureResult()) return unconfigure_response + def _internal_cleanup(self, request): + """Cleanup operation wrapper. + + Executed when deleting an existing virtual source. + This plugin operation is run after unconfigure. + + Args: + request (VirtualCleanupRequest): Cleanup operation arguments. + + Returns: + VirtualCleanupResponse: A response containing VirtualCleanupResult + if successful or PluginErrorResult in case of an error. + """ + # Reasoning for method imports are in this file's docstring. + from generated.definitions import VirtualSourceDefinition + from generated.definitions import RepositoryDefinition + from generated.definitions import SourceConfigDefinition + + # + # While virtual.cleanup() is not a required operation, this should + # not be called if it wasn't implemented. + # + if not self.cleanup_impl: + raise OperationNotDefinedError(Op.VIRTUAL_CLEANUP) + + virtual_source_definition = VirtualSourceDefinition.from_dict( + json.loads(request.virtual_source.parameters.json)) + mounts = [ + VirtualOperations._from_protobuf_single_subset_mount(m) + for m in request.virtual_source.mounts + ] + + virtual_source = VirtualSource(guid=request.virtual_source.guid, + connection=RemoteConnection.from_proto( + request.virtual_source.connection), + parameters=virtual_source_definition, + mounts=mounts) + + repository = RepositoryDefinition.from_dict( + json.loads(request.repository.parameters.json)) + source_config = SourceConfigDefinition.from_dict( + json.loads(request.source_config.parameters.json)) + + self.cleanup_impl( + repository=repository, source_config=source_config, + virtual_source=virtual_source) + + virtual_cleanup_response = platform_pb2.VirtualCleanupResponse() + virtual_cleanup_response.return_value.CopyFrom( + platform_pb2.VirtualCleanupResult()) + return virtual_cleanup_response + def _internal_reconfigure(self, request): """Reconfigure operation wrapper. @@ -627,8 +690,8 @@ def _internal_initialize(self, request): repository = RepositoryDefinition.from_dict( json.loads(request.repository.parameters.json)) - config = self.initialize_impl(repository=repository, - virtual_source=virtual_source) + config = self.initialize_impl( + repository=repository, virtual_source=virtual_source) # Validate that this is a SourceConfigDefinition object. if not isinstance(config, SourceConfigDefinition): diff --git a/platform/src/main/python/dlpx/virtualization/platform/import_validations.py b/platform/src/main/python/dlpx/virtualization/platform/import_validations.py index 12960511..d0249d9a 100644 --- a/platform/src/main/python/dlpx/virtualization/platform/import_validations.py +++ b/platform/src/main/python/dlpx/virtualization/platform/import_validations.py @@ -1,7 +1,8 @@ # -# Copyright (c) 2020 by Delphix. All rights reserved. +# Copyright (c) 2020, 2021 by Delphix. All rights reserved. # import inspect +import six from dlpx.virtualization.platform import exceptions from dlpx.virtualization.platform.import_util import (import_check, @@ -25,7 +26,6 @@ def validate_entry_point(plugin_module): if plugin_module.entry_point is None: raise exceptions.IncorrectPluginCodeError( 'Plugin entry point object is None.') - if not hasattr(plugin_module.module_content, plugin_module.entry_point): raise exceptions.UserError( 'Entry point \'{}:{}\' does not exist. \'{}\' is not a symbol' @@ -83,7 +83,10 @@ def validate_named_args(plugin_module): for op_name_key, op_name in plugin_attrib.__dict__.items(): if op_name is None: continue - actual_args = inspect.getargspec(op_name) + if six.PY2: + actual_args = inspect.getargspec(op_name) + else: + actual_args = inspect.getfullargspec(op_name) warnings.extend( _check_args(method_name=op_name.__name__, expected_args=_lookup_expected_args( diff --git a/platform/src/main/python/dlpx/virtualization/platform/migration_helper.py b/platform/src/main/python/dlpx/virtualization/platform/migration_helper.py index b53e7def..92b16337 100644 --- a/platform/src/main/python/dlpx/virtualization/platform/migration_helper.py +++ b/platform/src/main/python/dlpx/virtualization/platform/migration_helper.py @@ -1,8 +1,9 @@ # -# Copyright (c) 2019 by Delphix. All rights reserved. +# Copyright (c) 2019, 2021 by Delphix. All rights reserved. # import re +import six from dlpx.virtualization.platform import validation_util as v from dlpx.virtualization.platform.exceptions import ( @@ -142,7 +143,7 @@ def __add(self, migration_id, impl_name): @staticmethod def __validate_migration_id(migration_id, impl_name): # First validate that the id is a string - if not isinstance(migration_id, basestring): + if not isinstance(migration_id, six.string_types): raise MigrationIdIncorrectTypeError(migration_id, impl_name) # Next check if the id is the right format @@ -260,7 +261,7 @@ def add_snapshot(self, migration_id, snapshot_impl): def __validate_lua_major_minor_version(migration_id, impl_name, decorator_name, impl_getter): # First validate that the major minor version is a string - if not isinstance(migration_id, basestring): + if not isinstance(migration_id, six.string_types): raise MigrationIdIncorrectTypeError(migration_id, impl_name) # Next check if the id already exists in this particular dictionary @@ -308,6 +309,7 @@ def __get_sorted_impls(migration_id, impl_dict): # if not migration_id: return [] + # # First filter out all ids less than the migration id. We need to do # this because even after sorting, we wouldn't know where in the list diff --git a/platform/src/main/python/dlpx/virtualization/platform/operation.py b/platform/src/main/python/dlpx/virtualization/platform/operation.py index 12f1fb6d..87a06c31 100644 --- a/platform/src/main/python/dlpx/virtualization/platform/operation.py +++ b/platform/src/main/python/dlpx/virtualization/platform/operation.py @@ -21,6 +21,7 @@ class Operation(Enum): VIRTUAL_CONFIGURE = 'virtual.configure()' VIRTUAL_UNCONFIGURE = 'virtual.unconfigure()' VIRTUAL_RECONFIGURE = 'virtual.reconfigure()' + VIRTUAL_CLEANUP = 'virtual.cleanup()' VIRTUAL_START = 'virtual.start()' VIRTUAL_STOP = 'virtual.stop()' VIRTUAL_PRE_SNAPSHOT = 'virtual.pre_snapshot()' diff --git a/platform/src/main/python/dlpx/virtualization/platform/util.py b/platform/src/main/python/dlpx/virtualization/platform/util.py index d80a8718..79037b9e 100644 --- a/platform/src/main/python/dlpx/virtualization/platform/util.py +++ b/platform/src/main/python/dlpx/virtualization/platform/util.py @@ -1,7 +1,8 @@ # -# Copyright (c) 2019 by Delphix. All rights reserved. +# Copyright (c) 2019, 2021 by Delphix. All rights reserved. # import dlpx.virtualization.api +from dlpx.virtualization.common.util import to_str def get_virtualization_api_version(): @@ -9,4 +10,4 @@ def get_virtualization_api_version(): :return: version string """ - return dlpx.virtualization.api.__version__ + return to_str(dlpx.virtualization.api.__version__) diff --git a/platform/src/test/python/dlpx/virtualization/fake_generated_definitions.py b/platform/src/test/python/dlpx/virtualization/fake_generated_definitions.py index 90fbcef6..e4ef3419 100644 --- a/platform/src/test/python/dlpx/virtualization/fake_generated_definitions.py +++ b/platform/src/test/python/dlpx/virtualization/fake_generated_definitions.py @@ -1,3 +1,6 @@ +import six + + class Model(object): # swaggerTypes: The key is attribute name and the # value is attribute type. @@ -10,7 +13,7 @@ class Model(object): class RepositoryDefinition(Model): def __init__(self, name): - self.swagger_types = {'name': str} + self.swagger_types = {'name': six.string_types[0]} self.attribute_map = {'name': 'name'} self._name = name @@ -29,7 +32,7 @@ def to_dict(self): class SourceConfigDefinition(Model): def __init__(self, name): - self.swagger_types = {'name': str} + self.swagger_types = {'name': six.string_types[0]} self.attribute_map = {'name': 'name'} self._name = name @@ -48,7 +51,7 @@ def to_dict(self): class LinkedSourceDefinition(Model): def __init__(self, name): - self.swagger_types = {'name': str} + self.swagger_types = {'name': six.string_types[0]} self.attribute_map = {'name': 'name'} self._name = name @@ -64,7 +67,7 @@ def from_dict(input_dict): class VirtualSourceDefinition(Model): def __init__(self, name): - self.swagger_types = {'name': str} + self.swagger_types = {'name': six.string_types[0]} self.attribute_map = {'name': 'name'} self._name = name @@ -83,7 +86,7 @@ def to_dict(self): class SnapshotDefinition(Model): def __init__(self, name): - self.swagger_types = {'name': str} + self.swagger_types = {'name': six.string_types[0]} self.attribute_map = {'name': 'name'} self._name = name diff --git a/platform/src/test/python/dlpx/virtualization/test_migration_helper.py b/platform/src/test/python/dlpx/virtualization/test_migration_helper.py index de90d246..ac5ba436 100644 --- a/platform/src/test/python/dlpx/virtualization/test_migration_helper.py +++ b/platform/src/test/python/dlpx/virtualization/test_migration_helper.py @@ -1,9 +1,10 @@ +from __future__ import absolute_import # # Copyright (c) 2019 by Delphix. All rights reserved. # import pytest -import conftest +from . import conftest from dlpx.virtualization.platform import migration_helper as m from dlpx.virtualization.platform.exceptions import ( MigrationIdAlreadyUsedError, MigrationIdIncorrectFormatError, diff --git a/platform/src/test/python/dlpx/virtualization/test_plugin.py b/platform/src/test/python/dlpx/virtualization/test_plugin.py index 8cfde728..f69258ca 100755 --- a/platform/src/test/python/dlpx/virtualization/test_plugin.py +++ b/platform/src/test/python/dlpx/virtualization/test_plugin.py @@ -1,22 +1,24 @@ +from __future__ import absolute_import # -# Copyright (c) 2019 by Delphix. All rights reserved. +# Copyright (c) 2019, 2021 by Delphix. All rights reserved. # import json import pytest +import re +import six from dlpx.virtualization.api import common_pb2, platform_pb2 -from dlpx.virtualization.common import (RemoteConnection, RemoteEnvironment, - RemoteHost, RemoteUser) +from dlpx.virtualization.common import ( + RemoteConnection, RemoteEnvironment, RemoteHost, RemoteUser) from dlpx.virtualization.platform.exceptions import ( IncorrectReturnTypeError, IncorrectUpgradeObjectTypeError, OperationAlreadyDefinedError, PluginRuntimeError) from mock import MagicMock, patch -import fake_generated_definitions -from fake_generated_definitions import (RepositoryDefinition, - SnapshotDefinition, - SourceConfigDefinition) +from . import fake_generated_definitions +from .fake_generated_definitions import ( + RepositoryDefinition, SnapshotDefinition, SourceConfigDefinition) TEST_BINARY_PATH = '/binary/path' TEST_SCRATCH_PATH = '/scratch/path' @@ -63,9 +65,10 @@ })) TEST_POST_UPGRADE_PARAMS = ({ u'obj': - '"{\\"obj\\": {\\"prettyName\\": \\"prettyUpgrade\\", ' - '\\"name\\": \\"upgrade\\", \\"metadata\\": \\"metadata\\"}}"' + '"{\\"obj\\": {\\"name\\": \\"upgrade\\", ' + '\\"prettyName\\": \\"prettyUpgrade\\", \\"metadata\\": \\"metadata\\"}}"' }) + MIGRATION_IDS = ('2020.1.1', '2020.2.2') @@ -92,8 +95,8 @@ def configure_impl(): with pytest.raises(OperationAlreadyDefinedError): - @my_plugin.virtual.configure() - def configure_impl(): + @my_plugin.virtual.configure() # noqa F811 + def configure_impl(): # noqa F811 pass class NotModel1: @@ -467,10 +470,16 @@ def virtual_configure_impl(virtual_source, repository, snapshot): my_plugin.virtual._internal_configure(configure_request) message = err_info.value.message - assert message == ( - "The returned object for the virtual.configure() operation was" - " type 'unicode' but should be of class 'dlpx.virtualization." - "fake_generated_definitions.SourceConfigDefinition'.") + if six.PY2: + assert message == ( + "The returned object for the virtual.configure() operation was" + " type 'unicode' but should be of class 'dlpx.virtualization." + "fake_generated_definitions.SourceConfigDefinition'.") + else: + assert message == ( + "The returned object for the virtual.configure() operation was" + " class 'str' but should be of class 'dlpx.virtualization." + "fake_generated_definitions.SourceConfigDefinition'.") @staticmethod def test_virtual_unconfigure(my_plugin, virtual_source, repository, @@ -526,14 +535,11 @@ def virtual_reconfigure_impl(virtual_source, repository, source_config, assert config.parameters.json == expected_source_config @staticmethod - def test_virtual_reconfigure_return_incorrect_type(my_plugin, - virtual_source, - repository, - source_config, - snapshot): + def test_virtual_reconfigure_return_incorrect_type( + my_plugin, virtual_source, repository, source_config, snapshot): @my_plugin.virtual.reconfigure() - def virtual_reconfigure_impl(virtual_source, repository, source_config, - snapshot): + def virtual_reconfigure_impl( + virtual_source, repository, source_config, snapshot): TestPlugin.assert_plugin_args(virtual_source=virtual_source, source_config=source_config, repository=repository, @@ -553,14 +559,43 @@ def virtual_reconfigure_impl(virtual_source, repository, source_config, my_plugin.virtual._internal_reconfigure(reconfigure_request) message = err_info.value.message - assert message == ( - "The returned object for the virtual.reconfigure() operation was" - " type 'unicode' but should be of class 'dlpx.virtualization." - "fake_generated_definitions.SourceConfigDefinition'.") + if six.PY2: + assert message == ( + "The returned object for the virtual.reconfigure() operation was" + " type 'unicode' but should be of class 'dlpx.virtualization." + "fake_generated_definitions.SourceConfigDefinition'.") + else: + assert message == ( + "The returned object for the virtual.reconfigure() operation was" + " class 'str' but should be of class 'dlpx.virtualization." + "fake_generated_definitions.SourceConfigDefinition'.") + + @staticmethod + def test_virtual_cleanup(my_plugin, virtual_source, repository, source_config): + @my_plugin.virtual.cleanup() + def virtual_cleanup_impl(virtual_source, repository, source_config): + TestPlugin.assert_plugin_args(virtual_source=virtual_source, + repository=repository, + source_config=source_config) + return + + virtual_cleanup_request = platform_pb2.VirtualCleanupRequest() + TestPlugin.setup_request(request=virtual_cleanup_request, + virtual_source=virtual_source, + repository=repository, + source_config=source_config) + + expected_result = platform_pb2.VirtualCleanupResult() + + virtual_cleanup_response = my_plugin.virtual._internal_cleanup( + virtual_cleanup_request) + + # Check that the response's oneof is set to return_value and not error + assert virtual_cleanup_response.WhichOneof('result') == 'return_value' + assert virtual_cleanup_response.return_value == expected_result @staticmethod - def test_virtual_start(my_plugin, virtual_source, repository, - source_config): + def test_virtual_start(my_plugin, virtual_source, repository, source_config): @my_plugin.virtual.start() def virtual_start_impl(virtual_source, repository, source_config): TestPlugin.assert_plugin_args(virtual_source=virtual_source, @@ -695,8 +730,6 @@ def virtual_initialize_impl(virtual_source, repository): TestPlugin.setup_request(request=initialize_request, virtual_source=virtual_source, repository=repository) - - expected_source_config = TEST_REPOSITORY_JSON initialize_response = my_plugin.virtual._internal_initialize( initialize_request) @@ -722,10 +755,16 @@ def virtual_initialize_impl(virtual_source, repository): with pytest.raises(IncorrectReturnTypeError) as err_info: my_plugin.virtual._internal_initialize(initialize_request) message = err_info.value.message - assert message == ( - "The returned object for the virtual.initialize() operation was" - " type 'NoneType' but should be of class 'dlpx.virtualization." - "fake_generated_definitions.SourceConfigDefinition'.") + if six.PY2: + assert message == ( + "The returned object for the virtual.initialize() operation was" + " type 'NoneType' but should be of class 'dlpx.virtualization." + "fake_generated_definitions.SourceConfigDefinition'.") + else: + assert message == ( + "The returned object for the virtual.initialize() operation was" + " class 'NoneType' but should be of class 'dlpx.virtualization." + "fake_generated_definitions.SourceConfigDefinition'.") @staticmethod def test_virtual_mount_spec(my_plugin, virtual_source, repository): @@ -802,12 +841,20 @@ def repository_discovery_impl(source_connection): repository_discovery_request) message = err_info.value.message - assert message == ( - "The returned object for the discovery.repository() operation was" - " a list of [type 'str', class 'dlpx.virtualization" - ".fake_generated_definitions.RepositoryDefinition'] but should" - " be of type 'list of dlpx.virtualization" - ".fake_generated_definitions.RepositoryDefinition'.") + if six.PY2: + assert message == ( + "The returned object for the discovery.repository() operation was" + " a list of [type 'str', class 'dlpx.virtualization" + ".fake_generated_definitions.RepositoryDefinition'] but should" + " be of type 'list of dlpx.virtualization" + ".fake_generated_definitions.RepositoryDefinition'.") + else: + assert message == ( + "The returned object for the discovery.repository() operation was" + " a list of [class 'str', class 'dlpx.virtualization" + ".fake_generated_definitions.RepositoryDefinition'] but should" + " be of type 'list of dlpx.virtualization" + ".fake_generated_definitions.RepositoryDefinition'.") @staticmethod def test_source_config_discovery(my_plugin, connection, repository): @@ -867,8 +914,9 @@ def mock_direct_pre_snapshot(direct_source, repository, source_config, def test_direct_pre_snapshot_null_snapparams(my_plugin, direct_source, repository, source_config): @my_plugin.linked.pre_snapshot() - def mock_direct_pre_snapshot(direct_source, repository, source_config, - optional_snapshot_parameters): + def mock_direct_pre_snapshot( + direct_source, repository, source_config, + optional_snapshot_parameters): TestPlugin.assert_direct_source(direct_source) TestPlugin.assert_repository(repository) TestPlugin.assert_source_config(source_config) @@ -924,11 +972,11 @@ def direct_post_snapshot_impl(direct_source, repository, source_config, assert snapshot.parameters.json == expected_snapshot @staticmethod - def test_direct_post_snapshot_null_snapparams(my_plugin, direct_source, - repository, source_config): + def test_direct_post_snapshot_null_snapparams( + my_plugin, direct_source, repository, source_config): @my_plugin.linked.post_snapshot() def direct_post_snapshot_impl(direct_source, repository, source_config, - optional_snapshot_parameters): + optional_snapshot_parameters): TestPlugin.assert_direct_source(direct_source) TestPlugin.assert_repository(repository) TestPlugin.assert_source_config(source_config) @@ -987,7 +1035,7 @@ def test_staged_pre_snapshot_null_snapparams(my_plugin, staged_source, repository, source_config): @my_plugin.linked.pre_snapshot() def staged_pre_snapshot_impl(staged_source, repository, source_config, - optional_snapshot_parameters): + optional_snapshot_parameters): TestPlugin.assert_staged_source(staged_source) TestPlugin.assert_repository(repository) TestPlugin.assert_source_config(source_config) @@ -1045,7 +1093,7 @@ def test_staged_post_snapshot_null_snapparams(my_plugin, staged_source, repository, source_config): @my_plugin.linked.post_snapshot() def staged_post_snapshot_impl(staged_source, repository, source_config, - optional_snapshot_parameters): + optional_snapshot_parameters): TestPlugin.assert_staged_source(staged_source) TestPlugin.assert_repository(repository) TestPlugin.assert_source_config(source_config) @@ -1235,8 +1283,8 @@ def test_upgrade_repository_success(my_plugin): def upgrade_repository(old_repository): return TEST_POST_MIGRATION_METADATA_1 - @my_plugin.upgrade.repository('2020.2.2') - def upgrade_repository(old_repository): + @my_plugin.upgrade.repository('2020.2.2') # noqa F811 + def upgrade_repository(old_repository): # noqa F811 return TEST_POST_MIGRATION_METADATA_2 upgrade_request = platform_pb2.UpgradeRequest() @@ -1251,7 +1299,17 @@ def upgrade_repository(old_repository): expected_response.return_value.post_upgrade_parameters\ .update(TEST_POST_UPGRADE_PARAMS) - assert expected_response == upgrade_response + pat = re.compile( + 'return_value{post_upgrade_parameters{key:"obj"value:""{"obj":{("' + 'prettyName":"prettyUpgrade"|"name":"upgrade"),("prettyName":"' + 'prettyUpgrade"|"name":"upgrade"),"metadata":"metadata"}}""}}' + ) + assert re.match( + pat, str(expected_response).replace( + "\n", "").replace("\\", "").replace(" ", "")) + assert re.match( + pat, str(upgrade_response).replace( + "\n", "").replace("\\", "").replace(" ", "")) @staticmethod def test_upgrade_source_config_success(my_plugin): @@ -1259,8 +1317,8 @@ def test_upgrade_source_config_success(my_plugin): def upgrade_source_config(old_source_config): return TEST_POST_MIGRATION_METADATA_1 - @my_plugin.upgrade.source_config('2020.2.2') - def upgrade_source_config(old_source_config): + @my_plugin.upgrade.source_config('2020.2.2') # noqa F811 + def upgrade_source_config(old_source_config): # noqa F811 return TEST_POST_MIGRATION_METADATA_2 upgrade_request = platform_pb2.UpgradeRequest() @@ -1275,7 +1333,17 @@ def upgrade_source_config(old_source_config): expected_response.return_value.post_upgrade_parameters \ .update(TEST_POST_UPGRADE_PARAMS) - assert expected_response == upgrade_response + pat = re.compile( + 'return_value{post_upgrade_parameters{key:"obj"value:""{"obj":' + '{("prettyName":"prettyUpgrade"|"name":"upgrade"),("prettyName":"' + 'prettyUpgrade"|"name":"upgrade"),"metadata":"metadata"}}""}}' + ) + assert re.match( + pat, str(expected_response).replace( + "\n", "").replace("\\", "").replace(" ", "")) + assert re.match( + pat, str(upgrade_response).replace( + "\n", "").replace("\\", "").replace(" ", "")) @staticmethod def test_upgrade_linked_source_success(my_plugin): @@ -1283,8 +1351,8 @@ def test_upgrade_linked_source_success(my_plugin): def upgrade_linked_source(old_linked_source): return TEST_POST_MIGRATION_METADATA_1 - @my_plugin.upgrade.linked_source('2020.2.2') - def upgrade_linked_source(old_linked_source): + @my_plugin.upgrade.linked_source('2020.2.2') # noqa F811 + def upgrade_linked_source(old_linked_source): # noqa F811 return TEST_POST_MIGRATION_METADATA_2 upgrade_request = platform_pb2.UpgradeRequest() @@ -1299,7 +1367,17 @@ def upgrade_linked_source(old_linked_source): expected_response.return_value.post_upgrade_parameters \ .update(TEST_POST_UPGRADE_PARAMS) - assert expected_response == upgrade_response + pat = re.compile( + 'return_value{post_upgrade_parameters{key:"obj"value:""{"obj":{("' + 'prettyName":"prettyUpgrade"|"name":"upgrade"),("prettyName":"' + 'prettyUpgrade"|"name":"upgrade"),"metadata":"metadata"}}""}}' + ) + assert re.match( + pat, str(expected_response).replace( + "\n", "").replace("\\", "").replace(" ", "")) + assert re.match( + pat, str(upgrade_response).replace( + "\n", "").replace("\\", "").replace(" ", "")) @staticmethod def test_upgrade_virtual_source_success(my_plugin): @@ -1307,8 +1385,8 @@ def test_upgrade_virtual_source_success(my_plugin): def upgrade_virtual_source(old_virtual_source): return TEST_POST_MIGRATION_METADATA_1 - @my_plugin.upgrade.virtual_source('2020.2.2') - def upgrade_virtual_source(old_virtual_source): + @my_plugin.upgrade.virtual_source('2020.2.2') # noqa F811 + def upgrade_virtual_source(old_virtual_source): # noqa F811 return TEST_POST_MIGRATION_METADATA_2 upgrade_request = platform_pb2.UpgradeRequest() @@ -1323,7 +1401,17 @@ def upgrade_virtual_source(old_virtual_source): expected_response.return_value.post_upgrade_parameters \ .update(TEST_POST_UPGRADE_PARAMS) - assert expected_response == upgrade_response + pat = re.compile( + 'return_value{post_upgrade_parameters{key:"obj"value:""{"obj":{("name":' + '"upgrade"|"prettyName":"prettyUpgrade"),("name":"upgrade"|"prettyName":' + '"prettyUpgrade"),"metadata":"metadata"}}""}}' + ) + assert re.match( + pat, str(expected_response).replace( + "\n", "").replace("\\", "").replace(" ", "")) + assert re.match( + pat, str(upgrade_response).replace( + "\n", "").replace("\\", "").replace(" ", "")) @staticmethod def test_upgrade_snapshot_success(my_plugin): @@ -1331,8 +1419,8 @@ def test_upgrade_snapshot_success(my_plugin): def upgrade_snapshot(old_snapshot): return TEST_POST_MIGRATION_METADATA_1 - @my_plugin.upgrade.snapshot('2020.2.2') - def upgrade_snapshot(old_snapshot): + @my_plugin.upgrade.snapshot('2020.2.2') # noqa F811 + def upgrade_snapshot(old_snapshot): # noqa F811 return TEST_POST_MIGRATION_METADATA_2 upgrade_request = platform_pb2.UpgradeRequest() @@ -1347,7 +1435,17 @@ def upgrade_snapshot(old_snapshot): expected_response.return_value.post_upgrade_parameters \ .update(TEST_POST_UPGRADE_PARAMS) - assert expected_response == upgrade_response + pat = re.compile( + 'return_value{post_upgrade_parameters{key:"obj"value:""{"obj":{("' + 'prettyName":"prettyUpgrade"|"name":"upgrade"),("name":"' + 'upgrade"|"prettyName":"prettyUpgrade"),"metadata":"metadata"}}""}}' + ) + assert re.match( + pat, str(expected_response).replace( + "\n", "").replace("\\", "").replace(" ", "")) + assert re.match( + pat, str(upgrade_response).replace( + "\n", "").replace("\\", "").replace(" ", "")) @staticmethod def test_upgrade_repository_incorrect_upgrade_object_type(my_plugin): @@ -1415,8 +1513,8 @@ def test_upgrade_snapshot_fail_with_runtime_error(my_plugin): def upgrade_snapshot(old_snapshot): raise RuntimeError('RuntimeError in snapshot migration') - @my_plugin.upgrade.snapshot('2020.2.2') - def upgrade_snapshot(old_snapshot): + @my_plugin.upgrade.snapshot('2020.2.2') # noqa F811 + def upgrade_snapshot(old_snapshot): # noqa F811 raise RuntimeError('RuntimeError in snapshot migration') upgrade_request = platform_pb2.UpgradeRequest() diff --git a/platform/src/test/python/dlpx/virtualization/test_plugin_classes.py b/platform/src/test/python/dlpx/virtualization/test_plugin_classes.py index c1e5215a..fb12496a 100644 --- a/platform/src/test/python/dlpx/virtualization/test_plugin_classes.py +++ b/platform/src/test/python/dlpx/virtualization/test_plugin_classes.py @@ -1,8 +1,9 @@ # -# Copyright (c) 2019 by Delphix. All rights reserved. +# Copyright (c) 2019, 2021 by Delphix. All rights reserved. # import pytest +import six from dlpx.virtualization.common._common_classes import (RemoteEnvironment, RemoteHost) from dlpx.virtualization.common.exceptions import IncorrectTypeError @@ -40,17 +41,27 @@ def test_init_mount_bad_remote_env(): def test_init_mount_bad_mount_path(remote_environment): with pytest.raises(IncorrectTypeError) as err_info: Mount(remote_environment, 10000, 'shared_path') - assert err_info.value.message == ( - "Mount's parameter 'mount_path' was type 'int' but should" - " be of type 'basestring'.") + if six.PY2: + assert err_info.value.message == ( + "Mount's parameter 'mount_path' was type 'int' but should" + " be of type 'basestring'.") + else: + assert err_info.value.message == ( + "Mount's parameter 'mount_path' was class 'int' but should" + " be of class 'str'.") @staticmethod def test_init_mount_bad_shared_path(remote_environment): with pytest.raises(IncorrectTypeError) as err_info: Mount(remote_environment, 'mount_path', 10000) - assert err_info.value.message == ( - "Mount's parameter 'shared_path' was type 'int' but should" - " be of type 'basestring' if defined.") + if six.PY2: + assert err_info.value.message == ( + "Mount's parameter 'shared_path' was type 'int' but should" + " be of type 'basestring' if defined.") + else: + assert err_info.value.message == ( + "Mount's parameter 'shared_path' was class 'int' but should" + " be of class 'str' if defined.") @staticmethod def test_init_ownership_spec(): @@ -60,17 +71,27 @@ def test_init_ownership_spec(): def test_init_ownership_spec_bad_uid(): with pytest.raises(IncorrectTypeError) as err_info: OwnershipSpecification('10', 10) - assert err_info.value.message == ( - "OwnershipSpecification's parameter 'uid' was type 'str' but" - " should be of type 'int'.") + if six.PY2: + assert err_info.value.message == ( + "OwnershipSpecification's parameter 'uid' was type 'str' but" + " should be of type 'int'.") + else: + assert err_info.value.message == ( + "OwnershipSpecification's parameter 'uid' was class 'str' but" + " should be of class 'int'.") @staticmethod def test_init_ownership_spec_bad_gid(): with pytest.raises(IncorrectTypeError) as err_info: OwnershipSpecification(10, '10') - assert err_info.value.message == ( - "OwnershipSpecification's parameter 'gid' was type 'str' but" - " should be of type 'int'.") + if six.PY2: + assert err_info.value.message == ( + "OwnershipSpecification's parameter 'gid' was type 'str' but" + " should be of type 'int'.") + else: + assert err_info.value.message == ( + "OwnershipSpecification's parameter 'gid' was class 'str' but" + " should be of class 'int'.") @staticmethod def test_init_mount_spec(remote_environment): @@ -108,29 +129,48 @@ def test_init_mount_incorrect_format_reference_string(reference_string): def test_init_mount_invalid_reference_type(reference): with pytest.raises(IncorrectTypeError) as err_info: Mount(reference, 'mount_path', 'shared_path') - assert err_info.value.message == ( - "Mount's parameter 'remote_environment' was type '{}' but " - "should be of any one of the following types: " - "'['dlpx.virtualization.common._common_classes.RemoteEnvironment'," - " 'basestring']'.".format(type(reference).__name__)) + if six.PY2: + assert err_info.value.message == ( + "Mount's parameter 'remote_environment' was type '{}' but " + "should be of any one of the following types: " + "'['dlpx.virtualization.common._common_classes.RemoteEnvironment'," + " 'basestring']'.".format(type(reference).__name__)) + else: + assert err_info.value.message == ( + "Mount's parameter 'remote_environment' was class '{}' but " + "should be of any one of the following types: " + "'['dlpx.virtualization.common._common_classes.RemoteEnvironment'," + " 'str']'.".format(type(reference).__name__)) @staticmethod def test_init_mount_spec_mounts_not_list(): with pytest.raises(IncorrectTypeError) as err_info: MountSpecification('string', OwnershipSpecification(10, 10)) - assert err_info.value.message == ( - "MountSpecification's parameter 'mounts' was type 'str' but" - " should be of type 'list of dlpx.virtualization.platform" - "._plugin_classes.Mount'.") + if six.PY2: + assert err_info.value.message == ( + "MountSpecification's parameter 'mounts' was type 'str' but" + " should be of type 'list of dlpx.virtualization.platform" + "._plugin_classes.Mount'.") + else: + assert err_info.value.message == ( + "MountSpecification's parameter 'mounts' was class 'str' but" + " should be of type 'list of dlpx.virtualization.platform" + "._plugin_classes.Mount'.") @staticmethod def test_init_mount_spec_bad_mounts(): with pytest.raises(IncorrectTypeError) as err_info: MountSpecification(['string'], OwnershipSpecification(10, 10)) - assert err_info.value.message == ( - "MountSpecification's parameter 'mounts' was a list of" - " [type 'str'] but should be of type 'list of dlpx.virtualization" - ".platform._plugin_classes.Mount'.") + if six.PY2: + assert err_info.value.message == ( + "MountSpecification's parameter 'mounts' was a list of" + " [type 'str'] but should be of type 'list of dlpx.virtualization" + ".platform._plugin_classes.Mount'.") + else: + assert err_info.value.message == ( + "MountSpecification's parameter 'mounts' was a list of" + " [class 'str'] but should be of type 'list of dlpx.virtualization" + ".platform._plugin_classes.Mount'.") @staticmethod def test_init_mount_spec_bad_owner_spec(remote_environment): @@ -138,8 +178,15 @@ def test_init_mount_spec_bad_owner_spec(remote_environment): with pytest.raises(IncorrectTypeError) as err_info: MountSpecification([mount], 'string') - assert err_info.value.message == ( - "MountSpecification's parameter 'ownership_specification' was" - " type 'str' but should be of class 'dlpx.virtualization" - ".platform._plugin_classes.OwnershipSpecification'" - " if defined.") + if six.PY2: + assert err_info.value.message == ( + "MountSpecification's parameter 'ownership_specification' was" + " type 'str' but should be of class 'dlpx.virtualization" + ".platform._plugin_classes.OwnershipSpecification'" + " if defined.") + else: + assert err_info.value.message == ( + "MountSpecification's parameter 'ownership_specification' was" + " class 'str' but should be of class 'dlpx.virtualization" + ".platform._plugin_classes.OwnershipSpecification'" + " if defined.") diff --git a/platform/src/test/python/dlpx/virtualization/test_upgrade.py b/platform/src/test/python/dlpx/virtualization/test_upgrade.py index f5801e28..7990d384 100755 --- a/platform/src/test/python/dlpx/virtualization/test_upgrade.py +++ b/platform/src/test/python/dlpx/virtualization/test_upgrade.py @@ -1,5 +1,5 @@ # -# Copyright (c) 2019 by Delphix. All rights reserved. +# Copyright (c) 2019, 2021 by Delphix. All rights reserved. # import copy @@ -254,8 +254,8 @@ def repo_upgrade_one(input_dict): output_dict['migrations'].append('lua repo 1.1') return output_dict - @upgrade_type_decorator('1.2', MigrationType.LUA) - def repo_upgrade_one(input_dict): + @upgrade_type_decorator('1.2', MigrationType.LUA) # noqa F811 + def repo_upgrade_one(input_dict): # noqa F811 output_dict = copy.deepcopy(input_dict) output_dict['migrations'].append('lua repo 1.2') return output_dict @@ -266,14 +266,14 @@ def repo_upgrade_two(input_dict): output_dict['migrations'].append('platform repo 2020.4.2') return output_dict - @upgrade_type_decorator('2020.4.3') - def repo_upgrade_two(input_dict): + @upgrade_type_decorator('2020.4.3') # noqa F811 + def repo_upgrade_two(input_dict): # noqa F811 output_dict = copy.deepcopy(input_dict) output_dict['migrations'].append('platform repo 2020.4.3') return output_dict - @upgrade_type_decorator('2020.4.4') - def repo_upgrade_two(input_dict): + @upgrade_type_decorator('2020.4.4') # noqa F811 + def repo_upgrade_two(input_dict): # noqa F811 output_dict = copy.deepcopy(input_dict) output_dict['migrations'].append('platform repo 2020.4.4') return output_dict diff --git a/tools/.python-version b/tools/.python-version deleted file mode 100644 index 43c4dbe6..00000000 --- a/tools/.python-version +++ /dev/null @@ -1 +0,0 @@ -2.7.17 diff --git a/tools/setup.cfg b/tools/setup.cfg index 690c13de..91544edc 100644 --- a/tools/setup.cfg +++ b/tools/setup.cfg @@ -1,25 +1,25 @@ # -# Copyright (c) 2019 by Delphix. All rights reserved. +# Copyright (c) 2019, 2021 by Delphix. All rights reserved. # [metadata] -Metadata-Version: 1.2 -Author: Delphix -Author-email: virtualization-plugins@delphix.com -Home-page: https://developer.delphix.com -Summary: Delphix Virtualization SDK Tools -Long-description: file: README.md -Long-description-content-type: text/markdown -Classifiers: +metadata_version: 1.2 +author: Delphix +author_email: virtualization-plugins@delphix.com +home_page: https://developer.delphix.com +summary: Delphix Virtualization SDK Tools +long_description: file: README.md +long_description_content_type: text/markdown +classifiers: Development Status :: 5 - Production/Stable Programming Language :: Python - Programming Language :: Python :: 2.7 + Programming Language :: Python :: 3.8 License :: OSI Approved :: Apache Software License Operating System :: OS Independent [options] include_package_data = True -Requires-Python: 2.7 +requires_python: 3.8 [options.entry_points] console_scripts = diff --git a/tools/setup.py b/tools/setup.py index b5e0edc1..0e91ae4f 100644 --- a/tools/setup.py +++ b/tools/setup.py @@ -7,15 +7,16 @@ version = version_file.read().strip() install_requires = [ - "click >= 7.1", - "click-configfile == 0.2.3", - "dvp-platform == {}".format(version), - "enum34 >= 1.1.6", - "flake8 >= 3.6", - "jinja2 >= 2.10", - "jsonschema >= 3", - "pyyaml >= 3", - "requests >= 2.21.0", + "click == 7.1.2", + "click-configfile == 0.2.3", + "dvp-platform == {}".format(version), + "enum34 >= 1.1.6", + "flake8 >= 3.6", + "jinja2 >= 2.10", + "jsonschema >= 3", + "pyyaml >= 3", + "requests >= 2.21.0", + "httpretty == 0.9.7", ] setuptools.setup(name='dvp-tools', @@ -23,4 +24,5 @@ install_requires=install_requires, package_dir={'': PYTHON_SRC}, packages=setuptools.find_packages(PYTHON_SRC), + python_requires='>=3.8, <3.9', ) diff --git a/tools/src/main/python/dlpx/virtualization/_internal/VERSION b/tools/src/main/python/dlpx/virtualization/_internal/VERSION index fd2a0186..fcdb2e10 100644 --- a/tools/src/main/python/dlpx/virtualization/_internal/VERSION +++ b/tools/src/main/python/dlpx/virtualization/_internal/VERSION @@ -1 +1 @@ -3.1.0 +4.0.0 diff --git a/tools/src/main/python/dlpx/virtualization/_internal/cli.py b/tools/src/main/python/dlpx/virtualization/_internal/cli.py index 7d785591..ea7094b5 100644 --- a/tools/src/main/python/dlpx/virtualization/_internal/cli.py +++ b/tools/src/main/python/dlpx/virtualization/_internal/cli.py @@ -1,5 +1,5 @@ # -# Copyright (c) 2019, 2020 by Delphix. All rights reserved. +# Copyright (c) 2019, 2021 by Delphix. All rights reserved. # import logging @@ -45,8 +45,7 @@ # adding it to CONTEXT_SETTINGS to avoid any side-effects on other commands. # CONTEXT_SETTINGS_INIT = dict(help_option_names=['-h', '--help'], - obj=click_util.ConfigFileProcessor.read_config(), - token_normalize_func=lambda x: x.encode("ascii")) + obj=click_util.ConfigFileProcessor.read_config()) DVP_CONFIG_MAP = CONTEXT_SETTINGS['obj'] @@ -96,11 +95,11 @@ def delphix_sdk(verbose, quiet): # will be printed to the console until this is executed. # logging_util.add_console_handler(console_logging_level) - - if sys.version_info[:2] != (2, 7): + if sys.version_info[:2] != (3, 8): raise exceptions.UserError( 'Python version check failed.' - 'Supported version is 2.7.x, found {}'.format(sys.version_info)) + 'Supported versions are 2.7.x and 3.8.x, found {}' + .format(sys.version_info)) @delphix_sdk.command(context_settings=CONTEXT_SETTINGS_INIT) diff --git a/tools/src/main/python/dlpx/virtualization/_internal/codegen.py b/tools/src/main/python/dlpx/virtualization/_internal/codegen.py index 30b1529c..cfc2d91d 100644 --- a/tools/src/main/python/dlpx/virtualization/_internal/codegen.py +++ b/tools/src/main/python/dlpx/virtualization/_internal/codegen.py @@ -1,5 +1,5 @@ # -# Copyright (c) 2019, 2020 by Delphix. All rights reserved. +# Copyright (c) 2019, 2021 by Delphix. All rights reserved. # import copy @@ -8,9 +8,11 @@ import logging import os import shutil +import six import subprocess from dlpx.virtualization._internal import const, exceptions, file_util +from dlpx.virtualization.common.util import to_str logger = logging.getLogger(__name__) UNKNOWN_ERR = 'UNKNOWN_ERR' @@ -96,8 +98,9 @@ def generate_python(name, source_dir, plugin_config_dir, schema_content): # def _make_url_refs_opaque(json): if isinstance(json, dict): - for key in json: - if key == '$ref' and isinstance(json[key], basestring)\ + keys = list(json.keys()) + for key in keys: + if key == '$ref' and isinstance(json[key], six.string_types)\ and json[key].startswith('https://delphix.com/platform/api#'): json.pop(key) json['type'] = 'object' @@ -181,8 +184,7 @@ def _execute_swagger_codegen(swagger_file, output_dir): output_dir ] - logger.info('Running process with arguments: {!r}'.format( - ' '.join(process_inputs))) + logger.info(f"Running process with arguments: \'{' '.join(process_inputs)}\'") process = subprocess.Popen(process_inputs, stdout=subprocess.PIPE, stderr=subprocess.PIPE) @@ -191,12 +193,13 @@ def _execute_swagger_codegen(swagger_file, output_dir): raise exceptions.UserError('Swagger python code generation failed.' ' Make sure java is on the PATH.') raise exceptions.UserError( - 'Unable to run {!r} to generate python code.' - '\nError code: {}. Error message: {}'.format( - jar, err.errno, os.strerror(err.errno))) + f"Unable to run '{jar}' to generate python code." + f"\nError code: {err.errno}. Error message: {os.strerror(err.errno)}") # Get the pipes pointed so we have access to them. stdout, stderr = process.communicate() + stdout = to_str(stdout) + stderr = to_str(stderr) # # Wait for the process to end and take the results. If res then we know diff --git a/tools/src/main/python/dlpx/virtualization/_internal/codegen/templates/base_model_.mustache b/tools/src/main/python/dlpx/virtualization/_internal/codegen/templates/base_model_.mustache index 80f49d0b..ec6631a3 100644 --- a/tools/src/main/python/dlpx/virtualization/_internal/codegen/templates/base_model_.mustache +++ b/tools/src/main/python/dlpx/virtualization/_internal/codegen/templates/base_model_.mustache @@ -1,10 +1,10 @@ # -# Copyright (c) 2019 by Delphix. All rights reserved. +# Copyright (c) 2019, 2021 by Delphix. All rights reserved. # import pprint - import six + {{^supportPython2}} import typing {{/supportPython2}} @@ -37,7 +37,7 @@ class Model(object): """ result = {} - for attr, _ in six.iteritems(self.swagger_types): + for attr, _ in self.swagger_types.items(): value = getattr(self, attr) if value is None: # Plugins use the JSON schema specification to define their @@ -170,7 +170,7 @@ class GeneratedClassesTypeError(GeneratedClassesError): container then we assume there is one element in it and that type is the expected type of the container. (For dicts this is the key) ie: if expected_type = {str} then the returned expected string with - be something like "type 'dict with key basestring'" + be something like "type 'dict with key str'" Returns: tuple (str, str): the actual and expected strings used for the @@ -180,13 +180,16 @@ class GeneratedClassesTypeError(GeneratedClassesError): def _remove_angle_brackets(type_string): return type_string.replace('<', '').replace('>', '') + # In py3 the builtins module will be named 'builtins', in py2 it will + # be '__builtin__'. + builtins = ['__builtin__', 'builtins'] if isinstance(expected_type, list): if len(expected_type) != 1: raise ValueError('The thrown GeneratedClassesTypeError should' ' have had a list of size 1 as the' ' expected_type') single_type = expected_type[0] - if single_type.__module__ != '__builtin__': + if single_type.__module__ not in builtins: type_name = '{}.{}'.format( single_type.__module__, single_type.__name__) else: @@ -198,7 +201,10 @@ class GeneratedClassesTypeError(GeneratedClassesError): ' have had a set of size 1 as the' ' expected_type') single_type = expected_type.pop() - if single_type.__module__ != '__builtin__': + + + st_module = single_type.__module__ + if st_module not in builtins: type_name = '{}.{}'.format( single_type.__module__, single_type.__name__) else: @@ -209,14 +215,14 @@ class GeneratedClassesTypeError(GeneratedClassesError): raise ValueError('The thrown GeneratedClassesTypeError should' ' have had a dict of size 1 as the' ' expected_type') - key_type = expected_type.keys()[0] - value_type = expected_type.values()[0] - if key_type.__module__ != '__builtin__': + key_type = list(expected_type.keys())[0] + value_type = list(expected_type.values())[0] + if key_type.__module__ not in builtins: key_type_name = '{}.{}'.format( key_type.__module__, key_type.__name__) else: key_type_name = key_type.__name__ - if value_type.__module__ != '__builtin__': + if value_type.__module__ not in builtins: value_type_name = '{}.{}'.format( value_type.__module__, value_type.__name__) else: @@ -224,11 +230,11 @@ class GeneratedClassesTypeError(GeneratedClassesError): expected = "type 'dict of {}:{}'".format( key_type_name, value_type_name) else: - expected = _remove_angle_brackets(str(expected_type)) + expected = _remove_angle_brackets(six.string_types[0](expected_type)) if isinstance(actual_type, list): actual = 'a list of [{}]'.format( - ', '.join(_remove_angle_brackets(str(single_type)) + ', '.join(_remove_angle_brackets(six.string_types[0](single_type)) for single_type in actual_type)) elif isinstance(actual_type, set): # @@ -241,19 +247,19 @@ class GeneratedClassesTypeError(GeneratedClassesError): for type_tuple in actual_type)): actual = 'a dict with keys of {}{}{}'.format( '{', - ', '.join(_remove_angle_brackets(str(single_type)) + ', '.join(_remove_angle_brackets(six.string_types[0](single_type)) for single_type in actual_type), '}') else: actual = 'a dict of {}{}{}'.format( '{', ', '.join(['{0}:{1}'.format( - _remove_angle_brackets(str(k)), - _remove_angle_brackets(str(v))) for k, v in actual_type]), + _remove_angle_brackets(six.string_types[0](k)), + _remove_angle_brackets(six.string_types[0](v))) for k, v in actual_type]), '}') else: - actual = _remove_angle_brackets(str(actual_type)) + actual = _remove_angle_brackets(six.string_types[0](actual_type)) return actual, expected @@ -276,19 +282,20 @@ class GeneratedClassesTypeError(GeneratedClassesError): if not required and parameter is None: return None # Now check if the types are incorrect. + num_types = (float, complex, int) if expected_type == float: - if not isinstance(parameter, (float, int, long, complex)): + if not isinstance(parameter, num_types): return GeneratedClassesTypeError(object_type, parameter_name, type(parameter), float, required) - elif expected_type == str: - if not isinstance(parameter, basestring): + elif expected_type == six.string_types[0]: + if not isinstance(parameter, six.string_types): return GeneratedClassesTypeError(object_type, parameter_name, type(parameter), - basestring, + six.string_types[0], required) elif expected_type == list: if element_type: @@ -300,7 +307,7 @@ class GeneratedClassesTypeError(GeneratedClassesError): required) if element_type == float: - check = all(isinstance(elem, (float, int, long, complex)) + check = all(isinstance(elem, num_types) for elem in parameter) else: check = all(isinstance(elem, element_type) @@ -324,7 +331,7 @@ class GeneratedClassesTypeError(GeneratedClassesError): return GeneratedClassesTypeError(object_type, parameter_name, type(parameter), - {basestring}, + {six.string_types[0]}, required) # @@ -336,26 +343,26 @@ class GeneratedClassesTypeError(GeneratedClassesError): if element_type: if element_type == float: value_check = all(isinstance(v, - (float, int, long, complex)) + num_types) for v in parameter.values()) else: value_check = all(isinstance(v, element_type) for v in parameter.values()) - if (not all(isinstance(k, basestring) + if (not all(isinstance(k, six.string_types) for k in parameter.keys()) or not value_check): return GeneratedClassesTypeError( object_type, parameter_name, {(type(k), type(v)) for k, v in parameter.items()}, - {basestring: element_type}, + {six.string_types[0]: element_type}, required) else: - if not all(isinstance(k, basestring) for k in parameter.keys()): + if not all(isinstance(k, six.string_types) for k in parameter.keys()): return GeneratedClassesTypeError( object_type, parameter_name, {type(k) for k in parameter.keys()}, - {basestring}, + {six.string_types[0]}, required) else: diff --git a/tools/src/main/python/dlpx/virtualization/_internal/codegen/templates/util.mustache b/tools/src/main/python/dlpx/virtualization/_internal/codegen/templates/util.mustache index 008528ae..b6a59aa2 100644 --- a/tools/src/main/python/dlpx/virtualization/_internal/codegen/templates/util.mustache +++ b/tools/src/main/python/dlpx/virtualization/_internal/codegen/templates/util.mustache @@ -1,7 +1,9 @@ +# +# Copyright (c) 2019, 2021 by Delphix. All rights reserved. +# import datetime import pydoc import re - import six @@ -33,9 +35,9 @@ def get_contained_type(type_string): for pattern in patterns: match = re.search(pattern, type_string) if match and match.group(1) != 'ERRORUNKNOWN': - # Convert the type to basestring here. + # Convert the type to string here. if match.group(1) == 'str': - return basestring + return six.string_types[0] return pydoc.locate(match.group(1)) return None @@ -53,7 +55,7 @@ def deserialize_model(data, klass): if not instance.swagger_types: return data - for attr, attr_type in six.iteritems(instance.swagger_types): + for attr, attr_type in instance.swagger_types.items(): if (data is not None and instance.attribute_map[attr] in data and isinstance(data, dict)): value = data[instance.attribute_map[attr]] @@ -73,7 +75,8 @@ def _deserialize(data, klass): if data is None: return None - if issubclass(klass, (int, float, long, complex, basestring, bool)): + primitive_types = {float, complex, bool, int, six.text_type, six.binary_type} + if issubclass(klass, tuple(primitive_types)): return _deserialize_primitive(data, klass) elif klass == datetime.date: return deserialize_date(data) @@ -93,21 +96,20 @@ def _deserialize_primitive(data, klass): :param data: data to deserialize. :param klass: class literal. - :return: int, float, long, complex, basestring, bool. - :rtype: int | float | long | complex | basestring | bool + :return: int, float, long, complex, str, bool. + :rtype: int | float | long | complex | str | bool """ try: value = klass(data) except UnicodeEncodeError: - if isinstance(data, str): + if isinstance(data, six.binary_type): # # Ignore errors even if the string is not proper UTF-8 or has - # broken marker bytes. The builtin unicode function can do this. + # broken marker bytes. # - value = unicode(data, 'utf-8', errors='ignore') + value = str(data, "utf-8", errors='ignore') else: - # Assume the value object has proper __unicode__() method. - value = unicode(data) + value = six.text_type(data) except TypeError: value = data return value @@ -166,4 +168,4 @@ def _deserialize_dict(data): :return: deserialized dict. :rtype: dict """ - return {k: _deserialize(v, type(v)) for k, v in six.iteritems(data)} + return {k: _deserialize(v, type(v)) for k, v in data.items()} diff --git a/tools/src/main/python/dlpx/virtualization/_internal/commands/build.py b/tools/src/main/python/dlpx/virtualization/_internal/commands/build.py index 30eff7ab..38c46d02 100644 --- a/tools/src/main/python/dlpx/virtualization/_internal/commands/build.py +++ b/tools/src/main/python/dlpx/virtualization/_internal/commands/build.py @@ -1,19 +1,20 @@ # -# Copyright (c) 2019, 2020 by Delphix. All rights reserved. +# Copyright (c) 2019, 2021 by Delphix. All rights reserved. # import base64 import compileall import copy +import io import json import logging import os -import StringIO import zipfile from dlpx.virtualization._internal import (codegen, exceptions, file_util, package_util, plugin_dependency_util, plugin_util) +from dlpx.virtualization.common.util import to_str logger = logging.getLogger(__name__) @@ -56,8 +57,16 @@ def build(plugin_config, 'Build parameters include plugin_config: %s, upload_artifact: %s,' ' generate_only: %s', plugin_config, upload_artifact, generate_only) + # Click handles the conversions for us, so we need not run inputs through to_str in + # the cli build function. However, this function may be called from places other + # than its corresponding cli function, such as unit tests. As such, we should ensure + # all appropriate inputs at this point are properly converted to unicode strings as + # soon as they enter the program. + plugin_config = to_str(plugin_config) + upload_artifact = to_str(upload_artifact) + if local_vsdk_root: - local_vsdk_root = os.path.expanduser(local_vsdk_root) + local_vsdk_root = to_str(os.path.expanduser(local_vsdk_root)) # Read content of the plugin config file provided and perform validations logger.info('Validating plugin config file %s', plugin_config) @@ -100,7 +109,6 @@ def build(plugin_config, os.path.dirname(plugin_config), schemas) except exceptions.UserError as err: raise exceptions.BuildFailedError(err) - if generate_only: # # If the generate_only flag is set then just return after generation @@ -140,6 +148,10 @@ def build(plugin_config, # Install dependencies in the plugin's source root in the build directory. plugin_dependency_util.install_deps(build_src_dir, local_vsdk_root=local_vsdk_root) + virtualization_dir = os.path.join(build_src_dir, "dlpx", "virtualization") + + for pkg in ['api', 'common', 'libs', 'platform']: + plugin_dependency_util.compile_py_files(os.path.join(virtualization_dir, pkg)) # Patch dependencies. patch_dependencies(build_src_dir) @@ -160,7 +172,7 @@ def build(plugin_config, logger.info('Successfully generated artifact file at %s.', upload_artifact) - logger.warn('\nBUILD SUCCESSFUL.') + logger.warning('\nBUILD SUCCESSFUL.') def patch_dependencies(build_src_dir): @@ -174,7 +186,7 @@ def patch_dependencies(build_src_dir): json_format_path = os.path.join( build_src_dir, 'google', 'protobuf', 'json_format.py') with open(json_format_path, 'r') as f: - json_format_text = f.read() + json_format_text = to_str(f.read()) json_format_text = json_format_text\ .replace(UNPAIRED_SURROGATE_DEFINITION, '')\ .replace(UNPAIRED_SURROGATE_SEARCH, '') @@ -187,6 +199,7 @@ def prepare_upload_artifact(plugin_config_content, src_dir, schemas, manifest): # This is the output dictionary that will be written # to the upload_artifact. # + artifact = { # Hard code the type to a set default. 'type': @@ -198,7 +211,7 @@ def prepare_upload_artifact(plugin_config_content, src_dir, schemas, manifest): # set default value of locale to en-us 'defaultLocale': plugin_config_content.get('defaultLocale', LOCALE_DEFAULT), - # set default value of language to PYTHON27 + # set default value of language to PYTHON38 'language': plugin_config_content['language'], 'hostTypes': @@ -252,7 +265,7 @@ def prepare_upload_artifact(plugin_config_content, src_dir, schemas, manifest): artifact['minimumLuaVersion'] = plugin_config_content[ 'minimumLuaVersion'] - return artifact + return to_str(artifact) def get_linked_source_definition_type(plugin_config_content): @@ -332,7 +345,7 @@ def zip_and_encode_source_files(source_code_dir): Jython creates a class loader to import .py files which the security manager prohibits. """ - + source_code_dir = to_str(source_code_dir) # # The contents of the zip should have relative and not absolute paths or # else the imports won't work as expected. @@ -348,7 +361,8 @@ def zip_and_encode_source_files(source_code_dir): raise exceptions.UserError( 'Failed to compile source code in the directory {}.'.format( source_code_dir)) - out_file = StringIO.StringIO() + + out_file = io.BytesIO() with zipfile.ZipFile(out_file, 'w', zipfile.ZIP_DEFLATED) as zip_file: for root, _, files in os.walk('.'): for filename in files: diff --git a/tools/src/main/python/dlpx/virtualization/_internal/commands/download_logs.py b/tools/src/main/python/dlpx/virtualization/_internal/commands/download_logs.py index 7000b701..b29f5e17 100644 --- a/tools/src/main/python/dlpx/virtualization/_internal/commands/download_logs.py +++ b/tools/src/main/python/dlpx/virtualization/_internal/commands/download_logs.py @@ -1,10 +1,11 @@ # -# Copyright (c) 2019 by Delphix. All rights reserved. +# Copyright (c) 2019, 2021 by Delphix. All rights reserved. # import logging from dlpx.virtualization._internal import delphix_client, package_util +from dlpx.virtualization.common.util import to_str logger = logging.getLogger(__name__) @@ -34,6 +35,17 @@ def download_logs(engine, plugin_config, user, password, directory): logger.info('Downloading plugin logs from {} to: {}'.format( engine, directory)) + # Click handles the conversions for us, so we need not run inputs through to_str in + # the cli download_logs function. However, to be safe, this function may be called + # from places other than its corresponding cli function, such as unit tests. As + # such, we should ensure all appropriate inputs at this point are properly + # converted to unicode strings as soon as they enter the program. + engine = to_str(engine) + plugin_config = to_str(plugin_config) + user = to_str(user) + password = to_str(password) + directory = to_str(directory) + # Create a new delphix session. client = delphix_client.DelphixClient(engine) client.login(package_util.get_engine_api_version(), user, password) diff --git a/tools/src/main/python/dlpx/virtualization/_internal/commands/initialize.py b/tools/src/main/python/dlpx/virtualization/_internal/commands/initialize.py index 7d6d04b5..0975d822 100644 --- a/tools/src/main/python/dlpx/virtualization/_internal/commands/initialize.py +++ b/tools/src/main/python/dlpx/virtualization/_internal/commands/initialize.py @@ -1,10 +1,11 @@ # -# Copyright (c) 2019, 2020 by Delphix. All rights reserved. +# Copyright (c) 2019, 2021 by Delphix. All rights reserved. # import logging import os import shutil +import six import uuid from collections import OrderedDict @@ -12,6 +13,7 @@ import yaml from dlpx.virtualization._internal import (codegen, const, exceptions, file_util, plugin_util) +from dlpx.virtualization.common.util import to_bytes, to_str logger = logging.getLogger(__name__) @@ -60,6 +62,16 @@ def init(root, ingestion_strategy, name, host_type): 'Host Types': host_type }) + # Click handles the conversions for us, so we need not run inputs through to_str in + # the cli init function. However, this function may be called from places other + # than its corresponding cli function, such as unit tests. As such, we should ensure + # all appropriate inputs at this point are properly converted to unicode strings as + # soon as they enter the program. + root = to_str(root) + ingestion_strategy = to_str(ingestion_strategy) + name = to_str(name) + host_type = to_str(host_type) + # Files paths based on 'root' to be used throughout src_dir_path = os.path.join(root, DEFAULT_SRC_DIRECTORY) config_file_path = os.path.join(root, DEFAULT_PLUGIN_CONFIG_FILE) @@ -207,17 +219,30 @@ def _get_default_plugin_config(plugin_id, ingestion_strategy, name, OrderedDict: A valid plugin configuration roughly ordered from most interesting to a new plugin author to least interesting. """ - # Ensure values are type 'str'. If they are type unicode yaml prints - # them with '!!python/unicode' prepended to the value. - config = OrderedDict([('id', plugin_id.encode('utf-8')), - ('name', name.encode('utf-8')), - ('language', 'PYTHON27'), ('hostTypes', ['UNIX']), - ('pluginType', ingestion_strategy.encode('utf-8')), - ('entryPoint', entry_point.encode('utf-8')), - ('srcDir', src_dir_path.encode('utf-8')), - ('schemaFile', schema_file_path.encode('utf-8')), - ('hostTypes', [host_type.encode('utf-8')]), - ('buildNumber', default_build_number.encode('utf-8')) + if six.PY2: + # + # Ensure values are type 'str'. If they are type unicode yaml prints + # them with '!!python/unicode' prepended to the value. + # + # In Py3 yaml will print bytes with `!!binary |` prepended to them, so we + # should leave the as strings. + # + plugin_id = to_bytes(plugin_id) + name = to_bytes(name) + ingestion_strategy = to_bytes(ingestion_strategy) + entry_point = to_bytes(entry_point) + src_dir_path = to_bytes(src_dir_path) + schema_file_path = to_bytes(schema_file_path) + host_type = to_bytes(host_type) + default_build_number = to_bytes(default_build_number) + config = OrderedDict([('id', plugin_id), + ('name', name), + ('language', 'PYTHON38'), ('hostTypes', ['UNIX']), + ('pluginType', ingestion_strategy), + ('entryPoint', entry_point), + ('srcDir', src_dir_path), + ('schemaFile', schema_file_path), + ('hostTypes', [host_type]), + ('buildNumber', default_build_number) ]) - return config diff --git a/tools/src/main/python/dlpx/virtualization/_internal/commands/upload.py b/tools/src/main/python/dlpx/virtualization/_internal/commands/upload.py index 1395f4ae..fbfa58a3 100644 --- a/tools/src/main/python/dlpx/virtualization/_internal/commands/upload.py +++ b/tools/src/main/python/dlpx/virtualization/_internal/commands/upload.py @@ -1,5 +1,5 @@ # -# Copyright (c) 2019 by Delphix. All rights reserved. +# Copyright (c) 2019, 2021 by Delphix. All rights reserved. # import errno @@ -8,6 +8,7 @@ import os from dlpx.virtualization._internal import delphix_client, exceptions +from dlpx.virtualization.common.util import to_str logger = logging.getLogger(__name__) UNKNOWN_ERR = 'UNKNOWN_ERR' @@ -36,6 +37,18 @@ def upload(engine, user, upload_artifact, password, wait): ' wait: {}'.format(engine, user, upload_artifact, wait)) logger.info('Uploading plugin artifact {} ...'.format(upload_artifact)) + # + # Click handles the conversions for us, so we need not run inputs through to_str in + # the cli upload function. However, this function may be called from places other + # than its corresponding cli function, such as unit tests. As such, we should ensure + # all appropriate inputs at this point are properly converted to unicode strings as + # soon as they enter the program. + # + engine = to_str(engine) + user = to_str(user) + upload_artifact = to_str(upload_artifact) + password = to_str(password) + # Read content of upload artifact try: with open(upload_artifact, 'rb') as f: diff --git a/tools/src/main/python/dlpx/virtualization/_internal/delphix_client.py b/tools/src/main/python/dlpx/virtualization/_internal/delphix_client.py index 03b5799e..55d941f0 100644 --- a/tools/src/main/python/dlpx/virtualization/_internal/delphix_client.py +++ b/tools/src/main/python/dlpx/virtualization/_internal/delphix_client.py @@ -1,5 +1,5 @@ # -# Copyright (c) 2019 by Delphix. All rights reserved. +# Copyright (c) 2019, 2021 by Delphix. All rights reserved. # import json @@ -9,6 +9,7 @@ import requests from dlpx.virtualization._internal import exceptions, plugin_util +from dlpx.virtualization.common.util import to_bytes, to_str logger = logging.getLogger(__name__) @@ -36,12 +37,11 @@ def login(self, engine_api, user, password): Takes in the engine_api, user, and password and attempts to login to the engine. Can raise HttpPostError and UnexpectedError. """ - logger.info('Logging onto the Delphix Engine {!r}.'.format( - self.__engine)) + logger.info(f"Logging onto the Delphix Engine '{self.__engine}'.") self.__post('delphix/session', data={ - 'type': 'APISession', - 'version': engine_api + 'version': engine_api, + 'type': 'APISession' }) logger.debug('Session started successfully.') self.__post('delphix/login', @@ -50,7 +50,7 @@ def login(self, engine_api, user, password): 'username': user, 'password': password }) - logger.info('Successfully logged in as {!r}.'.format(user)) + logger.info(f"Successfully logged in as '{user}'.") @staticmethod def get_engine_api(artifact_content): @@ -70,7 +70,7 @@ def get_engine_api(artifact_content): json.dumps(engine_api))) return engine_api logger.debug( - 'engineApi found but malformed: {!r}'.format(engine_api)) + f"engineApi found but malformed: '{engine_api}'") raise exceptions.InvalidArtifactError() def __post(self, resource, content_type='application/json', data=None): @@ -93,7 +93,8 @@ def __post(self, resource, content_type='application/json', data=None): # Issue post request that was passed in, if data is a dict then convert # it to a json string. # - if data is not None and not isinstance(data, (str, bytes, unicode)): + if data is not None and not isinstance(data, (str, bytes)): + data = to_str(data) data = json.dumps(data) try: response = requests.post(url=url, data=data, headers=headers) @@ -211,7 +212,7 @@ def __download_logs(self, plugin_name, token, directory): "dlpx-plugin-logs-{}-{}.tar.gz".format(plugin_name, token)) with open(download_zip_name, "wb") as f: for chunk in download_zip_data: - f.write(chunk) + f.write(to_bytes(chunk)) def upload_plugin(self, name, content, wait): """ @@ -223,9 +224,9 @@ def upload_plugin(self, name, content, wait): logger.debug('Getting token to do upload.') response = self.__post('delphix/toolkit/requestUploadToken') token = response['result']['token'] - logger.debug('Got token {!r} successfully.'.format(token)) + logger.debug(f"Got token '{token}' successfully.") - logger.info('Uploading plugin {!r}.'.format(name)) + logger.info(f"Uploading plugin '{name}'.") # Encode plugin content. upload_response = self.__post('delphix/data/upload', content_type=self.__UPLOAD_CONTENT, @@ -306,8 +307,8 @@ def download_plugin_logs(self, directory, plugin_config): } response = self.__post('delphix/service/support/bundle/generate', data=data) - token = response['result'].encode('utf-8').strip() - logger.debug('Got token {!r} successfully.'.format(token)) + token = to_str(response['result'].encode('utf-8').strip()) + logger.debug(f"Got token '{token}' successfully.") self.__download_logs(plugin_name, token, directory) diff --git a/tools/src/main/python/dlpx/virtualization/_internal/exceptions.py b/tools/src/main/python/dlpx/virtualization/_internal/exceptions.py index 5013e2d5..decff5cd 100644 --- a/tools/src/main/python/dlpx/virtualization/_internal/exceptions.py +++ b/tools/src/main/python/dlpx/virtualization/_internal/exceptions.py @@ -1,11 +1,13 @@ # -# Copyright (c) 2019 by Delphix. All rights reserved. +# Copyright (c) 2019, 2021 by Delphix. All rights reserved. # import collections import json import re +from dlpx.virtualization.common.util import to_str + class SDKToolingError(Exception): """ @@ -255,14 +257,15 @@ def __format_error(err): 'type': 'object'} """ # - # Validation error message could be unicode encoded string. Strip out - # any leading unicode characters for proper display and logging. + # Validation error message could be byte string. Strip out + # any leading byte characters for proper display and logging. # - err_msg = re.compile(r'\bu\b', re.IGNORECASE) + err_msg = re.compile(r'\bb\b', re.IGNORECASE) err_msg = err_msg.sub("", err.message) + map_func = to_str error_string = 'Error: {} on {}'.format( - err_msg, map(str, list(err.schema_path))) + err_msg, list(map(map_func, list(err.schema_path)))) return error_string diff --git a/tools/src/main/python/dlpx/virtualization/_internal/package_util.py b/tools/src/main/python/dlpx/virtualization/_internal/package_util.py index 1eb75725..298256e5 100644 --- a/tools/src/main/python/dlpx/virtualization/_internal/package_util.py +++ b/tools/src/main/python/dlpx/virtualization/_internal/package_util.py @@ -1,5 +1,5 @@ # -# Copyright (c) 2019 by Delphix. All rights reserved. +# Copyright (c) 2019, 2021 by Delphix. All rights reserved. # import functools @@ -38,7 +38,7 @@ def _get_settings(): This assumes that the settings file is in the root of dlpx.virtualization._internal. """ - parser = configparser.SafeConfigParser() + parser = configparser.ConfigParser() parser.read(os.path.join(get_internal_package_root(), SETTINGS_FILE_NAME)) return parser diff --git a/tools/src/main/python/dlpx/virtualization/_internal/plugin_dependency_util.py b/tools/src/main/python/dlpx/virtualization/_internal/plugin_dependency_util.py index 1a90a154..4ae8ea88 100644 --- a/tools/src/main/python/dlpx/virtualization/_internal/plugin_dependency_util.py +++ b/tools/src/main/python/dlpx/virtualization/_internal/plugin_dependency_util.py @@ -1,14 +1,17 @@ # -# Copyright (c) 2019 by Delphix. All rights reserved. +# Copyright (c) 2019, 2021 by Delphix. All rights reserved. # +import compileall import logging import os +import py_compile import subprocess import sys from dlpx.virtualization._internal import file_util, package_util from dlpx.virtualization._internal.exceptions import SubprocessFailedError +from dlpx.virtualization.common.util import to_str logger = logging.getLogger(__name__) @@ -110,12 +113,12 @@ def _execute_pip(pip_args): """ args = [sys.executable, '-m', 'pip'] args.extend(pip_args) - logger.debug('Executing %s', ' '.join(args)) proc = subprocess.Popen(args, stdout=subprocess.PIPE, stderr=subprocess.STDOUT) all_output, _ = proc.communicate() + all_output = to_str(all_output) exit_code = proc.wait() # @@ -168,9 +171,29 @@ def _build_wheel(package_root, target_dir=None): cwd=package_root) all_output, _ = proc.communicate() + all_output = to_str(all_output) exit_code = proc.wait() if exit_code != 0: raise SubprocessFailedError(' '.join(args), exit_code, all_output) else: logger.debug(all_output) + + +def compile_py_files(dpath): + """ + Compiles the python files in the given directory, generating the pyc files in the + same directory. + """ + compileall.compile_dir(dpath, force=True, quiet=1, legacy=True, ddir=".") + + +def compile_py_file(fpath: str): + """ + Compiles the python file at the given path, generating the pyc file in the + same directory. + + :param fpath: + :return: + """ + py_compile.compile(fpath, cfile=f"{fpath}c") diff --git a/tools/src/main/python/dlpx/virtualization/_internal/plugin_importer.py b/tools/src/main/python/dlpx/virtualization/_internal/plugin_importer.py index f3ad2119..858ef5a9 100644 --- a/tools/src/main/python/dlpx/virtualization/_internal/plugin_importer.py +++ b/tools/src/main/python/dlpx/virtualization/_internal/plugin_importer.py @@ -1,7 +1,6 @@ # -# Copyright (c) 2019, 2020 by Delphix. All rights reserved. +# Copyright (c) 2019, 2021 by Delphix. All rights reserved. # -import importlib import logging import os import sys @@ -11,6 +10,7 @@ import yaml from dlpx.virtualization._internal import const, exceptions from dlpx.virtualization.platform import import_util +from dlpx.virtualization._internal import plugin_dependency_util logger = logging.getLogger(__name__) @@ -97,7 +97,6 @@ def __internal_import(self): self.__plugin_entry_point, self.__src_dir, err)) warnings['exception'].append(exception_msg) - return plugin_manifest, warnings @staticmethod @@ -161,7 +160,7 @@ def __run_checks(self, warnings): # warning_msg = exceptions.ValidationFailedError( warnings).message - logger.warn(warning_msg) + logger.warning(warning_msg) def __check_for_required_methods(self): """ @@ -189,8 +188,6 @@ def _import_module_and_get_manifest(queue, src_dir, module, entry_point, """ Imports the plugin module, runs validations and returns the manifest. """ - module_content = None - try: module_content = _import_helper(queue, src_dir, module) except exceptions.UserError: @@ -248,10 +245,29 @@ def _import_helper(queue, src_dir, module): exceptions. """ module_content = None - sys.path.append(src_dir) + try: + # + # Compile the plugin_runner module to be copied over to the docker container's + # plugin dir. + # The module comes in the format + # + appended_paths = _add_dirs_to_sys_path(src_dir) + plugin_dependency_util.compile_py_files(src_dir) + except FileNotFoundError: + # + # If the module couldn't be found, the user's entry point has the incorrect + # module name specified. + # + error = exceptions.UserError("No module named {}".format(module)) + queue.put({'exception': error}) try: - module_content = importlib.import_module(module) + # + # Module comes in the format pkg.[subpkg1.]*module. Since we've added the + # necessary paths to sys.path, we can import the module directly to retrieve + # its contents + # + module_content = __import__(module.split(".")[-1]) except (ImportError, TypeError) as err: queue.put({'exception': err}) except Exception as err: @@ -274,14 +290,32 @@ def _import_helper(queue, src_dir, module): error = exceptions.SDKToolingError(str(err)) queue.put({'sdk exception': error}) finally: - sys.path.remove(src_dir) + for p in appended_paths: + sys.path.remove(p) if not module_content: raise exceptions.UserError("Plugin module content is None") - return module_content +def _add_dirs_to_sys_path(src_dir, appended_paths=None): + if appended_paths is None: + appended_paths = [] + if src_dir not in sys.path: + sys.path.append(src_dir) + appended_paths.append(src_dir) + for root, dirs, _ in os.walk(src_dir): + for dir_name in dirs: + if dir_name in ["__pycache__"]: + continue + src_dir = os.path.sep.join([root, dir_name]) + if src_dir not in sys.path: + sys.path.append(src_dir) + appended_paths.append(src_dir) + _add_dirs_to_sys_path(src_dir, appended_paths=appended_paths) + return appended_paths + + def _process_warnings(queue, warnings): for warning in warnings: queue.put({'exception': warning}) @@ -330,6 +364,8 @@ def _prepare_manifest(entry_point, module_content): bool(plugin_object.virtual.unconfigure_impl), 'hasVirtualReconfigure': bool(plugin_object.virtual.reconfigure_impl), + 'hasVirtualCleanup': + bool(plugin_object.virtual.cleanup_impl), 'hasVirtualStart': bool(plugin_object.virtual.start_impl), 'hasVirtualStop': diff --git a/tools/src/main/python/dlpx/virtualization/_internal/plugin_validator.py b/tools/src/main/python/dlpx/virtualization/_internal/plugin_validator.py index 9608a2db..821b3587 100644 --- a/tools/src/main/python/dlpx/virtualization/_internal/plugin_validator.py +++ b/tools/src/main/python/dlpx/virtualization/_internal/plugin_validator.py @@ -1,5 +1,5 @@ # -# Copyright (c) 2019 by Delphix. All rights reserved. +# Copyright (c) 2019, 2021 by Delphix. All rights reserved. # import json @@ -12,6 +12,7 @@ from dlpx.virtualization._internal.codegen import CODEGEN_PACKAGE from flake8.api import legacy as flake8 from jsonschema import Draft7Validator +from dlpx.virtualization.common.util import to_bytes, to_str logger = logging.getLogger(__name__) @@ -33,6 +34,10 @@ def __init__(self, plugin_config, plugin_config_schema, plugin_config_content=None): + plugin_config = to_str(plugin_config) + plugin_config_schema = to_str(plugin_config_schema) + if plugin_config_content is not None: + plugin_config_content = to_str(plugin_config_content) self.__plugin_config = plugin_config self.__plugin_config_schema = plugin_config_schema self.__plugin_config_content = plugin_config_content @@ -83,7 +88,7 @@ def __read_plugin_config_file(self): try: with open(self.__plugin_config, 'rb') as f: try: - return yaml.safe_load(f) + return to_str(yaml.safe_load(f)) except yaml.YAMLError as err: if hasattr(err, 'problem_mark'): mark = err.problem_mark @@ -126,7 +131,7 @@ def __validate_plugin_config_content(self): schemaFile: the file containing defined schemas in the plugin manualDiscovery whether or not manual discovery is supported pluginType whether the plugin is DIRECT or STAGED - language language of the source code(ex: PYTHON27 for python2.7) + language language of the source code(ex: PYTHON38 for python3.8) Args: plugin_config_content (dict): A dictionary representing a plugin @@ -140,7 +145,7 @@ def __validate_plugin_config_content(self): try: with open(self.__plugin_config_schema, 'r') as f: try: - plugin_schema = json.load(f) + plugin_schema = to_str(json.load(f)) except ValueError as err: raise exceptions.UserError( 'Failed to load schemas because {} is not a ' @@ -155,8 +160,8 @@ def __validate_plugin_config_content(self): os.strerror(err.errno))) # Convert plugin config content to json - plugin_config_json = json.loads( - json.dumps(self.__plugin_config_content)) + plugin_config_json = to_str(json.loads( + to_bytes(json.dumps(self.__plugin_config_content)))) # Validate the plugin config against the schema v = Draft7Validator(plugin_schema) diff --git a/tools/src/main/python/dlpx/virtualization/_internal/schema_validator.py b/tools/src/main/python/dlpx/virtualization/_internal/schema_validator.py index 46354fce..75388f4b 100644 --- a/tools/src/main/python/dlpx/virtualization/_internal/schema_validator.py +++ b/tools/src/main/python/dlpx/virtualization/_internal/schema_validator.py @@ -1,5 +1,5 @@ # -# Copyright (c) 2019 by Delphix. All rights reserved. +# Copyright (c) 2019, 2021 by Delphix. All rights reserved. # import json @@ -8,6 +8,7 @@ from collections import namedtuple from dlpx.virtualization._internal import exceptions +from dlpx.virtualization.common.util import to_str from jsonschema import Draft7Validator logger = logging.getLogger(__name__) @@ -25,9 +26,9 @@ class SchemaValidator: back. """ def __init__(self, schema_file, plugin_meta_schema, schemas=None): - self.__schema_file = schema_file - self.__plugin_meta_schema = plugin_meta_schema - self.__plugin_schemas = schemas + self.__schema_file = to_str(schema_file) + self.__plugin_meta_schema = to_str(plugin_meta_schema) + self.__plugin_schemas = to_str(schemas) @property def result(self): @@ -55,7 +56,7 @@ def __read_schema_file(self): try: with open(self.__schema_file, 'r') as f: try: - return json.load(f) + return to_str(json.load(f)) except ValueError as err: raise exceptions.UserError( 'Failed to load schemas because \'{}\' is not a ' @@ -77,7 +78,7 @@ def __validate_schemas(self): try: with open(self.__plugin_meta_schema, 'r') as f: try: - plugin_meta_schema = json.load(f) + plugin_meta_schema = to_str(json.load(f)) except ValueError as err: raise exceptions.UserError( 'Failed to load schemas because \'{}\' is not a ' @@ -98,8 +99,12 @@ def __validate_schemas(self): # This will do lazy validation so that we can consolidate all the # validation errors and report everything wrong with the schema. # - validation_errors = sorted(v.iter_errors(self.__plugin_schemas), - key=lambda e: e.path) + # In Python 3.8, we are using jsonschema 4.X.X. This version of jsonschema + # breaks when we pass a dictionary to Draft7Validator.iter_errors(). + # Instead it expects a list. + # + errors = v.iter_errors(self.__plugin_schemas) + validation_errors = sorted(errors, key=lambda e: e.path) if validation_errors: raise exceptions.SchemaValidationError(self.__schema_file, diff --git a/tools/src/main/python/dlpx/virtualization/_internal/settings.cfg b/tools/src/main/python/dlpx/virtualization/_internal/settings.cfg index 700afb91..d71b6a5d 100644 --- a/tools/src/main/python/dlpx/virtualization/_internal/settings.cfg +++ b/tools/src/main/python/dlpx/virtualization/_internal/settings.cfg @@ -20,7 +20,7 @@ # versions in those packages until they are shipped out of band. # [General] -engine_api_version = 1.11.6 +engine_api_version = 1.11.11 distribution_name = dvp-tools package_author = Delphix namespace_package = dlpx diff --git a/tools/src/main/python/dlpx/virtualization/_internal/validation_schemas/plugin_config_schema.json b/tools/src/main/python/dlpx/virtualization/_internal/validation_schemas/plugin_config_schema.json index 15f80871..4c6f3f3f 100644 --- a/tools/src/main/python/dlpx/virtualization/_internal/validation_schemas/plugin_config_schema.json +++ b/tools/src/main/python/dlpx/virtualization/_internal/validation_schemas/plugin_config_schema.json @@ -38,7 +38,7 @@ }, "language": { "type": "string", - "enum": ["PYTHON27"] + "enum": ["PYTHON38"] }, "rootSquashEnabled": { "type": "boolean" diff --git a/tools/src/main/python/dlpx/virtualization/_internal/validation_schemas/plugin_importer.yaml b/tools/src/main/python/dlpx/virtualization/_internal/validation_schemas/plugin_importer.yaml index d769cd62..b04e88fc 100644 --- a/tools/src/main/python/dlpx/virtualization/_internal/validation_schemas/plugin_importer.yaml +++ b/tools/src/main/python/dlpx/virtualization/_internal/validation_schemas/plugin_importer.yaml @@ -52,6 +52,10 @@ EXPECTED_STAGED_ARGS_BY_OP: - repository - source_config - snapshot + cleanup_impl: + - virtual_source + - repository + - source_config start_impl: - virtual_source - repository @@ -112,6 +116,10 @@ EXPECTED_DIRECT_ARGS_BY_OP: - repository - source_config - snapshot + cleanup_impl: + - virtual_source + - repository + - source_config start_impl: - virtual_source - repository diff --git a/tools/src/main/python/dlpx/virtualization/_internal/validation_schemas/plugin_schema.json b/tools/src/main/python/dlpx/virtualization/_internal/validation_schemas/plugin_schema.json index b8be1836..edbfeab6 100644 --- a/tools/src/main/python/dlpx/virtualization/_internal/validation_schemas/plugin_schema.json +++ b/tools/src/main/python/dlpx/virtualization/_internal/validation_schemas/plugin_schema.json @@ -37,10 +37,6 @@ }, "type": ["object", "boolean"], "properties": { - "$id": { - "type": "string", - "format": "uri-reference" - }, "$schema": { "type": "string", "format": "uri" diff --git a/tools/src/test/python/dlpx/virtualization/_internal/commands/test_build.py b/tools/src/test/python/dlpx/virtualization/_internal/commands/test_build.py index bcc5a9ca..084e914e 100644 --- a/tools/src/test/python/dlpx/virtualization/_internal/commands/test_build.py +++ b/tools/src/test/python/dlpx/virtualization/_internal/commands/test_build.py @@ -1,5 +1,5 @@ # -# Copyright (c) 2019, 2020 by Delphix. All rights reserved. +# Copyright (c) 2019, 2021 by Delphix. All rights reserved. # import json @@ -35,7 +35,9 @@ class TestBuild: @mock.patch( 'dlpx.virtualization._internal.plugin_dependency_util.install_deps') @mock.patch('os.path.isabs', return_value=False) - def test_build_success(mock_relative_path, mock_install_deps, + @mock.patch( + 'dlpx.virtualization._internal.plugin_dependency_util.compile_py_files') + def test_build_success(mock_compile_py_files, mock_relative_path, mock_install_deps, mock_generate_python, mock_plugin_manifest, mock_patch_dependencies, plugin_config_file, artifact_file, artifact_content, @@ -72,12 +74,13 @@ def test_build_success(mock_relative_path, mock_install_deps, @mock.patch( 'dlpx.virtualization._internal.plugin_dependency_util.install_deps') @mock.patch('os.path.isabs', return_value=False) + @mock.patch( + 'dlpx.virtualization._internal.plugin_dependency_util.compile_py_files') def test_build_success_with_patched_dependencies( - mock_relative_path, mock_install_deps, + mock_compile_py_files, mock_relative_path, mock_install_deps, mock_generate_python, mock_plugin_manifest, plugin_config_file, artifact_file, codegen_gen_py_inputs, json_format_file_patched, json_format_content): - build.build(plugin_config_file, artifact_file, False, False) with open(json_format_file_patched, 'r') as f: @@ -95,10 +98,12 @@ def test_build_success_with_patched_dependencies( @mock.patch( 'dlpx.virtualization._internal.plugin_dependency_util.install_deps') @mock.patch('os.path.isabs', return_value=False) - def test_build_success_from_init(mock_relative_path, mock_install_deps, - mock_patch_dependencies, - tmpdir, ingestion_strategy, host_type, - plugin_name, artifact_file): + @mock.patch( + 'dlpx.virtualization._internal.plugin_dependency_util.compile_py_files') + def test_build_success_from_init( + mock_relative_path, mock_compile_py_files, mock_install_deps, + mock_patch_dependencies, tmpdir, ingestion_strategy, host_type, + plugin_name, artifact_file): # Initialize an empty directory. init.init(tmpdir.strpath, ingestion_strategy, plugin_name, host_type) plugin_config_file = os.path.join( @@ -124,11 +129,13 @@ def test_build_success_from_init(mock_relative_path, mock_install_deps, @mock.patch( 'dlpx.virtualization._internal.plugin_dependency_util.install_deps') @mock.patch('os.path.isabs', return_value=False) + @mock.patch( + 'dlpx.virtualization._internal.plugin_dependency_util.compile_py_files') def test_build_success_non_default_output_file( - mock_relative_path, mock_install_deps, mock_generate_python, - mock_import_plugin, mock_patch_dependencies, - plugin_config_file, artifact_file, - artifact_content, codegen_gen_py_inputs): + mock_relative_path, mock_compile_py_files, mock_install_deps, + mock_generate_python, mock_import_plugin, mock_patch_dependencies, + plugin_config_file, artifact_file, artifact_content, + codegen_gen_py_inputs): gen_py = codegen_gen_py_inputs # Before running build assert that the artifact file does not exist. @@ -236,13 +243,13 @@ def test_build_manifest_fail(mock_relative_path, mock_install_deps, @mock.patch( 'dlpx.virtualization._internal.plugin_dependency_util.install_deps') @mock.patch('os.path.isabs', return_value=False) - def test_build_prepare_artifact_fail(mock_relative_path, mock_install_deps, - mock_generate_python, - mock_plugin_manifest, - mock_patch_dependencies, - mock_prep_artifact, - plugin_config_file, artifact_file, - codegen_gen_py_inputs): + @mock.patch( + 'dlpx.virtualization._internal.plugin_dependency_util.compile_py_files') + def test_build_prepare_artifact_fail( + mock_compile_py_files, mock_relative_path, mock_install_deps, + mock_generate_python, mock_plugin_manifest, mock_patch_dependencies, + mock_prep_artifact, plugin_config_file, artifact_file, + codegen_gen_py_inputs): gen_py = codegen_gen_py_inputs # Before running build assert that the artifact file does not exist. @@ -281,11 +288,13 @@ def test_build_prepare_artifact_fail(mock_relative_path, mock_install_deps, @mock.patch( 'dlpx.virtualization._internal.plugin_dependency_util.install_deps') @mock.patch('os.path.isabs', return_value=False) + @mock.patch( + 'dlpx.virtualization._internal.plugin_dependency_util.compile_py_files') def test_build_generate_artifact_fail( - mock_relative_path, mock_install_deps, mock_generate_python, - mock_plugin_manifest, mock_patch_dependencies, - mock_gen_artifact, plugin_config_file, - artifact_file, codegen_gen_py_inputs): + mock_compile_py_files, mock_relative_path, mock_install_deps, + mock_generate_python, mock_plugin_manifest, mock_patch_dependencies, + mock_gen_artifact, plugin_config_file, artifact_file, + codegen_gen_py_inputs): gen_py = codegen_gen_py_inputs # Before running build assert that the artifact file does not exist. @@ -419,10 +428,12 @@ def test_zip_and_encode_source_files_encode_fail(mock_encode, src_dir): @mock.patch( 'dlpx.virtualization._internal.plugin_dependency_util.install_deps') @mock.patch('os.path.isabs', return_value=False) + @mock.patch( + 'dlpx.virtualization._internal.plugin_dependency_util.compile_py_files') @pytest.mark.parametrize('plugin_id', ['77f18ce4-4425-4cd6-b9a7-23653254d660']) - def test_id_validation_positive(mock_relative_path, mock_install_deps, - mock_patch_dependencies, + def test_id_validation_positive(mock_compile_py_files, mock_relative_path, + mock_install_deps, mock_patch_dependencies, mock_import_plugin, plugin_config_file, artifact_file): build.build(plugin_config_file, artifact_file, False) @@ -497,7 +508,7 @@ def test_plugin_bad_language(mock_generate_python, plugin_config_file, build.build(plugin_config_file, artifact_file, False, False) message = err_info.value.message - assert "'BAD_LANGUAGE' is not one of ['PYTHON27']" in message + assert "'BAD_LANGUAGE' is not one of ['PYTHON38']" in message assert not mock_generate_python.called @@ -603,9 +614,9 @@ def test_schema_bad_format(mock_generate_python, plugin_config_file, message = err_info.value.message assert ( - 'Failed to load schemas because \'{}\' is not a valid json file.' - ' Error: Extra data: line 2 column 1 - line 2 column 9' - ' (char 19 - 27)'.format(schema_file)) in message + "Failed to load schemas because '{}' is not a valid json file." + " Error: Extra data: line 2 column 1 (char 19) \n\nBUILD" + " FAILED.").format(schema_file) in message assert not mock_generate_python.called @@ -829,5 +840,5 @@ def test_non_existing_entry_file(mock_relative_path, plugin_config_file, build.build(plugin_config_file, artifact_file, False, False) message = err_info.value.message - exp_message = "No module named {module}".format(module=entry_module) + exp_message = "No module named \'{module}\'".format(module=entry_module) assert exp_message in message diff --git a/tools/src/test/python/dlpx/virtualization/_internal/commands/test_codegen.py b/tools/src/test/python/dlpx/virtualization/_internal/commands/test_codegen.py index ca6e3f84..b0fce144 100644 --- a/tools/src/test/python/dlpx/virtualization/_internal/commands/test_codegen.py +++ b/tools/src/test/python/dlpx/virtualization/_internal/commands/test_codegen.py @@ -1,5 +1,5 @@ # -# Copyright (c) 2019, 2020 by Delphix. All rights reserved. +# Copyright (c) 2019, 2021 by Delphix. All rights reserved. # import errno @@ -287,9 +287,9 @@ def test_execute_swagger_codegen_jar_issue(tmpdir, schema_content, codegen._execute_swagger_codegen(swagger_file, tmpdir.strpath) message = err_info.value.message - assert message == ('Unable to run {!r} to generate python code.' - '\nError code: 23. Error message: Too many open' - ' files in system'.format(popen_helper.jar)) + assert message == ( + f"Unable to run '{popen_helper.jar}' to generate python code.\nError" + " code: 23. Error message: Too many open files in system") assert popen_helper.stdout_input == subprocess.PIPE assert popen_helper.stderr_input == subprocess.PIPE diff --git a/tools/src/test/python/dlpx/virtualization/_internal/commands/test_delphix_client.py b/tools/src/test/python/dlpx/virtualization/_internal/commands/test_delphix_client.py index 8396089a..e21fb584 100644 --- a/tools/src/test/python/dlpx/virtualization/_internal/commands/test_delphix_client.py +++ b/tools/src/test/python/dlpx/virtualization/_internal/commands/test_delphix_client.py @@ -1,5 +1,5 @@ # -# Copyright (c) 2019 by Delphix. All rights reserved. +# Copyright (c) 2019, 2021 by Delphix. All rights reserved. # import json @@ -7,6 +7,7 @@ import requests from dlpx.virtualization._internal import delphix_client, exceptions +from dlpx.virtualization.common.util import to_str import httpretty import mock @@ -403,11 +404,11 @@ def test_delphix_client_unknown_error(engine_api): assert err_info.value.status_code == 404 assert err_info.value.response == ( - '{\n "status": "UNKNOWN", \n "blob": "Unknown"\n}') + '{\n "blob": "Unknown",\n "status": "UNKNOWN"\n}') + assert err_info.value.message == ( 'Received an unexpected error with HTTP Status 404,\nDumping full' - ' response:\n{\n "status": "UNKNOWN", \n "blob": "Unknown"\n}') - + ' response:\n{\n "blob": "Unknown",\n "status": "UNKNOWN"\n}') history = httpretty.HTTPretty.latest_requests assert history[-1].path == u'/resources/json/delphix/session' @@ -471,17 +472,18 @@ def test_delphix_client_wrong_login_no_detail(engine_api): message = err_info.value.message assert err_info.value.status_code == 401 - assert message == ('API request failed with HTTP Status 401' - '\nUnable to parse details of error.' - ' Dumping full response: {' - '\n "commandOutput": null, ' - '\n "diagnoses": [], ' - '\n "type": "APIError", ' - '\n "id": "exception.webservices.login.failed", ' - '\n "error": "Not a real error: Invalid username' - ' or password. Try with a different set of' - ' credentials."' - '\n}') + expected_message = ('API request failed with HTTP Status 401' + '\nUnable to parse details of error.' + ' Dumping full response: {' + '\n "type": "APIError",' + '\n "error": "Not a real error: Invalid username' + ' or password. Try with a different set of' + ' credentials.",' + '\n "id": "exception.webservices.login.failed",' + '\n "commandOutput": null,' + '\n "diagnoses": []' + '\n}') + assert message == expected_message history = httpretty.HTTPretty.latest_requests assert history[-1].path == u'/resources/json/delphix/login' @@ -726,10 +728,12 @@ def test_delphix_client_download_success(engine_api, src_dir, dc.download_plugin_logs(src_dir, plugin_config_file) history = httpretty.HTTPretty.latest_requests - assert (history[-1].path == + to_str(history[-1].__dict__) + + assert (to_str(history[-1].path) == u'/resources/json/delphix/data/downloadOutputStream' u'?token=5d6d5bb8-0f71-4304-8922-49c4c95c2387') - assert history[-2].path == ( + assert to_str(history[-2].path) == ( u'/resources/json/delphix/service/support/bundle/generate') assert history[-3].path == u'/resources/json/delphix/toolkit' assert history[-4].path == u'/resources/json/delphix/login' @@ -760,14 +764,15 @@ def test_validate_fail(artifact_content): delphix_client.DelphixClient.get_engine_api(artifact_content) message = err_info.value.message - assert message == ( + expected_message = ( 'The engineApi field is either missing or malformed.' ' The field must be of the form:' '\n{' '\n "type": "APIVersion",' - ' \n "major": 1,' - ' \n "minor": 7,' - ' \n "micro": 0' + '\n "major": 1,' + '\n "minor": 7,' + '\n "micro": 0' '\n}' '\nVerify that the artifact passed in was generated' ' by the build function.') + assert message == expected_message diff --git a/tools/src/test/python/dlpx/virtualization/_internal/commands/test_initialize.py b/tools/src/test/python/dlpx/virtualization/_internal/commands/test_initialize.py index 62fa9dfe..fe54a998 100644 --- a/tools/src/test/python/dlpx/virtualization/_internal/commands/test_initialize.py +++ b/tools/src/test/python/dlpx/virtualization/_internal/commands/test_initialize.py @@ -1,5 +1,5 @@ # -# Copyright (c) 2019, 2020 by Delphix. All rights reserved. +# Copyright (c) 2019, 2021 by Delphix. All rights reserved. # import ast @@ -12,6 +12,7 @@ from dlpx.virtualization._internal import (const, exceptions, plugin_util, plugin_validator, schema_validator) from dlpx.virtualization._internal.commands import initialize as init +from dlpx.virtualization.common.util import to_str @pytest.fixture @@ -63,6 +64,29 @@ def format_template(plugin_name, ingestion_strategy, host_type): raise RuntimeError( 'Got unrecognized ingestion strategy: {}'.format( ingestion_strategy)) + + # + # PY2 VS PY3 STRING/BYTES/UNICODE REPRESENTATIONS + # + # repr(("")) + # | PY2 | PY3 | + # -------------------------------------------------------- + # repr(str("1")) | "'1'" | '1' | + # -------------------------------------------------------- + # repr(bytes("1", "utf-8")) | N/A | b'1' | + # -------------------------------------------------------- + # repr(bytes("1")) | '1' | N/A | + # -------------------------------------------------------- + # repr(unicode("1")) | u'1' | N/A | + # -------------------------------------------------------- + # + # Looking at the above table, we need to do one of the following to get a + # representation that is not prepended with `u` or `b`: + # 1) if six.PY2: plugin_name = to_bytes(plugin_name) + # 2) if six.PY3: plugin_name = to_str(plugin_name) (to_str will return a + # unicode string in PY2) + # + plugin_name = to_str(plugin_name) return template.render(name=repr(plugin_name), linked_operations=operations, default_mount_path=default_mount_path) @@ -110,9 +134,10 @@ def test_init(tmpdir, ingestion_strategy, host_type, schema_template, entry_file_path = os.path.join(tmpdir.strpath, config['srcDir'], entry_file) with open(entry_file_path, 'r') as f: - contents = f.read() - assert contents == format_entry_point_template( - config['id'], ingestion_strategy, host_type) + contents = to_str(f.read()) + expected_contents = to_str(format_entry_point_template( + config['id'], ingestion_strategy, host_type)) + assert contents == expected_contents @staticmethod def test_init_with_relative_path(tmpdir): @@ -217,9 +242,8 @@ def test_init_calls_cleanup_on_failure(mock_cleanup, mock_yaml_dump, @staticmethod def test_default_schema_definition(schema_template): validator = schema_validator.SchemaValidator(None, const.PLUGIN_SCHEMA, - schema_template) + schemas=schema_template) validator.validate() - # Validate the repository schema only has the 'name' property. assert len(schema_template['repositoryDefinition'] ['properties']) == 1, json.dumps( diff --git a/tools/src/test/python/dlpx/virtualization/_internal/commands/test_templates.py b/tools/src/test/python/dlpx/virtualization/_internal/commands/test_templates.py index bb7e9d89..7da59ca3 100644 --- a/tools/src/test/python/dlpx/virtualization/_internal/commands/test_templates.py +++ b/tools/src/test/python/dlpx/virtualization/_internal/commands/test_templates.py @@ -1,15 +1,17 @@ # -# Copyright (c) 2019 by Delphix. All rights reserved. +# Copyright (c) 2019, 2021 by Delphix. All rights reserved. # import importlib import itertools import json import os +import re import subprocess import sys from dlpx.virtualization._internal import codegen +from dlpx.virtualization.common.util import to_bytes, to_str import pytest @@ -35,7 +37,7 @@ def module(tmp_factory, schema_content): # create the config file to point to the tmpdir config_dict = {'packageName': tmpdir.name} config_file = basedir.joinpath('codegen-config.json') - config_file.write_bytes(json.dumps(config_dict, indent=2)) + config_file.write_bytes(to_bytes(json.dumps(config_dict, indent=2))) execute_swagger_codegen(swagger_file, str(config_file), str(basedir)) return importlib.import_module('.definitions', package=tmpdir.name) @@ -60,6 +62,8 @@ def execute_swagger_codegen(swagger_file, config_file, output_dir): # Get the pipes pointed so we have access to them. stdout, stderr = process.communicate() + stdout = to_str(stdout) + stderr = to_str(stderr) # # Wait for the process to end and take the results. If res then we know @@ -109,14 +113,14 @@ def schema_content(): 'type': 'object', 'additionalProperties': False, 'properties': { + 'requiredStringProperty': { + 'type': 'string', + 'pattern': "^test.*" + }, 'stringProperty': { 'type': 'string', 'minLength': 5, 'maxLength': 10 - }, - 'requiredStringProperty': { - 'type': 'string', - 'pattern': "^test.*" } }, 'required': ['requiredStringProperty'] @@ -198,7 +202,7 @@ def test_required_param_missing(module): @staticmethod def test_required_param_missing_setter(module): - test_object = module.TestDefinition('test string') + test_object = module.TestDefinition(required_string_property='test string') with pytest.raises(module.GeneratedClassesError) as err_info: test_object.required_string_property = None @@ -209,16 +213,17 @@ def test_required_param_missing_setter(module): @staticmethod def test_not_string(module): with pytest.raises(module.GeneratedClassesTypeError) as err_info: - module.TestDefinition('test string', 10) + module.TestDefinition( + required_string_property='test string', string_property=10) message = err_info.value.message assert message == ( "TestDefinition's parameter 'string_property' was" - " type 'int' but should be of type 'basestring' if defined.") + " class 'int' but should be of class 'str' if defined.") @staticmethod def test_not_string_setter(module): - test_object = module.TestDefinition('test string') + test_object = module.TestDefinition(required_string_property='test string') with pytest.raises(module.GeneratedClassesTypeError) as err_info: test_object.string_property = 10 @@ -226,12 +231,13 @@ def test_not_string_setter(module): message = err_info.value.message assert message == ( "TestDefinition's parameter 'string_property' was" - " type 'int' but should be of type 'basestring' if defined.") + " class 'int' but should be of class 'str' if defined.") @staticmethod def test_min_length(module): with pytest.raises(module.GeneratedClassesError) as err_info: - module.TestDefinition('test string', 'test') + module.TestDefinition( + required_string_property='test string', string_property='test') message = err_info.value.message assert message == ("Invalid value for 'string_property', length was 4" @@ -239,7 +245,7 @@ def test_min_length(module): @staticmethod def test_min_length_setter(module): - test_object = module.TestDefinition('test string') + test_object = module.TestDefinition(required_string_property='test string') with pytest.raises(module.GeneratedClassesError) as err_info: test_object.string_property = 'test' @@ -251,7 +257,9 @@ def test_min_length_setter(module): @staticmethod def test_max_length(module): with pytest.raises(module.GeneratedClassesError) as err_info: - module.TestDefinition('test string', 'test too long of string') + module.TestDefinition( + required_string_property='test string', + string_property='test too long of string') message = err_info.value.message assert message == ("Invalid value for 'string_property', length was 23" @@ -259,7 +267,7 @@ def test_max_length(module): @staticmethod def test_max_length_setter(module): - test_object = module.TestDefinition('test string') + test_object = module.TestDefinition(required_string_property='test string') with pytest.raises(module.GeneratedClassesError) as err_info: test_object.string_property = 'test too long of string' @@ -271,7 +279,7 @@ def test_max_length_setter(module): @staticmethod def test_bad_pattern(module): with pytest.raises(module.GeneratedClassesError) as err_info: - module.TestDefinition('bad test string') + module.TestDefinition(required_string_property='bad test string') message = err_info.value.message assert message == ("Invalid value for 'required_string_property'," @@ -280,7 +288,7 @@ def test_bad_pattern(module): @staticmethod def test_bad_pattern_setter(module): - test_object = module.TestDefinition('test string') + test_object = module.TestDefinition(required_string_property='test string') with pytest.raises(module.GeneratedClassesError) as err_info: test_object.required_string_property = 'bad test string' @@ -327,7 +335,9 @@ def schema_content(): @staticmethod def test_success(module): - test_object = module.TestDefinition(200.5, None, -50) + test_object = module.TestDefinition( + required_number_property=200.5, number_property=None, + required_integer_property=-50) assert test_object.required_number_property == 200.5 assert not test_object.number_property @@ -344,7 +354,9 @@ def test_success(module): @staticmethod def test_success_setter(module): - test_object = module.TestDefinition(200.5, None, -50) + test_object = module.TestDefinition( + required_number_property=200.5, number_property=None, + required_integer_property=-50) test_object.number_property = 13.5 test_object.integer_property = 18 @@ -365,7 +377,9 @@ def test_success_setter(module): @staticmethod def test_success_number_is_int(module): - test_object = module.TestDefinition(200, 13, -50, 18) + test_object = module.TestDefinition( + required_number_property=200, number_property=13, + required_integer_property=-50, integer_property=18) assert test_object.required_number_property == 200 assert test_object.number_property == 13 @@ -375,12 +389,13 @@ def test_success_number_is_int(module): @staticmethod def test_int_passed_in_as_float(module): with pytest.raises(module.GeneratedClassesTypeError) as err_info: - module.TestDefinition(200.5, 13.5, -50.5, 18.5) + module.TestDefinition( + required_number_property=200.5, required_integer_property=2.2) message = err_info.value.message assert message == ( "TestDefinition's parameter 'required_integer_property' was" - " type 'float' but should be of type 'int'.") + " class 'float' but should be of class 'int'.") @staticmethod def test_required_param_missing(module): @@ -392,21 +407,25 @@ def test_required_param_missing(module): " must not be 'None'.") with pytest.raises(module.GeneratedClassesError) as err_info: - module.TestDefinition(200.5) + module.TestDefinition(required_number_property=200.5) message = err_info.value.message assert message == ("The required parameter 'required_integer_property'" " must not be 'None'.") with pytest.raises(module.GeneratedClassesError) as err_info: - module.TestDefinition(None, 13.5, -50, 18) + module.TestDefinition( + required_number_property=None, number_property=13.5, + required_integer_property=-50, integer_property=18) message = err_info.value.message assert message == ("The required parameter 'required_number_property'" " must not be 'None'.") with pytest.raises(module.GeneratedClassesError) as err_info: - module.TestDefinition(200.5, 13.5, None, 18) + module.TestDefinition( + required_number_property=200.5, number_property=13.5, + required_integer_property=None, integer_property=18) message = err_info.value.message assert message == ("The required parameter 'required_integer_property'" @@ -414,7 +433,9 @@ def test_required_param_missing(module): @staticmethod def test_required_param_missing_setter(module): - test_object = module.TestDefinition(200.5, None, -50) + test_object = module.TestDefinition( + required_number_property=200.5, number_property=None, + required_integer_property=-50) with pytest.raises(module.GeneratedClassesError) as err_info: test_object.required_number_property = None @@ -432,45 +453,53 @@ def test_required_param_missing_setter(module): @staticmethod def test_not_number(module): with pytest.raises(module.GeneratedClassesTypeError) as err_info: - module.TestDefinition('string', None, -50) + module.TestDefinition( + required_number_property='string', number_property=None, + required_integer_property=-50) message = err_info.value.message assert message == ( "TestDefinition's parameter 'required_number_property' was" - " type 'str' but should be of type 'float'.") + " class 'str' but should be of class 'float'.") with pytest.raises(module.GeneratedClassesTypeError) as err_info: - module.TestDefinition(200.5, None, 'string') + module.TestDefinition( + required_number_property=200.5, number_property=None, + required_integer_property='string') message = err_info.value.message assert message == ( - "TestDefinition's parameter 'required_integer_property' was" - " type 'str' but should be of type 'int'.") + "TestDefinition's parameter 'required_integer_property' was" + " class 'str' but should be of class 'int'.") @staticmethod def test_not_number_setter(module): - test_object = module.TestDefinition(200.5, None, -50) + test_object = module.TestDefinition( + required_number_property=200.5, number_property=None, + required_integer_property=-50) with pytest.raises(module.GeneratedClassesTypeError) as err_info: test_object.number_property = 'string' message = err_info.value.message assert message == ( - "TestDefinition's parameter 'number_property' was" - " type 'str' but should be of type 'float' if defined.") + "TestDefinition's parameter 'number_property' was" + " class 'str' but should be of class 'float' if defined.") with pytest.raises(module.GeneratedClassesTypeError) as err_info: test_object.integer_property = 'string' message = err_info.value.message assert message == ( - "TestDefinition's parameter 'integer_property' was" - " type 'str' but should be of type 'int' if defined.") + "TestDefinition's parameter 'integer_property' was" + " class 'str' but should be of class 'int' if defined.") @staticmethod def test_minimum(module): with pytest.raises(module.GeneratedClassesError) as err_info: - module.TestDefinition(200.5, 1.0, -50) + module.TestDefinition( + required_number_property=200.5, number_property=1.0, + required_integer_property=-50) message = err_info.value.message assert message == ("Invalid value for 'number_property', value was 1.0" @@ -478,7 +507,9 @@ def test_minimum(module): @staticmethod def test_minimum_setter(module): - test_object = module.TestDefinition(200.5, None, -50) + test_object = module.TestDefinition( + required_number_property=200.5, number_property=None, + required_integer_property=-50) with pytest.raises(module.GeneratedClassesError) as err_info: test_object.number_property = 1.0 @@ -490,7 +521,9 @@ def test_minimum_setter(module): @staticmethod def test_maximum(module): with pytest.raises(module.GeneratedClassesError) as err_info: - module.TestDefinition(200.5, 13.5, -50, 21) + module.TestDefinition( + required_number_property=200.5, number_property=13.5, + required_integer_property=-50, integer_property=21) message = err_info.value.message assert message == ("Invalid value for 'integer_property', value was 21" @@ -498,7 +531,9 @@ def test_maximum(module): @staticmethod def test_maximum_setter(module): - test_object = module.TestDefinition(200.5, None, -50, None) + test_object = module.TestDefinition( + required_number_property=200.5, number_property=None, + required_integer_property=-50, integer_property=None) with pytest.raises(module.GeneratedClassesError) as err_info: test_object.integer_property = 21 @@ -510,7 +545,9 @@ def test_maximum_setter(module): @staticmethod def test_exclusive_minimum(module): with pytest.raises(module.GeneratedClassesError) as err_info: - module.TestDefinition(2.0, 13.5, -50) + module.TestDefinition( + required_number_property=2.0, number_property=13.5, + required_integer_property=-50) message = err_info.value.message assert message == ("Invalid value for 'required_number_property'," @@ -518,7 +555,9 @@ def test_exclusive_minimum(module): @staticmethod def test_exclusive_minimum_setter(module): - test_object = module.TestDefinition(200.5, None, -50) + test_object = module.TestDefinition( + required_number_property=200.5, number_property=None, + required_integer_property=-50) with pytest.raises(module.GeneratedClassesError) as err_info: test_object.required_number_property = 2.0 @@ -530,7 +569,7 @@ def test_exclusive_minimum_setter(module): @staticmethod def test_exclusive_maximum(module): with pytest.raises(module.GeneratedClassesError) as err_info: - module.TestDefinition(200.5, 13.5, 100) + module.TestDefinition(200.5, 13.5, required_integer_property=100) message = err_info.value.message assert message == ("Invalid value for 'required_integer_property'," @@ -538,7 +577,9 @@ def test_exclusive_maximum(module): @staticmethod def test_exclusive_maximum_setter(module): - test_object = module.TestDefinition(200.5, None, -50) + test_object = module.TestDefinition( + required_number_property=200.5, number_property=None, + required_integer_property=-50) with pytest.raises(module.GeneratedClassesError) as err_info: test_object.required_integer_property = 100 @@ -698,8 +739,8 @@ def test_object_is_string(module): message = err_info.value.message assert message == ( - "TestDefinition's parameter 'required_object_property' was type" - " 'str' but should be of a dict with keys type 'basestring'.") + "TestDefinition's parameter 'required_object_property' was class" + " 'str' but should be of a dict with keys type 'str'.") @staticmethod def test_object_is_string_setter(module): @@ -711,8 +752,8 @@ def test_object_is_string_setter(module): message = err_info.value.message assert message == ( - "TestDefinition's parameter 'required_object_property' was type" - " 'str' but should be of a dict with keys type 'basestring'.") + "TestDefinition's parameter 'required_object_property' was class" + " 'str' but should be of a dict with keys type 'str'.") @staticmethod def test_object_is_array(module): @@ -722,8 +763,8 @@ def test_object_is_array(module): message = err_info.value.message assert message == ( - "TestDefinition's parameter 'required_object_property' was type" - " 'list' but should be of a dict with keys type 'basestring'.") + "TestDefinition's parameter 'required_object_property' was class" + " 'list' but should be of a dict with keys type 'str'.") @staticmethod def test_object_is_array_setter(module): @@ -735,8 +776,8 @@ def test_object_is_array_setter(module): message = err_info.value.message assert message == ( - "TestDefinition's parameter 'required_object_property' was type" - " 'list' but should be of a dict with keys type 'basestring'.") + "TestDefinition's parameter 'required_object_property' was class" + " 'list' but should be of a dict with keys type 'str'.") @staticmethod def test_object_dict_with_bad_key_type(module): @@ -748,9 +789,9 @@ def test_object_dict_with_bad_key_type(module): }) expected_msg_template = ( - "TestDefinition's parameter 'required_object_property' was a dict" - " with keys of {{{}}} but should be" - " of a dict with keys type 'basestring'.") + "TestDefinition's parameter 'required_object_property' was a dict" + " with keys of {{{}}} but should be" + " of a dict with keys type 'str'.") possible_messages = create_possible_expected_messages( expected_msg_template, [str, int, bool]) @@ -774,9 +815,10 @@ def test_object_dict_with_bad_key_type_setter(module): } expected_msg_template = ( - "TestDefinition's parameter 'required_object_property' was a dict" - " with keys of {{{}}} but should be" - " of a dict with keys type 'basestring'.") + "TestDefinition's parameter 'required_object_property' was a dict" + " with keys of {{{}}} but should be" + " of a dict with keys type 'str'.") + possible_messages = create_possible_expected_messages( expected_msg_template, [str, int, bool]) @@ -795,9 +837,9 @@ def test_semi_defined_dict_not_bool_value_type(module): message = err_info.value.message assert message == ( - "TestDefinition's parameter 'boolean_dict_property' was a" - " dict of {type 'str':type 'str'} but should be of type 'dict of" - " basestring:bool' if defined.") + "TestDefinition's parameter 'boolean_dict_property' was a" + " dict of {class 'str':class 'str'} but should be of type 'dict of" + " str:bool' if defined.") @staticmethod def test_semi_defined_dict_not_bool_value_type_setter(module): @@ -809,9 +851,9 @@ def test_semi_defined_dict_not_bool_value_type_setter(module): message = err_info.value.message assert message == ( - "TestDefinition's parameter 'boolean_dict_property' was a" - " dict of {type 'str':type 'str'} but should be of type 'dict of" - " basestring:bool' if defined.") + "TestDefinition's parameter 'boolean_dict_property' was a" + " dict of {class 'str':class 'str'} but should be of type 'dict of" + " str:bool' if defined.") @staticmethod def test_semi_defined_dict_not_number_value_type(module): @@ -822,8 +864,8 @@ def test_semi_defined_dict_not_number_value_type(module): message = err_info.value.message assert message == ( "TestDefinition's parameter 'number_dict_property' was a" - " dict of {type 'str':type 'str'} but should be of type 'dict of" - " basestring:float' if defined.") + " dict of {class 'str':class 'str'} but should be of type 'dict of" + " str:float' if defined.") @staticmethod def test_semi_defined_dict_not_number_value_type_setter(module): @@ -836,8 +878,8 @@ def test_semi_defined_dict_not_number_value_type_setter(module): message = err_info.value.message assert message == ( "TestDefinition's parameter 'number_dict_property' was a" - " dict of {type 'str':type 'str'} but should be of type 'dict of" - " basestring:float' if defined.") + " dict of {class 'str':class 'str'} but should be of type 'dict of" + " str:float' if defined.") @staticmethod def test_internal_class_is_string(module): @@ -847,7 +889,7 @@ def test_internal_class_is_string(module): message = err_info.value.message assert message == ( - "TestDefinition's parameter 'defined_object_property' was type" + "TestDefinition's parameter 'defined_object_property' was class" " 'str' but should be of class" " '{}.TestDefinitionDefinedObjectProperty' if defined.".format( module.TestDefinitionDefinedObjectProperty.__module__)) @@ -862,7 +904,7 @@ def test_internal_class_is_string_setter(module): message = err_info.value.message assert message == ( - "TestDefinition's parameter 'defined_object_property' was type" + "TestDefinition's parameter 'defined_object_property' was class" " 'str' but should be of class" " '{}.TestDefinitionDefinedObjectProperty' if defined.".format( module.TestDefinitionDefinedObjectProperty.__module__)) @@ -878,8 +920,10 @@ class TestOtherClass: message = err_info.value.message assert message == ( - "TestDefinition's parameter 'defined_object_property' was type" - " 'instance' but should be of class" + "TestDefinition's parameter 'defined_object_property' was class" + " 'dlpx.virtualization._internal.commands.test_templates" + ".TestTemplateObjectProperty.test_internal_class_is_other_class" + ".locals.TestOtherClass' but should be of class" " '{}.TestDefinitionDefinedObjectProperty' if defined.".format( module.TestDefinitionDefinedObjectProperty.__module__)) @@ -896,9 +940,12 @@ class TestOtherClass: message = err_info.value.message assert message == ( - "TestDefinition's parameter 'defined_object_property' was type" - " 'instance' but should be of class" - " '{}.TestDefinitionDefinedObjectProperty' if defined.".format( + "TestDefinition's parameter 'defined_object_property' was class" + " 'dlpx.virtualization._internal.commands.test_templates" + ".TestTemplateObjectProperty" + ".test_internal_class_is_other_class_setter.locals.TestOtherClass' " + "but should be of class '{}.TestDefinitionDefinedObjectProperty' " + "if defined.".format( module.TestDefinitionDefinedObjectProperty.__module__)) @@ -999,8 +1046,8 @@ def test_array_is_string(module): message = err_info.value.message assert message == ( - "TestDefinition's parameter 'required_array_property' was type" - " 'str' but should be of type 'list of float'.") + "TestDefinition's parameter 'required_array_property' was class" + " 'str' but should be of type 'list of float'.") @staticmethod def test_array_is_string_setter(module): @@ -1011,8 +1058,8 @@ def test_array_is_string_setter(module): message = err_info.value.message assert message == ( - "TestDefinition's parameter 'required_array_property' was type" - " 'str' but should be of type 'list of float'.") + "TestDefinition's parameter 'required_array_property' was class" + " 'str' but should be of type 'list of float'.") @staticmethod def test_array_is_object(module): @@ -1025,8 +1072,8 @@ def test_array_is_object(module): message = err_info.value.message assert message == ( - "TestDefinition's parameter 'array_property' was type" - " 'dict' but should be of type 'list' if defined.") + "TestDefinition's parameter 'array_property' was class" + " 'dict' but should be of class 'list' if defined.") @staticmethod def test_array_is_object_setter(module): @@ -1037,8 +1084,8 @@ def test_array_is_object_setter(module): message = err_info.value.message assert message == ( - "TestDefinition's parameter 'array_property' was type" - " 'dict' but should be of type 'list' if defined.") + "TestDefinition's parameter 'array_property' was class" + " 'dict' but should be of class 'list' if defined.") @staticmethod def test_number_array_wrong_elem_types(module): @@ -1047,9 +1094,9 @@ def test_number_array_wrong_elem_types(module): message = err_info.value.message assert message == ( - "TestDefinition's parameter 'required_array_property' was a list" - " of [type 'str', type 'bool'] but should be of type 'list of" - " float'.") + "TestDefinition's parameter 'required_array_property' was a list" + " of [class 'str', class 'bool'] but should be of type 'list of" + " float'.") @staticmethod def test_number_array_wrong_elem_types_setter(module): @@ -1060,9 +1107,9 @@ def test_number_array_wrong_elem_types_setter(module): message = err_info.value.message assert message == ( - "TestDefinition's parameter 'required_array_property' was a list" - " of [type 'str', type 'bool'] but should be of type 'list of" - " float'.") + "TestDefinition's parameter 'required_array_property' was a list" + " of [class 'str', class 'bool'] but should be of type 'list of" + " float'.") @staticmethod def test_string_array_wrong_elem_types(module): @@ -1073,8 +1120,8 @@ def test_string_array_wrong_elem_types(module): message = err_info.value.message assert message == ( "TestDefinition's parameter 'string_array_property' was a list of" - " [type 'str', type 'int', type 'float'] but should be of type" - " 'list of basestring' if defined.") + " [class 'str', class 'int', class 'float'] but should be of type" + " 'list of str' if defined.") @staticmethod def test_string_array_wrong_elem_types_setter(module): @@ -1085,9 +1132,9 @@ def test_string_array_wrong_elem_types_setter(module): message = err_info.value.message assert message == ( - "TestDefinition's parameter 'string_array_property' was a list of" - " [type 'str', type 'int', type 'float'] but should be of type" - " 'list of basestring' if defined.") + "TestDefinition's parameter 'string_array_property' was a list of" + " [class 'str', class 'int', class 'float'] but should be of type" + " 'list of str' if defined.") class TestTemplateBooleanProperty: @@ -1171,8 +1218,8 @@ def test_boolean_is_string(module): message = err_info.value.message assert message == ( - "TestDefinition's parameter 'required_boolean_property' was type" - " 'str' but should be of type 'bool'.") + "TestDefinition's parameter 'required_boolean_property' was class" + " 'str' but should be of class 'bool'.") @staticmethod def test_boolean_is_string_setter(module): @@ -1183,8 +1230,8 @@ def test_boolean_is_string_setter(module): message = err_info.value.message assert message == ( - "TestDefinition's parameter 'required_boolean_property' was type" - " 'str' but should be of type 'bool'.") + "TestDefinition's parameter 'required_boolean_property' was class" + " 'str' but should be of class 'bool'.") class TestTemplateEnumProperty: @@ -1334,8 +1381,8 @@ def test_required_string_not_string(module): message = err_info.value.message assert message == ( - "TestDefinition's parameter 'required_string_property' was type" - " 'int' but should be of type 'basestring'.") + "TestDefinition's parameter 'required_string_property' was class" + " 'int' but should be of class 'str'.") @staticmethod def test_required_string_not_string_setter(module): @@ -1348,8 +1395,8 @@ def test_required_string_not_string_setter(module): message = err_info.value.message assert message == ( - "TestDefinition's parameter 'required_string_property' was type" - " 'int' but should be of type 'basestring'.") + "TestDefinition's parameter 'required_string_property' was class" + " 'int' but should be of class 'str'.") @staticmethod def test_string_incorrect_enum(module): @@ -1387,8 +1434,8 @@ def test_string_not_string(module): message = err_info.value.message assert message == ( - "TestDefinition's parameter 'string_property' was type 'int' but" - " should be of type 'basestring' if defined.") + "TestDefinition's parameter 'string_property' was class 'int' but" + " should be of class 'str' if defined.") @staticmethod def test_string_not_string_setter(module): @@ -1401,8 +1448,8 @@ def test_string_not_string_setter(module): message = err_info.value.message assert message == ( - "TestDefinition's parameter 'string_property' was type 'int' but" - " should be of type 'basestring' if defined.") + "TestDefinition's parameter 'string_property' was class 'int' but" + " should be of class 'str' if defined.") @staticmethod def test_required_object_incorrect_enum(module): @@ -1439,8 +1486,8 @@ def test_required_object_not_object(module): message = err_info.value.message assert message == ( - "TestDefinition's parameter 'required_object_property' was type" - " 'str' but should be of a dict with keys type 'basestring'.") + "TestDefinition's parameter 'required_object_property' was class" + " 'str' but should be of a dict with keys type 'str'.") @staticmethod def test_required_object_not_object_setter(module): @@ -1453,8 +1500,8 @@ def test_required_object_not_object_setter(module): message = err_info.value.message assert message == ( - "TestDefinition's parameter 'required_object_property' was type" - " 'str' but should be of a dict with keys type 'basestring'.") + "TestDefinition's parameter 'required_object_property' was class" + " 'str' but should be of a dict with keys type 'str'.") @staticmethod def test_object_incorrect_enum(module): @@ -1493,8 +1540,8 @@ def test_object_not_object(module): message = err_info.value.message assert message == ( - "TestDefinition's parameter 'object_property' was type 'str' but" - " should be of a dict with keys type 'basestring' if defined.") + "TestDefinition's parameter 'object_property' was class 'str' but" + " should be of a dict with keys type 'str' if defined.") @staticmethod def test_object_not_object_setter(module): @@ -1507,8 +1554,8 @@ def test_object_not_object_setter(module): message = err_info.value.message assert message == ( - "TestDefinition's parameter 'object_property' was type 'str' but" - " should be of a dict with keys type 'basestring' if defined.") + "TestDefinition's parameter 'object_property' was class 'str' but" + " should be of a dict with keys type 'str' if defined.") @staticmethod def test_required_array_incorrect_enum(module): @@ -1518,9 +1565,12 @@ def test_required_array_incorrect_enum(module): required_array_property=['FA', 'SO']) message = err_info.value.message - assert message == ( - "Invalid values for 'required_array_property'. Was [FA, SO]" - " but must be a subset of [DO, RE, MI].") + # In Py3, the "FA" and "SO" are not always listed in the same order. Use a + # regex to check test success regardless of this ordering. + pattern = re.compile( + "Invalid values for 'required_array_property'. Was \\[(FA, SO|SO, FA)\\] " + "but must be a subset of \\[DO, RE, MI\\].") + assert re.match(pattern, message) is not None @staticmethod def test_required_array_incorrect_enum_setter(module): @@ -1532,9 +1582,10 @@ def test_required_array_incorrect_enum_setter(module): test_object.required_array_property = ['FA', 'SO'] message = err_info.value.message - assert message == ( - "Invalid values for 'required_array_property'. Was [FA, SO]" - " but must be a subset of [DO, RE, MI].") + pattern = re.compile( + "Invalid values for 'required_array_property'. Was \\[(FA, SO|SO, FA)\\] " + "but must be a subset of \\[DO, RE, MI\\].") + assert re.match(pattern, message) is not None @staticmethod def test_required_array_not_array(module): @@ -1545,8 +1596,8 @@ def test_required_array_not_array(module): message = err_info.value.message assert message == ( - "TestDefinition's parameter 'required_array_property' was type" - " 'str' but should be of type 'list of basestring'.") + "TestDefinition's parameter 'required_array_property' was class" + " 'str' but should be of type 'list of str'.") @staticmethod def test_required_array_not_array_setter(module): @@ -1559,8 +1610,8 @@ def test_required_array_not_array_setter(module): message = err_info.value.message assert message == ( - "TestDefinition's parameter 'required_array_property' was type" - " 'str' but should be of type 'list of basestring'.") + "TestDefinition's parameter 'required_array_property' was class" + " 'str' but should be of type 'list of str'.") @staticmethod def test_array_incorrect_enum(module): @@ -1571,8 +1622,10 @@ def test_array_incorrect_enum(module): array_property=['FA', 'SO']) message = err_info.value.message - assert message == ("Invalid values for 'array_property'. Was [FA, SO]" - " but must be a subset of [DO, RE, MI] if defined.") + pattern = re.compile( + "Invalid values for 'array_property'. Was \\[(FA, SO|SO, FA)\\] but must" + " be a subset of \\[DO, RE, MI\\] if defined.") + assert re.match(pattern, message) is not None @staticmethod def test_array_incorrect_enum_setter(module): @@ -1584,8 +1637,11 @@ def test_array_incorrect_enum_setter(module): test_object.array_property = ['FA', 'SO'] message = err_info.value.message - assert message == ("Invalid values for 'array_property'. Was [FA, SO]" - " but must be a subset of [DO, RE, MI] if defined.") + pattern = re.compile( + "Invalid values for 'array_property'. Was \\[(FA, SO|SO, FA)\\] but must" + " be a subset of \\[DO, RE, MI\\] if defined.") + + assert re.match(pattern, message) is not None @staticmethod def test_array_not_array(module): @@ -1597,8 +1653,8 @@ def test_array_not_array(module): message = err_info.value.message assert message == ( - "TestDefinition's parameter 'array_property' was type 'str' but" - " should be of type 'list of basestring' if defined.") + "TestDefinition's parameter 'array_property' was class 'str' but" + " should be of type 'list of str' if defined.") @staticmethod def test_array_not_array_setter(module): @@ -1611,5 +1667,5 @@ def test_array_not_array_setter(module): message = err_info.value.message assert message == ( - "TestDefinition's parameter 'array_property' was type 'str' but" - " should be of type 'list of basestring' if defined.") + "TestDefinition's parameter 'array_property' was class 'str' but" + " should be of type 'list of str' if defined.") diff --git a/tools/src/test/python/dlpx/virtualization/_internal/conftest.py b/tools/src/test/python/dlpx/virtualization/_internal/conftest.py index 39a4460f..43c43602 100644 --- a/tools/src/test/python/dlpx/virtualization/_internal/conftest.py +++ b/tools/src/test/python/dlpx/virtualization/_internal/conftest.py @@ -1,5 +1,5 @@ # -# Copyright (c) 2019, 2020 by Delphix. All rights reserved. +# Copyright (c) 2019, 2021 by Delphix. All rights reserved. # import configparser @@ -7,9 +7,11 @@ import json import os +from importlib import reload import yaml from dlpx.virtualization._internal import cli, click_util, const, package_util from dlpx.virtualization._internal.commands import build +from dlpx.virtualization.common.util import to_bytes, to_str import pytest @@ -39,7 +41,7 @@ def plugin_config_file(tmpdir, plugin_config_filename, plugin_config_content): f = tmpdir.join(plugin_config_filename) if plugin_config_content: - f.write(plugin_config_content) + f.write(to_str(plugin_config_content)) return f.strpath @@ -113,8 +115,8 @@ def _write_dvp_config_file(tmpdir, if dev_config_properties: parser['dev'] = dev_config_properties - with open(dvp_config_filepath, 'wb') as config_file: - parser.write(config_file) + with open(dvp_config_filepath, 'w') as config_file: + parser.write(to_bytes(config_file)) # # Add temp_dir to list of config files the ConfigFileProcessor will @@ -156,7 +158,7 @@ def artifact_file(tmpdir, artifact_content, artifact_filename, # Only write the artifact if we want to actually create it. if isinstance(artifact_content, dict): artifact_content = json.dumps(artifact_content, indent=4) - f.write(artifact_content) + f.write(to_bytes(artifact_content)) return f.strpath @@ -249,7 +251,7 @@ def external_version(): @pytest.fixture def language(): - return 'PYTHON27' + return 'PYTHON38' @pytest.fixture @@ -404,6 +406,7 @@ def mount_specification(virtual_source, repository): virtual.mount_specification_impl = mount_specification virtual.status_impl = None virtual.initialize_impl = None + virtual.cleanup_impl = None return virtual @@ -447,7 +450,8 @@ def plugin_manifest(upgrade_operation): 'hasVirtualMountSpecification': True, 'hasVirtualStatus': False, 'hasInitialize': False, - 'migrationIdList': upgrade_operation.migration_id_list + 'migrationIdList': upgrade_operation.migration_id_list, + 'hasVirtualCleanup': False, } return manifest @@ -642,7 +646,7 @@ def artifact_content(engine_api, virtual_source_definition, 'name': 'python_vfiles', 'externalVersion': '2.0.0', 'defaultLocale': 'en-us', - 'language': 'PYTHON27', + 'language': 'PYTHON38', 'hostTypes': ['UNIX'], 'entryPoint': 'python_vfiles:vfiles', 'buildApi': package_util.get_build_api_version(), @@ -691,7 +695,7 @@ def artifact_content(engine_api, virtual_source_definition, @pytest.fixture def engine_api(): - return {'type': 'APIVersion', 'major': 1, 'minor': 11, 'micro': 6} + return {'type': 'APIVersion', 'major': 1, 'minor': 11, 'micro': 11} @pytest.fixture diff --git a/tools/src/test/python/dlpx/virtualization/_internal/fake_plugin/direct/multiple_warnings.py b/tools/src/test/python/dlpx/virtualization/_internal/fake_plugin/direct/multiple_warnings.py index 56f4d299..1ee0d307 100644 --- a/tools/src/test/python/dlpx/virtualization/_internal/fake_plugin/direct/multiple_warnings.py +++ b/tools/src/test/python/dlpx/virtualization/_internal/fake_plugin/direct/multiple_warnings.py @@ -75,6 +75,11 @@ def unconfigure(repository, source_config, virtual_source): pass +@vfiles.virtual.cleanup() +def cleanup(repository, source_config, virtual_source, bad_arg): + pass + + @vfiles.upgrade.repository('2019.10.30') def repo_upgrade(old_repository): return old_repository diff --git a/tools/src/test/python/dlpx/virtualization/_internal/fake_plugin/direct/successful.py b/tools/src/test/python/dlpx/virtualization/_internal/fake_plugin/direct/successful.py index cb355933..da568732 100644 --- a/tools/src/test/python/dlpx/virtualization/_internal/fake_plugin/direct/successful.py +++ b/tools/src/test/python/dlpx/virtualization/_internal/fake_plugin/direct/successful.py @@ -81,6 +81,11 @@ def unconfigure(repository, source_config, virtual_source): pass +@direct.virtual.cleanup() +def cleanup(repository, source_config, virtual_source): + pass + + @direct.upgrade.repository('1.3', MigrationType.LUA) def repo_upgrade(old_repository): return old_repository diff --git a/tools/src/test/python/dlpx/virtualization/_internal/fake_plugin/direct/upgrade_warnings.py b/tools/src/test/python/dlpx/virtualization/_internal/fake_plugin/direct/upgrade_warnings.py index 0046fe49..0f018bf2 100644 --- a/tools/src/test/python/dlpx/virtualization/_internal/fake_plugin/direct/upgrade_warnings.py +++ b/tools/src/test/python/dlpx/virtualization/_internal/fake_plugin/direct/upgrade_warnings.py @@ -74,6 +74,11 @@ def unconfigure(repository, source_config, virtual_source): pass +@direct.virtual.cleanup() +def cleanup(repository, source_config, virtual_source): + pass + + @direct.upgrade.repository('2019.11.20') def repo_upgrade(old_repository): return old_repository diff --git a/tools/src/test/python/dlpx/virtualization/_internal/test_cli.py b/tools/src/test/python/dlpx/virtualization/_internal/test_cli.py index 4e8e215a..49c673bf 100644 --- a/tools/src/test/python/dlpx/virtualization/_internal/test_cli.py +++ b/tools/src/test/python/dlpx/virtualization/_internal/test_cli.py @@ -1,8 +1,9 @@ # -# Copyright (c) 2019, 2020 by Delphix. All rights reserved. +# Copyright (c) 2019, 2021 by Delphix. All rights reserved. # import os +import re import click.testing as click_testing import yaml @@ -172,7 +173,7 @@ def test_blank_ingestion_strategy(plugin_name): ['init', '-n', plugin_name, '-s', '']) assert result.exit_code != 0 - assert "invalid choice" in result.output + assert "Invalid value" in result.output @staticmethod def test_non_existent_root_dir(plugin_name): @@ -214,7 +215,7 @@ def test_multiple_host_types(): ]) assert result.exit_code != 0 - assert "invalid choice" in result.output + assert "Invalid value" in result.output @staticmethod @mock.patch('dlpx.virtualization._internal.commands.initialize.init') @@ -235,7 +236,7 @@ def test_invalid_host_type(): result = runner.invoke(cli.delphix_sdk, ['init', '-t', 'UNI']) assert result.exit_code != 0 - assert "invalid choice" in result.output + assert "Invalid value" in result.output class TestBuildCli: @@ -576,12 +577,10 @@ def test_with_config_file_fail(artifact_file): os.chdir(cwd) assert result.exit_code == 2 - assert result.output == (u'Usage: delphix-sdk upload [OPTIONS]\n' - u'\n' - u'Error: Invalid value for \'-e\' / ' - u'\'--engine\': Option is required ' - u'and must be specified via the command line.' - u'\n') + output = result.output.replace("\n", "") + pattern = re.compile( + r"Usage: delphix-sdk upload \[OPTIONS\].*Error: Invalid value for '-e.*") + assert re.match(pattern, output) is not None class TestDownloadCli: @@ -640,13 +639,11 @@ def test_missing_params(): ]) assert result.exit_code == 2 - assert result.output == ( - u"Usage: delphix-sdk download-logs [OPTIONS]\n" - u"\n" - u"Error: Invalid value for '-e' / " - u"'--engine': Option is required " - u"and must be specified via the command line." - u"\n") + output = result.output.replace("\n", "") + pattern = re.compile( + r"Usage: delphix-sdk download-logs \[OPTIONS\].*" + r"Error: Invalid value for '-e.*") + assert re.match(pattern, output) is not None @staticmethod @mock.patch( @@ -789,10 +786,9 @@ def test_with_config_file_fail(plugin_config_file, dvp_config_file): os.chdir(cwd) assert result.exit_code == 2 - assert result.output == ( - u"Usage: delphix-sdk download-logs [OPTIONS]\n" - u"\n" - u"Error: Invalid value for '-e' / " - u"'--engine': Option is required " - u"and must be specified via the command line." - u"\n") + + output = result.output.replace("\n", "") + pattern = re.compile( + r"Usage: delphix-sdk download-logs \[OPTIONS\].*" + r"Error: Invalid value for '-e.*") + assert re.match(pattern, output) is not None diff --git a/tools/src/test/python/dlpx/virtualization/_internal/test_package_util.py b/tools/src/test/python/dlpx/virtualization/_internal/test_package_util.py index 87f63c08..0477487f 100644 --- a/tools/src/test/python/dlpx/virtualization/_internal/test_package_util.py +++ b/tools/src/test/python/dlpx/virtualization/_internal/test_package_util.py @@ -10,23 +10,23 @@ class TestPackageUtil: @staticmethod def test_get_version(): - assert package_util.get_version() == '3.1.0' + assert package_util.get_version() == '4.0.0' @staticmethod def test_get_virtualization_api_version(): - assert package_util.get_virtualization_api_version() == '1.5.0' + assert package_util.get_virtualization_api_version() == '1.6.1' @staticmethod def test_get_engine_api_version(): - assert package_util.get_engine_api_version_from_settings() == '1.11.6' + assert package_util.get_engine_api_version_from_settings() == '1.11.11' @staticmethod def test_get_build_api_version_json(): build_api_version = { 'type': 'APIVersion', 'major': 1, - 'minor': 5, - 'micro': 0 + 'minor': 6, + 'micro': 1 } assert package_util.get_build_api_version() == build_api_version @@ -36,7 +36,7 @@ def test_get_engine_api_version_json(): 'type': 'APIVersion', 'major': 1, 'minor': 11, - 'micro': 6 + 'micro': 11 } assert package_util.get_engine_api_version() == engine_api_version diff --git a/tools/src/test/python/dlpx/virtualization/_internal/test_plugin_importer.py b/tools/src/test/python/dlpx/virtualization/_internal/test_plugin_importer.py index a8795765..1d20ee9c 100644 --- a/tools/src/test/python/dlpx/virtualization/_internal/test_plugin_importer.py +++ b/tools/src/test/python/dlpx/virtualization/_internal/test_plugin_importer.py @@ -1,14 +1,14 @@ # -# Copyright (c) 2019, 2020 by Delphix. All rights reserved. +# Copyright (c) 2019, 2021 by Delphix. All rights reserved. # -import exceptions import os +import re import uuid from collections import OrderedDict from multiprocessing import Queue -from dlpx.virtualization._internal import (file_util, plugin_util, - plugin_validator, plugin_importer) +from dlpx.virtualization._internal import ( + file_util, plugin_util, plugin_validator, plugin_importer, exceptions) from dlpx.virtualization._internal.plugin_importer import PluginImporter import mock @@ -137,14 +137,15 @@ def test_successful_validation(mock_file_util, plugin_config_file, ('multiple_warnings:vfiles', 'DIRECT', [ 'Error: Number of arguments do not match in method status', 'Error: Named argument mismatch in method status', + 'Error: Number of arguments do not match in method cleanup', + 'Error: Named argument mismatch in method cleanup', 'Warning: Implementation missing for required method' - ' virtual.reconfigure().', '1 Warning(s). 2 Error(s).' + ' virtual.reconfigure().', '1 Warning(s). 4 Error(s).' ])]) @mock.patch('dlpx.virtualization._internal.file_util.get_src_dir_path') def test_multiple_warnings(mock_file_util, plugin_config_file, fake_src_dir, expected_errors): mock_file_util.return_value = fake_src_dir - with pytest.raises(exceptions.UserError) as err_info: importer = get_plugin_importer(plugin_config_file) importer.validate_plugin_module() @@ -231,12 +232,12 @@ def test_plugin_info_warn_mode(mock_import, mock_relative_path, plugin_config_file, src_dir, plugin_module_content): plugin_config_content = OrderedDict([ - ('id', str(uuid.uuid4())), ('name', 'staged'.encode('utf-8')), - ('version', '0.1.0'), ('language', 'PYTHON27'), - ('hostTypes', ['UNIX']), ('pluginType', 'STAGED'.encode('utf-8')), + ('id', str(uuid.uuid4())), ('name', 'staged'), + ('version', '0.1.0'), ('language', 'PYTHON38'), + ('hostTypes', ['UNIX']), ('pluginType', 'STAGED'), ('manualDiscovery', True), - ('entryPoint', 'staged_plugin:staged'.encode('utf-8')), - ('srcDir', src_dir), ('schemaFile', 'schema.json'.encode('utf-8')) + ('entryPoint', 'staged_plugin:staged'), + ('srcDir', src_dir), ('schemaFile', 'schema.json') ]) mock_import.return_value = plugin_module_content try: @@ -280,7 +281,8 @@ def test_import_error(mock_file_util, plugin_config_file, importer.validate_plugin_module() message = err_info.value.message - assert expected_error in message + pattern = re.compile("^Error: No module named (')?dlpxxx(')?.*") + assert re.match(pattern, message) is not None @staticmethod @pytest.mark.parametrize( @@ -307,7 +309,6 @@ def test_bad_syntax(mock_file_util, plugin_config_file, def test_undefined_name_error(mock_file_util, plugin_config_file, fake_src_dir, expected_error): mock_file_util.return_value = fake_src_dir - with pytest.raises(exceptions.SDKToolingError) as err_info: importer = get_plugin_importer(plugin_config_file) importer.validate_plugin_module() diff --git a/tools/src/test/python/dlpx/virtualization/_internal/test_plugin_validator.py b/tools/src/test/python/dlpx/virtualization/_internal/test_plugin_validator.py index e81ce7da..4b419768 100644 --- a/tools/src/test/python/dlpx/virtualization/_internal/test_plugin_validator.py +++ b/tools/src/test/python/dlpx/virtualization/_internal/test_plugin_validator.py @@ -1,5 +1,5 @@ # -# Copyright (c) 2019, 2020 by Delphix. All rights reserved. +# Copyright (c) 2019, 2021 by Delphix. All rights reserved. # import json @@ -24,8 +24,8 @@ def test_plugin_bad_schema(plugin_config_file, plugin_config_content, message = err_info.value.message assert ('Failed to load schemas because {} is not a valid json file.' - ' Error: Extra data: line 2 column 1 - line 2 column 9' - ' (char 19 - 27)'.format(schema_file)) in message + ' Error: Extra data: line 2 column 1 (char 19)' + .format(schema_file)) in message @staticmethod @pytest.mark.parametrize('plugin_config_file', ['/dir/plugin_config.yml']) diff --git a/tools/src/test/python/dlpx/virtualization/_internal/test_schema_validator.py b/tools/src/test/python/dlpx/virtualization/_internal/test_schema_validator.py index 2901c0cf..5f735a79 100644 --- a/tools/src/test/python/dlpx/virtualization/_internal/test_schema_validator.py +++ b/tools/src/test/python/dlpx/virtualization/_internal/test_schema_validator.py @@ -1,5 +1,5 @@ # -# Copyright (c) 2019, 2020 by Delphix. All rights reserved. +# Copyright (c) 2019, 2021 by Delphix. All rights reserved. # import json @@ -22,8 +22,8 @@ def test_bad_meta_schema(schema_file, tmpdir, schema_filename): message = err_info.value.message assert ("Failed to load schemas because '{}' is not a valid json file." - " Error: Extra data: line 2 column 1 - line 2 column 9" - " (char 19 - 27)".format(schema_file)) in message + " Error: Extra data: line 2 column 1 (char 19)" + .format(schema_file)) in message @staticmethod def test_bad_schema_file(schema_file): From 9342f1ae87d6df641d9321763a0d0144c7a65abe Mon Sep 17 00:00:00 2001 From: "zachary.burke" Date: Wed, 12 Jan 2022 18:30:50 -0800 Subject: [PATCH 02/21] manual merge of release into master --- .bumpversion.cfg | 2 +- common/setup.py | 2 +- common/src/main/python/dlpx/virtualization/common/VERSION | 2 +- docs/docs/Best_Practices/Message_Limits.md | 2 +- dvp/src/main/python/dlpx/virtualization/VERSION | 2 +- libs/setup.py | 2 +- libs/src/main/python/dlpx/virtualization/libs/VERSION | 2 +- platform/setup.py | 2 +- .../src/main/python/dlpx/virtualization/platform/VERSION | 2 +- .../src/main/python/dlpx/virtualization/_internal/VERSION | 2 +- .../dlpx/virtualization/_internal/test_package_util.py | 8 ++++---- 11 files changed, 14 insertions(+), 14 deletions(-) diff --git a/.bumpversion.cfg b/.bumpversion.cfg index afb68a8c..2d18460a 100644 --- a/.bumpversion.cfg +++ b/.bumpversion.cfg @@ -1,5 +1,5 @@ [bumpversion] -current_version = 4.0.0 +current_version = 4.0.2 commit = False tag = False parse = (?P\d+)\.(?P\d+)\.(?P\d+)(\.(?P[a-z]+)(?P\d+))? diff --git a/common/setup.py b/common/setup.py index 68fc5cdd..d3593d72 100644 --- a/common/setup.py +++ b/common/setup.py @@ -4,7 +4,7 @@ PYTHON_SRC = 'src/main/python' install_requires = [ - "dvp-api == 1.6.1", + "dvp-api == 1.6.3", ] with open(os.path.join(PYTHON_SRC, 'dlpx/virtualization/common/VERSION')) as version_file: diff --git a/common/src/main/python/dlpx/virtualization/common/VERSION b/common/src/main/python/dlpx/virtualization/common/VERSION index fcdb2e10..4d54dadd 100644 --- a/common/src/main/python/dlpx/virtualization/common/VERSION +++ b/common/src/main/python/dlpx/virtualization/common/VERSION @@ -1 +1 @@ -4.0.0 +4.0.2 diff --git a/docs/docs/Best_Practices/Message_Limits.md b/docs/docs/Best_Practices/Message_Limits.md index a17d352e..0389b1c5 100644 --- a/docs/docs/Best_Practices/Message_Limits.md +++ b/docs/docs/Best_Practices/Message_Limits.md @@ -41,4 +41,4 @@ Discovery of "my_plugin" failed: Plugin operation "Repository Discovery" got a R ## What to do if the maximum metadata or message size is exceeded -Reach out to us via the [Virtualization SDK GitHub repository](https://github.com/delphix/virtualization-sdk/) for guidance. \ No newline at end of file +Reach out to us via the [Virtualization SDK GitHub repository](https://github.com/delphix/virtualization-sdk/) for guidance. diff --git a/dvp/src/main/python/dlpx/virtualization/VERSION b/dvp/src/main/python/dlpx/virtualization/VERSION index fcdb2e10..4d54dadd 100644 --- a/dvp/src/main/python/dlpx/virtualization/VERSION +++ b/dvp/src/main/python/dlpx/virtualization/VERSION @@ -1 +1 @@ -4.0.0 +4.0.2 diff --git a/libs/setup.py b/libs/setup.py index 7ab8eff5..f443840a 100644 --- a/libs/setup.py +++ b/libs/setup.py @@ -7,7 +7,7 @@ version = version_file.read().strip() install_requires = [ - "dvp-api == 1.6.1", + "dvp-api == 1.6.3", "dvp-common == {}".format(version) ] diff --git a/libs/src/main/python/dlpx/virtualization/libs/VERSION b/libs/src/main/python/dlpx/virtualization/libs/VERSION index fcdb2e10..4d54dadd 100644 --- a/libs/src/main/python/dlpx/virtualization/libs/VERSION +++ b/libs/src/main/python/dlpx/virtualization/libs/VERSION @@ -1 +1 @@ -4.0.0 +4.0.2 diff --git a/platform/setup.py b/platform/setup.py index 5a696bb3..6cd76b71 100644 --- a/platform/setup.py +++ b/platform/setup.py @@ -7,7 +7,7 @@ version = version_file.read().strip() install_requires = [ - "dvp-api == 1.6.1", + "dvp-api == 1.6.3", "dvp-common == {}".format(version), "enum34;python_version < '3.4'", ] diff --git a/platform/src/main/python/dlpx/virtualization/platform/VERSION b/platform/src/main/python/dlpx/virtualization/platform/VERSION index fcdb2e10..4d54dadd 100644 --- a/platform/src/main/python/dlpx/virtualization/platform/VERSION +++ b/platform/src/main/python/dlpx/virtualization/platform/VERSION @@ -1 +1 @@ -4.0.0 +4.0.2 diff --git a/tools/src/main/python/dlpx/virtualization/_internal/VERSION b/tools/src/main/python/dlpx/virtualization/_internal/VERSION index fcdb2e10..4d54dadd 100644 --- a/tools/src/main/python/dlpx/virtualization/_internal/VERSION +++ b/tools/src/main/python/dlpx/virtualization/_internal/VERSION @@ -1 +1 @@ -4.0.0 +4.0.2 diff --git a/tools/src/test/python/dlpx/virtualization/_internal/test_package_util.py b/tools/src/test/python/dlpx/virtualization/_internal/test_package_util.py index 0477487f..81485f12 100644 --- a/tools/src/test/python/dlpx/virtualization/_internal/test_package_util.py +++ b/tools/src/test/python/dlpx/virtualization/_internal/test_package_util.py @@ -1,5 +1,5 @@ # -# Copyright (c) 2019, 2020 by Delphix. All rights reserved. +# Copyright (c) 2019, 2022 by Delphix. All rights reserved. # import os from dlpx.virtualization._internal import package_util @@ -10,11 +10,11 @@ class TestPackageUtil: @staticmethod def test_get_version(): - assert package_util.get_version() == '4.0.0' + assert package_util.get_version() == '4.0.2' @staticmethod def test_get_virtualization_api_version(): - assert package_util.get_virtualization_api_version() == '1.6.1' + assert package_util.get_virtualization_api_version() == '1.6.3' @staticmethod def test_get_engine_api_version(): @@ -26,7 +26,7 @@ def test_get_build_api_version_json(): 'type': 'APIVersion', 'major': 1, 'minor': 6, - 'micro': 1 + 'micro': 3 } assert package_util.get_build_api_version() == build_api_version From 647728a698298ba679e52e4016e5107f07ad30ca Mon Sep 17 00:00:00 2001 From: Sourabh Jain <45626610+SumoSourabh@users.noreply.github.com> Date: Mon, 6 Jun 2022 11:18:12 +0530 Subject: [PATCH 03/21] Updating code owners for the github repository (#439) Updating code owners Co-authored-by: Sourabh Jain --- .github/CODEOWNERS | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index b9a5cc1a..2da0a40c 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -1 +1 @@ -* @ankursarin @nhlien93 @mothslaw +* @nhlien93 @SumoSourabh @vimleshmishra From 5cf8cb97a2894e6ff13c6d2269708e1ca09e4b4a Mon Sep 17 00:00:00 2001 From: Terraform User Date: Tue, 28 Jun 2022 04:53:48 +0000 Subject: [PATCH 04/21] Create CodeOwners File [Terraform] --- .github/CODEOWNERS | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index 2da0a40c..30c28255 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -1 +1,5 @@ -* @nhlien93 @SumoSourabh @vimleshmishra +# +# Copyright (c) 2022 by Delphix. All rights reserved. +# + +* @delphix/gatekeepers-appdata From b3696ba1934dd08ced9557960410f689141d503d Mon Sep 17 00:00:00 2001 From: vimleshmishra Date: Tue, 12 Jul 2022 13:11:49 +0530 Subject: [PATCH 05/21] CP-8174 VSDK Changes to support source sizing decorators (#441) * __hash__ is needed in python 3 * DLPX-79760 SnapshotDefinitionDataPaths object is not hashable since py2-3 transition * DOC-2234 Update new logo in developer.delphix.com * Resolved COnflicts * revert logo chnages * revert logo chnages * Resolve Conflicts * revert changes * revert logo.png * Resolve Conflicts * DOC-2234 Update new logo in developer.delphix.com * logo chnages * DOC-2234 Update new logo in developer.delphix.com * new favicon * incorporated review comments * DLPX-81195 develop version of vSDK needs to be bumped up to 4.0.4 for testing to use latest vSDK code * DLPX-81195 develop version of vSDK needs to be bumped up to 4.0.4 for testing to use latest vSDK code * Changes suggested by Aleks * CP-8174 VSDK Changes to support source sizing decorators * reverted dvp version related changes * reverted extra comma * update engine API version * Fixed Formatting Issues * Fixed Formatting Issues * Fixed Unit Tests * Fixed Unit Tests --- .../dlpx/virtualization/platform/_linked.py | 125 +++++++++++++++++- .../dlpx/virtualization/platform/_virtual.py | 65 +++++++++ .../dlpx/virtualization/platform/operation.py | 2 + .../_internal/plugin_importer.py | 4 + .../virtualization/_internal/settings.cfg | 2 +- .../validation_schemas/plugin_importer.yaml | 17 +++ .../dlpx/virtualization/_internal/conftest.py | 6 +- .../_internal/test_package_util.py | 6 +- 8 files changed, 220 insertions(+), 7 deletions(-) diff --git a/platform/src/main/python/dlpx/virtualization/platform/_linked.py b/platform/src/main/python/dlpx/virtualization/platform/_linked.py index 4e6c07ee..adecb6fc 100644 --- a/platform/src/main/python/dlpx/virtualization/platform/_linked.py +++ b/platform/src/main/python/dlpx/virtualization/platform/_linked.py @@ -32,6 +32,7 @@ def __init__(self): self.status_impl = None self.worker_impl = None self.mount_specification_impl = None + self.source_size_impl = None def pre_snapshot(self): def pre_snapshot_decorator(pre_snapshot_impl): @@ -101,6 +102,16 @@ def mount_specification_decorator(mount_specification_impl): return mount_specification_decorator + def source_size(self): + def source_size_decorator(source_size_impl): + if self.source_size_impl: + raise OperationAlreadyDefinedError(Op.LINKED_SOURCE_SIZE) + self.source_size_impl = v.check_function(source_size_impl, + Op.LINKED_SOURCE_SIZE) + return source_size_impl + + return source_size_decorator + def _internal_direct_pre_snapshot(self, request): """Pre Snapshot Wrapper for direct plugins. @@ -237,6 +248,57 @@ def to_protobuf(snapshot): return direct_post_snapshot_response + def _internal_direct_source_size(self, request): + """Direct Source Size Wrapper for direct plugins. + + Executed as part of several operations to get the source size + of a direct source + + Run source_size operation for a direct source. + + Args: + request (DirectSourceSizeRequest): Source Size Request arguments. + + Returns: + DirectSourceSizeResponse: A response containing the return value - + DirectSourceSizeResult which has source size. In + case of errors, response object will contain PluginErrorResult. + """ + # Reasoning for method imports are in this file's docstring. + from generated.definitions import RepositoryDefinition + from generated.definitions import LinkedSourceDefinition + from generated.definitions import SourceConfigDefinition + + # + # While linked.source_size() is not a required operation, this should + # not be called if it wasn't implemented. + # + if not self.source_size_impl: + raise OperationNotDefinedError(Op.LINKED_SOURCE_SIZE) + + direct_source_definition = LinkedSourceDefinition.from_dict( + json.loads(request.direct_source.linked_source.parameters.json)) + direct_source = DirectSource( + guid=request.direct_source.linked_source.guid, + connection=RemoteConnection.from_proto( + request.direct_source.connection), + parameters=direct_source_definition) + + repository = RepositoryDefinition.from_dict( + json.loads(request.repository.parameters.json)) + source_config = SourceConfigDefinition.from_dict( + json.loads(request.source_config.parameters.json)) + + size = self.source_size_impl( + direct_source=direct_source, + repository=repository, + source_config=source_config) + + direct_source_size_response = platform_pb2.DirectSourceSizeResponse() + direct_source_size_response.return_value.database_size = size + + return direct_source_size_response + def _internal_staged_pre_snapshot(self, request): """Pre Snapshot Wrapper for staged plugins. @@ -272,8 +334,8 @@ def _internal_staged_pre_snapshot(self, request): staged_mount = request.staged_source.staged_mount mount = Mount(remote_environment=RemoteEnvironment.from_proto( staged_mount.remote_environment), - mount_path=staged_mount.mount_path, - shared_path=staged_mount.shared_path) + mount_path=staged_mount.mount_path, + shared_path=staged_mount.shared_path) staged_source = StagedSource( guid=linked_source.guid, source_connection=RemoteConnection.from_proto( @@ -719,3 +781,62 @@ def to_protobuf_ownership_spec(ownership_spec): ownership_spec) return staged_mount_spec_response + + def _internal_staged_source_size(self, request): + """Staged Source Size Wrapper for staged plugins. + + Executed as part of several operations to get the source size + of a staged source + + Run source_size operation for a staged source. + + Args: + request (StagedSourceSizeRequest): Source Size Request arguments. + + Returns: + StagedSourceSizeResponse: A response containing the return value - + StagedSourceSizeResult which has source size. In + case of errors, response object will contain PluginErrorResult. + """ + # Reasoning for method imports are in this file's docstring. + from generated.definitions import RepositoryDefinition + from generated.definitions import LinkedSourceDefinition + from generated.definitions import SourceConfigDefinition + + # + # While linked.source_size() is not a required operation, this should + # not be called if it wasn't implemented. + # + if not self.source_size_impl: + raise OperationNotDefinedError(Op.LINKED_SOURCE_SIZE) + + staged_source_definition = LinkedSourceDefinition.from_dict( + json.loads(request.staged_source.linked_source.parameters.json)) + mount = Mount( + remote_environment=(RemoteEnvironment.from_proto( + request.staged_source.staged_mount.remote_environment)), + mount_path=request.staged_source.staged_mount.mount_path, + shared_path=request.staged_source.staged_mount.shared_path) + staged_source = StagedSource( + guid=request.staged_source.linked_source.guid, + source_connection=RemoteConnection.from_proto( + request.staged_source.source_connection), + parameters=staged_source_definition, + mount=mount, + staged_connection=RemoteConnection.from_proto( + request.staged_source.staged_connection)) + + repository = RepositoryDefinition.from_dict( + json.loads(request.repository.parameters.json)) + source_config = SourceConfigDefinition.from_dict( + json.loads(request.source_config.parameters.json)) + + size = self.source_size_impl( + staged_source=staged_source, + repository=repository, + source_config=source_config) + + staged_source_size_response = platform_pb2.StagedSourceSizeResponse() + staged_source_size_response.return_value.database_size = size + + return staged_source_size_response diff --git a/platform/src/main/python/dlpx/virtualization/platform/_virtual.py b/platform/src/main/python/dlpx/virtualization/platform/_virtual.py index 02d8826e..9f8ca084 100644 --- a/platform/src/main/python/dlpx/virtualization/platform/_virtual.py +++ b/platform/src/main/python/dlpx/virtualization/platform/_virtual.py @@ -34,6 +34,7 @@ def __init__(self): self.status_impl = None self.initialize_impl = None self.mount_specification_impl = None + self.source_size_impl = None def configure(self): def configure_decorator(configure_impl): @@ -142,6 +143,16 @@ def mount_specification_decorator(mount_specification_impl): return mount_specification_decorator + def source_size(self): + def source_size_decorator(source_size_impl): + if self.source_size_impl: + raise OperationAlreadyDefinedError(Op.VIRTUAL_SOURCE_SIZE) + self.source_size_impl = v.check_function(source_size_impl, + Op.VIRTUAL_SOURCE_SIZE) + return source_size_impl + + return source_size_decorator + @staticmethod def _from_protobuf_single_subset_mount(single_subset_mount): return Mount(remote_environment=RemoteEnvironment.from_proto( @@ -783,3 +794,57 @@ def to_protobuf_ownership_spec(ownership_spec): ] virtual_mount_spec_response.return_value.mounts.extend(mounts_list) return virtual_mount_spec_response + + def _internal_virtual_source_size(self, request): + """Virtual Source Size Wrapper. + + Executed as part of several operations to get the virtual source size + of a virtual source + + Run source_size operation for a virtual source. + + Args: + request (VirtualSourceSizeRequest): Source Size Request arguments. + + Returns: + VirtualSourceSizeResponse: A response containing the return value - + VirtualSourceSizeResult which has source size. In + case of errors, response object will contain PluginErrorResult. + """ + # Reasoning for method imports are in this file's docstring. + from generated.definitions import VirtualSourceDefinition + from generated.definitions import RepositoryDefinition + from generated.definitions import SourceConfigDefinition + + # + # While virtual.source_size() is not a required operation, this should + # not be called if it wasn't implemented. + # + if not self.source_size_impl: + raise OperationNotDefinedError(Op.VIRTUAL_SOURCE_SIZE) + + virtual_source_definition = VirtualSourceDefinition.from_dict( + json.loads(request.virtual_source.parameters.json)) + mounts = [ + VirtualOperations._from_protobuf_single_subset_mount(m) + for m in request.virtual_source.mounts + ] + virtual_source = VirtualSource(guid=request.virtual_source.guid, + connection=RemoteConnection.from_proto( + request.virtual_source.connection), + parameters=virtual_source_definition, + mounts=mounts) + repository = RepositoryDefinition.from_dict( + json.loads(request.repository.parameters.json)) + source_config = SourceConfigDefinition.from_dict( + json.loads(request.source_config.parameters.json)) + + size = self.source_size_impl( + virtual_source=virtual_source, + repository=repository, + source_config=source_config) + + virtual_source_size_response = platform_pb2.VirtualSourceSizeResponse() + virtual_source_size_response.return_value.database_size = size + + return virtual_source_size_response diff --git a/platform/src/main/python/dlpx/virtualization/platform/operation.py b/platform/src/main/python/dlpx/virtualization/platform/operation.py index 87a06c31..6d033d21 100644 --- a/platform/src/main/python/dlpx/virtualization/platform/operation.py +++ b/platform/src/main/python/dlpx/virtualization/platform/operation.py @@ -17,6 +17,7 @@ class Operation(Enum): LINKED_STATUS = 'linked.status()' LINKED_WORKER = 'linked.worker()' LINKED_MOUNT_SPEC = 'linked.mount_specification()' + LINKED_SOURCE_SIZE = 'linked.source_size()' VIRTUAL_CONFIGURE = 'virtual.configure()' VIRTUAL_UNCONFIGURE = 'virtual.unconfigure()' @@ -29,6 +30,7 @@ class Operation(Enum): VIRTUAL_STATUS = 'virtual.status()' VIRTUAL_INITIALIZE = 'virtual.initialize()' VIRTUAL_MOUNT_SPEC = 'virtual.mount_specification()' + VIRTUAL_SOURCE_SIZE = 'virtual.source_size()' UPGRADE_REPOSITORY = 'upgrade.repository()' UPGRADE_SOURCE_CONFIG = 'upgrade.source_config()' diff --git a/tools/src/main/python/dlpx/virtualization/_internal/plugin_importer.py b/tools/src/main/python/dlpx/virtualization/_internal/plugin_importer.py index 858ef5a9..f569c120 100644 --- a/tools/src/main/python/dlpx/virtualization/_internal/plugin_importer.py +++ b/tools/src/main/python/dlpx/virtualization/_internal/plugin_importer.py @@ -358,6 +358,8 @@ def _prepare_manifest(entry_point, module_content): bool(plugin_object.linked.worker_impl), 'hasLinkedMountSpecification': bool(plugin_object.linked.mount_specification_impl), + 'hasLinkedSourceSize': + bool(plugin_object.linked.source_size_impl), 'hasVirtualConfigure': bool(plugin_object.virtual.configure_impl), 'hasVirtualUnconfigure': @@ -380,6 +382,8 @@ def _prepare_manifest(entry_point, module_content): bool(plugin_object.virtual.status_impl), 'hasInitialize': bool(plugin_object.virtual.initialize_impl), + 'hasVirtualSourceSize': + bool(plugin_object.virtual.source_size_impl), 'migrationIdList': plugin_object.upgrade.migration_id_list } diff --git a/tools/src/main/python/dlpx/virtualization/_internal/settings.cfg b/tools/src/main/python/dlpx/virtualization/_internal/settings.cfg index d71b6a5d..93477cdc 100644 --- a/tools/src/main/python/dlpx/virtualization/_internal/settings.cfg +++ b/tools/src/main/python/dlpx/virtualization/_internal/settings.cfg @@ -20,7 +20,7 @@ # versions in those packages until they are shipped out of band. # [General] -engine_api_version = 1.11.11 +engine_api_version = 1.12.0 distribution_name = dvp-tools package_author = Delphix namespace_package = dlpx diff --git a/tools/src/main/python/dlpx/virtualization/_internal/validation_schemas/plugin_importer.yaml b/tools/src/main/python/dlpx/virtualization/_internal/validation_schemas/plugin_importer.yaml index b04e88fc..9193e581 100644 --- a/tools/src/main/python/dlpx/virtualization/_internal/validation_schemas/plugin_importer.yaml +++ b/tools/src/main/python/dlpx/virtualization/_internal/validation_schemas/plugin_importer.yaml @@ -38,6 +38,10 @@ EXPECTED_STAGED_ARGS_BY_OP: mount_specification_impl: - staged_source - repository + source_size_impl: + - staged_source + - repository + - source_config VirtualOperations: configure_impl: - virtual_source @@ -82,6 +86,11 @@ EXPECTED_STAGED_ARGS_BY_OP: mount_specification_impl: - virtual_source - repository + source_size_impl: + - virtual_source + - repository + - source_config + EXPECTED_DIRECT_ARGS_BY_OP: @@ -102,6 +111,10 @@ EXPECTED_DIRECT_ARGS_BY_OP: - repository - source_config - optional_snapshot_parameters + source_size_impl: + - direct_source + - repository + - source_config VirtualOperations: configure_impl: - virtual_source @@ -146,6 +159,10 @@ EXPECTED_DIRECT_ARGS_BY_OP: mount_specification_impl: - virtual_source - repository + source_size_impl: + - virtual_source + - repository + - source_config EXPECTED_UPGRADE_ARGS: _repository_id_to_impl: diff --git a/tools/src/test/python/dlpx/virtualization/_internal/conftest.py b/tools/src/test/python/dlpx/virtualization/_internal/conftest.py index de6c79eb..3b05daae 100644 --- a/tools/src/test/python/dlpx/virtualization/_internal/conftest.py +++ b/tools/src/test/python/dlpx/virtualization/_internal/conftest.py @@ -368,6 +368,7 @@ def post_snapshot(direct_source, repository, source_config): linked.status_impl = None linked.worker_impl = None linked.mount_specification_impl = None + linked.source_size_impl = None return linked @@ -408,6 +409,7 @@ def mount_specification(virtual_source, repository): virtual.status_impl = None virtual.initialize_impl = None virtual.cleanup_impl = None + virtual.source_size_impl = None return virtual @@ -441,6 +443,7 @@ def plugin_manifest(upgrade_operation): 'hasLinkedStatus': False, 'hasLinkedWorker': False, 'hasLinkedMountSpecification': False, + 'hasLinkedSourceSize': False, 'hasVirtualConfigure': True, 'hasVirtualUnconfigure': False, 'hasVirtualReconfigure': True, @@ -450,6 +453,7 @@ def plugin_manifest(upgrade_operation): 'hasVirtualPostSnapshot': True, 'hasVirtualMountSpecification': True, 'hasVirtualStatus': False, + 'hasVirtualSourceSize': False, 'hasInitialize': False, 'migrationIdList': upgrade_operation.migration_id_list, 'hasVirtualCleanup': False, @@ -706,7 +710,7 @@ def artifact_content(engine_api, virtual_source_definition, @pytest.fixture def engine_api(): - return {'type': 'APIVersion', 'major': 1, 'minor': 11, 'micro': 11} + return {'type': 'APIVersion', 'major': 1, 'minor': 12, 'micro': 0} @pytest.fixture diff --git a/tools/src/test/python/dlpx/virtualization/_internal/test_package_util.py b/tools/src/test/python/dlpx/virtualization/_internal/test_package_util.py index 49671c5e..11b8b726 100644 --- a/tools/src/test/python/dlpx/virtualization/_internal/test_package_util.py +++ b/tools/src/test/python/dlpx/virtualization/_internal/test_package_util.py @@ -18,7 +18,7 @@ def test_get_virtualization_api_version(): @staticmethod def test_get_engine_api_version(): - assert package_util.get_engine_api_version_from_settings() == '1.11.11' + assert package_util.get_engine_api_version_from_settings() == '1.12.0' @staticmethod def test_get_build_api_version_json(): @@ -35,8 +35,8 @@ def test_get_engine_api_version_json(): engine_api_version = { 'type': 'APIVersion', 'major': 1, - 'minor': 11, - 'micro': 11 + 'minor': 12, + 'micro': 0 } assert package_util.get_engine_api_version() == engine_api_version From 00c359d87874a959aee89ef0b531d06c51c81b95 Mon Sep 17 00:00:00 2001 From: vimleshmishra Date: Mon, 18 Jul 2022 20:55:16 +0530 Subject: [PATCH 06/21] CP-8293 Update version to 4.0.5.dev0 for virtualization-sdk develop branch before updating github actions (#442) * __hash__ is needed in python 3 * DLPX-79760 SnapshotDefinitionDataPaths object is not hashable since py2-3 transition * DOC-2234 Update new logo in developer.delphix.com * Resolved COnflicts * revert logo chnages * revert logo chnages * Resolve Conflicts * revert changes * revert logo.png * Resolve Conflicts * DOC-2234 Update new logo in developer.delphix.com * logo chnages * DOC-2234 Update new logo in developer.delphix.com * new favicon * incorporated review comments * DLPX-81195 develop version of vSDK needs to be bumped up to 4.0.4 for testing to use latest vSDK code * DLPX-81195 develop version of vSDK needs to be bumped up to 4.0.4 for testing to use latest vSDK code * Changes suggested by Aleks * CP-8174 VSDK Changes to support source sizing decorators * reverted dvp version related changes * reverted extra comma * update engine API version * Fixed Formatting Issues * Fixed Formatting Issues * Fixed Unit Tests * Fixed Unit Tests * Bumping up dvp-api, vsdk * uppdate formatting --- .bumpversion.cfg | 2 +- common/setup.py | 2 +- .../python/dlpx/virtualization/common/VERSION | 2 +- .../main/python/dlpx/virtualization/VERSION | 2 +- libs/setup.py | 2 +- .../python/dlpx/virtualization/libs/VERSION | 2 +- platform/setup.py | 2 +- .../dlpx/virtualization/platform/VERSION | 2 +- .../dlpx/virtualization/platform/_linked.py | 8 +++--- .../dlpx/virtualization/platform/_virtual.py | 4 +-- .../python/dlpx/virtualization/test_plugin.py | 25 ++++++++++++++++--- .../dlpx/virtualization/_internal/VERSION | 2 +- .../fake_plugin/direct/successful.py | 12 +++++++++ .../_internal/test_package_util.py | 8 +++--- 14 files changed, 53 insertions(+), 22 deletions(-) diff --git a/.bumpversion.cfg b/.bumpversion.cfg index c3164d8b..dd520fbc 100644 --- a/.bumpversion.cfg +++ b/.bumpversion.cfg @@ -1,5 +1,5 @@ [bumpversion] -current_version = 4.0.4 +current_version = 4.0.5.dev0 commit = False tag = False parse = (?P\d+)\.(?P\d+)\.(?P\d+)(\.(?P[a-z]+)(?P\d+))? diff --git a/common/setup.py b/common/setup.py index d3593d72..8943b6cc 100644 --- a/common/setup.py +++ b/common/setup.py @@ -4,7 +4,7 @@ PYTHON_SRC = 'src/main/python' install_requires = [ - "dvp-api == 1.6.3", + "dvp-api == 1.7.0.dev0", ] with open(os.path.join(PYTHON_SRC, 'dlpx/virtualization/common/VERSION')) as version_file: diff --git a/common/src/main/python/dlpx/virtualization/common/VERSION b/common/src/main/python/dlpx/virtualization/common/VERSION index c5106e6d..eaf58ac5 100644 --- a/common/src/main/python/dlpx/virtualization/common/VERSION +++ b/common/src/main/python/dlpx/virtualization/common/VERSION @@ -1 +1 @@ -4.0.4 +4.0.5.dev0 diff --git a/dvp/src/main/python/dlpx/virtualization/VERSION b/dvp/src/main/python/dlpx/virtualization/VERSION index c5106e6d..eaf58ac5 100644 --- a/dvp/src/main/python/dlpx/virtualization/VERSION +++ b/dvp/src/main/python/dlpx/virtualization/VERSION @@ -1 +1 @@ -4.0.4 +4.0.5.dev0 diff --git a/libs/setup.py b/libs/setup.py index f443840a..21f0aad2 100644 --- a/libs/setup.py +++ b/libs/setup.py @@ -7,7 +7,7 @@ version = version_file.read().strip() install_requires = [ - "dvp-api == 1.6.3", + "dvp-api == 1.7.0.dev0", "dvp-common == {}".format(version) ] diff --git a/libs/src/main/python/dlpx/virtualization/libs/VERSION b/libs/src/main/python/dlpx/virtualization/libs/VERSION index c5106e6d..eaf58ac5 100644 --- a/libs/src/main/python/dlpx/virtualization/libs/VERSION +++ b/libs/src/main/python/dlpx/virtualization/libs/VERSION @@ -1 +1 @@ -4.0.4 +4.0.5.dev0 diff --git a/platform/setup.py b/platform/setup.py index 6cd76b71..29b4ba6f 100644 --- a/platform/setup.py +++ b/platform/setup.py @@ -7,7 +7,7 @@ version = version_file.read().strip() install_requires = [ - "dvp-api == 1.6.3", + "dvp-api == 1.7.0.dev0", "dvp-common == {}".format(version), "enum34;python_version < '3.4'", ] diff --git a/platform/src/main/python/dlpx/virtualization/platform/VERSION b/platform/src/main/python/dlpx/virtualization/platform/VERSION index c5106e6d..eaf58ac5 100644 --- a/platform/src/main/python/dlpx/virtualization/platform/VERSION +++ b/platform/src/main/python/dlpx/virtualization/platform/VERSION @@ -1 +1 @@ -4.0.4 +4.0.5.dev0 diff --git a/platform/src/main/python/dlpx/virtualization/platform/_linked.py b/platform/src/main/python/dlpx/virtualization/platform/_linked.py index adecb6fc..5b780674 100644 --- a/platform/src/main/python/dlpx/virtualization/platform/_linked.py +++ b/platform/src/main/python/dlpx/virtualization/platform/_linked.py @@ -289,13 +289,13 @@ def _internal_direct_source_size(self, request): source_config = SourceConfigDefinition.from_dict( json.loads(request.source_config.parameters.json)) - size = self.source_size_impl( + source_size = self.source_size_impl( direct_source=direct_source, repository=repository, source_config=source_config) direct_source_size_response = platform_pb2.DirectSourceSizeResponse() - direct_source_size_response.return_value.database_size = size + direct_source_size_response.return_value.database_size = source_size return direct_source_size_response @@ -831,12 +831,12 @@ def _internal_staged_source_size(self, request): source_config = SourceConfigDefinition.from_dict( json.loads(request.source_config.parameters.json)) - size = self.source_size_impl( + source_size = self.source_size_impl( staged_source=staged_source, repository=repository, source_config=source_config) staged_source_size_response = platform_pb2.StagedSourceSizeResponse() - staged_source_size_response.return_value.database_size = size + staged_source_size_response.return_value.database_size = source_size return staged_source_size_response diff --git a/platform/src/main/python/dlpx/virtualization/platform/_virtual.py b/platform/src/main/python/dlpx/virtualization/platform/_virtual.py index 9f8ca084..05fa5cd2 100644 --- a/platform/src/main/python/dlpx/virtualization/platform/_virtual.py +++ b/platform/src/main/python/dlpx/virtualization/platform/_virtual.py @@ -839,12 +839,12 @@ def _internal_virtual_source_size(self, request): source_config = SourceConfigDefinition.from_dict( json.loads(request.source_config.parameters.json)) - size = self.source_size_impl( + source_size = self.source_size_impl( virtual_source=virtual_source, repository=repository, source_config=source_config) virtual_source_size_response = platform_pb2.VirtualSourceSizeResponse() - virtual_source_size_response.return_value.database_size = size + virtual_source_size_response.return_value.database_size = source_size return virtual_source_size_response diff --git a/platform/src/test/python/dlpx/virtualization/test_plugin.py b/platform/src/test/python/dlpx/virtualization/test_plugin.py index f69258ca..5dd65f3c 100755 --- a/platform/src/test/python/dlpx/virtualization/test_plugin.py +++ b/platform/src/test/python/dlpx/virtualization/test_plugin.py @@ -65,8 +65,8 @@ })) TEST_POST_UPGRADE_PARAMS = ({ u'obj': - '"{\\"obj\\": {\\"name\\": \\"upgrade\\", ' - '\\"prettyName\\": \\"prettyUpgrade\\", \\"metadata\\": \\"metadata\\"}}"' + '"{\\"obj\\": {\\"name\\": \\"upgrade\\", ' + '\\"prettyName\\": \\"prettyUpgrade\\", \\"metadata\\": \\"metadata\\"}}"' }) MIGRATION_IDS = ('2020.1.1', '2020.2.2') @@ -94,7 +94,6 @@ def configure_impl(): pass with pytest.raises(OperationAlreadyDefinedError): - @my_plugin.virtual.configure() # noqa F811 def configure_impl(): # noqa F811 pass @@ -1189,6 +1188,26 @@ def staged_status_impl(staged_source, repository, source_config): assert staged_status_response.return_value.status == expected_status + @staticmethod + def test_source_size(my_plugin, staged_source, repository, + source_config): + + @my_plugin.linked.source_size() + def staged_source_size_impl(staged_source, repository, source_config): + TestPlugin.assert_plugin_args(staged_source=staged_source, + repository=repository, + source_config=source_config) + return 0 + staged_source_size_request = platform_pb2.StagedSourceSizeRequest() + TestPlugin.setup_request(request=staged_source_size_request, + staged_source=staged_source, + repository=repository, + source_config=source_config) + staged_source_size_response = my_plugin.linked._internal_staged_source_size( + staged_source_size_request) + + assert staged_source_size_response.return_value.database_size == 0 + @staticmethod def test_staged_worker(my_plugin, staged_source, repository, source_config): diff --git a/tools/src/main/python/dlpx/virtualization/_internal/VERSION b/tools/src/main/python/dlpx/virtualization/_internal/VERSION index c5106e6d..eaf58ac5 100644 --- a/tools/src/main/python/dlpx/virtualization/_internal/VERSION +++ b/tools/src/main/python/dlpx/virtualization/_internal/VERSION @@ -1 +1 @@ -4.0.4 +4.0.5.dev0 diff --git a/tools/src/test/python/dlpx/virtualization/_internal/fake_plugin/direct/successful.py b/tools/src/test/python/dlpx/virtualization/_internal/fake_plugin/direct/successful.py index da568732..c03c499a 100644 --- a/tools/src/test/python/dlpx/virtualization/_internal/fake_plugin/direct/successful.py +++ b/tools/src/test/python/dlpx/virtualization/_internal/fake_plugin/direct/successful.py @@ -29,18 +29,25 @@ def direct_post_snapshot(direct_source, repository, source_config, return None +@direct.linked.source_size() +def source_size(repository, source_config, direct_source): + return 0 + + @direct.virtual.configure() def configure(virtual_source, repository, snapshot): path = virtual_source.parameters.path name = "VDB mounted to " + path return None + @direct.virtual.initialize() def initialize(virtual_source, repository): path = virtual_source.parameters.path name = "VDB mounted to " + path return None + @direct.virtual.mount_specification() def mount_specification(repository, virtual_source): return None @@ -86,6 +93,11 @@ def cleanup(repository, source_config, virtual_source): pass +@direct.virtual.source_size() +def source_size(repository, source_config, virtual_source): + return 0 + + @direct.upgrade.repository('1.3', MigrationType.LUA) def repo_upgrade(old_repository): return old_repository diff --git a/tools/src/test/python/dlpx/virtualization/_internal/test_package_util.py b/tools/src/test/python/dlpx/virtualization/_internal/test_package_util.py index 11b8b726..109935c5 100644 --- a/tools/src/test/python/dlpx/virtualization/_internal/test_package_util.py +++ b/tools/src/test/python/dlpx/virtualization/_internal/test_package_util.py @@ -10,11 +10,11 @@ class TestPackageUtil: @staticmethod def test_get_version(): - assert package_util.get_version() == '4.0.4' + assert package_util.get_version() == '4.0.5.dev0' @staticmethod def test_get_virtualization_api_version(): - assert package_util.get_virtualization_api_version() == '1.6.3' + assert package_util.get_virtualization_api_version() == '1.7.0' @staticmethod def test_get_engine_api_version(): @@ -25,8 +25,8 @@ def test_get_build_api_version_json(): build_api_version = { 'type': 'APIVersion', 'major': 1, - 'minor': 6, - 'micro': 3 + 'minor': 7, + 'micro': 0 } assert package_util.get_build_api_version() == build_api_version From 7a619d2b7100c758293716b5d7df7e84c5c39760 Mon Sep 17 00:00:00 2001 From: vimleshmishra Date: Mon, 18 Jul 2022 21:27:37 +0530 Subject: [PATCH 07/21] Revert "CP-8293 Update version to 4.0.5.dev0 for virtualization-sdk develop branch before updating github actions (#442)" (#443) This reverts commit 222517e1b5917ff1f28879050582fe02f8f85ae8. --- .bumpversion.cfg | 2 +- common/setup.py | 2 +- .../python/dlpx/virtualization/common/VERSION | 2 +- .../main/python/dlpx/virtualization/VERSION | 2 +- libs/setup.py | 2 +- .../python/dlpx/virtualization/libs/VERSION | 2 +- platform/setup.py | 2 +- .../dlpx/virtualization/platform/VERSION | 2 +- .../dlpx/virtualization/platform/_linked.py | 8 +++--- .../dlpx/virtualization/platform/_virtual.py | 4 +-- .../python/dlpx/virtualization/test_plugin.py | 25 +++---------------- .../dlpx/virtualization/_internal/VERSION | 2 +- .../fake_plugin/direct/successful.py | 12 --------- .../_internal/test_package_util.py | 8 +++--- 14 files changed, 22 insertions(+), 53 deletions(-) diff --git a/.bumpversion.cfg b/.bumpversion.cfg index dd520fbc..c3164d8b 100644 --- a/.bumpversion.cfg +++ b/.bumpversion.cfg @@ -1,5 +1,5 @@ [bumpversion] -current_version = 4.0.5.dev0 +current_version = 4.0.4 commit = False tag = False parse = (?P\d+)\.(?P\d+)\.(?P\d+)(\.(?P[a-z]+)(?P\d+))? diff --git a/common/setup.py b/common/setup.py index 8943b6cc..d3593d72 100644 --- a/common/setup.py +++ b/common/setup.py @@ -4,7 +4,7 @@ PYTHON_SRC = 'src/main/python' install_requires = [ - "dvp-api == 1.7.0.dev0", + "dvp-api == 1.6.3", ] with open(os.path.join(PYTHON_SRC, 'dlpx/virtualization/common/VERSION')) as version_file: diff --git a/common/src/main/python/dlpx/virtualization/common/VERSION b/common/src/main/python/dlpx/virtualization/common/VERSION index eaf58ac5..c5106e6d 100644 --- a/common/src/main/python/dlpx/virtualization/common/VERSION +++ b/common/src/main/python/dlpx/virtualization/common/VERSION @@ -1 +1 @@ -4.0.5.dev0 +4.0.4 diff --git a/dvp/src/main/python/dlpx/virtualization/VERSION b/dvp/src/main/python/dlpx/virtualization/VERSION index eaf58ac5..c5106e6d 100644 --- a/dvp/src/main/python/dlpx/virtualization/VERSION +++ b/dvp/src/main/python/dlpx/virtualization/VERSION @@ -1 +1 @@ -4.0.5.dev0 +4.0.4 diff --git a/libs/setup.py b/libs/setup.py index 21f0aad2..f443840a 100644 --- a/libs/setup.py +++ b/libs/setup.py @@ -7,7 +7,7 @@ version = version_file.read().strip() install_requires = [ - "dvp-api == 1.7.0.dev0", + "dvp-api == 1.6.3", "dvp-common == {}".format(version) ] diff --git a/libs/src/main/python/dlpx/virtualization/libs/VERSION b/libs/src/main/python/dlpx/virtualization/libs/VERSION index eaf58ac5..c5106e6d 100644 --- a/libs/src/main/python/dlpx/virtualization/libs/VERSION +++ b/libs/src/main/python/dlpx/virtualization/libs/VERSION @@ -1 +1 @@ -4.0.5.dev0 +4.0.4 diff --git a/platform/setup.py b/platform/setup.py index 29b4ba6f..6cd76b71 100644 --- a/platform/setup.py +++ b/platform/setup.py @@ -7,7 +7,7 @@ version = version_file.read().strip() install_requires = [ - "dvp-api == 1.7.0.dev0", + "dvp-api == 1.6.3", "dvp-common == {}".format(version), "enum34;python_version < '3.4'", ] diff --git a/platform/src/main/python/dlpx/virtualization/platform/VERSION b/platform/src/main/python/dlpx/virtualization/platform/VERSION index eaf58ac5..c5106e6d 100644 --- a/platform/src/main/python/dlpx/virtualization/platform/VERSION +++ b/platform/src/main/python/dlpx/virtualization/platform/VERSION @@ -1 +1 @@ -4.0.5.dev0 +4.0.4 diff --git a/platform/src/main/python/dlpx/virtualization/platform/_linked.py b/platform/src/main/python/dlpx/virtualization/platform/_linked.py index 5b780674..adecb6fc 100644 --- a/platform/src/main/python/dlpx/virtualization/platform/_linked.py +++ b/platform/src/main/python/dlpx/virtualization/platform/_linked.py @@ -289,13 +289,13 @@ def _internal_direct_source_size(self, request): source_config = SourceConfigDefinition.from_dict( json.loads(request.source_config.parameters.json)) - source_size = self.source_size_impl( + size = self.source_size_impl( direct_source=direct_source, repository=repository, source_config=source_config) direct_source_size_response = platform_pb2.DirectSourceSizeResponse() - direct_source_size_response.return_value.database_size = source_size + direct_source_size_response.return_value.database_size = size return direct_source_size_response @@ -831,12 +831,12 @@ def _internal_staged_source_size(self, request): source_config = SourceConfigDefinition.from_dict( json.loads(request.source_config.parameters.json)) - source_size = self.source_size_impl( + size = self.source_size_impl( staged_source=staged_source, repository=repository, source_config=source_config) staged_source_size_response = platform_pb2.StagedSourceSizeResponse() - staged_source_size_response.return_value.database_size = source_size + staged_source_size_response.return_value.database_size = size return staged_source_size_response diff --git a/platform/src/main/python/dlpx/virtualization/platform/_virtual.py b/platform/src/main/python/dlpx/virtualization/platform/_virtual.py index 05fa5cd2..9f8ca084 100644 --- a/platform/src/main/python/dlpx/virtualization/platform/_virtual.py +++ b/platform/src/main/python/dlpx/virtualization/platform/_virtual.py @@ -839,12 +839,12 @@ def _internal_virtual_source_size(self, request): source_config = SourceConfigDefinition.from_dict( json.loads(request.source_config.parameters.json)) - source_size = self.source_size_impl( + size = self.source_size_impl( virtual_source=virtual_source, repository=repository, source_config=source_config) virtual_source_size_response = platform_pb2.VirtualSourceSizeResponse() - virtual_source_size_response.return_value.database_size = source_size + virtual_source_size_response.return_value.database_size = size return virtual_source_size_response diff --git a/platform/src/test/python/dlpx/virtualization/test_plugin.py b/platform/src/test/python/dlpx/virtualization/test_plugin.py index 5dd65f3c..f69258ca 100755 --- a/platform/src/test/python/dlpx/virtualization/test_plugin.py +++ b/platform/src/test/python/dlpx/virtualization/test_plugin.py @@ -65,8 +65,8 @@ })) TEST_POST_UPGRADE_PARAMS = ({ u'obj': - '"{\\"obj\\": {\\"name\\": \\"upgrade\\", ' - '\\"prettyName\\": \\"prettyUpgrade\\", \\"metadata\\": \\"metadata\\"}}"' + '"{\\"obj\\": {\\"name\\": \\"upgrade\\", ' + '\\"prettyName\\": \\"prettyUpgrade\\", \\"metadata\\": \\"metadata\\"}}"' }) MIGRATION_IDS = ('2020.1.1', '2020.2.2') @@ -94,6 +94,7 @@ def configure_impl(): pass with pytest.raises(OperationAlreadyDefinedError): + @my_plugin.virtual.configure() # noqa F811 def configure_impl(): # noqa F811 pass @@ -1188,26 +1189,6 @@ def staged_status_impl(staged_source, repository, source_config): assert staged_status_response.return_value.status == expected_status - @staticmethod - def test_source_size(my_plugin, staged_source, repository, - source_config): - - @my_plugin.linked.source_size() - def staged_source_size_impl(staged_source, repository, source_config): - TestPlugin.assert_plugin_args(staged_source=staged_source, - repository=repository, - source_config=source_config) - return 0 - staged_source_size_request = platform_pb2.StagedSourceSizeRequest() - TestPlugin.setup_request(request=staged_source_size_request, - staged_source=staged_source, - repository=repository, - source_config=source_config) - staged_source_size_response = my_plugin.linked._internal_staged_source_size( - staged_source_size_request) - - assert staged_source_size_response.return_value.database_size == 0 - @staticmethod def test_staged_worker(my_plugin, staged_source, repository, source_config): diff --git a/tools/src/main/python/dlpx/virtualization/_internal/VERSION b/tools/src/main/python/dlpx/virtualization/_internal/VERSION index eaf58ac5..c5106e6d 100644 --- a/tools/src/main/python/dlpx/virtualization/_internal/VERSION +++ b/tools/src/main/python/dlpx/virtualization/_internal/VERSION @@ -1 +1 @@ -4.0.5.dev0 +4.0.4 diff --git a/tools/src/test/python/dlpx/virtualization/_internal/fake_plugin/direct/successful.py b/tools/src/test/python/dlpx/virtualization/_internal/fake_plugin/direct/successful.py index c03c499a..da568732 100644 --- a/tools/src/test/python/dlpx/virtualization/_internal/fake_plugin/direct/successful.py +++ b/tools/src/test/python/dlpx/virtualization/_internal/fake_plugin/direct/successful.py @@ -29,25 +29,18 @@ def direct_post_snapshot(direct_source, repository, source_config, return None -@direct.linked.source_size() -def source_size(repository, source_config, direct_source): - return 0 - - @direct.virtual.configure() def configure(virtual_source, repository, snapshot): path = virtual_source.parameters.path name = "VDB mounted to " + path return None - @direct.virtual.initialize() def initialize(virtual_source, repository): path = virtual_source.parameters.path name = "VDB mounted to " + path return None - @direct.virtual.mount_specification() def mount_specification(repository, virtual_source): return None @@ -93,11 +86,6 @@ def cleanup(repository, source_config, virtual_source): pass -@direct.virtual.source_size() -def source_size(repository, source_config, virtual_source): - return 0 - - @direct.upgrade.repository('1.3', MigrationType.LUA) def repo_upgrade(old_repository): return old_repository diff --git a/tools/src/test/python/dlpx/virtualization/_internal/test_package_util.py b/tools/src/test/python/dlpx/virtualization/_internal/test_package_util.py index 109935c5..11b8b726 100644 --- a/tools/src/test/python/dlpx/virtualization/_internal/test_package_util.py +++ b/tools/src/test/python/dlpx/virtualization/_internal/test_package_util.py @@ -10,11 +10,11 @@ class TestPackageUtil: @staticmethod def test_get_version(): - assert package_util.get_version() == '4.0.5.dev0' + assert package_util.get_version() == '4.0.4' @staticmethod def test_get_virtualization_api_version(): - assert package_util.get_virtualization_api_version() == '1.7.0' + assert package_util.get_virtualization_api_version() == '1.6.3' @staticmethod def test_get_engine_api_version(): @@ -25,8 +25,8 @@ def test_get_build_api_version_json(): build_api_version = { 'type': 'APIVersion', 'major': 1, - 'minor': 7, - 'micro': 0 + 'minor': 6, + 'micro': 3 } assert package_util.get_build_api_version() == build_api_version From 6533a5f39e31e4b0e4f8658692343574735d3491 Mon Sep 17 00:00:00 2001 From: vimleshmishra Date: Fri, 22 Jul 2022 17:04:55 +0530 Subject: [PATCH 08/21] CP-8293 Update version to 4.0.5.dev0 for virtualization-sdk develop branch before updating github actions (#444) * __hash__ is needed in python 3 * DLPX-79760 SnapshotDefinitionDataPaths object is not hashable since py2-3 transition * DOC-2234 Update new logo in developer.delphix.com * Resolved COnflicts * revert logo chnages * revert logo chnages * Resolve Conflicts * revert changes * revert logo.png * Resolve Conflicts * DOC-2234 Update new logo in developer.delphix.com * logo chnages * DOC-2234 Update new logo in developer.delphix.com * new favicon * incorporated review comments * DLPX-81195 develop version of vSDK needs to be bumped up to 4.0.4 for testing to use latest vSDK code * DLPX-81195 develop version of vSDK needs to be bumped up to 4.0.4 for testing to use latest vSDK code * Changes suggested by Aleks * CP-8174 VSDK Changes to support source sizing decorators * reverted dvp version related changes * reverted extra comma * update engine API version * Fixed Formatting Issues * Fixed Formatting Issues * Fixed Unit Tests * Fixed Unit Tests * Bumping up dvp-api, vsdk * uppdate formatting * for rerun the failed runs * bumpversion support dev bumping * revert bumpversion support dev bumping * bump up version from dev0 to dev1 * bump up version from dev0 to dev1 --- .bumpversion.cfg | 2 +- common/setup.py | 2 +- .../python/dlpx/virtualization/common/VERSION | 2 +- .../main/python/dlpx/virtualization/VERSION | 2 +- libs/setup.py | 2 +- .../python/dlpx/virtualization/libs/VERSION | 2 +- platform/setup.py | 2 +- .../dlpx/virtualization/platform/VERSION | 2 +- .../dlpx/virtualization/platform/_linked.py | 8 +++--- .../dlpx/virtualization/platform/_virtual.py | 4 +-- .../python/dlpx/virtualization/test_plugin.py | 25 ++++++++++++++++--- .../dlpx/virtualization/_internal/VERSION | 2 +- .../fake_plugin/direct/successful.py | 12 +++++++++ .../_internal/test_package_util.py | 8 +++--- 14 files changed, 53 insertions(+), 22 deletions(-) diff --git a/.bumpversion.cfg b/.bumpversion.cfg index c3164d8b..fedf6332 100644 --- a/.bumpversion.cfg +++ b/.bumpversion.cfg @@ -1,5 +1,5 @@ [bumpversion] -current_version = 4.0.4 +current_version = 4.0.5.dev1 commit = False tag = False parse = (?P\d+)\.(?P\d+)\.(?P\d+)(\.(?P[a-z]+)(?P\d+))? diff --git a/common/setup.py b/common/setup.py index d3593d72..8943b6cc 100644 --- a/common/setup.py +++ b/common/setup.py @@ -4,7 +4,7 @@ PYTHON_SRC = 'src/main/python' install_requires = [ - "dvp-api == 1.6.3", + "dvp-api == 1.7.0.dev0", ] with open(os.path.join(PYTHON_SRC, 'dlpx/virtualization/common/VERSION')) as version_file: diff --git a/common/src/main/python/dlpx/virtualization/common/VERSION b/common/src/main/python/dlpx/virtualization/common/VERSION index c5106e6d..1b348fba 100644 --- a/common/src/main/python/dlpx/virtualization/common/VERSION +++ b/common/src/main/python/dlpx/virtualization/common/VERSION @@ -1 +1 @@ -4.0.4 +4.0.5.dev1 diff --git a/dvp/src/main/python/dlpx/virtualization/VERSION b/dvp/src/main/python/dlpx/virtualization/VERSION index c5106e6d..1b348fba 100644 --- a/dvp/src/main/python/dlpx/virtualization/VERSION +++ b/dvp/src/main/python/dlpx/virtualization/VERSION @@ -1 +1 @@ -4.0.4 +4.0.5.dev1 diff --git a/libs/setup.py b/libs/setup.py index f443840a..21f0aad2 100644 --- a/libs/setup.py +++ b/libs/setup.py @@ -7,7 +7,7 @@ version = version_file.read().strip() install_requires = [ - "dvp-api == 1.6.3", + "dvp-api == 1.7.0.dev0", "dvp-common == {}".format(version) ] diff --git a/libs/src/main/python/dlpx/virtualization/libs/VERSION b/libs/src/main/python/dlpx/virtualization/libs/VERSION index c5106e6d..1b348fba 100644 --- a/libs/src/main/python/dlpx/virtualization/libs/VERSION +++ b/libs/src/main/python/dlpx/virtualization/libs/VERSION @@ -1 +1 @@ -4.0.4 +4.0.5.dev1 diff --git a/platform/setup.py b/platform/setup.py index 6cd76b71..29b4ba6f 100644 --- a/platform/setup.py +++ b/platform/setup.py @@ -7,7 +7,7 @@ version = version_file.read().strip() install_requires = [ - "dvp-api == 1.6.3", + "dvp-api == 1.7.0.dev0", "dvp-common == {}".format(version), "enum34;python_version < '3.4'", ] diff --git a/platform/src/main/python/dlpx/virtualization/platform/VERSION b/platform/src/main/python/dlpx/virtualization/platform/VERSION index c5106e6d..1b348fba 100644 --- a/platform/src/main/python/dlpx/virtualization/platform/VERSION +++ b/platform/src/main/python/dlpx/virtualization/platform/VERSION @@ -1 +1 @@ -4.0.4 +4.0.5.dev1 diff --git a/platform/src/main/python/dlpx/virtualization/platform/_linked.py b/platform/src/main/python/dlpx/virtualization/platform/_linked.py index adecb6fc..5b780674 100644 --- a/platform/src/main/python/dlpx/virtualization/platform/_linked.py +++ b/platform/src/main/python/dlpx/virtualization/platform/_linked.py @@ -289,13 +289,13 @@ def _internal_direct_source_size(self, request): source_config = SourceConfigDefinition.from_dict( json.loads(request.source_config.parameters.json)) - size = self.source_size_impl( + source_size = self.source_size_impl( direct_source=direct_source, repository=repository, source_config=source_config) direct_source_size_response = platform_pb2.DirectSourceSizeResponse() - direct_source_size_response.return_value.database_size = size + direct_source_size_response.return_value.database_size = source_size return direct_source_size_response @@ -831,12 +831,12 @@ def _internal_staged_source_size(self, request): source_config = SourceConfigDefinition.from_dict( json.loads(request.source_config.parameters.json)) - size = self.source_size_impl( + source_size = self.source_size_impl( staged_source=staged_source, repository=repository, source_config=source_config) staged_source_size_response = platform_pb2.StagedSourceSizeResponse() - staged_source_size_response.return_value.database_size = size + staged_source_size_response.return_value.database_size = source_size return staged_source_size_response diff --git a/platform/src/main/python/dlpx/virtualization/platform/_virtual.py b/platform/src/main/python/dlpx/virtualization/platform/_virtual.py index 9f8ca084..05fa5cd2 100644 --- a/platform/src/main/python/dlpx/virtualization/platform/_virtual.py +++ b/platform/src/main/python/dlpx/virtualization/platform/_virtual.py @@ -839,12 +839,12 @@ def _internal_virtual_source_size(self, request): source_config = SourceConfigDefinition.from_dict( json.loads(request.source_config.parameters.json)) - size = self.source_size_impl( + source_size = self.source_size_impl( virtual_source=virtual_source, repository=repository, source_config=source_config) virtual_source_size_response = platform_pb2.VirtualSourceSizeResponse() - virtual_source_size_response.return_value.database_size = size + virtual_source_size_response.return_value.database_size = source_size return virtual_source_size_response diff --git a/platform/src/test/python/dlpx/virtualization/test_plugin.py b/platform/src/test/python/dlpx/virtualization/test_plugin.py index f69258ca..5dd65f3c 100755 --- a/platform/src/test/python/dlpx/virtualization/test_plugin.py +++ b/platform/src/test/python/dlpx/virtualization/test_plugin.py @@ -65,8 +65,8 @@ })) TEST_POST_UPGRADE_PARAMS = ({ u'obj': - '"{\\"obj\\": {\\"name\\": \\"upgrade\\", ' - '\\"prettyName\\": \\"prettyUpgrade\\", \\"metadata\\": \\"metadata\\"}}"' + '"{\\"obj\\": {\\"name\\": \\"upgrade\\", ' + '\\"prettyName\\": \\"prettyUpgrade\\", \\"metadata\\": \\"metadata\\"}}"' }) MIGRATION_IDS = ('2020.1.1', '2020.2.2') @@ -94,7 +94,6 @@ def configure_impl(): pass with pytest.raises(OperationAlreadyDefinedError): - @my_plugin.virtual.configure() # noqa F811 def configure_impl(): # noqa F811 pass @@ -1189,6 +1188,26 @@ def staged_status_impl(staged_source, repository, source_config): assert staged_status_response.return_value.status == expected_status + @staticmethod + def test_source_size(my_plugin, staged_source, repository, + source_config): + + @my_plugin.linked.source_size() + def staged_source_size_impl(staged_source, repository, source_config): + TestPlugin.assert_plugin_args(staged_source=staged_source, + repository=repository, + source_config=source_config) + return 0 + staged_source_size_request = platform_pb2.StagedSourceSizeRequest() + TestPlugin.setup_request(request=staged_source_size_request, + staged_source=staged_source, + repository=repository, + source_config=source_config) + staged_source_size_response = my_plugin.linked._internal_staged_source_size( + staged_source_size_request) + + assert staged_source_size_response.return_value.database_size == 0 + @staticmethod def test_staged_worker(my_plugin, staged_source, repository, source_config): diff --git a/tools/src/main/python/dlpx/virtualization/_internal/VERSION b/tools/src/main/python/dlpx/virtualization/_internal/VERSION index c5106e6d..1b348fba 100644 --- a/tools/src/main/python/dlpx/virtualization/_internal/VERSION +++ b/tools/src/main/python/dlpx/virtualization/_internal/VERSION @@ -1 +1 @@ -4.0.4 +4.0.5.dev1 diff --git a/tools/src/test/python/dlpx/virtualization/_internal/fake_plugin/direct/successful.py b/tools/src/test/python/dlpx/virtualization/_internal/fake_plugin/direct/successful.py index da568732..c03c499a 100644 --- a/tools/src/test/python/dlpx/virtualization/_internal/fake_plugin/direct/successful.py +++ b/tools/src/test/python/dlpx/virtualization/_internal/fake_plugin/direct/successful.py @@ -29,18 +29,25 @@ def direct_post_snapshot(direct_source, repository, source_config, return None +@direct.linked.source_size() +def source_size(repository, source_config, direct_source): + return 0 + + @direct.virtual.configure() def configure(virtual_source, repository, snapshot): path = virtual_source.parameters.path name = "VDB mounted to " + path return None + @direct.virtual.initialize() def initialize(virtual_source, repository): path = virtual_source.parameters.path name = "VDB mounted to " + path return None + @direct.virtual.mount_specification() def mount_specification(repository, virtual_source): return None @@ -86,6 +93,11 @@ def cleanup(repository, source_config, virtual_source): pass +@direct.virtual.source_size() +def source_size(repository, source_config, virtual_source): + return 0 + + @direct.upgrade.repository('1.3', MigrationType.LUA) def repo_upgrade(old_repository): return old_repository diff --git a/tools/src/test/python/dlpx/virtualization/_internal/test_package_util.py b/tools/src/test/python/dlpx/virtualization/_internal/test_package_util.py index 11b8b726..501815f5 100644 --- a/tools/src/test/python/dlpx/virtualization/_internal/test_package_util.py +++ b/tools/src/test/python/dlpx/virtualization/_internal/test_package_util.py @@ -10,11 +10,11 @@ class TestPackageUtil: @staticmethod def test_get_version(): - assert package_util.get_version() == '4.0.4' + assert package_util.get_version() == '4.0.5.dev1' @staticmethod def test_get_virtualization_api_version(): - assert package_util.get_virtualization_api_version() == '1.6.3' + assert package_util.get_virtualization_api_version() == '1.7.0' @staticmethod def test_get_engine_api_version(): @@ -25,8 +25,8 @@ def test_get_build_api_version_json(): build_api_version = { 'type': 'APIVersion', 'major': 1, - 'minor': 6, - 'micro': 3 + 'minor': 7, + 'micro': 0 } assert package_util.get_build_api_version() == build_api_version From 1e3348fc8c2ef276eb158f607240dd97bc7f89d3 Mon Sep 17 00:00:00 2001 From: Vimlesh Mishra Date: Mon, 8 Aug 2022 17:47:02 +0530 Subject: [PATCH 09/21] virtualization-sdk master branch will always refer dlpx-appgate stage/release --- .../src/main/python/dlpx/virtualization/_internal/settings.cfg | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tools/src/main/python/dlpx/virtualization/_internal/settings.cfg b/tools/src/main/python/dlpx/virtualization/_internal/settings.cfg index 93477cdc..c80f2be1 100644 --- a/tools/src/main/python/dlpx/virtualization/_internal/settings.cfg +++ b/tools/src/main/python/dlpx/virtualization/_internal/settings.cfg @@ -20,7 +20,7 @@ # versions in those packages until they are shipped out of band. # [General] -engine_api_version = 1.12.0 +engine_api_version = 1.11.16 distribution_name = dvp-tools package_author = Delphix namespace_package = dlpx From 1c32d8647e730c345d5a1a709a52c01776fea3f7 Mon Sep 17 00:00:00 2001 From: Vimlesh Mishra Date: Mon, 8 Aug 2022 17:47:02 +0530 Subject: [PATCH 10/21] virtualization-sdk master branch will always refer dlpx-appgate stage/release --- tools/src/test/python/dlpx/virtualization/_internal/conftest.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tools/src/test/python/dlpx/virtualization/_internal/conftest.py b/tools/src/test/python/dlpx/virtualization/_internal/conftest.py index 3b05daae..57059fa9 100644 --- a/tools/src/test/python/dlpx/virtualization/_internal/conftest.py +++ b/tools/src/test/python/dlpx/virtualization/_internal/conftest.py @@ -710,7 +710,7 @@ def artifact_content(engine_api, virtual_source_definition, @pytest.fixture def engine_api(): - return {'type': 'APIVersion', 'major': 1, 'minor': 12, 'micro': 0} + return {'type': 'APIVersion', 'major': 1, 'minor': 11, 'micro': 16} @pytest.fixture From 859ba36b5f98c23a834a9dbb699c2ea3584929d0 Mon Sep 17 00:00:00 2001 From: Vimlesh Mishra Date: Mon, 8 Aug 2022 17:47:02 +0530 Subject: [PATCH 11/21] virtualization-sdk master branch will always refer dlpx-appgate stage/release --- .../python/dlpx/virtualization/_internal/test_package_util.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tools/src/test/python/dlpx/virtualization/_internal/test_package_util.py b/tools/src/test/python/dlpx/virtualization/_internal/test_package_util.py index 501815f5..4135a0a4 100644 --- a/tools/src/test/python/dlpx/virtualization/_internal/test_package_util.py +++ b/tools/src/test/python/dlpx/virtualization/_internal/test_package_util.py @@ -18,7 +18,7 @@ def test_get_virtualization_api_version(): @staticmethod def test_get_engine_api_version(): - assert package_util.get_engine_api_version_from_settings() == '1.12.0' + assert package_util.get_engine_api_version_from_settings() == '1.11.16' @staticmethod def test_get_build_api_version_json(): @@ -52,4 +52,4 @@ def test_get_internal_package_root(): ]) def test_get_external_version_string(version_string): assert package_util.get_external_version_string( - version_string) == '1.1.0' + version_string) == '1.11.16' From e825c03499919d57a239ca09fea4a6fe1b193da4 Mon Sep 17 00:00:00 2001 From: Vimlesh Mishra Date: Mon, 8 Aug 2022 17:47:02 +0530 Subject: [PATCH 12/21] virtualization-sdk master branch will always refer dlpx-appgate stage/release --- .../python/dlpx/virtualization/_internal/test_package_util.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tools/src/test/python/dlpx/virtualization/_internal/test_package_util.py b/tools/src/test/python/dlpx/virtualization/_internal/test_package_util.py index 4135a0a4..a72586d6 100644 --- a/tools/src/test/python/dlpx/virtualization/_internal/test_package_util.py +++ b/tools/src/test/python/dlpx/virtualization/_internal/test_package_util.py @@ -35,8 +35,8 @@ def test_get_engine_api_version_json(): engine_api_version = { 'type': 'APIVersion', 'major': 1, - 'minor': 12, - 'micro': 0 + 'minor': 11, + 'micro': 16 } assert package_util.get_engine_api_version() == engine_api_version From 64c7907c6afa10ae8d8bca0363d06ef4b5c30033 Mon Sep 17 00:00:00 2001 From: Vimlesh Mishra Date: Mon, 8 Aug 2022 17:47:02 +0530 Subject: [PATCH 13/21] virtualization-sdk master branch will always refer dlpx-appgate stage/release --- .../python/dlpx/virtualization/_internal/test_package_util.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tools/src/test/python/dlpx/virtualization/_internal/test_package_util.py b/tools/src/test/python/dlpx/virtualization/_internal/test_package_util.py index a72586d6..930d1a65 100644 --- a/tools/src/test/python/dlpx/virtualization/_internal/test_package_util.py +++ b/tools/src/test/python/dlpx/virtualization/_internal/test_package_util.py @@ -52,4 +52,4 @@ def test_get_internal_package_root(): ]) def test_get_external_version_string(version_string): assert package_util.get_external_version_string( - version_string) == '1.11.16' + version_string) == '1.1.0' From 8fd66cd870665ed89be58b2cd14ccfabaf97624e Mon Sep 17 00:00:00 2001 From: Vimlesh Mishra Date: Mon, 8 Aug 2022 18:12:02 +0530 Subject: [PATCH 14/21] Fixed Issue: Lint common - Python38 --- .../python/dlpx/virtualization/common/_common_classes.py | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/common/src/main/python/dlpx/virtualization/common/_common_classes.py b/common/src/main/python/dlpx/virtualization/common/_common_classes.py index 57a698f4..ad701824 100644 --- a/common/src/main/python/dlpx/virtualization/common/_common_classes.py +++ b/common/src/main/python/dlpx/virtualization/common/_common_classes.py @@ -65,7 +65,8 @@ def user(self): return self.__user def to_proto(self): - """Converts plugin class RemoteConnection to protobuf class common_pb2.RemoteConnection + """Converts plugin class RemoteConnection to protobuf class + common_pb2.RemoteConnection """ remote_connection = common_pb2.RemoteConnection() remote_connection.environment.CopyFrom(self.environment.to_proto()) @@ -74,7 +75,8 @@ def to_proto(self): @staticmethod def from_proto(connection): - """Converts protobuf class common_pb2.RemoteConnection to plugin class RemoteConnection + """Converts protobuf class common_pb2.RemoteConnection to plugin class + RemoteConnection """ if not isinstance(connection, common_pb2.RemoteConnection): raise IncorrectTypeError( @@ -411,7 +413,8 @@ def public_key(self): @staticmethod def from_proto(credentials_result): - """Converts protobuf class libs_pb2.CredentialsResult to plugin class KeyPairCredentials + """Converts protobuf class libs_pb2.CredentialsResult to plugin class + KeyPairCredentials """ if not isinstance(credentials_result, libs_pb2.CredentialsResult): raise IncorrectTypeError( From 0a08915a06c8d65a48a27431ed4997eb218371c2 Mon Sep 17 00:00:00 2001 From: Rakshanda Bhat <111853524+rakshandabhat@users.noreply.github.com> Date: Fri, 8 Sep 2023 10:53:26 +0530 Subject: [PATCH 15/21] CP-10502 Jython Removal documentation changes (#528) CP-10488 Jython Removal documentation changes (#527) * CP-10488 Jython Removal documentation changes * CP-10488 Jython Removal documentation changes * CP-10488 Jython Removal documentation changes --- docs/docs/References/Version_Compatibility.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/docs/References/Version_Compatibility.md b/docs/docs/References/Version_Compatibility.md index 9a5da617..48d79947 100644 --- a/docs/docs/References/Version_Compatibility.md +++ b/docs/docs/References/Version_Compatibility.md @@ -11,7 +11,7 @@ | [3.0.0](../Release_Notes/3.0.0/3.0.0.md) | 6.0.6.0 | [Latest Release](https://cd.delphix.com/docs/latest/new-features) | | [2.1.0](../Release_Notes/2.1.0/2.1.0.md) | 6.0.3.0 | [Latest Release](https://cd.delphix.com/docs/latest/new-features) | | [2.0.0](../Release_Notes/2.0.0/2.0.0.md) | 6.0.2.0 | [Latest Release](https://cd.delphix.com/docs/latest/new-features) | -| [1.0.0](../Release_Notes/1.0.0/1.0.0.md) | 6.0.2.0 | [Latest Release](https://cd.delphix.com/docs/latest/new-features) | +| [1.0.0](../Release_Notes/1.0.0/1.0.0.md) | 6.0.2.0 | 14.0.0.0 | | [0.4.0](../Release_Notes/0.4.0/0.4.0.md) | 5.3.5.0 | 6.0.1.0 | ## Virtualization SDK and Python Compatibility Map From 5df3b7df2430fb4964f31841f3f26b571b44ae9d Mon Sep 17 00:00:00 2001 From: Rakshanda Bhat <111853524+rakshandabhat@users.noreply.github.com> Date: Wed, 9 Apr 2025 11:46:53 +0530 Subject: [PATCH 16/21] CP-12356 DVP Record size validation changes - Release DVP on PyPi (#561) * Configure Mend for GitHub.com (#543) * Add .whitesource configuration file * Configure Mend for GitHub.com --------- Co-authored-by: mend-for-github-com[bot] <50673670+mend-for-github-com[bot]@users.noreply.github.com> Co-authored-by: Shibasish Nandi <119420781+ShibasishDelphix@users.noreply.github.com> * CP-12279 Documentation for ZFS record size change (#554) * CP-12230 DVP Record size validation changes for setting ZFS record size (#556) * CP-12230 DVP Record size validation changes for setting ZFS record size * CP-12367 Record size validation - Bump DVP version (#558) * CP-12377 Record size validation - DVP version documentation (#560) Record size validation - DVP version documentation --------- Co-authored-by: mend-for-github-com[bot] <50673670+mend-for-github-com[bot]@users.noreply.github.com> Co-authored-by: Shibasish Nandi <119420781+ShibasishDelphix@users.noreply.github.com> --- .bumpversion.cfg | 2 +- .whitesource | 48 +++++++++++++++++ .../python/dlpx/virtualization/common/VERSION | 2 +- docs/docs/References/Glossary.md | 3 ++ .../Schemas_and_Autogenerated_Classes.md | 28 ++++++++++ docs/docs/References/Version_Compatibility.md | 2 + docs/docs/Release_Notes/.pages | 1 + docs/docs/Release_Notes/5.0.1/5.0.1.md | 11 ++++ docs/mkdocs.yml | 4 +- .../main/python/dlpx/virtualization/VERSION | 2 +- .../python/dlpx/virtualization/libs/VERSION | 2 +- .../dlpx/virtualization/platform/VERSION | 2 +- .../dlpx/virtualization/_internal/VERSION | 2 +- .../validation_schemas/plugin_schema.json | 27 +++++++++- .../_internal/test_package_util.py | 2 +- .../_internal/test_schema_validator.py | 53 +++++++++++++++++++ 16 files changed, 180 insertions(+), 11 deletions(-) create mode 100644 .whitesource create mode 100644 docs/docs/Release_Notes/5.0.1/5.0.1.md diff --git a/.bumpversion.cfg b/.bumpversion.cfg index 72126112..2f9a3bc7 100644 --- a/.bumpversion.cfg +++ b/.bumpversion.cfg @@ -1,5 +1,5 @@ [bumpversion] -current_version = 5.0.0 +current_version = 5.0.1 commit = False tag = False parse = (?P\d+)\.(?P\d+)\.(?P\d+)(\.(?P[a-z]+)(?P\d+))? diff --git a/.whitesource b/.whitesource new file mode 100644 index 00000000..e82e82f3 --- /dev/null +++ b/.whitesource @@ -0,0 +1,48 @@ +{ + "scanSettings": { + "configMode": "AUTO", + "configExternalURL": "", + "projectToken": "", + "enableLicenseViolations": "true", + "displayLicenseViolations": "true", + "enableIaC": "true", + "baseBranches": [] + }, + "scanSettingsSAST": { + "enableScan": true, + "scanPullRequests": true, + "incrementalScan": true, + "baseBranches": [], + "snippetSize": 10 + }, + "checkRunSettings": { + "vulnerableCheckRunConclusionLevel": "failure", + "displayMode": "diff", + "useMendCheckNames": true + }, + "checkRunSettingsSAST": { + "checkRunConclusionLevel": "failure", + "severityThreshold": "high" + }, + "issueSettings": { + "minSeverityLevel": "LOW", + "issueType": "DEPENDENCY" + }, + "issueSettingsSAST": { + "minSeverityLevel": "high", + "issueType": "repo" + }, + "remediateSettings": { + "workflowRules": { + "enabled": true + } + }, + "imageSettings":{ + "imageTracing":{ + "enableImageTracingPR": false, + "addRepositoryCoordinate": false, + "addDockerfilePath": false, + "addMendIdentifier": false + } + } +} \ No newline at end of file diff --git a/common/src/main/python/dlpx/virtualization/common/VERSION b/common/src/main/python/dlpx/virtualization/common/VERSION index 0062ac97..6b244dcd 100644 --- a/common/src/main/python/dlpx/virtualization/common/VERSION +++ b/common/src/main/python/dlpx/virtualization/common/VERSION @@ -1 +1 @@ -5.0.0 +5.0.1 diff --git a/docs/docs/References/Glossary.md b/docs/docs/References/Glossary.md index cf10b678..7aba6042 100644 --- a/docs/docs/References/Glossary.md +++ b/docs/docs/References/Glossary.md @@ -78,6 +78,9 @@ For example, a MySQL plugin might provide an operation called "stop" which knows ## Provisioning The process of making a virtual copy of a dataset and making it available for use on a target environment. +## Record Size +Delphix allows plugins to override the default record size for the linked source and virtual source datafiles. If the “recordSizeInKB” parameter is passed, it will be used to override the default record size and set the new record size for the respective dSources and empty VDBs. VDBs with parent will be inheriting the record size from their parent dSource or VDB. If the parameter is not passed, then default 8K record size will be set for the datafiles. + ## Replication Delphix allows end users to replicate data objects between Delphix Engines by creating a replication profile. Data objects that belong to a plugin can also be part of the replication profile. Refer to the [Delphix Engine Documentation](https://cd.delphix.com/docs/latest/) for more details. diff --git a/docs/docs/References/Schemas_and_Autogenerated_Classes.md b/docs/docs/References/Schemas_and_Autogenerated_Classes.md index 2c993835..d396d6e6 100644 --- a/docs/docs/References/Schemas_and_Autogenerated_Classes.md +++ b/docs/docs/References/Schemas_and_Autogenerated_Classes.md @@ -122,6 +122,20 @@ Defines properties used to identify [linked sources](Glossary.md#linked-source). } ``` +To set [record size](Glossary.md#record-size) for the datafiles of the linked source, pass the property "recordSizeInKB" in the [LinkedSourceDefinition Schema](#linkedsourcedefinition-schema). The value of the recordSizeInKB property must be an integer and power of 2 ranging from 1 to 16384. It will set the value of record size in KB. If this parameter is not passed, then default 8K record size will be set. +```json +{ + "type": "object", + "required": ["name", "port"], + "additionalProperties": false, + "properties": { + "name": { "type": "string" }, + "port": { "type": "integer" }, + "recordSizeInKB": {"type": "integer"} + } +} +``` + ### LinkedSourceDefinition Class Autogenerated based on the [LinkedSourceDefinition Schema](#linkedsourcedefinition-schema). @@ -164,6 +178,20 @@ Defines properties used to identify [virtual sources](Glossary.md#virtual-source } ``` +To set [record size](Glossary.md#record-size) for the datafiles of the empty VDB, pass the property "recordSizeInKB" in the [VirtualSourceDefinition Schema](#virtualsourcedefinition-schema). The value of the recordSizeInKB property must be an integer and power of 2 ranging from 1 to 16384. It will set the value of record size in KB. If this parameter is not passed, then default 8K record size will be set. VDB with parent source will inherit the record size value from its parent dSource or VDB. +```json +{ + "type": "object", + "required": ["name", "port"], + "additionalProperties": false, + "properties": { + "name": { "type": "string" }, + "port": { "type": "integer" }, + "recordSizeInKB": {"type": "integer"} + } +} +``` + ### VirtualSourceDefinition Class Autogenerated based on the [VirtualSourceDefinition Schema](#virtualsourcedefinition-schema). diff --git a/docs/docs/References/Version_Compatibility.md b/docs/docs/References/Version_Compatibility.md index c855eb33..0507dc89 100644 --- a/docs/docs/References/Version_Compatibility.md +++ b/docs/docs/References/Version_Compatibility.md @@ -4,6 +4,7 @@ | vSDK Version | Earliest Supported DE Version | Latest Supported DE Version | |------------------------------------------|:-----------------------------:|:-----------------------------------------------------------------:| +| [5.0.1](../Release_Notes/5.0.1/5.0.1.md) | 2025.2.0.0 | [Latest Release](https://cd.delphix.com/docs/latest/new-features) | | [5.0.0](../Release_Notes/5.0.0/5.0.0.md) | 29.0.0.0 | [Latest Release](https://cd.delphix.com/docs/latest/new-features) | | [4.1.0](../Release_Notes/4.1.0/4.1.0.md) | 12.0.0.0 | [Latest Release](https://cd.delphix.com/docs/latest/new-features) | | [4.0.5](../Release_Notes/4.0.5/4.0.5.md) | 6.0.16.0 | [Latest Release](https://cd.delphix.com/docs/latest/new-features) | @@ -19,6 +20,7 @@ | vSDK Version | Python Version | |------------------------------------------|:--------------:| +| [5.0.1](../Release_Notes/5.0.1/5.0.1.md) | 3.11 | | [5.0.0](../Release_Notes/5.0.0/5.0.0.md) | 3.11 | | [4.1.0](../Release_Notes/4.1.0/4.1.0.md) | 3.8 | | [4.0.5](../Release_Notes/4.0.5/4.0.5.md) | 3.8 | diff --git a/docs/docs/Release_Notes/.pages b/docs/docs/Release_Notes/.pages index 9f67f5fe..17365274 100644 --- a/docs/docs/Release_Notes/.pages +++ b/docs/docs/Release_Notes/.pages @@ -1,4 +1,5 @@ arrange: + - 5.0.1 - 5.0.0 - 4.1.0 - 4.0.5 diff --git a/docs/docs/Release_Notes/5.0.1/5.0.1.md b/docs/docs/Release_Notes/5.0.1/5.0.1.md new file mode 100644 index 00000000..a37a0437 --- /dev/null +++ b/docs/docs/Release_Notes/5.0.1/5.0.1.md @@ -0,0 +1,11 @@ +# Release - v5.0.1 + +To install or upgrade the SDK, refer to instructions [here](../../Getting_Started.md#installation). + +## New & Improved + +* Enhanced schema validation to set [record size](../../References/Glossary.md#record-size). + +## Breaking Changes + +* No breaking changes in this release! \ No newline at end of file diff --git a/docs/mkdocs.yml b/docs/mkdocs.yml index 147990ad..35f13b30 100644 --- a/docs/mkdocs.yml +++ b/docs/mkdocs.yml @@ -1,4 +1,4 @@ -site_name: Delphix Virtualization SDK 5.0.0 +site_name: Delphix Virtualization SDK 5.0.1 repo_name: Delphix Virtualization SDK repo_url: https://github.com/delphix/virtualization-sdk/ @@ -20,7 +20,7 @@ theme: - content.code.copy - content.code.annotate -copyright: Copyright © 2024 Delphix Corp. +copyright: Copyright © 2025 Delphix Corp. extra: analytics: diff --git a/dvp/src/main/python/dlpx/virtualization/VERSION b/dvp/src/main/python/dlpx/virtualization/VERSION index 0062ac97..6b244dcd 100644 --- a/dvp/src/main/python/dlpx/virtualization/VERSION +++ b/dvp/src/main/python/dlpx/virtualization/VERSION @@ -1 +1 @@ -5.0.0 +5.0.1 diff --git a/libs/src/main/python/dlpx/virtualization/libs/VERSION b/libs/src/main/python/dlpx/virtualization/libs/VERSION index 0062ac97..6b244dcd 100644 --- a/libs/src/main/python/dlpx/virtualization/libs/VERSION +++ b/libs/src/main/python/dlpx/virtualization/libs/VERSION @@ -1 +1 @@ -5.0.0 +5.0.1 diff --git a/platform/src/main/python/dlpx/virtualization/platform/VERSION b/platform/src/main/python/dlpx/virtualization/platform/VERSION index 0062ac97..6b244dcd 100644 --- a/platform/src/main/python/dlpx/virtualization/platform/VERSION +++ b/platform/src/main/python/dlpx/virtualization/platform/VERSION @@ -1 +1 @@ -5.0.0 +5.0.1 diff --git a/tools/src/main/python/dlpx/virtualization/_internal/VERSION b/tools/src/main/python/dlpx/virtualization/_internal/VERSION index 0062ac97..6b244dcd 100644 --- a/tools/src/main/python/dlpx/virtualization/_internal/VERSION +++ b/tools/src/main/python/dlpx/virtualization/_internal/VERSION @@ -1 +1 @@ -5.0.0 +5.0.1 diff --git a/tools/src/main/python/dlpx/virtualization/_internal/validation_schemas/plugin_schema.json b/tools/src/main/python/dlpx/virtualization/_internal/validation_schemas/plugin_schema.json index edbfeab6..59c39057 100644 --- a/tools/src/main/python/dlpx/virtualization/_internal/validation_schemas/plugin_schema.json +++ b/tools/src/main/python/dlpx/virtualization/_internal/validation_schemas/plugin_schema.json @@ -166,6 +166,15 @@ "required": ["nameField", "identityFields"], "nameField": { "type": "string" }, "identityFields": { "type": "array", "minItems": 1, "items": { "type": "string" } } + }, + "recordSizeObject": { + "type": "object", + "properties": { + "properties": { + "properties": { + "recordSizeInKB" : {"properties": {"type": {"enum" : ["integer"]}}}} + } + } } }, "type": "object", @@ -200,10 +209,24 @@ ] }, "virtualSourceDefinition": { - "$ref": "#/definitions/jsonSchema" + "allOf": [ + { + "$ref": "#/definitions/jsonSchema" + }, + { + "$ref": "#/definitions/recordSizeObject" + } + ] }, "linkedSourceDefinition": { - "$ref": "#/definitions/jsonSchema" + "allOf": [ + { + "$ref": "#/definitions/jsonSchema" + }, + { + "$ref": "#/definitions/recordSizeObject" + } + ] }, "snapshotDefinition": { "$ref": "#/definitions/jsonSchema" diff --git a/tools/src/test/python/dlpx/virtualization/_internal/test_package_util.py b/tools/src/test/python/dlpx/virtualization/_internal/test_package_util.py index 669fc919..22779c64 100644 --- a/tools/src/test/python/dlpx/virtualization/_internal/test_package_util.py +++ b/tools/src/test/python/dlpx/virtualization/_internal/test_package_util.py @@ -7,7 +7,7 @@ from dlpx.virtualization._internal import package_util -DVP_VERSION = '5.0.0' +DVP_VERSION = '5.0.1' DVP_API_VERSION = '1.9.0' diff --git a/tools/src/test/python/dlpx/virtualization/_internal/test_schema_validator.py b/tools/src/test/python/dlpx/virtualization/_internal/test_schema_validator.py index 5f735a79..3dbed667 100644 --- a/tools/src/test/python/dlpx/virtualization/_internal/test_schema_validator.py +++ b/tools/src/test/python/dlpx/virtualization/_internal/test_schema_validator.py @@ -356,3 +356,56 @@ def test_bad_type_in_array(schema_file): message = err_info.value.message assert ("'strings' is not valid under any of the given schemas" in message) + + @staticmethod + @pytest.mark.parametrize('linked_source_definition', + [{ + 'type': 'object', + 'additionalProperties': False, + 'properties': { + 'recordSizeInKB': { + 'type': 'string' + } + } + }]) + def test_bad_recordSizeInKB_type_string(schema_file): + with pytest.raises(exceptions.SchemaValidationError) as err_info: + validator = SchemaValidator(schema_file, const.PLUGIN_SCHEMA) + validator.validate() + + message = err_info.value.message + assert "'string' is not one of ['integer']" in message + + @staticmethod + @pytest.mark.parametrize('virtual_source_definition', + [{ + 'type': 'object', + 'additionalProperties': False, + 'properties': { + 'recordSizeInKB': { + 'type': 'array' + } + } + }]) + def test_bad_recordSizeInKB_type_array_virtual(schema_file): + with pytest.raises(exceptions.SchemaValidationError) as err_info: + validator = SchemaValidator(schema_file, const.PLUGIN_SCHEMA) + validator.validate() + + message = err_info.value.message + assert "'array' is not one of ['integer']" in message + + @staticmethod + @pytest.mark.parametrize('virtual_source_definition', + [{ + 'type': 'object', + 'additionalProperties': False, + 'properties': { + 'recordSizeInKB': { + 'type': 'integer' + } + } + }]) + def test_recordSizeInKB_type_virtual(schema_file): + validator = SchemaValidator(schema_file, const.PLUGIN_SCHEMA) + validator.validate() From 1c80e9cfc39dfeda638bf4f9b754c80946a445e3 Mon Sep 17 00:00:00 2001 From: vimleshmishra Date: Thu, 26 Jun 2025 20:30:05 +0530 Subject: [PATCH 17/21] DLPXDOC-3433 Typo in instructions for virtualization sdk docs (#563) --- docs/docs/Getting_Started.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/docs/Getting_Started.md b/docs/docs/Getting_Started.md index 59e59864..9bc518c6 100644 --- a/docs/docs/Getting_Started.md +++ b/docs/docs/Getting_Started.md @@ -3,7 +3,7 @@ The Virtualization SDK is a Python package on [PyPI](https://pypi.org/project/dv The SDK consists of three parts: -- The `dlpx.virtulization.platform` module +- The `dlpx.virtualization.platform` module - The `dlpx.virtualization.libs` module - A CLI From d5eef3bdc723d9efcbcd93e95ffbadb36fc35c20 Mon Sep 17 00:00:00 2001 From: Rakshanda Bhat <111853524+rakshandabhat@users.noreply.github.com> Date: Thu, 7 May 2026 15:37:15 +0530 Subject: [PATCH 18/21] CP-13245 Add azure vault support in developer.delphix.com (#639) * CP-13245 Add azure vault support in developer.delphix.com * Updated old links to new ones --- .../Overview.md | 8 +-- docs/docs/References/Glossary.md | 2 +- docs/docs/References/Schemas.md | 61 +++++++++++++++++-- docs/docs/References/Version_Compatibility.md | 20 +++--- .../Versioning_And_Upgrade/Replication.md | 6 +- 5 files changed, 74 insertions(+), 23 deletions(-) diff --git a/docs/docs/References/Dynamic_UI_Schema_Configuration/Overview.md b/docs/docs/References/Dynamic_UI_Schema_Configuration/Overview.md index 2b3c3cd4..2dad668b 100644 --- a/docs/docs/References/Dynamic_UI_Schema_Configuration/Overview.md +++ b/docs/docs/References/Dynamic_UI_Schema_Configuration/Overview.md @@ -28,8 +28,8 @@ Table below lists all the allowed attributes within [dxFormProperties](../Schema | Dynamic UI Schema Configuration | Earliest Supported vSDK Version | Latest Supported vSDK Version | Earliest Supported DE Version | Latest Supported DE Version | |:---------------------------------------------:|:--------------------------------------------:|:-----------------------------------------------:|:-----------------------------:|:-----------------------------------------------------------------:| -| [Collapsible Section](Collapsible_Section.md) | [4.0.2](https://pypi.org/project/dvp/4.0.2/) | [Latest Release](https://pypi.org/project/dvp/) | 14.0.0.0 | [Latest Release](https://cd.delphix.com/docs/latest/new-features) | -| [Hidden Fields](Hidden_Fields.md) | [4.0.2](https://pypi.org/project/dvp/4.0.2/) | [Latest Release](https://pypi.org/project/dvp/) | 14.0.0.0 | [Latest Release](https://cd.delphix.com/docs/latest/new-features) | -| [Text Area](Text_Area.md) | [4.0.2](https://pypi.org/project/dvp/4.0.2/) | [Latest Release](https://pypi.org/project/dvp/) | 14.0.0.0 | [Latest Release](https://cd.delphix.com/docs/latest/new-features) | -| [Validation Messages](Validation_Messages.md) | [4.0.2](https://pypi.org/project/dvp/4.0.2/) | [Latest Release](https://pypi.org/project/dvp/) | 14.0.0.0 | [Latest Release](https://cd.delphix.com/docs/latest/new-features) | +| [Collapsible Section](Collapsible_Section.md) | [4.0.2](https://pypi.org/project/dvp/4.0.2/) | [Latest Release](https://pypi.org/project/dvp/) | 14.0.0.0 | [Latest Release](https://help.delphix.com/cd/current/content/release_notes.htm) | +| [Hidden Fields](Hidden_Fields.md) | [4.0.2](https://pypi.org/project/dvp/4.0.2/) | [Latest Release](https://pypi.org/project/dvp/) | 14.0.0.0 | [Latest Release](https://help.delphix.com/cd/current/content/release_notes.htm) | +| [Text Area](Text_Area.md) | [4.0.2](https://pypi.org/project/dvp/4.0.2/) | [Latest Release](https://pypi.org/project/dvp/) | 14.0.0.0 | [Latest Release](https://help.delphix.com/cd/current/content/release_notes.htm) | +| [Validation Messages](Validation_Messages.md) | [4.0.2](https://pypi.org/project/dvp/4.0.2/) | [Latest Release](https://pypi.org/project/dvp/) | 14.0.0.0 | [Latest Release](https://help.delphix.com/cd/current/content/release_notes.htm) | diff --git a/docs/docs/References/Glossary.md b/docs/docs/References/Glossary.md index a1b949c6..e8af27b5 100644 --- a/docs/docs/References/Glossary.md +++ b/docs/docs/References/Glossary.md @@ -88,7 +88,7 @@ The process of making a virtual copy of a dataset and making it available for us Delphix allows plugins to override the default record size for the linked source and virtual source datafiles. If the “recordSizeInKB” parameter is passed, it will be used to override the default record size and set the new record size for the respective dSources and empty VDBs. VDBs with parent will be inheriting the record size from their parent dSource or VDB. If the parameter is not passed, then default 8K record size will be set for the datafiles. ## Replication -Delphix allows end users to replicate data objects between Delphix Engines by creating a replication profile. Data objects that belong to a plugin can also be part of the replication profile. Refer to the [Delphix Engine Documentation](https://cd.delphix.com/docs/latest/) for more details. +Delphix allows end users to replicate data objects between Delphix Engines by creating a replication profile. Data objects that belong to a plugin can also be part of the replication profile. Refer to the [Delphix Engine Documentation](https://help.delphix.com/cd/current/content/home.htm) for more details. ## Repository Information that represents a set of dependencies that a dataset requires in order to be functional. For example, a particular Postgres database might require an installed Postgres 9.6 DBMS, and so its associated repository would contain all the information required to interact with that DBMS. diff --git a/docs/docs/References/Schemas.md b/docs/docs/References/Schemas.md index 745d867f..5972db2c 100644 --- a/docs/docs/References/Schemas.md +++ b/docs/docs/References/Schemas.md @@ -434,7 +434,7 @@ An example of a JSON schema using this type is: ``` where `credentialsSupplier` is a definition in the external schema `https://delphix.com/platform/api`. -When providing data for a property of this type, the user has the following four options. +When providing data for a property of this type, the user has the following five options. ##### Option 1: Username and password @@ -541,7 +541,7 @@ For this option, the user must provide data that satisfies this definition: } } ``` -where `type` is a constant that the user interface will submit automatically on behalf of the user, `vault` is a reference to a CyberArk vault configured in the system, and `queryString` is a parameter for locating the credentials in the vault. For details on configuring and using CyberArk vaults, see the [password-vaults documentation for the Delphix engine](https://cd.delphix.com/docs/latest/password-vault-support). +where `type` is a constant that the user interface will submit automatically on behalf of the user, `vault` is a reference to a CyberArk vault configured in the system, and `queryString` is a parameter for locating the credentials in the vault. For details on configuring and using CyberArk vaults, see the [password-vaults documentation for the Delphix engine](https://help.delphix.com/cd/current/content/password_vault_support.htm). Optionally, `expectedSecretType` lets the user constrain the secret returned by the vault to passwords or keys (the default is to allow `any` of those two types of secret). An unexpected type of secret returned by the vault will result in a runtime exception. @@ -593,7 +593,7 @@ For this option, the user must provide data that satisfies this definition: } } ``` -where `type` is a constant that the user interface will submit automatically on behalf of the user, `vault` is a reference to a HashiCorp vault configured in the system, and `engine`, `path`, `usernameKey` and `secretKey` are parameters for locating the credentials in the vault. For details on configuring and using HashiCorp vaults, see the [password-vaults documentation for the Delphix engine](https://cd.delphix.com/docs/latest/password-vault-support). +where `type` is a constant that the user interface will submit automatically on behalf of the user, `vault` is a reference to a HashiCorp vault configured in the system, and `engine`, `path`, `usernameKey` and `secretKey` are parameters for locating the credentials in the vault. For details on configuring and using HashiCorp vaults, see the [password-vaults documentation for the Delphix engine](https://help.delphix.com/cd/current/content/password_vault_support.htm). Optionally, `expectedSecretType` lets the user constrain the secret returned by the vault to passwords or keys (the default is to allow `any` of those two types of secret). An unexpected type of secret returned by the vault will result in a runtime exception. @@ -610,13 +610,64 @@ For example, the user, or the user interface on behalf of the user, can provide: } ``` +##### Option 5: Azure Vault credentials + +For this option, the user must provide data that satisfies this definition: +```json +{ + "type": "object", + "required": ["type", "vault", "azureVaultName", "usernameKey", "secretKey"], + "properties": { + "type": { + "type": "string", + "const": "AzureVaultCredential" + }, + "vault": { + "type": "string", + "format": "reference", + "referenceType": "AzureVault" + }, + "azureVaultName": { + "type": "string" + }, + "usernameKey": { + "type": "string" + }, + "secretKey": { + "type": "string" + }, + "expectedSecretType": { + "type": "string", + "enum": ["any", "password", "keyPair"], + "default": "any" + } + } +} +``` +where `type` is a constant that the user interface will submit automatically on behalf of the user, `vault` is a reference to an Azure Vault configured in the system, and `azureVaultName`, `usernameKey` and `secretKey` are parameters for locating the credentials in the vault. For details on configuring and using Azure vaults, see the [password-vaults documentation for the Delphix engine](https://help.delphix.com/cd/current/content/password_vault_support.htm). + +Optionally, `expectedSecretType` lets the user constrain the secret returned by the vault to passwords or keys (the default is to allow `any` of those two types of secret). An unexpected type of secret returned by the vault will result in a runtime exception. + +For example, the user, or the user interface on behalf of the user, can provide: +```json +"properties": { + "myCredentials": { + "type": "AzureVaultCredential", + "vault": "AZURE_VAULT-1", + "azureVaultName": "my-azure-vault", + "usernameKey": "username", + "secretKey": "password" + } +} +``` + #### `keyCredentialsSupplier` -This object type is identical to `credentialsSupplier` but requires the secrets to be keys. The available options are [keys](#option-2-username-and-keys), [CyberArk vaults](#option-3-cyberark-vault-credentials) and [HashiCorp vaults](#option-4-hashicorp-vault-credentials). The property `expectedSecretType` is required in all cases and must have the value `keyPair`. +This object type is identical to `credentialsSupplier` but requires the secrets to be keys. The available options are [keys](#option-2-username-and-keys), [CyberArk vaults](#option-3-cyberark-vault-credentials), [HashiCorp vaults](#option-4-hashicorp-vault-credentials) and [Azure vaults](#option-5-azure-vault-credentials). The property `expectedSecretType` is required in all cases and must have the value `keyPair`. #### `passwordCredentialsSupplier` -This object type is identical to `credentialsSupplier` but requires the secrets to be passwords. The available options are [passwords](#option-1-username-and-password), [CyberArk vaults](#option-3-cyberark-vault-credentials) and [HashiCorp vaults](#option-4-hashicorp-vault-credentials). The property `expectedSecretType` is required in all cases and must have the value `keyPair`. +This object type is identical to `credentialsSupplier` but requires the secrets to be passwords. The available options are [passwords](#option-1-username-and-password), [CyberArk vaults](#option-3-cyberark-vault-credentials), [HashiCorp vaults](#option-4-hashicorp-vault-credentials) and [Azure vaults](#option-5-azure-vault-credentials). The property `expectedSecretType` is required in all cases and must have the value `keyPair`. ## JSON Schema Limitations diff --git a/docs/docs/References/Version_Compatibility.md b/docs/docs/References/Version_Compatibility.md index 0af640b6..e5365419 100644 --- a/docs/docs/References/Version_Compatibility.md +++ b/docs/docs/References/Version_Compatibility.md @@ -4,16 +4,16 @@ | vSDK Version | Earliest Supported DE Version | Latest Supported DE Version | |------------------------------------------|:-----------------------------:|:-----------------------------------------------------------------:| -| [5.1.0](../Release_Notes/5.1.0/5.1.0.md) | 2026.4.0.0 | [Latest Release](https://cd.delphix.com/docs/latest/new-features) | -| [5.0.1](../Release_Notes/5.0.1/5.0.1.md) | 2025.2.0.0 | [Latest Release](https://cd.delphix.com/docs/latest/new-features) | -| [5.0.0](../Release_Notes/5.0.0/5.0.0.md) | 29.0.0.0 | [Latest Release](https://cd.delphix.com/docs/latest/new-features) | -| [4.1.0](../Release_Notes/4.1.0/4.1.0.md) | 12.0.0.0 | [Latest Release](https://cd.delphix.com/docs/latest/new-features) | -| [4.0.5](../Release_Notes/4.0.5/4.0.5.md) | 6.0.16.0 | [Latest Release](https://cd.delphix.com/docs/latest/new-features) | -| [4.0.2](../Release_Notes/4.0.2/4.0.2.md) | 6.0.12.0 | [Latest Release](https://cd.delphix.com/docs/latest/new-features) | -| [3.1.0](../Release_Notes/3.1.0/3.1.0.md) | 6.0.7.0 | [Latest Release](https://cd.delphix.com/docs/latest/new-features) | -| [3.0.0](../Release_Notes/3.0.0/3.0.0.md) | 6.0.6.0 | [Latest Release](https://cd.delphix.com/docs/latest/new-features) | -| [2.1.0](../Release_Notes/2.1.0/2.1.0.md) | 6.0.3.0 | [Latest Release](https://cd.delphix.com/docs/latest/new-features) | -| [2.0.0](../Release_Notes/2.0.0/2.0.0.md) | 6.0.2.0 | [Latest Release](https://cd.delphix.com/docs/latest/new-features) | +| [5.1.0](../Release_Notes/5.1.0/5.1.0.md) | 2026.4.0.0 | [Latest Release](https://help.delphix.com/cd/current/content/release_notes.htm) | +| [5.0.1](../Release_Notes/5.0.1/5.0.1.md) | 2025.2.0.0 | [Latest Release](https://help.delphix.com/cd/current/content/release_notes.htm) | +| [5.0.0](../Release_Notes/5.0.0/5.0.0.md) | 29.0.0.0 | [Latest Release](https://help.delphix.com/cd/current/content/release_notes.htm) | +| [4.1.0](../Release_Notes/4.1.0/4.1.0.md) | 12.0.0.0 | [Latest Release](https://help.delphix.com/cd/current/content/release_notes.htm) | +| [4.0.5](../Release_Notes/4.0.5/4.0.5.md) | 6.0.16.0 | [Latest Release](https://help.delphix.com/cd/current/content/release_notes.htm) | +| [4.0.2](../Release_Notes/4.0.2/4.0.2.md) | 6.0.12.0 | [Latest Release](https://help.delphix.com/cd/current/content/release_notes.htm) | +| [3.1.0](../Release_Notes/3.1.0/3.1.0.md) | 6.0.7.0 | [Latest Release](https://help.delphix.com/cd/current/content/release_notes.htm) | +| [3.0.0](../Release_Notes/3.0.0/3.0.0.md) | 6.0.6.0 | [Latest Release](https://help.delphix.com/cd/current/content/release_notes.htm) | +| [2.1.0](../Release_Notes/2.1.0/2.1.0.md) | 6.0.3.0 | [Latest Release](https://help.delphix.com/cd/current/content/release_notes.htm) | +| [2.0.0](../Release_Notes/2.0.0/2.0.0.md) | 6.0.2.0 | [Latest Release](https://help.delphix.com/cd/current/content/release_notes.htm) | | [1.0.0](../Release_Notes/1.0.0/1.0.0.md) | 6.0.2.0 | 14.0.0.0 | | [0.4.0](../Release_Notes/0.4.0/0.4.0.md) | 5.3.5.0 | 6.0.1.0 | diff --git a/docs/docs/Versioning_And_Upgrade/Replication.md b/docs/docs/Versioning_And_Upgrade/Replication.md index af5f5433..9471f81c 100644 --- a/docs/docs/Versioning_And_Upgrade/Replication.md +++ b/docs/docs/Versioning_And_Upgrade/Replication.md @@ -1,7 +1,7 @@ # Replication A Delphix Engine (source) can be setup to replicate data objects to another Delphix Engine (target). Plugins built using the Virtualization SDK work seamlessly with Delphix Engine replication with no additional development required from plugin developers. -Only a single version of a plugin can be active on a Delphix Engine at a time. We discuss some basic scenarios below. For more detailed information refer to the [Delphix Engine Documentation](https://cd.delphix.com/docs/latest/). +Only a single version of a plugin can be active on a Delphix Engine at a time. We discuss some basic scenarios below. For more detailed information refer to the [Delphix Engine Documentation](https://help.delphix.com/cd/current/content/home.htm). ## Replica Provisioning Replicated dSource or VDB snapshots can be used to provision new VDBs onto a target Delphix Engine, without failing over any of the objects. When provisioning a VDB from a replicated snapshot: @@ -9,7 +9,7 @@ Replicated dSource or VDB snapshots can be used to provision new VDBs onto a tar * A version of the plugin has to be installed on the target Delphix Engine. * The versions of the plugins installed on the source and target Delphix Engines have to be [compatible](Compatibility.md). -Once provisioned, the VDB on the target Delphix Engine will be associated with the version of the plugin installed on the target Delphix Engine, any required data migrations will be run as part of the provisioning process. For more details refer to the [Delphix Engine Documentation](https://cd.delphix.com/docs/latest/). +Once provisioned, the VDB on the target Delphix Engine will be associated with the version of the plugin installed on the target Delphix Engine, any required data migrations will be run as part of the provisioning process. For more details refer to the [Delphix Engine Documentation](https://help.delphix.com/cd/current/content/home.htm). ## Replication Failover On failover, there are three scenarios for each plugin: @@ -18,4 +18,4 @@ On failover, there are three scenarios for each plugin: | -------- | ------- Source plugin **not installed** on target Delphix Engine | The plugin will be failed over and marked as `active` on the target Delphix Engine. Source plugin version **is equal to** the target plugin version | The plugin from the source will be merged with the plugin on the target Delphix Engine. -Source plugin version **is not equal to** the target plugin version | The plugin from the source will be marked `inactive` on the target Delphix Engine. An `inactive` plugin can be subsequently activated, after failover, if it is [compatible](Compatibility.md) with the existing `active` plugin. Activating a plugin will do an upgrade and merge the `inactive` plugin, and all its associated objects, with the `active` plugin. For more details refer to the [Delphix Engine Documentation](https://cd.delphix.com/docs/latest/). \ No newline at end of file +Source plugin version **is not equal to** the target plugin version | The plugin from the source will be marked `inactive` on the target Delphix Engine. An `inactive` plugin can be subsequently activated, after failover, if it is [compatible](Compatibility.md) with the existing `active` plugin. Activating a plugin will do an upgrade and merge the `inactive` plugin, and all its associated objects, with the `active` plugin. For more details refer to the [Delphix Engine Documentation](https://help.delphix.com/cd/current/content/home.htm). \ No newline at end of file From b34a7ecbd5fec08b525e1fee177bf04c96f7c40c Mon Sep 17 00:00:00 2001 From: Sourabh Jain <45626610+SumoSourabh@users.noreply.github.com> Date: Tue, 2 Jun 2026 14:39:42 +0530 Subject: [PATCH 19/21] DLPX-97348 depbump: bump pytest from 9.0.2 to 9.0.3 (consolidates 9 PRs) (#666) --- common/requirements.txt | 2 +- dvp/requirements.txt | 2 +- libs/requirements.txt | 2 +- platform/requirements.txt | 2 +- tools/requirements.txt | 2 +- 5 files changed, 5 insertions(+), 5 deletions(-) diff --git a/common/requirements.txt b/common/requirements.txt index 822c250f..95767ed1 100644 --- a/common/requirements.txt +++ b/common/requirements.txt @@ -2,6 +2,6 @@ bump2version==1.0.1 packaging==26.0 pluggy==1.6.0 pyparsing==3.3.2 -pytest==9.0.2 +pytest==9.0.3 six==1.17.0 zipp==3.23.0 diff --git a/dvp/requirements.txt b/dvp/requirements.txt index 822c250f..95767ed1 100644 --- a/dvp/requirements.txt +++ b/dvp/requirements.txt @@ -2,6 +2,6 @@ bump2version==1.0.1 packaging==26.0 pluggy==1.6.0 pyparsing==3.3.2 -pytest==9.0.2 +pytest==9.0.3 six==1.17.0 zipp==3.23.0 diff --git a/libs/requirements.txt b/libs/requirements.txt index 8b37cc01..3ee2d9b7 100644 --- a/libs/requirements.txt +++ b/libs/requirements.txt @@ -4,6 +4,6 @@ mock==5.2.0 packaging==26.0 pluggy==1.6.0 pyparsing==3.3.2 -pytest==9.0.2 +pytest==9.0.3 six==1.17.0 zipp==3.23.0 diff --git a/platform/requirements.txt b/platform/requirements.txt index 8b37cc01..3ee2d9b7 100644 --- a/platform/requirements.txt +++ b/platform/requirements.txt @@ -4,6 +4,6 @@ mock==5.2.0 packaging==26.0 pluggy==1.6.0 pyparsing==3.3.2 -pytest==9.0.2 +pytest==9.0.3 six==1.17.0 zipp==3.23.0 diff --git a/tools/requirements.txt b/tools/requirements.txt index 5ba42cdb..76829ac3 100644 --- a/tools/requirements.txt +++ b/tools/requirements.txt @@ -16,7 +16,7 @@ pycodestyle==2.14.0 pyflakes==3.4.0 pyparsing==3.3.2 pytest-cov==7.1.0 -pytest==9.0.2 +pytest==9.0.3 six==1.17.0 yapf==0.43.0 zipp==3.23.0 From 80ed9995b2de8195574908b9c0b799bb31a66fbd Mon Sep 17 00:00:00 2001 From: Sourabh Jain <45626610+SumoSourabh@users.noreply.github.com> Date: Wed, 3 Jun 2026 08:02:21 +0530 Subject: [PATCH 20/21] CP-13290 Migrate Python packages from setup.cfg/setup.py to pyproject.toml (#662) PR URL: https://github.com/delphix/virtualization-sdk/pull/662 --------- Co-authored-by: Claude Opus 4.7 (1M context) --- .bumpversion.cfg | 18 ++-- .github/dependabot.yml | 4 + .github/workflows/pre-commit.yml | 74 +++++++++------ .github/workflows/publish-python-packages.yml | 17 ++-- LICENSE | 2 +- README-dev.md | 8 +- bin/build_project.sh | 53 +++++++---- common/LICENSE | 2 +- common/MANIFEST.in | 3 +- common/pyproject.toml | 48 ++++++++++ common/requirements.txt | 7 -- common/setup.cfg | 20 ----- common/setup.py | 20 ----- .../python/dlpx/virtualization/common/VERSION | 1 - docs/Pipfile | 4 + docs/build.sh | 3 + dvp/LICENSE | 2 +- dvp/MANIFEST.in | 5 +- dvp/pyproject.toml | 53 +++++++++++ dvp/requirements.txt | 7 -- dvp/setup.cfg | 22 ----- dvp/setup.py | 22 ----- dvp/src/main/python/dlpx/__init__.py | 4 +- .../main/python/dlpx/virtualization/VERSION | 1 - .../python/dlpx/virtualization/__init__.py | 4 +- dvp/src/test/python/test_not_used.py | 3 +- libs/LICENSE | 2 +- libs/MANIFEST.in | 3 +- libs/pyproject.toml | 52 +++++++++++ libs/requirements.txt | 9 -- libs/setup.cfg | 21 ----- libs/setup.py | 21 ----- .../python/dlpx/virtualization/libs/VERSION | 1 - .../dlpx/virtualization/_engine/__init__.py | 3 + linkcheck-skip.txt | 3 + platform/LICENSE | 2 +- platform/MANIFEST.in | 3 +- platform/pyproject.toml | 52 +++++++++++ platform/requirements.txt | 9 -- platform/setup.cfg | 21 ----- platform/setup.py | 21 ----- .../dlpx/virtualization/platform/VERSION | 1 - .../fake_generated_definitions.py | 4 + tools/LICENSE | 2 +- tools/MANIFEST.in | 5 +- tools/README-dev.md | 4 +- tools/pyproject.toml | 89 +++++++++++++++++++ tools/requirements.txt | 22 ----- tools/setup.cfg | 26 ------ tools/setup.py | 43 --------- .../dlpx/virtualization/_internal/VERSION | 1 - .../virtualization/_internal/package_util.py | 14 +-- .../_internal/plugin_dependency_util.py | 27 +++--- .../_internal/engine_version.cfg | 2 +- .../dlpx/virtualization/_internal/test_cli.py | 9 +- .../_internal/test_plugin_dependency_util.py | 31 ++++--- 56 files changed, 492 insertions(+), 418 deletions(-) create mode 100644 common/pyproject.toml delete mode 100644 common/requirements.txt delete mode 100644 common/setup.cfg delete mode 100644 common/setup.py delete mode 100644 common/src/main/python/dlpx/virtualization/common/VERSION create mode 100644 dvp/pyproject.toml delete mode 100644 dvp/requirements.txt delete mode 100644 dvp/setup.cfg delete mode 100644 dvp/setup.py delete mode 100644 dvp/src/main/python/dlpx/virtualization/VERSION create mode 100644 libs/pyproject.toml delete mode 100644 libs/requirements.txt delete mode 100644 libs/setup.cfg delete mode 100644 libs/setup.py delete mode 100644 libs/src/main/python/dlpx/virtualization/libs/VERSION create mode 100644 platform/pyproject.toml delete mode 100644 platform/requirements.txt delete mode 100644 platform/setup.cfg delete mode 100644 platform/setup.py delete mode 100644 platform/src/main/python/dlpx/virtualization/platform/VERSION create mode 100644 tools/pyproject.toml delete mode 100644 tools/requirements.txt delete mode 100644 tools/setup.cfg delete mode 100644 tools/setup.py delete mode 100644 tools/src/main/python/dlpx/virtualization/_internal/VERSION diff --git a/.bumpversion.cfg b/.bumpversion.cfg index 2f9a3bc7..c8ea28be 100644 --- a/.bumpversion.cfg +++ b/.bumpversion.cfg @@ -1,5 +1,9 @@ +# +# Copyright (c) 2026 by Delphix. All rights reserved. +# + [bumpversion] -current_version = 5.0.1 +current_version = 5.1.0 commit = False tag = False parse = (?P\d+)\.(?P\d+)\.(?P\d+)(\.(?P[a-z]+)(?P\d+))? @@ -17,15 +21,17 @@ values = [bumpversion:part:dev] -[bumpversion:file:./dvp/src/main/python/dlpx/virtualization/VERSION] +# Each pyproject.toml's [project].version and sibling-package pins are matched +# by the default {current_version} search; both get updated atomically. +[bumpversion:file:./common/pyproject.toml] -[bumpversion:file:./common/src/main/python/dlpx/virtualization/common/VERSION] +[bumpversion:file:./dvp/pyproject.toml] -[bumpversion:file:./platform/src/main/python/dlpx/virtualization/platform/VERSION] +[bumpversion:file:./libs/pyproject.toml] -[bumpversion:file:./libs/src/main/python/dlpx/virtualization/libs/VERSION] +[bumpversion:file:./platform/pyproject.toml] -[bumpversion:file:./tools/src/main/python/dlpx/virtualization/_internal/VERSION] +[bumpversion:file:./tools/pyproject.toml] [bumpversion:file:./tools/src/test/python/dlpx/virtualization/_internal/test_package_util.py] search = DVP_VERSION = '{current_version}' diff --git a/.github/dependabot.yml b/.github/dependabot.yml index 373d4f42..22c07c21 100644 --- a/.github/dependabot.yml +++ b/.github/dependabot.yml @@ -1,3 +1,7 @@ +# +# Copyright (c) 2026 by Delphix. All rights reserved. +# + version: 2 updates: - package-ecosystem: pip diff --git a/.github/workflows/pre-commit.yml b/.github/workflows/pre-commit.yml index 3ac1ca99..287175b3 100644 --- a/.github/workflows/pre-commit.yml +++ b/.github/workflows/pre-commit.yml @@ -1,5 +1,5 @@ # -# Copyright (c) 2020, 2022 by Delphix. All rights reserved. +# Copyright (c) 2020, 2026 by Delphix. All rights reserved. # name: "Pre-commit actions for Delphix Virtualization SDK" @@ -14,53 +14,69 @@ on: jobs: pytest311: - name: Test ${{ matrix.package }} on ${{ matrix.os }} using pytest (Python 3.11) + name: Test SDK on ${{ matrix.os }} (Python ${{ matrix.python-version }}) runs-on: ${{ matrix.os }} strategy: - max-parallel: 4 + max-parallel: 3 matrix: python-version: [ 3.11 ] os: [ ubuntu-latest, macos-latest, windows-latest ] - package: [ common, libs, platform, tools ] steps: - - name: Checkout ${{ matrix.package }} project - uses: actions/checkout@v3 + - uses: actions/checkout@v3 - name: Set up Python ${{ matrix.python-version }} uses: actions/setup-python@v4 with: python-version: ${{ matrix.python-version }} - - name: Install ${{ matrix.package }} dependencies - working-directory: ${{ matrix.package }} + # Install all SDK packages once, in dependency order. Sibling packages + # need to be present in the env before subsequent installs can resolve + # their PEP 621 pins (e.g. dvp-libs needs dvp-common already installed). + - name: Install SDK packages run: | python -m pip install --upgrade pip - pip install -r requirements.txt --find-links https://test.pypi.org/simple/dvp-api/ + pip install -e "./common[dev]" --find-links https://test.pypi.org/simple/dvp-api/ + pip install -e "./libs[dev]" --find-links https://test.pypi.org/simple/dvp-api/ + pip install -e "./platform[dev]" --find-links https://test.pypi.org/simple/dvp-api/ + pip install -e "./tools[dev]" --find-links https://test.pypi.org/simple/dvp-api/ - - name: Install ${{ matrix.package }} project - working-directory: ${{ matrix.package }} - run: | - pip install . --find-links https://test.pypi.org/simple/dvp-api/ + - name: Test common + run: python -m pytest ./common/src/test/python - # Run all the test cases part of the package. - - name: Test ${{ matrix.package }} project with pytest - working-directory: ${{ matrix.package }} - run: | - python -m pytest src/test/python + - name: Test libs + run: python -m pytest ./libs/src/test/python + + - name: Test platform + run: python -m pytest ./platform/src/test/python + + - name: Test tools + run: python -m pytest ./tools/src/test/python - # Install flake8 and run linting on src and test for linting if OS is ubuntu. - name: Install flake8 - if: ${{ matrix.os == 'ubuntu-latest' }} + if: matrix.os == 'ubuntu-latest' + run: pip install flake8 + + - name: Lint common + if: matrix.os == 'ubuntu-latest' run: | - pip install flake8 + python -m flake8 ./common/src/main/python --max-line-length 88 + python -m flake8 ./common/src/test/python --max-line-length 88 - - name: Run flake8 on src directory - if: ${{ matrix.os == 'ubuntu-latest' }} - working-directory: ${{ matrix.package }} - run: python -m flake8 src/main/python --max-line-length 88 + - name: Lint libs + if: matrix.os == 'ubuntu-latest' + run: | + python -m flake8 ./libs/src/main/python --max-line-length 88 + python -m flake8 ./libs/src/test/python --max-line-length 88 + + - name: Lint platform + if: matrix.os == 'ubuntu-latest' + run: | + python -m flake8 ./platform/src/main/python --max-line-length 88 + python -m flake8 ./platform/src/test/python --max-line-length 88 - - name: Run flake8 on test directory - if: ${{ matrix.os == 'ubuntu-latest' }} - working-directory: ${{ matrix.package }} - run: python -m flake8 src/test/python --max-line-length 88 + - name: Lint tools + if: matrix.os == 'ubuntu-latest' + run: | + python -m flake8 ./tools/src/main/python --max-line-length 88 + python -m flake8 ./tools/src/test/python --max-line-length 88 \ No newline at end of file diff --git a/.github/workflows/publish-python-packages.yml b/.github/workflows/publish-python-packages.yml index 2be43842..0c472f57 100644 --- a/.github/workflows/publish-python-packages.yml +++ b/.github/workflows/publish-python-packages.yml @@ -1,14 +1,19 @@ +# +# Copyright (c) 2026 by Delphix. All rights reserved. +# + name: Publish Python packages to Test PyPi on: - # Run on push when the version file has changed on selected branches. + # Run on push when a package version (declared in pyproject.toml) has + # changed on selected branches, or when this workflow itself changes. push: branches: - master - develop - release paths: - - 'dvp/src/main/python/dlpx/virtualization/VERSION' + - '**/pyproject.toml' - '.github/workflows/publish-python-packages.yml' jobs: @@ -21,15 +26,15 @@ jobs: package: [common, dvp, libs, platform, tools] steps: - - uses: actions/checkout@v2 + - uses: actions/checkout@v3 - name: Set up Python ${{ matrix.python-version }} - uses: actions/setup-python@v2 + uses: actions/setup-python@v4 with: python-version: ${{ matrix.python-version }} # Install dependencies necessary for building and publishing the package. - name: Install dependencies run: | - pip install setuptools wheel twine + pip install build twine # Build each Python package and publish it to Test PyPi. - name: Build and publish ${{ matrix.package }} package working-directory: ${{ matrix.package }} @@ -38,5 +43,5 @@ jobs: TWINE_PASSWORD: ${{ secrets.VSDK_PYPI_TOKEN }} TWINE_REPOSITORY_URL: https://test.pypi.org/legacy/ run: | - python setup.py sdist bdist_wheel + python -m build twine upload dist/* diff --git a/LICENSE b/LICENSE index d6456956..3d63c06a 100644 --- a/LICENSE +++ b/LICENSE @@ -187,7 +187,7 @@ same "printed page" as the copyright notice for easier identification within third-party archives. - Copyright [yyyy] [name of copyright owner] + Copyright 2026 Delphix Corp., a Perforce company. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. diff --git a/README-dev.md b/README-dev.md index 6a82a865..1581c2b8 100644 --- a/README-dev.md +++ b/README-dev.md @@ -1,4 +1,4 @@ -# Copyright (c) 2019, 2022 by Delphix. All rights reserved. +# Copyright (c) 2019, 2026 by Delphix. All rights reserved. # Delphix Virtualization SDK @@ -45,7 +45,7 @@ These steps are described in more detail below. ### Development environment Development should be done in a personal virtualenv. To setup the virtual environment: -1. `virtualenv /path/to/env/root`. This should be a Python 2.7 virtualenv. +1. `virtualenv /path/to/env/root`. This should be a Python 3.11 virtualenv. 2. `source ~/path/to/env/root/bin/activate`. ### Installing the SDK from source @@ -66,6 +66,10 @@ To install the SDK, follow these steps: 2. To build the project, run `sh bin/build_project.sh -b`. 3. For more information on the script options, use `sh bin/build_project.sh -h`. +To work on a single package in editable mode, install it together with its development +dependencies via the `dev` extra, e.g. `pip install -e ".[dev]"` from the package directory. +This replaces the old `requirements.txt`-based dev install. + ### CLI changes diff --git a/bin/build_project.sh b/bin/build_project.sh index bd938694..93980072 100644 --- a/bin/build_project.sh +++ b/bin/build_project.sh @@ -1,6 +1,6 @@ #!/bin/bash # -# Copyright (c) 2022 by Delphix. All rights reserved. +# Copyright (c) 2026 by Delphix. All rights reserved. # # This script provides functionality to build and run test cases for all python packages. The same script can be used @@ -15,13 +15,23 @@ should_test=false verbose=false coverage=false flake8=false -screenSize=$(tput cols) equalFiller="=" -greenColor=$(tput setaf 10) -orangeColor=$(tput setaf 208) -noColor=$(tput sgr0) -blackColor=$(tput setaf 0) -blueColor=$(tput setaf 31) +failed_steps=() +if [ -t 1 ]; then + screenSize=$(tput cols) + greenColor=$(tput setaf 10) + orangeColor=$(tput setaf 208) + noColor=$(tput sgr0) + blackColor=$(tput setaf 0) + blueColor=$(tput setaf 31) +else + screenSize=100 + greenColor="" + orangeColor="" + noColor="" + blackColor="" + blueColor="" +fi ############################################################ # Help # @@ -86,23 +96,23 @@ run_operations() { if [ "$should_build" = true ]; then echo print_as_per_screen_size " $module_name build starts " "${orangeColor}" ${equalFiller} "${screenSize}" - build_module + build_module || failed_steps+=("$module_name build") print_as_per_screen_size " $module_name build complete " "${greenColor}" ${equalFiller} "${screenSize}" fi if [ "$flake8" = true ]; then echo print_as_per_screen_size " $module_name Flake8 Main starts " "${orangeColor}" ${equalFiller} "${screenSize}" - python -m flake8 "$module_path/src/test/python" --max-line-length 88 + python -m flake8 "$module_path/src/test/python" --max-line-length 88 || failed_steps+=("$module_name flake8 (test)") print_as_per_screen_size " $module_name Flake8 Main complete " "${greenColor}" ${equalFiller} "${screenSize}" echo print_as_per_screen_size " $module_name Flake8 Test starts " "${orangeColor}" ${equalFiller} "${screenSize}" - python -m flake8 "$module_path/src/main/python" --max-line-length 88 + python -m flake8 "$module_path/src/main/python" --max-line-length 88 || failed_steps+=("$module_name flake8 (main)") print_as_per_screen_size " $module_name Flake8 Test complete " "${greenColor}" ${equalFiller} "${screenSize}" fi if [ "$should_test" = true ]; then echo print_as_per_screen_size " $module_name tests starts " "${orangeColor}" ${equalFiller} "${screenSize}" - test_module + test_module || failed_steps+=("$module_name tests") print_as_per_screen_size " $module_name tests complete " "${greenColor}" ${equalFiller} "${screenSize}" fi cd "$current_path" || exit @@ -118,13 +128,11 @@ get_project_path() { # Build the module build_module() { - python setup.py clean --all + rm -rf build/ ./*.egg-info src/main/python/*.egg-info if [ "$verbose" = true ]; then - pip install -r requirements.txt -v - pip install -e . -v + pip install -e ".[dev]" -v else - pip install -r requirements.txt -q - pip install -e . -q + pip install -e ".[dev]" -q fi } @@ -209,7 +217,16 @@ if [ "$should_build" = true ] || [ "$should_test" = true ] || [ "$flake8" = true if [ "$coverage" = true ]; then echo "Paths to combine for coverage are [${paths[*]}]." - coverage combine ${paths[@]} - coverage report -m -i + coverage combine ${paths[@]} || failed_steps+=("coverage combine") + coverage report -m || failed_steps+=("coverage report") + fi + + if [ ${#failed_steps[@]} -gt 0 ]; then + echo + echo "${#failed_steps[@]} step(s) FAILED:" + for step in "${failed_steps[@]}"; do + echo " - $step" + done + exit 1 fi fi diff --git a/common/LICENSE b/common/LICENSE index d6456956..3d63c06a 100644 --- a/common/LICENSE +++ b/common/LICENSE @@ -187,7 +187,7 @@ same "printed page" as the copyright notice for easier identification within third-party archives. - Copyright [yyyy] [name of copyright owner] + Copyright 2026 Delphix Corp., a Perforce company. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. diff --git a/common/MANIFEST.in b/common/MANIFEST.in index c39fba12..d0dda99f 100644 --- a/common/MANIFEST.in +++ b/common/MANIFEST.in @@ -1,6 +1,5 @@ # -# Copyright (c) 2019 by Delphix. All rights reserved. +# Copyright (c) 2019, 2026 by Delphix. All rights reserved. # include LICENSE -include src/main/python/dlpx/virtualization/common/VERSION diff --git a/common/pyproject.toml b/common/pyproject.toml new file mode 100644 index 00000000..8519e61b --- /dev/null +++ b/common/pyproject.toml @@ -0,0 +1,48 @@ +# +# Copyright (c) 2026 by Delphix. All rights reserved. +# + +[build-system] +requires = ["setuptools>=77"] +build-backend = "setuptools.build_meta" + +[project] +name = "dvp-common" +version = "5.1.0" +readme = "README.md" +requires-python = ">=3.11, <3.12" +authors = [ + {name = "Delphix", email = "virtualization-plugins@delphix.com"}, +] +license = "Apache-2.0" +license-files = ["LICENSE"] +classifiers = [ + "Development Status :: 5 - Production/Stable", + "Programming Language :: Python", + "Programming Language :: Python :: 3.11", + "Operating System :: OS Independent", +] +dependencies = [ + "dvp-api == 1.10.0.dev3", + "six >= 1.17, < 1.18", +] + +[project.optional-dependencies] +dev = [ + "bump2version == 1.0.1", + "packaging == 26.0", + "pluggy == 1.6.0", + "pyparsing == 3.3.2", + "pytest == 9.0.3", + "six == 1.17.0", + "zipp == 3.23.0", +] + +[project.urls] +Homepage = "https://developer.delphix.com" + +[tool.setuptools] +package-dir = {"" = "src/main/python"} + +[tool.setuptools.packages.find] +where = ["src/main/python"] \ No newline at end of file diff --git a/common/requirements.txt b/common/requirements.txt deleted file mode 100644 index 95767ed1..00000000 --- a/common/requirements.txt +++ /dev/null @@ -1,7 +0,0 @@ -bump2version==1.0.1 -packaging==26.0 -pluggy==1.6.0 -pyparsing==3.3.2 -pytest==9.0.3 -six==1.17.0 -zipp==3.23.0 diff --git a/common/setup.cfg b/common/setup.cfg deleted file mode 100644 index c282c94f..00000000 --- a/common/setup.cfg +++ /dev/null @@ -1,20 +0,0 @@ -# -# Copyright (c) 2019, 2022 by Delphix. All rights reserved. -# - -[metadata] -metadata_version: 1.2 -author: Delphix -author_email: virtualization-plugins@delphix.com -home_page: https://developer.delphix.com -long_description: file: README.md -long_description_content_type: text/markdown -classifiers: - Development Status :: 5 - Production/Stable - Programming Language :: Python - Programming Language :: Python :: 3.11 - License :: OSI Approved :: Apache Software License - Operating System :: OS Independent - -[options] -requires_python: >=3.11, <3.12 diff --git a/common/setup.py b/common/setup.py deleted file mode 100644 index 7ecfe15a..00000000 --- a/common/setup.py +++ /dev/null @@ -1,20 +0,0 @@ -import os -import setuptools - -PYTHON_SRC = 'src/main/python' - -install_requires = [ - "dvp-api == 1.10.0.dev0", - "six >= 1.17, < 1.18", -] - -with open(os.path.join(PYTHON_SRC, 'dlpx/virtualization/common/VERSION')) as version_file: - version = version_file.read().strip() - -setuptools.setup(name='dvp-common', - version=version, - install_requires=install_requires, - package_dir={'': PYTHON_SRC}, - packages=setuptools.find_packages(PYTHON_SRC), - python_requires='>=3.11, <3.12', - ) diff --git a/common/src/main/python/dlpx/virtualization/common/VERSION b/common/src/main/python/dlpx/virtualization/common/VERSION deleted file mode 100644 index 831446cb..00000000 --- a/common/src/main/python/dlpx/virtualization/common/VERSION +++ /dev/null @@ -1 +0,0 @@ -5.1.0 diff --git a/docs/Pipfile b/docs/Pipfile index d6b58fd2..7437f4f4 100644 --- a/docs/Pipfile +++ b/docs/Pipfile @@ -1,3 +1,7 @@ +# +# Copyright (c) 2026 by Delphix. All rights reserved. +# + [[source]] url = "https://pypi.org/simple" verify_ssl = true diff --git a/docs/build.sh b/docs/build.sh index f256f923..27023683 100755 --- a/docs/build.sh +++ b/docs/build.sh @@ -1,4 +1,7 @@ #!/bin/bash +# +# Copyright (c) 2026 by Delphix. All rights reserved. +# git fetch pipenv run mkdocs build --clean diff --git a/dvp/LICENSE b/dvp/LICENSE index d6456956..3d63c06a 100644 --- a/dvp/LICENSE +++ b/dvp/LICENSE @@ -187,7 +187,7 @@ same "printed page" as the copyright notice for easier identification within third-party archives. - Copyright [yyyy] [name of copyright owner] + Copyright 2026 Delphix Corp., a Perforce company. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. diff --git a/dvp/MANIFEST.in b/dvp/MANIFEST.in index 0479bae9..40806cc6 100644 --- a/dvp/MANIFEST.in +++ b/dvp/MANIFEST.in @@ -1,6 +1,5 @@ # -# Copyright (c) 2019 by Delphix. All rights reserved. +# Copyright (c) 2019, 2026 by Delphix. All rights reserved. # -include LICENSE -include src/main/python/dlpx/virtualization/VERSION \ No newline at end of file +include LICENSE \ No newline at end of file diff --git a/dvp/pyproject.toml b/dvp/pyproject.toml new file mode 100644 index 00000000..df862ab7 --- /dev/null +++ b/dvp/pyproject.toml @@ -0,0 +1,53 @@ +# +# Copyright (c) 2026 by Delphix. All rights reserved. +# + +[build-system] +requires = ["setuptools>=77"] +build-backend = "setuptools.build_meta" + +[project] +name = "dvp" +version = "5.1.0" +description = "Delphix Virtualization Platform SDK" +readme = "README.md" +requires-python = ">=3.11, <3.12" +authors = [ + {name = "Delphix", email = "virtualization-plugins@delphix.com"}, +] +license = "Apache-2.0" +license-files = ["LICENSE"] +keywords = ["virtualization", "plugin"] +classifiers = [ + "Development Status :: 5 - Production/Stable", + "Programming Language :: Python", + "Programming Language :: Python :: 3.11", + "Operating System :: OS Independent", +] +# Sibling-package pins; .bumpversion.cfg keeps these in sync with [project].version. +dependencies = [ + "dvp-common == 5.1.0", + "dvp-libs == 5.1.0", + "dvp-platform == 5.1.0", + "dvp-tools == 5.1.0", +] + +[project.optional-dependencies] +dev = [ + "bump2version == 1.0.1", + "packaging == 26.0", + "pluggy == 1.6.0", + "pyparsing == 3.3.2", + "pytest == 9.0.3", + "six == 1.17.0", + "zipp == 3.23.0", +] + +[project.urls] +Homepage = "https://developer.delphix.com" + +[tool.setuptools] +package-dir = {"" = "src/main/python"} + +[tool.setuptools.packages.find] +where = ["src/main/python"] \ No newline at end of file diff --git a/dvp/requirements.txt b/dvp/requirements.txt deleted file mode 100644 index 95767ed1..00000000 --- a/dvp/requirements.txt +++ /dev/null @@ -1,7 +0,0 @@ -bump2version==1.0.1 -packaging==26.0 -pluggy==1.6.0 -pyparsing==3.3.2 -pytest==9.0.3 -six==1.17.0 -zipp==3.23.0 diff --git a/dvp/setup.cfg b/dvp/setup.cfg deleted file mode 100644 index 3952a0c5..00000000 --- a/dvp/setup.cfg +++ /dev/null @@ -1,22 +0,0 @@ -# -# Copyright (c) 2019, 2021 by Delphix. All rights reserved. -# - -[metadata] -metadata_version: 1.2 -author: Delphix -author_email: virtualization-plugins@delphix.com -home_page: https://developer.delphix.com -summary: Delphix Virtualization Platform SDK -long_description: file: README.md -long_description_content_type: text/markdown -keywords: virtualization plugin -classifiers: - Development Status :: 5 - Production/Stable - Programming Language :: Python - Programming Language :: Python :: 3.11 - License :: OSI Approved :: Apache Software License - Operating System :: OS Independent - -[options] -requires_python: >=3.11, <3.12 diff --git a/dvp/setup.py b/dvp/setup.py deleted file mode 100644 index bf0858f5..00000000 --- a/dvp/setup.py +++ /dev/null @@ -1,22 +0,0 @@ -import os -import setuptools - -PYTHON_SRC = 'src/main/python' - -with open(os.path.join(PYTHON_SRC, 'dlpx/virtualization/VERSION')) as version_file: - version = version_file.read().strip() - -install_requires = [ - "dvp-common == {}".format(version), - "dvp-libs == {}".format(version), - "dvp-platform == {}".format(version), - "dvp-tools == {}".format(version) -] - -setuptools.setup(name='dvp', - version=version, - install_requires=install_requires, - package_dir={'': PYTHON_SRC}, - packages=setuptools.find_packages(PYTHON_SRC), - python_requires='>=3.11, <3.12', - ) diff --git a/dvp/src/main/python/dlpx/__init__.py b/dvp/src/main/python/dlpx/__init__.py index c7fd3fc1..2d4d83c5 100644 --- a/dvp/src/main/python/dlpx/__init__.py +++ b/dvp/src/main/python/dlpx/__init__.py @@ -1,5 +1,5 @@ # -# Copyright (c) 2019 by Delphix. All rights reserved. +# Copyright (c) 2019, 2026 by Delphix. All rights reserved. # -__path__ = __import__('pkgutil').extend_path(__path__, __name__) \ No newline at end of file +__path__ = __import__('pkgutil').extend_path(__path__, __name__) diff --git a/dvp/src/main/python/dlpx/virtualization/VERSION b/dvp/src/main/python/dlpx/virtualization/VERSION deleted file mode 100644 index 831446cb..00000000 --- a/dvp/src/main/python/dlpx/virtualization/VERSION +++ /dev/null @@ -1 +0,0 @@ -5.1.0 diff --git a/dvp/src/main/python/dlpx/virtualization/__init__.py b/dvp/src/main/python/dlpx/virtualization/__init__.py index c7fd3fc1..2d4d83c5 100644 --- a/dvp/src/main/python/dlpx/virtualization/__init__.py +++ b/dvp/src/main/python/dlpx/virtualization/__init__.py @@ -1,5 +1,5 @@ # -# Copyright (c) 2019 by Delphix. All rights reserved. +# Copyright (c) 2019, 2026 by Delphix. All rights reserved. # -__path__ = __import__('pkgutil').extend_path(__path__, __name__) \ No newline at end of file +__path__ = __import__('pkgutil').extend_path(__path__, __name__) diff --git a/dvp/src/test/python/test_not_used.py b/dvp/src/test/python/test_not_used.py index 1b369533..62c77b15 100644 --- a/dvp/src/test/python/test_not_used.py +++ b/dvp/src/test/python/test_not_used.py @@ -1,5 +1,5 @@ # -# Copyright (c) 2019 by Delphix. All rights reserved. +# Copyright (c) 2019, 2026 by Delphix. All rights reserved. # @@ -8,4 +8,3 @@ def test_not_used(): The build will fail if there are no tests. This is an empty package needed to tie together the other dvp packages so there's nothing to test. """ - diff --git a/libs/LICENSE b/libs/LICENSE index d6456956..3d63c06a 100644 --- a/libs/LICENSE +++ b/libs/LICENSE @@ -187,7 +187,7 @@ same "printed page" as the copyright notice for easier identification within third-party archives. - Copyright [yyyy] [name of copyright owner] + Copyright 2026 Delphix Corp., a Perforce company. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. diff --git a/libs/MANIFEST.in b/libs/MANIFEST.in index e99a4c2d..d0dda99f 100644 --- a/libs/MANIFEST.in +++ b/libs/MANIFEST.in @@ -1,6 +1,5 @@ # -# Copyright (c) 2019 by Delphix. All rights reserved. +# Copyright (c) 2019, 2026 by Delphix. All rights reserved. # include LICENSE -include src/main/python/dlpx/virtualization/libs/VERSION diff --git a/libs/pyproject.toml b/libs/pyproject.toml new file mode 100644 index 00000000..85905b3d --- /dev/null +++ b/libs/pyproject.toml @@ -0,0 +1,52 @@ +# +# Copyright (c) 2026 by Delphix. All rights reserved. +# + +[build-system] +requires = ["setuptools>=77"] +build-backend = "setuptools.build_meta" + +[project] +name = "dvp-libs" +version = "5.1.0" +description = "Delphix Virtualization Platform Libraries" +readme = "README.md" +requires-python = ">=3.11, <3.12" +authors = [ + {name = "Delphix", email = "virtualization-plugins@delphix.com"}, +] +license = "Apache-2.0" +license-files = ["LICENSE"] +classifiers = [ + "Development Status :: 5 - Production/Stable", + "Programming Language :: Python", + "Programming Language :: Python :: 3.11", + "Operating System :: OS Independent", +] +# dvp-common pin is kept in sync with [project].version via .bumpversion.cfg. +dependencies = [ + "dvp-api == 1.10.0.dev3", + "dvp-common == 5.1.0", + "six >= 1.17, < 1.18", +] + +[project.optional-dependencies] +dev = [ + "bump2version == 1.0.1", + "mock == 5.2.0", + "packaging == 26.0", + "pluggy == 1.6.0", + "pyparsing == 3.3.2", + "pytest == 9.0.3", + "six == 1.17.0", + "zipp == 3.23.0", +] + +[project.urls] +Homepage = "https://developer.delphix.com" + +[tool.setuptools] +package-dir = {"" = "src/main/python"} + +[tool.setuptools.packages.find] +where = ["src/main/python"] \ No newline at end of file diff --git a/libs/requirements.txt b/libs/requirements.txt deleted file mode 100644 index 3ee2d9b7..00000000 --- a/libs/requirements.txt +++ /dev/null @@ -1,9 +0,0 @@ -./../common -bump2version==1.0.1 -mock==5.2.0 -packaging==26.0 -pluggy==1.6.0 -pyparsing==3.3.2 -pytest==9.0.3 -six==1.17.0 -zipp==3.23.0 diff --git a/libs/setup.cfg b/libs/setup.cfg deleted file mode 100644 index e934522d..00000000 --- a/libs/setup.cfg +++ /dev/null @@ -1,21 +0,0 @@ -# -# Copyright (c) 2019, 2022 by Delphix. All rights reserved. -# - -[metadata] -metadata_version: 1.2 -author: Delphix -author_email: virtualization-plugins@delphix.com -home_page: https://developer.delphix.com -summary: Delphix Virtualization Platform Libraries -long_description: file: README.md -long_description_content_type: text/markdown -classifiers: - Development Status :: 5 - Production/Stable - Programming Language :: Python - Programming Language :: Python :: 3.11 - License :: OSI Approved :: Apache Software License - Operating System :: OS Independent - -[options] -requires_python: >=3.11, <3.12 diff --git a/libs/setup.py b/libs/setup.py deleted file mode 100644 index 4f69b703..00000000 --- a/libs/setup.py +++ /dev/null @@ -1,21 +0,0 @@ -import os -import setuptools - -PYTHON_SRC = 'src/main/python' - -with open(os.path.join(PYTHON_SRC, 'dlpx/virtualization/libs/VERSION')) as version_file: - version = version_file.read().strip() - -install_requires = [ - "dvp-api == 1.10.0.dev0", - "dvp-common == {}".format(version), - "six >= 1.17, < 1.18", -] - -setuptools.setup(name='dvp-libs', - version=version, - install_requires=install_requires, - package_dir={'': PYTHON_SRC}, - packages=setuptools.find_packages(PYTHON_SRC), - python_requires='>=3.11, <3.12', - ) diff --git a/libs/src/main/python/dlpx/virtualization/libs/VERSION b/libs/src/main/python/dlpx/virtualization/libs/VERSION deleted file mode 100644 index 831446cb..00000000 --- a/libs/src/main/python/dlpx/virtualization/libs/VERSION +++ /dev/null @@ -1 +0,0 @@ -5.1.0 diff --git a/libs/src/test/python/dlpx/virtualization/_engine/__init__.py b/libs/src/test/python/dlpx/virtualization/_engine/__init__.py index e69de29b..f256b614 100644 --- a/libs/src/test/python/dlpx/virtualization/_engine/__init__.py +++ b/libs/src/test/python/dlpx/virtualization/_engine/__init__.py @@ -0,0 +1,3 @@ +# +# Copyright (c) 2026 by Delphix. All rights reserved. +# diff --git a/linkcheck-skip.txt b/linkcheck-skip.txt index 58bb60b0..5f5e382a 100644 --- a/linkcheck-skip.txt +++ b/linkcheck-skip.txt @@ -1,3 +1,6 @@ +# +# Copyright (c) 2026 by Delphix. All rights reserved. +# # URLs to skip when running linkcheck (https://github.com/filiph/linkcheck) # Skip certain external links fonts.gstatic.com diff --git a/platform/LICENSE b/platform/LICENSE index d6456956..3d63c06a 100644 --- a/platform/LICENSE +++ b/platform/LICENSE @@ -187,7 +187,7 @@ same "printed page" as the copyright notice for easier identification within third-party archives. - Copyright [yyyy] [name of copyright owner] + Copyright 2026 Delphix Corp., a Perforce company. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. diff --git a/platform/MANIFEST.in b/platform/MANIFEST.in index 5cbde639..d0dda99f 100644 --- a/platform/MANIFEST.in +++ b/platform/MANIFEST.in @@ -1,6 +1,5 @@ # -# Copyright (c) 2019 by Delphix. All rights reserved. +# Copyright (c) 2019, 2026 by Delphix. All rights reserved. # include LICENSE -include src/main/python/dlpx/virtualization/platform/VERSION diff --git a/platform/pyproject.toml b/platform/pyproject.toml new file mode 100644 index 00000000..e9f4c5c4 --- /dev/null +++ b/platform/pyproject.toml @@ -0,0 +1,52 @@ +# +# Copyright (c) 2026 by Delphix. All rights reserved. +# + +[build-system] +requires = ["setuptools>=77"] +build-backend = "setuptools.build_meta" + +[project] +name = "dvp-platform" +version = "5.1.0" +description = "Delphix Virtualization Platform APIs" +readme = "README.md" +requires-python = ">=3.11, <3.12" +authors = [ + {name = "Delphix", email = "virtualization-plugins@delphix.com"}, +] +license = "Apache-2.0" +license-files = ["LICENSE"] +classifiers = [ + "Development Status :: 5 - Production/Stable", + "Programming Language :: Python", + "Programming Language :: Python :: 3.11", + "Operating System :: OS Independent", +] +# dvp-common pin is kept in sync with [project].version via .bumpversion.cfg. +dependencies = [ + "dvp-api == 1.10.0.dev3", + "dvp-common == 5.1.0", + "six >= 1.17, < 1.18", +] + +[project.optional-dependencies] +dev = [ + "bump2version == 1.0.1", + "mock == 5.2.0", + "packaging == 26.0", + "pluggy == 1.6.0", + "pyparsing == 3.3.2", + "pytest == 9.0.3", + "six == 1.17.0", + "zipp == 3.23.0", +] + +[project.urls] +Homepage = "https://developer.delphix.com" + +[tool.setuptools] +package-dir = {"" = "src/main/python"} + +[tool.setuptools.packages.find] +where = ["src/main/python"] \ No newline at end of file diff --git a/platform/requirements.txt b/platform/requirements.txt deleted file mode 100644 index 3ee2d9b7..00000000 --- a/platform/requirements.txt +++ /dev/null @@ -1,9 +0,0 @@ -./../common -bump2version==1.0.1 -mock==5.2.0 -packaging==26.0 -pluggy==1.6.0 -pyparsing==3.3.2 -pytest==9.0.3 -six==1.17.0 -zipp==3.23.0 diff --git a/platform/setup.cfg b/platform/setup.cfg deleted file mode 100644 index 7194761a..00000000 --- a/platform/setup.cfg +++ /dev/null @@ -1,21 +0,0 @@ -# -# Copyright (c) 2019, 2022 by Delphix. All rights reserved. -# - -[metadata] -metadata_version: 1.2 -author: Delphix -author_email: virtualization-plugins@delphix.com -home_page: https://developer.delphix.com -summary: Delphix Virtualization Platform APIs -long_description: file: README.md -long_description_content_type: text/markdown -classifiers: - Development Status :: 5 - Production/Stable - Programming Language :: Python - Programming Language :: Python :: 3.11 - License :: OSI Approved :: Apache Software License - Operating System :: OS Independent - -[options] -requires_python: >=3.11, <3.12 diff --git a/platform/setup.py b/platform/setup.py deleted file mode 100644 index 3377837b..00000000 --- a/platform/setup.py +++ /dev/null @@ -1,21 +0,0 @@ -import os -import setuptools - -PYTHON_SRC = 'src/main/python' - -with open(os.path.join(PYTHON_SRC, 'dlpx/virtualization/platform/VERSION')) as version_file: - version = version_file.read().strip() - -install_requires = [ - "dvp-api == 1.10.0.dev0", - "dvp-common == {}".format(version), - "six >= 1.17, < 1.18", -] - -setuptools.setup(name='dvp-platform', - version=version, - install_requires=install_requires, - package_dir={'': PYTHON_SRC}, - packages=setuptools.find_packages(PYTHON_SRC), - python_requires='>=3.11, <3.12', - ) diff --git a/platform/src/main/python/dlpx/virtualization/platform/VERSION b/platform/src/main/python/dlpx/virtualization/platform/VERSION deleted file mode 100644 index 831446cb..00000000 --- a/platform/src/main/python/dlpx/virtualization/platform/VERSION +++ /dev/null @@ -1 +0,0 @@ -5.1.0 diff --git a/platform/src/test/python/dlpx/virtualization/fake_generated_definitions.py b/platform/src/test/python/dlpx/virtualization/fake_generated_definitions.py index 9b5d087c..0c770fc8 100644 --- a/platform/src/test/python/dlpx/virtualization/fake_generated_definitions.py +++ b/platform/src/test/python/dlpx/virtualization/fake_generated_definitions.py @@ -1,3 +1,7 @@ +# +# Copyright (c) 2026 by Delphix. All rights reserved. +# + import six diff --git a/tools/LICENSE b/tools/LICENSE index d6456956..3d63c06a 100644 --- a/tools/LICENSE +++ b/tools/LICENSE @@ -187,7 +187,7 @@ same "printed page" as the copyright notice for easier identification within third-party archives. - Copyright [yyyy] [name of copyright owner] + Copyright 2026 Delphix Corp., a Perforce company. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. diff --git a/tools/MANIFEST.in b/tools/MANIFEST.in index a9138fbd..e5ec846d 100644 --- a/tools/MANIFEST.in +++ b/tools/MANIFEST.in @@ -1,9 +1,7 @@ # -# Copyright (c) 2019 by Delphix. All rights reserved. +# Copyright (c) 2019, 2026 by Delphix. All rights reserved. # -exclude README-dev.md - include LICENSE include src/main/python/dlpx/virtualization/_internal/codegen/swagger-codegen-cli-2.3.1.jar include src/main/python/dlpx/virtualization/_internal/codegen/codegen-config.json @@ -11,4 +9,3 @@ recursive-include src/main/python/dlpx/virtualization/_internal/codegen/template recursive-include src/main/python/dlpx/virtualization/_internal/commands/plugin_template * recursive-include src/main/python/dlpx/virtualization/_internal/validation_schemas * recursive-include src/main/python *.cfg -include src/main/python/dlpx/virtualization/_internal/VERSION diff --git a/tools/README-dev.md b/tools/README-dev.md index 740a1e1d..303f4b8c 100644 --- a/tools/README-dev.md +++ b/tools/README-dev.md @@ -1,3 +1,5 @@ +# Copyright (c) 2026 by Delphix. All rights reserved. + # Delphix Virtualization SDK Tools ## Purpose @@ -9,7 +11,7 @@ and upload virtualization plugins. ### Development Environment To setup the development environment, follow the instructions in [README-dev.md](https://github.com/delphix/virtualization-sdk/blob/develop/README-dev.md) -For quick iterations, install the `tools` package in editable mode (`pip install -e .`). This means that changes to the +For quick iterations, install the `tools` package and its development dependencies in editable mode (`pip install -e ".[dev]"`). This means that changes to the code will automatically be reflected in your environment. You will not need to reinstall the tools module each time a change is made. diff --git a/tools/pyproject.toml b/tools/pyproject.toml new file mode 100644 index 00000000..9a60701a --- /dev/null +++ b/tools/pyproject.toml @@ -0,0 +1,89 @@ +# +# Copyright (c) 2026 by Delphix. All rights reserved. +# + +[build-system] +requires = ["setuptools>=77"] +build-backend = "setuptools.build_meta" + +[project] +name = "dvp-tools" +version = "5.1.0" +description = "Delphix Virtualization SDK Tools" +readme = "README.md" +requires-python = ">=3.11, <3.12" +authors = [ + {name = "Delphix", email = "virtualization-plugins@delphix.com"}, +] +license = "Apache-2.0" +license-files = ["LICENSE"] +classifiers = [ + "Development Status :: 5 - Production/Stable", + "Programming Language :: Python", + "Programming Language :: Python :: 3.11", + "Operating System :: OS Independent", +] +# Sibling-package pins (dvp-libs, dvp-platform) are kept in sync via .bumpversion.cfg. +dependencies = [ + "attrs >= 25.3, < 25.4", + "certifi >= 2024, < 2025", + "click == 7.1.2", + "click-configfile == 0.2.3", + "configparser >= 7.2, < 7.3", + "dvp-libs == 5.1.0", + "dvp-platform == 5.1.0", + "flake8 >= 7.3, < 7.4", + "httpretty >= 1.0, < 1.1", + "importlib-resources >= 6.5, < 6.6", + "jinja2 >= 3.1, < 3.2", + "jsonschema >= 4.25, < 4.26", + "MarkupSafe >= 3.0, < 3.1", + "pkgutil_resolve_name == 1.3.10", + "pyyaml >= 6, < 7", + "requests >= 2.32, < 2.33", + "six >= 1.17, < 1.18", + "zipp >= 3.23, < 3.24", +] + +[project.optional-dependencies] +dev = [ + "bump2version == 1.0.1", + "coverage == 7.13.5", + "entrypoints == 0.4", + "flake8 == 7.3.0", + "httpretty == 1.0.5", + "isort == 8.0.1", + "mccabe == 0.7.0", + "mock == 5.2.0", + "more-itertools == 10.8.0", + "packaging == 26.0", + "pluggy == 1.6.0", + "pycodestyle == 2.14.0", + "pyflakes == 3.4.0", + "pyparsing == 3.3.2", + "pytest == 9.0.3", + "pytest-cov == 7.1.0", + "six == 1.17.0", + "yapf == 0.43.0", + "zipp == 3.23.0", +] + +[project.urls] +Homepage = "https://developer.delphix.com" + +[project.scripts] +dvp = "dlpx.virtualization._internal.cli:delphix_sdk" + +[tool.setuptools] +package-dir = {"" = "src/main/python"} +include-package-data = true + +[tool.setuptools.packages.find] +where = ["src/main/python"] + +[tool.coverage.run] +source = ["src/main/python"] +omit = [ + "*/plugin_template/*", + "*.template", +] \ No newline at end of file diff --git a/tools/requirements.txt b/tools/requirements.txt deleted file mode 100644 index 76829ac3..00000000 --- a/tools/requirements.txt +++ /dev/null @@ -1,22 +0,0 @@ -./../common -./../libs -./../platform -bump2version==1.0.1 -coverage==7.13.5 -entrypoints==0.4 -flake8==7.3.0 -httpretty==1.0.5 -isort==8.0.1 -mccabe==0.7.0 -mock==5.2.0 -more-itertools==10.8.0 -packaging==26.0 -pluggy==1.6.0 -pycodestyle==2.14.0 -pyflakes==3.4.0 -pyparsing==3.3.2 -pytest-cov==7.1.0 -pytest==9.0.3 -six==1.17.0 -yapf==0.43.0 -zipp==3.23.0 diff --git a/tools/setup.cfg b/tools/setup.cfg deleted file mode 100644 index d4c1bf04..00000000 --- a/tools/setup.cfg +++ /dev/null @@ -1,26 +0,0 @@ -# -# Copyright (c) 2019, 2021 by Delphix. All rights reserved. -# - -[metadata] -metadata_version: 1.2 -author: Delphix -author_email: virtualization-plugins@delphix.com -home_page: https://developer.delphix.com -summary: Delphix Virtualization SDK Tools -long_description: file: README.md -long_description_content_type: text/markdown -classifiers: - Development Status :: 5 - Production/Stable - Programming Language :: Python - Programming Language :: Python :: 3.11 - License :: OSI Approved :: Apache Software License - Operating System :: OS Independent - -[options] -include_package_data = True -requires_python: >=3.11, <3.12 - -[options.entry_points] -console_scripts = - dvp = dlpx.virtualization._internal.cli:delphix_sdk diff --git a/tools/setup.py b/tools/setup.py deleted file mode 100644 index d3eb0ed6..00000000 --- a/tools/setup.py +++ /dev/null @@ -1,43 +0,0 @@ -import os -import setuptools - -PYTHON_SRC = 'src/main/python' - -with open(os.path.join(PYTHON_SRC, 'dlpx/virtualization/_internal/VERSION')) as version_file: - version = version_file.read().strip() - -# -# Update the dependency using below use cases -# 1. Dependency version change does not break test cases or have code issues -# - Only update the maximum version (<). -# 2. Dependency version changes break test cases or have code issues -# - Update the minimum as well as maximum version along with code changes. -# -install_requires = [ - "attrs >= 25.3, < 25.4", - "certifi >= 2024, < 2025", - "click == 7.1.2", - "click-configfile == 0.2.3", - "configparser >= 7.2, < 7.3", - "dvp-libs == {}".format(version), - "dvp-platform == {}".format(version), - "flake8 >= 7.3, < 7.4", - "httpretty >= 1.0, < 1.1", - "importlib-resources >= 6.5, < 6.6", - "jinja2 >= 3.1, < 3.2", - "jsonschema >= 4.25, < 4.26", - "MarkupSafe >= 3.0, < 3.1", - "pkgutil_resolve_name == 1.3.10", - "pyyaml >= 6, < 7", - "requests >= 2.32, < 2.33", - "six >= 1.17, < 1.18", - "zipp >= 3.23, < 3.24", -] - -setuptools.setup(name='dvp-tools', - version=version, - install_requires=install_requires, - package_dir={'': PYTHON_SRC}, - packages=setuptools.find_packages(PYTHON_SRC), - python_requires='>=3.11, <3.12', - ) diff --git a/tools/src/main/python/dlpx/virtualization/_internal/VERSION b/tools/src/main/python/dlpx/virtualization/_internal/VERSION deleted file mode 100644 index 831446cb..00000000 --- a/tools/src/main/python/dlpx/virtualization/_internal/VERSION +++ /dev/null @@ -1 +0,0 @@ -5.1.0 diff --git a/tools/src/main/python/dlpx/virtualization/_internal/package_util.py b/tools/src/main/python/dlpx/virtualization/_internal/package_util.py index b065a380..ce2fee0b 100644 --- a/tools/src/main/python/dlpx/virtualization/_internal/package_util.py +++ b/tools/src/main/python/dlpx/virtualization/_internal/package_util.py @@ -6,8 +6,11 @@ import logging import os import re +from importlib.metadata import PackageNotFoundError +from importlib.metadata import version as _pkg_version from dlpx.virtualization import _internal as virtualization_internal +from dlpx.virtualization._internal import exceptions from dlpx.virtualization.platform import util from six.moves import configparser @@ -45,11 +48,12 @@ def _get_settings(): @_run_once def get_version(): - """Returns the version of the dlpx.virtualization._internal package.""" - with open(os.path.join(get_internal_package_root(), - 'VERSION')) as version_file: - version = version_file.read().strip() - return version + """Returns the version of the installed dvp-tools package.""" + try: + return _pkg_version("dvp-tools") + except PackageNotFoundError: + raise exceptions.UserError( + "dvp-tools is not installed. Run 'pip install dvp'.") def get_external_version_string(version_string): diff --git a/tools/src/main/python/dlpx/virtualization/_internal/plugin_dependency_util.py b/tools/src/main/python/dlpx/virtualization/_internal/plugin_dependency_util.py index 4ae8ea88..ff269092 100644 --- a/tools/src/main/python/dlpx/virtualization/_internal/plugin_dependency_util.py +++ b/tools/src/main/python/dlpx/virtualization/_internal/plugin_dependency_util.py @@ -1,5 +1,5 @@ # -# Copyright (c) 2019, 2021 by Delphix. All rights reserved. +# Copyright (c) 2019, 2021, 2026 by Delphix. All rights reserved. # import compileall @@ -45,12 +45,12 @@ def install_deps(target_dir, local_vsdk_root=None): # # Build the wheels for each package in a temporary directory. # - # Pip supports installing directly from a setup.py file but this + # Pip supports installing directly from a package source tree but this # proved to be incredibly slow due to how it copies source files. # If that issue is resolved, it would likely be better to use pip to - # install directly from the setup.py file instead of needing to build - # the wheels first. This would remove the need for a temp directory - # as well. + # install directly from the package directory instead of needing to + # build the wheels first. This would remove the need for a temp + # directory as well. # with file_util.tmpdir() as wheel_dir: for package in package_names: @@ -147,22 +147,23 @@ def _pip_install_to_dir(dependencies, target_dir): def _build_wheel(package_root, target_dir=None): """ - Uses the 'setup.py' file in package_root to build a wheel distribution. If - target_dir is present, the wheel is built into it. Raises a - SubprocessFailedError if it fails. + Uses the 'pyproject.toml' file in package_root to build a wheel + distribution via pip (PEP 517). If target_dir is present, the wheel is + built into it. Raises a SubprocessFailedError if it fails. Args: package_root: The path to the root of the package to build. It is - assumed there is a setup.py file in this directory. + assumed there is a pyproject.toml file in this directory. target_dir: The directory to build the wheel into. """ - if not os.path.exists(os.path.join(package_root, 'setup.py')): + if not os.path.exists(os.path.join(package_root, 'pyproject.toml')): raise RuntimeError( - 'No setup.py file exists in directory {}'.format(package_root)) + 'No pyproject.toml file exists in directory {}'.format( + package_root)) - args = [sys.executable, 'setup.py', 'bdist_wheel'] + args = [sys.executable, '-m', 'pip', 'wheel', '--no-deps', '.'] if target_dir: - args.extend(['-d', target_dir]) + args.extend(['-w', target_dir]) logger.debug('Executing %s', ' '.join(args)) proc = subprocess.Popen(args, diff --git a/tools/src/test/python/dlpx/virtualization/_internal/engine_version.cfg b/tools/src/test/python/dlpx/virtualization/_internal/engine_version.cfg index a3b69e2e..b9046630 100644 --- a/tools/src/test/python/dlpx/virtualization/_internal/engine_version.cfg +++ b/tools/src/test/python/dlpx/virtualization/_internal/engine_version.cfg @@ -1,5 +1,5 @@ # -# Copyright (c) 2022 by Delphix. All rights reserved. +# Copyright (c) 2026 by Delphix. All rights reserved. # # diff --git a/tools/src/test/python/dlpx/virtualization/_internal/test_cli.py b/tools/src/test/python/dlpx/virtualization/_internal/test_cli.py index 49c673bf..2ec8ad0b 100644 --- a/tools/src/test/python/dlpx/virtualization/_internal/test_cli.py +++ b/tools/src/test/python/dlpx/virtualization/_internal/test_cli.py @@ -198,12 +198,17 @@ def test_empty_root_dir(plugin_name): assert "Invalid value for '-r'" in result.output @staticmethod - def test_name_required(): + @mock.patch('dlpx.virtualization._internal.commands.initialize.init') + def test_name_optional(mock_init): + # --plugin-name is optional (PYT-536); when omitted, the plugin id + # (auto-generated UUID) is used as the display name. runner = click_testing.CliRunner() result = runner.invoke(cli.delphix_sdk, ['init']) - assert result.exit_code != 0 + assert result.exit_code == 0, 'Output: {}'.format(result.output) + mock_init.assert_called_once_with(os.getcwd(), const.DIRECT_TYPE, + None, const.UNIX_HOST_TYPE) @staticmethod def test_multiple_host_types(): diff --git a/tools/src/test/python/dlpx/virtualization/_internal/test_plugin_dependency_util.py b/tools/src/test/python/dlpx/virtualization/_internal/test_plugin_dependency_util.py index 2555ed87..1b705900 100644 --- a/tools/src/test/python/dlpx/virtualization/_internal/test_plugin_dependency_util.py +++ b/tools/src/test/python/dlpx/virtualization/_internal/test_plugin_dependency_util.py @@ -1,5 +1,5 @@ # -# Copyright (c) 2019, 2020 by Delphix. All rights reserved. +# Copyright (c) 2019, 2020, 2026 by Delphix. All rights reserved. # import os @@ -114,8 +114,8 @@ def test_install_to_dir(mock_popen): @staticmethod @mock.patch.object(subprocess, 'Popen') def test_build_wheel(mock_popen, tmp_path): - setup_file = tmp_path / 'setup.py' - setup_file.touch() + pyproject_file = tmp_path / 'pyproject.toml' + pyproject_file.touch() mock_popen.return_value.communicate.return_value = ('output', '') mock_popen.return_value.wait.return_value = 0 @@ -123,24 +123,25 @@ def test_build_wheel(mock_popen, tmp_path): pdu._build_wheel(tmp_path.as_posix()) mock_popen.assert_called_once_with( - [sys.executable, 'setup.py', 'bdist_wheel'], + [sys.executable, '-m', 'pip', 'wheel', '--no-deps', '.'], cwd=tmp_path.as_posix(), stdout=subprocess.PIPE, stderr=subprocess.STDOUT) @staticmethod - def test_build_wheel_fails_with_no_setup_file(tmp_path): + def test_build_wheel_fails_with_no_pyproject_file(tmp_path): with pytest.raises(RuntimeError) as excinfo: pdu._build_wheel(tmp_path.as_posix()) - assert str(excinfo.value) == ('No setup.py file exists in directory ' - '{}'.format(tmp_path.as_posix())) + assert str(excinfo.value) == ('No pyproject.toml file exists in ' + 'directory {}'.format( + tmp_path.as_posix())) @staticmethod @mock.patch.object(subprocess, 'Popen') def test_build_wheel_non_zero_exit(mock_popen, tmp_path): - setup_file = tmp_path / 'setup.py' - setup_file.touch() + pyproject_file = tmp_path / 'pyproject.toml' + pyproject_file.touch() mock_popen.return_value.communicate.return_value = ('output', '') mock_popen.return_value.wait.return_value = 1 @@ -150,8 +151,10 @@ def test_build_wheel_non_zero_exit(mock_popen, tmp_path): e = excinfo.value - expected_args = [sys.executable, 'setup.py', 'bdist_wheel'] - mock_popen.asesrt_called_once_with(expected_args, + expected_args = [ + sys.executable, '-m', 'pip', 'wheel', '--no-deps', '.' + ] + mock_popen.assert_called_once_with(expected_args, cwd=tmp_path.as_posix(), stdout=subprocess.PIPE, stderr=subprocess.STDOUT) @@ -164,10 +167,10 @@ def test_build_wheel_non_zero_exit(mock_popen, tmp_path): @mock.patch.object(subprocess, 'Popen') def test_build_wheel_target_dir(mock_popen, tmp_path): package_dir = tmp_path / 'pkg' - setup_file = package_dir / 'setup.py' + pyproject_file = package_dir / 'pyproject.toml' target_dir = tmp_path / 'tgt' package_dir.mkdir() - setup_file.touch() + pyproject_file.touch() target_dir.mkdir() mock_popen.return_value.communicate.return_value = ('output', '') @@ -177,7 +180,7 @@ def test_build_wheel_target_dir(mock_popen, tmp_path): target_dir=target_dir.as_posix()) expected_args = [ - sys.executable, 'setup.py', 'bdist_wheel', '-d', + sys.executable, '-m', 'pip', 'wheel', '--no-deps', '.', '-w', target_dir.as_posix() ] mock_popen.assert_called_once_with(expected_args, From 8bf151af44a85d026b9482ae92ae978559562093 Mon Sep 17 00:00:00 2001 From: Rakshanda Bhat <111853524+rakshandabhat@users.noreply.github.com> Date: Wed, 24 Jun 2026 09:37:12 +0530 Subject: [PATCH 21/21] CP-13446 Claude md file for vSDK repo (#679) * CP-13446 Claude md file for vSDK repo * Updated some additional details from PIT claude md file --- CLAUDE.md | 228 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 228 insertions(+) create mode 100644 CLAUDE.md diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 00000000..29a7ab00 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,228 @@ +[//]: # (Copyright (c) 2026 by Delphix. All rights reserved.) + +# CLAUDE.md + +This file provides guidance to AI agents working with code in this repository. + +## Project Overview + +The **Delphix Virtualization SDK** — the Python toolkit plugin developers use to build AppData plugins for the Delphix Engine. The repo produces **five Python distributions** that are versioned and shipped together: + +| Package (dist name) | Directory | Purpose | +| --- | --- | --- | +| `dvp` | `dvp/` | Umbrella/meta package | +| `dvp-common` | `common/` | Shared classes/exceptions used by `libs` + `platform` | +| `dvp-libs` | `libs/` | Runtime **Libs** API plugins call on the engine (`run_bash`/`run_powershell`/`run_expect`/`run_sync`, credential helpers) | +| `dvp-platform` | `platform/` | The **Plugin** programming model — `Plugin()` + Discovery/Linked/Virtual/Upgrade operations | +| `dvp-tools` | `tools/` | The `dvp` CLI (plugin build/test/distribution) | + +Two conceptually distinct parts, with different workflows: + +- **`tools`** — the SDK's CLI (`dvp`). Aids plugin development, testing, and distribution. Changes here are isolated from the Delphix Engine. +- **`common` / `libs` / `platform`** — collectively the **"wrappers"**: vanilla Python classes that abstract the Virtualization API protobuf messages (`dvp-api`, **published by app-gate**) away from plugin developers. This is the API plugin developers write against. Changes here affect the plugin build. + +A plugin must package all its dependencies — including `dvp-api` and the wrappers — and `dvp build` does this automatically. + +## Plugin Programming Model + +This is the API the wrappers expose and what `platform`/`libs` code must keep stable for plugin authors: + +- A plugin instantiates `Plugin()` (`from dlpx.virtualization.platform import Plugin`), composed of four operation groups (`platform/_plugin.py`): **`discovery`** (`_discovery.py`), **`linked`** (direct/staged linking, `_linked.py`), **`virtual`** (provisioning virtual datasets, `_virtual.py`), and **`upgrade`** (`_upgrade.py`). +- Authors implement an operation by decorating a method with the plugin object's group, e.g. `@my_plugin.virtual.configure()`, `@my_plugin.discovery.repository()`. The decorator name must start with the plugin variable's name. +- At runtime, plugin code calls the **`libs`** API (`from dlpx.virtualization.libs import ...`) to do remote work on the host: `run_bash` / `run_powershell` / `run_expect`, `run_sync` (rsync), and `retrieve_credentials` / `upgrade_password`. +- **Namespace packages:** all five packages share the `dlpx.virtualization` namespace via `pkgutil.extend_path` in each `__init__.py`, so the separately-installed packages import under one namespace. `platform/__init__.py` re-exports the public symbols — **only symbols exported there are part of the plugin-author API surface.** +- **Proto conversion:** authors never touch protobuf directly. Wrapper data classes implement `to_proto()` / `from_proto()` (operation classes also have `to_protobuf*` helpers); the operation groups' `_internal_*` methods do the protobuf conversion + input validation on the engine side. +- **Adding an operation:** define the constant in `platform/operation.py`, add the decorator + `_internal_*` wrapper to the relevant operations class, export any new public symbols from `platform/__init__.py`, add tests under `platform/src/test/python/`, and update `plugin_validator.py` if the operation should be required for a valid plugin. + +## Repository Layout + +- `common/`, `libs/`, `platform/`, `tools/`, `dvp/` — the five packages, each with its own `pyproject.toml`. +- Each package uses a **Maven-style source layout**: + - main: `/src/main/python/dlpx/virtualization/...` + - tests: `/src/test/python/dlpx/virtualization/...` (with a `conftest.py` per package) +- `bin/` — build/test tooling (`build_project.sh`). +- `docs/` — SDK documentation, built with **MkDocs** (Material theme; `docs/mkdocs.yml`) in its own `pipenv` env (`docs/Pipfile`); `docs/build.sh` runs `pipenv run mkdocs build` and publishes to S3. Public user docs live at https://developer.delphix.com. +- `.bumpversion.cfg` — version config spanning all five packages. + +## Environment & Install + +- **End users** install the published CLI from PyPI: `pip install dvp` (or `dvp==`). Everything below is for SDK *developers*. +- **Python 3.11 only** (`requires-python = ">=3.11, <3.12"`; current series is vSDK 5.x). Develop in a Python 3.11 virtualenv. +- `dvp-api` is hosted on **TestPyPI**, so configure pip with an extra index. Create `/pip.conf`: + ``` + [install] + index-url=https://pypi.org/simple/ + extra-index-url=https://test.pypi.org/simple/ + ``` +- **Editable install of a single package** (with dev tooling): from the package directory, + ``` + pip install -e ".[dev]" + ``` + (The `dev` extra replaces the old `requirements.txt`-based dev install.) + +## Build & Test Commands + +All from the repo root via `bin/build_project.sh` (operates on `common`, `libs`, `platform`, `tools`, `dvp`): + +```bash +sh bin/build_project.sh -h # help +sh bin/build_project.sh -b # build all modules +sh bin/build_project.sh -t # run unit tests for all modules +sh bin/build_project.sh -f # flake8 validation +sh bin/build_project.sh -c # test-coverage mode +sh bin/build_project.sh -v # verbose +sh bin/build_project.sh -m common # restrict to module(s); repeatable: -m common -m libs +# flags combine, e.g.: +sh bin/build_project.sh -bt -m tools # build + test just tools +sh bin/build_project.sh -bft -m tools # build + flake8 + test +sh bin/build_project.sh -bct -m tools # build + coverage + test +``` + +**Single package's unit tests** — from that package directory: +```bash +python -m pytest src/test/python +``` + +## CLI (`tools` / `dvp`) + +- Built with **Click**. The console-script entry point (`tools/pyproject.toml` → `[project.scripts]`) is `dvp = "dlpx.virtualization._internal.cli:delphix_sdk"` (a `@click.group`). Add a subcommand by defining a method in `dlpx.virtualization._internal.cli` annotated `@delphix_sdk.command()` — the method name becomes the command name. +- **Keep Click confined to `cli.py`** — it's the single source of truth for the CLI and must hold **no business logic**; each command delegates immediately into a `_internal/commands/.py` module. +- Commands: `dvp init` (scaffold a plugin), `dvp build` (bundle the plugin + wrappers + `dvp-api` into an upload artifact), `dvp upload` (push the artifact to a Delphix Engine), `dvp download-logs`. +- `dvp build` bundles the wrappers (`common`/`libs`/`platform`) with the plugin: pass `--dev` to build them **from source**, otherwise they're fetched from PyPI. Building from source also needs `~/.dvp/config`: + ``` + [dev] + vsdk_root = /path/to/virtualization-sdk + ``` +- Manually exercise the CLI with `dvp ` after an editable install of `tools`. +- User config `~/.dvp/config`: `[defaults]` (engine/user/password) and `[dev]` (`vsdk_root` for local wrapper builds). Global flags `-v`/`--verbose`, `-q`/`--quiet`. + +Key modules under `tools/src/main/python/dlpx/virtualization/_internal/` (verified to exist; purposes summarized): + +| Module | Purpose | +| --- | --- | +| `cli.py` | Click group + subcommands (declarations only) | +| `commands/build.py` | Build: config validation, codegen, dependency packaging | +| `commands/upload.py` | Upload artifact to the engine + job polling | +| `commands/initialize.py` | Plugin scaffold generation (`dvp init`) | +| `commands/download_logs.py` | Log retrieval from the engine | +| `codegen.py` | Generates Python classes from the plugin's JSON schemas | +| `plugin_util.py` / `plugin_validator.py` / `plugin_importer.py` | Plugin config parsing, decorator/validation, dynamic import | +| `delphix_client.py` | HTTP client for the Delphix Engine REST API | + +**Adding a CLI command:** add a `@delphix_sdk.command()` function in `cli.py`, put the logic in a new `commands/.py`, add tests under `tools/src/test/python/.../commands/`. + +## Conventions + +- **Language**: Python 3.11. Maven-style `src/main/python` + `src/test/python` tree under `dlpx/virtualization/`. +- **Linting**: `flake8` (run via `build_project.sh -f`). +- **Formatting / imports**: `yapf` and `isort` (both in each package's `dev` extra). +- **Testing**: `pytest` (+ `pytest-cov`, `coverage`); `mock`; `httpretty` for HTTP stubbing. Tests live in `src/test/python/...` with per-package `conftest.py`. Provide tests with changes where appropriate. +- **Packaging**: `setuptools` build backend; metadata in each `pyproject.toml`. License is Apache-2.0; sibling-package version pins (e.g. `dvp-tools` → `dvp-libs`/`dvp-platform`) are kept in sync via `.bumpversion.cfg`. +- **Copyright header**: every source file carries `Copyright (c) by Delphix. All rights reserved.` Use `, ` for files edited across years. +- **Lint/format config**: `flake8`/`isort`/`yapf` run on **tool defaults** — there are no repo-level `[tool.flake8]`/`[tool.isort]`/`.flake8` overrides (only `[tool.coverage.run]` in `tools/pyproject.toml`). + +## Exception Hierarchy + +Each package defines its own exceptions (verified in each package's `exceptions.py` / classes): + +- **`common`** — base types reused elsewhere: `PluginRuntimeError`, `IncorrectTypeError`, `PlatformError`. +- **`libs`** — remote-execution failures: `LibraryError`, `PluginScriptError`, `IncorrectArgumentTypeError`. +- **`platform`** — `UserError` plus operation-validation errors: `IncorrectReturnTypeError`, `OperationNotDefinedError`, `OperationAlreadyDefinedError`, `DecoratorNotFunctionError`, `IncorrectUpgradeObjectTypeError`, the `MigrationId*` errors, etc. +- **`tools`** — `SDKToolingError` (internal) and `UserError` (user-facing/actionable), plus CLI-specific errors (`BuildFailedError`, `SchemaValidationError`, `HttpError`, `PluginUploadJobFailed`, `PluginUploadWaitTimedOut`, …). + +## Versioning + +- All five packages (`dvp`, `dvp-common`, `dvp-libs`, `dvp-platform`, `dvp-tools`) are versioned and released **together**, using **semantic versioning** managed by **`bump2version`**. +- Version format: `MAJOR.MINOR.PATCH` (released) or `MAJOR.MINOR.PATCH.dev` (dev builds); config in `.bumpversion.cfg`. + ```bash + bumpversion dev # 1.1.0.dev7 -> 1.1.0.dev8 + bumpversion [major|minor|patch] + bumpversion release # 1.1.0.dev7 -> 1.1.0 + ``` +- A bump updates the `[project].version` (and sibling-package pins) in all five `pyproject.toml`s **and** the `DVP_VERSION` constant in `tools/.../_internal/test_package_util.py` atomically — these targets are listed in `.bumpversion.cfg`. It does **not** commit or tag (`commit = False`, `tag = False`), so commit all the bumped files together yourself. Current version: `5.1.0`. + +## Testing Tiers + +1. **Unit** — `pytest` per package (or `build_project.sh -t`). No engine required. +2. **Manual** — wrapper changes are exercised by building a plugin, uploading to a Delphix Engine, and running standard workflows. +3. **Functional (blackbox)** — run from **app-gate** via `git blackbox` against your pushed SDK branch, e.g.: + ``` + git blackbox -s appdata_python_samples \ + --extra-params="-p virt-sdk-repo=https://github.com//virtualization-sdk.git -p virt-sdk-branch=" + ``` + **Minimum per PR**: `appdata_python_samples` and `appdata_basic` (direct or staged plugin). CLI-focused changes also run the `virtualization_sdk` suite. A non-dev version bump requires QA to create a matching `sdk-x-y-z` toolkit branch first. + +## CI, Gate & Release + +- **CI (GitHub Actions):** the PR check `.github/workflows/pre-commit.yml` runs the **full `pytest` suite across Python 3.11 on Ubuntu, macOS, and Windows** for PRs targeting `master`/`develop`/`release`. It installs the five packages in dependency order (`common` → `libs` → `platform` → `tools`/`dvp`) with `--find-links` to TestPyPI for `dvp-api`. Keep tests green on all three OSes. (Despite its name, this workflow runs cross-OS pytest, not the pre-commit framework.) +- Other workflows: `publish-python-packages.yml` (publishes the five packages), `publish-docs.yml` (publishes docs); `dependabot.yml` manages dependency bumps. +- **Push gate:** `.hooksconfig` defines the Delphix gate for this repo — gatekeeper approval group, Slack push notifications, allowed Jira issue types per branch, and review/comment checks. +- **Manual release (Artifactory):** `sh bin/upload.sh` publishes to the internal dev PyPI (`dvp-local-pypi`); `sh bin/upload.sh --prod` to production (`delphix-local`). Requires `ARTIFACTORY_PYPI_USER` / `ARTIFACTORY_PYPI_PASS`; it reads the version from `.bumpversion.cfg` and uploads with `twine`. + +## Contributing / Posting Code for Review + +- Fork-based workflow: fork → clone → change → test → bump version → push to a branch on your fork → open a PR to `delphix/virtualization-sdk`. +- **PRs must be based on the current `master` branch** and apply without conflicts. (Default branch is `develop`; CI gates PRs to `master`/`develop`/`release`.) +- **Limit each PR to a single commit that resolves one issue** — squash and rebase onto `master`; for large changes, use a stack of logically independent patches. +- **Commit message format** (see `CONTRIBUTING.md`): start with the GitHub issue id and its title, e.g. `Fixes #123 Format of error is incorrect`, followed by an optional description (each line ≤ 72 chars). If it doesn't address an issue, describe the changes. +- Merges require approval from a **code owner** (per `CODEOWNERS`). +- Bugs and features are filed as **GitHub issues** on `delphix/virtualization-sdk` using the Bug Report / Feature Request templates. + +## Relationship to app-gate + +The Virtualization API protobuf messages (`dvp-api`) are **defined and published by the app-gate repo** (`appliance/server/virtualizationApi`); the wrappers here abstract them for plugin authors. Blackbox tests for this repo are driven from app-gate. Keep wrapper changes compatible with the `dvp-api` version they target. + +## Plugin Operations Reference + +> ⚠ **Verification note:** the decorator names and their operation groups below are **verified against the code** (`platform/_discovery.py` / `_linked.py` / `_virtual.py` / `_upgrade.py`). The **Required / Arguments / Returns** columns are taken from the public plugin-operations docs (https://developer.delphix.com/References/Plugin_Operations/) and were **not** re-verified line-by-line against the operation signatures in this pass — confirm there before relying on exact argument names/return types. Argument names are contractual (must match exactly). Also present in code but omitted from the tables: `linked.source_to_physical()` and `virtual.source_to_physical()`. + +### Discovery +| Decorator | Required | Arguments | Returns | +|---|---|---|---| +| `discovery.repository()` | Yes | `source_connection` | `list[RepositoryDefinition]` | +| `discovery.source_config()` | Yes | `source_connection`, `repository` | `list[SourceConfigDefinition]` | + +### Linked source (dSource) +| Decorator | Required | Arguments | Returns | +|---|---|---|---| +| `linked.pre_snapshot()` | No | `direct_source`\|`staged_source`, `repository`, `source_config`, `optional_snapshot_parameters` | None | +| `linked.post_snapshot()` | Yes | `direct_source`\|`staged_source`, `repository`, `source_config`, `optional_snapshot_parameters` | `SnapshotDefinition` | +| `linked.start_staging()` | No | `staged_source`, `repository`, `source_config` | None | +| `linked.stop_staging()` | No | `staged_source`, `repository`, `source_config` | None | +| `linked.status()` | No | `staged_source`, `repository`, `source_config` | `Status` (defaults `ACTIVE`) | +| `linked.worker()` | No | `staged_source`, `repository`, `source_config` | None | +| `linked.mount_specification()` | Yes (staged) | `staged_source`, `repository` | `MountSpecification` | +| `linked.source_size()` | No | `direct_source`\|`staged_source`, `repository`, `source_config` | numeric | + +### Virtual source (VDB) +| Decorator | Required | Arguments | Returns | +|---|---|---|---| +| `virtual.initialize()` | No | `virtual_source`, `repository` | `SourceConfigDefinition` | +| `virtual.configure()` | Yes | `virtual_source`, `snapshot`, `repository` | `SourceConfigDefinition` | +| `virtual.unconfigure()` | No | `virtual_source`, `repository`, `source_config` | None | +| `virtual.reconfigure()` | Yes | `virtual_source`, `repository`, `source_config`, `snapshot` | `SourceConfigDefinition` | +| `virtual.cleanup()` | No | `virtual_source`, `repository`, `source_config` | None | +| `virtual.start()` / `virtual.stop()` | No | `virtual_source`, `repository`, `source_config` | None | +| `virtual.pre_snapshot()` | No | `virtual_source`, `repository`, `source_config` | None | +| `virtual.post_snapshot()` | Yes | `virtual_source`, `repository`, `source_config` | `SnapshotDefinition` | +| `virtual.mount_specification()` | Yes | `virtual_source`, `repository` | `MountSpecification` | +| `virtual.status()` | No | `virtual_source`, `repository`, `source_config` | `Status` (defaults `ACTIVE`) | +| `virtual.source_size()` | No | `virtual_source`, `repository`, `source_config` | numeric | + +### Data migration (upgrade) +`upgrade.repository(migration_id)`, `upgrade.source_config(...)`, `upgrade.linked_source(...)`, `upgrade.virtual_source(...)`, `upgrade.snapshot(...)` — all optional; each takes the old object **as a plain dict** (property names match the previous schema verbatim) and returns a dict. Migrations run in `migration_id` order. + +Behavioral notes (from the docs): `optional_snapshot_parameters` is `None` for scheduled-policy snapshots (set only on user-triggered syncs); `virtual.unconfigure()` runs on Refresh/Delete/Disable (not just Delete); `virtual.cleanup()` runs after `unconfigure()` in the Delete flow; `virtual.mount_specification()` is the most-triggered required op (Enable/Provision/Refresh/Rollback/Start); `MountSpecification.ownership_specification` is Unix-only and optional. + +## Engine ↔ Plugin Data Flows + +> ⚠ **Verification note:** these describe **engine-side orchestration** (app-gate / Delphix Engine) — the *order* in which the engine invokes plugin operations and how it stores results. They are **not** code in this repo and were **not** verified here; they're carried over from prior documentation. Confirm against the engine docs / SDD before relying on exact ordering or storage details. Useful as a starting map for "trace the full data flow before deciding where a change belongs." + +- **dSource link:** `linked.mount_specification` → `linked.start_staging` → `linked.pre_snapshot` → engine ingests (DIRECT: engine pulls; STAGED: plugin controls transfer) → `linked.post_snapshot` returns a `SnapshotDefinition` the engine persists as snapshot metadata. +- **dSource sync:** `linked.pre_snapshot` (optional) → ingest → `linked.post_snapshot` (new snapshot on the timeflow). +- **VDB provision:** `virtual.mount_specification` → engine clones+mounts → `virtual.configure(virtual_source, snapshot, repository)` → returns `SourceConfigDefinition`. +- **VDB refresh:** `virtual.unconfigure` → `virtual.mount_specification` → `virtual.configure` (newer snapshot). +- **VDB rollback / enable:** `virtual.mount_specification` → `virtual.reconfigure(...)` (+ `virtual.status` to verify on enable). +- **VDB snapshot:** `virtual.pre_snapshot` (optional) → engine snapshots → `virtual.post_snapshot` → `SnapshotDefinition`. +- **VDB delete:** `virtual.stop` (if running) → `virtual.unconfigure` → `virtual.cleanup` → engine unmounts/destroys the clone. +- **Plugin upgrade:** for each stored object the engine calls the matching `upgrade.*` migration (old dict in → new dict out) in `migration_id` order.