diff --git a/README.md b/README.md index f03f0d0..e195fa8 100644 --- a/README.md +++ b/README.md @@ -30,14 +30,14 @@ $ graphtage original.json modified.json ``` ```json { - "z̟b̶ab̟r̶": "testing", "foo": [ 1̶,̶ 2, 3, 4,̟ 5̟ - ],̟ + ], + "b̶z̟ar̶b̟": "testing",̟ "̟w̟o̟o̟"̟:̟ ̟[̟ "̟f̟o̟o̟b̟a̟r̟"̟ ]̟ @@ -46,12 +46,34 @@ $ graphtage original.json modified.json ## Installation +Graphtage requires Python 3.10 or later. It is tested against Python 3.10 through 3.14. + ```console $ pip3 install graphtage ``` +Installing the package puts two commands on your `PATH`: `graphtage`, the diff utility, and `graphtage-git-diff`, the +external diff driver described in [Git Integration](#git-integration). + +To work on Graphtage itself, install the `dev` extra, which adds pytest, Ruff, and Sphinx: + +```console +$ pip3 install 'graphtage[dev]' +``` + ## Command Line Usage +### Input File Types +Graphtage infers the type of each input file from its extension. To state the type instead of inferring it, use +`--from-` for the first file and `--to-` for the second. Both flags exist for every format Graphtage +supports: `csv`, `html`, `ini`, `json`, `json5`, `pickle`, `plist`, `toml`, `xml`, and `yaml`. For example, to read a +JSON document whose name does not end in `.json`: +```console +$ graphtage --from-json config.txt config.json +``` +`--from-mime` and `--to-mime` do the same thing but take a MIME type rather than a format name, which matters for the +formats that Graphtage registers under more than one type. Run `graphtage --help` for the accepted values. + ### Output Formatting Graphtage performs an analysis on an intermediate representation of the trees that is divorced from the filetypes of the input files. This means, for example, that you can diff a JSON file against a YAML file. Also, the output format can be @@ -60,35 +82,39 @@ first input file. But one could, for example, diff two JSON files and format the command-line arguments to specify these transformations, such as `--format`; please check the `--help` output for more information. +Graphtage sorts the keys of every dictionary it reads, so the output orders keys alphabetically no matter how the +input files order them. The examples in this section all render the file `{"foo": [1, 2, 3], "bar": "baz"}` diffed +against itself. + By default, Graphtage pretty-prints its output with as many line breaks and indents as possible. ```json { + "bar": "baz", "foo": [ 1, 2, 3 - ], - "bar": "baz" + ] } ``` Use the `--join-lists` or `-jl` option to suppress linebreaks after list items: ```json { - "foo": [1, 2, 3], - "bar": "baz" + "bar": "baz", + "foo": [1,2,3] } ``` Likewise, use the `--join-dict-items` or `-jd` option to suppress linebreaks after key/value pairs in a dict: ```json -{"foo": [ - 1, - 2, - 3 -], "bar": "baz"} +{"bar": "baz","foo": [ + 1, + 2, + 3 + ]} ``` Use `--condensed` or `-j` to apply both of these options: ```json -{"foo": [1, 2, 3], "bar": "baz"} +{"bar": "baz","foo": [1,2,3]} ``` The `--only-edits` or `-e` option will print out a list of edits rather than applying them to the input file in place. @@ -97,7 +123,8 @@ The `--edit-digest` or `-d` option is like `--only-edits` but prints a more conc human-readable. ### Matching Options -By default, Graphtage tries to match all possible pairs of elements in a dictionary. +By default, Graphtage matches the values of key/value pairs that share a key, and tries to match all possible +pairs of the remaining elements. Matching two dictionaries with each other is hard. Although computationally tractable, this can sometimes be onerous for input files with huge dictionaries. Graphtage has three different strategies for matching dictionaries: @@ -108,6 +135,9 @@ input files with huge dictionaries. Graphtage has three different strategies for 3. `--dict-strategy auto` (the default) will automatically match the values of any key-value pairs that have identical keys and then use the `match` strategy for the remainder of key/value pairs. +`--dict-strategy` also has the short form `-ds`. The `--no-key-edits` or `-k` option is equivalent to +`--dict-strategy none`. + See [Pull Request #51](https://github.com/trailofbits/graphtage/pull/51) for some examples of how these strategies affect output. @@ -133,6 +163,75 @@ a comparison is large enough for this to matter. `--ignore-list-order` cannot be combined with `--no-list-edits` or `--no-list-edits-when-same-length`, because those two options apply only to ordered lists. +### Match Constraints +The `--match-unless` or `-u` and `--match-if` or `-m` options take an expression that decides whether Graphtage may +pair two nodes. Graphtage evaluates the expression once for each pair of nodes it considers, with `from` bound to the +node from the first file and `to` bound to the node from the second. `--match-unless` refuses the pair when the +expression is true; `--match-if` refuses it unless the expression is true. A refused pair is reported as a wholesale +replacement rather than compared element by element. + +Expressions are parsed by the `graphtage.expressions` module rather than by `eval`. They support arithmetic and +comparison operators, indexing, attribute lookup, and calls to a fixed set of builtins such as `len` and `sorted`. + +Say two files describe the same two servers, each identified by an `id`: +```console +$ echo Original: && cat servers.json && echo Modified: && cat servers.new.json +``` +```json +Original: +{ + "primary": {"id": 1, "host": "alpha"}, + "replica": {"id": 2, "host": "beta"} +} +Modified: +{ + "primary": {"id": 1, "host": "alphas"}, + "replica": {"id": 3, "host": "gamma"} +} +``` +By default Graphtage pairs the two `replica` records and reports the differences between them, even though they +describe different servers: +```console +$ graphtage servers.json servers.new.json +``` +```json +{ + "primary": { + "host": "alpha++s++", + "id": 1 + }, + "replica": { + "host": "~~bet~~++gamm++a", + "id": 2 -> 3 + } +} +``` +Refusing to pair records whose `id` differs reports the second record as replaced instead: +```console +$ graphtage --match-unless "from['id'] != to['id']" servers.json servers.new.json +``` +```json +{ + "primary": { + "host": "alpha++s++", + "id": 1 + }, + "replica": { + "host": "beta", + "id": 2 + } -> { + "host": "gamma", + "id": 3 + } +} +``` +The two options differ in more than the sense of the test. `--match-unless` binds `from` and `to` to plain Python +values, and leaves a pair unconstrained when the expression raises an error, which is what makes the example above +work on the records without also constraining the strings and integers underneath them. `--match-if` binds `from` and +`to` to Graphtage node objects, and refuses a pair when the expression raises an error. Because the constraint applies +to every node in the tree, including the two roots, an expression that reads a key such as `from['id'] == to['id']` +refuses every pair and collapses the whole diff into one replacement. Prefer `--match-unless`. + ### ANSI Color By default, Graphtage will only use ANSI color in its output if it is run from a TTY. If, for example, you would like to have Graphtage emit colorized output from a script or pipe, use the `--color` or `-c` argument. To disable color even @@ -147,7 +246,27 @@ $ graphtage --html original.json modified.json > diff.html ### Status and Logging By default, Graphtage prints status messages and a progress bar to STDERR. To suppress this, use the `--no-status` option. To additionally suppress all but critical log messages, use `--quiet`. Fine-grained control of log messages is -via the `--log-level` option. +via the `--log-level` option. `--debug` is equivalent to `--log-level=DEBUG`, and `--quiet` is equivalent to +`--log-level=CRITICAL --no-status`. + +### Version Information +`--version` or `-v` writes a line such as `Graphtage version 0.3.1` to STDERR. If you pass it without any input files, +Graphtage prints the version and exits; if you pass input files as well, it prints the version and then computes the +diff. `-dumpversion` writes the raw version to STDOUT and exits without reading any input. + +### Exit Status +`graphtage` exits with one of three statuses, so a script can tell the three outcomes apart: + +| Status | Meaning | +|--------|---------| +| `0` | The two inputs are semantically identical. | +| `1` | The two inputs differ. | +| `2` | Graphtage could not compute a diff, for example because a file did not parse or its type was not recognized. | + +Interrupting Graphtage with `SIGINT` returns `-2`, which a POSIX shell reports as `254`. + +Because a status of `1` means "the inputs differ" rather than "something went wrong", a CI job that treats any non-zero +status as a failure will fail on every diff Graphtage finds. Test for `2` to detect an error. ### Git Integration Graphtage installs a `graphtage-git-diff` command that implements git's external diff interface, so `git diff` can @@ -184,7 +303,7 @@ original.json 4, ++5++ ], - "++z++~~b~~a++b++~~r~~": "testing", + "~~b~~++z++a~~r~~++b++": "testing", ++"woo": [ "foobar" ]++ @@ -268,4 +387,4 @@ This research was developed by [Trail of Bits](https://www.trailofbits.com/) wit Advanced Research Projects Agency (DARPA) under the SafeDocs program as a subcontractor to [Galois](https://galois.com). It is licensed under the [GNU Lesser General Public License v3.0](LICENSE). [Contact us](mailto:opensource@trailofbits.com) if you're looking for an exception to the terms. -© 2020–2023, Trail of Bits. +© 2020–2026, Trail of Bits. diff --git a/docs/filetypes.rst b/docs/filetypes.rst index aebb210..1714408 100644 --- a/docs/filetypes.rst +++ b/docs/filetypes.rst @@ -5,48 +5,118 @@ Defining New Filetypes Implementing support for a new Graphtage filetype entails extending the :class:`graphtage.Filetype` class. Subclassing :class:`graphtage.Filetype` automatically registers it with Graphtage. +Subclassing is only the first of several steps, though. A filetype that Graphtage can name but cannot recognize on disk, or one whose formatter cannot resolve every node type it emits, fails at runtime rather than at import. The sections below cover each step in turn. + Filetype Matching ----------------- -Input files are matched to an associated :class:`graphtage.Filetype` using MIME types. Each :class:`graphtage.Filetype` registers one or more MIME types for which it will be responsible. Input file MIME types are classified using the :mod:`mimetypes` module. Sometimes a filetype does not have a standardized MIME type or is not properly classified by the :mod:`mimetypes` module. For example, Graphtage's :class:`graphtage.pickle.Pickle` filetype has neither. You can add support for such a filetype as follows: +Input files are matched to an associated :class:`graphtage.Filetype` using MIME types. Each :class:`graphtage.Filetype` registers one or more MIME types for which it will be responsible. Input file MIME types are classified using the :mod:`mimetypes` module: :func:`graphtage.get_filetype` calls :func:`mimetypes.guess_type` on the path and looks the result up in :data:`graphtage.FILETYPES_BY_MIME`. + +Sometimes a filetype does not have a standardized MIME type or is not properly classified by the :mod:`mimetypes` module. For example, Graphtage's :class:`graphtage.pickle.Pickle` filetype has neither. Graphtage registers the missing types in :func:`graphtage.__main__.register_mimetypes`, which the command line entry point calls once, after it has parsed its arguments and therefore after every filetype module has been imported: .. code-block:: python - import mimetypes + def register_mimetypes(): + mimetypes.init() + ... + if '.pkl' not in mimetypes.types_map and '.pickle' not in mimetypes.types_map: + mimetypes.add_type('application/x-python-pickle', '.pkl') + mimetypes.suffix_map['.pickle'] = '.pkl' + +The first line is the reason a filetype cannot register its own extension at import time. :func:`mimetypes.init` rebuilds the module's global ``types_map`` from scratch, discarding anything an earlier :func:`mimetypes.add_type` call added. A module that calls :func:`mimetypes.add_type` when it is imported therefore loses its registration as soon as :func:`register_mimetypes` runs, and every diff of that filetype fails with ``Could not determine the filetype``. + +There are two places to register an extension, depending on where your filetype lives: + +* A filetype inside Graphtage adds its extension to :func:`graphtage.__main__.register_mimetypes`, alongside the ones already there. +* A filetype outside Graphtage calls :func:`mimetypes.add_type` after :func:`graphtage.__main__.register_mimetypes` has run. Calling it while your module is imported is not enough. From your own entry point, call :func:`graphtage.__main__.register_mimetypes` first and add your type afterward: + + .. code-block:: python + + from graphtage.__main__ import register_mimetypes + + register_mimetypes() + mimetypes.add_type("application/x-widget", ".widget") - if '.pkl' not in mimetypes.types_map and '.pickle' not in mimetypes.types_map: - mimetypes.add_type('application/x-python-pickle', '.pkl') - mimetypes.suffix_map['.pickle'] = '.pkl' + Under the ``graphtage`` command, where you have no chance to run code between the two calls, name the type explicitly with the ``--from-mime`` and ``--to-mime`` options, which skip the guess entirely. Implementing a New Filetype --------------------------- -With the MIME type registered, here is a sketch of how one might define the Pickle filetype: +With the MIME type registered, here is how the Pickle filetype is defined. This is :mod:`graphtage.pickle`, with its docstrings removed and its imports written as an out-of-tree filetype would write them: .. code-block:: python - from graphtage import BuildOptions, Filetype, GraphtageFormatter, TreeNode + import os + + from fickling.fickle import Interpreter, Pickled, PickleDecodeError + + from graphtage import BuildOptions, Filetype, TreeNode + from graphtage.pydiff import PyDiffFormatter, ast_to_tree + class Pickle(Filetype): def __init__(self): super().__init__( - "pickle", # a unique identifier - "application/python-pickle", # the primary MIME type - "application/x-python-pickle" # an optional secondary MIME type + "pickle", # a unique identifier + "application/python-pickle", # the primary MIME type + "application/x-python-pickle", # an optional secondary MIME type ) def build_tree(self, path: str, options: BuildOptions | None = None) -> TreeNode: - # return the root node of the tree built from the given pickle file + with open(path, "rb") as f: + pickled = Pickled.load(f) + ast = Interpreter(pickled).to_ast() + return ast_to_tree(ast, options) def build_tree_handling_errors(self, path: str, options: BuildOptions | None = None) -> str | TreeNode: - # the same as the build_tree() function, - # but on error return a string containing the error message - # - # for example: + # the same as build_tree(), but on error return a string containing the error message try: return self.build_tree(path=path, options=options) except PickleDecodeError as e: return f"Error deserializing {os.path.basename(path)}: {e!s}" - def get_default_formatter(self) -> GraphtageFormatter: - # return the formatter associated with this file type + def get_default_formatter(self) -> PyDiffFormatter: + return PyDiffFormatter.DEFAULT_INSTANCE + +:class:`graphtage.FiletypeWatcher` instantiates the subclass at class definition time, so :meth:`__init__` must take no arguments beyond ``self``, and all three methods must be implemented. Registration into :data:`graphtage.FILETYPES_BY_TYPENAME` and :data:`graphtage.FILETYPES_BY_MIME` happens automatically, and that registration is what generates the ``--from-pickle``, ``--to-pickle``, and ``--format pickle`` command line options. + +Because the class statement itself creates and registers the instance, retrieve your filetype from the registry rather than constructing it: + +.. code-block:: python + + filetype = graphtage.FILETYPES_BY_TYPENAME["pickle"] + tree = filetype.build_tree("example.pkl") + +Calling ``Pickle()`` a second time raises ``ValueError: MIME type application/python-pickle is already assigned``. Running the snippet above verbatim raises the same error, since Graphtage already ships :mod:`graphtage.pickle` under those three identifiers; change the type name and MIME types to try it out. + +If your parser already produces plain Python objects, :func:`graphtage.json.build_tree` converts one into a tree that satisfies the whole :class:`graphtage.TreeNode` contract, so :meth:`build_tree` can be a single call. + +Registering the Module +---------------------- + +Defining the class is not enough on its own: nothing imports it. A filetype inside Graphtage is added to the ``from . import ...`` statement in ``graphtage/__init__.py``, which is also where ``docs/build_api.py`` discovers the modules it generates API pages for. A filetype outside Graphtage must be imported by whatever code runs the diff, before :func:`graphtage.get_filetype` is called. + +Writing the Formatter +--------------------- + +:meth:`Filetype.get_default_formatter` returns the :class:`graphtage.GraphtageFormatter` that renders your nodes. Four rules govern how formatters compose; see :mod:`graphtage.json` and :mod:`graphtage.ini` for worked examples. + +**Route sequence printing through** :meth:`SequenceFormatter.print_SequenceNode`. That method is where insertion and removal edits are turned into output. A formatter that iterates a node's children directly renders the unedited tree correctly and silently drops every insertion and removal, so define ``print_ListNode`` and its siblings as thin wrappers that call ``super().print_SequenceNode(*args, **kwargs)``. A sub-formatter overrides ``print_SequenceNode`` itself to hand any other sequence back to the parent formatter with ``self.parent.print(*args, **kwargs)``, which is how a list containing a dictionary reaches the dictionary sub-formatter. + +**Alias** ``print_UnorderedListNode`` **to** ``print_ListNode``. A :class:`graphtage.UnorderedListNode` reaches your formatter whenever the user passes ``--ignore-list-order``, and it does not inherit from :class:`graphtage.ListNode`, so ``print_ListNode`` alone does not match it. Without the alias, the lookup walks the node's method resolution order down to :class:`graphtage.sequences.SequenceNode` and settles on ``print_SequenceNode``, which delegates back to the parent formatter, which resolves the node the same way again. The print recurses until the interpreter runs out of stack: + +.. code-block:: python + + def print_ListNode(self, *args, **kwargs): + super().print_SequenceNode(*args, **kwargs) + + print_UnorderedListNode = print_ListNode + +**Mark helper formatters** ``is_partial = True``. :class:`graphtage.formatter.FormatterChecker` appends every non-partial formatter to the global :data:`graphtage.formatter.FORMATTERS` list, which :func:`graphtage.formatter.get_formatter` searches when no closer match is found. A helper that is registered globally can therefore be chosen to print a node in an unrelated format. Only the one formatter that :meth:`get_default_formatter` returns should be non-partial. + +**Define each** ``print_`` **method once per formatter tree.** :func:`graphtage.formatter.get_formatter` returns the first match it finds while walking a formatter, its sub-formatters, and then its parents, so a second definition of the same method elsewhere in the tree is unreachable. When two nesting levels need different output for the same node type, branch inside a single key/value formatter rather than splitting the work across two formatters by node type. + +Testing a New Filetype +---------------------- + +Graphtage's own test suite requires a ``test__formatting`` method in ``test/test_formatting.py`` for every registered filetype; ``test_formatter_coverage`` fails without one. The ``@filetype_test`` decorator only round-trips an *unedited* tree, so it cannot catch a formatter that mishandles edits. Add a separate test that diffs two documents and checks that insertions and removals appear in the output. diff --git a/docs/index.rst b/docs/index.rst index 47a300e..e514d28 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -10,7 +10,7 @@ There are several reasons why you might be here… .. topic:: You want to learn how to use Graphtage as a command line utility. - This documentation focuses on Graphtage’ use as a library, specifically how to extend it by implementing new file + This documentation focuses on Graphtage’s use as a library, specifically how to extend it by implementing new file formats. For instructions on using Graphtage as a utility, see the documentation in its `GitHub page`_. .. topic:: You want to programmatically interact with Graphtage as a library. diff --git a/pyproject.toml b/pyproject.toml index a3783ac..2d8b50e 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "hatchling.build" [project] name = "graphtage" -description = "A utility to diff tree-like files such as JSON, XML, YAML, TOML, INI, CSV, and plist." +description = "A utility to diff tree-like files such as JSON, JSON5, XML, HTML, YAML, TOML, INI, CSV, plist, and Python pickle." readme = "README.md" requires-python = ">=3.10" license = {text = "LGPL-3.0-or-later"}