Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
151 changes: 135 additions & 16 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,14 +30,14 @@ $ graphtage original.json modified.json
```
```json
{
"z̟b̶ab̟r̶": "testing",
"foo": [
1̶,̶
2,
3,
4,̟
5̟
],̟
],
"b̶z̟ar̶b̟": "testing",̟
"̟w̟o̟o̟"̟:̟ ̟[̟
"̟f̟o̟o̟b̟a̟r̟"̟
]̟
Expand All @@ -46,12 +46,34 @@ $ graphtage original.json modified.json

## Installation

Graphtage requires Python 3.10 or later. It is tested against Python 3.10 through 3.14.

```console
$ pip3 install graphtage
```

Installing the package puts two commands on your `PATH`: `graphtage`, the diff utility, and `graphtage-git-diff`, the
external diff driver described in [Git Integration](#git-integration).

To work on Graphtage itself, install the `dev` extra, which adds pytest, Ruff, and Sphinx:

```console
$ pip3 install 'graphtage[dev]'
```

## Command Line Usage

### Input File Types
Graphtage infers the type of each input file from its extension. To state the type instead of inferring it, use
`--from-<type>` for the first file and `--to-<type>` for the second. Both flags exist for every format Graphtage
supports: `csv`, `html`, `ini`, `json`, `json5`, `pickle`, `plist`, `toml`, `xml`, and `yaml`. For example, to read a
JSON document whose name does not end in `.json`:
```console
$ graphtage --from-json config.txt config.json
```
`--from-mime` and `--to-mime` do the same thing but take a MIME type rather than a format name, which matters for the
formats that Graphtage registers under more than one type. Run `graphtage --help` for the accepted values.

### Output Formatting
Graphtage performs an analysis on an intermediate representation of the trees that is divorced from the filetypes of the
input files. This means, for example, that you can diff a JSON file against a YAML file. Also, the output format can be
Expand All @@ -60,35 +82,39 @@ first input file. But one could, for example, diff two JSON files and format the
command-line arguments to specify these transformations, such as `--format`; please check the `--help` output for more
information.

Graphtage sorts the keys of every dictionary it reads, so the output orders keys alphabetically no matter how the
input files order them. The examples in this section all render the file `{"foo": [1, 2, 3], "bar": "baz"}` diffed
against itself.

By default, Graphtage pretty-prints its output with as many line breaks and indents as possible.
```json
{
"bar": "baz",
"foo": [
1,
2,
3
],
"bar": "baz"
]
}
```
Use the `--join-lists` or `-jl` option to suppress linebreaks after list items:
```json
{
"foo": [1, 2, 3],
"bar": "baz"
"bar": "baz",
"foo": [1,2,3]
}
```
Likewise, use the `--join-dict-items` or `-jd` option to suppress linebreaks after key/value pairs in a dict:
```json
{"foo": [
1,
2,
3
], "bar": "baz"}
{"bar": "baz","foo": [
1,
2,
3
]}
```
Use `--condensed` or `-j` to apply both of these options:
```json
{"foo": [1, 2, 3], "bar": "baz"}
{"bar": "baz","foo": [1,2,3]}
```

The `--only-edits` or `-e` option will print out a list of edits rather than applying them to the input file in place.
Expand All @@ -97,7 +123,8 @@ The `--edit-digest` or `-d` option is like `--only-edits` but prints a more conc
human-readable.

### Matching Options
By default, Graphtage tries to match all possible pairs of elements in a dictionary.
By default, Graphtage matches the values of key/value pairs that share a key, and tries to match all possible
pairs of the remaining elements.

Matching two dictionaries with each other is hard. Although computationally tractable, this can sometimes be onerous for
input files with huge dictionaries. Graphtage has three different strategies for matching dictionaries:
Expand All @@ -108,6 +135,9 @@ input files with huge dictionaries. Graphtage has three different strategies for
3. `--dict-strategy auto` (the default) will automatically match the values of any key-value pairs that have identical
keys and then use the `match` strategy for the remainder of key/value pairs.

`--dict-strategy` also has the short form `-ds`. The `--no-key-edits` or `-k` option is equivalent to
`--dict-strategy none`.

See [Pull Request #51](https://github.com/trailofbits/graphtage/pull/51) for some examples of how these strategies
affect output.

Expand All @@ -133,6 +163,75 @@ a comparison is large enough for this to matter.
`--ignore-list-order` cannot be combined with `--no-list-edits` or `--no-list-edits-when-same-length`, because those two
options apply only to ordered lists.

### Match Constraints
The `--match-unless` or `-u` and `--match-if` or `-m` options take an expression that decides whether Graphtage may
pair two nodes. Graphtage evaluates the expression once for each pair of nodes it considers, with `from` bound to the
node from the first file and `to` bound to the node from the second. `--match-unless` refuses the pair when the
expression is true; `--match-if` refuses it unless the expression is true. A refused pair is reported as a wholesale
replacement rather than compared element by element.

Expressions are parsed by the `graphtage.expressions` module rather than by `eval`. They support arithmetic and
comparison operators, indexing, attribute lookup, and calls to a fixed set of builtins such as `len` and `sorted`.

Say two files describe the same two servers, each identified by an `id`:
```console
$ echo Original: && cat servers.json && echo Modified: && cat servers.new.json
```
```json
Original:
{
"primary": {"id": 1, "host": "alpha"},
"replica": {"id": 2, "host": "beta"}
}
Modified:
{
"primary": {"id": 1, "host": "alphas"},
"replica": {"id": 3, "host": "gamma"}
}
```
By default Graphtage pairs the two `replica` records and reports the differences between them, even though they
describe different servers:
```console
$ graphtage servers.json servers.new.json
```
```json
{
"primary": {
"host": "alpha++s++",
"id": 1
},
"replica": {
"host": "~~bet~~++gamm++a",
"id": 2 -> 3
}
}
```
Refusing to pair records whose `id` differs reports the second record as replaced instead:
```console
$ graphtage --match-unless "from['id'] != to['id']" servers.json servers.new.json
```
```json
{
"primary": {
"host": "alpha++s++",
"id": 1
},
"replica": {
"host": "beta",
"id": 2
} -> {
"host": "gamma",
"id": 3
}
}
```
The two options differ in more than the sense of the test. `--match-unless` binds `from` and `to` to plain Python
values, and leaves a pair unconstrained when the expression raises an error, which is what makes the example above
work on the records without also constraining the strings and integers underneath them. `--match-if` binds `from` and
`to` to Graphtage node objects, and refuses a pair when the expression raises an error. Because the constraint applies
to every node in the tree, including the two roots, an expression that reads a key such as `from['id'] == to['id']`
refuses every pair and collapses the whole diff into one replacement. Prefer `--match-unless`.

### ANSI Color
By default, Graphtage will only use ANSI color in its output if it is run from a TTY. If, for example, you would like
to have Graphtage emit colorized output from a script or pipe, use the `--color` or `-c` argument. To disable color even
Expand All @@ -147,7 +246,27 @@ $ graphtage --html original.json modified.json > diff.html
### Status and Logging
By default, Graphtage prints status messages and a progress bar to STDERR. To suppress this, use the `--no-status`
option. To additionally suppress all but critical log messages, use `--quiet`. Fine-grained control of log messages is
via the `--log-level` option.
via the `--log-level` option. `--debug` is equivalent to `--log-level=DEBUG`, and `--quiet` is equivalent to
`--log-level=CRITICAL --no-status`.

### Version Information
`--version` or `-v` writes a line such as `Graphtage version 0.3.1` to STDERR. If you pass it without any input files,
Graphtage prints the version and exits; if you pass input files as well, it prints the version and then computes the
diff. `-dumpversion` writes the raw version to STDOUT and exits without reading any input.

### Exit Status
`graphtage` exits with one of three statuses, so a script can tell the three outcomes apart:

| Status | Meaning |
|--------|---------|
| `0` | The two inputs are semantically identical. |
| `1` | The two inputs differ. |
| `2` | Graphtage could not compute a diff, for example because a file did not parse or its type was not recognized. |

Interrupting Graphtage with `SIGINT` returns `-2`, which a POSIX shell reports as `254`.

Because a status of `1` means "the inputs differ" rather than "something went wrong", a CI job that treats any non-zero
status as a failure will fail on every diff Graphtage finds. Test for `2` to detect an error.

### Git Integration
Graphtage installs a `graphtage-git-diff` command that implements git's external diff interface, so `git diff` can
Expand Down Expand Up @@ -184,7 +303,7 @@ original.json
4,
++5++
],
"++z++~~b~~a++b++~~r~~": "testing",
"~~b~~++z++a~~r~~++b++": "testing",
++"woo": [
"foobar"
]++
Expand Down Expand Up @@ -268,4 +387,4 @@ This research was developed by [Trail of Bits](https://www.trailofbits.com/) wit
Advanced Research Projects Agency (DARPA) under the SafeDocs program as a subcontractor to [Galois](https://galois.com).
It is licensed under the [GNU Lesser General Public License v3.0](LICENSE).
[Contact us](mailto:opensource@trailofbits.com) if you're looking for an exception to the terms.
© 2020–2023, Trail of Bits.
© 2020–2026, Trail of Bits.
Loading