Support Snappy compression and a configurable GZIP level for make_examples output#1088
Support Snappy compression and a configurable GZIP level for make_examples output#1088tfenne wants to merge 2 commits into
Conversation
…amples make_examples has always written its tf.Example output as GZIP-compressed TFRecords at the zlib default level. On many-core machines the GZIP step is a meaningful slice of make_examples CPU, and when output throughput is provisioned generously the cheaper Snappy codec is the better trade. This adds that control. The compression codec is inferred from the examples file-name suffix, so the file name is the single source of truth for both writing and reading: a ".snappy" examples path selects Snappy, any other suffix keeps the historical GZIP behaviour. The C++ writer (nucleus::ExampleWriter, via the new CompressionTypeForPath) and every examples reader (call_variants, data_providers, show_examples, and the shape probe in dv_utils, via dv_utils.compression_type_for_examples_path) derive the codec the same way, so a file can never be written with one codec and read with another. A new --examples_compression_level flag (MakeExamplesOptions.examples_compression_level) exposes the GZIP level (-1 or 0..9; -1 = library default). It only affects GZIP output; the proto field is declared optional so an unset value is distinguishable from a deliberate level 0, and a level supplied alongside a Snappy path is ignored with a warning. The auxiliary make_examples outputs (call_variant_outputs, small_model_examples) are written through the Python TFRecord writer, which cannot emit Snappy, so they remain GZIP and their file names are forced to ".gz" to stay consistent with their bytes. Tested: C++ example_writer_test (codec detection, Snappy round-trip, GZIP level applied), dv_utils_test (suffix detection), make_examples_core_test (side-output renaming), and an end-to-end make_examples -> call_variants round-trip for both Snappy and GZIP examples.
tfenne
force-pushed
the
tf_examples-compression
branch
from
d9fe074 to
65cf535
Compare
The compression work added a suffix-inferred codec and --examples_compression_level to make_examples, but the run_deepvariant wrapper still hardcoded .gz example names and did not surface the level, so Snappy was reachable only by invoking make_examples directly. This wires both controls through the orchestration wrapper.
--examples_compression {GZIP,SNAPPY} selects the suffix of the single shared examples path, which feeds both make_examples (writer) and call_variants (reader) so they stay aligned; the auxiliary outputs (gVCF, call_variant_outputs, small_model) remain GZIP. --examples_compression_level forwards the GZIP level to make_examples for GZIP output only, and check_flags validates the range and warns if a level is supplied alongside SNAPPY.
|
Pushed an extra commit this morning that exposes the compression options in |
|
Hi @tfenne , Thanks for the PR! Since I believe you're already familiar with our process, I'll go ahead and start the review. As a reminder, because of the way our project is set up, we aren't able to merge GitHub PRs directly. If the changes look good, I will commit them, crediting your GitHub username and referencing this PR in the commit description. Please let me know if you have any concerns with this approach. -pichuan |
|
I re-benchmarked this PR's changes vs. r1.10 through the standard docker build process, running the baseline and modified versions in the resulting docker containers on the chr20 short-read WGS set on a baseline runtime: 108.6 The baseline is different from my other PR, I think largely run-to-run variance; the above is the average of multiple interleaved replicates. Also, the improvement shown above is only seen if a) the option is supplied to switch to snappy intermediate compression, and b) the storage has enough throughput to absorb the extra writes (the snappy compressed outputs are ~3x the size of the gzip level 6, but with the small model on, the absolute size is not so bad). |
| new_file = os.path.join(file_dir, new_file_base) | ||
| return new_file | ||
|
|
||
| @staticmethod |
There was a problem hiding this comment.
I'm changing this part to be:
@classmethod
def _as_gzip_path(cls, file_path):
| agree with the codec detection in | ||
| dv_utils.compression_type_for_examples_path and the C++ writer (both | ||
| lower-case the path before comparing). | ||
| """ |
There was a problem hiding this comment.
I'm adding more comment here:
Args:
file_path: The file path to potentially rewrite.
Returns:
The path with '.snappy' replaced by '.gz', or the original path.
| #include <vector> | ||
|
|
||
| #include "tensorflow/core/lib/io/record_reader.h" | ||
| #include "tensorflow/core/platform/env.h" |
There was a problem hiding this comment.
I'll add two more include here for tstring.h and types.h
|
|
||
| // High-level options that encapsulates all of the parameters needed to run | ||
| // DeepVariant end-to-end. | ||
| // Next ID: 102. |
pichuan
left a comment
pichuan
left a comment
There was a problem hiding this comment.
As I incorporate changes internally, I'm adding some minor edits.
Here are a few examples.
Going forward I might not mention all of them if they're minor ones like this. FYI.
| tensorflow::io::RecordReaderOptions opts = | ||
| tensorflow::io::RecordReaderOptions::CreateRecordReaderOptions("SNAPPY"); | ||
| tensorflow::io::RecordReader reader(file.get(), opts); | ||
| tensorflow::uint64 offset = 0; |
There was a problem hiding this comment.
I'm changing this to uint64_t
|
Hi @tfenne, An update on testing. I ran your PR with Good news first: On On the runtime improvement, my results so far are mixed:
I'm currently re-running the experiment on I noticed in your earlier comment you saw a ~5.2% improvement on chr20 WGS (c8a.4xlarge, 16 cores), and you mentioned that storage throughput matters — the benefit only appears if storage can absorb the ~3× larger intermediate writes. That's an interesting clue. In my setup (Google Batch VMs writing to local SSD), I'm not sure if storage throughput is the bottleneck. Do you have a sense of what other factors might explain why I'm not seeing a clear improvement at full-genome scale? -pichuan PR #1088 (SNAPPY vs Noop) Runtime Report —
|
| uid | noop (mean) | snappy (mean) | diff (sec) | diff (%) | t-stat | p-value | significant? |
|---|---|---|---|---|---|---|---|
| pacbio | 8915.50s (2h 28m) | 9096.95s (2h 31m) | +181.45s | +2.04% | -0.47 | 0.6566 | ❌ Not significant |
| wgs | 11245.00s (3h 7m) | 11728.52s (3h 15m) | +483.52s | +4.30% | -0.97 | 0.3708 | ❌ Not significant |
| hybrid | 13956.67s (3h 52m) | 14588.14s (4h 3m) | +631.47s | +4.52% | -5.21 | 0.0057 | |
| ont-r104 | 12868.34s (3h 34m) | 13541.70s (3h 45m) | +673.36s | +5.23% | -1.87 | 0.1333 | ❌ Not significant |
| exome | 481.07s (8m 1s) | 499.10s (8m 19s) | +18.03s | +3.75% | -1.08 | 0.3242 | ❌ Not significant |
| rnaseq | 1461.44s (24m 21s) | 1676.24s (27m 56s) | +214.80s | +14.70% | -3.39 | 0.0255 |
Caution
SNAPPY is slower on every single dataset. Not a single make_examples improvement. The direction is 6/6 regressions.
However, the SNAPPY run had dramatically higher runtime variance across all stages (up to 18× on ont-r104), including stages unrelated to example compression (
vcf_stats,postprocess_variants), suggesting VM-level instability rather than a SNAPPY-specific regression. See the raw tables below for the std_dev values.
VCF Output Verification
All md5sums are byte-identical:
| uid | md5sum (both runs) | match? |
|---|---|---|
| exome | a564eb2e51d9ba927e8d2c406b045cd6 |
✅ |
| hybrid-pacbio-illumina | eba191e9407907e660317ccbd49f420d |
✅ |
| ont-r104 | 46aeecf822164751ade92d05c64c61cd |
✅ |
| pacbio | e70fb870b3d4b8e82f3dab6dbc6aeee3 |
✅ |
| rnaseq | 1296173f2fd5b6f328a37130186021ba |
✅ |
| wgs | b0f1e53dbc9bc1b9b44dc653ae5f9cef |
✅ |
Note
The c3d-standard-16 md5sums differ from the n2-standard-96 md5sums — this is expected since num_shards differs (16 vs 96), and DeepVariant's sharding is not deterministic across different shard counts.
Summary
SNAPPY appears slower on every dataset on c3d-standard-16, but this result is almost certainly unreliable due to:
- Catastrophic variance in the SNAPPY run (up to 18× higher than noop)
- The slowdown affects all stages equally — including stages that don't touch example compression at all (
vcf_stats,postprocess_variants) - The pattern is consistent with the SNAPPY VMs landing on degraded hardware or hitting severe noisy-neighbor effects
This data cannot be used to conclude that SNAPPY is harmful. The runs would need to be repeated, ideally with more trials or with the VMs pinned to specific hardware to reduce placement variance.
Click to view raw snappy-c3d16 runtime table
| uid | sample | stage | mean_runtime | std_runtime | n_trials | mean_hruntime |
|---|---|---|---|---|---|---|
| exome | HG003 | make_examples | 499.1 | 33.992 | 5 | 8m 19s |
| exome | HG003 | call_variants | 188.28 | 34.675 | 5 | 3m 8s |
| exome | HG003 | postprocess_variants | 32.81 | 2.989 | 5 | 32s |
| exome | HG003 | vcf_stats | 6.19 | 0.589 | 5 | 6s |
| exome | HG003 | total | 720.19 | 71.534 | 5 | 12m 0s |
| hybrid-pacbio-illumina | HG003 | make_examples | 14588.14 | 267.518 | 5 | 4h 3m 8s |
| hybrid-pacbio-illumina | HG003 | call_variants | 24897.76 | 382.958 | 5 | 6h 54m 57s |
| hybrid-pacbio-illumina | HG003 | postprocess_variants | 287.11 | 11.679 | 5 | 4m 47s |
| hybrid-pacbio-illumina | HG003 | vcf_stats | 240.92 | 1.755 | 5 | 4m 0s |
| hybrid-pacbio-illumina | HG003 | total | 39773.02 | 120.736 | 5 | 11h 2m 53s |
| ont-r104 | HG003 | make_examples | 13541.7 | 800.865 | 5 | 3h 45m 41s |
| ont-r104 | HG003 | call_variants | 12762.13 | 604.424 | 5 | 3h 32m 42s |
| ont-r104 | HG003 | postprocess_variants | 1122.57 | 72.831 | 5 | 18m 42s |
| ont-r104 | HG003 | vcf_stats | 376.11 | 9.419 | 5 | 6m 16s |
| ont-r104 | HG003 | total | 27426.4 | 1326.791 | 5 | 7h 37m 6s |
| pacbio | HG003 | make_examples | 9096.95 | 780.255 | 5 | 2h 31m 36s |
| pacbio | HG003 | call_variants | 6656.72 | 417.249 | 5 | 1h 50m 56s |
| pacbio | HG003 | postprocess_variants | 551.59 | 30.105 | 5 | 9m 11s |
| pacbio | HG003 | vcf_stats | 283.55 | 4.095 | 5 | 4m 43s |
| pacbio | HG003 | total | 16305.27 | 1226.954 | 5 | 4h 31m 45s |
| rnaseq | HG005 | make_examples | 1676.24 | 139.908 | 5 | 27m 56s |
| rnaseq | HG005 | call_variants | 116.3 | 11.928 | 5 | 1m 56s |
| rnaseq | HG005 | postprocess_variants | 238.74 | 19.31 | 5 | 3m 58s |
| rnaseq | HG005 | vcf_stats | 5.69 | 0.549 | 5 | 5s |
| rnaseq | HG005 | total | 2031.27 | 170.952 | 5 | 33m 51s |
| wgs | HG003 | make_examples | 11728.52 | 1008.781 | 5 | 3h 15m 28s |
| wgs | HG003 | call_variants | 6156.89 | 303.598 | 5 | 1h 42m 36s |
| wgs | HG003 | postprocess_variants | 414.25 | 15.42 | 5 | 6m 54s |
| wgs | HG003 | vcf_stats | 253.21 | 1.886 | 5 | 4m 13s |
| wgs | HG003 | total | 18299.66 | 1327.511 | 5 | 5h 4m 59s |
Click to view raw noop-c3d16 runtime table
| uid | sample | stage | mean_runtime | std_runtime | n_trials | mean_hruntime |
|---|---|---|---|---|---|---|
| exome | HG003 | make_examples | 481.07 | 15.308 | 5 | 8m 1s |
| exome | HG003 | call_variants | 156.24 | 7.811 | 5 | 2m 36s |
| exome | HG003 | postprocess_variants | 30.02 | 0.75 | 5 | 30s |
| exome | HG003 | vcf_stats | 5.66 | 0.114 | 5 | 5s |
| exome | HG003 | total | 667.34 | 23.065 | 5 | 11m 7s |
| hybrid-pacbio-illumina | HG003 | make_examples | 13956.67 | 41.849 | 5 | 3h 52m 36s |
| hybrid-pacbio-illumina | HG003 | call_variants | 23576.81 | 50.241 | 5 | 6h 32m 56s |
| hybrid-pacbio-illumina | HG003 | postprocess_variants | 291.82 | 4.216 | 5 | 4m 51s |
| hybrid-pacbio-illumina | HG003 | vcf_stats | 241.65 | 2.887 | 5 | 4m 1s |
| hybrid-pacbio-illumina | HG003 | total | 37825.3 | 73.059 | 5 | 10h 30m 25s |
| ont-r104 | HG003 | make_examples | 12868.34 | 74.442 | 5 | 3h 34m 28s |
| ont-r104 | HG003 | call_variants | 11858.91 | 33.492 | 5 | 3h 17m 38s |
| ont-r104 | HG003 | postprocess_variants | 1012.97 | 6.896 | 5 | 16m 52s |
| ont-r104 | HG003 | vcf_stats | 356.39 | 1.438 | 5 | 5m 56s |
| ont-r104 | HG003 | total | 25740.22 | 105.546 | 5 | 7h 9m 0s |
| pacbio | HG003 | make_examples | 8915.5 | 395.291 | 5 | 2h 28m 35s |
| pacbio | HG003 | call_variants | 6326.91 | 191.306 | 5 | 1h 45m 26s |
| pacbio | HG003 | postprocess_variants | 521.59 | 8.425 | 5 | 8m 41s |
| pacbio | HG003 | vcf_stats | 278.0 | 2.267 | 5 | 4m 38s |
| pacbio | HG003 | total | 15763.99 | 571.737 | 5 | 4h 22m 43s |
| rnaseq | HG005 | make_examples | 1461.44 | 22.379 | 5 | 24m 21s |
| rnaseq | HG005 | call_variants | 96.44 | 1.544 | 5 | 1m 36s |
| rnaseq | HG005 | postprocess_variants | 205.45 | 1.356 | 5 | 3m 25s |
| rnaseq | HG005 | vcf_stats | 4.97 | 0.069 | 5 | 4s |
| rnaseq | HG005 | total | 1763.33 | 22.935 | 5 | 29m 23s |
| wgs | HG003 | make_examples | 11245.0 | 468.325 | 5 | 3h 7m 25s |
| wgs | HG003 | call_variants | 5924.95 | 156.029 | 5 | 1h 38m 44s |
| wgs | HG003 | postprocess_variants | 392.92 | 2.839 | 5 | 6m 32s |
| wgs | HG003 | vcf_stats | 255.12 | 6.976 | 5 | 4m 15s |
| wgs | HG003 | total | 17562.87 | 612.448 | 5 | 4h 52m 42s |
2 participants
Summary
make_exampleshas always written itstf.Exampleoutput as GZIP-compressed TFRecords at zlib's default level, with the codec hardcoded on both the write and read sides. On many-core machines the GZIP step is a meaningful slice ofmake_examplesCPU time, and when output throughput is provisioned generously the cheaper Snappy codec is often the better trade. This PR makes the examples codec selectable and exposes the GZIP level. Output compression type is keyed off filenames, compression level by a new option.Going from gzip level 6 to level 1 on chr20 saved ~7% of make_examples runtime, at the cost of about doubling the output size. Going gzip level 6 to snappy saved 10% at the cost of about 4x more storage. The tradeoff is user selectable based on available storage, and is much more modest with the small-model handling a large fraction of calls now.
Design: the file name is the single source of truth
The compression codec is inferred from the examples file-name suffix rather than from a separate flag:
*.snappyexamples path selects Snappy;The C++ writer (
nucleus::ExampleWriter, via the newCompressionTypeForPath) and every examples reader (call_variants,data_providers,show_examples, and the shape probe indv_utils, via the newdv_utils.compression_type_for_examples_path) derive the codec the same way. Detection is case-insensitive and handles the usual sharded (@N,-ddddd-of-ddddd) and comma-separated path forms.A new
--examples_compression_levelflag (protoMakeExamplesOptions.examples_compression_level) exposes the GZIP level (-1or0..9;-1= library default). It only affects GZIP output. The proto field is declaredoptionalso an unset value is distinguishable from a deliberate level 0 (otherwise a non-flag caller would silently get level 0 / no deflate), and a level supplied alongside a.snappypath is ignored with a warning (Snappy has no levels).Backward compatibility
Default behaviour is unchanged: with no
.snappysuffix and no level flag, output is GZIP at the library default level, exactly as before. The auxiliarymake_examplesoutputs (call_variant_outputs,small_model_examples) are written through the Python TFRecord writer, which cannot emit Snappy, so they remain GZIP — and their file names are forced to.gzeven when the main examples are Snappy, so each file's name matches its actual bytes.What changed
third_party/nucleus/io/example_writer.{h,cc}: codec inferred from the path;compression_levelplumbed intoRecordWriterOptions.zlib_options; new publicCompressionTypeForPath.deepvariant/protos/deepvariant.proto:optional int32 examples_compression_level.deepvariant/make_examples_options.py:--examples_compression_levelflag + range validator.deepvariant/make_examples_native.cc: passes the level (library default when unset).deepvariant/dv_utils.py,call_variants.py,data_providers.py,show_examples.py: readers autodetect the codec from the suffix.deepvariant/make_examples_core.py: side outputs forced to.gz.Testing
third_party/nucleus/io:example_writer_test— codec detection, Snappy write/read round-trip, and GZIP level actually applied (level 0 file larger than level 9).deepvariant:dv_utils_test— suffix detection across casing and sharding forms.deepvariant:make_examples_core_test— side-output.snappy→.gzrenaming (incl. uppercase).make_examples→call_variantsround-trip on a chr20 region for both Snappy and GZIP examples (and an explicit GZIP level), confirmingcall_variantsreads Snappy output and that an out-of-range level is rejected.