Implement --config for various commands

[victorlin]

Description of proposed changes

Adds the --config option to align, ancestral, frequencies, mask, traits, translate, and tree using patterns from #2040.

Docs preview: https://nextstrain--2041.org.readthedocs.build/projects/augur/en/2041/usage/cli/ancestral.html#configuration

Review threads

• add generic get_referenced_files function? yes
• export v2: YAML JSON, and CLI options

Checklist

• Automated checks pass
• Check if you need to add a changelog message
• Check if you need to add tests
• Check if you need to update docs
• PR nextstrain.org for schema routes

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 74.56%. Comparing base (895e092) to head (130695c).

Additional details and impacted files

@@                     Coverage Diff                     @@
##           victorlin/refine-config    #2041      +/-   ##
===========================================================
+ Coverage                    74.54%   74.56%   +0.01%     
===========================================================
  Files                           86       86              
  Lines                        10874    10882       +8     
  Branches                      2123     2123              
===========================================================
+ Hits                          8106     8114       +8     
  Misses                        2373     2373              
  Partials                       395      395              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[joverlee521]

Looking at the addition of color_by_metadata in nextstrain/measles#148 and this PR, I think the --config option for export really highlights the need for cleaning up + documenting the interactions between the Auspice config JSON and the CLI/YAML options. However, that's a bigger project that I don't want to block this work...

For now, maybe there's a way to add the description from the argument group to the schema (with minor edits to refer to config YAML options):

augur/augur/export_v2.py

Lines 912 to 915 in 0d28749

	description="These control the display settings for auspice. \
	Config JSON(s) allow customisation of all available options whereas individual command line arguments are more limited but great to get started. \
	Supplying both is fine too, command line args will overrule what is set in the config file. \
	Multiple JSONs will be merged together, and lists present in multiple configs will be merged by extending the original list."

[jameshadfield]

The auspice-config JSON is (conceptually!) the config YAML we're now introducing elsewhere, and in the long run we could/should probably replace the auspice-config JSON with a --config YAML. Pragmatically, here are the CLI args not currently exposed in the JSON:

Files:

• tree, metadata, output, node-data # part of the typical snakemake pipeline
• description markdown, warning text # should be in auspice-config json:
• colors TSV, auspice-config JSON

Misc:

• minification
• validation mode
• metadata strain column / delimeters

If we put the description markdown & warning text files into the auspice-config JSON, then we're down to a handful of possible options we (the workflow) want to expose as user-config. I think these are feasible to do the old-fashioned way. Which is to say, short term I don't think we need a config YAML at the moment and long term the config YAML can replace the auspice config JSON

[victorlin]

Because the JSON is nested, a full switch from JSON to YAML is not possible with the ConfigArgParse implementation, just like how augur subsample --config must be handled separately. I think we all agree the full switch is the best solution. We have the following short term options:

1. Leave export as CLI + JSON only, with some CLI options exposed through YAML on the workflow level.
2. Expose all CLI options through YAML with --config. JSON is still the only option for data_provenance, etc.

f9e1672 is (2). It allows for more flexibility, and our workflow config is a half-baked implementation of this (nextstrain/measles#148 baked it a little more). But it is also confusing to have both --config and --auspice-config under augur export v2 --help.
(1) makes export v2 a loner when everything else has been moved over.

Maybe it's reasonable to block the --config work on the full switch from JSON to YAML for export v2.

[joverlee521]

Because the JSON is nested, a full switch from JSON to YAML is not possible with the ConfigArgParse implementation, just like how augur subsample --config must be handled separately.

Oh hmm, does this mean we would also remove the CLI options for params that are provided via the YAML?

---

An alternative to (1) (similar to what James is proposing), is making sure all Auspice configs are available via the JSON. Then in the YAML exclude all display configuration options that are not auspice_config, i.e. disabling CLI overrides in workflows that use --config. In practice, this would probably look like replacing nextstrain/measles#148 with multiple layered JSONs that are provided via auspice_config.

[victorlin]

Oh hmm, does this mean we would also remove the CLI options for params that are provided via the YAML?

We could keep them. It won't be as simple as with ConfigArgParse, but we could apply the overrides in a similar fashion as the existing implementation.

Alternatively, we could implement something like Snakemake's --config to allow generic overrides via CLI, but that would be a new pattern and we'd need another name for the option.