Use augur subsample

[victorlin]

Description of proposed changes

The previous subsampling implementation was fixed to a two-sample recent+background split with some hardcoded parameters. Replacing it with augur subsample allows for more flexible configuration.

In Snakemake, implementation is mostly copied from pathogen repos that have switched over to augur subsample. One notable difference is that the combine_samples rule must stay to handle output from the enrich_antibody_escape rule.

This is a breaking change and the old configuration will no longer work.

Related issue(s)

Closes #101

Checklist

• Update changelog
• Draft necessary changes on NW-PaGe fork: 0582db5
• Manually tested that validation fails with proximal sample entry
• Merge Configuration schema and validation #139 and Add schema for unaligned sequences augur#2038
• Checks pass

old implementations

• YAML anchors and aliases: b0d6728
• matrix: d24a8d1
• dataset-first glob syntax 1: 5741f0f
• option-first glob syntax 1: bff7e53
• option-first glob syntax 2: 1df4e05
• dataset-first glob syntax 2: 397850d

[victorlin]

I'll wait for a decision in nextstrain/public#27 before continuing here.

[joverlee521]

Thank you for continuing to push through this! I've only added small comments for docs/changelog edits.

[victorlin]

c33cf92 is a significant change, basically option (1) from nextstrain/public#23.

[rneher]

Thanks for pushing this, Victor. I am a bit on the fence here. What bugs me most about this is that if one now wants to change something in the workflow, one has to trace the config that ends up as input in the rules through even more layers of input functions and config generators. I am wondering whether part of this problem could be avoided if instead of using write_subsample_config to pull out a specific section of the main file, we just use the generate_default_config to produce that section and place it as subsample_config into the corresponding results directory? This would also avoid having to commit the generated config and the resulting duplication parameters. The config could then specify custom subsampling configs if desired and fall back on the autogenerated ones.

[victorlin]

Users currently rely on values in the default config file for inheritance and documentation. There are better ways to provide documentation, but values in the default config file is necessary for the inheritance aspect. So I don't think we can save the generated default subsample values directly to results without breaking config inheritance.

That said, I take your point on increased complexity in how config is passed around, though for this PR I think that comes more from the switch to augur subsample – the pattern of providing config via an exported YAML file – than from the switch to generating default config via script.

I can drop c33cf92 if you/others are ok with hand-editing the default config file, similar to how things are done with ncov's builds.yaml.