(prototype) SimCLR v2 refactor - #2031
Closed
gabrielfruet wants to merge 9 commits into
Closed
Conversation
Contributor
Author
|
Split into a stack of seven, one thing each, since this was too big to read as one diff:
The |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
SimCLR existed four times here and nothing compared the copies: the two shipped heads differed by 4,922,112 parameters, split and fused forwards gave gradients at cosine similarity 0.0837, and both examples took
NTXentLoss()'s default 0.5 while the benchmark passed 0.1.First vertical slice of the 2.0 design, SimCLR only. The other 29 methods are untouched.
lightly/data/sample.py: a batch is aSampleof typedViews, one generic collate, pluslegacy_collate.lightly/backbones/:BackboneandDenseBackboneprotocols, a torchvision ResNet adapter, a conformance suite.lightly/nn/axes.py:encode(..., group_by="shape"), so BatchNorm sees 2N.lightly/functional/ntxent.pyandlightly/optim/: the equation and the optimiser pieces get homes; old paths still import.examples/simclr.pyandbenchmarks/simclr/replace the four old copies.tests/test_simclr_agrees.pycompares the method, not the run: blocks, view contract, fused forward, head width, and that each side states its temperature. Batch size, backbone and temperature values are free to differ, and do.Breaking:
SimCLRTransform,MoCoV1/V2TransformandDenseCLTransformreturnlist[View]. Readviews[0].data, or passcollate_fn=legacy_collate.Testing: format-check, mypy and pytest clean bar one pre-existing DCL distributed failure; the example runs an epoch;
--fast-dev-runlogstrain_lossand both probes; both gate mutations fail as intended.