Implement masking - #225
Conversation
Add --mask-dir option to specify a directory containing binary mask images (0=exclude, 1=include). Masks are matched to images by filename stem and applied during loss computation (L1 and SSIM) so only foreground pixels contribute to training. Features: - Binary mask loading with automatic resizing to match image dimensions - Mask pyramid caching (like image pyramids) using nearest-neighbor interpolation - Masked L1 loss: weighted mean over foreground pixels only - Masked SSIM: post-masking of SSIM map before averaging - Validation loss also respects masks
|
Thanks @JorisGoosen. Can you test with https://drive.google.com/file/d/1XzRI7kKKx3hvql7KKhvaIWGRwebltm9N/view?usp=sharing and post some screenshots? |
|
After 9000 or so iterations: Ill let it run a bit longer now. |
|
After 40000: |
|
Thanks Joris. You can see the rendering is no good. Compare the result with a run without masks and you'll see the difference. But even just looking at the renderer should give a clue that this is not the right approach: 
|
|
Well, the thing is, the part outside of the mask is not being updated (properly) now. Maybe adding a very small loss to parts that are consistently masked might help? Ive been running with a less tight mask and I get better results then. Edit: definitely need to tweak this, but just have to find what to tweak. 
|
So with a bit of a wide mask its already almost usable for me. |
|
Right, one cannot simply zero out the loss around the mask. I would start by looking at how other implementations do it for possible approaches (e.g. https://github.com/MrNeRF/LichtFeld-Studio/blob/master/src/training/trainer.cpp#L246) I think depending on the approach you'll need to track the outAlpha in the backward step (https://github.com/nerfstudio-project/gsplat/pull/70/changes) as well, as I currently always initialize it to zero: |
e278076 to
6047beb
Compare
|
So last week I did look into the code at lichtfeld and seem to have a better idea on how to get this used. |
|
(Haven't forgotten about this, its just that im in the middle of moving houses) |
Quite untested and probably doesn't work, but it does seem to compile. To be continued
|
Hi Joris, are you still working on this? |
|
Not actively. I got a bit stuck with forwarding the gradient of the masked pixels properly through the pipeline. |
|
|
||
| //maskPenalty = torch::zeros_like(expandedMask) | ||
|
|
||
| const torch::Tensor bg_mask = 1.0 - expandedMask, | ||
| penalty_weights = bg_mask.pow(__mask_opacity_penalty_power), | ||
| penalty = (expandedMask * penalty_weights).mean() * __mask_opacity_penalty_weight; // de penalty wordt bepaald aan de hand penalty gewicht * penalty_weights | ||
|
|
||
| const float inv_pixels = __mask_opacity_penalty_weight / static_cast<float>(expandedMask.numel()); //deel de opacity weight door het aantal elementen in de alpha lijst | ||
| maskPenalty = penalty_weights * inv_pixels; //Grad alpha wordt dan de 2d penalty kaart teruggebracht tot een | ||
| //loss = loss + penalty; | ||
|
|
||
| // hier moet ik alpha in krijgen zodat ik met de inverse daarvan hier vervolgens het gradient mee kan maken |
There was a problem hiding this comment.
Here is some code I based on lichtenfelds implementation. The alpha needs to be propagated here for things to progress.
|
I think the mainpart missing here right now is the backwardpass using the maskpenalty properly. Id need to reread the code of at least lichtenfeld again to have a better and fresh idea of what inspired me though. And while I do plan to continue on this when I get the chance that is problem not yet this summer. But who knows ^^ |
|
Closing in favor of #245 |
4 participants
The code in #220 actually looked like a good base to implement 2d masking, which I wanted opensplat to support for a while.
Ive fixed some loose ends in it and changed the application of the mask.
Implements #35
Ive tested this with some datasets of mine, any suggestion on other things to test would be welcome.
So far it seems to be working fine.
Any suggestions for possible datasets to test things on?