Reproducibility issue

[valeryozenne]

Hello,

First, thank you for this project and for making all the nnUNet tools available. I'd like to contact you because I'm having a few (technical) problems for reproducing the results.
It's probably a mistake on my own but I didn't find the solution.

Thanks in advance for your help.

Valéry

Step I followed

Reading the documentation, here are the steps I followed:

creation of a Docker images

compatible with the code (with two layers: one for inference, one for training) (the docker file is enclosed)

docker build --target particle_train -t particleseg3d-train .
docker build --target particle_test -t particleseg3d-test .

check that libs are working:

start docker container

docker run --gpus all -v /yourmountingpoint/nnUNet_v1/:/opt/database/nnUNet_v1/  --shm-size 10Gb  -it  particleseg3d-train bash
docker run --gpus all -v /yourmountingpoint/nnUNet_v1/:/opt/database/nnUNet_v1/  --shm-size 10Gb  -it  particleseg3d-test bash

basic check

root@68896d0ddc93:/# python3 -c "import torch; print('PyTorch version:', torch.__version__)"
PyTorch version: 1.11.0+cu113
root@68896d0ddc93:/# python3 -c "import torch; print('cuDNN version:', torch.backends.cudnn.version())"
cuDNN version: 8200
root@68896d0ddc93:/# python3 -c "import torch; print('CUDA available:', torch.cuda.is_available())"
CUDA available: True

Probleme 1 : inference

[ok] dowload the data and model and put everything at the right location

[ok] convert the nifti data to zarr

ps3d_nifti2zarr -i /opt/database/nnUNet_v1/nnUNet_raw_data_base/Task001_Particles/Patches/test/images/Ore1-Comp3-Concentrate_patch_1.nii.gz -o /opt/database/nn
UNet_v1/nnUNet_raw_data_base/Task001_Particles/Patches/testzarr/images/Ore1-Comp3-Concentrate_patch_1.zarr

[not ok] run the inference

ps3d_inference -i /opt/nnUNet_v1/nnUNet_raw_data_base/Task001_Particles/Patches/testzarr/ -o /opt/TestInference/ -m /opt/nnUNet_v1/nnUNet_trained_models/Task310_particle_seg/ -n Ore1-Comp3-Concentrate_patch_1

return

ps3d_inference -i /opt/database/nnUNet_v1/nnUNet_raw_data_base/Task001_Particles/Patches/testzarr/ -o /opt/TestInference/ -m /opt/database/nnUNet_v1/nnUNet_trained_models/Task310_particle_seg/ -n Ore1-Comp3-Concentrate_patch_1
/usr/local/lib/python3.8/dist-packages/pytorch_lightning/trainer/connectors/accelerator_connector.py:441: LightningDeprecationWarning: Setting `Trainer(gpus=1)` is deprecated in v1.7 and will be removed in v2.0. Please use `Trainer(accelerator='gpu', devices=1)` instead.
  rank_zero_deprecation(
Using 16bit native Automatic Mixed Precision (AMP)
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
Samples:  ['Ore1-Comp3-Concentrate_patch_1']
Inference Query:   0%|                                                                                                                                                       | 0/1 [00:00<?, ?it/s]Starting inference of sample:  Ore1-Comp3-Concentrate_patch_1
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
Predicting DataLoader 0:  26%|██████████████████████████████▌                                                                                       | 14/54 [02:30<07:11, 10.78s/it]Predicting DataLoader 0: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 54/54 [09:49<00:00, 10.92s/it]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 286/286 [00:00<00:00, 1228.60it/s]
Inference Query:   0%|                                                                                                                                                       | 0/1 [10:03<?, ?it/s]
Traceback (most recent call last):
  File "/usr/local/bin/ps3d_inference", line 8, in <module>
    sys.exit(main())
  File "/opt/code/ParticleSeg3D/particleseg3d/inference/inference.py", line 527, in main
    predict_cases(args.input, args.output, args.name, trainer, model, config, args.target_particle_size, args.target_spacing, args.batch_size, args.processes, args.min_rel_particle_size, args.zscore)
  File "/opt/code/ParticleSeg3D/particleseg3d/inference/inference.py", line 93, in predict_cases
    predict_case(image_dir, save_dir, name, metadata_filepath, zscore, trainer, model, config, target_particle_size, target_spacing, processes, min_rel_particle_size, batch_size)
  File "/opt/code/ParticleSeg3D/particleseg3d/inference/inference.py", line 143, in predict_case
    predict(load_filepath, pred_softmax_filepath, pred_border_core_filepath, pred_border_core_tmp_filepath, pred_instance_filepath, target_spacing, target_particle_size_in_mm, target_patch_size_in_pixel,
  File "/opt/code/ParticleSeg3D/particleseg3d/inference/inference.py", line 232, in predict
    instance_pred = filter_small_particles(instance_pred, min_rel_particle_size)
  File "/opt/code/ParticleSeg3D/particleseg3d/inference/inference.py", line 395, in filter_small_particles
    particle_voxels = cc3d.statistics(instance_pred)["voxel_counts"]
  File "cc3d/fastcc3d.pyx", line 750, in fastcc3d.statistics
  File "cc3d/fastcc3d.pyx", line 831, in fastcc3d.__pyx_fused_cpdef
TypeError: Function call with ambiguous argument types

Probleme 2 : training

[ok] download the data and model and put everything at the right location

[ok] preprocessing

ps3d_train_preprocess -i /opt/nnUNet_v1/nnUNet_raw_data_base/Task001_Particles/Patches/train/ -o /opt/nnUNet_v1/nnUNet_preprocessed/ -z 0.12345 0.6789

training

I did install the modified nnunet_v1 using

pip install git+https://github.com/MIC-DKFZ/nnUNet.git@ParticleSeg3D

I'm a bit familiar with the nnUNet_v2 version but there are three binary for the training:

which one should I used?

nnUNet_train
nnUNet_train_DP
nnUNet_train_DDP

[valeryozenne]

With the attachment, I had to modify the name and extension.

ParticleDockerfile.txt

[Karol-G]

Hey,

Regarding inference, it seems like you are using an incompatible version of connected-components-3d maybe. Can you try with this version instead connected-components-3d==3.11.0?
Regarding training, you have to use the command nnUNet_train. But if you are unsure how to use nnU-Net I would recommend to read the nnU-Net readme in their repo.

Best,
Karol

[valeryozenne]

Hello Karol,

Thanks a lot for your message. I give a try and get another error message:

root@0a9c797ef3aa:/# ps3d_inference -i /opt/nnUNet_v1/nnUNet_raw_data_base/Task001_Particles/Patches/testzarr/ -o /opt/TestInference/ -m /opt/nnUNet_v1/nnUNet_trained_models/Task310_particle_seg/ -n Ore1-Comp3-Concentrate_patch_1
Traceback (most recent call last):
  File "/usr/local/bin/ps3d_inference", line 8, in <module>
    sys.exit(main())
  File "/opt/code/ParticleSeg3D/particleseg3d/inference/inference.py", line 526, in main
    trainer, model, config = setup_model(args.model, args.fold)
  File "/opt/code/ParticleSeg3D/particleseg3d/inference/inference.py", line 41, in setup_model
    with open(join(model_dir, trainer, "plans.pkl"), 'rb') as handle:
FileNotFoundError: [Errno 2] No such file or directory: '/opt/nnUNet_v1/nnUNet_trained_models/Task310_particle_seg/nnUNetTrainerV2_slimDA5_touchV5__nnUNetPlansv2.1/plans.pkl'
root@0a9c797ef3aa:/# ls /opt/database/nnUNet_v1/nnUNet^C
root@0a9c797ef3aa:/# ps3d_inference -i /opt/database/nnUNet_v1/nnUNet_raw_data_base/Task001_Particles/Patches/testzarr/ -o /opt/TestInference/ -m /opt/database/nnUNet_v1/nnUNet_tra
ined_models/Task310_particle_seg/ -n Ore1-Comp3-Concentrate_patch_1
TestInference/  code/           database/       layer_info.txt  nvidia/         
root@0a9c797ef3aa:/# ps3d_inference -i /opt/database/nnUNet_v1/nnUNet_raw_data_base/Task001_Particles/Patches/testzarr/ -o /opt/TestInference/ -m /opt/database/nnUNet_v1/nnUNet_tra
ined_models/Task310_particle_seg/ -n Ore1-Comp3-Concentrate_patch_1
/usr/local/lib/python3.8/dist-packages/pytorch_lightning/trainer/connectors/accelerator_connector.py:441: LightningDeprecationWarning: Setting `Trainer(gpus=1)` is deprecated in v1.7 and will be removed in v2.0. Please use `Trainer(accelerator='gpu', devices=1)` instead.
  rank_zero_deprecation(
Using 16bit native Automatic Mixed Precision (AMP)
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
Samples:  ['Ore1-Comp3-Concentrate_patch_1']
Inference Query:   0%|                                                                                                                                        | 0/1 [00:00<?, ?it/s]Starting inference of sample:  Ore1-Comp3-Concentrate_patch_1
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
Predicting DataLoader 0: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 54/54 [09:59<00:00, 11.11s/it]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 286/286 [00:00<00:00, 1314.27it/s]
Inference Query:   0%|                                                                                                                                        | 0/1 [10:23<?, ?it/s]
Traceback (most recent call last):
  File "/usr/local/bin/ps3d_inference", line 8, in <module>
    sys.exit(main())
  File "/opt/code/ParticleSeg3D/particleseg3d/inference/inference.py", line 527, in main
    predict_cases(args.input, args.output, args.name, trainer, model, config, args.target_particle_size, args.target_spacing, args.batch_size, args.processes, args.min_rel_particle_size, args.zscore)
  File "/opt/code/ParticleSeg3D/particleseg3d/inference/inference.py", line 93, in predict_cases
    predict_case(image_dir, save_dir, name, metadata_filepath, zscore, trainer, model, config, target_particle_size, target_spacing, processes, min_rel_particle_size, batch_size)
  File "/opt/code/ParticleSeg3D/particleseg3d/inference/inference.py", line 143, in predict_case
    predict(load_filepath, pred_softmax_filepath, pred_border_core_filepath, pred_border_core_tmp_filepath, pred_instance_filepath, target_spacing, target_particle_size_in_mm, target_patch_size_in_pixel,
  File "/opt/code/ParticleSeg3D/particleseg3d/inference/inference.py", line 232, in predict
    instance_pred = filter_small_particles(instance_pred, min_rel_particle_size)
  File "/opt/code/ParticleSeg3D/particleseg3d/inference/inference.py", line 395, in filter_small_particles
    particle_voxels = cc3d.statistics(instance_pred)["voxel_counts"]
  File "cc3d.pyx", line 619, in cc3d.statistics
  File "cc3d.pyx", line 625, in cc3d.__pyx_fused_cpdef
TypeError: No matching signature found

Google mentioned it might be related to a datatype issue. But I haven't yet found the solution.
I haven't had much time over the last few days but I'm going to save the variable instance_pred used in particle_voxels = cc3d.statistics(instance_pred)["voxel_counts"] to see if there's a problem upstream.

Thanks in advance for your help, I will let you know.

Valéry