Lapras gpu1 drop

[jhgoh]

gpu-brun 테스트중 lapras의 GPU1이 드랍됨.

[jhgoh]

리붓 후 재 테스트중에도 GPU1이 drop됨.

[slowmoyang]

❯ hostname
lapras.lo

❯ date
Mon Aug 31 10:06:39 AM KST 2026

❯ nvidia-smi -i 1
Unable to determine the device handle for GPU1: 0000:21:00.0: Unknown Error
No devices were found

❯ uv run python -c "import torch;print(torch.__version__);print(torch.torch.cuda.device_count())"
2.13.0+cu130
/users/hep/slowmoyang/work/test-gpu/.venv/lib/python3.12/site-packages/torch/cuda/__init__.py:1112: UserWarning: Can't initialize NVML
  raw_cnt = _raw_device_count_nvml()
/users/hep/slowmoyang/work/test-gpu/.venv/lib/python3.12/site-packages/torch/cuda/__init__.py:1170: UserWarning: CUDA initialization: CUDA unknown error - this may be due to an incorrectly set up environment, e.g. changing env variable CUDA_VISIBLE_DEVICES after program start. Setting the available devices to be zero. (Triggered internally at /__w/pytorch/pytorch/c10/cuda/CUDAFunctions.cpp:119.)
  r = torch._C._cuda_getDeviceCount() if nvml_count < 0 else nvml_count
0

❯ CUDA_VISIBLE_DEVICES=0,2,3 uv run python -c "import torch;print(torch.__version__);print(torch.torch.cuda.device_count())"
2.13.0+cu130
/users/hep/slowmoyang/work/test-gpu/.venv/lib/python3.12/site-packages/torch/cuda/__init__.py:1112: UserWarning: Can't initialize NVML
  raw_cnt = _raw_device_count_nvml()
/users/hep/slowmoyang/work/test-gpu/.venv/lib/python3.12/site-packages/torch/cuda/__init__.py:1170: UserWarning: CUDA initialization: CUDA unknown error - this may be due to an incorrectly set up environment, e.g. changing env variable CUDA_VISIBLE_DEVICES after program start. Setting the available devices to be zero. (Triggered internally at /__w/pytorch/pytorch/c10/cuda/CUDAFunctions.cpp:119.)
  r = torch._C._cuda_getDeviceCount() if nvml_count < 0 else nvml_count
0

❯ CUDA_VISIBLE_DEVICES=0 uv run python -c "import torch;print(torch.__version__);print(torch.torch.cuda.device_count())"
2.13.0+cu130
/users/hep/slowmoyang/work/test-gpu/.venv/lib/python3.12/site-packages/torch/cuda/__init__.py:1112: UserWarning: Can't initialize NVML
  raw_cnt = _raw_device_count_nvml()
/users/hep/slowmoyang/work/test-gpu/.venv/lib/python3.12/site-packages/torch/cuda/__init__.py:1170: UserWarning: CUDA initialization: CUDA unknown error - this may be due to an incorrectly set up environment, e.g. changing env variable CUDA_VISIBLE_DEVICES after program start. Setting the available devices to be zero. (Triggered internally at /__w/pytorch/pytorch/c10/cuda/CUDAFunctions.cpp:119.)
  r = torch._C._cuda_getDeviceCount() if nvml_count < 0 else nvml_count
0

[slowmoyang]

reboot 해볼까요?

[slowmoyang]

@jhgoh 직접 하신건가요?

[jhgoh]

리붓완료