running eval_multipro.py error:cuda runtime error

[sunwillz]

Hi,
when i finished my training, i run the eval_multipro.py and get below error:
nr_dev: 1
0%| | 0/5145 [00:00<?, ?it/s]process:0, start_idx:0, end_idx:5145
THCudaCheck FAIL file=/pytorch/torch/csrc/cuda/Module.cpp line=34 error=3 : initialization error
Process Process-1:
Traceback (most recent call last):
File "/usr/lib/python3.5/multiprocessing/process.py", line 249, in _bootstrap
self.run()
File "/usr/lib/python3.5/multiprocessing/process.py", line 93, in run
self._target(*self._args, **self._kwargs)
File "eval_multipro.py", line 129, in worker
torch.cuda.set_device(dev_id)
File "/home/sunyangwei/.local/lib/python3.5/site-packages/torch/cuda/init.py", line 264, in set_device
torch._C._cuda_setDevice(device)
RuntimeError: cuda runtime error (3) : initialization error at /pytorch/torch/csrc/cuda/Module.cpp:34

could you please tell me how to solve it?
I run it with pytorch 1.0 on ubuntu16.04

[Jongchan]

Change the starting of eval_multipro.py as, then it should work. I use Ubuntu 16.04 and PyTorch 1.0.1

# System libs
import os
from tqdm import tqdm
import datetime
import argparse
from distutils.version import LooseVersion
#from multiprocessing import Queue, Process
# Numerical libs
import numpy as np
import math
import torch
import cv2
import torch.nn as nn
from scipy.io import loadmat
import torch.multiprocessing as mp
from torch.multiprocessing import Queue, Process
mp.set_start_method('spawn', force=True)

[zhihengli-UR]

You can also add multiprocessing.set_start_method('spawn') following if __name__ == '__main__':