Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

--n_events option in simulate_pixels.py not working #193

Open
krwood opened this issue Jan 20, 2024 · 2 comments
Open

--n_events option in simulate_pixels.py not working #193

krwood opened this issue Jan 20, 2024 · 2 comments

Comments

@krwood
Copy link
Member

krwood commented Jan 20, 2024 •
edited
Loading

Specifying certain values for the --n_events option in simulate_pixels.py crashes the simulation. When running on a file with 200 events (= NuMI spills in 2x2), it runs fine when not specifying --n_events; however, when specifying --n_events = 1, there is a cuda illegal address error:

Loading track segments...Selecting only the first 1 events for simulation.
Data preparation time: 2.73 s
******************
RUNNING SIMULATION
******************
Quenching electrons... 0.42 s
Drifting electrons... 0.20 s

Calculating optical responses... 34.23 s
Simulating batches...:   0%|                              | 0/1 [00:00<?, ?it/s]ERROR:numba.cuda.cudadrv.driver:Call to cuMemcpyDtoH results in UNKNOWN_CUDA_ERROR
Simulating batches...:   0%|                              | 0/1 [00:05<?, ?it/s]
Traceback (most recent call last):
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/commit/clean/larnd-sim/cli/simulate_pixels.py", line 1143, in <module>
    fire.Fire(run_simulation)
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/fire/core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/fire/core.py", line 475, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/fire/core.py", line 691, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/commit/clean/larnd-sim/cli/simulate_pixels.py", line 993, in run_simulation
    light_sim.sum_light_signals[BPG, TPB](
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/dispatcher.py", line 539, in __call__
    return self.dispatcher.call(args, self.griddim, self.blockdim,
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/dispatcher.py", line 675, in call
    kernel.launch(args, griddim, blockdim, stream, sharedmem)
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/dispatcher.py", line 371, in launch
    wb()
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/args.py", line 62, in <lambda>
    retr.append(lambda: devary.copy_to_host(self.value, stream=stream))
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/devices.py", line 232, in _require_cuda_context
    return fn(*args, **kws)
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/devicearray.py", line 277, in copy_to_host
    _driver.device_to_host(hostary, self, self.alloc_size,
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 3145, in device_to_host
    fn(host_pointer(dst), device_pointer(src), size, *varargs)
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 327, in safe_cuda_api_call
    self._check_ctypes_error(fname, retcode)
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 395, in _check_ctypes_error
    raise CudaAPIError(retcode, msg)
numba.cuda.cudadrv.driver.CudaAPIError: [700] Call to cuMemcpyDtoH results in UNKNOWN_CUDA_ERROR
ERROR:numba.cuda.cudadrv.driver:Call to cuMemFree results in UNKNOWN_CUDA_ERROR
Traceback (most recent call last):
  File "/global/common/software/nersc/pm-2022q3/sw/python/3.9-anaconda-2021.11/lib/python3.9/weakref.py", line 667, in _exitfunc
    f()
  File "/global/common/software/nersc/pm-2022q3/sw/python/3.9-anaconda-2021.11/lib/python3.9/weakref.py", line 591, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 2173, in deref
    mem.free()
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 1962, in free
    self._finalizer()
  File "/global/common/software/nersc/pm-2022q3/sw/python/3.9-anaconda-2021.11/lib/python3.9/weakref.py", line 591, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 1614, in core
    deallocations.add_item(driver.cuMemFree, ptr, size)
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 1180, in add_item
    self.clear()
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 1191, in clear
    dtor(handle)
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 327, in safe_cuda_api_call
    self._check_ctypes_error(fname, retcode)
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 395, in _check_ctypes_error
    raise CudaAPIError(retcode, msg)
numba.cuda.cudadrv.driver.CudaAPIError: [700] Call to cuMemFree results in UNKNOWN_CUDA_ERROR
ERROR:numba.cuda.cudadrv.driver:Call to cuMemFree results in UNKNOWN_CUDA_ERROR
Traceback (most recent call last):
  File "/global/common/software/nersc/pm-2022q3/sw/python/3.9-anaconda-2021.11/lib/python3.9/weakref.py", line 667, in _exitfunc
    f()
  File "/global/common/software/nersc/pm-2022q3/sw/python/3.9-anaconda-2021.11/lib/python3.9/weakref.py", line 591, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 2173, in deref
    mem.free()
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 1962, in free
    self._finalizer()
  File "/global/common/software/nersc/pm-2022q3/sw/python/3.9-anaconda-2021.11/lib/python3.9/weakref.py", line 591, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 1614, in core
    deallocations.add_item(driver.cuMemFree, ptr, size)
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 1180, in add_item
    self.clear()
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 1191, in clear
    dtor(handle)
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 327, in safe_cuda_api_call
    self._check_ctypes_error(fname, retcode)
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 395, in _check_ctypes_error
    raise CudaAPIError(retcode, msg)
numba.cuda.cudadrv.driver.CudaAPIError: [700] Call to cuMemFree results in UNKNOWN_CUDA_ERROR
ERROR:numba.cuda.cudadrv.driver:Call to cuMemFree results in UNKNOWN_CUDA_ERROR
Traceback (most recent call last):
  File "/global/common/software/nersc/pm-2022q3/sw/python/3.9-anaconda-2021.11/lib/python3.9/weakref.py", line 667, in _exitfunc
    f()
  File "/global/common/software/nersc/pm-2022q3/sw/python/3.9-anaconda-2021.11/lib/python3.9/weakref.py", line 591, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 2173, in deref
    mem.free()
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 1962, in free
    self._finalizer()
  File "/global/common/software/nersc/pm-2022q3/sw/python/3.9-anaconda-2021.11/lib/python3.9/weakref.py", line 591, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 1614, in core
    deallocations.add_item(driver.cuMemFree, ptr, size)
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 1180, in add_item
    self.clear()
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 1191, in clear
    dtor(handle)
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 327, in safe_cuda_api_call
    self._check_ctypes_error(fname, retcode)
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 395, in _check_ctypes_error
    raise CudaAPIError(retcode, msg)
numba.cuda.cudadrv.driver.CudaAPIError: [700] Call to cuMemFree results in UNKNOWN_CUDA_ERROR
ERROR:numba.cuda.cudadrv.driver:Call to cuMemFree results in UNKNOWN_CUDA_ERROR
Traceback (most recent call last):
  File "/global/common/software/nersc/pm-2022q3/sw/python/3.9-anaconda-2021.11/lib/python3.9/weakref.py", line 667, in _exitfunc
    f()
  File "/global/common/software/nersc/pm-2022q3/sw/python/3.9-anaconda-2021.11/lib/python3.9/weakref.py", line 591, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 2173, in deref
    mem.free()
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 1962, in free
    self._finalizer()
  File "/global/common/software/nersc/pm-2022q3/sw/python/3.9-anaconda-2021.11/lib/python3.9/weakref.py", line 591, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 1614, in core
    deallocations.add_item(driver.cuMemFree, ptr, size)
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 1180, in add_item
    self.clear()
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 1191, in clear
    dtor(handle)
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 327, in safe_cuda_api_call
    self._check_ctypes_error(fname, retcode)
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 395, in _check_ctypes_error
    raise CudaAPIError(retcode, msg)
numba.cuda.cudadrv.driver.CudaAPIError: [700] Call to cuMemFree results in UNKNOWN_CUDA_ERROR
ERROR:numba.cuda.cudadrv.driver:Call to cuModuleUnload results in UNKNOWN_CUDA_ERROR
Traceback (most recent call last):
  File "/global/common/software/nersc/pm-2022q3/sw/python/3.9-anaconda-2021.11/lib/python3.9/weakref.py", line 667, in _exitfunc
    f()
  File "/global/common/software/nersc/pm-2022q3/sw/python/3.9-anaconda-2021.11/lib/python3.9/weakref.py", line 591, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 1698, in core
    dealloc.add_item(module_unload, handle)
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 1180, in add_item
    self.clear()
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 1191, in clear
    dtor(handle)
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 1696, in module_unload
    driver.cuModuleUnload(handle)
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 327, in safe_cuda_api_call
    self._check_ctypes_error(fname, retcode)
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 395, in _check_ctypes_error
    raise CudaAPIError(retcode, msg)
numba.cuda.cudadrv.driver.CudaAPIError: [700] Call to cuModuleUnload results in UNKNOWN_CUDA_ERROR
ERROR:numba.cuda.cudadrv.driver:Call to cuMemFree results in UNKNOWN_CUDA_ERROR
Traceback (most recent call last):
  File "/global/common/software/nersc/pm-2022q3/sw/python/3.9-anaconda-2021.11/lib/python3.9/weakref.py", line 667, in _exitfunc
    f()
  File "/global/common/software/nersc/pm-2022q3/sw/python/3.9-anaconda-2021.11/lib/python3.9/weakref.py", line 591, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 2173, in deref
    mem.free()
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 1962, in free
    self._finalizer()
  File "/global/common/software/nersc/pm-2022q3/sw/python/3.9-anaconda-2021.11/lib/python3.9/weakref.py", line 591, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 1614, in core
    deallocations.add_item(driver.cuMemFree, ptr, size)
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 1180, in add_item
    self.clear()
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 1191, in clear
    dtor(handle)
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 327, in safe_cuda_api_call
    self._check_ctypes_error(fname, retcode)
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 395, in _check_ctypes_error
    raise CudaAPIError(retcode, msg)
numba.cuda.cudadrv.driver.CudaAPIError: [700] Call to cuMemFree results in UNKNOWN_CUDA_ERROR
ERROR:numba.cuda.cudadrv.driver:Call to cuModuleUnload results in UNKNOWN_CUDA_ERROR
Traceback (most recent call last):
  File "/global/common/software/nersc/pm-2022q3/sw/python/3.9-anaconda-2021.11/lib/python3.9/weakref.py", line 667, in _exitfunc
    f()
  File "/global/common/software/nersc/pm-2022q3/sw/python/3.9-anaconda-2021.11/lib/python3.9/weakref.py", line 591, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 1698, in core
    dealloc.add_item(module_unload, handle)
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 1180, in add_item
    self.clear()
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 1191, in clear
    dtor(handle)
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 1696, in module_unload
    driver.cuModuleUnload(handle)
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 327, in safe_cuda_api_call
    self._check_ctypes_error(fname, retcode)
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 395, in _check_ctypes_error
    raise CudaAPIError(retcode, msg)
numba.cuda.cudadrv.driver.CudaAPIError: [700] Call to cuModuleUnload results in UNKNOWN_CUDA_ERROR
ERROR:numba.cuda.cudadrv.driver:Call to cuModuleUnload results in UNKNOWN_CUDA_ERROR
Traceback (most recent call last):
  File "/global/common/software/nersc/pm-2022q3/sw/python/3.9-anaconda-2021.11/lib/python3.9/weakref.py", line 667, in _exitfunc
    f()
  File "/global/common/software/nersc/pm-2022q3/sw/python/3.9-anaconda-2021.11/lib/python3.9/weakref.py", line 591, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 1698, in core
    dealloc.add_item(module_unload, handle)
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 1180, in add_item
    self.clear()
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 1191, in clear
    dtor(handle)
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 1696, in module_unload
    driver.cuModuleUnload(handle)
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 327, in safe_cuda_api_call
    self._check_ctypes_error(fname, retcode)
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 395, in _check_ctypes_error
    raise CudaAPIError(retcode, msg)
numba.cuda.cudadrv.driver.CudaAPIError: [700] Call to cuModuleUnload results in UNKNOWN_CUDA_ERROR
ERROR:numba.cuda.cudadrv.driver:Call to cuMemFree results in UNKNOWN_CUDA_ERROR
Traceback (most recent call last):
  File "/global/common/software/nersc/pm-2022q3/sw/python/3.9-anaconda-2021.11/lib/python3.9/weakref.py", line 667, in _exitfunc
    f()
  File "/global/common/software/nersc/pm-2022q3/sw/python/3.9-anaconda-2021.11/lib/python3.9/weakref.py", line 591, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 2173, in deref
    mem.free()
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 1962, in free
    self._finalizer()
  File "/global/common/software/nersc/pm-2022q3/sw/python/3.9-anaconda-2021.11/lib/python3.9/weakref.py", line 591, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 1614, in core
    deallocations.add_item(driver.cuMemFree, ptr, size)
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 1180, in add_item
    self.clear()
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 1191, in clear
    dtor(handle)
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 327, in safe_cuda_api_call
    self._check_ctypes_error(fname, retcode)
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 395, in _check_ctypes_error
    raise CudaAPIError(retcode, msg)
numba.cuda.cudadrv.driver.CudaAPIError: [700] Call to cuMemFree results in UNKNOWN_CUDA_ERROR
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Exception ignored in: 'cupy.cuda.function.Module.__dealloc__'
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Exception ignored in: 'cupy.cuda.function.Module.__dealloc__'
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Exception ignored in: 'cupy.cuda.function.Module.__dealloc__'
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Exception ignored in: 'cupy.cuda.function.Module.__dealloc__'
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Exception ignored in: 'cupy.cuda.function.Module.__dealloc__'
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Exception ignored in: 'cupy.cuda.function.Module.__dealloc__'
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Exception ignored in: 'cupy.cuda.function.Module.__dealloc__'
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Error in sys.excepthook:
Traceback (most recent call last):
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/exceptiongroup/_formatting.py", line 71, in exceptiongroup_excepthook
TypeError: 'NoneType' object is not callable

Original exception was:
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Exception ignored in: 'cupy.cuda.function.Module.__dealloc__'
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Error in sys.excepthook:
Traceback (most recent call last):
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/exceptiongroup/_formatting.py", line 71, in exceptiongroup_excepthook
TypeError: 'NoneType' object is not callable

Original exception was:
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Exception ignored in: 'cupy.cuda.function.Module.__dealloc__'
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Error in sys.excepthook:
Traceback (most recent call last):
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/exceptiongroup/_formatting.py", line 71, in exceptiongroup_excepthook
TypeError: 'NoneType' object is not callable

Original exception was:
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Exception ignored in: 'cupy.cuda.function.Module.__dealloc__'
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Error in sys.excepthook:
Traceback (most recent call last):
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/exceptiongroup/_formatting.py", line 71, in exceptiongroup_excepthook
TypeError: 'NoneType' object is not callable

Original exception was:
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Exception ignored in: 'cupy.cuda.function.Module.__dealloc__'
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Error in sys.excepthook:
Traceback (most recent call last):
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/exceptiongroup/_formatting.py", line 71, in exceptiongroup_excepthook
TypeError: 'NoneType' object is not callable

Original exception was:
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Exception ignored in: 'cupy.cuda.function.Module.__dealloc__'
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Error in sys.excepthook:
Traceback (most recent call last):
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/exceptiongroup/_formatting.py", line 71, in exceptiongroup_excepthook
TypeError: 'NoneType' object is not callable

Original exception was:
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Exception ignored in: 'cupy.cuda.function.Module.__dealloc__'
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Error in sys.excepthook:
Traceback (most recent call last):
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/exceptiongroup/_formatting.py", line 71, in exceptiongroup_excepthook
TypeError: 'NoneType' object is not callable

Original exception was:
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Exception ignored in: 'cupy.cuda.function.Module.__dealloc__'
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Error in sys.excepthook:
Traceback (most recent call last):
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/exceptiongroup/_formatting.py", line 71, in exceptiongroup_excepthook
TypeError: 'NoneType' object is not callable

Original exception was:
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Exception ignored in: 'cupy.cuda.function.Module.__dealloc__'
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Error in sys.excepthook:
Traceback (most recent call last):
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/exceptiongroup/_formatting.py", line 71, in exceptiongroup_excepthook
TypeError: 'NoneType' object is not callable

Original exception was:
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Exception ignored in: 'cupy.cuda.function.Module.__dealloc__'
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Error in sys.excepthook:
Traceback (most recent call last):
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/exceptiongroup/_formatting.py", line 71, in exceptiongroup_excepthook
TypeError: 'NoneType' object is not callable

Original exception was:
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Exception ignored in: 'cupy.cuda.function.Module.__dealloc__'
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Error in sys.excepthook:
Traceback (most recent call last):
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/exceptiongroup/_formatting.py", line 71, in exceptiongroup_excepthook
TypeError: 'NoneType' object is not callable

Original exception was:
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Exception ignored in: 'cupy.cuda.function.Module.__dealloc__'
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Error in sys.excepthook:
Traceback (most recent call last):
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/exceptiongroup/_formatting.py", line 71, in exceptiongroup_excepthook
TypeError: 'NoneType' object is not callable

Original exception was:
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Exception ignored in: 'cupy.cuda.function.Module.__dealloc__'
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Error in sys.excepthook:
Traceback (most recent call last):
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/exceptiongroup/_formatting.py", line 71, in exceptiongroup_excepthook
TypeError: 'NoneType' object is not callable

Original exception was:
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Exception ignored in: 'cupy.cuda.function.Module.__dealloc__'
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Error in sys.excepthook:
Traceback (most recent call last):
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/exceptiongroup/_formatting.py", line 71, in exceptiongroup_excepthook
TypeError: 'NoneType' object is not callable

Original exception was:
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Exception ignored in: 'cupy.cuda.function.Module.__dealloc__'
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Error in sys.excepthook:
Traceback (most recent call last):
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/exceptiongroup/_formatting.py", line 71, in exceptiongroup_excepthook
TypeError: 'NoneType' object is not callable

Original exception was:
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Exception ignored in: 'cupy.cuda.function.Module.__dealloc__'
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Error in sys.excepthook:
Traceback (most recent call last):
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/exceptiongroup/_formatting.py", line 71, in exceptiongroup_excepthook
TypeError: 'NoneType' object is not callable

Original exception was:
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Exception ignored in: 'cupy.cuda.function.Module.__dealloc__'
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Error in sys.excepthook:
Traceback (most recent call last):
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/exceptiongroup/_formatting.py", line 71, in exceptiongroup_excepthook
TypeError: 'NoneType' object is not callable

Original exception was:
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Exception ignored in: 'cupy.cuda.function.Module.__dealloc__'
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Error in sys.excepthook:
Traceback (most recent call last):
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/exceptiongroup/_formatting.py", line 71, in exceptiongroup_excepthook
TypeError: 'NoneType' object is not callable

Original exception was:
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Exception ignored in: 'cupy.cuda.function.Module.__dealloc__'
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Error in sys.excepthook:

Original exception was:
Error in sys.excepthook:

Original exception was:
Error in sys.excepthook:

Original exception was:
Error in sys.excepthook:

Original exception was:
Error in sys.excepthook:

Original exception was:
Error in sys.excepthook:

Original exception was:
Error in sys.excepthook:

Original exception was:
Error in sys.excepthook:

Original exception was:
Error in sys.excepthook:

Original exception was:

This has been observed on both Perlmutter and SLAC machines.
@liviocali
Copy link
Member

n_events=10 works

@sam-fogarty
Copy link
Member

@krwood @liviocali Is this still an issue? I got nevents=1 and nevents=5 to run on 2x2_mod2mod_variation config and a miniRun4 file. (I haven't tested on the 2x2 config yet as I was running into an unrelated error.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@krwood @liviocali @sam-fogarty