[bug] failed to install torch-tensorrt #3903
Comments
|
Hi @geraldstanje, we have recently updated torchServe version to 0.11.0. Please pull the latest images to use them. For tensor-rt, we'll require a repro steps to do so. However, we suggest taking a look at DJL TensorRT containers if you would be interested in that. For extending DLCs, you can do so as you outlined. Model artifacts are copied into the container at runtime from the Python SDK (which I am assuming is what you're using) through a For the image tag, the two images you outlined are the same image even though the tags are different. I want to note though that |
whats the name of that pytorch image? e.g. is that what you refer to? https://github.com/aws/deep-learning-containers/tree/master/pytorch/inference/docker/2.2/py3
why switch to a different image? torch-tensorrt and tensorrt can be used with torchServe... |
|
Any supported PyTorch (PT 1.13, 2.1, 2.2) inference image would work. They all have torchserve 0.11.0. Generally, you can pull images with the following tags: These tags would pull our latest release which will be moved to the latest image every time we release a patch. However, you may see some tags such as These tags represent the specific patch releases. so using these tags would pull in specific image that was released at certain date.
Yes, our gpu inference image is using cuda 11.8.
We don't expect any installation error with tensor-rt but you're welcomed to outline repro steps if you encounter issues and we'll be happy to reproduce and assist.
DJL containers offer tensorrt out of the box while our regular DLCs do not. DJL containers can also be used similarly to extend your own custom containers. For more information about DJL containers |
|
@sirutBuasai do you also going to release a new pytorch-inference image with cuda 12.x? |
|
Not for PyTorch 2.1 and 2.2 Inference. However, we are working on PyTorch 2.3 Inference with CUDA 12.1. |
|
@sirutBuasai any timeline when do you also update the triton inference image for cuda 12.x soon? |
|
We are aiming for 6/7 for PyTorch 2.3 Inference with CUDA 12.1. Which triton image are you referring to? |
|
@sirutBuasai i mean NVIDIA Triton Inference Server: https://github.com/aws/deep-learning-containers/blob/master/available_images.md#nvidia-triton-inference-containers-sm-support-only - can someone build the Triton Inference Server Release 24.05? dont see the nvidia-triton-inference-containers image in this github repo... can you send me the link? cc @nskool |
|
@nskool Could you assist with triton image questions? |
|
@sirutBuasai - if you go the following link it says: https://github.com/pytorch/TensorRT i use torch-tensorrt 2.2.0 with dlc 763104351884.dkr.ecr.us-west-2.amazonaws.com/pytorch-inference:2.2.0-gpu-py310-cu118-ubuntu20.04-sagemaker-v1.10 and get error: but when i run it on ec2 with cuda - it works fine - it seems i cannot use cuda 11 and require cuda 12.x for torch-tensorrt 2.2.0... |
|
regarding NVIDIA Triton Inference Server
cc @nskool @sirutBuasai |
|
For tensorrt installation error, could you provide the following:
|
Checklist
Concise Description:
i get error: failed to install torch-tensorrt.
Error Message:
entire log:
Logs:
code/requirements.txt:
DLC image/dockerfile:
763104351884.dkr.ecr.us-west-2.amazonaws.com/pytorch-inference:2.1-gpu-py310
Current behavior:
error during installing torch-tensorrt
Expected behavior:
no error
Additional context:
Can i extend the deep learning image for sagemaker as follows, push this image to aws ecr and use that image to deploy my sagemaker inference endpoint? how does the model artifact (code/inference.py code/requirements.txt model etc.) get copied into the docker container?
i see there are 2 images - can i use both for sagemaker - or only the second one?
vs.
also the torch-tensorrt 2.2.0 whl file is available here: https://pypi.org/project/torch-tensorrt/2.2.0/ - why it cant find it?
cc @tejaschumbalkar @joaopcm1996
also, torchServe is already at version 0.10 - how can i use that version with 763104351884.dkr.ecr.us-west-2.amazonaws.com/pytorch-inference:2.1-gpu-py310 or 763104351884.dkr.ecr.us-east-1.amazonaws.com/pytorch-inference:2.1.0-gpu-py310-cu118-ubuntu20.04-sagemaker? cc @sirutBuasai
The text was updated successfully, but these errors were encountered: