-
Notifications
You must be signed in to change notification settings - Fork 142
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
make install
insufficient for running llama3-8B-Instruct
#484
Comments
I'll note that the docker install worked perfectly. I just happen to be testing in an environment where I can't run docker. |
I successfully got everything working by installing all the low level libraries. Further, I found that for some Loras it triggers a flow that depends on
After that my rank 256 lora successfully ran. |
Need to update the docs to reflect the steps you took to get it to work. Are you blocked on anything? |
Not blocked~ But updated docs (or a unified install target) would help a lot. Now that I figured it out (a lot was from semi-copying from the dockerfile) I unblocked myself, but I imagine others who want to repeat this step will probably stumble. I'll definitely try to avoid re-doing this however. |
System Info
lorax-launcher-env
output:cargo version
output:Model being used:
meta-llama/Meta-Llama-3-70B-Instruct
GPUs: 8 A100s on Coreweave (can't get more details since I broke nvidia accidentally).
Cuda is 12.2 I believe.
Information
Tasks
Reproduction
make install
lorax-launcher --model-id meta-llama/Meta-Llama-3-70B-Instruct --port 8080
Initial failure reports that module
dropout_layer_norm
can't be found.From reading the docker instructions, I believe the full installation is something like:
However when doing this the
install-vllm
step ran into an issue whereby it expectedtorch==2.2.1
howevermake install
actually runspip install torch==2.2.0
which breaks the vllm step.Expected behavior
The following steps work successfully:
make install
lorax-launcher --model-id meta-llama/Meta-Llama-3-70B-Instruct --port 8080
Alternatively, 2 could be something like
make install-comprehensive
to include the full vllm and flash attention set of dependencies.The text was updated successfully, but these errors were encountered: