Skip to content

Commit

Permalink
ollama tweaks
Browse files Browse the repository at this point in the history
  • Loading branch information
julien-c committed Oct 16, 2024
1 parent 481c63a commit a7f304e
Showing 1 changed file with 5 additions and 3 deletions.
8 changes: 5 additions & 3 deletions docs/hub/ollama.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,9 @@ ollama run hf.co/bartowski/Humanish-LLama3-8B-Instruct-GGUF

## Custom Quantization

By default, the `Q4_K_M` quantization scheme is used. To select a different scheme, simply add a tag:
By default, the `Q4_K_M` quantization scheme is used, when it's present inside the model repo. If not, we default to picking one reasonable quant type present inside the repo.

To select a different scheme, simply add a tag:

```sh
ollama run hf.co/{username}/{repository}:{quantization}
Expand All @@ -40,15 +42,15 @@ ollama run hf.co/bartowski/Llama-3.2-3B-Instruct-GGUF:Q8_0
# the quantization name is case-insensitive, this will also work
ollama run hf.co/bartowski/Llama-3.2-3B-Instruct-GGUF:iq3_m

# you can also select a specific file
# you can also directly use the full filename as a tag
ollama run hf.co/bartowski/Llama-3.2-3B-Instruct-GGUF:Llama-3.2-3B-Instruct-IQ3_M.gguf
```

## Custom Chat Template and Parameters

By default, a template will be selected automatically from a list of commonly used templates. It will be selected based on the built-in `tokenizer.chat_template` metadata stored inside the GGUF file.

If your GGUF file doesn't have a built-in template or uses a custom chat template, you can create a new file called `template` in the repository. The template must be a Go template, not a Jinja template. Here's an example:
If your GGUF file doesn't have a built-in template or if you want to customize your chat template, you can create a new file called `template` in the repository. The template must be a Go template, not a Jinja template. Here's an example:

```
{{ if .System }}<|system|>
Expand Down

0 comments on commit a7f304e

Please sign in to comment.