Skip to content

Commit

Permalink
Update gh-pages to output generated at d81dc64
Browse files Browse the repository at this point in the history
  • Loading branch information
slobentanzer committed Aug 15, 2024
1 parent ed5775f commit fd09539
Show file tree
Hide file tree
Showing 15 changed files with 69 additions and 41 deletions.
40 changes: 20 additions & 20 deletions api-docs/llm_connect/index.html

Large diffs are not rendered by default.

68 changes: 48 additions & 20 deletions features/chat/index.html
Original file line number Diff line number Diff line change
Expand Up @@ -539,6 +539,15 @@
</span>
</a>

</li>

<li class="md-nav__item">
<a href="#using-anthropic-models-claude" class="md-nav__link">
<span class="md-ellipsis">
Using Anthropic models (Claude)
</span>
</a>

</li>

<li class="md-nav__item">
Expand Down Expand Up @@ -1223,6 +1232,15 @@
</span>
</a>

</li>

<li class="md-nav__item">
<a href="#using-anthropic-models-claude" class="md-nav__link">
<span class="md-ellipsis">
Using Anthropic models (Claude)
</span>
</a>

</li>

<li class="md-nav__item">
Expand Down Expand Up @@ -1326,40 +1344,50 @@ <h2 id="using-openai-models">Using OpenAI models</h2>
designate the <code>OPENAI_API_KEY</code> variable in your environment directly (<code>export
OPENAI_API_KEY=sk-...</code>) by adding it to your shell configuration (e.g., the
<code>zshrc</code>).</p>
<h2 id="using-anthropic-models-claude">Using Anthropic models (Claude)</h2>
<p>Similarly, to use an Anthropic model, you need a billable account with Anthropic
API access, and to set the <code>ANTHROPIC_API_KEY</code> variable in your environment.</p>
<div class="language-python highlight"><pre><span></span><code><span id="__span-3-1"><a id="__codelineno-3-1" name="__codelineno-3-1" href="#__codelineno-3-1"></a><span class="kn">from</span> <span class="nn">biochatter.llm_connect</span> <span class="kn">import</span> <span class="n">AnthropicConversation</span>
</span><span id="__span-3-2"><a id="__codelineno-3-2" name="__codelineno-3-2" href="#__codelineno-3-2"></a>
</span><span id="__span-3-3"><a id="__codelineno-3-3" name="__codelineno-3-3" href="#__codelineno-3-3"></a><span class="n">conversation</span> <span class="o">=</span> <span class="n">AnthropicConversation</span><span class="p">(</span>
</span><span id="__span-3-4"><a id="__codelineno-3-4" name="__codelineno-3-4" href="#__codelineno-3-4"></a> <span class="n">model_name</span><span class="o">=</span><span class="s2">&quot;claude-3-5-sonnet-20240620&quot;</span><span class="p">,</span>
</span><span id="__span-3-5"><a id="__codelineno-3-5" name="__codelineno-3-5" href="#__codelineno-3-5"></a> <span class="n">prompts</span><span class="o">=</span><span class="p">{},</span>
</span><span id="__span-3-6"><a id="__codelineno-3-6" name="__codelineno-3-6" href="#__codelineno-3-6"></a><span class="p">)</span>
</span></code></pre></div>
<h2 id="multimodal-models-text-and-image">Multimodal models - Text and image</h2>
<p>We support multimodal queries in models that offer these capabilities after the
blueprint of the OpenAI API. We can either add an image-containing message to
the conversation using the <code>append_image_message</code> method, or we can pass an
image URL directly to the <code>query</code> method:</p>
<div class="language-python highlight"><pre><span></span><code><span id="__span-3-1"><a id="__codelineno-3-1" name="__codelineno-3-1" href="#__codelineno-3-1"></a><span class="c1"># Either: Append image message</span>
</span><span id="__span-3-2"><a id="__codelineno-3-2" name="__codelineno-3-2" href="#__codelineno-3-2"></a><span class="n">conversation</span><span class="o">.</span><span class="n">append_image_message</span><span class="p">(</span>
</span><span id="__span-3-3"><a id="__codelineno-3-3" name="__codelineno-3-3" href="#__codelineno-3-3"></a> <span class="n">message</span><span class="o">=</span><span class="s2">&quot;Here is an attached image&quot;</span><span class="p">,</span>
</span><span id="__span-3-4"><a id="__codelineno-3-4" name="__codelineno-3-4" href="#__codelineno-3-4"></a> <span class="n">image_url</span><span class="o">=</span><span class="s2">&quot;https://example.com/image.jpg&quot;</span>
</span><span id="__span-3-5"><a id="__codelineno-3-5" name="__codelineno-3-5" href="#__codelineno-3-5"></a><span class="p">)</span>
</span><span id="__span-3-6"><a id="__codelineno-3-6" name="__codelineno-3-6" href="#__codelineno-3-6"></a>
</span><span id="__span-3-7"><a id="__codelineno-3-7" name="__codelineno-3-7" href="#__codelineno-3-7"></a><span class="c1"># Or: Query with image included</span>
</span><span id="__span-3-8"><a id="__codelineno-3-8" name="__codelineno-3-8" href="#__codelineno-3-8"></a><span class="n">msg</span><span class="p">,</span> <span class="n">token_usage</span><span class="p">,</span> <span class="n">correction</span> <span class="o">=</span> <span class="n">conversation</span><span class="o">.</span><span class="n">query</span><span class="p">(</span>
</span><span id="__span-3-9"><a id="__codelineno-3-9" name="__codelineno-3-9" href="#__codelineno-3-9"></a> <span class="s2">&quot;What&#39;s in this image?&quot;</span><span class="p">,</span>
</span><span id="__span-3-10"><a id="__codelineno-3-10" name="__codelineno-3-10" href="#__codelineno-3-10"></a> <span class="n">image_url</span><span class="o">=</span><span class="s2">&quot;https://example.com/image.jpg&quot;</span>
</span><span id="__span-3-11"><a id="__codelineno-3-11" name="__codelineno-3-11" href="#__codelineno-3-11"></a><span class="p">)</span>
<div class="language-python highlight"><pre><span></span><code><span id="__span-4-1"><a id="__codelineno-4-1" name="__codelineno-4-1" href="#__codelineno-4-1"></a><span class="c1"># Either: Append image message</span>
</span><span id="__span-4-2"><a id="__codelineno-4-2" name="__codelineno-4-2" href="#__codelineno-4-2"></a><span class="n">conversation</span><span class="o">.</span><span class="n">append_image_message</span><span class="p">(</span>
</span><span id="__span-4-3"><a id="__codelineno-4-3" name="__codelineno-4-3" href="#__codelineno-4-3"></a> <span class="n">message</span><span class="o">=</span><span class="s2">&quot;Here is an attached image&quot;</span><span class="p">,</span>
</span><span id="__span-4-4"><a id="__codelineno-4-4" name="__codelineno-4-4" href="#__codelineno-4-4"></a> <span class="n">image_url</span><span class="o">=</span><span class="s2">&quot;https://example.com/image.jpg&quot;</span>
</span><span id="__span-4-5"><a id="__codelineno-4-5" name="__codelineno-4-5" href="#__codelineno-4-5"></a><span class="p">)</span>
</span><span id="__span-4-6"><a id="__codelineno-4-6" name="__codelineno-4-6" href="#__codelineno-4-6"></a>
</span><span id="__span-4-7"><a id="__codelineno-4-7" name="__codelineno-4-7" href="#__codelineno-4-7"></a><span class="c1"># Or: Query with image included</span>
</span><span id="__span-4-8"><a id="__codelineno-4-8" name="__codelineno-4-8" href="#__codelineno-4-8"></a><span class="n">msg</span><span class="p">,</span> <span class="n">token_usage</span><span class="p">,</span> <span class="n">correction</span> <span class="o">=</span> <span class="n">conversation</span><span class="o">.</span><span class="n">query</span><span class="p">(</span>
</span><span id="__span-4-9"><a id="__codelineno-4-9" name="__codelineno-4-9" href="#__codelineno-4-9"></a> <span class="s2">&quot;What&#39;s in this image?&quot;</span><span class="p">,</span>
</span><span id="__span-4-10"><a id="__codelineno-4-10" name="__codelineno-4-10" href="#__codelineno-4-10"></a> <span class="n">image_url</span><span class="o">=</span><span class="s2">&quot;https://example.com/image.jpg&quot;</span>
</span><span id="__span-4-11"><a id="__codelineno-4-11" name="__codelineno-4-11" href="#__codelineno-4-11"></a><span class="p">)</span>
</span></code></pre></div>
<h3 id="using-local-images">Using local images</h3>
<p>Following the recommendations by OpenAI, we can pass local images as
base64-encoded strings. We allow this by setting the <code>local</code> flag to <code>True</code> in
the <code>append_image_message</code> method:</p>
<div class="language-python highlight"><pre><span></span><code><span id="__span-4-1"><a id="__codelineno-4-1" name="__codelineno-4-1" href="#__codelineno-4-1"></a><span class="n">conversation</span><span class="o">.</span><span class="n">append_image_message</span><span class="p">(</span>
</span><span id="__span-4-2"><a id="__codelineno-4-2" name="__codelineno-4-2" href="#__codelineno-4-2"></a> <span class="n">message</span><span class="o">=</span><span class="s2">&quot;Here is an attached image&quot;</span><span class="p">,</span>
</span><span id="__span-4-3"><a id="__codelineno-4-3" name="__codelineno-4-3" href="#__codelineno-4-3"></a> <span class="n">image_url</span><span class="o">=</span><span class="s2">&quot;my/local/image.jpg&quot;</span><span class="p">,</span>
</span><span id="__span-4-4"><a id="__codelineno-4-4" name="__codelineno-4-4" href="#__codelineno-4-4"></a> <span class="n">local</span><span class="o">=</span><span class="kc">True</span>
</span><span id="__span-4-5"><a id="__codelineno-4-5" name="__codelineno-4-5" href="#__codelineno-4-5"></a><span class="p">)</span>
<div class="language-python highlight"><pre><span></span><code><span id="__span-5-1"><a id="__codelineno-5-1" name="__codelineno-5-1" href="#__codelineno-5-1"></a><span class="n">conversation</span><span class="o">.</span><span class="n">append_image_message</span><span class="p">(</span>
</span><span id="__span-5-2"><a id="__codelineno-5-2" name="__codelineno-5-2" href="#__codelineno-5-2"></a> <span class="n">message</span><span class="o">=</span><span class="s2">&quot;Here is an attached image&quot;</span><span class="p">,</span>
</span><span id="__span-5-3"><a id="__codelineno-5-3" name="__codelineno-5-3" href="#__codelineno-5-3"></a> <span class="n">image_url</span><span class="o">=</span><span class="s2">&quot;my/local/image.jpg&quot;</span><span class="p">,</span>
</span><span id="__span-5-4"><a id="__codelineno-5-4" name="__codelineno-5-4" href="#__codelineno-5-4"></a> <span class="n">local</span><span class="o">=</span><span class="kc">True</span>
</span><span id="__span-5-5"><a id="__codelineno-5-5" name="__codelineno-5-5" href="#__codelineno-5-5"></a><span class="p">)</span>
</span></code></pre></div>
<p>We also support the use of local images in the <code>query</code> method by detecting the
netloc of the image URL. If the netloc is empty, we assume that the image is
local and read it as a base64-encoded string:</p>
<div class="language-python highlight"><pre><span></span><code><span id="__span-5-1"><a id="__codelineno-5-1" name="__codelineno-5-1" href="#__codelineno-5-1"></a><span class="n">msg</span><span class="p">,</span> <span class="n">token_usage</span><span class="p">,</span> <span class="n">correction</span> <span class="o">=</span> <span class="n">conversation</span><span class="o">.</span><span class="n">query</span><span class="p">(</span>
</span><span id="__span-5-2"><a id="__codelineno-5-2" name="__codelineno-5-2" href="#__codelineno-5-2"></a> <span class="s2">&quot;What&#39;s in this image?&quot;</span><span class="p">,</span>
</span><span id="__span-5-3"><a id="__codelineno-5-3" name="__codelineno-5-3" href="#__codelineno-5-3"></a> <span class="n">image_url</span><span class="o">=</span><span class="s2">&quot;my/local/image.jpg&quot;</span>
</span><span id="__span-5-4"><a id="__codelineno-5-4" name="__codelineno-5-4" href="#__codelineno-5-4"></a><span class="p">)</span>
<div class="language-python highlight"><pre><span></span><code><span id="__span-6-1"><a id="__codelineno-6-1" name="__codelineno-6-1" href="#__codelineno-6-1"></a><span class="n">msg</span><span class="p">,</span> <span class="n">token_usage</span><span class="p">,</span> <span class="n">correction</span> <span class="o">=</span> <span class="n">conversation</span><span class="o">.</span><span class="n">query</span><span class="p">(</span>
</span><span id="__span-6-2"><a id="__codelineno-6-2" name="__codelineno-6-2" href="#__codelineno-6-2"></a> <span class="s2">&quot;What&#39;s in this image?&quot;</span><span class="p">,</span>
</span><span id="__span-6-3"><a id="__codelineno-6-3" name="__codelineno-6-3" href="#__codelineno-6-3"></a> <span class="n">image_url</span><span class="o">=</span><span class="s2">&quot;my/local/image.jpg&quot;</span>
</span><span id="__span-6-4"><a id="__codelineno-6-4" name="__codelineno-6-4" href="#__codelineno-6-4"></a><span class="p">)</span>
</span></code></pre></div>
<h3 id="open-source-multimodal-models">Open-source multimodal models</h3>
<p>While OpenAI models work seamlessly, open-source multimodal models can be buggy
Expand Down
Binary file modified images/boxplot-naive-vs-biochatter.pdf
Binary file not shown.
Binary file modified images/dotplot-per-task.pdf
Binary file not shown.
Binary file modified images/dotplot-per-task.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified images/scatter-per-quantisation-name.pdf
Binary file not shown.
Binary file modified images/scatter-per-quantisation-name.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified images/scatter-quantisation-accuracy.pdf
Binary file not shown.
Binary file modified images/scatter-size-accuracy.pdf
Binary file not shown.
Binary file modified images/stripplot-extraction-tasks.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified images/stripplot-per-model.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified images/stripplot-rag-tasks.pdf
Binary file not shown.
Binary file modified images/stripplot-rag-tasks.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified scripts/__pycache__/hooks.cpython-312.pyc
Binary file not shown.
2 changes: 1 addition & 1 deletion search/search_index.json

Large diffs are not rendered by default.

0 comments on commit fd09539

Please sign in to comment.