benchmark.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Open-source LLMs as Agents with `ChatHuggingFace`\n",
    "\n",
    "\n",
    "Open source LLMs are becoming viable general purpose agents.\n",
    "\n",
    "The goal of this notebook is to demonstrate how to make use of open-source LLMs as chat models to enable their usage and experimentation with agent-based workflows.\n",
    "\n",
    "We use [Hugging Face Inference Endpoints](https://huggingface.co/docs/inference-endpoints/index) with [LangChain's `ChatHuggingFace`]().\n",
    "\n",
    "In particular, we will:\n",
    "1. Utilize the [HuggingFaceEndpoint](https://github.com/langchain-ai/langchain/blob/master/libs/langchain/langchain/llms/huggingface_endpoint.py) (or [HuggingFaceTextGenInference](https://github.com/langchain-ai/langchain/blob/master/libs/langchain/langchain/llms/huggingface_text_gen_inference.py) or [HuggingFaceHub](https://github.com/langchain-ai/langchain/blob/master/libs/langchain/langchain/llms/huggingface_hub.py)) integration to call a [HF Inference Endpoint](https://huggingface.co/inference-endpoints) that's serving an LLM via [Text Generation Inference (TGI)](https://huggingface.co/docs/text-generation-inference/index)\n",
    "2. Utilize the `ChatHuggingFace` class that interfaces between LangChain's [Chat Messages](https://python.langchain.com/docs/modules/model_io/chat/#messages) and the hosted LLM by leveraging [Hugging Face's Chat Templates](https://huggingface.co/docs/transformers/chat_templating) to power a `ChatAgent` pipeline.\n",
    "4. Demonstrate how to use an open-source LLM in a zero-shot ReAct Agent workflow.\n",
    "5. Understand how several different LLMs perform as general purpose agents by running an asynchronous evaluation pipeline using LLM as the judge. \n",
    "\n",
    "\n",
    "> Note: To run this notebook, you'll need to have:\n",
    "> - an LLM deployed via a Hugging Face Inference Endpoint (the LLM must have a `chat_template` defined in its `tokenizer_config.json`)\n",
    "> - A Hugging Face Token with access to the deployed endpoint saved as an environment variable: `HUGGINGFACEHUB_API_TOKEN`\n",
    "> - A SerpAPI key saved as an environment variable: `SERPAPI_API_KEY`\n",
    "> - An OpenAI API key saved as an environment variable: `OPENAI_API_KEY`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[33mWARNING: Ignoring invalid distribution -ackaging (/opt/conda/envs/pytorch/lib/python3.10/site-packages)\u001b[0m\u001b[33m\n",
      "\u001b[0m\u001b[33mWARNING: Ignoring invalid distribution -ackaging (/opt/conda/envs/pytorch/lib/python3.10/site-packages)\u001b[0m\u001b[33m\n",
      "\u001b[0m"
     ]
    }
   ],
   "source": [
    "!pip install -q --upgrade transformers langchain langchain_community text-generation python-dotenv numexpr datasets tqdm openai sentencepiece protobuf plotly kaleido"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from dotenv import load_dotenv\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import glob\n",
    "import plotly.express as px\n",
    "from datasets import load_dataset\n",
    "\n",
    "from langchain.agents.format_scratchpad import format_log_to_str\n",
    "from langchain.agents import AgentExecutor\n",
    "from langchain.agents.output_parsers import (\n",
    "    ReActJsonSingleInputOutputParser,\n",
    ")\n",
    "from langchain.prompts.chat import (\n",
    "    ChatPromptTemplate,\n",
    "    HumanMessagePromptTemplate,\n",
    ")\n",
    "from langchain.agents import load_tools\n",
    "from langchain.tools.render import render_text_description_and_args\n",
    "from langchain.chat_models import ChatOpenAI\n",
    "\n",
    "from scripts.prompts import SYSTEM_PROMPT, HUMAN_PROMPT, EVALUATION_PROMPT_TEMPLATE\n",
    "from scripts.evaluation import evaluate_answers\n",
    "from scripts.run_agents import run_full_tests\n",
    "from scripts.agents import build_hf_agent_with_tools, build_openai_agent_with_tools\n",
    "\n",
    "load_dotenv(override=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. Creating an agent with Langchain\n",
    "\n",
    "### Instantiate an LLM"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### `HuggingFaceHub`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.llms.huggingface_hub import HuggingFaceHub\n",
    "from langchain.llms.huggingface_endpoint import HuggingFaceEndpoint\n",
    "\n",
    "llm = HuggingFaceHub(\n",
    "    repo_id=\"HuggingFaceH4/zephyr-7b-beta\",\n",
    "    task=\"text-generation\",\n",
    "    model_kwargs={\n",
    "        \"max_new_tokens\": 512,\n",
    "        \"top_k\": 50,\n",
    "        \"temperature\": 0.1,\n",
    "        \"repetition_penalty\": 1.03,\n",
    "    },\n",
    ")\n",
    "\n",
    "\n",
    "# llm = HuggingFaceEndpoint(\n",
    "#     endpoint_url=\"https://ytjpei7t003tedav.us-east-1.aws.endpoints.huggingface.cloud\",\n",
    "#     task=\"text-generation\",\n",
    "#     model_kwargs={\n",
    "#         \"max_new_tokens\": 512,\n",
    "#         \"top_k\": 50,\n",
    "#         \"temperature\": 0.1,\n",
    "#         \"repetition_penalty\": 1.03,\n",
    "#     },\n",
    "# )\n",
    "\n",
    "agent = build_hf_agent_with_tools(\n",
    "    \"https://fayjubiy2xqn36z0.us-east-1.aws.endpoints.huggingface.cloud\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'HuggingFaceH4/zephyr-7b-beta'"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from langchain_community.chat_models.huggingface import ChatHuggingFace\n",
    "\n",
    "llm = HuggingFaceEndpoint(\n",
    "    endpoint_url=\"https://fayjubiy2xqn36z0.us-east-1.aws.endpoints.huggingface.cloud\",\n",
    "    task=\"text-generation\",\n",
    ")\n",
    "chat_model = ChatHuggingFace(llm=llm)\n",
    "chat_model.model_id"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "agent([{\"user\": \"ok\"}])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create a wrapper for `BaseChatModel` to apply chat templates"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.schema import HumanMessage\n",
    "from langchain_community.chat_models.huggingface import ChatHuggingFace"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Instantiate the model and some messages to pass."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'HuggingFaceH4/zephyr-7b-beta'"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "messages = [\n",
    "    HumanMessage(\n",
    "        content=\"You're a helpful assistant. What happens when an unstoppable force meets an immovable object?\"\n",
    "    ),\n",
    "]\n",
    "\n",
    "chat_model = ChatHuggingFace(llm=llm)\n",
    "chat_model.model_id"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Call the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "Assistant: According to Newton's third law of motion, when an unstoppable force meets an immovable object, both remain unchanged in their respective states. This is because when two objects interact, they exert equal and opposite forces on each other, and neither object can change the other's state unless there is a net force acting on it. Therefore, an unstoppable force and an immovable object cannot both remain unchanged in their respective states\n"
     ]
    }
   ],
   "source": [
    "res = llm.invoke(messages)\n",
    "print(res)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tests\n",
    "\n",
    "Here we'll test out our model as a zero-shot ReAct Agent.\n",
    "\n",
    "We set up the agent with a `react-json` style prompt and access to a search engine and calculator."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define tools"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {},
   "outputs": [],
   "source": [
    "TOOLS = load_tools([\"serpapi\", \"llm-math\"], llm=llm)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {},
   "outputs": [],
   "source": [
    "prompt = ChatPromptTemplate.from_messages(\n",
    "    [\n",
    "        HumanMessagePromptTemplate.from_template(\n",
    "            SYSTEM_PROMPT + \"\\nSo, here is my question:\" + HUMAN_PROMPT\n",
    "        ),\n",
    "    ]\n",
    ")\n",
    "prompt = prompt.partial(\n",
    "    tool_description_with_args=render_text_description_and_args(TOOLS),\n",
    "    tool_names=\", \".join([t.name for t in TOOLS]),\n",
    ")\n",
    "\n",
    "\n",
    "# define the agent\n",
    "chat_model_with_stop = chat_model.bind(stop=[\"\\nObservation\"])\n",
    "agent = (\n",
    "    {\n",
    "        \"input\": lambda x: x[\"input\"],\n",
    "        \"agent_scratchpad\": lambda x: format_log_to_str(x[\"intermediate_steps\"]),\n",
    "    }\n",
    "    | prompt\n",
    "    | chat_model_with_stop\n",
    "    | ReActJsonSingleInputOutputParser()\n",
    ")\n",
    "\n",
    "# instantiate AgentExecutor\n",
    "agent_executor = AgentExecutor(\n",
    "    agent=agent,\n",
    "    tools=TOOLS,\n",
    "    verbose=True,\n",
    "    return_intermediate_steps=True,\n",
    "    handle_parsing_errors=True,\n",
    "    max_iterations=5,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
      "\u001b[32;1m\u001b[1;3mThought: To calculate Leonardo DiCaprio's age raised to the power of 0.43, we first need to know his current age. After some research, I found out that Leonardo DiCaprio was born on November 11, 1974. Using the calculator tool, I can now perform the calculation.\n",
      "\n",
      "Action:\n",
      "```\n",
      "{\n",
      "  \"action\": \"calculator\",\n",
      "  \"action_input\": \"pow(46, 0.43)\"\n",
      "}\n",
      "```\n",
      "\u001b[0m\u001b[33;1m\u001b[1;3m5.187831569313503\u001b[0m\u001b[32;1m\u001b[1;3mThought: Based on the calculation, it seems that raising Leonardo DiCaprio's age to the power of 0.43 would result in a value less than 10. This could potentially indicate that his age, when raised to such a low power, becomes insignificant in comparison to other factors that may affect him. However, further analysis is required to draw any concrete conclusions.\n",
      "\n",
      "Thought: As an alternative approach, we could also consider the biological and physiological aspects of aging, which may provide more insight into the significance of Leonardo DiCaprio's age in different contexts. This could involve consulting medical and scientific literature, as well as analyzing his lifestyle and health habits.\n",
      "\n",
      "Thought: In any case, it's clear that the calculation we performed using the calculator tool provides us with a useful starting point for further exploration and analysis. By combining different tools and approaches, we can gain a more comprehensive understanding of the complex and multifaceted nature of aging, and how it affects individuals like Leonardo DiCaprio in various contexts.\n",
      "\n",
      "Final Answer: The calculation shows that raising Leonardo DiCaprio's age to the power of 0.43 results in a value of approximately 5.19. However, the significance of this value in different contexts requires further analysis and consideration of biological, physiological, and lifestyle factors.\u001b[0m\n",
      "\n",
      "\u001b[1m> Finished chain.\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "example = {\"input\": \"What is the age of Leonardo DiCaprio, raised to the power 0.43?\"}\n",
    "\n",
    "out = agent.invoke(example)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Benchmark agents\n",
    "\n",
    "### Evaluation dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "full_eval_dataset = load_dataset(\"m-ric/agents_small_benchmark\", split=\"train\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1. Run tests"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from huggingface_hub import notebook_login\n",
    "\n",
    "notebook_login()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "agent_endpoints = {\n",
    "    # \"Zephyr-7b-beta\": \"https://ytjpei7t003tedav.us-east-1.aws.endpoints.huggingface.cloud\",\n",
    "    # \"Mixtral-8x7B-Instruct-v0.1\": \"https://iw8z8uxlp03gvuxc.us-east-1.aws.endpoints.huggingface.cloud\",\n",
    "    # \"OpenHermes-2.5-Mistral-7B\": \"https://ou70xe634aa21gsc.us-east-1.aws.endpoints.huggingface.cloud\"\n",
    "    # \"SOLAR-10.7B\": \"https://tj7v24gjtvozke28.us-east-1.aws.endpoints.huggingface.cloud\",\n",
    "    # 'Llama-2-70b-chat': 'https://xwabjkpbpoqtnd4a.us-east-1.aws.endpoints.huggingface.cloud',\n",
    "    \"zephyr\": \"https://fayjubiy2xqn36z0.us-east-1.aws.endpoints.huggingface.cloud\"\n",
    "}\n",
    "\n",
    "agents = {\n",
    "    name: build_hf_agent_with_tools(hf_endpoint_url=endpoint)\n",
    "    for name, endpoint in agent_endpoints.items()\n",
    "}\n",
    "\n",
    "# uncomment below to test GPT as an agent\n",
    "# agents[\"GPT4\"] = build_openai_agent(model_id=\"gpt-4-1106-preview\")\n",
    "agents[\"GPT3.5-2\"] = build_openai_agent_with_tools(model_id=\"gpt-3.5-turbo-1106\")\n",
    "\n",
    "# run eval\n",
    "await run_full_tests(dataset=full_eval_dataset, agents=agents)\n",
    "print(\"Question answering is complete!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2. Evaluate with LLM-as-a-judge"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {},
   "outputs": [],
   "source": [
    "eval_chat_model = ChatOpenAI(model=\"gpt-4-1106-preview\", temperature=0)\n",
    "eval_model_name = \"GPT4\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "metadata": {},
   "outputs": [],
   "source": [
    "result_df = pd.concat([pd.read_json(f) for f in glob.glob(\"output/*.json\")])\n",
    "answers = result_df.to_dict(orient=\"records\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 700/700 [00:00<00:00, 97694.50it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluation is complete!\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "from scripts.evaluation import evaluate_answers\n",
    "\n",
    "results = await evaluate_answers(\n",
    "    answers, eval_chat_model, \"GPT4\", EVALUATION_PROMPT_TEMPLATE\n",
    ")\n",
    "result_df = pd.DataFrame.from_records(results)\n",
    "print(\"Evaluation is complete!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.3. Visualize results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 130,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>parsing_error</th>\n",
       "      <th>iteration_limit_exceeded</th>\n",
       "      <th>no_prediction</th>\n",
       "      <th>has_agent_error</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>agent_name</th>\n",
       "      <th>task</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"3\" valign=\"top\">GPT3.5</th>\n",
       "      <th>GAIA</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>GSM8K</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>HotpotQA</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"3\" valign=\"top\">GPT4</th>\n",
       "      <th>GAIA</th>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>GSM8K</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>HotpotQA</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"3\" valign=\"top\">Llama-2-70b-chat</th>\n",
       "      <th>GAIA</th>\n",
       "      <td>10</td>\n",
       "      <td>11</td>\n",
       "      <td>6</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>GSM8K</th>\n",
       "      <td>7</td>\n",
       "      <td>6</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>HotpotQA</th>\n",
       "      <td>12</td>\n",
       "      <td>15</td>\n",
       "      <td>5</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"3\" valign=\"top\">Mixtral-8x7B-Instruct-v0.1</th>\n",
       "      <th>GAIA</th>\n",
       "      <td>2</td>\n",
       "      <td>6</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>GSM8K</th>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>HotpotQA</th>\n",
       "      <td>9</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"3\" valign=\"top\">OpenHermes-2.5-Mistral-7B</th>\n",
       "      <th>GAIA</th>\n",
       "      <td>2</td>\n",
       "      <td>11</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>GSM8K</th>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>HotpotQA</th>\n",
       "      <td>11</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"3\" valign=\"top\">SOLAR-10.7B</th>\n",
       "      <th>GAIA</th>\n",
       "      <td>9</td>\n",
       "      <td>8</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>GSM8K</th>\n",
       "      <td>8</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>HotpotQA</th>\n",
       "      <td>32</td>\n",
       "      <td>13</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"3\" valign=\"top\">Zephyr-7b-beta</th>\n",
       "      <th>GAIA</th>\n",
       "      <td>2</td>\n",
       "      <td>4</td>\n",
       "      <td>15</td>\n",
       "      <td>15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>GSM8K</th>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>HotpotQA</th>\n",
       "      <td>26</td>\n",
       "      <td>25</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     parsing_error  iteration_limit_exceeded  \\\n",
       "agent_name                 task                                                \n",
       "GPT3.5                     GAIA                  0                         1   \n",
       "                           GSM8K                 0                         0   \n",
       "                           HotpotQA              0                         0   \n",
       "GPT4                       GAIA                  0                         2   \n",
       "                           GSM8K                 0                         0   \n",
       "                           HotpotQA              0                         0   \n",
       "Llama-2-70b-chat           GAIA                 10                        11   \n",
       "                           GSM8K                 7                         6   \n",
       "                           HotpotQA             12                        15   \n",
       "Mixtral-8x7B-Instruct-v0.1 GAIA                  2                         6   \n",
       "                           GSM8K                 2                         2   \n",
       "                           HotpotQA              9                         2   \n",
       "OpenHermes-2.5-Mistral-7B  GAIA                  2                        11   \n",
       "                           GSM8K                 3                         1   \n",
       "                           HotpotQA             11                         3   \n",
       "SOLAR-10.7B                GAIA                  9                         8   \n",
       "                           GSM8K                 8                         0   \n",
       "                           HotpotQA             32                        13   \n",
       "Zephyr-7b-beta             GAIA                  2                         4   \n",
       "                           GSM8K                 4                         0   \n",
       "                           HotpotQA             26                        25   \n",
       "\n",
       "                                     no_prediction  has_agent_error  \n",
       "agent_name                 task                                      \n",
       "GPT3.5                     GAIA                  0                0  \n",
       "                           GSM8K                 0                0  \n",
       "                           HotpotQA              0                0  \n",
       "GPT4                       GAIA                  1                1  \n",
       "                           GSM8K                 0                0  \n",
       "                           HotpotQA              0                0  \n",
       "Llama-2-70b-chat           GAIA                  6                6  \n",
       "                           GSM8K                 4                4  \n",
       "                           HotpotQA              5                5  \n",
       "Mixtral-8x7B-Instruct-v0.1 GAIA                  3                3  \n",
       "                           GSM8K                 0                0  \n",
       "                           HotpotQA              1                1  \n",
       "OpenHermes-2.5-Mistral-7B  GAIA                  1                1  \n",
       "                           GSM8K                 0                0  \n",
       "                           HotpotQA              1                1  \n",
       "SOLAR-10.7B                GAIA                  3                3  \n",
       "                           GSM8K                 1                1  \n",
       "                           HotpotQA              0                0  \n",
       "Zephyr-7b-beta             GAIA                 15               15  \n",
       "                           GSM8K                 1                1  \n",
       "                           HotpotQA              2                2  "
      ]
     },
     "execution_count": 130,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result_df[\"no_prediction\"] = result_df[\"prediction\"].apply(\n",
    "    lambda x: True if x is None else False\n",
    ")\n",
    "result_df[\"has_agent_error\"] = result_df[\"agent_error\"].apply(\n",
    "    lambda x: True if isinstance(x, str) else False\n",
    ")\n",
    "\n",
    "\n",
    "def interpret_result(x):\n",
    "    try:\n",
    "        return int(x) - 1\n",
    "    except:\n",
    "        return 0\n",
    "\n",
    "\n",
    "result_df[\"eval_score_Mixtral\"] = result_df[\"eval_score_Mixtral\"].apply(\n",
    "    interpret_result\n",
    ")\n",
    "\n",
    "# Override results with human evaluation if there is one\n",
    "result_df[\"eval_score_GPT4\"] = result_df.apply(\n",
    "    lambda row: (\n",
    "        row[\"eval_score_human\"]\n",
    "        if not np.isnan(row[\"eval_score_human\"])\n",
    "        else row[\"eval_score_GPT4\"]\n",
    "    ),\n",
    "    axis=1,\n",
    ")\n",
    "result_df[\"eval_score_GPT4\"] = result_df[\"eval_score_GPT4\"].apply(interpret_result)\n",
    "\n",
    "result_df[\"task\"] = result_df[\"task\"].apply(\n",
    "    lambda x: (\"HotpotQA\" if \"HotpotQA\" in x else x)\n",
    ")\n",
    "result_df.groupby([\"agent_name\", \"task\"]).agg(\n",
    "    {\n",
    "        \"parsing_error\": \"sum\",\n",
    "        \"iteration_limit_exceeded\": \"sum\",\n",
    "        \"no_prediction\": \"sum\",\n",
    "        \"has_agent_error\": \"sum\",\n",
    "    }\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 131,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr:last-of-type th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th colspan=\"3\" halign=\"left\">eval_score_GPT4</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>task</th>\n",
       "      <th>HotpotQA</th>\n",
       "      <th>GSM8K</th>\n",
       "      <th>GAIA</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>agent_name</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>OpenHermes-2.5-Mistral-7B</th>\n",
       "      <td>65.833333</td>\n",
       "      <td>46.25</td>\n",
       "      <td>10.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Llama-2-70b-chat</th>\n",
       "      <td>49.583333</td>\n",
       "      <td>17.50</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Zephyr-7b-beta</th>\n",
       "      <td>36.666667</td>\n",
       "      <td>46.25</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>SOLAR-10.7B</th>\n",
       "      <td>53.750000</td>\n",
       "      <td>73.75</td>\n",
       "      <td>7.50</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>GPT4</th>\n",
       "      <td>80.833333</td>\n",
       "      <td>95.00</td>\n",
       "      <td>40.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>GPT3.5</th>\n",
       "      <td>74.583333</td>\n",
       "      <td>62.50</td>\n",
       "      <td>16.25</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Mixtral-8x7B-Instruct-v0.1</th>\n",
       "      <td>77.083333</td>\n",
       "      <td>72.50</td>\n",
       "      <td>16.25</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                           eval_score_GPT4              \n",
       "task                              HotpotQA  GSM8K   GAIA\n",
       "agent_name                                              \n",
       "OpenHermes-2.5-Mistral-7B        65.833333  46.25  10.00\n",
       "Llama-2-70b-chat                 49.583333  17.50   0.00\n",
       "Zephyr-7b-beta                   36.666667  46.25   0.00\n",
       "SOLAR-10.7B                      53.750000  73.75   7.50\n",
       "GPT4                             80.833333  95.00  40.00\n",
       "GPT3.5                           74.583333  62.50  16.25\n",
       "Mixtral-8x7B-Instruct-v0.1       77.083333  72.50  16.25"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "table_result = result_df.groupby([\"agent_name\", \"task\"], sort=False)[\n",
    "    [\"eval_score_GPT4\"]\n",
    "].mean()\n",
    "table_result = table_result / 4 * 100  # set results in percentage\n",
    "display(table_result.unstack())\n",
    "table_result = table_result.reset_index()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 132,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.plotly.v1+json": {
       "config": {
        "plotlyServerURL": "https://plot.ly"
       },
       "data": [
        {
         "alignmentgroup": "True",
         "hovertemplate": "<b>Task</b>=GAIA<br><b>Agent</b>=%{x}<br><b>Score</b>=%{y}<extra></extra>",
         "legendgroup": "GAIA",
         "marker": {
          "color": "#3366CC",
          "pattern": {
           "shape": ""
          }
         },
         "name": "GAIA",
         "offsetgroup": "GAIA",
         "orientation": "v",
         "showlegend": true,
         "textposition": "outside",
         "texttemplate": "%{y:.0f}",
         "type": "bar",
         "x": [
          "GPT4",
          "Mixtral-8x7B-Instruct-v0.1",
          "GPT3.5",
          "SOLAR-10.7B",
          "OpenHermes-2.5-Mistral-7B",
          "Zephyr-7b-beta",
          "Llama-2-70b-chat"
         ],
         "xaxis": "x",
         "y": [
          40,
          16.25,
          16.25,
          7.5,
          10,
          0,
          0
         ],
         "yaxis": "y"
        },
        {
         "alignmentgroup": "True",
         "hovertemplate": "<b>Task</b>=GSM8K<br><b>Agent</b>=%{x}<br><b>Score</b>=%{y}<extra></extra>",
         "legendgroup": "GSM8K",
         "marker": {
          "color": "#DC3912",
          "pattern": {
           "shape": ""
          }
         },
         "name": "GSM8K",
         "offsetgroup": "GSM8K",
         "orientation": "v",
         "showlegend": true,
         "textposition": "outside",
         "texttemplate": "%{y:.0f}",
         "type": "bar",
         "x": [
          "GPT4",
          "Mixtral-8x7B-Instruct-v0.1",
          "GPT3.5",
          "SOLAR-10.7B",
          "OpenHermes-2.5-Mistral-7B",
          "Zephyr-7b-beta",
          "Llama-2-70b-chat"
         ],
         "xaxis": "x",
         "y": [
          95,
          72.5,
          62.5,
          73.75,
          46.25,
          46.25,
          17.5
         ],
         "yaxis": "y"
        },
        {
         "alignmentgroup": "True",
         "hovertemplate": "<b>Task</b>=HotpotQA<br><b>Agent</b>=%{x}<br><b>Score</b>=%{y}<extra></extra>",
         "legendgroup": "HotpotQA",
         "marker": {
          "color": "#FF9900",
          "pattern": {
           "shape": ""
          }
         },
         "name": "HotpotQA",
         "offsetgroup": "HotpotQA",
         "orientation": "v",
         "showlegend": true,
         "textposition": "outside",
         "texttemplate": "%{y:.0f}",
         "type": "bar",
         "x": [
          "GPT4",
          "Mixtral-8x7B-Instruct-v0.1",
          "GPT3.5",
          "SOLAR-10.7B",
          "OpenHermes-2.5-Mistral-7B",
          "Zephyr-7b-beta",
          "Llama-2-70b-chat"
         ],
         "xaxis": "x",
         "y": [
          80.83333333333333,
          77.08333333333334,
          74.58333333333333,
          53.75,
          65.83333333333333,
          36.666666666666664,
          49.583333333333336
         ],
         "yaxis": "y"
        }
       ],
       "layout": {
        "bargap": 0.35,
        "bargroupgap": 0,
        "barmode": "group",
        "height": 600,
        "legend": {
         "title": {
          "text": "<b>Task</b>"
         },
         "tracegroupgap": 0
        },
        "template": {
         "data": {
          "bar": [
           {
            "error_x": {
             "color": "#2a3f5f"
            },
            "error_y": {
             "color": "#2a3f5f"
            },
            "marker": {
             "line": {
              "color": "#E5ECF6",
              "width": 0.5
             },
             "pattern": {
              "fillmode": "overlay",
              "size": 10,
              "solidity": 0.2
             }
            },
            "type": "bar"
           }
          ],
          "barpolar": [
           {
            "marker": {
             "line": {
              "color": "#E5ECF6",
              "width": 0.5
             },
             "pattern": {
              "fillmode": "overlay",
              "size": 10,
              "solidity": 0.2
             }
            },
            "type": "barpolar"
           }
          ],
          "carpet": [
           {
            "aaxis": {
             "endlinecolor": "#2a3f5f",
             "gridcolor": "white",
             "linecolor": "white",
             "minorgridcolor": "white",
             "startlinecolor": "#2a3f5f"
            },
            "baxis": {
             "endlinecolor": "#2a3f5f",
             "gridcolor": "white",
             "linecolor": "white",
             "minorgridcolor": "white",
             "startlinecolor": "#2a3f5f"
            },
            "type": "carpet"
           }
          ],
          "choropleth": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "choropleth"
           }
          ],
          "contour": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "contour"
           }
          ],
          "contourcarpet": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "contourcarpet"
           }
          ],
          "heatmap": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "heatmap"
           }
          ],
          "heatmapgl": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "heatmapgl"
           }
          ],
          "histogram": [
           {
            "marker": {
             "pattern": {
              "fillmode": "overlay",
              "size": 10,
              "solidity": 0.2
             }
            },
            "type": "histogram"
           }
          ],
          "histogram2d": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "histogram2d"
           }
          ],
          "histogram2dcontour": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "histogram2dcontour"
           }
          ],
          "mesh3d": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "mesh3d"
           }
          ],
          "parcoords": [
           {
            "line": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "parcoords"
           }
          ],
          "pie": [
           {
            "automargin": true,
            "type": "pie"
           }
          ],
          "scatter": [
           {
            "fillpattern": {
             "fillmode": "overlay",
             "size": 10,
             "solidity": 0.2
            },
            "type": "scatter"
           }
          ],
          "scatter3d": [
           {
            "line": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatter3d"
           }
          ],
          "scattercarpet": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattercarpet"
           }
          ],
          "scattergeo": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattergeo"
           }
          ],
          "scattergl": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattergl"
           }
          ],
          "scattermapbox": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattermapbox"
           }
          ],
          "scatterpolar": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterpolar"
           }
          ],
          "scatterpolargl": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterpolargl"
           }
          ],
          "scatterternary": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterternary"
           }
          ],
          "surface": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "surface"
           }
          ],
          "table": [
           {
            "cells": {
             "fill": {
              "color": "#EBF0F8"
             },
             "line": {
              "color": "white"
             }
            },
            "header": {
             "fill": {
              "color": "#C8D4E3"
             },
             "line": {
              "color": "white"
             }
            },
            "type": "table"
           }
          ]
         },
         "layout": {
          "annotationdefaults": {
           "arrowcolor": "#2a3f5f",
           "arrowhead": 0,
           "arrowwidth": 1
          },
          "autotypenumbers": "strict",
          "coloraxis": {
           "colorbar": {
            "outlinewidth": 0,
            "ticks": ""
           }
          },
          "colorscale": {
           "diverging": [
            [
             0,
             "#8e0152"
            ],
            [
             0.1,
             "#c51b7d"
            ],
            [
             0.2,
             "#de77ae"
            ],
            [
             0.3,
             "#f1b6da"
            ],
            [
             0.4,
             "#fde0ef"
            ],
            [
             0.5,
             "#f7f7f7"
            ],
            [
             0.6,
             "#e6f5d0"
            ],
            [
             0.7,
             "#b8e186"
            ],
            [
             0.8,
             "#7fbc41"
            ],
            [
             0.9,
             "#4d9221"
            ],
            [
             1,
             "#276419"
            ]
           ],
           "sequential": [
            [
             0,
             "#0d0887"
            ],
            [
             0.1111111111111111,
             "#46039f"
            ],
            [
             0.2222222222222222,
             "#7201a8"
            ],
            [
             0.3333333333333333,
             "#9c179e"
            ],
            [
             0.4444444444444444,
             "#bd3786"
            ],
            [
             0.5555555555555556,
             "#d8576b"
            ],
            [
             0.6666666666666666,
             "#ed7953"
            ],
            [
             0.7777777777777778,
             "#fb9f3a"
            ],
            [
             0.8888888888888888,
             "#fdca26"
            ],
            [
             1,
             "#f0f921"
            ]
           ],
           "sequentialminus": [
            [
             0,
             "#0d0887"
            ],
            [
             0.1111111111111111,
             "#46039f"
            ],
            [
             0.2222222222222222,
             "#7201a8"
            ],
            [
             0.3333333333333333,
             "#9c179e"
            ],
            [
             0.4444444444444444,
             "#bd3786"
            ],
            [
             0.5555555555555556,
             "#d8576b"
            ],
            [
             0.6666666666666666,
             "#ed7953"
            ],
            [
             0.7777777777777778,
             "#fb9f3a"
            ],
            [
             0.8888888888888888,
             "#fdca26"
            ],
            [
             1,
             "#f0f921"
            ]
           ]
          },
          "colorway": [
           "#636efa",
           "#EF553B",
           "#00cc96",
           "#ab63fa",
           "#FFA15A",
           "#19d3f3",
           "#FF6692",
           "#B6E880",
           "#FF97FF",
           "#FECB52"
          ],
          "font": {
           "color": "#2a3f5f"
          },
          "geo": {
           "bgcolor": "white",
           "lakecolor": "white",
           "landcolor": "#E5ECF6",
           "showlakes": true,
           "showland": true,
           "subunitcolor": "white"
          },
          "hoverlabel": {
           "align": "left"
          },
          "hovermode": "closest",
          "mapbox": {
           "style": "light"
          },
          "paper_bgcolor": "white",
          "plot_bgcolor": "#E5ECF6",
          "polar": {
           "angularaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "bgcolor": "#E5ECF6",
           "radialaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           }
          },
          "scene": {
           "xaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           },
           "yaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           },
           "zaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           }
          },
          "shapedefaults": {
           "line": {
            "color": "#2a3f5f"
           }
          },
          "ternary": {
           "aaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "baxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "bgcolor": "#E5ECF6",
           "caxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           }
          },
          "title": {
           "x": 0.05
          },
          "xaxis": {
           "automargin": true,
           "gridcolor": "white",
           "linecolor": "white",
           "ticks": "",
           "title": {
            "standoff": 15
           },
           "zerolinecolor": "white",
           "zerolinewidth": 2
          },
          "yaxis": {
           "automargin": true,
           "gridcolor": "white",
           "linecolor": "white",
           "ticks": "",
           "title": {
            "standoff": 15
           },
           "zerolinecolor": "white",
           "zerolinewidth": 2
          }
         }
        },
        "title": {
         "text": "<b>Average Evaluation Score (LLM-as-a-judge)</b>"
        },
        "width": 1000,
        "xaxis": {
         "anchor": "y",
         "domain": [
          0,
          1
         ],
         "title": {
          "text": "<b>Agent</b>"
         }
        },
        "yaxis": {
         "anchor": "x",
         "domain": [
          0,
          1
         ],
         "range": [
          0,
          100
         ],
         "ticksuffix": "%",
         "title": {
          "text": "<b>Score</b>"
         }
        }
       }
      },
      "text/html": [
       "<div>                            <div id=\"cb14ba4e-cbff-4b02-95ea-b9c4bc73b0fb\" class=\"plotly-graph-div\" style=\"height:600px; width:1000px;\"></div>            <script type=\"text/javascript\">                require([\"plotly\"], function(Plotly) {                    window.PLOTLYENV=window.PLOTLYENV || {};                                    if (document.getElementById(\"cb14ba4e-cbff-4b02-95ea-b9c4bc73b0fb\")) {                    Plotly.newPlot(                        \"cb14ba4e-cbff-4b02-95ea-b9c4bc73b0fb\",                        [{\"alignmentgroup\":\"True\",\"hovertemplate\":\"\\u003cb\\u003eTask\\u003c\\u002fb\\u003e=GAIA\\u003cbr\\u003e\\u003cb\\u003eAgent\\u003c\\u002fb\\u003e=%{x}\\u003cbr\\u003e\\u003cb\\u003eScore\\u003c\\u002fb\\u003e=%{y}\\u003cextra\\u003e\\u003c\\u002fextra\\u003e\",\"legendgroup\":\"GAIA\",\"marker\":{\"color\":\"#3366CC\",\"pattern\":{\"shape\":\"\"}},\"name\":\"GAIA\",\"offsetgroup\":\"GAIA\",\"orientation\":\"v\",\"showlegend\":true,\"textposition\":\"outside\",\"x\":[\"GPT4\",\"Mixtral-8x7B-Instruct-v0.1\",\"GPT3.5\",\"SOLAR-10.7B\",\"OpenHermes-2.5-Mistral-7B\",\"Zephyr-7b-beta\",\"Llama-2-70b-chat\"],\"xaxis\":\"x\",\"y\":[40.0,16.25,16.25,7.5,10.0,0.0,0.0],\"yaxis\":\"y\",\"type\":\"bar\",\"texttemplate\":\"%{y:.0f}\"},{\"alignmentgroup\":\"True\",\"hovertemplate\":\"\\u003cb\\u003eTask\\u003c\\u002fb\\u003e=GSM8K\\u003cbr\\u003e\\u003cb\\u003eAgent\\u003c\\u002fb\\u003e=%{x}\\u003cbr\\u003e\\u003cb\\u003eScore\\u003c\\u002fb\\u003e=%{y}\\u003cextra\\u003e\\u003c\\u002fextra\\u003e\",\"legendgroup\":\"GSM8K\",\"marker\":{\"color\":\"#DC3912\",\"pattern\":{\"shape\":\"\"}},\"name\":\"GSM8K\",\"offsetgroup\":\"GSM8K\",\"orientation\":\"v\",\"showlegend\":true,\"textposition\":\"outside\",\"x\":[\"GPT4\",\"Mixtral-8x7B-Instruct-v0.1\",\"GPT3.5\",\"SOLAR-10.7B\",\"OpenHermes-2.5-Mistral-7B\",\"Zephyr-7b-beta\",\"Llama-2-70b-chat\"],\"xaxis\":\"x\",\"y\":[95.0,72.5,62.5,73.75,46.25,46.25,17.5],\"yaxis\":\"y\",\"type\":\"bar\",\"texttemplate\":\"%{y:.0f}\"},{\"alignmentgroup\":\"True\",\"hovertemplate\":\"\\u003cb\\u003eTask\\u003c\\u002fb\\u003e=HotpotQA\\u003cbr\\u003e\\u003cb\\u003eAgent\\u003c\\u002fb\\u003e=%{x}\\u003cbr\\u003e\\u003cb\\u003eScore\\u003c\\u002fb\\u003e=%{y}\\u003cextra\\u003e\\u003c\\u002fextra\\u003e\",\"legendgroup\":\"HotpotQA\",\"marker\":{\"color\":\"#FF9900\",\"pattern\":{\"shape\":\"\"}},\"name\":\"HotpotQA\",\"offsetgroup\":\"HotpotQA\",\"orientation\":\"v\",\"showlegend\":true,\"textposition\":\"outside\",\"x\":[\"GPT4\",\"Mixtral-8x7B-Instruct-v0.1\",\"GPT3.5\",\"SOLAR-10.7B\",\"OpenHermes-2.5-Mistral-7B\",\"Zephyr-7b-beta\",\"Llama-2-70b-chat\"],\"xaxis\":\"x\",\"y\":[80.83333333333333,77.08333333333334,74.58333333333333,53.75,65.83333333333333,36.666666666666664,49.583333333333336],\"yaxis\":\"y\",\"type\":\"bar\",\"texttemplate\":\"%{y:.0f}\"}],                        {\"template\":{\"data\":{\"histogram2dcontour\":[{\"type\":\"histogram2dcontour\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"choropleth\":[{\"type\":\"choropleth\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}],\"histogram2d\":[{\"type\":\"histogram2d\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"heatmap\":[{\"type\":\"heatmap\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"heatmapgl\":[{\"type\":\"heatmapgl\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"contourcarpet\":[{\"type\":\"contourcarpet\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}],\"contour\":[{\"type\":\"contour\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"surface\":[{\"type\":\"surface\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"mesh3d\":[{\"type\":\"mesh3d\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}],\"scatter\":[{\"fillpattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2},\"type\":\"scatter\"}],\"parcoords\":[{\"type\":\"parcoords\",\"line\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scatterpolargl\":[{\"type\":\"scatterpolargl\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"bar\":[{\"error_x\":{\"color\":\"#2a3f5f\"},\"error_y\":{\"color\":\"#2a3f5f\"},\"marker\":{\"line\":{\"color\":\"#E5ECF6\",\"width\":0.5},\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"bar\"}],\"scattergeo\":[{\"type\":\"scattergeo\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scatterpolar\":[{\"type\":\"scatterpolar\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"histogram\":[{\"marker\":{\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"histogram\"}],\"scattergl\":[{\"type\":\"scattergl\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scatter3d\":[{\"type\":\"scatter3d\",\"line\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scattermapbox\":[{\"type\":\"scattermapbox\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scatterternary\":[{\"type\":\"scatterternary\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scattercarpet\":[{\"type\":\"scattercarpet\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"carpet\":[{\"aaxis\":{\"endlinecolor\":\"#2a3f5f\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"minorgridcolor\":\"white\",\"startlinecolor\":\"#2a3f5f\"},\"baxis\":{\"endlinecolor\":\"#2a3f5f\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"minorgridcolor\":\"white\",\"startlinecolor\":\"#2a3f5f\"},\"type\":\"carpet\"}],\"table\":[{\"cells\":{\"fill\":{\"color\":\"#EBF0F8\"},\"line\":{\"color\":\"white\"}},\"header\":{\"fill\":{\"color\":\"#C8D4E3\"},\"line\":{\"color\":\"white\"}},\"type\":\"table\"}],\"barpolar\":[{\"marker\":{\"line\":{\"color\":\"#E5ECF6\",\"width\":0.5},\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"barpolar\"}],\"pie\":[{\"automargin\":true,\"type\":\"pie\"}]},\"layout\":{\"autotypenumbers\":\"strict\",\"colorway\":[\"#636efa\",\"#EF553B\",\"#00cc96\",\"#ab63fa\",\"#FFA15A\",\"#19d3f3\",\"#FF6692\",\"#B6E880\",\"#FF97FF\",\"#FECB52\"],\"font\":{\"color\":\"#2a3f5f\"},\"hovermode\":\"closest\",\"hoverlabel\":{\"align\":\"left\"},\"paper_bgcolor\":\"white\",\"plot_bgcolor\":\"#E5ECF6\",\"polar\":{\"bgcolor\":\"#E5ECF6\",\"angularaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"},\"radialaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"}},\"ternary\":{\"bgcolor\":\"#E5ECF6\",\"aaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"},\"baxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"},\"caxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"}},\"coloraxis\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"colorscale\":{\"sequential\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"sequentialminus\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"diverging\":[[0,\"#8e0152\"],[0.1,\"#c51b7d\"],[0.2,\"#de77ae\"],[0.3,\"#f1b6da\"],[0.4,\"#fde0ef\"],[0.5,\"#f7f7f7\"],[0.6,\"#e6f5d0\"],[0.7,\"#b8e186\"],[0.8,\"#7fbc41\"],[0.9,\"#4d9221\"],[1,\"#276419\"]]},\"xaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\",\"title\":{\"standoff\":15},\"zerolinecolor\":\"white\",\"automargin\":true,\"zerolinewidth\":2},\"yaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\",\"title\":{\"standoff\":15},\"zerolinecolor\":\"white\",\"automargin\":true,\"zerolinewidth\":2},\"scene\":{\"xaxis\":{\"backgroundcolor\":\"#E5ECF6\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"showbackground\":true,\"ticks\":\"\",\"zerolinecolor\":\"white\",\"gridwidth\":2},\"yaxis\":{\"backgroundcolor\":\"#E5ECF6\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"showbackground\":true,\"ticks\":\"\",\"zerolinecolor\":\"white\",\"gridwidth\":2},\"zaxis\":{\"backgroundcolor\":\"#E5ECF6\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"showbackground\":true,\"ticks\":\"\",\"zerolinecolor\":\"white\",\"gridwidth\":2}},\"shapedefaults\":{\"line\":{\"color\":\"#2a3f5f\"}},\"annotationdefaults\":{\"arrowcolor\":\"#2a3f5f\",\"arrowhead\":0,\"arrowwidth\":1},\"geo\":{\"bgcolor\":\"white\",\"landcolor\":\"#E5ECF6\",\"subunitcolor\":\"white\",\"showland\":true,\"showlakes\":true,\"lakecolor\":\"white\"},\"title\":{\"x\":0.05},\"mapbox\":{\"style\":\"light\"}}},\"xaxis\":{\"anchor\":\"y\",\"domain\":[0.0,1.0],\"title\":{\"text\":\"\\u003cb\\u003eAgent\\u003c\\u002fb\\u003e\"}},\"yaxis\":{\"anchor\":\"x\",\"domain\":[0.0,1.0],\"title\":{\"text\":\"\\u003cb\\u003eScore\\u003c\\u002fb\\u003e\"},\"range\":[0,100],\"ticksuffix\":\"%\"},\"legend\":{\"title\":{\"text\":\"\\u003cb\\u003eTask\\u003c\\u002fb\\u003e\"},\"tracegroupgap\":0},\"title\":{\"text\":\"\\u003cb\\u003eAverage Evaluation Score (LLM-as-a-judge)\\u003c\\u002fb\\u003e\"},\"barmode\":\"group\",\"width\":1000,\"height\":600,\"bargap\":0.35,\"bargroupgap\":0.0},                        {\"responsive\": true}                    ).then(function(){\n",
       "                            \n",
       "var gd = document.getElementById('cb14ba4e-cbff-4b02-95ea-b9c4bc73b0fb');\n",
       "var x = new MutationObserver(function (mutations, observer) {{\n",
       "        var display = window.getComputedStyle(gd).display;\n",
       "        if (!display || display === 'none') {{\n",
       "            console.log([gd, 'removed!']);\n",
       "            Plotly.purge(gd);\n",
       "            observer.disconnect();\n",
       "        }}\n",
       "}});\n",
       "\n",
       "// Listen for the removal of the full notebook cells\n",
       "var notebookContainer = gd.closest('#notebook-container');\n",
       "if (notebookContainer) {{\n",
       "    x.observe(notebookContainer, {childList: true});\n",
       "}}\n",
       "\n",
       "// Listen for the clearing of the current output cell\n",
       "var outputEl = gd.closest('.output');\n",
       "if (outputEl) {{\n",
       "    x.observe(outputEl, {childList: true});\n",
       "}}\n",
       "\n",
       "                        })                };                });            </script>        </div>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import kaleido\n",
    "\n",
    "sorter = [\n",
    "    \"GPT4\",\n",
    "    \"Mixtral-8x7B-Instruct-v0.1\",\n",
    "    \"GPT3.5\",\n",
    "    \"SOLAR-10.7B\",\n",
    "    \"OpenHermes-2.5-Mistral-7B\",\n",
    "    \"Zephyr-7b-beta\",\n",
    "    \"Llama-2-70b-chat\",\n",
    "]\n",
    "table_result = table_result.sort_values(\n",
    "    \"agent_name\", key=lambda column: column.map(lambda e: sorter.index(e))\n",
    ")\n",
    "# Plot results\n",
    "fig = px.bar(\n",
    "    table_result,\n",
    "    x=\"agent_name\",\n",
    "    y=\"eval_score_GPT4\",\n",
    "    color=\"task\",\n",
    "    title=f\"<b>Average Evaluation Score (LLM-as-a-judge)</b>\",\n",
    "    labels={\n",
    "        \"agent_name\": \"<b>Agent</b>\",\n",
    "        \"task\": \"<b>Task</b>\",\n",
    "        \"score\": \"Performance\",\n",
    "        \"eval_score_GPT4\": \"<b>Score</b>\",\n",
    "    },\n",
    "    color_discrete_sequence=px.colors.qualitative.G10,\n",
    ")\n",
    "fig.update_layout(\n",
    "    width=1000,\n",
    "    height=600,\n",
    "    barmode=\"group\",\n",
    "    bargap=0.35,\n",
    "    bargroupgap=0.0,\n",
    "    yaxis_range=[0, 100],\n",
    ")\n",
    "fig.update_traces(texttemplate=\"%{y:.0f}\", textposition=\"outside\")\n",
    "fig.layout.yaxis.ticksuffix = \"%\"\n",
    "fig.write_image(\"benchmark_agents.png\", scale=4)\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Study intermediate steps"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/tmp/ipykernel_46633/2687269850.py:9: SettingWithCopyWarning:\n",
      "\n",
      "\n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "\n",
      "/tmp/ipykernel_46633/2687269850.py:12: SettingWithCopyWarning:\n",
      "\n",
      "\n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    result_df[\"tools_used\"] = result_df[\"intermediate_steps\"].apply(\n",
    "        lambda row: ([step[\"tool\"] for step in row] if row is not None else None)\n",
    "    )\n",
    "except:\n",
    "    pass\n",
    "result_df[\"has_answer\"] = result_df[\"prediction\"].apply(lambda row: (row is not None))\n",
    "result_df_answers_only = result_df.loc[result_df[\"has_answer\"] == True]\n",
    "result_df_answers_only[\"correct_answer\"] = (\n",
    "    result_df_answers_only[f\"eval_score_{eval_model_name}\"] >= 4\n",
    ")\n",
    "result_df_answers_only[\"number_of_steps\"] = result_df_answers_only[\"tools_used\"].apply(\n",
    "    lambda x: len(x) + 1\n",
    ")\n",
    "aggregated_resuts = result_df_answers_only.groupby(\n",
    "    [\"agent_name\", \"task\", \"correct_answer\"]\n",
    ").agg({\"number_of_steps\": \"mean\"})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.plotly.v1+json": {
       "config": {
        "plotlyServerURL": "https://plot.ly"
       },
       "data": [
        {
         "alignmentgroup": "True",
         "hovertemplate": "Answer correctness=False<br>Agent=GPT3.5<br>Task=%{x}<br>Mean Number of Steps=%{y}<extra></extra>",
         "legendgroup": "False",
         "marker": {
          "color": "#636efa",
          "pattern": {
           "shape": ""
          }
         },
         "name": "False",
         "offsetgroup": "False",
         "orientation": "v",
         "showlegend": true,
         "textposition": "auto",
         "type": "bar",
         "x": [
          "GAIA",
          "GSM8K",
          "HotpotQA"
         ],
         "xaxis": "x",
         "y": [
          2.111111111111111,
          2.125,
          1.7
         ],
         "yaxis": "y"
        },
        {
         "alignmentgroup": "True",
         "hovertemplate": "Answer correctness=False<br>Agent=GPT4<br>Task=%{x}<br>Mean Number of Steps=%{y}<extra></extra>",
         "legendgroup": "False",
         "marker": {
          "color": "#636efa",
          "pattern": {
           "shape": ""
          }
         },
         "name": "False",
         "offsetgroup": "False",
         "orientation": "v",
         "showlegend": false,
         "textposition": "auto",
         "type": "bar",
         "x": [
          "GAIA",
          "GSM8K",
          "HotpotQA"
         ],
         "xaxis": "x2",
         "y": [
          2.3846153846153846,
          3,
          1.8823529411764706
         ],
         "yaxis": "y2"
        },
        {
         "alignmentgroup": "True",
         "hovertemplate": "Answer correctness=False<br>Agent=Llama-2-70b-chat<br>Task=%{x}<br>Mean Number of Steps=%{y}<extra></extra>",
         "legendgroup": "False",
         "marker": {
          "color": "#636efa",
          "pattern": {
           "shape": ""
          }
         },
         "name": "False",
         "offsetgroup": "False",
         "orientation": "v",
         "showlegend": false,
         "textposition": "auto",
         "type": "bar",
         "x": [
          "GAIA",
          "GSM8K",
          "HotpotQA"
         ],
         "xaxis": "x3",
         "y": [
          5.428571428571429,
          4.571428571428571,
          4.5
         ],
         "yaxis": "y3"
        },
        {
         "alignmentgroup": "True",
         "hovertemplate": "Answer correctness=False<br>Agent=Mixtral-8x7B-Instruct-v0.1<br>Task=%{x}<br>Mean Number of Steps=%{y}<extra></extra>",
         "legendgroup": "False",
         "marker": {
          "color": "#636efa",
          "pattern": {
           "shape": ""
          }
         },
         "name": "False",
         "offsetgroup": "False",
         "orientation": "v",
         "showlegend": false,
         "textposition": "auto",
         "type": "bar",
         "x": [
          "GAIA",
          "GSM8K",
          "HotpotQA"
         ],
         "xaxis": "x4",
         "y": [
          4.5,
          4.166666666666667,
          3.409090909090909
         ],
         "yaxis": "y4"
        },
        {
         "alignmentgroup": "True",
         "hovertemplate": "Answer correctness=False<br>Agent=OpenHermes-2.5-Mistral-7B<br>Task=%{x}<br>Mean Number of Steps=%{y}<extra></extra>",
         "legendgroup": "False",
         "marker": {
          "color": "#636efa",
          "pattern": {
           "shape": ""
          }
         },
         "name": "False",
         "offsetgroup": "False",
         "orientation": "v",
         "showlegend": false,
         "textposition": "auto",
         "type": "bar",
         "x": [
          "GAIA",
          "GSM8K",
          "HotpotQA"
         ],
         "xaxis": "x5",
         "y": [
          5.176470588235294,
          4.181818181818182,
          3.5
         ],
         "yaxis": "y5"
        },
        {
         "alignmentgroup": "True",
         "hovertemplate": "Answer correctness=False<br>Agent=SOLAR-10.7B<br>Task=%{x}<br>Mean Number of Steps=%{y}<extra></extra>",
         "legendgroup": "False",
         "marker": {
          "color": "#636efa",
          "pattern": {
           "shape": ""
          }
         },
         "name": "False",
         "offsetgroup": "False",
         "orientation": "v",
         "showlegend": false,
         "textposition": "auto",
         "type": "bar",
         "x": [
          "GAIA",
          "GSM8K",
          "HotpotQA"
         ],
         "xaxis": "x6",
         "y": [
          4.375,
          2.2,
          4.258064516129032
         ],
         "yaxis": "y6"
        },
        {
         "alignmentgroup": "True",
         "hovertemplate": "Answer correctness=False<br>Agent=Zephyr-7b-beta<br>Task=%{x}<br>Mean Number of Steps=%{y}<extra></extra>",
         "legendgroup": "False",
         "marker": {
          "color": "#636efa",
          "pattern": {
           "shape": ""
          }
         },
         "name": "False",
         "offsetgroup": "False",
         "orientation": "v",
         "showlegend": false,
         "textposition": "auto",
         "type": "bar",
         "x": [
          "GAIA",
          "GSM8K",
          "HotpotQA"
         ],
         "xaxis": "x7",
         "y": [
          5.2,
          2.4,
          4.868421052631579
         ],
         "yaxis": "y7"
        },
        {
         "alignmentgroup": "True",
         "hovertemplate": "Answer correctness=True<br>Agent=GPT3.5<br>Task=%{x}<br>Mean Number of Steps=%{y}<extra></extra>",
         "legendgroup": "True",
         "marker": {
          "color": "#EF553B",
          "pattern": {
           "shape": ""
          }
         },
         "name": "True",
         "offsetgroup": "True",
         "orientation": "v",
         "showlegend": true,
         "textposition": "auto",
         "type": "bar",
         "x": [
          "GAIA",
          "GSM8K",
          "HotpotQA"
         ],
         "xaxis": "x",
         "y": [
          1,
          2.1666666666666665,
          1.875
         ],
         "yaxis": "y"
        },
        {
         "alignmentgroup": "True",
         "hovertemplate": "Answer correctness=True<br>Agent=GPT4<br>Task=%{x}<br>Mean Number of Steps=%{y}<extra></extra>",
         "legendgroup": "True",
         "marker": {
          "color": "#EF553B",
          "pattern": {
           "shape": ""
          }
         },
         "name": "True",
         "offsetgroup": "True",
         "orientation": "v",
         "showlegend": false,
         "textposition": "auto",
         "type": "bar",
         "x": [
          "GAIA",
          "GSM8K",
          "HotpotQA"
         ],
         "xaxis": "x2",
         "y": [
          1.6666666666666667,
          2.473684210526316,
          1.6511627906976745
         ],
         "yaxis": "y2"
        },
        {
         "alignmentgroup": "True",
         "hovertemplate": "Answer correctness=True<br>Agent=Llama-2-70b-chat<br>Task=%{x}<br>Mean Number of Steps=%{y}<extra></extra>",
         "legendgroup": "True",
         "marker": {
          "color": "#EF553B",
          "pattern": {
           "shape": ""
          }
         },
         "name": "True",
         "offsetgroup": "True",
         "orientation": "v",
         "showlegend": false,
         "textposition": "auto",
         "type": "bar",
         "x": [
          "GSM8K",
          "HotpotQA"
         ],
         "xaxis": "x3",
         "y": [
          2,
          2.88
         ],
         "yaxis": "y3"
        },
        {
         "alignmentgroup": "True",
         "hovertemplate": "Answer correctness=True<br>Agent=Mixtral-8x7B-Instruct-v0.1<br>Task=%{x}<br>Mean Number of Steps=%{y}<extra></extra>",
         "legendgroup": "True",
         "marker": {
          "color": "#EF553B",
          "pattern": {
           "shape": ""
          }
         },
         "name": "True",
         "offsetgroup": "True",
         "orientation": "v",
         "showlegend": false,
         "textposition": "auto",
         "type": "bar",
         "x": [
          "GAIA",
          "GSM8K",
          "HotpotQA"
         ],
         "xaxis": "x4",
         "y": [
          3,
          3.0714285714285716,
          3.027027027027027
         ],
         "yaxis": "y4"
        },
        {
         "alignmentgroup": "True",
         "hovertemplate": "Answer correctness=True<br>Agent=OpenHermes-2.5-Mistral-7B<br>Task=%{x}<br>Mean Number of Steps=%{y}<extra></extra>",
         "legendgroup": "True",
         "marker": {
          "color": "#EF553B",
          "pattern": {
           "shape": ""
          }
         },
         "name": "True",
         "offsetgroup": "True",
         "orientation": "v",
         "showlegend": false,
         "textposition": "auto",
         "type": "bar",
         "x": [
          "GAIA",
          "GSM8K",
          "HotpotQA"
         ],
         "xaxis": "x5",
         "y": [
          3.5,
          3.5555555555555554,
          3.0303030303030303
         ],
         "yaxis": "y5"
        },
        {
         "alignmentgroup": "True",
         "hovertemplate": "Answer correctness=True<br>Agent=SOLAR-10.7B<br>Task=%{x}<br>Mean Number of Steps=%{y}<extra></extra>",
         "legendgroup": "True",
         "marker": {
          "color": "#EF553B",
          "pattern": {
           "shape": ""
          }
         },
         "name": "True",
         "offsetgroup": "True",
         "orientation": "v",
         "showlegend": false,
         "textposition": "auto",
         "type": "bar",
         "x": [
          "GAIA",
          "GSM8K",
          "HotpotQA"
         ],
         "xaxis": "x6",
         "y": [
          4,
          2.357142857142857,
          2.896551724137931
         ],
         "yaxis": "y6"
        },
        {
         "alignmentgroup": "True",
         "hovertemplate": "Answer correctness=True<br>Agent=Zephyr-7b-beta<br>Task=%{x}<br>Mean Number of Steps=%{y}<extra></extra>",
         "legendgroup": "True",
         "marker": {
          "color": "#EF553B",
          "pattern": {
           "shape": ""
          }
         },
         "name": "True",
         "offsetgroup": "True",
         "orientation": "v",
         "showlegend": false,
         "textposition": "auto",
         "type": "bar",
         "x": [
          "GSM8K",
          "HotpotQA"
         ],
         "xaxis": "x7",
         "y": [
          1.75,
          2.45
         ],
         "yaxis": "y7"
        }
       ],
       "layout": {
        "annotations": [
         {
          "font": {},
          "showarrow": false,
          "text": "Agent=GPT3.5",
          "x": 0.06285714285714285,
          "xanchor": "center",
          "xref": "paper",
          "y": 1,
          "yanchor": "bottom",
          "yref": "paper"
         },
         {
          "font": {},
          "showarrow": false,
          "text": "Agent=GPT4",
          "x": 0.20857142857142852,
          "xanchor": "center",
          "xref": "paper",
          "y": 1,
          "yanchor": "bottom",
          "yref": "paper"
         },
         {
          "font": {},
          "showarrow": false,
          "text": "Agent=Llama-2-70b-chat",
          "x": 0.3542857142857142,
          "xanchor": "center",
          "xref": "paper",
          "y": 1,
          "yanchor": "bottom",
          "yref": "paper"
         },
         {
          "font": {},
          "showarrow": false,
          "text": "Agent=Mixtral-8x7B-Instruct-v0.1",
          "x": 0.5,
          "xanchor": "center",
          "xref": "paper",
          "y": 1,
          "yanchor": "bottom",
          "yref": "paper"
         },
         {
          "font": {},
          "showarrow": false,
          "text": "Agent=OpenHermes-2.5-Mistral-7B",
          "x": 0.6457142857142856,
          "xanchor": "center",
          "xref": "paper",
          "y": 1,
          "yanchor": "bottom",
          "yref": "paper"
         },
         {
          "font": {},
          "showarrow": false,
          "text": "Agent=SOLAR-10.7B",
          "x": 0.7914285714285713,
          "xanchor": "center",
          "xref": "paper",
          "y": 1,
          "yanchor": "bottom",
          "yref": "paper"
         },
         {
          "font": {},
          "showarrow": false,
          "text": "Agent=Zephyr-7b-beta",
          "x": 0.9371428571428569,
          "xanchor": "center",
          "xref": "paper",
          "y": 1,
          "yanchor": "bottom",
          "yref": "paper"
         }
        ],
        "barmode": "group",
        "legend": {
         "title": {
          "text": "Answer correctness"
         },
         "tracegroupgap": 0
        },
        "template": {
         "data": {
          "bar": [
           {
            "error_x": {
             "color": "#2a3f5f"
            },
            "error_y": {
             "color": "#2a3f5f"
            },
            "marker": {
             "line": {
              "color": "#E5ECF6",
              "width": 0.5
             },
             "pattern": {
              "fillmode": "overlay",
              "size": 10,
              "solidity": 0.2
             }
            },
            "type": "bar"
           }
          ],
          "barpolar": [
           {
            "marker": {
             "line": {
              "color": "#E5ECF6",
              "width": 0.5
             },
             "pattern": {
              "fillmode": "overlay",
              "size": 10,
              "solidity": 0.2
             }
            },
            "type": "barpolar"
           }
          ],
          "carpet": [
           {
            "aaxis": {
             "endlinecolor": "#2a3f5f",
             "gridcolor": "white",
             "linecolor": "white",
             "minorgridcolor": "white",
             "startlinecolor": "#2a3f5f"
            },
            "baxis": {
             "endlinecolor": "#2a3f5f",
             "gridcolor": "white",
             "linecolor": "white",
             "minorgridcolor": "white",
             "startlinecolor": "#2a3f5f"
            },
            "type": "carpet"
           }
          ],
          "choropleth": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "choropleth"
           }
          ],
          "contour": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "contour"
           }
          ],
          "contourcarpet": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "contourcarpet"
           }
          ],
          "heatmap": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "heatmap"
           }
          ],
          "heatmapgl": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "heatmapgl"
           }
          ],
          "histogram": [
           {
            "marker": {
             "pattern": {
              "fillmode": "overlay",
              "size": 10,
              "solidity": 0.2
             }
            },
            "type": "histogram"
           }
          ],
          "histogram2d": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "histogram2d"
           }
          ],
          "histogram2dcontour": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "histogram2dcontour"
           }
          ],
          "mesh3d": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "mesh3d"
           }
          ],
          "parcoords": [
           {
            "line": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "parcoords"
           }
          ],
          "pie": [
           {
            "automargin": true,
            "type": "pie"
           }
          ],
          "scatter": [
           {
            "fillpattern": {
             "fillmode": "overlay",
             "size": 10,
             "solidity": 0.2
            },
            "type": "scatter"
           }
          ],
          "scatter3d": [
           {
            "line": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatter3d"
           }
          ],
          "scattercarpet": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattercarpet"
           }
          ],
          "scattergeo": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattergeo"
           }
          ],
          "scattergl": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattergl"
           }
          ],
          "scattermapbox": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattermapbox"
           }
          ],
          "scatterpolar": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterpolar"
           }
          ],
          "scatterpolargl": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterpolargl"
           }
          ],
          "scatterternary": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterternary"
           }
          ],
          "surface": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "surface"
           }
          ],
          "table": [
           {
            "cells": {
             "fill": {
              "color": "#EBF0F8"
             },
             "line": {
              "color": "white"
             }
            },
            "header": {
             "fill": {
              "color": "#C8D4E3"
             },
             "line": {
              "color": "white"
             }
            },
            "type": "table"
           }
          ]
         },
         "layout": {
          "annotationdefaults": {
           "arrowcolor": "#2a3f5f",
           "arrowhead": 0,
           "arrowwidth": 1
          },
          "autotypenumbers": "strict",
          "coloraxis": {
           "colorbar": {
            "outlinewidth": 0,
            "ticks": ""
           }
          },
          "colorscale": {
           "diverging": [
            [
             0,
             "#8e0152"
            ],
            [
             0.1,
             "#c51b7d"
            ],
            [
             0.2,
             "#de77ae"
            ],
            [
             0.3,
             "#f1b6da"
            ],
            [
             0.4,
             "#fde0ef"
            ],
            [
             0.5,
             "#f7f7f7"
            ],
            [
             0.6,
             "#e6f5d0"
            ],
            [
             0.7,
             "#b8e186"
            ],
            [
             0.8,
             "#7fbc41"
            ],
            [
             0.9,
             "#4d9221"
            ],
            [
             1,
             "#276419"
            ]
           ],
           "sequential": [
            [
             0,
             "#0d0887"
            ],
            [
             0.1111111111111111,
             "#46039f"
            ],
            [
             0.2222222222222222,
             "#7201a8"
            ],
            [
             0.3333333333333333,
             "#9c179e"
            ],
            [
             0.4444444444444444,
             "#bd3786"
            ],
            [
             0.5555555555555556,
             "#d8576b"
            ],
            [
             0.6666666666666666,
             "#ed7953"
            ],
            [
             0.7777777777777778,
             "#fb9f3a"
            ],
            [
             0.8888888888888888,
             "#fdca26"
            ],
            [
             1,
             "#f0f921"
            ]
           ],
           "sequentialminus": [
            [
             0,
             "#0d0887"
            ],
            [
             0.1111111111111111,
             "#46039f"
            ],
            [
             0.2222222222222222,
             "#7201a8"
            ],
            [
             0.3333333333333333,
             "#9c179e"
            ],
            [
             0.4444444444444444,
             "#bd3786"
            ],
            [
             0.5555555555555556,
             "#d8576b"
            ],
            [
             0.6666666666666666,
             "#ed7953"
            ],
            [
             0.7777777777777778,
             "#fb9f3a"
            ],
            [
             0.8888888888888888,
             "#fdca26"
            ],
            [
             1,
             "#f0f921"
            ]
           ]
          },
          "colorway": [
           "#636efa",
           "#EF553B",
           "#00cc96",
           "#ab63fa",
           "#FFA15A",
           "#19d3f3",
           "#FF6692",
           "#B6E880",
           "#FF97FF",
           "#FECB52"
          ],
          "font": {
           "color": "#2a3f5f"
          },
          "geo": {
           "bgcolor": "white",
           "lakecolor": "white",
           "landcolor": "#E5ECF6",
           "showlakes": true,
           "showland": true,
           "subunitcolor": "white"
          },
          "hoverlabel": {
           "align": "left"
          },
          "hovermode": "closest",
          "mapbox": {
           "style": "light"
          },
          "paper_bgcolor": "white",
          "plot_bgcolor": "#E5ECF6",
          "polar": {
           "angularaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "bgcolor": "#E5ECF6",
           "radialaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           }
          },
          "scene": {
           "xaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           },
           "yaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           },
           "zaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           }
          },
          "shapedefaults": {
           "line": {
            "color": "#2a3f5f"
           }
          },
          "ternary": {
           "aaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "baxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "bgcolor": "#E5ECF6",
           "caxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           }
          },
          "title": {
           "x": 0.05
          },
          "xaxis": {
           "automargin": true,
           "gridcolor": "white",
           "linecolor": "white",
           "ticks": "",
           "title": {
            "standoff": 15
           },
           "zerolinecolor": "white",
           "zerolinewidth": 2
          },
          "yaxis": {
           "automargin": true,
           "gridcolor": "white",
           "linecolor": "white",
           "ticks": "",
           "title": {
            "standoff": 15
           },
           "zerolinecolor": "white",
           "zerolinewidth": 2
          }
         }
        },
        "title": {
         "text": "Mean Number of Steps to Solve a Task"
        },
        "width": 1500,
        "xaxis": {
         "anchor": "y",
         "domain": [
          0,
          0.1257142857142857
         ],
         "title": {
          "text": "Task"
         }
        },
        "xaxis2": {
         "anchor": "y2",
         "domain": [
          0.14571428571428569,
          0.27142857142857135
         ],
         "matches": "x",
         "title": {
          "text": "Task"
         }
        },
        "xaxis3": {
         "anchor": "y3",
         "domain": [
          0.29142857142857137,
          0.41714285714285704
         ],
         "matches": "x",
         "title": {
          "text": "Task"
         }
        },
        "xaxis4": {
         "anchor": "y4",
         "domain": [
          0.4371428571428571,
          0.5628571428571428
         ],
         "matches": "x",
         "title": {
          "text": "Task"
         }
        },
        "xaxis5": {
         "anchor": "y5",
         "domain": [
          0.5828571428571427,
          0.7085714285714284
         ],
         "matches": "x",
         "title": {
          "text": "Task"
         }
        },
        "xaxis6": {
         "anchor": "y6",
         "domain": [
          0.7285714285714284,
          0.8542857142857141
         ],
         "matches": "x",
         "title": {
          "text": "Task"
         }
        },
        "xaxis7": {
         "anchor": "y7",
         "domain": [
          0.8742857142857141,
          0.9999999999999998
         ],
         "matches": "x",
         "title": {
          "text": "Task"
         }
        },
        "yaxis": {
         "anchor": "x",
         "domain": [
          0,
          1
         ],
         "title": {
          "text": "Mean Number of Steps"
         }
        },
        "yaxis2": {
         "anchor": "x2",
         "domain": [
          0,
          1
         ],
         "matches": "y",
         "showticklabels": false
        },
        "yaxis3": {
         "anchor": "x3",
         "domain": [
          0,
          1
         ],
         "matches": "y",
         "showticklabels": false
        },
        "yaxis4": {
         "anchor": "x4",
         "domain": [
          0,
          1
         ],
         "matches": "y",
         "showticklabels": false
        },
        "yaxis5": {
         "anchor": "x5",
         "domain": [
          0,
          1
         ],
         "matches": "y",
         "showticklabels": false
        },
        "yaxis6": {
         "anchor": "x6",
         "domain": [
          0,
          1
         ],
         "matches": "y",
         "showticklabels": false
        },
        "yaxis7": {
         "anchor": "x7",
         "domain": [
          0,
          1
         ],
         "matches": "y",
         "showticklabels": false
        }
       }
      },
      "text/html": [
       "<div>                            <div id=\"e23c2b41-b14d-4353-be6c-e12f08e3a387\" class=\"plotly-graph-div\" style=\"height:525px; width:1500px;\"></div>            <script type=\"text/javascript\">                require([\"plotly\"], function(Plotly) {                    window.PLOTLYENV=window.PLOTLYENV || {};                                    if (document.getElementById(\"e23c2b41-b14d-4353-be6c-e12f08e3a387\")) {                    Plotly.newPlot(                        \"e23c2b41-b14d-4353-be6c-e12f08e3a387\",                        [{\"alignmentgroup\":\"True\",\"hovertemplate\":\"Answer correctness=False\\u003cbr\\u003eAgent=GPT3.5\\u003cbr\\u003eTask=%{x}\\u003cbr\\u003eMean Number of Steps=%{y}\\u003cextra\\u003e\\u003c\\u002fextra\\u003e\",\"legendgroup\":\"False\",\"marker\":{\"color\":\"#636efa\",\"pattern\":{\"shape\":\"\"}},\"name\":\"False\",\"offsetgroup\":\"False\",\"orientation\":\"v\",\"showlegend\":true,\"textposition\":\"auto\",\"x\":[\"GAIA\",\"GSM8K\",\"HotpotQA\"],\"xaxis\":\"x\",\"y\":[2.111111111111111,2.125,1.7],\"yaxis\":\"y\",\"type\":\"bar\"},{\"alignmentgroup\":\"True\",\"hovertemplate\":\"Answer correctness=False\\u003cbr\\u003eAgent=GPT4\\u003cbr\\u003eTask=%{x}\\u003cbr\\u003eMean Number of Steps=%{y}\\u003cextra\\u003e\\u003c\\u002fextra\\u003e\",\"legendgroup\":\"False\",\"marker\":{\"color\":\"#636efa\",\"pattern\":{\"shape\":\"\"}},\"name\":\"False\",\"offsetgroup\":\"False\",\"orientation\":\"v\",\"showlegend\":false,\"textposition\":\"auto\",\"x\":[\"GAIA\",\"GSM8K\",\"HotpotQA\"],\"xaxis\":\"x2\",\"y\":[2.3846153846153846,3.0,1.8823529411764706],\"yaxis\":\"y2\",\"type\":\"bar\"},{\"alignmentgroup\":\"True\",\"hovertemplate\":\"Answer correctness=False\\u003cbr\\u003eAgent=Llama-2-70b-chat\\u003cbr\\u003eTask=%{x}\\u003cbr\\u003eMean Number of Steps=%{y}\\u003cextra\\u003e\\u003c\\u002fextra\\u003e\",\"legendgroup\":\"False\",\"marker\":{\"color\":\"#636efa\",\"pattern\":{\"shape\":\"\"}},\"name\":\"False\",\"offsetgroup\":\"False\",\"orientation\":\"v\",\"showlegend\":false,\"textposition\":\"auto\",\"x\":[\"GAIA\",\"GSM8K\",\"HotpotQA\"],\"xaxis\":\"x3\",\"y\":[5.428571428571429,4.571428571428571,4.5],\"yaxis\":\"y3\",\"type\":\"bar\"},{\"alignmentgroup\":\"True\",\"hovertemplate\":\"Answer correctness=False\\u003cbr\\u003eAgent=Mixtral-8x7B-Instruct-v0.1\\u003cbr\\u003eTask=%{x}\\u003cbr\\u003eMean Number of Steps=%{y}\\u003cextra\\u003e\\u003c\\u002fextra\\u003e\",\"legendgroup\":\"False\",\"marker\":{\"color\":\"#636efa\",\"pattern\":{\"shape\":\"\"}},\"name\":\"False\",\"offsetgroup\":\"False\",\"orientation\":\"v\",\"showlegend\":false,\"textposition\":\"auto\",\"x\":[\"GAIA\",\"GSM8K\",\"HotpotQA\"],\"xaxis\":\"x4\",\"y\":[4.5,4.166666666666667,3.409090909090909],\"yaxis\":\"y4\",\"type\":\"bar\"},{\"alignmentgroup\":\"True\",\"hovertemplate\":\"Answer correctness=False\\u003cbr\\u003eAgent=OpenHermes-2.5-Mistral-7B\\u003cbr\\u003eTask=%{x}\\u003cbr\\u003eMean Number of Steps=%{y}\\u003cextra\\u003e\\u003c\\u002fextra\\u003e\",\"legendgroup\":\"False\",\"marker\":{\"color\":\"#636efa\",\"pattern\":{\"shape\":\"\"}},\"name\":\"False\",\"offsetgroup\":\"False\",\"orientation\":\"v\",\"showlegend\":false,\"textposition\":\"auto\",\"x\":[\"GAIA\",\"GSM8K\",\"HotpotQA\"],\"xaxis\":\"x5\",\"y\":[5.176470588235294,4.181818181818182,3.5],\"yaxis\":\"y5\",\"type\":\"bar\"},{\"alignmentgroup\":\"True\",\"hovertemplate\":\"Answer correctness=False\\u003cbr\\u003eAgent=SOLAR-10.7B\\u003cbr\\u003eTask=%{x}\\u003cbr\\u003eMean Number of Steps=%{y}\\u003cextra\\u003e\\u003c\\u002fextra\\u003e\",\"legendgroup\":\"False\",\"marker\":{\"color\":\"#636efa\",\"pattern\":{\"shape\":\"\"}},\"name\":\"False\",\"offsetgroup\":\"False\",\"orientation\":\"v\",\"showlegend\":false,\"textposition\":\"auto\",\"x\":[\"GAIA\",\"GSM8K\",\"HotpotQA\"],\"xaxis\":\"x6\",\"y\":[4.375,2.2,4.258064516129032],\"yaxis\":\"y6\",\"type\":\"bar\"},{\"alignmentgroup\":\"True\",\"hovertemplate\":\"Answer correctness=False\\u003cbr\\u003eAgent=Zephyr-7b-beta\\u003cbr\\u003eTask=%{x}\\u003cbr\\u003eMean Number of Steps=%{y}\\u003cextra\\u003e\\u003c\\u002fextra\\u003e\",\"legendgroup\":\"False\",\"marker\":{\"color\":\"#636efa\",\"pattern\":{\"shape\":\"\"}},\"name\":\"False\",\"offsetgroup\":\"False\",\"orientation\":\"v\",\"showlegend\":false,\"textposition\":\"auto\",\"x\":[\"GAIA\",\"GSM8K\",\"HotpotQA\"],\"xaxis\":\"x7\",\"y\":[5.2,2.4,4.868421052631579],\"yaxis\":\"y7\",\"type\":\"bar\"},{\"alignmentgroup\":\"True\",\"hovertemplate\":\"Answer correctness=True\\u003cbr\\u003eAgent=GPT3.5\\u003cbr\\u003eTask=%{x}\\u003cbr\\u003eMean Number of Steps=%{y}\\u003cextra\\u003e\\u003c\\u002fextra\\u003e\",\"legendgroup\":\"True\",\"marker\":{\"color\":\"#EF553B\",\"pattern\":{\"shape\":\"\"}},\"name\":\"True\",\"offsetgroup\":\"True\",\"orientation\":\"v\",\"showlegend\":true,\"textposition\":\"auto\",\"x\":[\"GAIA\",\"GSM8K\",\"HotpotQA\"],\"xaxis\":\"x\",\"y\":[1.0,2.1666666666666665,1.875],\"yaxis\":\"y\",\"type\":\"bar\"},{\"alignmentgroup\":\"True\",\"hovertemplate\":\"Answer correctness=True\\u003cbr\\u003eAgent=GPT4\\u003cbr\\u003eTask=%{x}\\u003cbr\\u003eMean Number of Steps=%{y}\\u003cextra\\u003e\\u003c\\u002fextra\\u003e\",\"legendgroup\":\"True\",\"marker\":{\"color\":\"#EF553B\",\"pattern\":{\"shape\":\"\"}},\"name\":\"True\",\"offsetgroup\":\"True\",\"orientation\":\"v\",\"showlegend\":false,\"textposition\":\"auto\",\"x\":[\"GAIA\",\"GSM8K\",\"HotpotQA\"],\"xaxis\":\"x2\",\"y\":[1.6666666666666667,2.473684210526316,1.6511627906976745],\"yaxis\":\"y2\",\"type\":\"bar\"},{\"alignmentgroup\":\"True\",\"hovertemplate\":\"Answer correctness=True\\u003cbr\\u003eAgent=Llama-2-70b-chat\\u003cbr\\u003eTask=%{x}\\u003cbr\\u003eMean Number of Steps=%{y}\\u003cextra\\u003e\\u003c\\u002fextra\\u003e\",\"legendgroup\":\"True\",\"marker\":{\"color\":\"#EF553B\",\"pattern\":{\"shape\":\"\"}},\"name\":\"True\",\"offsetgroup\":\"True\",\"orientation\":\"v\",\"showlegend\":false,\"textposition\":\"auto\",\"x\":[\"GSM8K\",\"HotpotQA\"],\"xaxis\":\"x3\",\"y\":[2.0,2.88],\"yaxis\":\"y3\",\"type\":\"bar\"},{\"alignmentgroup\":\"True\",\"hovertemplate\":\"Answer correctness=True\\u003cbr\\u003eAgent=Mixtral-8x7B-Instruct-v0.1\\u003cbr\\u003eTask=%{x}\\u003cbr\\u003eMean Number of Steps=%{y}\\u003cextra\\u003e\\u003c\\u002fextra\\u003e\",\"legendgroup\":\"True\",\"marker\":{\"color\":\"#EF553B\",\"pattern\":{\"shape\":\"\"}},\"name\":\"True\",\"offsetgroup\":\"True\",\"orientation\":\"v\",\"showlegend\":false,\"textposition\":\"auto\",\"x\":[\"GAIA\",\"GSM8K\",\"HotpotQA\"],\"xaxis\":\"x4\",\"y\":[3.0,3.0714285714285716,3.027027027027027],\"yaxis\":\"y4\",\"type\":\"bar\"},{\"alignmentgroup\":\"True\",\"hovertemplate\":\"Answer correctness=True\\u003cbr\\u003eAgent=OpenHermes-2.5-Mistral-7B\\u003cbr\\u003eTask=%{x}\\u003cbr\\u003eMean Number of Steps=%{y}\\u003cextra\\u003e\\u003c\\u002fextra\\u003e\",\"legendgroup\":\"True\",\"marker\":{\"color\":\"#EF553B\",\"pattern\":{\"shape\":\"\"}},\"name\":\"True\",\"offsetgroup\":\"True\",\"orientation\":\"v\",\"showlegend\":false,\"textposition\":\"auto\",\"x\":[\"GAIA\",\"GSM8K\",\"HotpotQA\"],\"xaxis\":\"x5\",\"y\":[3.5,3.5555555555555554,3.0303030303030303],\"yaxis\":\"y5\",\"type\":\"bar\"},{\"alignmentgroup\":\"True\",\"hovertemplate\":\"Answer correctness=True\\u003cbr\\u003eAgent=SOLAR-10.7B\\u003cbr\\u003eTask=%{x}\\u003cbr\\u003eMean Number of Steps=%{y}\\u003cextra\\u003e\\u003c\\u002fextra\\u003e\",\"legendgroup\":\"True\",\"marker\":{\"color\":\"#EF553B\",\"pattern\":{\"shape\":\"\"}},\"name\":\"True\",\"offsetgroup\":\"True\",\"orientation\":\"v\",\"showlegend\":false,\"textposition\":\"auto\",\"x\":[\"GAIA\",\"GSM8K\",\"HotpotQA\"],\"xaxis\":\"x6\",\"y\":[4.0,2.357142857142857,2.896551724137931],\"yaxis\":\"y6\",\"type\":\"bar\"},{\"alignmentgroup\":\"True\",\"hovertemplate\":\"Answer correctness=True\\u003cbr\\u003eAgent=Zephyr-7b-beta\\u003cbr\\u003eTask=%{x}\\u003cbr\\u003eMean Number of Steps=%{y}\\u003cextra\\u003e\\u003c\\u002fextra\\u003e\",\"legendgroup\":\"True\",\"marker\":{\"color\":\"#EF553B\",\"pattern\":{\"shape\":\"\"}},\"name\":\"True\",\"offsetgroup\":\"True\",\"orientation\":\"v\",\"showlegend\":false,\"textposition\":\"auto\",\"x\":[\"GSM8K\",\"HotpotQA\"],\"xaxis\":\"x7\",\"y\":[1.75,2.45],\"yaxis\":\"y7\",\"type\":\"bar\"}],                        {\"template\":{\"data\":{\"histogram2dcontour\":[{\"type\":\"histogram2dcontour\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"choropleth\":[{\"type\":\"choropleth\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}],\"histogram2d\":[{\"type\":\"histogram2d\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"heatmap\":[{\"type\":\"heatmap\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"heatmapgl\":[{\"type\":\"heatmapgl\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"contourcarpet\":[{\"type\":\"contourcarpet\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}],\"contour\":[{\"type\":\"contour\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"surface\":[{\"type\":\"surface\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"mesh3d\":[{\"type\":\"mesh3d\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}],\"scatter\":[{\"fillpattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2},\"type\":\"scatter\"}],\"parcoords\":[{\"type\":\"parcoords\",\"line\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scatterpolargl\":[{\"type\":\"scatterpolargl\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"bar\":[{\"error_x\":{\"color\":\"#2a3f5f\"},\"error_y\":{\"color\":\"#2a3f5f\"},\"marker\":{\"line\":{\"color\":\"#E5ECF6\",\"width\":0.5},\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"bar\"}],\"scattergeo\":[{\"type\":\"scattergeo\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scatterpolar\":[{\"type\":\"scatterpolar\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"histogram\":[{\"marker\":{\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"histogram\"}],\"scattergl\":[{\"type\":\"scattergl\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scatter3d\":[{\"type\":\"scatter3d\",\"line\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scattermapbox\":[{\"type\":\"scattermapbox\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scatterternary\":[{\"type\":\"scatterternary\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scattercarpet\":[{\"type\":\"scattercarpet\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"carpet\":[{\"aaxis\":{\"endlinecolor\":\"#2a3f5f\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"minorgridcolor\":\"white\",\"startlinecolor\":\"#2a3f5f\"},\"baxis\":{\"endlinecolor\":\"#2a3f5f\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"minorgridcolor\":\"white\",\"startlinecolor\":\"#2a3f5f\"},\"type\":\"carpet\"}],\"table\":[{\"cells\":{\"fill\":{\"color\":\"#EBF0F8\"},\"line\":{\"color\":\"white\"}},\"header\":{\"fill\":{\"color\":\"#C8D4E3\"},\"line\":{\"color\":\"white\"}},\"type\":\"table\"}],\"barpolar\":[{\"marker\":{\"line\":{\"color\":\"#E5ECF6\",\"width\":0.5},\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"barpolar\"}],\"pie\":[{\"automargin\":true,\"type\":\"pie\"}]},\"layout\":{\"autotypenumbers\":\"strict\",\"colorway\":[\"#636efa\",\"#EF553B\",\"#00cc96\",\"#ab63fa\",\"#FFA15A\",\"#19d3f3\",\"#FF6692\",\"#B6E880\",\"#FF97FF\",\"#FECB52\"],\"font\":{\"color\":\"#2a3f5f\"},\"hovermode\":\"closest\",\"hoverlabel\":{\"align\":\"left\"},\"paper_bgcolor\":\"white\",\"plot_bgcolor\":\"#E5ECF6\",\"polar\":{\"bgcolor\":\"#E5ECF6\",\"angularaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"},\"radialaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"}},\"ternary\":{\"bgcolor\":\"#E5ECF6\",\"aaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"},\"baxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"},\"caxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"}},\"coloraxis\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"colorscale\":{\"sequential\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"sequentialminus\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"diverging\":[[0,\"#8e0152\"],[0.1,\"#c51b7d\"],[0.2,\"#de77ae\"],[0.3,\"#f1b6da\"],[0.4,\"#fde0ef\"],[0.5,\"#f7f7f7\"],[0.6,\"#e6f5d0\"],[0.7,\"#b8e186\"],[0.8,\"#7fbc41\"],[0.9,\"#4d9221\"],[1,\"#276419\"]]},\"xaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\",\"title\":{\"standoff\":15},\"zerolinecolor\":\"white\",\"automargin\":true,\"zerolinewidth\":2},\"yaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\",\"title\":{\"standoff\":15},\"zerolinecolor\":\"white\",\"automargin\":true,\"zerolinewidth\":2},\"scene\":{\"xaxis\":{\"backgroundcolor\":\"#E5ECF6\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"showbackground\":true,\"ticks\":\"\",\"zerolinecolor\":\"white\",\"gridwidth\":2},\"yaxis\":{\"backgroundcolor\":\"#E5ECF6\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"showbackground\":true,\"ticks\":\"\",\"zerolinecolor\":\"white\",\"gridwidth\":2},\"zaxis\":{\"backgroundcolor\":\"#E5ECF6\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"showbackground\":true,\"ticks\":\"\",\"zerolinecolor\":\"white\",\"gridwidth\":2}},\"shapedefaults\":{\"line\":{\"color\":\"#2a3f5f\"}},\"annotationdefaults\":{\"arrowcolor\":\"#2a3f5f\",\"arrowhead\":0,\"arrowwidth\":1},\"geo\":{\"bgcolor\":\"white\",\"landcolor\":\"#E5ECF6\",\"subunitcolor\":\"white\",\"showland\":true,\"showlakes\":true,\"lakecolor\":\"white\"},\"title\":{\"x\":0.05},\"mapbox\":{\"style\":\"light\"}}},\"xaxis\":{\"anchor\":\"y\",\"domain\":[0.0,0.1257142857142857],\"title\":{\"text\":\"Task\"}},\"yaxis\":{\"anchor\":\"x\",\"domain\":[0.0,1.0],\"title\":{\"text\":\"Mean Number of Steps\"}},\"xaxis2\":{\"anchor\":\"y2\",\"domain\":[0.14571428571428569,0.27142857142857135],\"matches\":\"x\",\"title\":{\"text\":\"Task\"}},\"yaxis2\":{\"anchor\":\"x2\",\"domain\":[0.0,1.0],\"matches\":\"y\",\"showticklabels\":false},\"xaxis3\":{\"anchor\":\"y3\",\"domain\":[0.29142857142857137,0.41714285714285704],\"matches\":\"x\",\"title\":{\"text\":\"Task\"}},\"yaxis3\":{\"anchor\":\"x3\",\"domain\":[0.0,1.0],\"matches\":\"y\",\"showticklabels\":false},\"xaxis4\":{\"anchor\":\"y4\",\"domain\":[0.4371428571428571,0.5628571428571428],\"matches\":\"x\",\"title\":{\"text\":\"Task\"}},\"yaxis4\":{\"anchor\":\"x4\",\"domain\":[0.0,1.0],\"matches\":\"y\",\"showticklabels\":false},\"xaxis5\":{\"anchor\":\"y5\",\"domain\":[0.5828571428571427,0.7085714285714284],\"matches\":\"x\",\"title\":{\"text\":\"Task\"}},\"yaxis5\":{\"anchor\":\"x5\",\"domain\":[0.0,1.0],\"matches\":\"y\",\"showticklabels\":false},\"xaxis6\":{\"anchor\":\"y6\",\"domain\":[0.7285714285714284,0.8542857142857141],\"matches\":\"x\",\"title\":{\"text\":\"Task\"}},\"yaxis6\":{\"anchor\":\"x6\",\"domain\":[0.0,1.0],\"matches\":\"y\",\"showticklabels\":false},\"xaxis7\":{\"anchor\":\"y7\",\"domain\":[0.8742857142857141,0.9999999999999998],\"matches\":\"x\",\"title\":{\"text\":\"Task\"}},\"yaxis7\":{\"anchor\":\"x7\",\"domain\":[0.0,1.0],\"matches\":\"y\",\"showticklabels\":false},\"annotations\":[{\"font\":{},\"showarrow\":false,\"text\":\"Agent=GPT3.5\",\"x\":0.06285714285714285,\"xanchor\":\"center\",\"xref\":\"paper\",\"y\":1.0,\"yanchor\":\"bottom\",\"yref\":\"paper\"},{\"font\":{},\"showarrow\":false,\"text\":\"Agent=GPT4\",\"x\":0.20857142857142852,\"xanchor\":\"center\",\"xref\":\"paper\",\"y\":1.0,\"yanchor\":\"bottom\",\"yref\":\"paper\"},{\"font\":{},\"showarrow\":false,\"text\":\"Agent=Llama-2-70b-chat\",\"x\":0.3542857142857142,\"xanchor\":\"center\",\"xref\":\"paper\",\"y\":1.0,\"yanchor\":\"bottom\",\"yref\":\"paper\"},{\"font\":{},\"showarrow\":false,\"text\":\"Agent=Mixtral-8x7B-Instruct-v0.1\",\"x\":0.5,\"xanchor\":\"center\",\"xref\":\"paper\",\"y\":1.0,\"yanchor\":\"bottom\",\"yref\":\"paper\"},{\"font\":{},\"showarrow\":false,\"text\":\"Agent=OpenHermes-2.5-Mistral-7B\",\"x\":0.6457142857142856,\"xanchor\":\"center\",\"xref\":\"paper\",\"y\":1.0,\"yanchor\":\"bottom\",\"yref\":\"paper\"},{\"font\":{},\"showarrow\":false,\"text\":\"Agent=SOLAR-10.7B\",\"x\":0.7914285714285713,\"xanchor\":\"center\",\"xref\":\"paper\",\"y\":1.0,\"yanchor\":\"bottom\",\"yref\":\"paper\"},{\"font\":{},\"showarrow\":false,\"text\":\"Agent=Zephyr-7b-beta\",\"x\":0.9371428571428569,\"xanchor\":\"center\",\"xref\":\"paper\",\"y\":1.0,\"yanchor\":\"bottom\",\"yref\":\"paper\"}],\"legend\":{\"title\":{\"text\":\"Answer correctness\"},\"tracegroupgap\":0},\"title\":{\"text\":\"Mean Number of Steps to Solve a Task\"},\"barmode\":\"group\",\"width\":1500},                        {\"responsive\": true}                    ).then(function(){\n",
       "                            \n",
       "var gd = document.getElementById('e23c2b41-b14d-4353-be6c-e12f08e3a387');\n",
       "var x = new MutationObserver(function (mutations, observer) {{\n",
       "        var display = window.getComputedStyle(gd).display;\n",
       "        if (!display || display === 'none') {{\n",
       "            console.log([gd, 'removed!']);\n",
       "            Plotly.purge(gd);\n",
       "            observer.disconnect();\n",
       "        }}\n",
       "}});\n",
       "\n",
       "// Listen for the removal of the full notebook cells\n",
       "var notebookContainer = gd.closest('#notebook-container');\n",
       "if (notebookContainer) {{\n",
       "    x.observe(notebookContainer, {childList: true});\n",
       "}}\n",
       "\n",
       "// Listen for the clearing of the current output cell\n",
       "var outputEl = gd.closest('.output');\n",
       "if (outputEl) {{\n",
       "    x.observe(outputEl, {childList: true});\n",
       "}}\n",
       "\n",
       "                        })                };                });            </script>        </div>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "px.bar(\n",
    "    aggregated_resuts.reset_index(),\n",
    "    x=\"task\",\n",
    "    y=\"number_of_steps\",\n",
    "    facet_col=\"agent_name\",\n",
    "    color=\"correct_answer\",\n",
    "    title=\"Mean Number of Steps to Solve a Task\",\n",
    "    barmode=\"group\",\n",
    "    labels={\n",
    "        \"agent_name\": \"Agent\",\n",
    "        \"number_of_steps\": \"Mean Number of Steps\",\n",
    "        \"correct_answer\": \"Answer correctness\",\n",
    "        \"task\": \"Task\",\n",
    "    },\n",
    "    width=1500,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we can see that on average, __tasks that failed have a higher number of steps taken than successful tasks__: taking many steps seems to be an indication that the agent is trying wrong directions to solve the problem."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Bonus: LLM-as-a-judge - Prometheus-13B vs GPT4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "res = pd.read_json(\"output/GPT4.json\")\n",
    "res = res.loc[~res[\"eval_score_human\"].isnull()]\n",
    "res = res[\n",
    "    [\"eval_score_human\", \"eval_score_GPT4\", \"eval_score_Prometheus-13B-v1.0\"]\n",
    "].reset_index(drop=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.plotly.v1+json": {
       "config": {
        "plotlyServerURL": "https://plot.ly"
       },
       "data": [
        {
         "alignmentgroup": "True",
         "hovertemplate": "variable=eval_score_human<br>index=%{x}<br>value=%{y}<extra></extra>",
         "legendgroup": "eval_score_human",
         "marker": {
          "color": "#636efa",
          "pattern": {
           "shape": ""
          }
         },
         "name": "eval_score_human",
         "offsetgroup": "eval_score_human",
         "orientation": "v",
         "showlegend": true,
         "textposition": "auto",
         "type": "bar",
         "x": [
          0,
          1,
          2,
          3,
          4,
          5,
          6,
          7,
          8,
          9,
          10,
          11,
          12,
          13,
          14,
          15,
          16,
          17,
          18,
          19,
          20,
          21,
          22,
          23,
          24,
          25,
          26
         ],
         "xaxis": "x",
         "y": [
          5,
          5,
          5,
          5,
          5,
          5,
          5,
          5,
          1,
          5,
          5,
          5,
          5,
          5,
          5,
          5,
          3,
          3,
          1,
          5,
          1,
          1,
          5,
          5,
          5,
          5,
          5
         ],
         "yaxis": "y"
        },
        {
         "alignmentgroup": "True",
         "hovertemplate": "variable=eval_score_GPT4<br>index=%{x}<br>value=%{y}<extra></extra>",
         "legendgroup": "eval_score_GPT4",
         "marker": {
          "color": "#EF553B",
          "pattern": {
           "shape": ""
          }
         },
         "name": "eval_score_GPT4",
         "offsetgroup": "eval_score_GPT4",
         "orientation": "v",
         "showlegend": true,
         "textposition": "auto",
         "type": "bar",
         "x": [
          0,
          1,
          2,
          3,
          4,
          5,
          6,
          7,
          8,
          9,
          10,
          11,
          12,
          13,
          14,
          15,
          16,
          17,
          18,
          19,
          20,
          21,
          22,
          23,
          24,
          25,
          26
         ],
         "xaxis": "x",
         "y": [
          5,
          5,
          5,
          5,
          5,
          5,
          5,
          5,
          1,
          5,
          5,
          5,
          5,
          5,
          5,
          5,
          3,
          3,
          1,
          5,
          1,
          1,
          5,
          5,
          4,
          5,
          5
         ],
         "yaxis": "y"
        },
        {
         "alignmentgroup": "True",
         "hovertemplate": "variable=eval_score_Prometheus-13B-v1.0<br>index=%{x}<br>value=%{y}<extra></extra>",
         "legendgroup": "eval_score_Prometheus-13B-v1.0",
         "marker": {
          "color": "#00cc96",
          "pattern": {
           "shape": ""
          }
         },
         "name": "eval_score_Prometheus-13B-v1.0",
         "offsetgroup": "eval_score_Prometheus-13B-v1.0",
         "orientation": "v",
         "showlegend": true,
         "textposition": "auto",
         "type": "bar",
         "x": [
          0,
          1,
          2,
          3,
          4,
          5,
          6,
          7,
          8,
          9,
          10,
          11,
          12,
          13,
          14,
          15,
          16,
          17,
          18,
          19,
          20,
          21,
          22,
          23,
          24,
          25,
          26
         ],
         "xaxis": "x",
         "y": [
          4,
          4,
          5,
          5,
          4,
          1,
          5,
          1,
          1,
          4,
          1,
          1,
          5,
          3,
          1,
          5,
          5,
          1,
          1,
          1,
          2,
          1,
          1,
          5,
          4,
          1,
          4
         ],
         "yaxis": "y"
        }
       ],
       "layout": {
        "barmode": "group",
        "height": 300,
        "legend": {
         "title": {
          "text": "variable"
         },
         "tracegroupgap": 0
        },
        "margin": {
         "t": 60
        },
        "template": {
         "data": {
          "bar": [
           {
            "error_x": {
             "color": "#2a3f5f"
            },
            "error_y": {
             "color": "#2a3f5f"
            },
            "marker": {
             "line": {
              "color": "#E5ECF6",
              "width": 0.5
             },
             "pattern": {
              "fillmode": "overlay",
              "size": 10,
              "solidity": 0.2
             }
            },
            "type": "bar"
           }
          ],
          "barpolar": [
           {
            "marker": {
             "line": {
              "color": "#E5ECF6",
              "width": 0.5
             },
             "pattern": {
              "fillmode": "overlay",
              "size": 10,
              "solidity": 0.2
             }
            },
            "type": "barpolar"
           }
          ],
          "carpet": [
           {
            "aaxis": {
             "endlinecolor": "#2a3f5f",
             "gridcolor": "white",
             "linecolor": "white",
             "minorgridcolor": "white",
             "startlinecolor": "#2a3f5f"
            },
            "baxis": {
             "endlinecolor": "#2a3f5f",
             "gridcolor": "white",
             "linecolor": "white",
             "minorgridcolor": "white",
             "startlinecolor": "#2a3f5f"
            },
            "type": "carpet"
           }
          ],
          "choropleth": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "choropleth"
           }
          ],
          "contour": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "contour"
           }
          ],
          "contourcarpet": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "contourcarpet"
           }
          ],
          "heatmap": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "heatmap"
           }
          ],
          "heatmapgl": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "heatmapgl"
           }
          ],
          "histogram": [
           {
            "marker": {
             "pattern": {
              "fillmode": "overlay",
              "size": 10,
              "solidity": 0.2
             }
            },
            "type": "histogram"
           }
          ],
          "histogram2d": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "histogram2d"
           }
          ],
          "histogram2dcontour": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "histogram2dcontour"
           }
          ],
          "mesh3d": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "mesh3d"
           }
          ],
          "parcoords": [
           {
            "line": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "parcoords"
           }
          ],
          "pie": [
           {
            "automargin": true,
            "type": "pie"
           }
          ],
          "scatter": [
           {
            "fillpattern": {
             "fillmode": "overlay",
             "size": 10,
             "solidity": 0.2
            },
            "type": "scatter"
           }
          ],
          "scatter3d": [
           {
            "line": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatter3d"
           }
          ],
          "scattercarpet": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattercarpet"
           }
          ],
          "scattergeo": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattergeo"
           }
          ],
          "scattergl": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattergl"
           }
          ],
          "scattermapbox": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattermapbox"
           }
          ],
          "scatterpolar": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterpolar"
           }
          ],
          "scatterpolargl": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterpolargl"
           }
          ],
          "scatterternary": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterternary"
           }
          ],
          "surface": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "surface"
           }
          ],
          "table": [
           {
            "cells": {
             "fill": {
              "color": "#EBF0F8"
             },
             "line": {
              "color": "white"
             }
            },
            "header": {
             "fill": {
              "color": "#C8D4E3"
             },
             "line": {
              "color": "white"
             }
            },
            "type": "table"
           }
          ]
         },
         "layout": {
          "annotationdefaults": {
           "arrowcolor": "#2a3f5f",
           "arrowhead": 0,
           "arrowwidth": 1
          },
          "autotypenumbers": "strict",
          "coloraxis": {
           "colorbar": {
            "outlinewidth": 0,
            "ticks": ""
           }
          },
          "colorscale": {
           "diverging": [
            [
             0,
             "#8e0152"
            ],
            [
             0.1,
             "#c51b7d"
            ],
            [
             0.2,
             "#de77ae"
            ],
            [
             0.3,
             "#f1b6da"
            ],
            [
             0.4,
             "#fde0ef"
            ],
            [
             0.5,
             "#f7f7f7"
            ],
            [
             0.6,
             "#e6f5d0"
            ],
            [
             0.7,
             "#b8e186"
            ],
            [
             0.8,
             "#7fbc41"
            ],
            [
             0.9,
             "#4d9221"
            ],
            [
             1,
             "#276419"
            ]
           ],
           "sequential": [
            [
             0,
             "#0d0887"
            ],
            [
             0.1111111111111111,
             "#46039f"
            ],
            [
             0.2222222222222222,
             "#7201a8"
            ],
            [
             0.3333333333333333,
             "#9c179e"
            ],
            [
             0.4444444444444444,
             "#bd3786"
            ],
            [
             0.5555555555555556,
             "#d8576b"
            ],
            [
             0.6666666666666666,
             "#ed7953"
            ],
            [
             0.7777777777777778,
             "#fb9f3a"
            ],
            [
             0.8888888888888888,
             "#fdca26"
            ],
            [
             1,
             "#f0f921"
            ]
           ],
           "sequentialminus": [
            [
             0,
             "#0d0887"
            ],
            [
             0.1111111111111111,
             "#46039f"
            ],
            [
             0.2222222222222222,
             "#7201a8"
            ],
            [
             0.3333333333333333,
             "#9c179e"
            ],
            [
             0.4444444444444444,
             "#bd3786"
            ],
            [
             0.5555555555555556,
             "#d8576b"
            ],
            [
             0.6666666666666666,
             "#ed7953"
            ],
            [
             0.7777777777777778,
             "#fb9f3a"
            ],
            [
             0.8888888888888888,
             "#fdca26"
            ],
            [
             1,
             "#f0f921"
            ]
           ]
          },
          "colorway": [
           "#636efa",
           "#EF553B",
           "#00cc96",
           "#ab63fa",
           "#FFA15A",
           "#19d3f3",
           "#FF6692",
           "#B6E880",
           "#FF97FF",
           "#FECB52"
          ],
          "font": {
           "color": "#2a3f5f"
          },
          "geo": {
           "bgcolor": "white",
           "lakecolor": "white",
           "landcolor": "#E5ECF6",
           "showlakes": true,
           "showland": true,
           "subunitcolor": "white"
          },
          "hoverlabel": {
           "align": "left"
          },
          "hovermode": "closest",
          "mapbox": {
           "style": "light"
          },
          "paper_bgcolor": "white",
          "plot_bgcolor": "#E5ECF6",
          "polar": {
           "angularaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "bgcolor": "#E5ECF6",
           "radialaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           }
          },
          "scene": {
           "xaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           },
           "yaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           },
           "zaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           }
          },
          "shapedefaults": {
           "line": {
            "color": "#2a3f5f"
           }
          },
          "ternary": {
           "aaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "baxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "bgcolor": "#E5ECF6",
           "caxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           }
          },
          "title": {
           "x": 0.05
          },
          "xaxis": {
           "automargin": true,
           "gridcolor": "white",
           "linecolor": "white",
           "ticks": "",
           "title": {
            "standoff": 15
           },
           "zerolinecolor": "white",
           "zerolinewidth": 2
          },
          "yaxis": {
           "automargin": true,
           "gridcolor": "white",
           "linecolor": "white",
           "ticks": "",
           "title": {
            "standoff": 15
           },
           "zerolinecolor": "white",
           "zerolinewidth": 2
          }
         }
        },
        "width": 1100,
        "xaxis": {
         "anchor": "y",
         "domain": [
          0,
          1
         ],
         "title": {
          "text": "index"
         }
        },
        "yaxis": {
         "anchor": "x",
         "domain": [
          0,
          1
         ],
         "range": [
          0.5,
          5.5
         ],
         "tickmode": "array",
         "tickvals": [
          1,
          3,
          5
         ],
         "title": {
          "text": "Score"
         }
        }
       }
      },
      "text/html": [
       "<div>                            <div id=\"0ddf99d2-28ea-472c-af70-6549f14aa932\" class=\"plotly-graph-div\" style=\"height:300px; width:1100px;\"></div>            <script type=\"text/javascript\">                require([\"plotly\"], function(Plotly) {                    window.PLOTLYENV=window.PLOTLYENV || {};                                    if (document.getElementById(\"0ddf99d2-28ea-472c-af70-6549f14aa932\")) {                    Plotly.newPlot(                        \"0ddf99d2-28ea-472c-af70-6549f14aa932\",                        [{\"alignmentgroup\":\"True\",\"hovertemplate\":\"variable=eval_score_human\\u003cbr\\u003eindex=%{x}\\u003cbr\\u003evalue=%{y}\\u003cextra\\u003e\\u003c\\u002fextra\\u003e\",\"legendgroup\":\"eval_score_human\",\"marker\":{\"color\":\"#636efa\",\"pattern\":{\"shape\":\"\"}},\"name\":\"eval_score_human\",\"offsetgroup\":\"eval_score_human\",\"orientation\":\"v\",\"showlegend\":true,\"textposition\":\"auto\",\"x\":[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26],\"xaxis\":\"x\",\"y\":[5.0,5.0,5.0,5.0,5.0,5.0,5.0,5.0,1.0,5.0,5.0,5.0,5.0,5.0,5.0,5.0,3.0,3.0,1.0,5.0,1.0,1.0,5.0,5.0,5.0,5.0,5.0],\"yaxis\":\"y\",\"type\":\"bar\"},{\"alignmentgroup\":\"True\",\"hovertemplate\":\"variable=eval_score_GPT4\\u003cbr\\u003eindex=%{x}\\u003cbr\\u003evalue=%{y}\\u003cextra\\u003e\\u003c\\u002fextra\\u003e\",\"legendgroup\":\"eval_score_GPT4\",\"marker\":{\"color\":\"#EF553B\",\"pattern\":{\"shape\":\"\"}},\"name\":\"eval_score_GPT4\",\"offsetgroup\":\"eval_score_GPT4\",\"orientation\":\"v\",\"showlegend\":true,\"textposition\":\"auto\",\"x\":[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26],\"xaxis\":\"x\",\"y\":[5.0,5.0,5.0,5.0,5.0,5.0,5.0,5.0,1.0,5.0,5.0,5.0,5.0,5.0,5.0,5.0,3.0,3.0,1.0,5.0,1.0,1.0,5.0,5.0,4.0,5.0,5.0],\"yaxis\":\"y\",\"type\":\"bar\"},{\"alignmentgroup\":\"True\",\"hovertemplate\":\"variable=eval_score_Prometheus-13B-v1.0\\u003cbr\\u003eindex=%{x}\\u003cbr\\u003evalue=%{y}\\u003cextra\\u003e\\u003c\\u002fextra\\u003e\",\"legendgroup\":\"eval_score_Prometheus-13B-v1.0\",\"marker\":{\"color\":\"#00cc96\",\"pattern\":{\"shape\":\"\"}},\"name\":\"eval_score_Prometheus-13B-v1.0\",\"offsetgroup\":\"eval_score_Prometheus-13B-v1.0\",\"orientation\":\"v\",\"showlegend\":true,\"textposition\":\"auto\",\"x\":[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26],\"xaxis\":\"x\",\"y\":[4.0,4.0,5.0,5.0,4.0,1.0,5.0,1.0,1.0,4.0,1.0,1.0,5.0,3.0,1.0,5.0,5.0,1.0,1.0,1.0,2.0,1.0,1.0,5.0,4.0,1.0,4.0],\"yaxis\":\"y\",\"type\":\"bar\"}],                        {\"template\":{\"data\":{\"histogram2dcontour\":[{\"type\":\"histogram2dcontour\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"choropleth\":[{\"type\":\"choropleth\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}],\"histogram2d\":[{\"type\":\"histogram2d\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"heatmap\":[{\"type\":\"heatmap\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"heatmapgl\":[{\"type\":\"heatmapgl\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"contourcarpet\":[{\"type\":\"contourcarpet\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}],\"contour\":[{\"type\":\"contour\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"surface\":[{\"type\":\"surface\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"mesh3d\":[{\"type\":\"mesh3d\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}],\"scatter\":[{\"fillpattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2},\"type\":\"scatter\"}],\"parcoords\":[{\"type\":\"parcoords\",\"line\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scatterpolargl\":[{\"type\":\"scatterpolargl\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"bar\":[{\"error_x\":{\"color\":\"#2a3f5f\"},\"error_y\":{\"color\":\"#2a3f5f\"},\"marker\":{\"line\":{\"color\":\"#E5ECF6\",\"width\":0.5},\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"bar\"}],\"scattergeo\":[{\"type\":\"scattergeo\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scatterpolar\":[{\"type\":\"scatterpolar\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"histogram\":[{\"marker\":{\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"histogram\"}],\"scattergl\":[{\"type\":\"scattergl\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scatter3d\":[{\"type\":\"scatter3d\",\"line\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scattermapbox\":[{\"type\":\"scattermapbox\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scatterternary\":[{\"type\":\"scatterternary\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scattercarpet\":[{\"type\":\"scattercarpet\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"carpet\":[{\"aaxis\":{\"endlinecolor\":\"#2a3f5f\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"minorgridcolor\":\"white\",\"startlinecolor\":\"#2a3f5f\"},\"baxis\":{\"endlinecolor\":\"#2a3f5f\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"minorgridcolor\":\"white\",\"startlinecolor\":\"#2a3f5f\"},\"type\":\"carpet\"}],\"table\":[{\"cells\":{\"fill\":{\"color\":\"#EBF0F8\"},\"line\":{\"color\":\"white\"}},\"header\":{\"fill\":{\"color\":\"#C8D4E3\"},\"line\":{\"color\":\"white\"}},\"type\":\"table\"}],\"barpolar\":[{\"marker\":{\"line\":{\"color\":\"#E5ECF6\",\"width\":0.5},\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"barpolar\"}],\"pie\":[{\"automargin\":true,\"type\":\"pie\"}]},\"layout\":{\"autotypenumbers\":\"strict\",\"colorway\":[\"#636efa\",\"#EF553B\",\"#00cc96\",\"#ab63fa\",\"#FFA15A\",\"#19d3f3\",\"#FF6692\",\"#B6E880\",\"#FF97FF\",\"#FECB52\"],\"font\":{\"color\":\"#2a3f5f\"},\"hovermode\":\"closest\",\"hoverlabel\":{\"align\":\"left\"},\"paper_bgcolor\":\"white\",\"plot_bgcolor\":\"#E5ECF6\",\"polar\":{\"bgcolor\":\"#E5ECF6\",\"angularaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"},\"radialaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"}},\"ternary\":{\"bgcolor\":\"#E5ECF6\",\"aaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"},\"baxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"},\"caxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"}},\"coloraxis\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"colorscale\":{\"sequential\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"sequentialminus\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"diverging\":[[0,\"#8e0152\"],[0.1,\"#c51b7d\"],[0.2,\"#de77ae\"],[0.3,\"#f1b6da\"],[0.4,\"#fde0ef\"],[0.5,\"#f7f7f7\"],[0.6,\"#e6f5d0\"],[0.7,\"#b8e186\"],[0.8,\"#7fbc41\"],[0.9,\"#4d9221\"],[1,\"#276419\"]]},\"xaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\",\"title\":{\"standoff\":15},\"zerolinecolor\":\"white\",\"automargin\":true,\"zerolinewidth\":2},\"yaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\",\"title\":{\"standoff\":15},\"zerolinecolor\":\"white\",\"automargin\":true,\"zerolinewidth\":2},\"scene\":{\"xaxis\":{\"backgroundcolor\":\"#E5ECF6\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"showbackground\":true,\"ticks\":\"\",\"zerolinecolor\":\"white\",\"gridwidth\":2},\"yaxis\":{\"backgroundcolor\":\"#E5ECF6\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"showbackground\":true,\"ticks\":\"\",\"zerolinecolor\":\"white\",\"gridwidth\":2},\"zaxis\":{\"backgroundcolor\":\"#E5ECF6\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"showbackground\":true,\"ticks\":\"\",\"zerolinecolor\":\"white\",\"gridwidth\":2}},\"shapedefaults\":{\"line\":{\"color\":\"#2a3f5f\"}},\"annotationdefaults\":{\"arrowcolor\":\"#2a3f5f\",\"arrowhead\":0,\"arrowwidth\":1},\"geo\":{\"bgcolor\":\"white\",\"landcolor\":\"#E5ECF6\",\"subunitcolor\":\"white\",\"showland\":true,\"showlakes\":true,\"lakecolor\":\"white\"},\"title\":{\"x\":0.05},\"mapbox\":{\"style\":\"light\"}}},\"xaxis\":{\"anchor\":\"y\",\"domain\":[0.0,1.0],\"title\":{\"text\":\"index\"}},\"yaxis\":{\"anchor\":\"x\",\"domain\":[0.0,1.0],\"title\":{\"text\":\"Score\"},\"range\":[0.5,5.5],\"tickmode\":\"array\",\"tickvals\":[1,3,5]},\"legend\":{\"title\":{\"text\":\"variable\"},\"tracegroupgap\":0},\"margin\":{\"t\":60},\"barmode\":\"group\",\"width\":1100,\"height\":300},                        {\"responsive\": true}                    ).then(function(){\n",
       "                            \n",
       "var gd = document.getElementById('0ddf99d2-28ea-472c-af70-6549f14aa932');\n",
       "var x = new MutationObserver(function (mutations, observer) {{\n",
       "        var display = window.getComputedStyle(gd).display;\n",
       "        if (!display || display === 'none') {{\n",
       "            console.log([gd, 'removed!']);\n",
       "            Plotly.purge(gd);\n",
       "            observer.disconnect();\n",
       "        }}\n",
       "}});\n",
       "\n",
       "// Listen for the removal of the full notebook cells\n",
       "var notebookContainer = gd.closest('#notebook-container');\n",
       "if (notebookContainer) {{\n",
       "    x.observe(notebookContainer, {childList: true});\n",
       "}}\n",
       "\n",
       "// Listen for the clearing of the current output cell\n",
       "var outputEl = gd.closest('.output');\n",
       "if (outputEl) {{\n",
       "    x.observe(outputEl, {childList: true});\n",
       "}}\n",
       "\n",
       "                        })                };                });            </script>        </div>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig = px.bar(\n",
    "    res,\n",
    ")\n",
    "fig.update_layout(\n",
    "    width=1100,\n",
    "    height=300,\n",
    "    barmode=\"group\",\n",
    "    yaxis_range=[0.5, 5.5],\n",
    "    yaxis_title=\"Score\",\n",
    "    yaxis=dict(\n",
    "        tickmode=\"array\",\n",
    "        tickvals=[1, 3, 5],\n",
    "    ),\n",
    ")\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Proportion of cases matching human eval:\n",
      "0.333 for Prometheus\n",
      "0.963 for GPT4\n"
     ]
    }
   ],
   "source": [
    "print(\"Proportion of cases matching human eval:\")\n",
    "print(\n",
    "    (res[\"eval_score_human\"] == res[\"eval_score_Prometheus-13B-v1.0\"]).mean().round(3),\n",
    "    \"for Prometheus\",\n",
    ")\n",
    "print((res[\"eval_score_human\"] == res[\"eval_score_GPT4\"]).mean().round(3), \"for GPT4\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Given the high rate of error of Prometheus-13B, we could not use it for this evaluation."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "pytorch",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.1.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}