langchain-intro.ipynb

{"cells": [{"metadata": {}, "id": "032112c4", "cell_type": "markdown", "source": "## 1. Programmatically using WatsonX.ai models"}, {"metadata": {}, "id": "7199da06", "cell_type": "code", "source": "!pip install langchain==0.0.312\n!pip install python-dotenv==1.0.0\n!pip install ibm-watson==7.0.1\n!pip install ibm-watson-machine-learning==1.0.327\n!pip install chromadb==0.4.2\n!pip install pypdf==3.12.2\n!pip install sentence-transformers==2.2.2\n!pip install transformers==4.34.0\n!pip install flask-sqlalchemy\n!pip install wget", "execution_count": 1, "outputs": [{"output_type": "stream", "text": "Collecting langchain==0.0.312\n  Downloading langchain-0.0.312-py3-none-any.whl.metadata (15 kB)\nRequirement already satisfied: PyYAML>=5.3 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from langchain==0.0.312) (6.0)\nRequirement already satisfied: SQLAlchemy<3,>=1.4 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from langchain==0.0.312) (1.4.39)\nRequirement already satisfied: aiohttp<4.0.0,>=3.8.3 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from langchain==0.0.312) (3.8.5)\nRequirement already satisfied: anyio<4.0 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from langchain==0.0.312) (3.7.1)\nRequirement already satisfied: async-timeout<5.0.0,>=4.0.0 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from langchain==0.0.312) (4.0.2)\nRequirement already satisfied: dataclasses-json<0.7,>=0.5.7 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from langchain==0.0.312) (0.6.2)\nRequirement already satisfied: jsonpatch<2.0,>=1.33 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from langchain==0.0.312) (1.33)\nRequirement already satisfied: langsmith<0.1.0,>=0.0.43 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from langchain==0.0.312) (0.0.66)\nRequirement already satisfied: numpy<2,>=1 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from langchain==0.0.312) (1.23.5)\nRequirement already satisfied: pydantic<3,>=1 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from langchain==0.0.312) (2.5.2)\nRequirement already satisfied: requests<3,>=2 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from langchain==0.0.312) (2.31.0)\nRequirement already satisfied: tenacity<9.0.0,>=8.1.0 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from langchain==0.0.312) (8.2.3)\nRequirement already satisfied: attrs>=17.3.0 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from aiohttp<4.0.0,>=3.8.3->langchain==0.0.312) (22.1.0)\nRequirement already satisfied: charset-normalizer<4.0,>=2.0 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from aiohttp<4.0.0,>=3.8.3->langchain==0.0.312) (2.0.4)\nRequirement already satisfied: multidict<7.0,>=4.5 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from aiohttp<4.0.0,>=3.8.3->langchain==0.0.312) (6.0.2)\nRequirement already satisfied: yarl<2.0,>=1.0 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from aiohttp<4.0.0,>=3.8.3->langchain==0.0.312) (1.8.1)\nRequirement already satisfied: frozenlist>=1.1.1 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from aiohttp<4.0.0,>=3.8.3->langchain==0.0.312) (1.3.3)\nRequirement already satisfied: aiosignal>=1.1.2 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from aiohttp<4.0.0,>=3.8.3->langchain==0.0.312) (1.2.0)\nRequirement already satisfied: idna>=2.8 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from anyio<4.0->langchain==0.0.312) (3.4)\nRequirement already satisfied: sniffio>=1.1 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from anyio<4.0->langchain==0.0.312) (1.3.0)\nRequirement already satisfied: exceptiongroup in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from anyio<4.0->langchain==0.0.312) (1.2.0)\nRequirement already satisfied: marshmallow<4.0.0,>=3.18.0 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from dataclasses-json<0.7,>=0.5.7->langchain==0.0.312) (3.20.1)\nRequirement already satisfied: typing-inspect<1,>=0.4.0 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from dataclasses-json<0.7,>=0.5.7->langchain==0.0.312) (0.9.0)\nRequirement already satisfied: jsonpointer>=1.9 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from jsonpatch<2.0,>=1.33->langchain==0.0.312) (2.4)\nRequirement already satisfied: annotated-types>=0.4.0 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from pydantic<3,>=1->langchain==0.0.312) (0.6.0)\nRequirement already satisfied: pydantic-core==2.14.5 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from pydantic<3,>=1->langchain==0.0.312) (2.14.5)\nRequirement already satisfied: typing-extensions>=4.6.1 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from pydantic<3,>=1->langchain==0.0.312) (4.8.0)\nRequirement already satisfied: urllib3<3,>=1.21.1 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from requests<3,>=2->langchain==0.0.312) (1.26.18)\nRequirement already satisfied: certifi>=2017.4.17 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from requests<3,>=2->langchain==0.0.312) (2023.7.22)\nRequirement already satisfied: greenlet!=0.4.17 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from SQLAlchemy<3,>=1.4->langchain==0.0.312) (2.0.1)\nRequirement already satisfied: packaging>=17.0 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from marshmallow<4.0.0,>=3.18.0->dataclasses-json<0.7,>=0.5.7->langchain==0.0.312) (23.0)\nRequirement already satisfied: mypy-extensions>=0.3.0 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from typing-inspect<1,>=0.4.0->dataclasses-json<0.7,>=0.5.7->langchain==0.0.312) (0.4.3)\nDownloading langchain-0.0.312-py3-none-any.whl (1.8 MB)\n\u001b[2K   \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m1.8/1.8 MB\u001b[0m \u001b[31m81.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n\u001b[?25hInstalling collected packages: langchain\n  Attempting uninstall: langchain\n    Found existing installation: langchain 0.0.340\n    Uninstalling langchain-0.0.340:\n      Successfully uninstalled langchain-0.0.340\nSuccessfully installed langchain-0.0.312\nRequirement already satisfied: python-dotenv==1.0.0 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (1.0.0)\nCollecting ibm-watson==7.0.1\n  Downloading ibm-watson-7.0.1.tar.gz (389 kB)\n\u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m389.3/389.3 kB\u001b[0m \u001b[31m37.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n\u001b[?25h  Installing build dependencies ... \u001b[?25ldone\n\u001b[?25h  Getting requirements to build wheel ... \u001b[?25ldone\n\u001b[?25h  Preparing metadata (pyproject.toml) ... \u001b[?25ldone\n\u001b[?25hRequirement already satisfied: requests<3.0,>=2.0 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from ibm-watson==7.0.1) (2.31.0)\nRequirement already satisfied: python-dateutil>=2.5.3 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from ibm-watson==7.0.1) (2.8.2)\nRequirement already satisfied: websocket-client>=1.1.0 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from ibm-watson==7.0.1) (1.6.4)\nRequirement already satisfied: ibm-cloud-sdk-core==3.*,>=3.3.6 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from ibm-watson==7.0.1) (3.16.5)\nRequirement already satisfied: urllib3<2.0.0,>=1.26.0 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from ibm-cloud-sdk-core==3.*,>=3.3.6->ibm-watson==7.0.1) (1.26.18)\nRequirement already satisfied: PyJWT<3.0.0,>=2.4.0 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from ibm-cloud-sdk-core==3.*,>=3.3.6->ibm-watson==7.0.1) (2.4.0)\nRequirement already satisfied: six>=1.5 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from python-dateutil>=2.5.3->ibm-watson==7.0.1) (1.16.0)\nRequirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from requests<3.0,>=2.0->ibm-watson==7.0.1) (2.0.4)\nRequirement already satisfied: idna<4,>=2.5 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from requests<3.0,>=2.0->ibm-watson==7.0.1) (3.4)\nRequirement already satisfied: certifi>=2017.4.17 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from requests<3.0,>=2.0->ibm-watson==7.0.1) (2023.7.22)\nBuilding wheels for collected packages: ibm-watson\n  Building wheel for ibm-watson (pyproject.toml) ... \u001b[?25ldone\n\u001b[?25h  Created wheel for ibm-watson: filename=ibm_watson-7.0.1-py3-none-any.whl size=389784 sha256=63c584eb92b96750f9e2299431d165befc8a26609cf14b9f0e81618c88654fab\n  Stored in directory: /tmp/wsuser/.cache/pip/wheels/34/df/f4/f8edc5ba0637dd4bfb2029741ae20402976a49d1b6bc113553\nSuccessfully built ibm-watson\n", "name": "stdout"}, {"output_type": "stream", "text": "Installing collected packages: ibm-watson\nSuccessfully installed ibm-watson-7.0.1\nCollecting ibm-watson-machine-learning==1.0.327\n  Downloading ibm_watson_machine_learning-1.0.327-py3-none-any.whl.metadata (8.6 kB)\nRequirement already satisfied: requests in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from ibm-watson-machine-learning==1.0.327) (2.31.0)\nRequirement already satisfied: urllib3 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from ibm-watson-machine-learning==1.0.327) (1.26.18)\nRequirement already satisfied: pandas<1.6.0,>=0.24.2 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from ibm-watson-machine-learning==1.0.327) (1.5.3)\nRequirement already satisfied: certifi in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from ibm-watson-machine-learning==1.0.327) (2023.7.22)\nRequirement already satisfied: lomond in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from ibm-watson-machine-learning==1.0.327) (0.3.3)\nRequirement already satisfied: tabulate in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from ibm-watson-machine-learning==1.0.327) (0.8.10)\nRequirement already satisfied: packaging in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from ibm-watson-machine-learning==1.0.327) (23.0)\nRequirement already satisfied: importlib-metadata in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from ibm-watson-machine-learning==1.0.327) (6.0.0)\nRequirement already satisfied: ibm-cos-sdk<2.14.0,>=2.12.0 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from ibm-watson-machine-learning==1.0.327) (2.12.0)\nRequirement already satisfied: ibm-cos-sdk-core==2.12.0 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from ibm-cos-sdk<2.14.0,>=2.12.0->ibm-watson-machine-learning==1.0.327) (2.12.0)\nRequirement already satisfied: ibm-cos-sdk-s3transfer==2.12.0 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from ibm-cos-sdk<2.14.0,>=2.12.0->ibm-watson-machine-learning==1.0.327) (2.12.0)\nRequirement already satisfied: jmespath<1.0.0,>=0.10.0 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from ibm-cos-sdk<2.14.0,>=2.12.0->ibm-watson-machine-learning==1.0.327) (0.10.0)\nRequirement already satisfied: python-dateutil<3.0.0,>=2.8.2 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from ibm-cos-sdk-core==2.12.0->ibm-cos-sdk<2.14.0,>=2.12.0->ibm-watson-machine-learning==1.0.327) (2.8.2)\nRequirement already satisfied: pytz>=2020.1 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from pandas<1.6.0,>=0.24.2->ibm-watson-machine-learning==1.0.327) (2022.7)\nRequirement already satisfied: numpy>=1.21.0 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from pandas<1.6.0,>=0.24.2->ibm-watson-machine-learning==1.0.327) (1.23.5)\nRequirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from requests->ibm-watson-machine-learning==1.0.327) (2.0.4)\nRequirement already satisfied: idna<4,>=2.5 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from requests->ibm-watson-machine-learning==1.0.327) (3.4)\nRequirement already satisfied: zipp>=0.5 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from importlib-metadata->ibm-watson-machine-learning==1.0.327) (3.11.0)\nRequirement already satisfied: six>=1.10.0 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from lomond->ibm-watson-machine-learning==1.0.327) (1.16.0)\nDownloading ibm_watson_machine_learning-1.0.327-py3-none-any.whl (1.7 MB)\n\u001b[2K   \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m1.7/1.7 MB\u001b[0m \u001b[31m36.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m:00:01\u001b[0m\n\u001b[?25hInstalling collected packages: ibm-watson-machine-learning\n  Attempting uninstall: ibm-watson-machine-learning\n    Found existing installation: ibm-watson-machine-learning 1.0.330\n    Uninstalling ibm-watson-machine-learning-1.0.330:\n      Successfully uninstalled ibm-watson-machine-learning-1.0.330\nSuccessfully installed ibm-watson-machine-learning-1.0.327\nCollecting chromadb==0.4.2\n  Downloading chromadb-0.4.2-py3-none-any.whl.metadata (6.9 kB)\nRequirement already satisfied: pandas>=1.3 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from chromadb==0.4.2) (1.5.3)\nRequirement already satisfied: requests>=2.28 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from chromadb==0.4.2) (2.31.0)\nCollecting pydantic<2.0,>=1.9 (from chromadb==0.4.2)\n  Downloading pydantic-1.10.13-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (149 kB)\n\u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m149.6/149.6 kB\u001b[0m \u001b[31m22.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n\u001b[?25hCollecting chroma-hnswlib==0.7.1 (from chromadb==0.4.2)\n  Downloading chroma-hnswlib-0.7.1.tar.gz (30 kB)\n  Installing build dependencies ... \u001b[?25ldone\n\u001b[?25h  Getting requirements to build wheel ... \u001b[?25ldone\n\u001b[?25h  Preparing metadata (pyproject.toml) ... \u001b[?25ldone\n\u001b[?25hCollecting fastapi<0.100.0,>=0.95.2 (from chromadb==0.4.2)\n  Downloading fastapi-0.99.1-py3-none-any.whl.metadata (23 kB)\nRequirement already satisfied: uvicorn>=0.18.3 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from uvicorn[standard]>=0.18.3->chromadb==0.4.2) (0.24.0.post1)\nRequirement already satisfied: numpy>=1.21.6 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from chromadb==0.4.2) (1.23.5)\nRequirement already satisfied: posthog>=2.4.0 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from chromadb==0.4.2) (3.0.2)\nRequirement already satisfied: typing-extensions>=4.5.0 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from chromadb==0.4.2) (4.8.0)\nRequirement already satisfied: pulsar-client>=3.1.0 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from chromadb==0.4.2) (3.3.0)\nRequirement already satisfied: onnxruntime>=1.14.1 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from chromadb==0.4.2) (1.16.3)\nRequirement already satisfied: tokenizers>=0.13.2 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from chromadb==0.4.2) (0.15.0)\nRequirement already satisfied: pypika>=0.48.9 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from chromadb==0.4.2) (0.48.9)\nRequirement already satisfied: tqdm>=4.65.0 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from chromadb==0.4.2) (4.65.0)\nRequirement already satisfied: overrides>=7.3.1 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from chromadb==0.4.2) (7.4.0)\nRequirement already satisfied: importlib-resources in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from chromadb==0.4.2) (6.1.1)\nRequirement already satisfied: starlette<0.28.0,>=0.27.0 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from fastapi<0.100.0,>=0.95.2->chromadb==0.4.2) (0.27.0)\nRequirement already satisfied: coloredlogs in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from onnxruntime>=1.14.1->chromadb==0.4.2) (15.0.1)\nRequirement already satisfied: flatbuffers in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from onnxruntime>=1.14.1->chromadb==0.4.2) (2.0)\nRequirement already satisfied: packaging in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from onnxruntime>=1.14.1->chromadb==0.4.2) (23.0)\nRequirement already satisfied: protobuf in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from onnxruntime>=1.14.1->chromadb==0.4.2) (4.21.12)\nRequirement already satisfied: sympy in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from onnxruntime>=1.14.1->chromadb==0.4.2) (1.11.1)\nRequirement already satisfied: python-dateutil>=2.8.1 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from pandas>=1.3->chromadb==0.4.2) (2.8.2)\nRequirement already satisfied: pytz>=2020.1 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from pandas>=1.3->chromadb==0.4.2) (2022.7)\nRequirement already satisfied: six>=1.5 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from posthog>=2.4.0->chromadb==0.4.2) (1.16.0)\nRequirement already satisfied: monotonic>=1.5 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from posthog>=2.4.0->chromadb==0.4.2) (1.6)\nRequirement already satisfied: backoff>=1.10.0 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from posthog>=2.4.0->chromadb==0.4.2) (2.2.1)\nRequirement already satisfied: certifi in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from pulsar-client>=3.1.0->chromadb==0.4.2) (2023.7.22)\n", "name": "stdout"}, {"output_type": "stream", "text": "Requirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from requests>=2.28->chromadb==0.4.2) (2.0.4)\nRequirement already satisfied: idna<4,>=2.5 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from requests>=2.28->chromadb==0.4.2) (3.4)\nRequirement already satisfied: urllib3<3,>=1.21.1 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from requests>=2.28->chromadb==0.4.2) (1.26.18)\nRequirement already satisfied: huggingface_hub<1.0,>=0.16.4 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from tokenizers>=0.13.2->chromadb==0.4.2) (0.19.4)\nRequirement already satisfied: click>=7.0 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from uvicorn>=0.18.3->uvicorn[standard]>=0.18.3->chromadb==0.4.2) (8.0.4)\nRequirement already satisfied: h11>=0.8 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from uvicorn>=0.18.3->uvicorn[standard]>=0.18.3->chromadb==0.4.2) (0.14.0)\nRequirement already satisfied: httptools>=0.5.0 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from uvicorn[standard]>=0.18.3->chromadb==0.4.2) (0.6.1)\nRequirement already satisfied: python-dotenv>=0.13 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from uvicorn[standard]>=0.18.3->chromadb==0.4.2) (1.0.0)\nRequirement already satisfied: pyyaml>=5.1 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from uvicorn[standard]>=0.18.3->chromadb==0.4.2) (6.0)\nRequirement already satisfied: uvloop!=0.15.0,!=0.15.1,>=0.14.0 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from uvicorn[standard]>=0.18.3->chromadb==0.4.2) (0.19.0)\nRequirement already satisfied: watchfiles>=0.13 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from uvicorn[standard]>=0.18.3->chromadb==0.4.2) (0.21.0)\nRequirement already satisfied: websockets>=10.4 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from uvicorn[standard]>=0.18.3->chromadb==0.4.2) (12.0)\nRequirement already satisfied: filelock in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from huggingface_hub<1.0,>=0.16.4->tokenizers>=0.13.2->chromadb==0.4.2) (3.9.0)\nRequirement already satisfied: fsspec>=2023.5.0 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from huggingface_hub<1.0,>=0.16.4->tokenizers>=0.13.2->chromadb==0.4.2) (2023.10.0)\nRequirement already satisfied: anyio<5,>=3.4.0 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from starlette<0.28.0,>=0.27.0->fastapi<0.100.0,>=0.95.2->chromadb==0.4.2) (3.7.1)\nRequirement already satisfied: humanfriendly>=9.1 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from coloredlogs->onnxruntime>=1.14.1->chromadb==0.4.2) (10.0)\nRequirement already satisfied: mpmath>=0.19 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from sympy->onnxruntime>=1.14.1->chromadb==0.4.2) (1.3.0)\nRequirement already satisfied: sniffio>=1.1 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from anyio<5,>=3.4.0->starlette<0.28.0,>=0.27.0->fastapi<0.100.0,>=0.95.2->chromadb==0.4.2) (1.3.0)\nRequirement already satisfied: exceptiongroup in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from anyio<5,>=3.4.0->starlette<0.28.0,>=0.27.0->fastapi<0.100.0,>=0.95.2->chromadb==0.4.2) (1.2.0)\nDownloading chromadb-0.4.2-py3-none-any.whl (399 kB)\n\u001b[2K   \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m399.3/399.3 kB\u001b[0m \u001b[31m62.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n\u001b[?25hDownloading fastapi-0.99.1-py3-none-any.whl (58 kB)\n\u001b[2K   \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m58.4/58.4 kB\u001b[0m \u001b[31m17.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n\u001b[?25hDownloading pydantic-1.10.13-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.1 MB)\n\u001b[2K   \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m3.1/3.1 MB\u001b[0m \u001b[31m67.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m:00:01\u001b[0m\n\u001b[?25hBuilding wheels for collected packages: chroma-hnswlib\n  Building wheel for chroma-hnswlib (pyproject.toml) ... \u001b[?25ldone\n\u001b[?25h  Created wheel for chroma-hnswlib: filename=chroma_hnswlib-0.7.1-cp310-cp310-linux_x86_64.whl size=181706 sha256=874e6b076e67006cdfdc2fd3c6aff452f265790d736eae48d637416ea32c4c74\n  Stored in directory: /tmp/wsuser/.cache/pip/wheels/ad/f2/d2/3f32228e9f4713a9f32a468de8bbc3c642f7805ebef888418b\nSuccessfully built chroma-hnswlib\nInstalling collected packages: pydantic, chroma-hnswlib, fastapi, chromadb\n  Attempting uninstall: pydantic\n    Found existing installation: pydantic 2.5.2\n    Uninstalling pydantic-2.5.2:\n      Successfully uninstalled pydantic-2.5.2\n  Attempting uninstall: chroma-hnswlib\n    Found existing installation: chroma-hnswlib 0.7.3\n    Uninstalling chroma-hnswlib-0.7.3:\n      Successfully uninstalled chroma-hnswlib-0.7.3\n  Attempting uninstall: fastapi\n    Found existing installation: fastapi 0.104.1\n    Uninstalling fastapi-0.104.1:\n      Successfully uninstalled fastapi-0.104.1\n  Attempting uninstall: chromadb\n    Found existing installation: chromadb 0.4.18\n    Uninstalling chromadb-0.4.18:\n      Successfully uninstalled chromadb-0.4.18\nSuccessfully installed chroma-hnswlib-0.7.1 chromadb-0.4.2 fastapi-0.99.1 pydantic-1.10.13\nCollecting pypdf==3.12.2\n  Downloading pypdf-3.12.2-py3-none-any.whl.metadata (6.8 kB)\nDownloading pypdf-3.12.2-py3-none-any.whl (254 kB)\n\u001b[2K   \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m255.0/255.0 kB\u001b[0m \u001b[31m28.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n\u001b[?25hInstalling collected packages: pypdf\nSuccessfully installed pypdf-3.12.2\nCollecting sentence-transformers==2.2.2\n  Downloading sentence-transformers-2.2.2.tar.gz (85 kB)\n\u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m86.0/86.0 kB\u001b[0m \u001b[31m16.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25ldone\n\u001b[?25hRequirement already satisfied: transformers<5.0.0,>=4.6.0 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from sentence-transformers==2.2.2) (4.35.2)\nRequirement already satisfied: tqdm in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from sentence-transformers==2.2.2) (4.65.0)\nRequirement already satisfied: torch>=1.6.0 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from sentence-transformers==2.2.2) (2.0.1)\nRequirement already satisfied: torchvision in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from sentence-transformers==2.2.2) (0.15.2)\nRequirement already satisfied: numpy in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from sentence-transformers==2.2.2) (1.23.5)\nRequirement already satisfied: scikit-learn in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from sentence-transformers==2.2.2) (1.1.1)\nRequirement already satisfied: scipy in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from sentence-transformers==2.2.2) (1.10.1)\nCollecting nltk (from sentence-transformers==2.2.2)\n  Downloading nltk-3.8.1-py3-none-any.whl (1.5 MB)\n\u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m1.5/1.5 MB\u001b[0m \u001b[31m96.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n\u001b[?25hRequirement already satisfied: sentencepiece in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from sentence-transformers==2.2.2) (0.1.97)\nRequirement already satisfied: huggingface-hub>=0.4.0 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from sentence-transformers==2.2.2) (0.19.4)\nRequirement already satisfied: filelock in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from huggingface-hub>=0.4.0->sentence-transformers==2.2.2) (3.9.0)\nRequirement already satisfied: fsspec>=2023.5.0 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from huggingface-hub>=0.4.0->sentence-transformers==2.2.2) (2023.10.0)\nRequirement already satisfied: requests in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from huggingface-hub>=0.4.0->sentence-transformers==2.2.2) (2.31.0)\nRequirement already satisfied: pyyaml>=5.1 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from huggingface-hub>=0.4.0->sentence-transformers==2.2.2) (6.0)\nRequirement already satisfied: typing-extensions>=3.7.4.3 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from huggingface-hub>=0.4.0->sentence-transformers==2.2.2) (4.8.0)\nRequirement already satisfied: packaging>=20.9 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from huggingface-hub>=0.4.0->sentence-transformers==2.2.2) (23.0)\nRequirement already satisfied: sympy in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from torch>=1.6.0->sentence-transformers==2.2.2) (1.11.1)\nRequirement already satisfied: networkx in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from torch>=1.6.0->sentence-transformers==2.2.2) (2.8.4)\nRequirement already satisfied: jinja2 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from torch>=1.6.0->sentence-transformers==2.2.2) (3.1.2)\n", "name": "stdout"}, {"output_type": "stream", "text": "Requirement already satisfied: regex!=2019.12.17 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from transformers<5.0.0,>=4.6.0->sentence-transformers==2.2.2) (2023.10.3)\nRequirement already satisfied: tokenizers<0.19,>=0.14 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from transformers<5.0.0,>=4.6.0->sentence-transformers==2.2.2) (0.15.0)\nRequirement already satisfied: safetensors>=0.3.1 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from transformers<5.0.0,>=4.6.0->sentence-transformers==2.2.2) (0.4.1)\nRequirement already satisfied: click in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from nltk->sentence-transformers==2.2.2) (8.0.4)\nRequirement already satisfied: joblib in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from nltk->sentence-transformers==2.2.2) (1.1.1)\nRequirement already satisfied: threadpoolctl>=2.0.0 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from scikit-learn->sentence-transformers==2.2.2) (2.2.0)\nRequirement already satisfied: pillow!=8.3.*,>=5.3.0 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from torchvision->sentence-transformers==2.2.2) (10.0.1)\nRequirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from jinja2->torch>=1.6.0->sentence-transformers==2.2.2) (2.1.1)\nRequirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from requests->huggingface-hub>=0.4.0->sentence-transformers==2.2.2) (2.0.4)\nRequirement already satisfied: idna<4,>=2.5 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from requests->huggingface-hub>=0.4.0->sentence-transformers==2.2.2) (3.4)\nRequirement already satisfied: urllib3<3,>=1.21.1 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from requests->huggingface-hub>=0.4.0->sentence-transformers==2.2.2) (1.26.18)\nRequirement already satisfied: certifi>=2017.4.17 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from requests->huggingface-hub>=0.4.0->sentence-transformers==2.2.2) (2023.7.22)\nRequirement already satisfied: mpmath>=0.19 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from sympy->torch>=1.6.0->sentence-transformers==2.2.2) (1.3.0)\nBuilding wheels for collected packages: sentence-transformers\n  Building wheel for sentence-transformers (setup.py) ... \u001b[?25ldone\n\u001b[?25h  Created wheel for sentence-transformers: filename=sentence_transformers-2.2.2-py3-none-any.whl size=125925 sha256=862e2811dd39a4a0d51399cb9f86232624b732b3399ab8673478f0e7f4d2db5d\n  Stored in directory: /tmp/wsuser/.cache/pip/wheels/62/f2/10/1e606fd5f02395388f74e7462910fe851042f97238cbbd902f\nSuccessfully built sentence-transformers\nInstalling collected packages: nltk, sentence-transformers\nSuccessfully installed nltk-3.8.1 sentence-transformers-2.2.2\nCollecting transformers==4.34.0\n  Downloading transformers-4.34.0-py3-none-any.whl.metadata (121 kB)\n\u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m121.5/121.5 kB\u001b[0m \u001b[31m16.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n\u001b[?25hRequirement already satisfied: filelock in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from transformers==4.34.0) (3.9.0)\nRequirement already satisfied: huggingface-hub<1.0,>=0.16.4 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from transformers==4.34.0) (0.19.4)\nRequirement already satisfied: numpy>=1.17 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from transformers==4.34.0) (1.23.5)\nRequirement already satisfied: packaging>=20.0 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from transformers==4.34.0) (23.0)\nRequirement already satisfied: pyyaml>=5.1 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from transformers==4.34.0) (6.0)\nRequirement already satisfied: regex!=2019.12.17 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from transformers==4.34.0) (2023.10.3)\nRequirement already satisfied: requests in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from transformers==4.34.0) (2.31.0)\nCollecting tokenizers<0.15,>=0.14 (from transformers==4.34.0)\n  Downloading tokenizers-0.14.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.7 kB)\nRequirement already satisfied: safetensors>=0.3.1 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from transformers==4.34.0) (0.4.1)\nRequirement already satisfied: tqdm>=4.27 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from transformers==4.34.0) (4.65.0)\nRequirement already satisfied: fsspec>=2023.5.0 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from huggingface-hub<1.0,>=0.16.4->transformers==4.34.0) (2023.10.0)\nRequirement already satisfied: typing-extensions>=3.7.4.3 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from huggingface-hub<1.0,>=0.16.4->transformers==4.34.0) (4.8.0)\nCollecting huggingface-hub<1.0,>=0.16.4 (from transformers==4.34.0)\n  Downloading huggingface_hub-0.17.3-py3-none-any.whl.metadata (13 kB)\nRequirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from requests->transformers==4.34.0) (2.0.4)\nRequirement already satisfied: idna<4,>=2.5 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from requests->transformers==4.34.0) (3.4)\nRequirement already satisfied: urllib3<3,>=1.21.1 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from requests->transformers==4.34.0) (1.26.18)\nRequirement already satisfied: certifi>=2017.4.17 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from requests->transformers==4.34.0) (2023.7.22)\nDownloading transformers-4.34.0-py3-none-any.whl (7.7 MB)\n\u001b[2K   \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m7.7/7.7 MB\u001b[0m \u001b[31m71.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n\u001b[?25hDownloading tokenizers-0.14.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.8 MB)\n\u001b[2K   \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m3.8/3.8 MB\u001b[0m \u001b[31m104.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m\n\u001b[?25hDownloading huggingface_hub-0.17.3-py3-none-any.whl (295 kB)\n\u001b[2K   \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m295.0/295.0 kB\u001b[0m \u001b[31m54.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n\u001b[?25hInstalling collected packages: huggingface-hub, tokenizers, transformers\n  Attempting uninstall: huggingface-hub\n    Found existing installation: huggingface-hub 0.19.4\n    Uninstalling huggingface-hub-0.19.4:\n      Successfully uninstalled huggingface-hub-0.19.4\n  Attempting uninstall: tokenizers\n    Found existing installation: tokenizers 0.15.0\n    Uninstalling tokenizers-0.15.0:\n      Successfully uninstalled tokenizers-0.15.0\n  Attempting uninstall: transformers\n    Found existing installation: transformers 4.35.2\n    Uninstalling transformers-4.35.2:\n      Successfully uninstalled transformers-4.35.2\nSuccessfully installed huggingface-hub-0.17.3 tokenizers-0.14.1 transformers-4.34.0\nCollecting flask-sqlalchemy\n  Downloading flask_sqlalchemy-3.1.1-py3-none-any.whl.metadata (3.4 kB)\nCollecting flask>=2.2.5 (from flask-sqlalchemy)\n  Downloading flask-3.0.0-py3-none-any.whl.metadata (3.6 kB)\nCollecting sqlalchemy>=2.0.16 (from flask-sqlalchemy)\n  Downloading SQLAlchemy-2.0.23-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (9.6 kB)\nCollecting Werkzeug>=3.0.0 (from flask>=2.2.5->flask-sqlalchemy)\n  Downloading werkzeug-3.0.1-py3-none-any.whl.metadata (4.1 kB)\nRequirement already satisfied: Jinja2>=3.1.2 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from flask>=2.2.5->flask-sqlalchemy) (3.1.2)\nCollecting itsdangerous>=2.1.2 (from flask>=2.2.5->flask-sqlalchemy)\n  Downloading itsdangerous-2.1.2-py3-none-any.whl (15 kB)\nCollecting click>=8.1.3 (from flask>=2.2.5->flask-sqlalchemy)\n  Downloading click-8.1.7-py3-none-any.whl.metadata (3.0 kB)\nCollecting blinker>=1.6.2 (from flask>=2.2.5->flask-sqlalchemy)\n  Downloading blinker-1.7.0-py3-none-any.whl.metadata (1.9 kB)\nRequirement already satisfied: typing-extensions>=4.2.0 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from sqlalchemy>=2.0.16->flask-sqlalchemy) (4.8.0)\nRequirement already satisfied: greenlet!=0.4.17 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from sqlalchemy>=2.0.16->flask-sqlalchemy) (2.0.1)\nRequirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/Python-RT23.1/lib/python3.10/site-packages (from Jinja2>=3.1.2->flask>=2.2.5->flask-sqlalchemy) (2.1.1)\n", "name": "stdout"}, {"output_type": "stream", "text": "Downloading flask_sqlalchemy-3.1.1-py3-none-any.whl (25 kB)\nDownloading flask-3.0.0-py3-none-any.whl (99 kB)\n\u001b[2K   \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m99.7/99.7 kB\u001b[0m \u001b[31m28.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n\u001b[?25hDownloading SQLAlchemy-2.0.23-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.0 MB)\n\u001b[2K   \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m3.0/3.0 MB\u001b[0m \u001b[31m108.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n\u001b[?25hDownloading blinker-1.7.0-py3-none-any.whl (13 kB)\nDownloading click-8.1.7-py3-none-any.whl (97 kB)\n\u001b[2K   \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m97.9/97.9 kB\u001b[0m \u001b[31m27.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n\u001b[?25hDownloading werkzeug-3.0.1-py3-none-any.whl (226 kB)\n\u001b[2K   \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m226.7/226.7 kB\u001b[0m \u001b[31m49.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n\u001b[?25hInstalling collected packages: Werkzeug, sqlalchemy, itsdangerous, click, blinker, flask, flask-sqlalchemy\n  Attempting uninstall: Werkzeug\n    Found existing installation: Werkzeug 2.2.3\n    Uninstalling Werkzeug-2.2.3:\n      Successfully uninstalled Werkzeug-2.2.3\n  Attempting uninstall: sqlalchemy\n    Found existing installation: SQLAlchemy 1.4.39\n    Uninstalling SQLAlchemy-1.4.39:\n      Successfully uninstalled SQLAlchemy-1.4.39\n  Attempting uninstall: click\n    Found existing installation: click 8.0.4\n    Uninstalling click-8.0.4:\n      Successfully uninstalled click-8.0.4\n  Attempting uninstall: blinker\n    Found existing installation: blinker 1.4\n    Uninstalling blinker-1.4:\n      Successfully uninstalled blinker-1.4\n\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\nlale 0.7.7 requires click==8.0.4, but you have click 8.1.7 which is incompatible.\u001b[0m\u001b[31m\n\u001b[0mSuccessfully installed Werkzeug-3.0.1 blinker-1.7.0 click-8.1.7 flask-3.0.0 flask-sqlalchemy-3.1.1 itsdangerous-2.1.2 sqlalchemy-2.0.23\nCollecting wget\n  Downloading wget-3.2.zip (10 kB)\n  Preparing metadata (setup.py) ... \u001b[?25ldone\n\u001b[?25hBuilding wheels for collected packages: wget\n  Building wheel for wget (setup.py) ... \u001b[?25ldone\n\u001b[?25h  Created wheel for wget: filename=wget-3.2-py3-none-any.whl size=9656 sha256=79e31f7f512cb028aef4247385795eac15c5bad9ba49f783c11daeddf886b584\n  Stored in directory: /tmp/wsuser/.cache/pip/wheels/8b/f1/7f/5c94f0a7a505ca1c81cd1d9208ae2064675d97582078e6c769\nSuccessfully built wget\nInstalling collected packages: wget\nSuccessfully installed wget-3.2\n", "name": "stdout"}]}, {"metadata": {}, "id": "4adcdb35", "cell_type": "code", "source": "# First import the dependencies we need:\nimport os\nfrom dotenv import load_dotenv\nfrom time import sleep\ntry:\n    from langchain import PromptTemplate\n    from langchain.chains import LLMChain, SimpleSequentialChain\n    from langchain.document_loaders import PyPDFLoader\n    from langchain.indexes import VectorstoreIndexCreator # Vectorize db index with chromadb\n    from langchain.embeddings import HuggingFaceEmbeddings # For using HuggingFace embedding models\n    from langchain.text_splitter import CharacterTextSplitter # Text splitter\n\n    from ibm_watson_machine_learning.metanames import GenTextParamsMetaNames as GenParams\n    from ibm_watson_machine_learning.foundation_models import Model\n    from ibm_watson_machine_learning.foundation_models.extensions.langchain import WatsonxLLM\n    import wget\nexcept ImportError as e:\n    print(e)\n\nprint(\"Done importing dependencies.\")\n", "execution_count": 2, "outputs": [{"output_type": "stream", "text": "Done importing dependencies.\n", "name": "stdout"}]}, {"metadata": {}, "id": "96339420", "cell_type": "code", "source": "# Get our API key and URL from .env\nload_dotenv()\napi_key = 'mXMwkYwWPgTl2ORNtFEfhCOlkOFQQt6f92hZITvDSKEy'\nibm_cloud_url = 'https://us-south.ml.cloud.ibm.com'\nproject_id = os.getenv(\"PROJECT_ID\", None)\n\nif api_key is None or ibm_cloud_url is None or project_id is None:\n    raise Exception(\"One or more environment variables are missing!\")\nelse:\n    creds = {\n        \"url\": ibm_cloud_url,\n        \"apikey\": api_key \n    }\nprint(\"Done getting env variables.\")\n", "execution_count": 3, "outputs": [{"output_type": "stream", "text": "Done getting env variables.\n", "name": "stdout"}]}, {"metadata": {}, "id": "a51cbd27", "cell_type": "code", "source": "# Initialize the WatsonX model\n\n# parameters = {\n#     \"decoding_method\": \"sample\",\n#     \"min_new_tokens\": 1,\n#     \"max_new_tokens\": 100,\n#     \"temperature\": 0.2,\n#     \"top_k\": 25,\n#     \"top_p\": 1,\n#     \"repetition_penalty\": 1.0\n# }\n\nparams = {\n    GenParams.DECODING_METHOD: \"sample\",\n    GenParams.TEMPERATURE: 0.2,\n    GenParams.TOP_P: 1,\n    GenParams.TOP_K: 25,\n    GenParams.REPETITION_PENALTY: 1.0,\n    GenParams.MIN_NEW_TOKENS: 1,\n    GenParams.MAX_NEW_TOKENS: 20\n}\n\nllm_model = Model(\n    model_id=\"google/flan-ul2\",\n    params=params,\n    credentials=creds,\n    project_id=project_id\n)\nprint(\"Done initializing LLM.\")\n", "execution_count": 4, "outputs": [{"output_type": "stream", "text": "Done initializing LLM.\n", "name": "stdout"}]}, {"metadata": {}, "id": "1b25c008", "cell_type": "code", "source": "# Predict with the model\ncountries = [\"France\", \"Japan\", \"Australia\"]\n\ntry:\n  for country in countries:\n    question = f\"What is the capital of {country}\"\n    res = llm_model.generate_text(question)\n    print(f\"The capital of {country} is {res.capitalize()}\")\nexcept Exception as e:\n  print(e)\n", "execution_count": 5, "outputs": [{"output_type": "stream", "text": "The capital of France is Paris\nThe capital of Japan is Tokyo\nThe capital of Australia is Canberra\n", "name": "stdout"}]}, {"metadata": {}, "id": "a52e2e8f", "cell_type": "markdown", "source": "## 2. Prompt Templates & Chains\n\nIn the previous example, the user input is sent directly to the Watsonx LLM, without using Langchain. This is a basic use case, but real applications are rarely so simple. When using an LLM in an application, you will usually need to reuse the same prompt across multiple scenarios. We will now replicate the previous example, but use an LLM chain. This allows us to:\n\n- Accept user input and contruct a prompt\n- Generate multiple prompts from a collection of data points in a dataset "}, {"metadata": {}, "id": "310c2bbc", "cell_type": "code", "source": "# Define the prompt template\nprompt = PromptTemplate(\n  input_variables=[\"country\"],\n  template= \"What is the capital of {country}?\",\n)\n\ntry:\n  # In order to use Langchain, we need to instantiate Langchain extension\n  lc_llm_model = WatsonxLLM(model=llm_model)\n  \n  # Define a chain based on model and prompt\n  chain = LLMChain(llm=lc_llm_model, prompt=prompt)\n\n  # Getting predictions\n  countries = [\"Sweden\", \"Mexico\", \"Vietnam\"]\n  for country in countries:\n    response = chain.run(country)\n    print(prompt.format(country=country) + \" = \" + response.capitalize())\n    sleep(0.5)\nexcept Exception as e:\n  print(e)\n", "execution_count": 6, "outputs": [{"output_type": "stream", "text": "What is the capital of Sweden? = Stockholm\nWhat is the capital of Mexico? = Mexico city\nWhat is the capital of Vietnam? = Hanoi\n", "name": "stdout"}]}, {"metadata": {}, "id": "918a9df3", "cell_type": "markdown", "source": "## 3. Simple sequential chains\nThe utility of LangChain becomes apparent as we chain outputs of one model as input to another model. Here's a simple example where one generates a question which the other model answers.\n\nLangChain determines a model's output based on its response.  In our examples, the first model creates a response to the end prompt of \"Question:\" which LangChain maps as an input variable called \"question\" which it passes to the 2nd model."}, {"metadata": {}, "id": "ffda7c24", "cell_type": "code", "source": "# Create two sequential prompts \npt1 = PromptTemplate(input_variables=[\"topic\"], template=\"Generate a random question about {topic}: Question: \")\npt2 = PromptTemplate(\n    input_variables=[\"question\"],\n    template=\"Answer the following question: {question}\",\n)\nprint(\"Done.\")\n", "execution_count": 7, "outputs": [{"output_type": "stream", "text": "Done.\n", "name": "stdout"}]}, {"metadata": {}, "id": "23e4e1ee", "cell_type": "code", "source": "# Instantiate 2 models (Note, these could be different models depending on use case)\n# Note the .to_langchain() method which returns a WatsonxLLM wrapper, like above.\nmodel_1 = Model(\n    model_id=\"google/flan-ul2\",\n    params=params,\n    credentials=creds,\n    project_id=project_id\n).to_langchain()\nmodel_2 = Model(\n    model_id=\"google/flan-ul2\",\n    credentials=creds,\n    project_id=project_id\n).to_langchain()\n\nprint(\"Done.\")\n", "execution_count": 8, "outputs": [{"output_type": "stream", "text": "Done.\n", "name": "stdout"}]}, {"metadata": {}, "id": "35de1e4d", "cell_type": "code", "source": "# Construct the sequential chain\nprompt_to_model_1 = LLMChain(llm=model_1, prompt=pt1)\nprompt_to_model_2 = LLMChain(llm=model_2, prompt=pt2)\nqa = SimpleSequentialChain(chains=[prompt_to_model_1, prompt_to_model_2], verbose=True)\nprint(\"Done.\")\n", "execution_count": 9, "outputs": [{"output_type": "stream", "text": "Done.\n", "name": "stdout"}]}, {"metadata": {}, "id": "34586549", "cell_type": "code", "source": "# Run our chain with the topic: \"an animal\"\n# Play around with providing different topics to see the output. eg. cars, the Roman empire\ntry:\n  qa.run(\"an animal\")\nexcept Exception as e:\n  print(e)\n", "execution_count": 15, "outputs": [{"output_type": "stream", "text": "\n\n\u001b[1m> Entering new SimpleSequentialChain chain...\u001b[0m\n\u001b[36;1m\u001b[1;3mWhat is the name of the largest lizard in the world?\u001b[0m\n\u001b[33;1m\u001b[1;3mgila monster\u001b[0m\n\n\u001b[1m> Finished chain.\u001b[0m\n", "name": "stdout"}]}, {"metadata": {}, "id": "ed7c152d", "cell_type": "markdown", "source": "## 4. Easy Loading of Documents Using Lang Chain\nLangChain makes it easy to extract passages from documents so that you can answer questions based on your document's content. First download the example PDF file to your working folder: [what is generative ai.pdf](https://github.com/ibm-build-lab/VAD-VAR-Workshop/blob/main/content/Watsonx/WatsonxAI/105/what%20is%20generative%20ai.pdf)"}, {"metadata": {}, "id": "bf2642d8", "cell_type": "code", "source": "!wget https://vest-watsonx.s3.us-south.cloud-object-storage.appdomain.cloud/what%20is%20generative%20ai.pdf\n!ls", "execution_count": 19, "outputs": [{"output_type": "stream", "text": "--2023-11-27 14:13:51--  https://vest-watsonx.s3.us-south.cloud-object-storage.appdomain.cloud/what%20is%20generative%20ai.pdf\nResolving vest-watsonx.s3.us-south.cloud-object-storage.appdomain.cloud (vest-watsonx.s3.us-south.cloud-object-storage.appdomain.cloud)... 169.46.118.100\nConnecting to vest-watsonx.s3.us-south.cloud-object-storage.appdomain.cloud (vest-watsonx.s3.us-south.cloud-object-storage.appdomain.cloud)|169.46.118.100|:443... connected.\nHTTP request sent, awaiting response... 200 OK\nLength: 310867 (304K) [application/pdf]\nSaving to: \u2018what is generative ai.pdf\u2019\n\nwhat is generative  100%[===================>] 303.58K  --.-KB/s    in 0.002s  \n\n2023-11-27 14:13:51 (186 MB/s) - \u2018what is generative ai.pdf\u2019 saved [310867/310867]\n\n'what is generative ai.pdf'\n", "name": "stdout"}]}, {"metadata": {}, "id": "b1743ccd", "cell_type": "code", "source": "# Load PDF document\npdf='what is generative ai.pdf'\n\nloaders = [PyPDFLoader(pdf)]\nprint(\"Done.\")\n", "execution_count": 20, "outputs": [{"output_type": "stream", "text": "Done.\n", "name": "stdout"}]}, {"metadata": {}, "id": "9348f8c4", "cell_type": "code", "source": "# Index loaded PDF\nindex = VectorstoreIndexCreator(\n    embedding=HuggingFaceEmbeddings(),\n    text_splitter=CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)).from_loaders(loaders)\nprint(\"Done.\")\n", "execution_count": 21, "outputs": [{"output_type": "display_data", "data": {"text/plain": "Downloading .gitattributes:   0%|          | 0.00/1.18k [00:00<?, ?B/s]", "application/vnd.jupyter.widget-view+json": {"version_major": 2, "version_minor": 0, "model_id": "0dab96edddeb4a068749d7739b7281a7"}}, "metadata": {}}, {"output_type": "display_data", "data": {"text/plain": "Downloading 1_Pooling/config.json:   0%|          | 0.00/190 [00:00<?, ?B/s]", "application/vnd.jupyter.widget-view+json": {"version_major": 2, "version_minor": 0, "model_id": "389b08ccb99145e381faf6425641d11f"}}, "metadata": {}}, {"output_type": "display_data", "data": {"text/plain": "Downloading README.md:   0%|          | 0.00/10.6k [00:00<?, ?B/s]", "application/vnd.jupyter.widget-view+json": {"version_major": 2, "version_minor": 0, "model_id": "9e9fdea677134d4385cf8ee9cd3aef13"}}, "metadata": {}}, {"output_type": "display_data", "data": {"text/plain": "Downloading config.json:   0%|          | 0.00/571 [00:00<?, ?B/s]", "application/vnd.jupyter.widget-view+json": {"version_major": 2, "version_minor": 0, "model_id": "457b95efb6c2417ab3045b6c930786ac"}}, "metadata": {}}, {"output_type": "display_data", "data": {"text/plain": "Downloading (\u2026)ce_transformers.json:   0%|          | 0.00/116 [00:00<?, ?B/s]", "application/vnd.jupyter.widget-view+json": {"version_major": 2, "version_minor": 0, "model_id": "e4fbec1672bd4f27ad35337c60567b35"}}, "metadata": {}}, {"output_type": "display_data", "data": {"text/plain": "Downloading data_config.json:   0%|          | 0.00/39.3k [00:00<?, ?B/s]", "application/vnd.jupyter.widget-view+json": {"version_major": 2, "version_minor": 0, "model_id": "ff599cbb587841d48c1e24f841e56d85"}}, "metadata": {}}, {"output_type": "display_data", "data": {"text/plain": "Downloading pytorch_model.bin:   0%|          | 0.00/438M [00:00<?, ?B/s]", "application/vnd.jupyter.widget-view+json": {"version_major": 2, "version_minor": 0, "model_id": "15010a94f424458fb3002ca4cb024db6"}}, "metadata": {}}, {"output_type": "display_data", "data": {"text/plain": "Downloading (\u2026)nce_bert_config.json:   0%|          | 0.00/53.0 [00:00<?, ?B/s]", "application/vnd.jupyter.widget-view+json": {"version_major": 2, "version_minor": 0, "model_id": "120bb6d3fe7f4397b37875d82c70cb66"}}, "metadata": {}}, {"output_type": "display_data", "data": {"text/plain": "Downloading (\u2026)cial_tokens_map.json:   0%|          | 0.00/239 [00:00<?, ?B/s]", "application/vnd.jupyter.widget-view+json": {"version_major": 2, "version_minor": 0, "model_id": "8162624c93934f7f92358bcb9d9c4fc7"}}, "metadata": {}}, {"output_type": "display_data", "data": {"text/plain": "Downloading tokenizer.json:   0%|          | 0.00/466k [00:00<?, ?B/s]", "application/vnd.jupyter.widget-view+json": {"version_major": 2, "version_minor": 0, "model_id": "87892e09bb9d4ff1af23d8c7f3e3fd09"}}, "metadata": {}}, {"output_type": "display_data", "data": {"text/plain": "Downloading tokenizer_config.json:   0%|          | 0.00/363 [00:00<?, ?B/s]", "application/vnd.jupyter.widget-view+json": {"version_major": 2, "version_minor": 0, "model_id": "c1948eaf5c844f5b8ac7bb30f5e04417"}}, "metadata": {}}, {"output_type": "display_data", "data": {"text/plain": "Downloading train_script.py:   0%|          | 0.00/13.1k [00:00<?, ?B/s]", "application/vnd.jupyter.widget-view+json": {"version_major": 2, "version_minor": 0, "model_id": "52840f2cbe3c4b84a9e8d18fea3fceee"}}, "metadata": {}}, {"output_type": "display_data", "data": {"text/plain": "Downloading vocab.txt:   0%|          | 0.00/232k [00:00<?, ?B/s]", "application/vnd.jupyter.widget-view+json": {"version_major": 2, "version_minor": 0, "model_id": "aaa67210350c4bf180ee0574075025a3"}}, "metadata": {}}, {"output_type": "display_data", "data": {"text/plain": "Downloading modules.json:   0%|          | 0.00/349 [00:00<?, ?B/s]", "application/vnd.jupyter.widget-view+json": {"version_major": 2, "version_minor": 0, "model_id": "e6e4173a8fc24f8aaefcc1b9ca7c47b0"}}, "metadata": {}}, {"output_type": "stream", "text": "Done.\n", "name": "stdout"}]}, {"metadata": {}, "id": "43a890f9", "cell_type": "code", "source": "# Initialize watsonx google/flan-ul2 model\nparams = {\n    GenParams.DECODING_METHOD: \"sample\",\n    GenParams.TEMPERATURE: 0.2,\n    GenParams.TOP_P: 1,\n    GenParams.TOP_K: 100,\n    GenParams.MIN_NEW_TOKENS: 50,\n    GenParams.MAX_NEW_TOKENS: 300\n}\nmodel = Model(\n    model_id=\"google/flan-ul2\",\n    params=params,\n    credentials=creds,\n    project_id=project_id\n).to_langchain()\n\nprint(\"Done.\")\n", "execution_count": 22, "outputs": [{"output_type": "stream", "text": "Done.\n", "name": "stdout"}]}, {"metadata": {}, "id": "7d1a50d6", "cell_type": "code", "source": "# Init RAG chain\nfrom langchain.chains import RetrievalQA\n\nchain = RetrievalQA.from_chain_type(llm=model, \n                                    chain_type=\"stuff\", \n                                    retriever=index.vectorstore.as_retriever(), \n                                    input_key=\"question\")\nprint(\"Done.\")\n", "execution_count": 23, "outputs": [{"output_type": "stream", "text": "Done.\n", "name": "stdout"}]}, {"metadata": {}, "id": "2b6efed8", "cell_type": "code", "source": "# Answer based on the document\nres = chain.run(\"what is Machine Learning?\")\nprint(res)\n", "execution_count": 24, "outputs": [{"output_type": "stream", "text": "Machine learning is a type of artificial intelligence. Through machine learning, practitioners develop artificial intelligence through models that can \u201clearn\u201d from data patterns without human direction. The unmanageably huge volume and complexity of data (unmanageable by humans, anyway) that is now being generated has increased the potential of machine learning, as well as the need for it.\n", "name": "stdout"}]}, {"metadata": {}, "id": "cc2679f2", "cell_type": "markdown", "source": "Retrieval Augmented Generation (RAG) is a common AI use case. Many companies have vast amounts of data about which they want an AI system to answer questions, do searches or perform summarization tasks. We will learn more about RAG in lab 106."}], "metadata": {"kernelspec": {"name": "python3", "display_name": "Python 3.10", "language": "python"}, "language_info": {"name": "python", "version": "3.10.13", "mimetype": "text/x-python", "codemirror_mode": {"name": "ipython", "version": 3}, "pygments_lexer": "ipython3", "nbconvert_exporter": "python", "file_extension": ".py"}}, "nbformat": 4, "nbformat_minor": 5}