UseCases/Natural_Language_Processing/Natural_Language_Processing_Python.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "9abfac3c-6662-4b57-b5a8-82f0f6505d71",
   "metadata": {},
   "source": [
    "<header>\n",
    "   <p  style='font-size:36px;font-family:Arial; color:#F0F0F0; background-color: #00233c; padding-left: 20pt; padding-top: 20pt;padding-bottom: 10pt; padding-right: 20pt;'>\n",
    "       Natural Languauge Processing in Vantage\n",
    "  <br>\n",
    "       <img id=\"teradata-logo\" src=\"https://storage.googleapis.com/clearscape_analytics_demo_data/DEMO_Logo/teradata.svg\" alt=\"Teradata\" style=\"width: 125px; height: auto; margin-top: 20pt;\">\n",
    "    </p>\n",
    "</header>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "507bfa51-059d-49a3-91d0-6a1de1d9f510",
   "metadata": {},
   "source": [
    "<p style = 'font-size:20px;font-family:Arial;color:#00233C'><b>Introduction</b></p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>\n",
    "Natural Language Processing (NLP) involves teaching computers to understand, interpret, and generate human language, just like people do. It's about enabling computers to read and understand text, so they can perform tasks that involve language, such as answering questions, understanding customer feedback, or even generating human-like responses.<br>Think of NLP as a translator between humans and computers. It allows computers to analyze and make sense of text data in a way that's meaningful for businesses. There are many uses of NLP that can be used in business for example <br><b>Customer Insights</b>: NLP helps businesses understand what their customers are saying across different channels like emails, reviews, or social media. It can analyze this text to identify trends, sentiments, and common issues, helping companies tailor their products and services to meet customer needs better.<br><b>Automated Support</b>: NLP powers chatbots and virtual assistants that can understand and respond to customer queries in real-time. These assistants can handle routine inquiries, provide product recommendations, or even troubleshoot problems, freeing up human agents for more complex tasks.<br><b>Information Extraction</b>: NLP can extract valuable information from unstructured text data, such as contracts, legal documents, or research papers. It helps businesses quickly find relevant information, identify key insights, and make informed decisions based on this data.<br><b>Personalization</b>: By analyzing customer interactions and preferences expressed in text, NLP enables businesses to personalize their marketing messages, offers, and user experiences. This personalized approach can lead to higher customer engagement and loyalty.<br><br>In essence, NLP empowers businesses to leverage the power of language to improve customer experiences, streamline operations, and drive better decision-making. \n",
    "<p style = 'font-size:18px;font-family:Arial;color:#00233C'><b>Business Values</b></p>\n",
    "<ul style = 'font-size:16px;font-family:Arial;color:#00233C'>\n",
    "    <li>Efficiency: Automating sentiment analysis saves time and resources compared to manual analysis.</li>\n",
    "    <li>Insights: Gain valuable insights into customer sentiment and preferences to drive strategic decision-making.</li>\n",
    "    <li>Proactive Response: Identify and address customer concerns and issues in real-time to improve customer satisfaction and loyalty.</li>\n",
    "    <li>Competitive Advantage: Stay ahead of competitors by continuously monitoring and adapting to changing customer sentiments and market trends.</li>\n",
    " </ul>\n",
    "<p style = 'font-size:18px;font-family:Arial;color:#00233C'><b>Why Vantage?</b></p>  \n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Natural Language Processing deals with huge amounts of data; doing pre-processing and using Vantage's InDB text processing functions saves time and can be easily scaled to meet business needs. Moreover, using Clearscape Analytics it is very easy to integrate widely used 3rd party LLM models like GPT with trusted business data.<br>In this demo we will process the comments received by a retail store using Vantage's InDb functions.  </p>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9a047ba1-e8ce-4228-83c8-fe3a6f8379fc",
   "metadata": {},
   "source": [
    "<hr style='height:2px;border:none;background-color:#00233C;'>\n",
    "<b style = 'font-size:20px;font-family:Arial;color:#00233c'>1. Configuring the environment</b>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9290b9db-a1cc-43e3-8639-3a7dd8dfd200",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "# '%%capture' suppresses the display of installation steps of the following packages\n",
    "!pip install -r requirements.txt --quiet"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ef6f671b-a24c-4911-87a8-10cd17fadfb1",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info\">\n",
    "    <p style = 'font-size:16px;font-family:Arial;color:#00233C'><b>Note: </b><i>The above statements will install the required libraries to run this demo. Be sure to restart the kernel after executing the above lines to bring the installed libraries into memory. The simplest way to restart the Kernel is by typing zero zero: <b> 0 0</b></i></p>\n",
    "    </div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "63fd7658-b33c-481f-80ee-0beff2ee2663",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<p style = 'font-size:20px;font-family:Arial;color:#00233C'><b>2. Connect to Vantage</b></p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "10130700-f95c-4832-8db7-39f21a679f9e",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Suppress warnings\n",
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\", category=DeprecationWarning) \n",
    "\n",
    "\n",
    "#import libraries\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt \n",
    "import getpass\n",
    "\n",
    "import timeit\n",
    "import tqdm\n",
    "from tqdm.notebook import *\n",
    "tqdm_notebook.pandas()\n",
    "\n",
    "# teradata lib\n",
    "from teradataml import *\n",
    "import teradataml\n",
    "from teradataml import configure\n",
    "from teradataml.analytics.valib import *\n",
    "configure.val_install_location = \"val\"\n",
    "\n",
    "# helper\n",
    "from utils.sql_helper_func import *\n",
    "\n",
    "display.max_rows = 5\n",
    "display.print_sqlmr_query=False\n",
    "display.suppress_vantage_runtime_warnings=True\n",
    "\n",
    "# markdown\n",
    "from IPython.display import display, Markdown"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b0969ef7-3e8d-4e11-a6ea-e6e5dcf11ff6",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>We will be prompted to provide the password. We will enter the password, press the Enter key, and then use the down arrow to go to the next cell.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0f334bfe-a214-49ec-86d8-06b027a5a2c4",
   "metadata": {},
   "outputs": [],
   "source": [
    "%run -i ../startup.ipynb\n",
    "eng = create_context(host = 'host.docker.internal', username = 'demo_user', password = password)\n",
    "print(eng)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5f822383-bb3c-447f-9a7e-730f98e74872",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%capture\n",
    "execute_sql('''SET query_band='DEMO=Natural_Language_Processing_Python.ipynb;' UPDATE FOR SESSION; ''')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2e0175c5-820e-4a8f-9353-6308c63f7715",
   "metadata": {},
   "source": [
    "<p style = 'font-size:18px;font-family:Arial;color:#00233C'> <b>Getting Data for This Demo</b></p>\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>We have provided data for this demo on cloud storage. We have the option of either running the demo using foreign tables to access the data without using any storage on our environment or downloading the data to local storage, which may yield somewhat faster execution. However, we need to consider available storage. There are two statements in the following cell, and one is commented out. We may switch which mode we choose by changing the comment string.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8b1a4c01-edd1-4c06-aa47-5451df2d0f42",
   "metadata": {},
   "outputs": [],
   "source": [
    "%run -i ../run_procedure.py \"call get_data('DEMO_Retail_local');\"\n",
    " # takes about 2 minute 30 seconds, estimated space: 90 MB\n",
    "#%run -i ../run_procedure.py \"call get_data('DEMO_Retail_cloud');\" \n",
    "# takes about 30 seconds, estimated space: 0 MB"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6451ecbf-6b2f-4193-b44f-eb472a0dbc7f",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Optional step – We should execute the below step only if we want to see the status of databases/tables created and space used.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0d650f5c-006a-4fb9-b6d7-1e705511b0da",
   "metadata": {},
   "outputs": [],
   "source": [
    "%run -i ../run_procedure.py \"call space_report();\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e1d38a9c-a891-45be-90b1-164085dd4597",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<p style = 'font-size:20px;font-family:Arial;color:#00233C'><b>3. Data Exploration</b></p>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2266bc7e-138a-408d-a772-66f0ca5ddbea",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Please check the db version should be above 17.20.03.21 for the functions to work correctly. If the database version is less than that, please create a new VM in Clearscape Analytics Experience.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5a8c997c-804b-4112-aa2a-6c1f6994cef7",
   "metadata": {},
   "outputs": [],
   "source": [
    "configure.database_version"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c1c0c277-c6f0-4365-bc51-c498f1bf771b",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Let us start by creating a \"Virtual DataFrame\" that points directly to the dataset in Vantage.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "49f69153-cd20-43bf-bd67-a55c888a75a1",
   "metadata": {},
   "outputs": [],
   "source": [
    "tdf_reviews = DataFrame('\"DEMO_Retail\".\"Web_Comment\"')\n",
    "tdf_reviews"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f24ad14a-ebf9-4937-94e2-d72d8f59f8d1",
   "metadata": {},
   "outputs": [],
   "source": [
    "tdf_reviews.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1b6b76d1-eb8a-4992-8159-e1d8f10bf337",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>We have around 23k comments in our dataset. Let us first remove the null comments.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b394a7f4-5a97-4978-a1bd-20710b937bcb",
   "metadata": {},
   "outputs": [],
   "source": [
    "tdf_nonull = tdf_reviews[tdf_reviews.comment_text.isnull() == 0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "702529d6-fa8c-4630-9b8f-c16d7b2977fd",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'> For demo purpose we will use 5k comments for our analysis.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0f4fb627-0fa4-4f60-af10-5db210c85b61",
   "metadata": {},
   "outputs": [],
   "source": [
    "tdf_sample = tdf_nonull.iloc[:5000, :]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2ea8d513-49b7-412e-a6bd-f2e7a4d0a204",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<p style = 'font-size:20px;font-family:Arial;color:#00233C'><b>4. Sentiment Extraction</b></p>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "18ba237e-bb17-44c0-9a23-bce43a0dbcb2",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Sentiment Extraction is the process of analyzing large volumes of text to determine whether it expresses a positive, negative, or neutral sentiment.<br> Clearscape Analytics SentimentExtractor \n",
    "uses a dictionary model to extract the sentiment (positive, negative, or neutral) of each input document or sentence. The  dictionary model consists of WordNet( a lexical database of the English language).The function handles negated sentiments as follows:<ul style = 'font-size:16px;font-family:Arial;color:#00233C'>\n",
    "    <li>-1 if the sentiment is negated. For example, I am not happy.</li>\n",
    "    <li>-1 if one word separates the sentiment and a negation word. For example, I am not very happy.</li>\n",
    "    <li>+1 if two or more words separate the sentiment and a negation word. For example, I am not saying I am happy.</li></ul>\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Let us use this function to see the overall sentiments of comments received in our sample dataset.</p>     "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2a7a4f72-7a87-4ca8-b304-c1c4fefcc01d",
   "metadata": {},
   "outputs": [],
   "source": [
    "sentimentextractor_out = SentimentExtractor(text_column=\"comment_text\",\n",
    "                                                data=tdf_sample,\n",
    "                                                accumulate=['comment_id', 'comment_text']\n",
    "                                                )\n",
    "\n",
    "senti = sentimentextractor_out.result\n",
    "senti"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "be4b8fab-465b-4463-bc54-227803b52474",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>As we can see that the function outputs the polarity , sentiment score and the sentiment words on which it calculated the score. In the sentiment_words output column the function returns the overall positive and negative scores, the words which are used for soring the sentiment -1 for negative and +1 for positive sentiment and in the brackets it also displays how many times the word is repeated in the comment. e.g beautiful 1 (2) means beautiful is positive sentiment word and has occured twice in the comment. We also have an option of providing the custom dictonary to the function.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4ef16958-285e-4682-9b3b-058cc32bc7bc",
   "metadata": {},
   "outputs": [],
   "source": [
    "d1=senti.select(['comment_id','polarity']).groupby('polarity').count()\n",
    "d1 = d1.assign(drop_columns=True,\n",
    "          Polarity=d1.polarity,\n",
    "          Count=d1.count_comment_id)\n",
    "d1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "043b7451-d83a-4ea0-be3e-9e41dc1afb94",
   "metadata": {},
   "outputs": [],
   "source": [
    "plot1 = d1.to_pandas()\n",
    "# Create a bar plot\n",
    "ax = plot1.plot(kind='bar', x='Polarity', y='Count', color='skyblue', edgecolor='black', figsize=(8, 6))\n",
    "\n",
    "# Add count labels on top of the bars\n",
    "for p in ax.patches:\n",
    "    ax.annotate(str(p.get_height()), (p.get_x() + p.get_width() / 2., p.get_height()),\n",
    "                ha='center', va='center', xytext=(0, 10), textcoords='offset points')\n",
    "\n",
    "# Rotate x-labels by 45 degrees\n",
    "ax.set_xticklabels(ax.get_xticklabels(), rotation=45, ha='right')\n",
    "\n",
    "# Add labels and title\n",
    "plt.title('Polarity of comments')\n",
    "plt.xlabel('Polarity')\n",
    "plt.ylabel('Counts')\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "503ab3bf-be96-463a-94cb-7bcd08882d58",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>From the above chart we can see that the comments are largely postive in sentiment.</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "40afb547-9610-4d9c-82d8-2e1ab1aa0e71",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<p style = 'font-size:20px;font-family:Arial;color:#00233C'><b>5. Integrating with OpenSource LLM and create Word Embeddings</b></p>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "14f793dc-c5af-4fd2-9c96-856892567ca9",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>\n",
    "Word embedding is a technique used in natural language processing (NLP) to represent words as dense vectors. This allows words with similar meanings to have similar representations. Word embeddings capture semantic relationships between words, enabling NLP models to better understand and process human language.<br><br>Traditional methods of representing words, such as one-hot encoding or bag-of-words, represent each word as a sparse vector where most elements are zero and only one element is one (for one-hot encoding) or a count of occurrences (for bag-of-words). These representations do not capture semantic similarity between words and can result in high-dimensional and sparse feature spaces.<br> <br>Word embeddings represent words as dense vectors of fixed dimensionality (e.g., 100, 200, or 300 dimensions) where each dimension represents a different aspect of the word's meaning. These vectors are learned from large corpora of text using techniques like Word2Vec, GloVe, or FastText.The key idea behind word embeddings is that words that occur in similar contexts tend to have similar meanings. By training word embeddings on large text corpora, the model learns to map words with similar meanings to nearby points in the vector space. For example, in a well-trained word embedding model, the vectors for \"male\" and \"female\" are expected to be closer to each other than to the vector for \"apple\".\n",
    "</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "93c32045-7331-4fa6-811a-9a2046c91a03",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'> Large Language Models are trained on huge amounts of data enabling them to learn patterns, grammar, and context from a wide range of topics. They can be fine-tuned for specific tasks, such as question-answering, natural language understanding, and text generation etc and have a wide range of uses across various domains due to their ability to understand and generate human-like text.<br>In this demo we will use text-embedding-3-small model from OpenAI for our embeddings generation. Clearscape analytics can integrate with any opensource, OpenAI or cloud provider specific LLM (AWS Sagemaker/AWS Bedrock). Please refer demo index for demos on other integrations with TDApiClient. </p>\n",
    "\n",
    "\n",
    "<p style='font-size:16px;font-family:Arial;color:#00233C'>OpenAI and Azure OpenAI, provide multiple APIs for our hosted models. We introduce integration with the embedding API, which can be used in various types of applications: Classification, Search, Recommendations, and Anomaly detection. For more information on our Teradata API Integration, click <a href='https://docs.teradata.com/r/Enterprise_IntelliFlex_VMware/Teradata-VantageTM-API-Integration-Guide-for-Cloud-Machine-Learning/Teradata-Partner-API/Welcome-to-Teradata-API-Integration'>here.</a></p>\n",
    "\n",
    "<p style='font-size:16px;font-family:Arial;color:#00233C'>Under the hood, we will utilize the OpenAI embeddings method to generate the embeddings. OpenAI embeddings are a type of word embedding that we can use to represent products in a way that captures their semantic meaning. To generate embeddings for a product table, we will use the product name field. We will employ the OpenAI Embeddings API to generate embeddings for each product. Please refer to the <a href=\"https://platform.openai.com/docs/guides/embeddings\"> Embeddings documentation</a> for more information about embeddings and types of models available.</p>\n",
    "\n",
    "<p style='font-size:16px;font-family:Arial;color:#00233C'>The OpenAI Embeddings API takes a text string as input and returns a vector of numbers that represent the embedding. The length of the vector depends on the model that we are using. For example, the <b>text-embedding-3-small</b> model returns a vector of 1536 numbers.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aac9013d-b38a-4d64-a83e-1c9f5c76dbcb",
   "metadata": {},
   "outputs": [],
   "source": [
    "df1=tdf_sample.drop(['comment_summary'],axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a69d915a-b4c1-47fc-ae75-36540a454caa",
   "metadata": {},
   "source": [
    "\n",
    "<div class=\"alert alert-block alert-warning\">\n",
    "    <p style = 'font-size:16px;font-family:Arial;color:#00233C'><i><b>Note:</b> In this section, we are creating the OpenAI embeddings for 5000 comment texts. It will cost us a few dollars on our OpenAI account.</i></p>\n",
    "</div>\n",
    "\n",
    "<hr style='height:1px;border:none;background-color:#00233C;'>\n",
    "<p style = 'font-size:18px;font-family:Arial;color:#00233c'><b>5.1 Get the OpenAI API key</b></p>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "06dfad04-ff27-4631-a5b4-cbc3360bc9ca",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>We will first create function which will create embeddings from the opensouce model we are using. We are using <b>text-embedding-3-small</b> model you are free to change the model as per your needs.</p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>In order to utilize this demo, you will need an OpenAI API key. If you do not have one, please refer to the instructions provided in this guide to obtain your OpenAI API key: </p>\n",
    "\n",
    "\n",
    "<a href=\"..//Openai_setup_api_key/Openai_setup_api_key.md\" style=\"text-decoration:none;\" target=\"_blank\"><button style=\"font-size:16px;font-family:Arial;color:#fff;background-color:#00233C;border:none;border-radius:5px;cursor:pointer;height:50px;line-height:50px;display:flex;align-items:center;\">OpenAI API Key Guide <span style=\"margin-left:10px;\">&#8658;</span></button>\n",
    "</a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c8ef1f89-55fc-43fa-afb9-57e08862a7e0",
   "metadata": {},
   "outputs": [],
   "source": [
    "import getpass\n",
    "\n",
    "# enter your openai api key\n",
    "api_key = getpass.getpass(\"\\n Please Enter OpenAI API key: \")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "328fe31a-2b47-4bcd-a6d6-8838c77de6e4",
   "metadata": {},
   "outputs": [],
   "source": [
    "from tdapiclient import TDApiClient, create_tdapi_context\n",
    "\n",
    "# set embedding model from openai models\n",
    "embedding_model = \"text-embedding-3-small\"\n",
    "\n",
    "def generate_embeddings_tdapiclient(tdf, api_key, text_column):\n",
    "    return TDApiClient.API_Request(dataframe=tdf, \n",
    "                                   api_type=\"open-ai-embedding\",\n",
    "                                   model_name=embedding_model,\n",
    "                                   authorization ='''{{\"Key\":\"{}\"}}'''.format(api_key),\n",
    "                                   text_column=text_column)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "44335f15-6957-4abe-ad80-35d30e14cd14",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-warning\">\n",
    "    <p style = 'font-size:16px;font-family:Arial;color:#00233C'><i><b>Note:</b> In this section, we are creating the OpenAI embeddings for 5000 comment text. It will cost us a few dollars on our OpenAI account.</i></p>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae4841f6-9827-46cc-a928-6d8dbf6a0045",
   "metadata": {},
   "source": [
    "<hr style='height:1px;border:none;background-color:#00233C;'>\n",
    "\n",
    "<p style = 'font-size:18px;font-family:Arial;color:#00233c'><b>5.2 Do you want to generate the embeddings?</b></p>    \n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>We have already generated embeddings for the comment text and stored them in <b>Vantage</b> table.</p>\n",
    "\n",
    "<center><img src=\"images/decision_emb_gen_cmt2.svg\" alt=\"embeddings_decision\" width=300 height=400/></center>\n",
    "\n",
    "<div class=\"alert alert-block alert-info\">\n",
    "    <p style = 'font-size:16px;font-family:Arial;color:#00233C'><i><b>Note: If you would like to skip the embedding generation step to save the time and move quickly to nect step, please enter \"No\" in the next prompt.</b></i></p>\n",
    "</div>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>To save time, you can move to the already generated embeddings section. However, if you would like to see how we generate the embeddings, or if you need to generate the embeddings for a different dataset, then continue to the following section.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "48c8c10d-73e7-4599-a24d-062abf2c33bc",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_section52_desc_start():\n",
    "    return \"\"\"<div class=\"alert alert-block alert-warning\">\n",
    "    <p style = 'font-size:16px;font-family:Arial;color:#00233C'><i><b>Please be patient, The embedding generation step is estimated to take approximately 5 minutes to complete. </b></i></p>\n",
    "</div>\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9ef47cfd-54f2-476e-ab8b-fcc70f49caa9",
   "metadata": {},
   "outputs": [],
   "source": [
    "user_conformation = input(\"Do you want to generate the embeddings? (yes/no): \")\n",
    "\n",
    "if user_conformation.lower() == 'yes':\n",
    "    print(\"Generating the embeddings...\")\n",
    "    display(Markdown(get_section52_desc_start()))\n",
    "    comment_embeddings = generate_embeddings_tdapiclient(df1, api_key, \"comment_text\")\n",
    "    copy_to_sql(comment_embeddings, table_name=\"comment_embeddings\",primary_index='comment_id', if_exists='append')\n",
    "    comment_embeddings = DataFrame(\"comment_embeddings\")\n",
    "    print(\"Embeddings generated successfully.\")\n",
    "else:\n",
    "    print(\"Loading the embeddings from Vantage table...\")\n",
    "    comment_embeddings = DataFrame(in_schema('DEMO_Retail', 'Comment_Embeddings'))  \n",
    "    print(\"Existing embeddings loaded to Comment_Embeddings table.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "21336cea-cbd4-4702-9d1a-40ecb3d1835f",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "print(\"Data information: \\n\",comment_embeddings.shape)\n",
    "comment_embeddings"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b8d84c7c-6df4-4a61-8b66-b9af97d83e3b",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<p style = 'font-size:20px;font-family:Arial;color:#00233C'><b>6. Kmeans clustering using the embeddings</b></p>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a7f094f7-2ec9-488d-bd17-b1ab11853e10",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>In the previous step we have created embeddings from the text data we have. The generated embeddings are the features that can be used in machine learning algorithms. We will use Kmeans clustering to categorize the comments in different clusters.<br>First let us start by generating columnlist to be used in KMeans function.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "28d79e8e-3e7f-4346-8931-73912c584b9e",
   "metadata": {},
   "outputs": [],
   "source": [
    "embedding_column_list = comment_embeddings.columns\n",
    "embedding_column_list.remove(\"comment_id\")\n",
    "embedding_column_list.remove(\"customer_id\")\n",
    "embedding_column_list.remove(\"comment_text\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cc5ab58d-6dcc-47eb-a585-419300568c36",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run KMeans to find the clustering based on embeddings.\n",
    "kmeans_out = KMeans(\n",
    "    id_column=\"comment_id\",\n",
    "    data=comment_embeddings,\n",
    "    target_columns=embedding_column_list,\n",
    "    output_cluster_assignment=True,\n",
    "    num_clusters=7\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f875e0b2-c664-4c48-9330-4e4a44e65139",
   "metadata": {},
   "outputs": [],
   "source": [
    "kmeans_df=kmeans_out.result\n",
    "kmeans_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3b4254ba-bb23-43d6-8b83-b90fdb399cbf",
   "metadata": {},
   "outputs": [],
   "source": [
    "d2 = kmeans_df.groupby('td_clusterid_kmeans').count()\n",
    "d2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a262d769-7f7e-4619-b237-f566fe4df279",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Convert to pandas DataFrame \n",
    "plot2 = d2.to_pandas()\n",
    "\n",
    "# Plotting the bar chart\n",
    "ax = plot2.plot(kind='bar', x='td_clusterid_kmeans', y='count_comment_id', color='skyblue', edgecolor='black', figsize=(8, 6))\n",
    "\n",
    "# Add labels and title\n",
    "plt.title('Comments in each cluster')\n",
    "plt.xlabel('Cluster_Id')\n",
    "plt.ylabel('Counts')\n",
    "\n",
    "# Show count on top of bars\n",
    "for p in ax.patches:\n",
    "    ax.annotate(str(p.get_height()), (p.get_x() + p.get_width() / 2., p.get_height()), ha='center', va='center', xytext=(0, 10), textcoords='offset points')\n",
    "\n",
    "# Rotate x-labels by 45 degrees\n",
    "plt.xticks(rotation=360)\n",
    "\n",
    "# Show the plot\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a863299a-231b-429b-a7e3-7a4d49013682",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<p style = 'font-size:20px;font-family:Arial;color:#00233C'><b>7. PCA</b></p>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8b5bc2e1-4654-4e66-b615-7c3a025a9604",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Principal Component Analysis (PCA) is a technique used for dimensionality reduction in data analysis and machine learning. It works by transforming the original high-dimensional data into a lower-dimensional space while retaining as much of the original variance as possible. PCA achieves this by identifying the principal components, which are the directions in feature space along which the data varies the most. These principal components are computed as the eigenvectors of the covariance matrix of the standardized data, and they represent the most significant sources of variation in the data. By selecting a subset of the principal components that capture the most variance, PCA allows for a more compact representation of the data while preserving its essential structure and relationships. The transformed data can be used for visualization, feature extraction, noise reduction, and other analysis tasks, making PCA a powerful tool for data exploration and dimensionality reduction.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1a32c236-a419-43b1-8612-216b0fcd9153",
   "metadata": {},
   "outputs": [],
   "source": [
    "pca_df=comment_embeddings.join(other = kmeans_df, on = [\"comment_id\"], how = \"inner\",lprefix = \"emb\", rprefix = \"kmeans\")\n",
    "pca_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fd982e8a-6d46-4acf-950c-6e95b6669b24",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "pca_obj = valib.PCA(data=pca_df,\n",
    "                        columns=embedding_column_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "52f8060a-7c5b-462d-9c6b-5c12b7b81bf3",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info\" id=\"no-azure\">\n",
    "    <p style = 'font-size:16px;font-family:Arial;color:#00233C'><i><b>Note</b>: Please be patient. We are currently performing some mathematical calculations. This process may take 3-5 minutes.</i></p>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b4a1754a-04d1-4be1-87f7-02b202464893",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get PCA scores using the model generated above\n",
    "obj = valib.PCAPredict(data=pca_df,\n",
    "                           model=pca_obj.result,\n",
    "                           index_columns=\"emb_comment_id\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "084b6803-ce54-4330-9faf-d5f01d0c2141",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Print the results\n",
    "obj.result"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6139e1f9-e563-4344-8cf4-fb18b4eecb2a",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>As we see from above the PCA has reduced 300+ embeddings to 80+ embeddings. We will plot the first 2 factors to see how our clusters looks.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4e207d5e-17bd-4e5b-bbda-a06ced5a3214",
   "metadata": {},
   "outputs": [],
   "source": [
    "out_reduced_df = obj.result.select(['emb_comment_id','Factor 1','Factor 2'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bb1e213c-b534-4826-9bcc-0775e9dd4239",
   "metadata": {},
   "outputs": [],
   "source": [
    "out_reduced_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b6e0003d-1f35-4f5d-9e07-b2b18c790ce2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Join the KMeans output with dataframe with reduced number of columns.\n",
    "final_df=kmeans_df.join(other = out_reduced_df, on = [\"comment_id = emb_comment_id\"], how = \"inner\",lprefix = \"l\", rprefix = \"r\")\n",
    "final_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cd9d467e-fc89-4bb8-b39b-15614cdbe5c9",
   "metadata": {},
   "outputs": [],
   "source": [
    "plot3=final_df.to_pandas().reset_index()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0c9c857c-f3da-440f-aace-a7294335dafd",
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(10, 8))\n",
    "scatter = plt.scatter(plot3['Factor 1'], plot3['Factor 2'], c=plot3['td_clusterid_kmeans'], cmap='viridis')\n",
    "plt.title('PCA Visualization of Clusters')\n",
    "plt.legend(*scatter.legend_elements(), title='Clusters')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3ee75544-f3cf-4a82-ae02-501d69f2031b",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<p style = 'font-size:20px;font-family:Arial;color:#00233C'><b>8. Term Frequency-Inverse Document Frequency</b></p>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "39fece81-f82c-4e95-8efa-73fe39fc94e8",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>\n",
    "TF-IDF (Term Frequency-Inverse Document Frequency) is a numerical statistic used in NLP to evaluate the importance of a word in a document relative to a collection of documents. It is calculated by multiplying two factors: the term frequency (TF), which measures how frequently a word occurs in a document, and the inverse document frequency (IDF), which penalizes words that are common across multiple documents in the collection. TF-IDF assigns higher weights to words that are frequent in a document but rare in other documents, allowing it to capture the discriminative power of words in distinguishing documents. This technique is commonly used for text mining, document classification, search engine ranking, and other tasks where the relevance of words needs to be assessed within a corpus of text data.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c175d66c-b15e-4363-9a19-45bbb9920f3a",
   "metadata": {},
   "outputs": [],
   "source": [
    "title_input=tdf_sample.join(other = kmeans_df, on = [\"comment_id\"],how = \"inner\",lprefix = \"l\", rprefix = \"r\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b865ed50-8c1f-4f11-b4e1-2f39f09c584d",
   "metadata": {},
   "outputs": [],
   "source": [
    "title_df = title_input.assign(\n",
    "    drop_columns=True,\n",
    "    comment_id=title_input.l_comment_id,\n",
    "    customer_id=title_input.customer_id,\n",
    "    comment_text=title_input.comment_text,\n",
    "    comment_summary=title_input.comment_summary,\n",
    "    cluster_id=title_input.td_clusterid_kmeans\n",
    " )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "55946d49-562b-4615-a9eb-7d45b4b716e9",
   "metadata": {},
   "outputs": [],
   "source": [
    "title_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4636d93e-b095-4b50-858a-c4a09e0eb08e",
   "metadata": {},
   "outputs": [],
   "source": [
    "copy_to_sql(df = title_df, table_name = 'title_comments', if_exists = 'replace',primary_index = \"comment_id\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dcc9c2c1-a810-4d51-827b-8463fddb917e",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>\n",
    "A text parser, also known as a text tokenizer, breaks a text into its constituent parts, such as words, phrases, sentences, or other meaningful units. The <b>TD_TextParser</b> function performs the following operations:\n",
    "<ul style = 'font-size:16px;font-family:Arial;color:#00233C'><li>Tokenizes the text in the specified column</li>\n",
    "    <li>Removes the punctuations from the text and converts the text to lowercase</li>\n",
    "    <li>Removes stop words from the text and converts the text to their root forms</li>\n",
    "    <li>Creates a row for each word in the output table</li>\n",
    "    <li>Performs stemming; that is, the function identifies the common root form of a word by removing or replacing word suffixes</li>\n",
    "    </ul>\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'> The output table generated from the TD_TextParser is fed to the <b>TD_TFIDF</b> function. TD_TFIDF function represents each document as an N-dimensional vector, where N is the number of terms in the document set (therefore, the document vector is sparse). Each entry in the document vector is the TF-IDF score of a term.</p>    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3208bc3d-1967-4e47-8c67-0b99ffcbd647",
   "metadata": {},
   "outputs": [],
   "source": [
    "qry1='''\n",
    "CREATE MULTISET TABLE tfidf_input_tokenized AS (\n",
    "SELECT comment_id, cast(token as varchar(15)) as token, cluster_id FROM TD_TextParser (\n",
    "ON title_comments AS InputTable\n",
    "USING\n",
    "TextColumn ('comment_text')\n",
    "ConvertToLowerCase ('true')\n",
    "OutputByWord ('true')\n",
    "Punctuation ('\\[.,-?\\!\\]')\n",
    "RemoveStopWords ('true')\n",
    "StemTokens ('true')\n",
    "Accumulate ('comment_id','cluster_id')\n",
    ") AS dt ) WITH DATA;\n",
    "'''\n",
    "\n",
    "qry2='''CREATE MULTISET TABLE tfidf_comments AS (\n",
    "SELECT * FROM TD_TFIDF (\n",
    "   ON tfidf_input_tokenized  AS InputTable\n",
    "   USING\n",
    "   DocIdColumn ('cluster_id')\n",
    "   TokenColumn ('token')\n",
    "   TFNormalization ('LOG')\n",
    "   IDFNormalization ('SMOOTH')\n",
    "   Regularization ('L2')\n",
    "   --Accumulate ('cluster_id')\n",
    ") AS dt ) WITH DATA;\n",
    "'''\n",
    "\n",
    "# Execute the query\n",
    "execute_sql(qry1)\n",
    "execute_sql(qry2)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a5cd1fa0-7194-4f95-b977-888116b9029a",
   "metadata": {},
   "outputs": [],
   "source": [
    "tfidf_comments = DataFrame(\"tfidf_comments\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8ce69281-a675-46ce-8090-836a66e70cd9",
   "metadata": {},
   "outputs": [],
   "source": [
    "tfidf_comments"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d89f4ed2-cb87-4a47-8606-e2ac7ecb7811",
   "metadata": {},
   "outputs": [],
   "source": [
    "window = tfidf_comments.window(partition_columns=\"cluster_id\",\n",
    "                               order_columns=\"TD_TF_IDF\"\n",
    "                              )\n",
    "\n",
    "# Execute rank() on a window.\n",
    "df = window.rank()\n",
    "df.sort('col_rank')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "40c3b77a-de5b-4a39-ab5b-707519b6bf72",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>From above we can see the frequency and importance of each word in the cluster.</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4cc47e14-a7e5-41c9-a637-e0fbc240250f",
   "metadata": {},
   "source": [
    "<p style = 'font-size:20px;font-family:Arial;color:#00233C'><b>Conclusion</b></p>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ecd35fef-d012-418f-bb61-9e9619ca50ee",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>In this demo we have seen how we can do analysis and pre-processing of the text data in Vantage using InDb functions and integrating with 3rd party LLM models. </p>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f78894af-5444-4d7b-afc9-93b0c139fd20",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<p style = 'font-size:20px;font-family:Arial;color:#00233C'><b>9. Cleanup</b></p>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6b9fda39-aa2d-4b70-8fa6-027c7aac735d",
   "metadata": {},
   "source": [
    "<p style = 'font-size:18px;font-family:Arial;color:#00233C;color:#00233C'><b>Work Tables</b></p>\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C;'>\n",
    "We need to clean up our work tables to prevent errors next time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "825f607d-db0a-46c4-b463-31f82fbec57f",
   "metadata": {},
   "outputs": [],
   "source": [
    "tables = ['tfidf_comments','tfidf_input_tokenized','title_comments','comment_embeddings']\n",
    "\n",
    "# Loop through the list of tables and execute the drop table command for each table\n",
    "for table in tables:\n",
    "    try:\n",
    "        db_drop_table(table_name = table)\n",
    "    except:\n",
    "        pass"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6cd94a98-d3e7-40d3-84a2-57d3931c1e8d",
   "metadata": {},
   "source": [
    "<p style = 'font-size:18px;font-family:Arial;color:#00233C;color:#00233C'><b>Databases and Tables</b></p>\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>We will use the following code to clean up tables and databases created for this demonstration.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8d1284f1-8b92-4e65-a681-04ee478247ec",
   "metadata": {},
   "outputs": [],
   "source": [
    "%run -i ../run_procedure.py \"call remove_data('DEMO_Retail');\" \n",
    "#Takes 20 seconds"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c4c470f7-5747-4777-b1c3-8ae4a2433218",
   "metadata": {},
   "outputs": [],
   "source": [
    "remove_context()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "767393be-e89c-4d6d-9e8f-fc440fc28901",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0788928e-d332-4e24-984b-e88bb05fb8c1",
   "metadata": {},
   "source": [
    "<b style = 'font-size:20px;font-family:Arial;color:#00233C'>Required Materials</b>\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Let’s look at the elements we have available for reference for this notebook:</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "60a109c3-cf1d-4419-ba7e-34e989a5c7ac",
   "metadata": {},
   "source": [
    "<p style = 'font-size:18px;font-family:Arial;color:#00233C'><b>Filters:</b></p>\n",
    "    <ul style = 'font-size:16px;font-family:Arial;color:#00233C'>\n",
    "    <li><b>Industry:</b> Retail</li>\n",
    "    <li><b>Functionality:</b> Text Analysis</li>\n",
    "    <li><b>Use Case:</b> Natural Language Processing</li>\n",
    "    </ul>\n",
    "    <p style = 'font-size:18px;font-family:Arial;color:#00233C'><b>Related Resources:</b></p>\n",
    "    <ul style = 'font-size:16px;font-family:Arial;color:#00233C'>\n",
    "    <li><a href = 'https://www.teradata.com/Blogs/NPS-is-a-metric-not-the-goal'>·In the fight to improve customer experience, NPS is a metric, not the goal</a></li>\n",
    "    <li><a href = 'https://www.teradata.com/insights/ai-and-machine-learning/using-natural-language-to-query-teradata-vantagecloud-with-llms'>·Using Natural Language to query Teradata Vantage Cloud with LLMs</a></li>\n",
    "    </ul>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7fa577ba-8825-428c-8e90-686e900e9574",
   "metadata": {},
   "source": [
    "<p style = 'font-size:18px;font-family:Arial;color:#00233C'><b>Reference Links:</b></p>\n",
    "<ul style = 'font-size:16px;font-family:Arial;color:#00233C'> \n",
    "       <li>Teradata Vantage™ - Analytics Database Analytic Functions - 17.20: <a href = 'https://docs.teradata.com/r/Enterprise_IntelliFlex_VMware/Teradata-VantageTM-Analytics-Database-Analytic-Functions-17.20/Introduction-to-Analytics-Database-Analytic-Functions '>https://docs.teradata.com/r/Enterprise_IntelliFlex_VMware/Teradata-VantageTM-Analytics-Database-Analytic-Functions-17.20/Introduction-to-Analytics-Database-Analytic-Functions </a></li>    \n",
    "  <li>Teradata® Package for Python User Guide - 17.20: <a href = 'https://docs.teradata.com/r/Enterprise_IntelliFlex_VMware/Teradata-Package-for-Python-User-Guide-17.20/Introduction-to-Teradata-Package-for-Python'>https://docs.teradata.com/r/Enterprise_IntelliFlex_VMware/Teradata-Package-for-Python-User-Guide-17.20/Introduction-to-Teradata-Package-for-Python</a></li>\n",
    "  <li>Teradata® Package for Python Function Reference - 17.20: <a href = 'https://docs.teradata.com/r/Enterprise/Teradata-Package-for-Python-Function-Reference-17.20/Teradata-Package-for-Python-Function-Reference'>https://docs.teradata.com/r/Enterprise/Teradata-Package-for-Python-Function-Reference-17.20/Teradata-Package-for-Python-Function-Reference</a></li>      \n",
    "  <li>Teradata® API Integration Guide for Cloud Machine Learning: <a href = 'https://docs.teradata.com/r/Enterprise_IntelliFlex_VMware/Teradata-VantageTM-API-Integration-Guide-for-Cloud-Machine-Learning/Teradata-Partner-API'>https://docs.teradata.com/r/Enterprise_IntelliFlex_VMware/Teradata-VantageTM-API-Integration-Guide-for-Cloud-Machine-Learning/Teradata-Partner-API</a></li>    \n",
    "</ul>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "91320552-fa4b-4378-8d87-8b8477800766",
   "metadata": {},
   "source": [
    "<footer style=\"padding-bottom:35px; background:#f9f9f9; border-bottom:3px solid #00233C\">\n",
    "    <div style=\"float:left;margin-top:14px\">ClearScape Analytics™</div>\n",
    "    <div style=\"float:right;\">\n",
    "        <div style=\"float:left; margin-top:14px\">\n",
    "            Copyright © Teradata Corporation - 2024. All Rights Reserved\n",
    "        </div>\n",
    "    </div>\n",
    "</footer>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}