07_module.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# The Naive Bayes Algorithm\n",
    "\n",
    "## A Spam Filter\n",
    "\n",
    "Spam is most commonly associated with emails. For instance, unwanted and unsolicited advertising emails are usually classified as spam. Spamming, however, occurs in ways and environments that don't necessarily relate to emails:\n",
    "\n",
    "+ Articles or blog posts can be spammed with comments — the comments are ads or they are repetitive.\n",
    "+ An educational forum may be spammed with posts that are, in fact, ads.\n",
    "+ Mobile phone users may receive unwanted and unsolicited SMS messages, usually about advertising.\n",
    "\n",
    "In our lessons, we're going to build a spam filter specifically directed at preventing mobile phone spam. The filter will be able to analyze new messages and tell whether they are spam or not — this way, we might be able to prevent spam from bothering mobile phone users.\n",
    "\n",
    "To build the spam filter, we're going to use an algorithm called Naive Bayes — as the name suggests, the algorithm is based on Bayes' theorem.\n",
    "\n",
    "This lesson explores the theoretical side of the algorithm and is dedicated to helping you understand how the algorithm works. In the project lesson, which is a guided project, we'll apply the algorithm to a dataset of over 5,000 SMS messages.\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Naive Bayes Overview\n",
    "\n",
    "Imagine we just got a new SMS message:\n",
    "\n",
    "+ \"WINNER! You have 8 hours to claim your money by calling 090061701461. Claim code: KL341.\"\n",
    "\n",
    "This must be spam, but how could we create an algorithm that reaches the same conclusion? One thing we might think of is to create a list of words that occur frequently in spam messages, and then write a bunch of if statements:\n",
    "\n",
    "+ If the word \"money\" is in the message, then classify the message as spam.\n",
    "+ If the words \"secret\" and \"money\" are both in the message, then classify the message as spam; etc.\n",
    "\n",
    "However, as messages become numerous and more complex, coming up with the right if statements will slowly become very difficult.\n",
    "\n",
    "Another solution would be to classify a couple of messages ourselves and make the computer learn from our classification. And this is exactly what the Naive Bayes algorithm is about: It makes the computer learn from the classification a humans does, and then the computer uses that knowledge to classify new messages.\n",
    "\n",
    "The computer uses the specifications of the Naive Bayes algorithm to learn how we classify messages (what counts as spam and non-spam for us), and then it uses that human knowledge to estimate probabilities for new messages. Following the specifications of the algorithm, the computer tries to answer two conditional probability questions:\n",
    "\n",
    "\\begin{equation}\n",
    "P(Spam | New\\ message) = ?\n",
    "\\end{equation}\n",
    "\n",
    "\\begin{equation}\n",
    "P(Spam^C |New\\ message) = ?\n",
    "\\end{equation}"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In plain English, these two questions are:\n",
    "\n",
    "+ What's the probability that this new message is spam, given its content (its words, punctuation, letter case, etc.)?\n",
    "+ What's the probability that this new message is non-spam, given its content?\n",
    "\n",
    "Once it has an answer to these two questions, the computer classifies the message as spam or non-spam based on the probability values. If the probability for spam is greater, then the message is classified as spam. Otherwise, it goes into the non-spam category."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using Bayes' Theorem\n",
    "\n",
    "On the previous lesson, we saw an overview of how the computer may classify new messages using the Naive Bayes algorithm:\n",
    "\n",
    "+ The computer learns how humans classify messages.\n",
    "+ Then it uses that human knowledge to estimate probabilities for new messages — probabilities for spam and non-spam.\n",
    "+ Finally, the computer classifies a new message based on the probability values it calculated in step 2 — if the probability for spam is greater, then it classifies the message as spam. Otherwise, it classifies it as non-spam (if the two probability values are equal, then we may want a human to classify the message — we'll come back to this issue in the guided project).\n",
    "\n",
    "We saw on the previous lesson that when a new message comes in, the algorithm requires the computer to calculate the following probabilities:\n",
    "\n",
    "\\begin{equation}\n",
    "P(Spam | New\\ message) =\\ ?\n",
    "\\end{equation}\n",
    "\n",
    "\\begin{equation}\n",
    "P(Spam^C |New\\ message) =\\ ?\n",
    "\\end{equation}\n",
    "\n",
    "Let's take the first equation and expand it using Bayes' theorem:\n",
    "\n",
    "\\begin{equation}\n",
    "P(Spam | New\\ message) = \\frac{P(Spam) \\cdot P(New\\ Message | Spam)}{P(New\\ message)}\n",
    "\\end{equation}\n",
    "\n",
    "Now let's do the same for the second equation:\n",
    "\n",
    "\\begin{equation}\n",
    "P(Spam^C | New\\ message) = \\frac{P(Spam^C) \\cdot P(New\\ Message | Spam^C)}{P(New\\ message)}\n",
    "\\end{equation}\n",
    "\n",
    "For the sake of example, let's assume the following probabilities are already known:"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\begin{aligned}\n",
    "&P(Spam) = 0.5 \\\\\n",
    "&P(Spam^C) = 0.5 \\\\\n",
    "&P(New\\ message) = 0.4167 \\\\\n",
    "&P(New\\ Message | Spam) = 0.5 \\\\\n",
    "&P(New\\ Message | Spam^C) = 0.3334\n",
    "\\end{aligned}"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If the computer knows these values, then it can calculate the probabilities it needs to classify a new message:\n",
    "\n",
    "\\begin{equation}\n",
    "P(Spam | New\\ message) = \\frac{0.5 \\cdot 0.5}{0.4167} = 0.6\n",
    "\\end{equation}\n",
    "\n",
    "\\begin{equation}\n",
    "P(Spam^C | New\\ message) = \\frac{0.5 \\cdot 0.3334}{0.4167} = 0.4\n",
    "\\end{equation}"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "since\n",
    "\n",
    "\\begin{equation}\n",
    "P(Spam | New\\ message) > P(Spam^C | New\\ message)\n",
    "\\end{equation}"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " the computer will classify the new message as spam."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task\n",
    "\n",
    "A new mobile message has been received: \"URGENT!! You have one day left to claim your $873 prize.\" The following probabilities are known:\n",
    "\n",
    "\\begin{aligned}\n",
    "&P(Spam) = 0.5 \\\\\n",
    "&P(Spam^C) = 0.5 \\\\\n",
    "&P(New\\ message) = 0.5417 \\\\\n",
    "&P(New\\ Message | Spam) = 0.75 \\\\\n",
    "&P(New\\ Message | Spam^C) = 0.3334\n",
    "\\end{aligned}\n",
    "\n",
    "Classify this new message as spam or non-spam:\n",
    "\n",
    "1. Calculate P(Spam|New Message). Assign your answer to p_spam_given_new_message.\n",
    "2. Calculate P(Spam<sup>c</sup>|New Message). Assign your answer to p_non_spam_given_new_message.\n",
    "3. Classify the message by comparing the probability values. If the message is spam, then assign the string 'spam' to the variable classification. Otherwise, assign the string 'non-spam'.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True\n"
     ]
    }
   ],
   "source": [
    "p_spam = 0.5\n",
    "p_non_spam = 0.5\n",
    "p_new_message = 0.5417\n",
    "p_new_message_given_spam = 0.75\n",
    "p_new_message_given_non_spam = 0.3334\n",
    "\n",
    "p_spam_given_new_message = p_spam * p_new_message_given_spam/p_new_message\n",
    "p_non_spam_given_new_message = p_non_spam * p_new_message_given_non_spam/p_new_message\n",
    "\n",
    "print(p_spam_given_new_message > p_non_spam_given_new_message)\n",
    "\n",
    "classification = 'spam'"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Ignoring the Division\n",
    "\n",
    "On the last lesson, we saw the computer can use these two equations to calculate the probabilities it needs to classify new messages:\n",
    "\n",
    "\\begin{equation}\n",
    "P(Spam | New\\ message) = \\frac{P(Spam) \\cdot P(New\\ Message | Spam)}{P(New\\ message)}\n",
    "\\end{equation}\n",
    "\n",
    "\\begin{equation}\n",
    "P(Spam^C | New\\ message) = \\frac{P(Spam^C) \\cdot P(New\\ Message | Spam^C)}{P(New\\ message)}\n",
    "\\end{equation}"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This was just the first step, the actual Naive Bayes equation are different, we will develop them step by step in this lesson.  Let's start by pointing out that both equations above have the same denominator: P(New message).\n",
    "\n",
    "When a new message comes in, P(New message) has the same value for both equations. Since we only need to compare the results of the two equations to classify a new message, we can ignore the division:\n",
    "\n",
    "\\begin{equation}\n",
    "\\frac{P(Spam) \\cdot P(New\\ Message | Spam)}{P(New\\ message)}\\ \\ \\ \\  \\xrightarrow[]{becomes}\\ \\ \\ \\  P(Spam) \\cdot P(New\\ Message | Spam) \\\\\n",
    "\\end{equation}\n",
    "\n",
    "\\begin{equation}\n",
    "\\frac{P(Spam^C) \\cdot P(New\\ Message | Spam^C)}{P(New\\ message)}\\ \\ \\  \\xrightarrow[]{becomes}\\ \\ \\ \\  P(Spam^C) \\cdot P(New\\ Message | Spam^C)\n",
    "\\end{equation}"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This means our two equations reduce to:\n",
    "\n",
    "\\begin{equation}\n",
    "P(Spam | New\\ message) = P(Spam) \\cdot P(New\\ Message | Spam) \\\\\n",
    "\\end{equation}\n",
    "\n",
    "\\begin{equation}\n",
    "P(Spam^C | New\\ message) = P(Spam^C) \\cdot P(New\\ Message | Spam^C)\n",
    "\\end{equation}"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ignoring the division doesn't affect the algorithm's ability to classify new messages. For instance, let's repeat the classification we did on the previous screen using the new equations above. Recall that we assumed we already know these values:"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\begin{aligned}\n",
    "&P(Spam) = 0.5 \\\\\n",
    "&P(Spam^C) = 0.5 \\\\\n",
    "&P(New\\ message) = 0.4167 \\\\\n",
    "&P(New\\ Message | Spam) = 0.5 \\\\\n",
    "&P(New\\ Message | Spam^C) = 0.3334\n",
    "\\end{aligned}"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Previously, the algorithm classified the new message as spam. Using the new equations, we see the conclusion is identical — the new message is spam because \n",
    "\n",
    "\\begin{aligned}\n",
    "P(Spam | New\\ message) > P(Spam^C | New\\ message)\n",
    "\\end{aligned}"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\begin{aligned}\n",
    "P(Spam | New\\ message) &= P(Spam) \\cdot P(New\\ Message | Spam) \\\\\n",
    "&= 0.5 \\cdot 0.5 = 0.25\n",
    "\\end{aligned}\n",
    "\n",
    "\\begin{aligned}\n",
    "P(Spam^C | New\\ message) &= P(Spam^C) \\cdot P(New\\ Message | Spam^C) \\\\\n",
    "&= 0.5 \\cdot 0.3334 = 0.1667\n",
    "\\end{aligned}"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The classification works fine, but ignoring the division changes the probability values, and some probability rules also begin to break. For instance, let's take this conditional probability rule that we've learned about in a previous lesson:\n",
    "\n",
    "\\begin{equation}\n",
    "P(A|B) + P(A^C|B) = 1\n",
    "\\end{equation}\n",
    "\n",
    "On the previous lesson, we saw `P(Spam|New message)=0.6` and P(Spam<sup>c</sup>|New message)=0.4, and the rule holds with these values:\n",
    "\n",
    "\\begin{equation}\n",
    "P(Spam | New\\ message) + P(Spam^C | New\\ message) = 0.6 + 0.4 = 1\n",
    "\\end{equation}"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the values we got from the new equations, however, the law breaks:\n",
    "\n",
    "\\begin{equation}\n",
    "P(Spam | New\\ message) + P(Spam^C | New\\ message) = 0.25 + 0.1667 = 0.4167 \\not = 1\n",
    "\\end{equation}\n",
    "\n",
    "Even though probability rules break, the Naive Bayes algorithm still requires us to ignore the division by P(New message). This might not make a lot of sense, but there's actually a very good reason we do that.\n",
    "\n",
    "The main goal of the algorithm is to classify new messages, not to calculate probabilities — calculating probabilities is just a means to an end. Ignoring the division by P(New message) means less calculations, which can make a lot of difference when we use the algorithm to classify 500,000 new messages.\n",
    "\n",
    "It's true the probability values are not accurate anymore. However, this is not important with respect to the the goal of the algorithm — correctly classifying new messages (not to accurately estimate probabilities).\n",
    "\n",
    "The classification itself remains completely unaffected because we ignore division for both equations (not just for one). The probability values change, but they change directly proportional with one another, so the result of the comparison doesn't change.\n",
    "\n",
    "For instance, $\\frac{8}{4}$ > $\\frac{4}{4}$.  If we ignore the division, the values change directly proportional with respect to one another such that the result of the comparison stays the same: 8>4. \n",
    "\n",
    "The symbol for directly proportional is ∝, and it's more accurate to replace the equality sign with ∝ in our two equations:\n",
    "\n",
    "\\begin{equation}\n",
    "P(Spam | New\\ message) \\propto P(Spam) \\cdot P(New\\ Message | Spam) \\\\\n",
    "\\end{equation}\n",
    "\n",
    "\\begin{equation}\n",
    "P(Spam^C | New\\ message) \\propto P(Spam^C) \\cdot P(New\\ Message | Spam^C)\n",
    "\\end{equation}"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task\n",
    "\n",
    "A new mobile message has been received: \"URGENT!! You have one day left to claim your $873 prize.\" The following probabilities are known:\n",
    "\n",
    "\\begin{aligned}\n",
    "&P(Spam) = 0.5 \\\\\n",
    "&P(Spam^C) = 0.5 \\\\\n",
    "&P(New\\ Message | Spam) = 0.75 \\\\\n",
    "&P(New\\ Message | Spam^C) = 0.3334\n",
    "\\end{aligned}\n",
    "\n",
    "Use the new equations we learned on this screen, and classify the new message as spam or non-spam:\n",
    "\n",
    "1. Calculate P(Spam|New Message). Assign your answer to p_spam_given_new_message.\n",
    "2. Calculate P(Spam<sup>c</sup>|New Message). Assign your answer to p_non_spam_given_new_message.\n",
    "3. Classify the message by comparing the probability values — if the message is spam, then assign the string 'spam' to the variable classification. Otherwise, assign the string 'non-spam'.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "p_spam = 0.5\n",
    "p_non_spam = 0.5\n",
    "p_new_message_given_spam = 0.75\n",
    "p_new_message_given_non_spam = 0.3334\\\n",
    "\n",
    "p_spam_given_new_message = p_spam * p_new_message_given_spam\n",
    "\n",
    "p_non_spam_given_new_message = p_non_spam * p_new_message_given_non_spam\n",
    "\n",
    "classification = 'spam'"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## One Word New Message\n",
    "\n",
    "We'll now look at how the algorithm can use messages that are already classified by humans to calculate the values it needs for:\n",
    "\n",
    "+ P(Spam) and P(SpamC)\n",
    "+ P(New message|Spam) and P(New message|SpamC).\n",
    "\n",
    "We'll start with some examples that may look a bit too simplistic and unrealistic, but they will make it easier to understand the mathematics behind the algorithm.\n",
    "\n",
    "Let's say we have three messages that are already classified\n",
    "\n",
    "![spam](img/cpm4_viz1.1.png)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's say the one-word message \"secret\" comes in and we want to use the Naive Bayes algorithm to classify it — to tell whether it's spam or non-spam.\n",
    "\n",
    "![secret](img/cpm4_viz2.2.png)\n",
    "\n",
    "As we learned, we first need to answer these two probability questions (note that we changed New Message to \"secret\" inside the notation below) and then compare the values (recall that the ∝ symbol replaces the equal sign):\n",
    "\n",
    "\\begin{equation}\n",
    "P(Spam | \\text{\"secret\"}) \\propto P(Spam) \\cdot P(\\text{\"secret\"} | Spam) \\\\\n",
    "\\end{equation}\n",
    "\n",
    "\\begin{equation}\n",
    "P(Spam^C | \\text{\"secret\"}) \\propto P(Spam^C) \\cdot P(\\text{\"secret\"} | Spam^C)\n",
    "\\end{equation}"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's begin with the first equation, for which we need to find the values of P(Spam) and P(\"secret\"|Spam). To find P(Spam), we use the messages that are already classified and divide the number of spam messages by the total number of messages:\n",
    "\n",
    "\\begin{equation}\n",
    "P(Spam) = \\frac{\\text{number of spam messages}}{\\text{total number of messages}} = \\frac{2}{3}\n",
    "\\end{equation}\n",
    "\n",
    "To calculate P(\"secret\"|Spam), we only look at the spam messages and divide the number of times the word \"secret\" occurred in all the spam messages by the total number of words.\n",
    "\n",
    "\\begin{equation}\n",
    "P(\\text{\"secret\"}| Spam) = \\frac{\\text{number of times the word \"secret\" occurs}}{\\text{total number of words in all spam messages}}\n",
    "\\end{equation}"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that \"secret\" occurs four times in the spam messages:\n",
    "\n",
    "![secret](img/cpm4_viz3.1.png)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have two spam messages and there's a total of seven words in all of them, so P(\"secret\"|Spam) is:\n",
    "\n",
    "\\begin{equation}\n",
    "P(\\text{\"secret\"}| Spam) = \\frac{\\text{number of times the word \"secret\" occurs}}{\\text{total number of words in all spam messages}} = \\frac{4}{7}\n",
    "\\end{equation}\n",
    "\n",
    "Now that we know the values for P(Spam) and P(\"secret\"|Spam), we have all we need to calculate P(Spam|\"secret\"):\n",
    "\n",
    "\\begin{aligned}\n",
    "P(Spam | \\text{\"secret\"}) &\\propto P(Spam) \\cdot P(\\text{\"secret\"} | Spam) \\\\\n",
    "&= \\frac{2}{3} \\cdot \\frac{4}{7} = \\frac{8}{21}\n",
    "\\end{aligned}\n",
    "\n",
    "For the task below, we'll take the same steps as above to calculate P(SpamC|\"secret\"). Then, we can compare the values of P(SpamC|\"secret\") and P(Spam|\"secret\") to classify the message \"secret\" as spam or non-spam."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task\n",
    "\n",
    "Using the table below (there are the same messages as above), classify the message \"secret\" as spam or non-spam.\n",
    "\n",
    "![secret](img/cpm4_viz2.1.png)\n",
    "\n",
    "\n",
    "1. Calculate P(Spam<sup>c</sup>) and assign the answer to p_non_spam.\n",
    "2. Calculate P(\"secret\"|Spam<sup>c</sup>) and assign the answer to p_secret_given_non_spam.\n",
    "3. Calculate P(Spam<sup>c</sup>|\"secret\") and assign the answer to p_non_spam_given_secret.\n",
    "    Compare P(Spam<sup>c</sup>|\"secret\") with P(Spam|\"secret\") and classify the message \"secret\" — if the message is spam, then assign the string 'spam' to the variable classification, otherwise assign the string 'non-spam'.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "p_spam_given_secret = 8/21\n",
    "p_non_spam = 1/3\n",
    "p_secret_given_non_spam = 1/4\n",
    "p_non_spam_given_secret = p_non_spam * p_secret_given_non_spam\n",
    "classification = 'spam'"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Multiple Words\n",
    "\n",
    "On the previous screen, we used our algorithm to classify the message \"secret\", and we concluded it's spam. The message \"secret\" has only one word, but what about the situation where we have to classify messages that have more words?\n",
    "\n",
    "Let's say we want to classify the message \"secret place secret secret\" based on four messages that are already classified (the four messages below are different than what what we saw on the previous screen):\n",
    "\n",
    "![multiple](img/cpm4_viz4.222.png)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To calculate the probabilities we need, we'll treat each word in our new message separately. This means that the word \"secret\" at the beginning is different and separate from the word \"secret\" at the end. There are four words in the message \"secret place secret secret\", and we're going to abbreviate them \"w1\", \"w2\", \"w3\" and \"w4\" (the \"w\" comes from \"word\").\n",
    "\n",
    "![words](img/cpm4_viz5.png)\n",
    "\n",
    "Since we treat each word separately, these are the two equations we can use to calculate the probabilities:\n",
    "\n",
    "\\begin{equation}\n",
    "P(Spam | w_1,w_2,w_3,w_4) \\propto P(Spam) \\cdot P(w_1|Spam) \\cdot P(w_2|Spam) \\cdot P(w_3|Spam) \\cdot P(w_4|Spam) \\\\\n",
    "P(Spam^C | w_1,w_2,w_3,w_4) \\propto P(Spam^C) \\cdot P(w_1|Spam^C) \\cdot P(w_2|Spam^C) \\cdot P(w_3|Spam^C) \\cdot P(w_4|Spam^C) \\\\\n",
    "\\end{equation}\n",
    "\n",
    "Let's begin with calculating P(Spam|w1, w2, w3, w4). To calculate the probabilities we need, we'll look at the four messages that are already classified. We have four messages and two of them are spam, so:\n",
    "\n",
    "\\begin{equation}\n",
    "P(Spam) = \\frac{2}{4} = \\frac{1}{2}\n",
    "\\end{equation}\n",
    "\n",
    "The first word, w1, is \"secret\", and we see that \"secret\" occurs four times in all spam messages. There's a total of seven words in all the spam messages, so:\n",
    "\n",
    "\\begin{equation}\n",
    "P(w_1|Spam) = \\frac{4}{7} \n",
    "\\end{equation}\n",
    "\n",
    "Applying a similar reasoning, we have:\n",
    "\n",
    "\\begin{equation}\n",
    "P(w_2|Spam) = \\frac{1}{7} \\\\\n",
    "\\end{equation}\n",
    "\n",
    "\\begin{equation}\n",
    "P(w_3|Spam) = \\frac{4}{7} \\\\\n",
    "\\end{equation}\n",
    "\n",
    "\\begin{equation}\n",
    "P(w_4|Spam) = \\frac{4}{7}\n",
    "\\end{equation}\n",
    "\n",
    "We now have all the probabilities we need to calculate P(Spam|w1, w2, w3, w4):\n",
    "\n",
    "\\begin{aligned}\n",
    "P(Spam | w_1,w_2,w_3,w_4) &\\propto P(Spam) \\cdot P(w_1|Spam) \\cdot P(w_2|Spam) \\cdot P(w_3|Spam) \\cdot P(w_4|Spam) \\\\\n",
    "&= \\frac{1}{2} \\cdot \\frac{4}{7} \\cdot \\frac{1}{7} \\cdot \\frac{4}{7} \\cdot \\frac{4}{7} = \\frac{64}{4802} = 0.01333\n",
    "\\end{aligned}\n",
    "\n",
    "Let's now take similar steps to calculate P(Spam<sup>c</sup>|w1, w2, w3, w4), and then classify the message \"secret place secret secret\" as spam or non-spam.\n",
    "\n",
    "\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task\n",
    "\n",
    "Using the table below (the same as above), classify the message \"secret place secret secret\" as spam or non-spam.\n",
    "\n",
    "![many_words](img/cpm4_viz4.222.png)\n",
    "\n",
    "\n",
    "1. Calculate P(Spam<sup>c</sup>|w1, w2, w3, w4). Assign the answer to p_non_spam_given_w1_w2_w3_w4. Check the hint if you get stuck.\n",
    "2. Compare P(Spam<sup>c</sup>|w1, w2, w3, w4) with P(Spam|w1, w2, w3, w4) and classify the message \"secret place secret secret\" — if the message is spam, then assign the string 'spam' to the variable classification. Otherwise, assign the string 'non-spam'.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "p_spam_given_w1_w2_w3_w4 = 64/4802\n",
    "p_non_spam = 2/4\n",
    "p_w1_given_non_spam = 2/9\n",
    "p_w2_given_non_spam = 1/9\n",
    "p_w3_given_non_spam = 2/9\n",
    "p_w4_given_non_spam = 2/9\n",
    "\n",
    "p_non_spam_given_w1_w2_w3_w4 = (p_non_spam *\n",
    "                                p_w1_given_non_spam * p_w2_given_non_spam *\n",
    "                                p_w3_given_non_spam * p_w4_given_non_spam\n",
    "                               )\n",
    "\n",
    "classification = 'spam'"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conditional Independence\n",
    "\n",
    "On the previous lesson and task, we introduced these two equations without much explanation:\n",
    "\n",
    "\\begin{equation}\n",
    "P(Spam | w_1,w_2,w_3,w_4) \\propto P(Spam) \\cdot P(w_1|Spam) \\cdot P(w_2|Spam) \\cdot P(w_3|Spam) \\cdot P(w_4|Spam) \\\\\n",
    "P(Spam^C | w_1,w_2,w_3,w_4) \\propto P(Spam^C) \\cdot P(w_1|Spam^C) \\cdot P(w_2|Spam^C) \\cdot P(w_3|Spam^C) \\cdot P(w_4|Spam^C) \\\\\n",
    "\\end{equation}\n",
    "\n",
    "To explain the mathematics behind these equations, let's start by looking at P(Spam|w1, w2, w3, w4). Using the conditional probability formula, we can expand P(Spam|w1, w2, w3, w4) like this (below, make sure you notice the \\cap symbol in the numerator):\n",
    "\n",
    "\\begin{equation}\n",
    "P(Spam | w_1,w_2,w_3,w_4) = \\frac{P(Spam \\cap (w_1, w_2, w_3, w_4))}{P(w_1, w_2, w_3, w_4)}\n",
    "\\end{equation}\n",
    "\n",
    "Recall that we learned in a previous screen that we can ignore the division, which means we can drop P(w1, w2, w3, w4) to avoid redundant calculations (when we ignore the division, we also replace the equals sign with \\propto, which means directly proportional):\n",
    "\n",
    "\\begin{equation}\n",
    "P(Spam | w_1,w_2,w_3,w_4) \\propto P(Spam \\cap (w_1, w_2, w_3, w_4))\n",
    "\\end{equation}\n",
    "\n",
    "Note that (w1, w2, w3, w4) can be modeled as an intersection of four events:\n",
    "\n",
    "\\begin{equation}\n",
    "w_1,w_2,w_3,w_4 = w_1 \\cap w_2 \\cap w_3 \\cap w_4\n",
    "\\end{equation}\n",
    "\n",
    "For instance, we could think of a message like \"thanks for your help\" as the intersection of four words inside a single message: \"thanks\", \"for, \"your\", and \"help\". In probability jargon, finding the value of P(w_1 \\cap w_2 \\cap w_3 \\cap w_4) means finding the probability that the four words w1, w2, w3, w4 occur together in a single message — this is similar to P(A \\cap B \\cap C \\cap D), which is the probability that events A, B, C, and D occur together.\n",
    "\n",
    "With this in mind, our equation above transforms to:\n",
    "\n",
    "\\begin{equation}\n",
    "P(Spam | w_1,w_2,w_3,w_4) \\propto P(Spam \\cap \\underbrace{(w_1 \\cap w_2 \\cap w_3 \\cap w_4)}_{\\displaystyle (w_1,w_2,w_3,w_4)})\n",
    "\\end{equation}\n",
    "\n",
    "From set theory, we know that\n",
    "\n",
    " \\begin{equation}\n",
    "  A \\cap (B \\cap C) = A \\cap B \\cap C = C \\cap B \\cap A \n",
    " \\end{equation},\n",
    " \n",
    "  which means we can transform\n",
    " \\begin{equation}\n",
    "   P(Spam \\cap (w_1 \\cap w_2 \\cap w_3 \\cap w_4)) \n",
    "\\end{equation}\n",
    "   in our equation above to make it suitable for further expansion:\n",
    "\n",
    "\\begin{aligned}\n",
    "P(Spam \\cap (w_1 \\cap w_2 \\cap w_3 \\cap w_4)) &= P(Spam \\cap w_1 \\cap w_2 \\cap w_3 \\cap w_4) \\\\\n",
    "&= P(w_1 \\cap w_2 \\cap w_3 \\cap w_4 \\cap Spam)\n",
    "\\end{aligned}\n",
    "\n",
    "Now let's use the multiplication rule to expand P(w_1 \\cap w_2 \\cap w_3 \\cap w_4 \\cap Spam):\n",
    "\n",
    "\\begin{equation}\n",
    "P(w_1 \\cap w_2 \\cap w_3 \\cap w_4 \\cap Spam) = P(w_1 | w_2 \\cap w_3 \\cap w_4 \\cap Spam) \\cdot P(w_2 \\cap w_3 \\cap w_4 \\cap Spam)\n",
    "\\end{equation}\n",
    "\n",
    "We can use the multiplication rule again to expand P(w2 ∩ w3 ∩ w4 ∩ Spam):\n",
    "\n",
    "\\begin{equation}\n",
    "P(w_1 \\cap w_2 \\cap w_3 \\cap w_4 \\cap Spam) = P(w_1 | w_2 \\cap w_3 \\cap w_4 \\cap Spam) \\cdot \\underbrace{P(w_2 | w_3 \\cap w_4 \\cap Spam) \\cdot P(w_3 \\cap w_4 \\cap Spam)}_{\\displaystyle P(w_2 \\cap w_3 \\cap w_4 \\cap Spam)}\n",
    "\\end{equation}\n",
    "\n",
    "We can use the multiplication rule successively, until there's nothing more left to expand:\n",
    "\n",
    "\\begin{aligned}\n",
    "P(w_1 \\cap w_2 \\cap w_3 \\cap w_4 \\cap Spam) &= P(w_1 | w_2 \\cap w_3 \\cap w_4 \\cap Spam) \\cdot P(w_2 \\cap w_3 \\cap w_4 \\cap Spam) \\\\\n",
    "&= P(w_1 | w_2 \\cap w_3 \\cap w_4 \\cap Spam) \\cdot P(w_2 | w_3 \\cap w_4 \\cap Spam) \\cdot P(w_3 \\cap w_4 \\cap Spam) \\\\\n",
    "&= P(w_1 | w_2 \\cap w_3 \\cap w_4 \\cap Spam) \\cdot P(w_2 | w_3 \\cap w_4 \\cap Spam) \\cdot P(w_3 | w_4 \\cap Spam) \\cdot P(w_4 \\cap Spam) \\\\\n",
    "&= P(w_1 | w_2 \\cap w_3 \\cap w_4 \\cap Spam) \\cdot P(w_2 | w_3 \\cap w_4 \\cap Spam) \\cdot P(w_3 | w_4 \\cap Spam) \\cdot P(w_4|Spam) \\cdot P(Spam) \\\\\n",
    "\\end{aligned}\n",
    "\n",
    "In theory, the last equation you see above is what we'd have to use if we wanted to calculate P(Spam|w1, w2, w3, w4). However, the equation is pretty long for just four words. Also, imagine how would the equation look for a 50-word message — just think of how many calculations we'd have to perform!!\n",
    "\n",
    "To make the calculations tractable for messages of all kinds of lengths, we can assume conditional independence between w1, w2, w3, and w4. This implies that:\n",
    "\n",
    "\\begin{aligned}\n",
    "&P(w_1 | w_2 \\cap w_3 \\cap w_4 \\cap Spam) = P(w_1|Spam) \\\\\n",
    "&P(w_2 | w_3 \\cap w_4 \\cap Spam) = P(w_2|Spam) \\\\ \n",
    "&P(w_3 | w_4 \\cap Spam) = P(w_3|Spam) \\\\\n",
    "&P(w_4|Spam) = P(w_4|Spam) \\\\\n",
    "\\end{aligned}\n",
    "\n",
    "Under the assumption of independence, our lengthy equation above reduces to:\n",
    "\n",
    "\\begin{equation}\n",
    "P(w_1 \\cap w_2 \\cap w_3 \\cap w_4 \\cap Spam) = P(w_1|Spam) \\cdot P(w_2|Spam) \\cdot P(w_3|Spam) \\cdot P(w_4|Spam) \\cdot P(Spam)\n",
    "\\end{equation}\n",
    "\n",
    "The assumption of conditional independence is unrealistic in practice because words are often in a relationship of dependence. For instance, if you see the word \"WINNER\" in a message, the probability of seeing the word \"money\" is very likely to increase, so \"WINNER\" and \"money\" are most likely dependent. The assumption of conditional independence between words is thus naive since it rarely holds in practice, and this is why the algorithm is called __Naive Bayes__ (also called __simple Bayes or independence Bayes__).\n",
    "\n",
    "Despite this simplifying assumption, the algorithm works quite well in many real-word situations, and we'll see that ourselves in the guided project.\n",
    "\n",
    "That being said, on the previous screen we assumed conditional independence when we introduced these two equations:\n",
    "\n",
    "\\begin{equation}\n",
    "P(Spam | w_1,w_2,w_3,w_4) \\propto P(Spam) \\cdot P(w_1|Spam) \\cdot P(w_2|Spam) \\cdot P(w_3|Spam) \\cdot P(w_4|Spam) \\\\\n",
    "\\end{equation}\n",
    "\n",
    "\\begin{equation}\n",
    "P(Spam^C | w_1,w_2,w_3,w_4) \\propto P(Spam^C) \\cdot P(w_1|Spam^C) \\cdot P(w_2|Spam^C) \\cdot P(w_3|Spam^C) \\cdot P(w_4|Spam^C) \\\\\n",
    "\\end{equation}"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A General Equation\n",
    "\n",
    "On the previous screen, we learned about the conditional independence assumption, which is central to the Naive Bayes algorithm. As a result of the assumption, we saw we can use these simplified equations:\n",
    "\n",
    "\\begin{equation}\n",
    "P(Spam | w_1,w_2,w_3,w_4) \\propto P(Spam) \\cdot P(w_1|Spam) \\cdot P(w_2|Spam) \\cdot P(w_3|Spam) \\cdot P(w_4|Spam) \\\\\n",
    "P(Spam^C | w_1,w_2,w_3,w_4) \\propto P(Spam^C) \\cdot P(w_1|Spam^C) \\cdot P(w_2|Spam^C) \\cdot P(w_3|Spam^C) \\cdot P(w_4|Spam^C) \\\\\n",
    "\\end{equation}\n",
    "\n",
    "The equations above work for messages that have four words, but we need a more general form to use with messages of various word lengths.\n",
    "\n",
    "A new message has n words, where n can be any positive integer (1, 2, 3, ..., 50, 51, 53, ...). If we wanted to find P(Spam|w1, w2, ..., wn), then this is an equation we could use:\n",
    "\n",
    "\\begin{equation}\n",
    "P(Spam | w_1,w_2, \\ldots, w_n) \\propto P(Spam) \\cdot P(w_1|Spam) \\cdot P(w_2|Spam) \\cdot \\ldots \\cdot P(w_n|Spam)\n",
    "\\end{equation}\n",
    "\n",
    "Notice that there's a certain pattern in the equation above — after P(Spam), the only thing that changes is the word number.\n",
    "\n",
    "\\begin{equation}\n",
    "P(Spam | w_1,w_2, \\ldots, w_n) \\propto P(Spam) \\cdot P(\\overbrace{w_1}^{1}|Spam) \\cdot P(\\overbrace{w_2}^{2}|Spam) \\cdot \\ldots \\cdot P(\\overbrace{w_n}^{n}|Spam)\n",
    "\\end{equation}\n",
    "\n",
    "Whenever we have a product that follows a pattern like that, it's common to use the \\prod symbol (this is the uppercase Greek letter \"pi\"). So the equation above simplifies to:\n",
    "\n",
    "\\begin{equation}\n",
    "P(Spam | w_1,w_2, \\ldots, w_n) \\propto P(Spam) \\cdot \\prod_{i=1}^{n}P(w_i|Spam)\n",
    "\\end{equation}\n",
    "\n",
    "The equation above is the same as:\n",
    "\n",
    "\\begin{equation}\n",
    "P(Spam | w_1,w_2, \\ldots, w_n) \\propto P(Spam) \\cdot \\overbrace{P(w_1|Spam) \\cdot P(w_2|Spam) \\cdot \\ldots \\cdot P(w_n|Spam)}^{\\displaystyle \\prod_{i=1}^{n}P(w_i|Spam)}\n",
    "\\end{equation}\n",
    "\n",
    "Applying the same reasoning to P(SpamC|w1, w2, ..., wn), we have:\n",
    "\n",
    "\\begin{equation}\n",
    "P(Spam^C | w_1,w_2, \\ldots, w_n) \\propto P(Spam^C) \\cdot \\prod_{i=1}^{n}P(w_i|Spam^C)\n",
    "\\end{equation}\n",
    "\n",
    "Now that we have these general equations, we're going to discuss a few edge cases on the next few screens. After this, we'll be ready to start working on the guided project, where we'll work with over 5,000 real messages."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Edge Words\n",
    "\n",
    "\n",
    "On a previous screen, we looked at a few messages that were already classified\n",
    "\n",
    "![towards edge](img/cpm4_viz6.png)\n",
    "\n",
    "Above, we have four messages and nine unique words: \"secret\", \"party\", \"at\", \"my\", \"place\", \"money\", \"you\", \"know\", \"the\". We call the set of unique words a vocabulary.\n",
    "\n",
    "Now, what if we receive a new message that contains words which are not part of the vocabulary? How do we calculate probabilities for these kind of words?\n",
    "\n",
    "For instance, say we received the message \"secret code to unlock the money\".\n",
    "\n",
    "![non_voc](img/cpm4_viz7.png)\n",
    "\n",
    "Notice that for this new message:\n",
    "\n",
    "+ The words \"code\", \"to\", and \"unlock\" are not part of the vocabulary.\n",
    "+ The word \"secret\" is part of both spam and non-spam messages.\n",
    "+ The word \"money\" is only part of the spam messages and is missing from the non-spam messages.\n",
    "+ The word \"the\" is missing from the spam messages and is only part of the non-spam messages.\n",
    "\n",
    "Whenever we have to deal with words that are not part of the vocabulary, one solution is to ignore them when we're calculating probabilities. If we wanted to calculate P(Spam|\"secret code to unlock the money\"), we could skip calculating P(\"code\"|Spam), P(\"to\"|Spam), and P(\"unlock\"|Spam) because \"code\", \"to\", and \"unlock\" are not part of the vocabulary:\n",
    "\n",
    "\\begin{equation}\n",
    "P(Spam|\\text{\"secret code to unlock the money\"}) \\propto P(Spam) \\cdot {P(\\text{\"secret\"}|Spam) \\cdot P(\\text{\"the\"}|Spam) \\cdot P(\\text{\"money\"}|Spam)}\n",
    "\\end{equation}\n",
    "\n",
    "We can also apply the same reasoning for calculating P(SpamC|\"secret code to unlock the money\"):\n",
    "\n",
    "\\begin{equation}\n",
    "P(Spam^C|\\text{\"secret code to unlock the money\"}) \\propto P(Spam^C) \\cdot P(\\text{\"secret\"}|Spam^C) \\cdot P(\\text{\"the\"}|Spam^C) \\cdot P(\\text{\"money\"}|Spam^C)\n",
    "\\end{equation}\n",
    "\n",
    "Let's now calculate P(Spam|\"secret code to unlock the money\") and P(SpamC|\"secret code to unlock the money\"), and see what we get.\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task\n",
    "\n",
    "P(Spam|\"secret code to unlock the money\") is already calculated for you. Use the table below (the same as above) to calculate P(SpamC|\"secret code to unlock the money\").\n",
    "\n",
    "![skip](img/cpm4_viz7.2.png)\n",
    "\n",
    "\n",
    "+ Calculate P(SpamC|\"secret code to unlock the money\"). Assign your answer to p_non_spam_given_message.\n",
    "+ Print p_spam_given_message and p_non_spam_given_message. Why do you think we got these values? We'll discuss more about this in the next screen.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.0\n",
      "0.0\n"
     ]
    }
   ],
   "source": [
    "p_spam = 2/4\n",
    "p_secret_given_spam = 4/7\n",
    "p_the_given_spam = 0/7\n",
    "p_money_given_spam = 2/7\n",
    "p_spam_given_message = (p_spam * p_secret_given_spam *\n",
    "                        p_the_given_spam * p_money_given_spam)\n",
    "p_non_spam = 2/4\n",
    "p_secret_given_non_spam = 2/9\n",
    "p_the_given_non_spam = 1/9\n",
    "p_money_given_non_spam = 0/9\n",
    "p_non_spam_given_message = (p_non_spam * p_secret_given_non_spam *\n",
    "                            p_the_given_non_spam * p_money_given_non_spam)\n",
    "\n",
    "\n",
    "print(p_spam_given_message)\n",
    "print(p_non_spam_given_message)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Additive Smoothing\n",
    "\n",
    "In the previous exercise, we saw that both P(Spam|\"secret code to unlock the money\") and P(SpamC|\"secret code to unlock the money\") were equal to 0. This will always happen when we have words that occur in only one category — \"money\" occurs only in spam messages, while \"the\" only occurs in non-spam messages.\n",
    "\n",
    "![smoothing](img/cpm4_viz8.png)\n",
    "\n",
    "When we calculate P(Spam|\"secret code to unlock the money\"), we can see that P(\"the\"|Spam) is equal to 0 because \"the\" is not part of the spam messages. Unfortunately, that single value of 0 has the drawback of turning the result of the entire equation to 0:"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\begin{aligned}\n",
    "P(Spam|\\text{\"secret code to unlock the money\"}) &\\propto P(Spam) \\cdot P(\\text{\"secret\"}|Spam) \\cdot P(\\text{\"the\"}|Spam) \\cdot P(\\text{\"money\"}|Spam) \\\\\n",
    "&= \\frac{2}{4} \\cdot \\frac{4}{7} \\cdot \\frac{0}{7} \\cdot \\frac{2}{7} = 0\n",
    "\\end{aligned}\n",
    "\n",
    "To fix this problem, we need to find a way to avoid these cases where we get probabilities of 0. Let's start by laying out the equation we're using to calculate P(\"the\"|Spam):\n",
    "\n",
    "\\begin{equation}\n",
    "P(\\text{\"the\"}|Spam) = \\frac{\\text{total number of times \"the\" occurs in spam messages}}{\\text{total number of words in spam messages}} = \\frac{0}{7}\n",
    "\\end{equation}\n",
    "\n",
    "We're going to add some notation and rewrite the equation above as:\n",
    "\n",
    "\\begin{equation}\n",
    "P(\\text{\"the\"}|Spam) = \\frac{N_{\\text{\"the\"}|Spam}}{N_{Spam}} = \\frac{0}{7}\n",
    "\\end{equation}\n",
    "\n",
    "To fix the problem, we're going to use a technique called __additive smoothing__, where we add a smoothing parameter α. In the equation below, we'll use α=1 (below, N<sub>Vocabulary</sub> represents the number of unique words in all the messages — both spam and non-spam)."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\begin{equation}\n",
    "P(\\text{\"the\"}|Spam) = \\frac{N_{\\text{\"the\"}|Spam} + \\alpha}{N_{Spam} + \\alpha \\cdot N_{Vocabulary}} = \\frac{0 + 1}{7 + 1 \\cdot 9} = \\frac{1}{16}\n",
    "\\end{equation}\n",
    "\n",
    "The additive smoothing technique solves the issue and gets us a non-zero result, but it introduces another problem. We're now calculating probabilities differently depending on the word — take P(\"the\"|Spam) and P(\"secret\"|Spam) for instance:\n",
    "\n",
    "\\begin{equation}\n",
    "P(\\text{\"the\"}|Spam) = \\frac{N_{\\text{\"the\"}|Spam} + \\alpha}{N_{Spam} + \\alpha \\cdot N_{Vocabulary}}\n",
    "\\end{equation}\n",
    "\n",
    "\\begin{equation}\n",
    " P(\\text{\"secret\"}|Spam) = \\frac{N_{\\text{\"secret\"}|Spam}}{N_{Spam}}\n",
    "\\end{equation}"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Words like \"the\" are thus given special treatment and their probability are increased artificially to avoid non-zero cases, while words like \"secret\" are treated normally. To keep the probability values proportional across all words, we're going to use the additive smoothing for every word:\n",
    "\n",
    "\\begin{equation}\n",
    "P(\\text{\"the\"}|Spam) = \\frac{N_{\\text{\"the\"}|Spam} + \\alpha}{N_{Spam} + \\alpha \\cdot N_{Vocabulary}} \n",
    "\\end{equation}\n",
    "\n",
    "\\begin{equation}\n",
    "P(\\text{\"secret\"}|Spam) = \\frac{N_{\\text{\"secret\"}|Spam} + \\alpha}{N_{Spam} + \\alpha \\cdot N_{Vocabulary}}\n",
    "\\end{equation}"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In more general terms, this is the equation that we'll need to use for every word:\n",
    "\n",
    "\\begin{equation}\n",
    "P(word|Spam) = \\frac{N_{\\text{word}|Spam} + \\alpha}{N_{Spam} + \\alpha \\cdot N_{Vocabulary}}\n",
    "\\end{equation}\n",
    "\n",
    "> when `α=1`, the additive smoothing technique is most commonly known as __Laplace smoothing__ (or __add-one smoothing__). However, it is also possible to use `α<1`, in which case the technique is called __Lidstone smoothing__. "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task\n",
    "\n",
    "P(Spam|\"secret code to unlock the money\") is already calculated for you. Use the table below (the same as above) to calculate P(SpamC|\"secret code to unlock the money\").\n",
    "\n",
    "![smoothing](img/cpm4_viz722.png)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. Using the additive smoothing technique, calculate P(Spam<sup>c</sup>|\"secret code to unlock the money\"). Assign your answer to p_non_spam_given_message.\n",
    "2. Compare p_spam_given_message and p_non_spam_given_message to classify the message as spam or non-spam. If you think it's spam, then assign the string 'spam' to classification. Otherwise, assign 'non-spam'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "p_spam = 2/4\n",
    "p_secret_given_spam = (4 + 1) / (7 + 9)\n",
    "p_the_given_spam = (0 + 1) / (7 + 9)\n",
    "p_money_given_spam = (2 + 1) / (7 + 9)\n",
    "p_spam_given_message = (p_spam * p_secret_given_spam *\n",
    "                        p_the_given_spam * p_money_given_spam)\n",
    "p_non_spam = 2/4\n",
    "p_secret_given_non_spam = (2 + 1) / (9 + 9)\n",
    "p_the_given_non_spam = (1 + 1) / (9 + 9)\n",
    "p_money_given_non_spam = (0 + 1) / (9 + 9)\n",
    "p_non_spam_given_message = (p_non_spam * p_secret_given_non_spam *\n",
    "                            p_the_given_non_spam * p_money_given_non_spam)\n",
    "\n",
    "classification = 'spam'"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Multinomial Naive Bayes\n",
    "\n",
    "The Naive Bayes algorithm can be used for more than just building spam filters. For instance, we could use it to perform sentiment analysis for Twitter messages — the input is a Twitter message, and the output is the sentiment type (positive or negative). This follows the same pattern we saw with our spam filter, where the input is a new SMS message and the output is the message type (spam or non-spam).\n",
    "\n",
    "Depending on the math and the assumptions used, the Naive Bayes algorithm has a few variations. The three most popular Naive Bayes algorithms are:\n",
    "\n",
    "+ Multinomial Naive Bayes\n",
    "+ Gaussian Naive Bayes\n",
    "+  Bernoulli Naive Bayes\n",
    "\n",
    "In this lesson, we learned the multinomial Naive Bayes version of the algorithm. Explaining the mathematical differences between the various versions is out of the scope of this course, but it's important to keep in mind that all the Naive Bayes algorithms build on the (naive) conditional independence assumption we learned about earlier in this lesson. "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.9"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "0e1f169f5711a90ff09b1ce155e8835a9bac5ac4c242468736f19be364069406"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}