03- Hypothesis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Hypothesis Testing\n",
    "\n",
    "## One Sample Significance Tests\n",
    "\n",
    "The purpose of One Sample Significance Tests is to check if a sample of observations could have been generated by a process with a specific mean or proportion.\n",
    "\n",
    "Some questions that can be answered by one sample significance tests are:\n",
    "* Is there equal representation of men and women in a particular industry?\n",
    "* Is the normal human body temperature 98.6 F?\n",
    "\n",
    "We will try and apply this test to a few real world problems in this notebook.\n",
    "\n",
    "The Suicide dataset was obtained from Kaggle courtesy Rajanand Illangovan. You can download it here: https://www.kaggle.com/rajanand/suicides-in-india"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from scipy import stats"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Analyzing Suicides in India by Gender\n",
    "\n",
    "Are men as likely to commit suicide as women?\n",
    "\n",
    "This is the question we will attempt at answering in this section. To answer this question, we will use suicide statistics shared by the National Crime Records Bureau (NCRB), Govt of India. To perform this analysis, we need to know the sex ratio in India. The Census 2011 report states that there are 940 females for every 1000 males in India.\n",
    "\n",
    "Let p denote the fraction of women in India."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.4845360824742268"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "p = 940/(940+1000)\n",
    "p"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If there is no correlation between gender and suicide, then the sex ratio of people committing suicides should closely reflect that of the general population. \n",
    "\n",
    "Let us now get our data into a Pandas dataframe for analysis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>State</th>\n",
       "      <th>Year</th>\n",
       "      <th>Type_code</th>\n",
       "      <th>Type</th>\n",
       "      <th>Gender</th>\n",
       "      <th>Age_group</th>\n",
       "      <th>Total</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>A &amp; N Islands</td>\n",
       "      <td>2001</td>\n",
       "      <td>Causes</td>\n",
       "      <td>Illness (Aids/STD)</td>\n",
       "      <td>Female</td>\n",
       "      <td>0-14</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>A &amp; N Islands</td>\n",
       "      <td>2001</td>\n",
       "      <td>Causes</td>\n",
       "      <td>Bankruptcy or Sudden change in Economic</td>\n",
       "      <td>Female</td>\n",
       "      <td>0-14</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>A &amp; N Islands</td>\n",
       "      <td>2001</td>\n",
       "      <td>Causes</td>\n",
       "      <td>Cancellation/Non-Settlement of Marriage</td>\n",
       "      <td>Female</td>\n",
       "      <td>0-14</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>A &amp; N Islands</td>\n",
       "      <td>2001</td>\n",
       "      <td>Causes</td>\n",
       "      <td>Physical Abuse (Rape/Incest Etc.)</td>\n",
       "      <td>Female</td>\n",
       "      <td>0-14</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>A &amp; N Islands</td>\n",
       "      <td>2001</td>\n",
       "      <td>Causes</td>\n",
       "      <td>Dowry Dispute</td>\n",
       "      <td>Female</td>\n",
       "      <td>0-14</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           State  Year Type_code                                     Type  \\\n",
       "0  A & N Islands  2001    Causes                       Illness (Aids/STD)   \n",
       "1  A & N Islands  2001    Causes  Bankruptcy or Sudden change in Economic   \n",
       "2  A & N Islands  2001    Causes  Cancellation/Non-Settlement of Marriage   \n",
       "3  A & N Islands  2001    Causes        Physical Abuse (Rape/Incest Etc.)   \n",
       "4  A & N Islands  2001    Causes                            Dowry Dispute   \n",
       "\n",
       "   Gender Age_group  Total  \n",
       "0  Female      0-14      0  \n",
       "1  Female      0-14      0  \n",
       "2  Female      0-14      0  \n",
       "3  Female      0-14      0  \n",
       "4  Female      0-14      0  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv('data/suicides.csv')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(237519, 7)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Male      118879\n",
       "Female    118640\n",
       "Name: Gender, dtype: int64"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['Gender'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that the number of female suicides is slightly lesser than the number of male suicides. There are also fewer females than males. How do we prove that females are as likely to commit suicide as males? This can be answered through hypothesis testing.|"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Step 1: Formulate the hypothesis and decide on confidence level\n",
    "\n",
    "The null hypothesis, as stated in the slides, is the default state. Therefore, I will state my null and alternate hypothesis as follows.\n",
    "\n",
    "* **Null Hypothesis (H0)**: Men and women are equally likely to commit suicide.\n",
    "* **Alternate Hypothesis (H1)**: Men and women are not equally likely to commit suicide.\n",
    "\n",
    "If the null hypothesis is true, it would mean that the fraction of women committing suicide would be the same as the fraction of women in the general population. We now need to use a suitable statistica test to find out if this is indeed is the case.\n",
    "\n",
    "Our statistical test will generate a p-value which has to be compared to a significance level ($\\alpha$). If p is less than alpha, then it is extremely unlikely that the event must have occurred by chance and we would be reasonable in rejecting the null hypothesis. On the contrary, if the p-value is higher than $\\alpha$, we will not be in a position to reject the null hypothesis.\n",
    "\n",
    "Let us assume, $\\alpha$ = 0.05"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Step 2: Decide on the Statsitical Test\n",
    "\n",
    "We will be using the One Sample Z-Test here. How to decide upon a test will be discussed in another notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Step 3: Compute the p-value"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.4845360824742268"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "h0_prop = p\n",
    "h0_prop"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.49949688235467476"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "h1_prop = df['Gender'].value_counts()['Female']/len(df)\n",
    "h1_prop"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.0010254465276083747"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sigma_prop = np.sqrt((h0_prop * (1 - h0_prop))/len(df))\n",
    "sigma_prop"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "14.589546580591277"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "z = (h1_prop - h0_prop)/sigma_prop\n",
    "z"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def pvalue(z):\n",
    "    return 2 * (1 - stats.norm.cdf(z))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.0"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "p_val = (1-stats.norm.cdf(z))*2\n",
    "p_val"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The p value is so small that Python has effectively rounded it to zero.\n",
    "\n",
    "#### Step 4: Comparison and Decision\n",
    "\n",
    "The p value obtained is extremely strong evidence to suggest that it is much lower than our significance level $\\alpha$. We can thus safely disregard the null hypothesis and accept the alternate hypothesis (since it is the negation of the null hypothesis).\n",
    "\n",
    "**Men and women are not equally likely to commit suicide.**\n",
    "\n",
    "Note that this test says nothing about if men are more likely than women to commit suicide or vice versa. It just states that they are not equally likely. The reader is encouraged to form their own hypothesis tests to check these results."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Analyzing the average heights of NBA Players\n",
    "\n",
    "I was interested in knowing the average height of NBA playes. A quick Google search tells me that the average height of players between 1985-2006 was **6'7\"** or 200.66 cm. Is this still the case?\n",
    "\n",
    "To answer this question, we will be using the NBA Players Stats - 2014-2015 dataset on Kaggle courtesy DrGuillermo. The dataset can be downloaded here: https://www.kaggle.com/drgilermo/nba-players-stats-20142015"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Name</th>\n",
       "      <th>Games Played</th>\n",
       "      <th>MIN</th>\n",
       "      <th>PTS</th>\n",
       "      <th>FGM</th>\n",
       "      <th>FGA</th>\n",
       "      <th>FG%</th>\n",
       "      <th>3PM</th>\n",
       "      <th>3PA</th>\n",
       "      <th>3P%</th>\n",
       "      <th>...</th>\n",
       "      <th>Age</th>\n",
       "      <th>Birth_Place</th>\n",
       "      <th>Birthdate</th>\n",
       "      <th>Collage</th>\n",
       "      <th>Experience</th>\n",
       "      <th>Height</th>\n",
       "      <th>Pos</th>\n",
       "      <th>Team</th>\n",
       "      <th>Weight</th>\n",
       "      <th>BMI</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>AJ Price</td>\n",
       "      <td>26</td>\n",
       "      <td>324</td>\n",
       "      <td>133</td>\n",
       "      <td>51</td>\n",
       "      <td>137</td>\n",
       "      <td>37.2</td>\n",
       "      <td>15</td>\n",
       "      <td>57</td>\n",
       "      <td>26.3</td>\n",
       "      <td>...</td>\n",
       "      <td>29.0</td>\n",
       "      <td>us</td>\n",
       "      <td>October 7, 1986</td>\n",
       "      <td>University of Connecticut</td>\n",
       "      <td>5</td>\n",
       "      <td>185.0</td>\n",
       "      <td>PG</td>\n",
       "      <td>PHO</td>\n",
       "      <td>81.45</td>\n",
       "      <td>23.798393</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Aaron Brooks</td>\n",
       "      <td>82</td>\n",
       "      <td>1885</td>\n",
       "      <td>954</td>\n",
       "      <td>344</td>\n",
       "      <td>817</td>\n",
       "      <td>42.1</td>\n",
       "      <td>121</td>\n",
       "      <td>313</td>\n",
       "      <td>38.7</td>\n",
       "      <td>...</td>\n",
       "      <td>30.0</td>\n",
       "      <td>us</td>\n",
       "      <td>January 14, 1985</td>\n",
       "      <td>University of Oregon</td>\n",
       "      <td>6</td>\n",
       "      <td>180.0</td>\n",
       "      <td>PG</td>\n",
       "      <td>CHI</td>\n",
       "      <td>72.45</td>\n",
       "      <td>22.361111</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Aaron Gordon</td>\n",
       "      <td>47</td>\n",
       "      <td>797</td>\n",
       "      <td>243</td>\n",
       "      <td>93</td>\n",
       "      <td>208</td>\n",
       "      <td>44.7</td>\n",
       "      <td>13</td>\n",
       "      <td>48</td>\n",
       "      <td>27.1</td>\n",
       "      <td>...</td>\n",
       "      <td>20.0</td>\n",
       "      <td>us</td>\n",
       "      <td>September 16, 1995</td>\n",
       "      <td>University of Arizona</td>\n",
       "      <td>R</td>\n",
       "      <td>202.5</td>\n",
       "      <td>PF</td>\n",
       "      <td>ORL</td>\n",
       "      <td>99.00</td>\n",
       "      <td>24.142661</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Adreian Payne</td>\n",
       "      <td>32</td>\n",
       "      <td>740</td>\n",
       "      <td>213</td>\n",
       "      <td>91</td>\n",
       "      <td>220</td>\n",
       "      <td>41.4</td>\n",
       "      <td>1</td>\n",
       "      <td>9</td>\n",
       "      <td>11.1</td>\n",
       "      <td>...</td>\n",
       "      <td>24.0</td>\n",
       "      <td>us</td>\n",
       "      <td>February 19, 1991</td>\n",
       "      <td>Michigan State University</td>\n",
       "      <td>R</td>\n",
       "      <td>205.0</td>\n",
       "      <td>PF</td>\n",
       "      <td>ATL</td>\n",
       "      <td>106.65</td>\n",
       "      <td>25.377751</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Al Horford</td>\n",
       "      <td>76</td>\n",
       "      <td>2318</td>\n",
       "      <td>1156</td>\n",
       "      <td>519</td>\n",
       "      <td>965</td>\n",
       "      <td>53.8</td>\n",
       "      <td>11</td>\n",
       "      <td>36</td>\n",
       "      <td>30.6</td>\n",
       "      <td>...</td>\n",
       "      <td>29.0</td>\n",
       "      <td>do</td>\n",
       "      <td>June 3, 1986</td>\n",
       "      <td>University of Florida</td>\n",
       "      <td>7</td>\n",
       "      <td>205.0</td>\n",
       "      <td>C</td>\n",
       "      <td>ATL</td>\n",
       "      <td>110.25</td>\n",
       "      <td>26.234384</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 34 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "            Name  Games Played   MIN   PTS  FGM  FGA   FG%  3PM  3PA   3P%  \\\n",
       "0       AJ Price            26   324   133   51  137  37.2   15   57  26.3   \n",
       "1   Aaron Brooks            82  1885   954  344  817  42.1  121  313  38.7   \n",
       "2   Aaron Gordon            47   797   243   93  208  44.7   13   48  27.1   \n",
       "3  Adreian Payne            32   740   213   91  220  41.4    1    9  11.1   \n",
       "4     Al Horford            76  2318  1156  519  965  53.8   11   36  30.6   \n",
       "\n",
       "     ...       Age  Birth_Place           Birthdate  \\\n",
       "0    ...      29.0           us     October 7, 1986   \n",
       "1    ...      30.0           us    January 14, 1985   \n",
       "2    ...      20.0           us  September 16, 1995   \n",
       "3    ...      24.0           us   February 19, 1991   \n",
       "4    ...      29.0           do        June 3, 1986   \n",
       "\n",
       "                     Collage  Experience  Height  Pos  Team  Weight        BMI  \n",
       "0  University of Connecticut           5   185.0   PG   PHO   81.45  23.798393  \n",
       "1       University of Oregon           6   180.0   PG   CHI   72.45  22.361111  \n",
       "2      University of Arizona           R   202.5   PF   ORL   99.00  24.142661  \n",
       "3  Michigan State University           R   205.0   PF   ATL  106.65  25.377751  \n",
       "4      University of Florida           7   205.0    C   ATL  110.25  26.234384  \n",
       "\n",
       "[5 rows x 34 columns]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df2 = pd.read_csv('data/players_stats.csv')\n",
    "df2.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(490, 34)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df2.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Hypothesis Testing\n",
    "\n",
    "One Sample Significance Test for Mean is extremely similar to that for Proportion. We will go through almost an identical process.\n",
    "\n",
    "The hypotheses are defined as follows:\n",
    "* **Null Hypothesis**: The average height of an NBA player is 200.66 cm.\n",
    "* **Alternate Hypothesis**: The average height of an NBA player is not 200.66 cm.\n",
    "\n",
    "Significance Level, $\\alpha$ is at 0.05. Assuming Null Hypothesis to be true."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "h0_mean = 200.66"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "197.44075829383885"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "h1_mean = df2['Height'].mean()\n",
    "h1_mean"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.39484424472376178"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sigma = df2['Height'].std()/np.sqrt(len(df2))\n",
    "sigma"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-8.1531939471812898"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "z = (h1_mean - h0_mean)/sigma\n",
    "z"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4.4408920985006262e-16"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "p_val = (1 - stats.norm.cdf(abs(z))) * 2\n",
    "p_val"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The p value obtained is much lesser than the significance level $\\alpha$. We therefore reject the null hypothesis and accept the alternate hypothesis (the negation). We can therefore arrive at the following conclusion from this analysis:\n",
    "\n",
    "**The average height of NBA Players is NOT 6'7\"**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Two Sample Significance Tests\n",
    "\n",
    "In the last section, we saw how one sample significance tests could be used to test if the proportion or the mean of a certain feature of a population is equal to a predefined proportion or mean respectively. In other words, we were comparing A sample with a prdefined value.\n",
    "\n",
    "Two sample significance tests, on the other hand, allow us to compare two different populations and check if there is any meaningful difference in their means or proportions. The steps involved and the tools used are almost identical to the one sample significance test with one critical difference. The null hypothesis mean or proportion is assumed to be the difference of the means or proportions of the two populations and is set to zero.\n",
    "\n",
    "Using two sample significance tests, we can answer questions such as:\n",
    "* Is there racial discrimination when it comes to recruitment for white collar jobs?\n",
    "* Is there a pay gap between men and women in the industry? Are women, on average, paid less?\n",
    "* Do some universities involve in conscious racial discrimination? That is, are they more inclined to accept a student of a particular race as compared to another?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Analyzing Literacy Rates\n",
    "\n",
    "In this section, we will try and compare the literacy rates in the major areas of Punjab and Delhi ICT and discern if there is any meaningful difference between the two aforementioned quantities.\n",
    "\n",
    "To answer this question, we will be using the 'Top 500 Indian Cities' dataset made available on Kaggle courtesy Arijit Mukherjee. The dataset can be found here: https://www.kaggle.com/zed9941/top-500-indian-cities"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name_of_city</th>\n",
       "      <th>state_code</th>\n",
       "      <th>state_name</th>\n",
       "      <th>dist_code</th>\n",
       "      <th>population_total</th>\n",
       "      <th>population_male</th>\n",
       "      <th>population_female</th>\n",
       "      <th>0-6_population_total</th>\n",
       "      <th>0-6_population_male</th>\n",
       "      <th>0-6_population_female</th>\n",
       "      <th>...</th>\n",
       "      <th>literates_female</th>\n",
       "      <th>sex_ratio</th>\n",
       "      <th>child_sex_ratio</th>\n",
       "      <th>effective_literacy_rate_total</th>\n",
       "      <th>effective_literacy_rate_male</th>\n",
       "      <th>effective_literacy_rate_female</th>\n",
       "      <th>location</th>\n",
       "      <th>total_graduates</th>\n",
       "      <th>male_graduates</th>\n",
       "      <th>female_graduates</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Abohar</td>\n",
       "      <td>3</td>\n",
       "      <td>PUNJAB</td>\n",
       "      <td>9</td>\n",
       "      <td>145238</td>\n",
       "      <td>76840</td>\n",
       "      <td>68398</td>\n",
       "      <td>15870</td>\n",
       "      <td>8587</td>\n",
       "      <td>7283</td>\n",
       "      <td>...</td>\n",
       "      <td>44972</td>\n",
       "      <td>890</td>\n",
       "      <td>848</td>\n",
       "      <td>79.86</td>\n",
       "      <td>85.49</td>\n",
       "      <td>73.59</td>\n",
       "      <td>30.1452928,74.1993043</td>\n",
       "      <td>16287</td>\n",
       "      <td>8612</td>\n",
       "      <td>7675</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Achalpur</td>\n",
       "      <td>27</td>\n",
       "      <td>MAHARASHTRA</td>\n",
       "      <td>7</td>\n",
       "      <td>112293</td>\n",
       "      <td>58256</td>\n",
       "      <td>54037</td>\n",
       "      <td>11810</td>\n",
       "      <td>6186</td>\n",
       "      <td>5624</td>\n",
       "      <td>...</td>\n",
       "      <td>43086</td>\n",
       "      <td>928</td>\n",
       "      <td>909</td>\n",
       "      <td>91.99</td>\n",
       "      <td>94.77</td>\n",
       "      <td>89.00</td>\n",
       "      <td>21.257584,77.5086754</td>\n",
       "      <td>8863</td>\n",
       "      <td>5269</td>\n",
       "      <td>3594</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Adilabad</td>\n",
       "      <td>28</td>\n",
       "      <td>ANDHRA PRADESH</td>\n",
       "      <td>1</td>\n",
       "      <td>117388</td>\n",
       "      <td>59232</td>\n",
       "      <td>58156</td>\n",
       "      <td>13103</td>\n",
       "      <td>6731</td>\n",
       "      <td>6372</td>\n",
       "      <td>...</td>\n",
       "      <td>37660</td>\n",
       "      <td>982</td>\n",
       "      <td>947</td>\n",
       "      <td>80.51</td>\n",
       "      <td>88.18</td>\n",
       "      <td>72.73</td>\n",
       "      <td>19.0809075,79.560344</td>\n",
       "      <td>10565</td>\n",
       "      <td>6797</td>\n",
       "      <td>3768</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Adityapur</td>\n",
       "      <td>20</td>\n",
       "      <td>JHARKHAND</td>\n",
       "      <td>24</td>\n",
       "      <td>173988</td>\n",
       "      <td>91495</td>\n",
       "      <td>82493</td>\n",
       "      <td>23042</td>\n",
       "      <td>12063</td>\n",
       "      <td>10979</td>\n",
       "      <td>...</td>\n",
       "      <td>54515</td>\n",
       "      <td>902</td>\n",
       "      <td>910</td>\n",
       "      <td>83.46</td>\n",
       "      <td>89.98</td>\n",
       "      <td>76.23</td>\n",
       "      <td>22.7834741,86.1576889</td>\n",
       "      <td>19225</td>\n",
       "      <td>12189</td>\n",
       "      <td>7036</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Adoni</td>\n",
       "      <td>28</td>\n",
       "      <td>ANDHRA PRADESH</td>\n",
       "      <td>21</td>\n",
       "      <td>166537</td>\n",
       "      <td>82743</td>\n",
       "      <td>83794</td>\n",
       "      <td>18406</td>\n",
       "      <td>9355</td>\n",
       "      <td>9051</td>\n",
       "      <td>...</td>\n",
       "      <td>45089</td>\n",
       "      <td>1013</td>\n",
       "      <td>968</td>\n",
       "      <td>68.38</td>\n",
       "      <td>76.58</td>\n",
       "      <td>60.33</td>\n",
       "      <td>15.6322227,77.2728368</td>\n",
       "      <td>11902</td>\n",
       "      <td>7871</td>\n",
       "      <td>4031</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 22 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "  name_of_city  state_code      state_name  dist_code  population_total  \\\n",
       "0      Abohar            3          PUNJAB          9            145238   \n",
       "1    Achalpur           27     MAHARASHTRA          7            112293   \n",
       "2    Adilabad           28  ANDHRA PRADESH          1            117388   \n",
       "3   Adityapur           20       JHARKHAND         24            173988   \n",
       "4       Adoni           28  ANDHRA PRADESH         21            166537   \n",
       "\n",
       "   population_male  population_female  0-6_population_total  \\\n",
       "0            76840              68398                 15870   \n",
       "1            58256              54037                 11810   \n",
       "2            59232              58156                 13103   \n",
       "3            91495              82493                 23042   \n",
       "4            82743              83794                 18406   \n",
       "\n",
       "   0-6_population_male  0-6_population_female        ...         \\\n",
       "0                 8587                   7283        ...          \n",
       "1                 6186                   5624        ...          \n",
       "2                 6731                   6372        ...          \n",
       "3                12063                  10979        ...          \n",
       "4                 9355                   9051        ...          \n",
       "\n",
       "   literates_female  sex_ratio  child_sex_ratio  \\\n",
       "0             44972        890              848   \n",
       "1             43086        928              909   \n",
       "2             37660        982              947   \n",
       "3             54515        902              910   \n",
       "4             45089       1013              968   \n",
       "\n",
       "   effective_literacy_rate_total  effective_literacy_rate_male  \\\n",
       "0                          79.86                         85.49   \n",
       "1                          91.99                         94.77   \n",
       "2                          80.51                         88.18   \n",
       "3                          83.46                         89.98   \n",
       "4                          68.38                         76.58   \n",
       "\n",
       "   effective_literacy_rate_female               location  total_graduates  \\\n",
       "0                           73.59  30.1452928,74.1993043            16287   \n",
       "1                           89.00   21.257584,77.5086754             8863   \n",
       "2                           72.73   19.0809075,79.560344            10565   \n",
       "3                           76.23  22.7834741,86.1576889            19225   \n",
       "4                           60.33  15.6322227,77.2728368            11902   \n",
       "\n",
       "  male_graduates  female_graduates  \n",
       "0           8612              7675  \n",
       "1           5269              3594  \n",
       "2           6797              3768  \n",
       "3          12189              7036  \n",
       "4           7871              4031  \n",
       "\n",
       "[5 rows x 22 columns]"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df3 = pd.read_csv('data/cities.csv')\n",
    "df3.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "UTTAR PRADESH                63\n",
       "WEST BENGAL                  61\n",
       "MAHARASHTRA                  43\n",
       "ANDHRA PRADESH               42\n",
       "MADHYA PRADESH               32\n",
       "TAMIL NADU                   32\n",
       "GUJARAT                      29\n",
       "RAJASTHAN                    29\n",
       "KARNATAKA                    26\n",
       "BIHAR                        26\n",
       "HARYANA                      20\n",
       "PUNJAB                       16\n",
       "NCT OF DELHI                 15\n",
       "JHARKHAND                    10\n",
       "ORISSA                       10\n",
       "CHHATTISGARH                  9\n",
       "KERALA                        7\n",
       "UTTARAKHAND                   6\n",
       "ASSAM                         4\n",
       "JAMMU & KASHMIR               3\n",
       "PUDUCHERRY                    2\n",
       "CHANDIGARH                    1\n",
       "MANIPUR                       1\n",
       "NAGALAND                      1\n",
       "MIZORAM                       1\n",
       "HIMACHAL PRADESH              1\n",
       "ANDAMAN & NICOBAR ISLANDS     1\n",
       "TRIPURA                       1\n",
       "MEGHALAYA                     1\n",
       "Name: state_name, dtype: int64"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df3['state_name'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "punjab = df3[df3['state_name'] == 'PUNJAB']['effective_literacy_rate_total']\n",
    "delhi = df3[df3['state_name'] == 'NCT OF DELHI']['effective_literacy_rate_total']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(83.440624999999997, 5.381935796408821)"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "punjab_mean = punjab.mean()\n",
    "punjab_std = punjab.std()\n",
    "\n",
    "punjab_mean, punjab_std"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(83.658000000000001, 4.6569551671206195)"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "delhi_mean = delhi.mean()\n",
    "delhi_std = delhi.std()\n",
    "\n",
    "delhi_mean, delhi_std"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the above calculations, it can be seen that the mean and the standard deviations of Punjab and Delhi literacy rates differ slightly. The next step is to determine if this difference is a statistically significant one."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For hypothesis testing, the following are defined:\n",
    "\n",
    "* **Null Hypothesis:** The true mean literacy rate for Punjab and Delhi are the same.\n",
    "* **Alternate Hypothesis:** The true mean literacy rate for Punjab and Delhi are not the same.\n",
    "\n",
    "The threshold value of $\\alpha$ is assumed to be 0.05.\n",
    "Assuming Null Hypothesis is true."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.21737500000000409, 1.8044784525904138)"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "h0_mean = 0\n",
    "mean_diff = delhi_mean - punjab_mean\n",
    "sigma_diff = np.sqrt((delhi_std**2)/len(delhi)  + (punjab_std**2)/len(punjab))\n",
    "mean_diff, sigma_diff"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since we are dealing with sample sizes less than 30, using the t-statistic will be more appropriate. To use student's t though, we need to calculate the degree of freedom. This is done as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "28.826817888400029"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "deg = (((delhi_std**2)/len(delhi)  + (punjab_std**2)/len(punjab)) ** 2) / ((((delhi_std**2)/len(delhi))**2)/(len(delhi)-1)  + (((punjab_std**2)/len(punjab))**2)/(len(punjab) - 1))\n",
    "deg"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.12046417051306545"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "z = (mean_diff - h0_mean) / sigma_diff\n",
    "z"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.90495118045088319"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "p = (1-stats.t.cdf(z, deg))*2\n",
    "p"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The value of p obtained here is much higher than the significance level $\\alpha$. Therefore, we cannot reject the null hypothesis. It stands.\n",
    "\n",
    "**The true mean literacy rate for Punjab and Delhi are the same.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Chi Squared Significance Test\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let us check out the Chi Squared Significance Test. The Chi Square test is used to check if there is a dependency between two ordinal or categorical variables. \n",
    "\n",
    "For instance, let's say we want to know the preference of ice cream between men and women. We give them three choices of ice creams and ask them to choose their favorite. Is there a gender preference for a certain type of ice cream? This is something that this test can answer for us."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Analysing Airbnb Booking Trends by Gender\n",
    "\n",
    "Do men and women have certain preferences when it comes to traveling to a certain country. To answer this question, we will be using the Airbnb Bookings dataset available on Kaggle. It can be downloaded here: https://www.kaggle.com/c/airbnb-recruiting-new-user-bookings\n",
    "\n",
    "The steps to be followed are the same as above. The only difference is in the statistical method that we are going to use.\n",
    "\n",
    "For the hypothesis testing, we define the following:\n",
    "\n",
    "* **Null Hypothesis:** There is no relationship between country preference and the sex of the customer.\n",
    "* **Alternate Hypothesis:** There is a relationship between country preference and the sex of the customer.\n",
    "\n",
    "We will assume our significance level, $\\alpha$ to be 0.05. Let's begin!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>date_account_created</th>\n",
       "      <th>timestamp_first_active</th>\n",
       "      <th>date_first_booking</th>\n",
       "      <th>gender</th>\n",
       "      <th>age</th>\n",
       "      <th>signup_method</th>\n",
       "      <th>signup_flow</th>\n",
       "      <th>language</th>\n",
       "      <th>affiliate_channel</th>\n",
       "      <th>affiliate_provider</th>\n",
       "      <th>first_affiliate_tracked</th>\n",
       "      <th>signup_app</th>\n",
       "      <th>first_device_type</th>\n",
       "      <th>first_browser</th>\n",
       "      <th>country_destination</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>gxn3p5htnn</td>\n",
       "      <td>2010-06-28</td>\n",
       "      <td>20090319043255</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-unknown-</td>\n",
       "      <td>NaN</td>\n",
       "      <td>facebook</td>\n",
       "      <td>0</td>\n",
       "      <td>en</td>\n",
       "      <td>direct</td>\n",
       "      <td>direct</td>\n",
       "      <td>untracked</td>\n",
       "      <td>Web</td>\n",
       "      <td>Mac Desktop</td>\n",
       "      <td>Chrome</td>\n",
       "      <td>NDF</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>820tgsjxq7</td>\n",
       "      <td>2011-05-25</td>\n",
       "      <td>20090523174809</td>\n",
       "      <td>NaN</td>\n",
       "      <td>MALE</td>\n",
       "      <td>38.0</td>\n",
       "      <td>facebook</td>\n",
       "      <td>0</td>\n",
       "      <td>en</td>\n",
       "      <td>seo</td>\n",
       "      <td>google</td>\n",
       "      <td>untracked</td>\n",
       "      <td>Web</td>\n",
       "      <td>Mac Desktop</td>\n",
       "      <td>Chrome</td>\n",
       "      <td>NDF</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>4ft3gnwmtx</td>\n",
       "      <td>2010-09-28</td>\n",
       "      <td>20090609231247</td>\n",
       "      <td>2010-08-02</td>\n",
       "      <td>FEMALE</td>\n",
       "      <td>56.0</td>\n",
       "      <td>basic</td>\n",
       "      <td>3</td>\n",
       "      <td>en</td>\n",
       "      <td>direct</td>\n",
       "      <td>direct</td>\n",
       "      <td>untracked</td>\n",
       "      <td>Web</td>\n",
       "      <td>Windows Desktop</td>\n",
       "      <td>IE</td>\n",
       "      <td>US</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>bjjt8pjhuk</td>\n",
       "      <td>2011-12-05</td>\n",
       "      <td>20091031060129</td>\n",
       "      <td>2012-09-08</td>\n",
       "      <td>FEMALE</td>\n",
       "      <td>42.0</td>\n",
       "      <td>facebook</td>\n",
       "      <td>0</td>\n",
       "      <td>en</td>\n",
       "      <td>direct</td>\n",
       "      <td>direct</td>\n",
       "      <td>untracked</td>\n",
       "      <td>Web</td>\n",
       "      <td>Mac Desktop</td>\n",
       "      <td>Firefox</td>\n",
       "      <td>other</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>87mebub9p4</td>\n",
       "      <td>2010-09-14</td>\n",
       "      <td>20091208061105</td>\n",
       "      <td>2010-02-18</td>\n",
       "      <td>-unknown-</td>\n",
       "      <td>41.0</td>\n",
       "      <td>basic</td>\n",
       "      <td>0</td>\n",
       "      <td>en</td>\n",
       "      <td>direct</td>\n",
       "      <td>direct</td>\n",
       "      <td>untracked</td>\n",
       "      <td>Web</td>\n",
       "      <td>Mac Desktop</td>\n",
       "      <td>Chrome</td>\n",
       "      <td>US</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           id date_account_created  timestamp_first_active date_first_booking  \\\n",
       "0  gxn3p5htnn           2010-06-28          20090319043255                NaN   \n",
       "1  820tgsjxq7           2011-05-25          20090523174809                NaN   \n",
       "2  4ft3gnwmtx           2010-09-28          20090609231247         2010-08-02   \n",
       "3  bjjt8pjhuk           2011-12-05          20091031060129         2012-09-08   \n",
       "4  87mebub9p4           2010-09-14          20091208061105         2010-02-18   \n",
       "\n",
       "      gender   age signup_method  signup_flow language affiliate_channel  \\\n",
       "0  -unknown-   NaN      facebook            0       en            direct   \n",
       "1       MALE  38.0      facebook            0       en               seo   \n",
       "2     FEMALE  56.0         basic            3       en            direct   \n",
       "3     FEMALE  42.0      facebook            0       en            direct   \n",
       "4  -unknown-  41.0         basic            0       en            direct   \n",
       "\n",
       "  affiliate_provider first_affiliate_tracked signup_app first_device_type  \\\n",
       "0             direct               untracked        Web       Mac Desktop   \n",
       "1             google               untracked        Web       Mac Desktop   \n",
       "2             direct               untracked        Web   Windows Desktop   \n",
       "3             direct               untracked        Web       Mac Desktop   \n",
       "4             direct               untracked        Web       Mac Desktop   \n",
       "\n",
       "  first_browser country_destination  \n",
       "0        Chrome                 NDF  \n",
       "1        Chrome                 NDF  \n",
       "2            IE                  US  \n",
       "3       Firefox               other  \n",
       "4        Chrome                  US  "
      ]
     },
     "execution_count": 96,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df4 = pd.read_csv('data/airbnb.csv')\n",
    "df4.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "FEMALE    63041\n",
       "MALE      54440\n",
       "Name: gender, dtype: int64"
      ]
     },
     "execution_count": 97,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df4 = df4[(df4['gender'] != '-unknown-') & (df4['gender'] != 'OTHER')]\n",
    "df4['gender'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "NDF      57767\n",
       "US       42151\n",
       "other     6603\n",
       "FR        3297\n",
       "IT        1790\n",
       "GB        1563\n",
       "ES        1530\n",
       "CA         932\n",
       "DE         774\n",
       "NL         532\n",
       "AU         395\n",
       "PT         147\n",
       "Name: country_destination, dtype: int64"
      ]
     },
     "execution_count": 98,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df4['country_destination'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To keep our calculations simple, we'll just be looking at France, Italy and Great Britain. However, please note that this test can theoretically be used for an arbritrarily large number of categories."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>gender</th>\n",
       "      <th>country_destination</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>gdka1q5ktd</td>\n",
       "      <td>FEMALE</td>\n",
       "      <td>FR</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>al8bcetz0g</td>\n",
       "      <td>FEMALE</td>\n",
       "      <td>FR</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>74</th>\n",
       "      <td>h53mz152m4</td>\n",
       "      <td>FEMALE</td>\n",
       "      <td>FR</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>87</th>\n",
       "      <td>wqj8o7gv3y</td>\n",
       "      <td>FEMALE</td>\n",
       "      <td>IT</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>107</th>\n",
       "      <td>2c543qkuic</td>\n",
       "      <td>FEMALE</td>\n",
       "      <td>FR</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             id  gender country_destination\n",
       "19   gdka1q5ktd  FEMALE                  FR\n",
       "25   al8bcetz0g  FEMALE                  FR\n",
       "74   h53mz152m4  FEMALE                  FR\n",
       "87   wqj8o7gv3y  FEMALE                  IT\n",
       "107  2c543qkuic  FEMALE                  FR"
      ]
     },
     "execution_count": 99,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df4 = df4[(df4['country_destination'] == 'FR') | (df4['country_destination'] == 'IT') | (df4['country_destination'] == 'GB')][['id', 'gender', 'country_destination']]\n",
    "df4.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us now construct the contingency table."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 161,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>FR</th>\n",
       "      <th>GB</th>\n",
       "      <th>IT</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>gender</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>FEMALE</th>\n",
       "      <td>1962</td>\n",
       "      <td>881</td>\n",
       "      <td>1091</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>MALE</th>\n",
       "      <td>1335</td>\n",
       "      <td>682</td>\n",
       "      <td>699</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          FR   GB    IT\n",
       "gender                 \n",
       "FEMALE  1962  881  1091\n",
       "MALE    1335  682   699"
      ]
     },
     "execution_count": 161,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "observed = df4.pivot_table('id', ['gender'], 'country_destination', aggfunc='count').reset_index()\n",
    "del observed.columns.name\n",
    "observed = observed.set_index('gender')\n",
    "observed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 162,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "chi2, p, dof, expected = stats.chi2_contingency(observed)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 163,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "7.5890123075794911"
      ]
     },
     "execution_count": 163,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "chi2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 164,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.022494011656631389"
      ]
     },
     "execution_count": 164,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "p"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The p-value that we have obtained is less than our chosen significance level. Therefore, we reject the null hypothesis and accept the negating alterate hypothesis.\n",
    "\n",
    "**There is a relationship between country preference and the sex of the customer.**\n",
    "\n",
    "Note that had we chosen our $\\alpha$ to be 0.01, then we wouldn't have been able to reject our null hypothesis. Choosing the value of $\\aplha$ depends on your relative tolerance of Type I and Type II error. We will not discuss this in this talk."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using Different Tests to arrive at the same result\n",
    "\n",
    "### Racial Discrimination in Industry Recruitments\n",
    "\n",
    "In this section, we are going to check for racial discrimination using two different tests: The test of two proportions (the two sample significance test) and the Chi Square significance test. We will also compare the statistical results arrived at by both these tests (specifically, the p-value).\n",
    "\n",
    "The question we are trying to answer is if there is racial discrimination when it comes to giving callbacks to candidates. The first step, as usual, is to formulate our hypothesis.\n",
    "\n",
    "* **Null Hypothesis:** There is no relation between race and callbacks\n",
    "* **Alternate Hypothesis:** There is a relationship between race and callbacks\n",
    "\n",
    "We are going to assume that the null hypothesis is true. Also, the significance level $\\alpha$ is assumed to be 10% or 0.1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 165,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>ad</th>\n",
       "      <th>education</th>\n",
       "      <th>ofjobs</th>\n",
       "      <th>yearsexp</th>\n",
       "      <th>honors</th>\n",
       "      <th>volunteer</th>\n",
       "      <th>military</th>\n",
       "      <th>empholes</th>\n",
       "      <th>occupspecific</th>\n",
       "      <th>...</th>\n",
       "      <th>compreq</th>\n",
       "      <th>orgreq</th>\n",
       "      <th>manuf</th>\n",
       "      <th>transcom</th>\n",
       "      <th>bankreal</th>\n",
       "      <th>trade</th>\n",
       "      <th>busservice</th>\n",
       "      <th>othservice</th>\n",
       "      <th>missind</th>\n",
       "      <th>ownership</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>b</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "      <td>6</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>17</td>\n",
       "      <td>...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>b</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>6</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>316</td>\n",
       "      <td>...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>b</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>6</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>19</td>\n",
       "      <td>...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>b</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>6</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>313</td>\n",
       "      <td>...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>b</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>22</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>313</td>\n",
       "      <td>...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Nonprofit</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 65 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "  id ad  education  ofjobs  yearsexp  honors  volunteer  military  empholes  \\\n",
       "0  b  1          4       2         6       0          0         0         1   \n",
       "1  b  1          3       3         6       0          1         1         0   \n",
       "2  b  1          4       1         6       0          0         0         0   \n",
       "3  b  1          3       4         6       0          1         0         1   \n",
       "4  b  1          3       3        22       0          0         0         0   \n",
       "\n",
       "   occupspecific    ...      compreq  orgreq  manuf  transcom  bankreal trade  \\\n",
       "0             17    ...          1.0     0.0    1.0       0.0       0.0   0.0   \n",
       "1            316    ...          1.0     0.0    1.0       0.0       0.0   0.0   \n",
       "2             19    ...          1.0     0.0    1.0       0.0       0.0   0.0   \n",
       "3            313    ...          1.0     0.0    1.0       0.0       0.0   0.0   \n",
       "4            313    ...          1.0     1.0    0.0       0.0       0.0   0.0   \n",
       "\n",
       "  busservice othservice  missind  ownership  \n",
       "0        0.0        0.0      0.0             \n",
       "1        0.0        0.0      0.0             \n",
       "2        0.0        0.0      0.0             \n",
       "3        0.0        0.0      0.0             \n",
       "4        0.0        1.0      0.0  Nonprofit  \n",
       "\n",
       "[5 rows x 65 columns]"
      ]
     },
     "execution_count": 165,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df5 = pd.io.stata.read_stata('data/race.dta')\n",
    "df5.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 167,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Black</th>\n",
       "      <th>White</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Failure</th>\n",
       "      <td>2278</td>\n",
       "      <td>2200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Success</th>\n",
       "      <td>157</td>\n",
       "      <td>235</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         Black  White\n",
       "Failure   2278   2200\n",
       "Success    157    235"
      ]
     },
     "execution_count": 167,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "callback_df = df5.pivot_table('id', ['call'], 'race', aggfunc='count')\n",
    "callback_df.index = ['Failure', 'Success']\n",
    "callback_df.columns = ['Black', 'White']\n",
    "callback_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Chi Square Test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 179,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "chi2, p, dof, expected = stats.chi2_contingency(callback_df, correction=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 180,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "16.879050414270221"
      ]
     },
     "execution_count": 180,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "chi2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 181,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3.983886837585076e-05"
      ]
     },
     "execution_count": 181,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "p"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The p-value obtained is much lesser than our significance level. Therefore, we reject the null hypothesis and accept the negating alternate hypothesis.\n",
    "\n",
    "**There is a relationship between race and callbacks.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Two Sample Significance Test\n",
    "\n",
    "Let us now do the same hypothesis testing using the two sample significance test for proportions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 182,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Black</th>\n",
       "      <th>White</th>\n",
       "      <th>Total</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Failure</th>\n",
       "      <td>2278</td>\n",
       "      <td>2200</td>\n",
       "      <td>4478</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Success</th>\n",
       "      <td>157</td>\n",
       "      <td>235</td>\n",
       "      <td>392</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Total</th>\n",
       "      <td>2435</td>\n",
       "      <td>2435</td>\n",
       "      <td>4870</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         Black  White  Total\n",
       "Failure   2278   2200   4478\n",
       "Success    157    235    392\n",
       "Total     2435   2435   4870"
      ]
     },
     "execution_count": 182,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "callback_df.loc['Total'] = [ callback_df['Black'].sum(), callback_df['White'].sum()]\n",
    "callback_df['Total'] = callback_df['Black'] + callback_df['White']\n",
    "callback_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 185,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.096509240246406572, 0.29528834517039093)"
      ]
     },
     "execution_count": 185,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "white_prop = callback_df.loc['Success', 'White']/callback_df.loc['Total', 'White']\n",
    "white_std = callback_df.loc['Success', 'White'] * ((1 - white_prop) ** 2) + callback_df.loc['Failure', 'White'] * ((0 - white_prop) ** 2)\n",
    "white_std = np.sqrt(white_std/callback_df.loc['Total', 'White'])\n",
    "\n",
    "white_prop, white_std"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 187,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.064476386036960986, 0.24559963697158382)"
      ]
     },
     "execution_count": 187,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "black_prop = callback_df.loc['Success', 'Black']/callback_df.loc['Total', 'Black']\n",
    "black_std = callback_df.loc['Success', 'Black'] * ((1 - black_prop) ** 2) + callback_df.loc['Failure', 'Black'] * ((0 - black_prop) ** 2)\n",
    "black_std = np.sqrt(black_std/callback_df.loc['Total', 'Black'])\n",
    "\n",
    "black_prop, black_std"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 188,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.032032854209445585, 0.0077968940361704568)"
      ]
     },
     "execution_count": 188,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "h0_prop = 0\n",
    "\n",
    "prop_diff = white_prop - black_prop\n",
    "p_hat = (callback_df.loc['Success', 'Black'] + callback_df.loc['Success', 'White'])/(callback_df.loc['Total', 'Black'] + callback_df.loc['Total', 'White']) \n",
    "var_diff = p_hat * (1- p_hat) * (1/callback_df.loc['Total', 'Black'] + 1/callback_df.loc['Total', 'White'])\n",
    "sigma_diff = np.sqrt(var_diff)\n",
    "\n",
    "prop_diff, sigma_diff"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 189,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4.1084121524343464"
      ]
     },
     "execution_count": 189,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "z = (prop_diff - h0_mean) / sigma_diff\n",
    "z"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 190,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3.9838868375774439e-05"
      ]
     },
     "execution_count": 190,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "p = (1-stats.norm.cdf(z))*2\n",
    "p"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We notice that the p-value arrived at is **identical** to what we got with the Chi Squared Independence test. Here again, obviously, we reject the null hypothesis and accept the alternate hypothesis. We can conclude that at a 0.1 significance level, there is a relationship between race and callbacks. \n",
    "\n",
    "When we are dealing with binary categorical variables, we have a choice between using the two aforementioned tests. We will arrive at (near) identical results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}