lab6.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Artificial Intelligence\n",
    "## L2 International, Univ. Bordeaux\n",
    "### Lab #4, Supervised Learning (2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Decision Trees\n",
    "A decision tree is a tree structure where an internal node represents feature, the branch represents a decision rule, and each leaf node represents the outcome. The topmost node in a decision tree is known as the root node. It learns to partition on the basis of the attribute value. It partitions the tree in recursively manner call recursive partitioning. This tree structure helps you in decision making. It's visualization like a flowchart diagram which easily mimics the human level thinking. That is why decision trees are easy to understand and interpret.\n",
    "\n",
    "For this lab, we will use pima-indians-diabetes dataset. It is well described in the following address:\n",
    "\n",
    "https://www.kaggle.com/kumargh/pimaindiansdiabetescsv\n",
    "\n",
    "As usual, we need to import necessary python modules:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Two additional modules for data visualization:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "from matplotlib import pyplot as plt\n",
    "import seaborn as sns\n",
    "sns.set()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And the, load the dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>pregnant</th>\n",
       "      <th>glucose</th>\n",
       "      <th>bp</th>\n",
       "      <th>skin</th>\n",
       "      <th>insulin</th>\n",
       "      <th>bml</th>\n",
       "      <th>pedigree</th>\n",
       "      <th>age</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>6</td>\n",
       "      <td>148</td>\n",
       "      <td>72</td>\n",
       "      <td>35</td>\n",
       "      <td>0</td>\n",
       "      <td>33.6</td>\n",
       "      <td>0.627</td>\n",
       "      <td>50</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>85</td>\n",
       "      <td>66</td>\n",
       "      <td>29</td>\n",
       "      <td>0</td>\n",
       "      <td>26.6</td>\n",
       "      <td>0.351</td>\n",
       "      <td>31</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>8</td>\n",
       "      <td>183</td>\n",
       "      <td>64</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>23.3</td>\n",
       "      <td>0.672</td>\n",
       "      <td>32</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>89</td>\n",
       "      <td>66</td>\n",
       "      <td>23</td>\n",
       "      <td>94</td>\n",
       "      <td>28.1</td>\n",
       "      <td>0.167</td>\n",
       "      <td>21</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>137</td>\n",
       "      <td>40</td>\n",
       "      <td>35</td>\n",
       "      <td>168</td>\n",
       "      <td>43.1</td>\n",
       "      <td>2.288</td>\n",
       "      <td>33</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   pregnant  glucose  bp  skin  insulin   bml  pedigree  age  label\n",
       "0         6      148  72    35        0  33.6     0.627   50      1\n",
       "1         1       85  66    29        0  26.6     0.351   31      0\n",
       "2         8      183  64     0        0  23.3     0.672   32      1\n",
       "3         1       89  66    23       94  28.1     0.167   21      0\n",
       "4         0      137  40    35      168  43.1     2.288   33      1"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "col_names = ['pregnant', 'glucose', 'bp', 'skin', 'insulin', 'bml', 'pedigree', 'age', 'label']\n",
    "pima = pd.read_csv('https://www.labri.fr/~zemmari/datasets/pima-indians-diabetes.csv',\n",
    "                  header=None, names=col_names)\n",
    "pima.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "feature_cols = ['pregnant', 'insulin', 'bml', 'age', 'glucose', 'bp', 'pedigree']\n",
    "X=pima[feature_cols]\n",
    "y=pima.label"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>pregnant</th>\n",
       "      <th>insulin</th>\n",
       "      <th>bml</th>\n",
       "      <th>age</th>\n",
       "      <th>glucose</th>\n",
       "      <th>bp</th>\n",
       "      <th>pedigree</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>6</td>\n",
       "      <td>0</td>\n",
       "      <td>33.6</td>\n",
       "      <td>50</td>\n",
       "      <td>148</td>\n",
       "      <td>72</td>\n",
       "      <td>0.627</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>26.6</td>\n",
       "      <td>31</td>\n",
       "      <td>85</td>\n",
       "      <td>66</td>\n",
       "      <td>0.351</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>8</td>\n",
       "      <td>0</td>\n",
       "      <td>23.3</td>\n",
       "      <td>32</td>\n",
       "      <td>183</td>\n",
       "      <td>64</td>\n",
       "      <td>0.672</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>94</td>\n",
       "      <td>28.1</td>\n",
       "      <td>21</td>\n",
       "      <td>89</td>\n",
       "      <td>66</td>\n",
       "      <td>0.167</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>168</td>\n",
       "      <td>43.1</td>\n",
       "      <td>33</td>\n",
       "      <td>137</td>\n",
       "      <td>40</td>\n",
       "      <td>2.288</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   pregnant  insulin   bml  age  glucose  bp  pedigree\n",
       "0         6        0  33.6   50      148  72     0.627\n",
       "1         1        0  26.6   31       85  66     0.351\n",
       "2         8        0  23.3   32      183  64     0.672\n",
       "3         1       94  28.1   21       89  66     0.167\n",
       "4         0      168  43.1   33      137  40     2.288"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 768 entries, 0 to 767\n",
      "Data columns (total 7 columns):\n",
      " #   Column    Non-Null Count  Dtype  \n",
      "---  ------    --------------  -----  \n",
      " 0   pregnant  768 non-null    int64  \n",
      " 1   insulin   768 non-null    int64  \n",
      " 2   bml       768 non-null    float64\n",
      " 3   age       768 non-null    int64  \n",
      " 4   glucose   768 non-null    int64  \n",
      " 5   bp        768 non-null    int64  \n",
      " 6   pedigree  768 non-null    float64\n",
      "dtypes: float64(2), int64(5)\n",
      "memory usage: 42.1 KB\n"
     ]
    }
   ],
   "source": [
    "X.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    1\n",
       "1    0\n",
       "2    1\n",
       "3    0\n",
       "4    1\n",
       "Name: label, dtype: int64"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we will split the dataset into two subsets: one for the training and the other for the test. For this, we will import the necessary function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=.3, random_state=109)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we can train our supervised learning model using Decision Tree classifier:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DecisionTreeClassifier(presort=False)"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn import tree\n",
    "from sklearn import metrics\n",
    "dt = tree.DecisionTreeClassifier(class_weight=None, criterion='gini', max_depth=None,\n",
    "                       max_features=None, max_leaf_nodes=None,\n",
    "                       min_impurity_decrease=0.0, min_impurity_split=None,\n",
    "                       min_samples_leaf=1, min_samples_split=2,\n",
    "                       min_weight_fraction_leaf=0.0, presort=False,\n",
    "                       random_state=None, splitter='best')\n",
    "dt.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "DecisionTreeClassifier(class_weight=None, criterion='gini', max_depth=None,\n",
    "                       max_features=None, max_leaf_nodes=None,\n",
    "                       min_impurity_decrease=0.0, min_impurity_split=None,\n",
    "                       min_samples_leaf=1, min_samples_split=2,\n",
    "                       min_weight_fraction_leaf=0.0, presort=False,\n",
    "                       random_state=None, splitter='best')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(231,)\n",
      "(231,)\n"
     ]
    }
   ],
   "source": [
    "y_pred = dt.predict(X_test)\n",
    "print(y_pred.shape)\n",
    "print(y_test.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[123  26]\n",
      " [ 44  38]]\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import confusion_matrix\n",
    "cm = confusion_matrix(y_test, y_pred)\n",
    "print(cm)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy:  69.70%\n"
     ]
    }
   ],
   "source": [
    "scores = metrics.accuracy_score(y_test, y_pred)\n",
    "print('Accuracy: ','{:2.2%}'.format(scores))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "ename": "ModuleNotFoundError",
     "evalue": "No module named 'pydotplus'",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mModuleNotFoundError\u001b[0m                       Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-35-a3698ee28b16>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m      3\u001b[0m \u001b[1;32mfrom\u001b[0m \u001b[0msix\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0mStringIO\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      4\u001b[0m \u001b[1;32mfrom\u001b[0m \u001b[0mIPython\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mdisplay\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0mImage\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 5\u001b[1;33m \u001b[1;32mimport\u001b[0m \u001b[0mpydotplus\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m      6\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      7\u001b[0m \u001b[0mdot_data\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mStringIO\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mModuleNotFoundError\u001b[0m: No module named 'pydotplus'"
     ]
    }
   ],
   "source": [
    "from sklearn.tree import export_graphviz\n",
    "#from sklearn.externals.six import StringIO  \n",
    "from six import StringIO\n",
    "from IPython.display import Image  \n",
    "import pydotplus\n",
    "\n",
    "dot_data = StringIO()\n",
    "export_graphviz(dt, out_file=dot_data,  \n",
    "                filled=True, rounded=True,\n",
    "                special_characters=True,feature_names = feature_cols,class_names=['0','1'])\n",
    "graph = pydotplus.graph_from_dot_data(dot_data.getvalue())  \n",
    "graph.write_png('diabetes.png')\n",
    "Image(graph.create_png())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Random Forests\n",
    "Random forests is a supervised learning algorithm. It is the most flexible and easy to use algorithm. A forest is comprised of trees. It is said that the more trees it has, the more robust a forest is. Random forests creates decision trees on randomly selected data samples, gets prediction from each tree and selects the best solution by means of voting. It also provides a pretty good indicator of the feature importance.\n",
    "\n",
    "In the previous section, we used a decision tree classifier and got an accuracy of 69.70%.\n",
    "\n",
    "Let's try to do better and use random forest classifier:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy:  66.67%\n"
     ]
    }
   ],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier\n",
    "rf = RandomForestClassifier(bootstrap=True, class_weight=None, criterion='gini',\n",
    "                       max_depth=None, max_features='auto', max_leaf_nodes=None,\n",
    "                       min_impurity_decrease=0.0, min_impurity_split=None,\n",
    "                       min_samples_leaf=1, min_samples_split=2,\n",
    "                       min_weight_fraction_leaf=0.0, n_estimators=10,\n",
    "                       n_jobs=None, oob_score=False, random_state=None,\n",
    "                       verbose=0, warm_start=False)\n",
    "rf.fit(X_train, y_train)\n",
    "y_pred = rf.predict(X_test)\n",
    "scores = metrics.accuracy_score(y_test, y_pred)\n",
    "print('Accuracy: ','{:2.2%}'.format(scores))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "RandomForestClassifier(n_estimators=10)"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy:  66.67%\n"
     ]
    }
   ],
   "source": [
    "rf = RandomForestClassifier(bootstrap=True, max_depth=None)\n",
    "rf.fit(X_train, y_train)\n",
    "y_pred = rf.predict(X_test)\n",
    "scores = metrics.accuracy_score(y_test, y_pred)\n",
    "print('Accuracy: ','{:2.2%}'.format(scores))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'n_estimators': [200, 400, 600, 800, 1000, 1200, 1400, 1600, 1800, 2000], 'max_features': ['auto', 'sqrt'], 'max_depth': [10, 20, 30, 40, 50, 60, 70, 80, 90, 100, 110, None], 'min_samples_split': [2, 5, 10], 'min_samples_leaf': [1, 2, 4], 'bootstrap': [True, False]}\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "from sklearn.model_selection import RandomizedSearchCV\n",
    "# Number of trees in random forest\n",
    "n_estimators = [int(x) for x in np.linspace(start = 200, stop = 2000, num = 10)]\n",
    "# Number of features to consider at every split\n",
    "max_features = ['auto', 'sqrt']\n",
    "# Maximum number of levels in tree\n",
    "max_depth = [int(x) for x in np.linspace(10, 110, num = 11)]\n",
    "max_depth.append(None)\n",
    "# Minimum number of samples required to split a node\n",
    "min_samples_split = [2, 5, 10]\n",
    "# Minimum number of samples required at each leaf node\n",
    "min_samples_leaf = [1, 2, 4]\n",
    "# Method of selecting samples for training each tree\n",
    "bootstrap = [True, False]\n",
    "# Create the random grid\n",
    "random_grid = {'n_estimators': n_estimators,\n",
    "               'max_features': max_features,\n",
    "               'max_depth': max_depth,\n",
    "               'min_samples_split': min_samples_split,\n",
    "               'min_samples_leaf': min_samples_leaf,\n",
    "               'bootstrap': bootstrap}\n",
    "print(random_grid)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}