lab8.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Artificial Intelligence\n",
    "## L2 International, Univ. Bordeaux\n",
    "\n",
    "### Lab #5, Supervised Learning (4)\n",
    "In this lab, we will see how to build a neural network with the Keras framework in Python. As usual, we give an incomplete python program. Some instructions will be used as they are and you do not need to change them. We will comment some of them but we do not need to understand them all. We will use the dataset MNIST, one of the most used datasets in machine learning research. This dataset consists of 70000 images of handwritten digits (of size 28x28).\n",
    "\n",
    "### Artificial neural networks\n",
    "Recall that an artificial neural network (ANN for short) consists of a set of layers disposed linearly, and each layer is a set of (artificial) neurons. The model is simplified so that signals can only circulate from the bottom layer to the top layer. Each neuron in the $k$-th layer of the neural network is connected to all the neurons in the $(k−1)$-th layer, and the neurons in a given layer are all independent from each other.\n",
    "\n",
    "A neural network consists of the following components\n",
    "\n",
    "<ul>\n",
    "    <li>An input layer, $x$</li>\n",
    "    <li>An arbitrary amount of hidden layers</li>\n",
    "    <li>An output layer, $y$</li>\n",
    "    <li>A set of weights and biases between each layer, $w$ and $b$</li>\n",
    "    <li>A choice of activation function for each hidden layer, $σ$.</li>\n",
    "</ul>\n",
    "\n",
    "As usual, we need to import necessary python modules:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "from keras.datasets import mnist\n",
    "(X_train, y_train), (X_test, y_test) = mnist.load_data()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.datasets import mnist\n",
    "(X_train, y_train), (X_test, y_test) = mnist.load_data()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X_train shape:  (60000, 28, 28)\n",
      "Y_train shape:  (60000,)\n"
     ]
    }
   ],
   "source": [
    "print('X_train shape: ', X_train.shape)\n",
    "print('Y_train shape: ', y_train.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAD4CAYAAAAq5pAIAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAAAOKUlEQVR4nO3de4xc9XnG8efBXhtiTGJDbYxjgYusFhQVU1aGlBYRQahDLpAEKiwaUcmNuaaxSi8IWoU/GslNEyhqEpAdKKYCojSJi6WgEOIQ0fSCWBxjG5zEXBwwdm1SJwESbK/x2z/2UC2w5zfrubPv9yOtZua8c+a8Gu/jMzu/c87PESEAE99hvW4AQHcQdiAJwg4kQdiBJAg7kMTkbm5siqfG4ZrWzU0CqezVr7Q/9nmsWktht71Y0i2SJkn6SkSsKD3/cE3T6T6nlU0CKHgk1tXWmv4Yb3uSpC9J+oCkkyUtsX1ys68HoLNa+Zt9kaSnIuKZiNgv6auSLmhPWwDarZWwz5X0/KjH26tlb2B7me0h20PD2tfC5gC0opWwj/UlwFuOvY2IlRExGBGDA5rawuYAtKKVsG+XNG/U43dL2tFaOwA6pZWwPyppge35tqdIukTS2va0BaDdmh56i4gDtq+R9IBGht7uiIgn2tYZgLZqaZw9Iu6XdH+begHQQRwuCyRB2IEkCDuQBGEHkiDsQBKEHUiCsANJEHYgCcIOJEHYgSQIO5AEYQeSIOxAEoQdSIKwA0kQdiAJwg4kQdiBJAg7kARhB5Ig7EAShB1IgrADSRB2IAnCDiRB2IEkCDuQBGEHkiDsQBItzeKKie+whScX6z9efkSxvvX9q2prk1ze1/z64P5i/b2fX16sH3fb+trawb17i+tORC2F3fY2SS9Lek3SgYgYbEdTANqvHXv290XEz9rwOgA6iL/ZgSRaDXtI+o7tx2wvG+sJtpfZHrI9NKx9LW4OQLNa/Rh/ZkTssD1L0oO2fxQRD49+QkSslLRSko7yzGhxewCa1NKePSJ2VLe7Ja2RtKgdTQFov6bDbnua7emv35d0nqTN7WoMQHu18jF+tqQ1tl9/nXsi4ttt6Qpt48nlf+Idf1b+MPaVT91SrJ82ZdIh9/S6/9h7sFg/Y2q59/V/+cVi/UMPXVpffHxLcd2JqOmwR8Qzkk5pYy8AOoihNyAJwg4kQdiBJAg7kARhB5LgFNcJYPdVv1db+8XC4eK6T32wPHwllYfW3rf548X6wVWzamvTf/TL4ronr/5Jsf65Y4eK9aNv3Vlbe7H+LZuw2LMDSRB2IAnCDiRB2IEkCDuQBGEHkiDsQBKMs78NPP835UHhx6/8p9raYXJx3Q37DxTrf7X0ymL9iIfqL9csSYpna0vlE1ylLefOKD+hwdUT/vn4dbW18xZfUVx3yrcfLb/42xB7diAJwg4kQdiBJAg7kARhB5Ig7EAShB1IgnH2PjBpRnk8efml/1asl8bSd7726+K6f3HF8mJ9yvfK54x3Urz6arH+5V/ML9avelf9GH+UDz+YkNizA0kQdiAJwg4kQdiBJAg7kARhB5Ig7EASjLP3Ac94Z7G+9KjtTb/2WfddW6wveOCRpl+70w7u3Vus3/Xs6cX6VafWj7Nn1HDPbvsO27ttbx61bKbtB21vrW4bXGUAQK+N52P8nZIWv2nZdZLWRcQCSeuqxwD6WMOwR8TDkva8afEFklZX91dLurC9bQFot2a/oJsdETslqbqtndDL9jLbQ7aHhrWvyc0BaFXHv42PiJURMRgRgwOa2unNAajRbNh32Z4jSdXt7va1BKATmg37WkmXVfcvk3Rfe9oB0CkNx9lt3yvpbEnH2N4u6TOSVkj6mu2lkp6TdHEnm5zohue8q6X1Xyics/5bq8pzoDe6djsmjoZhj4glNaVz2twLgA7icFkgCcIOJEHYgSQIO5AEYQeS4BTXPvD0RYe3tP55/10/rfLxGze19NqYONizA0kQdiAJwg4kQdiBJAg7kARhB5Ig7EASjLN3weS5xxXrt3749pZef9IPp7e0fr867B3vKNY/+9trutTJxMCeHUiCsANJEHYgCcIOJEHYgSQIO5AEYQeSYJy9C351ytxi/ZwjWpsWa+rPo6X1+5Unl389G71v/3vw1drawCsHmurp7Yw9O5AEYQeSIOxAEoQdSIKwA0kQdiAJwg4kwTj7BDD77s21tcxTMq/+5e/U1g779x92sZP+0HDPbvsO27ttbx617EbbL9jeUP2c39k2AbRqPB/j75S0eIzlN0fEwurn/va2BaDdGoY9Ih6WtKcLvQDooFa+oLvG9sbqY/6MuifZXmZ7yPbQsFo7BhxA85oN+62STpS0UNJOSV+oe2JErIyIwYgYHNDUJjcHoFVNhT0idkXEaxFxUNIqSYva2xaAdmsq7LbnjHr4UUn1Yz8A+kLDcXbb90o6W9IxtrdL+oyks20vlBSStkm6vHMtIqufXv2eBs/4frF6z21/WFubpf889Ibe5hqGPSKWjLG4tVkNAHQdh8sCSRB2IAnCDiRB2IEkCDuQBKe4dsHh6zYW63e/PKtYv3T67na20zcmzz++WP/Sn97W0usf960Xamv5LiTNnh1Ig7ADSRB2IAnCDiRB2IEkCDuQBGEHkmCcvQtiX/lyXHtjSpc66S+7zj2uWP+Dw8uj4fuiwWh5TMyprJvFnh1IgrADSRB2IAnCDiRB2IEkCDuQBGEHkmCcfSI4cV59bcOT3etjDJOPr+/tY5/6XnHdRuPo7/2H5cX6sdvyXS66hD07kARhB5Ig7EAShB1IgrADSRB2IAnCDiTBOHsf+PsHPlKsL734y8X605e8s7Y2f0MzHY2fJ5d/hZ684dja2tqj7yuu+/29RxTrx97COPqhaLhntz3P9kO2t9h+wvanq+UzbT9oe2t1O6Pz7QJo1ng+xh+QdG1EnCTpDElX2z5Z0nWS1kXEAknrqscA+lTDsEfEzohYX91/WdIWSXMlXSBpdfW01ZIu7FCPANrgkL6gs32CpFMlPSJpdkTslEb+Q5A05oRltpfZHrI9NKzytdgAdM64w277SEnfkLQ8Il4a73oRsTIiBiNicEBTm+kRQBuMK+y2BzQS9Lsj4pvV4l2251T1OZIm5lSjwATRcOjNtiXdLmlLRNw0qrRW0mWSVlS35XEU1Jqx2eUnXFwu/93H7qmtrf7HM4rrHvifXeUXb2DXFYuK9ac++MXa2qb9w8V1P3v5J4v1AT1WrOONxjPOfqakT0jaZHtDtex6jYT8a7aXSnpODX8lAfRSw7BHxA8k1e16zmlvOwA6hcNlgSQIO5AEYQeSIOxAEoQdSIJTXPvA7G89W6xvuKF8SeWPT/t5be26vz2huO5JKwaK9a1XFS5TLenrS24q1qX66agv+vry4ponfve/Grw2DgV7diAJwg4kQdiBJAg7kARhB5Ig7EAShB1IwhHRtY0d5ZlxujlR7lANn3tasb7mzvpzxo90+epAj+1/rVg/pX6YXJI0WZOK9bM2XVRbm/6h54rrxoHy8QV4q0dinV6KPWOepcqeHUiCsANJEHYgCcIOJEHYgSQIO5AEYQeS4Hz2t4GB75avj77ozj+vrf3rH99cXPe0KQ0G0htYsObKYv2kFdtrawcYR+8q9uxAEoQdSIKwA0kQdiAJwg4kQdiBJAg7kETD89ltz5N0l6RjJR2UtDIibrF9o6RPSnqxeur1EXF/6bU4nx3orNL57OM5qOaApGsjYr3t6ZIes/1gVbs5Ij7frkYBdM545mffKWlndf9l21skze10YwDa65D+Zrd9gqRTJT1SLbrG9kbbd9ieUbPOMttDtoeGta+1bgE0bdxht32kpG9IWh4RL0m6VdKJkhZqZM//hbHWi4iVETEYEYMDKl8PDUDnjCvstgc0EvS7I+KbkhQRuyLitYg4KGmVpEWdaxNAqxqG3bYl3S5pS0TcNGr5nFFP+6ikze1vD0C7jOfb+DMlfULSJtsbqmXXS1pie6GkkLRN0uUd6A9Am4zn2/gfSBpr3K44pg6gv3AEHZAEYQeSIOxAEoQdSIKwA0kQdiAJwg4kQdiBJAg7kARhB5Ig7EAShB1IgrADSRB2IImGl5Ju68bsFyX9dNSiYyT9rGsNHJp+7a1f+5LorVnt7O34iPiNsQpdDftbNm4PRcRgzxoo6Nfe+rUvid6a1a3e+BgPJEHYgSR6HfaVPd5+Sb/21q99SfTWrK701tO/2QF0T6/37AC6hLADSfQk7LYX2/6x7adsX9eLHurY3mZ7k+0Ntod63Msdtnfb3jxq2UzbD9reWt2OOcdej3q70fYL1Xu3wfb5Peptnu2HbG+x/YTtT1fLe/reFfrqyvvW9b/ZbU+S9BNJ75e0XdKjkpZExJNdbaSG7W2SBiOi5wdg2D5L0iuS7oqI91TLPidpT0SsqP6jnBERf90nvd0o6ZVeT+NdzVY0Z/Q045IulPQn6uF7V+jrj9SF960Xe/ZFkp6KiGciYr+kr0q6oAd99L2IeFjSnjctvkDS6ur+ao38snRdTW99ISJ2RsT66v7Lkl6fZryn712hr67oRdjnSnp+1OPt6q/53kPSd2w/ZntZr5sZw+yI2CmN/PJImtXjft6s4TTe3fSmacb75r1rZvrzVvUi7GNNJdVP439nRsTvSvqApKurj6sYn3FN490tY0wz3heanf68Vb0I+3ZJ80Y9frekHT3oY0wRsaO63S1pjfpvKupdr8+gW93u7nE//6+fpvEea5px9cF718vpz3sR9kclLbA93/YUSZdIWtuDPt7C9rTqixPZnibpPPXfVNRrJV1W3b9M0n097OUN+mUa77ppxtXj967n059HRNd/JJ2vkW/kn5Z0Qy96qOnrNyU9Xv080eveJN2rkY91wxr5RLRU0tGS1knaWt3O7KPe/kXSJkkbNRKsOT3q7fc18qfhRkkbqp/ze/3eFfrqyvvG4bJAEhxBByRB2IEkCDuQBGEHkiDsQBKEHUiCsANJ/B8y5CnV98nfjQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\n"
     ]
    }
   ],
   "source": [
    "from matplotlib import pyplot as plt\n",
    "%matplotlib inline\n",
    "plt.imshow(X_train[1000])\n",
    "plt.show()\n",
    "\n",
    "print(y_train[1000])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(60000, 784)\n"
     ]
    }
   ],
   "source": [
    "num_pixels = X_train.shape[1] * X_train.shape[2]\n",
    "X_train = X_train.reshape(X_train.shape[0], num_pixels)\n",
    "X_test = X_test.reshape(X_test.shape[0], num_pixels)\n",
    "\n",
    "y_train = y_train.reshape(y_train.shape[0], 1)\n",
    "y_test = y_test.reshape(y_test.shape[0], 1)\n",
    "print(X_train.shape)\n",
    "# 에러나기 전부분까지는 실행된 채로 변수에 저장됨"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train = X_train.astype('float32')\n",
    "X_test = X_test.astype('float32')\n",
    "y_train = y_train.astype('float32')\n",
    "y_test = y_test.astype('float32')\n",
    "X_train  = X_train / 255\n",
    "X_test  = X_test / 255"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.models import Sequential\n",
    "from keras.layers import Dense\n",
    "from keras.optimizers import adam"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "import keras\n",
    "import keras.utils\n",
    "from keras import utils as np_utils"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_train = keras.utils.to_categorical(y_train)\n",
    "y_test = keras.utils.to_categorical(y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "num_classes = y_train.shape[1]\n",
    "def neural_network():\n",
    "    model = Sequential()\n",
    "    model.add(Dense(num_pixels, input_dim = num_pixels, kernel_initializer = 'normal', activation = 'relu'))\n",
    "    model.add(Dense(num_classes, kernel_initializer = 'normal', activation='softmax'))\n",
    "    model.compile(loss = 'categorical_crossentropy', optimizer = 'adam', metrics = ['accuracy'])\n",
    "    return model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_1\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "dense_1 (Dense)              (None, 784)               615440    \n",
      "_________________________________________________________________\n",
      "dense_2 (Dense)              (None, 10)                7850      \n",
      "=================================================================\n",
      "Total params: 623,290\n",
      "Trainable params: 623,290\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model = neural_network()\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 60000 samples, validate on 10000 samples\n",
      "Epoch 1/1\n",
      "60000/60000 [==============================] - 5s 78us/step - loss: 0.2291 - accuracy: 0.9346 - val_loss: 0.1171 - val_accuracy: 0.9659\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.callbacks.History at 0x2a940ca39c8>"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(X_train, y_train, validation_data=(X_test, y_test), epochs=1, batch_size=100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10000/10000 [==============================] - 0s 42us/step\n",
      "Neural network accuracy: 96.59%\n"
     ]
    }
   ],
   "source": [
    "scores = model.evaluate(X_test, y_test)\n",
    "print('Neural network accuracy: %.2f%%' % (scores[1]*100))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}