3_timeseries_features_model.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Random Forest with TS-Fresh Feature Selection\n",
    "\n",
    "The tsfresh python package is designed specifically for selecting features from time series data. With the help of having the best, most useful features, a random forest classifier achieves **79% accuracy** in predicting which students will ultimately succeed or fail based on the data from the first quarter of course timelines. Importantly, it also achieves **87% precision on class 0 (failing students)**. Depending on the nature of the interjection that this model would support, it may be worth it in a real-world setting to sacrifice precision in order to improve recall on class 0. (For example, if the interjection was polite and optional, it would probably be more important to reach students in danger of failing than to not bother students that are actually fine.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 213,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tsfresh import extract_features, select_features, extract_relevant_features\n",
    "from tsfresh.utilities.dataframe_functions import impute\n",
    "from tsfresh.feature_extraction.settings import MinimalFCParameters, ComprehensiveFCParameters\n",
    "from tsfresh.transformers import RelevantFeatureAugmenter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import classification_report\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.pipeline import Pipeline\n",
    "import xgboost as xgb"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "stu_vle = pd.read_csv('data/stvl_ccc14b.csv')\n",
    "stu_as = pd.read_csv('data/stas_ccc14b.csv')\n",
    "ass = pd.read_csv('data/ass_ccc14b.csv')\n",
    "vle = pd.read_csv('data/vle_ccc14b.csv')\n",
    "stu_info = pd.read_csv('data/stuinfo_ccc14b.csv')\n",
    "stu_reg = pd.read_csv('data/stureg_ccc14b.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Preprocessing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## NOTE: This is a demonstration of the preprocessing that needs to be done for using ts-fresh transformer. All of this preprocessing in encapsulated in the ts_preprocess function in src.util, for easier use of the pipeline in the future."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Drop early withdrawal students"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or rather, determine which students to keep."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>code_module</th>\n",
       "      <th>code_presentation</th>\n",
       "      <th>id_student</th>\n",
       "      <th>date_registration</th>\n",
       "      <th>date_unregistration</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>CCC</td>\n",
       "      <td>2014B</td>\n",
       "      <td>28418</td>\n",
       "      <td>-37.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>CCC</td>\n",
       "      <td>2014B</td>\n",
       "      <td>29764</td>\n",
       "      <td>-34.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>CCC</td>\n",
       "      <td>2014B</td>\n",
       "      <td>29820</td>\n",
       "      <td>-57.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>CCC</td>\n",
       "      <td>2014B</td>\n",
       "      <td>40333</td>\n",
       "      <td>-30.0</td>\n",
       "      <td>17.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>CCC</td>\n",
       "      <td>2014B</td>\n",
       "      <td>40604</td>\n",
       "      <td>-17.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  code_module code_presentation  id_student  date_registration  \\\n",
       "0         CCC             2014B       28418              -37.0   \n",
       "1         CCC             2014B       29764              -34.0   \n",
       "2         CCC             2014B       29820              -57.0   \n",
       "3         CCC             2014B       40333              -30.0   \n",
       "4         CCC             2014B       40604              -17.0   \n",
       "\n",
       "   date_unregistration  \n",
       "0                  NaN  \n",
       "1                  NaN  \n",
       "2                  NaN  \n",
       "3                 17.0  \n",
       "4                  NaN  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stu_reg.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "pop_of_interest = stu_reg.drop(stu_reg[stu_reg.date_unregistration <= 67].index)['id_student'].unique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1300"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(stu_vle.drop(stu_vle[~stu_vle.id_student.isin(pop_of_interest)].index)['id_student'].unique()) # num students left in course who clicked"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "clkd_pop_of_interest = stu_vle.drop(stu_vle[~stu_vle.id_student.isin(pop_of_interest)].index)['id_student'].unique()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Make y (targets) column"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>code_module</th>\n",
       "      <th>code_presentation</th>\n",
       "      <th>id_student</th>\n",
       "      <th>gender</th>\n",
       "      <th>region</th>\n",
       "      <th>highest_education</th>\n",
       "      <th>imd_band</th>\n",
       "      <th>age_band</th>\n",
       "      <th>num_of_prev_attempts</th>\n",
       "      <th>studied_credits</th>\n",
       "      <th>disability</th>\n",
       "      <th>final_result</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>CCC</td>\n",
       "      <td>2014B</td>\n",
       "      <td>28418</td>\n",
       "      <td>F</td>\n",
       "      <td>West Midlands Region</td>\n",
       "      <td>A Level or Equivalent</td>\n",
       "      <td>20-30%</td>\n",
       "      <td>0-35</td>\n",
       "      <td>0</td>\n",
       "      <td>30</td>\n",
       "      <td>N</td>\n",
       "      <td>Fail</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>CCC</td>\n",
       "      <td>2014B</td>\n",
       "      <td>29764</td>\n",
       "      <td>M</td>\n",
       "      <td>East Anglian Region</td>\n",
       "      <td>A Level or Equivalent</td>\n",
       "      <td>50-60%</td>\n",
       "      <td>0-35</td>\n",
       "      <td>0</td>\n",
       "      <td>90</td>\n",
       "      <td>N</td>\n",
       "      <td>Distinction</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>CCC</td>\n",
       "      <td>2014B</td>\n",
       "      <td>29820</td>\n",
       "      <td>M</td>\n",
       "      <td>East Anglian Region</td>\n",
       "      <td>HE Qualification</td>\n",
       "      <td>40-50%</td>\n",
       "      <td>0-35</td>\n",
       "      <td>0</td>\n",
       "      <td>60</td>\n",
       "      <td>N</td>\n",
       "      <td>Pass</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>CCC</td>\n",
       "      <td>2014B</td>\n",
       "      <td>40333</td>\n",
       "      <td>M</td>\n",
       "      <td>North Region</td>\n",
       "      <td>HE Qualification</td>\n",
       "      <td>0-10%</td>\n",
       "      <td>35-55</td>\n",
       "      <td>0</td>\n",
       "      <td>30</td>\n",
       "      <td>N</td>\n",
       "      <td>Withdrawn</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>CCC</td>\n",
       "      <td>2014B</td>\n",
       "      <td>40604</td>\n",
       "      <td>M</td>\n",
       "      <td>Ireland</td>\n",
       "      <td>A Level or Equivalent</td>\n",
       "      <td>NaN</td>\n",
       "      <td>35-55</td>\n",
       "      <td>0</td>\n",
       "      <td>30</td>\n",
       "      <td>N</td>\n",
       "      <td>Pass</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  code_module code_presentation  id_student gender                region  \\\n",
       "0         CCC             2014B       28418      F  West Midlands Region   \n",
       "1         CCC             2014B       29764      M   East Anglian Region   \n",
       "2         CCC             2014B       29820      M   East Anglian Region   \n",
       "3         CCC             2014B       40333      M          North Region   \n",
       "4         CCC             2014B       40604      M               Ireland   \n",
       "\n",
       "       highest_education imd_band age_band  num_of_prev_attempts  \\\n",
       "0  A Level or Equivalent   20-30%     0-35                     0   \n",
       "1  A Level or Equivalent   50-60%     0-35                     0   \n",
       "2       HE Qualification   40-50%     0-35                     0   \n",
       "3       HE Qualification    0-10%    35-55                     0   \n",
       "4  A Level or Equivalent      NaN    35-55                     0   \n",
       "\n",
       "   studied_credits disability final_result  \n",
       "0               30          N         Fail  \n",
       "1               90          N  Distinction  \n",
       "2               60          N         Pass  \n",
       "3               30          N    Withdrawn  \n",
       "4               30          N         Pass  "
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stu_info.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_col = stu_info.drop(stu_info[~stu_info.id_student.isin(clkd_pop_of_interest)].index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_col = y_col.drop(['code_module', 'code_presentation', 'gender', 'region', 'highest_education', 'imd_band', 'age_band', 'num_of_prev_attempts', 'studied_credits', 'disability'], axis=1)\n",
    "y_col.final_result.replace(to_replace=dict(Pass=1, Distinction=1, Fail=0, Withdrawn=0), inplace=True)\n",
    "y_col = y_col.sort_values(by=['id_student'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id_student</th>\n",
       "      <th>final_result</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>28418</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>29764</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>29820</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>40604</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>42638</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   id_student  final_result\n",
       "0       28418             0\n",
       "1       29764             1\n",
       "2       29820             1\n",
       "4       40604             1\n",
       "5       42638             1"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_col.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_col = y_col.set_index('id_student')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [],
   "source": [
    "y = y_col['final_result']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train-test split BEFORE feature-extraction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Very important or else this would perform significantly worse in production."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_train, y_test = train_test_split(y, test_size=0.2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1040, 260)"
      ]
     },
     "execution_count": 88,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(y_train), len(y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Make students click stream data (from which to generate features)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_train = stu_vle.loc[stu_vle.id_student.isin(y_train.index)]\n",
    "df_test = stu_vle.loc[stu_vle.id_student.isin(y_test.index)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_train = df_train.drop(['code_module', 'code_presentation', 'id_site'], axis=1)\n",
    "df_train = df_train.groupby(['id_student', 'date']).sum() #sum daily clicks per student\n",
    "df_train = df_train.reset_index()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_test = df_test.drop(['code_module', 'code_presentation', 'id_site'], axis=1)\n",
    "df_test = df_test.groupby(['id_student', 'date']).sum() #sum daily clicks per student\n",
    "df_test = df_test.reset_index()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((66932, 3), (15815, 3))"
      ]
     },
     "execution_count": 99,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_train.shape, df_test.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Make X input for tsfresh feature augmenter (just student IDs of each group)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train = pd.DataFrame(index=y_train.index)\n",
    "X_test = pd.DataFrame(index=y_test.index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>id_student</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>147675</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>601949</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>251385</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2016517</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>383158</th>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Empty DataFrame\n",
       "Columns: []\n",
       "Index: [147675, 601949, 251385, 2016517, 383158]"
      ]
     },
     "execution_count": 103,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Create and run pipeline with feature extraction and classifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This all may look strange but setting the parameters like this is precisely how the ts docs outline, for the use case of extracting features on the train set, and then only using those features when evaluating the test set. (that is, NOT extracting features from the test set) (see docs at https://github.com/blue-yonder/tsfresh/blob/master/notebooks/pipeline_with_two_datasets.ipynb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {},
   "outputs": [],
   "source": [
    "ppl = Pipeline([('fresh', RelevantFeatureAugmenter(column_id='id_student', column_sort='date', \n",
    "                                                   default_fc_parameters=MinimalFCParameters())),\n",
    "                ('clf', RandomForestClassifier())])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Feature Extraction: 100%|██████████| 20/20 [00:00<00:00, 97.19it/s]\n",
      "WARNING:tsfresh.feature_selection.relevance:Infered classification as machine learning task\n",
      "Feature Extraction: 100%|██████████| 20/20 [00:00<00:00, 257.30it/s]\n",
      "//anaconda3/lib/python3.7/site-packages/sklearn/ensemble/forest.py:245: FutureWarning: The default value of n_estimators will change from 10 in version 0.20 to 100 in 0.22.\n",
      "  \"10 in version 0.20 to 100 in 0.22.\", FutureWarning)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Pipeline(memory=None,\n",
       "         steps=[('fresh',\n",
       "                 RelevantFeatureAugmenter(chunksize=None,\n",
       "                                          column_id='id_student',\n",
       "                                          column_kind=None, column_sort='date',\n",
       "                                          column_value=None,\n",
       "                                          default_fc_parameters={'length': None,\n",
       "                                                                 'maximum': None,\n",
       "                                                                 'mean': None,\n",
       "                                                                 'median': None,\n",
       "                                                                 'minimum': None,\n",
       "                                                                 'standard_deviation': None,\n",
       "                                                                 'sum_values': None,\n",
       "                                                                 'variance': None},\n",
       "                                          disable_progressbar=False,\n",
       "                                          fdr_level=0.05,\n",
       "                                          f...\n",
       "                 RandomForestClassifier(bootstrap=True, class_weight=None,\n",
       "                                        criterion='gini', max_depth=None,\n",
       "                                        max_features='auto',\n",
       "                                        max_leaf_nodes=None,\n",
       "                                        min_impurity_decrease=0.0,\n",
       "                                        min_impurity_split=None,\n",
       "                                        min_samples_leaf=1, min_samples_split=2,\n",
       "                                        min_weight_fraction_leaf=0.0,\n",
       "                                        n_estimators=10, n_jobs=None,\n",
       "                                        oob_score=False, random_state=None,\n",
       "                                        verbose=0, warm_start=False))],\n",
       "         verbose=False)"
      ]
     },
     "execution_count": 105,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# for the fit on the train test set, we set the fresh__timeseries_container to `df_train`\n",
    "ppl.set_params(fresh__timeseries_container=df_train)\n",
    "ppl.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Feature Extraction: 100%|██████████| 20/20 [00:00<00:00, 782.50it/s]\n"
     ]
    }
   ],
   "source": [
    "# for the predict on the test test set, we set the fresh__timeseries_container to `df_test`\n",
    "ppl.set_params(fresh__timeseries_container=df_test)\n",
    "y_pred = ppl.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.79      0.74      0.76       131\n",
      "           1       0.75      0.80      0.77       129\n",
      "\n",
      "    accuracy                           0.77       260\n",
      "   macro avg       0.77      0.77      0.77       260\n",
      "weighted avg       0.77      0.77      0.77       260\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(classification_report(y_test, y_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Again but with diffs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>date</th>\n",
       "      <th>-18</th>\n",
       "      <th>-17</th>\n",
       "      <th>-16</th>\n",
       "      <th>-15</th>\n",
       "      <th>-14</th>\n",
       "      <th>-13</th>\n",
       "      <th>-12</th>\n",
       "      <th>-11</th>\n",
       "      <th>-10</th>\n",
       "      <th>-9</th>\n",
       "      <th>...</th>\n",
       "      <th>232</th>\n",
       "      <th>233</th>\n",
       "      <th>234</th>\n",
       "      <th>235</th>\n",
       "      <th>236</th>\n",
       "      <th>237</th>\n",
       "      <th>238</th>\n",
       "      <th>239</th>\n",
       "      <th>240</th>\n",
       "      <th>241</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>id_student</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>28418</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29820</th>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>40604</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>9.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>24.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>42638</th>\n",
       "      <td>20.0</td>\n",
       "      <td>22.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>5.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>7.0</td>\n",
       "      <td>32.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>5.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>18.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45664</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 258 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "date        -18   -17   -16   -15   -14   -13   -12   -11   -10   -9    ...   \\\n",
       "id_student                                                              ...    \n",
       "28418        NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN  ...    \n",
       "29820        2.0   NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN  ...    \n",
       "40604        NaN   NaN   4.0   9.0   NaN   NaN   9.0   NaN  24.0  10.0  ...    \n",
       "42638       20.0  22.0   7.0   9.0   NaN   5.0   NaN   5.0   2.0   3.0  ...    \n",
       "45664        NaN   NaN   NaN   NaN   NaN   NaN   NaN   2.0   1.0   NaN  ...    \n",
       "\n",
       "date         232   233   234   235   236   237   238   239   240   241  \n",
       "id_student                                                              \n",
       "28418        NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN  \n",
       "29820        NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN  \n",
       "40604        NaN   NaN   NaN   NaN   1.0   NaN   NaN   NaN   NaN   NaN  \n",
       "42638        NaN   7.0  32.0   9.0   NaN   5.0   NaN  18.0   9.0   NaN  \n",
       "45664        NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN  \n",
       "\n",
       "[5 rows x 258 columns]"
      ]
     },
     "execution_count": 112,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_trainpiv = df_train.pivot(index='id_student', columns='date', values='sum_click')\n",
    "df_trainpiv.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_trainpiv = df_trainpiv.fillna(0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_trpivdif = df_trainpiv.diff(axis=1)\n",
    "df_trpivdif.drop(-18, axis=1, inplace=True) #drop first column which becomes all NaNs after diff"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>date</th>\n",
       "      <th>-17</th>\n",
       "      <th>-16</th>\n",
       "      <th>-15</th>\n",
       "      <th>-14</th>\n",
       "      <th>-13</th>\n",
       "      <th>-12</th>\n",
       "      <th>-11</th>\n",
       "      <th>-10</th>\n",
       "      <th>-9</th>\n",
       "      <th>-8</th>\n",
       "      <th>...</th>\n",
       "      <th>232</th>\n",
       "      <th>233</th>\n",
       "      <th>234</th>\n",
       "      <th>235</th>\n",
       "      <th>236</th>\n",
       "      <th>237</th>\n",
       "      <th>238</th>\n",
       "      <th>239</th>\n",
       "      <th>240</th>\n",
       "      <th>241</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>id_student</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>28418</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29820</th>\n",
       "      <td>-2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>40604</th>\n",
       "      <td>0.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>-9.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>-9.0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>-14.0</td>\n",
       "      <td>-3.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>42638</th>\n",
       "      <td>2.0</td>\n",
       "      <td>-15.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>-9.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>-5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>-3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>-3.0</td>\n",
       "      <td>...</td>\n",
       "      <td>-6.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>25.0</td>\n",
       "      <td>-23.0</td>\n",
       "      <td>-9.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>-5.0</td>\n",
       "      <td>18.0</td>\n",
       "      <td>-9.0</td>\n",
       "      <td>-9.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45664</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 257 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "date        -17   -16   -15   -14   -13   -12   -11   -10   -9    -8    ...   \\\n",
       "id_student                                                              ...    \n",
       "28418        0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0  ...    \n",
       "29820       -2.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0  ...    \n",
       "40604        0.0   4.0   5.0  -9.0   0.0   9.0  -9.0  24.0 -14.0  -3.0  ...    \n",
       "42638        2.0 -15.0   2.0  -9.0   5.0  -5.0   5.0  -3.0   1.0  -3.0  ...    \n",
       "45664        0.0   0.0   0.0   0.0   0.0   0.0   2.0  -1.0  -1.0   0.0  ...    \n",
       "\n",
       "date         232   233   234   235   236   237   238   239   240   241  \n",
       "id_student                                                              \n",
       "28418        0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0  \n",
       "29820        0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0  \n",
       "40604        0.0   0.0   0.0   0.0   1.0  -1.0   0.0   0.0   0.0   0.0  \n",
       "42638       -6.0   7.0  25.0 -23.0  -9.0   5.0  -5.0  18.0  -9.0  -9.0  \n",
       "45664        0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0  \n",
       "\n",
       "[5 rows x 257 columns]"
      ]
     },
     "execution_count": 117,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_trpivdif.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 130,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_trpivdif = df_trpivdif.stack() #back to tsfresh format"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 133,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_trpivdif = df_trpivdif.reset_index()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 134,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_testpiv = df_test.pivot(index='id_student', columns='date', values='sum_click')\n",
    "df_testpiv = df_testpiv.fillna(0)\n",
    "df_tspivdif = df_testpiv.diff(axis=1)\n",
    "df_tspivdif.drop(-18, axis=1, inplace=True) #drop first column which becomes all NaNs after diff\n",
    "df_tspivdif = df_tspivdif.stack() #back to tsfresh format\n",
    "df_tspivdif = df_tspivdif.reset_index()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 135,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id_student</th>\n",
       "      <th>date</th>\n",
       "      <th>0</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>29764</td>\n",
       "      <td>-17</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>29764</td>\n",
       "      <td>-16</td>\n",
       "      <td>-14.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>29764</td>\n",
       "      <td>-15</td>\n",
       "      <td>-1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>29764</td>\n",
       "      <td>-14</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>29764</td>\n",
       "      <td>-13</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   id_student  date     0\n",
       "0       29764   -17   2.0\n",
       "1       29764   -16 -14.0\n",
       "2       29764   -15  -1.0\n",
       "3       29764   -14   0.0\n",
       "4       29764   -13   0.0"
      ]
     },
     "execution_count": 135,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_tspivdif.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 136,
   "metadata": {},
   "outputs": [],
   "source": [
    "ppl2 = Pipeline([('fresh', RelevantFeatureAugmenter(column_id='id_student', column_sort='date', \n",
    "                                                   default_fc_parameters=MinimalFCParameters())),\n",
    "                ('clf', RandomForestClassifier())])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 137,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Feature Extraction: 100%|██████████| 20/20 [00:00<00:00, 209.00it/s]\n",
      "WARNING:tsfresh.feature_selection.relevance:Infered classification as machine learning task\n",
      "WARNING:tsfresh.feature_selection.relevance:[test_feature_significance] Feature 0__length is constant\n",
      "WARNING:tsfresh.feature_selection.significance_tests:[target_binary_feature_binary_test] A binary feature should have only values 1 and 0 (incl. True and False). Instead found {0.0, -1.0} in feature ''0__median''.\n",
      "WARNING:tsfresh.feature_selection.significance_tests:[target_binary_feature_binary_test] A binary feature should have only values 1 and 0 (incl. True and False). Instead found {0.0, -1.0} in feature ''0__median''.\n",
      "Feature Extraction: 100%|██████████| 20/20 [00:00<00:00, 229.00it/s]\n",
      "//anaconda3/lib/python3.7/site-packages/sklearn/ensemble/forest.py:245: FutureWarning: The default value of n_estimators will change from 10 in version 0.20 to 100 in 0.22.\n",
      "  \"10 in version 0.20 to 100 in 0.22.\", FutureWarning)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Pipeline(memory=None,\n",
       "         steps=[('fresh',\n",
       "                 RelevantFeatureAugmenter(chunksize=None,\n",
       "                                          column_id='id_student',\n",
       "                                          column_kind=None, column_sort='date',\n",
       "                                          column_value=None,\n",
       "                                          default_fc_parameters={'length': None,\n",
       "                                                                 'maximum': None,\n",
       "                                                                 'mean': None,\n",
       "                                                                 'median': None,\n",
       "                                                                 'minimum': None,\n",
       "                                                                 'standard_deviation': None,\n",
       "                                                                 'sum_values': None,\n",
       "                                                                 'variance': None},\n",
       "                                          disable_progressbar=False,\n",
       "                                          fdr_level=0.05,\n",
       "                                          f...\n",
       "                 RandomForestClassifier(bootstrap=True, class_weight=None,\n",
       "                                        criterion='gini', max_depth=None,\n",
       "                                        max_features='auto',\n",
       "                                        max_leaf_nodes=None,\n",
       "                                        min_impurity_decrease=0.0,\n",
       "                                        min_impurity_split=None,\n",
       "                                        min_samples_leaf=1, min_samples_split=2,\n",
       "                                        min_weight_fraction_leaf=0.0,\n",
       "                                        n_estimators=10, n_jobs=None,\n",
       "                                        oob_score=False, random_state=None,\n",
       "                                        verbose=0, warm_start=False))],\n",
       "         verbose=False)"
      ]
     },
     "execution_count": 137,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# for the fit on the train test set, we set the fresh__timeseries_container to `df_train`\n",
    "ppl2.set_params(fresh__timeseries_container=df_trpivdif)\n",
    "ppl2.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 138,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Feature Extraction: 100%|██████████| 20/20 [00:00<00:00, 794.44it/s]\n"
     ]
    }
   ],
   "source": [
    "# for the predict on the test test set, we set the fresh__timeseries_container to `df_test`\n",
    "ppl2.set_params(fresh__timeseries_container=df_tspivdif)\n",
    "y_pred2 = ppl2.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 139,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.74      0.73      0.74       131\n",
      "           1       0.73      0.74      0.74       129\n",
      "\n",
      "    accuracy                           0.74       260\n",
      "   macro avg       0.74      0.74      0.74       260\n",
      "weighted avg       0.74      0.74      0.74       260\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(classification_report(y_test, y_pred2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Much worse with diffs this time!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Parameter play"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 214,
   "metadata": {},
   "outputs": [],
   "source": [
    "ppl3 = Pipeline([('fresh', RelevantFeatureAugmenter(column_id='id_student', column_sort='date', \n",
    "                                                   default_fc_parameters=ComprehensiveFCParameters())),\n",
    "                ('clf', RandomForestClassifier(n_estimators = 200,\n",
    "                                max_depth=5,\n",
    "                                random_state = 21))])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 215,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Feature Extraction: 100%|██████████| 20/20 [00:39<00:00,  1.37s/it]\n",
      "WARNING:tsfresh.feature_selection.relevance:Infered classification as machine learning task\n",
      "WARNING:tsfresh.feature_selection.relevance:[test_feature_significance] Feature sum_click__change_quantiles__f_agg_\"mean\"__isabs_False__qh_0.2__ql_0.2 is constant\n",
      "WARNING:tsfresh.feature_selection.relevance:[test_feature_significance] Feature sum_click__change_quantiles__f_agg_\"mean\"__isabs_True__qh_0.2__ql_0.2 is constant\n",
      "WARNING:tsfresh.feature_selection.relevance:[test_feature_significance] Feature sum_click__change_quantiles__f_agg_\"mean\"__isabs_False__qh_0.2__ql_0.4 is constant\n",
      "WARNING:tsfresh.feature_selection.relevance:[test_feature_significance] Feature sum_click__change_quantiles__f_agg_\"mean\"__isabs_True__qh_0.2__ql_0.4 is constant\n",
      "WARNING:tsfresh.feature_selection.relevance:[test_feature_significance] Feature sum_click__change_quantiles__f_agg_\"mean\"__isabs_True__qh_0.2__ql_0.6 is constant\n",
      "WARNING:tsfresh.feature_selection.relevance:[test_feature_significance] Feature sum_click__change_quantiles__f_agg_\"mean\"__isabs_False__qh_0.2__ql_0.6 is constant\n",
      "WARNING:tsfresh.feature_selection.relevance:[test_feature_significance] Feature sum_click__change_quantiles__f_agg_\"mean\"__isabs_False__qh_0.2__ql_0.8 is constant\n",
      "WARNING:tsfresh.feature_selection.relevance:[test_feature_significance] Feature sum_click__change_quantiles__f_agg_\"mean\"__isabs_True__qh_0.2__ql_0.8 is constant\n",
      "WARNING:tsfresh.feature_selection.relevance:[test_feature_significance] Feature sum_click__change_quantiles__f_agg_\"mean\"__isabs_True__qh_0.4__ql_0.4 is constant\n",
      "WARNING:tsfresh.feature_selection.relevance:[test_feature_significance] Feature sum_click__change_quantiles__f_agg_\"mean\"__isabs_False__qh_0.4__ql_0.4 is constant\n",
      "WARNING:tsfresh.feature_selection.relevance:[test_feature_significance] Feature sum_click__change_quantiles__f_agg_\"mean\"__isabs_True__qh_0.4__ql_0.6 is constant\n",
      "WARNING:tsfresh.feature_selection.relevance:[test_feature_significance] Feature sum_click__change_quantiles__f_agg_\"mean\"__isabs_False__qh_0.4__ql_0.6 is constant\n",
      "WARNING:tsfresh.feature_selection.relevance:[test_feature_significance] Feature sum_click__change_quantiles__f_agg_\"var\"__isabs_True__qh_0.2__ql_0.2 is constant\n",
      "WARNING:tsfresh.feature_selection.relevance:[test_feature_significance] Feature sum_click__change_quantiles__f_agg_\"mean\"__isabs_True__qh_0.4__ql_0.8 is constant\n",
      "WARNING:tsfresh.feature_selection.relevance:[test_feature_significance] Feature sum_click__change_quantiles__f_agg_\"mean\"__isabs_False__qh_0.4__ql_0.8 is constant\n",
      "WARNING:tsfresh.feature_selection.relevance:[test_feature_significance] Feature sum_click__change_quantiles__f_agg_\"var\"__isabs_True__qh_0.2__ql_0.4 is constant\n",
      "WARNING:tsfresh.feature_selection.relevance:[test_feature_significance] Feature sum_click__change_quantiles__f_agg_\"var\"__isabs_True__qh_0.2__ql_0.6 is constant\n",
      "WARNING:tsfresh.feature_selection.relevance:[test_feature_significance] Feature sum_click__change_quantiles__f_agg_\"mean\"__isabs_True__qh_0.6__ql_0.6 is constant\n",
      "WARNING:tsfresh.feature_selection.relevance:[test_feature_significance] Feature sum_click__change_quantiles__f_agg_\"mean\"__isabs_False__qh_0.6__ql_0.6 is constant\n",
      "WARNING:tsfresh.feature_selection.relevance:[test_feature_significance] Feature sum_click__change_quantiles__f_agg_\"mean\"__isabs_True__qh_0.6__ql_0.8 is constant\n",
      "WARNING:tsfresh.feature_selection.relevance:[test_feature_significance] Feature sum_click__change_quantiles__f_agg_\"mean\"__isabs_False__qh_0.6__ql_0.8 is constant\n",
      "WARNING:tsfresh.feature_selection.relevance:[test_feature_significance] Feature sum_click__change_quantiles__f_agg_\"var\"__isabs_True__qh_0.2__ql_0.8 is constant\n",
      "WARNING:tsfresh.feature_selection.relevance:[test_feature_significance] Feature sum_click__change_quantiles__f_agg_\"var\"__isabs_True__qh_0.4__ql_0.4 is constant\n",
      "WARNING:tsfresh.feature_selection.relevance:[test_feature_significance] Feature sum_click__change_quantiles__f_agg_\"mean\"__isabs_False__qh_0.8__ql_0.8 is constant\n",
      "WARNING:tsfresh.feature_selection.relevance:[test_feature_significance] Feature sum_click__change_quantiles__f_agg_\"mean\"__isabs_True__qh_0.8__ql_0.8 is constant\n",
      "WARNING:tsfresh.feature_selection.relevance:[test_feature_significance] Feature sum_click__change_quantiles__f_agg_\"var\"__isabs_True__qh_0.4__ql_0.6 is constant\n",
      "WARNING:tsfresh.feature_selection.relevance:[test_feature_significance] Feature sum_click__change_quantiles__f_agg_\"var\"__isabs_True__qh_0.4__ql_0.8 is constant\n",
      "WARNING:tsfresh.feature_selection.relevance:[test_feature_significance] Feature sum_click__change_quantiles__f_agg_\"var\"__isabs_False__qh_0.2__ql_0.4 is constant\n",
      "WARNING:tsfresh.feature_selection.relevance:[test_feature_significance] Feature sum_click__change_quantiles__f_agg_\"var\"__isabs_False__qh_0.2__ql_0.2 is constant\n",
      "WARNING:tsfresh.feature_selection.relevance:[test_feature_significance] Feature sum_click__change_quantiles__f_agg_\"var\"__isabs_True__qh_0.6__ql_0.6 is constant\n",
      "WARNING:tsfresh.feature_selection.relevance:[test_feature_significance] Feature sum_click__change_quantiles__f_agg_\"var\"__isabs_True__qh_0.6__ql_0.8 is constant\n",
      "WARNING:tsfresh.feature_selection.relevance:[test_feature_significance] Feature sum_click__change_quantiles__f_agg_\"var\"__isabs_False__qh_0.2__ql_0.6 is constant\n",
      "WARNING:tsfresh.feature_selection.relevance:[test_feature_significance] Feature sum_click__fft_coefficient__coeff_0__attr_\"angle\" is constant\n",
      "WARNING:tsfresh.feature_selection.relevance:[test_feature_significance] Feature sum_click__fft_coefficient__coeff_0__attr_\"imag\" is constant\n",
      "WARNING:tsfresh.feature_selection.relevance:[test_feature_significance] Feature sum_click__change_quantiles__f_agg_\"var\"__isabs_False__qh_0.2__ql_0.8 is constant\n",
      "WARNING:tsfresh.feature_selection.relevance:[test_feature_significance] Feature sum_click__change_quantiles__f_agg_\"var\"__isabs_True__qh_0.8__ql_0.8 is constant\n",
      "WARNING:tsfresh.feature_selection.relevance:[test_feature_significance] Feature sum_click__change_quantiles__f_agg_\"var\"__isabs_False__qh_0.4__ql_0.4 is constant\n",
      "WARNING:tsfresh.feature_selection.relevance:[test_feature_significance] Feature sum_click__change_quantiles__f_agg_\"var\"__isabs_False__qh_0.4__ql_0.6 is constant\n",
      "WARNING:tsfresh.feature_selection.relevance:[test_feature_significance] Feature sum_click__change_quantiles__f_agg_\"var\"__isabs_False__qh_0.4__ql_0.8 is constant\n",
      "WARNING:tsfresh.feature_selection.relevance:[test_feature_significance] Feature sum_click__change_quantiles__f_agg_\"var\"__isabs_False__qh_0.6__ql_0.6 is constant\n",
      "WARNING:tsfresh.feature_selection.relevance:[test_feature_significance] Feature sum_click__change_quantiles__f_agg_\"var\"__isabs_False__qh_0.6__ql_0.8 is constant\n",
      "WARNING:tsfresh.feature_selection.relevance:[test_feature_significance] Feature sum_click__change_quantiles__f_agg_\"var\"__isabs_False__qh_0.8__ql_0.8 is constant\n",
      "WARNING:tsfresh.feature_selection.relevance:[test_feature_significance] Feature sum_click__large_standard_deviation__r_0.5 is constant\n",
      "WARNING:tsfresh.feature_selection.relevance:[test_feature_significance] Feature sum_click__large_standard_deviation__r_0.55 is constant\n",
      "WARNING:tsfresh.feature_selection.relevance:[test_feature_significance] Feature sum_click__number_crossing_m__m_-1 is constant\n",
      "WARNING:tsfresh.feature_selection.relevance:[test_feature_significance] Feature sum_click__large_standard_deviation__r_0.6000000000000001 is constant\n",
      "WARNING:tsfresh.feature_selection.relevance:[test_feature_significance] Feature sum_click__number_crossing_m__m_0 is constant\n",
      "WARNING:tsfresh.feature_selection.relevance:[test_feature_significance] Feature sum_click__large_standard_deviation__r_0.65 is constant\n",
      "WARNING:tsfresh.feature_selection.relevance:[test_feature_significance] Feature sum_click__large_standard_deviation__r_0.7000000000000001 is constant\n",
      "WARNING:tsfresh.feature_selection.relevance:[test_feature_significance] Feature sum_click__large_standard_deviation__r_0.75 is constant\n",
      "WARNING:tsfresh.feature_selection.relevance:[test_feature_significance] Feature sum_click__partial_autocorrelation__lag_0 is constant\n",
      "WARNING:tsfresh.feature_selection.relevance:[test_feature_significance] Feature sum_click__symmetry_looking__r_0.0 is constant\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:tsfresh.feature_selection.relevance:[test_feature_significance] Feature sum_click__large_standard_deviation__r_0.8 is constant\n",
      "WARNING:tsfresh.feature_selection.relevance:[test_feature_significance] Feature sum_click__large_standard_deviation__r_0.8500000000000001 is constant\n",
      "WARNING:tsfresh.feature_selection.relevance:[test_feature_significance] Feature sum_click__large_standard_deviation__r_0.9 is constant\n",
      "WARNING:tsfresh.feature_selection.relevance:[test_feature_significance] Feature sum_click__range_count__max_0__min_1000000000000.0 is constant\n",
      "WARNING:tsfresh.feature_selection.relevance:[test_feature_significance] Feature sum_click__value_count__value_-1 is constant\n",
      "WARNING:tsfresh.feature_selection.relevance:[test_feature_significance] Feature sum_click__large_standard_deviation__r_0.9500000000000001 is constant\n",
      "WARNING:tsfresh.feature_selection.relevance:[test_feature_significance] Feature sum_click__range_count__max_1__min_-1 is constant\n",
      "WARNING:tsfresh.feature_selection.relevance:[test_feature_significance] Feature sum_click__value_count__value_0 is constant\n",
      "Feature Extraction: 100%|██████████| 20/20 [00:26<00:00,  1.01it/s]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Pipeline(memory=None,\n",
       "         steps=[('fresh',\n",
       "                 RelevantFeatureAugmenter(chunksize=None,\n",
       "                                          column_id='id_student',\n",
       "                                          column_kind=None, column_sort='date',\n",
       "                                          column_value=None,\n",
       "                                          default_fc_parameters={'abs_energy': None,\n",
       "                                                                 'absolute_sum_of_changes': None,\n",
       "                                                                 'agg_autocorrelation': [{'f_agg': 'mean',\n",
       "                                                                                          'maxlag': 40},\n",
       "                                                                                         {'f_agg': 'median',\n",
       "                                                                                          'maxlag': 40},\n",
       "                                                                                         {'f_agg': 'var',\n",
       "                                                                                          'maxlag': 40}],\n",
       "                                                                 'agg_linear_...\n",
       "                ('clf',\n",
       "                 RandomForestClassifier(bootstrap=True, class_weight=None,\n",
       "                                        criterion='gini', max_depth=5,\n",
       "                                        max_features='auto',\n",
       "                                        max_leaf_nodes=None,\n",
       "                                        min_impurity_decrease=0.0,\n",
       "                                        min_impurity_split=None,\n",
       "                                        min_samples_leaf=1, min_samples_split=2,\n",
       "                                        min_weight_fraction_leaf=0.0,\n",
       "                                        n_estimators=200, n_jobs=None,\n",
       "                                        oob_score=False, random_state=21,\n",
       "                                        verbose=0, warm_start=False))],\n",
       "         verbose=False)"
      ]
     },
     "execution_count": 215,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# for the fit on the train test set, we set the fresh__timeseries_container to `df_train`\n",
    "ppl3.set_params(fresh__timeseries_container=df_train)\n",
    "ppl3.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 216,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Feature Extraction: 100%|██████████| 20/20 [00:05<00:00,  4.54it/s]\n"
     ]
    }
   ],
   "source": [
    "# for the predict on the test test set, we set the fresh__timeseries_container to `df_test`\n",
    "ppl3.set_params(fresh__timeseries_container=df_test)\n",
    "y_pred3 = ppl3.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 217,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.87      0.69      0.77       131\n",
      "           1       0.74      0.89      0.81       129\n",
      "\n",
      "    accuracy                           0.79       260\n",
      "   macro avg       0.80      0.79      0.79       260\n",
      "weighted avg       0.80      0.79      0.79       260\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(classification_report(y_test, y_pred3))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**That's more like it.** This model achieved 79% accuracy, and 87% precision on class 0 (failing students). Depending on the nature of the interjection that this model would support, it may be worth it in a real-world setting to sacrifice precision to improve recall on class 0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 210,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.391     , 0.31416356, 0.11127336, 0.05598614, 0.04973959,\n",
       "       0.04213917, 0.01146875, 0.02422944])"
      ]
     },
     "execution_count": 210,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ppl3.named_steps['clf'].feature_importances_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 201,
   "metadata": {},
   "outputs": [],
   "source": [
    "ppl4 = Pipeline([('fresh', RelevantFeatureAugmenter(column_id='id_student', column_sort='date', \n",
    "                                                   default_fc_parameters=MinimalFCParameters())),\n",
    "                ('clf', xgb.XGBClassifier(n_estimators    = 300, \n",
    "                              max_depth    =  5,\n",
    "                              random_state = 21))])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 202,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Feature Extraction: 100%|██████████| 20/20 [00:00<00:00, 241.92it/s]\n",
      "WARNING:tsfresh.feature_selection.relevance:Infered classification as machine learning task\n",
      "Feature Extraction: 100%|██████████| 20/20 [00:00<00:00, 257.60it/s]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Pipeline(memory=None,\n",
       "         steps=[('fresh',\n",
       "                 RelevantFeatureAugmenter(chunksize=None,\n",
       "                                          column_id='id_student',\n",
       "                                          column_kind=None, column_sort='date',\n",
       "                                          column_value=None,\n",
       "                                          default_fc_parameters={'length': None,\n",
       "                                                                 'maximum': None,\n",
       "                                                                 'mean': None,\n",
       "                                                                 'median': None,\n",
       "                                                                 'minimum': None,\n",
       "                                                                 'standard_deviation': None,\n",
       "                                                                 'sum_values': None,\n",
       "                                                                 'variance': None},\n",
       "                                          disable_progressbar=False,\n",
       "                                          fdr_level=0.05,\n",
       "                                          f...\n",
       "                 XGBClassifier(base_score=0.5, booster='gbtree',\n",
       "                               colsample_bylevel=1, colsample_bynode=1,\n",
       "                               colsample_bytree=1, gamma=0, learning_rate=0.1,\n",
       "                               max_delta_step=0, max_depth=5,\n",
       "                               min_child_weight=1, missing=None,\n",
       "                               n_estimators=300, n_jobs=1, nthread=None,\n",
       "                               objective='binary:logistic', random_state=21,\n",
       "                               reg_alpha=0, reg_lambda=1, scale_pos_weight=1,\n",
       "                               seed=None, silent=None, subsample=1,\n",
       "                               verbosity=1))],\n",
       "         verbose=False)"
      ]
     },
     "execution_count": 202,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# for the fit on the train test set, we set the fresh__timeseries_container to `df_train`\n",
    "ppl4.set_params(fresh__timeseries_container=df_train)\n",
    "ppl4.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 203,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Feature Extraction: 100%|██████████| 20/20 [00:00<00:00, 8312.96it/s]\n"
     ]
    }
   ],
   "source": [
    "# for the predict on the test test set, we set the fresh__timeseries_container to `df_test`\n",
    "ppl4.set_params(fresh__timeseries_container=df_test)\n",
    "y_pred4 = ppl4.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 204,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.82      0.73      0.77       131\n",
      "           1       0.75      0.84      0.79       129\n",
      "\n",
      "    accuracy                           0.78       260\n",
      "   macro avg       0.78      0.78      0.78       260\n",
      "weighted avg       0.78      0.78      0.78       260\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(classification_report(y_test, y_pred4))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}