Lab+1+Solutions.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lab 1 - Hello Spark\n",
    "This lab will introduce you to Apache Spark.  It will be written in Python and run in IBM's Data Science Experience environment through a Jupyter notebook.  While you work, it will be valuable to reference the [Apache Spark Documentation](http://spark.apache.org/docs/latest/programming-guide.html).  Since it is Python, be careful of whitespace!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 1 - Working with Spark Context\n",
    "### Step 1.1 - Invoke the spark context: <i>version</i> will return the working version of Apache Spark<br><br>\n",
    " <div class=\"panel-group\" id=\"accordion-11\">\n",
    "  <div class=\"panel panel-default\">\n",
    "    <div class=\"panel-heading\">\n",
    "      <h4 class=\"panel-title\">\n",
    "        <a data-toggle=\"collapse\" data-parent=\"#accordion-11\" href=\"#collapse1-11\">\n",
    "        Hint 1</a>\n",
    "      </h4>\n",
    "    </div>\n",
    "    <div id=\"collapse1-11\" class=\"panel-collapse collapse\">\n",
    "      <div class=\"panel-body\">The spark context is automatically set in a Jupyter notebook.   It is called: sc</div>\n",
    "    </div>\n",
    "  </div>\n",
    "  <div class=\"panel panel-default\">\n",
    "    <div class=\"panel-heading\">\n",
    "      <h4 class=\"panel-title\">\n",
    "        <a data-toggle=\"collapse\" data-parent=\"#accordion-11\" href=\"#collapse2-11\">\n",
    "        Hint 2</a>\n",
    "      </h4>\n",
    "    </div>\n",
    "    <div id=\"collapse2-11\" class=\"panel-collapse collapse\">\n",
    "      <div class=\"panel-body\">Type:<br>&nbsp;&nbsp;&nbsp;&nbsp;sc.version</div>\n",
    "    </div>\n",
    "  </div>\n",
    "  <div class=\"panel panel-default\">\n",
    "    <div class=\"panel-heading\">\n",
    "      <h4 class=\"panel-title\">\n",
    "        <a data-toggle=\"collapse\" data-parent=\"#accordion-11\" href=\"#collapse3-11\">\n",
    "        Optional</a>\n",
    "      </h4>\n",
    "    </div>\n",
    "    <div id=\"collapse3-11\" class=\"panel-collapse collapse\">\n",
    "      <div class=\"panel-body\">Jupyter notebooks have command completion which can be invoked via the TAB key.<br>Type:<br>&nbsp;&nbsp;&nbsp;&nbsp;<i>sc.&lt;TAB&gt;</i><br>to see all the possible options within the Spark context</div>\n",
    "    </div>\n",
    "  </div>\n",
    "</div> "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "u'2.0.1'"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Step 1 - Check spark version\n",
    "sc.version"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2 - Working with Resilient Distributed Datasets (RDD)\n",
    "\n",
    "### Step 2.1 - Create an RDD with numbers 1 to 10\n",
    "\n",
    "RDDs are the basic abstraction unit in Spark.   An RDD represents an immutable, partitioned, fault-tolerant collection of elements that can be operated on in parallel.<br>\n",
    "There are three ways to create an RDD: parallelizing an existing collection, referencing a dataset in an external storage system which offers a Hadoop InputFormat -- or transforming an existing RDD.<br>\n",
    "<br>\n",
    "Create an iterable or collection in your program with numbers 1 to 10 and then invoke the Spark Context's (sc) <i>parallelize()</i> method on it.<br>\n",
    "\n",
    " <div class=\"panel-group\" id=\"accordion-21\">\n",
    "  <div class=\"panel panel-default\">\n",
    "    <div class=\"panel-heading\">\n",
    "      <h4 class=\"panel-title\">\n",
    "        <a data-toggle=\"collapse\" data-parent=\"#accordion-21\" href=\"#collapse1-21\">\n",
    "        Hint 1</a>\n",
    "      </h4>\n",
    "    </div>\n",
    "    <div id=\"collapse1-21\" class=\"panel-collapse collapse\">\n",
    "      <div class=\"panel-body\">Type:<br>\n",
    "&nbsp;&nbsp;&nbsp;&nbsp;\n",
    "x = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]<br><br>\n",
    "Or we can try to be a little clever by typing:<br>\n",
    "&nbsp;&nbsp;&nbsp;&nbsp;\n",
    "x = range(1, 11)\n",
    "      </div>\n",
    "    </div>\n",
    "  </div>\n",
    "  <div class=\"panel panel-default\">\n",
    "    <div class=\"panel-heading\">\n",
    "      <h4 class=\"panel-title\">\n",
    "        <a data-toggle=\"collapse\" data-parent=\"#accordion-21\" href=\"#collapse2-21\">\n",
    "        Hint 2</a>\n",
    "      </h4>\n",
    "    </div>\n",
    "    <div id=\"collapse2-21\" class=\"panel-collapse collapse\">\n",
    "      <div class=\"panel-body\">Type:<br>\n",
    "&nbsp;&nbsp;&nbsp;&nbsp;\n",
    "x = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]<br>\n",
    "&nbsp;&nbsp;&nbsp;&nbsp; x_nbr_rdd = sc.parallelize(x)\n",
    "      </div>\n",
    "    </div>\n",
    "  </div>\n",
    "  <div class=\"panel panel-default\">\n",
    "    <div class=\"panel-heading\">\n",
    "      <h4 class=\"panel-title\">\n",
    "        <a data-toggle=\"collapse\" data-parent=\"#accordion-21\" href=\"#collapse3-21\">\n",
    "        Optional Advanced</a>\n",
    "      </h4>\n",
    "    </div>\n",
    "    <div id=\"collapse3-21\" class=\"panel-collapse collapse\">\n",
    "      <div class=\"panel-body\">An optional parameter to parallelize is the number of partitions to cut the dataset into.   Spark will run one task for each partition.   Typically you want 2-4 partitions for each CPU.   Normally, Spark will set it automatically, but you can control this by specifying it manually as a second parameter to the parallelize method.<br><br>\n",
    "You can obtain the partitions size by calling <i>&lt;RDD&gt;.getNumPartitions()</i><br>\n",
    "Try experimenting with different partitions sizes -- including ones higher than the number of values.   To see how the values are distributed use:<br><br>\n",
    "<i>\n",
    "def f(iterator):<br>\n",
    "    &nbsp;&nbsp;&nbsp;&nbsp;\n",
    "    count = 0<br>\n",
    "    &nbsp;&nbsp;&nbsp;&nbsp;\n",
    "    for value in iterator:<br>\n",
    "    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;\n",
    "        count = count + 1<br>\n",
    "    &nbsp;&nbsp;&nbsp;&nbsp;\n",
    "    yield count<br>\n",
    "x_nbr_rdd.mapPartitions(f).collect()</i><br>\n",
    "      </div>\n",
    "    </div>\n",
    "  </div>\n",
    "</div> "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#Step 2.1 - Create RDD of numbers 1-10\n",
    "x = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]\n",
    "x_nbr_rdd = sc.parallelize(x) \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 2.2 - Return the first element<br><br>\n",
    " <div class=\"panel-group\" id=\"accordion-22\">\n",
    "  <div class=\"panel panel-default\">\n",
    "    <div class=\"panel-heading\">\n",
    "      <h4 class=\"panel-title\">\n",
    "        <a data-toggle=\"collapse\" data-parent=\"#accordion-22\" href=\"#collapse1-22\">\n",
    "        Hint 1</a>\n",
    "      </h4>\n",
    "    </div>\n",
    "    <div id=\"collapse1-22\" class=\"panel-collapse collapse\">\n",
    "      <div class=\"panel-body\">Use the <i>first()</i> method on the RDD to return the first element in an RDD.   You could also use the <i>take()</i> method with a parameter of 1.   first() and take(1) are equivalent.   Both will take the first element in the RDD's 0th partition.</div>\n",
    "    </div>\n",
    "  </div>\n",
    "  <div class=\"panel panel-default\">\n",
    "    <div class=\"panel-heading\">\n",
    "      <h4 class=\"panel-title\">\n",
    "        <a data-toggle=\"collapse\" data-parent=\"#accordion-22\" href=\"#collapse2-22\">\n",
    "        Hint 2</a>\n",
    "      </h4>\n",
    "    </div>\n",
    "    <div id=\"collapse2-22\" class=\"panel-collapse collapse\">\n",
    "      <div class=\"panel-body\">Type: <br>\n",
    "&nbsp;&nbsp;&nbsp;&nbsp;x_nbr_rdd.first()</div>\n",
    "    </div>\n",
    "  </div>\n",
    "</div> "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Step 2.2 - Return first element\n",
    "x_nbr_rdd.first()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 2.3 - Return an array of the first five elements<br><br>\n",
    " <div class=\"panel-group\" id=\"accordion-23\">\n",
    "  <div class=\"panel panel-default\">\n",
    "    <div class=\"panel-heading\">\n",
    "      <h4 class=\"panel-title\">\n",
    "        <a data-toggle=\"collapse\" data-parent=\"#accordion-23\" href=\"#collapse1-23\">\n",
    "        Hint 1</a>\n",
    "      </h4>\n",
    "    </div>\n",
    "    <div id=\"collapse1-23\" class=\"panel-collapse collapse\">\n",
    "      <div class=\"panel-body\">Use the <i>take()</i> method</div>\n",
    "    </div>\n",
    "  </div>\n",
    "  <div class=\"panel panel-default\">\n",
    "    <div class=\"panel-heading\">\n",
    "      <h4 class=\"panel-title\">\n",
    "        <a data-toggle=\"collapse\" data-parent=\"#accordion-23\" href=\"#collapse2-23\">\n",
    "        Hint 2</a>\n",
    "      </h4>\n",
    "    </div>\n",
    "    <div id=\"collapse2-23\" class=\"panel-collapse collapse\">\n",
    "      <div class=\"panel-body\">Type:<br>\n",
    "&nbsp;&nbsp;&nbsp;&nbsp;x_nbr_rdd.take(5)</div>\n",
    "    </div>\n",
    "  </div>\n",
    "  <div class=\"panel panel-default\">\n",
    "    <div class=\"panel-heading\">\n",
    "      <h4 class=\"panel-title\">\n",
    "        <a data-toggle=\"collapse\" data-parent=\"#accordion-23\" href=\"#collapse3-23\">\n",
    "        Optional Advanced</a>\n",
    "      </h4>\n",
    "    </div>\n",
    "    <div id=\"collapse3-23\" class=\"panel-collapse collapse\">\n",
    "      <div class=\"panel-body\">How would you get the 5th-7th elements?   <i>take()</i> only accepts one parameter so <i>take(5,7)</i> will not work.<br>\n",
    "      </div>\n",
    "    </div>\n",
    "  </div>\n",
    "</div> \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[1, 2, 3, 4, 5]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Step 2.3 - Return an array of the first five elements\n",
    "x_nbr_rdd.take(5)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 2.4 - Perform a map transformation to increment each element of the array by 1.  The map function creates a new RDD by applying the function provided in the argument to each element.  For more information go to [Transformations](http://spark.apache.org/docs/latest/programming-guide.html#transformations)<br><br>\n",
    " <div class=\"panel-group\" id=\"accordion-24\">\n",
    "  <div class=\"panel panel-default\">\n",
    "    <div class=\"panel-heading\">\n",
    "      <h4 class=\"panel-title\">\n",
    "        <a data-toggle=\"collapse\" data-parent=\"#accordion-24\" href=\"#collapse1-24\">\n",
    "        Hint 1</a>\n",
    "      </h4>\n",
    "    </div>\n",
    "    <div id=\"collapse1-24\" class=\"panel-collapse collapse\">\n",
    "      <div class=\"panel-body\">Use the <i>map(func)</i> function on the RDD.   Map invokes function <i>func</i> on each element of the RDD.   You can also use a inline (or lambda) function.   The syntax for a lambda function is:<br>\n",
    "&nbsp;&nbsp;&nbsp;&nbsp;\n",
    "lambda &lt;var&gt;: &lt;myCode&gt;\n",
    "</div>\n",
    "    </div>\n",
    "  </div>\n",
    "  <div class=\"panel panel-default\">\n",
    "    <div class=\"panel-heading\">\n",
    "      <h4 class=\"panel-title\">\n",
    "        <a data-toggle=\"collapse\" data-parent=\"#accordion-24\" href=\"#collapse2-24\">\n",
    "        Hint 2</a>\n",
    "      </h4>\n",
    "    </div>\n",
    "    <div id=\"collapse2-24\" class=\"panel-collapse collapse\">\n",
    "      <div class=\"panel-body\">Type:<br>\n",
    "&nbsp;&nbsp;&nbsp;&nbsp;x_nbr_rdd_2 = x_nbr_rdd.map(lambda x: x+1)</div>\n",
    "    </div>\n",
    "  </div>\n",
    "  <div class=\"panel panel-default\">\n",
    "    <div class=\"panel-heading\">\n",
    "      <h4 class=\"panel-title\">\n",
    "        <a data-toggle=\"collapse\" data-parent=\"#accordion-24\" href=\"#collapse3-24\">\n",
    "        Optional Advanced</a>\n",
    "      </h4>\n",
    "    </div>\n",
    "    <div id=\"collapse3-24\" class=\"panel-collapse collapse\">\n",
    "      <div class=\"panel-body\">Write a function which increments the value by 1 and pass that function to map()</div>\n",
    "    </div>\n",
    "  </div>\n",
    "</div> \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#Step 2.4 - Write your map function\n",
    "x_nbr_rdd_2 = x_nbr_rdd.map(lambda x: x+1)\n",
    "    \n",
    "#Optional Advanced\n",
    "#def f(var):\n",
    "#    return var + 1\n",
    "    \n",
    "#x_nbr_rdd_2 = x_nbr_rdd.map(f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 2.5 - Note that there was no result for step 2.4.  Why was this?  Take a look at all the elements of the new RDD.<br>\n",
    "Type:<br>\n",
    "&nbsp;&nbsp;&nbsp;&nbsp; x_nbr_rdd_2.collect()   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[2, 3, 4, 5, 6, 7, 8, 9, 10, 11]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Step 2.5 - Check out the elements of the new RDD. Warning: Be careful with this in real life! Collect returns everything!\n",
    "\n",
    "x_nbr_rdd_2.collect()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 2.6 - Create a new RDD with one string \"Hello Spark\" and print it by getting the first element.<br><br>\n",
    " <div class=\"panel-group\" id=\"accordion-26\">\n",
    "  <div class=\"panel panel-default\">\n",
    "    <div class=\"panel-heading\">\n",
    "      <h4 class=\"panel-title\">\n",
    "        <a data-toggle=\"collapse\" data-parent=\"#accordion-26\" href=\"#collapse1-26\">\n",
    "        Hint 1</a>\n",
    "      </h4>\n",
    "    </div>\n",
    "    <div id=\"collapse1-26\" class=\"panel-collapse collapse\">\n",
    "      <div class=\"panel-body\">Create a variable with the String \"Hello Spark\" and turn it into an RDD with the parallelize() function.   Remember that parallelize() is invoked from the Spark context!</div>\n",
    "    </div>\n",
    "  </div>\n",
    "  <div class=\"panel panel-default\">\n",
    "    <div class=\"panel-heading\">\n",
    "      <h4 class=\"panel-title\">\n",
    "        <a data-toggle=\"collapse\" data-parent=\"#accordion-26\" href=\"#collapse2-26\">\n",
    "        Hint 2</a>\n",
    "      </h4>\n",
    "    </div>\n",
    "    <div id=\"collapse2-26\" class=\"panel-collapse collapse\">\n",
    "      <div class=\"panel-body\">Type:<br>\n",
    "&nbsp;&nbsp;&nbsp;&nbsp; y = \"Hello Spark\"<br>\n",
    "&nbsp;&nbsp;&nbsp;&nbsp; y_str_rdd = sc.parallelize(y)<br>\n",
    "&nbsp;&nbsp;&nbsp;&nbsp; y_str_rdd.first()<br></div>\n",
    "    </div>\n",
    "  </div>\n",
    "  <div class=\"panel panel-default\">\n",
    "    <div class=\"panel-heading\">\n",
    "      <h4 class=\"panel-title\">\n",
    "        <a data-toggle=\"collapse\" data-parent=\"#accordion-26\" href=\"#collapse3-26\">\n",
    "        Optional Advanced</a>\n",
    "      </h4>\n",
    "    </div>\n",
    "    <div id=\"collapse3-26\" class=\"panel-collapse collapse\">\n",
    "      <div class=\"panel-body\">Why did getting the first element only print 'H' instead of \"Hello Spark\"?   What does <i>collect()</i> do?   Is there a way to have the first element be the full string instead of an individual character?</div>\n",
    "    </div>\n",
    "  </div>\n",
    "</div> "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Hello Spark'"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Step 2.6 - Create a string y, then turn it into an RDD\n",
    "y = [\"Hello Spark\"]\n",
    "foo = sc.parallelize(y)\n",
    "foo.first()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 2.7 - Create a third RDD with the following strings and extract the first line.\n",
    "&nbsp;&nbsp;&nbsp;&nbsp;IBM Data Science Experience is built for enterprise-scale deployment.<br>\n",
    "&nbsp;&nbsp;&nbsp;&nbsp;Manage your data, your analytical assets, and your projects in a secured cloud environment.<br>\n",
    "&nbsp;&nbsp;&nbsp;&nbsp;When you create an account in the IBM Data Science Experience, we deploy for you a Spark as a Service instance to power your analysis and 5 GB of IBM Object Storage to store your data.<br><br>\n",
    " <div class=\"panel-group\" id=\"accordion-27\">\n",
    "  <div class=\"panel panel-default\">\n",
    "    <div class=\"panel-heading\">\n",
    "      <h4 class=\"panel-title\">\n",
    "        <a data-toggle=\"collapse\" data-parent=\"#accordion-27\" href=\"#collapse1-27\">\n",
    "        Hint 1</a>\n",
    "      </h4>\n",
    "    </div>\n",
    "    <div id=\"collapse1-27\" class=\"panel-collapse collapse\">\n",
    "      <div class=\"panel-body\">Use an array -- [] -- to contain all three strings.   Don't forget to enclose them in quotes!</div>\n",
    "    </div>\n",
    "  </div>\n",
    "  <div class=\"panel panel-default\">\n",
    "    <div class=\"panel-heading\">\n",
    "      <h4 class=\"panel-title\">\n",
    "        <a data-toggle=\"collapse\" data-parent=\"#accordion-27\" href=\"#collapse2-27\">\n",
    "        Hint 2</a>\n",
    "      </h4>\n",
    "    </div>\n",
    "    <div id=\"collapse2-27\" class=\"panel-collapse collapse\">\n",
    "      <div class=\"panel-body\">&nbsp;&nbsp;&nbsp;&nbsp; z = [ \"IBM Data Science Experience is built for enterprise-scale deployment.\", \"Manage your data, your analytical assets, and your projects in a secured cloud environment.\", \"When you create an account in the IBM Data Science Experience, we deploy for you a Spark as a Service instance to power your analysis and 5 GB of IBM Object Storage to store your data.\" ]<br>\n",
    "&nbsp;&nbsp;&nbsp;&nbsp; z_str_rdd = sc.parallelize(z)<br>\n",
    "&nbsp;&nbsp;&nbsp;&nbsp; z_str_rdd.first()      \n",
    "      </div>\n",
    "    </div>\n",
    "  </div>\n",
    "</div> "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'IBM Data Science Experience is built for enterprise-scale deployment.'"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Step 2.7 - Create String RDD with many lines / entries, Extract first line\n",
    "z = [ \"IBM Data Science Experience is built for enterprise-scale deployment.\", \"Manage your data, your analytical assets, and your projects in a secured cloud environment.\", \"When you create an account in the IBM Data Science Experience, we deploy for you a Spark as a Service instance to power your analysis and 5 GB of IBM Object Storage to store your data.\" ]\n",
    "z_str_rdd = sc.parallelize(z)\n",
    "z_str_rdd.first()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 2.8 - Count the number of entries in this RDD\n",
    "<br>\n",
    " <div class=\"panel-group\" id=\"accordion-28\">\n",
    "  <div class=\"panel panel-default\">\n",
    "    <div class=\"panel-heading\">\n",
    "      <h4 class=\"panel-title\">\n",
    "        <a data-toggle=\"collapse\" data-parent=\"#accordion-28\" href=\"#collapse1-28\">\n",
    "        Hint</a>\n",
    "      </h4>\n",
    "    </div>\n",
    "    <div id=\"collapse1-28\" class=\"panel-collapse collapse\">\n",
    "      <div class=\"panel-body\">Type:<br>\n",
    "&nbsp;&nbsp;&nbsp;&nbsp; z_str_rdd.count()<br></div>\n",
    "    </div>\n",
    "  </div>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Step 2.8 - Count the number of entries in the RDD\n",
    "z_str_rdd.count()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 2.9 - Inspect the elements of this RDD by collecting all the values\n",
    "<br>\n",
    " <div class=\"panel-group\" id=\"accordion-29\">\n",
    "  <div class=\"panel panel-default\">\n",
    "    <div class=\"panel-heading\">\n",
    "      <h4 class=\"panel-title\">\n",
    "        <a data-toggle=\"collapse\" data-parent=\"#accordion-29\" href=\"#collapse1-29\">\n",
    "        Hint</a>\n",
    "      </h4>\n",
    "    </div>\n",
    "    <div id=\"collapse1-29\" class=\"panel-collapse collapse\">\n",
    "      <div class=\"panel-body\">Type:<br>\n",
    "&nbsp;&nbsp;&nbsp;&nbsp;z_str_rdd.collect()<br></div>\n",
    "    </div>\n",
    "  </div>\n",
    "</div> "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['IBM Data Science Experience is built for enterprise-scale deployment.',\n",
       " 'Manage your data, your analytical assets, and your projects in a secured cloud environment.',\n",
       " 'When you create an account in the IBM Data Science Experience, we deploy for you a Spark as a Service instance to power your analysis and 5 GB of IBM Object Storage to store your data.']"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Step 2.9 - Show all the entries in the RDD\n",
    "z_str_rdd.collect()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 2.10 - Split all the entries in the RDD on the spaces.  Then print it out.  Pay careful attention to the new format.\n",
    "<br>\n",
    " <div class=\"panel-group\" id=\"accordion-210\">\n",
    "  <div class=\"panel panel-default\">\n",
    "    <div class=\"panel-heading\">\n",
    "      <h4 class=\"panel-title\">\n",
    "        <a data-toggle=\"collapse\" data-parent=\"#accordion-210\" href=\"#collapse1-210\">\n",
    "        Hint 1</a>\n",
    "      </h4>\n",
    "    </div>\n",
    "    <div id=\"collapse1-210\" class=\"panel-collapse collapse\">\n",
    "      <div class=\"panel-body\">To split on spaces, use the <a href=\"https://docs.python.org/2/library/stdtypes.html#string-methods\"><i>split()</i></a> function.</div>\n",
    "    </div>\n",
    "  </div>\n",
    "  <div class=\"panel panel-default\">\n",
    "    <div class=\"panel-heading\">\n",
    "      <h4 class=\"panel-title\">\n",
    "        <a data-toggle=\"collapse\" data-parent=\"#accordion-210\" href=\"#collapse2-210\">\n",
    "        Hint 2</a>\n",
    "      </h4>\n",
    "    </div>\n",
    "    <div id=\"collapse2-210\" class=\"panel-collapse collapse\">\n",
    "      <div class=\"panel-body\">Since you want to run on every line, use <i>map()</i> on the RDD and write a lambda function to call <i>split()</i></div>\n",
    "    </div>\n",
    "  </div>\n",
    "  <div class=\"panel panel-default\">\n",
    "    <div class=\"panel-heading\">\n",
    "      <h4 class=\"panel-title\">\n",
    "        <a data-toggle=\"collapse\" data-parent=\"#accordion-210\" href=\"#collapse3-210\">\n",
    "        Hint 3</a>\n",
    "      </h4>\n",
    "    </div>\n",
    "    <div id=\"collapse3-210\" class=\"panel-collapse collapse\">\n",
    "      <div class=\"panel-body\">Type: <br>\n",
    "&nbsp;&nbsp;&nbsp;&nbsp;z_str_rdd_split = z_str_rdd.map(lambda line: line.split(\" \"))<br>\n",
    "&nbsp;&nbsp;&nbsp;&nbsp;z_str_rdd_split.collect()<br><br>\n",
    "Question: Is there any difference between split(\" \") and split()?</div>\n",
    "    </div>\n",
    "  </div>\n",
    "</div> "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[['IBM',\n",
       "  'Data',\n",
       "  'Science',\n",
       "  'Experience',\n",
       "  'is',\n",
       "  'built',\n",
       "  'for',\n",
       "  'enterprise-scale',\n",
       "  'deployment.'],\n",
       " ['Manage',\n",
       "  'your',\n",
       "  'data,',\n",
       "  'your',\n",
       "  'analytical',\n",
       "  'assets,',\n",
       "  'and',\n",
       "  'your',\n",
       "  'projects',\n",
       "  'in',\n",
       "  'a',\n",
       "  'secured',\n",
       "  'cloud',\n",
       "  'environment.'],\n",
       " ['When',\n",
       "  'you',\n",
       "  'create',\n",
       "  'an',\n",
       "  'account',\n",
       "  'in',\n",
       "  'the',\n",
       "  'IBM',\n",
       "  'Data',\n",
       "  'Science',\n",
       "  'Experience,',\n",
       "  'we',\n",
       "  'deploy',\n",
       "  'for',\n",
       "  'you',\n",
       "  'a',\n",
       "  'Spark',\n",
       "  'as',\n",
       "  'a',\n",
       "  'Service',\n",
       "  'instance',\n",
       "  'to',\n",
       "  'power',\n",
       "  'your',\n",
       "  'analysis',\n",
       "  'and',\n",
       "  '5',\n",
       "  'GB',\n",
       "  'of',\n",
       "  'IBM',\n",
       "  'Object',\n",
       "  'Storage',\n",
       "  'to',\n",
       "  'store',\n",
       "  'your',\n",
       "  'data.']]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Step 2.10 - Perform a map transformation to split all entries in the RDD\n",
    "#Check out the entries in the new RDD\n",
    "z_str_rdd_split = z_str_rdd.map(lambda line: line.split())\n",
    "z_str_rdd_split.collect()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 2.11 - Explore a new transformation: <a href=\"https://spark.apache.org/docs/1.6.0/api/python/pyspark#pyspark.RDD.flatMap\">flatMap</a>\n",
    "<br>\n",
    "We want to count the words in <b>all</b> the lines, but currently they are split by line.   We need to 'flatten' the line return values into one object.<br>\n",
    "flatMap will \"flatten\" all the elements of an RDD element into 0 or more output terms.<br><br>\n",
    " <div class=\"panel-group\" id=\"accordion-211\">\n",
    "  <div class=\"panel panel-default\">\n",
    "    <div class=\"panel-heading\">\n",
    "      <h4 class=\"panel-title\">\n",
    "        <a data-toggle=\"collapse\" data-parent=\"#accordion-211\" href=\"#collapse1-211\">\n",
    "        Hint 1</a>\n",
    "      </h4>\n",
    "    </div>\n",
    "    <div id=\"collapse1-211\" class=\"panel-collapse collapse\">\n",
    "      <div class=\"panel-body\"><i>flatmap()</i> parameters work the same way as in <i>map()</i></div>\n",
    "    </div>\n",
    "  </div>\n",
    "  <div class=\"panel panel-default\">\n",
    "    <div class=\"panel-heading\">\n",
    "      <h4 class=\"panel-title\">\n",
    "        <a data-toggle=\"collapse\" data-parent=\"#accordion-211\" href=\"#collapse2-211\">\n",
    "        Hint 2</a>\n",
    "      </h4>\n",
    "    </div>\n",
    "    <div id=\"collapse2-211\" class=\"panel-collapse collapse\">\n",
    "      <div class=\"panel-body\">Type:<br>\n",
    "&nbsp;&nbsp;&nbsp;&nbsp; z_str_rdd_split_flatmap = z_str_rdd.flatMap(lambda line: line.split(\" \"))<br>\n",
    "&nbsp;&nbsp;&nbsp;&nbsp; z_str_rdd_split_flatmap.collect()<br></div>\n",
    "    </div>\n",
    "  </div>\n",
    "  <div class=\"panel panel-default\">\n",
    "    <div class=\"panel-heading\">\n",
    "      <h4 class=\"panel-title\">\n",
    "        <a data-toggle=\"collapse\" data-parent=\"#accordion-211\" href=\"#collapse3-211\">\n",
    "        Optional Advanced</a>\n",
    "      </h4>\n",
    "    </div>\n",
    "    <div id=\"collapse3-211\" class=\"panel-collapse collapse\">\n",
    "      <div class=\"panel-body\">Use the replace() and lower() methods to remove all commas and periods then make everything lower-case</div>\n",
    "    </div>\n",
    "  </div>\n",
    "</div> "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['ibm',\n",
       " 'data',\n",
       " 'science',\n",
       " 'experience',\n",
       " 'is',\n",
       " 'built',\n",
       " 'for',\n",
       " 'enterprise-scale',\n",
       " 'deployment',\n",
       " 'manage',\n",
       " 'your',\n",
       " 'data',\n",
       " 'your',\n",
       " 'analytical',\n",
       " 'assets',\n",
       " 'and',\n",
       " 'your',\n",
       " 'projects',\n",
       " 'in',\n",
       " 'a',\n",
       " 'secured',\n",
       " 'cloud',\n",
       " 'environment',\n",
       " 'when',\n",
       " 'you',\n",
       " 'create',\n",
       " 'an',\n",
       " 'account',\n",
       " 'in',\n",
       " 'the',\n",
       " 'ibm',\n",
       " 'data',\n",
       " 'science',\n",
       " 'experience',\n",
       " 'we',\n",
       " 'deploy',\n",
       " 'for',\n",
       " 'you',\n",
       " 'a',\n",
       " 'spark',\n",
       " 'as',\n",
       " 'a',\n",
       " 'service',\n",
       " 'instance',\n",
       " 'to',\n",
       " 'power',\n",
       " 'your',\n",
       " 'analysis',\n",
       " 'and',\n",
       " '5',\n",
       " 'gb',\n",
       " 'of',\n",
       " 'ibm',\n",
       " 'object',\n",
       " 'storage',\n",
       " 'to',\n",
       " 'store',\n",
       " 'your',\n",
       " 'data']"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Step 2.11 - Learn the difference between two transformations: map and flatMap.\n",
    "z_str_rdd_split_flatmap = z_str_rdd.flatMap(lambda line: line.split())\n",
    "z_str_rdd_split_flatmap.collect()\n",
    "#What do you notice? How are the outputs of 2.10 and 2.11 different?\n",
    "z_str_rdd_split_advanced = z_str_rdd_split_flatmap.map(lambda x: x.replace(',','').replace('.','').lower())\n",
    "z_str_rdd_split_advanced.collect()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 2.12 - Augment each entry in the previous RDD with the number \"1\" to create pairs or tuples. The first element of the tuple will be the word and the second elements of the tuple will be the digit \"1\".  This is a common step in performing a count as we need values to sum.\n",
    "<br>\n",
    " <div class=\"panel-group\" id=\"accordion-212\">\n",
    "  <div class=\"panel panel-default\">\n",
    "    <div class=\"panel-heading\">\n",
    "      <h4 class=\"panel-title\">\n",
    "        <a data-toggle=\"collapse\" data-parent=\"#accordion-212\" href=\"#collapse1-212\">\n",
    "        Hint 1</a>\n",
    "      </h4>\n",
    "    </div>\n",
    "    <div id=\"collapse1-212\" class=\"panel-collapse collapse\">\n",
    "      <div class=\"panel-body\">Maps don't always have to perform calculations, they can just echo values as well.   Simply echo the value and a 1<br></div>\n",
    "    </div>\n",
    "  </div>\n",
    "  <div class=\"panel panel-default\">\n",
    "    <div class=\"panel-heading\">\n",
    "      <h4 class=\"panel-title\">\n",
    "        <a data-toggle=\"collapse\" data-parent=\"#accordion-212\" href=\"#collapse2-212\">\n",
    "        Hint 2</a>\n",
    "      </h4>\n",
    "    </div>\n",
    "    <div id=\"collapse2-212\" class=\"panel-collapse collapse\">\n",
    "      <div class=\"panel-body\">We need to create tuples which are values enclosed in parenthesis, so you'll need to enclose the value, 1 in parens.   For example: (x, 1)<br></div>\n",
    "    </div>\n",
    "  </div>\n",
    "  <div class=\"panel panel-default\">\n",
    "    <div class=\"panel-heading\">\n",
    "      <h4 class=\"panel-title\">\n",
    "        <a data-toggle=\"collapse\" data-parent=\"#accordion-212\" href=\"#collapse3-212\">\n",
    "        Hint 3</a>\n",
    "      </h4>\n",
    "    </div>\n",
    "    <div id=\"collapse3-212\" class=\"panel-collapse collapse\">\n",
    "      <div class=\"panel-body\">Type:<br>\n",
    "&nbsp;&nbsp;&nbsp;&nbsp; countWords = z_str_rdd_split_flatmap.map(lambda word:(word,1))<br>\n",
    "&nbsp;&nbsp;&nbsp;&nbsp; countWords.collect()<br></div>\n",
    "    </div>\n",
    "  </div>\n",
    "</div> "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('IBM', 1),\n",
       " ('Data', 1),\n",
       " ('Science', 1),\n",
       " ('Experience', 1),\n",
       " ('is', 1),\n",
       " ('built', 1),\n",
       " ('for', 1),\n",
       " ('enterprise-scale', 1),\n",
       " ('deployment.', 1),\n",
       " ('Manage', 1),\n",
       " ('your', 1),\n",
       " ('data,', 1),\n",
       " ('your', 1),\n",
       " ('analytical', 1),\n",
       " ('assets,', 1),\n",
       " ('and', 1),\n",
       " ('your', 1),\n",
       " ('projects', 1),\n",
       " ('in', 1),\n",
       " ('a', 1),\n",
       " ('secured', 1),\n",
       " ('cloud', 1),\n",
       " ('environment.', 1),\n",
       " ('When', 1),\n",
       " ('you', 1),\n",
       " ('create', 1),\n",
       " ('an', 1),\n",
       " ('account', 1),\n",
       " ('in', 1),\n",
       " ('the', 1),\n",
       " ('IBM', 1),\n",
       " ('Data', 1),\n",
       " ('Science', 1),\n",
       " ('Experience,', 1),\n",
       " ('we', 1),\n",
       " ('deploy', 1),\n",
       " ('for', 1),\n",
       " ('you', 1),\n",
       " ('a', 1),\n",
       " ('Spark', 1),\n",
       " ('as', 1),\n",
       " ('a', 1),\n",
       " ('Service', 1),\n",
       " ('instance', 1),\n",
       " ('to', 1),\n",
       " ('power', 1),\n",
       " ('your', 1),\n",
       " ('analysis', 1),\n",
       " ('and', 1),\n",
       " ('5', 1),\n",
       " ('GB', 1),\n",
       " ('of', 1),\n",
       " ('IBM', 1),\n",
       " ('Object', 1),\n",
       " ('Storage', 1),\n",
       " ('to', 1),\n",
       " ('store', 1),\n",
       " ('your', 1),\n",
       " ('data.', 1)]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Step 2.12 - Create pairs or tuple RDD and print it.\n",
    "countWords = z_str_rdd_split_flatmap.map(lambda word:(word, 1))\n",
    "countWords.collect()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 2.13 Now we have above what is known as a [Pair RDD](http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.rdd.PairRDDFunctions). Each entry in the RDD has a KEY and a VALUE.<br>\n",
    "The KEY is the word (Light, of, the, ...) and the value is the number \"1\".  \n",
    "We can now AGGREGATE this RDD by summing up all the values BY KEY<br><br>\n",
    " <div class=\"panel-group\" id=\"accordion-213\">\n",
    "  <div class=\"panel panel-default\">\n",
    "    <div class=\"panel-heading\">\n",
    "      <h4 class=\"panel-title\">\n",
    "        <a data-toggle=\"collapse\" data-parent=\"#accordion-213\" href=\"#collapse1-213\">\n",
    "        Hint 1</a>\n",
    "      </h4>\n",
    "    </div>\n",
    "    <div id=\"collapse1-213\" class=\"panel-collapse collapse in\">\n",
    "      <div class=\"panel-body\">We want to sum all values by key in the key-value pairs.  The generic function to do this is <i>reduceByKey(func)</i>:<br>\n",
    "      &nbsp;&nbsp;&nbsp;&nbsp;When called on a dataset of (K [Key], V [Value]) pairs, returns a dataset of (K, V) pairs where the values for each key are aggregated using the given reduce function func, which must be of type (V,V) => V.<br><br>Which means func(v1, v2) runs across all values for a specific key.  Think of v1 as the output (initialized as 0 or \"\") and v2 as the iterated value over each value in the set with the same key.  With each iterated value, v1 is updated.<br>\n",
    "      Use a lambda function to sum up the values just as you wrote for <i>map()</i></div>\n",
    "    </div>\n",
    "  </div>\n",
    "  <div class=\"panel panel-default\">\n",
    "    <div class=\"panel-heading\">\n",
    "      <h4 class=\"panel-title\">\n",
    "        <a data-toggle=\"collapse\" data-parent=\"#accordion-213\" href=\"#collapse2-213\">\n",
    "         Hint 2</a>\n",
    "      </h4>\n",
    "    </div>\n",
    "    <div id=\"collapse2-213\" class=\"panel-collapse collapse\">\n",
    "      <div class=\"panel-body\">Type:<br>\n",
    "&nbsp;&nbsp;&nbsp;&nbsp;countWords2 = countWords.reduceByKey(lambda x,y: x+y)<br>\n",
    "&nbsp;&nbsp;&nbsp;&nbsp;countWords2.collect()<br></div>\n",
    "    </div>\n",
    "  </div>\n",
    "  <div class=\"panel panel-default\">\n",
    "    <div class=\"panel-heading\">\n",
    "      <h4 class=\"panel-title\">\n",
    "        <a data-toggle=\"collapse\" data-parent=\"#accordion-213\" href=\"#collapse3-213\">\n",
    "        Optional Advanced</a>\n",
    "      </h4>\n",
    "    </div>\n",
    "    <div id=\"collapse3-213\" class=\"panel-collapse collapse\">\n",
    "      <div class=\"panel-body\">Sort the results by the count.   You could call <i>sortByKey()</i> on the result, but it works on the key....<br>\n",
    "      Also, while the function used in <i>map()</i> has only one parameter, when working with Pair RDDs, that parameter is an array of two values....\n",
    "      </div>\n",
    "    </div>\n",
    "  </div>\n",
    "</div> \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(5, 'your'),\n",
       " (3, 'a'),\n",
       " (3, 'IBM'),\n",
       " (2, 'and'),\n",
       " (2, 'for'),\n",
       " (2, 'you'),\n",
       " (2, 'Data'),\n",
       " (2, 'Science'),\n",
       " (2, 'to'),\n",
       " (2, 'in'),\n",
       " (1, 'enterprise-scale'),\n",
       " (1, 'Service'),\n",
       " (1, 'is'),\n",
       " (1, 'Storage'),\n",
       " (1, 'assets,'),\n",
       " (1, 'as'),\n",
       " (1, 'cloud'),\n",
       " (1, 'secured'),\n",
       " (1, '5'),\n",
       " (1, 'store'),\n",
       " (1, 'we'),\n",
       " (1, 'power'),\n",
       " (1, 'When'),\n",
       " (1, 'Experience'),\n",
       " (1, 'Spark'),\n",
       " (1, 'projects'),\n",
       " (1, 'account'),\n",
       " (1, 'analysis'),\n",
       " (1, 'deployment.'),\n",
       " (1, 'analytical'),\n",
       " (1, 'the'),\n",
       " (1, 'create'),\n",
       " (1, 'data,'),\n",
       " (1, 'data.'),\n",
       " (1, 'instance'),\n",
       " (1, 'Experience,'),\n",
       " (1, 'Manage'),\n",
       " (1, 'an'),\n",
       " (1, 'environment.'),\n",
       " (1, 'Object'),\n",
       " (1, 'GB'),\n",
       " (1, 'deploy'),\n",
       " (1, 'of'),\n",
       " (1, 'built')]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Step 2.13 - Check out the results of the aggregation\n",
    "countWords2 = countWords.reduceByKey(lambda x,y: x+y)\n",
    "countWords3 = countWords2.map(lambda x: (x[1],x[0]))\n",
    "countWords3.sortByKey(False).collect()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3 - Reading a file and counting words\n",
    "### Step 3.1 - Read the Apache Spark README.md file from Github.  The ! allows you to embed file system commands\n",
    "<br>\n",
    "We remove README.md in case there was an updated version -- but also for another reason you will discover in Lab 2<br><br>\n",
    "Type:<br>\n",
    "\n",
    "&nbsp;&nbsp;&nbsp;&nbsp;!rm README.md* -f<br>\n",
    "&nbsp;&nbsp;&nbsp;&nbsp;!wget https://raw.githubusercontent.com/apache/spark/master/README.md<br>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2016-11-16 14:15:30--  https://raw.githubusercontent.com/apache/spark/master/README.md\n",
      "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.48.133\n",
      "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.48.133|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 4034 (3.9K) [text/plain]\n",
      "Saving to: 'README.md'\n",
      "\n",
      "100%[======================================>] 4,034       --.-K/s   in 0s      \n",
      "\n",
      "2016-11-16 14:15:30 (29.9 MB/s) - 'README.md' saved [4034/4034]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "#Step 3.1 - Pull data file into workbench\n",
    "!rm README.md* -f\n",
    "!wget https://raw.githubusercontent.com/apache/spark/master/README.md"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 3.2 - Create an RDD by reading from the local filesystem and count the number of lines  Here is the [textfile()](http://spark.apache.org/docs/latest/api/python/pyspark.html?highlight=textfile#pyspark.SparkContext.textFile) documentation.<br><br>\n",
    " <div class=\"panel-group\" id=\"accordion-32\">\n",
    "  <div class=\"panel panel-default\">\n",
    "    <div class=\"panel-heading\">\n",
    "      <h4 class=\"panel-title\">\n",
    "        <a data-toggle=\"collapse\" data-parent=\"#accordion-32\" href=\"#collapse1-32\">\n",
    "        Hint 1</a>\n",
    "      </h4>\n",
    "    </div>\n",
    "    <div id=\"collapse1-32\" class=\"panel-collapse collapse\">\n",
    "      <div class=\"panel-body\">README.md has been loaded into local storage so there is no path needed.   <i>textFile()</i> returns an RDD -- you do not have to parallelize the result.</div>\n",
    "    </div>\n",
    "  </div>\n",
    "  <div class=\"panel panel-default\">\n",
    "    <div class=\"panel-heading\">\n",
    "      <h4 class=\"panel-title\">\n",
    "        <a data-toggle=\"collapse\" data-parent=\"#accordion-32\" href=\"#collapse2-32\">\n",
    "        Hint 2</a>\n",
    "      </h4>\n",
    "    </div>\n",
    "    <div id=\"collapse2-32\" class=\"panel-collapse collapse\">\n",
    "      <div class=\"panel-body\">Type:<br>\n",
    "&nbsp;&nbsp;&nbsp;&nbsp;textfile_rdd = sc.textFile(\"README.md\")<br>\n",
    "&nbsp;&nbsp;&nbsp;&nbsp;textfile_rdd.count()<br></div>\n",
    "    </div>\n",
    "  </div>\n",
    "  <div class=\"panel panel-default\">\n",
    "    <div class=\"panel-heading\">\n",
    "      <h4 class=\"panel-title\">\n",
    "        <a data-toggle=\"collapse\" data-parent=\"#accordion-32\" href=\"#collapse3-32\">\n",
    "        Optional Advanced 3</a>\n",
    "      </h4>\n",
    "    </div>\n",
    "    <div id=\"collapse3-32\" class=\"panel-collapse collapse\">\n",
    "      <div class=\"panel-body\">By default, <i>textFile()</i> uses UTF-8 format.   Read the file as UNICODE.</div>\n",
    "    </div>\n",
    "  </div>\n",
    "</div> \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "104"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Step 3.2 - Create RDD from data file\n",
    "textfile_rdd = sc.textFile(\"README.md\")\n",
    "textfile_rdd.count()\n",
    "\n",
    "#Optional Advanced 3\n",
    "#textfile_rdd = sc.textFile(\"README.md\", None, True)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 3.3 - Filter out lines that contain \"Spark\". This will be achieved using the [filter](http://spark.apache.org/docs/latest/api/python/pyspark.html?highlight=filter#pyspark.RDD.filter) transformation.  Python allows us to use the 'in' syntax to search strings.<br>\n",
    "We will also take a look at the first line in the newly filtered RDD. <br><br>\n",
    " <div class=\"panel-group\" id=\"accordion-33\">\n",
    "  <div class=\"panel panel-default\">\n",
    "    <div class=\"panel-heading\">\n",
    "      <h4 class=\"panel-title\">\n",
    "        <a data-toggle=\"collapse\" data-parent=\"#accordion-33\" href=\"#collapse1-33\">\n",
    "        Hint 1</a>\n",
    "      </h4>\n",
    "    </div>\n",
    "    <div id=\"collapse1-33\" class=\"panel-collapse collapse\">\n",
    "      <div class=\"panel-body\"><i>filter()</i>, just like <i>map()</i> can take a lambda function as its input</div>\n",
    "    </div>\n",
    "  </div>\n",
    "  <div class=\"panel panel-default\">\n",
    "    <div class=\"panel-heading\">\n",
    "      <h4 class=\"panel-title\">\n",
    "        <a data-toggle=\"collapse\" data-parent=\"#accordion-33\" href=\"#collapse2-33\">\n",
    "        Hint 2</a>\n",
    "      </h4>\n",
    "    </div>\n",
    "    <div id=\"collapse2-33\" class=\"panel-collapse collapse\">\n",
    "      <div class=\"panel-body\">Type:<br>\n",
    "&nbsp;&nbsp;&nbsp;&nbsp;Spark_lines = textfile_rdd.filter(lambda line: \"Spark\" in line)<br>\n",
    "&nbsp;&nbsp;&nbsp;&nbsp;Spark_lines.first()<br></div>\n",
    "    </div>\n",
    "  </div>\n",
    "  <div class=\"panel panel-default\">\n",
    "    <div class=\"panel-heading\">\n",
    "      <h4 class=\"panel-title\">\n",
    "        <a data-toggle=\"collapse\" data-parent=\"#accordion-33\" href=\"#collapse3-33\">\n",
    "        Advanced Optional</a>\n",
    "      </h4>\n",
    "    </div>\n",
    "    <div id=\"collapse3-33\" class=\"panel-collapse collapse\">\n",
    "      <div class=\"panel-body\">There are 19 lines which contain the word \"Spark\".   Find all lines which contain it when case-insensitive<br></div>\n",
    "    </div>\n",
    "  </div>\n",
    "</div> "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#Step 3.3 - Filter for only lines with word Spark\n",
    "Spark_lines = textfile_rdd.filter(lambda line: \"spark\" in line.lower())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 3.4 - Print the number of Spark lines in this filtered RDD out of the total number and print the result as a concatenated string.<br><br>\n",
    " <div class=\"panel-group\" id=\"accordion-34\">\n",
    "  <div class=\"panel panel-default\">\n",
    "    <div class=\"panel-heading\">\n",
    "      <h4 class=\"panel-title\">\n",
    "        <a data-toggle=\"collapse\" data-parent=\"#accordion-34\" href=\"#collapse1-34\">\n",
    "        Hint 1</a>\n",
    "      </h4>\n",
    "    </div>\n",
    "    <div id=\"collapse1-34\" class=\"panel-collapse collapse\">\n",
    "      <div class=\"panel-body\">The <i>print()</i> statement prints to the console.  (Note: be careful on a cluster because a print on a distributed machine will not be seen).  You can cast integers to string by using the <i>str()</i> method.</div>\n",
    "    </div>\n",
    "  </div>\n",
    "  <div class=\"panel panel-default\">\n",
    "    <div class=\"panel-heading\">\n",
    "      <h4 class=\"panel-title\">\n",
    "        <a data-toggle=\"collapse\" data-parent=\"#accordion-34\" href=\"#collapse2-34\">\n",
    "        Hint 2</a>\n",
    "      </h4>\n",
    "    </div>\n",
    "    <div id=\"collapse2-34\" class=\"panel-collapse collapse\">\n",
    "      <div class=\"panel-body\">Strings can be concatenated together with the + sign.   You can mark a statement as spanning multiple lines by putting a \\ at the end of the line.</div>\n",
    "    </div>\n",
    "  </div>\n",
    "  <div class=\"panel panel-default\">\n",
    "    <div class=\"panel-heading\">\n",
    "      <h4 class=\"panel-title\">\n",
    "        <a data-toggle=\"collapse\" data-parent=\"#accordion-34\" href=\"#collapse3-34\">\n",
    "        Hint 3</a>\n",
    "      </h4>\n",
    "    </div>\n",
    "    <div id=\"collapse3-34\" class=\"panel-collapse collapse\">\n",
    "      <div class=\"panel-body\">Type:<br>\n",
    "&nbsp;&nbsp;&nbsp;&nbsp;print \"The file README.md has \" + str(Spark_lines.count()) + \\<br>\n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;\" of \" + str(textfile_rdd.count()) + \\<br>\n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;\" lines with the word Spark in it.\"<br></div>\n",
    "    </div>\n",
    "  </div>\n",
    "</div> "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The file README.md has 30 of 104 lines with the word Spark in it.\n"
     ]
    }
   ],
   "source": [
    "#Step 3.4 - count the number of lines\n",
    "print \"The file README.md has \" + str(Spark_lines.count()) + \\\n",
    "        \" of \" + str(textfile_rdd.count()) + \\\n",
    "        \" lines with the word Spark in it.\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 3.5 - Now count the number of times the word Spark appears in the original text, not just the number of lines that contain it.\n",
    "Looking back at previous exercises, you will need to: <br>\n",
    "&nbsp;&nbsp;&nbsp;&nbsp;1 - Execute a flatMap transformation on the original RDD Spark_lines and split on white space.<br>\n",
    "&nbsp;&nbsp;&nbsp;&nbsp;2 - Filter out all instances of the word Spark<br>\n",
    "&nbsp;&nbsp;&nbsp;&nbsp;3 - Count all instances<br>\n",
    "&nbsp;&nbsp;&nbsp;&nbsp;4 - Print the total count<br><br>\n",
    " <div class=\"panel-group\" id=\"accordion-35\">\n",
    "  <div class=\"panel panel-default\">\n",
    "    <div class=\"panel-heading\">\n",
    "      <h4 class=\"panel-title\">\n",
    "        <a data-toggle=\"collapse\" data-parent=\"#accordion-35\" href=\"#collapse1-35\">\n",
    "        Hint 1</a>\n",
    "      </h4>\n",
    "    </div>\n",
    "    <div id=\"collapse1-35\" class=\"panel-collapse collapse\">\n",
    "      <div class=\"panel-body\"><i>str</i> not in <i>string</i> is how to filter out</div>\n",
    "    </div>\n",
    "  </div>\n",
    "  <div class=\"panel panel-default\">\n",
    "    <div class=\"panel-heading\">\n",
    "      <h4 class=\"panel-title\">\n",
    "        <a data-toggle=\"collapse\" data-parent=\"#accordion-35\" href=\"#collapse2-35\">\n",
    "        Hint 2</a>\n",
    "      </h4>\n",
    "    </div>\n",
    "    <div id=\"collapse2-35\" class=\"panel-collapse collapse\">\n",
    "      <div class=\"panel-body\">flatMapRDD = textfile_rdd.flatMap(lambda line: line.split())<br>\n",
    "      flatMapRDDFilter = flatMapRDD.filter(lambda line: \"Spark\" not in line)<br>\n",
    "      flatMapRDDFilterCount = flatMapRDDFilter.count()<br>\n",
    "      print flatMapRDDFilterCount<br>\n",
    "      </div>\n",
    "    </div>\n",
    "  </div>\n",
    "  <div class=\"panel panel-default\">\n",
    "    <div class=\"panel-heading\">\n",
    "      <h4 class=\"panel-title\">\n",
    "        <a data-toggle=\"collapse\" data-parent=\"#accordion-35\" href=\"#collapse3-35\">\n",
    "        Optional Advanced</a>\n",
    "      </h4>\n",
    "    </div>\n",
    "    <div id=\"collapse3-35\" class=\"panel-collapse collapse\">\n",
    "      <div class=\"panel-body\">Put the entire statement on one line and make the filter case-insensitive.</div>\n",
    "    </div>\n",
    "  </div>\n",
    "</div> "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(u'Spark', 15)]"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Step 3.5 - Count the number of instances of\"Spark\"\n",
    "#flatMapRDD = textfile_rdd.flatMap(lambda line: line.split())\n",
    "#flatMapFiltered = flatMapRDD.filter(lambda x: x==\"Spark\")\n",
    "#flatMapFilteredMapped = flatMapFiltered.map(lambda x: (x,1))\n",
    "#FMFMReduced = flatMapFilteredMapped.reduceByKey(lambda x,y: x+y)\n",
    "#print FMFMReduced.collect()\n",
    "\n",
    "#Optional Advanced\n",
    "textfile_rdd.flatMap(lambda line: line.split()).filter(lambda x: x.lower()==\"spark\").map(lambda x: (x,1)).reduceByKey(lambda x,y: x+y).collect()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 4 - Perform analysis on a data file\n",
    "This part is a little more open ended and there are a few ways to complete it.  Scroll up to previous examples for some guidance.  You will download a data file, transform the data, and then average the prices.  The data file will be a sample of tech stock prices over six days. <br>\n",
    "\n",
    "Data Location: https://raw.githubusercontent.com/JosephKambourakisIBM/SparkPoT/master/StockPrices.csv<br>\n",
    "The data file is a csv<br>\n",
    "Here is a sample of the file:<br>\n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;\"IBM\",\"159.720001\" ,\"159.399994\" ,\"158.880005\",\"159.539993\", \"159.550003\", \"160.350006\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2016-11-16 14:15:32--  https://raw.githubusercontent.com/JosephKambourakisIBM/SparkPoT/master/StockPrices.csv\n",
      "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.48.133\n",
      "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.48.133|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 244 [text/plain]\n",
      "Saving to: 'StockPrices.csv'\n",
      "\n",
      "100%[======================================>] 244         --.-K/s   in 0s      \n",
      "\n",
      "2016-11-16 14:15:32 (54.9 MB/s) - 'StockPrices.csv' saved [244/244]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "#Step 4.1 - Delete the file if it exists, download a new copy and load it into an RDD\n",
    "!rm StockPrices.csv -f\n",
    "!wget https://raw.githubusercontent.com/JosephKambourakisIBM/SparkPoT/master/StockPrices.csv\n",
    "    \n",
    "SP = sc.textFile(\"StockPrices.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(u'IBM', 159.720001, 159.399994, 158.880005, 159.539993, 159.550003),\n",
       " (u'MSFT', 58.099998, 57.889999, 57.459999, 57.59, 57.669998),\n",
       " (u'AAPL', 106.82, 106.0, 106.099998, 106.730003, 107.730003),\n",
       " (u'ORCL', 41.310001, 41.310001, 41.220001, 41.16, 41.25)]"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Step 4.2 - Transform the data to extract the stock ticker symbol and the prices.  \n",
    "SP_mapped = SP.map(lambda line: line.split(','))\n",
    "SP_Keyed = SP_mapped.map(lambda x: (x[0], float(x[1]), float(x[2]), float(x[3]), float(x[4]), float(x[5])))\n",
    "SP_Keyed.collect()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(u'IBM', 159.4179992),\n",
       " (u'MSFT', 57.7419988),\n",
       " (u'AAPL', 106.6760008),\n",
       " (u'ORCL', 41.2500006)]"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Step 4.3 - Compute the averages and print them for each symbol.\n",
    "SP_Mean = SP_Keyed.map(lambda x: (x[0], (x[1]+x[2]+ x[3]+ x[4]+x[5])/5)) #There are other ways to do this such as importing a mean function from numpy\n",
    "SP_Mean.collect()\n",
    "\n",
    "#Using a mean function\n",
    "#import numpy\n",
    "#SP_Mean = SP_Keyed.map(lambda x: (x[0], numpy.mean([x[1],x[2], x[3], x[4], x[5]]))) \n",
    "#SP_Mean.collect()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2 with Spark 2.0",
   "language": "python",
   "name": "python2-spark20"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}