Python Tutorial: Managing Data with Generators

Want to learn more? Take the full course at https://campus.datacamp.com/courses/parallel-programming-with-dask-in-python at your own pace. More than a video, you'll learn hands-on coding & quickly apply skills to your daily work.

---
We've seen that breaking files into chunks can alleviate problems caused by large datasets. Before introducing Dask, let's see how to use generators to help.

Recall our work from before: we iterated over 1,000-line chunks of lines extracted from a CSV file of taxi-ride data. We then filtered the individual chunks applying a function within a list comprehension.

If we replace the enclosing brackets with parentheses in a list comprehension, the result is a generator expression. Generator expressions resemble comprehensions, but use lazy evaluation. This means elements are generated one-at-a-time, so they are never in memory simultaneously This is extremely helpful when operating at the limits of available memory. We can quickly build another generator distances whose elements are totals of the trip_distance column from each chunk. No actual computation is done until we iterate over the chained generators explicitly (in this case, by applying the function sum to distances). To reiterate - no reading or work is done until the very last step.

The generators chunks & distances persist after the computation. However, they have been consumed at this point. That is, trying to next function on either produces a StopIteration exception (which tells users that the generator is exhausted).

Let's use generators now to read many files. We use a slightly different collection of CSV files that describe Yellow Cab rides in New York City. We use the string format() method and a generator filenames to generate names of CSV files for each month of 2015. Imagine here that, rather than having one large file to read in chunks, we have many large individual files that cannot fit in memory simultaneously.

Let's examine one of the CSV files using `pd.read_csv`. We force columns 1 & 2 to be read as datetime objects. For this data, we have to calculate the trip duration explicitly.

We embed this calculation within a function `count_long_trips` that filters trips longer than 20 minutes, and counts the total number of trips and long trips. These two values are returned in a DataFrame with one row.

With the function count_long_trips in place, we can organize our work into a pipeline using generators. We recreate filenames, this time as a list comprehension (it's just a list of 12 strings so a generator is not required). We create a generator dataframes to load the files listed in filenames one-by-one. We create another generator totals that applies count_long_trips to each DataFrame from dataframes. Finally, the actual computation takes about 13.4 seconds on my laptop.

For now, notice all computation are deferred until we compute sum(totals). Using the builtin sum function consumes the generator.

The computed DataFrame annual_totals has two columns, n_long and n_total. Their ratio, then, is the fraction of trips over 20 minutes in duration over the year 2015 (roughly 20%).

Okay, now it's time for you to experiment with generators yourself in the exercises.

#PythonTutorial #Python #DataCamp #BigData #parallelprogramming #dask #Managing #Data #Generators