Paul Ingram - Calling Distributed Python from Inside SQL

PyData Dallas 2015
"We set out to build a fully scalable distributed SQL platform but quickly realized that use cases at that scale were much more complex that simply joining data and easy to digest select statements. To support our data science team and our users, we implemented a distributed python layer that allows our programmers to quickly construct distributed functions that enhance SQL’s native capabilities, including implementing statistical and machine learning algorithms for use across massively parallel data sets. Our talk will cover the motivation behind implementing the python layer as well as the technical challenges we ran into while extending the language and enabling it to run in a truly parallelized fashion. Abstract : The SQL language allows many data scientists to quickly manipulate raw data into data sets useful for analysis. However, SQL has a number of restrictions that make simple tasks such as efficiently parsing rows of data difficult to do without writing custom procedures. Couple that with the increasingly varied data types and data sets, and SQL is quickly eschewed as the language of choice for complex, big data manipulation. Enter Python — a language perfectly suited to the task. Marrying python and SQL has allowed our data science teams to pick the language that best fits their specific needs and construct functions that allow us to implement repeatable processes at scale and make complex calls accessible to clients and team members who are only familiar with SQL. We will introduce 3 key paradigms for parallelizing python and building the SQL integration and discuss the technical thought process behind each : 1. Implementing a master-worker communication paradigm for Python 2. Developing a custom distributed data structure 3. Distributed package access At the conclusion of the talk, we will walk through a basic use case and demonstrate how the interplay between SQL and python is made real time. " 00:00 Welcome!
00:10 Help us add time stamps or captions to this video! See the description for details.

Want to help add timestamps to our YouTube videos to help with discoverability? Find out more here: https://github.com/numfocus/YouTubeVideoTimestamps