Many-task Computing for Everyone: How Python is Making Parallel Computing Accessible

University of Colorado Boulder - Wednesday September 18, 2013 @ 6:00pm MST
Location: ATLAS - 1125 18th St Bldg 223, Boulder, CO - Room 100
http://www.meetup.com/Data-Science-Business-Analytics/events/137505342/

Many-task Computing for Everyone: How Python is Making Parallel Computing Accessible - Abstract

Researchers in life and social sciences are increasingly utilizing high performance compute and storage systems to advance their work. One common question that continues to surface for these scientists is: what's the most efficient way to execute thousands of independent tasks on a cluster? This body of research is called Many-Task Computing (MTC) and in this talk we compare the characteristics and scaling of several different ways to address this problem using Python.

A decade ago, distributed computing usually implied using the Message Passing Interface (MPI) standard with a language like C, C++, or Fortran, possibly running as a hybrid code using OpenMP. But today more accessible languages, such as Python, give users different options for utilizing an HPC resource in very efficient ways. The talk will include many examples from different domains. Our results suggest that Python is an excellent way to manage many-task computing and that no single technique is the obvious choice in every situation.

Bio

Monte Lunacek is a consultant and educator in the Research Computing group at the University of Colorado Boulder. Monte received his PhD in Computer Science from Colorado State University and was a postdoc in the Computational Science group at the National Renewable Energy Lab (NREL) before joining University of Colorado. His expertise are in high performance computing and parameter optimization.