2.3 Data formats - Python for Scientific Computing 2021

Input/output bandwidth is just as much a consumable resource as CPU or memory. And data formats are important for both ease (human effort) and time taken to read data. Here, we will go over the basic concepts and some important data formats which you may need in your work.

https://aaltoscicomp.github.io/python-for-scicomp/data-formats/


00:56 Why are data formats important?
22:38 Two most common data formats (tidy and array)
28:25 File formats, considerations
34:38 csv
44:08 Feather
48:01 Parquet
50:53 HDF5
52:36 NetCDF
54:13 npy (numpy data format)
55:23 Exercises
55:38 Why binary formats over text formats? Main considerations
57:23 Things to remember / questions to ask yourself
58:18 sqlite
1:00:20 Discussion
1:02:50 Structural format vs semantic formats
1:03:50 Conclusion, Relation of data formats to the rest of workflow: reducing bottlenecks

-----

Python for Scientific Computing is a bridge between basic Python courses and scientific work with Python. This is a basic to intermediate course in Python tools such as NumPy, SciPy, Matplotlib, and Pandas. It also covers some more advanced tools, such as Binder, releasing software, data formats, etc. It is suitable for people who have a basic understanding of Python and want to know some internals and important libraries for science. We don't cover anything in too much depth, but we do introduce you to all of the main tools you will need.

This course was put on as a collaboration between partners in Finland, Norway, and Sweden, coordinated by Aalto Scientific Computing.

Links:
- Playlist: https://www.youtube.com/playlist?list=PLZLVmS9rf3nOS7bHNmbcDoyTnMYaz_TJW
- Course material: https://aaltoscicomp.github.io/python-for-scicomp/
- Workshop webpage: https://scicomp.aalto.fi/training/scip/python-for-scicomp/
- Aalto Scientific Computing: https://scicomp.aalto.fi/
- CodeRefinery: https://coderefinery.org