DocumentCloud Batch Upload Script Tutorial | macOS version

This video provides an overview of how to use the Batch Upload Script on a computer using the macOS operating system. It was written to upload the CIA Crest files, a collection containing almost 1 million files.

It keeps track of which files were uploaded successfully, As a result, it can be stopped and restarted and pick up where it left off and errors can be retried. It uploads files in batches. It can be stopped gracefully by pressing Ctrl+C (once) while running.

As a reminder, your DocumentCloud account must be verified in order to use the script. The following link will take you to the submission form to start the process: https://airtable.com/shrZrgdmuOwW0ZLPM

0:00 Finding the batch upload script
https://github.com/MuckRock
0:35 Verification link for DocumentCloud
0:40 Installing Python
1:20 Running Install Certificates.command to install SSL for the script to run properly on macOS X
1:52 Installing pip
python3 get-pip.py
2:08 Grabbing the script from GitHub
2:30 Setting environment variables DC_USERNAME and DC_PASSWORD for the script to run properly
export DC_USERNAME =
export DC_PASSWORD =
2:58 Locating the project ID for the script
3:15 Installing pip packages from requirements.txt in script folder
pip install -r requirements.txt
3:52 Finding the file path to the directory of documents we are seeking to upload
4:40 Generating a CSV file that describes our uploadable content using the script
5:45 Uploading documents using the script once CSV is generated
5:56 Note about delayed processing/indexing of documents by using the batch upload script
6:43 Verified that documents uploaded properly