This python script downloads all Creative Commons licensed videos for a given YouTube channel. Downloads are saved to FILEPATH/{video_id}.{ext}, where {video_id} is the YouTube id of the video, and {ext} is the filetype extension. The metadata is also stored in FILEPATH/{video_id}.info.json as a JSON file. An additional utility file at FILEPATH/dl_status keeps track of the download status, and an informational file about the channel is stored at FILEPATH/channel_summary.json
usage: python get_vids.py [-h] -c CHANNELID [-f FILEPATH] [-w] [-a]
optional arguments:
-h, --help show this help message and exit
-c CHANNELID, --channelid CHANNELID
id for the channel of the form
https://www.youtube.com/channel/{channel_id}
-f FILEPATH, --filepath FILEPATH
file path where the downloaded videos should be
downloaded. Defaults to local directory
-w, --overwrite if specified, will overwrite existing videos with the
same name when downloading. By default, it will skip
over existing files
-a, --getaudio if specified, will also try to get audio file
(separately)
-v, --verbose print extra details
-d, --dry-run don't download files, only print information about
channel and videos
Python requirements listed in environment.yml conda environment file. Requires conda. Create an environment with the required packages using:
conda env create -f environment.yml
Important packages:
- youtube-dl is used for downloading the videos
- google-api-python-client is used for authentication and youtube API requests
In order to use the YouTube API, the user must register an API key with Google. See this tutorial for instructions on how to get an API key. Users should replace the client_secret.json_demo with their respective client_secret.json.
The first time the script is run on a machine, it will prompt for user authentication via a URL. Users should copy the URL into a browser, and authenticate using their Google/Youtube credentials. An access token will be stored in the tokens/credentials.json file, which should automatically be used for future runs.
The client_secret.json and credentials.json files should be kept secret.
ffmpeg or a similar tool is necessary to combine audio and high resolution videos. By default, YouTube stores high resolution (1080p+) videos separately from audio files. If available, we download the best video and audio files separately. They can then be combined by the user as needed using a tool such as ffmpeg.
Some media may require VLC or similar software for playing different formats.
This software, funded by PSIAP 2017, supports the development of datasets representative of public safety operations, whose need was formally identified in NIST TN 1917: Public Safety Analytics R&D Roadmap:
One of the most fundamental barriers to seamless data integration is simply a lack of awareness or access to datasets that are accurate, current, and relevant to improving response.
This effort is motivated by the increasing importance of video and imagery for public safety operations. The goal is to accelerate technology innovation for network providers, application providers, and public safety agencies. Videos with a Creative Commons (CC) license are routinely uploaded to YouTube. As part of the PSIAP dataset effort, we identified specific YouTube channels with representative public safety content. YouTube channels operate independently on the PSIAP effort and will continue to collect and post videos beyond the PSIAP effort. This results in an ever growing dataset. This growth is a realization discussed in NISTIR 8164: First Workshop on Video Analytics in Public Safety:
Thoughts on future development: Increasingly, software developers will need to take advantage of new hardware innovations. With the increasing reliance on machine learning methods such as deep learning, developers will require access to ever-increasing quantities of data for both training and evaluation purposes. Going beyond static datasets (which can be overlearned), future algorithms will require constant novelty allowing for a state of never-ending unsupervised learning. This will require a migration from data-sets to data-sites and, perhaps, data-cities.
- Creative Commons
- Creative Commons - YouTube Help
- Public Safety Innovation Accelerator Program 2017
- ResearchGate - NIST PSIAP: Representative Public Safety Video Dataset
- New Jersey Office of Homeland Security and Preparedness
- MIT Lincoln Laboratory - Humanitarian Assistance and Disaster Relief Systems
- Jeffrey Liu (MITLL)
- Andrew Weinert (MITLL)
- Gabriela Barrera (MITLL)
- Dieter Schuldt (MITLL)
- Steven Talpas (NJOHSP)
- William Drew (NJOHSP)
This work was performed under the following financial assistance award 70NANB17Hl69 from U.S. Department of Commerce, National Institute of Standards and Technology.
