Function improvement (continued) (#7)

* Tweak text descriptions.
* Move TODO from different files to a single file.
* Add examples with both steam and smoke on the index page.
* Add text description of user score to the labeling page.
* Add a link to the introduction video to the index page.
* Add a link to the viewer for each video in the gallery when in the researcher mode.
* Replace 'Collection' to 'Contribution'.
* Add raw score (the number of viewed unlabeled videos) for each user to the database.
* Display the raw score in the front-end.
* Remove user_raw_score and user_score from the connection table.
* Add the api call to get label statistics.
* Show the label statistics on the front-end when labeling.
* Add more positive videos.
* Improve the flow for adding videos.
* When querying batches, make sure that half of them are partially labeled. In this way, we can speed up labeling.
* Now the gold standard videos have at least one negative and one positive videos, which prevents spamming.
* Fix the migration script problem.
* Upgrade python packages.
* Add loading icon for the galley.
* Improve loading icon logic for the labeling tool.
* Increase the countdown duration when labeling.
* Reduce the number of videos at the first page to 16 because of video number restrictions before mobile safari version 12.
* Fix a problem that when google sign-in api failed, the system get stuck in the loading state.
* Update website template to improve the functionality of the dialog box.
* Update website template to improve accessibility and usability.
* Fix a css bug in the submit button when counting down.
* Improve the automatic check for video autoplay.
* Fix a bug that in track protection mode (Firefox), the page will hang forever because the browser is unable to load the google analytics.
* Improve the track protection detection in the google analytics tracker.
* Add browser support checks.
* Add a script to mark bad videos.
* Improve video autoplay detection.
* Add frequently asked questions (FAQ) page.
* Add a google form to collect feedback on the FAQ page.
* Change max-age in the apache2 cache control to 5 seconds (this used to be 60 seconds, and the browser will cache the label statistics, which can be wrong).
* Fix other miscellaneous bugs.

README.md

@@ -1,11 +1,27 @@
 # video-labeling-tool
 Demo: http://smoke.createlab.org
 
-A tool for labeling video clips (both front-end and back-end). The back-end depends on a [thumbnail server](https://github.com/CMU-CREATE-Lab/timemachine-thumbnail-server) to provides video urls. The back-end is based on [flask](http://flask.pocoo.org/). A flask tutorial can be found on [this blog](https://blog.miguelgrinberg.com/post/the-flask-mega-tutorial-part-i-hello-world). 
+A tool for labeling video clips (both front-end and back-end). The back-end depends on the [thumbnail server](https://github.com/CMU-CREATE-Lab/timemachine-thumbnail-server) to provide video urls. The back-end is based on [flask](http://flask.pocoo.org/). A flask tutorial can be found on [this blog](https://blog.miguelgrinberg.com/post/the-flask-mega-tutorial-part-i-hello-world). This tool is tested and worked on:
+- macOS Mojave
+  - Chrome 76
+  - Safari 12
+  - Firefox 68
+- Windows 10
+  - Chrome 76
+  - Firefox 68
+  - Edge 44
+- Android 8
+  - Chrome 76
+  - Firefox 68
+- iOS 12
+  - Chrome 76
+  - Safari 12
+  - Firefox 18
 
 ### Table of Content
 - [Install MySQL](#install-mysql)
 - [Setup back-end](#setup-back-end)
+- [Prepare gold standards for quality check](#prepare-gold-standards)
 - [Dump and import MySQL database](#dump-and-import-mysql)
 - [Deploy back-end using uwsgi](#deploy-back-end-using-uwsgi)
 - [Connect uwsgi to apache](#connect-uwsgi-to-apache)
@@ -79,10 +95,15 @@ echo 'export PATH="/usr/local/miniconda3/bin:$PATH"' >> ~/.bash_profile
 echo '. /usr/local/miniconda3/etc/profile.d/conda.sh' >> ~/.bash_profile
 source ~/.bash_profile
 ```
-Clone this repository.
+Clone this repository and set the permission.
 ```sh
 git clone https://github.com/CMU-CREATE-Lab/video-labeling-tool.git
-sudo chown -R $USER video-labeling-tool
+sudo chown -R $USER video-labeling-tool/
+sudo addgroup [group_name]
+sudo usermod -a -G [group_name] [user_name]
+groups [user_name]
+sudo chmod -R 775 video-labeling-tool/
+sudo chgrp -R [group_name] video-labeling-tool/
 ```
 Create conda environment and install packages. It is important to install pip first inside the newly created conda environment.
 ```sh
@@ -141,6 +162,20 @@ Run server in the conda environment for development purpose.
 sh development.sh
 ```
 
+# <a name="prepare-gold-standards"></a>Prepare gold standards for quality check
+The system uses gold standards (videos with known labels) to check the quality of each labeled batch. If a user did not label the gold standards correctly, the corresponding batch would be discarded. Initially, there are no gold standards, and the backend will not return videos for labeling. To solve this issue, give yourself the researcher permission by using
+```sh
+python set_client_type.py [user_id] 0
+```
+where user_id can be found on the "Account" tab on the top right of the "label.html" page after logging in with Google. The number 0 that follows the user_id is the researcher permission. For more information about the permission, please refer to the client_type variable in the "User" class in the "application.py" file. The system will not run the quality check for users with the researcher permission. In this way, you can start labeling first.
+
+To assign gold standards videos, go to the "gallery.html" page when logging in with the account that has the researcher permission. On the gallery, you will find "P*" and "N*" buttons. Clicking on these buttons shows the positive and negative videos that the researcher labeled. You can now use the dropdown below each video to change the label to Gold Pos (positive gold standards) or Gold Neg (negative gold standards). Once there is a sufficient number of gold standards (more than 4), normal users will be able to label videos. I recommend having at least 100 gold standards to start.
+
+If you found that some videos are not suitable for labeling (e.g., due to incorrect image stitching), you can get the url of the video and use the following command to mark similar ones (with the same date and bounding box) as "bad" videos. This process does not remove videos. Instead it gives all bad videos a label state -2.
+```sh
+python set_client_type.py [video_url]
+```
+
 # <a name="dump-and-import-mysql"></a>Dump and import MySQL database
 This section assumes that you want to dump the production database to a file and import it to the development database. First, SSH to the production server and dump the database to the /tmp/ directory.
 ```sh
@@ -244,7 +279,7 @@ sudo vim /etc/apache2/sites-available/[BACK_END_DOMAIN].conf
   ServerName [BACK_END_DOMAIN]
   Header always set Access-Control-Allow-Origin "http://[FRONT_END_DOMAIN]"
   Header set Access-Control-Allow-Headers "Content-Type"
-  Header set Cache-Control "max-age=60, public, must-revalidate"
+  Header set Cache-Control "max-age=5, public, must-revalidate"
   ProxyPreserveHost On
   ProxyRequests Off
   ProxyVia Off
@@ -270,7 +305,7 @@ sudo vim /etc/apache2/sites-available/[FRONT_END_DOMAIN].conf
   ServerName [FRONT_END_DOMAIN]
   DocumentRoot /[PATH]/video-labeling-tool/front-end
   Header always set Access-Control-Allow-Origin "*"
-  Header set Cache-Control "max-age=60, public, must-revalidate"
+  Header set Cache-Control "max-age=5, public, must-revalidate"
   <Directory "/[PATH]/video-labeling-tool/front-end">
     Options FollowSymLinks
     AllowOverride None
@@ -280,7 +315,7 @@ sudo vim /etc/apache2/sites-available/[FRONT_END_DOMAIN].conf
   CustomLog ${APACHE_LOG_DIR}/[FRONT_END_DOMAIN].access.log combined
 </VirtualHost>
 ```
-Use the following if you only want to access the server from an IP address with a port. Remember to tell the apache server to listen to the port number.
+Use the following if you only want to access the server from an IP address with a port (e.g., http://192.168.1.72:8080). Remember to tell the apache server to listen to the port number.
 ```sh
 sudo vim /etc/apache2/sites-available/video-labeling-tool-front-end.conf
 # Add the following lines to this file
@@ -314,8 +349,14 @@ Give permissions so that the Certbot and apache can modify the website. This ass
 cd /var/www/
 sudo mkdir html # only run this if the html directory did not exist
 sudo chmod 775 html
-sudo chgrp www-data html
-sudo chgrp www-data [CLONED_REPOSITORY]
+sudo chmod 775 [CLONED_REPOSITORY]
+sudo chgrp -R www-data html
+sudo chgrp -R www-data [CLONED_REPOSITORY]
+```
+If other users need to modify this repository, add them to the www-data group.
+```sh
+sudo usermod -a -G www-data [user_name]
+groups [user_name]
 ```
 Run the Certbot.
 ```sh
@@ -338,7 +379,7 @@ sudo vim /etc/apache2/sites-available/[BACK_END_DOMAIN].conf
   Header always set Access-Control-Allow-Origin "https://[FRONT_END_DOMAIN]"
   Header set Access-Control-Allow-Headers "Content-Type"
   # The following line forces the browser to break the cache
-  Header set Cache-Control "max-age=60, public, must-revalidate"
+  Header set Cache-Control "max-age=5, public, must-revalidate"
   # Reverse proxy to the uwsgi server
   ProxyPreserveHost On
   ProxyRequests Off
@@ -374,7 +415,7 @@ sudo vim /etc/apache2/sites-available/[FRONT_END_DOMAIN].conf
   # The following line enables cors
   Header always set Access-Control-Allow-Origin "*"
   # The following line forces the browser to break the cache
-  Header set Cache-Control "max-age=60, public, must-revalidate"
+  Header set Cache-Control "max-age=5, public, must-revalidate"
   <Directory "/[PATH]/video-labeling-tool/front-end">
     Options FollowSymLinks
     AllowOverride None
@@ -612,3 +653,23 @@ $.ajax({
 # curl example
 curl -d 'user_token=your_user_token' -H 'Content-Type: application/x-www-form-urlencoded; charset=UTF-8' -X POST http://localhost:5000/api/v1/get_all_labels
 ```
+### Get the statistics of labels
+Get the number of all videos (excluding the videos that were marked as "bad" data), the number of fully labeled videos (confirmed by multiple users), and the number of partially labeled videos.
+- Paths:
+  - **/api/v1/get_label_statistics**
+- Available methods:
+  - GET
+- Returned fields:
+  - "num_all_videos": number of all videos (excluding bad data)
+  - "num_fully_labeled": number of fully labeled videos
+  - "num_partially_labeled": number of partially labeled videos
+```JavaScript
+// jQuery examples
+$.getJSON("http://localhost:5000/api/v1/get_label_statistics", function (data) {
+  console.log(data);
+});
+```
+```sh
+# curl example
+curl http://localhost:5000/api/v1/get_label_statistics
+```

TODO

@@ -0,0 +1,35 @@
+TODO: add interactive tutorial
+
+TODO: design user feedback system after labeling a batch (e.g., dialog box)
+  - poor performance when labels are rejected
+  - if the user made too many bad batches, ask the user to retake the tutorial
+  - detect and mark spamming (e.g., all not selected, all selected) in the database
+
+TODO: design features for sharing achievements
+  - add a leaderboard for showing user id and scores
+  - generate PDF certificates of label statistics for each user upon request
+  - allow users to share the badge with their achievement on social media
+
+TODO: add the feature for labeling smoke opacity
+  - add a compass to show sun direction for each video (when labeling opacity)
+TODO: add the feature for cropping videos to a region
+
+TODO: refactor code based on https://codeburst.io/jwt-authorization-in-flask-c63c1acf4eeb
+
+TODO: as users gain enough scores, advance them to the harder mode
+  - laypeople mode: select videos that have smoke
+  - amateur mode: label smoke opacity (low, medium, high)
+  - expert mode: crop videos to a region
+  - how to invalidate previous user tokens with different permissions?
+  - need to add a table to record the promotion history
+  - need to encode client type in the user token, and check if this matches the database record
+  - for a user that did not login via google, always treat them as laypeople
+
+BUG: when changing the client type of a user, previous tokens with different permissions are still working
+  - need to invalidate previous issued tokens?
+
+OPTIONAL: graphically display the label statistics (instead of using text)
+OPTIONAL: add a playback timeline bar to show the video playback time
+OPTIONAL: add a link back to time machine viewer (or a larger video) on the labeling page (also gallery page)
+OPTIONAL: prevent the case that multiple people label the same data
+  - add the last_queried_time to video and query the ones with last_queried_time <= current_time - lock_time