M24.md

Installation

Using Poetry

1. Install poetry based on the instructions provided in their documentation.
2. Clone timesformer along with additional dependencies using:

    git clone [email protected]:darpa-sail-on/TimeSformer.git
    git clone [email protected]/darpa-sail-on/ND-Activity-Recognition-Feeback.git
   

   This would create TimeSformer, and ND-Activity-Recognition-Feeback directories in your working directory
3. Create a virtual environment and install the components using the following commands:

    cd TimeSformer
    git checkout m24-agent
    poetry install
    poetry run pip install ../ND-Activity-Recognition-Feeback
    poetry shell
   

Using Conda

1. Create a conda virtual environment and activate it:

   conda create -n timesformer python=3.8 -y
   source activate timesformer
   

2. Install the following packages:

   • torchvision: pip install torchvision or conda install torchvision -c pytorch
   • fvcore: pip install 'git+https://github.com/facebookresearch/fvcore'
   • simplejson: pip install simplejson
   • einops: pip install einops
   • timm: pip install timm
   • PyAV: conda install av -c conda-forge
   • psutil: pip install psutil
   • scikit-learn: pip install scikit-learn
   • OpenCV: pip install opencv-python
   • tensorboard: pip install tensorboard
   • sail-on-client: pip install sail-on-client

3. Build the TimeSformer codebase by running:

   git clone [email protected]:darpa-sail-on/TimeSformer.git
   cd TimeSformer
   git checkout m24-agent
   python -m pip install .
   

4. Install Additional dependencies using:

   pip install ../ND-Activity-Recognition-Feeback
   

Usage

Dry Run

1. Download the checkpoint_epoch_00015.pyth from google drive
2. Download the evm model (HDF5 File) from google drive in the same directory as the model from the previous step.
3. If you are using the files on your machine use the following command

     HYDRA_FULL_ERROR=1 sail-on-client --config-dir <your working directory>/TimeSformer/configs/ \
                                       --config-name dry_run_local \
                                       test_root=<your working directory>/TimeSformer/data \
                                       protocol.smqtk.config.dataset_root=<root directory for videos from first prerequisites> \
                                       model_root=<root directory where models were downloaded from step 1 and 2> \
                                       [email protected]=[timesformer_base] \
                                       protocol.smqtk.config.test_ids=[OND.0.10001.6438158]
   

M-24 Evaluation

Feature Extraction

1. Download the checkpoint_epoch_00015.pyth from google drive
2. If you are using the evaluation use the following command

   HYDRA_FULL_ERROR=1 sail-on-client --config-dir <your working directory>/TimeSformer/configs/ \
                                     --config-name feature_extraction_par \
                                     server_url=<url for server> \
                                     protocol.smqtk.config.dataset_root=<root directory for videos> \
                                     model_root=<root directory for models> \
                                     protocol.smqtk.config.feature_dir=<root directory where features are saved> \
                                     [email protected]=[timesformer_base] \
                                     protocol.smqtk.config.test_ids=[<comma seperated list of test ids>]
   

3. If you are using the files on your machine use the following command

   HYDRA_FULL_ERROR=1 sail-on-client --config-dir <your working directory>/TimeSformer/configs/ \
                                     --config-name feature_extraction_local \
                                     test_root=<root directory for tests> \
                                     protocol.smqtk.config.dataset_root=<root directory for videos> \
                                     model_root=<root directory for models> \
                                     protocol.smqtk.config.feature_dir=<root directory where features are saved> \
                                     [email protected]=[timesformer_base] \
                                     protocol.smqtk.config.test_ids=[<comma seperated list of test ids>]
   

4. [Optional] To use slurm with the feature extraction use the following command

    HYDRA_FULL_ERROR=1 sail-on-client --config-dir configs/ \
                                      --config-name feature_extraction_local \
                                      --multirun protocol.smqtk.config.test_ids=["OND.9.99999.0"],["OND.9.99999.1"],["OND.9.99999.2"],["OND.9.99999.3"],["OND.9.99999.4"],["OND.9.99999.5"],["OND.9.99999.6"],["OND.9.99999.7"] \
                                      test_root=/data/datasets/m24-activity-test/feature_extraction_tests \
                                      protocol.smqtk.config.dataset_root=/data/datasets/m24-activity-test/1115_2021 \
                                      model_root=/home/khq.kitware.com/ameya.shringi/models/timesformer-m24 \
                                      protocol.smqtk.config.feature_dir=/home/khq.kitware.com/ameya.shringi/features/timesformer-m24 \
                                      [email protected]=[timesformer_base] \
                                      hydra/launcher=veydrus \
   

System Detection

1. Download the features from google drive

2. Download the evm model (HDF5 File) from google drive

3. With the evaluation server use the following command

     HYDRA_FULL_ERROR=1 sail-on-client --config-dir configs/ \
                                       --config-name system_detection_par \
                                       server_url=<url for server> \
                                       model_root=<root directory where models are stored> \
                                       protocol.smqtk.config.feature_dir=<root directory where features are stored> \
                                       protocol.smqtk.config.dataset_root=<root directory of vidoes> \
                                       [email protected]=[timesformer_base] \
                                       protocol.smqtk.config.test_ids=[<comma seperated test ids>]
   

4. With files on the machine using the following command

     HYDRA_FULL_ERROR=1 sail-on-client --config-dir configs/ \
                                       --config-name system_detection_local \
                                       test_root=<root directory with tests> \
                                       protocol.smqtk.config.feature_dir=<root directory with features> \
                                       protocol.smqtk.config.dataset_root=<root directory with videos> \
                                       [email protected]=[timesformer_base]
                                       protocol.smqtk.config.test_ids=[<comma seperate test ids>]
   

Given Detection

1. Download the features from google drive

2. Download the evm model (HDF5 File) from google drive

3. With the evaluation server use the following command

     HYDRA_FULL_ERROR=1 sail-on-client --config-dir configs/ \
                                       --config-name given_detection_par \
                                       server_url=<url for server> \
                                       model_root=<root directory where models are stored> \
                                       protocol.smqtk.config.feature_dir=<root directory where features are stored> \
                                       protocol.smqtk.config.dataset_root=<root directory of vidoes> \
                                       [email protected]=[timesformer_rd] \
                                       protocol.smqtk.config.test_ids=[<comma seperated test ids>]
   

4. With files on the machine using the following command

     HYDRA_FULL_ERROR=1 sail-on-client --config-dir configs/ \
                                       --config-name given_detection_local \
                                       test_root=<root directory with tests> \
                                       protocol.smqtk.config.feature_dir=<root directory with features> \
                                       protocol.smqtk.config.dataset_root=<root directory with videos> \
                                       [email protected]=[timesformer_rd]
                                       protocol.smqtk.config.test_ids=[<comma seperate test ids>]
   

System Detection With Classification Feedback

1. Download the features from google drive
2. Download the evm model (HDF5 File) from google drive
3. Download additional file available in the following links:
   • clip path
   • clip templates
   • pred known map
   • pred label encs
   • feedback known map
   • feedback label encs
4. With the evaluation server use the following command

     HYDRA_FULL_ERROR=1 sail-on-client --config-dir configs/ \
                                       --config-name system_detection_classification_feedback_par \
                                       server_url=<url for server> \
                                       model_root=<root directory where models are stored> \
                                       protocol.smqtk.config.feature_dir=<root directory where features are stored> \
                                       protocol.smqtk.config.dataset_root=<root directory of vidoes> \
                                       [email protected]=[timesformer_feedback] \
                                       protocol.smqtk.config.test_ids=[<comma seperated test ids>]
   

5. With files on the machine using the following command

     HYDRA_FULL_ERROR=1 sail-on-client --config-dir configs/ \
                                       --config-name system_detection_classification_feedback_local \
                                       test_root=<root directory with tests> \
                                       protocol.smqtk.config.feature_dir=<root directory with features> \
                                       protocol.smqtk.config.dataset_root=<root directory with videos> \
                                       [email protected]=[timesformer_feedback]
                                       protocol.smqtk.config.test_ids=[<comma seperate test ids>]
   

Given Detection With Detection Feedback

1. Download the features from google drive
2. Download the evm model (HDF5 File) from google drive
3. With the evaluation server use the following command

     HYDRA_FULL_ERROR=1 sail-on-client --config-dir configs/ \
                                       --config-name given_detection_detection_feedback_par \
                                       server_url=<url for server> \
                                       model_root=<root directory where models are stored> \
                                       protocol.smqtk.config.feature_dir=<root directory where features are stored> \
                                       protocol.smqtk.config.dataset_root=<root directory of vidoes> \
                                       [email protected]=[timesformer_detection_feedback] \
                                       protocol.smqtk.config.test_ids=[<comma seperated test ids>]
   

4. With files on the machine using the following command

     HYDRA_FULL_ERROR=1 sail-on-client --config-dir configs/ \
                                       --config-name given_detection_detection_feedback_local \
                                       test_root=<root directory with tests> \
                                       protocol.smqtk.config.feature_dir=<root directory with features> \
                                       protocol.smqtk.config.dataset_root=<root directory with videos> \
                                       [email protected]=[timesformer_detection_feedback]
                                       protocol.smqtk.config.test_ids=[<comma seperate test ids>]