Introducing multi object tracking - adding Joint Detection and Embedding Tracker (JDETracker)

[shardulparab97]

Motivation and Context

Multi-object tracking(MOT) would be an exciting functionality which can be added to the pytorchvideo.

Likewise, I propose to add the Joint Detection and Embedding (JDE) Tracker introduced in the paper 'Towards Real-Time Multi-Object Tracking' (https://arxiv.org/abs/1909.12605).

This tracker can work with any model which can provide the following two inputs:
(i) pred_dets: Detection results of the image i.e. x1, y1, x2, y2, object_conf (The detections should/could be passed through NMS or a similar technique followed by scaling the coordinates to the original image size - This way allowing more flexibility for the tracker)
(ii) pred_embs: Embedding results of the image.

How Has This Been Tested

Have added a test file at tests/test_models_mot_tracker_jde_tracker.py

Types of changes

• Docs change / refactoring / dependency upgrade
• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)

Checklist

• My code follows the code style of this project.
• My change requires a change to the documentation.
• I have updated the documentation accordingly.
• I have read the CONTRIBUTING document.
• I have completed my CLA (see CONTRIBUTING)
• I have added tests to cover my changes.
• All new and existing tests passed.

[shardulparab97]

@kalyanvasudev have shifted the multi object tracking component to the projects folder and have also the following:
(i) Readme file for reference
(ii) A detector utils file for utilizing a sample detector which outputs detections and embeddings, which can used for running a demo.
(ii) A demo file for taking video input and outputting a video with tracked objects.

Would love to know your thoughts on the same, thank you.

[austinmw]

Hey @shardulparab97 is multi-object tracking still planned here?

[shardulparab97]

Hey @austinmw , yes I believe that multi-object tracking would be pretty good addition. Currently awaiting the repo maintainers response for the next steps! :)

[austinmw]

@shardulparab97 Hey, responding to #174, in my opinion this would be a cool addition as an example of joint-detection-association tracking.

I would also personally be very interested in being able to perform tracking-by-detection by adding a modular tracker to an existing detector (like ByteTrack or StrongSORT for example). This would allow users to swap the detector and tracker components separately, depending on the relative difficulty of the detection and tracking tasks.