Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add option to use legacy planning metric definition #173

Merged
merged 1 commit into from
Jul 8, 2024

Conversation

wljungbergh
Copy link
Contributor

As described in #166, there is a discrepancy between how planning metrics (L2 and collision rate) are computed in this repo vs previous works, such as VAD and ST-P3.

However, in the paper, these values are compared in the same table, leading to confusion. This PR enables computation of the UniAD metrics using the legacy definition used by VAD/ST-P3, making comparison easier. However, it still defaults to the "original" UniAD way of evaluating. This results in a ranking shift when comparing the methods, see the table below

Method L2 1s (m) L2 2s (m) L2 3s (m) CR 1s (%) CR 2s (%) CR 3s (%)
ST-P3 1.33 2.11 2.90 0.23 0.62 1.27
UniAD (your current metric) 0.51 0.98 1.65 0.10 0.15 0.61
UniAD (legacy ST-P3 metric) 0.42 0.64 0.91 0.07 0.10 0.22
VAD-Tiny 0.46 0.76 1.12 0.21 0.35 0.58
VAD-Base 0.41 0.70 1.05 0.07 0.17 0.41

Copy link
Collaborator

@YTEP-ZHI YTEP-ZHI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Update planning metrics

@YTEP-ZHI YTEP-ZHI merged commit 7531063 into OpenDriveLab:main Jul 8, 2024
@wljungbergh wljungbergh deleted the legacy_metrics branch September 25, 2024 06:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants