current position：Home>MOT：A Higher Order Metric for Evaluating Multi-object Tracking

MOT：A Higher Order Metric for Evaluating Multi-object Tracking

2022-05-15 07:35:34【chaibubble】

brief introduction

HOTA: A Higher Order Metric for Evaluating Multi-object Tracking yes IJCV 2020 Of paper, Before that, use MOTChallenge Mainly multi-target tracking benchmark Has been used to MOTA Evaluation criteria for ranking , although MOTChallenge Of metrics There are also IDF1, But the ranking is still based on MOTA Subject to . however MOTA In some cases, it is not enough to measure the performance of multi-target tracking , It's not even as good as IDF1, So this article reconsiders the multi-target tracking task , And a method is proposed Higher Order Tracking Accuracy Of Metric.HOTA It can better align the evaluation score with people's visual perception . MOTA The main evaluation is 2006 It was proposed in , And pass by MOTChallenge Blessing , It is still the mainstream multi-target tracking evaluation standard , and HOTA It's just been put forward , At present, only KITTI MOT In the use of . Even if it does replace MOTA, It will also take a long time .

MOTA The problem of

The proportion of detection is greater than that of tracking

MOTA The evaluation overemphasizes the effect of detection , according to MOTA Calculation method of , One extreme case is , The performance of the test is excellent , But all detected targets are not tracked , Instead, all are assigned the same track id, At this time MOTA It's going to be very high , because IDsw=0. But obviously , The tracking performance of this extreme case is 0.

\begin{aligned} \text {MOTA} = 1 - \frac{|\text {FN}| + |\text {FP}| + |\text {IDSW}|}{|\text {gtDet}|} \end{aligned}

\begin{aligned} \text {MOTP} = \frac{1}{|\text {TP}|}\sum _{\text {TP}}{ \mathcal {S}} \end{aligned}

MOTP Even more so , The root cause is that there is no tracking of anything , Instead, only evaluate the test results . although IDF1 The tracking effect can be evaluated , But the ranking depends on MOTA Of .

\begin{aligned}&\text {ID-Recall} = \frac{|\text {IDTP}|}{|\text {IDTP}| + |\text {IDFN}|} \end{aligned}

\begin{aligned}&\text {ID-Precision} = \frac{|\text {IDTP}|}{|\text {IDTP}| + |\text {IDFP}|} \end{aligned}

\begin{aligned}&\text {IDF1} = \frac{|\text {IDTP}|}{|\text {IDTP}| + 0.5 \, |\text {IDFN}| + 0.5 \, |\text {IDFP}|} \end{aligned}

Pictured above ,gt The length of is 100, Tracking performance C hold gt Divided into 4 paragraph , In fact, the performance is poor , however MOTA the height is 97%.

Precision The specific gravity of is greater than Recall

There is no definition IDsw Of MOTA by MODA, That is, the accuracy of multi-target detection （Multi Object Detection Accuracy）, The formula is as follows ：

\begin{aligned} \begin{aligned} \text {MODA}&= 1 - \frac{|\text {FN}| + |\text {FP}|}{|\text {gtDet}|}&\\&= \frac{|\text {TP}| - |\text {FP}|}{|\text {TP}| + |\text {FN}|}&\\&= \text {DetRe} \cdot (2 - \frac{1}{\text {DetPr}})&\end{aligned} \end{aligned}

You can find , If it's tested Precision Less than or equal to 0.5 Words ,MODA Will be for 0, Even negative values , And the test Recall Less than or equal to 0.5 But it won't have such an impact .

Evaluation Metric

DetA

DetA For the accuracy of detection , Evaluate the performance of detector in multi-target tracking , The functions and Precision and Recall almost , Total of all categories acc The following formula represents ：

\begin{aligned}&\text {DetA}_\alpha = \frac{|\text {TP}|}{|\text {TP}| + |\text {FN}| + |\text {FP}|}&\end{aligned}

AssA

AssA For the accuracy of correlation , Evaluate the accuracy of correlation , The formula is as follows ：

\begin{aligned}&\mathcal {A}(c) = \frac{|\text {TPA}(c)|}{|\text {TPA}(c)| + |\text {FNA}(c)| + |\text {FPA}(c)|}&\end{aligned}

\begin{aligned}&\text {AssA}_\alpha = \frac{1}{|\text {TP}|} \sum _{c \in \{\text {TP}\}} \mathcal {A}(c)&\end{aligned}

DetA,AssA The role of , And Precision,Recall,IDP,IDR,IDF1 Very similar Precision,Recall It is used to evaluate the accuracy and recall of detection , and DetA Used to evaluate the accuracy of detection . IDP,IDR,IDF1 Used to evaluate the accuracy of matching , Recall rate and F1-score, and AssA Used to evaluate the accuracy of matching . This needs to know \text {TPA}(c) ,\text {FNA}(c) ,\text {FPA}(c) These numbers mean , First c It belongs to TP The point of , It can be TP Any one of , According to this point , We can always identify a unique GT The trajectory , At the same time, if there is pred Track and GT If the trajectory intersects at this point , We can also identify one pred The trajectory . It should be noted that , Even if it's the same GT Different trajectories c, It will also produce different \text {TPA}(c) ,\text {FNA}(c) ,\text {FPA}(c) , Therefore, these three values can only be bound with sampling , Not bound to dataset . This is related to 《Evaluating Multiple Object Tracking Performance: The CLEAR MOT Metrics》 Different , Not for one GT The trajectory is assigned a maximum matching degree pred The trajectory .

And here you need

HOTA

Single index evaluation
Evaluate long-term high-order tracking correlation
Decompose into sub indicators , Allows analysis of different components of tracker performance .

\begin{aligned}&\text {HOTA}_{\alpha } = \sqrt{\frac{\sum _{c \in \{\text {TP}\}} \mathcal {A}(c) }{|\text {TP}| + |\text {FN}| + |\text {FP}|}}&\end{aligned}

HOTA Evaluation is a double jacquard coefficient , That is, I took it twice and compared it , First of all \mathcal {A}(c)

For the current interest-c Corresponding GT tracklet, Calculated True Positive Associations,False Positive Associations And False Negative Associations, This is the jackard coefficient on the first floor , It should be noted that interest-c It's not worth a , All the needs SUM. As shown in the figure below . The jacquard coefficient of the second layer is SUM After \mathcal {A}(c)

Compared with the results of the previous test TP,FN,FP. Last ,\alpha Is a fixed threshold , therefore \text {HOTA}_{\alpha } Is the result of a fixed threshold , and HOTA yes ：

\begin{aligned} \text {HOTA} = \int _{0}^{1}{ \text {HOTA}_\alpha \; d\alpha } \approx \frac{1}{19} \sum _{\alpha \in \{ \begin{array}{c} 0.05, \; 0.1, \; ... \\ 0.9, \; 0.95 \end{array} \} } \text {HOTA}_\alpha \end{aligned}

It's like coco Of AP Calculation . Last , according to DetA and AssA,HOTA It can be calculated by ：

\begin{aligned}&\begin{aligned} \text {HOTA}_\alpha&\quad = \sqrt{\frac{\sum _{c \in \{\text {TP}\}} \mathcal {A}(c) }{|\text {TP}| + |\text {FN}| + |\text {FP}|}}&\\&\quad = \sqrt{\text {DetA}_\alpha \cdot \text {AssA}_\alpha }&\end{aligned}&\end{aligned}

HOTA Decompose into sub-metric

HOTA Decompose into detection and association

\begin{aligned}&\text {DetA}_\alpha = \frac{|\text {TP}|}{|\text {TP}| + |\text {FN}| + |\text {FP}|}&\end{aligned}

\begin{aligned}&\text {AssA}_\alpha = \frac{1}{|\text {TP}|} \sum _{c \in \{\text {TP}\}} \mathcal {A}(c)&\end{aligned}

\begin{aligned}&\mathcal {A}(c) = \frac{|\text {TPA}(c)|}{|\text {TPA}(c)| + |\text {FNA}(c)| + |\text {FPA}(c)|}&\end{aligned}

detection Decompose into precision and recall

\begin{aligned} \text {DetRe}_\alpha&= \frac{|\text {TP}|}{|\text {TP}| + |\text {FN}| } \end{aligned}

\begin{aligned} \text {DetPr}_\alpha&= \frac{|\text {TP}|}{|\text {TP}| + |\text {FP}| } \end{aligned}

\begin{aligned} \text {DetA}_\alpha&= \frac{\text {DetRe}_\alpha \cdot \text {DetPr}_\alpha }{\text {DetRe}_\alpha + \text {DetPr}_\alpha - \text {DetRe}_\alpha .\text {DetPr}_\alpha } \end{aligned}

association Decompose into precision and recall

\begin{aligned} \text {AssRe}_\alpha&= \frac{1}{|\text {TP}|} \; \sum _{c \in \{\text {TP}\}} \frac{|\text {TPA}(c)|}{|\text {TPA}(c)| + |\text {FNA}(c)|} \end{aligned}

\begin{aligned} \text {AssPr}_\alpha&= \frac{1}{|\text {TP}|} \; \sum _{c \in \{\text {TP}\}} \frac{|\text {TPA}(c)|}{|\text {TPA}(c)| + |\text {FPA}(c)|} \end{aligned}

\begin{aligned} \text {AssA}_\alpha&= \frac{\text {AssRe}_\alpha \cdot \text {AssPr}_\alpha }{\text {AssRe}_\alpha + \text {AssPr}_\alpha - \text {AssRe}_\alpha \cdot \text {AssPr}_\alpha } \end{aligned}

Reference

copyright notice
author[chaibubble],Please bring the original link to reprint, thank you.
https://en.chowdera.com/2022/131/202205102135064718.html