Distance-IoU Loss: Faster and Better Learning for Bounding Box Regression

In [\citeauthoryearSong et al.2018], the central axis line is applied in pedestrian detection. CornerNet [\citeauthoryearLaw and Deng2018] suggested predicting a pair of corners to replace a rectangular box for locating object. In RepPoints [\citeauthoryearYang et al.2019], a rectangular box is formed by predicting several points. Recently, FSAF [\citeauthoryearZhu, He, and Savvides2019] proposed anchor-free branch to tackle the issues of non-optimality in online feature selection. There are also several loss functions for object detection, e.g., focal loss [\citeauthoryearLin et al.2017], class-balanced loss [\citeauthoryearCui et al.2019], balanced loss for classification and bounding box regression [\citeauthoryearPang et al.2019], and gradient flow balancing loss [\citeauthoryearLi, Liu, and Wang2019]. Nevertheless, the regression of rectangular boxes is still the most popular manner in the state-of-the-art object detection algorithms [\citeauthoryearRedmon and Farhadi2018, \citeauthoryearHe et al.2017, \citeauthoryearFu et al.2017, \citeauthoryearLiu et al.2016, \citeauthoryearTian et al.2019].

The $\ell_{n}$-norm loss functions are usually adopted in bounding box regression, but are sensitive to variant scales. In YOLO v1 [\citeauthoryearRedmon et al.2016], square roots for $w$ and $h$ are adopted to mitigate this effect, while YOLO v3 [\citeauthoryearRedmon and Farhadi2018] uses $2-wh$. IoU loss is also used since Unitbox [\citeauthoryearYu et al.2016], which is invariant to the scale. GIoU [\citeauthoryearRezatofighi et al.2019] loss is proposed to tackle the issues of gradient vanishing for non-overlapping cases, but is still facing the problems of slow convergence and inaccurate regression. In comparison, we propose DIoU and CIoU losses with faster convergence and better regression accuracy.

NMS is the last step in most object detection algorithms, in which redundant detection boxes are removed as long as its overlap with the highest score box exceeds a threshold. Soft-NMS [\citeauthoryearBodla et al.2017] penalizes the detection score of neighbors by a continuous function w.r.t. IoU, yielding softer and more robust suppression than original NMS. IoU-Net [\citeauthoryearJiang et al.2018] introduces a new network branch to predict the localization confidence to guide NMS. Recently, adaptive NMS [\citeauthoryearLiu, Huang, and Wang2019] and Softer-NMS [\citeauthoryearHe et al.2019] are proposed to respectively study proper threshold and weighted average strategies. In this work, DIoU is simply deployed as the criterion in original NMS, in which the overlap area and the distance between two central points of bounding boxes are simultaneously considered when suppressing redundant boxes.

In the simulation experiments, we try to cover most of the relationships between bounding boxes in terms of distance, scale and aspect ratio, as shown in Fig. 3(a). In particular, we choose 7 unit boxes (i.e., the area of each box is 1) with different aspect ratios (i.e., 1:4, 1:3, 1:2, 1:1, 2:1, 3:1 and 4:1) as target boxes. Without loss of generality, the central points of the 7 target boxes are fixed at $(10,10)$. The anchor boxes are uniformly scattered at 5,000 points. (i) Distance: In the circular region centered at $(10,10)$ with radius 3, 5,000 points are uniformly chosen to place anchor boxes with 7 scales and 7 aspect ratios. In these cases, overlapping and non-overlapping boxes are included. (ii) Scale: For each point, the areas of anchor boxes are set as $0.5$, $0.67$, $0.75$, $1$, $1.33$, $1.5$ and $2$. (iii) Aspect ratio: For a given point and scale, 7 aspect ratios are adopted, i.e., following the same setting with target boxes (i.e., 1:4, 1:3, 1:2, 1:1, 2:1, 3:1 and 4:1). All the $5,000\times 7\times 7$ anchor boxes should be fitted to each target box. To sum up, there are totally $1,715,000=7\times 7\times 7\times 5,000$ regression cases.

Then given a loss function $\mathcal{L}$, we can simulate the procedure of bounding box regression for each case using gradient descent algorithm. For predicted box $B_{i}$, the current prediction can be obtained by

where $B_{i}^{t}$ is the predicted box at iteration $t$, $\nabla B_{i}^{t-1}$ denotes the gradient of loss $\mathcal{L}$ w.r.t. $B_{i}$ at iteration $t-1$, and $\eta$ is the step. It is worth noting that in our implementation, the gradient is multiplied by $2-IoU_{i}^{t-1}$ to accelerate the convergence. The performance of bounding box regression is evaluated using $\ell_{1}$-norm. For each loss function, the simulation experiment is terminated when reaching iteration $T=200$, and the error curves are shown in Fig. 3(b).

In Fig. 4, we visualize the final regression errors at iteration $T$ for 5,000 scattered points. From Fig. 4(a), it is easy to see that IoU loss only works for the cases of overlapping with target boxes. The anchor boxes without overlap will not move due to that $\nabla B$ is always 0.

By adding a penalty term as Eqn. (3), GIoU loss can better relieve the issues of non-overlapping cases. From Fig. 4(b), GIoU loss significantly enlarges the basin, i.e., the area that GIoU works. But the cases at horizontal and vertical orientations are likely to still have large errors. This is because that the penalty term in GIoU loss is used to minimize $|C-A\cup B|$, but the area of $C-A\cup B$ is often small or 0 (when two boxes have inclusion relationships), and then GIoU almost degrades to IoU loss. GIoU loss would converge to good solution as long as running sufficient iterations with proper learning rates, but the convergence rate is indeed very slow. Geometrically speaking, from the regression steps as shown in Fig. 1, one can see that GIoU actually increases the predicted box size to overlap with target box, and then the IoU term will make the predicted box match with the target box, yielding a very slow convergence.

To sum up, IoU loss converges to bad solutions for non-overlapping cases, while GIoU loss is with slow convergence especially for the boxes at horizontal and vertical orientations. And when incorporating into object detection pipeline, both IoU and GIoU losses cannot guarantee the accuracy of regression. It is natural to ask that: First, is it feasible to directly minimize the normalized distance between predicted box and target box for achieving faster convergence? Second, how to make the regression more accurate and faster when having overlap even inclusion with target box?

To answer the first question, we propose to minimize the normalized distance between central points of two bounding boxes, and the penalty term can be defined as

where $\mathbf{b}$ and $\mathbf{b}^{gt}$ denote the central points of $B$ and $B^{gt}$, $\rho(\cdot)$ is the Euclidean distance, and $c$ is the diagonal length of the smallest enclosing box covering the two boxes. And then the DIoU loss function can be defined as

As shown in Fig. 5, the penalty term of DIoU loss directly minimizes the distance between two central points, while GIoU loss aims to reduce the area of $C-B\cup B^{gt}$.

Then we answer the second question, by suggesting that a good loss for bounding box regression should consider three important geometric factors, i.e., overlap area, central point distance and aspect ratio. By uniting the coordinates, IoU loss considers the overlap area, and GIoU loss heavily relies on IoU loss. Our proposed DIoU loss aims at considering simultaneously the overlap area and central point distance of bounding boxes. However, the consistency of aspect ratios for bounding boxes is also an important geometric factor.

Therefore, based on DIoU loss, the CIoU loss is proposed by imposing the consistency of aspect ratio,

where $\alpha$ is a positive trade-off parameter, and $v$ measures the consistency of aspect ratio,

Then the loss function can be defined as

And the trade-off parameter $\alpha$ is defined as

by which the overlap area factor is given higher priority for regression, especially for non-overlapping cases.

Finally, the optimization of CIoU loss is same with that of DIoU loss, except that the gradient of $v$ w.r.t. $w$ and $h$ should be specified,

The dominator $w^{2}+h^{2}$ is usually a small value for the cases $h$ and $w$ ranging in $[0,1]$, which is likely to yield gradient explosion. And thus in our implementation, the dominator $w^{2}+h^{2}$ is simply removed for stable convergence, by which the step size $\frac{1}{w^{2}+h^{2}}$ is replaced by 1 and the gradient direction is still consistent with Eqn. (12).

In original NMS, the IoU metric is used to suppress the redundant detection boxes, where the overlap area is the unique factor, often yielding false suppression for the cases with occlusion. We in this work suggest that DIoU is a better criterion for NMS, because not only overlap area but also central point distance between two boxes should also be considered in the suppression criterion. For the predicted box $\mathcal{M}$ with the highest score, the DIoU-NMS can be formally defined as

where box $B_{i}$ is removed by simultaneously considering the IoU and the distance between central points of two boxes, $s_{i}$ is the classification score and $\varepsilon$ is the NMS threshold. We suggest that two boxes with distant central points probably locate different objects, and should not be removed. Moreover, the DIoU-NMS is very flexible to be integrated into any object detection pipeline with only a few lines of code.

PASCAL VOC [\citeauthoryearEveringham et al.2010] is one of the most popular dataset for object detection. YOLO v3 is trained on PASCAL VOC using DIoU and CIoU losses in comparison with IoU and GIoU losses. We use VOC 07+12 (the union of VOC 2007 trainval and VOC 2012 trainval) as training set, containing 16,551 images from 20 classes. And the testing set is VOC 2007 test, which consists of 4,952 images. The backbone network is Darknet608. We follow exactly the GDarknet¹¹ 1 https://github.com/generalized-iou/g-darknet training protocol released from [\citeauthoryearRezatofighi et al.2019], and the maximum iteration is set to 50K. The performance for each loss has been reported in Table 1. We use the same performance measure, i.e., AP (the average of 10 mAP across different IoU thresholds) = (AP50 + AP55 + $\dots$ + AP95) / 10 and AP75 (mAP@0.75). We also report the evaluation results using GIoU metric.

As shown in Table 1, GIoU as a generalized version of IoU, it indeed achieves a certain degree of performance improvement. While DIoU loss can improve the performance with gains of 3.29% AP and 6.02% AP75 using IoU as evaluation metric. CIoU loss takes the three important geometric factors of two bounding boxes into account, which brings an amazing performance gains, i.e., 5.67% AP and 8.95% AP75. From Fig. 6, one can see that the detection box by CIoU loss is more accurate than that by GIoU loss. Finally, CIoU loss combined with DIoU-NMS brings marvelous improvements of 5.91% AP and 9.88% AP75. Also in terms of GIoU metric, we can come to the same conclusion, validating the effectiveness of the proposed methods. We note that GIoU metric is actually consistent with IoU metric, and thus we only report the IoU metric for the following experiments.

We use another popular one-stage method SSD to further conduct evaluation experiments. The latest PyTorch implementation of SSD²² 2 https://github.com/JaryHuang/awesome˙SSD˙FPN˙GIoU is adopted. Both the training set and testing set share the same setting with YOLO v3 on PASCAL VOC. Following the default training protocol, the max iteration is set to 120K. The backbone network is ResNet-50-FPN. The default bounding box regression loss is smooth $\ell_{1}$-norm, which has different magnitudes with IoU-based losses. And thus there should be a more appropriate trade-off weight for the regression loss to balance with the classification loss. We have observed that for dense anchor algorithms, increasing the regression loss properly can improve the performance. Therefore, for a fair comparison, we fix the weight on regression loss as 5 for these IoU-based losses. And then we train the models using IoU, GIoU, DIoU and CIoU losses. Table 2 gives the quantitative comparison, in which AP and AP75 of IoU metric are reported. For SSD, we can see the consistent improvements of DIoU and CIoU losses in comparison with IoU and GIoU losses.

We also evaluate the proposed method on another more difficult and complex dataset MS COCO 2017 [\citeauthoryearLin et al.2014] using Faster R-CNN³³ 3 https://github.com/generalized-iou/Detectron.pytorch. MS COCO is a large-scale dataset, containing more than 118K images for training and 5K images for evaluation. Following the same training protocol of [\citeauthoryearRezatofighi et al.2019], we trained the models using DIoU and CIoU losses in comparison with IoU and GIoU losses. The backbone network is ResNet-50-FPN. Besides AP and AP75 metrics, the evaluation metrics in terms of large, medium and small scale objects are also included. As for the trade-off weight for regression loss, we set the weight as 12 for all losses for a fair comparison. Table 3 reports the quantitative comparison.

Faster R-CNN is a detection algorithm with dense anchor boxes, and is usually with high IoU levels in the initial situation. Geometrically speaking, the regression cases of Faster R-CNN are likely to place in the basins of Fig. 4, where IoU, GIoU and DIoU losses all have good performance. Therefore, GIoU loss has very small gain than the baseline IoU loss, as shown in Table 3. But our DIoU and CIoU losses still contribute to performance improvements than IoU and GIoU losses in terms of AP, AP75, APmedium and APlarge. Especially the gains by CIoU loss are very significant. From Fig. 7, one can easily find more accurate detection boxes by CIoU loss than those by GIoU loss. One may have noticed that in terms of APsmall, CIoU loss is a little inferior to the original IoU loss, while DIoU loss is better than all the other losses. That is to say the consistency of aspect ratio may not contribute to the regression accuracy for small objects. Actually it is reasonable that for small objects, the central point distance is more important than aspect ratio for regression, and the aspect ratio may weaken the effect of normalized distance between the two boxes. Nevertheless, CIoU loss performs much better for medium and large objects, and for small objects, the adverse effects can be relieved by DIoU-NMS.

In Tables 1, 2 and 3, we report the results of CIoU loss cooperating with original NMS ($\mathcal{L}_{CIoU}$) and DIoU-NMS ($\mathcal{L}_{CIoU}(D)$), where the thresholds follow the default settings of original NMS, i.e., $\varepsilon=0.45$ for YOLO v3 and SSD, and $\varepsilon=0.50$ for Faster R-CNN. One can find that DIoU-NMS makes further performance improvements than original NMS for most cases. Fig. 8 shows that DIoU-NMS can better preserve the correct detection boxes, where YOLO v3 trained on PASCAL VOC is adopted to detect objects on MS COCO. To further validate the superiority of DIoU-NMS over original NMS, we conduct comparison experiments, where original NMS and DIoU-NMS are cooperated with YOLO v3 and SSD trained using CIoU loss. We present the comparison of original NMS and DIoU-NMS within a wide range of thresholds $[0.43,0.48]$. From Fig. 9, one can see that DIoU-NMS is better than original NMS for every threshold. Furthermore, it is worth noting that even the worst performance of DIoU-NMS is at least comparable or better than the best performance of original NMS. That is to say our DIoU-NMS can generally perform better than original NMS even without carefully tuning the threshold $\varepsilon$.