SiamMask

NEW: now including code for both training and inference!

This is the official implementation with training code for SiamMask (CVPR2019). For technical details, please refer to:

SiamMask: A Framework for Fast Online Object Tracking and Segmentation <br />
Weiming Hu, Qiang Wang*, Li Zhang*, Luca Bertinetto*, Philip H.S. Torr (* denotes equal contribution) <br />
TPAMI 2023 <br />
[Paper] [ArXiv] <br />

Fast Online Object Tracking and Segmentation: A Unifying Approach <br />
Qiang Wang*, Li Zhang*, Luca Bertinetto*, Weiming Hu, Philip H.S. Torr (* denotes equal contribution) <br />
CVPR 2019 <br />
[Paper] [Video] [Project Page] <br />

<div align="center"> <img src="http://www.robots.ox.ac.uk/~qwang/SiamMask/img/SiamMask.jpg" width="600px" /> </div>

Bibtex

If you find this code useful, please consider citing:

@article{hu2023siammask,
  title={Siammask: A framework for fast online object tracking and segmentation},
  author={Hu, Weiming and Wang, Qiang and Zhang, Li and Bertinetto, Luca and Torr, Philip HS},
  journal={IEEE Transactions on Pattern Analysis and Machine Intelligence},
  volume={45},
  number={3},
  pages={3072--3089},
  year={2023},
  publisher={IEEE}
}

@inproceedings{wang2019fast,
    title={Fast online object tracking and segmentation: A unifying approach},
    author={Wang, Qiang and Zhang, Li and Bertinetto, Luca and Hu, Weiming and Torr, Philip HS},
    booktitle={Proceedings of the IEEE conference on computer vision and pattern recognition},
    year={2019}
}

Contents

1. Environment Setup
2. Demo
3. Testing Models
4. Training Models

Environment setup

This code has been tested on Ubuntu 16.04, Python 3.6, Pytorch 0.4.1, CUDA 9.2, RTX 2080 GPUs

• Clone the repository

git clone https://github.com/foolwood/SiamMask.git && cd SiamMask
export SiamMask=$PWD

• Setup python environment

conda create -n siammask python=3.6
source activate siammask
pip install -r requirements.txt
bash make.sh

• Add the project to your PYTHONPATH

export PYTHONPATH=$PWD:$PYTHONPATH

Demo

• Setup your environment
• Download the SiamMask model

shellCopy
cd $SiamMask/experiments/siammask_sharp
wget http://www.robots.ox.ac.uk/~qwang/SiamMask_VOT.pth
wget http://www.robots.ox.ac.uk/~qwang/SiamMask_DAVIS.pth

• Run demo.py

shellCopy
cd $SiamMask/experiments/siammask_sharp
export PYTHONPATH=$PWD:$PYTHONPATH
python ../../tools/demo.py --resume SiamMask_DAVIS.pth --config config_davis.json

<div align="center"> <img src="http://www.robots.ox.ac.uk/~qwang/SiamMask/img/SiamMask_demo.gif" width="500px" /> </div>

Testing

• Setup your environment
• Download test data

shellCopy
cd $SiamMask/data
sudo apt-get install jq
bash get_test_data.sh

• Download pretrained models

shellCopy
cd $SiamMask/experiments/siammask_sharp
wget http://www.robots.ox.ac.uk/~qwang/SiamMask_VOT.pth
wget http://www.robots.ox.ac.uk/~qwang/SiamMask_VOT_LD.pth
wget http://www.robots.ox.ac.uk/~qwang/SiamMask_DAVIS.pth

• Evaluate performance on VOT

shellCopy
bash test_mask_refine.sh config_vot.json SiamMask_VOT.pth VOT2016 0
bash test_mask_refine.sh config_vot.json SiamMask_VOT.pth VOT2018 0
bash test_mask_refine.sh config_vot.json SiamMask_VOT.pth VOT2019 0
bash test_mask_refine.sh config_vot18.json SiamMask_VOT_LD.pth VOT2016 0
bash test_mask_refine.sh config_vot18.json SiamMask_VOT_LD.pth VOT2018 0
python ../../tools/eval.py --dataset VOT2016 --tracker_prefix C --result_dir ./test/VOT2016
python ../../tools/eval.py --dataset VOT2018 --tracker_prefix C --result_dir ./test/VOT2018
python ../../tools/eval.py --dataset VOT2019 --tracker_prefix C --result_dir ./test/VOT2019

• Evaluate performance on DAVIS (less than 50s)

shellCopy
bash test_mask_refine.sh config_davis.json SiamMask_DAVIS.pth DAVIS2016 0
bash test_mask_refine.sh config_davis.json SiamMask_DAVIS.pth DAVIS2017 0

• Evaluate performance on Youtube-VOS (need download data from website)

shellCopy
bash test_mask_refine.sh config_davis.json SiamMask_DAVIS.pth ytb_vos 0

Results

These are the reproduction results from this repository. All results can be downloaded from our project page.

<sub>Tracker</sub>	<sub>VOT2016</br>EAO / A / R</sub>	<sub>VOT2018</br>EAO / A / R</sub>	<sub>DAVIS2016</br>J / F</sub>	<sub>DAVIS2017</br>J / F</sub>	<sub>Youtube-VOS</br>J_s / J_u / F_s / F_u</sub>	<sub>Speed</sub>
<sub>SiamMask-box</sub>	<sub>0.412/0.623/0.233</sub>	<sub>0.363/0.584/0.300</sub>	- / -	- / -	- / - / - / -	<sub>77 FPS</sub>
<sub>SiamMask</sub>	<sub>0.433/0.639/0.214</sub>	<sub>0.380/0.609/0.276</sub>	<sub>0.713/0.674</sub>	<sub>0.543/0.585</sub>	<sub>0.602/0.451/0.582/0.477</sub>	<sub>56 FPS</sub>
<sub>SiamMask-LD</sub>	<sub>0.455/0.634/0.219</sub>	<sub>0.423/0.615/0.248</sub>	- / -	- / -	- / - / - / -	<sub>56 FPS</sub>

Note:

• Speed are tested on a NVIDIA RTX 2080.
• -box reports an axis-aligned bounding box from the box branch.
• -LD means training with large dataset (ytb-bb+ytb-vos+vid+coco+det).

Training

Training Data

• Download the Youtube-VOS,
  COCO,
  ImageNet-DET,
  and ImageNet-VID.
• Preprocess each datasets according the readme files.

Download the pre-trained model (174 MB)

(This model was trained on the ImageNet-1k Dataset)

cd $SiamMask/experiments
wget http://www.robots.ox.ac.uk/~qwang/resnet.model
ls | grep siam | xargs -I {} cp resnet.model {}

Training SiamMask base model

• Setup your environment
• From the experiment directory, run

cd $SiamMask/experiments/siammask_base/
bash run.sh

• Training takes about 10 hours in our 4 Tesla V100 GPUs.
• If you experience out-of-memory errors, you can reduce the batch size in run.sh.
• You can view progress on Tensorboard (logs are at <experiment_dir>/logs/)
• After training, you can test checkpoints on VOT dataset.

shellCopy
bash test_all.sh -s 1 -e 20 -d VOT2018 -g 4  # test all snapshots with 4 GPUs

• Select best model for hyperparametric search.

shellCopy
#bash test_all.sh -m [best_test_model] -d VOT2018 -n [thread_num] -g [gpu_num] # 8 threads with 4 GPUS
bash test_all.sh -m snapshot/checkpoint_e12.pth -d VOT2018 -n 8 -g 4 # 8 threads with 4 GPUS

Training SiamMask model with the Refine module

• Setup your environment
• In the experiment file, train with the best SiamMask base model

cd $SiamMask/experiments/siammask_sharp
bash run.sh <best_base_model>
bash run.sh checkpoint_e12.pth

• You can view progress on Tensorboard (logs are at <experiment_dir>/logs/)
• After training, you can test checkpoints on VOT dataset

shellCopy
bash test_all.sh -s 1 -e 20 -d VOT2018 -g 4

Training SiamRPN++ model (unofficial)

• Setup your environment
• From the experiment directory, run

cd $SiamMask/experiments/siamrpn_resnet
bash run.sh

• You can view progress on Tensorboard (logs are at <experiment_dir>/logs/)
• After training, you can test checkpoints on VOT dataset

shellCopy
bash test_all.sh -h
bash test_all.sh -s 1 -e 20 -d VOT2018 -g 4

License

Licensed under an MIT license.