Project Website | arXiv | Dataset
- Clone C3Po.
git clone --recursive git@github.com:c3po-correspondence/C3Po.git
cd C3Po- Create the environment, here we show an example using conda.
conda create -n c3po python=3.11 cmake=3.14.0
conda activate c3po
pip install torch==2.5.1 torchvision==0.20.1 torchaudio==2.5.1 --index-url https://download.pytorch.org/whl/cu121 # use the correct version for you
pip install -r requirements.txt- Compile the CUDA kernels for RoPE.
# C3Po relies on RoPE positional embeddings for which you can compile some CUDA kernels for faster runtime
cd croco/models/curope/
python setup.py build_ext --inplace
cd ../../../The full dataset is available on HuggingFace. It provides a complete overview of the dataset, including its structure, the contents of each file, and code for visualizing photo-plan pairs and their correspondences and camera poses.
Pre-trained model weights:
C3Po.pth
To run the demo, download the model checkpoint and save the weights in checkpoints/.
After saving the model weights in checkpoints/, run demo.ipynb to visualize plan-photo pairs, predicted correspondences, and ground truth correspondences.
- Download DUSt3R_ViTLarge_BaseDecoder_512_dpt.pth to
checkpoints/. - Run the following command to start training.
torchrun --nproc_per_node 8 train.py \
--train_dataset="C3(image_pairs_dir='C3/image_pairs', geometric_dir='C3/geometric', visual_dir='C3/visual', split='train', resolution=[(512, 512)], augmentation_factor=3)" \
--val_dataset="C3(image_pairs_dir='C3/image_pairs', geometric_dir='C3/geometric', visual_dir='C3/visual', split='val', resolution=[(512, 512)], augmentation_factor=1)" \
--test_dataset="C3(image_pairs_dir='C3/image_pairs', geometric_dir='C3/geometric', visual_dir='C3/visual', split='test', resolution=[(512, 512)], augmentation_factor=1)" \
--train_criterion="ConfLoss(CorrespondenceLoss(L21), alpha=0.2)" \
--test_criterion="CorrespondenceLoss(L21)" \
--model="AsymmetricCroCo3DStereo(pos_embed='RoPE100', patch_embed_cls='ManyAR_PatchEmbed', img_size=(512, 512), head_type='dpt', output_mode='pts3d', depth_mode=('exp', -inf, inf), conf_mode=('exp', 1, inf), enc_embed_dim=1024, enc_depth=24, enc_num_heads=16, dec_embed_dim=768, dec_depth=12, dec_num_heads=12)" \
--pretrained="checkpoints/DUSt3R_ViTLarge_BaseDecoder_512_dpt.pth" \
--lr=1e-04 --min_lr=1e-06 --warmup_epochs=3 --epochs=500 --train_batch_size=2 --test_batch_size=1 --accum_iter=3 \
--save_freq=1 --keep_freq=1 --eval_freq=1 --num_workers=12 \
--output_dir="checkpoints/c3po"Use the command below to evaluate correspondence accuracy. To additionally evaluate camera pose estimation performance, include the --eval-camera-poses flag.
python eval.py \
--weights "./checkpoints/C3Po.pth" \
--image-pairs-dir "C3/image_pairs" \
--geometric-dir "C3/geometric" \
--visual-dir "C3/visual" \
--split test \
--batch-size 1 \
# --eval-camera-poses \
--output "./results.json"@inproceedings{
huang2025c3po,
title={C3Po: Cross-View Cross-Modality Correspondence by Pointmap Prediction},
author={Huang, Kuan Wei and Li, Brandon and Hariharan, Bharath and Snavely, Noah},
booktitle={Advances in Neural Information Processing Systems},
volume={38},
year={2025}
}
