Low-Level Vision

Let's enhance! Our team has been working on various image/video restoration and enhancement problems such as super-resolution, denoising, low-light enhancement etc. Some notable methods developed by us include SRCNN, ESRGAN, EDVR, BasicVSR, GLEAN, CodeFormer and Zero-DCE.

Super-Resolution

• Exploiting Diffusion Prior for Real-World Image Super-Resolution
  J. Wang, Z. Yue, S. Zhou, K. C. K. Chan, C. C. Loy
  Technical report, arXiv:2305.07015, 2023
  [arXiv] [Project Page]
• Learning Generative Structure Prior for Blind Text Image Super-resolution
  X. Li, W. Zuo, C. C. Loy
  in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2023 (CVPR)
  [PDF] [arXiv] [Supplementary Material] [Project Page]
• Reference-based Image and Video Super-Resolution via C²-Matching
  Y. Jiang, K. C. K. Chan, X. Wang, C. C. Loy, Z. Liu
  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023 (TPAMI)
  [arXiv] [Project Page]
• DifFace: Blind Face Restoration with Diffused Error Contraction
  Z. Yue, C. C. Loy
  Technical report, arXiv:2212.06512, 2022
  [arXiv] [Project Page] [Demo]
• Towards Robust Blind Face Restoration with Codebook Lookup Transformer
  S. Zhou, K. C. K. Chan, C. Li, C. C. Loy
  in Proceedings of Neural Information Processing Systems, 2022 (NeurIPS)
  [arXiv] [Project Page] [Demo]
• GLEAN: Generative Latent Bank for Image Super-Resolution and Beyond
  K. C. K. Chan, X. Wang, X. Xu, J. Gu, C. C. Loy
  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022 (TPAMI)
  [DOI] [arXiv] [Project Page]
• Investigating Trade-offs in Real-World Video Super-Resolution
  K. C. K. Chan, S. Zhou, X. Xu, C. C. Loy
  in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2022 (CVPR)
  [PDF] [arXiv] [Supplementary Material] [Project Page]
• BasicVSR++: Improving Video Super-Resolution with Enhanced Propagation and Alignment
  K. C. K. Chan, S. Zhou, X. Xu, C. C. Loy
  in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2022 (CVPR)
  [PDF] [arXiv] [Supplementary Material] [Project Page]
• GLEAN: Generative Latent Bank for Large-Factor Image Super-Resolution
  K. C. K. Chan, X. Wang, X. Xu, J. Gu, C. C. Loy
  in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2021 (CVPR, Oral)
  [PDF] [arXiv] [Supplementary Material] [Project Page]
• BasicVSR: The Search for Essential Components in Video Super-Resolution and Beyond
  K. C. K. Chan, X. Wang, K. Yu, C. Dong, C. C. Loy
  in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2021 (CVPR)
  [PDF] [arXiv] [Supplementary Material] [Project Page]
• Robust Reference-based Super-Resolution via C²-Matching
  Y. Jiang, K. C. K. Chan, X. Wang, C. C. Loy, Z. Liu
  in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2021 (CVPR)
  [PDF] [arXiv] [Supplementary Material] [Project Page]
• Understanding Deformable Alignment in Video Super-Resolution
  K. C. K. Chan, X. Wang, K. Yu, C. Dong, C. C. Loy
  in Proceedings of AAAI Conference on Artificial Intelligence, 2021 (AAAI)
  [arXiv] [Project Page]
• Cross-Scale Internal Graph Neural Network for Image Super-Resolution
  S. Zhou, J. Zhang, W. Zuo, C. C. Loy
  in Proceedings of Neural Information Processing Systems, 2020 (NeurIPS)
  [PDF] [arXiv] [Supplementary Material] [Project Page]
• Non-Local Recurrent Network for Image Restoration
  D. Liu, B. Wen, Y. Fan, C. C. Loy, T. S. Huang
  in Proceedings of Neural Information Processing Systems, 2018 (NeurIPS)
  [PDF] [arXiv] [Project Page]
• ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks
  X. Wang, K. Yu, S. Wu, J. Gu, Y. Liu, C. Dong, Y. Qiao, C. C. Loy
  in Workshop Proceedings of European Conference on Computer Vision, 2018 (ECCVW)
  [PDF] [arXiv] [Project Page]

Restoration | Enhancement

• Flare7K++: Mixing Synthetic and Real Datasets for Nighttime Flare Removal and Beyond
  Y. Dai, C. Li, S. Zhou, R. Feng, Y. Luo, C. C. Loy
  Technical report, arXiv:2306.04236, 2023
  [arXiv] [Project Page]
• Iterative Prompt Learning for Unsupervised Backlit Image Enhancement
  Z. Liang, C. Li, S. Zhou, R. Feng, C. C. Loy
  Technical report, arXiv:2303.17569, 2023
  [arXiv] [Project Page]
• Nighttime Smartphone Reflective Flare Removal using Optical Center Symmetry Prior
  Y. Dai, Y. Luo, S. Zhou, C. Li, C. C. Loy
  in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2023 (CVPR, Highlight)
  [PDF] [arXiv] [Supplementary Material] [Project Page]
• Generating Aligned Pseudo-Supervision from Non-Aligned Data for Image Restoration in Under-Display Camera
  R. Feng, C. Li, H. Chen, S. Li, J. Gu, C. C. Loy
  in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2023 (CVPR)
  [PDF] [arXiv] [Supplementary Material] [Project Page]
• Flexible Piecewise Curves Estimation for Photo Enhancement
  C. Li, C. Guo, Q. Ai, S. Zhou, C. C. Loy
  in Workshop Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, NTIRE, 2023 (CVPRW)
  [arXiv]
• Embedding Fourier for Ultra-High-Definition Low-Light Image Enhancement
  C. Li, C. Guo, M. Zhou, Z. Liang, S. Zhou, R. Feng, C. C. Loy
  International Conference on Learning Representations, 2023 (ICLR, Oral)
  [arXiv] [Project Page]
• Exploring CLIP for Assessing the Look and Feel of Images
  J. Wang, K. C. K. Chan, C. C. Loy
  in Proceedings of AAAI Conference on Artificial Intelligence, 2023 (AAAI)
  [arXiv] [Project Page]
• AnimeRun: 2D Animation Visual Correspondence from Open Source 3D Movies
  S-Y. Li, Y. Li, B. Li, C. Dong, Z. Liu, C. C. Loy
  in Proceedings of Neural Information Processing Systems, 2022 (NeurIPS)
  [arXiv] [Project Page]
• Flare7K: A Phenomenological Nighttime Flare Removal Dataset
  Y. Dai, C. Li, S. Zhou, R. Feng, C. C. Loy
  in Proceedings of Neural Information Processing Systems, 2022 (NeurIPS)
  [arXiv] [Project Page]
• Deep Fourier Up-Sampling
  M. Zhou, H. Yu, J. Huang, F. Zhao, J. Gu, C. C. Loy, D. Meng, C. Li
  in Proceedings of Neural Information Processing Systems, 2022 (NeurIPS)
  [arXiv] [Project Page]
• CuDi: Curve Distillation for Efficient and Controllable Exposure Adjustment
  C. Li, C. Guo, R. Feng, S. Zhou, C. C. Loy
  Technical report, arXiv:2207.14273, 2022
  [arXiv] [Project Page]
• LEDNet: Joint Low-light Enhancement and Deblurring in the Dark
  S. Zhou, C. Li, C. C. Loy
  European Conference on Computer Vision, 2022 (ECCV)
  [PDF] [arXiv] [Supplementary Material] [Project Page]
• On the Generalization of BasicVSR++ to Video Deblurring and Denoising
  K. C. K. Chan, S. Zhou, X. Xu, C. C. Loy
  Technical report, arXiv:2204.05308, 2022
  [arXiv] [Project Page]
• Low-Light Image and Video Enhancement Using Deep Learning: A Survey
  C. Li, C. Guo, L. Han, J. Jiang, M. Cheng, J. Gu, C. C. Loy
  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021 (TPAMI)
  [DOI] [arXiv] [Project Page]
• Temporally Consistent Video Colorization with Deep Feature Propagation and Self-regularization Learning
  Y. Liu, H. Zhao, K. C. K. Chan, X. Wang, C. C. Loy, Y. Qiao, C. Dong
  Technical report, arXiv:2110.04562, 2021
  [arXiv]
• Exploiting Deep Generative Prior for Versatile Image Restoration and Manipulation
  X. Pan, X. Zhan, B. Dai, D. Lin, C. C. Loy, P. Luo
  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021 (TPAMI)
  [DOI] [Project Page]
• ReconfigISP: Reconfigurable Camera Image Processing Pipeline
  K. Yu, Z. Li, Y. Peng, C. C. Loy, J. Gu
  in Proceedings of IEEE/CVF International Conference on Computer Vision, 2021 (ICCV)
  [PDF] [arXiv] [Supplementary Material] [Project Page]
• Path-Restore: Learning Network Path Selection for Image Restoration
  K. Yu, X. Wang, C. Dong, X. Tang, C. C. Loy
  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021 (TPAMI)
  [DOI] [arXiv] [Project Page]
• Learning to Enhance Low-Light Image via Zero-Reference Deep Curve Estimation
  C. Li, C. Guo, C. C. Loy
  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021 (TPAMI)
  [DOI] [arXiv] [Project Page]
• Removing Diffraction Image Artifacts in Under-Display Camera via Dynamic Skip Connection Network
  R. Feng, C. Li, H. Chen, S. Li, C. C. Loy, J. Gu
  in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2021 (CVPR)
  [PDF] [arXiv] [Supplementary Material] [Project Page]
• Deep Animation Video Interpolation in the Wild
  S-Y. Li, S. Zhao, W. Yu, W. Sun, D. Metaxas, C. C. Loy, Z. Liu
  in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2021 (CVPR)
  [PDF] [arXiv] [Supplementary Material] [Project Page]
• Exploiting Deep Generative Prior for Versatile Image Restoration and Manipulation
  X. Pan, X. Zhan, B. Dai, D. Lin, C. C. Loy, P. Luo
  European Conference on Computer Vision, 2020 (ECCV, Oral)
  [PDF] [arXiv] [Project Page]
• Zero-Reference Deep Curve Estimation for Low-Light Image Enhancement
  C. Guo, C. Li, J. Guo, C. C. Loy, J. Hou, S. Kwong, R. Cong
  in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2020 (CVPR)
  [PDF] [arXiv] [Supplementary Material] [Project Page]
• EDVR: Video Restoration with Enhanced Deformable Convolutional Networks
  X. Wang, C. K. Chan, K. Yu, C. Dong, X. Tang, C. C. Loy
  in Workshop Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, NTIRE, 2019 (CVPRW)
  [PDF] [arXiv] [Project Page]
• Deep Network Interpolation for Continuous Imagery Effect Transition
  X. Wang, K. Yu, C. Dong, X. Tang, C. C. Loy
  in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2019 (CVPR)
  [PDF] [arXiv] [Project Page]

Optical Flow Estimation

• LiteFlowNet3: Resolving Correspondence Ambiguity for More Accurate Optical Flow Estimation
  T.-W. Hui, C. C. Loy
  European Conference on Computer Vision, 2020 (ECCV)
  [PDF] [arXiv] [Supplementary Material] [Project Page]
• A Lightweight Optical Flow CNN - Revisiting Data Fidelity and Regularization
  T.-W. Hui, X. Tang, C. C. Loy
  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020 (TPAMI)
  [DOI] [arXiv] [Project Page]

Editing and Generation

We like algorithms that could generate new visual contents, e.g., face generation, face reenactment, image inpainting, scene de-occlusion, etc.

Face Editing

• Collaborative Diffusion for Multi-Modal Face Generation and Editing
  Z. Huang, K.C.K. Chan, Y. Jiang, Z. Liu
  in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2023 (CVPR)
  [PDF] [arXiv] [Supplementary Material] [Project Page]
• CelebV-Text: A Large-Scale Facial Text-Video Dataset
  J. Yu, H. Zhu, L. Jiang, C. C. Loy, W. Cai, W. Wu
  in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2023 (CVPR)
  [PDF] [arXiv] [Supplementary Material] [Project Page]
• BeautyREC: Robust, Efficient, and Content-preserving Makeup Transfer
  Q. Yan, C. Guo, J. Zhao, Y. Dai, C. C. Loy, C. Li
  in Workshop Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, NTIRE, 2023 (CVPRW)
  [arXiv] [Project Page]
• StyleGANEX: StyleGAN-Based Manipulation Beyond Cropped Aligned Faces
  S. Yang, L. Jiang, Z. Liu, C. C. Loy
  Technical report, arXiv:2303.06146, 2023
  [arXiv] [Project Page]
• VToonify: Controllable High-Resolution Portrait Video Style Transfer
  S. Yang, L. Jiang, Z. Liu, C. C. Loy
  ACM Transactions on Graphics, 2022 (SIGGRAPH ASIA - TOG)
  [DOI] [arXiv] [Project Page] [YouTube]
• Audio-driven Dubbing for User Generated Contents via Style-aware Semi-parametric Synthesis
  L. Song, W. Wu, C. Fu, C. C. Loy, R. He
  IEEE Transactions on Circuits and Systems for Video Technology, 2022 (TCVST)
  [DOI]
• CelebV-HQ: A Large-Scale Video Facial Attributes Dataset
  H. Zhu, W. Wu, W. Zhu, L. Jiang, S. Tang, L. Zhang, Z. Liu, C. C. Loy
  European Conference on Computer Vision, 2022 (ECCV)
  [PDF] [arXiv] [Supplementary Material] [Project Page]
• TransEditor: Transformer-Based Dual-Space GAN for Highly Controllable Facial Editing
  Y. Xu, Y. Yin, L. Jiang, Q. Wu, C. Zheng, C. C. Loy, B. Dai, W. Wu
  in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2022 (CVPR)
  [PDF] [arXiv] [Supplementary Material] [Project Page]
• Talking Faces: Audio-to-Video Face Generation
  Y. Wang, L. Song, W. Wu, C. Qian, R. He, C. C. Loy
  In C. Rathgeb, R. Tolosana, R. Vera-Rodriguez, C. Busch (Eds.), Handbook of Digital Face Manipulation and Detection, Springer, 2022
  [Book Link]
• Everybody’s Talkin’: Let Me Talk as You Want
  L. Song, W. Wu, C. Qian, R. He, C. C. Loy
  IEEE Transactions on Information Forensics and Security, 2022 (TIFS)
  [DOI] [arXiv] [Project Page]
• Talk-to-Edit: Fine-Grained Facial Editing via Dialog
  Y. Jiang, Z. Huang, X. Pan, C. C. Loy, Z. Liu
  in Proceedings of IEEE/CVF International Conference on Computer Vision, 2021 (ICCV)
  [PDF] [arXiv] [Supplementary Material] [Project Page]
• Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representation
  H. Zhou, Y. Sun, W. Wu, C. C. Loy, X. Wang, Z. Liu
  in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2021 (CVPR)
  [PDF] [arXiv] [Supplementary Material] [Project Page]
• Pareidolia Face Reenactment
  L. Song, W. Wu, C. Fu, C. Qian, C. C. Loy, R. He
  in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2021 (CVPR)
  [PDF] [arXiv] [Supplementary Material] [Project Page] [YouTube]
• Audio-Driven Emotional Video Portraits
  X. Ji, H. Zhou, K. Wang, W. Wu, X. Cao, C. C. Loy, F. Xu
  in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2021 (CVPR)
  [PDF] [arXiv] [Supplementary Material] [Project Page]
• MEAD: A Large-scale Audio-visual Dataset for Emotional Talking Face Generation
  K. Wang, Q. Wu, L. Song, Z. Yang, W. Wu, C. Qian, R. He, Y. Qiao, C. C. Loy
  European Conference on Computer Vision, 2020 (ECCV)
  [PDF] [Supplementary Material] [Project Page]
• One-shot Face Reenactment
  Y. Zhang, S. Zhang, Y. He, C. Li, C. C. Loy, Z. Liu
  in Proceedings of British Machine Vision Conference, 2019 (BMVC, Spotlight)
  [PDF] [arXiv] [Project Page]
• Instance-level Facial Attributes Transfer with Geometry-aware Flow
  W. Yin, Z. Liu, C. C. Loy
  in Proceedings of AAAI Conference on Artificial Intelligence, 2019 (AAAI, Spotlight)
  [PDF] [arXiv] [Project Page]
• ReenactGAN: Learning to Reenact Faces via Boundary Transfer
  W. Wu, Y. Zhang, C. Li, C. Qian, C. C. Loy
  in Proceedings of European Conference on Computer Vision, 2018 (ECCV)
  [PDF] [arXiv] [Project Page] [YouTube]

Image and Video Generation

• GP-UNIT: Generative Prior for Versatile Unsupervised Image-to-Image Translation
  S. Yang, L. Jiang, Z. Liu, C. C. Loy
  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023 (TPAMI)
  [arXiv] [Project Page]
• Text2Performer: Text-Driven Human Video Generation
  Y. Jiang, S. Yang, T. L. Koh, W. Wu, C. C. Loy, Z. Liu
  Technical report, arXiv:2304.08483, 2023
  [arXiv] [Project Page]
• ReVersion: Diffusion-Based Relation Inversion from Images
  Z. Huang, T. Wu, Y. Jiang, K. C. K. Chan, Z. Liu
  Technical report, arXiv:2303.13509, 2023
  [arXiv] [Project Page]
• The Nuts and Bolts of Adopting Transformer in GANs
  R. Xu and X. Xu and K. Chen and B. Zhou and C. C. Loy
  in Workshop Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, AI4CC, 2023 (CVPRW)
  [arXiv] [Project Page]
• Text2Light: Zero-Shot Text-Driven HDR Panorama Generation
  Z. Chen, G. Wang, Z. Liu
  ACM Transactions on Graphics, 2022 (SIGGRAPH ASIA - TOG)
  [arXiv] [Project Page] [YouTube]
• StyleFaceV: Face Video Generation via Decomposing and Recomposing Pretrained StyleGAN3
  H. Qiu, Y. Jiang, H. Zhou, W. Wu, Z. Liu
  Technical report, arXiv:2208.07862, 2022
  [arXiv] [Project Page]
• BRACE: The Breakdancing Competition Dataset for Dance Motion Synthesis
  D. Moltisanti, J. Wu, B. Dai, C. C. Loy
  European Conference on Computer Vision, 2022 (ECCV)
  [PDF] [arXiv] [Supplementary Material] [Project Page]
• StyleLight: HDR Panorama Generation for Lighting Estimation and Editing
  G. Wang, Y. Yang, C. C. Loy, Z. Liu
  European Conference on Computer Vision, 2022 (ECCV)
  [PDF] [arXiv] [Supplementary Material] [Project Page]
• Fast-Vid2Vid: Spatial-Temporal Compression for Video-to-Video Synthesis
  L. Zhuo, G. Wang, S. Li, W. Wu, Z. Liu
  European Conference on Computer Vision, 2022 (ECCV)
  [PDF] [arXiv] [Supplementary Material] [Project Page]
• StyleGAN-Human: A Data-Centric Odyssey of Human Generation
  J. Fu, S. Li, Y. Jiang, K.-Y. Lin, C. Qian, C. C. Loy, W. Wu, Z. Liu
  European Conference on Computer Vision, 2022 (ECCV)
  [PDF] [arXiv] [Supplementary Material] [Project Page] [YouTube]
• Text2Human: Text-Driven Controllable Human Image Generation
  Y. Jiang, S. Yang, H. Qiu, W. Wu, C. C. Loy, Z. Liu
  ACM Transactions on Graphics, 2022 (SIGGRAPH - TOG)
  [DOI] [arXiv] [Project Page] [YouTube]
• Pastiche Master: Exemplar-Based High-Resolution Portrait Style Transfer
  S. Yang, L. Jiang, Z. Liu, C. C. Loy
  in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2022 (CVPR)
  [PDF] [arXiv] [Supplementary Material] [Project Page] [YouTube]
• Unsupervised Image-to-Image Translation with Generative Prior
  S. Yang, L. Jiang, Z. Liu, C. C. Loy
  in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2022 (CVPR)
  [PDF] [arXiv] [Supplementary Material] [Project Page]
• Bailando: 3D Dance Generation by Actor-Critic GPT with Choreographic Memory
  S-Y. Li, W. Yu, T. Gu, C. Lin, Q. Wang, C. Qian, C. C. Loy, Z. Liu
  in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2022 (CVPR, Oral)
  [PDF] [arXiv] [Supplementary Material] [Project Page] [YouTube]
• Full-Range Virtual Try-On with Recurrent Tri-Level Transformation
  H. Yang, X. Yu, Z. Liu
  in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2022 (CVPR)
  [PDF] [Supplementary Material]
• Towards Diverse and Natural Scene-aware 3D Human Motion Synthesis
  J. Wang, Y. Rong, J. Liu, S. Yan, D. Lin, B. Dai
  in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2022 (CVPR)
  [PDF] [arXiv] [Supplementary Material]
• MoCaNet: Motion Retargeting in-the-wild via Canonicalization Networks
  W. Zhu, Z. Yang, Z. Di, W. Wu, Y. Wang, C. C. Loy
  in Proceedings of AAAI Conference on Artificial Intelligence, 2022 (AAAI)
  [arXiv] [Project Page]
• Deceive D: Adaptive Pseudo Augmentation for GAN Training with Limited Data
  L. Jiang, B. Dai, W. Wu, C. C. Loy
  in Proceedings of Neural Information Processing Systems, 2021 (NeurIPS)
  [PDF] [arXiv] [Project Page] [YouTube]
• Focal Frequency Loss for Image Reconstruction and Synthesis
  L. Jiang, B. Dai, W. Wu, C. C. Loy
  in Proceedings of IEEE/CVF International Conference on Computer Vision, 2021 (ICCV)
  [PDF] [arXiv] [Supplementary Material] [Project Page]
• BlockPlanner: City Block Generation with Vectorized Graph Representation
  L. Xu, Y. Xiangli, A. Rao, N. Zhao, B. Dai, Z. Liu, D. Lin
  in Proceedings of IEEE/CVF International Conference on Computer Vision, 2021 (ICCV)
  [PDF] [Supplementary Material]
• Positional Encoding as Spatial Inductive Bias in GANs
  R. Xu, X. Wang, K. Chen, B. Zhou, C. C. Loy
  in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2021 (CVPR)
  [PDF] [arXiv] [Supplementary Material] [Project Page] [YouTube]
• Scene-aware Generative Network for Human Motion Synthesis
  J. Wang, S. Yan, B. Dai, D. Lin
  in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2021 (CVPR)
  [PDF] [Supplementary Material] [Project Page]
• Do 2D GANs Know 3D Shape? Unsupervised 3D Shape Reconstruction from 2D Image GANs
  X. Pan, B. Dai, Z. Liu, C. C. Loy, P. Luo
  International Conference on Learning Representations, 2021 (ICLR, Oral)
  [PDF] [arXiv] [Project Page]
• Texture Memory-Augmented Deep Patch-Based Image Inpainting
  R. Xu, M. Guo, J. Wang, X. Li, B. Zhou, C. C. Loy
  IEEE Transactions on Image Processing, 2021 (TIP)
  [DOI] [arXiv]
• Exploiting Deep Generative Prior for Versatile Image Restoration and Manipulation
  X. Pan, X. Zhan, B. Dai, D. Lin, C. C. Loy, P. Luo
  European Conference on Computer Vision, 2020 (ECCV, Oral)
  [PDF] [arXiv] [Project Page]
• TSIT: A Simple and Versatile Framework for Image-to-Image Translation
  L. Jiang, C. Zhang, M. Huang, C. Liu, J. Shi, C. C. Loy
  European Conference on Computer Vision, 2020 (ECCV, Spotlight)
  [PDF] [arXiv] [Supplementary Material] [Project Page]
• Self-Supervised Scene De-occlusion
  X. Zhan, X. Pan, B. Dai, Z. Liu, D. Lin, C. C. Loy
  in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2020 (CVPR, Oral)
  [PDF] [arXiv] [Supplementary Material] [Project Page] [YouTube]
• TransMoMo: Invariance-Driven Unsupervised Video Motion Retargeting
  Z. Yang, W. Zhu, W. Wu, C. Qian, Q. Zhou, B. Zhou, C. C. Loy
  in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2020 (CVPR)
  [PDF] [arXiv] [Supplementary Material] [Project Page]
• Real or Not Real, That is the Question
  Y. Xiangli, Y. Deng, B. Dai, C. C. Loy, D. Lin
  International Conference on Learning Representations, 2020 (ICLR, Spotlight)
  [PDF] [Project Page]
• High-Quality Video Generation from Static Structural Annotations
  L. Sheng, J. Pan, J. Guo, J. Shao, C. C. Loy
  International Journal of Computer Vision, 2020 (IJCV)
  [DOI]
• TransGaGa: Geometry-Aware Unsupervised Image-to-Image Translation
  W. Wu, K. Cao, C. Li, C. Qian, C. C. Loy
  in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2019 (CVPR)
  [PDF] [arXiv] [Supplementary Material] [Project Page]
• Deep Flow-Guided Video Inpainting
  R. Xu, X. Li, B. Zhou, C. C. Loy
  in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2019 (CVPR)
  [PDF] [arXiv] [Project Page]
• Dense Intrinsic Appearance Flow for Human Pose Transfer
  Y. Li, C. Huang, C. C. Loy
  in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2019 (CVPR)
  [PDF] [Supplementary Material] [Project Page]
• Disentangling Content and Style via Unsupervised Geometry Distillation
  W. Wu, K. Cao, C. Li, C. Qian, C. C. Loy
  International Conference on Learning Representations Workshop, 2019 (ICLRW)
  [PDF]

Simulation

• Towards Multi-Layered 3D Garments Animation
  Y. Shao, C. C. Loy, B. Dai
  Technical report, arXiv:2305.10418, 2023
  [arXiv] [Project Page]
• Transformer with Implicit Edges for Particle-based Physics Simulation
  Y. Shao, C. C. Loy, B. Dai
  European Conference on Computer Vision, 2022 (ECCV)
  [PDF] [arXiv] [Project Page]

Visual and Sound

• Learning Hierarchical Cross-Modal Association for Co-Speech Gesture Generation
  X. Liu, Q. Wu, H. Zhou, Y. Xu, R. Qian, X. Lin, X. Zhou, W. Wu, B. Dai, B. Zhou
  in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2022 (CVPR)
  [PDF] [arXiv] [Supplementary Material] [Project Page]
• Visual Sound Localization in-the-Wild by Cross-Modal Interference Erasing
  X. Liu, R. Qian, H. Zhou, W. lin, Z. Liu, B. Zhou, X. Zhou
  in Proceedings of AAAI Conference on Artificial Intelligence, 2022 (AAAI)
  [PDF] [arXiv]
• SepFusion: Finding Optimal Fusion Structures for Visual Sound Separation
  D. Zhou, X. Zhou, D. Hu, H. Zhou, L. Bai, Z. Liu, W. Ouyang
  in Proceedings of AAAI Conference on Artificial Intelligence, 2022 (AAAI)
  [PDF]
• Visually Informed Binaural Audio Generation without Binaural Audios
  X. Xu, H. Zhou, Z. Liu, B. Dai, X. Wang, D. Lin
  in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2021 (CVPR)
  [PDF] [arXiv] [Supplementary Material] [Project Page]

Image and Video Understanding

We explore effective and efficient methods to detect, segment and recognize objects in complex scenes.

Image Recognition

• Panoptic Video Scene Graph Generation
  J. Yang, W. Peng, X. Li, Z. Guo, L. Chen, Bo Li, Z. Ma, W. Zhang, K. Zhou, C. C. Loy, Z. Liu
  in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2023 (CVPR)
  [PDF]
• X-Learner: Learning Cross Sources and Tasks for Universal Visual Representation
  Y. He, Ge. Huang, S. Chen, J. Teng, K. Wang, Z. Yin, L. Sheng, Z. Liu, Y. Qiao, J. Shao
  European Conference on Computer Vision, 2022 (ECCV)
  [PDF] [arXiv] [Supplementary Material]
• Benchmarking Omni-Vision Representation through the Lens of Visual Realms
  Y. Zhang, Z. Yin, J. Shao, Z. Liu
  European Conference on Computer Vision, 2022 (ECCV)
  [PDF] [arXiv] [Supplementary Material][arXiv] [Project Page]
• Panoptic Scene Graph Generation
  J. Yang, Y. Z. Ang, Z. Guo, K. Zhou, W. Zhang, Z. Liu
  European Conference on Computer Vision, 2022 (ECCV)
  [PDF] [arXiv] [Supplementary Material] [Project Page]
• Bamboo: Building Mega-Scale Vision Dataset Continually with Human-Machine Synergy
  Y. Zhang, Q. Sun, Y. Zhou, Z. He, Z. Yin, K. Wang, L. Sheng, Y. Qiao, J. Shao, Z. Liu
  Technical report, arXiv:2203.07845, 2022
  [arXiv] [Project Page]
• Incorporating Convolution Designs into Visual Transformers
  K. Yuan, S. Guo, Z. Liu, A. Zhou, F. Yu, W. Wu
  in Proceedings of IEEE/CVF International Conference on Computer Vision, 2021 (ICCV)
  [PDF] [arXiv]
• Differentiable Dynamic Wirings for Neural Networks
  K. Yuan, Q. Li, S. Guo, D. Chen, A. Zhou, F. Yu, Z. Liu
  in Proceedings of IEEE/CVF International Conference on Computer Vision, 2021 (ICCV)
  [PDF]

Object Detection

• Aligning Bag of Regions for Open-Vocabulary Object Detection
  S. Wu, W. Zhang, S. Jin, W. Liu, C. C. Loy
  in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2023 (CVPR)
  [PDF] [arXiv] [Supplementary Material] [Project Page]
• Siamese DETR
  Z. Chen, G. Huang, W. Li, J. Teng, K. Wang, J. Shao, C. C. Loy, L. Sheng
  in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2023 (CVPR)
  [PDF] [arXiv] [Supplementary Material] [Project Page]
• Semi-Supervised and Long-Tailed Object Detection with CascadeMatch
  Y. Zang, K. Zhou, C. Huang, C. C. Loy
  International Journal of Computer Vision, 2022 (IJCV)
  [PDF] [arXiv] [Project Page]
• Open-Vocabulary DETR with Conditional Matching
  Y. Zang, W. Li, K. Zhou, C. Huang, C. C. Loy
  European Conference on Computer Vision, 2022 (ECCV, Oral)
  [PDF] [arXiv] [Supplementary Material] [Project Page]
• Few-Shot Object Detection via Association and Discrimination
  Y. Cao, J. Wang, Y. Jin, T. Wu, K. Chen, Z. Liu, D. Lin
  in Proceedings of Neural Information Processing Systems, 2021 (NeurIPS)
  [PDF] [arXiv] [Project Page]
• CARAFE++: Unified Content-Aware ReAssembly of FEatures
  J. Wang, K. Chen, R. Xu, Z. Liu, C. C. Loy, D. Lin
  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021 (TPAMI)
  [DOI] [arXiv]
• Side-Aware Boundary Localization for More Precise Object Detection
  J. Wang, W. Zhang, Y. Cao, K. Chen, J. Pang, T. Gong, J. Shi, C. C. Loy, D. Lin
  European Conference on Computer Vision, 2020 (ECCV)
  [PDF] [arXiv] [Supplementary Material]
• RGB-D Salient Object Detection with Cross-Modality Modulation and Selection
  C. Li, R. Cong, Y. Piao, Q. Xu, C. C. Loy
  European Conference on Computer Vision, 2020 (ECCV)
  [PDF] [arXiv] [Supplementary Material] [Project Page]
• Feature Pyramid Grids
  K. Chen, Y. Cao, C. C. Loy, D. Lin, C. Feichtenhofer
  Technical report, arXiv:2004.03580, 2020
  [arXiv]
• Prime Sample Attention in Object Detection
  Y. Cao, K. Chen, C. C. Loy, D. Lin
  in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2020 (CVPR)
  [PDF] [arXiv] [Supplementary Material] [Project Page]
• CARAFE: Content-Aware ReAssembly of FEatures
  J. Wang, K. Chen, R. Xu, Z. Liu, C. C. Loy, D. Lin
  in Proceedings of International Conference on Computer Vision, 2019 (ICCV, Oral)
  [PDF] [arXiv] [Supplementary Material]
• Region Proposal by Guided Anchoring
  J. Wang, K. Chen, S. Yang, C. C. Loy, D. Lin
  in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2019 (CVPR)
  [PDF] [arXiv] [Project Page]

Segmentation

• SAD: Segment Any RGBD
  J. Cen, Y. Wu, K. Wang, X. Li, J. Yang, Y. Pei, L. Kong, Z. Liu, Q. Chen
  Technical report, arXiv:2305.14207, 2023
  [arXiv] [Project Page]
• Transformer-Based Visual Segmentation: A Survey
  X. Li, H. Ding, W. Zhang, H. Yuan, J. Pang, G. Cheng, K. Chen, Z. Liu, C. C. Loy
  Technical report, arXiv:2304.09854, 2023
  [arXiv] [Project Page]
• Tube-Link: A Flexible Cross Tube Baseline for Universal Video Segmentation
  X. Li, H. Yuan, W. Zhang, G. Cheng, J. Pang, C. C. Loy
  Technical report, arXiv:2303.12782, 2023
  [arXiv] [Project Page]
• Betrayed by Captions: Joint Caption Grounding and Generation for Open Vocabulary Instance Segmentation
  J. Wu, X. Li, H. Ding, X. Li, G. Cheng, Y. Tong, C. C. Loy
  Technical report, arXiv:2301.00805, 2023
  [arXiv]
• Rethinking Range View Representation for LiDAR Segmentation
  L. Kong, Y. Liu, R. Chen, Y. Ma, X. Zhu, Y. Li, Y. Hou, Y. Qiao, Z. Liu
  Technical report, arXiv:2303.05367, 2023
  [arXiv] [Project Page]
• LaserMix for Semi-Supervised LiDAR Semantic Segmentation
  L. Kong, J. Ren, L. Pan, Z. Liu
  in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2023 (CVPR, Highlight)
  [PDF] [arXiv] [Supplementary Material] [Project Page]
• Video K-Net: A Simple, Strong, and Unified Baseline For End-to-End Dense Video Segmentation
  X. Li, W. Zhang, J. Pang, K. Chen, G. Cheng, Y. Tong, C. C. Loy
  in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2022 (CVPR, Oral)
  [PDF] [arXiv] [Supplementary Material] [Project Page]
• Delving into High-Quality Synthetic Face Occlusion Segmentation Datasets
  K. T. R. Voo, L. Jiang, C. C. Loy
  in Workshop Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, VDU, 2022 (CVPRW)
  [PDF] [arXiv] [Supplementary Material] [Project Page]
• K-Net: Towards Unified Image Segmentation
  W. Zhang, J. Pang, K. Chen, C. C. Loy
  in Proceedings of Neural Information Processing Systems, 2021 (NeurIPS)
  [PDF] [arXiv] [Project Page]
• FASA: Feature Augmentation and Sampling Adaptation for Long-Tailed Instance Segmentation
  Y. Zang, C. Huang, C. C. Loy
  in Proceedings of IEEE/CVF International Conference on Computer Vision, 2021 (ICCV)
  [PDF] [arXiv] [Supplementary Material] [Project Page]
• Seesaw Loss for Long-Tailed Instance Segmentation
  J. Wang, W. Zhang, Y. Zang, Y. Cao, J. Pang, T. Gong, K. Chen, Z. Liu, C. C. Loy, D. Lin
  in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2021 (CVPR)
  [PDF] [arXiv] [Supplementary Material]
• Hybrid Task Cascade for Instance Segmentation
  K. Chen, J. Pang, J. Wang, Y. Xiong, X. Li, S. Sun, W. Feng, Z. Liu, J. Shi, W. Ouyang, C. C. Loy, D. Lin
  in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2019 (CVPR)
  [PDF] [arXiv] [Project Page]
• Video Object Segmentation with Joint Re-identification and Attention-Aware Mask Propagation
  X. Li, C. C. Loy
  in Proceedings of European Conference on Computer Vision, 2018 (ECCV)
  [PDF] [arXiv]
• PSANet: Point-wise Spatial Attention Network for Scene Parsing
  H. Zhao, Y. Zhang, S. Liu, J. Shi, C. C. Loy, D. Lin, J. Jia
  in Proceedings of European Conference on Computer Vision, 2018 (ECCV)
  [PDF] [Project Page]

Tracking and Association

• TCTrack: Temporal Contexts for Aerial Tracking
  Z. Cao, Z. Huang, L. Pan, S. Zhang, Z. Liu, C. Fu
  in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2022 (CVPR)
  [PDF] [arXiv] [Supplementary Material] [Project Page]
• MessyTable: Instance Association in Multiple Camera Views
  Z. Cai, J. Zhang, D. Ren, C. Yu, H. Zhao, S. Yi, C. K. Yeo, C. C. Loy
  European Conference on Computer Vision, 2020 (ECCV)
  [PDF] [arXiv] [Supplementary Material] [Project Page]
• Robust Multi-Modality Multi-Object Tracking
  W. Zhang, H. Zhou, S. Sun, Z. Wang, J. Shi, C. C. Loy
  in Proceedings of International Conference on Computer Vision, 2019 (ICCV)
  [PDF] [arXiv] [Project Page]

Action Recognition

• Cross-Model Pseudo-Labeling for Semi-Supervised Action Recognition
  Y. Xu, F. Wei, X. Sun, C. Yang, Y. Shen, B. Dai, B. Zhou, S. Lin
  in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2022 (CVPR, Oral)
  [PDF] [arXiv] [Supplementary Material] [Project Page]
• Revisiting Skeleton-based Action Recognition
  H. Duan, Y. Zhao, K. Chen, D. Lin, B. Dai
  in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2022 (CVPR, Oral)
  [PDF] [arXiv] [Supplementary Material] [Project Page]
• TAda! Temporally-Adaptive Convolutions for Video Understanding
  Z. Huang, S. Zhang, L. Pan, Z. Qing, M. Tang, Z. Liu, M. H. Ang Jr
  International Conference on Learning Representations, 2022 (ICLR)
  [arXiv]

3D Scene Understanding and Reconstruction

Our team has been working on various tasks related to 3D reconstruction and perception, e.g, 3D shape generation and 3D human recovery

3D Reconstruction | Generation

• RenderMe-360: A Large Digital Asset Library and Benchmarks Towards High-fidelity Head Avatars
  D. Pan, L. Zhuo, J. Piao, H. Luo, W. Cheng, Y. Wang, S. Fan, S. Liu, L. Yang, B. Dai, Z. Liu, C. C. Loy, C. Qian, W. Wu, D. Lin, K.-Y. Lin
  Technical report, arXiv:2305.13353, 2023
  [arXiv] [Project Page]
• Variational Relational Point Completion Network for Robust 3D Classification
  L. Pan, X. Chen, Z. Cai, J. Zhang, H. Zhao, S. Yi, Z. Liu
  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023 (TPAMI)
  [arXiv] [Project Page]
• ConsistentNeRF: Enhancing Neural Radiance Fields with 3D Consistency for Sparse View Synthesis
  S. Hu, K. Zhou, K. Li, L. Yu, L. Hong, T. Hu, Z. Li, G. H. Lee, Z. Liu
  Technical report, arXiv:2305.11031, 2023
  [arXiv] [Project Page]
• ReMoDiffuse: Retrieval-Augmented Motion Diffusion Model
  M. Zhang, X. Guo, L. Pan, Z. Cai, F. Hong, H. Li, L. Yang, Z. Liu
  Technical report, arXiv:2304.01116, 2023
  [arXiv] [Project Page]
• SynBody: Synthetic Dataset with Layered Human Models for 3D Human Perception and Modeling
  Z. Yang, Z. Cai, H. Mei, S. Liu, Z. Chen, W. Xiao, Y. Wei, Z. Qing, C. Wei, B. Dai, W. Wu, C. Qian, D. Lin, Z. Liu, L. Yang
  Technical report, arXiv:2303.17368, 2023
  [arXiv] [Project Page]
• SparseNeRF: Distilling Depth Ranking for Few-shot Novel View Synthesis
  G. Wang, Z. Chen, C. C. Loy, Z. Liu
  Technical report, arXiv:2303.16196, 2023
  [arXiv] [Project Page]
• SHERF: Generalizable Human NeRF from a Single Image
  S. Hu, F. Hong, L. Pan, H. Mei, L. Yang, Z. Liu
  Technical report, arXiv:2303.12791, 2023
  [arXiv] [Project Page]
• Task-Oriented Human-Object Interactions Generation with Implicit Neural Representations
  Q. Li, J. Wang, C. C. Loy, B. Dai
  Technical report, arXiv:2303.13129, 2023
  [arXiv]
• F2-NeRF: Fast Neural Radiance Field Training with Free Camera Trajectories
  P. Wang, Y. Liu, Z. Chen, L. Liu, Z. Liu, T. Komura, C. Theobalt, W. Wang
  in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2023 (CVPR, Highlight)
  [PDF] [arXiv] [Supplementary Material] [Project Page]
• OmniObject3D: Large-Vocabulary 3D Object Dataset for Realistic Perception, Reconstruction and Generation
  T. Wu, J. Zhang, X. Fu, Y. Wang, J. Ren, L. Pan, W. Wu, L. Yang, J. Wang, C. Qian, D. Lin, Z. Liu
  in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2023 (CVPR, Highlight)
  [PDF] [arXiv] [Supplementary Material] [Project Page]
• Self-Supervised Geometry-Aware Encoder for Style-Based 3D GAN Inversion
  Y. Lan, X. Meng, S. Yang, C. C. Loy, B. Dai
  in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2023 (CVPR)
  [PDF] [arXiv] [Supplementary Material] [Project Page]
• SceneDreamer: Unbounded 3D Scene Generation from 2D Image Collections
  Z. Chen, G. Wang, Z. Liu
  Technical report, arXiv:2302.01330, 2023
  [arXiv] [Project Page]
• EVA3D: Compositional 3D Human Generation from 2D Image Collections
  F. Hong, Z. Chen, Y. Lan, L. Pan, Z. Liu
  International Conference on Learning Representations, 2023 (ICLR, Spotlight)
  [arXiv] [Project Page] [YouTube]
• DiffMimic: Efficient Motion Mimicking with Differentiable Physics
  J. Ren, C. Yu, S. Chen, X. Ma, L. Pan, Z. Liu
  International Conference on Learning Representations, 2023 (ICLR)
  [PDF] [arXiv] [Project Page]
• Correspondence Distillation from NeRF-based GAN
  Y. Lan, C. C. Loy, B. Dai
  Technical report, arXiv:2212.09735, 2022
  [arXiv] [Project Page]
• MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Model
  M. Zhang, Z. Cai, L. Pan, F. Hong, X. Guo, L. Yang, Z. Liu
  Technical report, arXiv:2208.15001, 2022
  [arXiv] [Project Page]
• Relighting4D: Neural Relightable Human from Videos
  Z. Chen, Z. Liu
  European Conference on Computer Vision, 2022 (ECCV)
  [PDF] [arXiv] [Supplementary Material] [Project Page]
• Monocular 3D Object Reconstruction with GAN Inversion
  J. Zhang, D. Ren, Z. Cai, C. K. Yeo, B. Dai, C. C. Loy
  European Conference on Computer Vision, 2022 (ECCV)
  [PDF] [arXiv] [Supplementary Material] [Project Page]
• HuMMan: Multi-Modal 4D Human Dataset for Versatile Sensing and Modeling
  Z. Cai, D. Ren, A. Zeng, Z. Lin, T. Yu, W. Wang, X. Fan, Y. Gao, Y. Yu, L. Pan, F. Hong, M. Zhang, C. C. Loy, L. Yang, Z. Liu
  European Conference on Computer Vision, 2022 (ECCV, Oral)
  [PDF] [arXiv] [Supplementary Material] [Project Page] [YouTube]
• AvatarCLIP: Zero-Shot Text-Driven Generation and Animation of 3D Avatars
  F. Hong, M. Zhang, L. Pan, Z. Cai, L. Yang, Z. Liu
  ACM Transactions on Graphics, 2022 (SIGGRAPH - TOG)
  [DOI] [arXiv] [Project Page]
• Chasing the Tail in Monocular 3D Human Reconstruction with Prototype Memory
  Y. Rong, Z. Liu, C. C. Loy
  IEEE Transactions on Image Processing, 2022 (TIP)
  [DOI] [arXiv] [Project Page]
• CityNeRF: Building NeRF at City Scale
  Y. Xiangli, L. Xu, X. Pan, N. Zhao, A. Rao, C. Theobalt, B. Dai, D. Lin
  Technical report, arXiv:2112.05504, 2021
  [arXiv] [Project Page]
• Robust Partial-to-Partial Point Cloud Registration in a Full Range
  L. Pan, Z. Cai, Z. Liu
  Technical report, arXiv:2111.15606, 2021
  [arXiv] [Project Page]
• Playing for 3D Human Recovery
  Z. Cai, M. Zhang, J. Ren, C. Wei, D. Ren, J. Li, Z. Lin, H. Zhao, S. Yi, L. Yang, C. C. Loy, Z. Liu
  Technical report, arXiv:2110.07588, 2021
  [arXiv] [Project Page]
• A Shading-Guided Generative Implicit Model for Shape-Accurate 3D-Aware Image Synthesis
  X. Pan, X. Xu, C. C. Loy, C. Theobalt, B. Dai
  in Proceedings of Neural Information Processing Systems, 2021 (NeurIPS)
  [PDF] [arXiv] [Project Page]
• Garment4D: Garment Reconstruction from Point Cloud Sequences
  F. Hong, L. Pan, Z. Cai, Z. Liu
  in Proceedings of Neural Information Processing Systems, 2021 (NeurIPS)
  [PDF] [Project Page]
• Density-aware Chamfer Distance as a Comprehensive Metric for Point Cloud Completion
  T. Wu, L. Pan, J. Zhang, T. Wang, Z. Liu, D. Lin
  in Proceedings of Neural Information Processing Systems, 2021 (NeurIPS)
  [PDF] [arXiv] [Project Page]
• Generative Occupancy Fields for 3D Surface-Aware Image Synthesis
  X. Xu, X. Pan, D. Lin, B. Dai
  in Proceedings of Neural Information Processing Systems, 2021 (NeurIPS)
  [PDF] [arXiv] [Project Page]
• Monocular 3D Reconstruction of Interacting Hands via Collision-Aware Factorized Refinements
  Y. Rong, J. Wang, Z. Liu, C. C. Loy
  in Proceedings of International Conference on 3D Vision, 2021 (3DV)
  [arXiv] [Project Page]
• 3D Human Texture Estimation from a Single Image with Transformers
  X. Xu, C. C. Loy
  in Proceedings of IEEE/CVF International Conference on Computer Vision, 2021 (ICCV, Oral)
  [PDF] [arXiv] [Project Page]
• Variational Relational Point Completion Network
  L. Pan, X. Chen, Z. Cai, J. Zhang, H. Zhao, S. Yi, Z. Liu
  in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2021 (CVPR, Oral)
  [PDF] [arXiv] [Supplementary Material] [Project Page]
• Unsupervised 3D Shape Completion through GAN Inversion
  J. Zhang, X. Chen, Z. Cai, L. Pan, H. Zhao, S. Yi, C. K. Yeo, B. Dai, C. C. Loy
  in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2021 (CVPR)
  [PDF] [arXiv] [Supplementary Material] [Project Page]
• Do 2D GANs Know 3D Shape? Unsupervised 3D Shape Reconstruction from 2D Image GANs
  X. Pan, B. Dai, Z. Liu, C. C. Loy, P. Luo
  International Conference on Learning Representations, 2021 (ICLR, Oral)
  [PDF] [arXiv] [Project Page]
• Delving Deep into Hybrid Annotations for 3D Human Recovery in the Wild
  Y. Rong, Z. Liu, C. Li, K. Cao, C. C. Loy
  in Proceedings of International Conference on Computer Vision, 2019 (ICCV)
  [PDF] [arXiv] [Project Page] [Supplementary Material]

3D Perception

• RoboBEV: Towards Robust Bird's Eye View Perception under Corruptions
  S. Xie, L. Kong, W. Zhang, J. Ren, L. Pan, K. Chen, Z. Liu
  Technical report, arXiv:2304.06719, 2023
  [arXiv] [Project Page]
• Robo3D: Towards Robust and Reliable 3D Perception against Corruptions
  L. Kong, Y. Liu, X. Li, R. Chen, W. Zhang, J. Ren, L. Pan, K. Chen, Z. Liu
  Technical report, arXiv:2303.17597, 2023
  [arXiv] [Project Page]
• Position-Guided Point Cloud Panoptic Segmentation Transformer
  Z. Xiao, W. Zhang, T. Wang, C. C. Loy, D. Lin, J. Pang
  Technical report, arXiv:2303.13509, 2023
  [arXiv] [Project Page]
• Improving Data Augmentation for Multi-Modality 3D Object Detection
  W. Zhang, Z. Wang, C. C. Loy
  International Conference on Learning Representations Workshop, 2023 (ICLRW)
  [arXiv] [Project Page]
• Benchmarking and Analyzing 3D Human Pose and Shape Estimation Beyond Algorithms
  H. E. Pang, Z. Cai, L. Yang, T. Zhang, Z. Liu
  in Proceedings of Neural Information Processing Systems, 2022 (NeurIPS)
  [arXiv] [Project Page]
• Benchmarking and Analyzing Point Cloud Classification under Corruptions
  J. Ren, L. Pan, Z. Liu
  in Proceedings of International Conference on Machine Learning, 2022 (ICML)
  [arXiv] [Project Page]
• Versatile Multi-Modal Pre-Training for Human-Centric Perception
  F. Hong, L. Pan, Z. Cai, Z. Liu
  in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2022 (CVPR, Oral)
  [PDF] [arXiv] [Project Page]
• LiDAR-based 4D Panoptic Segmentation via Dynamic Shifting Network
  F. Hong, H. Zhou, X. Zhu, H. Li, Z. Liu
  Technical report, arXiv:2203.07186, 2022
  [arXiv] [Project Page]
• Unsupervised Domain Adaptive 3D Detection with Multi-Level Consistency
  Z. Luo, Z. Cai, C. Zhou, G. Zhang, H. Zhao, S. Yi, S. Lu, H. Li, S. Zhang, Z. Liu
  in Proceedings of IEEE/CVF International Conference on Computer Vision, 2021 (ICCV)
  [PDF] [arXiv] [Supplementary Material]
• LiDAR-based Panoptic Segmentation via Dynamic Shifting Network
  F. Hong, H. Zhou, X. Zhu, H. Li, Z. Liu
  in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2021 (CVPR)
  [PDF] [arXiv] [Supplementary Material] [Project Page]

Deep Learning

We investigate new deep learning methods that are more efficient, robust, accurate, scalable, transferable, and explainable.

Multimodal Large Language Models

• Contextual Object Detection with Multimodal Large Language Models
  Y. Zang, W. Li, J. Han, K. Zhou, C. C. Loy
  Technical report, arXiv:2305.14207, 2023
  [arXiv] [Project Page]
• Otter: A Multi-Modal Model with In-Context Instruction Tuning
  B. Li, Y. Zhang, L. Chen, J. Wang, J. Yang, Z. Liu
  Technical report, arXiv:2305.03726, 2023
  [arXiv] [Project Page]

Unsupervised | Self-Supervised Learning

• Correlational Image Modeling for Self-Supervised Visual Pre-Training
  W. Li, J. Xie, C. C. Loy
  in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2023 (CVPR)
  [PDF] [arXiv] [Supplementary Material] [Project Page]
• Masked Frequency Modeling for Self-Supervised Visual Pre-Training
  J. Xie, W. Li, X. Zhan, Z. Liu, Y. S. Ong, C. C. Loy
  International Conference on Learning Representations, 2023 (ICLR)
  [arXiv] [Project Page]
• Delving into Inter-Image Invariance for Unsupervised Visual Representations
  J. Xie, X. Zhan, Z. Liu, Y. S. Ong, C. C. Loy
  International Journal of Computer Vision, 2022 (IJCV)
  [PDF] [DOI] [arXiv] [Project Page]
• Dense Siamese Network for Dense Unsupervised Learning
  W. Zhang, J. Pang, K. Chen, C. C. Loy
  European Conference on Computer Vision, 2022 (ECCV)
  [PDF] [arXiv] [Supplementary Material] [Project Page]
• Self-Supervised Representation Learning: Introduction, Advances and Challenges
  L. Ericsson, H. Gouk, C. C. Loy, T. M. Hospedales
  IEEE Signal Processing Magazine, 2022 (SPM)
  [DOI] [arXiv]
• Unsupervised Object-Level Representation Learning from Scene Images
  J. Xie, X. Zhan, Z. Liu, Y. S. Ong, C. C. Loy
  in Proceedings of Neural Information Processing Systems, 2021 (NeurIPS)
  [PDF] [arXiv] [Project Page]
• Unsupervised Feature Learning by Cross-Level Instance-Group Discrimination
  X. Wang, Z. Liu, S. X. Yu
  in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2021 (CVPR)
  [PDF] [arXiv] [Supplementary Material] [Project Page]
• Online Deep Clustering for Unsupervised Representation Learning
  X. Zhan, J. Xie, Z. Liu, Y. S. Ong, C. C. Loy
  in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2020 (CVPR)
  [PDF] [arXiv] [Project Page]
• Self-Supervised Learning via Conditional Motion Propagation
  X. Zhan, X. Pan, Z. Liu, D. Lin, C. C. Loy
  in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2019 (CVPR)
  [PDF] [arXiv] [Project Page]

Knowledge Distillation

• Computation-Efficient Knowledge Distillation via Uncertainty-Aware Mixup
  G. Xu, Z. Liu, C. C. Loy
  Pattern Recognition, 2023 (PR)
  [DOI] [arXiv] [Project Page]
• Mind the Gap in Distilling StyleGANs
  G. Xu, Y. Hou, Z. Liu, C. C. Loy
  European Conference on Computer Vision, 2022 (ECCV)
  [PDF] [arXiv] [Supplementary Material] [Project Page]
• Knowledge Distillation Meets Self-Supervision
  G. Xu, Z. Liu, X. Li, C. C. Loy
  European Conference on Computer Vision, 2020 (ECCV)
  [PDF] [arXiv] [Supplementary Material] [Project Page]
• Residual Knowledge Distillation
  M. Gao, Y. Shen, Q. Li, C. C. Loy
  Technical report, arXiv:2002.09168, 2020
  [arXiv]
• Inter-Region Affinity Distillation for Road Marking Segmentation
  Y. Hou, Z. Ma, C. Liu, T.-W. Hui, C. C. Loy
  in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2020 (CVPR)
  [PDF] [arXiv] [Supplementary Material] [Project Page]
• Learning Lightweight Lane Detection CNNs by Self Attention Distillation
  Y. Hou, Z. Ma, C, Liu, C. C. Loy
  in Proceedings of International Conference on Computer Vision, 2019 (ICCV)
  [PDF] [arXiv] [Supplementary Material] [Project Page]
• Learning to Steer by Mimicking Features from Heterogeneous Auxiliary Networks
  Y. Hou, Z. Ma, C. Liu, C. C. Loy
  in Proceedings of AAAI Conference on Artificial Intelligence, 2019 (AAAI, Oral)
  [PDF] [arXiv] [Project Page]
• An Embarrassingly Simple Approach for Knowledge Distillation
  M. Gao, Y. Shen, Q. Li, C. C. Loy, X. Tang
  Technical report, arXiv:1812.01819, 2018
  [arXiv]

Continual Learning

• Learning without Forgetting for Vision-Language Models
  D.-W. Zhou, Y. Zhang, J. Ning, H.-J. Ye, D.-C. Zhan, Z. Liu
  Technical report, arXiv:2305.19270, 2023
  [arXiv]
• Retrospective Class Incremental Learning
  Q. Tao, C. C. Loy, J. Cai, Z. Ge, S. See
  in Proceedings of IEEE International Conference on Multimedia and Expo, 2021 (ICME)
  [PDF]
• Learning a Unified Classifier Incrementally via Rebalancing
  S. Hou, X. Pan, C. C. Loy, Z. Wang, D. Lin
  in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2019 (CVPR)
  [PDF] [Project Page]
• Lifelong Learning via Progressive Distillation and Retrospection
  S. Hou, X. Pan, C. C. Loy, Z. Wang, D. Lin
  in Proceedings of European Conference on Computer Vision, 2018 (ECCV)
  [PDF] [Project Page]

Long-Tailed Recognition

• Open Long-Tailed Recognition in a Dynamic World
  Z. Liu, Z. Miao, X. Zhan, J. Wang, B. Gong, S. X. Yu
  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022 (TPAMI)
  [arXiv]
• Iterative Human and Automated Identification of Wildlife Images
  Z. Miao, Z. Liu, K. M. Gaynor, M. S. Palmer, S. X. Yu, W. M. Getz
  Nature Machine Intelligence, vol. 3, pp. 885–895, 2021 (Nat Mach Intell)
  [DOI] [arXiv]
• FASA: Feature Augmentation and Sampling Adaptation for Long-Tailed Instance Segmentation
  Y. Zang, C. Huang, C. C. Loy
  in Proceedings of IEEE/CVF International Conference on Computer Vision, 2021 (ICCV)
  [PDF] [arXiv] [Supplementary Material] [Project Page]
• Adversarial Robustness under Long-Tailed Distribution
  T. Wu, Z. Liu, Q. Huang, Y. Wang, D. Lin
  in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2021 (CVPR, Oral)
  [PDF] [arXiv] [Supplementary Material] [Project Page]
• Long-Tailed Recognition by Routing Diverse Distribution-Aware Experts
  X. Wang, L. Lian, Z. Miao, Z. Liu, S. X. Yu
  International Conference on Learning Representations, 2021 (ICLR)
  [PDF] [arXiv] [Project Page]

Model Uncertainty and Robustness

• Semi-Supervised Domain Generalization with Stochastic StyleMatch
  K. Zhou, C. C. Loy, Z. Liu
  International Journal of Computer Vision, 2023 (IJCV)
  [PDF] [DOI] [arXiv] [Project Page]
• Sparse Mixture-of-Experts are Domain Generalizable Learners
  B. Li, Y. Shen, J. Yang, Y. Wang, J. Ren, T. Che, J. Zhang, Z. Liu
  International Conference on Learning Representations, 2023 (ICLR, Oral)
  [arXiv] [Project Page]
• OpenOOD: Benchmarking Generalized Out-of-Distribution Detection
  J. Yang, P. Wang, D. Zou, Z. Zhou, K. Ding, W. Peng, H. Wang, G. Chen, B. Li, Y. Sun, X. Du, K. Zhou, W. Zhang, D. Hendrycks, Y. Li, Z. Liu
  in Proceedings of Neural Information Processing Systems, 2022 (NeurIPS)
  [arXiv] [Project Page]
• On-Device Domain Generalization
  K. Zhou, Y. Zhang, Y. Zang, J. Yang, C. C. Loy, Z. Liu
  Technical report, arXiv:2209.07521, 2022
  [arXiv]
• Domain Generalization: A Survey
  K. Zhou, Z. Liu, Y. Qiao, T. Xiang, C. C. Loy
  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022 (TPAMI)
  [DOI] [arXiv]
• Robust Face Anti-Spoofing with Dual Probabilistic Modeling
  Y. Zhang, Y. Wu, Z. Yin, J. Shao, Z. Liu
  Technical report, arXiv:2204.12685, 2022
  [arXiv]
• Full-Spectrum Out-of-Distribution Detection
  J. Yang · K. Zhou · Z. Liu
  Technical report, arXiv:2204.05306, 2022
  [arXiv] [Project Page]
• Balanced MSE for Imbalanced Visual Regression
  J. Ren, M. Zhang, C. Yu, Z. Liu
  in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2022 (CVPR, Oral)
  [PDF] [arXiv] [Supplementary Material] [Project Page]
• Delving Deep into the Generalization of Vision Transformers under Distribution Shifts
  C. Zhang, M. Zhang, S. Zhang, D. Jin, Q. Zhou, Z. Cai, H. Zhao, S. Yi, X. Liu, Z. Liu
  in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2022 (CVPR)
  [PDF] [arXiv] [Project Page]
• Generalized Out-of-Distribution Detection: A Survey
  J. Yang, K. Zhou, Y. Li, Z. Liu
  Technical report, arXiv:2110.11334, 2021
  [arXiv] [Project Page]
• Semantically Coherent Out-of-Distribution Detection
  J. Yang, H. Wang, L. Feng, X. Yan, H. Zheng, W. Zhang, Z. Liu
  in Proceedings of IEEE/CVF International Conference on Computer Vision, 2021 (ICCV)
  [PDF] [arXiv] [Supplementary Material] [Project Page]
• Energy-Based Open-World Uncertainty Modeling for Confidence Calibration
  Y. Wang, B. Li, T. Che, K. Zhou, D. Li, Z. Liu
  in Proceedings of IEEE/CVF International Conference on Computer Vision, 2021 (ICCV)
  [PDF] [arXiv]
• Optimization Variance: Exploring Generalization Properties of DNNs
  X. Zhang, D. Wu, H. Xiong, B. Dai
  Technical report, arXiv:2106.01714, 2021
  [arXiv]

Network Compression

• Network Pruning via Resource Reallocation
  Y. Hou, Z. Ma, C. Liu, Z. Wang, C. C. Loy
  Technical report, arXiv:2103.01847, 2021
  [arXiv]
• EcoNAS: Finding Proxies for Economical Neural Architecture Search
  D. Zhou, X. Zhou, W. Zhang, C. C. Loy, S. Yi, X. Zhang, W. Ouyang
  in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2020 (CVPR)
  [PDF] [arXiv] [Supplementary Material]

Zero/Few-shot Learning

• What Makes Good Examples for Visual In-Context Learning?
  Y. Zhang, K. Zhou, Z. Liu
  Technical report, arXiv:2301.13670, 2023
  [arXiv] [Project Page]
• Unified Vision and Language Prompt Learning
  Y. Zang, W. Li, K. Zhou, C. Huang, C. C. Loy
  Technical report, arXiv:2210.07225, 2022
  [arXiv]
• Extract Free Dense Labels from CLIP
  C. Zhou, C. C. Loy, B. Dai
  European Conference on Computer Vision, 2022 (ECCV, Oral)
  [PDF] [arXiv] [Supplementary Material] [Project Page]
• Neural Prompt Search
  Y. Zhang, K. Zhou, Z. Liu
  Technical report, arXiv:2206.04673, 2022
  [arXiv] [Project Page]
• Conditional Prompt Learning for Vision-Language Models
  K. Zhou, J. Yang, C. C. Loy, Z. Liu
  in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2022 (CVPR)
  [PDF] [arXiv] [Project Page]
• Learning to Prompt for Vision-Language Models
  K. Zhou, J. Yang, C. C. Loy, Z. Liu
  International Journal of Computer Vision, 2022 (IJCV)
  [DOI] [arXiv] [Project Page]

Media Forensics

We collect large-scale datasets and develop new methods for face forgery detection.

Forgery Detection and Anti-Deepfake

• DeepFake-Adapter: Dual-Level Adapter for DeepFake Detection
  R. Shao, T. Wu, L. Nie, Z. Liu
  Technical report, arXiv:2306.00863, 2023
  [arXiv] [Project Page]
• Detecting and Grounding Multi-Modal Media Manipulation
  R. Shao, T. Wu, Z. Liu
  in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2023 (CVPR)
  [PDF] [arXiv] [Supplementary Material] [Project Page]
• Detecting and Recovering Sequential DeepFake Manipulation
  R. Shao, T. Wu, Z. Liu
  European Conference on Computer Vision, 2022 (ECCV)
  [PDF] [arXiv] [Supplementary Material] [Project Page]
• Few-shot Forgery Detection via Guided Adversarial Interpolation
  H. Qiu, S. Chen, B. Gan, K. Wang, H. Shi, J. Shao, Z. Liu
  Technical report, arXiv:2204.05905, 2022
  [arXiv]
• DeepFakes Detection: the DeeperForensics Dataset and Challenge
  L. Jiang, W. Wu, C. Qian, C. C. Loy
  In C. Rathgeb, R. Tolosana, R. Vera-Rodriguez, C. Busch (Eds.), Handbook of Digital Face Manipulation and Detection, Springer, 2022
  [Book Link]
• ForgeryNet: A Versatile Benchmark for Comprehensive Forgery Analysis
  Y. He, B. Gan, S. Chen, Y. Zhou, G. Yin, L. Song, L. Sheng, J. Shao, Z. Liu
  in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2021 (CVPR, Oral)
  [PDF] [arXiv] [Supplementary Material] [Project Page]
• DeeperForensics-1.0: A Large-Scale Dataset for Real-World Face Forgery Detection
  L. Jiang, R. Li, W. Wu, C. Qian, C. C. Loy
  in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2020 (CVPR)
  [PDF] [arXiv] [Supplementary Material] [Project Page]