Chen Change Loy

Nanyang Associate Professor, SCSE, NTU
Director, MMLab@NTU
Co-Associate Director, S-Lab

Blk N4-02c-108, Nanyang Avenue
School of Computer Science and Engineering
Nanyang Technological University
Singapore 639798

Chen Change Loy is a Nanyang Associate Professor with the School of Computer Science and Engineering, Nanyang Technological University, Singapore. He is also an Adjunct Associate Professor at The Chinese University of Hong Kong. He is the Lab Director of MMLab@NTU and Co-associate Director of S-Lab. He received his Ph.D. (2010) in Computer Science from the Queen Mary University of London. Prior to joining NTU, he served as a Research Assistant Professor at the MMLab of The Chinese University of Hong Kong, from 2013 to 2018.

His research interests include computer vision and deep learning with a focus on image/video restoration and enhancement, generative tasks, and representation learning. He serves as an Associate Editor of the International Journal of Computer Vision (IJCV) and IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI). He also serves/served as an Area Chair of top conferences such as ICCV 2021, CVPR 2021, CVPR 2019, ECCV 2022 and ECCV 2018. He is a senior member of IEEE.

Full Biography

MMLab@NTU

MMLab@NTU was formed by Chen Change Loy on the 1 August 2018, with a research focus on computer vision and deep learning. Its sister lab is MMLab@CUHK. It is now a group with three faculty members and more than 40 members including research fellows, research assistants, and PhD students. Members in MMLab@NTU conduct research primarily in low-level vision, image and video understanding, creative content creation, 3D scene understanding and reconstruction.
Visit MMLab@NTU Homepage

Recent

Papers

Investigating Trade-offs in Real-World Video Super-Resolution
K. C. K. Chan, S. Zhou, X. Xu, C. C. Loy
in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2022 (CVPR)
[PDF] [arXiv] [Supplementary Material] [Project Page]

We examine the contributions of temporal propagation in real-world VSR and find that long-term information is also beneficial to this task but do not come for free, due to the diverse and complicated degradations in the wild. As an explorational study, we reveal several challenges in real-world VSR. We find that the domain gap on degradations and the increased computational costs result in various challenges and tradeoffs.

BasicVSR++: Effective Video Super-Resolution via Enhanced Propagation and Alignment
K. C. K. Chan, S. Zhou, X. Xu, C. C. Loy
in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2022 (CVPR)
[PDF] [arXiv] [Supplementary Material] [Project Page]

BasicVSR++ won three champions in NTIRE 2021 Video Restoration and Enhancement Challenge. BasicVSR++ consists of two effective modifications for improving propagation and alignment. The proposed second-order grid propagation and flow-guided deformable alignment allow BasicVSR++ to significantly outperform existing state of the arts with comparable runtime.

Unsupervised Image-to-Image Translation with Generative Prior
S. Yang, L. Jiang, Z. Liu, C. C. Loy
in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2022 (CVPR)
[PDF] [arXiv] [Supplementary Material] [Project Page]

We explore the use of GAN generative prior to build a versatile unsupervised image-to-image translation framework. In particular, we present a two-stage framework that is able to characterize content correspondences at a high semantic level for challenging multi-modal translations between distant domains. Such content correspondences can be discovered with only domain supervision.

Pastiche Master: Exemplar-Based High-Resolution Portrait Style Transfer
S. Yang, L. Jiang, Z. Liu, C. C. Loy
in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2022 (CVPR)
[PDF] [arXiv] [Supplementary Material] [Project Page] [YouTube]

We extend StyleGAN to accept style condition from new domains while preserving its style control in the original domain. This results in an interesting application of high-resolution exemplar-based portrait style transfer with a friendly data requirement. DualStyleGAN, with an additional style path to StyleGAN, can effectively model and modulate the intrinsic and extrinsic styles for flexible and diverse artistic portrait generation.

TransEditor: Transformer-Based Dual-Space GAN for Highly Controllable Facial Editing
Y. Xu, Y. Yin, L. Jiang, Q. Wu, C. Zheng, C. C. Loy, B. Dai, W. Wu
in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2022 (CVPR)
[arXiv] [Supplementary Material] [Project Page]

We present TransEditor, a dual-space GAN architecture with a Cross-Space Interaction mechanism based on the Transformer. The approach employs a new dual-space image editing and inversion strategy for highly controllable facial editing.

Bailando: 3D Dance Generation by Actor-Critic GPT with Choreographic Memory
S-Y. Li, W. Yu, T. Gu, C. Lin, Q. Wang, C. Qian, C. C. Loy, Z. Liu
in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2022 (CVPR)
[PDF] [arXiv] [Supplementary Material] [Project Page] [YouTube]

We address the spatial and temporal challenges of 3D dance generation by proposing a novel framework named Bailando, which is composed of a choreographic memory to address the spatial constraint by encoding and quantizing dancing-style poses, and an actor-critic GPT to realize the temporal coherency with music that translates and aligns various motion tempos and music beats.

Video K-Net: A Simple, Strong, and Unified Baseline For End-to-End Dense Video Segmentation
X. Li, W. Zhang, J. Pang, K. Chen, G. Cheng, Y. Tong, C. C. Loy
in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2022 (CVPR)
[PDF] [arXiv] [Supplementary Material] [Project Page]

Video K-Net is a simple, strong and unified framework for fully end-to-end video panoptic segmentation. It achieves state-of-the-art resuls on popular benchmarks including Cityscapes-VPS and KITTI-STEP.

Point-to-Voxel Knowledge Distillation for LiDAR Semantic Segmentation
Y. Hou, X. Zhu, Y. Ma, C. C. Loy, Y. Li
in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2022 (CVPR)
[PDF] [Supplementary Material] [Project Page]

We propose a novel point-to-voxel knowledge distillation approach (PVD) tailored for LiDAR semantic segmentation. PVD is comprised of the point-to-voxel output distillation and affinity distillation. The supervoxel partition and difficulty-aware sampling strategy are further proposed to improve the learning efficiency of affinity distillation.

Conditional Prompt Learning for Vision-Language Models
K. Zhou, J. Yang, C. C. Loy, Z. Liu
in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2022 (CVPR)
[PDF] [arXiv] [Project Page]

Conditional Context Optimization (CoCoOp) extends CoOp by further learning a lightweight neural network to generate for each image an input-conditional token (vector). Compared to CoOp’s static prompts, the dynamic prompts adapt to each instance and are thus less sensitive to class shift.

We are looking for Research Fellows who want to do exciting research in the area of machine leanring and computer vision.