Welcome to Xiaoyu Zhu's homepage

			Xiaoyu Zhu
			
			Ph.D. Student in Artificial Intelligence
		
			Language Technologies Institute
		
			School of Computer Science
		
			Carnegie Mellon University

Publications Google Scholar Github Linkedin Resume

Introduction

Greetings! I'm a Ph.D. student at Language Technologies Institute under the supervision of Prof. Alexander Hauptmann.
My research focuses on robust feature representation learning for visual perception and generation. I have been developing novel representation learning methods including: (1) Contrastive/Siamese Learning for video and 3D scene understanding; (2) Masked Visual Modeling for human action analysis from videos and 3D point clouds/meshes; and (3) Generatively Pretrained Vision-Language Models for open-vocabulary 3D scene understanding and generation. I have published first-authored papers in top computer vision conferences such as CVPR, ICCV, and ECCV. My research has been featured by CMU News and Science Daily, and deployed by the Federal Emergency Management Agency.
I'm actively seeking full-time industrial positions starting in Fall 2024 or Spring 2025. Please reach out via email if you are interested!

News

[07/2024] One paper has been accepted by ECCV 2024.
[02/2024] One paper has been accepted by SPIE DCS 2024 (Oral).
[02/2023] One paper has been accepted by CVPR 2023.
[01/2023] One paper has been accepted by SPIE DCS 2023.
[06/2022] Featured in a news report by Washington Post using shooter localization technologies.
[07/2021] CIVANet paper has been accepted by ICCV 2021.
[03/2021] We developed an AI solution to improve cleaning schedules at Pittsburgh International Airport. Press Coverage: , ,
[09/2020] We won the Automated Streams Analysis for Public Safety Challenge with a $30k prize.
[08/2020] Our automatic damage assessment system has been successfully tested on Hurricane Laura. Press Coverage: , , , ,
[07/2020] MSNet paper has been accepted by WACV 2021 (with strong-accept).

Publications

Open Vocabulary 3D Semantic Segmentation with Text-to-Image Diffusion Models
Zhu, X., Zhou, H., Xing, P., Zhao L., Xu H., Liang J., Hauptmann, A., Liu, T., Gallagher, A.

ECCV 2024

[Paper] [Project Page]
STMT: A Spatial-Temporal Mesh Transformer for MoCap-Based Action Recognition
Zhu, X., Huang, P.*, Liang, J.*, De Melo, C., Hauptmann, A.

CVPR 2023

[Paper] [Dataset/Code]
Leveraging Body Pose Estimation for Gesture Recognition Using Synthetic Data
Zhu, X., De Melo, C., Hauptmann, A.

SPIE DCS 2023

[Paper] [Dataset/Code]
Weakly Supervised 3D Semantic Segmentation Using Cross-Image Consensus and Inter-Voxel Affinity Relations
Zhu, X., Chen, J., Zeng, X., Liang, J., Li, C., Liu, S., Xu, M.

ICCV 2021

[Paper]
MSNet: A Multilevel Instance Segmentation Network for Natural Disaster Damage Assessment in Aerial Videos
Zhu, X., Liang, J., Hauptmann, A.

WACV 2021

[Paper] [Dataset/Code]

Selected Media

Carnegie Mellon University News. Students Use AI To Improve Cleaning Schedules at Pittsburgh Airport, March 11, 2021.
Carnegie Mellon University News. Amateur Drone Videos Could Aid in Natural Disaster Damage Assessment, August 28, 2020.
Science Daily. Amateur Drone Videos Could Aid in Natural Disaster Damage Assessment, August 28, 2020.
CBS. CMU: Amateur Drone Videos Posted To Social Media Could Be Used To Assess Storm Damage. August 29, 2020.
Microsoft News. CMU: Amateur Drone Videos Posted To Social Media Could Be Used To Assess Storm Damage. August 31, 2020.
Yahoo News. CMU Developing Program To Assess Hurricane Damage. August 29, 2020.
AZO Robotics. New AI System Helps Detect Damage Caused to Buildings by Hurricanes. August 31, 2020.