Xiaoyu Zhu

Xiaoyu Zhu

Machine Learning Researcher, Apple

Ph.D. in Artificial Intelligence, Carnegie Mellon University

Greetings! I’m a Machine Learning Researcher at Apple, where I work on multimodal agentic post-training, with a focus on image/video reasoning and long-horizon visual agents. I obtained my Ph.D. in Artificial Intelligence from Carnegie Mellon University under the supervision of Prof. Alexander Hauptmann.

News

  • [08/2026] Check out our latest technical report: Beyond Visual CoT: Internalized Visual Thinking for Proactive Video Reasoning.
  • [03/2026] One paper has been accepted by ICML 2026, and selected for an oral presentation at the CVPR 2026 Workshop.
  • [12/2024] Completed my Ph.D. in Artificial Intelligence at Carnegie Mellon University, and started at Apple as a Machine Learning Researcher.
  • [07/2024] One paper has been accepted by ECCV 2024.
  • [02/2023] One paper has been accepted by CVPR 2023, and selected for an oral presentation at the SPIE 2024.

Selected Publications

Figure 1: three post-training paradigms for proactive video reasoning

Beyond Visual CoT: Internalized Visual Thinking for Proactive Video Reasoning

Zhu, X., Deng, X., Taddewadikar, S., Mondal, A. K., Jiang, Z., Liebelt, J.

[Paper] [Blog Post]
Figure 1: stress-testing VLMs with targeted textual perturbations

On Robustness and Chain-of-Thought Consistency of RL-Finetuned VLMs

Zhao, R., Shah, A., Zhu, X., Deng, X., Jiang, Z., Yang, Y., Liebelt, J., Mondal, A. K.

ICML 2026

Oral — CVPR 2026 Workshop
[Paper]
Diff2Scene

Open Vocabulary 3D Semantic Segmentation with Text-to-Image Diffusion Models

Zhu, X., Zhou, H., Xing, P., Zhao, L., Xu, H., Liang, J., Hauptmann, A., Liu, T., Gallagher, A.

ECCV 2024

[Paper] [Project Page]
STMT

STMT: A Spatial-Temporal Mesh Transformer for MoCap-Based Action Recognition

Zhu, X., Huang, P.*, Liang, J.*, De Melo, C., Hauptmann, A.

CVPR 2023

Oral — SPIE 2024
[Paper] [Dataset/Code]
CIVANet

Weakly Supervised 3D Semantic Segmentation Using Cross-Image Consensus and Inter-Voxel Affinity Relations

Zhu, X., Chen, J., Zeng, X., Liang, J., Li, C., Behpour, S., Xu, M.

ICCV 2021

[Paper]
MSNet

MSNet: A Multilevel Instance Segmentation Network for Natural Disaster Damage Assessment in Aerial Videos

Zhu, X., Liang, J., Hauptmann, A.

WACV 2021

[Paper] [Dataset/Code]

Won the Automated Streams Analysis for Public Safety Challenge with a $30k prize; the damage assessment system was successfully tested on Hurricane Laura. Press coverage: Carnegie Mellon University News CBS Pittsburgh Microsoft News Science Daily AZO Robotics

See all publications →

Education

Ph.D. in Artificial Intelligence (Language and Information Technologies)
Carnegie Mellon University, School of Computer Science · Pittsburgh, PA

Advisor: Prof. Alexander Hauptmann · Informedia Lab

Aug. 2021 – Dec. 2024
M.S. in Artificial Intelligence and Innovation
Carnegie Mellon University, School of Computer Science · Pittsburgh, PA
May 2019 – May 2021

Experience

Apple Inc.
Machine Learning Researcher
Dec. 2024 – Present
Carnegie Mellon University, School of Computer Science
Research Assistant
Sep. 2019 – Dec. 2024
Meta GenAI
AI Research Scientist Intern
May 2024 – Nov. 2024
Google DeepMind & Google Research
Student Researcher
May 2023 – Nov. 2023