ML

Publications

Highlighted

Multi-modal Agent Tuning: Building a VLM-Driven Agent for Efficient Tool Usage
Multi-modal Agent Tuning: Building a VLM-Driven Agent for Efficient Tool Usage
Zhi Gao, Bofei Zhang, Pengxiang Li, Xiaojian Ma, Tao Yuan, Yue Fan, Yuwei Wu, Yunde Jia, Song-Chun Zhu, Qing Li
International Conference on Learning Representations (ICLR)  ·  2025
Website
spotlight
Move to Understand a 3D Scene: Bridging Visual Grounding and Exploration for Efficient and Versatile Embodied Navigation
Move to Understand a 3D Scene: Bridging Visual Grounding and Exploration for Efficient and Versatile Embodied Navigation
Ziyu Zhu, Xilin Wang, Yixuan Li, Zhuofan Zhang, Xiaojian Ma, …, Wei Liang, Qian Yu, Zhidong Deng, Siyuan Huang, Qing Li
International Conference on Computer Vision (ICCV)  ·  2025
Website
spotlight
Embodied VideoAgent: Persistent Memory from Egocentric Videos and Embodied Sensors Enables Dynamic Scene Understanding
Embodied VideoAgent: Persistent Memory from Egocentric Videos and Embodied Sensors Enables Dynamic Scene Understanding
Yue Fan, Xiaojian Ma, Rongpeng Su, Jun Guo, Rujie Wu, Xi Chen, Qing Li
International Conference on Computer Vision (ICCV)  ·  2025
Website
spotlight
End-to-End Neuro-Symbolic Reinforcement Learning with Textual Explanations
End-to-End Neuro-Symbolic Reinforcement Learning with Textual Explanations
Lirui Luo, Guoxi Zhang, Hongming Xu, Yaodong Yang, Cong Fang, Qing Li
International Conference on Machine Learning (ICML)  ·  2024
Website
spotlight

All

2025

Move to Understand a 3D Scene: Bridging Visual Grounding and Exploration for Efficient and Versatile Embodied Navigation
Move to Understand a 3D Scene: Bridging Visual Grounding and Exploration for Efficient and Versatile Embodied Navigation
Ziyu Zhu, Xilin Wang, Yixuan Li, Zhuofan Zhang, Xiaojian Ma, …, Wei Liang, Qian Yu, Zhidong Deng, Siyuan Huang, Qing Li
International Conference on Computer Vision (ICCV)  ·  2025
Website
spotlight
From Objects to Anywhere: A Holistic Benchmark for Multi-level Visual Grounding in 3D Scenes
From Objects to Anywhere: A Holistic Benchmark for Multi-level Visual Grounding in 3D Scenes
Tianxu Wang, Zhuofan Zhang, Ziyu Zhu, Yue Fan, Jing Xiong, Pengxiang Li, Xiaojian Ma, Qing Li
Neural Information Processing Systems (NeurIPS D&B)  ·  2025
Website
Iterative Tool Usage Exploration for Multimodal Agents via Step-wise Preference Tuning
Iterative Tool Usage Exploration for Multimodal Agents via Step-wise Preference Tuning
Pengxiang Li, Zhi Gao, Bofei Zhang, Yapeng Mi, Xiaojian Ma, …, Tao Yuan, Yuwei Wu, Yunde Jia, Song-Chun Zhu, Qing Li
Neural Information Processing Systems (NeurIPS)  ·  2025
Website
Embodied VideoAgent: Persistent Memory from Egocentric Videos and Embodied Sensors Enables Dynamic Scene Understanding
Embodied VideoAgent: Persistent Memory from Egocentric Videos and Embodied Sensors Enables Dynamic Scene Understanding
Yue Fan, Xiaojian Ma, Rongpeng Su, Jun Guo, Rujie Wu, Xi Chen, Qing Li
International Conference on Computer Vision (ICCV)  ·  2025
Website
spotlight
MMKE-Bench: A Multimodal Editing Benchmark for Diverse Visual Knowledge
MMKE-Bench: A Multimodal Editing Benchmark for Diverse Visual Knowledge
Yuntao Du, Kailin Jiang, Zhi Gao, Chenrui Shi, Zilong Zheng, Siyuan Qi, Qing Li
International Conference on Learning Representations (ICLR)  ·  2025
Website
Multi-modal Agent Tuning: Building a VLM-Driven Agent for Efficient Tool Usage
Multi-modal Agent Tuning: Building a VLM-Driven Agent for Efficient Tool Usage
Zhi Gao, Bofei Zhang, Pengxiang Li, Xiaojian Ma, Tao Yuan, Yue Fan, Yuwei Wu, Yunde Jia, Song-Chun Zhu, Qing Li
International Conference on Learning Representations (ICLR)  ·  2025
Website
spotlight

2024

FIRE: A Dataset for Feedback Integration and Refinement Evaluation of Multimodal Models
FIRE: A Dataset for Feedback Integration and Refinement Evaluation of Multimodal Models
Pengxiang Li, Zhi Gao, Bofei Zhang, Tao Yuan, Yuwei Wu, Mehrtash Harandi, Yunde Jia, Song-Chun Zhu, Qing Li
Neural Information Processing Systems (NeurIPS D&B)  ·  2024
Website
UltraEdit: Instruction-based Fine-Grained Image Editing at Scale
UltraEdit: Instruction-based Fine-Grained Image Editing at Scale
Haozhe Zhao, Xiaojian Ma, Liang Chen, Shuzheng Si, Rujie Wu, Kaikai An, Peiyu Yu, Minjia Zhang, Qing Li, Baobao Chang
Neural Information Processing Systems (NeurIPS D&B)  ·  2024
Website
Bongard-OpenWorld: Few-Shot Reasoning for Free-form Visual Concepts in the Real World
Bongard-OpenWorld: Few-Shot Reasoning for Free-form Visual Concepts in the Real World
Rujie Wu, Xiaojian Ma, Zhenliang Zhang, Wei Wang, Qing Li, Song-Chun Zhu, Yizhou Wang
International Conference on Learning Representations (ICLR)  ·  2024
Website
CLOVA: A Closed-Loop Visual Assistant with Tool Usage and Update
CLOVA: A Closed-Loop Visual Assistant with Tool Usage and Update
Zhi Gao, Yuntao Du, Xintong Zhang, Xiaojian Ma, Wenjuan Han, Song-Chun Zhu, Qing Li
The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)  ·  2024
Website
VideoAgent: A Memory-augmented Multimodal Agent for Video Understanding
VideoAgent: A Memory-augmented Multimodal Agent for Video Understanding
Yue Fan, Xiaojian Ma, Rujie Wu, Yuntao Du, Jiaqi Li, Zhi Gao, Qing Li
European Conference on Computer Vision (ECCV)  ·  2024
Website
MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learning
MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learning
Haozhe Zhao, Zefan Cai, Shuzheng Si, Xiaojian Ma, Kaikai An, Liang Chen, Zixuan Liu, Sheng Wang, Wenjuan Han, Baobao Chang
International Conference on Learning Representations (ICLR)  ·  2024
Website
An Embodied Generalist Agent in 3D World
An Embodied Generalist Agent in 3D World
Jiangyong Huang, Silong Yong, Xiaojian Ma, Xiongkun Linghu, Puhao Li, Yan Wang, Qing Li, Song-Chun Zhu, Baoxiong Jia, Siyuan Huang
International Conference on Machine Learning (ICML)  ·  2024
Website
End-to-End Neuro-Symbolic Reinforcement Learning with Textual Explanations
End-to-End Neuro-Symbolic Reinforcement Learning with Textual Explanations
Lirui Luo, Guoxi Zhang, Hongming Xu, Yaodong Yang, Cong Fang, Qing Li
International Conference on Machine Learning (ICML)  ·  2024
Website
spotlight
SceneVerse: Scaling 3D Vision-Language Learning for Grounded Scene Understanding
SceneVerse: Scaling 3D Vision-Language Learning for Grounded Scene Understanding
Baoxiong Jia, Yixin Chen, Huangyue Yu, Yan Wang, Xuesong Niu, Tengyu Liu, Qing Li, Siyuan Huang
European Conference on Computer Vision (ECCV)  ·  2024
Website
Neural-Symbolic Recursive Machine for Systematic Generalization
Neural-Symbolic Recursive Machine for Systematic Generalization
Qing Li, Yixin Zhu, Yitao Liang, Ying Nian Wu, Song-Chun Zhu, Siyuan Huang
International Conference on Learning Representations (ICLR)  ·  2024
Website
JARVIS-1: Open-World Multi-task Agents with Memory-Augmented Multimodal Language Models
JARVIS-1: Open-World Multi-task Agents with Memory-Augmented Multimodal Language Models
Zihao Wang, Shaofei Cai, Anji Liu, Yonggang Jin, Jinbing Hou, …, Zhaofeng He, Zilong Zheng, Yaodong Yang, Xiaojian Ma, Yitao Liang
Transactions on Pattern Analysis and Machine Intelligence (TPAMI)  ·  2024
Website

2023

3D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignment
3D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignment
Ziyu Zhu, Xiaojian Ma, Yixin Chen, Zhidong Deng, Siyuan Huang, Qing Li
International Conference on Computer Vision (ICCV)  ·  2023
Website
A Minimalist Dataset for Systematic Generalization of Perception, Syntax, and Semantics
A Minimalist Dataset for Systematic Generalization of Perception, Syntax, and Semantics
Qing Li, Siyuan Huang, Yining Hong, Yixin Zhu, Ying Nian Wu, Song-Chun Zhu
International Conference on Learning Representations (ICLR)  ·  2023
Website
Exploring Data Geometry for Continual Learning
Exploring Data Geometry for Continual Learning
Zhi Gao, Chen Xu, Feng Li, Yunde Jia, Mehrtash Harandi, Yuwei Wu
The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)  ·  2023
Website
SQA3D: Situated Question Answering in 3D Scenes
SQA3D: Situated Question Answering in 3D Scenes
Xiaojian Ma, Silong Yong, Zilong Zheng, Qing Li, Yitao Liang, Song-Chun Zhu, Siyuan Huang
International Conference on Learning Representations (ICLR)  ·  2023
Website
Learning non-Markovian Decision-Making from State-only Sequences
Learning non-Markovian Decision-Making from State-only Sequences
Aoyang Qin, Feng Gao, Qing Li, Song-Chun Zhu, Sirui Xie
Advances in Neural Information Processing Systems (NeurIPS)  ·  2023
Website
Meta-causal Learning for Single Domain Generalization
Meta-causal Learning for Single Domain Generalization
Jin Chen, Zhi Gao, Xinxiao Wu, Jiebo Luo
The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)  ·  2023
Website
Learning to Optimize on Riemannian Manifolds
Learning to Optimize on Riemannian Manifolds
Zhi Gao, Yuwei Wu, Xiaomeng Fan, Mehrtash Harandi, Yunde Jia
Transactions on Pattern Analysis and Machine Intelligence (TPAMI)  ·  2023
Website
Curvature-Adaptive Meta-Learning for Fast Adaptation to Manifold Data
Curvature-Adaptive Meta-Learning for Fast Adaptation to Manifold Data
Zhi Gao, Yuwei Wu, Mehrtash Harandi, Yunde Jia
Transactions on Pattern Analysis and Machine Intelligence (TPAMI)  ·  2023
Website
Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agents
Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agents
Zihao Wang, Shaofei Cai, Guanzhou Chen, Anji Liu, Xiaojian Ma, Yitao Liang
Advances in Neural Information Processing Systems (NeurIPS)  ·  2023
Website