Magma: A Foundation Model for Multimodal AI Agents
Jianwei Yang, Reuben Tan, Qianhui Wu, Ruijie Zheng, Baolin Peng, Yongyuan Liang, Yu Gu, Mu Cai, Seonghyeon Ye, Joel Jang, Yuquan Deng, Lars Liden, Jianfeng Gao
February 2025
Embodied AI
Developing intelligent systems that perform complex tasks through understanding and engaging with physical and virtual environments
Publications
Publications by Year
-
VLM-driven Behavior Tree for Context-aware Task Planning
Naoki Wake, Atsushi Kanehira, Jun Takamatsu, Kazuhiro Sasabuchi, Katsushi Ikeuchi
January 2025
-
Modality-Driven Design for Multi-Step Dexterous Manipulation: Insights from Neuroscience
Naoki Wake, Atsushi Kanehira, Daichi Saito, Jun Takamatsu, Kazuhiro Sasabuchi, Hideki Koike, Katsushi Ikeuchi
December 2024
UniGraspTransformer: Simplified Policy Distillation for Scalable Dexterous Robotic Grasping
Wenbo Wang, Fangyun Wei, Lei Zhou, Xi Chen, Lin Luo, Xiaohan Yi, Yizhong Zhang, Yaobo Liang, Chang Xu, Yan Lu, Jiaolong Yang, Baining Guo
CVPR 2025 | December 2024
CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulation
Qixiu Li, Yaobo Liang, Zeyu Wang, Lin Luo, Xi Chen, Mozheng Liao, Fangyun Wei, Yu Deng, Sicheng Xu, Yizhong Zhang, Xiaofan Wang, Bei Liu, Jianlong Fu, Jianmin Bao, Dong Chen, Yuanchun Shi, Jiaolong Yang, Baining Guo
November 2024
Scaling Laws for Pre-training Agents and World Models
Tim Pearce, Tabish Rashid, Dave Bignell, Raluca Stevenson, Sam Devlin, Katja Hofmann
November 2024
Self-maintaining [networked] systems: The rise of datacenter robotics!
Freddie Hong, Iason Sarantopoulos, Elliott Hogg, David Richardson, Yizhong Zhang, Hugh Williams, David Sweeney, Andromachi Chatzieleftheriou, Ant Rowstron
2024 Hot Topics in Networks | November 2024
IGOR: Image-GOal Representations are the Atomic Control Units for Foundation Models in Embodied AI
Xiaoyu Chen, Junliang Guo, Tianyu He, Chuheng Zhang, Pushi Zhang, Derek Yang, Li Zhao, Jiang Bian
October 2024
APriCoT: Action Primitives based on Contact-state Transition for In-Hand Tool Manipulation
Daichi Saito, Atsushi Kanehira, Kazuhiro Sasabuchi, Naoki Wake, Jun Takamatsu, Hideki Koike, Katsushi Ikeuchi
2024 IEEE-RAS 23rd International Conference on Humanoid Robots (Humanoids) | July 2024
PRISE: LLM-Style Sequence Compression for Learning Temporal Action Abstractions in Control
Ruijie Zheng, Ching-An Cheng, Hal Daumé III, Furong Huang, Andrey Kolobov
ICML 2024 | June 2024
ORAL
Rapidly Adapting Policies to the Real World via Simulation-Guided Fine-Tuning
Patrick Yin, Tyler Westenbroek, Simran Bagaria, Kevin Huang, Ching-An Cheng, Andrey Kolobov, Abhishek Gupta
Under submission | June 2024
RASCAL: A Scalable, High-redundancy Robot for Automated Storage and Retrieval Systems
Richard Black, Marco Caballero, Andromachi Chatzieleftheriou, Tim Deegan, Philip Heard, Freddie Hong, Russell Joyce, Sergey Legtchenko, Ant Rowstron, Adam Smith, David Sweeney, Hugh Williams
2024 IEEE International Conference on Robotics and Automation | May 2024
Interaction-Shaping Robotics: Robots that Influence Interactions between Other Agents
Sarah Gillet, Marynel Vázquez, Sean Andrist, Iolanda Leite, S. Sebo
ACM Transactions on Human-Robot Interaction | February 2024
-
Semantic constraints to represent common sense required in household actions for multimodal learning-from-observation robot
Katsushi Ikeuchi, Naoki Wake, Kazuhiro Sasabuchi, Jun Takamatsu
Int. J. Robotics Res. | November 2023, Vol 43(2): pp. 134-170
GPT-4V(ision) for Robotics: Multimodal Task Planning From Human Demonstration
Naoki Wake, Atsushi Kanehira, Kazuhiro Sasabuchi, Jun Takamatsu, Katsushi Ikeuchi
IEEE Robotics and Automation Letters | November 2023, Vol 9(11): pp. 10567-10574
Comparing the Efficacy of Fine-Tuning and Meta-Learning for Few-Shot Policy Imitation
Massimiliano Patacchiola, Mingfei Sun, Katja Hofmann, Richard E Turner
Conference on Lifelong Learning Agents | November 2023
Interactive Robot Learning from Verbal Correction
Huihan Liu, Alice Chen, Yuke Zhu, Adith Swaminathan, Andrey Kolobov, Ching-An Cheng
CoRL 2023 Workshop on Language and Robot Learning (LangRob) | November 2023
CASSPR: Cross Attention Single Scan Place Recognition
Yan Xia, Mariia Gladkova, Rui Wang, Qianyun Li, Uwe Stilla, Joao F. Henriques, Daniel Cremers
ICCV | October 2023
ChatGPT Empowered Long-Step Robot Control in Various Environments: A Case Application
Naoki Wake, Atsushi Kanehira, Kazuhiro Sasabuchi, Jun Takamatsu, Katsushi Ikeuchi
IEEE Access | August 2023, Vol 1: pp. 95060-95078
Open Access https://doi.org/10.1109/ACCESS.2023.3310935
R3D3: Dense 3D Reconstruction of Dynamic Scenes from Multiple Cameras
Aron Schmied, Tobias Fischer, Martin Danelljan, Marc Pollefeys, Fisher Yu
ICCV 2023 | August 2023
Harnessing AI and robotics in humanitarian assistance and disaster response
Thomas Manzini, Robin R. Murphy, Eric Heim, Caleb Robinson, Guido Zarrella, Ritwik Gupta
Science Robotics | July 2023
Provable Safe Reinforcement Learning with Binary Feedback
Andrew Bennett, Dipendra Misra, Nathan Kallus
2023 International Conference on Artificial Intelligence and Statistics | April 2023
ChatGPT for Robotics: Design Principles and Model Abilities
Sai Vemprala, Rogerio Bonatti, Arthur Bucker, Ashish Kapoor
MSR-TR-2023-8 | February 2023
Published by Microsoft
Guaranteed Discovery of Control-Endogenous Latent States with Multi-Step Inverse Models
Alex Lamb, Riashat Islam, Yonathan Efroni, Aniket Didolkar, Dipendra Misra, Dylan Foster, Olalekan Ogunmolu, Rajan Chari, Akshay Krishnamurthy, John Langford
Transactions on Machine Learning Research | February 2023
-
M³ViT: Mixture-of-Experts Vision Transformer for Efficient Multi-task Learning with Model-Accelerator Co-design
Hanxue Liang, Zhiwen Fan, Rishov Sarkar, Ziyu Jiang, Tianlong Chen, Yu Cheng, Cong Hao, Zhangyang Wang
36th Annual Conference on Neural Information Processing System (NeurIPS 2022) | December 2022
PACT: Perception-Action Causal Transformer for Autoregressive Robotics Pre-Training
Rogerio Bonatti, Sai Vemprala, Shuang Ma, Felipe Vieira Frujeri, Shuhang Chen, Ashish Kapoor
IROS 2023 | September 2022
MoBT 17.1 / MoBIP 17.1
Provably sample-efficient RL with side information about latent dynamics
Yao Liu, Dipendra Misra, Miro Dudík, Robert E. Schapire
2022 Neural Information Processing Systems | May 2022
Panoptic Multi-TSDFs: a Flexible Representation for Online Multi-resolution Volumetric Mapping and Long-term Dynamic Scene Consistency
Lukas Schmid, Jeffrey Delmerico, Johannes L. Schönberger, Juan Nieto, Marc Pollefeys, Roland Siegwart, Cesar Cadena
2022 International Conference on Robotics and Automation (ICRA) | May 2022
Latent Variable Sequential Set Transformers For Joint Multi-Agent Motion Prediction
Roger Girgis, Florian Golemo, Felipe Codevilla, Martin Weiss, Jim Aldon D'Souza, Samira Ebrahimi Kahou, Felix Heide, Christopher Pal
ICLR 2022 | April 2022
Design strategies for controlling neuron-connected robots using reinforcement learning
Haruto Sawada, Naoki Wake, Kazuhiro Sasabuchi, Jun Takamatsu, Hirokazu Takahashi, Katsushi Ikeuchi
March 2022
SMOP: Scaling Up Real-time Collaborative Visual SLAM at the Edge
Jingao Xu, Hao Cao, Zheng Yang, Longfei Shangguan, Jialin Zhang, Xiaowu He, Yunhao Liu
NSDI 2022 | March 2022
Spatial Computing and Intuitive Interaction: Bringing Mixed Reality and Robotics Together
Jeffrey Delmerico, Roi Poranne, Federica Bogo, Helen Oleynikova, Eric Vollenweider, Stelian Coros, Juan Nieto, Marc Pollefeys
February 2022
Integration of Gesture Generation System Using Gesture Library with DIY Robot Design Kit
Hitoshi Teshima, Naoki Wake, Diego Thomas, Yuta Nakashima, David Baumert, Hiroshi Kawasaki, Katsushi Ikeuchi
2022 IEEE/SICE International Symposium on System Integration | January 2022
A Multimodal Learning-from-Observation Towards All-at-once Robot Teaching using Task Cohesion
Iori Yanokura, Naoki Wake, Kazuhiro Sasabuchi, Riku Arakawa, Kei Okada, Jun Takamatsu, Masayuki Inaba, Katsushi Ikeuchi
2022 IEEE/SICE International Symposium on System Integration | January 2022
-
ORBIT: A Real-World Few-Shot Dataset for Teachable Object Recognition
Daniela Massiceti, Luisa Zintgraf, John Bronskill, Lida Theodorou, Matthew Tobias Harris, Ed Cutrell, Cecily Morrison, Katja Hofmann, Simone Stumpf
ICCV 2021 | October 2021
HapticBots: Distributed Encountered-type Haptics for VR with Multiple Shape-changing Mobile Robots
Ryo Suzuki, Eyal Ofek, Mike Sinclair, Daniel Leithinger, Mar Gonzalez Franco
UIST 2021 | October 2021
Verbal Focus-of-Attention System for Learning-from-Observation
Naoki Wake, Iori Yanokura, Kazuhiro Sasabuchi, Katsushi Ikeuchi
2021 International Conference on Robotics and Automation | May 2021
Training Structured Mechanical Models by Minimizing Discrete Euler-Lagrange Residual
Kunal Menda, Jayesh K. Gupta, Zachary Manchester, Mykel J. Kochenderfer
MSR-TR-2021-35 | May 2021
Published by Microsoft
RMP2: A Structured Composable Policy Class for Robot Learning.
Anqi Li, Ching-An Cheng, Muhammad Asif Rana, Man Xie, Karl Van Wyk, Nathan D. Ratliff, Byron Boots
RSS | March 2021
Contact Web Status Presentation for Freehand Grasping in MR-based Robot-teaching
Daichi Saito, Naoki Wake, Kazuhiro Sasabuchi, Hideki Koike, Katsushi Ikeuchi
2021 Human-Robot Interaction | March 2021
A Computational Stack for Cross-Domain Acceleration
Sean Kinzer, Joon Kyung Kim, Soroush Ghodrati, Brahmendra Yatham, Alric Althoff, Divya Mahajan, Sorin Lerner, Hadi Esmailzadeh
International Symposium on High Performance Computer Architecture | February 2021
A Learning-from-Observation Framework: One-Shot Robot Teaching for Grasp-Manipulation-Release Household Operations
Naoki Wake, Riku Arakawa, Iori Yanokura, Takuya Kiyokawa, Kazuhiro Sasabuchi, Jun Takamatsu, Katsushi Ikeuchi
2021 IEEE/SICE International Symposium on System Integration | January 2021
SICE Young Investigator Award
-
Adversarial Attacks on Optimization based Planners
Sai Vemprala, Ashish Kapoor
October 2020
Freetures: Localization in Signed Distance Function Maps
Alexander Millane, Helen Oleynikova, Christian Lanegger, Jeffrey Delmerico, Juan Nieto, Roland Siegwart, Marc Pollefeys, Cesar Cadena
October 2020
Remote Observation of Field Work on the Farm
Wendy Ju, Ilan Mandel, Kevin Weatherwax, Leila Takayama, Nikolas Martelaro, Denis Willett
August 2020
The future of work and wellbeing: A preliminary report from a series of conversations
Andrew L. Kun, Orit Shaer, Raffaella Sadun, Linda Ng Boyle, John D. Lee
August 2020
gDLS*: Generalized Pose-and-Scale Estimation Given Scale and Gravity Priors
Victor Fragoso, Joseph DeGol, Gang Hua
CVPR 2020 | June 2020
Variational Recurrent Models for Solving Partially Observable Control Tasks
Dongqi Han, Kenji Doya, Jun Tani
2020 International Conference on Learning Representations | May 2020
VR-Enabled Telepresence as a Bridge for People, Environments, and Experiences
Brennan Jones, Yaying Zhang, Priscilla N. Y. Wong, Sean Rintel, Yasamin Heshmat
CHI 2020 | April 2020
Workshop on Social VR Workshop
VROOM: Virtual Robot Overlay for Online Meetings
Brennan Jones, Yaying Zhang, Priscilla Wong, Sean Rintel
CHI 2020 | April 2020
CHI 2020 Extended Abstracts
PDF DOI Video Github Project Project Project Project Project
Automatic Curriculum Learning for Deep RL: A Short Survey
Rémy Portelas, Cédric Colas, Lilian Weng, Katja Hofmann, Pierre-Yves Oudeyer
International Joint Conference on Artificial Intelligence | March 2020
-
Enhancing listening capability of humanoid robot by reduction of stationary ego-noise
Naoki Wake, Masaaki Fukumoto, Hirokazu Takahashi, Katsushi Ikeuchi
IEEJ Transactions on Electrical and Electronic Engineering | December 2019, Vol 14(12): pp. 1815-1822
Visual prosody supports reading aloud expressively
Ann Bessemans, Maarten Renckens, Kevin Bormans, Erik Nuyts, Kevin Larson
Visible Language | December 2019, Vol 53(3): pp. 28-49
Night-to-day image translation for retrieval-based localization
Asha Anoosheh, Torsten Sattler, Radu Timofte, Marc Pollefeys, Luc Van Gool
2019 International Conference on Robotics and Automation (ICRA) | May 2019
-
Describing upper-body motions based on Labanotation for learning-from-observation robot
Katsushi Ikeuchi, Zhaoyuan Ma, Zengqiang Yan, Shunsuke Kudoh, Minako Nakamura
International Journal of Computer Vision | December 2018, Vol 126(12): pp. 1415-1429
A Convergence Theory for Deep Learning via Over-Parameterization
Zeyuan Allen-Zhu, Yuanzhi Li, Zhao Song
September 2018
Automating Robotic Furniture with A Collaborative Vision-based Sensing Scheme
Zhizheng Zhang, Zhibo Chen, Weiping Li
IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN 2018) | August 2018
Robox: an end-to-end solution to accelerate autonomous control in robotics
Jacob Sacks, Divya Mahajan, Richard C. Lawson, Hadi Esmaeilzadeh
2018 International Symposium on Computer Architecture | June 2018
Reinforcement learning with optimized reward function for stealth applications
Matheus R. F. Mendonça, Heder S. Bernardino, Raul Fonseca Neto
Entertainment Computing | February 2018, Vol 25: pp. 37-47
-
Publications by Research Area
-
Magma: A Foundation Model for Multimodal AI Agents
Jianwei Yang, Reuben Tan, Qianhui Wu, Ruijie Zheng, Baolin Peng, Yongyuan Liang, Yu Gu, Mu Cai, Seonghyeon Ye, Joel Jang, Yuquan Deng, Lars Liden, Jianfeng Gao
February 2025
VLM-driven Behavior Tree for Context-aware Task Planning
Naoki Wake, Atsushi Kanehira, Jun Takamatsu, Kazuhiro Sasabuchi, Katsushi Ikeuchi
January 2025
Modality-Driven Design for Multi-Step Dexterous Manipulation: Insights from Neuroscience
Naoki Wake, Atsushi Kanehira, Daichi Saito, Jun Takamatsu, Kazuhiro Sasabuchi, Hideki Koike, Katsushi Ikeuchi
December 2024
UniGraspTransformer: Simplified Policy Distillation for Scalable Dexterous Robotic Grasping
Wenbo Wang, Fangyun Wei, Lei Zhou, Xi Chen, Lin Luo, Xiaohan Yi, Yizhong Zhang, Yaobo Liang, Chang Xu, Yan Lu, Jiaolong Yang, Baining Guo
CVPR 2025 | December 2024
CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulation
Qixiu Li, Yaobo Liang, Zeyu Wang, Lin Luo, Xi Chen, Mozheng Liao, Fangyun Wei, Yu Deng, Sicheng Xu, Yizhong Zhang, Xiaofan Wang, Bei Liu, Jianlong Fu, Jianmin Bao, Dong Chen, Yuanchun Shi, Jiaolong Yang, Baining Guo
November 2024
Scaling Laws for Pre-training Agents and World Models
Tim Pearce, Tabish Rashid, Dave Bignell, Raluca Stevenson, Sam Devlin, Katja Hofmann
November 2024
IGOR: Image-GOal Representations are the Atomic Control Units for Foundation Models in Embodied AI
Xiaoyu Chen, Junliang Guo, Tianyu He, Chuheng Zhang, Pushi Zhang, Derek Yang, Li Zhao, Jiang Bian
October 2024
APriCoT: Action Primitives based on Contact-state Transition for In-Hand Tool Manipulation
Daichi Saito, Atsushi Kanehira, Kazuhiro Sasabuchi, Naoki Wake, Jun Takamatsu, Hideki Koike, Katsushi Ikeuchi
2024 IEEE-RAS 23rd International Conference on Humanoid Robots (Humanoids) | July 2024
PRISE: LLM-Style Sequence Compression for Learning Temporal Action Abstractions in Control
Ruijie Zheng, Ching-An Cheng, Hal Daumé III, Furong Huang, Andrey Kolobov
ICML 2024 | June 2024
ORAL
Rapidly Adapting Policies to the Real World via Simulation-Guided Fine-Tuning
Patrick Yin, Tyler Westenbroek, Simran Bagaria, Kevin Huang, Ching-An Cheng, Andrey Kolobov, Abhishek Gupta
Under submission | June 2024
Interaction-Shaping Robotics: Robots that Influence Interactions between Other Agents
Sarah Gillet, Marynel Vázquez, Sean Andrist, Iolanda Leite, S. Sebo
ACM Transactions on Human-Robot Interaction | February 2024
Semantic constraints to represent common sense required in household actions for multimodal learning-from-observation robot
Katsushi Ikeuchi, Naoki Wake, Kazuhiro Sasabuchi, Jun Takamatsu
Int. J. Robotics Res. | November 2023, Vol 43(2): pp. 134-170
GPT-4V(ision) for Robotics: Multimodal Task Planning From Human Demonstration
Naoki Wake, Atsushi Kanehira, Kazuhiro Sasabuchi, Jun Takamatsu, Katsushi Ikeuchi
IEEE Robotics and Automation Letters | November 2023, Vol 9(11): pp. 10567-10574
Comparing the Efficacy of Fine-Tuning and Meta-Learning for Few-Shot Policy Imitation
Massimiliano Patacchiola, Mingfei Sun, Katja Hofmann, Richard E Turner
Conference on Lifelong Learning Agents | November 2023
Interactive Robot Learning from Verbal Correction
Huihan Liu, Alice Chen, Yuke Zhu, Adith Swaminathan, Andrey Kolobov, Ching-An Cheng
CoRL 2023 Workshop on Language and Robot Learning (LangRob) | November 2023
CASSPR: Cross Attention Single Scan Place Recognition
Yan Xia, Mariia Gladkova, Rui Wang, Qianyun Li, Uwe Stilla, Joao F. Henriques, Daniel Cremers
ICCV | October 2023
ChatGPT Empowered Long-Step Robot Control in Various Environments: A Case Application
Naoki Wake, Atsushi Kanehira, Kazuhiro Sasabuchi, Jun Takamatsu, Katsushi Ikeuchi
IEEE Access | August 2023, Vol 1: pp. 95060-95078
Open Access https://doi.org/10.1109/ACCESS.2023.3310935
R3D3: Dense 3D Reconstruction of Dynamic Scenes from Multiple Cameras
Aron Schmied, Tobias Fischer, Martin Danelljan, Marc Pollefeys, Fisher Yu
ICCV 2023 | August 2023
Harnessing AI and robotics in humanitarian assistance and disaster response
Thomas Manzini, Robin R. Murphy, Eric Heim, Caleb Robinson, Guido Zarrella, Ritwik Gupta
Science Robotics | July 2023
Provable Safe Reinforcement Learning with Binary Feedback
Andrew Bennett, Dipendra Misra, Nathan Kallus
2023 International Conference on Artificial Intelligence and Statistics | April 2023
ChatGPT for Robotics: Design Principles and Model Abilities
Sai Vemprala, Rogerio Bonatti, Arthur Bucker, Ashish Kapoor
MSR-TR-2023-8 | February 2023
Published by Microsoft
Guaranteed Discovery of Control-Endogenous Latent States with Multi-Step Inverse Models
Alex Lamb, Riashat Islam, Yonathan Efroni, Aniket Didolkar, Dipendra Misra, Dylan Foster, Olalekan Ogunmolu, Rajan Chari, Akshay Krishnamurthy, John Langford
Transactions on Machine Learning Research | February 2023
M³ViT: Mixture-of-Experts Vision Transformer for Efficient Multi-task Learning with Model-Accelerator Co-design
Hanxue Liang, Zhiwen Fan, Rishov Sarkar, Ziyu Jiang, Tianlong Chen, Yu Cheng, Cong Hao, Zhangyang Wang
36th Annual Conference on Neural Information Processing System (NeurIPS 2022) | December 2022
PACT: Perception-Action Causal Transformer for Autoregressive Robotics Pre-Training
Rogerio Bonatti, Sai Vemprala, Shuang Ma, Felipe Vieira Frujeri, Shuhang Chen, Ashish Kapoor
IROS 2023 | September 2022
MoBT 17.1 / MoBIP 17.1
Provably sample-efficient RL with side information about latent dynamics
Yao Liu, Dipendra Misra, Miro Dudík, Robert E. Schapire
2022 Neural Information Processing Systems | May 2022
Latent Variable Sequential Set Transformers For Joint Multi-Agent Motion Prediction
Roger Girgis, Florian Golemo, Felipe Codevilla, Martin Weiss, Jim Aldon D'Souza, Samira Ebrahimi Kahou, Felix Heide, Christopher Pal
ICLR 2022 | April 2022
Design strategies for controlling neuron-connected robots using reinforcement learning
Haruto Sawada, Naoki Wake, Kazuhiro Sasabuchi, Jun Takamatsu, Hirokazu Takahashi, Katsushi Ikeuchi
March 2022
Integration of Gesture Generation System Using Gesture Library with DIY Robot Design Kit
Hitoshi Teshima, Naoki Wake, Diego Thomas, Yuta Nakashima, David Baumert, Hiroshi Kawasaki, Katsushi Ikeuchi
2022 IEEE/SICE International Symposium on System Integration | January 2022
A Multimodal Learning-from-Observation Towards All-at-once Robot Teaching using Task Cohesion
Iori Yanokura, Naoki Wake, Kazuhiro Sasabuchi, Riku Arakawa, Kei Okada, Jun Takamatsu, Masayuki Inaba, Katsushi Ikeuchi
2022 IEEE/SICE International Symposium on System Integration | January 2022
ORBIT: A Real-World Few-Shot Dataset for Teachable Object Recognition
Daniela Massiceti, Luisa Zintgraf, John Bronskill, Lida Theodorou, Matthew Tobias Harris, Ed Cutrell, Cecily Morrison, Katja Hofmann, Simone Stumpf
ICCV 2021 | October 2021
Verbal Focus-of-Attention System for Learning-from-Observation
Naoki Wake, Iori Yanokura, Kazuhiro Sasabuchi, Katsushi Ikeuchi
2021 International Conference on Robotics and Automation | May 2021
Training Structured Mechanical Models by Minimizing Discrete Euler-Lagrange Residual
Kunal Menda, Jayesh K. Gupta, Zachary Manchester, Mykel J. Kochenderfer
MSR-TR-2021-35 | May 2021
Published by Microsoft
RMP2: A Structured Composable Policy Class for Robot Learning.
Anqi Li, Ching-An Cheng, Muhammad Asif Rana, Man Xie, Karl Van Wyk, Nathan D. Ratliff, Byron Boots
RSS | March 2021
Contact Web Status Presentation for Freehand Grasping in MR-based Robot-teaching
Daichi Saito, Naoki Wake, Kazuhiro Sasabuchi, Hideki Koike, Katsushi Ikeuchi
2021 Human-Robot Interaction | March 2021
A Computational Stack for Cross-Domain Acceleration
Sean Kinzer, Joon Kyung Kim, Soroush Ghodrati, Brahmendra Yatham, Alric Althoff, Divya Mahajan, Sorin Lerner, Hadi Esmailzadeh
International Symposium on High Performance Computer Architecture | February 2021
A Learning-from-Observation Framework: One-Shot Robot Teaching for Grasp-Manipulation-Release Household Operations
Naoki Wake, Riku Arakawa, Iori Yanokura, Takuya Kiyokawa, Kazuhiro Sasabuchi, Jun Takamatsu, Katsushi Ikeuchi
2021 IEEE/SICE International Symposium on System Integration | January 2021
SICE Young Investigator Award
Adversarial Attacks on Optimization based Planners
Sai Vemprala, Ashish Kapoor
October 2020
Variational Recurrent Models for Solving Partially Observable Control Tasks
Dongqi Han, Kenji Doya, Jun Tani
2020 International Conference on Learning Representations | May 2020
Enhancing listening capability of humanoid robot by reduction of stationary ego-noise
Naoki Wake, Masaaki Fukumoto, Hirokazu Takahashi, Katsushi Ikeuchi
IEEJ Transactions on Electrical and Electronic Engineering | December 2019, Vol 14(12): pp. 1815-1822
A Convergence Theory for Deep Learning via Over-Parameterization
Zeyuan Allen-Zhu, Yuanzhi Li, Zhao Song
September 2018
Automating Robotic Furniture with A Collaborative Vision-based Sensing Scheme
Zhizheng Zhang, Zhibo Chen, Weiping Li
IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN 2018) | August 2018
Robox: an end-to-end solution to accelerate autonomous control in robotics
Jacob Sacks, Divya Mahajan, Richard C. Lawson, Hadi Esmaeilzadeh
2018 International Symposium on Computer Architecture | June 2018
Reinforcement learning with optimized reward function for stealth applications
Matheus R. F. Mendonça, Heder S. Bernardino, Raul Fonseca Neto
Entertainment Computing | February 2018, Vol 25: pp. 37-47
A Review of Eye Gaze in Virtual Agents, Social Robotics and HCI: Behaviour Generation, User Interaction and Perception
Sean Andrist, et al
February 2015
-
Magma: A Foundation Model for Multimodal AI Agents
Jianwei Yang, Reuben Tan, Qianhui Wu, Ruijie Zheng, Baolin Peng, Yongyuan Liang, Yu Gu, Mu Cai, Seonghyeon Ye, Joel Jang, Yuquan Deng, Lars Liden, Jianfeng Gao
February 2025
VLM-driven Behavior Tree for Context-aware Task Planning
Naoki Wake, Atsushi Kanehira, Jun Takamatsu, Kazuhiro Sasabuchi, Katsushi Ikeuchi
January 2025
Modality-Driven Design for Multi-Step Dexterous Manipulation: Insights from Neuroscience
Naoki Wake, Atsushi Kanehira, Daichi Saito, Jun Takamatsu, Kazuhiro Sasabuchi, Hideki Koike, Katsushi Ikeuchi
December 2024
UniGraspTransformer: Simplified Policy Distillation for Scalable Dexterous Robotic Grasping
Wenbo Wang, Fangyun Wei, Lei Zhou, Xi Chen, Lin Luo, Xiaohan Yi, Yizhong Zhang, Yaobo Liang, Chang Xu, Yan Lu, Jiaolong Yang, Baining Guo
CVPR 2025 | December 2024
CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulation
Qixiu Li, Yaobo Liang, Zeyu Wang, Lin Luo, Xi Chen, Mozheng Liao, Fangyun Wei, Yu Deng, Sicheng Xu, Yizhong Zhang, Xiaofan Wang, Bei Liu, Jianlong Fu, Jianmin Bao, Dong Chen, Yuanchun Shi, Jiaolong Yang, Baining Guo
November 2024
Semantic constraints to represent common sense required in household actions for multimodal learning-from-observation robot
Katsushi Ikeuchi, Naoki Wake, Kazuhiro Sasabuchi, Jun Takamatsu
Int. J. Robotics Res. | November 2023, Vol 43(2): pp. 134-170
GPT-4V(ision) for Robotics: Multimodal Task Planning From Human Demonstration
Naoki Wake, Atsushi Kanehira, Kazuhiro Sasabuchi, Jun Takamatsu, Katsushi Ikeuchi
IEEE Robotics and Automation Letters | November 2023, Vol 9(11): pp. 10567-10574
CASSPR: Cross Attention Single Scan Place Recognition
Yan Xia, Mariia Gladkova, Rui Wang, Qianyun Li, Uwe Stilla, Joao F. Henriques, Daniel Cremers
ICCV | October 2023
R3D3: Dense 3D Reconstruction of Dynamic Scenes from Multiple Cameras
Aron Schmied, Tobias Fischer, Martin Danelljan, Marc Pollefeys, Fisher Yu
ICCV 2023 | August 2023
M³ViT: Mixture-of-Experts Vision Transformer for Efficient Multi-task Learning with Model-Accelerator Co-design
Hanxue Liang, Zhiwen Fan, Rishov Sarkar, Ziyu Jiang, Tianlong Chen, Yu Cheng, Cong Hao, Zhangyang Wang
36th Annual Conference on Neural Information Processing System (NeurIPS 2022) | December 2022
PACT: Perception-Action Causal Transformer for Autoregressive Robotics Pre-Training
Rogerio Bonatti, Sai Vemprala, Shuang Ma, Felipe Vieira Frujeri, Shuhang Chen, Ashish Kapoor
IROS 2023 | September 2022
MoBT 17.1 / MoBIP 17.1
Panoptic Multi-TSDFs: a Flexible Representation for Online Multi-resolution Volumetric Mapping and Long-term Dynamic Scene Consistency
Lukas Schmid, Jeffrey Delmerico, Johannes L. Schönberger, Juan Nieto, Marc Pollefeys, Roland Siegwart, Cesar Cadena
2022 International Conference on Robotics and Automation (ICRA) | May 2022
Spatial Computing and Intuitive Interaction: Bringing Mixed Reality and Robotics Together
Jeffrey Delmerico, Roi Poranne, Federica Bogo, Helen Oleynikova, Eric Vollenweider, Stelian Coros, Juan Nieto, Marc Pollefeys
February 2022
A Multimodal Learning-from-Observation Towards All-at-once Robot Teaching using Task Cohesion
Iori Yanokura, Naoki Wake, Kazuhiro Sasabuchi, Riku Arakawa, Kei Okada, Jun Takamatsu, Masayuki Inaba, Katsushi Ikeuchi
2022 IEEE/SICE International Symposium on System Integration | January 2022
ORBIT: A Real-World Few-Shot Dataset for Teachable Object Recognition
Daniela Massiceti, Luisa Zintgraf, John Bronskill, Lida Theodorou, Matthew Tobias Harris, Ed Cutrell, Cecily Morrison, Katja Hofmann, Simone Stumpf
ICCV 2021 | October 2021
Verbal Focus-of-Attention System for Learning-from-Observation
Naoki Wake, Iori Yanokura, Kazuhiro Sasabuchi, Katsushi Ikeuchi
2021 International Conference on Robotics and Automation | May 2021
Contact Web Status Presentation for Freehand Grasping in MR-based Robot-teaching
Daichi Saito, Naoki Wake, Kazuhiro Sasabuchi, Hideki Koike, Katsushi Ikeuchi
2021 Human-Robot Interaction | March 2021
A Learning-from-Observation Framework: One-Shot Robot Teaching for Grasp-Manipulation-Release Household Operations
Naoki Wake, Riku Arakawa, Iori Yanokura, Takuya Kiyokawa, Kazuhiro Sasabuchi, Jun Takamatsu, Katsushi Ikeuchi
2021 IEEE/SICE International Symposium on System Integration | January 2021
SICE Young Investigator Award
Freetures: Localization in Signed Distance Function Maps
Alexander Millane, Helen Oleynikova, Christian Lanegger, Jeffrey Delmerico, Juan Nieto, Roland Siegwart, Marc Pollefeys, Cesar Cadena
October 2020
gDLS*: Generalized Pose-and-Scale Estimation Given Scale and Gravity Priors
Victor Fragoso, Joseph DeGol, Gang Hua
CVPR 2020 | June 2020
Night-to-day image translation for retrieval-based localization
Asha Anoosheh, Torsten Sattler, Radu Timofte, Marc Pollefeys, Luc Van Gool
2019 International Conference on Robotics and Automation (ICRA) | May 2019
Describing upper-body motions based on Labanotation for learning-from-observation robot
Katsushi Ikeuchi, Zhaoyuan Ma, Zengqiang Yan, Shunsuke Kudoh, Minako Nakamura
International Journal of Computer Vision | December 2018, Vol 126(12): pp. 1415-1429
Automating Robotic Furniture with A Collaborative Vision-based Sensing Scheme
Zhizheng Zhang, Zhibo Chen, Weiping Li
IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN 2018) | August 2018
-
VLM-driven Behavior Tree for Context-aware Task Planning
Naoki Wake, Atsushi Kanehira, Jun Takamatsu, Kazuhiro Sasabuchi, Katsushi Ikeuchi
January 2025
Modality-Driven Design for Multi-Step Dexterous Manipulation: Insights from Neuroscience
Naoki Wake, Atsushi Kanehira, Daichi Saito, Jun Takamatsu, Kazuhiro Sasabuchi, Hideki Koike, Katsushi Ikeuchi
December 2024
UniGraspTransformer: Simplified Policy Distillation for Scalable Dexterous Robotic Grasping
Wenbo Wang, Fangyun Wei, Lei Zhou, Xi Chen, Lin Luo, Xiaohan Yi, Yizhong Zhang, Yaobo Liang, Chang Xu, Yan Lu, Jiaolong Yang, Baining Guo
CVPR 2025 | December 2024
CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulation
Qixiu Li, Yaobo Liang, Zeyu Wang, Lin Luo, Xi Chen, Mozheng Liao, Fangyun Wei, Yu Deng, Sicheng Xu, Yizhong Zhang, Xiaofan Wang, Bei Liu, Jianlong Fu, Jianmin Bao, Dong Chen, Yuanchun Shi, Jiaolong Yang, Baining Guo
November 2024
Semantic constraints to represent common sense required in household actions for multimodal learning-from-observation robot
Katsushi Ikeuchi, Naoki Wake, Kazuhiro Sasabuchi, Jun Takamatsu
Int. J. Robotics Res. | November 2023, Vol 43(2): pp. 134-170
GPT-4V(ision) for Robotics: Multimodal Task Planning From Human Demonstration
Naoki Wake, Atsushi Kanehira, Kazuhiro Sasabuchi, Jun Takamatsu, Katsushi Ikeuchi
IEEE Robotics and Automation Letters | November 2023, Vol 9(11): pp. 10567-10574
ChatGPT for Robotics: Design Principles and Model Abilities
Sai Vemprala, Rogerio Bonatti, Arthur Bucker, Ashish Kapoor
MSR-TR-2023-8 | February 2023
Published by Microsoft
Integration of Gesture Generation System Using Gesture Library with DIY Robot Design Kit
Hitoshi Teshima, Naoki Wake, Diego Thomas, Yuta Nakashima, David Baumert, Hiroshi Kawasaki, Katsushi Ikeuchi
2022 IEEE/SICE International Symposium on System Integration | January 2022
A Multimodal Learning-from-Observation Towards All-at-once Robot Teaching using Task Cohesion
Iori Yanokura, Naoki Wake, Kazuhiro Sasabuchi, Riku Arakawa, Kei Okada, Jun Takamatsu, Masayuki Inaba, Katsushi Ikeuchi
2022 IEEE/SICE International Symposium on System Integration | January 2022
ORBIT: A Real-World Few-Shot Dataset for Teachable Object Recognition
Daniela Massiceti, Luisa Zintgraf, John Bronskill, Lida Theodorou, Matthew Tobias Harris, Ed Cutrell, Cecily Morrison, Katja Hofmann, Simone Stumpf
ICCV 2021 | October 2021
HapticBots: Distributed Encountered-type Haptics for VR with Multiple Shape-changing Mobile Robots
Ryo Suzuki, Eyal Ofek, Mike Sinclair, Daniel Leithinger, Mar Gonzalez Franco
UIST 2021 | October 2021
Verbal Focus-of-Attention System for Learning-from-Observation
Naoki Wake, Iori Yanokura, Kazuhiro Sasabuchi, Katsushi Ikeuchi
2021 International Conference on Robotics and Automation | May 2021
Contact Web Status Presentation for Freehand Grasping in MR-based Robot-teaching
Daichi Saito, Naoki Wake, Kazuhiro Sasabuchi, Hideki Koike, Katsushi Ikeuchi
2021 Human-Robot Interaction | March 2021
A Learning-from-Observation Framework: One-Shot Robot Teaching for Grasp-Manipulation-Release Household Operations
Naoki Wake, Riku Arakawa, Iori Yanokura, Takuya Kiyokawa, Kazuhiro Sasabuchi, Jun Takamatsu, Katsushi Ikeuchi
2021 IEEE/SICE International Symposium on System Integration | January 2021
SICE Young Investigator Award
The future of work and wellbeing: A preliminary report from a series of conversations
Andrew L. Kun, Orit Shaer, Raffaella Sadun, Linda Ng Boyle, John D. Lee
August 2020
Remote Observation of Field Work on the Farm
Wendy Ju, Ilan Mandel, Kevin Weatherwax, Leila Takayama, Nikolas Martelaro, Denis Willett
August 2020
VR-Enabled Telepresence as a Bridge for People, Environments, and Experiences
Brennan Jones, Yaying Zhang, Priscilla N. Y. Wong, Sean Rintel, Yasamin Heshmat
CHI 2020 | April 2020
Workshop on Social VR Workshop
VROOM: Virtual Robot Overlay for Online Meetings
Brennan Jones, Yaying Zhang, Priscilla Wong, Sean Rintel
CHI 2020 | April 2020
CHI 2020 Extended Abstracts
PDF DOI Video Github Project Project Project Project Project
Automatic Curriculum Learning for Deep RL: A Short Survey
Rémy Portelas, Cédric Colas, Lilian Weng, Katja Hofmann, Pierre-Yves Oudeyer
International Joint Conference on Artificial Intelligence | March 2020
Visual prosody supports reading aloud expressively
Ann Bessemans, Maarten Renckens, Kevin Bormans, Erik Nuyts, Kevin Larson
Visible Language | December 2019, Vol 53(3): pp. 28-49
A Review of Eye Gaze in Virtual Agents, Social Robotics and HCI: Behaviour Generation, User Interaction and Perception
Sean Andrist, et al
February 2015
-
Self-maintaining [networked] systems: The rise of datacenter robotics!
Freddie Hong, Iason Sarantopoulos, Elliott Hogg, David Richardson, Yizhong Zhang, Hugh Williams, David Sweeney, Andromachi Chatzieleftheriou, Ant Rowstron
2024 Hot Topics in Networks | November 2024
RASCAL: A Scalable, High-redundancy Robot for Automated Storage and Retrieval Systems
Richard Black, Marco Caballero, Andromachi Chatzieleftheriou, Tim Deegan, Philip Heard, Freddie Hong, Russell Joyce, Sergey Legtchenko, Ant Rowstron, Adam Smith, David Sweeney, Hugh Williams
2024 IEEE International Conference on Robotics and Automation | May 2024
SMOP: Scaling Up Real-time Collaborative Visual SLAM at the Edge
Jingao Xu, Hao Cao, Zheng Yang, Longfei Shangguan, Jialin Zhang, Xiaowu He, Yunhao Liu
NSDI 2022 | March 2022
-
RASCAL: A Scalable, High-redundancy Robot for Automated Storage and Retrieval Systems
Richard Black, Marco Caballero, Andromachi Chatzieleftheriou, Tim Deegan, Philip Heard, Freddie Hong, Russell Joyce, Sergey Legtchenko, Ant Rowstron, Adam Smith, David Sweeney, Hugh Williams
2024 IEEE International Conference on Robotics and Automation | May 2024
HapticBots: Distributed Encountered-type Haptics for VR with Multiple Shape-changing Mobile Robots
Ryo Suzuki, Eyal Ofek, Mike Sinclair, Daniel Leithinger, Mar Gonzalez Franco
UIST 2021 | October 2021
A Computational Stack for Cross-Domain Acceleration
Sean Kinzer, Joon Kyung Kim, Soroush Ghodrati, Brahmendra Yatham, Alric Althoff, Divya Mahajan, Sorin Lerner, Hadi Esmailzadeh
International Symposium on High Performance Computer Architecture | February 2021
Robox: an end-to-end solution to accelerate autonomous control in robotics
Jacob Sacks, Divya Mahajan, Richard C. Lawson, Hadi Esmaeilzadeh
2018 International Symposium on Computer Architecture | June 2018
-
M³ViT: Mixture-of-Experts Vision Transformer for Efficient Multi-task Learning with Model-Accelerator Co-design
Hanxue Liang, Zhiwen Fan, Rishov Sarkar, Ziyu Jiang, Tianlong Chen, Yu Cheng, Cong Hao, Zhangyang Wang
36th Annual Conference on Neural Information Processing System (NeurIPS 2022) | December 2022
Visual prosody supports reading aloud expressively
Ann Bessemans, Maarten Renckens, Kevin Bormans, Erik Nuyts, Kevin Larson
Visible Language | December 2019, Vol 53(3): pp. 28-49
-
Reinforcement learning with optimized reward function for stealth applications
Matheus R. F. Mendonça, Heder S. Bernardino, Raul Fonseca Neto
Entertainment Computing | February 2018, Vol 25: pp. 37-47
-
Enhancing listening capability of humanoid robot by reduction of stationary ego-noise
Naoki Wake, Masaaki Fukumoto, Hirokazu Takahashi, Katsushi Ikeuchi
IEEJ Transactions on Electrical and Electronic Engineering | December 2019, Vol 14(12): pp. 1815-1822
-
A Computational Stack for Cross-Domain Acceleration
Sean Kinzer, Joon Kyung Kim, Soroush Ghodrati, Brahmendra Yatham, Alric Althoff, Divya Mahajan, Sorin Lerner, Hadi Esmailzadeh
International Symposium on High Performance Computer Architecture | February 2021
-
The future of work and wellbeing: A preliminary report from a series of conversations
Andrew L. Kun, Orit Shaer, Raffaella Sadun, Linda Ng Boyle, John D. Lee
August 2020
Remote Observation of Field Work on the Farm
Wendy Ju, Ilan Mandel, Kevin Weatherwax, Leila Takayama, Nikolas Martelaro, Denis Willett
August 2020
VR-Enabled Telepresence as a Bridge for People, Environments, and Experiences
Brennan Jones, Yaying Zhang, Priscilla N. Y. Wong, Sean Rintel, Yasamin Heshmat
CHI 2020 | April 2020
Workshop on Social VR Workshop
VROOM: Virtual Robot Overlay for Online Meetings
Brennan Jones, Yaying Zhang, Priscilla Wong, Sean Rintel
CHI 2020 | April 2020
CHI 2020 Extended Abstracts
PDF DOI Video Github Project Project Project Project Project
Publications by Type
-
UniGraspTransformer: Simplified Policy Distillation for Scalable Dexterous Robotic Grasping
Wenbo Wang, Fangyun Wei, Lei Zhou, Xi Chen, Lin Luo, Xiaohan Yi, Yizhong Zhang, Yaobo Liang, Chang Xu, Yan Lu, Jiaolong Yang, Baining Guo
CVPR 2025 | December 2024
Self-maintaining [networked] systems: The rise of datacenter robotics!
Freddie Hong, Iason Sarantopoulos, Elliott Hogg, David Richardson, Yizhong Zhang, Hugh Williams, David Sweeney, Andromachi Chatzieleftheriou, Ant Rowstron
2024 Hot Topics in Networks | November 2024
IGOR: Image-GOal Representations are the Atomic Control Units for Foundation Models in Embodied AI
Xiaoyu Chen, Junliang Guo, Tianyu He, Chuheng Zhang, Pushi Zhang, Derek Yang, Li Zhao, Jiang Bian
October 2024
APriCoT: Action Primitives based on Contact-state Transition for In-Hand Tool Manipulation
Daichi Saito, Atsushi Kanehira, Kazuhiro Sasabuchi, Naoki Wake, Jun Takamatsu, Hideki Koike, Katsushi Ikeuchi
2024 IEEE-RAS 23rd International Conference on Humanoid Robots (Humanoids) | July 2024
PRISE: LLM-Style Sequence Compression for Learning Temporal Action Abstractions in Control
Ruijie Zheng, Ching-An Cheng, Hal Daumé III, Furong Huang, Andrey Kolobov
ICML 2024 | June 2024
ORAL
Rapidly Adapting Policies to the Real World via Simulation-Guided Fine-Tuning
Patrick Yin, Tyler Westenbroek, Simran Bagaria, Kevin Huang, Ching-An Cheng, Andrey Kolobov, Abhishek Gupta
Under submission | June 2024
RASCAL: A Scalable, High-redundancy Robot for Automated Storage and Retrieval Systems
Richard Black, Marco Caballero, Andromachi Chatzieleftheriou, Tim Deegan, Philip Heard, Freddie Hong, Russell Joyce, Sergey Legtchenko, Ant Rowstron, Adam Smith, David Sweeney, Hugh Williams
2024 IEEE International Conference on Robotics and Automation | May 2024
Comparing the Efficacy of Fine-Tuning and Meta-Learning for Few-Shot Policy Imitation
Massimiliano Patacchiola, Mingfei Sun, Katja Hofmann, Richard E Turner
Conference on Lifelong Learning Agents | November 2023
CASSPR: Cross Attention Single Scan Place Recognition
Yan Xia, Mariia Gladkova, Rui Wang, Qianyun Li, Uwe Stilla, Joao F. Henriques, Daniel Cremers
ICCV | October 2023
R3D3: Dense 3D Reconstruction of Dynamic Scenes from Multiple Cameras
Aron Schmied, Tobias Fischer, Martin Danelljan, Marc Pollefeys, Fisher Yu
ICCV 2023 | August 2023
Provable Safe Reinforcement Learning with Binary Feedback
Andrew Bennett, Dipendra Misra, Nathan Kallus
2023 International Conference on Artificial Intelligence and Statistics | April 2023
M³ViT: Mixture-of-Experts Vision Transformer for Efficient Multi-task Learning with Model-Accelerator Co-design
Hanxue Liang, Zhiwen Fan, Rishov Sarkar, Ziyu Jiang, Tianlong Chen, Yu Cheng, Cong Hao, Zhangyang Wang
36th Annual Conference on Neural Information Processing System (NeurIPS 2022) | December 2022
PACT: Perception-Action Causal Transformer for Autoregressive Robotics Pre-Training
Rogerio Bonatti, Sai Vemprala, Shuang Ma, Felipe Vieira Frujeri, Shuhang Chen, Ashish Kapoor
IROS 2023 | September 2022
MoBT 17.1 / MoBIP 17.1
Provably sample-efficient RL with side information about latent dynamics
Yao Liu, Dipendra Misra, Miro Dudík, Robert E. Schapire
2022 Neural Information Processing Systems | May 2022
Panoptic Multi-TSDFs: a Flexible Representation for Online Multi-resolution Volumetric Mapping and Long-term Dynamic Scene Consistency
Lukas Schmid, Jeffrey Delmerico, Johannes L. Schönberger, Juan Nieto, Marc Pollefeys, Roland Siegwart, Cesar Cadena
2022 International Conference on Robotics and Automation (ICRA) | May 2022
Latent Variable Sequential Set Transformers For Joint Multi-Agent Motion Prediction
Roger Girgis, Florian Golemo, Felipe Codevilla, Martin Weiss, Jim Aldon D'Souza, Samira Ebrahimi Kahou, Felix Heide, Christopher Pal
ICLR 2022 | April 2022
SMOP: Scaling Up Real-time Collaborative Visual SLAM at the Edge
Jingao Xu, Hao Cao, Zheng Yang, Longfei Shangguan, Jialin Zhang, Xiaowu He, Yunhao Liu
NSDI 2022 | March 2022
Spatial Computing and Intuitive Interaction: Bringing Mixed Reality and Robotics Together
Jeffrey Delmerico, Roi Poranne, Federica Bogo, Helen Oleynikova, Eric Vollenweider, Stelian Coros, Juan Nieto, Marc Pollefeys
February 2022
Integration of Gesture Generation System Using Gesture Library with DIY Robot Design Kit
Hitoshi Teshima, Naoki Wake, Diego Thomas, Yuta Nakashima, David Baumert, Hiroshi Kawasaki, Katsushi Ikeuchi
2022 IEEE/SICE International Symposium on System Integration | January 2022
A Multimodal Learning-from-Observation Towards All-at-once Robot Teaching using Task Cohesion
Iori Yanokura, Naoki Wake, Kazuhiro Sasabuchi, Riku Arakawa, Kei Okada, Jun Takamatsu, Masayuki Inaba, Katsushi Ikeuchi
2022 IEEE/SICE International Symposium on System Integration | January 2022
ORBIT: A Real-World Few-Shot Dataset for Teachable Object Recognition
Daniela Massiceti, Luisa Zintgraf, John Bronskill, Lida Theodorou, Matthew Tobias Harris, Ed Cutrell, Cecily Morrison, Katja Hofmann, Simone Stumpf
ICCV 2021 | October 2021
HapticBots: Distributed Encountered-type Haptics for VR with Multiple Shape-changing Mobile Robots
Ryo Suzuki, Eyal Ofek, Mike Sinclair, Daniel Leithinger, Mar Gonzalez Franco
UIST 2021 | October 2021
Verbal Focus-of-Attention System for Learning-from-Observation
Naoki Wake, Iori Yanokura, Kazuhiro Sasabuchi, Katsushi Ikeuchi
2021 International Conference on Robotics and Automation | May 2021
RMP2: A Structured Composable Policy Class for Robot Learning.
Anqi Li, Ching-An Cheng, Muhammad Asif Rana, Man Xie, Karl Van Wyk, Nathan D. Ratliff, Byron Boots
RSS | March 2021
Contact Web Status Presentation for Freehand Grasping in MR-based Robot-teaching
Daichi Saito, Naoki Wake, Kazuhiro Sasabuchi, Hideki Koike, Katsushi Ikeuchi
2021 Human-Robot Interaction | March 2021
A Computational Stack for Cross-Domain Acceleration
Sean Kinzer, Joon Kyung Kim, Soroush Ghodrati, Brahmendra Yatham, Alric Althoff, Divya Mahajan, Sorin Lerner, Hadi Esmailzadeh
International Symposium on High Performance Computer Architecture | February 2021
A Learning-from-Observation Framework: One-Shot Robot Teaching for Grasp-Manipulation-Release Household Operations
Naoki Wake, Riku Arakawa, Iori Yanokura, Takuya Kiyokawa, Kazuhiro Sasabuchi, Jun Takamatsu, Katsushi Ikeuchi
2021 IEEE/SICE International Symposium on System Integration | January 2021
SICE Young Investigator Award
gDLS*: Generalized Pose-and-Scale Estimation Given Scale and Gravity Priors
Victor Fragoso, Joseph DeGol, Gang Hua
CVPR 2020 | June 2020
Variational Recurrent Models for Solving Partially Observable Control Tasks
Dongqi Han, Kenji Doya, Jun Tani
2020 International Conference on Learning Representations | May 2020
VR-Enabled Telepresence as a Bridge for People, Environments, and Experiences
Brennan Jones, Yaying Zhang, Priscilla N. Y. Wong, Sean Rintel, Yasamin Heshmat
CHI 2020 | April 2020
Workshop on Social VR Workshop
VROOM: Virtual Robot Overlay for Online Meetings
Brennan Jones, Yaying Zhang, Priscilla Wong, Sean Rintel
CHI 2020 | April 2020
CHI 2020 Extended Abstracts
PDF DOI Video Github Project Project Project Project Project
Automatic Curriculum Learning for Deep RL: A Short Survey
Rémy Portelas, Cédric Colas, Lilian Weng, Katja Hofmann, Pierre-Yves Oudeyer
International Joint Conference on Artificial Intelligence | March 2020
Night-to-day image translation for retrieval-based localization
Asha Anoosheh, Torsten Sattler, Radu Timofte, Marc Pollefeys, Luc Van Gool
2019 International Conference on Robotics and Automation (ICRA) | May 2019
Automating Robotic Furniture with A Collaborative Vision-based Sensing Scheme
Zhizheng Zhang, Zhibo Chen, Weiping Li
IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN 2018) | August 2018
Robox: an end-to-end solution to accelerate autonomous control in robotics
Jacob Sacks, Divya Mahajan, Richard C. Lawson, Hadi Esmaeilzadeh
2018 International Symposium on Computer Architecture | June 2018
-
Magma: A Foundation Model for Multimodal AI Agents
Jianwei Yang, Reuben Tan, Qianhui Wu, Ruijie Zheng, Baolin Peng, Yongyuan Liang, Yu Gu, Mu Cai, Seonghyeon Ye, Joel Jang, Yuquan Deng, Lars Liden, Jianfeng Gao
February 2025
VLM-driven Behavior Tree for Context-aware Task Planning
Naoki Wake, Atsushi Kanehira, Jun Takamatsu, Kazuhiro Sasabuchi, Katsushi Ikeuchi
January 2025
Modality-Driven Design for Multi-Step Dexterous Manipulation: Insights from Neuroscience
Naoki Wake, Atsushi Kanehira, Daichi Saito, Jun Takamatsu, Kazuhiro Sasabuchi, Hideki Koike, Katsushi Ikeuchi
December 2024
CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulation
Qixiu Li, Yaobo Liang, Zeyu Wang, Lin Luo, Xi Chen, Mozheng Liao, Fangyun Wei, Yu Deng, Sicheng Xu, Yizhong Zhang, Xiaofan Wang, Bei Liu, Jianlong Fu, Jianmin Bao, Dong Chen, Yuanchun Shi, Jiaolong Yang, Baining Guo
November 2024
Freetures: Localization in Signed Distance Function Maps
Alexander Millane, Helen Oleynikova, Christian Lanegger, Jeffrey Delmerico, Juan Nieto, Roland Siegwart, Marc Pollefeys, Cesar Cadena
October 2020
A Convergence Theory for Deep Learning via Over-Parameterization
Zeyuan Allen-Zhu, Yuanzhi Li, Zhao Song
September 2018
A Review of Eye Gaze in Virtual Agents, Social Robotics and HCI: Behaviour Generation, User Interaction and Perception
Sean Andrist, et al
February 2015
-
Interaction-Shaping Robotics: Robots that Influence Interactions between Other Agents
Sarah Gillet, Marynel Vázquez, Sean Andrist, Iolanda Leite, S. Sebo
ACM Transactions on Human-Robot Interaction | February 2024
Semantic constraints to represent common sense required in household actions for multimodal learning-from-observation robot
Katsushi Ikeuchi, Naoki Wake, Kazuhiro Sasabuchi, Jun Takamatsu
Int. J. Robotics Res. | November 2023, Vol 43(2): pp. 134-170
GPT-4V(ision) for Robotics: Multimodal Task Planning From Human Demonstration
Naoki Wake, Atsushi Kanehira, Kazuhiro Sasabuchi, Jun Takamatsu, Katsushi Ikeuchi
IEEE Robotics and Automation Letters | November 2023, Vol 9(11): pp. 10567-10574
Interactive Robot Learning from Verbal Correction
Huihan Liu, Alice Chen, Yuke Zhu, Adith Swaminathan, Andrey Kolobov, Ching-An Cheng
CoRL 2023 Workshop on Language and Robot Learning (LangRob) | November 2023
ChatGPT Empowered Long-Step Robot Control in Various Environments: A Case Application
Naoki Wake, Atsushi Kanehira, Kazuhiro Sasabuchi, Jun Takamatsu, Katsushi Ikeuchi
IEEE Access | August 2023, Vol 1: pp. 95060-95078
Open Access https://doi.org/10.1109/ACCESS.2023.3310935
Harnessing AI and robotics in humanitarian assistance and disaster response
Thomas Manzini, Robin R. Murphy, Eric Heim, Caleb Robinson, Guido Zarrella, Ritwik Gupta
Science Robotics | July 2023
Guaranteed Discovery of Control-Endogenous Latent States with Multi-Step Inverse Models
Alex Lamb, Riashat Islam, Yonathan Efroni, Aniket Didolkar, Dipendra Misra, Dylan Foster, Olalekan Ogunmolu, Rajan Chari, Akshay Krishnamurthy, John Langford
Transactions on Machine Learning Research | February 2023
Enhancing listening capability of humanoid robot by reduction of stationary ego-noise
Naoki Wake, Masaaki Fukumoto, Hirokazu Takahashi, Katsushi Ikeuchi
IEEJ Transactions on Electrical and Electronic Engineering | December 2019, Vol 14(12): pp. 1815-1822
Visual prosody supports reading aloud expressively
Ann Bessemans, Maarten Renckens, Kevin Bormans, Erik Nuyts, Kevin Larson
Visible Language | December 2019, Vol 53(3): pp. 28-49
Describing upper-body motions based on Labanotation for learning-from-observation robot
Katsushi Ikeuchi, Zhaoyuan Ma, Zengqiang Yan, Shunsuke Kudoh, Minako Nakamura
International Journal of Computer Vision | December 2018, Vol 126(12): pp. 1415-1429
Reinforcement learning with optimized reward function for stealth applications
Matheus R. F. Mendonça, Heder S. Bernardino, Raul Fonseca Neto
Entertainment Computing | February 2018, Vol 25: pp. 37-47
-
Scaling Laws for Pre-training Agents and World Models
Tim Pearce, Tabish Rashid, Dave Bignell, Raluca Stevenson, Sam Devlin, Katja Hofmann
November 2024
Design strategies for controlling neuron-connected robots using reinforcement learning
Haruto Sawada, Naoki Wake, Kazuhiro Sasabuchi, Jun Takamatsu, Hirokazu Takahashi, Katsushi Ikeuchi
March 2022
Adversarial Attacks on Optimization based Planners
Sai Vemprala, Ashish Kapoor
October 2020
The future of work and wellbeing: A preliminary report from a series of conversations
Andrew L. Kun, Orit Shaer, Raffaella Sadun, Linda Ng Boyle, John D. Lee
August 2020
Remote Observation of Field Work on the Farm
Wendy Ju, Ilan Mandel, Kevin Weatherwax, Leila Takayama, Nikolas Martelaro, Denis Willett
August 2020
-
ChatGPT for Robotics: Design Principles and Model Abilities
Sai Vemprala, Rogerio Bonatti, Arthur Bucker, Ashish Kapoor
MSR-TR-2023-8 | February 2023
Published by Microsoft
Training Structured Mechanical Models by Minimizing Discrete Euler-Lagrange Residual
Kunal Menda, Jayesh K. Gupta, Zachary Manchester, Mykel J. Kochenderfer
MSR-TR-2021-35 | May 2021
Published by Microsoft