Credit: Nvidia
NVIDIA engineers have created an AI controller called SONIC that translates natural language commands, video, and VR glove movements into coordinated actions for a humanoid robot. The work has been published in Science Robotics journal.
Essentially, this is a foundational model for controlling the movements of a humanoid robot. It translates high-level commands into coordinated whole-body movements.
To train the model, researchers used over 100 million frames of human motion data. SONIC was then tested in physics simulations across various tasks involving locomotion and object manipulation. During testing, the system was able to reproduce not only movements found in the training data but also novel variations.
Notably, SONIC does not require retraining for each new input method. The researchers demonstrated the controller operating via VR-based telepresence, using motion extracted from video, and in conjunction with vision-language-action models that link visual information, language, and actions.
Trials with a humanoid robot showed that a single controller could generate a wide range of stable, natural movements. The researchers also demonstrated remote robot control and the conversion of written instructions into physical actions.
The authors believe this approach could simplify the development of humanoid systems tasked with performing diverse physical operations. They cite manufacturing, warehousing, and logistics as potential application areas.
The team has already integrated SONIC with other components to demonstrate remote teleoperation of the robot and the translation of written instructions into specific actions. Moving forward, the researchers plan to enhance the robot’s environmental awareness to reduce collision risks, improve navigation over uneven terrain, refine contact-based manipulation, and facilitate the transfer of skills from simulation to the real world, as well as more tightly integrate SONIC with the NVIDIA Isaac GR00T platform.
