How Robots are Helping ChatGPT in the Real World

Microsoft researchers are using ChatGPT to educate robots and drones. Be ready for these Human-to-Robot conversations.

In addition to producing textually coherent responses to natural language questions, OpenAI's ChatGPT can also participate in human-to-robot conversations and use sensor data to generate program code for robot activities. To assist with robotics chores, Microsoft has performed research to "see if ChatGPT can go beyond text and reason about the actual world." Now Robots are helping ChatGPT.

The experiment's goal was to determine whether users of ChatGPT could control robots without having any prior knowledge of robotics or programming languages. A team of researchers at Microsoft is now pushing AI technology even further by allowing ChatGPT to leave the limits of the computer and be free in the real world. Microsoft's cooperation with OpenAI, the company that created ChatGPT, has already sparked an AI-fueled frenzy in the tech industry. The purpose of this project, according to Microsoft researchers, is to determine whether ChatGPT is capable of reasoning about the physical world in addition to text to assist with robotics tasks. The researchers at Microsoft envision a time when the typical person may yell commands like "please warm up my lunch" at a home robot assistant and see it accomplish the task from beginning to end.

Because the AI model was trained on enormous libraries of human language, including the code for software applications, ChatGPT is capable of doing this. Based on text-based requests, ChatGPT has previously demonstrated that it can develop and debug programs in a variety of languages. The researchers at Microsoft then decided to test whether they could use the same talents to create software for robotics hardware. The researchers concluded that while ChatGPT can accomplish a lot on its own, it still needs assistance. The researchers initially described to the AI software the various commands it could use to manage a certain robot to aid ChatGPT in writing the computer code. We provide a text prompt for ChatGPT that expressly lists the functions from the high-level library that are available while also outlining the task's objective. The question may also specify task restrictions or the format in which ChatGPT should formulate its responses, the researchers continued.

The researchers used the method in various demonstrations, one of which involved writing computer code to control an aerial drone using ChatGPT. The AI chatbot was first given a lengthy prompt from Microsoft researchers outlining the computer commands it could enter to operate the drone. Following that, the researchers could issue commands to ChatGPT to tell it how to control the robot in different ways.

This included requesting ChatGPT to recognize drinks like coconut water and beer using the drone's camera. In their article, Microsoft researchers offer some recommendations for crafting a useful ChatGPT prompt for managing robots. Also, the team developed an open-source GitHub portal "where anyone may share examples of prompting tactics for various robotics categories."

Researchers are currently cautioning people not to use ChatGPT to control robots. We stress that these tools shouldn't have complete control over the robotics pipeline, especially for safety-critical applications, they said in their paper. Because that large language models (LLMs) often produce inaccurate answers, it is crucial to check the solution's quality and the safety of the code under human supervision before running it on a robot.