Why do we make our robots shop for groceries |  By Toyota Research Institute |  Toyota Research Institute |  October 2022

Why do we make our robots shop for groceries | By Toyota Research Institute | Toyota Research Institute | October 2022

How Challenge Missions are driving the development of our mobile manipulation systems

By the TRI Mobile Manipulation team, including James Borders, Richard Cheng, Dan Helmick, Lukas Kaul, Dan Kruse, John Leichty, Carolyn Matl, Chavdar Papazov and Mark Tjersland

Toyota Research Institute robot
The latest version of TRI’s fully customized two-armed mobile manipulation robot

At TRI, we are developing robotic capabilities with the goal of improving the quality of everyday life for everyone. To reach this goal, we define exciting “challenge tasks” to work on, which drive our development towards general-purpose robot capabilities, and that allow for rigorous quantitative testing

Fulfilling autonomous orders in grocery stores is a particularly good way to drive our development of mobile manipulation capabilities because it involves a set of challenging challenges for robots, including perceiving and handling a large variety of objects, navigating an ever-changing environment, and interacting with unforeseen circumstances. A single shopping process can have a long list of items, so this task requires the system to be reliable and encourages a focus on overall execution speed. We use many intuitive metrics to measure progress. How many items did the bot retrieve correctly? How much did you take by mistake? How long did it take? Best of all, we are able to recreate representative shopping aisles within our robotics labs, allowing us to quickly iterate between tests in real grocery stores.

Today, we provide an insight into the work that goes into enabling our bots to do self-grocery shopping. We reveal the current iteration of our mobile processing bot platform and highlight some of the key technologies and techniques we’ve developed. These developments can be used for much more than grocery shopping.

One use case that in particular motivates us is helping the elderly with physical tasks. We envision that our developments will lead to advanced robots that can support older adults autonomously in their homes and enable them to lead more fulfilling lives, without completely replacing their decisions.[1].

We are constantly improving our two-arm mobile manipulation robots. Since the creation of our first robot platform (shown in [2]), we have developed a fully customized bot platform for advanced mobile manipulation. This bot relies on TRI’s custom bot actuators of various sizes, as well as the software ecosystem to support it. These actuators are the components that drive every joint of the robot – they direct the wheels, move the 5 degrees of freedom (DOF) torso, move the seven DOF arms, and control the neck. Our actuators encapsulate most of the complexity of a mechanical system and allow us to quickly iterate over different robot designs. They have a unique high torque density that enables us to build sticks that are very thin and strong enough to carry even the heaviest of everyday objects. The robot is fully self-contained with a high-performance computing system, over 2 kWh of fast charging and a hot swappable battery capacity to support extensive testing without the need for a power cord. We can remove the battery modules and reinsert them into the robot while the system is still running.

Cross-section: A cross-section of one of our custom common triggers
One of our robots is holding an eight-pound oil pan

Our bots rely on a pre-built map of the grocery store to find the items they’re looking for. The map contains a detailed 3D geometric reconstruction of the shop as well as the locations of a large number of items. To this end, we manually move a dedicated data collection cart through the store and record a stream of images captured by multiple stereo cameras. Based on this data, our system creates a map in two main steps. First, it performs a detailed 3D geometric reconstruction of the entire space. Next, it detects the objects in the captured images and compares them to a set of items stored in a database. If the system recognizes an object, it adds the object to the map at the correct location in the space given by the 3D geometry calculated in the first step. To successfully solve these challenging tasks, we have developed a pipeline that combines the robustness of modern deep learning approaches with the precision of classical engineering algorithms.

Top view of a 3D map of a grocery store where we’re testing our systems
Detailed 3D map view of grocery shelves in our lab, with coordination frames at each particular item location

Equipped with a grocery store map, we test our system by generating a random shopping list. Next, we ask the bot to return as many items from the shopping list as possible. From now on, the robot becomes completely autonomous and unrestricted, running all the required calculations on board. He plans an efficient itinerary to visit all the items on the shopping list and then begins driving to the first item on the path. As the design of real grocery stores is constantly changing, the robot uses its own stereoscopic vision system [3,4] To navigate around special displays, wet floor signs, or other obstructions that may have appeared since the map was created.

Once the robot reaches the location of the particular item, it uses its stereo cameras to verify that the item is still there and to determine its exact position. A variety of circumstances can arise, and our detection algorithms and bot behaviors all need to be aggressively addressed. For example, the item may be unavailable, or it may have been moved to a different shelf. Packaging may change seasonally or similar to that of a new item. Items may not fit perfectly in front of the camera if they are rotated, placed on the top or bottom shelf, or placed farther toward the back of the shelf. All of these differences are wonderful test cases for our methods of cognition. One way we improve our chances of getting a good discrimination of things is by running the same neural networks on data from two pairs of stereo cameras, one in the head and one in the mobile base of the robot. This helps greatly in increasing coverage across the entire height of the grocery shelf.

Engineering our element classification system
Grocery rack from a robot’s point of view (it has two!)

Once the robot has successfully located the item, it plans how to catch it. Given the variety of items (in terms of weight, shape, size, and stiffness) represented in the grocery store, we outfitted the robot with a custom suction handle and a two-finger parallel handle that was ready to use.

The robot uses a ready-to-use parallel gripper and a dedicated suction tool to grip a variety of items

During operation, the robot uses its stereo cameras to obtain a rich 3D geometric representation of the object and infer properties such as the element’s dimensions, position, and surface curvature. We take advantage of a PointNet-based neural network model to determine which tool to use and what kind of understanding to use. The system uses the output of this grid with processed 3D geometric information to understand the element. He might grab the bottle cap instead of the body or place the rubber suction cup on a flat area of ​​the jar, rather than the curve.

The robot automatically determines the best spot on an item to achieve a successful suction grip

To quickly and successfully put the tool into the enhanced grip position, we have developed a custom high-powered motion chart. Our scheme combines concepts from dynamic probabilistic roadmaps with the benefit of GPU acceleration and custom inverse motion analyzers to quickly create motion plans for our 29 DOF robot, even in narrow grocery aisles. Because tool placement is critical to success, the robot checks and corrects the relative position of the tool and items as necessary, using an iterative nearest-point algorithm before closing the clutch or operating the suction pump. If the sensors in the gadget indicate that the object has been successfully captured, the robot will put it in its cart and move on to the next item in its path. If not, try again.

A selection of random shopping lists during a field test at a local grocery store (3x speed)

We’ve been doing multi-day field tests at a local grocery store in Mountain View, California every three months for the past year. In these field tests, members of the TRI Prototyping and Research (PROPS) team send the bot to shop for several hours each night, gathering invaluable data that allows us to measure our progress, learn from detailed analysis of failures, and quickly test new ideas on the ground. World data. As a result, our robots are constantly getting better and faster in dealing with an ever-increasing variety of items.

JC Hancook of TRI and colleagues on TRI’s PROPS team conduct rigorous testing and data collection that allows capabilities to be developed rapidly

Fulfilling grocery orders continues to challenge and inspire us to devise new approaches to the challenging problems facing mobile robots, and we believe it has already brought us closer to our vision of a practical and reliable robot companion that can improve quality of life. We’ve made breakthroughs in powerful motion perception, manipulation, and planning methods that advance the field of robotics in beneficial ways. We are excited to apply our technologies to other areas to continue to innovate quickly and maximize our impact.

You can learn more about our work in this exclusive CNET video. And if our goals and methods sound like something you’ll enjoy working on, then think Join our team!

#robots #shop #groceries #Toyota #Research #Institute #Toyota #Research #Institute #October

Leave a Comment

Your email address will not be published. Required fields are marked *