29 September 2015

This is the first part of several essays on the principle of grounding. I feel that this principle helps me communicate and think better. I’ll try to keep each essay short

A few years ago I was doing my masters, with the goal of using language to command robots. I was very excited because the research was very interesting to me - I needed to understand how to write software for robots, how the robot works as a system, and various fascinating aspects of human-robot interaction and linguistics. It felt super futuristic to be at the meeting point of so many disciplines.

Specifically, I was trying to teach a robot to follow some simple commands such as Pick up the cup. I was working with a robot which had two hands, and a 3D scanning sensor for eyes, and was usually facing a table with a bunch of standard household objects on top of it. If the robot was facing a specific table with specific object configuration, it is possible to program and hard-code the robot arm to move towards the cup, grab the cup, and then lift it. However, as soon as something tiny changes in the configuration, the position of the cup the robot, or the command, then the hard-coded solution would no longer work, instead making the robot wave its hands in air clumsily. Programming the robot in a single situation may be fun, but doing the same thing over and over will get boring and annoying quite fast, and on top of it wouldn’t be effective.

My task was to make the robot succeed when the cup looks differently, or is at a different location, or even if the command asked for a different item. In each new environment, the robot needed to connect the abstract symbol for cup, bowl, bottle from the command to the sensory information from the environment, figure out the right motions and execute them.

Luckily, I didn’t have to start from scratch. I stepped on the shoulders of giants, my professor and the senior graduate students, who had already invented algorithms for this meeting point of disciplines. They created techniques for connecting the symbolic command to the physical reality, albeit for a very different robot, in a different setting. My task was to port the algorithm to this new robot and make sure it works, helping justify the algorithm’s ability to adapt to new situations.

The algorithm connected the text of the command and the sensory perceptions of the robot and its imaginary motions. This is grounding, as I came to know it. Grounding is the connecting between something completely abstract and symbolic, such as the symbols for the action pick up and the object cup and on the other hand the specific sets of 3D pixels and specific hand trajectories and motor settings. The latter ones exist in the reality, they are very exact - they are solid ground. The former are just the names of items or motions and could resolve differently depending on the situation.

In this project, grounding is the same as name resolution. Finding the best object which corresponds to the meaning of the cup, given the whole context of the command gets to be named the cup. Finding the best trajectory of the robot hand that corresponds to the action of pick up, again gets to be named pick up.

This connection, the grounding, bridged the gap between the abstract, clean and imaginary part of the robot’s understanding, the name of each object and trajectory, and the concrete, noisy and complicated physical world. Without crossing this divide somehow, the robot can’t to do anything of physical importance correctly. These physical actions are what made the robot a ROBOT, instead of just a normal computer. The algorithm built a bridge for the robot between the abstract and the concrete, so that the robot can cross it every time it needs to do something of physical importance.

The specific algorithm may have been clever, but what I found to be the really wise part, was the step of connecting the abstract to the specific. Every time I was trying explain something and provided specific examples of my abstract ideas I noticed that not only I was able to communicate these ideas more clearly, I also understood my ideas better. Grounding became a tool for better communicating and a tool for clearer thinking.

The next several essays will dive more deeply into the various aspects and applications of this tool.

You'll receive emails containing the text and images of the essay, once or twice per week. Usually less often.

I write these posts to clarify my thoughts and improve my communication. They are mainly written for myself, and my future self, but I'd love to hear any comments you might have, especially if you disagree with what I've written, or if you've found it valuable in any way. Message me on Twitter @themitak or email me at comments@dimitarsimeonov.com.