Jiayuan Mao
MIT
jiayuanm@mit.edu
Bio
Jiayuan Mao is a Ph.D. student at MIT, advised by Professors Josh Tenenbaum and Leslie Kaelbling. Her research agenda is to build machines that can continually learn concepts (e.g., properties, relations, rules, and skills) from their experiences and apply them for reasoning and planning in the physical world. Her research topics include visual reasoning, robotic manipulation, scene and activity understanding, and language acquisition. Her work is supported by the MIT presidential fellowship.
Areas of Research
- Artificial Intelligence
Learning, Reasoning and Planning with Neuro-Symbolic Concepts
I want to build machines that can continually learn new knowledge from their experiences and apply this knowledge to their reasoning in the physical world across different tasks, modalities, and environments. The desired capabilities of such agents include, but are not limited to, describing perceived scenes, answering queries about scenes, making plans to achieve certain goals, and executing plans in the physical world, all while learning from multiple modalities. The key idea is to learn a vocabulary of neuro-symbolic concepts and apply them compositionally. As understood by philosophers and cognitive scientists, concepts are the basic building unit of thought — humans acquire them from past experiences and build relations among them to form sophisticated thoughts. One of the principled ways to construct a set of useful concepts for reasoning is to derive them from word meanings in language. In particular, my work focuses on the granularity of concepts at the level of word meanings (meanings of nouns, verbs, etc.). These concepts are represented as a combination of symbolic programs, to capture how they can be structurally combined similarly to the way words combine into sentences in human languages, and modular neural networks, to ground the abstract names of the concepts onto 2D/3D visual features and robotic actuation commands.