NeurIPS 2021 Lead Author Spotlight
Andrew Szot, PhD Machine Learning student
We introduce Habitat 2.0 (H2.0), a simulation platform for training virtual robots in interactive 3D environments and complex physics-enabled scenarios. We make comprehensive contributions to all levels of the embodied AI stack – data, simulation, and benchmark tasks. Specifically, we present: (i) ReplicaCAD: an artist-authored, annotated, reconfigurable 3D dataset of apartments (matching real spaces) with articulated objects (e.g. cabinets and drawers that can open/close); (ii) H2.0: a high-performance physics-enabled 3D simulator with speeds exceeding 25,000 simulation steps per second (850x real-time) on an 8-GPU node, representing 100x speed-ups over prior work; and, (iii) Home Assistant Benchmark (HAB): a suite of common tasks for assistive robots (tidy the house, stock groceries, set the table) that test a range of mobile manipulation capabilities. These large-scale engineering contributions allow us to systematically compare deep reinforcement learning (RL) at scale and classical sense-plan-act (SPA) pipelines in long-horizon structured tasks, with an emphasis on generalization to new objects, receptacles, and layouts. We find that (1) flat RL policies struggle on HAB compared to hierarchical ones; (2) a hierarchy with independent skills suffers from ‘hand-off problems’, and (3) SPA pipelines are more brittle than RL policies.
Q&A with Andrew Szot
(click question to show answer)
What motivated your work on this paper?
I was motivated to work on this paper to accelerate embodied AI towards being able to complete realistic tasks in the home such as cooking meals, loading the dishwasher, or cleaning up. Previously, working towards this goal was challenging due to the lack of datasets, slow simulators, and missing benchmarks to measure progress. Motivated by these three deficiencies, our work proposed a dataset of interactive house-scale scenes, a simulator that is 100x faster than prior work, and a benchmark for realistic tasks in the home.
If readers remember one takeaway from the paper, what should it be and why?
One takeaway is that Habitat 2.0 is the perfect test bed for developing agents in interactive, 3D, and physics-enabled tasks in the home. Also that researchers should get started using the dataset, simulator, and benchmark at https://aihabitat.org.
Were there any “aha” moments or lessons that you’ll use to inform your future work?
A part of the project was benchmarking how prior approaches performed on long and complex tasks in the home such as preparing groceries or setting the table. An “aha” moment was realizing that all previous methods achieved zero success on the hardest version of the benchmark. This finding informs my future research in learning algorithms that address such long and compositional task structures.
What are you most excited for at NeurIPS and what do you hope to take away from the experience?
I am excited to meet other researchers in the space. I hope to take away new connections from the experience.