Organized by Krishna Murthy Jatavallabhula, Florian Golemo, Dishank Bansal, Rey Reza Wiyatno, Zhen Liu and Liam Paull.

	Speaker	Talk	YouTube link
	Steven Waslander Associate professor University of Toronto	Probabilistic Object Detection for Autonomous Driving Fri, Jan 15, 2021 · 4:00 PM ET Abstract Modern object detection has relentlessly pursued perfection on a single metric: mean average precision, with impressive gains in performance across datasets and sensor types over that last few years. I will discuss our multiple contributions in this domain, particularly in 3D object detection with monocular, stereo and LIDAR/camera fusion. Our work has regularly topped the Kitti Vision Benchmark, and emphasizes the value of attention focused on object shape to enhance localization and extent estimation. Despite the strong progress in this domain, current network outputs are primarily deterministic, providing little visibility beyond class confidence as to the probability a detection is accurate. This makes current object detectors a black box for downstream processes such as tracking and prediction, and can lead to over confidence in low-quality detection. To help address this challenge, I will discuss our recent work on probabilistic object detectors (PODs) on two fronts. First, I will describe efforts to place the evaluation of PODs on a secure footing, by introducing proper scoring rules with both local and global extent that can determine whether a predictive distribution is both well calibrated and discriminative. Then, I will discuss our work BayesOD, a novel probabilistic object detector that exhibits strong output distribution prediction capabilities and outperforms existing PODs in terms of calibration and sharpness.
	Animesh Garg Assistant professor University of Toronto Senior research scientist NVIDIA	Building blocks of Generalizable Autonomy in Robotics Fri, Feb 5, 2021 · 4:00 PM ET Abstract My approach to Generalizable Autonomy posits that interactive learning across families of tasks is essential for discovering efficient representation and inference mechanisms. Arguably, a cognitive concept or a dexterous skill should be reusable across task instances to avoid constant relearning. It is insufficient to learn to “open a door”, and then have to re-learn it for a new door, or even windows & cupboards. Thus, I focus on three key questions: (1) Representational biases for embodied reasoning, (2) Causal Inference in abstract sequential domains, and (3) Interactive Policy Learning under uncertainty. In this talk I will first through example lay bare the need for structured biases in modern RL algorithms in the context of robotics. This will span state, actions, learning mechanisms and network architectures. Secondly, we will talk about discovery of latent causal structure in dynamics for planning. Finally I will demonstrate how large scale data generation combined with insights from structure learning can enable sample efficient algorithms for practical systems. In this talk I will focus mainly on manipulation, but my work has been applied to surgical robotics and legged locomotion as well.
	Sylvia Herbert Assistant professor University of California San Diego	Safe Real-World Autonomy in Uncertain and Unstructured Environments Fri, Feb 12, 2021 · 4:00 PM ET Abstract In this talk I will present my current and future work towards enabling safe real-world autonomy. My core focus is to enable efficient and safe decision-making in complex autonomous systems, while reasoning about uncertainty in real-world environments, including those involving human interactions. These methods draw from control theory, cognitive science, and reinforcement learning, and are backed by both rigorous theory and physical testing on robotic platforms. First I will discuss safety for complex systems in simple environments. Traditional methods for generating safety analyses and safe controllers struggle to handle realistic complex models of autonomous systems, and therefore are stuck with simplistic models that are less accurate. I have developed scalable techniques for theoretically sound safety guarantees that can reduce computation by orders of magnitude for high-dimensional systems, resulting in better safety analyses and paving the way for safety in real-world autonomy. Next I will add in complex environments. Safety analyses depend on pre-defined assumptions that will often be wrong in practice, as real-world systems will inevitably encounter incomplete knowledge of the environment and other agents. Reasoning efficiently and safely in unstructured environments is an area where humans excel compared to current autonomous systems. Inspired by this, I have used models of human decision-making from cognitive science to develop algorithms that allow autonomous systems to navigate quickly and safely, adapt to new information, and reason over the uncertainty inherent in predicting humans and other agents. Combining these techniques brings us closer to the goal of safe real-world autonomy.
	Georgia Chalvatzaki Postdoctoral researcher TU Darmstadt	Robot Learning for Intelligent Robotic Assistants Fri, Feb 19, 2021 · 4:00 PM ET Abstract Societal facts like the increase in life expectancy, the lack of nursing staff, the hectic rhythms of everyday life, and the recent unprecedented situation of the Covid-19 pandemic, make the need for intelligent robotic assistants more urgent than ever. Spanning their applications from home-environments to hospitals, workhouses to agricultural development, etc., the embodied AI robotic assistants are in the epicenter of modern robotics and AI research. In this talk, I will go through my research work for developing intelligent methods for such assistive agents. We will draw the big picture of intelligent service robots, and we will specifically focus on sub-problems that I have tackled in the last few years. The main research areas we will cover consider: the perception and recognition of human activities, combining classical methods like tracking with machine learning for extracting useful multi-sensor human-related information for robot action planning and control; algorithms for encoding object-features that allow 6D tracking and grasp-planning; we will discuss methods that can leverage human-centered information for learning intelligent robot behavior using reinforcement learning; and, I will elaborate on our recent work about accelerated policy learning of manipulation tasks both through the effective combination of imitation and reinforcement learning, but also through a novel method for model-predictive policy optimization. While these topics cover only partial aspects of the bigger problem, we will discuss the open research questions on the combination of learning, reasoning, and planning in unstructured environments using mobile manipulator robots. Mobile manipulators are the most emblematic systems to encapsulate the benefits of embodied AI research towards achieving the long-term vision of developing intelligent robotic assistants.
	Deepak Pathak Assistant professor Robotics Institute, CMU	Robots that learn to generalize beyond training Fri, Mar 5, 2021 · 4:00 PM ET Abstract Generalization, i.e., the ability to adapt to novel scenarios, is the hallmark of human intelligence. While we have systems that excel at cleaning floors, playing complex games, and occasionally beating humans, they are incredibly specific in that they only perform the tasks they are trained for and are miserable at generalization. One of the fundamental reasons is that, unlike humans, most of these artificial agents start tabula-rasa without any prior knowledge and learn only towards a fixed goal. In this talk, I will present our initial efforts towards building a framework for learning general-purpose embodied intelligence. The framework brings together ideas from four areas: machine learning, control theory, computer vision, and developmental psychology to achieve end-to-end sensorimotor learning in embodied agents. I will present results from case studies of robots that manipulate deformable objects, navigate in office environments, perform dynamic manipulation tasks like writing digits on white-board, scooping etc., display drastically diverse locomotion styles across unseen robot shapes, and perceive the real-world in 3D from just a single 2D image.
	Pulkit Agrawal Assistant professor MIT	Self-Supervised Robot Learning Fri, Mar 12, 2021 · 12:00 PM ET Abstract A grand goal of robot learning is to build systems that can perform "common-sense" tasks. Achieving this requires circumventing many roadblocks: reward definition and measurement, data inefficiency, dexterity, and long-term planning to name a few. We believe that by analyzing these challenges at all levels of the stack: hardware, control, and perception, we can simplify the problem and find solutions that are not obvious by looking at only a single level. At the center of our approach is the learning methodology of robots conducting experiments to collect data, which is then synthesized into models of how the world works. These models in turn inform what the robot explores next. The iterative cycle of exploration and model-building is further guided by social learning and forms a never-ending learning loop. Through several case studies spanning manipulation, locomotion, and navigation, I will expand on our attempts to overcome the central challenges in robot learning.
	Lilian Weng Research scientist OpenAI	The power of diverse training environments for solving robotic manipulation tasks Fri, Mar 19, 2021 · 12:00 PM ET Abstract I'm gonna talk about two robotic manipulation projects we have done at the OpenAI Robotics team. In the project of solving Rubik's cube with a single human-like robot hand, we trained a deep reinforcement learning control policy 100% in simulation and deployed it on the real physical robot to solve a fully scrambled Rubik's cube. The policy was trained with automatic domain randomization (ADR) and exhibited vastly improved sim2real transfer capability. In the other project, we trained a single goal-conditioned policy to solve various rearrange tasks on the tabletop setting. We relied on asymmetric self-play to generate a diverse distribution of training tasks and at test time this policy was able to generalize to many unseen tasks such as setting a table, stacking blocks, and solving simple puzzles. We believe a memory-augmented model trained on an ever-growing distribution of training environment, either generated by ADR or self-play, can do meta-learning at test time and achieve amazing generalizability in environments or tasks that are not seen during training.
	Kelsey Allen Research scientist DeepMind	The power of structured action and dynamics models Fri, Mar 26, 2021 · 12:00 PM ET Abstract The world is structured in countless ways. When cognitive and machine models respect these structures, by factorizing their modules and parameters, they can achieve remarkable accuracy and generalization. In this talk, I will discuss our work investigating the factorizations of objects, relations, and modes in both humans and machines. Focusing on problems in physical problem-solving including construction and tool use, I will show how to harness object and relational structure in the form of graph networks and probabilistic programs to improve machine generalization, and how to harness dynamic modes represented as constraints to enable a robot to plan with tools. To go from harnessing structure to discovering it, I will then talk about some initial steps examining how to learn constraints and dynamic modes from relatively little experience. By taking better advantage of problem structure, and combining it with general-purpose methods for statistical learning, we can develop more robust and data-efficient machine agents, and better explain how natural intelligence learns so much from so little.
	Manolis Savva Assistant professor Simon Fraser University	3D simulation for embodied AI - Three emerging directions Fri, Apr 9, 2021 · 12:00 PM ET Abstract 3D simulators are increasingly being used to develop and evaluate "embodied AI" (agents perceiving and acting in realistic environments). Much of the prior work in this space has treated simulation platforms as "black boxes" within which learning algorithms are to be deployed. However, the design choices and resulting system characteristics of the simulation platforms themselves can greatly impact both the feasibility and the outcomes of experiments involving simulation. In this talk, I will describe a number of recent projects that outline three emerging directions for 3D simulation platforms: scalability, predictivity, and interactivity.
	Jiajun Wu Assistant professor Stanford University	Learning to see the physical world Fri, Apr 16, 2021 · 1:00 PM ET Abstract Human intelligence is beyond pattern recognition. From a single image, we're able to explain what we see, reconstruct the scene in 3D, predict what's going to happen, and plan our actions accordingly. In this talk, I will present our recent work on physical scene understanding---building versatile, data-efficient, and generalizable machines that learn to see, reason about, and interact with the physical world. The core idea is to exploit the generic, causal structure behind the world, including knowledge from computer graphics, physics, and language, in the form of approximate simulation engines, and to integrate them with deep learning. Here, deep learning plays two major roles: first, it learns to invert simulation engines for efficient inference; second, it learns to augment simulation engines for constructing powerful forward models. I'll focus on a few topics to demonstrate this idea: building scene representation for both object geometry and physics; learning expressive dynamics models for planning and control; perception and reasoning beyond vision.