Autonomous driving has seen incredible progress of-late. Recent workshops at top conferences in robotics, computer vision, and machine learning have primarily showcased the technological advancements in the field. This workshop provides an platform to investigate and discuss the methods by which progress in autonomous driving is evaluated, benchmarked, and verified.

Workshop Format

Given that the workshop was held virtually, we used a format that is engaging and concise. We held a sequence of four 1.5-hour moderated round-table discussions (including an introduction) centered around 4 themes.

This workshop took place on October 25, 2020 in conjunction with IROS 2020


Theme 1: Assessing progress for the field of autonomous vehicles (AVs)

At present, regulators rely on self-driving car companies to report statistics based on metrics that they use to make decisions about how safe the technology is.

Moderator: Andrea Censi

Invited Panelists:

Theme 2: How to evaluate AV risk from the perspective of real world deployment (public acceptance, insurance, liability, …)?

There is a difference between metrics used by the AV industry for the purpose of development, and the metrics that will be used to evaluate AV risks “externally”, for example for the purpose of obtaining insurance premiums, which are likely going to be standardized and of a black-box nature.

Moderator: Jacopo Tani

Invited Panelists:

Accepted Abstracts:

Theme 3: Best practices for AV benchmarking

An alternative but related concept to creating metrics is to create benchmarks that must be passed. Ideally, not all of the benchmarks would require evaluation on the real hardware platform, but could include the use of logs and simulations.

Moderator: Liam Paull

Invited Panelists:

Accepted Abstracts:

Theme 4: Do we need new paradigms for AV development?

While our focus here is not centrally on the algorithms developed for self-driving cars, the types of algorithms and paradigms used will have an impact on our ability to benchmark and evaluate them. This facet of algorithms is often forsaken for performance.

Moderator: Matt Walter

Invited Panelists:

Wrap Up