Openpilot

The goal of this project is to implement open-source data collection and training pipeline for comma.ai Openpilot. This involves:

• implementing custom comma.ai API for data collection,
• creating high-precision GNSS trajectories with Laika,
• implementing neural network training,
• integrating the trained network into Openpilot.

Organization

The students will pick roles in the team:

• data engineer - data collection and preprocessing trajectories
• web engineer - comma.ai API, Explorer, Cabana
• NN engineer - neural network training, Qualcomm SDK for neural network inference
• device engineer - everything that happens in the device

If there are not enough people, data and web engineer roles can be skipped. In this case the training will be performed on existing comma2k19 dataset.

Schedule

Date	Topic	Presenter	Materials
8.09.2021	Openpilot intro	Tambet	Towards a superhuman driving agent
15.09.2021	How does Openpilot work?	Rustam (video)	How does openpilot work? A Tour Through openpilot How Do We Control The Car? Building a Super Human Driving Agent
skipped	CAN bus and DBC	???	Opendbc How to write a car port for openpilot Openpilot port guide for Toyota models Toyota interface used for Lexus
22.09.2021	Explorer and Cabana	Mike	Explorer Web Explorer GitHub Cabana Web Cabana GitHub
29.09.2021	Openpilot data format	Nikita (slides)	Replay driving data fetch_image_from_route.py Open Sourcing openpilot Development Tools Code for training on Openpilot data
6.10.2021	comma.ai API	Mike	comma API spec
13.10.2021	(skipped)
20.10.2021	GNSS postprocessing for high precision trajectories	Nikita (video)	comma2k19 Laika Rednose Reference Frames in Openpilot
27.10.2021	Openpilot driving model	Gautam	Decoding comma.ai/openpilot End-to-end lateral planning Reverse engineer openpilot Extracted standalone Openpilot model Code for training on Openpilot data Building a Super Human Driving Agent
3.11.2021	Running Openpilot against Carla	Rustam	Run openpilot in the driving simulator CARLA Openpilot in simulator
10.11.2021	Qualcomm Neural Processing SDK	Gautam	Decoding comma.ai/openpilot Qualcomm Neural Processing SDK for AI
17.11.2021	Joystick steering test	???	Debug car controls Open Sourcing openpilot Development Tools
24.11.2021	Running Openpilot from laptop with webcam	???	Self-Driving Car For Free Run openpilot with webcam on PC/laptop
1.12.2021
8.12.2021
15.12.2021
22.12.2021	Project demo

Links

Other useful sources of information:

• Openpilot wiki
• comma.ai blog
• comma.ai Discord
• comma.ai YouTube channel, including COMMA_CON presentations
• George Hotz YouTube channel

Deliverables

• Each student should present one topic to others. The presentation should include live demo.
• All students together should train one neural network model and demonstrate it driving in comma two device. The driving does not have to be perfect. Either self-collected data or comma2k19 dataset can be used for training.

Contact

Autonomous Driving Lab (room 3095)
Tambet Matiisen
tambet.matiisen@ut.ee