Embodied AI - Fundamentals and Applications

This course aims at introducing the fundamentals in algorithms, data and systems of the autonomous intelligent systems, which often refers to the embodied and robotics applications. As the fast advances in the field of AI, how to utilize the learning-based, data-driven approaches to improve the applications for the better human life, becomes very pivotal. We will address the key challenges in this domain, such as:

• (i) How to formulate a system that is equipped with generalization, intelligence and reliability merits?
• (ii) How to balance the data distribution between simulation and real-world data?
• (iii) Is scaling law the only pathway towards high-level AGI?

We will introduce the concepts, principles and knowhow to build the embodied intelligent systems. The basic fundamentals would be detailed in the lectures, with tutorials and hands-on training sessions. All the important topics will be covered, such as imitation learning, reinforcement learning, and so on. The highlights in this course would consist of several guest lectures from outside renowned speakers from both industry and academia to address the latest advances in this field. The hands-on session is akin to tutorials or hackathons where students learn the recipe of technologies from scratch quickly. These features would be complementary to the main lecture and facilitate the final group presentation.

The course includes 13 lectures (by instructor or guest lecture) and 4 tutorial/hands-on session (by TA). Homework includes both written exercises and programming.

The basic programming skill is needed, e.g. python and C++. We require students to have prior knowledge in undergraduate linear algebra, statistics, and probability. Background in machine learning, and computer vision may allow you to better appreciate certain aspects of the course material, but not necessarily all at once.

The course is open to research postgraduates (Rpg, MPhil, PhD). If you're curious about whether you would benefit from this course, contact the instructor for details.

We do not have a fixed textbook. The following resources are for references.

Textbook:

• Foundations of Computer Vision, by Antonio Torralba et al.
• Understanding Deep Learning, by Simon Prince

Online courses (some contents in our course are by courtesy of them):

• Introduction to Robot Learning, by Guanya Shi
• Deep Learning for Robotics, by Deepak Pathak

Conferences (where you get the latest advances in this field):

• Robotics: Science and Systems (RSS)
• Conference on Robot Learning (CoRL)
• IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
• Annual Conference on Neural Information Processing Systems (NeurIPS)

• Course website: https://opendrivelab.com/AP0001
  • We in general will use the Piazza as well to post lecture materials, homework, data, code examples, etc.
• Computing tools: We will use the python and (a little bit) C++, and Google Colab environments for many of the in-lecture demos, examples and homework. Hugging Face for hosting the dataset would be recommended. Cloud services would be introduced to run some GPU experiments.