Synthetic Data: The future of ADAS for EV’s AI models development
In the ever-evolving landscape of electric vehicle technology, the majority of EV’s on the market already have some level of Advanced Driver Assistance Systems (ADAS) or autonomous driving (AD) capabilities. These systems function by using a combination of exterior and interior cameras, sensors, and vision processing software to analyse their surroundings and adequately respond. The goal of these assistance and autonomous systems is to create safer, smarter, and more efficient roads.
To delve into this transformative journey, it’s essential first to understand the levels of autonomy that vehicles are progressing through, known as the SAE levels, ranging from L0 (basic driver assistance) to L5 (full automation).
Tesla is an example of this paradigm, it has become synonymous with advanced automotive technology, particularly in the ADAS and autonomous driving space. Tesla’s Autopilot combines adaptive cruise control and lane-keeping systems to provide L1 and L2 automation. Moreover, Tesla’s Full Self-Driving (FSD) package takes a step closer to L3 autonomy, with features like Navigate on Autopilot, which can handle highway on-ramps and off-ramps.
On the other end of the autonomy spectrum, robotaxi companies like Waymo are pushing the boundaries of L4 autonomy. Waymo has deployed a fleet of fully autonomous vehicles in select cities for ride-sharing services. These vehicles can operate in defined geographic areas, providing rides to passengers without human drivers.
The success of companies like Waymo underscores the potential of autonomous vehicles in transforming urban mobility. However, reaching L4 autonomy required extensive training, testing, and validation of the perception systems that enable these vehicles to navigate complex environments safely.
EV’s and perception systems
The higher the level of autonomy EV’s want to achieve, the more sophisticated their perception systems need to be, and the better their understanding of the real world needs to be. First, they need different sources of information through different sensors such as cameras, LiDAR, and radar. In turn, the brains behind every ADAS and Autonomous Driving (AD) system supporting the different levels of autonomy, are Machine Learning (ML) models that consume this information, this data coming from the sensors, to solve particular computer vision problems. Object detection and recognition, road line tracking, distance estimation, and driver behaviour are some of those problems that, once solved, help EV’s understand the world around them to make the right decision in every situation.
The key for the success of these perception systems is the ML model’s performance in solving the different problems. The goal during development is to have the highest performance when validating and testing the system in real-world scenarios. There are several metrics to measure the performance of ML models, the most widely used for the type of problems for automotive (classification) are precision and recall that focus on minimizing false positives and false negatives respectively. Now, what knobs do machine learning engineers have to improve model performance? DATA. The data used in the process of model hyper-parameter tuning.
This is an iterative process using a dataset split in training, validation, and test sets. Put simply, it may look like this:
● First, you train the model with the training set and validate it with the validation set
● Iterate until you find a set of model hyper-parameters that gives you the best model’s performance on the validation set (based on the metrics you decided, precision and recall, for example). Train the model using both, training and validation sets
● Measure performance using the test set (data that the model has not seen during the process)
● If you are getting the performance your system needs you can deploy your system
● If not, you can check if the model is over-fitting or is struggling to perform with some specific case. You need to analyse your dataset and find gaps that you may need to fill
Thus the data you use is the key to the process and can make your model perform to levels that are acceptable for your systems or not. It’s important to remember that for EV’s ADAS or AD systems a poor performance can have fatal implications.
Let’s talk about data
What is the data you need to train, validate, and test EV’s ADAS and AD systems? Where do you get this precious data needed to get the best performance out of your system? The most obvious answer is to get it from the real world. In fact, that is what most people do: rig a car with all the sensors that will capture the data and start driving it around saving everything the sensors see.
After thousands of miles of capturing data, now you have to curate that data and annotate it. ML models ADAS and AD perception systems need huge amounts of images or video, and they have to be varied and annotated with descriptions of what there is in those images, so during training and validation the ML models “know” what they are seeing. With this information they can learn to solve a classification problem, for example. Annotations or ground truth data is not captured by the sensors, it is created by humans or applications that understand the images. Typical ground truth data for a classification problem are bounding boxes, they mean “this specific area in the image is a car”. Depending on the problem the ground truth data you need is different, a perception system estimating distances needs to know the distance of the objects it sees during training, so bounding boxes are not enough.
All this capturing, curating, and annotating process is very costly and time-consuming. In fact, if you have to iterate the hyper-parameter tuning and you need more data, you have to do it all over again. This can seriously cripple your development process.
Let’s talk about synthetic data
What if you can generate all the data you want automatically based on a set of specifications, fully annotated with all the ground truth you need for your specific perception problem? That is exactly what synthetic data can provide. It is the promise of generating data that simulates real-world scenarios and sensors, allowing the perception system’s ML models to learn and adapt faster. This accelerates the development and validation of ADAS and autonomous systems, making them safer and more reliable.
Synthetic data creation is a combination of 3D computer graphics with physically based simulation technologies providing a new field for data scientists to create exactly the data they need when they need it, with pixel-perfect ground truth data. However, not all is as perfect as it sounds. How do synthetic data trained and validated perception systems perform when they face real-world data?
This depends on the nature of the synthetic data technologies used and how they are in reducing the so-called “domain gap” for the ML perception systems. The closer the synthetic data is to the real-world images the system will face in production the smaller the domain gap and the better the performance. To achieve this it is key to have a physically based simulation of the camera sensors when producing the data and this can only be achieved if the 3D render an accurate simulation of light transport and produces hyper-spectral information of the scene it is capturing. This hyper-spectral information is the amount of energy reaching every pixel of the camera sensor for many different wavelengths. With it, you can simulate the sensor to produce a raw image that is later processed to produce a final colour image, very close to what a real sensor would capture. Not all 3D renders out there have this capability, but it is important to consider if you are experiencing the domain gap in your models.
In any case, it is always the data scientists’ and ML engineers’ responsibility to decide what is best for their particular perception system, even a combination of real and synthetic data has been proven successful in improving model performance. It’s your call.
Javier Salado – Technical Product Manager – Anyverse
