Adverse Weather Conditions in the Validation of ADAS/AD Sensors

Armin Engstle

 

Introduction

The sensor technology of future automated driving functions and advanced driver assistance systems must work safely in all weather conditions. Currently, certification tests like e.g. Euro NCAP AEB VRU are only performed in good weather conditions, but not in rain and fog, for example. However, a significant proportion of serious and fatal accidents, especially with VRUs, occur in the context of bad weather and/or poor visibility.

In order to be able to test weather effects independently of real outdoor conditions, an indoor sensor centre for scenario based, reproduceable sensor testing and validation in adverse weather conditions is required. The following article describes the basic characterisation of natural rain as well as fog and presents an example for an indoor sensor centre that artificially generates realistic weather conditions and provides the advantage of reproducibility.

Rain Characterization

Raindrop diameters range between 0.1 and 9 mm, and the terminal velocities vary from 0.1 to greater than 9 m/s. Typically, a volume of 1 cubic meter of air contains about 10³ raindrops with less large drops than smaller ones. Small drops are close to spherical, whereas larger ones become oblate spheroidal [1].

The three most important parameters characterizing natural rain are rain intensity measured in mm/h, drop size distribution (DSD) representing the total number of drops in a unit volume as a function of drop diameter and terminal velocity of rain drops as a function of drop diameter. Obviously, a specific rain intensity can be generated by multiple DSDs whereas a given DSD results in a biunique rain intensity.

Artificially generated rain intensity and drop size distribution are mainly influenced by the following three parameters:

1. selected nozzle type,
2. the pressure in the hydraulic system and
3. the spatial distribution of the nozzle outlets additionally influencing the homogeneity of the rain in the sprinkled area.

Terminal velocity of raindrops is mainly influenced by the installation height of the rain facility. This correlation has been analysed precisely in [2] the results of this analysis are displayed in Figure 1.

Sketching falling distance needed over the droplet diameter it can clearly be stated that with an installation height of the rain facility of 6 meter more than 90% of all drops reach the terminal velocity measured in natural rain.

It has to be mentioned, that the portion of raindrops reaching their terminal velocity is even higher in a hydraulic installation as the pressure presses the droplets out of the nozzle and the resulting starting velocity of the droplets is unequal zero. As this effect however is a function of pressure it is difficult to further quantify it.

Fog Characterization

Fog is present when water droplets suspended in the air decrease the visibility to less than 1 km. Generally, a volume of 1 cubic meter of air contains more than 10⁶ drops, which is several orders of magnitude greater than in rain. The diameter of fog droplets typically ranges from 5 to 20 μm and depends on fog type [1].

The most important parameter characterizing natural fog is the DSD similar to natural rain. Obviously, this simplified correlation neglects the specific DSDs of fog and its influence on the undefined concept “visibility.” As elaborated earlier on rain however the visibility range not only depends on the rain rate measured in mm/h but strongly depends on the specific DSD of the available rain. The same logic applies to fog meaning a rough correlation between fog density and visibility can be stated but for precise sensor testing and validation the analysis has to be carried out on a more detailed level.

Sensor Test Bench

Based on the analysis of natural rain and fog requirements for an indoor sensor testbench have been derived. Major design criterion being:

• Highest degree of reproducibility: the repetition of artificially generated weather always yields the same results (e.g., rain intensity and DSD)
• High spatial homogeneity of rain and fog: independent of the measurement point in the weather hall, weather characteristics are the same everywhere (e.g., rain intensity and DSD)
• Sophisticated control system: changing rain intensity can work within minutes
• Environmental protection: e.g., rainwater can be recycled as much as possible.

The result of this design process can be seen below: – Site-Overview Sensor Testbench

On an area of 84 m x 19 m the sensor test bench provides an asphalt road surface with defined reflectivity and standard road markings. For water recycling purpose as well as continuous operation capability two drainage lines are incorporated into the road surface that collect the water and return the water to the hydrological cycle of the indoor facility. 560 nozzles generate homogeneous rain on an area of approximately 1000 qm with an adjustable, reproduceable rain intensity ranging from 15 mm/h to 115 mm/h and realistic drop size distributions. A high-pressure fog machine consisting of 173 nozzles that are arranged in the same area as the rain facility produces fog particles of 5 – 20 µm. For adapting the illuminance during the rain and fog experiments in the weather hall, a light facility featuring 60 LED spotlights is installed in the hall. In order to provide indirect light to the hall, the spotlights are directed to the ceiling where the light is reflected by a Lambertian reflector. With this unrealistic light reflections from LED-bands on the wet ground are avoided.

Take-over maneuver of a semitrailer truck by a passenger car during intensive rain

Testing Approaches

For testing and validation of sensors in artificially generated adverse weather in an indoor facility several approaches with different targets of analysis are possible. In general we can distinguish between

1. Single sensor tests (camera, radar, lidar) on raw data level
2. Sensor or sensor system tests on perception level and
3. Vehicle tests, where the sensor(s) is / are geometrically integrated in the vehicle and the vehicle driving through the weather respectively the hall.

Obviously, the target objects differ between the three approaches.

1. For raw data level tests typically reflectivity boards are being used for lidar, corner reflectors for radar and MTF targets or colorchecker for camera. Variation parameters are mainly the distance between sensor and target object, rain intensity respectively visibility range for fog and corresponding illuminance and colour temperature of light.
2. For sensor or sensor system tests, the set-up is still static, but the target objects are usually typical dynamic (e.g. pedestrians, cars, etc.) or static (e.g. traffic sign, traffic light, etc.) objects appearing in real world traffic. The question to be answered in this test setup is clearly more binary: Can the sensor detect, classify and position the respective target object still at distance of e.g. 40 m when rain intensity is 50 mm/h and illuminance is 20 lux at a colour temperature of 4500 K?
3. Finally vehicle tests include the complete vehicle reaction on a target object or scenario like Euro NCAP. It has to be noted, that due to safety reasons the vehicle velocity in the sensor hall is limited to 50 kph, however this velocity serves for the majority of urban scenarios.

Summary

Scenario-based, reproduceable testing and validation of sensors in adverse weather conditions is indispensable for the development of robust and reliable ADAS / AD functionalities. The EURO NCAP Vison 2030: “a safer future for mobility” [5] clearly addresses this request as well as the SOTIF standard released in 2022.

The AVL sensor center in Roding provides a unique testing environment for adverse weather conditions. The reproducibility of rain, fog and light does not only allow the repetition of experiments but also enables the precise development of virtual sensor models. By that the test depth can be increased significantly. As especially perception functions are based on neural networks and therefore are non-deterministic by definition, a high number of test-variations is absolutely core for making these functions safe, robust and reliable. Dr. Georg Schwab, Managing Director of AVL Software and Functions GmbH, commented: “For the introduction and technical optimization of autonomous systems, it is essential to reproducibly test them under different environmental conditions. These tests must be carried out in an efficient manner and combined with innovative simulation methods. Both will be available at AVL in Roding

Armin Engstle Site Manager AVL Roding: Center for Mobility and Sensor testing