GOLD

a complete framework for developing artificial vision applications for intelligent vehicles

M. Bertozzi, L. Bombini, A. Broggi , P. Cerri, A. Fascioli, P. Grisleri, P. Medici and P. Zani

VisLab — – Università di Parma, ITALY

The developing real time vision applications to be used on highly dynamic environments, like the automotive one, researchers need to gather large amounts of data coming from different sensors, different systems, end at a different rate.

A software able to perform data synchronization, data logging and data processing and visualization in real time is of basic importance, to increment the researcher efficiency.

The GOLD framework, thanks to hits support different devices and for the simplicity of adding new functionalities, is an efficient software layer that can easily become the engine for many automotive applications.

The GOLD framework

The use of vision for Intelligent Transportation Systems developed in the last 20 years and, after the first experiments, rapidly evolved and is now widely considered as one of the most convenient perception technologies.

In order to effectively develop vision based systems for intelligent vehicles, both hardware and software issues have to be considered.

The hardware side affects both the sensing technologies -namely the cameras, in case of vision- and the computing engine. Different camera technologies can be exploited: daylight, far infrared (thermal), near infrared, or even range cameras; the choice mainly depends on the specific application and on a cost versus benefit analysis.

On the software side, two different issues have to be examined: the software for the off-line development and debugging phase and the software for the final system. In order to enable programmers to focus on the specific vision problem without having to care about other details, a rapid application development tool needs to be used. In the last 10 years, VisLab developed a tool named GOLD [1], which provides programmers with a rich set of functionalities and APIs for commonly required tasks like I/O operations (acquisition from cameras and other sensors as well, sensors synchronization, network and file management, data graphical display,…) and low-level image processing functions. A convenient graphical user interface is also used to support programming activities.

The name GOLD is the acronym of General Obstacle and Lane Detection, since when it was first conceived, it was used as the main engine to detect lanes and obstacles on board of the ARGO vehicle [2]. After fifteen years of continuous research and development efforts, now GOLD has evolved into a framework aimed at providing a complete set of tools to allow fast development of computer vision applications. Moreover, GOLD is also used to run the on-board final system. In spite of the complexity of this framework, GOLD has a user-friendly GUI (Fig.1.a) that permits to manage all the applications. It is possible to visualize the data coming from different sensors, such as a laserscanner (Fig. 1.b) or stereo camera system, and show the intermediate (Fig. 1.c,d) and final processing results (Fig. 1.e).

Fig. 1. The graphical user interface of GOLD (a), laser scanner data (b),

top view of the area in front of vehicle (c) showing obstacles and lane markings, laser data mapped on the image coming from a camera (d), and the final results of the application in real-time (e).

Applications

GOLD allows to build vision-based applications as independent plug-ins. Each application can take advantage from an increasingly rich set of tools and debugging facilities. These components are shown in Fig. 2 and described in the following.

The rationale behind GOLD is to free the developer from the burden of performing low-level hardware management, offering an abstraction layer over real devices. Various classes of devices are currently supported, such as several camera types (analog and digital), laserscanners, radars,vehicular CAN data, and inertial systems. Data can be acquired from multiple sensors with or without synchronization. This feature allows to easily develop data-fusion oriented algorithms. Moreover, virtual devices are also implemented to access pre-recorded data. GOLD also allows to preprocess acquired data performing common operations (such as image stabilization or distortion removal) and sharing the results among all the active plug-in applications.

A graphical user interface layer has been developed to provide graphic widgets and controls for interacting with applications allowing programmers to forget about available graphic backends. A rich set of APIs allow programmers to output final or partial results, producing graphics with very low resource consumption, thanks to its multithreaded client-server architecture.

Developers may use GOLD to record data sequences from multiple streams in order to allow a later playback for in-lab development. Captured data can be stored in different formats, depending on their nature and on the application constraints. GOLD allows advanced playback capabilities (such as looping, stepping, jumping, moving forward or backwards at a given frame rate and setting bookmarks), which guarantee high flexibility for the development and testing of algorithms since it has been engineered by the very same team that developed applications at VisLab in the last ten years. During playback, acquired data are fed to the application through the same APIs used for acquiring from real devices. Recorded data comes with an index file with a timestamp for each event, which is used to cope with problems involving time, such as tracking and synchronization, allowing a time based replay mode.

In this case, playback is performed considering the actual processing time and therefore emulating the system behavior in real working conditions.

GOLD architecture

GOLD architecture has been conceived to ensure scalability and independence among its components. The whole system is organized into several subsystem layers, which provide the different functionalities to the developers. Thanks to a cleanly defined interface, subsystems can be easily added or removed depending on the specific needs of the target project. All applications are seen as plug-ins by the framework, and can be easily removed or independently connected and ported to different platforms.

The hardware abstraction layers masks the complexity of input devices to the programmer.

Fig. 2. The architecture of GOLD.

Developers can also easily produce highly interactive and consistent user interfaces, through the use of a set of widgets specifically designed to seamlessly integrate with the processing code. A windowing subsystem provides a set of simple and powerful drawing primitives that can be used to display intermediate and final results during the debugging phases, or as the final output.

Examples of vision systems based on GOLD

GOLD has been used to power many system developed at VisLab in the last decade, like pedestrian detectors, using both visible (US Army Tacom, US) and FIR and NIR images (VW, Germany), traffic signs recognition (Marelli, Italy), Vehicle Detection (Fiat, Italy), Short range Obstacle Decettors (Volvo Tech, Sweden).

In the following, two different examples are given: the DARPA Grand Challenge and Urban Challenge sensing architecture and a surveillance system for boats speed estimation.

DARPA Grand & Urban Challenges

VisLab is now developing the artificial vision systems to sense the environment for the 2007 DARPA Urban Challenge. VisLab is working together with Oshkosh Truck Corp, Teledyne, Auburn Univ, and IBEO.

The vision software is spread over 4 high performance dual core PCs, and it is used to sense the environment thanks to 11 cameras, providing a wide range of information: drivable path, lane position, far frontal obstacle, close rear obstacles [3], close frontal and rear obstacles [4], overtaking vehicles and traffic at junctions.

In such complex scenarios, the design efforts are totally focused on reaching the needed levels of accuracy, robustness, speed, and completeness. Issues regarding power consumption, hardware size, etc. are not taken into account. The PCs were chosen among the most powerful ones. Each one is hosting a full-featured copy of GOLD for both acquisition and processing.

Boat Speed Monitor Vision system

In the city of Venice, a lot of watercrafts pass too fast through water channels, generating waves that erode seacoasts. The preservation of this coast is really important in order to save an ecosystem that is typical of Venice.

For such a problem, a system that can evaluate the boat speed and display it with the intent to deter excessive speeds has been developed.

A high resolution camera mounted on the top of a 10 m pole is used to monitor a large area up to a distance of 200 m. The camera is connected to an off-the-shelf PC together with a large led panel used to show the boat speed. The system stores all the data of the identified watercrafts (speed, date, time) in a log file and can act as a web server, providing information in real time.

Fig. 3. The boat system output.

For this application only part of the GOLD system is needed: low level vision filters, the hardware abstraction layer, and real time management. The system has been designed using GOLD, with the ultimate goal of porting the whole processing on a smart camera powered by an ARM processor and a Linux embedded operating system.

Conclusion

The GOLD platform not only allows rapid application development and debugging, but also the final fielding of complex vehicular systems; it also allows to build applications as independent plug-ins, and therefore applications can be extracted and ported towards any architecture.

Moreover, facilities to add support for new devices and new algorithm libraries are provided to easily expand the framework. Similar projects [5], [6] do not have this kind of flexibility, they are usually devoted to support only one phase of the product development.

After fifteen years of software development, GOLD represents a fundamental instrument for vision algorithms development, and it is the key to VisLab success in the international research arena.

References

1. M. Bertozzi and A. Broggi. GOLD: a Parallel Real-Time Stereo Vision System for Generic Obstacle and Lane Detection. IEEE Trans. on Image Processing, 7(1):62–81, Jan. 1998.

2. A. Broggi, M. Bertozzi, A. Fascioli, G. Conte, "Automatic vehicle guidance: Experience of the ARGO autonomous vehicle", World Scientific, 1999.

3. A. Broggi, C. Caraffi, P. P. Porta, and P. Zani. A Single Frame Stereo Vision System for Reliable Obstacle Detection during Darpa Grand Challenge 2005. In Procs. IEEE Intl. Conf. on Intelligent Transportation Systems 2006, pages 745– 752, Toronto, Canada, Sept. 2006.

4. M. Bertozzi, A. Broggi, P. Medici, P. P. Porta, and A. Sj¨ogren. Stereo Vision-Based Start-Inhibit for Heavy Goods Vehicles. In Procs. IEEE Intelligent Vehicles Symposium 2006, pages 350–355, Tokyo, Japan, June 2006.

5.The MOOS,Simple, open-source, cross-platform software for mobile robotics research , Robotics Research Group Department of Engineering Science, Oxford, United Kingdom.

6. MIDAS 4.0 Video and Data Capture, , Excitex.