Human-Machine Interaction: HBKU’s Smart Vision Sensor Aims for Next-Level Intelligence

Sight is one of the five human senses but today, technology has evolved so that artificial intelligence (AI)-enabled sight is also essential to robotics, optics, machine vision systems, and automation.

Several industries are taking advantage of machine vision systems, including self-driving vehicles, industrial automation such as food packaging and beverage bottling; semiconductor assemblies; and pharmaceutical companies, for automation tasks such as assisted driving, counting, quality control, and the detection of defects.

At the College of Science and Engineering (CSE), part of Hamad Bin Khalifa University (HBKU), the Sensor Research Group brings together world-class technology experts under the leadership of Dr. Amine Bermak, Professor and Associate Dean of CSE. The research group is unique to the region and is putting efforts into harnessing new technological capabilities to enable the next level of intelligence in human-machine interaction. For instance, the ongoing pandemic saw the group advance the work on their prototype “smart mask”, which is designed to be worn like today’s commonly used medical face masks. A major area of their work is vision sensors and Dr. Bermak is currently leading a project to design a smart vision sensor. 

What is a vision sensor?

Vision sensors, sometimes known as “machine vision”, are a key component in today’s emerging AI systems as they enable these systems to “see” the environment they are in and to interact with the external world. An example of where we can observe this capability is the use of machine vision systems in advanced driver assistance systems (ADAS). 

Machine vision and cars

ADAS computer systems use a range of different optical image sensors to enable cars to learn to perceive the environment and respond autonomously or alert the driver. Computer vision algorithms such as object recognition, road sign detection, pedestrian detection, lane tracking, parking assistance, vehicle control, and collision warning assist in safe and smart driving.

After capturing a high-quality image, the machine vision system must analyze what is going on, and in real-time detect if there has been a change in road conditions. For example, if another vehicle is braking or a pedestrian is stepping into its way. The system then has to make a corrective driving decision: whether it needs to brake or adjust either the speed or direction of the vehicle.

What is the challenge and the goal?

Vision-based sensors are considerably quicker and more accurate than humans employed for a comparable task, but have high demands on real-time control. Most image sensors only process information from a scene in a limited manner. In the case of ADAS, because the vehicle must often react very swiftly, computer vision algorithms should process image frames quickly to identify and avoid danger. The algorithm processing time is directly related to computing power.

How are vision sensors becoming smart?

The smart vision sensor currently being developed by Dr. Bermak and his team in the Sensor Research Group aims at bringing the smart vision system into the next intelligence level with high robustness, low cost, low power, and low latency. 

The team is working on designing an energy-efficient smart vision sensor featuring AI capabilities integrated side-by-side with the image sensory array. The vision sensor will feature for the first time the co-design of novel AI algorithms and computationally efficient hardware with the multi-feature vision sensor. The team is receiving support for the project under Qatar National Research Fund’s National Priorities Research Program (NPRP). 

What are the novel aspects?

Emerging AI applications rely heavily on hardware as a core enabler of innovation, especially for logic and memory functions. CSE’s proposed smart vision sensor includes several innovative aspects. 

The first is its hardware-friendly AI algorithms with event-based processing. Event-based sensors can achieve very short latencies that are only constrained by the minimal response time of the sensor because they only collect data when something is happening.

An event-based camera (EBC), also known as a silicon retina, dynamic vision sensor (DVS), or neuromorphic camera, is a biologically-inspired vision system that acquires visual information in a different way than normal cameras. Event-based vision systems function much like the eye and the brain to perceive the vitality of the scene and overlook what is irrelevant. They produce significantly less data than a conventional sensor while achieving a higher equivalent temporal resolution to capture movements seamlessly. Utilizing an EBC in early image recognition can provide a major advantage to real time image processing applications.

By bypassing inherent limitations in conventional computer vision, event-based vision is disrupting the current technology in fields such as automotive vehicles, AI and deep learning, industrial automation, the Internet of Things (IoT), among others.

Second, the sensor’s new deep computing architecture with energy-efficient mixed-signal processing. The highly reconfigurable and fully dynamic mixed-signal processing architecture extends the processing capabilities with the highest energy efficiencies. 

Third, the event-based vision sensor will have embedded intelligence (i.e. built-in feature extraction), and fourth, it will use novel vision system integration methodologies with application-specific optimization. 

The result: a shift in machine vision

The innovative new hardware being deployed for the smart vision sensor will help overcome the limitations of machine vision by enabling the speed, performance, efficiency, and safety to allow the vision sensor to react more intelligently, more autonomously, infinitely faster, and safer than ever.

Potential applications

The smart vision sensor being developed at CSE can be deployed for a wide range of mission-critical applications from security and surveillance, to logistics, medical, and assisted living functions (the visually impaired and blind). It can offer advanced solutions for road safety through ADAS in future autonomous driving vehicles. 

The expected findings strongly relate to the wide application of AI that directly aligns with the priority theme of the Qatar National Research Strategy, and with the National Artificial Intelligence Strategy for Qatar, which was developed by Qatar Center for Artificial Intelligence (QCAI), a part of the Qatar Computing Research Institute at HBKU.

Dr. Bermak anticipates that the project will lead to patentable outcomes, a high-quality publishable set of contributions, and the training of postgraduate students in an increasingly popular and emerging research area. 

He says: “This project will provide not only a unique smart vision sensor but also novel insight for designing other multifunctional electronics for the human-machine interface. We are continually advancing in our application of smart technology and AI in people’s everyday lives to improve quality of life in the 21 century. The project is an example of how CSE’s specialized capabilities will support an emerging AI industrial sector for the benefit of Qatar’s knowledge-based economy and community.”