because of their high computational requirements. Time-critical
inference tasks should be performed on the mobile device,
whenever they don’t exceed its processing or storage capacity.
Other inferences that are less time-critical, or require a larger
knowledge model may be shipped off to the cloud.
Likewise, some of the sensor fusion tasks in cars, drones,
robots and other autonomously-mobile products can be done
remotely, or efficiently handled by a pre-learned knowledge
base, but most sensor fusion must be done onboard and in real
time in a centralized hub on the vehicle or robot itself.
NVIDIA has addressed the issue of segmentation with
a multi-pronged approach, developing both an onboard
processor for AI-based sensor fusion and autonomous
guidance and control, and an architecture (together with
Microsoft) for a cloud-based AI data center, where the
deployable GPUs can be trained, and where the extensive
knowledge model can reside. The onboard processing system
is based on the Xavier SoC mentioned earlier in this story.
Billed as “the world’s 1st AI car superchip,” Xavier will be
trained offline in a datacenter.
AI-Capable GPUs Enable Sensor Fusion
Since nearly all AI-enabled edge applications require some
type of sensor fusion technology, many vendors are scrambling
to deliver products and platforms to meet the demand. Three of
the world’s leading chip makers (Intel, Qualcomm, and NVIDIA)
and specialty-focused Mentor Graphics Corp. have recently
released integrated sensor fusion platforms. These platforms
differ from previous sensor-processing systems in two major
ways: (1) they accept raw data from multiple sensors, and
( 2) use neural networks (often deep learning) and other AI
algorithms to process and integrate it into a useable form.
Qualcomm’s Drive Data Platform is designed for easy
insertion into the autonomous vehicle supply chain. Based on
the Snapdragon 820, it makes extensive use of the Snapdragon
Neural Processing Engine (SNPE) to process, fuse, and
interpret multiple streams of imaging data generated by camera,
radar, and LIDAR.
Meanwhile, Intel has made its own bid to gain traction in the
vehicular sensor fusion platform market with the acquisition of
Israeli-based Mobileye. Their 5th-generation System-on-a-Chip
(SoC) is expected to be deployed in autonomous vehicles by
NVIDIA’s Drive PX2 platform is the product of a collaboration
with Bosch, and is based on its forthcoming Xavier device.
AI for Home Security and Appliances
One of other large markets anticipated for AI technology
is in home security and home appliance devices. For these
household-level edge devices, sensor fusion will be as
important as it is in autonomous vehicles. It will also introduce
some interesting new types of sensor modalities. For example,
Audio Analytic has created a technology that can discern
sounds that have “non-communicable intent” from both ambient
noise as well as intentional sounds, such as speech and music.
This technology enables a sensor fusion AI system
equipped with microphones to monitor a household, office
or industrial building and identify any sounds that require an
active response, such as crashing glass inside a home, or a
siren heard outside a car, even when they are faint, or nearly
drowned out by background noise. The system’s response
could range from collecting more data to sounding an alert to
driving a car to the side of the road and allowing an emergency
vehicle to pass.
Robotics and Smart Manufacturing
AI and its associated technologies will also accelerate
the evolution of the industrial robots that are already making
inroads into the factory labor pool. One example is NVIDIA’s
Isaac robot simulator that changes how robots learn to perform
complex tasks (Figure 1). The AI-based software platform lets
development teams train and test robots in highly realistic
virtual environments. Once trained in the virtual environment, its
knowledge can be imported to other models and variants of the
“AI everywhere” is becoming a common phrase as AI finds
its way into applications as diverse as personal assistants,
personalized recommendation apps in online vendors,
home security, robotics, and smart manufacturing. Of all the
technology evolutions over the past decade, AI is the one most
likely to impact the greatest range of products and their design.
Reader Note: An extended version of this story, complete
with longer explanations, more graphics, and extensive
references, is available on the PD&D web site at