As we approach Nvidia GTC, its worth noting that there is another player in town. Or south of town, in San Diego: Qualcomm. The company has been building AI expertise and technology for over a decade, and we believe its lead over mobile rivals in both AI hardware and software is significant. The company has a broad SoC family that share core AI accelerator tech, including the new Snapdragon X Elite, the Snapdragon 8 Gen 3 for mobile, and the Cloud AI100 Ultra for data center AI.

I first got interested in Qualcomm AI at a luncheon the company sponsored in San Francisco back in 2017. The engineers seemed genuinely excited to talk about both the AI on Snapdragon for mobile and about what became the CloudAI 100, which boasted some 400 TOPS at only 75 watts when it launched in 2020. Thats an amazing amount of AI performance at low power. In fact, it is still the market leader. Let’s look at why we think Qualcomm now has the pole position in inference processing at the edge.

The Qualcomm logo outside its San Diego headquarters.

Qualcomm

Reasons for Qualcomm’s Leadership in Edge AI

I recently discussed Qualcomm’s AI strategy and products with Ziad Asghar, Qualcomm’s VP of product marketing. Qualcomm thinks that much of the AI inferencing being conducted on the cloud today will migrate to edge devices. After all, if the edge device has enough processing power and memory to do the work, why pay for time on a CSP’s infrastructure when you can run the job on the device you already own? Its free, right? And Ziad even predicted that AI apps such as Microsoft Co-Pilot, currently only available via the Azure cloud, will eventually run on-device!

But getting large language models to run on a phone or automotive SoC is not just a simple compile-and-go. You need to reduce the size of the model to fit in memory (quantization). You need to prune the network to increase performance (harvest sparsity). Using compression can further reduce the network size (MX6 compression). And new techniques such as speculative decoding can speed time to text by using two LLMs in parallel; a fast one to generate a possible answer and a more complete one to check that answer. These are all areas being researched and deployed by Qualcomm on Snapdragon and on Cloud AI100.

The X Elite is designed for laptop Windows PCs.

Qualcomm

Overall, here are a few reasons Qualcomm is well-positioned in these early days of AI on the edge.

1. Specialized Hardware for AI: Qualcomm’s Snapdragon series of processors include dedicated AI engines (such as the Hexagon DSP) designed to handle AI and machine learning tasks efficiently. This allows for faster processing of AI tasks on-device without the need to send data to the cloud.
2. Energy Efficiency: Qualcomm’s processors are designed to deliver high performance while managing power consumption effectively. This is crucial for edge AI applications where battery life and thermal management are important considerations.
3. Widespread Adoption in Mobile Devices: Qualcomm’s Snapdragon processors are used in a wide range of smartphones and devices. This installed base has driven a robust ecosystem of developers and applications optimized for Qualcomm’s hardware.
4. AI Software Tools and SDKs: Qualcomm provides comprehensive software support for AI development, including the AI Engine and specific SDKs for developers. This support simplifies the integration of AI capabilities into applications and services. To this end, Qualcomm introduced the AI Hub at CES this year; more on that in a minute.

Snapdragon Performance

But lets start with performance. Good performance is just the start of the AI journey, but without it, you go nowhere.

Tom’s Hardware and Wccftech have published articles (links below) discussing benchmarks showcasing the Snapdragon X Elite’s performance in single-threaded, multi-threaded CPU tasks, and AI workloads. Hopefully we will see some MLPerf benchmarks coming soon.

Here are some key points:

• AI Performance: The Snapdragon X Elite’s Hexagon NPU boasts 45 TOPS ( tera operations per second) performance, exceeding Intel and AMD offerings in AI inferencing according to Qualcomm’s benchmarks.The Snapdragon X Elite’s NPU significantly outperforms the Intel chip in AI inferencing tasks using the UL Procyon benchmark
• CPU Performance: The Snapdragon X Elite also demonstrates competitive CPU performance against Intel Core i7 and AMD Ryzen chips in benchmarks like Geekbench and Cinebench. And versus the Apple M3, the X Elite was measured to be 21% faster.
• The performance and power efficiency of the Cloud AI 100 has attracted partners looking to run inference processing at a lower cost and power consumption, including Amazon AWS, HP Enterprise, Dell, and Lenovo. Cerebras, the inventor of the Wafer Scale Engine 3, announced this week that they, too, will partner with Qualcomm and are realizing a 10X cost advantage using this platform for their customers.
• We note that the X Elite uses a completely new core, designed by Nuvia which Qualcomm acquired in 2021. We expect this new core to be used in future data center and mobile devices soon. We also should point out that X Elite, while a great chip for the age of the AI PC, has a big software hill to climb for Windows applications.

The Cloud AI 100 Ultra, a four-chip PCI card now supported by major server OEMs.

Qualcomm
XDA DevelopersSnapdragon X Elite vs Intel Core Ultra 7 155H: We ran the benchmarks
Tom’s HardwareEarly Snapdragon X Elite benchmark shows Arm CPU is faster than AMD’s top-end mobile APU

AI Software

The Qualcomm AI Hub is a resource specifically designed for developers working on devices powered by Qualcomm’s Snapdragon and other platforms. It functions as a central location for developers to access and utilize tools for on-device AI development. Here’s a breakdown of its key features:

• AI Model Library: The hub offers a collection of over 75 pre-optimized AI models. These models cover various tasks like image recognition, object detection, speech processing, and more. Developers can easily integrate these models into their applications, reducing development time and effort.
• Focus on On-Device AI: The AI Hub emphasizes on-device AI, where AI processing happens directly on the device itself, rather than relying on the cloud. This approach offers benefits like faster response times, improved privacy, and lower reliance on internet connectivity.
• Pre-Optimization for Performance: The AI models offered in the hub are specifically optimized to work efficiently on Qualcomm processors. This optimization ensures smooth performance and efficient use of device resources.
• Accessibility: The Qualcomm AI Hub makes these models available on multiple platforms besides their own. Developers can find them on the Qualcomm AI Hub itself, Hugging Face, and GitHub.

The Qualcomm AI Hub

Qualcomm

Overall, the Qualcomm AI Hub aims to simplify and accelerate the development of AI-powered applications for devices running on Qualcomm Snapdragon and other platforms by providing pre-optimized models and resources tailored for on-device AI.

Conclusions

As Qualcomm continues to evolve, they see AI as the core differentiator going forward. AI can help everything from photography to personalized productivity services. Armed with their own in-house research organization for long-term innovation and SoC teams that reliably execute to plan, Qualcomm is set to lead Edge AI hardware and software. AMD and Intel don’t participate in the mobile space, and Apple doesn’t even seem to be aware that an AI revolution is already underway.

Qualcomm must develop and execute a winning ecosystem strategy for Windows on X Elite, but Microsoft is there to help out. Nobody wants to run an application in emulation mode, at least not for long.

While Nvidia is by far the leader in data center AI, we believe Qualcomm has the lead position on the Edge, where a lot of action is taking place. And some really cool phones are already in market to demonstrate the value of AI on the Edge.

Disclosures: This article expresses the author’s opinions and

should not be taken as advice to purchase from or invest in the companies mentioned. Cambrian AI Research is fortunate to have many, if not most, semiconductor firms as our clients, including Blaize, BrainChip, CadenceDesign, Cerebras, D-Matrix, Eliyan, Esperanto, FuriosaAI, Graphcore, GML, IBM, Intel, Mythic, NVIDIA, Qualcomm Technologies, Si-Five, SiMa.ai, Synopsys, Ventana Microsystems, Tenstorrent and scores of investment clients. We have no investment positions in any of the companies mentioned in this article and do not plan to initiate any in the near future. For more information, please visit our website at https://cambrian-AI.com.