It’s been 8 years of telephone AI chips — they usually’re nonetheless losing their potential

[ad_1] It’s been a bit over eight years since we first began speaking about Neural Processing Items (NPUs) inside our smartphones and the early prospects of on-device AI. Large factors should you keep in mind that the HUAWEI Mate 10’s Kirin 970 processor was the primary, although comparable concepts had been floating round, significantly in imaging, earlier than then.

After all, lots has modified within the final eight years — Apple has lastly embraced AI, albeit with blended outcomes, and Google has clearly leaned closely into its Tensor Processor Unit for every part from imaging to on-device language translation. Ask any of the large tech firms, from Arm and Qualcomm to Apple and Samsung, they usually’ll all let you know that AI is the way forward for smartphone {hardware} and software program.

And but the panorama for cellular AI nonetheless feels fairly confined; we’re restricted to a small however rising pool of on-device AI options, curated principally by Google, with little or no in the way in which of a inventive developer panorama, and NPUs are partly guilty — not as a result of they’re ineffective, however as a result of they’ve by no means been uncovered as an actual platform. Which begs the query, what precisely is that this silicon sitting in our telephones actually good for?

What's an NPU anyway?

<img class="e_jg" decoding="async" loading="lazy" sizes="(min-width: 64rem) 51.25rem, 80vw" title="SoC resting on Google Pixel phone" srcset="https://www.androidauthority.com/wp-content/uploads/2023/06/SoC-resting-on-Google-Pixel-phone.jpg 1920w, https://www.androidauthority.com/wp-content/uploads/2023/06/SoC-resting-on-Google-Pixel-phone-1536w-864h.jpg 1536w, https://www.androidauthority.com/wp-content/uploads/2023/06/SoC-resting-on-Google-Pixel-phone-675w-380h.jpg 675w, https://www.androidauthority.com/wp-content/uploads/2023/06/SoC-resting-on-Google-Pixel-phone-64w-36h.jpg 64w, https://www.androidauthority.com/wp-content/uploads/2023/06/SoC-resting-on-Google-Pixel-phone-1000w-563h.jpg 1000w, https://www.androidauthority.com/wp-content/uploads/2023/06/SoC-resting-on-Google-Pixel-phone-300w-170h.jpg 300w, https://www.androidauthority.com/wp-content/uploads/2023/06/SoC-resting-on-Google-Pixel-phone-1280w-720h.jpg 1280w, https://www.androidauthority.com/wp-content/uploads/2023/06/SoC-resting-on-Google-Pixel-phone-840w-472h.jpg 840w" alt="SoC resting on Google Pixel phone" src="https://www.androidauthority.com/wp-content/uploads/2023/06/SoC-resting-on-Google-Pixel-phone.jpg"/> Robert Triggs / Android Authority

Earlier than we are able to decisively reply whether or not telephones actually “want” an NPU, we must always most likely acquaint ourselves with what it truly does.

Similar to your telephone’s general-purpose CPU for operating apps, GPU for rendering video games, or its ISP devoted to crunching picture and video knowledge, an NPU is a purpose-built processor for operating AI workloads as shortly and effectively as attainable. Easy sufficient.

Particularly, an NPU is designed to deal with smaller knowledge sizes (equivalent to tiny 4-bit and even 2-bit fashions), particular reminiscence patterns, and extremely parallel mathematical operations, equivalent to fused multiply-add and fused multiply–accumulate.

Cell NPUs have taken maintain to run AI workloads that conventional processors battle with.

Now, as I mentioned again in 2017, you don’t strictly want an NPU to run machine studying workloads; a number of smaller algorithms can run on even a modest CPU, whereas the information facilities powering varied Massive Language Fashions run on {hardware} that’s nearer to an NVIDIA graphics card than the NPU in your telephone.

Nonetheless, a devoted NPU will help you run fashions that your CPU or GPU can’t deal with at tempo, and it will probably usually carry out duties extra effectively. What this heterogeneous strategy to computing can value when it comes to complexity and silicon space, it will probably acquire again in energy and efficiency, that are clearly key for smartphones. Nobody needs their telephone’s AI instruments to eat up their battery.

Wait, however doesn’t AI additionally run on graphics playing cards?

<img class="e_jg" decoding="async" loading="lazy" sizes="(min-width: 64rem) 51.25rem, 80vw" title="NVIDIA GeForce 4090 in box" srcset="https://www.androidauthority.com/wp-content/uploads/2023/09/NVIDIA-GeForce-4090-in-box-scaled.jpeg 1920w, https://www.androidauthority.com/wp-content/uploads/2023/09/NVIDIA-GeForce-4090-in-box-scaled-1536w-864h.jpeg 1536w, https://www.androidauthority.com/wp-content/uploads/2023/09/NVIDIA-GeForce-4090-in-box-scaled-675w-380h.jpeg 675w, https://www.androidauthority.com/wp-content/uploads/2023/09/NVIDIA-GeForce-4090-in-box-scaled-64w-36h.jpeg 64w, https://www.androidauthority.com/wp-content/uploads/2023/09/NVIDIA-GeForce-4090-in-box-scaled-1000w-563h.jpeg 1000w, https://www.androidauthority.com/wp-content/uploads/2023/09/NVIDIA-GeForce-4090-in-box-scaled-300w-170h.jpeg 300w, https://www.androidauthority.com/wp-content/uploads/2023/09/NVIDIA-GeForce-4090-in-box-scaled-1280w-720h.jpeg 1280w, https://www.androidauthority.com/wp-content/uploads/2023/09/NVIDIA-GeForce-4090-in-box-scaled-840w-472h.jpeg 840w" alt="NVIDIA GeForce 4090 in box" src="https://www.androidauthority.com/wp-content/uploads/2023/09/NVIDIA-GeForce-4090-in-box-scaled.jpeg"/> Oliver Cragg / Android Authority

If you happen to’ve been following the continuing RAM value disaster, you’ll know that AI knowledge facilities and the demand for highly effective AI and GPU accelerators, significantly these from NVIDIA, are driving the shortages.

What makes NVIDIA’s CUDA structure so efficient for AI workloads (in addition to graphics) is that it’s massively parallelized, with tensor cores that deal with extremely fused multiply–accumulate (MMA) operations throughout a variety of matrix and knowledge codecs, together with the tiny bit-depths used for contemporary quantized fashions.

Whereas fashionable cellular GPUs, like Arm’s Mali and Qualcomm’s Adreno lineup, can help 16-bit and more and more 8-bit knowledge sorts with extremely parallel math, they don’t execute very small, closely quantized fashions — equivalent to INT4 or decrease — with wherever close to the identical effectivity. Likewise, regardless of supporting these codecs on paper and providing substantial parallelism, they aren’t optimized for AI as a main workload.

Cell GPUs give attention to effectivity; they're far much less highly effective for AI than desktop rivals.

Not like beefy desktop graphics chips, cellular GPU architectures are designed before everything for energy effectivity, utilizing ideas equivalent to tile-based rendering pipelines and sliced execution items that aren’t solely conducive to sustained, compute-intensive workloads. Cell GPUs can positively carry out AI compute and are fairly good in some conditions, however for extremely specialised operations, there are sometimes extra power-efficient choices.

Software program improvement is the opposite equally essential half of the equation. NVIDIA’s CUDA exposes key architectural attributes to builders, permitting for deep, kernel-level optimizations when operating AI workloads. Cell platforms lack comparable low-level entry for builders and gadget producers, as a substitute counting on higher-level and infrequently vendor-specific abstractions equivalent to Qualcomm’s Neural Processing SDK or Arm’s Compute Library.

This highlights a big ache level for the cellular AI improvement atmosphere. Whereas desktop improvement has principally settled on CUDA (although AMD’s ROCm is gaining traction), smartphones run a wide range of NPU architectures. There’s Google’s proprietary Tensor, Snapdragon Hexagon, Apple’s Neural Engine, and extra, every with its personal capabilities and improvement platforms.

NPUs haven’t solved the platform drawback

<img class="e_jg" decoding="async" loading="lazy" sizes="(min-width: 64rem) 51.25rem, 80vw" title="gemini image generation disney openai" srcset="https://www.androidauthority.com/wp-content/uploads/2025/12/gemini-image-generation-disney-openai.jpg 1920w, https://www.androidauthority.com/wp-content/uploads/2025/12/gemini-image-generation-disney-openai-64w-36h.jpg 64w, https://www.androidauthority.com/wp-content/uploads/2025/12/gemini-image-generation-disney-openai-1000w-563h.jpg 1000w, https://www.androidauthority.com/wp-content/uploads/2025/12/gemini-image-generation-disney-openai-1536w-864h.jpg 1536w, https://www.androidauthority.com/wp-content/uploads/2025/12/gemini-image-generation-disney-openai-675w-380h.jpg 675w, https://www.androidauthority.com/wp-content/uploads/2025/12/gemini-image-generation-disney-openai-300w-170h.jpg 300w, https://www.androidauthority.com/wp-content/uploads/2025/12/gemini-image-generation-disney-openai-1280w-720h.jpg 1280w, https://www.androidauthority.com/wp-content/uploads/2025/12/gemini-image-generation-disney-openai-840w-472h.jpg 840w" alt="gemini image generation disney openai" src="https://www.androidauthority.com/wp-content/uploads/2025/12/gemini-image-generation-disney-openai.jpg"/> Taylor Kerns / Android Authority

Smartphone chipsets that boast NPU capabilities (which is basically all of them) are constructed to resolve one drawback — supporting smaller knowledge values, advanced math, and difficult reminiscence patterns in an environment friendly method with out having to retool GPU architectures. Nonetheless, discrete NPUs introduce new challenges, particularly on the subject of third-party improvement.

Whereas APIs and SDKs can be found for Apple, Snapdragon, and MediaTek chips, builders historically needed to construct and optimize their purposes individually for every platform. Even Google doesn’t but present simple, normal developer entry for its AI showcase Pixels: the Tensor ML SDK stays in experimental entry, with no assure of normal launch. Builders can experiment with higher-level Gemini Nano options by way of Google’s ML Equipment, however that stops nicely in need of true, low-level entry to the underlying {hardware}.

Worse, Samsung withdrew help for its Neural SDK altogether, and Google’s extra common Android NNAPI has since been deprecated. The result's a labyrinth of specs and deserted APIs that make environment friendly third-party cellular AI improvement exceedingly tough. Vendor-specific optimizations have been by no means going to scale, leaving us caught with cloud-based and in-house compact fashions managed by a number of main distributors, equivalent to Google.

LiteRT runs on-device AI on Android, iOS, Net, IoT, and PC environments.

Fortunately, Google launched LiteRT in 2024 — successfully repositioning TensorFlow Lite — as a single on-device runtime that helps CPU, GPU, and vendor NPUs (presently Qualcomm and MediaTek). It was particularly designed to maximise {hardware} acceleration at runtime, leaving the software program to decide on probably the most appropriate technique, addressing NNAPI’s greatest flaw. Whereas NNAPI was meant to summary away vendor-specific {hardware}, it finally standardized the interface reasonably than the conduct, leaving efficiency and reliability to vendor drivers — a spot LiteRT makes an attempt to shut by proudly owning the runtime itself.

Apparently, LiteRT is designed to run inference solely on-device throughout Android, iOS, embedded programs, and even desktop-class environments, signaling Google’s ambition to make it a very cross-platform runtime for compact fashions. Nonetheless, in contrast to desktop AI frameworks or diffusion pipelines that expose dozens of runtime tuning parameters, a TensorFlow Lite mannequin represents a totally specified mannequin, with precision, quantization, and execution constraints determined forward of time so it will probably run predictably on constrained cellular {hardware}.

<img class="e_jg" decoding="async" loading="lazy" sizes="(min-width: 64rem) 51.25rem, 80vw" title="LiteRT Hardware Accelerator Table" srcset="https://www.androidauthority.com/wp-content/uploads/2026/01/LiteRT-Hardware-Accelerator-Table.png 546w, https://www.androidauthority.com/wp-content/uploads/2026/01/LiteRT-Hardware-Accelerator-Table-64w-24h.png 64w" alt="LiteRT Hardware Accelerator Table" src="https://www.androidauthority.com/wp-content/uploads/2026/01/LiteRT-Hardware-Accelerator-Table.png"/> Whereas abstracting away the vendor-NPU drawback is a significant perk of LiteRT, it’s nonetheless value contemplating whether or not NPUs will stay as central as they as soon as have been in mild of different fashionable developments.

As an example, Arm’s new SME2 exterior extension for its newest C1 collection of CPUs gives as much as 4x CPU-side AI acceleration for some workloads, with vast framework help and no want for devoted SDKs. It’s additionally attainable that cellular GPU architectures will shift to raised help superior machine studying workloads, presumably lowering the necessity for devoted NPUs altogether. Samsung is reportedly exploring its personal GPU structure particularly to raised leverage on-device AI, which may debut as early because the Galaxy S28 collection. Likewise, Immagination’s E-series is particularly constructed for AI acceleration, debuting help for FP8 and INT8. Possibly Pixel will undertake this chip, finally.

LiteRT enhances these developments, liberating builders to fret much less about precisely how the {hardware} market shakes out. The advance of advanced instruction help on CPUs could make them more and more environment friendly instruments for operating machine studying workloads reasonably than a fallback. In the meantime, GPUs with superior quantization help would possibly finally transfer to grow to be the default accelerators as a substitute of NPUs, and LiteRT can deal with the transition. That makes LiteRT really feel nearer to the mobile-side equal of CUDA we’ve been lacking — not as a result of it exposes {hardware}, however as a result of it lastly abstracts it correctly.

Devoted cellular NPUs are unlikely to vanish however apps could lastly begin leveraging them.

Devoted cellular NPUs are unlikely to vanish any time quickly, however the NPU-centric, vendor-locked strategy that outlined the primary wave of on-device AI clearly isn’t the endgame. For many third-party purposes, CPUs and GPUs will proceed to shoulder a lot of the sensible workload, significantly as they acquire extra environment friendly help for contemporary machine studying operations. What issues greater than any single block of silicon is the software program layer that decides how — and if — that {hardware} is used.

If LiteRT succeeds, NPUs grow to be accelerators reasonably than gatekeepers, and on-device cellular AI lastly turns into one thing builders can goal with out betting on a particular chip vendor’s roadmap. With that in thoughts, there’s most likely nonetheless some option to go earlier than on-device AI has a vibrant ecosystem of third-party options to take pleasure in, however we're lastly inching a bit bit nearer.

Don’t need to miss the most effective from Android Authority?

<img class="e_jg" decoding="async" loading="lazy" sizes="9.375rem" title="google preferred source badge light@2x" srcset="https://www.androidauthority.com/wp-content/uploads/2025/09/google_preferred_source_badge_light@2x.png 676w, https://www.androidauthority.com/wp-content/uploads/2025/09/google_preferred_source_badge_light@2x-64w-20h.png 64w" alt="google preferred source badge light@2x" src="https://www.androidauthority.com/wp-content/uploads/2025/09/google_preferred_source_badge_light@2x.png"/><img class="e_jg" decoding="async" loading="lazy" sizes="9.375rem" title="google preferred source badge dark@2x" srcset="https://www.androidauthority.com/wp-content/uploads/2025/09/google_preferred_source_badge_dark@2x.png 676w, https://www.androidauthority.com/wp-content/uploads/2025/09/google_preferred_source_badge_dark@2x-64w-20h.png 64w" alt="google preferred source badge dark@2x" src="https://www.androidauthority.com/wp-content/uploads/2025/09/google_preferred_source_badge_dark@2x.png"/> Thanks for being a part of our neighborhood. Learn our Remark Coverage earlier than posting.