The rapid adoption of AI has spurred the development of a new breed of AI chip with the performance needed to meet the high processing and low power requirements of machine learning and deep learning applications. There are now more than 80 start-up companies around the world developing AI chips for training and inference.
These start-ups are focused on developing chips designed for specific AI workloads and use cases using new processor architectures which address the so-called “memory bottleneck” by adopting a much better balance between memory and on-chip compute. As a result, they hope to deliver dramatic improvements in performance and efficiency compared to today’s CPUs and GPUs.
AI-friendly memories being adopted by start-ups include near-chip memories (DRAM and NAND flash), on-chip memories (SRAM) and emerging NV memories. “Some of these also start-ups use analog memory technologies which promise to reduce power consumption from milliwatts to microwatts” said Gareth Owen, Associate Director of Emerging Technologies at Counterpoint Research, “This is ideal for the always-on inference market, for example, for smart speakers, potentially increasing battery life from days to months. However, there are still many challenges, particularly associated with mass production.”
The current AI landscape is still in flux. Although a host of AI start-ups are developing new hardware architectures and software stacks, client companies do not want to be locked to any specific AI acceleration architecture, particularly with consolidation likely in the next few years. Also, they want to ensure that solutions are scalable.
In the training market, competition will be particularly tough. Start-ups will have to compete against Nvidia as well as home-grown products from potential customers such as Amazon, Alibaba, Baidu, Facebook, Google and Tencent. In addition, the cloud training market will be a smaller market than the inference market.
A small number of vendors, such as Graphcore and Habana Labs, have already launched commercial products, mostly PCI-type plug-in accelerators. However, they are competing against Nvidia, who will be tough to beat for several reasons, but particularly due to its strong software ecosystem. As a result, many start-ups are staying out of Nvidia’s way and focusing on the inference market.
In contrast, the edge-based inference market is still in its infancy with many new players and new approaches. It will be a much bigger market opportunity than the cloud data market, with demand for AI chips in potentially billions of industrial and consumer devices and hence with a wide range of different requirements. Like the server chip vendors, however, edge-based chip start-ups will need to focus on very specific markets differentiated, primarily, by processing capability, power and cost requirements.
Key start-ups in the inference market include Gyrfalcon, Efinix and Syntiant which have already launched commercial products, and Flex Logix, Mythic and Blaize which are sampling and will offer commercial products later this year.
However, these start-ups face their own unique set of challenges. To be successful, they must offer chipsets that are highly scalable and flexible, achieve the right balance between performance and power budget but also feature strong ecosystem support and a comprehensive software stack. Time to market will also be key.
They will also face tough competition from established players such as Intel, Nvidia, Xilinx and Microsoft’s Altera who will be under pressure to acquire emerging winners in order to maintain and increase overall market share. This has already started to happen with Intel’s recent acquisition of Habana Labs for $2 billion.
At present, it is not clear who will be the winners or losers, but the industry will not accommodate many new chip suppliers. A few will get acquired, but most will disappear.
“We believe that there will be consolidation in the market during the next two years, due predominantly to a mismatch between actual performance from working silicon and predicted theoretical performance,” said Peter Richardson, Research Director at Counterpoint Research. “Many of these start-ups have not developed software in tandem with hardware and thus are not able to perform accurate performance modelling, resulting in shortfalls in actual performance achieved.” he added.
Ultimately, the differentiator may not be the company with the best hardware solution but the one with best software/hardware mix. Established players such as Intel and Nvidia will continue to dominate the market landscape due to the robust developer ecosystems they have created around their AI chipsets.
