To reflect exactly how fast a quantum computer can carry out a program, IBM is launching a new quantum speed metric, in a move that is likely to add a dose of competitiveness across an industry that is growing at pace.
Named CLOPS (Circuit Layer Operations Per Second), the metric is the first to measure the number of quantum circuits a quantum processing unit (QPU) can execute per unit of time, and is designed to provide an objective understanding of the amount of work a quantum system can do in a particular period.
Quantum volume is a metric that is now widely adopted across the industry, with major players like Honeywell basing performance measurements on the benchmark. IBM hopes that CLOPS will follow a similar path and this way enable quantum computing companies to put numbers on all three aspects of performance.
“Without progress on all three, we will not have practical and useful quantum computing systems,” Bob Sutor, chief quantum exponent at IBM, tells ZDNet. “Would you want a slow, poor quality, but big machine? How about a high-quality system that does not have enough qubits to tackle your problem? Would you use a system that was hundreds of times slower than a competitor’s system?”
CLOPS measures the speed of quantum circuits, which are the basic unit of computation for quantum computers, and which include the sequence of quantum operations, but also the interaction of the quantum system with a classical computer.
A quantum program, in effect, includes a degree of classical computing: developers use classical hardware, like a laptop, to convert instructions into a form consumable by the QPU, as well as to retrieve results from the calculation. With each “query” that is sent to the quantum computer, a few tens of thousands of quantum circuits are realized – which is why the speed at which the quantum-classical interaction happens is key to the performance of the overall system.
This speed is what CLOPS measures, which includes the time spent actually running the circuit on the device, but also the delay time between each shot of each circuit on the system, and the time spent preparing the circuits to run.
IBM has run CLOPS to benchmark several of the company’s quantum processors, ranging from five-qubit systems to 65-qubit devices. Although all of the quantum computers had a similar quantum volume, IBM’s researchers found vast differences in speed: the largest machine performed the slowest, at a CLOPS of 753 layers per second, compared to 1,419 for the smallest processor.
The company hopes, therefore, that CLOPS will provide a wider-ranging understanding of the performance of quantum hardware, which is not limited to the number of qubits or to quantum volume alone. “The three quantum performance metrics allow quantum computing providers to share the true performance of their systems to the world,” says Sutor.
This is particularly important, continues Sutor, as the quantum computing ecosystem continues to develop rapidly, and it becomes difficult to discern hype from reality.
Companies large and small, in effect, are increasingly taking interest in building quantum computing hardware and software, and the industry is moving fast. But this has raised concerns among experts that the technology is being overhyped, and that many promises will fail to be kept – which in turn, could lead to the entire industry crashing down.
“Hype happens when people make predictions about the future without the technical proof to back up their claims,” says Sutor. “Without milestones on a detailed roadmap, and demonstrations of meeting or exceeding those milestones, we are only left with inflated claims.”
“Publicly published metrics like number of qubits, quantum volume and now CLOPS show the true state of quantum systems today. They also measure progress over time.”
IBM, for its part, is working on improving the CLOPS performance of its current systems. The company recently unveiled a new architecture for quantum computing, named Qiskit Runtime, which is key to reducing latencies during quantum operations.
Runtime creates an execution environment located beside the quantum hardware, where developers can send their entire program – meaning that the loops that happen between the classical and the quantum environment are contained within a space that is physically near the quantum computer. This in turn slashes the latencies that emerge from communicating between a user’s computer and the quantum processor.
Using Qiskit Runtime, IBM’s team managed to reduce the time taken to simulate the behavior of a small molecule called lithium hydride (LiH) from 45 days to nine hours. CLOPS now enables the researchers to carry out timing measurements and identify specific speed bottlenecks in Runtime, such as idle times between consecutive circuits.
Improving CLOPS performance is now a point of focus for IBM, alongside scaling the number of qubits in the company’s processors, and increasing quantum volume. Sutor explains that, to maintain healthy competition, all quantum computing hardware providers should transparently share their results on all three aspects of performance.
“We believe it is the responsibility of quantum computer providers and those who have access to them to provide CLOPS and other metrics,” says Sutor. “That is, we all need to publish our technical results so that users and adopters of quantum computing can make their own informed decisions.”
If it is adopted as widely as quantum volume, the metric could help customers identify the pros and cons of different architectures such as superconducting qubits, trapped ions or cold atoms. Honeywell, for instance, recently claimed that the company has launched the highest performing quantum computing system in the world, boasting a quantum volume of 128 thanks to a trapped ions system. It remains to be seen if the processor’s speed can match the rest of the system’s performance.
