This document describes The NEASQC Benchmark Suite (TNBS). The objective of the document is to define the benchmarks that compose it, and the methodology for executing them and reporting their results. It includes also a short description of the TNBS website and the associated repository of submitted benchmark results.
TNBS has been designed to take into account four main objectives:
• Objective 1: the test cases which compose the suite must help computer architects, programmers and researchers to design future quantum computers, taking into account the variability in the performance introduced by the different components of the stack.
• Objective 2: the test cases must help to understand the evolution of quantum computers (more qubits, better topologies) or to improve the current one (reduction of noise, better compilers, etc.).
• Objective 3: the test cases must allow to compare the performance of different platforms. Currently, there are many different proposals (as transmons, ions, neutral atoms, etc.) which use different one- and two-qubit gates. The benchmarks must allow users and researchers to compare the performance of different platforms, and find their bottlenecks. For users, they should allow them to find the best platforms for their application.
• Objective 4: the test cases should consider other metrics which may be important to understand the quantum computing advantage (such as energy consumption, throughput or better scalability).
To achieve these objectives, TNBS has found representative kernels among the NEASQC uses cases to define a set of well-defined tests. Currently, TNBS is composed of four cases, each of them with:
1. A full and detailed documentation of the suite and of each microbenchmark. This documentation will include the definition of the microbenchmarks, the rules for executing them, the methods for reporting the results, and the methods for evaluating the final results.
2. An Eviden myQLM reference implementation, that allows the analysis of their complete results.
This document includes sections for describing the general rules for executing the benchmarks, the methods for gener-ating the results, the process to submit them, the list of benchmarks that compose the Suite, and a brief summary of the capabilities of the repository. As appendices, the JSON schema for reporting the results, the template for defining new cases, and the four documents that describe the current benchmarks are included. All the cases included a reference version based on Eviden myQLM framework.
In the future, the TNBS will increase the number of cases, from NEASQC use cases or from external proposals. So, this document will be a live document, with continuous improvements.