Neutral, standardized benchmarking of zkVM proving times

Aligned Layer

Aligned Layer

— 2 min read
Neutral, standardized benchmarking of zkVM proving times
zkbenchmarks.com

Zero-knowledge virtual machines (zkVMs) are virtual machines that let you write and execute code in common languages like Rust or C and generate proofs demonstrating the integrity of computations.

This makes building with ZK much more accessible to developers, and is a key part of Ethereum’s scaling roadmap. While initiatives like Ethproofs are benchmarking the progress of different zkVMs towards real time proving of Ethereum blocks, there are other useful computational workloads that we believe should also be used for benchmarking.

There are multiple zkVMs in the market or in active development, and many zkVM vendors claim to be the fastest according to their own benchmarking. Because we believe a neutral and standardized approach is necessary to accurately assess the performance of each zkVM, we have open-sourced our benchmarking work for RISC-V zkVMs at zkbenchmarks.com to help the community determine the best zkVM for their use case. 

These benchmarks are carried out on standardized GPU or CPU servers, and test each zkVM’s efficiency for proving specific programs. 

The full details, results, repo, and a fully reproducible Jupyter notebook are available at zkbenchmarks.com. Currently, the benchmarks test the following programs:

  • Fibonacci
  • Keccak
  • ️RSP (Reth Succinct Processor—proving Ethereum block execution using Reth)

The benchmarks have been run on the following zkVMs:

  • SP1
  • Risc0
  • Pico

We are actively collaborating with the different zkVM teams to determine the optimal configurations for different tasks, and thank these teams for their help so far.

Some observations from initial data

While our benchmarking work is ongoing, you can already see the results at zkbenchmarks.com. We discussed some initial observations on our most recent weekly Alignment Sessions stream, such as that SP1 currently outperforms RISC0 for proving Ethereum blocks (using the RSP benchmark), and that the relationship between proving time and the composition and size of the block being proved have a non-linear relationship. More tests with different Ethereum blocks will be needed.

Block proving performance (Block Megagas vs Time, on GPU) (note: The 27Mgas block has fewer transactions than the other blocks, but one of them is a high CPU usage transaction.)

Our testing shows us that specific optimizations used for each zkVM have a large impact on performance, and that different zkVMs may be better suited for different workloads. For example, Pico is the best performing zkVM for proving keccak hashes (when proving using CPUs, not GPUs).

Keccak proving performance comparison (CPU)

Next steps

We will be adding additional zkVMs like Valida, Nexus VM, and Zisk in the near future, as well as adding more proving workloads as benchmarks. These benchmarks are important for informing our product development pipeline and we hope that these results are beneficial to the wider community. We invite others to contribute to this open source effort by adding code to benchmark new zkVMs or to run new benchmarks (see open issues here). Small grants may be available to support this work, but in the meantime, if you are interested in helping with any of these issues please let us know in the Aligned Telegram group. 

Stay tuned:  🐦 Twitter | 🗨️ Telegram | 👾 Discord | 🌐 Website | 📝 Manifesto

Read more