Benchmarking httomo / Nabu / TomoCuPy on the Komondor supercomputer

We have built a benchmark suite to comparatively assess the performance of the httomo, Nabu and TomoCuPy computed tomography toolkits. The executed pipelines are matching in the input data, algorithms and parameters, therefore the benchmarks are suitable to compare the resource utilization and efficiency of each toolkit. For reproducibility and potential spinoff work, the suite was made publicly available on GitHub. We hope that the members of the open-source tomography software community find this work interesting and informative. It was definitely a worthy learning experience for us.

The benchmark suite is composed of the following parts:

• Definition of a common Python environment which accommodates all three toolkits
• A tool to generate input data suitable for all three toolkits
• Pipeline definitions that match features and configuration between the toolkits
• An Apptainer/Singularity container definition to build a container image containing the environment
• SLURM scripts to execute the reconstruction jobs on the Komondor cluster

Common Python environment

The specification of the Conda environment can be found in the repository. It contains the dependencies for all three benchmarked toolkits. We found it important to use the same environment in all benchmarks, to exclude the potential performance impact of a different stack. Notable dependencies include:

• python=3.12
• numpy=2.4
• cuda-version=12.9
• cupy=14.0.1
• openmpi=4.1.6

Additionally, some packages were installed using pip on top of the Conda environment. Most notably, the toolkits under benchmark were installed this way:

• httomo suite
  • httomo=3.2.1
  • httomo-backends=1.2.0
  • httomolibgpu=5.8.1
  • httomolib=4.2
  • ToMoBAR=2026.3.1.0
• nabu=2025.2.6
• tomocupy: Our fork based on version 1.1.0 was used, which fixes a problem in the log-polar reconstruction

To recreate the environment on your local machine, cd to the cloned repository root, and invoke the following command. By default, this overwrites the Conda environment named tomobenchmarks.

$ just recreate-env

To build an Apptainer/Singularity image containing the dependencies and toolkits, execute the following. This produces the tomobenchmarks.sif image based on Ubuntu 24.04.

$ apptainer --verbose build tomobenchmarks.sif tomobenchmarks.def

Generating input data

We provide the means to generate a synthetic input data file that can be read by all three toolkits. While all of those work with the HDF5 format, the expected layout of the file is different for each. Nabu and httomo employ the NeXus standard, albeit in slightly different ways, whereas TomoCuPy is completely different. To resolve this, we provide the script nxs_generator.py that generates a HDF5 file that fulfills all toolkits’ structural requirements. It is achieved using HDF5 internal links, so that the file size does not increase.

The synthetic sinogram data is generated using the TomoPhantom library and depicts the well-known Shepp-Logan phantom.

Two input sizes are defined for the benchmarks on synthetic data. The file synthetic.nx contains 1 GiB of data and is suitable to be processed on most workstation machines. synthetic-huge.nx is over 30 GiB in size, and is used in the multi-GPU benchmarks. Additionally, the multi-GPU benchmarks were executed on a third dataset, which is a real-world capture of a sandstone rock. This dataset can be acquired from Zenodo.org. The following table summarizes the properties of the three input datasets.

Filename	synthetic.nx	synthetic-huge.nx	119647.nx
Data	Synthetic Shepp-Logan phantom	Synthetic Shepp-Logan phantom	Sandstone rock captured at DLS
Format	uint16	uint16	uint16
Detector width	1024	2048	2560
Detector height	1024	4096	2160
Number of angles	512	2000	1801
Data size [GiB]	1	31.3	18.5

To generate synthetic.nx, run the following in the repository root:

$ just generate-input

To generate both synthetic datafiles, run the following in the repository root:

$ just generate-all

The sandstone dataset must be modified to conform to Nabu toolkit’s format standard. Importantly, the file extension must be nx. We provide a script to perform the required modifications to the file contents:

$ just preprocess-dataset path/to/119647.nx

Notably, all input and output data is read/written to the /scratch partition of the Komondor cluster, which is the fastest available filesystem.

Benchmarks

In total, the runtime of 16 benchmark cases were measured. The following table lists all measured benchmark cases:

Toolkit	Input datafile	Pipeline	Hardware
httomo	synthetic.nx	fbp.yaml	single-GPU
httomo	synthetic-huge.nx	fbp.yaml	multi-GPU
httomo	119647.nx	fbp.yaml	multi-GPU
httomo	synthetic.nx	fbp-preproc.yaml	single-GPU
httomo	synthetic-huge.nx	fbp-preproc.yaml	multi-GPU
httomo	synthetic.nx	lprec.yaml	single-GPU
httomo	synthetic-huge.nx	lprec.yaml	multi-GPU
httomo	119647.nx	lprec.yaml	multi-GPU
Nabu	synthetic.nx	fbp.conf	single-GPU
Nabu	synthetic-huge.nx	fbp.conf	multi-GPU
Nabu	119647.nx	fbp.conf	multi-GPU
Nabu	synthetic.nx	fbp-preproc.conf	single-GPU
Nabu	synthetic-huge.nx	fbp-preproc.conf	multi-GPU
TomoCuPy	synthetic.nx	fbp.conf	single-GPU
TomoCuPy	synthetic.nx	fbp-preproc.conf	single-GPU
TomoCuPy	synthetic.nx	lprec.conf	single-GPU

The performance is assessed based on the end-to-end runtime which is measured with the time command. In case of multiple MPI processes launched, the longest is considered. Since the containerized runs lack kernel caching, in each run, the pipeline was executed twice, and the second result is considered to be valid.

Pipeline definitions

Three tomography pipelines are defined to be benchmarked, with their configuration files checked in to the repository. The sub-algorithms and parameters are chosen to be as close as possible across the toolkits.

• FBP pipeline: reconstruction based on the filtered backprojection algorithm. No preprocessing applied.
• FBP-preproc pipeline: reconstruction based on the filtered backprojection algorithm. Preprocessing steps are added: median-based outlier removal, Paganin phase-retrival and Fourier-Wavelet-based stripe removal algorithm.
• LPRec pipeline: Log-Polar reconstruction without preprocessing. This pipeline was executed with httomo and TomoCuPy only, since this algorithm is not implemented in Nabu.

Single- and multi-GPU setups

The measurements were conducted on the GPU partition of the Komondor supercomputer.

• For the single-GPU setup, a node partition of 16 CPUs, 64 GiB of RAM and a single Nvidia A100 40 GB accelerator.
• The multi-GPU measurements were occupying a full node of an AMD EPYC 7763 64-core CPU, 256 GiB RAM and four A100 accelerators.

Benchmark results

Small synthetic data

The following table and figure list the benchmark execution times when using the small synthetic dataset (synthetic.nx):

Pipeline	httomo	nabu	TomoCuPy
FBP	14.042	12.291	10.07
FBP /w preprocessing	20.094	12.945	14.006
LPRec	13.667	–	8.634

Huge synthetic data

The following table and figure list the benchmark execution times when using the huge synthetic dataset (synthetic-huge.nx):

Pipeline	httomo	Nabu
FBP	121.758	67.153
FBP /w preprocessing	209.73	69.141
LPRec	112.033	–

Sandstone rock dataset

The following table and figure list the benchmark execution times when using the real-world capture of a sandstone rock (119647.nx):

Pipeline	httomo	Nabu
FBP	93.821	64.503
LPRec	74.455	–

Summary of results

In this work, we applied a gray-box method in evaluating the results. We did not compare the toolkits on the source code level, though from tracing and the benchmark results some conclusions can be drawn. Also, we found a few items that can be a basis for further investigation and optimization work.

• We have shown that it is possible to build a Python environment that can accommodate all three toolkits.
• Comparing the FBP-preprocessed results has limited explanation value. It is likely that httomo does more preprocessing work, since there is a large increase in execution time when turning it on. Whereas in the case of Nabu, the runtime doesn’t change much. It would be interesting to assess the preprocessing qualitatively, and compare the performance of the algorithmic steps individually.
• While previously we have shown very significant advantage of the LPRec method versus the FBP in httomo, the above results do not reflect that. In this case, LPRec is only slightly faster than FBP. It would be interesting to re-run the same experiment on newer GPUs.
• In the multi-GPU benchmarks, Nabu shows a significant speed advantage over httomo. We suspect that this difference is not the result of difference in the implementation quality of the FBP compute kernels, but instead a result of architectural differences. A significant difference is that httomo spawns a process for each GPU, and coordinates the processes via MPI. This architecture is suitable for multi-node operation as well. Conversely, Nabu doesn’t support multi-node, and all GPUs are fed from a single process. Each process httomo launches executes serially, meaning that loading the data from disk blocks compute. In Nabu (and also TomoCuPy), a task-based approach is used, that achieves parallel load/store and compute. Further investigation is required.

Acknowledgements

We acknowledge the Digital Government Development and Project Management Ltd. for awarding us access to the Komondor HPC facility based in Hungary.