BenchIT

As the time for development of new hard- and software decreases, the variety of available system components for HPC and advanced server systems increases. The result is a wide variety of available computer systems. This variety does, however, make the choice of which system will fit to a given application more difficult.
Common benchmarks provide a useful set of approaches symplifying the choice to a set of scores. Multi-experiment approaches, often as part of complex system analysis and system tuning tools, extend this view significantly. Parameter studies offer the possibility to separate different system characteristics.

Summary

The BenchIT-project approaches a performance analysis of a given computer system in the following way: It runs an extendable set of small measurements (kernels) to produce expressive results in a short period of time. The measurement environment is prepared to deal with time-based measurements and measurements based on the evaluation of built-in hardware performance counters. It is possible to correlate a set of peculiarities with respect to one variable parameter in one measurement run.

One major design goal for the BenchIT measurement suite is a high portability among the different UNIX derivates. The system detection and measurement setup (available compilers and libraries) is done mostly automatically using a set of shell scripts. The only prompt requiring user input regards information about the system architecture to include it in the generated result files enabling the results to be explainable. The compilation, linking and execution of the measurement kernels is also handled automatically using shell scripts and the previously generated system setup. This process can be interrupted after the linking stage to allow for cross-compilation platforms. The produced binaries can then be transferred to the target system where the measurements will be run. The execution of the various measurement kernels is driven by one main program providing an interface to the kernels. It is possible to execute all kernels, a subset of kernels or just a single kernel. Furthermore it is possible to reduce the amount of time available for one measurement kernel as well as the degree of reliability of the measurement to produce results faster.

BenchIT is designed to be extendable by measurement kernels contributed by external users.

ZIH Contact

Daniel Hackenberg

Additional Links

• BenchIT-Wiki
• BenchIT-Repository

Selected Publications

• Markus Velten,Robert Schöne,Thomas Ilsche,Daniel Hackenberg, "Memory Performance of AMD EPYC Rome and Intel Cascade Lake SP Server Processors" In:
  International Conference on Performance Engineering (ICPE), 2022. DOI: 10.1145/3489525.3511689
• Daniel Molka, Robert Schöne, Daniel Hackenberg, Wolfgang E. Nagel. "Detecting Memory-Boundedness with Hardware Performance Counters", In: International Conference on Performance Engineering (ICPE), 2017. DOI: 10.1145/3030207.3030223
• Daniel Molka, Daniel Hackenberg, Robert Schöne, Wolfgang E. Nagel “Cache Coherence Protocol and Memory Performance of the Intel Haswell-EP Architecture”, Proceedings of the 44th International Conference on Parallel Processing (ICPP'15), IEEE, 2015
• Daniel Molka, Daniel Hackenberg, Robert Schöne, “Main Memory and Cache Performance of Intel Sandy Bridge and AMD Bulldozer”, In Proceedings of the 2014 ACM SIGPLAN workshop on Memory Systems Performance and Correctness (MSPC'14), ACM, 2014
• Michael Kluge, Stephen Simms, Thomas William, Robert Henschel, Andy Georgi, Christian Meyer, Matthias S. Mueller, Craig A. Stewart, Wolfgang Wünsch, Wolfgang E. Nagel, Performance and quality of service of data and video movement over a 100 Gbps testbed, Future Generation Computer Systems, Volume 29, Issue 1, January 2013, Pages 230-240, ISSN 0167-739X, http://dx.doi.org/10.1016/j.future.2012.05.028.
• R.Schöne, D. Hackenberg, D. Molka, “Memory performance at reduced CPU clock speeds: an analysis of current x86_64 processors”, In Proceedings of the 2012 USENIX conference on Power-Aware Computing and Systems (HotPower'12), October 7, 2012, Hollywood, USA
• R.Schöne, D. Hackenberg, D. Molka, “Simultaneous Multithreading on x86_64 Systems: An Energy Efficiency Evaluation”, In Procedings of the 4th Workshop on Power-Aware Computing and Systems (HotPower'11), October 23-26, 2011, Cascais, Portugal
• Andy Georgi , Thomas William and Wolfgang E. Nagel, Synthetische Lasttests auf dem 100-Gigabit-Testbed zwischen der TU Dresden und der TU Bergakademie Freiberg, Gesellschaft für Informatik, Bonn, ISBN 978-3-88579-281-9
• D. Molka, R. Schöne, D. Hackenberg, M. S. Müller, “Memory Performance and SPEC OpenMP Scalability on Quad-Socket x86 64 Systems”, In Procedings of the 11th International Conference on Algorithms and Architectures for Parallel Processing (ICA3PP'11), October 24-26 2011, Melbourne, Australia
• D. Molka, D. Hackenberg, R. Schöne and M. S. Müller, “Characterizing the Energy Consumption of Data Transfers and Arithmetic Operations on x86-64 Processors”, In Proceedings of the 1st International Green Computing Conference (IGCC'10), pages 123-133, IEEE, 2010
• D. Hackenberg, D. Molka and W. E. Nagel, “Comparing Cache Architectures and Coherency Protocols on x86-64 Multicore SMP Systems”, In Proceedings of the 42nd International Symposium on Microarchitecture (MICRO'09), pages 413-422, ACM, 2009
• D. Molka, D. Hackenberg, R. Schöne and M. S. Müller, “Memory Performance and Cache Coherency Effects on an Intel Nehalem Multiprocessor System”, In Proceedings of the 18th International Conference on Parallel Architectures and Compilation Techniques (PACT'09), pages 261-270, IEEE, 2009
• Robert Schöne, Wolfgang E. Nagel and Stefan Pflüger, Analyzing Cache Bandwidth on the Intel Core 2 Architecture, In proceeding of: Parallel Computing: Architectures, Algorithms and Applications, ParCo 2007, Forschungszentrum Jülich and RWTH Aachen University, Germany, 4-7 September 2007, ISBN 158603796X, 978-1-58603-796-3
• Daniel Hackenberg, Robert Schöne, Wolfgang E Nagel, Stefan Pflüger, Optimizing OpenMP parallelized DGEMM calls on SGI Altix 3700, Lecture Notes in Computer Science for Euro-Par 2006 Parallel Processing, pages 145-154, Januar 01 2006, http://dx.doi.org/10.1007/11823285_15
• Robert Schöne, Guido Juckeland, Wolfgang E. Nagel, Stefan Pflüger and Robert Wloch, Performance Comparison and Optimization: Case Studies using BenchIT, In proceeding of: Parallel Computing: Architectures, Algorithms and Applications, ParCo 2006, pages 877-884, 2006
• Guido Juckeland, Michael Kluge, Wolfgang E. Nagel and Stefan Pflüger, Performance analysis with BenchIT: portable, flexible, easy to use, in: 1st International Conference on Quantitative Evaluation of Systems (QEST 2004), Proceedings, pages 320--321, IEEE Computer Society, 2004
• Guido Juckeland, S. Borner, Michael Kluge, S. Kolling, Wolfgang E. Nagel, Stefan Pflüger and H. Roding, BenchIT - Performance Measurements and Comparison for Scientific Applications, In proceeding of: Parallel Computing: Architectures, Algorithms and Applications, ParCo 2003, pages 501-508, 2003