Homepage of Technische Universität Dresden

Personal tools
Home » ... » Research » Projects » BenchIT
Sections

BenchIT

BenchIT Logo

Motivation

As the time for development of new hard- and software decreases, the variety of available system components for HPC and advanced server systems increases. The result is a wide variety of available computer systems. This variety does, however, make the choice of which system will fit to a given application more difficult.
Common benchmarks provide a useful set of approaches symplifying the choice to a set of scores. Multi-experiment approaches, often as part of complex system analysis and system tuning tools, extend this view significantly. Parameter studies offer the possibility to separate different system characteristics.

Summary

The BenchIT-project approaches a performance analysis of a given computer system in the following way: It runs an extendable set of small measurements (kernels) to produce expressive results in a short period of time. The measurement environment is prepared to deal with time-based measurements and measurements based on the evaluation of built-in hardware performance counters. It is possible to correlate a set of peculiarities with respect to one variable parameter in one measurement run.

One major design goal for the BenchIT measurement suite is a high portability among the different UNIX derivates. The system detection and measurement setup (available compilers and libraries) is done mostly automatically using a set of shell scripts. The only prompt requiring user input regards information about the system architecture to include it in the generated result files enabling the results to be explainable. The compilation, linking and execution of the measurement kernels is also handled automatically using shell scripts and the previously generated system setup. This process can be interrupted after the linking stage to allow for cross-compilation platforms. The produced binaries can then be transferred to the target system where the measurements will be run. The execution of the various measurement kernels is driven by one main program providing an interface to the kernels. It is possible to execute all kernels, a subset of kernels or just a single kernel. Furthermore it is possible to reduce the amount of time available for one measurement kernel as well as the degree of reliability of the measurement to produce results faster.

The measurement results are placed together with all other relevant data concerning the measurement in one result file. To compare these results comfortably, it is possible to upload the results onto a central web-server also holding a PostgreSQL database. While the plain results remain in the uploaded result files, the measurement information (such as architecture and compiler information) is placed into the database for faster access. The database also holds an extended user interface to allow access for different user groups.

Using the means provided by the web server, one can analyze the results alone or in comparison with each other. In order to do this, two general approaches in filtering the available results are offered: Selection by architectural feature or by measurement kernel. The first way allows for a direct comparison of different computer architecture features using a set of measurement kernels. A faster way to plot specific results is the direct selection of a measurement kernel and a selection from the available results for this kernel. The data filtering leads to a point where the measurement data will be displayed using gnuplot and a large set of display options. It is possible to store, modify and export plots.

BenchIT is designed to be extendable by measurement kernels and measurement results contributed by external users. The results are then also available online.

ZIH CONTACT

Thomas William

Additional LINKS

 

Publications

  • Daniel Molka, Daniel Hackenberg, Robert Schöne, Main Memory and Cache Performance of Intel Sandy Bridge and AMD Bulldozer, In Proceedings of the 2014 ACM SIGPLAN workshop on Memory Systems Performance and Correctness (MSPC'14), ACM, 2014, http://dl.acm.org/citation.cfm?doid=2618128.2618129
  • Michael Kluge, Stephen Simms, Thomas William, Robert Henschel, Andy Georgi, Christian Meyer, Matthias S. Mueller, Craig A. Stewart, Wolfgang Wünsch, Wolfgang E. Nagel, Performance and quality of service of data and video movement over a 100 Gbps testbed, Future Generation Computer Systems, Volume 29, Issue 1, January 2013, Pages 230-240, ISSN 0167-739X, http://dx.doi.org/10.1016/j.future.2012.05.028. (http://www.sciencedirect.com/science/article/pii/S0167739X12001380)
  • R.Schöne, D. Hackenberg, D. Molka, Memory performance at reduced CPU clock speeds: an analysis of current x86_64 processors, In Proceedings of the 2012 USENIX conference on Power-Aware Computing and Systems (HotPower'12), October 7, 2012, Hollywood, USA, http://dl.acm.org/citation.cfm?id=2387869.2387878
  • R.Schöne, D. Hackenberg, D. Molka, Simultaneous Multithreading on x86_64 Systems: An Energy Efficiency Evaluation, In Procedings of the 4th Workshop on Power-Aware Computing and Systems (HotPower'11), October 23-26, 2011, Cascais, Portugal, http://dl.acm.org/citation.cfm?doid=2039252.2039262
  • Andy Georgi , Thomas William and Wolfgang E. Nagel, Synthetische Lasttests auf dem 100-Gigabit-Testbed zwischen der TU Dresden und der TU Bergakademie Freiberg, Gesellschaft für Informatik, Bonn, ISBN 978-3-88579-281-9
  • D. Molka, R. Schöne, D. Hackenberg, M. S. Müller, Memory Performance and SPEC OpenMP Scalability on Quad-Socket x86 64 Systems, In Procedings of the 11th International Conference on Algorithms and Architectures for Parallel Processing (ICA3PP'11), October 24-26 2011, Melbourne, Australia, http://www.springerlink.com/content/2k52134047538083/
  • D. Molka, D. Hackenberg, R. Schöne and M. S. Müller, Characterizing the Energy Consumption of Data Transfers and Arithmetic Operations on x86-64 Processors, In Proceedings of the 1st International Green Computing Conference (IGCC'10), pages 123-133, IEEE, 2010, http://ieeexplore.ieee.org/search/srchabstract.jsp?tp=&arnumber=5598316
  • D. Hackenberg, D. Molka and W. E. Nagel, Comparing Cache Architectures and Coherency Protocols on x86-64 Multicore SMP Systems, In Proceedings of the 42nd International Symposium on Microarchitecture (MICRO'09), pages 413-422, ACM, 2009, http://portal.acm.org/citation.cfm?id=1669165&dl=GUIDE&coll=GUIDE&CFID=81286494&CFTOKEN=18541064
  • Daniel Hackenberg, Robert Schöne, Wolfgang E Nagel, Stefan Pflüger, Optimizing OpenMP parallelized DGEMM calls on SGI Altix 3700, Lecture Notes in Computer Science for Euro-Par 2006 Parallel Processing, pages 145-154, Januar 01 2006, http://dx.doi.org/10.1007/11823285_15
  • Robert Schöne, Wolfgang E. Nagel and Stefan Pflüger, Analyzing Cache Bandwidth on the Intel Core 2 Architecture, In proceeding of: Parallel Computing: Architectures, Algorithms and Applications, ParCo 2007, Forschungszentrum Jülich and RWTH Aachen University, Germany, 4-7 September 2007, ISBN 158603796X, 978-1-58603-796-3
  • D. Molka, D. Hackenberg, R. Schöne and M. S. Müller, Memory Performance and Cache Coherency Effects on an Intel Nehalem Multiprocessor System, In Proceedings of the 18th International Conference on Parallel Architectures and Compilation Techniques (PACT'09), pages 261-270, IEEE, 2009, http://ieeexplore.ieee.org/search/srchabstract.jsp?tp=&arnumber=5260544
  • Robert Schöne, Guido Juckeland, Wolfgang E. Nagel, Stefan Pflüger and Robert Wloch, Performance Comparison and Optimization: Case Studies using BenchIT, In proceeding of: Parallel Computing: Architectures, Algorithms and Applications, ParCo 2006, pages 877-884, 2006
  • Guido Juckeland, Michael Kluge, Wolfgang E. Nagel and Stefan Pflüger, Performance analysis with BenchIT: portable, flexible, easy to use, in: 1st International Conference on Quantitative Evaluation of Systems (QEST 2004), Proceedings, pages 320--321, IEEE Computer Society, 2004
  • Guido Juckeland, S. Borner, Michael Kluge, S. Kolling, Wolfgang E. Nagel, Stefan Pflüger and H. Roding, BenchIT - Performance Measurements and Comparison for Scientific Applications, In proceeding of: Parallel Computing: Architectures, Algorithms and Applications, ParCo 2003, pages 501-508, 2003

 

Last modified: 06.08.2014 09:42
Author: Daniel Hackenberg



Contact
Service Desk:
Room:
      Nöthnitzer Str. 46
Dep. of Computer Science,  E036
Mo-Fr 08:00-19:00
Phone:
      +49 351 463-40000
Fax:
  +49 351 463-42328
Announcement of Troubles:
  +49 351 463-31888
email iconservicedesk@tu-dresden.de
 
Secretary:
Room: Willers-Bau A 207
Phone:
      +49 351 463-35450
Fax:
  +49 351 463-37773
email iconzih@tu-dresden.de
 

 

Visitor's address:
Willers-Bau, A-Flügel
Zellescher Weg 12-14
 

Postal address:
TU Dresden
Center for Information Services and High Performance Computing (ZIH)
01062 Dresden
Germany

Parcels:
TU Dresden
Center for Information Services and High Performance Computing (ZIH)
Helmholtzstr. 10
01069 Dresden
Germany