Benchmarking in the Data Center: Expanding to the Cloud

Workshop held in conjunction with ICPE 2025: the 15th ACM/SPEC International Conference on Performance Engineering 2025

Workshop Scope

High performance computing (HPC) is no longer confined to universities and national research laboratories, it is increasingly used in industry and in the cloud. Education of users also needs to take this into account. Users need to be able to evaluate what benefits HPC can bring to their companies, what type of computational resources (e.g. multi-, many-core CPUs, GPUs, hybrid systems) would be best for their workloads and how they can evaluate what they should pay for these resources. Another issue that arises in shared computing environments is privacy: in commercial HPC environments, data produced and software used typically has commercial value, and hence needs to be protected.

Recent general adoption of machine learning has motivated migration of HPC workloads to cloud data centers, and there is a growing interest by the community on performance evaluation in this area, especially for end-to-end workflows. In addition to traditional performance benchmarking and high performance system evaluation (including absolute performance, energy efficiency), as well as configuration optimizations, this workshop will discuss issues that are of particular importance in commercial HPC. Benchmarking has typically involved running specific workloads that are reflective of typical HPC workloads, yet with growing diversity of workloads, theoretical performance modeling is also of interest to allow for performance prediction given a minimal set of measurements. The workshop will be composed of submitted papers, invited talks and a panel composed of representatives from industry.

Submission

We invite novel, unpublished research paper submission within the scope of this workshop. Paper submission topics include, but are not limited to, the following areas:

Authors are invited to submit work as regular paper (up to 8 pages including references). All papers must be prepared in ACM Primary Article Template format: proceedings template. The submitted work shall be in the English language. Inclusion of artefact evaluation/description appendix is encouraged and does not count towards the page limit.

Submitted papers will be peer-reviewed by the technical program committee (TPC). Review of supplementary material is at the discretion of the reviewers; papers must be complete and self-contained. We will be following single blind review process.

All accepted papers will be indexed and published in the ACM Digital Library after the workshop as part of the ICPE’25’s workshop track.

Workshop submission site: https://easychair.org/conferences/?conf=bid2025

Schedule

Paper submission deadline: 01. February 2025 (23:59 AoE)
Author notification: 15. February 2025
Camera ready deadline: 26. February 2025
Workshop date: 5. May 5th 2025 from 2 PM to 5:30 PM EDT at Room A in Second Student Centre, York University, 4700 Keele St, Toronto.

Location


Co-located with ICPE 2025

Workshop Program


Zoom link has been emailed to the ICPE registered attendees.

In person location : Room A in Second Student Centre, York University, 4700 Keele St, Toronto.

Please refer to the full ICPE programme for other co-located events.
Time
(EDT Toronto; UTC−04:00)
Title Location
02:00 PM - 02:15 PM Welcome remarks
Awais Khan, Oak Ridge National Laboratory, USA
Session 1
[Room A]
02:15 PM - 02:45 PM
Invited Talk 1
Speaker: Alok Mishra, Hewlett Packard Enterprise (HPE), USA
HeteroBench: Multi-kernel Benchmarks for Heterogeneous Systems.
02:45 PM - 03:15 PM
Invited Talk 2
Speaker: Manoj Nambiar, Tata Consultancy Services Research, INDIA
Benchmarking with objectives
03:15 PM - 03:30 PM
Open Questions / Discussions
03:30 PM - 04:00 PM Coffee Break
04:00 PM - 04:30 PM
Paper 1
Authors : Arjun Parab, Amir Raoofy, Leon Spörl, Stefan Dimitrov, Matthew Tovey, Josef Weidendorfer
Affiliation : Leibniz Supercomputing Centre, Germany
Title: AutoBench: A Holistic Platform for Automated and Reproducible Benchmarking in HPC Testbeds
Session 2
[Room A]
04:30 PM - 05:00 PM
Paper 2
Authors : Christian von Elm, Robert Schöne
Affiliation : Technische Universität Dresden, Germany
Title: Performance Tools for the NEC SX-Aurora Tsubasa
05:00 PM - 05:10 PM
Concluding Remarks
Awais Khan, Oak Ridge National Laboratory, USA

Registration

ICPE 2025 Registration

Attending

We will support presenters with travel restrictions by setting up a way to present remotely. However, regular registration fees, as stated on ICPE website, will still apply.

Workshop agenda will be updated soon!

Organizing Committee

Awais Khan ( Oak Ridge National Laboratory, USA)

Kaushik Velusamy ( Argonne National Laboratory, USA)

Contact: chairs2025(at)parallel.computer

Program Committee

Joseph Schuchart, University of Tennessee

Ahmad Maroof Karimi, Oak Ridge National Lab

Youngjae Kim, Sogang University

Swapna Raj, NVIDIA

Ravi Reddy, UCD

Filippo Spiga, NVIDIA

Aleksandar Ilic, Universidade de Lisboa

Wei-Chen Lin, University of Bristol

Mubarak Ojewale, CISTER Research Centre, ISEP

Advisory Committee

Samar Aseeri (King Abdullah University of Science and Technology)

Juan (Jenny) Chen (National University of Defense Technology, China)

Benson Muite (Kichakato Kizito)

Kevin A. Brown (Argonne National Laboratory, US)