TY - GEN
T1 - Improving MPI Language Support Through Custom Datatype Serialization
AU - Tronge, Jake
AU - Schuchart, Joseph
AU - Dalcin, Lisandro
AU - Pritchard, Howard
N1 - Publisher Copyright:
© 2024 IEEE.
PY - 2024
Y1 - 2024
N2 - Exascale applications are being increasingly written in modern languages such as Python, Julia, C++, and Rust. The Message-Passing Interface (MPI), the de facto standard for parallel computing, only defines interfaces for C and Fortran, languages that are very different from these modern languages, often containing more complex types and representations incompatible with MPI. The existing derived datatype interface is widely used for older applications, but fails to work efficiently for types containing multiple pointers, requiring application-specific initialization, or serialization. Applications written in these languages can still use MPI, but at the cost of complicated address manipulation or high overhead. This work proposes a new datatype interface for MPI giving more control to the application over buffer packing and the wire representation. We built a prototype for this interface, demonstrating it with Rust, Python, and C++, highlighting key concerns of each language and showing the improvements provided.
AB - Exascale applications are being increasingly written in modern languages such as Python, Julia, C++, and Rust. The Message-Passing Interface (MPI), the de facto standard for parallel computing, only defines interfaces for C and Fortran, languages that are very different from these modern languages, often containing more complex types and representations incompatible with MPI. The existing derived datatype interface is widely used for older applications, but fails to work efficiently for types containing multiple pointers, requiring application-specific initialization, or serialization. Applications written in these languages can still use MPI, but at the cost of complicated address manipulation or high overhead. This work proposes a new datatype interface for MPI giving more control to the application over buffer packing and the wire representation. We built a prototype for this interface, demonstrating it with Rust, Python, and C++, highlighting key concerns of each language and showing the improvements provided.
KW - C++
KW - Data serialization
KW - distributed datatypes
KW - Julia
KW - MPI
KW - Python
KW - Rust
UR - https://www.scopus.com/pages/publications/85217164277
U2 - 10.1109/SCW63240.2024.00062
DO - 10.1109/SCW63240.2024.00062
M3 - Conference contribution
AN - SCOPUS:85217164277
T3 - Proceedings of SC 2024-W: Workshops of the International Conference for High Performance Computing, Networking, Storage and Analysis
SP - 414
EP - 424
BT - Proceedings of SC 2024-W
PB - Institute of Electrical and Electronics Engineers Inc.
T2 - 2024 Workshops of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC Workshops 2024
Y2 - 17 November 2024 through 22 November 2024
ER -