Simulated performance evaluation of a 6D mesh/iBT interconnect

A new 6D mesh/torus (Tofu) interconnect is implemented in the K Computer for achieving a performance of more than 10 petaflops. As expected, among its many interesting properties, such interconnect has large variations of latencies for communications within a node-group (intra-node-group) or between them (inter-node-group). Inspired by their works while recognizing the superior features of the recently published interlaced bypass torus (iBT) network that can help reduce such variations, we propose a new 6D mesh/iBT interconnect by replacing the 3D torus of Tofu by a 3D iBT. Our new interconnect suppresses the latencies for inter-node-group communications to dramatically improve the overall performance for the entire network. Adapting an industrial standard network simulator NS-3, we study different communication patterns of the added iBT networks under different bypass schemes. We find that the simulated average latencies of our network are 40% less than those of the Tofu interconnect for all tested patterns.