Multi-protocol active messages on a cluster of SMP's

Clusters of multiprocessors, or Clumps, promise to be the supercomputers of the fu- ture, but obtaining high performance on these architectures requires an understanding of interactions between the multiple levels of interconnection. In this paper, we present the rst multi-protocol implemen tation of a lightweight message layer|a v ersion of Ac- tive Messages-II running on a cluster of Sun En terprise 5000 servers connected with Myrinet. This researc h brings together several pieces of high-performance interconnec- tion technology: bus backplanes for symmetric m ultiprocessors, low-latency networks for connections between machines, and simple, user-level primitives for comm unication. The paper describes the shared memory message-passing protocol and analyzes the multi-protocol implemen tation with both microbenchmarks and Split-C applications. Three aspects of the comm unication layer are critical to performance: The overhead of cache-coherence mechanisms, the method of managing concurren t access, and the cost of accessing state with the slower protocol. Through the use of an adaptive polling strategy, the multi-protocol implemen tation limits performance in teractions between the protocols, delivering up to 160 MB/s of bandwidth with 3.6 microsecond end-to-end latency. Applications within an SMP benefit from this fast comm unication, running up to 75% faster than on a network of uniprocessor workstations. Applications running on the entire Clump are limited b y the balance of NIC's to processors in our system, and are typically slower than on the NOW. These results illustrate sev eral potential pitfalls for the Clumps architecture.