LAM/MPI logo

LAM FAQ: Interoperable MPI (IMPI)

  |   Home   |   Download   |   Documentation   |   FAQ   |   all just the FAQ
Table of contents:
  1. What is Interoperable MPI (IMPI)?
  2. What is an IMPI server?
  3. What is an IMPI client?
  4. What is an IMPI host?
  5. What is an IMPI proc?
  6. What MPI implementations currently support IMPI?
  7. How much of IMPI does LAM support?
  8. How do I configure/compile support for IMPI in LAM/MPI?
  9. How do I run an IMPI job in LAM?
  10. Can the IMPI version of LAM be used to run regular (i.e., non-IMPI) MPI jobs?
  11. What is this impid entry that I see in the output of mpitask?
  12. Do I need to modify my MPI program to make it utilize IMPI?
  13. Yow! Message passing performance between MPI implementations is horrible! Why? Can anything be done about it?
  14. Where can I get more information about IMPI?

[ Return to FAQ ]


1. What is Interoperable MPI (IMPI)?

Interoperable MPI (IMPI) is a standard created by a forum led by the National Institute of Standards and Technology (NIST). IMPI defines a protocol by which vendor implementations of MPI can speak to one another in a single MPI job.

This is accomplished via a server that collects and distributes meta-information among different MPI implementations, thus allowing them to establish communcations for MPI calls. A typical scenario for running an IMPI job across multiple MPI implementations could be:

  • start an IMPI server
  • use the impirun command to launch a job on one computing resource (a cluster of Sun's, for example)
  • use the impirun command to launch a job on another computing resource (a Beowulf cluster, for example)

[ Top of page | Return to FAQ ]


2. What is an IMPI server?

An IMPI server is a rendezvous point for different MPI implementations to first meet and exchange meta data about the job that is about to be run. An IMPI server only lasts for the duration of one IMPI job.

The IMPI server is not tied to any particular MPI implementation -- it actually has nothing to do with MPI at all. Indiana University has a freeware IMPI server available at the following URL:

http://www.osl.iu.edu/research/impi/

This package (or some other IMPI server) must be installed in order to be able to run IMPI jobs with LAM.

LAM/MPI will probably include the IMPI server someday, just as it includes Argonne National Lab's ROMIO package. We just haven't gotten to it yet. :-)

[ Top of page | Return to FAQ ]


3. What is an IMPI client?

IMPI clients can be thought of as a representative process of each of the MPI implentations that will be running an IMPI job. In the case of LAM's implementation, an IMPI client is a special process named the impid. There will be one impid per LAM cluster running the IMPI job.

An IMPI job can have a maximum of 32 IMPI clients.

[ Top of page | Return to FAQ ]


4. What is an IMPI host?

IMPI hosts are representative processes of an IMPI client; the exact definition of an IMPI host is implementation dependant. There may be one or more IMPI hosts per IMPI client. The rationale behind having potentially multiple IMPI hosts is that there could be one IMPI host per machine in a cluster of SMPs.

In the case of LAM, there is only one host per client. Actually, the IMPI client and IMPI host are both combined into a single process named impid. The impid will be launched automatically when an IMPI job is run.

[ Top of page | Return to FAQ ]


5. What is an IMPI proc?

An IMPI proc is an individual user process - a rank in MPI_COMM_WORLD.

So an IMPI job is composed of 1-32 IMPI clients, each of which will have one or more IMPI hosts, each of which will have one or more IMPI procs.

[ Top of page | Return to FAQ ]


6. What MPI implementations currently support IMPI?

Hewlett Packard and MPI Software Technologies announced support for IMPI at the SC'2000 conference. Cool demos were presented showing full interoperability between LAM/MPI, HP/MPI, and MPIPro. HP and MST expect to have shipping IMPI products in 1Q 2001. However, it should be noted that many vendors were involved in the standardization process, including Fujitsu, Hewlett Packard, NEC, Silicon Graphics, and Sun Microsystems. You should contact your vendor representative to learn the status of their implementation.

One use of IMPI can be to join physically distant LAM/MPI clusters. In this way, there will only be one high-latency channel open between the two clusters (instead of many).

[ Top of page | Return to FAQ ]


7. How much of IMPI does LAM support?
Applies to LAM 7.0 and above

The current level of support for IMPI in LAM/MPI includes all of the point-to-point MPI functions (all modes of sending and receiving). MPI_PROBE and MPI_IPROBE are not supported. With the exception of MPI_BARRIER, which is necessary to complete the call to MPI_FINALIZE, MPI_BCAST, MPI_GATHER, and MPI_ALLGATHER, no other collectives are supported (this includes the functions for creating communicators).

The color attributes on communicators are supported; they can be used by each rank telling it which IMPI client and host it resides on. This can be used to identify high latency links between MPI implementations.

[ Top of page | Return to FAQ ]


8. How do I configure/compile support for IMPI in LAM/MPI?
Applies to LAM 7.0 and above

When invoking the ./configure in the top level LAM 7.1.1 directory, the default is to build the IMPI support into LAM/MPI. Currently, compiling IMPI requires a C++ compiler that includes support for the Standard Template Library (STL). configure will abort if it cannot find a C++ compiler, or finds a C++ compiler that does not have the STL.

If you do not have an STL-enabled C++ compiler, we suggest the latest version of the GNU C and C++ compilers as a freeware solution. These can be downloaded from the GNU webpage.

[ Top of page | Return to FAQ ]


9. How do I run an IMPI job in LAM?
Applies to LAM 7.0 and above

Launch the IMPI server on a machine. Start up the LAM/MPI client by issuing the impirun with the -client flag and the output from the IMPI server command. impirun takes all the same flags as mpirun.

The following is an example of a shell script which could be used to launch and IMPI job. It assumes the use of the Notre Dame IMPI Server package:

#!/bin/csh

-- Launch the IMPI server and capture the host and port to pass to clients

set hostport='impi-server -server 2 -fork'

-- Launch client 0 on the host bar

rsh bar impirun -client 0 $hostport -np 8 a.out &

-- Launch client 1 on the host baz

rsh baz impirun -client 1 $hostport -np 4 b.out &

wait

[ Top of page | Return to FAQ ]


10. Can the IMPI version of LAM be used to run regular (i.e., non-IMPI) MPI jobs?
Applies to LAM 7.0 and above

Of course. The IMPI extensions in LAM/MPI are only activated if you specifically run an IMPI job (with the -client option to impirun).

If you do not supply the -client option to mpirun, you are effectively using the 6.3 series of LAM/MPI.

[ Top of page | Return to FAQ ]


11. What is this impid entry that I see in the output of mpitask?
Applies to LAM 7.0 and above

The impid is the IMPI daemon that LAM/MPI uses for inter-MPI implementation communication. It is a similar concept to the lamd, which should be familiar to long-time users of LAM/MPI.

The impid is both the IMPI host and the IMPI client for the LAM/MPI cluster, meaning that all traffic that is going from the local LAM/MPI to a remote MPI implementation will be routed through the impid.

The impid will die when MPI_FINALIZE is called on all ranks in the local cluster, or can be killed by lamclean if a process exits abnormally.

[ Top of page | Return to FAQ ]


12. Do I need to modify my MPI program to make it utilize IMPI?
Applies to LAM 7.0 and above

Yes and no. All MPI functions still have the same semantics as described in the MPI standard. So technically, no - you do not need to modify your MPI source code to have it run with IMPI.

However, once an IMPI job starts, MPI_COMM_WORLD spans multiple MPI implementations. Thus, the inter-implementation communication will likely be slower than intra-implementation communication because of the extra overhead of being relayed through IMPI hosts. In addition, it is possible that the different IMPI hosts will be physically distant from each another, thereby increasing the latency of messages between source and destination.

Programmers will need to make decisions in their code that minimize the communcation between these physically distanced machines, or suffer performance penalties associated with the extra communication. The design of IMPI imposes a 2-level hierarchy of communication performance; it is to the programmer's advantage to recognize this hierarchy and code appropriately.

[ Top of page | Return to FAQ ]


13. Yow! Message passing performance between MPI implementations is horrible! Why? Can anything be done about it?
Applies to LAM 7.0 and above

Unfortunately, this is by design. All inter-MPI implementation messages must go through the impid, so each message travels are an average of 3 hops:

  • source rank to theimpid
  • impid to remote IMPI host
  • remote IMPI host to destination rank

The main goal of IMPI is not to directly achieve high performance through inter-implementation links - the goal of IMPI is to connect multiple highly-tuned implementations of MPI such that multiple "pools" of computational resources can be used in a single job. That is, well-written user programs will minimize this inter-MPI implementation communication, and maximize "local" computation/communication within each implementation.

"It's not remarkable that elephants can't dance well; it's remarkable that they can dance at all."

[ Top of page | Return to FAQ ]


14. Where can I get more information about IMPI?
Applies to LAM 7.0 and above

The National Institute of Standards and Technology headed the standardization effort. They maintain the IMPI web site. The IMPI standard and errata can be downloaded from their site.

[ Top of page | Return to FAQ ]