High-Performance Computing - Episode 1 - Introducing MPI

hi and welcome to the first episode of the software carpentry lectures on MPI in this episode we'll explain what MPI is and what it's good for as computing tasks get larger and larger whether we want to do bigger and bigger simulations or analyze larger and larger piles of data our needs can quickly exceed the capabilities of any one computer we might want to do bigger problems requiring more memory or storage than is available in one machine we might need the computation to go a lot faster than it would on one computer so we need access to more processors or we might just want to do more computations than this feasible in one computer running enormous parameter sweeps that would take months or years have done sequentially on one desktop computer for instance message-passing is one programming model which is proven very useful for arranging computations on multiple computers in message passing each process has its own data which can't be seen by the other processes when data needs to be communicated the processes send and receive messages to and from one another from these basic communication primitives scientific programmers can build up multi computer computations by the way when multiple computers are used together like this it's common to refer to the separate independent computers as nodes I'll use that term often a given node likely has more than one processor just as your desktop or laptop might MPI is a standard library for message passing it's ubiquitous widely and easily available and quite possibly already installed or whatever computers you have access to it runs on the world's largest supercomputers but it can also run efficiently and usefully on your laptop or desktop nearly every big academic or commercial package for simulation or data analysis that uses multiple nodes simultaneously uses MPI directly or indirectly so if message passing is just sending and receiving messages between computers why do we bother with MPI this just sounds like Network communications and there are already dozens of libraries out there to handle such things why would we need another first the MPI interface is designed and implemented with performance in mind secondly the MPI interface will take advantage of the fastest network transport available to it when sending messages so for instance communicating two different processes within a node MPI we'll use shared memory to send and receive messages rather than Network communications on fast interconnects within a high-performance computer cluster it already knows how to take advantage of transports like InfiniBand or mere net for communications to processes on other nodes and only if all else fails will it use the standard Internet tcp/ip this represents a huge body of networking code for many interfaces and protocols that you don't have to implement yourself MPI enforces other guarantees like guaranteeing messages arrive rather than being lost in requiring retries and that messages arrive in order this enormous ly simplifies programming in the end MPI is designed for multi node technical computing and that means we can spend our time figuring out how to decompose our scientific problem rather than having to worry about Network protocols it is based on the standards process and is the most widely used interface of its type widely and easily available in addition and a particular interest to those of us doing technical computing it comes with specialized routines for collective communications of the sort frequently needed in science or engineering computations the usual message-passing stuff is point-to-point communications where communications is one-to-one one process has some data which sends to another process possibly on another node this is useful since we don't have to worry about network capabilities but there are other modes communication that are frequently needed for parallel computations for instance it's often useful to broadcast data which is a one-to-many operation one process has a piece of data and broadcasts it to many or even all of the other processes a close relative of broadcast is scattered where one process divides values between many others the inverse of scatter is gather in which many processes have different parts of the overall picture which are then brought together to one process finally there is reduction which combines communication and computation many useful sorts of operations finding a global minimum maximum sum or product are fundamentally reduction operations consider doing a global sum of data each process calculates its partial sum and then these are combined into a global sum on one process it's often said that MPI is a very low-level way to program but abstracting away networking details and providing global operations sounds fairly high-level so which is it compared to other networking libraries it's quite high level you don't have to think about transport details and it implements those nice collectives but from a technical computing point of view it's still very low level we're concerned with doing things like fluid simulations are doing bioinformatics using MPI means we have to figure out how to break up the problem amongst multiple computers and write code for each communication that has to occur between them so should we use MPI to do everything no it's probably the best choice for doing a single large computation across similar machines across a reliable network such as a number of desktops in a lab or in a specialized computer cluster but it would be a nightmare to try to use to write a distributed chat client or BitTorrent Network if it were possible to do so at all message passing type approaches can work very well for such systems Erlang is an example of a programming language using message passing that's very good at these sorts of tasks but MPI is not clients constantly connecting and disconnecting introduces a lot of complexity that MPI isn't designed to handle at the other extreme if you have lots of independent computations and you just want to parcel them out to run in parallel as a task farm MPI can work for that but it's quite a big tool to use for a fairly simple task there may be other approaches which are easier so now that we know what MPI is and isn't how would we go about using it MPI is an interface to a library so it consists of function calls and constants that can be used for many programming languages it comes with bindings that is native ways to access these functions and constants for Fortran and for see there are many third parties who have taken these tools and built very useful bindings for other languages - such as Python R and most other languages that scientists and engineers use the basic workflow for using MPI is to build your tool using the libraries and the appropriate bindings and run the resulting program with a utility called MPI exec to see how this works make sure you already have MPI installed if you're using a computer system where other people routinely run multi node jobs it probably already is but you can also run MPI programs on your desktop or laptop quite simply if you don't already have it it's easy enough to install there are two main MPI distributions at the moment open MPI and M pitch - both are excellent open-source implementations of the MPI standard interface there's a friendly rivalry between the two groups ensuring that development proceeds quickly and neither implementation has significant performance advantages over the other for very long both are very good and while they each have their strengths when starting out there's no a priority reason to prefer one over the other so to start whichever one is easiest to install say whichever you can first find binary installers for your operating system for is a fine choice so now that you've got MPI installed let's try testing the installation and running something simple in the Linux or Mac OS X terminal or on Windows with cygwin the hostname command simply prints the name of the host it's run on running it on my laptop it produces output like this we can run hostname with MPI's MPI exec running MPI exec n for hostname gives me the same output four times it should do the same for you so what just happened here running MPI exact - n for host name tells mdpi exec to launch four processes on your computer and in each of these processes to run the hostname program each ran independently in its own environment and as a result we were informed of our computer's hostname four times you can do the same thing with - n 8 or 27 or 93 and you'll get that many outputs for performance reasons though with MPI on any given node we usually run at most as many processes as there are processors of course the real power of MPI is being able to use multiple computers if you have access to other computers that you can log into without passwords say by using SSH keys this is easy to do you create a file with the list of machines to use and specify that list of hosts using the - machine file option to MPI exec here we're launching four processes on a list of 3 hosts MPI exec again lunches four processes this time split up as evenly as it can manage over the three hosts each of these processes then run host name and print their output this is a toy example but actually running host name with MPI exec is a useful way of making sure that MPI is distributing processes across hosts the way you think of it of course host name isn't a real MPI program an actual MPI program would be launched with knowledge of the existence of the other processes and knowing which in the list of processes it is we will learn how to write simple but real MPI programs using Python in our next episode thank you for listening

Channel: softwarecarpentry

Views: 64,884

Rating: 4.9071565 out of 5

Keywords: software carpentry, High-Performance Computing, Introducing MPI, science, programming, software

Id: kHV6wmG35po

Length: 11min 21sec (681 seconds)

Published: Wed Jun 08 2011