1 Parallel Programming Models 49

when an error is detected in subsequent execution without having to restart the

entire program.
3. Advanced input/output constructs will be used for generating “information prod-
ucts.” It will become increasingly impractical to attempt to store the full raw data
from a simulation because the data sets will become prohibitively large. Also,
the data itself, even if visualized, may not be useful in understanding the impli-
cations and consequences of the results. An output layer to process the raw data
may be necessary to generate information products that can be many orders of
magnitude smaller than the basic data values but far more meaningful to the sci-
entist or engineer. Future languages will emphasize high-level information rather
than raw data sets as the principal output content, and the I/O semantics of the
language will reflect this new usage.

1.6 Final Thoughts

The information in this chapter touches only the tip of the iceberg with respect to
the issues of writing parallel programs. Even long-term practitioners fall into the
many pitfalls of developing parallel codes. Overall, writing parallel programs is best
learned by “getting your hands dirty.”
It is important to use the technology needed to get the job done, but it is also
important to think about what changes might come in the future. The software de-
velopment research community is producing new technologies rapidly and some of
these technologies may be useful in high-performance application development. Al-
though implementing object-oriented technology in Fortran 77 is impossible, some
of the object-oriented concepts can build better-structured Fortran 77 codes. For ex-
ample, abstraction and data-hiding are easily implemented with solid APIs for the
functionality required.
What will come in the future? In this book, the chapter on common component
architecture technology [2] discusses how the CCA framework has been used suc-
cessfully to integrate functionality among multiple computational chemistry codes
on parallel platforms. Also, the cross-cutting technologies of aspect-oriented pro-
gramming [22] could change the way in which we construct software infrastructure
for event logging, performance monitoring, or computational steering.
One additional comment is in order. Readers new to parallel computing on clus-
ters might ask which is the best programming model with respect to performance
and scalability. These are only two aspects of the interaction with a programming
model and an application code with many different algorithms. The programming
model also determines the ease of algorithmic development and thus application de-
velopment and maintenance, Asking which is best is similar to asking which pre-
conditioner, which Kyrlov subspace method, or which editor is the best to use. All
programming models have strengths and weaknesses, and the choice is best made by
those actually using the programming model for their particular purpose. MPI offers
the greatest availability, portability, and scalability to large systems. OpenMP offers
very good portability and availability with reasonable scalability on SMP systems.
50 R. A. Kendall et al.

The distributed shared-memory programming models are best when long-term avail-
ability is possible and there is an appropriate match to the algorithms or applications
involved. Clearly, programming models and their associated execution models will
have to evolve to be able to reach sustained petaflops levels of computing, which will
in time move to computational resources known as clusters.
Finally, we will put together a series of examples of “working” code for many
of the programming models discussed in this chapter. These will be designed around
small computational kernels or simple applications in order to illustrate each model.
The examples will be available at the Center for Programming Models for Scalable
Parallel Computing website13 .

Acknowledgments

This work was supported by the Mathematical, Information, and Computational Sci-
ences Division subprogram of the Office of Advanced Scientific Computing Re-
search, Office of Science, U.S. Department of Energy, under Contract W-31-109-
ENG-38 with Argonne National Laboratory and under Contract W-7405-ENG-82 at
Ames Laboratory. The U.S. Government retains for itself, and others acting on its be-
half, a paid-up, non-exclusive, irrevocable worldwide license in said article to repro-
duce, prepare derivative works, distribute copies to the public, and perform publicly
and display publicly, by or on behalf of the Government. We thank the members of
the Center for Programming Models for Scalable Parallel Computing [8] who have
helped us better understand many of the issues of parallel software development and
the associated programming models. We thank Brent Gorda, Angie Kendall, Gail W.
Pieper, Douglas Fuller, and Professor Gary T. Leavens for reviewing the manuscript.
We also thank the book series editors and referees for their many helpful comments.

References
1. Alphaserver SC user guide, 2000. Bristol, Quadrics Supercomputer World Ltd.
2. R. Armstrong, D. Gannon, A. Geist, K. Keahey, S. R. Kohn, L. McInnes, S. R. Parker,
and B. A. Smolinski. Toward a common component architecture for high-performance
scientific computing. In Proceedings of the 8th High Performance Distributed Computing
(HPDC’99), 1999. URL: http://www.cca-forum.org.
3. S. Balay, W. D. Gropp, L. C. McInnes, and B. F. Smith. PETSc users manual. Technical
Report ANL-95/11 - Revision 2.1.0, Argonne National Laboratory, 2001.
4. R. Bariuso and A. Knies. SHMEM’s User’s Guide. SN-2515 Rev. 2.2, Cray Research,
Inc., Eagan, MN, USA, 1994.
5. M. Bull. OpenMP 2.5 and 3.0. In Proceedings of the Workshop on OpenMP Applications
and Tools, WOMPAT 2004, Houston, TX, May 17-18 2004. (Invited talk).
6. P. M. Burton, B. Carruthers, G. S. Fischer, B. H. Johnson, and R. W. Numrich. Converting
the halo-update subroutine in the MET Office unified model to Co-Array Fortran. In

13
At this URL: http://www.pmodels.org/ppde.
 1 Parallel Programming Models 51

W. Zwieflhofer and N. Kreitz, editors, Developments in Teracomputing: Proceedings of

the Ninth ECMWF Workshop on the Use of High Performance Computing in Meteorology,
pp. 177–188. World Scientific Publishing, 2001.
7. J. B. Carter, J. K. Bennett, and W. Zwaenepoel. Implementation and performance of
Munin. In Proceedings of the 13th ACM Symp. on Operating Systems Principles (SOSP-
13), pages 152–164, 1991.
8. Center for Programming Models for Scalable Parallel Computing. URL: http://www.
pmodels.org.
9. R. Chandra, L. Dagum, D. Kohr, D. Maydan, J. McDonald, and R. Menon. Parallel
Programming in OpenMP. Morgan Kaufmann Publishers, San Francisco, CA, 2001.
10. D. Chen, S. Dwarkadas, S. Parthasarathy, E. Pinheiro, and M. L. Scott. Interweave: A
middleware system for distributed shared state. In Languages, Compilers, and Run-Time
Systems for Scalable Computers, pages 207–220, 2000.
11. E. Chow, A. Cleary, and R. Falgout. HYPRE User’s manual, version 1.6.0. Technical
Report UCRL-MA-137155, Lawrence Livermore National Laboratory, Livermore, CA,
1998.
12. D. Clark. OpenMP: A parallel standard for the masses. IEEE Concurrency, 6(1):10–12,
January–March 1998.
13. C. Coarfa, Y. Dotsenko, J. L. Eckhardt, and J. Mellor-Crummey. Co-array Fortran perfor-
mance and potential: An NPB experimental study. In The 16th International Workshop on
Languages and Compilers for Parallel Computing (LCPC 2003), College Station, Texas,
October 2003.
14. Cray Research. Application Programmer’s Library Reference Manual, 2nd edition, Nov.
1995. Publication SR-2165.
15. L. Dagum and R. Menon. OpenMP: An industry standard API for shared-memory pro-
gramming. IEEE Computational Science & Engineering, 5(1):46–55, January–March
1998.
16. S. Dong and G. E. Karniadakis. Dual-level parallelism for deterministic and stochastic
CFD problems. In Proceedings of Supercomputing, SC02, Baltimore, MD, 2002.
17. J. Dongarra, I. Foster, G. Fox, W. D. Gropp, K. Kennedy, L. Torczon, and A. White,
editors. Sourcebook of Parallel Computing. Morgan Kaufmann, 2003.
18. P. F. Dubois. Ten Good Practices In Scientific Programming. Computing in Science &
Engineering, 1(1), January-February 1999.
19. S. Dwarkadas, N. Hardavellas, L. Kontothanassis, R. Nikhil, and R. Stets. Cashmere-
VLM: Remote memory paging for software distributed shared memory. In Proceedings
of the 13th International Parallel Processing Symposium and 10th Symposium on Parallel
and Distributed Processing, pages 153–159. IEEE Computer Society, Apr. 1999.
20. Earth Simulator home page, http://www.es.jamstec.go.jp.
21. T. A. El-Ghazawi, W. W. Carlson, and J. M. Draper. UPC Language Specifica-
tions Version 1.1.1, October 2003. URL: http://www.gwu.edu/∼upc/docs/
upc spec 1.1.1.pdf.
22. T. Elrad, R. E. Filman, and A. Bader. Aspect-Oriented Programming. Communications
of the ACM, 44(10):29–32, October 2001.
23. R. D. Falgout, J. E. Jones, and U. M. Yang. The design and implementation of hypre,
a library of parallel high performance preconditioners. In A. M. Bruaset and A. Tveito,
editors, Numerical Solution of Partial Differential Equations on Parallel Computers, vol-
ume 51 of Lecture Notes in Computational Science and Engineering, pages 267–294.
Springer-Verlag, 2005.
52 R. A. Kendall et al.

24. M. Folk, A. Cheng, and K. Yates. HDF5: A file format and I/O library for high perfor-
mance computing applications. In Proceedings of Supercomputing’99 (CD-ROM). ACM
SIGARCH and IEEE, Nov. 1999.
25. FORTRAN 77 Binding of X3H5 Model for Parallel Programming Constructs. Draft Ver-
sion, ANSI X3H5, 1992.
26. P. C. Forum. PCF Parallel FORTRAN Extensions. FORTRAN Forum, 10(3), September
1991. (Special issue).
27. Global Array Project. URL: http://www.emsl.pnl.gov/docs/global.
28. W. D. Gropp. Learning from the success of MPI. In B. Monien, V. K. Prasanna, and S. Va-
japeyam, editors, High Performance Computing – HiPC 2001, number 2228 in Lecture
Notes in Computer Science, pages 81–92. Springer, Dec. 2001.
29. W. D. Gropp, S. Huss-Lederman, A. Lumsdaine, E. Lusk, B. Nitzberg, W. Saphir, and
M. Snir. MPI—The Complete Reference: Volume 2, The MPI-2 Extensions. MIT Press,
Cambridge, MA, 1998.
30. W. D. Gropp, E. Lusk, and A. Skjellum. Using MPI: Portable Parallel Programming with
the Message Passing Interface, 2nd edition. MIT Press, Cambridge, MA, 1999.
31. W. D. Gropp, E. Lusk, and T. Sterling, editors. Beowulf Cluster Computing with Linux.
MIT Press, 2nd edition, 2003.
32. W. D. Gropp, E. Lusk, and R. Thakur. Using MPI-2: Advanced Features of the Message-
Passing Interface. MIT Press, Cambridge, MA, 1999.
33. R. Hempel and D. W. Walker. The emergence of the MPI message passing standard for
parallel computing. Computer Standards and Interfaces, 21(1):51–62, 1999.
34. High Performance Fortran Forum. High Performance Fortran language specification.
Scientific Programming, 2(1–2):1–170, 1993.
35. J. M. D. Hill, B. McColl, D. C. Stefanescu, M. W. Goudreau, K. Lang, S. B. Rao, T. Suel,
T. Tsantilas, and R. H. Bisseling. BSPlib: The BSP programming library. Parallel Com-
puting, 24(14):1947–1980, Dec. 1998.
36. C. A. R. Hoare. Communicating sequential processes. Communications of the ACM,
21(8):666–677, Aug. 1978.
37. J. Hoeflinger. Towards industry adoption of OpenMP. In Proceedings of the Workshop
on OpenMP Applications and Tools, WOMPAT 2004, Houston, TX, May 17–18 2004.
Invited Talk.
38. F. Hoffman. Writing hybrid MPI/OpenMP code. Linux Magazine, 6(4):44–48, April
2004. URL: http://www.linux-mag.com/2004-04/extreme 01.html.
39. Y. Hu, H. Lu, A. L. Cox, and W. Zwaenepoel. OpenMP for networks of SMPs. In
Proceedings of the 13th International Parallel Processing Symposium, April 1999.
40. P. Hyde. Java Thread Programming. SAMS, 1999.
41. IEEE Standard for Information Technology-Portable Operating System Interface
(POSIX). IEEE Standard No.: 1003.1, 2004.
42. W. Jiang, J. Liu, H.-W. Jin, D. K. Panda, W. D. Gropp, and R. Thakur. High performance
MPI-2 one-sided communication over InfiniBand. Technical Report ANL/MCS-P1119-
0104, Mathematics and Computer Science Division, Argonne National Laboratory, 2004.
43. G. Jost, J. Labarta, and J. Gimenez. What multilevel parallel programs do when you are
not watching: A performance analysis case study comparing MPI/OpenMP, MLP, and
nested OpenMP. In Proceedings of the Workshop on OpenMP Applications and Tools,
WOMPAT 2004, pages 29–40, Houston, TX, May 17-18 2004. (Invited talk).
44. P. Keleher, A. L. Cox, S. Dwarkadas, and W. Zwaenepoel. TreadMarks: Distributed
shared memory on standard workstations and operating systems. In Proceedings of the
Winter 94 Usenix Conference, pages 115–131, January 1994.
 1 Parallel Programming Models 53

45. R. A. Kendall, E. Aprà, D. E. Bernholdt, E. J. Bylaska, M. Dupuis, G. I. Fann, R. J. Har-

rison, J. Ju, J. A. Nichols, J. Nieplocha, T. P. Straatsma, T. L. Windus, and A. T. Wong.
High performance computational chemistry; an overview of NWChem a distributed par-
allel application. Computer Physics Communications, 128:260–283, 2002.
46. M. G. Knepley, R. F. Katz, and B. Smith. Developing a geodynamics simulator with
petsc. In A. M. Bruaset and A. Tveito, editors, Numerical Solution of Partial Differential
Equations on Parallel Computers, volume 51 of Lecture Notes in Computational Science
and Engineering, pages 413–438. Springer-Verlag, 2005.
47. C. Koelbel, D. B. Loveman, R. S. Schreiber, G. L. Steele, and M. E. Zosel. The High
Performance Fortran Handbook. MIT Press, 1994.
48. B. Leasure, editor. PCF Fortran: Language Definitons, Version 3.1. The Parallel Com-
puting Forum, Champaign, IL, 1990.
49. J. Li, W. Liao, A. Choudhary, R. Ross, R. Thakur, W. D. Gropp, R. Latham, A. Siegel,
B. Gallagher, and M. Zingale. Parallel netCDF: A high-performance scientific I/O inter-
face. In Proceedings of SC2003, Nov. 2003.
50. Z. Li, Y. Saad, and M. Sosonkina. pARMS: A parallel version of the algebraic recursive
multilevel solver. Numerical Linear Algebra with Applications, 10:485–509, 2003.
51. R. K. Lie Huang, Barbara Chapman. OpenMP on distributed memory via global arrays.
In Proceedings of Parallel Computing 2003 (ParCo2003), Dresden, Germany, September
2–5 2003.
52. Message Passing Interface Forum. MPI: A Message-Passing Interface standard. Interna-
tional Journal of Supercomputer Applications, 8(3/4):165–414, 1994.
53. Message Passing Interface Forum. MPI2: A Message Passing Interface standard. Inter-
national Journal of High Performance Computing Applications, 12(1–2):1–299, 1998.
54. Message Passing Toolkit: MPI programmer’s manual, document number : 007-3687-010,
2003. Mountain View, CA, Silicon Graphics Inc.
55. Mpi papers. URL: http://www.mcs.anl.gov/mpi/papers.
56. K. Nakajima and H. Okuda. Parallel Iterative Solvers for Unstructured Grids Using and
OpenMP/MPI Hybrid Programming Model for GeoFEM Platfrom on SMP Cluster Ar-
chitectures. Lecture Notes in Computer Science, 2327:437–448, 2002.
57. B. Nichols, D. Buttlar, and J. P. Farrel. Pthreads Programming. O’Reilly & Associates,
Inc, 1996.
58. J. Nieplocha, R. Harrison, M. Krishnan, B. Palmer, , and V. Tipparaju. Combining shared
and distributed memory models: Evolution and recent advancements of the Global Array
Toolkit. In Proceedings of POOHL’2002 workshop of ICS-2002, New York, NY, 2002.
59. J. Nieplocha, R. J. Harrison, and R. J. Littlefield. Global Arrays: A portable “shared
memory” programming model for distributed memory computers. In Proceedings of Su-
percomputing 1994, SC94, pages 340–349, 1994.
60. J. Nieplocha, R. J. Harrison, and R. J. Littlefield. Global Arrays: A nonuniform memory
access programming model for high-performance computers. The Journal of Supercom-
puting, 10:197–220, 1996.
61. R. W. Numrich, J. Reid, and K. Kim. Writing a multigrid solver using Co-Array For-
tran. In B. Kågström, J. Dongarra, E. Elmroth, and J. Waśniewski, editors, Applied Par-
allel Computing: Large Scale Scientific and Industrial Problems, volume 1541 of Lecture
Notes in Computer Science, pages 390–399. Springer, 1998.
62. R. W. Numrich and J. K. Reid. Co-Array Fortran for parallel programming. ACM Fortran
Forum, 17(2):1–31, 1998.
63. OpenMP Architecture Review Board. OpenMP Fortran Application Program Inter-
face, Version 2.0. November 2000. URL: http://www.openmp.org/drupal/
mp-documents/fspec20.pdf.