Introduction to Parallel and Distributed Computing (SS 2010)

Section outline

• Select section General

  General

  Collapse all Expand all

  Introduction to Parallel and Distributed Computing (326.603, SS 2010)
  

  ---

  Time: Monday, 8:30-10:00, Room: T 642, Start: March 1, 2010
  

  ---

  The efficient application of parallel and distributed systems (multi-processors and computer networks) is nowadays an important task for computer scientists and mathematicians. The goal of this course is to provide an integrated view of the various facets of software development on such systems: hardware architectures, programming languages and models, software development tools, software engineering concepts and design patterns, performance modeling and analysis, experimenting and measuring.
  

  Class presentation will be accompanied by hands-on experience on an SGI Altix 4700 distributed shared memory multiprocessor using the following programming models:
  

  • Auto-parallelization and OpenMP (Intel C/C++ or Fortran),
    
  • Java networking and multi-threading API,
  • Message Programming Interface (MPI-C/C++ or Fortran).

  Some background in programming in C/C++ and/or Fortran and/or Java is assumed. Course grades will be based on a couple of programming exercises.
  

  ---

  To take part in the course, you have to enrol in the KUSSS system. Since the exercises will be submitted via Moodle, you also have to login in Moodle and register as a course participant. You will then also receive per email all messages posted in the News forum.
  
  

  • Select activity News forum

    

    News forum
• Select section Topic 1

  Topic 1

  Topics
  

  • Introduction
  • The SGI Altix System
  • Semi-Automatic Parallelization
  • Performance of Parallel Programs
  • Multi-Threaded Client/Server Programming
    
  • Parallel Program Design
    
  • The Message Passing Interface MPI
  • Distributed Memory Algorithms

  Literature
  
  

  Ian Foster: Designing and Building Parallel Programs -- Concepts and Tools for Parallel Software Engineering, Addison Wesley, Reading, MA, 1995. Amazon page.
  Michael J. Quinn: Parallel Programming in C with with MPI and OpenMP, McGraw-Hill, New York, 2003. Source code, Amazon page.
  Kai Hwang: Advanced Computer Architecture: Parallelism, Scalability, Programmability, McGraw-Hill, New York, 1993. Amazon page.
  
  

  Restricted Material
  The password to this section is handed out in class.
  
  

  • Select activity How to Use the Parallel Computing Server

    

    How to Use the Parallel Computing Server Page
  • Select activity Informationen zum Compute Server (in German)

    

    Informationen zum Compute Server (in German) Page
  • Select activity Example Programs

    

    Example Programs File
• Select section Topic 2

  Topic 2

  Introduction
  

  A quick cross-section of the course.
  

  • Software and the Concurrency Revolution (restricted)
    
  • Presentation
  • mpimatmult.f
• Select section Topic 3

  Topic 3

  The SGI Altix System
  Architecture and operating system.
  

  • SGI Altix 4700 (datasheet)
  • The SGI Altix 3000 Global Shared Memory Architecture
    
  • SGI Altix Hardware Architecture
  • SGI Altix Linux and Compiling Environment
• Select section Topic 4

  Topic 4

  Auto-Parallelization and OpenMP
  Basic concepts and their application on the Altix; material partially taken from the SGI developer documentation.
  

  • Automatic Parallelism (Compiler Output)
  • Automatic Parallelization Example
  • OpenMP Example
    
  • Linux Application Tuning Guide
  • SGI Altix Applications Development and Optimization (restricted)
    
  • Programming Documentation
    
  • Intel C++ Compiler User and Reference Guide
  • Intel Fortran Compiler User and Reference Guide
  • Using OpenMP to Parallelize a Program
  • Pragma Directives for Parallel Processing
    
  • OpenMP Tutorial
    
• Select section Topic 5

  Topic 5

  Performance of Parallel Programs
  Modeling and analyzing the performance of parallel programs, some material from Chapter 3 of Foster's Designing and Building Parallel Programs.
  

  • Presentation
  • Figures (restricted)
• Select section Topic 6

  Topic 6

  Multi-Threaded Client/Server Programming
  Developing Internet-based parallel programs by simple language/operating system mechanisms.
  

  • Alle Fäden in der Hand (in German)
  • Sockets - The GNU C Library
    
  • Thread-Level Parallelism (Section from "Topics in IRIX Programming")
    
  • Java networking and multi-threading API
  • Numerische Aufgaben mit Java lösen (restricted)
• Select section Topic 7

  Topic 7

  Parallel Program Design
  The engineering of parallel programs based on Chapter 2 of Foster's Designing and Building Parallel Programs.

  • Presentation
  • Figures (restricted)
• Select section Topic 8

  Topic 8

  The Message Passing Interface MPI
  Programming in MPI based on Chapter 8 of Foster's Designing and Building Parallel Programs.
  

  • Presentation (restricted)
  • Message Passing Interface (restricted)
    
  • MPI Home Page
  • SGI Altix MPI - Programming and Tuning
  • Message Passing Toolkit: MPI Programmer's Manual
• Select section Topic 9

  Topic 9

  Distributed Memory Algorithms
  

  Some algorithms for distributed memory processors.
  

  • Presentation
  • Figures (restricted)
• Select section Topic 10

  Topic 10

  Berkeley UPC - Unified Parallel C

  • Language Specification (PDF)
  • Berkeley UPC Tutorial (PDF)
  • Community Tutorials
    
• Select section Topic 11

  Topic 11

  Exercises
  Four programming/benchmarking exercises will be handed out which will be used for grading.

  • Select activity Exercise 2 (May 7)

    

    Exercise 2 (May 7) Assignment
  • Select activity Exercise 3 (May 28)

    

    Exercise 3 (May 28) Assignment
  • Select activity Exercise 1 (April 14)

    

    Exercise 1 (April 14) Assignment
  • Select activity Exercise 4 (June 25)

    

    Exercise 4 (June 25) Assignment