Parallel Programming and Heterogeneous Computing 2020

Prof. Dr. Andreas Polze

Max Plauth
Sven Köhler
Lukas Wenzel
Felix Eberhardt

In this lecture you will learn theoretical and practical skills necessary to develop parallel software and target heterogeneous hardware.

Since the very beginning of computers, processors were build with ever-increasing clock frequencies and instruction-level optimizations for faster serial code execution, such as ILP, caches, or speculative engines. Software developers and industry got used to the fact that applications get faster by just exchanging the underlying hardware. For several years now, these rules are proven to apply no longer. Moore's law about the ever-increasing number of transistors per die may still be valid, but power consumption limits stall the continued increase of clock frequencies. Due to this development, serial execution performance no longer improves automatically with the next processor generation.

In the many-core era that happens now, additional transistors are used not to speed up serial code paths, but to offer multiple execution engines (cores) per processor. This changes every desktop-, server-, or even mobile system into a parallel computer. The exploitation of additional transistors is therefore now the responsibility of software, which makes parallel programming a mandatory approach for all software with scalability demands.

The following topics and technologies will be covered:

  • Theory: Flynn's Taxonomy, Scalability Laws, Memory Models
  • Shared Memory: PThreads, C++11 Threads, Futures, OpenMP, TBB
  • Non-Uniform Memory Access: libnuma, PGASUS
  • On-Chip Accelerators: SIMD (SSE, AVX, AltiVec), Compression, LLVM-IR
  • External Accelerators: GPUs (CUDA), FGPAs (CAPI SNAP/MetalFS)
  • Shared Nothing: MPI, CloudCL, Actors (Erlang)
  • Profiling and monitoring with Hardware Performance Counters
  • Energy-Aware Computing

Intro

Download video

Lecture

This lecture will follow a flipped classroom approach, with the following time distribution:

  • 75 %: Watch pre-recorded lecture videos
  • 25 %: Participate in online meetings

We publish lecture videos by Tuesday evenings, so you can watch them in your own time every Wednesday. As there is no direct feedback channel in this format, please be sure to make notes of questions or comments as you watch.

The online meetings every Thursday will be used to discuss the material of the preceeding lecture videos. Each meeting will begin with a recap session for the lecture material and there is room for questions and comments. You will also receive organizational updates concerning the lecture, hands-on demos and tipps for your assignments.

Apart from the online meetings you can always post questions on GitLab.

All slides and lecture videos will be available in English.

Meetings

Online meetings are hosted on Zoom and will be held in German, unless English is requested.

Time Thursdays, 11:00 ~ 12:30
Meeting ID 945-9492-0566
Password 08152020
Link https://uni-potsdam.zoom.us/j/94594920566
Mobile +496950502596,,94594920566#,,#,08152020#
Mobile +496971049922,,94594920566#,,#,08152020#
Phone +49 695 050 2596
Phone +49 69 7104 9922
Phone +49 30 5679 5800
SIP 94594920566@zoomcrc.com
H.323 213.19.144.110

To join the meeting, it is not necessary to register a Zoom account. Nevertheless, students of the University of Potsdam can create an account using their email address first.last@uni-potsdam.de (note the missing hpi). You may use either the web client, desktop client or your telephone to dial in. Please note that the University license does not include toll-free dial-in numbers, so this option might result in some costs.

This preliminary setup may change in the course of the semester if other meeting platforms become available.

As some participants may not be able to attend all meetings, we offer to make local recordings of each meeting. We will not use the Zoom built-in record functionality - which is administratively disabled anyway - but capture audio and video as received by our Zoom clients. Thus you are free to mute your audio and video streams if you do not want to appear in the recording. If you want to ask questions without being recorded, you can give us an indication and we will endeavor to cut your part from the audio track, but keep in mind that this may become infeasible if you talk while others talk as well. All recordings will be accessible with a lecture-specific password that we will send to participants and request you to keep private.

We hope this solution offers an adequate balance between convenience and privacy. If you have suggestions, we are very open for further discussion.

Assignments

This lecture includes five pratical assignments.

Assignments have to be solved in teams of three persons. The oral exam admittance is achieved if 50% of each assignment is solved correctly. We will provide one non-mandatory assignment at the end of the semester, that can be used to cancel out one failed assignment.

Assignments materials will be available on Gitlab at gitlab.hpi.de/parprog20/parprog20. There we will also prepare a way for you to find team partners if you have not found them already.

All assigments are submitted to and validated by our submission system at osm.hpi.de/submit. Submissions failing the validation script will be treated as not handed in.

Schedule

Week 1 (20.04. ~ 24.04.)

  • A0 - Introduction
    video slides

  • Thursday 23.04.2020 11:00
    Open pre-semester meeting for organizational questions
    meeting


Week 2 (27.04. ~ 01.05.)


Week 3 (04.05. ~ 08.05.)


Week 4 (11.05. ~ 15.05.)


Week 5 (18.05. ~ 22.05.)


Week 6 (25.05. ~ 29.05.)


Week 7 (01.06. ~ 05.06.)


Week 8 (08.06. ~ 12.06.)


Week 9 (15.06. ~ 19.06.)


Week 10 (22.06. ~ 26.06.)


Week 11 (29.06. ~ 03.07.)


Week 12 (06.07. ~ 10.07.)


Week 13 (13.07. ~ 17.07.)


Week 14 (20.07. ~ 24.07.)

  • E2 - Summary Ask us your most urgent questions
    slides

  • Thursday 23.07.2020 11:00 Online Summary Session
    meeting recording


Post-Semester Week (27.07. ~ 31.07.)