Parallel Programming and Heterogeneous Computing 2021

Prof. Dr. Andreas Polze
Sven Köhler, Lukas Wenzel, Max Plauth

Note: First meeting on Wednesday, 14.04.2021. No meetings on Mondays.

In this lecture you will learn theoretical and practical skills necessary to develop parallel software and target heterogeneous hardware.

Since the very beginning of computers, processors were build with ever-increasing clock frequencies and instruction-level optimizations for faster serial code execution, such as ILP, caches, or speculative engines. Software developers and industry got used to the fact that applications get faster by just exchanging the underlying hardware. For several years now, these rules are proven to apply no longer. Moore's law about the ever-increasing number of transistors per die may still be valid, but power consumption limits stall the continued increase of clock frequencies. Due to this development, serial execution performance no longer improves automatically with the next processor generation.

In the many-core era that happens now, additional transistors are used not to speed up serial code paths, but to offer multiple execution engines (cores) per processor. This changes every desktop-, server-, or even mobile system into a parallel computer. The exploitation of additional transistors is therefore now the responsibility of software, which makes parallel programming a mandatory approach for all software with scalability demands.

The following topics and technologies will be covered:

Theory: Flynn's Taxonomy, Scalability Laws, Memory Models
Shared Memory: PThreads, C++11 Threads, Futures, OpenMP, TBB
Non-Uniform Memory Access: libnuma, PGASUS
On-Chip Accelerators: SIMD (SSE, AVX, AltiVec), Compression, LLVM-IR
External Accelerators: GPUs (CUDA), FGPAs (CAPI SNAP/MetalFS)
Shared Nothing: MPI, CloudCL, Actors (Erlang)
Profiling and monitoring with Hardware Performance Counters
Energy-Aware Computing

Lecture

This lecture will follow a flipped classroom approach, with the following time distribution:

75 %: Watch pre-recorded lecture videos
25 %: Participate in online meetings

We publish lecture videos by Sunday evenings, so you can watch them in a timeslot to your convenience. As there is no direct feedback channel in this format, please be sure to make notes of questions or comments as you watch.

The online meetings every Wednesday will be used to discuss the material of the preceding lecture videos. Each meeting will begin with a recap session for the lecture material and there is room for questions and comments. You will also receive organizational updates concerning the lecture, hands-on demos and tipps for your assignments.

Apart from the online meetings you can always post questions on GitLab.

All slides and lecture videos will be available in English.

Meetings

Online meetings are hosted on Zoom and will be held in German, unless English is requested.

Time	Wednesday, 11:00 ~ 12:30
Meeting ID	668-3017-5448
Password	75226074
Link	https://uni-potsdam.zoom.us/j/66830175448
Mobile	+496971049922,,66830175448#,,,,*75226074#
Mobile	+493056795800,,66830175448#,,,,*75226074#
SIP	66830175448@fr.zmeu.us
H.323	213.244.140.110

To join the meeting, it is not necessary to register a Zoom account. Nevertheless, students of the University of Potsdam can create an account using their email address first.last@uni-potsdam.de (note the missing hpi). You may use either the web client, desktop client or your telephone to dial in. Please note that the University license does not include toll-free dial-in numbers, so this option might result in some costs.

This preliminary setup may change in the course of the semester if other meeting platforms become available.

As some participants may not be able to attend all meetings, we offer to make local recordings of each meeting. We will not use the Zoom built-in record functionality - which is administratively disabled anyway - but capture audio and video as received by our Zoom clients. Thus you are free to mute your audio and video streams if you do not want to appear in the recording. If you want to ask questions without being recorded, you can give us an indication and we will endeavor to cut your part from the audio track, but keep in mind that this may become infeasible if you talk while others talk as well. All recordings will be accessible with a lecture-specific password that we will send to participants and request you to keep private.

We hope this solution offers an adequate balance between convenience and privacy. If you have suggestions, we are very open for further discussion.

Assignments

This lecture includes five practical assignments.

Assignments have to be solved in teams of three persons. The oral exam admittance is granted, if you achieve at least 50% in each assignment. We may provide one non-mandatory assignment at the end of the semester, that can be used to cancel out one failed assignment.

Assignments materials will be available on GitLab at gitlab.hpi.de/parprog21/parprog21. There we will also prepare a way for you to find team partners if you have not found them already.

All assignments are submitted to and validated by our submission system at osm.hpi.de/submit. Submissions failing the validation script will be treated as not handed in.

Parallel Programming and Heterogeneous Computing 2021

Lecture

Meetings

Assignments

Schedule