Parallel Programming and Heterogeneous Computing 2021
Prof. Dr. Andreas Polze
Note: First meeting on Wednesday, 14.04.2021. No meetings on Mondays.
In this lecture you will learn theoretical and practical skills necessary to develop parallel software and target heterogeneous hardware.
Since the very beginning of computers, processors were build with ever-increasing clock frequencies and instruction-level optimizations for faster serial code execution, such as ILP, caches, or speculative engines. Software developers and industry got used to the fact that applications get faster by just exchanging the underlying hardware. For several years now, these rules are proven to apply no longer. Moore's law about the ever-increasing number of transistors per die may still be valid, but power consumption limits stall the continued increase of clock frequencies. Due to this development, serial execution performance no longer improves automatically with the next processor generation.
In the many-core era that happens now, additional transistors are used not to speed up serial code paths, but to offer multiple execution engines (cores) per processor. This changes every desktop-, server-, or even mobile system into a parallel computer. The exploitation of additional transistors is therefore now the responsibility of software, which makes parallel programming a mandatory approach for all software with scalability demands.
The following topics and technologies will be covered:
- Theory: Flynn's Taxonomy, Scalability Laws, Memory Models
- Shared Memory: PThreads, C++11 Threads, Futures, OpenMP, TBB
- Non-Uniform Memory Access: libnuma, PGASUS
- On-Chip Accelerators: SIMD (SSE, AVX, AltiVec), Compression, LLVM-IR
- External Accelerators: GPUs (CUDA), FGPAs (CAPI SNAP/MetalFS)
- Shared Nothing: MPI, CloudCL, Actors (Erlang)
- Profiling and monitoring with Hardware Performance Counters
- Energy-Aware Computing
This lecture will follow a flipped classroom approach, with the following time distribution:
- 75 %: Watch pre-recorded lecture videos
- 25 %: Participate in online meetings
We publish lecture videos by Sunday evenings, so you can watch them in a timeslot to your convenience. As there is no direct feedback channel in this format, please be sure to make notes of questions or comments as you watch.
The online meetings every Wednesday will be used to discuss the material of the preceding lecture videos. Each meeting will begin with a recap session for the lecture material and there is room for questions and comments. You will also receive organizational updates concerning the lecture, hands-on demos and tipps for your assignments.
Apart from the online meetings you can always post questions on GitLab.
All slides and lecture videos will be available in English.
Online meetings are hosted on Zoom and will be held in German, unless English is requested.
|Time||Wednesday, 11:00 ~ 12:30|
To join the meeting, it is not necessary to register a Zoom account. Nevertheless, students of the University of Potsdam can create an account using their email address firstname.lastname@example.org (note the missing hpi). You may use either the web client, desktop client or your telephone to dial in. Please note that the University license does not include toll-free dial-in numbers, so this option might result in some costs.
This preliminary setup may change in the course of the semester if other meeting platforms become available.
As some participants may not be able to attend all meetings, we offer to make local recordings of each meeting. We will not use the Zoom built-in record functionality - which is administratively disabled anyway - but capture audio and video as received by our Zoom clients. Thus you are free to mute your audio and video streams if you do not want to appear in the recording. If you want to ask questions without being recorded, you can give us an indication and we will endeavor to cut your part from the audio track, but keep in mind that this may become infeasible if you talk while others talk as well. All recordings will be accessible with a lecture-specific password that we will send to participants and request you to keep private.
We hope this solution offers an adequate balance between convenience and privacy. If you have suggestions, we are very open for further discussion.
This lecture includes five practical assignments.
Assignments have to be solved in teams of three persons. The oral exam admittance is granted, if you achieve at least 50% in each assignment. We may provide one non-mandatory assignment at the end of the semester, that can be used to cancel out one failed assignment.
Assignments materials will be available on GitLab at gitlab.hpi.de/parprog21/parprog21. There we will also prepare a way for you to find team partners if you have not found them already.
All assignments are submitted to and validated by our submission system at osm.hpi.de/submit. Submissions failing the validation script will be treated as not handed in.
Week 1 (12.04 ~ 16.04)
Wednesday 14.04.2020 11:00 Online Recap Session
Week 2 (19.04 ~ 23.04)
Week 3 (26.04 ~ 30.04)
Week 4 (03.05 ~ 07.05)
Week 5 (10.05 ~ 14.05)
Wednesday 12.05.2021 11:00 Online Recap Session
Week 6 (17.05 ~ 21.05)
Non-Uniform Memory Access
Wednesday 19.05.2021 11:00 Online Recap Session
Week 7 (24.05 ~ 28.05)
Unit C: Accelerators
Integrated SIMD Accelerators
Wednesday 26.05.2021 11:00 Online Recap Session
Week 8 (31.05 ~ 04.06)
Accelerators: GPUs and CUDA
Wednesday 02.06.2021 11:00 Online Recap Session
Week 9 (07.06 ~ 11.06)
Wednesday 09.06.2021 11:00 Online Recap Session
Week 10 (14.06 ~ 18.06)
Unit D: Shared-Nothing Systems
Wednesday 16.06.2021 11:00 Online Recap Session
Week 11 (21.06 ~ 25.06)
Wednesday 23.06.2021 11:00 Online Recap Session
Week 12 (28.06 ~ 02.07)
Unit E: Energy-Aware Computing (Beyond Performance Centric Optimization...)
Performance vs. Energy Heuristics
Wednesday 30.06.2021 11:00 Online Recap Session
Week 13 (05.07 ~ 09.07)
Unit F: Outlook
Wednesday 07.07.2021 11:00 Online Recap Session
Week 14 (12.07 ~ 16.07)
Wednesday 14.07.2021 11:00 Online Recap Session
Week 15 (19.07 ~ 23.07)
Summary: Ask us your most urgent questions
Wednesday 21.07.2021 11:00 Online Recap Session