Parallel Programming and Heterogeneous Computing 2020

Prof. Dr. Andreas Polze
Max Plauth, Sven Köhler, Lukas Wenzel, Felix Eberhardt

In this lecture you will learn theoretical and practical skills necessary to develop parallel software and target heterogeneous hardware.

Since the very beginning of computers, processors were build with ever-increasing clock frequencies and instruction-level optimizations for faster serial code execution, such as ILP, caches, or speculative engines. Software developers and industry got used to the fact that applications get faster by just exchanging the underlying hardware. For several years now, these rules are proven to apply no longer. Moore's law about the ever-increasing number of transistors per die may still be valid, but power consumption limits stall the continued increase of clock frequencies. Due to this development, serial execution performance no longer improves automatically with the next processor generation.

In the many-core era that happens now, additional transistors are used not to speed up serial code paths, but to offer multiple execution engines (cores) per processor. This changes every desktop-, server-, or even mobile system into a parallel computer. The exploitation of additional transistors is therefore now the responsibility of software, which makes parallel programming a mandatory approach for all software with scalability demands.

The following topics and technologies will be covered:

Theory: Flynn's Taxonomy, Scalability Laws, Memory Models
Shared Memory: PThreads, C++11 Threads, Futures, OpenMP, TBB
Non-Uniform Memory Access: libnuma, PGASUS
On-Chip Accelerators: SIMD (SSE, AVX, AltiVec), Compression, LLVM-IR
External Accelerators: GPUs (CUDA), FGPAs (CAPI SNAP/MetalFS)
Shared Nothing: MPI, CloudCL, Actors (Erlang)
Profiling and monitoring with Hardware Performance Counters
Energy-Aware Computing

Intro

video

Lecture

This lecture will follow a flipped classroom approach, with the following time distribution:

75 %: Watch pre-recorded lecture videos
25 %: Participate in online meetings

We publish lecture videos by Tuesday evenings, so you can watch them in your own time every Wednesday. As there is no direct feedback channel in this format, please be sure to make notes of questions or comments as you watch.

The online meetings every Thursday will be used to discuss the material of the preceeding lecture videos. Each meeting will begin with a recap session for the lecture material and there is room for questions and comments. You will also receive organizational updates concerning the lecture, hands-on demos and tipps for your assignments.

Apart from the online meetings you can always post questions on GitLab.

All slides and lecture videos will be available in English.

Meetings

Online meetings are hosted on Zoom and will be held in German, unless English is requested.

Time	Thursdays, 11:00 ~ 12:30
Meeting ID	945-9492-0566
Password	08152020
Link	https://uni-potsdam.zoom.us/j/94594920566
Mobile	+496950502596,,94594920566#,,#,08152020#
Mobile	+496971049922,,94594920566#,,#,08152020#
Phone	+49 695 050 2596
Phone	+49 69 7104 9922
Phone	+49 30 5679 5800
SIP	94594920566@zoomcrc.com
H.323	213.19.144.110

To join the meeting, it is not necessary to register a Zoom account. Nevertheless, students of the University of Potsdam can create an account using their email address first.last@uni-potsdam.de (note the missing hpi). You may use either the web client, desktop client or your telephone to dial in. Please note that the University license does not include toll-free dial-in numbers, so this option might result in some costs.

This preliminary setup may change in the course of the semester if other meeting platforms become available.

As some participants may not be able to attend all meetings, we offer to make local recordings of each meeting. We will not use the Zoom built-in record functionality - which is administratively disabled anyway - but capture audio and video as received by our Zoom clients. Thus you are free to mute your audio and video streams if you do not want to appear in the recording. If you want to ask questions without being recorded, you can give us an indication and we will endeavor to cut your part from the audio track, but keep in mind that this may become infeasible if you talk while others talk as well. All recordings will be accessible with a lecture-specific password that we will send to participants and request you to keep private.

We hope this solution offers an adequate balance between convenience and privacy. If you have suggestions, we are very open for further discussion.

Assignments

This lecture includes five pratical assignments.

Assignments have to be solved in teams of three persons. The oral exam admittance is achieved if 50% of each assignment is solved correctly. We will provide one non-mandatory assignment at the end of the semester, that can be used to cancel out one failed assignment.

Assignments materials will be available on Gitlab at gitlab.hpi.de/parprog20/parprog20. There we will also prepare a way for you to find team partners if you have not found them already.

All assigments are submitted to and validated by our submission system at osm.hpi.de/submit. Submissions failing the validation script will be treated as not handed in.

Parallel Programming and Heterogeneous Computing 2020

Intro

Lecture

Meetings

Assignments

Schedule