CS 213 Multiprocessor Architecture and Programming, Winter 2021

Time: Monday and Wednesday 6:30 pm to 7:50 pm
Office hours: by appointment
Instructor: Elaheh Sadredini, elaheh@cs.ucr.edu

Course Overview

The inevitable and rapidly growing adoption of multi-core parallel architectures within a processor chip by all of the computer industry pushes explicit parallelism to the forefront of computing for all applications and scales, and makes the challenge of parallel programming and system understanding all the more crucial. This course is designed to build a strong understanding of the fundamentals of the architecture of parallel computers and the tradeoffs made in their design. We will examine how architectures are designed to exploit different types of parallelism. We will study multi-core architectures, parallel memory systems, cache coherence in shared-memory architectures, memory consistency, vector architectures, dataflow machines, interconnection networks, and also programming models deployed on parallel machines (e.g., message passing, data-parallel, and shared memory models).

Course Prerequisite(s)

CS 203 or consent of instructor

Textbooks and Reading Material

The following books could be useful as supplements to lectures. However, they are not required.

◆ “Parallel Computer Architecture: A Hardware/Software Approach” by Culler, Singh, and Gupta.

◆ “Principles and Practices of Interconnection Networks” by Dally and Towles.

◆ “Computer Architecture: A Quantitative Approach” by Hennesssy and Patterson.

We will be using Piazza as our class forum, and our primary way of communication outside of class. All general inquiries must be made on Piazza. For group-specific questions or private questions, you can either email me or post a private question on Piazza.

Grading and Policies

Grading Breakdown

- Homework assignments (two): 30%
- Programming assignments (Two): 40%
- Quizzes (four): 20%
- Paper presentation (one): 10%
- Bonus points for in-class activities: 10%

Grading Policies

- Late policy: 10% per day up to 5 days (no credit after 5 days)
- Assignments are due at 11:59pm of the submission deadline
- Everything must be your own work (unless otherwise specified)
- Cheating == failing grade (no exceptions)

Grading Scale

A+: 96 and above
A: 93-95.9
A-: 90-92.9
B+: 87-89.9
B: 83-86.9
B-: 80-82.9
C+: 77-79.9
C: 73-76.9
C-: 70-72.9
D+: 67-69.9
D: 63-66.9
D-: 60-62.9
F: Below 60

Academic Integrity

Here at UCR we are committed to upholding and promoting the values of the Tartan Soul: Integrity, Accountability, Excellence, and Respect. As a student in this class, it is your responsibility to act in accordance with these values by completing all assignments in the manner described, and by informing the instructor of suspected acts of academic misconduct by your peers. By doing so, you will not only affirm your own integrity, but also the integrity of the intellectual work of this University, and the degree which it represents. Should you choose to commit academic misconduct in this class, you will be held accountable according to the policies set forth by the University, and will incur appropriate consequences both in this class and from Student Conduct and Academic Integrity Programs. For more information regarding University policy and its enforcement, please visit: http://conduct.ucr.edu.

Student Resources

◆ If you need special accomodation, please either contact me or find more information here: "UCR campus resources"

Course Schedule

The following schedule is tentative and is subject to change.

Week	Date	Topic	Note/Due/Assignment
1	Jan. 4th Jan. 6th	Introduction Parallel Processing Basics
2	Jan. 11th Jan. 13th	Shared Memory Basics Programming Models (I)
3	Jan. 18th Jan. 20th	Holiday - No Class Programming Models (II) - Memory Organization	Quiz 1 (in class), HW1 out (due on Jan 27th at 11:59pm)
4	Jan. 25th Jan. 27th	Cache Coherence (I) Cache Coherence (II)
5	Feb. 1st Feb. 3rd	Cache Coherence (III) Shared Memory Synchronization	Quiz 2 (in class), Programming Assignment #1 out (due on Feb 13th at 11:59pm)
6	Feb. 8th Feb. 10th	Shared Memory Synchronization Memory Consistency Models (I)
7	Feb. 15th Feb. 17th	Holiday - No Class Memory Consistency Models (II)	Quiz 3 (in class), Makeup class on Feb 18th
8	Feb. 22nd Feb. 24th	Memory Consistency Models (III) Interconnection Networks (I)	HW2 out (due on March 1st at 11:59pm)
9	March 1st March 3rd	Interconnection Networks (II) Interconnection Networks (III)	Programming Assignment #2 out (due on March 15th at 11:59pm) Quiz 4 (in class)
10	March 8th March 10th	Paper Presentations Paper Presentations

Assignments, Quizzes, and Bonus Questions

Reading Materials

Topic	Readings
Parallel Processing Basics	Reading(s): The Task of the Referee, by Smith. Amdahl’s Law in the Multicore Era, by Hill and Marty. The Landscape of Parallel Computing Research: A View from Berkeley, by Asanovic et al. The Future of Microprocessors, by Borkar and Chien. Hardware Trends: Challenges and Opportunities in Distributed Computing by Harris. Software and the Concurrency Revolution, by Sutter and Larus. Cost-Effective Parallel Computing, by Wood and Hill. Book Chapter(s): Parallel Computer Architecture: A Hardware/Software Approach, Chapter 1.
Shared Memory Basics	Reading(s): How to make a multiprocessor computer that correctly executes multiprocess programs, by Lamport. Shared Memory Consistency Models: A Tutorial, by Adve and Gharachorloo. Foundations of the C++ concurrency memory model, by Boehm and Adve. The Java memory model, by Manson et al. Book Chapter(s): H&P, Chapter 5 (in fifth edition). Parallel Computer Architecture: A Hardware/Software Approach, Chapter 5. A Primer on Memory Consistency and Cache Coherence, Chapters 1-3.
Programming Models and Architectures	Book Chapter(s): Multiprocessors and Multicomputers, pp. 551-560. Parallel Computer Architecture: A Hardware/Software Approach, Chapter 1.
Cache Coherence	Reading(s): A Survey of Cache Coherence Schemes for Multiprocessors, by Stenstrom. A Class of Compatible Cache Consistency Protocols and their Support by the IEEE Futurebus, by Sweazey and Smith. Starfire: Extending the SMP Envelope, by Charlesworth. The Directory-Based Cache Coherence Protocol for the DASH Multiprocessor, by Lenoski et al. Performance Optimizations, Implementation, and Verification of the SGI Challenge Multiprocessor, by Galles and Williams. Specifying and Verifying a Broadcast and a Multicast Snooping Cache Coherence Protocol, by Sorin et al. So Many States, So Little Time: Verifying Memory Coherence in the Cray X1, by Abts et al. Reducing Memory and Traffic Requirements for Scalable Directory-Based Cache Coherence Schemes, by Gupta et al. Cache Invalidation Patterns in Shared-Memory Multiprocessors, by Gupta and Weber False Sharing and Spatial Locality in Multiprocessor Caches, by Torrellas et al. Book Chapter(s): H&P, Chapter 5.2, 5.3, 5.4 (in fifth edition). Parallel Computer Architecture: A Hardware/Software Approach, Chapters 6 & 8. A Primer on Memory Consistency and Cache Coherence, Chapters 6-9.
Shared Memory Synchronization	Reading(s): Efficient Synchronization Primitives for Large-Scale Cache-Coherent Multiprocessors, by Goodman et al. Algorithms for Scalable Synchronization on Shared-Memory Multiprocessors, by Mellor-Crummey and Scott. Efficient Synchronization: Let Them Eat QOLB, by Kägi et al. Synchronization and Communication in the T3E Multiprocessor, by Scott. Book Chapter(s): Shared-Memory Synchronization, Chapters 1-5 & 7.
Memory Consistency Models	Reading(s): Shared Memory Consistency Models: A Tutorial, by Adve and Gharachorloo. Memory Consistency and Event Ordering in Scalable Shared-Memory Multiprocessors, by Gharachorloo et al. Two Techniques to Enhance the Performance of Memory Consistency Models, by Gharachorloo et al. An Evaluation of Memory Consistency Models for Shared-Memory Systems with ILP Processors, by Pai et al. Threads Cannot Be Implemented As a Library, by Boehm. Book Chapter(s): A Primer on Memory Consistency and Cache Coherence, Chapters 3-5.
Interconnection Networks	Reading(s): The Alpha 21364 Network Architecture, by Mukherjee et al. On-Chip Interconnection Architecture of the Tile Processor, by Wentzlaff et al. Flattened Butterfly : A Cost-Efficient Topology for High-Radix Networks, by Kim et al. The Cray T3E Network: Adaptive Routing in a High Performance 3D Torus, by Scott and Thorson. Microarchitecture of a High-Radix Router, by Kim et al. SeaStar Interconnect: Balanced Bandwidth for Scalable Performance, by Brightwell et al. The Gemini System Interconnect, by Alverson et al. Book Chapter(s): High Performance Datacenter Networks: Architectures, Algorithms, and Opportunities, All Chapters (It's a short synthesis lecture). On-Chip Networks, Chapters 2, 5 & 6. Principles and Practices of Interconnection Networks, the whole book (highly recommended).
Vector, SIMD, and SIMT Processing	Reading(s): The Cray-1 Computer System, by Cray Research, Inc. Tarantula: A Vector Extension to the Alpha Architecture, by Espasa et al. NVIDIA’s Next Generation CUDA Compute Architecture: Kepler GK110, by NVIDIA. NVIDIA GeForce GTX 980 Featuring Maxwell, The Most Advanced GPU Ever Made, by NVIDIA. Book Chapter(s): Computer Architecture: A Quantitative Approach (5th Ed), Chapter 4 and Appendix G.

Papers Presentation

Resources

◆ WWW Computer Architecture Home Page: a comprehensive guide to research, tools, and general information on computer architecture.