CS 213 Multiprocessor Architecture and Programming, Fall 2022

Time: Monday/Wednesday 5:00 pm to 6:20 pm
Instructor: Elaheh Sadredini, elaheh@cs.ucr.edu - Office hours: by appointment
TA: Jingyao Zhang, jzhan502@ucr.edu - Office hours: Fridays at 1-2pm, WCH 110

Course Overview

The inevitable and rapidly growing adoption of multi-core parallel architectures within a processor chip by all of the computer industry pushes explicit parallelism to the forefront of computing for all applications and scales, and makes the challenge of parallel programming and system understanding more crucial. This course is designed to build a strong understanding of the fundamentals of the architecture of parallel computers and the tradeoffs made in their design. We will examine how architectures are designed to exploit different types of parallelism. We will study multi-core architectures, parallel memory systems, cache coherence in shared-memory architectures, memory consistency, vector architectures, dataflow machines, interconnection networks, and also programming models deployed on parallel machines (e.g., message passing, data-parallel, and shared memory models).

Course Prerequisite(s)

CS 203 or consent of instructor

Textbooks and Reading Material

The following books could be useful as supplements to lectures. However, they are not required.

◆ “Parallel Computer Architecture: A Hardware/Software Approach” by Culler, Singh, and Gupta.

◆ “Principles and Practices of Interconnection Networks” by Dally and Towles.

◆ “Computer Architecture: A Quantitative Approach” by Hennesssy and Patterson.

Grading and Policies

Grading Breakdown

- Homework assignments (two): 20%

- Programming assignments (two): 40%

- Quizzes (six): 30%

	 - Quizzes will be taken throughout the course. 
	 - We will have quizzes on eLearn (Canvas). 
	 - The lowest score will be dropped.

- Paper presentation (one): 10%

- Bonus points for in-class activities: 10%

Class Policies

Forum: We will be using Piazza as our class forum, and our primary way of communication outside of class. All general inquiries must be made on Piazza. For group-specific questions or private questions, you can either email me or post a private question on Piazza.

Late submission: 10% penalty per day up to 5 days for homworks and programming assigments (no credit after 5 days). We do not accept any late submission for quizzes. You are allowed to skip one of the quizzes with no penalty. Use these freebie prudently. If you choose to submit all the quizzes, the lowest score will be dropped.

Cheating policy: Working with others on assignments is a good way to learn the material and is encouraged. However, there are limits to the degree of cooperation that is permitted. Students may discuss among themselves the meaning of homework problems and possible approaches to solving them. Any written portion of an assignment, however, is to be done strictly on an individual basis. You may not copy from another student or from any other source, and you may not allow another student to copy your work. Exams should be done individually and no collaboration in any form with others are permitted. Any violation of the above is considered to be cheating and will result in a failing grade in the class (no exceptions).

Errors in grading: If you feel there has been an error in how an assignment or test was graded, you have one week from when the assignment is returned to bring it to our attention. You must submit (via email to the instructor) a written description of the problem. We will discuss regrades without receiving an email from you about it first.

Dues: All assignments are due at 11:59pm of the submission deadline.

Grades: Your score will be available on Canvas.

Grading Scale

A+: 96 and above
A: 93-95.9
A-: 90-92.9
B+: 87-89.9
B: 83-86.9
B-: 80-82.9
C+: 77-79.9
C: 73-76.9
C-: 70-72.9
D+: 67-69.9
D: 63-66.9
D-: 60-62.9
F: Below 60

Academic Integrity

Here at UCR we are committed to upholding and promoting the values of the Tartan Soul: Integrity, Accountability, Excellence, and Respect. As a student in this class, it is your responsibility to act in accordance with these values by completing all assignments in the manner described, and by informing the instructor of suspected acts of academic misconduct by your peers. By doing so, you will not only affirm your own integrity, but also the integrity of the intellectual work of this University, and the degree which it represents. Should you choose to commit academic misconduct in this class, you will be held accountable according to the policies set forth by the University, and will incur appropriate consequences both in this class and from Student Conduct and Academic Integrity Programs. For more information regarding University policy and its enforcement, please visit: http://conduct.ucr.edu.

Student Resources

◆ If you need special accomodation, please either contact me or find more information here: "UCR campus resources"

Course Schedule

The following schedule is tentative and is subject to change.

Week	Date	Topic	Note/Due/Assignment
1	Sep. 26 Sep. 28	Introduction Parallel Processing Basics
2	Oct. 3 Oct 5	Shared Memory Basics Shared Memory Basics	Quiz #1 (on Parallel Processing Basics), HW1 out
3	Oct. 10 Oct. 12	Memory Organization Programming Models (I)	Programming Assignment 1 out
4	Oct. 17 Oct. 19	Programming Models (II) Cache Coherence (I)	Quiz #2 (on Shared Memory Basics)
5	Oct. 24 Oct. 26	No Class Cache Coherence (II)
6	Oct. 31 Nov. 2	Cache Coherence (III) Shared Memory Synchronization	Programming Assignment 2 out, HW2 out
7	Nov. 7 Nov. 9	Memory Consistency Models (I) Memory Consistency Models (II)	Quiz #3 (on Cache Coherency)
8	Nov. 14 Nov. 16	Interconnection Networks (I) Interconnection Networks (II)	Quiz #4 (on Shared Memory Synchronization)
9	Nov. 21 Nov. 23	Interconnection Networks (III) Paper Presentations	Quiz #5 (Memory Consistency Models)
10	Nov. 28 Nov. 30	Paper Presentations Paper Presentations	Quiz #6 (Interconnection Networks)

Reading Materials

Topic	Readings
Parallel Processing Basics	Reading(s): The Task of the Referee, by Smith. Amdahl’s Law in the Multicore Era, by Hill and Marty. The Landscape of Parallel Computing Research: A View from Berkeley, by Asanovic et al. The Future of Microprocessors, by Borkar and Chien. Hardware Trends: Challenges and Opportunities in Distributed Computing by Harris. Software and the Concurrency Revolution, by Sutter and Larus. Cost-Effective Parallel Computing, by Wood and Hill. Book Chapter(s): Parallel Computer Architecture: A Hardware/Software Approach, Chapter 1.
Shared Memory Basics	Reading(s): How to make a multiprocessor computer that correctly executes multiprocess programs, by Lamport. Shared Memory Consistency Models: A Tutorial, by Adve and Gharachorloo. Foundations of the C++ concurrency memory model, by Boehm and Adve. The Java memory model, by Manson et al. Book Chapter(s): H&P, Chapter 5 (in fifth edition). Parallel Computer Architecture: A Hardware/Software Approach, Chapter 5. A Primer on Memory Consistency and Cache Coherence, Chapters 1-3.
Memory Organization	Reading(s): A case for exploiting subarray-level parallelism (SALP) in DRAM, by Yoongu et al. Low-cost inter-linked subarrays (LISA): Enabling fast inter-subarray data movement in DRAM, by Chang et al. Book Chapter(s): Memory Systems: Cache, DRAM, Disk: Bruce Jacob, David T. Wang, and Spencer Ng, Morgan Kaufman,2007, Chapter 10
Programming Models and Architectures	Book Chapter(s): Multiprocessors and Multicomputers, pp. 551-560. Parallel Computer Architecture: A Hardware/Software Approach, Chapter 1.
Cache Coherence	Reading(s): A Survey of Cache Coherence Schemes for Multiprocessors, by Stenstrom. A Class of Compatible Cache Consistency Protocols and their Support by the IEEE Futurebus, by Sweazey and Smith. Starfire: Extending the SMP Envelope, by Charlesworth. The Directory-Based Cache Coherence Protocol for the DASH Multiprocessor, by Lenoski et al. Performance Optimizations, Implementation, and Verification of the SGI Challenge Multiprocessor, by Galles and Williams. Specifying and Verifying a Broadcast and a Multicast Snooping Cache Coherence Protocol, by Sorin et al. So Many States, So Little Time: Verifying Memory Coherence in the Cray X1, by Abts et al. Reducing Memory and Traffic Requirements for Scalable Directory-Based Cache Coherence Schemes, by Gupta et al. Cache Invalidation Patterns in Shared-Memory Multiprocessors, by Gupta and Weber False Sharing and Spatial Locality in Multiprocessor Caches, by Torrellas et al. Book Chapter(s): H&P, Chapter 5.2, 5.3, 5.4 (in fifth edition). Parallel Computer Architecture: A Hardware/Software Approach, Chapters 6 & 8. A Primer on Memory Consistency and Cache Coherence, Chapters 6-9.
Shared Memory Synchronization	Reading(s): Efficient Synchronization Primitives for Large-Scale Cache-Coherent Multiprocessors, by Goodman et al. Algorithms for Scalable Synchronization on Shared-Memory Multiprocessors, by Mellor-Crummey and Scott. Efficient Synchronization: Let Them Eat QOLB, by Kägi et al. Synchronization and Communication in the T3E Multiprocessor, by Scott. Book Chapter(s): Shared-Memory Synchronization, Chapters 1-5 & 7.
Memory Consistency Models	Reading(s): Shared Memory Consistency Models: A Tutorial, by Adve and Gharachorloo. Memory Consistency and Event Ordering in Scalable Shared-Memory Multiprocessors, by Gharachorloo et al. Two Techniques to Enhance the Performance of Memory Consistency Models, by Gharachorloo et al. An Evaluation of Memory Consistency Models for Shared-Memory Systems with ILP Processors, by Pai et al. Threads Cannot Be Implemented As a Library, by Boehm. Book Chapter(s): A Primer on Memory Consistency and Cache Coherence, Chapters 3-5.
Interconnection Networks	Reading(s): The Alpha 21364 Network Architecture, by Mukherjee et al. On-Chip Interconnection Architecture of the Tile Processor, by Wentzlaff et al. Flattened Butterfly : A Cost-Efficient Topology for High-Radix Networks, by Kim et al. The Cray T3E Network: Adaptive Routing in a High Performance 3D Torus, by Scott and Thorson. Microarchitecture of a High-Radix Router, by Kim et al. SeaStar Interconnect: Balanced Bandwidth for Scalable Performance, by Brightwell et al. The Gemini System Interconnect, by Alverson et al. Book Chapter(s): High Performance Datacenter Networks: Architectures, Algorithms, and Opportunities, All Chapters (It's a short synthesis lecture). On-Chip Networks, Chapters 2, 5 & 6. Principles and Practices of Interconnection Networks, the whole book (highly recommended).
Vector, SIMD, and SIMT Processing	Reading(s): The Cray-1 Computer System, by Cray Research, Inc. Tarantula: A Vector Extension to the Alpha Architecture, by Espasa et al. NVIDIA’s Next Generation CUDA Compute Architecture: Kepler GK110, by NVIDIA. NVIDIA GeForce GTX 980 Featuring Maxwell, The Most Advanced GPU Ever Made, by NVIDIA. Book Chapter(s): Computer Architecture: A Quantitative Approach (5th Ed), Chapter 4 and Appendix G.

Papers Presentation

TBA ...

Resources

◆ WWW Computer Architecture Home Page: a comprehensive guide to research, tools, and general information on computer architecture.