Welcome to CS 250: Software Security.

My name is Heng Yin, and I’m a professor in the Computer Science Department. This course is directly related to my research area, so I’m excited to share it with you.

We’ll begin with some logistics—let’s talk about the syllabus.

Please check the course website. I’ll be releasing the schedule, lecture notes, and the papers we’ll be studying, presenting, and discussing—all on that page. Let me show you what it looks like so you get a sense of it.

You don’t need to memorize the URL. Just Google my name, go to my homepage, and find the teaching section. The course link will be there. It’s your go-to source for everything essential.

That page already contains a lot of useful information: the catalog description, communication channels, covered topics, paper review guidelines, grading policy, and a preliminary schedule. Make sure to check it regularly.

So, what problem are we trying to solve in this course?

Software security is a broad field—it can mean many different things. To help us focus, I’ll use two DARPA competitions as motivation for our discussions. Much of this course revolves around the types of problems these competitions present and the techniques developed to solve them.

The first event is called the Cyber Grand Challenge (CGC). It started in 2014, giving teams two years to build their systems, with a final competition in 2016.

The core idea was to evaluate how well systems could automatically identify software vulnerabilities, patch them, and generate exploits to prove that the vulnerabilities existed—or even attack other systems. Exploit generation, in this context, means crafting input to take advantage of these flaws.

An important aspect of CGC is that it focused solely on binary code—because in many real-world situations, source code isn’t available, especially for proprietary or closed-source systems. Verifying whether a vulnerability exists—and whether it can be exploited—must be done at the binary level. So this competition emphasized binary analysis.

The second event is called AIxCC (AI Cyber Challenge), and it’s still ongoing. This time, DARPA added an AI component, especially in light of recent breakthroughs in large language models (LLMs), which are proving to be very powerful. The idea is similar: use AI to identify and patch vulnerabilities—but this time, at the source-code level, rather than binaries.

This reflects a shift in focus: AIxCC targets the open-source ecosystem, where developers frequently reuse components from public repositories. Unfortunately, there aren’t many strong tools for automatically securing such code. If we could build AI systems to help with that, it could significantly improve Internet security overall.

We use these two events to motivate our exploration of current techniques and recent advances in software security. Although AIxCC is focused on source code, our course will keep a strong focus on binary-level security, because binary analysis remains a fundamental part of modern software security.

Background Skills and Expectations

Let me list some basic skills you should have coming into this course. These are fairly common for computer science students, but I want to lay them out explicitly so you know what to expect.

We’ll primarily be working with Linux distributions—not so much macOS or Windows. Most of the binaries we analyze will be compiled from C or C++ code, so we’ll focus heavily on those languages.

We won’t be dealing much with Java or Python as targets of analysis, but you might need to use Python for scripting in labs and projects. Some tools or tasks may involve writing or modifying Python scripts.

Many of the tools and projects we’ll use are open source. That means you’re expected to navigate documentation and examples on your own to some extent. Since they’re not commercial products, the documentation can be spotty. That’s part of the challenge—we’ll figure things out together. And of course, feel free to reach out to me with any questions.

You should also know how to read assembly code, particularly x86. If you’re a bit rusty, that’s okay—just take some time to review the basics. The instructions we’ll work with aren’t overly complex, but they are essential background knowledge.

I expect that you’ve taken an undergraduate computer security course—CS 165 would be ideal. If not, you might find some material challenging, but it’s still possible to catch up if you’re motivated.

Your first lab assignment will be released soon, and it will be due in one week. It involves analyzing binary code and constructing a basic buffer overflow attack. It’s designed to test your understanding of several foundational skills.

If you find it very difficult to complete this lab on time, that might be a sign the course could be too advanced for you at this point. I’m not saying you have to finish it perfectly within a week—but consider it a gauge. If you can complete at least part of it, that’s a good indicator you’ll be able to keep up and learn as we go.

Course Topics Overview

In this course, we’ll cover several major topics in software security:

  • Vulnerability discovery, including:

    • Fuzzing (a key technique we’ll explore in depth)
    • Symbolic execution (some of you may have seen this in software testing courses—it’s similar in spirit but different in focus)
    • Combinations of techniques (hybrid approaches)

  • Binary-level analysis, especially static analysis, which is significantly harder than analyzing source code. For binaries, static analysis is often limited due to lack of structure and semantics.

  • Source code analysis with large language models (LLMs). We’ll briefly discuss how LLMs can be used to review source code and detect possible vulnerabilities. While they show promise with source code, applying them to binaries is much harder.

  • Patching strategies: Once we identify vulnerabilities—or even if we don’t know exactly where they are—we’ll talk about techniques to harden binaries and protect executables from attacks.

  • Exploit generation: This is an extremely difficult area that often requires deep domain knowledge. We’ll still explore it conceptually and work through practical exercises. Generating exploits automatically requires a lot of domain expertise. While we’d like to fully automate exploit generation, in practice we can only go so far. The problem is extremely challenging and often demands deep technical knowledge.

  • software supply chain security. This is increasingly important because modern software development heavily relies on open-source components. These components may contain vulnerabilities, or even malware and backdoors. One specific issue we’ll focus on is called “one-day” (or sometimes “n-day”) vulnerabilities. These are vulnerabilities that are already known and documented, yet for various reasons still end up in software builds. In other words, your program might unknowingly include code with known flaws.

So how do we tackle these problems? We’ll rely on a combination of program analysis and, increasingly, AI-based techniques. AI has become powerful enough to help with various aspects of vulnerability detection and mitigation, and we’ll explore how it can be integrated with traditional analysis methods.

Course Timeline and Structure

In terms of schedule, I’ll use the first 4–5 weeks to give you essential background.

Each lecture—or a pair of lectures—will focus on a specific topic. These will be structured like survey-style talks, where I share my perspective on an area, especially things I’ve worked on, along with relevant work in the field. My goal is to provide enough context so that you can confidently read research papers and give your own presentation later in the course.

Labs

During this same period, you’ll also complete three lab assignments. Each one is intentionally lightweight—both in scale and difficulty—so that you can focus on learning.

Here’s a breakdown:

  1. Lab 1: Simple Exploit Construction
    If you’ve taken an undergraduate security course like CS 150 or CS 165, you’ve probably seen buffer overflow attacks and return-to-libc exploits. This first lab is mostly a review. If you’ve never done it before but have some background knowledge, you should be able to pick it up quickly. But if you find yourself struggling or don’t know where to start, it may be a sign that the rest of the labs will also be difficult.

  2. Lab 2: Fuzzing and Symbolic Execution
    I’ll provide one or two small programs and ask you to try out a few tools. You’ll document your experience—what you did, what you observed, and what you learned. This should be very manageable.

  3. Lab 3: Implementing Control-Flow Integrity (CFI)
    If you’ve taken a class like CS 255, you’ve probably encountered CFI at the source-code or conceptual level. In this lab, you’ll apply those ideas directly to binaries. I’ll introduce some binary instrumentation techniques, and you’ll implement basic CFI logic and test it on a few small programs. That’s it.

These three labs will be completed in the first half of the quarter.

Final Term Paper

In the second half, you’ll work on a small-scale research project of your choice. You’ll submit a brief proposal (around two pages) describing what you’d like to explore, your plan, and any goals or hypotheses you might have. By the end of the quarter—basically the last day of finals week—you’ll submit your term paper.

I’m not expecting a full-fledged conference paper. It doesn’t have to be groundbreaking or span dozens of pages. What I want to see is that you explored a topic, tried something out, and learned something through the process. That’s enough.

If you’re not sure what to work on, feel free to come to my office hours—I’ll announce the schedule. We can talk through some ideas together.

Some simple project ideas could be: - Try out two tools that are supposed to do the same thing, and compare your experience. - Pick a topic we’ve covered and go deeper: replicate something, modify a tool, or evaluate a method.

If you genuinely run out of ideas or try something that turns out to be too difficult, there’s a backup plan: write a survey paper. Since I’ll have already given overview lectures and recommended papers on most topics, you’ll have enough background to write a solid summary and analysis.

Paper Reviews

During the second half of the quarter, as we start student paper presentations, you’ll be expected to write reviews for the papers being presented.

Sometimes we may have two presentations in one class. In that case, you’ll only need to review one of the two papers—so the workload stays manageable.

Here are some review guidelines: - Minimum 400 words - Do not copy/paste from the paper or use tools like ChatGPT or GPT-based summarizers to generate the review. - Yes, I have ways to check. - Academic integrity matters. - No bullet points—write in natural, flowing text, like a mini-essay or commentary. - Your writing style doesn’t have to be fully academic or formal. If you prefer a more personal tone, that’s totally fine—actually, I like it. Let your voice and opinion show through.

What I’m looking for is your thought process. The review doesn’t have to be perfectly “correct” or balanced. It’s okay to be critical, or enthusiastic, or even confused—just explain yourself.

Here are some questions to guide your review: - What problem is the paper trying to solve? - Why is that problem important? - What are existing approaches, and why aren’t they sufficient? - How does this paper attempt to solve the problem? - What are the key ideas? Can you explain them clearly? - Are there specific details or design choices that seem critical to the approach?

Try not to just restate the abstract. Instead, if you can identify and explain what makes this paper work, that’s great. For example: “This optimization in section 3 is essential—without it, the rest of the approach probably wouldn’t scale.” That kind of insight is exactly what I’d like to see.

But don’t worry—if you can’t reach that level of analysis, that’s okay. Just try your best.

Also, bring questions to class for discussion! If something puzzled you or you think a certain claim deserves more scrutiny, bring it up during the Q&A.

Absolutely! Here's a polished and clear version of that section:

Paper Presentations

Each of you will prepare a 25-minute presentation on a research paper (not including Q&A). After your talk, we’ll have about 15 minutes for discussion. Since we have 80 minutes per class, we can usually fit two presentations per session, giving each paper about 40 minutes total.

So, aim for a 25-minute talk, and prepare a few discussion questions to help spark conversation afterward. If you’re able to lead the discussion, even better—but don’t worry if you find that part difficult. Just try your best to explain the paper at a meaningful level.

If you’ve never done this kind of presentation before, that’s totally fine—raise your hand if it’s new to you. You’ll get the hang of it quickly by observing your classmates and learning as we go. Presenting research papers is very common in graduate-level courses, and this will be great practice.

Grading Breakdown

Here’s how your grade will be distributed:

  • Participation: 10%
  • Lab Assignments (3 total): 30%
  • Paper Reviews: 10%
  • Paper Presentation: 20%
  • Final Term Project and Paper: 30%

The grading is fairly balanced, and each component contributes meaningfully.

All assignments, labs, and the final project are expected to be done individually. However, if you have a strong reason to work as a group, especially for the final project, come talk to me. For example, if two of you share a common research interest and working together would allow for a more substantial result, that’s a good reason. I’m open to it—just let me know.

Introduction to Lab 1: Binary Code Comprehension

The first part of this lab focuses on binary code comprehension.

Now, don’t be intimidated if you’re thinking, “I don’t know how to read binary code!”—that’s completely fine. We’ll work through it gradually, and that’s the whole point of starting with a simple example. The goal is to ease you into the fundamentals.

There’s a lot of fragmented knowledge involved in understanding binary: things like executable formats, calling conventions, stack layouts, relocation tables, and position-independent code. I’ll provide you with some helpful resources on all of these topics.

But again—if you don’t know any of this yet, it’s okay. I started there too. And for the first lab, you really don’t need to know all of it in detail. You can get pretty far with just a few key ideas.

Even better: nowadays, we have tools like ChatGPT, which are great for filling in the gaps. You can ask it specific questions, and it’ll give you solid explanations—often better and faster than trying to Google around for answers. The trick is learning to ask the right questions.

So, don’t be afraid. With modern AI tools, it's easier than ever to get familiar with low-level topics that used to feel obscure or inaccessible.

What You'll Be Doing

We’ll start by constructing a very simple buffer overflow attack. I’ve given you a minimal example01.c file with just two functions.

#include <stdio.h>
#include <string.h>

int IsPasswordOkay(void)
{
  char Password[12];
  gets(Password);

  if (!strcmp(Password, "goodpass"))
    return 1;

  return 0;
}

int main(void)
{
  int PwStatus;

  puts("Enter password:");
  PwStatus = IsPasswordOkay();
  if (PwStatus == 0) {
    puts("Access denied");
    return -1;
  }
  puts("Access granted");
  return 0;
}

Here’s how it works: - The main() function asks the user to enter a password. - It calls another function to check the password against a hardcoded string. - If the password matches, it returns true; otherwise, false. - Based on the return value, main() either prints "Access Granted" or "Access Denied."

Now, in that password-checking function, there’s a very small buffer—just two bytes—allocated for the input. The comparison uses a hardcoded string, and the input is read using the unsafe gets() function.

That’s a problem.

Because gets() doesn’t check the input length, if you enter a string that’s too long, it will overflow the buffer and start overwriting the stack.

That’s where the vulnerability lies—and it’s what you’ll exploit.

So yes, it’s a simple toy example, but it contains the essence of a real stack-based buffer overflow attack. You’ll use it to learn how overflows work, how control flow can be hijacked, and why these bugs are dangerous.

We’ll also begin to look at the binary code itself. You’ll disassemble the program, see how the stack is laid out, and understand how the function behaves at the machine level.

This lab is a great opportunity to learn by doing, and start getting comfortable with working at the binary level.

Setting Up the Lab Environment & Introduction to Ghidra

So, let’s talk about how to compile the program and set up your environment for the lab.

The first step is to disable address space layout randomization (ASLR). There’s a way to do this using sudo, but that requires root privileges, which might not be available to you on all machines.

Don’t worry—I’ve included an easier alternative in the lab instructions. It doesn’t require root access and allows you to do the lab on any lab computer, even if you don’t have Linux installed on your own machine.

When compiling the source code, you’ll also use specific compiler flags to disable certain security features, like the stack protector. This makes it easier to successfully construct an exploit, which is the goal of this lab.

To examine the binary code, I’ll be using Ghidra. If you’ve never used it before, I highly encourage you to try it. It’s free, open source, and surprisingly user-friendly—unlike tools like IDA Pro, which are commercial and expensive.

Once you load the binary into Ghidra, you’ll see a user interface with a function tree on the left-hand side. This lists all the functions Ghidra has identified in the binary.

In this case, you should easily spot the main() function and the password-checking function (is_password_ok). However, in some binaries—especially stripped binaries—functions may appear only as numeric addresses, making them harder to identify. In those cases, some reverse engineering is required.

But for our example, things are simple and labeled.

Exploring the Decompiler View

Let’s look at the main() function. Ghidra will show you both the disassembly and a decompiled view of the code. The decompiled view is especially helpful—it often looks very close to the original C source code.

Ghidra does a great job of reconstructing high-level logic from assembly. In this example, the decompiled output almost exactly matches the original source. This is great, because it means you don’t have to spend too much time reading raw assembly—unless you want to dig deeper.

Focusing on the Vulnerability

Now, let’s look at the is_password_ok() function. This is where the vulnerability lives.

In the decompiled view, you’ll see: - A buffer (just a few bytes in size) - A call to the unsafe gets() function - A comparison between the input and a hardcoded password

Because gets() doesn’t perform bounds checking, a long enough input will overflow the buffer and start overwriting memory on the stack. That’s the essence of a stack-based buffer overflow vulnerability.

And again, thanks to Ghidra’s decompiler, you can clearly see the structure of the code and understand what's happening—even if you're not fluent in assembly.

Understanding the Stack Layout and Buffer Overflow

Let’s take a closer look at the stack layout and how the buffer overflow works in this example.

When you call the gets() function in the password-checking function, the program first performs some standard stack setup instructions. You’ll typically see something like this:

push   rbp
mov    rbp, rsp
sub    rsp, 0x10

Let’s break this down step by step.

How the Stack Works

First, recall that in most diagrams of the stack, we draw memory addresses in reverse order—lower addresses are at the bottom, and higher addresses are at the top.

So imagine a vertical stack diagram:

↑ Higher memory addresses
|
| [ ... other data ... ]
| [ Return Address     ] ← top of the current frame
| [ Old RBP            ]
| [ Local Variables    ] ← buffer, etc.
↓ Lower memory addresses

When the main() function calls is_password_ok(), the return address is pushed onto the stack, followed by the old base pointer (RBP). Then, a new stack frame is set up with:

  • push rbp — saves the old base pointer
  • mov rbp, rsp — sets the base pointer to the current top of the stack
  • sub rsp, 0x10 — allocates 16 bytes on the stack for local variables

This gives us 16 bytes of space, and within that space is the buffer we’re going to overflow.

Tracking the Buffer

Next, we see an instruction like:

lea rax, [rbp - 0xc]

This computes the effective address of the buffer and stores it in rax. Since rbp is the base pointer, rbp - 0xc (or 12 bytes below rbp) is where the buffer starts.

Then:

mov rdi, rax
call gets

This means the address of the buffer is passed to gets() via the RDI register, which is the first argument register on x86-64 according to the System V calling convention.

This matches the original source code, where gets() is called with the buffer as the argument.

Why This Matters

This layout is important because we now know the exact position of the buffer relative to the return address on the stack.

  • The buffer starts at [rbp - 0xc] (12 bytes)
  • The saved RBP is at [rbp] (8 bytes)
  • The return address is right above that

So, to overwrite the return address, we need to write:

  • 12 bytes to reach the end of the buffer
  • 8 more bytes to overwrite the saved RBP
  • And then the next 8 bytes will overwrite the return address

That’s 20 bytes before we hit the return address. So, for a basic exploit, we can construct an input like:

"AAAAAAAAAAAAAAAAAAAA"  (20 'A's)

This sets the stage for buffer overflow, and now we can control what comes next—potentially overwriting the return address with an address of our choosing.

Crafting the Exploit Payload

After the 20 bytes of padding we use to reach the return address, we need to provide 8 more bytes—and this is where the actual exploit payload goes.

These 8 bytes will overwrite the return address. When the function returns, instead of jumping back to where it’s supposed to (i.e., the next instruction in main()), it will jump to whatever address we’ve placed there.

This is the core idea behind a buffer overflow attack: once you control the return address, you control the flow of execution. You can redirect the program to any address within the process's memory space.

So, in this case, the goal is to replace the return address with a new one—maybe pointing somewhere useful in the code to bypass authentication, for example.

A Note on Endianness

Before we do that, there’s an important detail we need to consider: endianness.

Endianness describes how bytes are ordered in memory. Most of the architectures we use today are little-endian, which means the least significant byte is stored at the lowest memory address.

Let’s say we want to overwrite the return address with the value:
0x0000000000054321

In memory (little-endian), we would write the bytes in reverse order:

21 43 05 00 00 00 00 00

So when you craft your input payload, you’ll need to lay out the bytes in little-endian format. This is a critical detail—if you get the byte order wrong, your program will crash or jump to an invalid address.

Exploiting the Vulnerability to Bypass Authentication

Let’s start with something simple: say you just want to bypass the password check, even without knowing the correct password.

Even though the password is hardcoded in the binary—and yes, you could technically discover it by reverse engineering—we’ll assume you can’t or don’t want to crack it directly.

Instead, you can use the overflow to skip the authentication check altogether.

Here’s the idea: after calling the password check function, the program looks at the return value: - If it’s true, it prints "Access Granted" - If it’s false, it prints "Access Denied"

So, if we can find the instruction address corresponding to the “Access Granted” branch, we can overwrite the return address to jump directly there.

How Do You Find That Address?

This requires a little binary analysis.

By examining the main() function in Ghidra or a disassembler, you can: 1. Identify the call to the password-checking function 2. Look at the next few instructions, which handle the return value 3. Locate the conditional jump that determines whether to print "Access Granted" or "Access Denied"

For example, you might see something like:

call is_password_ok
mov  eax, [rbp-0x10]
cmp  eax, 0
jne  0x4012d0   ; jump if not equal (i.e., password is correct)

In this case, the instruction at address 0x4012d0 is where the program jumps to if the password check passes. That’s the address you want to overwrite the return address with.

If you can point execution directly to that instruction, you bypass the entire check and trick the program into granting access—even with the wrong password.

Identifying the Jump Target

Ghidra does a great job of making the binary more readable. Still, to confirm the control flow, let’s walk through what’s happening in the disassembly.

After the call to the password-checking function, the return value is stored in a local variable on the stack. In x86-64, that return value comes back in the EAX register (or RAX, depending on context). You’ll then see something like:

mov    [rbp-0x10], eax
cmp    [rbp-0x10], 0
jne    0x40120d

Here’s what’s happening: - The result of the password check (eax) is saved to the stack. - It’s then compared to 0—essentially checking whether it’s true or false. - The instruction jne (jump if not equal) means: if the result is not zero (i.e., password is correct), jump to address 0x40120d.

And that jump goes directly to the part of the code that prints:

puts("Access Granted");

Ghidra will even annotate the instruction for you, showing that RDI is loaded with the string "Access Granted", and then puts() is called.

Overwriting the Return Address

This gives us a target. Instead of returning to the normal instruction after the function call, we want the program to return to 0x40120d. That way, it skips the password check logic entirely and jumps straight to access being granted.

To do that, we overwrite the return address with 0x000000000040120d.

But remember: this is a little-endian system, so you have to write that address in reverse byte order in your exploit input. That means the correct byte sequence will be:

0d 12 40 00 00 00 00 00

So after your 20 bytes of padding (e.g., "A" * 20), you append these 8 bytes to redirect execution to the "Access Granted" code.

You can try this in your exploit input and see if it works. It's a great first demonstration of how buffer overflows can be used to bypass control flow.

Injecting Shellcode (Part 2 of the Lab)

The next part of the lab focuses on injecting arbitrary code (i.e., shellcode). You’ll be working with a template I’ve provided, named something like:

shellcode_injection.bin

The shellcode is already included—you don’t have to write it from scratch—but you’ll need to tweak things to make it work.

Stack Layout and Shellcode Placement

In my version of the binary, I know the distance between the start of the buffer and the return address is 20 bytes. So again, the plan is:

  • Fill the first 20 bytes with junk (padding)
  • Overwrite the return address with the address of your shellcode

But where do we place the shellcode?

You have two main options: 1. Right before the return address — if your shellcode is small, you might be able to fit it within the padding space (before those final 8 bytes) 2. After the return address — append it after your payload, and jump to it from the overwritten return address

The second approach is often easier, since you can give your shellcode more space.

So imagine your payload looks like this:

[ padding (20 bytes) ] [ return address (8 bytes) ] [ shellcode (variable length) ]

Finding the Address to Jump To

Here’s the key part: you’ll need to figure out the exact memory address where your shellcode resides on the stack so you can overwrite the return address with that value.

To do that: - Use GDB to debug the program. - Set a breakpoint at the entry of the vulnerable function. - Examine the value of RSP or RBP after the stack frame is set up.

For example:

(gdb) break is_password_ok
(gdb) run
(gdb) info registers rsp rbp

Once you know where the stack is, you can estimate or directly observe the location of the injected shellcode. Use that address in little-endian format as the new return address.

A Note on Debugger vs. Real Execution

Be aware: sometimes the addresses you see in GDB differ slightly from when the program runs normally (due to differences in the execution environment, memory layout, etc.). So your exploit might need a bit of tweaking.

You can try inserting a NOP sled before the shellcode—this gives you a bit of slack, so even if your jump is off by a few bytes, it still lands in the sled and slides into the shellcode.

Return-to-Shellcode with a NOP Sled

Now, I don’t want to get too deep into the weeds here, but it’s important to note a tiny detail about stack addresses when you’re using a debugger.

When you run the program inside a debugger, it may push some additional values onto the stack—like environment variables or internal metadata—which slightly shifts the stack layout. So the addresses you observe in GDB may not perfectly match what happens when the binary runs outside the debugger.

To handle this uncertainty, a common technique is to use a NOP sled.

A NOP sled (short for No Operation) is a long sequence of 0x90 instructions, which do nothing but consume CPU cycles. They’re one byte each and allow the CPU to “slide” into the shellcode even if the jump address is slightly off.

So, your exploit payload might look like:

[ Padding ] [ Return Address ] [ NOP NOP NOP ... ] [ Shellcode ]

The idea is that even if the jump lands somewhere in the middle of the NOPs, the CPU will slide forward until it hits the shellcode and executes it.

In this case, because the binary accepts a fairly long input, you can inject a very large NOP sled. Then, set the return address to land somewhere in the middle of that sled—giving you a pretty generous margin of error. If everything lines up, your exploit will work reliably.

Return-to-libc Attack

The next attack you’ll attempt is the classic return-to-libc technique.

Buffer overflow attacks have been around for decades. As a defense, modern systems often mark stack memory as non-executable, meaning you can’t jump into shellcode placed on the stack. So even if your exploit overwrites the return address, the CPU will crash if it tries to execute instructions in a non-executable region.

To bypass this restriction, attackers came up with return-to-libc. Instead of injecting new code, you jump to existing code—specifically, to a known library function in libc (like system()).

Here’s the basic idea: - Overwrite the return address with the address of system() - Place the argument (like "ps") somewhere on the stack - When the function returns, it behaves as if the program called system("ps")

In 32-bit systems, this was fairly straightforward because function arguments were passed on the stack, so you could place the string "ps" right after the return address.

Challenges on 64-bit Systems

However, in 64-bit binaries—like the one you're working with—things are more complicated.

Function arguments in 64-bit systems (System V calling convention) are passed via registers, not the stack: - The first argument goes into RDI - The second goes into RSI, and so on

This makes it difficult to pass a custom argument (like "ps") to system() via a stack overflow, because you don’t control the RDI register directly through a simple overflow.

In other words, you can still overwrite the return address to call system(), but you have no easy way to set up RDI with the string you want to execute.

This is why return-to-libc attacks are more challenging in 64-bit environments. There are ways to solve it (e.g., chaining gadgets in ROP), but they’re more advanced and go beyond the scope of this first lab.

Wrapping Up Lab 1

So, to recap: - The first exploit is a return-to-shellcode using a buffer overflow and a NOP sled - The second exploit attempts a return-to-libc attack by redirecting control to system("ps") - These are foundational techniques taught in introductory security courses like CS 165 - If you haven’t taken such a course, don’t worry—try it anyway - If one week feels tight, just focus on completing one of the two techniques. That’s a good start!

Wrapping Up and Lab Setup Summary

Again, depending on your background, don’t worry if you can’t complete both attacks perfectly in the first week. This isn’t about filtering people out. If you can get at least one working, that’s a strong indication that you can learn quickly—even if you haven’t taken a formal security class before.

32-bit Compilation Required

Make sure you compile a 32-bit binary. For the first attack (code injection), you can technically use a 64-bit binary, but for the second attack (return-to-libc), you’ll need 32-bit because it relies on passing arguments via the stack—something that doesn’t work the same way in 64-bit systems.

So yes, your binary might differ slightly from mine. You may see different memory layouts, offsets, or addresses—that's normal. Just take notes and adapt accordingly.

Lab Environment & Platform Notes

Let me go over a few environment recommendations:

  • The preferred OS is Linux—specifically a Linux distribution like Ubuntu, which is widely used and well-supported.
  • If you’re using Windows, WSL (Windows Subsystem for Linux) works fine.
  • But note that WSL installs a 64-bit environment by default, so to compile 32-bit binaries, you need to install the gcc-multilib package and use the -m32 flag.

Compilation and Security Flags

When compiling, use these flags:

  • -m32 to generate a 32-bit binary
  • -fno-stack-protector to disable stack canaries
  • -z execstack to make the stack executable

These are necessary for your exploit to work properly. Otherwise, built-in protections will stop your code from executing or crashing at runtime.

Disabling ASLR

ASLR (Address Space Layout Randomization) randomizes memory addresses every time you run a program—stack, heap, and code segment starting points are all different. This makes memory-based attacks unreliable.

To keep things simple for the lab, we disable ASLR. You have two options:

  1. System-wide method using sudo:

    • Requires root access (e.g., echo 0 > /proc/sys/kernel/randomize_va_space)
    • Works if you're on your own machine or have admin rights

  2. Temporary shell method (recommended for lab computers):

    • Run a special script I’ll provide
    • This launches a new shell with ASLR disabled
    • Any child processes launched from this shell will also have ASLR off
    • Just exit the shell when you’re done

This method doesn’t require sudo, so it works well on shared lab machines.

Lab Files and Next Steps

You’ll be working with a provided binary called something like:

codeinjection.bin

The download link will be on the course website. After downloading it, you can begin inspecting it in Ghidra or GDB.

There are two tasks:

  1. Code Injection Attack using buffer overflow and NOP sled
  2. Return-to-libc Attack calling system("ps") by overwriting the return address

Try both if you can—but again, even completing just the first task is a solid start.

Final Notes on Lab 1: Input Crafting and Workflow

One question you might be wondering is: how do I actually inject this payload into the binary?

There are a couple of ways to do it:

  1. Write a script — You can use a simple shell script or Python script to generate the payload and write it to a file.
  2. Use a hex editor — There are tools like hexedit that allow you to edit binary files directly. You can search for one online and use it to modify the input at the byte level.

Once you’ve crafted your payload, you’ll feed it to the program like this:

./vulnerable_binary < code_injection_input.bin

If your attack is successful, the program might print "yes" and then crash—that’s expected. A crash after successful output often indicates that your exploit worked.

Debugging Tips

You can also attach a debugger (GDB) to the running process. In fact, attaching to a process gives you the most accurate view of the actual runtime memory layout.

Instead of running the program inside GDB (which can introduce slight shifts), do this:

  1. Start the program in one terminal.
  2. Find its PID (process ID).
  3. In another terminal, attach GDB to it:
gdb -p <PID>

This way, GDB attaches to the running process without altering the environment, so the stack addresses you see match what you'd get during normal execution.

Note: You may need special permissions to attach to a process, but typically you can attach to a program you launched yourself.

Return-to-libc Task

The second task in the lab is a return-to-libc attack.

Here, you’ll construct an exploit that calls the system() function from libc with the argument "ps", so the process list is printed—without injecting any custom code.

To do this, you’ll need to: - Overwrite the return address with the address of system() - Provide the argument ("ps") on the stack

You can find the address of system() using GDB:

(gdb) p system

GDB will return the runtime address of system(), which you’ll use to overwrite the return address.

I’ve provided you with a template binary. The string "ps" is already embedded—you just need to: - Insert the address of system() as the new return address - Insert the address of the "ps" string in the correct place on the stack (often right after the return address)

There are two values you’ll typically need to modify: 1. The address to jump into system() 2. The address of the "ps" argument

Submitting the Lab

You don’t need to write any code for this lab. All you need to do is: - Modify the binary inputs as described - Generate and test your exploits - Submit a report that includes: - Screenshots - Explanations of what you did - What worked or didn’t work - Any observations from using GDB, Ghidra, etc.

Think of this lab as an exercise in understanding low-level program behavior and how exploits work in practice.

A Slack Workspace for Help

I’ve also created a Slack workspace for the class. You should all be enrolled by default.

If you get stuck, feel free to message me there—for example: - “I’m not sure what to change in the template…” - “I can’t locate the correct return address…” - “GDB shows a different address than expected…”

I’ll try to respond quickly and help you get unblocked.

Final Thoughts

This lab is more about exploring and understanding, not writing code from scratch.

  • For the first task, you're just editing input files to trigger code execution.
  • For the second, you're observing tool behavior and tweaking a prebuilt template.

If you know what you're doing, you might finish quickly. But if you're new to this, it may take some time to think through the mechanics—and that’s the point. This lab challenges your understanding of binary-level behavior.

So, take your time, ask questions, and enjoy the process!

That wraps up our first lecture. Any final questions before we call it a day?