Overview
Understanding Linux beneath the shell — how the kernel schedules processes, manages memory, serves filesystems, and moves packets — is what separates "it works on my machine" from being able to reason about any production system. It is also the substrate of containers: namespaces and cgroups are the container. Reference anchors: Brendan Gregg (Systems Performance), Love (Linux Kernel Development), LWN.
Key points
- Processes & scheduling: fork/exec, process states, signals; CFS/EEVDF scheduler, nice and cgroup weights; threads as tasks sharing an address space.
- Memory: virtual memory, page tables, page cache (why "free" memory isn't), demand
paging, OOM killer, swap;
mmap, huge pages. - Filesystems & I/O: VFS abstraction, ext4/XFS/Btrfs/ZFS, inodes, the page cache again, fsync semantics, io_uring as the modern async I/O interface.
- Networking stack: sockets, TCP state machine, netfilter/iptables/nftables, epoll (the engine under every event loop), eBPF reshaping all of it.
- Namespaces & cgroups: PID/net/mount/user namespaces + cgroup v2 resource control = containers demystified; systemd as PID 1 and its units.
- Observability toolbox:
strace/ltrace,perf, eBPF tools (bcc, bpftrace), /proc and /sys, the USE method (utilization, saturation, errors) for triage. - To explore: boot process (UEFI → initramfs → systemd), syscall mechanics (vDSO), kernel builds, real-time tuning.
Practice
- OverTheWire Bandit (source) — learn the Linux CLI and filesystem by solving level puzzles; the warm-up every sysadmin skill builds on.
- The Missing Semester (source) — MIT's exercises on the shell, scripting, tooling, debugging, and profiling — the tacit craft, made explicit.
- SadServers (source) — troubleshoot deliberately-broken Linux servers in a live sandbox; the USE method applied under pressure.
- Build your own shell (exercise) — fork/exec, pipes, redirection, and signals — the process model's syscalls, by hand.
- Linux From Scratch (source) — assemble a working Linux from source and understand every layer from bootloader to userland.
Related
- Check which process is listening on a given port
- Hardening Linux — the security angle on the same internals.
- Concurrency and Parallelism — OS threads and the scheduler are its foundation.
- Networking fundamentals — the protocol side of the kernel's network stack.
- Performance engineering — the tooling here applied with methodology.
- Kubernetes & containers deep dive — namespaces and cgroups productized.
- How a computer works — the hardware layer the kernel abstracts.