Linux interviews for SRE, DevOps, backend and data engineering roles are mostly troubleshooting. You get a symptom (a full disk that du cannot explain, a load average of 16 with idle CPUs, a container that ignores docker stop) and you are expected to explain the mechanism and the next command to run.
Candidates slip on a predictable set of details: the difference between TIME_WAIT and CLOSE_WAIT, what load average actually counts, why RSS overstates memory, why PID 1 ignores SIGTERM, and how a deleted file can still fill a disk. Each one is a single fact about the kernel that turns a guess into a diagnosis.
This page has 30 practice questions in five areas: processes and signals, files and disks, memory and performance, networking and DNS, and shell, systemd and containers. Answers cite real defaults and commands, with shell snippets you can run on a Linux host.
Intermediate30 questions
Linux Interview Questions
Practice Linux interview questions on processes, signals, file systems, memory, load, TCP, DNS, systemd and containers, with worked answers and commands.
Site Reliability EngineerDevOps EngineerBackend EngineerPlatform EngineerData Engineer
Process questions are the opening move in most Linux rounds because they test whether you understand what the kernel is doing beneath every service you run. Expect graceful shutdown, zombies, PID 1 in containers and processes that ignore kill -9.
One idea covers most of them: a process is created by fork(), transformed by execve(), and must be reaped by its parent with wait(). Signals are the only way to interrupt one from outside, and each signal has a default action that a process may or may not override.
Processes and Signals
During a deploy, kill <pid> leaves the old worker running for almost a minute, so someone changes the script to kill -9. Explain the difference between SIGTERM and SIGKILL and write the stop logic you would ship.
EasyProcesses and Signals
Sample Answer
kill sends SIGTERM (15) by default. A process can catch it, stop accepting work, finish in-flight requests, flush buffers and exit cleanly. SIGKILL (9) cannot be caught, blocked or ignored: the kernel ends the process immediately, so nothing is flushed, temp and lock files stay behind, and clients see reset connections.
The standard pattern is SIGTERM, wait with a deadline, then SIGKILL as a last resort. systemd does exactly this (TimeoutStopSec defaults to 90 s), docker stop waits 10 s, and Kubernetes waits terminationGracePeriodSeconds (30 s by default). If a worker needs a minute to drain, fix its shutdown handler or raise the grace period rather than skipping straight to SIGKILL.
Bash
1kill -TERM"$pid"2for_in$(seq30);do3kill -0 "$pid"2>/dev/null ||exit0# process is gone: clean stop4sleep15done6echo"still running after 30 s, sending SIGKILL">&27kill -KILL "$pid"
ps shows hundreds of processes in state Z, marked <defunct>, all with the same parent, and kill -9 on them does nothing. What are they, are they dangerous, and how do you clear them?
MediumProcesses and Signals
Sample Answer
A zombie is a child that has already exited but whose parent has not called wait() or waitpid() to collect its exit status. The kernel frees its memory and other resources but keeps the process table entry so the status can still be read. kill does nothing because the process is already dead.
The cost is PIDs, not memory. Enough zombies exhaust kernel.pid_max or a cgroup's pids.max, and then fork() fails with EAGAIN for everything on the host or in the container.
The fix is in the parent: reap children in a SIGCHLD handler, or set SIGCHLD to SIG_IGN so the kernel reaps them automatically. For immediate relief, restart the parent. Its zombies are re-parented to init (or a subreaper), which reaps them.
Your container ignores docker stop: it is always killed after 10 seconds, and zombie processes pile up inside it. The Dockerfile ends with CMD ./start.sh. Explain both symptoms and fix them.
HardProcesses and Signals
Sample Answer
CMD ./start.sh is shell form, so the runtime runs /bin/sh -c ./start.sh and a shell becomes PID 1 in the container's PID namespace, with your app as its child. PID 1 is special: from inside its namespace, and from the host for anything but SIGKILL and SIGSTOP, the kernel delivers only signals for which PID 1 installed a handler. The shell installed none for SIGTERM and does not forward it, so nothing happens until the runtime sends SIGKILL after its grace period.
Orphans are re-parented to PID 1, and a PID 1 that never calls wait() leaves them as zombies.
Fix: use exec form CMD ["./start.sh"], end the script with exec so the app replaces the shell, and run a tiny init (docker run --init) if the app spawns children.
Explain what happens at the system call level when you type ls -l > out.txt in bash. Then explain why cd has to be a shell builtin rather than a program like /usr/bin/cd.
MediumProcesses and Signals
Sample Answer
Bash calls fork() (implemented with clone() on Linux) to create a child that is a copy of the shell. Memory is shared copy-on-write, so the copy is cheap. In the child, bash opens out.txt with O_WRONLY|O_CREAT|O_TRUNC and uses dup2() to put it on file descriptor 1. Then execve("/usr/bin/ls", ...) replaces the program image. The PID stays the same and open descriptors survive, except those marked close-on-exec. The parent calls waitpid(), which reaps the child and sets $?.
The working directory is a per-process attribute. A child that calls chdir() changes only its own directory and then exits, leaving the shell where it was. That is why cd, export, ulimit and source must run inside the shell process itself.
A process reading from an NFS mount shows state D in ps. kill -9 has no effect, and the load average keeps climbing while the CPUs sit idle. Explain what you are seeing and how you investigate.
MediumProcesses and Signals
Sample Answer
D is uninterruptible sleep: the task is blocked inside the kernel, usually waiting on I/O, in a code path that cannot safely be interrupted. Signals, SIGKILL included, stay pending until the wait ends. Some waits are written as killable and do respond to SIGKILL, but a hard NFS mount to an unreachable server can block indefinitely.
On Linux the load average counts tasks in state D as well as runnable ones, so blocked I/O raises load with zero CPU use.
Investigate with ps and the wchan column to see where tasks wait, cat /proc/<pid>/stack as root, and dmesg for hung task warnings ("blocked for more than 120 seconds" by default). The fix is the storage path itself: restore the NFS server or disk, or lazily unmount with umount -l.
Bash
ps -eo pid,stat,wchan:32,cmd |awk'NR == 1 || $2 ~ /^D/'sudocat /proc/<pid>/stack
dmesg -T |grep -i 'blocked for more than'
You start a long backfill over SSH with ./backfill.sh &, close your laptop, and the job dies. Why, and what are the right ways to run it?
EasyProcesses and Signals
Sample Answer
Closing the connection hangs up the terminal, and the session's shell receives SIGHUP. Bash exits on SIGHUP, and before exiting an interactive shell resends SIGHUP to all its jobs. The default action for SIGHUP is to terminate. & only puts the job in the background of the same session, so it gets the signal too.
Options, from quick to robust: nohup ./backfill.sh > backfill.log 2>&1 & ignores SIGHUP and keeps output; disown -h removes the job from bash's SIGHUP list; setsid starts it in a new session with no terminal; tmux keeps a terminal you can reattach to. For anything that matters, run it under systemd with systemd-run --unit=backfill ./backfill.sh, which gives you supervision, journalctl -u backfill logs and a clean way to stop it.
"The disk is full" is one of the most common pages an on-call engineer gets, and interviewers love it because the obvious answer is usually wrong. These questions cover inodes, links, deleted files that still hold space, permissions and reading disk latency.
The key idea is that a file name is just a directory entry pointing at an inode. Space is freed only when no names and no open descriptors refer to the inode, and permissions apply to every directory along the path.
Files, Disks and Permissions
Explain the difference between a hard link and a symbolic link in terms of inodes. Which one survives deleting the original file, and which one can point to another filesystem?
EasyFiles, Disks and Permissions
Sample Answer
A directory entry maps a name to an inode number. The inode holds metadata and the location of the data, but not the name. A hard link is a second name for the same inode, so the link count goes up. rm calls unlink(), which removes one name; the data is freed only when the count reaches zero and no process still has the file open. So the hard link survives deleting the original.
Hard links cannot cross filesystems, because inode numbers are only unique within one, and cannot point to directories. A symbolic link is a separate inode whose content is a path. It can cross filesystems and point to directories, but it dangles when the target is removed or renamed.
Bash
1echo data > a.txt
2ln a.txt hard.txt # same inode, link count becomes 23ln -s a.txt soft.txt # new inode that stores the path "a.txt"4ls -li a.txt hard.txt soft.txt
5rm a.txt
6cat hard.txt # data7cat soft.txt # No such file or directory
Alerts say /var is 100% full. df -h agrees, but du -sh /var adds up to 30 GB less than df reports. Someone already deleted a huge log file. Where did the space go, and how do you get it back?
MediumFiles, Disks and Permissions
Sample Answer
Deleting a file removes its name, but the inode and blocks stay allocated while any process holds it open. A logger still writing to the deleted file keeps the space in use. du walks names, so it cannot see the file, while df reads the filesystem's block counts, so it still sees the space. The gap is the deleted-but-open file.
lsof +L1 lists open files with a link count below one. Restarting or signalling the process to reopen its log releases the space. When you cannot restart, truncate the file through /proc/<pid>/fd/<n>.
Prevent it with log rotation that tells the process to reopen files (a postrotate signal), or copytruncate when the app cannot reopen.
Bash
1df -h /var;sudodu -sh /var 2>/dev/null # numbers disagree2sudolsof +L1 /var # open files that have been deleted3# free the space without a restart: truncate through the open descriptor4sudo truncate -s 0 /proc/<pid>/fd/<fd>
Writes fail with No space left on device, but df -h shows the filesystem 40% free. What else can be exhausted, how do you confirm it, and how do you find the cause?
MediumFiles, Disks and Permissions
Sample Answer
The usual answer is inodes. Every file and directory needs one, and on ext4 the inode count is fixed when the filesystem is created (for most sizes mke2fs defaults to one inode per 16 KiB of space). Millions of tiny files, such as session files, cache entries or mail spool messages, can use every inode while most blocks are free. df -i shows IUse% at 100%.
Find the directory with the most entries and clean or redesign it, for example by sharding or using an object store. XFS allocates inodes dynamically, so this mostly bites ext4.
Other causes worth naming: you checked a different mount than the one the app writes to, such as a small tmpfs on /tmp, or a disk quota, which reports Disk quota exceeded instead.
Bash
df -i /var # IUse% at 100% means no inodes leftsudofind /var -xdev -type f |awk -F/ '{print "/"$2"/"$3"/"$4}'\|sort|uniq -c |sort -rn |head# directories holding the most files
A deploy user can run ls /srv/app/releases and see file names, but cd /srv/app/releases fails with Permission denied. Explain what read, write and execute mean on a directory, and how you debug a path like this.
MediumFiles, Disks and Permissions
Sample Answer
On a directory, r lets you list the names (read the directory), x lets you traverse it (look up entries by name, cd into it, open files beneath it), and w lets you create, delete and rename entries, which also requires x. With r but not x, ls shows names but cannot stat them, and cd fails.
Every component of a path needs x for the user, so /srv, /srv/app and /srv/app/releases all matter. namei -l prints the owner and mode of each component in one go.
Two related traps: deleting a file depends on the directory's permissions, not the file's, unless the sticky bit is set (as on /tmp, mode 1777). And the default umask of 022 gives new files 644 and new directories 755.
On a database host, iostat -x 1 shows the NVMe device at 100% %util, r_await 0.2 ms, w_await 45 ms and aqu-sz around 30. Is the disk saturated? What do you check next?
HardFiles, Disks and Permissions
Sample Answer
%util is the share of time the device had at least one request in flight. The iostat man page says it shows saturation only for devices that serve requests serially; for RAID arrays and modern SSDs, which serve many requests in parallel, 100% does not mean the device is at its limit. Do not stop at that column.
The await columns include time queued plus time in service. Fast reads with 45 ms writes and a queue of 30 point at the write path: fsync storms, a checkpoint, or the SSD slowing under sustained writes.
Next, find the writer with pidstat -d 1 or iotop, compare w/s and wkB/s with what the device is rated for, and read /proc/pressure/io to see how much time tasks actually stall on I/O.
A service fails with Too many open files after running for about a day. How do you tell a descriptor leak from a limit that is simply too low, and where do you raise the limit correctly?
MediumFiles, Disks and Permissions
Sample Answer
Every open file, socket, pipe and epoll instance uses a descriptor, capped per process by RLIMIT_NOFILE, and hitting it returns EMFILE. Under systemd the default soft limit is 1024, with a hard limit of 524288.
Compare the count in /proc/<pid>/fd with the limit in /proc/<pid>/limits over time. A count that climbs steadily while traffic is flat is a leak; grouping descriptors by type usually shows thousands of sockets, often stuck in CLOSE_WAIT because the code never closes them.
To raise it for a service, set LimitNOFILE= in the unit file. /etc/security/limits.conf applies to PAM login sessions, not to systemd services. In containers the runtime sets the limit. Raising the limit on a leak only delays the outage.
Troubleshooting rounds hand you tool output and ask what it means. You are expected to read free, ps, vmstat, load average and OOM messages correctly, and to know which tool answers which question without slowing the host down.
The most useful habit is to separate busy from blocked. A busy system has runnable tasks and high CPU; a blocked one has tasks waiting on I/O, locks or memory. Load average, vmstat and pressure stall information all tell you which one you are looking at.
Memory, CPU and Performance Triage
free -h on a 32 GiB server shows only 0.5 GiB free and 24 GiB under buff/cache. A teammate wants to add RAM. What do you tell them, and what would real memory pressure look like?
EasyMemory, CPU and Performance Triage
Sample Answer
Linux uses otherwise idle memory as page cache for file data, because empty RAM does nothing useful. Clean cached pages are dropped the moment an application needs memory, so a low free column is normal on a healthy server.
Read the available column instead. It is the kernel's estimate (MemAvailable) of memory that can be handed to new allocations without swapping, and it includes reclaimable cache.
Real pressure shows up as low available, continuous swap activity in the si and so columns of vmstat 1, rising major page faults, non-zero stall times in /proc/pressure/memory, and OOM kills in dmesg. Used swap on its own is not a problem; steady swapping in and out is. Dropping caches with drop_caches fixes nothing and makes the next reads slower.
Your dashboard sums RSS across 40 pre-forked worker processes of a web server and reports 48 GB on a 32 GB machine that is not swapping. Explain VSZ, RSS and PSS, and pick the right metric.
MediumMemory, CPU and Performance Triage
Sample Answer
VSZ is all virtual address space the process has mapped, including reserved ranges never touched and mapped files. Runtimes such as the JVM and Go reserve large ranges, so VSZ is usually meaningless for capacity.
RSS counts pages currently resident in RAM for that process, including shared ones: libraries, shared memory and copy-on-write pages inherited from the master process at fork(). Pre-forked workers share most of their pages, so summing RSS counts the same memory up to 40 times.
PSS (proportional set size) divides each shared page by the number of processes mapping it, so PSS values add up correctly. Read it from /proc/<pid>/smaps_rollup. For a container, the number the limit enforces is the cgroup's memory.current, which also includes page cache.
Bash
forpin$(pgrep -f gunicorn);doawk'/^Pss:/ {print $2}' /proc/$p/smaps_rollup # kBdone|awk'{sum += $1} END {printf "total PSS: %.1f GiB\n", sum / 1048576}'
A containerized service died with exit code 137 and no stack trace. How do you confirm it was the OOM killer, tell a container limit from host-wide memory exhaustion, and explain which process the kernel picks?
HardMemory, CPU and Performance Triage
Sample Answer
Exit code 137 is 128 + 9: the process received SIGKILL. The OOM killer logs to the kernel ring buffer, so check dmesg or journalctl -k on the node for Out of memory or oom-kill lines. Kubernetes also reports OOMKilled in the container status.
In cgroup v2, a cgroup that hits memory.max and cannot reclaim enough invokes the OOM killer inside that cgroup, even if the host has plenty free. The oom_kill counter in that cgroup's memory.events increments. A global OOM has no such limit involved.
The victim is the task with the highest badness score, 0 to 1000 based on memory and swap use relative to what it may use, plus oom_score_adj (from -1000 to 1000; -1000 exempts the task). memory.high throttles before the hard limit kills.
An 8-core host shows load average: 16.2, 15.8, 12.1, yet top reports the CPUs about 70% idle. How can both be true, and what do you check?
MediumMemory, CPU and Performance Triage
Sample Answer
The Linux load average is an exponentially damped average over 1, 5 and 15 minutes of tasks that are runnable (state R) or in uninterruptible sleep (state D). Counting D means tasks blocked on disk, NFS or some kernel locks raise the load without using any CPU. 16.2 against 12.1 also tells you the load is rising.
So high load with idle CPUs points at blocked tasks. In vmstat 1, compare r (runnable) with b (blocked) and look at wa. List D tasks with ps and their wchan. Check iostat -x 1 for the device they wait on.
Load above the core count with busy CPUs and a large r column is the other case: genuine CPU saturation. /proc/pressure/cpu and /proc/pressure/io separate the two directly.
You are paged: API latency is high on one Linux host, and you have SSH and five minutes. Which commands do you run, in what order, and what is each one for?
MediumMemory, CPU and Performance Triage
Sample Answer
Go from broad to specific and check each resource for utilization, saturation and errors.
uptime gives the load trend. dmesg -T | tail shows OOM kills, disk errors or NIC resets. vmstat 1 shows run queue, blocked tasks, swapping and the CPU split. mpstat -P ALL 1 catches one hot core (a single-threaded bottleneck or interrupt load). pidstat 1 names the process. iostat -xz 1 covers disk latency and queues. free -m covers memory. sar -n DEV 1 shows NIC throughput, and sar -n TCP,ETCP 1 shows retransmits.
Most of these come from the sysstat package. In an interview, say what you expect each command to show before you run it, and say what you have ruled out. The narration is graded as much as the commands.
Bash
1uptime2dmesg -T |tail -20
3vmstat154mpstat -P ALL 135pidstat 136iostat -xz 137free -m
8sar -n DEV 139sar -n TCP,ETCP 13
One process sits at 100% of a core, and another is hung at 0% CPU. Which tool do you use on each, what do you look for, and what is the production risk of each tool?
HardMemory, CPU and Performance Triage
Sample Answer
For the busy process, sample it with perf record -F 99 -g -p <pid> for 30 seconds and read perf report. Sampling at 99 Hz is cheap and shows hot functions with call stacks (JIT runtimes need perf map support).
A process at 0% CPU is blocked, not busy. strace -p <pid> shows the system call it is stuck in: futex means a lock or condition wait, read or recvfrom means waiting on a descriptor (resolve it in /proc/<pid>/fd), connect means a network dial. /proc/<pid>/wchan shows the kernel wait point.
The risk is strace itself. It uses ptrace and stops the tracee on every traced syscall, which can slow a syscall-heavy process by an order of magnitude or more. Filter with -e trace= and keep sessions short.
Most production incidents that are not code bugs are network or DNS problems, so SRE and backend interviews spend real time here: socket states, port exhaustion, name resolution and firewalls, usually framed as "connections are failing, find out why".
Learn the TCP state machine well enough to know which side owns each state. TIME_WAIT belongs to whoever closed first and expires on its own. CLOSE_WAIT belongs to whoever has not yet called close() and never expires, so it always points at an application.
Networking and DNS
Your service fails to start with bind: Address already in use on port 8080. Find what holds the port. Then explain why the service can be listening and still refuse connections from other hosts.
EasyNetworking and DNS
Sample Answer
ss -ltnp lists listening TCP sockets with numeric ports and the owning process; run it as root to see processes of other users. Filter with 'sport = :8080'. The holder is often a previous instance still shutting down, or a stray copy started by hand.
The second problem is the bind address. A server bound to 127.0.0.1:8080 accepts only connections that originate on the same host. Binding to 0.0.0.0:8080 (or [::]:8080 for IPv6) accepts on every interface. This bites most often in containers, where an app bound to loopback inside the container cannot be reached through the published port.
If the address is right and remote clients still fail, check the firewall next.
Bash
sudo ss -ltnp 'sport = :8080'# LISTEN 0 4096 127.0.0.1:8080 0.0.0.0:* users:(("api",pid=4121,fd=7))# 127.0.0.1 means local clients only; bind 0.0.0.0 or [::] for remote clients
A batch job opens a new HTTP connection for each of 50,000 requests to one upstream. After a while connect() fails with Cannot assign requested address, and ss -s shows about 28,000 sockets in TIME_WAIT. Explain and fix.
HardNetworking and DNS
Sample Answer
The side that closes a TCP connection first holds it in TIME_WAIT for 2 x MSL, which Linux fixes at 60 seconds. It exists to absorb late segments and resend the final ACK.
Every outgoing connection to the same destination IP and port needs its own local port. The default ip_local_port_range is 32768 to 60999, or 28,232 ports, so a client can sustain only about 470 new connections per second to one destination before ports run out.
The real fix is connection reuse: HTTP keep-alive and a pool. Kernel-side, net.ipv4.tcp_tw_reuse=1 lets new outgoing connections reuse TIME_WAIT ports when it is safe (the default, 2, enables this only for loopback), and widening the port range helps. tcp_fin_timeout is the classic wrong answer: it controls FIN_WAIT_2, not TIME_WAIT.
Bash
1ss -s # TCP: ... timewait 281142sysctl net.ipv4.ip_local_port_range # 32768 60999 = 28,232 ports3echo$(((60999-32768+1)/60))# ~470 new connections/s per destination4sysctl net.ipv4.tcp_tw_reuse # 2 = loopback only
ss -tan state close-wait shows 4,000 sockets on your service, and the number only grows. Which side is at fault, and why will no sysctl fix it?
MediumNetworking and DNS
Sample Answer
CLOSE_WAIT means the peer sent FIN, the kernel acknowledged it, and the connection is waiting for the local application to call close(). The kernel does not time this state out on its own, because a half-closed connection may legitimately still be used to send data. A count that only grows therefore means your application is not closing sockets.
Typical causes: an error path that skips close(), an HTTP response body that is never read or closed, or a connection pool that never discards connections the server has closed. Each one also holds a descriptor, so the next symptom is Too many open files.
Group the sockets by peer to find which dependency is affected, then fix the code. A restart clears them only until they build up again.
Bash
# CLOSE_WAIT sockets grouped by remote address (last column without -p)ss -Htan state close-wait |awk'{print $NF}'|sed's/:[0-9]*$//'|sort|uniq -c |sort -rn |headsudo ss -tanp state close-wait |head -5 # which process owns them
A pod in Kubernetes calls api.partner.com and sees occasional 5-second stalls. The cluster DNS logs show lookups for names like api.partner.com.default.svc.cluster.local. Walk through the resolution path and fix it.
MediumNetworking and DNS
Sample Answer
An application lookup through glibc's getaddrinfo reads /etc/nsswitch.conf (typically hosts: files dns), checks /etc/hosts, then queries the nameservers in /etc/resolv.conf. Kubernetes writes a resolv.conf with several search domains and options ndots:5.
A name with fewer dots than ndots is tried with each search suffix before being tried as-is. api.partner.com has two dots, so every lookup first produces several NXDOMAIN queries, for both A and AAAA. Any dropped UDP packet then costs a full resolver timeout, 5 seconds by default.
Fixes: use a trailing dot (api.partner.com.) to mark the name absolute, lower ndots in the pod's dnsConfig, or run a node-local DNS cache. Test with getent hosts, which follows the application's path; dig skips nsswitch and the search list by default.
curl 10.0.3.7:8443/health fails. In one environment it reports Connection refused instantly; in another it hangs until it times out. What does each tell you, and how do you confirm it?
MediumNetworking and DNS
Sample Answer
Refused means the SYN reached a host that answered with a RST. The host is up and routable, but nothing is listening on that address and port, or a firewall rule rejects with a reset. Check ss -ltn on the server; often the app is down or bound to 127.0.0.1 only.
A timeout means the SYN got no answer at all: a firewall or security group dropping packets, a wrong route, a host that is down, or an overloaded server whose accept queue is full (by default Linux silently drops the handshake packets and lets the client retry).
Confirm with tcpdump on both ends (does the SYN arrive, does a SYN-ACK leave?) and the rule counters in nft list ruleset or iptables -L -n -v.
Bash
nc -vz -w 310.0.3.7 8443# refused vs timed out, in 3 ssudo ss -ltn 'sport = :8443'# on the server: is anything listening?sudo tcpdump -nn -i any 'tcp port 8443'# does the SYN arrive, does a SYN-ACK leave?
Write a minimal nftables ruleset for a web server: allow SSH and HTTPS in, allow replies to outbound traffic, drop everything else. Then explain an outage where dmesg showed nf_conntrack: table full, dropping packet.
HardNetworking and DNS
Sample Answer
The ruleset is stateful: ct state established,related accept admits replies to connections the kernel's connection tracking already knows, so you only open new inbound ports explicitly. Accept loopback and ICMPv6 too, or IPv6 neighbor discovery breaks.
Connection tracking keeps an entry per flow in a table capped by net.netfilter.nf_conntrack_max. When it is full, packets that would create new entries are dropped, which clients see as random timeouts. Established TCP entries live 432,000 seconds (5 days) by default, so many short-lived or abandoned flows fill it.
Compare nf_conntrack_count with nf_conntrack_max, then raise the max (each entry costs memory), shorten timeouts, or exempt high-volume traffic with notrack. On current distros iptables is often a front end that writes nftables rules.
Day-to-day Linux work is pipelines, service units and containers. Interviewers check that you can process logs with standard tools, run services under systemd without crash loops, and explain containers as kernel features rather than magic.
The underlying idea is composition. Small tools joined by pipes, units that systemd supervises, and containers built from namespaces plus cgroups all behave predictably once you know the rules of each part, and fail in confusing ways when you do not.
Shell, systemd and Containers
Given an nginx access log in the default combined format, print the 10 client IPs with the most 5xx responses. Then explain why uniq -c needs sort in front of it.
EasyShell, systemd and Containers
Sample Answer
In the combined format, whitespace splitting puts the client IP in $1 and the status code in $9, because the quoted request line splits into three fields. Filter on a 5xx status, print the IP, then count.
uniq collapses only adjacent duplicate lines, so unsorted input gives wrong counts. sort | uniq -c | sort -rn is the standard count-and-rank idiom. The second form counts in one awk pass with an associative array and only sorts the distinct IPs, which is faster on large files. LC_ALL=C sort also speeds up sorting a lot.
The trap: a malformed request line with extra spaces shifts the fields. For production analysis, log JSON or split on the quote character.
Bash
1awk'$9 ~ /^5[0-9][0-9]$/ {print $1}' access.log |sort|uniq -c |sort -rn |head -10
23# one pass, sort only the distinct IPs4awk'$9 ~ /^5/ {n[$1]++} END {for (ip in n) print n[ip], ip}' access.log |sort -rn |head -10
Compress every .log file older than 7 days under /var/log/app, where some file names contain spaces, using all CPU cores. Why does ls *.log | xargs gzip break?
MediumShell, systemd and Containers
Sample Answer
xargs splits its input on whitespace and treats quotes and backslashes specially, so my app.log becomes two arguments, my and app.log. Parsing ls output has the same problem plus any other odd characters.
Use find -print0 with xargs -0. Names are separated by a NUL byte, the one byte that cannot appear in a file name. -r (GNU) skips running gzip when nothing matches, -n 50 passes 50 files per gzip call, and -P "$(nproc)" runs one gzip per core. find ... -exec gzip {} + is the simpler single-process alternative.
One detail interviewers check: -mtime +7 ignores fractional days, so it matches files modified at least 8 full days ago.
A systemd service crash-loops, then stops restarting and systemctl status says start request repeated too quickly. Explain what systemd is doing, how you investigate, and write a unit that restarts sensibly.
MediumShell, systemd and Containers
Sample Answer
Restart=on-failure restarts after a non-zero exit, an unclean signal, a timeout or a watchdog failure. Exits by SIGTERM, SIGINT, SIGHUP or SIGPIPE count as clean and are not restarted. Separately, start rate limiting refuses more than StartLimitBurst starts (default 5) within StartLimitIntervalSec (default 10 s). With the default RestartSec of 100 ms, a service that crashes on startup hits that limit within a second or two and is left failed.
Investigate with systemctl status app for the exit code, and journalctl -u app -b --since "15 min ago" for the output. After fixing, clear the counter with systemctl reset-failed app.
A sensible unit waits between restarts and allows a few attempts over minutes. The StartLimit settings belong in [Unit], not [Service].
Explain what a container is in terms of Linux kernel features. With only a root shell on the host, show that a containerized process is an ordinary host process.
HardShell, systemd and Containers
Sample Answer
The kernel has no container object. A container is a process tree with its own namespaces, which control what it can see (mount, PID, network, UTS, IPC, user, cgroup and time), placed in a cgroup, which controls what it can use. In cgroup v2 that means files such as memory.max, cpu.max and pids.max in a single hierarchy under /sys/fs/cgroup. Add a root filesystem from the image (usually overlayfs), dropped capabilities and a seccomp filter.
From the host, the container's processes appear in ps with host PIDs. /proc/<pid>/ns/ shows which namespaces they belong to, /proc/<pid>/cgroup shows their cgroup, and nsenter runs a command inside their namespaces. Because every container shares the host kernel, a kernel vulnerability can break isolation, which is the main difference from a VM.
Bash
1pid=$(pgrep -o -f 'nginx: master')# a containerized process, seen from the host2sudols -l /proc/$pid/ns/ |awk'NR > 1 {print $9, $11}'# compare with /proc/self/ns3cat /proc/$pid/cgroup # 0::/system.slice/docker-<id>.scope4sudo nsenter -t $pid -n ip -brief addr # run ip inside its network namespace5sudo unshare --pid --fork --mount-proc sh -c 'echo "my pid: $$"'# prints 1
A latency-sensitive service in Kubernetes has a CPU limit of 2 cores. Average usage is 40% of that limit, yet p99 latency spikes every few seconds. What is happening, and how do you prove it?
HardShell, systemd and Containers
Sample Answer
A CPU limit becomes CFS bandwidth control: in cgroup v2, cpu.max reads 200000 100000, meaning 200 ms of CPU time per 100 ms period, summed across all cores. A process with 16 busy threads (a GC phase, a thread pool burst) can burn the whole 200 ms in about 12.5 ms of wall time, then gets throttled for the rest of the period. Averages over a minute look fine while individual requests stall for up to about 88 ms.
Prove it with cpu.stat: compare nr_throttled with nr_periods, and watch throttled_usec grow during the spikes.
Fixes: raise or remove the CPU limit while keeping the request, size thread pools and runtime CPU counts to the limit rather than the host's cores, or allow short bursts with cpu.max.burst.
Bash
1cat /sys/fs/cgroup/cpu.max # 200000 100000 = 2 CPUs per 100 ms period2cat /sys/fs/cgroup/cpu.stat # nr_periods, nr_throttled, throttled_usec3awk'/^nr_periods/ {p=$2} /^nr_throttled/ {t=$2}
4 END {printf "throttled in %.1f%% of periods\n", 100 * t / p}' /sys/fs/cgroup/cpu.stat
Review this deploy script line: cd $DEPLOY_DIR; rm -rf build/*; tar xzvf $ARTIFACT | tee extract.log. What can go wrong, and how do you harden it?
EasyShell, systemd and Containers
Sample Answer
If DEPLOY_DIR is empty or wrong, cd fails, the script carries on, and rm -rf build/* runs in whatever directory the script started in. Unquoted variables are split on spaces and glob-expanded. And a pipeline's exit status is the last command's, so a corrupt archive that makes tar fail still reports success because tee succeeded.
Harden it with set -euo pipefail: exit on errors, fail on unset variables, and fail a pipeline if any stage fails. Quote every expansion, guard required variables with ${VAR:?message}, and use -- before paths.
Know the limits of -e: it does not fire for commands tested in if or while conditions, or for any command in an && or || list except the last, so explicit checks still matter. Run shellcheck in CI.
macOS and BSD tools differ from Linux in ps, sed, xargs and /proc, which does not exist there. Spin up a small VM or container and reproduce each symptom on this page: a zombie, a deleted-but-open file, a full inode table. Processes and signals and Files and descriptors walk through these step by step.
Learn to read tool output cold
Interviewers paste output from vmstat, iostat, free or ss and ask what is wrong. Practice naming every column you rely on and the number that would worry you. CPU and load, Memory and Disk and I/O use real output throughout.
Know the TCP state machine and the DNS path
Draw the TCP close sequence from memory and say which side holds each state and for how long. Then trace a name lookup from getaddrinfo to the nameserver. TCP in production and DNS and load balancing cover both with production failure cases.
Rehearse a troubleshooting method out loud
In a live round, a structured approach beats a memorized command list. State the symptom, form a hypothesis, pick the command that tests it, and say what you ruled out. The troubleshooting round teaches the method, and Lab: the website is down lets you practice it on a broken system.
Get fast at text processing
Many loops include a practical scripting task such as parsing a log, counting errors per host or finding the slowest endpoints. Be fluent with awk, sort, uniq, xargs and grep, and know when to switch to Python. Log parsing at scale and The coding round have timed exercises.
How Ready Are You for Linux Interviews?
1 / 6
Disks
df says /var is 100% full but du reports 30 GB less. What is the most likely cause?
Frequently Asked Questions
What Linux topics come up most in SRE and DevOps interviews?
Expect processes and signals, file systems and full disks, memory and the OOM killer, load average and CPU, TCP connection states, DNS resolution, and systemd services. Container roles add namespaces and cgroups. Most of it is asked as troubleshooting: you are given a symptom or a block of tool output and asked what is happening and what you would run next, so practice explaining the mechanism, not just naming the command.
Do data engineers and backend engineers get Linux questions?
Yes, though usually lighter than SRE loops. Backend interviews often include connection problems, file descriptor limits, memory usage of services and reading logs from the command line. Data engineers get questions about disks filling up, processes killed by the OOM killer, long-running jobs that die when an SSH session closes, and text processing with awk, sort and xargs on large files.
How deep do I need to know Linux internals?
For most roles you need the kernel concepts that explain symptoms: inodes, file descriptors, process states, page cache, virtual versus resident memory, the TCP state machine, namespaces and cgroups. You do not need to read kernel source. Senior SRE and platform roles go further into performance tools such as perf, pressure stall information, cgroup v2 controllers and how the scheduler enforces CPU limits.
Which Linux commands should I know by heart?
Be fluent with ps, top, kill, df, du, lsof, free, vmstat, iostat, ss, ip, dig, curl, tcpdump, strace, journalctl, systemctl, and the text tools grep, awk, sed, sort, uniq and xargs. More important than flags is knowing which command answers which question and how to read its output, since interviewers usually paste output and ask you to interpret it.
How long does it take to prepare for a Linux interview?
If you already operate Linux servers, two to three weeks of focused practice usually covers it: one area from this page every few days, plus reproducing each failure on a VM. Coming mainly from application development, plan four to six weeks, because reading vmstat, iostat and ss output fluently takes repetition on a real system rather than reading about it.
Can I practice Linux interview questions on a Mac?
Only partly. macOS shares the shell and many command names, but ps, sed, xargs and top behave differently, and /proc, systemd, cgroups and most of the networking tools are missing. Use a Linux VM or a container for practice, and use a full VM for anything involving systemd, the OOM killer or kernel settings, since containers share the host kernel.
Written by
Dan Lee
Data & AI Lead
Dan is a seasoned data scientist and ML coach with 10+ years of experience at Google, PayPal, and startups. He has helped candidates land top-paying roles and offers personalized guidance to accelerate your data career.