Secure the Ground Before You Build the Pipeline — Linux Hardening for DevOps Engineers

I have reviewed a lot of DevOps setups over the years. The pattern I keep seeing is the same: careful, well-thought-out pipeline security — secrets scanning, SAST, container image scanning, signed artefacts — running on a Linux host where root login over SSH is still enabled and auditd has never been touched.

The pipeline controls are real. The work that went into them is real. But they are sitting on ground that has not been hardened, and that matters more than most teams want to admit.

Why Teams Lock Down the Pipeline and Ignore the Host

The pipeline is visible. Every CI/CD tool gives you dashboards, pass/fail gates, scan reports. You can see the controls working. The host OS is quiet by comparison — it just runs. Nobody opens a ticket to say "the SSH config on the build server looks a bit loose."

There is also a specialisation gap. DevOps engineers know pipelines. They may not have come through a sysadmin route, and deep Linux hardening is traditionally sysadmin territory. The result is a gap at exactly the layer where an attacker who gets past your pipeline controls — or bypasses them entirely — would land.

The host is the last line of defence for your pipeline, and in most environments I have seen, it is the weakest one.

Linux Hardening Essentials

None of what follows is exotic. These are baseline controls. The point is not that they are advanced — it is that they are skipped.

SSH Configuration

SSH is almost always the initial access vector on a Linux build server that gets compromised. The default OpenSSH config is permissive enough to cause real problems.

Open /etc/ssh/sshd_config and enforce these settings:

# Disable root login entirely
PermitRootLogin no

# Key-based authentication only — no passwords
PasswordAuthentication no
PubkeyAuthentication yes

# Change the default port (reduces automated scan noise)
Port 2222

# Restrict to specific users if your setup allows it
AllowUsers deployuser cirunner

# Disable unused auth methods
ChallengeResponseAuthentication no
UsePAM yes

# Set an idle timeout — 5 minutes of inactivity disconnects the session
ClientAliveInterval 300
ClientAliveCountMax 0

After editing, restart the service and verify the new config before closing your existing session:

sudo sshd -t          # Validate config syntax — do this before restarting
sudo systemctl restart sshd

Changing the default port does not stop a determined attacker. It does dramatically reduce the volume of automated brute-force attempts hitting your logs, which makes genuine anomalies easier to spot.

Disable Unused Services

Every running service that you do not need is an attack surface you are maintaining for free. On a build server, the list of things that genuinely need to be running is short. Audit what is active:

systemctl list-units --type=service --state=running

Anything you cannot justify — disable it:

sudo systemctl disable --now avahi-daemon
sudo systemctl disable --now cups
sudo systemctl disable --now bluetooth

The principle is minimum necessary. A CI runner does not need a print service or a network discovery daemon. Remove what is not needed, and document what you left running and why.

auditd — Logging What Matters

auditd is the Linux kernel's audit framework. It records system calls, file access, privilege escalation, and authentication events in a tamper-resistant log. If something happens on the host, auditd is how you find out what it was.

Install and enable it:

sudo apt install auditd audispd-plugins   # Debian/Ubuntu
sudo systemctl enable --now auditd

A minimal but meaningful ruleset in /etc/audit/rules.d/hardening.rules:

# Monitor authentication events
-w /etc/passwd -p wa -k identity
-w /etc/shadow -p wa -k identity
-w /etc/sudoers -p wa -k sudoers

# Monitor SSH authorised keys changes
-w /home -p wa -k home_changes

# Log all privilege escalation
-a always,exit -F arch=b64 -S execve -F euid=0 -k root_commands

# Monitor crontab changes
-w /etc/cron.d/ -p wa -k cron
-w /var/spool/cron/ -p wa -k cron

Reload the rules with auditctl -R /etc/audit/rules.d/hardening.rules and verify with auditctl -l. Now you have a log trail that a security team — or a SIEM — can actually use.

fail2ban — Automated Brute-Force Protection

Even with key-only SSH auth, you want fail2ban running. It parses authentication logs and automatically blocks IP addresses that trigger repeated failures. It also works across other services.

sudo apt install fail2ban

Create a local override at /etc/fail2ban/jail.local:

[DEFAULT]
bantime  = 1h
findtime = 10m
maxretry = 5

[sshd]
enabled  = true
port     = 2222
logpath  = %(sshd_log)s
backend  = %(syslog_backend)s

sudo systemctl enable --now fail2ban
sudo fail2ban-client status sshd   # Verify it is watching the right service

These four controls — SSH hardening, service reduction, auditd, and fail2ban — are not a complete hardening programme. They are the floor. They are what I would put in place before anything else ran on that host.

Where CI/CD Introduces Its Own Attack Surface

Hardening the host closes the obvious OS-level gaps. But CI/CD pipelines introduce their own attack surface that sits on top of — and sometimes undermines — host-level controls.

Secrets in environment variables. Environment variables are convenient and widely used for passing credentials into build jobs. They are also visible in process listings, often echoed in verbose build logs, and accessible to any code running in the build context. Use a secrets manager — HashiCorp Vault, AWS Secrets Manager, or your CI platform's native secrets store — and inject credentials only at the point of use, not as ambient environment variables for the entire job.

Over-permissioned runners and build agents. A CI runner that has write access to your production AWS account, broad IAM permissions, or the ability to push to main without a review gate is not a convenience — it is a lateral movement path. Scope runner permissions to exactly what each job requires. Use separate runners for build, test, and deploy stages. Apply IAM least privilege to the roles those runners assume.

Unscanned base images. A pipeline that scans your application code but pulls an unscanned base image in the Dockerfile is not a hardened pipeline. Scan images with Trivy or Grype in the pipeline, and pin base images to specific digests — not latest. An unpinned ubuntu:latest in production is an uncontrolled dependency.

No image signing. If you are not signing container images, you cannot verify that the image running in production is the image your pipeline built. Cosign and Sigstore provide supply chain verification at the image layer. If image signing is not in your pipeline today, it belongs on the backlog.

Defence in Depth — How They Reinforce Each Other

Host hardening and pipeline security are not alternatives. They are layers in a defence-in-depth model, and they only work properly together.

A hardened host limits what an attacker can do if they reach the underlying OS — whether through a compromised dependency, a vulnerable runner, or direct exploitation. A secured pipeline limits what an attacker can do if they compromise the build process — injecting malicious code, exfiltrating secrets, or pivoting into the deployment environment.

If either layer is missing, the other layer is carrying more weight than it was designed to. A pipeline with excellent secret scanning but running on an unhardened host is one compromised SSH session away from an attacker who can read those secrets directly from disk. A hardened host running a pipeline with over-permissioned runners still gives an attacker production access the moment they compromise a build job.

The MITRE ATT&CK framework is useful here. Look at Initial Access (T1078 — valid accounts, T1190 — exploit public-facing application), Privilege Escalation (T1548), and Lateral Movement (T1021). Host hardening specifically addresses initial access and privilege escalation. Pipeline security specifically addresses supply chain compromise (T1195) and credential access (T1552). Neither is sufficient on its own.

Practical Checklist

Work through this before you call your pipeline hardened.

Host — Linux Baseline

• [ ] Root SSH login disabled (PermitRootLogin no)
• [ ] Password authentication disabled (PasswordAuthentication no)
• [ ] SSH running on a non-default port
• [ ] SSH AllowUsers restricted to named accounts
• [ ] Unused system services disabled and documented
• [ ] auditd installed, enabled, and ruleset configured
• [ ] fail2ban installed and watching SSH
• [ ] Unattended security updates enabled (unattended-upgrades)
• [ ] Firewall active with default-deny inbound (ufw or iptables)
• [ ] No world-writable files in critical directories (find / -perm -002 -type f)

Pipeline — CI/CD Security

• [ ] Secrets injected via secrets manager — not hardcoded or in plain env vars
• [ ] CI runner permissions scoped per job, not shared globally
• [ ] Build logs reviewed — no secrets echoed in output
• [ ] Base images pinned to digest, not tag
• [ ] Container images scanned with Trivy or Grype in every build
• [ ] Image signing in place (Cosign / Sigstore)
• [ ] SAST running on every pull request
• [ ] Dependency scanning (SCA) in the pipeline
• [ ] No direct push to main from the runner — review gate enforced
• [ ] Pipeline configuration (.github/workflows, .gitlab-ci.yml) reviewed for over-permission

---

The controls above are not aspirational. They are achievable in a working environment without significant overhead. The gap is not tooling — the tooling exists and most of it is free. The gap is that host hardening sits in a blind spot between sysadmin tradition and DevOps practice, and nobody owns it clearly.

Own it. The pipeline you have spent time securing deserves ground that will hold.

---

Where does your setup stand? If you are running CI/CD on Linux and have not worked through a host hardening baseline, start with SSH config and auditd. They take less than an hour and close the most common initial access paths. Everything else builds on that foundation.

---

#Linux #DevSecOps #Hardening #CICD #DevOps #SSH #Cybersecurity #PipelineSecurity #DefenceInDepth #CloudSecurity #SecurityEngineering #SOC #InfrastructureSecurity