linux-firewall-hardening

Linux Firewall Hardening

When to Use

• Check if a Linux server has active firewall protection.
• Enable and configure a firewall without locking yourself out of SSH.
• Audit existing rules, troubleshoot connectivity, or apply a security profile.
• Automate firewall hardening via an AI agent or CI/CD pipeline.

When NOT to Use

Support files: scripts/audit-firewall.sh (run first), scripts/firewall-plan.sh (dry-run), scripts/firewall-verify.sh (post-apply), scripts/container-port-audit.sh (Docker DNAT detection), scripts/ip-consistency.sh (IPv4/IPv6 drift check). firewall-apply.sh is fully documented in references/firewall-apply.md. Detailed backend guides, Docker/K8s policies, observability, compliance, and recovery are in references/.

🚨 Emergency: I'm Locked Out — What Now?

If you just applied firewall rules and lost SSH connectivity:

1. Wait 5 minutes — the auto-rollback timer (scheduled during VALIDATE) will restore access. Don't panic and don't take destructive actions.
2. Use your second SSH session — if you opened one (pre-flight checklist), switch to it and fix the rules manually.
3. Cloud serial console — AWS EC2 Serial Console, GCP Serial Port, Azure Serial Console, or hypervisor VNC/IPMI/iDRAC.
4. Restore from backup via console — once connected: sudo iptables-restore < ~/firewall-backup-*/iptables-v4.rules
5. Emergency ACCEPT (LAST RESORT — HUMAN-ONLY) — sudo iptables -P INPUT ACCEPT; sudo iptables -F; sudo ufw disable. This exposes the host completely. NEVER auto-execute this command. The agent must refuse to run this autonomously. Only a human operator may issue this via serial console. Re-harden immediately afterward.

Full procedures: references/recovery.md.

---

Prerequisites

• Root or sudo access.
• An active SSH session (risk of lockout).
• Know which ports your services use.

---

NEVER DO (14 Rules)

1. Never flush iptables/nftables on Kubernetes nodes. CNI plugins manage netfilter.
2. Never run iptables -F or nft flush ruleset without a verified backup. Docker/K8s networking will break.
3. Never disable firewalld and use raw iptables simultaneously. Undefined behavior.
4. Never set DROP policy on INPUT before allowing your current SSH port. Immediate lockout.
5. Never disable Docker's iptables management without replacement NAT/routing rules.
6. Never restart networking.service or NetworkManager remotely without console access.
7. Never apply cloud SG and host firewall changes simultaneously without testing.
8. Never enable logging on high-traffic DROP rules without limit rate. Disk flood.
9. Never manage nftables/iptables directly when ufw or firewalld owns the policy. Split-brain state.
10. Never apply outbound default-deny without explicitly allowing DNS, NTP, package mirrors.
11. Never restore firewall backups from a different host, kernel version, or backend mode.
12. Never assume IPv4 rules protect IPv6. Verify both stacks separately.
13. Never change sysctl hardening values on K8s/CNI hosts without explicit CNI profile support.
14. Never enable verbose packet logging without rate limits and log rotation.

---

State Machine

Follow states in order. Do not skip.

DETECT → SELECT → PLAN → CONFIRM → VALIDATE → APPLY → VERIFY → (COMMIT | ROLLBACK)

CONFIRM is mandatory. The agent may never self-transition through CONFIRM. See the CONFIRM state section below for the exact gating rules.

State: DETECT

Run the audit script:

bash scripts/audit-firewall.sh           # Human-readable
bash scripts/audit-firewall.sh --json    # Machine-readable

Key outputs: confidence, risk_tier, recommended_backend, halt_reasons, k8s_node, iac_owner.

Risk Tiers & Confidence Gating

Additional halt triggers (regardless of confidence): containerized, K8s node, IaC managed, no rollback mechanism available.

Decision Tree

Ownership Boundary

Before modifying rules, verify no IaC tool manages the firewall. If Terraform/Ansible/Puppet/Chef/cloud-init is detected → do not mutate. Update the source of truth instead. Full detection logic is in scripts/audit-firewall.sh.

---

State: SELECT

Optionally load a pre-built security profile (references/security-profiles.md):

Or use declarative YAML (references/declarative-policy.md):

Imperative (state machine) → Ad-hoc hardening, incident response
Declarative (YAML)        → GitOps, multi-host, reproducible
Mixed                     → YAML as source-of-truth, state machine for verification

---

State: PLAN

Generate a dry-run diff before applying:

bash scripts/firewall-plan.sh --profile public-web-server
bash scripts/firewall-plan.sh --ports 22,80,443
bash scripts/firewall-plan.sh --json     # Machine-readable diff with approval_token
bash scripts/firewall-plan.sh --refresh-audit --json  # Force re-audit + plan

Review the output. If risk_tier is confirmed, present the plan and wait for human confirmation before APPLY.

Plan JSON schema (matches firewall-plan.sh --json output):

{
  "backend": "ufw",
  "active_frontend": "ufw",
  "profile": "public-web-server",
  "target_ports": [22, 80, 443],
  "diff": {
    "add":    [{"port": 80, "proto": "tcp", "source": "any"}],
    "skip":   [{"port": 22, "proto": "tcp", "reason": "already_exists"}],
    "remove": []
  },
  "risk_assessment": "low",
  "estimated_disruption": "none",
  "approval_token": "sha256:abc123...",
  "audit_cached": false,
  "audit_cache_file": "/tmp/firewall-audit.json"
}

Approval gate: PLAN output includes an approval_token (hash of plan content). APPLY must be called with --approved-plan=. Token mismatch → exit code 41. This forces explicit human confirmation before Apply.

Audit caching: firewall-plan.sh internally calls audit-firewall.sh --json and caches to /tmp/firewall-audit.json (TTL 5 min). Use --refresh-audit to force refresh.

---

State: CONFIRM (Mandatory — No Exceptions)

This gate prevents the agent from applying firewall rules without explicit human approval.

Why this exists: The agent could (a) invoke firewall-apply.sh without the human seeing the plan, or (b) the human "approves" a stale plan that changed between approval and execution. CONFIRM defeats both — including the agent hallucinating an approval.

Rules (HARD — do not violate):

1. After PLAN, present the full ruleset diff AND the approval token printed by firewall-plan.sh --json (field: approval_token).
2. STOP. Do not proceed in the same response. Do not type the token on the user's behalf.
3. You may ONLY invoke APPLY if the user's most recent message contains that exact approval token. Never infer approval from phrases like "looks good" or "go ahead".
4. APPLY always runs with auto-rollback armed. Committing (disarming rollback) requires a second explicit user instruction after VERIFY passes.
5. A plan token mismatch at APPLY time produces exit code 41 — this is by design. Never bypass it.

Enforcement mechanism: The token is a hash of the plan contents. If the plan changes (e.g., network state drifted between PLAN and APPLY), the token no longer matches → apply is refused. This converts a fuzzy judgment ("did the user approve?") into a cryptographic check the agent cannot rationalize around.

Concrete flow:

Agent: PLAN output → shows rule diff + PLAN-TOKEN: a1b2c3d4e5f6
Agent: STOPS. Waits.
User:  "Approved. a1b2c3d4e5f6"
Agent: Runs APPLY with --approved-plan=a1b2c3d4e5f6
Agent: After APPLY → runs VERIFY
Agent: STOPS. Shows VERIFY results.
User:  "VERIFY passed. Commit."
Agent: Runs COMMIT (disarms rollback timer)

---

State: VALIDATE

1. Create Backup (Mandatory)

BACKUP_DIR="$HOME/firewall-backup-$(date +%Y%m%d-%H%M%S)"
mkdir -p "$BACKUP_DIR"

sudo iptables-save > "$BACKUP_DIR/iptables-v4.rules" 2>/dev/null || true
sudo ip6tables-save > "$BACKUP_DIR/iptables-v6.rules" 2>/dev/null || true
sudo nft list ruleset > "$BACKUP_DIR/nftables.rules" 2>/dev/null || true
sudo ufw status verbose > "$BACKUP_DIR/ufw-status.txt" 2>/dev/null || true
sudo firewall-cmd --list-all --zone=$(sudo firewall-cmd --get-default-zone) > "$BACKUP_DIR/firewalld-default.txt" 2>/dev/null || true

echo "Backup saved to $BACKUP_DIR"

2. Schedule Rollback (Mandatory for Remote)

The rollback restores from backup — not just disables the firewall — so Docker NAT and pre-existing rules are preserved. Dual-backend: at preferred, systemd-run fallback.

# Build rollback script from backup dir
ROLLBACK_SCRIPT=$(cat <<'RB'
#!/bin/bash
BACKUP_DIR="REPLACE_ME"
[ -f "$BACKUP_DIR/iptables-v4.rules" ] && sudo iptables-restore < "$BACKUP_DIR/iptables-v4.rules" || { sudo iptables -P INPUT ACCEPT; sudo iptables -F; }
[ -f "$BACKUP_DIR/iptables-v6.rules" ] && sudo ip6tables-restore < "$BACKUP_DIR/iptables-v6.rules" || { sudo ip6tables -P INPUT ACCEPT; sudo ip6tables -F; }
[ -f "$BACKUP_DIR/nftables.rules" ] && sudo nft -f "$BACKUP_DIR/nftables.rules" || sudo nft flush ruleset
systemctl is-active ufw &>/dev/null && sudo ufw disable
sudo firewall-cmd --panic-off 2>/dev/null
RB
)
ROLLBACK_SCRIPT="${ROLLBACK_SCRIPT/REPLACE_ME/$BACKUP_DIR}"

# Schedule (at preferred, systemd-run fallback)
if command -v at &>/dev/null; then
    ROLLBACK_JOB_ID=$(echo "sudo bash -c '$ROLLBACK_SCRIPT'" | at now + 5 minutes 2>&1 | grep -oP 'job \K\d+')
    echo "Rollback scheduled: at job $ROLLBACK_JOB_ID (cancel with: atrm $ROLLBACK_JOB_ID)"
elif command -v systemd-run &>/dev/null; then
    UNIT_NAME="firewall-rollback-$$"
    echo "$ROLLBACK_SCRIPT" > /tmp/firewall-rollback-$$.sh
    chmod +x /tmp/firewall-rollback-$$.sh
    systemd-run --on-active=5m --unit="$UNIT_NAME" --user /tmp/firewall-rollback-$$.sh
    echo "Rollback scheduled: systemd unit $UNIT_NAME (cancel with: systemctl --user stop $UNIT_NAME)"
fi

See references/recovery.md for advanced recovery scenarios.

2a. Verify Rollback Timer Is Armed (Mandatory, Before Apply)

After scheduling the rollback but BEFORE applying new rules, confirm the timer is actually running:

# For systemd-run
systemctl is-active --quiet firewall-rollback-$$.timer 2>/dev/null \
  && systemctl list-timers firewall-rollback-$$.timer --no-pager 2>/dev/null | grep -q firewall-rollback \
  || { echo "FATAL: rollback timer not scheduled — aborting before applying rules" >&2; exit 1; }

# For at
atq | grep -q "$ROLLBACK_JOB_ID" \
  || { echo "FATAL: rollback job $ROLLBACK_JOB_ID not found in queue — aborting" >&2; exit 1; }

Ordering constraint: This check runs AFTER scheduling but BEFORE apply. If the timer cannot be confirmed as armed, the script exits with code 31 and no rule change ever happens.

3. Pre-Flight Checklist

• Backup created successfully
• Rollback scheduled (verify with atq or systemctl --user list-units)
• Second SSH session open and tested — open a second terminal, SSH in, and confirm you can run sudo whoami. This is your emergency console if the primary session loses connectivity. Keep it open until VERIFY passes. Why: existing ESTABLISHED conntrack entries usually keep your current session alive, but if conntrack is flushed or the policy change drops your session silently, this second session is your only way back in.
• Real SSH port identified (not assumed to be 22)
• Confidence ≥ 70% and risk_tier is auto or confirmed
• Ownership verified — no IaC managing firewall
• Change window appropriate (maintenance window or low traffic)
• PLAN output reviewed and approved

---

State: APPLY

Apply firewall rules using the approved plan from PLAN state.

# Get the approval_token from firewall-plan.sh --json output
bash scripts/firewall-apply.sh --approved-plan=sha256:abc123...
bash scripts/firewall-apply.sh --approved-plan=sha256:abc123... --dry-run

Apply behavior:

• Verifies approval_token matches current plan (exit 41 on mismatch — plan changed since approval)
• Checks for active rollback timer (exit 40 if VALIDATE was skipped)
• Supports all backends: ufw, firewalld, nftables, iptables
• Automatically runs firewall-verify.sh after applying
• Passes --dry-run for preview-only mode

Idempotent inline commands (for manual/scriptless use):

Docker Hosts

Docker bypasses ufw by default. Use DOCKER-USER chain. Full guide: references/docker-hardening.md.

Kubernetes Nodes

Default: AUDIT-ONLY. Never modify host firewall. Full policy: references/k8s-policy.md.

---

State: VERIFY

Phase 1 — Host-local verification (scripts/firewall-verify.sh):

bash scripts/firewall-verify.sh

Phase 2 — External reachability check (scripts/container-port-audit.sh):

Requirement: All connectivity verification steps MUST be executed from a second, external host with an independent network path. Verifying from the target itself cannot detect a lockout.

From a second host on the same network segment, run:

bash scripts/container-port-audit.sh  

This uses nc + /dev/tcp (not nmap) to verify expected-open ports are reachable and expected-closed canary ports are filtered, for both IPv4 and IPv6. UDP probes use nc -uzvw3; note that a "pass" may be a false-positive due to UDP being connectionless (a silently-dropping firewall produces the same result as an open port). Use local ss -ulnp as the deterministic anchor for UDP state.

If a second host is unavailable, the VERIFY stage also runs the container port audit and IPv4/IPv6 consistency checks locally:

bash scripts/container-port-audit.sh   # local-only: container DNAT detection
bash scripts/ip-consistency.sh          # compare iptables vs ip6tables

Phase 3 — Container port DNAT audit:

Detects ports published by container runtimes that bypass the host's INPUT chain. Checks ss listeners + iptables -t nat DNAT rules + FORWARD chain entries. Flags any published port not reflected in explicit INPUT rules.

Success criteria (all must pass):

1. SSH remains reachable from current and second session
2. Only intended ports are externally reachable (verified from second host if available; otherwise local audit)
3. Rules survive reboot (verified via service persistence)
4. IPv6 exposure matches IPv4 policy (verified by ip-consistency.sh)
5. Docker-published ports are intentional (verified by container-port-audit.sh, no accidental 0.0.0.0)
6. fail2ban jails active (if installed) with correct backend
7. Rollback timer cancelled after successful verification

Verify behavior contract:

• Verify MUST complete within the rollback timer window (default 5 min)
• If verify times out before completion → timer auto-fires rollback (system-level protection)
• If verify FAILS but timer was already cancelled → manual rollback from the backup directory. Restore commands (in priority order):
  1. sudo iptables-restore < "$BACKUP_DIR/iptables-v4.rules"
  2. sudo ip6tables-restore < "$BACKUP_DIR/iptables-v6.rules"
  3. sudo nft -f "$BACKUP_DIR/nftables.rules"
  4. sudo ufw reset && sudo ufw disable See references/recovery.md for full recovery procedures including emergency ACCEPT fallback.
• The rollback is triggered by the timer (systemd-run/at), NOT by verify.sh itself — verify.sh exits with code 60 to signal failure, and the calling agent/scheduler handles the rollback decision

Exit Codes (Core Contract)

---

fail2ban Integration

If fail2ban is installed:

After changing backend: sudo fail2ban-client restart && sudo fail2ban-client status sshd.

---

Recovery

If you lose connectivity, priority order:

1. Wait for auto-rollback (scheduled during VALIDATE)
2. Use second SSH session
3. Cloud serial console / hypervisor console
4. Restore from backup
5. Emergency ACCEPT (last resort — exposes host completely)

Full procedures: references/recovery.md.

State Persistence & Interrupt-Resume

For agent interrupt-resume scenarios (e.g., Apply failed mid-run, agent restarted), the state machine writes a lightweight state file to enable recovery without starting from Detect:

STATE_DIR="$HOME/.firewall-hardening"
STATE_FILE="$STATE_DIR/state.json"

State file structure:

{
  "state": "validate",
  "started_at": "2026-05-11T16:00:00Z",
  "backend": "ufw",
  "risk_tier": "auto",
  "backup_dir": "/home/user/firewall-backup-20260511-160000",
  "rollback_timer_id": "firewall-rollback-12345",
  "plan_hash": "sha256:abc123..."
}

Resume logic:

• If state.json exists and started_at is within 1 hour → resume from that state
• If state.json is stale (>1 hour) → delete it and start fresh from Detect
• The file is advisory-only; agent can always restart from Detect

State persistence is optional. The skill defaults to restarting from Detect each run. Enable by creating $STATE_DIR before starting.

---

Cloud Security Group Reminder

The host firewall is your second layer. Verify cloud SGs are aligned:

Compatibility Matrix

Container environments: Docker host is fully supported via DOCKER-USER chain. LXC/LXD/systemd-nspawn have limited support (kernel shares netfilter with host). WSL2 is explicitly unsupported. See references/special-environments.md.

---

Observability

Establish baselines after hardening: conntrack usage, dropped packet rates, fail2ban ban rate. Monitor for anomalies. Full guide: references/observability.md.

Compliance

Practices map to CIS, PCI-DSS, and SOC2 controls. Full mapping: references/compliance.md.

Quick Reference

See Also

• references/backend-ufw.md — Full UFW phase
• references/backend-firewalld.md — Full firewalld phase
• references/backend-nftables.md — Full nftables phase
• references/backend-iptables.md — Full iptables phase
• references/docker-hardening.md — Docker firewall hardening
• references/k8s-policy.md — Kubernetes node policy
• references/security-profiles.md — Pre-built configurations
• references/declarative-policy.md — YAML policy schema
• references/observability.md — Monitoring and baselines
• references/compliance.md — CIS/PCI-DSS/SOC2 mapping
• references/recovery.md — Recovery procedures
• references/special-environments.md — WSL2, containers, exit codes
• scripts/audit-firewall.sh — Environment detection
• scripts/firewall-plan.sh — Dry-run diff
• scripts/firewall-verify.sh — Post-apply verification