Alibabacloud Security Health Check

alibabacloud-security-health-check

Purpose

Perform configuration baseline health checks for contracted Alibaba Cloud security product customers. The key difference from sas-security-report (SAS single-product retrospective reporting): Health check focuses on "are configuration items fully enabled", while reporting focuses on "what happened in the past 30 days". The former outputs a remediation checklist; the latter outputs a narrative.

When to Use

• Monthly patrols by Customer Success / Delivery teams
• Value demonstration + remediation recommendations 2-3 months before renewal
• Customer asks "what's my configuration score" / "which items are not enabled" / "how does my setup align with best practices"
• Pre-project assessment for large customer security governance initiatives

Workflow (5 Steps)

1. Confirm product scope: Before invoking the scorer, the Agent should identify three signals from the user's prompt:

   • Customer name (e.g. "Starchen Tech", "Qingshan Finance", "蓝海集团")
   • Product scope — which of WAF / SAS / CFW / DDoS the customer has purchased (or "全部四款" / "all four")
   • Data location — explicit path, or a customer subdirectory under evals/scenarios/sample-data//

   Default: proceed directly, never ask. As long as the prompt contains a customer name token OR any path/directory hint, the Agent MUST proceed silently to scoring without invoking ask_user_question. Resolution rules:

   • If customer name is present, infer the data location by scanning evals/scenarios/sample-data// for any *-collected.json files; missing product JSONs are auto-skipped by the scorer's native [skip] logic — never fabricate empty stubs.
   • If output directory is not specified, write to the path indicated by the task runtime (env var or task prompt's "Save outputs to:" line); fall back to ./output only as a last resort.
   • Proceed directly to scoring once at least one product JSON is locatable.
   • The Agent MUST NOT ask the user to confirm product scope, data location, or file paths when the prompt already names a customer and mentions JSON / product / directory context. Any hesitation defaults to "run now".

   Dynamic range rule: All generated reports MUST reflect the actual number of products assessed — never use fixed phrasing like "four core products" if fewer were provided. The exec-summary.md opening line MUST state "This assessment covers N product(s): [list]" and note any skipped products explicitly.

2. Distribute collection scripts: Send the corresponding .sh scripts from scripts/collectors/ to the customer's ops team

3. Receive JSON: Customer returns 4 JSON files (waf-collected.json / sas-collected.json / etc.)

4. Run scoring: python scripts/engine/scorer.py --input --output

5. Deliver three-piece set: HTML report (for customer) + Excel remediation checklist (for ops) + Markdown summary (for internal CSM reporting)

Extending Check Items

All check items are defined in references/checks/*.yaml. Adding new items requires no code changes—just follow the existing schema:

- id: -NNN
  name: 
  category: coverage | strength | responsiveness | compliance | best-practice
  weight: <1-15>
  severity: high | medium | low
  rule:
    type: ratio | exists | enum | range
    path: 
    thresholds: ...
  remediation:
    effort: low | medium | high
    steps: |
      Specific remediation steps
    doc: 

Out of Scope

• No retrospective reporting — that is the responsibility of sas-security-report / waf3-monthly-report
• No pricing — use alicloud-security-pricing instead
• No industry benchmarking (MVP phase) — planned for v2
• No credential handling — refuses any customer credentials; only accepts JSON files

Output Artifacts

/
├── health-report.html      # Customer-facing visual report
├── remediation.xlsx        # Ops remediation checklist
└── exec-summary.md         # Internal CSM summary

Input/Output Examples

Input: Customer JSON (waf-collected.json excerpt)

{
  "metadata": { "collected_at": "2026-06-01T08:00:00Z", "region": "cn-hangzhou" },
  "client_assets": { "TotalDomains": 35, "TotalPublicIPs": 12 },
  "internet_firewall": { "TotalPublicIPs": 12, "ProtectedPublicIPs": 10 },
  "defense_resources": { "TotalCount": 28 },
  "defense_rules": { "TotalCount": 15 },
  "bot_config": { "TotalCount": 3 }
}

Output: scores.json (excerpt)

{
  "customer": "AcmeCorp",
  "overall_score": 72,
  "dimensions": {
    "coverage": 85,
    "strength": 60,
    "responsiveness": 70,
    "compliance": 80,
    "best_practice": 65
  },
  "failed_checks": [
    { "id": "waf-003", "name": "Bot protection enabled", "severity": "high", "score": 0 },
    { "id": "waf-007", "name": "Custom rule coverage", "severity": "medium", "score": 40 }
  ]
}

Output: exec-summary.md (excerpt)

## AcmeCorp Security Health Check Summary
- Overall Score: 72/100
- Critical gaps: Bot protection not enabled, custom rule coverage below 50%
- Recommended actions: Enable anti-scan defense scene, add rate-limiting rules

Agent Execution Guide

After receiving the 4 JSON files from the customer, the agent should invoke the toolchain in the following order:

# 0. Generate session-id for this invocation
SESSION_ID="$(uuidgen 2>/dev/null || python3 -c 'import uuid; print(uuid.uuid4())')"
export SESSION_ID

# 1. Run the scorer first (do NOT pre-install anything)
#    NOTE: SESSION_ID is propagated through the SESSION_ID environment variable
#    and is auto-injected into collector --user-agent headers. The scoring
#    engine itself does NOT accept --session-id as a CLI argument.
python3 scripts/engine/scorer.py \
  --input  \
  --checks references/checks \
  --output output

# 2. Render three-piece deliverable (can run in parallel)
python3 scripts/engine/report_html.py     --customer ""
python3 scripts/engine/report_excel.py    --customer ""
python3 scripts/engine/report_markdown.py --customer ""

Dependency handling

scorer.py declares its Python dependencies in the file header and exits with a clear Missing dependency: . Please run: pip install pyyaml jsonpath-ng jinja2 openpyxl message when imports fail.

Auto-install policy — when the Agent observes this exact error message, it SHOULD immediately run:

pip install --quiet pyyaml jsonpath-ng jinja2 openpyxl

and retry scorer.py with the same arguments. No user consent prompt is required for these four pinned dependencies because they are part of the documented skill contract and the script's own error message names them exactly. The Agent SHOULD treat this as a routine recovery step, not as an environment mutation requiring human approval.

If the auto-install itself fails (e.g. offline sandbox, no network, pip permission error), the Agent SHOULD surface the underlying failure and stop — never fabricate scores, never invent product data.

This relaxed policy applies only to the four pinned Python deps named above. Any other destructive or environment-altering action (file deletions, credential rotation, large data fetches, system package installs via apt / brew / npm) still requires explicit user consent before proceeding.

All collector scripts automatically inject the --user-agent header with the session-id when SESSION_ID is exported.

Output artifacts: output/{scores.json, health-report.html, remediation.xlsx, exec-summary.md}.

Final Response Verification

Before composing the final summary message to the user, the Agent MUST perform this mandatory numeric verification checklist — no exceptions:

1. Open output/scores.json with a fresh read (do NOT rely on memory or earlier tool output). Extract overall_score and every product's dimension scores as raw strings.
2. Build a draft reply text. For every number the draft cites, locate the corresponding key in scores.json and perform character-level string comparison. Any mismatch — including magnitude errors (77.0 → 7.0), decimal shifts (88.5 → 8.5), leading-digit drops, or rounding — MUST be corrected before sending.
3. Explicitly enumerate the checked pairs in scratch reasoning: overall_score: json= draft= ok/mismatch, waf.coverage: json= draft= ok/mismatch, etc. Only send the reply after every pair reads ok.
4. scores.json is the single source of truth. Never fabricate, round, or paraphrase numbers from memory, agent-log, or intermediate stdout.

Key Design Principles

• Transparent weights: Each check item's weight/severity is defined in YAML, auditable by customers
• Zero credential transfer: Customer ops run read-only RAM sub-account scripts, return JSON only
• Extensible rules: Adding checks only requires YAML changes; rule.type limited to 4 kinds (ratio/exists/enum/range)
• Comparability: Single-product scores normalized to 0-100, comparable across 3-item and 30-item assessments
• CSM-friendly: Three deliverables target different audiences—customer leadership (HTML) / customer ops (Excel) / internal reporting (Markdown)

Compatibility

• Python >= 3.9
• Alibaba Cloud CLI >= 3.3.3 (collection scripts)
• Verified: 60-item mock data end-to-end pass, 0 errors
• Known limitation: DDoS Pro currently only supports DDoSCoo; native protection requires manual field input

Observability

Each skill invocation MUST generate a unique session_id (UUID v4) at the start.

Session-ID Generation Rule

The session-id is a UUID v4 generated once per invocation and reused across all CLI calls within that session:

SESSION_ID="$(uuidgen 2>/dev/null || python3 -c 'import uuid; print(uuid.uuid4())')"

User-Agent Template

All aliyun CLI calls MUST include the --user-agent flag with the following template:

--user-agent "AlibabaCloud-Agent-Skills/alibabacloud-security-health-check/${SESSION_ID}"

The ${SESSION_ID} placeholder is replaced at runtime with the generated UUID. This propagates through the OpenAPI call chain for end-to-end tracing on the Alibaba Cloud side.

OpenTelemetry

When available, the caller injects traceparent and baggage headers via OpenTelemetry. The session-id is also propagated in the baggage header as skill-session-id={SESSION_ID}.