Huawei Cloud Cce Autoscaling Diagnoser

Huawei Cloud CCE Autoscaling Diagnoser

⚠️ Execution Method (Must Read): This skill executes queries via the local Python dispatcher script. Using hcloud, openstack, or other CLI tools or direct API calls is prohibited.

• The dispatcher script is located at scripts/huawei-cloud.py within the skill directory
• All scripts and environment check scripts are inside the skill package. You must use skill action=exec to execute them. Do not run them directly in a shell.
• Do not attempt hcloud, openstack, curl IAM, or any other CLI/API methods. This skill does not depend on those tools.
• All paths are relative to the skill directory, which is the directory where this SKILL.md is located.

Overview

This skill diagnoses CCE autoscaling link failures across two closed-loop layers: (1) whether HPA increases workload replica count from N to N+1, and (2) whether CCE elastic engine / Cluster Autoscaler increases node count from M to M+1 after resource-insufficient Pending Pods appear. It outputs a complete Markdown diagnosis report with process, evidence, root cause conclusion, confidence, data gaps, and recommendations.

Architecture: Python dispatcher (scripts/huawei-cloud.py) → Huawei Cloud Python SDK + Kubernetes client → HPA/CA/Addon/NodePool/Pod/Events/Metrics → Gateway intent routing → Path A/B/C diagnosis → Structured evidence + Markdown report

Related Skills:

Capabilities:

1. One-shot autoscaling diagnosis with Gateway intent routing, capability discovery, and Path A/B/C evidence collection (huawei_autoscaling_diagnose)
2. HPA object inspection: spec, currentReplicas, desiredReplicas, minReplicas, maxReplicas, conditions, metrics (huawei_list_cce_hpas)
3. CCE addon and nodepool autoscaling discovery (huawei_list_cce_addons, huawei_list_cce_nodepools)
4. CA Pod log analysis: automatic discovery of kube-system autoscaler Pods, log retrieval, and 16 diagnostic signal pattern matching
5. Pending Pod and scheduling constraint analysis (huawei_get_cce_pods, huawei_get_cce_events)
6. AOM/Prometheus custom metric evidence (huawei_get_aom_metrics)
7. Complete Markdown report generation with evidence, conclusion, confidence, and recommendations

Typical Use Cases:

• "HPA is not scaling my Deployment, what's wrong?"
• "Why isn't the Cluster Autoscaler adding nodes when Pods are Pending?"
• "My workload replicas aren't increasing despite high CPU usage"
• "Diagnose why autoscaling is not working in my CCE cluster"
• "HPA shows desiredReplicas equals currentReplicas, why no scaling?"
• "Pods are Pending with Insufficient cpu/memory but no new nodes appear"
• "Check if autoscaling is properly configured for my workload"
• "Analyze CA logs for node scaling failure signals"

Prerequisites

1. Python Requirements (MANDATORY)

• Python >= 3.6 installed
• Required packages: huaweicloudsdkcore, huaweicloudsdkcce, huaweicloudsdkaom, kubernetes
• Verify: python3 --version
• Install packages: pip3 install huaweicloudsdkcore huaweicloudsdkcce huaweicloudsdkaom kubernetes

2. Credential Configuration

• Valid Huawei Cloud credentials (AK/SK mode)
• Security Rules:
  • 🚫 Never expose AK/SK values in code, conversation, or commands
  • 🚫 Never use echo $HUAWEI_AK or echo $HUAWEI_SK to check credentials
  • 🚫 Never write credentials to files, logs, or responses
  • ✅ Use environment variables: HUAWEI_AK, HUAWEI_SK, HUAWEI_REGION
  • ✅ Credentials exist only in the current request call stack and are released after each invocation
  • ✅ Prefer IAM users over root account for cloud operations

Configuration Method (Environment Variables Only):

export HUAWEI_AK=
export HUAWEI_SK=
export HUAWEI_REGION=cn-north-4

Additional Variables:

3. IAM Permission Requirements

Permission Failure Handling:

1. When any action fails due to IAM permission errors, display the required permission list
2. Guide the user to create custom policies in the IAM console for Huawei Cloud permissions
3. Pause execution and wait for user confirmation that permissions have been granted
4. Retry the failed action

Core Commands

All actions are invoked via the dispatcher script:

python3 scripts/huawei-cloud.py  region= cluster_id= [key=value ...]

1. Primary Diagnosis Action

python3 scripts/huawei-cloud.py huawei_autoscaling_diagnose \
  region=cn-north-4 cluster_id= \
  namespace=default workload_name=my-app workload_type=Deployment \
  question="Why isn't HPA scaling my workload?"

Returns structured evidence + report_markdown (complete Markdown diagnosis report). When report_markdown is present, use it as the final report body. You may add clarifications the user requests, but do not discard evidence tables.

2. Evidence Collection Actions (Read-Only)

3. CA Pod Log Analysis (Manual Fallback)

# Step 1: Locate CA component Pods in kube-system
python3 scripts/huawei-cloud.py huawei_get_cce_pods \
  region=cn-north-4 cluster_id= namespace=kube-system

# Step 2: Retrieve CA Pod logs (find pods with names containing autoscaler/cce-elastic/elastic-engine)
python3 scripts/huawei-cloud.py huawei_get_pod_logs \
  region=cn-north-4 cluster_id= namespace=kube-system \
  pod_name=cce-cluster-autoscaler-abc123 container=autoscaler tail_lines=200

CA Log Signal Quick Reference:

Parameter Reference

huawei_autoscaling_diagnose (Primary Action)

Common Parameters

Common Region IDs

Output Format

The primary action huawei_autoscaling_diagnose returns structured evidence and a Markdown report. See Output Schema for the full JSON response schema.

Key output fields:

Required Markdown report sections:

1. # CCE Autoscaling Automated Diagnosis Report
2. ## 1. Diagnosis Overview: region, cluster, intent, scale direction, route, conclusion, confidence
3. ## 2. Capability Discovery & Routing: Has_HPA, Has_CA, metric link, routing basis
4. ## 3. Investigation Process: Gateway, Path A/B/C actual execution steps
5. ## 4. Key Evidence: HPA status, nodepool/addon, Pending Pod, FailedScheduling evidence
6. ## 5. Issues & Root Cause Convergence: issues ranked by severity with evidence and recommendations
7. ## 6. Next-Step Recommendations: read-only verification and remediation suggestions only
8. ## 7. Data Gaps: collection failures and items that could not be confirmed

Verification

See Verification Method for step-by-step verification.

Best Practices

1. Primary action first: Always call huawei_autoscaling_diagnose first; use manual fallback only if the primary action fails
2. Gateway routing: Do not skip the Gateway phase — intent routing and capability discovery determine the correct Path A/B/C
3. CA logs are critical: CA Pod logs are the highest-confidence evidence source for node scaling failures; the primary tool automatically collects them, but manual fallback must prioritize this step
4. Cascade diagnosis: When HPA has scaled but new Pods are Pending, trace from HPA → CA as a cascade (Path C), not as separate isolated issues
5. Metric prerequisites: CPU/memory utilization-based HPA requires corresponding resources.requests on Pod containers; missing requests are a common critical root cause
6. Read-only boundary: This skill is read-only diagnosis; never create/modify HPA, scale workloads, modify nodepool min/max, install/upgrade addons, expand subnets, or apply for quota
7. Hand off remediation: When remediation is needed, hand off to huawei-cloud-cce-auto-remediation-runner and require user confirmation
8. Log sanitization: Never copy raw passwords, tokens, AK/SK, or Authorization headers from CA logs into output

Reference Documents

Notes

• Read-only by design — this skill does NOT create/modify HPA, scale workloads, modify nodepool min/max, install/upgrade addons, expand subnets, or apply for quota
• One-call preferred — huawei_autoscaling_diagnose is the primary tool; raw queries are for targeted evidence when the user requests specific information or when the primary tool fails
• Log sanitization — only sanitized tail excerpts are included; raw secrets, tokens, and credentials must never appear in output
• Gateway routing mandatory — do not skip intent identification and capability discovery before entering Path A/B/C
• CA Pod log analysis — the primary tool automatically discovers and analyzes CA component Pod logs; manual fallback must prioritize this step
• Cross-skill handoff — when diagnosis reveals issues beyond autoscaling scope (Pod runtime failure, workload rollout failure, node NotReady), escalate to the appropriate skill

Common Pitfalls