Huawei Cloud Cce Node Failure Diagnoser

Huawei Cloud CCE Node Failure Diagnoser

⚠️ Execution Method (Must Read): This skill executes queries via the local Python dispatcher script. Using hcloud, openstack, or other CLI tools or direct API calls is prohibited.

• The dispatcher script is located at scripts/huawei-cloud.py within the skill directory
• All scripts and environment check scripts are inside the skill package. You must use skill action=exec to execute them. Do not run them directly in a shell.
• Do not attempt hcloud, openstack, curl IAM, or any other CLI/API methods. This skill does not depend on those tools.
• All paths are relative to the skill directory, which is the directory where this SKILL.md is located.

Overview

This skill diagnoses CCE/Kubernetes node failures and produces a complete Markdown diagnosis report. It uses the local Python dispatcher (scripts/huawei-cloud.py) to call the Huawei Cloud Python SDK and Kubernetes client APIs, collecting node status, kube-node-lease evidence, events, pod symptoms, and AOM metrics.

The skill covers: node NotReady, disk/memory/CPU pressure, network abnormalities, kubelet/CRI failures, NPD events, and workload impact on the affected node.

Related Skills

Capabilities

1. One-shot node failure diagnosis with structured evidence and Markdown report (huawei_node_failure_diagnose)
2. Kubernetes node status and conditions collection (huawei_get_kubernetes_nodes)
3. Node/Pod event timeline retrieval (huawei_get_cce_events)
4. Pod phase/reason/state aggregation per node (huawei_get_cce_pods)
5. Node and Pod AOM metric queries (huawei_get_cce_node_metrics, huawei_get_cce_pod_metrics_topN)
6. Node inspection items (status, resource, vulnerability) (huawei_node_status_inspection, huawei_node_resource_inspection, huawei_node_vul_inspection)
7. Security group and HSS vulnerability correlation (huawei_list_security_groups, huawei_hss_list_hosts, huawei_hss_list_host_vuls_all)

Typical Use Cases

• Diagnose a CCE node that transitioned to NotReady state
• Investigate node disk or memory pressure conditions
• Analyze kubelet or container runtime failures on a node
• Check node network connectivity issues (CNI sandbox failures)
• Assess pod impact when a node becomes unhealthy
• Review NPD events and node-level security findings

---

Prerequisites

Python Dependencies

The dispatcher script requires Python >= 3.6 and the following packages:

• huaweicloudsdkcore
• huaweicloudsdkcce
• huaweicloudsdkaom
• huaweicloudsdkhss
• huaweicloudsdkvpc
• huaweicloudsdkecs
• huaweicloudsdkces
• huaweicloudsdkevs
• huaweicloudsdkeip
• huaweicloudsdkelb
• huaweicloudsdkiam
• kubernetes

Credential Configuration

🚫 Never expose or log AK/SK values. Credentials exist only in the current request call stack and are released after each invocation. Do not write credentials to files, logs, or responses.

✅ Use environment variables HUAWEI_AK / HUAWEI_SK for authentication. The dispatcher reads them automatically.

IAM Permissions

---

Core Tools

All actions are invoked via the dispatcher script:

python3 scripts/huawei-cloud.py  region= cluster_id= [key=value ...]

Primary Diagnosis Action

python3 scripts/huawei-cloud.py huawei_node_failure_diagnose \
  region=cn-north-4 cluster_id= \
  node_name= lease_timeout_seconds=40 \
  event_limit=500 hours=1 include_metrics=true

Returns structured evidence + report_markdown (complete Markdown diagnosis report).

Evidence Collection Actions

Inspection Actions

Security Correlation Actions

---

Parameter Reference

huawei_node_failure_diagnose

*At least one of node_name or node_ip must be provided. If both are omitted, the action returns an error.

Common Parameters

---

Output Format

The primary action huawei_node_failure_diagnose returns structured evidence and a Markdown report. See references/output-schema.md for the full JSON response schema.

Key output fields:

When report_markdown is present, use it as the final report body. You may add clarifications the user requests, but do not discard evidence tables.

---

Verification

1. Run the dispatcher with a known cluster and node to confirm connectivity:

   python3 scripts/huawei-cloud.py huawei_get_kubernetes_nodes region=cn-north-4 cluster_id=
   

2. Execute huawei_node_failure_diagnose on a healthy node; expect root_category=Healthy and confidence=High
3. Verify report_markdown contains all required sections (see references/output-schema.md)
4. Compare node conditions in the output with the CCE console

---

Best Practices

1. Always call huawei_node_failure_diagnose first; use manual fallback actions only if the primary action fails
2. When Ready=Unknown and lease is stale, conclude "control plane disconnected from node" rather than prematurely attributing to kubelet or network alone
3. When pressure conditions are Unknown with NodeStatusUnknown reason, label them "indeterminate" — do not mark as "normal"
4. Correlate Event signals with Pod symptoms before forming conclusions; evidence strength determines confidence level
5. Include security group and HSS checks only when network or vulnerability hypotheses are strong
6. Do not single-point metric peaks as root cause; validate with trend data

---

Reference Documents

---

Notes

1. This skill is read-only diagnosis only — it does not cordon, uncordon, drain, reboot, or modify vulnerability status
2. When remediation actions are needed, hand off to huawei-cloud-cce-auto-remediation-runner and require user confirmation
3. Never expose or log AK/SK or environment variable values
4. All actions are executed via python3 scripts/huawei-cloud.py ; do not use hcloud CLI or direct API calls
5. If the primary action huawei_node_failure_diagnose fails, follow the manual fallback workflow in references/workflow.md

---

Common Pitfalls