huawei-cloud-mrs-host-fault-diagnose

Huawei Cloud MRS Host Fault Diagnosis Skill

Overview

This skill diagnoses Huawei Cloud MRS (MapReduce Service) cluster faults. Given a service name and/or node name, it progressively localizes the root cause: quick log scan first, host troubleshooting when host issues are found, detailed investigation when no conclusion is reached.

Architecture: Caller (Agent) -> lakewatch_api_client.py (Python, scripts/) -> LakeWatch API -> MRS cluster (node resource data, logs, MRS Manager proxy); per-component knowledge base (components/.md) drives the diagnosis flow; three fault layers (host -> instance -> service) with propagation chain tracing.

Note on language: This SKILL.md and the documents under references/ are written in English per the repository spec. The knowledge base documents under fault_layer/, scenarios/, components/, and propagation.md are also in English. Commands and code blocks are English throughout.

Applicable Scenarios:

• A service is reported unhealthy and the root cause must be localized
• An instance is reported faulty on a specific node
• A host is reported unreachable or abnormal
• Progressive fault triage from quick scan to deep investigation

Typical Use Cases:

• "KrbServer出问题了，帮忙诊断一下" (service fault, no node specified)
• "8-5-225-6上的KrbServer挂了" (instance fault, service + node specified)
• "8-5-225-6出问题了" (host fault, node only)
• "MRS集群KrbServer启动失败，集群ID xxx"
• "DBService停止异常，节点8-5-225-6"

Critical Constraints

Important constraints:

1. Read-only: This skill only runs information-gathering commands (view logs, query status, collect resource data). It MUST NOT run any start/stop, modify, or delete operations.
2. User confirmation for repair: The skill only provides executable repair suggestions; it MUST NOT directly execute any repair operation. All repair actions require user confirmation.
3. Strict execution: Diagnose strictly according to the knowledge base content under this skill directory. Fabricating diagnostic commands outside the knowledge base is prohibited.

Prerequisites

1. Python Requirements

• Python >= 3.7
• Dependencies: pyyaml (YAML parsing), cryptography (Windows AES password encryption only)
• Linux uses CryptoAPI for password encryption (no cryptography dependency)
• Verify installation: python3 --version (Linux) / python --version (Windows)

This skill does NOT require KooCLI (hcloud). It calls the LakeWatch API through scripts/lakewatch_api_client.py. For the LakeWatch client setup, see CLI Installation Guide.

2. LakeWatch Credential Configuration

• A valid LakeWatch service account (username + password)
• The password MUST be encrypted with --encrypt-password and stored in scripts/lakewatch_api_config.yaml (auth.encrypted_password). Never store the plaintext password.
• Security Rules:
  • Never expose the LakeWatch password in conversation or command output
  • Never ask the user to input the plaintext password in conversation; use the interactive --encrypt-password flow
  • The token is cached locally with owner-only file permissions (Win: %TEMP%\lakewatch_token\, Linux: /tmp/lakewatch_token/)

3. Access Permissions

• Reachability to the LakeWatch service endpoint (configured in scripts/lakewatch_api_config.yaml server.host/port)
• The LakeWatch account must have permission to call the MRS Manager proxy and collect node resource/log data on the target cluster
• See IAM Policies for the access model and required roles

4. Dependent Skill: huawei-cloud-mrs-host-alarm-diagnose

This skill references the per-alarm diagnosis knowledge base from the huawei-cloud-mrs-host-alarm-diagnose skill (sibling directory under skills/bigdata/mrs/). When the fault diagnosis flow encounters a known alarm (12006/12007/25000/25500/27001), it loads the corresponding document from ../huawei-cloud-mrs-host-alarm-diagnose/alarms/.md.

• If the alarm skill exists, load the referenced document and follow its diagnosis flow
• If NOT exist, inform the user and proceed with the generic fault diagnosis flow
• The dependency is document-level reference only (loading markdown by relative path), NOT a direct skill call. Both skills share the same LakeWatch API client and config format.

Command Format Standard

This skill uses the LakeWatch API client instead of KooCLI. The unified command format is:

# Linux
python3 /scripts/lakewatch_api_client.py -a  -p 'key1=value1' -p 'key2=value2'

# Windows
python /scripts/lakewatch_api_client.py -a  -p 'key1=value1' -p 'key2=value2'

Windows (PowerShell) quote rule: every " inside a value must be replaced with """ (including " inside [] and {}), otherwise the server returns {"message":"Unknown exception","success":false,"code":"500"}:

# Correct on Windows
-p 'keywords=["""ERROR"""]'
-p 'env={"""PID""":"""123"""}'

# Wrong on Windows (will fail)
-p 'keywords=["ERROR"]'

Linux (bash) quote rule: keep " as-is inside the value, wrap the whole value in single quotes:

# Correct on Linux
-p 'keywords=["ERROR","Exception"]'
-p 'env={"PID":"123"}'

For the full API catalog, parameters, and the token/encryption mechanism, see LakeWatch API Client.

Workflow

Step 1: Determine Fault Entry

Extract fault information from the user input and determine the diagnosis entry:

Step 2: Locate the Fault Object

Entry A: Service Fault (has service_name, no node_name)

Load components/.md for component config. Query OMS primary/standby nodes, check process on each node:

python3 lakewatch_api_client.py -a query-management-node-info \
  -p 'cluster_id='

python3 lakewatch_api_client.py -a collect_alarm_node_res_data \
  -p 'cluster_id=' \
  -p 'strategy_name=process-basic-info' \
  -p 'env={"process_name":""}' \
  -p 'node_name='

Decision:

Entry B: Instance Fault (has service_name + node_name)

Load components/.md. Directly check process on that node:

python3 lakewatch_api_client.py -a collect_alarm_node_res_data \
  -p 'cluster_id=' \
  -p 'strategy_name=process-basic-info' \
  -p 'env={"process_name":""}' \
  -p 'node_name='

Decision:

Entry C: Host Fault (has node_name, no service_name)

Query OMS primary/standby nodes, query node IP, ping the faulty node from OMS active node:

python3 lakewatch_api_client.py -a query-management-node-info \
  -p 'cluster_id='

python3 lakewatch_api_client.py -a query-node-ip \
  -p 'cluster_id=' \
  -p 'node_name='

python3 lakewatch_api_client.py -a collect_alarm_node_res_data \
  -p 'cluster_id=' \
  -p 'strategy_name=ping-check' \
  -p 'env={"TARGET_IP":""}' \
  -p 'node_name='

Decision:

Step 3: Quick Log Scan

For the faulty node, quickly scan three layers of logs (Controller -> NodeAgent -> component), looking for clear ERROR:

# Controller log
python3 lakewatch_api_client.py -a collect_alarm_log_data \
  -p 'cluster_id=' \
  -p 'alarm_time=' \
  -p 'log_directory=/var/log/Bigdata/controller' \
  -p 'log_file_name=exe.log*' \
  -p 'keywords=["","ERROR","fail","timeout","Exception"]' \
  -p 'log_type=local' \
  -p 'node_name='

# NodeAgent script log
python3 lakewatch_api_client.py -a collect_alarm_log_data \
  -p 'cluster_id=' \
  -p 'alarm_time=' \
  -p 'log_directory=/var/log/Bigdata/nodeagent/scriptlog' \
  -p 'log_file_name=*.log*' \
  -p 'keywords=["","ERROR","fail","exit"]' \
  -p 'log_type=local' \
  -p 'node_name='

If service_name is known, also check the component's own log (path from components/.md):

python3 lakewatch_api_client.py -a collect_alarm_log_data \
  -p 'cluster_id=' \
  -p 'alarm_time=' \
  -p 'log_directory=' \
  -p 'log_file_name=' \
  -p 'keywords=["ERROR","Exception","FATAL","fail","OOM"]' \
  -p 'log_type=local' \
  -p 'node_name='

Decision:

Step 4: Detailed Investigation

When the quick log scan yields no conclusion, collect complete data:

1. Load Data Collection to collect process/port/HA/resource/alarm/framework logs
2. Load Instance Fault Diagnosis for instance-level diagnosis (includes scenario identification)
3. If needed, load Service Fault Diagnosis for service-level diagnosis
4. If host issue is found, load Host Fault Diagnosis for host-level diagnosis

Step 5: Propagation Chain Tracing

Load Propagation Chain to trace the root cause propagation path and impact scope.

Step 6: Output Diagnosis Conclusion

## Diagnosis Result

| Item | Content |
|------|---------|
| Diagnosis time | [time] |
| Cluster ID | [cluster_id] |
| Faulty component | [service_name] |
| Faulty node | [node_name] |

### Diagnosis Process

| Step | Result |
|------|--------|
| Instance status | [which nodes normal/abnormal] |
| Quick log scan | [found/not found clear ERROR] |
| Host troubleshooting | [normal/abnormal: ...] |
| Detailed investigation | [process/port/HA/resource results] |

### Propagation Path

[root cause] -> [propagation] -> [symptom] (single-layer root cause if no propagation)

### Root Cause Analysis

**Root cause layer**: [host/instance/service]
**Root cause type**: [specific reason]

### Repair Suggestion

| Priority | Operation | Description | Needs user confirmation |
|----------|-----------|-------------|-------------------------|
| 1 | [operation] | [description] | Yes |

Core Commands

Query OMS Primary/Standby Nodes

python3 lakewatch_api_client.py -a query-management-node-info \
  -p 'cluster_id='

Query Node IP

python3 lakewatch_api_client.py -a query-node-ip \
  -p 'cluster_id=' \
  -p 'node_name='

Collect Node Resource Data

# Process basic info
python3 lakewatch_api_client.py -a collect_alarm_node_res_data \
  -p 'cluster_id=' \
  -p 'strategy_name=process-basic-info' \
  -p 'env={"process_name":""}' \
  -p 'node_name='

# Port check
python3 lakewatch_api_client.py -a collect_alarm_node_res_data \
  -p 'cluster_id=' \
  -p 'strategy_name=port-check' \
  -p 'env={"PORT":""}' \
  -p 'node_name='

# HA resource status
python3 lakewatch_api_client.py -a collect_alarm_node_res_data \
  -p 'cluster_id=' \
  -p 'strategy_name=ha-resource-status' \
  -p 'node_name='

# Disk space / Memory / CPU load
python3 lakewatch_api_client.py -a collect_alarm_node_res_data \
  -p 'cluster_id=' \
  -p 'strategy_name=disk-space' \
  -p 'node_name='

Supported strategy_name values include: system-load, memory-usage, disk-space, disk-io, network-io, file-handle, port-check, high-cpu-processes, high-memory-process, zombie-process, dns-check, network-connectivity-test, process-basic-info, process-file-descriptor, jstack-thread-dump, disk-health-check, disk-smart-info, ha-resource-status, omm-process-tree, and more. See LakeWatch API Client for the full list.

Collect Alarm Log Data

python3 lakewatch_api_client.py -a collect_alarm_log_data \
  -p 'cluster_id=' \
  -p 'alarm_time=' \
  -p 'log_directory=' \
  -p 'log_file_name=' \
  -p 'keywords=["ERROR","Exception"]' \
  -p 'log_type=local'

When the log time format is non-standard ISO (e.g. [2026-07-07 20:54:25,171]), pass time_pattern:

python3 lakewatch_api_client.py -a collect_alarm_log_data \
  -p 'cluster_id=' \
  -p 'alarm_time=2026/07/07 20:54:00 GMT+08:00' \
  -p 'log_directory=/var/log/Bigdata/omm/oms/pms' \
  -p 'log_file_name=pms*.log' \
  -p 'keywords=["ERROR","Exception"]' \
  -p 'log_type=local' \
  -p 'time_pattern=^\[([0-9]{4})-([0-9]{2})-([0-9]{2}) ([0-9]{2}):([0-9]{2}):([0-9]{2})||ymdHMS'

Proxy MRS Manager GET API

# Query cluster services
python3 lakewatch_api_client.py -a access_manager_get \
  -p 'cluster_id=' \
  -p 'target_url=api/v2/clusters//services'

# Query host processes
python3 lakewatch_api_client.py -a access_manager_get \
  -p 'cluster_id=' \
  -p 'target_url=api/v2/clusters//hosts//processes'

# Query active alarms
python3 lakewatch_api_client.py -a access_manager_get \
  -p 'cluster_id=' \
  -p 'target_url=api/v2/clusters//alarms'

target_url MUST NOT start with /. The proxy requires Agent >= 1.0.5 and reported OMS node info. Only GET is supported currently.

Parameter Confirmation

Output Format

The diagnosis report is output in Markdown, containing:

• Diagnosis result table: diagnosis time, cluster ID, faulty component, faulty node
• Diagnosis process: step-by-step results (instance status, quick log scan, host troubleshooting, detailed investigation)
• Propagation path: root cause -> propagation -> symptom (single-layer if no propagation)
• Root cause analysis: root cause layer (host/instance/service) + root cause type
• Repair suggestion table: priority, operation, description, needs-user-confirmation (all repair actions require user confirmation)

See the template in the Workflow -> Step 6 section.

Verification Method

See Verification Method for the installation, configuration, and function verification steps.

Best Practices

1. Determine entry first: Based on user-provided information (service_name, node_name), determine whether the entry is service fault, instance fault, or host fault before starting diagnosis.
2. Progressive investigation: Always start with quick log scan (Step 3); only escalate to detailed investigation (Step 4) when no clear conclusion is reached.
3. Substitute placeholders: Replace , , , , ``, etc. with actual user-provided values; never hardcode them.
4. Quote parameters: Always wrap -p values in single quotes; on Windows PowerShell, escape " as """ to avoid code:500 errors.
5. Time format: alarm_time must follow yyyy/MM/dd HH:mm:ss GMT+X:XX; for non-standard log time formats, pass time_pattern.
6. Summarize results: Use a summarization tool to condense command output before analysis; large raw outputs should not be analyzed directly.
7. Reflect after diagnosis: After completing the checks, reflect on whether the root cause is confirmed; if not, re-check for missed steps.
8. Read-only: All commands are read-only; repair steps are suggestions only and require user confirmation before execution.
9. Command failure handling: When a command fails, skip the current check item and continue with the other checks; do not abort the whole diagnosis.

References

Notes

• Security: This skill is read-only. It never exposes the LakeWatch password; the password is encrypted via --encrypt-password and stored in lakewatch_api_config.yaml. Repair steps are suggestions only.
• No KooCLI: This skill does not use hcloud; it calls the LakeWatch API through lakewatch_api_client.py. Do not mix in hcloud commands.
• Command failure: When a command fails, skip the current check item and continue with the other checks; do not abort the whole diagnosis.
• Known limitations: The access_manager_get proxy only supports GET requests (PUT is not yet available on the Agent side); collect_alarm_log_data requires log_directory to be under /var/log/; some strategy_name values require extra env parameters.
• Cross-skill dependency: This skill references alarm diagnosis documents from the huawei-cloud-mrs-host-alarm-diagnose skill (e.g. ../huawei-cloud-mrs-host-alarm-diagnose/alarms/12006.md, 12007.md). See Prerequisites section 4 for the dependency declaration and handling rules. If the alarm skill is not installed, inform the user and proceed with the generic fault diagnosis flow.