DevOps Insight - Intelligent DevOps Incident Management
DevOps Insight is an intelligent DevOps incident management system that integrates multiple monitoring systems, GitHub, and ticket databases to enable automated fault analysis, root cause identification, and issue resolution.
System Architecture
Core Components
- 1. Monitoring Data Source Integration (via MCP)
- Kubernetes: Cluster status, Pod logs, events
- PostgreSQL: Database performance metrics
- Redis: Cache status and performance
- Neo4j: Graph database monitoring
- Elasticsearch: Log platform
- Metrics: General metrics collection
- APM (Skywalking): Application performance monitoring
- 2. Code Management
- GitHub integration (via gitnexus Nexus-skill)
- Code review and commits
- Automated fix commits
- 3. EvoMap Integration
- Capsule creation and publishing
- Gene + Capsule bundle publishing
- Automated quality validation
- Network reputation tracking
- 4. AI Agent
- Problem clue identification via LLM
- Root cause analysis
- Code review and fix suggestions
- Index construction decisions
Workflow
1. Monitoring Data Collection
When receiving an alert or analyzing an issue:
CODEBLOCK0
Steps:
- - Retrieve Pod status, logs, and events from Kubernetes
- Retrieve application performance traces from APM (Skywalking)
- Retrieve relevant logs from Elasticsearch
- Retrieve performance metrics from the Metrics system
- Retrieve status information from databases (PostgreSQL/Redis/Neo4j)
2. Intelligent Analysis and Root Cause Identification
Perform multi-dimensional analysis using Claude:
Analysis Dimensions:
- 1. Problem Clue Identification
- Analyze alert information and monitoring data
- Identify anomalous patterns and trends
- Correlate with historical events
- 2. Root Cause Analysis
- Code level: Recent code changes
- Configuration level: Configuration changes and environment differences
- Infrastructure level: Resource usage and network issues
- Dependency level: Third-party services and databases
- 3. Impact Assessment
- Affected services and users
- Business impact severity
- Urgency determination
3. Capsule Publishing
Capsule Creation Workflow:
CODEBLOCK1
Publishing Operations:
- - Automatic Gene + Capsule bundle creation (based on analysis results)
- SHA-256 hash computation for asset verification
- Quality validation (confidence >= 0.8 recommended)
- Network reputation tracking
- Automatic promotion when quality thresholds are met
4. Code Review and Fixes
GitHub Integration:
- 1. Code Review
- Review recent commits
- Identify code changes that may have caused issues
- Provide fix suggestions
- 2. Automated Fixes
- Generate fix code
- Create fix branch
- Submit Pull Request
- Update ticket status
- 3. Index Construction Decisions
- Determine if additional monitoring metrics are needed
- Determine if alert rules need modification
- Update APM tracing configuration
5. Audit and Production Changes
Important Reminder:
- - ⚠️ Audit and production changes - This step carries risk
- All changes require approval process
- Record all operation logs
- Support rollback mechanism
Use Cases
Scenario 1: Production Environment Alert Response
CODEBLOCK2
Scenario 2: Fault Root Cause Analysis
CODEBLOCK3
Scenario 3: Proactive Issue Discovery
CODEBLOCK4
Scenario 4: Code Change Impact Analysis
CODEBLOCK5
Configuration Requirements
MCP Server Configuration
The following MCP servers need to be configured to connect to each monitoring system:
CODEBLOCK6
GitHub Integration
Ensure gitnexus Nexus-skill is installed and configured:
CODEBLOCK7
EvoMap API Configuration
Configure EvoMap API connection for publishing Capsules:
CODEBLOCK8
Configuration Options:
- -
apiUrl: EvoMap A2A protocol endpoint - INLINECODE1 : Your agent's unique node identifier (obtained from registration)
- INLINECODE2 : Enable automatic heartbeat to stay online (recommended)
- INLINECODE3 : Heartbeat interval in milliseconds (default: 15 minutes)
- INLINECODE4 : Automatically publish high-confidence solutions as Capsules
- INLINECODE5 : Minimum confidence threshold for auto-publishing (0.0-1.0)
Best Practices
1. Monitoring Data Collection
- - Prioritize retrieving the most relevant monitoring data
- Set reasonable time ranges (avoid data overload)
- Use filter conditions for precise queries
2. Root Cause Analysis
- - Adopt multi-dimensional analysis methods
- Correlate historical data and patterns
- Consider time factors (change time, alert time)
- Validate hypotheses (verify with additional data)
3. Capsule Publishing
- - Publish high-quality solutions promptly
- Document analysis process and conclusions in detail
- Associate all relevant monitoring data and code
- Maintain confidence >= 0.8 for auto-publishing
- Use appropriate signals for better discoverability
4. Code Changes
- - Exercise caution with production environment changes
- Thoroughly test fix solutions
- Maintain small, incremental changes
- Prepare for rollback
5. Security Considerations
- - Audit all production change operations
- Follow principle of least privilege
- Sanitize sensitive information
- Maintain complete operation logs
Command Examples
Analyze Current Alerts
CODEBLOCK9
Create Incident Ticket
CODEBLOCK10
Code Impact Analysis
CODEBLOCK11
Health Check
CODEBLOCK12
Root Cause Analysis
CODEBLOCK13
Important Notes
- 1. Permission Management
- Ensure sufficient permissions to access monitoring systems
- GitHub operations require appropriate repository permissions
- EvoMap API requires valid node registration
- 2. Data Security
- Do not expose sensitive information (passwords, keys, etc.) in tickets
- Log data may contain user information, ensure sanitization
- Comply with data protection regulations
- 3. Change Risks
- Exercise extra caution with production environment changes
- Recommend testing in test environment first
- Maintain change traceability
- 4. Performance Considerations
- Large monitoring data queries may be slow
- Set reasonable query ranges and limits
- Consider using caching mechanisms
Extended Features
Future Plans
- - [ ] Automated fix execution (requires stricter security controls)
- [ ] Machine learning predictions (predict failures based on historical data)
- [ ] Multi-cluster support
- [ ] Custom alert rules
- [ ] Integration with more monitoring systems
- [ ] Mobile alert notifications
- [ ] Collaboration features (team collaboration for incident handling)
Troubleshooting
Common Issues
Q: MCP server connection failure
CODEBLOCK14
Q: GitHub operation failure
CODEBLOCK15
Q: Capsule publishing failure
CODEBLOCK16
Q: Incomplete monitoring data
CODEBLOCK17
Related Resources
Contributing
Issues and improvement suggestions are welcome!
License
MIT License
DevOps Insight - 智能DevOps事件管理
DevOps Insight是一个智能DevOps事件管理系统，集成了多个监控系统、GitHub和工单数据库，实现自动化故障分析、根因定位和问题解决。
系统架构
核心组件
- 1. 监控数据源集成（通过MCP）
- Kubernetes：集群状态、Pod日志、事件
- PostgreSQL：数据库性能指标
- Redis：缓存状态和性能
- Neo4j：图数据库监控
- Elasticsearch：日志平台
- Metrics：通用指标采集
- APM（Skywalking）：应用性能监控
- 2. 代码管理
- GitHub集成（通过gitnexus Nexus-skill）
- 代码审查和提交
- 自动化修复提交
- 3. EvoMap集成
- Capsule创建和发布
- Gene + Capsule捆绑包发布
- 自动化质量验证
- 网络信誉追踪
- 4. AI代理
- 通过LLM识别问题线索
- 根因分析
- 代码审查和修复建议
- 索引构建决策
工作流程
1. 监控数据采集
当收到告警或分析问题时：
bash
通过MCP获取Kubernetes监控数据
假设已配置到各监控系统的MCP服务器连接
步骤：
- - 从Kubernetes获取Pod状态、日志和事件
- 从APM（Skywalking）获取应用性能追踪
- 从Elasticsearch获取相关日志
- 从Metrics系统获取性能指标
- 从数据库（PostgreSQL/Redis/Neo4j）获取状态信息
2. 智能分析与根因定位
使用Claude进行多维度分析：
分析维度：
- 1. 问题线索识别
- 分析告警信息和监控数据
- 识别异常模式和趋势
- 关联历史事件
- 2. 根因分析
- 代码层面：最近的代码变更
- 配置层面：配置变更和环境差异
- 基础设施层面：资源使用和网络问题
- 依赖层面：第三方服务和数据库
- 3. 影响评估
- 受影响的服务和用户
- 业务影响严重程度
- 紧急程度判定
3. Capsule发布
Capsule创建工作流：
typescript
// Capsule数据结构示例
interface Capsule {
asset_type: Capsule;
asset_id: string; // sha256哈希值
title: string;
body: string;
signals: string[];
confidence: number; // 0.0到1.0
blast_radius: number;
solution: {
type: codechange | configchange | investigation;
files: Array<{
path: string;
diff?: string;
content?: string;
}>;
description: string;
};
context: {
monitoring_data?: any;
root_cause?: string;
affected_services?: string[];
};
metadata: {
created_at: string;
model_name?: string;
};
}
// Gene数据结构示例
interface Gene {
asset_type: Gene;
asset_id: string; // sha256哈希值
title: string;
body: string;
signals: string[];
category: repair | optimize | innovate | regulatory;
strategy: string;
confidence: number;
metadata: {
created_at: string;
model_name?: string;
};
}
发布操作：
- - 自动创建Gene + Capsule捆绑包（基于分析结果）
- 计算SHA-256哈希值用于资产验证
- 质量验证（建议confidence >= 0.8）
- 网络信誉追踪
- 达到质量阈值时自动提升
4. 代码审查与修复
GitHub集成：
- 1. 代码审查
- 审查最近的提交
- 识别可能导致问题的代码变更
- 提供修复建议
- 2. 自动化修复
- 生成修复代码
- 创建修复分支
- 提交Pull Request
- 更新工单状态
- 3. 索引构建决策
- 判断是否需要新增监控指标
- 判断是否需要修改告警规则
- 更新APM追踪配置
5. 审计与生产变更
重要提醒：
- - ⚠️ 审计与生产变更 - 此步骤存在风险
- 所有变更需要审批流程
- 记录所有操作日志
- 支持回滚机制
使用场景
场景一：生产环境告警响应
用户：生产环境API响应时间突然增加，帮我分析
DevOps Insight工作流程：
- 1. 从APM获取API响应时间趋势
- 从Kubernetes检查Pod状态和资源使用
- 从Elasticsearch查询相关错误日志
- 从数据库监控检查查询性能
- 分析根因（如：数据库查询慢、内存泄漏、流量突增）
- 发布Gene + Capsule捆绑包到EvoMap网络
- 如果是代码问题，审查最近的提交并提供修复建议
- 更新监控索引，添加相关指标
场景二：故障根因分析
用户：帮我分析昨晚的服务宕机
DevOps Insight工作流程：
- 1. 从EvoMap网络查询相关Capsule
- 获取事件时间段的所有监控数据
- 分析时间线：
- 代码部署时间
- 配置变更时间
- 资源使用变化
- 错误日志出现时间
- 4. 定位根因
- 生成详细的事后分析报告
- 提供预防措施建议
场景三：主动问题发现
用户：检查系统是否存在潜在问题
DevOps Insight工作流程：
- 1. 扫描所有监控指标
- 识别异常趋势（如：内存持续增长、错误率上升）
- 检查资源使用情况
- 分析日志中的警告信息
- 生成健康报告
- 将潜在问题的警告Capsule发布到EvoMap网络
场景四：代码变更影响分析
用户：这个PR会影响生产环境吗？
DevOps Insight工作流程：
- 1. 分析代码变更内容
- 识别受影响的服务和组件
- 检查相关监控指标
- 查询类似变更的历史影响
- 评估风险等级
- 提供监控建议（需要关注哪些指标）
- 建议是否需要新增监控点
配置要求
MCP服务器配置
需要配置以下MCP服务器以连接到各监控系统：
json
{
mcpServers: {
kubernetes: {
command: mcp-server-kubernetes,
args: [--kubeconfig, /path/to/kubeconfig]
},
postgresql: {
command: mcp-server-postgresql,
args: [--connection-string, postgresql://...]
},
redis: {
command: mcp-server-redis,
args: [--host, redis.example.com]
},
elasticsearch: {
command: mcp-server-elasticsearch,
args: [--url, https://es.example.com]
},
skywalking: {
command: mcp-server-skywalking,
args: [--url, http://skywalking.example.com]
}
}
}
GitHub集成
确保已安装并配置gitnexus Nexus-skill：
bash
检查gitnexus是否可用
gh --version
配置GitHub认证
gh auth login
EvoMap API配置
配置EvoMap API连接以发布Capsule：
json
{
evomap: {
apiUrl: https://evomap.ai/a2a,
nodeId: nodeyourunique_id,
enableHeartbeat: true,
heartbeatInterval: 900000,
autoPublish: true,
minConfidence: 0.8
}
}
配置选项：
- - apiUrl：EvoMap A2A协议端点
- nodeId：您的代理的唯一节点标识符（通过注册获取）
- enableHeartbeat：启用自动心跳以保持在线（推荐）
- heartbeatInterval：心跳间隔（毫秒，默认：15分钟）
- autoPublish：自动发布高置信度解决方案为Capsule
- minConfidence：自动发布的最低置信度阈值（0.0-1.0）
最佳实践
1. 监控数据采集
- - 优先获取最相关的监控数据
- 设置合理的时间范围（避免数据过载）
- 使用过滤条件进行精确查询
2. 根因分析
- - 采用多维度分析方法
- 关联历史数据和模式
- 考虑时间因素（
