代码拉取完成,页面将自动刷新
同步操作将从 KubeSphere/kubeeye 强制同步,此操作会覆盖自 Fork 仓库以来所做的任何修改,且无法恢复!!!
确定后同步将在后台操作,完成时将刷新页面,请耐心等待。
English | 中文
KubeEye 旨在发现 Kubernetes 上的各种问题,比如应用配置错误(使用Polaris)、集群组件不健康和节点问题(使用Node-Problem-Detector)。除了预定义的规则,它还支持自定义规则。
KubeEye 通过调用Kubernetes API,通过常规匹配日志中的关键错误信息和容器语法的规则匹配来获取集群诊断数据,详见架构。
机器上安装 KubeEye
git clone https://github.com/kubesphere/kubeeye.git
cd kubeeye
make install
[可选] 安装 Node-problem-Detector
注意:这一行将在你的集群上安装 npd,只有当你想要详细的报告时才需要。
ke install npd
KubeEye 执行
root@node1:# ke diag
NODENAME SEVERITY HEARTBEATTIME REASON MESSAGE
node18 Fatal 2020-11-19T10:32:03+08:00 NodeStatusUnknown Kubelet stopped posting node status.
node19 Fatal 2020-11-19T10:31:37+08:00 NodeStatusUnknown Kubelet stopped posting node status.
node2 Fatal 2020-11-19T10:31:14+08:00 NodeStatusUnknown Kubelet stopped posting node status.
node3 Fatal 2020-11-27T17:36:53+08:00 KubeletNotReady Container runtime not ready: RuntimeReady=false reason:DockerDaemonNotReady message:docker: failed to get docker version: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
NAME SEVERITY TIME MESSAGE
scheduler Fatal 2020-11-27T17:09:59+08:00 Get http://127.0.0.1:10251/healthz: dial tcp 127.0.0.1:10251: connect: connection refused
etcd-0 Fatal 2020-11-27T17:56:37+08:00 Get https://192.168.13.8:2379/health: dial tcp 192.168.13.8:2379: connect: connection refused
NAMESPACE SEVERITY PODNAME EVENTTIME REASON MESSAGE
default Warning node3.164b53d23ea79fc7 2020-11-27T17:37:34+08:00 ContainerGCFailed rpc error: code = Unknown desc = Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
default Warning node3.164b553ca5740aae 2020-11-27T18:03:31+08:00 FreeDiskSpaceFailed failed to garbage collect required amount of images. Wanted to free 5399374233 bytes, but freed 416077545 bytes
default Warning nginx-b8ffcf679-q4n9v.16491643e6b68cd7 2020-11-27T17:09:24+08:00 Failed Error: ImagePullBackOff
default Warning node3.164b5861e041a60e 2020-11-27T19:01:09+08:00 SystemOOM System OOM encountered, victim process: stress, pid: 16713
default Warning node3.164b58660f8d4590 2020-11-27T19:01:27+08:00 OOMKilling Out of memory: Kill process 16711 (stress) score 205 or sacrifice child Killed process 16711 (stress), UID 0, total-vm:826516kB, anon-rss:819296kB, file-rss:0kB, shmem-rss:0kB
insights-agent Warning workloads-1606467120.164b519ca8c67416 2020-11-27T16:57:05+08:00 DeadlineExceeded Job was active longer than specified deadline
kube-system Warning calico-node-zvl9t.164b3dc50580845d 2020-11-27T17:09:35+08:00 DNSConfigForming Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 100.64.11.3 114.114.114.114 119.29.29.29
kube-system Warning kube-proxy-4bnn7.164b3dc4f4c4125d 2020-11-27T17:09:09+08:00 DNSConfigForming Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 100.64.11.3 114.114.114.114 119.29.29.29
kube-system Warning nodelocaldns-2zbhh.164b3dc4f42d358b 2020-11-27T17:09:14+08:00 DNSConfigForming Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 100.64.11.3 114.114.114.114 119.29.29.29
NAMESPACE SEVERITY NAME KIND TIME MESSAGE
kube-system Warning node-problem-detector DaemonSet 2020-11-27T17:09:59+08:00 [livenessProbeMissing runAsPrivileged]
kube-system Warning calico-node DaemonSet 2020-11-27T17:09:59+08:00 [runAsPrivileged cpuLimitsMissing]
kube-system Warning nodelocaldns DaemonSet 2020-11-27T17:09:59+08:00 [cpuLimitsMissing runAsPrivileged]
default Warning nginx Deployment 2020-11-27T17:09:59+08:00 [cpuLimitsMissing livenessProbeMissing tagNotSpecified]
insights-agent Warning workloads CronJob 2020-11-27T17:09:59+08:00 [livenessProbeMissing]
insights-agent Warning cronjob-executor Job 2020-11-27T17:09:59+08:00 [livenessProbeMissing]
kube-system Warning calico-kube-controllers Deployment 2020-11-27T17:09:59+08:00 [cpuLimitsMissing livenessProbeMissing]
kube-system Warning coredns Deployment 2020-11-27T17:09:59+08:00 [cpuLimitsMissing]
您可以参考常见FAQ内容来优化您的集群。
是/否 | 检查项 | 描述 |
---|---|---|
ETCDHealthStatus | 如果 etcd 启动并正常运行 | |
ControllerManagerHealthStatus | 如果 kubernetes kube-controller-manager 正常启动并运行 | |
SchedulerHealthStatus | 如果 kubernetes kube-schedule 正常启动并运行 | |
NodeMemory | 如果节点内存使用量超过阈值 | |
DockerHealthStatus | 如果 docker 正常运行 | |
NodeDisk | 如果节点磁盘使用量超过阈值 | |
KubeletHealthStatus | 如果 kubelet 激活状态且正常运行 | |
NodeCPU | 如果节点 CPU 使用量超过阈值 | |
NodeCorruptOverlay2 | Overlay2 不可用 | |
NodeKernelNULLPointer | node 显示 NotReady | |
NodeDeadlock | 死锁是指两个或两个以上的进程在争夺资源时互相等待的现象。 | |
NodeOOM | 监控那些消耗过多内存的进程,尤其是那些消耗大量内存非常快的进程,内核会杀掉它们,防止它们耗尽内存 | |
NodeExt4Error | Ext4 挂载失败 | |
NodeTaskHung | 检查D状态下是否有超过 120s 的进程 | |
NodeUnregisterNetDevice | 检查对应网络 | |
NodeCorruptDockerImage | 检查 docker 镜像 | |
NodeAUFSUmountHung | 检查存储 | |
NodeDockerHung | Docker hang住, 检查 docker 的日志 | |
PodSetLivenessProbe | 如果为pod中的每一个容器设置了 livenessProbe | |
PodSetTagNotSpecified | 镜像地址没有声明标签或标签是最新 | |
PodSetRunAsPrivileged | 以特权模式运行 Pod 意味着 Pod 可以访问主机的资源和内核功能 | |
PodSetImagePullBackOff | Pod 无法正确拉出镜像,因此可以在相应节点上手动拉出镜像 | |
PodSetImageRegistry | 检查镜像形式是否在相应仓库 | |
PodSetCpuLimitsMissing | 未声明 CPU 资源限制 | |
PodNoSuchFileOrDirectory | 进入容器查看相应文件是否存在 | |
PodIOError | 这通常是由于文件 IO 性能瓶颈 | |
PodNoSuchDeviceOrAddress | 检查对应网络 | |
PodInvalidArgument | 检查对应存储 | |
PodDeviceOrResourceBusy | 检查对应的目录和 PID | |
PodFileExists | 检查现有文件 | |
PodTooManyOpenFiles | 程序打开的文件/套接字连接数超过系统设置值 | |
PodNoSpaceLeftOnDevice | 检查磁盘和索引节点的使用情况 | |
NodeApiServerExpiredPeriod | 将检查 ApiServer 证书的到期日期少于30天 | |
PodSetCpuRequestsMissing | 未声明 CPU 资源请求值 | |
PodSetHostIPCSet | 设置主机 IP | |
PodSetHostNetworkSet | 设置主机网络 | |
PodHostPIDSet | 设置主机 PID | |
PodMemoryRequestsMiss | 没有声明内存资源请求值 | |
PodSetHostPort | 设置主机端口 | |
PodSetMemoryLimitsMissing | 没有声明内存资源限制值 | |
PodNotReadOnlyRootFiles | 文件系统未设置为只读 | |
PodSetPullPolicyNotAlways | 镜像拉策略并非总是如此 | |
PodSetRunAsRootAllowed | 以 root 用户执行 | |
PodDangerousCapabilities | 您在 ALL / SYS_ADMIN / NET_ADMIN 等功能中有危险的选择 | |
PodlivenessProbeMissing | 未声明 ReadinessProbe | |
privilegeEscalationAllowed | 允许特权升级 | |
NodeNotReadyAndUseOfClosedNetworkConnection | http 2-max-streams-per-connection | |
NodeNotReady | 无法启动 ContainerManager 无法设置属性 TasksAccounting 或未知属性 |
未标注的项目正在开发中
ke install npd
kubectl edit cm -n kube-system node-problem-detector-config
checks:
imageFromUnauthorizedRegistry: warning
customChecks:
imageFromUnauthorizedRegistry:
promptMessage: When the corresponding rule does not match. Show that image from an unauthorized registry.
category: Images
target: Container
schema:
'$schema': http://json-schema.org/draft-07/schema
type: object
properties:
image:
type: string
not:
pattern: ^quay.io
root:# ke diag -f rule.yaml --kubeconfig ~/.kube/config
NAMESPACE SEVERITY NAME KIND TIME MESSAGE
default Warning nginx Deployment 2020-11-27T17:18:31+08:00 [imageFromUnauthorizedRegistry]
kube-system Warning node-problem-detector DaemonSet 2020-11-27T17:18:31+08:00 [livenessProbeMissing runAsPrivileged]
kube-system Warning calico-node DaemonSet 2020-11-27T17:18:31+08:00 [cpuLimitsMissing runAsPrivileged]
kube-system Warning calico-kube-controllers Deployment 2020-11-27T17:18:31+08:00 [cpuLimitsMissing livenessProbeMissing]
kube-system Warning nodelocaldns DaemonSet 2020-11-27T17:18:31+08:00 [runAsPrivileged cpuLimitsMissing]
default Warning nginx Deployment 2020-11-27T17:18:31+08:00 [livenessProbeMissing cpuLimitsMissing]
kube-system Warning coredns Deployment 2020-11-27T17:18:31+08:00 [cpuLimitsMissing]
此处可能存在不合适展示的内容,页面不予展示。您可通过相关编辑功能自查并修改。
如您确认内容无涉及 不当用语 / 纯广告导流 / 暴力 / 低俗色情 / 侵权 / 盗版 / 虚假 / 无价值内容或违法国家有关法律法规的内容,可点击提交进行申诉,我们将尽快为您处理。