暗能星系

zhanglu

#!/bin/bash

1. 定义日志文件路径（默认是当前目录的nohup.out，可根据实际路径修改）

LOG_FILE="./nohup.out"

2. 检查日志文件是否存在

if [ ! -f "$LOG_FILE" ]; then
echo "错误：日志文件 $LOG_FILE 不存在！请检查路径是否正确。"
exit 1
fi

3. 实时监听日志 + 提取目标任务ID（UUID）

echo "=== 开始监听日志 $LOG_FILE，提取含 Restarting 的任务ID ==="
echo "=== 按 Ctrl+C 停止监听 ==="
echo "=========================="

核心逻辑：

- tail -f：实时跟踪日志新增内容

- grep "Restarting"：筛选包含“Restarting”的行

- sed 正则：提取“WorkflowActor-”后的36位UUID（格式：8-4-4-4-12位字符）

- sort -u：去重（避免同一任务多次重启导致重复输出）

tail -f "$LOG_FILE" |
grep --line-buffered "Restarting" |
sed -n 's/.WorkflowActor-([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12})./\1/p' |
sort -u

zhanglu

estarting micro_dy_gro.upstream
[INFO] [09/23/2025 06:46:37.396] [cromwell-system-akka.dispatchers.engine-dispatcher-9] [akka://cromwell-system/user/cromwell-service/WorkflowManagerActor/WorkflowActor-fa3a52b6-19db-4435-ac0f-a5c1fbeec385/WorkflowExecutionActor-fa3a52b6-19db-4435-ac0f-a5c1fbeec385] WorkflowExecutionActor-fa3a52b6-19db-4435-ac0f-a5c1fbeec385 [UUID(fa3a52b6)]: Restarting blood_meta.jsonFile, blood_meta.reportNoFile, blood_meta.resFile
################# retry : Some(9999) ################################## retry : Some(9999) ################################## retry : Some(9999) ################################## retry : Some(9999) ################################## retry : Some(9999) ################################## retry : Some(9999) ################################## retry : Some(9999) ################################## retry : Some(9998) ################################## retry : Some(9998) ################################## retry : Some(9999) ################################## retry : Some(9999) ################################## retry : Some(9999) ###############

zhanglu

Type Reason Age From Message

Normal NodeReady 47m (x11 over 6h56m) kubelet Node node1 status is now: NodeReady
Normal NodeNotReady 44m (x12 over 7h3m) kubelet Node node1 status is now: NodeNotReady
Normal Starting 37m kubelet Starting kubelet.
Normal NodeHasSufficientMemory 37m kubelet Node node1 status is now: NodeHasSufficientMemory
Normal NodeHasNoDiskPressure 37m kubelet Node node1 status is now: NodeHasNoDiskPressure
Normal NodeHasSufficientPID 37m kubelet Node node1 status is now: NodeHasSufficientPID
Normal NodeAllocatableEnforced 37m kubelet Updated Node Allocatable limit across pods
Normal NodeReady 37m kubelet Node node1 status is now: NodeReady
Normal NodeNotReady 34m kubelet Node node1 status is now: NodeNotReady
Normal Starting 31m kubelet Starting kubelet.
Normal NodeHasSufficientMemory 31m kubelet Node node1 status is now: NodeHasSufficientMemory
Normal NodeHasNoDiskPressure 31m kubelet Node node1 status is now: NodeHasNoDiskPressure
Normal NodeHasSufficientPID 31m kubelet Node node1 status is now: NodeHasSufficientPID
Normal NodeAllocatableEnforced 31m kubelet Updated Node Allocatable limit across pods
Normal NodeReady 9m29s (x2 over 31m) kubelet Node node1 status is now: NodeReady
Normal NodeNotReady 6m28s (x2 over 28m) kubelet Node node1 status is now: NodeNotReady

zhanglu

安装NVIDIA仓库配置包（适用于CentOS 8）

distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.repo | sudo tee /etc/yum.repos.d/nvidia-docker.repo

sudo dnf install -y nvidia-container-toolkit

zhanglu

nvidia-ctk runtime configure --runtime=docker

zhanglu

kubectl get pods --all-namespaces -o json | jq -r '.items[] | select(.spec.containers[].resources.limits["<gpu-resource-name>"] != null) | .metadata.namespace + " " + .metadata.name'

zhanglu

2025/10/13 02:31:43 Starting FS watcher.
2025/10/13 02:31:43 Starting OS watcher.
2025/10/13 02:31:43 Starting Plugins.
2025/10/13 02:31:43 Loading configuration.
2025/10/13 02:31:43 Initializing NVML.
2025/10/13 02:31:43 Failed to initialize NVML: could not load NVML library.
2025/10/13 02:31:43 If this is a GPU node, did you set the docker default runtime to nvidia?
2025/10/13 02:31:43 You can check the prerequisites at: https://github.com/NVIDIA/k8s-device-plugin#prerequisites
2025/10/13 02:31:43 You can learn how to set the runtime at: https://github.com/NVIDIA/k8s-device-plugin#quick-start
2025/10/13 02:31:43 If this is not a GPU node, you should set up a toleration or nodeSelector to only deploy this plugin on GPU nodes

zhanglu

https://cn.download.nvidia.com/XFree86/Linux-x86_64/580.95.05/NVIDIA-Linux-x86_64-580.95.05.run

zhanglu

2025/10/13 07:23:31 Failed to initialize NVML: could not load NVML library.
2025/10/13 07:23:31 If this is a GPU node, did you set the docker default runtime to nvidia?

zhanglu

wget https://developer.download.nvidia.com/compute/cuda/13.0.2/local_installers/cuda_13.0.2_580.95.05_linux.run

zhanglu

wget https://developer.download.nvidia.com/compute/cuda/11.6.2/local_installers/cuda_11.6.2_510.47.03_linux.run

zhanglu

Failed to initialize NVML: could not load NVML library

zhanglu

https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/v0.17.1/deployments/static/nvidia-device-plugin.yml

zhanglu

https://www.nvidia.cn/drivers/details/252785/

zhanglu

$ curl -s -L https://nvidia.github.io/nvidia-docker/centos8/nvidia-docker.repo | sudo tee /etc/yum.repos.d/nvidia-docker.repo
yum install -y nvidia-container-toolkit

zhanglu

d?????????? ? ? ? ? ? cephfs_data

zhanglu

root
Tzzs@2025*
220.185.228.106
30001

zhanglu

http://192.168.30.202:31237/api/workflows/v1/8a6aa1f8-0ab1-4518-bc22-06390d1c7494/abort

zhanglu

curl -X POST "http://192.168.30.202:31237/api/workflows/v1/8a6aa1f8-0ab1-4518-bc22-06390d1c7494/abort" -H "accept: application/json"

zhanglu

tmp