暗能星系

zhanglu

kubectl -n rook-ceph get secret rook-ceph-dashboard-password -o jsonpath="{['data']['password']}" | base64 --decode && echo

zhanglu

mount -t ceph 10.233.31.47:6789,10.233.36.144:6789,10.233.33.7:6789:/ /cephfs_data -o name=admin,secret=AQDw/2VjAJmTBxAAezkJzzXVJ4VCxiYtx49L+w==

zhanglu

docker pull dockerhub.genostack.com:8090/library/web_socket:latest

zhanglu

PostgreSQL Database directory appears to contain a database; Skipping initialization

2025-04-02 13:24:07.154 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
2025-04-02 13:24:07.154 UTC [1] LOG: listening on IPv6 address "::", port 5432
2025-04-02 13:24:07.157 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2025-04-02 13:24:07.411 UTC [26] LOG: database system was shut down at 2025-04-02 11:47:48 UTC
2025-04-02 13:24:07.411 UTC [26] LOG: invalid record length at 8/DBE56CD8: wanted 24, got 0
2025-04-02 13:24:07.411 UTC [26] LOG: invalid primary checkpoint record
2025-04-02 13:24:07.411 UTC [26] LOG: invalid resource manager ID in secondary checkpoint record
2025-04-02 13:24:07.411 UTC [26] PANIC: could not locate a valid checkpoint record
2025-04-02 13:24:08.323 UTC [1] LOG: startup process (PID 26) was terminated by signal 6: Aborted
2025-04-02 13:24:08.323 UTC [1] LOG: aborting startup due to startup process failure
2025-04-02 13:24:08.338 UTC [1] LOG: database system is shut down

zhanglu

PostgreSQL Database directory appears to contain a database; Skipping initialization

2025-04-07 04:24:18.838 UTC [1] FATAL: lock file "postmaster.pid" is empty
2025-04-07 04:24:18.838 UTC [1] HINT: Either another server is starting, or the lock file is the remnant of a previous server startup crash.

zhanglu

kill -9 lsof | grep delete | awk '{print $2}'

zhanglu

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDW8bqVdBe8okVnEaWxn/GCBn2ngPWezwomczuS+tcj/T3odSjNPe6NQVRb+KkumnwMk8CHLq41v1J+C9BZp2x/w90pCKtSjotl94CbNzqnEFJK+tfrKG38twMlPnCeHfsGXEeagbT0Td4nqhvvM2oz1gEzYwoNsY3VV/hufLbj0qNBeoKOqltTUtdRBaNyuxDxIzh+uHy5LrJAWjftGkjOHbJDXp+HEgXsUDyhr13lnAm2s3WPT5jLLiZvPiTBnrEscLZEoLkfelUJMwGOOIfTakd7illHLUtyRO9UcG3cYvUOTc2Pfp/SDLcuxrHbfpsRSz9ZD4ZzsX50SjTxQR3O44RGk2b4SQK7JNhEIDIzbhXbGTRPQVK4yI+ox6NKKYgj/GkxxMCErcAWa/o9PGLYEHJSuyK8vaE/Dh0rHYgUtq0EIolQ0DLPHa4vzlhkxK9j5qUtu6oZkYcqIhevz+blwnQ2h8DIWmK+9AL/QmqvbfWysMyPzXj7/475Bmt5aZ0= admin@example.com

zhanglu

/cephfs_data/genostack_v3/genostack_core/genostack_php_new/project_data/project_data/320/Mus_musculus.GRCm39.dna.primary_assembly.fa.gz

zhanglu

task-33146a1d
task-5dfacee4
task-7fed90b9
task-267924fd
task-02b7953c
task-56f81fc9
task-482572b9
task-9c85fa81
task-10a5f9fb
task-ce516c8b
task-d93f84e3
task-d6fac1ef
task-dde9bd01
task-e8bfb791

zhanglu

'task-33146a1d','task-5dfacee4','task-7fed90b9','task-267924fd','task-02b7953c','task-56f81fc9','task-482572b9','task-9c85fa81','task-10a5f9fb','task-ce516c8b','task-d93f84e3','task-d6fac1ef','task-dde9bd01','task-e8bfb791'

zhanglu

http://192.168.30.202:30280/d/StarsL_cn_K8S/k8s-dashboard-cn-20240513-starsl-cn?orgId=1&from=now-30m&to=now&timezone=browser&var-origin_prometheus=&var-Node=$__all&var-NameSpace=$__all&var-Container=$__all&var-Pod=$__all

zhanglu

zhanglu

调整内核脏页参数，触发自动写入
通过降低 “脏页阈值”，让内核更积极地将脏页写入磁盘：
bash

临时生效（重启后恢复默认）

脏页占总内存比例超过 5% 时，内核强制同步（默认 20%）

sysctl -w vm.dirty_ratio=5

后台进程在脏页占比超过 3% 时开始同步（默认 10%）

sysctl -w vm.dirty_background_ratio=3

脏页存活时间超过 10 秒后强制同步（默认 30 秒）

sysctl -w vm.dirty_expire_centisecs=1000 # 1000 厘秒 = 10 秒

zhanglu

ceph status
cluster:
id: 807d820b-5c5b-451c-9f52-41b93d5d905a
health: HEALTH_ERR
1 MDSs report oversized cache
1 MDSs report slow metadata IOs
2 MDSs behind on trimming
mon bh is low on available space
10 backfillfull osd(s)
1 OSDs or CRUSH {nodes, device-classes} have {NOUP,NODOWN,NOIN,NOOUT} flags set
full ratio(s) out of order
Low space hindering backfill (add storage if this doesn't resolve itself): 22 pgs backfill_toofull
Degraded data redundancy: 715/452014992 objects degraded (0.000%), 150 pgs degraded, 2 pgs undersized
206 pgs not deep-scrubbed in time
128 pgs not scrubbed in time
4 pool(s) backfillfull

zhanglu

[WARN] [09/06/2025 06:54:11.355] [cromwell-system-akka.actor.default-dispatcher-3] [cromwell-system/Pool(shared->http://tesk-api.default.svc.cluster.local:8080)] [1 (WaitingForResponseEntitySubscription)] Response entity was not subscribed after 1 second. Make sure to read the response entity body or call discardBytes() on it. POST /ga4gh/tes/v1/tasks/task-68f2dfc5:cancel Empty -> 400 Bad Request Chunked
[WARN] [09/06/2025 06:54:16.926] [cromwell-system-akka.actor.default-dispatcher-24] [cromwell-system/Pool(shared->http://tesk-api.default.svc.cluster.local:8080)] [0 (WaitingForResponseEntitySubscription)] Response entity was not subscribed after 1 second. Make sure to read the response entity body or call discardBytes() on it. POST /ga4gh/tes/v1/tasks/task-3a62617f:cancel Empty -> 400 Bad Request Chunked
[WARN] [09/06/2025 06:54:17.945] [cromwell-system-akka.actor.default-dispatcher-26] [cromwell-system/Pool(shared->http://tesk-api.default.svc.cluster.local:8080)] [1 (WaitingForResponseEntitySubscription)] Response entity was not subscribed after 1 second. Make sure to read the response entity body or call discardBytes() on it. POST /ga4gh/tes/v1/tasks/task-d38cd4a6:cancel Empty -> 400 Bad Request Chunked
[WARN] [09/06/2025 06:54:18.095] [cromwell-system-akka.actor.default-dispatcher-24] [cromwell-system/Pool(shared->http://tesk-api.default.svc.cluster.local:8080)] [3 (WaitingForResponseEntitySubscription)] Response entity was not subscribed after 1 second. Make sure to read the response entity body or call discardBytes() on it. POST /ga4gh/tes/v1/tasks/task-d38cd4a6:cancel Empty -> 400 Bad Request Chunked
[WARN] [09/06/2025 06:54:22.325] [cromwell-system-akka.actor.default-dispatcher-4] [cromwell-system/Pool(shared->http://tesk-api.default.svc.cluster.local:8080)] [2 (WaitingForResponseEntitySubscription)] Response entity was not subscribed after 1 second. Make sure to read the response entity body or call discardBytes() on it. POST /ga4gh/tes/v1/tasks/task-3299d1e5:cancel Empty -> 400 Bad Request Chunked
[WARN] [09/06/2025 06:54:25.545] [cromwell-system-akka.actor.default-dispatcher-2] [cromwell-system/Pool(shared->http://tesk-api.default.svc.cluster.local:8080)] [1 (WaitingForResponseEntitySubscription)] Response entity was not subscribed after 1 second. Make sure to read the response entity body or call discardBytes() on it. POST /ga4gh/tes/v1/tasks/task-68f2dfc5:cancel Empty -> 400 Bad Request Chunked
[WARN] [09/06/2025 06:54:25.575] [cromwell-system-akka.actor.default-dispatcher-3] [cromwell-system/Pool(shared->http://tesk-api.default.svc.cluster.local:8080)] [0 (WaitingForResponseEntitySubscription)] Response entity was not subscribed after 1 second. Make sure to read the response entity body or call discardBytes() on it. POST /ga4gh/tes/v1/tasks/task-3299d1e5:cancel Empty -> 400 Bad Request Chunked
[WARN] [09/06/2025 06:54:30.316] [cromwell-system-akka.actor.default-dispatcher-4] [cromwell-system/Pool(shared->http://tesk-api.default.svc.cluster.local:8080)] [3 (WaitingForResponseEntitySubscription)] Response entity was not subscribed after 1 second. Make sure to read the response entity body or call discardBytes() on it. POST /ga4gh/tes/v1/tasks/task-3a62617f:cancel Empty -> 400 Bad Request Chunked
[WARN] [09/06/2025 06:54:32.715] [cromwell-system-akka.actor.default-dispatcher-4] [cromwell-system/Pool(shared->http://tesk-api.default.svc.cluster.local:8080)] [0 (WaitingForResponseEntitySubscription)] Response entity was not subscribed after 1 second. Make sure to read the response entity body or call discardBytes() on it. POST /ga4gh/tes/v1/tasks/task-d38cd4a6:cancel Empty -> 400 Bad Request Chunked
[WARN] [09/06/2025 06:54:32.865] [cromwell-system-akka.actor.default-dispatcher-24] [cromwell-system/Pool(shared->http://tesk-api.default.svc.cluster.local:8080)] [1 (WaitingForResponseEntitySubscription)] Response entity was not subscribed after 1 second. Make sure to read the response entity body or call discardBytes() on it. POST /ga4gh/tes/v1/tasks/task-d38cd4a6:cancel Empty -> 400 Bad Request Chunked
[WARN] [09/06/2025 06:54:37.295] [cromwell-system-akka.actor.default-dispatcher-2] [cromwell-system/Pool(shared->http://tesk-api.default.svc.cluster.local:8080)] [3 (WaitingForResponseEntitySubscription)] Response entity was not subscribed after 1 second. Make sure to read the response entity body or call discardBytes() on it. POST /ga4gh/tes/v1/tasks/task-3299d1e5:cancel Empty -> 400 Bad Request Chunked

zhanglu

import time
import re
import os

def follow_log(file_path):
"""实时跟踪日志文件的新内容，类似tail -f命令"""
# 打开文件并移动到文件末尾
with open(file_path, 'r') as f:
f.seek(0, os.SEEK_END)

    while True:
        line = f.readline()
        if not line:
            # 没有新内容时短暂休眠
            time.sleep(0.1)
            continue
        yield line

def extract_task_id(line):
"""从日志行中提取task ID"""
# 正则表达式匹配task-开头的ID
pattern = r'task-[0-9a-f]+'
match = re.search(pattern, line)
if match:
return match.group()
return None

def main(log_file_path):
print(f"开始监控日志文件: {log_file_path}")
print("正在提取task ID... (按Ctrl+C停止)")

try:
    for line in follow_log(log_file_path):
        # 检查行中是否包含目标POST请求
        if 'POST /ga4gh/tes/v1/tasks/' in line and ':cancel' in line:
            task_id = extract_task_id(line)
            if task_id:
                print(f"提取到task ID: {task_id}")
except KeyboardInterrupt:
    print("\n程序已停止")
except FileNotFoundError:
    print(f"错误: 找不到日志文件 {log_file_path}")
except Exception as e:
    print(f"发生错误: {str(e)}")

if name == "main":
# 日志文件路径，可根据实际情况修改
log_file = "nohup.out"
main(log_file)

zhanglu

delete from "JOB_KEY_VALUE_ENTRY" where "STORE_VALUE" = '195'

zhanglu

cde89041-fe8a-4712-8385-0cb8afc0efcc, 7cf42b54-9888-486e-9460-9d5b607c02fd, 3178dfab-75f8-4ab3-95a4-408af8e3d7c1, 170c1ac2-e068-456b-b5ea-c948f432e87d, fbafa7ff-408b-4869-b9c5-c66a45d9843f, 9c35008e-0f1a-433d-a974-266d0ac5038d, 0caaa4b6-6a32-4973-8e4f-9f65fb0ece2b, 1b3b877d-cd72-4694-bfad-07fde451e6ff, cd08e0b4-a0b7-4ce3-ae67-79650a6a6bb2, 601d6df9-ab99-4a65-b790-0a2b6cc5c191, 971df97e-e631-4f4a-9d78-4704eb0a7591, c712c011-afd1-4e6c-b1d1-ef6532ff48f4, a66c9c15-22c8-4bb3-920b-9354b408862b, 58d69c6a-df71-4780-8b18-b3020532997f, 1cc9b767-573a-447d-b257-968f2c91df17, dab528d2-1eda-46e3-a232-622cb188a7c5, 2026b96c-019b-4a6c-bc13-e4cc3bb78e98, edabc69e-72d1-48df-942e-528cc60b835d, 846a9eb7-5c60-4171-98db-c7d4b370cf27, d33d18c7-293c-4d0b-ab3b-38f049a305c2, 351735e1-31cd-4577-82d6-e3a4d10d679c, 93acc752-31bf-4538-b3ce-59c0762d00f6, b0599234-046e-4914-9941-00aadf0a292e, 524ddb2b-e921-4483-b6c1-bc449047b6a4, f09dd720-5851-47d5-92d0-a36aa7d2be72, 35d9125b-cee8-4cea-abd9-717e7ba75ce2, 81df380e-50a9-434d-ba6e-c74500038d8d, 2f853868-6d6c-4b51-88b7-506dffd0e4ae, c1eb19f2-9ce4-4ccd-986e-a8abb7c5b737, 3e21be2c-b074-4a4c-9802-173be8ea45eb, 0cf1a30b-4031-4789-819e-f5a3e6f58b50, 042e5576-988e-4bdc-b782-2e212ad88e8c, 4766114f-efee-4200-a3fa-dd109a4ee9df, 3f84c54c-0e03-4ea3-a258-ab71691cee60, 41278df2-f761-4463-8df9-0c9223585348, f47e22e1-a046-4f93-ac9d-65e2801ab9ac, a1611293-04c2-4b81-847a-6a53defa7399, a164dc0d-388f-4b3e-ba7a-5f9543b34185, 98b1d8ba-757d-4599-ad01-b703d58b5d0d, 1ac95950-d62c-4fe8-ba7d-7df645dcebbf, bb3fad51-a03e-4a47-95b3-f74711577c64, beb7ba8d-0909-4f04-9223-1ec72ae7fac4, 057ed284-0b60-4ede-ae87-d7db476c1882, c9c98451-9773-4245-a636-e7dcb3c19282, d7b55685-cdd6-483b-8ea9-38131ad5ea92, 486875ec-76a0-45c5-a734-fd63bc599af3, fcd11583-245f-42a9-94c9-1c4524217923, ecd92665-0550-48d6-99b0-0588b3367469, 071e9c51-931b-4c05-b3e8-a227449f3b9e

zhanglu

curl -X POST "http://192.168.30.202:31237/api/workflows/v1/edabc69e-72d1-48df-942e-528cc60b835d/abort" -H "accept: application/json"

zhanglu

tmp

临时生效（重启后恢复默认）

脏页占总内存比例超过 5% 时，内核强制同步（默认 20%）

后台进程在脏页占比超过 3% 时开始同步（默认 10%）

脏页存活时间超过 10 秒后强制同步（默认 30 秒）