暗能星系

zhanglu

The request operation to custom alerting rule could not be done because thanos ruler is not enabled

zhanglu

akka.http.server.request-timeout = 20000000s
akka.http.server.idle-timeout = 20000000s

zhanglu

docker.io/library/nginx:1.25.2-alpine

zhanglu

zhanglu

nerdctl pull dockerhub.genostack.com:8090/library/csi-snapshotter:v6.3.2
nerdctl tag dockerhub.genostack.com:8090/library/csi-snapshotter:v6.3.2 registry.k8s.io/sig-storage/csi-snapshotter:v6.3.2

zhanglu

kubectl -n rook-ceph get secret rook-ceph-dashboard-password -o jsonpath="{['data']['password']}" | base64 --decode && echo

zhanglu

mount -t ceph 10.233.31.47:6789,10.233.36.144:6789,10.233.33.7:6789:/ /cephfs_data -o name=admin,secret=AQDw/2VjAJmTBxAAezkJzzXVJ4VCxiYtx49L+w==

zhanglu

docker pull dockerhub.genostack.com:8090/library/web_socket:latest

zhanglu

PostgreSQL Database directory appears to contain a database; Skipping initialization

2025-04-02 13:24:07.154 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
2025-04-02 13:24:07.154 UTC [1] LOG: listening on IPv6 address "::", port 5432
2025-04-02 13:24:07.157 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2025-04-02 13:24:07.411 UTC [26] LOG: database system was shut down at 2025-04-02 11:47:48 UTC
2025-04-02 13:24:07.411 UTC [26] LOG: invalid record length at 8/DBE56CD8: wanted 24, got 0
2025-04-02 13:24:07.411 UTC [26] LOG: invalid primary checkpoint record
2025-04-02 13:24:07.411 UTC [26] LOG: invalid resource manager ID in secondary checkpoint record
2025-04-02 13:24:07.411 UTC [26] PANIC: could not locate a valid checkpoint record
2025-04-02 13:24:08.323 UTC [1] LOG: startup process (PID 26) was terminated by signal 6: Aborted
2025-04-02 13:24:08.323 UTC [1] LOG: aborting startup due to startup process failure
2025-04-02 13:24:08.338 UTC [1] LOG: database system is shut down

zhanglu

PostgreSQL Database directory appears to contain a database; Skipping initialization

2025-04-07 04:24:18.838 UTC [1] FATAL: lock file "postmaster.pid" is empty
2025-04-07 04:24:18.838 UTC [1] HINT: Either another server is starting, or the lock file is the remnant of a previous server startup crash.

zhanglu

kill -9 lsof | grep delete | awk '{print $2}'

zhanglu

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDW8bqVdBe8okVnEaWxn/GCBn2ngPWezwomczuS+tcj/T3odSjNPe6NQVRb+KkumnwMk8CHLq41v1J+C9BZp2x/w90pCKtSjotl94CbNzqnEFJK+tfrKG38twMlPnCeHfsGXEeagbT0Td4nqhvvM2oz1gEzYwoNsY3VV/hufLbj0qNBeoKOqltTUtdRBaNyuxDxIzh+uHy5LrJAWjftGkjOHbJDXp+HEgXsUDyhr13lnAm2s3WPT5jLLiZvPiTBnrEscLZEoLkfelUJMwGOOIfTakd7illHLUtyRO9UcG3cYvUOTc2Pfp/SDLcuxrHbfpsRSz9ZD4ZzsX50SjTxQR3O44RGk2b4SQK7JNhEIDIzbhXbGTRPQVK4yI+ox6NKKYgj/GkxxMCErcAWa/o9PGLYEHJSuyK8vaE/Dh0rHYgUtq0EIolQ0DLPHa4vzlhkxK9j5qUtu6oZkYcqIhevz+blwnQ2h8DIWmK+9AL/QmqvbfWysMyPzXj7/475Bmt5aZ0= admin@example.com

zhanglu

/cephfs_data/genostack_v3/genostack_core/genostack_php_new/project_data/project_data/320/Mus_musculus.GRCm39.dna.primary_assembly.fa.gz

zhanglu

task-33146a1d
task-5dfacee4
task-7fed90b9
task-267924fd
task-02b7953c
task-56f81fc9
task-482572b9
task-9c85fa81
task-10a5f9fb
task-ce516c8b
task-d93f84e3
task-d6fac1ef
task-dde9bd01
task-e8bfb791

zhanglu

'task-33146a1d','task-5dfacee4','task-7fed90b9','task-267924fd','task-02b7953c','task-56f81fc9','task-482572b9','task-9c85fa81','task-10a5f9fb','task-ce516c8b','task-d93f84e3','task-d6fac1ef','task-dde9bd01','task-e8bfb791'

zhanglu

http://192.168.30.202:30280/d/StarsL_cn_K8S/k8s-dashboard-cn-20240513-starsl-cn?orgId=1&from=now-30m&to=now&timezone=browser&var-origin_prometheus=&var-Node=$__all&var-NameSpace=$__all&var-Container=$__all&var-Pod=$__all

zhanglu

zhanglu

调整内核脏页参数，触发自动写入
通过降低 “脏页阈值”，让内核更积极地将脏页写入磁盘：
bash

临时生效（重启后恢复默认）

脏页占总内存比例超过 5% 时，内核强制同步（默认 20%）

sysctl -w vm.dirty_ratio=5

后台进程在脏页占比超过 3% 时开始同步（默认 10%）

sysctl -w vm.dirty_background_ratio=3

脏页存活时间超过 10 秒后强制同步（默认 30 秒）

sysctl -w vm.dirty_expire_centisecs=1000 # 1000 厘秒 = 10 秒

zhanglu

ceph status
cluster:
id: 807d820b-5c5b-451c-9f52-41b93d5d905a
health: HEALTH_ERR
1 MDSs report oversized cache
1 MDSs report slow metadata IOs
2 MDSs behind on trimming
mon bh is low on available space
10 backfillfull osd(s)
1 OSDs or CRUSH {nodes, device-classes} have {NOUP,NODOWN,NOIN,NOOUT} flags set
full ratio(s) out of order
Low space hindering backfill (add storage if this doesn't resolve itself): 22 pgs backfill_toofull
Degraded data redundancy: 715/452014992 objects degraded (0.000%), 150 pgs degraded, 2 pgs undersized
206 pgs not deep-scrubbed in time
128 pgs not scrubbed in time
4 pool(s) backfillfull

zhanglu

[WARN] [09/06/2025 06:54:11.355] [cromwell-system-akka.actor.default-dispatcher-3] [cromwell-system/Pool(shared->http://tesk-api.default.svc.cluster.local:8080)] [1 (WaitingForResponseEntitySubscription)] Response entity was not subscribed after 1 second. Make sure to read the response entity body or call discardBytes() on it. POST /ga4gh/tes/v1/tasks/task-68f2dfc5:cancel Empty -> 400 Bad Request Chunked
[WARN] [09/06/2025 06:54:16.926] [cromwell-system-akka.actor.default-dispatcher-24] [cromwell-system/Pool(shared->http://tesk-api.default.svc.cluster.local:8080)] [0 (WaitingForResponseEntitySubscription)] Response entity was not subscribed after 1 second. Make sure to read the response entity body or call discardBytes() on it. POST /ga4gh/tes/v1/tasks/task-3a62617f:cancel Empty -> 400 Bad Request Chunked
[WARN] [09/06/2025 06:54:17.945] [cromwell-system-akka.actor.default-dispatcher-26] [cromwell-system/Pool(shared->http://tesk-api.default.svc.cluster.local:8080)] [1 (WaitingForResponseEntitySubscription)] Response entity was not subscribed after 1 second. Make sure to read the response entity body or call discardBytes() on it. POST /ga4gh/tes/v1/tasks/task-d38cd4a6:cancel Empty -> 400 Bad Request Chunked
[WARN] [09/06/2025 06:54:18.095] [cromwell-system-akka.actor.default-dispatcher-24] [cromwell-system/Pool(shared->http://tesk-api.default.svc.cluster.local:8080)] [3 (WaitingForResponseEntitySubscription)] Response entity was not subscribed after 1 second. Make sure to read the response entity body or call discardBytes() on it. POST /ga4gh/tes/v1/tasks/task-d38cd4a6:cancel Empty -> 400 Bad Request Chunked
[WARN] [09/06/2025 06:54:22.325] [cromwell-system-akka.actor.default-dispatcher-4] [cromwell-system/Pool(shared->http://tesk-api.default.svc.cluster.local:8080)] [2 (WaitingForResponseEntitySubscription)] Response entity was not subscribed after 1 second. Make sure to read the response entity body or call discardBytes() on it. POST /ga4gh/tes/v1/tasks/task-3299d1e5:cancel Empty -> 400 Bad Request Chunked
[WARN] [09/06/2025 06:54:25.545] [cromwell-system-akka.actor.default-dispatcher-2] [cromwell-system/Pool(shared->http://tesk-api.default.svc.cluster.local:8080)] [1 (WaitingForResponseEntitySubscription)] Response entity was not subscribed after 1 second. Make sure to read the response entity body or call discardBytes() on it. POST /ga4gh/tes/v1/tasks/task-68f2dfc5:cancel Empty -> 400 Bad Request Chunked
[WARN] [09/06/2025 06:54:25.575] [cromwell-system-akka.actor.default-dispatcher-3] [cromwell-system/Pool(shared->http://tesk-api.default.svc.cluster.local:8080)] [0 (WaitingForResponseEntitySubscription)] Response entity was not subscribed after 1 second. Make sure to read the response entity body or call discardBytes() on it. POST /ga4gh/tes/v1/tasks/task-3299d1e5:cancel Empty -> 400 Bad Request Chunked
[WARN] [09/06/2025 06:54:30.316] [cromwell-system-akka.actor.default-dispatcher-4] [cromwell-system/Pool(shared->http://tesk-api.default.svc.cluster.local:8080)] [3 (WaitingForResponseEntitySubscription)] Response entity was not subscribed after 1 second. Make sure to read the response entity body or call discardBytes() on it. POST /ga4gh/tes/v1/tasks/task-3a62617f:cancel Empty -> 400 Bad Request Chunked
[WARN] [09/06/2025 06:54:32.715] [cromwell-system-akka.actor.default-dispatcher-4] [cromwell-system/Pool(shared->http://tesk-api.default.svc.cluster.local:8080)] [0 (WaitingForResponseEntitySubscription)] Response entity was not subscribed after 1 second. Make sure to read the response entity body or call discardBytes() on it. POST /ga4gh/tes/v1/tasks/task-d38cd4a6:cancel Empty -> 400 Bad Request Chunked
[WARN] [09/06/2025 06:54:32.865] [cromwell-system-akka.actor.default-dispatcher-24] [cromwell-system/Pool(shared->http://tesk-api.default.svc.cluster.local:8080)] [1 (WaitingForResponseEntitySubscription)] Response entity was not subscribed after 1 second. Make sure to read the response entity body or call discardBytes() on it. POST /ga4gh/tes/v1/tasks/task-d38cd4a6:cancel Empty -> 400 Bad Request Chunked
[WARN] [09/06/2025 06:54:37.295] [cromwell-system-akka.actor.default-dispatcher-2] [cromwell-system/Pool(shared->http://tesk-api.default.svc.cluster.local:8080)] [3 (WaitingForResponseEntitySubscription)] Response entity was not subscribed after 1 second. Make sure to read the response entity body or call discardBytes() on it. POST /ga4gh/tes/v1/tasks/task-3299d1e5:cancel Empty -> 400 Bad Request Chunked

tmp

临时生效（重启后恢复默认）

脏页占总内存比例超过 5% 时，内核强制同步（默认 20%）

后台进程在脏页占比超过 3% 时开始同步（默认 10%）

脏页存活时间超过 10 秒后强制同步（默认 30 秒）