networkPolicy:
  enabled: true

禁止pod 和pod相互访问
在需要pod之间相互通讯是需要设置成

networkPolicy:
  enabled: false

原理介绍：
该配置是k8s 中的

(base) root@node1:/opt/app/genostack_v3_service/jupyter# kubectl get netpol -n jhub
NAME         POD-SELECTOR                                              AGE
hub          app=jupyterhub,component=hub,release=jhub                 19m
proxy        app=jupyterhub,component=proxy,release=jhub               19m
singleuser   app=jupyterhub,component=singleuser-server,release=jhub   19m

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: restrict-pod-access
  namespace: default
spec:
  podSelector:
    matchLabels:
      app: restricted-app
  policyTypes:
    - Egress
  egress:
    - to: []
      ports: []

通过“开始 > 运行... > regedit”打开注册表编辑器。
"HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\W32Time\TimeProviders\NtpServer".
设置 "Enable" 为1
"HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\W32Time\Config".
AnnounceFlags 为5 可以直接搜索一下这键 不同的操作系统版本可能路径有差别。
其他Windows time服务并设置为自动
2.Ubuntu是用timedatectl作为时钟同步服务的
/etc/systemd/timesyncd.conf
[Time]
NTP=192.168.10.11 Windows的IP
RootDistanceMaxSec=50
Systemctl restart system-timesyncd
3.OSD自动会恢复为Running状态

PostgreSQL Database directory appears to contain a database; Skipping initialization
2025-04-02 13:24:07.154 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
2025-04-02 13:24:07.154 UTC [1] LOG: listening on IPv6 address "::", port 5432
2025-04-02 13:24:07.157 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2025-04-02 13:24:07.411 UTC [26] LOG: database system was shut down at 2025-04-02 11:47:48 UTC
2025-04-02 13:24:07.411 UTC [26] LOG: invalid record length at 8/DBE56CD8: wanted 24, got 0
2025-04-02 13:24:07.411 UTC [26] LOG: invalid primary checkpoint record
2025-04-02 13:24:07.411 UTC [26] LOG: invalid resource manager ID in secondary checkpoint record
2025-04-02 13:24:07.411 UTC [26] PANIC: could not locate a valid checkpoint record
2025-04-02 13:24:08.323 UTC [1] LOG: startup process (PID 26) was terminated by signal 6: Aborted
2025-04-02 13:24:08.323 UTC [1] LOG: aborting startup due to startup process failure
2025-04-02 13:24:08.338 UTC [1] LOG: database system is shut down

解决措施：
https://stackoverflow.com/questions/71258094/invalid-resource-manager-id-in-primary-checkpoint-record

pg_resetwal /var/lib/postgresql/data/pgdata

目前代码backend中有个congfig.py文件，打开增加一行代码

WEBUI_NAME = _get_env_by_type("WEBUI_NAME", dft="灵岸")

然后在.env中才能生效

2、配置语言

目前docker有映射一个dada目录，在目录中找到config.json文件，修改如下 "default_locale": "zh-CN",：

{
    "version": 0,
    "ui": {
        "default_locale": "zh-CN",
        "prompt_suggestions": [
            {
                "title": [
                    "Help me study",
                    "vocabulary for a college entrance exam"
                ],
                "content": "Help me study vocabulary: write a sentence for me to fill in the blank, and I'll try to pick the correct option."
            }
        ]
    }
}

最后重启灵岸docker服务

1、报错
docker内部ng

2025/03/12 18:43:34 [warn] 29#29: *218120 an upstream response is buffered to a temporary file /tmp/fastcgi_temp/0000006707 while reading upstream, client: 10.233.90.0, server: _, request: "GET /api/dow_report/c656a052-9ea1-8de2-efd0-11c7a051eb6c?type=raw HTTP/1.0", upstream: "fastcgi://127.0.0.1:8991", host: "omic.sanshugroup.com"
10.233.90.0 - - [12/Mar/2025:18:47:51 +0800] "GET /api/dow_report/c656a052-9ea1-8de2-efd0-11c7a051eb6c?type=raw HTTP/1.0" 200 52940341857 "-" "Wget/1.19.4 (linux-gnu)" "103.114.101.5" 265.672 265.672 . -

服务器NG

2025/03/12 18:43:20 [error] 2052141#0: *166617 readv() failed (104: Connection reset by peer) while reading upstream, client: 103.114.101.5, server: omic.sanshugroup.com, request: "GET /api/dow_report/c656a052-9ea1-8de2-efd0-11c7a051eb6c?type=raw HTTP/1.1", upstream: "http://192.168.30.202:30000/api/dow_report/c656a052-9ea1-8de2-efd0-11c7a051eb6c?type=raw", host: "omic.sanshugroup.com"
2025/03/12 18:53:32 [error] 2052141#0: *166073 readv() failed (104: Connection reset by peer) while reading upstream, client: 113.132.179.7, server: omic.sanshugroup.com, request: "GET /api/dow_report/c656a052-9ea1-8de2-efd0-11c7a051eb6c?type=raw HTTP/1.1", upstream: "http://192.168.30.202:30000/api/dow_report/c656a052-9ea1-8de2-efd0-11c7a051eb6c?type=raw", host: "omic.sanshugroup.com", referrer: "https://omic.sanshugroup.com/"

2、问题排查
根据上放问题是临时文件的缓冲区不够
修改NGINX配置

	proxy_buffer_size   32m;
	proxy_buffers       12 16m;
	proxy_busy_buffers_size   64m;
        fastcgi_connect_timeout 24h;
        fastcgi_send_timeout 24h;
        fastcgi_read_timeout 24h;

问题排查初期（第二天）：

1、每次下载执行到49.30G就无故断开，连续测试多次

secure_download?key=54d4e2deca0598059daa524255a9c685     [                                                                                                                  <=> ]  49.30G  4.57MB/s    in 3h 21m 

2、修改php配置
但是实际观察占用内存并不是很高

ini_set('memory_limit', '100G');

测试还是和第二天一样

问题排查初期（第三天）：

找到外部nginx配置，修改如下：

        client_header_buffer_size 20m;
	large_client_header_buffers 4 18m;
	client_max_body_size 100g;
	proxy_max_temp_file_size 1001024m;

晚上同时执行两个下载

浏览器和命令行，两个下载均已成功

总结：

问题报错与断开原因有两个
1、缓冲区的配置不够，配置如下

	proxy_buffer_size   32m;
	proxy_buffers       12 16m;
	proxy_busy_buffers_size   64m;

2、固定下载到49.30G断开，配置如下

	client_max_body_size 100g;
	proxy_max_temp_file_size 1001024m;

吉凯集群(node2,node3, node4)中，node2的磁盘sdb松动，导致osd一直无法正常启动， 重新插拔重启

    resources:
      limits:
        cpu: 100m

参考： https://github.com/DaoCloud/public-image-mirror

删除node3, sdc

删除sdc的osd, 状态才会更新
kubectl delete pod rook-ceph-osd-4-64b4f4f9b4-lpgts -n rook-ceph

将osd4移除

ceph osd out osd.4
ceph osd purge 4 --yes-i-really-mean-it
ceph osd crush remove osd.4
ceph auth rm osd.4
ceph osd rm osd.4

kubectl delete deploy rook-ceph-osd-4 -n rook-ceph

1、进入rook ceph toolbox工具pod

kubectl exec -it rook-ceph-tools-68754fc9dd-sj9h6 bash -n rook-ceph

2、命名用ceph命令查询并删除osd

#查询状态，找到要移除的osd id

ceph osd status
#标记移除的osd
ceph osd out osd.1
ceph osd purge 1 --yes-i-really-mean-it
ceph osd crush remove osd.1
ceph auth rm osd.1
ceph osd rm osd.1

3、删除相关osd节点的deployment

kubectl delete deploy rook-ceph-osd-1 -n rook-ceph

4、登录要删除osd所在的服务器，格式化osd硬盘

#检查硬盘路径
fdisk -l
#删除硬盘分区信息
DISK="/dev/sdb"
sgdisk --zap-all $DISK
#清理硬盘数据（hdd硬盘使用dd，ssd硬盘使用blkdiscard，二选一）
dd if=/dev/zero of="$DISK" bs=1M count=100 oflag=direct,dsync
blkdiscard $DISK
#删除原osd的lvm信息（如果单个节点有多个osd，那么就不能用*拼配模糊删除，而根据lsblk -f查询出明确的lv映射信息再具体删除，参照第5项操作）
ls /dev/mapper/ceph-* | xargs -I% -- dmsetup remove %
rm -rf /dev/ceph-*
#重启，sgdisk –zzap-all需要重启后才生效
reboot

报错错误：cannot open /dev/sdb: Device or resource busy

5、手动查看并删除原osd创建的lvm信息

（可选，根据第4步执行情况决定），否则格式化时会报 cannot open /dev/sdb: Device or resource busy 异常

#查看lvm设备信息

dmsetup ls;
#删除ceph osd lvm映射关系
dmsetup remove ceph--5a4cb4bb--70b3--40bd--9da7--09d4f264a513-osd-xxxxxxxxx
#移动lv
lvremove /dev/mapper/ceph--5a4cb4bb--70b3--40bd--9da7--09d4f264a513-osd—xxxxxxxxx
#删除相关文件
rm –rf /dev/ceph--5a4cb4bb--70b3--40bd--9da7--09d4f264a513

5、重启ceph operator调度，使检测到格式化后的osd硬盘，osd启动后ceph集群会自动平衡数据

注：如果新osd pod无法执行起来可以通过查询osd prepare日志找问题

kubectl -n rook-ceph logs rook-ceph-osd-prepare-node1-fvmrp provision

使用delete命令将死机的节点先删除 重启k8s

/etc/resolv.conf
nameserver 210.22.84.3

{
    "job": {
        "name": "slurm_test220240428_13_55_34",
        "current_working_directory": "/cephfs_data/slurm_hpc/genostack_v3/genostack_core/genostack_php/slurm_run/c1/09c/42a-c99c-37ed-0e3f-2b2c8ddf3860",
        "standard_output": "/cephfs_data/slurm_hpc/genostack_v3/genostack_core/genostack_php/slurm_run/c1/09c/42a-c99c-37ed-0e3f-2b2c8ddf3860/output.out",
        "standard_error": "/cephfs_data/slurm_hpc/genostack_v3/genostack_core/genostack_php/slurm_run/c1/09c/42a-c99c-37ed-0e3f-2b2c8ddf3860/error.out",
        "environment": {
            "PATH": "/bin:/usr/bin/:/usr/local/bin/",
            "LD_LIBRARY_PATH": "/lib/:/lib64/:/usr/local/lib"
        },
        "partition": "debug",
        "cpus_per_task": 1,
        "memory_per_node":100,
        "required_nodes":["node3"]
    },
    "script": "#!/bin/bash \nsbatch test.sh"
}

原因分析：通过slurm rest api 提交任务，实际是用sbatch做了封装，在script中只需要写运行的job命令，不能用sbatch提交sh脚本。
这种错误做法等价于： 在test.sh 中写入 sbatch sleep 20s, 然后再用 sbatch test.sh

1. 下载simhei字体（https://github.com/zhangsheng377/stats_stock/blob/master/simhei.ttf)，存到 /usr/share/fonts 目录下，可新建文件夹。

2. 然后刷新字体：

sudo fc-cache -f -v

可从回显中检查有无载入刚才的字体文件。

3. 然后删除matplotlib字体缓存：

rm -rf /home/xxx/.cache/matplotlib

xxx替换为自己的账号

4. 然后在python中指定字体，即可正常显示中文：

import matplotlib.pyplot as plt
plt.rcParams['font.sans-serif'] = ['SimHei']