暗能星系

zhanglu

https://repo.almalinux.org/almalinux/8/AppStream/x86_64/os/Packages/ipvsadm-1.31-1.el8.x86_64.rpm

zhanglu

libnetfilter_queue-1.0.4-3.el8.x86_64.rpm

zhanglu

https://repo.almalinux.org/almalinux/8/BaseOS/x86_64/os/Packages/libnetfilter_queue-1.0.4-3.el8.x86_64.rpm

zhanglu

libnetfilter_cttimeout-1.0.0-11.el8.x86_64.rpm

zhanglu

https://repo.almalinux.org/almalinux/8/BaseOS/x86_64/os/Packages/libnetfilter_cttimeout-1.0.0-11.el8.x86_64.rpm

zhanglu

socat-1.7.4.1-1.el8.x86_64.rpm

zhanglu

cd /etc/yum.repos.d/
创建新文件夹并将源文件备份为repo.bak
mkdir backup && mv repo backup/
下载国内yum源文件
curl -o /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-8.repo
更新下载yum源地址
sed -i -e"s|mirrors.cloud.aliyuncs.com|mirrors.aliyun.com|g " /etc/yum.repos.d/CentOS-
sed -i -e "s|releasever|releasever-stream|g" /etc/yum.repos.d/CentOS-*
生成缓存
yum clean all && yum makecache

zhanglu

rrors during downloading metadata for repository 'appstream':

Curl error (60): Peer certificate cannot be authenticated with given CA certificates for https://vault.centos.org/centos/8-stream/AppStream/x86_64/os/repodata/repomd.xml [SSL certificate problem: certificate has expired]
Error: Failed to download metadata for repo 'appstream': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried

zhanglu

name': 'containerd.io-1.3.9-3.1.el8', 'repo': 'docker-ce'}) => {"ansible_loop_var": "item", "attempts": 4, "changed": false, "item": {"name": "containerd.io-1.3.9-3.1.el8", "repo": "docker-ce"}, "msg": "Failed to download metadata for repo 'docker-ce': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried", "rc": 1, "results": []}

zhanglu

yum install -y docker-ce-19.03.15-3.el8

zhanglu

yum -y install docker-ce-19.03.6-3.el8 docker-ce-cli-19.03.6-3.el8 containerd.io

zhanglu

yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

yum install containerd.io-1.3.9-3.1.el8 docker-ce-cli-19.03.14-3.el8 docker-ce-19.03.14-3.el8

zhanglu

failed: [node5] (item={'name': 'containerd.io-1.3.9-3.1.el8', 'repo': 'docker-ce'}) => {"ansible_loop_var": "item", "attempts": 4, "changed": false, "item": {"name": "containerd.io-1.3.9-3.1.el8", "repo": "docker-ce"}, "msg": "Failed to download metadata for repo 'docker-ce': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried", "rc": 1, "results": []}

zhanglu

docker重启失败,重装docker和恢复服务操作：

yum remove docker*
yum install docker-ce
配置/etc/docker/daimon.json
systemctl start docker
systemctl enable docker
拉取镜像

docker pull dockerhub.genostack.com:8090/rook/cephcsi/cephcsi:v3.7.2
docker pull dockerhub.genostack.com:8090/rook/csi-node-driver-registrar:v2.5.1
docker pull dockerhub.genostack.com:8090/rook/csi-resizer:v1.6.0
docker pull dockerhub.genostack.com:8090/rook/csi-provisioner:v3.3.0
docker pull dockerhub.genostack.com:8090/rook/csi-snapshotter:v6.1.0
docker pull dockerhub.genostack.com:8090/rook/csi-attacher:v4.0.0
docker pull dockerhub.genostack.com:8090/rook/ceph/ceph:v17.2.5

docker tag dockerhub.genostack.com:8090/rook/cephcsi/cephcsi:v3.7.2 quay.io/cephcsi/cephcsi:v3.7.2
docker tag dockerhub.genostack.com:8090/rook/csi-node-driver-registrar:v2.5.1 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1
docker tag dockerhub.genostack.com:8090/rook/csi-resizer:v1.6.0 registry.k8s.io/sig-storage/csi-resizer:v1.6.0
docker tag dockerhub.genostack.com:8090/rook/csi-provisioner:v3.3.0 registry.k8s.io/sig-storage/csi-provisioner:v3.3.0
docker tag dockerhub.genostack.com:8090/rook/csi-snapshotter:v6.1.0 registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0
docker tag dockerhub.genostack.com:8090/rook/csi-attacher:v4.0.0 registry.k8s.io/sig-storage/csi-attacher:v4.0.0
docker tag dockerhub.genostack.com:8090/rook/ceph/ceph:v17.2.5 quay.io/ceph/ceph:v17.2.5

启动服务：
（1）检查swap内存，需要处于关闭状态，关闭命令：
swapoff -a
(2) 重启kubelet:
systemctl restart kubelet
(3) 等待集群重启，正常情况下执行docker ps,会有很多容器启动，
查看服务： kubectl get pod -n kube-system
存储服务： kubectl get pod -n rook-ceph

zhanglu

fatal: [node5]: FAILED! => {"changed": true, "cmd": ["timeout", "-k", "120s", "120s", "/usr/local/bin/kubeadm", "join", "--config", "/etc/kubernetes/kubeadm-client.conf", "--ignore-preflight-errors=all"], "delta": "0:01:00.404492", "end": "2098-01-03 15:47:51.813810", "msg": "non-zero return code", "rc": 1, "start": "2098-01-03 15:46:51.409318", "stderr": "\t[WARNING DirAvailable--etc-kubernetes-manifests]: /etc/kubernetes/manifests is not empty\nerror execution phase preflight: couldn't validate the identity of the API Server: Get "https://192.168.10.10:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s": x509: certificate has expired or is not yet valid: current time 2098-01-03T15:47:48+08:00 is after 2024-12-17T10:42:50Z\nTo see the stack trace of this error execute with --v=5 or higher", "stderr_lines": ["\t[WARNING DirAvailable--etc-kubernetes-manifests]: /etc/kubernetes/manifests is not empty", "error execution phase preflight: couldn't validate the identity of the API Server: Get "https://192.168.10.10:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s": x509: certificate has expired or is not yet valid: current time 2098-01-03T15:47:48+08:00 is after 2024-12-17T10:42:50Z", "To see the stack trace of this error execute with --v=5 or higher"], "stdout": "[preflight] Running pre-flight checks", "stdout_lines": ["[preflight] Running pre-flight checks"]

zhanglu

k: [node5] => {
"msg": "Joined with warnings\n['\t[WARNING DirAvailable--etc-kubernetes-manifests]: /etc/kubernetes/manifests is not empty', 'error execution phase preflight: couldn\'t validate the identity of the API Server: Get "https://192.168.10.10:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s": x509: certificate has expired or is not yet valid: current time 2098-01-03T15:47:48+08:00 is after 2024-12-17T10:42:50Z', 'To see the stack trace of this error execute with --v=5 or higher']\n"

zhanglu

systemctl start chronyd
systemctl enable chronyd

ansible-playbook -i inventory/mycluster/hosts.yaml remove-node.yml -b -v -e "node=node5"

zhanglu

admin_socket: exception getting command descriptions: [Errno 2] No such file or directory

zhanglu

Warning FailedMount 22m kubelet MountVolume.SetUp failed for volume "ceph-admin-secret" : failed to sync secret cache: timed out waiting for the condition
Warning FailedMount 22m kubelet MountVolume.SetUp failed for volume "rook-ceph-mgr-b-keyring" : failed to sync secret cache: timed out waiting for the condition
Warning FailedMount 22m kubelet MountVolume.SetUp failed for volume "rook-ceph-mgr-token-wnmk6" : failed to sync secret cache: timed out waiting for the condition
Normal Pulled 22m kubelet Container image "quay.io/ceph/ceph:v17.2.5" already present on machine
Normal Created 22m kubelet Created container chown-container-data-dir
Normal Started 22m kubelet Started container chown-container-data-dir
Normal Created 22m kubelet Created container watch-active
Normal Pulled 22m kubelet Container image "rook/ceph:v1.10.8" already present on machine
Normal Created 22m kubelet Created container log-collector
Normal Pulled 22m kubelet Container image "quay.io/ceph/ceph:v17.2.5" already present on machine
Normal Started 22m kubelet Started container log-collector
Normal Started 22m kubelet Started container watch-active
Normal Killing 20m kubelet Container mgr failed startup probe, will be restarted
Normal Pulled 20m (x2 over 22m) kubelet Container image "quay.io/ceph/ceph:v17.2.5" already present on machine
Normal Started 20m (x2 over 22m) kubelet Started container mgr
Normal Created 20m (x2 over 22m) kubelet Created container mgr
Warning Unhealthy 7m11s (x44 over 22m) kubelet Startup probe failed: ceph daemon health check failed with the following output:

admin_socket: exception getting command descriptions: [Errno 2] No such file or directory

zhanglu

apiVersion: v1
data:
keyring: ClttZ3IuYV0KCWtleSA9IEFRQ2ROTFZqN3FBREtCQUFKbHA4dFI4ZkY4YkhTdHBlZzd6eFVRPT0KCWNhcHMgbW9uID0gImFsbG93IHByb2ZpbGUgbWdyIgoJY2FwcyBtZHMgPSAiYWxsb3cgKiIKCWNhcHMgb3NkID0gImFsbG93ICoiCg==
kind: Secret
metadata:
creationTimestamp: "2023-01-04T08:11:09Z"
name: rook-ceph-mgr-a-keyring
namespace: rook-ceph
ownerReferences:

apiVersion: ceph.rook.io/v1
blockOwnerDeletion: true
controller: true
kind: CephCluster
name: rook-ceph
uid: 38bd931c-d505-4d51-aa8b-9414d834017c
resourceVersion: "5167"
selfLink: /api/v1/namespaces/rook-ceph/secrets/rook-ceph-mgr-a-keyring
uid: d42c8443-3979-40bf-8c5d-7f8053c6a34e
type: kubernetes.io/rook

tmp