暗能星系

zhanglu

yum -y install docker-ce-19.03.6-3.el8 docker-ce-cli-19.03.6-3.el8 containerd.io

zhanglu

yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

yum install containerd.io-1.3.9-3.1.el8 docker-ce-cli-19.03.14-3.el8 docker-ce-19.03.14-3.el8

zhanglu

failed: [node5] (item={'name': 'containerd.io-1.3.9-3.1.el8', 'repo': 'docker-ce'}) => {"ansible_loop_var": "item", "attempts": 4, "changed": false, "item": {"name": "containerd.io-1.3.9-3.1.el8", "repo": "docker-ce"}, "msg": "Failed to download metadata for repo 'docker-ce': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried", "rc": 1, "results": []}

zhanglu

docker重启失败,重装docker和恢复服务操作：

yum remove docker*
yum install docker-ce
配置/etc/docker/daimon.json
systemctl start docker
systemctl enable docker
拉取镜像

docker pull dockerhub.genostack.com:8090/rook/cephcsi/cephcsi:v3.7.2
docker pull dockerhub.genostack.com:8090/rook/csi-node-driver-registrar:v2.5.1
docker pull dockerhub.genostack.com:8090/rook/csi-resizer:v1.6.0
docker pull dockerhub.genostack.com:8090/rook/csi-provisioner:v3.3.0
docker pull dockerhub.genostack.com:8090/rook/csi-snapshotter:v6.1.0
docker pull dockerhub.genostack.com:8090/rook/csi-attacher:v4.0.0
docker pull dockerhub.genostack.com:8090/rook/ceph/ceph:v17.2.5

docker tag dockerhub.genostack.com:8090/rook/cephcsi/cephcsi:v3.7.2 quay.io/cephcsi/cephcsi:v3.7.2
docker tag dockerhub.genostack.com:8090/rook/csi-node-driver-registrar:v2.5.1 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1
docker tag dockerhub.genostack.com:8090/rook/csi-resizer:v1.6.0 registry.k8s.io/sig-storage/csi-resizer:v1.6.0
docker tag dockerhub.genostack.com:8090/rook/csi-provisioner:v3.3.0 registry.k8s.io/sig-storage/csi-provisioner:v3.3.0
docker tag dockerhub.genostack.com:8090/rook/csi-snapshotter:v6.1.0 registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0
docker tag dockerhub.genostack.com:8090/rook/csi-attacher:v4.0.0 registry.k8s.io/sig-storage/csi-attacher:v4.0.0
docker tag dockerhub.genostack.com:8090/rook/ceph/ceph:v17.2.5 quay.io/ceph/ceph:v17.2.5

启动服务：
（1）检查swap内存，需要处于关闭状态，关闭命令：
swapoff -a
(2) 重启kubelet:
systemctl restart kubelet
(3) 等待集群重启，正常情况下执行docker ps,会有很多容器启动，
查看服务： kubectl get pod -n kube-system
存储服务： kubectl get pod -n rook-ceph

zhanglu

fatal: [node5]: FAILED! => {"changed": true, "cmd": ["timeout", "-k", "120s", "120s", "/usr/local/bin/kubeadm", "join", "--config", "/etc/kubernetes/kubeadm-client.conf", "--ignore-preflight-errors=all"], "delta": "0:01:00.404492", "end": "2098-01-03 15:47:51.813810", "msg": "non-zero return code", "rc": 1, "start": "2098-01-03 15:46:51.409318", "stderr": "\t[WARNING DirAvailable--etc-kubernetes-manifests]: /etc/kubernetes/manifests is not empty\nerror execution phase preflight: couldn't validate the identity of the API Server: Get "https://192.168.10.10:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s": x509: certificate has expired or is not yet valid: current time 2098-01-03T15:47:48+08:00 is after 2024-12-17T10:42:50Z\nTo see the stack trace of this error execute with --v=5 or higher", "stderr_lines": ["\t[WARNING DirAvailable--etc-kubernetes-manifests]: /etc/kubernetes/manifests is not empty", "error execution phase preflight: couldn't validate the identity of the API Server: Get "https://192.168.10.10:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s": x509: certificate has expired or is not yet valid: current time 2098-01-03T15:47:48+08:00 is after 2024-12-17T10:42:50Z", "To see the stack trace of this error execute with --v=5 or higher"], "stdout": "[preflight] Running pre-flight checks", "stdout_lines": ["[preflight] Running pre-flight checks"]

zhanglu

k: [node5] => {
"msg": "Joined with warnings\n['\t[WARNING DirAvailable--etc-kubernetes-manifests]: /etc/kubernetes/manifests is not empty', 'error execution phase preflight: couldn\'t validate the identity of the API Server: Get "https://192.168.10.10:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s": x509: certificate has expired or is not yet valid: current time 2098-01-03T15:47:48+08:00 is after 2024-12-17T10:42:50Z', 'To see the stack trace of this error execute with --v=5 or higher']\n"

zhanglu

systemctl start chronyd
systemctl enable chronyd

ansible-playbook -i inventory/mycluster/hosts.yaml remove-node.yml -b -v -e "node=node5"

zhanglu

admin_socket: exception getting command descriptions: [Errno 2] No such file or directory

zhanglu

Warning FailedMount 22m kubelet MountVolume.SetUp failed for volume "ceph-admin-secret" : failed to sync secret cache: timed out waiting for the condition
Warning FailedMount 22m kubelet MountVolume.SetUp failed for volume "rook-ceph-mgr-b-keyring" : failed to sync secret cache: timed out waiting for the condition
Warning FailedMount 22m kubelet MountVolume.SetUp failed for volume "rook-ceph-mgr-token-wnmk6" : failed to sync secret cache: timed out waiting for the condition
Normal Pulled 22m kubelet Container image "quay.io/ceph/ceph:v17.2.5" already present on machine
Normal Created 22m kubelet Created container chown-container-data-dir
Normal Started 22m kubelet Started container chown-container-data-dir
Normal Created 22m kubelet Created container watch-active
Normal Pulled 22m kubelet Container image "rook/ceph:v1.10.8" already present on machine
Normal Created 22m kubelet Created container log-collector
Normal Pulled 22m kubelet Container image "quay.io/ceph/ceph:v17.2.5" already present on machine
Normal Started 22m kubelet Started container log-collector
Normal Started 22m kubelet Started container watch-active
Normal Killing 20m kubelet Container mgr failed startup probe, will be restarted
Normal Pulled 20m (x2 over 22m) kubelet Container image "quay.io/ceph/ceph:v17.2.5" already present on machine
Normal Started 20m (x2 over 22m) kubelet Started container mgr
Normal Created 20m (x2 over 22m) kubelet Created container mgr
Warning Unhealthy 7m11s (x44 over 22m) kubelet Startup probe failed: ceph daemon health check failed with the following output:

admin_socket: exception getting command descriptions: [Errno 2] No such file or directory

zhanglu

apiVersion: v1
data:
keyring: ClttZ3IuYV0KCWtleSA9IEFRQ2ROTFZqN3FBREtCQUFKbHA4dFI4ZkY4YkhTdHBlZzd6eFVRPT0KCWNhcHMgbW9uID0gImFsbG93IHByb2ZpbGUgbWdyIgoJY2FwcyBtZHMgPSAiYWxsb3cgKiIKCWNhcHMgb3NkID0gImFsbG93ICoiCg==
kind: Secret
metadata:
creationTimestamp: "2023-01-04T08:11:09Z"
name: rook-ceph-mgr-a-keyring
namespace: rook-ceph
ownerReferences:

apiVersion: ceph.rook.io/v1
blockOwnerDeletion: true
controller: true
kind: CephCluster
name: rook-ceph
uid: 38bd931c-d505-4d51-aa8b-9414d834017c
resourceVersion: "5167"
selfLink: /api/v1/namespaces/rook-ceph/secrets/rook-ceph-mgr-a-keyring
uid: d42c8443-3979-40bf-8c5d-7f8053c6a34e
type: kubernetes.io/rook

zhanglu

Unable to attach or mount volumes: unmounted volumes=[coredns-token-dltf5 config-volume], unattached volumes=[coredns-token-dltf5 config-volume]: timed out waiting for the condition

zhanglu

networkPlugin cni failed to teardown po

zhanglu

docker images --format '{{.Repository}}: {{.Size}}' | sort -h

zhanglu

zhanglu

zhanglu

docker volume create portainer_data
docker run -d --name=portainer -p 9000:9000 -v /var/run/docker.sock:/var/run/docker.sock -v portainer_data:/data dockerhub.genostack.com:8090/library/portainer

zhanglu

[resolver] failed to query external DNS server

zhanglu

docker save goharbor/harbor-exporter:v2.7.4 -o exporter.tar.gz
docker save goharbor/chartmuseum-photon:v2.7.4 -o chart.tar.gz
docker save goharbor/redis-photon:v2.7.4 -o redis.tar.gz
docker save goharbor/trivy-adapter-photon:v2.7.4 -o trivy.tar.gz
docker save goharbor/notary-server-photon:v2.7.4 -o notary.tar.gz
docker save goharbor/notary-signer-photon:v2.7.4 -o nota2.tar.gz
docker save goharbor/harbor-registryctl:v2.7.4 -o harbor-regi.tar.gz
docker save goharbor/registry-photon:v2.7.4 -o registry-photon.tar.gz
docker save goharbor/nginx-photon:v2.7.4 -o nginx-photon.tar.gz
docker save goharbor/harbor-log:v2.7.4 -o harbor-log.tar.gz.tar.gz
docker save goharbor/harbor-jobservice:v2.7.4 -o harbor-jobservice.tar.gz
docker save goharbor/harbor-core:v2.7.4 -o harbor-core.tar.gz
docker save goharbor/harbor-portal:v2.7.4 -o harbor-portal.tar.gz
docker save goharbor/harbor-db:v2.7.4 -o harbor-db.tar.gz
docker save goharbor/prepare:v2.7.4 -o prepare.tar.gz

zhanglu

docker save goharbor/harbor-exporter:v2.7.4 -o exporter.tar.gz
docker save goharbor/chartmuseum-photon:v2.7.4 -o chart.tar.gz
docker save goharbor/redis-photon:v2.7.4 -o redis.tar.gz
docker save goharbor/trivy-adapter-photon:v2.7.4 -o trivy.tar.gz
docker save goharbor/notary-server-photon:v2.7.4 -o notary.tar.gz
docker save goharbor/notary-signer-photon:v2.7.4 -o nota2.tar.gz
docker save goharbor/harbor-registryctl:v2.7.4 -o harbor-regi.tar.gz
docker save goharbor/registry-photon:v2.7.4 -o registry-photon.tar.gz
docker save goharbor/nginx-photon:v2.7.4 -o nginx-photon.tar.gz
docker save goharbor/harbor-log:v2.7.4 -o harbor-log.tar.gz.tar.gz
docker save goharbor/harbor-jobservice:v2.7.4 -o harbor-jobservice.tar.gz
docker save goharbor/harbor-core:v2.7.4 -o harbor-core.tar.gz
docker save goharbor/harbor-portal:v2.7.4 -o harbor-portal.tar.gz
docker save goharbor/harbor-db:v2.7.4 -o harbor-db.tar.gz
docker save goharbor/prepare:v2.7.4 -o prepare.tar.gz

zhanglu

-rw-------. 1 root root 231080448 6月 25 10:26 chart.tar.gz
-rw-------. 1 root root 99730944 6月 25 10:26 exporter.tar.gz
-rw-------. 1 root root 226446336 6月 25 10:26 harbor-core.tar.gz
-rw-------. 1 root root 179054592 6月 25 10:26 harbor-db.tar.gz
-rw-------. 1 root root 264910848 6月 25 10:26 harbor-jobservice.tar.gz
-rw-------. 1 root root 130537984 6月 25 10:26 harbor-log.tar.gz.tar.gz
-rw-------. 1 root root 128102400 6月 25 10:26 harbor-portal.tar.gz
-rw-------. 1 root root 142263296 6月 25 10:26 harbor-regi.tar.gz
-rw-------. 1 root root 119507456 6月 25 10:26 nginx-photon.tar.gz
-rw-------. 1 root root 111452160 6月 25 10:26 nota2.tar.gz
-rw-------. 1 root root 114264064 6月 25 10:26 notary.tar.gz
-rw-------. 1 root root 173484544 6月 25 10:18 prepare.tar
-rw-------. 1 root root 173484544 6月 25 10:26 prepare.tar.gz
-rw-------. 1 root root 122591232 6月 25 10:26 redis.tar.gz
-rw-------. 1 root root 79580672 6月 25 10:26 registry-photon.tar.gz
-rw-------. 1 root root 466467840 6月 25 10:26 trivy.tar.gz

tmp