暗能星系
    • 登录
    • 搜索

    tmp

    张渌
    2
    262
    1755
    正在加载更多帖子
    • 从旧到新
    • 从新到旧
    • 最多赞同
    回复
    • 在新帖中回复
    登录后回复
    此主题已被删除。只有拥有主题管理权限的用户可以查看。
    • Z
      zhanglu 最后由 编辑

      18 node3 9771G 1177G 3 0 0 0 exists,full,up

      1 条回复 最后回复 回复 引用 0
      • Z
        zhanglu 最后由 编辑

        Network v274_harbor Created 0.0s
        ⠹ Container harbor-log Starting 0.6s
        ✔ Container harbor-db Created 0.2s
        ✔ Container registry Created 0.2s
        ✔ Container registryctl Created 0.2s
        ✔ Container harbor-portal Created 0.2s
        ✔ Container redis Created 0.2s
        ✔ Container harbor-core Created 0.1s
        ✔ Container nginx Created 0.1s
        ✔ Container harbor-jobservice Created 0.1s
        Error response from daemon: driver failed programming external connectivity on endpoint harbor-log (3fa8de62ae840ea23fe6dd59902fc1cd0fdf2ccd585a0c3a4ecf1cf515268535): Bind for 127.0.0.1:1514 failed: port is already allocated
        [root@node1 v2.7.4]# vim docker-compose.yml

        1 条回复 最后回复 回复 引用 0
        • Z
          zhanglu 最后由 编辑

          ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS
          0 hdd 10.69240 0.95000 11 TiB 7.6 TiB 7.6 TiB 6.9 GiB 20 GiB 3.1 TiB 71.36 0.84 54 up
          1 hdd 10.69240 0.95000 11 TiB 9.1 TiB 9.0 TiB 14 GiB 23 GiB 1.6 TiB 84.89 1.00 65 up
          2 hdd 10.69240 0.95000 11 TiB 8.4 TiB 8.4 TiB 6.9 GiB 20 GiB 2.3 TiB 78.40 0.93 62 up
          3 hdd 10.69240 0.95000 11 TiB 8.0 TiB 7.9 TiB 14 GiB 21 GiB 2.7 TiB 74.50 0.88 59 up
          4 hdd 10.69240 0.95000 11 TiB 9.1 TiB 9.1 TiB 6.9 GiB 22 GiB 1.6 TiB 85.08 1.01 66 up
          5 hdd 10.69240 0.95000 11 TiB 8.1 TiB 8.1 TiB 14 GiB 22 GiB 2.6 TiB 75.90 0.90 61 up
          6 hdd 10.69240 0.95000 11 TiB 8.2 TiB 8.2 TiB 6 KiB 17 GiB 2.5 TiB 76.97 0.91 60 up
          7 hdd 10.69240 0.95000 11 TiB 9.0 TiB 9.0 TiB 14 GiB 23 GiB 1.7 TiB 84.53 1.00 67 up
          8 hdd 10.69240 0.95000 11 TiB 9.7 TiB 9.7 TiB 7.3 GiB 23 GiB 1011 GiB 90.77 1.07 70 up
          9 hdd 10.69240 0.95000 11 TiB 9.0 TiB 8.9 TiB 6.8 GiB 21 GiB 1.7 TiB 83.72 0.99 65 up
          10 hdd 9.00000 0.95000 11 TiB 9.3 TiB 9.3 TiB 8.9 MiB 21 GiB 1.4 TiB 87.07 1.03 66 up
          11 hdd 10.69240 0.95000 11 TiB 9.3 TiB 9.3 TiB 1 KiB 21 GiB 1.4 TiB 87.25 1.03 67 up
          12 hdd 10.69240 0.95000 11 TiB 9.2 TiB 9.2 TiB 116 KiB 20 GiB 1.5 TiB 85.76 1.01 64 up
          13 hdd 10.69240 0.95000 11 TiB 9.1 TiB 9.0 TiB 7.2 GiB 22 GiB 1.6 TiB 84.66 1.00 65 up
          14 hdd 9.00000 0.95000 11 TiB 9.4 TiB 9.4 TiB 327 KiB 22 GiB 1.3 TiB 88.18 1.04 64 up
          15 hdd 10.69240 0.95000 11 TiB 9.5 TiB 9.5 TiB 14 GiB 25 GiB 1.2 TiB 88.77 1.05 69 up
          16 hdd 10.69240 0.95000 11 TiB 8.5 TiB 8.5 TiB 1 KiB 19 GiB 2.1 TiB 79.95 0.95 60 up
          17 hdd 10.69240 0.95000 11 TiB 9.1 TiB 9.1 TiB 4.1 MiB 21 GiB 1.6 TiB 85.42 1.01 66 up
          18 hdd 10.69240 0.95000 11 TiB 9.6 TiB 9.6 TiB 14 GiB 26 GiB 1.1 TiB 90.17 1.07 69 up
          19 hdd 10.69240 0.95000 11 TiB 8.5 TiB 8.5 TiB 21 GiB 26 GiB 2.1 TiB 79.96 0.95 64 up
          20 hdd 10.69240 0.95000 11 TiB 9.1 TiB 9.1 TiB 1 KiB 21 GiB 1.6 TiB 85.46 1.01 64 up
          27 hdd 14.55269 1.00000 15 TiB 12 TiB 12 TiB 735 KiB 27 GiB 2.8 TiB 80.92 0.96 83 up
          28 hdd 14.55269 1.00000 15 TiB 12 TiB 12 TiB 21 GiB 31 GiB 2.4 TiB 83.18 0.98 87 up
          29 hdd 14.55269 1.00000 15 TiB 13 TiB 13 TiB 7.1 GiB 31 GiB 1.2 TiB 91.87 1.09 95 up
          30 hdd 14.55269 1.00000 15 TiB 12 TiB 12 TiB 14 GiB 30 GiB 2.7 TiB 81.47 0.96 87 up
          31 hdd 14.55269 1.00000 15 TiB 13 TiB 13 TiB 1 KiB 28 GiB 1.8 TiB 87.89 1.04 95 up
          32 hdd 14.55269 1.00000 15 TiB 12 TiB 12 TiB 2.6 MiB 26 GiB 2.8 TiB 80.64 0.95 87 up
          33 hdd 14.55269 1.00000 15 TiB 12 TiB 12 TiB 14 GiB 30 GiB 2.4 TiB 83.45 0.99 91 up
          34 hdd 14.55269 1.00000 15 TiB 13 TiB 12 TiB 6.9 GiB 29 GiB 2.0 TiB 86.13 1.02 91 up
          35 hdd 14.55269 1.00000 15 TiB 12 TiB 12 TiB 14 GiB 31 GiB 2.2 TiB 84.90 1.00 90 up
          36 hdd 14.55269 1.00000 15 TiB 13 TiB 13 TiB 1 KiB 28 GiB 1.6 TiB 88.94 1.05 95 up
          37 hdd 14.55269 1.00000 15 TiB 13 TiB 13 TiB 767 KiB 28 GiB 1.9 TiB 86.81 1.03 91 up
          38 hdd 14.55269 1.00000 15 TiB 12 TiB 12 TiB 21 MiB 27 GiB 2.1 TiB 85.36 1.01 88 up
          39 hdd 14.55269 1.00000 15 TiB 13 TiB 13 TiB 467 KiB 28 GiB 1.6 TiB 89.31 1.06 91 up
          40 hdd 14.55269 1.00000 15 TiB 12 TiB 12 TiB 14 GiB 30 GiB 2.1 TiB 85.53 1.01 92 up
          22 hdd 14.55269 1.00000 15 TiB 13 TiB 13 TiB 7.2 GiB 29 GiB 1.9 TiB 86.71 1.03 90 up
          23 hdd 14.55269 1.00000 15 TiB 12 TiB 12 TiB 1 KiB 27 GiB 2.3 TiB 84.04 0.99 87 up
          24 hdd 14.55269 1.00000 15 TiB 13 TiB 13 TiB 6.9 GiB 30 GiB 1.2 TiB 92.03 1.09 95 up
          25 hdd 14.55269 1.00000 15 TiB 12 TiB 12 TiB 1.5 MiB 27 GiB 2.3 TiB 84.45 1.00 86 up
          26 hdd 14.55269 1.00000 15 TiB 13 TiB 13 TiB 6.9 GiB 30 GiB 1.6 TiB 88.82 1.05 90 up
          21 hdd 18.19040 1.00000 18 TiB 16 TiB 16 TiB 14 GiB 37 GiB 2.6 TiB 85.92 1.02 114 up
          41 hdd 18.19040 1.00000 18 TiB 16 TiB 16 TiB 14 GiB 38 GiB 2.3 TiB 87.56 1.04 113 up
          42 hdd 18.19040 1.00000 18 TiB 15 TiB 15 TiB 7.2 GiB 34 GiB 3.1 TiB 83.17 0.98 110 up
          43 hdd 18.19040 1.00000 18 TiB 16 TiB 15 TiB 1 KiB 33 GiB 2.7 TiB 85.23 1.01 113 up
          44 hdd 18.19040 1.00000 18 TiB 15 TiB 15 TiB 14 GiB 37 GiB 3.2 TiB 82.21 0.97 99 up
          45 hdd 18.19040 1.00000 18 TiB 15 TiB 15 TiB 1 KiB 33 GiB 3.2 TiB 82.50 0.98 105 up
          46 hdd 18.19040 1.00000 18 TiB 15 TiB 15 TiB 14 GiB 37 GiB 2.7 TiB 84.92 1.00 107 up
          47 hdd 18.19040 1.00000 18 TiB 15 TiB 15 TiB 6.8 GiB 35 GiB 3.5 TiB 80.89 0.96 103 up
          48 hdd 16.00000 1.00000 18 TiB 17 TiB 16 TiB 2.2 MiB 35 GiB 1.7 TiB 90.79 1.07 113 up
          49 hdd 18.19040 1.00000 18 TiB 15 TiB 15 TiB 7.1 GiB 33 GiB 3.5 TiB 80.81 0.96 105 up
          50 hdd 18.19040 1.00000 18 TiB 15 TiB 15 TiB 7 KiB 33 GiB 3.4 TiB 81.03 0.96 101 up
          TOTAL 701 TiB 593 TiB 591 TiB 336 GiB 1.4 TiB 108 TiB 84.55

          1 条回复 最后回复 回复 引用 0
          • Z
            zhanglu 最后由 编辑

            bash-4.4$ ceph -s
            cluster:
            id: 807d820b-5c5b-451c-9f52-41b93d5d905a
            health: HEALTH_WARN
            1 large omap objects
            mon bv is low on available space
            Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull
            Degraded data redundancy: 1166183/520639685 objects degraded (0.224%), 2 pgs degraded, 2 pgs undersized
            582 pgs not deep-scrubbed in time
            427 pgs not scrubbed in time
            1 mgr modules have recently crashed

            services:
            mon: 3 daemons, quorum bt,bu,bv (age 2h)
            mgr: a(active, since 44h), standbys: b
            mds: 3/3 daemons up, 3 hot standby
            osd: 51 osds: 51 up (since 118m), 51 in (since 43h); 147 remapped pgs

            data:
            volumes: 1/1 healthy
            pools: 4 pools, 2097 pgs
            objects: 244.76M objects, 290 TiB
            usage: 593 TiB used, 108 TiB / 701 TiB avail
            pgs: 1166183/520639685 objects degraded (0.224%)
            7755848/520639685 objects misplaced (1.490%)
            1950 active+clean
            142 active+remapped+backfilling
            3 active+remapped+backfill_toofull
            2 active+undersized+degraded+remapped+backfilling

            io:
            client: 5.7 MiB/s rd, 689 MiB/s wr, 290 op/s rd, 736 op/s wr
            recovery: 151 MiB/s, 0 keys/s, 104 objects/s

            1 条回复 最后回复 回复 引用 0
            • Z
              zhanglu 最后由 编辑

              稍微调低最满 OSD 的权重，让数据往外迁（例如从 0.95 调到 0.90）

              ceph osd reweight 24 0.90
              ceph osd reweight 29 0.90

              1 条回复 最后回复 回复 引用 0
              • Z
                zhanglu 最后由 编辑

                kubectl -n rook-ceph rollout restart deployment rook-ceph-operator

                1 条回复 最后回复 回复 引用 0
                • Z
                  zhanglu 最后由 编辑

                  ceph -s
                  cluster:
                  id: 807d820b-5c5b-451c-9f52-41b93d5d905a
                  health: HEALTH_ERR
                  1 large omap objects
                  mon bv is low on available space
                  full ratio(s) out of order
                  Degraded data redundancy: 1694907/494508308 objects degraded (0.343%), 1 pg degraded, 1 pg undersized
                  715 pgs not deep-scrubbed in time
                  621 pgs not scrubbed in time
                  1 mgr modules have recently crashed

                  services:
                  mon: 3 daemons, quorum bt,bu,bv (age 16h)
                  mgr: b(active, since 16h), standbys: a
                  mds: 3/3 daemons up, 3 hot standby
                  osd: 51 osds: 51 up (since 12h), 51 in (since 2d); 106 remapped pgs

                  data:
                  volumes: 1/1 healthy
                  pools: 4 pools, 2097 pgs
                  objects: 232.41M objects, 279 TiB
                  usage: 570 TiB used, 132 TiB / 701 TiB avail
                  pgs: 1694907/494508308 objects degraded (0.343%)
                  5201810/494508308 objects misplaced (1.052%)
                  1990 active+clean
                  105 active+remapped+backfilling
                  1 active+undersized+degraded+remapped+backfilling
                  1 active+clean+scrubbing+deep

                  io:
                  client: 73 MiB/s rd, 588 MiB/s wr, 159 op/s rd, 855 op/s wr
                  recovery: 97 MiB/s, 27 keys/s, 68 objects/s

                  1 条回复 最后回复 回复 引用 0
                  • Z
                    zhanglu 最后由 编辑

                    bash-4.4$ ceph status
                    cluster:
                    id: 807d820b-5c5b-451c-9f52-41b93d5d905a
                    health: HEALTH_WARN
                    1 large omap objects
                    Degraded data redundancy: 1683124/491472480 objects degraded (0.342%), 1 pg degraded, 1 pg undersized
                    729 pgs not deep-scrubbed in time
                    671 pgs not scrubbed in time
                    1 mgr modules have recently crashed

                    services:
                    mon: 3 daemons, quorum bt,bu,bv (age 19h)
                    mgr: b(active, since 19h), standbys: a
                    mds: 3/3 daemons up, 3 hot standby
                    osd: 51 osds: 51 up (since 15h), 51 in (since 2d); 91 remapped pgs

                    data:
                    volumes: 1/1 healthy
                    pools: 4 pools, 2097 pgs
                    objects: 231.00M objects, 278 TiB
                    usage: 567 TiB used, 135 TiB / 701 TiB avail
                    pgs: 1683124/491472480 objects degraded (0.342%)
                    4532435/491472480 objects misplaced (0.922%)
                    2004 active+clean
                    90 active+remapped+backfilling
                    2 active+clean+scrubbing+deep
                    1 active+undersized+degraded+remapped+backfilling

                    io:
                    client: 22 MiB/s rd, 5.7 MiB/s wr, 137 op/s rd, 1.20k op/s wr
                    recovery: 84 MiB/s, 10 keys/s, 60 objects/s

                    1 条回复 最后回复 回复 引用 0
                    • Z
                      zhanglu 最后由 编辑

                      bash-4.4$ ceph osd blacklist ls
                      10.233.92.40:6801/1911285772 2026-05-16T06:42:45.415305+0000
                      10.233.95.126:6801/423702819 2026-05-16T06:39:29.555227+0000
                      10.233.92.40:6800/1911285772 2026-05-16T06:42:45.415305+0000
                      10.233.95.126:6800/423702819 2026-05-16T06:39:29.555227+0000
                      10.233.96.102:6800/263402740 2026-05-16T06:37:42.398974+0000
                      10.233.95.112:6800/1005951981 2026-05-15T11:36:21.160738+0000
                      10.233.95.0:0/1064179451 2026-05-15T11:36:14.378873+0000
                      10.233.95.0:6801/3909978274 2026-05-15T11:36:14.378873+0000
                      10.233.95.0:6800/3909978274 2026-05-15T11:36:14.378873+0000
                      10.233.95.112:6801/1005951981 2026-05-15T11:36:21.160738+0000
                      10.233.95.0:0/4149376748 2026-05-15T11:36:14.378873+0000
                      10.233.95.0:0/888298246 2026-05-15T11:36:14.378873+0000
                      10.233.92.42:6801/4254687 2026-05-15T11:35:40.127336+0000
                      10.233.92.208:0/3664263079 2026-05-15T11:20:14.004549+0000
                      10.233.108.135:6801/1731972526 2026-05-15T11:34:04.983882+0000
                      10.233.92.208:0/2326052718 2026-05-15T11:20:14.004549+0000
                      10.233.92.208:0/3801167330 2026-05-15T11:20:14.004549+0000
                      10.233.90.67:6801/835486982 2026-05-16T06:41:10.430971+0000
                      10.233.96.102:6801/263402740 2026-05-16T06:37:42.398974+0000
                      10.233.92.208:0/3997222985 2026-05-15T11:20:14.004549+0000
                      10.233.92.208:6801/3458710516 2026-05-15T11:20:14.004549+0000
                      10.233.90.67:6800/835486982 2026-05-16T06:41:10.430971+0000
                      10.233.69.0:0/3888805416 2026-05-15T07:38:43.071704+0000
                      10.233.95.0:0/2053524312 2026-05-15T11:36:14.378873+0000
                      10.233.92.208:0/3913625702 2026-05-15T11:20:14.004549+0000
                      10.233.70.84:6800/2865481930 2026-05-15T11:34:04.953630+0000
                      10.233.95.253:6800/1685858956 2026-05-15T10:24:56.891807+0000
                      10.233.70.84:6801/2865481930 2026-05-15T11:34:04.953630+0000
                      10.233.95.253:6801/1685858956 2026-05-15T10:24:56.891807+0000
                      10.233.108.135:6800/1731972526 2026-05-15T11:34:04.983882+0000
                      10.233.69.0:0/779221205 2026-05-15T07:38:43.071521+0000
                      10.233.92.208:6800/3458710516 2026-05-15T11:20:14.004549+0000
                      10.233.92.42:6800/4254687 2026-05-15T11:35:40.127336+0000
                      listed 33 entries
                      bash-4.4$
                      bash-4.4$ ceph osd blacklist rm 10.233.92.40:6801/1911285772
                      un-blocklisting 10.233.92.40:6801/1911285772

                      1 条回复 最后回复 回复 引用 0
                      • Z
                        zhanglu 最后由 编辑

                        ceph osd blacklist rm 10.233.92.40:6801/1911285772
                        ceph osd blacklist rm 10.233.95.126:6801/423702819
                        ceph osd blacklist rm 10.233.92.40:6800/1911285772
                        ceph osd blacklist rm 10.233.95.126:6800/423702819
                        ceph osd blacklist rm 10.233.96.102:6800/263402740
                        ceph osd blacklist rm 10.233.95.112:6800/1005951981
                        ceph osd blacklist rm 10.233.95.0:0/1064179451
                        ceph osd blacklist rm 10.233.95.0:6801/3909978274
                        ceph osd blacklist rm 10.233.95.0:6800/3909978274
                        ceph osd blacklist rm 10.233.95.112:6801/1005951981
                        ceph osd blacklist rm 10.233.95.0:0/4149376748
                        ceph osd blacklist rm 10.233.95.0:0/888298246
                        ceph osd blacklist rm 10.233.92.42:6801/4254687
                        ceph osd blacklist rm 10.233.92.208:0/3664263079
                        ceph osd blacklist rm 10.233.108.135:6801/1731972526
                        ceph osd blacklist rm 10.233.92.208:0/2326052718
                        ceph osd blacklist rm 10.233.92.208:0/3801167330
                        ceph osd blacklist rm 10.233.90.67:6801/835486982
                        ceph osd blacklist rm 10.233.96.102:6801/263402740
                        ceph osd blacklist rm 10.233.92.208:0/3997222985
                        ceph osd blacklist rm 10.233.92.208:6801/3458710516
                        ceph osd blacklist rm 10.233.90.67:6800/835486982
                        ceph osd blacklist rm 10.233.69.0:0/3888805416
                        ceph osd blacklist rm 10.233.95.0:0/2053524312
                        ceph osd blacklist rm 10.233.92.208:0/3913625702
                        ceph osd blacklist rm 10.233.70.84:6800/2865481930
                        ceph osd blacklist rm 10.233.95.253:6800/1685858956
                        ceph osd blacklist rm 10.233.70.84:6801/2865481930
                        ceph osd blacklist rm 10.233.95.253:6801/1685858956
                        ceph osd blacklist rm 10.233.108.135:6800/1731972526
                        ceph osd blacklist rm 10.233.69.0:0/779221205
                        ceph osd blacklist rm 10.233.92.208:6800/3458710516
                        ceph osd blacklist rm 10.233.92.42:6800/4254687

                        1 条回复 最后回复 回复 引用 0
                        • Z
                          zhanglu 最后由 编辑

                          2026-05-15 07:27:09.479984 I | op-osd: waiting... 5 of 6 OSD prepare jobs have finished processing and 49 of 51 OSDs have been updated
                          2026-05-15 07:27:10.683717 I | op-osd: OSD 18 is not ok-to-stop. will try updating it again later
                          2026-05-15 07:27:11.167655 I | clusterdisruption-controller: all "host" failure domains: [node1 node2 node3 node5 node6 node7 node8]. osd is down in failure domain: "". active node drains: false. pg health: "cluster is not fully clean. PGs: [{StateName:active+clean Count:2007} {StateName:active+remapped+backfilling Count:87} {StateName:active+clean+scrubbing+deep Count:2} {StateName:active+undersized+degraded+remapped+backfilling Count:1}]"
                          2026-05-15 07:27:11.901951 I | op-osd: OSD 46 is not ok-to-stop. will try updating it again later
                          2026-05-15 07:27:12.683286 I | clusterdisruption-controller: all "host" failure domains: [node1 node2 node3 node5 node6 node7 node8]. osd is down in failure domain: "". active node drains: false. pg health: "cluster is not fully clean. PGs: [{StateName:active+clean Count:2007} {StateName:active+remapped+backfilling Count:87} {StateName:active+clean+scrubbing+deep Count:2} {StateName:active+undersized+degraded+remapped+backfilling Count:1}]"
                          2026-05-15 07:27:13.194962 I | op-osd: OSD 18 is not ok-to-stop. will try updating it again later
                          2026-05-15 07:27:14.432436 I | op-osd: OSD 46 is not ok-to-stop. will try updating it again later
                          2026-05-15 07:27:15.627441 I | op-osd: OSD 18 is not ok-to-stop. will try updating it again later
                          2026-05-15 07:27:16.947802 I | op-osd: OSD 46 is not ok-to-stop. will try updating it again later
                          2026-05-15 07:27:18.279735 I | op-osd: OSD 18 is not ok-to-stop. will try updating it again later
                          2026-05-15 07:27:19.444455 I | op-osd: OSD 46 is not ok-to-stop. will try updating it again later
                          2026-05-15 07:27:20.563726 I | op-osd: OSD 18 is not ok-to-stop. will try updating it

                          1 条回复 最后回复 回复 引用 0
                          • First post
                            Last post
                          Powered by 暗能星系