<![CDATA[tmp1]]>bash-4.4$ ceph osd status
ID HOST USED AVAIL WR OPS WR DATA RD OPS RD DATA STATE
0 node1 8912G 2036G 0 0 0 0 exists,up
1 node1 9.90T 809G 0 18 12 25.0k exists,up
2 node1 9360G 1588G 0 0 4 10.3k exists,up
3 node1 9229G 1719G 0 0 0 0 exists,up
4 node1 9980G 968G 0 0 0 0 exists,up
5 node1 9081G 1867G 0 0 0 0 exists,up
6 node1 9099G 1849G 0 0 0 0 exists,up
7 node2 9682G 1266G 0 0 0 0 exists,up
8 node2 10.0T 616G 0 0 5 12.2k exists,up
9 node2 10.1T 514G 0 0 0 0 exists,full,up
10 node2 10.0T 675G 0 0 0 0 exists,up
11 node2 10.0T 657G 0 0 0 0 exists,up
12 node2 9.87T 835G 0 0 0 0 exists,up
13 node2 10.1T 570G 0 0 0 0 exists,up
14 node3 10.0T 689G 0 0 0 0 exists,up
15 node3 10.4T 277G 0 0 0 0 exists,full,up
16 node3 9654G 1294G 0 0 0 0 exists,up
17 node3 9665G 1283G 0 0 0 0 exists,up
18 node3 9983G 965G 0 0 0 0 exists,up
19 node3 9638G 1310G 0 0 0 0 exists,up
20 node3 9346G 1602G 0 0 0 0 exists,up
21 node8 15.9T 2279G 9 66.0k 5 56.9k exists,up
22 node7 13.0T 1571G 0 0 4 10.6k exists,up
23 node7 12.4T 2121G 0 0 0 0 exists,up
24 node7 13.5T 990G 0 0 0 0 exists,up
25 node7 11.9T 2676G 0 0 0 0 exists,up
26 node7 13.4T 1171G 0 0 0 0 exists,up
27 node5 12.4T 2174G 0 0 0 0 exists,up
28 node5 12.3T 2296G 0 0 9 22.8k exists,up
29 node5 13.0T 1516G 0 0 0 0 exists,up
30 node5 12.0T 2584G 18 191k 10 218k exists,up
31 node5 13.1T 1417G 0 0 0 0 exists,up
32 node5 12.1T 2438G 0 0 0 0 exists,up
33 node5 12.7T 1869G 0 0 5 12.0k exists,up
34 node6 12.7T 1870G 0 0 0 0 exists,up
35 node6 12.1T 2491G 0 0 0 0 exists,up
36 node6 13.3T 1215G 0 0 0 0 exists,up
37 node6 13.3T 1264G 0 0 0 0 exists,up
38 node6 12.5T 2073G 0 0 0 0 exists,up
39 node6 12.9T 1625G 0 0 0 0 exists,up
40 node6 12.9T 1686G 0 5492 11 25.9k exists,up
41 node8 16.3T 1892G 0 0 4 10.7k exists,up
42 node8 15.8T 2365G 0 0 0 0 exists,up
43 node8 15.8T 2389G 0 0 0 0 exists,up
44 node8 15.2T 3028G 0 0 4 8535 exists,up
45 node8 15.2T 2973G 0 0 0 0 exists,up
46 node8 15.4T 2851G 0 0 5 13.5k exists,up
47 node8 15.3T 2861G 0 0 0 0 exists,up
48 node8 16.6T 1583G 0 0 0 0 exists,up
49 node8 15.4T 2844G 0 0 0 0 exists,up
50 node8 15.5T 2694G 0 0 0 0 exists,up
bash-4.4$ ceph status
cluster:
id: 807d820b-5c5b-451c-9f52-41b93d5d905a
health: HEALTH_ERR
1 MDSs report slow metadata IOs
2 MDSs behind on trimming
2 full osd(s)
Low space hindering backfill (add storage if this doesn't resolve itself): 19 pgs backfill_toofull
Degraded data redundancy: 2/601565166 objects degraded (0.000%), 2 pgs degraded
184 pgs not deep-scrubbed in time
124 pgs not scrubbed in time
Full OSDs blocking recovery: 2 pgs recovery_toofull
4 pool(s) full

services:
mon: 3 daemons, quorum bm,bo,bp (age 3h)
mgr: a(active, since 3h), standbys: b
mds: 1/1 daemons up, 1 hot standby
osd: 51 osds: 51 up (since 3h), 51 in (since 5h); 54 remapped pgs

data:
volumes: 1/1 healthy
pools: 4 pools, 2097 pgs
objects: 279.97M objects, 306 TiB
usage: 617 TiB used, 84 TiB / 701 TiB avail
pgs: 2/601565166 objects degraded (0.000%)
4525904/601565166 objects misplaced (0.752%)
2032 active+clean
35 active+remapped+backfilling
19 active+remapped+backfill_toofull
8 active+clean+scrubbing+deep
2 active+recovery_toofull+degraded
1 active+clean+scrubbing

io:
client: 500 KiB/s rd, 213 KiB/s wr, 237 op/s rd, 21 op/s wr
recovery: 33 MiB/s, 26 objects/s

]]>http://an.forum.genostack.com/topic/1130/tmp1RSS for NodeSat, 13 Jun 2026 09:22:54 GMTMon, 23 Feb 2026 10:28:36 GMT60<![CDATA[Reply to tmp1 on Tue, 21 Apr 2026 10:07:49 GMT]]>akka.http.host-connection-pool.max-open-requests = 512
akka.http.host-connection-pool.max-connections = 512

]]>http://an.forum.genostack.com/post/2920http://an.forum.genostack.com/post/2920Tue, 21 Apr 2026 10:07:49 GMT<![CDATA[Reply to tmp1 on Mon, 20 Apr 2026 05:50:41 GMT]]>select count(1) from "CALL_CACHING_AGGREGATION_ENTRY" ;
select count(1) from "CALL_CACHING_DETRITUS_ENTRY" ;
select count(1) from "CALL_CACHING_ENTRY" ;
select count(1) from "CALL_CACHING_ENTRY_CALL_CACHING_ENTRY_ID_seq" ;
select count(1) from "CALL_CACHING_HASH_ENTRY" ;
select count(1) from "CALL_CACHING_SIMPLETON_ENTRY" ;
select count(1) from "CUSTOM_LABEL_ENTRY" ;
select count(1) from "DOCKER_HASH_STORE_ENTRY" ;
select count(1) from "GROUP_METRICS_ENTRY" ;
select count(1) from "JOB_KEY_VALUE_ENTRY" ;
select count(1) from "JOB_STORE_ENTRY" ;
select count(1) from "JOB_STORE_SIMPLETON_ENTRY" ;
select count(1) from "METADATA_ENTRY" ;
select count(1) from "SUB_WORKFLOW_STORE_ENTRY" ;
select count(1) from "SUMMARY_QUEUE_ENTRY" ;
select count(1) from "SUMMARY_STATUS_ENTRY" ;
select count(1) from "WORKFLOW_METADATA_SUMMARY_ENTRY" ;
select count(1) from "WORKFLOW_STORE_ENTRY" ;
select count(1) from "databasechangelog" ;
select count(1) from "databasechangeloglock" ;
select count(1) from "sqlmetadatadatabasechangelog" ;
select count(1) from "sqlmetadatadatabasechangeloglock" ;

]]>http://an.forum.genostack.com/post/2919http://an.forum.genostack.com/post/2919Mon, 20 Apr 2026 05:50:41 GMT<![CDATA[Reply to tmp1 on Mon, 20 Apr 2026 05:49:52 GMT]]>select count() from "CALL_CACHING_AGGREGATION_ENTRY" ;
select count() from "CALL_CACHING_DETRITUS_ENTRY" ;
select count() from "CALL_CACHING_ENTRY" ;
select count() from "CALL_CACHING_ENTRY_CALL_CACHING_ENTRY_ID_seq" ;
select count() from "CALL_CACHING_HASH_ENTRY" ;
select count() from "CALL_CACHING_SIMPLETON_ENTRY" ;
select count() from "CUSTOM_LABEL_ENTRY" ;
select count() from "DOCKER_HASH_STORE_ENTRY" ;
select count() from "GROUP_METRICS_ENTRY" ;
select count() from "JOB_KEY_VALUE_ENTRY" ;
select count() from "JOB_STORE_ENTRY" ;
select count() from "JOB_STORE_SIMPLETON_ENTRY" ;
select count() from "METADATA_ENTRY" ;
select count() from "SUB_WORKFLOW_STORE_ENTRY" ;
select count() from "SUMMARY_QUEUE_ENTRY" ;
select count() from "SUMMARY_STATUS_ENTRY" ;
select count() from "WORKFLOW_METADATA_SUMMARY_ENTRY" ;
select count() from "WORKFLOW_STORE_ENTRY" ;
select count() from "databasechangelog" ;
select count() from "databasechangeloglock" ;
select count() from "sqlmetadatadatabasechangelog" ;
select count() from "sqlmetadatadatabasechangeloglock" ;

]]>http://an.forum.genostack.com/post/2918http://an.forum.genostack.com/post/2918Mon, 20 Apr 2026 05:49:52 GMT<![CDATA[Reply to tmp1 on Mon, 20 Apr 2026 05:45:22 GMT]]>public | CALL_CACHING_AGGREGATION_ENTRY | table | postgres
public | CALL_CACHING_AGGREGATION_ENTR_CALL_CACHING_AGGREGATION_ENTR_seq | sequence | postgres
public | CALL_CACHING_DETRITUS_ENTRY | table | postgres
public | CALL_CACHING_DETRITUS_ENTRY_CALL_CACHING_DETRITUS_ENTRY_ID_seq | sequence | postgres
public | CALL_CACHING_ENTRY | table | postgres
public | CALL_CACHING_ENTRY_CALL_CACHING_ENTRY_ID_seq | sequence | postgres
public | CALL_CACHING_HASH_ENTRY | table | postgres
public | CALL_CACHING_HASH_ENTRY_CALL_CACHING_HASH_ENTRY_ID_seq | sequence | postgres
public | CALL_CACHING_SIMPLETON_ENTRY | table | postgres
public | CALL_CACHING_SIMPLETON_ENTRY_CALL_CACHING_SIMPLETON_ENTRY_I_seq | sequence | postgres
public | CUSTOM_LABEL_ENTRY | table | postgres
public | CUSTOM_LABEL_ENTRY_CUSTOM_LABEL_ENTRY_ID_seq | sequence | postgres
public | DOCKER_HASH_STORE_ENTRY | table | postgres
public | DOCKER_HASH_STORE_ENTRY_DOCKER_HASH_STORE_ENTRY_ID_seq | sequence | postgres
public | GROUP_METRICS_ENTRY | table | postgres
public | GROUP_METRICS_ENTRY_GROUP_METRICS_ENTRY_ID_seq | sequence | postgres
public | JOB_KEY_VALUE_ENTRY | table | postgres
public | JOB_KEY_VALUE_ENTRY_JOB_KEY_VALUE_ENTRY_ID_seq | sequence | postgres
public | JOB_STORE_ENTRY | table | postgres
public | JOB_STORE_ENTRY_JOB_STORE_ENTRY_ID_seq | sequence | postgres
public | JOB_STORE_SIMPLETON_ENTRY | table | postgres
public | JOB_STORE_SIMPLETON_ENTRY_JOB_STORE_SIMPLETON_ENTRY_ID_seq | sequence | postgres
public | METADATA_ENTRY | table | postgres
public | METADATA_ENTRY_METADATA_JOURNAL_ID_seq | sequence | postgres
public | SUB_WORKFLOW_STORE_ENTRY | table | postgres
public | SUB_WORKFLOW_STORE_ENTRY_SUB_WORKFLOW_STORE_ENTRY_ID_seq | sequence | postgres
public | SUMMARY_QUEUE_ENTRY | table | postgres
public | SUMMARY_STATUS_ENTRY | table | postgres
public | SUMMARY_STATUS_ENTRY_SUMMARY_STATUS_ENTRY_ID_seq | sequence | postgres
public | WORKFLOW_METADATA_SUMMARY_ENTRY | table | postgres
public | WORKFLOW_METADATA_SUMMARY_ENT_WORKFLOW_METADATA_SUMMARY_ENT_seq | sequence | postgres
public | WORKFLOW_STORE_ENTRY | table | postgres
public | WORKFLOW_STORE_ENTRY_WORKFLOW_STORE_ENTRY_ID_seq | sequence | postgres
public | databasechangelog | table | postgres
public | databasechangeloglock | table | postgres
public | sqlmetadatadatabasechangelog | table | postgres
public | sqlmetadatadatabasechangeloglock | table | postgres

]]>http://an.forum.genostack.com/post/2917http://an.forum.genostack.com/post/2917Mon, 20 Apr 2026 05:45:22 GMT<![CDATA[Reply to tmp1 on Mon, 20 Apr 2026 05:38:11 GMT]]>SELECT count(*) FROM "WORKFLOW_METADATA_SUMMARY_ENTRY"

]]>http://an.forum.genostack.com/post/2916http://an.forum.genostack.com/post/2916Mon, 20 Apr 2026 05:38:11 GMT<![CDATA[Reply to tmp1 on Fri, 10 Apr 2026 07:28:15 GMT]]>[INFO] [04/10/2026 05:37:55.134] [cromwell-system-akka.dispatchers.api-dispatcher-1095] [akka://cromwell-system/user/cromwell-service/WorkflowStoreActor/WorkflowStoreSubmitActor] Unspecified type (Unspecified version) workflow 050f21d7-5e2a-4e38-9c6d-142f276de0eb submitted
[INFO] [04/10/2026 05:39:08.039] [cromwell-system-akka.dispatchers.engine-dispatcher-5] [akka://cromwell-system/user/cromwell-service/WorkflowStoreActor/WorkflowStoreEngineActor] 1 new workflows fetched by cromid-508afbd: 050f21d7-5e2a-4e38-9c6d-142f276de0eb
[INFO] [04/10/2026 05:39:08.039] [cromwell-system-akka.dispatchers.engine-dispatcher-5] [akka://cromwell-system/user/cromwell-service/WorkflowManagerActor] WorkflowManagerActor: Starting workflow UUID(050f21d7-5e2a-4e38-9c6d-142f276de0eb)
[INFO] [04/10/2026 05:39:08.039] [cromwell-system-akka.dispatchers.engine-dispatcher-5] [akka://cromwell-system/user/cromwell-service/WorkflowManagerActor] WorkflowManagerActor: Successfully started WorkflowActor-050f21d7-5e2a-4e38-9c6d-142f276de0eb
[INFO] [04/10/2026 05:39:08.040] [cromwell-system-akka.dispatchers.engine-dispatcher-5] [akka://cromwell-system/user/cromwell-service/WorkflowManagerActor/WorkflowActor-050f21d7-5e2a-4e38-9c6d-142f276de0eb/MaterializeWorkflowDescriptorActor] MaterializeWorkflowDescriptorActor [UUID(050f21d7)]: Parsing workflow as WDL draft-2
[INFO] [04/10/2026 05:39:08.051] [cromwell-system-akka.dispatchers.engine-dispatcher-5] [akka://cromwell-system/user/cromwell-service/WorkflowManagerActor/WorkflowActor-050f21d7-5e2a-4e38-9c6d-142f276de0eb/MaterializeWorkflowDescriptorActor] MaterializeWorkflowDescriptorActor [UUID(050f21d7)]: Call-to-Backend assignments: rnaseq_eukaryonanalysis.quantificationsummarycorrelation -> TESK
[INFO] [04/10/2026 05:39:10.094] [cromwell-system-akka.dispatchers.engine-dispatcher-97] [akka://cromwell-system/user/cromwell-service/WorkflowManagerActor/WorkflowActor-050f21d7-5e2a-4e38-9c6d-142f276de0eb/WorkflowExecutionActor-050f21d7-5e2a-4e38-9c6d-142f276de0eb] WorkflowExecutionActor-050f21d7-5e2a-4e38-9c6d-142f276de0eb [UUID(050f21d7)]: Starting rnaseq_eukaryonanalysis.quantificationsummarycorrelation
[WARN] [04/10/2026 05:39:17.208] [cromwell-system-akka.dispatchers.engine-dispatcher-39] [akka://cromwell-system/user/cromwell-service/WorkflowManagerActor/WorkflowActor-050f21d7-5e2a-4e38-9c6d-142f276de0eb/WorkflowExecutionActor-050f21d7-5e2a-4e38-9c6d-142f276de0eb/050f21d7-5e2a-4e38-9c6d-142f276de0eb-EngineJobExecutionActor-rnaseq_eukaryonanalysis.quantificationsummarycorrelation:NA:1/BackendPreparationActor_for_050f21d7:rnaseq_eukaryonanalysis.quantificationsummarycorrelation:-1:1] BackendPreparationActor_for_050f21d7:rnaseq_eukaryonanalysis.quantificationsummarycorrelation:-1:1 [UUID(050f21d7)]: Docker lookup failed
[INFO] [04/10/2026 05:39:17.348] [cromwell-system-akka.dispatchers.backend-dispatcher-5522] [akka://cromwell-system/user/cromwell-service/WorkflowManagerActor/WorkflowActor-050f21d7-5e2a-4e38-9c6d-142f276de0eb/WorkflowExecutionActor-050f21d7-5e2a-4e38-9c6d-142f276de0eb/050f21d7-5e2a-4e38-9c6d-142f276de0eb-EngineJobExecutionActor-rnaseq_eukaryonanalysis.quantificationsummarycorrelation:NA:1/050f21d7-5e2a-4e38-9c6d-142f276de0eb-BackendJobExecutionActor-rnaseq_eukaryonanalysis.quantificationsummarycorrelation:NA:1/TesAsyncBackendJobExecutionActor] TesAsyncBackendJobExecutionActor [UUID(050f21d7)rnaseq_eukaryonanalysis.quantificationsummarycorrelation:NA:1]: conda run -n RNASeq --no-capture-output \ [INFO] [04/10/2026 05:39:17.348] [cromwell-system-akka.dispatchers.backend-dispatcher-5522] [akka://cromwell-system/user/cromwell-service/WorkflowManagerActor/WorkflowActor-050f21d7-5e2a-4e38-9c6d-142f276de0eb/WorkflowExecutionActor-050f21d7-5e2a-4e38-9c6d-142f276de0eb/050f21d7-5e2a-4e38-9c6d-142f276de0eb-EngineJobExecutionActor-rnaseq_eukaryonanalysis.quantificationsummarycorrelation:NA:1/050f21d7-5e2a-4e38-9c6d-142f276de0eb-BackendJobExecutionActor-rnaseq_eukaryonanalysis.quantificationsummarycorrelation:NA:1/TesAsyncBackendJobExecutionActor] TesAsyncBackendJobExecutionActor [UUID(050f21d7)rnaseq_eukaryonanalysis.quantificationsummarycorrelation:NA:1]: Calculated TES outputs (found 7): Output(Some(rc),Some(rnaseq_eukaryonanalysis.quantificationsummarycorrelation.rc),Some(/cephfs_data/genostack_v3/genostack_cromwell/cromwell-executions/rnaseq_eukaryonanalysis/050f21d7-5e2a-4e38-9c6d-142f276de0eb/call-quantificationsummarycorrelation/execution/rc),/cephfs_data/genostack_v3/genostack_cromwell/cromwell-executions/rnaseq_eukaryonanalysis/050f21d7-5e2a-4e38-9c6d-142f276de0eb/call-quantificationsummarycorrelation/execution/rc,Some(FILE)) Output(Some(stdout),Some(rnaseq_eukaryonanalysis.quantificationsummarycorrelation.stdout),Some(/cephfs_data/genostack_v3/genostack_cromwell/cromwell-executions/rnaseq_eukaryonanalysis/050f21d7-5e2a-4e38-9c6d-142f276de0eb/call-quantificationsummarycorrelation/execution/stdout),/cephfs_data/genostack_v3/genostack_cromwell/cromwell-executions/rnaseq_eukaryonanalysis/050f21d7-5e2a-4e38-9c6d-142f276de0eb/call-quantificationsummarycorrelation/execution/stdout,Some(FILE)) Output(Some(stderr),Some(rnaseq_eukaryonanalysis.quantificationsummarycorrelation.stderr),Some(/cephfs_data/genostack_v3/genostack_cromwell/cromwell-executions/rnaseq_eukaryonanalysis/050f21d7-5e2a-4e38-9c6d-142f276de0eb/call-quantificationsummarycorrelation/execution/stderr),/cephfs_data/genostack_v3/genostack_cromwell/cromwell-executions/rnaseq_eukaryonanalysis/050f21d7-5e2a-4e38-9c6d-142f276de0eb/call-quantificationsummarycorrelation/execution/stderr,Some(FILE)) Output(Some(commandScript),Some(rnaseq_eukaryonanalysis.quantificationsummarycorrelation.commandScript),Some(/cephfs_data/genostack_v3/genostack_cromwell/cromwell-executions/rnaseq_eukaryonanalysis/050f21d7-5e2a-4e38-9c6d-142f276de0eb/call-quantificationsummarycorrelation/execution/script),/cephfs_data/genostack_v3/genostack_cromwell/cromwell-executions/rnaseq_eukaryonanalysis/050f21d7-5e2a-4e38-9c6d-142f276de0eb/call-quantificationsummarycorrelation/execution/script,Some(FILE)) Output(Some(rnaseq_eukaryonanalysis.quantificationsummarycorrelation.output.0),Some(rnaseq_eukaryonanalysis.quantificationsummarycorrelation.output.0),Some(/cephfs_data/genostack_v3/genostack_cromwell/cromwell-executions/rnaseq_eukaryonanalysis/050f21d7-5e2a-4e38-9c6d-142f276de0eb/call-quantificationsummarycorrelation/execution/temp),/cephfs_data/genostack_v3/genostack_cromwell/cromwell-executions/rnaseq_eukaryonanalysis/050f21d7-5e2a-4e38-9c6d-142f276de0eb/call-quantificationsummarycorrelation/execution/temp,Some(FILE)) Output(Some(rnaseq_eukaryonanalysis.quantificationsummarycorrelation.output.1),Some(rnaseq_eukaryonanalysis.quantificationsummarycorrelation.output.1),Some(/cephfs_data/genostack_v3/genostack_cromwell/cromwell-executions/rnaseq_eukaryonanalysis/050f21d7-5e2a-4e38-9c6d-142f276de0eb/call-quantificationsummarycorrelation/execution/report),/cephfs_data/genostack_v3/genostack_cromwell/cromwell-executions/rnaseq_eukaryonanalysis/050f21d7-5e2a-4e38-9c6d-142f276de0eb/call-quantificationsummarycorrelation/execution/report,Some(FILE)) Output(Some(rnaseq_eukaryonanalysis.quantificationsummarycorrelation.output.2),Some(rnaseq_eukaryonanalysis.quantificationsummarycorrelation.output.2),Some(/cephfs_data/genostack_v3/genostack_cromwell/cromwell-executions/rnaseq_eukaryonanalysis/050f21d7-5e2a-4e38-9c6d-142f276de0eb/call-quantificationsummarycorrelation/execution/result),/cephfs_data/genostack_v3/genostack_cromwell/cromwell-executions/rnaseq_eukaryonanalysis/050f21d7-5e2a-4e38-9c6d-142f276de0eb/call-quantificationsummarycorrelation/execution/result,Some(FILE)) [INFO] [04/10/2026 05:39:17.348] [cromwell-system-akka.dispatchers.backend-dispatcher-5522] [akka://cromwell-system/user/cromwell-service/WorkflowManagerActor/WorkflowActor-050f21d7-5e2a-4e38-9c6d-142f276de0eb/WorkflowExecutionActor-050f21d7-5e2a-4e38-9c6d-142f276de0eb/050f21d7-5e2a-4e38-9c6d-142f276de0eb-EngineJobExecutionActor-rnaseq_eukaryonanalysis.quantificationsummarycorrelation:NA:1/050f21d7-5e2a-4e38-9c6d-142f276de0eb-BackendJobExecutionActor-rnaseq_eukaryonanalysis.quantificationsummarycorrelation:NA:1/TesAsyncBackendJobExecutionActor] TesAsyncBackendJobExecutionActor [UUID(050f21d7)rnaseq_eukaryonanalysis.quantificationsummarycorrelation:NA:1]: Calculated TES inputs (found 1): cromwell.backend.impl.tes.Input(Some(commandScript),Some(rnaseq_eukaryonanalysis.quantificationsummarycorrelation.commandScript),Some(/cephfs_data/genostack_v3/genostack_cromwell/cromwell-executions/rnaseq_eukaryonanalysis/050f21d7-5e2a-4e38-9c6d-142f276de0eb/call-quantificationsummarycorrelation/execution/script),/cephfs_data/genostack_v3/genostack_cromwell/cromwell-executions/rnaseq_eukaryonanalysis/050f21d7-5e2a-4e38-9c6d-142f276de0eb/call-quantificationsummarycorrelation/execution/script,Some(FILE),None) [INFO] [04/10/2026 05:39:17.348] [cromwell-system-akka.dispatchers.backend-dispatcher-5522] [akka://cromwell-system/user/cromwell-service/WorkflowManagerActor/WorkflowActor-050f21d7-5e2a-4e38-9c6d-142f276de0eb/WorkflowExecutionActor-050f21d7-5e2a-4e38-9c6d-142f276de0eb/050f21d7-5e2a-4e38-9c6d-142f276de0eb-EngineJobExecutionActor-rnaseq_eukaryonanalysis.quantificationsummarycorrelation:NA:1/050f21d7-5e2a-4e38-9c6d-142f276de0eb-BackendJobExecutionActor-rnaseq_eukaryonanalysis.quantificationsummarycorrelation:NA:1/TesAsyncBackendJobExecutionActor] TesAsyncBackendJobExecutionActor [UUID(050f21d7)rnaseq_eukaryonanalysis.quantificationsummarycorrelation:NA:1]: conda run -n RNASeq --no-capture-output
[INFO] [04/10/2026 05:39:22.027] [cromwell-system-akka.dispatchers.backend-dispatcher-5523] [akka://cromwell-system/user/cromwell-service/WorkflowManagerActor/WorkflowActor-050f21d7-5e2a-4e38-9c6d-142f276de0eb/WorkflowExecutionActor-050f21d7-5e2a-4e38-9c6d-142f276de0eb/050f21d7-5e2a-4e38-9c6d-142f276de0eb-EngineJobExecutionActor-rnaseq_eukaryonanalysis.quantificationsummarycorrelation:NA:1/050f21d7-5e2a-4e38-9c6d-142f276de0eb-BackendJobExecutionActor-rnaseq_eukaryonanalysis.quantificationsummarycorrelation:NA:1/TesAsyncBackendJobExecutionActor] TesAsyncBackendJobExecutionActor [UUID(050f21d7)rnaseq_eukaryonanalysis.quantificationsummarycorrelation:NA:1]: job id: task-9c1b11c6
[INFO] [04/10/2026 05:39:22.278] [cromwell-system-akka.dispatchers.backend-dispatcher-5529] [akka://cromwell-system/user/cromwell-service/WorkflowManagerActor/WorkflowActor-050f21d7-5e2a-4e38-9c6d-142f276de0eb/WorkflowExecutionActor-050f21d7-5e2a-4e38-9c6d-142f276de0eb/050f21d7-5e2a-4e38-9c6d-142f276de0eb-EngineJobExecutionActor-rnaseq_eukaryonanalysis.quantificationsummarycorrelation:NA:1/050f21d7-5e2a-4e38-9c6d-142f276de0eb-BackendJobExecutionActor-rnaseq_eukaryonanalysis.quantificationsummarycorrelation:NA:1/TesAsyncBackendJobExecutionActor] TesAsyncBackendJobExecutionActor [UUID(050f21d7)rnaseq_eukaryonanalysis.quantificationsummarycorrelation:NA:1]: Status change from - to Running
[INFO] [04/10/2026 05:52:07.204] [cromwell-system-akka.dispatchers.backend-dispatcher-5501] [akka://cromwell-system/user/cromwell-service/WorkflowManagerActor/WorkflowActor-050f21d7-5e2a-4e38-9c6d-142f276de0eb/WorkflowExecutionActor-050f21d7-5e2a-4e38-9c6d-142f276de0eb/050f21d7-5e2a-4e38-9c6d-142f276de0eb-EngineJobExecutionActor-rnaseq_eukaryonanalysis.quantificationsummarycorrelation:NA:1/050f21d7-5e2a-4e38-9c6d-142f276de0eb-BackendJobExecutionActor-rnaseq_eukaryonanalysis.quantificationsummarycorrelation:NA:1/TesAsyncBackendJobExecutionActor] TesAsyncBackendJobExecutionActor [UUID(050f21d7)rnaseq_eukaryonanalysis.quantificationsummarycorrelation:NA:1]: Job task-9c1b11c6 is complete
[INFO] [04/10/2026 05:52:07.204] [cromwell-system-akka.dispatchers.backend-dispatcher-5529] [akka://cromwell-system/user/cromwell-service/WorkflowManagerActor/WorkflowActor-050f21d7-5e2a-4e38-9c6d-142f276de0eb/WorkflowExecutionActor-050f21d7-5e2a-4e38-9c6d-142f276de0eb/050f21d7-5e2a-4e38-9c6d-142f276de0eb-EngineJobExecutionActor-rnaseq_eukaryonanalysis.quantificationsummarycorrelation:NA:1/050f21d7-5e2a-4e38-9c6d-142f276de0eb-BackendJobExecutionActor-rnaseq_eukaryonanalysis.quantificationsummarycorrelation:NA:1/TesAsyncBackendJobExecutionActor] TesAsyncBackendJobExecutionActor [UUID(050f21d7)rnaseq_eukaryonanalysis.quantificationsummarycorrelation:NA:1]: Status change from Running to Complete
[INFO] [04/10/2026 05:52:09.375] [cromwell-system-akka.dispatchers.engine-dispatcher-95] [akka://cromwell-system/user/cromwell-service/WorkflowManagerActor/WorkflowActor-050f21d7-5e2a-4e38-9c6d-142f276de0eb/WorkflowExecutionActor-050f21d7-5e2a-4e38-9c6d-142f276de0eb] WorkflowExecutionActor-050f21d7-5e2a-4e38-9c6d-142f276de0eb [UUID(050f21d7)]: Workflow rnaseq_eukaryonanalysis complete. Final Outputs:
"rnaseq_eukaryonanalysis.quantificationsummarycorrelation.outresultdatadir": "/cephfs_data/genostack_v3/genostack_cromwell/cromwell-executions/rnaseq_eukaryonanalysis/050f21d7-5e2a-4e38-9c6d-142f276de0eb/call-quantificationsummarycorrelation/execution/result",
"rnaseq_eukaryonanalysis.quantificationsummarycorrelation.outempdir": "/cephfs_data/genostack_v3/genostack_cromwell/cromwell-executions/rnaseq_eukaryonanalysis/050f21d7-5e2a-4e38-9c6d-142f276de0eb/call-quantificationsummarycorrelation/execution/temp",
"rnaseq_eukaryonanalysis.quantificationsummarycorrelation.outreportdir": "/cephfs_data/genostack_v3/genostack_cromwell/cromwell-executions/rnaseq_eukaryonanalysis/050f21d7-5e2a-4e38-9c6d-142f276de0eb/call-quantificationsummarycorrelation/execution/report"
[INFO] [04/10/2026 05:52:12.036] [cromwell-system-akka.dispatchers.engine-dispatcher-77] [akka://cromwell-system/user/cromwell-service/WorkflowManagerActor] WorkflowManagerActor: Workflow actor for 050f21d7-5e2a-4e38-9c6d-142f276de0eb completed with status 'Succeeded'. The workflow will be removed from the workflow store.

]]>http://an.forum.genostack.com/post/2915http://an.forum.genostack.com/post/2915Fri, 10 Apr 2026 07:28:15 GMT<![CDATA[Reply to tmp1 on Fri, 27 Mar 2026 02:24:19 GMT]]>[root@node1 cephfs_data]# kubectl get pod | grep 3624
task-36247356-ex-00-4jb48 0/1 Error 0 25h
task-36247356-ex-00-b6cf5 0/1 Error 0 25h
task-36247356-ex-00-b7ck2 0/1 Error 0 25h
task-36247356-ex-00-bkvkm 0/1 Error 0 25h
task-36247356-ex-00-ftlp8 0/1 Error 0 25h
task-36247356-ex-00-ht7cv 0/1 Error 0 25h
task-36247356-ex-00-jvkxm 0/1 Error 0 25h
task-36247356-vqwdt 0/1 Completed 0 25h

]]>http://an.forum.genostack.com/post/2913http://an.forum.genostack.com/post/2913Fri, 27 Mar 2026 02:24:19 GMT<![CDATA[Reply to tmp1 on Tue, 24 Mar 2026 01:27:15 GMT]]>cluster:
id: 807d820b-5c5b-451c-9f52-41b93d5d905a
health: HEALTH_WARN
1 MDSs report oversized cache
mon bq is low on available space
Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull
804 pgs not deep-scrubbed in time
818 pgs not scrubbed in time

services:
mon: 3 daemons, quorum bo,bp,bq (age 11d)
mgr: a(active, since 4w), standbys: b
mds: 1/1 daemons up, 1 hot standby
osd: 51 osds: 51 up (since 24h), 51 in (since 4w); 58 remapped pgs

data:
volumes: 1/1 healthy
pools: 4 pools, 2097 pgs
objects: 261.90M objects, 302 TiB
usage: 611 TiB used, 90 TiB / 701 TiB avail
pgs: 3836403/563068733 objects misplaced (0.681%)
2030 active+clean
56 active+remapped+backfilling
9 active+clean+scrubbing+deep
2 active+remapped+backfill_toofull

io:
client: 19 MiB/s rd, 17 MiB/s wr, 15 op/s rd, 1.01k op/s wr
recovery: 58 MiB/s, 42 objects/s

]]>http://an.forum.genostack.com/post/2912http://an.forum.genostack.com/post/2912Tue, 24 Mar 2026 01:27:15 GMT<![CDATA[Reply to tmp1 on Thu, 12 Mar 2026 01:50:00 GMT]]>a.txt

]]>http://an.forum.genostack.com/post/2911http://an.forum.genostack.com/post/2911Thu, 12 Mar 2026 01:50:00 GMT<![CDATA[Reply to tmp1 on Thu, 26 Feb 2026 02:15:30 GMT]]>e74b452e-db0c-4282-958c-928960298115

]]>http://an.forum.genostack.com/post/2909http://an.forum.genostack.com/post/2909Thu, 26 Feb 2026 02:15:30 GMT<![CDATA[Reply to tmp1 on Mon, 23 Feb 2026 11:06:19 GMT]]>bash-4.4$ ceph status
cluster:
id: 807d820b-5c5b-451c-9f52-41b93d5d905a
health: HEALTH_ERR
1 OSDs or CRUSH {nodes, device-classes} have {NOUP,NODOWN,NOIN,NOOUT} flags set
full ratio(s) out of order
Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull
186 pgs not deep-scrubbed in time
128 pgs not scrubbed in time

services:
mon: 3 daemons, quorum bm,bo,bp (age 12m)
mgr: a(active, since 12m), standbys: b
mds: 1/1 daemons up, 1 hot standby
osd: 51 osds: 51 up (since 10m), 51 in (since 6h); 54 remapped pgs

data:
volumes: 1/1 healthy
pools: 4 pools, 2097 pgs
objects: 279.91M objects, 306 TiB
usage: 617 TiB used, 84 TiB / 701 TiB avail
pgs: 4438013/601423453 objects misplaced (0.738%)
2031 active+clean
52 active+remapped+backfilling
12 active+clean+scrubbing+deep
2 active+remapped+backfill_toofull

io:
client: 185 KiB/s rd, 32 KiB/s wr, 63 op/s rd, 193 op/s wr
recovery: 69 MiB/s, 51 objects/s

]]>http://an.forum.genostack.com/post/2907http://an.forum.genostack.com/post/2907Mon, 23 Feb 2026 11:06:19 GMT