climateprediction.net home page
Task 22437314

Task 22437314

Name oifs_43r3_bl_a2zq_2016092300_20_1017_12285705_0
Workunit 12285705
Created 7 Jun 2024, 16:27:09 UTC
Sent 7 Jun 2024, 16:31:19 UTC
Report deadline 6 Aug 2024, 16:31:19 UTC
Received 10 Jun 2024, 18:36:33 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 0 (0x00000000)
Computer ID 1533094
Run time 19 hours 49 min 33 sec
CPU time 19 hours 14 min 10 sec
Validate state Invalid
Credit 1,976.32
Device peak FLOPS 7.75 GFLOPS
Application version OpenIFS 43r3 Baroclinic Lifecycle v1.13
x86_64-pc-linux-gnu
Peak working set size 5,539.37 MB
Peak swap size 5,980.80 MB
Peak disk usage 1,362.35 MB
Stderr
<core_client_version>7.24.1</core_client_version>
<![CDATA[
<stderr_txt>
59.980
  07:38:47 STEP 1017 H= 254:15 +CPU= 60.707
  07:39:54 STEP 1018 H= 254:30 +CPU= 64.861
  07:40:59 STEP 1019 H= 254:45 +CPU= 62.388
  07:42:11 STEP 1020 H= 255:00 +CPU= 69.609
Moving to projects directory: /var/lib/boinc/slots/19/ICMGGh7zg+001020
Moving to projects directory: /var/lib/boinc/slots/19/ICMSHh7zg+001020
Moving to projects directory: /var/lib/boinc/slots/19/ICMUAh7zg+001020
  07:43:18 STEP 1021 H= 255:15 +CPU= 64.158
  07:44:18 STEP 1022 H= 255:30 +CPU= 58.231
  07:45:19 STEP 1023 H= 255:45 +CPU= 58.834
  07:46:17 STEP 1024 H= 256:00 +CPU= 57.002
  07:47:22 STEP 1025 H= 256:15 +CPU= 62.079
  07:48:20 STEP 1026 H= 256:30 +CPU= 56.619
  07:49:18 STEP 1027 H= 256:45 +CPU= 56.924
  07:50:19 STEP 1028 H= 257:00 +CPU= 58.713
  07:51:21 STEP 1029 H= 257:15 +CPU= 58.955
  07:52:23 STEP 1030 H= 257:30 +CPU= 60.562
  07:53:26 STEP 1031 H= 257:45 +CPU= 59.429
  07:54:40 STEP 1032 H= 258:00 +CPU= 71.147
Moving to projects directory: /var/lib/boinc/slots/19/ICMGGh7zg+001032
Moving to projects directory: /var/lib/boinc/slots/19/ICMSHh7zg+001032
Moving to projects directory: /var/lib/boinc/slots/19/ICMUAh7zg+001032
  07:55:47 STEP 1033 H= 258:15 +CPU= 64.430
  07:56:51 STEP 1034 H= 258:30 +CPU= 61.430
  07:57:54 STEP 1035 H= 258:45 +CPU= 60.926
  07:58:59 STEP 1036 H= 259:00 +CPU= 61.649
  08:00:01 STEP 1037 H= 259:15 +CPU= 60.099
  08:01:02 STEP 1038 H= 259:30 +CPU= 60.115
  08:02:02 STEP 1039 H= 259:45 +CPU= 58.093
  08:03:04 STEP 1040 H= 260:00 +CPU= 60.499
  08:04:07 STEP 1041 H= 260:15 +CPU= 61.642
  08:05:14 STEP 1042 H= 260:30 +CPU= 63.932
  08:06:19 STEP 1043 H= 260:45 +CPU= 62.728
  08:07:31 STEP 1044 H= 261:00 +CPU= 68.997
Moving to projects directory: /var/lib/boinc/slots/19/ICMGGh7zg+001044
Moving to projects directory: /var/lib/boinc/slots/19/ICMSHh7zg+001044
Moving to projects directory: /var/lib/boinc/slots/19/ICMUAh7zg+001044
  08:08:29 STEP 1045 H= 261:15 +CPU= 56.277
  08:09:32 STEP 1046 H= 261:30 +CPU= 60.945
  08:10:34 STEP 1047 H= 261:45 +CPU= 59.460
  08:11:39 STEP 1048 H= 262:00 +CPU= 62.543
  08:12:41 STEP 1049 H= 262:15 +CPU= 60.105
  08:13:40 STEP 1050 H= 262:30 +CPU= 57.046
  08:14:42 STEP 1051 H= 262:45 +CPU= 58.878
  08:15:45 STEP 1052 H= 263:00 +CPU= 61.273
  08:16:46 STEP 1053 H= 263:15 +CPU= 57.998
  08:17:51 STEP 1054 H= 263:30 +CPU= 61.590
  08:18:51 STEP 1055 H= 263:45 +CPU= 57.671
  08:20:18 STEP 1056 H= 264:00 +CPU= 83.234
Moving to projects directory: /var/lib/boinc/slots/19/ICMGGh7zg+001056
Moving to projects directory: /var/lib/boinc/slots/19/ICMSHh7zg+001056
Moving to projects directory: /var/lib/boinc/slots/19/ICMUAh7zg+001056
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMGGh7zg+000960
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMSHh7zg+000960
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMUAh7zg+000960
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMGGh7zg+000972
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMSHh7zg+000972
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMUAh7zg+000972
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMGGh7zg+000984
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMSHh7zg+000984
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMUAh7zg+000984
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMGGh7zg+000996
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMSHh7zg+000996
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMUAh7zg+000996
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMGGh7zg+001008
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMSHh7zg+001008
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMUAh7zg+001008
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMGGh7zg+001020
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMSHh7zg+001020
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMUAh7zg+001020
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMGGh7zg+001032
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMSHh7zg+001032
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMUAh7zg+001032
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMGGh7zg+001044
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMSHh7zg+001044
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMUAh7zg+001044
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_a2zq_2016092300_20_1017_12285705_0_r295099571_10.zip
Uploading the intermediate file: upload_file_10.zip
  08:21:19 STEP 1057 H= 264:15 +CPU= 58.555
  08:22:22 STEP 1058 H= 264:30 +CPU= 60.012
  08:23:23 STEP 1059 H= 264:45 +CPU= 59.855
  08:24:27 STEP 1060 H= 265:00 +CPU= 60.694
  08:25:30 STEP 1061 H= 265:15 +CPU= 60.195
  08:26:29 STEP 1062 H= 265:30 +CPU= 56.822
  08:27:28 STEP 1063 H= 265:45 +CPU= 57.229
  08:28:30 STEP 1064 H= 266:00 +CPU= 60.424
  08:29:34 STEP 1065 H= 266:15 +CPU= 61.826
  08:30:39 STEP 1066 H= 266:30 +CPU= 62.425
  08:31:45 STEP 1067 H= 266:45 +CPU= 62.062
  08:33:05 STEP 1068 H= 267:00 +CPU= 76.698
Moving to projects directory: /var/lib/boinc/slots/19/ICMGGh7zg+001068
Moving to projects directory: /var/lib/boinc/slots/19/ICMSHh7zg+001068
Moving to projects directory: /var/lib/boinc/slots/19/ICMUAh7zg+001068
  08:34:08 STEP 1069 H= 267:15 +CPU= 60.156
  08:35:10 STEP 1070 H= 267:30 +CPU= 59.554
  08:36:12 STEP 1071 H= 267:45 +CPU= 58.476
  08:37:17 STEP 1072 H= 268:00 +CPU= 62.099
  08:38:19 STEP 1073 H= 268:15 +CPU= 60.241
  08:39:22 STEP 1074 H= 268:30 +CPU= 60.011
  08:40:25 STEP 1075 H= 268:45 +CPU= 60.657
  08:41:28 STEP 1076 H= 269:00 +CPU= 60.323
  08:42:27 STEP 1077 H= 269:15 +CPU= 57.842
  08:43:31 STEP 1078 H= 269:30 +CPU= 61.382
  08:44:28 STEP 1079 H= 269:45 +CPU= 55.380
  08:45:43 STEP 1080 H= 270:00 +CPU= 73.532
Moving to projects directory: /var/lib/boinc/slots/19/ICMGGh7zg+001080
Moving to projects directory: /var/lib/boinc/slots/19/ICMSHh7zg+001080
Moving to projects directory: /var/lib/boinc/slots/19/ICMUAh7zg+001080
  08:46:47 STEP 1081 H= 270:15 +CPU= 60.846
  08:47:48 STEP 1082 H= 270:30 +CPU= 58.838
  08:48:51 STEP 1083 H= 270:45 +CPU= 60.808
  08:49:50 STEP 1084 H= 271:00 +CPU= 56.721
  08:50:51 STEP 1085 H= 271:15 +CPU= 58.447
  08:51:57 STEP 1086 H= 271:30 +CPU= 61.935
  08:52:57 STEP 1087 H= 271:45 +CPU= 57.840
  08:53:59 STEP 1088 H= 272:00 +CPU= 60.774
  08:55:07 STEP 1089 H= 272:15 +CPU= 64.186
  08:56:08 STEP 1090 H= 272:30 +CPU= 58.609
  08:57:10 STEP 1091 H= 272:45 +CPU= 59.794
  08:58:25 STEP 1092 H= 273:00 +CPU= 72.641
Moving to projects directory: /var/lib/boinc/slots/19/ICMGGh7zg+001092
Moving to projects directory: /var/lib/boinc/slots/19/ICMSHh7zg+001092
Moving to projects directory: /var/lib/boinc/slots/19/ICMUAh7zg+001092
  08:59:23 STEP 1093 H= 273:15 +CPU= 56.172
  09:00:26 STEP 1094 H= 273:30 +CPU= 61.605
  09:01:27 STEP 1095 H= 273:45 +CPU= 58.222
  09:02:30 STEP 1096 H= 274:00 +CPU= 60.829
  09:03:36 STEP 1097 H= 274:15 +CPU= 61.943
  09:04:40 STEP 1098 H= 274:30 +CPU= 62.059
  09:05:43 STEP 1099 H= 274:45 +CPU= 60.210
  09:06:46 STEP 1100 H= 275:00 +CPU= 61.253
  09:07:48 STEP 1101 H= 275:15 +CPU= 60.257
  09:08:53 STEP 1102 H= 275:30 +CPU= 61.827
  09:09:54 STEP 1103 H= 275:45 +CPU= 58.450
  09:11:17 STEP 1104 H= 276:00 +CPU= 80.336
Moving to projects directory: /var/lib/boinc/slots/19/ICMGGh7zg+001104
Moving to projects directory: /var/lib/boinc/slots/19/ICMSHh7zg+001104
Moving to projects directory: /var/lib/boinc/slots/19/ICMUAh7zg+001104
  09:12:23 STEP 1105 H= 276:15 +CPU= 63.114
  09:13:24 STEP 1106 H= 276:30 +CPU= 59.610
  09:14:24 STEP 1107 H= 276:45 +CPU= 57.671
  09:15:26 STEP 1108 H= 277:00 +CPU= 59.639
  09:16:26 STEP 1109 H= 277:15 +CPU= 57.747
  09:17:29 STEP 1110 H= 277:30 +CPU= 60.174
  09:18:29 STEP 1111 H= 277:45 +CPU= 57.665
  09:19:30 STEP 1112 H= 278:00 +CPU= 59.091
  09:20:31 STEP 1113 H= 278:15 +CPU= 58.852
  09:21:34 STEP 1114 H= 278:30 +CPU= 61.276
  09:22:38 STEP 1115 H= 278:45 +CPU= 60.300
  09:23:56 STEP 1116 H= 279:00 +CPU= 74.081
Moving to projects directory: /var/lib/boinc/slots/19/ICMGGh7zg+001116
Moving to projects directory: /var/lib/boinc/slots/19/ICMSHh7zg+001116
Moving to projects directory: /var/lib/boinc/slots/19/ICMUAh7zg+001116
  09:24:59 STEP 1117 H= 279:15 +CPU= 60.346
  09:26:00 STEP 1118 H= 279:30 +CPU= 59.940
  09:27:01 STEP 1119 H= 279:45 +CPU= 58.658
  09:28:03 STEP 1120 H= 280:00 +CPU= 59.263
  09:29:06 STEP 1121 H= 280:15 +CPU= 61.001
  09:30:08 STEP 1122 H= 280:30 +CPU= 60.168
  09:31:10 STEP 1123 H= 280:45 +CPU= 58.662
  09:32:19 STEP 1124 H= 281:00 +CPU= 62.919
  09:33:24 STEP 1125 H= 281:15 +CPU= 58.301
  09:34:32 STEP 1126 H= 281:30 +CPU= 60.752
  09:35:40 STEP 1127 H= 281:45 +CPU= 64.008
  09:37:03 STEP 1128 H= 282:00 +CPU= 77.710
Moving to projects directory: /var/lib/boinc/slots/19/ICMGGh7zg+001128
Moving to projects directory: /var/lib/boinc/slots/19/ICMSHh7zg+001128
Moving to projects directory: /var/lib/boinc/slots/19/ICMUAh7zg+001128
  09:38:10 STEP 1129 H= 282:15 +CPU= 61.517
  09:39:13 STEP 1130 H= 282:30 +CPU= 60.049
  09:40:18 STEP 1131 H= 282:45 +CPU= 61.091
  09:41:23 STEP 1132 H= 283:00 +CPU= 61.668
  09:42:28 STEP 1133 H= 283:15 +CPU= 60.081
  09:43:31 STEP 1134 H= 283:30 +CPU= 58.776
  09:44:39 STEP 1135 H= 283:45 +CPU= 62.932
  09:45:43 STEP 1136 H= 284:00 +CPU= 59.019
  09:46:49 STEP 1137 H= 284:15 +CPU= 61.047
  09:47:52 STEP 1138 H= 284:30 +CPU= 59.209
  09:48:58 STEP 1139 H= 284:45 +CPU= 62.879
  09:50:13 STEP 1140 H= 285:00 +CPU= 69.968
Moving to projects directory: /var/lib/boinc/slots/19/ICMGGh7zg+001140
Moving to projects directory: /var/lib/boinc/slots/19/ICMSHh7zg+001140
Moving to projects directory: /var/lib/boinc/slots/19/ICMUAh7zg+001140
  09:51:16 STEP 1141 H= 285:15 +CPU= 60.737
  09:52:20 STEP 1142 H= 285:30 +CPU= 60.759
  09:53:24 STEP 1143 H= 285:45 +CPU= 61.208
  09:54:26 STEP 1144 H= 286:00 +CPU= 58.871
  09:55:31 STEP 1145 H= 286:15 +CPU= 61.911
  09:56:33 STEP 1146 H= 286:30 +CPU= 58.879
  09:57:34 STEP 1147 H= 286:45 +CPU= 58.263
  09:58:41 STEP 1148 H= 287:00 +CPU= 62.963
  09:59:41 STEP 1149 H= 287:15 +CPU= 58.103
  10:00:45 STEP 1150 H= 287:30 +CPU= 60.745
  10:01:48 STEP 1151 H= 287:45 +CPU= 60.575
  10:03:15 STEP 1152 H= 288:00 +CPU= 82.874
Moving to projects directory: /var/lib/boinc/slots/19/ICMGGh7zg+001152
Moving to projects directory: /var/lib/boinc/slots/19/ICMSHh7zg+001152
Moving to projects directory: /var/lib/boinc/slots/19/ICMUAh7zg+001152
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMGGh7zg+001056
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMSHh7zg+001056
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMUAh7zg+001056
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMGGh7zg+001068
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMSHh7zg+001068
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMUAh7zg+001068
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMGGh7zg+001080
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMSHh7zg+001080
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMUAh7zg+001080
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMGGh7zg+001092
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMSHh7zg+001092
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMUAh7zg+001092
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMGGh7zg+001104
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMSHh7zg+001104
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMUAh7zg+001104
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMGGh7zg+001116
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMSHh7zg+001116
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMUAh7zg+001116
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMGGh7zg+001128
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMSHh7zg+001128
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMUAh7zg+001128
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMGGh7zg+001140
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMSHh7zg+001140
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMUAh7zg+001140
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_a2zq_2016092300_20_1017_12285705_0_r295099571_11.zip
Uploading the intermediate file: upload_file_11.zip
  10:04:20 STEP 1153 H= 288:15 +CPU= 60.460
  10:05:23 STEP 1154 H= 288:30 +CPU= 60.910
  10:06:28 STEP 1155 H= 288:45 +CPU= 63.106
  10:07:32 STEP 1156 H= 289:00 +CPU= 61.262
  10:08:28 STEP 1157 H= 289:15 +CPU= 55.021
  10:09:35 STEP 1158 H= 289:30 +CPU= 64.623
  10:10:36 STEP 1159 H= 289:45 +CPU= 59.714
  10:11:42 STEP 1160 H= 290:00 +CPU= 63.687
  10:12:47 STEP 1161 H= 290:15 +CPU= 61.465
  10:13:49 STEP 1162 H= 290:30 +CPU= 60.242
  10:14:50 STEP 1163 H= 290:45 +CPU= 59.252
  10:16:01 STEP 1164 H= 291:00 +CPU= 68.523
Moving to projects directory: /var/lib/boinc/slots/19/ICMGGh7zg+001164
Moving to projects directory: /var/lib/boinc/slots/19/ICMSHh7zg+001164
Moving to projects directory: /var/lib/boinc/slots/19/ICMUAh7zg+001164
  10:17:04 STEP 1165 H= 291:15 +CPU= 60.538
  10:18:06 STEP 1166 H= 291:30 +CPU= 60.690
  10:19:08 STEP 1167 H= 291:45 +CPU= 60.344
  10:20:11 STEP 1168 H= 292:00 +CPU= 60.278
  10:21:12 STEP 1169 H= 292:15 +CPU= 58.209
  10:22:09 STEP 1170 H= 292:30 +CPU= 54.279
  10:23:14 STEP 1171 H= 292:45 +CPU= 62.151
  10:24:16 STEP 1172 H= 293:00 +CPU= 59.386
  10:25:15 STEP 1173 H= 293:15 +CPU= 57.085
  10:26:19 STEP 1174 H= 293:30 +CPU= 60.571
  10:27:23 STEP 1175 H= 293:45 +CPU= 61.546
  10:28:34 STEP 1176 H= 294:00 +CPU= 68.824
Moving to projects directory: /var/lib/boinc/slots/19/ICMGGh7zg+001176
Moving to projects directory: /var/lib/boinc/slots/19/ICMSHh7zg+001176
Moving to projects directory: /var/lib/boinc/slots/19/ICMUAh7zg+001176
  10:29:31 STEP 1177 H= 294:15 +CPU= 55.128
  10:30:33 STEP 1178 H= 294:30 +CPU= 59.127
  10:31:35 STEP 1179 H= 294:45 +CPU= 59.979
  10:32:42 STEP 1180 H= 295:00 +CPU= 62.104
  10:33:45 STEP 1181 H= 295:15 +CPU= 60.330
  10:34:50 STEP 1182 H= 295:30 +CPU= 61.406
  10:35:53 STEP 1183 H= 295:45 +CPU= 58.804
  10:36:57 STEP 1184 H= 296:00 +CPU= 60.270
  10:38:00 STEP 1185 H= 296:15 +CPU= 59.759
  10:39:07 STEP 1186 H= 296:30 +CPU= 62.164
  10:40:11 STEP 1187 H= 296:45 +CPU= 60.247
  10:41:30 STEP 1188 H= 297:00 +CPU= 75.999
Moving to projects directory: /var/lib/boinc/slots/19/ICMGGh7zg+001188
Moving to projects directory: /var/lib/boinc/slots/19/ICMSHh7zg+001188
Moving to projects directory: /var/lib/boinc/slots/19/ICMUAh7zg+001188
  10:42:31 STEP 1189 H= 297:15 +CPU= 57.519
  10:43:34 STEP 1190 H= 297:30 +CPU= 57.859
  10:44:38 STEP 1191 H= 297:45 +CPU= 59.718
  10:45:46 STEP 1192 H= 298:00 +CPU= 64.392
  10:46:48 STEP 1193 H= 298:15 +CPU= 58.156
  10:47:48 STEP 1194 H= 298:30 +CPU= 57.659
  10:48:52 STEP 1195 H= 298:45 +CPU= 59.986
  10:50:03 STEP 1196 H= 299:00 +CPU= 62.220
  10:51:11 STEP 1197 H= 299:15 +CPU= 59.715
  10:52:13 STEP 1198 H= 299:30 +CPU= 59.989
  10:53:18 STEP 1199 H= 299:45 +CPU= 61.250
  10:54:41 STEP 1200 H= 300:00 +CPU= 79.708
Moving to projects directory: /var/lib/boinc/slots/19/ICMGGh7zg+001200
Moving to projects directory: /var/lib/boinc/slots/19/ICMSHh7zg+001200
Moving to projects directory: /var/lib/boinc/slots/19/ICMUAh7zg+001200
  10:55:43 STEP 1201 H= 300:15 +CPU= 58.888
  10:56:43 STEP 1202 H= 300:30 +CPU= 57.511
  10:57:47 STEP 1203 H= 300:45 +CPU= 60.791
  10:58:52 STEP 1204 H= 301:00 +CPU= 63.091
  10:59:54 STEP 1205 H= 301:15 +CPU= 60.564
  11:00:54 STEP 1206 H= 301:30 +CPU= 58.552
  11:01:56 STEP 1207 H= 301:45 +CPU= 59.815
  11:02:56 STEP 1208 H= 302:00 +CPU= 59.238
  11:03:58 STEP 1209 H= 302:15 +CPU= 59.756
  11:05:00 STEP 1210 H= 302:30 +CPU= 59.201
  11:06:00 STEP 1211 H= 302:45 +CPU= 58.446
  11:07:17 STEP 1212 H= 303:00 +CPU= 73.482
Moving to projects directory: /var/lib/boinc/slots/19/ICMGGh7zg+001212
Moving to projects directory: /var/lib/boinc/slots/19/ICMSHh7zg+001212
Moving to projects directory: /var/lib/boinc/slots/19/ICMUAh7zg+001212
  11:08:19 STEP 1213 H= 303:15 +CPU= 58.642
  11:09:18 STEP 1214 H= 303:30 +CPU= 57.576
[EC_DRHOOK:antares:1:1:28605:28605] [20240608:110953:1717837793:49619.984] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 3958MB (heap), 5307MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:antares:1:1:28605:28605] [20240608:110953:1717837793:49619.984] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:antares:1:1:28605:28605] [20240608:110953:1717837793:49619.984] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x820890 (old at (nil))
[EC_DRHOOK:antares:1:1:28605:28605] [20240608:110953:1717837793:49619.984] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1
[EC_DRHOOK:antares:1:1:28605:28605] [20240608:110953:1717837793:49619.984] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1
[EC_DRHOOK:antares:1:1:28605:28605] [20240608:110953:1717837793:49619.984] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3843] 3958 MB (maxheap), 5307 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:antares:1:1:28605:28605] [20240608:110953:1717837793:49619.984] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:antares:1:1:28605:28605] [20240608:110953:1717837793:49619.984] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:antares:1:1:28605:28605] [20240608:110953:1717837793:49619.984] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:antares:1:1:28605:28605] [20240608:110953:1717837793:49619.984] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:antares:1:1:28605:28605] [20240608:110953:1717837793:49619.984] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:antares:1:1:28605:28605] [20240608:110953:1717837793:49619.984] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :       CNT4 
[EC_DRHOOK:antares:1:1:28605:28605] [20240608:110953:1717837793:49619.984] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :        STEPO 
[EC_DRHOOK:antares:1:1:28605:28605] [20240608:110953:1717837793:49619.984] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :         SCAN2M 
[EC_DRHOOK:antares:1:1:28605:28605] [20240608:110953:1717837793:49619.984] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :          GP_MODEL_HEAP 
[EC_DRHOOK:antares:1:1:28605:28605] [20240608:110953:1717837793:49619.984] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :           GP_MODEL 
[EC_DRHOOK:antares:1:1:28605:28605] [20240608:110953:1717837793:49619.984] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :            EC_PHYS_DRV 
[EC_DRHOOK:antares:1:1:28605:28605] [20240608:110953:1717837793:49619.984] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :             >OMP-PHYSICS              (1001) 
[EC_DRHOOK:antares:1:1:28605:28605] [20240608:110953:1717837793:49619.984] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :              EC_PHYS 
[EC_DRHOOK:antares:1:1:28605:28605] [20240608:110953:1717837793:49619.984] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :               CALLPAR 
[EC_DRHOOK:antares:1:1:28605:28605] [20240608:110953:1717837793:49619.984] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :                CONVECTION_LAYER 
[EC_DRHOOK:antares:1:1:28605:28605] [20240608:110953:1717837793:49619.984] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :                 CUCALLN 
[EC_DRHOOK:antares:1:1:28605:28605] [20240608:110953:1717837793:49619.984] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :                  CUMASTRN 
[EC_DRHOOK:antares:1:1:28605:28605] [20240608:110953:1717837793:49619.984] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :                   CUDDRAFN 
[EC_DRHOOK:antares:1:1:28605:28605] [20240608:110954:1717837794:49620.984] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1
[EC_DRHOOK:antares:1:1:28605:28605] [20240608:110954:1717837794:49620.984] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1cf8da0 for signal#15, nsigs = 1
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source             
oifs_43r3_model.e  0000000001CF935B  Unknown               Unknown  Unknown
oifs_43r3_model.e  00000000008214C1  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DD5490  Unknown               Unknown  Unknown
oifs_43r3_model.e  00000000011E4339  cuddrafn_                 165  cuddrafn.F90
oifs_43r3_model.e  00000000011DC51E  cumastrn_                 619  cumastrn.F90
oifs_43r3_model.e  00000000011D02C2  cucalln_                  352  cucalln.F90
oifs_43r3_model.e  00000000011CD745  convection_layer_         121  convection_layer.F90
oifs_43r3_model.e  0000000000FB288B  callpar_                 1064  callpar.F90
oifs_43r3_model.e  0000000000F5FA6B  ec_phys_                  670  ec_phys.F90
oifs_43r3_model.e  0000000000F4DBC4  ec_phys_drv_              599  ec_phys_drv.F90
oifs_43r3_model.e  0000000000BD7131  gp_model_                 613  gp_model.F90
oifs_43r3_model.e  00000000012C5002  gp_model_heap_             74  gp_model_heap.F90
oifs_43r3_model.e  0000000000BACBAE  scan2m_                   535  scan2m.F90
oifs_43r3_model.e  0000000000688153  stepo_                    327  stepo.F90
oifs_43r3_model.e  00000000004CD1DC  cnt4_                    1133  cnt4.F90
oifs_43r3_model.e  00000000004B5D85  cnt3_                     267  cnt3.F90
oifs_43r3_model.e  00000000004B4F52  cnt2_                      88  cnt2.F90
oifs_43r3_model.e  000000000049D638  cnt1_                      92  cnt1.F90
oifs_43r3_model.e  0000000000406FBD  cnt0_                     146  cnt0.F90
oifs_43r3_model.e  0000000000401E7F  MAIN__                     96  master.F90
oifs_43r3_model.e  0000000000401E12  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCC34A  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCDBE7  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000000401CD5  Unknown               Unknown  Unknown
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.13_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a2zq
(argv4) batchid: 1017
(argv5) wuid: 12285705
(argv6) fclen: 20
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/19
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.13
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/19/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/19/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_6a6a82c13b0b464590d0b484bda89d1f to: /var/lib/boinc/slots/19/oifs_43r3_bl_a2zq_2016092300_20_1017_12285705.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/19/oifs_43r3_bl_a2zq_2016092300_20_1017_12285705.zip
ic_ancil_file: ic_ancil_12285705
ifsdata_file: ifsdata_12285705
climate_data_file: clim_data_12285705
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /var/lib/boinc/slots/19/ic_ancil_12285705.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/19/ic_ancil_12285705.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_db8c37e7f66a4f26248b064014fe903d to: /var/lib/boinc/slots/19/ifsdata/ifsdata_12285705.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/19/ifsdata/ifsdata_12285705.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/19/159l_2/clim_data_12285705.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/19/159l_2/clim_data_12285705.zip
Checking for progress XML file: /var/lib/boinc/slots/19/progress_file_12285705.xml
Opened progress file ok : /var/lib/boinc/slots/19/progress_file_12285705.xml
-- Model is restarting --
Adjusting last_iter, 1215, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/19/progress_file_12285705.xml
last_cpu_time: 55956
upload_file_number: 12
last_iter: 1201
last_upload: 1036800
model_completed: 0
total_length_of_simulation: 1728000
result_base_name: oifs_43r3_bl_a2zq_2016092300_20_1017_12285705_0_r295099571
The child process has been launched with process id: 8059
Executing the command: /var/lib/boinc/slots/19/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:antares:1:1:8059:8059] [20240610:162012:1718029212:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2091] fp = 0x314f440
[EC_DRHOOK:antares:1:1:8059:8059] [20240610:162012:1718029212:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:antares:1:1:8059:8059] [20240610:162012:1718029212:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:antares:1:1:8059:8059] [20240610:162012:1718029212:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:antares:1:1:8059:8059] [20240610:162012:1718029212:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:antares:1:1:8059:8059] [20240610:162012:1718029212:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:antares:1:1:8059:8059] [20240610:162012:1718029212:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:antares:1:1:8059:8059] [20240610:162012:1718029212:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:antares:1:1:8059:8059] [20240610:162012:1718029212:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:antares:1:1:8059:8059] [20240610:162012:1718029212:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:antares:1:1:8059:8059] [20240610:162012:1718029212:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:antares:1:1:8059:8059] [20240610:162012:1718029212:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:antares:1:1:8059:8059] [20240610:162012:1718029212:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:antares:1:1:8059:8059] [20240610:162012:1718029212:0.000] [ignore_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:antares:1:1:8059:8059] [20240610:162012:1718029212:0.000] [restore_default_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:antares:1:1:8059:8059] [20240610:162012:1718029212:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:antares:1:1:8059:8059] [20240610:162012:1718029212:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:antares:1:1:8059:8059] [20240610:162012:1718029212:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:antares:1:1:8059:8059] [20240610:162012:1718029212:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x820cf0 (old at (nil))
[EC_DRHOOK:antares:1:1:8059:8059] [20240610:162012:1718029212:0.000] [trapfpe_treatment@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:antares:1:1:8059:8059] [20240610:162012:1718029212:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:antares:1:1:8059:8059] [20240610:162012:1718029212:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:antares:1:1:8059:8059] [20240610:162012:1718029212:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x820cf0 (old at (nil))
[EC_DRHOOK:antares:1:1:8059:8059] [20240610:162012:1718029212:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:antares:1:1:8059:8059] [20240610:162012:1718029212:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:antares:1:1:8059:8059] [20240610:162012:1718029212:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:antares:1:1:8059:8059] [20240610:162012:1718029212:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x820cf0 (old at (nil))
[EC_DRHOOK:antares:1:1:8059:8059] [20240610:162012:1718029212:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x820cf0 (old at (nil))
[EC_DRHOOK:antares:1:1:8059:8059] [20240610:162012:1718029212:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:antares:1:1:8059:8059] [20240610:162012:1718029212:0.001] [catch_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  16:20:56 STEP 1200 H= 300:00 +CPU= 35.510
  16:21:25 STEP 1201 H= 300:15 +CPU= 29.268
  16:21:59 STEP 1202 H= 300:30 +CPU= 33.990
  16:22:26 STEP 1203 H= 300:45 +CPU= 26.092
  16:23:08 STEP 1204 H= 301:00 +CPU= 40.307
  16:23:58 STEP 1205 H= 301:15 +CPU= 45.171
  16:24:51 STEP 1206 H= 301:30 +CPU= 49.716
  16:25:47 STEP 1207 H= 301:45 +CPU= 49.852
  16:26:43 STEP 1208 H= 302:00 +CPU= 51.527
  16:27:46 STEP 1209 H= 302:15 +CPU= 51.792
  16:28:45 STEP 1210 H= 302:30 +CPU= 53.053
  16:29:43 STEP 1211 H= 302:45 +CPU= 51.025
  16:30:54 STEP 1212 H= 303:00 +CPU= 63.997
Moving to projects directory: /var/lib/boinc/slots/19/ICMGGh7zg+001212
Moving to projects directory: /var/lib/boinc/slots/19/ICMSHh7zg+001212
Moving to projects directory: /var/lib/boinc/slots/19/ICMUAh7zg+001212
  16:31:51 STEP 1213 H= 303:15 +CPU= 51.014
  16:32:51 STEP 1214 H= 303:30 +CPU= 52.073
  16:33:51 STEP 1215 H= 303:45 +CPU= 53.175
  16:34:51 STEP 1216 H= 304:00 +CPU= 52.951
  16:35:52 STEP 1217 H= 304:15 +CPU= 53.234
  16:36:50 STEP 1218 H= 304:30 +CPU= 51.824
  16:37:54 STEP 1219 H= 304:45 +CPU= 54.099
  16:38:55 STEP 1220 H= 305:00 +CPU= 54.502
  16:39:59 STEP 1221 H= 305:15 +CPU= 56.727
  16:41:02 STEP 1222 H= 305:30 +CPU= 56.519
  16:42:03 STEP 1223 H= 305:45 +CPU= 51.624
  16:43:18 STEP 1224 H= 306:00 +CPU= 65.356
Moving to projects directory: /var/lib/boinc/slots/19/ICMGGh7zg+001224
Moving to projects directory: /var/lib/boinc/slots/19/ICMSHh7zg+001224
Moving to projects directory: /var/lib/boinc/slots/19/ICMUAh7zg+001224
  16:44:14 STEP 1225 H= 306:15 +CPU= 51.327
  16:45:08 STEP 1226 H= 306:30 +CPU= 49.309
  16:46:04 STEP 1227 H= 306:45 +CPU= 50.549
  16:47:02 STEP 1228 H= 307:00 +CPU= 51.247
  16:48:01 STEP 1229 H= 307:15 +CPU= 52.716
  16:48:58 STEP 1230 H= 307:30 +CPU= 52.395
  16:49:55 STEP 1231 H= 307:45 +CPU= 50.674
  16:50:53 STEP 1232 H= 308:00 +CPU= 52.795
  16:51:51 STEP 1233 H= 308:15 +CPU= 52.576
  16:52:49 STEP 1234 H= 308:30 +CPU= 52.373
  16:53:49 STEP 1235 H= 308:45 +CPU= 54.110
  16:55:01 STEP 1236 H= 309:00 +CPU= 64.811
Moving to projects directory: /var/lib/boinc/slots/19/ICMGGh7zg+001236
Moving to projects directory: /var/lib/boinc/slots/19/ICMSHh7zg+001236
Moving to projects directory: /var/lib/boinc/slots/19/ICMUAh7zg+001236
  16:56:00 STEP 1237 H= 309:15 +CPU= 53.529
  16:57:00 STEP 1238 H= 309:30 +CPU= 53.647
  16:57:59 STEP 1239 H= 309:45 +CPU= 53.067
  16:59:01 STEP 1240 H= 310:00 +CPU= 55.143
  17:00:00 STEP 1241 H= 310:15 +CPU= 53.133
  17:01:00 STEP 1242 H= 310:30 +CPU= 54.584
  17:01:58 STEP 1243 H= 310:45 +CPU= 52.567
  17:02:58 STEP 1244 H= 311:00 +CPU= 55.074
  17:03:54 STEP 1245 H= 311:15 +CPU= 51.653
  17:04:49 STEP 1246 H= 311:30 +CPU= 54.058
  17:05:45 STEP 1247 H= 311:45 +CPU= 52.974
  17:06:56 STEP 1248 H= 312:00 +CPU= 67.962
Moving to projects directory: /var/lib/boinc/slots/19/ICMGGh7zg+001248
Moving to projects directory: /var/lib/boinc/slots/19/ICMSHh7zg+001248
Moving to projects directory: /var/lib/boinc/slots/19/ICMUAh7zg+001248
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMGGh7zg+001152
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMSHh7zg+001152
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMUAh7zg+001152
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMGGh7zg+001164
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMSHh7zg+001164
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMUAh7zg+001164
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMGGh7zg+001176
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMSHh7zg+001176
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMUAh7zg+001176
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMGGh7zg+001188
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMSHh7zg+001188
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMUAh7zg+001188
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMGGh7zg+001200
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMSHh7zg+001200
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMUAh7zg+001200
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMGGh7zg+001212
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMSHh7zg+001212
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMUAh7zg+001212
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMGGh7zg+001224
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMSHh7zg+001224
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMUAh7zg+001224
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMGGh7zg+001236
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMSHh7zg+001236
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMUAh7zg+001236
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_a2zq_2016092300_20_1017_12285705_0_r295099571_12.zip
Uploading the intermediate file: upload_file_12.zip
  17:07:54 STEP 1249 H= 312:15 +CPU= 54.537
  17:08:51 STEP 1250 H= 312:30 +CPU= 54.890
  17:09:51 STEP 1251 H= 312:45 +CPU= 57.615
  17:10:54 STEP 1252 H= 313:00 +CPU= 59.749
  17:11:47 STEP 1253 H= 313:15 +CPU= 49.760
  17:12:44 STEP 1254 H= 313:30 +CPU= 53.546
  17:13:39 STEP 1255 H= 313:45 +CPU= 52.243
  17:14:36 STEP 1256 H= 314:00 +CPU= 55.309
  17:15:34 STEP 1257 H= 314:15 +CPU= 55.276
  17:16:33 STEP 1258 H= 314:30 +CPU= 56.859
  17:17:28 STEP 1259 H= 314:45 +CPU= 53.486
  17:18:38 STEP 1260 H= 315:00 +CPU= 66.759
Moving to projects directory: /var/lib/boinc/slots/19/ICMGGh7zg+001260
Moving to projects directory: /var/lib/boinc/slots/19/ICMSHh7zg+001260
Moving to projects directory: /var/lib/boinc/slots/19/ICMUAh7zg+001260
  17:19:33 STEP 1261 H= 315:15 +CPU= 53.017
  17:20:35 STEP 1262 H= 315:30 +CPU= 59.739
  17:21:35 STEP 1263 H= 315:45 +CPU= 57.256
  17:22:35 STEP 1264 H= 316:00 +CPU= 57.656
  17:23:33 STEP 1265 H= 316:15 +CPU= 56.050
  17:24:34 STEP 1266 H= 316:30 +CPU= 57.329
  17:25:30 STEP 1267 H= 316:45 +CPU= 54.572
  17:26:32 STEP 1268 H= 317:00 +CPU= 60.284
  17:27:33 STEP 1269 H= 317:15 +CPU= 58.491
  17:28:34 STEP 1270 H= 317:30 +CPU= 58.082
  17:29:30 STEP 1271 H= 317:45 +CPU= 54.053
  17:30:48 STEP 1272 H= 318:00 +CPU= 75.514
Moving to projects directory: /var/lib/boinc/slots/19/ICMGGh7zg+001272
Moving to projects directory: /var/lib/boinc/slots/19/ICMSHh7zg+001272
Moving to projects directory: /var/lib/boinc/slots/19/ICMUAh7zg+001272
  17:31:48 STEP 1273 H= 318:15 +CPU= 57.455
  17:32:45 STEP 1274 H= 318:30 +CPU= 55.936
  17:33:48 STEP 1275 H= 318:45 +CPU= 59.760
  17:34:46 STEP 1276 H= 319:00 +CPU= 55.182
  17:35:49 STEP 1277 H= 319:15 +CPU= 60.887
  17:36:49 STEP 1278 H= 319:30 +CPU= 58.136
  17:37:51 STEP 1279 H= 319:45 +CPU= 59.462
  17:38:51 STEP 1280 H= 320:00 +CPU= 57.097
  17:39:51 STEP 1281 H= 320:15 +CPU= 57.671
  17:40:52 STEP 1282 H= 320:30 +CPU= 58.954
  17:41:54 STEP 1283 H= 320:45 +CPU= 58.396
  17:43:13 STEP 1284 H= 321:00 +CPU= 75.872
Moving to projects directory: /var/lib/boinc/slots/19/ICMGGh7zg+001284
Moving to projects directory: /var/lib/boinc/slots/19/ICMSHh7zg+001284
Moving to projects directory: /var/lib/boinc/slots/19/ICMUAh7zg+001284
  17:44:16 STEP 1285 H= 321:15 +CPU= 59.897
  17:45:13 STEP 1286 H= 321:30 +CPU= 55.794
  17:46:09 STEP 1287 H= 321:45 +CPU= 54.487
  17:47:12 STEP 1288 H= 322:00 +CPU= 59.484
  17:48:13 STEP 1289 H= 322:15 +CPU= 58.878
  17:49:13 STEP 1290 H= 322:30 +CPU= 57.521
  17:50:11 STEP 1291 H= 322:45 +CPU= 55.841
  17:51:06 STEP 1292 H= 323:00 +CPU= 53.466
  17:52:05 STEP 1293 H= 323:15 +CPU= 56.682
  17:53:06 STEP 1294 H= 323:30 +CPU= 58.348
  17:54:05 STEP 1295 H= 323:45 +CPU= 56.589
  17:55:30 STEP 1296 H= 324:00 +CPU= 80.318
Moving to projects directory: /var/lib/boinc/slots/19/ICMGGh7zg+001296
Moving to projects directory: /var/lib/boinc/slots/19/ICMSHh7zg+001296
Moving to projects directory: /var/lib/boinc/slots/19/ICMUAh7zg+001296
  17:56:31 STEP 1297 H= 324:15 +CPU= 56.916
  17:57:36 STEP 1298 H= 324:30 +CPU= 61.976
  17:58:35 STEP 1299 H= 324:45 +CPU= 57.177
  17:59:36 STEP 1300 H= 325:00 +CPU= 58.806
  18:00:36 STEP 1301 H= 325:15 +CPU= 57.161
  18:01:42 STEP 1302 H= 325:30 +CPU= 61.649
  18:02:42 STEP 1303 H= 325:45 +CPU= 58.208
  18:03:45 STEP 1304 H= 326:00 +CPU= 59.684
  18:04:45 STEP 1305 H= 326:15 +CPU= 57.338
  18:05:44 STEP 1306 H= 326:30 +CPU= 55.520
  18:06:41 STEP 1307 H= 326:45 +CPU= 55.818
  18:07:53 STEP 1308 H= 327:00 +CPU= 68.959
Moving to projects directory: /var/lib/boinc/slots/19/ICMGGh7zg+001308
Moving to projects directory: /var/lib/boinc/slots/19/ICMSHh7zg+001308
Moving to projects directory: /var/lib/boinc/slots/19/ICMUAh7zg+001308
  18:08:55 STEP 1309 H= 327:15 +CPU= 59.156
  18:09:57 STEP 1310 H= 327:30 +CPU= 59.413
  18:10:55 STEP 1311 H= 327:45 +CPU= 56.044
  18:11:56 STEP 1312 H= 328:00 +CPU= 58.487
  18:12:52 STEP 1313 H= 328:15 +CPU= 54.323
  18:13:53 STEP 1314 H= 328:30 +CPU= 58.186
  18:14:51 STEP 1315 H= 328:45 +CPU= 55.945
  18:15:49 STEP 1316 H= 329:00 +CPU= 55.656
  18:16:47 STEP 1317 H= 329:15 +CPU= 56.132
  18:17:47 STEP 1318 H= 329:30 +CPU= 57.135
  18:18:45 STEP 1319 H= 329:45 +CPU= 56.811
  18:19:59 STEP 1320 H= 330:00 +CPU= 70.568
Moving to projects directory: /var/lib/boinc/slots/19/ICMGGh7zg+001320
Moving to projects directory: /var/lib/boinc/slots/19/ICMSHh7zg+001320
Moving to projects directory: /var/lib/boinc/slots/19/ICMUAh7zg+001320
  18:21:01 STEP 1321 H= 330:15 +CPU= 58.574
  18:22:02 STEP 1322 H= 330:30 +CPU= 55.122
  18:23:07 STEP 1323 H= 330:45 +CPU= 61.351
  18:24:08 STEP 1324 H= 331:00 +CPU= 58.692
  18:25:10 STEP 1325 H= 331:15 +CPU= 59.950
  18:26:10 STEP 1326 H= 331:30 +CPU= 57.275
  18:27:11 STEP 1327 H= 331:45 +CPU= 58.986
  18:28:12 STEP 1328 H= 332:00 +CPU= 59.217
  18:29:14 STEP 1329 H= 332:15 +CPU= 59.908
  18:30:12 STEP 1330 H= 332:30 +CPU= 56.218
  18:31:10 STEP 1331 H= 332:45 +CPU= 56.700
  18:32:27 STEP 1332 H= 333:00 +CPU= 72.411
Moving to projects directory: /var/lib/boinc/slots/19/ICMGGh7zg+001332
Moving to projects directory: /var/lib/boinc/slots/19/ICMSHh7zg+001332
Moving to projects directory: /var/lib/boinc/slots/19/ICMUAh7zg+001332
  18:33:33 STEP 1333 H= 333:15 +CPU= 62.949
  18:34:30 STEP 1334 H= 333:30 +CPU= 54.535
  18:35:28 STEP 1335 H= 333:45 +CPU= 55.850
  18:36:29 STEP 1336 H= 334:00 +CPU= 58.444
  18:37:27 STEP 1337 H= 334:15 +CPU= 56.177
  18:38:24 STEP 1338 H= 334:30 +CPU= 55.154
  18:39:24 STEP 1339 H= 334:45 +CPU= 57.914
  18:40:27 STEP 1340 H= 335:00 +CPU= 60.004
  18:41:20 STEP 1341 H= 335:15 +CPU= 51.988
  18:42:21 STEP 1342 H= 335:30 +CPU= 58.371
  18:43:23 STEP 1343 H= 335:45 +CPU= 59.354
  18:44:50 STEP 1344 H= 336:00 +CPU= 81.920
Moving to projects directory: /var/lib/boinc/slots/19/ICMGGh7zg+001344
Moving to projects directory: /var/lib/boinc/slots/19/ICMSHh7zg+001344
Moving to projects directory: /var/lib/boinc/slots/19/ICMUAh7zg+001344
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMGGh7zg+001248
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMSHh7zg+001248
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMUAh7zg+001248
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMGGh7zg+001260
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMSHh7zg+001260
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMUAh7zg+001260
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMGGh7zg+001272
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMSHh7zg+001272
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMUAh7zg+001272
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMGGh7zg+001284
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMSHh7zg+001284
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMUAh7zg+001284
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMGGh7zg+001296
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMSHh7zg+001296
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMUAh7zg+001296
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMGGh7zg+001308
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMSHh7zg+001308
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMUAh7zg+001308
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMGGh7zg+001320
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMSHh7zg+001320
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMUAh7zg+001320
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMGGh7zg+001332
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMSHh7zg+001332
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMUAh7zg+001332
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_a2zq_2016092300_20_1017_12285705_0_r295099571_13.zip
Uploading the intermediate file: upload_file_13.zip
  18:45:49 STEP 1345 H= 336:15 +CPU= 56.294
  18:46:51 STEP 1346 H= 336:30 +CPU= 59.219
  18:47:53 STEP 1347 H= 336:45 +CPU= 59.239
  18:48:52 STEP 1348 H= 337:00 +CPU= 56.652
  18:49:53 STEP 1349 H= 337:15 +CPU= 57.590
  18:50:58 STEP 1350 H= 337:30 +CPU= 62.143
  18:51:58 STEP 1351 H= 337:45 +CPU= 57.591
  18:53:01 STEP 1352 H= 338:00 +CPU= 59.849
  18:54:03 STEP 1353 H= 338:15 +CPU= 59.044
  18:55:01 STEP 1354 H= 338:30 +CPU= 56.643
  18:56:01 STEP 1355 H= 338:45 +CPU= 57.235
  18:57:21 STEP 1356 H= 339:00 +CPU= 72.537
Moving to projects directory: /var/lib/boinc/slots/19/ICMGGh7zg+001356
Moving to projects directory: /var/lib/boinc/slots/19/ICMSHh7zg+001356
Moving to projects directory: /var/lib/boinc/slots/19/ICMUAh7zg+001356
  18:58:25 STEP 1357 H= 339:15 +CPU= 57.319
  18:59:29 STEP 1358 H= 339:30 +CPU= 59.943
  19:00:28 STEP 1359 H= 339:45 +CPU= 55.036
  19:01:30 STEP 1360 H= 340:00 +CPU= 58.367
  19:02:32 STEP 1361 H= 340:15 +CPU= 58.786
  19:03:34 STEP 1362 H= 340:30 +CPU= 58.684
  19:04:38 STEP 1363 H= 340:45 +CPU= 60.908
  19:05:37 STEP 1364 H= 341:00 +CPU= 57.514
  19:06:39 STEP 1365 H= 341:15 +CPU= 60.485
  19:07:37 STEP 1366 H= 341:30 +CPU= 56.737
  19:08:35 STEP 1367 H= 341:45 +CPU= 56.024
  19:09:44 STEP 1368 H= 342:00 +CPU= 67.308
Moving to projects directory: /var/lib/boinc/slots/19/ICMGGh7zg+001368
Moving to projects directory: /var/lib/boinc/slots/19/ICMSHh7zg+001368
Moving to projects directory: /var/lib/boinc/slots/19/ICMUAh7zg+001368
  19:10:44 STEP 1369 H= 342:15 +CPU= 57.915
  19:11:40 STEP 1370 H= 342:30 +CPU= 54.178
  19:12:47 STEP 1371 H= 342:45 +CPU= 64.293
  19:13:50 STEP 1372 H= 343:00 +CPU= 60.378
  19:14:54 STEP 1373 H= 343:15 +CPU= 61.670
  19:15:57 STEP 1374 H= 343:30 +CPU= 60.285
  19:16:52 STEP 1375 H= 343:45 +CPU= 53.438
  19:17:52 STEP 1376 H= 344:00 +CPU= 58.717
  19:18:54 STEP 1377 H= 344:15 +CPU= 58.727
  19:19:51 STEP 1378 H= 344:30 +CPU= 55.437
  19:20:50 STEP 1379 H= 344:45 +CPU= 55.847
  19:22:13 STEP 1380 H= 345:00 +CPU= 69.965
Moving to projects directory: /var/lib/boinc/slots/19/ICMGGh7zg+001380
Moving to projects directory: /var/lib/boinc/slots/19/ICMSHh7zg+001380
Moving to projects directory: /var/lib/boinc/slots/19/ICMUAh7zg+001380
  19:23:25 STEP 1381 H= 345:15 +CPU= 55.917
  19:24:28 STEP 1382 H= 345:30 +CPU= 56.759
  19:25:34 STEP 1383 H= 345:45 +CPU= 60.401
  19:26:38 STEP 1384 H= 346:00 +CPU= 62.102
  19:27:41 STEP 1385 H= 346:15 +CPU= 60.169
  19:28:41 STEP 1386 H= 346:30 +CPU= 58.097
  19:29:42 STEP 1387 H= 346:45 +CPU= 57.925
  19:30:40 STEP 1388 H= 347:00 +CPU= 56.285
  19:31:42 STEP 1389 H= 347:15 +CPU= 60.064
  19:32:40 STEP 1390 H= 347:30 +CPU= 55.780
  19:33:39 STEP 1391 H= 347:45 +CPU= 57.770
  19:34:55 STEP 1392 H= 348:00 +CPU= 73.999
Moving to projects directory: /var/lib/boinc/slots/19/ICMGGh7zg+001392
Moving to projects directory: /var/lib/boinc/slots/19/ICMSHh7zg+001392
Moving to projects directory: /var/lib/boinc/slots/19/ICMUAh7zg+001392
  19:35:50 STEP 1393 H= 348:15 +CPU= 54.075
  19:36:47 STEP 1394 H= 348:30 +CPU= 56.175
  19:37:42 STEP 1395 H= 348:45 +CPU= 53.427
  19:38:34 STEP 1396 H= 349:00 +CPU= 51.187
  19:39:23 STEP 1397 H= 349:15 +CPU= 48.886
  19:40:09 STEP 1398 H= 349:30 +CPU= 45.223
  19:40:51 STEP 1399 H= 349:45 +CPU= 41.654
  19:41:38 STEP 1400 H= 350:00 +CPU= 45.757
  19:42:23 STEP 1401 H= 350:15 +CPU= 44.392
  19:43:14 STEP 1402 H= 350:30 +CPU= 49.244
  19:43:58 STEP 1403 H= 350:45 +CPU= 43.354
  19:44:53 STEP 1404 H= 351:00 +CPU= 53.424
Moving to projects directory: /var/lib/boinc/slots/19/ICMGGh7zg+001404
Moving to projects directory: /var/lib/boinc/slots/19/ICMSHh7zg+001404
Moving to projects directory: /var/lib/boinc/slots/19/ICMUAh7zg+001404
  19:45:44 STEP 1405 H= 351:15 +CPU= 49.009
  19:46:33 STEP 1406 H= 351:30 +CPU= 48.019
  19:47:21 STEP 1407 H= 351:45 +CPU= 47.413
  19:48:10 STEP 1408 H= 352:00 +CPU= 47.733
  19:48:55 STEP 1409 H= 352:15 +CPU= 44.164
  19:49:42 STEP 1410 H= 352:30 +CPU= 46.082
  19:50:31 STEP 1411 H= 352:45 +CPU= 47.638
  19:51:18 STEP 1412 H= 353:00 +CPU= 46.296
  19:52:10 STEP 1413 H= 353:15 +CPU= 50.188
  19:52:55 STEP 1414 H= 353:30 +CPU= 43.948
  19:53:46 STEP 1415 H= 353:45 +CPU= 49.707
  19:54:42 STEP 1416 H= 354:00 +CPU= 55.768
Moving to projects directory: /var/lib/boinc/slots/19/ICMGGh7zg+001416
Moving to projects directory: /var/lib/boinc/slots/19/ICMSHh7zg+001416
Moving to projects directory: /var/lib/boinc/slots/19/ICMUAh7zg+001416
  19:55:29 STEP 1417 H= 354:15 +CPU= 45.742
  19:56:21 STEP 1418 H= 354:30 +CPU= 50.137
  19:57:10 STEP 1419 H= 354:45 +CPU= 48.310
  19:57:59 STEP 1420 H= 355:00 +CPU= 48.154
  19:58:46 STEP 1421 H= 355:15 +CPU= 45.750
  19:59:34 STEP 1422 H= 355:30 +CPU= 46.825
  20:00:20 STEP 1423 H= 355:45 +CPU= 45.361
  20:01:01 STEP 1424 H= 356:00 +CPU= 40.756
  20:01:52 STEP 1425 H= 356:15 +CPU= 49.473
  20:02:33 STEP 1426 H= 356:30 +CPU= 40.141
  20:03:19 STEP 1427 H= 356:45 +CPU= 46.012
  20:04:14 STEP 1428 H= 357:00 +CPU= 53.739
Moving to projects directory: /var/lib/boinc/slots/19/ICMGGh7zg+001428
Moving to projects directory: /var/lib/boinc/slots/19/ICMSHh7zg+001428
Moving to projects directory: /var/lib/boinc/slots/19/ICMUAh7zg+001428
  20:05:04 STEP 1429 H= 357:15 +CPU= 48.377
  20:05:55 STEP 1430 H= 357:30 +CPU= 50.284
  20:06:44 STEP 1431 H= 357:45 +CPU= 47.988
  20:07:37 STEP 1432 H= 358:00 +CPU= 51.762
  20:08:21 STEP 1433 H= 358:15 +CPU= 43.542
  20:09:06 STEP 1434 H= 358:30 +CPU= 43.987
  20:09:44 STEP 1435 H= 358:45 +CPU= 38.061
  20:10:24 STEP 1436 H= 359:00 +CPU= 39.334
  20:11:10 STEP 1437 H= 359:15 +CPU= 44.672
  20:11:59 STEP 1438 H= 359:30 +CPU= 48.421
  20:12:43 STEP 1439 H= 359:45 +CPU= 43.225
  20:13:48 STEP 1440 H= 360:00 +CPU= 60.368
..The child process terminated with status: 0
>>> Printing last 70 lines from file: NODE.001_01

 
DDH------------------------------------------------------------------------
I                                                                         I
I TIME =  360.000H DDH * Diagnostic Var.,     AREA                GLOBAL  I
I                                                                         I
I-------------------------------------------------------------------------I
I   VARIABLE   I      Initial value         I        Current value        I
I-------------------------------------------------------------------------I
I  OMEGA (P/S) I     -0.4308327369E-21      I       0.4092587381E-07      I
ICloud fractionI      0.0000000000E+00      I       0.7108308181E-02      I
I  Relat. Hum. I      0.9126445569E-01      I       0.1336878307E+00      I
I  PASS 01     I      0.0000000000E+00      I       0.2798617147E+02      I
DDH------------------------------------------------------------------------
  
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1440 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001440 MODE=a
 IO-STREAM CLOSED - ICMGGh7zg+001440
 MPI-TASK:   1 -    313415828 BYTES IN      7 RECORDS TRANSFERRED IN    0.0047 SECONDS 66684.2 Mbytes/s, TOTAL TIME=    0.0964( 4.9%)
 20:13:48 STEP 1440 H= 360:00 +CPU= 60.368
 END CNT3
 NSTEP =  1440 CNT0   000000000  43.883  43.883  47.238   0.929               0               0               0               0               0
IO-STREAM STATISTICS, TOTAL NO OF REC/BYTES READ           46/  1069125708 WRITTEN         7150/************
 -TASK-OPENED-OPEN-RECS IN -KBYTE IN    -RECS OUT-KBYTE OUT-WALL   -WALL IN-WALL
  OU-TOT IN -TOT OUT
   1    128    0       46      1044068     7150 ************    24.4     3.8    20.6     6.0    47.8
===-=== START OF TIMING STATISTICS ===-===
 
STATS FOR ALL TASKS
 NUM ROUTINE                                     CALLS  MEAN(ms)   MAX(ms)   FRAC(%)  UNBAL(%)
   0 CNT0     - COMPLETE EXECUTION                   1 ********* *********    100.00      0.00
   1 CNT4     - FORWARD INTEGRATION                  1 ********* *********     99.94      0.00
   8 SCAN2M - GRID-POINT DYNAMICS                  261   10834.0   10834.0     20.17      0.00
   9 SPCM     - SPECTRAL COMP.                     240    2826.6    2826.6      4.84      0.00
  10 SCAN2M - PHYSICS                              241   25184.9   25184.9     43.30      0.00
  11 IOPACK   - OUTPUT P.P. RESULTS                 20    1612.3    1612.3      0.23      0.00
  12 SPNORM   - SPECTRAL NORM COMP.                 11     122.5     122.5      0.01      0.00
  17 GRIDFPOS IN CNT4                               20      78.8      78.8      0.01      0.00
  27 RESTART FILES - WRITING                         5    6447.9    6447.9      0.23      0.00
  28 RESTART FILES - READING                         1    8028.0    8028.0      0.06      0.00
  29 SU4FPOS IN CNT4                                20       1.4       1.4      0.00      0.00
  30 DYNFPOS IN CNT4                                20   11516.4   11516.4      1.64      0.00
  31 POSDDH IN STEPO                                 2       4.8       4.8      0.00      0.00
  37 CPGLAG   - SL COMPUTATIONS                    241   11122.4   11122.4     19.12      0.00
  39 SU0YOMB                                         1     632.7     632.7      0.00      0.00
  51 SCAN2M   - SL COMM. PART 1                    241      44.1      44.1      0.08      0.00
  54 SPCM     - M TO S/S TO M TRANSP.              240    1298.1    1298.1      2.22      0.00
  55 SPCIMPF  - S TO M/M TO S TRANSP.              240     142.5     142.5      0.24      0.00
  56 SPNORM   - SPECTRAL NORM COMM.                 11       2.2       2.2      0.00      0.00
 102 LTINV_CTL   - INVERSE LEGENDRE TRANSFORM      524     809.1     809.1      3.02      0.00
 103 LTDIR_CTL   - DIRECT LEGENDRE TRANSFORM       500     448.6     448.6      1.60      0.00
 106 FTDIR_CTL   - DIRECT FOURIER TRANSFORM        500     162.7     162.7      0.58      0.00
 107 FTINV_CTL   - INVERSE FOURIER TRANSFORM       524     264.5     264.5      0.99      0.00
 140 SULEG       - COMP. OF LEGENDRE POL.            1      54.1      54.1      0.00      0.00
 152 LTINV_CTL   - M TO L TRANSPOSITION            524     143.4     143.4      0.54      0.00
 153 LTDIR_CTL   - L TO M TRANSPOSITION            500      91.2      91.2      0.33      0.00
 157 FTINV_CTL   - L TO G TRANSPOSITION            524     199.4     199.4      0.75      0.00
 158 FTDIR_CTL   - G TO L TRANSPOSITION            500     100.8     100.8      0.36      0.00
 400 GSTATS                                      28864       0.0       0.0      0.00      0.00
 401 GSTATS HOOK                                 26802       0.0       0.0      0.00      0.00
TOTAL MEASURED IMBALANCE =       0.0 SECONDS,  0.0 PERCENT
TOTAL WALLCLOCK TIME    14016.0 CPU TIME   13294.2 VECTOR TIME    13294.2


===-=== END   OF TIMING STATISTICS ===-===
FORECAST DAYS PER DAY       92.5
  *** END CNT0 *** 
------------------------------------------------
Moving to projects directory: /var/lib/boinc/slots/19/ICMGGh7zg+001440
Moving to projects directory: /var/lib/boinc/slots/19/ICMSHh7zg+001440
Moving to projects directory: /var/lib/boinc/slots/19/ICMUAh7zg+001440
Adding to the zip: /var/lib/boinc/slots/19/NODE.001_01
Adding to the zip: /var/lib/boinc/slots/19/ifs.stat
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMGGh7zg+001344
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMSHh7zg+001344
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMUAh7zg+001344
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMGGh7zg+001356
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMSHh7zg+001356
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMUAh7zg+001356
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMGGh7zg+001368
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMSHh7zg+001368
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMUAh7zg+001368
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMGGh7zg+001380
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMSHh7zg+001380
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMUAh7zg+001380
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMGGh7zg+001392
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMSHh7zg+001392
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMUAh7zg+001392
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMGGh7zg+001404
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMSHh7zg+001404
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMUAh7zg+001404
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMGGh7zg+001416
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMSHh7zg+001416
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMUAh7zg+001416
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMGGh7zg+001428
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMSHh7zg+001428
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMUAh7zg+001428
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMGGh7zg+001440
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMSHh7zg+001440
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285705/ICMUAh7zg+001440
Zipping up the final file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_a2zq_2016092300_20_1017_12285705_0_r295099571_14.zip
Uploading the final file: upload_file_14.zip
Uploading trickle at timestep: 1295100
20:17:16 (7981): called boinc_finish(0)

</stderr_txt>
<message>
upload failure: <file_xfer_error>
  <file_name>oifs_43r3_bl_a2zq_2016092300_20_1017_12285705_0_r295099571_15.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a2zq_2016092300_20_1017_12285705_0_r295099571_16.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a2zq_2016092300_20_1017_12285705_0_r295099571_17.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a2zq_2016092300_20_1017_12285705_0_r295099571_18.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a2zq_2016092300_20_1017_12285705_0_r295099571_19.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
</message>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
10 Jun 2024 18:37:16 1533094 22437314 oifs_43r3_bl_a2zq_2016092300_20_1017_12285705_0 1,295,100 69,250 0.0535


©2024 climateprediction.net