climateprediction.net home page
Task 22435578

Task 22435578

Name oifs_43r3_bl_a1oc_2016092300_20_1017_12283999_0
Workunit 12283999
Created 7 Jun 2024, 13:59:19 UTC
Sent 7 Jun 2024, 14:02:49 UTC
Report deadline 6 Aug 2024, 14:02:49 UTC
Received 8 Jun 2024, 19:15:37 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 0 (0x00000000)
Computer ID 1532456
Run time 10 hours 33 min 10 sec
CPU time 10 hours 15 min 56 sec
Validate state Invalid
Credit 1,976.32
Device peak FLOPS 6.55 GFLOPS
Application version OpenIFS 43r3 Baroclinic Lifecycle v1.13
x86_64-pc-linux-gnu
Peak working set size 5,011.59 MB
Peak swap size 5,980.80 MB
Peak disk usage 1,269.49 MB
Stderr
<core_client_version>7.18.1</core_client_version>
<![CDATA[
<stderr_txt>
5/ICMUAh7zg+001032
  16:36:06 STEP 1033 H= 258:15 +CPU= 25.962
  16:36:32 STEP 1034 H= 258:30 +CPU= 26.120
  16:36:59 STEP 1035 H= 258:45 +CPU= 26.097
  16:37:27 STEP 1036 H= 259:00 +CPU= 27.409
  16:37:54 STEP 1037 H= 259:15 +CPU= 26.168
  16:38:21 STEP 1038 H= 259:30 +CPU= 26.595
  16:38:48 STEP 1039 H= 259:45 +CPU= 26.466
  16:39:16 STEP 1040 H= 260:00 +CPU= 26.912
  16:39:42 STEP 1041 H= 260:15 +CPU= 25.294
  16:40:08 STEP 1042 H= 260:30 +CPU= 25.955
  16:40:34 STEP 1043 H= 260:45 +CPU= 25.084
  16:41:05 STEP 1044 H= 261:00 +CPU= 30.785
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001044
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001044
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001044
  16:41:32 STEP 1045 H= 261:15 +CPU= 26.175
  16:41:58 STEP 1046 H= 261:30 +CPU= 26.063
  16:42:25 STEP 1047 H= 261:45 +CPU= 25.658
  16:42:52 STEP 1048 H= 262:00 +CPU= 26.875
  16:43:19 STEP 1049 H= 262:15 +CPU= 26.081
  16:43:45 STEP 1050 H= 262:30 +CPU= 25.793
  16:44:11 STEP 1051 H= 262:45 +CPU= 25.079
  16:44:38 STEP 1052 H= 263:00 +CPU= 26.311
  16:45:05 STEP 1053 H= 263:15 +CPU= 26.218
  16:45:31 STEP 1054 H= 263:30 +CPU= 26.011
  16:45:58 STEP 1055 H= 263:45 +CPU= 26.257
  16:46:36 STEP 1056 H= 264:00 +CPU= 36.679
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001056
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001056
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMGGh7zg+000960
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMSHh7zg+000960
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMUAh7zg+000960
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMGGh7zg+000972
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMSHh7zg+000972
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMUAh7zg+000972
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMGGh7zg+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMSHh7zg+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMUAh7zg+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMGGh7zg+000996
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMSHh7zg+000996
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMUAh7zg+000996
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMGGh7zg+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMSHh7zg+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMUAh7zg+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMGGh7zg+001020
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMSHh7zg+001020
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMUAh7zg+001020
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMGGh7zg+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMSHh7zg+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMUAh7zg+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMGGh7zg+001044
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMSHh7zg+001044
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMUAh7zg+001044
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a1oc_2016092300_20_1017_12283999_0_r542734340_10.zip
Uploading the intermediate file: upload_file_10.zip
  16:47:02 STEP 1057 H= 264:15 +CPU= 26.187
  16:47:29 STEP 1058 H= 264:30 +CPU= 26.205
  16:47:56 STEP 1059 H= 264:45 +CPU= 26.663
  16:48:24 STEP 1060 H= 265:00 +CPU= 26.945
  16:48:51 STEP 1061 H= 265:15 +CPU= 26.668
  16:49:18 STEP 1062 H= 265:30 +CPU= 26.581
  16:49:44 STEP 1063 H= 265:45 +CPU= 25.666
  16:50:12 STEP 1064 H= 266:00 +CPU= 26.776
  16:50:38 STEP 1065 H= 266:15 +CPU= 25.955
  16:51:04 STEP 1066 H= 266:30 +CPU= 25.143
  16:51:29 STEP 1067 H= 266:45 +CPU= 24.984
  16:52:01 STEP 1068 H= 267:00 +CPU= 31.193
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001068
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001068
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001068
  16:52:28 STEP 1069 H= 267:15 +CPU= 26.408
  16:52:54 STEP 1070 H= 267:30 +CPU= 25.416
  16:53:20 STEP 1071 H= 267:45 +CPU= 25.057
  16:53:47 STEP 1072 H= 268:00 +CPU= 26.515
  16:54:14 STEP 1073 H= 268:15 +CPU= 25.616
  16:54:40 STEP 1074 H= 268:30 +CPU= 25.412
  16:55:06 STEP 1075 H= 268:45 +CPU= 25.713
  16:55:33 STEP 1076 H= 269:00 +CPU= 26.200
  16:55:59 STEP 1077 H= 269:15 +CPU= 25.120
  16:56:24 STEP 1078 H= 269:30 +CPU= 25.098
  16:56:49 STEP 1079 H= 269:45 +CPU= 24.530
  16:57:22 STEP 1080 H= 270:00 +CPU= 32.039
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001080
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001080
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001080
  16:57:49 STEP 1081 H= 270:15 +CPU= 26.344
  16:58:15 STEP 1082 H= 270:30 +CPU= 25.462
  16:58:42 STEP 1083 H= 270:45 +CPU= 26.052
  16:59:10 STEP 1084 H= 271:00 +CPU= 26.854
  16:59:36 STEP 1085 H= 271:15 +CPU= 25.846
  17:00:03 STEP 1086 H= 271:30 +CPU= 26.109
  17:00:29 STEP 1087 H= 271:45 +CPU= 25.498
  17:00:56 STEP 1088 H= 272:00 +CPU= 26.007
  17:01:21 STEP 1089 H= 272:15 +CPU= 24.729
  17:01:45 STEP 1090 H= 272:30 +CPU= 23.583
  17:02:08 STEP 1091 H= 272:45 +CPU= 22.832
  17:02:37 STEP 1092 H= 273:00 +CPU= 28.876
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001092
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001092
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001092
  17:03:03 STEP 1093 H= 273:15 +CPU= 25.327
  17:03:29 STEP 1094 H= 273:30 +CPU= 25.011
  17:03:55 STEP 1095 H= 273:45 +CPU= 25.279
  17:04:21 STEP 1096 H= 274:00 +CPU= 25.887
  17:04:47 STEP 1097 H= 274:15 +CPU= 25.214
  17:05:14 STEP 1098 H= 274:30 +CPU= 25.777
  17:05:41 STEP 1099 H= 274:45 +CPU= 25.856
  17:06:08 STEP 1100 H= 275:00 +CPU= 26.123
  17:06:34 STEP 1101 H= 275:15 +CPU= 25.749
  17:07:00 STEP 1102 H= 275:30 +CPU= 25.534
  17:07:27 STEP 1103 H= 275:45 +CPU= 25.709
  17:08:04 STEP 1104 H= 276:00 +CPU= 36.423
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001104
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001104
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001104
  17:08:31 STEP 1105 H= 276:15 +CPU= 25.831
  17:08:57 STEP 1106 H= 276:30 +CPU= 25.398
  17:09:23 STEP 1107 H= 276:45 +CPU= 25.169
  17:09:49 STEP 1108 H= 277:00 +CPU= 25.102
  17:10:14 STEP 1109 H= 277:15 +CPU= 24.951
  17:10:41 STEP 1110 H= 277:30 +CPU= 25.613
  17:11:06 STEP 1111 H= 277:45 +CPU= 25.131
  17:11:32 STEP 1112 H= 278:00 +CPU= 25.089
  17:11:58 STEP 1113 H= 278:15 +CPU= 24.934
  17:12:23 STEP 1114 H= 278:30 +CPU= 24.940
  17:12:49 STEP 1115 H= 278:45 +CPU= 24.609
  17:13:20 STEP 1116 H= 279:00 +CPU= 30.875
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001116
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001116
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001116
  17:13:46 STEP 1117 H= 279:15 +CPU= 25.251
  17:14:13 STEP 1118 H= 279:30 +CPU= 26.171
  17:14:39 STEP 1119 H= 279:45 +CPU= 25.064
  17:15:05 STEP 1120 H= 280:00 +CPU= 25.736
  17:15:31 STEP 1121 H= 280:15 +CPU= 25.392
  17:15:58 STEP 1122 H= 280:30 +CPU= 25.481
  17:16:23 STEP 1123 H= 280:45 +CPU= 25.173
  17:16:50 STEP 1124 H= 281:00 +CPU= 25.500
  17:17:16 STEP 1125 H= 281:15 +CPU= 25.439
  17:17:41 STEP 1126 H= 281:30 +CPU= 24.869
  17:18:07 STEP 1127 H= 281:45 +CPU= 25.011
  17:18:39 STEP 1128 H= 282:00 +CPU= 30.774
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001128
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001128
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001128
  17:19:04 STEP 1129 H= 282:15 +CPU= 25.066
  17:19:29 STEP 1130 H= 282:30 +CPU= 24.536
  17:19:55 STEP 1131 H= 282:45 +CPU= 24.702
  17:20:21 STEP 1132 H= 283:00 +CPU= 25.131
  17:20:46 STEP 1133 H= 283:15 +CPU= 24.898
  17:21:12 STEP 1134 H= 283:30 +CPU= 24.877
  17:21:37 STEP 1135 H= 283:45 +CPU= 24.797
  17:22:03 STEP 1136 H= 284:00 +CPU= 25.261
  17:22:28 STEP 1137 H= 284:15 +CPU= 24.917
  17:22:54 STEP 1138 H= 284:30 +CPU= 25.006
  17:23:20 STEP 1139 H= 284:45 +CPU= 24.859
  17:23:52 STEP 1140 H= 285:00 +CPU= 31.613
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001140
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001140
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001140
  17:24:18 STEP 1141 H= 285:15 +CPU= 24.981
  17:24:43 STEP 1142 H= 285:30 +CPU= 24.935
  17:25:09 STEP 1143 H= 285:45 +CPU= 24.726
  17:25:35 STEP 1144 H= 286:00 +CPU= 25.367
  17:26:00 STEP 1145 H= 286:15 +CPU= 24.785
  17:26:26 STEP 1146 H= 286:30 +CPU= 24.967
  17:26:51 STEP 1147 H= 286:45 +CPU= 24.888
  17:27:17 STEP 1148 H= 287:00 +CPU= 24.654
  17:27:42 STEP 1149 H= 287:15 +CPU= 24.738
  17:28:07 STEP 1150 H= 287:30 +CPU= 24.527
  17:28:32 STEP 1151 H= 287:45 +CPU= 24.663
  17:29:08 STEP 1152 H= 288:00 +CPU= 34.525
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001152
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001152
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMGGh7zg+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMSHh7zg+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMUAh7zg+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMGGh7zg+001068
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMSHh7zg+001068
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMUAh7zg+001068
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMGGh7zg+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMSHh7zg+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMUAh7zg+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMGGh7zg+001092
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMSHh7zg+001092
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMUAh7zg+001092
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMGGh7zg+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMSHh7zg+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMUAh7zg+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMGGh7zg+001116
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMSHh7zg+001116
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMUAh7zg+001116
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMGGh7zg+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMSHh7zg+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMUAh7zg+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMGGh7zg+001140
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMSHh7zg+001140
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMUAh7zg+001140
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a1oc_2016092300_20_1017_12283999_0_r542734340_11.zip
Uploading the intermediate file: upload_file_11.zip
  17:29:34 STEP 1153 H= 288:15 +CPU= 25.356
  17:29:59 STEP 1154 H= 288:30 +CPU= 24.981
  17:30:25 STEP 1155 H= 288:45 +CPU= 24.890
  17:30:51 STEP 1156 H= 289:00 +CPU= 25.099
  17:31:17 STEP 1157 H= 289:15 +CPU= 25.080
  17:31:42 STEP 1158 H= 289:30 +CPU= 25.154
  17:32:08 STEP 1159 H= 289:45 +CPU= 25.117
  17:32:34 STEP 1160 H= 290:00 +CPU= 25.497
  17:33:00 STEP 1161 H= 290:15 +CPU= 24.904
  17:33:25 STEP 1162 H= 290:30 +CPU= 24.870
  17:33:51 STEP 1163 H= 290:45 +CPU= 25.271
  17:34:24 STEP 1164 H= 291:00 +CPU= 31.360
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001164
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001164
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001164
  17:34:50 STEP 1165 H= 291:15 +CPU= 25.209
  17:35:15 STEP 1166 H= 291:30 +CPU= 25.102
  17:35:41 STEP 1167 H= 291:45 +CPU= 24.893
  17:36:08 STEP 1168 H= 292:00 +CPU= 26.033
  17:36:34 STEP 1169 H= 292:15 +CPU= 25.590
  17:37:00 STEP 1170 H= 292:30 +CPU= 25.289
  17:37:26 STEP 1171 H= 292:45 +CPU= 25.245
  17:37:52 STEP 1172 H= 293:00 +CPU= 25.575
  17:38:18 STEP 1173 H= 293:15 +CPU= 24.587
  17:38:43 STEP 1174 H= 293:30 +CPU= 24.792
  17:39:09 STEP 1175 H= 293:45 +CPU= 24.888
  17:39:40 STEP 1176 H= 294:00 +CPU= 30.701
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001176
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001176
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001176
  17:40:07 STEP 1177 H= 294:15 +CPU= 25.548
  17:40:32 STEP 1178 H= 294:30 +CPU= 25.160
  17:40:58 STEP 1179 H= 294:45 +CPU= 24.712
  17:41:24 STEP 1180 H= 295:00 +CPU= 25.806
  17:41:51 STEP 1181 H= 295:15 +CPU= 25.693
  17:42:17 STEP 1182 H= 295:30 +CPU= 25.506
  17:42:43 STEP 1183 H= 295:45 +CPU= 24.901
  17:43:09 STEP 1184 H= 296:00 +CPU= 25.690
  17:43:35 STEP 1185 H= 296:15 +CPU= 25.388
  17:44:01 STEP 1186 H= 296:30 +CPU= 25.015
  17:44:27 STEP 1187 H= 296:45 +CPU= 24.969
  17:44:58 STEP 1188 H= 297:00 +CPU= 30.638
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001188
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001188
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001188
  17:45:24 STEP 1189 H= 297:15 +CPU= 25.178
  17:45:50 STEP 1190 H= 297:30 +CPU= 24.898
  17:46:16 STEP 1191 H= 297:45 +CPU= 25.900
  17:46:43 STEP 1192 H= 298:00 +CPU= 26.093
  17:47:09 STEP 1193 H= 298:15 +CPU= 24.974
  17:47:34 STEP 1194 H= 298:30 +CPU= 24.768
  17:48:00 STEP 1195 H= 298:45 +CPU= 24.713
  17:48:25 STEP 1196 H= 299:00 +CPU= 24.750
  17:48:51 STEP 1197 H= 299:15 +CPU= 25.547
  17:49:18 STEP 1198 H= 299:30 +CPU= 25.658
  17:49:44 STEP 1199 H= 299:45 +CPU= 25.544
  17:50:20 STEP 1200 H= 300:00 +CPU= 35.415
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001200
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001200
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001200
  17:50:46 STEP 1201 H= 300:15 +CPU= 25.483
  17:51:12 STEP 1202 H= 300:30 +CPU= 25.042
  17:51:38 STEP 1203 H= 300:45 +CPU= 25.219
  17:52:05 STEP 1204 H= 301:00 +CPU= 26.363
  17:52:31 STEP 1205 H= 301:15 +CPU= 26.001
  17:52:59 STEP 1206 H= 301:30 +CPU= 27.021
  17:53:27 STEP 1207 H= 301:45 +CPU= 27.282
  17:53:54 STEP 1208 H= 302:00 +CPU= 27.077
  17:54:21 STEP 1209 H= 302:15 +CPU= 25.750
  17:54:47 STEP 1210 H= 302:30 +CPU= 25.539
  17:55:13 STEP 1211 H= 302:45 +CPU= 25.372
  17:55:46 STEP 1212 H= 303:00 +CPU= 31.645
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001212
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001212
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001212
  17:56:12 STEP 1213 H= 303:15 +CPU= 25.401
  17:56:38 STEP 1214 H= 303:30 +CPU= 25.903
  17:57:05 STEP 1215 H= 303:45 +CPU= 25.859
  17:57:32 STEP 1216 H= 304:00 +CPU= 26.042
  17:57:58 STEP 1217 H= 304:15 +CPU= 25.791
  17:58:25 STEP 1218 H= 304:30 +CPU= 26.171
  17:58:52 STEP 1219 H= 304:45 +CPU= 25.807
  17:59:19 STEP 1220 H= 305:00 +CPU= 26.292
  17:59:45 STEP 1221 H= 305:15 +CPU= 26.043
  18:00:12 STEP 1222 H= 305:30 +CPU= 25.433
  18:00:38 STEP 1223 H= 305:45 +CPU= 25.683
  18:01:11 STEP 1224 H= 306:00 +CPU= 31.967
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001224
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001224
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001224
  18:01:38 STEP 1225 H= 306:15 +CPU= 25.980
  18:02:01 STEP 1226 H= 306:30 +CPU= 22.976
  18:02:25 STEP 1227 H= 306:45 +CPU= 24.274
  18:02:52 STEP 1228 H= 307:00 +CPU= 25.721
  18:03:18 STEP 1229 H= 307:15 +CPU= 25.208
  18:03:43 STEP 1230 H= 307:30 +CPU= 24.809
  18:04:08 STEP 1231 H= 307:45 +CPU= 24.716
  18:04:34 STEP 1232 H= 308:00 +CPU= 25.624
  18:05:00 STEP 1233 H= 308:15 +CPU= 25.312
  18:05:27 STEP 1234 H= 308:30 +CPU= 25.711
  18:05:53 STEP 1235 H= 308:45 +CPU= 25.575
  18:06:26 STEP 1236 H= 309:00 +CPU= 31.868
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001236
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001236
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001236
  18:06:52 STEP 1237 H= 309:15 +CPU= 25.501
  18:07:18 STEP 1238 H= 309:30 +CPU= 25.849
  18:07:44 STEP 1239 H= 309:45 +CPU= 25.392
  18:08:11 STEP 1240 H= 310:00 +CPU= 25.600
  18:08:36 STEP 1241 H= 310:15 +CPU= 25.087
  18:09:02 STEP 1242 H= 310:30 +CPU= 25.060
  18:09:28 STEP 1243 H= 310:45 +CPU= 25.066
  18:09:54 STEP 1244 H= 311:00 +CPU= 25.486
  18:10:19 STEP 1245 H= 311:15 +CPU= 25.392
  18:10:45 STEP 1246 H= 311:30 +CPU= 24.969
  18:11:10 STEP 1247 H= 311:45 +CPU= 24.940
  18:11:47 STEP 1248 H= 312:00 +CPU= 35.190
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001248
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001248
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMGGh7zg+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMSHh7zg+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMUAh7zg+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMGGh7zg+001164
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMSHh7zg+001164
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMUAh7zg+001164
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMGGh7zg+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMSHh7zg+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMUAh7zg+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMGGh7zg+001188
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMSHh7zg+001188
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMUAh7zg+001188
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMGGh7zg+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMSHh7zg+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMUAh7zg+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMGGh7zg+001212
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMSHh7zg+001212
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMUAh7zg+001212
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMGGh7zg+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMSHh7zg+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMUAh7zg+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMGGh7zg+001236
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMSHh7zg+001236
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMUAh7zg+001236
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a1oc_2016092300_20_1017_12283999_0_r542734340_12.zip
Uploading the intermediate file: upload_file_12.zip
  18:12:13 STEP 1249 H= 312:15 +CPU= 25.481
  18:12:38 STEP 1250 H= 312:30 +CPU= 25.319
  18:13:05 STEP 1251 H= 312:45 +CPU= 25.600
  18:13:31 STEP 1252 H= 313:00 +CPU= 25.482
  18:13:57 STEP 1253 H= 313:15 +CPU= 25.519
  18:14:23 STEP 1254 H= 313:30 +CPU= 25.679
  18:14:49 STEP 1255 H= 313:45 +CPU= 25.466
  18:15:15 STEP 1256 H= 314:00 +CPU= 25.931
  18:15:42 STEP 1257 H= 314:15 +CPU= 26.008
  18:16:08 STEP 1258 H= 314:30 +CPU= 25.515
  18:16:34 STEP 1259 H= 314:45 +CPU= 25.122
  18:17:05 STEP 1260 H= 315:00 +CPU= 30.748
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001260
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001260
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001260
  18:17:31 STEP 1261 H= 315:15 +CPU= 25.160
  18:17:57 STEP 1262 H= 315:30 +CPU= 25.174
  18:18:22 STEP 1263 H= 315:45 +CPU= 25.021
  18:18:48 STEP 1264 H= 316:00 +CPU= 25.511
  18:19:13 STEP 1265 H= 316:15 +CPU= 24.143
  18:19:37 STEP 1266 H= 316:30 +CPU= 24.112
  18:20:01 STEP 1267 H= 316:45 +CPU= 23.816
  18:20:26 STEP 1268 H= 317:00 +CPU= 24.158
  18:20:50 STEP 1269 H= 317:15 +CPU= 23.935
  18:21:13 STEP 1270 H= 317:30 +CPU= 23.334
  18:21:38 STEP 1271 H= 317:45 +CPU= 23.999
  18:22:11 STEP 1272 H= 318:00 +CPU= 32.692
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001272
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001272
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001272
  18:22:38 STEP 1273 H= 318:15 +CPU= 25.887
  18:23:04 STEP 1274 H= 318:30 +CPU= 25.614
  18:23:31 STEP 1275 H= 318:45 +CPU= 25.969
  18:23:58 STEP 1276 H= 319:00 +CPU= 26.135
  18:24:24 STEP 1277 H= 319:15 +CPU= 25.631
  18:24:50 STEP 1278 H= 319:30 +CPU= 25.515
  18:25:17 STEP 1279 H= 319:45 +CPU= 25.759
  18:25:44 STEP 1280 H= 320:00 +CPU= 26.197
  18:26:11 STEP 1281 H= 320:15 +CPU= 26.487
  18:26:39 STEP 1282 H= 320:30 +CPU= 26.462
  18:27:06 STEP 1283 H= 320:45 +CPU= 26.477
  18:27:39 STEP 1284 H= 321:00 +CPU= 31.949
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001284
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001284
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001284
  18:28:06 STEP 1285 H= 321:15 +CPU= 26.181
  18:28:33 STEP 1286 H= 321:30 +CPU= 25.765
  18:28:59 STEP 1287 H= 321:45 +CPU= 25.923
  18:29:27 STEP 1288 H= 322:00 +CPU= 26.510
  18:29:54 STEP 1289 H= 322:15 +CPU= 26.080
  18:30:21 STEP 1290 H= 322:30 +CPU= 26.234
  18:30:47 STEP 1291 H= 322:45 +CPU= 26.009
  18:31:15 STEP 1292 H= 323:00 +CPU= 26.874
  18:31:42 STEP 1293 H= 323:15 +CPU= 25.892
  18:32:05 STEP 1294 H= 323:30 +CPU= 22.906
  18:32:31 STEP 1295 H= 323:45 +CPU= 25.222
  18:33:08 STEP 1296 H= 324:00 +CPU= 36.440
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001296
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001296
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001296
  18:33:35 STEP 1297 H= 324:15 +CPU= 25.927
  18:34:02 STEP 1298 H= 324:30 +CPU= 26.203
  18:34:29 STEP 1299 H= 324:45 +CPU= 26.236
  18:34:57 STEP 1300 H= 325:00 +CPU= 26.807
  18:35:24 STEP 1301 H= 325:15 +CPU= 25.924
  18:35:51 STEP 1302 H= 325:30 +CPU= 26.279
  18:36:18 STEP 1303 H= 325:45 +CPU= 26.548
  18:36:45 STEP 1304 H= 326:00 +CPU= 26.341
  18:37:12 STEP 1305 H= 326:15 +CPU= 25.893
  18:37:39 STEP 1306 H= 326:30 +CPU= 25.939
  18:38:05 STEP 1307 H= 326:45 +CPU= 25.813
  18:38:34 STEP 1308 H= 327:00 +CPU= 28.788
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001308
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001308
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001308
  18:38:58 STEP 1309 H= 327:15 +CPU= 23.208
  18:39:22 STEP 1310 H= 327:30 +CPU= 23.736
  18:39:45 STEP 1311 H= 327:45 +CPU= 23.508
  18:40:13 STEP 1312 H= 328:00 +CPU= 26.546
  18:40:39 STEP 1313 H= 328:15 +CPU= 25.368
  18:41:06 STEP 1314 H= 328:30 +CPU= 25.998
  18:41:32 STEP 1315 H= 328:45 +CPU= 25.785
  18:42:00 STEP 1316 H= 329:00 +CPU= 26.459
  18:42:26 STEP 1317 H= 329:15 +CPU= 26.129
  18:42:53 STEP 1318 H= 329:30 +CPU= 26.071
  18:43:20 STEP 1319 H= 329:45 +CPU= 25.669
  18:43:49 STEP 1320 H= 330:00 +CPU= 28.688
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001320
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001320
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001320
  18:44:14 STEP 1321 H= 330:15 +CPU= 25.219
  18:44:42 STEP 1322 H= 330:30 +CPU= 26.498
  18:45:09 STEP 1323 H= 330:45 +CPU= 26.645
  18:45:37 STEP 1324 H= 331:00 +CPU= 26.459
  18:46:04 STEP 1325 H= 331:15 +CPU= 26.267
  18:46:31 STEP 1326 H= 331:30 +CPU= 26.445
  18:46:57 STEP 1327 H= 331:45 +CPU= 25.751
  18:47:25 STEP 1328 H= 332:00 +CPU= 26.237
  18:47:51 STEP 1329 H= 332:15 +CPU= 26.071
  18:48:18 STEP 1330 H= 332:30 +CPU= 26.054
  18:48:45 STEP 1331 H= 332:45 +CPU= 25.951
  18:49:18 STEP 1332 H= 333:00 +CPU= 32.006
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001332
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001332
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001332
  18:49:46 STEP 1333 H= 333:15 +CPU= 26.570
  18:50:13 STEP 1334 H= 333:30 +CPU= 26.325
  18:50:40 STEP 1335 H= 333:45 +CPU= 26.403
  18:51:08 STEP 1336 H= 334:00 +CPU= 26.573
  18:51:35 STEP 1337 H= 334:15 +CPU= 26.767
  18:52:03 STEP 1338 H= 334:30 +CPU= 26.832
  18:52:30 STEP 1339 H= 334:45 +CPU= 26.101
  18:52:58 STEP 1340 H= 335:00 +CPU= 27.041
  18:53:25 STEP 1341 H= 335:15 +CPU= 26.314
  18:53:52 STEP 1342 H= 335:30 +CPU= 26.250
  18:54:20 STEP 1343 H= 335:45 +CPU= 26.489
  18:54:54 STEP 1344 H= 336:00 +CPU= 33.445
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001344
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001344
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMGGh7zg+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMSHh7zg+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMUAh7zg+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMGGh7zg+001260
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMSHh7zg+001260
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMUAh7zg+001260
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMGGh7zg+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMSHh7zg+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMUAh7zg+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMGGh7zg+001284
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMSHh7zg+001284
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMUAh7zg+001284
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMGGh7zg+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMSHh7zg+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMUAh7zg+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMGGh7zg+001308
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMSHh7zg+001308
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMUAh7zg+001308
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMGGh7zg+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMSHh7zg+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMUAh7zg+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMGGh7zg+001332
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMSHh7zg+001332
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMUAh7zg+001332
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a1oc_2016092300_20_1017_12283999_0_r542734340_13.zip
Uploading the intermediate file: upload_file_13.zip
  18:55:21 STEP 1345 H= 336:15 +CPU= 26.082
  18:55:48 STEP 1346 H= 336:30 +CPU= 26.715
  18:56:17 STEP 1347 H= 336:45 +CPU= 27.189
  18:56:45 STEP 1348 H= 337:00 +CPU= 27.036
  18:57:11 STEP 1349 H= 337:15 +CPU= 25.994
  18:57:39 STEP 1350 H= 337:30 +CPU= 26.394
  18:58:06 STEP 1351 H= 337:45 +CPU= 26.019
  18:58:33 STEP 1352 H= 338:00 +CPU= 26.949
  18:59:00 STEP 1353 H= 338:15 +CPU= 25.806
  18:59:26 STEP 1354 H= 338:30 +CPU= 25.115
  18:59:52 STEP 1355 H= 338:45 +CPU= 25.471
  19:00:24 STEP 1356 H= 339:00 +CPU= 31.554
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001356
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001356
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001356
  19:00:50 STEP 1357 H= 339:15 +CPU= 25.773
  19:01:17 STEP 1358 H= 339:30 +CPU= 25.761
  19:01:43 STEP 1359 H= 339:45 +CPU= 25.823
  19:02:10 STEP 1360 H= 340:00 +CPU= 26.182
  19:02:36 STEP 1361 H= 340:15 +CPU= 25.672
  19:02:59 STEP 1362 H= 340:30 +CPU= 22.353
  19:03:24 STEP 1363 H= 340:45 +CPU= 25.022
  19:03:52 STEP 1364 H= 341:00 +CPU= 26.434
  19:04:18 STEP 1365 H= 341:15 +CPU= 25.847
  19:04:44 STEP 1366 H= 341:30 +CPU= 25.481
  19:05:10 STEP 1367 H= 341:45 +CPU= 25.534
  19:05:42 STEP 1368 H= 342:00 +CPU= 31.114
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001368
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001368
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001368
  19:06:08 STEP 1369 H= 342:15 +CPU= 25.161
  19:06:34 STEP 1370 H= 342:30 +CPU= 25.135
  19:07:00 STEP 1371 H= 342:45 +CPU= 25.541
  19:07:27 STEP 1372 H= 343:00 +CPU= 26.320
  19:07:54 STEP 1373 H= 343:15 +CPU= 25.979
  19:08:20 STEP 1374 H= 343:30 +CPU= 25.857
  19:08:46 STEP 1375 H= 343:45 +CPU= 25.554
  19:09:13 STEP 1376 H= 344:00 +CPU= 25.938
  19:09:39 STEP 1377 H= 344:15 +CPU= 25.069
  19:10:05 STEP 1378 H= 344:30 +CPU= 25.751
  19:10:31 STEP 1379 H= 344:45 +CPU= 25.534
  19:11:04 STEP 1380 H= 345:00 +CPU= 31.492
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001380
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001380
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001380
  19:11:30 STEP 1381 H= 345:15 +CPU= 25.965
  19:11:57 STEP 1382 H= 345:30 +CPU= 25.640
  19:12:22 STEP 1383 H= 345:45 +CPU= 25.201
  19:12:49 STEP 1384 H= 346:00 +CPU= 26.199
  19:13:16 STEP 1385 H= 346:15 +CPU= 25.997
  19:13:43 STEP 1386 H= 346:30 +CPU= 26.105
  19:14:09 STEP 1387 H= 346:45 +CPU= 25.855
  19:14:36 STEP 1388 H= 347:00 +CPU= 26.551
  19:15:03 STEP 1389 H= 347:15 +CPU= 25.748
  19:15:57 STEP 1390 H= 347:30 +CPU= 34.558
  19:16:23 STEP 1391 H= 347:45 +CPU= 25.041
  19:16:59 STEP 1392 H= 348:00 +CPU= 34.912
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001392
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001392
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001392
  19:17:26 STEP 1393 H= 348:15 +CPU= 25.908
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
[EC_DRHOOK:Dinger4712:1:1:71393:71393] [20240608:191934:1717867174:36879.065] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 4411MB (heap), 5023MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:Dinger4712:1:1:71393:71393] [20240608:191934:1717867174:36879.069] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:Dinger4712:1:1:71393:71393] [20240608:191934:1717867174:36879.070] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x820890 (old at (nil))
[EC_DRHOOK:Dinger4712:1:1:71393:71393] [20240608:191934:1717867174:36879.070] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1
[EC_DRHOOK:Dinger4712:1:1:71393:71393] [20240608:191934:1717867174:36879.073] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1
[EC_DRHOOK:Dinger4712:1:1:71393:71393] [20240608:191934:1717867174:36879.073] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3843] 4411 MB (maxheap), 5023 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:Dinger4712:1:1:71393:71393] [20240608:191934:1717867174:36879.077] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:Dinger4712:1:1:71393:71393] [20240608:191934:1717867174:36879.077] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:Dinger4712:1:1:71393:71393] [20240608:191934:1717867174:36879.078] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:Dinger4712:1:1:71393:71393] [20240608:191934:1717867174:36879.078] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:Dinger4712:1:1:71393:71393] [20240608:191934:1717867174:36879.078] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:Dinger4712:1:1:71393:71393] [20240608:191934:1717867174:36879.078] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :       CNT4 
[EC_DRHOOK:Dinger4712:1:1:71393:71393] [20240608:191934:1717867174:36879.078] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :        STEPO 
[EC_DRHOOK:Dinger4712:1:1:71393:71393] [20240608:191934:1717867174:36879.078] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :         SCAN2M 
[EC_DRHOOK:Dinger4712:1:1:71393:71393] [20240608:191934:1717867174:36879.078] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :          GP_MODEL_HEAP 
[EC_DRHOOK:Dinger4712:1:1:71393:71393] [20240608:191934:1717867174:36879.078] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :           GP_MODEL 
[EC_DRHOOK:Dinger4712:1:1:71393:71393] [20240608:191934:1717867174:36879.078] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :            EC_PHYS_DRV 
[EC_DRHOOK:Dinger4712:1:1:71393:71393] [20240608:191934:1717867174:36879.078] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :             >OMP-PHYSICS              (1001) 
[EC_DRHOOK:Dinger4712:1:1:71393:71393] [20240608:191934:1717867174:36879.078] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :              EC_PHYS 
[EC_DRHOOK:Dinger4712:1:1:71393:71393] [20240608:191934:1717867174:36879.078] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :               CALLPAR 
[EC_DRHOOK:Dinger4712:1:1:71393:71393] [20240608:191934:1717867174:36879.078] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :                TURBULENCE_LAYER 
[EC_DRHOOK:Dinger4712:1:1:71393:71393] [20240608:191934:1717867174:36879.078] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :                 VDFOUTER 
[EC_DRHOOK:Dinger4712:1:1:71393:71393] [20240608:191934:1717867174:36879.078] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :                  VDFMAIN 
[EC_DRHOOK:Dinger4712:1:1:71393:71393] [20240608:191934:1717867174:36879.078] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :                   VDFHGHTN 
[EC_DRHOOK:Dinger4712:1:1:71393:71393] [20240608:191935:1717867175:36880.078] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1
[EC_DRHOOK:Dinger4712:1:1:71393:71393] [20240608:191935:1717867175:36880.078] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1cf8da0 for signal#15, nsigs = 1
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source             
oifs_43r3_model.e  0000000001CF935B  Unknown               Unknown  Unknown
oifs_43r3_model.e  00000000008214C1  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DD5490  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001232263  vdfhghtn_                 658  vdfhghtn.F90
oifs_43r3_model.e  0000000001220C63  vdfmain_                  971  vdfmain.F90
oifs_43r3_model.e  0000000001207E29  vdfouter_                 739  vdfouter.F90
oifs_43r3_model.e  00000000011FDCB2  turbulence_layer_         158  turbulence_layer.F90
oifs_43r3_model.e  0000000000FAC003  callpar_                  869  callpar.F90
oifs_43r3_model.e  0000000000F5FA6B  ec_phys_                  670  ec_phys.F90
oifs_43r3_model.e  0000000000F4DBC4  ec_phys_drv_              599  ec_phys_drv.F90
oifs_43r3_model.e  0000000000BD7131  gp_model_                 613  gp_model.F90
oifs_43r3_model.e  00000000012C5002  gp_model_heap_             74  gp_model_heap.F90
oifs_43r3_model.e  0000000000BACBAE  scan2m_                   535  scan2m.F90
oifs_43r3_model.e  0000000000688153  stepo_                    327  stepo.F90
oifs_43r3_model.e  00000000004CD1DC  cnt4_                    1133  cnt4.F90
oifs_43r3_model.e  00000000004B5D85  cnt3_                     267  cnt3.F90
oifs_43r3_model.e  00000000004B4F52  cnt2_                      88  cnt2.F90
oifs_43r3_model.e  000000000049D638  cnt1_                      92  cnt1.F90
oifs_43r3_model.e  0000000000406FBD  cnt0_                     146  cnt0.F90
oifs_43r3_model.e  0000000000401E7F  MAIN__                     96  master.F90
oifs_43r3_model.e  0000000000401E12  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCC34A  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCDBE7  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000000401CD5  Unknown               Unknown  Unknown
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.13_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a1oc
(argv4) batchid: 1017
(argv5) wuid: 12283999
(argv6) fclen: 20
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/5
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.13
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/5/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/5/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_e84e11518a9509a9e097a74760b94840 to: /var/lib/boinc-client/slots/5/oifs_43r3_bl_a1oc_2016092300_20_1017_12283999.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/5/oifs_43r3_bl_a1oc_2016092300_20_1017_12283999.zip
ic_ancil_file: ic_ancil_12283999
ifsdata_file: ifsdata_12283999
climate_data_file: clim_data_12283999
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /var/lib/boinc-client/slots/5/ic_ancil_12283999.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/5/ic_ancil_12283999.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_40fc044db8ef0fbbea240b0842e506cc to: /var/lib/boinc-client/slots/5/ifsdata/ifsdata_12283999.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/5/ifsdata/ifsdata_12283999.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/5/159l_2/clim_data_12283999.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/5/159l_2/clim_data_12283999.zip
Checking for progress XML file: /var/lib/boinc-client/slots/5/progress_file_12283999.xml
Opened progress file ok : /var/lib/boinc-client/slots/5/progress_file_12283999.xml
-- Model is restarting --
Adjusting last_iter, 1394, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/5/progress_file_12283999.xml
last_cpu_time: 35858
upload_file_number: 14
last_iter: 1393
last_upload: 1209600
model_completed: 0
total_length_of_simulation: 1728000
result_base_name: oifs_43r3_bl_a1oc_2016092300_20_1017_12283999_0_r542734340
The child process has been launched with process id: 3230
Executing the command: /var/lib/boinc-client/slots/5/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:Dinger4712:1:1:3230:3230] [20240608:201459:1717870499:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2091] fp = 0x314f440
[EC_DRHOOK:Dinger4712:1:1:3230:3230] [20240608:201459:1717870499:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:Dinger4712:1:1:3230:3230] [20240608:201459:1717870499:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:Dinger4712:1:1:3230:3230] [20240608:201459:1717870499:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:Dinger4712:1:1:3230:3230] [20240608:201459:1717870499:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:Dinger4712:1:1:3230:3230] [20240608:201459:1717870499:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:Dinger4712:1:1:3230:3230] [20240608:201459:1717870499:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:Dinger4712:1:1:3230:3230] [20240608:201459:1717870499:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:Dinger4712:1:1:3230:3230] [20240608:201459:1717870499:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:Dinger4712:1:1:3230:3230] [20240608:201459:1717870499:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:Dinger4712:1:1:3230:3230] [20240608:201459:1717870499:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:Dinger4712:1:1:3230:3230] [20240608:201459:1717870499:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:Dinger4712:1:1:3230:3230] [20240608:201459:1717870499:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:Dinger4712:1:1:3230:3230] [20240608:201459:1717870499:0.000] [ignore_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:Dinger4712:1:1:3230:3230] [20240608:201459:1717870499:0.000] [restore_default_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:Dinger4712:1:1:3230:3230] [20240608:201459:1717870499:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Dinger4712:1:1:3230:3230] [20240608:201459:1717870499:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Dinger4712:1:1:3230:3230] [20240608:201459:1717870499:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Dinger4712:1:1:3230:3230] [20240608:201459:1717870499:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Dinger4712:1:1:3230:3230] [20240608:201459:1717870499:0.000] [trapfpe_treatment@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:Dinger4712:1:1:3230:3230] [20240608:201459:1717870499:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Dinger4712:1:1:3230:3230] [20240608:201459:1717870499:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Dinger4712:1:1:3230:3230] [20240608:201459:1717870499:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Dinger4712:1:1:3230:3230] [20240608:201459:1717870499:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Dinger4712:1:1:3230:3230] [20240608:201459:1717870499:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Dinger4712:1:1:3230:3230] [20240608:201459:1717870499:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Dinger4712:1:1:3230:3230] [20240608:201459:1717870499:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Dinger4712:1:1:3230:3230] [20240608:201459:1717870499:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Dinger4712:1:1:3230:3230] [20240608:201459:1717870499:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Dinger4712:1:1:3230:3230] [20240608:201459:1717870499:0.000] [catch_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  20:15:29 STEP 1392 H= 348:00 +CPU= 24.474
  20:15:52 STEP 1393 H= 348:15 +CPU= 22.671
  20:16:14 STEP 1394 H= 348:30 +CPU= 21.599
  20:16:35 STEP 1395 H= 348:45 +CPU= 21.047
  20:16:55 STEP 1396 H= 349:00 +CPU= 19.887
  20:17:14 STEP 1397 H= 349:15 +CPU= 19.321
  20:17:35 STEP 1398 H= 349:30 +CPU= 20.501
  20:17:55 STEP 1399 H= 349:45 +CPU= 19.909
  20:18:15 STEP 1400 H= 350:00 +CPU= 19.772
  20:18:36 STEP 1401 H= 350:15 +CPU= 20.924
  20:18:57 STEP 1402 H= 350:30 +CPU= 20.959
  20:19:17 STEP 1403 H= 350:45 +CPU= 20.165
  20:19:42 STEP 1404 H= 351:00 +CPU= 24.498
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001404
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001404
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001404
  20:20:02 STEP 1405 H= 351:15 +CPU= 19.946
  20:20:23 STEP 1406 H= 351:30 +CPU= 20.663
  20:20:43 STEP 1407 H= 351:45 +CPU= 20.367
  20:21:05 STEP 1408 H= 352:00 +CPU= 21.572
  20:21:27 STEP 1409 H= 352:15 +CPU= 21.779
  20:21:48 STEP 1410 H= 352:30 +CPU= 21.034
  20:22:10 STEP 1411 H= 352:45 +CPU= 21.253
  20:22:32 STEP 1412 H= 353:00 +CPU= 21.367
  20:22:52 STEP 1413 H= 353:15 +CPU= 20.568
  20:23:15 STEP 1414 H= 353:30 +CPU= 22.356
  20:23:39 STEP 1415 H= 353:45 +CPU= 22.759
  20:24:07 STEP 1416 H= 354:00 +CPU= 27.927
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001416
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001416
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001416
  20:24:30 STEP 1417 H= 354:15 +CPU= 22.616
  20:24:53 STEP 1418 H= 354:30 +CPU= 22.305
  20:25:16 STEP 1419 H= 354:45 +CPU= 22.456
  20:25:40 STEP 1420 H= 355:00 +CPU= 23.243
  20:26:03 STEP 1421 H= 355:15 +CPU= 23.095
  20:26:26 STEP 1422 H= 355:30 +CPU= 22.495
  20:26:49 STEP 1423 H= 355:45 +CPU= 22.743
  20:27:13 STEP 1424 H= 356:00 +CPU= 23.308
  20:27:36 STEP 1425 H= 356:15 +CPU= 22.782
  20:27:59 STEP 1426 H= 356:30 +CPU= 22.559
  20:28:23 STEP 1427 H= 356:45 +CPU= 22.843
  20:28:51 STEP 1428 H= 357:00 +CPU= 27.882
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001428
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001428
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001428
  20:29:14 STEP 1429 H= 357:15 +CPU= 22.374
  20:29:37 STEP 1430 H= 357:30 +CPU= 22.594
  20:30:00 STEP 1431 H= 357:45 +CPU= 22.440
  20:30:23 STEP 1432 H= 358:00 +CPU= 23.109
  20:30:46 STEP 1433 H= 358:15 +CPU= 22.500
  20:31:09 STEP 1434 H= 358:30 +CPU= 22.659
  20:31:33 STEP 1435 H= 358:45 +CPU= 22.625
  20:31:56 STEP 1436 H= 359:00 +CPU= 23.180
  20:32:19 STEP 1437 H= 359:15 +CPU= 22.428
  20:32:42 STEP 1438 H= 359:30 +CPU= 22.832
  20:33:05 STEP 1439 H= 359:45 +CPU= 22.327
  20:33:35 STEP 1440 H= 360:00 +CPU= 29.598
..The child process terminated with status: 0
>>> Printing last 70 lines from file: NODE.001_01

 
DDH------------------------------------------------------------------------
I                                                                         I
I TIME =  360.000H DDH * Diagnostic Var.,     AREA                GLOBAL  I
I                                                                         I
I-------------------------------------------------------------------------I
I   VARIABLE   I      Initial value         I        Current value        I
I-------------------------------------------------------------------------I
I  OMEGA (P/S) I     -0.6655317937E-21      I       0.3707232169E-05      I
ICloud fractionI      0.0000000000E+00      I       0.2598311446E-01      I
I  Relat. Hum. I      0.1360828022E+00      I       0.2675130922E+00      I
I  PASS 01     I      0.0000000000E+00      I       0.4641523736E+03      I
DDH------------------------------------------------------------------------
  
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1440 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001440 MODE=a
 IO-STREAM CLOSED - ICMGGh7zg+001440
 MPI-TASK:   1 -    319930468 BYTES IN      7 RECORDS TRANSFERRED IN    0.0008 SECONDS399913.1 Mbytes/s, TOTAL TIME=    0.0124( 6.5%)
 20:33:35 STEP 1440 H= 360:00 +CPU= 29.598
 END CNT3
 NSTEP =  1440 CNT0   000000000  21.846  21.846  22.263   0.981               0               0               0               0               0
IO-STREAM STATISTICS, TOTAL NO OF REC/BYTES READ           46/  1069125708 WRITTEN         1430/************
 -TASK-OPENED-OPEN-RECS IN -KBYTE IN    -RECS OUT-KBYTE OUT-WALL   -WALL IN-WALL
  OU-TOT IN -TOT OUT
   1     28    0       46      1044068     1430 ************     5.6     3.0     2.6     3.8     4.7
===-=== START OF TIMING STATISTICS ===-===
 
STATS FOR ALL TASKS
 NUM ROUTINE                                     CALLS  MEAN(ms)   MAX(ms)   FRAC(%)  UNBAL(%)
   0 CNT0     - COMPLETE EXECUTION                   1 1115955.6 1115955.6    100.00      0.00
   1 CNT4     - FORWARD INTEGRATION                  1 1110712.9 1110712.9     99.53      0.00
   8 SCAN2M - GRID-POINT DYNAMICS                   53    3923.7    3923.7     18.63      0.00
   9 SPCM     - SPECTRAL COMP.                      48     627.1     627.1      2.70      0.00
  10 SCAN2M - PHYSICS                               49   10674.7   10674.7     46.87      0.00
  11 IOPACK   - OUTPUT P.P. RESULTS                  4     827.9     827.9      0.30      0.00
  12 SPNORM   - SPECTRAL NORM COMP.                  3      46.1      46.1      0.01      0.00
  17 GRIDFPOS IN CNT4                                4      22.9      22.9      0.01      0.00
  27 RESTART FILES - WRITING                         1    3310.4    3310.4      0.30      0.00
  28 RESTART FILES - READING                         1    4693.2    4693.2      0.42      0.00
  29 SU4FPOS IN CNT4                                 4       3.5       3.5      0.00      0.00
  30 DYNFPOS IN CNT4                                 4    4520.5    4520.5      1.62      0.00
  31 POSDDH IN STEPO                                 1      16.4      16.4      0.00      0.00
  37 CPGLAG   - SL COMPUTATIONS                     49    4669.5    4669.5     20.50      0.00
  39 SU0YOMB                                         1     349.3     349.3      0.03      0.00
  51 SCAN2M   - SL COMM. PART 1                     49      19.5      19.5      0.09      0.00
  54 SPCM     - M TO S/S TO M TRANSP.               48     328.3     328.3      1.41      0.00
  55 SPCIMPF  - S TO M/M TO S TRANSP.               48      52.6      52.6      0.23      0.00
  56 SPNORM   - SPECTRAL NORM COMM.                  3       0.2       0.2      0.00      0.00
 102 LTINV_CTL   - INVERSE LEGENDRE TRANSFORM      108     287.8     287.8      2.79      0.00
 103 LTDIR_CTL   - DIRECT LEGENDRE TRANSFORM       100     161.3     161.3      1.45      0.00
 106 FTDIR_CTL   - DIRECT FOURIER TRANSFORM        100      58.2      58.2      0.52      0.00
 107 FTINV_CTL   - INVERSE FOURIER TRANSFORM       108      83.3      83.3      0.81      0.00
 140 SULEG       - COMP. OF LEGENDRE POL.            1      32.8      32.8      0.00      0.00
 152 LTINV_CTL   - M TO L TRANSPOSITION            108      56.9      56.9      0.55      0.00
 153 LTDIR_CTL   - L TO M TRANSPOSITION            100      30.0      30.0      0.27      0.00
 157 FTINV_CTL   - L TO G TRANSPOSITION            108      57.2      57.2      0.55      0.00
 158 FTDIR_CTL   - G TO L TRANSPOSITION            100      33.6      33.6      0.30      0.00
 400 GSTATS                                       6667       0.0       0.0      0.00      0.00
 401 GSTATS HOOK                                  6149       0.0       0.0      0.00      0.00
TOTAL MEASURED IMBALANCE =       0.0 SECONDS,  0.0 PERCENT
TOTAL WALLCLOCK TIME     1116.0 CPU TIME    1098.3 VECTOR TIME     1098.3


===-=== END   OF TIMING STATISTICS ===-===
FORECAST DAYS PER DAY     1166.8
  *** END CNT0 *** 
------------------------------------------------
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001440
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001440
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001440
Adding to the zip: /var/lib/boinc-client/slots/5/NODE.001_01
Adding to the zip: /var/lib/boinc-client/slots/5/ifs.stat
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMGGh7zg+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMUAh7zg+001428
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMSHh7zg+001380
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMSHh7zg+001404
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMSHh7zg+001428
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMSHh7zg+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMGGh7zg+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMGGh7zg+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMSHh7zg+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMUAh7zg+001404
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMUAh7zg+001380
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMGGh7zg+001404
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMUAh7zg+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMSHh7zg+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMGGh7zg+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMUAh7zg+001356
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMUAh7zg+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMUAh7zg+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMGGh7zg+001356
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMSHh7zg+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMSHh7zg+001356
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMGGh7zg+001380
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMGGh7zg+001428
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMUAh7zg+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMSHh7zg+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMUAh7zg+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283999/ICMGGh7zg+001368
Zipping up the final file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a1oc_2016092300_20_1017_12283999_0_r542734340_14.zip
Uploading the final file: upload_file_14.zip
Uploading trickle at timestep: 1295100
20:37:03 (2831): called boinc_finish(0)

</stderr_txt>
<message>
upload failure: <file_xfer_error>
  <file_name>oifs_43r3_bl_a1oc_2016092300_20_1017_12283999_0_r542734340_15.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a1oc_2016092300_20_1017_12283999_0_r542734340_16.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a1oc_2016092300_20_1017_12283999_0_r542734340_17.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a1oc_2016092300_20_1017_12283999_0_r542734340_18.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a1oc_2016092300_20_1017_12283999_0_r542734340_19.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
</message>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
08 Jun 2024 19:16:38 1532456 22435578 oifs_43r3_bl_a1oc_2016092300_20_1017_12283999_0 1,295,100 36,956 0.0285
08 Jun 2024 14:49:12 1532456 22435578 oifs_43r3_bl_a1oc_2016092300_20_1017_12283999_0 864,000 24,547 0.0284


©2024 climateprediction.net