climateprediction.net home page
Task 22435464

Task 22435464

Name oifs_43r3_bl_a1l6_2016092300_20_1017_12283885_0
Workunit 12283885
Created 7 Jun 2024, 13:49:28 UTC
Sent 8 Jun 2024, 5:45:33 UTC
Report deadline 7 Aug 2024, 5:45:33 UTC
Received 8 Jun 2024, 22:38:07 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 0 (0x00000000)
Computer ID 1517679
Run time 16 hours 28 min 17 sec
CPU time 14 hours 20 min 30 sec
Validate state Invalid
Credit 1,976.32
Device peak FLOPS 5.26 GFLOPS
Application version OpenIFS 43r3 Baroclinic Lifecycle v1.13
x86_64-pc-linux-gnu
Peak working set size 5,025.44 MB
Peak swap size 5,980.80 MB
Peak disk usage 1,346.47 MB
Stderr
<core_client_version>7.18.1</core_client_version>
<![CDATA[
<stderr_txt>
5 STEP 1035 H= 258:45 +CPU= 34.408
  11:11:23 STEP 1036 H= 259:00 +CPU= 34.943
  11:12:00 STEP 1037 H= 259:15 +CPU= 34.126
  11:12:38 STEP 1038 H= 259:30 +CPU= 34.094
  11:13:16 STEP 1039 H= 259:45 +CPU= 34.166
  11:13:54 STEP 1040 H= 260:00 +CPU= 35.084
  11:14:31 STEP 1041 H= 260:15 +CPU= 34.655
  11:15:08 STEP 1042 H= 260:30 +CPU= 33.783
  11:15:46 STEP 1043 H= 260:45 +CPU= 34.180
  11:16:33 STEP 1044 H= 261:00 +CPU= 42.468
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMGGh7zg+001044
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMSHh7zg+001044
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMUAh7zg+001044
  11:17:10 STEP 1045 H= 261:15 +CPU= 33.294
  11:17:47 STEP 1046 H= 261:30 +CPU= 33.732
  11:18:22 STEP 1047 H= 261:45 +CPU= 33.506
  11:18:58 STEP 1048 H= 262:00 +CPU= 33.843
  11:19:34 STEP 1049 H= 262:15 +CPU= 33.522
  11:20:10 STEP 1050 H= 262:30 +CPU= 33.717
  11:20:46 STEP 1051 H= 262:45 +CPU= 34.367
  11:21:24 STEP 1052 H= 263:00 +CPU= 34.480
  11:22:00 STEP 1053 H= 263:15 +CPU= 34.056
  11:22:37 STEP 1054 H= 263:30 +CPU= 33.858
  11:23:12 STEP 1055 H= 263:45 +CPU= 33.117
  11:23:59 STEP 1056 H= 264:00 +CPU= 43.171
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMGGh7zg+001056
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMSHh7zg+001056
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMUAh7zg+001056
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMGGh7zg+000960
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMSHh7zg+000960
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMUAh7zg+000960
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMGGh7zg+000972
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMSHh7zg+000972
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMUAh7zg+000972
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMGGh7zg+000984
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMSHh7zg+000984
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMUAh7zg+000984
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMGGh7zg+000996
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMSHh7zg+000996
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMUAh7zg+000996
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMGGh7zg+001008
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMSHh7zg+001008
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMUAh7zg+001008
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMGGh7zg+001020
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMSHh7zg+001020
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMUAh7zg+001020
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMGGh7zg+001032
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMSHh7zg+001032
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMUAh7zg+001032
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMGGh7zg+001044
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMSHh7zg+001044
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMUAh7zg+001044
Zipping up the intermediate file: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a1l6_2016092300_20_1017_12283885_0_r1611978764_10.zip
Uploading the intermediate file: upload_file_10.zip
  11:24:34 STEP 1057 H= 264:15 +CPU= 32.826
  11:25:09 STEP 1058 H= 264:30 +CPU= 32.643
  11:25:45 STEP 1059 H= 264:45 +CPU= 33.098
  11:26:21 STEP 1060 H= 265:00 +CPU= 32.940
  11:26:57 STEP 1061 H= 265:15 +CPU= 31.578
  11:27:32 STEP 1062 H= 265:30 +CPU= 32.188
  11:28:08 STEP 1063 H= 265:45 +CPU= 32.809
  11:28:47 STEP 1064 H= 266:00 +CPU= 33.498
  11:29:23 STEP 1065 H= 266:15 +CPU= 32.207
  11:30:00 STEP 1066 H= 266:30 +CPU= 33.126
  11:30:36 STEP 1067 H= 266:45 +CPU= 32.317
  11:31:22 STEP 1068 H= 267:00 +CPU= 39.917
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMGGh7zg+001068
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMSHh7zg+001068
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMUAh7zg+001068
  11:31:59 STEP 1069 H= 267:15 +CPU= 32.382
  11:32:34 STEP 1070 H= 267:30 +CPU= 31.762
  11:33:10 STEP 1071 H= 267:45 +CPU= 32.230
  11:33:46 STEP 1072 H= 268:00 +CPU= 32.147
  11:34:24 STEP 1073 H= 268:15 +CPU= 32.841
  11:34:59 STEP 1074 H= 268:30 +CPU= 32.562
  11:35:37 STEP 1075 H= 268:45 +CPU= 32.932
  11:36:14 STEP 1076 H= 269:00 +CPU= 33.515
  11:36:50 STEP 1077 H= 269:15 +CPU= 33.031
  11:37:27 STEP 1078 H= 269:30 +CPU= 33.270
  11:38:03 STEP 1079 H= 269:45 +CPU= 32.622
  11:38:48 STEP 1080 H= 270:00 +CPU= 40.180
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMGGh7zg+001080
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMSHh7zg+001080
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMUAh7zg+001080
  11:39:23 STEP 1081 H= 270:15 +CPU= 31.667
  11:39:59 STEP 1082 H= 270:30 +CPU= 31.655
  11:40:35 STEP 1083 H= 270:45 +CPU= 31.770
  11:41:12 STEP 1084 H= 271:00 +CPU= 33.170
  11:41:48 STEP 1085 H= 271:15 +CPU= 31.769
  11:42:26 STEP 1086 H= 271:30 +CPU= 32.298
  11:43:01 STEP 1087 H= 271:45 +CPU= 31.652
  11:43:38 STEP 1088 H= 272:00 +CPU= 32.731
  11:44:13 STEP 1089 H= 272:15 +CPU= 32.304
  11:44:50 STEP 1090 H= 272:30 +CPU= 31.951
  11:45:26 STEP 1091 H= 272:45 +CPU= 31.986
  11:46:10 STEP 1092 H= 273:00 +CPU= 40.746
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMGGh7zg+001092
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMSHh7zg+001092
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMUAh7zg+001092
  11:46:47 STEP 1093 H= 273:15 +CPU= 33.494
  11:47:23 STEP 1094 H= 273:30 +CPU= 32.688
  11:48:00 STEP 1095 H= 273:45 +CPU= 32.908
  11:48:38 STEP 1096 H= 274:00 +CPU= 33.979
  11:49:14 STEP 1097 H= 274:15 +CPU= 32.161
  11:49:50 STEP 1098 H= 274:30 +CPU= 32.638
  11:50:27 STEP 1099 H= 274:45 +CPU= 32.992
  11:51:04 STEP 1100 H= 275:00 +CPU= 34.420
  11:51:41 STEP 1101 H= 275:15 +CPU= 32.670
  11:52:16 STEP 1102 H= 275:30 +CPU= 32.813
  11:52:54 STEP 1103 H= 275:45 +CPU= 33.406
  11:53:43 STEP 1104 H= 276:00 +CPU= 43.968
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMGGh7zg+001104
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMSHh7zg+001104
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMUAh7zg+001104
  11:54:20 STEP 1105 H= 276:15 +CPU= 32.844
  11:54:56 STEP 1106 H= 276:30 +CPU= 32.915
  11:55:34 STEP 1107 H= 276:45 +CPU= 32.820
  11:56:13 STEP 1108 H= 277:00 +CPU= 33.852
  11:56:50 STEP 1109 H= 277:15 +CPU= 32.896
  11:57:27 STEP 1110 H= 277:30 +CPU= 32.503
  11:58:04 STEP 1111 H= 277:45 +CPU= 33.499
  11:58:40 STEP 1112 H= 278:00 +CPU= 33.963
  11:59:16 STEP 1113 H= 278:15 +CPU= 33.781
  11:59:52 STEP 1114 H= 278:30 +CPU= 33.568
  12:00:29 STEP 1115 H= 278:45 +CPU= 33.590
  12:01:13 STEP 1116 H= 279:00 +CPU= 41.292
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMGGh7zg+001116
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMSHh7zg+001116
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMUAh7zg+001116
  12:01:49 STEP 1117 H= 279:15 +CPU= 33.543
  12:02:25 STEP 1118 H= 279:30 +CPU= 33.551
  12:03:00 STEP 1119 H= 279:45 +CPU= 33.660
  12:03:37 STEP 1120 H= 280:00 +CPU= 34.975
  12:04:15 STEP 1121 H= 280:15 +CPU= 34.701
  12:04:51 STEP 1122 H= 280:30 +CPU= 33.658
  12:05:27 STEP 1123 H= 280:45 +CPU= 33.514
  12:06:04 STEP 1124 H= 281:00 +CPU= 34.471
  12:06:40 STEP 1125 H= 281:15 +CPU= 34.260
  12:07:16 STEP 1126 H= 281:30 +CPU= 34.046
  12:07:53 STEP 1127 H= 281:45 +CPU= 33.400
  12:08:38 STEP 1128 H= 282:00 +CPU= 40.916
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMGGh7zg+001128
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMSHh7zg+001128
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMUAh7zg+001128
  12:09:14 STEP 1129 H= 282:15 +CPU= 33.440
  12:09:50 STEP 1130 H= 282:30 +CPU= 33.133
  12:10:26 STEP 1131 H= 282:45 +CPU= 33.726
  12:11:03 STEP 1132 H= 283:00 +CPU= 34.006
  12:11:39 STEP 1133 H= 283:15 +CPU= 33.094
  12:12:15 STEP 1134 H= 283:30 +CPU= 33.348
  12:12:51 STEP 1135 H= 283:45 +CPU= 33.506
  12:13:27 STEP 1136 H= 284:00 +CPU= 34.129
  12:14:04 STEP 1137 H= 284:15 +CPU= 32.944
  12:14:42 STEP 1138 H= 284:30 +CPU= 32.714
  12:15:21 STEP 1139 H= 284:45 +CPU= 32.890
  12:16:06 STEP 1140 H= 285:00 +CPU= 40.298
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMGGh7zg+001140
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMSHh7zg+001140
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMUAh7zg+001140
  12:16:44 STEP 1141 H= 285:15 +CPU= 32.628
  12:17:21 STEP 1142 H= 285:30 +CPU= 32.346
  12:17:57 STEP 1143 H= 285:45 +CPU= 33.898
  12:18:33 STEP 1144 H= 286:00 +CPU= 33.729
  12:19:09 STEP 1145 H= 286:15 +CPU= 33.798
  12:19:44 STEP 1146 H= 286:30 +CPU= 33.583
  12:20:19 STEP 1147 H= 286:45 +CPU= 33.471
  12:20:55 STEP 1148 H= 287:00 +CPU= 34.724
  12:21:30 STEP 1149 H= 287:15 +CPU= 33.729
  12:22:06 STEP 1150 H= 287:30 +CPU= 33.510
  12:22:41 STEP 1151 H= 287:45 +CPU= 34.000
  12:23:29 STEP 1152 H= 288:00 +CPU= 45.572
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMGGh7zg+001152
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMSHh7zg+001152
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMUAh7zg+001152
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMGGh7zg+001056
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMSHh7zg+001056
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMUAh7zg+001056
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMGGh7zg+001068
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMSHh7zg+001068
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMUAh7zg+001068
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMGGh7zg+001080
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMSHh7zg+001080
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMUAh7zg+001080
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMGGh7zg+001092
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMSHh7zg+001092
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMUAh7zg+001092
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMGGh7zg+001104
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMSHh7zg+001104
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMUAh7zg+001104
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMGGh7zg+001116
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMSHh7zg+001116
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMUAh7zg+001116
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMGGh7zg+001128
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMSHh7zg+001128
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMUAh7zg+001128
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMGGh7zg+001140
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMSHh7zg+001140
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMUAh7zg+001140
Zipping up the intermediate file: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a1l6_2016092300_20_1017_12283885_0_r1611978764_11.zip
Uploading the intermediate file: upload_file_11.zip
  12:24:05 STEP 1153 H= 288:15 +CPU= 33.451
  12:24:40 STEP 1154 H= 288:30 +CPU= 33.518
  12:25:15 STEP 1155 H= 288:45 +CPU= 33.475
  12:25:51 STEP 1156 H= 289:00 +CPU= 34.291
  12:26:28 STEP 1157 H= 289:15 +CPU= 33.746
  12:27:04 STEP 1158 H= 289:30 +CPU= 33.072
  12:27:41 STEP 1159 H= 289:45 +CPU= 32.809
  12:28:18 STEP 1160 H= 290:00 +CPU= 33.690
  12:28:56 STEP 1161 H= 290:15 +CPU= 32.864
  12:29:32 STEP 1162 H= 290:30 +CPU= 33.076
  12:30:09 STEP 1163 H= 290:45 +CPU= 33.270
  12:30:54 STEP 1164 H= 291:00 +CPU= 41.021
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMGGh7zg+001164
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMSHh7zg+001164
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMUAh7zg+001164
  12:31:30 STEP 1165 H= 291:15 +CPU= 33.451
  12:32:06 STEP 1166 H= 291:30 +CPU= 32.986
  12:32:43 STEP 1167 H= 291:45 +CPU= 33.146
  12:33:20 STEP 1168 H= 292:00 +CPU= 33.604
  12:33:57 STEP 1169 H= 292:15 +CPU= 33.135
  12:34:33 STEP 1170 H= 292:30 +CPU= 34.115
  12:35:08 STEP 1171 H= 292:45 +CPU= 33.613
  12:35:45 STEP 1172 H= 293:00 +CPU= 34.056
  12:36:21 STEP 1173 H= 293:15 +CPU= 34.104
  12:36:56 STEP 1174 H= 293:30 +CPU= 33.871
  12:37:32 STEP 1175 H= 293:45 +CPU= 33.990
  12:38:16 STEP 1176 H= 294:00 +CPU= 41.287
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMGGh7zg+001176
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMSHh7zg+001176
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMUAh7zg+001176
  12:38:51 STEP 1177 H= 294:15 +CPU= 32.986
  12:39:27 STEP 1178 H= 294:30 +CPU= 33.852
  12:40:02 STEP 1179 H= 294:45 +CPU= 33.469
  12:40:39 STEP 1180 H= 295:00 +CPU= 34.743
  12:41:14 STEP 1181 H= 295:15 +CPU= 33.665
  12:41:50 STEP 1182 H= 295:30 +CPU= 33.841
  12:42:25 STEP 1183 H= 295:45 +CPU= 33.996
  12:43:02 STEP 1184 H= 296:00 +CPU= 34.652
  12:43:37 STEP 1185 H= 296:15 +CPU= 33.464
  12:44:13 STEP 1186 H= 296:30 +CPU= 33.957
  12:44:49 STEP 1187 H= 296:45 +CPU= 33.751
  12:45:32 STEP 1188 H= 297:00 +CPU= 41.275
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMGGh7zg+001188
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMSHh7zg+001188
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMUAh7zg+001188
  12:46:08 STEP 1189 H= 297:15 +CPU= 33.473
  12:46:44 STEP 1190 H= 297:30 +CPU= 33.801
  12:47:20 STEP 1191 H= 297:45 +CPU= 33.630
  12:47:58 STEP 1192 H= 298:00 +CPU= 33.992
  12:48:33 STEP 1193 H= 298:15 +CPU= 33.621
  12:49:09 STEP 1194 H= 298:30 +CPU= 34.071
  12:49:44 STEP 1195 H= 298:45 +CPU= 33.769
  12:50:21 STEP 1196 H= 299:00 +CPU= 34.084
  12:50:57 STEP 1197 H= 299:15 +CPU= 33.641
  12:51:34 STEP 1198 H= 299:30 +CPU= 32.958
  12:52:10 STEP 1199 H= 299:45 +CPU= 33.479
  12:52:59 STEP 1200 H= 300:00 +CPU= 44.978
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMGGh7zg+001200
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMSHh7zg+001200
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMUAh7zg+001200
  12:53:36 STEP 1201 H= 300:15 +CPU= 33.556
  12:54:12 STEP 1202 H= 300:30 +CPU= 33.251
  12:54:47 STEP 1203 H= 300:45 +CPU= 33.827
  12:55:23 STEP 1204 H= 301:00 +CPU= 34.325
  12:56:00 STEP 1205 H= 301:15 +CPU= 33.701
  12:56:36 STEP 1206 H= 301:30 +CPU= 33.142
  12:57:12 STEP 1207 H= 301:45 +CPU= 33.066
  12:57:49 STEP 1208 H= 302:00 +CPU= 34.497
  12:58:25 STEP 1209 H= 302:15 +CPU= 33.457
  12:59:00 STEP 1210 H= 302:30 +CPU= 33.721
  12:59:36 STEP 1211 H= 302:45 +CPU= 33.327
  13:00:20 STEP 1212 H= 303:00 +CPU= 41.090
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMGGh7zg+001212
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMSHh7zg+001212
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMUAh7zg+001212
  13:00:58 STEP 1213 H= 303:15 +CPU= 33.313
  13:01:36 STEP 1214 H= 303:30 +CPU= 33.543
  13:02:14 STEP 1215 H= 303:45 +CPU= 33.714
  13:02:54 STEP 1216 H= 304:00 +CPU= 34.319
  13:03:31 STEP 1217 H= 304:15 +CPU= 34.103
  13:04:08 STEP 1218 H= 304:30 +CPU= 33.615
  13:04:44 STEP 1219 H= 304:45 +CPU= 33.210
  13:05:21 STEP 1220 H= 305:00 +CPU= 34.149
  13:05:56 STEP 1221 H= 305:15 +CPU= 32.953
  13:06:31 STEP 1222 H= 305:30 +CPU= 33.027
  13:07:07 STEP 1223 H= 305:45 +CPU= 34.059
  13:07:53 STEP 1224 H= 306:00 +CPU= 42.166
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMGGh7zg+001224
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMSHh7zg+001224
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMUAh7zg+001224
  13:08:29 STEP 1225 H= 306:15 +CPU= 33.848
  13:09:05 STEP 1226 H= 306:30 +CPU= 33.653
  13:09:41 STEP 1227 H= 306:45 +CPU= 33.725
  13:10:20 STEP 1228 H= 307:00 +CPU= 33.695
  13:10:57 STEP 1229 H= 307:15 +CPU= 32.826
  13:11:35 STEP 1230 H= 307:30 +CPU= 33.198
  13:12:12 STEP 1231 H= 307:45 +CPU= 32.555
  13:12:48 STEP 1232 H= 308:00 +CPU= 33.451
  13:13:24 STEP 1233 H= 308:15 +CPU= 33.329
  13:14:00 STEP 1234 H= 308:30 +CPU= 33.530
  13:14:37 STEP 1235 H= 308:45 +CPU= 32.430
  13:15:23 STEP 1236 H= 309:00 +CPU= 41.512
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMGGh7zg+001236
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMSHh7zg+001236
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMUAh7zg+001236
  13:16:00 STEP 1237 H= 309:15 +CPU= 32.872
  13:16:38 STEP 1238 H= 309:30 +CPU= 32.429
  13:17:13 STEP 1239 H= 309:45 +CPU= 33.110
  13:17:49 STEP 1240 H= 310:00 +CPU= 33.947
  13:18:23 STEP 1241 H= 310:15 +CPU= 33.292
  13:18:58 STEP 1242 H= 310:30 +CPU= 33.060
  13:19:31 STEP 1243 H= 310:45 +CPU= 32.700
  13:20:06 STEP 1244 H= 311:00 +CPU= 33.528
  13:20:42 STEP 1245 H= 311:15 +CPU= 33.089
  13:21:18 STEP 1246 H= 311:30 +CPU= 33.365
  13:21:54 STEP 1247 H= 311:45 +CPU= 33.547
  13:22:41 STEP 1248 H= 312:00 +CPU= 44.172
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMGGh7zg+001248
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMSHh7zg+001248
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMUAh7zg+001248
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMGGh7zg+001152
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMSHh7zg+001152
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMUAh7zg+001152
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMGGh7zg+001164
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMSHh7zg+001164
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMUAh7zg+001164
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMGGh7zg+001176
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMSHh7zg+001176
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMUAh7zg+001176
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMGGh7zg+001188
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMSHh7zg+001188
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMUAh7zg+001188
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMGGh7zg+001200
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMSHh7zg+001200
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMUAh7zg+001200
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMGGh7zg+001212
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMSHh7zg+001212
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMUAh7zg+001212
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMGGh7zg+001224
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMSHh7zg+001224
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMUAh7zg+001224
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMGGh7zg+001236
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMSHh7zg+001236
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMUAh7zg+001236
Zipping up the intermediate file: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a1l6_2016092300_20_1017_12283885_0_r1611978764_12.zip
Uploading the intermediate file: upload_file_12.zip
  13:23:17 STEP 1249 H= 312:15 +CPU= 33.495
  13:23:53 STEP 1250 H= 312:30 +CPU= 33.206
  13:24:28 STEP 1251 H= 312:45 +CPU= 32.562
  13:25:04 STEP 1252 H= 313:00 +CPU= 33.139
  13:25:38 STEP 1253 H= 313:15 +CPU= 32.629
  13:26:14 STEP 1254 H= 313:30 +CPU= 33.305
  13:26:50 STEP 1255 H= 313:45 +CPU= 33.078
  13:27:27 STEP 1256 H= 314:00 +CPU= 33.303
  13:28:02 STEP 1257 H= 314:15 +CPU= 32.847
  13:28:38 STEP 1258 H= 314:30 +CPU= 32.635
  13:29:15 STEP 1259 H= 314:45 +CPU= 33.668
  13:30:00 STEP 1260 H= 315:00 +CPU= 40.622
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMGGh7zg+001260
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMSHh7zg+001260
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMUAh7zg+001260
  13:30:35 STEP 1261 H= 315:15 +CPU= 32.343
  13:31:11 STEP 1262 H= 315:30 +CPU= 32.406
  13:31:46 STEP 1263 H= 315:45 +CPU= 32.177
  13:32:23 STEP 1264 H= 316:00 +CPU= 32.991
  13:32:57 STEP 1265 H= 316:15 +CPU= 32.125
  13:33:31 STEP 1266 H= 316:30 +CPU= 32.158
  13:34:04 STEP 1267 H= 316:45 +CPU= 31.814
  13:34:37 STEP 1268 H= 317:00 +CPU= 32.632
  13:35:10 STEP 1269 H= 317:15 +CPU= 31.244
  13:35:42 STEP 1270 H= 317:30 +CPU= 31.627
  13:36:15 STEP 1271 H= 317:45 +CPU= 31.578
  13:36:57 STEP 1272 H= 318:00 +CPU= 39.218
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMGGh7zg+001272
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMSHh7zg+001272
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMUAh7zg+001272
  13:37:33 STEP 1273 H= 318:15 +CPU= 31.993
  13:38:08 STEP 1274 H= 318:30 +CPU= 31.953
  13:38:43 STEP 1275 H= 318:45 +CPU= 31.656
  13:39:19 STEP 1276 H= 319:00 +CPU= 32.570
  13:39:55 STEP 1277 H= 319:15 +CPU= 32.461
  13:40:31 STEP 1278 H= 319:30 +CPU= 32.148
  13:41:06 STEP 1279 H= 319:45 +CPU= 31.616
  13:41:41 STEP 1280 H= 320:00 +CPU= 31.886
  13:42:15 STEP 1281 H= 320:15 +CPU= 31.286
  13:42:49 STEP 1282 H= 320:30 +CPU= 31.136
  13:43:23 STEP 1283 H= 320:45 +CPU= 31.042
  13:44:05 STEP 1284 H= 321:00 +CPU= 38.021
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMGGh7zg+001284
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMSHh7zg+001284
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMUAh7zg+001284
  13:44:40 STEP 1285 H= 321:15 +CPU= 31.039
  13:45:16 STEP 1286 H= 321:30 +CPU= 32.164
  13:45:52 STEP 1287 H= 321:45 +CPU= 32.492
  13:46:28 STEP 1288 H= 322:00 +CPU= 33.167
  13:47:03 STEP 1289 H= 322:15 +CPU= 32.269
  13:47:40 STEP 1290 H= 322:30 +CPU= 32.661
  13:48:15 STEP 1291 H= 322:45 +CPU= 32.862
  13:48:50 STEP 1292 H= 323:00 +CPU= 32.593
  13:49:25 STEP 1293 H= 323:15 +CPU= 31.864
  13:50:00 STEP 1294 H= 323:30 +CPU= 31.834
  13:50:34 STEP 1295 H= 323:45 +CPU= 31.545
  13:51:21 STEP 1296 H= 324:00 +CPU= 43.444
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMGGh7zg+001296
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMSHh7zg+001296
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMUAh7zg+001296
  13:51:55 STEP 1297 H= 324:15 +CPU= 32.177
  13:52:30 STEP 1298 H= 324:30 +CPU= 32.426
  13:53:04 STEP 1299 H= 324:45 +CPU= 32.145
  13:53:39 STEP 1300 H= 325:00 +CPU= 32.311
  13:54:14 STEP 1301 H= 325:15 +CPU= 32.093
  13:54:48 STEP 1302 H= 325:30 +CPU= 31.338
  13:55:23 STEP 1303 H= 325:45 +CPU= 32.159
  13:55:58 STEP 1304 H= 326:00 +CPU= 32.093
  13:56:32 STEP 1305 H= 326:15 +CPU= 31.602
  13:57:06 STEP 1306 H= 326:30 +CPU= 31.687
  13:57:41 STEP 1307 H= 326:45 +CPU= 32.446
  13:58:24 STEP 1308 H= 327:00 +CPU= 38.947
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMGGh7zg+001308
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMSHh7zg+001308
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMUAh7zg+001308
  13:58:59 STEP 1309 H= 327:15 +CPU= 31.934
  13:59:35 STEP 1310 H= 327:30 +CPU= 31.992
  14:00:10 STEP 1311 H= 327:45 +CPU= 31.564
  14:00:45 STEP 1312 H= 328:00 +CPU= 32.390
  14:01:20 STEP 1313 H= 328:15 +CPU= 31.730
  14:01:55 STEP 1314 H= 328:30 +CPU= 31.535
  14:02:30 STEP 1315 H= 328:45 +CPU= 31.959
  14:03:05 STEP 1316 H= 329:00 +CPU= 32.123
  14:03:40 STEP 1317 H= 329:15 +CPU= 32.221
  14:04:16 STEP 1318 H= 329:30 +CPU= 32.189
  14:04:50 STEP 1319 H= 329:45 +CPU= 31.449
  14:05:34 STEP 1320 H= 330:00 +CPU= 39.847
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMGGh7zg+001320
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMSHh7zg+001320
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMUAh7zg+001320
  14:06:09 STEP 1321 H= 330:15 +CPU= 31.815
  14:06:44 STEP 1322 H= 330:30 +CPU= 32.410
  14:07:19 STEP 1323 H= 330:45 +CPU= 32.560
  14:07:55 STEP 1324 H= 331:00 +CPU= 32.968
  14:08:31 STEP 1325 H= 331:15 +CPU= 32.230
  14:09:06 STEP 1326 H= 331:30 +CPU= 31.718
  14:09:41 STEP 1327 H= 331:45 +CPU= 31.613
  14:10:17 STEP 1328 H= 332:00 +CPU= 32.611
  14:10:52 STEP 1329 H= 332:15 +CPU= 32.375
  14:11:28 STEP 1330 H= 332:30 +CPU= 32.286
  14:12:03 STEP 1331 H= 332:45 +CPU= 31.838
  14:12:47 STEP 1332 H= 333:00 +CPU= 39.645
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMGGh7zg+001332
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMSHh7zg+001332
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMUAh7zg+001332
  14:13:22 STEP 1333 H= 333:15 +CPU= 32.368
  14:13:57 STEP 1334 H= 333:30 +CPU= 32.524
  14:14:32 STEP 1335 H= 333:45 +CPU= 31.889
  14:15:08 STEP 1336 H= 334:00 +CPU= 33.119
  14:15:43 STEP 1337 H= 334:15 +CPU= 32.452
  14:16:18 STEP 1338 H= 334:30 +CPU= 32.204
  14:16:50 STEP 1339 H= 334:45 +CPU= 31.151
  14:17:24 STEP 1340 H= 335:00 +CPU= 32.135
  14:17:58 STEP 1341 H= 335:15 +CPU= 32.109
  14:18:31 STEP 1342 H= 335:30 +CPU= 32.571
  14:19:05 STEP 1343 H= 335:45 +CPU= 32.266
  14:19:50 STEP 1344 H= 336:00 +CPU= 43.197
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMGGh7zg+001344
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMSHh7zg+001344
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMUAh7zg+001344
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMGGh7zg+001248
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMSHh7zg+001248
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMUAh7zg+001248
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMGGh7zg+001260
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMSHh7zg+001260
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMUAh7zg+001260
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMGGh7zg+001272
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMSHh7zg+001272
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMUAh7zg+001272
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMGGh7zg+001284
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMSHh7zg+001284
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMUAh7zg+001284
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMGGh7zg+001296
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMSHh7zg+001296
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMUAh7zg+001296
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMGGh7zg+001308
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMSHh7zg+001308
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMUAh7zg+001308
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMGGh7zg+001320
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMSHh7zg+001320
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMUAh7zg+001320
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMGGh7zg+001332
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMSHh7zg+001332
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMUAh7zg+001332
Zipping up the intermediate file: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a1l6_2016092300_20_1017_12283885_0_r1611978764_13.zip
Uploading the intermediate file: upload_file_13.zip
  14:20:25 STEP 1345 H= 336:15 +CPU= 32.662
  14:21:00 STEP 1346 H= 336:30 +CPU= 32.351
  14:21:35 STEP 1347 H= 336:45 +CPU= 32.370
  14:22:10 STEP 1348 H= 337:00 +CPU= 32.840
  14:22:44 STEP 1349 H= 337:15 +CPU= 31.657
  14:23:17 STEP 1350 H= 337:30 +CPU= 31.633
  14:23:52 STEP 1351 H= 337:45 +CPU= 31.110
  14:24:27 STEP 1352 H= 338:00 +CPU= 32.048
  14:25:01 STEP 1353 H= 338:15 +CPU= 31.132
  14:25:35 STEP 1354 H= 338:30 +CPU= 30.763
  14:26:10 STEP 1355 H= 338:45 +CPU= 30.330
  14:26:53 STEP 1356 H= 339:00 +CPU= 38.347
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMGGh7zg+001356
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMSHh7zg+001356
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMUAh7zg+001356
  14:27:26 STEP 1357 H= 339:15 +CPU= 31.116
  14:27:59 STEP 1358 H= 339:30 +CPU= 31.667
  14:28:33 STEP 1359 H= 339:45 +CPU= 31.793
  14:29:06 STEP 1360 H= 340:00 +CPU= 32.455
  14:29:40 STEP 1361 H= 340:15 +CPU= 31.875
  14:30:13 STEP 1362 H= 340:30 +CPU= 31.829
  14:30:45 STEP 1363 H= 340:45 +CPU= 30.849
  14:31:19 STEP 1364 H= 341:00 +CPU= 31.581
  14:31:52 STEP 1365 H= 341:15 +CPU= 31.220
  14:32:25 STEP 1366 H= 341:30 +CPU= 31.652
  14:32:58 STEP 1367 H= 341:45 +CPU= 30.883
  14:33:39 STEP 1368 H= 342:00 +CPU= 38.447
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMGGh7zg+001368
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMSHh7zg+001368
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMUAh7zg+001368
  14:34:11 STEP 1369 H= 342:15 +CPU= 31.094
  14:34:45 STEP 1370 H= 342:30 +CPU= 30.715
  14:35:19 STEP 1371 H= 342:45 +CPU= 30.609
  14:35:53 STEP 1372 H= 343:00 +CPU= 31.317
  14:36:27 STEP 1373 H= 343:15 +CPU= 30.554
  14:37:02 STEP 1374 H= 343:30 +CPU= 30.735
  14:37:36 STEP 1375 H= 343:45 +CPU= 30.853
  14:38:13 STEP 1376 H= 344:00 +CPU= 31.684
  14:38:48 STEP 1377 H= 344:15 +CPU= 30.699
  14:39:22 STEP 1378 H= 344:30 +CPU= 30.844
  14:39:57 STEP 1379 H= 344:45 +CPU= 31.146
  14:40:41 STEP 1380 H= 345:00 +CPU= 39.049
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMGGh7zg+001380
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMSHh7zg+001380
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMUAh7zg+001380
  14:41:14 STEP 1381 H= 345:15 +CPU= 31.236
  14:41:49 STEP 1382 H= 345:30 +CPU= 31.534
  14:42:24 STEP 1383 H= 345:45 +CPU= 31.072
  14:42:59 STEP 1384 H= 346:00 +CPU= 31.527
  14:43:33 STEP 1385 H= 346:15 +CPU= 30.904
  14:44:07 STEP 1386 H= 346:30 +CPU= 30.665
  14:44:41 STEP 1387 H= 346:45 +CPU= 31.067
  14:45:16 STEP 1388 H= 347:00 +CPU= 31.218
  14:45:50 STEP 1389 H= 347:15 +CPU= 30.403
  14:46:25 STEP 1390 H= 347:30 +CPU= 30.847
  14:46:59 STEP 1391 H= 347:45 +CPU= 30.534
  14:47:44 STEP 1392 H= 348:00 +CPU= 41.524
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMGGh7zg+001392
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMSHh7zg+001392
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMUAh7zg+001392
  14:48:19 STEP 1393 H= 348:15 +CPU= 31.139
  14:48:54 STEP 1394 H= 348:30 +CPU= 31.528
  14:49:28 STEP 1395 H= 348:45 +CPU= 30.958
  14:50:03 STEP 1396 H= 349:00 +CPU= 31.377
  14:50:38 STEP 1397 H= 349:15 +CPU= 31.299
  14:51:13 STEP 1398 H= 349:30 +CPU= 30.899
  14:51:45 STEP 1399 H= 349:45 +CPU= 30.861
  14:52:20 STEP 1400 H= 350:00 +CPU= 30.924
  14:52:53 STEP 1401 H= 350:15 +CPU= 30.029
  14:53:26 STEP 1402 H= 350:30 +CPU= 30.194
  14:53:59 STEP 1403 H= 350:45 +CPU= 30.156
  14:54:40 STEP 1404 H= 351:00 +CPU= 37.443
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMGGh7zg+001404
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMSHh7zg+001404
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMUAh7zg+001404
  14:55:14 STEP 1405 H= 351:15 +CPU= 29.754
Quit request received from BOINC client, ending the child process
>>> Printing last 70 lines from file: NODE.001_01

         AVE   0.920961709366938E-03 0.100000000000000E-07 0.112793869818539E-01
 GPNORM  LIQUID WATER        AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.622013337104064E-08 0.000000000000000E+00 0.160965356007289E-03
 GPNORM  ICE WATER           AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.612345888939527E-09 0.000000000000000E+00 0.260386513537809E-05
 GPNORM SNOW                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.196762797006077E-11 0.000000000000000E+00 0.102880340520403E-06
 GPNORM RAIN                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.316040713320001E-12 0.000000000000000E+00 0.388870641703876E-07
 GPNORM  CLOUD FRACTION      AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.110544051125829E-01 0.000000000000000E+00 0.100000000000000E+01
 GPNORM CLW                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.618842295640307E-08 0.000000000000000E+00 0.159766655204847E-03
 GPNORM CIW                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.611709691010888E-09 0.000000000000000E+00 0.276142907058069E-05
 GPNORM CC                   AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.110540384376572E-01 0.000000000000000E+00 0.100000000000000E+01
 GPNORM RFL                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.294720869987170E-11 0.000000000000000E+00 0.185742832017722E-06
 GPNORM SFL                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.968082302056685E-11 0.000000000000000E+00 0.446607016256246E-06
 GPNORM CRFL                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00
 GPNORM CSFL                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00
 GPNORM PFRC                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.688650346108416E-02 0.000000000000000E+00 0.100000000000000E+01
 NSTEP =  1404 STEPO   A00000000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1404 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001404 MODE=w
 IO-STREAM CLOSED - ICMGGh7zg+001404
 MPI-TASK:   1 -    328407636 BYTES IN      6 RECORDS TRANSFERRED IN    0.0002 SECONDS******** Mbytes/s, TOTAL TIME=    0.0105( 1.9%)
 NSTEP =  1404 STEPO   0AA000000
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMSHh7zg+001404 MODE=w
 IO-STREAM CLOSED - ./ICMSHh7zg+001404
 MPI-TASK:   1 -    707399250 BYTES IN    202 RECORDS TRANSFERRED IN    0.0068 SECONDS104029.3 Mbytes/s, TOTAL TIME=    0.3240( 2.1%)
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMUAh7zg+001404 MODE=w
 IO-STREAM CLOSED - ./ICMUAh7zg+001404
 MPI-TASK:   1 -   5254522176 BYTES IN    125 RECORDS TRANSFERRED IN    0.0036 SECONDS******** Mbytes/s, TOTAL TIME=    0.1740( 2.1%)
 STEPO_FPOS WAS : B00000F00E
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMUAh7zg+001404 MODE=a
 IO-STREAM CLOSED - ./ICMUAh7zg+001404
 MPI-TASK:   1 -           10 BYTES IN      2 RECORDS TRANSFERRED IN    0.0002 SECONDS     0.1 Mbytes/s, TOTAL TIME=    0.0171( 1.2%)
 STEPO_FPOS WAS : V00000000Y
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMUAh7zg+001404 MODE=a
 IO-STREAM CLOSED - ./ICMUAh7zg+001404
 MPI-TASK:   1 -    319827907 BYTES IN      4 RECORDS TRANSFERRED IN    0.0003 SECONDS******** Mbytes/s, TOTAL TIME=    0.0075( 4.0%)
 STEPO_FPOS WAS : T00000000M
 NSTEP =  1404 STEPO   0AAA00AAA
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1404 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001404 MODE=a
 IO-STREAM CLOSED - ICMGGh7zg+001404
 MPI-TASK:   1 -    328407636 BYTES IN      7 RECORDS TRANSFERRED IN    0.0003 SECONDS******** Mbytes/s, TOTAL TIME=    0.0128( 2.3%)
 14:54:40 STEP 1404 H= 351:00 +CPU= 37.443
ISTEP=    1405 RSOLINC= 1361.4482 RI0= 1365.0000 ZI0= 1361.4482 RIP0= 1361.4482
ISTEP=    1405 ZI0= 1361.4482 ZSEASON= 1.000000000 REA=149597870000.00 RDEASO=149597870000.00
 NSTEP =  1405 STEPO   0AAA00AAA
 14:55:14 STEP 1405 H= 351:15 +CPU= 29.754
 NSTEP =  1406 STEPO   0AAA00AAA
------------------------------------------------
CNT0 not found; string returned was: 'STEPO'
>>> Printing last 8 lines from file: ifs.stat

 14:54:00 0AA000000 STEPO     1404     0.027    0.027    0.027    837:17    968:09 0.70384906287946E-07       2GB       0MB
 14:54:02 FULLPOS-B DYNFPOS   1404     1.778    1.778    1.871    837:18    968:11 0.70384906287946E-07       2GB       0MB
 14:54:05 FULLPOS-V DYNFPOS   1404     3.595    3.595    3.751    837:22    968:15 0.70384906287946E-07       2GB       0MB
 14:54:06 FULLPOS-T DYNFPOS   1404     0.721    0.721    0.795    837:23    968:16 0.70384906287946E-07       2GB       0MB
 14:54:07 0AAA00AAA STEPO     1404     0.553    0.553    0.620    837:23    968:16 0.70384906287946E-07       2GB       0MB
 14:54:40 0AAA00AAA STEPO     1405    30.660   30.660   33.695    837:54    968:50 0.70384906287946E-07       2GB       0MB
 14:55:14 0AAA00AAA STEPO     1406    29.754   29.754   33.112    838:24    969:23 0.70384906287946E-07       2GB       0MB
------------------------------------------------
>>> Printing last 11 lines from file: rcf

 CSTEP   = "    1392",
 CTIME   = "00141200      ",
 NSTEPLPP        =         201,         202,          49,      228226,      228227, 2*0,        1380, 2*0,
 IPRGPNSRES      =           1,
 IPRGPEWRES      =           1,
 IPRTRWRES       =           1,
 IPRTRVRES       =           1,
 GMASS0  =  0.000000000000000E+000,
 GMASSI  =  0.000000000000000E+000
 /
------------------------------------------------
>>> Printing last 8 lines from file: /mydisks/b/boinc-client/slots/31/progress_file_12283885.xml

<running_values>
  <last_cpu_time>50304.450000</last_cpu_time>
  <upload_file_number>14</upload_file_number>
  <last_iter>1406</last_iter>
  <last_upload>1209600</last_upload>
  <model_completed>0</model_completed>
</running_values>
------------------------------------------------
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.13_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a1l6
(argv4) batchid: 1017
(argv5) wuid: 12283885
(argv6) fclen: 20
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /mydisks/b/boinc-client/slots/31
Project directory is: /mydisks/b/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.13
Location of temp folder: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885
..mkdir for temp folder for results failed
Copying: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip to: /mydisks/b/boinc-client/slots/31/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /mydisks/b/boinc-client/slots/31/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_725d91043aa010ad7d054f5e562f8a7e to: /mydisks/b/boinc-client/slots/31/oifs_43r3_bl_a1l6_2016092300_20_1017_12283885.zip
Unzipping the namelist zip file: /mydisks/b/boinc-client/slots/31/oifs_43r3_bl_a1l6_2016092300_20_1017_12283885.zip
ic_ancil_file: ic_ancil_12283885
ifsdata_file: ifsdata_12283885
climate_data_file: clim_data_12283885
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /mydisks/b/boinc-client/slots/31/ic_ancil_12283885.zip
Unzipping the IC ancils zip file: /mydisks/b/boinc-client/slots/31/ic_ancil_12283885.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_d8f8ff7076096af105d0b5a71f3e06c1 to: /mydisks/b/boinc-client/slots/31/ifsdata/ifsdata_12283885.zip
Unzipping the ifsdata_zip file: /mydisks/b/boinc-client/slots/31/ifsdata/ifsdata_12283885.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /mydisks/b/boinc-client/slots/31/159l_2/clim_data_12283885.zip
Unzipping the climate data zip file: /mydisks/b/boinc-client/slots/31/159l_2/clim_data_12283885.zip
Checking for progress XML file: /mydisks/b/boinc-client/slots/31/progress_file_12283885.xml
Opened progress file ok : /mydisks/b/boinc-client/slots/31/progress_file_12283885.xml
-- Model is restarting --
Adjusting last_iter, 1406, to previous model restart step.
Creating progress file: /mydisks/b/boinc-client/slots/31/progress_file_12283885.xml
last_cpu_time: 50304
upload_file_number: 14
last_iter: 1393
last_upload: 1209600
model_completed: 0
total_length_of_simulation: 1728000
result_base_name: oifs_43r3_bl_a1l6_2016092300_20_1017_12283885_0_r1611978764
The child process has been launched with process id: 358460
Executing the command: /mydisks/b/boinc-client/slots/31/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:centurion:1:1:358460:358460] [20240608:145822:1717883902:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2091] fp = 0x314f440
[EC_DRHOOK:centurion:1:1:358460:358460] [20240608:145822:1717883902:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:centurion:1:1:358460:358460] [20240608:145822:1717883902:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:centurion:1:1:358460:358460] [20240608:145822:1717883902:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:centurion:1:1:358460:358460] [20240608:145822:1717883902:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:centurion:1:1:358460:358460] [20240608:145822:1717883902:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:centurion:1:1:358460:358460] [20240608:145822:1717883902:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:centurion:1:1:358460:358460] [20240608:145822:1717883902:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:centurion:1:1:358460:358460] [20240608:145822:1717883902:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:centurion:1:1:358460:358460] [20240608:145822:1717883902:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:centurion:1:1:358460:358460] [20240608:145822:1717883902:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:centurion:1:1:358460:358460] [20240608:145822:1717883902:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:centurion:1:1:358460:358460] [20240608:145822:1717883902:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:centurion:1:1:358460:358460] [20240608:145822:1717883902:0.000] [ignore_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:centurion:1:1:358460:358460] [20240608:145822:1717883902:0.000] [restore_default_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:centurion:1:1:358460:358460] [20240608:145822:1717883902:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:centurion:1:1:358460:358460] [20240608:145822:1717883902:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:centurion:1:1:358460:358460] [20240608:145822:1717883902:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:centurion:1:1:358460:358460] [20240608:145822:1717883902:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x820cf0 (old at (nil))
[EC_DRHOOK:centurion:1:1:358460:358460] [20240608:145822:1717883902:0.000] [trapfpe_treatment@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:centurion:1:1:358460:358460] [20240608:145822:1717883902:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:centurion:1:1:358460:358460] [20240608:145822:1717883902:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:centurion:1:1:358460:358460] [20240608:145822:1717883902:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x820cf0 (old at (nil))
[EC_DRHOOK:centurion:1:1:358460:358460] [20240608:145822:1717883902:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:centurion:1:1:358460:358460] [20240608:145822:1717883902:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:centurion:1:1:358460:358460] [20240608:145822:1717883902:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:centurion:1:1:358460:358460] [20240608:145822:1717883902:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x820cf0 (old at (nil))
[EC_DRHOOK:centurion:1:1:358460:358460] [20240608:145822:1717883902:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x820cf0 (old at (nil))
[EC_DRHOOK:centurion:1:1:358460:358460] [20240608:145822:1717883902:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:centurion:1:1:358460:358460] [20240608:145822:1717883902:0.000] [catch_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  14:58:53 STEP 1392 H= 348:00 +CPU= 26.845
  14:59:18 STEP 1393 H= 348:15 +CPU= 24.888
  14:59:44 STEP 1394 H= 348:30 +CPU= 25.249
  15:00:11 STEP 1395 H= 348:45 +CPU= 25.467
  15:00:38 STEP 1396 H= 349:00 +CPU= 26.158
  15:01:05 STEP 1397 H= 349:15 +CPU= 25.431
  15:01:31 STEP 1398 H= 349:30 +CPU= 24.903
  15:01:57 STEP 1399 H= 349:45 +CPU= 24.894
  15:02:23 STEP 1400 H= 350:00 +CPU= 24.987
  15:02:49 STEP 1401 H= 350:15 +CPU= 24.971
  15:03:15 STEP 1402 H= 350:30 +CPU= 25.105
  15:03:41 STEP 1403 H= 350:45 +CPU= 25.037
  15:04:14 STEP 1404 H= 351:00 +CPU= 32.135
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMGGh7zg+001404
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMSHh7zg+001404
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMUAh7zg+001404
  15:04:41 STEP 1405 H= 351:15 +CPU= 26.423
  15:05:08 STEP 1406 H= 351:30 +CPU= 26.898
  15:05:36 STEP 1407 H= 351:45 +CPU= 26.858
  15:06:04 STEP 1408 H= 352:00 +CPU= 27.406
  15:06:31 STEP 1409 H= 352:15 +CPU= 26.984
  15:06:58 STEP 1410 H= 352:30 +CPU= 26.663
  15:07:24 STEP 1411 H= 352:45 +CPU= 25.569
  15:07:47 STEP 1412 H= 353:00 +CPU= 22.090
  15:08:08 STEP 1413 H= 353:15 +CPU= 21.717
  15:08:30 STEP 1414 H= 353:30 +CPU= 21.837
  15:08:53 STEP 1415 H= 353:45 +CPU= 22.407
  15:09:29 STEP 1416 H= 354:00 +CPU= 35.398
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMGGh7zg+001416
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMSHh7zg+001416
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMUAh7zg+001416
  15:09:59 STEP 1417 H= 354:15 +CPU= 28.955
  15:10:29 STEP 1418 H= 354:30 +CPU= 29.067
  15:10:58 STEP 1419 H= 354:45 +CPU= 29.231
  15:11:27 STEP 1420 H= 355:00 +CPU= 28.183
  15:11:55 STEP 1421 H= 355:15 +CPU= 27.829
  15:12:23 STEP 1422 H= 355:30 +CPU= 27.187
  15:12:53 STEP 1423 H= 355:45 +CPU= 28.443
  15:13:22 STEP 1424 H= 356:00 +CPU= 29.101
  15:13:51 STEP 1425 H= 356:15 +CPU= 27.882
  15:14:20 STEP 1426 H= 356:30 +CPU= 28.254
  15:14:48 STEP 1427 H= 356:45 +CPU= 27.577
  15:15:22 STEP 1428 H= 357:00 +CPU= 33.271
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMGGh7zg+001428
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMSHh7zg+001428
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMUAh7zg+001428
  15:15:50 STEP 1429 H= 357:15 +CPU= 27.631
  15:16:19 STEP 1430 H= 357:30 +CPU= 28.009
  15:16:47 STEP 1431 H= 357:45 +CPU= 27.500
  15:17:15 STEP 1432 H= 358:00 +CPU= 27.471
  15:17:43 STEP 1433 H= 358:15 +CPU= 26.995
  15:18:10 STEP 1434 H= 358:30 +CPU= 27.055
  15:18:38 STEP 1435 H= 358:45 +CPU= 26.947
  15:19:06 STEP 1436 H= 359:00 +CPU= 27.111
  15:19:33 STEP 1437 H= 359:15 +CPU= 26.118
  15:20:00 STEP 1438 H= 359:30 +CPU= 26.492
  15:20:27 STEP 1439 H= 359:45 +CPU= 26.446
  15:21:01 STEP 1440 H= 360:00 +CPU= 33.254
..The child process terminated with status: 0
>>> Printing last 70 lines from file: NODE.001_01

 
DDH------------------------------------------------------------------------
I                                                                         I
I TIME =  360.000H DDH * Diagnostic Var.,     AREA                GLOBAL  I
I                                                                         I
I-------------------------------------------------------------------------I
I   VARIABLE   I      Initial value         I        Current value        I
I-------------------------------------------------------------------------I
I  OMEGA (P/S) I     -0.4247964626E-21      I       0.2119281721E-07      I
ICloud fractionI      0.0000000000E+00      I       0.6077363489E-03      I
I  Relat. Hum. I      0.2624483536E+00      I       0.2835323972E+00      I
I  PASS 01     I      0.0000000000E+00      I       0.7480479886E+01      I
DDH------------------------------------------------------------------------
  
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1440 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001440 MODE=a
 IO-STREAM CLOSED - ICMGGh7zg+001440
 MPI-TASK:   1 -    312099220 BYTES IN      7 RECORDS TRANSFERRED IN    0.0003 SECONDS******** Mbytes/s, TOTAL TIME=    0.0115( 2.6%)
 15:21:01 STEP 1440 H= 360:00 +CPU= 33.254
 END CNT3
 NSTEP =  1440 CNT0   000000000  24.765  24.765  25.300   0.979               0               0               0               0               0
IO-STREAM STATISTICS, TOTAL NO OF REC/BYTES READ           46/  1069125708 WRITTEN         1430/************
 -TASK-OPENED-OPEN-RECS IN -KBYTE IN    -RECS OUT-KBYTE OUT-WALL   -WALL IN-WALL
  OU-TOT IN -TOT OUT
   1     28    0       46      1044068     1430 ************     3.7     1.6     2.1     2.3     4.4
===-=== START OF TIMING STATISTICS ===-===
 
STATS FOR ALL TASKS
 NUM ROUTINE                                     CALLS  MEAN(ms)   MAX(ms)   FRAC(%)  UNBAL(%)
   0 CNT0     - COMPLETE EXECUTION                   1 1359177.5 1359177.5    100.00      0.00
   1 CNT4     - FORWARD INTEGRATION                  1 1355233.3 1355233.3     99.71      0.00
   8 SCAN2M - GRID-POINT DYNAMICS                   53    4239.0    4239.0     16.53      0.00
   9 SPCM     - SPECTRAL COMP.                      48     794.3     794.3      2.81      0.00
  10 SCAN2M - PHYSICS                               49   13286.2   13286.2     47.90      0.00
  11 IOPACK   - OUTPUT P.P. RESULTS                  4     743.5     743.5      0.22      0.00
  12 SPNORM   - SPECTRAL NORM COMP.                  3      36.7      36.7      0.01      0.00
  17 GRIDFPOS IN CNT4                                4      23.1      23.1      0.01      0.00
  27 RESTART FILES - WRITING                         1    2973.8    2973.8      0.22      0.00
  28 RESTART FILES - READING                         1    3353.5    3353.5      0.25      0.00
  29 SU4FPOS IN CNT4                                 4       0.1       0.1      0.00      0.00
  30 DYNFPOS IN CNT4                                 4    5437.1    5437.1      1.60      0.00
  31 POSDDH IN STEPO                                 1       0.6       0.6      0.00      0.00
  37 CPGLAG   - SL COMPUTATIONS                     49    5547.8    5547.8     20.00      0.00
  39 SU0YOMB                                         1     363.7     363.7      0.03      0.00
  51 SCAN2M   - SL COMM. PART 1                     49      37.6      37.6      0.14      0.00
  54 SPCM     - M TO S/S TO M TRANSP.               48     355.1     355.1      1.25      0.00
  55 SPCIMPF  - S TO M/M TO S TRANSP.               48      70.8      70.8      0.25      0.00
  56 SPNORM   - SPECTRAL NORM COMM.                  3       0.3       0.3      0.00      0.00
 102 LTINV_CTL   - INVERSE LEGENDRE TRANSFORM      108     475.7     475.7      3.78      0.00
 103 LTDIR_CTL   - DIRECT LEGENDRE TRANSFORM       100     265.4     265.4      1.95      0.00
 106 FTDIR_CTL   - DIRECT FOURIER TRANSFORM        100      85.2      85.2      0.63      0.00
 107 FTINV_CTL   - INVERSE FOURIER TRANSFORM       108     138.7     138.7      1.10      0.00
 140 SULEG       - COMP. OF LEGENDRE POL.            1      39.9      39.9      0.00      0.00
 152 LTINV_CTL   - M TO L TRANSPOSITION            108      57.4      57.4      0.46      0.00
 153 LTDIR_CTL   - L TO M TRANSPOSITION            100      30.3      30.3      0.22      0.00
 157 FTINV_CTL   - L TO G TRANSPOSITION            108      78.8      78.8      0.63      0.00
 158 FTDIR_CTL   - G TO L TRANSPOSITION            100      61.9      61.9      0.46      0.00
 400 GSTATS                                       6667       0.0       0.0      0.00      0.00
 401 GSTATS HOOK                                  6149       0.0       0.0      0.00      0.00
TOTAL MEASURED IMBALANCE =       0.0 SECONDS,  0.0 PERCENT
TOTAL WALLCLOCK TIME     1359.2 CPU TIME    1326.2 VECTOR TIME     1326.2


===-=== END   OF TIMING STATISTICS ===-===
FORECAST DAYS PER DAY      956.3
  *** END CNT0 *** 
------------------------------------------------
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMGGh7zg+001440
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMSHh7zg+001440
Moving to projects directory: /mydisks/b/boinc-client/slots/31/ICMUAh7zg+001440
Adding to the zip: /mydisks/b/boinc-client/slots/31/NODE.001_01
Adding to the zip: /mydisks/b/boinc-client/slots/31/ifs.stat
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMGGh7zg+001344
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMSHh7zg+001344
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMUAh7zg+001344
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMGGh7zg+001356
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMSHh7zg+001356
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMUAh7zg+001356
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMGGh7zg+001368
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMSHh7zg+001368
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMUAh7zg+001368
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMGGh7zg+001380
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMSHh7zg+001380
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMUAh7zg+001380
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMGGh7zg+001392
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMSHh7zg+001392
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMUAh7zg+001392
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMGGh7zg+001404
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMSHh7zg+001404
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMUAh7zg+001404
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMGGh7zg+001416
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMSHh7zg+001416
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMUAh7zg+001416
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMGGh7zg+001428
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMSHh7zg+001428
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMUAh7zg+001428
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMGGh7zg+001440
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMSHh7zg+001440
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283885/ICMUAh7zg+001440
Zipping up the final file: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a1l6_2016092300_20_1017_12283885_0_r1611978764_14.zip
Uploading the final file: upload_file_14.zip
Uploading trickle at timestep: 1295100
15:24:29 (357572): called boinc_finish(0)

</stderr_txt>
<message>
upload failure: <file_xfer_error>
  <file_name>oifs_43r3_bl_a1l6_2016092300_20_1017_12283885_0_r1611978764_15.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a1l6_2016092300_20_1017_12283885_0_r1611978764_16.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a1l6_2016092300_20_1017_12283885_0_r1611978764_17.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a1l6_2016092300_20_1017_12283885_0_r1611978764_18.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a1l6_2016092300_20_1017_12283885_0_r1611978764_19.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
</message>
]]
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
08 Jun 2024 22:38:31 1517679 22435464 oifs_43r3_bl_a1l6_2016092300_20_1017_12283885_0 1,295,100 51,630 0.0399
08 Jun 2024 17:22:16 1517679 22435464 oifs_43r3_bl_a1l6_2016092300_20_1017_12283885_0 864,000 35,279 0.0408


©2024 climateprediction.net