climateprediction.net home page
Task 22437259

Task 22437259

Name oifs_43r3_bl_a2y9_2016092300_20_1017_12285652_0
Workunit 12285652
Created 7 Jun 2024, 16:22:34 UTC
Sent 7 Jun 2024, 16:31:19 UTC
Report deadline 6 Aug 2024, 16:31:19 UTC
Received 10 Jun 2024, 19:37:15 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 0 (0x00000000)
Computer ID 1533094
Run time 20 hours 25 min 30 sec
CPU time 20 hours 14 min 17 sec
Validate state Invalid
Credit 1,976.32
Device peak FLOPS 7.75 GFLOPS
Application version OpenIFS 43r3 Baroclinic Lifecycle v1.13
x86_64-pc-linux-gnu
Peak working set size 5,533.09 MB
Peak swap size 5,980.80 MB
Peak disk usage 1,391.77 MB
Stderr
<core_client_version>7.24.1</core_client_version>
<![CDATA[
<stderr_txt>
30 +CPU= 64.398
  08:04:15 STEP 1015 H= 253:45 +CPU= 60.420
  08:05:19 STEP 1016 H= 254:00 +CPU= 62.353
  08:06:24 STEP 1017 H= 254:15 +CPU= 62.311
  08:07:23 STEP 1018 H= 254:30 +CPU= 58.400
  08:08:31 STEP 1019 H= 254:45 +CPU= 64.638
  08:09:51 STEP 1020 H= 255:00 +CPU= 77.012
Moving to projects directory: /var/lib/boinc/slots/18/ICMGGh7zg+001020
Moving to projects directory: /var/lib/boinc/slots/18/ICMSHh7zg+001020
Moving to projects directory: /var/lib/boinc/slots/18/ICMUAh7zg+001020
  08:10:56 STEP 1021 H= 255:15 +CPU= 62.132
  08:12:02 STEP 1022 H= 255:30 +CPU= 63.265
  08:13:10 STEP 1023 H= 255:45 +CPU= 65.083
  08:14:15 STEP 1024 H= 256:00 +CPU= 62.443
  08:15:21 STEP 1025 H= 256:15 +CPU= 62.674
  08:16:20 STEP 1026 H= 256:30 +CPU= 57.542
  08:17:24 STEP 1027 H= 256:45 +CPU= 61.705
  08:18:30 STEP 1028 H= 257:00 +CPU= 62.796
  08:19:34 STEP 1029 H= 257:15 +CPU= 61.217
  08:20:42 STEP 1030 H= 257:30 +CPU= 65.629
  08:21:47 STEP 1031 H= 257:45 +CPU= 61.430
  08:23:09 STEP 1032 H= 258:00 +CPU= 78.897
Moving to projects directory: /var/lib/boinc/slots/18/ICMGGh7zg+001032
Moving to projects directory: /var/lib/boinc/slots/18/ICMSHh7zg+001032
Moving to projects directory: /var/lib/boinc/slots/18/ICMUAh7zg+001032
  08:24:13 STEP 1033 H= 258:15 +CPU= 61.855
  08:25:13 STEP 1034 H= 258:30 +CPU= 58.791
  08:26:20 STEP 1035 H= 258:45 +CPU= 62.813
  08:27:24 STEP 1036 H= 259:00 +CPU= 62.936
  08:28:26 STEP 1037 H= 259:15 +CPU= 60.124
  08:29:33 STEP 1038 H= 259:30 +CPU= 64.739
  08:30:40 STEP 1039 H= 259:45 +CPU= 64.870
  08:31:45 STEP 1040 H= 260:00 +CPU= 62.925
  08:32:48 STEP 1041 H= 260:15 +CPU= 61.000
  08:33:55 STEP 1042 H= 260:30 +CPU= 64.200
  08:34:59 STEP 1043 H= 260:45 +CPU= 61.598
  08:36:21 STEP 1044 H= 261:00 +CPU= 78.624
Moving to projects directory: /var/lib/boinc/slots/18/ICMGGh7zg+001044
Moving to projects directory: /var/lib/boinc/slots/18/ICMSHh7zg+001044
Moving to projects directory: /var/lib/boinc/slots/18/ICMUAh7zg+001044
  08:37:26 STEP 1045 H= 261:15 +CPU= 61.830
  08:38:32 STEP 1046 H= 261:30 +CPU= 63.722
  08:39:39 STEP 1047 H= 261:45 +CPU= 65.549
  08:40:40 STEP 1048 H= 262:00 +CPU= 59.141
  08:41:41 STEP 1049 H= 262:15 +CPU= 59.347
  08:42:47 STEP 1050 H= 262:30 +CPU= 63.086
  08:43:52 STEP 1051 H= 262:45 +CPU= 64.083
  08:44:59 STEP 1052 H= 263:00 +CPU= 64.846
  08:46:10 STEP 1053 H= 263:15 +CPU= 67.069
  08:47:15 STEP 1054 H= 263:30 +CPU= 63.759
  08:48:19 STEP 1055 H= 263:45 +CPU= 61.450
  08:49:47 STEP 1056 H= 264:00 +CPU= 84.379
Moving to projects directory: /var/lib/boinc/slots/18/ICMGGh7zg+001056
Moving to projects directory: /var/lib/boinc/slots/18/ICMSHh7zg+001056
Moving to projects directory: /var/lib/boinc/slots/18/ICMUAh7zg+001056
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMGGh7zg+000960
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMSHh7zg+000960
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMUAh7zg+000960
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMGGh7zg+000972
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMSHh7zg+000972
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMUAh7zg+000972
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMGGh7zg+000984
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMSHh7zg+000984
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMUAh7zg+000984
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMGGh7zg+000996
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMSHh7zg+000996
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMUAh7zg+000996
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMGGh7zg+001008
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMSHh7zg+001008
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMUAh7zg+001008
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMGGh7zg+001020
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMSHh7zg+001020
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMUAh7zg+001020
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMGGh7zg+001032
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMSHh7zg+001032
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMUAh7zg+001032
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMGGh7zg+001044
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMSHh7zg+001044
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMUAh7zg+001044
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_a2y9_2016092300_20_1017_12285652_0_r1821875991_10.zip
Uploading the intermediate file: upload_file_10.zip
  08:50:52 STEP 1057 H= 264:15 +CPU= 63.111
  08:51:56 STEP 1058 H= 264:30 +CPU= 61.703
  08:53:02 STEP 1059 H= 264:45 +CPU= 63.770
  08:54:10 STEP 1060 H= 265:00 +CPU= 65.150
  08:55:19 STEP 1061 H= 265:15 +CPU= 65.318
  08:56:27 STEP 1062 H= 265:30 +CPU= 63.633
  08:57:34 STEP 1063 H= 265:45 +CPU= 64.153
  08:58:42 STEP 1064 H= 266:00 +CPU= 65.546
  08:59:46 STEP 1065 H= 266:15 +CPU= 61.170
  09:00:50 STEP 1066 H= 266:30 +CPU= 61.398
  09:01:55 STEP 1067 H= 266:45 +CPU= 62.054
  09:03:11 STEP 1068 H= 267:00 +CPU= 73.293
Moving to projects directory: /var/lib/boinc/slots/18/ICMGGh7zg+001068
Moving to projects directory: /var/lib/boinc/slots/18/ICMSHh7zg+001068
Moving to projects directory: /var/lib/boinc/slots/18/ICMUAh7zg+001068
  09:04:15 STEP 1069 H= 267:15 +CPU= 61.867
  09:05:19 STEP 1070 H= 267:30 +CPU= 62.013
  09:06:25 STEP 1071 H= 267:45 +CPU= 63.099
  09:07:33 STEP 1072 H= 268:00 +CPU= 64.644
  09:08:36 STEP 1073 H= 268:15 +CPU= 60.140
  09:09:39 STEP 1074 H= 268:30 +CPU= 60.452
  09:10:47 STEP 1075 H= 268:45 +CPU= 65.676
  09:11:57 STEP 1076 H= 269:00 +CPU= 67.767
  09:13:06 STEP 1077 H= 269:15 +CPU= 66.466
  09:14:11 STEP 1078 H= 269:30 +CPU= 62.403
  09:15:18 STEP 1079 H= 269:45 +CPU= 64.127
  09:16:42 STEP 1080 H= 270:00 +CPU= 80.188
Moving to projects directory: /var/lib/boinc/slots/18/ICMGGh7zg+001080
Moving to projects directory: /var/lib/boinc/slots/18/ICMSHh7zg+001080
Moving to projects directory: /var/lib/boinc/slots/18/ICMUAh7zg+001080
  09:17:46 STEP 1081 H= 270:15 +CPU= 62.022
  09:18:51 STEP 1082 H= 270:30 +CPU= 62.239
  09:19:59 STEP 1083 H= 270:45 +CPU= 64.379
  09:21:04 STEP 1084 H= 271:00 +CPU= 62.630
  09:22:12 STEP 1085 H= 271:15 +CPU= 64.151
  09:23:22 STEP 1086 H= 271:30 +CPU= 65.991
  09:24:30 STEP 1087 H= 271:45 +CPU= 65.287
  09:25:37 STEP 1088 H= 272:00 +CPU= 64.589
  09:26:43 STEP 1089 H= 272:15 +CPU= 63.007
  09:27:47 STEP 1090 H= 272:30 +CPU= 62.272
  09:28:56 STEP 1091 H= 272:45 +CPU= 65.860
  09:30:15 STEP 1092 H= 273:00 +CPU= 75.765
Moving to projects directory: /var/lib/boinc/slots/18/ICMGGh7zg+001092
Moving to projects directory: /var/lib/boinc/slots/18/ICMSHh7zg+001092
Moving to projects directory: /var/lib/boinc/slots/18/ICMUAh7zg+001092
  09:31:23 STEP 1093 H= 273:15 +CPU= 64.978
  09:32:34 STEP 1094 H= 273:30 +CPU= 64.116
  09:33:45 STEP 1095 H= 273:45 +CPU= 63.163
  09:34:53 STEP 1096 H= 274:00 +CPU= 63.420
  09:36:00 STEP 1097 H= 274:15 +CPU= 63.308
  09:37:10 STEP 1098 H= 274:30 +CPU= 64.997
  09:38:18 STEP 1099 H= 274:45 +CPU= 64.487
  09:39:26 STEP 1100 H= 275:00 +CPU= 64.184
  09:40:34 STEP 1101 H= 275:15 +CPU= 64.789
  09:41:42 STEP 1102 H= 275:30 +CPU= 62.549
  09:42:54 STEP 1103 H= 275:45 +CPU= 67.208
  09:44:29 STEP 1104 H= 276:00 +CPU= 87.511
Moving to projects directory: /var/lib/boinc/slots/18/ICMGGh7zg+001104
Moving to projects directory: /var/lib/boinc/slots/18/ICMSHh7zg+001104
Moving to projects directory: /var/lib/boinc/slots/18/ICMUAh7zg+001104
  09:45:37 STEP 1105 H= 276:15 +CPU= 63.367
  09:46:47 STEP 1106 H= 276:30 +CPU= 64.207
  09:48:00 STEP 1107 H= 276:45 +CPU= 68.173
  09:49:10 STEP 1108 H= 277:00 +CPU= 66.795
  09:50:17 STEP 1109 H= 277:15 +CPU= 62.951
  09:51:24 STEP 1110 H= 277:30 +CPU= 63.083
  09:52:31 STEP 1111 H= 277:45 +CPU= 64.730
  09:53:40 STEP 1112 H= 278:00 +CPU= 65.314
  09:54:44 STEP 1113 H= 278:15 +CPU= 60.810
  09:55:55 STEP 1114 H= 278:30 +CPU= 65.959
  09:57:01 STEP 1115 H= 278:45 +CPU= 63.235
  09:58:20 STEP 1116 H= 279:00 +CPU= 75.789
Moving to projects directory: /var/lib/boinc/slots/18/ICMGGh7zg+001116
Moving to projects directory: /var/lib/boinc/slots/18/ICMSHh7zg+001116
Moving to projects directory: /var/lib/boinc/slots/18/ICMUAh7zg+001116
  09:59:30 STEP 1117 H= 279:15 +CPU= 66.586
  10:00:38 STEP 1118 H= 279:30 +CPU= 65.055
  10:01:43 STEP 1119 H= 279:45 +CPU= 62.715
  10:02:50 STEP 1120 H= 280:00 +CPU= 63.806
  10:03:57 STEP 1121 H= 280:15 +CPU= 62.578
  10:05:06 STEP 1122 H= 280:30 +CPU= 64.706
  10:06:06 STEP 1123 H= 280:45 +CPU= 58.860
  10:07:13 STEP 1124 H= 281:00 +CPU= 64.996
  10:08:16 STEP 1125 H= 281:15 +CPU= 60.303
  10:09:21 STEP 1126 H= 281:30 +CPU= 62.683
  10:10:25 STEP 1127 H= 281:45 +CPU= 61.104
  10:11:42 STEP 1128 H= 282:00 +CPU= 76.205
Moving to projects directory: /var/lib/boinc/slots/18/ICMGGh7zg+001128
Moving to projects directory: /var/lib/boinc/slots/18/ICMSHh7zg+001128
Moving to projects directory: /var/lib/boinc/slots/18/ICMUAh7zg+001128
  10:12:45 STEP 1129 H= 282:15 +CPU= 60.506
  10:13:53 STEP 1130 H= 282:30 +CPU= 65.661
  10:14:55 STEP 1131 H= 282:45 +CPU= 60.521
  10:16:05 STEP 1132 H= 283:00 +CPU= 66.624
  10:17:04 STEP 1133 H= 283:15 +CPU= 57.589
  10:18:09 STEP 1134 H= 283:30 +CPU= 62.160
  10:19:17 STEP 1135 H= 283:45 +CPU= 66.025
  10:20:22 STEP 1136 H= 284:00 +CPU= 62.408
  10:21:27 STEP 1137 H= 284:15 +CPU= 62.390
  10:22:34 STEP 1138 H= 284:30 +CPU= 64.097
  10:23:43 STEP 1139 H= 284:45 +CPU= 65.514
  10:25:02 STEP 1140 H= 285:00 +CPU= 74.701
Moving to projects directory: /var/lib/boinc/slots/18/ICMGGh7zg+001140
Moving to projects directory: /var/lib/boinc/slots/18/ICMSHh7zg+001140
Moving to projects directory: /var/lib/boinc/slots/18/ICMUAh7zg+001140
  10:26:11 STEP 1141 H= 285:15 +CPU= 65.732
  10:27:15 STEP 1142 H= 285:30 +CPU= 62.200
  10:28:17 STEP 1143 H= 285:45 +CPU= 60.270
  10:29:19 STEP 1144 H= 286:00 +CPU= 60.689
  10:30:27 STEP 1145 H= 286:15 +CPU= 65.332
  10:31:32 STEP 1146 H= 286:30 +CPU= 62.240
  10:32:36 STEP 1147 H= 286:45 +CPU= 61.276
  10:33:41 STEP 1148 H= 287:00 +CPU= 61.526
  10:34:44 STEP 1149 H= 287:15 +CPU= 60.709
  10:35:48 STEP 1150 H= 287:30 +CPU= 60.337
  10:36:55 STEP 1151 H= 287:45 +CPU= 63.985
  10:38:27 STEP 1152 H= 288:00 +CPU= 87.017
Moving to projects directory: /var/lib/boinc/slots/18/ICMGGh7zg+001152
Moving to projects directory: /var/lib/boinc/slots/18/ICMSHh7zg+001152
Moving to projects directory: /var/lib/boinc/slots/18/ICMUAh7zg+001152
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMGGh7zg+001056
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMSHh7zg+001056
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMUAh7zg+001056
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMGGh7zg+001068
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMSHh7zg+001068
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMUAh7zg+001068
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMGGh7zg+001080
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMSHh7zg+001080
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMUAh7zg+001080
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMGGh7zg+001092
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMSHh7zg+001092
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMUAh7zg+001092
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMGGh7zg+001104
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMSHh7zg+001104
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMUAh7zg+001104
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMGGh7zg+001116
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMSHh7zg+001116
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMUAh7zg+001116
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMGGh7zg+001128
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMSHh7zg+001128
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMUAh7zg+001128
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMGGh7zg+001140
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMSHh7zg+001140
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMUAh7zg+001140
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_a2y9_2016092300_20_1017_12285652_0_r1821875991_11.zip
Uploading the intermediate file: upload_file_11.zip
  10:39:29 STEP 1153 H= 288:15 +CPU= 59.004
  10:40:28 STEP 1154 H= 288:30 +CPU= 56.833
  10:41:30 STEP 1155 H= 288:45 +CPU= 60.295
  10:42:35 STEP 1156 H= 289:00 +CPU= 61.412
  10:43:43 STEP 1157 H= 289:15 +CPU= 61.874
  10:44:48 STEP 1158 H= 289:30 +CPU= 61.022
  10:45:59 STEP 1159 H= 289:45 +CPU= 66.390
  10:47:06 STEP 1160 H= 290:00 +CPU= 62.712
  10:48:12 STEP 1161 H= 290:15 +CPU= 62.691
  10:49:18 STEP 1162 H= 290:30 +CPU= 59.735
  10:50:31 STEP 1163 H= 290:45 +CPU= 64.206
  10:51:50 STEP 1164 H= 291:00 +CPU= 73.829
Moving to projects directory: /var/lib/boinc/slots/18/ICMGGh7zg+001164
Moving to projects directory: /var/lib/boinc/slots/18/ICMSHh7zg+001164
Moving to projects directory: /var/lib/boinc/slots/18/ICMUAh7zg+001164
  10:53:01 STEP 1165 H= 291:15 +CPU= 67.226
  10:54:07 STEP 1166 H= 291:30 +CPU= 62.945
  10:55:12 STEP 1167 H= 291:45 +CPU= 61.128
  10:56:21 STEP 1168 H= 292:00 +CPU= 65.186
  10:57:26 STEP 1169 H= 292:15 +CPU= 62.143
  10:58:28 STEP 1170 H= 292:30 +CPU= 59.726
  10:59:32 STEP 1171 H= 292:45 +CPU= 63.356
  11:00:38 STEP 1172 H= 293:00 +CPU= 64.291
  11:01:42 STEP 1173 H= 293:15 +CPU= 61.028
  11:02:49 STEP 1174 H= 293:30 +CPU= 64.351
  11:03:53 STEP 1175 H= 293:45 +CPU= 60.699
  11:05:12 STEP 1176 H= 294:00 +CPU= 75.969
Moving to projects directory: /var/lib/boinc/slots/18/ICMGGh7zg+001176
Moving to projects directory: /var/lib/boinc/slots/18/ICMSHh7zg+001176
Moving to projects directory: /var/lib/boinc/slots/18/ICMUAh7zg+001176
  11:06:20 STEP 1177 H= 294:15 +CPU= 64.233
  11:07:26 STEP 1178 H= 294:30 +CPU= 61.473
  11:08:31 STEP 1179 H= 294:45 +CPU= 61.541
  11:09:39 STEP 1180 H= 295:00 +CPU= 64.378
[EC_DRHOOK:antares:1:1:28702:28702] [20240608:110953:1717837793:49598.585] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 3536MB (heap), 5314MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:antares:1:1:28702:28702] [20240608:110953:1717837793:49598.585] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:antares:1:1:28702:28702] [20240608:110953:1717837793:49598.585] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x820890 (old at (nil))
[EC_DRHOOK:antares:1:1:28702:28702] [20240608:110953:1717837793:49598.585] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1
[EC_DRHOOK:antares:1:1:28702:28702] [20240608:110953:1717837793:49598.585] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1
[EC_DRHOOK:antares:1:1:28702:28702] [20240608:110953:1717837793:49598.585] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3843] 3536 MB (maxheap), 5314 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:antares:1:1:28702:28702] [20240608:110953:1717837793:49598.585] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:antares:1:1:28702:28702] [20240608:110953:1717837793:49598.585] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:antares:1:1:28702:28702] [20240608:110953:1717837793:49598.585] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:antares:1:1:28702:28702] [20240608:110953:1717837793:49598.585] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:antares:1:1:28702:28702] [20240608:110953:1717837793:49598.585] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:antares:1:1:28702:28702] [20240608:110953:1717837793:49598.585] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :       CNT4 
[EC_DRHOOK:antares:1:1:28702:28702] [20240608:110953:1717837793:49598.585] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :        STEPO 
[EC_DRHOOK:antares:1:1:28702:28702] [20240608:110953:1717837793:49598.585] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :         SCAN2M 
[EC_DRHOOK:antares:1:1:28702:28702] [20240608:110953:1717837793:49598.585] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :          GP_MODEL_HEAP 
[EC_DRHOOK:antares:1:1:28702:28702] [20240608:110953:1717837793:49598.585] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :           GP_MODEL 
[EC_DRHOOK:antares:1:1:28702:28702] [20240608:110953:1717837793:49598.585] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :            CPG_DRV 
[EC_DRHOOK:antares:1:1:28702:28702] [20240608:110953:1717837793:49598.585] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :             >OMP-CPG                  (1025) 
[EC_DRHOOK:antares:1:1:28702:28702] [20240608:110953:1717837793:49598.586] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :              CPG 
[EC_DRHOOK:antares:1:1:28702:28702] [20240608:110953:1717837793:49598.586] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :               CPG_DIA 
[EC_DRHOOK:antares:1:1:28702:28702] [20240608:110953:1717837793:49598.586] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :                CPDYDDH 
[EC_DRHOOK:antares:1:1:28702:28702] [20240608:110954:1717837794:49599.586] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1
[EC_DRHOOK:antares:1:1:28702:28702] [20240608:110954:1717837794:49599.586] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1cf8da0 for signal#15, nsigs = 1
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source             
oifs_43r3_model.e  0000000001CF935B  Unknown               Unknown  Unknown
oifs_43r3_model.e  00000000008214C1  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DD5490  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000000CEE2CB  cpdyddh_                  959  cpdyddh.F90
oifs_43r3_model.e  0000000000CBEE1D  cpg_dia_                  618  cpg_dia.F90
oifs_43r3_model.e  0000000000C2519F  cpg_                      718  cpg.F90
oifs_43r3_model.e  0000000000C22238  cpg_drv_                  351  cpg_drv.F90
oifs_43r3_model.e  0000000000BCC914  gp_model_                 419  gp_model.F90
oifs_43r3_model.e  00000000012C5002  gp_model_heap_             74  gp_model_heap.F90
oifs_43r3_model.e  0000000000BACBAE  scan2m_                   535  scan2m.F90
oifs_43r3_model.e  0000000000688153  stepo_                    327  stepo.F90
oifs_43r3_model.e  00000000004CD1DC  cnt4_                    1133  cnt4.F90
oifs_43r3_model.e  00000000004B5D85  cnt3_                     267  cnt3.F90
oifs_43r3_model.e  00000000004B4F52  cnt2_                      88  cnt2.F90
oifs_43r3_model.e  000000000049D638  cnt1_                      92  cnt1.F90
oifs_43r3_model.e  0000000000406FBD  cnt0_                     146  cnt0.F90
oifs_43r3_model.e  0000000000401E7F  MAIN__                     96  master.F90
oifs_43r3_model.e  0000000000401E12  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCC34A  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCDBE7  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000000401CD5  Unknown               Unknown  Unknown
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.13_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a2y9
(argv4) batchid: 1017
(argv5) wuid: 12285652
(argv6) fclen: 20
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/18
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.13
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/18/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/18/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_6217e2621fb85cdbc9f4f8dd7906d4f2 to: /var/lib/boinc/slots/18/oifs_43r3_bl_a2y9_2016092300_20_1017_12285652.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/18/oifs_43r3_bl_a2y9_2016092300_20_1017_12285652.zip
ic_ancil_file: ic_ancil_12285652
ifsdata_file: ifsdata_12285652
climate_data_file: clim_data_12285652
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /var/lib/boinc/slots/18/ic_ancil_12285652.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/18/ic_ancil_12285652.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_4ff2a3106759ee25e06a32583f5c3e0a to: /var/lib/boinc/slots/18/ifsdata/ifsdata_12285652.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/18/ifsdata/ifsdata_12285652.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/18/159l_2/clim_data_12285652.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/18/159l_2/clim_data_12285652.zip
Checking for progress XML file: /var/lib/boinc/slots/18/progress_file_12285652.xml
Opened progress file ok : /var/lib/boinc/slots/18/progress_file_12285652.xml
-- Model is restarting --
Adjusting last_iter, 1181, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/18/progress_file_12285652.xml
last_cpu_time: 56415
upload_file_number: 12
last_iter: 1153
last_upload: 1036800
model_completed: 0
total_length_of_simulation: 1728000
result_base_name: oifs_43r3_bl_a2y9_2016092300_20_1017_12285652_0_r1821875991
The child process has been launched with process id: 8050
Executing the command: /var/lib/boinc/slots/18/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:antares:1:1:8050:8050] [20240610:162012:1718029212:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2091] fp = 0x314f440
[EC_DRHOOK:antares:1:1:8050:8050] [20240610:162012:1718029212:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:antares:1:1:8050:8050] [20240610:162012:1718029212:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:antares:1:1:8050:8050] [20240610:162012:1718029212:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:antares:1:1:8050:8050] [20240610:162012:1718029212:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:antares:1:1:8050:8050] [20240610:162012:1718029212:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:antares:1:1:8050:8050] [20240610:162012:1718029212:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:antares:1:1:8050:8050] [20240610:162012:1718029212:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:antares:1:1:8050:8050] [20240610:162012:1718029212:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:antares:1:1:8050:8050] [20240610:162012:1718029212:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:antares:1:1:8050:8050] [20240610:162012:1718029212:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:antares:1:1:8050:8050] [20240610:162012:1718029212:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:antares:1:1:8050:8050] [20240610:162012:1718029212:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:antares:1:1:8050:8050] [20240610:162012:1718029212:0.000] [ignore_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:antares:1:1:8050:8050] [20240610:162012:1718029212:0.000] [restore_default_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:antares:1:1:8050:8050] [20240610:162012:1718029212:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:antares:1:1:8050:8050] [20240610:162012:1718029212:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:antares:1:1:8050:8050] [20240610:162012:1718029212:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:antares:1:1:8050:8050] [20240610:162012:1718029212:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x820cf0 (old at (nil))
[EC_DRHOOK:antares:1:1:8050:8050] [20240610:162012:1718029212:0.000] [trapfpe_treatment@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:antares:1:1:8050:8050] [20240610:162012:1718029212:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:antares:1:1:8050:8050] [20240610:162012:1718029212:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:antares:1:1:8050:8050] [20240610:162012:1718029212:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x820cf0 (old at (nil))
[EC_DRHOOK:antares:1:1:8050:8050] [20240610:162012:1718029212:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:antares:1:1:8050:8050] [20240610:162012:1718029212:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:antares:1:1:8050:8050] [20240610:162012:1718029212:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:antares:1:1:8050:8050] [20240610:162012:1718029212:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x820cf0 (old at (nil))
[EC_DRHOOK:antares:1:1:8050:8050] [20240610:162012:1718029212:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x820cf0 (old at (nil))
[EC_DRHOOK:antares:1:1:8050:8050] [20240610:162012:1718029212:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:antares:1:1:8050:8050] [20240610:162012:1718029212:0.000] [catch_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  16:20:57 STEP 1152 H= 288:00 +CPU= 36.437
  16:21:30 STEP 1153 H= 288:15 +CPU= 32.730
  16:22:03 STEP 1154 H= 288:30 +CPU= 32.782
  16:22:32 STEP 1155 H= 288:45 +CPU= 29.004
  16:23:22 STEP 1156 H= 289:00 +CPU= 45.625
  16:24:20 STEP 1157 H= 289:15 +CPU= 52.504
  16:25:20 STEP 1158 H= 289:30 +CPU= 55.239
  16:26:18 STEP 1159 H= 289:45 +CPU= 53.121
  16:27:20 STEP 1160 H= 290:00 +CPU= 54.890
  16:28:23 STEP 1161 H= 290:15 +CPU= 54.739
  16:29:24 STEP 1162 H= 290:30 +CPU= 54.399
  16:30:25 STEP 1163 H= 290:45 +CPU= 54.027
  16:31:39 STEP 1164 H= 291:00 +CPU= 66.105
Moving to projects directory: /var/lib/boinc/slots/18/ICMGGh7zg+001164
Moving to projects directory: /var/lib/boinc/slots/18/ICMSHh7zg+001164
Moving to projects directory: /var/lib/boinc/slots/18/ICMUAh7zg+001164
  16:32:43 STEP 1165 H= 291:15 +CPU= 56.257
  16:33:46 STEP 1166 H= 291:30 +CPU= 56.466
  16:34:46 STEP 1167 H= 291:45 +CPU= 52.731
  16:35:50 STEP 1168 H= 292:00 +CPU= 55.151
  16:36:54 STEP 1169 H= 292:15 +CPU= 56.277
  16:37:56 STEP 1170 H= 292:30 +CPU= 53.930
  16:39:00 STEP 1171 H= 292:45 +CPU= 57.527
  16:40:05 STEP 1172 H= 293:00 +CPU= 58.804
  16:41:09 STEP 1173 H= 293:15 +CPU= 55.702
  16:42:14 STEP 1174 H= 293:30 +CPU= 55.639
  16:43:20 STEP 1175 H= 293:45 +CPU= 56.895
  16:44:33 STEP 1176 H= 294:00 +CPU= 66.691
Moving to projects directory: /var/lib/boinc/slots/18/ICMGGh7zg+001176
Moving to projects directory: /var/lib/boinc/slots/18/ICMSHh7zg+001176
Moving to projects directory: /var/lib/boinc/slots/18/ICMUAh7zg+001176
  16:45:31 STEP 1177 H= 294:15 +CPU= 53.025
  16:46:31 STEP 1178 H= 294:30 +CPU= 53.417
  16:47:32 STEP 1179 H= 294:45 +CPU= 55.134
  16:48:33 STEP 1180 H= 295:00 +CPU= 55.129
  16:49:35 STEP 1181 H= 295:15 +CPU= 55.208
  16:50:36 STEP 1182 H= 295:30 +CPU= 54.939
  16:51:35 STEP 1183 H= 295:45 +CPU= 53.843
  16:52:42 STEP 1184 H= 296:00 +CPU= 60.289
  16:53:46 STEP 1185 H= 296:15 +CPU= 57.106
  16:54:50 STEP 1186 H= 296:30 +CPU= 58.706
  16:55:53 STEP 1187 H= 296:45 +CPU= 56.215
  16:57:09 STEP 1188 H= 297:00 +CPU= 69.587
Moving to projects directory: /var/lib/boinc/slots/18/ICMGGh7zg+001188
Moving to projects directory: /var/lib/boinc/slots/18/ICMSHh7zg+001188
Moving to projects directory: /var/lib/boinc/slots/18/ICMUAh7zg+001188
  16:58:12 STEP 1189 H= 297:15 +CPU= 56.184
  16:59:14 STEP 1190 H= 297:30 +CPU= 55.911
  17:00:19 STEP 1191 H= 297:45 +CPU= 58.818
  17:01:24 STEP 1192 H= 298:00 +CPU= 58.010
  17:02:25 STEP 1193 H= 298:15 +CPU= 55.009
  17:03:28 STEP 1194 H= 298:30 +CPU= 56.950
  17:04:27 STEP 1195 H= 298:45 +CPU= 56.966
  17:05:28 STEP 1196 H= 299:00 +CPU= 59.012
  17:06:29 STEP 1197 H= 299:15 +CPU= 58.852
  17:07:30 STEP 1198 H= 299:30 +CPU= 57.997
  17:08:24 STEP 1199 H= 299:45 +CPU= 52.679
  17:09:43 STEP 1200 H= 300:00 +CPU= 76.120
Moving to projects directory: /var/lib/boinc/slots/18/ICMGGh7zg+001200
Moving to projects directory: /var/lib/boinc/slots/18/ICMSHh7zg+001200
Moving to projects directory: /var/lib/boinc/slots/18/ICMUAh7zg+001200
  17:10:43 STEP 1201 H= 300:15 +CPU= 57.575
  17:11:48 STEP 1202 H= 300:30 +CPU= 60.295
  17:12:45 STEP 1203 H= 300:45 +CPU= 54.683
  17:13:44 STEP 1204 H= 301:00 +CPU= 56.475
  17:14:41 STEP 1205 H= 301:15 +CPU= 54.845
  17:15:49 STEP 1206 H= 301:30 +CPU= 64.615
  17:16:53 STEP 1207 H= 301:45 +CPU= 60.827
  17:17:55 STEP 1208 H= 302:00 +CPU= 59.692
  17:18:56 STEP 1209 H= 302:15 +CPU= 58.810
  17:19:59 STEP 1210 H= 302:30 +CPU= 61.052
  17:21:03 STEP 1211 H= 302:45 +CPU= 59.685
  17:22:14 STEP 1212 H= 303:00 +CPU= 68.426
Moving to projects directory: /var/lib/boinc/slots/18/ICMGGh7zg+001212
Moving to projects directory: /var/lib/boinc/slots/18/ICMSHh7zg+001212
Moving to projects directory: /var/lib/boinc/slots/18/ICMUAh7zg+001212
  17:23:13 STEP 1213 H= 303:15 +CPU= 57.150
  17:24:08 STEP 1214 H= 303:30 +CPU= 54.142
  17:25:10 STEP 1215 H= 303:45 +CPU= 59.290
  17:26:14 STEP 1216 H= 304:00 +CPU= 61.612
  17:27:19 STEP 1217 H= 304:15 +CPU= 62.697
  17:28:21 STEP 1218 H= 304:30 +CPU= 60.155
  17:29:30 STEP 1219 H= 304:45 +CPU= 65.357
  17:30:35 STEP 1220 H= 305:00 +CPU= 62.384
  17:31:38 STEP 1221 H= 305:15 +CPU= 60.889
  17:32:44 STEP 1222 H= 305:30 +CPU= 62.963
  17:33:53 STEP 1223 H= 305:45 +CPU= 65.369
  17:35:12 STEP 1224 H= 306:00 +CPU= 75.370
Moving to projects directory: /var/lib/boinc/slots/18/ICMGGh7zg+001224
Moving to projects directory: /var/lib/boinc/slots/18/ICMSHh7zg+001224
Moving to projects directory: /var/lib/boinc/slots/18/ICMUAh7zg+001224
  17:36:17 STEP 1225 H= 306:15 +CPU= 62.678
  17:37:24 STEP 1226 H= 306:30 +CPU= 63.828
  17:38:27 STEP 1227 H= 306:45 +CPU= 60.937
  17:39:30 STEP 1228 H= 307:00 +CPU= 60.635
  17:40:38 STEP 1229 H= 307:15 +CPU= 64.163
  17:41:41 STEP 1230 H= 307:30 +CPU= 60.733
  17:42:42 STEP 1231 H= 307:45 +CPU= 58.832
  17:43:43 STEP 1232 H= 308:00 +CPU= 58.675
  17:44:49 STEP 1233 H= 308:15 +CPU= 62.817
  17:45:51 STEP 1234 H= 308:30 +CPU= 59.766
  17:46:55 STEP 1235 H= 308:45 +CPU= 61.841
  17:48:06 STEP 1236 H= 309:00 +CPU= 68.755
Moving to projects directory: /var/lib/boinc/slots/18/ICMGGh7zg+001236
Moving to projects directory: /var/lib/boinc/slots/18/ICMSHh7zg+001236
Moving to projects directory: /var/lib/boinc/slots/18/ICMUAh7zg+001236
  17:49:05 STEP 1237 H= 309:15 +CPU= 56.930
  17:50:05 STEP 1238 H= 309:30 +CPU= 58.515
  17:51:06 STEP 1239 H= 309:45 +CPU= 59.189
  17:52:13 STEP 1240 H= 310:00 +CPU= 64.911
  17:53:17 STEP 1241 H= 310:15 +CPU= 60.321
  17:54:19 STEP 1242 H= 310:30 +CPU= 60.596
  17:55:26 STEP 1243 H= 310:45 +CPU= 63.105
  17:56:31 STEP 1244 H= 311:00 +CPU= 61.245
  17:57:32 STEP 1245 H= 311:15 +CPU= 57.631
  17:58:36 STEP 1246 H= 311:30 +CPU= 61.800
  17:59:39 STEP 1247 H= 311:45 +CPU= 60.181
  18:01:01 STEP 1248 H= 312:00 +CPU= 78.058
Moving to projects directory: /var/lib/boinc/slots/18/ICMGGh7zg+001248
Moving to projects directory: /var/lib/boinc/slots/18/ICMSHh7zg+001248
Moving to projects directory: /var/lib/boinc/slots/18/ICMUAh7zg+001248
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMGGh7zg+001152
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMSHh7zg+001152
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMUAh7zg+001152
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMGGh7zg+001164
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMSHh7zg+001164
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMUAh7zg+001164
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMGGh7zg+001176
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMSHh7zg+001176
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMUAh7zg+001176
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMGGh7zg+001188
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMSHh7zg+001188
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMUAh7zg+001188
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMGGh7zg+001200
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMSHh7zg+001200
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMUAh7zg+001200
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMGGh7zg+001212
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMSHh7zg+001212
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMUAh7zg+001212
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMGGh7zg+001224
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMSHh7zg+001224
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMUAh7zg+001224
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMGGh7zg+001236
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMSHh7zg+001236
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMUAh7zg+001236
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_a2y9_2016092300_20_1017_12285652_0_r1821875991_12.zip
Uploading the intermediate file: upload_file_12.zip
  18:02:03 STEP 1249 H= 312:15 +CPU= 58.523
  18:03:05 STEP 1250 H= 312:30 +CPU= 59.615
  18:04:09 STEP 1251 H= 312:45 +CPU= 61.323
  18:05:18 STEP 1252 H= 313:00 +CPU= 65.535
  18:06:20 STEP 1253 H= 313:15 +CPU= 59.576
  18:07:28 STEP 1254 H= 313:30 +CPU= 64.558
  18:08:33 STEP 1255 H= 313:45 +CPU= 62.221
  18:09:38 STEP 1256 H= 314:00 +CPU= 61.498
  18:10:43 STEP 1257 H= 314:15 +CPU= 62.004
  18:11:46 STEP 1258 H= 314:30 +CPU= 60.409
  18:12:42 STEP 1259 H= 314:45 +CPU= 55.238
  18:13:59 STEP 1260 H= 315:00 +CPU= 73.254
Moving to projects directory: /var/lib/boinc/slots/18/ICMGGh7zg+001260
Moving to projects directory: /var/lib/boinc/slots/18/ICMSHh7zg+001260
Moving to projects directory: /var/lib/boinc/slots/18/ICMUAh7zg+001260
  18:15:02 STEP 1261 H= 315:15 +CPU= 60.436
  18:16:05 STEP 1262 H= 315:30 +CPU= 61.441
  18:17:07 STEP 1263 H= 315:45 +CPU= 59.720
  18:18:11 STEP 1264 H= 316:00 +CPU= 62.056
  18:19:12 STEP 1265 H= 316:15 +CPU= 58.856
  18:20:16 STEP 1266 H= 316:30 +CPU= 61.610
  18:21:22 STEP 1267 H= 316:45 +CPU= 61.326
  18:22:27 STEP 1268 H= 317:00 +CPU= 61.273
  18:23:31 STEP 1269 H= 317:15 +CPU= 61.366
  18:24:34 STEP 1270 H= 317:30 +CPU= 61.237
  18:25:38 STEP 1271 H= 317:45 +CPU= 62.287
  18:26:58 STEP 1272 H= 318:00 +CPU= 76.475
Moving to projects directory: /var/lib/boinc/slots/18/ICMGGh7zg+001272
Moving to projects directory: /var/lib/boinc/slots/18/ICMSHh7zg+001272
Moving to projects directory: /var/lib/boinc/slots/18/ICMUAh7zg+001272
  18:27:59 STEP 1273 H= 318:15 +CPU= 59.083
  18:29:00 STEP 1274 H= 318:30 +CPU= 59.323
  18:30:00 STEP 1275 H= 318:45 +CPU= 58.369
  18:31:01 STEP 1276 H= 319:00 +CPU= 60.115
  18:32:00 STEP 1277 H= 319:15 +CPU= 56.559
  18:33:00 STEP 1278 H= 319:30 +CPU= 58.724
  18:33:57 STEP 1279 H= 319:45 +CPU= 55.306
  18:35:00 STEP 1280 H= 320:00 +CPU= 60.548
  18:35:59 STEP 1281 H= 320:15 +CPU= 57.288
  18:37:01 STEP 1282 H= 320:30 +CPU= 59.264
  18:38:10 STEP 1283 H= 320:45 +CPU= 65.951
  18:39:25 STEP 1284 H= 321:00 +CPU= 72.274
Moving to projects directory: /var/lib/boinc/slots/18/ICMGGh7zg+001284
Moving to projects directory: /var/lib/boinc/slots/18/ICMSHh7zg+001284
Moving to projects directory: /var/lib/boinc/slots/18/ICMUAh7zg+001284
  18:40:34 STEP 1285 H= 321:15 +CPU= 66.270
  18:41:38 STEP 1286 H= 321:30 +CPU= 61.540
  18:42:42 STEP 1287 H= 321:45 +CPU= 61.272
  18:43:46 STEP 1288 H= 322:00 +CPU= 61.798
  18:44:49 STEP 1289 H= 322:15 +CPU= 59.654
  18:45:55 STEP 1290 H= 322:30 +CPU= 62.918
  18:46:58 STEP 1291 H= 322:45 +CPU= 61.392
  18:48:07 STEP 1292 H= 323:00 +CPU= 65.344
  18:49:06 STEP 1293 H= 323:15 +CPU= 56.919
  18:50:09 STEP 1294 H= 323:30 +CPU= 60.389
  18:51:14 STEP 1295 H= 323:45 +CPU= 60.912
  18:52:44 STEP 1296 H= 324:00 +CPU= 86.163
Moving to projects directory: /var/lib/boinc/slots/18/ICMGGh7zg+001296
Moving to projects directory: /var/lib/boinc/slots/18/ICMSHh7zg+001296
Moving to projects directory: /var/lib/boinc/slots/18/ICMUAh7zg+001296
  18:53:50 STEP 1297 H= 324:15 +CPU= 62.614
  18:54:48 STEP 1298 H= 324:30 +CPU= 55.183
  18:55:50 STEP 1299 H= 324:45 +CPU= 59.917
  18:57:01 STEP 1300 H= 325:00 +CPU= 63.249
  18:58:12 STEP 1301 H= 325:15 +CPU= 60.686
  18:59:15 STEP 1302 H= 325:30 +CPU= 59.620
  19:00:18 STEP 1303 H= 325:45 +CPU= 58.737
  19:01:21 STEP 1304 H= 326:00 +CPU= 58.963
  19:02:27 STEP 1305 H= 326:15 +CPU= 61.305
  19:03:33 STEP 1306 H= 326:30 +CPU= 62.564
  19:04:35 STEP 1307 H= 326:45 +CPU= 59.027
  19:05:55 STEP 1308 H= 327:00 +CPU= 76.828
Moving to projects directory: /var/lib/boinc/slots/18/ICMGGh7zg+001308
Moving to projects directory: /var/lib/boinc/slots/18/ICMSHh7zg+001308
Moving to projects directory: /var/lib/boinc/slots/18/ICMUAh7zg+001308
  19:07:00 STEP 1309 H= 327:15 +CPU= 62.149
  19:08:02 STEP 1310 H= 327:30 +CPU= 59.987
  19:09:05 STEP 1311 H= 327:45 +CPU= 60.063
  19:10:09 STEP 1312 H= 328:00 +CPU= 61.350
  19:11:16 STEP 1313 H= 328:15 +CPU= 62.787
  19:12:22 STEP 1314 H= 328:30 +CPU= 63.484
  19:13:27 STEP 1315 H= 328:45 +CPU= 62.025
  19:14:26 STEP 1316 H= 329:00 +CPU= 57.477
  19:15:28 STEP 1317 H= 329:15 +CPU= 59.638
  19:16:34 STEP 1318 H= 329:30 +CPU= 62.542
  19:17:37 STEP 1319 H= 329:45 +CPU= 60.316
  19:18:53 STEP 1320 H= 330:00 +CPU= 74.057
Moving to projects directory: /var/lib/boinc/slots/18/ICMGGh7zg+001320
Moving to projects directory: /var/lib/boinc/slots/18/ICMSHh7zg+001320
Moving to projects directory: /var/lib/boinc/slots/18/ICMUAh7zg+001320
  19:19:56 STEP 1321 H= 330:15 +CPU= 60.549
  19:20:57 STEP 1322 H= 330:30 +CPU= 57.647
  19:22:10 STEP 1323 H= 330:45 +CPU= 59.318
  19:23:26 STEP 1324 H= 331:00 +CPU= 61.204
  19:24:33 STEP 1325 H= 331:15 +CPU= 61.691
  19:25:39 STEP 1326 H= 331:30 +CPU= 61.331
  19:26:40 STEP 1327 H= 331:45 +CPU= 59.143
  19:27:45 STEP 1328 H= 332:00 +CPU= 62.374
  19:28:53 STEP 1329 H= 332:15 +CPU= 64.907
  19:29:53 STEP 1330 H= 332:30 +CPU= 57.631
  19:30:54 STEP 1331 H= 332:45 +CPU= 58.537
  19:32:13 STEP 1332 H= 333:00 +CPU= 75.369
Moving to projects directory: /var/lib/boinc/slots/18/ICMGGh7zg+001332
Moving to projects directory: /var/lib/boinc/slots/18/ICMSHh7zg+001332
Moving to projects directory: /var/lib/boinc/slots/18/ICMUAh7zg+001332
  19:33:15 STEP 1333 H= 333:15 +CPU= 59.491
  19:34:16 STEP 1334 H= 333:30 +CPU= 57.779
  19:35:18 STEP 1335 H= 333:45 +CPU= 61.065
  19:36:16 STEP 1336 H= 334:00 +CPU= 56.445
  19:37:19 STEP 1337 H= 334:15 +CPU= 61.699
  19:38:15 STEP 1338 H= 334:30 +CPU= 55.263
  19:39:09 STEP 1339 H= 334:45 +CPU= 53.189
  19:39:59 STEP 1340 H= 335:00 +CPU= 48.927
  19:40:56 STEP 1341 H= 335:15 +CPU= 56.877
  19:41:53 STEP 1342 H= 335:30 +CPU= 54.960
  19:42:50 STEP 1343 H= 335:45 +CPU= 54.055
  19:44:03 STEP 1344 H= 336:00 +CPU= 70.182
Moving to projects directory: /var/lib/boinc/slots/18/ICMGGh7zg+001344
Moving to projects directory: /var/lib/boinc/slots/18/ICMSHh7zg+001344
Moving to projects directory: /var/lib/boinc/slots/18/ICMUAh7zg+001344
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMGGh7zg+001248
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMSHh7zg+001248
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMUAh7zg+001248
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMGGh7zg+001260
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMSHh7zg+001260
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMUAh7zg+001260
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMGGh7zg+001272
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMSHh7zg+001272
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMUAh7zg+001272
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMGGh7zg+001284
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMSHh7zg+001284
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMUAh7zg+001284
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMGGh7zg+001296
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMSHh7zg+001296
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMUAh7zg+001296
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMGGh7zg+001308
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMSHh7zg+001308
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMUAh7zg+001308
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMGGh7zg+001320
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMSHh7zg+001320
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMUAh7zg+001320
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMGGh7zg+001332
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMSHh7zg+001332
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMUAh7zg+001332
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_a2y9_2016092300_20_1017_12285652_0_r1821875991_13.zip
Uploading the intermediate file: upload_file_13.zip
  19:44:57 STEP 1345 H= 336:15 +CPU= 51.730
  19:45:49 STEP 1346 H= 336:30 +CPU= 50.130
  19:46:44 STEP 1347 H= 336:45 +CPU= 52.693
  19:47:33 STEP 1348 H= 337:00 +CPU= 47.175
  19:48:26 STEP 1349 H= 337:15 +CPU= 51.735
  19:49:18 STEP 1350 H= 337:30 +CPU= 49.876
  19:50:14 STEP 1351 H= 337:45 +CPU= 54.318
  19:51:07 STEP 1352 H= 338:00 +CPU= 51.045
  19:52:04 STEP 1353 H= 338:15 +CPU= 55.045
  19:52:57 STEP 1354 H= 338:30 +CPU= 51.180
  19:53:50 STEP 1355 H= 338:45 +CPU= 50.617
  19:54:55 STEP 1356 H= 339:00 +CPU= 62.192
Moving to projects directory: /var/lib/boinc/slots/18/ICMGGh7zg+001356
Moving to projects directory: /var/lib/boinc/slots/18/ICMSHh7zg+001356
Moving to projects directory: /var/lib/boinc/slots/18/ICMUAh7zg+001356
  19:55:50 STEP 1357 H= 339:15 +CPU= 52.329
  19:56:47 STEP 1358 H= 339:30 +CPU= 55.145
  19:57:42 STEP 1359 H= 339:45 +CPU= 52.961
  19:58:32 STEP 1360 H= 340:00 +CPU= 48.947
  19:59:21 STEP 1361 H= 340:15 +CPU= 47.384
  20:00:14 STEP 1362 H= 340:30 +CPU= 50.776
  20:01:14 STEP 1363 H= 340:45 +CPU= 56.931
  20:02:12 STEP 1364 H= 341:00 +CPU= 56.239
  20:03:05 STEP 1365 H= 341:15 +CPU= 51.366
  20:03:56 STEP 1366 H= 341:30 +CPU= 49.166
  20:04:53 STEP 1367 H= 341:45 +CPU= 54.964
  20:05:57 STEP 1368 H= 342:00 +CPU= 63.139
Moving to projects directory: /var/lib/boinc/slots/18/ICMGGh7zg+001368
Moving to projects directory: /var/lib/boinc/slots/18/ICMSHh7zg+001368
Moving to projects directory: /var/lib/boinc/slots/18/ICMUAh7zg+001368
  20:06:52 STEP 1369 H= 342:15 +CPU= 52.332
  20:07:44 STEP 1370 H= 342:30 +CPU= 50.750
  20:08:41 STEP 1371 H= 342:45 +CPU= 53.726
  20:09:31 STEP 1372 H= 343:00 +CPU= 48.541
  20:10:33 STEP 1373 H= 343:15 +CPU= 59.604
  20:11:24 STEP 1374 H= 343:30 +CPU= 48.359
  20:12:15 STEP 1375 H= 343:45 +CPU= 50.453
  20:13:12 STEP 1376 H= 344:00 +CPU= 53.821
  20:14:07 STEP 1377 H= 344:15 +CPU= 52.679
  20:15:05 STEP 1378 H= 344:30 +CPU= 55.688
  20:16:03 STEP 1379 H= 344:45 +CPU= 56.861
  20:16:59 STEP 1380 H= 345:00 +CPU= 55.837
Moving to projects directory: /var/lib/boinc/slots/18/ICMGGh7zg+001380
Moving to projects directory: /var/lib/boinc/slots/18/ICMSHh7zg+001380
Moving to projects directory: /var/lib/boinc/slots/18/ICMUAh7zg+001380
  20:17:55 STEP 1381 H= 345:15 +CPU= 53.927
  20:18:51 STEP 1382 H= 345:30 +CPU= 53.013
  20:19:44 STEP 1383 H= 345:45 +CPU= 50.686
  20:20:33 STEP 1384 H= 346:00 +CPU= 46.985
  20:21:25 STEP 1385 H= 346:15 +CPU= 50.738
  20:22:19 STEP 1386 H= 346:30 +CPU= 51.437
  20:23:07 STEP 1387 H= 346:45 +CPU= 46.828
  20:23:59 STEP 1388 H= 347:00 +CPU= 50.623
  20:24:50 STEP 1389 H= 347:15 +CPU= 49.479
  20:25:46 STEP 1390 H= 347:30 +CPU= 53.668
  20:26:40 STEP 1391 H= 347:45 +CPU= 52.814
  20:27:52 STEP 1392 H= 348:00 +CPU= 68.110
Moving to projects directory: /var/lib/boinc/slots/18/ICMGGh7zg+001392
Moving to projects directory: /var/lib/boinc/slots/18/ICMSHh7zg+001392
Moving to projects directory: /var/lib/boinc/slots/18/ICMUAh7zg+001392
  20:28:46 STEP 1393 H= 348:15 +CPU= 52.295
  20:29:41 STEP 1394 H= 348:30 +CPU= 52.544
  20:30:37 STEP 1395 H= 348:45 +CPU= 53.702
  20:31:34 STEP 1396 H= 349:00 +CPU= 55.137
  20:32:23 STEP 1397 H= 349:15 +CPU= 47.639
  20:33:15 STEP 1398 H= 349:30 +CPU= 49.957
  20:34:08 STEP 1399 H= 349:45 +CPU= 51.197
  20:35:01 STEP 1400 H= 350:00 +CPU= 50.691
  20:35:54 STEP 1401 H= 350:15 +CPU= 50.628
  20:36:44 STEP 1402 H= 350:30 +CPU= 49.356
  20:37:38 STEP 1403 H= 350:45 +CPU= 52.407
  20:38:36 STEP 1404 H= 351:00 +CPU= 56.141
Moving to projects directory: /var/lib/boinc/slots/18/ICMGGh7zg+001404
Moving to projects directory: /var/lib/boinc/slots/18/ICMSHh7zg+001404
Moving to projects directory: /var/lib/boinc/slots/18/ICMUAh7zg+001404
  20:39:27 STEP 1405 H= 351:15 +CPU= 49.787
  20:40:18 STEP 1406 H= 351:30 +CPU= 48.408
  20:41:06 STEP 1407 H= 351:45 +CPU= 47.201
  20:42:00 STEP 1408 H= 352:00 +CPU= 51.434
  20:42:51 STEP 1409 H= 352:15 +CPU= 48.979
  20:43:36 STEP 1410 H= 352:30 +CPU= 43.825
  20:44:26 STEP 1411 H= 352:45 +CPU= 49.379
  20:45:17 STEP 1412 H= 353:00 +CPU= 48.819
  20:46:04 STEP 1413 H= 353:15 +CPU= 46.627
  20:46:51 STEP 1414 H= 353:30 +CPU= 45.827
  20:47:43 STEP 1415 H= 353:45 +CPU= 50.674
  20:48:43 STEP 1416 H= 354:00 +CPU= 58.498
Moving to projects directory: /var/lib/boinc/slots/18/ICMGGh7zg+001416
Moving to projects directory: /var/lib/boinc/slots/18/ICMSHh7zg+001416
Moving to projects directory: /var/lib/boinc/slots/18/ICMUAh7zg+001416
  20:49:38 STEP 1417 H= 354:15 +CPU= 54.172
  20:50:30 STEP 1418 H= 354:30 +CPU= 50.903
  20:51:16 STEP 1419 H= 354:45 +CPU= 44.483
  20:51:54 STEP 1420 H= 355:00 +CPU= 37.849
  20:52:46 STEP 1421 H= 355:15 +CPU= 51.370
  20:53:30 STEP 1422 H= 355:30 +CPU= 43.040
  20:54:14 STEP 1423 H= 355:45 +CPU= 43.397
  20:55:04 STEP 1424 H= 356:00 +CPU= 48.587
  20:55:47 STEP 1425 H= 356:15 +CPU= 41.739
  20:56:40 STEP 1426 H= 356:30 +CPU= 51.577
  20:57:24 STEP 1427 H= 356:45 +CPU= 43.159
  20:58:26 STEP 1428 H= 357:00 +CPU= 60.097
Moving to projects directory: /var/lib/boinc/slots/18/ICMGGh7zg+001428
Moving to projects directory: /var/lib/boinc/slots/18/ICMSHh7zg+001428
Moving to projects directory: /var/lib/boinc/slots/18/ICMUAh7zg+001428
  20:59:15 STEP 1429 H= 357:15 +CPU= 47.493
  21:00:08 STEP 1430 H= 357:30 +CPU= 51.156
  21:00:53 STEP 1431 H= 357:45 +CPU= 45.150
  21:01:45 STEP 1432 H= 358:00 +CPU= 49.937
  21:02:25 STEP 1433 H= 358:15 +CPU= 39.259
  21:03:08 STEP 1434 H= 358:30 +CPU= 42.222
  21:03:57 STEP 1435 H= 358:45 +CPU= 47.118
  21:04:45 STEP 1436 H= 359:00 +CPU= 47.252
  21:05:25 STEP 1437 H= 359:15 +CPU= 39.772
  21:06:21 STEP 1438 H= 359:30 +CPU= 55.042
  21:07:26 STEP 1439 H= 359:45 +CPU= 63.662
  21:08:24 STEP 1440 H= 360:00 +CPU= 57.176
..The child process terminated with status: 0
>>> Printing last 70 lines from file: NODE.001_01

 
DDH------------------------------------------------------------------------
I                                                                         I
I TIME =  360.000H DDH * Diagnostic Var.,     AREA                GLOBAL  I
I                                                                         I
I-------------------------------------------------------------------------I
I   VARIABLE   I      Initial value         I        Current value        I
I-------------------------------------------------------------------------I
I  OMEGA (P/S) I     -0.1764572380E-20      I      -0.2303947284E-04      I
ICloud fractionI      0.0000000000E+00      I       0.9500970302E-01      I
I  Relat. Hum. I      0.5228859684E+00      I       0.4997645229E+00      I
I  PASS 01     I      0.0000000000E+00      I       0.1409617159E+04      I
DDH------------------------------------------------------------------------
  
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1440 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001440 MODE=a
 IO-STREAM CLOSED - ICMGGh7zg+001440
 MPI-TASK:   1 -    325358276 BYTES IN      7 RECORDS TRANSFERRED IN    0.0011 SECONDS295780.3 Mbytes/s, TOTAL TIME=    0.0140( 7.9%)
 21:08:24 STEP 1440 H= 360:00 +CPU= 57.176
 END CNT3
 NSTEP =  1440 CNT0   000000000  41.285  41.285  41.656   0.991               0               0               0               0               0
IO-STREAM STATISTICS, TOTAL NO OF REC/BYTES READ           46/  1069125708 WRITTEN         8580/************
 -TASK-OPENED-OPEN-RECS IN -KBYTE IN    -RECS OUT-KBYTE OUT-WALL   -WALL IN-WALL
  OU-TOT IN -TOT OUT
   1    153    0       46      1044068     8580 ************    28.8     4.0    24.8     6.3    56.6
===-=== START OF TIMING STATISTICS ===-===
 
STATS FOR ALL TASKS
 NUM ROUTINE                                     CALLS  MEAN(ms)   MAX(ms)   FRAC(%)  UNBAL(%)
   0 CNT0     - COMPLETE EXECUTION                   1 ********* *********    100.00      0.00
   1 CNT4     - FORWARD INTEGRATION                  1 ********* *********     99.95      0.00
   8 SCAN2M - GRID-POINT DYNAMICS                  313   10606.4   10606.4     19.20      0.00
   9 SPCM     - SPECTRAL COMP.                     288    2675.9    2675.9      4.46      0.00
  10 SCAN2M - PHYSICS                              289   27017.5   27017.5     45.15      0.00
  11 IOPACK   - OUTPUT P.P. RESULTS                 24    1622.1    1622.1      0.23      0.00
  12 SPNORM   - SPECTRAL NORM COMP.                 13     131.1     131.1      0.01      0.00
  17 GRIDFPOS IN CNT4                               24      58.7      58.7      0.01      0.00
  27 RESTART FILES - WRITING                         6    6483.7    6483.7      0.22      0.00
  28 RESTART FILES - READING                         1    8601.2    8601.2      0.05      0.00
  29 SU4FPOS IN CNT4                                24       0.5       0.5      0.00      0.00
  30 DYNFPOS IN CNT4                                24   11087.6   11087.6      1.54      0.00
  31 POSDDH IN STEPO                                 2       3.1       3.1      0.00      0.00
  37 CPGLAG   - SL COMPUTATIONS                    289   11557.9   11557.9     19.32      0.00
  39 SU0YOMB                                         1     425.9     425.9      0.00      0.00
  51 SCAN2M   - SL COMM. PART 1                    289      42.9      42.9      0.07      0.00
  54 SPCM     - M TO S/S TO M TRANSP.              288    1249.8    1249.8      2.08      0.00
  55 SPCIMPF  - S TO M/M TO S TRANSP.              288     127.3     127.3      0.21      0.00
  56 SPNORM   - SPECTRAL NORM COMM.                 13       2.2       2.2      0.00      0.00
 102 LTINV_CTL   - INVERSE LEGENDRE TRANSFORM      628     792.8     792.8      2.88      0.00
 103 LTDIR_CTL   - DIRECT LEGENDRE TRANSFORM       600     429.4     429.4      1.49      0.00
 106 FTDIR_CTL   - DIRECT FOURIER TRANSFORM        600     157.8     157.8      0.55      0.00
 107 FTINV_CTL   - INVERSE FOURIER TRANSFORM       628     258.7     258.7      0.94      0.00
 140 SULEG       - COMP. OF LEGENDRE POL.            1      26.4      26.4      0.00      0.00
 152 LTINV_CTL   - M TO L TRANSPOSITION            628     143.1     143.1      0.52      0.00
 153 LTDIR_CTL   - L TO M TRANSPOSITION            600      89.1      89.1      0.31      0.00
 157 FTINV_CTL   - L TO G TRANSPOSITION            628     198.5     198.5      0.72      0.00
 158 FTDIR_CTL   - G TO L TRANSPOSITION            600      99.4      99.4      0.35      0.00
 400 GSTATS                                      34412       0.0       0.0      0.00      0.00
 401 GSTATS HOOK                                 31964       0.0       0.0      0.00      0.00
TOTAL MEASURED IMBALANCE =       0.0 SECONDS,  0.0 PERCENT
TOTAL WALLCLOCK TIME    17292.0 CPU TIME   16442.5 VECTOR TIME    16442.5


===-=== END   OF TIMING STATISTICS ===-===
FORECAST DAYS PER DAY       75.0
  *** END CNT0 *** 
------------------------------------------------
Moving to projects directory: /var/lib/boinc/slots/18/ICMGGh7zg+001440
Moving to projects directory: /var/lib/boinc/slots/18/ICMSHh7zg+001440
Moving to projects directory: /var/lib/boinc/slots/18/ICMUAh7zg+001440
Adding to the zip: /var/lib/boinc/slots/18/NODE.001_01
Adding to the zip: /var/lib/boinc/slots/18/ifs.stat
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMGGh7zg+001344
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMSHh7zg+001344
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMUAh7zg+001344
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMGGh7zg+001356
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMSHh7zg+001356
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMUAh7zg+001356
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMGGh7zg+001368
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMSHh7zg+001368
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMUAh7zg+001368
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMGGh7zg+001380
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMSHh7zg+001380
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMUAh7zg+001380
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMGGh7zg+001392
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMSHh7zg+001392
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMUAh7zg+001392
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMGGh7zg+001404
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMSHh7zg+001404
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMUAh7zg+001404
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMGGh7zg+001416
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMSHh7zg+001416
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMUAh7zg+001416
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMGGh7zg+001428
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMSHh7zg+001428
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMUAh7zg+001428
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMGGh7zg+001440
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMSHh7zg+001440
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285652/ICMUAh7zg+001440
Zipping up the final file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_a2y9_2016092300_20_1017_12285652_0_r1821875991_14.zip
Uploading the final file: upload_file_14.zip
Uploading trickle at timestep: 1295100
21:11:58 (7980): called boinc_finish(0)

</stderr_txt>
<message>
upload failure: <file_xfer_error>
  <file_name>oifs_43r3_bl_a2y9_2016092300_20_1017_12285652_0_r1821875991_15.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a2y9_2016092300_20_1017_12285652_0_r1821875991_16.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a2y9_2016092300_20_1017_12285652_0_r1821875991_17.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a2y9_2016092300_20_1017_12285652_0_r1821875991_18.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a2y9_2016092300_20_1017_12285652_0_r1821875991_19.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
</message>
]]
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
10 Jun 2024 19:37:51 1533094 22437259 oifs_43r3_bl_a2y9_2016092300_20_1017_12285652_0 1,295,100 72,857 0.0563


©2024 climateprediction.net