climateprediction.net home page
Task 22437318

Task 22437318

Name oifs_43r3_bl_a2zu_2016092300_20_1017_12285709_0
Workunit 12285709
Created 7 Jun 2024, 16:27:29 UTC
Sent 7 Jun 2024, 16:31:19 UTC
Report deadline 6 Aug 2024, 16:31:19 UTC
Received 10 Jun 2024, 19:37:15 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 0 (0x00000000)
Computer ID 1533094
Run time 20 hours 29 min 27 sec
CPU time 20 hours 12 min 50 sec
Validate state Invalid
Credit 1,976.32
Device peak FLOPS 7.75 GFLOPS
Application version OpenIFS 43r3 Baroclinic Lifecycle v1.13
x86_64-pc-linux-gnu
Peak working set size 5,540.26 MB
Peak swap size 5,980.80 MB
Peak disk usage 2,045.81 MB
Stderr
<core_client_version>7.24.1</core_client_version>
<![CDATA[
<stderr_txt>
ts/14/ICMGGh7zg+001008
Moving to projects directory: /var/lib/boinc/slots/14/ICMSHh7zg+001008
Moving to projects directory: /var/lib/boinc/slots/14/ICMUAh7zg+001008
  08:02:10 STEP 1009 H= 252:15 +CPU= 65.201
  08:03:15 STEP 1010 H= 252:30 +CPU= 62.560
  08:04:25 STEP 1011 H= 252:45 +CPU= 66.146
  08:05:35 STEP 1012 H= 253:00 +CPU= 67.422
  08:06:41 STEP 1013 H= 253:15 +CPU= 63.535
  08:07:48 STEP 1014 H= 253:30 +CPU= 63.977
  08:08:55 STEP 1015 H= 253:45 +CPU= 63.824
  08:10:01 STEP 1016 H= 254:00 +CPU= 65.285
  08:11:05 STEP 1017 H= 254:15 +CPU= 61.947
  08:12:08 STEP 1018 H= 254:30 +CPU= 61.132
  08:13:13 STEP 1019 H= 254:45 +CPU= 61.166
  08:14:35 STEP 1020 H= 255:00 +CPU= 77.957
Moving to projects directory: /var/lib/boinc/slots/14/ICMGGh7zg+001020
Moving to projects directory: /var/lib/boinc/slots/14/ICMSHh7zg+001020
Moving to projects directory: /var/lib/boinc/slots/14/ICMUAh7zg+001020
  08:15:42 STEP 1021 H= 255:15 +CPU= 64.281
  08:16:43 STEP 1022 H= 255:30 +CPU= 58.899
  08:17:47 STEP 1023 H= 255:45 +CPU= 62.095
  08:18:52 STEP 1024 H= 256:00 +CPU= 62.431
  08:19:58 STEP 1025 H= 256:15 +CPU= 64.364
  08:21:07 STEP 1026 H= 256:30 +CPU= 66.553
  08:22:12 STEP 1027 H= 256:45 +CPU= 63.115
  08:23:18 STEP 1028 H= 257:00 +CPU= 64.016
  08:24:24 STEP 1029 H= 257:15 +CPU= 61.795
  08:25:32 STEP 1030 H= 257:30 +CPU= 65.165
  08:26:34 STEP 1031 H= 257:45 +CPU= 59.651
  08:27:52 STEP 1032 H= 258:00 +CPU= 75.321
Moving to projects directory: /var/lib/boinc/slots/14/ICMGGh7zg+001032
Moving to projects directory: /var/lib/boinc/slots/14/ICMSHh7zg+001032
Moving to projects directory: /var/lib/boinc/slots/14/ICMUAh7zg+001032
  08:28:55 STEP 1033 H= 258:15 +CPU= 59.625
  08:29:58 STEP 1034 H= 258:30 +CPU= 61.716
  08:31:01 STEP 1035 H= 258:45 +CPU= 61.193
  08:32:11 STEP 1036 H= 259:00 +CPU= 67.048
  08:33:19 STEP 1037 H= 259:15 +CPU= 65.147
  08:34:29 STEP 1038 H= 259:30 +CPU= 65.323
  08:35:36 STEP 1039 H= 259:45 +CPU= 65.126
  08:36:38 STEP 1040 H= 260:00 +CPU= 61.897
  08:37:46 STEP 1041 H= 260:15 +CPU= 65.813
  08:38:50 STEP 1042 H= 260:30 +CPU= 62.443
  08:39:54 STEP 1043 H= 260:45 +CPU= 62.785
  08:41:16 STEP 1044 H= 261:00 +CPU= 79.492
Moving to projects directory: /var/lib/boinc/slots/14/ICMGGh7zg+001044
Moving to projects directory: /var/lib/boinc/slots/14/ICMSHh7zg+001044
Moving to projects directory: /var/lib/boinc/slots/14/ICMUAh7zg+001044
  08:42:19 STEP 1045 H= 261:15 +CPU= 62.090
  08:43:27 STEP 1046 H= 261:30 +CPU= 65.574
  08:44:29 STEP 1047 H= 261:45 +CPU= 61.035
  08:45:33 STEP 1048 H= 262:00 +CPU= 63.221
  08:46:33 STEP 1049 H= 262:15 +CPU= 59.045
  08:47:39 STEP 1050 H= 262:30 +CPU= 63.642
  08:48:41 STEP 1051 H= 262:45 +CPU= 60.977
  08:49:41 STEP 1052 H= 263:00 +CPU= 59.226
  08:50:46 STEP 1053 H= 263:15 +CPU= 62.543
  08:51:50 STEP 1054 H= 263:30 +CPU= 63.033
  08:52:57 STEP 1055 H= 263:45 +CPU= 65.259
  08:54:24 STEP 1056 H= 264:00 +CPU= 84.385
Moving to projects directory: /var/lib/boinc/slots/14/ICMGGh7zg+001056
Moving to projects directory: /var/lib/boinc/slots/14/ICMSHh7zg+001056
Moving to projects directory: /var/lib/boinc/slots/14/ICMUAh7zg+001056
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMGGh7zg+000960
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMSHh7zg+000960
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMUAh7zg+000960
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMGGh7zg+000972
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMSHh7zg+000972
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMUAh7zg+000972
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMGGh7zg+000984
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMSHh7zg+000984
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMUAh7zg+000984
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMGGh7zg+000996
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMSHh7zg+000996
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMUAh7zg+000996
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMGGh7zg+001008
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMSHh7zg+001008
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMUAh7zg+001008
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMGGh7zg+001020
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMSHh7zg+001020
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMUAh7zg+001020
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMGGh7zg+001032
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMSHh7zg+001032
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMUAh7zg+001032
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMGGh7zg+001044
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMSHh7zg+001044
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMUAh7zg+001044
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_a2zu_2016092300_20_1017_12285709_0_r295598715_10.zip
Uploading the intermediate file: upload_file_10.zip
  08:55:30 STEP 1057 H= 264:15 +CPU= 62.890
  08:56:36 STEP 1058 H= 264:30 +CPU= 64.056
  08:57:42 STEP 1059 H= 264:45 +CPU= 64.861
  08:58:50 STEP 1060 H= 265:00 +CPU= 66.119
  08:59:51 STEP 1061 H= 265:15 +CPU= 59.975
  09:00:53 STEP 1062 H= 265:30 +CPU= 60.264
  09:02:00 STEP 1063 H= 265:45 +CPU= 65.896
  09:03:03 STEP 1064 H= 266:00 +CPU= 61.719
  09:04:05 STEP 1065 H= 266:15 +CPU= 60.766
  09:05:12 STEP 1066 H= 266:30 +CPU= 65.398
  09:06:17 STEP 1067 H= 266:45 +CPU= 62.737
  09:07:38 STEP 1068 H= 267:00 +CPU= 78.647
Moving to projects directory: /var/lib/boinc/slots/14/ICMGGh7zg+001068
Moving to projects directory: /var/lib/boinc/slots/14/ICMSHh7zg+001068
Moving to projects directory: /var/lib/boinc/slots/14/ICMUAh7zg+001068
  09:08:46 STEP 1069 H= 267:15 +CPU= 64.237
  09:09:53 STEP 1070 H= 267:30 +CPU= 65.107
  09:11:02 STEP 1071 H= 267:45 +CPU= 65.231
  09:12:06 STEP 1072 H= 268:00 +CPU= 61.917
  09:13:08 STEP 1073 H= 268:15 +CPU= 59.941
  09:14:10 STEP 1074 H= 268:30 +CPU= 59.364
  09:15:18 STEP 1075 H= 268:45 +CPU= 66.324
  09:16:25 STEP 1076 H= 269:00 +CPU= 64.275
  09:17:30 STEP 1077 H= 269:15 +CPU= 62.433
  09:18:39 STEP 1078 H= 269:30 +CPU= 65.370
  09:19:47 STEP 1079 H= 269:45 +CPU= 65.440
  09:21:06 STEP 1080 H= 270:00 +CPU= 75.096
Moving to projects directory: /var/lib/boinc/slots/14/ICMGGh7zg+001080
Moving to projects directory: /var/lib/boinc/slots/14/ICMSHh7zg+001080
Moving to projects directory: /var/lib/boinc/slots/14/ICMUAh7zg+001080
  09:22:13 STEP 1081 H= 270:15 +CPU= 64.120
  09:23:17 STEP 1082 H= 270:30 +CPU= 61.712
  09:24:24 STEP 1083 H= 270:45 +CPU= 64.024
  09:25:29 STEP 1084 H= 271:00 +CPU= 61.893
  09:26:37 STEP 1085 H= 271:15 +CPU= 63.662
  09:27:41 STEP 1086 H= 271:30 +CPU= 63.175
  09:28:45 STEP 1087 H= 271:45 +CPU= 62.386
  09:29:53 STEP 1088 H= 272:00 +CPU= 64.218
  09:30:59 STEP 1089 H= 272:15 +CPU= 62.905
  09:32:08 STEP 1090 H= 272:30 +CPU= 64.054
  09:33:19 STEP 1091 H= 272:45 +CPU= 64.679
  09:34:47 STEP 1092 H= 273:00 +CPU= 79.559
Moving to projects directory: /var/lib/boinc/slots/14/ICMGGh7zg+001092
Moving to projects directory: /var/lib/boinc/slots/14/ICMSHh7zg+001092
Moving to projects directory: /var/lib/boinc/slots/14/ICMUAh7zg+001092
  09:35:55 STEP 1093 H= 273:15 +CPU= 63.627
  09:37:04 STEP 1094 H= 273:30 +CPU= 64.477
  09:38:14 STEP 1095 H= 273:45 +CPU= 64.545
  09:39:26 STEP 1096 H= 274:00 +CPU= 67.260
  09:40:33 STEP 1097 H= 274:15 +CPU= 63.408
  09:41:46 STEP 1098 H= 274:30 +CPU= 66.792
  09:42:52 STEP 1099 H= 274:45 +CPU= 63.530
  09:44:01 STEP 1100 H= 275:00 +CPU= 64.694
  09:45:08 STEP 1101 H= 275:15 +CPU= 63.493
  09:46:16 STEP 1102 H= 275:30 +CPU= 63.973
  09:47:26 STEP 1103 H= 275:45 +CPU= 65.897
  09:48:58 STEP 1104 H= 276:00 +CPU= 87.401
Moving to projects directory: /var/lib/boinc/slots/14/ICMGGh7zg+001104
Moving to projects directory: /var/lib/boinc/slots/14/ICMSHh7zg+001104
Moving to projects directory: /var/lib/boinc/slots/14/ICMUAh7zg+001104
  09:50:05 STEP 1105 H= 276:15 +CPU= 63.289
  09:51:15 STEP 1106 H= 276:30 +CPU= 66.004
  09:52:21 STEP 1107 H= 276:45 +CPU= 63.847
  09:53:33 STEP 1108 H= 277:00 +CPU= 68.134
  09:54:42 STEP 1109 H= 277:15 +CPU= 65.221
  09:55:48 STEP 1110 H= 277:30 +CPU= 63.326
  09:56:53 STEP 1111 H= 277:45 +CPU= 63.523
  09:57:57 STEP 1112 H= 278:00 +CPU= 62.029
  09:59:04 STEP 1113 H= 278:15 +CPU= 63.839
  10:00:12 STEP 1114 H= 278:30 +CPU= 65.196
  10:01:17 STEP 1115 H= 278:45 +CPU= 62.991
  10:02:34 STEP 1116 H= 279:00 +CPU= 74.210
Moving to projects directory: /var/lib/boinc/slots/14/ICMGGh7zg+001116
Moving to projects directory: /var/lib/boinc/slots/14/ICMSHh7zg+001116
Moving to projects directory: /var/lib/boinc/slots/14/ICMUAh7zg+001116
  10:03:42 STEP 1117 H= 279:15 +CPU= 64.868
  10:04:51 STEP 1118 H= 279:30 +CPU= 65.016
  10:05:57 STEP 1119 H= 279:45 +CPU= 63.545
  10:07:03 STEP 1120 H= 280:00 +CPU= 64.214
  10:08:07 STEP 1121 H= 280:15 +CPU= 60.856
  10:09:12 STEP 1122 H= 280:30 +CPU= 62.782
  10:10:16 STEP 1123 H= 280:45 +CPU= 62.847
  10:11:23 STEP 1124 H= 281:00 +CPU= 64.019
  10:12:29 STEP 1125 H= 281:15 +CPU= 63.094
  10:13:32 STEP 1126 H= 281:30 +CPU= 61.633
  10:14:37 STEP 1127 H= 281:45 +CPU= 62.980
  10:15:55 STEP 1128 H= 282:00 +CPU= 74.473
Moving to projects directory: /var/lib/boinc/slots/14/ICMGGh7zg+001128
Moving to projects directory: /var/lib/boinc/slots/14/ICMSHh7zg+001128
Moving to projects directory: /var/lib/boinc/slots/14/ICMUAh7zg+001128
  10:16:54 STEP 1129 H= 282:15 +CPU= 57.646
  10:17:58 STEP 1130 H= 282:30 +CPU= 61.263
  10:19:06 STEP 1131 H= 282:45 +CPU= 66.079
  10:20:14 STEP 1132 H= 283:00 +CPU= 64.440
  10:21:22 STEP 1133 H= 283:15 +CPU= 64.029
  10:22:25 STEP 1134 H= 283:30 +CPU= 60.331
  10:23:29 STEP 1135 H= 283:45 +CPU= 60.818
  10:24:36 STEP 1136 H= 284:00 +CPU= 63.708
  10:25:46 STEP 1137 H= 284:15 +CPU= 66.314
  10:26:50 STEP 1138 H= 284:30 +CPU= 61.871
  10:27:55 STEP 1139 H= 284:45 +CPU= 63.319
  10:29:12 STEP 1140 H= 285:00 +CPU= 75.644
Moving to projects directory: /var/lib/boinc/slots/14/ICMGGh7zg+001140
Moving to projects directory: /var/lib/boinc/slots/14/ICMSHh7zg+001140
Moving to projects directory: /var/lib/boinc/slots/14/ICMUAh7zg+001140
  10:30:17 STEP 1141 H= 285:15 +CPU= 63.031
  10:31:24 STEP 1142 H= 285:30 +CPU= 63.757
  10:32:31 STEP 1143 H= 285:45 +CPU= 64.020
  10:33:40 STEP 1144 H= 286:00 +CPU= 65.545
  10:34:47 STEP 1145 H= 286:15 +CPU= 64.264
  10:35:59 STEP 1146 H= 286:30 +CPU= 66.762
  10:37:05 STEP 1147 H= 286:45 +CPU= 63.489
  10:38:12 STEP 1148 H= 287:00 +CPU= 64.346
  10:39:17 STEP 1149 H= 287:15 +CPU= 61.914
  10:40:22 STEP 1150 H= 287:30 +CPU= 61.629
  10:41:22 STEP 1151 H= 287:45 +CPU= 59.466
  10:42:51 STEP 1152 H= 288:00 +CPU= 83.065
Moving to projects directory: /var/lib/boinc/slots/14/ICMGGh7zg+001152
Moving to projects directory: /var/lib/boinc/slots/14/ICMSHh7zg+001152
Moving to projects directory: /var/lib/boinc/slots/14/ICMUAh7zg+001152
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMGGh7zg+001056
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMSHh7zg+001056
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMUAh7zg+001056
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMGGh7zg+001068
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMSHh7zg+001068
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMUAh7zg+001068
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMGGh7zg+001080
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMSHh7zg+001080
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMUAh7zg+001080
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMGGh7zg+001092
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMSHh7zg+001092
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMUAh7zg+001092
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMGGh7zg+001104
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMSHh7zg+001104
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMUAh7zg+001104
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMGGh7zg+001116
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMSHh7zg+001116
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMUAh7zg+001116
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMGGh7zg+001128
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMSHh7zg+001128
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMUAh7zg+001128
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMGGh7zg+001140
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMSHh7zg+001140
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMUAh7zg+001140
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_a2zu_2016092300_20_1017_12285709_0_r295598715_11.zip
Uploading the intermediate file: upload_file_11.zip
  10:43:56 STEP 1153 H= 288:15 +CPU= 61.243
  10:45:03 STEP 1154 H= 288:30 +CPU= 64.041
  10:46:06 STEP 1155 H= 288:45 +CPU= 60.873
  10:47:18 STEP 1156 H= 289:00 +CPU= 68.411
  10:48:26 STEP 1157 H= 289:15 +CPU= 64.018
  10:49:37 STEP 1158 H= 289:30 +CPU= 63.622
  10:50:45 STEP 1159 H= 289:45 +CPU= 61.628
  10:51:55 STEP 1160 H= 290:00 +CPU= 64.643
  10:52:58 STEP 1161 H= 290:15 +CPU= 60.900
  10:54:07 STEP 1162 H= 290:30 +CPU= 65.741
  10:55:12 STEP 1163 H= 290:45 +CPU= 61.797
  10:56:33 STEP 1164 H= 291:00 +CPU= 77.419
Moving to projects directory: /var/lib/boinc/slots/14/ICMGGh7zg+001164
Moving to projects directory: /var/lib/boinc/slots/14/ICMSHh7zg+001164
Moving to projects directory: /var/lib/boinc/slots/14/ICMUAh7zg+001164
  10:57:42 STEP 1165 H= 291:15 +CPU= 65.178
  10:58:48 STEP 1166 H= 291:30 +CPU= 63.289
  10:59:53 STEP 1167 H= 291:45 +CPU= 63.037
  11:00:59 STEP 1168 H= 292:00 +CPU= 64.021
  11:02:07 STEP 1169 H= 292:15 +CPU= 64.550
  11:03:15 STEP 1170 H= 292:30 +CPU= 65.912
  11:04:21 STEP 1171 H= 292:45 +CPU= 63.407
  11:05:31 STEP 1172 H= 293:00 +CPU= 65.895
  11:06:35 STEP 1173 H= 293:15 +CPU= 61.311
  11:07:41 STEP 1174 H= 293:30 +CPU= 62.062
  11:08:50 STEP 1175 H= 293:45 +CPU= 65.329
[EC_DRHOOK:antares:1:1:28720:28720] [20240608:110953:1717837793:49598.214] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 3958MB (heap), 5314MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:antares:1:1:28720:28720] [20240608:110953:1717837793:49598.216] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:antares:1:1:28720:28720] [20240608:110953:1717837793:49598.216] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x820890 (old at (nil))
[EC_DRHOOK:antares:1:1:28720:28720] [20240608:110953:1717837793:49598.216] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1
[EC_DRHOOK:antares:1:1:28720:28720] [20240608:110953:1717837793:49598.216] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1
[EC_DRHOOK:antares:1:1:28720:28720] [20240608:110953:1717837793:49598.216] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3843] 3958 MB (maxheap), 5314 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:antares:1:1:28720:28720] [20240608:110953:1717837793:49598.216] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:antares:1:1:28720:28720] [20240608:110953:1717837793:49598.216] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:antares:1:1:28720:28720] [20240608:110953:1717837793:49598.216] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:antares:1:1:28720:28720] [20240608:110953:1717837793:49598.216] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:antares:1:1:28720:28720] [20240608:110953:1717837793:49598.216] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:antares:1:1:28720:28720] [20240608:110953:1717837793:49598.216] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :       CNT4 
[EC_DRHOOK:antares:1:1:28720:28720] [20240608:110953:1717837793:49598.216] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :        STEPO 
[EC_DRHOOK:antares:1:1:28720:28720] [20240608:110953:1717837793:49598.216] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :         SCAN2M 
[EC_DRHOOK:antares:1:1:28720:28720] [20240608:110953:1717837793:49598.216] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :          GP_MODEL_HEAP 
[EC_DRHOOK:antares:1:1:28720:28720] [20240608:110953:1717837793:49598.216] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :           GP_MODEL 
[EC_DRHOOK:antares:1:1:28720:28720] [20240608:110953:1717837793:49598.216] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :            EC_PHYS_DRV 
[EC_DRHOOK:antares:1:1:28720:28720] [20240608:110953:1717837793:49598.216] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :             >OMP-PHYSICS CLDPP T/S    (1002) 
[EC_DRHOOK:antares:1:1:28720:28720] [20240608:110953:1717837793:49598.216] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :              EC_PHYS 
[EC_DRHOOK:antares:1:1:28720:28720] [20240608:110953:1717837793:49598.216] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :               CALLPAR 
[EC_DRHOOK:antares:1:1:28720:28720] [20240608:110953:1717837793:49598.216] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :                STATE_UPDATE 
[EC_DRHOOK:antares:1:1:28720:28720] [20240608:110954:1717837794:49599.217] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1
[EC_DRHOOK:antares:1:1:28720:28720] [20240608:110954:1717837794:49599.217] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1cf8da0 for signal#15, nsigs = 1
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source             
oifs_43r3_model.e  0000000001CF935B  Unknown               Unknown  Unknown
oifs_43r3_model.e  00000000008214C1  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DD5490  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001149226  state_update_              36  state_update.F90
oifs_43r3_model.e  0000000000FB27BD  callpar_                 1045  callpar.F90
oifs_43r3_model.e  0000000000F5FA6B  ec_phys_                  670  ec_phys.F90
oifs_43r3_model.e  0000000000F4DBC4  ec_phys_drv_              599  ec_phys_drv.F90
oifs_43r3_model.e  0000000000BD7131  gp_model_                 613  gp_model.F90
oifs_43r3_model.e  00000000012C5002  gp_model_heap_             74  gp_model_heap.F90
oifs_43r3_model.e  0000000000BACBAE  scan2m_                   535  scan2m.F90
oifs_43r3_model.e  0000000000688153  stepo_                    327  stepo.F90
oifs_43r3_model.e  00000000004CD1DC  cnt4_                    1133  cnt4.F90
oifs_43r3_model.e  00000000004B5D85  cnt3_                     267  cnt3.F90
oifs_43r3_model.e  00000000004B4F52  cnt2_                      88  cnt2.F90
oifs_43r3_model.e  000000000049D638  cnt1_                      92  cnt1.F90
oifs_43r3_model.e  0000000000406FBD  cnt0_                     146  cnt0.F90
oifs_43r3_model.e  0000000000401E7F  MAIN__                     96  master.F90
oifs_43r3_model.e  0000000000401E12  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCC34A  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCDBE7  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000000401CD5  Unknown               Unknown  Unknown
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.13_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a2zu
(argv4) batchid: 1017
(argv5) wuid: 12285709
(argv6) fclen: 20
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/14
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.13
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/14/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/14/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_4b98fd961e5fa891d8aee96fd98e8c56 to: /var/lib/boinc/slots/14/oifs_43r3_bl_a2zu_2016092300_20_1017_12285709.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/14/oifs_43r3_bl_a2zu_2016092300_20_1017_12285709.zip
ic_ancil_file: ic_ancil_12285709
ifsdata_file: ifsdata_12285709
climate_data_file: clim_data_12285709
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /var/lib/boinc/slots/14/ic_ancil_12285709.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/14/ic_ancil_12285709.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_ea0abab7ea789decfd517e267302909f to: /var/lib/boinc/slots/14/ifsdata/ifsdata_12285709.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/14/ifsdata/ifsdata_12285709.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/14/159l_2/clim_data_12285709.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/14/159l_2/clim_data_12285709.zip
Checking for progress XML file: /var/lib/boinc/slots/14/progress_file_12285709.xml
Opened progress file ok : /var/lib/boinc/slots/14/progress_file_12285709.xml
-- Model is restarting --
Adjusting last_iter, 1176, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/14/progress_file_12285709.xml
last_cpu_time: 56465
upload_file_number: 12
last_iter: 1153
last_upload: 1036800
model_completed: 0
total_length_of_simulation: 1728000
result_base_name: oifs_43r3_bl_a2zu_2016092300_20_1017_12285709_0_r295598715
The child process has been launched with process id: 8044
Executing the command: /var/lib/boinc/slots/14/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:antares:1:1:8044:8044] [20240610:162012:1718029212:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2091] fp = 0x314f440
[EC_DRHOOK:antares:1:1:8044:8044] [20240610:162012:1718029212:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:antares:1:1:8044:8044] [20240610:162012:1718029212:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:antares:1:1:8044:8044] [20240610:162012:1718029212:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:antares:1:1:8044:8044] [20240610:162012:1718029212:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:antares:1:1:8044:8044] [20240610:162012:1718029212:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:antares:1:1:8044:8044] [20240610:162012:1718029212:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:antares:1:1:8044:8044] [20240610:162012:1718029212:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:antares:1:1:8044:8044] [20240610:162012:1718029212:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:antares:1:1:8044:8044] [20240610:162012:1718029212:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:antares:1:1:8044:8044] [20240610:162012:1718029212:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:antares:1:1:8044:8044] [20240610:162012:1718029212:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:antares:1:1:8044:8044] [20240610:162012:1718029212:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:antares:1:1:8044:8044] [20240610:162012:1718029212:0.000] [ignore_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:antares:1:1:8044:8044] [20240610:162012:1718029212:0.000] [restore_default_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:antares:1:1:8044:8044] [20240610:162012:1718029212:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:antares:1:1:8044:8044] [20240610:162012:1718029212:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:antares:1:1:8044:8044] [20240610:162012:1718029212:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:antares:1:1:8044:8044] [20240610:162012:1718029212:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x820cf0 (old at (nil))
[EC_DRHOOK:antares:1:1:8044:8044] [20240610:162012:1718029212:0.000] [trapfpe_treatment@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:antares:1:1:8044:8044] [20240610:162012:1718029212:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:antares:1:1:8044:8044] [20240610:162012:1718029212:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:antares:1:1:8044:8044] [20240610:162012:1718029212:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x820cf0 (old at (nil))
[EC_DRHOOK:antares:1:1:8044:8044] [20240610:162012:1718029212:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:antares:1:1:8044:8044] [20240610:162012:1718029212:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:antares:1:1:8044:8044] [20240610:162012:1718029212:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:antares:1:1:8044:8044] [20240610:162012:1718029212:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x820cf0 (old at (nil))
[EC_DRHOOK:antares:1:1:8044:8044] [20240610:162012:1718029212:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x820cf0 (old at (nil))
[EC_DRHOOK:antares:1:1:8044:8044] [20240610:162012:1718029212:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:antares:1:1:8044:8044] [20240610:162012:1718029212:0.000] [catch_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  16:20:57 STEP 1152 H= 288:00 +CPU= 37.235
  16:21:30 STEP 1153 H= 288:15 +CPU= 32.391
  16:22:02 STEP 1154 H= 288:30 +CPU= 32.162
  16:22:36 STEP 1155 H= 288:45 +CPU= 32.943
  16:23:24 STEP 1156 H= 289:00 +CPU= 44.566
  16:24:23 STEP 1157 H= 289:15 +CPU= 53.792
  16:25:22 STEP 1158 H= 289:30 +CPU= 53.958
  16:26:23 STEP 1159 H= 289:45 +CPU= 55.464
  16:27:28 STEP 1160 H= 290:00 +CPU= 55.766
  16:28:29 STEP 1161 H= 290:15 +CPU= 53.933
  16:29:31 STEP 1162 H= 290:30 +CPU= 54.767
  16:30:33 STEP 1163 H= 290:45 +CPU= 54.347
  16:31:50 STEP 1164 H= 291:00 +CPU= 69.280
Moving to projects directory: /var/lib/boinc/slots/14/ICMGGh7zg+001164
Moving to projects directory: /var/lib/boinc/slots/14/ICMSHh7zg+001164
Moving to projects directory: /var/lib/boinc/slots/14/ICMUAh7zg+001164
  16:32:51 STEP 1165 H= 291:15 +CPU= 54.620
  16:33:53 STEP 1166 H= 291:30 +CPU= 55.021
  16:34:53 STEP 1167 H= 291:45 +CPU= 53.235
  16:35:58 STEP 1168 H= 292:00 +CPU= 55.848
  16:37:02 STEP 1169 H= 292:15 +CPU= 56.570
  16:38:05 STEP 1170 H= 292:30 +CPU= 54.254
  16:39:06 STEP 1171 H= 292:45 +CPU= 54.923
  16:40:10 STEP 1172 H= 293:00 +CPU= 58.302
  16:41:16 STEP 1173 H= 293:15 +CPU= 58.965
  16:42:21 STEP 1174 H= 293:30 +CPU= 54.672
  16:43:21 STEP 1175 H= 293:45 +CPU= 53.224
  16:44:34 STEP 1176 H= 294:00 +CPU= 67.010
Moving to projects directory: /var/lib/boinc/slots/14/ICMGGh7zg+001176
Moving to projects directory: /var/lib/boinc/slots/14/ICMSHh7zg+001176
Moving to projects directory: /var/lib/boinc/slots/14/ICMUAh7zg+001176
  16:45:35 STEP 1177 H= 294:15 +CPU= 55.229
  16:46:37 STEP 1178 H= 294:30 +CPU= 55.031
  16:47:37 STEP 1179 H= 294:45 +CPU= 54.434
  16:48:39 STEP 1180 H= 295:00 +CPU= 54.917
  16:49:40 STEP 1181 H= 295:15 +CPU= 54.933
  16:50:40 STEP 1182 H= 295:30 +CPU= 54.191
  16:51:40 STEP 1183 H= 295:45 +CPU= 54.288
  16:52:44 STEP 1184 H= 296:00 +CPU= 56.964
  16:53:48 STEP 1185 H= 296:15 +CPU= 57.455
  16:54:49 STEP 1186 H= 296:30 +CPU= 55.091
  16:55:53 STEP 1187 H= 296:45 +CPU= 57.360
  16:57:06 STEP 1188 H= 297:00 +CPU= 66.737
Moving to projects directory: /var/lib/boinc/slots/14/ICMGGh7zg+001188
Moving to projects directory: /var/lib/boinc/slots/14/ICMSHh7zg+001188
Moving to projects directory: /var/lib/boinc/slots/14/ICMUAh7zg+001188
  16:58:10 STEP 1189 H= 297:15 +CPU= 57.756
  16:59:10 STEP 1190 H= 297:30 +CPU= 54.412
  17:00:14 STEP 1191 H= 297:45 +CPU= 57.256
  17:01:16 STEP 1192 H= 298:00 +CPU= 56.630
  17:02:18 STEP 1193 H= 298:15 +CPU= 56.031
  17:03:20 STEP 1194 H= 298:30 +CPU= 55.955
  17:04:21 STEP 1195 H= 298:45 +CPU= 57.191
  17:05:23 STEP 1196 H= 299:00 +CPU= 60.285
  17:06:23 STEP 1197 H= 299:15 +CPU= 58.251
  17:07:22 STEP 1198 H= 299:30 +CPU= 54.636
  17:08:17 STEP 1199 H= 299:45 +CPU= 53.866
  17:09:35 STEP 1200 H= 300:00 +CPU= 74.096
Moving to projects directory: /var/lib/boinc/slots/14/ICMGGh7zg+001200
Moving to projects directory: /var/lib/boinc/slots/14/ICMSHh7zg+001200
Moving to projects directory: /var/lib/boinc/slots/14/ICMUAh7zg+001200
  17:10:35 STEP 1201 H= 300:15 +CPU= 57.047
  17:11:34 STEP 1202 H= 300:30 +CPU= 56.284
  17:12:30 STEP 1203 H= 300:45 +CPU= 52.594
  17:13:29 STEP 1204 H= 301:00 +CPU= 57.241
  17:14:25 STEP 1205 H= 301:15 +CPU= 53.781
  17:15:28 STEP 1206 H= 301:30 +CPU= 60.149
  17:16:28 STEP 1207 H= 301:45 +CPU= 58.354
  17:17:27 STEP 1208 H= 302:00 +CPU= 57.789
  17:18:26 STEP 1209 H= 302:15 +CPU= 56.706
  17:19:30 STEP 1210 H= 302:30 +CPU= 60.642
  17:20:31 STEP 1211 H= 302:45 +CPU= 58.816
  17:21:40 STEP 1212 H= 303:00 +CPU= 65.803
Moving to projects directory: /var/lib/boinc/slots/14/ICMGGh7zg+001212
Moving to projects directory: /var/lib/boinc/slots/14/ICMSHh7zg+001212
Moving to projects directory: /var/lib/boinc/slots/14/ICMUAh7zg+001212
  17:22:44 STEP 1213 H= 303:15 +CPU= 61.268
  17:23:45 STEP 1214 H= 303:30 +CPU= 58.893
  17:24:46 STEP 1215 H= 303:45 +CPU= 59.311
  17:25:52 STEP 1216 H= 304:00 +CPU= 63.304
  17:26:55 STEP 1217 H= 304:15 +CPU= 59.373
  17:27:53 STEP 1218 H= 304:30 +CPU= 56.397
  17:29:00 STEP 1219 H= 304:45 +CPU= 64.155
  17:30:03 STEP 1220 H= 305:00 +CPU= 60.171
  17:31:05 STEP 1221 H= 305:15 +CPU= 59.741
  17:32:14 STEP 1222 H= 305:30 +CPU= 66.058
  17:33:19 STEP 1223 H= 305:45 +CPU= 61.660
  17:34:34 STEP 1224 H= 306:00 +CPU= 71.914
Moving to projects directory: /var/lib/boinc/slots/14/ICMGGh7zg+001224
Moving to projects directory: /var/lib/boinc/slots/14/ICMSHh7zg+001224
Moving to projects directory: /var/lib/boinc/slots/14/ICMUAh7zg+001224
  17:35:33 STEP 1225 H= 306:15 +CPU= 57.946
  17:36:39 STEP 1226 H= 306:30 +CPU= 62.970
  17:37:47 STEP 1227 H= 306:45 +CPU= 64.490
  17:38:50 STEP 1228 H= 307:00 +CPU= 61.529
  17:39:52 STEP 1229 H= 307:15 +CPU= 59.572
  17:40:55 STEP 1230 H= 307:30 +CPU= 60.632
  17:41:59 STEP 1231 H= 307:45 +CPU= 60.764
  17:43:04 STEP 1232 H= 308:00 +CPU= 62.624
  17:43:58 STEP 1233 H= 308:15 +CPU= 52.615
  17:45:00 STEP 1234 H= 308:30 +CPU= 60.309
  17:46:09 STEP 1235 H= 308:45 +CPU= 65.316
  17:47:33 STEP 1236 H= 309:00 +CPU= 80.201
Moving to projects directory: /var/lib/boinc/slots/14/ICMGGh7zg+001236
Moving to projects directory: /var/lib/boinc/slots/14/ICMSHh7zg+001236
Moving to projects directory: /var/lib/boinc/slots/14/ICMUAh7zg+001236
  17:48:34 STEP 1237 H= 309:15 +CPU= 58.643
  17:49:36 STEP 1238 H= 309:30 +CPU= 60.074
  17:50:36 STEP 1239 H= 309:45 +CPU= 57.583
  17:51:37 STEP 1240 H= 310:00 +CPU= 59.444
  17:52:35 STEP 1241 H= 310:15 +CPU= 56.244
  17:53:41 STEP 1242 H= 310:30 +CPU= 63.035
  17:54:42 STEP 1243 H= 310:45 +CPU= 58.201
  17:55:41 STEP 1244 H= 311:00 +CPU= 56.008
  17:56:40 STEP 1245 H= 311:15 +CPU= 56.384
  17:57:45 STEP 1246 H= 311:30 +CPU= 61.871
  17:58:47 STEP 1247 H= 311:45 +CPU= 59.842
  18:00:06 STEP 1248 H= 312:00 +CPU= 75.422
Moving to projects directory: /var/lib/boinc/slots/14/ICMGGh7zg+001248
Moving to projects directory: /var/lib/boinc/slots/14/ICMSHh7zg+001248
Moving to projects directory: /var/lib/boinc/slots/14/ICMUAh7zg+001248
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMGGh7zg+001152
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMSHh7zg+001152
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMUAh7zg+001152
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMGGh7zg+001164
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMSHh7zg+001164
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMUAh7zg+001164
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMGGh7zg+001176
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMSHh7zg+001176
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMUAh7zg+001176
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMGGh7zg+001188
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMSHh7zg+001188
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMUAh7zg+001188
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMGGh7zg+001200
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMSHh7zg+001200
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMUAh7zg+001200
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMGGh7zg+001212
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMSHh7zg+001212
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMUAh7zg+001212
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMGGh7zg+001224
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMSHh7zg+001224
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMUAh7zg+001224
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMGGh7zg+001236
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMSHh7zg+001236
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMUAh7zg+001236
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_a2zu_2016092300_20_1017_12285709_0_r295598715_12.zip
Uploading the intermediate file: upload_file_12.zip
  18:01:09 STEP 1249 H= 312:15 +CPU= 60.536
  18:02:14 STEP 1250 H= 312:30 +CPU= 61.598
  18:03:21 STEP 1251 H= 312:45 +CPU= 63.521
  18:04:22 STEP 1252 H= 313:00 +CPU= 58.539
  18:05:22 STEP 1253 H= 313:15 +CPU= 59.032
  18:06:26 STEP 1254 H= 313:30 +CPU= 61.872
  18:07:31 STEP 1255 H= 313:45 +CPU= 61.778
  18:08:33 STEP 1256 H= 314:00 +CPU= 59.883
  18:09:39 STEP 1257 H= 314:15 +CPU= 62.730
  18:10:39 STEP 1258 H= 314:30 +CPU= 57.844
  18:11:39 STEP 1259 H= 314:45 +CPU= 58.502
  18:12:57 STEP 1260 H= 315:00 +CPU= 75.110
Moving to projects directory: /var/lib/boinc/slots/14/ICMGGh7zg+001260
Moving to projects directory: /var/lib/boinc/slots/14/ICMSHh7zg+001260
Moving to projects directory: /var/lib/boinc/slots/14/ICMUAh7zg+001260
  18:13:55 STEP 1261 H= 315:15 +CPU= 56.493
  18:14:56 STEP 1262 H= 315:30 +CPU= 58.545
  18:16:01 STEP 1263 H= 315:45 +CPU= 62.324
  18:17:04 STEP 1264 H= 316:00 +CPU= 60.606
  18:18:07 STEP 1265 H= 316:15 +CPU= 60.609
  18:19:13 STEP 1266 H= 316:30 +CPU= 63.163
  18:20:18 STEP 1267 H= 316:45 +CPU= 61.925
  18:21:27 STEP 1268 H= 317:00 +CPU= 63.276
  18:22:31 STEP 1269 H= 317:15 +CPU= 58.967
  18:23:36 STEP 1270 H= 317:30 +CPU= 64.049
  18:24:43 STEP 1271 H= 317:45 +CPU= 63.780
  18:26:03 STEP 1272 H= 318:00 +CPU= 75.177
Moving to projects directory: /var/lib/boinc/slots/14/ICMGGh7zg+001272
Moving to projects directory: /var/lib/boinc/slots/14/ICMSHh7zg+001272
Moving to projects directory: /var/lib/boinc/slots/14/ICMUAh7zg+001272
  18:27:03 STEP 1273 H= 318:15 +CPU= 58.667
  18:28:07 STEP 1274 H= 318:30 +CPU= 62.176
  18:29:08 STEP 1275 H= 318:45 +CPU= 58.661
  18:30:13 STEP 1276 H= 319:00 +CPU= 61.490
  18:31:17 STEP 1277 H= 319:15 +CPU= 60.822
  18:32:15 STEP 1278 H= 319:30 +CPU= 56.236
  18:33:16 STEP 1279 H= 319:45 +CPU= 59.400
  18:34:18 STEP 1280 H= 320:00 +CPU= 59.491
  18:35:24 STEP 1281 H= 320:15 +CPU= 63.672
  18:36:17 STEP 1282 H= 320:30 +CPU= 51.840
  18:37:17 STEP 1283 H= 320:45 +CPU= 58.342
  18:38:30 STEP 1284 H= 321:00 +CPU= 70.029
Moving to projects directory: /var/lib/boinc/slots/14/ICMGGh7zg+001284
Moving to projects directory: /var/lib/boinc/slots/14/ICMSHh7zg+001284
Moving to projects directory: /var/lib/boinc/slots/14/ICMUAh7zg+001284
  18:39:41 STEP 1285 H= 321:15 +CPU= 66.648
  18:40:47 STEP 1286 H= 321:30 +CPU= 63.798
  18:41:52 STEP 1287 H= 321:45 +CPU= 62.722
  18:42:56 STEP 1288 H= 322:00 +CPU= 61.182
  18:44:00 STEP 1289 H= 322:15 +CPU= 59.665
  18:44:56 STEP 1290 H= 322:30 +CPU= 54.121
  18:46:03 STEP 1291 H= 322:45 +CPU= 63.778
  18:47:06 STEP 1292 H= 323:00 +CPU= 60.399
  18:48:10 STEP 1293 H= 323:15 +CPU= 60.815
  18:49:10 STEP 1294 H= 323:30 +CPU= 57.400
  18:50:14 STEP 1295 H= 323:45 +CPU= 60.406
  18:51:39 STEP 1296 H= 324:00 +CPU= 79.839
Moving to projects directory: /var/lib/boinc/slots/14/ICMGGh7zg+001296
Moving to projects directory: /var/lib/boinc/slots/14/ICMSHh7zg+001296
Moving to projects directory: /var/lib/boinc/slots/14/ICMUAh7zg+001296
  18:52:43 STEP 1297 H= 324:15 +CPU= 61.390
  18:53:46 STEP 1298 H= 324:30 +CPU= 60.529
  18:54:52 STEP 1299 H= 324:45 +CPU= 62.108
  18:55:58 STEP 1300 H= 325:00 +CPU= 63.036
  18:57:05 STEP 1301 H= 325:15 +CPU= 59.810
  18:58:11 STEP 1302 H= 325:30 +CPU= 58.429
  18:59:17 STEP 1303 H= 325:45 +CPU= 61.071
  19:00:23 STEP 1304 H= 326:00 +CPU= 62.593
  19:01:28 STEP 1305 H= 326:15 +CPU= 60.948
  19:02:30 STEP 1306 H= 326:30 +CPU= 58.314
  19:03:35 STEP 1307 H= 326:45 +CPU= 61.612
  19:04:52 STEP 1308 H= 327:00 +CPU= 72.478
Moving to projects directory: /var/lib/boinc/slots/14/ICMGGh7zg+001308
Moving to projects directory: /var/lib/boinc/slots/14/ICMSHh7zg+001308
Moving to projects directory: /var/lib/boinc/slots/14/ICMUAh7zg+001308
  19:05:54 STEP 1309 H= 327:15 +CPU= 60.218
  19:06:55 STEP 1310 H= 327:30 +CPU= 59.032
  19:07:56 STEP 1311 H= 327:45 +CPU= 59.254
  19:08:55 STEP 1312 H= 328:00 +CPU= 57.298
  19:09:58 STEP 1313 H= 328:15 +CPU= 60.143
  19:11:01 STEP 1314 H= 328:30 +CPU= 61.039
  19:11:55 STEP 1315 H= 328:45 +CPU= 52.817
  19:13:00 STEP 1316 H= 329:00 +CPU= 62.195
  19:13:59 STEP 1317 H= 329:15 +CPU= 57.294
  19:14:58 STEP 1318 H= 329:30 +CPU= 57.243
  19:15:58 STEP 1319 H= 329:45 +CPU= 58.346
  19:17:11 STEP 1320 H= 330:00 +CPU= 70.559
Moving to projects directory: /var/lib/boinc/slots/14/ICMGGh7zg+001320
Moving to projects directory: /var/lib/boinc/slots/14/ICMSHh7zg+001320
Moving to projects directory: /var/lib/boinc/slots/14/ICMUAh7zg+001320
  19:18:09 STEP 1321 H= 330:15 +CPU= 55.779
  19:19:11 STEP 1322 H= 330:30 +CPU= 59.518
  19:20:15 STEP 1323 H= 330:45 +CPU= 62.466
  19:21:20 STEP 1324 H= 331:00 +CPU= 59.851
  19:22:33 STEP 1325 H= 331:15 +CPU= 58.684
  19:23:46 STEP 1326 H= 331:30 +CPU= 58.178
  19:24:52 STEP 1327 H= 331:45 +CPU= 60.580
  19:25:53 STEP 1328 H= 332:00 +CPU= 58.238
  19:26:53 STEP 1329 H= 332:15 +CPU= 58.635
  19:27:55 STEP 1330 H= 332:30 +CPU= 59.635
  19:28:59 STEP 1331 H= 332:45 +CPU= 61.105
  19:30:14 STEP 1332 H= 333:00 +CPU= 72.297
Moving to projects directory: /var/lib/boinc/slots/14/ICMGGh7zg+001332
Moving to projects directory: /var/lib/boinc/slots/14/ICMSHh7zg+001332
Moving to projects directory: /var/lib/boinc/slots/14/ICMUAh7zg+001332
  19:31:21 STEP 1333 H= 333:15 +CPU= 63.939
  19:32:23 STEP 1334 H= 333:30 +CPU= 59.647
  19:33:24 STEP 1335 H= 333:45 +CPU= 58.020
  19:34:22 STEP 1336 H= 334:00 +CPU= 55.133
  19:35:26 STEP 1337 H= 334:15 +CPU= 62.732
  19:36:23 STEP 1338 H= 334:30 +CPU= 54.893
  19:37:25 STEP 1339 H= 334:45 +CPU= 61.630
  19:38:21 STEP 1340 H= 335:00 +CPU= 55.363
  19:39:16 STEP 1341 H= 335:15 +CPU= 53.878
  19:40:06 STEP 1342 H= 335:30 +CPU= 48.975
  19:40:52 STEP 1343 H= 335:45 +CPU= 45.656
  19:42:08 STEP 1344 H= 336:00 +CPU= 74.200
Moving to projects directory: /var/lib/boinc/slots/14/ICMGGh7zg+001344
Moving to projects directory: /var/lib/boinc/slots/14/ICMSHh7zg+001344
Moving to projects directory: /var/lib/boinc/slots/14/ICMUAh7zg+001344
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMGGh7zg+001248
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMSHh7zg+001248
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMUAh7zg+001248
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMGGh7zg+001260
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMSHh7zg+001260
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMUAh7zg+001260
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMGGh7zg+001272
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMSHh7zg+001272
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMUAh7zg+001272
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMGGh7zg+001284
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMSHh7zg+001284
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMUAh7zg+001284
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMGGh7zg+001296
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMSHh7zg+001296
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMUAh7zg+001296
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMGGh7zg+001308
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMSHh7zg+001308
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMUAh7zg+001308
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMGGh7zg+001320
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMSHh7zg+001320
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMUAh7zg+001320
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMGGh7zg+001332
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMSHh7zg+001332
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMUAh7zg+001332
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_a2zu_2016092300_20_1017_12285709_0_r295598715_13.zip
Uploading the intermediate file: upload_file_13.zip
  19:43:07 STEP 1345 H= 336:15 +CPU= 55.273
  19:43:59 STEP 1346 H= 336:30 +CPU= 50.358
  19:44:55 STEP 1347 H= 336:45 +CPU= 53.203
  19:45:52 STEP 1348 H= 337:00 +CPU= 55.141
  19:46:46 STEP 1349 H= 337:15 +CPU= 51.794
  19:47:35 STEP 1350 H= 337:30 +CPU= 47.598
  19:48:33 STEP 1351 H= 337:45 +CPU= 56.132
  19:49:24 STEP 1352 H= 338:00 +CPU= 49.174
  19:50:17 STEP 1353 H= 338:15 +CPU= 50.364
  19:51:10 STEP 1354 H= 338:30 +CPU= 51.520
  19:52:05 STEP 1355 H= 338:45 +CPU= 53.585
  19:53:04 STEP 1356 H= 339:00 +CPU= 57.399
Moving to projects directory: /var/lib/boinc/slots/14/ICMGGh7zg+001356
Moving to projects directory: /var/lib/boinc/slots/14/ICMSHh7zg+001356
Moving to projects directory: /var/lib/boinc/slots/14/ICMUAh7zg+001356
  19:53:53 STEP 1357 H= 339:15 +CPU= 48.206
  19:54:40 STEP 1358 H= 339:30 +CPU= 46.533
  19:55:30 STEP 1359 H= 339:45 +CPU= 48.385
  19:56:22 STEP 1360 H= 340:00 +CPU= 50.068
  19:57:14 STEP 1361 H= 340:15 +CPU= 50.362
  19:58:11 STEP 1362 H= 340:30 +CPU= 54.951
  19:59:06 STEP 1363 H= 340:45 +CPU= 52.895
  20:00:02 STEP 1364 H= 341:00 +CPU= 53.601
  20:00:59 STEP 1365 H= 341:15 +CPU= 54.058
  20:01:52 STEP 1366 H= 341:30 +CPU= 51.880
  20:02:54 STEP 1367 H= 341:45 +CPU= 59.089
  20:04:01 STEP 1368 H= 342:00 +CPU= 63.407
Moving to projects directory: /var/lib/boinc/slots/14/ICMGGh7zg+001368
Moving to projects directory: /var/lib/boinc/slots/14/ICMSHh7zg+001368
Moving to projects directory: /var/lib/boinc/slots/14/ICMUAh7zg+001368
  20:05:01 STEP 1369 H= 342:15 +CPU= 57.900
  20:05:54 STEP 1370 H= 342:30 +CPU= 51.561
  20:06:41 STEP 1371 H= 342:45 +CPU= 45.143
  20:07:32 STEP 1372 H= 343:00 +CPU= 49.988
  20:08:30 STEP 1373 H= 343:15 +CPU= 54.886
  20:09:23 STEP 1374 H= 343:30 +CPU= 50.735
  20:10:08 STEP 1375 H= 343:45 +CPU= 44.155
  20:11:00 STEP 1376 H= 344:00 +CPU= 50.556
  20:11:52 STEP 1377 H= 344:15 +CPU= 50.240
  20:12:49 STEP 1378 H= 344:30 +CPU= 54.936
  20:13:44 STEP 1379 H= 344:45 +CPU= 51.290
  20:14:48 STEP 1380 H= 345:00 +CPU= 61.895
Moving to projects directory: /var/lib/boinc/slots/14/ICMGGh7zg+001380
Moving to projects directory: /var/lib/boinc/slots/14/ICMSHh7zg+001380
Moving to projects directory: /var/lib/boinc/slots/14/ICMUAh7zg+001380
  20:15:42 STEP 1381 H= 345:15 +CPU= 52.750
  20:16:38 STEP 1382 H= 345:30 +CPU= 54.320
  20:17:35 STEP 1383 H= 345:45 +CPU= 56.165
  20:18:29 STEP 1384 H= 346:00 +CPU= 50.591
  20:19:19 STEP 1385 H= 346:15 +CPU= 48.437
  20:20:06 STEP 1386 H= 346:30 +CPU= 45.613
  20:20:58 STEP 1387 H= 346:45 +CPU= 48.900
  20:21:49 STEP 1388 H= 347:00 +CPU= 49.939
  20:22:38 STEP 1389 H= 347:15 +CPU= 47.919
  20:23:25 STEP 1390 H= 347:30 +CPU= 46.460
  20:24:14 STEP 1391 H= 347:45 +CPU= 46.937
  20:25:23 STEP 1392 H= 348:00 +CPU= 67.497
Moving to projects directory: /var/lib/boinc/slots/14/ICMGGh7zg+001392
Moving to projects directory: /var/lib/boinc/slots/14/ICMSHh7zg+001392
Moving to projects directory: /var/lib/boinc/slots/14/ICMUAh7zg+001392
  20:26:10 STEP 1393 H= 348:15 +CPU= 45.268
  20:26:58 STEP 1394 H= 348:30 +CPU= 46.290
  20:27:49 STEP 1395 H= 348:45 +CPU= 49.688
  20:28:45 STEP 1396 H= 349:00 +CPU= 54.222
  20:29:39 STEP 1397 H= 349:15 +CPU= 52.143
  20:30:34 STEP 1398 H= 349:30 +CPU= 52.439
  20:31:29 STEP 1399 H= 349:45 +CPU= 52.341
  20:32:20 STEP 1400 H= 350:00 +CPU= 50.210
  20:33:08 STEP 1401 H= 350:15 +CPU= 46.316
  20:33:53 STEP 1402 H= 350:30 +CPU= 43.490
  20:34:38 STEP 1403 H= 350:45 +CPU= 43.650
  20:35:37 STEP 1404 H= 351:00 +CPU= 57.740
Moving to projects directory: /var/lib/boinc/slots/14/ICMGGh7zg+001404
Moving to projects directory: /var/lib/boinc/slots/14/ICMSHh7zg+001404
Moving to projects directory: /var/lib/boinc/slots/14/ICMUAh7zg+001404
  20:36:32 STEP 1405 H= 351:15 +CPU= 52.805
  20:37:23 STEP 1406 H= 351:30 +CPU= 49.554
  20:38:13 STEP 1407 H= 351:45 +CPU= 48.851
  20:39:06 STEP 1408 H= 352:00 +CPU= 50.225
  20:40:02 STEP 1409 H= 352:15 +CPU= 53.704
  20:40:50 STEP 1410 H= 352:30 +CPU= 46.776
  20:41:35 STEP 1411 H= 352:45 +CPU= 44.287
  20:42:30 STEP 1412 H= 353:00 +CPU= 53.271
  20:43:28 STEP 1413 H= 353:15 +CPU= 55.432
  20:44:19 STEP 1414 H= 353:30 +CPU= 50.608
  20:45:11 STEP 1415 H= 353:45 +CPU= 49.584
  20:46:10 STEP 1416 H= 354:00 +CPU= 57.946
Moving to projects directory: /var/lib/boinc/slots/14/ICMGGh7zg+001416
Moving to projects directory: /var/lib/boinc/slots/14/ICMSHh7zg+001416
Moving to projects directory: /var/lib/boinc/slots/14/ICMUAh7zg+001416
  20:46:55 STEP 1417 H= 354:15 +CPU= 44.609
  20:47:45 STEP 1418 H= 354:30 +CPU= 48.397
  20:48:44 STEP 1419 H= 354:45 +CPU= 57.595
  20:49:38 STEP 1420 H= 355:00 +CPU= 52.353
  20:50:34 STEP 1421 H= 355:15 +CPU= 54.555
  20:51:24 STEP 1422 H= 355:30 +CPU= 49.753
  20:52:23 STEP 1423 H= 355:45 +CPU= 58.404
  20:53:07 STEP 1424 H= 356:00 +CPU= 43.398
  20:54:03 STEP 1425 H= 356:15 +CPU= 54.919
  20:54:48 STEP 1426 H= 356:30 +CPU= 44.440
  20:55:34 STEP 1427 H= 356:45 +CPU= 44.013
  20:56:21 STEP 1428 H= 357:00 +CPU= 46.524
Moving to projects directory: /var/lib/boinc/slots/14/ICMGGh7zg+001428
Moving to projects directory: /var/lib/boinc/slots/14/ICMSHh7zg+001428
Moving to projects directory: /var/lib/boinc/slots/14/ICMUAh7zg+001428
  20:57:08 STEP 1429 H= 357:15 +CPU= 45.864
  20:57:55 STEP 1430 H= 357:30 +CPU= 45.542
  20:58:42 STEP 1431 H= 357:45 +CPU= 46.370
  20:59:29 STEP 1432 H= 358:00 +CPU= 45.655
  21:00:15 STEP 1433 H= 358:15 +CPU= 44.516
  21:00:56 STEP 1434 H= 358:30 +CPU= 40.517
  21:01:43 STEP 1435 H= 358:45 +CPU= 46.125
  21:02:31 STEP 1436 H= 359:00 +CPU= 46.566
  21:03:18 STEP 1437 H= 359:15 +CPU= 46.082
  21:04:01 STEP 1438 H= 359:30 +CPU= 41.820
  21:04:48 STEP 1439 H= 359:45 +CPU= 46.229
  21:05:55 STEP 1440 H= 360:00 +CPU= 64.656
..The child process terminated with status: 0
>>> Printing last 70 lines from file: NODE.001_01

 
DDH------------------------------------------------------------------------
I                                                                         I
I TIME =  360.000H DDH * Diagnostic Var.,     AREA                GLOBAL  I
I                                                                         I
I-------------------------------------------------------------------------I
I   VARIABLE   I      Initial value         I        Current value        I
I-------------------------------------------------------------------------I
I  OMEGA (P/S) I     -0.5113626359E-21      I       0.2371489124E-04      I
ICloud fractionI      0.0000000000E+00      I       0.8016902129E-01      I
I  Relat. Hum. I      0.4704424567E+00      I       0.4609506193E+00      I
I  PASS 01     I      0.0000000000E+00      I       0.1292402758E+04      I
DDH------------------------------------------------------------------------
  
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1440 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001440 MODE=a
 IO-STREAM CLOSED - ICMGGh7zg+001440
 MPI-TASK:   1 -    327254724 BYTES IN      7 RECORDS TRANSFERRED IN    0.0014 SECONDS233753.4 Mbytes/s, TOTAL TIME=    0.0197( 7.1%)
 21:05:55 STEP 1440 H= 360:00 +CPU= 64.656
 END CNT3
 NSTEP =  1440 CNT0   000000000  47.332  47.332  49.088   0.964               0               0               0               0               0
IO-STREAM STATISTICS, TOTAL NO OF REC/BYTES READ           46/  1069125708 WRITTEN         8580/************
 -TASK-OPENED-OPEN-RECS IN -KBYTE IN    -RECS OUT-KBYTE OUT-WALL   -WALL IN-WALL
  OU-TOT IN -TOT OUT
   1    153    0       46      1044068     8580 ************    30.9     3.8    27.1     6.3    59.5
===-=== START OF TIMING STATISTICS ===-===
 
STATS FOR ALL TASKS
 NUM ROUTINE                                     CALLS  MEAN(ms)   MAX(ms)   FRAC(%)  UNBAL(%)
   0 CNT0     - COMPLETE EXECUTION                   1 ********* *********    100.00      0.00
   1 CNT4     - FORWARD INTEGRATION                  1 ********* *********     99.95      0.00
   8 SCAN2M - GRID-POINT DYNAMICS                  313   10334.7   10334.7     18.87      0.00
   9 SPCM     - SPECTRAL COMP.                     288    2753.6    2753.6      4.63      0.00
  10 SCAN2M - PHYSICS                              289   26782.6   26782.6     45.15      0.00
  11 IOPACK   - OUTPUT P.P. RESULTS                 24    1754.8    1754.8      0.25      0.00
  12 SPNORM   - SPECTRAL NORM COMP.                 13     133.6     133.6      0.01      0.00
  17 GRIDFPOS IN CNT4                               24      67.1      67.1      0.01      0.00
  27 RESTART FILES - WRITING                         6    7019.2    7019.2      0.25      0.00
  28 RESTART FILES - READING                         1    8405.4    8405.4      0.05      0.00
  29 SU4FPOS IN CNT4                                24       0.4       0.4      0.00      0.00
  30 DYNFPOS IN CNT4                                24   10641.0   10641.0      1.49      0.00
  31 POSDDH IN STEPO                                 2       8.5       8.5      0.00      0.00
  37 CPGLAG   - SL COMPUTATIONS                    289   11510.9   11510.9     19.41      0.00
  39 SU0YOMB                                         1     252.5     252.5      0.00      0.00
  51 SCAN2M   - SL COMM. PART 1                    289      41.7      41.7      0.07      0.00
  54 SPCM     - M TO S/S TO M TRANSP.              288    1274.1    1274.1      2.14      0.00
  55 SPCIMPF  - S TO M/M TO S TRANSP.              288     136.3     136.3      0.23      0.00
  56 SPNORM   - SPECTRAL NORM COMM.                 13       1.3       1.3      0.00      0.00
 102 LTINV_CTL   - INVERSE LEGENDRE TRANSFORM      628     780.3     780.3      2.86      0.00
 103 LTDIR_CTL   - DIRECT LEGENDRE TRANSFORM       600     437.5     437.5      1.53      0.00
 106 FTDIR_CTL   - DIRECT FOURIER TRANSFORM        600     158.9     158.9      0.56      0.00
 107 FTINV_CTL   - INVERSE FOURIER TRANSFORM       628     254.7     254.7      0.93      0.00
 140 SULEG       - COMP. OF LEGENDRE POL.            1      20.1      20.1      0.00      0.00
 152 LTINV_CTL   - M TO L TRANSPOSITION            628     140.9     140.9      0.52      0.00
 153 LTDIR_CTL   - L TO M TRANSPOSITION            600      91.3      91.3      0.32      0.00
 157 FTINV_CTL   - L TO G TRANSPOSITION            628     195.3     195.3      0.72      0.00
 158 FTDIR_CTL   - G TO L TRANSPOSITION            600     100.4     100.4      0.35      0.00
 400 GSTATS                                      34412       0.0       0.0      0.00      0.00
 401 GSTATS HOOK                                 31964       0.0       0.0      0.00      0.00
TOTAL MEASURED IMBALANCE =       0.0 SECONDS,  0.0 PERCENT
TOTAL WALLCLOCK TIME    17143.1 CPU TIME   16305.8 VECTOR TIME    16305.8


===-=== END   OF TIMING STATISTICS ===-===
FORECAST DAYS PER DAY       75.6
  *** END CNT0 *** 
------------------------------------------------
Moving to projects directory: /var/lib/boinc/slots/14/ICMGGh7zg+001440
Moving to projects directory: /var/lib/boinc/slots/14/ICMSHh7zg+001440
Moving to projects directory: /var/lib/boinc/slots/14/ICMUAh7zg+001440
Adding to the zip: /var/lib/boinc/slots/14/NODE.001_01
Adding to the zip: /var/lib/boinc/slots/14/ifs.stat
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMGGh7zg+001344
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMSHh7zg+001344
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMUAh7zg+001344
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMGGh7zg+001356
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMSHh7zg+001356
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMUAh7zg+001356
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMGGh7zg+001368
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMSHh7zg+001368
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMUAh7zg+001368
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMGGh7zg+001380
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMSHh7zg+001380
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMUAh7zg+001380
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMGGh7zg+001392
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMSHh7zg+001392
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMUAh7zg+001392
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMGGh7zg+001404
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMSHh7zg+001404
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMUAh7zg+001404
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMGGh7zg+001416
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMSHh7zg+001416
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMUAh7zg+001416
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMGGh7zg+001428
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMSHh7zg+001428
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMUAh7zg+001428
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMGGh7zg+001440
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMSHh7zg+001440
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285709/ICMUAh7zg+001440
Zipping up the final file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_a2zu_2016092300_20_1017_12285709_0_r295598715_14.zip
Uploading the final file: upload_file_14.zip
Uploading trickle at timestep: 1295100
21:09:27 (7976): called boinc_finish(0)

</stderr_txt>
<message>
upload failure: <file_xfer_error>
  <file_name>oifs_43r3_bl_a2zu_2016092300_20_1017_12285709_0_r295598715_15.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a2zu_2016092300_20_1017_12285709_0_r295598715_16.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a2zu_2016092300_20_1017_12285709_0_r295598715_17.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a2zu_2016092300_20_1017_12285709_0_r295598715_18.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a2zu_2016092300_20_1017_12285709_0_r295598715_19.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
</message>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
10 Jun 2024 19:37:51 1533094 22437318 oifs_43r3_bl_a2zu_2016092300_20_1017_12285709_0 1,295,100 72,770 0.0562


©2024 climateprediction.net