climateprediction.net home page
Task 22437271

Task 22437271

Name oifs_43r3_bl_a2yj_2016092300_20_1017_12285662_0
Workunit 12285662
Created 7 Jun 2024, 16:23:24 UTC
Sent 7 Jun 2024, 16:31:19 UTC
Report deadline 6 Aug 2024, 16:31:19 UTC
Received 10 Jun 2024, 18:36:33 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 0 (0x00000000)
Computer ID 1533094
Run time 19 hours 23 min 51 sec
CPU time 18 hours 42 min 47 sec
Validate state Invalid
Credit 1,976.32
Device peak FLOPS 7.75 GFLOPS
Application version OpenIFS 43r3 Baroclinic Lifecycle v1.13
x86_64-pc-linux-gnu
Peak working set size 5,541.07 MB
Peak swap size 5,980.80 MB
Peak disk usage 1,357.56 MB
Stderr
<core_client_version>7.24.1</core_client_version>
<![CDATA[
<stderr_txt>
6/ICMSHh7zg+001008
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAh7zg+001008
  06:39:19 STEP 1009 H= 252:15 +CPU= 58.531
  06:40:24 STEP 1010 H= 252:30 +CPU= 62.529
  06:41:25 STEP 1011 H= 252:45 +CPU= 58.960
  06:42:26 STEP 1012 H= 253:00 +CPU= 57.885
  06:43:24 STEP 1013 H= 253:15 +CPU= 56.032
  06:44:19 STEP 1014 H= 253:30 +CPU= 54.095
  06:45:19 STEP 1015 H= 253:45 +CPU= 58.560
  06:46:16 STEP 1016 H= 254:00 +CPU= 55.548
  06:47:22 STEP 1017 H= 254:15 +CPU= 63.827
  06:48:21 STEP 1018 H= 254:30 +CPU= 56.569
  06:49:24 STEP 1019 H= 254:45 +CPU= 60.359
  06:50:37 STEP 1020 H= 255:00 +CPU= 70.126
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGh7zg+001020
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHh7zg+001020
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAh7zg+001020
  06:51:40 STEP 1021 H= 255:15 +CPU= 59.438
  06:52:42 STEP 1022 H= 255:30 +CPU= 59.396
  06:53:44 STEP 1023 H= 255:45 +CPU= 59.548
  06:54:45 STEP 1024 H= 256:00 +CPU= 59.773
  06:55:48 STEP 1025 H= 256:15 +CPU= 61.113
  06:56:46 STEP 1026 H= 256:30 +CPU= 55.976
  06:57:45 STEP 1027 H= 256:45 +CPU= 57.311
  06:58:40 STEP 1028 H= 257:00 +CPU= 54.135
  06:59:42 STEP 1029 H= 257:15 +CPU= 58.910
  07:00:37 STEP 1030 H= 257:30 +CPU= 54.355
  07:01:36 STEP 1031 H= 257:45 +CPU= 56.339
  07:02:46 STEP 1032 H= 258:00 +CPU= 68.601
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGh7zg+001032
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHh7zg+001032
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAh7zg+001032
  07:03:43 STEP 1033 H= 258:15 +CPU= 55.418
  07:04:45 STEP 1034 H= 258:30 +CPU= 59.497
  07:05:43 STEP 1035 H= 258:45 +CPU= 55.495
  07:06:44 STEP 1036 H= 259:00 +CPU= 59.178
  07:07:45 STEP 1037 H= 259:15 +CPU= 57.824
  07:08:45 STEP 1038 H= 259:30 +CPU= 57.622
  07:09:41 STEP 1039 H= 259:45 +CPU= 54.705
  07:10:34 STEP 1040 H= 260:00 +CPU= 51.596
  07:11:34 STEP 1041 H= 260:15 +CPU= 57.898
  07:12:38 STEP 1042 H= 260:30 +CPU= 61.362
  07:13:38 STEP 1043 H= 260:45 +CPU= 58.152
  07:14:51 STEP 1044 H= 261:00 +CPU= 69.736
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGh7zg+001044
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHh7zg+001044
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAh7zg+001044
  07:15:50 STEP 1045 H= 261:15 +CPU= 56.873
  07:16:52 STEP 1046 H= 261:30 +CPU= 59.728
  07:17:56 STEP 1047 H= 261:45 +CPU= 60.975
  07:18:54 STEP 1048 H= 262:00 +CPU= 55.745
  07:19:56 STEP 1049 H= 262:15 +CPU= 58.987
  07:20:59 STEP 1050 H= 262:30 +CPU= 60.975
  07:21:55 STEP 1051 H= 262:45 +CPU= 54.992
  07:22:54 STEP 1052 H= 263:00 +CPU= 57.588
  07:23:54 STEP 1053 H= 263:15 +CPU= 58.377
  07:24:52 STEP 1054 H= 263:30 +CPU= 56.963
  07:25:53 STEP 1055 H= 263:45 +CPU= 59.227
  07:27:07 STEP 1056 H= 264:00 +CPU= 72.996
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGh7zg+001056
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHh7zg+001056
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAh7zg+001056
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMGGh7zg+000960
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMSHh7zg+000960
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMUAh7zg+000960
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMGGh7zg+000972
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMSHh7zg+000972
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMUAh7zg+000972
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMGGh7zg+000984
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMSHh7zg+000984
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMUAh7zg+000984
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMGGh7zg+000996
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMSHh7zg+000996
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMUAh7zg+000996
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMGGh7zg+001008
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMSHh7zg+001008
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMUAh7zg+001008
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMGGh7zg+001020
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMSHh7zg+001020
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMUAh7zg+001020
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMGGh7zg+001032
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMSHh7zg+001032
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMUAh7zg+001032
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMGGh7zg+001044
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMSHh7zg+001044
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMUAh7zg+001044
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_a2yj_2016092300_20_1017_12285662_0_r826087009_10.zip
Uploading the intermediate file: upload_file_10.zip
  07:28:06 STEP 1057 H= 264:15 +CPU= 57.384
  07:29:05 STEP 1058 H= 264:30 +CPU= 57.341
  07:30:03 STEP 1059 H= 264:45 +CPU= 57.748
  07:31:07 STEP 1060 H= 265:00 +CPU= 61.915
  07:32:08 STEP 1061 H= 265:15 +CPU= 59.637
  07:33:12 STEP 1062 H= 265:30 +CPU= 62.519
  07:34:15 STEP 1063 H= 265:45 +CPU= 60.021
  07:35:23 STEP 1064 H= 266:00 +CPU= 63.257
  07:36:26 STEP 1065 H= 266:15 +CPU= 59.813
  07:37:28 STEP 1066 H= 266:30 +CPU= 58.993
  07:38:29 STEP 1067 H= 266:45 +CPU= 58.825
  07:39:44 STEP 1068 H= 267:00 +CPU= 72.254
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGh7zg+001068
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHh7zg+001068
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAh7zg+001068
  07:40:46 STEP 1069 H= 267:15 +CPU= 59.595
  07:41:50 STEP 1070 H= 267:30 +CPU= 60.893
  07:42:51 STEP 1071 H= 267:45 +CPU= 59.311
  07:43:53 STEP 1072 H= 268:00 +CPU= 59.798
  07:44:54 STEP 1073 H= 268:15 +CPU= 58.877
  07:45:57 STEP 1074 H= 268:30 +CPU= 60.686
  07:46:56 STEP 1075 H= 268:45 +CPU= 56.635
  07:47:55 STEP 1076 H= 269:00 +CPU= 57.354
  07:48:53 STEP 1077 H= 269:15 +CPU= 56.478
  07:49:57 STEP 1078 H= 269:30 +CPU= 59.773
  07:50:58 STEP 1079 H= 269:45 +CPU= 59.132
  07:52:19 STEP 1080 H= 270:00 +CPU= 77.915
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGh7zg+001080
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHh7zg+001080
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAh7zg+001080
  07:53:23 STEP 1081 H= 270:15 +CPU= 61.520
  07:54:28 STEP 1082 H= 270:30 +CPU= 62.410
  07:55:32 STEP 1083 H= 270:45 +CPU= 61.192
  07:56:30 STEP 1084 H= 271:00 +CPU= 57.095
  07:57:33 STEP 1085 H= 271:15 +CPU= 59.792
  07:58:35 STEP 1086 H= 271:30 +CPU= 59.586
  07:59:35 STEP 1087 H= 271:45 +CPU= 56.798
  08:00:41 STEP 1088 H= 272:00 +CPU= 62.911
  08:01:43 STEP 1089 H= 272:15 +CPU= 59.237
  08:02:42 STEP 1090 H= 272:30 +CPU= 57.288
  08:03:44 STEP 1091 H= 272:45 +CPU= 59.404
  08:04:54 STEP 1092 H= 273:00 +CPU= 67.238
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGh7zg+001092
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHh7zg+001092
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAh7zg+001092
  08:05:57 STEP 1093 H= 273:15 +CPU= 60.576
  08:06:59 STEP 1094 H= 273:30 +CPU= 58.963
  08:07:58 STEP 1095 H= 273:45 +CPU= 57.402
  08:08:59 STEP 1096 H= 274:00 +CPU= 59.248
  08:09:59 STEP 1097 H= 274:15 +CPU= 57.790
  08:11:02 STEP 1098 H= 274:30 +CPU= 60.615
  08:12:06 STEP 1099 H= 274:45 +CPU= 60.828
  08:13:04 STEP 1100 H= 275:00 +CPU= 56.544
  08:14:06 STEP 1101 H= 275:15 +CPU= 58.684
  08:15:05 STEP 1102 H= 275:30 +CPU= 57.207
  08:16:08 STEP 1103 H= 275:45 +CPU= 60.302
  08:17:32 STEP 1104 H= 276:00 +CPU= 80.488
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGh7zg+001104
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHh7zg+001104
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAh7zg+001104
  08:18:30 STEP 1105 H= 276:15 +CPU= 55.890
  08:19:32 STEP 1106 H= 276:30 +CPU= 59.482
  08:20:36 STEP 1107 H= 276:45 +CPU= 60.496
  08:21:38 STEP 1108 H= 277:00 +CPU= 59.511
  08:22:38 STEP 1109 H= 277:15 +CPU= 57.787
  08:23:42 STEP 1110 H= 277:30 +CPU= 59.864
  08:24:47 STEP 1111 H= 277:45 +CPU= 62.546
  08:25:48 STEP 1112 H= 278:00 +CPU= 59.679
  08:26:51 STEP 1113 H= 278:15 +CPU= 60.636
  08:27:49 STEP 1114 H= 278:30 +CPU= 55.907
  08:28:47 STEP 1115 H= 278:45 +CPU= 56.066
  08:30:02 STEP 1116 H= 279:00 +CPU= 72.679
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGh7zg+001116
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHh7zg+001116
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAh7zg+001116
  08:31:00 STEP 1117 H= 279:15 +CPU= 55.931
  08:32:03 STEP 1118 H= 279:30 +CPU= 60.573
  08:33:04 STEP 1119 H= 279:45 +CPU= 58.264
  08:34:07 STEP 1120 H= 280:00 +CPU= 61.106
  08:35:11 STEP 1121 H= 280:15 +CPU= 60.504
  08:36:14 STEP 1122 H= 280:30 +CPU= 59.288
  08:37:19 STEP 1123 H= 280:45 +CPU= 62.709
  08:38:21 STEP 1124 H= 281:00 +CPU= 60.169
  08:39:27 STEP 1125 H= 281:15 +CPU= 62.383
  08:40:27 STEP 1126 H= 281:30 +CPU= 58.613
  08:41:26 STEP 1127 H= 281:45 +CPU= 57.238
  08:42:43 STEP 1128 H= 282:00 +CPU= 73.140
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGh7zg+001128
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHh7zg+001128
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAh7zg+001128
  08:43:44 STEP 1129 H= 282:15 +CPU= 58.862
  08:44:43 STEP 1130 H= 282:30 +CPU= 57.819
  08:45:47 STEP 1131 H= 282:45 +CPU= 60.564
  08:46:46 STEP 1132 H= 283:00 +CPU= 58.828
  08:47:48 STEP 1133 H= 283:15 +CPU= 58.266
  08:48:45 STEP 1134 H= 283:30 +CPU= 55.526
  08:49:49 STEP 1135 H= 283:45 +CPU= 60.946
  08:50:53 STEP 1136 H= 284:00 +CPU= 60.652
  08:51:52 STEP 1137 H= 284:15 +CPU= 56.789
  08:52:52 STEP 1138 H= 284:30 +CPU= 58.112
  08:53:53 STEP 1139 H= 284:45 +CPU= 59.842
  08:55:14 STEP 1140 H= 285:00 +CPU= 76.753
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGh7zg+001140
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHh7zg+001140
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAh7zg+001140
  08:56:16 STEP 1141 H= 285:15 +CPU= 59.606
  08:57:19 STEP 1142 H= 285:30 +CPU= 59.890
  08:58:23 STEP 1143 H= 285:45 +CPU= 61.916
  08:59:27 STEP 1144 H= 286:00 +CPU= 61.842
  09:00:31 STEP 1145 H= 286:15 +CPU= 60.988
  09:01:36 STEP 1146 H= 286:30 +CPU= 61.939
  09:02:38 STEP 1147 H= 286:45 +CPU= 59.066
  09:03:42 STEP 1148 H= 287:00 +CPU= 60.236
  09:04:44 STEP 1149 H= 287:15 +CPU= 59.841
  09:05:45 STEP 1150 H= 287:30 +CPU= 58.807
  09:06:47 STEP 1151 H= 287:45 +CPU= 58.310
  09:08:14 STEP 1152 H= 288:00 +CPU= 83.478
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGh7zg+001152
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHh7zg+001152
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAh7zg+001152
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMGGh7zg+001056
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMSHh7zg+001056
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMUAh7zg+001056
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMGGh7zg+001068
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMSHh7zg+001068
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMUAh7zg+001068
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMGGh7zg+001080
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMSHh7zg+001080
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMUAh7zg+001080
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMGGh7zg+001092
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMSHh7zg+001092
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMUAh7zg+001092
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMGGh7zg+001104
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMSHh7zg+001104
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMUAh7zg+001104
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMGGh7zg+001116
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMSHh7zg+001116
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMUAh7zg+001116
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMGGh7zg+001128
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMSHh7zg+001128
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMUAh7zg+001128
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMGGh7zg+001140
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMSHh7zg+001140
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMUAh7zg+001140
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_a2yj_2016092300_20_1017_12285662_0_r826087009_11.zip
Uploading the intermediate file: upload_file_11.zip
  09:09:15 STEP 1153 H= 288:15 +CPU= 58.683
  09:10:15 STEP 1154 H= 288:30 +CPU= 57.927
  09:11:15 STEP 1155 H= 288:45 +CPU= 57.310
  09:12:20 STEP 1156 H= 289:00 +CPU= 62.647
  09:13:21 STEP 1157 H= 289:15 +CPU= 59.117
  09:14:22 STEP 1158 H= 289:30 +CPU= 58.304
  09:15:24 STEP 1159 H= 289:45 +CPU= 60.476
  09:16:27 STEP 1160 H= 290:00 +CPU= 60.932
  09:17:30 STEP 1161 H= 290:15 +CPU= 59.376
  09:18:30 STEP 1162 H= 290:30 +CPU= 58.313
  09:19:32 STEP 1163 H= 290:45 +CPU= 59.128
  09:20:43 STEP 1164 H= 291:00 +CPU= 70.341
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGh7zg+001164
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHh7zg+001164
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAh7zg+001164
  09:21:48 STEP 1165 H= 291:15 +CPU= 61.615
  09:22:52 STEP 1166 H= 291:30 +CPU= 60.773
  09:23:56 STEP 1167 H= 291:45 +CPU= 60.511
  09:24:58 STEP 1168 H= 292:00 +CPU= 60.423
  09:26:02 STEP 1169 H= 292:15 +CPU= 62.004
  09:26:59 STEP 1170 H= 292:30 +CPU= 55.560
  09:27:59 STEP 1171 H= 292:45 +CPU= 59.018
  09:29:02 STEP 1172 H= 293:00 +CPU= 60.907
  09:30:02 STEP 1173 H= 293:15 +CPU= 57.774
  09:31:06 STEP 1174 H= 293:30 +CPU= 61.086
  09:32:11 STEP 1175 H= 293:45 +CPU= 59.500
  09:33:34 STEP 1176 H= 294:00 +CPU= 74.935
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGh7zg+001176
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHh7zg+001176
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAh7zg+001176
  09:34:39 STEP 1177 H= 294:15 +CPU= 58.797
  09:35:43 STEP 1178 H= 294:30 +CPU= 59.226
  09:36:48 STEP 1179 H= 294:45 +CPU= 61.803
  09:37:53 STEP 1180 H= 295:00 +CPU= 59.466
  09:38:59 STEP 1181 H= 295:15 +CPU= 62.858
  09:40:05 STEP 1182 H= 295:30 +CPU= 62.530
  09:41:07 STEP 1183 H= 295:45 +CPU= 58.451
  09:42:13 STEP 1184 H= 296:00 +CPU= 61.083
  09:43:17 STEP 1185 H= 296:15 +CPU= 60.549
  09:44:19 STEP 1186 H= 296:30 +CPU= 57.208
  09:45:25 STEP 1187 H= 296:45 +CPU= 61.024
  09:46:44 STEP 1188 H= 297:00 +CPU= 74.157
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGh7zg+001188
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHh7zg+001188
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAh7zg+001188
  09:47:49 STEP 1189 H= 297:15 +CPU= 61.279
  09:48:55 STEP 1190 H= 297:30 +CPU= 62.072
  09:49:58 STEP 1191 H= 297:45 +CPU= 59.895
  09:51:03 STEP 1192 H= 298:00 +CPU= 62.285
  09:52:04 STEP 1193 H= 298:15 +CPU= 58.699
  09:53:08 STEP 1194 H= 298:30 +CPU= 61.169
  09:54:10 STEP 1195 H= 298:45 +CPU= 58.223
  09:55:14 STEP 1196 H= 299:00 +CPU= 59.858
  09:56:15 STEP 1197 H= 299:15 +CPU= 58.553
  09:57:16 STEP 1198 H= 299:30 +CPU= 58.775
  09:58:14 STEP 1199 H= 299:45 +CPU= 56.879
  09:59:41 STEP 1200 H= 300:00 +CPU= 82.381
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGh7zg+001200
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHh7zg+001200
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAh7zg+001200
  10:00:42 STEP 1201 H= 300:15 +CPU= 60.008
  10:01:45 STEP 1202 H= 300:30 +CPU= 59.990
  10:02:44 STEP 1203 H= 300:45 +CPU= 57.944
  10:03:50 STEP 1204 H= 301:00 +CPU= 62.182
  10:04:56 STEP 1205 H= 301:15 +CPU= 62.542
  10:05:55 STEP 1206 H= 301:30 +CPU= 56.696
  10:06:59 STEP 1207 H= 301:45 +CPU= 61.782
  10:07:58 STEP 1208 H= 302:00 +CPU= 57.129
  10:09:01 STEP 1209 H= 302:15 +CPU= 60.211
  10:10:01 STEP 1210 H= 302:30 +CPU= 58.651
  10:11:00 STEP 1211 H= 302:45 +CPU= 57.096
  10:12:13 STEP 1212 H= 303:00 +CPU= 70.644
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGh7zg+001212
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHh7zg+001212
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAh7zg+001212
  10:13:15 STEP 1213 H= 303:15 +CPU= 59.226
  10:14:16 STEP 1214 H= 303:30 +CPU= 59.006
  10:15:19 STEP 1215 H= 303:45 +CPU= 60.997
  10:16:22 STEP 1216 H= 304:00 +CPU= 59.846
  10:17:24 STEP 1217 H= 304:15 +CPU= 59.690
  10:18:24 STEP 1218 H= 304:30 +CPU= 58.237
  10:19:30 STEP 1219 H= 304:45 +CPU= 62.518
  10:20:28 STEP 1220 H= 305:00 +CPU= 56.652
  10:21:35 STEP 1221 H= 305:15 +CPU= 63.611
  10:22:40 STEP 1222 H= 305:30 +CPU= 61.145
  10:23:42 STEP 1223 H= 305:45 +CPU= 59.027
  10:24:59 STEP 1224 H= 306:00 +CPU= 73.218
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGh7zg+001224
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHh7zg+001224
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAh7zg+001224
  10:26:03 STEP 1225 H= 306:15 +CPU= 60.809
  10:27:01 STEP 1226 H= 306:30 +CPU= 57.029
  10:27:58 STEP 1227 H= 306:45 +CPU= 55.647
  10:29:01 STEP 1228 H= 307:00 +CPU= 61.692
  10:30:01 STEP 1229 H= 307:15 +CPU= 58.769
  10:31:04 STEP 1230 H= 307:30 +CPU= 59.660
  10:32:08 STEP 1231 H= 307:45 +CPU= 60.444
  10:33:09 STEP 1232 H= 308:00 +CPU= 59.433
  10:34:15 STEP 1233 H= 308:15 +CPU= 63.006
  10:35:18 STEP 1234 H= 308:30 +CPU= 59.402
  10:36:21 STEP 1235 H= 308:45 +CPU= 59.812
  10:37:36 STEP 1236 H= 309:00 +CPU= 72.316
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGh7zg+001236
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHh7zg+001236
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAh7zg+001236
  10:38:36 STEP 1237 H= 309:15 +CPU= 57.407
  10:39:40 STEP 1238 H= 309:30 +CPU= 60.274
  10:40:43 STEP 1239 H= 309:45 +CPU= 60.562
  10:41:43 STEP 1240 H= 310:00 +CPU= 58.730
  10:42:48 STEP 1241 H= 310:15 +CPU= 61.115
  10:43:52 STEP 1242 H= 310:30 +CPU= 59.366
  10:44:54 STEP 1243 H= 310:45 +CPU= 58.213
  10:45:59 STEP 1244 H= 311:00 +CPU= 61.535
  10:47:01 STEP 1245 H= 311:15 +CPU= 59.223
  10:48:05 STEP 1246 H= 311:30 +CPU= 60.540
  10:49:10 STEP 1247 H= 311:45 +CPU= 61.062
  10:50:38 STEP 1248 H= 312:00 +CPU= 77.119
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGh7zg+001248
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHh7zg+001248
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAh7zg+001248
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMGGh7zg+001152
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMSHh7zg+001152
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMUAh7zg+001152
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMGGh7zg+001164
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMSHh7zg+001164
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMUAh7zg+001164
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMGGh7zg+001176
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMSHh7zg+001176
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMUAh7zg+001176
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMGGh7zg+001188
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMSHh7zg+001188
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMUAh7zg+001188
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMGGh7zg+001200
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMSHh7zg+001200
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMUAh7zg+001200
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMGGh7zg+001212
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMSHh7zg+001212
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMUAh7zg+001212
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMGGh7zg+001224
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMSHh7zg+001224
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMUAh7zg+001224
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMGGh7zg+001236
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMSHh7zg+001236
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMUAh7zg+001236
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_a2yj_2016092300_20_1017_12285662_0_r826087009_12.zip
Uploading the intermediate file: upload_file_12.zip
  10:51:42 STEP 1249 H= 312:15 +CPU= 58.128
  10:52:48 STEP 1250 H= 312:30 +CPU= 62.128
  10:53:50 STEP 1251 H= 312:45 +CPU= 59.312
  10:54:50 STEP 1252 H= 313:00 +CPU= 56.765
  10:55:51 STEP 1253 H= 313:15 +CPU= 58.606
  10:56:51 STEP 1254 H= 313:30 +CPU= 58.050
  10:57:51 STEP 1255 H= 313:45 +CPU= 57.870
  10:58:49 STEP 1256 H= 314:00 +CPU= 55.990
  10:59:48 STEP 1257 H= 314:15 +CPU= 57.434
  11:00:46 STEP 1258 H= 314:30 +CPU= 56.174
  11:01:51 STEP 1259 H= 314:45 +CPU= 61.029
  11:03:05 STEP 1260 H= 315:00 +CPU= 72.021
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGh7zg+001260
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHh7zg+001260
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAh7zg+001260
  11:04:07 STEP 1261 H= 315:15 +CPU= 59.474
  11:05:09 STEP 1262 H= 315:30 +CPU= 60.354
  11:06:13 STEP 1263 H= 315:45 +CPU= 60.378
  11:07:17 STEP 1264 H= 316:00 +CPU= 59.916
  11:08:19 STEP 1265 H= 316:15 +CPU= 58.308
  11:09:20 STEP 1266 H= 316:30 +CPU= 57.352
[EC_DRHOOK:antares:1:1:28703:28703] [20240608:110953:1717837793:49598.542] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 3958MB (heap), 5306MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:antares:1:1:28703:28703] [20240608:110953:1717837793:49598.542] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:antares:1:1:28703:28703] [20240608:110953:1717837793:49598.542] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x820890 (old at (nil))
[EC_DRHOOK:antares:1:1:28703:28703] [20240608:110953:1717837793:49598.542] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1
[EC_DRHOOK:antares:1:1:28703:28703] [20240608:110953:1717837793:49598.542] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1
[EC_DRHOOK:antares:1:1:28703:28703] [20240608:110953:1717837793:49598.542] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3843] 3958 MB (maxheap), 5306 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:antares:1:1:28703:28703] [20240608:110953:1717837793:49598.542] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:antares:1:1:28703:28703] [20240608:110953:1717837793:49598.542] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:antares:1:1:28703:28703] [20240608:110953:1717837793:49598.542] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:antares:1:1:28703:28703] [20240608:110953:1717837793:49598.542] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:antares:1:1:28703:28703] [20240608:110953:1717837793:49598.542] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:antares:1:1:28703:28703] [20240608:110953:1717837793:49598.542] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :       CNT4 
[EC_DRHOOK:antares:1:1:28703:28703] [20240608:110953:1717837793:49598.542] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :        STEPO 
[EC_DRHOOK:antares:1:1:28703:28703] [20240608:110953:1717837793:49598.542] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :         SCAN2M 
[EC_DRHOOK:antares:1:1:28703:28703] [20240608:110953:1717837793:49598.542] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :          GP_MODEL_HEAP 
[EC_DRHOOK:antares:1:1:28703:28703] [20240608:110953:1717837793:49598.542] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :           GP_MODEL 
[EC_DRHOOK:antares:1:1:28703:28703] [20240608:110953:1717837793:49598.542] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :            EC_PHYS_DRV 
[EC_DRHOOK:antares:1:1:28703:28703] [20240608:110953:1717837793:49598.542] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :             >OMP-PHYSICS              (1001) 
[EC_DRHOOK:antares:1:1:28703:28703] [20240608:110953:1717837793:49598.542] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :              EC_PHYS 
[EC_DRHOOK:antares:1:1:28703:28703] [20240608:110953:1717837793:49598.542] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :               CALLPAR 
[EC_DRHOOK:antares:1:1:28703:28703] [20240608:110953:1717837793:49598.542] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :                CLOUD_LAYER 
[EC_DRHOOK:antares:1:1:28703:28703] [20240608:110953:1717837793:49598.542] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :                 CLOUDSC 
[EC_DRHOOK:antares:1:1:28703:28703] [20240608:110954:1717837794:49599.542] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1
[EC_DRHOOK:antares:1:1:28703:28703] [20240608:110954:1717837794:49599.542] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1cf8da0 for signal#15, nsigs = 1
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source             
oifs_43r3_model.e  0000000001CF935B  Unknown               Unknown  Unknown
oifs_43r3_model.e  00000000008214C1  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DD5490  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000117CFEC  cloudsc_                 3157  cloudsc.F90
oifs_43r3_model.e  000000000115C847  cloud_layer_              131  cloud_layer.F90
oifs_43r3_model.e  0000000000FBB064  callpar_                 1223  callpar.F90
oifs_43r3_model.e  0000000000F5FA6B  ec_phys_                  670  ec_phys.F90
oifs_43r3_model.e  0000000000F4DBC4  ec_phys_drv_              599  ec_phys_drv.F90
oifs_43r3_model.e  0000000000BD7131  gp_model_                 613  gp_model.F90
oifs_43r3_model.e  00000000012C5002  gp_model_heap_             74  gp_model_heap.F90
oifs_43r3_model.e  0000000000BACBAE  scan2m_                   535  scan2m.F90
oifs_43r3_model.e  0000000000688153  stepo_                    327  stepo.F90
oifs_43r3_model.e  00000000004CD1DC  cnt4_                    1133  cnt4.F90
oifs_43r3_model.e  00000000004B5D85  cnt3_                     267  cnt3.F90
oifs_43r3_model.e  00000000004B4F52  cnt2_                      88  cnt2.F90
oifs_43r3_model.e  000000000049D638  cnt1_                      92  cnt1.F90
oifs_43r3_model.e  0000000000406FBD  cnt0_                     146  cnt0.F90
oifs_43r3_model.e  0000000000401E7F  MAIN__                     96  master.F90
oifs_43r3_model.e  0000000000401E12  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCC34A  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCDBE7  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000000401CD5  Unknown               Unknown  Unknown
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.13_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a2yj
(argv4) batchid: 1017
(argv5) wuid: 12285662
(argv6) fclen: 20
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/16
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.13
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/16/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/16/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_743b6e8e94ed4a61535542a247f8a63c to: /var/lib/boinc/slots/16/oifs_43r3_bl_a2yj_2016092300_20_1017_12285662.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/16/oifs_43r3_bl_a2yj_2016092300_20_1017_12285662.zip
ic_ancil_file: ic_ancil_12285662
ifsdata_file: ifsdata_12285662
climate_data_file: clim_data_12285662
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /var/lib/boinc/slots/16/ic_ancil_12285662.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/16/ic_ancil_12285662.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_bc976c137a4f5f923e82cbeadf3d71a0 to: /var/lib/boinc/slots/16/ifsdata/ifsdata_12285662.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/16/ifsdata/ifsdata_12285662.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/16/159l_2/clim_data_12285662.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/16/159l_2/clim_data_12285662.zip
Checking for progress XML file: /var/lib/boinc/slots/16/progress_file_12285662.xml
Opened progress file ok : /var/lib/boinc/slots/16/progress_file_12285662.xml
-- Model is restarting --
Adjusting last_iter, 1267, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/16/progress_file_12285662.xml
last_cpu_time: 56418
upload_file_number: 13
last_iter: 1249
last_upload: 1123200
model_completed: 0
total_length_of_simulation: 1728000
result_base_name: oifs_43r3_bl_a2yj_2016092300_20_1017_12285662_0_r826087009
The child process has been launched with process id: 8049
Executing the command: /var/lib/boinc/slots/16/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:antares:1:1:8049:8049] [20240610:162012:1718029212:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2091] fp = 0x314f440
[EC_DRHOOK:antares:1:1:8049:8049] [20240610:162012:1718029212:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:antares:1:1:8049:8049] [20240610:162012:1718029212:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:antares:1:1:8049:8049] [20240610:162012:1718029212:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:antares:1:1:8049:8049] [20240610:162012:1718029212:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:antares:1:1:8049:8049] [20240610:162012:1718029212:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:antares:1:1:8049:8049] [20240610:162012:1718029212:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:antares:1:1:8049:8049] [20240610:162012:1718029212:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:antares:1:1:8049:8049] [20240610:162012:1718029212:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:antares:1:1:8049:8049] [20240610:162012:1718029212:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:antares:1:1:8049:8049] [20240610:162012:1718029212:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:antares:1:1:8049:8049] [20240610:162012:1718029212:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:antares:1:1:8049:8049] [20240610:162012:1718029212:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:antares:1:1:8049:8049] [20240610:162012:1718029212:0.000] [ignore_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:antares:1:1:8049:8049] [20240610:162012:1718029212:0.000] [restore_default_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:antares:1:1:8049:8049] [20240610:162012:1718029212:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:antares:1:1:8049:8049] [20240610:162012:1718029212:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:antares:1:1:8049:8049] [20240610:162012:1718029212:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:antares:1:1:8049:8049] [20240610:162012:1718029212:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x820cf0 (old at (nil))
[EC_DRHOOK:antares:1:1:8049:8049] [20240610:162012:1718029212:0.000] [trapfpe_treatment@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:antares:1:1:8049:8049] [20240610:162012:1718029212:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:antares:1:1:8049:8049] [20240610:162012:1718029212:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:antares:1:1:8049:8049] [20240610:162012:1718029212:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x820cf0 (old at (nil))
[EC_DRHOOK:antares:1:1:8049:8049] [20240610:162012:1718029212:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:antares:1:1:8049:8049] [20240610:162012:1718029212:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:antares:1:1:8049:8049] [20240610:162012:1718029212:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:antares:1:1:8049:8049] [20240610:162012:1718029212:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x820cf0 (old at (nil))
[EC_DRHOOK:antares:1:1:8049:8049] [20240610:162012:1718029212:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x820cf0 (old at (nil))
[EC_DRHOOK:antares:1:1:8049:8049] [20240610:162012:1718029212:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:antares:1:1:8049:8049] [20240610:162012:1718029212:0.000] [catch_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  16:20:55 STEP 1248 H= 312:00 +CPU= 34.594
  16:21:26 STEP 1249 H= 312:15 +CPU= 31.330
  16:21:57 STEP 1250 H= 312:30 +CPU= 30.430
  16:22:26 STEP 1251 H= 312:45 +CPU= 28.741
  16:23:05 STEP 1252 H= 313:00 +CPU= 36.261
  16:23:58 STEP 1253 H= 313:15 +CPU= 47.555
  16:24:52 STEP 1254 H= 313:30 +CPU= 50.203
  16:25:49 STEP 1255 H= 313:45 +CPU= 51.423
  16:26:46 STEP 1256 H= 314:00 +CPU= 52.218
  16:27:46 STEP 1257 H= 314:15 +CPU= 50.860
  16:28:45 STEP 1258 H= 314:30 +CPU= 52.399
  16:29:44 STEP 1259 H= 314:45 +CPU= 52.910
  16:30:56 STEP 1260 H= 315:00 +CPU= 64.922
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGh7zg+001260
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHh7zg+001260
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAh7zg+001260
  16:31:56 STEP 1261 H= 315:15 +CPU= 53.535
  16:32:55 STEP 1262 H= 315:30 +CPU= 51.601
  16:33:52 STEP 1263 H= 315:45 +CPU= 50.957
  16:34:54 STEP 1264 H= 316:00 +CPU= 53.785
  16:35:53 STEP 1265 H= 316:15 +CPU= 50.653
  16:36:50 STEP 1266 H= 316:30 +CPU= 50.568
  16:37:53 STEP 1267 H= 316:45 +CPU= 53.802
  16:38:53 STEP 1268 H= 317:00 +CPU= 52.993
  16:39:53 STEP 1269 H= 317:15 +CPU= 52.890
  16:40:56 STEP 1270 H= 317:30 +CPU= 55.813
  16:41:56 STEP 1271 H= 317:45 +CPU= 51.098
  16:43:09 STEP 1272 H= 318:00 +CPU= 63.161
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGh7zg+001272
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHh7zg+001272
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAh7zg+001272
  16:44:07 STEP 1273 H= 318:15 +CPU= 52.380
  16:45:03 STEP 1274 H= 318:30 +CPU= 51.714
  16:46:02 STEP 1275 H= 318:45 +CPU= 52.489
  16:47:00 STEP 1276 H= 319:00 +CPU= 51.933
  16:47:56 STEP 1277 H= 319:15 +CPU= 50.329
  16:48:53 STEP 1278 H= 319:30 +CPU= 51.837
  16:49:51 STEP 1279 H= 319:45 +CPU= 51.280
  16:50:49 STEP 1280 H= 320:00 +CPU= 52.576
  16:51:46 STEP 1281 H= 320:15 +CPU= 51.815
  16:52:47 STEP 1282 H= 320:30 +CPU= 54.443
  16:53:46 STEP 1283 H= 320:45 +CPU= 52.777
  16:54:59 STEP 1284 H= 321:00 +CPU= 67.091
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGh7zg+001284
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHh7zg+001284
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAh7zg+001284
  16:55:57 STEP 1285 H= 321:15 +CPU= 52.561
  16:56:57 STEP 1286 H= 321:30 +CPU= 54.314
  16:57:57 STEP 1287 H= 321:45 +CPU= 53.595
  16:58:55 STEP 1288 H= 322:00 +CPU= 52.541
  16:59:55 STEP 1289 H= 322:15 +CPU= 54.138
  17:00:54 STEP 1290 H= 322:30 +CPU= 53.460
  17:01:54 STEP 1291 H= 322:45 +CPU= 53.912
  17:02:53 STEP 1292 H= 323:00 +CPU= 53.078
  17:03:51 STEP 1293 H= 323:15 +CPU= 54.424
  17:04:45 STEP 1294 H= 323:30 +CPU= 51.832
  17:05:42 STEP 1295 H= 323:45 +CPU= 54.184
  17:06:54 STEP 1296 H= 324:00 +CPU= 69.938
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGh7zg+001296
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHh7zg+001296
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAh7zg+001296
  17:07:48 STEP 1297 H= 324:15 +CPU= 51.341
  17:08:45 STEP 1298 H= 324:30 +CPU= 54.212
  17:09:39 STEP 1299 H= 324:45 +CPU= 52.646
  17:10:33 STEP 1300 H= 325:00 +CPU= 51.683
  17:11:28 STEP 1301 H= 325:15 +CPU= 52.054
  17:12:25 STEP 1302 H= 325:30 +CPU= 52.588
  17:13:18 STEP 1303 H= 325:45 +CPU= 50.346
  17:14:14 STEP 1304 H= 326:00 +CPU= 52.941
  17:15:16 STEP 1305 H= 326:15 +CPU= 59.430
  17:16:14 STEP 1306 H= 326:30 +CPU= 55.707
  17:17:13 STEP 1307 H= 326:45 +CPU= 55.980
  17:18:26 STEP 1308 H= 327:00 +CPU= 70.491
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGh7zg+001308
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHh7zg+001308
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAh7zg+001308
  17:19:23 STEP 1309 H= 327:15 +CPU= 54.426
  17:20:17 STEP 1310 H= 327:30 +CPU= 52.198
  17:21:17 STEP 1311 H= 327:45 +CPU= 55.507
  17:22:13 STEP 1312 H= 328:00 +CPU= 54.197
  17:23:16 STEP 1313 H= 328:15 +CPU= 60.091
  17:24:17 STEP 1314 H= 328:30 +CPU= 58.119
  17:25:10 STEP 1315 H= 328:45 +CPU= 51.542
  17:26:10 STEP 1316 H= 329:00 +CPU= 58.044
  17:27:08 STEP 1317 H= 329:15 +CPU= 55.913
  17:28:08 STEP 1318 H= 329:30 +CPU= 57.978
  17:29:08 STEP 1319 H= 329:45 +CPU= 57.425
  17:30:20 STEP 1320 H= 330:00 +CPU= 68.780
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGh7zg+001320
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHh7zg+001320
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAh7zg+001320
  17:31:27 STEP 1321 H= 330:15 +CPU= 63.504
  17:32:27 STEP 1322 H= 330:30 +CPU= 57.200
  17:33:28 STEP 1323 H= 330:45 +CPU= 59.444
  17:34:33 STEP 1324 H= 331:00 +CPU= 60.905
  17:35:29 STEP 1325 H= 331:15 +CPU= 54.346
  17:36:33 STEP 1326 H= 331:30 +CPU= 60.464
  17:37:34 STEP 1327 H= 331:45 +CPU= 59.483
  17:38:35 STEP 1328 H= 332:00 +CPU= 59.158
  17:39:31 STEP 1329 H= 332:15 +CPU= 55.202
  17:40:35 STEP 1330 H= 332:30 +CPU= 61.149
  17:41:31 STEP 1331 H= 332:45 +CPU= 53.216
  17:42:49 STEP 1332 H= 333:00 +CPU= 74.700
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGh7zg+001332
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHh7zg+001332
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAh7zg+001332
  17:43:48 STEP 1333 H= 333:15 +CPU= 56.739
  17:44:51 STEP 1334 H= 333:30 +CPU= 60.020
  17:45:50 STEP 1335 H= 333:45 +CPU= 56.137
  17:46:51 STEP 1336 H= 334:00 +CPU= 58.299
  17:47:52 STEP 1337 H= 334:15 +CPU= 59.369
  17:48:55 STEP 1338 H= 334:30 +CPU= 59.279
  17:49:57 STEP 1339 H= 334:45 +CPU= 60.235
  17:50:57 STEP 1340 H= 335:00 +CPU= 58.156
  17:51:53 STEP 1341 H= 335:15 +CPU= 54.898
  17:52:58 STEP 1342 H= 335:30 +CPU= 61.742
  17:54:03 STEP 1343 H= 335:45 +CPU= 61.425
  17:55:26 STEP 1344 H= 336:00 +CPU= 78.530
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGh7zg+001344
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHh7zg+001344
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAh7zg+001344
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMGGh7zg+001248
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMSHh7zg+001248
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMUAh7zg+001248
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMGGh7zg+001260
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMSHh7zg+001260
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMUAh7zg+001260
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMGGh7zg+001272
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMSHh7zg+001272
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMUAh7zg+001272
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMGGh7zg+001284
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMSHh7zg+001284
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMUAh7zg+001284
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMGGh7zg+001296
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMSHh7zg+001296
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMUAh7zg+001296
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMGGh7zg+001308
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMSHh7zg+001308
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMUAh7zg+001308
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMGGh7zg+001320
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMSHh7zg+001320
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMUAh7zg+001320
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMGGh7zg+001332
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMSHh7zg+001332
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMUAh7zg+001332
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_a2yj_2016092300_20_1017_12285662_0_r826087009_13.zip
Uploading the intermediate file: upload_file_13.zip
  17:56:26 STEP 1345 H= 336:15 +CPU= 57.110
  17:57:29 STEP 1346 H= 336:30 +CPU= 59.231
  17:58:28 STEP 1347 H= 336:45 +CPU= 56.381
  17:59:30 STEP 1348 H= 337:00 +CPU= 59.461
  18:00:24 STEP 1349 H= 337:15 +CPU= 51.023
  18:01:23 STEP 1350 H= 337:30 +CPU= 56.387
  18:02:26 STEP 1351 H= 337:45 +CPU= 59.135
  18:03:33 STEP 1352 H= 338:00 +CPU= 63.742
  18:04:31 STEP 1353 H= 338:15 +CPU= 56.161
  18:05:32 STEP 1354 H= 338:30 +CPU= 57.509
  18:06:35 STEP 1355 H= 338:45 +CPU= 59.747
  18:07:50 STEP 1356 H= 339:00 +CPU= 72.633
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGh7zg+001356
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHh7zg+001356
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAh7zg+001356
  18:08:49 STEP 1357 H= 339:15 +CPU= 56.650
  18:09:54 STEP 1358 H= 339:30 +CPU= 61.678
  18:10:54 STEP 1359 H= 339:45 +CPU= 57.470
  18:11:56 STEP 1360 H= 340:00 +CPU= 59.276
  18:12:55 STEP 1361 H= 340:15 +CPU= 56.490
  18:13:53 STEP 1362 H= 340:30 +CPU= 55.898
  18:14:53 STEP 1363 H= 340:45 +CPU= 57.656
  18:15:53 STEP 1364 H= 341:00 +CPU= 57.123
  18:16:53 STEP 1365 H= 341:15 +CPU= 57.957
  18:17:54 STEP 1366 H= 341:30 +CPU= 58.405
  18:18:53 STEP 1367 H= 341:45 +CPU= 56.192
  18:20:10 STEP 1368 H= 342:00 +CPU= 73.878
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGh7zg+001368
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHh7zg+001368
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAh7zg+001368
  18:21:13 STEP 1369 H= 342:15 +CPU= 59.112
  18:22:16 STEP 1370 H= 342:30 +CPU= 57.462
  18:23:19 STEP 1371 H= 342:45 +CPU= 58.893
  18:24:21 STEP 1372 H= 343:00 +CPU= 59.398
  18:25:17 STEP 1373 H= 343:15 +CPU= 54.085
  18:26:16 STEP 1374 H= 343:30 +CPU= 56.725
  18:27:15 STEP 1375 H= 343:45 +CPU= 57.008
  18:28:19 STEP 1376 H= 344:00 +CPU= 60.733
  18:29:21 STEP 1377 H= 344:15 +CPU= 59.775
  18:30:23 STEP 1378 H= 344:30 +CPU= 59.990
  18:31:20 STEP 1379 H= 344:45 +CPU= 54.694
  18:32:40 STEP 1380 H= 345:00 +CPU= 75.500
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGh7zg+001380
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHh7zg+001380
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAh7zg+001380
  18:33:34 STEP 1381 H= 345:15 +CPU= 51.757
  18:34:37 STEP 1382 H= 345:30 +CPU= 60.092
  18:35:33 STEP 1383 H= 345:45 +CPU= 54.717
  18:36:39 STEP 1384 H= 346:00 +CPU= 61.958
  18:37:35 STEP 1385 H= 346:15 +CPU= 53.853
  18:38:31 STEP 1386 H= 346:30 +CPU= 53.534
  18:39:30 STEP 1387 H= 346:45 +CPU= 56.881
  18:40:28 STEP 1388 H= 347:00 +CPU= 56.185
  18:41:29 STEP 1389 H= 347:15 +CPU= 58.538
  18:42:28 STEP 1390 H= 347:30 +CPU= 57.218
  18:43:28 STEP 1391 H= 347:45 +CPU= 57.827
  18:44:52 STEP 1392 H= 348:00 +CPU= 79.567
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGh7zg+001392
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHh7zg+001392
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAh7zg+001392
  18:45:56 STEP 1393 H= 348:15 +CPU= 59.994
  18:46:59 STEP 1394 H= 348:30 +CPU= 61.262
  18:48:02 STEP 1395 H= 348:45 +CPU= 58.568
  18:49:03 STEP 1396 H= 349:00 +CPU= 58.547
  18:50:03 STEP 1397 H= 349:15 +CPU= 57.134
  18:51:02 STEP 1398 H= 349:30 +CPU= 56.116
  18:52:01 STEP 1399 H= 349:45 +CPU= 57.202
  18:53:00 STEP 1400 H= 350:00 +CPU= 56.554
  18:54:03 STEP 1401 H= 350:15 +CPU= 60.055
  18:55:04 STEP 1402 H= 350:30 +CPU= 57.685
  18:56:04 STEP 1403 H= 350:45 +CPU= 57.025
  18:57:23 STEP 1404 H= 351:00 +CPU= 70.760
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGh7zg+001404
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHh7zg+001404
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAh7zg+001404
  18:58:28 STEP 1405 H= 351:15 +CPU= 58.148
  18:59:31 STEP 1406 H= 351:30 +CPU= 58.865
  19:00:31 STEP 1407 H= 351:45 +CPU= 56.695
  19:01:38 STEP 1408 H= 352:00 +CPU= 62.193
  19:02:40 STEP 1409 H= 352:15 +CPU= 58.607
  19:03:41 STEP 1410 H= 352:30 +CPU= 56.966
  19:04:42 STEP 1411 H= 352:45 +CPU= 57.708
  19:05:45 STEP 1412 H= 353:00 +CPU= 59.933
  19:06:45 STEP 1413 H= 353:15 +CPU= 57.476
  19:07:44 STEP 1414 H= 353:30 +CPU= 57.396
  19:08:45 STEP 1415 H= 353:45 +CPU= 59.412
  19:10:05 STEP 1416 H= 354:00 +CPU= 75.541
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGh7zg+001416
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHh7zg+001416
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAh7zg+001416
  19:11:02 STEP 1417 H= 354:15 +CPU= 55.087
  19:12:04 STEP 1418 H= 354:30 +CPU= 59.158
  19:13:02 STEP 1419 H= 354:45 +CPU= 55.944
  19:14:05 STEP 1420 H= 355:00 +CPU= 59.879
  19:15:05 STEP 1421 H= 355:15 +CPU= 57.586
  19:16:04 STEP 1422 H= 355:30 +CPU= 57.271
  19:17:04 STEP 1423 H= 355:45 +CPU= 57.672
  19:17:57 STEP 1424 H= 356:00 +CPU= 51.649
  19:18:46 STEP 1425 H= 356:15 +CPU= 48.666
  19:19:38 STEP 1426 H= 356:30 +CPU= 50.733
  19:20:36 STEP 1427 H= 356:45 +CPU= 56.650
  19:21:52 STEP 1428 H= 357:00 +CPU= 68.139
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGh7zg+001428
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHh7zg+001428
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAh7zg+001428
  19:23:03 STEP 1429 H= 357:15 +CPU= 54.756
  19:24:07 STEP 1430 H= 357:30 +CPU= 54.205
  19:25:04 STEP 1431 H= 357:45 +CPU= 52.583
  19:26:02 STEP 1432 H= 358:00 +CPU= 56.344
  19:26:59 STEP 1433 H= 358:15 +CPU= 55.976
  19:27:57 STEP 1434 H= 358:30 +CPU= 55.573
  19:28:57 STEP 1435 H= 358:45 +CPU= 57.431
  19:29:58 STEP 1436 H= 359:00 +CPU= 58.804
  19:31:00 STEP 1437 H= 359:15 +CPU= 59.193
  19:31:58 STEP 1438 H= 359:30 +CPU= 55.382
  19:32:56 STEP 1439 H= 359:45 +CPU= 55.842
  19:34:13 STEP 1440 H= 360:00 +CPU= 73.274
..The child process terminated with status: 0
>>> Printing last 70 lines from file: NODE.001_01

 
DDH------------------------------------------------------------------------
I                                                                         I
I TIME =  360.000H DDH * Diagnostic Var.,     AREA                GLOBAL  I
I                                                                         I
I-------------------------------------------------------------------------I
I   VARIABLE   I      Initial value         I        Current value        I
I-------------------------------------------------------------------------I
I  OMEGA (P/S) I     -0.6267679131E-21      I       0.5644348782E-07      I
ICloud fractionI      0.0000000000E+00      I       0.6806022299E-03      I
I  Relat. Hum. I      0.2320040958E+00      I       0.2556868859E+00      I
I  PASS 01     I      0.0000000000E+00      I       0.7246092220E+01      I
DDH------------------------------------------------------------------------
  
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1440 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001440 MODE=a
 IO-STREAM CLOSED - ICMGGh7zg+001440
 MPI-TASK:   1 -    317866692 BYTES IN      7 RECORDS TRANSFERRED IN    0.0010 SECONDS317866.7 Mbytes/s, TOTAL TIME=    0.0179( 5.6%)
 19:34:13 STEP 1440 H= 360:00 +CPU= 73.274
 END CNT3
 NSTEP =  1440 CNT0   000000000  52.625  52.625  55.316   0.951               0               0               0               0               0
IO-STREAM STATISTICS, TOTAL NO OF REC/BYTES READ           46/  1069125708 WRITTEN         5720/************
 -TASK-OPENED-OPEN-RECS IN -KBYTE IN    -RECS OUT-KBYTE OUT-WALL   -WALL IN-WALL
  OU-TOT IN -TOT OUT
   1    103    0       46      1044068     5720 ************    21.7     3.5    18.2     5.9    40.2
===-=== START OF TIMING STATISTICS ===-===
 
STATS FOR ALL TASKS
 NUM ROUTINE                                     CALLS  MEAN(ms)   MAX(ms)   FRAC(%)  UNBAL(%)
   0 CNT0     - COMPLETE EXECUTION                   1 ********* *********    100.00      0.00
   1 CNT4     - FORWARD INTEGRATION                  1 ********* *********     99.92      0.00
   8 SCAN2M - GRID-POINT DYNAMICS                  209   11259.5   11259.5     20.21      0.00
   9 SPCM     - SPECTRAL COMP.                     192    2904.5    2904.5      4.79      0.00
  10 SCAN2M - PHYSICS                              193   25883.0   25883.0     42.91      0.00
  11 IOPACK   - OUTPUT P.P. RESULTS                 16    1758.0    1758.0      0.24      0.00
  12 SPNORM   - SPECTRAL NORM COMP.                  9     120.6     120.6      0.01      0.00
  17 GRIDFPOS IN CNT4                               16      72.9      72.9      0.01      0.00
  27 RESTART FILES - WRITING                         4    7030.8    7030.8      0.24      0.00
  28 RESTART FILES - READING                         1    8153.5    8153.5      0.07      0.00
  29 SU4FPOS IN CNT4                                16       0.7       0.7      0.00      0.00
  30 DYNFPOS IN CNT4                                16   12123.5   12123.5      1.67      0.00
  31 POSDDH IN STEPO                                 1      18.8      18.8      0.00      0.00
  37 CPGLAG   - SL COMPUTATIONS                    193   11825.6   11825.6     19.61      0.00
  39 SU0YOMB                                         1     480.4     480.4      0.00      0.00
  51 SCAN2M   - SL COMM. PART 1                    193      43.7      43.7      0.07      0.00
  54 SPCM     - M TO S/S TO M TRANSP.              192    1334.4    1334.4      2.20      0.00
  55 SPCIMPF  - S TO M/M TO S TRANSP.              192     142.4     142.4      0.23      0.00
  56 SPNORM   - SPECTRAL NORM COMM.                  9       1.9       1.9      0.00      0.00
 102 LTINV_CTL   - INVERSE LEGENDRE TRANSFORM      420     827.3     827.3      2.98      0.00
 103 LTDIR_CTL   - DIRECT LEGENDRE TRANSFORM       400     455.1     455.1      1.56      0.00
 106 FTDIR_CTL   - DIRECT FOURIER TRANSFORM        400     169.5     169.5      0.58      0.00
 107 FTINV_CTL   - INVERSE FOURIER TRANSFORM       420     269.7     269.7      0.97      0.00
 140 SULEG       - COMP. OF LEGENDRE POL.            1      26.6      26.6      0.00      0.00
 152 LTINV_CTL   - M TO L TRANSPOSITION            420     148.7     148.7      0.54      0.00
 153 LTDIR_CTL   - L TO M TRANSPOSITION            400      95.2      95.2      0.33      0.00
 157 FTINV_CTL   - L TO G TRANSPOSITION            420     206.7     206.7      0.75      0.00
 158 FTDIR_CTL   - G TO L TRANSPOSITION            400     105.2     105.2      0.36      0.00
 400 GSTATS                                      23311       0.0       0.0      0.00      0.00
 401 GSTATS HOOK                                 21635       0.0       0.0      0.00      0.00
TOTAL MEASURED IMBALANCE =       0.0 SECONDS,  0.0 PERCENT
TOTAL WALLCLOCK TIME    11641.4 CPU TIME   10949.6 VECTOR TIME    10949.6


===-=== END   OF TIMING STATISTICS ===-===
FORECAST DAYS PER DAY      111.4
  *** END CNT0 *** 
------------------------------------------------
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGh7zg+001440
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHh7zg+001440
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAh7zg+001440
Adding to the zip: /var/lib/boinc/slots/16/NODE.001_01
Adding to the zip: /var/lib/boinc/slots/16/ifs.stat
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMGGh7zg+001344
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMSHh7zg+001344
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMUAh7zg+001344
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMGGh7zg+001356
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMSHh7zg+001356
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMUAh7zg+001356
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMGGh7zg+001368
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMSHh7zg+001368
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMUAh7zg+001368
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMGGh7zg+001380
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMSHh7zg+001380
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMUAh7zg+001380
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMGGh7zg+001392
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMSHh7zg+001392
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMUAh7zg+001392
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMGGh7zg+001404
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMSHh7zg+001404
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMUAh7zg+001404
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMGGh7zg+001416
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMSHh7zg+001416
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMUAh7zg+001416
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMGGh7zg+001428
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMSHh7zg+001428
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMUAh7zg+001428
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMGGh7zg+001440
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMSHh7zg+001440
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12285662/ICMUAh7zg+001440
Zipping up the final file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_a2yj_2016092300_20_1017_12285662_0_r826087009_14.zip
Uploading the final file: upload_file_14.zip
Uploading trickle at timestep: 1295100
19:37:46 (7978): called boinc_finish(0)

</stderr_txt>
<message>
upload failure: <file_xfer_error>
  <file_name>oifs_43r3_bl_a2yj_2016092300_20_1017_12285662_0_r826087009_15.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a2yj_2016092300_20_1017_12285662_0_r826087009_16.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a2yj_2016092300_20_1017_12285662_0_r826087009_17.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a2yj_2016092300_20_1017_12285662_0_r826087009_18.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a2yj_2016092300_20_1017_12285662_0_r826087009_19.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
</message>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
10 Jun 2024 17:35:52 1533094 22437271 oifs_43r3_bl_a2yj_2016092300_20_1017_12285662_0 1,295,100 67,367 0.0520


©2024 climateprediction.net