climateprediction.net home page
Task 22309391

Task 22309391

Name oifs_43r3_bl_a1v2_2016092300_15_991_12209201_0
Workunit 12209201
Created 13 Feb 2023, 16:27:23 UTC
Sent 13 Feb 2023, 16:28:15 UTC
Report deadline 14 Apr 2023, 16:28:15 UTC
Received 14 Feb 2023, 23:14:50 UTC
Server state Over
Outcome Success
Client state Done
Exit status 0 (0x00000000)
Computer ID 1536331
Run time 12 hours 40 min 49 sec
CPU time 12 hours 40 min 49 sec
Validate state Valid
Credit 1,813.14
Device peak FLOPS 6.13 GFLOPS
Application version OpenIFS 43r3 Baroclinic Lifecycle v1.11
x86_64-pc-linux-gnu
Peak working set size 5,075.96 MB
Peak swap size 5,930.79 MB
Peak disk usage 1,376.10 MB
Stderr
<core_client_version>7.18.1</core_client_version>
<![CDATA[
<stderr_txt>
pport/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f520 (old at (nil))
[EC_DRHOOK:W32Ub22:1:1:1803:1803] [20230214:192725:1676399245:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:W32Ub22:1:1:1803:1803] [20230214:192725:1676399245:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:W32Ub22:1:1:1803:1803] [20230214:192725:1676399245:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:W32Ub22:1:1:1803:1803] [20230214:192725:1676399245:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f520 (old at (nil))
[EC_DRHOOK:W32Ub22:1:1:1803:1803] [20230214:192725:1676399245:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f520 (old at (nil))
[EC_DRHOOK:W32Ub22:1:1:1803:1803] [20230214:192725:1676399245:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f520 (old at (nil))
[EC_DRHOOK:W32Ub22:1:1:1803:1803] [20230214:192725:1676399245:0.000] [catch_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  19:28:49 STEP 1056 H= 264:00 +CPU= 61.274
  19:29:32 STEP 1057 H= 264:15 +CPU= 42.630
  19:30:12 STEP 1058 H= 264:30 +CPU= 40.525
  19:30:51 STEP 1059 H= 264:45 +CPU= 38.872
  19:31:31 STEP 1060 H= 265:00 +CPU= 39.763
  19:32:12 STEP 1061 H= 265:15 +CPU= 41.278
  19:32:51 STEP 1062 H= 265:30 +CPU= 39.219
  19:33:32 STEP 1063 H= 265:45 +CPU= 40.204
  19:34:13 STEP 1064 H= 266:00 +CPU= 41.369
  19:34:54 STEP 1065 H= 266:15 +CPU= 40.849
  19:35:34 STEP 1066 H= 266:30 +CPU= 39.662
  19:36:15 STEP 1067 H= 266:45 +CPU= 41.102
  19:37:06 STEP 1068 H= 267:00 +CPU= 51.046
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001068
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001068
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001068
  19:37:47 STEP 1069 H= 267:15 +CPU= 41.171
  19:38:27 STEP 1070 H= 267:30 +CPU= 39.807
  19:39:07 STEP 1071 H= 267:45 +CPU= 39.699
  19:39:49 STEP 1072 H= 268:00 +CPU= 42.504
  19:40:31 STEP 1073 H= 268:15 +CPU= 41.885
  19:41:13 STEP 1074 H= 268:30 +CPU= 41.850
  19:41:55 STEP 1075 H= 268:45 +CPU= 42.004
  19:42:36 STEP 1076 H= 269:00 +CPU= 41.223
  19:43:20 STEP 1077 H= 269:15 +CPU= 43.375
  19:44:00 STEP 1078 H= 269:30 +CPU= 40.349
  19:44:40 STEP 1079 H= 269:45 +CPU= 39.810
  19:45:31 STEP 1080 H= 270:00 +CPU= 51.075
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001080
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001080
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001080
  19:46:13 STEP 1081 H= 270:15 +CPU= 42.050
  19:46:55 STEP 1082 H= 270:30 +CPU= 41.890
  19:47:36 STEP 1083 H= 270:45 +CPU= 40.741
  19:48:19 STEP 1084 H= 271:00 +CPU= 42.737
  19:48:59 STEP 1085 H= 271:15 +CPU= 40.821
  19:49:40 STEP 1086 H= 271:30 +CPU= 40.342
  19:50:19 STEP 1087 H= 271:45 +CPU= 38.871
  19:51:01 STEP 1088 H= 272:00 +CPU= 41.824
  19:51:41 STEP 1089 H= 272:15 +CPU= 40.384
  19:52:22 STEP 1090 H= 272:30 +CPU= 40.470
  19:53:01 STEP 1091 H= 272:45 +CPU= 39.493
  19:53:51 STEP 1092 H= 273:00 +CPU= 50.331
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001092
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001092
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001092
  19:54:31 STEP 1093 H= 273:15 +CPU= 39.395
  19:55:09 STEP 1094 H= 273:30 +CPU= 38.491
  19:55:50 STEP 1095 H= 273:45 +CPU= 40.228
  19:56:31 STEP 1096 H= 274:00 +CPU= 41.165
  19:57:10 STEP 1097 H= 274:15 +CPU= 39.141
  19:57:50 STEP 1098 H= 274:30 +CPU= 39.720
  19:58:30 STEP 1099 H= 274:45 +CPU= 40.765
  19:59:11 STEP 1100 H= 275:00 +CPU= 40.516
  19:59:52 STEP 1101 H= 275:15 +CPU= 41.413
  20:00:34 STEP 1102 H= 275:30 +CPU= 41.291
  20:01:16 STEP 1103 H= 275:45 +CPU= 42.133
  20:02:24 STEP 1104 H= 276:00 +CPU= 67.687
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001104
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001104
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001104
  20:03:09 STEP 1105 H= 276:15 +CPU= 45.059
  20:03:50 STEP 1106 H= 276:30 +CPU= 41.265
  20:04:31 STEP 1107 H= 276:45 +CPU= 41.104
  20:05:14 STEP 1108 H= 277:00 +CPU= 43.297
  20:05:56 STEP 1109 H= 277:15 +CPU= 41.950
  20:06:38 STEP 1110 H= 277:30 +CPU= 41.829
  20:07:20 STEP 1111 H= 277:45 +CPU= 41.954
  20:08:03 STEP 1112 H= 278:00 +CPU= 42.338
  20:08:43 STEP 1113 H= 278:15 +CPU= 40.677
  20:09:25 STEP 1114 H= 278:30 +CPU= 41.836
  20:10:05 STEP 1115 H= 278:45 +CPU= 40.343
  20:10:54 STEP 1116 H= 279:00 +CPU= 48.421
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001116
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001116
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001116
  20:11:35 STEP 1117 H= 279:15 +CPU= 41.079
  20:12:16 STEP 1118 H= 279:30 +CPU= 41.129
  20:12:57 STEP 1119 H= 279:45 +CPU= 40.940
  20:13:37 STEP 1120 H= 280:00 +CPU= 40.206
  20:14:19 STEP 1121 H= 280:15 +CPU= 41.332
  20:15:02 STEP 1122 H= 280:30 +CPU= 42.983
  20:15:42 STEP 1123 H= 280:45 +CPU= 39.883
  20:16:23 STEP 1124 H= 281:00 +CPU= 41.339
  20:17:05 STEP 1125 H= 281:15 +CPU= 42.484
  20:17:47 STEP 1126 H= 281:30 +CPU= 42.040
  20:18:27 STEP 1127 H= 281:45 +CPU= 39.874
  20:19:17 STEP 1128 H= 282:00 +CPU= 50.156
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001128
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001128
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001128
  20:19:59 STEP 1129 H= 282:15 +CPU= 41.097
  20:20:40 STEP 1130 H= 282:30 +CPU= 41.531
  20:21:21 STEP 1131 H= 282:45 +CPU= 40.517
  20:22:03 STEP 1132 H= 283:00 +CPU= 42.093
  20:22:44 STEP 1133 H= 283:15 +CPU= 41.121
  20:23:25 STEP 1134 H= 283:30 +CPU= 41.470
  20:24:07 STEP 1135 H= 283:45 +CPU= 41.717
  20:24:49 STEP 1136 H= 284:00 +CPU= 41.600
  20:25:28 STEP 1137 H= 284:15 +CPU= 39.413
  20:26:08 STEP 1138 H= 284:30 +CPU= 39.384
  20:26:47 STEP 1139 H= 284:45 +CPU= 39.719
  20:27:37 STEP 1140 H= 285:00 +CPU= 49.332
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001140
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001140
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001140
  20:28:18 STEP 1141 H= 285:15 +CPU= 41.534
  20:29:00 STEP 1142 H= 285:30 +CPU= 41.345
  20:29:40 STEP 1143 H= 285:45 +CPU= 40.461
  20:30:22 STEP 1144 H= 286:00 +CPU= 42.181
  20:31:03 STEP 1145 H= 286:15 +CPU= 40.272
  20:31:43 STEP 1146 H= 286:30 +CPU= 40.551
  20:32:24 STEP 1147 H= 286:45 +CPU= 41.335
  20:33:05 STEP 1148 H= 287:00 +CPU= 40.916
  20:33:47 STEP 1149 H= 287:15 +CPU= 41.383
  20:34:27 STEP 1150 H= 287:30 +CPU= 40.011
  20:35:10 STEP 1151 H= 287:45 +CPU= 43.002
  20:36:22 STEP 1152 H= 288:00 +CPU= 72.600
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001152
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001152
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209201/ICMGGh7zg+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209201/ICMSHh7zg+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209201/ICMUAh7zg+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209201/ICMGGh7zg+001068
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209201/ICMSHh7zg+001068
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209201/ICMUAh7zg+001068
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209201/ICMGGh7zg+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209201/ICMSHh7zg+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209201/ICMUAh7zg+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209201/ICMGGh7zg+001092
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209201/ICMSHh7zg+001092
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209201/ICMUAh7zg+001092
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209201/ICMGGh7zg+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209201/ICMSHh7zg+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209201/ICMUAh7zg+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209201/ICMGGh7zg+001116
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209201/ICMSHh7zg+001116
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209201/ICMUAh7zg+001116
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209201/ICMGGh7zg+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209201/ICMSHh7zg+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209201/ICMUAh7zg+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209201/ICMGGh7zg+001140
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209201/ICMSHh7zg+001140
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209201/ICMUAh7zg+001140
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a1v2_2016092300_15_991_12209201_0_r1635881709_11.zip
Uploading the intermediate file: upload_file_11.zip
  20:37:10 STEP 1153 H= 288:15 +CPU= 47.376
  20:37:53 STEP 1154 H= 288:30 +CPU= 43.064
  20:38:35 STEP 1155 H= 288:45 +CPU= 41.673
  20:39:17 STEP 1156 H= 289:00 +CPU= 42.319
  20:39:59 STEP 1157 H= 289:15 +CPU= 41.901
  20:40:43 STEP 1158 H= 289:30 +CPU= 44.170
  20:41:27 STEP 1159 H= 289:45 +CPU= 43.881
  20:42:10 STEP 1160 H= 290:00 +CPU= 43.082
  20:42:52 STEP 1161 H= 290:15 +CPU= 42.021
  20:43:36 STEP 1162 H= 290:30 +CPU= 43.547
  20:44:18 STEP 1163 H= 290:45 +CPU= 42.255
  20:45:11 STEP 1164 H= 291:00 +CPU= 52.705
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001164
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001164
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001164
  20:45:56 STEP 1165 H= 291:15 +CPU= 44.835
  20:46:38 STEP 1166 H= 291:30 +CPU= 42.698
  20:47:24 STEP 1167 H= 291:45 +CPU= 45.998
  20:48:10 STEP 1168 H= 292:00 +CPU= 45.720
  20:48:54 STEP 1169 H= 292:15 +CPU= 43.755
  20:49:35 STEP 1170 H= 292:30 +CPU= 40.792
  20:50:14 STEP 1171 H= 292:45 +CPU= 39.692
  20:50:58 STEP 1172 H= 293:00 +CPU= 43.572
  20:51:43 STEP 1173 H= 293:15 +CPU= 44.881
  20:52:26 STEP 1174 H= 293:30 +CPU= 43.336
  20:53:06 STEP 1175 H= 293:45 +CPU= 39.777
  20:53:56 STEP 1176 H= 294:00 +CPU= 49.945
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001176
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001176
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001176
  20:54:35 STEP 1177 H= 294:15 +CPU= 39.443
  20:55:15 STEP 1178 H= 294:30 +CPU= 39.762
  20:55:55 STEP 1179 H= 294:45 +CPU= 39.421
  20:56:33 STEP 1180 H= 295:00 +CPU= 38.758
  20:57:13 STEP 1181 H= 295:15 +CPU= 39.503
  20:57:52 STEP 1182 H= 295:30 +CPU= 38.873
  20:58:31 STEP 1183 H= 295:45 +CPU= 39.136
  20:59:15 STEP 1184 H= 296:00 +CPU= 43.819
  20:59:55 STEP 1185 H= 296:15 +CPU= 40.229
  21:00:34 STEP 1186 H= 296:30 +CPU= 39.013
[EC_DRHOOK:W32Ub22:1:1:1803:1803] [20230214:210100:1676404860:5615.865] [signal_drhook@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 4057MB (heap), 4897MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:W32Ub22:1:1:1803:1803] [20230214:210100:1676404860:5615.865] [signal_drhook@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:W32Ub22:1:1:1803:1803] [20230214:210100:1676404860:5615.865] [signal_drhook@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f0c0 (old at (nil))
[EC_DRHOOK:W32Ub22:1:1:1803:1803] [20230214:210100:1676404860:5615.865] [signal_drhook@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1
[EC_DRHOOK:W32Ub22:1:1:1803:1803] [20230214:210100:1676404860:5615.865] [signal_drhook@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1
[EC_DRHOOK:W32Ub22:1:1:1803:1803] [20230214:210100:1676404860:5615.865] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3843] 4057 MB (maxheap), 4897 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:W32Ub22:1:1:1803:1803] [20230214:210100:1676404860:5615.865] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:W32Ub22:1:1:1803:1803] [20230214:210100:1676404860:5615.865] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:W32Ub22:1:1:1803:1803] [20230214:210100:1676404860:5615.865] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:W32Ub22:1:1:1803:1803] [20230214:210100:1676404860:5615.865] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:W32Ub22:1:1:1803:1803] [20230214:210100:1676404860:5615.865] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:W32Ub22:1:1:1803:1803] [20230214:210100:1676404860:5615.865] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] :       CNT4 
[EC_DRHOOK:W32Ub22:1:1:1803:1803] [20230214:210100:1676404860:5615.865] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] :        STEPO 
[EC_DRHOOK:W32Ub22:1:1:1803:1803] [20230214:210100:1676404860:5615.865] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] :         SCAN2M 
[EC_DRHOOK:W32Ub22:1:1:1803:1803] [20230214:210100:1676404860:5615.865] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] :          GP_MODEL_HEAP 
[EC_DRHOOK:W32Ub22:1:1:1803:1803] [20230214:210100:1676404860:5615.865] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] :           GP_MODEL 
[EC_DRHOOK:W32Ub22:1:1:1803:1803] [20230214:210100:1676404860:5615.865] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] :            EC_PHYS_DRV 
[EC_DRHOOK:W32Ub22:1:1:1803:1803] [20230214:210100:1676404860:5615.865] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] :             >OMP-PHYSICS              (1001) 
[EC_DRHOOK:W32Ub22:1:1:1803:1803] [20230214:210100:1676404860:5615.865] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] :              EC_PHYS 
[EC_DRHOOK:W32Ub22:1:1:1803:1803] [20230214:210100:1676404860:5615.865] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] :               CALLPAR 
[EC_DRHOOK:W32Ub22:1:1:1803:1803] [20230214:210100:1676404860:5615.865] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] :                TURBULENCE_LAYER 
[EC_DRHOOK:W32Ub22:1:1:1803:1803] [20230214:210100:1676404860:5615.865] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] :                 VDFOUTER 
[EC_DRHOOK:W32Ub22:1:1:1803:1803] [20230214:210100:1676404860:5615.865] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] :                  VDFMAIN 
[EC_DRHOOK:W32Ub22:1:1:1803:1803] [20230214:210100:1676404860:5615.865] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] :                   SURFEXCDRIVER 
[EC_DRHOOK:W32Ub22:1:1:1803:1803] [20230214:210100:1676404860:5615.865] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] :                    SURFEXCDRIVER_CTL_MOD:SURFEXCDRIVER_CTL 
[EC_DRHOOK:W32Ub22:1:1:1803:1803] [20230214:210100:1676404860:5615.865] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] :                     VUPDZ0_MOD:VUPDZ0 
[EC_DRHOOK:W32Ub22:1:1:1803:1803] [20230214:210101:1676404861:5616.870] [signal_drhook@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1
[EC_DRHOOK:W32Ub22:1:1:1803:1803] [20230214:210101:1676404861:5616.870] [signal_drhook@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x401140 for signal#15, nsigs = 1
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source             
oifs_43r3_model.e  000000000081FCF1  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001D8BB20  Unknown               Unknown  Unknown
oifs_43r3_model.e  00000000017DB607  vupdz0_mod_mp_vup         288  vupdz0_mod.F90
oifs_43r3_model.e  00000000017A60F9  surfexcdriver_ctl         499  surfexcdriver_ctl_mod.F90
oifs_43r3_model.e  000000000164526E  surfexcdriver_            847  surfexcdriver.F90
oifs_43r3_model.e  00000000012D1794  vdfmain_                  738  vdfmain.F90
oifs_43r3_model.e  00000000012BA8D9  vdfouter_                 739  vdfouter.F90
oifs_43r3_model.e  00000000012B0762  turbulence_layer_         158  turbulence_layer.F90
oifs_43r3_model.e  0000000001163FC3  callpar_                  869  callpar.F90
oifs_43r3_model.e  00000000010B682F  ec_phys_                  670  ec_phys.F90
oifs_43r3_model.e  00000000010A915C  ec_phys_drv_              599  ec_phys_drv.F90
oifs_43r3_model.e  0000000000E680C3  gp_model_                 613  gp_model.F90
oifs_43r3_model.e  00000000015C1E62  gp_model_heap_             74  gp_model_heap.F90
oifs_43r3_model.e  00000000015AC40E  scan2m_                   535  scan2m.F90
oifs_43r3_model.e  00000000006620B4  stepo_                    327  stepo.F90
oifs_43r3_model.e  00000000004A0E7C  cnt4_                    1133  cnt4.F90
oifs_43r3_model.e  000000000049DC29  cnt3_                     267  cnt3.F90
oifs_43r3_model.e  000000000049CE72  cnt2_                      88  cnt2.F90
oifs_43r3_model.e  000000000049CD68  cnt1_                      92  cnt1.F90
oifs_43r3_model.e  00000000004078AD  cnt0_                     146  cnt0.F90
oifs_43r3_model.e  000000000040276F  MAIN__                     96  master.F90
oifs_43r3_model.e  000000000040270D  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001D8287A  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001D840D7  Unknown               Unknown  Unknown
oifs_43r3_model.e  00000000004025E5  Unknown               Unknown  Unknown
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.11_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a1v2
(argv4) batchid: 991
(argv5) wuid: 12209201
(argv6) fclen: 15
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/4
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.11
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209201
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/4/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/4/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_9c6b1ae893f589c7cc5e0c116b06ea24 to: /var/lib/boinc-client/slots/4/oifs_43r3_bl_a1v2_2016092300_15_991_12209201.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/4/oifs_43r3_bl_a1v2_2016092300_15_991_12209201.zip
ic_ancil_file: ic_ancil_12209201
ifsdata_file: ifsdata_12209201
climate_data_file: clim_data_12209201
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /var/lib/boinc-client/slots/4/ic_ancil_12209201.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/4/ic_ancil_12209201.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_77a4d58b4ad54753d1099da420940c21 to: /var/lib/boinc-client/slots/4/ifsdata/ifsdata_12209201.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/4/ifsdata/ifsdata_12209201.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/4/159l_2/clim_data_12209201.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/4/159l_2/clim_data_12209201.zip
Checking for progress XML file: /var/lib/boinc-client/slots/4/progress_file_12209201.xml
Opened progress file ok : /var/lib/boinc-client/slots/4/progress_file_12209201.xml
-- Model is restarting --
Adjusting last_iter, 1187, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/4/progress_file_12209201.xml
last_cpu_time: 36417
upload_file_number: 12
last_iter: 1153
last_upload: 1036800
model_completed: 0
total_length_of_simulation: 1296000
result_base_name: oifs_43r3_bl_a1v2_2016092300_15_991_12209201_0_r1635881709
The child process has been launched with process id: 1794
Executing the command: /var/lib/boinc-client/slots/4/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:W32Ub22:1:1:1794:1794] [20230214:210232:1676404952:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2091] fp = 0x30e4a00
[EC_DRHOOK:W32Ub22:1:1:1794:1794] [20230214:210232:1676404952:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:W32Ub22:1:1:1794:1794] [20230214:210232:1676404952:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:W32Ub22:1:1:1794:1794] [20230214:210232:1676404952:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:W32Ub22:1:1:1794:1794] [20230214:210232:1676404952:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:W32Ub22:1:1:1794:1794] [20230214:210232:1676404952:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:W32Ub22:1:1:1794:1794] [20230214:210232:1676404952:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:W32Ub22:1:1:1794:1794] [20230214:210232:1676404952:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:W32Ub22:1:1:1794:1794] [20230214:210232:1676404952:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:W32Ub22:1:1:1794:1794] [20230214:210232:1676404952:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:W32Ub22:1:1:1794:1794] [20230214:210232:1676404952:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:W32Ub22:1:1:1794:1794] [20230214:210232:1676404952:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:W32Ub22:1:1:1794:1794] [20230214:210232:1676404952:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:W32Ub22:1:1:1794:1794] [20230214:210232:1676404952:0.000] [ignore_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:W32Ub22:1:1:1794:1794] [20230214:210232:1676404952:0.000] [restore_default_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:W32Ub22:1:1:1794:1794] [20230214:210232:1676404952:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:W32Ub22:1:1:1794:1794] [20230214:210232:1676404952:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f520 (old at (nil))
[EC_DRHOOK:W32Ub22:1:1:1794:1794] [20230214:210232:1676404952:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:W32Ub22:1:1:1794:1794] [20230214:210232:1676404952:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f520 (old at (nil))
[EC_DRHOOK:W32Ub22:1:1:1794:1794] [20230214:210232:1676404952:0.001] [trapfpe_treatment@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:W32Ub22:1:1:1794:1794] [20230214:210232:1676404952:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:W32Ub22:1:1:1794:1794] [20230214:210232:1676404952:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:W32Ub22:1:1:1794:1794] [20230214:210232:1676404952:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f520 (old at (nil))
[EC_DRHOOK:W32Ub22:1:1:1794:1794] [20230214:210232:1676404952:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:W32Ub22:1:1:1794:1794] [20230214:210232:1676404952:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:W32Ub22:1:1:1794:1794] [20230214:210232:1676404952:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:W32Ub22:1:1:1794:1794] [20230214:210232:1676404952:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f520 (old at (nil))
[EC_DRHOOK:W32Ub22:1:1:1794:1794] [20230214:210232:1676404952:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f520 (old at (nil))
[EC_DRHOOK:W32Ub22:1:1:1794:1794] [20230214:210232:1676404952:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f520 (old at (nil))
[EC_DRHOOK:W32Ub22:1:1:1794:1794] [20230214:210232:1676404952:0.001] [catch_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  21:03:33 STEP 1152 H= 288:00 +CPU= 43.229
  21:04:04 STEP 1153 H= 288:15 +CPU= 31.562
  21:04:34 STEP 1154 H= 288:30 +CPU= 30.014
  21:05:05 STEP 1155 H= 288:45 +CPU= 30.354
  21:05:35 STEP 1156 H= 289:00 +CPU= 30.423
  21:06:05 STEP 1157 H= 289:15 +CPU= 29.908
  21:06:35 STEP 1158 H= 289:30 +CPU= 29.914
  21:07:05 STEP 1159 H= 289:45 +CPU= 29.843
  21:07:36 STEP 1160 H= 290:00 +CPU= 30.611
  21:08:06 STEP 1161 H= 290:15 +CPU= 30.382
  21:08:36 STEP 1162 H= 290:30 +CPU= 30.299
  21:09:06 STEP 1163 H= 290:45 +CPU= 29.617
  21:09:43 STEP 1164 H= 291:00 +CPU= 36.660
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001164
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001164
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001164
  21:10:13 STEP 1165 H= 291:15 +CPU= 30.387
  21:10:44 STEP 1166 H= 291:30 +CPU= 30.464
  21:11:14 STEP 1167 H= 291:45 +CPU= 29.991
  21:11:44 STEP 1168 H= 292:00 +CPU= 30.354
  21:12:14 STEP 1169 H= 292:15 +CPU= 30.269
  21:12:44 STEP 1170 H= 292:30 +CPU= 30.013
  21:13:15 STEP 1171 H= 292:45 +CPU= 30.220
  21:13:45 STEP 1172 H= 293:00 +CPU= 30.688
  21:14:15 STEP 1173 H= 293:15 +CPU= 30.126
  21:14:46 STEP 1174 H= 293:30 +CPU= 30.309
  21:15:16 STEP 1175 H= 293:45 +CPU= 29.795
  21:15:53 STEP 1176 H= 294:00 +CPU= 37.228
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001176
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001176
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001176
  21:16:23 STEP 1177 H= 294:15 +CPU= 30.423
  21:16:54 STEP 1178 H= 294:30 +CPU= 30.452
  21:17:23 STEP 1179 H= 294:45 +CPU= 29.774
  21:17:54 STEP 1180 H= 295:00 +CPU= 30.859
  21:18:25 STEP 1181 H= 295:15 +CPU= 30.260
  21:18:55 STEP 1182 H= 295:30 +CPU= 30.416
  21:19:26 STEP 1183 H= 295:45 +CPU= 30.741
  21:19:57 STEP 1184 H= 296:00 +CPU= 30.796
  21:20:27 STEP 1185 H= 296:15 +CPU= 30.177
  21:20:57 STEP 1186 H= 296:30 +CPU= 30.298
  21:21:28 STEP 1187 H= 296:45 +CPU= 30.888
  21:22:06 STEP 1188 H= 297:00 +CPU= 37.437
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001188
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001188
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001188
  21:22:37 STEP 1189 H= 297:15 +CPU= 30.938
  21:23:08 STEP 1190 H= 297:30 +CPU= 31.004
  21:23:39 STEP 1191 H= 297:45 +CPU= 31.095
  21:24:11 STEP 1192 H= 298:00 +CPU= 32.006
  21:24:42 STEP 1193 H= 298:15 +CPU= 31.388
  21:25:14 STEP 1194 H= 298:30 +CPU= 31.714
  21:25:45 STEP 1195 H= 298:45 +CPU= 31.419
  21:26:17 STEP 1196 H= 299:00 +CPU= 31.977
  21:26:49 STEP 1197 H= 299:15 +CPU= 31.406
  21:27:20 STEP 1198 H= 299:30 +CPU= 30.932
  21:27:54 STEP 1199 H= 299:45 +CPU= 34.526
  21:28:42 STEP 1200 H= 300:00 +CPU= 47.473
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001200
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001200
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001200
  21:29:13 STEP 1201 H= 300:15 +CPU= 31.568
  21:29:45 STEP 1202 H= 300:30 +CPU= 31.712
  21:30:16 STEP 1203 H= 300:45 +CPU= 31.360
  21:30:48 STEP 1204 H= 301:00 +CPU= 31.565
  21:31:19 STEP 1205 H= 301:15 +CPU= 30.964
  21:31:50 STEP 1206 H= 301:30 +CPU= 30.766
  21:32:20 STEP 1207 H= 301:45 +CPU= 30.310
  21:32:52 STEP 1208 H= 302:00 +CPU= 31.332
  21:33:22 STEP 1209 H= 302:15 +CPU= 30.408
  21:33:52 STEP 1210 H= 302:30 +CPU= 30.459
  21:34:23 STEP 1211 H= 302:45 +CPU= 30.374
  21:35:01 STEP 1212 H= 303:00 +CPU= 37.985
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001212
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001212
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001212
  21:35:32 STEP 1213 H= 303:15 +CPU= 30.929
  21:36:02 STEP 1214 H= 303:30 +CPU= 30.477
  21:36:33 STEP 1215 H= 303:45 +CPU= 30.259
  21:37:04 STEP 1216 H= 304:00 +CPU= 30.910
  21:37:34 STEP 1217 H= 304:15 +CPU= 30.271
  21:38:04 STEP 1218 H= 304:30 +CPU= 30.367
  21:38:35 STEP 1219 H= 304:45 +CPU= 30.470
  21:39:06 STEP 1220 H= 305:00 +CPU= 30.939
  21:39:36 STEP 1221 H= 305:15 +CPU= 30.408
  21:40:07 STEP 1222 H= 305:30 +CPU= 30.780
  21:40:38 STEP 1223 H= 305:45 +CPU= 30.849
  21:41:15 STEP 1224 H= 306:00 +CPU= 37.623
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001224
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001224
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001224
  21:41:47 STEP 1225 H= 306:15 +CPU= 31.133
  21:42:18 STEP 1226 H= 306:30 +CPU= 31.179
  21:42:50 STEP 1227 H= 306:45 +CPU= 32.064
  21:43:22 STEP 1228 H= 307:00 +CPU= 32.289
  21:43:54 STEP 1229 H= 307:15 +CPU= 32.251
  21:44:27 STEP 1230 H= 307:30 +CPU= 32.106
  21:44:58 STEP 1231 H= 307:45 +CPU= 31.656
  21:45:30 STEP 1232 H= 308:00 +CPU= 32.003
  21:46:02 STEP 1233 H= 308:15 +CPU= 31.880
  21:46:34 STEP 1234 H= 308:30 +CPU= 31.816
  21:47:06 STEP 1235 H= 308:45 +CPU= 31.852
  21:47:45 STEP 1236 H= 309:00 +CPU= 38.626
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001236
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001236
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001236
  21:48:16 STEP 1237 H= 309:15 +CPU= 30.934
  21:48:46 STEP 1238 H= 309:30 +CPU= 30.934
  21:49:17 STEP 1239 H= 309:45 +CPU= 30.773
  21:49:48 STEP 1240 H= 310:00 +CPU= 31.100
  21:50:19 STEP 1241 H= 310:15 +CPU= 30.710
  21:50:50 STEP 1242 H= 310:30 +CPU= 30.563
  21:51:20 STEP 1243 H= 310:45 +CPU= 30.466
  21:51:51 STEP 1244 H= 311:00 +CPU= 30.536
  21:52:22 STEP 1245 H= 311:15 +CPU= 31.128
  21:52:55 STEP 1246 H= 311:30 +CPU= 33.139
  21:53:34 STEP 1247 H= 311:45 +CPU= 39.253
  21:54:18 STEP 1248 H= 312:00 +CPU= 44.072
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001248
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001248
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209201/ICMGGh7zg+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209201/ICMSHh7zg+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209201/ICMUAh7zg+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209201/ICMGGh7zg+001164
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209201/ICMSHh7zg+001164
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209201/ICMUAh7zg+001164
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209201/ICMGGh7zg+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209201/ICMSHh7zg+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209201/ICMUAh7zg+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209201/ICMGGh7zg+001188
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209201/ICMSHh7zg+001188
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209201/ICMUAh7zg+001188
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209201/ICMGGh7zg+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209201/ICMSHh7zg+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209201/ICMUAh7zg+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209201/ICMGGh7zg+001212
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209201/ICMSHh7zg+001212
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209201/ICMUAh7zg+001212
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209201/ICMGGh7zg+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209201/ICMSHh7zg+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209201/ICMUAh7zg+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209201/ICMGGh7zg+001236
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209201/ICMSHh7zg+001236
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209201/ICMUAh7zg+001236
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a1v2_2016092300_15_991_12209201_0_r1635881709_12.zip
Uploading the intermediate file: upload_file_12.zip
  21:54:50 STEP 1249 H= 312:15 +CPU= 31.180
  21:55:20 STEP 1250 H= 312:30 +CPU= 30.494
  21:55:51 STEP 1251 H= 312:45 +CPU= 30.694
  21:56:22 STEP 1252 H= 313:00 +CPU= 31.308
  21:56:53 STEP 1253 H= 313:15 +CPU= 30.800
  21:57:24 STEP 1254 H= 313:30 +CPU= 30.773
  21:57:55 STEP 1255 H= 313:45 +CPU= 30.725
  21:58:26 STEP 1256 H= 314:00 +CPU= 31.146
  21:58:56 STEP 1257 H= 314:15 +CPU= 30.538
  21:59:27 STEP 1258 H= 314:30 +CPU= 30.451
  21:59:58 STEP 1259 H= 314:45 +CPU= 31.254
  22:00:36 STEP 1260 H= 315:00 +CPU= 37.570
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001260
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001260
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001260
  22:01:06 STEP 1261 H= 315:15 +CPU= 30.655
  22:01:37 STEP 1262 H= 315:30 +CPU= 30.939
  22:02:09 STEP 1263 H= 315:45 +CPU= 31.159
  22:02:40 STEP 1264 H= 316:00 +CPU= 31.593
  22:03:11 STEP 1265 H= 316:15 +CPU= 30.884
  22:03:42 STEP 1266 H= 316:30 +CPU= 31.006
  22:04:14 STEP 1267 H= 316:45 +CPU= 31.404
  22:04:45 STEP 1268 H= 317:00 +CPU= 31.850
  22:05:17 STEP 1269 H= 317:15 +CPU= 31.851
  22:05:48 STEP 1270 H= 317:30 +CPU= 30.539
  22:06:20 STEP 1271 H= 317:45 +CPU= 31.980
  22:06:59 STEP 1272 H= 318:00 +CPU= 39.552
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001272
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001272
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001272
  22:07:31 STEP 1273 H= 318:15 +CPU= 31.914
  22:08:03 STEP 1274 H= 318:30 +CPU= 31.844
  22:08:35 STEP 1275 H= 318:45 +CPU= 31.756
  22:09:07 STEP 1276 H= 319:00 +CPU= 32.004
  22:09:38 STEP 1277 H= 319:15 +CPU= 31.440
  22:10:10 STEP 1278 H= 319:30 +CPU= 31.656
  22:10:41 STEP 1279 H= 319:45 +CPU= 31.145
  22:11:13 STEP 1280 H= 320:00 +CPU= 31.548
  22:11:43 STEP 1281 H= 320:15 +CPU= 30.281
  22:12:15 STEP 1282 H= 320:30 +CPU= 31.832
  22:12:47 STEP 1283 H= 320:45 +CPU= 31.931
  22:13:25 STEP 1284 H= 321:00 +CPU= 38.270
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001284
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001284
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001284
  22:13:56 STEP 1285 H= 321:15 +CPU= 30.190
  22:14:26 STEP 1286 H= 321:30 +CPU= 30.556
  22:14:57 STEP 1287 H= 321:45 +CPU= 30.719
  22:15:28 STEP 1288 H= 322:00 +CPU= 30.712
  22:15:58 STEP 1289 H= 322:15 +CPU= 30.175
  22:16:28 STEP 1290 H= 322:30 +CPU= 30.420
  22:16:59 STEP 1291 H= 322:45 +CPU= 30.383
  22:17:31 STEP 1292 H= 323:00 +CPU= 31.922
  22:18:03 STEP 1293 H= 323:15 +CPU= 32.831
  22:18:36 STEP 1294 H= 323:30 +CPU= 32.992
  22:19:08 STEP 1295 H= 323:45 +CPU= 31.919
  22:19:53 STEP 1296 H= 324:00 +CPU= 44.925
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001296
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001296
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001296
  22:20:25 STEP 1297 H= 324:15 +CPU= 31.338
  22:20:56 STEP 1298 H= 324:30 +CPU= 31.404
  22:21:27 STEP 1299 H= 324:45 +CPU= 31.293
  22:22:00 STEP 1300 H= 325:00 +CPU= 32.345
  22:22:32 STEP 1301 H= 325:15 +CPU= 31.970
  22:23:04 STEP 1302 H= 325:30 +CPU= 32.149
  22:23:36 STEP 1303 H= 325:45 +CPU= 31.757
  22:24:08 STEP 1304 H= 326:00 +CPU= 31.801
  22:24:39 STEP 1305 H= 326:15 +CPU= 31.839
  22:25:11 STEP 1306 H= 326:30 +CPU= 31.771
  22:25:43 STEP 1307 H= 326:45 +CPU= 31.658
  22:26:22 STEP 1308 H= 327:00 +CPU= 38.760
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001308
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001308
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001308
  22:26:53 STEP 1309 H= 327:15 +CPU= 31.807
  22:27:25 STEP 1310 H= 327:30 +CPU= 31.328
  22:27:56 STEP 1311 H= 327:45 +CPU= 31.212
  22:28:27 STEP 1312 H= 328:00 +CPU= 31.427
  22:28:58 STEP 1313 H= 328:15 +CPU= 31.003
  22:29:29 STEP 1314 H= 328:30 +CPU= 30.708
  22:30:00 STEP 1315 H= 328:45 +CPU= 30.342
  22:30:30 STEP 1316 H= 329:00 +CPU= 30.752
  22:31:02 STEP 1317 H= 329:15 +CPU= 31.456
  22:31:33 STEP 1318 H= 329:30 +CPU= 31.270
  22:32:04 STEP 1319 H= 329:45 +CPU= 31.041
  22:32:42 STEP 1320 H= 330:00 +CPU= 37.633
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001320
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001320
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001320
  22:33:12 STEP 1321 H= 330:15 +CPU= 30.254
  22:33:42 STEP 1322 H= 330:30 +CPU= 30.196
  22:34:13 STEP 1323 H= 330:45 +CPU= 30.612
  22:34:44 STEP 1324 H= 331:00 +CPU= 30.930
  22:35:14 STEP 1325 H= 331:15 +CPU= 30.221
  22:35:45 STEP 1326 H= 331:30 +CPU= 30.700
  22:36:15 STEP 1327 H= 331:45 +CPU= 30.692
  22:36:46 STEP 1328 H= 332:00 +CPU= 30.967
  22:37:17 STEP 1329 H= 332:15 +CPU= 30.589
  22:37:47 STEP 1330 H= 332:30 +CPU= 30.143
  22:38:18 STEP 1331 H= 332:45 +CPU= 30.551
  22:38:55 STEP 1332 H= 333:00 +CPU= 37.079
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001332
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001332
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001332
  22:39:26 STEP 1333 H= 333:15 +CPU= 31.375
  22:39:57 STEP 1334 H= 333:30 +CPU= 31.339
  22:40:29 STEP 1335 H= 333:45 +CPU= 31.276
  22:41:01 STEP 1336 H= 334:00 +CPU= 31.823
  22:41:32 STEP 1337 H= 334:15 +CPU= 31.637
  22:42:04 STEP 1338 H= 334:30 +CPU= 31.361
  22:42:38 STEP 1339 H= 334:45 +CPU= 34.043
  22:43:16 STEP 1340 H= 335:00 +CPU= 38.408
  22:43:49 STEP 1341 H= 335:15 +CPU= 33.163
  22:44:21 STEP 1342 H= 335:30 +CPU= 31.724
  22:44:53 STEP 1343 H= 335:45 +CPU= 31.774
  22:45:36 STEP 1344 H= 336:00 +CPU= 43.069
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001344
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001344
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209201/ICMGGh7zg+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209201/ICMSHh7zg+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209201/ICMUAh7zg+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209201/ICMGGh7zg+001260
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209201/ICMSHh7zg+001260
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209201/ICMUAh7zg+001260
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209201/ICMGGh7zg+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209201/ICMSHh7zg+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209201/ICMUAh7zg+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209201/ICMGGh7zg+001284
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209201/ICMSHh7zg+001284
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209201/ICMUAh7zg+001284
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209201/ICMGGh7zg+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209201/ICMSHh7zg+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209201/ICMUAh7zg+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209201/ICMGGh7zg+001308
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209201/ICMSHh7zg+001308
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209201/ICMUAh7zg+001308
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209201/ICMGGh7zg+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209201/ICMSHh7zg+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209201/ICMUAh7zg+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209201/ICMGGh7zg+001332
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209201/ICMSHh7zg+001332
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209201/ICMUAh7zg+001332
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a1v2_2016092300_15_991_12209201_0_r1635881709_13.zip
Uploading the intermediate file: upload_file_13.zip
  22:46:08 STEP 1345 H= 336:15 +CPU= 32.380
  22:46:41 STEP 1346 H= 336:30 +CPU= 32.289
  22:47:12 STEP 1347 H= 336:45 +CPU= 31.176
  22:47:44 STEP 1348 H= 337:00 +CPU= 32.225
  22:48:16 STEP 1349 H= 337:15 +CPU= 31.572
  22:48:46 STEP 1350 H= 337:30 +CPU= 30.436
  22:49:16 STEP 1351 H= 337:45 +CPU= 30.508
  22:49:49 STEP 1352 H= 338:00 +CPU= 32.701
  22:50:21 STEP 1353 H= 338:15 +CPU= 32.165
  22:50:53 STEP 1354 H= 338:30 +CPU= 31.320
  22:51:24 STEP 1355 H= 338:45 +CPU= 31.351
  22:52:02 STEP 1356 H= 339:00 +CPU= 38.246
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001356
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001356
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001356
  22:52:33 STEP 1357 H= 339:15 +CPU= 31.151
  22:53:04 STEP 1358 H= 339:30 +CPU= 30.717
  22:53:35 STEP 1359 H= 339:45 +CPU= 30.961
  22:54:07 STEP 1360 H= 340:00 +CPU= 31.386
  22:54:37 STEP 1361 H= 340:15 +CPU= 30.793
  22:55:08 STEP 1362 H= 340:30 +CPU= 30.939
  22:55:39 STEP 1363 H= 340:45 +CPU= 31.159
  22:56:11 STEP 1364 H= 341:00 +CPU= 31.698
  22:56:42 STEP 1365 H= 341:15 +CPU= 31.103
  22:57:13 STEP 1366 H= 341:30 +CPU= 30.295
  22:57:43 STEP 1367 H= 341:45 +CPU= 30.434
  22:58:21 STEP 1368 H= 342:00 +CPU= 37.530
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001368
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001368
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001368
  22:58:52 STEP 1369 H= 342:15 +CPU= 30.981
  22:59:23 STEP 1370 H= 342:30 +CPU= 31.242
  22:59:54 STEP 1371 H= 342:45 +CPU= 31.317
  23:00:26 STEP 1372 H= 343:00 +CPU= 31.549
  23:00:57 STEP 1373 H= 343:15 +CPU= 31.672
  23:01:28 STEP 1374 H= 343:30 +CPU= 30.944
  23:01:59 STEP 1375 H= 343:45 +CPU= 30.695
  23:02:31 STEP 1376 H= 344:00 +CPU= 31.664
  23:03:02 STEP 1377 H= 344:15 +CPU= 31.170
  23:03:33 STEP 1378 H= 344:30 +CPU= 31.264
  23:04:04 STEP 1379 H= 344:45 +CPU= 30.611
  23:04:42 STEP 1380 H= 345:00 +CPU= 37.945
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001380
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001380
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001380
  23:05:13 STEP 1381 H= 345:15 +CPU= 31.432
  23:05:45 STEP 1382 H= 345:30 +CPU= 31.372
  23:06:16 STEP 1383 H= 345:45 +CPU= 31.398
  23:06:48 STEP 1384 H= 346:00 +CPU= 32.031
  23:07:20 STEP 1385 H= 346:15 +CPU= 31.553
  23:07:55 STEP 1386 H= 346:30 +CPU= 35.035
  23:08:28 STEP 1387 H= 346:45 +CPU= 33.746
  23:09:01 STEP 1388 H= 347:00 +CPU= 32.805
  23:09:34 STEP 1389 H= 347:15 +CPU= 32.455
  23:10:06 STEP 1390 H= 347:30 +CPU= 32.115
  23:10:38 STEP 1391 H= 347:45 +CPU= 31.894
  23:11:21 STEP 1392 H= 348:00 +CPU= 43.090
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001392
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001392
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001392
  23:11:52 STEP 1393 H= 348:15 +CPU= 31.110
  23:12:23 STEP 1394 H= 348:30 +CPU= 30.973
  23:12:53 STEP 1395 H= 348:45 +CPU= 30.585
  23:13:24 STEP 1396 H= 349:00 +CPU= 30.456
  23:13:54 STEP 1397 H= 349:15 +CPU= 30.332
  23:14:25 STEP 1398 H= 349:30 +CPU= 30.475
  23:14:55 STEP 1399 H= 349:45 +CPU= 30.536
  23:15:26 STEP 1400 H= 350:00 +CPU= 31.139
  23:15:57 STEP 1401 H= 350:15 +CPU= 30.246
  23:16:27 STEP 1402 H= 350:30 +CPU= 30.172
  23:16:57 STEP 1403 H= 350:45 +CPU= 30.450
  23:17:34 STEP 1404 H= 351:00 +CPU= 37.105
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001404
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001404
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001404
  23:18:05 STEP 1405 H= 351:15 +CPU= 30.825
  23:18:36 STEP 1406 H= 351:30 +CPU= 30.770
  23:19:07 STEP 1407 H= 351:45 +CPU= 30.782
  23:19:39 STEP 1408 H= 352:00 +CPU= 31.792
  23:20:10 STEP 1409 H= 352:15 +CPU= 31.198
  23:20:41 STEP 1410 H= 352:30 +CPU= 30.934
  23:21:12 STEP 1411 H= 352:45 +CPU= 30.907
  23:21:43 STEP 1412 H= 353:00 +CPU= 31.028
  23:22:14 STEP 1413 H= 353:15 +CPU= 30.914
  23:22:44 STEP 1414 H= 353:30 +CPU= 30.672
  23:23:16 STEP 1415 H= 353:45 +CPU= 31.397
  23:23:54 STEP 1416 H= 354:00 +CPU= 38.599
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001416
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001416
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001416
  23:24:26 STEP 1417 H= 354:15 +CPU= 31.589
  23:24:57 STEP 1418 H= 354:30 +CPU= 31.195
  23:25:28 STEP 1419 H= 354:45 +CPU= 31.363
  23:26:01 STEP 1420 H= 355:00 +CPU= 32.348
  23:26:32 STEP 1421 H= 355:15 +CPU= 30.985
  23:27:03 STEP 1422 H= 355:30 +CPU= 31.529
  23:27:35 STEP 1423 H= 355:45 +CPU= 31.544
  23:28:07 STEP 1424 H= 356:00 +CPU= 31.972
  23:28:38 STEP 1425 H= 356:15 +CPU= 31.152
  23:29:09 STEP 1426 H= 356:30 +CPU= 31.282
  23:29:41 STEP 1427 H= 356:45 +CPU= 31.355
  23:30:18 STEP 1428 H= 357:00 +CPU= 37.579
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001428
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001428
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001428
  23:30:49 STEP 1429 H= 357:15 +CPU= 30.448
  23:31:19 STEP 1430 H= 357:30 +CPU= 30.549
  23:31:50 STEP 1431 H= 357:45 +CPU= 30.630
  23:32:22 STEP 1432 H= 358:00 +CPU= 32.076
  23:32:58 STEP 1433 H= 358:15 +CPU= 36.188
  23:33:34 STEP 1434 H= 358:30 +CPU= 35.614
  23:34:04 STEP 1435 H= 358:45 +CPU= 30.606
  23:34:35 STEP 1436 H= 359:00 +CPU= 30.413
  23:35:04 STEP 1437 H= 359:15 +CPU= 29.300
  23:35:34 STEP 1438 H= 359:30 +CPU= 29.677
  23:36:04 STEP 1439 H= 359:45 +CPU= 30.191
  23:36:42 STEP 1440 H= 360:00 +CPU= 37.798
..The child process terminated with status: 0
>>> Printing last 70 lines from file: NODE.001_01

 
DDH------------------------------------------------------------------------
I                                                                         I
I TIME =  360.000H DDH * Diagnostic Var.,     AREA                GLOBAL  I
I                                                                         I
I-------------------------------------------------------------------------I
I   VARIABLE   I      Initial value         I        Current value        I
I-------------------------------------------------------------------------I
I  OMEGA (P/S) I      0.8862346914E-21      I       0.9725958197E-06      I
ICloud fractionI      0.0000000000E+00      I       0.5013057618E-01      I
I  Relat. Hum. I      0.3830746790E+00      I       0.4471523918E+00      I
I  PASS 01     I      0.0000000000E+00      I       0.4489275557E+03      I
DDH------------------------------------------------------------------------
  
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1440 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001440 MODE=a
 IO-STREAM CLOSED - ICMGGh7zg+001440
 MPI-TASK:   1 -    311621761 BYTES IN      7 RECORDS TRANSFERRED IN    0.0006 SECONDS519369.6 Mbytes/s, TOTAL TIME=    0.0095( 6.3%)
 23:36:42 STEP 1440 H= 360:00 +CPU= 37.798
 END CNT3
 NSTEP =  1440 CNT0   000000000  28.670  28.670  28.674   1.000               0               0               0               0               0
IO-STREAM STATISTICS, TOTAL NO OF REC/BYTES READ           34/  1069125708 WRITTEN         8508/************
 -TASK-OPENED-OPEN-RECS IN -KBYTE IN    -RECS OUT-KBYTE OUT-WALL   -WALL IN-WALL
  OU-TOT IN -TOT OUT
   1    153    0       34      1044068     8508 ************    31.3    14.1    17.2    15.4    35.7
===-=== START OF TIMING STATISTICS ===-===
 
STATS FOR ALL TASKS
 NUM ROUTINE                                     CALLS  MEAN(ms)   MAX(ms)   FRAC(%)  UNBAL(%)
   0 CNT0     - COMPLETE EXECUTION                   1 9250039.1 9250039.1    100.00      0.00
   1 CNT4     - FORWARD INTEGRATION                  1 9232423.9 9232423.9     99.81      0.00
   8 SCAN2M - GRID-POINT DYNAMICS                  313    6243.4    6243.4     21.13      0.00
   9 SPCM     - SPECTRAL COMP.                     288     886.7     886.7      2.76      0.00
  10 SCAN2M - PHYSICS                              289   14327.3   14327.3     44.76      0.00
  11 IOPACK   - OUTPUT P.P. RESULTS                 24    1078.8    1078.8      0.28      0.00
  12 SPNORM   - SPECTRAL NORM COMP.                 13      66.2      66.2      0.01      0.00
  17 GRIDFPOS IN CNT4                               24      30.9      30.9      0.01      0.00
  27 RESTART FILES - WRITING                         6    4315.1    4315.1      0.28      0.00
  28 RESTART FILES - READING                         1   16649.6   16649.6      0.18      0.00
  29 SU4FPOS IN CNT4                                24       0.1       0.1      0.00      0.00
  30 DYNFPOS IN CNT4                                24    6234.1    6234.1      1.62      0.00
  31 POSDDH IN STEPO                                 2       1.4       1.4      0.00      0.00
  37 CPGLAG   - SL COMPUTATIONS                    289    6440.6    6440.6     20.12      0.00
  39 SU0YOMB                                         1     700.5     700.5      0.01      0.00
  51 SCAN2M   - SL COMM. PART 1                    289      39.4      39.4      0.12      0.00
  54 SPCM     - M TO S/S TO M TRANSP.              288     327.3     327.3      1.02      0.00
  55 SPCIMPF  - S TO M/M TO S TRANSP.              288      71.3      71.3      0.22      0.00
  56 SPNORM   - SPECTRAL NORM COMM.                 13       0.2       0.2      0.00      0.00
 102 LTINV_CTL   - INVERSE LEGENDRE TRANSFORM      628     480.1     480.1      3.26      0.00
 103 LTDIR_CTL   - DIRECT LEGENDRE TRANSFORM       600     286.3     286.3      1.86      0.00
 106 FTDIR_CTL   - DIRECT FOURIER TRANSFORM        600      88.2      88.2      0.57      0.00
 107 FTINV_CTL   - INVERSE FOURIER TRANSFORM       628     138.4     138.4      0.94      0.00
 140 SULEG       - COMP. OF LEGENDRE POL.            1      69.5      69.5      0.00      0.00
 152 LTINV_CTL   - M TO L TRANSPOSITION            628      66.6      66.6      0.45      0.00
 153 LTDIR_CTL   - L TO M TRANSPOSITION            600      35.7      35.7      0.23      0.00
 157 FTINV_CTL   - L TO G TRANSPOSITION            628      60.1      60.1      0.41      0.00
 158 FTDIR_CTL   - G TO L TRANSPOSITION            600      35.4      35.4      0.23      0.00
 400 GSTATS                                      34304       0.0       0.0      0.00      0.00
 401 GSTATS HOOK                                 31868       0.0       0.0      0.00      0.00
TOTAL MEASURED IMBALANCE =       0.0 SECONDS,  0.0 PERCENT
TOTAL WALLCLOCK TIME     9250.0 CPU TIME    9232.5 VECTOR TIME     9232.5


===-=== END   OF TIMING STATISTICS ===-===
FORECAST DAYS PER DAY      140.4
  *** END CNT0 *** 
------------------------------------------------
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001440
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001440
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001440
Adding to the zip: /var/lib/boinc-client/slots/4/NODE.001_01
Adding to the zip: /var/lib/boinc-client/slots/4/ifs.stat
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209201/ICMSHh7zg+001428
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209201/ICMUAh7zg+001428
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209201/ICMSHh7zg+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209201/ICMUAh7zg+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209201/ICMGGh7zg+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209201/ICMGGh7zg+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209201/ICMUAh7zg+001356
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209201/ICMUAh7zg+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209201/ICMGGh7zg+001404
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209201/ICMSHh7zg+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209201/ICMGGh7zg+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209201/ICMGGh7zg+001428
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209201/ICMUAh7zg+001404
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209201/ICMGGh7zg+001356
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209201/ICMSHh7zg+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209201/ICMSHh7zg+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209201/ICMGGh7zg+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209201/ICMSHh7zg+001404
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209201/ICMGGh7zg+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209201/ICMSHh7zg+001356
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209201/ICMUAh7zg+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209201/ICMSHh7zg+001380
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209201/ICMGGh7zg+001380
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209201/ICMUAh7zg+001380
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209201/ICMUAh7zg+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209201/ICMSHh7zg+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209201/ICMUAh7zg+001440
Zipping up the final file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a1v2_2016092300_15_991_12209201_0_r1635881709_14.zip
Uploading the final file: upload_file_14.zip
Uploading trickle at timestep: 1295100
23:40:14 (1657): called boinc_finish(0)

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:51:26 1536331 22309391 oifs_43r3_bl_a1v2_2016092300_15_991_12209201_0 1,295,100 45,649 0.0352
09 May 2023 16:51:21 1536331 22309391 oifs_43r3_bl_a1v2_2016092300_15_991_12209201_0 864,000 27,989 0.0324


©2024 climateprediction.net