climateprediction.net home page
Task 22309383

Task 22309383

Name oifs_43r3_bl_a1uu_2016092300_15_991_12209193_0
Workunit 12209193
Created 13 Feb 2023, 16:26:28 UTC
Sent 13 Feb 2023, 16:28:15 UTC
Report deadline 14 Apr 2023, 16:28:15 UTC
Received 14 Feb 2023, 23:14:50 UTC
Server state Over
Outcome Success
Client state Done
Exit status 0 (0x00000000)
Computer ID 1536331
Run time 12 hours 37 min
CPU time 12 hours 37 min
Validate state Valid
Credit 1,813.14
Device peak FLOPS 6.13 GFLOPS
Application version OpenIFS 43r3 Baroclinic Lifecycle v1.11
x86_64-pc-linux-gnu
Peak working set size 5,075.95 MB
Peak swap size 5,930.79 MB
Peak disk usage 1,684.44 MB
Stderr
<core_client_version>7.18.1</core_client_version>
<![CDATA[
<stderr_txt>
] [20230214:192724:1676399244:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f520 (old at (nil))
[EC_DRHOOK:W32Ub22:1:1:1778:1778] [20230214:192724:1676399244:0.000] [trapfpe_treatment@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:W32Ub22:1:1:1778:1778] [20230214:192724:1676399244:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:W32Ub22:1:1:1778:1778] [20230214:192724:1676399244:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:W32Ub22:1:1:1778:1778] [20230214:192724:1676399244:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f520 (old at (nil))
[EC_DRHOOK:W32Ub22:1:1:1778:1778] [20230214:192724:1676399244:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:W32Ub22:1:1:1778:1778] [20230214:192724:1676399244:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:W32Ub22:1:1:1778:1778] [20230214:192724:1676399244:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:W32Ub22:1:1:1778:1778] [20230214:192724:1676399244:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f520 (old at (nil))
[EC_DRHOOK:W32Ub22:1:1:1778:1778] [20230214:192724:1676399244:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f520 (old at (nil))
[EC_DRHOOK:W32Ub22:1:1:1778:1778] [20230214:192724:1676399244:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f520 (old at (nil))
[EC_DRHOOK:W32Ub22:1:1:1778:1778] [20230214:192724:1676399244:0.000] [catch_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  19:28:44 STEP 1056 H= 264:00 +CPU= 61.150
  19:29:26 STEP 1057 H= 264:15 +CPU= 42.206
  19:30:06 STEP 1058 H= 264:30 +CPU= 39.623
  19:30:45 STEP 1059 H= 264:45 +CPU= 38.581
  19:31:24 STEP 1060 H= 265:00 +CPU= 38.974
  19:32:04 STEP 1061 H= 265:15 +CPU= 40.461
  19:32:44 STEP 1062 H= 265:30 +CPU= 39.591
  19:33:24 STEP 1063 H= 265:45 +CPU= 40.557
  19:34:05 STEP 1064 H= 266:00 +CPU= 41.144
  19:34:46 STEP 1065 H= 266:15 +CPU= 40.684
  19:35:26 STEP 1066 H= 266:30 +CPU= 39.976
  19:36:07 STEP 1067 H= 266:45 +CPU= 40.532
  19:36:58 STEP 1068 H= 267:00 +CPU= 51.342
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGh7zg+001068
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHh7zg+001068
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAh7zg+001068
  19:37:39 STEP 1069 H= 267:15 +CPU= 40.969
  19:38:20 STEP 1070 H= 267:30 +CPU= 40.357
  19:38:59 STEP 1071 H= 267:45 +CPU= 38.954
  19:39:41 STEP 1072 H= 268:00 +CPU= 42.374
  19:40:23 STEP 1073 H= 268:15 +CPU= 41.773
  19:41:05 STEP 1074 H= 268:30 +CPU= 41.868
  19:41:46 STEP 1075 H= 268:45 +CPU= 41.121
  19:42:28 STEP 1076 H= 269:00 +CPU= 41.704
  19:43:08 STEP 1077 H= 269:15 +CPU= 40.551
  19:43:48 STEP 1078 H= 269:30 +CPU= 40.259
  19:44:29 STEP 1079 H= 269:45 +CPU= 41.031
  19:45:19 STEP 1080 H= 270:00 +CPU= 49.582
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGh7zg+001080
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHh7zg+001080
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAh7zg+001080
  19:46:01 STEP 1081 H= 270:15 +CPU= 41.683
  19:46:43 STEP 1082 H= 270:30 +CPU= 41.893
  19:47:24 STEP 1083 H= 270:45 +CPU= 41.250
  19:48:06 STEP 1084 H= 271:00 +CPU= 41.842
  19:48:48 STEP 1085 H= 271:15 +CPU= 41.705
  19:49:28 STEP 1086 H= 271:30 +CPU= 40.737
  19:50:07 STEP 1087 H= 271:45 +CPU= 38.903
  19:50:48 STEP 1088 H= 272:00 +CPU= 40.872
  19:51:29 STEP 1089 H= 272:15 +CPU= 40.517
  19:52:10 STEP 1090 H= 272:30 +CPU= 41.181
  19:52:50 STEP 1091 H= 272:45 +CPU= 40.081
  19:53:40 STEP 1092 H= 273:00 +CPU= 49.424
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGh7zg+001092
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHh7zg+001092
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAh7zg+001092
  19:54:20 STEP 1093 H= 273:15 +CPU= 39.962
  19:54:58 STEP 1094 H= 273:30 +CPU= 38.541
  19:55:38 STEP 1095 H= 273:45 +CPU= 39.792
  19:56:19 STEP 1096 H= 274:00 +CPU= 40.865
  19:56:58 STEP 1097 H= 274:15 +CPU= 39.423
  19:57:38 STEP 1098 H= 274:30 +CPU= 39.769
  19:58:19 STEP 1099 H= 274:45 +CPU= 40.514
  19:58:59 STEP 1100 H= 275:00 +CPU= 39.974
  19:59:40 STEP 1101 H= 275:15 +CPU= 41.357
  20:00:21 STEP 1102 H= 275:30 +CPU= 40.838
  20:01:01 STEP 1103 H= 275:45 +CPU= 39.780
  20:02:10 STEP 1104 H= 276:00 +CPU= 69.452
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGh7zg+001104
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHh7zg+001104
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAh7zg+001104
  20:02:57 STEP 1105 H= 276:15 +CPU= 46.858
  20:03:39 STEP 1106 H= 276:30 +CPU= 41.653
  20:04:20 STEP 1107 H= 276:45 +CPU= 41.142
  20:05:03 STEP 1108 H= 277:00 +CPU= 42.764
  20:05:45 STEP 1109 H= 277:15 +CPU= 42.494
  20:06:27 STEP 1110 H= 277:30 +CPU= 41.738
  20:07:10 STEP 1111 H= 277:45 +CPU= 43.130
  20:07:53 STEP 1112 H= 278:00 +CPU= 43.051
  20:08:33 STEP 1113 H= 278:15 +CPU= 39.800
  20:09:15 STEP 1114 H= 278:30 +CPU= 42.134
  20:09:57 STEP 1115 H= 278:45 +CPU= 41.599
  20:10:45 STEP 1116 H= 279:00 +CPU= 48.241
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGh7zg+001116
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHh7zg+001116
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAh7zg+001116
  20:11:26 STEP 1117 H= 279:15 +CPU= 40.805
  20:12:07 STEP 1118 H= 279:30 +CPU= 41.015
  20:12:47 STEP 1119 H= 279:45 +CPU= 40.387
  20:13:28 STEP 1120 H= 280:00 +CPU= 40.745
  20:14:08 STEP 1121 H= 280:15 +CPU= 39.809
  20:14:51 STEP 1122 H= 280:30 +CPU= 43.363
  20:15:31 STEP 1123 H= 280:45 +CPU= 39.950
  20:16:12 STEP 1124 H= 281:00 +CPU= 40.827
  20:16:54 STEP 1125 H= 281:15 +CPU= 41.871
  20:17:34 STEP 1126 H= 281:30 +CPU= 39.781
  20:18:14 STEP 1127 H= 281:45 +CPU= 39.913
  20:19:03 STEP 1128 H= 282:00 +CPU= 49.724
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGh7zg+001128
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHh7zg+001128
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAh7zg+001128
  20:19:45 STEP 1129 H= 282:15 +CPU= 41.230
  20:20:25 STEP 1130 H= 282:30 +CPU= 40.261
  20:21:07 STEP 1131 H= 282:45 +CPU= 41.742
  20:21:48 STEP 1132 H= 283:00 +CPU= 41.219
  20:22:29 STEP 1133 H= 283:15 +CPU= 41.314
  20:23:11 STEP 1134 H= 283:30 +CPU= 41.680
  20:23:53 STEP 1135 H= 283:45 +CPU= 42.400
  20:24:35 STEP 1136 H= 284:00 +CPU= 41.421
  20:25:15 STEP 1137 H= 284:15 +CPU= 39.851
  20:25:54 STEP 1138 H= 284:30 +CPU= 38.976
  20:26:33 STEP 1139 H= 284:45 +CPU= 39.562
  20:27:22 STEP 1140 H= 285:00 +CPU= 49.304
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGh7zg+001140
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHh7zg+001140
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAh7zg+001140
  20:28:04 STEP 1141 H= 285:15 +CPU= 41.019
  20:28:43 STEP 1142 H= 285:30 +CPU= 39.891
  20:29:25 STEP 1143 H= 285:45 +CPU= 41.485
  20:30:06 STEP 1144 H= 286:00 +CPU= 41.029
  20:30:46 STEP 1145 H= 286:15 +CPU= 40.245
  20:31:27 STEP 1146 H= 286:30 +CPU= 40.909
  20:32:08 STEP 1147 H= 286:45 +CPU= 40.534
  20:32:49 STEP 1148 H= 287:00 +CPU= 41.430
  20:33:30 STEP 1149 H= 287:15 +CPU= 40.934
  20:34:10 STEP 1150 H= 287:30 +CPU= 40.323
  20:34:50 STEP 1151 H= 287:45 +CPU= 39.586
  20:36:00 STEP 1152 H= 288:00 +CPU= 70.241
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGh7zg+001152
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHh7zg+001152
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAh7zg+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209193/ICMGGh7zg+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209193/ICMSHh7zg+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209193/ICMUAh7zg+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209193/ICMGGh7zg+001068
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209193/ICMSHh7zg+001068
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209193/ICMUAh7zg+001068
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209193/ICMGGh7zg+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209193/ICMSHh7zg+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209193/ICMUAh7zg+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209193/ICMGGh7zg+001092
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209193/ICMSHh7zg+001092
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209193/ICMUAh7zg+001092
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209193/ICMGGh7zg+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209193/ICMSHh7zg+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209193/ICMUAh7zg+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209193/ICMGGh7zg+001116
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209193/ICMSHh7zg+001116
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209193/ICMUAh7zg+001116
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209193/ICMGGh7zg+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209193/ICMSHh7zg+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209193/ICMUAh7zg+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209193/ICMGGh7zg+001140
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209193/ICMSHh7zg+001140
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209193/ICMUAh7zg+001140
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a1uu_2016092300_15_991_12209193_0_r1235926258_11.zip
Uploading the intermediate file: upload_file_11.zip
  20:36:53 STEP 1153 H= 288:15 +CPU= 52.551
  20:37:35 STEP 1154 H= 288:30 +CPU= 41.658
  20:38:17 STEP 1155 H= 288:45 +CPU= 42.211
  20:38:59 STEP 1156 H= 289:00 +CPU= 41.724
  20:39:40 STEP 1157 H= 289:15 +CPU= 41.474
  20:40:20 STEP 1158 H= 289:30 +CPU= 40.430
  20:41:03 STEP 1159 H= 289:45 +CPU= 42.080
  20:41:45 STEP 1160 H= 290:00 +CPU= 42.394
  20:42:26 STEP 1161 H= 290:15 +CPU= 40.812
  20:43:06 STEP 1162 H= 290:30 +CPU= 40.414
  20:43:48 STEP 1163 H= 290:45 +CPU= 41.963
  20:44:38 STEP 1164 H= 291:00 +CPU= 49.626
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGh7zg+001164
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHh7zg+001164
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAh7zg+001164
  20:45:20 STEP 1165 H= 291:15 +CPU= 42.464
  20:46:03 STEP 1166 H= 291:30 +CPU= 42.464
  20:46:44 STEP 1167 H= 291:45 +CPU= 40.922
  20:47:28 STEP 1168 H= 292:00 +CPU= 44.616
  20:48:11 STEP 1169 H= 292:15 +CPU= 42.947
  20:48:53 STEP 1170 H= 292:30 +CPU= 42.060
  20:49:33 STEP 1171 H= 292:45 +CPU= 39.408
  20:50:12 STEP 1172 H= 293:00 +CPU= 39.420
  20:50:53 STEP 1173 H= 293:15 +CPU= 40.996
  20:51:34 STEP 1174 H= 293:30 +CPU= 40.384
  20:52:17 STEP 1175 H= 293:45 +CPU= 43.672
  20:53:07 STEP 1176 H= 294:00 +CPU= 49.347
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGh7zg+001176
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHh7zg+001176
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAh7zg+001176
  20:53:46 STEP 1177 H= 294:15 +CPU= 39.438
  20:54:25 STEP 1178 H= 294:30 +CPU= 38.854
  20:55:05 STEP 1179 H= 294:45 +CPU= 39.805
  20:55:45 STEP 1180 H= 295:00 +CPU= 40.211
  20:56:24 STEP 1181 H= 295:15 +CPU= 38.345
  20:57:02 STEP 1182 H= 295:30 +CPU= 38.682
  20:57:42 STEP 1183 H= 295:45 +CPU= 39.543
  20:58:21 STEP 1184 H= 296:00 +CPU= 39.342
  20:59:04 STEP 1185 H= 296:15 +CPU= 42.372
  20:59:44 STEP 1186 H= 296:30 +CPU= 39.961
  21:00:23 STEP 1187 H= 296:45 +CPU= 39.098
[EC_DRHOOK:W32Ub22:1:1:1778:1778] [20230214:210100:1676404860:5616.117] [signal_drhook@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 4057MB (heap), 4897MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:W32Ub22:1:1:1778:1778] [20230214:210100:1676404860:5616.117] [signal_drhook@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:W32Ub22:1:1:1778:1778] [20230214:210100:1676404860:5616.117] [signal_drhook@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f0c0 (old at (nil))
[EC_DRHOOK:W32Ub22:1:1:1778:1778] [20230214:210100:1676404860:5616.117] [signal_drhook@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1
[EC_DRHOOK:W32Ub22:1:1:1778:1778] [20230214:210100:1676404860:5616.117] [signal_drhook@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1
[EC_DRHOOK:W32Ub22:1:1:1778:1778] [20230214:210100:1676404860:5616.117] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3843] 4057 MB (maxheap), 4897 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:W32Ub22:1:1:1778:1778] [20230214:210100:1676404860:5616.117] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:W32Ub22:1:1:1778:1778] [20230214:210100:1676404860:5616.117] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:W32Ub22:1:1:1778:1778] [20230214:210100:1676404860:5616.117] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:W32Ub22:1:1:1778:1778] [20230214:210100:1676404860:5616.117] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:W32Ub22:1:1:1778:1778] [20230214:210100:1676404860:5616.117] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:W32Ub22:1:1:1778:1778] [20230214:210100:1676404860:5616.117] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] :       CNT4 
[EC_DRHOOK:W32Ub22:1:1:1778:1778] [20230214:210100:1676404860:5616.117] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] :        STEPO 
[EC_DRHOOK:W32Ub22:1:1:1778:1778] [20230214:210100:1676404860:5616.117] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] :         SCAN2M 
[EC_DRHOOK:W32Ub22:1:1:1778:1778] [20230214:210100:1676404860:5616.117] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] :          GP_MODEL_HEAP 
[EC_DRHOOK:W32Ub22:1:1:1778:1778] [20230214:210100:1676404860:5616.117] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] :           GP_MODEL 
[EC_DRHOOK:W32Ub22:1:1:1778:1778] [20230214:210100:1676404860:5616.117] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] :            EC_PHYS_DRV 
[EC_DRHOOK:W32Ub22:1:1:1778:1778] [20230214:210100:1676404860:5616.117] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] :             >OMP-PHYSICS CLDPP T/S    (1002) 
[EC_DRHOOK:W32Ub22:1:1:1778:1778] [20230214:210100:1676404860:5616.117] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] :              EC_PHYS 
[EC_DRHOOK:W32Ub22:1:1:1778:1778] [20230214:210100:1676404860:5616.117] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] :               CALLPAR 
[EC_DRHOOK:W32Ub22:1:1:1778:1778] [20230214:210100:1676404860:5616.117] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] :                CLOUD_LAYER 
[EC_DRHOOK:W32Ub22:1:1:1778:1778] [20230214:210100:1676404860:5616.117] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] :                 CLOUDSC 
[EC_DRHOOK:W32Ub22:1:1:1778:1778] [20230214:210101:1676404861:5617.121] [signal_drhook@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1
[EC_DRHOOK:W32Ub22:1:1:1778:1778] [20230214:210101:1676404861:5617.121] [signal_drhook@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x401140 for signal#15, nsigs = 1
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source             
oifs_43r3_model.e  000000000081FCF1  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001D8BB20  Unknown               Unknown  Unknown
oifs_43r3_model.e  00000000011DDDFA  cloudsc_                  747  cloudsc.F90
oifs_43r3_model.e  00000000011D9EE7  cloud_layer_              131  cloud_layer.F90
oifs_43r3_model.e  000000000116955B  callpar_                 1011  callpar.F90
oifs_43r3_model.e  00000000010B682F  ec_phys_                  670  ec_phys.F90
oifs_43r3_model.e  00000000010A915C  ec_phys_drv_              599  ec_phys_drv.F90
oifs_43r3_model.e  0000000000E680C3  gp_model_                 613  gp_model.F90
oifs_43r3_model.e  00000000015C1E62  gp_model_heap_             74  gp_model_heap.F90
oifs_43r3_model.e  00000000015AC40E  scan2m_                   535  scan2m.F90
oifs_43r3_model.e  00000000006620B4  stepo_                    327  stepo.F90
oifs_43r3_model.e  00000000004A0E7C  cnt4_                    1133  cnt4.F90
oifs_43r3_model.e  000000000049DC29  cnt3_                     267  cnt3.F90
oifs_43r3_model.e  000000000049CE72  cnt2_                      88  cnt2.F90
oifs_43r3_model.e  000000000049CD68  cnt1_                      92  cnt1.F90
oifs_43r3_model.e  00000000004078AD  cnt0_                     146  cnt0.F90
oifs_43r3_model.e  000000000040276F  MAIN__                     96  master.F90
oifs_43r3_model.e  000000000040270D  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001D8287A  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001D840D7  Unknown               Unknown  Unknown
oifs_43r3_model.e  00000000004025E5  Unknown               Unknown  Unknown
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.11_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a1uu
(argv4) batchid: 991
(argv5) wuid: 12209193
(argv6) fclen: 15
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/6
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.11
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209193
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/6/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/6/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_94d4ebbd31e5256b8debbf882a9b4450 to: /var/lib/boinc-client/slots/6/oifs_43r3_bl_a1uu_2016092300_15_991_12209193.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/6/oifs_43r3_bl_a1uu_2016092300_15_991_12209193.zip
ic_ancil_file: ic_ancil_12209193
ifsdata_file: ifsdata_12209193
climate_data_file: clim_data_12209193
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /var/lib/boinc-client/slots/6/ic_ancil_12209193.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/6/ic_ancil_12209193.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_b271e86f742965756d76f271e62c090a to: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12209193.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12209193.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/6/159l_2/clim_data_12209193.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/6/159l_2/clim_data_12209193.zip
Checking for progress XML file: /var/lib/boinc-client/slots/6/progress_file_12209193.xml
Opened progress file ok : /var/lib/boinc-client/slots/6/progress_file_12209193.xml
-- Model is restarting --
Adjusting last_iter, 1188, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/6/progress_file_12209193.xml
last_cpu_time: 36415
upload_file_number: 12
last_iter: 1153
last_upload: 1036800
model_completed: 0
total_length_of_simulation: 1296000
result_base_name: oifs_43r3_bl_a1uu_2016092300_15_991_12209193_0_r1235926258
The child process has been launched with process id: 1791
Executing the command: /var/lib/boinc-client/slots/6/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:W32Ub22:1:1:1791:1791] [20230214:210232:1676404952:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2091] fp = 0x30e4a00
[EC_DRHOOK:W32Ub22:1:1:1791:1791] [20230214:210232:1676404952:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:W32Ub22:1:1:1791:1791] [20230214:210232:1676404952:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:W32Ub22:1:1:1791:1791] [20230214:210232:1676404952:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:W32Ub22:1:1:1791:1791] [20230214:210232:1676404952:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:W32Ub22:1:1:1791:1791] [20230214:210232:1676404952:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:W32Ub22:1:1:1791:1791] [20230214:210232:1676404952:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:W32Ub22:1:1:1791:1791] [20230214:210232:1676404952:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:W32Ub22:1:1:1791:1791] [20230214:210232:1676404952:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:W32Ub22:1:1:1791:1791] [20230214:210232:1676404952:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:W32Ub22:1:1:1791:1791] [20230214:210232:1676404952:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:W32Ub22:1:1:1791:1791] [20230214:210232:1676404952:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:W32Ub22:1:1:1791:1791] [20230214:210232:1676404952:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:W32Ub22:1:1:1791:1791] [20230214:210232:1676404952:0.000] [ignore_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:W32Ub22:1:1:1791:1791] [20230214:210232:1676404952:0.000] [restore_default_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:W32Ub22:1:1:1791:1791] [20230214:210232:1676404952:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:W32Ub22:1:1:1791:1791] [20230214:210232:1676404952:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f520 (old at (nil))
[EC_DRHOOK:W32Ub22:1:1:1791:1791] [20230214:210232:1676404952:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:W32Ub22:1:1:1791:1791] [20230214:210232:1676404952:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f520 (old at (nil))
[EC_DRHOOK:W32Ub22:1:1:1791:1791] [20230214:210232:1676404952:0.000] [trapfpe_treatment@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:W32Ub22:1:1:1791:1791] [20230214:210232:1676404952:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:W32Ub22:1:1:1791:1791] [20230214:210232:1676404952:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:W32Ub22:1:1:1791:1791] [20230214:210232:1676404952:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f520 (old at (nil))
[EC_DRHOOK:W32Ub22:1:1:1791:1791] [20230214:210232:1676404952:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:W32Ub22:1:1:1791:1791] [20230214:210232:1676404952:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:W32Ub22:1:1:1791:1791] [20230214:210232:1676404952:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:W32Ub22:1:1:1791:1791] [20230214:210232:1676404952:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f520 (old at (nil))
[EC_DRHOOK:W32Ub22:1:1:1791:1791] [20230214:210232:1676404952:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f520 (old at (nil))
[EC_DRHOOK:W32Ub22:1:1:1791:1791] [20230214:210232:1676404952:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f520 (old at (nil))
[EC_DRHOOK:W32Ub22:1:1:1791:1791] [20230214:210232:1676404952:0.000] [catch_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  21:03:33 STEP 1152 H= 288:00 +CPU= 42.764
  21:04:04 STEP 1153 H= 288:15 +CPU= 31.344
  21:04:34 STEP 1154 H= 288:30 +CPU= 29.676
  21:05:04 STEP 1155 H= 288:45 +CPU= 30.473
  21:05:35 STEP 1156 H= 289:00 +CPU= 30.227
  21:06:04 STEP 1157 H= 289:15 +CPU= 29.153
  21:06:33 STEP 1158 H= 289:30 +CPU= 29.369
  21:07:03 STEP 1159 H= 289:45 +CPU= 29.427
  21:07:33 STEP 1160 H= 290:00 +CPU= 29.854
  21:08:02 STEP 1161 H= 290:15 +CPU= 29.854
  21:08:32 STEP 1162 H= 290:30 +CPU= 29.726
  21:09:02 STEP 1163 H= 290:45 +CPU= 29.552
  21:09:38 STEP 1164 H= 291:00 +CPU= 36.007
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGh7zg+001164
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHh7zg+001164
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAh7zg+001164
  21:10:08 STEP 1165 H= 291:15 +CPU= 29.735
  21:10:38 STEP 1166 H= 291:30 +CPU= 30.103
  21:11:07 STEP 1167 H= 291:45 +CPU= 29.519
  21:11:38 STEP 1168 H= 292:00 +CPU= 30.444
  21:12:08 STEP 1169 H= 292:15 +CPU= 29.837
  21:12:37 STEP 1170 H= 292:30 +CPU= 29.682
  21:13:07 STEP 1171 H= 292:45 +CPU= 29.833
  21:13:37 STEP 1172 H= 293:00 +CPU= 30.349
  21:14:07 STEP 1173 H= 293:15 +CPU= 29.872
  21:14:37 STEP 1174 H= 293:30 +CPU= 30.022
  21:15:07 STEP 1175 H= 293:45 +CPU= 29.962
  21:15:44 STEP 1176 H= 294:00 +CPU= 36.777
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGh7zg+001176
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHh7zg+001176
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAh7zg+001176
  21:16:15 STEP 1177 H= 294:15 +CPU= 30.363
  21:16:45 STEP 1178 H= 294:30 +CPU= 30.114
  21:17:15 STEP 1179 H= 294:45 +CPU= 29.914
  21:17:45 STEP 1180 H= 295:00 +CPU= 30.617
  21:18:15 STEP 1181 H= 295:15 +CPU= 30.097
  21:18:45 STEP 1182 H= 295:30 +CPU= 29.803
  21:19:16 STEP 1183 H= 295:45 +CPU= 30.529
  21:19:49 STEP 1184 H= 296:00 +CPU= 33.058
  21:20:20 STEP 1185 H= 296:15 +CPU= 31.501
  21:20:51 STEP 1186 H= 296:30 +CPU= 31.101
  21:21:21 STEP 1187 H= 296:45 +CPU= 29.257
  21:21:57 STEP 1188 H= 297:00 +CPU= 36.407
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGh7zg+001188
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHh7zg+001188
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAh7zg+001188
  21:22:27 STEP 1189 H= 297:15 +CPU= 29.745
  21:22:57 STEP 1190 H= 297:30 +CPU= 29.556
  21:23:26 STEP 1191 H= 297:45 +CPU= 29.528
  21:23:56 STEP 1192 H= 298:00 +CPU= 29.902
  21:24:26 STEP 1193 H= 298:15 +CPU= 29.603
  21:24:55 STEP 1194 H= 298:30 +CPU= 29.705
  21:25:25 STEP 1195 H= 298:45 +CPU= 29.700
  21:25:55 STEP 1196 H= 299:00 +CPU= 30.083
  21:26:25 STEP 1197 H= 299:15 +CPU= 29.922
  21:26:55 STEP 1198 H= 299:30 +CPU= 29.641
  21:27:25 STEP 1199 H= 299:45 +CPU= 29.885
  21:28:11 STEP 1200 H= 300:00 +CPU= 45.776
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGh7zg+001200
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHh7zg+001200
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAh7zg+001200
  21:28:44 STEP 1201 H= 300:15 +CPU= 32.944
  21:29:14 STEP 1202 H= 300:30 +CPU= 30.604
  21:29:45 STEP 1203 H= 300:45 +CPU= 30.410
  21:30:16 STEP 1204 H= 301:00 +CPU= 30.941
  21:30:46 STEP 1205 H= 301:15 +CPU= 30.427
  21:31:16 STEP 1206 H= 301:30 +CPU= 30.326
  21:31:47 STEP 1207 H= 301:45 +CPU= 30.344
  21:32:17 STEP 1208 H= 302:00 +CPU= 30.652
  21:32:48 STEP 1209 H= 302:15 +CPU= 30.558
  21:33:18 STEP 1210 H= 302:30 +CPU= 30.017
  21:33:48 STEP 1211 H= 302:45 +CPU= 30.227
  21:34:25 STEP 1212 H= 303:00 +CPU= 36.750
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGh7zg+001212
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHh7zg+001212
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAh7zg+001212
  21:34:55 STEP 1213 H= 303:15 +CPU= 30.032
  21:35:25 STEP 1214 H= 303:30 +CPU= 29.838
  21:35:55 STEP 1215 H= 303:45 +CPU= 29.824
  21:36:25 STEP 1216 H= 304:00 +CPU= 30.447
  21:36:55 STEP 1217 H= 304:15 +CPU= 29.747
  21:37:25 STEP 1218 H= 304:30 +CPU= 30.182
  21:37:55 STEP 1219 H= 304:45 +CPU= 29.849
  21:38:26 STEP 1220 H= 305:00 +CPU= 30.612
  21:38:56 STEP 1221 H= 305:15 +CPU= 30.049
  21:39:26 STEP 1222 H= 305:30 +CPU= 30.301
  21:39:56 STEP 1223 H= 305:45 +CPU= 30.106
  21:40:33 STEP 1224 H= 306:00 +CPU= 37.098
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGh7zg+001224
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHh7zg+001224
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAh7zg+001224
  21:41:04 STEP 1225 H= 306:15 +CPU= 30.269
  21:41:34 STEP 1226 H= 306:30 +CPU= 30.484
  21:42:04 STEP 1227 H= 306:45 +CPU= 30.276
  21:42:35 STEP 1228 H= 307:00 +CPU= 30.717
  21:43:06 STEP 1229 H= 307:15 +CPU= 30.529
  21:43:36 STEP 1230 H= 307:30 +CPU= 30.019
  21:44:06 STEP 1231 H= 307:45 +CPU= 30.477
  21:44:37 STEP 1232 H= 308:00 +CPU= 30.266
  21:45:07 STEP 1233 H= 308:15 +CPU= 30.065
  21:45:37 STEP 1234 H= 308:30 +CPU= 29.876
  21:46:07 STEP 1235 H= 308:45 +CPU= 30.107
  21:46:43 STEP 1236 H= 309:00 +CPU= 36.609
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGh7zg+001236
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHh7zg+001236
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAh7zg+001236
  21:47:14 STEP 1237 H= 309:15 +CPU= 30.261
  21:47:44 STEP 1238 H= 309:30 +CPU= 30.136
  21:48:14 STEP 1239 H= 309:45 +CPU= 30.101
  21:48:45 STEP 1240 H= 310:00 +CPU= 30.819
  21:49:15 STEP 1241 H= 310:15 +CPU= 29.841
  21:49:45 STEP 1242 H= 310:30 +CPU= 29.925
  21:50:15 STEP 1243 H= 310:45 +CPU= 30.323
  21:50:46 STEP 1244 H= 311:00 +CPU= 30.984
  21:51:16 STEP 1245 H= 311:15 +CPU= 30.261
  21:51:46 STEP 1246 H= 311:30 +CPU= 29.576
  21:52:16 STEP 1247 H= 311:45 +CPU= 30.181
  21:53:02 STEP 1248 H= 312:00 +CPU= 46.442
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGh7zg+001248
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHh7zg+001248
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAh7zg+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209193/ICMGGh7zg+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209193/ICMSHh7zg+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209193/ICMUAh7zg+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209193/ICMGGh7zg+001164
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209193/ICMSHh7zg+001164
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209193/ICMUAh7zg+001164
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209193/ICMGGh7zg+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209193/ICMSHh7zg+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209193/ICMUAh7zg+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209193/ICMGGh7zg+001188
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209193/ICMSHh7zg+001188
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209193/ICMUAh7zg+001188
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209193/ICMGGh7zg+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209193/ICMSHh7zg+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209193/ICMUAh7zg+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209193/ICMGGh7zg+001212
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209193/ICMSHh7zg+001212
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209193/ICMUAh7zg+001212
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209193/ICMGGh7zg+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209193/ICMSHh7zg+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209193/ICMUAh7zg+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209193/ICMGGh7zg+001236
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209193/ICMSHh7zg+001236
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209193/ICMUAh7zg+001236
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a1uu_2016092300_15_991_12209193_0_r1235926258_12.zip
Uploading the intermediate file: upload_file_12.zip
  21:53:40 STEP 1249 H= 312:15 +CPU= 37.146
  21:54:11 STEP 1250 H= 312:30 +CPU= 31.469
  21:54:42 STEP 1251 H= 312:45 +CPU= 31.265
  21:55:13 STEP 1252 H= 313:00 +CPU= 30.699
  21:55:44 STEP 1253 H= 313:15 +CPU= 30.549
  21:56:14 STEP 1254 H= 313:30 +CPU= 30.400
  21:56:45 STEP 1255 H= 313:45 +CPU= 30.500
  21:57:16 STEP 1256 H= 314:00 +CPU= 31.027
  21:57:46 STEP 1257 H= 314:15 +CPU= 30.461
  21:58:17 STEP 1258 H= 314:30 +CPU= 30.390
  21:58:47 STEP 1259 H= 314:45 +CPU= 30.400
  21:59:23 STEP 1260 H= 315:00 +CPU= 36.410
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGh7zg+001260
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHh7zg+001260
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAh7zg+001260
  21:59:54 STEP 1261 H= 315:15 +CPU= 30.720
  22:00:25 STEP 1262 H= 315:30 +CPU= 30.637
  22:00:55 STEP 1263 H= 315:45 +CPU= 30.357
  22:01:27 STEP 1264 H= 316:00 +CPU= 31.705
  22:01:59 STEP 1265 H= 316:15 +CPU= 32.255
  22:02:32 STEP 1266 H= 316:30 +CPU= 32.955
  22:03:04 STEP 1267 H= 316:45 +CPU= 32.133
  22:03:34 STEP 1268 H= 317:00 +CPU= 30.067
  22:04:04 STEP 1269 H= 317:15 +CPU= 29.812
  22:04:34 STEP 1270 H= 317:30 +CPU= 29.969
  22:05:05 STEP 1271 H= 317:45 +CPU= 30.240
  22:05:41 STEP 1272 H= 318:00 +CPU= 36.825
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGh7zg+001272
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHh7zg+001272
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAh7zg+001272
  22:06:12 STEP 1273 H= 318:15 +CPU= 30.132
  22:06:41 STEP 1274 H= 318:30 +CPU= 29.915
  22:07:11 STEP 1275 H= 318:45 +CPU= 29.914
  22:07:42 STEP 1276 H= 319:00 +CPU= 30.560
  22:08:12 STEP 1277 H= 319:15 +CPU= 29.959
  22:08:42 STEP 1278 H= 319:30 +CPU= 30.133
  22:09:12 STEP 1279 H= 319:45 +CPU= 29.932
  22:09:43 STEP 1280 H= 320:00 +CPU= 30.442
  22:10:13 STEP 1281 H= 320:15 +CPU= 30.388
  22:10:43 STEP 1282 H= 320:30 +CPU= 29.699
  22:11:12 STEP 1283 H= 320:45 +CPU= 29.640
  22:11:49 STEP 1284 H= 321:00 +CPU= 36.146
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGh7zg+001284
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHh7zg+001284
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAh7zg+001284
  22:12:19 STEP 1285 H= 321:15 +CPU= 30.510
  22:12:50 STEP 1286 H= 321:30 +CPU= 30.665
  22:13:20 STEP 1287 H= 321:45 +CPU= 30.171
  22:13:51 STEP 1288 H= 322:00 +CPU= 30.514
  22:14:21 STEP 1289 H= 322:15 +CPU= 29.972
  22:14:51 STEP 1290 H= 322:30 +CPU= 29.957
  22:15:20 STEP 1291 H= 322:45 +CPU= 29.658
  22:15:51 STEP 1292 H= 323:00 +CPU= 30.601
  22:16:21 STEP 1293 H= 323:15 +CPU= 30.334
  22:16:51 STEP 1294 H= 323:30 +CPU= 29.749
  22:17:21 STEP 1295 H= 323:45 +CPU= 30.075
  22:18:08 STEP 1296 H= 324:00 +CPU= 46.957
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGh7zg+001296
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHh7zg+001296
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAh7zg+001296
  22:18:39 STEP 1297 H= 324:15 +CPU= 31.232
  22:19:10 STEP 1298 H= 324:30 +CPU= 30.634
  22:19:41 STEP 1299 H= 324:45 +CPU= 30.572
  22:20:12 STEP 1300 H= 325:00 +CPU= 31.000
  22:20:42 STEP 1301 H= 325:15 +CPU= 30.642
  22:21:13 STEP 1302 H= 325:30 +CPU= 30.242
  22:21:43 STEP 1303 H= 325:45 +CPU= 30.440
  22:22:14 STEP 1304 H= 326:00 +CPU= 30.976
  22:22:45 STEP 1305 H= 326:15 +CPU= 30.780
  22:23:15 STEP 1306 H= 326:30 +CPU= 30.492
  22:23:45 STEP 1307 H= 326:45 +CPU= 30.121
  22:24:22 STEP 1308 H= 327:00 +CPU= 36.482
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGh7zg+001308
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHh7zg+001308
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAh7zg+001308
  22:24:52 STEP 1309 H= 327:15 +CPU= 30.316
  22:25:23 STEP 1310 H= 327:30 +CPU= 30.365
  22:25:53 STEP 1311 H= 327:45 +CPU= 30.600
  22:26:24 STEP 1312 H= 328:00 +CPU= 30.866
  22:26:55 STEP 1313 H= 328:15 +CPU= 30.566
  22:27:25 STEP 1314 H= 328:30 +CPU= 30.154
  22:27:55 STEP 1315 H= 328:45 +CPU= 30.373
  22:28:26 STEP 1316 H= 329:00 +CPU= 30.799
  22:28:57 STEP 1317 H= 329:15 +CPU= 30.546
  22:29:27 STEP 1318 H= 329:30 +CPU= 30.185
  22:29:56 STEP 1319 H= 329:45 +CPU= 29.686
  22:30:33 STEP 1320 H= 330:00 +CPU= 36.488
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGh7zg+001320
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHh7zg+001320
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAh7zg+001320
  22:31:03 STEP 1321 H= 330:15 +CPU= 30.222
  22:31:33 STEP 1322 H= 330:30 +CPU= 30.174
  22:32:03 STEP 1323 H= 330:45 +CPU= 30.074
  22:32:34 STEP 1324 H= 331:00 +CPU= 30.440
  22:33:04 STEP 1325 H= 331:15 +CPU= 30.499
  22:33:34 STEP 1326 H= 331:30 +CPU= 29.584
  22:34:04 STEP 1327 H= 331:45 +CPU= 29.846
  22:34:34 STEP 1328 H= 332:00 +CPU= 30.223
  22:35:04 STEP 1329 H= 332:15 +CPU= 29.954
  22:35:34 STEP 1330 H= 332:30 +CPU= 29.732
  22:36:04 STEP 1331 H= 332:45 +CPU= 30.159
  22:36:41 STEP 1332 H= 333:00 +CPU= 37.100
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGh7zg+001332
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHh7zg+001332
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAh7zg+001332
  22:37:11 STEP 1333 H= 333:15 +CPU= 29.947
  22:37:41 STEP 1334 H= 333:30 +CPU= 29.683
  22:38:11 STEP 1335 H= 333:45 +CPU= 30.031
  22:38:41 STEP 1336 H= 334:00 +CPU= 30.562
  22:39:12 STEP 1337 H= 334:15 +CPU= 30.505
  22:39:42 STEP 1338 H= 334:30 +CPU= 30.525
  22:40:13 STEP 1339 H= 334:45 +CPU= 30.178
  22:40:43 STEP 1340 H= 335:00 +CPU= 30.678
  22:41:13 STEP 1341 H= 335:15 +CPU= 30.010
  22:41:43 STEP 1342 H= 335:30 +CPU= 29.809
  22:42:13 STEP 1343 H= 335:45 +CPU= 30.095
  22:42:59 STEP 1344 H= 336:00 +CPU= 46.045
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGh7zg+001344
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHh7zg+001344
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAh7zg+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209193/ICMGGh7zg+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209193/ICMSHh7zg+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209193/ICMUAh7zg+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209193/ICMGGh7zg+001260
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209193/ICMSHh7zg+001260
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209193/ICMUAh7zg+001260
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209193/ICMGGh7zg+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209193/ICMSHh7zg+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209193/ICMUAh7zg+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209193/ICMGGh7zg+001284
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209193/ICMSHh7zg+001284
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209193/ICMUAh7zg+001284
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209193/ICMGGh7zg+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209193/ICMSHh7zg+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209193/ICMUAh7zg+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209193/ICMGGh7zg+001308
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209193/ICMSHh7zg+001308
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209193/ICMUAh7zg+001308
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209193/ICMGGh7zg+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209193/ICMSHh7zg+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209193/ICMUAh7zg+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209193/ICMGGh7zg+001332
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209193/ICMSHh7zg+001332
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209193/ICMUAh7zg+001332
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a1uu_2016092300_15_991_12209193_0_r1235926258_13.zip
Uploading the intermediate file: upload_file_13.zip
  22:43:35 STEP 1345 H= 336:15 +CPU= 35.503
  22:44:05 STEP 1346 H= 336:30 +CPU= 30.678
  22:44:36 STEP 1347 H= 336:45 +CPU= 30.924
  22:45:07 STEP 1348 H= 337:00 +CPU= 30.804
  22:45:38 STEP 1349 H= 337:15 +CPU= 30.709
  22:46:09 STEP 1350 H= 337:30 +CPU= 30.949
  22:46:40 STEP 1351 H= 337:45 +CPU= 31.002
  22:47:11 STEP 1352 H= 338:00 +CPU= 30.850
  22:47:41 STEP 1353 H= 338:15 +CPU= 30.536
  22:48:12 STEP 1354 H= 338:30 +CPU= 30.905
  22:48:42 STEP 1355 H= 338:45 +CPU= 29.676
  22:49:19 STEP 1356 H= 339:00 +CPU= 36.893
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGh7zg+001356
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHh7zg+001356
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAh7zg+001356
  22:49:49 STEP 1357 H= 339:15 +CPU= 30.648
  22:50:20 STEP 1358 H= 339:30 +CPU= 31.068
  22:50:51 STEP 1359 H= 339:45 +CPU= 30.491
  22:51:22 STEP 1360 H= 340:00 +CPU= 31.077
  22:51:53 STEP 1361 H= 340:15 +CPU= 30.493
  22:52:24 STEP 1362 H= 340:30 +CPU= 31.342
  22:52:55 STEP 1363 H= 340:45 +CPU= 31.077
  22:53:26 STEP 1364 H= 341:00 +CPU= 31.070
  22:53:57 STEP 1365 H= 341:15 +CPU= 30.598
  22:54:27 STEP 1366 H= 341:30 +CPU= 30.265
  22:54:57 STEP 1367 H= 341:45 +CPU= 30.492
  22:55:35 STEP 1368 H= 342:00 +CPU= 37.608
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGh7zg+001368
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHh7zg+001368
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAh7zg+001368
  22:56:06 STEP 1369 H= 342:15 +CPU= 30.680
  22:56:36 STEP 1370 H= 342:30 +CPU= 30.640
  22:57:07 STEP 1371 H= 342:45 +CPU= 30.109
  22:57:37 STEP 1372 H= 343:00 +CPU= 30.714
  22:58:07 STEP 1373 H= 343:15 +CPU= 30.170
  22:58:38 STEP 1374 H= 343:30 +CPU= 30.199
  22:59:08 STEP 1375 H= 343:45 +CPU= 30.227
  22:59:39 STEP 1376 H= 344:00 +CPU= 30.762
  23:00:09 STEP 1377 H= 344:15 +CPU= 30.617
  23:00:39 STEP 1378 H= 344:30 +CPU= 30.204
  23:01:10 STEP 1379 H= 344:45 +CPU= 30.119
  23:01:46 STEP 1380 H= 345:00 +CPU= 36.671
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGh7zg+001380
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHh7zg+001380
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAh7zg+001380
  23:02:17 STEP 1381 H= 345:15 +CPU= 30.568
  23:02:47 STEP 1382 H= 345:30 +CPU= 30.470
  23:03:18 STEP 1383 H= 345:45 +CPU= 31.007
  23:03:49 STEP 1384 H= 346:00 +CPU= 30.496
  23:04:19 STEP 1385 H= 346:15 +CPU= 30.366
  23:04:49 STEP 1386 H= 346:30 +CPU= 30.120
  23:05:19 STEP 1387 H= 346:45 +CPU= 29.998
  23:05:50 STEP 1388 H= 347:00 +CPU= 30.728
  23:06:20 STEP 1389 H= 347:15 +CPU= 30.120
  23:06:51 STEP 1390 H= 347:30 +CPU= 30.376
  23:07:21 STEP 1391 H= 347:45 +CPU= 30.083
  23:08:07 STEP 1392 H= 348:00 +CPU= 46.523
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGh7zg+001392
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHh7zg+001392
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAh7zg+001392
  23:08:38 STEP 1393 H= 348:15 +CPU= 31.147
  23:09:09 STEP 1394 H= 348:30 +CPU= 30.717
  23:09:40 STEP 1395 H= 348:45 +CPU= 30.615
  23:10:11 STEP 1396 H= 349:00 +CPU= 31.353
  23:10:42 STEP 1397 H= 349:15 +CPU= 30.489
  23:11:12 STEP 1398 H= 349:30 +CPU= 30.199
  23:11:43 STEP 1399 H= 349:45 +CPU= 30.743
  23:12:14 STEP 1400 H= 350:00 +CPU= 31.425
  23:12:44 STEP 1401 H= 350:15 +CPU= 30.408
  23:13:14 STEP 1402 H= 350:30 +CPU= 29.964
  23:13:44 STEP 1403 H= 350:45 +CPU= 29.398
  23:14:20 STEP 1404 H= 351:00 +CPU= 36.699
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGh7zg+001404
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHh7zg+001404
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAh7zg+001404
  23:14:51 STEP 1405 H= 351:15 +CPU= 30.071
  23:15:21 STEP 1406 H= 351:30 +CPU= 30.023
  23:15:51 STEP 1407 H= 351:45 +CPU= 30.120
  23:16:21 STEP 1408 H= 352:00 +CPU= 30.582
  23:16:52 STEP 1409 H= 352:15 +CPU= 30.244
  23:17:22 STEP 1410 H= 352:30 +CPU= 30.100
  23:17:52 STEP 1411 H= 352:45 +CPU= 30.091
  23:18:22 STEP 1412 H= 353:00 +CPU= 30.658
  23:18:54 STEP 1413 H= 353:15 +CPU= 31.854
  23:19:26 STEP 1414 H= 353:30 +CPU= 31.317
  23:19:55 STEP 1415 H= 353:45 +CPU= 29.816
  23:20:32 STEP 1416 H= 354:00 +CPU= 36.593
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGh7zg+001416
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHh7zg+001416
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAh7zg+001416
  23:21:02 STEP 1417 H= 354:15 +CPU= 29.716
  23:21:31 STEP 1418 H= 354:30 +CPU= 29.557
  23:22:01 STEP 1419 H= 354:45 +CPU= 29.779
  23:22:31 STEP 1420 H= 355:00 +CPU= 30.079
  23:23:01 STEP 1421 H= 355:15 +CPU= 30.038
  23:23:31 STEP 1422 H= 355:30 +CPU= 30.058
  23:24:01 STEP 1423 H= 355:45 +CPU= 29.988
  23:24:32 STEP 1424 H= 356:00 +CPU= 30.659
  23:25:02 STEP 1425 H= 356:15 +CPU= 29.729
  23:25:32 STEP 1426 H= 356:30 +CPU= 29.926
  23:26:02 STEP 1427 H= 356:45 +CPU= 30.278
  23:26:39 STEP 1428 H= 357:00 +CPU= 36.642
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGh7zg+001428
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHh7zg+001428
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAh7zg+001428
  23:27:08 STEP 1429 H= 357:15 +CPU= 29.878
  23:27:39 STEP 1430 H= 357:30 +CPU= 30.023
  23:28:09 STEP 1431 H= 357:45 +CPU= 30.223
  23:28:40 STEP 1432 H= 358:00 +CPU= 30.937
  23:29:11 STEP 1433 H= 358:15 +CPU= 31.163
  23:29:41 STEP 1434 H= 358:30 +CPU= 30.068
  23:30:11 STEP 1435 H= 358:45 +CPU= 29.871
  23:30:41 STEP 1436 H= 359:00 +CPU= 30.430
  23:31:11 STEP 1437 H= 359:15 +CPU= 30.093
  23:31:41 STEP 1438 H= 359:30 +CPU= 29.791
  23:32:11 STEP 1439 H= 359:45 +CPU= 30.242
  23:32:55 STEP 1440 H= 360:00 +CPU= 43.392
..The child process terminated with status: 0
>>> Printing last 70 lines from file: NODE.001_01

 
DDH------------------------------------------------------------------------
I                                                                         I
I TIME =  360.000H DDH * Diagnostic Var.,     AREA                GLOBAL  I
I                                                                         I
I-------------------------------------------------------------------------I
I   VARIABLE   I      Initial value         I        Current value        I
I-------------------------------------------------------------------------I
I  OMEGA (P/S) I     -0.5086353400E-21      I       0.2077511228E-06      I
ICloud fractionI      0.0000000000E+00      I       0.3191029128E-02      I
I  Relat. Hum. I      0.5219044776E+00      I       0.5255609345E+00      I
I  PASS 01     I      0.0000000000E+00      I       0.4748513857E+02      I
DDH------------------------------------------------------------------------
  
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1440 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001440 MODE=a
 IO-STREAM CLOSED - ICMGGh7zg+001440
 MPI-TASK:   1 -    307370113 BYTES IN      7 RECORDS TRANSFERRED IN    0.0006 SECONDS512283.5 Mbytes/s, TOTAL TIME=    0.0159( 3.8%)
 23:32:55 STEP 1440 H= 360:00 +CPU= 43.392
 END CNT3
 NSTEP =  1440 CNT0   000000000  30.289  30.289  30.312   0.999               0               0               0               0               0
IO-STREAM STATISTICS, TOTAL NO OF REC/BYTES READ           34/  1069125708 WRITTEN         8508/************
 -TASK-OPENED-OPEN-RECS IN -KBYTE IN    -RECS OUT-KBYTE OUT-WALL   -WALL IN-WALL
  OU-TOT IN -TOT OUT
   1    153    0       34      1044068     8508 ************    40.1    15.3    24.7    16.5    44.4
===-=== START OF TIMING STATISTICS ===-===
 
STATS FOR ALL TASKS
 NUM ROUTINE                                     CALLS  MEAN(ms)   MAX(ms)   FRAC(%)  UNBAL(%)
   0 CNT0     - COMPLETE EXECUTION                   1 9023020.4 9023020.4    100.00      0.00
   1 CNT4     - FORWARD INTEGRATION                  1 9004765.8 9004765.8     99.80      0.00
   8 SCAN2M - GRID-POINT DYNAMICS                  313    6449.3    6449.3     22.37      0.00
   9 SPCM     - SPECTRAL COMP.                     288     991.7     991.7      3.17      0.00
  10 SCAN2M - PHYSICS                              289   13416.3   13416.3     42.97      0.00
  11 IOPACK   - OUTPUT P.P. RESULTS                 24    1516.9    1516.9      0.40      0.00
  12 SPNORM   - SPECTRAL NORM COMP.                 13      71.2      71.2      0.01      0.00
  17 GRIDFPOS IN CNT4                               24      34.6      34.6      0.01      0.00
  27 RESTART FILES - WRITING                         6    6067.4    6067.4      0.40      0.00
  28 RESTART FILES - READING                         1   17212.1   17212.1      0.19      0.00
  29 SU4FPOS IN CNT4                                24       0.3       0.3      0.00      0.00
  30 DYNFPOS IN CNT4                                24    6423.8    6423.8      1.71      0.00
  31 POSDDH IN STEPO                                 2       4.4       4.4      0.00      0.00
  37 CPGLAG   - SL COMPUTATIONS                    289    5794.5    5794.5     18.56      0.00
  39 SU0YOMB                                         1     708.4     708.4      0.01      0.00
  51 SCAN2M   - SL COMM. PART 1                    289      37.6      37.6      0.12      0.00
  54 SPCM     - M TO S/S TO M TRANSP.              288     396.4     396.4      1.27      0.00
  55 SPCIMPF  - S TO M/M TO S TRANSP.              288      89.8      89.8      0.29      0.00
  56 SPNORM   - SPECTRAL NORM COMM.                 13       0.2       0.2      0.00      0.00
 102 LTINV_CTL   - INVERSE LEGENDRE TRANSFORM      628     531.3     531.3      3.70      0.00
 103 LTDIR_CTL   - DIRECT LEGENDRE TRANSFORM       600     314.7     314.7      2.09      0.00
 106 FTDIR_CTL   - DIRECT FOURIER TRANSFORM        600     103.0     103.0      0.68      0.00
 107 FTINV_CTL   - INVERSE FOURIER TRANSFORM       628     148.0     148.0      1.03      0.00
 140 SULEG       - COMP. OF LEGENDRE POL.            1      72.2      72.2      0.00      0.00
 152 LTINV_CTL   - M TO L TRANSPOSITION            628      83.6      83.6      0.58      0.00
 153 LTDIR_CTL   - L TO M TRANSPOSITION            600      46.2      46.2      0.31      0.00
 157 FTINV_CTL   - L TO G TRANSPOSITION            628      65.9      65.9      0.46      0.00
 158 FTDIR_CTL   - G TO L TRANSPOSITION            600      38.8      38.8      0.26      0.00
 400 GSTATS                                      34304       0.0       0.0      0.00      0.00
 401 GSTATS HOOK                                 31868       0.0       0.0      0.00      0.00
TOTAL MEASURED IMBALANCE =       0.0 SECONDS,  0.0 PERCENT
TOTAL WALLCLOCK TIME     9023.0 CPU TIME    9004.8 VECTOR TIME     9004.8


===-=== END   OF TIMING STATISTICS ===-===
FORECAST DAYS PER DAY      143.9
  *** END CNT0 *** 
------------------------------------------------
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGh7zg+001440
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHh7zg+001440
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAh7zg+001440
Adding to the zip: /var/lib/boinc-client/slots/6/NODE.001_01
Adding to the zip: /var/lib/boinc-client/slots/6/ifs.stat
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209193/ICMSHh7zg+001428
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209193/ICMUAh7zg+001428
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209193/ICMSHh7zg+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209193/ICMUAh7zg+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209193/ICMGGh7zg+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209193/ICMGGh7zg+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209193/ICMUAh7zg+001356
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209193/ICMUAh7zg+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209193/ICMGGh7zg+001404
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209193/ICMSHh7zg+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209193/ICMGGh7zg+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209193/ICMGGh7zg+001428
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209193/ICMUAh7zg+001404
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209193/ICMGGh7zg+001356
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209193/ICMSHh7zg+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209193/ICMSHh7zg+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209193/ICMGGh7zg+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209193/ICMSHh7zg+001404
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209193/ICMGGh7zg+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209193/ICMSHh7zg+001356
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209193/ICMUAh7zg+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209193/ICMSHh7zg+001380
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209193/ICMGGh7zg+001380
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209193/ICMUAh7zg+001380
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209193/ICMUAh7zg+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209193/ICMSHh7zg+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209193/ICMUAh7zg+001440
Zipping up the final file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a1uu_2016092300_15_991_12209193_0_r1235926258_14.zip
Uploading the final file: upload_file_14.zip
Uploading trickle at timestep: 1295100
23:36:27 (1659): called boinc_finish(0)

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:51:26 1536331 22309383 oifs_43r3_bl_a1uu_2016092300_15_991_12209193_0 1,295,100 45,420 0.0351
09 May 2023 16:51:21 1536331 22309383 oifs_43r3_bl_a1uu_2016092300_15_991_12209193_0 864,000 27,850 0.0322


©2024 climateprediction.net