climateprediction.net home page
Task 22264183

Task 22264183

Name oifs_43r3_ps_0637_1993050100_123_962_12179281_0
Workunit 12179281
Created 22 Dec 2022, 3:38:54 UTC
Sent 26 Dec 2022, 20:30:59 UTC
Report deadline 25 Jan 2023, 20:30:59 UTC
Received 29 Dec 2022, 2:02:37 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1467890
Run time 23 hours 23 min 31 sec
CPU time 23 hours 20 min 18 sec
Validate state Invalid
Credit 10,039.68
Device peak FLOPS 4.20 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,407.76 MB
Peak swap size 4,978.07 MB
Peak disk usage 7,550.59 MB
Stderr 
<core_client_version>7.16.6</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
 [20221228:223729:1672263449:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1446033:1446033] [20221228:223729:1672263449:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  22:38:34 STEP 2208 H=2208:00 +CPU= 53.765
  22:38:59 STEP 2209 H=2209:00 +CPU= 24.478
  22:39:29 STEP 2210 H=2210:00 +CPU= 30.433
  22:40:17 STEP 2211 H=2211:00 +CPU= 47.089
  22:40:47 STEP 2212 H=2212:00 +CPU= 30.814
  22:41:12 STEP 2213 H=2213:00 +CPU= 24.377
  22:41:54 STEP 2214 H=2214:00 +CPU= 42.647
  22:42:24 STEP 2215 H=2215:00 +CPU= 29.651
  22:42:52 STEP 2216 H=2216:00 +CPU= 27.631
  22:43:37 STEP 2217 H=2217:00 +CPU= 45.365
  22:44:08 STEP 2218 H=2218:00 +CPU= 30.158
  22:44:37 STEP 2219 H=2219:00 +CPU= 29.852
  22:45:22 STEP 2220 H=2220:00 +CPU= 44.725
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002220
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002220
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002220
  22:45:47 STEP 2221 H=2221:00 +CPU= 24.874
  22:46:19 STEP 2222 H=2222:00 +CPU= 31.640
  22:47:06 STEP 2223 H=2223:00 +CPU= 47.206
  22:47:37 STEP 2224 H=2224:00 +CPU= 30.925
  22:48:07 STEP 2225 H=2225:00 +CPU= 29.533
  22:48:51 STEP 2226 H=2226:00 +CPU= 44.500
  22:49:22 STEP 2227 H=2227:00 +CPU= 29.768
  22:49:52 STEP 2228 H=2228:00 +CPU= 30.294
  22:50:35 STEP 2229 H=2229:00 +CPU= 42.815
  22:51:00 STEP 2230 H=2230:00 +CPU= 24.821
  22:51:26 STEP 2231 H=2231:00 +CPU= 26.146
  22:52:23 STEP 2232 H=2232:00 +CPU= 55.512
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002232
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002232
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002232
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179281/ICMGGhq0f+002208
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179281/ICMSHhq0f+002208
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179281/ICMUAhq0f+002208
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179281/ICMGGhq0f+002220
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179281/ICMSHhq0f+002220
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179281/ICMUAhq0f+002220
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0637_1993050100_123_962_12179281_0_r1385864353_92.zip
Uploading the intermediate file: upload_file_92.zip
  22:52:49 STEP 2233 H=2233:00 +CPU= 26.368
  22:53:21 STEP 2234 H=2234:00 +CPU= 31.185
  22:54:02 STEP 2235 H=2235:00 +CPU= 40.968
  22:54:31 STEP 2236 H=2236:00 +CPU= 29.170
  22:54:55 STEP 2237 H=2237:00 +CPU= 23.657
  22:55:36 STEP 2238 H=2238:00 +CPU= 41.438
  22:56:06 STEP 2239 H=2239:00 +CPU= 30.081
  22:56:34 STEP 2240 H=2240:00 +CPU= 28.239
  22:57:19 STEP 2241 H=2241:00 +CPU= 44.777
  22:57:50 STEP 2242 H=2242:00 +CPU= 30.325
  22:58:14 STEP 2243 H=2243:00 +CPU= 24.254
  22:59:04 STEP 2244 H=2244:00 +CPU= 49.851
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002244
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002244
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002244
  22:59:28 STEP 2245 H=2245:00 +CPU= 24.448
  22:59:58 STEP 2246 H=2246:00 +CPU= 29.810
  23:00:45 STEP 2247 H=2247:00 +CPU= 47.143
  23:01:12 STEP 2248 H=2248:00 +CPU= 26.096
  23:01:42 STEP 2249 H=2249:00 +CPU= 29.917
  23:02:30 STEP 2250 H=2250:00 +CPU= 48.484
  23:03:00 STEP 2251 H=2251:00 +CPU= 29.942
  23:03:27 STEP 2252 H=2252:00 +CPU= 27.037
  23:04:12 STEP 2253 H=2253:00 +CPU= 44.342
  23:04:37 STEP 2254 H=2254:00 +CPU= 24.909
  23:05:05 STEP 2255 H=2255:00 +CPU= 28.083
  23:05:59 STEP 2256 H=2256:00 +CPU= 50.450
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002256
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002256
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002256
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179281/ICMGGhq0f+002232
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179281/ICMSHhq0f+002232
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179281/ICMUAhq0f+002232
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179281/ICMGGhq0f+002244
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179281/ICMSHhq0f+002244
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179281/ICMUAhq0f+002244
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0637_1993050100_123_962_12179281_0_r1385864353_93.zip
Uploading the intermediate file: upload_file_93.zip
  23:06:23 STEP 2257 H=2257:00 +CPU= 24.543
  23:06:54 STEP 2258 H=2258:00 +CPU= 31.182
  23:07:42 STEP 2259 H=2259:00 +CPU= 47.175
  23:08:13 STEP 2260 H=2260:00 +CPU= 30.821
  23:08:36 STEP 2261 H=2261:00 +CPU= 23.708
  23:09:18 STEP 2262 H=2262:00 +CPU= 41.768
  23:09:48 STEP 2263 H=2263:00 +CPU= 30.272
  23:10:19 STEP 2264 H=2264:00 +CPU= 30.516
  23:11:03 STEP 2265 H=2265:00 +CPU= 44.040
  23:11:34 STEP 2266 H=2266:00 +CPU= 30.340
  23:12:04 STEP 2267 H=2267:00 +CPU= 30.365
  23:12:49 STEP 2268 H=2268:00 +CPU= 45.313
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002268
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002268
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002268
  23:13:16 STEP 2269 H=2269:00 +CPU= 26.622
  23:13:47 STEP 2270 H=2270:00 +CPU= 31.054
  23:14:29 STEP 2271 H=2271:00 +CPU= 41.667
  23:15:00 STEP 2272 H=2272:00 +CPU= 31.019
  23:15:30 STEP 2273 H=2273:00 +CPU= 29.688
  23:16:13 STEP 2274 H=2274:00 +CPU= 43.415
  23:16:36 STEP 2275 H=2275:00 +CPU= 23.170
  23:17:02 STEP 2276 H=2276:00 +CPU= 25.242
  23:17:48 STEP 2277 H=2277:00 +CPU= 46.672
  23:18:19 STEP 2278 H=2278:00 +CPU= 30.227
  23:18:50 STEP 2279 H=2279:00 +CPU= 31.220
  23:19:37 STEP 2280 H=2280:00 +CPU= 46.573
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002280
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002280
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179281/ICMGGhq0f+002256
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179281/ICMSHhq0f+002256
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179281/ICMUAhq0f+002256
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179281/ICMGGhq0f+002268
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179281/ICMSHhq0f+002268
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179281/ICMUAhq0f+002268
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0637_1993050100_123_962_12179281_0_r1385864353_94.zip
Uploading the intermediate file: upload_file_94.zip
  23:20:03 STEP 2281 H=2281:00 +CPU= 26.091
  23:20:34 STEP 2282 H=2282:00 +CPU= 31.152
  23:21:22 STEP 2283 H=2283:00 +CPU= 47.297
  23:21:53 STEP 2284 H=2284:00 +CPU= 30.827
  23:22:16 STEP 2285 H=2285:00 +CPU= 23.625
  23:23:01 STEP 2286 H=2286:00 +CPU= 44.556
  23:23:30 STEP 2287 H=2287:00 +CPU= 29.475
  23:24:01 STEP 2288 H=2288:00 +CPU= 30.468
  23:24:43 STEP 2289 H=2289:00 +CPU= 42.566
  23:25:09 STEP 2290 H=2290:00 +CPU= 25.223
  23:25:37 STEP 2291 H=2291:00 +CPU= 27.843
  23:26:29 STEP 2292 H=2292:00 +CPU= 52.273
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002292
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002292
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002292
  23:26:59 STEP 2293 H=2293:00 +CPU= 29.917
  23:27:25 STEP 2294 H=2294:00 +CPU= 25.926
  23:28:05 STEP 2295 H=2295:00 +CPU= 39.863
  23:28:37 STEP 2296 H=2296:00 +CPU= 31.622
  23:29:06 STEP 2297 H=2297:00 +CPU= 29.466
  23:29:50 STEP 2298 H=2298:00 +CPU= 43.722
  23:30:19 STEP 2299 H=2299:00 +CPU= 29.505
  23:30:50 STEP 2300 H=2300:00 +CPU= 30.430
  23:31:36 STEP 2301 H=2301:00 +CPU= 45.884
  23:32:00 STEP 2302 H=2302:00 +CPU= 24.544
  23:32:29 STEP 2303 H=2303:00 +CPU= 28.560
  23:33:16 STEP 2304 H=2304:00 +CPU= 46.333
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002304
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002304
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002304
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179281/ICMGGhq0f+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179281/ICMSHhq0f+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179281/ICMUAhq0f+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179281/ICMGGhq0f+002292
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179281/ICMSHhq0f+002292
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179281/ICMUAhq0f+002292
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0637_1993050100_123_962_12179281_0_r1385864353_95.zip
Uploading the intermediate file: upload_file_95.zip
  23:33:47 STEP 2305 H=2305:00 +CPU= 30.345
  23:34:17 STEP 2306 H=2306:00 +CPU= 30.453
  23:35:00 STEP 2307 H=2307:00 +CPU= 42.735
  23:35:32 STEP 2308 H=2308:00 +CPU= 31.421
  23:36:01 STEP 2309 H=2309:00 +CPU= 29.602
  23:36:46 STEP 2310 H=2310:00 +CPU= 44.948
  23:37:11 STEP 2311 H=2311:00 +CPU= 24.836
  23:37:37 STEP 2312 H=2312:00 +CPU= 25.279
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1993050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0637
(argv4) batchid: 962
(argv5) wuid: 12179281
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/1
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179281
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_dd73c8be961a559612d07e6095a116cd to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0637_1993050100_123_962_12179281.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0637_1993050100_123_962_12179281.zip
ic_ancil_file: ic_ancil_12179281
ifsdata_file: ifsdata_12179281
climate_data_file: clim_data_12179281
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_4f3491678436a4ee3d150d782e39b67a to: /var/lib/boinc-client/slots/1/ic_ancil_12179281.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/1/ic_ancil_12179281.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_19da254ae066118bd1fbecde01667126 to: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12179281.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12179281.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/1/159l_2/clim_data_12179281.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/1/159l_2/clim_data_12179281.zip
Checking for progress XML file: /var/lib/boinc-client/slots/1/progress_file_12179281.xml
Opened progress file ok : /var/lib/boinc-client/slots/1/progress_file_12179281.xml
-- Model is restarting --
Adjusting last_iter, 2313, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/1/progress_file_12179281.xml
last_cpu_time: 75613
upload_file_number: 96
last_iter: 2303
last_upload: 8294400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0637_1993050100_123_962_12179281_0_r1385864353
The child process has been launched with process id: 1448972
Executing the command: /var/lib/boinc-client/slots/1/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1448972:1448972] [20221228:233904:1672267144:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1448972:1448972] [20221228:233904:1672267144:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1448972:1448972] [20221228:233904:1672267144:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1448972:1448972] [20221228:233904:1672267144:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1448972:1448972] [20221228:233904:1672267144:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1448972:1448972] [20221228:233904:1672267144:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1448972:1448972] [20221228:233904:1672267144:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1448972:1448972] [20221228:233904:1672267144:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1448972:1448972] [20221228:233904:1672267144:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1448972:1448972] [20221228:233904:1672267144:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1448972:1448972] [20221228:233904:1672267144:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1448972:1448972] [20221228:233904:1672267144:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1448972:1448972] [20221228:233904:1672267144:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1448972:1448972] [20221228:233904:1672267144:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1448972:1448972] [20221228:233904:1672267144:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1448972:1448972] [20221228:233904:1672267144:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1448972:1448972] [20221228:233904:1672267144:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1448972:1448972] [20221228:233904:1672267144:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1448972:1448972] [20221228:233904:1672267144:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1448972:1448972] [20221228:233904:1672267144:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1448972:1448972] [20221228:233904:1672267144:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1448972:1448972] [20221228:233904:1672267144:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1448972:1448972] [20221228:233904:1672267144:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1448972:1448972] [20221228:233904:1672267144:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1448972:1448972] [20221228:233904:1672267144:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1448972:1448972] [20221228:233904:1672267144:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1448972:1448972] [20221228:233904:1672267144:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1448972:1448972] [20221228:233904:1672267144:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1448972:1448972] [20221228:233904:1672267144:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1448972:1448972] [20221228:233904:1672267144:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  23:40:05 STEP 2304 H=2304:00 +CPU= 50.616
  23:40:30 STEP 2305 H=2305:00 +CPU= 24.487
  23:41:01 STEP 2306 H=2306:00 +CPU= 30.371
  23:41:48 STEP 2307 H=2307:00 +CPU= 47.119
  23:42:19 STEP 2308 H=2308:00 +CPU= 30.913
  23:42:44 STEP 2309 H=2309:00 +CPU= 25.210
  23:43:27 STEP 2310 H=2310:00 +CPU= 42.543
  23:43:57 STEP 2311 H=2311:00 +CPU= 30.188
  23:44:23 STEP 2312 H=2312:00 +CPU= 25.811
  23:45:07 STEP 2313 H=2313:00 +CPU= 44.252
  23:45:38 STEP 2314 H=2314:00 +CPU= 30.376
  23:46:08 STEP 2315 H=2315:00 +CPU= 30.398
  23:46:50 STEP 2316 H=2316:00 +CPU= 41.795
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002316
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002316
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002316
  23:47:16 STEP 2317 H=2317:00 +CPU= 25.972
  23:47:47 STEP 2318 H=2318:00 +CPU= 31.215
  23:48:35 STEP 2319 H=2319:00 +CPU= 47.202
  23:49:06 STEP 2320 H=2320:00 +CPU= 30.976
  23:49:35 STEP 2321 H=2321:00 +CPU= 29.483
  23:50:20 STEP 2322 H=2322:00 +CPU= 45.091
  23:50:44 STEP 2323 H=2323:00 +CPU= 24.173
  23:51:10 STEP 2324 H=2324:00 +CPU= 25.853
  23:51:58 STEP 2325 H=2325:00 +CPU= 47.706
  23:52:22 STEP 2326 H=2326:00 +CPU= 23.349
  23:52:51 STEP 2327 H=2327:00 +CPU= 29.696
  23:53:46 STEP 2328 H=2328:00 +CPU= 54.820
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002328
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002328
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002328
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179281/ICMGGhq0f+002304
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179281/ICMSHhq0f+002304
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179281/ICMUAhq0f+002304
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179281/ICMGGhq0f+002316
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179281/ICMSHhq0f+002316
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179281/ICMUAhq0f+002316
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0637_1993050100_123_962_12179281_0_r1385864353_96.zip
Uploading the intermediate file: upload_file_96.zip
  23:54:17 STEP 2329 H=2329:00 +CPU= 29.727
  23:54:41 STEP 2330 H=2330:00 +CPU= 24.062
  23:55:21 STEP 2331 H=2331:00 +CPU= 40.148
  23:55:52 STEP 2332 H=2332:00 +CPU= 31.010
  23:56:22 STEP 2333 H=2333:00 +CPU= 29.515
  23:57:10 STEP 2334 H=2334:00 +CPU= 47.658
  23:57:33 STEP 2335 H=2335:00 +CPU= 23.537
  23:57:59 STEP 2336 H=2336:00 +CPU= 26.084
  23:58:48 STEP 2337 H=2337:00 +CPU= 48.180
  23:59:18 STEP 2338 H=2338:00 +CPU= 30.353
  23:59:49 STEP 2339 H=2339:00 +CPU= 30.507
  00:00:38 STEP 2340 H=2340:00 +CPU= 49.475
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002340
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002340
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002340
  00:01:07 STEP 2341 H=2341:00 +CPU= 28.356
  00:01:32 STEP 2342 H=2342:00 +CPU= 25.236
  00:02:19 STEP 2343 H=2343:00 +CPU= 46.656
  00:02:49 STEP 2344 H=2344:00 +CPU= 30.393
  00:03:19 STEP 2345 H=2345:00 +CPU= 29.507
  00:04:06 STEP 2346 H=2346:00 +CPU= 47.601
  00:04:30 STEP 2347 H=2347:00 +CPU= 23.342
  00:04:56 STEP 2348 H=2348:00 +CPU= 26.248
  00:05:44 STEP 2349 H=2349:00 +CPU= 47.977
  00:06:15 STEP 2350 H=2350:00 +CPU= 30.325
  00:06:46 STEP 2351 H=2351:00 +CPU= 31.127
  00:07:34 STEP 2352 H=2352:00 +CPU= 47.723
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002352
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002352
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002352
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179281/ICMGGhq0f+002328
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179281/ICMSHhq0f+002328
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179281/ICMUAhq0f+002328
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179281/ICMGGhq0f+002340
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179281/ICMSHhq0f+002340
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179281/ICMUAhq0f+002340
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0637_1993050100_123_962_12179281_0_r1385864353_97.zip
Uploading the intermediate file: upload_file_97.zip
  00:08:04 STEP 2353 H=2353:00 +CPU= 30.667
  00:08:35 STEP 2354 H=2354:00 +CPU= 30.449
  00:09:19 STEP 2355 H=2355:00 +CPU= 43.842
  00:09:44 STEP 2356 H=2356:00 +CPU= 25.379
  00:10:10 STEP 2357 H=2357:00 +CPU= 25.777
  00:10:59 STEP 2358 H=2358:00 +CPU= 48.831
  00:11:28 STEP 2359 H=2359:00 +CPU= 29.439
  00:11:58 STEP 2360 H=2360:00 +CPU= 29.832
  00:12:38 STEP 2361 H=2361:00 +CPU= 39.445
  00:13:07 STEP 2362 H=2362:00 +CPU= 28.879
  00:13:37 STEP 2363 H=2363:00 +CPU= 29.786
  00:14:23 STEP 2364 H=2364:00 +CPU= 46.715
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002364
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002364
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002364
  00:14:53 STEP 2365 H=2365:00 +CPU= 29.633
  00:15:24 STEP 2366 H=2366:00 +CPU= 30.699
  00:16:08 STEP 2367 H=2367:00 +CPU= 43.897
  00:16:33 STEP 2368 H=2368:00 +CPU= 25.619
  00:17:02 STEP 2369 H=2369:00 +CPU= 28.569
  00:17:51 STEP 2370 H=2370:00 +CPU= 48.371
  00:18:21 STEP 2371 H=2371:00 +CPU= 30.061
  00:18:49 STEP 2372 H=2372:00 +CPU= 28.439
  00:19:29 STEP 2373 H=2373:00 +CPU= 39.954
  00:20:00 STEP 2374 H=2374:00 +CPU= 30.361
  00:20:26 STEP 2375 H=2375:00 +CPU= 25.924
  00:21:18 STEP 2376 H=2376:00 +CPU= 52.016
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002376
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002376
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002376
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179281/ICMGGhq0f+002352
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179281/ICMSHhq0f+002352
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179281/ICMUAhq0f+002352
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179281/ICMGGhq0f+002364
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179281/ICMSHhq0f+002364
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179281/ICMUAhq0f+002364
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0637_1993050100_123_962_12179281_0_r1385864353_98.zip
Uploading the intermediate file: upload_file_98.zip
  00:21:47 STEP 2377 H=2377:00 +CPU= 29.381
  00:22:19 STEP 2378 H=2378:00 +CPU= 32.091
  00:23:05 STEP 2379 H=2379:00 +CPU= 45.554
  00:23:30 STEP 2380 H=2380:00 +CPU= 24.892
  00:24:00 STEP 2381 H=2381:00 +CPU= 30.380
  00:24:45 STEP 2382 H=2382:00 +CPU= 45.092
  00:25:14 STEP 2383 H=2383:00 +CPU= 28.140
  00:25:44 STEP 2384 H=2384:00 +CPU= 30.470
  00:26:27 STEP 2385 H=2385:00 +CPU= 43.302
  00:26:58 STEP 2386 H=2386:00 +CPU= 30.365
  00:27:23 STEP 2387 H=2387:00 +CPU= 24.638
  00:28:15 STEP 2388 H=2388:00 +CPU= 52.202
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002388
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002388
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002388
  00:28:39 STEP 2389 H=2389:00 +CPU= 24.445
  00:29:08 STEP 2390 H=2390:00 +CPU= 29.014
  00:29:56 STEP 2391 H=2391:00 +CPU= 47.103
  00:30:27 STEP 2392 H=2392:00 +CPU= 31.042
  00:30:50 STEP 2393 H=2393:00 +CPU= 22.946
  00:31:32 STEP 2394 H=2394:00 +CPU= 42.038
  00:32:02 STEP 2395 H=2395:00 +CPU= 29.951
  00:32:33 STEP 2396 H=2396:00 +CPU= 30.564
  00:33:21 STEP 2397 H=2397:00 +CPU= 47.916
  00:33:47 STEP 2398 H=2398:00 +CPU= 26.376
  00:34:14 STEP 2399 H=2399:00 +CPU= 27.220
  00:35:08 STEP 2400 H=2400:00 +CPU= 50.966
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002400
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002400
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002400
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179281/ICMGGhq0f+002376
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179281/ICMSHhq0f+002376
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179281/ICMUAhq0f+002376
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179281/ICMGGhq0f+002388
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179281/ICMSHhq0f+002388
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179281/ICMUAhq0f+002388
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0637_1993050100_123_962_12179281_0_r1385864353_99.zip
Uploading the intermediate file: upload_file_99.zip
  00:35:32 STEP 2401 H=2401:00 +CPU= 24.352
  00:36:02 STEP 2402 H=2402:00 +CPU= 30.127
  00:36:49 STEP 2403 H=2403:00 +CPU= 46.314
  00:37:16 STEP 2404 H=2404:00 +CPU= 27.693
  00:37:46 STEP 2405 H=2405:00 +CPU= 29.396
  00:38:34 STEP 2406 H=2406:00 +CPU= 48.555
  00:38:59 STEP 2407 H=2407:00 +CPU= 24.655
  00:39:24 STEP 2408 H=2408:00 +CPU= 24.770
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1993050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0637
(argv4) batchid: 962
(argv5) wuid: 12179281
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/1
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179281
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_dd73c8be961a559612d07e6095a116cd to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0637_1993050100_123_962_12179281.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0637_1993050100_123_962_12179281.zip
ic_ancil_file: ic_ancil_12179281
ifsdata_file: ifsdata_12179281
climate_data_file: clim_data_12179281
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_4f3491678436a4ee3d150d782e39b67a to: /var/lib/boinc-client/slots/1/ic_ancil_12179281.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/1/ic_ancil_12179281.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_19da254ae066118bd1fbecde01667126 to: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12179281.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12179281.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/1/159l_2/clim_data_12179281.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/1/159l_2/clim_data_12179281.zip
Checking for progress XML file: /var/lib/boinc-client/slots/1/progress_file_12179281.xml
Opened progress file ok : /var/lib/boinc-client/slots/1/progress_file_12179281.xml
-- Model is restarting --
Adjusting last_iter, 2409, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/1/progress_file_12179281.xml
last_cpu_time: 79246
upload_file_number: 100
last_iter: 2399
last_upload: 8640000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0637_1993050100_123_962_12179281_0_r1385864353
The child process has been launched with process id: 1452014
Executing the command: /var/lib/boinc-client/slots/1/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1452014:1452014] [20221229:004107:1672270867:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1452014:1452014] [20221229:004107:1672270867:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1452014:1452014] [20221229:004107:1672270867:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1452014:1452014] [20221229:004107:1672270867:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1452014:1452014] [20221229:004107:1672270867:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1452014:1452014] [20221229:004107:1672270867:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1452014:1452014] [20221229:004107:1672270867:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1452014:1452014] [20221229:004107:1672270867:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1452014:1452014] [20221229:004107:1672270867:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1452014:1452014] [20221229:004107:1672270867:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1452014:1452014] [20221229:004107:1672270867:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1452014:1452014] [20221229:004107:1672270867:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1452014:1452014] [20221229:004107:1672270867:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1452014:1452014] [20221229:004107:1672270867:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1452014:1452014] [20221229:004107:1672270867:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1452014:1452014] [20221229:004107:1672270867:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1452014:1452014] [20221229:004107:1672270867:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1452014:1452014] [20221229:004107:1672270867:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1452014:1452014] [20221229:004107:1672270867:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1452014:1452014] [20221229:004107:1672270867:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1452014:1452014] [20221229:004107:1672270867:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1452014:1452014] [20221229:004107:1672270867:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1452014:1452014] [20221229:004107:1672270867:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1452014:1452014] [20221229:004107:1672270867:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1452014:1452014] [20221229:004107:1672270867:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1452014:1452014] [20221229:004107:1672270867:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1452014:1452014] [20221229:004107:1672270867:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1452014:1452014] [20221229:004107:1672270867:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1452014:1452014] [20221229:004107:1672270867:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1452014:1452014] [20221229:004107:1672270867:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  00:42:11 STEP 2400 H=2400:00 +CPU= 53.708
  00:42:41 STEP 2401 H=2401:00 +CPU= 29.415
  00:43:09 STEP 2402 H=2402:00 +CPU= 28.457
  00:43:49 STEP 2403 H=2403:00 +CPU= 39.424
  00:44:17 STEP 2404 H=2404:00 +CPU= 28.249
  00:44:43 STEP 2405 H=2405:00 +CPU= 26.024
  00:45:30 STEP 2406 H=2406:00 +CPU= 46.194
  00:45:59 STEP 2407 H=2407:00 +CPU= 29.391
  00:46:27 STEP 2408 H=2408:00 +CPU= 27.997
  00:47:14 STEP 2409 H=2409:00 +CPU= 46.688
  00:47:44 STEP 2410 H=2410:00 +CPU= 30.211
  00:48:14 STEP 2411 H=2411:00 +CPU= 29.668
  00:48:57 STEP 2412 H=2412:00 +CPU= 42.423
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002412
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002412
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002412
  00:49:24 STEP 2413 H=2413:00 +CPU= 26.740
  00:49:55 STEP 2414 H=2414:00 +CPU= 30.966
  00:50:38 STEP 2415 H=2415:00 +CPU= 42.977
  00:51:10 STEP 2416 H=2416:00 +CPU= 31.467
  00:51:39 STEP 2417 H=2417:00 +CPU= 29.514
  00:52:25 STEP 2418 H=2418:00 +CPU= 45.710
  00:52:50 STEP 2419 H=2419:00 +CPU= 24.740
  00:53:21 STEP 2420 H=2420:00 +CPU= 31.383
  00:54:03 STEP 2421 H=2421:00 +CPU= 41.347
  00:54:28 STEP 2422 H=2422:00 +CPU= 25.059
  00:54:59 STEP 2423 H=2423:00 +CPU= 31.448
  00:55:52 STEP 2424 H=2424:00 +CPU= 51.796
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002424
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002424
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002424
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179281/ICMGGhq0f+002400
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179281/ICMSHhq0f+002400
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179281/ICMUAhq0f+002400
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179281/ICMGGhq0f+002412
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179281/ICMSHhq0f+002412
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179281/ICMUAhq0f+002412
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0637_1993050100_123_962_12179281_0_r1385864353_100.zip
Uploading the intermediate file: upload_file_100.zip
  00:56:22 STEP 2425 H=2425:00 +CPU= 30.207
  00:56:50 STEP 2426 H=2426:00 +CPU= 27.272
  00:57:32 STEP 2427 H=2427:00 +CPU= 42.123
  00:58:02 STEP 2428 H=2428:00 +CPU= 30.263
  00:58:28 STEP 2429 H=2429:00 +CPU= 25.654
  00:59:15 STEP 2430 H=2430:00 +CPU= 46.655
  00:59:38 STEP 2431 H=2431:00 +CPU= 23.408
  01:00:05 STEP 2432 H=2432:00 +CPU= 26.778
  01:00:53 STEP 2433 H=2433:00 +CPU= 48.529
  01:01:19 STEP 2434 H=2434:00 +CPU= 25.334
  01:01:50 STEP 2435 H=2435:00 +CPU= 30.877
  01:02:35 STEP 2436 H=2436:00 +CPU= 45.663
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002436
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002436
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002436
  01:03:05 STEP 2437 H=2437:00 +CPU= 29.206
  01:03:35 STEP 2438 H=2438:00 +CPU= 30.603
  01:04:21 STEP 2439 H=2439:00 +CPU= 45.960
  01:04:47 STEP 2440 H=2440:00 +CPU= 25.044
  01:05:12 STEP 2441 H=2441:00 +CPU= 25.205
  01:06:01 STEP 2442 H=2442:00 +CPU= 49.432
  01:06:29 STEP 2443 H=2443:00 +CPU= 27.582
  01:06:58 STEP 2444 H=2444:00 +CPU= 29.180
  01:07:42 STEP 2445 H=2445:00 +CPU= 43.656
  01:08:09 STEP 2446 H=2446:00 +CPU= 26.650
  01:08:40 STEP 2447 H=2447:00 +CPU= 31.456
  01:09:36 STEP 2448 H=2448:00 +CPU= 55.620
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002448
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002448
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002448
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179281/ICMGGhq0f+002424
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179281/ICMSHhq0f+002424
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179281/ICMUAhq0f+002424
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179281/ICMGGhq0f+002436
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179281/ICMSHhq0f+002436
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179281/ICMUAhq0f+002436
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0637_1993050100_123_962_12179281_0_r1385864353_101.zip
Uploading the intermediate file: upload_file_101.zip
  01:10:06 STEP 2449 H=2449:00 +CPU= 29.561
  01:10:32 STEP 2450 H=2450:00 +CPU= 26.347
  01:11:20 STEP 2451 H=2451:00 +CPU= 47.709
  01:11:44 STEP 2452 H=2452:00 +CPU= 23.971
  01:12:10 STEP 2453 H=2453:00 +CPU= 26.199
  01:12:58 STEP 2454 H=2454:00 +CPU= 47.541
  01:13:22 STEP 2455 H=2455:00 +CPU= 24.340
  01:13:54 STEP 2456 H=2456:00 +CPU= 31.852
  01:14:42 STEP 2457 H=2457:00 +CPU= 47.213
  01:15:13 STEP 2458 H=2458:00 +CPU= 30.932
  01:15:39 STEP 2459 H=2459:00 +CPU= 25.937
  01:16:23 STEP 2460 H=2460:00 +CPU= 44.466
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002460
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002460
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002460
  01:16:53 STEP 2461 H=2461:00 +CPU= 29.453
  01:17:23 STEP 2462 H=2462:00 +CPU= 30.600
  01:18:11 STEP 2463 H=2463:00 +CPU= 47.697
  01:18:35 STEP 2464 H=2464:00 +CPU= 23.660
  01:19:04 STEP 2465 H=2465:00 +CPU= 29.371
  01:19:52 STEP 2466 H=2466:00 +CPU= 47.542
  01:20:18 STEP 2467 H=2467:00 +CPU= 26.482
  01:20:49 STEP 2468 H=2468:00 +CPU= 30.652
  01:21:37 STEP 2469 H=2469:00 +CPU= 47.775
  01:22:05 STEP 2470 H=2470:00 +CPU= 27.902
  01:22:31 STEP 2471 H=2471:00 +CPU= 25.722
  01:23:26 STEP 2472 H=2472:00 +CPU= 54.868
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002472
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002472
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179281/ICMGGhq0f+002448
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179281/ICMSHhq0f+002448
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179281/ICMUAhq0f+002448
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179281/ICMGGhq0f+002460
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179281/ICMSHhq0f+002460
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179281/ICMUAhq0f+002460
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0637_1993050100_123_962_12179281_0_r1385864353_102.zip
Uploading the intermediate file: upload_file_102.zip
  01:23:51 STEP 2473 H=2473:00 +CPU= 24.861
  01:24:18 STEP 2474 H=2474:00 +CPU= 27.458
  01:25:06 STEP 2475 H=2475:00 +CPU= 48.000
  01:25:31 STEP 2476 H=2476:00 +CPU= 24.659
  01:26:01 STEP 2477 H=2477:00 +CPU= 30.330
  01:26:46 STEP 2478 H=2478:00 +CPU= 44.286
  01:27:15 STEP 2479 H=2479:00 +CPU= 28.781
  01:27:45 STEP 2480 H=2480:00 +CPU= 30.511
  01:28:34 STEP 2481 H=2481:00 +CPU= 48.293
  01:29:00 STEP 2482 H=2482:00 +CPU= 26.367
  01:29:27 STEP 2483 H=2483:00 +CPU= 27.187
  01:30:20 STEP 2484 H=2484:00 +CPU= 52.523
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002484
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002484
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002484
  01:30:43 STEP 2485 H=2485:00 +CPU= 23.496
  01:31:11 STEP 2486 H=2486:00 +CPU= 27.751
  01:31:58 STEP 2487 H=2487:00 +CPU= 47.140
  01:32:29 STEP 2488 H=2488:00 +CPU= 30.938
  01:32:54 STEP 2489 H=2489:00 +CPU= 24.620
  01:33:35 STEP 2490 H=2490:00 +CPU= 40.996
  01:34:06 STEP 2491 H=2491:00 +CPU= 30.357
  01:34:36 STEP 2492 H=2492:00 +CPU= 30.407
  01:35:20 STEP 2493 H=2493:00 +CPU= 44.370
  01:35:51 STEP 2494 H=2494:00 +CPU= 30.082
  01:36:16 STEP 2495 H=2495:00 +CPU= 25.001
  01:37:12 STEP 2496 H=2496:00 +CPU= 55.710
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002496
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002496
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179281/ICMGGhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179281/ICMSHhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179281/ICMUAhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179281/ICMGGhq0f+002484
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179281/ICMSHhq0f+002484
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179281/ICMUAhq0f+002484
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0637_1993050100_123_962_12179281_0_r1385864353_103.zip
Uploading the intermediate file: upload_file_103.zip
  01:37:40 STEP 2497 H=2497:00 +CPU= 27.611
  01:38:11 STEP 2498 H=2498:00 +CPU= 30.825
  01:38:54 STEP 2499 H=2499:00 +CPU= 42.864
  01:39:22 STEP 2500 H=2500:00 +CPU= 28.745
  01:39:52 STEP 2501 H=2501:00 +CPU= 29.447
  01:40:36 STEP 2502 H=2502:00 +CPU= 44.485
  01:41:07 STEP 2503 H=2503:00 +CPU= 30.383
  01:41:37 STEP 2504 H=2504:00 +CPU= 29.757
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1993050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0637
(argv4) batchid: 962
(argv5) wuid: 12179281
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/1
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179281
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_dd73c8be961a559612d07e6095a116cd to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0637_1993050100_123_962_12179281.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0637_1993050100_123_962_12179281.zip
ic_ancil_file: ic_ancil_12179281
ifsdata_file: ifsdata_12179281
climate_data_file: clim_data_12179281
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_4f3491678436a4ee3d150d782e39b67a to: /var/lib/boinc-client/slots/1/ic_ancil_12179281.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/1/ic_ancil_12179281.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_19da254ae066118bd1fbecde01667126 to: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12179281.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12179281.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/1/159l_2/clim_data_12179281.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/1/159l_2/clim_data_12179281.zip
Checking for progress XML file: /var/lib/boinc-client/slots/1/progress_file_12179281.xml
Opened progress file ok : /var/lib/boinc-client/slots/1/progress_file_12179281.xml
-- Model is restarting --
Adjusting last_iter, 2505, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/1/progress_file_12179281.xml
last_cpu_time: 82889
upload_file_number: 104
last_iter: 2495
last_upload: 8985600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0637_1993050100_123_962_12179281_0_r1385864353
The child process has been launched with process id: 1455231
Executing the command: /var/lib/boinc-client/slots/1/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1455231:1455231] [20221229:014318:1672274598:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1455231:1455231] [20221229:014318:1672274598:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1455231:1455231] [20221229:014318:1672274598:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1455231:1455231] [20221229:014318:1672274598:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1455231:1455231] [20221229:014318:1672274598:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1455231:1455231] [20221229:014318:1672274598:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1455231:1455231] [20221229:014318:1672274598:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1455231:1455231] [20221229:014318:1672274598:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1455231:1455231] [20221229:014318:1672274598:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1455231:1455231] [20221229:014318:1672274598:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1455231:1455231] [20221229:014318:1672274598:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1455231:1455231] [20221229:014318:1672274598:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1455231:1455231] [20221229:014318:1672274598:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1455231:1455231] [20221229:014318:1672274598:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1455231:1455231] [20221229:014318:1672274598:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1455231:1455231] [20221229:014318:1672274598:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1455231:1455231] [20221229:014318:1672274598:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1455231:1455231] [20221229:014318:1672274598:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1455231:1455231] [20221229:014318:1672274598:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1455231:1455231] [20221229:014318:1672274598:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1455231:1455231] [20221229:014318:1672274598:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1455231:1455231] [20221229:014318:1672274598:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1455231:1455231] [20221229:014318:1672274598:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1455231:1455231] [20221229:014318:1672274598:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1455231:1455231] [20221229:014318:1672274598:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1455231:1455231] [20221229:014318:1672274598:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1455231:1455231] [20221229:014318:1672274598:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1455231:1455231] [20221229:014318:1672274598:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1455231:1455231] [20221229:014318:1672274598:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1455231:1455231] [20221229:014318:1672274598:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  01:44:22 STEP 2496 H=2496:00 +CPU= 53.295
  01:44:49 STEP 2497 H=2497:00 +CPU= 27.229
  01:45:20 STEP 2498 H=2498:00 +CPU= 30.588
  01:46:04 STEP 2499 H=2499:00 +CPU= 44.332
  01:46:34 STEP 2500 H=2500:00 +CPU= 29.697
  01:47:03 STEP 2501 H=2501:00 +CPU= 28.906
  01:47:44 STEP 2502 H=2502:00 +CPU= 41.206
  01:48:09 STEP 2503 H=2503:00 +CPU= 24.838
  01:48:34 STEP 2504 H=2504:00 +CPU= 25.204
  01:49:19 STEP 2505 H=2505:00 +CPU= 44.581
  01:49:49 STEP 2506 H=2506:00 +CPU= 30.179
  01:50:19 STEP 2507 H=2507:00 +CPU= 29.736
  01:51:12 STEP 2508 H=2508:00 +CPU= 53.345
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002508
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002508
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002508
  01:51:42 STEP 2509 H=2509:00 +CPU= 29.482
  01:52:09 STEP 2510 H=2510:00 +CPU= 27.548
  01:52:49 STEP 2511 H=2511:00 +CPU= 39.644
  01:53:18 STEP 2512 H=2512:00 +CPU= 28.727
  01:53:47 STEP 2513 H=2513:00 +CPU= 29.606
  01:54:36 STEP 2514 H=2514:00 +CPU= 48.836
  01:55:01 STEP 2515 H=2515:00 +CPU= 24.430
  01:55:26 STEP 2516 H=2516:00 +CPU= 25.542
  01:56:14 STEP 2517 H=2517:00 +CPU= 47.395
  01:56:44 STEP 2518 H=2518:00 +CPU= 30.142
  01:57:15 STEP 2519 H=2519:00 +CPU= 30.961
  01:58:04 STEP 2520 H=2520:00 +CPU= 48.216
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002520
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002520
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179281/ICMGGhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179281/ICMSHhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179281/ICMUAhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179281/ICMGGhq0f+002508
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179281/ICMSHhq0f+002508
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179281/ICMUAhq0f+002508
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0637_1993050100_123_962_12179281_0_r1385864353_104.zip
Uploading the intermediate file: upload_file_104.zip
  01:58:32 STEP 2521 H=2521:00 +CPU= 27.956
  01:59:02 STEP 2522 H=2522:00 +CPU= 30.528
  01:59:44 STEP 2523 H=2523:00 +CPU= 41.841
  02:00:16 STEP 2524 H=2524:00 +CPU= 31.141
  02:00:41 STEP 2525 H=2525:00 +CPU= 25.233
  02:01:27 STEP 2526 H=2526:00 +CPU= 45.682
  02:01:56 STEP 2527 H=2527:00 +CPU= 29.309
Abort request received from BOINC client, ending the child process

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 15:59:40 1467890 22264183 oifs_43r3_ps_0637_1993050100_123_962_12179281_0 7,171,200 63,640 0.0089
09 May 2023 15:59:34 1467890 22264183 oifs_43r3_ps_0637_1993050100_123_962_12179281_0 6,307,200 55,961 0.0089
09 May 2023 15:59:23 1467890 22264183 oifs_43r3_ps_0637_1993050100_123_962_12179281_0 5,443,200 48,291 0.0089
09 May 2023 15:59:17 1467890 22264183 oifs_43r3_ps_0637_1993050100_123_962_12179281_0 4,579,200 40,603 0.0089
09 May 2023 15:58:13 1467890 22264183 oifs_43r3_ps_0637_1993050100_123_962_12179281_0 3,456,000 30,904 0.0089
09 May 2023 15:58:05 1467890 22264183 oifs_43r3_ps_0637_1993050100_123_962_12179281_0 2,592,000 23,094 0.0089
09 May 2023 15:57:57 1467890 22264183 oifs_43r3_ps_0637_1993050100_123_962_12179281_0 1,728,000 15,284 0.0088
09 May 2023 15:57:48 1467890 22264183 oifs_43r3_ps_0637_1993050100_123_962_12179281_0 864,000 7,456 0.0086


©2024 climateprediction.net