climateprediction.net home page
Task 22286368

Task 22286368

Name oifs_43r3_ps_0509_2014050100_123_983_12200153_0
Workunit 12200153
Created 24 Dec 2022, 10:21:48 UTC
Sent 5 Jan 2023, 4:12:19 UTC
Report deadline 4 Feb 2023, 4:12:19 UTC
Received 6 Jan 2023, 3:00:16 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1496322
Run time 22 hours 36 min 43 sec
CPU time 22 hours 15 min 13 sec
Validate state Invalid
Credit 11,612.16
Device peak FLOPS 4.33 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,416.96 MB
Peak swap size 4,978.07 MB
Peak disk usage 7,672.57 MB
Stderr 
<core_client_version>7.16.6</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
U= 23.646
  02:01:25 STEP 2393 H=2393:00 +CPU= 22.865
  02:02:02 STEP 2394 H=2394:00 +CPU= 36.955
  02:02:25 STEP 2395 H=2395:00 +CPU= 23.005
  02:02:49 STEP 2396 H=2396:00 +CPU= 23.376
  02:03:25 STEP 2397 H=2397:00 +CPU= 36.029
  02:03:49 STEP 2398 H=2398:00 +CPU= 23.480
  02:04:12 STEP 2399 H=2399:00 +CPU= 23.364
  02:04:55 STEP 2400 H=2400:00 +CPU= 42.292
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002400
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002400
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002400
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200153/ICMGGhq0f+002376
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200153/ICMSHhq0f+002376
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200153/ICMUAhq0f+002376
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200153/ICMGGhq0f+002388
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200153/ICMSHhq0f+002388
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200153/ICMUAhq0f+002388
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0509_2014050100_123_983_12200153_0_r743208141_99.zip
Uploading the intermediate file: upload_file_99.zip
  02:05:18 STEP 2401 H=2401:00 +CPU= 23.249
  02:05:41 STEP 2402 H=2402:00 +CPU= 23.424
  02:06:18 STEP 2403 H=2403:00 +CPU= 36.247
  02:06:42 STEP 2404 H=2404:00 +CPU= 23.622
  02:07:04 STEP 2405 H=2405:00 +CPU= 22.635
  02:07:41 STEP 2406 H=2406:00 +CPU= 36.889
  02:08:04 STEP 2407 H=2407:00 +CPU= 22.855
  02:08:28 STEP 2408 H=2408:00 +CPU= 23.363
  02:09:04 STEP 2409 H=2409:00 +CPU= 36.138
  02:09:28 STEP 2410 H=2410:00 +CPU= 23.622
  02:09:50 STEP 2411 H=2411:00 +CPU= 22.689
  02:10:31 STEP 2412 H=2412:00 +CPU= 40.165
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002412
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002412
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002412
  02:10:54 STEP 2413 H=2413:00 +CPU= 23.371
  02:11:18 STEP 2414 H=2414:00 +CPU= 23.629
  02:11:54 STEP 2415 H=2415:00 +CPU= 36.418
  02:12:18 STEP 2416 H=2416:00 +CPU= 23.864
  02:12:41 STEP 2417 H=2417:00 +CPU= 22.794
  02:13:19 STEP 2418 H=2418:00 +CPU= 37.201
  02:13:42 STEP 2419 H=2419:00 +CPU= 23.059
  02:14:05 STEP 2420 H=2420:00 +CPU= 23.534
  02:14:42 STEP 2421 H=2421:00 +CPU= 36.252
  02:15:06 STEP 2422 H=2422:00 +CPU= 23.675
  02:15:29 STEP 2423 H=2423:00 +CPU= 23.577
  02:16:12 STEP 2424 H=2424:00 +CPU= 42.631
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002424
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002424
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002424
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200153/ICMGGhq0f+002400
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200153/ICMSHhq0f+002400
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200153/ICMUAhq0f+002400
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200153/ICMGGhq0f+002412
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200153/ICMSHhq0f+002412
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200153/ICMUAhq0f+002412
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0509_2014050100_123_983_12200153_0_r743208141_100.zip
Uploading the intermediate file: upload_file_100.zip
  02:16:35 STEP 2425 H=2425:00 +CPU= 23.338
  02:16:59 STEP 2426 H=2426:00 +CPU= 23.420
  02:17:35 STEP 2427 H=2427:00 +CPU= 35.409
  02:17:57 STEP 2428 H=2428:00 +CPU= 22.129
  02:18:18 STEP 2429 H=2429:00 +CPU= 21.499
  02:18:55 STEP 2430 H=2430:00 +CPU= 36.887
  02:19:19 STEP 2431 H=2431:00 +CPU= 23.154
  02:19:43 STEP 2432 H=2432:00 +CPU= 23.976
  02:20:19 STEP 2433 H=2433:00 +CPU= 36.487
  02:20:43 STEP 2434 H=2434:00 +CPU= 24.049
  02:21:07 STEP 2435 H=2435:00 +CPU= 23.333
  02:21:47 STEP 2436 H=2436:00 +CPU= 40.230
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002436
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002436
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002436
  02:22:10 STEP 2437 H=2437:00 +CPU= 23.235
  02:22:34 STEP 2438 H=2438:00 +CPU= 23.861
  02:23:11 STEP 2439 H=2439:00 +CPU= 36.336
  02:23:35 STEP 2440 H=2440:00 +CPU= 23.845
  02:23:58 STEP 2441 H=2441:00 +CPU= 23.049
  02:24:35 STEP 2442 H=2442:00 +CPU= 37.187
  02:24:59 STEP 2443 H=2443:00 +CPU= 23.151
  02:25:22 STEP 2444 H=2444:00 +CPU= 23.853
  02:25:59 STEP 2445 H=2445:00 +CPU= 36.463
  02:26:23 STEP 2446 H=2446:00 +CPU= 23.885
  02:26:47 STEP 2447 H=2447:00 +CPU= 23.918
  02:27:30 STEP 2448 H=2448:00 +CPU= 42.724
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002448
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002448
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002448
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200153/ICMGGhq0f+002424
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200153/ICMSHhq0f+002424
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200153/ICMUAhq0f+002424
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200153/ICMGGhq0f+002436
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200153/ICMSHhq0f+002436
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200153/ICMUAhq0f+002436
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0509_2014050100_123_983_12200153_0_r743208141_101.zip
Uploading the intermediate file: upload_file_101.zip
  02:27:54 STEP 2449 H=2449:00 +CPU= 23.442
  02:28:17 STEP 2450 H=2450:00 +CPU= 23.703
  02:28:54 STEP 2451 H=2451:00 +CPU= 36.346
  02:29:18 STEP 2452 H=2452:00 +CPU= 23.968
  02:29:41 STEP 2453 H=2453:00 +CPU= 23.249
  02:30:19 STEP 2454 H=2454:00 +CPU= 37.259
  02:30:42 STEP 2455 H=2455:00 +CPU= 23.405
  02:31:06 STEP 2456 H=2456:00 +CPU= 24.032
  02:31:43 STEP 2457 H=2457:00 +CPU= 36.521
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2014050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0509
(argv4) batchid: 983
(argv5) wuid: 12200153
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/1
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200153
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_38d986791ed2f1996f0373e647684d40 to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0509_2014050100_123_983_12200153.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0509_2014050100_123_983_12200153.zip
ic_ancil_file: ic_ancil_12200153
ifsdata_file: ifsdata_12200153
climate_data_file: clim_data_12200153
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_0ee9f5c9322d2ee30e3755006a729c6e to: /var/lib/boinc-client/slots/1/ic_ancil_12200153.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/1/ic_ancil_12200153.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_1dc0d26401b5c901e0e650fe2c5991c8 to: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12200153.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12200153.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/1/159l_2/clim_data_12200153.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/1/159l_2/clim_data_12200153.zip
Checking for progress XML file: /var/lib/boinc-client/slots/1/progress_file_12200153.xml
Opened progress file ok : /var/lib/boinc-client/slots/1/progress_file_12200153.xml
-- Model is restarting --
Adjusting last_iter, 2458, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/1/progress_file_12200153.xml
last_cpu_time: 71586
upload_file_number: 102
last_iter: 2447
last_upload: 8812800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0509_2014050100_123_983_12200153_0_r743208141
The child process has been launched with process id: 3989040
Executing the command: /var/lib/boinc-client/slots/1/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:Particle:1:1:3989040:3989040] [20230106:023304:1672965184:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:Particle:1:1:3989040:3989040] [20230106:023304:1672965184:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:Particle:1:1:3989040:3989040] [20230106:023304:1672965184:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:Particle:1:1:3989040:3989040] [20230106:023304:1672965184:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:Particle:1:1:3989040:3989040] [20230106:023304:1672965184:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:Particle:1:1:3989040:3989040] [20230106:023304:1672965184:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:Particle:1:1:3989040:3989040] [20230106:023304:1672965184:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:Particle:1:1:3989040:3989040] [20230106:023304:1672965184:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:Particle:1:1:3989040:3989040] [20230106:023304:1672965184:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:Particle:1:1:3989040:3989040] [20230106:023304:1672965184:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:Particle:1:1:3989040:3989040] [20230106:023304:1672965184:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:Particle:1:1:3989040:3989040] [20230106:023304:1672965184:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:Particle:1:1:3989040:3989040] [20230106:023304:1672965184:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:Particle:1:1:3989040:3989040] [20230106:023304:1672965184:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:Particle:1:1:3989040:3989040] [20230106:023304:1672965184:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:Particle:1:1:3989040:3989040] [20230106:023304:1672965184:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Particle:1:1:3989040:3989040] [20230106:023304:1672965184:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Particle:1:1:3989040:3989040] [20230106:023304:1672965184:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Particle:1:1:3989040:3989040] [20230106:023304:1672965184:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:Particle:1:1:3989040:3989040] [20230106:023304:1672965184:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:Particle:1:1:3989040:3989040] [20230106:023304:1672965184:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Particle:1:1:3989040:3989040] [20230106:023304:1672965184:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Particle:1:1:3989040:3989040] [20230106:023304:1672965184:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:Particle:1:1:3989040:3989040] [20230106:023304:1672965184:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Particle:1:1:3989040:3989040] [20230106:023304:1672965184:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Particle:1:1:3989040:3989040] [20230106:023304:1672965184:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Particle:1:1:3989040:3989040] [20230106:023304:1672965184:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:Particle:1:1:3989040:3989040] [20230106:023304:1672965184:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:Particle:1:1:3989040:3989040] [20230106:023304:1672965184:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Particle:1:1:3989040:3989040] [20230106:023304:1672965184:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  02:33:48 STEP 2448 H=2448:00 +CPU= 41.394
  02:34:11 STEP 2449 H=2449:00 +CPU= 22.937
  02:34:35 STEP 2450 H=2450:00 +CPU= 23.687
  02:35:12 STEP 2451 H=2451:00 +CPU= 36.552
  02:35:35 STEP 2452 H=2452:00 +CPU= 23.748
  02:35:58 STEP 2453 H=2453:00 +CPU= 22.723
  02:36:36 STEP 2454 H=2454:00 +CPU= 37.484
  02:36:59 STEP 2455 H=2455:00 +CPU= 23.022
  02:37:23 STEP 2456 H=2456:00 +CPU= 23.658
  02:37:59 STEP 2457 H=2457:00 +CPU= 36.356
  02:38:23 STEP 2458 H=2458:00 +CPU= 24.087
  02:38:46 STEP 2459 H=2459:00 +CPU= 22.888
  02:39:27 STEP 2460 H=2460:00 +CPU= 40.198
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002460
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002460
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002460
  02:39:50 STEP 2461 H=2461:00 +CPU= 23.289
  02:40:14 STEP 2462 H=2462:00 +CPU= 23.772
  02:40:51 STEP 2463 H=2463:00 +CPU= 36.583
  02:41:15 STEP 2464 H=2464:00 +CPU= 23.890
  02:41:38 STEP 2465 H=2465:00 +CPU= 22.872
  02:42:16 STEP 2466 H=2466:00 +CPU= 37.830
  02:42:39 STEP 2467 H=2467:00 +CPU= 23.087
  02:43:03 STEP 2468 H=2468:00 +CPU= 23.406
  02:43:39 STEP 2469 H=2469:00 +CPU= 36.561
  02:44:03 STEP 2470 H=2470:00 +CPU= 23.889
  02:44:27 STEP 2471 H=2471:00 +CPU= 23.540
  02:45:10 STEP 2472 H=2472:00 +CPU= 42.721
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002472
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002472
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200153/ICMGGhq0f+002448
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200153/ICMSHhq0f+002448
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200153/ICMUAhq0f+002448
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200153/ICMGGhq0f+002460
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200153/ICMSHhq0f+002460
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200153/ICMUAhq0f+002460
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0509_2014050100_123_983_12200153_0_r743208141_102.zip
Uploading the intermediate file: upload_file_102.zip
  02:45:33 STEP 2473 H=2473:00 +CPU= 23.161
  02:45:57 STEP 2474 H=2474:00 +CPU= 23.629
  02:46:33 STEP 2475 H=2475:00 +CPU= 36.618
  02:46:57 STEP 2476 H=2476:00 +CPU= 23.763
  02:47:20 STEP 2477 H=2477:00 +CPU= 22.967
  02:47:58 STEP 2478 H=2478:00 +CPU= 37.165
  02:48:21 STEP 2479 H=2479:00 +CPU= 23.163
  02:48:45 STEP 2480 H=2480:00 +CPU= 23.688
  02:49:21 STEP 2481 H=2481:00 +CPU= 36.657
  02:49:45 STEP 2482 H=2482:00 +CPU= 23.782
  02:50:08 STEP 2483 H=2483:00 +CPU= 22.941
  02:50:49 STEP 2484 H=2484:00 +CPU= 40.482
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002484
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002484
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002484
  02:51:12 STEP 2485 H=2485:00 +CPU= 23.199
  02:51:36 STEP 2486 H=2486:00 +CPU= 23.880
  02:52:13 STEP 2487 H=2487:00 +CPU= 36.518
  02:52:37 STEP 2488 H=2488:00 +CPU= 23.957
  02:53:00 STEP 2489 H=2489:00 +CPU= 22.915
  02:53:38 STEP 2490 H=2490:00 +CPU= 37.712
  02:54:01 STEP 2491 H=2491:00 +CPU= 23.202
  02:54:25 STEP 2492 H=2492:00 +CPU= 23.489
  02:55:02 STEP 2493 H=2493:00 +CPU= 36.673
  02:55:25 STEP 2494 H=2494:00 +CPU= 23.775
  02:55:49 STEP 2495 H=2495:00 +CPU= 23.658
  02:56:32 STEP 2496 H=2496:00 +CPU= 42.733
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002496
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002496
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200153/ICMGGhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200153/ICMSHhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200153/ICMUAhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200153/ICMGGhq0f+002484
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200153/ICMSHhq0f+002484
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200153/ICMUAhq0f+002484
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0509_2014050100_123_983_12200153_0_r743208141_103.zip
Uploading the intermediate file: upload_file_103.zip
  02:56:55 STEP 2497 H=2497:00 +CPU= 23.286
  02:57:19 STEP 2498 H=2498:00 +CPU= 23.806
  02:57:56 STEP 2499 H=2499:00 +CPU= 36.689
  02:58:20 STEP 2500 H=2500:00 +CPU= 23.851
  02:58:43 STEP 2501 H=2501:00 +CPU= 22.951
  02:59:21 STEP 2502 H=2502:00 +CPU= 37.519
  02:59:44 STEP 2503 H=2503:00 +CPU= 23.118
  03:00:08 STEP 2504 H=2504:00 +CPU= 23.670
  03:00:45 STEP 2505 H=2505:00 +CPU= 36.621
  03:01:09 STEP 2506 H=2506:00 +CPU= 23.901
  03:01:32 STEP 2507 H=2507:00 +CPU= 23.221
  03:02:13 STEP 2508 H=2508:00 +CPU= 40.444
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002508
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002508
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002508
  03:02:36 STEP 2509 H=2509:00 +CPU= 23.087
  03:03:00 STEP 2510 H=2510:00 +CPU= 23.730
  03:03:36 STEP 2511 H=2511:00 +CPU= 36.666
  03:04:00 STEP 2512 H=2512:00 +CPU= 23.802
  03:04:23 STEP 2513 H=2513:00 +CPU= 22.874
  03:05:01 STEP 2514 H=2514:00 +CPU= 37.740
  03:05:25 STEP 2515 H=2515:00 +CPU= 23.388
  03:05:48 STEP 2516 H=2516:00 +CPU= 23.457
  03:06:25 STEP 2517 H=2517:00 +CPU= 36.622
  03:06:49 STEP 2518 H=2518:00 +CPU= 23.922
  03:07:13 STEP 2519 H=2519:00 +CPU= 23.604
  03:07:55 STEP 2520 H=2520:00 +CPU= 42.498
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002520
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002520
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200153/ICMGGhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200153/ICMSHhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200153/ICMUAhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200153/ICMGGhq0f+002508
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200153/ICMSHhq0f+002508
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200153/ICMUAhq0f+002508
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0509_2014050100_123_983_12200153_0_r743208141_104.zip
Uploading the intermediate file: upload_file_104.zip
  03:08:19 STEP 2521 H=2521:00 +CPU= 23.282
  03:08:42 STEP 2522 H=2522:00 +CPU= 23.625
  03:09:19 STEP 2523 H=2523:00 +CPU= 36.647
  03:09:43 STEP 2524 H=2524:00 +CPU= 23.779
  03:10:06 STEP 2525 H=2525:00 +CPU= 22.827
  03:10:43 STEP 2526 H=2526:00 +CPU= 37.432
  03:11:07 STEP 2527 H=2527:00 +CPU= 23.094
  03:11:31 STEP 2528 H=2528:00 +CPU= 23.815
  03:12:07 STEP 2529 H=2529:00 +CPU= 36.727
  03:12:31 STEP 2530 H=2530:00 +CPU= 23.872
  03:12:54 STEP 2531 H=2531:00 +CPU= 22.856
  03:13:35 STEP 2532 H=2532:00 +CPU= 40.417
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002532
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002532
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002532
  03:13:58 STEP 2533 H=2533:00 +CPU= 22.986
  03:14:22 STEP 2534 H=2534:00 +CPU= 23.621
  03:14:59 STEP 2535 H=2535:00 +CPU= 36.688
  03:15:22 STEP 2536 H=2536:00 +CPU= 23.857
  03:15:45 STEP 2537 H=2537:00 +CPU= 22.803
  03:16:23 STEP 2538 H=2538:00 +CPU= 37.479
  03:16:46 STEP 2539 H=2539:00 +CPU= 23.152
  03:17:10 STEP 2540 H=2540:00 +CPU= 23.469
  03:17:47 STEP 2541 H=2541:00 +CPU= 36.620
  03:18:11 STEP 2542 H=2542:00 +CPU= 23.950
  03:18:34 STEP 2543 H=2543:00 +CPU= 23.551
  03:19:17 STEP 2544 H=2544:00 +CPU= 42.531
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002544
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002544
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200153/ICMGGhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200153/ICMSHhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200153/ICMUAhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200153/ICMGGhq0f+002532
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200153/ICMSHhq0f+002532
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200153/ICMUAhq0f+002532
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0509_2014050100_123_983_12200153_0_r743208141_105.zip
Uploading the intermediate file: upload_file_105.zip
  03:19:39 STEP 2545 H=2545:00 +CPU= 21.898
  03:20:01 STEP 2546 H=2546:00 +CPU= 21.981
  03:20:36 STEP 2547 H=2547:00 +CPU= 34.748
  03:20:59 STEP 2548 H=2548:00 +CPU= 23.149
  03:21:22 STEP 2549 H=2549:00 +CPU= 22.949
  03:21:59 STEP 2550 H=2550:00 +CPU= 36.893
  03:22:22 STEP 2551 H=2551:00 +CPU= 22.508
  03:22:45 STEP 2552 H=2552:00 +CPU= 23.501
  03:23:22 STEP 2553 H=2553:00 +CPU= 36.240
  03:23:45 STEP 2554 H=2554:00 +CPU= 23.199
  03:24:08 STEP 2555 H=2555:00 +CPU= 22.823
  03:24:48 STEP 2556 H=2556:00 +CPU= 40.249
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002556
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002556
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002556
  03:25:12 STEP 2557 H=2557:00 +CPU= 22.981
  03:25:36 STEP 2558 H=2558:00 +CPU= 23.721
  03:26:12 STEP 2559 H=2559:00 +CPU= 36.380
  03:26:35 STEP 2560 H=2560:00 +CPU= 23.168
  03:26:58 STEP 2561 H=2561:00 +CPU= 22.862
  03:27:36 STEP 2562 H=2562:00 +CPU= 37.173
  03:27:58 STEP 2563 H=2563:00 +CPU= 22.446
  03:28:22 STEP 2564 H=2564:00 +CPU= 23.581
  03:28:58 STEP 2565 H=2565:00 +CPU= 36.015
  03:29:21 STEP 2566 H=2566:00 +CPU= 23.295
  03:29:45 STEP 2567 H=2567:00 +CPU= 23.678
  03:30:27 STEP 2568 H=2568:00 +CPU= 41.948
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002568
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002568
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200153/ICMGGhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200153/ICMSHhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200153/ICMUAhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200153/ICMGGhq0f+002556
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200153/ICMSHhq0f+002556
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200153/ICMUAhq0f+002556
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0509_2014050100_123_983_12200153_0_r743208141_106.zip
Uploading the intermediate file: upload_file_106.zip
  03:30:50 STEP 2569 H=2569:00 +CPU= 22.832
  03:31:14 STEP 2570 H=2570:00 +CPU= 23.552
  03:31:50 STEP 2571 H=2571:00 +CPU= 36.135
  03:32:14 STEP 2572 H=2572:00 +CPU= 23.453
  03:32:37 STEP 2573 H=2573:00 +CPU= 23.068
  03:33:14 STEP 2574 H=2574:00 +CPU= 37.269
  03:33:37 STEP 2575 H=2575:00 +CPU= 22.756
  03:34:01 STEP 2576 H=2576:00 +CPU= 23.601
  03:34:37 STEP 2577 H=2577:00 +CPU= 36.335
  03:35:01 STEP 2578 H=2578:00 +CPU= 23.423
  03:35:24 STEP 2579 H=2579:00 +CPU= 22.956
  03:36:04 STEP 2580 H=2580:00 +CPU= 40.176
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002580
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002580
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002580
  03:36:27 STEP 2581 H=2581:00 +CPU= 22.746
  03:36:51 STEP 2582 H=2582:00 +CPU= 23.616
  03:37:27 STEP 2583 H=2583:00 +CPU= 36.273
  03:37:50 STEP 2584 H=2584:00 +CPU= 23.358
  03:38:13 STEP 2585 H=2585:00 +CPU= 22.876
  03:38:50 STEP 2586 H=2586:00 +CPU= 36.924
  03:39:13 STEP 2587 H=2587:00 +CPU= 22.750
  03:39:37 STEP 2588 H=2588:00 +CPU= 23.620
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2014050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0509
(argv4) batchid: 983
(argv5) wuid: 12200153
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/1
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200153
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_38d986791ed2f1996f0373e647684d40 to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0509_2014050100_123_983_12200153.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0509_2014050100_123_983_12200153.zip
ic_ancil_file: ic_ancil_12200153
ifsdata_file: ifsdata_12200153
climate_data_file: clim_data_12200153
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_0ee9f5c9322d2ee30e3755006a729c6e to: /var/lib/boinc-client/slots/1/ic_ancil_12200153.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/1/ic_ancil_12200153.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_1dc0d26401b5c901e0e650fe2c5991c8 to: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12200153.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12200153.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/1/159l_2/clim_data_12200153.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/1/159l_2/clim_data_12200153.zip
Checking for progress XML file: /var/lib/boinc-client/slots/1/progress_file_12200153.xml
Opened progress file ok : /var/lib/boinc-client/slots/1/progress_file_12200153.xml
-- Model is restarting --
Adjusting last_iter, 2589, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/1/progress_file_12200153.xml
last_cpu_time: 75571
upload_file_number: 107
last_iter: 2567
last_upload: 9244800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0509_2014050100_123_983_12200153_0_r743208141
The child process has been launched with process id: 3997583
Executing the command: /var/lib/boinc-client/slots/1/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:Particle:1:1:3997583:3997583] [20230106:034100:1672969260:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:Particle:1:1:3997583:3997583] [20230106:034100:1672969260:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:Particle:1:1:3997583:3997583] [20230106:034100:1672969260:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:Particle:1:1:3997583:3997583] [20230106:034100:1672969260:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:Particle:1:1:3997583:3997583] [20230106:034100:1672969260:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:Particle:1:1:3997583:3997583] [20230106:034100:1672969260:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:Particle:1:1:3997583:3997583] [20230106:034100:1672969260:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:Particle:1:1:3997583:3997583] [20230106:034100:1672969260:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:Particle:1:1:3997583:3997583] [20230106:034100:1672969260:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:Particle:1:1:3997583:3997583] [20230106:034100:1672969260:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:Particle:1:1:3997583:3997583] [20230106:034100:1672969260:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:Particle:1:1:3997583:3997583] [20230106:034100:1672969260:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:Particle:1:1:3997583:3997583] [20230106:034100:1672969260:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:Particle:1:1:3997583:3997583] [20230106:034100:1672969260:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:Particle:1:1:3997583:3997583] [20230106:034100:1672969260:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:Particle:1:1:3997583:3997583] [20230106:034100:1672969260:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Particle:1:1:3997583:3997583] [20230106:034100:1672969260:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Particle:1:1:3997583:3997583] [20230106:034100:1672969260:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Particle:1:1:3997583:3997583] [20230106:034100:1672969260:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:Particle:1:1:3997583:3997583] [20230106:034100:1672969260:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:Particle:1:1:3997583:3997583] [20230106:034100:1672969260:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Particle:1:1:3997583:3997583] [20230106:034100:1672969260:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Particle:1:1:3997583:3997583] [20230106:034100:1672969260:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:Particle:1:1:3997583:3997583] [20230106:034100:1672969260:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Particle:1:1:3997583:3997583] [20230106:034100:1672969260:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Particle:1:1:3997583:3997583] [20230106:034100:1672969260:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Particle:1:1:3997583:3997583] [20230106:034100:1672969260:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:Particle:1:1:3997583:3997583] [20230106:034100:1672969260:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:Particle:1:1:3997583:3997583] [20230106:034100:1672969260:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Particle:1:1:3997583:3997583] [20230106:034100:1672969260:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  03:41:44 STEP 2568 H=2568:00 +CPU= 41.001
  03:42:06 STEP 2569 H=2569:00 +CPU= 22.677
  03:42:30 STEP 2570 H=2570:00 +CPU= 23.485
  03:43:06 STEP 2571 H=2571:00 +CPU= 36.361
  03:43:30 STEP 2572 H=2572:00 +CPU= 23.147
  03:43:53 STEP 2573 H=2573:00 +CPU= 22.956
  03:44:30 STEP 2574 H=2574:00 +CPU= 37.065
  03:44:53 STEP 2575 H=2575:00 +CPU= 22.563
  03:45:16 STEP 2576 H=2576:00 +CPU= 23.581
  03:45:53 STEP 2577 H=2577:00 +CPU= 36.222
  03:46:16 STEP 2578 H=2578:00 +CPU= 23.394
  03:46:39 STEP 2579 H=2579:00 +CPU= 22.926
  03:47:19 STEP 2580 H=2580:00 +CPU= 40.047
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002580
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002580
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002580
  03:47:42 STEP 2581 H=2581:00 +CPU= 22.687
  03:48:06 STEP 2582 H=2582:00 +CPU= 23.462
  03:48:42 STEP 2583 H=2583:00 +CPU= 36.197
  03:49:05 STEP 2584 H=2584:00 +CPU= 23.308
  03:49:28 STEP 2585 H=2585:00 +CPU= 22.916
  03:50:06 STEP 2586 H=2586:00 +CPU= 37.134
  03:50:28 STEP 2587 H=2587:00 +CPU= 22.586
  03:50:52 STEP 2588 H=2588:00 +CPU= 23.544
  03:51:28 STEP 2589 H=2589:00 +CPU= 36.272
  03:51:52 STEP 2590 H=2590:00 +CPU= 23.275
  03:52:15 STEP 2591 H=2591:00 +CPU= 23.596
  03:52:58 STEP 2592 H=2592:00 +CPU= 42.247
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002592
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002592
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200153/ICMGGhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200153/ICMSHhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200153/ICMUAhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200153/ICMGGhq0f+002580
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200153/ICMSHhq0f+002580
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200153/ICMUAhq0f+002580
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0509_2014050100_123_983_12200153_0_r743208141_107.zip
Uploading the intermediate file: upload_file_107.zip
  03:53:21 STEP 2593 H=2593:00 +CPU= 22.944
  03:53:44 STEP 2594 H=2594:00 +CPU= 23.654
  03:54:21 STEP 2595 H=2595:00 +CPU= 36.275
  03:54:44 STEP 2596 H=2596:00 +CPU= 23.386
  03:55:08 STEP 2597 H=2597:00 +CPU= 23.271
  03:55:45 STEP 2598 H=2598:00 +CPU= 37.183
  03:56:08 STEP 2599 H=2599:00 +CPU= 22.759
  03:56:32 STEP 2600 H=2600:00 +CPU= 23.557
  03:57:08 STEP 2601 H=2601:00 +CPU= 36.265
  03:57:32 STEP 2602 H=2602:00 +CPU= 23.572
  03:57:54 STEP 2603 H=2603:00 +CPU= 22.801
  03:58:35 STEP 2604 H=2604:00 +CPU= 39.973
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002604
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002604
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002604
  03:58:57 STEP 2605 H=2605:00 +CPU= 22.683
  03:59:21 STEP 2606 H=2606:00 +CPU= 23.536
  03:59:57 STEP 2607 H=2607:00 +CPU= 36.120
  04:00:21 STEP 2608 H=2608:00 +CPU= 23.422
  04:00:44 STEP 2609 H=2609:00 +CPU= 22.829
  04:01:21 STEP 2610 H=2610:00 +CPU= 36.838
  04:01:43 STEP 2611 H=2611:00 +CPU= 22.550
  04:02:07 STEP 2612 H=2612:00 +CPU= 23.523
  04:02:43 STEP 2613 H=2613:00 +CPU= 36.423
  04:03:07 STEP 2614 H=2614:00 +CPU= 23.190
  04:03:30 STEP 2615 H=2615:00 +CPU= 23.600
  04:04:13 STEP 2616 H=2616:00 +CPU= 42.267
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002616
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002616
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200153/ICMGGhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200153/ICMSHhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200153/ICMUAhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200153/ICMGGhq0f+002604
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200153/ICMSHhq0f+002604
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200153/ICMUAhq0f+002604
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0509_2014050100_123_983_12200153_0_r743208141_108.zip
Uploading the intermediate file: upload_file_108.zip
  04:04:36 STEP 2617 H=2617:00 +CPU= 22.995
  04:05:00 STEP 2618 H=2618:00 +CPU= 23.484
  04:05:36 STEP 2619 H=2619:00 +CPU= 36.355
  04:05:59 STEP 2620 H=2620:00 +CPU= 23.265
  04:06:22 STEP 2621 H=2621:00 +CPU= 22.985
  04:07:00 STEP 2622 H=2622:00 +CPU= 36.969
  04:07:22 STEP 2623 H=2623:00 +CPU= 22.625
  04:07:46 STEP 2624 H=2624:00 +CPU= 23.505
  04:08:22 STEP 2625 H=2625:00 +CPU= 36.295
  04:08:46 STEP 2626 H=2626:00 +CPU= 23.376
  04:09:09 STEP 2627 H=2627:00 +CPU= 22.889
  04:09:49 STEP 2628 H=2628:00 +CPU= 40.162
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002628
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002628
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002628
  04:10:12 STEP 2629 H=2629:00 +CPU= 22.838
  04:10:36 STEP 2630 H=2630:00 +CPU= 23.573
  04:11:12 STEP 2631 H=2631:00 +CPU= 36.260
  04:11:36 STEP 2632 H=2632:00 +CPU= 23.412
  04:11:59 STEP 2633 H=2633:00 +CPU= 22.860
  04:12:36 STEP 2634 H=2634:00 +CPU= 36.873
  04:12:58 STEP 2635 H=2635:00 +CPU= 22.687
  04:13:22 STEP 2636 H=2636:00 +CPU= 23.497
  04:13:58 STEP 2637 H=2637:00 +CPU= 36.035
  04:14:21 STEP 2638 H=2638:00 +CPU= 23.306
  04:14:45 STEP 2639 H=2639:00 +CPU= 23.672
  04:15:27 STEP 2640 H=2640:00 +CPU= 42.149
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002640
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002640
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200153/ICMGGhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200153/ICMSHhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200153/ICMUAhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200153/ICMGGhq0f+002628
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200153/ICMSHhq0f+002628
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200153/ICMUAhq0f+002628
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0509_2014050100_123_983_12200153_0_r743208141_109.zip
Uploading the intermediate file: upload_file_109.zip
  04:15:50 STEP 2641 H=2641:00 +CPU= 22.929
  04:16:14 STEP 2642 H=2642:00 +CPU= 23.595
  04:16:50 STEP 2643 H=2643:00 +CPU= 36.205
  04:17:14 STEP 2644 H=2644:00 +CPU= 23.458
  04:17:37 STEP 2645 H=2645:00 +CPU= 23.127
  04:18:14 STEP 2646 H=2646:00 +CPU= 36.956
  04:18:37 STEP 2647 H=2647:00 +CPU= 22.886
  04:19:01 STEP 2648 H=2648:00 +CPU= 23.502
  04:19:37 STEP 2649 H=2649:00 +CPU= 36.192
  04:20:01 STEP 2650 H=2650:00 +CPU= 23.543
  04:20:24 STEP 2651 H=2651:00 +CPU= 22.955
  04:21:04 STEP 2652 H=2652:00 +CPU= 39.775
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002652
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002652
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002652
  04:21:27 STEP 2653 H=2653:00 +CPU= 22.872
  04:21:50 STEP 2654 H=2654:00 +CPU= 23.425
  04:22:27 STEP 2655 H=2655:00 +CPU= 36.166
  04:22:50 STEP 2656 H=2656:00 +CPU= 23.460
  04:23:13 STEP 2657 H=2657:00 +CPU= 22.910
  04:23:50 STEP 2658 H=2658:00 +CPU= 36.859
  04:24:13 STEP 2659 H=2659:00 +CPU= 22.659
  04:24:37 STEP 2660 H=2660:00 +CPU= 23.641
  04:25:13 STEP 2661 H=2661:00 +CPU= 36.142
  04:25:36 STEP 2662 H=2662:00 +CPU= 23.361
  04:26:00 STEP 2663 H=2663:00 +CPU= 23.540
  04:26:42 STEP 2664 H=2664:00 +CPU= 42.060
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002664
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002664
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200153/ICMGGhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200153/ICMSHhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200153/ICMUAhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200153/ICMGGhq0f+002652
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200153/ICMSHhq0f+002652
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200153/ICMUAhq0f+002652
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0509_2014050100_123_983_12200153_0_r743208141_110.zip
Uploading the intermediate file: upload_file_110.zip
  04:27:05 STEP 2665 H=2665:00 +CPU= 23.009
  04:27:29 STEP 2666 H=2666:00 +CPU= 23.810
  04:28:10 STEP 2667 H=2667:00 +CPU= 40.449
  04:28:34 STEP 2668 H=2668:00 +CPU= 23.492
  04:29:00 STEP 2669 H=2669:00 +CPU= 26.354
  04:29:35 STEP 2670 H=2670:00 +CPU= 35.223
  04:29:58 STEP 2671 H=2671:00 +CPU= 22.389
  04:30:21 STEP 2672 H=2672:00 +CPU= 23.420
  04:30:58 STEP 2673 H=2673:00 +CPU= 36.244
  04:31:22 STEP 2674 H=2674:00 +CPU= 23.625
  04:31:44 STEP 2675 H=2675:00 +CPU= 22.692
  04:32:25 STEP 2676 H=2676:00 +CPU= 40.350
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002676
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002676
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002676
  04:32:48 STEP 2677 H=2677:00 +CPU= 22.849
  04:33:11 STEP 2678 H=2678:00 +CPU= 23.476
  04:33:48 STEP 2679 H=2679:00 +CPU= 36.366
  04:34:12 STEP 2680 H=2680:00 +CPU= 23.494
  04:34:34 STEP 2681 H=2681:00 +CPU= 22.714
  04:35:12 STEP 2682 H=2682:00 +CPU= 37.097
  04:35:35 STEP 2683 H=2683:00 +CPU= 22.939
  04:35:58 STEP 2684 H=2684:00 +CPU= 23.252
  04:36:34 STEP 2685 H=2685:00 +CPU= 36.427
  04:36:58 STEP 2686 H=2686:00 +CPU= 23.576
  04:37:22 STEP 2687 H=2687:00 +CPU= 23.424
  04:38:04 STEP 2688 H=2688:00 +CPU= 42.710
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002688
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002688
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200153/ICMGGhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200153/ICMSHhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200153/ICMUAhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200153/ICMGGhq0f+002676
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200153/ICMSHhq0f+002676
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200153/ICMUAhq0f+002676
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0509_2014050100_123_983_12200153_0_r743208141_111.zip
Uploading the intermediate file: upload_file_111.zip
  04:38:28 STEP 2689 H=2689:00 +CPU= 23.204
  04:38:51 STEP 2690 H=2690:00 +CPU= 23.492
  04:39:28 STEP 2691 H=2691:00 +CPU= 36.374
  04:39:51 STEP 2692 H=2692:00 +CPU= 23.476
  04:40:14 STEP 2693 H=2693:00 +CPU= 22.848
  04:40:52 STEP 2694 H=2694:00 +CPU= 37.220
  04:41:15 STEP 2695 H=2695:00 +CPU= 22.931
  04:41:38 STEP 2696 H=2696:00 +CPU= 23.349
  04:42:15 STEP 2697 H=2697:00 +CPU= 36.717
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2014050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0509
(argv4) batchid: 983
(argv5) wuid: 12200153
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/1
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200153
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_38d986791ed2f1996f0373e647684d40 to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0509_2014050100_123_983_12200153.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0509_2014050100_123_983_12200153.zip
ic_ancil_file: ic_ancil_12200153
ifsdata_file: ifsdata_12200153
climate_data_file: clim_data_12200153
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_0ee9f5c9322d2ee30e3755006a729c6e to: /var/lib/boinc-client/slots/1/ic_ancil_12200153.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/1/ic_ancil_12200153.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_1dc0d26401b5c901e0e650fe2c5991c8 to: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12200153.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12200153.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/1/159l_2/clim_data_12200153.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/1/159l_2/clim_data_12200153.zip
Checking for progress XML file: /var/lib/boinc-client/slots/1/progress_file_12200153.xml
Opened progress file ok : /var/lib/boinc-client/slots/1/progress_file_12200153.xml
-- Model is restarting --
Adjusting last_iter, 2697, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/1/progress_file_12200153.xml
last_cpu_time: 79229
upload_file_number: 112
last_iter: 2687
last_upload: 9676800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0509_2014050100_123_983_12200153_0_r743208141
The child process has been launched with process id: 4006692
Executing the command: /var/lib/boinc-client/slots/1/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:Particle:1:1:4006692:4006692] [20230106:044326:1672973006:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:Particle:1:1:4006692:4006692] [20230106:044326:1672973006:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:Particle:1:1:4006692:4006692] [20230106:044326:1672973006:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:Particle:1:1:4006692:4006692] [20230106:044326:1672973006:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:Particle:1:1:4006692:4006692] [20230106:044326:1672973006:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:Particle:1:1:4006692:4006692] [20230106:044326:1672973006:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:Particle:1:1:4006692:4006692] [20230106:044326:1672973006:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:Particle:1:1:4006692:4006692] [20230106:044326:1672973006:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:Particle:1:1:4006692:4006692] [20230106:044326:1672973006:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:Particle:1:1:4006692:4006692] [20230106:044326:1672973006:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:Particle:1:1:4006692:4006692] [20230106:044326:1672973006:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:Particle:1:1:4006692:4006692] [20230106:044326:1672973006:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:Particle:1:1:4006692:4006692] [20230106:044326:1672973006:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:Particle:1:1:4006692:4006692] [20230106:044326:1672973006:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:Particle:1:1:4006692:4006692] [20230106:044326:1672973006:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:Particle:1:1:4006692:4006692] [20230106:044326:1672973006:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Particle:1:1:4006692:4006692] [20230106:044326:1672973006:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Particle:1:1:4006692:4006692] [20230106:044326:1672973006:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Particle:1:1:4006692:4006692] [20230106:044326:1672973006:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:Particle:1:1:4006692:4006692] [20230106:044326:1672973006:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:Particle:1:1:4006692:4006692] [20230106:044326:1672973006:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Particle:1:1:4006692:4006692] [20230106:044326:1672973006:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Particle:1:1:4006692:4006692] [20230106:044326:1672973006:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:Particle:1:1:4006692:4006692] [20230106:044326:1672973006:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Particle:1:1:4006692:4006692] [20230106:044326:1672973006:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Particle:1:1:4006692:4006692] [20230106:044326:1672973006:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Particle:1:1:4006692:4006692] [20230106:044326:1672973006:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:Particle:1:1:4006692:4006692] [20230106:044326:1672973006:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:Particle:1:1:4006692:4006692] [20230106:044326:1672973006:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Particle:1:1:4006692:4006692] [20230106:044326:1672973006:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  04:44:10 STEP 2688 H=2688:00 +CPU= 40.722
  04:44:33 STEP 2689 H=2689:00 +CPU= 22.845
  04:44:56 STEP 2690 H=2690:00 +CPU= 23.462
  04:45:32 STEP 2691 H=2691:00 +CPU= 35.987
  04:45:56 STEP 2692 H=2692:00 +CPU= 23.495
  04:46:19 STEP 2693 H=2693:00 +CPU= 22.889
  04:46:56 STEP 2694 H=2694:00 +CPU= 36.891
  04:47:19 STEP 2695 H=2695:00 +CPU= 23.028
  04:47:43 STEP 2696 H=2696:00 +CPU= 23.638
  04:48:19 STEP 2697 H=2697:00 +CPU= 35.928
  04:48:42 STEP 2698 H=2698:00 +CPU= 23.446
  04:49:05 STEP 2699 H=2699:00 +CPU= 23.008
  04:49:45 STEP 2700 H=2700:00 +CPU= 39.861
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002700
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002700
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002700
  04:50:08 STEP 2701 H=2701:00 +CPU= 22.827
  04:50:32 STEP 2702 H=2702:00 +CPU= 23.500
  04:51:08 STEP 2703 H=2703:00 +CPU= 36.113
  04:51:32 STEP 2704 H=2704:00 +CPU= 23.467
  04:51:55 STEP 2705 H=2705:00 +CPU= 22.870
  04:52:32 STEP 2706 H=2706:00 +CPU= 36.910
  04:52:55 STEP 2707 H=2707:00 +CPU= 22.852
  04:53:19 STEP 2708 H=2708:00 +CPU= 23.831
  04:53:55 STEP 2709 H=2709:00 +CPU= 36.139
  04:54:18 STEP 2710 H=2710:00 +CPU= 23.413
  04:54:42 STEP 2711 H=2711:00 +CPU= 23.582
  04:55:24 STEP 2712 H=2712:00 +CPU= 41.924
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002712
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002712
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200153/ICMGGhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200153/ICMSHhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200153/ICMUAhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200153/ICMGGhq0f+002700
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200153/ICMSHhq0f+002700
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200153/ICMUAhq0f+002700
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0509_2014050100_123_983_12200153_0_r743208141_112.zip
Uploading the intermediate file: upload_file_112.zip
  04:55:47 STEP 2713 H=2713:00 +CPU= 23.183
  04:56:11 STEP 2714 H=2714:00 +CPU= 23.585
  04:56:47 STEP 2715 H=2715:00 +CPU= 35.983
  04:57:11 STEP 2716 H=2716:00 +CPU= 23.559
  04:57:34 STEP 2717 H=2717:00 +CPU= 22.935
  04:58:11 STEP 2718 H=2718:00 +CPU= 36.851
Abort request received from BOINC client, ending the child process
..Failed, model did not complete successfully

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:09:38 1496322 22286368 oifs_43r3_ps_0509_2014050100_123_983_12200153_0 8,294,400 67,287 0.0081
09 May 2023 16:09:24 1496322 22286368 oifs_43r3_ps_0509_2014050100_123_983_12200153_0 6,652,800 53,927 0.0081
09 May 2023 16:09:17 1496322 22286368 oifs_43r3_ps_0509_2014050100_123_983_12200153_0 5,788,800 47,114 0.0081
09 May 2023 16:09:10 1496322 22286368 oifs_43r3_ps_0509_2014050100_123_983_12200153_0 4,924,800 40,430 0.0082
09 May 2023 16:09:03 1496322 22286368 oifs_43r3_ps_0509_2014050100_123_983_12200153_0 4,060,800 33,767 0.0083
09 May 2023 16:08:57 1496322 22286368 oifs_43r3_ps_0509_2014050100_123_983_12200153_0 3,196,800 27,104 0.0085
09 May 2023 16:08:49 1496322 22286368 oifs_43r3_ps_0509_2014050100_123_983_12200153_0 2,332,800 20,256 0.0087
09 May 2023 16:08:36 1496322 22286368 oifs_43r3_ps_0509_2014050100_123_983_12200153_0 864,000 6,784 0.0079


©2024 climateprediction.net