climateprediction.net home page
Task 22262446

Task 22262446

Name oifs_43r3_ps_0989_1991050100_123_960_12177633_0
Workunit 12177633
Created 21 Dec 2022, 23:14:44 UTC
Sent 21 Dec 2022, 23:33:32 UTC
Report deadline 20 Jan 2023, 23:33:32 UTC
Received 22 Dec 2022, 23:53:31 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1532237
Run time 20 hours 11 min 42 sec
CPU time 18 hours 41 min 50 sec
Validate state Invalid
Credit 9,797.76
Device peak FLOPS 7.29 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,418.65 MB
Peak swap size 4,978.16 MB
Peak disk usage 7,690.52 MB
Stderr 
<core_client_version>7.16.6</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
U= 19.053
  14:16:27 STEP 2618 H=2618:00 +CPU= 19.447
  14:16:54 STEP 2619 H=2619:00 +CPU= 26.620
  14:17:13 STEP 2620 H=2620:00 +CPU= 18.734
  14:17:32 STEP 2621 H=2621:00 +CPU= 18.978
  14:18:02 STEP 2622 H=2622:00 +CPU= 29.284
  14:18:18 STEP 2623 H=2623:00 +CPU= 16.140
  14:18:37 STEP 2624 H=2624:00 +CPU= 18.930
  14:19:07 STEP 2625 H=2625:00 +CPU= 29.399
  14:19:25 STEP 2626 H=2626:00 +CPU= 17.977
  14:19:43 STEP 2627 H=2627:00 +CPU= 18.305
  14:20:15 STEP 2628 H=2628:00 +CPU= 31.596
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002628
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002628
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002628
  14:20:33 STEP 2629 H=2629:00 +CPU= 17.553
  14:20:51 STEP 2630 H=2630:00 +CPU= 17.249
  14:21:19 STEP 2631 H=2631:00 +CPU= 28.482
  14:21:39 STEP 2632 H=2632:00 +CPU= 19.622
  14:21:58 STEP 2633 H=2633:00 +CPU= 18.249
  14:22:25 STEP 2634 H=2634:00 +CPU= 27.383
  14:22:44 STEP 2635 H=2635:00 +CPU= 18.726
  14:23:03 STEP 2636 H=2636:00 +CPU= 18.387
  14:23:32 STEP 2637 H=2637:00 +CPU= 28.183
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1991050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0989
(argv4) batchid: 960
(argv5) wuid: 12177633
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/10
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177633
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/10/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/10/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_389b3119e473ef444aff8c413a17f421 to: /var/lib/boinc-client/slots/10/oifs_43r3_ps_0989_1991050100_123_960_12177633.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/10/oifs_43r3_ps_0989_1991050100_123_960_12177633.zip
ic_ancil_file: ic_ancil_12177633
ifsdata_file: ifsdata_12177633
climate_data_file: clim_data_12177633
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_dd9c00bfb38c43f6c4ab06ab4871302e to: /var/lib/boinc-client/slots/10/ic_ancil_12177633.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/10/ic_ancil_12177633.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_fd0a06630c5b3ab903ad947d0a9c9d27 to: /var/lib/boinc-client/slots/10/ifsdata/ifsdata_12177633.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/10/ifsdata/ifsdata_12177633.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/10/159l_2/clim_data_12177633.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/10/159l_2/clim_data_12177633.zip
Checking for progress XML file: /var/lib/boinc-client/slots/10/progress_file_12177633.xml
Opened progress file ok : /var/lib/boinc-client/slots/10/progress_file_12177633.xml
-- Model is restarting --
Adjusting last_iter, 2638, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/10/progress_file_12177633.xml
last_cpu_time: 59606
upload_file_number: 109
last_iter: 2615
last_upload: 9417600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0989_1991050100_123_960_12177633_0_r1484119340
The child process has been launched with process id: 320995
Executing the command: /var/lib/boinc-client/slots/10/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:320995:320995] [20221222:142450:1671744290:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:320995:320995] [20221222:142450:1671744290:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:320995:320995] [20221222:142450:1671744290:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:320995:320995] [20221222:142450:1671744290:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:320995:320995] [20221222:142450:1671744290:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:320995:320995] [20221222:142450:1671744290:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:320995:320995] [20221222:142450:1671744290:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:320995:320995] [20221222:142450:1671744290:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:320995:320995] [20221222:142450:1671744290:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:320995:320995] [20221222:142450:1671744290:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:320995:320995] [20221222:142450:1671744290:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:320995:320995] [20221222:142450:1671744290:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:320995:320995] [20221222:142450:1671744290:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:320995:320995] [20221222:142450:1671744290:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:320995:320995] [20221222:142450:1671744290:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:320995:320995] [20221222:142450:1671744290:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:320995:320995] [20221222:142450:1671744290:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:320995:320995] [20221222:142450:1671744290:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:320995:320995] [20221222:142450:1671744290:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:320995:320995] [20221222:142450:1671744290:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:320995:320995] [20221222:142450:1671744290:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:320995:320995] [20221222:142450:1671744290:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:320995:320995] [20221222:142450:1671744290:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:320995:320995] [20221222:142450:1671744290:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:320995:320995] [20221222:142450:1671744290:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:320995:320995] [20221222:142450:1671744290:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:320995:320995] [20221222:142450:1671744290:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:320995:320995] [20221222:142450:1671744290:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:320995:320995] [20221222:142450:1671744290:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:320995:320995] [20221222:142450:1671744290:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  14:25:26 STEP 2616 H=2616:00 +CPU= 32.610
  14:25:43 STEP 2617 H=2617:00 +CPU= 16.800
  14:26:02 STEP 2618 H=2618:00 +CPU= 18.750
  14:26:30 STEP 2619 H=2619:00 +CPU= 27.777
  14:26:50 STEP 2620 H=2620:00 +CPU= 19.117
  14:27:07 STEP 2621 H=2621:00 +CPU= 17.414
  14:27:36 STEP 2622 H=2622:00 +CPU= 28.835
  14:27:54 STEP 2623 H=2623:00 +CPU= 17.196
  14:28:13 STEP 2624 H=2624:00 +CPU= 18.546
  14:28:38 STEP 2625 H=2625:00 +CPU= 24.997
  14:28:56 STEP 2626 H=2626:00 +CPU= 17.346
  14:29:15 STEP 2627 H=2627:00 +CPU= 18.976
  14:29:49 STEP 2628 H=2628:00 +CPU= 33.732
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002628
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002628
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002628
  14:30:07 STEP 2629 H=2629:00 +CPU= 17.918
  14:30:26 STEP 2630 H=2630:00 +CPU= 19.143
  14:30:54 STEP 2631 H=2631:00 +CPU= 27.707
  14:31:13 STEP 2632 H=2632:00 +CPU= 18.662
  14:31:31 STEP 2633 H=2633:00 +CPU= 17.452
  14:32:00 STEP 2634 H=2634:00 +CPU= 28.777
  14:32:19 STEP 2635 H=2635:00 +CPU= 18.738
  14:32:38 STEP 2636 H=2636:00 +CPU= 18.242
  14:33:07 STEP 2637 H=2637:00 +CPU= 28.729
  14:33:23 STEP 2638 H=2638:00 +CPU= 16.272
  14:33:41 STEP 2639 H=2639:00 +CPU= 16.999
  14:34:15 STEP 2640 H=2640:00 +CPU= 33.604
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002640
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002640
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177633/ICMGGhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177633/ICMSHhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177633/ICMUAhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177633/ICMGGhq0f+002628
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177633/ICMSHhq0f+002628
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177633/ICMUAhq0f+002628
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0989_1991050100_123_960_12177633_0_r1484119340_109.zip
Uploading the intermediate file: upload_file_109.zip
  14:34:34 STEP 2641 H=2641:00 +CPU= 19.005
  14:34:52 STEP 2642 H=2642:00 +CPU= 17.565
  14:35:20 STEP 2643 H=2643:00 +CPU= 28.261
  14:35:40 STEP 2644 H=2644:00 +CPU= 18.960
  14:35:57 STEP 2645 H=2645:00 +CPU= 17.320
  14:36:24 STEP 2646 H=2646:00 +CPU= 26.936
  14:36:43 STEP 2647 H=2647:00 +CPU= 18.455
  14:37:04 STEP 2648 H=2648:00 +CPU= 20.273
  14:37:31 STEP 2649 H=2649:00 +CPU= 26.917
  14:37:52 STEP 2650 H=2650:00 +CPU= 20.167
  14:38:09 STEP 2651 H=2651:00 +CPU= 17.019
  14:38:40 STEP 2652 H=2652:00 +CPU= 30.398
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002652
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002652
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002652
  14:38:58 STEP 2653 H=2653:00 +CPU= 18.204
  14:39:17 STEP 2654 H=2654:00 +CPU= 18.771
  14:39:46 STEP 2655 H=2655:00 +CPU= 28.867
  14:40:03 STEP 2656 H=2656:00 +CPU= 16.928
  14:40:21 STEP 2657 H=2657:00 +CPU= 17.250
  14:40:51 STEP 2658 H=2658:00 +CPU= 29.112
  14:41:09 STEP 2659 H=2659:00 +CPU= 17.740
  14:41:27 STEP 2660 H=2660:00 +CPU= 18.083
  14:41:55 STEP 2661 H=2661:00 +CPU= 27.942
  14:42:13 STEP 2662 H=2662:00 +CPU= 17.192
  14:42:32 STEP 2663 H=2663:00 +CPU= 18.972
  14:43:03 STEP 2664 H=2664:00 +CPU= 29.984
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002664
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002664
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177633/ICMGGhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177633/ICMSHhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177633/ICMUAhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177633/ICMGGhq0f+002652
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177633/ICMSHhq0f+002652
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177633/ICMUAhq0f+002652
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0989_1991050100_123_960_12177633_0_r1484119340_110.zip
Uploading the intermediate file: upload_file_110.zip
  14:43:23 STEP 2665 H=2665:00 +CPU= 20.019
  14:43:41 STEP 2666 H=2666:00 +CPU= 18.094
  14:44:09 STEP 2667 H=2667:00 +CPU= 27.314
  14:44:28 STEP 2668 H=2668:00 +CPU= 18.273
  14:44:46 STEP 2669 H=2669:00 +CPU= 18.250
  14:45:15 STEP 2670 H=2670:00 +CPU= 28.330
  14:45:30 STEP 2671 H=2671:00 +CPU= 15.661
  14:45:49 STEP 2672 H=2672:00 +CPU= 17.983
  14:46:19 STEP 2673 H=2673:00 +CPU= 29.881
  14:46:40 STEP 2674 H=2674:00 +CPU= 20.145
  14:46:59 STEP 2675 H=2675:00 +CPU= 18.746
  14:47:31 STEP 2676 H=2676:00 +CPU= 32.050
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002676
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002676
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002676
  14:47:48 STEP 2677 H=2677:00 +CPU= 17.117
  14:48:06 STEP 2678 H=2678:00 +CPU= 17.928
  14:48:36 STEP 2679 H=2679:00 +CPU= 29.150
  14:48:58 STEP 2680 H=2680:00 +CPU= 21.071
  14:49:18 STEP 2681 H=2681:00 +CPU= 19.593
  14:49:47 STEP 2682 H=2682:00 +CPU= 28.840
  14:50:07 STEP 2683 H=2683:00 +CPU= 19.320
  14:50:27 STEP 2684 H=2684:00 +CPU= 20.170
  14:50:57 STEP 2685 H=2685:00 +CPU= 29.290
  14:51:17 STEP 2686 H=2686:00 +CPU= 19.567
  14:51:36 STEP 2687 H=2687:00 +CPU= 19.235
  14:52:11 STEP 2688 H=2688:00 +CPU= 33.499
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002688
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002688
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177633/ICMGGhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177633/ICMSHhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177633/ICMUAhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177633/ICMGGhq0f+002676
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177633/ICMSHhq0f+002676
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177633/ICMUAhq0f+002676
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0989_1991050100_123_960_12177633_0_r1484119340_111.zip
Uploading the intermediate file: upload_file_111.zip
  14:52:28 STEP 2689 H=2689:00 +CPU= 17.171
  14:52:47 STEP 2690 H=2690:00 +CPU= 18.932
  14:53:15 STEP 2691 H=2691:00 +CPU= 27.718
  14:53:37 STEP 2692 H=2692:00 +CPU= 20.776
  14:53:54 STEP 2693 H=2693:00 +CPU= 17.420
  14:54:23 STEP 2694 H=2694:00 +CPU= 28.928
  14:54:43 STEP 2695 H=2695:00 +CPU= 18.929
  14:55:02 STEP 2696 H=2696:00 +CPU= 19.057
  14:55:28 STEP 2697 H=2697:00 +CPU= 26.061
  14:55:48 STEP 2698 H=2698:00 +CPU= 19.488
  14:56:05 STEP 2699 H=2699:00 +CPU= 16.512
  14:56:38 STEP 2700 H=2700:00 +CPU= 32.851
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002700
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002700
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002700
  14:56:58 STEP 2701 H=2701:00 +CPU= 19.872
  14:57:17 STEP 2702 H=2702:00 +CPU= 18.506
  14:57:46 STEP 2703 H=2703:00 +CPU= 28.763
  14:58:04 STEP 2704 H=2704:00 +CPU= 17.235
  14:58:22 STEP 2705 H=2705:00 +CPU= 17.745
  14:58:53 STEP 2706 H=2706:00 +CPU= 30.924
  14:59:13 STEP 2707 H=2707:00 +CPU= 19.124
  14:59:32 STEP 2708 H=2708:00 +CPU= 18.850
  15:00:02 STEP 2709 H=2709:00 +CPU= 29.387
  15:00:20 STEP 2710 H=2710:00 +CPU= 18.336
  15:00:40 STEP 2711 H=2711:00 +CPU= 19.104
  15:01:14 STEP 2712 H=2712:00 +CPU= 33.389
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002712
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002712
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177633/ICMGGhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177633/ICMSHhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177633/ICMUAhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177633/ICMGGhq0f+002700
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177633/ICMSHhq0f+002700
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177633/ICMUAhq0f+002700
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0989_1991050100_123_960_12177633_0_r1484119340_112.zip
Uploading the intermediate file: upload_file_112.zip
  15:01:32 STEP 2713 H=2713:00 +CPU= 17.992
  15:01:50 STEP 2714 H=2714:00 +CPU= 17.690
  15:02:19 STEP 2715 H=2715:00 +CPU= 28.424
  15:02:39 STEP 2716 H=2716:00 +CPU= 20.531
  15:02:56 STEP 2717 H=2717:00 +CPU= 16.509
  15:03:25 STEP 2718 H=2718:00 +CPU= 28.057
  15:03:43 STEP 2719 H=2719:00 +CPU= 17.890
  15:04:02 STEP 2720 H=2720:00 +CPU= 19.507
  15:04:32 STEP 2721 H=2721:00 +CPU= 29.331
  15:04:53 STEP 2722 H=2722:00 +CPU= 20.084
  15:05:12 STEP 2723 H=2723:00 +CPU= 19.177
  15:05:43 STEP 2724 H=2724:00 +CPU= 30.913
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002724
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002724
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002724
  15:06:01 STEP 2725 H=2725:00 +CPU= 17.521
  15:06:19 STEP 2726 H=2726:00 +CPU= 17.787
  15:06:48 STEP 2727 H=2727:00 +CPU= 28.816
  15:07:07 STEP 2728 H=2728:00 +CPU= 18.162
  15:07:26 STEP 2729 H=2729:00 +CPU= 18.569
  15:07:56 STEP 2730 H=2730:00 +CPU= 29.864
  15:08:15 STEP 2731 H=2731:00 +CPU= 18.198
  15:08:32 STEP 2732 H=2732:00 +CPU= 16.981
  15:09:00 STEP 2733 H=2733:00 +CPU= 27.474
  15:09:20 STEP 2734 H=2734:00 +CPU= 19.164
  15:09:38 STEP 2735 H=2735:00 +CPU= 17.938
  15:10:11 STEP 2736 H=2736:00 +CPU= 32.617
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002736
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002736
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177633/ICMGGhq0f+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177633/ICMSHhq0f+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177633/ICMUAhq0f+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177633/ICMGGhq0f+002724
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177633/ICMSHhq0f+002724
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177633/ICMUAhq0f+002724
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0989_1991050100_123_960_12177633_0_r1484119340_113.zip
Uploading the intermediate file: upload_file_113.zip
  15:10:29 STEP 2737 H=2737:00 +CPU= 17.896
  15:10:49 STEP 2738 H=2738:00 +CPU= 19.426
  15:11:18 STEP 2739 H=2739:00 +CPU= 28.929
  15:11:37 STEP 2740 H=2740:00 +CPU= 18.653
  15:11:55 STEP 2741 H=2741:00 +CPU= 18.207
  15:12:21 STEP 2742 H=2742:00 +CPU= 25.230
  15:12:41 STEP 2743 H=2743:00 +CPU= 20.129
  15:13:00 STEP 2744 H=2744:00 +CPU= 18.805
  15:13:31 STEP 2745 H=2745:00 +CPU= 30.086
  15:13:50 STEP 2746 H=2746:00 +CPU= 18.804
  15:14:07 STEP 2747 H=2747:00 +CPU= 16.707
  15:14:38 STEP 2748 H=2748:00 +CPU= 30.714
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002748
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002748
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002748
  15:14:57 STEP 2749 H=2749:00 +CPU= 18.578
  15:15:15 STEP 2750 H=2750:00 +CPU= 17.758
  15:15:42 STEP 2751 H=2751:00 +CPU= 26.622
  15:16:04 STEP 2752 H=2752:00 +CPU= 21.260
  15:16:23 STEP 2753 H=2753:00 +CPU= 18.559
  15:16:53 STEP 2754 H=2754:00 +CPU= 29.914
  15:17:11 STEP 2755 H=2755:00 +CPU= 17.879
  15:17:29 STEP 2756 H=2756:00 +CPU= 17.444
  15:17:58 STEP 2757 H=2757:00 +CPU= 29.084
  15:18:19 STEP 2758 H=2758:00 +CPU= 19.621
  15:18:38 STEP 2759 H=2759:00 +CPU= 19.130
  15:19:12 STEP 2760 H=2760:00 +CPU= 33.469
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002760
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002760
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177633/ICMGGhq0f+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177633/ICMSHhq0f+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177633/ICMUAhq0f+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177633/ICMGGhq0f+002748
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177633/ICMSHhq0f+002748
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177633/ICMUAhq0f+002748
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0989_1991050100_123_960_12177633_0_r1484119340_114.zip
Uploading the intermediate file: upload_file_114.zip
  15:19:31 STEP 2761 H=2761:00 +CPU= 19.151
  15:19:50 STEP 2762 H=2762:00 +CPU= 18.672
  15:20:19 STEP 2763 H=2763:00 +CPU= 28.527
  15:20:39 STEP 2764 H=2764:00 +CPU= 19.558
  15:20:59 STEP 2765 H=2765:00 +CPU= 19.293
  15:21:27 STEP 2766 H=2766:00 +CPU= 28.037
  15:21:45 STEP 2767 H=2767:00 +CPU= 18.311
  15:22:05 STEP 2768 H=2768:00 +CPU= 18.977
  15:22:34 STEP 2769 H=2769:00 +CPU= 28.705
  15:22:55 STEP 2770 H=2770:00 +CPU= 21.034
  15:23:15 STEP 2771 H=2771:00 +CPU= 19.828
  15:23:46 STEP 2772 H=2772:00 +CPU= 30.681
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002772
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002772
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002772
  15:24:04 STEP 2773 H=2773:00 +CPU= 18.072
  15:24:22 STEP 2774 H=2774:00 +CPU= 17.476
  15:24:52 STEP 2775 H=2775:00 +CPU= 29.083
  15:25:11 STEP 2776 H=2776:00 +CPU= 19.406
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1991050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0989
(argv4) batchid: 960
(argv5) wuid: 12177633
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/10
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177633
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/10/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/10/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_389b3119e473ef444aff8c413a17f421 to: /var/lib/boinc-client/slots/10/oifs_43r3_ps_0989_1991050100_123_960_12177633.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/10/oifs_43r3_ps_0989_1991050100_123_960_12177633.zip
ic_ancil_file: ic_ancil_12177633
ifsdata_file: ifsdata_12177633
climate_data_file: clim_data_12177633
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_dd9c00bfb38c43f6c4ab06ab4871302e to: /var/lib/boinc-client/slots/10/ic_ancil_12177633.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/10/ic_ancil_12177633.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_fd0a06630c5b3ab903ad947d0a9c9d27 to: /var/lib/boinc-client/slots/10/ifsdata/ifsdata_12177633.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/10/ifsdata/ifsdata_12177633.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/10/159l_2/clim_data_12177633.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/10/159l_2/clim_data_12177633.zip
Checking for progress XML file: /var/lib/boinc-client/slots/10/progress_file_12177633.xml
Opened progress file ok : /var/lib/boinc-client/slots/10/progress_file_12177633.xml
-- Model is restarting --
Adjusting last_iter, 2776, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/10/progress_file_12177633.xml
last_cpu_time: 63173
upload_file_number: 115
last_iter: 2759
last_upload: 9936000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0989_1991050100_123_960_12177633_0_r1484119340
The child process has been launched with process id: 324005
Executing the command: /var/lib/boinc-client/slots/10/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:324005:324005] [20221222:152615:1671747975:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:324005:324005] [20221222:152615:1671747975:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:324005:324005] [20221222:152615:1671747975:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:324005:324005] [20221222:152615:1671747975:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:324005:324005] [20221222:152615:1671747975:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:324005:324005] [20221222:152615:1671747975:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:324005:324005] [20221222:152615:1671747975:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:324005:324005] [20221222:152615:1671747975:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:324005:324005] [20221222:152615:1671747975:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:324005:324005] [20221222:152615:1671747975:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:324005:324005] [20221222:152615:1671747975:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:324005:324005] [20221222:152615:1671747975:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:324005:324005] [20221222:152615:1671747975:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:324005:324005] [20221222:152615:1671747975:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:324005:324005] [20221222:152615:1671747975:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:324005:324005] [20221222:152615:1671747975:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:324005:324005] [20221222:152615:1671747975:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:324005:324005] [20221222:152615:1671747975:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:324005:324005] [20221222:152615:1671747975:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:324005:324005] [20221222:152615:1671747975:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:324005:324005] [20221222:152615:1671747975:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:324005:324005] [20221222:152615:1671747975:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:324005:324005] [20221222:152615:1671747975:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:324005:324005] [20221222:152615:1671747975:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:324005:324005] [20221222:152615:1671747975:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:324005:324005] [20221222:152615:1671747975:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:324005:324005] [20221222:152615:1671747975:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:324005:324005] [20221222:152615:1671747975:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:324005:324005] [20221222:152615:1671747975:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:324005:324005] [20221222:152615:1671747975:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  15:26:50 STEP 2760 H=2760:00 +CPU= 32.273
  15:27:10 STEP 2761 H=2761:00 +CPU= 19.520
  15:27:29 STEP 2762 H=2762:00 +CPU= 18.377
  15:27:57 STEP 2763 H=2763:00 +CPU= 27.422
  15:28:15 STEP 2764 H=2764:00 +CPU= 18.261
  15:28:34 STEP 2765 H=2765:00 +CPU= 19.380
  15:29:05 STEP 2766 H=2766:00 +CPU= 29.938
  15:29:23 STEP 2767 H=2767:00 +CPU= 17.933
  15:29:43 STEP 2768 H=2768:00 +CPU= 19.952
  15:30:13 STEP 2769 H=2769:00 +CPU= 29.093
  15:30:32 STEP 2770 H=2770:00 +CPU= 18.778
  15:30:50 STEP 2771 H=2771:00 +CPU= 18.242
  15:31:19 STEP 2772 H=2772:00 +CPU= 28.917
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002772
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002772
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002772
  15:31:39 STEP 2773 H=2773:00 +CPU= 18.984
  15:31:59 STEP 2774 H=2774:00 +CPU= 19.597
  15:32:28 STEP 2775 H=2775:00 +CPU= 28.627
  15:32:47 STEP 2776 H=2776:00 +CPU= 18.778
  15:33:07 STEP 2777 H=2777:00 +CPU= 19.503
  15:33:37 STEP 2778 H=2778:00 +CPU= 29.440
  15:33:56 STEP 2779 H=2779:00 +CPU= 18.952
  15:34:15 STEP 2780 H=2780:00 +CPU= 18.498
  15:34:43 STEP 2781 H=2781:00 +CPU= 27.529
  15:35:02 STEP 2782 H=2782:00 +CPU= 18.931
  15:35:23 STEP 2783 H=2783:00 +CPU= 20.280
  15:35:58 STEP 2784 H=2784:00 +CPU= 34.264
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002784
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002784
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177633/ICMGGhq0f+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177633/ICMSHhq0f+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177633/ICMUAhq0f+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177633/ICMGGhq0f+002772
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177633/ICMSHhq0f+002772
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177633/ICMUAhq0f+002772
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0989_1991050100_123_960_12177633_0_r1484119340_115.zip
Uploading the intermediate file: upload_file_115.zip
  15:36:17 STEP 2785 H=2785:00 +CPU= 19.812
  15:36:37 STEP 2786 H=2786:00 +CPU= 19.354
  15:37:06 STEP 2787 H=2787:00 +CPU= 28.244
  15:37:23 STEP 2788 H=2788:00 +CPU= 17.291
  15:37:40 STEP 2789 H=2789:00 +CPU= 17.035
  15:38:11 STEP 2790 H=2790:00 +CPU= 29.779
  15:38:29 STEP 2791 H=2791:00 +CPU= 18.069
  15:38:49 STEP 2792 H=2792:00 +CPU= 19.953
  15:39:20 STEP 2793 H=2793:00 +CPU= 29.826
  15:39:38 STEP 2794 H=2794:00 +CPU= 18.661
  15:39:58 STEP 2795 H=2795:00 +CPU= 19.031
  15:40:30 STEP 2796 H=2796:00 +CPU= 31.256
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002796
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002796
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002796
  15:40:48 STEP 2797 H=2797:00 +CPU= 18.021
  15:41:07 STEP 2798 H=2798:00 +CPU= 18.962
  15:41:37 STEP 2799 H=2799:00 +CPU= 29.299
  15:41:55 STEP 2800 H=2800:00 +CPU= 17.712
  15:42:14 STEP 2801 H=2801:00 +CPU= 18.957
  15:42:44 STEP 2802 H=2802:00 +CPU= 29.350
  15:43:01 STEP 2803 H=2803:00 +CPU= 16.585
  15:43:20 STEP 2804 H=2804:00 +CPU= 18.585
  15:43:50 STEP 2805 H=2805:00 +CPU= 29.316
  15:44:08 STEP 2806 H=2806:00 +CPU= 18.004
  15:44:26 STEP 2807 H=2807:00 +CPU= 18.028
  15:45:01 STEP 2808 H=2808:00 +CPU= 34.163
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002808
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002808
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177633/ICMGGhq0f+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177633/ICMSHhq0f+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177633/ICMUAhq0f+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177633/ICMGGhq0f+002796
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177633/ICMSHhq0f+002796
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177633/ICMUAhq0f+002796
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0989_1991050100_123_960_12177633_0_r1484119340_116.zip
Uploading the intermediate file: upload_file_116.zip
  15:45:21 STEP 2809 H=2809:00 +CPU= 19.596
  15:45:39 STEP 2810 H=2810:00 +CPU= 18.460
  15:46:06 STEP 2811 H=2811:00 +CPU= 26.636
  15:46:26 STEP 2812 H=2812:00 +CPU= 18.996
  15:46:45 STEP 2813 H=2813:00 +CPU= 18.766
  15:47:15 STEP 2814 H=2814:00 +CPU= 30.214
  15:47:34 STEP 2815 H=2815:00 +CPU= 18.228
  15:47:53 STEP 2816 H=2816:00 +CPU= 19.076
  15:48:23 STEP 2817 H=2817:00 +CPU= 29.326
  15:48:41 STEP 2818 H=2818:00 +CPU= 17.770
  15:48:59 STEP 2819 H=2819:00 +CPU= 18.090
  15:49:34 STEP 2820 H=2820:00 +CPU= 34.376
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002820
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002820
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002820
  15:49:51 STEP 2821 H=2821:00 +CPU= 16.997
  15:50:10 STEP 2822 H=2822:00 +CPU= 18.233
  15:50:39 STEP 2823 H=2823:00 +CPU= 28.646
  15:50:57 STEP 2824 H=2824:00 +CPU= 17.429
  15:51:15 STEP 2825 H=2825:00 +CPU= 17.948
  15:51:45 STEP 2826 H=2826:00 +CPU= 29.322
  15:52:02 STEP 2827 H=2827:00 +CPU= 16.915
  15:52:22 STEP 2828 H=2828:00 +CPU= 18.931
  15:52:51 STEP 2829 H=2829:00 +CPU= 29.516
  15:53:12 STEP 2830 H=2830:00 +CPU= 19.603
  15:53:32 STEP 2831 H=2831:00 +CPU= 20.361
  15:54:04 STEP 2832 H=2832:00 +CPU= 31.250
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002832
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002832
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177633/ICMGGhq0f+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177633/ICMSHhq0f+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177633/ICMUAhq0f+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177633/ICMGGhq0f+002820
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177633/ICMSHhq0f+002820
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177633/ICMUAhq0f+002820
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0989_1991050100_123_960_12177633_0_r1484119340_117.zip
Uploading the intermediate file: upload_file_117.zip
  15:54:22 STEP 2833 H=2833:00 +CPU= 17.701
  15:54:39 STEP 2834 H=2834:00 +CPU= 17.074
  15:55:08 STEP 2835 H=2835:00 +CPU= 28.721
  15:55:28 STEP 2836 H=2836:00 +CPU= 19.031
  15:55:48 STEP 2837 H=2837:00 +CPU= 19.867
  15:56:18 STEP 2838 H=2838:00 +CPU= 29.573
  15:56:36 STEP 2839 H=2839:00 +CPU= 18.154
  15:56:54 STEP 2840 H=2840:00 +CPU= 18.128
  15:57:23 STEP 2841 H=2841:00 +CPU= 28.038
  15:57:42 STEP 2842 H=2842:00 +CPU= 19.279
  15:58:03 STEP 2843 H=2843:00 +CPU= 20.136
  15:58:35 STEP 2844 H=2844:00 +CPU= 31.474
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002844
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002844
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002844
  15:58:52 STEP 2845 H=2845:00 +CPU= 17.533
  15:59:13 STEP 2846 H=2846:00 +CPU= 20.318
  15:59:43 STEP 2847 H=2847:00 +CPU= 29.172
  15:59:59 STEP 2848 H=2848:00 +CPU= 16.360
  16:00:16 STEP 2849 H=2849:00 +CPU= 16.094
  16:00:47 STEP 2850 H=2850:00 +CPU= 30.289
  16:01:05 STEP 2851 H=2851:00 +CPU= 18.310
  16:01:23 STEP 2852 H=2852:00 +CPU= 17.457
  16:01:50 STEP 2853 H=2853:00 +CPU= 26.837
  16:02:11 STEP 2854 H=2854:00 +CPU= 20.013
  16:02:32 STEP 2855 H=2855:00 +CPU= 20.911
  16:03:06 STEP 2856 H=2856:00 +CPU= 33.605
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002856
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002856
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002856
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177633/ICMGGhq0f+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177633/ICMSHhq0f+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177633/ICMUAhq0f+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177633/ICMGGhq0f+002844
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177633/ICMSHhq0f+002844
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177633/ICMUAhq0f+002844
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0989_1991050100_123_960_12177633_0_r1484119340_118.zip
Uploading the intermediate file: upload_file_118.zip
  16:03:26 STEP 2857 H=2857:00 +CPU= 19.760
  16:03:46 STEP 2858 H=2858:00 +CPU= 19.224
  16:04:14 STEP 2859 H=2859:00 +CPU= 27.428
  16:04:32 STEP 2860 H=2860:00 +CPU= 17.958
  16:04:50 STEP 2861 H=2861:00 +CPU= 18.170
  16:05:20 STEP 2862 H=2862:00 +CPU= 28.942
  16:05:38 STEP 2863 H=2863:00 +CPU= 18.263
  16:05:58 STEP 2864 H=2864:00 +CPU= 19.924
  16:06:28 STEP 2865 H=2865:00 +CPU= 29.349
  16:06:48 STEP 2866 H=2866:00 +CPU= 19.875
  16:07:06 STEP 2867 H=2867:00 +CPU= 17.910
  16:07:39 STEP 2868 H=2868:00 +CPU= 32.104
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002868
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002868
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002868
  16:07:57 STEP 2869 H=2869:00 +CPU= 18.174
  16:08:14 STEP 2870 H=2870:00 +CPU= 17.038
  16:08:44 STEP 2871 H=2871:00 +CPU= 28.963
  16:09:04 STEP 2872 H=2872:00 +CPU= 19.370
  16:09:24 STEP 2873 H=2873:00 +CPU= 19.690
  16:09:51 STEP 2874 H=2874:00 +CPU= 26.964
  16:10:10 STEP 2875 H=2875:00 +CPU= 18.648
  16:10:29 STEP 2876 H=2876:00 +CPU= 18.404
  16:10:54 STEP 2877 H=2877:00 +CPU= 24.836
  16:11:12 STEP 2878 H=2878:00 +CPU= 18.000
  16:11:32 STEP 2879 H=2879:00 +CPU= 19.552
  16:12:06 STEP 2880 H=2880:00 +CPU= 33.670
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002880
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002880
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002880
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177633/ICMGGhq0f+002856
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177633/ICMSHhq0f+002856
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177633/ICMUAhq0f+002856
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177633/ICMGGhq0f+002868
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177633/ICMSHhq0f+002868
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177633/ICMUAhq0f+002868
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0989_1991050100_123_960_12177633_0_r1484119340_119.zip
Uploading the intermediate file: upload_file_119.zip
  16:12:23 STEP 2881 H=2881:00 +CPU= 16.750
  16:12:43 STEP 2882 H=2882:00 +CPU= 19.495
  16:13:12 STEP 2883 H=2883:00 +CPU= 28.740
  16:13:28 STEP 2884 H=2884:00 +CPU= 15.933
  16:13:47 STEP 2885 H=2885:00 +CPU= 18.640
  16:14:14 STEP 2886 H=2886:00 +CPU= 26.563
  16:14:33 STEP 2887 H=2887:00 +CPU= 18.777
  16:14:53 STEP 2888 H=2888:00 +CPU= 19.210
  16:15:20 STEP 2889 H=2889:00 +CPU= 27.423
  16:15:41 STEP 2890 H=2890:00 +CPU= 20.369
  16:16:01 STEP 2891 H=2891:00 +CPU= 19.797
  16:16:33 STEP 2892 H=2892:00 +CPU= 31.300
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002892
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002892
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002892
  16:16:52 STEP 2893 H=2893:00 +CPU= 18.756
  16:17:12 STEP 2894 H=2894:00 +CPU= 19.292
  16:17:40 STEP 2895 H=2895:00 +CPU= 27.514
  16:18:00 STEP 2896 H=2896:00 +CPU= 19.262
  16:18:18 STEP 2897 H=2897:00 +CPU= 18.581
  16:18:50 STEP 2898 H=2898:00 +CPU= 30.772
  16:19:09 STEP 2899 H=2899:00 +CPU= 19.605
  16:19:30 STEP 2900 H=2900:00 +CPU= 20.557
  16:19:59 STEP 2901 H=2901:00 +CPU= 27.702
  16:20:18 STEP 2902 H=2902:00 +CPU= 19.425
  16:20:39 STEP 2903 H=2903:00 +CPU= 20.254
  16:21:16 STEP 2904 H=2904:00 +CPU= 36.252
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002904
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002904
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002904
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177633/ICMGGhq0f+002880
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177633/ICMSHhq0f+002880
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177633/ICMUAhq0f+002880
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177633/ICMGGhq0f+002892
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177633/ICMSHhq0f+002892
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177633/ICMUAhq0f+002892
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0989_1991050100_123_960_12177633_0_r1484119340_120.zip
Uploading the intermediate file: upload_file_120.zip
  16:21:34 STEP 2905 H=2905:00 +CPU= 18.474
  16:21:52 STEP 2906 H=2906:00 +CPU= 17.607
  16:22:20 STEP 2907 H=2907:00 +CPU= 27.568
  16:22:39 STEP 2908 H=2908:00 +CPU= 19.076
  16:22:59 STEP 2909 H=2909:00 +CPU= 19.141
  16:23:28 STEP 2910 H=2910:00 +CPU= 28.704
  16:23:48 STEP 2911 H=2911:00 +CPU= 19.707
  16:24:07 STEP 2912 H=2912:00 +CPU= 19.192
  16:24:37 STEP 2913 H=2913:00 +CPU= 29.388
  16:24:58 STEP 2914 H=2914:00 +CPU= 20.764
  16:25:21 STEP 2915 H=2915:00 +CPU= 19.619
  16:25:51 STEP 2916 H=2916:00 +CPU= 29.697
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002916
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002916
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002916
  16:26:11 STEP 2917 H=2917:00 +CPU= 19.315
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1991050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0989
(argv4) batchid: 960
(argv5) wuid: 12177633
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/10
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177633
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/10/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/10/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_389b3119e473ef444aff8c413a17f421 to: /var/lib/boinc-client/slots/10/oifs_43r3_ps_0989_1991050100_123_960_12177633.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/10/oifs_43r3_ps_0989_1991050100_123_960_12177633.zip
ic_ancil_file: ic_ancil_12177633
ifsdata_file: ifsdata_12177633
climate_data_file: clim_data_12177633
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_dd9c00bfb38c43f6c4ab06ab4871302e to: /var/lib/boinc-client/slots/10/ic_ancil_12177633.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/10/ic_ancil_12177633.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_fd0a06630c5b3ab903ad947d0a9c9d27 to: /var/lib/boinc-client/slots/10/ifsdata/ifsdata_12177633.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/10/ifsdata/ifsdata_12177633.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/10/159l_2/clim_data_12177633.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/10/159l_2/clim_data_12177633.zip
Checking for progress XML file: /var/lib/boinc-client/slots/10/progress_file_12177633.xml
Opened progress file ok : /var/lib/boinc-client/slots/10/progress_file_12177633.xml
-- Model is restarting --
Adjusting last_iter, 2917, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/10/progress_file_12177633.xml
last_cpu_time: 66708
upload_file_number: 121
last_iter: 2903
last_upload: 10454400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0989_1991050100_123_960_12177633_0_r1484119340
The child process has been launched with process id: 327074
Executing the command: /var/lib/boinc-client/slots/10/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:327074:327074] [20221222:162718:1671751638:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:327074:327074] [20221222:162718:1671751638:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:327074:327074] [20221222:162718:1671751638:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:327074:327074] [20221222:162718:1671751638:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:327074:327074] [20221222:162718:1671751638:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:327074:327074] [20221222:162718:1671751638:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:327074:327074] [20221222:162718:1671751638:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:327074:327074] [20221222:162718:1671751638:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:327074:327074] [20221222:162718:1671751638:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:327074:327074] [20221222:162718:1671751638:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:327074:327074] [20221222:162718:1671751638:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:327074:327074] [20221222:162718:1671751638:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:327074:327074] [20221222:162718:1671751638:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:327074:327074] [20221222:162718:1671751638:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:327074:327074] [20221222:162718:1671751638:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:327074:327074] [20221222:162718:1671751638:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:327074:327074] [20221222:162718:1671751638:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:327074:327074] [20221222:162718:1671751638:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:327074:327074] [20221222:162718:1671751638:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:327074:327074] [20221222:162718:1671751638:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:327074:327074] [20221222:162718:1671751638:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:327074:327074] [20221222:162718:1671751638:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:327074:327074] [20221222:162718:1671751638:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:327074:327074] [20221222:162718:1671751638:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:327074:327074] [20221222:162718:1671751638:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:327074:327074] [20221222:162718:1671751638:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:327074:327074] [20221222:162718:1671751638:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:327074:327074] [20221222:162718:1671751638:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:327074:327074] [20221222:162718:1671751638:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:327074:327074] [20221222:162718:1671751638:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  16:27:54 STEP 2904 H=2904:00 +CPU= 33.343
  16:28:14 STEP 2905 H=2905:00 +CPU= 19.090
  16:28:32 STEP 2906 H=2906:00 +CPU= 18.260
  16:29:00 STEP 2907 H=2907:00 +CPU= 27.412
  16:29:19 STEP 2908 H=2908:00 +CPU= 18.897
  16:29:39 STEP 2909 H=2909:00 +CPU= 19.406
  16:30:08 STEP 2910 H=2910:00 +CPU= 29.295
  16:30:28 STEP 2911 H=2911:00 +CPU= 19.246
  16:30:47 STEP 2912 H=2912:00 +CPU= 18.387
  16:31:16 STEP 2913 H=2913:00 +CPU= 28.305
  16:31:33 STEP 2914 H=2914:00 +CPU= 17.742
  16:31:53 STEP 2915 H=2915:00 +CPU= 19.406
  16:32:24 STEP 2916 H=2916:00 +CPU= 30.275
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002916
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002916
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002916
  16:32:42 STEP 2917 H=2917:00 +CPU= 17.882
  16:33:02 STEP 2918 H=2918:00 +CPU= 19.435
  16:33:30 STEP 2919 H=2919:00 +CPU= 28.016
  16:33:49 STEP 2920 H=2920:00 +CPU= 18.616
  16:34:09 STEP 2921 H=2921:00 +CPU= 19.095
  16:34:38 STEP 2922 H=2922:00 +CPU= 29.099
  16:34:56 STEP 2923 H=2923:00 +CPU= 17.622
  16:35:16 STEP 2924 H=2924:00 +CPU= 19.200
  16:35:46 STEP 2925 H=2925:00 +CPU= 30.086
  16:36:04 STEP 2926 H=2926:00 +CPU= 17.713
  16:36:23 STEP 2927 H=2927:00 +CPU= 18.849
  16:36:59 STEP 2928 H=2928:00 +CPU= 35.416
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002928
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002928
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002928
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177633/ICMGGhq0f+002904
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177633/ICMSHhq0f+002904
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177633/ICMUAhq0f+002904
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177633/ICMGGhq0f+002916
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177633/ICMSHhq0f+002916
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177633/ICMUAhq0f+002916
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0989_1991050100_123_960_12177633_0_r1484119340_121.zip
Uploading the intermediate file: upload_file_121.zip
  16:37:17 STEP 2929 H=2929:00 +CPU= 17.545
Abort request received from BOINC client, ending the child process

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 15:48:22 1532237 22262446 oifs_43r3_ps_0989_1991050100_123_960_12177633_0 6,998,400 44,033 0.0063
09 May 2023 15:48:17 1532237 22262446 oifs_43r3_ps_0989_1991050100_123_960_12177633_0 6,134,400 40,333 0.0066
09 May 2023 15:48:12 1532237 22262446 oifs_43r3_ps_0989_1991050100_123_960_12177633_0 5,270,400 36,435 0.0069
09 May 2023 15:48:07 1532237 22262446 oifs_43r3_ps_0989_1991050100_123_960_12177633_0 4,406,400 32,538 0.0074
09 May 2023 15:48:02 1532237 22262446 oifs_43r3_ps_0989_1991050100_123_960_12177633_0 3,542,400 28,404 0.0080
09 May 2023 15:47:56 1532237 22262446 oifs_43r3_ps_0989_1991050100_123_960_12177633_0 2,678,400 24,231 0.0090
09 May 2023 15:47:51 1532237 22262446 oifs_43r3_ps_0989_1991050100_123_960_12177633_0 1,814,400 20,054 0.0111
09 May 2023 15:47:40 1532237 22262446 oifs_43r3_ps_0989_1991050100_123_960_12177633_0 950,400 15,871 0.0167


©2024 climateprediction.net