<core_client_version>7.16.6</core_client_version>
<![CDATA[
<message>
aborted by user</message>
<stderr_txt>
b
(argv4) batchid: 991
(argv5) wuid: 12206834
(argv6) fclen: 15
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/23
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.11
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206834
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/23/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/23/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_82383272ecd2a6c68ef40d3b5dd703af to: /var/lib/boinc-client/slots/23/oifs_43r3_bl_a01b_2016092300_15_991_12206834.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/23/oifs_43r3_bl_a01b_2016092300_15_991_12206834.zip
ic_ancil_file: ic_ancil_12206834
ifsdata_file: ifsdata_12206834
climate_data_file: clim_data_12206834
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /var/lib/boinc-client/slots/23/ic_ancil_12206834.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/23/ic_ancil_12206834.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_f122ec64654a145d9fccb1c0419998b3 to: /var/lib/boinc-client/slots/23/ifsdata/ifsdata_12206834.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/23/ifsdata/ifsdata_12206834.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/23/159l_2/clim_data_12206834.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/23/159l_2/clim_data_12206834.zip
Checking for progress XML file: /var/lib/boinc-client/slots/23/progress_file_12206834.xml
Opened progress file ok : /var/lib/boinc-client/slots/23/progress_file_12206834.xml
-- Model is restarting --
Adjusting last_iter, 741, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/23/progress_file_12206834.xml
last_cpu_time: 43889
upload_file_number: 7
last_iter: 721
last_upload: 605700
model_completed: 0
total_length_of_simulation: 1296000
result_base_name: oifs_43r3_bl_a01b_2016092300_15_991_12206834_0_r2029125939
The child process has been launched with process id: 3880
Executing the command: /var/lib/boinc-client/slots/23/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:chonky:1:1:3880:3880] [20230214:142212:1676384532:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2091] fp = 0x30e4a00
[EC_DRHOOK:chonky:1:1:3880:3880] [20230214:142212:1676384532:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:chonky:1:1:3880:3880] [20230214:142212:1676384532:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:chonky:1:1:3880:3880] [20230214:142212:1676384532:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:chonky:1:1:3880:3880] [20230214:142212:1676384532:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:chonky:1:1:3880:3880] [20230214:142212:1676384532:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:chonky:1:1:3880:3880] [20230214:142212:1676384532:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:chonky:1:1:3880:3880] [20230214:142212:1676384532:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:chonky:1:1:3880:3880] [20230214:142212:1676384532:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:chonky:1:1:3880:3880] [20230214:142212:1676384532:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:chonky:1:1:3880:3880] [20230214:142212:1676384532:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:chonky:1:1:3880:3880] [20230214:142212:1676384532:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:chonky:1:1:3880:3880] [20230214:142212:1676384532:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:chonky:1:1:3880:3880] [20230214:142212:1676384532:0.000] [ignore_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:chonky:1:1:3880:3880] [20230214:142212:1676384532:0.001] [restore_default_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:chonky:1:1:3880:3880] [20230214:142212:1676384532:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:chonky:1:1:3880:3880] [20230214:142212:1676384532:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f520 (old at (nil))
[EC_DRHOOK:chonky:1:1:3880:3880] [20230214:142212:1676384532:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:chonky:1:1:3880:3880] [20230214:142212:1676384532:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f520 (old at (nil))
[EC_DRHOOK:chonky:1:1:3880:3880] [20230214:142212:1676384532:0.001] [trapfpe_treatment@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:chonky:1:1:3880:3880] [20230214:142212:1676384532:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:chonky:1:1:3880:3880] [20230214:142212:1676384532:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:chonky:1:1:3880:3880] [20230214:142212:1676384532:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f520 (old at (nil))
[EC_DRHOOK:chonky:1:1:3880:3880] [20230214:142212:1676384532:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:chonky:1:1:3880:3880] [20230214:142212:1676384532:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:chonky:1:1:3880:3880] [20230214:142212:1676384532:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:chonky:1:1:3880:3880] [20230214:142212:1676384532:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f520 (old at (nil))
[EC_DRHOOK:chonky:1:1:3880:3880] [20230214:142212:1676384532:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f520 (old at (nil))
[EC_DRHOOK:chonky:1:1:3880:3880] [20230214:142212:1676384532:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f520 (old at (nil))
[EC_DRHOOK:chonky:1:1:3880:3880] [20230214:142212:1676384532:0.001] [catch_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
14:24:40 STEP 720 H= 180:00 +CPU= 57.860
14:25:28 STEP 721 H= 180:15 +CPU= 48.401
14:26:19 STEP 722 H= 180:30 +CPU= 50.418
14:27:11 STEP 723 H= 180:45 +CPU= 50.651
14:28:03 STEP 724 H= 181:00 +CPU= 52.147
14:28:58 STEP 725 H= 181:15 +CPU= 53.709
14:29:52 STEP 726 H= 181:30 +CPU= 53.535
14:30:43 STEP 727 H= 181:45 +CPU= 50.732
14:31:35 STEP 728 H= 182:00 +CPU= 51.167
14:32:27 STEP 729 H= 182:15 +CPU= 52.198
14:33:20 STEP 730 H= 182:30 +CPU= 51.953
14:34:09 STEP 731 H= 182:45 +CPU= 48.691
14:35:15 STEP 732 H= 183:00 +CPU= 65.029
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGh7zg+000732
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHh7zg+000732
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAh7zg+000732
14:36:20 STEP 733 H= 183:15 +CPU= 52.015
14:37:13 STEP 734 H= 183:30 +CPU= 52.380
14:38:03 STEP 735 H= 183:45 +CPU= 49.449
14:38:56 STEP 736 H= 184:00 +CPU= 52.846
14:39:45 STEP 737 H= 184:15 +CPU= 49.036
14:40:34 STEP 738 H= 184:30 +CPU= 48.462
14:41:25 STEP 739 H= 184:45 +CPU= 51.063
14:42:17 STEP 740 H= 185:00 +CPU= 52.165
14:43:08 STEP 741 H= 185:15 +CPU= 50.655
14:43:59 STEP 742 H= 185:30 +CPU= 50.405
14:44:50 STEP 743 H= 185:45 +CPU= 51.191
14:45:55 STEP 744 H= 186:00 +CPU= 64.295
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGh7zg+000744
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHh7zg+000744
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAh7zg+000744
14:46:47 STEP 745 H= 186:15 +CPU= 51.954
14:47:40 STEP 746 H= 186:30 +CPU= 52.747
14:48:29 STEP 747 H= 186:45 +CPU= 49.490
14:49:21 STEP 748 H= 187:00 +CPU= 51.933
14:50:14 STEP 749 H= 187:15 +CPU= 52.627
14:51:03 STEP 750 H= 187:30 +CPU= 48.824
14:51:50 STEP 751 H= 187:45 +CPU= 46.422
14:52:41 STEP 752 H= 188:00 +CPU= 50.386
14:53:31 STEP 753 H= 188:15 +CPU= 49.756
14:54:19 STEP 754 H= 188:30 +CPU= 48.247
14:55:07 STEP 755 H= 188:45 +CPU= 47.523
14:56:07 STEP 756 H= 189:00 +CPU= 60.098
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGh7zg+000756
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHh7zg+000756
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAh7zg+000756
14:56:56 STEP 757 H= 189:15 +CPU= 48.516
14:57:46 STEP 758 H= 189:30 +CPU= 50.313
14:58:35 STEP 759 H= 189:45 +CPU= 48.315
14:59:27 STEP 760 H= 190:00 +CPU= 52.002
15:00:15 STEP 761 H= 190:15 +CPU= 48.236
15:01:07 STEP 762 H= 190:30 +CPU= 51.029
15:01:57 STEP 763 H= 190:45 +CPU= 49.573
15:02:49 STEP 764 H= 191:00 +CPU= 51.917
15:03:39 STEP 765 H= 191:15 +CPU= 49.994
15:04:28 STEP 766 H= 191:30 +CPU= 48.933
15:05:18 STEP 767 H= 191:45 +CPU= 50.074
15:06:32 STEP 768 H= 192:00 +CPU= 67.979
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGh7zg+000768
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHh7zg+000768
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAh7zg+000768
15:07:33 STEP 769 H= 192:15 +CPU= 49.172
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206834/ICMGGh7zg+000684
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206834/ICMSHh7zg+000684
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206834/ICMUAh7zg+000684
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206834/ICMGGh7zg+000696
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206834/ICMSHh7zg+000696
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206834/ICMUAh7zg+000696
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206834/ICMGGh7zg+000708
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206834/ICMSHh7zg+000708
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206834/ICMUAh7zg+000708
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206834/ICMGGh7zg+000720
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206834/ICMSHh7zg+000720
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206834/ICMUAh7zg+000720
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206834/ICMGGh7zg+000732
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206834/ICMSHh7zg+000732
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206834/ICMUAh7zg+000732
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206834/ICMGGh7zg+000744
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206834/ICMSHh7zg+000744
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206834/ICMUAh7zg+000744
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206834/ICMGGh7zg+000756
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206834/ICMSHh7zg+000756
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206834/ICMUAh7zg+000756
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206834/ICMGGh7zg+000768
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206834/ICMSHh7zg+000768
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206834/ICMUAh7zg+000768
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a01b_2016092300_15_991_12206834_0_r2029125939_7.zip
Uploading the intermediate file: upload_file_7.zip
15:08:21 STEP 770 H= 192:30 +CPU= 48.106
15:09:11 STEP 771 H= 192:45 +CPU= 49.705
15:10:00 STEP 772 H= 193:00 +CPU= 48.533
15:10:49 STEP 773 H= 193:15 +CPU= 48.588
15:11:38 STEP 774 H= 193:30 +CPU= 48.261
15:12:29 STEP 775 H= 193:45 +CPU= 50.913
15:13:23 STEP 776 H= 194:00 +CPU= 52.996
15:14:12 STEP 777 H= 194:15 +CPU= 48.958
15:15:02 STEP 778 H= 194:30 +CPU= 50.471
15:15:54 STEP 779 H= 194:45 +CPU= 51.703
15:16:59 STEP 780 H= 195:00 +CPU= 64.786
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGh7zg+000780
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHh7zg+000780
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAh7zg+000780
15:17:52 STEP 781 H= 195:15 +CPU= 52.131
15:18:44 STEP 782 H= 195:30 +CPU= 52.009
15:19:35 STEP 783 H= 195:45 +CPU= 50.617
15:20:26 STEP 784 H= 196:00 +CPU= 50.616
15:21:14 STEP 785 H= 196:15 +CPU= 47.707
15:22:02 STEP 786 H= 196:30 +CPU= 47.929
15:22:52 STEP 787 H= 196:45 +CPU= 49.522
15:23:43 STEP 788 H= 197:00 +CPU= 51.156
15:24:32 STEP 789 H= 197:15 +CPU= 48.359
15:25:22 STEP 790 H= 197:30 +CPU= 50.512
15:26:12 STEP 791 H= 197:45 +CPU= 49.888
15:27:14 STEP 792 H= 198:00 +CPU= 61.453
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGh7zg+000792
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHh7zg+000792
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAh7zg+000792
15:28:03 STEP 793 H= 198:15 +CPU= 48.724
15:28:52 STEP 794 H= 198:30 +CPU= 48.740
15:29:41 STEP 795 H= 198:45 +CPU= 48.960
15:30:33 STEP 796 H= 199:00 +CPU= 51.256
15:31:25 STEP 797 H= 199:15 +CPU= 51.840
15:32:16 STEP 798 H= 199:30 +CPU= 51.165
15:33:06 STEP 799 H= 199:45 +CPU= 49.462
15:34:00 STEP 800 H= 200:00 +CPU= 53.700
15:34:48 STEP 801 H= 200:15 +CPU= 48.498
15:35:38 STEP 802 H= 200:30 +CPU= 49.170
15:36:26 STEP 803 H= 200:45 +CPU= 48.632
15:37:31 STEP 804 H= 201:00 +CPU= 64.854
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGh7zg+000804
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHh7zg+000804
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAh7zg+000804
15:38:21 STEP 805 H= 201:15 +CPU= 48.968
15:39:10 STEP 806 H= 201:30 +CPU= 48.900
15:39:59 STEP 807 H= 201:45 +CPU= 48.952
15:40:50 STEP 808 H= 202:00 +CPU= 51.567
15:41:40 STEP 809 H= 202:15 +CPU= 49.523
15:42:30 STEP 810 H= 202:30 +CPU= 49.944
15:43:19 STEP 811 H= 202:45 +CPU= 49.227
15:44:11 STEP 812 H= 203:00 +CPU= 51.521
15:45:00 STEP 813 H= 203:15 +CPU= 48.656
15:45:52 STEP 814 H= 203:30 +CPU= 51.574
[EC_DRHOOK:chonky:1:1:3880:3880] [20230214:154604:1676389564:5031.765] [signal_drhook@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 5397MB (heap), 4884MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:chonky:1:1:3880:3880] [20230214:154604:1676389564:5031.765] [signal_drhook@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:chonky:1:1:3880:3880] [20230214:154604:1676389564:5031.765] [signal_drhook@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f0c0 (old at (nil))
[EC_DRHOOK:chonky:1:1:3880:3880] [20230214:154604:1676389564:5031.765] [signal_drhook@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1
[EC_DRHOOK:chonky:1:1:3880:3880] [20230214:154604:1676389564:5031.765] [signal_drhook@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1
[EC_DRHOOK:chonky:1:1:3880:3880] [20230214:154604:1676389564:5031.765] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3843] 5397 MB (maxheap), 4884 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:chonky:1:1:3880:3880] [20230214:154604:1676389564:5031.765] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] : MASTER
[EC_DRHOOK:chonky:1:1:3880:3880] [20230214:154604:1676389564:5031.765] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] : CNT0
[EC_DRHOOK:chonky:1:1:3880:3880] [20230214:154604:1676389564:5031.765] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] : CNT1
[EC_DRHOOK:chonky:1:1:3880:3880] [20230214:154604:1676389564:5031.765] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] : CNT2
[EC_DRHOOK:chonky:1:1:3880:3880] [20230214:154604:1676389564:5031.765] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] : CNT3
[EC_DRHOOK:chonky:1:1:3880:3880] [20230214:154604:1676389564:5031.765] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] : CNT4
[EC_DRHOOK:chonky:1:1:3880:3880] [20230214:154604:1676389564:5031.765] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] : STEPO
[EC_DRHOOK:chonky:1:1:3880:3880] [20230214:154604:1676389564:5031.766] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] : SCAN2M
[EC_DRHOOK:chonky:1:1:3880:3880] [20230214:154604:1676389564:5031.766] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] : GP_MODEL_HEAP
[EC_DRHOOK:chonky:1:1:3880:3880] [20230214:154604:1676389564:5031.766] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] : GP_MODEL
[EC_DRHOOK:chonky:1:1:3880:3880] [20230214:154604:1676389564:5031.766] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] : CALL_SL_HEAP
[EC_DRHOOK:chonky:1:1:3880:3880] [20230214:154604:1676389564:5031.766] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] : CALL_SL
[EC_DRHOOK:chonky:1:1:3880:3880] [20230214:154604:1676389564:5031.766] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] : >OMP-CALL_SL 1 (1004)
[EC_DRHOOK:chonky:1:1:3880:3880] [20230214:154604:1676389564:5031.766] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] : LAPINEA
[EC_DRHOOK:chonky:1:1:3880:3880] [20230214:154604:1676389564:5031.766] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] : LARMES
[EC_DRHOOK:chonky:1:1:3880:3880] [20230214:154604:1676389564:5031.766] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] : LARCINA
[EC_DRHOOK:chonky:1:1:3880:3880] [20230214:154604:1676389564:5031.766] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] : LASCAW
[EC_DRHOOK:chonky:1:1:3880:3880] [20230214:154605:1676389565:5032.766] [signal_drhook@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1
[EC_DRHOOK:chonky:1:1:3880:3880] [20230214:154605:1676389565:5032.766] [signal_drhook@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x401140 for signal#15, nsigs = 1
forrtl: error (78): process killed (SIGTERM)
Image PC Routine Line Source
oifs_43r3_model.e 000000000081FCF1 Unknown Unknown Unknown
oifs_43r3_model.e 0000000001D8BB20 Unknown Unknown Unknown
oifs_43r3_model.e 0000000000FAF537 lascaw_ 470 lascaw.F90
oifs_43r3_model.e 0000000001064965 larcina_ 319 larcina.F90
oifs_43r3_model.e 0000000001062038 larmes_ 603 larmes.F90
oifs_43r3_model.e 0000000001058F5D lapinea_ 270 lapinea.F90
oifs_43r3_model.e 0000000000FF4E6C call_sl_ 300 call_sl.F90
oifs_43r3_model.e 0000000001081A7D call_sl_heap_ 94 call_sl_heap.F90
oifs_43r3_model.e 0000000000E67648 gp_model_ 533 gp_model.F90
oifs_43r3_model.e 00000000015C1E62 gp_model_heap_ 74 gp_model_heap.F90
oifs_43r3_model.e 00000000015AC40E scan2m_ 535 scan2m.F90
oifs_43r3_model.e 00000000006620B4 stepo_ 327 stepo.F90
oifs_43r3_model.e 00000000004A0E7C cnt4_ 1133 cnt4.F90
oifs_43r3_model.e 000000000049DC29 cnt3_ 267 cnt3.F90
oifs_43r3_model.e 000000000049CE72 cnt2_ 88 cnt2.F90
oifs_43r3_model.e 000000000049CD68 cnt1_ 92 cnt1.F90
oifs_43r3_model.e 00000000004078AD cnt0_ 146 cnt0.F90
oifs_43r3_model.e 000000000040276F MAIN__ 96 master.F90
oifs_43r3_model.e 000000000040270D Unknown Unknown Unknown
oifs_43r3_model.e 0000000001D8287A Unknown Unknown Unknown
oifs_43r3_model.e 0000000001D840D7 Unknown Unknown Unknown
oifs_43r3_model.e 00000000004025E5 Unknown Unknown Unknown
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.11_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a01b
(argv4) batchid: 991
(argv5) wuid: 12206834
(argv6) fclen: 15
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/23
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.11
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206834
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/23/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/23/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_82383272ecd2a6c68ef40d3b5dd703af to: /var/lib/boinc-client/slots/23/oifs_43r3_bl_a01b_2016092300_15_991_12206834.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/23/oifs_43r3_bl_a01b_2016092300_15_991_12206834.zip
ic_ancil_file: ic_ancil_12206834
ifsdata_file: ifsdata_12206834
climate_data_file: clim_data_12206834
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /var/lib/boinc-client/slots/23/ic_ancil_12206834.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/23/ic_ancil_12206834.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_f122ec64654a145d9fccb1c0419998b3 to: /var/lib/boinc-client/slots/23/ifsdata/ifsdata_12206834.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/23/ifsdata/ifsdata_12206834.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/23/159l_2/clim_data_12206834.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/23/159l_2/clim_data_12206834.zip
Checking for progress XML file: /var/lib/boinc-client/slots/23/progress_file_12206834.xml
Opened progress file ok : /var/lib/boinc-client/slots/23/progress_file_12206834.xml
-- Model is restarting --
Adjusting last_iter, 815, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/23/progress_file_12206834.xml
last_cpu_time: 48781
upload_file_number: 8
last_iter: 769
last_upload: 692100
model_completed: 0
total_length_of_simulation: 1296000
result_base_name: oifs_43r3_bl_a01b_2016092300_15_991_12206834_0_r2029125939
The child process has been launched with process id: 3855
Executing the command: /var/lib/boinc-client/slots/23/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:chonky:1:1:3855:3855] [20230214:155101:1676389861:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2091] fp = 0x30e4a00
[EC_DRHOOK:chonky:1:1:3855:3855] [20230214:155101:1676389861:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:chonky:1:1:3855:3855] [20230214:155101:1676389861:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:chonky:1:1:3855:3855] [20230214:155101:1676389861:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:chonky:1:1:3855:3855] [20230214:155101:1676389861:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:chonky:1:1:3855:3855] [20230214:155101:1676389861:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:chonky:1:1:3855:3855] [20230214:155101:1676389861:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:chonky:1:1:3855:3855] [20230214:155101:1676389861:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:chonky:1:1:3855:3855] [20230214:155101:1676389861:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:chonky:1:1:3855:3855] [20230214:155101:1676389861:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:chonky:1:1:3855:3855] [20230214:155101:1676389861:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:chonky:1:1:3855:3855] [20230214:155101:1676389861:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:chonky:1:1:3855:3855] [20230214:155101:1676389861:0.001] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:chonky:1:1:3855:3855] [20230214:155101:1676389861:0.001] [ignore_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:chonky:1:1:3855:3855] [20230214:155101:1676389861:0.001] [restore_default_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:chonky:1:1:3855:3855] [20230214:155101:1676389861:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:chonky:1:1:3855:3855] [20230214:155101:1676389861:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f520 (old at (nil))
[EC_DRHOOK:chonky:1:1:3855:3855] [20230214:155101:1676389861:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:chonky:1:1:3855:3855] [20230214:155101:1676389861:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f520 (old at (nil))
[EC_DRHOOK:chonky:1:1:3855:3855] [20230214:155101:1676389861:0.001] [trapfpe_treatment@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:chonky:1:1:3855:3855] [20230214:155101:1676389861:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:chonky:1:1:3855:3855] [20230214:155101:1676389861:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:chonky:1:1:3855:3855] [20230214:155101:1676389861:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f520 (old at (nil))
[EC_DRHOOK:chonky:1:1:3855:3855] [20230214:155101:1676389861:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:chonky:1:1:3855:3855] [20230214:155101:1676389861:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:chonky:1:1:3855:3855] [20230214:155101:1676389861:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:chonky:1:1:3855:3855] [20230214:155101:1676389861:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f520 (old at (nil))
[EC_DRHOOK:chonky:1:1:3855:3855] [20230214:155101:1676389861:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f520 (old at (nil))
[EC_DRHOOK:chonky:1:1:3855:3855] [20230214:155101:1676389861:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f520 (old at (nil))
[EC_DRHOOK:chonky:1:1:3855:3855] [20230214:155101:1676389861:0.001] [catch_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
15:52:27 STEP 768 H= 192:00 +CPU= 51.603
15:53:15 STEP 769 H= 192:15 +CPU= 47.261
15:54:00 STEP 770 H= 192:30 +CPU= 44.260
15:54:44 STEP 771 H= 192:45 +CPU= 43.979
15:55:36 STEP 772 H= 193:00 +CPU= 51.181
15:56:25 STEP 773 H= 193:15 +CPU= 48.685
15:57:10 STEP 774 H= 193:30 +CPU= 45.205
15:57:55 STEP 775 H= 193:45 +CPU= 44.865
15:58:40 STEP 776 H= 194:00 +CPU= 44.533
15:59:27 STEP 777 H= 194:15 +CPU= 46.932
16:00:12 STEP 778 H= 194:30 +CPU= 44.991
16:00:57 STEP 779 H= 194:45 +CPU= 44.333
16:01:55 STEP 780 H= 195:00 +CPU= 57.181
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGh7zg+000780
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHh7zg+000780
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAh7zg+000780
16:02:42 STEP 781 H= 195:15 +CPU= 46.993
16:03:27 STEP 782 H= 195:30 +CPU= 45.213
16:04:15 STEP 783 H= 195:45 +CPU= 46.946
16:05:04 STEP 784 H= 196:00 +CPU= 49.162
16:05:52 STEP 785 H= 196:15 +CPU= 47.919
16:06:39 STEP 786 H= 196:30 +CPU= 46.966
16:07:30 STEP 787 H= 196:45 +CPU= 50.478
16:08:19 STEP 788 H= 197:00 +CPU= 49.006
16:09:07 STEP 789 H= 197:15 +CPU= 47.568
16:09:56 STEP 790 H= 197:30 +CPU= 48.515
16:10:48 STEP 791 H= 197:45 +CPU= 50.543
16:12:06 STEP 792 H= 198:00 +CPU= 64.866
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGh7zg+000792
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHh7zg+000792
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAh7zg+000792
16:12:57 STEP 793 H= 198:15 +CPU= 49.075
16:13:46 STEP 794 H= 198:30 +CPU= 48.825
16:14:34 STEP 795 H= 198:45 +CPU= 47.977
16:15:23 STEP 796 H= 199:00 +CPU= 48.605
16:16:14 STEP 797 H= 199:15 +CPU= 49.798
16:17:03 STEP 798 H= 199:30 +CPU= 49.181
16:17:54 STEP 799 H= 199:45 +CPU= 50.293
16:18:50 STEP 800 H= 200:00 +CPU= 54.345
16:19:46 STEP 801 H= 200:15 +CPU= 54.181
16:20:41 STEP 802 H= 200:30 +CPU= 53.494
16:21:34 STEP 803 H= 200:45 +CPU= 51.463
16:22:39 STEP 804 H= 201:00 +CPU= 63.805
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGh7zg+000804
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHh7zg+000804
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAh7zg+000804
16:23:33 STEP 805 H= 201:15 +CPU= 52.501
16:24:29 STEP 806 H= 201:30 +CPU= 53.532
16:25:22 STEP 807 H= 201:45 +CPU= 51.837
16:26:21 STEP 808 H= 202:00 +CPU= 54.741
16:27:14 STEP 809 H= 202:15 +CPU= 52.433
16:28:13 STEP 810 H= 202:30 +CPU= 55.715
16:29:06 STEP 811 H= 202:45 +CPU= 51.962
16:30:01 STEP 812 H= 203:00 +CPU= 53.561
16:30:54 STEP 813 H= 203:15 +CPU= 52.958
16:31:47 STEP 814 H= 203:30 +CPU= 52.288
16:32:41 STEP 815 H= 203:45 +CPU= 52.263
16:33:55 STEP 816 H= 204:00 +CPU= 71.845
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGh7zg+000816
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHh7zg+000816
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAh7zg+000816
16:34:50 STEP 817 H= 204:15 +CPU= 53.455
16:35:42 STEP 818 H= 204:30 +CPU= 51.806
16:36:35 STEP 819 H= 204:45 +CPU= 52.139
16:37:29 STEP 820 H= 205:00 +CPU= 53.240
16:38:18 STEP 821 H= 205:15 +CPU= 48.406
16:39:08 STEP 822 H= 205:30 +CPU= 50.011
16:40:00 STEP 823 H= 205:45 +CPU= 51.274
16:40:56 STEP 824 H= 206:00 +CPU= 54.986
16:41:50 STEP 825 H= 206:15 +CPU= 52.580
16:42:44 STEP 826 H= 206:30 +CPU= 52.859
16:43:38 STEP 827 H= 206:45 +CPU= 53.136
16:44:46 STEP 828 H= 207:00 +CPU= 67.394
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGh7zg+000828
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHh7zg+000828
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAh7zg+000828
16:45:39 STEP 829 H= 207:15 +CPU= 52.438
16:46:32 STEP 830 H= 207:30 +CPU= 51.405
16:47:24 STEP 831 H= 207:45 +CPU= 51.403
16:48:16 STEP 832 H= 208:00 +CPU= 52.049
16:49:12 STEP 833 H= 208:15 +CPU= 53.389
16:50:05 STEP 834 H= 208:30 +CPU= 51.905
16:50:59 STEP 835 H= 208:45 +CPU= 52.620
16:51:54 STEP 836 H= 209:00 +CPU= 54.443
16:52:50 STEP 837 H= 209:15 +CPU= 54.509
16:53:45 STEP 838 H= 209:30 +CPU= 54.321
16:54:42 STEP 839 H= 209:45 +CPU= 54.162
16:55:48 STEP 840 H= 210:00 +CPU= 64.833
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGh7zg+000840
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHh7zg+000840
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAh7zg+000840
16:56:43 STEP 841 H= 210:15 +CPU= 52.900
16:57:37 STEP 842 H= 210:30 +CPU= 52.120
16:58:30 STEP 843 H= 210:45 +CPU= 51.070
16:59:22 STEP 844 H= 211:00 +CPU= 51.545
17:00:19 STEP 845 H= 211:15 +CPU= 54.069
17:01:09 STEP 846 H= 211:30 +CPU= 49.291
17:01:55 STEP 847 H= 211:45 +CPU= 45.626
17:02:44 STEP 848 H= 212:00 +CPU= 47.977
17:03:34 STEP 849 H= 212:15 +CPU= 49.533
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.11_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a01b
(argv4) batchid: 991
(argv5) wuid: 12206834
(argv6) fclen: 15
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/23
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.11
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206834
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/23/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/23/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_82383272ecd2a6c68ef40d3b5dd703af to: /var/lib/boinc-client/slots/23/oifs_43r3_bl_a01b_2016092300_15_991_12206834.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/23/oifs_43r3_bl_a01b_2016092300_15_991_12206834.zip
ic_ancil_file: ic_ancil_12206834
ifsdata_file: ifsdata_12206834
climate_data_file: clim_data_12206834
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /var/lib/boinc-client/slots/23/ic_ancil_12206834.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/23/ic_ancil_12206834.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_f122ec64654a145d9fccb1c0419998b3 to: /var/lib/boinc-client/slots/23/ifsdata/ifsdata_12206834.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/23/ifsdata/ifsdata_12206834.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/23/159l_2/clim_data_12206834.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/23/159l_2/clim_data_12206834.zip
Checking for progress XML file: /var/lib/boinc-client/slots/23/progress_file_12206834.xml
Opened progress file ok : /var/lib/boinc-client/slots/23/progress_file_12206834.xml
-- Model is restarting --
Adjusting last_iter, 850, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/23/progress_file_12206834.xml
last_cpu_time: 53076
upload_file_number: 8
last_iter: 817
last_upload: 692100
model_completed: 0
total_length_of_simulation: 1296000
result_base_name: oifs_43r3_bl_a01b_2016092300_15_991_12206834_0_r2029125939
The child process has been launched with process id: 3861
Executing the command: /var/lib/boinc-client/slots/23/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:chonky:1:1:3861:3861] [20230214:185204:1676400724:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2091] fp = 0x30e4a00
[EC_DRHOOK:chonky:1:1:3861:3861] [20230214:185204:1676400724:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:chonky:1:1:3861:3861] [20230214:185204:1676400724:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:chonky:1:1:3861:3861] [20230214:185204:1676400724:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:chonky:1:1:3861:3861] [20230214:185204:1676400724:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:chonky:1:1:3861:3861] [20230214:185204:1676400724:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:chonky:1:1:3861:3861] [20230214:185204:1676400724:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:chonky:1:1:3861:3861] [20230214:185204:1676400724:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:chonky:1:1:3861:3861] [20230214:185204:1676400724:0.001] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:chonky:1:1:3861:3861] [20230214:185204:1676400724:0.001] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:chonky:1:1:3861:3861] [20230214:185204:1676400724:0.001] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:chonky:1:1:3861:3861] [20230214:185204:1676400724:0.001] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:chonky:1:1:3861:3861] [20230214:185204:1676400724:0.001] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:chonky:1:1:3861:3861] [20230214:185204:1676400724:0.001] [ignore_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:chonky:1:1:3861:3861] [20230214:185204:1676400724:0.001] [restore_default_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:chonky:1:1:3861:3861] [20230214:185204:1676400724:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:chonky:1:1:3861:3861] [20230214:185204:1676400724:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f520 (old at (nil))
[EC_DRHOOK:chonky:1:1:3861:3861] [20230214:185204:1676400724:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:chonky:1:1:3861:3861] [20230214:185204:1676400724:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f520 (old at (nil))
[EC_DRHOOK:chonky:1:1:3861:3861] [20230214:185204:1676400724:0.001] [trapfpe_treatment@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:chonky:1:1:3861:3861] [20230214:185204:1676400724:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:chonky:1:1:3861:3861] [20230214:185204:1676400724:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:chonky:1:1:3861:3861] [20230214:185204:1676400724:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f520 (old at (nil))
[EC_DRHOOK:chonky:1:1:3861:3861] [20230214:185204:1676400724:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:chonky:1:1:3861:3861] [20230214:185204:1676400724:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:chonky:1:1:3861:3861] [20230214:185204:1676400724:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:chonky:1:1:3861:3861] [20230214:185204:1676400724:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f520 (old at (nil))
[EC_DRHOOK:chonky:1:1:3861:3861] [20230214:185204:1676400724:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f520 (old at (nil))
[EC_DRHOOK:chonky:1:1:3861:3861] [20230214:185204:1676400724:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f520 (old at (nil))
[EC_DRHOOK:chonky:1:1:3861:3861] [20230214:185204:1676400724:0.001] [catch_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
18:54:15 STEP 816 H= 204:00 +CPU= 50.660
18:54:59 STEP 817 H= 204:15 +CPU= 43.562
18:55:47 STEP 818 H= 204:30 +CPU= 47.500
18:56:30 STEP 819 H= 204:45 +CPU= 43.197
18:57:13 STEP 820 H= 205:00 +CPU= 42.283
18:57:56 STEP 821 H= 205:15 +CPU= 43.095
18:58:43 STEP 822 H= 205:30 +CPU= 46.214
18:59:29 STEP 823 H= 205:45 +CPU= 45.994
19:00:12 STEP 824 H= 206:00 +CPU= 43.028
19:00:54 STEP 825 H= 206:15 +CPU= 41.224
19:01:44 STEP 826 H= 206:30 +CPU= 49.438
19:02:31 STEP 827 H= 206:45 +CPU= 46.973
19:03:28 STEP 828 H= 207:00 +CPU= 56.398
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGh7zg+000828
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHh7zg+000828
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAh7zg+000828
19:04:19 STEP 829 H= 207:15 +CPU= 50.240
19:05:02 STEP 830 H= 207:30 +CPU= 42.878
19:05:46 STEP 831 H= 207:45 +CPU= 43.644
19:06:35 STEP 832 H= 208:00 +CPU= 49.354
19:07:24 STEP 833 H= 208:15 +CPU= 48.320
19:08:11 STEP 834 H= 208:30 +CPU= 47.227
19:08:59 STEP 835 H= 208:45 +CPU= 47.335
19:09:53 STEP 836 H= 209:00 +CPU= 50.300
19:10:44 STEP 837 H= 209:15 +CPU= 50.327
19:11:39 STEP 838 H= 209:30 +CPU= 50.593
19:12:31 STEP 839 H= 209:45 +CPU= 51.466
19:13:35 STEP 840 H= 210:00 +CPU= 62.283
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGh7zg+000840
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHh7zg+000840
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAh7zg+000840
19:14:27 STEP 841 H= 210:15 +CPU= 49.395
19:15:17 STEP 842 H= 210:30 +CPU= 48.581
19:16:10 STEP 843 H= 210:45 +CPU= 50.817
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.11_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a01b
(argv4) batchid: 991
(argv5) wuid: 12206834
(argv6) fclen: 15
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/23
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.11
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206834
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/23/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/23/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_82383272ecd2a6c68ef40d3b5dd703af to: /var/lib/boinc-client/slots/23/oifs_43r3_bl_a01b_2016092300_15_991_12206834.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/23/oifs_43r3_bl_a01b_2016092300_15_991_12206834.zip
ic_ancil_file: ic_ancil_12206834
ifsdata_file: ifsdata_12206834
climate_data_file: clim_data_12206834
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /var/lib/boinc-client/slots/23/ic_ancil_12206834.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/23/ic_ancil_12206834.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_f122ec64654a145d9fccb1c0419998b3 to: /var/lib/boinc-client/slots/23/ifsdata/ifsdata_12206834.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/23/ifsdata/ifsdata_12206834.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/23/159l_2/clim_data_12206834.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/23/159l_2/clim_data_12206834.zip
Checking for progress XML file: /var/lib/boinc-client/slots/23/progress_file_12206834.xml
Opened progress file ok : /var/lib/boinc-client/slots/23/progress_file_12206834.xml
-- Model is restarting --
Adjusting last_iter, 844, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/23/progress_file_12206834.xml
last_cpu_time: 54446
upload_file_number: 8
last_iter: 817
last_upload: 692100
model_completed: 0
total_length_of_simulation: 1296000
result_base_name: oifs_43r3_bl_a01b_2016092300_15_991_12206834_0_r2029125939
The child process has been launched with process id: 3875
Executing the command: /var/lib/boinc-client/slots/23/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:chonky:1:1:3875:3875] [20230214:192219:1676402539:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2091] fp = 0x30e4a00
[EC_DRHOOK:chonky:1:1:3875:3875] [20230214:192219:1676402539:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:chonky:1:1:3875:3875] [20230214:192219:1676402539:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:chonky:1:1:3875:3875] [20230214:192219:1676402539:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:chonky:1:1:3875:3875] [20230214:192219:1676402539:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:chonky:1:1:3875:3875] [20230214:192219:1676402539:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:chonky:1:1:3875:3875] [20230214:192219:1676402539:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:chonky:1:1:3875:3875] [20230214:192219:1676402539:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:chonky:1:1:3875:3875] [20230214:192219:1676402539:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:chonky:1:1:3875:3875] [20230214:192219:1676402539:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:chonky:1:1:3875:3875] [20230214:192219:1676402539:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:chonky:1:1:3875:3875] [20230214:192219:1676402539:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:chonky:1:1:3875:3875] [20230214:192219:1676402539:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:chonky:1:1:3875:3875] [20230214:192219:1676402539:0.000] [ignore_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:chonky:1:1:3875:3875] [20230214:192219:1676402539:0.001] [restore_default_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:chonky:1:1:3875:3875] [20230214:192219:1676402539:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:chonky:1:1:3875:3875] [20230214:192219:1676402539:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f520 (old at (nil))
[EC_DRHOOK:chonky:1:1:3875:3875] [20230214:192219:1676402539:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:chonky:1:1:3875:3875] [20230214:192219:1676402539:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f520 (old at (nil))
[EC_DRHOOK:chonky:1:1:3875:3875] [20230214:192219:1676402539:0.001] [trapfpe_treatment@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:chonky:1:1:3875:3875] [20230214:192219:1676402539:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:chonky:1:1:3875:3875] [20230214:192219:1676402539:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:chonky:1:1:3875:3875] [20230214:192219:1676402539:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f520 (old at (nil))
[EC_DRHOOK:chonky:1:1:3875:3875] [20230214:192219:1676402539:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:chonky:1:1:3875:3875] [20230214:192219:1676402539:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:chonky:1:1:3875:3875] [20230214:192219:1676402539:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:chonky:1:1:3875:3875] [20230214:192219:1676402539:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f520 (old at (nil))
[EC_DRHOOK:chonky:1:1:3875:3875] [20230214:192219:1676402539:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f520 (old at (nil))
[EC_DRHOOK:chonky:1:1:3875:3875] [20230214:192219:1676402539:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f520 (old at (nil))
[EC_DRHOOK:chonky:1:1:3875:3875] [20230214:192219:1676402539:0.001] [catch_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
19:24:29 STEP 816 H= 204:00 +CPU= 50.222
19:25:16 STEP 817 H= 204:15 +CPU= 45.570
Abort request received from BOINC client, ending the child process
</stderr_txt>
]]>
