<core_client_version>7.16.6</core_client_version>
<![CDATA[
<message>
aborted by user</message>
<stderr_txt>
ction.net/oifs_43r3_bl_1.11_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a01h
(argv4) batchid: 991
(argv5) wuid: 12206840
(argv6) fclen: 15
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/22
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.11
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206840
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/22/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/22/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_d887486f6a3aabe4c55374c1725be1de to: /var/lib/boinc-client/slots/22/oifs_43r3_bl_a01h_2016092300_15_991_12206840.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/22/oifs_43r3_bl_a01h_2016092300_15_991_12206840.zip
ic_ancil_file: ic_ancil_12206840
ifsdata_file: ifsdata_12206840
climate_data_file: clim_data_12206840
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /var/lib/boinc-client/slots/22/ic_ancil_12206840.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/22/ic_ancil_12206840.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_155e44070232d2cfdcf7069aabe5bcf7 to: /var/lib/boinc-client/slots/22/ifsdata/ifsdata_12206840.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/22/ifsdata/ifsdata_12206840.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/22/159l_2/clim_data_12206840.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/22/159l_2/clim_data_12206840.zip
Checking for progress XML file: /var/lib/boinc-client/slots/22/progress_file_12206840.xml
Opened progress file ok : /var/lib/boinc-client/slots/22/progress_file_12206840.xml
-- Model is restarting --
Adjusting last_iter, 742, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/22/progress_file_12206840.xml
last_cpu_time: 43585
upload_file_number: 7
last_iter: 721
last_upload: 604800
model_completed: 0
total_length_of_simulation: 1296000
result_base_name: oifs_43r3_bl_a01h_2016092300_15_991_12206840_0_r968148541
The child process has been launched with process id: 3879
Executing the command: /var/lib/boinc-client/slots/22/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:chonky:1:1:3879:3879] [20230214:142212:1676384532:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2091] fp = 0x30e4a00
[EC_DRHOOK:chonky:1:1:3879:3879] [20230214:142212:1676384532:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:chonky:1:1:3879:3879] [20230214:142212:1676384532:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:chonky:1:1:3879:3879] [20230214:142212:1676384532:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:chonky:1:1:3879:3879] [20230214:142212:1676384532:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:chonky:1:1:3879:3879] [20230214:142212:1676384532:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:chonky:1:1:3879:3879] [20230214:142212:1676384532:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:chonky:1:1:3879:3879] [20230214:142212:1676384532:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:chonky:1:1:3879:3879] [20230214:142212:1676384532:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:chonky:1:1:3879:3879] [20230214:142212:1676384532:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:chonky:1:1:3879:3879] [20230214:142212:1676384532:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:chonky:1:1:3879:3879] [20230214:142212:1676384532:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:chonky:1:1:3879:3879] [20230214:142212:1676384532:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:chonky:1:1:3879:3879] [20230214:142212:1676384532:0.000] [ignore_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:chonky:1:1:3879:3879] [20230214:142212:1676384532:0.000] [restore_default_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:chonky:1:1:3879:3879] [20230214:142212:1676384532:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:chonky:1:1:3879:3879] [20230214:142212:1676384532:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f520 (old at (nil))
[EC_DRHOOK:chonky:1:1:3879:3879] [20230214:142212:1676384532:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:chonky:1:1:3879:3879] [20230214:142212:1676384532:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f520 (old at (nil))
[EC_DRHOOK:chonky:1:1:3879:3879] [20230214:142212:1676384532:0.000] [trapfpe_treatment@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:chonky:1:1:3879:3879] [20230214:142212:1676384532:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:chonky:1:1:3879:3879] [20230214:142212:1676384532:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:chonky:1:1:3879:3879] [20230214:142212:1676384532:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f520 (old at (nil))
[EC_DRHOOK:chonky:1:1:3879:3879] [20230214:142212:1676384532:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:chonky:1:1:3879:3879] [20230214:142212:1676384532:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:chonky:1:1:3879:3879] [20230214:142212:1676384532:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:chonky:1:1:3879:3879] [20230214:142212:1676384532:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f520 (old at (nil))
[EC_DRHOOK:chonky:1:1:3879:3879] [20230214:142212:1676384532:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f520 (old at (nil))
[EC_DRHOOK:chonky:1:1:3879:3879] [20230214:142212:1676384532:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f520 (old at (nil))
[EC_DRHOOK:chonky:1:1:3879:3879] [20230214:142212:1676384532:0.001] [catch_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
14:24:37 STEP 720 H= 180:00 +CPU= 54.691
14:25:27 STEP 721 H= 180:15 +CPU= 48.232
14:26:17 STEP 722 H= 180:30 +CPU= 49.043
14:27:09 STEP 723 H= 180:45 +CPU= 50.525
14:27:58 STEP 724 H= 181:00 +CPU= 47.083
14:28:46 STEP 725 H= 181:15 +CPU= 48.174
14:29:37 STEP 726 H= 181:30 +CPU= 49.556
14:30:23 STEP 727 H= 181:45 +CPU= 45.380
14:31:12 STEP 728 H= 182:00 +CPU= 47.207
14:31:55 STEP 729 H= 182:15 +CPU= 42.847
14:32:44 STEP 730 H= 182:30 +CPU= 48.972
14:33:30 STEP 731 H= 182:45 +CPU= 45.570
14:34:23 STEP 732 H= 183:00 +CPU= 53.079
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMGGh7zg+000732
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMSHh7zg+000732
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMUAh7zg+000732
14:35:08 STEP 733 H= 183:15 +CPU= 45.191
14:36:02 STEP 734 H= 183:30 +CPU= 47.440
14:36:53 STEP 735 H= 183:45 +CPU= 49.927
14:37:48 STEP 736 H= 184:00 +CPU= 55.340
14:38:39 STEP 737 H= 184:15 +CPU= 50.405
14:39:30 STEP 738 H= 184:30 +CPU= 51.123
14:40:22 STEP 739 H= 184:45 +CPU= 52.067
14:41:15 STEP 740 H= 185:00 +CPU= 52.280
14:42:05 STEP 741 H= 185:15 +CPU= 49.599
14:42:55 STEP 742 H= 185:30 +CPU= 50.552
14:43:47 STEP 743 H= 185:45 +CPU= 51.192
14:44:52 STEP 744 H= 186:00 +CPU= 65.127
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMGGh7zg+000744
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMSHh7zg+000744
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMUAh7zg+000744
14:45:45 STEP 745 H= 186:15 +CPU= 53.247
14:46:39 STEP 746 H= 186:30 +CPU= 53.608
14:47:31 STEP 747 H= 186:45 +CPU= 51.499
14:48:23 STEP 748 H= 187:00 +CPU= 52.603
14:49:16 STEP 749 H= 187:15 +CPU= 52.053
14:50:07 STEP 750 H= 187:30 +CPU= 51.615
14:50:59 STEP 751 H= 187:45 +CPU= 51.446
14:51:51 STEP 752 H= 188:00 +CPU= 51.915
14:52:46 STEP 753 H= 188:15 +CPU= 54.543
14:53:40 STEP 754 H= 188:30 +CPU= 54.470
14:54:32 STEP 755 H= 188:45 +CPU= 51.672
14:55:38 STEP 756 H= 189:00 +CPU= 65.571
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMGGh7zg+000756
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMSHh7zg+000756
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMUAh7zg+000756
14:56:31 STEP 757 H= 189:15 +CPU= 53.064
14:57:22 STEP 758 H= 189:30 +CPU= 50.601
14:58:14 STEP 759 H= 189:45 +CPU= 51.505
14:59:05 STEP 760 H= 190:00 +CPU= 51.212
14:59:54 STEP 761 H= 190:15 +CPU= 49.270
15:00:45 STEP 762 H= 190:30 +CPU= 50.887
15:01:36 STEP 763 H= 190:45 +CPU= 50.190
15:02:32 STEP 764 H= 191:00 +CPU= 56.039
15:03:23 STEP 765 H= 191:15 +CPU= 50.439
15:04:14 STEP 766 H= 191:30 +CPU= 50.635
15:05:04 STEP 767 H= 191:45 +CPU= 50.809
15:06:23 STEP 768 H= 192:00 +CPU= 71.712
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMGGh7zg+000768
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMSHh7zg+000768
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMUAh7zg+000768
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206840/ICMGGh7zg+000672
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206840/ICMSHh7zg+000672
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206840/ICMUAh7zg+000672
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206840/ICMGGh7zg+000684
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206840/ICMSHh7zg+000684
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206840/ICMUAh7zg+000684
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206840/ICMGGh7zg+000696
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206840/ICMSHh7zg+000696
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206840/ICMUAh7zg+000696
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206840/ICMGGh7zg+000708
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206840/ICMSHh7zg+000708
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206840/ICMUAh7zg+000708
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206840/ICMGGh7zg+000720
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206840/ICMSHh7zg+000720
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206840/ICMUAh7zg+000720
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206840/ICMGGh7zg+000732
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206840/ICMSHh7zg+000732
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206840/ICMUAh7zg+000732
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206840/ICMGGh7zg+000744
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206840/ICMSHh7zg+000744
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206840/ICMUAh7zg+000744
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206840/ICMGGh7zg+000756
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206840/ICMSHh7zg+000756
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206840/ICMUAh7zg+000756
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a01h_2016092300_15_991_12206840_0_r968148541_7.zip
15:07:17 STEP 769 H= 192:15 +CPU= 50.607
Uploading the intermediate file: upload_file_7.zip
15:08:09 STEP 770 H= 192:30 +CPU= 50.545
15:09:00 STEP 771 H= 192:45 +CPU= 51.547
15:09:53 STEP 772 H= 193:00 +CPU= 52.561
15:10:42 STEP 773 H= 193:15 +CPU= 49.381
15:11:38 STEP 774 H= 193:30 +CPU= 54.094
15:12:30 STEP 775 H= 193:45 +CPU= 52.097
15:13:20 STEP 776 H= 194:00 +CPU= 49.607
15:14:12 STEP 777 H= 194:15 +CPU= 51.391
15:15:02 STEP 778 H= 194:30 +CPU= 50.281
15:15:55 STEP 779 H= 194:45 +CPU= 52.947
15:17:02 STEP 780 H= 195:00 +CPU= 67.213
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMGGh7zg+000780
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMSHh7zg+000780
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMUAh7zg+000780
15:17:55 STEP 781 H= 195:15 +CPU= 51.890
15:18:49 STEP 782 H= 195:30 +CPU= 54.083
15:19:44 STEP 783 H= 195:45 +CPU= 55.252
15:20:40 STEP 784 H= 196:00 +CPU= 54.899
15:21:35 STEP 785 H= 196:15 +CPU= 54.514
15:22:27 STEP 786 H= 196:30 +CPU= 51.944
15:23:22 STEP 787 H= 196:45 +CPU= 55.120
15:24:20 STEP 788 H= 197:00 +CPU= 57.870
15:25:14 STEP 789 H= 197:15 +CPU= 53.750
15:26:06 STEP 790 H= 197:30 +CPU= 51.339
15:26:57 STEP 791 H= 197:45 +CPU= 51.357
15:28:07 STEP 792 H= 198:00 +CPU= 69.718
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMGGh7zg+000792
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMSHh7zg+000792
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMUAh7zg+000792
15:28:57 STEP 793 H= 198:15 +CPU= 49.672
15:29:48 STEP 794 H= 198:30 +CPU= 50.559
15:30:38 STEP 795 H= 198:45 +CPU= 50.257
15:31:30 STEP 796 H= 199:00 +CPU= 51.820
15:32:18 STEP 797 H= 199:15 +CPU= 47.707
15:33:10 STEP 798 H= 199:30 +CPU= 52.104
15:33:59 STEP 799 H= 199:45 +CPU= 49.210
15:34:54 STEP 800 H= 200:00 +CPU= 54.822
15:35:48 STEP 801 H= 200:15 +CPU= 53.750
15:36:38 STEP 802 H= 200:30 +CPU= 49.752
15:37:27 STEP 803 H= 200:45 +CPU= 48.790
15:38:30 STEP 804 H= 201:00 +CPU= 62.860
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMGGh7zg+000804
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMSHh7zg+000804
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMUAh7zg+000804
15:39:19 STEP 805 H= 201:15 +CPU= 48.831
15:40:07 STEP 806 H= 201:30 +CPU= 47.819
15:40:55 STEP 807 H= 201:45 +CPU= 47.845
15:41:45 STEP 808 H= 202:00 +CPU= 49.639
15:42:34 STEP 809 H= 202:15 +CPU= 48.647
15:43:22 STEP 810 H= 202:30 +CPU= 48.632
15:44:11 STEP 811 H= 202:45 +CPU= 48.475
15:45:01 STEP 812 H= 203:00 +CPU= 49.866
15:45:51 STEP 813 H= 203:15 +CPU= 49.756
[EC_DRHOOK:chonky:1:1:3879:3879] [20230214:154604:1676389564:5031.792] [signal_drhook@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 5397MB (heap), 4884MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:chonky:1:1:3879:3879] [20230214:154604:1676389564:5031.792] [signal_drhook@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:chonky:1:1:3879:3879] [20230214:154604:1676389564:5031.792] [signal_drhook@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f0c0 (old at (nil))
[EC_DRHOOK:chonky:1:1:3879:3879] [20230214:154604:1676389564:5031.792] [signal_drhook@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1
[EC_DRHOOK:chonky:1:1:3879:3879] [20230214:154604:1676389564:5031.792] [signal_drhook@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1
[EC_DRHOOK:chonky:1:1:3879:3879] [20230214:154604:1676389564:5031.792] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3843] 5397 MB (maxheap), 4884 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:chonky:1:1:3879:3879] [20230214:154604:1676389564:5031.792] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] : MASTER
[EC_DRHOOK:chonky:1:1:3879:3879] [20230214:154604:1676389564:5031.792] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] : CNT0
[EC_DRHOOK:chonky:1:1:3879:3879] [20230214:154604:1676389564:5031.792] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] : CNT1
[EC_DRHOOK:chonky:1:1:3879:3879] [20230214:154604:1676389564:5031.792] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] : CNT2
[EC_DRHOOK:chonky:1:1:3879:3879] [20230214:154604:1676389564:5031.792] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] : CNT3
[EC_DRHOOK:chonky:1:1:3879:3879] [20230214:154604:1676389564:5031.792] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] : CNT4
[EC_DRHOOK:chonky:1:1:3879:3879] [20230214:154604:1676389564:5031.792] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] : STEPO
[EC_DRHOOK:chonky:1:1:3879:3879] [20230214:154604:1676389564:5031.792] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] : SCAN2M
[EC_DRHOOK:chonky:1:1:3879:3879] [20230214:154604:1676389564:5031.792] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] : GP_MODEL_HEAP
[EC_DRHOOK:chonky:1:1:3879:3879] [20230214:154604:1676389564:5031.792] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] : GP_MODEL
[EC_DRHOOK:chonky:1:1:3879:3879] [20230214:154604:1676389564:5031.792] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] : CALL_SL_HEAP
[EC_DRHOOK:chonky:1:1:3879:3879] [20230214:154604:1676389564:5031.792] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] : CALL_SL
[EC_DRHOOK:chonky:1:1:3879:3879] [20230214:154604:1676389564:5031.792] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] : >OMP-CALL_SL 1 (1004)
[EC_DRHOOK:chonky:1:1:3879:3879] [20230214:154604:1676389564:5031.792] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] : LAPINEA
[EC_DRHOOK:chonky:1:1:3879:3879] [20230214:154604:1676389564:5031.792] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] : LARCINA
[EC_DRHOOK:chonky:1:1:3879:3879] [20230214:154604:1676389564:5031.792] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] : LASCAW
[EC_DRHOOK:chonky:1:1:3879:3879] [20230214:154604:1676389564:5031.792] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] : LASCAW_CLA
[EC_DRHOOK:chonky:1:1:3879:3879] [20230214:154605:1676389565:5032.793] [signal_drhook@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1
[EC_DRHOOK:chonky:1:1:3879:3879] [20230214:154605:1676389565:5032.793] [signal_drhook@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x401140 for signal#15, nsigs = 1
forrtl: error (78): process killed (SIGTERM)
Image PC Routine Line Source
oifs_43r3_model.e 000000000081FCF1 Unknown Unknown Unknown
oifs_43r3_model.e 0000000001D8BB20 Unknown Unknown Unknown
oifs_43r3_model.e 0000000000FCA684 lascaw_cla_ 198 lascaw_cla.F90
oifs_43r3_model.e 0000000000FB552E lascaw_ 866 lascaw.F90
oifs_43r3_model.e 0000000001064965 larcina_ 319 larcina.F90
oifs_43r3_model.e 0000000001059F2D lapinea_ 325 lapinea.F90
oifs_43r3_model.e 0000000000FF4E6C call_sl_ 300 call_sl.F90
oifs_43r3_model.e 0000000001081A7D call_sl_heap_ 94 call_sl_heap.F90
oifs_43r3_model.e 0000000000E67648 gp_model_ 533 gp_model.F90
oifs_43r3_model.e 00000000015C1E62 gp_model_heap_ 74 gp_model_heap.F90
oifs_43r3_model.e 00000000015AC40E scan2m_ 535 scan2m.F90
oifs_43r3_model.e 00000000006620B4 stepo_ 327 stepo.F90
oifs_43r3_model.e 00000000004A0E7C cnt4_ 1133 cnt4.F90
oifs_43r3_model.e 000000000049DC29 cnt3_ 267 cnt3.F90
oifs_43r3_model.e 000000000049CE72 cnt2_ 88 cnt2.F90
oifs_43r3_model.e 000000000049CD68 cnt1_ 92 cnt1.F90
oifs_43r3_model.e 00000000004078AD cnt0_ 146 cnt0.F90
oifs_43r3_model.e 000000000040276F MAIN__ 96 master.F90
oifs_43r3_model.e 000000000040270D Unknown Unknown Unknown
oifs_43r3_model.e 0000000001D8287A Unknown Unknown Unknown
oifs_43r3_model.e 0000000001D840D7 Unknown Unknown Unknown
oifs_43r3_model.e 00000000004025E5 Unknown Unknown Unknown
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.11_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a01h
(argv4) batchid: 991
(argv5) wuid: 12206840
(argv6) fclen: 15
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/22
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.11
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206840
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/22/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/22/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_d887486f6a3aabe4c55374c1725be1de to: /var/lib/boinc-client/slots/22/oifs_43r3_bl_a01h_2016092300_15_991_12206840.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/22/oifs_43r3_bl_a01h_2016092300_15_991_12206840.zip
ic_ancil_file: ic_ancil_12206840
ifsdata_file: ifsdata_12206840
climate_data_file: clim_data_12206840
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /var/lib/boinc-client/slots/22/ic_ancil_12206840.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/22/ic_ancil_12206840.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_155e44070232d2cfdcf7069aabe5bcf7 to: /var/lib/boinc-client/slots/22/ifsdata/ifsdata_12206840.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/22/ifsdata/ifsdata_12206840.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/22/159l_2/clim_data_12206840.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/22/159l_2/clim_data_12206840.zip
Checking for progress XML file: /var/lib/boinc-client/slots/22/progress_file_12206840.xml
Opened progress file ok : /var/lib/boinc-client/slots/22/progress_file_12206840.xml
-- Model is restarting --
Adjusting last_iter, 814, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/22/progress_file_12206840.xml
last_cpu_time: 48487
upload_file_number: 8
last_iter: 769
last_upload: 691200
model_completed: 0
total_length_of_simulation: 1296000
result_base_name: oifs_43r3_bl_a01h_2016092300_15_991_12206840_0_r968148541
The child process has been launched with process id: 3859
Executing the command: /var/lib/boinc-client/slots/22/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:chonky:1:1:3859:3859] [20230214:155101:1676389861:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2091] fp = 0x30e4a00
[EC_DRHOOK:chonky:1:1:3859:3859] [20230214:155101:1676389861:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:chonky:1:1:3859:3859] [20230214:155101:1676389861:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:chonky:1:1:3859:3859] [20230214:155101:1676389861:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:chonky:1:1:3859:3859] [20230214:155101:1676389861:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:chonky:1:1:3859:3859] [20230214:155101:1676389861:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:chonky:1:1:3859:3859] [20230214:155101:1676389861:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:chonky:1:1:3859:3859] [20230214:155101:1676389861:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:chonky:1:1:3859:3859] [20230214:155101:1676389861:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:chonky:1:1:3859:3859] [20230214:155101:1676389861:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:chonky:1:1:3859:3859] [20230214:155101:1676389861:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:chonky:1:1:3859:3859] [20230214:155101:1676389861:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:chonky:1:1:3859:3859] [20230214:155101:1676389861:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:chonky:1:1:3859:3859] [20230214:155101:1676389861:0.000] [ignore_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:chonky:1:1:3859:3859] [20230214:155101:1676389861:0.001] [restore_default_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:chonky:1:1:3859:3859] [20230214:155101:1676389861:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:chonky:1:1:3859:3859] [20230214:155101:1676389861:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f520 (old at (nil))
[EC_DRHOOK:chonky:1:1:3859:3859] [20230214:155101:1676389861:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:chonky:1:1:3859:3859] [20230214:155101:1676389861:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f520 (old at (nil))
[EC_DRHOOK:chonky:1:1:3859:3859] [20230214:155101:1676389861:0.001] [trapfpe_treatment@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:chonky:1:1:3859:3859] [20230214:155101:1676389861:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:chonky:1:1:3859:3859] [20230214:155101:1676389861:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:chonky:1:1:3859:3859] [20230214:155101:1676389861:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f520 (old at (nil))
[EC_DRHOOK:chonky:1:1:3859:3859] [20230214:155101:1676389861:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:chonky:1:1:3859:3859] [20230214:155101:1676389861:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:chonky:1:1:3859:3859] [20230214:155101:1676389861:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:chonky:1:1:3859:3859] [20230214:155101:1676389861:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f520 (old at (nil))
[EC_DRHOOK:chonky:1:1:3859:3859] [20230214:155101:1676389861:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f520 (old at (nil))
[EC_DRHOOK:chonky:1:1:3859:3859] [20230214:155101:1676389861:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f520 (old at (nil))
[EC_DRHOOK:chonky:1:1:3859:3859] [20230214:155101:1676389861:0.001] [catch_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
15:52:21 STEP 768 H= 192:00 +CPU= 47.831
15:53:12 STEP 769 H= 192:15 +CPU= 49.824
15:54:03 STEP 770 H= 192:30 +CPU= 50.418
15:54:47 STEP 771 H= 192:45 +CPU= 44.067
15:55:46 STEP 772 H= 193:00 +CPU= 55.935
15:56:36 STEP 773 H= 193:15 +CPU= 48.172
15:57:18 STEP 774 H= 193:30 +CPU= 42.145
15:58:04 STEP 775 H= 193:45 +CPU= 45.166
15:58:49 STEP 776 H= 194:00 +CPU= 45.112
15:59:36 STEP 777 H= 194:15 +CPU= 46.659
16:00:26 STEP 778 H= 194:30 +CPU= 49.661
16:01:14 STEP 779 H= 194:45 +CPU= 47.503
16:02:15 STEP 780 H= 195:00 +CPU= 60.574
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMGGh7zg+000780
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMSHh7zg+000780
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMUAh7zg+000780
16:03:05 STEP 781 H= 195:15 +CPU= 49.724
16:03:53 STEP 782 H= 195:30 +CPU= 47.976
16:04:42 STEP 783 H= 195:45 +CPU= 48.750
16:05:35 STEP 784 H= 196:00 +CPU= 53.222
16:06:24 STEP 785 H= 196:15 +CPU= 48.762
16:07:09 STEP 786 H= 196:30 +CPU= 44.422
16:07:57 STEP 787 H= 196:45 +CPU= 47.349
16:08:52 STEP 788 H= 197:00 +CPU= 53.347
16:09:43 STEP 789 H= 197:15 +CPU= 50.819
16:10:35 STEP 790 H= 197:30 +CPU= 50.962
16:11:40 STEP 791 H= 197:45 +CPU= 52.837
16:12:46 STEP 792 H= 198:00 +CPU= 63.344
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMGGh7zg+000792
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMSHh7zg+000792
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMUAh7zg+000792
16:13:36 STEP 793 H= 198:15 +CPU= 49.405
16:14:26 STEP 794 H= 198:30 +CPU= 49.123
16:15:15 STEP 795 H= 198:45 +CPU= 48.040
16:16:08 STEP 796 H= 199:00 +CPU= 52.037
16:17:00 STEP 797 H= 199:15 +CPU= 51.738
16:17:52 STEP 798 H= 199:30 +CPU= 49.076
16:18:50 STEP 799 H= 199:45 +CPU= 49.637
16:19:45 STEP 800 H= 200:00 +CPU= 51.294
16:20:42 STEP 801 H= 200:15 +CPU= 52.111
16:21:34 STEP 802 H= 200:30 +CPU= 49.710
16:22:25 STEP 803 H= 200:45 +CPU= 48.678
16:23:31 STEP 804 H= 201:00 +CPU= 63.472
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMGGh7zg+000804
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMSHh7zg+000804
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMUAh7zg+000804
16:24:24 STEP 805 H= 201:15 +CPU= 52.132
16:25:16 STEP 806 H= 201:30 +CPU= 50.512
16:26:14 STEP 807 H= 201:45 +CPU= 51.934
16:27:09 STEP 808 H= 202:00 +CPU= 51.388
16:28:07 STEP 809 H= 202:15 +CPU= 52.669
16:29:00 STEP 810 H= 202:30 +CPU= 51.227
16:29:54 STEP 811 H= 202:45 +CPU= 50.228
16:30:54 STEP 812 H= 203:00 +CPU= 54.800
16:31:51 STEP 813 H= 203:15 +CPU= 53.261
16:32:45 STEP 814 H= 203:30 +CPU= 51.980
16:33:37 STEP 815 H= 203:45 +CPU= 49.226
16:34:51 STEP 816 H= 204:00 +CPU= 69.507
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMGGh7zg+000816
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMSHh7zg+000816
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMUAh7zg+000816
16:35:43 STEP 817 H= 204:15 +CPU= 50.316
16:36:40 STEP 818 H= 204:30 +CPU= 51.031
16:37:31 STEP 819 H= 204:45 +CPU= 48.998
16:38:30 STEP 820 H= 205:00 +CPU= 54.346
16:39:23 STEP 821 H= 205:15 +CPU= 50.249
16:40:18 STEP 822 H= 205:30 +CPU= 51.627
16:41:14 STEP 823 H= 205:45 +CPU= 51.342
16:42:09 STEP 824 H= 206:00 +CPU= 51.337
16:43:03 STEP 825 H= 206:15 +CPU= 51.127
16:43:57 STEP 826 H= 206:30 +CPU= 51.544
16:44:46 STEP 827 H= 206:45 +CPU= 48.548
16:45:54 STEP 828 H= 207:00 +CPU= 65.021
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMGGh7zg+000828
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMSHh7zg+000828
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMUAh7zg+000828
16:46:44 STEP 829 H= 207:15 +CPU= 49.700
16:47:31 STEP 830 H= 207:30 +CPU= 46.911
16:48:20 STEP 831 H= 207:45 +CPU= 49.358
16:49:13 STEP 832 H= 208:00 +CPU= 51.665
16:50:01 STEP 833 H= 208:15 +CPU= 47.759
16:50:54 STEP 834 H= 208:30 +CPU= 51.983
16:51:42 STEP 835 H= 208:45 +CPU= 47.285
16:52:32 STEP 836 H= 209:00 +CPU= 50.321
16:53:22 STEP 837 H= 209:15 +CPU= 49.048
16:54:10 STEP 838 H= 209:30 +CPU= 47.701
16:55:01 STEP 839 H= 209:45 +CPU= 50.927
16:56:07 STEP 840 H= 210:00 +CPU= 64.470
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMGGh7zg+000840
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMSHh7zg+000840
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMUAh7zg+000840
16:57:01 STEP 841 H= 210:15 +CPU= 52.673
16:57:53 STEP 842 H= 210:30 +CPU= 50.754
16:58:46 STEP 843 H= 210:45 +CPU= 52.049
16:59:40 STEP 844 H= 211:00 +CPU= 52.488
17:00:34 STEP 845 H= 211:15 +CPU= 51.590
17:01:20 STEP 846 H= 211:30 +CPU= 46.431
17:02:05 STEP 847 H= 211:45 +CPU= 44.228
17:02:56 STEP 848 H= 212:00 +CPU= 51.058
17:03:46 STEP 849 H= 212:15 +CPU= 49.426
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.11_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a01h
(argv4) batchid: 991
(argv5) wuid: 12206840
(argv6) fclen: 15
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/22
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.11
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206840
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/22/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/22/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_d887486f6a3aabe4c55374c1725be1de to: /var/lib/boinc-client/slots/22/oifs_43r3_bl_a01h_2016092300_15_991_12206840.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/22/oifs_43r3_bl_a01h_2016092300_15_991_12206840.zip
ic_ancil_file: ic_ancil_12206840
ifsdata_file: ifsdata_12206840
climate_data_file: clim_data_12206840
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /var/lib/boinc-client/slots/22/ic_ancil_12206840.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/22/ic_ancil_12206840.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_155e44070232d2cfdcf7069aabe5bcf7 to: /var/lib/boinc-client/slots/22/ifsdata/ifsdata_12206840.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/22/ifsdata/ifsdata_12206840.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/22/159l_2/clim_data_12206840.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/22/159l_2/clim_data_12206840.zip
Checking for progress XML file: /var/lib/boinc-client/slots/22/progress_file_12206840.xml
Opened progress file ok : /var/lib/boinc-client/slots/22/progress_file_12206840.xml
-- Model is restarting --
Adjusting last_iter, 850, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/22/progress_file_12206840.xml
last_cpu_time: 52719
upload_file_number: 8
last_iter: 817
last_upload: 691200
model_completed: 0
total_length_of_simulation: 1296000
result_base_name: oifs_43r3_bl_a01h_2016092300_15_991_12206840_0_r968148541
The child process has been launched with process id: 3860
Executing the command: /var/lib/boinc-client/slots/22/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:chonky:1:1:3860:3860] [20230214:185204:1676400724:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2091] fp = 0x30e4a00
[EC_DRHOOK:chonky:1:1:3860:3860] [20230214:185204:1676400724:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:chonky:1:1:3860:3860] [20230214:185204:1676400724:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:chonky:1:1:3860:3860] [20230214:185204:1676400724:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:chonky:1:1:3860:3860] [20230214:185204:1676400724:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:chonky:1:1:3860:3860] [20230214:185204:1676400724:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:chonky:1:1:3860:3860] [20230214:185204:1676400724:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:chonky:1:1:3860:3860] [20230214:185204:1676400724:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:chonky:1:1:3860:3860] [20230214:185204:1676400724:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:chonky:1:1:3860:3860] [20230214:185204:1676400724:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:chonky:1:1:3860:3860] [20230214:185204:1676400724:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:chonky:1:1:3860:3860] [20230214:185204:1676400724:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:chonky:1:1:3860:3860] [20230214:185204:1676400724:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:chonky:1:1:3860:3860] [20230214:185204:1676400724:0.000] [ignore_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:chonky:1:1:3860:3860] [20230214:185204:1676400724:0.000] [restore_default_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:chonky:1:1:3860:3860] [20230214:185204:1676400724:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:chonky:1:1:3860:3860] [20230214:185204:1676400724:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f520 (old at (nil))
[EC_DRHOOK:chonky:1:1:3860:3860] [20230214:185204:1676400724:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:chonky:1:1:3860:3860] [20230214:185204:1676400724:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f520 (old at (nil))
[EC_DRHOOK:chonky:1:1:3860:3860] [20230214:185204:1676400724:0.001] [trapfpe_treatment@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:chonky:1:1:3860:3860] [20230214:185204:1676400724:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:chonky:1:1:3860:3860] [20230214:185204:1676400724:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:chonky:1:1:3860:3860] [20230214:185204:1676400724:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f520 (old at (nil))
[EC_DRHOOK:chonky:1:1:3860:3860] [20230214:185204:1676400724:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:chonky:1:1:3860:3860] [20230214:185204:1676400724:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:chonky:1:1:3860:3860] [20230214:185204:1676400724:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:chonky:1:1:3860:3860] [20230214:185204:1676400724:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f520 (old at (nil))
[EC_DRHOOK:chonky:1:1:3860:3860] [20230214:185204:1676400724:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f520 (old at (nil))
[EC_DRHOOK:chonky:1:1:3860:3860] [20230214:185204:1676400724:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f520 (old at (nil))
[EC_DRHOOK:chonky:1:1:3860:3860] [20230214:185204:1676400724:0.001] [catch_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
18:54:21 STEP 816 H= 204:00 +CPU= 55.939
18:55:11 STEP 817 H= 204:15 +CPU= 49.651
18:56:05 STEP 818 H= 204:30 +CPU= 52.957
18:57:01 STEP 819 H= 204:45 +CPU= 53.697
18:57:52 STEP 820 H= 205:00 +CPU= 50.425
18:58:42 STEP 821 H= 205:15 +CPU= 49.553
18:59:32 STEP 822 H= 205:30 +CPU= 49.522
19:00:20 STEP 823 H= 205:45 +CPU= 48.411
19:01:08 STEP 824 H= 206:00 +CPU= 47.381
19:01:55 STEP 825 H= 206:15 +CPU= 46.405
19:02:38 STEP 826 H= 206:30 +CPU= 42.856
19:03:22 STEP 827 H= 206:45 +CPU= 43.628
19:04:25 STEP 828 H= 207:00 +CPU= 63.272
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMGGh7zg+000828
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMSHh7zg+000828
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMUAh7zg+000828
19:05:08 STEP 829 H= 207:15 +CPU= 42.297
19:05:51 STEP 830 H= 207:30 +CPU= 42.911
19:06:35 STEP 831 H= 207:45 +CPU= 43.860
19:07:24 STEP 832 H= 208:00 +CPU= 48.931
19:08:12 STEP 833 H= 208:15 +CPU= 47.713
19:08:59 STEP 834 H= 208:30 +CPU= 46.308
19:09:51 STEP 835 H= 208:45 +CPU= 51.961
19:10:47 STEP 836 H= 209:00 +CPU= 53.749
19:11:45 STEP 837 H= 209:15 +CPU= 54.276
19:12:42 STEP 838 H= 209:30 +CPU= 53.887
19:13:36 STEP 839 H= 209:45 +CPU= 52.705
19:14:46 STEP 840 H= 210:00 +CPU= 66.733
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMGGh7zg+000840
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMSHh7zg+000840
Moving to projects directory: /var/lib/boinc-client/slots/22/ICMUAh7zg+000840
19:15:39 STEP 841 H= 210:15 +CPU= 51.820
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.11_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a01h
(argv4) batchid: 991
(argv5) wuid: 12206840
(argv6) fclen: 15
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/22
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.11
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206840
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/22/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/22/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_d887486f6a3aabe4c55374c1725be1de to: /var/lib/boinc-client/slots/22/oifs_43r3_bl_a01h_2016092300_15_991_12206840.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/22/oifs_43r3_bl_a01h_2016092300_15_991_12206840.zip
ic_ancil_file: ic_ancil_12206840
ifsdata_file: ifsdata_12206840
climate_data_file: clim_data_12206840
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /var/lib/boinc-client/slots/22/ic_ancil_12206840.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/22/ic_ancil_12206840.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_155e44070232d2cfdcf7069aabe5bcf7 to: /var/lib/boinc-client/slots/22/ifsdata/ifsdata_12206840.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/22/ifsdata/ifsdata_12206840.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/22/159l_2/clim_data_12206840.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/22/159l_2/clim_data_12206840.zip
Checking for progress XML file: /var/lib/boinc-client/slots/22/progress_file_12206840.xml
Opened progress file ok : /var/lib/boinc-client/slots/22/progress_file_12206840.xml
-- Model is restarting --
Adjusting last_iter, 842, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/22/progress_file_12206840.xml
last_cpu_time: 54087
upload_file_number: 8
last_iter: 817
last_upload: 691200
model_completed: 0
total_length_of_simulation: 1296000
result_base_name: oifs_43r3_bl_a01h_2016092300_15_991_12206840_0_r968148541
The child process has been launched with process id: 3879
Executing the command: /var/lib/boinc-client/slots/22/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:chonky:1:1:3879:3879] [20230214:192219:1676402539:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2091] fp = 0x30e4a00
[EC_DRHOOK:chonky:1:1:3879:3879] [20230214:192219:1676402539:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:chonky:1:1:3879:3879] [20230214:192219:1676402539:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:chonky:1:1:3879:3879] [20230214:192219:1676402539:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:chonky:1:1:3879:3879] [20230214:192219:1676402539:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:chonky:1:1:3879:3879] [20230214:192219:1676402539:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:chonky:1:1:3879:3879] [20230214:192219:1676402539:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:chonky:1:1:3879:3879] [20230214:192219:1676402539:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:chonky:1:1:3879:3879] [20230214:192219:1676402539:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:chonky:1:1:3879:3879] [20230214:192219:1676402539:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:chonky:1:1:3879:3879] [20230214:192219:1676402539:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:chonky:1:1:3879:3879] [20230214:192219:1676402539:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:chonky:1:1:3879:3879] [20230214:192219:1676402539:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:chonky:1:1:3879:3879] [20230214:192219:1676402539:0.000] [ignore_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:chonky:1:1:3879:3879] [20230214:192219:1676402539:0.000] [restore_default_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:chonky:1:1:3879:3879] [20230214:192219:1676402539:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:chonky:1:1:3879:3879] [20230214:192219:1676402539:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f520 (old at (nil))
[EC_DRHOOK:chonky:1:1:3879:3879] [20230214:192219:1676402539:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:chonky:1:1:3879:3879] [20230214:192219:1676402539:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f520 (old at (nil))
[EC_DRHOOK:chonky:1:1:3879:3879] [20230214:192219:1676402539:0.000] [trapfpe_treatment@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:chonky:1:1:3879:3879] [20230214:192219:1676402539:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:chonky:1:1:3879:3879] [20230214:192219:1676402539:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:chonky:1:1:3879:3879] [20230214:192219:1676402539:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f520 (old at (nil))
[EC_DRHOOK:chonky:1:1:3879:3879] [20230214:192219:1676402539:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:chonky:1:1:3879:3879] [20230214:192219:1676402539:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:chonky:1:1:3879:3879] [20230214:192219:1676402539:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:chonky:1:1:3879:3879] [20230214:192219:1676402539:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f520 (old at (nil))
[EC_DRHOOK:chonky:1:1:3879:3879] [20230214:192219:1676402539:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f520 (old at (nil))
[EC_DRHOOK:chonky:1:1:3879:3879] [20230214:192219:1676402539:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f520 (old at (nil))
[EC_DRHOOK:chonky:1:1:3879:3879] [20230214:192219:1676402539:0.001] [catch_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
19:24:33 STEP 816 H= 204:00 +CPU= 52.374
19:25:16 STEP 817 H= 204:15 +CPU= 43.298
Abort request received from BOINC client, ending the child process
</stderr_txt>
]]>
