climateprediction.net home page
Task 22254484

Task 22254484

Name oifs_43r3_ps_0524_1984050100_123_953_12170168_0
Workunit 12170168
Created 21 Dec 2022, 3:37:30 UTC
Sent 8 Jan 2023, 5:33:46 UTC
Report deadline 7 Feb 2023, 5:33:46 UTC
Received 10 Jan 2023, 17:15:19 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1537334
Run time 19 hours 49 min 31 sec
CPU time 19 hours 31 min 27 sec
Validate state Invalid
Credit 9,434.88
Device peak FLOPS 3.47 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,417.91 MB
Peak swap size 4,978.08 MB
Peak disk usage 7,179.26 MB
Stderr
<core_client_version>7.20.5</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
lient/slots/19/ICMGGhq0f+001560
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001560
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170168/ICMGGhq0f+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170168/ICMSHhq0f+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170168/ICMUAhq0f+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170168/ICMGGhq0f+001548
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170168/ICMSHhq0f+001548
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170168/ICMUAhq0f+001548
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0524_1984050100_123_953_12170168_0_r1550041224_64.zip
Uploading the intermediate file: upload_file_64.zip
  18:17:58 STEP 1561 H=1561:00 +CPU= 24.159
  18:18:23 STEP 1562 H=1562:00 +CPU= 24.756
  18:19:01 STEP 1563 H=1563:00 +CPU= 38.007
  18:19:25 STEP 1564 H=1564:00 +CPU= 24.284
  18:19:49 STEP 1565 H=1565:00 +CPU= 23.786
  18:20:28 STEP 1566 H=1566:00 +CPU= 39.178
  18:20:52 STEP 1567 H=1567:00 +CPU= 23.781
  18:21:16 STEP 1568 H=1568:00 +CPU= 24.508
  18:21:55 STEP 1569 H=1569:00 +CPU= 38.318
  18:22:19 STEP 1570 H=1570:00 +CPU= 24.545
  18:22:43 STEP 1571 H=1571:00 +CPU= 23.677
  18:23:25 STEP 1572 H=1572:00 +CPU= 41.950
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001572
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001572
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001572
  18:23:49 STEP 1573 H=1573:00 +CPU= 23.796
  18:24:13 STEP 1574 H=1574:00 +CPU= 24.540
  18:24:52 STEP 1575 H=1575:00 +CPU= 38.296
  18:25:16 STEP 1576 H=1576:00 +CPU= 24.415
  18:25:40 STEP 1577 H=1577:00 +CPU= 23.753
  18:26:19 STEP 1578 H=1578:00 +CPU= 38.819
  18:26:42 STEP 1579 H=1579:00 +CPU= 23.664
  18:27:07 STEP 1580 H=1580:00 +CPU= 24.292
  18:27:45 STEP 1581 H=1581:00 +CPU= 37.806
  18:28:09 STEP 1582 H=1582:00 +CPU= 23.961
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1984050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0524
(argv4) batchid: 953
(argv5) wuid: 12170168
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/19
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170168
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/19/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/19/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_9cfc2ce339eceed94293ba4cf4560c67 to: /var/lib/boinc-client/slots/19/oifs_43r3_ps_0524_1984050100_123_953_12170168.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/19/oifs_43r3_ps_0524_1984050100_123_953_12170168.zip
ic_ancil_file: ic_ancil_12170168
ifsdata_file: ifsdata_12170168
climate_data_file: clim_data_12170168
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_926f6e33cb192b14883e72faaafe4013 to: /var/lib/boinc-client/slots/19/ic_ancil_12170168.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/19/ic_ancil_12170168.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_63db628b747bb7c60924e4ab97d67e84 to: /var/lib/boinc-client/slots/19/ifsdata/ifsdata_12170168.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/19/ifsdata/ifsdata_12170168.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/19/159l_2/clim_data_12170168.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/19/159l_2/clim_data_12170168.zip
Checking for progress XML file: /var/lib/boinc-client/slots/19/progress_file_12170168.xml
Opened progress file ok : /var/lib/boinc-client/slots/19/progress_file_12170168.xml
-- Model is restarting --
Adjusting last_iter, 1583, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/19/progress_file_12170168.xml
last_cpu_time: 57657
upload_file_number: 65
last_iter: 1559
last_upload: 5616000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0524_1984050100_123_953_12170168_0_r1550041224
The child process has been launched with process id: 2698
Executing the command: /var/lib/boinc-client/slots/19/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:egon-VirtualBox:1:1:2698:2698] [20230109:183130:1673285490:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:egon-VirtualBox:1:1:2698:2698] [20230109:183130:1673285490:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:egon-VirtualBox:1:1:2698:2698] [20230109:183130:1673285490:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:egon-VirtualBox:1:1:2698:2698] [20230109:183130:1673285490:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:egon-VirtualBox:1:1:2698:2698] [20230109:183130:1673285490:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:egon-VirtualBox:1:1:2698:2698] [20230109:183130:1673285490:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:egon-VirtualBox:1:1:2698:2698] [20230109:183130:1673285490:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:egon-VirtualBox:1:1:2698:2698] [20230109:183130:1673285490:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:egon-VirtualBox:1:1:2698:2698] [20230109:183130:1673285490:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:egon-VirtualBox:1:1:2698:2698] [20230109:183130:1673285490:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:egon-VirtualBox:1:1:2698:2698] [20230109:183130:1673285490:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:egon-VirtualBox:1:1:2698:2698] [20230109:183130:1673285490:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:egon-VirtualBox:1:1:2698:2698] [20230109:183130:1673285490:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:egon-VirtualBox:1:1:2698:2698] [20230109:183130:1673285490:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:egon-VirtualBox:1:1:2698:2698] [20230109:183130:1673285490:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:egon-VirtualBox:1:1:2698:2698] [20230109:183130:1673285490:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:egon-VirtualBox:1:1:2698:2698] [20230109:183130:1673285490:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:egon-VirtualBox:1:1:2698:2698] [20230109:183130:1673285490:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:egon-VirtualBox:1:1:2698:2698] [20230109:183130:1673285490:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:egon-VirtualBox:1:1:2698:2698] [20230109:183130:1673285490:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:egon-VirtualBox:1:1:2698:2698] [20230109:183130:1673285490:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:egon-VirtualBox:1:1:2698:2698] [20230109:183130:1673285490:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:egon-VirtualBox:1:1:2698:2698] [20230109:183130:1673285490:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:egon-VirtualBox:1:1:2698:2698] [20230109:183130:1673285490:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:egon-VirtualBox:1:1:2698:2698] [20230109:183130:1673285490:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:egon-VirtualBox:1:1:2698:2698] [20230109:183130:1673285490:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:egon-VirtualBox:1:1:2698:2698] [20230109:183130:1673285490:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:egon-VirtualBox:1:1:2698:2698] [20230109:183130:1673285490:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:egon-VirtualBox:1:1:2698:2698] [20230109:183130:1673285490:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:egon-VirtualBox:1:1:2698:2698] [20230109:183130:1673285490:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  18:32:40 STEP 1560 H=1560:00 +CPU= 40.472
  18:33:09 STEP 1561 H=1561:00 +CPU= 27.949
  18:33:44 STEP 1562 H=1562:00 +CPU= 29.682
  18:34:43 STEP 1563 H=1563:00 +CPU= 47.526
  18:35:20 STEP 1564 H=1564:00 +CPU= 30.657
  18:35:52 STEP 1565 H=1565:00 +CPU= 28.915
  18:36:40 STEP 1566 H=1566:00 +CPU= 44.925
  18:37:11 STEP 1567 H=1567:00 +CPU= 28.862
  18:37:42 STEP 1568 H=1568:00 +CPU= 28.227
  18:38:30 STEP 1569 H=1569:00 +CPU= 45.218
  18:39:01 STEP 1570 H=1570:00 +CPU= 28.498
  18:39:34 STEP 1571 H=1571:00 +CPU= 30.006
  18:40:27 STEP 1572 H=1572:00 +CPU= 49.390
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001572
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001572
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001572
  18:40:56 STEP 1573 H=1573:00 +CPU= 27.502
  18:41:27 STEP 1574 H=1574:00 +CPU= 28.612
  18:42:14 STEP 1575 H=1575:00 +CPU= 44.199
  18:42:46 STEP 1576 H=1576:00 +CPU= 28.752
  18:43:15 STEP 1577 H=1577:00 +CPU= 27.496
  18:44:03 STEP 1578 H=1578:00 +CPU= 44.911
  18:44:32 STEP 1579 H=1579:00 +CPU= 27.525
  18:45:02 STEP 1580 H=1580:00 +CPU= 28.223
  18:45:49 STEP 1581 H=1581:00 +CPU= 43.941
  18:46:19 STEP 1582 H=1582:00 +CPU= 27.994
  18:46:49 STEP 1583 H=1583:00 +CPU= 28.395
  18:47:45 STEP 1584 H=1584:00 +CPU= 50.620
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001584
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001584
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170168/ICMGGhq0f+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170168/ICMSHhq0f+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170168/ICMUAhq0f+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170168/ICMGGhq0f+001572
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170168/ICMSHhq0f+001572
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170168/ICMUAhq0f+001572
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0524_1984050100_123_953_12170168_0_r1550041224_65.zip
Uploading the intermediate file: upload_file_65.zip
  18:48:15 STEP 1585 H=1585:00 +CPU= 27.357
  18:48:45 STEP 1586 H=1586:00 +CPU= 28.068
  18:49:34 STEP 1587 H=1587:00 +CPU= 45.524
  18:50:04 STEP 1588 H=1588:00 +CPU= 28.892
  18:50:33 STEP 1589 H=1589:00 +CPU= 27.309
  18:51:21 STEP 1590 H=1590:00 +CPU= 44.458
  18:51:52 STEP 1591 H=1591:00 +CPU= 28.500
  18:52:22 STEP 1592 H=1592:00 +CPU= 28.468
  18:53:11 STEP 1593 H=1593:00 +CPU= 45.255
  18:53:41 STEP 1594 H=1594:00 +CPU= 27.996
  18:54:11 STEP 1595 H=1595:00 +CPU= 27.245
  18:55:03 STEP 1596 H=1596:00 +CPU= 48.620
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001596
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001596
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001596
  18:55:33 STEP 1597 H=1597:00 +CPU= 27.947
  18:56:03 STEP 1598 H=1598:00 +CPU= 28.116
  18:56:50 STEP 1599 H=1599:00 +CPU= 43.653
  18:57:23 STEP 1600 H=1600:00 +CPU= 30.684
  18:57:53 STEP 1601 H=1601:00 +CPU= 28.077
  18:58:42 STEP 1602 H=1602:00 +CPU= 44.941
  18:59:14 STEP 1603 H=1603:00 +CPU= 29.688
  18:59:46 STEP 1604 H=1604:00 +CPU= 29.864
  19:00:35 STEP 1605 H=1605:00 +CPU= 45.652
  19:01:14 STEP 1606 H=1606:00 +CPU= 35.459
  19:01:48 STEP 1607 H=1607:00 +CPU= 31.184
  19:02:56 STEP 1608 H=1608:00 +CPU= 58.029
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001608
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001608
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170168/ICMGGhq0f+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170168/ICMSHhq0f+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170168/ICMUAhq0f+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170168/ICMGGhq0f+001596
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170168/ICMSHhq0f+001596
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170168/ICMUAhq0f+001596
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0524_1984050100_123_953_12170168_0_r1550041224_66.zip
Uploading the intermediate file: upload_file_66.zip
  19:03:38 STEP 1609 H=1609:00 +CPU= 33.473
  19:04:12 STEP 1610 H=1610:00 +CPU= 30.718
  19:05:15 STEP 1611 H=1611:00 +CPU= 54.855
  19:05:48 STEP 1612 H=1612:00 +CPU= 29.935
  19:06:21 STEP 1613 H=1613:00 +CPU= 29.867
  19:07:12 STEP 1614 H=1614:00 +CPU= 46.877
  19:07:45 STEP 1615 H=1615:00 +CPU= 29.918
  19:08:19 STEP 1616 H=1616:00 +CPU= 31.542
  19:09:12 STEP 1617 H=1617:00 +CPU= 48.802
  19:09:44 STEP 1618 H=1618:00 +CPU= 29.878
  19:10:21 STEP 1619 H=1619:00 +CPU= 30.182
  19:11:22 STEP 1620 H=1620:00 +CPU= 55.412
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001620
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001620
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001620
  19:11:54 STEP 1621 H=1621:00 +CPU= 29.183
  19:12:25 STEP 1622 H=1622:00 +CPU= 29.045
  19:13:13 STEP 1623 H=1623:00 +CPU= 44.194
  19:13:43 STEP 1624 H=1624:00 +CPU= 28.643
  19:14:13 STEP 1625 H=1625:00 +CPU= 27.711
  19:15:09 STEP 1626 H=1626:00 +CPU= 51.914
  19:15:46 STEP 1627 H=1627:00 +CPU= 33.343
  19:16:20 STEP 1628 H=1628:00 +CPU= 31.470
  19:17:11 STEP 1629 H=1629:00 +CPU= 47.107
  19:17:44 STEP 1630 H=1630:00 +CPU= 30.472
  19:18:18 STEP 1631 H=1631:00 +CPU= 30.729
  19:19:16 STEP 1632 H=1632:00 +CPU= 53.019
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001632
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001632
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170168/ICMGGhq0f+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170168/ICMSHhq0f+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170168/ICMUAhq0f+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170168/ICMGGhq0f+001620
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170168/ICMSHhq0f+001620
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170168/ICMUAhq0f+001620
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0524_1984050100_123_953_12170168_0_r1550041224_67.zip
Uploading the intermediate file: upload_file_67.zip
  19:19:48 STEP 1633 H=1633:00 +CPU= 28.279
  19:20:18 STEP 1634 H=1634:00 +CPU= 27.928
  19:21:04 STEP 1635 H=1635:00 +CPU= 43.291
  19:21:35 STEP 1636 H=1636:00 +CPU= 28.714
  19:22:12 STEP 1637 H=1637:00 +CPU= 33.832
  19:23:08 STEP 1638 H=1638:00 +CPU= 50.508
  19:23:39 STEP 1639 H=1639:00 +CPU= 28.945
  19:24:11 STEP 1640 H=1640:00 +CPU= 29.827
  19:25:02 STEP 1641 H=1641:00 +CPU= 46.854
  19:25:34 STEP 1642 H=1642:00 +CPU= 29.292
  19:26:08 STEP 1643 H=1643:00 +CPU= 31.499
  19:26:57 STEP 1644 H=1644:00 +CPU= 45.880
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001644
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001644
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001644
  19:27:26 STEP 1645 H=1645:00 +CPU= 27.137
  19:28:02 STEP 1646 H=1646:00 +CPU= 33.444
  19:28:52 STEP 1647 H=1647:00 +CPU= 46.100
  19:29:25 STEP 1648 H=1648:00 +CPU= 30.202
  19:29:57 STEP 1649 H=1649:00 +CPU= 29.063
  19:30:50 STEP 1650 H=1650:00 +CPU= 49.435
  19:31:23 STEP 1651 H=1651:00 +CPU= 30.769
  19:31:53 STEP 1652 H=1652:00 +CPU= 27.698
  19:32:55 STEP 1653 H=1653:00 +CPU= 53.589
  19:33:28 STEP 1654 H=1654:00 +CPU= 30.468
Suspend request received from the BOINC client, suspending the child process
[EC_DRHOOK:egon-VirtualBox:1:1:2698:2698] [20230109:221144:1673298704:13213.906] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 3996MB (heap), 4419MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:egon-VirtualBox:1:1:2698:2698] [20230109:221144:1673298704:13213.906] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:egon-VirtualBox:1:1:2698:2698] [20230109:221144:1673298704:13213.906] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f3c0 (old at (nil))
[EC_DRHOOK:egon-VirtualBox:1:1:2698:2698] [20230109:221144:1673298704:13213.906] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1
[EC_DRHOOK:egon-VirtualBox:1:1:2698:2698] [20230109:221144:1673298704:13213.906] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1
[EC_DRHOOK:egon-VirtualBox:1:1:2698:2698] [20230109:221144:1673298704:13213.906] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 3996 MB (maxheap), 4419 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:egon-VirtualBox:1:1:2698:2698] [20230109:221144:1673298704:13213.906] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:egon-VirtualBox:1:1:2698:2698] [20230109:221144:1673298704:13213.906] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:egon-VirtualBox:1:1:2698:2698] [20230109:221144:1673298704:13213.906] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:egon-VirtualBox:1:1:2698:2698] [20230109:221144:1673298704:13213.906] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:egon-VirtualBox:1:1:2698:2698] [20230109:221144:1673298704:13213.906] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:egon-VirtualBox:1:1:2698:2698] [20230109:221144:1673298704:13213.906] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :       CNT4 
[EC_DRHOOK:egon-VirtualBox:1:1:2698:2698] [20230109:221144:1673298704:13213.906] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :        STEPO 
[EC_DRHOOK:egon-VirtualBox:1:1:2698:2698] [20230109:221144:1673298704:13213.906] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :         SCAN2M 
[EC_DRHOOK:egon-VirtualBox:1:1:2698:2698] [20230109:221144:1673298704:13213.906] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :          GP_MODEL_HEAP 
[EC_DRHOOK:egon-VirtualBox:1:1:2698:2698] [20230109:221144:1673298704:13213.906] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :           GP_MODEL 
[EC_DRHOOK:egon-VirtualBox:1:1:2698:2698] [20230109:221144:1673298704:13213.906] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :            EC_PHYS_DRV 
[EC_DRHOOK:egon-VirtualBox:1:1:2698:2698] [20230109:221144:1673298704:13213.906] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :             >OMP-PHYSICS              (1001) 
[EC_DRHOOK:egon-VirtualBox:1:1:2698:2698] [20230109:221144:1673298704:13213.906] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :              EC_PHYS 
[EC_DRHOOK:egon-VirtualBox:1:1:2698:2698] [20230109:221144:1673298704:13213.906] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :               CALLPAR 
[EC_DRHOOK:egon-VirtualBox:1:1:2698:2698] [20230109:221144:1673298704:13213.906] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                SLTEND_LAYER 
[EC_DRHOOK:egon-VirtualBox:1:1:2698:2698] [20230109:221144:1673298704:13213.906] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                 SLTEND 
[EC_DRHOOK:egon-VirtualBox:1:1:2698:2698] [20230109:221144:1673298704:13213.906] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                  CUADJTQ 
[EC_DRHOOK:egon-VirtualBox:1:1:2698:2698] [20230109:221145:1673298705:13214.929] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1
[EC_DRHOOK:egon-VirtualBox:1:1:2698:2698] [20230109:221145:1673298705:13214.929] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1ce8bb0 for signal#15, nsigs = 1
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source             
oifs_43r3_model.e  0000000001CE916B  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000081FFF1  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DC9090  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001E3ED04  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000000402D7F  yomhook_mp_dr_hoo          58  yomhook.F90
oifs_43r3_model.e  0000000000FBCE3A  cuadjtq_                  549  cuadjtq.F90
oifs_43r3_model.e  000000000103E530  sltend_                   298  sltend.F90
oifs_43r3_model.e  000000000103CDC0  sltend_layer_              99  sltend_layer.F90
oifs_43r3_model.e  0000000000F410DA  callpar_                 1613  callpar.F90
oifs_43r3_model.e  0000000000E4940F  ec_phys_                  670  ec_phys.F90
oifs_43r3_model.e  0000000000E3610C  ec_phys_drv_              599  ec_phys_drv.F90
oifs_43r3_model.e  0000000000BFA3D3  gp_model_                 613  gp_model.F90
oifs_43r3_model.e  00000000012E9482  gp_model_heap_             74  gp_model_heap.F90
oifs_43r3_model.e  0000000000BBC1CE  scan2m_                   535  scan2m.F90
oifs_43r3_model.e  000000000057A664  stepo_                    327  stepo.F90
oifs_43r3_model.e  000000000055DEEC  cnt4_                    1133  cnt4.F90
oifs_43r3_model.e  00000000005471C9  cnt3_                     267  cnt3.F90
oifs_43r3_model.e  0000000000546412  cnt2_                      88  cnt2.F90
oifs_43r3_model.e  0000000000545E28  cnt1_                      92  cnt1.F90
oifs_43r3_model.e  000000000040708D  cnt0_                     146  cnt0.F90
oifs_43r3_model.e  000000000040220F  MAIN__                     96  master.F90
oifs_43r3_model.e  00000000004021A2  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCA390  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000040206E  Unknown               Unknown  Unknown
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1984050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0524
(argv4) batchid: 953
(argv5) wuid: 12170168
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/19
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170168
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/19/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/19/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_9cfc2ce339eceed94293ba4cf4560c67 to: /var/lib/boinc-client/slots/19/oifs_43r3_ps_0524_1984050100_123_953_12170168.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/19/oifs_43r3_ps_0524_1984050100_123_953_12170168.zip
ic_ancil_file: ic_ancil_12170168
ifsdata_file: ifsdata_12170168
climate_data_file: clim_data_12170168
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_926f6e33cb192b14883e72faaafe4013 to: /var/lib/boinc-client/slots/19/ic_ancil_12170168.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/19/ic_ancil_12170168.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_63db628b747bb7c60924e4ab97d67e84 to: /var/lib/boinc-client/slots/19/ifsdata/ifsdata_12170168.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/19/ifsdata/ifsdata_12170168.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/19/159l_2/clim_data_12170168.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/19/159l_2/clim_data_12170168.zip
Checking for progress XML file: /var/lib/boinc-client/slots/19/progress_file_12170168.xml
Opened progress file ok : /var/lib/boinc-client/slots/19/progress_file_12170168.xml
-- Model is restarting --
Adjusting last_iter, 1655, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/19/progress_file_12170168.xml
last_cpu_time: 61042
upload_file_number: 68
last_iter: 1631
last_upload: 5875200
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0524_1984050100_123_953_12170168_0_r1550041224
The child process has been launched with process id: 4137
Executing the command: /var/lib/boinc-client/slots/19/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:egon-VirtualBox:1:1:4137:4137] [20230110:144900:1673358540:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:egon-VirtualBox:1:1:4137:4137] [20230110:144900:1673358540:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:egon-VirtualBox:1:1:4137:4137] [20230110:144900:1673358540:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:egon-VirtualBox:1:1:4137:4137] [20230110:144900:1673358540:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:egon-VirtualBox:1:1:4137:4137] [20230110:144900:1673358540:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:egon-VirtualBox:1:1:4137:4137] [20230110:144900:1673358540:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:egon-VirtualBox:1:1:4137:4137] [20230110:144900:1673358540:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:egon-VirtualBox:1:1:4137:4137] [20230110:144900:1673358540:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:egon-VirtualBox:1:1:4137:4137] [20230110:144900:1673358540:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:egon-VirtualBox:1:1:4137:4137] [20230110:144900:1673358540:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:egon-VirtualBox:1:1:4137:4137] [20230110:144900:1673358540:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:egon-VirtualBox:1:1:4137:4137] [20230110:144900:1673358540:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:egon-VirtualBox:1:1:4137:4137] [20230110:144900:1673358540:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:egon-VirtualBox:1:1:4137:4137] [20230110:144900:1673358540:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:egon-VirtualBox:1:1:4137:4137] [20230110:144900:1673358540:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:egon-VirtualBox:1:1:4137:4137] [20230110:144900:1673358540:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:egon-VirtualBox:1:1:4137:4137] [20230110:144900:1673358540:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:egon-VirtualBox:1:1:4137:4137] [20230110:144900:1673358540:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:egon-VirtualBox:1:1:4137:4137] [20230110:144900:1673358540:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:egon-VirtualBox:1:1:4137:4137] [20230110:144900:1673358540:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:egon-VirtualBox:1:1:4137:4137] [20230110:144900:1673358540:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:egon-VirtualBox:1:1:4137:4137] [20230110:144900:1673358540:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:egon-VirtualBox:1:1:4137:4137] [20230110:144900:1673358540:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:egon-VirtualBox:1:1:4137:4137] [20230110:144900:1673358540:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:egon-VirtualBox:1:1:4137:4137] [20230110:144900:1673358540:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:egon-VirtualBox:1:1:4137:4137] [20230110:144900:1673358540:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:egon-VirtualBox:1:1:4137:4137] [20230110:144900:1673358540:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:egon-VirtualBox:1:1:4137:4137] [20230110:144900:1673358540:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:egon-VirtualBox:1:1:4137:4137] [20230110:144900:1673358540:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:egon-VirtualBox:1:1:4137:4137] [20230110:144900:1673358540:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  14:50:42 STEP 1632 H=1632:00 +CPU= 69.965
  14:51:45 STEP 1633 H=1633:00 +CPU= 41.297
  14:52:27 STEP 1634 H=1634:00 +CPU= 33.516
  14:53:17 STEP 1635 H=1635:00 +CPU= 46.550
  14:53:48 STEP 1636 H=1636:00 +CPU= 28.819
  14:54:18 STEP 1637 H=1637:00 +CPU= 27.726
  14:55:12 STEP 1638 H=1638:00 +CPU= 45.668
  14:55:42 STEP 1639 H=1639:00 +CPU= 27.837
  14:56:13 STEP 1640 H=1640:00 +CPU= 28.605
  14:57:05 STEP 1641 H=1641:00 +CPU= 47.913
  14:57:44 STEP 1642 H=1642:00 +CPU= 33.991
  14:58:18 STEP 1643 H=1643:00 +CPU= 30.611
  14:59:17 STEP 1644 H=1644:00 +CPU= 53.624
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001644
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001644
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001644
  14:59:49 STEP 1645 H=1645:00 +CPU= 29.742
  15:00:23 STEP 1646 H=1646:00 +CPU= 30.948
  15:01:15 STEP 1647 H=1647:00 +CPU= 47.633
  15:01:47 STEP 1648 H=1648:00 +CPU= 29.990
  15:02:19 STEP 1649 H=1649:00 +CPU= 29.178
  15:03:08 STEP 1650 H=1650:00 +CPU= 45.688
  15:03:38 STEP 1651 H=1651:00 +CPU= 28.157
  15:04:09 STEP 1652 H=1652:00 +CPU= 29.597
  15:04:58 STEP 1653 H=1653:00 +CPU= 45.270
  15:05:28 STEP 1654 H=1654:00 +CPU= 28.494
  15:05:59 STEP 1655 H=1655:00 +CPU= 28.681
  15:06:55 STEP 1656 H=1656:00 +CPU= 51.399
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001656
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001656
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170168/ICMGGhq0f+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170168/ICMSHhq0f+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170168/ICMUAhq0f+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170168/ICMGGhq0f+001644
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170168/ICMSHhq0f+001644
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170168/ICMUAhq0f+001644
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0524_1984050100_123_953_12170168_0_r1550041224_68.zip
Uploading the intermediate file: upload_file_68.zip
  15:07:28 STEP 1657 H=1657:00 +CPU= 28.450
  15:08:02 STEP 1658 H=1658:00 +CPU= 31.695
  15:08:51 STEP 1659 H=1659:00 +CPU= 45.396
  15:09:22 STEP 1660 H=1660:00 +CPU= 29.464
  15:09:52 STEP 1661 H=1661:00 +CPU= 27.808
  15:10:42 STEP 1662 H=1662:00 +CPU= 45.888
  15:11:17 STEP 1663 H=1663:00 +CPU= 31.511
  15:11:47 STEP 1664 H=1664:00 +CPU= 28.423
  15:12:37 STEP 1665 H=1665:00 +CPU= 46.089
  15:13:09 STEP 1666 H=1666:00 +CPU= 29.689
  15:13:39 STEP 1667 H=1667:00 +CPU= 28.129
  15:14:30 STEP 1668 H=1668:00 +CPU= 49.263
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001668
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001668
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001668
  15:14:58 STEP 1669 H=1669:00 +CPU= 27.675
  15:15:27 STEP 1670 H=1670:00 +CPU= 27.612
  15:16:12 STEP 1671 H=1671:00 +CPU= 43.745
  15:16:40 STEP 1672 H=1672:00 +CPU= 27.480
  15:17:09 STEP 1673 H=1673:00 +CPU= 27.864
  15:17:53 STEP 1674 H=1674:00 +CPU= 43.708
  15:18:21 STEP 1675 H=1675:00 +CPU= 27.140
  15:18:49 STEP 1676 H=1676:00 +CPU= 27.394
  15:19:34 STEP 1677 H=1677:00 +CPU= 44.221
  15:20:02 STEP 1678 H=1678:00 +CPU= 27.397
  15:20:31 STEP 1679 H=1679:00 +CPU= 27.586
  15:21:22 STEP 1680 H=1680:00 +CPU= 49.796
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001680
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001680
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170168/ICMGGhq0f+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170168/ICMSHhq0f+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170168/ICMUAhq0f+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170168/ICMGGhq0f+001668
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170168/ICMSHhq0f+001668
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170168/ICMUAhq0f+001668
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0524_1984050100_123_953_12170168_0_r1550041224_69.zip
Uploading the intermediate file: upload_file_69.zip
  15:21:49 STEP 1681 H=1681:00 +CPU= 26.577
  15:22:18 STEP 1682 H=1682:00 +CPU= 28.612
  15:23:02 STEP 1683 H=1683:00 +CPU= 43.080
  15:23:30 STEP 1684 H=1684:00 +CPU= 27.259
  15:23:58 STEP 1685 H=1685:00 +CPU= 26.918
  15:24:43 STEP 1686 H=1686:00 +CPU= 43.833
  15:25:10 STEP 1687 H=1687:00 +CPU= 27.091
  15:25:39 STEP 1688 H=1688:00 +CPU= 27.449
  15:26:23 STEP 1689 H=1689:00 +CPU= 43.146
  15:26:52 STEP 1690 H=1690:00 +CPU= 28.490
  15:27:22 STEP 1691 H=1691:00 +CPU= 29.443
  15:28:13 STEP 1692 H=1692:00 +CPU= 49.358
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001692
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001692
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001692
  15:28:42 STEP 1693 H=1693:00 +CPU= 28.222
  15:29:11 STEP 1694 H=1694:00 +CPU= 28.530
  15:29:56 STEP 1695 H=1695:00 +CPU= 42.830
  15:30:24 STEP 1696 H=1696:00 +CPU= 27.770
  15:30:57 STEP 1697 H=1697:00 +CPU= 32.044
  15:31:47 STEP 1698 H=1698:00 +CPU= 47.973
  15:32:15 STEP 1699 H=1699:00 +CPU= 28.190
  15:32:46 STEP 1700 H=1700:00 +CPU= 29.733
  15:33:33 STEP 1701 H=1701:00 +CPU= 45.615
  15:34:04 STEP 1702 H=1702:00 +CPU= 29.492
  15:34:35 STEP 1703 H=1703:00 +CPU= 29.696
  15:35:30 STEP 1704 H=1704:00 +CPU= 53.047
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001704
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001704
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170168/ICMGGhq0f+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170168/ICMSHhq0f+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170168/ICMUAhq0f+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170168/ICMGGhq0f+001692
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170168/ICMSHhq0f+001692
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170168/ICMUAhq0f+001692
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0524_1984050100_123_953_12170168_0_r1550041224_70.zip
Uploading the intermediate file: upload_file_70.zip
  15:35:55 STEP 1705 H=1705:00 +CPU= 24.907
  15:36:22 STEP 1706 H=1706:00 +CPU= 25.783
  15:37:03 STEP 1707 H=1707:00 +CPU= 40.104
  15:37:29 STEP 1708 H=1708:00 +CPU= 25.537
  15:37:54 STEP 1709 H=1709:00 +CPU= 24.931
  15:38:36 STEP 1710 H=1710:00 +CPU= 41.347
  15:39:02 STEP 1711 H=1711:00 +CPU= 25.052
  15:39:28 STEP 1712 H=1712:00 +CPU= 25.924
  15:40:09 STEP 1713 H=1713:00 +CPU= 40.009
  15:40:35 STEP 1714 H=1714:00 +CPU= 25.728
  15:41:01 STEP 1715 H=1715:00 +CPU= 24.917
  15:41:46 STEP 1716 H=1716:00 +CPU= 44.206
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001716
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001716
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001716
  15:42:12 STEP 1717 H=1717:00 +CPU= 25.019
  15:42:38 STEP 1718 H=1718:00 +CPU= 25.638
  15:43:19 STEP 1719 H=1719:00 +CPU= 40.163
  15:43:45 STEP 1720 H=1720:00 +CPU= 25.561
  15:44:11 STEP 1721 H=1721:00 +CPU= 25.252
  15:44:53 STEP 1722 H=1722:00 +CPU= 41.081
  15:45:18 STEP 1723 H=1723:00 +CPU= 24.987
  15:45:44 STEP 1724 H=1724:00 +CPU= 25.612
  15:46:25 STEP 1725 H=1725:00 +CPU= 40.165
  15:46:52 STEP 1726 H=1726:00 +CPU= 25.622
  15:47:18 STEP 1727 H=1727:00 +CPU= 25.723
  15:48:06 STEP 1728 H=1728:00 +CPU= 46.699
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001728
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001728
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170168/ICMGGhq0f+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170168/ICMSHhq0f+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170168/ICMUAhq0f+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170168/ICMGGhq0f+001716
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170168/ICMSHhq0f+001716
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170168/ICMUAhq0f+001716
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0524_1984050100_123_953_12170168_0_r1550041224_71.zip
Uploading the intermediate file: upload_file_71.zip
  15:48:31 STEP 1729 H=1729:00 +CPU= 24.915
  15:48:57 STEP 1730 H=1730:00 +CPU= 25.583
  15:49:38 STEP 1731 H=1731:00 +CPU= 40.088
  15:50:04 STEP 1732 H=1732:00 +CPU= 25.554
  15:50:30 STEP 1733 H=1733:00 +CPU= 25.376
  15:51:15 STEP 1734 H=1734:00 +CPU= 41.832
  15:51:43 STEP 1735 H=1735:00 +CPU= 26.037
  15:52:12 STEP 1736 H=1736:00 +CPU= 26.922
  15:52:56 STEP 1737 H=1737:00 +CPU= 41.683
  15:53:24 STEP 1738 H=1738:00 +CPU= 26.495
  15:53:52 STEP 1739 H=1739:00 +CPU= 26.198
  15:54:44 STEP 1740 H=1740:00 +CPU= 48.248
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001740
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001740
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001740
  15:55:12 STEP 1741 H=1741:00 +CPU= 26.514
  15:55:41 STEP 1742 H=1742:00 +CPU= 27.034
  15:56:26 STEP 1743 H=1743:00 +CPU= 42.134
  15:56:55 STEP 1744 H=1744:00 +CPU= 27.396
  15:57:24 STEP 1745 H=1745:00 +CPU= 26.804
  15:58:11 STEP 1746 H=1746:00 +CPU= 43.971
  15:58:39 STEP 1747 H=1747:00 +CPU= 26.427
  15:59:08 STEP 1748 H=1748:00 +CPU= 27.633
  15:59:53 STEP 1749 H=1749:00 +CPU= 41.828
  16:00:21 STEP 1750 H=1750:00 +CPU= 26.725
  16:00:51 STEP 1751 H=1751:00 +CPU= 28.269
  16:01:44 STEP 1752 H=1752:00 +CPU= 48.756
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001752
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001752
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170168/ICMGGhq0f+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170168/ICMSHhq0f+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170168/ICMUAhq0f+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170168/ICMGGhq0f+001740
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170168/ICMSHhq0f+001740
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170168/ICMUAhq0f+001740
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0524_1984050100_123_953_12170168_0_r1550041224_72.zip
Uploading the intermediate file: upload_file_72.zip
  16:02:11 STEP 1753 H=1753:00 +CPU= 25.819
  16:02:40 STEP 1754 H=1754:00 +CPU= 26.750
  16:03:25 STEP 1755 H=1755:00 +CPU= 42.256
  16:03:53 STEP 1756 H=1756:00 +CPU= 26.615
  16:04:21 STEP 1757 H=1757:00 +CPU= 26.146
  16:05:06 STEP 1758 H=1758:00 +CPU= 42.566
  16:05:34 STEP 1759 H=1759:00 +CPU= 25.911
  16:06:03 STEP 1760 H=1760:00 +CPU= 27.101
  16:06:49 STEP 1761 H=1761:00 +CPU= 42.765
  16:07:17 STEP 1762 H=1762:00 +CPU= 26.553
  16:07:45 STEP 1763 H=1763:00 +CPU= 26.074
  16:08:34 STEP 1764 H=1764:00 +CPU= 45.851
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001764
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001764
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001764
  16:09:01 STEP 1765 H=1765:00 +CPU= 26.055
  16:09:30 STEP 1766 H=1766:00 +CPU= 26.687
  16:10:14 STEP 1767 H=1767:00 +CPU= 41.821
  16:10:42 STEP 1768 H=1768:00 +CPU= 26.528
  16:11:12 STEP 1769 H=1769:00 +CPU= 27.187
  16:11:58 STEP 1770 H=1770:00 +CPU= 43.933
  16:12:27 STEP 1771 H=1771:00 +CPU= 26.986
  16:12:56 STEP 1772 H=1772:00 +CPU= 26.816
  16:13:40 STEP 1773 H=1773:00 +CPU= 41.648
  16:14:09 STEP 1774 H=1774:00 +CPU= 26.822
  16:14:37 STEP 1775 H=1775:00 +CPU= 26.983
  16:15:30 STEP 1776 H=1776:00 +CPU= 49.073
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001776
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001776
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170168/ICMGGhq0f+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170168/ICMSHhq0f+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170168/ICMUAhq0f+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170168/ICMGGhq0f+001764
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170168/ICMSHhq0f+001764
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170168/ICMUAhq0f+001764
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0524_1984050100_123_953_12170168_0_r1550041224_73.zip
Uploading the intermediate file: upload_file_73.zip
  16:15:59 STEP 1777 H=1777:00 +CPU= 26.639
  16:16:28 STEP 1778 H=1778:00 +CPU= 26.865
  16:17:14 STEP 1779 H=1779:00 +CPU= 43.311
  16:17:44 STEP 1780 H=1780:00 +CPU= 28.170
  16:18:13 STEP 1781 H=1781:00 +CPU= 27.164
  16:19:00 STEP 1782 H=1782:00 +CPU= 44.190
  16:19:28 STEP 1783 H=1783:00 +CPU= 25.922
  16:19:57 STEP 1784 H=1784:00 +CPU= 26.703
  16:20:41 STEP 1785 H=1785:00 +CPU= 41.840
  16:21:10 STEP 1786 H=1786:00 +CPU= 26.771
  16:21:37 STEP 1787 H=1787:00 +CPU= 26.098
  16:22:26 STEP 1788 H=1788:00 +CPU= 46.205
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001788
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001788
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001788
  16:22:54 STEP 1789 H=1789:00 +CPU= 26.161
  16:23:23 STEP 1790 H=1790:00 +CPU= 26.975
  16:24:08 STEP 1791 H=1791:00 +CPU= 42.155
  16:24:37 STEP 1792 H=1792:00 +CPU= 26.688
  16:25:06 STEP 1793 H=1793:00 +CPU= 27.260
  16:25:52 STEP 1794 H=1794:00 +CPU= 43.129
  16:26:20 STEP 1795 H=1795:00 +CPU= 26.155
  16:26:48 STEP 1796 H=1796:00 +CPU= 26.674
  16:27:32 STEP 1797 H=1797:00 +CPU= 41.814
  16:28:01 STEP 1798 H=1798:00 +CPU= 26.576
  16:28:29 STEP 1799 H=1799:00 +CPU= 26.821
  16:29:21 STEP 1800 H=1800:00 +CPU= 48.728
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001800
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001800
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170168/ICMGGhq0f+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170168/ICMSHhq0f+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170168/ICMUAhq0f+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170168/ICMGGhq0f+001788
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170168/ICMSHhq0f+001788
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170168/ICMUAhq0f+001788
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0524_1984050100_123_953_12170168_0_r1550041224_74.zip
Uploading the intermediate file: upload_file_74.zip
  16:29:49 STEP 1801 H=1801:00 +CPU= 25.932
  16:30:18 STEP 1802 H=1802:00 +CPU= 26.775
  16:31:03 STEP 1803 H=1803:00 +CPU= 41.789
  16:31:32 STEP 1804 H=1804:00 +CPU= 27.497
  16:32:00 STEP 1805 H=1805:00 +CPU= 26.094
  16:32:46 STEP 1806 H=1806:00 +CPU= 43.061
  16:33:16 STEP 1807 H=1807:00 +CPU= 27.382
  16:33:50 STEP 1808 H=1808:00 +CPU= 30.969
  16:34:39 STEP 1809 H=1809:00 +CPU= 44.330
  16:35:15 STEP 1810 H=1810:00 +CPU= 31.473
  16:35:44 STEP 1811 H=1811:00 +CPU= 27.001
  16:36:34 STEP 1812 H=1812:00 +CPU= 47.840
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001812
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001812
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001812
  16:37:06 STEP 1813 H=1813:00 +CPU= 28.611
  16:37:47 STEP 1814 H=1814:00 +CPU= 35.449
  16:38:36 STEP 1815 H=1815:00 +CPU= 45.002
  16:39:08 STEP 1816 H=1816:00 +CPU= 30.484
  16:39:50 STEP 1817 H=1817:00 +CPU= 35.048
  16:40:36 STEP 1818 H=1818:00 +CPU= 43.199
  16:41:04 STEP 1819 H=1819:00 +CPU= 26.073
  16:41:33 STEP 1820 H=1820:00 +CPU= 26.787
  16:42:26 STEP 1821 H=1821:00 +CPU= 48.975
  16:43:01 STEP 1822 H=1822:00 +CPU= 32.687
  16:43:34 STEP 1823 H=1823:00 +CPU= 30.240
  16:44:29 STEP 1824 H=1824:00 +CPU= 51.108
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001824
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001824
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170168/ICMGGhq0f+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170168/ICMSHhq0f+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170168/ICMUAhq0f+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170168/ICMGGhq0f+001812
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170168/ICMSHhq0f+001812
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170168/ICMUAhq0f+001812
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0524_1984050100_123_953_12170168_0_r1550041224_75.zip
Uploading the intermediate file: upload_file_75.zip
  16:45:00 STEP 1825 H=1825:00 +CPU= 28.119
  16:45:31 STEP 1826 H=1826:00 +CPU= 28.879
  16:46:19 STEP 1827 H=1827:00 +CPU= 45.249
  16:46:53 STEP 1828 H=1828:00 +CPU= 31.136
  16:47:27 STEP 1829 H=1829:00 +CPU= 31.912
  16:48:20 STEP 1830 H=1830:00 +CPU= 49.637
  16:48:53 STEP 1831 H=1831:00 +CPU= 30.408
  16:49:24 STEP 1832 H=1832:00 +CPU= 29.356
  16:50:14 STEP 1833 H=1833:00 +CPU= 45.909
  16:50:45 STEP 1834 H=1834:00 +CPU= 29.455
  16:51:19 STEP 1835 H=1835:00 +CPU= 30.750
  16:52:16 STEP 1836 H=1836:00 +CPU= 52.673
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001836
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001836
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001836
  16:52:46 STEP 1837 H=1837:00 +CPU= 27.637
  16:53:17 STEP 1838 H=1838:00 +CPU= 29.550
  16:54:07 STEP 1839 H=1839:00 +CPU= 46.126
  16:54:48 STEP 1840 H=1840:00 +CPU= 30.328
  16:55:18 STEP 1841 H=1841:00 +CPU= 26.804
  16:56:05 STEP 1842 H=1842:00 +CPU= 43.239
  16:56:33 STEP 1843 H=1843:00 +CPU= 26.851
  16:57:01 STEP 1844 H=1844:00 +CPU= 26.907
  16:57:46 STEP 1845 H=1845:00 +CPU= 42.112
  16:58:18 STEP 1846 H=1846:00 +CPU= 29.763
  16:58:49 STEP 1847 H=1847:00 +CPU= 28.393
  16:59:44 STEP 1848 H=1848:00 +CPU= 50.831
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001848
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001848
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170168/ICMGGhq0f+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170168/ICMSHhq0f+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170168/ICMUAhq0f+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170168/ICMGGhq0f+001836
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170168/ICMSHhq0f+001836
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170168/ICMUAhq0f+001836
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0524_1984050100_123_953_12170168_0_r1550041224_76.zip
Uploading the intermediate file: upload_file_76.zip
  17:00:13 STEP 1849 H=1849:00 +CPU= 26.720
  17:00:42 STEP 1850 H=1850:00 +CPU= 27.029
  17:01:27 STEP 1851 H=1851:00 +CPU= 42.709
  17:01:56 STEP 1852 H=1852:00 +CPU= 26.754
  17:02:24 STEP 1853 H=1853:00 +CPU= 26.457
  17:03:11 STEP 1854 H=1854:00 +CPU= 43.735
  17:03:39 STEP 1855 H=1855:00 +CPU= 26.247
  17:04:09 STEP 1856 H=1856:00 +CPU= 27.438
  17:04:56 STEP 1857 H=1857:00 +CPU= 43.646
  17:05:26 STEP 1858 H=1858:00 +CPU= 27.328
  17:05:54 STEP 1859 H=1859:00 +CPU= 26.288
  17:06:43 STEP 1860 H=1860:00 +CPU= 46.000
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001860
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001860
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001860
  17:07:12 STEP 1861 H=1861:00 +CPU= 27.286
  17:07:44 STEP 1862 H=1862:00 +CPU= 28.750
  17:08:32 STEP 1863 H=1863:00 +CPU= 44.978
  17:09:07 STEP 1864 H=1864:00 +CPU= 31.614
  17:09:39 STEP 1865 H=1865:00 +CPU= 29.918
  17:10:30 STEP 1866 H=1866:00 +CPU= 47.202
  17:11:00 STEP 1867 H=1867:00 +CPU= 28.580
  17:11:30 STEP 1868 H=1868:00 +CPU= 27.559
  17:12:15 STEP 1869 H=1869:00 +CPU= 42.271
  17:12:44 STEP 1870 H=1870:00 +CPU= 26.975
  17:13:14 STEP 1871 H=1871:00 +CPU= 28.620
  17:14:07 STEP 1872 H=1872:00 +CPU= 49.086
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001872
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001872
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170168/ICMGGhq0f+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170168/ICMSHhq0f+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170168/ICMUAhq0f+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170168/ICMGGhq0f+001860
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170168/ICMSHhq0f+001860
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170168/ICMUAhq0f+001860
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0524_1984050100_123_953_12170168_0_r1550041224_77.zip
Uploading the intermediate file: upload_file_77.zip
Uploading trickle at timestep: 6739200
  17:14:35 STEP 1873 H=1873:00 +CPU= 25.945
  17:15:07 STEP 1874 H=1874:00 +CPU= 28.280
  17:15:54 STEP 1875 H=1875:00 +CPU= 43.886
  17:16:28 STEP 1876 H=1876:00 +CPU= 30.439
  17:16:56 STEP 1877 H=1877:00 +CPU= 26.420
  17:17:52 STEP 1878 H=1878:00 +CPU= 50.532
  17:18:21 STEP 1879 H=1879:00 +CPU= 26.969
  17:18:52 STEP 1880 H=1880:00 +CPU= 27.886
  17:19:38 STEP 1881 H=1881:00 +CPU= 43.177
  17:20:08 STEP 1882 H=1882:00 +CPU= 27.430
  17:20:36 STEP 1883 H=1883:00 +CPU= 26.625
  17:21:26 STEP 1884 H=1884:00 +CPU= 46.485
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001884
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001884
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001884
  17:21:53 STEP 1885 H=1885:00 +CPU= 26.101
  17:22:23 STEP 1886 H=1886:00 +CPU= 27.796
  17:23:09 STEP 1887 H=1887:00 +CPU= 42.948
  17:23:38 STEP 1888 H=1888:00 +CPU= 26.913
  17:24:06 STEP 1889 H=1889:00 +CPU= 26.518
  17:24:54 STEP 1890 H=1890:00 +CPU= 44.420
  17:25:25 STEP 1891 H=1891:00 +CPU= 28.422
  17:25:54 STEP 1892 H=1892:00 +CPU= 27.734
  17:26:42 STEP 1893 H=1893:00 +CPU= 45.105
  17:27:17 STEP 1894 H=1894:00 +CPU= 32.088
  17:27:50 STEP 1895 H=1895:00 +CPU= 30.421
  17:28:45 STEP 1896 H=1896:00 +CPU= 51.534
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001896
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001896
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170168/ICMGGhq0f+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170168/ICMSHhq0f+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170168/ICMUAhq0f+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170168/ICMGGhq0f+001884
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170168/ICMSHhq0f+001884
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170168/ICMUAhq0f+001884
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0524_1984050100_123_953_12170168_0_r1550041224_78.zip
Uploading the intermediate file: upload_file_78.zip
  17:29:16 STEP 1897 H=1897:00 +CPU= 28.964
  17:29:50 STEP 1898 H=1898:00 +CPU= 30.947
  17:30:34 STEP 1899 H=1899:00 +CPU= 42.160
  17:31:03 STEP 1900 H=1900:00 +CPU= 26.608
  17:31:31 STEP 1901 H=1901:00 +CPU= 26.043
  17:32:17 STEP 1902 H=1902:00 +CPU= 43.169
  17:32:46 STEP 1903 H=1903:00 +CPU= 26.804
  17:33:17 STEP 1904 H=1904:00 +CPU= 28.490
Abort request received from BOINC client, ending the child process

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:14:49 1537334 22254484 oifs_43r3_ps_0524_1984050100_123_953_12170168_0 6,739,200 69,185 0.0103
09 May 2023 16:13:49 1537334 22254484 oifs_43r3_ps_0524_1984050100_123_953_12170168_0 4,838,400 50,264 0.0104
09 May 2023 16:13:05 1537334 22254484 oifs_43r3_ps_0524_1984050100_123_953_12170168_0 3,628,800 38,395 0.0106
09 May 2023 16:12:53 1537334 22254484 oifs_43r3_ps_0524_1984050100_123_953_12170168_0 2,419,200 25,731 0.0106
09 May 2023 16:12:49 1537334 22254484 oifs_43r3_ps_0524_1984050100_123_953_12170168_0 1,555,200 16,711 0.0107


©2024 climateprediction.net