climateprediction.net home page
Task 22267594

Task 22267594

Name oifs_43r3_ps_0847_1996050100_123_965_12182491_0
Workunit 12182491
Created 22 Dec 2022, 12:02:35 UTC
Sent 19 Jan 2023, 5:29:44 UTC
Report deadline 20 Mar 2023, 5:29:44 UTC
Received 20 Jan 2023, 3:44:00 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 148 (0x00000094) Unknown error code
Computer ID 1538241
Run time 16 hours 30 min 41 sec
CPU time 16 hours 18 min 35 sec
Validate state Invalid
Credit 7,257.60
Device peak FLOPS 6.24 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,418.43 MB
Peak swap size 4,978.08 MB
Peak disk usage 4,447.65 MB
Stderr
<core_client_version>7.20.2</core_client_version>
<![CDATA[
<message>
process exited with code 148 (0x94, -108)</message>
<stderr_txt>
' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:e3:1:1:1444:1444] [20230119:225716:1674169036:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:e3:1:1:1444:1444] [20230119:225716:1674169036:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e3:1:1:1444:1444] [20230119:225716:1674169036:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e3:1:1:1444:1444] [20230119:225716:1674169036:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:e3:1:1:1444:1444] [20230119:225716:1674169036:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e3:1:1:1444:1444] [20230119:225716:1674169036:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e3:1:1:1444:1444] [20230119:225716:1674169036:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e3:1:1:1444:1444] [20230119:225716:1674169036:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:e3:1:1:1444:1444] [20230119:225716:1674169036:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:e3:1:1:1444:1444] [20230119:225716:1674169036:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:e3:1:1:1444:1444] [20230119:225716:1674169036:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  22:59:47 STEP 1608 H=1608:00 +CPU= 42.069
  23:00:12 STEP 1609 H=1609:00 +CPU= 25.122
  23:00:36 STEP 1610 H=1610:00 +CPU= 23.263
  23:01:08 STEP 1611 H=1611:00 +CPU= 32.463
  23:01:31 STEP 1612 H=1612:00 +CPU= 23.133
  23:01:54 STEP 1613 H=1613:00 +CPU= 22.320
  23:02:31 STEP 1614 H=1614:00 +CPU= 37.440
  23:02:55 STEP 1615 H=1615:00 +CPU= 24.385
  23:03:20 STEP 1616 H=1616:00 +CPU= 24.859
  23:03:57 STEP 1617 H=1617:00 +CPU= 37.108
  23:04:23 STEP 1618 H=1618:00 +CPU= 25.534
  23:04:49 STEP 1619 H=1619:00 +CPU= 25.549
  23:05:30 STEP 1620 H=1620:00 +CPU= 41.872
Moving to projects directory: /var/lib/boinc-client_40001/slots/2/ICMGGhq0f+001620
Moving to projects directory: /var/lib/boinc-client_40001/slots/2/ICMSHhq0f+001620
Moving to projects directory: /var/lib/boinc-client_40001/slots/2/ICMUAhq0f+001620
  23:05:50 STEP 1621 H=1621:00 +CPU= 19.784
  23:06:15 STEP 1622 H=1622:00 +CPU= 24.928
  23:06:54 STEP 1623 H=1623:00 +CPU= 38.794
  23:07:19 STEP 1624 H=1624:00 +CPU= 24.828
  23:07:44 STEP 1625 H=1625:00 +CPU= 24.719
  23:08:21 STEP 1626 H=1626:00 +CPU= 37.016
  23:08:45 STEP 1627 H=1627:00 +CPU= 24.424
  23:09:08 STEP 1628 H=1628:00 +CPU= 22.804
  23:09:43 STEP 1629 H=1629:00 +CPU= 35.422
  23:10:08 STEP 1630 H=1630:00 +CPU= 24.590
  23:10:33 STEP 1631 H=1631:00 +CPU= 25.281
  23:11:15 STEP 1632 H=1632:00 +CPU= 41.659
Moving to projects directory: /var/lib/boinc-client_40001/slots/2/ICMGGhq0f+001632
Moving to projects directory: /var/lib/boinc-client_40001/slots/2/ICMSHhq0f+001632
Moving to projects directory: /var/lib/boinc-client_40001/slots/2/ICMUAhq0f+001632
Adding to the zip: /var/lib/boinc-client_40001/projects/climateprediction.net/oifs_43r3_ps_12182491/ICMGGhq0f+001608
Adding to the zip: /var/lib/boinc-client_40001/projects/climateprediction.net/oifs_43r3_ps_12182491/ICMSHhq0f+001608
Adding to the zip: /var/lib/boinc-client_40001/projects/climateprediction.net/oifs_43r3_ps_12182491/ICMUAhq0f+001608
Adding to the zip: /var/lib/boinc-client_40001/projects/climateprediction.net/oifs_43r3_ps_12182491/ICMGGhq0f+001620
Adding to the zip: /var/lib/boinc-client_40001/projects/climateprediction.net/oifs_43r3_ps_12182491/ICMSHhq0f+001620
Adding to the zip: /var/lib/boinc-client_40001/projects/climateprediction.net/oifs_43r3_ps_12182491/ICMUAhq0f+001620
Zipping up the intermediate file: /var/lib/boinc-client_40001/projects/climateprediction.net/oifs_43r3_ps_0847_1996050100_123_965_12182491_0_r200198354_67.zip
Uploading the intermediate file: upload_file_67.zip
  23:11:38 STEP 1633 H=1633:00 +CPU= 23.135
  23:12:02 STEP 1634 H=1634:00 +CPU= 23.988
  23:12:38 STEP 1635 H=1635:00 +CPU= 35.683
  23:13:03 STEP 1636 H=1636:00 +CPU= 24.695
  23:13:27 STEP 1637 H=1637:00 +CPU= 23.784
  23:14:03 STEP 1638 H=1638:00 +CPU= 35.887
  23:14:27 STEP 1639 H=1639:00 +CPU= 24.315
  23:14:52 STEP 1640 H=1640:00 +CPU= 25.415
  23:15:30 STEP 1641 H=1641:00 +CPU= 37.405
  23:15:54 STEP 1642 H=1642:00 +CPU= 24.708
  23:16:20 STEP 1643 H=1643:00 +CPU= 25.354
  23:17:01 STEP 1644 H=1644:00 +CPU= 41.580
Moving to projects directory: /var/lib/boinc-client_40001/slots/2/ICMGGhq0f+001644
Moving to projects directory: /var/lib/boinc-client_40001/slots/2/ICMSHhq0f+001644
Moving to projects directory: /var/lib/boinc-client_40001/slots/2/ICMUAhq0f+001644
  23:17:26 STEP 1645 H=1645:00 +CPU= 24.606
  23:17:51 STEP 1646 H=1646:00 +CPU= 24.889
  23:18:26 STEP 1647 H=1647:00 +CPU= 35.045
  23:18:49 STEP 1648 H=1648:00 +CPU= 23.054
  23:19:12 STEP 1649 H=1649:00 +CPU= 23.183
  23:19:49 STEP 1650 H=1650:00 +CPU= 36.623
  23:20:11 STEP 1651 H=1651:00 +CPU= 21.929
  23:20:33 STEP 1652 H=1652:00 +CPU= 21.957
  23:21:11 STEP 1653 H=1653:00 +CPU= 37.850
  23:21:35 STEP 1654 H=1654:00 +CPU= 24.719
  23:22:00 STEP 1655 H=1655:00 +CPU= 24.914
  23:22:44 STEP 1656 H=1656:00 +CPU= 43.529
Moving to projects directory: /var/lib/boinc-client_40001/slots/2/ICMGGhq0f+001656
Moving to projects directory: /var/lib/boinc-client_40001/slots/2/ICMSHhq0f+001656
Moving to projects directory: /var/lib/boinc-client_40001/slots/2/ICMUAhq0f+001656
Adding to the zip: /var/lib/boinc-client_40001/projects/climateprediction.net/oifs_43r3_ps_12182491/ICMGGhq0f+001632
Adding to the zip: /var/lib/boinc-client_40001/projects/climateprediction.net/oifs_43r3_ps_12182491/ICMSHhq0f+001632
Adding to the zip: /var/lib/boinc-client_40001/projects/climateprediction.net/oifs_43r3_ps_12182491/ICMUAhq0f+001632
Adding to the zip: /var/lib/boinc-client_40001/projects/climateprediction.net/oifs_43r3_ps_12182491/ICMGGhq0f+001644
Adding to the zip: /var/lib/boinc-client_40001/projects/climateprediction.net/oifs_43r3_ps_12182491/ICMSHhq0f+001644
Adding to the zip: /var/lib/boinc-client_40001/projects/climateprediction.net/oifs_43r3_ps_12182491/ICMUAhq0f+001644
Zipping up the intermediate file: /var/lib/boinc-client_40001/projects/climateprediction.net/oifs_43r3_ps_0847_1996050100_123_965_12182491_0_r200198354_68.zip
Uploading the intermediate file: upload_file_68.zip
  23:23:07 STEP 1657 H=1657:00 +CPU= 23.289
  23:23:31 STEP 1658 H=1658:00 +CPU= 23.260
  23:24:05 STEP 1659 H=1659:00 +CPU= 34.396
  23:24:30 STEP 1660 H=1660:00 +CPU= 24.339
  23:24:51 STEP 1661 H=1661:00 +CPU= 21.828
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1996050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0847
(argv4) batchid: 965
(argv5) wuid: 12182491
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client_40001/slots/2
Project directory is: /var/lib/boinc-client_40001/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client_40001/projects/climateprediction.net/oifs_43r3_ps_12182491
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client_40001/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client_40001/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client_40001/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_5dc4945452d8677271b2860d36d3cf37 to: /var/lib/boinc-client_40001/slots/2/oifs_43r3_ps_0847_1996050100_123_965_12182491.zip
Unzipping the namelist zip file: /var/lib/boinc-client_40001/slots/2/oifs_43r3_ps_0847_1996050100_123_965_12182491.zip
ic_ancil_file: ic_ancil_12182491
ifsdata_file: ifsdata_12182491
climate_data_file: clim_data_12182491
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_77ac4c35e84328580502d03cdc572a19 to: /var/lib/boinc-client_40001/slots/2/ic_ancil_12182491.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client_40001/slots/2/ic_ancil_12182491.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_900b549f2d50153362aa127af335777b to: /var/lib/boinc-client_40001/slots/2/ifsdata/ifsdata_12182491.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client_40001/slots/2/ifsdata/ifsdata_12182491.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client_40001/slots/2/159l_2/clim_data_12182491.zip
Unzipping the climate data zip file: /var/lib/boinc-client_40001/slots/2/159l_2/clim_data_12182491.zip
Checking for progress XML file: /var/lib/boinc-client_40001/slots/2/progress_file_12182491.xml
Opened progress file ok : /var/lib/boinc-client_40001/slots/2/progress_file_12182491.xml
-- Model is restarting --
Adjusting last_iter, 1662, to previous model restart step.
Creating progress file: /var/lib/boinc-client_40001/slots/2/progress_file_12182491.xml
last_cpu_time: 50189
upload_file_number: 69
last_iter: 1655
last_upload: 5961600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0847_1996050100_123_965_12182491_0_r200198354
The child process has been launched with process id: 1451
Executing the command: /var/lib/boinc-client_40001/slots/2/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:e3:1:1:1451:1451] [20230119:234434:1674171874:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:e3:1:1:1451:1451] [20230119:234434:1674171874:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:e3:1:1:1451:1451] [20230119:234434:1674171874:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:e3:1:1:1451:1451] [20230119:234434:1674171874:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:e3:1:1:1451:1451] [20230119:234434:1674171874:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:e3:1:1:1451:1451] [20230119:234434:1674171874:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:e3:1:1:1451:1451] [20230119:234434:1674171874:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:e3:1:1:1451:1451] [20230119:234434:1674171874:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:e3:1:1:1451:1451] [20230119:234434:1674171874:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:e3:1:1:1451:1451] [20230119:234434:1674171874:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:e3:1:1:1451:1451] [20230119:234434:1674171874:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:e3:1:1:1451:1451] [20230119:234434:1674171874:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:e3:1:1:1451:1451] [20230119:234434:1674171874:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:e3:1:1:1451:1451] [20230119:234434:1674171874:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:e3:1:1:1451:1451] [20230119:234434:1674171874:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:e3:1:1:1451:1451] [20230119:234434:1674171874:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e3:1:1:1451:1451] [20230119:234434:1674171874:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:e3:1:1:1451:1451] [20230119:234434:1674171874:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e3:1:1:1451:1451] [20230119:234434:1674171874:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:e3:1:1:1451:1451] [20230119:234434:1674171874:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:e3:1:1:1451:1451] [20230119:234434:1674171874:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e3:1:1:1451:1451] [20230119:234434:1674171874:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e3:1:1:1451:1451] [20230119:234434:1674171874:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:e3:1:1:1451:1451] [20230119:234434:1674171874:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e3:1:1:1451:1451] [20230119:234434:1674171874:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e3:1:1:1451:1451] [20230119:234434:1674171874:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e3:1:1:1451:1451] [20230119:234434:1674171874:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:e3:1:1:1451:1451] [20230119:234434:1674171874:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:e3:1:1:1451:1451] [20230119:234434:1674171874:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:e3:1:1:1451:1451] [20230119:234434:1674171874:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  23:47:02 STEP 1656 H=1656:00 +CPU= 38.144
  23:47:24 STEP 1657 H=1657:00 +CPU= 22.091
  23:47:46 STEP 1658 H=1658:00 +CPU= 22.469
  23:48:19 STEP 1659 H=1659:00 +CPU= 32.936
  23:48:40 STEP 1660 H=1660:00 +CPU= 20.340
  23:49:01 STEP 1661 H=1661:00 +CPU= 21.025
  23:49:34 STEP 1662 H=1662:00 +CPU= 33.134
  23:49:55 STEP 1663 H=1663:00 +CPU= 21.582
  23:50:17 STEP 1664 H=1664:00 +CPU= 22.004
  23:50:51 STEP 1665 H=1665:00 +CPU= 33.760
  23:51:12 STEP 1666 H=1666:00 +CPU= 20.484
  23:51:34 STEP 1667 H=1667:00 +CPU= 22.122
  23:52:11 STEP 1668 H=1668:00 +CPU= 37.199
Moving to projects directory: /var/lib/boinc-client_40001/slots/2/ICMGGhq0f+001668
Moving to projects directory: /var/lib/boinc-client_40001/slots/2/ICMSHhq0f+001668
Moving to projects directory: /var/lib/boinc-client_40001/slots/2/ICMUAhq0f+001668
  23:52:33 STEP 1669 H=1669:00 +CPU= 22.223
  23:52:56 STEP 1670 H=1670:00 +CPU= 22.747
  23:53:31 STEP 1671 H=1671:00 +CPU= 34.690
  23:53:54 STEP 1672 H=1672:00 +CPU= 22.975
  23:54:15 STEP 1673 H=1673:00 +CPU= 21.094
  23:54:47 STEP 1674 H=1674:00 +CPU= 32.369
  23:55:08 STEP 1675 H=1675:00 +CPU= 20.726
  23:55:29 STEP 1676 H=1676:00 +CPU= 21.574
  23:56:01 STEP 1677 H=1677:00 +CPU= 31.408
  23:56:24 STEP 1678 H=1678:00 +CPU= 23.251
  23:56:46 STEP 1679 H=1679:00 +CPU= 22.017
  23:57:25 STEP 1680 H=1680:00 +CPU= 39.171
Moving to projects directory: /var/lib/boinc-client_40001/slots/2/ICMGGhq0f+001680
Moving to projects directory: /var/lib/boinc-client_40001/slots/2/ICMSHhq0f+001680
Moving to projects directory: /var/lib/boinc-client_40001/slots/2/ICMUAhq0f+001680
Adding to the zip: /var/lib/boinc-client_40001/projects/climateprediction.net/oifs_43r3_ps_12182491/ICMGGhq0f+001656
Adding to the zip: /var/lib/boinc-client_40001/projects/climateprediction.net/oifs_43r3_ps_12182491/ICMSHhq0f+001656
Adding to the zip: /var/lib/boinc-client_40001/projects/climateprediction.net/oifs_43r3_ps_12182491/ICMUAhq0f+001656
Adding to the zip: /var/lib/boinc-client_40001/projects/climateprediction.net/oifs_43r3_ps_12182491/ICMGGhq0f+001668
Adding to the zip: /var/lib/boinc-client_40001/projects/climateprediction.net/oifs_43r3_ps_12182491/ICMSHhq0f+001668
Adding to the zip: /var/lib/boinc-client_40001/projects/climateprediction.net/oifs_43r3_ps_12182491/ICMUAhq0f+001668
Zipping up the intermediate file: /var/lib/boinc-client_40001/projects/climateprediction.net/oifs_43r3_ps_0847_1996050100_123_965_12182491_0_r200198354_69.zip
Uploading the intermediate file: upload_file_69.zip
  23:57:46 STEP 1681 H=1681:00 +CPU= 20.741
  23:58:08 STEP 1682 H=1682:00 +CPU= 21.875
  23:58:42 STEP 1683 H=1683:00 +CPU= 34.056
  23:59:04 STEP 1684 H=1684:00 +CPU= 21.788
  23:59:25 STEP 1685 H=1685:00 +CPU= 21.286
  23:59:59 STEP 1686 H=1686:00 +CPU= 33.749
  00:00:21 STEP 1687 H=1687:00 +CPU= 21.663
  00:00:43 STEP 1688 H=1688:00 +CPU= 22.520
  00:01:16 STEP 1689 H=1689:00 +CPU= 32.671
  00:01:38 STEP 1690 H=1690:00 +CPU= 22.203
  00:02:00 STEP 1691 H=1691:00 +CPU= 21.132
  00:02:38 STEP 1692 H=1692:00 +CPU= 38.545
Moving to projects directory: /var/lib/boinc-client_40001/slots/2/ICMGGhq0f+001692
Moving to projects directory: /var/lib/boinc-client_40001/slots/2/ICMSHhq0f+001692
Moving to projects directory: /var/lib/boinc-client_40001/slots/2/ICMUAhq0f+001692
  00:03:00 STEP 1693 H=1693:00 +CPU= 21.353
  00:03:23 STEP 1694 H=1694:00 +CPU= 23.347
  00:03:57 STEP 1695 H=1695:00 +CPU= 33.748
  00:04:19 STEP 1696 H=1696:00 +CPU= 22.398
  00:04:42 STEP 1697 H=1697:00 +CPU= 22.315
  00:05:14 STEP 1698 H=1698:00 +CPU= 32.266
  00:05:36 STEP 1699 H=1699:00 +CPU= 21.527
  00:05:56 STEP 1700 H=1700:00 +CPU= 20.477
  00:06:30 STEP 1701 H=1701:00 +CPU= 33.617
  00:06:52 STEP 1702 H=1702:00 +CPU= 21.872
  00:07:14 STEP 1703 H=1703:00 +CPU= 22.350
  00:07:54 STEP 1704 H=1704:00 +CPU= 39.479
Moving to projects directory: /var/lib/boinc-client_40001/slots/2/ICMGGhq0f+001704
Moving to projects directory: /var/lib/boinc-client_40001/slots/2/ICMSHhq0f+001704
Moving to projects directory: /var/lib/boinc-client_40001/slots/2/ICMUAhq0f+001704
Adding to the zip: /var/lib/boinc-client_40001/projects/climateprediction.net/oifs_43r3_ps_12182491/ICMGGhq0f+001680
Adding to the zip: /var/lib/boinc-client_40001/projects/climateprediction.net/oifs_43r3_ps_12182491/ICMSHhq0f+001680
Adding to the zip: /var/lib/boinc-client_40001/projects/climateprediction.net/oifs_43r3_ps_12182491/ICMUAhq0f+001680
Adding to the zip: /var/lib/boinc-client_40001/projects/climateprediction.net/oifs_43r3_ps_12182491/ICMGGhq0f+001692
Adding to the zip: /var/lib/boinc-client_40001/projects/climateprediction.net/oifs_43r3_ps_12182491/ICMSHhq0f+001692
Adding to the zip: /var/lib/boinc-client_40001/projects/climateprediction.net/oifs_43r3_ps_12182491/ICMUAhq0f+001692
Zipping up the intermediate file: /var/lib/boinc-client_40001/projects/climateprediction.net/oifs_43r3_ps_0847_1996050100_123_965_12182491_0_r200198354_70.zip
Uploading the intermediate file: upload_file_70.zip
  00:08:15 STEP 1705 H=1705:00 +CPU= 21.223
  00:08:38 STEP 1706 H=1706:00 +CPU= 22.518
  00:09:12 STEP 1707 H=1707:00 +CPU= 33.946
  00:09:34 STEP 1708 H=1708:00 +CPU= 22.105
  00:09:56 STEP 1709 H=1709:00 +CPU= 21.859
  00:10:29 STEP 1710 H=1710:00 +CPU= 33.814
  00:10:50 STEP 1711 H=1711:00 +CPU= 20.215
  00:11:11 STEP 1712 H=1712:00 +CPU= 21.765
  00:11:45 STEP 1713 H=1713:00 +CPU= 33.208
  00:12:06 STEP 1714 H=1714:00 +CPU= 21.645
  00:12:28 STEP 1715 H=1715:00 +CPU= 21.952
  00:13:05 STEP 1716 H=1716:00 +CPU= 37.111
Moving to projects directory: /var/lib/boinc-client_40001/slots/2/ICMGGhq0f+001716
Moving to projects directory: /var/lib/boinc-client_40001/slots/2/ICMSHhq0f+001716
Moving to projects directory: /var/lib/boinc-client_40001/slots/2/ICMUAhq0f+001716
  00:13:26 STEP 1717 H=1717:00 +CPU= 21.133
  00:13:49 STEP 1718 H=1718:00 +CPU= 22.607
  00:14:24 STEP 1719 H=1719:00 +CPU= 34.458
  00:14:46 STEP 1720 H=1720:00 +CPU= 22.903
  00:15:09 STEP 1721 H=1721:00 +CPU= 22.021
  00:15:43 STEP 1722 H=1722:00 +CPU= 34.066
  00:16:04 STEP 1723 H=1723:00 +CPU= 21.162
  00:16:24 STEP 1724 H=1724:00 +CPU= 20.108
  00:16:56 STEP 1725 H=1725:00 +CPU= 32.385
  00:17:17 STEP 1726 H=1726:00 +CPU= 20.934
  00:17:40 STEP 1727 H=1727:00 +CPU= 22.340
  00:18:20 STEP 1728 H=1728:00 +CPU= 40.449
Moving to projects directory: /var/lib/boinc-client_40001/slots/2/ICMGGhq0f+001728
Moving to projects directory: /var/lib/boinc-client_40001/slots/2/ICMSHhq0f+001728
Moving to projects directory: /var/lib/boinc-client_40001/slots/2/ICMUAhq0f+001728
Adding to the zip: /var/lib/boinc-client_40001/projects/climateprediction.net/oifs_43r3_ps_12182491/ICMGGhq0f+001704
Adding to the zip: /var/lib/boinc-client_40001/projects/climateprediction.net/oifs_43r3_ps_12182491/ICMSHhq0f+001704
Adding to the zip: /var/lib/boinc-client_40001/projects/climateprediction.net/oifs_43r3_ps_12182491/ICMUAhq0f+001704
Adding to the zip: /var/lib/boinc-client_40001/projects/climateprediction.net/oifs_43r3_ps_12182491/ICMGGhq0f+001716
Adding to the zip: /var/lib/boinc-client_40001/projects/climateprediction.net/oifs_43r3_ps_12182491/ICMSHhq0f+001716
Adding to the zip: /var/lib/boinc-client_40001/projects/climateprediction.net/oifs_43r3_ps_12182491/ICMUAhq0f+001716
Zipping up the intermediate file: /var/lib/boinc-client_40001/projects/climateprediction.net/oifs_43r3_ps_0847_1996050100_123_965_12182491_0_r200198354_71.zip
Uploading the intermediate file: upload_file_71.zip
  00:18:41 STEP 1729 H=1729:00 +CPU= 21.008
  00:19:03 STEP 1730 H=1730:00 +CPU= 21.963
  00:19:34 STEP 1731 H=1731:00 +CPU= 30.347
  00:19:56 STEP 1732 H=1732:00 +CPU= 22.057
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1996050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0847
(argv4) batchid: 965
(argv5) wuid: 12182491
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client_40001/slots/2
Project directory is: /var/lib/boinc-client_40001/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client_40001/projects/climateprediction.net/oifs_43r3_ps_12182491
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client_40001/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client_40001/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client_40001/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_5dc4945452d8677271b2860d36d3cf37 to: /var/lib/boinc-client_40001/slots/2/oifs_43r3_ps_0847_1996050100_123_965_12182491.zip
Unzipping the namelist zip file: /var/lib/boinc-client_40001/slots/2/oifs_43r3_ps_0847_1996050100_123_965_12182491.zip
ic_ancil_file: ic_ancil_12182491
ifsdata_file: ifsdata_12182491
climate_data_file: clim_data_12182491
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_77ac4c35e84328580502d03cdc572a19 to: /var/lib/boinc-client_40001/slots/2/ic_ancil_12182491.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client_40001/slots/2/ic_ancil_12182491.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_900b549f2d50153362aa127af335777b to: /var/lib/boinc-client_40001/slots/2/ifsdata/ifsdata_12182491.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client_40001/slots/2/ifsdata/ifsdata_12182491.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client_40001/slots/2/159l_2/clim_data_12182491.zip
Unzipping the climate data zip file: /var/lib/boinc-client_40001/slots/2/159l_2/clim_data_12182491.zip
Checking for progress XML file: /var/lib/boinc-client_40001/slots/2/progress_file_12182491.xml
Opened progress file ok : /var/lib/boinc-client_40001/slots/2/progress_file_12182491.xml
-- Model is restarting --
Adjusting last_iter, 1732, to previous model restart step.
Creating progress file: /var/lib/boinc-client_40001/slots/2/progress_file_12182491.xml
last_cpu_time: 52197
upload_file_number: 72
last_iter: 1727
last_upload: 6220800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0847_1996050100_123_965_12182491_0_r200198354
The child process has been launched with process id: 1449
Executing the command: /var/lib/boinc-client_40001/slots/2/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:e3:1:1:1449:1449] [20230120:005330:1674176010:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:e3:1:1:1449:1449] [20230120:005330:1674176010:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:e3:1:1:1449:1449] [20230120:005330:1674176010:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:e3:1:1:1449:1449] [20230120:005330:1674176010:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:e3:1:1:1449:1449] [20230120:005330:1674176010:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:e3:1:1:1449:1449] [20230120:005330:1674176010:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:e3:1:1:1449:1449] [20230120:005330:1674176010:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:e3:1:1:1449:1449] [20230120:005330:1674176010:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:e3:1:1:1449:1449] [20230120:005330:1674176010:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:e3:1:1:1449:1449] [20230120:005330:1674176010:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:e3:1:1:1449:1449] [20230120:005330:1674176010:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:e3:1:1:1449:1449] [20230120:005330:1674176010:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:e3:1:1:1449:1449] [20230120:005330:1674176010:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:e3:1:1:1449:1449] [20230120:005330:1674176010:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:e3:1:1:1449:1449] [20230120:005330:1674176010:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:e3:1:1:1449:1449] [20230120:005330:1674176010:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e3:1:1:1449:1449] [20230120:005330:1674176010:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:e3:1:1:1449:1449] [20230120:005330:1674176010:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e3:1:1:1449:1449] [20230120:005330:1674176010:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:e3:1:1:1449:1449] [20230120:005330:1674176010:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:e3:1:1:1449:1449] [20230120:005330:1674176010:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e3:1:1:1449:1449] [20230120:005330:1674176010:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e3:1:1:1449:1449] [20230120:005330:1674176010:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:e3:1:1:1449:1449] [20230120:005330:1674176010:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e3:1:1:1449:1449] [20230120:005330:1674176010:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e3:1:1:1449:1449] [20230120:005330:1674176010:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e3:1:1:1449:1449] [20230120:005330:1674176010:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:e3:1:1:1449:1449] [20230120:005330:1674176010:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:e3:1:1:1449:1449] [20230120:005330:1674176010:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:e3:1:1:1449:1449] [20230120:005330:1674176010:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  00:55:59 STEP 1728 H=1728:00 +CPU= 40.046
  00:56:21 STEP 1729 H=1729:00 +CPU= 22.085
  00:56:45 STEP 1730 H=1730:00 +CPU= 23.210
  00:57:18 STEP 1731 H=1731:00 +CPU= 33.264
  00:57:41 STEP 1732 H=1732:00 +CPU= 23.035
  00:58:03 STEP 1733 H=1733:00 +CPU= 22.465
  00:58:39 STEP 1734 H=1734:00 +CPU= 35.396
  00:59:00 STEP 1735 H=1735:00 +CPU= 20.926
  00:59:22 STEP 1736 H=1736:00 +CPU= 22.669
  00:59:57 STEP 1737 H=1737:00 +CPU= 34.328
  01:00:19 STEP 1738 H=1738:00 +CPU= 22.411
  01:00:42 STEP 1739 H=1739:00 +CPU= 22.757
  01:01:20 STEP 1740 H=1740:00 +CPU= 38.313
Moving to projects directory: /var/lib/boinc-client_40001/slots/2/ICMGGhq0f+001740
Moving to projects directory: /var/lib/boinc-client_40001/slots/2/ICMSHhq0f+001740
Moving to projects directory: /var/lib/boinc-client_40001/slots/2/ICMUAhq0f+001740
  01:01:41 STEP 1741 H=1741:00 +CPU= 20.552
  01:02:01 STEP 1742 H=1742:00 +CPU= 20.242
  01:02:32 STEP 1743 H=1743:00 +CPU= 31.171
  01:02:56 STEP 1744 H=1744:00 +CPU= 23.290
  01:03:18 STEP 1745 H=1745:00 +CPU= 22.547
  01:03:52 STEP 1746 H=1746:00 +CPU= 33.931
  01:04:14 STEP 1747 H=1747:00 +CPU= 22.226
  01:04:36 STEP 1748 H=1748:00 +CPU= 21.561
  01:05:09 STEP 1749 H=1749:00 +CPU= 33.535
  01:05:32 STEP 1750 H=1750:00 +CPU= 23.017
  01:05:55 STEP 1751 H=1751:00 +CPU= 22.435
  01:06:37 STEP 1752 H=1752:00 +CPU= 40.809
Moving to projects directory: /var/lib/boinc-client_40001/slots/2/ICMGGhq0f+001752
Moving to projects directory: /var/lib/boinc-client_40001/slots/2/ICMSHhq0f+001752
Moving to projects directory: /var/lib/boinc-client_40001/slots/2/ICMUAhq0f+001752
Adding to the zip: /var/lib/boinc-client_40001/projects/climateprediction.net/oifs_43r3_ps_12182491/ICMGGhq0f+001728
Adding to the zip: /var/lib/boinc-client_40001/projects/climateprediction.net/oifs_43r3_ps_12182491/ICMSHhq0f+001728
Adding to the zip: /var/lib/boinc-client_40001/projects/climateprediction.net/oifs_43r3_ps_12182491/ICMUAhq0f+001728
Adding to the zip: /var/lib/boinc-client_40001/projects/climateprediction.net/oifs_43r3_ps_12182491/ICMGGhq0f+001740
Adding to the zip: /var/lib/boinc-client_40001/projects/climateprediction.net/oifs_43r3_ps_12182491/ICMSHhq0f+001740
Adding to the zip: /var/lib/boinc-client_40001/projects/climateprediction.net/oifs_43r3_ps_12182491/ICMUAhq0f+001740
Zipping up the intermediate file: /var/lib/boinc-client_40001/projects/climateprediction.net/oifs_43r3_ps_0847_1996050100_123_965_12182491_0_r200198354_72.zip
Uploading the intermediate file: upload_file_72.zip
  01:06:59 STEP 1753 H=1753:00 +CPU= 21.794
  01:07:20 STEP 1754 H=1754:00 +CPU= 21.056
  01:07:54 STEP 1755 H=1755:00 +CPU= 33.842
  01:08:17 STEP 1756 H=1756:00 +CPU= 22.941
  01:08:39 STEP 1757 H=1757:00 +CPU= 21.923
  01:09:13 STEP 1758 H=1758:00 +CPU= 33.670
  01:09:34 STEP 1759 H=1759:00 +CPU= 21.046
  01:09:55 STEP 1760 H=1760:00 +CPU= 21.413
  01:10:30 STEP 1761 H=1761:00 +CPU= 34.825
  01:10:53 STEP 1762 H=1762:00 +CPU= 23.089
  01:11:15 STEP 1763 H=1763:00 +CPU= 21.264
  01:11:54 STEP 1764 H=1764:00 +CPU= 38.837
Moving to projects directory: /var/lib/boinc-client_40001/slots/2/ICMGGhq0f+001764
Moving to projects directory: /var/lib/boinc-client_40001/slots/2/ICMSHhq0f+001764
Moving to projects directory: /var/lib/boinc-client_40001/slots/2/ICMUAhq0f+001764
  01:12:17 STEP 1765 H=1765:00 +CPU= 22.387
  01:12:39 STEP 1766 H=1766:00 +CPU= 22.876
  01:13:15 STEP 1767 H=1767:00 +CPU= 35.928
  01:13:38 STEP 1768 H=1768:00 +CPU= 22.061
  01:13:58 STEP 1769 H=1769:00 +CPU= 20.905

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:29:21 1538241 22267594 oifs_43r3_ps_0847_1996050100_123_965_12182491_0 5,184,000 42,146 0.0081
09 May 2023 16:29:09 1538241 22267594 oifs_43r3_ps_0847_1996050100_123_965_12182491_0 4,320,000 33,986 0.0079
09 May 2023 16:28:58 1538241 22267594 oifs_43r3_ps_0847_1996050100_123_965_12182491_0 3,456,000 26,007 0.0075
09 May 2023 16:28:41 1538241 22267594 oifs_43r3_ps_0847_1996050100_123_965_12182491_0 2,592,000 18,490 0.0071
09 May 2023 16:28:35 1538241 22267594 oifs_43r3_ps_0847_1996050100_123_965_12182491_0 1,728,000 11,994 0.0069
09 May 2023 16:28:24 1538241 22267594 oifs_43r3_ps_0847_1996050100_123_965_12182491_0 864,000 6,061 0.0070


©2024 climateprediction.net