climateprediction.net home page
Task 22448576

Task 22448576

Name oifs_43r3_bl_a3b4_2016092300_20_1018_12292615_1
Workunit 12292615
Created 13 Jun 2024, 18:09:19 UTC
Sent 13 Jun 2024, 18:12:09 UTC
Report deadline 12 Aug 2024, 18:12:09 UTC
Received 17 Jun 2024, 20:18:30 UTC
Server state Over
Outcome Success
Client state Done
Exit status 0 (0x00000000)
Computer ID 1469580
Run time 11 hours 16 min 52 sec
CPU time 11 hours 12 min 30 sec
Validate state Valid
Credit 2,635.55
Device peak FLOPS 5.26 GFLOPS
Application version OpenIFS 43r3 Baroclinic Lifecycle v1.13
x86_64-pc-linux-gnu
Peak working set size 5,024.41 MB
Peak swap size 5,981.13 MB
Peak disk usage 1,742.33 MB
Stderr
<core_client_version>7.18.1</core_client_version>
<![CDATA[
<stderr_txt>
boinc-client/slots/12/ICMSHh7zg+001620
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAh7zg+001620
  19:14:53 STEP 1621 H= 405:15 +CPU= 23.464
  19:15:16 STEP 1622 H= 405:30 +CPU= 23.004
  19:15:38 STEP 1623 H= 405:45 +CPU= 21.921
  19:16:01 STEP 1624 H= 406:00 +CPU= 23.139
  19:16:23 STEP 1625 H= 406:15 +CPU= 21.968
  19:16:47 STEP 1626 H= 406:30 +CPU= 23.438
  19:17:09 STEP 1627 H= 406:45 +CPU= 22.651
  19:17:33 STEP 1628 H= 407:00 +CPU= 23.263
  19:17:56 STEP 1629 H= 407:15 +CPU= 23.549
  19:18:20 STEP 1630 H= 407:30 +CPU= 23.107
  19:18:42 STEP 1631 H= 407:45 +CPU= 22.633
  19:19:15 STEP 1632 H= 408:00 +CPU= 32.117
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGh7zg+001632
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHh7zg+001632
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAh7zg+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615/ICMGGh7zg+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615/ICMSHh7zg+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615/ICMUAh7zg+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615/ICMGGh7zg+001548
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615/ICMSHh7zg+001548
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615/ICMUAh7zg+001548
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615/ICMGGh7zg+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615/ICMSHh7zg+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615/ICMUAh7zg+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615/ICMGGh7zg+001572
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615/ICMSHh7zg+001572
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615/ICMUAh7zg+001572
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615/ICMGGh7zg+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615/ICMSHh7zg+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615/ICMUAh7zg+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615/ICMGGh7zg+001596
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615/ICMSHh7zg+001596
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615/ICMUAh7zg+001596
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615/ICMGGh7zg+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615/ICMSHh7zg+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615/ICMUAh7zg+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615/ICMGGh7zg+001620
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615/ICMSHh7zg+001620
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615/ICMUAh7zg+001620
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a3b4_2016092300_20_1018_12292615_1_r536513267_16.zip
Uploading the intermediate file: upload_file_16.zip
  19:19:38 STEP 1633 H= 408:15 +CPU= 22.670
  19:20:03 STEP 1634 H= 408:30 +CPU= 24.352
  19:20:25 STEP 1635 H= 408:45 +CPU= 22.704
  19:20:48 STEP 1636 H= 409:00 +CPU= 22.357
  19:21:12 STEP 1637 H= 409:15 +CPU= 23.836
  19:21:37 STEP 1638 H= 409:30 +CPU= 25.416
  19:22:00 STEP 1639 H= 409:45 +CPU= 22.655
  19:22:23 STEP 1640 H= 410:00 +CPU= 22.821
  19:22:46 STEP 1641 H= 410:15 +CPU= 22.847
  19:23:11 STEP 1642 H= 410:30 +CPU= 24.974
  19:23:34 STEP 1643 H= 410:45 +CPU= 22.834
  19:24:02 STEP 1644 H= 411:00 +CPU= 28.229
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGh7zg+001644
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHh7zg+001644
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAh7zg+001644
  19:24:25 STEP 1645 H= 411:15 +CPU= 22.612
  19:24:49 STEP 1646 H= 411:30 +CPU= 23.749
  19:25:11 STEP 1647 H= 411:45 +CPU= 22.026
  19:25:33 STEP 1648 H= 412:00 +CPU= 22.057
  19:25:56 STEP 1649 H= 412:15 +CPU= 22.696
  19:26:20 STEP 1650 H= 412:30 +CPU= 23.803
  19:26:41 STEP 1651 H= 412:45 +CPU= 21.940
  19:27:04 STEP 1652 H= 413:00 +CPU= 22.052
  19:27:27 STEP 1653 H= 413:15 +CPU= 23.058
  19:27:50 STEP 1654 H= 413:30 +CPU= 23.360
  19:28:14 STEP 1655 H= 413:45 +CPU= 24.037
  19:28:44 STEP 1656 H= 414:00 +CPU= 29.637
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGh7zg+001656
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHh7zg+001656
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAh7zg+001656
  19:29:07 STEP 1657 H= 414:15 +CPU= 23.388
  19:29:30 STEP 1658 H= 414:30 +CPU= 22.241
  19:29:52 STEP 1659 H= 414:45 +CPU= 22.443
  19:30:17 STEP 1660 H= 415:00 +CPU= 24.679
  19:30:40 STEP 1661 H= 415:15 +CPU= 23.294
  19:31:02 STEP 1662 H= 415:30 +CPU= 21.887
  19:31:24 STEP 1663 H= 415:45 +CPU= 22.006
  19:31:48 STEP 1664 H= 416:00 +CPU= 23.462
  19:32:12 STEP 1665 H= 416:15 +CPU= 24.445
  19:32:36 STEP 1666 H= 416:30 +CPU= 23.459
  19:32:59 STEP 1667 H= 416:45 +CPU= 23.260
  19:33:31 STEP 1668 H= 417:00 +CPU= 31.687
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGh7zg+001668
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHh7zg+001668
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAh7zg+001668
  19:33:54 STEP 1669 H= 417:15 +CPU= 23.339
  19:34:18 STEP 1670 H= 417:30 +CPU= 23.518
  19:34:43 STEP 1671 H= 417:45 +CPU= 25.383
  19:35:06 STEP 1672 H= 418:00 +CPU= 22.879
  19:35:27 STEP 1673 H= 418:15 +CPU= 21.329
  19:35:50 STEP 1674 H= 418:30 +CPU= 22.025
  19:36:14 STEP 1675 H= 418:45 +CPU= 24.408
  19:36:36 STEP 1676 H= 419:00 +CPU= 22.306
  19:36:58 STEP 1677 H= 419:15 +CPU= 22.013
  19:37:21 STEP 1678 H= 419:30 +CPU= 22.754
  19:37:45 STEP 1679 H= 419:45 +CPU= 23.675
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.13_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a3b4
(argv4) batchid: 1018
(argv5) wuid: 12292615
(argv6) fclen: 20
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/12
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.13
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/12/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/12/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_c5d0a6cab4482250225d38f560f1298d to: /var/lib/boinc-client/slots/12/oifs_43r3_bl_a3b4_2016092300_20_1018_12292615.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/12/oifs_43r3_bl_a3b4_2016092300_20_1018_12292615.zip
ic_ancil_file: ic_ancil_12292615
ifsdata_file: ifsdata_12292615
climate_data_file: clim_data_12292615
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /var/lib/boinc-client/slots/12/ic_ancil_12292615.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/12/ic_ancil_12292615.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_a6d21b067b4de9a3c175f3a00dc5ff65 to: /var/lib/boinc-client/slots/12/ifsdata/ifsdata_12292615.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/12/ifsdata/ifsdata_12292615.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/12/159l_2/clim_data_12292615.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/12/159l_2/clim_data_12292615.zip
Checking for progress XML file: /var/lib/boinc-client/slots/12/progress_file_12292615.xml
Opened progress file ok : /var/lib/boinc-client/slots/12/progress_file_12292615.xml
-- Model is restarting --
Adjusting last_iter, 1680, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/12/progress_file_12292615.xml
last_cpu_time: 34688
upload_file_number: 17
last_iter: 1681
last_upload: 1468800
model_completed: 0
total_length_of_simulation: 1728000
result_base_name: oifs_43r3_bl_a3b4_2016092300_20_1018_12292615_1_r536513267
The child process has been launched with process id: 13964
Executing the command: /var/lib/boinc-client/slots/12/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:server242:1:1:13964:13964] [20240617:193914:1718645954:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2091] fp = 0x314f440
[EC_DRHOOK:server242:1:1:13964:13964] [20240617:193914:1718645954:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:server242:1:1:13964:13964] [20240617:193914:1718645954:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:server242:1:1:13964:13964] [20240617:193914:1718645954:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:server242:1:1:13964:13964] [20240617:193914:1718645954:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:server242:1:1:13964:13964] [20240617:193914:1718645954:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:server242:1:1:13964:13964] [20240617:193914:1718645954:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:server242:1:1:13964:13964] [20240617:193914:1718645954:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:server242:1:1:13964:13964] [20240617:193914:1718645954:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:server242:1:1:13964:13964] [20240617:193914:1718645954:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:server242:1:1:13964:13964] [20240617:193914:1718645954:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:server242:1:1:13964:13964] [20240617:193914:1718645954:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:server242:1:1:13964:13964] [20240617:193914:1718645954:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:server242:1:1:13964:13964] [20240617:193914:1718645954:0.000] [ignore_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:server242:1:1:13964:13964] [20240617:193914:1718645954:0.000] [restore_default_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:server242:1:1:13964:13964] [20240617:193914:1718645954:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:server242:1:1:13964:13964] [20240617:193914:1718645954:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:server242:1:1:13964:13964] [20240617:193914:1718645954:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:server242:1:1:13964:13964] [20240617:193914:1718645954:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x820cf0 (old at (nil))
[EC_DRHOOK:server242:1:1:13964:13964] [20240617:193914:1718645954:0.000] [trapfpe_treatment@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:server242:1:1:13964:13964] [20240617:193914:1718645954:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:server242:1:1:13964:13964] [20240617:193914:1718645954:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:server242:1:1:13964:13964] [20240617:193914:1718645954:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x820cf0 (old at (nil))
[EC_DRHOOK:server242:1:1:13964:13964] [20240617:193914:1718645954:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:server242:1:1:13964:13964] [20240617:193914:1718645954:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:server242:1:1:13964:13964] [20240617:193914:1718645954:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:server242:1:1:13964:13964] [20240617:193914:1718645954:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x820cf0 (old at (nil))
[EC_DRHOOK:server242:1:1:13964:13964] [20240617:193914:1718645954:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x820cf0 (old at (nil))
[EC_DRHOOK:server242:1:1:13964:13964] [20240617:193914:1718645954:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:server242:1:1:13964:13964] [20240617:193914:1718645954:0.000] [catch_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  19:39:41 STEP 1680 H= 420:00 +CPU= 23.275
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGh7zg+001680
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHh7zg+001680
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAh7zg+001680
  19:40:04 STEP 1681 H= 420:15 +CPU= 22.319
  19:40:28 STEP 1682 H= 420:30 +CPU= 24.659
  19:40:52 STEP 1683 H= 420:45 +CPU= 24.046
  19:41:15 STEP 1684 H= 421:00 +CPU= 22.811
  19:41:37 STEP 1685 H= 421:15 +CPU= 21.666
  19:42:00 STEP 1686 H= 421:30 +CPU= 22.692
  19:42:23 STEP 1687 H= 421:45 +CPU= 23.260
  19:42:45 STEP 1688 H= 422:00 +CPU= 22.269
  19:43:07 STEP 1689 H= 422:15 +CPU= 21.290
  19:43:30 STEP 1690 H= 422:30 +CPU= 23.707
  19:43:53 STEP 1691 H= 422:45 +CPU= 22.176
  19:44:20 STEP 1692 H= 423:00 +CPU= 27.807
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGh7zg+001692
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHh7zg+001692
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAh7zg+001692
  19:44:43 STEP 1693 H= 423:15 +CPU= 22.376
  19:45:07 STEP 1694 H= 423:30 +CPU= 23.830
  19:45:29 STEP 1695 H= 423:45 +CPU= 22.426
  19:45:51 STEP 1696 H= 424:00 +CPU= 22.107
  19:46:14 STEP 1697 H= 424:15 +CPU= 22.345
  19:46:38 STEP 1698 H= 424:30 +CPU= 24.281
  19:47:00 STEP 1699 H= 424:45 +CPU= 21.651
  19:47:22 STEP 1700 H= 425:00 +CPU= 22.276
  19:47:45 STEP 1701 H= 425:15 +CPU= 23.318
  19:48:09 STEP 1702 H= 425:30 +CPU= 23.554
  19:48:31 STEP 1703 H= 425:45 +CPU= 21.443
  19:48:59 STEP 1704 H= 426:00 +CPU= 27.346
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGh7zg+001704
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHh7zg+001704
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAh7zg+001704
  19:49:23 STEP 1705 H= 426:15 +CPU= 23.794
  19:49:45 STEP 1706 H= 426:30 +CPU= 21.965
  19:50:07 STEP 1707 H= 426:45 +CPU= 22.077
  19:50:31 STEP 1708 H= 427:00 +CPU= 24.015
  19:50:54 STEP 1709 H= 427:15 +CPU= 23.362
  19:51:16 STEP 1710 H= 427:30 +CPU= 21.905
  19:51:38 STEP 1711 H= 427:45 +CPU= 22.018
  19:52:03 STEP 1712 H= 428:00 +CPU= 24.441
  19:52:26 STEP 1713 H= 428:15 +CPU= 23.357
  19:52:48 STEP 1714 H= 428:30 +CPU= 21.682
  19:53:10 STEP 1715 H= 428:45 +CPU= 21.743
  19:53:39 STEP 1716 H= 429:00 +CPU= 29.442
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGh7zg+001716
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHh7zg+001716
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAh7zg+001716
  19:54:02 STEP 1717 H= 429:15 +CPU= 22.783
  19:54:23 STEP 1718 H= 429:30 +CPU= 21.545
  19:54:46 STEP 1719 H= 429:45 +CPU= 22.355
  19:55:09 STEP 1720 H= 430:00 +CPU= 23.472
  19:55:31 STEP 1721 H= 430:15 +CPU= 21.703
  19:55:53 STEP 1722 H= 430:30 +CPU= 21.589
  19:56:15 STEP 1723 H= 430:45 +CPU= 21.940
  19:56:40 STEP 1724 H= 431:00 +CPU= 24.567
  19:57:03 STEP 1725 H= 431:15 +CPU= 23.201
  19:57:24 STEP 1726 H= 431:30 +CPU= 21.613
  19:57:47 STEP 1727 H= 431:45 +CPU= 22.595
  19:58:21 STEP 1728 H= 432:00 +CPU= 31.854
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGh7zg+001728
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHh7zg+001728
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAh7zg+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615/ICMGGh7zg+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615/ICMSHh7zg+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615/ICMUAh7zg+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615/ICMGGh7zg+001644
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615/ICMSHh7zg+001644
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615/ICMUAh7zg+001644
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615/ICMGGh7zg+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615/ICMSHh7zg+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615/ICMUAh7zg+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615/ICMGGh7zg+001668
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615/ICMSHh7zg+001668
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615/ICMUAh7zg+001668
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615/ICMGGh7zg+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615/ICMSHh7zg+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615/ICMUAh7zg+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615/ICMGGh7zg+001692
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615/ICMSHh7zg+001692
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615/ICMUAh7zg+001692
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615/ICMGGh7zg+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615/ICMSHh7zg+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615/ICMUAh7zg+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615/ICMGGh7zg+001716
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615/ICMSHh7zg+001716
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615/ICMUAh7zg+001716
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a3b4_2016092300_20_1018_12292615_1_r536513267_17.zip
Uploading the intermediate file: upload_file_17.zip
  19:58:44 STEP 1729 H= 432:15 +CPU= 23.020
  19:59:07 STEP 1730 H= 432:30 +CPU= 22.547
  19:59:29 STEP 1731 H= 432:45 +CPU= 22.013
  19:59:52 STEP 1732 H= 433:00 +CPU= 22.868
  20:00:15 STEP 1733 H= 433:15 +CPU= 23.351
  20:00:39 STEP 1734 H= 433:30 +CPU= 24.139
  20:01:01 STEP 1735 H= 433:45 +CPU= 21.542
  20:01:25 STEP 1736 H= 434:00 +CPU= 24.477
  20:01:48 STEP 1737 H= 434:15 +CPU= 22.060
  20:02:11 STEP 1738 H= 434:30 +CPU= 23.673
  20:02:33 STEP 1739 H= 434:45 +CPU= 22.095
  20:03:02 STEP 1740 H= 435:00 +CPU= 28.653
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGh7zg+001740
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHh7zg+001740
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAh7zg+001740
  20:03:25 STEP 1741 H= 435:15 +CPU= 22.514
  20:03:47 STEP 1742 H= 435:30 +CPU= 22.475
  20:04:10 STEP 1743 H= 435:45 +CPU= 23.003
  20:04:35 STEP 1744 H= 436:00 +CPU= 24.151
  20:04:58 STEP 1745 H= 436:15 +CPU= 22.790
  20:05:21 STEP 1746 H= 436:30 +CPU= 22.962
  20:05:44 STEP 1747 H= 436:45 +CPU= 22.833
  20:06:07 STEP 1748 H= 437:00 +CPU= 23.064
  20:06:29 STEP 1749 H= 437:15 +CPU= 22.286
  20:06:52 STEP 1750 H= 437:30 +CPU= 22.825
  20:07:15 STEP 1751 H= 437:45 +CPU= 22.925
  20:07:43 STEP 1752 H= 438:00 +CPU= 28.227
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGh7zg+001752
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHh7zg+001752
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAh7zg+001752
  20:08:06 STEP 1753 H= 438:15 +CPU= 22.426
  20:08:28 STEP 1754 H= 438:30 +CPU= 22.370
  20:08:51 STEP 1755 H= 438:45 +CPU= 23.012
  20:09:14 STEP 1756 H= 439:00 +CPU= 22.724
  20:09:37 STEP 1757 H= 439:15 +CPU= 23.063
  20:10:01 STEP 1758 H= 439:30 +CPU= 23.589
  20:10:27 STEP 1759 H= 439:45 +CPU= 25.650
  20:10:50 STEP 1760 H= 440:00 +CPU= 22.817
  20:11:13 STEP 1761 H= 440:15 +CPU= 22.827
  20:11:36 STEP 1762 H= 440:30 +CPU= 23.419
  20:11:59 STEP 1763 H= 440:45 +CPU= 22.500
  20:12:26 STEP 1764 H= 441:00 +CPU= 27.288
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGh7zg+001764
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHh7zg+001764
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAh7zg+001764
  20:12:50 STEP 1765 H= 441:15 +CPU= 23.226
  20:13:13 STEP 1766 H= 441:30 +CPU= 23.158
  20:13:36 STEP 1767 H= 441:45 +CPU= 22.910
  20:13:58 STEP 1768 H= 442:00 +CPU= 22.040
  20:14:21 STEP 1769 H= 442:15 +CPU= 23.400
  20:14:44 STEP 1770 H= 442:30 +CPU= 22.984
  20:15:07 STEP 1771 H= 442:45 +CPU= 22.945
  20:15:30 STEP 1772 H= 443:00 +CPU= 22.783
  20:15:53 STEP 1773 H= 443:15 +CPU= 22.402
  20:16:18 STEP 1774 H= 443:30 +CPU= 25.051
  20:16:40 STEP 1775 H= 443:45 +CPU= 22.441
  20:17:12 STEP 1776 H= 444:00 +CPU= 31.589
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGh7zg+001776
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHh7zg+001776
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAh7zg+001776
  20:17:36 STEP 1777 H= 444:15 +CPU= 23.098
  20:17:59 STEP 1778 H= 444:30 +CPU= 23.031
  20:18:22 STEP 1779 H= 444:45 +CPU= 23.398
  20:18:46 STEP 1780 H= 445:00 +CPU= 24.273
  20:19:09 STEP 1781 H= 445:15 +CPU= 22.099
  20:19:30 STEP 1782 H= 445:30 +CPU= 21.535
  20:19:53 STEP 1783 H= 445:45 +CPU= 22.978
  20:20:20 STEP 1784 H= 446:00 +CPU= 26.325
  20:20:42 STEP 1785 H= 446:15 +CPU= 21.945
  20:21:03 STEP 1786 H= 446:30 +CPU= 21.482
  20:21:27 STEP 1787 H= 446:45 +CPU= 23.914
  20:21:56 STEP 1788 H= 447:00 +CPU= 28.848
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGh7zg+001788
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHh7zg+001788
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAh7zg+001788
  20:22:18 STEP 1789 H= 447:15 +CPU= 21.905
  20:22:40 STEP 1790 H= 447:30 +CPU= 21.772
  20:23:03 STEP 1791 H= 447:45 +CPU= 23.605
  20:23:28 STEP 1792 H= 448:00 +CPU= 24.120
  20:23:51 STEP 1793 H= 448:15 +CPU= 23.075
  20:24:13 STEP 1794 H= 448:30 +CPU= 22.740
  20:24:36 STEP 1795 H= 448:45 +CPU= 22.652
  20:24:59 STEP 1796 H= 449:00 +CPU= 22.543
  20:25:21 STEP 1797 H= 449:15 +CPU= 22.541
  20:25:44 STEP 1798 H= 449:30 +CPU= 22.327
  20:26:06 STEP 1799 H= 449:45 +CPU= 22.473
  20:26:35 STEP 1800 H= 450:00 +CPU= 28.523
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGh7zg+001800
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHh7zg+001800
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAh7zg+001800
  20:26:56 STEP 1801 H= 450:15 +CPU= 21.524
  20:27:20 STEP 1802 H= 450:30 +CPU= 23.201
  20:27:43 STEP 1803 H= 450:45 +CPU= 23.214
  20:28:05 STEP 1804 H= 451:00 +CPU= 22.664
  20:28:27 STEP 1805 H= 451:15 +CPU= 21.393
  20:28:49 STEP 1806 H= 451:30 +CPU= 22.529
  20:29:14 STEP 1807 H= 451:45 +CPU= 24.628
  20:29:37 STEP 1808 H= 452:00 +CPU= 23.024
  20:29:59 STEP 1809 H= 452:15 +CPU= 21.620
  20:30:23 STEP 1810 H= 452:30 +CPU= 23.730
  20:30:47 STEP 1811 H= 452:45 +CPU= 24.300
  20:31:16 STEP 1812 H= 453:00 +CPU= 28.770
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGh7zg+001812
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHh7zg+001812
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAh7zg+001812
  20:31:38 STEP 1813 H= 453:15 +CPU= 21.782
  20:32:00 STEP 1814 H= 453:30 +CPU= 22.547
  20:32:24 STEP 1815 H= 453:45 +CPU= 23.363
  20:32:47 STEP 1816 H= 454:00 +CPU= 23.721
  20:33:09 STEP 1817 H= 454:15 +CPU= 21.923
  20:33:33 STEP 1818 H= 454:30 +CPU= 23.078
  20:33:56 STEP 1819 H= 454:45 +CPU= 22.998
  20:34:24 STEP 1820 H= 455:00 +CPU= 25.427
  20:34:47 STEP 1821 H= 455:15 +CPU= 22.552
  20:35:09 STEP 1822 H= 455:30 +CPU= 22.254
  20:35:33 STEP 1823 H= 455:45 +CPU= 24.259
  20:36:20 STEP 1824 H= 456:00 +CPU= 33.557
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGh7zg+001824
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHh7zg+001824
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAh7zg+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615/ICMGGh7zg+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615/ICMSHh7zg+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615/ICMUAh7zg+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615/ICMGGh7zg+001740
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615/ICMSHh7zg+001740
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615/ICMUAh7zg+001740
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615/ICMGGh7zg+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615/ICMSHh7zg+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615/ICMUAh7zg+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615/ICMGGh7zg+001764
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615/ICMSHh7zg+001764
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615/ICMUAh7zg+001764
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615/ICMGGh7zg+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615/ICMSHh7zg+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615/ICMUAh7zg+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615/ICMGGh7zg+001788
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615/ICMSHh7zg+001788
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615/ICMUAh7zg+001788
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615/ICMGGh7zg+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615/ICMSHh7zg+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615/ICMUAh7zg+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615/ICMGGh7zg+001812
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615/ICMSHh7zg+001812
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615/ICMUAh7zg+001812
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a3b4_2016092300_20_1018_12292615_1_r536513267_18.zip
Uploading the intermediate file: upload_file_18.zip
  20:36:43 STEP 1825 H= 456:15 +CPU= 23.228
  20:37:06 STEP 1826 H= 456:30 +CPU= 22.679
  20:37:28 STEP 1827 H= 456:45 +CPU= 22.199
  20:37:51 STEP 1828 H= 457:00 +CPU= 22.976
  20:38:14 STEP 1829 H= 457:15 +CPU= 22.296
  20:38:37 STEP 1830 H= 457:30 +CPU= 23.139
  20:39:00 STEP 1831 H= 457:45 +CPU= 23.138
  20:39:24 STEP 1832 H= 458:00 +CPU= 24.192
  20:39:47 STEP 1833 H= 458:15 +CPU= 22.462
  20:40:10 STEP 1834 H= 458:30 +CPU= 23.362
  20:40:35 STEP 1835 H= 458:45 +CPU= 24.843
  20:41:03 STEP 1836 H= 459:00 +CPU= 28.445
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGh7zg+001836
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHh7zg+001836
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAh7zg+001836
  20:41:26 STEP 1837 H= 459:15 +CPU= 22.014
  20:41:49 STEP 1838 H= 459:30 +CPU= 22.949
  20:42:12 STEP 1839 H= 459:45 +CPU= 23.107
  20:42:35 STEP 1840 H= 460:00 +CPU= 22.922
  20:42:57 STEP 1841 H= 460:15 +CPU= 21.755
  20:43:19 STEP 1842 H= 460:30 +CPU= 22.806
  20:43:43 STEP 1843 H= 460:45 +CPU= 23.218
  20:44:06 STEP 1844 H= 461:00 +CPU= 23.424
  20:44:28 STEP 1845 H= 461:15 +CPU= 21.848
  20:44:51 STEP 1846 H= 461:30 +CPU= 22.859
  20:45:14 STEP 1847 H= 461:45 +CPU= 22.840
  20:45:43 STEP 1848 H= 462:00 +CPU= 29.416
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGh7zg+001848
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHh7zg+001848
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAh7zg+001848
  20:46:07 STEP 1849 H= 462:15 +CPU= 24.014
  20:46:30 STEP 1850 H= 462:30 +CPU= 22.311
  20:46:51 STEP 1851 H= 462:45 +CPU= 21.809
  20:47:14 STEP 1852 H= 463:00 +CPU= 22.625
  20:47:39 STEP 1853 H= 463:15 +CPU= 24.745
  20:48:01 STEP 1854 H= 463:30 +CPU= 22.341
  20:48:23 STEP 1855 H= 463:45 +CPU= 21.797
  20:48:45 STEP 1856 H= 464:00 +CPU= 22.386
  20:49:09 STEP 1857 H= 464:15 +CPU= 23.930
  20:49:32 STEP 1858 H= 464:30 +CPU= 22.360
  20:49:54 STEP 1859 H= 464:45 +CPU= 21.634
  20:50:22 STEP 1860 H= 465:00 +CPU= 28.313
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGh7zg+001860
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHh7zg+001860
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAh7zg+001860
  20:50:48 STEP 1861 H= 465:15 +CPU= 26.249
  20:51:15 STEP 1862 H= 465:30 +CPU= 24.800
  20:51:40 STEP 1863 H= 465:45 +CPU= 23.800
  20:52:04 STEP 1864 H= 466:00 +CPU= 24.060
  20:52:26 STEP 1865 H= 466:15 +CPU= 22.393
  20:52:48 STEP 1866 H= 466:30 +CPU= 21.375
  20:53:10 STEP 1867 H= 466:45 +CPU= 22.388
  20:53:35 STEP 1868 H= 467:00 +CPU= 24.783
  20:53:58 STEP 1869 H= 467:15 +CPU= 22.909
  20:54:20 STEP 1870 H= 467:30 +CPU= 21.934
  20:54:43 STEP 1871 H= 467:45 +CPU= 22.762
Quit request received from BOINC client, ending the child process
>>> Printing last 70 lines from file: NODE.001_01

         810  9.999999999999998E-005
 NORMS AT NSTEP CNT4 1872
 SPECTRAL NORMS -   LOG(PREHYDS)    0.115261330119519E+02
 LEV     VORTICITY             DIVERGENCE           TEMPERATURE           KINETIC ENERGY    
 AVE 0.786232870755245E-05 0.127795382888281E-05 0.215492362422966E+03 0.273485849711824E+02
 GPNORM  HUMIDITY            AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.606955779873916E-03 0.100000000000000E-07 0.799842037370520E-02
 GPNORM  LIQUID WATER        AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.781799127513830E-06 0.000000000000000E+00 0.564497075276504E-03
 GPNORM  ICE WATER           AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.133959435919888E-06 0.000000000000000E+00 0.636565818611813E-04
 GPNORM SNOW                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.957050864795914E-07 0.000000000000000E+00 0.921058698649420E-04
 GPNORM RAIN                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.325591703695516E-08 0.000000000000000E+00 0.260228482453687E-04
 GPNORM  CLOUD FRACTION      AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.116917441540550E-01 0.000000000000000E+00 0.100000000000000E+01
 GPNORM CLW                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.781745238938009E-06 0.000000000000000E+00 0.565466337564427E-03
 GPNORM CIW                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.133517324610512E-06 0.000000000000000E+00 0.639095259573048E-04
 GPNORM CC                   AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.116690231447539E-01 0.000000000000000E+00 0.100000000000000E+01
 GPNORM RFL                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.154200807041113E-07 0.000000000000000E+00 0.125010258574918E-03
 GPNORM SFL                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.138078682281604E-06 0.000000000000000E+00 0.117298930487722E-03
 GPNORM CRFL                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.197333311829849E-07 0.000000000000000E+00 0.933566357332259E-04
 GPNORM CSFL                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.868066432325696E-09 0.000000000000000E+00 0.763874854550682E-04
 GPNORM PFRC                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.314023172830058E-01 0.000000000000000E+00 0.100000000000000E+01
 NSTEP =  1872 STEPO   F00000000
 CREATE RESTART FILE ./srf00191200.0001                                                                                                                                                                                                                                              
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./srf00191200.0001 MODE=w
 IO-STREAM CLOSED - ./srf00191200.0001
 MPI-TASK:   1 - ************ BYTES IN     46 RECORDS TRANSFERRED IN    4.0017 SECONDS******** Mbytes/s, TOTAL TIME=    4.7768(83.8%)
 CREATE RESTART CONTROL FILE ./rcf                                                                                                                   
 WRRESF: DELETED PREVIOUS RESTART FILE OK: ./srf00190000.0001
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1872 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001872 MODE=w
 IO-STREAM CLOSED - ICMGGh7zg+001872
 MPI-TASK:   1 -    312714708 BYTES IN      6 RECORDS TRANSFERRED IN    0.0014 SECONDS223367.6 Mbytes/s, TOTAL TIME=    5.7454( 0.0%)
 NSTEP =  1872 STEPO   0AA000000
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMSHh7zg+001872 MODE=w
 IO-STREAM CLOSED - ./ICMSHh7zg+001872
 MPI-TASK:   1 -    707399250 BYTES IN    202 RECORDS TRANSFERRED IN    0.0178 SECONDS 39741.5 Mbytes/s, TOTAL TIME=    0.3659( 4.9%)
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMUAh7zg+001872 MODE=w
 IO-STREAM CLOSED - ./ICMUAh7zg+001872
 MPI-TASK:   1 -   5003435328 BYTES IN    125 RECORDS TRANSFERRED IN    0.0082 SECONDS610175.0 Mbytes/s, TOTAL TIME=    0.1795( 4.6%)
 STEPO_FPOS WAS : B00000F00E
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMUAh7zg+001872 MODE=a
 IO-STREAM CLOSED - ./ICMUAh7zg+001872
 MPI-TASK:   1 -           10 BYTES IN      2 RECORDS TRANSFERRED IN    0.0003 SECONDS     0.0 Mbytes/s, TOTAL TIME=    0.0035( 8.6%)
 STEPO_FPOS WAS : V00000000Y
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMUAh7zg+001872 MODE=a
 IO-STREAM CLOSED - ./ICMUAh7zg+001872
 MPI-TASK:   1 -    304134979 BYTES IN      4 RECORDS TRANSFERRED IN    0.0005 SECONDS608270.0 Mbytes/s, TOTAL TIME=    0.0060( 8.3%)
 STEPO_FPOS WAS : T00000000M
 NSTEP =  1872 STEPO   0AAA00AAA
------------------------------------------------
CNT0 not found; string returned was: 'STEPO'
>>> Printing last 8 lines from file: ifs.stat

 20:54:20 0AAA00AAA STEPO     1871    21.934   21.934   21.961     74:33     75:05 0.12305138621107E-05       2GB       0MB
 20:54:43 F00000000 STEPO     1872    22.951   22.951   23.144     74:56     75:29 0.12779538288828E-05       2GB       0MB
 20:55:04 0AA000000 STEPO     1872     3.924    3.924   20.654     75:00     75:49 0.12779538288828E-05       2GB       0MB
 20:55:05 FULLPOS-B DYNFPOS   1872     1.373    1.373    1.414     75:01     75:51 0.12779538288828E-05       2GB       0MB
 20:55:09 FULLPOS-V DYNFPOS   1872     3.095    3.095    3.482     75:04     75:54 0.12779538288828E-05       2GB       0MB
 20:55:09 FULLPOS-T DYNFPOS   1872     0.604    0.604    0.607     75:05     75:55 0.12779538288828E-05       2GB       0MB
 20:55:10 0AAA00AAA STEPO     1872     0.532    0.532    0.537     75:05     75:55 0.12779538288828E-05       2GB       0MB
------------------------------------------------
>>> Printing last 11 lines from file: rcf

 CSTEP   = "    1872",
 CTIME   = "00191200      ",
 NSTEPLPP        =         201,         202,          49,      228226,      228227, 2*0,        1860, 2*0,
 IPRGPNSRES      =           1,
 IPRGPEWRES      =           1,
 IPRTRWRES       =           1,
 IPRTRVRES       =           1,
 GMASS0  =  0.000000000000000E+000,
 GMASSI  =  0.000000000000000E+000
 /
------------------------------------------------
>>> Printing last 8 lines from file: /var/lib/boinc-client/slots/12/progress_file_12292615.xml

<running_values>
  <last_cpu_time>39203.570000</last_cpu_time>
  <upload_file_number>19</upload_file_number>
  <last_iter>1872</last_iter>
  <last_upload>1641600</last_upload>
  <model_completed>0</model_completed>
</running_values>
------------------------------------------------
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.13_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a3b4
(argv4) batchid: 1018
(argv5) wuid: 12292615
(argv6) fclen: 20
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/12
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.13
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/12/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/12/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_c5d0a6cab4482250225d38f560f1298d to: /var/lib/boinc-client/slots/12/oifs_43r3_bl_a3b4_2016092300_20_1018_12292615.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/12/oifs_43r3_bl_a3b4_2016092300_20_1018_12292615.zip
ic_ancil_file: ic_ancil_12292615
ifsdata_file: ifsdata_12292615
climate_data_file: clim_data_12292615
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /var/lib/boinc-client/slots/12/ic_ancil_12292615.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/12/ic_ancil_12292615.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_a6d21b067b4de9a3c175f3a00dc5ff65 to: /var/lib/boinc-client/slots/12/ifsdata/ifsdata_12292615.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/12/ifsdata/ifsdata_12292615.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/12/159l_2/clim_data_12292615.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/12/159l_2/clim_data_12292615.zip
Checking for progress XML file: /var/lib/boinc-client/slots/12/progress_file_12292615.xml
Opened progress file ok : /var/lib/boinc-client/slots/12/progress_file_12292615.xml
-- Model is restarting --
Adjusting last_iter, 1872, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/12/progress_file_12292615.xml
last_cpu_time: 39203
upload_file_number: 19
last_iter: 1873
last_upload: 1641600
model_completed: 0
total_length_of_simulation: 1728000
result_base_name: oifs_43r3_bl_a3b4_2016092300_20_1018_12292615_1_r536513267
The child process has been launched with process id: 15534
Executing the command: /var/lib/boinc-client/slots/12/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:server242:1:1:15534:15534] [20240617:205630:1718650590:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2091] fp = 0x314f440
[EC_DRHOOK:server242:1:1:15534:15534] [20240617:205630:1718650590:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:server242:1:1:15534:15534] [20240617:205630:1718650590:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:server242:1:1:15534:15534] [20240617:205630:1718650590:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:server242:1:1:15534:15534] [20240617:205630:1718650590:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:server242:1:1:15534:15534] [20240617:205630:1718650590:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:server242:1:1:15534:15534] [20240617:205630:1718650590:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:server242:1:1:15534:15534] [20240617:205630:1718650590:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:server242:1:1:15534:15534] [20240617:205630:1718650590:0.001] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:server242:1:1:15534:15534] [20240617:205630:1718650590:0.001] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:server242:1:1:15534:15534] [20240617:205630:1718650590:0.001] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:server242:1:1:15534:15534] [20240617:205630:1718650590:0.001] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:server242:1:1:15534:15534] [20240617:205630:1718650590:0.001] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:server242:1:1:15534:15534] [20240617:205630:1718650590:0.001] [ignore_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:server242:1:1:15534:15534] [20240617:205630:1718650590:0.001] [restore_default_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:server242:1:1:15534:15534] [20240617:205630:1718650590:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:server242:1:1:15534:15534] [20240617:205630:1718650590:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:server242:1:1:15534:15534] [20240617:205630:1718650590:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:server242:1:1:15534:15534] [20240617:205630:1718650590:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x820cf0 (old at (nil))
[EC_DRHOOK:server242:1:1:15534:15534] [20240617:205630:1718650590:0.001] [trapfpe_treatment@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:server242:1:1:15534:15534] [20240617:205630:1718650590:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:server242:1:1:15534:15534] [20240617:205630:1718650590:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:server242:1:1:15534:15534] [20240617:205630:1718650590:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x820cf0 (old at (nil))
[EC_DRHOOK:server242:1:1:15534:15534] [20240617:205630:1718650590:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:server242:1:1:15534:15534] [20240617:205630:1718650590:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:server242:1:1:15534:15534] [20240617:205630:1718650590:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:server242:1:1:15534:15534] [20240617:205630:1718650590:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x820cf0 (old at (nil))
[EC_DRHOOK:server242:1:1:15534:15534] [20240617:205630:1718650590:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x820cf0 (old at (nil))
[EC_DRHOOK:server242:1:1:15534:15534] [20240617:205630:1718650590:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:server242:1:1:15534:15534] [20240617:205630:1718650590:0.001] [catch_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  20:57:09 STEP 1872 H= 468:00 +CPU= 24.776
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGh7zg+001872
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHh7zg+001872
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAh7zg+001872
  20:57:31 STEP 1873 H= 468:15 +CPU= 21.715
  20:57:53 STEP 1874 H= 468:30 +CPU= 21.813
  20:58:16 STEP 1875 H= 468:45 +CPU= 23.655
  20:58:42 STEP 1876 H= 469:00 +CPU= 24.968
  20:59:03 STEP 1877 H= 469:15 +CPU= 21.617
  20:59:25 STEP 1878 H= 469:30 +CPU= 21.766
  20:59:48 STEP 1879 H= 469:45 +CPU= 23.305
  21:00:13 STEP 1880 H= 470:00 +CPU= 24.538
  21:00:35 STEP 1881 H= 470:15 +CPU= 22.029
  21:00:57 STEP 1882 H= 470:30 +CPU= 21.618
  21:01:21 STEP 1883 H= 470:45 +CPU= 24.338
  21:01:53 STEP 1884 H= 471:00 +CPU= 28.835
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGh7zg+001884
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHh7zg+001884
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAh7zg+001884
  21:02:15 STEP 1885 H= 471:15 +CPU= 21.831
  21:02:37 STEP 1886 H= 471:30 +CPU= 22.350
  21:03:01 STEP 1887 H= 471:45 +CPU= 23.619
  21:03:24 STEP 1888 H= 472:00 +CPU= 23.262
  21:03:46 STEP 1889 H= 472:15 +CPU= 21.995
  21:04:09 STEP 1890 H= 472:30 +CPU= 22.364
  21:04:32 STEP 1891 H= 472:45 +CPU= 22.731
  21:04:55 STEP 1892 H= 473:00 +CPU= 23.160
  21:05:18 STEP 1893 H= 473:15 +CPU= 23.052
  21:05:40 STEP 1894 H= 473:30 +CPU= 22.271
  21:06:03 STEP 1895 H= 473:45 +CPU= 22.615
  21:06:32 STEP 1896 H= 474:00 +CPU= 28.698
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGh7zg+001896
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHh7zg+001896
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAh7zg+001896
  21:06:54 STEP 1897 H= 474:15 +CPU= 22.512
  21:07:16 STEP 1898 H= 474:30 +CPU= 21.266
  21:07:38 STEP 1899 H= 474:45 +CPU= 22.032
  21:08:01 STEP 1900 H= 475:00 +CPU= 23.017
  21:08:22 STEP 1901 H= 475:15 +CPU= 21.673
  21:08:45 STEP 1902 H= 475:30 +CPU= 22.324
  21:09:08 STEP 1903 H= 475:45 +CPU= 23.311
  21:09:32 STEP 1904 H= 476:00 +CPU= 23.784
  21:09:54 STEP 1905 H= 476:15 +CPU= 21.523
  21:10:17 STEP 1906 H= 476:30 +CPU= 23.466
  21:10:39 STEP 1907 H= 476:45 +CPU= 21.895
  21:11:12 STEP 1908 H= 477:00 +CPU= 28.843
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGh7zg+001908
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHh7zg+001908
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAh7zg+001908
  21:11:34 STEP 1909 H= 477:15 +CPU= 22.148
  21:11:57 STEP 1910 H= 477:30 +CPU= 23.295
  21:12:22 STEP 1911 H= 477:45 +CPU= 24.931
  21:12:45 STEP 1912 H= 478:00 +CPU= 22.847
  21:13:07 STEP 1913 H= 478:15 +CPU= 21.651
  21:13:29 STEP 1914 H= 478:30 +CPU= 22.043
  21:13:54 STEP 1915 H= 478:45 +CPU= 24.632
  21:14:17 STEP 1916 H= 479:00 +CPU= 22.904
  21:14:38 STEP 1917 H= 479:15 +CPU= 21.515
  21:15:01 STEP 1918 H= 479:30 +CPU= 22.620
  21:15:25 STEP 1919 H= 479:45 +CPU= 23.864
  21:16:08 STEP 1920 H= 480:00 +CPU= 30.026
..The child process terminated with status: 0
>>> Printing last 70 lines from file: NODE.001_01

 
DDH------------------------------------------------------------------------
I                                                                         I
I TIME =  480.000H DDH * Diagnostic Var.,     AREA                GLOBAL  I
I                                                                         I
I-------------------------------------------------------------------------I
I   VARIABLE   I      Initial value         I        Current value        I
I-------------------------------------------------------------------------I
I  OMEGA (P/S) I      0.2682306347E-21      I       0.2783013275E-05      I
ICloud fractionI      0.0000000000E+00      I       0.1950435888E-01      I
I  Relat. Hum. I      0.2519773146E+00      I       0.3122795622E+00      I
I  PASS 01     I      0.0000000000E+00      I       0.3735710512E+03      I
DDH------------------------------------------------------------------------
  
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1920 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001920 MODE=a
 IO-STREAM CLOSED - ICMGGh7zg+001920
 MPI-TASK:   1 -    325592388 BYTES IN      7 RECORDS TRANSFERRED IN    0.0010 SECONDS325592.4 Mbytes/s, TOTAL TIME=    0.0119( 8.4%)
 21:16:08 STEP 1920 H= 480:00 +CPU= 30.026
 END CNT3
 NSTEP =  1920 CNT0   000000000  21.049  21.049  21.122   0.997               0               0               0               0               0
IO-STREAM STATISTICS, TOTAL NO OF REC/BYTES READ           46/  1069125708 WRITTEN         1430/************
 -TASK-OPENED-OPEN-RECS IN -KBYTE IN    -RECS OUT-KBYTE OUT-WALL   -WALL IN-WALL
  OU-TOT IN -TOT OUT
   1     28    0       46      1044068     1430 ************    17.8    11.7     6.1    12.6    14.7
===-=== START OF TIMING STATISTICS ===-===
 
STATS FOR ALL TASKS
 NUM ROUTINE                                     CALLS  MEAN(ms)   MAX(ms)   FRAC(%)  UNBAL(%)
   0 CNT0     - COMPLETE EXECUTION                   1 1177905.1 1177905.1    100.00      0.00
   1 CNT4     - FORWARD INTEGRATION                  1 1163654.1 1163654.1     98.79      0.00
   8 SCAN2M - GRID-POINT DYNAMICS                   53    4139.8    4139.8     18.63      0.00
   9 SPCM     - SPECTRAL COMP.                      48     675.3     675.3      2.75      0.00
  10 SCAN2M - PHYSICS                               49   10450.1   10450.1     43.47      0.00
  11 IOPACK   - OUTPUT P.P. RESULTS                  4    1445.9    1445.9      0.49      0.00
  12 SPNORM   - SPECTRAL NORM COMP.                  3      50.4      50.4      0.01      0.00
  17 GRIDFPOS IN CNT4                                4      27.4      27.4      0.01      0.00
  27 RESTART FILES - WRITING                         1    5765.3    5765.3      0.49      0.00
  28 RESTART FILES - READING                         1   13511.2   13511.2      1.15      0.00
  29 SU4FPOS IN CNT4                                 4       2.6       2.6      0.00      0.00
  30 DYNFPOS IN CNT4                                 4    9394.8    9394.8      3.19      0.00
  31 POSDDH IN STEPO                                 1      43.2      43.2      0.00      0.00
  37 CPGLAG   - SL COMPUTATIONS                     49    5089.6    5089.6     21.17      0.00
  39 SU0YOMB                                         1     427.9     427.9      0.04      0.00
  51 SCAN2M   - SL COMM. PART 1                     49      20.2      20.2      0.08      0.00
  54 SPCM     - M TO S/S TO M TRANSP.               48     317.8     317.8      1.29      0.00
  55 SPCIMPF  - S TO M/M TO S TRANSP.               48      50.7      50.7      0.21      0.00
  56 SPNORM   - SPECTRAL NORM COMM.                  3       0.2       0.2      0.00      0.00
 102 LTINV_CTL   - INVERSE LEGENDRE TRANSFORM      108     315.6     315.6      2.89      0.00
 103 LTDIR_CTL   - DIRECT LEGENDRE TRANSFORM       100     174.8     174.8      1.48      0.00
 106 FTDIR_CTL   - DIRECT FOURIER TRANSFORM        100      63.8      63.8      0.54      0.00
 107 FTINV_CTL   - INVERSE FOURIER TRANSFORM       108      87.7      87.7      0.80      0.00
 140 SULEG       - COMP. OF LEGENDRE POL.            1      34.7      34.7      0.00      0.00
 152 LTINV_CTL   - M TO L TRANSPOSITION            108      62.4      62.4      0.57      0.00
 153 LTDIR_CTL   - L TO M TRANSPOSITION            100      33.0      33.0      0.28      0.00
 157 FTINV_CTL   - L TO G TRANSPOSITION            108      60.3      60.3      0.55      0.00
 158 FTDIR_CTL   - G TO L TRANSPOSITION            100      34.3      34.3      0.29      0.00
 400 GSTATS                                       6667       0.0       0.0      0.00      0.00
 401 GSTATS HOOK                                  6149       0.0       0.0      0.00      0.00
TOTAL MEASURED IMBALANCE =       0.0 SECONDS,  0.0 PERCENT
TOTAL WALLCLOCK TIME     1177.9 CPU TIME    1147.0 VECTOR TIME     1147.0


===-=== END   OF TIMING STATISTICS ===-===
FORECAST DAYS PER DAY     1485.0
  *** END CNT0 *** 
------------------------------------------------
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGh7zg+001920
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHh7zg+001920
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAh7zg+001920
Adding to the zip: /var/lib/boinc-client/slots/12/NODE.001_01
Adding to the zip: /var/lib/boinc-client/slots/12/ifs.stat
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615/ICMSHh7zg+001836
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615/ICMUAh7zg+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615/ICMSHh7zg+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615/ICMUAh7zg+001836
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615/ICMGGh7zg+001908
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615/ICMGGh7zg+001884
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615/ICMSHh7zg+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615/ICMSHh7zg+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615/ICMUAh7zg+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615/ICMUAh7zg+001860
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615/ICMUAh7zg+001884
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615/ICMSHh7zg+001860
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615/ICMGGh7zg+001836
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615/ICMGGh7zg+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615/ICMSHh7zg+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615/ICMGGh7zg+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615/ICMSHh7zg+001908
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615/ICMSHh7zg+001884
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615/ICMGGh7zg+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615/ICMUAh7zg+001908
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615/ICMGGh7zg+001860
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615/ICMGGh7zg+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615/ICMUAh7zg+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615/ICMUAh7zg+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615/ICMSHh7zg+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615/ICMGGh7zg+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292615/ICMUAh7zg+001848
Zipping up the final file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a3b4_2016092300_20_1018_12292615_1_r536513267_19.zip
Uploading the final file: upload_file_19.zip
Uploading trickle at timestep: 1727100
21:19:43 (15529): called boinc_finish(0)

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
17 Jun 2024 19:17:50 1469580 22448576 oifs_43r3_bl_a3b4_2016092300_20_1018_12292615_1 1,727,100 40,350 0.0234
17 Jun 2024 13:56:13 1469580 22448576 oifs_43r3_bl_a3b4_2016092300_20_1018_12292615_1 864,000 18,933 0.0219


©2024 climateprediction.net