climateprediction.net home page
Task 22444686

Task 22444686

Name oifs_43r3_bl_a24p_2016092300_20_1018_12291088_0
Workunit 12291088
Created 13 Jun 2024, 0:02:17 UTC
Sent 13 Jun 2024, 18:12:09 UTC
Report deadline 12 Aug 2024, 18:12:09 UTC
Received 18 Jun 2024, 5:50:24 UTC
Server state Over
Outcome Success
Client state Done
Exit status 0 (0x00000000)
Computer ID 1469580
Run time 12 hours 43 min 55 sec
CPU time 12 hours 43 min 55 sec
Validate state Valid
Credit 2,635.55
Device peak FLOPS 5.26 GFLOPS
Application version OpenIFS 43r3 Baroclinic Lifecycle v1.13
x86_64-pc-linux-gnu
Peak working set size 5,016.61 MB
Peak swap size 5,981.13 MB
Peak disk usage 1,371.96 MB
Stderr
<core_client_version>7.18.1</core_client_version>
<![CDATA[
<stderr_txt>
es: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /var/lib/boinc-client/slots/17/ic_ancil_12291088.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/17/ic_ancil_12291088.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_cebb04c1212c52df13fc0f82fbda623b to: /var/lib/boinc-client/slots/17/ifsdata/ifsdata_12291088.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/17/ifsdata/ifsdata_12291088.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/17/159l_2/clim_data_12291088.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/17/159l_2/clim_data_12291088.zip
Checking for progress XML file: /var/lib/boinc-client/slots/17/progress_file_12291088.xml
Opened progress file ok : /var/lib/boinc-client/slots/17/progress_file_12291088.xml
-- Model is restarting --
Adjusting last_iter, 1537, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/17/progress_file_12291088.xml
last_cpu_time: 37379
upload_file_number: 15
last_iter: 1537
last_upload: 1296900
model_completed: 0
total_length_of_simulation: 1728000
result_base_name: oifs_43r3_bl_a24p_2016092300_20_1018_12291088_0_r354668119
The child process has been launched with process id: 25883
Executing the command: /var/lib/boinc-client/slots/17/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:server242:1:1:25883:25883] [20240618:045743:1718679463:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2091] fp = 0x314f440
[EC_DRHOOK:server242:1:1:25883:25883] [20240618:045743:1718679463:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:server242:1:1:25883:25883] [20240618:045743:1718679463:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:server242:1:1:25883:25883] [20240618:045743:1718679463:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:server242:1:1:25883:25883] [20240618:045743:1718679463:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:server242:1:1:25883:25883] [20240618:045743:1718679463:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:server242:1:1:25883:25883] [20240618:045743:1718679463:0.001] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:server242:1:1:25883:25883] [20240618:045743:1718679463:0.001] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:server242:1:1:25883:25883] [20240618:045743:1718679463:0.001] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:server242:1:1:25883:25883] [20240618:045743:1718679463:0.001] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:server242:1:1:25883:25883] [20240618:045743:1718679463:0.001] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:server242:1:1:25883:25883] [20240618:045743:1718679463:0.001] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:server242:1:1:25883:25883] [20240618:045743:1718679463:0.001] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:server242:1:1:25883:25883] [20240618:045743:1718679463:0.001] [ignore_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:server242:1:1:25883:25883] [20240618:045743:1718679463:0.001] [restore_default_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:server242:1:1:25883:25883] [20240618:045743:1718679463:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:server242:1:1:25883:25883] [20240618:045743:1718679463:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:server242:1:1:25883:25883] [20240618:045743:1718679463:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:server242:1:1:25883:25883] [20240618:045743:1718679463:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x820cf0 (old at (nil))
[EC_DRHOOK:server242:1:1:25883:25883] [20240618:045743:1718679463:0.001] [trapfpe_treatment@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:server242:1:1:25883:25883] [20240618:045743:1718679463:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:server242:1:1:25883:25883] [20240618:045743:1718679463:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:server242:1:1:25883:25883] [20240618:045743:1718679463:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x820cf0 (old at (nil))
[EC_DRHOOK:server242:1:1:25883:25883] [20240618:045743:1718679463:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:server242:1:1:25883:25883] [20240618:045743:1718679463:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:server242:1:1:25883:25883] [20240618:045743:1718679463:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:server242:1:1:25883:25883] [20240618:045743:1718679463:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x820cf0 (old at (nil))
[EC_DRHOOK:server242:1:1:25883:25883] [20240618:045743:1718679463:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x820cf0 (old at (nil))
[EC_DRHOOK:server242:1:1:25883:25883] [20240618:045743:1718679463:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:server242:1:1:25883:25883] [20240618:045743:1718679463:0.001] [catch_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMGGh7zg+001452
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMSHh7zg+001452
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMUAh7zg+001452
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMGGh7zg+001464
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMSHh7zg+001464
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMUAh7zg+001464
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMGGh7zg+001476
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMSHh7zg+001476
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMUAh7zg+001476
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMGGh7zg+001488
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMSHh7zg+001488
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMUAh7zg+001488
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMGGh7zg+001500
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMSHh7zg+001500
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMUAh7zg+001500
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMGGh7zg+001512
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMSHh7zg+001512
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMUAh7zg+001512
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMGGh7zg+001524
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMSHh7zg+001524
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMUAh7zg+001524
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMGGh7zg+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMSHh7zg+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMUAh7zg+001536
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a24p_2016092300_20_1018_12291088_0_r354668119_15.zip
Uploading the intermediate file: upload_file_15.zip
  04:58:10 STEP 1536 H= 384:00 +CPU= 23.867
  04:58:31 STEP 1537 H= 384:15 +CPU= 21.117
  04:58:52 STEP 1538 H= 384:30 +CPU= 20.716
  04:59:15 STEP 1539 H= 384:45 +CPU= 22.360
  04:59:37 STEP 1540 H= 385:00 +CPU= 21.931
  04:59:58 STEP 1541 H= 385:15 +CPU= 21.145
  05:00:20 STEP 1542 H= 385:30 +CPU= 21.917
  05:00:43 STEP 1543 H= 385:45 +CPU= 22.920
  05:01:06 STEP 1544 H= 386:00 +CPU= 23.299
  05:01:27 STEP 1545 H= 386:15 +CPU= 20.705
  05:01:48 STEP 1546 H= 386:30 +CPU= 21.337
  05:02:10 STEP 1547 H= 386:45 +CPU= 21.858
  05:02:38 STEP 1548 H= 387:00 +CPU= 27.601
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMGGh7zg+001548
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMSHh7zg+001548
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMUAh7zg+001548
  05:02:59 STEP 1549 H= 387:15 +CPU= 20.781
  05:03:21 STEP 1550 H= 387:30 +CPU= 22.201
  05:03:45 STEP 1551 H= 387:45 +CPU= 23.765
  05:04:07 STEP 1552 H= 388:00 +CPU= 22.465
  05:04:28 STEP 1553 H= 388:15 +CPU= 21.166
  05:04:50 STEP 1554 H= 388:30 +CPU= 21.239
  05:05:14 STEP 1555 H= 388:45 +CPU= 24.180
  05:05:36 STEP 1556 H= 389:00 +CPU= 22.363
  05:05:56 STEP 1557 H= 389:15 +CPU= 19.467
  05:06:16 STEP 1558 H= 389:30 +CPU= 20.144
  05:06:37 STEP 1559 H= 389:45 +CPU= 20.661
  05:07:06 STEP 1560 H= 390:00 +CPU= 28.291
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMGGh7zg+001560
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMSHh7zg+001560
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMUAh7zg+001560
  05:07:27 STEP 1561 H= 390:15 +CPU= 21.255
  05:07:48 STEP 1562 H= 390:30 +CPU= 21.286
  05:08:15 STEP 1563 H= 390:45 +CPU= 24.082
  05:08:37 STEP 1564 H= 391:00 +CPU= 21.893
  05:08:58 STEP 1565 H= 391:15 +CPU= 21.079
  05:09:20 STEP 1566 H= 391:30 +CPU= 21.870
  05:09:44 STEP 1567 H= 391:45 +CPU= 23.266
  05:10:07 STEP 1568 H= 392:00 +CPU= 22.802
  05:10:28 STEP 1569 H= 392:15 +CPU= 21.350
  05:10:51 STEP 1570 H= 392:30 +CPU= 22.737
  05:11:14 STEP 1571 H= 392:45 +CPU= 22.630
  05:11:46 STEP 1572 H= 393:00 +CPU= 27.171
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMGGh7zg+001572
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMSHh7zg+001572
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMUAh7zg+001572
  05:12:07 STEP 1573 H= 393:15 +CPU= 20.907
  05:12:29 STEP 1574 H= 393:30 +CPU= 22.256
  05:12:51 STEP 1575 H= 393:45 +CPU= 22.519
  05:13:13 STEP 1576 H= 394:00 +CPU= 21.817
  05:13:34 STEP 1577 H= 394:15 +CPU= 20.742
  05:13:57 STEP 1578 H= 394:30 +CPU= 22.583
  05:14:20 STEP 1579 H= 394:45 +CPU= 23.770
  05:14:42 STEP 1580 H= 395:00 +CPU= 21.196
  05:15:03 STEP 1581 H= 395:15 +CPU= 20.949
  05:15:26 STEP 1582 H= 395:30 +CPU= 23.512
  05:15:49 STEP 1583 H= 395:45 +CPU= 22.456
  05:16:24 STEP 1584 H= 396:00 +CPU= 32.442
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMGGh7zg+001584
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMSHh7zg+001584
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMUAh7zg+001584
  05:16:45 STEP 1585 H= 396:15 +CPU= 20.888
  05:17:06 STEP 1586 H= 396:30 +CPU= 21.072
  05:17:27 STEP 1587 H= 396:45 +CPU= 21.354
  05:17:50 STEP 1588 H= 397:00 +CPU= 22.605
  05:18:11 STEP 1589 H= 397:15 +CPU= 21.379
  05:18:32 STEP 1590 H= 397:30 +CPU= 21.023
  05:18:54 STEP 1591 H= 397:45 +CPU= 21.432
  05:19:19 STEP 1592 H= 398:00 +CPU= 25.505
  05:19:42 STEP 1593 H= 398:15 +CPU= 21.583
  05:20:03 STEP 1594 H= 398:30 +CPU= 20.970
  05:20:26 STEP 1595 H= 398:45 +CPU= 22.144
  05:20:57 STEP 1596 H= 399:00 +CPU= 30.358
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMGGh7zg+001596
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMSHh7zg+001596
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMUAh7zg+001596
  05:21:19 STEP 1597 H= 399:15 +CPU= 21.863
  05:21:40 STEP 1598 H= 399:30 +CPU= 20.838
  05:22:02 STEP 1599 H= 399:45 +CPU= 21.962
  05:22:26 STEP 1600 H= 400:00 +CPU= 23.812
  05:22:47 STEP 1601 H= 400:15 +CPU= 20.841
  05:23:08 STEP 1602 H= 400:30 +CPU= 20.992
  05:23:30 STEP 1603 H= 400:45 +CPU= 22.182
  05:23:54 STEP 1604 H= 401:00 +CPU= 24.126
  05:24:15 STEP 1605 H= 401:15 +CPU= 20.925
  05:24:36 STEP 1606 H= 401:30 +CPU= 21.153
  05:24:58 STEP 1607 H= 401:45 +CPU= 21.470
  05:25:29 STEP 1608 H= 402:00 +CPU= 30.481
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMGGh7zg+001608
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMSHh7zg+001608
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMUAh7zg+001608
  05:25:50 STEP 1609 H= 402:15 +CPU= 21.320
  05:26:12 STEP 1610 H= 402:30 +CPU= 21.358
  05:26:34 STEP 1611 H= 402:45 +CPU= 22.555
  05:26:58 STEP 1612 H= 403:00 +CPU= 23.784
  05:27:20 STEP 1613 H= 403:15 +CPU= 21.481
  05:27:41 STEP 1614 H= 403:30 +CPU= 21.125
  05:28:06 STEP 1615 H= 403:45 +CPU= 22.499
  05:28:30 STEP 1616 H= 404:00 +CPU= 23.685
  05:28:56 STEP 1617 H= 404:15 +CPU= 20.619
  05:29:17 STEP 1618 H= 404:30 +CPU= 20.799
  05:29:40 STEP 1619 H= 404:45 +CPU= 22.818
  05:30:09 STEP 1620 H= 405:00 +CPU= 28.396
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMGGh7zg+001620
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMSHh7zg+001620
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMUAh7zg+001620
  05:30:30 STEP 1621 H= 405:15 +CPU= 21.721
  05:30:52 STEP 1622 H= 405:30 +CPU= 21.806
  05:31:15 STEP 1623 H= 405:45 +CPU= 23.157
  05:31:38 STEP 1624 H= 406:00 +CPU= 22.824
  05:32:00 STEP 1625 H= 406:15 +CPU= 21.216
  05:32:22 STEP 1626 H= 406:30 +CPU= 22.415
  05:32:44 STEP 1627 H= 406:45 +CPU= 22.224
  05:33:06 STEP 1628 H= 407:00 +CPU= 21.582
  05:33:27 STEP 1629 H= 407:15 +CPU= 20.746
  05:33:49 STEP 1630 H= 407:30 +CPU= 22.112
  05:34:10 STEP 1631 H= 407:45 +CPU= 21.598
  05:34:41 STEP 1632 H= 408:00 +CPU= 29.611
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMGGh7zg+001632
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMSHh7zg+001632
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMUAh7zg+001632
  05:35:03 STEP 1633 H= 408:15 +CPU= 21.583
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMGGh7zg+001548
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMSHh7zg+001548
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMUAh7zg+001548
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMGGh7zg+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMSHh7zg+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMUAh7zg+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMGGh7zg+001572
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMSHh7zg+001572
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMUAh7zg+001572
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMGGh7zg+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMSHh7zg+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMUAh7zg+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMGGh7zg+001596
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMSHh7zg+001596
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMUAh7zg+001596
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMGGh7zg+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMSHh7zg+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMUAh7zg+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMGGh7zg+001620
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMSHh7zg+001620
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMUAh7zg+001620
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMGGh7zg+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMSHh7zg+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMUAh7zg+001632
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a24p_2016092300_20_1018_12291088_0_r354668119_16.zip
  05:35:29 STEP 1634 H= 408:30 +CPU= 25.810
Uploading the intermediate file: upload_file_16.zip
  05:35:50 STEP 1635 H= 408:45 +CPU= 21.473
  05:36:12 STEP 1636 H= 409:00 +CPU= 21.385
  05:36:33 STEP 1637 H= 409:15 +CPU= 21.173
  05:36:58 STEP 1638 H= 409:30 +CPU= 23.845
  05:37:26 STEP 1639 H= 409:45 +CPU= 22.469
  05:37:48 STEP 1640 H= 410:00 +CPU= 21.880
  05:38:10 STEP 1641 H= 410:15 +CPU= 21.654
  05:38:33 STEP 1642 H= 410:30 +CPU= 22.446
  05:38:56 STEP 1643 H= 410:45 +CPU= 23.175
  05:39:28 STEP 1644 H= 411:00 +CPU= 28.297
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMGGh7zg+001644
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMSHh7zg+001644
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMUAh7zg+001644
  05:39:52 STEP 1645 H= 411:15 +CPU= 23.246
  05:40:15 STEP 1646 H= 411:30 +CPU= 23.148
  05:40:36 STEP 1647 H= 411:45 +CPU= 20.781
  05:40:57 STEP 1648 H= 412:00 +CPU= 21.314
  05:41:20 STEP 1649 H= 412:15 +CPU= 23.104
  05:41:42 STEP 1650 H= 412:30 +CPU= 21.984
  05:42:04 STEP 1651 H= 412:45 +CPU= 21.441
  05:42:26 STEP 1652 H= 413:00 +CPU= 21.754
  05:42:48 STEP 1653 H= 413:15 +CPU= 22.532
  05:43:12 STEP 1654 H= 413:30 +CPU= 23.199
  05:43:33 STEP 1655 H= 413:45 +CPU= 21.592
  05:44:00 STEP 1656 H= 414:00 +CPU= 26.418
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMGGh7zg+001656
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMSHh7zg+001656
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMUAh7zg+001656
  05:44:22 STEP 1657 H= 414:15 +CPU= 22.066
  05:44:45 STEP 1658 H= 414:30 +CPU= 22.716
  05:45:08 STEP 1659 H= 414:45 +CPU= 21.438
  05:45:29 STEP 1660 H= 415:00 +CPU= 21.678
  05:45:51 STEP 1661 H= 415:15 +CPU= 21.476
  05:46:14 STEP 1662 H= 415:30 +CPU= 22.461
  05:46:37 STEP 1663 H= 415:45 +CPU= 22.596
  05:47:00 STEP 1664 H= 416:00 +CPU= 22.570
  05:47:22 STEP 1665 H= 416:15 +CPU= 21.959
  05:47:42 STEP 1666 H= 416:30 +CPU= 20.671
  05:48:04 STEP 1667 H= 416:45 +CPU= 22.066
  05:48:33 STEP 1668 H= 417:00 +CPU= 28.255
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMGGh7zg+001668
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMSHh7zg+001668
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMUAh7zg+001668
  05:48:54 STEP 1669 H= 417:15 +CPU= 20.959
  05:49:15 STEP 1670 H= 417:30 +CPU= 21.416
  05:49:39 STEP 1671 H= 417:45 +CPU= 23.628
  05:50:00 STEP 1672 H= 418:00 +CPU= 21.484
  05:50:22 STEP 1673 H= 418:15 +CPU= 21.548
  05:50:44 STEP 1674 H= 418:30 +CPU= 21.910
  05:51:05 STEP 1675 H= 418:45 +CPU= 21.448
  05:51:27 STEP 1676 H= 419:00 +CPU= 21.623
  05:51:48 STEP 1677 H= 419:15 +CPU= 21.037
  05:52:10 STEP 1678 H= 419:30 +CPU= 22.153
  05:52:32 STEP 1679 H= 419:45 +CPU= 21.822
  05:53:04 STEP 1680 H= 420:00 +CPU= 29.827
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMGGh7zg+001680
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMSHh7zg+001680
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMUAh7zg+001680
  05:53:25 STEP 1681 H= 420:15 +CPU= 20.761
  05:53:47 STEP 1682 H= 420:30 +CPU= 22.019
  05:54:09 STEP 1683 H= 420:45 +CPU= 21.558
  05:54:30 STEP 1684 H= 421:00 +CPU= 21.438
  05:54:51 STEP 1685 H= 421:15 +CPU= 21.068
  05:55:13 STEP 1686 H= 421:30 +CPU= 22.193
  05:55:36 STEP 1687 H= 421:45 +CPU= 22.104
  05:55:58 STEP 1688 H= 422:00 +CPU= 22.201
  05:56:19 STEP 1689 H= 422:15 +CPU= 20.971
  05:56:40 STEP 1690 H= 422:30 +CPU= 20.885
  05:57:02 STEP 1691 H= 422:45 +CPU= 22.186
  05:57:29 STEP 1692 H= 423:00 +CPU= 26.455
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMGGh7zg+001692
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMSHh7zg+001692
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMUAh7zg+001692
  05:57:50 STEP 1693 H= 423:15 +CPU= 21.205
  05:58:11 STEP 1694 H= 423:30 +CPU= 20.804
  05:58:33 STEP 1695 H= 423:45 +CPU= 22.151
  05:58:55 STEP 1696 H= 424:00 +CPU= 22.237
  05:59:16 STEP 1697 H= 424:15 +CPU= 21.269
  05:59:37 STEP 1698 H= 424:30 +CPU= 20.625
  05:59:58 STEP 1699 H= 424:45 +CPU= 21.322
  06:00:22 STEP 1700 H= 425:00 +CPU= 23.849
  06:00:44 STEP 1701 H= 425:15 +CPU= 21.344
  06:01:05 STEP 1702 H= 425:30 +CPU= 21.271
  06:01:27 STEP 1703 H= 425:45 +CPU= 21.733
  06:01:54 STEP 1704 H= 426:00 +CPU= 27.476
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMGGh7zg+001704
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMSHh7zg+001704
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMUAh7zg+001704
  06:02:16 STEP 1705 H= 426:15 +CPU= 21.190
  06:02:37 STEP 1706 H= 426:30 +CPU= 21.115
  06:02:59 STEP 1707 H= 426:45 +CPU= 22.012
  06:03:21 STEP 1708 H= 427:00 +CPU= 22.585
  06:03:42 STEP 1709 H= 427:15 +CPU= 20.648
  06:04:03 STEP 1710 H= 427:30 +CPU= 20.660
  06:04:26 STEP 1711 H= 427:45 +CPU= 22.913
  06:04:49 STEP 1712 H= 428:00 +CPU= 22.848
  06:05:13 STEP 1713 H= 428:15 +CPU= 22.521
  06:05:37 STEP 1714 H= 428:30 +CPU= 23.577
  06:05:58 STEP 1715 H= 428:45 +CPU= 20.812
  06:06:23 STEP 1716 H= 429:00 +CPU= 25.359
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMGGh7zg+001716
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMSHh7zg+001716
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMUAh7zg+001716
  06:06:44 STEP 1717 H= 429:15 +CPU= 20.334
  06:07:05 STEP 1718 H= 429:30 +CPU= 21.266
  06:07:27 STEP 1719 H= 429:45 +CPU= 22.316
  06:07:49 STEP 1720 H= 430:00 +CPU= 22.195
  06:08:10 STEP 1721 H= 430:15 +CPU= 20.734
  06:08:31 STEP 1722 H= 430:30 +CPU= 21.042
  06:08:53 STEP 1723 H= 430:45 +CPU= 21.408
  06:09:15 STEP 1724 H= 431:00 +CPU= 22.374
  06:09:36 STEP 1725 H= 431:15 +CPU= 20.982
  06:09:57 STEP 1726 H= 431:30 +CPU= 20.795
  06:10:19 STEP 1727 H= 431:45 +CPU= 22.452
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.13_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a24p
(argv4) batchid: 1018
(argv5) wuid: 12291088
(argv6) fclen: 20
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/17
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.13
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/17/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/17/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_ebe6e41a42e401a67f8f9dd710f0bc4d to: /var/lib/boinc-client/slots/17/oifs_43r3_bl_a24p_2016092300_20_1018_12291088.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/17/oifs_43r3_bl_a24p_2016092300_20_1018_12291088.zip
ic_ancil_file: ic_ancil_12291088
ifsdata_file: ifsdata_12291088
climate_data_file: clim_data_12291088
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /var/lib/boinc-client/slots/17/ic_ancil_12291088.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/17/ic_ancil_12291088.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_cebb04c1212c52df13fc0f82fbda623b to: /var/lib/boinc-client/slots/17/ifsdata/ifsdata_12291088.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/17/ifsdata/ifsdata_12291088.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/17/159l_2/clim_data_12291088.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/17/159l_2/clim_data_12291088.zip
Checking for progress XML file: /var/lib/boinc-client/slots/17/progress_file_12291088.xml
Opened progress file ok : /var/lib/boinc-client/slots/17/progress_file_12291088.xml
-- Model is restarting --
Adjusting last_iter, 1728, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/17/progress_file_12291088.xml
last_cpu_time: 41687
upload_file_number: 17
last_iter: 1729
last_upload: 1469700
model_completed: 0
total_length_of_simulation: 1728000
result_base_name: oifs_43r3_bl_a24p_2016092300_20_1018_12291088_0_r354668119
The child process has been launched with process id: 27411
Executing the command: /var/lib/boinc-client/slots/17/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:server242:1:1:27411:27411] [20240618:061130:1718683890:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2091] fp = 0x314f440
[EC_DRHOOK:server242:1:1:27411:27411] [20240618:061130:1718683890:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:server242:1:1:27411:27411] [20240618:061130:1718683890:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:server242:1:1:27411:27411] [20240618:061130:1718683890:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:server242:1:1:27411:27411] [20240618:061130:1718683890:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:server242:1:1:27411:27411] [20240618:061130:1718683890:0.001] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:server242:1:1:27411:27411] [20240618:061130:1718683890:0.001] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:server242:1:1:27411:27411] [20240618:061130:1718683890:0.001] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:server242:1:1:27411:27411] [20240618:061130:1718683890:0.001] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:server242:1:1:27411:27411] [20240618:061130:1718683890:0.001] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:server242:1:1:27411:27411] [20240618:061130:1718683890:0.001] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:server242:1:1:27411:27411] [20240618:061130:1718683890:0.001] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:server242:1:1:27411:27411] [20240618:061130:1718683890:0.001] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:server242:1:1:27411:27411] [20240618:061130:1718683890:0.001] [ignore_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:server242:1:1:27411:27411] [20240618:061130:1718683890:0.001] [restore_default_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:server242:1:1:27411:27411] [20240618:061130:1718683890:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:server242:1:1:27411:27411] [20240618:061130:1718683890:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:server242:1:1:27411:27411] [20240618:061130:1718683890:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:server242:1:1:27411:27411] [20240618:061130:1718683890:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x820cf0 (old at (nil))
[EC_DRHOOK:server242:1:1:27411:27411] [20240618:061130:1718683890:0.001] [trapfpe_treatment@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:server242:1:1:27411:27411] [20240618:061130:1718683890:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:server242:1:1:27411:27411] [20240618:061130:1718683890:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:server242:1:1:27411:27411] [20240618:061130:1718683890:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x820cf0 (old at (nil))
[EC_DRHOOK:server242:1:1:27411:27411] [20240618:061130:1718683890:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:server242:1:1:27411:27411] [20240618:061130:1718683890:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:server242:1:1:27411:27411] [20240618:061130:1718683890:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:server242:1:1:27411:27411] [20240618:061130:1718683890:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x820cf0 (old at (nil))
[EC_DRHOOK:server242:1:1:27411:27411] [20240618:061130:1718683890:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x820cf0 (old at (nil))
[EC_DRHOOK:server242:1:1:27411:27411] [20240618:061130:1718683890:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:server242:1:1:27411:27411] [20240618:061130:1718683890:0.001] [catch_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMGGh7zg+001644
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMSHh7zg+001644
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMUAh7zg+001644
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMGGh7zg+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMSHh7zg+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMUAh7zg+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMGGh7zg+001668
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMSHh7zg+001668
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMUAh7zg+001668
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMGGh7zg+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMSHh7zg+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMUAh7zg+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMGGh7zg+001692
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMSHh7zg+001692
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMUAh7zg+001692
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMGGh7zg+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMSHh7zg+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMUAh7zg+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMGGh7zg+001716
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMSHh7zg+001716
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMUAh7zg+001716
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a24p_2016092300_20_1018_12291088_0_r354668119_17.zip
Uploading the intermediate file: upload_file_17.zip
  06:11:58 STEP 1728 H= 432:00 +CPU= 23.428
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMGGh7zg+001728
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMSHh7zg+001728
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMUAh7zg+001728
  06:12:19 STEP 1729 H= 432:15 +CPU= 21.231
  06:12:40 STEP 1730 H= 432:30 +CPU= 20.931
  06:13:01 STEP 1731 H= 432:45 +CPU= 21.152
  06:13:22 STEP 1732 H= 433:00 +CPU= 21.403
  06:13:43 STEP 1733 H= 433:15 +CPU= 20.915
  06:14:04 STEP 1734 H= 433:30 +CPU= 21.003
  06:14:26 STEP 1735 H= 433:45 +CPU= 21.922
  06:14:48 STEP 1736 H= 434:00 +CPU= 21.484
  06:15:09 STEP 1737 H= 434:15 +CPU= 21.078
  06:15:31 STEP 1738 H= 434:30 +CPU= 21.726
  06:15:52 STEP 1739 H= 434:45 +CPU= 20.800
  06:16:19 STEP 1740 H= 435:00 +CPU= 26.694
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMGGh7zg+001740
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMSHh7zg+001740
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMUAh7zg+001740
  06:16:40 STEP 1741 H= 435:15 +CPU= 21.593
  06:17:02 STEP 1742 H= 435:30 +CPU= 21.317
  06:17:23 STEP 1743 H= 435:45 +CPU= 21.332
  06:17:44 STEP 1744 H= 436:00 +CPU= 21.209
  06:18:06 STEP 1745 H= 436:15 +CPU= 21.710
  06:18:28 STEP 1746 H= 436:30 +CPU= 21.448
  06:18:49 STEP 1747 H= 436:45 +CPU= 20.997
  06:19:10 STEP 1748 H= 437:00 +CPU= 21.436
  06:19:32 STEP 1749 H= 437:15 +CPU= 22.115
  06:19:53 STEP 1750 H= 437:30 +CPU= 20.855
  06:20:15 STEP 1751 H= 437:45 +CPU= 21.629
  06:20:43 STEP 1752 H= 438:00 +CPU= 27.633
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMGGh7zg+001752
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMSHh7zg+001752
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMUAh7zg+001752
  06:21:05 STEP 1753 H= 438:15 +CPU= 21.907
  06:21:26 STEP 1754 H= 438:30 +CPU= 21.264
  06:21:47 STEP 1755 H= 438:45 +CPU= 20.811
  06:22:09 STEP 1756 H= 439:00 +CPU= 21.619
  06:22:30 STEP 1757 H= 439:15 +CPU= 21.288
  06:22:51 STEP 1758 H= 439:30 +CPU= 20.720
  06:23:12 STEP 1759 H= 439:45 +CPU= 21.136
  06:23:34 STEP 1760 H= 440:00 +CPU= 22.042
  06:23:55 STEP 1761 H= 440:15 +CPU= 20.723
  06:24:15 STEP 1762 H= 440:30 +CPU= 20.583
  06:24:35 STEP 1763 H= 440:45 +CPU= 20.029
  06:25:04 STEP 1764 H= 441:00 +CPU= 26.368
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMGGh7zg+001764
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMSHh7zg+001764
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMUAh7zg+001764
  06:25:26 STEP 1765 H= 441:15 +CPU= 21.496
  06:25:47 STEP 1766 H= 441:30 +CPU= 21.081
  06:26:08 STEP 1767 H= 441:45 +CPU= 20.866
  06:26:30 STEP 1768 H= 442:00 +CPU= 21.679
  06:26:51 STEP 1769 H= 442:15 +CPU= 20.850
  06:27:11 STEP 1770 H= 442:30 +CPU= 20.563
  06:27:32 STEP 1771 H= 442:45 +CPU= 20.637
  06:27:54 STEP 1772 H= 443:00 +CPU= 21.763
  06:28:15 STEP 1773 H= 443:15 +CPU= 21.118
  06:28:36 STEP 1774 H= 443:30 +CPU= 20.816
  06:28:57 STEP 1775 H= 443:45 +CPU= 21.235
  06:29:28 STEP 1776 H= 444:00 +CPU= 30.594
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMGGh7zg+001776
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMSHh7zg+001776
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMUAh7zg+001776
  06:29:48 STEP 1777 H= 444:15 +CPU= 20.608
  06:30:08 STEP 1778 H= 444:30 +CPU= 19.974
  06:30:29 STEP 1779 H= 444:45 +CPU= 20.607
  06:30:50 STEP 1780 H= 445:00 +CPU= 21.272
  06:31:11 STEP 1781 H= 445:15 +CPU= 21.164
  06:31:33 STEP 1782 H= 445:30 +CPU= 21.646
  06:31:54 STEP 1783 H= 445:45 +CPU= 20.879
  06:32:16 STEP 1784 H= 446:00 +CPU= 21.938
  06:32:37 STEP 1785 H= 446:15 +CPU= 21.051
  06:32:58 STEP 1786 H= 446:30 +CPU= 21.135
  06:33:19 STEP 1787 H= 446:45 +CPU= 20.838
  06:33:45 STEP 1788 H= 447:00 +CPU= 26.478
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMGGh7zg+001788
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMSHh7zg+001788
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMUAh7zg+001788
  06:34:06 STEP 1789 H= 447:15 +CPU= 21.065
  06:34:28 STEP 1790 H= 447:30 +CPU= 21.636
  06:34:49 STEP 1791 H= 447:45 +CPU= 20.990
  06:35:11 STEP 1792 H= 448:00 +CPU= 21.470
  06:35:30 STEP 1793 H= 448:15 +CPU= 19.483
  06:35:49 STEP 1794 H= 448:30 +CPU= 19.132
  06:36:09 STEP 1795 H= 448:45 +CPU= 19.511
  06:36:30 STEP 1796 H= 449:00 +CPU= 21.656
  06:36:51 STEP 1797 H= 449:15 +CPU= 20.854
  06:37:13 STEP 1798 H= 449:30 +CPU= 21.352
  06:37:34 STEP 1799 H= 449:45 +CPU= 20.934
  06:38:00 STEP 1800 H= 450:00 +CPU= 26.450
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMGGh7zg+001800
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMSHh7zg+001800
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMUAh7zg+001800
  06:38:21 STEP 1801 H= 450:15 +CPU= 20.817
  06:38:42 STEP 1802 H= 450:30 +CPU= 20.979
  06:39:03 STEP 1803 H= 450:45 +CPU= 21.018
  06:39:25 STEP 1804 H= 451:00 +CPU= 21.693
  06:39:46 STEP 1805 H= 451:15 +CPU= 20.876
  06:40:06 STEP 1806 H= 451:30 +CPU= 20.527
  06:40:26 STEP 1807 H= 451:45 +CPU= 20.113
  06:40:47 STEP 1808 H= 452:00 +CPU= 20.383
  06:41:07 STEP 1809 H= 452:15 +CPU= 19.856
  06:41:27 STEP 1810 H= 452:30 +CPU= 20.530
  06:41:48 STEP 1811 H= 452:45 +CPU= 21.013
  06:42:14 STEP 1812 H= 453:00 +CPU= 26.177
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMGGh7zg+001812
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMSHh7zg+001812
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMUAh7zg+001812
  06:42:35 STEP 1813 H= 453:15 +CPU= 20.933
  06:42:57 STEP 1814 H= 453:30 +CPU= 21.138
  06:43:18 STEP 1815 H= 453:45 +CPU= 21.517
  06:43:40 STEP 1816 H= 454:00 +CPU= 21.851
  06:44:01 STEP 1817 H= 454:15 +CPU= 20.768
  06:44:23 STEP 1818 H= 454:30 +CPU= 20.944
  06:44:44 STEP 1819 H= 454:45 +CPU= 21.224
  06:45:06 STEP 1820 H= 455:00 +CPU= 21.745
  06:45:27 STEP 1821 H= 455:15 +CPU= 21.241
  06:45:48 STEP 1822 H= 455:30 +CPU= 20.953
  06:46:09 STEP 1823 H= 455:45 +CPU= 20.822
  06:46:39 STEP 1824 H= 456:00 +CPU= 30.110
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMGGh7zg+001824
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMSHh7zg+001824
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMUAh7zg+001824
  06:46:59 STEP 1825 H= 456:15 +CPU= 19.879
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMGGh7zg+001740
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMSHh7zg+001740
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMUAh7zg+001740
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMGGh7zg+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMSHh7zg+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMUAh7zg+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMGGh7zg+001764
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMSHh7zg+001764
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMUAh7zg+001764
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMGGh7zg+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMSHh7zg+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMUAh7zg+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMGGh7zg+001788
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMSHh7zg+001788
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMUAh7zg+001788
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMGGh7zg+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMSHh7zg+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMUAh7zg+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMGGh7zg+001812
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMSHh7zg+001812
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMUAh7zg+001812
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMGGh7zg+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMSHh7zg+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMUAh7zg+001824
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a24p_2016092300_20_1018_12291088_0_r354668119_18.zip
Uploading the intermediate file: upload_file_18.zip
  06:47:19 STEP 1826 H= 456:30 +CPU= 19.741
  06:47:39 STEP 1827 H= 456:45 +CPU= 20.678
  06:48:01 STEP 1828 H= 457:00 +CPU= 21.216
  06:48:22 STEP 1829 H= 457:15 +CPU= 21.498
  06:48:43 STEP 1830 H= 457:30 +CPU= 20.819
  06:49:04 STEP 1831 H= 457:45 +CPU= 20.830
  06:49:26 STEP 1832 H= 458:00 +CPU= 21.578
  06:49:47 STEP 1833 H= 458:15 +CPU= 20.859
  06:50:08 STEP 1834 H= 458:30 +CPU= 21.301
  06:50:29 STEP 1835 H= 458:45 +CPU= 21.126
  06:50:56 STEP 1836 H= 459:00 +CPU= 26.405
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMGGh7zg+001836
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMSHh7zg+001836
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMUAh7zg+001836
  06:51:17 STEP 1837 H= 459:15 +CPU= 20.992
  06:51:38 STEP 1838 H= 459:30 +CPU= 21.192
  06:51:59 STEP 1839 H= 459:45 +CPU= 21.106
  06:52:20 STEP 1840 H= 460:00 +CPU= 21.432
  06:52:41 STEP 1841 H= 460:15 +CPU= 20.771
  06:53:02 STEP 1842 H= 460:30 +CPU= 20.881
  06:53:23 STEP 1843 H= 460:45 +CPU= 20.488
  06:53:44 STEP 1844 H= 461:00 +CPU= 21.572
  06:54:05 STEP 1845 H= 461:15 +CPU= 20.977
  06:54:26 STEP 1846 H= 461:30 +CPU= 20.748
  06:54:47 STEP 1847 H= 461:45 +CPU= 20.911
  06:55:13 STEP 1848 H= 462:00 +CPU= 26.526
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMGGh7zg+001848
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMSHh7zg+001848
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMUAh7zg+001848
  06:55:35 STEP 1849 H= 462:15 +CPU= 21.394
  06:55:56 STEP 1850 H= 462:30 +CPU= 20.901
  06:56:17 STEP 1851 H= 462:45 +CPU= 21.017
  06:56:39 STEP 1852 H= 463:00 +CPU= 21.791
  06:57:00 STEP 1853 H= 463:15 +CPU= 20.965
  06:57:20 STEP 1854 H= 463:30 +CPU= 20.745
  06:57:41 STEP 1855 H= 463:45 +CPU= 21.057
  06:58:03 STEP 1856 H= 464:00 +CPU= 21.487
  06:58:24 STEP 1857 H= 464:15 +CPU= 21.199
  06:58:45 STEP 1858 H= 464:30 +CPU= 20.785
  06:59:06 STEP 1859 H= 464:45 +CPU= 20.769
  06:59:32 STEP 1860 H= 465:00 +CPU= 26.697
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMGGh7zg+001860
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMSHh7zg+001860
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMUAh7zg+001860
  06:59:53 STEP 1861 H= 465:15 +CPU= 20.739
  07:00:15 STEP 1862 H= 465:30 +CPU= 21.451
  07:00:37 STEP 1863 H= 465:45 +CPU= 22.431
  07:00:59 STEP 1864 H= 466:00 +CPU= 21.768
  07:01:21 STEP 1865 H= 466:15 +CPU= 21.803
  07:01:42 STEP 1866 H= 466:30 +CPU= 21.138
  07:02:03 STEP 1867 H= 466:45 +CPU= 21.104
  07:02:25 STEP 1868 H= 467:00 +CPU= 21.756
  07:02:46 STEP 1869 H= 467:15 +CPU= 21.235
  07:03:07 STEP 1870 H= 467:30 +CPU= 20.219
  07:03:26 STEP 1871 H= 467:45 +CPU= 19.854
  07:03:55 STEP 1872 H= 468:00 +CPU= 28.580
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMGGh7zg+001872
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMSHh7zg+001872
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMUAh7zg+001872
  07:04:16 STEP 1873 H= 468:15 +CPU= 21.233
  07:04:38 STEP 1874 H= 468:30 +CPU= 21.334
  07:04:59 STEP 1875 H= 468:45 +CPU= 20.894
  07:05:21 STEP 1876 H= 469:00 +CPU= 22.110
  07:05:43 STEP 1877 H= 469:15 +CPU= 21.173
  07:06:04 STEP 1878 H= 469:30 +CPU= 21.273
  07:06:25 STEP 1879 H= 469:45 +CPU= 20.650
  07:06:47 STEP 1880 H= 470:00 +CPU= 21.919
  07:07:08 STEP 1881 H= 470:15 +CPU= 21.080
  07:07:29 STEP 1882 H= 470:30 +CPU= 20.785
  07:07:49 STEP 1883 H= 470:45 +CPU= 20.551
  07:08:15 STEP 1884 H= 471:00 +CPU= 26.112
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMGGh7zg+001884
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMSHh7zg+001884
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMUAh7zg+001884
  07:08:36 STEP 1885 H= 471:15 +CPU= 20.879
  07:08:57 STEP 1886 H= 471:30 +CPU= 21.167
  07:09:18 STEP 1887 H= 471:45 +CPU= 20.667
  07:09:40 STEP 1888 H= 472:00 +CPU= 21.650
  07:10:01 STEP 1889 H= 472:15 +CPU= 21.169
  07:10:23 STEP 1890 H= 472:30 +CPU= 21.847
  07:10:44 STEP 1891 H= 472:45 +CPU= 21.072
  07:11:06 STEP 1892 H= 473:00 +CPU= 21.712
  07:11:26 STEP 1893 H= 473:15 +CPU= 20.562
  07:11:48 STEP 1894 H= 473:30 +CPU= 21.396
  07:12:08 STEP 1895 H= 473:45 +CPU= 20.565
  07:12:35 STEP 1896 H= 474:00 +CPU= 26.478
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMGGh7zg+001896
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMSHh7zg+001896
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMUAh7zg+001896
  07:12:55 STEP 1897 H= 474:15 +CPU= 20.748
  07:13:16 STEP 1898 H= 474:30 +CPU= 20.980
  07:13:38 STEP 1899 H= 474:45 +CPU= 21.141
  07:13:59 STEP 1900 H= 475:00 +CPU= 21.693
  07:14:19 STEP 1901 H= 475:15 +CPU= 19.730
  07:14:39 STEP 1902 H= 475:30 +CPU= 19.893
  07:14:59 STEP 1903 H= 475:45 +CPU= 20.050
  07:15:20 STEP 1904 H= 476:00 +CPU= 20.809
  07:15:40 STEP 1905 H= 476:15 +CPU= 19.898
  07:15:59 STEP 1906 H= 476:30 +CPU= 19.403
  07:16:19 STEP 1907 H= 476:45 +CPU= 19.603
  07:16:44 STEP 1908 H= 477:00 +CPU= 25.000
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMGGh7zg+001908
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMSHh7zg+001908
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMUAh7zg+001908
  07:17:04 STEP 1909 H= 477:15 +CPU= 19.722
  07:17:23 STEP 1910 H= 477:30 +CPU= 19.559
  07:17:43 STEP 1911 H= 477:45 +CPU= 19.846
  07:18:04 STEP 1912 H= 478:00 +CPU= 20.088
  07:18:24 STEP 1913 H= 478:15 +CPU= 20.156
  07:18:44 STEP 1914 H= 478:30 +CPU= 20.043
  07:19:03 STEP 1915 H= 478:45 +CPU= 19.236
  07:19:23 STEP 1916 H= 479:00 +CPU= 19.976
  07:19:43 STEP 1917 H= 479:15 +CPU= 19.722
  07:20:03 STEP 1918 H= 479:30 +CPU= 19.730
  07:20:23 STEP 1919 H= 479:45 +CPU= 20.038
  07:20:50 STEP 1920 H= 480:00 +CPU= 27.590
..The child process terminated with status: 0
>>> Printing last 70 lines from file: NODE.001_01

 
DDH------------------------------------------------------------------------
I                                                                         I
I TIME =  480.000H DDH * Diagnostic Var.,     AREA                GLOBAL  I
I                                                                         I
I-------------------------------------------------------------------------I
I   VARIABLE   I      Initial value         I        Current value        I
I-------------------------------------------------------------------------I
I  OMEGA (P/S) I     -0.6240586652E-21      I      -0.9213592887E-09      I
ICloud fractionI      0.0000000000E+00      I       0.1251526590E-06      I
I  Relat. Hum. I      0.5408036528E+00      I       0.5492488798E+00      I
I  PASS 01     I      0.0000000000E+00      I       0.4248662357E+01      I
DDH------------------------------------------------------------------------
  
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1920 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001920 MODE=a
 IO-STREAM CLOSED - ICMGGh7zg+001920
 MPI-TASK:   1 -    325449028 BYTES IN      7 RECORDS TRANSFERRED IN    0.0008 SECONDS406811.3 Mbytes/s, TOTAL TIME=    0.0084( 9.5%)
 07:20:50 STEP 1920 H= 480:00 +CPU= 27.590
 END CNT3
 NSTEP =  1920 CNT0   000000000  19.415  19.415  19.494   0.996               0               0               0               0               0
IO-STREAM STATISTICS, TOTAL NO OF REC/BYTES READ           46/  1069125708 WRITTEN         5720/************
 -TASK-OPENED-OPEN-RECS IN -KBYTE IN    -RECS OUT-KBYTE OUT-WALL   -WALL IN-WALL
  OU-TOT IN -TOT OUT
   1    103    0       46      1044068     5720 ************    13.0     1.4    11.6     2.2    22.1
===-=== START OF TIMING STATISTICS ===-===
 
STATS FOR ALL TASKS
 NUM ROUTINE                                     CALLS  MEAN(ms)   MAX(ms)   FRAC(%)  UNBAL(%)
   0 CNT0     - COMPLETE EXECUTION                   1 4160062.6 4160062.6    100.00      0.00
   1 CNT4     - FORWARD INTEGRATION                  1 4156363.7 4156363.7     99.91      0.00
   8 SCAN2M - GRID-POINT DYNAMICS                  209    3919.2    3919.2     19.69      0.00
   9 SPCM     - SPECTRAL COMP.                     192     630.4     630.4      2.91      0.00
  10 SCAN2M - PHYSICS                              193    9359.7    9359.7     43.42      0.00
  11 IOPACK   - OUTPUT P.P. RESULTS                 16     902.6     902.6      0.35      0.00
  12 SPNORM   - SPECTRAL NORM COMP.                  9      41.2      41.2      0.01      0.00
  17 GRIDFPOS IN CNT4                               16      24.0      24.0      0.01      0.00
  27 RESTART FILES - WRITING                         4    3605.1    3605.1      0.35      0.00
  28 RESTART FILES - READING                         1    2991.9    2991.9      0.07      0.00
  29 SU4FPOS IN CNT4                                16       0.0       0.0      0.00      0.00
  30 DYNFPOS IN CNT4                                16    4979.8    4979.8      1.92      0.00
  31 POSDDH IN STEPO                                 1      37.9      37.9      0.00      0.00
  37 CPGLAG   - SL COMPUTATIONS                    193    4759.5    4759.5     22.08      0.00
  39 SU0YOMB                                         1     418.4     418.4      0.01      0.00
  51 SCAN2M   - SL COMM. PART 1                    193      19.9      19.9      0.09      0.00
  54 SPCM     - M TO S/S TO M TRANSP.              192     297.7     297.7      1.37      0.00
  55 SPCIMPF  - S TO M/M TO S TRANSP.              192      47.1      47.1      0.22      0.00
  56 SPNORM   - SPECTRAL NORM COMM.                  9       0.2       0.2      0.00      0.00
 102 LTINV_CTL   - INVERSE LEGENDRE TRANSFORM      420     303.0     303.0      3.06      0.00
 103 LTDIR_CTL   - DIRECT LEGENDRE TRANSFORM       400     165.0     165.0      1.59      0.00
 106 FTDIR_CTL   - DIRECT FOURIER TRANSFORM        400      62.9      62.9      0.60      0.00
 107 FTINV_CTL   - INVERSE FOURIER TRANSFORM       420      82.7      82.7      0.83      0.00
 140 SULEG       - COMP. OF LEGENDRE POL.            1      36.9      36.9      0.00      0.00
 152 LTINV_CTL   - M TO L TRANSPOSITION            420      61.8      61.8      0.62      0.00
 153 LTDIR_CTL   - L TO M TRANSPOSITION            400      30.8      30.8      0.30      0.00
 157 FTINV_CTL   - L TO G TRANSPOSITION            420      54.7      54.7      0.55      0.00
 158 FTDIR_CTL   - G TO L TRANSPOSITION            400      30.8      30.8      0.30      0.00
 400 GSTATS                                      23311       0.0       0.0      0.00      0.00
 401 GSTATS HOOK                                 21635       0.0       0.0      0.00      0.00
TOTAL MEASURED IMBALANCE =       0.0 SECONDS,  0.0 PERCENT
TOTAL WALLCLOCK TIME     4160.1 CPU TIME    4148.6 VECTOR TIME     4148.6


===-=== END   OF TIMING STATISTICS ===-===
FORECAST DAYS PER DAY      415.7
  *** END CNT0 *** 
------------------------------------------------
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMGGh7zg+001920
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMSHh7zg+001920
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMUAh7zg+001920
Adding to the zip: /var/lib/boinc-client/slots/17/NODE.001_01
Adding to the zip: /var/lib/boinc-client/slots/17/ifs.stat
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMSHh7zg+000768
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMSHh7zg+001836
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMUAh7zg+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMSHh7zg+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMGGh7zg+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMUAh7zg+001836
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMUAh7zg+000768
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMGGh7zg+001908
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMGGh7zg+001884
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMUAh7zg+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMSHh7zg+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMGGh7zg+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMUAh7zg+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMUAh7zg+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMUAh7zg+001860
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMUAh7zg+001884
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMGGh7zg+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMSHh7zg+001860
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMGGh7zg+001836
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMSHh7zg+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMGGh7zg+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMSHh7zg+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMGGh7zg+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMSHh7zg+001908
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMSHh7zg+001884
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMGGh7zg+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMUAh7zg+001908
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMGGh7zg+001860
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMGGh7zg+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMUAh7zg+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMSHh7zg+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMUAh7zg+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMSHh7zg+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291088/ICMGGh7zg+000768
Zipping up the final file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a24p_2016092300_20_1018_12291088_0_r354668119_19.zip
Uploading the final file: upload_file_19.zip
Uploading trickle at timestep: 1727100
07:24:23 (27408): called boinc_finish(0)

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
18 Jun 2024 05:51:34 1469580 22444686 oifs_43r3_bl_a24p_2016092300_20_1018_12291088_0 1,727,100 45,835 0.0265


©2024 climateprediction.net