climateprediction.net home page
Task 22438014

Task 22438014

Name oifs_43r3_bl_a3iz_2016092300_20_1017_12286398_0
Workunit 12286398
Created 7 Jun 2024, 17:27:18 UTC
Sent 7 Jun 2024, 17:40:49 UTC
Report deadline 6 Aug 2024, 17:40:49 UTC
Received 10 Jun 2024, 22:11:26 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 0 (0x00000000)
Computer ID 1533094
Run time 21 hours 21 min 20 sec
CPU time 20 hours 50 min 14 sec
Validate state Invalid
Credit 1,976.32
Device peak FLOPS 7.75 GFLOPS
Application version OpenIFS 43r3 Baroclinic Lifecycle v1.13
x86_64-pc-linux-gnu
Peak working set size 5,537.69 MB
Peak swap size 5,980.81 MB
Peak disk usage 1,408.54 MB
Stderr
<core_client_version>7.24.1</core_client_version>
<![CDATA[
<stderr_txt>
] [20240608:110953:1717837793:49702.790] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:antares:1:1:28569:28569] [20240608:110953:1717837793:49702.790] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :       CNT4 
[EC_DRHOOK:antares:1:1:28569:28569] [20240608:110953:1717837793:49702.790] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :        STEPO 
[EC_DRHOOK:antares:1:1:28569:28569] [20240608:110953:1717837793:49702.790] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :         SCAN2M 
[EC_DRHOOK:antares:1:1:28569:28569] [20240608:110953:1717837793:49702.790] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :          GP_MODEL_HEAP 
[EC_DRHOOK:antares:1:1:28569:28569] [20240608:110953:1717837793:49702.790] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :           GP_MODEL 
[EC_DRHOOK:antares:1:1:28569:28569] [20240608:110953:1717837793:49702.790] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :            EC_PHYS_DRV 
[EC_DRHOOK:antares:1:1:28569:28569] [20240608:110953:1717837793:49702.790] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :             >OMP-PHYSICS              (1001) 
[EC_DRHOOK:antares:1:1:28569:28569] [20240608:110953:1717837793:49702.790] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :              EC_PHYS 
[EC_DRHOOK:antares:1:1:28569:28569] [20240608:110953:1717837793:49702.790] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :               CALLPAR 
[EC_DRHOOK:antares:1:1:28569:28569] [20240608:110953:1717837793:49702.790] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :                TURBULENCE_LAYER 
[EC_DRHOOK:antares:1:1:28569:28569] [20240608:110953:1717837793:49702.790] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :                 VDFOUTER 
[EC_DRHOOK:antares:1:1:28569:28569] [20240608:110953:1717837793:49702.790] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :                  VDFMAIN 
[EC_DRHOOK:antares:1:1:28569:28569] [20240608:110953:1717837793:49702.790] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :                   VDFDIFM 
[EC_DRHOOK:antares:1:1:28569:28569] [20240608:110954:1717837794:49703.790] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1
[EC_DRHOOK:antares:1:1:28569:28569] [20240608:110954:1717837794:49703.790] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1cf8da0 for signal#15, nsigs = 1
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source             
oifs_43r3_model.e  0000000001CF935B  Unknown               Unknown  Unknown
oifs_43r3_model.e  00000000008214C1  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DD5490  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000124169F  vdfdifm_                  216  vdfdifm.F90
oifs_43r3_model.e  0000000001223411  vdfmain_                 1162  vdfmain.F90
oifs_43r3_model.e  0000000001207E29  vdfouter_                 739  vdfouter.F90
oifs_43r3_model.e  00000000011FDCB2  turbulence_layer_         158  turbulence_layer.F90
oifs_43r3_model.e  0000000000FAC003  callpar_                  869  callpar.F90
oifs_43r3_model.e  0000000000F5FA6B  ec_phys_                  670  ec_phys.F90
oifs_43r3_model.e  0000000000F4DBC4  ec_phys_drv_              599  ec_phys_drv.F90
oifs_43r3_model.e  0000000000BD7131  gp_model_                 613  gp_model.F90
oifs_43r3_model.e  00000000012C5002  gp_model_heap_             74  gp_model_heap.F90
oifs_43r3_model.e  0000000000BACBAE  scan2m_                   535  scan2m.F90
oifs_43r3_model.e  0000000000688153  stepo_                    327  stepo.F90
oifs_43r3_model.e  00000000004CD1DC  cnt4_                    1133  cnt4.F90
oifs_43r3_model.e  00000000004B5D85  cnt3_                     267  cnt3.F90
oifs_43r3_model.e  00000000004B4F52  cnt2_                      88  cnt2.F90
oifs_43r3_model.e  000000000049D638  cnt1_                      92  cnt1.F90
oifs_43r3_model.e  0000000000406FBD  cnt0_                     146  cnt0.F90
oifs_43r3_model.e  0000000000401E7F  MAIN__                     96  master.F90
oifs_43r3_model.e  0000000000401E12  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCC34A  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCDBE7  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000000401CD5  Unknown               Unknown  Unknown
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.13_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a3iz
(argv4) batchid: 1017
(argv5) wuid: 12286398
(argv6) fclen: 20
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/4
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.13
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/4/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/4/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_3993e865559f9250ca165b21431df8a5 to: /var/lib/boinc/slots/4/oifs_43r3_bl_a3iz_2016092300_20_1017_12286398.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/4/oifs_43r3_bl_a3iz_2016092300_20_1017_12286398.zip
ic_ancil_file: ic_ancil_12286398
ifsdata_file: ifsdata_12286398
climate_data_file: clim_data_12286398
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /var/lib/boinc/slots/4/ic_ancil_12286398.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/4/ic_ancil_12286398.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_d9abffc187b3391dc656d8c761e64fd6 to: /var/lib/boinc/slots/4/ifsdata/ifsdata_12286398.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/4/ifsdata/ifsdata_12286398.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/4/159l_2/clim_data_12286398.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/4/159l_2/clim_data_12286398.zip
Checking for progress XML file: /var/lib/boinc/slots/4/progress_file_12286398.xml
Opened progress file ok : /var/lib/boinc/slots/4/progress_file_12286398.xml
-- Model is restarting --
Adjusting last_iter, 980, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/4/progress_file_12286398.xml
last_cpu_time: 52165
upload_file_number: 10
last_iter: 961
last_upload: 864000
model_completed: 0
total_length_of_simulation: 1728000
result_base_name: oifs_43r3_bl_a3iz_2016092300_20_1017_12286398_0_r1164538962
The child process has been launched with process id: 8055
Executing the command: /var/lib/boinc/slots/4/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:antares:1:1:8055:8055] [20240610:162012:1718029212:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2091] fp = 0x314f440
[EC_DRHOOK:antares:1:1:8055:8055] [20240610:162012:1718029212:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:antares:1:1:8055:8055] [20240610:162012:1718029212:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:antares:1:1:8055:8055] [20240610:162012:1718029212:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:antares:1:1:8055:8055] [20240610:162012:1718029212:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:antares:1:1:8055:8055] [20240610:162012:1718029212:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:antares:1:1:8055:8055] [20240610:162012:1718029212:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:antares:1:1:8055:8055] [20240610:162012:1718029212:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:antares:1:1:8055:8055] [20240610:162012:1718029212:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:antares:1:1:8055:8055] [20240610:162012:1718029212:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:antares:1:1:8055:8055] [20240610:162012:1718029212:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:antares:1:1:8055:8055] [20240610:162012:1718029212:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:antares:1:1:8055:8055] [20240610:162012:1718029212:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:antares:1:1:8055:8055] [20240610:162012:1718029212:0.000] [ignore_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:antares:1:1:8055:8055] [20240610:162012:1718029212:0.000] [restore_default_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:antares:1:1:8055:8055] [20240610:162012:1718029212:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:antares:1:1:8055:8055] [20240610:162012:1718029212:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:antares:1:1:8055:8055] [20240610:162012:1718029212:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:antares:1:1:8055:8055] [20240610:162012:1718029212:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x820cf0 (old at (nil))
[EC_DRHOOK:antares:1:1:8055:8055] [20240610:162012:1718029212:0.001] [trapfpe_treatment@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:antares:1:1:8055:8055] [20240610:162012:1718029212:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:antares:1:1:8055:8055] [20240610:162012:1718029212:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:antares:1:1:8055:8055] [20240610:162012:1718029212:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x820cf0 (old at (nil))
[EC_DRHOOK:antares:1:1:8055:8055] [20240610:162012:1718029212:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:antares:1:1:8055:8055] [20240610:162012:1718029212:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:antares:1:1:8055:8055] [20240610:162012:1718029212:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:antares:1:1:8055:8055] [20240610:162012:1718029212:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x820cf0 (old at (nil))
[EC_DRHOOK:antares:1:1:8055:8055] [20240610:162012:1718029212:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x820cf0 (old at (nil))
[EC_DRHOOK:antares:1:1:8055:8055] [20240610:162012:1718029212:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:antares:1:1:8055:8055] [20240610:162012:1718029212:0.001] [catch_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  16:20:54 STEP  960 H= 240:00 +CPU= 33.966
  16:21:25 STEP  961 H= 240:15 +CPU= 30.882
  16:21:52 STEP  962 H= 240:30 +CPU= 26.574
  16:22:20 STEP  963 H= 240:45 +CPU= 27.692
  16:22:58 STEP  964 H= 241:00 +CPU= 36.218
  16:23:55 STEP  965 H= 241:15 +CPU= 51.697
  16:24:49 STEP  966 H= 241:30 +CPU= 50.443
  16:25:47 STEP  967 H= 241:45 +CPU= 52.841
  16:26:47 STEP  968 H= 242:00 +CPU= 53.475
  16:27:50 STEP  969 H= 242:15 +CPU= 52.615
  16:28:50 STEP  970 H= 242:30 +CPU= 53.982
  16:29:48 STEP  971 H= 242:45 +CPU= 51.577
  16:30:57 STEP  972 H= 243:00 +CPU= 62.058
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+000972
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+000972
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+000972
  16:31:54 STEP  973 H= 243:15 +CPU= 51.028
  16:32:54 STEP  974 H= 243:30 +CPU= 53.099
  16:33:52 STEP  975 H= 243:45 +CPU= 51.929
  16:34:50 STEP  976 H= 244:00 +CPU= 50.931
  16:35:52 STEP  977 H= 244:15 +CPU= 53.156
  16:36:52 STEP  978 H= 244:30 +CPU= 53.486
  16:37:53 STEP  979 H= 244:45 +CPU= 53.200
  16:38:52 STEP  980 H= 245:00 +CPU= 53.023
  16:39:54 STEP  981 H= 245:15 +CPU= 55.871
  16:40:55 STEP  982 H= 245:30 +CPU= 54.880
  16:41:57 STEP  983 H= 245:45 +CPU= 52.018
  16:43:10 STEP  984 H= 246:00 +CPU= 63.370
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+000984
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+000984
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+000984
  16:44:08 STEP  985 H= 246:15 +CPU= 52.442
  16:45:03 STEP  986 H= 246:30 +CPU= 50.452
  16:46:02 STEP  987 H= 246:45 +CPU= 53.129
  16:47:00 STEP  988 H= 247:00 +CPU= 51.778
  16:48:00 STEP  989 H= 247:15 +CPU= 52.930
  16:48:59 STEP  990 H= 247:30 +CPU= 53.404
  16:50:00 STEP  991 H= 247:45 +CPU= 54.458
  16:50:58 STEP  992 H= 248:00 +CPU= 52.640
  16:51:55 STEP  993 H= 248:15 +CPU= 51.354
  16:52:55 STEP  994 H= 248:30 +CPU= 54.487
  16:53:57 STEP  995 H= 248:45 +CPU= 54.932
  16:55:11 STEP  996 H= 249:00 +CPU= 65.787
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+000996
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+000996
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+000996
  16:56:07 STEP  997 H= 249:15 +CPU= 52.374
  16:57:09 STEP  998 H= 249:30 +CPU= 55.633
  16:58:11 STEP  999 H= 249:45 +CPU= 54.811
  16:59:14 STEP 1000 H= 250:00 +CPU= 57.397
  17:00:13 STEP 1001 H= 250:15 +CPU= 53.114
  17:01:13 STEP 1002 H= 250:30 +CPU= 54.225
  17:02:13 STEP 1003 H= 250:45 +CPU= 54.751
  17:03:13 STEP 1004 H= 251:00 +CPU= 54.485
  17:04:09 STEP 1005 H= 251:15 +CPU= 52.684
  17:05:05 STEP 1006 H= 251:30 +CPU= 54.737
  17:06:01 STEP 1007 H= 251:45 +CPU= 54.110
  17:07:13 STEP 1008 H= 252:00 +CPU= 68.260
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+001008
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+001008
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+001008
  17:08:12 STEP 1009 H= 252:15 +CPU= 54.856
  17:09:07 STEP 1010 H= 252:30 +CPU= 53.144
  17:10:02 STEP 1011 H= 252:45 +CPU= 53.233
  17:10:58 STEP 1012 H= 253:00 +CPU= 53.934
  17:11:52 STEP 1013 H= 253:15 +CPU= 51.202
  17:12:49 STEP 1014 H= 253:30 +CPU= 54.274
  17:13:49 STEP 1015 H= 253:45 +CPU= 56.816
  17:14:44 STEP 1016 H= 254:00 +CPU= 53.431
  17:15:44 STEP 1017 H= 254:15 +CPU= 57.185
  17:16:44 STEP 1018 H= 254:30 +CPU= 57.407
  17:17:45 STEP 1019 H= 254:45 +CPU= 57.707
  17:18:56 STEP 1020 H= 255:00 +CPU= 68.315
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+001020
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+001020
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+001020
  17:19:52 STEP 1021 H= 255:15 +CPU= 53.725
  17:20:48 STEP 1022 H= 255:30 +CPU= 53.107
  17:21:46 STEP 1023 H= 255:45 +CPU= 54.978
  17:22:46 STEP 1024 H= 256:00 +CPU= 58.115
  17:23:52 STEP 1025 H= 256:15 +CPU= 62.553
  17:24:53 STEP 1026 H= 256:30 +CPU= 58.207
  17:25:59 STEP 1027 H= 256:45 +CPU= 63.375
  17:27:02 STEP 1028 H= 257:00 +CPU= 59.743
  17:27:59 STEP 1029 H= 257:15 +CPU= 55.112
  17:29:02 STEP 1030 H= 257:30 +CPU= 60.649
  17:29:59 STEP 1031 H= 257:45 +CPU= 54.793
  17:31:16 STEP 1032 H= 258:00 +CPU= 73.963
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+001032
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+001032
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+001032
  17:32:16 STEP 1033 H= 258:15 +CPU= 58.018
  17:33:22 STEP 1034 H= 258:30 +CPU= 62.135
  17:34:15 STEP 1035 H= 258:45 +CPU= 51.013
  17:35:13 STEP 1036 H= 259:00 +CPU= 55.172
  17:36:11 STEP 1037 H= 259:15 +CPU= 56.130
  17:37:14 STEP 1038 H= 259:30 +CPU= 60.160
  17:38:18 STEP 1039 H= 259:45 +CPU= 60.520
  17:39:20 STEP 1040 H= 260:00 +CPU= 60.256
  17:40:20 STEP 1041 H= 260:15 +CPU= 58.443
  17:41:23 STEP 1042 H= 260:30 +CPU= 59.969
  17:42:23 STEP 1043 H= 260:45 +CPU= 57.242
  17:43:30 STEP 1044 H= 261:00 +CPU= 64.420
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+001044
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+001044
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+001044
  17:44:32 STEP 1045 H= 261:15 +CPU= 59.504
  17:45:38 STEP 1046 H= 261:30 +CPU= 62.848
  17:46:37 STEP 1047 H= 261:45 +CPU= 57.375
  17:47:30 STEP 1048 H= 262:00 +CPU= 51.831
  17:48:35 STEP 1049 H= 262:15 +CPU= 61.866
  17:49:35 STEP 1050 H= 262:30 +CPU= 58.254
  17:50:33 STEP 1051 H= 262:45 +CPU= 54.799
  17:51:31 STEP 1052 H= 263:00 +CPU= 55.920
  17:52:32 STEP 1053 H= 263:15 +CPU= 59.100
  17:53:37 STEP 1054 H= 263:30 +CPU= 62.587
  17:54:42 STEP 1055 H= 263:45 +CPU= 59.692
  17:56:02 STEP 1056 H= 264:00 +CPU= 75.413
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+001056
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+001056
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+001056
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMGGh7zg+000960
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMSHh7zg+000960
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMUAh7zg+000960
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMGGh7zg+000972
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMSHh7zg+000972
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMUAh7zg+000972
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMGGh7zg+000984
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMSHh7zg+000984
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMUAh7zg+000984
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMGGh7zg+000996
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMSHh7zg+000996
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMUAh7zg+000996
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMGGh7zg+001008
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMSHh7zg+001008
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMUAh7zg+001008
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMGGh7zg+001020
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMSHh7zg+001020
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMUAh7zg+001020
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMGGh7zg+001032
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMSHh7zg+001032
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMUAh7zg+001032
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMGGh7zg+001044
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMSHh7zg+001044
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMUAh7zg+001044
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_a3iz_2016092300_20_1017_12286398_0_r1164538962_10.zip
Uploading the intermediate file: upload_file_10.zip
  17:57:05 STEP 1057 H= 264:15 +CPU= 60.361
  17:58:06 STEP 1058 H= 264:30 +CPU= 58.424
  17:59:05 STEP 1059 H= 264:45 +CPU= 55.646
  18:00:11 STEP 1060 H= 265:00 +CPU= 62.451
  18:01:09 STEP 1061 H= 265:15 +CPU= 55.476
  18:02:08 STEP 1062 H= 265:30 +CPU= 56.755
  18:03:11 STEP 1063 H= 265:45 +CPU= 60.828
  18:04:17 STEP 1064 H= 266:00 +CPU= 62.549
  18:05:15 STEP 1065 H= 266:15 +CPU= 56.373
  18:06:15 STEP 1066 H= 266:30 +CPU= 57.398
  18:07:17 STEP 1067 H= 266:45 +CPU= 59.937
  18:08:30 STEP 1068 H= 267:00 +CPU= 69.664
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+001068
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+001068
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+001068
  18:09:27 STEP 1069 H= 267:15 +CPU= 55.726
  18:10:27 STEP 1070 H= 267:30 +CPU= 57.421
  18:11:23 STEP 1071 H= 267:45 +CPU= 54.716
  18:12:24 STEP 1072 H= 268:00 +CPU= 58.268
  18:13:24 STEP 1073 H= 268:15 +CPU= 58.515
  18:14:29 STEP 1074 H= 268:30 +CPU= 61.205
  18:15:29 STEP 1075 H= 268:45 +CPU= 58.108
  18:16:28 STEP 1076 H= 269:00 +CPU= 57.222
  18:17:32 STEP 1077 H= 269:15 +CPU= 60.543
  18:18:32 STEP 1078 H= 269:30 +CPU= 58.808
  18:19:32 STEP 1079 H= 269:45 +CPU= 58.331
  18:20:42 STEP 1080 H= 270:00 +CPU= 67.261
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+001080
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+001080
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+001080
  18:21:43 STEP 1081 H= 270:15 +CPU= 57.232
  18:22:41 STEP 1082 H= 270:30 +CPU= 55.709
  18:23:40 STEP 1083 H= 270:45 +CPU= 57.895
  18:24:37 STEP 1084 H= 271:00 +CPU= 55.504
  18:25:37 STEP 1085 H= 271:15 +CPU= 57.804
  18:26:37 STEP 1086 H= 271:30 +CPU= 59.493
  18:27:33 STEP 1087 H= 271:45 +CPU= 54.764
  18:28:36 STEP 1088 H= 272:00 +CPU= 60.519
  18:29:35 STEP 1089 H= 272:15 +CPU= 57.806
  18:30:33 STEP 1090 H= 272:30 +CPU= 56.792
  18:31:31 STEP 1091 H= 272:45 +CPU= 56.629
  18:32:36 STEP 1092 H= 273:00 +CPU= 63.395
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+001092
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+001092
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+001092
  18:33:37 STEP 1093 H= 273:15 +CPU= 59.392
  18:34:40 STEP 1094 H= 273:30 +CPU= 60.086
  18:35:40 STEP 1095 H= 273:45 +CPU= 57.634
  18:36:41 STEP 1096 H= 274:00 +CPU= 58.496
  18:37:35 STEP 1097 H= 274:15 +CPU= 52.779
  18:38:38 STEP 1098 H= 274:30 +CPU= 59.636
  18:39:39 STEP 1099 H= 274:45 +CPU= 58.656
  18:40:37 STEP 1100 H= 275:00 +CPU= 55.804
  18:41:34 STEP 1101 H= 275:15 +CPU= 55.655
  18:42:32 STEP 1102 H= 275:30 +CPU= 54.782
  18:43:30 STEP 1103 H= 275:45 +CPU= 56.453
  18:44:55 STEP 1104 H= 276:00 +CPU= 80.552
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+001104
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+001104
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+001104
  18:45:56 STEP 1105 H= 276:15 +CPU= 58.475
  18:46:58 STEP 1106 H= 276:30 +CPU= 59.099
  18:47:57 STEP 1107 H= 276:45 +CPU= 56.704
  18:48:59 STEP 1108 H= 277:00 +CPU= 59.493
  18:50:01 STEP 1109 H= 277:15 +CPU= 57.603
  18:51:05 STEP 1110 H= 277:30 +CPU= 60.026
  18:52:08 STEP 1111 H= 277:45 +CPU= 60.721
  18:53:13 STEP 1112 H= 278:00 +CPU= 61.579
  18:54:16 STEP 1113 H= 278:15 +CPU= 59.427
  18:55:13 STEP 1114 H= 278:30 +CPU= 54.935
  18:56:13 STEP 1115 H= 278:45 +CPU= 58.502
  18:57:31 STEP 1116 H= 279:00 +CPU= 69.436
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+001116
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+001116
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+001116
  18:58:36 STEP 1117 H= 279:15 +CPU= 58.541
  18:59:37 STEP 1118 H= 279:30 +CPU= 57.845
  19:00:38 STEP 1119 H= 279:45 +CPU= 58.397
  19:01:42 STEP 1120 H= 280:00 +CPU= 60.170
  19:02:42 STEP 1121 H= 280:15 +CPU= 56.795
  19:03:46 STEP 1122 H= 280:30 +CPU= 60.214
  19:04:47 STEP 1123 H= 280:45 +CPU= 58.102
  19:05:44 STEP 1124 H= 281:00 +CPU= 55.866
  19:06:40 STEP 1125 H= 281:15 +CPU= 55.045
  19:07:40 STEP 1126 H= 281:30 +CPU= 58.568
  19:08:39 STEP 1127 H= 281:45 +CPU= 57.614
  19:09:52 STEP 1128 H= 282:00 +CPU= 70.458
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+001128
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+001128
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+001128
  19:10:54 STEP 1129 H= 282:15 +CPU= 58.959
  19:11:51 STEP 1130 H= 282:30 +CPU= 55.889
  19:12:47 STEP 1131 H= 282:45 +CPU= 54.141
  19:13:49 STEP 1132 H= 283:00 +CPU= 61.321
  19:14:43 STEP 1133 H= 283:15 +CPU= 53.025
  19:15:42 STEP 1134 H= 283:30 +CPU= 57.428
  19:16:40 STEP 1135 H= 283:45 +CPU= 56.751
  19:17:38 STEP 1136 H= 284:00 +CPU= 56.330
  19:18:42 STEP 1137 H= 284:15 +CPU= 60.564
  19:19:46 STEP 1138 H= 284:30 +CPU= 61.595
  19:20:48 STEP 1139 H= 284:45 +CPU= 58.499
  19:22:13 STEP 1140 H= 285:00 +CPU= 70.301
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+001140
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+001140
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+001140
  19:23:22 STEP 1141 H= 285:15 +CPU= 54.339
  19:24:23 STEP 1142 H= 285:30 +CPU= 54.721
  19:25:21 STEP 1143 H= 285:45 +CPU= 54.208
  19:26:22 STEP 1144 H= 286:00 +CPU= 58.726
  19:27:23 STEP 1145 H= 286:15 +CPU= 59.558
  19:28:22 STEP 1146 H= 286:30 +CPU= 56.814
  19:29:24 STEP 1147 H= 286:45 +CPU= 59.364
  19:30:29 STEP 1148 H= 287:00 +CPU= 62.126
  19:31:24 STEP 1149 H= 287:15 +CPU= 53.599
  19:32:22 STEP 1150 H= 287:30 +CPU= 55.154
  19:33:27 STEP 1151 H= 287:45 +CPU= 61.300
  19:34:48 STEP 1152 H= 288:00 +CPU= 77.858
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+001152
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+001152
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+001152
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMGGh7zg+001056
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMSHh7zg+001056
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMUAh7zg+001056
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMGGh7zg+001068
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMSHh7zg+001068
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMUAh7zg+001068
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMGGh7zg+001080
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMSHh7zg+001080
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMUAh7zg+001080
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMGGh7zg+001092
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMSHh7zg+001092
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMUAh7zg+001092
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMGGh7zg+001104
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMSHh7zg+001104
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMUAh7zg+001104
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMGGh7zg+001116
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMSHh7zg+001116
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMUAh7zg+001116
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMGGh7zg+001128
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMSHh7zg+001128
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMUAh7zg+001128
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMGGh7zg+001140
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMSHh7zg+001140
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMUAh7zg+001140
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_a3iz_2016092300_20_1017_12286398_0_r1164538962_11.zip
Uploading the intermediate file: upload_file_11.zip
  19:35:47 STEP 1153 H= 288:15 +CPU= 57.718
  19:36:42 STEP 1154 H= 288:30 +CPU= 54.263
  19:37:37 STEP 1155 H= 288:45 +CPU= 53.644
  19:38:33 STEP 1156 H= 289:00 +CPU= 55.729
  19:39:28 STEP 1157 H= 289:15 +CPU= 53.413
  19:40:21 STEP 1158 H= 289:30 +CPU= 52.917
  19:41:11 STEP 1159 H= 289:45 +CPU= 49.512
  19:42:06 STEP 1160 H= 290:00 +CPU= 52.967
  19:43:00 STEP 1161 H= 290:15 +CPU= 50.697
  19:43:53 STEP 1162 H= 290:30 +CPU= 50.785
  19:44:41 STEP 1163 H= 290:45 +CPU= 47.310
  19:45:49 STEP 1164 H= 291:00 +CPU= 65.550
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+001164
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+001164
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+001164
  19:46:42 STEP 1165 H= 291:15 +CPU= 50.356
  19:47:35 STEP 1166 H= 291:30 +CPU= 51.677
  19:48:31 STEP 1167 H= 291:45 +CPU= 53.649
  19:49:30 STEP 1168 H= 292:00 +CPU= 56.100
  19:50:29 STEP 1169 H= 292:15 +CPU= 56.913
  19:51:18 STEP 1170 H= 292:30 +CPU= 48.119
  19:52:11 STEP 1171 H= 292:45 +CPU= 50.382
  19:53:07 STEP 1172 H= 293:00 +CPU= 54.074
  19:53:59 STEP 1173 H= 293:15 +CPU= 49.344
  19:54:55 STEP 1174 H= 293:30 +CPU= 53.982
  19:55:45 STEP 1175 H= 293:45 +CPU= 48.848
  19:56:48 STEP 1176 H= 294:00 +CPU= 61.192
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+001176
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+001176
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+001176
  19:57:42 STEP 1177 H= 294:15 +CPU= 51.026
  19:58:34 STEP 1178 H= 294:30 +CPU= 50.410
  19:59:22 STEP 1179 H= 294:45 +CPU= 45.970
  20:00:12 STEP 1180 H= 295:00 +CPU= 48.360
  20:01:05 STEP 1181 H= 295:15 +CPU= 51.039
  20:01:59 STEP 1182 H= 295:30 +CPU= 51.569
  20:02:48 STEP 1183 H= 295:45 +CPU= 47.952
  20:03:40 STEP 1184 H= 296:00 +CPU= 49.804
  20:04:26 STEP 1185 H= 296:15 +CPU= 45.072
  20:05:16 STEP 1186 H= 296:30 +CPU= 48.578
  20:06:07 STEP 1187 H= 296:45 +CPU= 49.084
  20:07:01 STEP 1188 H= 297:00 +CPU= 53.439
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+001188
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+001188
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+001188
  20:07:54 STEP 1189 H= 297:15 +CPU= 50.520
  20:08:52 STEP 1190 H= 297:30 +CPU= 55.467
  20:09:35 STEP 1191 H= 297:45 +CPU= 42.049
  20:10:30 STEP 1192 H= 298:00 +CPU= 52.393
  20:11:24 STEP 1193 H= 298:15 +CPU= 51.554
  20:12:16 STEP 1194 H= 298:30 +CPU= 50.972
  20:13:10 STEP 1195 H= 298:45 +CPU= 51.159
  20:14:03 STEP 1196 H= 299:00 +CPU= 49.533
  20:14:54 STEP 1197 H= 299:15 +CPU= 49.643
  20:15:43 STEP 1198 H= 299:30 +CPU= 48.463
  20:16:34 STEP 1199 H= 299:45 +CPU= 50.086
  20:17:46 STEP 1200 H= 300:00 +CPU= 70.203
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+001200
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+001200
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+001200
  20:18:36 STEP 1201 H= 300:15 +CPU= 48.238
  20:19:31 STEP 1202 H= 300:30 +CPU= 52.151
  20:20:25 STEP 1203 H= 300:45 +CPU= 52.640
  20:21:18 STEP 1204 H= 301:00 +CPU= 50.032
  20:22:06 STEP 1205 H= 301:15 +CPU= 47.198
  20:22:53 STEP 1206 H= 301:30 +CPU= 45.139
  20:23:39 STEP 1207 H= 301:45 +CPU= 44.590
  20:24:32 STEP 1208 H= 302:00 +CPU= 50.745
  20:25:16 STEP 1209 H= 302:15 +CPU= 43.178
  20:26:04 STEP 1210 H= 302:30 +CPU= 46.576
  20:26:51 STEP 1211 H= 302:45 +CPU= 44.834
  20:27:52 STEP 1212 H= 303:00 +CPU= 58.648
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+001212
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+001212
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+001212
  20:28:43 STEP 1213 H= 303:15 +CPU= 48.911
  20:29:32 STEP 1214 H= 303:30 +CPU= 47.093
  20:30:21 STEP 1215 H= 303:45 +CPU= 48.069
  20:31:08 STEP 1216 H= 304:00 +CPU= 45.538
  20:31:53 STEP 1217 H= 304:15 +CPU= 43.741
  20:32:44 STEP 1218 H= 304:30 +CPU= 48.433
  20:33:30 STEP 1219 H= 304:45 +CPU= 45.489
  20:34:19 STEP 1220 H= 305:00 +CPU= 47.322
  20:35:10 STEP 1221 H= 305:15 +CPU= 48.829
  20:35:58 STEP 1222 H= 305:30 +CPU= 47.065
  20:36:47 STEP 1223 H= 305:45 +CPU= 46.431
  20:37:41 STEP 1224 H= 306:00 +CPU= 53.368
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+001224
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+001224
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+001224
  20:38:30 STEP 1225 H= 306:15 +CPU= 48.114
  20:39:17 STEP 1226 H= 306:30 +CPU= 45.374
  20:40:04 STEP 1227 H= 306:45 +CPU= 45.863
  20:40:53 STEP 1228 H= 307:00 +CPU= 47.568
  20:41:39 STEP 1229 H= 307:15 +CPU= 44.660
  20:42:28 STEP 1230 H= 307:30 +CPU= 46.752
  20:43:15 STEP 1231 H= 307:45 +CPU= 45.775
  20:44:01 STEP 1232 H= 308:00 +CPU= 45.487
  20:44:46 STEP 1233 H= 308:15 +CPU= 44.125
  20:45:39 STEP 1234 H= 308:30 +CPU= 50.720
  20:46:28 STEP 1235 H= 308:45 +CPU= 47.642
  20:47:28 STEP 1236 H= 309:00 +CPU= 59.407
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+001236
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+001236
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+001236
  20:48:21 STEP 1237 H= 309:15 +CPU= 50.882
  20:49:07 STEP 1238 H= 309:30 +CPU= 45.428
  20:49:59 STEP 1239 H= 309:45 +CPU= 50.086
  20:50:50 STEP 1240 H= 310:00 +CPU= 49.358
  20:51:46 STEP 1241 H= 310:15 +CPU= 55.089
  20:52:23 STEP 1242 H= 310:30 +CPU= 37.062
  20:53:06 STEP 1243 H= 310:45 +CPU= 42.891
  20:53:50 STEP 1244 H= 311:00 +CPU= 43.247
  20:54:38 STEP 1245 H= 311:15 +CPU= 47.391
  20:55:21 STEP 1246 H= 311:30 +CPU= 40.937
  20:56:10 STEP 1247 H= 311:45 +CPU= 47.522
  20:57:10 STEP 1248 H= 312:00 +CPU= 58.613
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+001248
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+001248
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+001248
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMGGh7zg+001152
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMSHh7zg+001152
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMUAh7zg+001152
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMGGh7zg+001164
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMSHh7zg+001164
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMUAh7zg+001164
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMGGh7zg+001176
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMSHh7zg+001176
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMUAh7zg+001176
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMGGh7zg+001188
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMSHh7zg+001188
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMUAh7zg+001188
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMGGh7zg+001200
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMSHh7zg+001200
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMUAh7zg+001200
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMGGh7zg+001212
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMSHh7zg+001212
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMUAh7zg+001212
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMGGh7zg+001224
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMSHh7zg+001224
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMUAh7zg+001224
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMGGh7zg+001236
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMSHh7zg+001236
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMUAh7zg+001236
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_a3iz_2016092300_20_1017_12286398_0_r1164538962_12.zip
Uploading the intermediate file: upload_file_12.zip
  20:57:56 STEP 1249 H= 312:15 +CPU= 45.275
  20:58:40 STEP 1250 H= 312:30 +CPU= 42.287
  20:59:24 STEP 1251 H= 312:45 +CPU= 43.321
  21:00:12 STEP 1252 H= 313:00 +CPU= 46.549
  21:00:57 STEP 1253 H= 313:15 +CPU= 43.600
  21:01:40 STEP 1254 H= 313:30 +CPU= 42.678
  21:02:18 STEP 1255 H= 313:45 +CPU= 37.703
  21:02:59 STEP 1256 H= 314:00 +CPU= 39.966
  21:03:46 STEP 1257 H= 314:15 +CPU= 46.380
  21:04:33 STEP 1258 H= 314:30 +CPU= 45.882
  21:05:16 STEP 1259 H= 314:45 +CPU= 42.281
  21:06:06 STEP 1260 H= 315:00 +CPU= 48.817
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+001260
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+001260
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+001260
  21:06:35 STEP 1261 H= 315:15 +CPU= 28.316
  21:07:14 STEP 1262 H= 315:30 +CPU= 38.051
  21:07:58 STEP 1263 H= 315:45 +CPU= 43.466
  21:08:46 STEP 1264 H= 316:00 +CPU= 47.281
  21:09:32 STEP 1265 H= 316:15 +CPU= 46.037
  21:10:15 STEP 1266 H= 316:30 +CPU= 42.323
  21:11:01 STEP 1267 H= 316:45 +CPU= 45.259
  21:11:46 STEP 1268 H= 317:00 +CPU= 44.960
  21:12:33 STEP 1269 H= 317:15 +CPU= 45.868
  21:13:25 STEP 1270 H= 317:30 +CPU= 49.471
  21:14:11 STEP 1271 H= 317:45 +CPU= 45.033
  21:15:06 STEP 1272 H= 318:00 +CPU= 53.597
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+001272
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+001272
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+001272
  21:15:55 STEP 1273 H= 318:15 +CPU= 48.244
  21:16:45 STEP 1274 H= 318:30 +CPU= 49.080
  21:17:32 STEP 1275 H= 318:45 +CPU= 45.897
  21:18:13 STEP 1276 H= 319:00 +CPU= 40.191
  21:19:05 STEP 1277 H= 319:15 +CPU= 50.166
  21:19:52 STEP 1278 H= 319:30 +CPU= 45.772
  21:20:34 STEP 1279 H= 319:45 +CPU= 40.885
  21:21:23 STEP 1280 H= 320:00 +CPU= 46.779
  21:22:13 STEP 1281 H= 320:15 +CPU= 49.106
  21:23:01 STEP 1282 H= 320:30 +CPU= 46.836
  21:23:47 STEP 1283 H= 320:45 +CPU= 44.643
  21:24:46 STEP 1284 H= 321:00 +CPU= 57.265
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+001284
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+001284
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+001284
  21:25:33 STEP 1285 H= 321:15 +CPU= 46.241
  21:26:15 STEP 1286 H= 321:30 +CPU= 41.597
  21:27:01 STEP 1287 H= 321:45 +CPU= 44.463
  21:27:51 STEP 1288 H= 322:00 +CPU= 48.879
  21:28:37 STEP 1289 H= 322:15 +CPU= 43.811
  21:29:27 STEP 1290 H= 322:30 +CPU= 49.352
  21:30:16 STEP 1291 H= 322:45 +CPU= 46.552
  21:31:01 STEP 1292 H= 323:00 +CPU= 43.798
  21:31:42 STEP 1293 H= 323:15 +CPU= 41.030
  21:32:26 STEP 1294 H= 323:30 +CPU= 42.597
  21:33:16 STEP 1295 H= 323:45 +CPU= 47.736
  21:34:13 STEP 1296 H= 324:00 +CPU= 56.195
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+001296
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+001296
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+001296
  21:34:59 STEP 1297 H= 324:15 +CPU= 44.494
  21:35:41 STEP 1298 H= 324:30 +CPU= 40.903
  21:36:29 STEP 1299 H= 324:45 +CPU= 45.395
  21:37:15 STEP 1300 H= 325:00 +CPU= 44.233
  21:37:57 STEP 1301 H= 325:15 +CPU= 40.888
  21:38:39 STEP 1302 H= 325:30 +CPU= 40.345
  21:39:30 STEP 1303 H= 325:45 +CPU= 48.271
  21:40:20 STEP 1304 H= 326:00 +CPU= 48.612
  21:41:12 STEP 1305 H= 326:15 +CPU= 49.954
  21:42:03 STEP 1306 H= 326:30 +CPU= 49.099
  21:42:53 STEP 1307 H= 326:45 +CPU= 47.848
  21:43:48 STEP 1308 H= 327:00 +CPU= 52.875
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+001308
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+001308
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+001308
  21:44:32 STEP 1309 H= 327:15 +CPU= 41.764
  21:45:14 STEP 1310 H= 327:30 +CPU= 41.047
  21:46:03 STEP 1311 H= 327:45 +CPU= 47.547
  21:46:56 STEP 1312 H= 328:00 +CPU= 50.255
  21:47:31 STEP 1313 H= 328:15 +CPU= 34.861
  21:48:21 STEP 1314 H= 328:30 +CPU= 48.315
  21:49:06 STEP 1315 H= 328:45 +CPU= 43.934
  21:49:45 STEP 1316 H= 329:00 +CPU= 37.997
  21:50:25 STEP 1317 H= 329:15 +CPU= 39.047
  21:51:05 STEP 1318 H= 329:30 +CPU= 38.116
  21:51:50 STEP 1319 H= 329:45 +CPU= 41.904
  21:52:47 STEP 1320 H= 330:00 +CPU= 54.536
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+001320
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+001320
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+001320
  21:53:36 STEP 1321 H= 330:15 +CPU= 46.464
  21:54:26 STEP 1322 H= 330:30 +CPU= 46.519
  21:55:16 STEP 1323 H= 330:45 +CPU= 45.594
  21:56:06 STEP 1324 H= 331:00 +CPU= 44.097
  21:57:00 STEP 1325 H= 331:15 +CPU= 47.055
  21:57:52 STEP 1326 H= 331:30 +CPU= 44.929
  21:58:47 STEP 1327 H= 331:45 +CPU= 47.811
  21:59:44 STEP 1328 H= 332:00 +CPU= 48.757
  22:00:41 STEP 1329 H= 332:15 +CPU= 47.509
  22:01:31 STEP 1330 H= 332:30 +CPU= 45.451
  22:02:27 STEP 1331 H= 332:45 +CPU= 45.706
  22:03:34 STEP 1332 H= 333:00 +CPU= 56.770
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+001332
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+001332
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+001332
  22:04:29 STEP 1333 H= 333:15 +CPU= 46.351
  22:05:15 STEP 1334 H= 333:30 +CPU= 42.165
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:20:13 STEP 1335 H= 333:45 +CPU= 35.335
  22:20:49 STEP 1336 H= 334:00 +CPU= 33.343
  22:21:25 STEP 1337 H= 334:15 +CPU= 32.564
  22:21:59 STEP 1338 H= 334:30 +CPU= 31.812
  22:22:37 STEP 1339 H= 334:45 +CPU= 34.561
  22:23:32 STEP 1340 H= 335:00 +CPU= 35.527
  22:24:10 STEP 1341 H= 335:15 +CPU= 33.203
  22:24:45 STEP 1342 H= 335:30 +CPU= 32.368
  22:25:20 STEP 1343 H= 335:45 +CPU= 32.811
  22:26:11 STEP 1344 H= 336:00 +CPU= 45.724
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+001344
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+001344
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+001344
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMGGh7zg+001248
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMSHh7zg+001248
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMUAh7zg+001248
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMGGh7zg+001260
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMSHh7zg+001260
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMUAh7zg+001260
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMGGh7zg+001272
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMSHh7zg+001272
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMUAh7zg+001272
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMGGh7zg+001284
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMSHh7zg+001284
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMUAh7zg+001284
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMGGh7zg+001296
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMSHh7zg+001296
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMUAh7zg+001296
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMGGh7zg+001308
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMSHh7zg+001308
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMUAh7zg+001308
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMGGh7zg+001320
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMSHh7zg+001320
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMUAh7zg+001320
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMGGh7zg+001332
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMSHh7zg+001332
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMUAh7zg+001332
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_a3iz_2016092300_20_1017_12286398_0_r1164538962_13.zip
Uploading the intermediate file: upload_file_13.zip
  22:26:48 STEP 1345 H= 336:15 +CPU= 34.004
  22:27:24 STEP 1346 H= 336:30 +CPU= 33.132
  22:28:01 STEP 1347 H= 336:45 +CPU= 33.365
  22:28:38 STEP 1348 H= 337:00 +CPU= 34.197
  22:29:14 STEP 1349 H= 337:15 +CPU= 32.711
  22:29:51 STEP 1350 H= 337:30 +CPU= 33.956
  22:30:29 STEP 1351 H= 337:45 +CPU= 34.586
  22:31:07 STEP 1352 H= 338:00 +CPU= 34.691
  22:31:43 STEP 1353 H= 338:15 +CPU= 33.556
  22:32:22 STEP 1354 H= 338:30 +CPU= 34.926
  22:32:59 STEP 1355 H= 338:45 +CPU= 33.722
  22:33:42 STEP 1356 H= 339:00 +CPU= 40.138
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+001356
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+001356
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+001356
  22:34:19 STEP 1357 H= 339:15 +CPU= 33.887
  22:34:56 STEP 1358 H= 339:30 +CPU= 33.958
  22:35:33 STEP 1359 H= 339:45 +CPU= 33.728
  22:36:09 STEP 1360 H= 340:00 +CPU= 33.769
  22:36:45 STEP 1361 H= 340:15 +CPU= 33.235
  22:37:20 STEP 1362 H= 340:30 +CPU= 32.733
  22:37:57 STEP 1363 H= 340:45 +CPU= 33.567
  22:38:25 STEP 1364 H= 341:00 +CPU= 27.366
  22:38:52 STEP 1365 H= 341:15 +CPU= 26.930
  22:39:19 STEP 1366 H= 341:30 +CPU= 26.312
  22:39:48 STEP 1367 H= 341:45 +CPU= 28.078
  22:40:21 STEP 1368 H= 342:00 +CPU= 32.454
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+001368
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+001368
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+001368
  22:40:48 STEP 1369 H= 342:15 +CPU= 26.410
  22:41:15 STEP 1370 H= 342:30 +CPU= 26.501
  22:41:42 STEP 1371 H= 342:45 +CPU= 26.702
  22:42:10 STEP 1372 H= 343:00 +CPU= 26.999
  22:42:37 STEP 1373 H= 343:15 +CPU= 26.183
  22:43:03 STEP 1374 H= 343:30 +CPU= 25.954
  22:43:30 STEP 1375 H= 343:45 +CPU= 26.262
  22:43:57 STEP 1376 H= 344:00 +CPU= 26.291
  22:44:24 STEP 1377 H= 344:15 +CPU= 26.875
  22:44:49 STEP 1378 H= 344:30 +CPU= 24.327
  22:45:15 STEP 1379 H= 344:45 +CPU= 25.547
  22:45:45 STEP 1380 H= 345:00 +CPU= 29.594
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+001380
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+001380
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+001380
  22:46:10 STEP 1381 H= 345:15 +CPU= 24.200
  22:46:36 STEP 1382 H= 345:30 +CPU= 25.649
  22:46:59 STEP 1383 H= 345:45 +CPU= 23.414
  22:47:24 STEP 1384 H= 346:00 +CPU= 24.513
  22:47:47 STEP 1385 H= 346:15 +CPU= 22.761
  22:48:11 STEP 1386 H= 346:30 +CPU= 23.127
  22:48:34 STEP 1387 H= 346:45 +CPU= 22.977
  22:48:57 STEP 1388 H= 347:00 +CPU= 22.501
  22:49:20 STEP 1389 H= 347:15 +CPU= 22.579
  22:49:45 STEP 1390 H= 347:30 +CPU= 24.575
  22:50:07 STEP 1391 H= 347:45 +CPU= 21.774
  22:50:41 STEP 1392 H= 348:00 +CPU= 34.207
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+001392
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+001392
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+001392
  22:51:07 STEP 1393 H= 348:15 +CPU= 25.697
  22:51:34 STEP 1394 H= 348:30 +CPU= 26.493
  22:52:00 STEP 1395 H= 348:45 +CPU= 25.257
  22:52:28 STEP 1396 H= 349:00 +CPU= 27.422
  22:52:54 STEP 1397 H= 349:15 +CPU= 25.492
  22:53:18 STEP 1398 H= 349:30 +CPU= 24.183
  22:53:44 STEP 1399 H= 349:45 +CPU= 25.368
  22:54:11 STEP 1400 H= 350:00 +CPU= 26.431
  22:54:38 STEP 1401 H= 350:15 +CPU= 26.757
  22:55:04 STEP 1402 H= 350:30 +CPU= 24.762
  22:55:29 STEP 1403 H= 350:45 +CPU= 25.472
  22:55:59 STEP 1404 H= 351:00 +CPU= 28.719
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+001404
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+001404
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+001404
  22:56:25 STEP 1405 H= 351:15 +CPU= 26.167
  22:56:51 STEP 1406 H= 351:30 +CPU= 25.459
  22:57:18 STEP 1407 H= 351:45 +CPU= 26.253
  22:57:43 STEP 1408 H= 352:00 +CPU= 25.186
  22:58:08 STEP 1409 H= 352:15 +CPU= 24.526
  22:58:34 STEP 1410 H= 352:30 +CPU= 25.654
  22:59:00 STEP 1411 H= 352:45 +CPU= 24.813
  22:59:24 STEP 1412 H= 353:00 +CPU= 23.976
  22:59:49 STEP 1413 H= 353:15 +CPU= 24.325
  23:00:15 STEP 1414 H= 353:30 +CPU= 25.931
  23:00:41 STEP 1415 H= 353:45 +CPU= 25.279
  23:01:12 STEP 1416 H= 354:00 +CPU= 30.640
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+001416
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+001416
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+001416
  23:01:37 STEP 1417 H= 354:15 +CPU= 24.788
  23:02:04 STEP 1418 H= 354:30 +CPU= 26.087
  23:02:30 STEP 1419 H= 354:45 +CPU= 25.753
  23:02:56 STEP 1420 H= 355:00 +CPU= 25.468
  23:03:21 STEP 1421 H= 355:15 +CPU= 24.739
  23:03:46 STEP 1422 H= 355:30 +CPU= 24.888
  23:04:12 STEP 1423 H= 355:45 +CPU= 25.124
  23:04:37 STEP 1424 H= 356:00 +CPU= 25.324
  23:05:03 STEP 1425 H= 356:15 +CPU= 24.770
  23:05:29 STEP 1426 H= 356:30 +CPU= 25.978
  23:05:54 STEP 1427 H= 356:45 +CPU= 24.208
  23:06:27 STEP 1428 H= 357:00 +CPU= 32.599
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+001428
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+001428
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+001428
  23:07:16 STEP 1429 H= 357:15 +CPU= 48.292
  23:07:54 STEP 1430 H= 357:30 +CPU= 37.767
  23:08:39 STEP 1431 H= 357:45 +CPU= 44.342
  23:09:30 STEP 1432 H= 358:00 +CPU= 50.353
  23:10:11 STEP 1433 H= 358:15 +CPU= 40.240
  23:10:48 STEP 1434 H= 358:30 +CPU= 36.858
  23:11:24 STEP 1435 H= 358:45 +CPU= 35.356
  23:12:05 STEP 1436 H= 359:00 +CPU= 39.757
  23:12:40 STEP 1437 H= 359:15 +CPU= 35.625
  23:13:18 STEP 1438 H= 359:30 +CPU= 36.673
  23:13:55 STEP 1439 H= 359:45 +CPU= 35.616
  23:14:24 STEP 1440 H= 360:00 +CPU= 28.245
..The child process terminated with status: 0
>>> Printing last 70 lines from file: NODE.001_01

 
DDH------------------------------------------------------------------------
I                                                                         I
I TIME =  360.000H DDH * Diagnostic Var.,     AREA                GLOBAL  I
I                                                                         I
I-------------------------------------------------------------------------I
I   VARIABLE   I      Initial value         I        Current value        I
I-------------------------------------------------------------------------I
I  OMEGA (P/S) I     -0.1073350511E-20      I       0.1470873840E-05      I
ICloud fractionI      0.0000000000E+00      I       0.3580216966E-01      I
I  Relat. Hum. I      0.1242871055E+00      I       0.2129686713E+00      I
I  PASS 01     I      0.0000000000E+00      I       0.2586331532E+03      I
DDH------------------------------------------------------------------------
  
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1440 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001440 MODE=a
 IO-STREAM CLOSED - ICMGGh7zg+001440
 MPI-TASK:   1 -    307886116 BYTES IN      7 RECORDS TRANSFERRED IN    0.0006 SECONDS513143.5 Mbytes/s, TOTAL TIME=    0.0100( 6.0%)
 23:14:24 STEP 1440 H= 360:00 +CPU= 28.245
 END CNT3
 NSTEP =  1440 CNT0   000000000  20.995  20.995  21.268   0.987               0               0               0               0               0
IO-STREAM STATISTICS, TOTAL NO OF REC/BYTES READ           46/  1069125708 WRITTEN        14300/************
 -TASK-OPENED-OPEN-RECS IN -KBYTE IN    -RECS OUT-KBYTE OUT-WALL   -WALL IN-WALL
  OU-TOT IN -TOT OUT
   1    253    0       46      1044068    14300 ************    41.0     3.5    37.5     6.1    84.7
===-=== START OF TIMING STATISTICS ===-===
 
STATS FOR ALL TASKS
 NUM ROUTINE                                     CALLS  MEAN(ms)   MAX(ms)   FRAC(%)  UNBAL(%)
   0 CNT0     - COMPLETE EXECUTION                   1 ********* *********    100.00      0.00
   1 CNT4     - FORWARD INTEGRATION                  1 ********* *********     99.97      0.00
   8 SCAN2M - GRID-POINT DYNAMICS                  521    8897.5    8897.5     18.65      0.00
   9 SPCM     - SPECTRAL COMP.                     480    2309.2    2309.2      4.46      0.00
  10 SCAN2M - PHYSICS                              481   22064.7   22064.7     42.71      0.00
  11 IOPACK   - OUTPUT P.P. RESULTS                 40    1467.8    1467.8      0.24      0.00
  12 SPNORM   - SPECTRAL NORM COMP.                 21     113.5     113.5      0.01      0.00
  17 GRIDFPOS IN CNT4                               40      55.7      55.7      0.01      0.00
  27 RESTART FILES - WRITING                        10    5870.0    5870.0      0.24      0.00
  28 RESTART FILES - READING                         1    8243.8    8243.8      0.03      0.00
  29 SU4FPOS IN CNT4                                40       0.7       0.7      0.00      0.00
  30 DYNFPOS IN CNT4                                40    9358.1    9358.1      1.51      0.00
  31 POSDDH IN STEPO                                 3       1.6       1.6      0.00      0.00
  37 CPGLAG   - SL COMPUTATIONS                    481   11523.9   11523.9     22.30      0.00
  39 SU0YOMB                                         1     318.4     318.4      0.00      0.00
  51 SCAN2M   - SL COMM. PART 1                    481      37.9      37.9      0.07      0.00
  54 SPCM     - M TO S/S TO M TRANSP.              480    1108.0    1108.0      2.14      0.00
  55 SPCIMPF  - S TO M/M TO S TRANSP.              480     115.1     115.1      0.22      0.00
  56 SPNORM   - SPECTRAL NORM COMM.                 21       3.1       3.1      0.00      0.00
 102 LTINV_CTL   - INVERSE LEGENDRE TRANSFORM     1044     665.3     665.3      2.79      0.00
 103 LTDIR_CTL   - DIRECT LEGENDRE TRANSFORM      1000     380.7     380.7      1.53      0.00
 106 FTDIR_CTL   - DIRECT FOURIER TRANSFORM       1000     137.2     137.2      0.55      0.00
 107 FTINV_CTL   - INVERSE FOURIER TRANSFORM      1044     220.4     220.4      0.93      0.00
 140 SULEG       - COMP. OF LEGENDRE POL.            1      35.4      35.4      0.00      0.00
 152 LTINV_CTL   - M TO L TRANSPOSITION           1044     120.0     120.0      0.50      0.00
 153 LTDIR_CTL   - L TO M TRANSPOSITION           1000      77.3      77.3      0.31      0.00
 157 FTINV_CTL   - L TO G TRANSPOSITION           1044     175.7     175.7      0.74      0.00
 158 FTDIR_CTL   - G TO L TRANSPOSITION           1000      87.3      87.3      0.35      0.00
 400 GSTATS                                      56609       0.0       0.0      0.00      0.00
 401 GSTATS HOOK                                 52617       0.0       0.0      0.00      0.00
TOTAL MEASURED IMBALANCE =       0.0 SECONDS,  0.0 PERCENT
TOTAL WALLCLOCK TIME    24851.9 CPU TIME   22849.1 VECTOR TIME    22849.1


===-=== END   OF TIMING STATISTICS ===-===
FORECAST DAYS PER DAY       52.2
  *** END CNT0 *** 
------------------------------------------------
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+001440
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+001440
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+001440
Adding to the zip: /var/lib/boinc/slots/4/NODE.001_01
Adding to the zip: /var/lib/boinc/slots/4/ifs.stat
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMGGh7zg+001344
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMSHh7zg+001344
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMUAh7zg+001344
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMGGh7zg+001356
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMSHh7zg+001356
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMUAh7zg+001356
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMGGh7zg+001368
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMSHh7zg+001368
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMUAh7zg+001368
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMGGh7zg+001380
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMSHh7zg+001380
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMUAh7zg+001380
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMGGh7zg+001392
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMSHh7zg+001392
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMUAh7zg+001392
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMGGh7zg+001404
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMSHh7zg+001404
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMUAh7zg+001404
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMGGh7zg+001416
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMSHh7zg+001416
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMUAh7zg+001416
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMGGh7zg+001428
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMSHh7zg+001428
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMUAh7zg+001428
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMGGh7zg+001440
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMSHh7zg+001440
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12286398/ICMUAh7zg+001440
Zipping up the final file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_a3iz_2016092300_20_1017_12286398_0_r1164538962_14.zip
Uploading the final file: upload_file_14.zip
Uploading trickle at timestep: 1295100
23:17:51 (8006): called boinc_finish(0)

</stderr_txt>
<message>
upload failure: <file_xfer_error>
  <file_name>oifs_43r3_bl_a3iz_2016092300_20_1017_12286398_0_r1164538962_15.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a3iz_2016092300_20_1017_12286398_0_r1164538962_16.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a3iz_2016092300_20_1017_12286398_0_r1164538962_17.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a3iz_2016092300_20_1017_12286398_0_r1164538962_18.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a3iz_2016092300_20_1017_12286398_0_r1164538962_19.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
</message>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
10 Jun 2024 22:12:27 1533094 22438014 oifs_43r3_bl_a3iz_2016092300_20_1017_12286398_0 1,295,100 75,014 0.0579


©2024 climateprediction.net