climateprediction.net home page
Task 22272643

Task 22272643

Name oifs_43r3_ps_0585_2001050100_123_970_12187229_0
Workunit 12187229
Created 23 Dec 2022, 0:27:20 UTC
Sent 17 Jan 2023, 4:09:49 UTC
Report deadline 18 Mar 2023, 4:09:49 UTC
Received 20 Jan 2023, 8:01:53 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1520317
Run time 18 hours 13 min 58 sec
CPU time 17 hours 39 min 44 sec
Validate state Invalid
Credit 13,426.56
Device peak FLOPS 8.57 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,419.23 MB
Peak swap size 4,978.08 MB
Peak disk usage 7,293.38 MB
Stderr
<core_client_version>7.18.1</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
             (1001) 
[EC_DRHOOK:roo:1:1:34987:34987] [20230119:201316:1674119596:24831.102] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :              EC_PHYS 
[EC_DRHOOK:roo:1:1:34987:34987] [20230119:201316:1674119596:24831.102] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :               CALLPAR 
[EC_DRHOOK:roo:1:1:34987:34987] [20230119:201316:1674119596:24831.102] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                STATE_COPY 
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2001050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0585
(argv4) batchid: 970
(argv5) wuid: 12187229
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/13
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/13/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/13/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_b49c2439e4c3f3cbbe4f8ae37b5bf653 to: /var/lib/boinc-client/slots/13/oifs_43r3_ps_0585_2001050100_123_970_12187229.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/13/oifs_43r3_ps_0585_2001050100_123_970_12187229.zip
ic_ancil_file: ic_ancil_12187229
ifsdata_file: ifsdata_12187229
climate_data_file: clim_data_12187229
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_1cd981a5053c4f1b793f9d58405913eb to: /var/lib/boinc-client/slots/13/ic_ancil_12187229.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/13/ic_ancil_12187229.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_46fa7314a9701548051a528ae5693aca to: /var/lib/boinc-client/slots/13/ifsdata/ifsdata_12187229.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/13/ifsdata/ifsdata_12187229.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/13/159l_2/clim_data_12187229.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/13/159l_2/clim_data_12187229.zip
Checking for progress XML file: /var/lib/boinc-client/slots/13/progress_file_12187229.xml
Opened progress file ok : /var/lib/boinc-client/slots/13/progress_file_12187229.xml
-- Model is restarting --
Adjusting last_iter, 2444, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/13/progress_file_12187229.xml
last_cpu_time: 51481
upload_file_number: 101
last_iter: 2423
last_upload: 8726400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0585_2001050100_123_970_12187229_0_r1050872504
The child process has been launched with process id: 32946
Executing the command: /var/lib/boinc-client/slots/13/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:roo:1:1:32946:32946] [20230120:105516:1674172516:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:roo:1:1:32946:32946] [20230120:105516:1674172516:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:roo:1:1:32946:32946] [20230120:105516:1674172516:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:roo:1:1:32946:32946] [20230120:105516:1674172516:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:roo:1:1:32946:32946] [20230120:105516:1674172516:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:roo:1:1:32946:32946] [20230120:105516:1674172516:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:roo:1:1:32946:32946] [20230120:105516:1674172516:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:roo:1:1:32946:32946] [20230120:105516:1674172516:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:roo:1:1:32946:32946] [20230120:105516:1674172516:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:roo:1:1:32946:32946] [20230120:105516:1674172516:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:roo:1:1:32946:32946] [20230120:105516:1674172516:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:roo:1:1:32946:32946] [20230120:105516:1674172516:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:roo:1:1:32946:32946] [20230120:105516:1674172516:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:roo:1:1:32946:32946] [20230120:105516:1674172516:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:roo:1:1:32946:32946] [20230120:105516:1674172516:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:roo:1:1:32946:32946] [20230120:105516:1674172516:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:roo:1:1:32946:32946] [20230120:105516:1674172516:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:roo:1:1:32946:32946] [20230120:105516:1674172516:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:roo:1:1:32946:32946] [20230120:105516:1674172516:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:roo:1:1:32946:32946] [20230120:105516:1674172516:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:roo:1:1:32946:32946] [20230120:105516:1674172516:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:roo:1:1:32946:32946] [20230120:105516:1674172516:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:roo:1:1:32946:32946] [20230120:105516:1674172516:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:roo:1:1:32946:32946] [20230120:105516:1674172516:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:roo:1:1:32946:32946] [20230120:105516:1674172516:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:roo:1:1:32946:32946] [20230120:105516:1674172516:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:roo:1:1:32946:32946] [20230120:105516:1674172516:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:roo:1:1:32946:32946] [20230120:105516:1674172516:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:roo:1:1:32946:32946] [20230120:105516:1674172516:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:roo:1:1:32946:32946] [20230120:105516:1674172516:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  10:55:48 STEP 2424 H=2424:00 +CPU= 27.097
  10:56:04 STEP 2425 H=2425:00 +CPU= 15.828
  10:56:21 STEP 2426 H=2426:00 +CPU= 16.129
  10:56:45 STEP 2427 H=2427:00 +CPU= 23.838
  10:57:02 STEP 2428 H=2428:00 +CPU= 15.766
  10:57:18 STEP 2429 H=2429:00 +CPU= 15.833
  10:57:43 STEP 2430 H=2430:00 +CPU= 24.665
  10:57:59 STEP 2431 H=2431:00 +CPU= 14.870
  10:58:16 STEP 2432 H=2432:00 +CPU= 15.960
  10:58:42 STEP 2433 H=2433:00 +CPU= 24.585
  10:58:58 STEP 2434 H=2434:00 +CPU= 15.794
  10:59:15 STEP 2435 H=2435:00 +CPU= 15.604
  10:59:43 STEP 2436 H=2436:00 +CPU= 27.178
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+002436
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+002436
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+002436
  10:59:59 STEP 2437 H=2437:00 +CPU= 16.136
  11:00:15 STEP 2438 H=2438:00 +CPU= 15.890
  11:00:40 STEP 2439 H=2439:00 +CPU= 24.713
  11:00:57 STEP 2440 H=2440:00 +CPU= 15.703
  11:01:13 STEP 2441 H=2441:00 +CPU= 15.979
  11:01:39 STEP 2442 H=2442:00 +CPU= 25.255
  11:01:55 STEP 2443 H=2443:00 +CPU= 15.994
  11:02:13 STEP 2444 H=2444:00 +CPU= 16.276
  11:02:37 STEP 2445 H=2445:00 +CPU= 24.164
  11:02:55 STEP 2446 H=2446:00 +CPU= 16.393
  11:03:12 STEP 2447 H=2447:00 +CPU= 16.074
  11:03:41 STEP 2448 H=2448:00 +CPU= 29.099
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+002448
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+002448
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+002448
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMGGhq0f+002424
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMSHhq0f+002424
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMUAhq0f+002424
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMGGhq0f+002436
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMSHhq0f+002436
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMUAhq0f+002436
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0585_2001050100_123_970_12187229_0_r1050872504_101.zip
Uploading the intermediate file: upload_file_101.zip
  11:03:58 STEP 2449 H=2449:00 +CPU= 15.884
  11:04:14 STEP 2450 H=2450:00 +CPU= 15.857
  11:04:41 STEP 2451 H=2451:00 +CPU= 24.622
  11:04:58 STEP 2452 H=2452:00 +CPU= 16.411
  11:05:14 STEP 2453 H=2453:00 +CPU= 15.451
  11:05:39 STEP 2454 H=2454:00 +CPU= 24.474
  11:05:55 STEP 2455 H=2455:00 +CPU= 15.754
  11:06:11 STEP 2456 H=2456:00 +CPU= 16.031
  11:06:36 STEP 2457 H=2457:00 +CPU= 24.542
  11:06:53 STEP 2458 H=2458:00 +CPU= 15.909
  11:07:09 STEP 2459 H=2459:00 +CPU= 15.698
  11:07:38 STEP 2460 H=2460:00 +CPU= 27.283
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+002460
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+002460
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+002460
  11:07:53 STEP 2461 H=2461:00 +CPU= 15.357
  11:08:09 STEP 2462 H=2462:00 +CPU= 15.747
  11:08:34 STEP 2463 H=2463:00 +CPU= 24.670
  11:08:51 STEP 2464 H=2464:00 +CPU= 16.091
  11:09:07 STEP 2465 H=2465:00 +CPU= 15.289
  11:09:33 STEP 2466 H=2466:00 +CPU= 25.082
  11:09:49 STEP 2467 H=2467:00 +CPU= 15.280
  11:10:06 STEP 2468 H=2468:00 +CPU= 16.170
  11:10:31 STEP 2469 H=2469:00 +CPU= 24.638
  11:10:47 STEP 2470 H=2470:00 +CPU= 16.128
  11:11:04 STEP 2471 H=2471:00 +CPU= 16.594
  11:11:36 STEP 2472 H=2472:00 +CPU= 29.815
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+002472
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+002472
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMGGhq0f+002448
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMSHhq0f+002448
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMUAhq0f+002448
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMGGhq0f+002460
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMSHhq0f+002460
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMUAhq0f+002460
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0585_2001050100_123_970_12187229_0_r1050872504_102.zip
Uploading the intermediate file: upload_file_102.zip
  11:11:52 STEP 2473 H=2473:00 +CPU= 15.973
  11:12:08 STEP 2474 H=2474:00 +CPU= 15.768
  11:12:33 STEP 2475 H=2475:00 +CPU= 24.624
  11:12:50 STEP 2476 H=2476:00 +CPU= 16.510
  11:13:05 STEP 2477 H=2477:00 +CPU= 15.273
  11:13:31 STEP 2478 H=2478:00 +CPU= 25.137
  11:13:48 STEP 2479 H=2479:00 +CPU= 16.302
  11:14:05 STEP 2480 H=2480:00 +CPU= 16.629
  11:14:31 STEP 2481 H=2481:00 +CPU= 24.814
  11:14:48 STEP 2482 H=2482:00 +CPU= 16.249
  11:15:04 STEP 2483 H=2483:00 +CPU= 16.060
  11:15:33 STEP 2484 H=2484:00 +CPU= 27.460
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+002484
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+002484
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+002484
  11:15:49 STEP 2485 H=2485:00 +CPU= 15.774
  11:16:06 STEP 2486 H=2486:00 +CPU= 16.348
  11:16:31 STEP 2487 H=2487:00 +CPU= 24.564
  11:16:48 STEP 2488 H=2488:00 +CPU= 16.263
  11:17:05 STEP 2489 H=2489:00 +CPU= 15.926
  11:17:31 STEP 2490 H=2490:00 +CPU= 25.543
  11:17:48 STEP 2491 H=2491:00 +CPU= 15.695
  11:18:06 STEP 2492 H=2492:00 +CPU= 16.383
  11:18:32 STEP 2493 H=2493:00 +CPU= 24.815
  11:18:48 STEP 2494 H=2494:00 +CPU= 15.700
  11:19:04 STEP 2495 H=2495:00 +CPU= 15.729
  11:19:35 STEP 2496 H=2496:00 +CPU= 29.825
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+002496
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+002496
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMGGhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMSHhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMUAhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMGGhq0f+002484
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMSHhq0f+002484
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMUAhq0f+002484
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0585_2001050100_123_970_12187229_0_r1050872504_103.zip
Uploading the intermediate file: upload_file_103.zip
  11:19:52 STEP 2497 H=2497:00 +CPU= 16.371
  11:20:09 STEP 2498 H=2498:00 +CPU= 16.868
  11:20:34 STEP 2499 H=2499:00 +CPU= 25.051
  11:20:51 STEP 2500 H=2500:00 +CPU= 16.374
  11:21:08 STEP 2501 H=2501:00 +CPU= 16.015
  11:21:33 STEP 2502 H=2502:00 +CPU= 25.024
  11:21:49 STEP 2503 H=2503:00 +CPU= 15.516
  11:22:05 STEP 2504 H=2504:00 +CPU= 16.038
  11:22:31 STEP 2505 H=2505:00 +CPU= 24.747
  11:22:48 STEP 2506 H=2506:00 +CPU= 16.222
  11:23:04 STEP 2507 H=2507:00 +CPU= 15.930
  11:23:33 STEP 2508 H=2508:00 +CPU= 27.501
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+002508
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+002508
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+002508
  11:23:49 STEP 2509 H=2509:00 +CPU= 16.005
  11:24:05 STEP 2510 H=2510:00 +CPU= 16.215
  11:24:30 STEP 2511 H=2511:00 +CPU= 24.539
  11:24:47 STEP 2512 H=2512:00 +CPU= 16.585
  11:25:03 STEP 2513 H=2513:00 +CPU= 15.758
  11:25:28 STEP 2514 H=2514:00 +CPU= 25.037
  11:25:44 STEP 2515 H=2515:00 +CPU= 16.053
  11:26:00 STEP 2516 H=2516:00 +CPU= 16.141
  11:26:26 STEP 2517 H=2517:00 +CPU= 24.845
  11:26:44 STEP 2518 H=2518:00 +CPU= 16.751
  11:27:01 STEP 2519 H=2519:00 +CPU= 16.524
  11:27:33 STEP 2520 H=2520:00 +CPU= 30.522
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+002520
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+002520
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMGGhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMSHhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMUAhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMGGhq0f+002508
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMSHhq0f+002508
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMUAhq0f+002508
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0585_2001050100_123_970_12187229_0_r1050872504_104.zip
Uploading the intermediate file: upload_file_104.zip
  11:27:50 STEP 2521 H=2521:00 +CPU= 16.282
  11:28:07 STEP 2522 H=2522:00 +CPU= 16.396
  11:28:32 STEP 2523 H=2523:00 +CPU= 24.789
  11:28:48 STEP 2524 H=2524:00 +CPU= 16.305
  11:29:04 STEP 2525 H=2525:00 +CPU= 15.698
  11:29:30 STEP 2526 H=2526:00 +CPU= 25.693
  11:29:46 STEP 2527 H=2527:00 +CPU= 15.630
  11:30:03 STEP 2528 H=2528:00 +CPU= 16.598
  11:30:29 STEP 2529 H=2529:00 +CPU= 25.321
  11:30:46 STEP 2530 H=2530:00 +CPU= 16.457
  11:31:03 STEP 2531 H=2531:00 +CPU= 16.383
  11:31:32 STEP 2532 H=2532:00 +CPU= 27.976
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+002532
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+002532
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+002532
  11:31:48 STEP 2533 H=2533:00 +CPU= 15.945
  11:32:05 STEP 2534 H=2534:00 +CPU= 16.398
  11:32:31 STEP 2535 H=2535:00 +CPU= 24.577
  11:32:48 STEP 2536 H=2536:00 +CPU= 16.369
  11:33:04 STEP 2537 H=2537:00 +CPU= 15.704
  11:33:31 STEP 2538 H=2538:00 +CPU= 25.221
  11:33:47 STEP 2539 H=2539:00 +CPU= 15.738
  11:34:04 STEP 2540 H=2540:00 +CPU= 16.697
  11:34:30 STEP 2541 H=2541:00 +CPU= 24.778
  11:34:46 STEP 2542 H=2542:00 +CPU= 16.211
  11:35:03 STEP 2543 H=2543:00 +CPU= 15.957
  11:35:34 STEP 2544 H=2544:00 +CPU= 30.261
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+002544
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+002544
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMGGhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMSHhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMUAhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMGGhq0f+002532
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMSHhq0f+002532
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMUAhq0f+002532
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0585_2001050100_123_970_12187229_0_r1050872504_105.zip
Uploading the intermediate file: upload_file_105.zip
  11:35:50 STEP 2545 H=2545:00 +CPU= 15.711
  11:36:07 STEP 2546 H=2546:00 +CPU= 16.304
  11:36:32 STEP 2547 H=2547:00 +CPU= 25.394
  11:36:49 STEP 2548 H=2548:00 +CPU= 16.385
  11:37:05 STEP 2549 H=2549:00 +CPU= 16.593
  11:37:31 STEP 2550 H=2550:00 +CPU= 25.715
  11:37:48 STEP 2551 H=2551:00 +CPU= 16.096
  11:38:04 STEP 2552 H=2552:00 +CPU= 16.286
  11:38:29 STEP 2553 H=2553:00 +CPU= 24.720
  11:38:46 STEP 2554 H=2554:00 +CPU= 16.674
  11:39:02 STEP 2555 H=2555:00 +CPU= 15.707
  11:39:31 STEP 2556 H=2556:00 +CPU= 27.805
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+002556
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+002556
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+002556
  11:39:47 STEP 2557 H=2557:00 +CPU= 16.295
  11:40:05 STEP 2558 H=2558:00 +CPU= 17.217
  11:40:31 STEP 2559 H=2559:00 +CPU= 25.039
  11:40:48 STEP 2560 H=2560:00 +CPU= 16.473
  11:41:05 STEP 2561 H=2561:00 +CPU= 15.778
  11:41:30 STEP 2562 H=2562:00 +CPU= 24.967
  11:41:46 STEP 2563 H=2563:00 +CPU= 15.780
  11:42:03 STEP 2564 H=2564:00 +CPU= 16.028
  11:42:28 STEP 2565 H=2565:00 +CPU= 24.758
  11:42:45 STEP 2566 H=2566:00 +CPU= 16.549
  11:43:03 STEP 2567 H=2567:00 +CPU= 16.610
  11:43:34 STEP 2568 H=2568:00 +CPU= 30.046
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+002568
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+002568
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMGGhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMSHhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMUAhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMGGhq0f+002556
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMSHhq0f+002556
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMUAhq0f+002556
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0585_2001050100_123_970_12187229_0_r1050872504_106.zip
Uploading the intermediate file: upload_file_106.zip
  11:43:50 STEP 2569 H=2569:00 +CPU= 15.578
  11:44:07 STEP 2570 H=2570:00 +CPU= 16.301
  11:44:33 STEP 2571 H=2571:00 +CPU= 25.161
  11:44:51 STEP 2572 H=2572:00 +CPU= 16.641
  11:45:08 STEP 2573 H=2573:00 +CPU= 16.327
  11:45:35 STEP 2574 H=2574:00 +CPU= 25.585
  11:45:51 STEP 2575 H=2575:00 +CPU= 16.106
  11:46:09 STEP 2576 H=2576:00 +CPU= 16.436
  11:46:34 STEP 2577 H=2577:00 +CPU= 25.036
  11:46:52 STEP 2578 H=2578:00 +CPU= 16.570
  11:47:08 STEP 2579 H=2579:00 +CPU= 15.655
  11:47:36 STEP 2580 H=2580:00 +CPU= 27.486
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+002580
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+002580
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+002580
  11:47:53 STEP 2581 H=2581:00 +CPU= 16.096
  11:48:10 STEP 2582 H=2582:00 +CPU= 16.542
  11:48:36 STEP 2583 H=2583:00 +CPU= 24.807
  11:48:52 STEP 2584 H=2584:00 +CPU= 16.139
  11:49:09 STEP 2585 H=2585:00 +CPU= 16.446
  11:49:34 STEP 2586 H=2586:00 +CPU= 24.905
  11:49:51 STEP 2587 H=2587:00 +CPU= 15.676
  11:50:07 STEP 2588 H=2588:00 +CPU= 15.951
  11:50:31 STEP 2589 H=2589:00 +CPU= 23.822
  11:50:47 STEP 2590 H=2590:00 +CPU= 15.871
  11:51:04 STEP 2591 H=2591:00 +CPU= 16.076
  11:51:34 STEP 2592 H=2592:00 +CPU= 29.631
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+002592
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+002592
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMGGhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMSHhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMUAhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMGGhq0f+002580
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMSHhq0f+002580
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMUAhq0f+002580
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0585_2001050100_123_970_12187229_0_r1050872504_107.zip
Uploading the intermediate file: upload_file_107.zip
  11:51:51 STEP 2593 H=2593:00 +CPU= 15.974
  11:52:08 STEP 2594 H=2594:00 +CPU= 16.465
  11:52:33 STEP 2595 H=2595:00 +CPU= 24.803
  11:52:50 STEP 2596 H=2596:00 +CPU= 16.226
  11:53:06 STEP 2597 H=2597:00 +CPU= 15.440
  11:53:32 STEP 2598 H=2598:00 +CPU= 25.308
  11:53:48 STEP 2599 H=2599:00 +CPU= 15.489
  11:54:04 STEP 2600 H=2600:00 +CPU= 16.306
  11:54:29 STEP 2601 H=2601:00 +CPU= 25.134
  11:54:46 STEP 2602 H=2602:00 +CPU= 16.468
  11:55:01 STEP 2603 H=2603:00 +CPU= 15.414
  11:55:29 STEP 2604 H=2604:00 +CPU= 27.066
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+002604
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+002604
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+002604
  11:55:45 STEP 2605 H=2605:00 +CPU= 15.688
  11:56:02 STEP 2606 H=2606:00 +CPU= 16.357
  11:56:27 STEP 2607 H=2607:00 +CPU= 24.475
  11:56:43 STEP 2608 H=2608:00 +CPU= 15.809
  11:56:58 STEP 2609 H=2609:00 +CPU= 15.127
  11:57:23 STEP 2610 H=2610:00 +CPU= 24.460
  11:57:39 STEP 2611 H=2611:00 +CPU= 15.753
  11:57:56 STEP 2612 H=2612:00 +CPU= 16.278
  11:58:23 STEP 2613 H=2613:00 +CPU= 25.447
  11:58:40 STEP 2614 H=2614:00 +CPU= 16.260
  11:58:56 STEP 2615 H=2615:00 +CPU= 16.101
  11:59:26 STEP 2616 H=2616:00 +CPU= 29.320
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+002616
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+002616
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMGGhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMSHhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMUAhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMGGhq0f+002604
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMSHhq0f+002604
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMUAhq0f+002604
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0585_2001050100_123_970_12187229_0_r1050872504_108.zip
Uploading the intermediate file: upload_file_108.zip
  11:59:43 STEP 2617 H=2617:00 +CPU= 15.679
  11:59:59 STEP 2618 H=2618:00 +CPU= 16.217
  12:00:25 STEP 2619 H=2619:00 +CPU= 24.655
  12:00:42 STEP 2620 H=2620:00 +CPU= 16.434
  12:00:58 STEP 2621 H=2621:00 +CPU= 15.845
  12:01:23 STEP 2622 H=2622:00 +CPU= 25.010
  12:01:40 STEP 2623 H=2623:00 +CPU= 16.139
  12:01:57 STEP 2624 H=2624:00 +CPU= 16.072
  12:02:23 STEP 2625 H=2625:00 +CPU= 24.518
  12:02:39 STEP 2626 H=2626:00 +CPU= 16.083
  12:02:56 STEP 2627 H=2627:00 +CPU= 15.786
  12:03:24 STEP 2628 H=2628:00 +CPU= 27.871
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+002628
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+002628
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+002628
  12:03:41 STEP 2629 H=2629:00 +CPU= 15.845
  12:03:57 STEP 2630 H=2630:00 +CPU= 16.147
  12:04:22 STEP 2631 H=2631:00 +CPU= 24.444
  12:04:38 STEP 2632 H=2632:00 +CPU= 16.047
  12:04:53 STEP 2633 H=2633:00 +CPU= 15.495
  12:05:19 STEP 2634 H=2634:00 +CPU= 25.085
  12:05:36 STEP 2635 H=2635:00 +CPU= 15.714
  12:05:52 STEP 2636 H=2636:00 +CPU= 16.024
  12:06:18 STEP 2637 H=2637:00 +CPU= 24.720
  12:06:35 STEP 2638 H=2638:00 +CPU= 15.918
  12:06:52 STEP 2639 H=2639:00 +CPU= 16.205
  12:07:23 STEP 2640 H=2640:00 +CPU= 29.687
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+002640
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+002640
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMGGhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMSHhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMUAhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMGGhq0f+002628
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMSHhq0f+002628
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMUAhq0f+002628
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0585_2001050100_123_970_12187229_0_r1050872504_109.zip
Uploading the intermediate file: upload_file_109.zip
  12:07:39 STEP 2641 H=2641:00 +CPU= 15.629
  12:07:56 STEP 2642 H=2642:00 +CPU= 16.001
  12:08:21 STEP 2643 H=2643:00 +CPU= 24.331
  12:08:38 STEP 2644 H=2644:00 +CPU= 15.983
  12:08:54 STEP 2645 H=2645:00 +CPU= 15.650
  12:09:19 STEP 2646 H=2646:00 +CPU= 24.839
  12:09:35 STEP 2647 H=2647:00 +CPU= 15.612
  12:09:52 STEP 2648 H=2648:00 +CPU= 16.007
  12:10:17 STEP 2649 H=2649:00 +CPU= 24.431
  12:10:33 STEP 2650 H=2650:00 +CPU= 15.875
  12:10:49 STEP 2651 H=2651:00 +CPU= 15.503
  12:11:17 STEP 2652 H=2652:00 +CPU= 27.225
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+002652
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+002652
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+002652
  12:11:34 STEP 2653 H=2653:00 +CPU= 16.038
  12:11:51 STEP 2654 H=2654:00 +CPU= 16.639
  12:12:17 STEP 2655 H=2655:00 +CPU= 25.174
  12:12:33 STEP 2656 H=2656:00 +CPU= 16.197
  12:12:49 STEP 2657 H=2657:00 +CPU= 15.522
  12:13:15 STEP 2658 H=2658:00 +CPU= 25.017
  12:13:31 STEP 2659 H=2659:00 +CPU= 15.538
  12:13:48 STEP 2660 H=2660:00 +CPU= 15.797
  12:14:12 STEP 2661 H=2661:00 +CPU= 23.898
  12:14:27 STEP 2662 H=2662:00 +CPU= 15.293
  12:14:43 STEP 2663 H=2663:00 +CPU= 15.507
  12:15:12 STEP 2664 H=2664:00 +CPU= 28.829
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+002664
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+002664
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMGGhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMSHhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMUAhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMGGhq0f+002652
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMSHhq0f+002652
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMUAhq0f+002652
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0585_2001050100_123_970_12187229_0_r1050872504_110.zip
Uploading the intermediate file: upload_file_110.zip
  12:15:28 STEP 2665 H=2665:00 +CPU= 15.429
Uploading trickle at timestep: 9590400
  12:15:44 STEP 2666 H=2666:00 +CPU= 15.853
  12:16:09 STEP 2667 H=2667:00 +CPU= 24.276
  12:16:26 STEP 2668 H=2668:00 +CPU= 16.037
  12:16:42 STEP 2669 H=2669:00 +CPU= 15.370
  12:17:06 STEP 2670 H=2670:00 +CPU= 24.460
  12:17:22 STEP 2671 H=2671:00 +CPU= 15.754
  12:17:39 STEP 2672 H=2672:00 +CPU= 16.468
  12:18:04 STEP 2673 H=2673:00 +CPU= 24.193
  12:18:20 STEP 2674 H=2674:00 +CPU= 15.615
  12:18:36 STEP 2675 H=2675:00 +CPU= 15.131
  12:19:04 STEP 2676 H=2676:00 +CPU= 26.891
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+002676
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+002676
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+002676
  12:19:20 STEP 2677 H=2677:00 +CPU= 15.369
  12:19:36 STEP 2678 H=2678:00 +CPU= 15.790
  12:20:02 STEP 2679 H=2679:00 +CPU= 24.678
  12:20:18 STEP 2680 H=2680:00 +CPU= 16.140
  12:20:34 STEP 2681 H=2681:00 +CPU= 15.521
  12:20:58 STEP 2682 H=2682:00 +CPU= 24.043
  12:21:14 STEP 2683 H=2683:00 +CPU= 15.420
  12:21:30 STEP 2684 H=2684:00 +CPU= 15.774
  12:21:55 STEP 2685 H=2685:00 +CPU= 24.835
  12:22:12 STEP 2686 H=2686:00 +CPU= 15.992
  12:22:29 STEP 2687 H=2687:00 +CPU= 16.417
  12:23:00 STEP 2688 H=2688:00 +CPU= 29.801
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+002688
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+002688
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMGGhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMSHhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMUAhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMGGhq0f+002676
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMSHhq0f+002676
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMUAhq0f+002676
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0585_2001050100_123_970_12187229_0_r1050872504_111.zip
Uploading the intermediate file: upload_file_111.zip
  12:23:16 STEP 2689 H=2689:00 +CPU= 15.592
  12:23:32 STEP 2690 H=2690:00 +CPU= 15.698
  12:23:56 STEP 2691 H=2691:00 +CPU= 24.156
  12:24:13 STEP 2692 H=2692:00 +CPU= 15.936
  12:24:28 STEP 2693 H=2693:00 +CPU= 15.020
  12:24:53 STEP 2694 H=2694:00 +CPU= 24.713
  12:25:09 STEP 2695 H=2695:00 +CPU= 15.530
  12:25:26 STEP 2696 H=2696:00 +CPU= 16.139
  12:25:51 STEP 2697 H=2697:00 +CPU= 25.023
  12:26:08 STEP 2698 H=2698:00 +CPU= 16.476
  12:26:24 STEP 2699 H=2699:00 +CPU= 15.889
  12:26:52 STEP 2700 H=2700:00 +CPU= 27.141
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+002700
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+002700
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+002700
  12:27:07 STEP 2701 H=2701:00 +CPU= 15.406
  12:27:23 STEP 2702 H=2702:00 +CPU= 15.674
  12:27:49 STEP 2703 H=2703:00 +CPU= 24.567
  12:28:05 STEP 2704 H=2704:00 +CPU= 16.015
  12:28:21 STEP 2705 H=2705:00 +CPU= 15.553
  12:28:47 STEP 2706 H=2706:00 +CPU= 25.101
  12:29:03 STEP 2707 H=2707:00 +CPU= 15.654
  12:29:19 STEP 2708 H=2708:00 +CPU= 16.032
  12:29:44 STEP 2709 H=2709:00 +CPU= 24.568
  12:30:01 STEP 2710 H=2710:00 +CPU= 15.899
  12:30:17 STEP 2711 H=2711:00 +CPU= 15.872
  12:30:47 STEP 2712 H=2712:00 +CPU= 29.583
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+002712
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+002712
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMGGhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMSHhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMUAhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMGGhq0f+002700
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMSHhq0f+002700
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMUAhq0f+002700
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0585_2001050100_123_970_12187229_0_r1050872504_112.zip
Uploading the intermediate file: upload_file_112.zip
  12:31:03 STEP 2713 H=2713:00 +CPU= 15.978
  12:31:19 STEP 2714 H=2714:00 +CPU= 16.472
  12:31:44 STEP 2715 H=2715:00 +CPU= 24.574
  12:32:01 STEP 2716 H=2716:00 +CPU= 16.374
  12:32:17 STEP 2717 H=2717:00 +CPU= 15.931
  12:32:42 STEP 2718 H=2718:00 +CPU= 24.551
  12:32:58 STEP 2719 H=2719:00 +CPU= 15.354
  12:33:14 STEP 2720 H=2720:00 +CPU= 15.975
  12:33:38 STEP 2721 H=2721:00 +CPU= 24.066
  12:33:54 STEP 2722 H=2722:00 +CPU= 15.738
  12:34:10 STEP 2723 H=2723:00 +CPU= 15.443
  12:34:37 STEP 2724 H=2724:00 +CPU= 27.008
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+002724
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+002724
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+002724
  12:34:53 STEP 2725 H=2725:00 +CPU= 15.122
  12:35:08 STEP 2726 H=2726:00 +CPU= 15.534
  12:35:32 STEP 2727 H=2727:00 +CPU= 23.676
  12:35:48 STEP 2728 H=2728:00 +CPU= 15.544
  12:36:03 STEP 2729 H=2729:00 +CPU= 15.223
  12:36:28 STEP 2730 H=2730:00 +CPU= 24.446
  12:36:43 STEP 2731 H=2731:00 +CPU= 14.931
  12:36:59 STEP 2732 H=2732:00 +CPU= 16.020
  12:37:24 STEP 2733 H=2733:00 +CPU= 24.206
  12:37:40 STEP 2734 H=2734:00 +CPU= 15.815
  12:37:56 STEP 2735 H=2735:00 +CPU= 15.646
  12:38:27 STEP 2736 H=2736:00 +CPU= 29.371
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+002736
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+002736
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMGGhq0f+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMSHhq0f+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMUAhq0f+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMGGhq0f+002724
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMSHhq0f+002724
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMUAhq0f+002724
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0585_2001050100_123_970_12187229_0_r1050872504_113.zip
Uploading the intermediate file: upload_file_113.zip
  12:38:43 STEP 2737 H=2737:00 +CPU= 15.047
  12:38:59 STEP 2738 H=2738:00 +CPU= 15.640
  12:39:24 STEP 2739 H=2739:00 +CPU= 24.749
  12:39:42 STEP 2740 H=2740:00 +CPU= 16.747
  12:39:59 STEP 2741 H=2741:00 +CPU= 16.058
  12:40:25 STEP 2742 H=2742:00 +CPU= 25.435
  12:40:42 STEP 2743 H=2743:00 +CPU= 15.597
  12:40:58 STEP 2744 H=2744:00 +CPU= 15.968
  12:41:25 STEP 2745 H=2745:00 +CPU= 25.396
  12:41:42 STEP 2746 H=2746:00 +CPU= 16.288
  12:41:58 STEP 2747 H=2747:00 +CPU= 15.368
  12:42:25 STEP 2748 H=2748:00 +CPU= 27.349
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+002748
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+002748
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+002748
  12:42:41 STEP 2749 H=2749:00 +CPU= 16.113
  12:42:58 STEP 2750 H=2750:00 +CPU= 16.848
  12:43:24 STEP 2751 H=2751:00 +CPU= 25.040
  12:43:41 STEP 2752 H=2752:00 +CPU= 16.616
  12:43:58 STEP 2753 H=2753:00 +CPU= 16.239
  12:44:24 STEP 2754 H=2754:00 +CPU= 25.099
  12:44:40 STEP 2755 H=2755:00 +CPU= 15.220
  12:44:56 STEP 2756 H=2756:00 +CPU= 15.908
  12:45:20 STEP 2757 H=2757:00 +CPU= 24.661
  12:45:38 STEP 2758 H=2758:00 +CPU= 16.307
  12:45:54 STEP 2759 H=2759:00 +CPU= 16.070
  12:46:25 STEP 2760 H=2760:00 +CPU= 29.910
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+002760
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+002760
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMGGhq0f+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMSHhq0f+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMUAhq0f+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMGGhq0f+002748
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMSHhq0f+002748
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMUAhq0f+002748
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0585_2001050100_123_970_12187229_0_r1050872504_114.zip
Uploading the intermediate file: upload_file_114.zip
  12:46:42 STEP 2761 H=2761:00 +CPU= 15.941
  12:46:59 STEP 2762 H=2762:00 +CPU= 16.396
  12:47:24 STEP 2763 H=2763:00 +CPU= 24.466
  12:47:40 STEP 2764 H=2764:00 +CPU= 16.037
  12:47:55 STEP 2765 H=2765:00 +CPU= 15.486
  12:48:20 STEP 2766 H=2766:00 +CPU= 24.784
  12:48:36 STEP 2767 H=2767:00 +CPU= 15.708
  12:48:53 STEP 2768 H=2768:00 +CPU= 16.697
  12:49:19 STEP 2769 H=2769:00 +CPU= 24.436
  12:49:36 STEP 2770 H=2770:00 +CPU= 16.584
  12:49:53 STEP 2771 H=2771:00 +CPU= 16.183
  12:50:22 STEP 2772 H=2772:00 +CPU= 28.676
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+002772
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+002772
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+002772
  12:50:38 STEP 2773 H=2773:00 +CPU= 15.784
  12:50:55 STEP 2774 H=2774:00 +CPU= 16.827
  12:51:30 STEP 2775 H=2775:00 +CPU= 34.378
  12:52:08 STEP 2776 H=2776:00 +CPU= 35.726
  12:52:50 STEP 2777 H=2777:00 +CPU= 39.454
  12:53:49 STEP 2778 H=2778:00 +CPU= 57.857
  12:54:25 STEP 2779 H=2779:00 +CPU= 35.094
  12:55:06 STEP 2780 H=2780:00 +CPU= 40.029
  12:56:06 STEP 2781 H=2781:00 +CPU= 59.491
  12:56:47 STEP 2782 H=2782:00 +CPU= 40.199
  12:57:34 STEP 2783 H=2783:00 +CPU= 43.975
  12:59:03 STEP 2784 H=2784:00 +CPU= 81.348
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+002784
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+002784
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMGGhq0f+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMSHhq0f+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMUAhq0f+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMGGhq0f+002772
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMSHhq0f+002772
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMUAhq0f+002772
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0585_2001050100_123_970_12187229_0_r1050872504_115.zip
Uploading the intermediate file: upload_file_115.zip
  12:59:44 STEP 2785 H=2785:00 +CPU= 39.723
  13:00:34 STEP 2786 H=2786:00 +CPU= 46.656
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2001050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0585
(argv4) batchid: 970
(argv5) wuid: 12187229
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/13
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/13/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/13/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_b49c2439e4c3f3cbbe4f8ae37b5bf653 to: /var/lib/boinc-client/slots/13/oifs_43r3_ps_0585_2001050100_123_970_12187229.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/13/oifs_43r3_ps_0585_2001050100_123_970_12187229.zip
ic_ancil_file: ic_ancil_12187229
ifsdata_file: ifsdata_12187229
climate_data_file: clim_data_12187229
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_1cd981a5053c4f1b793f9d58405913eb to: /var/lib/boinc-client/slots/13/ic_ancil_12187229.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/13/ic_ancil_12187229.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_46fa7314a9701548051a528ae5693aca to: /var/lib/boinc-client/slots/13/ifsdata/ifsdata_12187229.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/13/ifsdata/ifsdata_12187229.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/13/159l_2/clim_data_12187229.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/13/159l_2/clim_data_12187229.zip
Checking for progress XML file: /var/lib/boinc-client/slots/13/progress_file_12187229.xml
Opened progress file ok : /var/lib/boinc-client/slots/13/progress_file_12187229.xml
-- Model is restarting --
Adjusting last_iter, 2787, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/13/progress_file_12187229.xml
last_cpu_time: 58793
upload_file_number: 116
last_iter: 2783
last_upload: 10022400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0585_2001050100_123_970_12187229_0_r1050872504
The child process has been launched with process id: 44127
Executing the command: /var/lib/boinc-client/slots/13/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:roo:1:1:44127:44127] [20230120:130200:1674180120:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:roo:1:1:44127:44127] [20230120:130200:1674180120:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:roo:1:1:44127:44127] [20230120:130200:1674180120:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:roo:1:1:44127:44127] [20230120:130200:1674180120:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:roo:1:1:44127:44127] [20230120:130200:1674180120:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:roo:1:1:44127:44127] [20230120:130200:1674180120:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:roo:1:1:44127:44127] [20230120:130200:1674180120:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:roo:1:1:44127:44127] [20230120:130200:1674180120:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:roo:1:1:44127:44127] [20230120:130200:1674180120:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:roo:1:1:44127:44127] [20230120:130200:1674180120:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:roo:1:1:44127:44127] [20230120:130200:1674180120:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:roo:1:1:44127:44127] [20230120:130200:1674180120:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:roo:1:1:44127:44127] [20230120:130200:1674180120:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:roo:1:1:44127:44127] [20230120:130200:1674180120:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:roo:1:1:44127:44127] [20230120:130200:1674180120:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:roo:1:1:44127:44127] [20230120:130200:1674180120:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:roo:1:1:44127:44127] [20230120:130200:1674180120:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:roo:1:1:44127:44127] [20230120:130200:1674180120:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:roo:1:1:44127:44127] [20230120:130200:1674180120:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:roo:1:1:44127:44127] [20230120:130200:1674180120:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:roo:1:1:44127:44127] [20230120:130200:1674180120:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:roo:1:1:44127:44127] [20230120:130200:1674180120:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:roo:1:1:44127:44127] [20230120:130200:1674180120:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:roo:1:1:44127:44127] [20230120:130200:1674180120:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:roo:1:1:44127:44127] [20230120:130200:1674180120:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:roo:1:1:44127:44127] [20230120:130200:1674180120:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:roo:1:1:44127:44127] [20230120:130200:1674180120:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:roo:1:1:44127:44127] [20230120:130200:1674180120:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:roo:1:1:44127:44127] [20230120:130200:1674180120:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:roo:1:1:44127:44127] [20230120:130200:1674180120:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  13:03:27 STEP 2784 H=2784:00 +CPU= 74.155
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:36:43 STEP 2785 H=2785:00 +CPU= 37.495
  16:37:21 STEP 2786 H=2786:00 +CPU= 37.223
  16:38:22 STEP 2787 H=2787:00 +CPU= 59.632
  16:39:09 STEP 2788 H=2788:00 +CPU= 46.866
  16:40:00 STEP 2789 H=2789:00 +CPU= 48.718
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:01:19 STEP 2790 H=2790:00 +CPU= 72.408
  17:02:06 STEP 2791 H=2791:00 +CPU= 45.840
  17:02:54 STEP 2792 H=2792:00 +CPU= 47.321
  17:03:59 STEP 2793 H=2793:00 +CPU= 63.819
  17:04:41 STEP 2794 H=2794:00 +CPU= 41.115
  17:05:27 STEP 2795 H=2795:00 +CPU= 45.111
  17:06:41 STEP 2796 H=2796:00 +CPU= 73.139
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+002796
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+002796
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+002796
  17:07:23 STEP 2797 H=2797:00 +CPU= 41.846
  17:08:11 STEP 2798 H=2798:00 +CPU= 46.593
  17:09:19 STEP 2799 H=2799:00 +CPU= 67.414
  17:09:59 STEP 2800 H=2800:00 +CPU= 39.580
  17:10:40 STEP 2801 H=2801:00 +CPU= 40.669
  17:11:48 STEP 2802 H=2802:00 +CPU= 66.339
  17:12:34 STEP 2803 H=2803:00 +CPU= 45.268
  17:13:19 STEP 2804 H=2804:00 +CPU= 44.513
  17:14:25 STEP 2805 H=2805:00 +CPU= 65.615
  17:15:08 STEP 2806 H=2806:00 +CPU= 42.783
  17:16:02 STEP 2807 H=2807:00 +CPU= 52.428
  17:17:19 STEP 2808 H=2808:00 +CPU= 76.673
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+002808
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+002808
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMGGhq0f+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMSHhq0f+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMUAhq0f+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMGGhq0f+002796
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMSHhq0f+002796
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMUAhq0f+002796
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0585_2001050100_123_970_12187229_0_r1050872504_116.zip
Uploading the intermediate file: upload_file_116.zip
  17:18:00 STEP 2809 H=2809:00 +CPU= 39.905
  17:18:40 STEP 2810 H=2810:00 +CPU= 40.129
  17:19:42 STEP 2811 H=2811:00 +CPU= 60.897
  17:20:24 STEP 2812 H=2812:00 +CPU= 42.144
  17:21:02 STEP 2813 H=2813:00 +CPU= 36.810
  17:22:03 STEP 2814 H=2814:00 +CPU= 60.255
  17:22:43 STEP 2815 H=2815:00 +CPU= 40.021
  17:23:26 STEP 2816 H=2816:00 +CPU= 41.825
  17:24:29 STEP 2817 H=2817:00 +CPU= 61.758
  17:25:08 STEP 2818 H=2818:00 +CPU= 38.691
  17:25:47 STEP 2819 H=2819:00 +CPU= 38.470
  17:26:54 STEP 2820 H=2820:00 +CPU= 66.117
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+002820
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+002820
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+002820
  17:27:34 STEP 2821 H=2821:00 +CPU= 38.895
  17:28:14 STEP 2822 H=2822:00 +CPU= 39.723
  17:29:17 STEP 2823 H=2823:00 +CPU= 62.045
  17:29:58 STEP 2824 H=2824:00 +CPU= 39.908
  17:30:37 STEP 2825 H=2825:00 +CPU= 39.098
  17:31:37 STEP 2826 H=2826:00 +CPU= 58.910
  17:32:18 STEP 2827 H=2827:00 +CPU= 40.080
  17:32:58 STEP 2828 H=2828:00 +CPU= 40.209
  17:33:59 STEP 2829 H=2829:00 +CPU= 59.425
  17:34:40 STEP 2830 H=2830:00 +CPU= 40.599
  17:35:21 STEP 2831 H=2831:00 +CPU= 40.953
  17:36:33 STEP 2832 H=2832:00 +CPU= 70.776
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+002832
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+002832
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMGGhq0f+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMSHhq0f+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMUAhq0f+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMGGhq0f+002820
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMSHhq0f+002820
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMUAhq0f+002820
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0585_2001050100_123_970_12187229_0_r1050872504_117.zip
Uploading the intermediate file: upload_file_117.zip
  17:37:16 STEP 2833 H=2833:00 +CPU= 42.175
  17:37:57 STEP 2834 H=2834:00 +CPU= 40.550
  17:38:57 STEP 2835 H=2835:00 +CPU= 59.449
  17:39:35 STEP 2836 H=2836:00 +CPU= 37.316
  17:40:13 STEP 2837 H=2837:00 +CPU= 37.114
  17:41:17 STEP 2838 H=2838:00 +CPU= 63.737
  17:41:59 STEP 2839 H=2839:00 +CPU= 41.486
  17:42:39 STEP 2840 H=2840:00 +CPU= 38.637
  17:43:42 STEP 2841 H=2841:00 +CPU= 62.077
  17:44:22 STEP 2842 H=2842:00 +CPU= 39.887
  17:45:01 STEP 2843 H=2843:00 +CPU= 38.501
  17:46:05 STEP 2844 H=2844:00 +CPU= 63.354
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+002844
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+002844
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+002844
  17:46:46 STEP 2845 H=2845:00 +CPU= 40.253
  17:47:28 STEP 2846 H=2846:00 +CPU= 41.723
  17:48:31 STEP 2847 H=2847:00 +CPU= 61.451
  17:49:11 STEP 2848 H=2848:00 +CPU= 39.670
  17:49:48 STEP 2849 H=2849:00 +CPU= 36.844
  17:50:49 STEP 2850 H=2850:00 +CPU= 60.025
  17:51:26 STEP 2851 H=2851:00 +CPU= 36.094
  17:52:08 STEP 2852 H=2852:00 +CPU= 42.053
  17:53:16 STEP 2853 H=2853:00 +CPU= 66.533
  17:53:56 STEP 2854 H=2854:00 +CPU= 39.761
  17:54:35 STEP 2855 H=2855:00 +CPU= 38.077
  17:55:47 STEP 2856 H=2856:00 +CPU= 70.938
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+002856
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+002856
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+002856
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMGGhq0f+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMSHhq0f+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMUAhq0f+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMGGhq0f+002844
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMSHhq0f+002844
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187229/ICMUAhq0f+002844
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0585_2001050100_123_970_12187229_0_r1050872504_118.zip
Uploading the intermediate file: upload_file_118.zip
  17:56:25 STEP 2857 H=2857:00 +CPU= 38.027
  17:57:04 STEP 2858 H=2858:00 +CPU= 38.132
  17:58:07 STEP 2859 H=2859:00 +CPU= 62.286
  17:58:47 STEP 2860 H=2860:00 +CPU= 39.478
  17:59:26 STEP 2861 H=2861:00 +CPU= 39.040
  18:00:27 STEP 2862 H=2862:00 +CPU= 59.628
  18:01:06 STEP 2863 H=2863:00 +CPU= 38.375
  18:01:48 STEP 2864 H=2864:00 +CPU= 42.051
  18:02:51 STEP 2865 H=2865:00 +CPU= 61.857
  18:03:34 STEP 2866 H=2866:00 +CPU= 42.334
  18:04:24 STEP 2867 H=2867:00 +CPU= 48.495
  18:05:42 STEP 2868 H=2868:00 +CPU= 76.404
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+002868
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+002868
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+002868
  18:06:26 STEP 2869 H=2869:00 +CPU= 43.407
  18:07:11 STEP 2870 H=2870:00 +CPU= 44.056
  18:08:18 STEP 2871 H=2871:00 +CPU= 65.548
  18:09:07 STEP 2872 H=2872:00 +CPU= 48.971
  18:09:51 STEP 2873 H=2873:00 +CPU= 43.429
  18:11:03 STEP 2874 H=2874:00 +CPU= 70.857
  18:11:52 STEP 2875 H=2875:00 +CPU= 47.745
  18:12:43 STEP 2876 H=2876:00 +CPU= 49.997
  18:13:58 STEP 2877 H=2877:00 +CPU= 73.833
  18:14:49 STEP 2878 H=2878:00 +CPU= 49.443
  18:15:45 STEP 2879 H=2879:00 +CPU= 54.933
Abort request received from BOINC client, ending the child process

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:29:48 1520317 22272643 oifs_43r3_ps_0585_2001050100_123_970_12187229_0 9,590,400 56,140 0.0059
09 May 2023 16:24:46 1520317 22272643 oifs_43r3_ps_0585_2001050100_123_970_12187229_0 6,998,400 40,086 0.0057
09 May 2023 16:24:32 1520317 22272643 oifs_43r3_ps_0585_2001050100_123_970_12187229_0 5,529,600 31,536 0.0057
09 May 2023 16:24:21 1520317 22272643 oifs_43r3_ps_0585_2001050100_123_970_12187229_0 4,320,000 24,350 0.0056
09 May 2023 16:24:11 1520317 22272643 oifs_43r3_ps_0585_2001050100_123_970_12187229_0 3,456,000 19,004 0.0055
09 May 2023 16:23:54 1520317 22272643 oifs_43r3_ps_0585_2001050100_123_970_12187229_0 2,592,000 14,174 0.0055
09 May 2023 16:23:49 1520317 22272643 oifs_43r3_ps_0585_2001050100_123_970_12187229_0 1,728,000 9,396 0.0054
09 May 2023 16:23:40 1520317 22272643 oifs_43r3_ps_0585_2001050100_123_970_12187229_0 864,000 4,738 0.0055


©2024 climateprediction.net