climateprediction.net home page
Task 22256673

Task 22256673

Name oifs_43r3_ps_0581_1986050100_123_955_12172225_0
Workunit 12172225
Created 21 Dec 2022, 9:00:39 UTC
Sent 21 Dec 2022, 9:08:16 UTC
Report deadline 20 Jan 2023, 9:08:16 UTC
Received 21 Dec 2022, 23:47:58 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1529888
Run time 13 hours 52 min 12 sec
CPU time 13 hours 42 min 49 sec
Validate state Invalid
Credit 8,588.16
Device peak FLOPS 5.49 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,413.77 MB
Peak swap size 4,978.16 MB
Peak disk usage 7,674.19 MB
Stderr 
<core_client_version>7.16.6</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
 STEP 1402 H=1402:00 +CPU= 21.478
  21:19:46 STEP 1403 H=1403:00 +CPU= 21.325
  21:20:23 STEP 1404 H=1404:00 +CPU= 36.167
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+001404
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+001404
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+001404
  21:20:39 STEP 1405 H=1405:00 +CPU= 15.667
  21:20:54 STEP 1406 H=1406:00 +CPU= 15.741
  21:21:26 STEP 1407 H=1407:00 +CPU= 31.440
  21:21:51 STEP 1408 H=1408:00 +CPU= 24.448
  21:22:14 STEP 1409 H=1409:00 +CPU= 23.119
  21:22:51 STEP 1410 H=1410:00 +CPU= 35.860
  21:23:15 STEP 1411 H=1411:00 +CPU= 23.823
  21:23:40 STEP 1412 H=1412:00 +CPU= 24.438
  21:24:21 STEP 1413 H=1413:00 +CPU= 40.520
  21:24:44 STEP 1414 H=1414:00 +CPU= 23.128
  21:25:08 STEP 1415 H=1415:00 +CPU= 23.345
  21:25:51 STEP 1416 H=1416:00 +CPU= 42.864
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+001416
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+001416
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172225/ICMGGhq0f+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172225/ICMSHhq0f+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172225/ICMUAhq0f+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172225/ICMGGhq0f+001404
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172225/ICMSHhq0f+001404
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172225/ICMUAhq0f+001404
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0581_1986050100_123_955_12172225_0_r1551694089_58.zip
Uploading the intermediate file: upload_file_58.zip
  21:26:13 STEP 1417 H=1417:00 +CPU= 21.860
Uploading trickle at timestep: 5097600
  21:26:37 STEP 1418 H=1418:00 +CPU= 24.347
  21:27:14 STEP 1419 H=1419:00 +CPU= 36.298
  21:27:39 STEP 1420 H=1420:00 +CPU= 25.432
  21:28:04 STEP 1421 H=1421:00 +CPU= 23.961
  21:28:37 STEP 1422 H=1422:00 +CPU= 33.663
  21:28:58 STEP 1423 H=1423:00 +CPU= 20.228
  21:29:23 STEP 1424 H=1424:00 +CPU= 25.307
  21:30:02 STEP 1425 H=1425:00 +CPU= 37.845
  21:30:28 STEP 1426 H=1426:00 +CPU= 26.506
  21:30:57 STEP 1427 H=1427:00 +CPU= 28.148
  21:31:35 STEP 1428 H=1428:00 +CPU= 37.985
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+001428
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+001428
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+001428
  21:31:58 STEP 1429 H=1429:00 +CPU= 22.491
  21:32:22 STEP 1430 H=1430:00 +CPU= 24.196
  21:32:53 STEP 1431 H=1431:00 +CPU= 30.718
  21:33:15 STEP 1432 H=1432:00 +CPU= 21.162
  21:33:38 STEP 1433 H=1433:00 +CPU= 23.342
  21:34:09 STEP 1434 H=1434:00 +CPU= 30.591
  21:34:31 STEP 1435 H=1435:00 +CPU= 22.233
  21:35:00 STEP 1436 H=1436:00 +CPU= 27.488
  21:35:37 STEP 1437 H=1437:00 +CPU= 36.696
  21:36:04 STEP 1438 H=1438:00 +CPU= 26.415
  21:36:30 STEP 1439 H=1439:00 +CPU= 26.207
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1986050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0581
(argv4) batchid: 955
(argv5) wuid: 12172225
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/9
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172225
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/9/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/9/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_5707a887078f9b57e254496de83113d5 to: /var/lib/boinc-client/slots/9/oifs_43r3_ps_0581_1986050100_123_955_12172225.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/9/oifs_43r3_ps_0581_1986050100_123_955_12172225.zip
ic_ancil_file: ic_ancil_12172225
ifsdata_file: ifsdata_12172225
climate_data_file: clim_data_12172225
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_37ad26ed28d3e1ad957325171ede5257 to: /var/lib/boinc-client/slots/9/ic_ancil_12172225.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/9/ic_ancil_12172225.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_85fca5dfb32442dd43ec173530b1d5fa to: /var/lib/boinc-client/slots/9/ifsdata/ifsdata_12172225.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/9/ifsdata/ifsdata_12172225.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/9/159l_2/clim_data_12172225.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/9/159l_2/clim_data_12172225.zip
Checking for progress XML file: /var/lib/boinc-client/slots/9/progress_file_12172225.xml
Opened progress file ok : /var/lib/boinc-client/slots/9/progress_file_12172225.xml
-- Model is restarting --
Adjusting last_iter, 1440, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/9/progress_file_12172225.xml
last_cpu_time: 40412
upload_file_number: 59
last_iter: 1439
last_upload: 5097600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0581_1986050100_123_955_12172225_0_r1551694089
The child process has been launched with process id: 109024
Executing the command: /var/lib/boinc-client/slots/9/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:Stargazer:1:1:109024:109024] [20221221:213811:1671655091:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:Stargazer:1:1:109024:109024] [20221221:213811:1671655091:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:Stargazer:1:1:109024:109024] [20221221:213811:1671655091:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:Stargazer:1:1:109024:109024] [20221221:213811:1671655091:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:Stargazer:1:1:109024:109024] [20221221:213811:1671655091:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:Stargazer:1:1:109024:109024] [20221221:213811:1671655091:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:Stargazer:1:1:109024:109024] [20221221:213811:1671655091:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:Stargazer:1:1:109024:109024] [20221221:213811:1671655091:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:Stargazer:1:1:109024:109024] [20221221:213811:1671655091:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:Stargazer:1:1:109024:109024] [20221221:213811:1671655091:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:Stargazer:1:1:109024:109024] [20221221:213811:1671655091:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:Stargazer:1:1:109024:109024] [20221221:213811:1671655091:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:Stargazer:1:1:109024:109024] [20221221:213811:1671655091:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:Stargazer:1:1:109024:109024] [20221221:213811:1671655091:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:Stargazer:1:1:109024:109024] [20221221:213811:1671655091:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:Stargazer:1:1:109024:109024] [20221221:213811:1671655091:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Stargazer:1:1:109024:109024] [20221221:213811:1671655091:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Stargazer:1:1:109024:109024] [20221221:213811:1671655091:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Stargazer:1:1:109024:109024] [20221221:213811:1671655091:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:Stargazer:1:1:109024:109024] [20221221:213811:1671655091:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:Stargazer:1:1:109024:109024] [20221221:213811:1671655091:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Stargazer:1:1:109024:109024] [20221221:213811:1671655091:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Stargazer:1:1:109024:109024] [20221221:213811:1671655091:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:Stargazer:1:1:109024:109024] [20221221:213811:1671655091:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Stargazer:1:1:109024:109024] [20221221:213811:1671655091:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Stargazer:1:1:109024:109024] [20221221:213811:1671655091:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Stargazer:1:1:109024:109024] [20221221:213811:1671655091:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:Stargazer:1:1:109024:109024] [20221221:213811:1671655091:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:Stargazer:1:1:109024:109024] [20221221:213811:1671655091:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Stargazer:1:1:109024:109024] [20221221:213811:1671655091:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  21:39:03 STEP 1440 H=1440:00 +CPU= 47.830
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+001440
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+001440
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172225/ICMGGhq0f+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172225/ICMSHhq0f+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172225/ICMUAhq0f+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172225/ICMGGhq0f+001428
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172225/ICMSHhq0f+001428
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172225/ICMUAhq0f+001428
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0581_1986050100_123_955_12172225_0_r1551694089_59.zip
Uploading the intermediate file: upload_file_59.zip
  21:39:24 STEP 1441 H=1441:00 +CPU= 21.308
  21:39:54 STEP 1442 H=1442:00 +CPU= 29.457
  21:40:32 STEP 1443 H=1443:00 +CPU= 37.827
  21:40:55 STEP 1444 H=1444:00 +CPU= 23.493
  21:41:15 STEP 1445 H=1445:00 +CPU= 19.910
  21:41:48 STEP 1446 H=1446:00 +CPU= 32.064
  21:42:12 STEP 1447 H=1447:00 +CPU= 24.103
  21:42:31 STEP 1448 H=1448:00 +CPU= 18.911
  21:42:58 STEP 1449 H=1449:00 +CPU= 26.134
  21:43:15 STEP 1450 H=1450:00 +CPU= 17.545
  21:43:42 STEP 1451 H=1451:00 +CPU= 25.844
  21:44:21 STEP 1452 H=1452:00 +CPU= 38.898
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+001452
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+001452
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+001452
  21:44:45 STEP 1453 H=1453:00 +CPU= 24.075
  21:45:07 STEP 1454 H=1454:00 +CPU= 22.243
  21:45:39 STEP 1455 H=1455:00 +CPU= 31.430
  21:46:02 STEP 1456 H=1456:00 +CPU= 23.090
  21:46:27 STEP 1457 H=1457:00 +CPU= 24.818
  21:47:05 STEP 1458 H=1458:00 +CPU= 36.957
  21:47:26 STEP 1459 H=1459:00 +CPU= 21.452
  21:47:48 STEP 1460 H=1460:00 +CPU= 21.199
  21:48:23 STEP 1461 H=1461:00 +CPU= 35.256
  21:48:49 STEP 1462 H=1462:00 +CPU= 24.622
  21:49:13 STEP 1463 H=1463:00 +CPU= 23.837
  21:49:54 STEP 1464 H=1464:00 +CPU= 40.761
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+001464
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+001464
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+001464
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172225/ICMGGhq0f+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172225/ICMSHhq0f+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172225/ICMUAhq0f+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172225/ICMGGhq0f+001452
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172225/ICMSHhq0f+001452
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172225/ICMUAhq0f+001452
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0581_1986050100_123_955_12172225_0_r1551694089_60.zip
Uploading the intermediate file: upload_file_60.zip
  21:50:17 STEP 1465 H=1465:00 +CPU= 22.765
  21:50:37 STEP 1466 H=1466:00 +CPU= 20.353
  21:51:10 STEP 1467 H=1467:00 +CPU= 32.454
  21:51:34 STEP 1468 H=1468:00 +CPU= 23.482
  21:51:56 STEP 1469 H=1469:00 +CPU= 21.571
  21:52:31 STEP 1470 H=1470:00 +CPU= 35.255
  21:52:57 STEP 1471 H=1471:00 +CPU= 25.278
  21:53:18 STEP 1472 H=1472:00 +CPU= 21.249
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1986050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0581
(argv4) batchid: 955
(argv5) wuid: 12172225
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/9
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172225
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/9/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/9/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_5707a887078f9b57e254496de83113d5 to: /var/lib/boinc-client/slots/9/oifs_43r3_ps_0581_1986050100_123_955_12172225.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/9/oifs_43r3_ps_0581_1986050100_123_955_12172225.zip
ic_ancil_file: ic_ancil_12172225
ifsdata_file: ifsdata_12172225
climate_data_file: clim_data_12172225
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_37ad26ed28d3e1ad957325171ede5257 to: /var/lib/boinc-client/slots/9/ic_ancil_12172225.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/9/ic_ancil_12172225.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_85fca5dfb32442dd43ec173530b1d5fa to: /var/lib/boinc-client/slots/9/ifsdata/ifsdata_12172225.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/9/ifsdata/ifsdata_12172225.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/9/159l_2/clim_data_12172225.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/9/159l_2/clim_data_12172225.zip
Checking for progress XML file: /var/lib/boinc-client/slots/9/progress_file_12172225.xml
Opened progress file ok : /var/lib/boinc-client/slots/9/progress_file_12172225.xml
-- Model is restarting --
Adjusting last_iter, 1473, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/9/progress_file_12172225.xml
last_cpu_time: 41317
upload_file_number: 61
last_iter: 1463
last_upload: 5270400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0581_1986050100_123_955_12172225_0_r1551694089
The child process has been launched with process id: 109203
Executing the command: /var/lib/boinc-client/slots/9/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:Stargazer:1:1:109203:109203] [20221221:215617:1671656177:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:Stargazer:1:1:109203:109203] [20221221:215617:1671656177:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:Stargazer:1:1:109203:109203] [20221221:215617:1671656177:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:Stargazer:1:1:109203:109203] [20221221:215617:1671656177:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:Stargazer:1:1:109203:109203] [20221221:215617:1671656177:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:Stargazer:1:1:109203:109203] [20221221:215617:1671656177:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:Stargazer:1:1:109203:109203] [20221221:215617:1671656177:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:Stargazer:1:1:109203:109203] [20221221:215617:1671656177:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:Stargazer:1:1:109203:109203] [20221221:215617:1671656177:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:Stargazer:1:1:109203:109203] [20221221:215617:1671656177:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:Stargazer:1:1:109203:109203] [20221221:215617:1671656177:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:Stargazer:1:1:109203:109203] [20221221:215617:1671656177:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:Stargazer:1:1:109203:109203] [20221221:215617:1671656177:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:Stargazer:1:1:109203:109203] [20221221:215617:1671656177:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:Stargazer:1:1:109203:109203] [20221221:215617:1671656177:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:Stargazer:1:1:109203:109203] [20221221:215617:1671656177:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Stargazer:1:1:109203:109203] [20221221:215617:1671656177:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Stargazer:1:1:109203:109203] [20221221:215617:1671656177:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Stargazer:1:1:109203:109203] [20221221:215617:1671656177:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:Stargazer:1:1:109203:109203] [20221221:215617:1671656177:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:Stargazer:1:1:109203:109203] [20221221:215617:1671656177:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Stargazer:1:1:109203:109203] [20221221:215617:1671656177:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Stargazer:1:1:109203:109203] [20221221:215617:1671656177:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:Stargazer:1:1:109203:109203] [20221221:215617:1671656177:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Stargazer:1:1:109203:109203] [20221221:215617:1671656177:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Stargazer:1:1:109203:109203] [20221221:215617:1671656177:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Stargazer:1:1:109203:109203] [20221221:215617:1671656177:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:Stargazer:1:1:109203:109203] [20221221:215617:1671656177:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:Stargazer:1:1:109203:109203] [20221221:215617:1671656177:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Stargazer:1:1:109203:109203] [20221221:215617:1671656177:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  21:57:01 STEP 1464 H=1464:00 +CPU= 40.672
  21:57:25 STEP 1465 H=1465:00 +CPU= 23.375
  21:57:50 STEP 1466 H=1466:00 +CPU= 24.886
  21:58:25 STEP 1467 H=1467:00 +CPU= 34.660
  21:58:51 STEP 1468 H=1468:00 +CPU= 25.905
  21:59:15 STEP 1469 H=1469:00 +CPU= 23.818
  21:59:47 STEP 1470 H=1470:00 +CPU= 32.119
  22:00:08 STEP 1471 H=1471:00 +CPU= 21.311
  22:00:32 STEP 1472 H=1472:00 +CPU= 22.990
  22:01:03 STEP 1473 H=1473:00 +CPU= 31.308
  22:01:24 STEP 1474 H=1474:00 +CPU= 20.634
  22:01:50 STEP 1475 H=1475:00 +CPU= 25.846
  22:02:27 STEP 1476 H=1476:00 +CPU= 36.035
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+001476
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+001476
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+001476
  22:02:49 STEP 1477 H=1477:00 +CPU= 22.356
  22:03:15 STEP 1478 H=1478:00 +CPU= 25.505
  22:03:46 STEP 1479 H=1479:00 +CPU= 31.484
  22:04:08 STEP 1480 H=1480:00 +CPU= 21.789
  22:04:32 STEP 1481 H=1481:00 +CPU= 23.496
  22:05:01 STEP 1482 H=1482:00 +CPU= 28.892
  22:05:16 STEP 1483 H=1483:00 +CPU= 14.633
  22:05:40 STEP 1484 H=1484:00 +CPU= 23.243
  22:06:15 STEP 1485 H=1485:00 +CPU= 34.789
  22:06:37 STEP 1486 H=1486:00 +CPU= 22.241
  22:07:07 STEP 1487 H=1487:00 +CPU= 28.460
  22:07:52 STEP 1488 H=1488:00 +CPU= 43.927
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+001488
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+001488
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+001488
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172225/ICMGGhq0f+001464
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172225/ICMSHhq0f+001464
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172225/ICMUAhq0f+001464
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172225/ICMGGhq0f+001476
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172225/ICMSHhq0f+001476
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172225/ICMUAhq0f+001476
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0581_1986050100_123_955_12172225_0_r1551694089_61.zip
Uploading the intermediate file: upload_file_61.zip
  22:08:14 STEP 1489 H=1489:00 +CPU= 22.178
  22:08:37 STEP 1490 H=1490:00 +CPU= 22.131
  22:09:13 STEP 1491 H=1491:00 +CPU= 35.644
  22:09:36 STEP 1492 H=1492:00 +CPU= 23.444
  22:10:01 STEP 1493 H=1493:00 +CPU= 24.986
  22:10:35 STEP 1494 H=1494:00 +CPU= 33.181
  22:10:56 STEP 1495 H=1495:00 +CPU= 20.593
  22:11:20 STEP 1496 H=1496:00 +CPU= 23.852
  22:11:57 STEP 1497 H=1497:00 +CPU= 36.861
  22:12:20 STEP 1498 H=1498:00 +CPU= 22.637
  22:12:45 STEP 1499 H=1499:00 +CPU= 24.305
  22:13:25 STEP 1500 H=1500:00 +CPU= 39.918
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+001500
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+001500
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+001500
  22:13:47 STEP 1501 H=1501:00 +CPU= 22.086
  22:14:08 STEP 1502 H=1502:00 +CPU= 20.202
  22:14:39 STEP 1503 H=1503:00 +CPU= 31.582
  22:15:03 STEP 1504 H=1504:00 +CPU= 23.687
  22:15:28 STEP 1505 H=1505:00 +CPU= 23.804
  22:16:02 STEP 1506 H=1506:00 +CPU= 34.040
  22:16:26 STEP 1507 H=1507:00 +CPU= 23.223
  22:16:47 STEP 1508 H=1508:00 +CPU= 21.154
  22:17:21 STEP 1509 H=1509:00 +CPU= 33.973
  22:17:41 STEP 1510 H=1510:00 +CPU= 19.985
  22:18:07 STEP 1511 H=1511:00 +CPU= 25.611
  22:18:50 STEP 1512 H=1512:00 +CPU= 42.560
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+001512
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+001512
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+001512
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172225/ICMGGhq0f+001488
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172225/ICMSHhq0f+001488
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172225/ICMUAhq0f+001488
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172225/ICMGGhq0f+001500
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172225/ICMSHhq0f+001500
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172225/ICMUAhq0f+001500
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0581_1986050100_123_955_12172225_0_r1551694089_62.zip
Uploading the intermediate file: upload_file_62.zip
  22:19:13 STEP 1513 H=1513:00 +CPU= 22.744
  22:19:42 STEP 1514 H=1514:00 +CPU= 27.754
  22:20:21 STEP 1515 H=1515:00 +CPU= 38.518
  22:20:48 STEP 1516 H=1516:00 +CPU= 26.450
  22:21:15 STEP 1517 H=1517:00 +CPU= 27.191
  22:21:54 STEP 1518 H=1518:00 +CPU= 38.918
  22:22:16 STEP 1519 H=1519:00 +CPU= 22.234
  22:22:42 STEP 1520 H=1520:00 +CPU= 25.840
  22:23:17 STEP 1521 H=1521:00 +CPU= 34.592
  22:23:42 STEP 1522 H=1522:00 +CPU= 24.440
  22:24:07 STEP 1523 H=1523:00 +CPU= 24.757
  22:24:47 STEP 1524 H=1524:00 +CPU= 39.351
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+001524
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+001524
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+001524
  22:25:09 STEP 1525 H=1525:00 +CPU= 22.300
  22:25:34 STEP 1526 H=1526:00 +CPU= 24.303
  22:26:05 STEP 1527 H=1527:00 +CPU= 30.561
  22:26:26 STEP 1528 H=1528:00 +CPU= 21.681
  22:26:53 STEP 1529 H=1529:00 +CPU= 26.517
  22:27:31 STEP 1530 H=1530:00 +CPU= 37.771
  22:27:51 STEP 1531 H=1531:00 +CPU= 19.858
  22:28:14 STEP 1532 H=1532:00 +CPU= 23.250
  22:28:47 STEP 1533 H=1533:00 +CPU= 32.384
  22:29:10 STEP 1534 H=1534:00 +CPU= 23.267
  22:29:31 STEP 1535 H=1535:00 +CPU= 20.880
  22:30:11 STEP 1536 H=1536:00 +CPU= 39.392
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+001536
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+001536
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172225/ICMGGhq0f+001512
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172225/ICMSHhq0f+001512
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172225/ICMUAhq0f+001512
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172225/ICMGGhq0f+001524
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172225/ICMSHhq0f+001524
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172225/ICMUAhq0f+001524
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0581_1986050100_123_955_12172225_0_r1551694089_63.zip
Uploading the intermediate file: upload_file_63.zip
  22:30:29 STEP 1537 H=1537:00 +CPU= 17.552
  22:30:47 STEP 1538 H=1538:00 +CPU= 17.967
  22:31:21 STEP 1539 H=1539:00 +CPU= 33.664
  22:31:42 STEP 1540 H=1540:00 +CPU= 21.365
  22:32:03 STEP 1541 H=1541:00 +CPU= 21.264
  22:32:38 STEP 1542 H=1542:00 +CPU= 34.392
  22:33:01 STEP 1543 H=1543:00 +CPU= 22.847
  22:33:26 STEP 1544 H=1544:00 +CPU= 24.814
  22:34:02 STEP 1545 H=1545:00 +CPU= 35.121
  22:34:24 STEP 1546 H=1546:00 +CPU= 22.206
  22:34:45 STEP 1547 H=1547:00 +CPU= 20.402
  22:35:29 STEP 1548 H=1548:00 +CPU= 43.091
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+001548
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+001548
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+001548
  22:35:52 STEP 1549 H=1549:00 +CPU= 22.875
  22:36:17 STEP 1550 H=1550:00 +CPU= 25.012
  22:36:52 STEP 1551 H=1551:00 +CPU= 34.698
  22:37:12 STEP 1552 H=1552:00 +CPU= 20.277
  22:37:37 STEP 1553 H=1553:00 +CPU= 24.132
  22:38:13 STEP 1554 H=1554:00 +CPU= 36.357
  22:38:35 STEP 1555 H=1555:00 +CPU= 21.569
  22:38:58 STEP 1556 H=1556:00 +CPU= 23.473
  22:39:32 STEP 1557 H=1557:00 +CPU= 33.043
  22:39:53 STEP 1558 H=1558:00 +CPU= 21.212
  22:40:18 STEP 1559 H=1559:00 +CPU= 24.478
  22:40:59 STEP 1560 H=1560:00 +CPU= 41.020
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+001560
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+001560
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172225/ICMGGhq0f+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172225/ICMSHhq0f+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172225/ICMUAhq0f+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172225/ICMGGhq0f+001548
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172225/ICMSHhq0f+001548
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172225/ICMUAhq0f+001548
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0581_1986050100_123_955_12172225_0_r1551694089_64.zip
Uploading the intermediate file: upload_file_64.zip
  22:41:19 STEP 1561 H=1561:00 +CPU= 19.832
  22:41:43 STEP 1562 H=1562:00 +CPU= 23.475
  22:42:21 STEP 1563 H=1563:00 +CPU= 37.243
  22:42:44 STEP 1564 H=1564:00 +CPU= 22.851
  22:43:07 STEP 1565 H=1565:00 +CPU= 22.354
  22:43:44 STEP 1566 H=1566:00 +CPU= 36.798
  22:44:04 STEP 1567 H=1567:00 +CPU= 20.103
  22:44:28 STEP 1568 H=1568:00 +CPU= 23.436
  22:45:05 STEP 1569 H=1569:00 +CPU= 36.064
  22:45:26 STEP 1570 H=1570:00 +CPU= 20.571
  22:45:47 STEP 1571 H=1571:00 +CPU= 21.252
  22:46:28 STEP 1572 H=1572:00 +CPU= 40.425
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+001572
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+001572
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+001572
  22:46:49 STEP 1573 H=1573:00 +CPU= 21.005
  22:47:13 STEP 1574 H=1574:00 +CPU= 23.776
  22:47:43 STEP 1575 H=1575:00 +CPU= 30.202
  22:48:06 STEP 1576 H=1576:00 +CPU= 22.883
  22:48:28 STEP 1577 H=1577:00 +CPU= 21.312
  22:49:07 STEP 1578 H=1578:00 +CPU= 38.195
  22:49:32 STEP 1579 H=1579:00 +CPU= 25.213
  22:49:58 STEP 1580 H=1580:00 +CPU= 25.716
  22:50:31 STEP 1581 H=1581:00 +CPU= 32.681
  22:50:55 STEP 1582 H=1582:00 +CPU= 23.939
  22:51:20 STEP 1583 H=1583:00 +CPU= 24.714
  22:52:02 STEP 1584 H=1584:00 +CPU= 41.516
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+001584
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+001584
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172225/ICMGGhq0f+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172225/ICMSHhq0f+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172225/ICMUAhq0f+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172225/ICMGGhq0f+001572
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172225/ICMSHhq0f+001572
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172225/ICMUAhq0f+001572
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0581_1986050100_123_955_12172225_0_r1551694089_65.zip
Uploading the intermediate file: upload_file_65.zip
  22:52:25 STEP 1585 H=1585:00 +CPU= 22.607
  22:52:49 STEP 1586 H=1586:00 +CPU= 23.116
  22:53:21 STEP 1587 H=1587:00 +CPU= 32.610
  22:53:46 STEP 1588 H=1588:00 +CPU= 24.322
  22:54:10 STEP 1589 H=1589:00 +CPU= 24.256
  22:54:47 STEP 1590 H=1590:00 +CPU= 36.214
  22:55:08 STEP 1591 H=1591:00 +CPU= 20.874
  22:55:31 STEP 1592 H=1592:00 +CPU= 23.731
  22:56:07 STEP 1593 H=1593:00 +CPU= 35.029
  22:56:29 STEP 1594 H=1594:00 +CPU= 21.476
  22:56:53 STEP 1595 H=1595:00 +CPU= 24.061
  22:57:33 STEP 1596 H=1596:00 +CPU= 39.987
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+001596
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+001596
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+001596
  22:57:53 STEP 1597 H=1597:00 +CPU= 20.158
  22:58:16 STEP 1598 H=1598:00 +CPU= 22.701
  22:58:52 STEP 1599 H=1599:00 +CPU= 35.256
  22:59:10 STEP 1600 H=1600:00 +CPU= 18.067
  22:59:30 STEP 1601 H=1601:00 +CPU= 19.277
  23:00:03 STEP 1602 H=1602:00 +CPU= 33.624
  23:00:24 STEP 1603 H=1603:00 +CPU= 20.122
  23:00:45 STEP 1604 H=1604:00 +CPU= 20.738
  23:01:24 STEP 1605 H=1605:00 +CPU= 38.519
  23:01:48 STEP 1606 H=1606:00 +CPU= 24.074
  23:02:13 STEP 1607 H=1607:00 +CPU= 24.688
  23:02:56 STEP 1608 H=1608:00 +CPU= 42.576
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+001608
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+001608
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172225/ICMGGhq0f+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172225/ICMSHhq0f+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172225/ICMUAhq0f+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172225/ICMGGhq0f+001596
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172225/ICMSHhq0f+001596
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172225/ICMUAhq0f+001596
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0581_1986050100_123_955_12172225_0_r1551694089_66.zip
Uploading the intermediate file: upload_file_66.zip
  23:03:17 STEP 1609 H=1609:00 +CPU= 21.310
  23:03:45 STEP 1610 H=1610:00 +CPU= 26.815
  23:04:19 STEP 1611 H=1611:00 +CPU= 33.674
  23:04:40 STEP 1612 H=1612:00 +CPU= 20.410
  23:05:01 STEP 1613 H=1613:00 +CPU= 21.189
  23:05:39 STEP 1614 H=1614:00 +CPU= 36.545
  23:06:00 STEP 1615 H=1615:00 +CPU= 21.135
  23:06:22 STEP 1616 H=1616:00 +CPU= 21.812
  23:07:00 STEP 1617 H=1617:00 +CPU= 37.072
  23:07:24 STEP 1618 H=1618:00 +CPU= 23.826
  23:07:46 STEP 1619 H=1619:00 +CPU= 22.377
  23:08:22 STEP 1620 H=1620:00 +CPU= 35.518
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+001620
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+001620
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+001620
  23:08:41 STEP 1621 H=1621:00 +CPU= 18.902
  23:09:04 STEP 1622 H=1622:00 +CPU= 23.024
  23:09:39 STEP 1623 H=1623:00 +CPU= 33.932
  23:10:03 STEP 1624 H=1624:00 +CPU= 23.896
  23:10:28 STEP 1625 H=1625:00 +CPU= 24.580
  23:11:06 STEP 1626 H=1626:00 +CPU= 37.674
  23:11:24 STEP 1627 H=1627:00 +CPU= 17.897
  23:11:44 STEP 1628 H=1628:00 +CPU= 19.544
  23:12:17 STEP 1629 H=1629:00 +CPU= 32.659
  23:12:38 STEP 1630 H=1630:00 +CPU= 21.243
  23:12:59 STEP 1631 H=1631:00 +CPU= 21.120
  23:13:42 STEP 1632 H=1632:00 +CPU= 42.242
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+001632
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+001632
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172225/ICMGGhq0f+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172225/ICMSHhq0f+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172225/ICMUAhq0f+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172225/ICMGGhq0f+001620
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172225/ICMSHhq0f+001620
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172225/ICMUAhq0f+001620
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0581_1986050100_123_955_12172225_0_r1551694089_67.zip
Uploading the intermediate file: upload_file_67.zip
  23:14:04 STEP 1633 H=1633:00 +CPU= 21.678
  23:14:25 STEP 1634 H=1634:00 +CPU= 20.453
  23:15:01 STEP 1635 H=1635:00 +CPU= 36.088
  23:15:29 STEP 1636 H=1636:00 +CPU= 27.448
  23:15:54 STEP 1637 H=1637:00 +CPU= 23.978
  23:16:27 STEP 1638 H=1638:00 +CPU= 32.738
  23:16:50 STEP 1639 H=1639:00 +CPU= 22.669
  23:17:14 STEP 1640 H=1640:00 +CPU= 23.772
  23:17:47 STEP 1641 H=1641:00 +CPU= 32.390
  23:18:10 STEP 1642 H=1642:00 +CPU= 22.639
  23:18:34 STEP 1643 H=1643:00 +CPU= 23.480
  23:19:17 STEP 1644 H=1644:00 +CPU= 43.082
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+001644
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+001644
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+001644
  23:19:40 STEP 1645 H=1645:00 +CPU= 23.104
  23:20:03 STEP 1646 H=1646:00 +CPU= 22.610
  23:20:38 STEP 1647 H=1647:00 +CPU= 34.583
  23:21:02 STEP 1648 H=1648:00 +CPU= 24.055
  23:21:24 STEP 1649 H=1649:00 +CPU= 21.921
  23:22:00 STEP 1650 H=1650:00 +CPU= 34.909
  23:22:22 STEP 1651 H=1651:00 +CPU= 22.220
  23:22:48 STEP 1652 H=1652:00 +CPU= 26.056
  23:23:25 STEP 1653 H=1653:00 +CPU= 35.591
  23:23:47 STEP 1654 H=1654:00 +CPU= 22.358
  23:24:12 STEP 1655 H=1655:00 +CPU= 24.779
  23:24:54 STEP 1656 H=1656:00 +CPU= 41.133
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+001656
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+001656
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172225/ICMGGhq0f+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172225/ICMSHhq0f+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172225/ICMUAhq0f+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172225/ICMGGhq0f+001644
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172225/ICMSHhq0f+001644
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172225/ICMUAhq0f+001644
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0581_1986050100_123_955_12172225_0_r1551694089_68.zip
Uploading the intermediate file: upload_file_68.zip
  23:25:14 STEP 1657 H=1657:00 +CPU= 19.702
  23:25:40 STEP 1658 H=1658:00 +CPU= 25.606
  23:26:14 STEP 1659 H=1659:00 +CPU= 33.742
  23:26:35 STEP 1660 H=1660:00 +CPU= 21.006
  23:27:00 STEP 1661 H=1661:00 +CPU= 25.214
  23:27:40 STEP 1662 H=1662:00 +CPU= 38.692
  23:28:02 STEP 1663 H=1663:00 +CPU= 22.680
  23:28:28 STEP 1664 H=1664:00 +CPU= 24.512
  23:29:06 STEP 1665 H=1665:00 +CPU= 37.370
  23:29:28 STEP 1666 H=1666:00 +CPU= 22.010
  23:29:54 STEP 1667 H=1667:00 +CPU= 25.387
  23:30:33 STEP 1668 H=1668:00 +CPU= 38.932
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+001668
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+001668
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+001668
  23:30:54 STEP 1669 H=1669:00 +CPU= 20.657
  23:31:13 STEP 1670 H=1670:00 +CPU= 18.078
  23:31:39 STEP 1671 H=1671:00 +CPU= 26.793
  23:31:55 STEP 1672 H=1672:00 +CPU= 15.838
  23:32:17 STEP 1673 H=1673:00 +CPU= 21.304
  23:32:51 STEP 1674 H=1674:00 +CPU= 34.036
  23:33:14 STEP 1675 H=1675:00 +CPU= 22.702
  23:33:39 STEP 1676 H=1676:00 +CPU= 24.219
  23:34:12 STEP 1677 H=1677:00 +CPU= 32.311
  23:34:33 STEP 1678 H=1678:00 +CPU= 20.533
  23:34:55 STEP 1679 H=1679:00 +CPU= 22.133
  23:35:37 STEP 1680 H=1680:00 +CPU= 42.157
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+001680
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+001680
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172225/ICMGGhq0f+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172225/ICMSHhq0f+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172225/ICMUAhq0f+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172225/ICMGGhq0f+001668
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172225/ICMSHhq0f+001668
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172225/ICMUAhq0f+001668
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0581_1986050100_123_955_12172225_0_r1551694089_69.zip
Uploading the intermediate file: upload_file_69.zip
  23:36:00 STEP 1681 H=1681:00 +CPU= 22.523
  23:36:24 STEP 1682 H=1682:00 +CPU= 23.330
  23:36:59 STEP 1683 H=1683:00 +CPU= 34.630
  23:37:23 STEP 1684 H=1684:00 +CPU= 23.901
  23:37:46 STEP 1685 H=1685:00 +CPU= 23.310
  23:38:17 STEP 1686 H=1686:00 +CPU= 30.758
  23:38:35 STEP 1687 H=1687:00 +CPU= 17.105
  23:38:51 STEP 1688 H=1688:00 +CPU= 16.232
  23:39:31 STEP 1689 H=1689:00 +CPU= 38.596
  23:39:54 STEP 1690 H=1690:00 +CPU= 23.501
  23:40:16 STEP 1691 H=1691:00 +CPU= 21.997
  23:40:53 STEP 1692 H=1692:00 +CPU= 36.298
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+001692
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+001692
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+001692
  23:41:16 STEP 1693 H=1693:00 +CPU= 22.230
  23:41:38 STEP 1694 H=1694:00 +CPU= 22.069
  23:42:08 STEP 1695 H=1695:00 +CPU= 29.566
  23:42:28 STEP 1696 H=1696:00 +CPU= 20.093
  23:42:51 STEP 1697 H=1697:00 +CPU= 22.304
  23:43:33 STEP 1698 H=1698:00 +CPU= 41.664
  23:44:01 STEP 1699 H=1699:00 +CPU= 27.737
  23:44:30 STEP 1700 H=1700:00 +CPU= 28.708
  23:45:12 STEP 1701 H=1701:00 +CPU= 42.357
  23:45:40 STEP 1702 H=1702:00 +CPU= 27.765
  23:46:02 STEP 1703 H=1703:00 +CPU= 21.836
  23:46:45 STEP 1704 H=1704:00 +CPU= 42.028
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+001704
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+001704
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172225/ICMGGhq0f+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172225/ICMSHhq0f+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172225/ICMUAhq0f+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172225/ICMGGhq0f+001692
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172225/ICMSHhq0f+001692
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172225/ICMUAhq0f+001692
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0581_1986050100_123_955_12172225_0_r1551694089_70.zip
Uploading the intermediate file: upload_file_70.zip
  23:47:09 STEP 1705 H=1705:00 +CPU= 24.312
Uploading trickle at timestep: 6134400
  23:47:29 STEP 1706 H=1706:00 +CPU= 20.137
  23:48:02 STEP 1707 H=1707:00 +CPU= 32.538
  23:48:26 STEP 1708 H=1708:00 +CPU= 23.729
  23:48:47 STEP 1709 H=1709:00 +CPU= 20.404
  23:49:19 STEP 1710 H=1710:00 +CPU= 32.220
  23:49:42 STEP 1711 H=1711:00 +CPU= 23.052
  23:50:04 STEP 1712 H=1712:00 +CPU= 21.665
  23:50:31 STEP 1713 H=1713:00 +CPU= 27.001
  23:50:55 STEP 1714 H=1714:00 +CPU= 23.700
  23:51:15 STEP 1715 H=1715:00 +CPU= 20.313
  23:51:46 STEP 1716 H=1716:00 +CPU= 30.481
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+001716
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+001716
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+001716
  23:52:11 STEP 1717 H=1717:00 +CPU= 24.263
  23:52:35 STEP 1718 H=1718:00 +CPU= 23.687
  23:53:07 STEP 1719 H=1719:00 +CPU= 32.184
  23:53:30 STEP 1720 H=1720:00 +CPU= 22.230
  23:53:53 STEP 1721 H=1721:00 +CPU= 23.376
  23:54:28 STEP 1722 H=1722:00 +CPU= 34.244
  23:54:52 STEP 1723 H=1723:00 +CPU= 24.073
  23:55:15 STEP 1724 H=1724:00 +CPU= 23.030
  23:55:46 STEP 1725 H=1725:00 +CPU= 30.629
  23:56:07 STEP 1726 H=1726:00 +CPU= 21.160
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1986050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0581
(argv4) batchid: 955
(argv5) wuid: 12172225
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/9
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172225
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/9/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/9/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_5707a887078f9b57e254496de83113d5 to: /var/lib/boinc-client/slots/9/oifs_43r3_ps_0581_1986050100_123_955_12172225.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/9/oifs_43r3_ps_0581_1986050100_123_955_12172225.zip
ic_ancil_file: ic_ancil_12172225
ifsdata_file: ifsdata_12172225
climate_data_file: clim_data_12172225
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_37ad26ed28d3e1ad957325171ede5257 to: /var/lib/boinc-client/slots/9/ic_ancil_12172225.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/9/ic_ancil_12172225.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_85fca5dfb32442dd43ec173530b1d5fa to: /var/lib/boinc-client/slots/9/ifsdata/ifsdata_12172225.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/9/ifsdata/ifsdata_12172225.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/9/159l_2/clim_data_12172225.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/9/159l_2/clim_data_12172225.zip
Checking for progress XML file: /var/lib/boinc-client/slots/9/progress_file_12172225.xml
Opened progress file ok : /var/lib/boinc-client/slots/9/progress_file_12172225.xml
-- Model is restarting --
Adjusting last_iter, 1727, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/9/progress_file_12172225.xml
last_cpu_time: 48442
upload_file_number: 71
last_iter: 1703
last_upload: 6134400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0581_1986050100_123_955_12172225_0_r1551694089
The child process has been launched with process id: 111355
Executing the command: /var/lib/boinc-client/slots/9/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:Stargazer:1:1:111355:111355] [20221221:235746:1671663466:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:Stargazer:1:1:111355:111355] [20221221:235746:1671663466:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:Stargazer:1:1:111355:111355] [20221221:235746:1671663466:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:Stargazer:1:1:111355:111355] [20221221:235746:1671663466:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:Stargazer:1:1:111355:111355] [20221221:235746:1671663466:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:Stargazer:1:1:111355:111355] [20221221:235746:1671663466:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:Stargazer:1:1:111355:111355] [20221221:235746:1671663466:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:Stargazer:1:1:111355:111355] [20221221:235746:1671663466:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:Stargazer:1:1:111355:111355] [20221221:235746:1671663466:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:Stargazer:1:1:111355:111355] [20221221:235746:1671663466:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:Stargazer:1:1:111355:111355] [20221221:235746:1671663466:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:Stargazer:1:1:111355:111355] [20221221:235746:1671663466:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:Stargazer:1:1:111355:111355] [20221221:235746:1671663466:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:Stargazer:1:1:111355:111355] [20221221:235746:1671663466:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:Stargazer:1:1:111355:111355] [20221221:235746:1671663466:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:Stargazer:1:1:111355:111355] [20221221:235746:1671663466:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Stargazer:1:1:111355:111355] [20221221:235746:1671663466:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Stargazer:1:1:111355:111355] [20221221:235746:1671663466:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Stargazer:1:1:111355:111355] [20221221:235746:1671663466:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:Stargazer:1:1:111355:111355] [20221221:235746:1671663466:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:Stargazer:1:1:111355:111355] [20221221:235746:1671663466:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Stargazer:1:1:111355:111355] [20221221:235746:1671663466:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Stargazer:1:1:111355:111355] [20221221:235746:1671663466:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:Stargazer:1:1:111355:111355] [20221221:235746:1671663466:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Stargazer:1:1:111355:111355] [20221221:235746:1671663466:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Stargazer:1:1:111355:111355] [20221221:235746:1671663466:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Stargazer:1:1:111355:111355] [20221221:235746:1671663466:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:Stargazer:1:1:111355:111355] [20221221:235746:1671663466:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:Stargazer:1:1:111355:111355] [20221221:235746:1671663466:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Stargazer:1:1:111355:111355] [20221221:235746:1671663466:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  23:58:29 STEP 1704 H=1704:00 +CPU= 39.501
  23:58:54 STEP 1705 H=1705:00 +CPU= 24.981
  23:59:23 STEP 1706 H=1706:00 +CPU= 28.531
  00:00:00 STEP 1707 H=1707:00 +CPU= 36.228
  00:00:26 STEP 1708 H=1708:00 +CPU= 25.944
  00:00:47 STEP 1709 H=1709:00 +CPU= 20.504
  00:01:17 STEP 1710 H=1710:00 +CPU= 29.426
  00:01:37 STEP 1711 H=1711:00 +CPU= 20.463
  00:02:00 STEP 1712 H=1712:00 +CPU= 22.723
  00:02:31 STEP 1713 H=1713:00 +CPU= 30.391
  00:02:53 STEP 1714 H=1714:00 +CPU= 22.587
  00:03:16 STEP 1715 H=1715:00 +CPU= 21.989
  00:03:54 STEP 1716 H=1716:00 +CPU= 38.238
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+001716
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+001716
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+001716
  00:04:18 STEP 1717 H=1717:00 +CPU= 22.656
  00:04:42 STEP 1718 H=1718:00 +CPU= 24.426
  00:05:18 STEP 1719 H=1719:00 +CPU= 35.925
  00:05:42 STEP 1720 H=1720:00 +CPU= 23.612
  00:06:04 STEP 1721 H=1721:00 +CPU= 21.999
  00:06:37 STEP 1722 H=1722:00 +CPU= 32.296
  00:07:00 STEP 1723 H=1723:00 +CPU= 23.225
  00:07:26 STEP 1724 H=1724:00 +CPU= 25.468
  00:08:02 STEP 1725 H=1725:00 +CPU= 36.303
  00:08:27 STEP 1726 H=1726:00 +CPU= 24.033
  00:08:52 STEP 1727 H=1727:00 +CPU= 25.072
  00:09:38 STEP 1728 H=1728:00 +CPU= 44.669
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+001728
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+001728
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172225/ICMGGhq0f+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172225/ICMSHhq0f+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172225/ICMUAhq0f+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172225/ICMGGhq0f+001716
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172225/ICMSHhq0f+001716
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172225/ICMUAhq0f+001716
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0581_1986050100_123_955_12172225_0_r1551694089_71.zip
Uploading the intermediate file: upload_file_71.zip
  00:10:03 STEP 1729 H=1729:00 +CPU= 25.540
  00:10:26 STEP 1730 H=1730:00 +CPU= 22.715
  00:11:03 STEP 1731 H=1731:00 +CPU= 35.743
  00:11:29 STEP 1732 H=1732:00 +CPU= 26.735
  00:11:52 STEP 1733 H=1733:00 +CPU= 22.203
  00:12:29 STEP 1734 H=1734:00 +CPU= 37.515
  00:12:53 STEP 1735 H=1735:00 +CPU= 23.118
  00:13:16 STEP 1736 H=1736:00 +CPU= 23.430
Abort request received from BOINC client, ending the child process

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 15:46:43 1529888 22256673 oifs_43r3_ps_0581_1986050100_123_955_12172225_0 6,134,400 47,884 0.0078
09 May 2023 15:46:31 1529888 22256673 oifs_43r3_ps_0581_1986050100_123_955_12172225_0 5,097,600 39,756 0.0078
09 May 2023 15:46:15 1529888 22256673 oifs_43r3_ps_0581_1986050100_123_955_12172225_0 3,542,400 27,319 0.0077
09 May 2023 15:46:05 1529888 22256673 oifs_43r3_ps_0581_1986050100_123_955_12172225_0 2,678,400 20,808 0.0078
09 May 2023 15:45:56 1529888 22256673 oifs_43r3_ps_0581_1986050100_123_955_12172225_0 1,728,000 13,545 0.0078
09 May 2023 15:45:51 1529888 22256673 oifs_43r3_ps_0581_1986050100_123_955_12172225_0 864,000 6,669 0.0077


©2024 climateprediction.net