climateprediction.net home page
Task 22304305

Task 22304305

Name oifs_43r3_ps_0384_2017050100_123_986_12203028_1
Workunit 12203028
Created 27 Jan 2023, 20:45:43 UTC
Sent 27 Jan 2023, 20:46:14 UTC
Report deadline 28 Mar 2023, 20:46:14 UTC
Received 29 Jan 2023, 1:59:48 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1531398
Run time 15 hours 55 min 31 sec
CPU time 15 hours 22 min 20 sec
Validate state Invalid
Credit 4,475.52
Device peak FLOPS 7.92 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,798.50 MB
Peak swap size 4,978.07 MB
Peak disk usage 7,674.36 MB
Stderr 
<core_client_version>7.20.2</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
320
Moving to projects directory: /var/lib/boinc/slots/8/ICMSHhq0f+001320
Moving to projects directory: /var/lib/boinc/slots/8/ICMUAhq0f+001320
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12203028/ICMGGhq0f+001296
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12203028/ICMSHhq0f+001296
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12203028/ICMUAhq0f+001296
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12203028/ICMGGhq0f+001308
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12203028/ICMSHhq0f+001308
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12203028/ICMUAhq0f+001308
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0384_2017050100_123_986_12203028_1_r1556625311_54.zip
Uploading the intermediate file: upload_file_54.zip
  19:06:55 STEP 1321 H=1321:00 +CPU= 28.181
  19:07:27 STEP 1322 H=1322:00 +CPU= 30.420
  19:08:07 STEP 1323 H=1323:00 +CPU= 39.042
  19:08:36 STEP 1324 H=1324:00 +CPU= 28.506
  19:09:03 STEP 1325 H=1325:00 +CPU= 25.977
  19:09:45 STEP 1326 H=1326:00 +CPU= 41.024
  19:10:15 STEP 1327 H=1327:00 +CPU= 28.362
  19:10:40 STEP 1328 H=1328:00 +CPU= 24.357
  19:11:24 STEP 1329 H=1329:00 +CPU= 43.673
  19:11:50 STEP 1330 H=1330:00 +CPU= 24.803
  19:12:17 STEP 1331 H=1331:00 +CPU= 26.708
  19:13:04 STEP 1332 H=1332:00 +CPU= 45.416
Moving to projects directory: /var/lib/boinc/slots/8/ICMGGhq0f+001332
Moving to projects directory: /var/lib/boinc/slots/8/ICMSHhq0f+001332
Moving to projects directory: /var/lib/boinc/slots/8/ICMUAhq0f+001332
  19:13:30 STEP 1333 H=1333:00 +CPU= 25.386
  19:13:59 STEP 1334 H=1334:00 +CPU= 28.941
  19:14:37 STEP 1335 H=1335:00 +CPU= 37.428
  19:15:07 STEP 1336 H=1336:00 +CPU= 28.646
  19:15:36 STEP 1337 H=1337:00 +CPU= 28.139
  19:16:17 STEP 1338 H=1338:00 +CPU= 39.859
  19:16:46 STEP 1339 H=1339:00 +CPU= 28.747
  19:17:12 STEP 1340 H=1340:00 +CPU= 25.723
  19:17:55 STEP 1341 H=1341:00 +CPU= 42.111
  19:18:24 STEP 1342 H=1342:00 +CPU= 27.906
  19:18:51 STEP 1343 H=1343:00 +CPU= 26.198
  19:19:41 STEP 1344 H=1344:00 +CPU= 48.569
Moving to projects directory: /var/lib/boinc/slots/8/ICMGGhq0f+001344
Moving to projects directory: /var/lib/boinc/slots/8/ICMSHhq0f+001344
Moving to projects directory: /var/lib/boinc/slots/8/ICMUAhq0f+001344
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12203028/ICMGGhq0f+001320
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12203028/ICMSHhq0f+001320
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12203028/ICMUAhq0f+001320
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12203028/ICMGGhq0f+001332
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12203028/ICMSHhq0f+001332
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12203028/ICMUAhq0f+001332
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0384_2017050100_123_986_12203028_1_r1556625311_55.zip
Uploading the intermediate file: upload_file_55.zip
  19:20:07 STEP 1345 H=1345:00 +CPU= 25.244
  19:20:36 STEP 1346 H=1346:00 +CPU= 28.290
  19:21:19 STEP 1347 H=1347:00 +CPU= 40.727
  19:21:47 STEP 1348 H=1348:00 +CPU= 26.956
  19:22:14 STEP 1349 H=1349:00 +CPU= 26.716
  19:22:55 STEP 1350 H=1350:00 +CPU= 39.894
  19:23:23 STEP 1351 H=1351:00 +CPU= 26.993
  19:23:52 STEP 1352 H=1352:00 +CPU= 27.898
  19:24:32 STEP 1353 H=1353:00 +CPU= 39.210
  19:25:02 STEP 1354 H=1354:00 +CPU= 28.389
  19:25:31 STEP 1355 H=1355:00 +CPU= 28.175
  19:26:16 STEP 1356 H=1356:00 +CPU= 43.883
Moving to projects directory: /var/lib/boinc/slots/8/ICMGGhq0f+001356
Moving to projects directory: /var/lib/boinc/slots/8/ICMSHhq0f+001356
Moving to projects directory: /var/lib/boinc/slots/8/ICMUAhq0f+001356
  19:26:46 STEP 1357 H=1357:00 +CPU= 29.067
  19:27:12 STEP 1358 H=1358:00 +CPU= 24.998
  19:27:55 STEP 1359 H=1359:00 +CPU= 42.187
  19:28:24 STEP 1360 H=1360:00 +CPU= 27.927
  19:28:51 STEP 1361 H=1361:00 +CPU= 26.051
  19:29:36 STEP 1362 H=1362:00 +CPU= 44.300
  19:30:02 STEP 1363 H=1363:00 +CPU= 24.386
  19:30:31 STEP 1364 H=1364:00 +CPU= 28.861
  19:31:15 STEP 1365 H=1365:00 +CPU= 41.727
  19:31:41 STEP 1366 H=1366:00 +CPU= 26.140
  19:32:12 STEP 1367 H=1367:00 +CPU= 30.074
  19:33:00 STEP 1368 H=1368:00 +CPU= 46.570
Moving to projects directory: /var/lib/boinc/slots/8/ICMGGhq0f+001368
Moving to projects directory: /var/lib/boinc/slots/8/ICMSHhq0f+001368
Moving to projects directory: /var/lib/boinc/slots/8/ICMUAhq0f+001368
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12203028/ICMGGhq0f+001344
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12203028/ICMSHhq0f+001344
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12203028/ICMUAhq0f+001344
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12203028/ICMGGhq0f+001356
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12203028/ICMSHhq0f+001356
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12203028/ICMUAhq0f+001356
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0384_2017050100_123_986_12203028_1_r1556625311_56.zip
Uploading the intermediate file: upload_file_56.zip
  19:33:32 STEP 1369 H=1369:00 +CPU= 29.863
  19:34:02 STEP 1370 H=1370:00 +CPU= 29.324
  19:34:44 STEP 1371 H=1371:00 +CPU= 40.318
  19:35:16 STEP 1372 H=1372:00 +CPU= 31.034
  19:35:40 STEP 1373 H=1373:00 +CPU= 23.505
  19:36:28 STEP 1374 H=1374:00 +CPU= 46.141
  19:36:54 STEP 1375 H=1375:00 +CPU= 25.682
  19:37:24 STEP 1376 H=1376:00 +CPU= 29.234
  19:38:08 STEP 1377 H=1377:00 +CPU= 42.577
  19:38:36 STEP 1378 H=1378:00 +CPU= 26.900
  19:39:07 STEP 1379 H=1379:00 +CPU= 29.445
  19:39:51 STEP 1380 H=1380:00 +CPU= 42.978
Moving to projects directory: /var/lib/boinc/slots/8/ICMGGhq0f+001380
Moving to projects directory: /var/lib/boinc/slots/8/ICMSHhq0f+001380
Moving to projects directory: /var/lib/boinc/slots/8/ICMUAhq0f+001380
  19:40:22 STEP 1381 H=1381:00 +CPU= 30.028
  19:40:51 STEP 1382 H=1382:00 +CPU= 28.436
  19:41:32 STEP 1383 H=1383:00 +CPU= 39.803
  19:42:02 STEP 1384 H=1384:00 +CPU= 28.608
  19:42:30 STEP 1385 H=1385:00 +CPU= 27.018
  19:43:13 STEP 1386 H=1386:00 +CPU= 42.423
  19:43:42 STEP 1387 H=1387:00 +CPU= 28.276
  19:44:09 STEP 1388 H=1388:00 +CPU= 25.877
  19:44:53 STEP 1389 H=1389:00 +CPU= 42.782
  19:45:22 STEP 1390 H=1390:00 +CPU= 27.607
  19:45:50 STEP 1391 H=1391:00 +CPU= 27.541
  19:46:42 STEP 1392 H=1392:00 +CPU= 49.997
Moving to projects directory: /var/lib/boinc/slots/8/ICMGGhq0f+001392
Moving to projects directory: /var/lib/boinc/slots/8/ICMSHhq0f+001392
Moving to projects directory: /var/lib/boinc/slots/8/ICMUAhq0f+001392
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12203028/ICMGGhq0f+001368
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12203028/ICMSHhq0f+001368
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12203028/ICMUAhq0f+001368
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12203028/ICMGGhq0f+001380
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12203028/ICMSHhq0f+001380
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12203028/ICMUAhq0f+001380
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0384_2017050100_123_986_12203028_1_r1556625311_57.zip
Uploading the intermediate file: upload_file_57.zip
  19:47:09 STEP 1393 H=1393:00 +CPU= 25.305
  19:47:39 STEP 1394 H=1394:00 +CPU= 29.188
  19:48:20 STEP 1395 H=1395:00 +CPU= 40.309
  19:48:48 STEP 1396 H=1396:00 +CPU= 27.230
  19:49:19 STEP 1397 H=1397:00 +CPU= 30.223
  19:49:59 STEP 1398 H=1398:00 +CPU= 39.036
  19:50:32 STEP 1399 H=1399:00 +CPU= 30.957
  19:50:59 STEP 1400 H=1400:00 +CPU= 27.161
  19:51:43 STEP 1401 H=1401:00 +CPU= 42.246
  19:52:13 STEP 1402 H=1402:00 +CPU= 28.875
  19:52:38 STEP 1403 H=1403:00 +CPU= 24.404
  19:53:28 STEP 1404 H=1404:00 +CPU= 49.032
Moving to projects directory: /var/lib/boinc/slots/8/ICMGGhq0f+001404
Moving to projects directory: /var/lib/boinc/slots/8/ICMSHhq0f+001404
Moving to projects directory: /var/lib/boinc/slots/8/ICMUAhq0f+001404
  19:53:54 STEP 1405 H=1405:00 +CPU= 24.905
  19:54:23 STEP 1406 H=1406:00 +CPU= 28.462
  19:55:05 STEP 1407 H=1407:00 +CPU= 40.405
  19:55:33 STEP 1408 H=1408:00 +CPU= 27.154
  19:56:03 STEP 1409 H=1409:00 +CPU= 29.648
  19:56:45 STEP 1410 H=1410:00 +CPU= 40.889
  19:57:15 STEP 1411 H=1411:00 +CPU= 29.169
  19:57:46 STEP 1412 H=1412:00 +CPU= 29.408
  19:58:28 STEP 1413 H=1413:00 +CPU= 40.783
  19:58:59 STEP 1414 H=1414:00 +CPU= 29.752
  19:59:28 STEP 1415 H=1415:00 +CPU= 27.883
  20:00:19 STEP 1416 H=1416:00 +CPU= 49.217
Moving to projects directory: /var/lib/boinc/slots/8/ICMGGhq0f+001416
Moving to projects directory: /var/lib/boinc/slots/8/ICMSHhq0f+001416
Moving to projects directory: /var/lib/boinc/slots/8/ICMUAhq0f+001416
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12203028/ICMGGhq0f+001392
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12203028/ICMSHhq0f+001392
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12203028/ICMUAhq0f+001392
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12203028/ICMGGhq0f+001404
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12203028/ICMSHhq0f+001404
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12203028/ICMUAhq0f+001404
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0384_2017050100_123_986_12203028_1_r1556625311_58.zip
Uploading the intermediate file: upload_file_58.zip
  20:00:47 STEP 1417 H=1417:00 +CPU= 27.287
  20:01:15 STEP 1418 H=1418:00 +CPU= 27.570
  20:02:01 STEP 1419 H=1419:00 +CPU= 44.651
  20:02:28 STEP 1420 H=1420:00 +CPU= 26.169
  20:02:56 STEP 1421 H=1421:00 +CPU= 27.652
  20:03:40 STEP 1422 H=1422:00 +CPU= 42.146
  20:04:06 STEP 1423 H=1423:00 +CPU= 26.057
  20:04:38 STEP 1424 H=1424:00 +CPU= 30.690
  20:05:20 STEP 1425 H=1425:00 +CPU= 40.268
  20:05:51 STEP 1426 H=1426:00 +CPU= 30.350
  20:06:18 STEP 1427 H=1427:00 +CPU= 25.943
  20:07:05 STEP 1428 H=1428:00 +CPU= 46.077
Moving to projects directory: /var/lib/boinc/slots/8/ICMGGhq0f+001428
Moving to projects directory: /var/lib/boinc/slots/8/ICMSHhq0f+001428
Moving to projects directory: /var/lib/boinc/slots/8/ICMUAhq0f+001428
  20:07:35 STEP 1429 H=1429:00 +CPU= 28.610
  20:08:02 STEP 1430 H=1430:00 +CPU= 26.181
  20:08:48 STEP 1431 H=1431:00 +CPU= 45.055
  20:09:14 STEP 1432 H=1432:00 +CPU= 25.329
  20:09:45 STEP 1433 H=1433:00 +CPU= 30.569
  20:10:29 STEP 1434 H=1434:00 +CPU= 41.980
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2017050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0384
(argv4) batchid: 986
(argv5) wuid: 12203028
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/8
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12203028
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/8/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/8/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_79309c237478e28dc003f376a90f63a2 to: /var/lib/boinc/slots/8/oifs_43r3_ps_0384_2017050100_123_986_12203028.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/8/oifs_43r3_ps_0384_2017050100_123_986_12203028.zip
ic_ancil_file: ic_ancil_12203028
ifsdata_file: ifsdata_12203028
climate_data_file: clim_data_12203028
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_7c6b7a5239e8fa5551203885b25978bb to: /var/lib/boinc/slots/8/ic_ancil_12203028.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/8/ic_ancil_12203028.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_0ec81e22fe81753eb3e03de66137c648 to: /var/lib/boinc/slots/8/ifsdata/ifsdata_12203028.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/8/ifsdata/ifsdata_12203028.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/8/159l_2/clim_data_12203028.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/8/159l_2/clim_data_12203028.zip
Checking for progress XML file: /var/lib/boinc/slots/8/progress_file_12203028.xml
Opened progress file ok : /var/lib/boinc/slots/8/progress_file_12203028.xml
-- Model is restarting --
Adjusting last_iter, 1435, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/8/progress_file_12203028.xml
last_cpu_time: 49770
upload_file_number: 59
last_iter: 1415
last_upload: 5097600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0384_2017050100_123_986_12203028_1_r1556625311
The child process has been launched with process id: 13841
Executing the command: /var/lib/boinc/slots/8/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:tweenis:1:1:13841:13841] [20230128:204329:1674938609:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:tweenis:1:1:13841:13841] [20230128:204329:1674938609:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:tweenis:1:1:13841:13841] [20230128:204329:1674938609:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:tweenis:1:1:13841:13841] [20230128:204329:1674938609:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:tweenis:1:1:13841:13841] [20230128:204329:1674938609:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:tweenis:1:1:13841:13841] [20230128:204329:1674938609:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:tweenis:1:1:13841:13841] [20230128:204329:1674938609:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:tweenis:1:1:13841:13841] [20230128:204329:1674938609:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:tweenis:1:1:13841:13841] [20230128:204329:1674938609:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:tweenis:1:1:13841:13841] [20230128:204329:1674938609:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:tweenis:1:1:13841:13841] [20230128:204329:1674938609:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:tweenis:1:1:13841:13841] [20230128:204329:1674938609:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:tweenis:1:1:13841:13841] [20230128:204329:1674938609:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:tweenis:1:1:13841:13841] [20230128:204329:1674938609:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:tweenis:1:1:13841:13841] [20230128:204329:1674938609:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:tweenis:1:1:13841:13841] [20230128:204329:1674938609:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:13841:13841] [20230128:204329:1674938609:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:13841:13841] [20230128:204329:1674938609:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:13841:13841] [20230128:204329:1674938609:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:13841:13841] [20230128:204329:1674938609:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:tweenis:1:1:13841:13841] [20230128:204329:1674938609:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:13841:13841] [20230128:204329:1674938609:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:13841:13841] [20230128:204329:1674938609:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:13841:13841] [20230128:204329:1674938609:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:13841:13841] [20230128:204329:1674938609:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:13841:13841] [20230128:204329:1674938609:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:13841:13841] [20230128:204329:1674938609:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:13841:13841] [20230128:204329:1674938609:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:13841:13841] [20230128:204329:1674938609:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:13841:13841] [20230128:204329:1674938609:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  20:44:07 STEP 1416 H=1416:00 +CPU= 33.028
  20:44:38 STEP 1417 H=1417:00 +CPU= 28.416
  20:45:09 STEP 1418 H=1418:00 +CPU= 29.580
  20:45:49 STEP 1419 H=1419:00 +CPU= 37.913
  20:46:20 STEP 1420 H=1420:00 +CPU= 29.297
  20:46:46 STEP 1421 H=1421:00 +CPU= 24.227
  20:47:31 STEP 1422 H=1422:00 +CPU= 42.643
  20:48:01 STEP 1423 H=1423:00 +CPU= 28.587
  20:48:27 STEP 1424 H=1424:00 +CPU= 24.761
  20:49:12 STEP 1425 H=1425:00 +CPU= 42.927
  20:49:38 STEP 1426 H=1426:00 +CPU= 24.893
  20:50:07 STEP 1427 H=1427:00 +CPU= 27.165
  20:50:52 STEP 1428 H=1428:00 +CPU= 42.542
Moving to projects directory: /var/lib/boinc/slots/8/ICMGGhq0f+001428
Moving to projects directory: /var/lib/boinc/slots/8/ICMSHhq0f+001428
Moving to projects directory: /var/lib/boinc/slots/8/ICMUAhq0f+001428
  20:51:21 STEP 1429 H=1429:00 +CPU= 27.591
  20:51:51 STEP 1430 H=1430:00 +CPU= 28.409
  20:52:32 STEP 1431 H=1431:00 +CPU= 39.266
  20:53:03 STEP 1432 H=1432:00 +CPU= 29.363
  20:53:31 STEP 1433 H=1433:00 +CPU= 26.703
  20:54:14 STEP 1434 H=1434:00 +CPU= 39.481
  20:54:44 STEP 1435 H=1435:00 +CPU= 28.305
  20:55:13 STEP 1436 H=1436:00 +CPU= 26.824
  20:55:56 STEP 1437 H=1437:00 +CPU= 40.377
  20:56:25 STEP 1438 H=1438:00 +CPU= 26.324
  20:56:53 STEP 1439 H=1439:00 +CPU= 26.522
  20:57:44 STEP 1440 H=1440:00 +CPU= 45.434
Moving to projects directory: /var/lib/boinc/slots/8/ICMGGhq0f+001440
Moving to projects directory: /var/lib/boinc/slots/8/ICMSHhq0f+001440
Moving to projects directory: /var/lib/boinc/slots/8/ICMUAhq0f+001440
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12203028/ICMGGhq0f+001416
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12203028/ICMSHhq0f+001416
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12203028/ICMUAhq0f+001416
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12203028/ICMGGhq0f+001428
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12203028/ICMSHhq0f+001428
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12203028/ICMUAhq0f+001428
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0384_2017050100_123_986_12203028_1_r1556625311_59.zip
Uploading the intermediate file: upload_file_59.zip
  20:58:15 STEP 1441 H=1441:00 +CPU= 27.491
  20:58:44 STEP 1442 H=1442:00 +CPU= 27.032
  20:59:28 STEP 1443 H=1443:00 +CPU= 39.736
  20:59:59 STEP 1444 H=1444:00 +CPU= 27.384
  21:00:27 STEP 1445 H=1445:00 +CPU= 26.261
  21:01:13 STEP 1446 H=1446:00 +CPU= 41.229
  21:01:43 STEP 1447 H=1447:00 +CPU= 26.819
  21:02:12 STEP 1448 H=1448:00 +CPU= 25.808
  21:02:58 STEP 1449 H=1449:00 +CPU= 40.472
  21:03:27 STEP 1450 H=1450:00 +CPU= 26.323
  21:03:59 STEP 1451 H=1451:00 +CPU= 28.234
  21:04:44 STEP 1452 H=1452:00 +CPU= 41.266
Moving to projects directory: /var/lib/boinc/slots/8/ICMGGhq0f+001452
Moving to projects directory: /var/lib/boinc/slots/8/ICMSHhq0f+001452
Moving to projects directory: /var/lib/boinc/slots/8/ICMUAhq0f+001452
  21:05:15 STEP 1453 H=1453:00 +CPU= 27.888
  21:05:45 STEP 1454 H=1454:00 +CPU= 26.982
  21:06:29 STEP 1455 H=1455:00 +CPU= 39.748
  21:07:01 STEP 1456 H=1456:00 +CPU= 28.903
  21:07:26 STEP 1457 H=1457:00 +CPU= 23.638
  21:08:13 STEP 1458 H=1458:00 +CPU= 44.049
  21:08:41 STEP 1459 H=1459:00 +CPU= 25.958
  21:09:10 STEP 1460 H=1460:00 +CPU= 26.675
  21:09:57 STEP 1461 H=1461:00 +CPU= 43.315
  21:10:24 STEP 1462 H=1462:00 +CPU= 24.587
  21:10:57 STEP 1463 H=1463:00 +CPU= 30.509
  21:11:46 STEP 1464 H=1464:00 +CPU= 45.353
Moving to projects directory: /var/lib/boinc/slots/8/ICMGGhq0f+001464
Moving to projects directory: /var/lib/boinc/slots/8/ICMSHhq0f+001464
Moving to projects directory: /var/lib/boinc/slots/8/ICMUAhq0f+001464
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12203028/ICMGGhq0f+001440
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12203028/ICMSHhq0f+001440
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12203028/ICMUAhq0f+001440
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12203028/ICMGGhq0f+001452
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12203028/ICMSHhq0f+001452
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12203028/ICMUAhq0f+001452
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0384_2017050100_123_986_12203028_1_r1556625311_60.zip
Uploading the intermediate file: upload_file_60.zip
  21:12:17 STEP 1465 H=1465:00 +CPU= 28.794
  21:12:49 STEP 1466 H=1466:00 +CPU= 29.470
  21:13:30 STEP 1467 H=1467:00 +CPU= 38.616
  21:14:03 STEP 1468 H=1468:00 +CPU= 29.530
  21:14:29 STEP 1469 H=1469:00 +CPU= 23.911
  21:15:16 STEP 1470 H=1470:00 +CPU= 44.060
  21:15:44 STEP 1471 H=1471:00 +CPU= 25.785
  21:16:13 STEP 1472 H=1472:00 +CPU= 27.147
  21:17:00 STEP 1473 H=1473:00 +CPU= 43.174
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2017050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0384
(argv4) batchid: 986
(argv5) wuid: 12203028
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/8
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12203028
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/8/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/8/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_79309c237478e28dc003f376a90f63a2 to: /var/lib/boinc/slots/8/oifs_43r3_ps_0384_2017050100_123_986_12203028.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/8/oifs_43r3_ps_0384_2017050100_123_986_12203028.zip
ic_ancil_file: ic_ancil_12203028
ifsdata_file: ifsdata_12203028
climate_data_file: clim_data_12203028
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_7c6b7a5239e8fa5551203885b25978bb to: /var/lib/boinc/slots/8/ic_ancil_12203028.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/8/ic_ancil_12203028.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_0ec81e22fe81753eb3e03de66137c648 to: /var/lib/boinc/slots/8/ifsdata/ifsdata_12203028.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/8/ifsdata/ifsdata_12203028.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/8/159l_2/clim_data_12203028.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/8/159l_2/clim_data_12203028.zip
Checking for progress XML file: /var/lib/boinc/slots/8/progress_file_12203028.xml
Opened progress file ok : /var/lib/boinc/slots/8/progress_file_12203028.xml
-- Model is restarting --
Adjusting last_iter, 1474, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/8/progress_file_12203028.xml
last_cpu_time: 51635
upload_file_number: 61
last_iter: 1463
last_upload: 5270400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0384_2017050100_123_986_12203028_1_r1556625311
The child process has been launched with process id: 7467
Executing the command: /var/lib/boinc/slots/8/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:tweenis:1:1:7467:7467] [20230128:223341:1674945221:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:tweenis:1:1:7467:7467] [20230128:223341:1674945221:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:tweenis:1:1:7467:7467] [20230128:223341:1674945221:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:tweenis:1:1:7467:7467] [20230128:223341:1674945221:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:tweenis:1:1:7467:7467] [20230128:223341:1674945221:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:tweenis:1:1:7467:7467] [20230128:223341:1674945221:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:tweenis:1:1:7467:7467] [20230128:223341:1674945221:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:tweenis:1:1:7467:7467] [20230128:223341:1674945221:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:tweenis:1:1:7467:7467] [20230128:223341:1674945221:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:tweenis:1:1:7467:7467] [20230128:223341:1674945221:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:tweenis:1:1:7467:7467] [20230128:223341:1674945221:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:tweenis:1:1:7467:7467] [20230128:223341:1674945221:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:tweenis:1:1:7467:7467] [20230128:223341:1674945221:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:tweenis:1:1:7467:7467] [20230128:223341:1674945221:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:tweenis:1:1:7467:7467] [20230128:223341:1674945221:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:tweenis:1:1:7467:7467] [20230128:223341:1674945221:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:7467:7467] [20230128:223341:1674945221:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:7467:7467] [20230128:223341:1674945221:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:7467:7467] [20230128:223341:1674945221:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:7467:7467] [20230128:223341:1674945221:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:tweenis:1:1:7467:7467] [20230128:223341:1674945221:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:7467:7467] [20230128:223341:1674945221:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:7467:7467] [20230128:223341:1674945221:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:7467:7467] [20230128:223341:1674945221:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:7467:7467] [20230128:223341:1674945221:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:7467:7467] [20230128:223341:1674945221:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:7467:7467] [20230128:223341:1674945221:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:7467:7467] [20230128:223341:1674945221:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:7467:7467] [20230128:223341:1674945221:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:7467:7467] [20230128:223341:1674945221:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  22:34:22 STEP 1464 H=1464:00 +CPU= 33.379
  22:34:54 STEP 1465 H=1465:00 +CPU= 26.024
  22:35:24 STEP 1466 H=1466:00 +CPU= 27.391
  22:36:07 STEP 1467 H=1467:00 +CPU= 39.150
  22:36:39 STEP 1468 H=1468:00 +CPU= 29.321
  22:37:06 STEP 1469 H=1469:00 +CPU= 25.656
  22:37:51 STEP 1470 H=1470:00 +CPU= 43.396
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2017050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0384
(argv4) batchid: 986
(argv5) wuid: 12203028
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/8
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12203028
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/8/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/8/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_79309c237478e28dc003f376a90f63a2 to: /var/lib/boinc/slots/8/oifs_43r3_ps_0384_2017050100_123_986_12203028.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/8/oifs_43r3_ps_0384_2017050100_123_986_12203028.zip
ic_ancil_file: ic_ancil_12203028
ifsdata_file: ifsdata_12203028
climate_data_file: clim_data_12203028
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_7c6b7a5239e8fa5551203885b25978bb to: /var/lib/boinc/slots/8/ic_ancil_12203028.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/8/ic_ancil_12203028.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_0ec81e22fe81753eb3e03de66137c648 to: /var/lib/boinc/slots/8/ifsdata/ifsdata_12203028.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/8/ifsdata/ifsdata_12203028.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/8/159l_2/clim_data_12203028.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/8/159l_2/clim_data_12203028.zip
Checking for progress XML file: /var/lib/boinc/slots/8/progress_file_12203028.xml
Opened progress file ok : /var/lib/boinc/slots/8/progress_file_12203028.xml
-- Model is restarting --
Adjusting last_iter, 1471, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/8/progress_file_12203028.xml
last_cpu_time: 51878
upload_file_number: 61
last_iter: 1463
last_upload: 5270400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0384_2017050100_123_986_12203028_1_r1556625311
The child process has been launched with process id: 9756
Executing the command: /var/lib/boinc/slots/8/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:tweenis:1:1:9756:9756] [20230128:233204:1674948724:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:tweenis:1:1:9756:9756] [20230128:233204:1674948724:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:tweenis:1:1:9756:9756] [20230128:233204:1674948724:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:tweenis:1:1:9756:9756] [20230128:233204:1674948724:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:tweenis:1:1:9756:9756] [20230128:233204:1674948724:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:tweenis:1:1:9756:9756] [20230128:233204:1674948724:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:tweenis:1:1:9756:9756] [20230128:233204:1674948724:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:tweenis:1:1:9756:9756] [20230128:233204:1674948724:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:tweenis:1:1:9756:9756] [20230128:233204:1674948724:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:tweenis:1:1:9756:9756] [20230128:233204:1674948724:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:tweenis:1:1:9756:9756] [20230128:233204:1674948724:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:tweenis:1:1:9756:9756] [20230128:233204:1674948724:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:tweenis:1:1:9756:9756] [20230128:233204:1674948724:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:tweenis:1:1:9756:9756] [20230128:233204:1674948724:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:tweenis:1:1:9756:9756] [20230128:233204:1674948724:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:tweenis:1:1:9756:9756] [20230128:233204:1674948724:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:9756:9756] [20230128:233204:1674948724:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:9756:9756] [20230128:233204:1674948724:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:9756:9756] [20230128:233204:1674948724:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:9756:9756] [20230128:233204:1674948724:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:tweenis:1:1:9756:9756] [20230128:233204:1674948724:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:9756:9756] [20230128:233204:1674948724:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:9756:9756] [20230128:233204:1674948724:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:9756:9756] [20230128:233204:1674948724:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:9756:9756] [20230128:233204:1674948724:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:9756:9756] [20230128:233204:1674948724:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:9756:9756] [20230128:233204:1674948724:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:9756:9756] [20230128:233204:1674948724:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:9756:9756] [20230128:233204:1674948724:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:9756:9756] [20230128:233204:1674948724:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  23:32:42 STEP 1464 H=1464:00 +CPU= 34.379
  23:33:15 STEP 1465 H=1465:00 +CPU= 29.717
  23:33:47 STEP 1466 H=1466:00 +CPU= 28.412
  23:34:30 STEP 1467 H=1467:00 +CPU= 38.033
  23:35:01 STEP 1468 H=1468:00 +CPU= 28.998
  23:35:28 STEP 1469 H=1469:00 +CPU= 25.573
  23:36:12 STEP 1470 H=1470:00 +CPU= 41.977
  23:36:43 STEP 1471 H=1471:00 +CPU= 27.778
  23:37:11 STEP 1472 H=1472:00 +CPU= 25.770
  23:37:54 STEP 1473 H=1473:00 +CPU= 40.914
  23:38:21 STEP 1474 H=1474:00 +CPU= 25.481
  23:38:51 STEP 1475 H=1475:00 +CPU= 28.405
  23:39:36 STEP 1476 H=1476:00 +CPU= 42.819
Moving to projects directory: /var/lib/boinc/slots/8/ICMGGhq0f+001476
Moving to projects directory: /var/lib/boinc/slots/8/ICMSHhq0f+001476
Moving to projects directory: /var/lib/boinc/slots/8/ICMUAhq0f+001476
  23:40:05 STEP 1477 H=1477:00 +CPU= 27.298
  23:40:35 STEP 1478 H=1478:00 +CPU= 28.556
  23:41:16 STEP 1479 H=1479:00 +CPU= 38.109
  23:41:48 STEP 1480 H=1480:00 +CPU= 28.407
  23:42:20 STEP 1481 H=1481:00 +CPU= 26.044
  23:43:11 STEP 1482 H=1482:00 +CPU= 41.169
  23:43:48 STEP 1483 H=1483:00 +CPU= 29.990
  23:44:18 STEP 1484 H=1484:00 +CPU= 26.579
  23:45:11 STEP 1485 H=1485:00 +CPU= 42.745
  23:45:44 STEP 1486 H=1486:00 +CPU= 26.412
  23:46:20 STEP 1487 H=1487:00 +CPU= 29.260
  23:47:16 STEP 1488 H=1488:00 +CPU= 45.713
Moving to projects directory: /var/lib/boinc/slots/8/ICMGGhq0f+001488
Moving to projects directory: /var/lib/boinc/slots/8/ICMSHhq0f+001488
Moving to projects directory: /var/lib/boinc/slots/8/ICMUAhq0f+001488
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12203028/ICMGGhq0f+001464
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12203028/ICMSHhq0f+001464
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12203028/ICMUAhq0f+001464
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12203028/ICMGGhq0f+001476
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12203028/ICMSHhq0f+001476
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12203028/ICMUAhq0f+001476
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0384_2017050100_123_986_12203028_1_r1556625311_61.zip
Uploading the intermediate file: upload_file_61.zip
  23:47:48 STEP 1489 H=1489:00 +CPU= 26.966
  23:48:25 STEP 1490 H=1490:00 +CPU= 29.393
  23:49:09 STEP 1491 H=1491:00 +CPU= 37.884
  23:49:44 STEP 1492 H=1492:00 +CPU= 28.353
  23:50:14 STEP 1493 H=1493:00 +CPU= 24.816
  23:51:02 STEP 1494 H=1494:00 +CPU= 39.827
  23:51:34 STEP 1495 H=1495:00 +CPU= 26.990
  23:52:03 STEP 1496 H=1496:00 +CPU= 24.265
  23:52:50 STEP 1497 H=1497:00 +CPU= 37.578
  23:53:20 STEP 1498 H=1498:00 +CPU= 24.858
  23:53:51 STEP 1499 H=1499:00 +CPU= 25.118
  23:54:41 STEP 1500 H=1500:00 +CPU= 41.329
Moving to projects directory: /var/lib/boinc/slots/8/ICMGGhq0f+001500
Moving to projects directory: /var/lib/boinc/slots/8/ICMSHhq0f+001500
Moving to projects directory: /var/lib/boinc/slots/8/ICMUAhq0f+001500
  23:55:11 STEP 1501 H=1501:00 +CPU= 24.350
  23:55:43 STEP 1502 H=1502:00 +CPU= 26.110
  23:56:28 STEP 1503 H=1503:00 +CPU= 36.791
  23:57:01 STEP 1504 H=1504:00 +CPU= 26.523
  23:57:32 STEP 1505 H=1505:00 +CPU= 26.418
  23:58:18 STEP 1506 H=1506:00 +CPU= 37.469
  23:58:51 STEP 1507 H=1507:00 +CPU= 26.259
  23:59:21 STEP 1508 H=1508:00 +CPU= 25.299
  00:00:08 STEP 1509 H=1509:00 +CPU= 38.422
  00:00:42 STEP 1510 H=1510:00 +CPU= 26.463
  00:01:14 STEP 1511 H=1511:00 +CPU= 26.439
  00:02:10 STEP 1512 H=1512:00 +CPU= 45.344
Moving to projects directory: /var/lib/boinc/slots/8/ICMGGhq0f+001512
Moving to projects directory: /var/lib/boinc/slots/8/ICMSHhq0f+001512
Moving to projects directory: /var/lib/boinc/slots/8/ICMUAhq0f+001512
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12203028/ICMGGhq0f+001488
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12203028/ICMSHhq0f+001488
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12203028/ICMUAhq0f+001488
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12203028/ICMGGhq0f+001500
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12203028/ICMSHhq0f+001500
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12203028/ICMUAhq0f+001500
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0384_2017050100_123_986_12203028_1_r1556625311_62.zip
Uploading the intermediate file: upload_file_62.zip
  00:02:41 STEP 1513 H=1513:00 +CPU= 25.400
  00:03:15 STEP 1514 H=1514:00 +CPU= 26.769
  00:04:04 STEP 1515 H=1515:00 +CPU= 38.627
  00:04:37 STEP 1516 H=1516:00 +CPU= 26.135
  00:05:10 STEP 1517 H=1517:00 +CPU= 26.173
  00:05:59 STEP 1518 H=1518:00 +CPU= 39.179
  00:06:30 STEP 1519 H=1519:00 +CPU= 25.662
  00:07:01 STEP 1520 H=1520:00 +CPU= 25.783
  00:07:45 STEP 1521 H=1521:00 +CPU= 38.102
  00:08:17 STEP 1522 H=1522:00 +CPU= 26.760
  00:08:47 STEP 1523 H=1523:00 +CPU= 26.462
  00:09:37 STEP 1524 H=1524:00 +CPU= 43.164
Moving to projects directory: /var/lib/boinc/slots/8/ICMGGhq0f+001524
Moving to projects directory: /var/lib/boinc/slots/8/ICMSHhq0f+001524
Moving to projects directory: /var/lib/boinc/slots/8/ICMUAhq0f+001524
  00:10:08 STEP 1525 H=1525:00 +CPU= 26.244
  00:10:38 STEP 1526 H=1526:00 +CPU= 26.367
  00:11:24 STEP 1527 H=1527:00 +CPU= 39.325
  00:11:55 STEP 1528 H=1528:00 +CPU= 26.667
  00:12:25 STEP 1529 H=1529:00 +CPU= 25.233
  00:13:12 STEP 1530 H=1530:00 +CPU= 40.040
  00:13:42 STEP 1531 H=1531:00 +CPU= 25.769
  00:14:13 STEP 1532 H=1532:00 +CPU= 26.856
  00:14:57 STEP 1533 H=1533:00 +CPU= 38.368
  00:15:29 STEP 1534 H=1534:00 +CPU= 26.707
  00:16:01 STEP 1535 H=1535:00 +CPU= 27.820
  00:16:56 STEP 1536 H=1536:00 +CPU= 46.792
Moving to projects directory: /var/lib/boinc/slots/8/ICMGGhq0f+001536
Moving to projects directory: /var/lib/boinc/slots/8/ICMSHhq0f+001536
Moving to projects directory: /var/lib/boinc/slots/8/ICMUAhq0f+001536
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12203028/ICMGGhq0f+001512
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12203028/ICMSHhq0f+001512
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12203028/ICMUAhq0f+001512
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12203028/ICMGGhq0f+001524
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12203028/ICMSHhq0f+001524
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12203028/ICMUAhq0f+001524
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0384_2017050100_123_986_12203028_1_r1556625311_63.zip
Uploading the intermediate file: upload_file_63.zip
  00:17:27 STEP 1537 H=1537:00 +CPU= 26.974
  00:17:57 STEP 1538 H=1538:00 +CPU= 26.569
  00:18:43 STEP 1539 H=1539:00 +CPU= 39.530
  00:19:14 STEP 1540 H=1540:00 +CPU= 26.709
  00:19:42 STEP 1541 H=1541:00 +CPU= 23.570
  00:20:25 STEP 1542 H=1542:00 +CPU= 39.279
  00:20:53 STEP 1543 H=1543:00 +CPU= 24.779
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2017050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0384
(argv4) batchid: 986
(argv5) wuid: 12203028
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/8
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12203028
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/8/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/8/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_79309c237478e28dc003f376a90f63a2 to: /var/lib/boinc/slots/8/oifs_43r3_ps_0384_2017050100_123_986_12203028.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/8/oifs_43r3_ps_0384_2017050100_123_986_12203028.zip
ic_ancil_file: ic_ancil_12203028
ifsdata_file: ifsdata_12203028
climate_data_file: clim_data_12203028
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_7c6b7a5239e8fa5551203885b25978bb to: /var/lib/boinc/slots/8/ic_ancil_12203028.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/8/ic_ancil_12203028.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_0ec81e22fe81753eb3e03de66137c648 to: /var/lib/boinc/slots/8/ifsdata/ifsdata_12203028.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/8/ifsdata/ifsdata_12203028.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/8/159l_2/clim_data_12203028.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/8/159l_2/clim_data_12203028.zip
Checking for progress XML file: /var/lib/boinc/slots/8/progress_file_12203028.xml
Opened progress file ok : /var/lib/boinc/slots/8/progress_file_12203028.xml
-- Model is restarting --
Adjusting last_iter, 1543, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/8/progress_file_12203028.xml
last_cpu_time: 54370
upload_file_number: 64
last_iter: 1535
last_upload: 5529600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0384_2017050100_123_986_12203028_1_r1556625311
The child process has been launched with process id: 15220
Executing the command: /var/lib/boinc/slots/8/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:tweenis:1:1:15220:15220] [20230129:004236:1674952956:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:tweenis:1:1:15220:15220] [20230129:004236:1674952956:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:tweenis:1:1:15220:15220] [20230129:004236:1674952956:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:tweenis:1:1:15220:15220] [20230129:004236:1674952956:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:tweenis:1:1:15220:15220] [20230129:004236:1674952956:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:tweenis:1:1:15220:15220] [20230129:004236:1674952956:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:tweenis:1:1:15220:15220] [20230129:004236:1674952956:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:tweenis:1:1:15220:15220] [20230129:004236:1674952956:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:tweenis:1:1:15220:15220] [20230129:004236:1674952956:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:tweenis:1:1:15220:15220] [20230129:004236:1674952956:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:tweenis:1:1:15220:15220] [20230129:004236:1674952956:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:tweenis:1:1:15220:15220] [20230129:004236:1674952956:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:tweenis:1:1:15220:15220] [20230129:004236:1674952956:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:tweenis:1:1:15220:15220] [20230129:004236:1674952956:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:tweenis:1:1:15220:15220] [20230129:004236:1674952956:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:tweenis:1:1:15220:15220] [20230129:004236:1674952956:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:15220:15220] [20230129:004236:1674952956:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:15220:15220] [20230129:004236:1674952956:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:15220:15220] [20230129:004236:1674952956:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:15220:15220] [20230129:004236:1674952956:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:tweenis:1:1:15220:15220] [20230129:004236:1674952956:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:15220:15220] [20230129:004236:1674952956:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:15220:15220] [20230129:004236:1674952956:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:15220:15220] [20230129:004236:1674952956:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:15220:15220] [20230129:004236:1674952956:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:15220:15220] [20230129:004236:1674952956:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:15220:15220] [20230129:004236:1674952956:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:15220:15220] [20230129:004236:1674952956:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:15220:15220] [20230129:004236:1674952956:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:15220:15220] [20230129:004236:1674952956:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  00:43:09 STEP 1536 H=1536:00 +CPU= 30.035
  00:43:38 STEP 1537 H=1537:00 +CPU= 28.267
  00:44:08 STEP 1538 H=1538:00 +CPU= 30.833
  00:44:42 STEP 1539 H=1539:00 +CPU= 33.626
  00:45:08 STEP 1540 H=1540:00 +CPU= 25.302
  00:45:31 STEP 1541 H=1541:00 +CPU= 23.235
  00:46:06 STEP 1542 H=1542:00 +CPU= 35.449
  00:46:35 STEP 1543 H=1543:00 +CPU= 28.702
  00:47:00 STEP 1544 H=1544:00 +CPU= 25.253
  00:47:42 STEP 1545 H=1545:00 +CPU= 41.850
  00:48:07 STEP 1546 H=1546:00 +CPU= 24.784
  00:48:35 STEP 1547 H=1547:00 +CPU= 28.271
  00:49:18 STEP 1548 H=1548:00 +CPU= 42.135
Moving to projects directory: /var/lib/boinc/slots/8/ICMGGhq0f+001548
Moving to projects directory: /var/lib/boinc/slots/8/ICMSHhq0f+001548
Moving to projects directory: /var/lib/boinc/slots/8/ICMUAhq0f+001548
  00:49:47 STEP 1549 H=1549:00 +CPU= 29.690
  00:50:15 STEP 1550 H=1550:00 +CPU= 27.167
  00:50:51 STEP 1551 H=1551:00 +CPU= 35.958
  00:51:21 STEP 1552 H=1552:00 +CPU= 29.919
  00:51:45 STEP 1553 H=1553:00 +CPU= 24.790
  00:52:31 STEP 1554 H=1554:00 +CPU= 45.428
  00:52:57 STEP 1555 H=1555:00 +CPU= 25.621
  00:53:25 STEP 1556 H=1556:00 +CPU= 28.497
  00:54:04 STEP 1557 H=1557:00 +CPU= 38.786
  00:54:26 STEP 1558 H=1558:00 +CPU= 21.527
  00:54:51 STEP 1559 H=1559:00 +CPU= 25.931
  00:55:32 STEP 1560 H=1560:00 +CPU= 40.219
Moving to projects directory: /var/lib/boinc/slots/8/ICMGGhq0f+001560
Moving to projects directory: /var/lib/boinc/slots/8/ICMSHhq0f+001560
Moving to projects directory: /var/lib/boinc/slots/8/ICMUAhq0f+001560
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12203028/ICMGGhq0f+001536
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12203028/ICMSHhq0f+001536
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12203028/ICMUAhq0f+001536
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12203028/ICMGGhq0f+001548
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12203028/ICMSHhq0f+001548
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12203028/ICMUAhq0f+001548
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0384_2017050100_123_986_12203028_1_r1556625311_64.zip
Uploading the intermediate file: upload_file_64.zip
  00:55:59 STEP 1561 H=1561:00 +CPU= 26.408
  00:56:28 STEP 1562 H=1562:00 +CPU= 29.531
  00:57:06 STEP 1563 H=1563:00 +CPU= 38.085
  00:57:37 STEP 1564 H=1564:00 +CPU= 30.044
  00:58:01 STEP 1565 H=1565:00 +CPU= 24.825
  00:58:44 STEP 1566 H=1566:00 +CPU= 42.622
Abort request received from BOINC client, ending the child process

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:44:09 1531398 22304305 oifs_43r3_ps_0384_2017050100_123_986_12203028_1 3,196,800 30,136 0.0094
09 May 2023 16:44:04 1531398 22304305 oifs_43r3_ps_0384_2017050100_123_986_12203028_1 2,332,800 21,402 0.0092
09 May 2023 16:43:58 1531398 22304305 oifs_43r3_ps_0384_2017050100_123_986_12203028_1 1,468,800 12,625 0.0086


©2024 climateprediction.net