climateprediction.net home page
Task 22304303

Task 22304303

Name oifs_43r3_ps_0310_2013050100_123_982_12198954_1
Workunit 12198954
Created 27 Jan 2023, 20:45:43 UTC
Sent 27 Jan 2023, 20:46:14 UTC
Report deadline 28 Mar 2023, 20:46:14 UTC
Received 29 Jan 2023, 1:59:48 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1531398
Run time 15 hours 56 min 19 sec
CPU time 15 hours 22 min 41 sec
Validate state Invalid
Credit 4,475.52
Device peak FLOPS 7.92 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,803.66 MB
Peak swap size 4,978.08 MB
Peak disk usage 7,674.18 MB
Stderr 
<core_client_version>7.20.2</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
nis:1:1:12850:12850] [20230128:190545:1674932745:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:12850:12850] [20230128:190545:1674932745:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:12850:12850] [20230128:190545:1674932745:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:12850:12850] [20230128:190545:1674932745:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  19:06:23 STEP 1320 H=1320:00 +CPU= 32.678
  19:06:53 STEP 1321 H=1321:00 +CPU= 27.882
  19:07:25 STEP 1322 H=1322:00 +CPU= 30.852
  19:08:05 STEP 1323 H=1323:00 +CPU= 38.867
  19:08:35 STEP 1324 H=1324:00 +CPU= 28.929
  19:09:02 STEP 1325 H=1325:00 +CPU= 26.256
  19:09:43 STEP 1326 H=1326:00 +CPU= 40.235
  19:10:12 STEP 1327 H=1327:00 +CPU= 28.355
  19:10:37 STEP 1328 H=1328:00 +CPU= 24.860
  19:11:20 STEP 1329 H=1329:00 +CPU= 41.582
  19:11:47 STEP 1330 H=1330:00 +CPU= 26.552
  19:12:14 STEP 1331 H=1331:00 +CPU= 25.639
  19:13:01 STEP 1332 H=1332:00 +CPU= 45.705
Moving to projects directory: /var/lib/boinc/slots/3/ICMGGhq0f+001332
Moving to projects directory: /var/lib/boinc/slots/3/ICMSHhq0f+001332
Moving to projects directory: /var/lib/boinc/slots/3/ICMUAhq0f+001332
  19:13:26 STEP 1333 H=1333:00 +CPU= 24.687
  19:13:56 STEP 1334 H=1334:00 +CPU= 29.462
  19:14:36 STEP 1335 H=1335:00 +CPU= 39.389
  19:15:07 STEP 1336 H=1336:00 +CPU= 29.770
  19:15:35 STEP 1337 H=1337:00 +CPU= 27.295
  19:16:16 STEP 1338 H=1338:00 +CPU= 40.329
  19:16:46 STEP 1339 H=1339:00 +CPU= 28.591
  19:17:12 STEP 1340 H=1340:00 +CPU= 26.331
  19:17:53 STEP 1341 H=1341:00 +CPU= 39.655
  19:18:22 STEP 1342 H=1342:00 +CPU= 28.453
  19:18:48 STEP 1343 H=1343:00 +CPU= 25.647
  19:19:38 STEP 1344 H=1344:00 +CPU= 48.277
Moving to projects directory: /var/lib/boinc/slots/3/ICMGGhq0f+001344
Moving to projects directory: /var/lib/boinc/slots/3/ICMSHhq0f+001344
Moving to projects directory: /var/lib/boinc/slots/3/ICMUAhq0f+001344
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12198954/ICMGGhq0f+001320
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12198954/ICMSHhq0f+001320
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12198954/ICMUAhq0f+001320
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12198954/ICMGGhq0f+001332
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12198954/ICMSHhq0f+001332
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12198954/ICMUAhq0f+001332
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0310_2013050100_123_982_12198954_1_r1385707654_55.zip
Uploading the intermediate file: upload_file_55.zip
  19:20:05 STEP 1345 H=1345:00 +CPU= 26.110
  19:20:35 STEP 1346 H=1346:00 +CPU= 28.545
  19:21:18 STEP 1347 H=1347:00 +CPU= 41.212
  19:21:45 STEP 1348 H=1348:00 +CPU= 26.602
  19:22:14 STEP 1349 H=1349:00 +CPU= 27.690
  19:22:55 STEP 1350 H=1350:00 +CPU= 40.505
  19:23:24 STEP 1351 H=1351:00 +CPU= 28.176
  19:23:54 STEP 1352 H=1352:00 +CPU= 29.266
  19:24:35 STEP 1353 H=1353:00 +CPU= 39.347
  19:25:04 STEP 1354 H=1354:00 +CPU= 27.955
  19:25:32 STEP 1355 H=1355:00 +CPU= 26.885
  19:26:17 STEP 1356 H=1356:00 +CPU= 43.987
Moving to projects directory: /var/lib/boinc/slots/3/ICMGGhq0f+001356
Moving to projects directory: /var/lib/boinc/slots/3/ICMSHhq0f+001356
Moving to projects directory: /var/lib/boinc/slots/3/ICMUAhq0f+001356
  19:26:47 STEP 1357 H=1357:00 +CPU= 28.953
  19:27:14 STEP 1358 H=1358:00 +CPU= 26.541
  19:27:57 STEP 1359 H=1359:00 +CPU= 41.439
  19:28:25 STEP 1360 H=1360:00 +CPU= 27.472
  19:28:52 STEP 1361 H=1361:00 +CPU= 25.973
  19:29:37 STEP 1362 H=1362:00 +CPU= 43.997
  19:30:03 STEP 1363 H=1363:00 +CPU= 24.527
  19:30:33 STEP 1364 H=1364:00 +CPU= 28.968
  19:31:14 STEP 1365 H=1365:00 +CPU= 40.790
  19:31:42 STEP 1366 H=1366:00 +CPU= 26.967
  19:32:14 STEP 1367 H=1367:00 +CPU= 30.919
  19:33:01 STEP 1368 H=1368:00 +CPU= 45.892
Moving to projects directory: /var/lib/boinc/slots/3/ICMGGhq0f+001368
Moving to projects directory: /var/lib/boinc/slots/3/ICMSHhq0f+001368
Moving to projects directory: /var/lib/boinc/slots/3/ICMUAhq0f+001368
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12198954/ICMGGhq0f+001344
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12198954/ICMSHhq0f+001344
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12198954/ICMUAhq0f+001344
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12198954/ICMGGhq0f+001356
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12198954/ICMSHhq0f+001356
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12198954/ICMUAhq0f+001356
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0310_2013050100_123_982_12198954_1_r1385707654_56.zip
Uploading the intermediate file: upload_file_56.zip
  19:33:31 STEP 1369 H=1369:00 +CPU= 28.281
  19:34:00 STEP 1370 H=1370:00 +CPU= 28.032
  19:34:41 STEP 1371 H=1371:00 +CPU= 39.674
  19:35:13 STEP 1372 H=1372:00 +CPU= 31.074
  19:35:37 STEP 1373 H=1373:00 +CPU= 23.796
  19:36:24 STEP 1374 H=1374:00 +CPU= 45.085
  19:36:50 STEP 1375 H=1375:00 +CPU= 25.279
  19:37:18 STEP 1376 H=1376:00 +CPU= 27.657
  19:38:02 STEP 1377 H=1377:00 +CPU= 42.717
  19:38:27 STEP 1378 H=1378:00 +CPU= 24.413
  19:38:58 STEP 1379 H=1379:00 +CPU= 29.953
  19:39:43 STEP 1380 H=1380:00 +CPU= 43.725
Moving to projects directory: /var/lib/boinc/slots/3/ICMGGhq0f+001380
Moving to projects directory: /var/lib/boinc/slots/3/ICMSHhq0f+001380
Moving to projects directory: /var/lib/boinc/slots/3/ICMUAhq0f+001380
  19:40:14 STEP 1381 H=1381:00 +CPU= 29.325
  19:40:44 STEP 1382 H=1382:00 +CPU= 29.347
  19:41:26 STEP 1383 H=1383:00 +CPU= 40.307
  19:41:56 STEP 1384 H=1384:00 +CPU= 29.039
  19:42:24 STEP 1385 H=1385:00 +CPU= 27.316
  19:43:08 STEP 1386 H=1386:00 +CPU= 42.817
  19:43:37 STEP 1387 H=1387:00 +CPU= 27.835
  19:44:03 STEP 1388 H=1388:00 +CPU= 25.604
  19:44:46 STEP 1389 H=1389:00 +CPU= 42.014
  19:45:16 STEP 1390 H=1390:00 +CPU= 28.609
  19:45:44 STEP 1391 H=1391:00 +CPU= 27.508
  19:46:36 STEP 1392 H=1392:00 +CPU= 49.594
Moving to projects directory: /var/lib/boinc/slots/3/ICMGGhq0f+001392
Moving to projects directory: /var/lib/boinc/slots/3/ICMSHhq0f+001392
Moving to projects directory: /var/lib/boinc/slots/3/ICMUAhq0f+001392
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12198954/ICMGGhq0f+001368
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12198954/ICMSHhq0f+001368
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12198954/ICMUAhq0f+001368
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12198954/ICMGGhq0f+001380
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12198954/ICMSHhq0f+001380
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12198954/ICMUAhq0f+001380
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0310_2013050100_123_982_12198954_1_r1385707654_57.zip
Uploading the intermediate file: upload_file_57.zip
  19:47:03 STEP 1393 H=1393:00 +CPU= 26.055
  19:47:33 STEP 1394 H=1394:00 +CPU= 28.657
  19:48:17 STEP 1395 H=1395:00 +CPU= 42.769
  19:48:44 STEP 1396 H=1396:00 +CPU= 26.138
  19:49:13 STEP 1397 H=1397:00 +CPU= 28.688
  19:49:54 STEP 1398 H=1398:00 +CPU= 39.481
  19:50:26 STEP 1399 H=1399:00 +CPU= 30.723
  19:50:54 STEP 1400 H=1400:00 +CPU= 27.559
  19:51:35 STEP 1401 H=1401:00 +CPU= 39.531
  19:52:06 STEP 1402 H=1402:00 +CPU= 29.992
  19:52:31 STEP 1403 H=1403:00 +CPU= 24.362
  19:53:22 STEP 1404 H=1404:00 +CPU= 48.960
Moving to projects directory: /var/lib/boinc/slots/3/ICMGGhq0f+001404
Moving to projects directory: /var/lib/boinc/slots/3/ICMSHhq0f+001404
Moving to projects directory: /var/lib/boinc/slots/3/ICMUAhq0f+001404
  19:53:49 STEP 1405 H=1405:00 +CPU= 26.473
  19:54:18 STEP 1406 H=1406:00 +CPU= 28.192
  19:55:03 STEP 1407 H=1407:00 +CPU= 43.650
  19:55:31 STEP 1408 H=1408:00 +CPU= 26.765
  19:56:01 STEP 1409 H=1409:00 +CPU= 29.121
  19:56:44 STEP 1410 H=1410:00 +CPU= 41.941
  19:57:12 STEP 1411 H=1411:00 +CPU= 27.423
  19:57:42 STEP 1412 H=1412:00 +CPU= 29.151
  19:58:25 STEP 1413 H=1413:00 +CPU= 41.188
  19:58:54 STEP 1414 H=1414:00 +CPU= 28.534
  19:59:23 STEP 1415 H=1415:00 +CPU= 28.553
  20:00:15 STEP 1416 H=1416:00 +CPU= 49.811
Moving to projects directory: /var/lib/boinc/slots/3/ICMGGhq0f+001416
Moving to projects directory: /var/lib/boinc/slots/3/ICMSHhq0f+001416
Moving to projects directory: /var/lib/boinc/slots/3/ICMUAhq0f+001416
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12198954/ICMGGhq0f+001392
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12198954/ICMSHhq0f+001392
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12198954/ICMUAhq0f+001392
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12198954/ICMGGhq0f+001404
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12198954/ICMSHhq0f+001404
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12198954/ICMUAhq0f+001404
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0310_2013050100_123_982_12198954_1_r1385707654_58.zip
Uploading the intermediate file: upload_file_58.zip
  20:00:44 STEP 1417 H=1417:00 +CPU= 28.206
  20:01:13 STEP 1418 H=1418:00 +CPU= 27.807
  20:01:58 STEP 1419 H=1419:00 +CPU= 43.045
  20:02:24 STEP 1420 H=1420:00 +CPU= 25.731
  20:02:53 STEP 1421 H=1421:00 +CPU= 28.260
  20:03:38 STEP 1422 H=1422:00 +CPU= 43.288
  20:04:05 STEP 1423 H=1423:00 +CPU= 26.807
  20:04:37 STEP 1424 H=1424:00 +CPU= 30.742
  20:05:19 STEP 1425 H=1425:00 +CPU= 41.121
  20:05:51 STEP 1426 H=1426:00 +CPU= 30.545
  20:06:19 STEP 1427 H=1427:00 +CPU= 27.215
  20:07:07 STEP 1428 H=1428:00 +CPU= 46.217
Moving to projects directory: /var/lib/boinc/slots/3/ICMGGhq0f+001428
Moving to projects directory: /var/lib/boinc/slots/3/ICMSHhq0f+001428
Moving to projects directory: /var/lib/boinc/slots/3/ICMUAhq0f+001428
  20:07:35 STEP 1429 H=1429:00 +CPU= 28.016
  20:08:01 STEP 1430 H=1430:00 +CPU= 25.027
  20:08:47 STEP 1431 H=1431:00 +CPU= 44.832
  20:09:12 STEP 1432 H=1432:00 +CPU= 24.802
  20:09:44 STEP 1433 H=1433:00 +CPU= 30.050
  20:10:27 STEP 1434 H=1434:00 +CPU= 42.326
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2013050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0310
(argv4) batchid: 982
(argv5) wuid: 12198954
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/3
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12198954
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_51e36f62412ac2d77d4e043d081073de to: /var/lib/boinc/slots/3/oifs_43r3_ps_0310_2013050100_123_982_12198954.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/3/oifs_43r3_ps_0310_2013050100_123_982_12198954.zip
ic_ancil_file: ic_ancil_12198954
ifsdata_file: ifsdata_12198954
climate_data_file: clim_data_12198954
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_664bc9e12ca318fb783a2a093a9e002c to: /var/lib/boinc/slots/3/ic_ancil_12198954.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/3/ic_ancil_12198954.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_eaae81093054a4fd34557e46600308cc to: /var/lib/boinc/slots/3/ifsdata/ifsdata_12198954.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/3/ifsdata/ifsdata_12198954.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/3/159l_2/clim_data_12198954.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/3/159l_2/clim_data_12198954.zip
Checking for progress XML file: /var/lib/boinc/slots/3/progress_file_12198954.xml
Opened progress file ok : /var/lib/boinc/slots/3/progress_file_12198954.xml
-- Model is restarting --
Adjusting last_iter, 1435, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/3/progress_file_12198954.xml
last_cpu_time: 49817
upload_file_number: 59
last_iter: 1415
last_upload: 5097600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0310_2013050100_123_982_12198954_1_r1385707654
The child process has been launched with process id: 13839
Executing the command: /var/lib/boinc/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:tweenis:1:1:13839:13839] [20230128:204329:1674938609:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:tweenis:1:1:13839:13839] [20230128:204329:1674938609:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:tweenis:1:1:13839:13839] [20230128:204329:1674938609:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:tweenis:1:1:13839:13839] [20230128:204329:1674938609:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:tweenis:1:1:13839:13839] [20230128:204329:1674938609:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:tweenis:1:1:13839:13839] [20230128:204329:1674938609:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:tweenis:1:1:13839:13839] [20230128:204329:1674938609:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:tweenis:1:1:13839:13839] [20230128:204329:1674938609:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:tweenis:1:1:13839:13839] [20230128:204329:1674938609:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:tweenis:1:1:13839:13839] [20230128:204329:1674938609:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:tweenis:1:1:13839:13839] [20230128:204329:1674938609:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:tweenis:1:1:13839:13839] [20230128:204329:1674938609:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:tweenis:1:1:13839:13839] [20230128:204329:1674938609:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:tweenis:1:1:13839:13839] [20230128:204329:1674938609:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:tweenis:1:1:13839:13839] [20230128:204329:1674938609:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:tweenis:1:1:13839:13839] [20230128:204329:1674938609:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:13839:13839] [20230128:204329:1674938609:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:13839:13839] [20230128:204329:1674938609:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:13839:13839] [20230128:204329:1674938609:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:13839:13839] [20230128:204329:1674938609:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:tweenis:1:1:13839:13839] [20230128:204329:1674938609:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:13839:13839] [20230128:204329:1674938609:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:13839:13839] [20230128:204329:1674938609:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:13839:13839] [20230128:204329:1674938609:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:13839:13839] [20230128:204329:1674938609:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:13839:13839] [20230128:204329:1674938609:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:13839:13839] [20230128:204329:1674938609:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:13839:13839] [20230128:204329:1674938609:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:13839:13839] [20230128:204329:1674938609:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:13839:13839] [20230128:204329:1674938609:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  20:44:07 STEP 1416 H=1416:00 +CPU= 33.348
  20:44:37 STEP 1417 H=1417:00 +CPU= 27.874
  20:45:08 STEP 1418 H=1418:00 +CPU= 29.768
  20:45:48 STEP 1419 H=1419:00 +CPU= 38.263
  20:46:19 STEP 1420 H=1420:00 +CPU= 28.369
  20:46:45 STEP 1421 H=1421:00 +CPU= 25.295
  20:47:29 STEP 1422 H=1422:00 +CPU= 41.812
  20:47:59 STEP 1423 H=1423:00 +CPU= 27.836
  20:48:24 STEP 1424 H=1424:00 +CPU= 24.253
  20:49:07 STEP 1425 H=1425:00 +CPU= 41.139
  20:49:35 STEP 1426 H=1426:00 +CPU= 26.392
  20:50:04 STEP 1427 H=1427:00 +CPU= 27.242
  20:50:51 STEP 1428 H=1428:00 +CPU= 43.407
Moving to projects directory: /var/lib/boinc/slots/3/ICMGGhq0f+001428
Moving to projects directory: /var/lib/boinc/slots/3/ICMSHhq0f+001428
Moving to projects directory: /var/lib/boinc/slots/3/ICMUAhq0f+001428
  20:51:19 STEP 1429 H=1429:00 +CPU= 26.414
  20:51:49 STEP 1430 H=1430:00 +CPU= 29.267
  20:52:30 STEP 1431 H=1431:00 +CPU= 38.753
  20:52:59 STEP 1432 H=1432:00 +CPU= 28.047
  20:53:29 STEP 1433 H=1433:00 +CPU= 27.960
  20:54:10 STEP 1434 H=1434:00 +CPU= 38.586
  20:54:41 STEP 1435 H=1435:00 +CPU= 28.809
  20:55:08 STEP 1436 H=1436:00 +CPU= 25.446
  20:55:53 STEP 1437 H=1437:00 +CPU= 40.964
  20:56:21 STEP 1438 H=1438:00 +CPU= 26.186
  20:56:52 STEP 1439 H=1439:00 +CPU= 28.126
  20:57:41 STEP 1440 H=1440:00 +CPU= 44.701
Moving to projects directory: /var/lib/boinc/slots/3/ICMGGhq0f+001440
Moving to projects directory: /var/lib/boinc/slots/3/ICMSHhq0f+001440
Moving to projects directory: /var/lib/boinc/slots/3/ICMUAhq0f+001440
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12198954/ICMGGhq0f+001416
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12198954/ICMSHhq0f+001416
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12198954/ICMUAhq0f+001416
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12198954/ICMGGhq0f+001428
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12198954/ICMSHhq0f+001428
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12198954/ICMUAhq0f+001428
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0310_2013050100_123_982_12198954_1_r1385707654_59.zip
Uploading the intermediate file: upload_file_59.zip
  20:58:13 STEP 1441 H=1441:00 +CPU= 27.626
  20:58:43 STEP 1442 H=1442:00 +CPU= 27.364
  20:59:27 STEP 1443 H=1443:00 +CPU= 40.118
  20:59:57 STEP 1444 H=1444:00 +CPU= 26.770
  21:00:26 STEP 1445 H=1445:00 +CPU= 26.514
  21:01:11 STEP 1446 H=1446:00 +CPU= 41.006
  21:01:42 STEP 1447 H=1447:00 +CPU= 26.879
  21:02:11 STEP 1448 H=1448:00 +CPU= 25.805
  21:02:55 STEP 1449 H=1449:00 +CPU= 39.651
  21:03:24 STEP 1450 H=1450:00 +CPU= 26.469
  21:03:56 STEP 1451 H=1451:00 +CPU= 27.873
  21:04:41 STEP 1452 H=1452:00 +CPU= 41.249
Moving to projects directory: /var/lib/boinc/slots/3/ICMGGhq0f+001452
Moving to projects directory: /var/lib/boinc/slots/3/ICMSHhq0f+001452
Moving to projects directory: /var/lib/boinc/slots/3/ICMUAhq0f+001452
  21:05:12 STEP 1453 H=1453:00 +CPU= 28.138
  21:05:42 STEP 1454 H=1454:00 +CPU= 27.432
  21:06:24 STEP 1455 H=1455:00 +CPU= 38.453
  21:06:56 STEP 1456 H=1456:00 +CPU= 28.767
  21:07:22 STEP 1457 H=1457:00 +CPU= 24.619
  21:08:06 STEP 1458 H=1458:00 +CPU= 41.642
  21:08:35 STEP 1459 H=1459:00 +CPU= 26.725
  21:09:03 STEP 1460 H=1460:00 +CPU= 25.872
  21:09:50 STEP 1461 H=1461:00 +CPU= 42.692
  21:10:16 STEP 1462 H=1462:00 +CPU= 24.696
  21:10:49 STEP 1463 H=1463:00 +CPU= 30.243
  21:11:38 STEP 1464 H=1464:00 +CPU= 45.277
Moving to projects directory: /var/lib/boinc/slots/3/ICMGGhq0f+001464
Moving to projects directory: /var/lib/boinc/slots/3/ICMSHhq0f+001464
Moving to projects directory: /var/lib/boinc/slots/3/ICMUAhq0f+001464
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12198954/ICMGGhq0f+001440
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12198954/ICMSHhq0f+001440
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12198954/ICMUAhq0f+001440
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12198954/ICMGGhq0f+001452
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12198954/ICMSHhq0f+001452
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12198954/ICMUAhq0f+001452
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0310_2013050100_123_982_12198954_1_r1385707654_60.zip
Uploading the intermediate file: upload_file_60.zip
  21:12:09 STEP 1465 H=1465:00 +CPU= 27.945
  21:12:41 STEP 1466 H=1466:00 +CPU= 29.625
  21:13:22 STEP 1467 H=1467:00 +CPU= 38.492
  21:13:52 STEP 1468 H=1468:00 +CPU= 28.084
  21:14:20 STEP 1469 H=1469:00 +CPU= 25.946
  21:15:03 STEP 1470 H=1470:00 +CPU= 40.348
  21:15:33 STEP 1471 H=1471:00 +CPU= 27.946
  21:16:00 STEP 1472 H=1472:00 +CPU= 24.415
  21:16:46 STEP 1473 H=1473:00 +CPU= 42.758
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2013050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0310
(argv4) batchid: 982
(argv5) wuid: 12198954
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/3
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12198954
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_51e36f62412ac2d77d4e043d081073de to: /var/lib/boinc/slots/3/oifs_43r3_ps_0310_2013050100_123_982_12198954.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/3/oifs_43r3_ps_0310_2013050100_123_982_12198954.zip
ic_ancil_file: ic_ancil_12198954
ifsdata_file: ifsdata_12198954
climate_data_file: clim_data_12198954
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_664bc9e12ca318fb783a2a093a9e002c to: /var/lib/boinc/slots/3/ic_ancil_12198954.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/3/ic_ancil_12198954.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_eaae81093054a4fd34557e46600308cc to: /var/lib/boinc/slots/3/ifsdata/ifsdata_12198954.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/3/ifsdata/ifsdata_12198954.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/3/159l_2/clim_data_12198954.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/3/159l_2/clim_data_12198954.zip
Checking for progress XML file: /var/lib/boinc/slots/3/progress_file_12198954.xml
Opened progress file ok : /var/lib/boinc/slots/3/progress_file_12198954.xml
-- Model is restarting --
Adjusting last_iter, 1474, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/3/progress_file_12198954.xml
last_cpu_time: 51683
upload_file_number: 61
last_iter: 1463
last_upload: 5270400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0310_2013050100_123_982_12198954_1_r1385707654
The child process has been launched with process id: 7466
Executing the command: /var/lib/boinc/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:tweenis:1:1:7466:7466] [20230128:223341:1674945221:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:tweenis:1:1:7466:7466] [20230128:223341:1674945221:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:tweenis:1:1:7466:7466] [20230128:223341:1674945221:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:tweenis:1:1:7466:7466] [20230128:223341:1674945221:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:tweenis:1:1:7466:7466] [20230128:223341:1674945221:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:tweenis:1:1:7466:7466] [20230128:223341:1674945221:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:tweenis:1:1:7466:7466] [20230128:223341:1674945221:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:tweenis:1:1:7466:7466] [20230128:223341:1674945221:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:tweenis:1:1:7466:7466] [20230128:223341:1674945221:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:tweenis:1:1:7466:7466] [20230128:223341:1674945221:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:tweenis:1:1:7466:7466] [20230128:223341:1674945221:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:tweenis:1:1:7466:7466] [20230128:223341:1674945221:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:tweenis:1:1:7466:7466] [20230128:223341:1674945221:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:tweenis:1:1:7466:7466] [20230128:223341:1674945221:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:tweenis:1:1:7466:7466] [20230128:223341:1674945221:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:tweenis:1:1:7466:7466] [20230128:223341:1674945221:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:7466:7466] [20230128:223341:1674945221:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:7466:7466] [20230128:223341:1674945221:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:7466:7466] [20230128:223341:1674945221:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:7466:7466] [20230128:223341:1674945221:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:tweenis:1:1:7466:7466] [20230128:223341:1674945221:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:7466:7466] [20230128:223341:1674945221:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:7466:7466] [20230128:223341:1674945221:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:7466:7466] [20230128:223341:1674945221:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:7466:7466] [20230128:223341:1674945221:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:7466:7466] [20230128:223341:1674945221:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:7466:7466] [20230128:223341:1674945221:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:7466:7466] [20230128:223341:1674945221:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:7466:7466] [20230128:223341:1674945221:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:7466:7466] [20230128:223341:1674945221:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  22:34:24 STEP 1464 H=1464:00 +CPU= 34.449
  22:34:57 STEP 1465 H=1465:00 +CPU= 28.317
  22:35:30 STEP 1466 H=1466:00 +CPU= 29.454
  22:36:16 STEP 1467 H=1467:00 +CPU= 40.546
  22:36:50 STEP 1468 H=1468:00 +CPU= 31.504
  22:37:16 STEP 1469 H=1469:00 +CPU= 25.525
  22:38:04 STEP 1470 H=1470:00 +CPU= 44.468
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2013050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0310
(argv4) batchid: 982
(argv5) wuid: 12198954
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/3
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12198954
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_51e36f62412ac2d77d4e043d081073de to: /var/lib/boinc/slots/3/oifs_43r3_ps_0310_2013050100_123_982_12198954.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/3/oifs_43r3_ps_0310_2013050100_123_982_12198954.zip
ic_ancil_file: ic_ancil_12198954
ifsdata_file: ifsdata_12198954
climate_data_file: clim_data_12198954
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_664bc9e12ca318fb783a2a093a9e002c to: /var/lib/boinc/slots/3/ic_ancil_12198954.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/3/ic_ancil_12198954.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_eaae81093054a4fd34557e46600308cc to: /var/lib/boinc/slots/3/ifsdata/ifsdata_12198954.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/3/ifsdata/ifsdata_12198954.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/3/159l_2/clim_data_12198954.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/3/159l_2/clim_data_12198954.zip
Checking for progress XML file: /var/lib/boinc/slots/3/progress_file_12198954.xml
Opened progress file ok : /var/lib/boinc/slots/3/progress_file_12198954.xml
-- Model is restarting --
Adjusting last_iter, 1471, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/3/progress_file_12198954.xml
last_cpu_time: 51925
upload_file_number: 61
last_iter: 1463
last_upload: 5270400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0310_2013050100_123_982_12198954_1_r1385707654
The child process has been launched with process id: 9758
Executing the command: /var/lib/boinc/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:tweenis:1:1:9758:9758] [20230128:233204:1674948724:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:tweenis:1:1:9758:9758] [20230128:233204:1674948724:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:tweenis:1:1:9758:9758] [20230128:233204:1674948724:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:tweenis:1:1:9758:9758] [20230128:233204:1674948724:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:tweenis:1:1:9758:9758] [20230128:233204:1674948724:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:tweenis:1:1:9758:9758] [20230128:233204:1674948724:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:tweenis:1:1:9758:9758] [20230128:233204:1674948724:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:tweenis:1:1:9758:9758] [20230128:233204:1674948724:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:tweenis:1:1:9758:9758] [20230128:233204:1674948724:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:tweenis:1:1:9758:9758] [20230128:233204:1674948724:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:tweenis:1:1:9758:9758] [20230128:233204:1674948724:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:tweenis:1:1:9758:9758] [20230128:233204:1674948724:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:tweenis:1:1:9758:9758] [20230128:233204:1674948724:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:tweenis:1:1:9758:9758] [20230128:233204:1674948724:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:tweenis:1:1:9758:9758] [20230128:233204:1674948724:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:tweenis:1:1:9758:9758] [20230128:233204:1674948724:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:9758:9758] [20230128:233204:1674948724:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:9758:9758] [20230128:233204:1674948724:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:9758:9758] [20230128:233204:1674948724:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:9758:9758] [20230128:233204:1674948724:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:tweenis:1:1:9758:9758] [20230128:233204:1674948724:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:9758:9758] [20230128:233204:1674948724:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:9758:9758] [20230128:233204:1674948724:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:9758:9758] [20230128:233204:1674948724:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:9758:9758] [20230128:233204:1674948724:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:9758:9758] [20230128:233204:1674948724:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:9758:9758] [20230128:233204:1674948724:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:9758:9758] [20230128:233204:1674948724:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:9758:9758] [20230128:233204:1674948724:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:9758:9758] [20230128:233204:1674948724:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  23:32:41 STEP 1464 H=1464:00 +CPU= 33.157
  23:33:13 STEP 1465 H=1465:00 +CPU= 29.270
  23:33:45 STEP 1466 H=1466:00 +CPU= 28.195
  23:34:28 STEP 1467 H=1467:00 +CPU= 38.501
  23:35:01 STEP 1468 H=1468:00 +CPU= 29.611
  23:35:27 STEP 1469 H=1469:00 +CPU= 25.617
  23:36:11 STEP 1470 H=1470:00 +CPU= 42.476
  23:36:42 STEP 1471 H=1471:00 +CPU= 27.908
  23:37:11 STEP 1472 H=1472:00 +CPU= 25.681
  23:37:55 STEP 1473 H=1473:00 +CPU= 41.820
  23:38:23 STEP 1474 H=1474:00 +CPU= 26.301
  23:38:53 STEP 1475 H=1475:00 +CPU= 29.133
  23:39:38 STEP 1476 H=1476:00 +CPU= 42.057
Moving to projects directory: /var/lib/boinc/slots/3/ICMGGhq0f+001476
Moving to projects directory: /var/lib/boinc/slots/3/ICMSHhq0f+001476
Moving to projects directory: /var/lib/boinc/slots/3/ICMUAhq0f+001476
  23:40:08 STEP 1477 H=1477:00 +CPU= 28.572
  23:40:40 STEP 1478 H=1478:00 +CPU= 29.960
  23:41:21 STEP 1479 H=1479:00 +CPU= 38.287
  23:41:53 STEP 1480 H=1480:00 +CPU= 28.781
  23:42:24 STEP 1481 H=1481:00 +CPU= 25.390
  23:43:17 STEP 1482 H=1482:00 +CPU= 41.962
  23:43:52 STEP 1483 H=1483:00 +CPU= 28.997
  23:44:24 STEP 1484 H=1484:00 +CPU= 27.438
  23:45:16 STEP 1485 H=1485:00 +CPU= 42.675
  23:45:48 STEP 1486 H=1486:00 +CPU= 25.630
  23:46:25 STEP 1487 H=1487:00 +CPU= 29.330
  23:47:24 STEP 1488 H=1488:00 +CPU= 46.009
Moving to projects directory: /var/lib/boinc/slots/3/ICMGGhq0f+001488
Moving to projects directory: /var/lib/boinc/slots/3/ICMSHhq0f+001488
Moving to projects directory: /var/lib/boinc/slots/3/ICMUAhq0f+001488
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12198954/ICMGGhq0f+001464
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12198954/ICMSHhq0f+001464
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12198954/ICMUAhq0f+001464
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12198954/ICMGGhq0f+001476
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12198954/ICMSHhq0f+001476
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12198954/ICMUAhq0f+001476
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0310_2013050100_123_982_12198954_1_r1385707654_61.zip
Uploading the intermediate file: upload_file_61.zip
  23:47:58 STEP 1489 H=1489:00 +CPU= 27.146
  23:48:33 STEP 1490 H=1490:00 +CPU= 27.796
  23:49:18 STEP 1491 H=1491:00 +CPU= 37.970
  23:49:52 STEP 1492 H=1492:00 +CPU= 27.795
  23:50:22 STEP 1493 H=1493:00 +CPU= 24.359
  23:51:13 STEP 1494 H=1494:00 +CPU= 40.903
  23:51:44 STEP 1495 H=1495:00 +CPU= 25.280
  23:52:14 STEP 1496 H=1496:00 +CPU= 24.365
  23:53:03 STEP 1497 H=1497:00 +CPU= 38.845
  23:53:32 STEP 1498 H=1498:00 +CPU= 24.395
  23:54:02 STEP 1499 H=1499:00 +CPU= 24.882
  23:54:51 STEP 1500 H=1500:00 +CPU= 40.280
Moving to projects directory: /var/lib/boinc/slots/3/ICMGGhq0f+001500
Moving to projects directory: /var/lib/boinc/slots/3/ICMSHhq0f+001500
Moving to projects directory: /var/lib/boinc/slots/3/ICMUAhq0f+001500
  23:55:22 STEP 1501 H=1501:00 +CPU= 24.883
  23:55:55 STEP 1502 H=1502:00 +CPU= 26.304
  23:56:42 STEP 1503 H=1503:00 +CPU= 37.123
  23:57:16 STEP 1504 H=1504:00 +CPU= 26.840
  23:57:47 STEP 1505 H=1505:00 +CPU= 25.067
  23:58:35 STEP 1506 H=1506:00 +CPU= 38.171
  23:59:08 STEP 1507 H=1507:00 +CPU= 25.986
  23:59:39 STEP 1508 H=1508:00 +CPU= 25.033
  00:00:29 STEP 1509 H=1509:00 +CPU= 39.710
  00:01:00 STEP 1510 H=1510:00 +CPU= 25.154
  00:01:33 STEP 1511 H=1511:00 +CPU= 26.958
  00:02:28 STEP 1512 H=1512:00 +CPU= 45.047
Moving to projects directory: /var/lib/boinc/slots/3/ICMGGhq0f+001512
Moving to projects directory: /var/lib/boinc/slots/3/ICMSHhq0f+001512
Moving to projects directory: /var/lib/boinc/slots/3/ICMUAhq0f+001512
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12198954/ICMGGhq0f+001488
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12198954/ICMSHhq0f+001488
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12198954/ICMUAhq0f+001488
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12198954/ICMGGhq0f+001500
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12198954/ICMSHhq0f+001500
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12198954/ICMUAhq0f+001500
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0310_2013050100_123_982_12198954_1_r1385707654_62.zip
Uploading the intermediate file: upload_file_62.zip
  00:03:00 STEP 1513 H=1513:00 +CPU= 25.634
  00:03:36 STEP 1514 H=1514:00 +CPU= 27.543
  00:04:24 STEP 1515 H=1515:00 +CPU= 38.420
  00:04:58 STEP 1516 H=1516:00 +CPU= 26.525
  00:05:29 STEP 1517 H=1517:00 +CPU= 25.107
  00:06:20 STEP 1518 H=1518:00 +CPU= 39.707
  00:06:51 STEP 1519 H=1519:00 +CPU= 25.471
  00:07:22 STEP 1520 H=1520:00 +CPU= 26.371
  00:08:07 STEP 1521 H=1521:00 +CPU= 39.187
  00:08:38 STEP 1522 H=1522:00 +CPU= 26.658
  00:09:08 STEP 1523 H=1523:00 +CPU= 26.158
  00:10:01 STEP 1524 H=1524:00 +CPU= 44.203
Moving to projects directory: /var/lib/boinc/slots/3/ICMGGhq0f+001524
Moving to projects directory: /var/lib/boinc/slots/3/ICMSHhq0f+001524
Moving to projects directory: /var/lib/boinc/slots/3/ICMUAhq0f+001524
  00:10:32 STEP 1525 H=1525:00 +CPU= 26.230
  00:11:02 STEP 1526 H=1526:00 +CPU= 25.713
  00:11:51 STEP 1527 H=1527:00 +CPU= 40.311
  00:12:21 STEP 1528 H=1528:00 +CPU= 26.404
  00:12:51 STEP 1529 H=1529:00 +CPU= 25.417
  00:13:39 STEP 1530 H=1530:00 +CPU= 40.300
  00:14:09 STEP 1531 H=1531:00 +CPU= 26.130
  00:14:39 STEP 1532 H=1532:00 +CPU= 25.667
  00:15:27 STEP 1533 H=1533:00 +CPU= 40.021
  00:15:59 STEP 1534 H=1534:00 +CPU= 27.441
  00:16:32 STEP 1535 H=1535:00 +CPU= 27.618
  00:17:28 STEP 1536 H=1536:00 +CPU= 46.889
Moving to projects directory: /var/lib/boinc/slots/3/ICMGGhq0f+001536
Moving to projects directory: /var/lib/boinc/slots/3/ICMSHhq0f+001536
Moving to projects directory: /var/lib/boinc/slots/3/ICMUAhq0f+001536
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12198954/ICMGGhq0f+001512
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12198954/ICMSHhq0f+001512
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12198954/ICMUAhq0f+001512
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12198954/ICMGGhq0f+001524
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12198954/ICMSHhq0f+001524
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12198954/ICMUAhq0f+001524
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0310_2013050100_123_982_12198954_1_r1385707654_63.zip
Uploading the intermediate file: upload_file_63.zip
  00:17:59 STEP 1537 H=1537:00 +CPU= 25.789
  00:18:29 STEP 1538 H=1538:00 +CPU= 25.459
  00:19:17 STEP 1539 H=1539:00 +CPU= 40.844
  00:19:45 STEP 1540 H=1540:00 +CPU= 24.235
  00:20:14 STEP 1541 H=1541:00 +CPU= 25.899
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2013050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0310
(argv4) batchid: 982
(argv5) wuid: 12198954
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/3
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12198954
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_51e36f62412ac2d77d4e043d081073de to: /var/lib/boinc/slots/3/oifs_43r3_ps_0310_2013050100_123_982_12198954.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/3/oifs_43r3_ps_0310_2013050100_123_982_12198954.zip
ic_ancil_file: ic_ancil_12198954
ifsdata_file: ifsdata_12198954
climate_data_file: clim_data_12198954
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_664bc9e12ca318fb783a2a093a9e002c to: /var/lib/boinc/slots/3/ic_ancil_12198954.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/3/ic_ancil_12198954.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_eaae81093054a4fd34557e46600308cc to: /var/lib/boinc/slots/3/ifsdata/ifsdata_12198954.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/3/ifsdata/ifsdata_12198954.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/3/159l_2/clim_data_12198954.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/3/159l_2/clim_data_12198954.zip
Checking for progress XML file: /var/lib/boinc/slots/3/progress_file_12198954.xml
Opened progress file ok : /var/lib/boinc/slots/3/progress_file_12198954.xml
-- Model is restarting --
Adjusting last_iter, 1542, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/3/progress_file_12198954.xml
last_cpu_time: 54393
upload_file_number: 64
last_iter: 1535
last_upload: 5529600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0310_2013050100_123_982_12198954_1_r1385707654
The child process has been launched with process id: 15218
Executing the command: /var/lib/boinc/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:tweenis:1:1:15218:15218] [20230129:004236:1674952956:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:tweenis:1:1:15218:15218] [20230129:004236:1674952956:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:tweenis:1:1:15218:15218] [20230129:004236:1674952956:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:tweenis:1:1:15218:15218] [20230129:004236:1674952956:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:tweenis:1:1:15218:15218] [20230129:004236:1674952956:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:tweenis:1:1:15218:15218] [20230129:004236:1674952956:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:tweenis:1:1:15218:15218] [20230129:004236:1674952956:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:tweenis:1:1:15218:15218] [20230129:004236:1674952956:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:tweenis:1:1:15218:15218] [20230129:004236:1674952956:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:tweenis:1:1:15218:15218] [20230129:004236:1674952956:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:tweenis:1:1:15218:15218] [20230129:004236:1674952956:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:tweenis:1:1:15218:15218] [20230129:004236:1674952956:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:tweenis:1:1:15218:15218] [20230129:004236:1674952956:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:tweenis:1:1:15218:15218] [20230129:004236:1674952956:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:tweenis:1:1:15218:15218] [20230129:004236:1674952956:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:tweenis:1:1:15218:15218] [20230129:004236:1674952956:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:15218:15218] [20230129:004236:1674952956:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:15218:15218] [20230129:004236:1674952956:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:15218:15218] [20230129:004236:1674952956:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:15218:15218] [20230129:004236:1674952956:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:tweenis:1:1:15218:15218] [20230129:004236:1674952956:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:15218:15218] [20230129:004236:1674952956:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:15218:15218] [20230129:004236:1674952956:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:15218:15218] [20230129:004236:1674952956:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:15218:15218] [20230129:004236:1674952956:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:15218:15218] [20230129:004236:1674952956:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:15218:15218] [20230129:004236:1674952956:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:15218:15218] [20230129:004236:1674952956:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:15218:15218] [20230129:004236:1674952956:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:15218:15218] [20230129:004236:1674952956:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  00:43:09 STEP 1536 H=1536:00 +CPU= 30.290
  00:43:38 STEP 1537 H=1537:00 +CPU= 28.214
  00:44:06 STEP 1538 H=1538:00 +CPU= 28.732
  00:44:45 STEP 1539 H=1539:00 +CPU= 38.576
  00:45:14 STEP 1540 H=1540:00 +CPU= 29.311
  00:45:38 STEP 1541 H=1541:00 +CPU= 23.374
  00:46:16 STEP 1542 H=1542:00 +CPU= 37.767
  00:46:40 STEP 1543 H=1543:00 +CPU= 24.114
  00:47:01 STEP 1544 H=1544:00 +CPU= 21.136
  00:47:41 STEP 1545 H=1545:00 +CPU= 39.609
  00:48:06 STEP 1546 H=1546:00 +CPU= 25.202
  00:48:33 STEP 1547 H=1547:00 +CPU= 26.664
  00:49:13 STEP 1548 H=1548:00 +CPU= 39.232
Moving to projects directory: /var/lib/boinc/slots/3/ICMGGhq0f+001548
Moving to projects directory: /var/lib/boinc/slots/3/ICMSHhq0f+001548
Moving to projects directory: /var/lib/boinc/slots/3/ICMUAhq0f+001548
  00:49:41 STEP 1549 H=1549:00 +CPU= 28.073
  00:50:09 STEP 1550 H=1550:00 +CPU= 28.187
  00:50:47 STEP 1551 H=1551:00 +CPU= 37.580
  00:51:13 STEP 1552 H=1552:00 +CPU= 25.901
  00:51:38 STEP 1553 H=1553:00 +CPU= 24.711
  00:52:14 STEP 1554 H=1554:00 +CPU= 35.888
  00:52:43 STEP 1555 H=1555:00 +CPU= 29.325
  00:53:08 STEP 1556 H=1556:00 +CPU= 24.529
  00:53:49 STEP 1557 H=1557:00 +CPU= 40.764
  00:54:13 STEP 1558 H=1558:00 +CPU= 24.289
  00:54:43 STEP 1559 H=1559:00 +CPU= 29.934
  00:55:25 STEP 1560 H=1560:00 +CPU= 41.947
Moving to projects directory: /var/lib/boinc/slots/3/ICMGGhq0f+001560
Moving to projects directory: /var/lib/boinc/slots/3/ICMSHhq0f+001560
Moving to projects directory: /var/lib/boinc/slots/3/ICMUAhq0f+001560
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12198954/ICMGGhq0f+001536
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12198954/ICMSHhq0f+001536
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12198954/ICMUAhq0f+001536
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12198954/ICMGGhq0f+001548
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12198954/ICMSHhq0f+001548
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12198954/ICMUAhq0f+001548
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0310_2013050100_123_982_12198954_1_r1385707654_64.zip
Uploading the intermediate file: upload_file_64.zip
  00:55:50 STEP 1561 H=1561:00 +CPU= 24.188
  00:56:17 STEP 1562 H=1562:00 +CPU= 26.565
  00:56:53 STEP 1563 H=1563:00 +CPU= 36.236
  00:57:19 STEP 1564 H=1564:00 +CPU= 26.328
  00:57:47 STEP 1565 H=1565:00 +CPU= 27.121
  00:58:23 STEP 1566 H=1566:00 +CPU= 36.490
Abort request received from BOINC client, ending the child process

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:44:07 1531398 22304303 oifs_43r3_ps_0310_2013050100_123_982_12198954_1 3,196,800 29,613 0.0093
09 May 2023 16:44:04 1531398 22304303 oifs_43r3_ps_0310_2013050100_123_982_12198954_1 2,332,800 20,998 0.0090
09 May 2023 16:43:59 1531398 22304303 oifs_43r3_ps_0310_2013050100_123_982_12198954_1 1,468,800 12,322 0.0084


©2024 climateprediction.net