climateprediction.net home page
Task 22304300

Task 22304300

Name oifs_43r3_ps_0725_1996050100_123_965_12182369_1
Workunit 12182369
Created 27 Jan 2023, 20:45:43 UTC
Sent 27 Jan 2023, 20:46:14 UTC
Report deadline 28 Mar 2023, 20:46:14 UTC
Received 29 Jan 2023, 1:59:48 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1531398
Run time 15 hours 58 min 48 sec
CPU time 15 hours 25 min 22 sec
Validate state Invalid
Credit 4,475.52
Device peak FLOPS 7.92 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,801.41 MB
Peak swap size 4,978.07 MB
Peak disk usage 7,674.29 MB
Stderr 
<core_client_version>7.20.2</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
slots/7/ICMUAhq0f+001320
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12182369/ICMGGhq0f+001296
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12182369/ICMSHhq0f+001296
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12182369/ICMUAhq0f+001296
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12182369/ICMGGhq0f+001308
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12182369/ICMSHhq0f+001308
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12182369/ICMUAhq0f+001308
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0725_1996050100_123_965_12182369_1_r881573072_54.zip
Uploading the intermediate file: upload_file_54.zip
  19:06:52 STEP 1321 H=1321:00 +CPU= 27.606
  19:07:24 STEP 1322 H=1322:00 +CPU= 29.968
  19:08:03 STEP 1323 H=1323:00 +CPU= 38.746
  19:08:34 STEP 1324 H=1324:00 +CPU= 29.794
  19:09:01 STEP 1325 H=1325:00 +CPU= 26.114
  19:09:42 STEP 1326 H=1326:00 +CPU= 40.808
  19:10:12 STEP 1327 H=1327:00 +CPU= 28.770
  19:10:37 STEP 1328 H=1328:00 +CPU= 24.090
  19:11:20 STEP 1329 H=1329:00 +CPU= 41.964
  19:11:46 STEP 1330 H=1330:00 +CPU= 26.132
  19:12:13 STEP 1331 H=1331:00 +CPU= 26.246
  19:13:00 STEP 1332 H=1332:00 +CPU= 45.432
Moving to projects directory: /var/lib/boinc/slots/7/ICMGGhq0f+001332
Moving to projects directory: /var/lib/boinc/slots/7/ICMSHhq0f+001332
Moving to projects directory: /var/lib/boinc/slots/7/ICMUAhq0f+001332
  19:13:26 STEP 1333 H=1333:00 +CPU= 25.875
  19:13:56 STEP 1334 H=1334:00 +CPU= 29.044
  19:14:35 STEP 1335 H=1335:00 +CPU= 37.653
  19:15:04 STEP 1336 H=1336:00 +CPU= 28.177
  19:15:32 STEP 1337 H=1337:00 +CPU= 26.883
  19:16:12 STEP 1338 H=1338:00 +CPU= 39.199
  19:16:43 STEP 1339 H=1339:00 +CPU= 29.650
  19:17:10 STEP 1340 H=1340:00 +CPU= 26.990
  19:17:52 STEP 1341 H=1341:00 +CPU= 41.213
  19:18:22 STEP 1342 H=1342:00 +CPU= 28.530
  19:18:49 STEP 1343 H=1343:00 +CPU= 25.755
  19:19:38 STEP 1344 H=1344:00 +CPU= 47.715
Moving to projects directory: /var/lib/boinc/slots/7/ICMGGhq0f+001344
Moving to projects directory: /var/lib/boinc/slots/7/ICMSHhq0f+001344
Moving to projects directory: /var/lib/boinc/slots/7/ICMUAhq0f+001344
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12182369/ICMGGhq0f+001320
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12182369/ICMSHhq0f+001320
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12182369/ICMUAhq0f+001320
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12182369/ICMGGhq0f+001332
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12182369/ICMSHhq0f+001332
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12182369/ICMUAhq0f+001332
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0725_1996050100_123_965_12182369_1_r881573072_55.zip
Uploading the intermediate file: upload_file_55.zip
  19:20:05 STEP 1345 H=1345:00 +CPU= 25.970
  19:20:34 STEP 1346 H=1346:00 +CPU= 28.127
  19:21:17 STEP 1347 H=1347:00 +CPU= 41.700
  19:21:42 STEP 1348 H=1348:00 +CPU= 25.171
  19:22:11 STEP 1349 H=1349:00 +CPU= 28.072
  19:22:54 STEP 1350 H=1350:00 +CPU= 41.202
  19:23:22 STEP 1351 H=1351:00 +CPU= 26.921
  19:23:52 STEP 1352 H=1352:00 +CPU= 29.522
  19:24:33 STEP 1353 H=1353:00 +CPU= 39.715
  19:25:03 STEP 1354 H=1354:00 +CPU= 29.227
  19:25:32 STEP 1355 H=1355:00 +CPU= 27.736
  19:26:18 STEP 1356 H=1356:00 +CPU= 44.857
Moving to projects directory: /var/lib/boinc/slots/7/ICMGGhq0f+001356
Moving to projects directory: /var/lib/boinc/slots/7/ICMSHhq0f+001356
Moving to projects directory: /var/lib/boinc/slots/7/ICMUAhq0f+001356
  19:26:48 STEP 1357 H=1357:00 +CPU= 29.529
  19:27:16 STEP 1358 H=1358:00 +CPU= 26.423
  19:27:59 STEP 1359 H=1359:00 +CPU= 42.096
  19:28:28 STEP 1360 H=1360:00 +CPU= 27.797
  19:28:54 STEP 1361 H=1361:00 +CPU= 25.262
  19:29:39 STEP 1362 H=1362:00 +CPU= 43.713
  19:30:05 STEP 1363 H=1363:00 +CPU= 24.980
  19:30:35 STEP 1364 H=1364:00 +CPU= 29.458
  19:31:17 STEP 1365 H=1365:00 +CPU= 40.916
  19:31:45 STEP 1366 H=1366:00 +CPU= 27.182
  19:32:17 STEP 1367 H=1367:00 +CPU= 31.284
  19:33:04 STEP 1368 H=1368:00 +CPU= 45.853
Moving to projects directory: /var/lib/boinc/slots/7/ICMGGhq0f+001368
Moving to projects directory: /var/lib/boinc/slots/7/ICMSHhq0f+001368
Moving to projects directory: /var/lib/boinc/slots/7/ICMUAhq0f+001368
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12182369/ICMGGhq0f+001344
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12182369/ICMSHhq0f+001344
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12182369/ICMUAhq0f+001344
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12182369/ICMGGhq0f+001356
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12182369/ICMSHhq0f+001356
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12182369/ICMUAhq0f+001356
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0725_1996050100_123_965_12182369_1_r881573072_56.zip
Uploading the intermediate file: upload_file_56.zip
  19:33:35 STEP 1369 H=1369:00 +CPU= 29.466
  19:34:04 STEP 1370 H=1370:00 +CPU= 27.987
  19:34:47 STEP 1371 H=1371:00 +CPU= 41.235
  19:35:17 STEP 1372 H=1372:00 +CPU= 29.849
  19:35:41 STEP 1373 H=1373:00 +CPU= 23.225
  19:36:28 STEP 1374 H=1374:00 +CPU= 45.663
  19:36:55 STEP 1375 H=1375:00 +CPU= 25.863
  19:37:25 STEP 1376 H=1376:00 +CPU= 29.532
  19:38:09 STEP 1377 H=1377:00 +CPU= 42.375
  19:38:38 STEP 1378 H=1378:00 +CPU= 28.326
  19:39:08 STEP 1379 H=1379:00 +CPU= 28.819
  19:39:52 STEP 1380 H=1380:00 +CPU= 43.601
Moving to projects directory: /var/lib/boinc/slots/7/ICMGGhq0f+001380
Moving to projects directory: /var/lib/boinc/slots/7/ICMSHhq0f+001380
Moving to projects directory: /var/lib/boinc/slots/7/ICMUAhq0f+001380
  19:40:24 STEP 1381 H=1381:00 +CPU= 30.268
  19:40:54 STEP 1382 H=1382:00 +CPU= 29.023
  19:41:37 STEP 1383 H=1383:00 +CPU= 41.567
  19:42:07 STEP 1384 H=1384:00 +CPU= 29.355
  19:42:34 STEP 1385 H=1385:00 +CPU= 26.543
  19:43:18 STEP 1386 H=1386:00 +CPU= 42.692
  19:43:47 STEP 1387 H=1387:00 +CPU= 27.970
  19:44:14 STEP 1388 H=1388:00 +CPU= 26.220
  19:44:59 STEP 1389 H=1389:00 +CPU= 43.555
  19:45:26 STEP 1390 H=1390:00 +CPU= 26.496
  19:45:56 STEP 1391 H=1391:00 +CPU= 28.368
  19:46:46 STEP 1392 H=1392:00 +CPU= 48.867
Moving to projects directory: /var/lib/boinc/slots/7/ICMGGhq0f+001392
Moving to projects directory: /var/lib/boinc/slots/7/ICMSHhq0f+001392
Moving to projects directory: /var/lib/boinc/slots/7/ICMUAhq0f+001392
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12182369/ICMGGhq0f+001368
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12182369/ICMSHhq0f+001368
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12182369/ICMUAhq0f+001368
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12182369/ICMGGhq0f+001380
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12182369/ICMSHhq0f+001380
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12182369/ICMUAhq0f+001380
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0725_1996050100_123_965_12182369_1_r881573072_57.zip
Uploading the intermediate file: upload_file_57.zip
  19:47:12 STEP 1393 H=1393:00 +CPU= 25.044
  19:47:44 STEP 1394 H=1394:00 +CPU= 31.107
  19:48:25 STEP 1395 H=1395:00 +CPU= 39.750
  19:48:55 STEP 1396 H=1396:00 +CPU= 29.485
  19:49:25 STEP 1397 H=1397:00 +CPU= 29.031
  19:50:07 STEP 1398 H=1398:00 +CPU= 40.853
  19:50:38 STEP 1399 H=1399:00 +CPU= 29.685
  19:51:05 STEP 1400 H=1400:00 +CPU= 26.424
  19:51:51 STEP 1401 H=1401:00 +CPU= 44.454
  19:52:18 STEP 1402 H=1402:00 +CPU= 26.586
  19:52:45 STEP 1403 H=1403:00 +CPU= 26.536
  19:53:34 STEP 1404 H=1404:00 +CPU= 47.938
Moving to projects directory: /var/lib/boinc/slots/7/ICMGGhq0f+001404
Moving to projects directory: /var/lib/boinc/slots/7/ICMSHhq0f+001404
Moving to projects directory: /var/lib/boinc/slots/7/ICMUAhq0f+001404
  19:53:59 STEP 1405 H=1405:00 +CPU= 24.201
  19:54:29 STEP 1406 H=1406:00 +CPU= 29.175
  19:55:12 STEP 1407 H=1407:00 +CPU= 41.443
  19:55:41 STEP 1408 H=1408:00 +CPU= 27.900
  19:56:11 STEP 1409 H=1409:00 +CPU= 29.364
  19:56:54 STEP 1410 H=1410:00 +CPU= 41.880
  19:57:23 STEP 1411 H=1411:00 +CPU= 28.625
  19:57:53 STEP 1412 H=1412:00 +CPU= 28.855
  19:58:35 STEP 1413 H=1413:00 +CPU= 40.894
  19:59:05 STEP 1414 H=1414:00 +CPU= 28.285
  19:59:33 STEP 1415 H=1415:00 +CPU= 26.953
  20:00:24 STEP 1416 H=1416:00 +CPU= 50.090
Moving to projects directory: /var/lib/boinc/slots/7/ICMGGhq0f+001416
Moving to projects directory: /var/lib/boinc/slots/7/ICMSHhq0f+001416
Moving to projects directory: /var/lib/boinc/slots/7/ICMUAhq0f+001416
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12182369/ICMGGhq0f+001392
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12182369/ICMSHhq0f+001392
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12182369/ICMUAhq0f+001392
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12182369/ICMGGhq0f+001404
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12182369/ICMSHhq0f+001404
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12182369/ICMUAhq0f+001404
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0725_1996050100_123_965_12182369_1_r881573072_58.zip
Uploading the intermediate file: upload_file_58.zip
  20:00:52 STEP 1417 H=1417:00 +CPU= 26.270
  20:01:23 STEP 1418 H=1418:00 +CPU= 30.515
  20:02:09 STEP 1419 H=1419:00 +CPU= 43.881
  20:02:37 STEP 1420 H=1420:00 +CPU= 27.424
  20:03:07 STEP 1421 H=1421:00 +CPU= 28.989
  20:03:48 STEP 1422 H=1422:00 +CPU= 40.070
  20:04:17 STEP 1423 H=1423:00 +CPU= 28.303
  20:04:48 STEP 1424 H=1424:00 +CPU= 29.979
  20:05:30 STEP 1425 H=1425:00 +CPU= 40.744
  20:06:01 STEP 1426 H=1426:00 +CPU= 29.701
  20:06:27 STEP 1427 H=1427:00 +CPU= 26.061
  20:07:17 STEP 1428 H=1428:00 +CPU= 48.327
Moving to projects directory: /var/lib/boinc/slots/7/ICMGGhq0f+001428
Moving to projects directory: /var/lib/boinc/slots/7/ICMSHhq0f+001428
Moving to projects directory: /var/lib/boinc/slots/7/ICMUAhq0f+001428
  20:07:45 STEP 1429 H=1429:00 +CPU= 26.872
  20:08:13 STEP 1430 H=1430:00 +CPU= 27.059
  20:08:57 STEP 1431 H=1431:00 +CPU= 43.119
  20:09:25 STEP 1432 H=1432:00 +CPU= 26.944
  20:09:57 STEP 1433 H=1433:00 +CPU= 30.931
  20:10:40 STEP 1434 H=1434:00 +CPU= 41.448
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1996050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0725
(argv4) batchid: 965
(argv5) wuid: 12182369
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/7
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12182369
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/7/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/7/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_611bb152d584b7ab14b51b5eb264bec9 to: /var/lib/boinc/slots/7/oifs_43r3_ps_0725_1996050100_123_965_12182369.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/7/oifs_43r3_ps_0725_1996050100_123_965_12182369.zip
ic_ancil_file: ic_ancil_12182369
ifsdata_file: ifsdata_12182369
climate_data_file: clim_data_12182369
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_a7991fb673670b4598075f2a4e0530a6 to: /var/lib/boinc/slots/7/ic_ancil_12182369.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/7/ic_ancil_12182369.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_c39c5297a8316fe84b15f53cc5aa22a4 to: /var/lib/boinc/slots/7/ifsdata/ifsdata_12182369.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/7/ifsdata/ifsdata_12182369.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/7/159l_2/clim_data_12182369.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/7/159l_2/clim_data_12182369.zip
Checking for progress XML file: /var/lib/boinc/slots/7/progress_file_12182369.xml
Opened progress file ok : /var/lib/boinc/slots/7/progress_file_12182369.xml
-- Model is restarting --
Adjusting last_iter, 1434, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/7/progress_file_12182369.xml
last_cpu_time: 49959
upload_file_number: 59
last_iter: 1415
last_upload: 5097600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0725_1996050100_123_965_12182369_1_r881573072
The child process has been launched with process id: 13838
Executing the command: /var/lib/boinc/slots/7/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:tweenis:1:1:13838:13838] [20230128:204329:1674938609:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:tweenis:1:1:13838:13838] [20230128:204329:1674938609:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:tweenis:1:1:13838:13838] [20230128:204329:1674938609:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:tweenis:1:1:13838:13838] [20230128:204329:1674938609:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:tweenis:1:1:13838:13838] [20230128:204329:1674938609:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:tweenis:1:1:13838:13838] [20230128:204329:1674938609:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:tweenis:1:1:13838:13838] [20230128:204329:1674938609:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:tweenis:1:1:13838:13838] [20230128:204329:1674938609:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:tweenis:1:1:13838:13838] [20230128:204329:1674938609:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:tweenis:1:1:13838:13838] [20230128:204329:1674938609:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:tweenis:1:1:13838:13838] [20230128:204329:1674938609:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:tweenis:1:1:13838:13838] [20230128:204329:1674938609:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:tweenis:1:1:13838:13838] [20230128:204329:1674938609:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:tweenis:1:1:13838:13838] [20230128:204329:1674938609:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:tweenis:1:1:13838:13838] [20230128:204329:1674938609:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:tweenis:1:1:13838:13838] [20230128:204329:1674938609:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:13838:13838] [20230128:204329:1674938609:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:13838:13838] [20230128:204329:1674938609:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:13838:13838] [20230128:204329:1674938609:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:13838:13838] [20230128:204329:1674938609:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:tweenis:1:1:13838:13838] [20230128:204329:1674938609:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:13838:13838] [20230128:204329:1674938609:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:13838:13838] [20230128:204329:1674938609:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:13838:13838] [20230128:204329:1674938609:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:13838:13838] [20230128:204329:1674938609:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:13838:13838] [20230128:204329:1674938609:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:13838:13838] [20230128:204329:1674938609:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:13838:13838] [20230128:204329:1674938609:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:13838:13838] [20230128:204329:1674938609:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:13838:13838] [20230128:204329:1674938609:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  20:44:07 STEP 1416 H=1416:00 +CPU= 32.549
  20:44:36 STEP 1417 H=1417:00 +CPU= 27.093
  20:45:06 STEP 1418 H=1418:00 +CPU= 28.027
  20:45:45 STEP 1419 H=1419:00 +CPU= 37.529
  20:46:15 STEP 1420 H=1420:00 +CPU= 27.550
  20:46:41 STEP 1421 H=1421:00 +CPU= 25.609
  20:47:21 STEP 1422 H=1422:00 +CPU= 37.547
  20:47:51 STEP 1423 H=1423:00 +CPU= 28.193
  20:48:16 STEP 1424 H=1424:00 +CPU= 23.674
  20:48:58 STEP 1425 H=1425:00 +CPU= 40.076
  20:49:26 STEP 1426 H=1426:00 +CPU= 26.011
  20:49:51 STEP 1427 H=1427:00 +CPU= 24.182
  20:50:39 STEP 1428 H=1428:00 +CPU= 44.274
Moving to projects directory: /var/lib/boinc/slots/7/ICMGGhq0f+001428
Moving to projects directory: /var/lib/boinc/slots/7/ICMSHhq0f+001428
Moving to projects directory: /var/lib/boinc/slots/7/ICMUAhq0f+001428
  20:51:04 STEP 1429 H=1429:00 +CPU= 24.031
  20:51:33 STEP 1430 H=1430:00 +CPU= 26.984
  20:52:13 STEP 1431 H=1431:00 +CPU= 37.897
  20:52:40 STEP 1432 H=1432:00 +CPU= 25.962
  20:53:08 STEP 1433 H=1433:00 +CPU= 26.981
  20:53:49 STEP 1434 H=1434:00 +CPU= 38.308
  20:54:16 STEP 1435 H=1435:00 +CPU= 24.607
  20:54:45 STEP 1436 H=1436:00 +CPU= 27.120
  20:55:25 STEP 1437 H=1437:00 +CPU= 37.598
  20:55:53 STEP 1438 H=1438:00 +CPU= 26.250
  20:56:21 STEP 1439 H=1439:00 +CPU= 25.690
  20:57:11 STEP 1440 H=1440:00 +CPU= 45.952
Moving to projects directory: /var/lib/boinc/slots/7/ICMGGhq0f+001440
Moving to projects directory: /var/lib/boinc/slots/7/ICMSHhq0f+001440
Moving to projects directory: /var/lib/boinc/slots/7/ICMUAhq0f+001440
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12182369/ICMGGhq0f+001416
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12182369/ICMSHhq0f+001416
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12182369/ICMUAhq0f+001416
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12182369/ICMGGhq0f+001428
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12182369/ICMSHhq0f+001428
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12182369/ICMUAhq0f+001428
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0725_1996050100_123_965_12182369_1_r881573072_59.zip
Uploading the intermediate file: upload_file_59.zip
  20:57:38 STEP 1441 H=1441:00 +CPU= 23.894
  20:58:08 STEP 1442 H=1442:00 +CPU= 26.672
  20:58:51 STEP 1443 H=1443:00 +CPU= 39.439
  20:59:17 STEP 1444 H=1444:00 +CPU= 23.511
  20:59:45 STEP 1445 H=1445:00 +CPU= 25.618
  21:00:27 STEP 1446 H=1446:00 +CPU= 39.334
  21:00:55 STEP 1447 H=1447:00 +CPU= 25.167
  21:01:24 STEP 1448 H=1448:00 +CPU= 26.357
  21:02:09 STEP 1449 H=1449:00 +CPU= 38.616
  21:02:39 STEP 1450 H=1450:00 +CPU= 26.762
  21:03:07 STEP 1451 H=1451:00 +CPU= 25.198
  21:03:55 STEP 1452 H=1452:00 +CPU= 42.850
Moving to projects directory: /var/lib/boinc/slots/7/ICMGGhq0f+001452
Moving to projects directory: /var/lib/boinc/slots/7/ICMSHhq0f+001452
Moving to projects directory: /var/lib/boinc/slots/7/ICMUAhq0f+001452
  21:04:21 STEP 1453 H=1453:00 +CPU= 24.218
  21:04:48 STEP 1454 H=1454:00 +CPU= 24.711
  21:05:34 STEP 1455 H=1455:00 +CPU= 41.160
  21:06:00 STEP 1456 H=1456:00 +CPU= 23.602
  21:06:28 STEP 1457 H=1457:00 +CPU= 25.675
  21:07:13 STEP 1458 H=1458:00 +CPU= 40.712
  21:07:38 STEP 1459 H=1459:00 +CPU= 23.970
  21:08:07 STEP 1460 H=1460:00 +CPU= 27.407
  21:08:49 STEP 1461 H=1461:00 +CPU= 38.212
  21:09:19 STEP 1462 H=1462:00 +CPU= 27.004
  21:09:49 STEP 1463 H=1463:00 +CPU= 28.030
  21:10:37 STEP 1464 H=1464:00 +CPU= 44.426
Moving to projects directory: /var/lib/boinc/slots/7/ICMGGhq0f+001464
Moving to projects directory: /var/lib/boinc/slots/7/ICMSHhq0f+001464
Moving to projects directory: /var/lib/boinc/slots/7/ICMUAhq0f+001464
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12182369/ICMGGhq0f+001440
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12182369/ICMSHhq0f+001440
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12182369/ICMUAhq0f+001440
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12182369/ICMGGhq0f+001452
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12182369/ICMSHhq0f+001452
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12182369/ICMUAhq0f+001452
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0725_1996050100_123_965_12182369_1_r881573072_60.zip
Uploading the intermediate file: upload_file_60.zip
  21:11:07 STEP 1465 H=1465:00 +CPU= 27.703
  21:11:33 STEP 1466 H=1466:00 +CPU= 23.823
  21:12:17 STEP 1467 H=1467:00 +CPU= 40.876
  21:12:48 STEP 1468 H=1468:00 +CPU= 28.194
  21:13:12 STEP 1469 H=1469:00 +CPU= 22.481
  21:13:58 STEP 1470 H=1470:00 +CPU= 42.897
  21:14:24 STEP 1471 H=1471:00 +CPU= 23.597
  21:14:52 STEP 1472 H=1472:00 +CPU= 26.268
  21:15:38 STEP 1473 H=1473:00 +CPU= 42.054
  21:16:03 STEP 1474 H=1474:00 +CPU= 23.896
  21:16:32 STEP 1475 H=1475:00 +CPU= 26.754
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1996050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0725
(argv4) batchid: 965
(argv5) wuid: 12182369
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/7
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12182369
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/7/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/7/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_611bb152d584b7ab14b51b5eb264bec9 to: /var/lib/boinc/slots/7/oifs_43r3_ps_0725_1996050100_123_965_12182369.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/7/oifs_43r3_ps_0725_1996050100_123_965_12182369.zip
ic_ancil_file: ic_ancil_12182369
ifsdata_file: ifsdata_12182369
climate_data_file: clim_data_12182369
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_a7991fb673670b4598075f2a4e0530a6 to: /var/lib/boinc/slots/7/ic_ancil_12182369.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/7/ic_ancil_12182369.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_c39c5297a8316fe84b15f53cc5aa22a4 to: /var/lib/boinc/slots/7/ifsdata/ifsdata_12182369.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/7/ifsdata/ifsdata_12182369.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/7/159l_2/clim_data_12182369.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/7/159l_2/clim_data_12182369.zip
Checking for progress XML file: /var/lib/boinc/slots/7/progress_file_12182369.xml
Opened progress file ok : /var/lib/boinc/slots/7/progress_file_12182369.xml
-- Model is restarting --
Adjusting last_iter, 1476, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/7/progress_file_12182369.xml
last_cpu_time: 51823
upload_file_number: 61
last_iter: 1463
last_upload: 5270400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0725_1996050100_123_965_12182369_1_r881573072
The child process has been launched with process id: 7464
Executing the command: /var/lib/boinc/slots/7/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:tweenis:1:1:7464:7464] [20230128:223341:1674945221:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:tweenis:1:1:7464:7464] [20230128:223341:1674945221:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:tweenis:1:1:7464:7464] [20230128:223341:1674945221:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:tweenis:1:1:7464:7464] [20230128:223341:1674945221:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:tweenis:1:1:7464:7464] [20230128:223341:1674945221:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:tweenis:1:1:7464:7464] [20230128:223341:1674945221:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:tweenis:1:1:7464:7464] [20230128:223341:1674945221:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:tweenis:1:1:7464:7464] [20230128:223341:1674945221:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:tweenis:1:1:7464:7464] [20230128:223341:1674945221:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:tweenis:1:1:7464:7464] [20230128:223341:1674945221:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:tweenis:1:1:7464:7464] [20230128:223341:1674945221:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:tweenis:1:1:7464:7464] [20230128:223341:1674945221:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:tweenis:1:1:7464:7464] [20230128:223341:1674945221:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:tweenis:1:1:7464:7464] [20230128:223341:1674945221:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:tweenis:1:1:7464:7464] [20230128:223341:1674945221:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:tweenis:1:1:7464:7464] [20230128:223341:1674945221:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:7464:7464] [20230128:223341:1674945221:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:7464:7464] [20230128:223341:1674945221:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:7464:7464] [20230128:223341:1674945221:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:7464:7464] [20230128:223341:1674945221:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:tweenis:1:1:7464:7464] [20230128:223341:1674945221:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:7464:7464] [20230128:223341:1674945221:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:7464:7464] [20230128:223341:1674945221:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:7464:7464] [20230128:223341:1674945221:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:7464:7464] [20230128:223341:1674945221:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:7464:7464] [20230128:223341:1674945221:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:7464:7464] [20230128:223341:1674945221:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:7464:7464] [20230128:223341:1674945221:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:7464:7464] [20230128:223341:1674945221:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:7464:7464] [20230128:223341:1674945221:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  22:34:22 STEP 1464 H=1464:00 +CPU= 33.784
  22:34:54 STEP 1465 H=1465:00 +CPU= 26.281
  22:35:23 STEP 1466 H=1466:00 +CPU= 27.504
  22:36:08 STEP 1467 H=1467:00 +CPU= 39.270
  22:36:42 STEP 1468 H=1468:00 +CPU= 30.090
  22:37:08 STEP 1469 H=1469:00 +CPU= 25.363
  22:37:56 STEP 1470 H=1470:00 +CPU= 45.068
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1996050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0725
(argv4) batchid: 965
(argv5) wuid: 12182369
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/7
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12182369
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/7/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/7/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_611bb152d584b7ab14b51b5eb264bec9 to: /var/lib/boinc/slots/7/oifs_43r3_ps_0725_1996050100_123_965_12182369.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/7/oifs_43r3_ps_0725_1996050100_123_965_12182369.zip
ic_ancil_file: ic_ancil_12182369
ifsdata_file: ifsdata_12182369
climate_data_file: clim_data_12182369
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_a7991fb673670b4598075f2a4e0530a6 to: /var/lib/boinc/slots/7/ic_ancil_12182369.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/7/ic_ancil_12182369.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_c39c5297a8316fe84b15f53cc5aa22a4 to: /var/lib/boinc/slots/7/ifsdata/ifsdata_12182369.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/7/ifsdata/ifsdata_12182369.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/7/159l_2/clim_data_12182369.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/7/159l_2/clim_data_12182369.zip
Checking for progress XML file: /var/lib/boinc/slots/7/progress_file_12182369.xml
Opened progress file ok : /var/lib/boinc/slots/7/progress_file_12182369.xml
-- Model is restarting --
Adjusting last_iter, 1471, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/7/progress_file_12182369.xml
last_cpu_time: 52065
upload_file_number: 61
last_iter: 1463
last_upload: 5270400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0725_1996050100_123_965_12182369_1_r881573072
The child process has been launched with process id: 9757
Executing the command: /var/lib/boinc/slots/7/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:tweenis:1:1:9757:9757] [20230128:233204:1674948724:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:tweenis:1:1:9757:9757] [20230128:233204:1674948724:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:tweenis:1:1:9757:9757] [20230128:233204:1674948724:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:tweenis:1:1:9757:9757] [20230128:233204:1674948724:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:tweenis:1:1:9757:9757] [20230128:233204:1674948724:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:tweenis:1:1:9757:9757] [20230128:233204:1674948724:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:tweenis:1:1:9757:9757] [20230128:233204:1674948724:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:tweenis:1:1:9757:9757] [20230128:233204:1674948724:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:tweenis:1:1:9757:9757] [20230128:233204:1674948724:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:tweenis:1:1:9757:9757] [20230128:233204:1674948724:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:tweenis:1:1:9757:9757] [20230128:233204:1674948724:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:tweenis:1:1:9757:9757] [20230128:233204:1674948724:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:tweenis:1:1:9757:9757] [20230128:233204:1674948724:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:tweenis:1:1:9757:9757] [20230128:233204:1674948724:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:tweenis:1:1:9757:9757] [20230128:233204:1674948724:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:tweenis:1:1:9757:9757] [20230128:233204:1674948724:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:9757:9757] [20230128:233204:1674948724:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:9757:9757] [20230128:233204:1674948724:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:9757:9757] [20230128:233204:1674948724:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:9757:9757] [20230128:233204:1674948724:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:tweenis:1:1:9757:9757] [20230128:233204:1674948724:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:9757:9757] [20230128:233204:1674948724:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:9757:9757] [20230128:233204:1674948724:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:9757:9757] [20230128:233204:1674948724:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:9757:9757] [20230128:233204:1674948724:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:9757:9757] [20230128:233204:1674948724:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:9757:9757] [20230128:233204:1674948724:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:9757:9757] [20230128:233204:1674948724:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:9757:9757] [20230128:233204:1674948724:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:9757:9757] [20230128:233204:1674948724:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  23:32:39 STEP 1464 H=1464:00 +CPU= 31.974
  23:33:11 STEP 1465 H=1465:00 +CPU= 29.174
  23:33:45 STEP 1466 H=1466:00 +CPU= 28.969
  23:34:27 STEP 1467 H=1467:00 +CPU= 37.692
  23:34:59 STEP 1468 H=1468:00 +CPU= 29.431
  23:35:25 STEP 1469 H=1469:00 +CPU= 25.954
  23:36:08 STEP 1470 H=1470:00 +CPU= 41.893
  23:36:40 STEP 1471 H=1471:00 +CPU= 28.172
  23:37:07 STEP 1472 H=1472:00 +CPU= 24.675
  23:37:52 STEP 1473 H=1473:00 +CPU= 42.614
  23:38:18 STEP 1474 H=1474:00 +CPU= 24.504
  23:38:48 STEP 1475 H=1475:00 +CPU= 28.805
  23:39:33 STEP 1476 H=1476:00 +CPU= 43.205
Moving to projects directory: /var/lib/boinc/slots/7/ICMGGhq0f+001476
Moving to projects directory: /var/lib/boinc/slots/7/ICMSHhq0f+001476
Moving to projects directory: /var/lib/boinc/slots/7/ICMUAhq0f+001476
  23:40:02 STEP 1477 H=1477:00 +CPU= 27.009
  23:40:34 STEP 1478 H=1478:00 +CPU= 29.528
  23:41:15 STEP 1479 H=1479:00 +CPU= 38.621
  23:41:47 STEP 1480 H=1480:00 +CPU= 28.512
  23:42:19 STEP 1481 H=1481:00 +CPU= 26.110
  23:43:09 STEP 1482 H=1482:00 +CPU= 40.906
  23:43:47 STEP 1483 H=1483:00 +CPU= 30.090
  23:44:17 STEP 1484 H=1484:00 +CPU= 26.201
  23:45:09 STEP 1485 H=1485:00 +CPU= 42.572
  23:45:41 STEP 1486 H=1486:00 +CPU= 26.574
  23:46:18 STEP 1487 H=1487:00 +CPU= 28.776
  23:47:13 STEP 1488 H=1488:00 +CPU= 45.269
Moving to projects directory: /var/lib/boinc/slots/7/ICMGGhq0f+001488
Moving to projects directory: /var/lib/boinc/slots/7/ICMSHhq0f+001488
Moving to projects directory: /var/lib/boinc/slots/7/ICMUAhq0f+001488
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12182369/ICMGGhq0f+001464
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12182369/ICMSHhq0f+001464
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12182369/ICMUAhq0f+001464
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12182369/ICMGGhq0f+001476
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12182369/ICMSHhq0f+001476
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12182369/ICMUAhq0f+001476
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0725_1996050100_123_965_12182369_1_r881573072_61.zip
Uploading the intermediate file: upload_file_61.zip
  23:47:46 STEP 1489 H=1489:00 +CPU= 26.564
  23:48:20 STEP 1490 H=1490:00 +CPU= 28.209
  23:49:05 STEP 1491 H=1491:00 +CPU= 38.122
  23:49:39 STEP 1492 H=1492:00 +CPU= 28.003
  23:50:11 STEP 1493 H=1493:00 +CPU= 25.731
  23:50:58 STEP 1494 H=1494:00 +CPU= 39.347
  23:51:30 STEP 1495 H=1495:00 +CPU= 27.072
  23:51:59 STEP 1496 H=1496:00 +CPU= 24.374
  23:52:46 STEP 1497 H=1497:00 +CPU= 37.561
  23:53:16 STEP 1498 H=1498:00 +CPU= 24.660
  23:53:45 STEP 1499 H=1499:00 +CPU= 24.412
  23:54:35 STEP 1500 H=1500:00 +CPU= 41.268
Moving to projects directory: /var/lib/boinc/slots/7/ICMGGhq0f+001500
Moving to projects directory: /var/lib/boinc/slots/7/ICMSHhq0f+001500
Moving to projects directory: /var/lib/boinc/slots/7/ICMUAhq0f+001500
  23:55:05 STEP 1501 H=1501:00 +CPU= 24.369
  23:55:36 STEP 1502 H=1502:00 +CPU= 25.501
  23:56:21 STEP 1503 H=1503:00 +CPU= 36.668
  23:56:52 STEP 1504 H=1504:00 +CPU= 26.029
  23:57:25 STEP 1505 H=1505:00 +CPU= 27.108
  23:58:12 STEP 1506 H=1506:00 +CPU= 37.655
  23:58:44 STEP 1507 H=1507:00 +CPU= 25.552
  23:59:15 STEP 1508 H=1508:00 +CPU= 25.989
  00:00:01 STEP 1509 H=1509:00 +CPU= 37.596
  00:00:34 STEP 1510 H=1510:00 +CPU= 27.150
  00:01:06 STEP 1511 H=1511:00 +CPU= 25.943
  00:02:02 STEP 1512 H=1512:00 +CPU= 45.442
Moving to projects directory: /var/lib/boinc/slots/7/ICMGGhq0f+001512
Moving to projects directory: /var/lib/boinc/slots/7/ICMSHhq0f+001512
Moving to projects directory: /var/lib/boinc/slots/7/ICMUAhq0f+001512
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12182369/ICMGGhq0f+001488
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12182369/ICMSHhq0f+001488
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12182369/ICMUAhq0f+001488
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12182369/ICMGGhq0f+001500
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12182369/ICMSHhq0f+001500
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12182369/ICMUAhq0f+001500
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0725_1996050100_123_965_12182369_1_r881573072_62.zip
Uploading the intermediate file: upload_file_62.zip
  00:02:33 STEP 1513 H=1513:00 +CPU= 24.876
  00:03:07 STEP 1514 H=1514:00 +CPU= 26.892
  00:03:55 STEP 1515 H=1515:00 +CPU= 39.114
  00:04:28 STEP 1516 H=1516:00 +CPU= 25.988
  00:05:01 STEP 1517 H=1517:00 +CPU= 25.970
  00:05:49 STEP 1518 H=1518:00 +CPU= 38.705
  00:06:21 STEP 1519 H=1519:00 +CPU= 26.111
  00:06:54 STEP 1520 H=1520:00 +CPU= 26.198
  00:07:39 STEP 1521 H=1521:00 +CPU= 38.491
  00:08:11 STEP 1522 H=1522:00 +CPU= 27.003
  00:08:41 STEP 1523 H=1523:00 +CPU= 26.036
  00:09:31 STEP 1524 H=1524:00 +CPU= 43.231
Moving to projects directory: /var/lib/boinc/slots/7/ICMGGhq0f+001524
Moving to projects directory: /var/lib/boinc/slots/7/ICMSHhq0f+001524
Moving to projects directory: /var/lib/boinc/slots/7/ICMUAhq0f+001524
  00:10:00 STEP 1525 H=1525:00 +CPU= 25.848
  00:10:31 STEP 1526 H=1526:00 +CPU= 26.778
  00:11:17 STEP 1527 H=1527:00 +CPU= 39.090
  00:11:48 STEP 1528 H=1528:00 +CPU= 26.869
  00:12:17 STEP 1529 H=1529:00 +CPU= 25.526
  00:13:03 STEP 1530 H=1530:00 +CPU= 38.988
  00:13:33 STEP 1531 H=1531:00 +CPU= 25.895
  00:14:03 STEP 1532 H=1532:00 +CPU= 26.876
  00:14:49 STEP 1533 H=1533:00 +CPU= 39.700
  00:15:19 STEP 1534 H=1534:00 +CPU= 25.456
  00:15:52 STEP 1535 H=1535:00 +CPU= 28.160
  00:16:48 STEP 1536 H=1536:00 +CPU= 46.525
Moving to projects directory: /var/lib/boinc/slots/7/ICMGGhq0f+001536
Moving to projects directory: /var/lib/boinc/slots/7/ICMSHhq0f+001536
Moving to projects directory: /var/lib/boinc/slots/7/ICMUAhq0f+001536
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12182369/ICMGGhq0f+001512
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12182369/ICMSHhq0f+001512
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12182369/ICMUAhq0f+001512
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12182369/ICMGGhq0f+001524
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12182369/ICMSHhq0f+001524
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12182369/ICMUAhq0f+001524
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0725_1996050100_123_965_12182369_1_r881573072_63.zip
Uploading the intermediate file: upload_file_63.zip
  00:17:21 STEP 1537 H=1537:00 +CPU= 27.350
  00:17:52 STEP 1538 H=1538:00 +CPU= 26.392
  00:18:39 STEP 1539 H=1539:00 +CPU= 38.663
  00:19:10 STEP 1540 H=1540:00 +CPU= 26.622
  00:19:39 STEP 1541 H=1541:00 +CPU= 24.485
  00:20:23 STEP 1542 H=1542:00 +CPU= 39.916
  00:20:51 STEP 1543 H=1543:00 +CPU= 24.769
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1996050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0725
(argv4) batchid: 965
(argv5) wuid: 12182369
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/7
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12182369
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/7/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/7/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_611bb152d584b7ab14b51b5eb264bec9 to: /var/lib/boinc/slots/7/oifs_43r3_ps_0725_1996050100_123_965_12182369.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/7/oifs_43r3_ps_0725_1996050100_123_965_12182369.zip
ic_ancil_file: ic_ancil_12182369
ifsdata_file: ifsdata_12182369
climate_data_file: clim_data_12182369
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_a7991fb673670b4598075f2a4e0530a6 to: /var/lib/boinc/slots/7/ic_ancil_12182369.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/7/ic_ancil_12182369.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_c39c5297a8316fe84b15f53cc5aa22a4 to: /var/lib/boinc/slots/7/ifsdata/ifsdata_12182369.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/7/ifsdata/ifsdata_12182369.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/7/159l_2/clim_data_12182369.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/7/159l_2/clim_data_12182369.zip
Checking for progress XML file: /var/lib/boinc/slots/7/progress_file_12182369.xml
Opened progress file ok : /var/lib/boinc/slots/7/progress_file_12182369.xml
-- Model is restarting --
Adjusting last_iter, 1543, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/7/progress_file_12182369.xml
last_cpu_time: 54556
upload_file_number: 64
last_iter: 1535
last_upload: 5529600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0725_1996050100_123_965_12182369_1_r881573072
The child process has been launched with process id: 15219
Executing the command: /var/lib/boinc/slots/7/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:tweenis:1:1:15219:15219] [20230129:004236:1674952956:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:tweenis:1:1:15219:15219] [20230129:004236:1674952956:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:tweenis:1:1:15219:15219] [20230129:004236:1674952956:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:tweenis:1:1:15219:15219] [20230129:004236:1674952956:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:tweenis:1:1:15219:15219] [20230129:004236:1674952956:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:tweenis:1:1:15219:15219] [20230129:004236:1674952956:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:tweenis:1:1:15219:15219] [20230129:004236:1674952956:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:tweenis:1:1:15219:15219] [20230129:004236:1674952956:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:tweenis:1:1:15219:15219] [20230129:004236:1674952956:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:tweenis:1:1:15219:15219] [20230129:004236:1674952956:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:tweenis:1:1:15219:15219] [20230129:004236:1674952956:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:tweenis:1:1:15219:15219] [20230129:004236:1674952956:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:tweenis:1:1:15219:15219] [20230129:004236:1674952956:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:tweenis:1:1:15219:15219] [20230129:004236:1674952956:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:tweenis:1:1:15219:15219] [20230129:004236:1674952956:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:tweenis:1:1:15219:15219] [20230129:004236:1674952956:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:15219:15219] [20230129:004236:1674952956:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:15219:15219] [20230129:004236:1674952956:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:15219:15219] [20230129:004236:1674952956:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:15219:15219] [20230129:004236:1674952956:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:tweenis:1:1:15219:15219] [20230129:004236:1674952956:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:15219:15219] [20230129:004236:1674952956:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:15219:15219] [20230129:004236:1674952956:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:15219:15219] [20230129:004236:1674952956:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:15219:15219] [20230129:004236:1674952956:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:15219:15219] [20230129:004236:1674952956:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:15219:15219] [20230129:004236:1674952956:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:15219:15219] [20230129:004236:1674952956:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:15219:15219] [20230129:004236:1674952956:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:15219:15219] [20230129:004236:1674952956:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  00:43:10 STEP 1536 H=1536:00 +CPU= 30.838
  00:43:35 STEP 1537 H=1537:00 +CPU= 25.455
  00:44:05 STEP 1538 H=1538:00 +CPU= 29.563
  00:44:43 STEP 1539 H=1539:00 +CPU= 38.387
  00:45:13 STEP 1540 H=1540:00 +CPU= 29.073
  00:45:35 STEP 1541 H=1541:00 +CPU= 22.240
  00:46:13 STEP 1542 H=1542:00 +CPU= 37.676
  00:46:42 STEP 1543 H=1543:00 +CPU= 28.984
  00:47:08 STEP 1544 H=1544:00 +CPU= 25.844
  00:47:50 STEP 1545 H=1545:00 +CPU= 42.246
  00:48:15 STEP 1546 H=1546:00 +CPU= 25.022
  00:48:44 STEP 1547 H=1547:00 +CPU= 28.554
  00:49:23 STEP 1548 H=1548:00 +CPU= 38.681
Moving to projects directory: /var/lib/boinc/slots/7/ICMGGhq0f+001548
Moving to projects directory: /var/lib/boinc/slots/7/ICMSHhq0f+001548
Moving to projects directory: /var/lib/boinc/slots/7/ICMUAhq0f+001548
  00:49:49 STEP 1549 H=1549:00 +CPU= 26.003
  00:50:19 STEP 1550 H=1550:00 +CPU= 30.000
  00:50:58 STEP 1551 H=1551:00 +CPU= 39.000
  00:51:25 STEP 1552 H=1552:00 +CPU= 26.670
  00:51:46 STEP 1553 H=1553:00 +CPU= 21.701
  00:52:26 STEP 1554 H=1554:00 +CPU= 40.047
  00:52:53 STEP 1555 H=1555:00 +CPU= 26.126
  00:53:16 STEP 1556 H=1556:00 +CPU= 23.261
  00:53:57 STEP 1557 H=1557:00 +CPU= 40.305
  00:54:18 STEP 1558 H=1558:00 +CPU= 21.024
  00:54:43 STEP 1559 H=1559:00 +CPU= 25.494
  00:55:27 STEP 1560 H=1560:00 +CPU= 43.695
Moving to projects directory: /var/lib/boinc/slots/7/ICMGGhq0f+001560
Moving to projects directory: /var/lib/boinc/slots/7/ICMSHhq0f+001560
Moving to projects directory: /var/lib/boinc/slots/7/ICMUAhq0f+001560
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12182369/ICMGGhq0f+001536
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12182369/ICMSHhq0f+001536
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12182369/ICMUAhq0f+001536
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12182369/ICMGGhq0f+001548
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12182369/ICMSHhq0f+001548
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12182369/ICMUAhq0f+001548
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0725_1996050100_123_965_12182369_1_r881573072_64.zip
Uploading the intermediate file: upload_file_64.zip
  00:55:55 STEP 1561 H=1561:00 +CPU= 26.941
  00:56:23 STEP 1562 H=1562:00 +CPU= 27.989
  00:57:00 STEP 1563 H=1563:00 +CPU= 37.349
  00:57:28 STEP 1564 H=1564:00 +CPU= 28.219
  00:57:55 STEP 1565 H=1565:00 +CPU= 26.402
  00:58:36 STEP 1566 H=1566:00 +CPU= 40.664
Abort request received from BOINC client, ending the child process
..Failed, model did not complete successfully

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:44:09 1531398 22304300 oifs_43r3_ps_0725_1996050100_123_965_12182369_1 3,196,800 30,088 0.0094
09 May 2023 16:44:04 1531398 22304300 oifs_43r3_ps_0725_1996050100_123_965_12182369_1 2,332,800 21,304 0.0091
09 May 2023 16:43:58 1531398 22304300 oifs_43r3_ps_0725_1996050100_123_965_12182369_1 1,468,800 12,486 0.0085


©2024 climateprediction.net