climateprediction.net home page
Task 22285643

Task 22285643

Name oifs_43r3_ps_0826_2013050100_123_982_12199470_0
Workunit 12199470
Created 24 Dec 2022, 8:34:53 UTC
Sent 25 Dec 2022, 11:54:18 UTC
Report deadline 24 Jan 2023, 11:54:18 UTC
Received 24 Jan 2023, 2:23:17 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 1 (0x00000001) Unknown error code
Computer ID 1507869
Run time 21 hours 42 min 47 sec
CPU time 18 hours 29 min 53 sec
Validate state Invalid
Credit 6,410.88
Device peak FLOPS 3.97 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,411.63 MB
Peak swap size 4,978.07 MB
Peak disk usage 6,430.67 MB
Stderr 
<core_client_version>7.16.6</core_client_version>
<![CDATA[
<message>
process exited with code 1 (0x1, -255)</message>
<stderr_txt>
0627200
result_base_name: oifs_43r3_ps_0826_2013050100_123_982_12199470_0_r1231106752
The child process has been launched with process id: 9840
Executing the command: /mnt/Cache/Linux/boinc-client/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:Leo-15:1:1:9840:9840] [20230123:151510:1674508510:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:Leo-15:1:1:9840:9840] [20230123:151510:1674508510:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:Leo-15:1:1:9840:9840] [20230123:151510:1674508510:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:Leo-15:1:1:9840:9840] [20230123:151510:1674508510:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:Leo-15:1:1:9840:9840] [20230123:151510:1674508510:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:Leo-15:1:1:9840:9840] [20230123:151510:1674508510:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:Leo-15:1:1:9840:9840] [20230123:151510:1674508510:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:Leo-15:1:1:9840:9840] [20230123:151510:1674508510:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:Leo-15:1:1:9840:9840] [20230123:151510:1674508510:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:Leo-15:1:1:9840:9840] [20230123:151510:1674508510:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:Leo-15:1:1:9840:9840] [20230123:151510:1674508510:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:Leo-15:1:1:9840:9840] [20230123:151510:1674508510:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:Leo-15:1:1:9840:9840] [20230123:151510:1674508510:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:Leo-15:1:1:9840:9840] [20230123:151510:1674508510:0.002] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:Leo-15:1:1:9840:9840] [20230123:151510:1674508510:0.002] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:Leo-15:1:1:9840:9840] [20230123:151510:1674508510:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Leo-15:1:1:9840:9840] [20230123:151510:1674508510:0.003] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Leo-15:1:1:9840:9840] [20230123:151510:1674508510:0.003] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Leo-15:1:1:9840:9840] [20230123:151510:1674508510:0.003] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:Leo-15:1:1:9840:9840] [20230123:151510:1674508510:0.003] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:Leo-15:1:1:9840:9840] [20230123:151510:1674508510:0.003] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Leo-15:1:1:9840:9840] [20230123:151510:1674508510:0.003] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Leo-15:1:1:9840:9840] [20230123:151510:1674508510:0.004] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:Leo-15:1:1:9840:9840] [20230123:151510:1674508510:0.004] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Leo-15:1:1:9840:9840] [20230123:151510:1674508510:0.004] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Leo-15:1:1:9840:9840] [20230123:151510:1674508510:0.004] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Leo-15:1:1:9840:9840] [20230123:151510:1674508510:0.004] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:Leo-15:1:1:9840:9840] [20230123:151510:1674508510:0.006] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:Leo-15:1:1:9840:9840] [20230123:151510:1674508510:0.006] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Leo-15:1:1:9840:9840] [20230123:151510:1674508510:0.006] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  15:16:55 STEP 1344 H=1344:00 +CPU= 54.773
  15:17:28 STEP 1345 H=1345:00 +CPU= 30.874
  15:18:02 STEP 1346 H=1346:00 +CPU= 31.848
  15:18:54 STEP 1347 H=1347:00 +CPU= 48.286
  15:19:28 STEP 1348 H=1348:00 +CPU= 31.719
  15:20:01 STEP 1349 H=1349:00 +CPU= 30.791
  15:20:53 STEP 1350 H=1350:00 +CPU= 49.243
  15:21:26 STEP 1351 H=1351:00 +CPU= 30.984
  15:22:00 STEP 1352 H=1352:00 +CPU= 31.737
  15:22:52 STEP 1353 H=1353:00 +CPU= 48.212
  15:23:27 STEP 1354 H=1354:00 +CPU= 31.752
  15:23:59 STEP 1355 H=1355:00 +CPU= 30.782
  15:24:56 STEP 1356 H=1356:00 +CPU= 53.112
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/3/ICMGGhq0f+001356
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/3/ICMSHhq0f+001356
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/3/ICMUAhq0f+001356
  15:25:29 STEP 1357 H=1357:00 +CPU= 30.736
  15:26:03 STEP 1358 H=1358:00 +CPU= 31.741
  15:26:54 STEP 1359 H=1359:00 +CPU= 48.353
  15:27:29 STEP 1360 H=1360:00 +CPU= 31.788
  15:28:01 STEP 1361 H=1361:00 +CPU= 30.989
  15:28:54 STEP 1362 H=1362:00 +CPU= 48.977
  15:29:27 STEP 1363 H=1363:00 +CPU= 30.872
  15:30:01 STEP 1364 H=1364:00 +CPU= 31.771
  15:30:53 STEP 1365 H=1365:00 +CPU= 48.243
  15:31:27 STEP 1366 H=1366:00 +CPU= 31.793
  15:32:01 STEP 1367 H=1367:00 +CPU= 31.683
  15:33:20 STEP 1368 H=1368:00 +CPU= 57.605
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/3/ICMGGhq0f+001368
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/3/ICMSHhq0f+001368
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/3/ICMUAhq0f+001368
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199470/ICMGGhq0f+001344
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199470/ICMSHhq0f+001344
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199470/ICMUAhq0f+001344
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199470/ICMGGhq0f+001356
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199470/ICMSHhq0f+001356
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199470/ICMUAhq0f+001356
Zipping up the intermediate file: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0826_2013050100_123_982_12199470_0_r1231106752_56.zip
Uploading the intermediate file: upload_file_56.zip
  15:33:54 STEP 1369 H=1369:00 +CPU= 30.971
  15:34:28 STEP 1370 H=1370:00 +CPU= 31.954
  15:35:19 STEP 1371 H=1371:00 +CPU= 48.085
  15:35:53 STEP 1372 H=1372:00 +CPU= 31.726
  15:36:26 STEP 1373 H=1373:00 +CPU= 30.958
  15:37:18 STEP 1374 H=1374:00 +CPU= 49.143
  15:37:51 STEP 1375 H=1375:00 +CPU= 31.053
  15:38:25 STEP 1376 H=1376:00 +CPU= 31.700
  15:39:16 STEP 1377 H=1377:00 +CPU= 48.290
  15:39:50 STEP 1378 H=1378:00 +CPU= 31.811
  15:40:23 STEP 1379 H=1379:00 +CPU= 31.220
  15:41:20 STEP 1380 H=1380:00 +CPU= 53.313
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/3/ICMGGhq0f+001380
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/3/ICMSHhq0f+001380
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/3/ICMUAhq0f+001380
  15:41:53 STEP 1381 H=1381:00 +CPU= 31.034
  15:42:29 STEP 1382 H=1382:00 +CPU= 32.073
  15:43:22 STEP 1383 H=1383:00 +CPU= 48.309
  15:43:55 STEP 1384 H=1384:00 +CPU= 31.658
  15:44:28 STEP 1385 H=1385:00 +CPU= 30.965
  15:45:20 STEP 1386 H=1386:00 +CPU= 49.017
  15:45:53 STEP 1387 H=1387:00 +CPU= 30.941
  15:46:27 STEP 1388 H=1388:00 +CPU= 31.734
  15:47:19 STEP 1389 H=1389:00 +CPU= 48.404
  15:47:52 STEP 1390 H=1390:00 +CPU= 31.904
  15:48:32 STEP 1391 H=1391:00 +CPU= 31.662
  15:49:48 STEP 1392 H=1392:00 +CPU= 57.694
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/3/ICMGGhq0f+001392
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/3/ICMSHhq0f+001392
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/3/ICMUAhq0f+001392
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199470/ICMGGhq0f+001368
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199470/ICMSHhq0f+001368
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199470/ICMUAhq0f+001368
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199470/ICMGGhq0f+001380
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199470/ICMSHhq0f+001380
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199470/ICMUAhq0f+001380
Zipping up the intermediate file: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0826_2013050100_123_982_12199470_0_r1231106752_57.zip
Uploading the intermediate file: upload_file_57.zip
  15:50:22 STEP 1393 H=1393:00 +CPU= 31.115
  15:50:56 STEP 1394 H=1394:00 +CPU= 31.690
  15:51:47 STEP 1395 H=1395:00 +CPU= 48.227
  15:52:21 STEP 1396 H=1396:00 +CPU= 31.875
  15:52:54 STEP 1397 H=1397:00 +CPU= 31.058
  15:53:47 STEP 1398 H=1398:00 +CPU= 49.249
  15:54:20 STEP 1399 H=1399:00 +CPU= 31.137
  15:54:54 STEP 1400 H=1400:00 +CPU= 31.765
  15:55:46 STEP 1401 H=1401:00 +CPU= 48.645
  15:56:20 STEP 1402 H=1402:00 +CPU= 31.910
  15:56:53 STEP 1403 H=1403:00 +CPU= 30.876
  15:57:50 STEP 1404 H=1404:00 +CPU= 53.522
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/3/ICMGGhq0f+001404
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/3/ICMSHhq0f+001404
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/3/ICMUAhq0f+001404
  15:58:24 STEP 1405 H=1405:00 +CPU= 31.128
  15:58:58 STEP 1406 H=1406:00 +CPU= 31.716
  15:59:50 STEP 1407 H=1407:00 +CPU= 48.144
  16:00:24 STEP 1408 H=1408:00 +CPU= 31.914
  16:00:56 STEP 1409 H=1409:00 +CPU= 30.567
  16:01:49 STEP 1410 H=1410:00 +CPU= 49.199
  16:02:22 STEP 1411 H=1411:00 +CPU= 30.974
  16:02:55 STEP 1412 H=1412:00 +CPU= 31.522
  16:03:47 STEP 1413 H=1413:00 +CPU= 48.112
  16:04:21 STEP 1414 H=1414:00 +CPU= 31.781
  16:04:56 STEP 1415 H=1415:00 +CPU= 31.253
  16:06:12 STEP 1416 H=1416:00 +CPU= 57.435
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/3/ICMGGhq0f+001416
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/3/ICMSHhq0f+001416
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/3/ICMUAhq0f+001416
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199470/ICMGGhq0f+001392
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199470/ICMSHhq0f+001392
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199470/ICMUAhq0f+001392
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199470/ICMGGhq0f+001404
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199470/ICMSHhq0f+001404
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199470/ICMUAhq0f+001404
Zipping up the intermediate file: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0826_2013050100_123_982_12199470_0_r1231106752_58.zip
Uploading the intermediate file: upload_file_58.zip
  16:06:45 STEP 1417 H=1417:00 +CPU= 30.846
  16:07:18 STEP 1418 H=1418:00 +CPU= 31.178
  16:08:09 STEP 1419 H=1419:00 +CPU= 48.286
  16:08:43 STEP 1420 H=1420:00 +CPU= 31.569
  16:09:16 STEP 1421 H=1421:00 +CPU= 30.485
  16:10:08 STEP 1422 H=1422:00 +CPU= 49.218
  16:10:40 STEP 1423 H=1423:00 +CPU= 30.843
  16:11:13 STEP 1424 H=1424:00 +CPU= 30.975
  16:12:05 STEP 1425 H=1425:00 +CPU= 48.481
  16:12:41 STEP 1426 H=1426:00 +CPU= 31.724
  16:13:13 STEP 1427 H=1427:00 +CPU= 30.441
  16:14:11 STEP 1428 H=1428:00 +CPU= 53.679
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/3/ICMGGhq0f+001428
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/3/ICMSHhq0f+001428
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/3/ICMUAhq0f+001428
  16:14:44 STEP 1429 H=1429:00 +CPU= 30.871
  16:15:17 STEP 1430 H=1430:00 +CPU= 31.162
  16:16:08 STEP 1431 H=1431:00 +CPU= 48.127
  16:16:42 STEP 1432 H=1432:00 +CPU= 31.470
  16:17:14 STEP 1433 H=1433:00 +CPU= 30.317
  16:18:06 STEP 1434 H=1434:00 +CPU= 49.243
  16:18:40 STEP 1435 H=1435:00 +CPU= 30.887
  16:19:13 STEP 1436 H=1436:00 +CPU= 31.213
  16:20:04 STEP 1437 H=1437:00 +CPU= 48.403
  16:20:39 STEP 1438 H=1438:00 +CPU= 31.308
  16:21:14 STEP 1439 H=1439:00 +CPU= 31.265
  16:22:29 STEP 1440 H=1440:00 +CPU= 57.748
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/3/ICMGGhq0f+001440
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/3/ICMSHhq0f+001440
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/3/ICMUAhq0f+001440
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199470/ICMGGhq0f+001416
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199470/ICMSHhq0f+001416
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199470/ICMUAhq0f+001416
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199470/ICMGGhq0f+001428
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199470/ICMSHhq0f+001428
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199470/ICMUAhq0f+001428
Zipping up the intermediate file: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0826_2013050100_123_982_12199470_0_r1231106752_59.zip
Uploading the intermediate file: upload_file_59.zip
  16:23:03 STEP 1441 H=1441:00 +CPU= 30.961
  16:23:37 STEP 1442 H=1442:00 +CPU= 31.734
  16:24:29 STEP 1443 H=1443:00 +CPU= 48.290
  16:25:02 STEP 1444 H=1444:00 +CPU= 31.493
  16:25:35 STEP 1445 H=1445:00 +CPU= 30.795
  16:26:28 STEP 1446 H=1446:00 +CPU= 49.348
  16:27:01 STEP 1447 H=1447:00 +CPU= 30.872
  16:27:35 STEP 1448 H=1448:00 +CPU= 31.688
  16:28:27 STEP 1449 H=1449:00 +CPU= 48.338
  16:29:00 STEP 1450 H=1450:00 +CPU= 31.415
  16:29:33 STEP 1451 H=1451:00 +CPU= 30.393
  16:30:29 STEP 1452 H=1452:00 +CPU= 53.075
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/3/ICMGGhq0f+001452
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/3/ICMSHhq0f+001452
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/3/ICMUAhq0f+001452
  16:31:01 STEP 1453 H=1453:00 +CPU= 30.684
  16:31:36 STEP 1454 H=1454:00 +CPU= 31.559
  16:32:27 STEP 1455 H=1455:00 +CPU= 48.057
  16:33:00 STEP 1456 H=1456:00 +CPU= 31.532
  16:33:33 STEP 1457 H=1457:00 +CPU= 30.654
  16:34:27 STEP 1458 H=1458:00 +CPU= 49.397
  16:35:00 STEP 1459 H=1459:00 +CPU= 30.672
  16:35:33 STEP 1460 H=1460:00 +CPU= 31.410
  16:36:24 STEP 1461 H=1461:00 +CPU= 47.992
  16:36:58 STEP 1462 H=1462:00 +CPU= 31.123
  16:37:33 STEP 1463 H=1463:00 +CPU= 31.186
  16:38:48 STEP 1464 H=1464:00 +CPU= 57.350
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/3/ICMGGhq0f+001464
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/3/ICMSHhq0f+001464
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/3/ICMUAhq0f+001464
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199470/ICMGGhq0f+001440
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199470/ICMSHhq0f+001440
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199470/ICMUAhq0f+001440
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199470/ICMGGhq0f+001452
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199470/ICMSHhq0f+001452
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199470/ICMUAhq0f+001452
Zipping up the intermediate file: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0826_2013050100_123_982_12199470_0_r1231106752_60.zip
Uploading the intermediate file: upload_file_60.zip
  16:39:21 STEP 1465 H=1465:00 +CPU= 30.646
  16:39:55 STEP 1466 H=1466:00 +CPU= 31.362
  16:40:46 STEP 1467 H=1467:00 +CPU= 48.015
  16:41:19 STEP 1468 H=1468:00 +CPU= 31.330
  16:41:52 STEP 1469 H=1469:00 +CPU= 30.857
  16:42:47 STEP 1470 H=1470:00 +CPU= 49.088
  16:43:20 STEP 1471 H=1471:00 +CPU= 31.052
  16:43:54 STEP 1472 H=1472:00 +CPU= 31.543
  16:44:45 STEP 1473 H=1473:00 +CPU= 47.903
  16:45:18 STEP 1474 H=1474:00 +CPU= 31.459
  16:45:51 STEP 1475 H=1475:00 +CPU= 30.589
  16:46:48 STEP 1476 H=1476:00 +CPU= 53.256
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/3/ICMGGhq0f+001476
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/3/ICMSHhq0f+001476
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/3/ICMUAhq0f+001476
  16:47:21 STEP 1477 H=1477:00 +CPU= 30.762
  16:47:54 STEP 1478 H=1478:00 +CPU= 31.527
  16:48:46 STEP 1479 H=1479:00 +CPU= 48.118
  16:49:19 STEP 1480 H=1480:00 +CPU= 31.444
  16:49:52 STEP 1481 H=1481:00 +CPU= 30.745
  16:50:44 STEP 1482 H=1482:00 +CPU= 48.868
  16:51:17 STEP 1483 H=1483:00 +CPU= 30.806
  16:51:51 STEP 1484 H=1484:00 +CPU= 31.676
  16:52:42 STEP 1485 H=1485:00 +CPU= 47.687
  16:53:16 STEP 1486 H=1486:00 +CPU= 31.482
  16:53:51 STEP 1487 H=1487:00 +CPU= 30.929
  16:55:10 STEP 1488 H=1488:00 +CPU= 58.314
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/3/ICMGGhq0f+001488
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/3/ICMSHhq0f+001488
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/3/ICMUAhq0f+001488
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199470/ICMGGhq0f+001464
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199470/ICMSHhq0f+001464
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199470/ICMUAhq0f+001464
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199470/ICMGGhq0f+001476
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199470/ICMSHhq0f+001476
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199470/ICMUAhq0f+001476
Zipping up the intermediate file: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0826_2013050100_123_982_12199470_0_r1231106752_61.zip
Uploading the intermediate file: upload_file_61.zip
  16:55:44 STEP 1489 H=1489:00 +CPU= 31.065
  16:56:20 STEP 1490 H=1490:00 +CPU= 32.164
  16:57:19 STEP 1491 H=1491:00 +CPU= 49.505
  16:57:55 STEP 1492 H=1492:00 +CPU= 32.077
  16:58:31 STEP 1493 H=1493:00 +CPU= 31.586
  16:59:30 STEP 1494 H=1494:00 +CPU= 50.313
  17:00:05 STEP 1495 H=1495:00 +CPU= 31.297
  17:00:42 STEP 1496 H=1496:00 +CPU= 32.417
  17:01:40 STEP 1497 H=1497:00 +CPU= 49.179
  17:02:16 STEP 1498 H=1498:00 +CPU= 31.979
  17:02:51 STEP 1499 H=1499:00 +CPU= 31.454
  17:03:52 STEP 1500 H=1500:00 +CPU= 54.400
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/3/ICMGGhq0f+001500
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/3/ICMSHhq0f+001500
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/3/ICMUAhq0f+001500
  17:04:27 STEP 1501 H=1501:00 +CPU= 31.411
  17:05:02 STEP 1502 H=1502:00 +CPU= 32.184
  17:05:57 STEP 1503 H=1503:00 +CPU= 48.899
  17:06:32 STEP 1504 H=1504:00 +CPU= 32.108
  17:07:08 STEP 1505 H=1505:00 +CPU= 31.613
  17:08:03 STEP 1506 H=1506:00 +CPU= 49.762
  17:08:38 STEP 1507 H=1507:00 +CPU= 31.096
  17:09:14 STEP 1508 H=1508:00 +CPU= 31.938
  17:10:09 STEP 1509 H=1509:00 +CPU= 49.086
  17:10:47 STEP 1510 H=1510:00 +CPU= 32.045
  17:11:23 STEP 1511 H=1511:00 +CPU= 31.570
  17:12:48 STEP 1512 H=1512:00 +CPU= 59.133
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/3/ICMGGhq0f+001512
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/3/ICMSHhq0f+001512
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/3/ICMUAhq0f+001512
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199470/ICMGGhq0f+001488
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199470/ICMSHhq0f+001488
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199470/ICMUAhq0f+001488
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199470/ICMGGhq0f+001500
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199470/ICMSHhq0f+001500
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199470/ICMUAhq0f+001500
Zipping up the intermediate file: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0826_2013050100_123_982_12199470_0_r1231106752_62.zip
Uploading the intermediate file: upload_file_62.zip
  17:13:23 STEP 1513 H=1513:00 +CPU= 31.155
  17:13:59 STEP 1514 H=1514:00 +CPU= 32.218
  17:14:59 STEP 1515 H=1515:00 +CPU= 49.662
  17:15:37 STEP 1516 H=1516:00 +CPU= 32.165
  17:16:16 STEP 1517 H=1517:00 +CPU= 31.900
  17:17:17 STEP 1518 H=1518:00 +CPU= 50.723
  17:17:52 STEP 1519 H=1519:00 +CPU= 31.340
  17:18:29 STEP 1520 H=1520:00 +CPU= 32.199
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2013050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0826
(argv4) batchid: 982
(argv5) wuid: 12199470
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /mnt/Cache/Linux/boinc-client/slots/3
Project directory is: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199470
..mkdir for temp folder for results failed
Copying: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /mnt/Cache/Linux/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /mnt/Cache/Linux/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_46d5b82a0c40108a3c508af5134097c3 to: /mnt/Cache/Linux/boinc-client/slots/3/oifs_43r3_ps_0826_2013050100_123_982_12199470.zip
Unzipping the namelist zip file: /mnt/Cache/Linux/boinc-client/slots/3/oifs_43r3_ps_0826_2013050100_123_982_12199470.zip
ic_ancil_file: ic_ancil_12199470
ifsdata_file: ifsdata_12199470
climate_data_file: clim_data_12199470
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_d816122801621a192f20e73b434add83 to: /mnt/Cache/Linux/boinc-client/slots/3/ic_ancil_12199470.zip
Unzipping the IC ancils zip file: /mnt/Cache/Linux/boinc-client/slots/3/ic_ancil_12199470.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_8cb1abfe7835fced7c2590f3696c8454 to: /mnt/Cache/Linux/boinc-client/slots/3/ifsdata/ifsdata_12199470.zip
Unzipping the ifsdata_zip file: /mnt/Cache/Linux/boinc-client/slots/3/ifsdata/ifsdata_12199470.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /mnt/Cache/Linux/boinc-client/slots/3/159l_2/clim_data_12199470.zip
Unzipping the climate data zip file: /mnt/Cache/Linux/boinc-client/slots/3/159l_2/clim_data_12199470.zip
Checking for progress XML file: /mnt/Cache/Linux/boinc-client/slots/3/progress_file_12199470.xml
Opened progress file ok : /mnt/Cache/Linux/boinc-client/slots/3/progress_file_12199470.xml
-- Model is restarting --
Adjusting last_iter, 1521, to previous model restart step.
Creating progress file: /mnt/Cache/Linux/boinc-client/slots/3/progress_file_12199470.xml
last_cpu_time: 61829
upload_file_number: 63
last_iter: 1511
last_upload: 5443200
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0826_2013050100_123_982_12199470_0_r1231106752
The child process has been launched with process id: 18145
Executing the command: /mnt/Cache/Linux/boinc-client/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:Leo-15:1:1:18145:18145] [20230123:172844:1674516524:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:Leo-15:1:1:18145:18145] [20230123:172844:1674516524:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:Leo-15:1:1:18145:18145] [20230123:172844:1674516524:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:Leo-15:1:1:18145:18145] [20230123:172844:1674516524:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:Leo-15:1:1:18145:18145] [20230123:172844:1674516524:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:Leo-15:1:1:18145:18145] [20230123:172844:1674516524:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:Leo-15:1:1:18145:18145] [20230123:172844:1674516524:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:Leo-15:1:1:18145:18145] [20230123:172844:1674516524:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:Leo-15:1:1:18145:18145] [20230123:172844:1674516524:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:Leo-15:1:1:18145:18145] [20230123:172844:1674516524:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:Leo-15:1:1:18145:18145] [20230123:172844:1674516524:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:Leo-15:1:1:18145:18145] [20230123:172844:1674516524:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:Leo-15:1:1:18145:18145] [20230123:172844:1674516524:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:Leo-15:1:1:18145:18145] [20230123:172844:1674516524:0.003] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:Leo-15:1:1:18145:18145] [20230123:172844:1674516524:0.003] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:Leo-15:1:1:18145:18145] [20230123:172844:1674516524:0.003] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Leo-15:1:1:18145:18145] [20230123:172844:1674516524:0.003] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Leo-15:1:1:18145:18145] [20230123:172844:1674516524:0.003] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Leo-15:1:1:18145:18145] [20230123:172844:1674516524:0.004] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:Leo-15:1:1:18145:18145] [20230123:172844:1674516524:0.004] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:Leo-15:1:1:18145:18145] [20230123:172844:1674516524:0.004] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Leo-15:1:1:18145:18145] [20230123:172844:1674516524:0.004] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Leo-15:1:1:18145:18145] [20230123:172844:1674516524:0.004] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:Leo-15:1:1:18145:18145] [20230123:172844:1674516524:0.004] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Leo-15:1:1:18145:18145] [20230123:172844:1674516524:0.004] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Leo-15:1:1:18145:18145] [20230123:172844:1674516524:0.004] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Leo-15:1:1:18145:18145] [20230123:172844:1674516524:0.004] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:Leo-15:1:1:18145:18145] [20230123:172844:1674516524:0.004] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:Leo-15:1:1:18145:18145] [20230123:172844:1674516524:0.004] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Leo-15:1:1:18145:18145] [20230123:172844:1674516524:0.004] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  17:29:53 STEP 1512 H=1512:00 +CPU= 55.947
  17:30:32 STEP 1513 H=1513:00 +CPU= 32.391
  17:31:12 STEP 1514 H=1514:00 +CPU= 33.232
  17:32:08 STEP 1515 H=1515:00 +CPU= 49.210
  17:32:44 STEP 1516 H=1516:00 +CPU= 32.601
  17:33:19 STEP 1517 H=1517:00 +CPU= 31.765
  17:34:14 STEP 1518 H=1518:00 +CPU= 50.397
  17:34:50 STEP 1519 H=1519:00 +CPU= 31.859
  17:35:26 STEP 1520 H=1520:00 +CPU= 32.760
  17:36:22 STEP 1521 H=1521:00 +CPU= 49.596
  17:37:02 STEP 1522 H=1522:00 +CPU= 32.870
  17:37:40 STEP 1523 H=1523:00 +CPU= 31.899
  17:38:48 STEP 1524 H=1524:00 +CPU= 54.980
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/3/ICMGGhq0f+001524
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/3/ICMSHhq0f+001524
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/3/ICMUAhq0f+001524
  17:39:25 STEP 1525 H=1525:00 +CPU= 31.919
  17:40:01 STEP 1526 H=1526:00 +CPU= 32.438
  17:40:56 STEP 1527 H=1527:00 +CPU= 49.529
  17:41:33 STEP 1528 H=1528:00 +CPU= 32.477
  17:42:10 STEP 1529 H=1529:00 +CPU= 31.709
  17:43:15 STEP 1530 H=1530:00 +CPU= 50.625
  17:43:55 STEP 1531 H=1531:00 +CPU= 32.135
  17:44:35 STEP 1532 H=1532:00 +CPU= 33.075
  17:45:35 STEP 1533 H=1533:00 +CPU= 49.972
  17:46:25 STEP 1534 H=1534:00 +CPU= 33.466
  17:47:10 STEP 1535 H=1535:00 +CPU= 33.233
  17:48:48 STEP 1536 H=1536:00 +CPU= 59.327
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/3/ICMGGhq0f+001536
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/3/ICMSHhq0f+001536
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/3/ICMUAhq0f+001536
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199470/ICMGGhq0f+001512
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199470/ICMSHhq0f+001512
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199470/ICMUAhq0f+001512
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199470/ICMGGhq0f+001524
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199470/ICMSHhq0f+001524
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199470/ICMUAhq0f+001524
Zipping up the intermediate file: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0826_2013050100_123_982_12199470_0_r1231106752_63.zip
Uploading the intermediate file: upload_file_63.zip
  17:49:29 STEP 1537 H=1537:00 +CPU= 29.780
  17:50:16 STEP 1538 H=1538:00 +CPU= 32.466
  17:51:23 STEP 1539 H=1539:00 +CPU= 50.694
  17:52:08 STEP 1540 H=1540:00 +CPU= 33.487
  17:52:50 STEP 1541 H=1541:00 +CPU= 32.513
  17:53:57 STEP 1542 H=1542:00 +CPU= 51.771
  17:54:40 STEP 1543 H=1543:00 +CPU= 32.611
  17:55:24 STEP 1544 H=1544:00 +CPU= 33.560
  17:56:29 STEP 1545 H=1545:00 +CPU= 50.728
  17:57:12 STEP 1546 H=1546:00 +CPU= 33.427
  17:57:56 STEP 1547 H=1547:00 +CPU= 32.619
  17:59:09 STEP 1548 H=1548:00 +CPU= 56.398
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/3/ICMGGhq0f+001548
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/3/ICMSHhq0f+001548
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/3/ICMUAhq0f+001548
  17:59:49 STEP 1549 H=1549:00 +CPU= 32.280
  18:00:25 STEP 1550 H=1550:00 +CPU= 32.132
  18:01:19 STEP 1551 H=1551:00 +CPU= 49.199
  18:01:55 STEP 1552 H=1552:00 +CPU= 32.469
  18:02:30 STEP 1553 H=1553:00 +CPU= 31.640
  18:03:30 STEP 1554 H=1554:00 +CPU= 50.760
  18:04:14 STEP 1555 H=1555:00 +CPU= 32.729
  18:04:58 STEP 1556 H=1556:00 +CPU= 33.724
  18:06:04 STEP 1557 H=1557:00 +CPU= 51.112
  18:06:47 STEP 1558 H=1558:00 +CPU= 33.563
  18:07:33 STEP 1559 H=1559:00 +CPU= 33.368
  18:09:05 STEP 1560 H=1560:00 +CPU= 60.325
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/3/ICMGGhq0f+001560
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/3/ICMSHhq0f+001560
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/3/ICMUAhq0f+001560
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199470/ICMGGhq0f+001536
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199470/ICMSHhq0f+001536
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199470/ICMUAhq0f+001536
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199470/ICMGGhq0f+001548
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199470/ICMSHhq0f+001548
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199470/ICMUAhq0f+001548
Zipping up the intermediate file: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0826_2013050100_123_982_12199470_0_r1231106752_64.zip
Uploading the intermediate file: upload_file_64.zip
  18:09:48 STEP 1561 H=1561:00 +CPU= 32.602
  18:10:34 STEP 1562 H=1562:00 +CPU= 32.267
  18:11:40 STEP 1563 H=1563:00 +CPU= 50.889
  18:12:27 STEP 1564 H=1564:00 +CPU= 33.592
  18:13:10 STEP 1565 H=1565:00 +CPU= 32.860
  18:14:17 STEP 1566 H=1566:00 +CPU= 51.680
  18:14:55 STEP 1567 H=1567:00 +CPU= 32.015
  18:15:32 STEP 1568 H=1568:00 +CPU= 32.476
  18:16:28 STEP 1569 H=1569:00 +CPU= 49.268
  18:17:05 STEP 1570 H=1570:00 +CPU= 32.358
  18:17:41 STEP 1571 H=1571:00 +CPU= 31.760
  18:18:44 STEP 1572 H=1572:00 +CPU= 54.508
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/3/ICMGGhq0f+001572
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/3/ICMSHhq0f+001572
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/3/ICMUAhq0f+001572
  18:19:20 STEP 1573 H=1573:00 +CPU= 31.617
  18:19:58 STEP 1574 H=1574:00 +CPU= 32.615
  18:20:54 STEP 1575 H=1575:00 +CPU= 49.202
  18:21:31 STEP 1576 H=1576:00 +CPU= 32.419
  18:22:07 STEP 1577 H=1577:00 +CPU= 31.633
  18:23:19 STEP 1578 H=1578:00 +CPU= 50.942
  18:24:04 STEP 1579 H=1579:00 +CPU= 32.596
  18:24:41 STEP 1580 H=1580:00 +CPU= 32.610
  18:25:41 STEP 1581 H=1581:00 +CPU= 49.874
  18:26:23 STEP 1582 H=1582:00 +CPU= 33.006
  18:27:05 STEP 1583 H=1583:00 +CPU= 32.482
  18:28:34 STEP 1584 H=1584:00 +CPU= 59.468
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/3/ICMGGhq0f+001584
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/3/ICMSHhq0f+001584
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/3/ICMUAhq0f+001584
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199470/ICMGGhq0f+001560
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199470/ICMSHhq0f+001560
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199470/ICMUAhq0f+001560
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199470/ICMGGhq0f+001572
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199470/ICMSHhq0f+001572
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199470/ICMUAhq0f+001572
Zipping up the intermediate file: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0826_2013050100_123_982_12199470_0_r1231106752_65.zip
Uploading the intermediate file: upload_file_65.zip
  18:29:14 STEP 1585 H=1585:00 +CPU= 31.986
  18:29:56 STEP 1586 H=1586:00 +CPU= 31.844
  18:30:57 STEP 1587 H=1587:00 +CPU= 49.357
  18:31:39 STEP 1588 H=1588:00 +CPU= 33.256
  18:32:19 STEP 1589 H=1589:00 +CPU= 32.427
  18:33:23 STEP 1590 H=1590:00 +CPU= 51.169
  18:34:05 STEP 1591 H=1591:00 +CPU= 32.401
  18:34:47 STEP 1592 H=1592:00 +CPU= 33.421
  18:35:49 STEP 1593 H=1593:00 +CPU= 50.197
  18:36:30 STEP 1594 H=1594:00 +CPU= 33.157
  18:37:10 STEP 1595 H=1595:00 +CPU= 32.656
  18:38:19 STEP 1596 H=1596:00 +CPU= 55.820
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/3/ICMGGhq0f+001596
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/3/ICMSHhq0f+001596
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/3/ICMUAhq0f+001596
  18:38:59 STEP 1597 H=1597:00 +CPU= 32.327
  18:39:35 STEP 1598 H=1598:00 +CPU= 32.255
  18:40:28 STEP 1599 H=1599:00 +CPU= 48.442
  18:41:04 STEP 1600 H=1600:00 +CPU= 32.295
  18:41:38 STEP 1601 H=1601:00 +CPU= 31.510
  18:42:35 STEP 1602 H=1602:00 +CPU= 49.899
  18:43:10 STEP 1603 H=1603:00 +CPU= 31.679
  18:43:46 STEP 1604 H=1604:00 +CPU= 32.390
  18:45:02 STEP 1605 H=1605:00 +CPU= 50.669
  18:45:58 STEP 1606 H=1606:00 +CPU= 34.301
  18:46:54 STEP 1607 H=1607:00 +CPU= 33.884
  18:48:35 STEP 1608 H=1608:00 +CPU= 60.193
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/3/ICMGGhq0f+001608
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/3/ICMSHhq0f+001608
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/3/ICMUAhq0f+001608
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199470/ICMGGhq0f+001584
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199470/ICMSHhq0f+001584
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199470/ICMUAhq0f+001584
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199470/ICMGGhq0f+001596
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199470/ICMSHhq0f+001596
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199470/ICMUAhq0f+001596
Zipping up the intermediate file: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0826_2013050100_123_982_12199470_0_r1231106752_66.zip
Uploading the intermediate file: upload_file_66.zip
  18:49:21 STEP 1609 H=1609:00 +CPU= 32.669
  18:50:07 STEP 1610 H=1610:00 +CPU= 32.391
  18:51:21 STEP 1611 H=1611:00 +CPU= 50.760
  18:52:11 STEP 1612 H=1612:00 +CPU= 33.825
  18:52:57 STEP 1613 H=1613:00 +CPU= 32.671
  18:54:17 STEP 1614 H=1614:00 +CPU= 52.382
  18:55:05 STEP 1615 H=1615:00 +CPU= 32.862
  18:55:53 STEP 1616 H=1616:00 +CPU= 33.650
  18:57:12 STEP 1617 H=1617:00 +CPU= 51.156
  18:58:03 STEP 1618 H=1618:00 +CPU= 33.831
  18:58:53 STEP 1619 H=1619:00 +CPU= 32.887
  19:00:12 STEP 1620 H=1620:00 +CPU= 56.639
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/3/ICMGGhq0f+001620
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/3/ICMSHhq0f+001620
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/3/ICMUAhq0f+001620
  19:01:01 STEP 1621 H=1621:00 +CPU= 32.713
  19:01:49 STEP 1622 H=1622:00 +CPU= 33.401
  19:03:19 STEP 1623 H=1623:00 +CPU= 51.011
  19:04:23 STEP 1624 H=1624:00 +CPU= 33.891
  19:05:25 STEP 1625 H=1625:00 +CPU= 32.992
  19:06:49 STEP 1626 H=1626:00 +CPU= 52.332
  19:07:37 STEP 1627 H=1627:00 +CPU= 32.730
  19:08:23 STEP 1628 H=1628:00 +CPU= 33.233
  19:09:32 STEP 1629 H=1629:00 +CPU= 50.759
  19:10:18 STEP 1630 H=1630:00 +CPU= 33.271
  19:11:06 STEP 1631 H=1631:00 +CPU= 33.234
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2013050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0826
(argv4) batchid: 982
(argv5) wuid: 12199470
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /mnt/Cache/Linux/boinc-client/slots/3
Project directory is: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199470
..mkdir for temp folder for results failed
Copying: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /mnt/Cache/Linux/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /mnt/Cache/Linux/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_46d5b82a0c40108a3c508af5134097c3 to: /mnt/Cache/Linux/boinc-client/slots/3/oifs_43r3_ps_0826_2013050100_123_982_12199470.zip
Unzipping the namelist zip file: /mnt/Cache/Linux/boinc-client/slots/3/oifs_43r3_ps_0826_2013050100_123_982_12199470.zip
ic_ancil_file: ic_ancil_12199470
ifsdata_file: ifsdata_12199470
climate_data_file: clim_data_12199470
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_d816122801621a192f20e73b434add83 to: /mnt/Cache/Linux/boinc-client/slots/3/ic_ancil_12199470.zip
Unzipping the IC ancils zip file: /mnt/Cache/Linux/boinc-client/slots/3/ic_ancil_12199470.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_8cb1abfe7835fced7c2590f3696c8454 to: /mnt/Cache/Linux/boinc-client/slots/3/ifsdata/ifsdata_12199470.zip
Unzipping the ifsdata_zip file: /mnt/Cache/Linux/boinc-client/slots/3/ifsdata/ifsdata_12199470.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /mnt/Cache/Linux/boinc-client/slots/3/159l_2/clim_data_12199470.zip
Unzipping the climate data zip file: /mnt/Cache/Linux/boinc-client/slots/3/159l_2/clim_data_12199470.zip
Checking for progress XML file: /mnt/Cache/Linux/boinc-client/slots/3/progress_file_12199470.xml
Opened progress file ok : /mnt/Cache/Linux/boinc-client/slots/3/progress_file_12199470.xml
-- Model is restarting --
Adjusting last_iter, 1631, to previous model restart step.
Creating progress file: /mnt/Cache/Linux/boinc-client/slots/3/progress_file_12199470.xml
last_cpu_time: 66532
upload_file_number: 67
last_iter: 1607
last_upload: 5788800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0826_2013050100_123_982_12199470_0_r1231106752
The child process has been launched with process id: 34870
Executing the command: /mnt/Cache/Linux/boinc-client/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:Leo-15:1:1:34870:34870] [20230123:201841:1674526721:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:Leo-15:1:1:34870:34870] [20230123:201841:1674526721:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:Leo-15:1:1:34870:34870] [20230123:201841:1674526721:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:Leo-15:1:1:34870:34870] [20230123:201841:1674526721:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:Leo-15:1:1:34870:34870] [20230123:201841:1674526721:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:Leo-15:1:1:34870:34870] [20230123:201841:1674526721:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:Leo-15:1:1:34870:34870] [20230123:201841:1674526721:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:Leo-15:1:1:34870:34870] [20230123:201841:1674526721:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:Leo-15:1:1:34870:34870] [20230123:201841:1674526721:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:Leo-15:1:1:34870:34870] [20230123:201841:1674526721:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:Leo-15:1:1:34870:34870] [20230123:201841:1674526721:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:Leo-15:1:1:34870:34870] [20230123:201841:1674526721:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:Leo-15:1:1:34870:34870] [20230123:201841:1674526721:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:Leo-15:1:1:34870:34870] [20230123:201841:1674526721:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:Leo-15:1:1:34870:34870] [20230123:201841:1674526721:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:Leo-15:1:1:34870:34870] [20230123:201841:1674526721:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Leo-15:1:1:34870:34870] [20230123:201841:1674526721:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Leo-15:1:1:34870:34870] [20230123:201841:1674526721:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Leo-15:1:1:34870:34870] [20230123:201841:1674526721:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:Leo-15:1:1:34870:34870] [20230123:201841:1674526721:0.002] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:Leo-15:1:1:34870:34870] [20230123:201841:1674526721:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Leo-15:1:1:34870:34870] [20230123:201841:1674526721:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Leo-15:1:1:34870:34870] [20230123:201841:1674526721:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:Leo-15:1:1:34870:34870] [20230123:201841:1674526721:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Leo-15:1:1:34870:34870] [20230123:201841:1674526721:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Leo-15:1:1:34870:34870] [20230123:201841:1674526721:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Leo-15:1:1:34870:34870] [20230123:201841:1674526721:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:Leo-15:1:1:34870:34870] [20230123:201841:1674526721:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:Leo-15:1:1:34870:34870] [20230123:201841:1674526721:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Leo-15:1:1:34870:34870] [20230123:201841:1674526721:0.002] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
 ABORT!    1 !! *** WAVE MODEL HAS ABORTED 
[EC_DRHOOK:Leo-15:1:1:34870:34870] [20230123:202037:1674526837:115.956] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 1721 MB (maxheap), 3520 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:Leo-15:1:1:34870:34870] [20230123:202037:1674526837:115.956] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:Leo-15:1:1:34870:34870] [20230123:202037:1674526837:115.956] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:Leo-15:1:1:34870:34870] [20230123:202037:1674526837:115.956] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:Leo-15:1:1:34870:34870] [20230123:202037:1674526837:115.956] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:Leo-15:1:1:34870:34870] [20230123:202037:1674526837:115.956] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:Leo-15:1:1:34870:34870] [20230123:202037:1674526837:115.956] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :       CNT4 
[EC_DRHOOK:Leo-15:1:1:34870:34870] [20230123:202037:1674526837:115.956] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :        WVCOUPLE 
[EC_DRHOOK:Leo-15:1:1:34870:34870] [20230123:202037:1674526837:115.956] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :         WAVEMDL 
[EC_DRHOOK:Leo-15:1:1:34870:34870] [20230123:202037:1674526837:115.956] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :          INITMDL 
[EC_DRHOOK:Leo-15:1:1:34870:34870] [20230123:202037:1674526837:115.956] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :           USERIN 
 SDL_TRACEBACK: Calling INTEL_TRBK, THRD =            1
 Process            1  thread            1  calling linux_trbk from intel_trbk()
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:240] Backtrace(s) for program '/mnt/Cache/Linux/boinc-client/slots/3/oifs_43r3_model.exe' : sigcontextptr=0x7ffdc34dd1a0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:277] Backtrace (size = 19) with addr2line-cmd
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:302] /usr/bin/addr2line -fs -e '/mnt/Cache/Linux/boinc-client/slots/3/oifs_43r3_model.exe' 0x839273 0x839879 0x8c2957 0x81d89e 0x15f8aca 0x15d51c8 0x166b07f 0x15d7ff0 0x8711e0 0x7e0165 0x55e4ad 0x5471c9 0x546412 0x545e28 0x40708d 0x40220f 0x4021a2 0x1dca390 0x40206e
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [0]: [0x839273] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [1]: [0x839879] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [2]: [0x8c2957] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [3]: [0x81d89e] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [4]: [0x15f8aca] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [5]: [0x15d51c8] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [6]: [0x166b07f] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [7]: [0x15d7ff0] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [8]: [0x8711e0] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [9]: [0x7e0165] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [10]: [0x55e4ad] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [11]: [0x5471c9] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [12]: [0x546412] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [13]: [0x545e28] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [14]: [0x40708d] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [15]: [0x40220f] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [16]: [0x4021a2] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [17]: [0x1dca390] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [18]: [0x40206e] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:370] End of backtrace(s)
 SDL_TRACEBACK: Done INTEL_TRBK, THRD =            1
[EC_DRHOOK:Leo-15:1:1:34870:34870] [20230123:202037:1674526837:116.058] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1538] Received signal#6 (SIGABRT) :: 1721MB (heap), 3520MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:Leo-15:1:1:34870:34870] [20230123:202037:1674526837:116.058] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:Leo-15:1:1:34870:34870] [20230123:202037:1674526837:116.058] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f3c0 (old at (nil))
[EC_DRHOOK:Leo-15:1:1:34870:34870] [20230123:202037:1674526837:116.058] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1617] Signal#6 was caused by unrecognized si_code [memaddr=0x8000008836], nsigs = 1
[EC_DRHOOK:Leo-15:1:1:34870:34870] [20230123:202037:1674526837:116.058] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#6, nsigs = 1
[EC_DRHOOK:Leo-15:1:1:34870:34870] [20230123:202038:1674526838:117.059] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#6, nsigs = 1
[EC_DRHOOK:Leo-15:1:1:34870:34870] [20230123:202038:1674526838:117.059] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1ce8bb0 for signal#6, nsigs = 1
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
oifs_43r3_model.e  0000000001CE916B  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000081FFF1  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DC9090  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DC903B  Unknown               Unknown  Unknown
oifs_43r3_model.e  00000000008C2880  sdl_mod_mp_sdl_sr         100  sdl_mod.F90
oifs_43r3_model.e  000000000081D8AF  abor1_                     41  abor1.F90
oifs_43r3_model.e  00000000015F8ACA  mpabort_                   50  mpabort.F
oifs_43r3_model.e  00000000015D51C8  abort1_                    44  abort1.F
oifs_43r3_model.e  000000000166B07F  userin_                   553  userin.F
oifs_43r3_model.e  00000000015D7FF0  initmdl_                  452  initmdl.F
oifs_43r3_model.e  00000000008711E0  wavemdl_                  393  wavemdl.F
oifs_43r3_model.e  00000000007E0165  wvcouple_                 260  wvcouple.F90
oifs_43r3_model.e  000000000055E4AD  cnt4_                    1283  cnt4.F90
oifs_43r3_model.e  00000000005471C9  cnt3_                     267  cnt3.F90
oifs_43r3_model.e  0000000000546412  cnt2_                      88  cnt2.F90
oifs_43r3_model.e  0000000000545E28  cnt1_                      92  cnt1.F90
oifs_43r3_model.e  000000000040708D  cnt0_                     146  cnt0.F90
oifs_43r3_model.e  000000000040220F  MAIN__                     96  master.F90
oifs_43r3_model.e  00000000004021A2  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCA390  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000040206E  Unknown               Unknown  Unknown
The child process terminated with status: 0
..Failed, model did not complete successfully

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:38:06 1507869 22285643 oifs_43r3_ps_0826_2013050100_123_982_12199470_0 4,579,200 51,409 0.0112
09 May 2023 16:37:49 1507869 22285643 oifs_43r3_ps_0826_2013050100_123_982_12199470_0 3,715,200 42,020 0.0113
09 May 2023 16:37:35 1507869 22285643 oifs_43r3_ps_0826_2013050100_123_982_12199470_0 2,851,200 32,779 0.0115
09 May 2023 16:36:51 1507869 22285643 oifs_43r3_ps_0826_2013050100_123_982_12199470_0 1,123,200 12,861 0.0115


©2024 climateprediction.net