climateprediction.net home page
Task 22268122

Task 22268122

Name oifs_43r3_ps_0330_1997050100_123_966_12182974_0
Workunit 12182974
Created 22 Dec 2022, 13:19:04 UTC
Sent 3 Jan 2023, 19:56:39 UTC
Report deadline 2 Feb 2023, 19:56:39 UTC
Received 5 Jan 2023, 7:33:20 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1519251
Run time 13 hours 25 min 9 sec
CPU time 11 hours 48 min 44 sec
Validate state Invalid
Credit 2,419.20
Device peak FLOPS 4.94 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,415.61 MB
Peak swap size 4,978.08 MB
Peak disk usage 7,769.62 MB
Stderr
<core_client_version>7.16.6</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
ok.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:p500:1:1:1187440:1187440] [20230105:111715:1672885035:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:p500:1:1:1187440:1187440] [20230105:111715:1672885035:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:p500:1:1:1187440:1187440] [20230105:111715:1672885035:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:p500:1:1:1187440:1187440] [20230105:111715:1672885035:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:1187440:1187440] [20230105:111715:1672885035:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:p500:1:1:1187440:1187440] [20230105:111715:1672885035:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:p500:1:1:1187440:1187440] [20230105:111715:1672885035:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:p500:1:1:1187440:1187440] [20230105:111715:1672885035:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:1187440:1187440] [20230105:111715:1672885035:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:1187440:1187440] [20230105:111715:1672885035:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1187440:1187440] [20230105:111715:1672885035:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:1187440:1187440] [20230105:111715:1672885035:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1187440:1187440] [20230105:111715:1672885035:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:1187440:1187440] [20230105:111715:1672885035:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:1187440:1187440] [20230105:111715:1672885035:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1187440:1187440] [20230105:111715:1672885035:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1187440:1187440] [20230105:111715:1672885035:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:1187440:1187440] [20230105:111715:1672885035:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1187440:1187440] [20230105:111715:1672885035:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1187440:1187440] [20230105:111715:1672885035:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1187440:1187440] [20230105:111715:1672885035:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:1187440:1187440] [20230105:111715:1672885035:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:1187440:1187440] [20230105:111715:1672885035:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:1187440:1187440] [20230105:111715:1672885035:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  11:17:56 STEP 1320 H=1320:00 +CPU= 36.550
  11:18:17 STEP 1321 H=1321:00 +CPU= 20.311
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:19:29 STEP 1322 H=1322:00 +CPU= 21.287
  11:20:01 STEP 1323 H=1323:00 +CPU= 31.977
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:21:24 STEP 1324 H=1324:00 +CPU= 21.304
  11:21:45 STEP 1325 H=1325:00 +CPU= 21.100
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:22:59 STEP 1326 H=1326:00 +CPU= 33.124
  11:23:20 STEP 1327 H=1327:00 +CPU= 20.709
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:24:43 STEP 1328 H=1328:00 +CPU= 21.746
  11:25:16 STEP 1329 H=1329:00 +CPU= 32.686
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:26:29 STEP 1330 H=1330:00 +CPU= 20.678
  11:26:50 STEP 1331 H=1331:00 +CPU= 21.399
  11:27:26 STEP 1332 H=1332:00 +CPU= 35.679
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+001332
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+001332
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+001332
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:28:47 STEP 1333 H=1333:00 +CPU= 20.292
  11:29:08 STEP 1334 H=1334:00 +CPU= 21.540
  11:29:41 STEP 1335 H=1335:00 +CPU= 32.624
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:31:02 STEP 1336 H=1336:00 +CPU= 20.907
  11:31:22 STEP 1337 H=1337:00 +CPU= 20.241
  11:31:56 STEP 1338 H=1338:00 +CPU= 33.410
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:33:28 STEP 1339 H=1339:00 +CPU= 20.289
  11:33:50 STEP 1340 H=1340:00 +CPU= 21.107
  11:34:22 STEP 1341 H=1341:00 +CPU= 32.357
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:35:55 STEP 1342 H=1342:00 +CPU= 22.032
  11:36:16 STEP 1343 H=1343:00 +CPU= 20.833
  11:36:54 STEP 1344 H=1344:00 +CPU= 37.487
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+001344
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+001344
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182974/ICMGGhq0f+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182974/ICMSHhq0f+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182974/ICMUAhq0f+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182974/ICMGGhq0f+001332
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182974/ICMSHhq0f+001332
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182974/ICMUAhq0f+001332
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0330_1997050100_123_966_12182974_0_r1282578787_55.zip
Uploading the intermediate file: upload_file_55.zip
  11:37:13 STEP 1345 H=1345:00 +CPU= 18.774
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:38:32 STEP 1346 H=1346:00 +CPU= 21.110
  11:39:04 STEP 1347 H=1347:00 +CPU= 31.698
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:40:16 STEP 1348 H=1348:00 +CPU= 20.740
  11:40:36 STEP 1349 H=1349:00 +CPU= 20.511
  11:41:09 STEP 1350 H=1350:00 +CPU= 32.643
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:42:41 STEP 1351 H=1351:00 +CPU= 20.420
  11:43:02 STEP 1352 H=1352:00 +CPU= 20.830
  11:43:34 STEP 1353 H=1353:00 +CPU= 32.117
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:44:57 STEP 1354 H=1354:00 +CPU= 21.539
  11:45:17 STEP 1355 H=1355:00 +CPU= 20.588
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:46:44 STEP 1356 H=1356:00 +CPU= 36.545
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+001356
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+001356
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+001356
  11:47:05 STEP 1357 H=1357:00 +CPU= 20.968
  11:47:26 STEP 1358 H=1358:00 +CPU= 20.425
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:49:10 STEP 1359 H=1359:00 +CPU= 33.138
  11:49:31 STEP 1360 H=1360:00 +CPU= 20.503
  11:49:51 STEP 1361 H=1361:00 +CPU= 20.198
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:51:17 STEP 1362 H=1362:00 +CPU= 33.907
  11:51:37 STEP 1363 H=1363:00 +CPU= 20.250
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:52:50 STEP 1364 H=1364:00 +CPU= 20.876
  11:53:23 STEP 1365 H=1365:00 +CPU= 32.236
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:54:34 STEP 1366 H=1366:00 +CPU= 21.151
  11:54:54 STEP 1367 H=1367:00 +CPU= 20.385
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:56:35 STEP 1368 H=1368:00 +CPU= 38.552
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+001368
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+001368
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182974/ICMGGhq0f+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182974/ICMSHhq0f+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182974/ICMUAhq0f+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182974/ICMGGhq0f+001356
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182974/ICMSHhq0f+001356
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182974/ICMUAhq0f+001356
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0330_1997050100_123_966_12182974_0_r1282578787_56.zip
Uploading the intermediate file: upload_file_56.zip
  11:56:57 STEP 1369 H=1369:00 +CPU= 21.001
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:57:59 STEP 1370 H=1370:00 +CPU= 21.277
  11:58:32 STEP 1371 H=1371:00 +CPU= 32.550
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:59:54 STEP 1372 H=1372:00 +CPU= 21.239
  12:00:14 STEP 1373 H=1373:00 +CPU= 20.060
  12:00:48 STEP 1374 H=1374:00 +CPU= 33.140
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:02:20 STEP 1375 H=1375:00 +CPU= 20.545
  12:02:41 STEP 1376 H=1376:00 +CPU= 20.748
  12:03:14 STEP 1377 H=1377:00 +CPU= 32.378
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:04:16 STEP 1378 H=1378:00 +CPU= 21.555
  12:04:37 STEP 1379 H=1379:00 +CPU= 20.157
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:06:15 STEP 1380 H=1380:00 +CPU= 37.078
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+001380
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+001380
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+001380
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:07:26 STEP 1381 H=1381:00 +CPU= 20.450
  12:07:48 STEP 1382 H=1382:00 +CPU= 21.387
  12:08:21 STEP 1383 H=1383:00 +CPU= 32.468
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:09:42 STEP 1384 H=1384:00 +CPU= 20.774
  12:10:03 STEP 1385 H=1385:00 +CPU= 20.380
  12:10:37 STEP 1386 H=1386:00 +CPU= 33.452
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:11:58 STEP 1387 H=1387:00 +CPU= 20.661
  12:12:19 STEP 1388 H=1388:00 +CPU= 20.985
  12:12:52 STEP 1389 H=1389:00 +CPU= 32.103
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:14:15 STEP 1390 H=1390:00 +CPU= 21.791
  12:14:36 STEP 1391 H=1391:00 +CPU= 21.257
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:16:16 STEP 1392 H=1392:00 +CPU= 38.770
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+001392
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+001392
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182974/ICMGGhq0f+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182974/ICMSHhq0f+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182974/ICMUAhq0f+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182974/ICMGGhq0f+001380
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182974/ICMSHhq0f+001380
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182974/ICMUAhq0f+001380
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0330_1997050100_123_966_12182974_0_r1282578787_57.zip
Uploading the intermediate file: upload_file_57.zip
  12:16:36 STEP 1393 H=1393:00 +CPU= 20.068
  12:16:57 STEP 1394 H=1394:00 +CPU= 20.817
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:18:41 STEP 1395 H=1395:00 +CPU= 32.812
  12:19:02 STEP 1396 H=1396:00 +CPU= 21.174
  12:19:22 STEP 1397 H=1397:00 +CPU= 19.924
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:20:48 STEP 1398 H=1398:00 +CPU= 33.609
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:21:41 STEP 1399 H=1399:00 +CPU= 20.207
  12:22:02 STEP 1400 H=1400:00 +CPU= 21.401
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:23:36 STEP 1401 H=1401:00 +CPU= 32.353
  12:23:56 STEP 1402 H=1402:00 +CPU= 20.631
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:25:09 STEP 1403 H=1403:00 +CPU= 20.585
  12:25:45 STEP 1404 H=1404:00 +CPU= 35.878
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+001404
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+001404
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+001404
  12:26:05 STEP 1405 H=1405:00 +CPU= 19.941
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:27:28 STEP 1406 H=1406:00 +CPU= 21.840
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:28:54 STEP 1407 H=1407:00 +CPU= 33.417
  12:29:15 STEP 1408 H=1408:00 +CPU= 21.204
  12:29:35 STEP 1409 H=1409:00 +CPU= 20.018
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:30:59 STEP 1410 H=1410:00 +CPU= 33.534
  12:31:19 STEP 1411 H=1411:00 +CPU= 20.243
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:32:41 STEP 1412 H=1412:00 +CPU= 20.666
  12:33:14 STEP 1413 H=1413:00 +CPU= 33.136
  12:33:35 STEP 1414 H=1414:00 +CPU= 20.567
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:34:58 STEP 1415 H=1415:00 +CPU= 20.818
  12:35:37 STEP 1416 H=1416:00 +CPU= 39.206
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+001416
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+001416
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182974/ICMGGhq0f+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182974/ICMSHhq0f+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182974/ICMUAhq0f+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182974/ICMGGhq0f+001404
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182974/ICMSHhq0f+001404
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182974/ICMUAhq0f+001404
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0330_1997050100_123_966_12182974_0_r1282578787_58.zip
Uploading the intermediate file: upload_file_58.zip
  12:36:59 STEP 1417 H=1417:00 +CPU= 20.898
  12:37:20 STEP 1418 H=1418:00 +CPU= 20.629
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:39:05 STEP 1419 H=1419:00 +CPU= 32.713
  12:39:26 STEP 1420 H=1420:00 +CPU= 21.225
  12:39:47 STEP 1421 H=1421:00 +CPU= 20.241
  12:40:20 STEP 1422 H=1422:00 +CPU= 33.162
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:41:52 STEP 1423 H=1423:00 +CPU= 21.191
  12:42:14 STEP 1424 H=1424:00 +CPU= 21.133
  12:42:46 STEP 1425 H=1425:00 +CPU= 32.233
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:44:19 STEP 1426 H=1426:00 +CPU= 21.364
  12:44:40 STEP 1427 H=1427:00 +CPU= 20.558
  12:45:16 STEP 1428 H=1428:00 +CPU= 35.978
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+001428
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+001428
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+001428
  12:46:28 STEP 1429 H=1429:00 +CPU= 21.107
  12:46:49 STEP 1430 H=1430:00 +CPU= 20.950
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:48:25 STEP 1431 H=1431:00 +CPU= 32.851
  12:48:46 STEP 1432 H=1432:00 +CPU= 21.262
  12:49:06 STEP 1433 H=1433:00 +CPU= 20.091
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:50:40 STEP 1434 H=1434:00 +CPU= 33.517
  12:51:00 STEP 1435 H=1435:00 +CPU= 20.314
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:52:22 STEP 1436 H=1436:00 +CPU= 21.353
  12:52:54 STEP 1437 H=1437:00 +CPU= 32.048
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:54:06 STEP 1438 H=1438:00 +CPU= 20.420
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:55:10 STEP 1439 H=1439:00 +CPU= 22.531
  12:55:48 STEP 1440 H=1440:00 +CPU= 38.080
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+001440
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+001440
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182974/ICMGGhq0f+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182974/ICMSHhq0f+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182974/ICMUAhq0f+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182974/ICMGGhq0f+001428
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182974/ICMSHhq0f+001428
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182974/ICMUAhq0f+001428
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0330_1997050100_123_966_12182974_0_r1282578787_59.zip
Uploading the intermediate file: upload_file_59.zip
  12:56:09 STEP 1441 H=1441:00 +CPU= 20.459
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:57:21 STEP 1442 H=1442:00 +CPU= 21.204
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:59:46 STEP 1443 H=1443:00 +CPU= 33.506
  13:00:13 STEP 1444 H=1444:00 +CPU= 25.565
  13:00:37 STEP 1445 H=1445:00 +CPU= 24.253
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  13:02:46 STEP 1446 H=1446:00 +CPU= 37.638
  13:03:09 STEP 1447 H=1447:00 +CPU= 22.546
  13:03:31 STEP 1448 H=1448:00 +CPU= 21.891
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  13:05:36 STEP 1449 H=1449:00 +CPU= 34.566
  13:05:59 STEP 1450 H=1450:00 +CPU= 22.639
  13:06:22 STEP 1451 H=1451:00 +CPU= 22.482
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1997050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0330
(argv4) batchid: 966
(argv5) wuid: 12182974
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/0
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182974
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_987e57a9ceb96035b02b9af7e354e042 to: /var/lib/boinc-client/slots/0/oifs_43r3_ps_0330_1997050100_123_966_12182974.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/0/oifs_43r3_ps_0330_1997050100_123_966_12182974.zip
ic_ancil_file: ic_ancil_12182974
ifsdata_file: ifsdata_12182974
climate_data_file: clim_data_12182974
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_8a2bcbaa7adb2422ec8b59d850d21b5e to: /var/lib/boinc-client/slots/0/ic_ancil_12182974.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/0/ic_ancil_12182974.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_d7f9b4096455331a68677c534044aad5 to: /var/lib/boinc-client/slots/0/ifsdata/ifsdata_12182974.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/0/ifsdata/ifsdata_12182974.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/0/159l_2/clim_data_12182974.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/0/159l_2/clim_data_12182974.zip
Checking for progress XML file: /var/lib/boinc-client/slots/0/progress_file_12182974.xml
Opened progress file ok : /var/lib/boinc-client/slots/0/progress_file_12182974.xml
-- Model is restarting --
Adjusting last_iter, 1452, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/0/progress_file_12182974.xml
last_cpu_time: 38535
upload_file_number: 60
last_iter: 1439
last_upload: 5184000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0330_1997050100_123_966_12182974_0_r1282578787
The child process has been launched with process id: 1207111
Executing the command: /var/lib/boinc-client/slots/0/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:p500:1:1:1207111:1207111] [20230105:131924:1672892364:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:p500:1:1:1207111:1207111] [20230105:131924:1672892364:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:p500:1:1:1207111:1207111] [20230105:131924:1672892364:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:p500:1:1:1207111:1207111] [20230105:131924:1672892364:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:p500:1:1:1207111:1207111] [20230105:131924:1672892364:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:p500:1:1:1207111:1207111] [20230105:131924:1672892364:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:p500:1:1:1207111:1207111] [20230105:131924:1672892364:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:p500:1:1:1207111:1207111] [20230105:131924:1672892364:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:p500:1:1:1207111:1207111] [20230105:131924:1672892364:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:p500:1:1:1207111:1207111] [20230105:131924:1672892364:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:1207111:1207111] [20230105:131924:1672892364:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:p500:1:1:1207111:1207111] [20230105:131924:1672892364:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:p500:1:1:1207111:1207111] [20230105:131924:1672892364:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:p500:1:1:1207111:1207111] [20230105:131924:1672892364:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:1207111:1207111] [20230105:131924:1672892364:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:1207111:1207111] [20230105:131924:1672892364:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1207111:1207111] [20230105:131924:1672892364:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:1207111:1207111] [20230105:131924:1672892364:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1207111:1207111] [20230105:131924:1672892364:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:1207111:1207111] [20230105:131924:1672892364:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:1207111:1207111] [20230105:131924:1672892364:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1207111:1207111] [20230105:131924:1672892364:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1207111:1207111] [20230105:131924:1672892364:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:1207111:1207111] [20230105:131924:1672892364:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1207111:1207111] [20230105:131924:1672892364:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1207111:1207111] [20230105:131924:1672892364:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1207111:1207111] [20230105:131924:1672892364:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:1207111:1207111] [20230105:131924:1672892364:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:1207111:1207111] [20230105:131924:1672892364:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:1207111:1207111] [20230105:131924:1672892364:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  13:21:41 STEP 1440 H=1440:00 +CPU= 40.772
  13:22:04 STEP 1441 H=1441:00 +CPU= 22.653
  13:22:28 STEP 1442 H=1442:00 +CPU= 23.315
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  13:24:16 STEP 1443 H=1443:00 +CPU= 35.860
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  13:27:02 STEP 1444 H=1444:00 +CPU= 24.624
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  13:39:19 STEP 1445 H=1445:00 +CPU= 23.728
  13:39:57 STEP 1446 H=1446:00 +CPU= 37.449
  13:40:20 STEP 1447 H=1447:00 +CPU= 21.990
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  13:43:35 STEP 1448 H=1448:00 +CPU= 23.465
  13:44:12 STEP 1449 H=1449:00 +CPU= 36.827
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  13:49:48 STEP 1450 H=1450:00 +CPU= 23.900
  13:50:11 STEP 1451 H=1451:00 +CPU= 22.457
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  13:53:03 STEP 1452 H=1452:00 +CPU= 39.780
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+001452
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+001452
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+001452
  13:53:25 STEP 1453 H=1453:00 +CPU= 22.123
  13:53:45 STEP 1454 H=1454:00 +CPU= 20.089
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  13:56:19 STEP 1455 H=1455:00 +CPU= 33.024
  13:56:41 STEP 1456 H=1456:00 +CPU= 21.152
  13:57:00 STEP 1457 H=1457:00 +CPU= 19.526
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  13:58:55 STEP 1458 H=1458:00 +CPU= 33.283
  13:59:15 STEP 1459 H=1459:00 +CPU= 20.051
  13:59:36 STEP 1460 H=1460:00 +CPU= 20.577
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:00:59 STEP 1461 H=1461:00 +CPU= 32.630
  14:01:20 STEP 1462 H=1462:00 +CPU= 20.705
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:02:43 STEP 1463 H=1463:00 +CPU= 20.763
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:04:04 STEP 1464 H=1464:00 +CPU= 38.695
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+001464
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+001464
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+001464
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182974/ICMGGhq0f+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182974/ICMSHhq0f+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182974/ICMUAhq0f+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182974/ICMGGhq0f+001452
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182974/ICMSHhq0f+001452
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182974/ICMUAhq0f+001452
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0330_1997050100_123_966_12182974_0_r1282578787_60.zip
Uploading the intermediate file: upload_file_60.zip
  14:04:24 STEP 1465 H=1465:00 +CPU= 20.351
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:05:37 STEP 1466 H=1466:00 +CPU= 20.542
  14:06:10 STEP 1467 H=1467:00 +CPU= 32.976
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:07:42 STEP 1468 H=1468:00 +CPU= 20.529
  14:08:03 STEP 1469 H=1469:00 +CPU= 21.143
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:10:09 STEP 1470 H=1470:00 +CPU= 33.730
  14:10:29 STEP 1471 H=1471:00 +CPU= 20.242
  14:10:50 STEP 1472 H=1472:00 +CPU= 20.613
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:12:24 STEP 1473 H=1473:00 +CPU= 32.960
  14:12:44 STEP 1474 H=1474:00 +CPU= 20.138
  14:13:04 STEP 1475 H=1475:00 +CPU= 19.922
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:16:04 STEP 1476 H=1476:00 +CPU= 36.527
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+001476
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+001476
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+001476
  14:16:25 STEP 1477 H=1477:00 +CPU= 20.637
  14:16:47 STEP 1478 H=1478:00 +CPU= 21.506
  14:17:18 STEP 1479 H=1479:00 +CPU= 31.505
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:19:00 STEP 1480 H=1480:00 +CPU= 20.454
  14:19:20 STEP 1481 H=1481:00 +CPU= 20.210
  14:19:53 STEP 1482 H=1482:00 +CPU= 32.870
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:21:24 STEP 1483 H=1483:00 +CPU= 20.281
  14:21:46 STEP 1484 H=1484:00 +CPU= 21.376
  14:22:18 STEP 1485 H=1485:00 +CPU= 32.206
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:23:50 STEP 1486 H=1486:00 +CPU= 21.152
  14:24:10 STEP 1487 H=1487:00 +CPU= 20.462
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:25:59 STEP 1488 H=1488:00 +CPU= 37.692
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+001488
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+001488
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+001488
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182974/ICMGGhq0f+001464
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182974/ICMSHhq0f+001464
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182974/ICMUAhq0f+001464
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182974/ICMGGhq0f+001476
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182974/ICMSHhq0f+001476
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182974/ICMUAhq0f+001476
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0330_1997050100_123_966_12182974_0_r1282578787_61.zip
Uploading the intermediate file: upload_file_61.zip
  14:26:19 STEP 1489 H=1489:00 +CPU= 19.672
  14:26:40 STEP 1490 H=1490:00 +CPU= 20.993
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:28:35 STEP 1491 H=1491:00 +CPU= 33.131
  14:28:56 STEP 1492 H=1492:00 +CPU= 20.529
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:30:18 STEP 1493 H=1493:00 +CPU= 20.791
  14:30:51 STEP 1494 H=1494:00 +CPU= 32.744
  14:31:10 STEP 1495 H=1495:00 +CPU= 19.705
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:32:42 STEP 1496 H=1496:00 +CPU= 21.081
  14:33:14 STEP 1497 H=1497:00 +CPU= 32.708
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:34:46 STEP 1498 H=1498:00 +CPU= 20.197
  14:35:07 STEP 1499 H=1499:00 +CPU= 21.212
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:36:24 STEP 1500 H=1500:00 +CPU= 35.603
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+001500
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+001500
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+001500
  14:36:44 STEP 1501 H=1501:00 +CPU= 20.461
  14:37:05 STEP 1502 H=1502:00 +CPU= 20.532
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:38:39 STEP 1503 H=1503:00 +CPU= 32.986
  14:39:00 STEP 1504 H=1504:00 +CPU= 20.610
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:40:22 STEP 1505 H=1505:00 +CPU= 21.014
  14:40:55 STEP 1506 H=1506:00 +CPU= 33.138
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:42:27 STEP 1507 H=1507:00 +CPU= 20.361
  14:42:48 STEP 1508 H=1508:00 +CPU= 21.086
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:44:12 STEP 1509 H=1509:00 +CPU= 32.749
  14:44:32 STEP 1510 H=1510:00 +CPU= 20.387
  14:44:52 STEP 1511 H=1511:00 +CPU= 20.222
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:46:32 STEP 1512 H=1512:00 +CPU= 38.768
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+001512
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+001512
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+001512
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182974/ICMGGhq0f+001488
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182974/ICMSHhq0f+001488
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182974/ICMUAhq0f+001488
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182974/ICMGGhq0f+001500
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182974/ICMSHhq0f+001500
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182974/ICMUAhq0f+001500
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0330_1997050100_123_966_12182974_0_r1282578787_62.zip
Uploading the intermediate file: upload_file_62.zip
  14:46:53 STEP 1513 H=1513:00 +CPU= 20.259
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:48:06 STEP 1514 H=1514:00 +CPU= 20.763
  14:48:38 STEP 1515 H=1515:00 +CPU= 32.483
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:49:51 STEP 1516 H=1516:00 +CPU= 20.798
  14:50:12 STEP 1517 H=1517:00 +CPU= 20.539
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:51:47 STEP 1518 H=1518:00 +CPU= 33.119
  14:52:07 STEP 1519 H=1519:00 +CPU= 19.978
  14:52:28 STEP 1520 H=1520:00 +CPU= 20.896
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:54:01 STEP 1521 H=1521:00 +CPU= 32.728
  14:54:22 STEP 1522 H=1522:00 +CPU= 20.550
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:55:42 STEP 1523 H=1523:00 +CPU= 20.363
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:57:21 STEP 1524 H=1524:00 +CPU= 36.160
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+001524
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+001524
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+001524
  14:57:41 STEP 1525 H=1525:00 +CPU= 20.551
  14:58:02 STEP 1526 H=1526:00 +CPU= 20.718
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:59:35 STEP 1527 H=1527:00 +CPU= 32.946
  14:59:56 STEP 1528 H=1528:00 +CPU= 20.653
  15:00:16 STEP 1529 H=1529:00 +CPU= 19.738
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:02:10 STEP 1530 H=1530:00 +CPU= 32.852
  15:02:29 STEP 1531 H=1531:00 +CPU= 19.649
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:03:42 STEP 1532 H=1532:00 +CPU= 21.032
  15:04:15 STEP 1533 H=1533:00 +CPU= 33.069
  15:04:35 STEP 1534 H=1534:00 +CPU= 20.109
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:06:08 STEP 1535 H=1535:00 +CPU= 21.458
  15:06:45 STEP 1536 H=1536:00 +CPU= 37.192
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+001536
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+001536
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182974/ICMGGhq0f+001512
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182974/ICMSHhq0f+001512
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182974/ICMUAhq0f+001512
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182974/ICMGGhq0f+001524
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182974/ICMSHhq0f+001524
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182974/ICMUAhq0f+001524
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0330_1997050100_123_966_12182974_0_r1282578787_63.zip
Uploading the intermediate file: upload_file_63.zip
  15:07:04 STEP 1537 H=1537:00 +CPU= 18.862
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:08:16 STEP 1538 H=1538:00 +CPU= 19.111
  15:08:48 STEP 1539 H=1539:00 +CPU= 32.267
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:10:09 STEP 1540 H=1540:00 +CPU= 20.360
  15:10:30 STEP 1541 H=1541:00 +CPU= 21.134
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:12:05 STEP 1542 H=1542:00 +CPU= 33.587
  15:12:26 STEP 1543 H=1543:00 +CPU= 21.008
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:13:49 STEP 1544 H=1544:00 +CPU= 20.998
  15:14:22 STEP 1545 H=1545:00 +CPU= 32.287
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:15:33 STEP 1546 H=1546:00 +CPU= 20.922
  15:15:53 STEP 1547 H=1547:00 +CPU= 20.498
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:18:23 STEP 1548 H=1548:00 +CPU= 36.592
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+001548
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+001548
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+001548
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:19:25 STEP 1549 H=1549:00 +CPU= 20.435
  15:19:46 STEP 1550 H=1550:00 +CPU= 20.970
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1997050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0330
(argv4) batchid: 966
(argv5) wuid: 12182974
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/0
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182974
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_987e57a9ceb96035b02b9af7e354e042 to: /var/lib/boinc-client/slots/0/oifs_43r3_ps_0330_1997050100_123_966_12182974.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/0/oifs_43r3_ps_0330_1997050100_123_966_12182974.zip
ic_ancil_file: ic_ancil_12182974
ifsdata_file: ifsdata_12182974
climate_data_file: clim_data_12182974
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_8a2bcbaa7adb2422ec8b59d850d21b5e to: /var/lib/boinc-client/slots/0/ic_ancil_12182974.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/0/ic_ancil_12182974.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_d7f9b4096455331a68677c534044aad5 to: /var/lib/boinc-client/slots/0/ifsdata/ifsdata_12182974.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/0/ifsdata/ifsdata_12182974.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/0/159l_2/clim_data_12182974.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/0/159l_2/clim_data_12182974.zip
Checking for progress XML file: /var/lib/boinc-client/slots/0/progress_file_12182974.xml
Opened progress file ok : /var/lib/boinc-client/slots/0/progress_file_12182974.xml
-- Model is restarting --
Adjusting last_iter, 1550, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/0/progress_file_12182974.xml
last_cpu_time: 41352
upload_file_number: 64
last_iter: 1535
last_upload: 5529600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0330_1997050100_123_966_12182974_0_r1282578787
The child process has been launched with process id: 1225350
Executing the command: /var/lib/boinc-client/slots/0/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:p500:1:1:1225350:1225350] [20230105:152101:1672899661:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:p500:1:1:1225350:1225350] [20230105:152101:1672899661:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:p500:1:1:1225350:1225350] [20230105:152101:1672899661:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:p500:1:1:1225350:1225350] [20230105:152101:1672899661:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:p500:1:1:1225350:1225350] [20230105:152101:1672899661:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:p500:1:1:1225350:1225350] [20230105:152101:1672899661:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:p500:1:1:1225350:1225350] [20230105:152101:1672899661:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:p500:1:1:1225350:1225350] [20230105:152101:1672899661:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:p500:1:1:1225350:1225350] [20230105:152101:1672899661:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:p500:1:1:1225350:1225350] [20230105:152101:1672899661:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:1225350:1225350] [20230105:152101:1672899661:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:p500:1:1:1225350:1225350] [20230105:152101:1672899661:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:p500:1:1:1225350:1225350] [20230105:152101:1672899661:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:p500:1:1:1225350:1225350] [20230105:152101:1672899661:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:1225350:1225350] [20230105:152101:1672899661:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:1225350:1225350] [20230105:152101:1672899661:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1225350:1225350] [20230105:152101:1672899661:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:1225350:1225350] [20230105:152101:1672899661:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1225350:1225350] [20230105:152101:1672899661:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:1225350:1225350] [20230105:152101:1672899661:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:1225350:1225350] [20230105:152101:1672899661:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1225350:1225350] [20230105:152101:1672899661:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1225350:1225350] [20230105:152101:1672899661:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:1225350:1225350] [20230105:152101:1672899661:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1225350:1225350] [20230105:152101:1672899661:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1225350:1225350] [20230105:152101:1672899661:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1225350:1225350] [20230105:152101:1672899661:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:1225350:1225350] [20230105:152101:1672899661:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:1225350:1225350] [20230105:152101:1672899661:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:1225350:1225350] [20230105:152101:1672899661:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  15:21:40 STEP 1536 H=1536:00 +CPU= 36.301
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:23:10 STEP 1537 H=1537:00 +CPU= 19.596
  15:23:32 STEP 1538 H=1538:00 +CPU= 21.815
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:25:25 STEP 1539 H=1539:00 +CPU= 31.954
  15:25:47 STEP 1540 H=1540:00 +CPU= 21.805
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:27:09 STEP 1541 H=1541:00 +CPU= 20.538
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:28:44 STEP 1542 H=1542:00 +CPU= 33.244
  15:29:05 STEP 1543 H=1543:00 +CPU= 20.644
  15:29:26 STEP 1544 H=1544:00 +CPU= 20.877
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:31:09 STEP 1545 H=1545:00 +CPU= 32.177
  15:31:30 STEP 1546 H=1546:00 +CPU= 21.224
  15:31:50 STEP 1547 H=1547:00 +CPU= 19.780
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:33:27 STEP 1548 H=1548:00 +CPU= 35.325
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+001548
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+001548
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+001548
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:34:50 STEP 1549 H=1549:00 +CPU= 21.028
  15:35:11 STEP 1550 H=1550:00 +CPU= 21.061
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:36:54 STEP 1551 H=1551:00 +CPU= 31.891
  15:37:15 STEP 1552 H=1552:00 +CPU= 20.942
  15:37:36 STEP 1553 H=1553:00 +CPU= 20.445
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:39:00 STEP 1554 H=1554:00 +CPU= 33.284
  15:39:20 STEP 1555 H=1555:00 +CPU= 19.808
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:40:40 STEP 1556 H=1556:00 +CPU= 20.322
  15:41:13 STEP 1557 H=1557:00 +CPU= 32.067
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:42:54 STEP 1558 H=1558:00 +CPU= 20.561
  15:43:15 STEP 1559 H=1559:00 +CPU= 20.965
  15:43:53 STEP 1560 H=1560:00 +CPU= 37.547
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+001560
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+001560
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182974/ICMGGhq0f+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182974/ICMSHhq0f+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182974/ICMUAhq0f+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182974/ICMGGhq0f+001548
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182974/ICMSHhq0f+001548
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182974/ICMUAhq0f+001548
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0330_1997050100_123_966_12182974_0_r1282578787_64.zip
Uploading the intermediate file: upload_file_64.zip
  15:44:13 STEP 1561 H=1561:00 +CPU= 19.492
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:45:37 STEP 1562 H=1562:00 +CPU= 19.744
  15:46:09 STEP 1563 H=1563:00 +CPU= 31.620
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:47:51 STEP 1564 H=1564:00 +CPU= 20.668
  15:48:11 STEP 1565 H=1565:00 +CPU= 20.192
  15:48:45 STEP 1566 H=1566:00 +CPU= 33.148
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:50:17 STEP 1567 H=1567:00 +CPU= 21.210
  15:50:38 STEP 1568 H=1568:00 +CPU= 20.647
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:52:32 STEP 1569 H=1569:00 +CPU= 32.083
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:53:45 STEP 1570 H=1570:00 +CPU= 20.755
  15:54:07 STEP 1571 H=1571:00 +CPU= 21.568
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:55:43 STEP 1572 H=1572:00 +CPU= 36.057
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+001572
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+001572
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+001572
  15:56:03 STEP 1573 H=1573:00 +CPU= 20.136
  15:56:24 STEP 1574 H=1574:00 +CPU= 20.844
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:58:17 STEP 1575 H=1575:00 +CPU= 32.504
  15:58:38 STEP 1576 H=1576:00 +CPU= 20.989
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:00:21 STEP 1577 H=1577:00 +CPU= 21.037
  16:00:53 STEP 1578 H=1578:00 +CPU= 32.479
  16:01:13 STEP 1579 H=1579:00 +CPU= 19.733
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:02:36 STEP 1580 H=1580:00 +CPU= 21.580
  16:03:08 STEP 1581 H=1581:00 +CPU= 31.646
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Abort request received from BOINC client, ending the child process

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:07:55 1519251 22268122 oifs_43r3_ps_0330_1997050100_123_966_12182974_0 1,728,000 11,893 0.0069
09 May 2023 16:07:55 1519251 22268122 oifs_43r3_ps_0330_1997050100_123_966_12182974_0 1,728,000 11,893 0.0069
09 May 2023 16:07:49 1519251 22268122 oifs_43r3_ps_0330_1997050100_123_966_12182974_0 864,000 6,110 0.0071


©2024 climateprediction.net