climateprediction.net home page
Task 22446464

Task 22446464

Name oifs_43r3_bl_a3hr_2016092300_20_1018_12292854_0
Workunit 12292854
Created 13 Jun 2024, 2:34:32 UTC
Sent 14 Jun 2024, 5:33:42 UTC
Report deadline 13 Aug 2024, 5:33:42 UTC
Received 15 Jun 2024, 11:01:02 UTC
Server state Over
Outcome Success
Client state Done
Exit status 0 (0x00000000)
Computer ID 1517679
Run time 16 hours 36 min 22 sec
CPU time 16 hours 24 min 21 sec
Validate state Valid
Credit 2,635.55
Device peak FLOPS 5.26 GFLOPS
Application version OpenIFS 43r3 Baroclinic Lifecycle v1.13
x86_64-pc-linux-gnu
Peak working set size 5,025.82 MB
Peak swap size 5,981.13 MB
Peak disk usage 1,825.37 MB
Stderr
<core_client_version>7.18.1</core_client_version>
<![CDATA[
<stderr_txt>
3:25:53 STEP 1527 H= 381:45 +CPU= 19.770
  23:26:12 STEP 1528 H= 382:00 +CPU= 19.472
  23:26:32 STEP 1529 H= 382:15 +CPU= 19.591
  23:26:57 STEP 1530 H= 382:30 +CPU= 24.685
  23:27:25 STEP 1531 H= 382:45 +CPU= 27.310
  23:27:53 STEP 1532 H= 383:00 +CPU= 27.530
  23:28:20 STEP 1533 H= 383:15 +CPU= 26.773
  23:28:48 STEP 1534 H= 383:30 +CPU= 26.584
  23:29:15 STEP 1535 H= 383:45 +CPU= 26.655
  23:29:49 STEP 1536 H= 384:00 +CPU= 34.042
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMGGh7zg+001536
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMSHh7zg+001536
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMUAh7zg+001536
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMGGh7zg+001440
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMSHh7zg+001440
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMUAh7zg+001440
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMGGh7zg+001452
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMSHh7zg+001452
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMUAh7zg+001452
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMGGh7zg+001464
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMSHh7zg+001464
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMUAh7zg+001464
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMGGh7zg+001476
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMSHh7zg+001476
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMUAh7zg+001476
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMGGh7zg+001488
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMSHh7zg+001488
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMUAh7zg+001488
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMGGh7zg+001500
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMSHh7zg+001500
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMUAh7zg+001500
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMGGh7zg+001512
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMSHh7zg+001512
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMUAh7zg+001512
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMGGh7zg+001524
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMSHh7zg+001524
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMUAh7zg+001524
Zipping up the intermediate file: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a3hr_2016092300_20_1018_12292854_0_r265557904_15.zip
Uploading the intermediate file: upload_file_15.zip
  23:30:21 STEP 1537 H= 384:15 +CPU= 31.655
  23:30:59 STEP 1538 H= 384:30 +CPU= 36.930
  23:31:38 STEP 1539 H= 384:45 +CPU= 35.428
  23:32:22 STEP 1540 H= 385:00 +CPU= 38.703
  23:33:05 STEP 1541 H= 385:15 +CPU= 38.049
  23:33:47 STEP 1542 H= 385:30 +CPU= 37.313
  23:34:30 STEP 1543 H= 385:45 +CPU= 37.677
  23:35:14 STEP 1544 H= 386:00 +CPU= 38.848
  23:35:54 STEP 1545 H= 386:15 +CPU= 35.130
  23:36:35 STEP 1546 H= 386:30 +CPU= 36.787
  23:37:13 STEP 1547 H= 386:45 +CPU= 34.092
  23:37:38 STEP 1548 H= 387:00 +CPU= 24.932
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMGGh7zg+001548
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMSHh7zg+001548
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMUAh7zg+001548
  23:37:58 STEP 1549 H= 387:15 +CPU= 19.923
  23:38:18 STEP 1550 H= 387:30 +CPU= 19.543
  23:38:41 STEP 1551 H= 387:45 +CPU= 22.655
  23:39:10 STEP 1552 H= 388:00 +CPU= 27.745
  23:39:39 STEP 1553 H= 388:15 +CPU= 27.538
  23:40:08 STEP 1554 H= 388:30 +CPU= 27.226
  23:40:36 STEP 1555 H= 388:45 +CPU= 27.104
  23:41:05 STEP 1556 H= 389:00 +CPU= 27.191
  23:41:33 STEP 1557 H= 389:15 +CPU= 26.298
  23:42:02 STEP 1558 H= 389:30 +CPU= 26.923
  23:42:31 STEP 1559 H= 389:45 +CPU= 26.907
  23:43:07 STEP 1560 H= 390:00 +CPU= 33.230
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMGGh7zg+001560
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMSHh7zg+001560
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMUAh7zg+001560
  23:43:35 STEP 1561 H= 390:15 +CPU= 26.710
  23:44:04 STEP 1562 H= 390:30 +CPU= 26.873
  23:44:33 STEP 1563 H= 390:45 +CPU= 27.027
  23:45:02 STEP 1564 H= 391:00 +CPU= 27.491
  23:45:30 STEP 1565 H= 391:15 +CPU= 26.972
  23:45:58 STEP 1566 H= 391:30 +CPU= 26.784
  23:46:26 STEP 1567 H= 391:45 +CPU= 26.469
  23:46:54 STEP 1568 H= 392:00 +CPU= 26.946
  23:47:23 STEP 1569 H= 392:15 +CPU= 26.997
  23:47:51 STEP 1570 H= 392:30 +CPU= 26.609
  23:48:19 STEP 1571 H= 392:45 +CPU= 26.410
  23:48:54 STEP 1572 H= 393:00 +CPU= 32.963
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMGGh7zg+001572
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMSHh7zg+001572
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMUAh7zg+001572
  23:49:22 STEP 1573 H= 393:15 +CPU= 26.561
  23:49:50 STEP 1574 H= 393:30 +CPU= 26.793
  23:50:18 STEP 1575 H= 393:45 +CPU= 26.576
  23:50:47 STEP 1576 H= 394:00 +CPU= 27.002
  23:51:15 STEP 1577 H= 394:15 +CPU= 26.491
  23:51:42 STEP 1578 H= 394:30 +CPU= 26.369
  23:52:10 STEP 1579 H= 394:45 +CPU= 26.587
  23:52:39 STEP 1580 H= 395:00 +CPU= 27.140
  23:53:08 STEP 1581 H= 395:15 +CPU= 26.521
  23:53:36 STEP 1582 H= 395:30 +CPU= 26.657
  23:54:05 STEP 1583 H= 395:45 +CPU= 26.727
  23:54:44 STEP 1584 H= 396:00 +CPU= 36.268
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMGGh7zg+001584
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMSHh7zg+001584
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMUAh7zg+001584
  23:55:14 STEP 1585 H= 396:15 +CPU= 26.668
  23:55:44 STEP 1586 H= 396:30 +CPU= 26.634
  23:56:13 STEP 1587 H= 396:45 +CPU= 26.673
  23:56:43 STEP 1588 H= 397:00 +CPU= 27.615
  23:57:11 STEP 1589 H= 397:15 +CPU= 26.573
  23:57:41 STEP 1590 H= 397:30 +CPU= 26.737
  23:58:09 STEP 1591 H= 397:45 +CPU= 26.622
  23:58:37 STEP 1592 H= 398:00 +CPU= 26.922
  23:59:05 STEP 1593 H= 398:15 +CPU= 26.361
  23:59:34 STEP 1594 H= 398:30 +CPU= 26.308
  00:00:01 STEP 1595 H= 398:45 +CPU= 26.307
  00:00:35 STEP 1596 H= 399:00 +CPU= 32.299
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMGGh7zg+001596
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMSHh7zg+001596
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMUAh7zg+001596
  00:01:04 STEP 1597 H= 399:15 +CPU= 26.384
  00:01:33 STEP 1598 H= 399:30 +CPU= 26.319
  00:02:02 STEP 1599 H= 399:45 +CPU= 26.623
  00:02:30 STEP 1600 H= 400:00 +CPU= 26.414
  00:02:58 STEP 1601 H= 400:15 +CPU= 26.416
  00:03:25 STEP 1602 H= 400:30 +CPU= 26.313
  00:03:54 STEP 1603 H= 400:45 +CPU= 26.628
  00:04:22 STEP 1604 H= 401:00 +CPU= 27.029
  00:04:50 STEP 1605 H= 401:15 +CPU= 26.139
  00:05:17 STEP 1606 H= 401:30 +CPU= 26.134
  00:05:44 STEP 1607 H= 401:45 +CPU= 25.863
  00:06:18 STEP 1608 H= 402:00 +CPU= 32.722
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMGGh7zg+001608
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMSHh7zg+001608
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMUAh7zg+001608
  00:06:45 STEP 1609 H= 402:15 +CPU= 26.347
  00:07:12 STEP 1610 H= 402:30 +CPU= 26.357
  00:07:39 STEP 1611 H= 402:45 +CPU= 26.292
  00:08:06 STEP 1612 H= 403:00 +CPU= 26.835
  00:08:33 STEP 1613 H= 403:15 +CPU= 26.264
  00:09:00 STEP 1614 H= 403:30 +CPU= 26.447
  00:09:27 STEP 1615 H= 403:45 +CPU= 26.229
  00:09:54 STEP 1616 H= 404:00 +CPU= 26.736
  00:10:21 STEP 1617 H= 404:15 +CPU= 26.621
  00:10:48 STEP 1618 H= 404:30 +CPU= 26.392
  00:11:15 STEP 1619 H= 404:45 +CPU= 26.071
  00:11:48 STEP 1620 H= 405:00 +CPU= 32.521
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMGGh7zg+001620
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMSHh7zg+001620
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMUAh7zg+001620
  00:12:14 STEP 1621 H= 405:15 +CPU= 26.121
  00:12:41 STEP 1622 H= 405:30 +CPU= 26.719
  00:13:08 STEP 1623 H= 405:45 +CPU= 26.193
  00:13:35 STEP 1624 H= 406:00 +CPU= 26.581
  00:14:02 STEP 1625 H= 406:15 +CPU= 26.094
  00:14:28 STEP 1626 H= 406:30 +CPU= 26.167
  00:14:55 STEP 1627 H= 406:45 +CPU= 25.917
  00:15:22 STEP 1628 H= 407:00 +CPU= 26.708
  00:15:49 STEP 1629 H= 407:15 +CPU= 26.148
  00:16:18 STEP 1630 H= 407:30 +CPU= 27.859
  00:16:44 STEP 1631 H= 407:45 +CPU= 26.068
  00:17:20 STEP 1632 H= 408:00 +CPU= 35.251
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMGGh7zg+001632
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMSHh7zg+001632
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMUAh7zg+001632
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMGGh7zg+001536
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMSHh7zg+001536
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMUAh7zg+001536
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMGGh7zg+001548
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMSHh7zg+001548
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMUAh7zg+001548
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMGGh7zg+001560
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMSHh7zg+001560
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMUAh7zg+001560
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMGGh7zg+001572
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMSHh7zg+001572
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMUAh7zg+001572
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMGGh7zg+001584
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMSHh7zg+001584
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMUAh7zg+001584
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMGGh7zg+001596
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMSHh7zg+001596
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMUAh7zg+001596
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMGGh7zg+001608
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMSHh7zg+001608
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMUAh7zg+001608
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMGGh7zg+001620
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMSHh7zg+001620
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMUAh7zg+001620
Zipping up the intermediate file: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a3hr_2016092300_20_1018_12292854_0_r265557904_16.zip
Uploading the intermediate file: upload_file_16.zip
  00:17:47 STEP 1633 H= 408:15 +CPU= 26.170
  00:18:14 STEP 1634 H= 408:30 +CPU= 26.590
  00:18:41 STEP 1635 H= 408:45 +CPU= 25.856
  00:19:07 STEP 1636 H= 409:00 +CPU= 25.962
  00:19:33 STEP 1637 H= 409:15 +CPU= 25.371
  00:19:59 STEP 1638 H= 409:30 +CPU= 25.424
  00:20:25 STEP 1639 H= 409:45 +CPU= 25.161
  00:20:52 STEP 1640 H= 410:00 +CPU= 25.744
  00:21:18 STEP 1641 H= 410:15 +CPU= 25.303
  00:21:44 STEP 1642 H= 410:30 +CPU= 25.193
  00:22:09 STEP 1643 H= 410:45 +CPU= 25.242
  00:22:41 STEP 1644 H= 411:00 +CPU= 31.361
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMGGh7zg+001644
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMSHh7zg+001644
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMUAh7zg+001644
  00:23:08 STEP 1645 H= 411:15 +CPU= 25.934
  00:23:34 STEP 1646 H= 411:30 +CPU= 26.146
  00:24:02 STEP 1647 H= 411:45 +CPU= 27.562
  00:24:33 STEP 1648 H= 412:00 +CPU= 30.414
  00:25:03 STEP 1649 H= 412:15 +CPU= 29.351
  00:25:33 STEP 1650 H= 412:30 +CPU= 28.697
  00:26:03 STEP 1651 H= 412:45 +CPU= 29.977
  00:26:34 STEP 1652 H= 413:00 +CPU= 30.325
  00:27:03 STEP 1653 H= 413:15 +CPU= 28.380
  00:27:34 STEP 1654 H= 413:30 +CPU= 29.978
  00:28:04 STEP 1655 H= 413:45 +CPU= 29.527
  00:28:40 STEP 1656 H= 414:00 +CPU= 35.238
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMGGh7zg+001656
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMSHh7zg+001656
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMUAh7zg+001656
  00:29:10 STEP 1657 H= 414:15 +CPU= 29.574
  00:29:40 STEP 1658 H= 414:30 +CPU= 29.446
  00:30:09 STEP 1659 H= 414:45 +CPU= 28.035
  00:30:39 STEP 1660 H= 415:00 +CPU= 30.047
  00:31:10 STEP 1661 H= 415:15 +CPU= 30.026
  00:31:41 STEP 1662 H= 415:30 +CPU= 29.884
  00:32:11 STEP 1663 H= 415:45 +CPU= 29.120
  00:32:41 STEP 1664 H= 416:00 +CPU= 29.597
  00:33:11 STEP 1665 H= 416:15 +CPU= 29.131
  00:33:39 STEP 1666 H= 416:30 +CPU= 27.242
  00:34:07 STEP 1667 H= 416:45 +CPU= 28.022
  00:34:41 STEP 1668 H= 417:00 +CPU= 32.659
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMGGh7zg+001668
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMSHh7zg+001668
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMUAh7zg+001668
  00:35:10 STEP 1669 H= 417:15 +CPU= 27.052
  00:35:38 STEP 1670 H= 417:30 +CPU= 27.708
  00:36:06 STEP 1671 H= 417:45 +CPU= 26.630
  00:36:35 STEP 1672 H= 418:00 +CPU= 28.075
  00:37:02 STEP 1673 H= 418:15 +CPU= 26.630
  00:37:30 STEP 1674 H= 418:30 +CPU= 27.064
  00:38:00 STEP 1675 H= 418:45 +CPU= 29.494
  00:38:36 STEP 1676 H= 419:00 +CPU= 34.636
  00:39:12 STEP 1677 H= 419:15 +CPU= 34.157
  00:39:49 STEP 1678 H= 419:30 +CPU= 33.743
  00:40:29 STEP 1679 H= 419:45 +CPU= 35.035
  00:41:21 STEP 1680 H= 420:00 +CPU= 46.571
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMGGh7zg+001680
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMSHh7zg+001680
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMUAh7zg+001680
  00:41:58 STEP 1681 H= 420:15 +CPU= 33.495
  00:42:38 STEP 1682 H= 420:30 +CPU= 35.388
  00:43:17 STEP 1683 H= 420:45 +CPU= 35.082
  00:43:54 STEP 1684 H= 421:00 +CPU= 34.101
  00:44:31 STEP 1685 H= 421:15 +CPU= 33.874
  00:45:05 STEP 1686 H= 421:30 +CPU= 32.113
  00:45:40 STEP 1687 H= 421:45 +CPU= 32.285
  00:46:15 STEP 1688 H= 422:00 +CPU= 33.427
  00:46:52 STEP 1689 H= 422:15 +CPU= 32.718
  00:47:27 STEP 1690 H= 422:30 +CPU= 32.232
  00:48:03 STEP 1691 H= 422:45 +CPU= 31.883
  00:48:45 STEP 1692 H= 423:00 +CPU= 38.121
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMGGh7zg+001692
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMSHh7zg+001692
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMUAh7zg+001692
  00:49:17 STEP 1693 H= 423:15 +CPU= 29.417
  00:49:50 STEP 1694 H= 423:30 +CPU= 29.586
  00:50:24 STEP 1695 H= 423:45 +CPU= 29.706
  00:50:56 STEP 1696 H= 424:00 +CPU= 29.785
  00:51:29 STEP 1697 H= 424:15 +CPU= 29.405
  00:52:02 STEP 1698 H= 424:30 +CPU= 29.999
  00:52:37 STEP 1699 H= 424:45 +CPU= 29.816
  00:53:10 STEP 1700 H= 425:00 +CPU= 29.753
  00:53:42 STEP 1701 H= 425:15 +CPU= 29.351
  00:54:16 STEP 1702 H= 425:30 +CPU= 29.741
  00:54:49 STEP 1703 H= 425:45 +CPU= 29.908
  00:55:32 STEP 1704 H= 426:00 +CPU= 36.892
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMGGh7zg+001704
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMSHh7zg+001704
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMUAh7zg+001704
  00:56:08 STEP 1705 H= 426:15 +CPU= 30.408
  00:56:45 STEP 1706 H= 426:30 +CPU= 30.340
  00:57:20 STEP 1707 H= 426:45 +CPU= 30.044
  00:57:55 STEP 1708 H= 427:00 +CPU= 30.813
  00:58:29 STEP 1709 H= 427:15 +CPU= 30.222
  00:59:02 STEP 1710 H= 427:30 +CPU= 29.627
  00:59:34 STEP 1711 H= 427:45 +CPU= 29.453
  01:00:08 STEP 1712 H= 428:00 +CPU= 30.475
  01:00:40 STEP 1713 H= 428:15 +CPU= 29.486
  01:01:13 STEP 1714 H= 428:30 +CPU= 29.537
  01:01:46 STEP 1715 H= 428:45 +CPU= 29.510
  01:02:24 STEP 1716 H= 429:00 +CPU= 36.396
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMGGh7zg+001716
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMSHh7zg+001716
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMUAh7zg+001716
  01:02:55 STEP 1717 H= 429:15 +CPU= 30.072
  01:03:31 STEP 1718 H= 429:30 +CPU= 32.696
  01:04:07 STEP 1719 H= 429:45 +CPU= 32.245
  01:04:41 STEP 1720 H= 430:00 +CPU= 31.605
  01:05:16 STEP 1721 H= 430:15 +CPU= 31.278
  01:05:52 STEP 1722 H= 430:30 +CPU= 33.156
  01:06:30 STEP 1723 H= 430:45 +CPU= 34.316
  01:07:04 STEP 1724 H= 431:00 +CPU= 31.384
  01:07:37 STEP 1725 H= 431:15 +CPU= 30.967
  01:08:08 STEP 1726 H= 431:30 +CPU= 29.987
  01:08:41 STEP 1727 H= 431:45 +CPU= 30.226
  01:09:25 STEP 1728 H= 432:00 +CPU= 40.894
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMGGh7zg+001728
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMSHh7zg+001728
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMUAh7zg+001728
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMGGh7zg+001632
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMSHh7zg+001632
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMUAh7zg+001632
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMGGh7zg+001644
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMSHh7zg+001644
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMUAh7zg+001644
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMGGh7zg+001656
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMSHh7zg+001656
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMUAh7zg+001656
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMGGh7zg+001668
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMSHh7zg+001668
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMUAh7zg+001668
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMGGh7zg+001680
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMSHh7zg+001680
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMUAh7zg+001680
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMGGh7zg+001692
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMSHh7zg+001692
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMUAh7zg+001692
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMGGh7zg+001704
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMSHh7zg+001704
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMUAh7zg+001704
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMGGh7zg+001716
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMSHh7zg+001716
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMUAh7zg+001716
Zipping up the intermediate file: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a3hr_2016092300_20_1018_12292854_0_r265557904_17.zip
Uploading the intermediate file: upload_file_17.zip
  01:09:59 STEP 1729 H= 432:15 +CPU= 30.673
  01:10:32 STEP 1730 H= 432:30 +CPU= 30.071
  01:11:05 STEP 1731 H= 432:45 +CPU= 30.151
  01:11:38 STEP 1732 H= 433:00 +CPU= 30.623
  01:12:10 STEP 1733 H= 433:15 +CPU= 29.784
  01:12:44 STEP 1734 H= 433:30 +CPU= 30.026
  01:13:16 STEP 1735 H= 433:45 +CPU= 30.251
  01:13:51 STEP 1736 H= 434:00 +CPU= 31.570
  01:14:24 STEP 1737 H= 434:15 +CPU= 30.438
  01:14:55 STEP 1738 H= 434:30 +CPU= 29.831
  01:15:28 STEP 1739 H= 434:45 +CPU= 29.947
  01:16:07 STEP 1740 H= 435:00 +CPU= 36.056
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMGGh7zg+001740
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMSHh7zg+001740
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMUAh7zg+001740
  01:16:39 STEP 1741 H= 435:15 +CPU= 29.353
  01:17:11 STEP 1742 H= 435:30 +CPU= 28.758
  01:17:42 STEP 1743 H= 435:45 +CPU= 28.449
  01:18:15 STEP 1744 H= 436:00 +CPU= 30.261
  01:18:46 STEP 1745 H= 436:15 +CPU= 28.947
  01:19:16 STEP 1746 H= 436:30 +CPU= 28.719
  01:19:46 STEP 1747 H= 436:45 +CPU= 28.889
  01:20:17 STEP 1748 H= 437:00 +CPU= 29.402
  01:20:48 STEP 1749 H= 437:15 +CPU= 28.884
  01:21:19 STEP 1750 H= 437:30 +CPU= 29.020
  01:21:50 STEP 1751 H= 437:45 +CPU= 28.730
  01:22:29 STEP 1752 H= 438:00 +CPU= 35.624
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMGGh7zg+001752
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMSHh7zg+001752
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMUAh7zg+001752
  01:22:58 STEP 1753 H= 438:15 +CPU= 28.325
  01:23:28 STEP 1754 H= 438:30 +CPU= 28.646
  01:23:58 STEP 1755 H= 438:45 +CPU= 28.757
  01:24:27 STEP 1756 H= 439:00 +CPU= 28.967
  01:24:57 STEP 1757 H= 439:15 +CPU= 28.285
  01:25:27 STEP 1758 H= 439:30 +CPU= 28.445
  01:25:57 STEP 1759 H= 439:45 +CPU= 28.640
  01:26:27 STEP 1760 H= 440:00 +CPU= 28.685
  01:26:56 STEP 1761 H= 440:15 +CPU= 28.485
  01:27:25 STEP 1762 H= 440:30 +CPU= 28.313
  01:27:55 STEP 1763 H= 440:45 +CPU= 28.594
  01:28:32 STEP 1764 H= 441:00 +CPU= 35.484
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMGGh7zg+001764
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMSHh7zg+001764
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMUAh7zg+001764
  01:29:02 STEP 1765 H= 441:15 +CPU= 28.661
  01:29:31 STEP 1766 H= 441:30 +CPU= 28.602
  01:30:01 STEP 1767 H= 441:45 +CPU= 28.251
  01:30:31 STEP 1768 H= 442:00 +CPU= 28.840
  01:31:02 STEP 1769 H= 442:15 +CPU= 29.162
  01:31:35 STEP 1770 H= 442:30 +CPU= 29.382
  01:32:08 STEP 1771 H= 442:45 +CPU= 29.652
  01:32:41 STEP 1772 H= 443:00 +CPU= 29.823
  01:33:14 STEP 1773 H= 443:15 +CPU= 29.383
  01:33:46 STEP 1774 H= 443:30 +CPU= 29.401
  01:34:18 STEP 1775 H= 443:45 +CPU= 29.129
  01:35:02 STEP 1776 H= 444:00 +CPU= 39.365
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMGGh7zg+001776
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMSHh7zg+001776
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMUAh7zg+001776
  01:35:34 STEP 1777 H= 444:15 +CPU= 29.196
  01:36:05 STEP 1778 H= 444:30 +CPU= 29.033
  01:36:37 STEP 1779 H= 444:45 +CPU= 29.728
  01:37:08 STEP 1780 H= 445:00 +CPU= 29.947
  01:37:38 STEP 1781 H= 445:15 +CPU= 29.366
  01:38:10 STEP 1782 H= 445:30 +CPU= 29.606
  01:38:41 STEP 1783 H= 445:45 +CPU= 29.551
  01:39:14 STEP 1784 H= 446:00 +CPU= 30.138
  01:39:46 STEP 1785 H= 446:15 +CPU= 29.680
  01:40:18 STEP 1786 H= 446:30 +CPU= 29.622
  01:40:50 STEP 1787 H= 446:45 +CPU= 29.264
  01:41:32 STEP 1788 H= 447:00 +CPU= 37.161
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMGGh7zg+001788
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMSHh7zg+001788
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMUAh7zg+001788
  01:42:04 STEP 1789 H= 447:15 +CPU= 29.194
  01:42:37 STEP 1790 H= 447:30 +CPU= 29.346
  01:43:10 STEP 1791 H= 447:45 +CPU= 30.004
  01:43:42 STEP 1792 H= 448:00 +CPU= 29.563
  01:44:14 STEP 1793 H= 448:15 +CPU= 29.161
  01:44:45 STEP 1794 H= 448:30 +CPU= 28.262
  01:45:12 STEP 1795 H= 448:45 +CPU= 26.419
  01:45:39 STEP 1796 H= 449:00 +CPU= 26.664
  01:46:05 STEP 1797 H= 449:15 +CPU= 25.421
  01:46:31 STEP 1798 H= 449:30 +CPU= 25.631
  01:46:58 STEP 1799 H= 449:45 +CPU= 26.386
  01:47:31 STEP 1800 H= 450:00 +CPU= 31.649
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMGGh7zg+001800
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMSHh7zg+001800
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMUAh7zg+001800
  01:47:57 STEP 1801 H= 450:15 +CPU= 26.119
  01:48:24 STEP 1802 H= 450:30 +CPU= 26.130
  01:48:51 STEP 1803 H= 450:45 +CPU= 26.126
  01:49:18 STEP 1804 H= 451:00 +CPU= 26.621
  01:49:45 STEP 1805 H= 451:15 +CPU= 26.169
  01:50:11 STEP 1806 H= 451:30 +CPU= 26.131
  01:50:38 STEP 1807 H= 451:45 +CPU= 26.060
  01:51:05 STEP 1808 H= 452:00 +CPU= 26.841
  01:51:32 STEP 1809 H= 452:15 +CPU= 26.256
  01:51:58 STEP 1810 H= 452:30 +CPU= 26.109
  01:52:25 STEP 1811 H= 452:45 +CPU= 26.153
  01:52:58 STEP 1812 H= 453:00 +CPU= 32.120
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMGGh7zg+001812
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMSHh7zg+001812
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMUAh7zg+001812
  01:53:24 STEP 1813 H= 453:15 +CPU= 25.798
  01:53:51 STEP 1814 H= 453:30 +CPU= 26.363
  01:54:18 STEP 1815 H= 453:45 +CPU= 26.140
  01:54:45 STEP 1816 H= 454:00 +CPU= 27.159
  01:55:12 STEP 1817 H= 454:15 +CPU= 26.319
  01:55:39 STEP 1818 H= 454:30 +CPU= 26.168
  01:56:05 STEP 1819 H= 454:45 +CPU= 26.026
  01:56:32 STEP 1820 H= 455:00 +CPU= 26.719
  01:56:58 STEP 1821 H= 455:15 +CPU= 26.257
  01:57:25 STEP 1822 H= 455:30 +CPU= 26.535
  01:57:53 STEP 1823 H= 455:45 +CPU= 26.742
  01:58:29 STEP 1824 H= 456:00 +CPU= 36.349
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMGGh7zg+001824
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMSHh7zg+001824
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMUAh7zg+001824
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMGGh7zg+001728
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMSHh7zg+001728
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMUAh7zg+001728
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMGGh7zg+001740
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMSHh7zg+001740
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMUAh7zg+001740
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMGGh7zg+001752
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMSHh7zg+001752
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMUAh7zg+001752
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMGGh7zg+001764
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMSHh7zg+001764
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMUAh7zg+001764
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMGGh7zg+001776
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMSHh7zg+001776
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMUAh7zg+001776
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMGGh7zg+001788
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMSHh7zg+001788
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMUAh7zg+001788
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMGGh7zg+001800
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMSHh7zg+001800
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMUAh7zg+001800
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMGGh7zg+001812
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMSHh7zg+001812
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMUAh7zg+001812
Zipping up the intermediate file: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a3hr_2016092300_20_1018_12292854_0_r265557904_18.zip
Uploading the intermediate file: upload_file_18.zip
  01:58:57 STEP 1825 H= 456:15 +CPU= 26.572
  01:59:23 STEP 1826 H= 456:30 +CPU= 25.803
  01:59:49 STEP 1827 H= 456:45 +CPU= 26.094
  02:00:17 STEP 1828 H= 457:00 +CPU= 26.392
  02:00:43 STEP 1829 H= 457:15 +CPU= 25.559
  02:01:09 STEP 1830 H= 457:30 +CPU= 25.511
  02:01:35 STEP 1831 H= 457:45 +CPU= 25.690
  02:02:03 STEP 1832 H= 458:00 +CPU= 28.386
  02:02:40 STEP 1833 H= 458:15 +CPU= 35.579
  02:03:19 STEP 1834 H= 458:30 +CPU= 36.983
  02:03:56 STEP 1835 H= 458:45 +CPU= 34.676
  02:04:47 STEP 1836 H= 459:00 +CPU= 45.914
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMGGh7zg+001836
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMSHh7zg+001836
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMUAh7zg+001836
  02:05:28 STEP 1837 H= 459:15 +CPU= 36.963
  02:06:07 STEP 1838 H= 459:30 +CPU= 35.005
  02:06:48 STEP 1839 H= 459:45 +CPU= 36.975
  02:07:28 STEP 1840 H= 460:00 +CPU= 35.959
  02:08:05 STEP 1841 H= 460:15 +CPU= 34.604
  02:08:45 STEP 1842 H= 460:30 +CPU= 37.017
  02:09:21 STEP 1843 H= 460:45 +CPU= 33.022
  02:10:01 STEP 1844 H= 461:00 +CPU= 37.069
  02:10:41 STEP 1845 H= 461:15 +CPU= 36.507
  02:11:21 STEP 1846 H= 461:30 +CPU= 35.280
  02:12:00 STEP 1847 H= 461:45 +CPU= 35.214
  02:12:46 STEP 1848 H= 462:00 +CPU= 42.375
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMGGh7zg+001848
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMSHh7zg+001848
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMUAh7zg+001848
  02:13:22 STEP 1849 H= 462:15 +CPU= 33.303
Quit request received from BOINC client, ending the child process
>>> Printing last 70 lines from file: NODE.001_01

         AVE   0.131153930801736E-02 0.100000000000000E-07 0.109192158687463E-01
 GPNORM  LIQUID WATER        AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.179935394459869E-05 0.000000000000000E+00 0.632611632364872E-03
 GPNORM  ICE WATER           AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.459879531420580E-06 0.000000000000000E+00 0.175975278139058E-03
 GPNORM SNOW                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.107085507086369E-05 0.000000000000000E+00 0.388455512502364E-03
 GPNORM RAIN                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.382665922374266E-07 0.000000000000000E+00 0.119705984965518E-03
 GPNORM  CLOUD FRACTION      AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.295121144617790E-01 0.000000000000000E+00 0.100000000000000E+01
 GPNORM CLW                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.180167176465040E-05 0.000000000000000E+00 0.634106134334995E-03
 GPNORM CIW                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.460248059826141E-06 0.000000000000000E+00 0.178826081352437E-03
 GPNORM CC                   AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.295076509995826E-01 0.000000000000000E+00 0.100000000000000E+01
 GPNORM RFL                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.176676851073264E-06 0.000000000000000E+00 0.560944076621080E-03
 GPNORM SFL                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.119699074414510E-05 0.000000000000000E+00 0.474550997186566E-03
 GPNORM CRFL                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.874748220817237E-07 0.000000000000000E+00 0.262472244190995E-03
 GPNORM CSFL                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.254975650085505E-08 0.000000000000000E+00 0.978206283129612E-04
 GPNORM PFRC                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.678374980817990E-01 0.000000000000000E+00 0.100000000000000E+01
 NSTEP =  1848 STEPO   A00000000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1848 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001848 MODE=w
 IO-STREAM CLOSED - ICMGGh7zg+001848
 MPI-TASK:   1 -    304963988 BYTES IN      6 RECORDS TRANSFERRED IN    0.0003 SECONDS******** Mbytes/s, TOTAL TIME=    0.0126( 2.4%)
 NSTEP =  1848 STEPO   0AA000000
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMSHh7zg+001848 MODE=w
 IO-STREAM CLOSED - ./ICMSHh7zg+001848
 MPI-TASK:   1 -    707399250 BYTES IN    202 RECORDS TRANSFERRED IN    0.0079 SECONDS 89544.2 Mbytes/s, TOTAL TIME=    0.4445( 1.8%)
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMUAh7zg+001848 MODE=w
 IO-STREAM CLOSED - ./ICMUAh7zg+001848
 MPI-TASK:   1 -   4879423808 BYTES IN    125 RECORDS TRANSFERRED IN    0.0043 SECONDS******** Mbytes/s, TOTAL TIME=    0.2393( 1.8%)
 STEPO_FPOS WAS : B00000F00E
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMUAh7zg+001848 MODE=a
 IO-STREAM CLOSED - ./ICMUAh7zg+001848
 MPI-TASK:   1 -           10 BYTES IN      2 RECORDS TRANSFERRED IN    0.0001 SECONDS     0.1 Mbytes/s, TOTAL TIME=    0.0052( 1.9%)
 STEPO_FPOS WAS : V00000000Y
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMUAh7zg+001848 MODE=a
 IO-STREAM CLOSED - ./ICMUAh7zg+001848
 MPI-TASK:   1 -    296384259 BYTES IN      4 RECORDS TRANSFERRED IN    0.0003 SECONDS987947.5 Mbytes/s, TOTAL TIME=    0.0081( 3.7%)
 STEPO_FPOS WAS : T00000000M
 NSTEP =  1848 STEPO   0AAA00AAA
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1848 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001848 MODE=a
 IO-STREAM CLOSED - ICMGGh7zg+001848
 MPI-TASK:   1 -    304963988 BYTES IN      7 RECORDS TRANSFERRED IN    0.0004 SECONDS762410.0 Mbytes/s, TOTAL TIME=    0.0126( 3.2%)
 02:12:46 STEP 1848 H= 462:00 +CPU= 42.375
 NSTEP =  1849 STEPO   0AAA00AAA
 02:13:22 STEP 1849 H= 462:15 +CPU= 33.303
ISTEP=    1850 RSOLINC= 1361.4482 RI0= 1365.0000 ZI0= 1361.4482 RIP0= 1361.4482
ISTEP=    1850 ZI0= 1361.4482 ZSEASON= 1.000000000 REA=149597870000.00 RDEASO=149597870000.00
 NSTEP =  1850 STEPO   0AAA00AAA
------------------------------------------------
CNT0 not found; string returned was: 'STEPO'
>>> Printing last 8 lines from file: ifs.stat

 02:12:01 0AA000000 STEPO     1848     0.033    0.033    0.033    200:05    212:18 0.30060541327814E-05       2GB       0MB
 02:12:03 FULLPOS-B DYNFPOS   1848     1.995    1.995    2.020    200:07    212:20 0.30060541327814E-05       2GB       0MB
 02:12:07 FULLPOS-V DYNFPOS   1848     3.967    3.967    4.222    200:11    212:24 0.30060541327814E-05       2GB       0MB
 02:12:07 FULLPOS-T DYNFPOS   1848     0.589    0.589    0.669    200:11    212:25 0.30060541327814E-05       2GB       0MB
 02:12:08 0AAA00AAA STEPO     1848     0.608    0.608    0.612    200:12    212:25 0.30060541327814E-05       2GB       0MB
 02:12:46 0AAA00AAA STEPO     1849    35.015   35.015   37.754    200:47    213:03 0.30060541327814E-05       2GB       0MB
 02:13:22 0AAA00AAA STEPO     1850    33.303   33.303   36.525    201:20    213:39 0.30060541327814E-05       2GB       0MB
------------------------------------------------
>>> Printing last 11 lines from file: rcf

 CSTEP   = "    1824",
 CTIME   = "00190000      ",
 NSTEPLPP        =         201,         202,          49,      228226,      228227, 2*0,        1812, 2*0,
 IPRGPNSRES      =           1,
 IPRGPEWRES      =           1,
 IPRTRWRES       =           1,
 IPRTRVRES       =           1,
 GMASS0  =  0.000000000000000E+000,
 GMASSI  =  0.000000000000000E+000
 /
------------------------------------------------
>>> Printing last 8 lines from file: /mydisks/b/boinc-client/slots/23/progress_file_12292854.xml

<running_values>
  <last_cpu_time>56137.300000</last_cpu_time>
  <upload_file_number>19</upload_file_number>
  <last_iter>1850</last_iter>
  <last_upload>1641600</last_upload>
  <model_completed>0</model_completed>
</running_values>
------------------------------------------------
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.13_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a3hr
(argv4) batchid: 1018
(argv5) wuid: 12292854
(argv6) fclen: 20
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /mydisks/b/boinc-client/slots/23
Project directory is: /mydisks/b/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.13
Location of temp folder: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854
..mkdir for temp folder for results failed
Copying: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip to: /mydisks/b/boinc-client/slots/23/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /mydisks/b/boinc-client/slots/23/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_884289748ebee2f5f3a040926af19fa5 to: /mydisks/b/boinc-client/slots/23/oifs_43r3_bl_a3hr_2016092300_20_1018_12292854.zip
Unzipping the namelist zip file: /mydisks/b/boinc-client/slots/23/oifs_43r3_bl_a3hr_2016092300_20_1018_12292854.zip
ic_ancil_file: ic_ancil_12292854
ifsdata_file: ifsdata_12292854
climate_data_file: clim_data_12292854
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /mydisks/b/boinc-client/slots/23/ic_ancil_12292854.zip
Unzipping the IC ancils zip file: /mydisks/b/boinc-client/slots/23/ic_ancil_12292854.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_bb3992d84bc65e2134aeffa1ff04bd89 to: /mydisks/b/boinc-client/slots/23/ifsdata/ifsdata_12292854.zip
Unzipping the ifsdata_zip file: /mydisks/b/boinc-client/slots/23/ifsdata/ifsdata_12292854.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /mydisks/b/boinc-client/slots/23/159l_2/clim_data_12292854.zip
Unzipping the climate data zip file: /mydisks/b/boinc-client/slots/23/159l_2/clim_data_12292854.zip
Checking for progress XML file: /mydisks/b/boinc-client/slots/23/progress_file_12292854.xml
Opened progress file ok : /mydisks/b/boinc-client/slots/23/progress_file_12292854.xml
-- Model is restarting --
Adjusting last_iter, 1850, to previous model restart step.
Creating progress file: /mydisks/b/boinc-client/slots/23/progress_file_12292854.xml
last_cpu_time: 56137
upload_file_number: 19
last_iter: 1825
last_upload: 1641600
model_completed: 0
total_length_of_simulation: 1728000
result_base_name: oifs_43r3_bl_a3hr_2016092300_20_1018_12292854_0_r265557904
The child process has been launched with process id: 469051
Executing the command: /mydisks/b/boinc-client/slots/23/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:centurion:1:1:469051:469051] [20240615:022350:1718443430:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2091] fp = 0x314f440
[EC_DRHOOK:centurion:1:1:469051:469051] [20240615:022350:1718443430:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:centurion:1:1:469051:469051] [20240615:022350:1718443430:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:centurion:1:1:469051:469051] [20240615:022350:1718443430:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:centurion:1:1:469051:469051] [20240615:022350:1718443430:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:centurion:1:1:469051:469051] [20240615:022350:1718443430:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:centurion:1:1:469051:469051] [20240615:022350:1718443430:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:centurion:1:1:469051:469051] [20240615:022350:1718443430:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:centurion:1:1:469051:469051] [20240615:022350:1718443430:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:centurion:1:1:469051:469051] [20240615:022350:1718443430:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:centurion:1:1:469051:469051] [20240615:022350:1718443430:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:centurion:1:1:469051:469051] [20240615:022350:1718443430:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:centurion:1:1:469051:469051] [20240615:022350:1718443430:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:centurion:1:1:469051:469051] [20240615:022350:1718443430:0.000] [ignore_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:centurion:1:1:469051:469051] [20240615:022350:1718443430:0.000] [restore_default_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:centurion:1:1:469051:469051] [20240615:022350:1718443430:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:centurion:1:1:469051:469051] [20240615:022350:1718443430:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:centurion:1:1:469051:469051] [20240615:022350:1718443430:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:centurion:1:1:469051:469051] [20240615:022350:1718443430:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x820cf0 (old at (nil))
[EC_DRHOOK:centurion:1:1:469051:469051] [20240615:022350:1718443430:0.000] [trapfpe_treatment@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:centurion:1:1:469051:469051] [20240615:022350:1718443430:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:centurion:1:1:469051:469051] [20240615:022350:1718443430:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:centurion:1:1:469051:469051] [20240615:022350:1718443430:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x820cf0 (old at (nil))
[EC_DRHOOK:centurion:1:1:469051:469051] [20240615:022350:1718443430:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:centurion:1:1:469051:469051] [20240615:022350:1718443430:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:centurion:1:1:469051:469051] [20240615:022350:1718443430:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:centurion:1:1:469051:469051] [20240615:022350:1718443430:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x820cf0 (old at (nil))
[EC_DRHOOK:centurion:1:1:469051:469051] [20240615:022350:1718443430:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x820cf0 (old at (nil))
[EC_DRHOOK:centurion:1:1:469051:469051] [20240615:022350:1718443430:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:centurion:1:1:469051:469051] [20240615:022350:1718443430:0.000] [catch_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  02:24:30 STEP 1824 H= 456:00 +CPU= 36.094
  02:25:08 STEP 1825 H= 456:15 +CPU= 35.848
  02:25:48 STEP 1826 H= 456:30 +CPU= 37.026
  02:26:29 STEP 1827 H= 456:45 +CPU= 38.780
  02:27:10 STEP 1828 H= 457:00 +CPU= 38.975
  02:27:47 STEP 1829 H= 457:15 +CPU= 35.631
  02:28:26 STEP 1830 H= 457:30 +CPU= 37.006
  02:29:06 STEP 1831 H= 457:45 +CPU= 37.422
  02:29:44 STEP 1832 H= 458:00 +CPU= 35.841
  02:30:24 STEP 1833 H= 458:15 +CPU= 36.421
  02:31:02 STEP 1834 H= 458:30 +CPU= 34.520
  02:31:41 STEP 1835 H= 458:45 +CPU= 34.857
  02:32:32 STEP 1836 H= 459:00 +CPU= 45.672
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMGGh7zg+001836
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMSHh7zg+001836
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMUAh7zg+001836
  02:33:10 STEP 1837 H= 459:15 +CPU= 34.522
  02:33:48 STEP 1838 H= 459:30 +CPU= 34.064
  02:34:25 STEP 1839 H= 459:45 +CPU= 34.716
  02:35:01 STEP 1840 H= 460:00 +CPU= 33.441
  02:35:35 STEP 1841 H= 460:15 +CPU= 32.102
  02:36:11 STEP 1842 H= 460:30 +CPU= 32.976
  02:36:46 STEP 1843 H= 460:45 +CPU= 31.689
  02:37:22 STEP 1844 H= 461:00 +CPU= 34.160
  02:37:55 STEP 1845 H= 461:15 +CPU= 32.201
  02:38:29 STEP 1846 H= 461:30 +CPU= 32.312
  02:39:02 STEP 1847 H= 461:45 +CPU= 31.618
  02:39:45 STEP 1848 H= 462:00 +CPU= 40.285
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMGGh7zg+001848
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMSHh7zg+001848
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMUAh7zg+001848
  02:40:20 STEP 1849 H= 462:15 +CPU= 33.397
  02:40:57 STEP 1850 H= 462:30 +CPU= 33.782
  02:41:34 STEP 1851 H= 462:45 +CPU= 34.512
  02:42:10 STEP 1852 H= 463:00 +CPU= 34.057
  02:42:51 STEP 1853 H= 463:15 +CPU= 36.832
  02:43:30 STEP 1854 H= 463:30 +CPU= 35.245
  02:44:11 STEP 1855 H= 463:45 +CPU= 36.924
  02:44:51 STEP 1856 H= 464:00 +CPU= 37.756
  02:45:32 STEP 1857 H= 464:15 +CPU= 37.526
  02:46:12 STEP 1858 H= 464:30 +CPU= 36.740
  02:46:50 STEP 1859 H= 464:45 +CPU= 35.185
  02:47:37 STEP 1860 H= 465:00 +CPU= 44.226
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMGGh7zg+001860
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMSHh7zg+001860
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMUAh7zg+001860
  02:48:13 STEP 1861 H= 465:15 +CPU= 34.197
  02:48:51 STEP 1862 H= 465:30 +CPU= 34.416
  02:49:29 STEP 1863 H= 465:45 +CPU= 34.796
  02:50:07 STEP 1864 H= 466:00 +CPU= 34.937
  02:50:42 STEP 1865 H= 466:15 +CPU= 32.408
  02:51:15 STEP 1866 H= 466:30 +CPU= 31.783
  02:51:48 STEP 1867 H= 466:45 +CPU= 31.154
  02:52:15 STEP 1868 H= 467:00 +CPU= 26.333
  02:52:37 STEP 1869 H= 467:15 +CPU= 22.370
  02:52:58 STEP 1870 H= 467:30 +CPU= 20.890
  02:53:24 STEP 1871 H= 467:45 +CPU= 24.660
  02:54:02 STEP 1872 H= 468:00 +CPU= 36.814
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMGGh7zg+001872
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMSHh7zg+001872
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMUAh7zg+001872
  02:54:30 STEP 1873 H= 468:15 +CPU= 26.395
  02:54:58 STEP 1874 H= 468:30 +CPU= 26.224
  02:55:26 STEP 1875 H= 468:45 +CPU= 26.292
  02:55:54 STEP 1876 H= 469:00 +CPU= 26.720
  02:56:22 STEP 1877 H= 469:15 +CPU= 25.944
  02:56:48 STEP 1878 H= 469:30 +CPU= 25.604
  02:57:15 STEP 1879 H= 469:45 +CPU= 25.498
  02:57:43 STEP 1880 H= 470:00 +CPU= 26.852
  02:58:10 STEP 1881 H= 470:15 +CPU= 25.524
  02:58:37 STEP 1882 H= 470:30 +CPU= 25.433
  02:59:04 STEP 1883 H= 470:45 +CPU= 25.737
  02:59:37 STEP 1884 H= 471:00 +CPU= 31.557
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMGGh7zg+001884
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMSHh7zg+001884
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMUAh7zg+001884
  03:00:04 STEP 1885 H= 471:15 +CPU= 26.016
  03:00:31 STEP 1886 H= 471:30 +CPU= 25.646
  03:00:57 STEP 1887 H= 471:45 +CPU= 25.345
  03:01:24 STEP 1888 H= 472:00 +CPU= 25.584
  03:01:51 STEP 1889 H= 472:15 +CPU= 24.934
  03:02:17 STEP 1890 H= 472:30 +CPU= 25.157
  03:02:40 STEP 1891 H= 472:45 +CPU= 22.368
  03:03:01 STEP 1892 H= 473:00 +CPU= 19.954
  03:03:20 STEP 1893 H= 473:15 +CPU= 19.517
  03:03:40 STEP 1894 H= 473:30 +CPU= 19.188
  03:04:00 STEP 1895 H= 473:45 +CPU= 20.006
  03:04:35 STEP 1896 H= 474:00 +CPU= 32.564
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMGGh7zg+001896
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMSHh7zg+001896
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMUAh7zg+001896
  03:05:05 STEP 1897 H= 474:15 +CPU= 27.608
  03:05:35 STEP 1898 H= 474:30 +CPU= 27.202
  03:06:03 STEP 1899 H= 474:45 +CPU= 26.448
  03:06:32 STEP 1900 H= 475:00 +CPU= 27.299
  03:07:01 STEP 1901 H= 475:15 +CPU= 26.462
  03:07:28 STEP 1902 H= 475:30 +CPU= 26.366
  03:07:56 STEP 1903 H= 475:45 +CPU= 26.277
  03:08:25 STEP 1904 H= 476:00 +CPU= 27.081
  03:08:53 STEP 1905 H= 476:15 +CPU= 26.306
  03:09:21 STEP 1906 H= 476:30 +CPU= 26.070
  03:09:49 STEP 1907 H= 476:45 +CPU= 26.543
  03:10:25 STEP 1908 H= 477:00 +CPU= 32.813
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMGGh7zg+001908
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMSHh7zg+001908
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMUAh7zg+001908
  03:10:53 STEP 1909 H= 477:15 +CPU= 26.253
  03:11:21 STEP 1910 H= 477:30 +CPU= 26.330
  03:11:48 STEP 1911 H= 477:45 +CPU= 25.923
  03:12:15 STEP 1912 H= 478:00 +CPU= 26.171
  03:12:43 STEP 1913 H= 478:15 +CPU= 26.369
  03:13:11 STEP 1914 H= 478:30 +CPU= 26.223
  03:13:38 STEP 1915 H= 478:45 +CPU= 25.964
  03:14:06 STEP 1916 H= 479:00 +CPU= 26.160
  03:14:28 STEP 1917 H= 479:15 +CPU= 22.099
  03:14:50 STEP 1918 H= 479:30 +CPU= 21.384
  03:15:10 STEP 1919 H= 479:45 +CPU= 20.722
  03:15:38 STEP 1920 H= 480:00 +CPU= 27.337
..The child process terminated with status: 0
>>> Printing last 70 lines from file: NODE.001_01

 
DDH------------------------------------------------------------------------
I                                                                         I
I TIME =  480.000H DDH * Diagnostic Var.,     AREA                GLOBAL  I
I                                                                         I
I-------------------------------------------------------------------------I
I   VARIABLE   I      Initial value         I        Current value        I
I-------------------------------------------------------------------------I
I  OMEGA (P/S) I     -0.8840855458E-21      I      -0.4620833489E-05      I
ICloud fractionI      0.0000000000E+00      I       0.4875318895E-01      I
I  Relat. Hum. I      0.4979081881E+00      I       0.4886005270E+00      I
I  PASS 01     I      0.0000000000E+00      I       0.8868570789E+03      I
DDH------------------------------------------------------------------------
  
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1920 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001920 MODE=a
 IO-STREAM CLOSED - ICMGGh7zg+001920
 MPI-TASK:   1 -    300101860 BYTES IN      7 RECORDS TRANSFERRED IN    0.0002 SECONDS******** Mbytes/s, TOTAL TIME=    0.0094( 2.1%)
 03:15:38 STEP 1920 H= 480:00 +CPU= 27.337
 END CNT3
 NSTEP =  1920 CNT0   000000000  20.214  20.214  20.480   0.987               0               0               0               0               0
IO-STREAM STATISTICS, TOTAL NO OF REC/BYTES READ           46/  1069125708 WRITTEN         2860/************
 -TASK-OPENED-OPEN-RECS IN -KBYTE IN    -RECS OUT-KBYTE OUT-WALL   -WALL IN-WALL
  OU-TOT IN -TOT OUT
   1     53    0       46      1044068     2860 ************     5.6     1.5     4.0     2.4     9.3
===-=== START OF TIMING STATISTICS ===-===
 
STATS FOR ALL TASKS
 NUM ROUTINE                                     CALLS  MEAN(ms)   MAX(ms)   FRAC(%)  UNBAL(%)
   0 CNT0     - COMPLETE EXECUTION                   1 3108487.5 3108487.5    100.00      0.00
   1 CNT4     - FORWARD INTEGRATION                  1 3104457.1 3104457.1     99.87      0.00
   8 SCAN2M - GRID-POINT DYNAMICS                  105    4703.3    4703.3     15.89      0.00
   9 SPCM     - SPECTRAL COMP.                      96     873.9     873.9      2.70      0.00
  10 SCAN2M - PHYSICS                               97   15786.6   15786.6     49.26      0.00
  11 IOPACK   - OUTPUT P.P. RESULTS                  8     712.5     712.5      0.18      0.00
  12 SPNORM   - SPECTRAL NORM COMP.                  5      52.8      52.8      0.01      0.00
  17 GRIDFPOS IN CNT4                                8      32.8      32.8      0.01      0.00
  27 RESTART FILES - WRITING                         2    2850.0    2850.0      0.18      0.00
  28 RESTART FILES - READING                         1    3433.4    3433.4      0.11      0.00
  29 SU4FPOS IN CNT4                                 8       0.1       0.1      0.00      0.00
  30 DYNFPOS IN CNT4                                 8    6154.7    6154.7      1.58      0.00
  31 POSDDH IN STEPO                                 1       0.6       0.6      0.00      0.00
  37 CPGLAG   - SL COMPUTATIONS                     97    6368.4    6368.4     19.87      0.00
  39 SU0YOMB                                         1     385.1     385.1      0.01      0.00
  51 SCAN2M   - SL COMM. PART 1                     97      43.8      43.8      0.14      0.00
  54 SPCM     - M TO S/S TO M TRANSP.               96     346.3     346.3      1.07      0.00
  55 SPCIMPF  - S TO M/M TO S TRANSP.               96      78.3      78.3      0.24      0.00
  56 SPNORM   - SPECTRAL NORM COMM.                  5       0.4       0.4      0.00      0.00
 102 LTINV_CTL   - INVERSE LEGENDRE TRANSFORM      212     542.3     542.3      3.70      0.00
 103 LTDIR_CTL   - DIRECT LEGENDRE TRANSFORM       200     309.8     309.8      1.99      0.00
 106 FTDIR_CTL   - DIRECT FOURIER TRANSFORM        200      98.9      98.9      0.64      0.00
 107 FTINV_CTL   - INVERSE FOURIER TRANSFORM       212     165.4     165.4      1.13      0.00
 140 SULEG       - COMP. OF LEGENDRE POL.            1      40.3      40.3      0.00      0.00
 152 LTINV_CTL   - M TO L TRANSPOSITION            212      64.6      64.6      0.44      0.00
 153 LTDIR_CTL   - L TO M TRANSPOSITION            200      33.1      33.1      0.21      0.00
 157 FTINV_CTL   - L TO G TRANSPOSITION            212      88.7      88.7      0.61      0.00
 158 FTDIR_CTL   - G TO L TRANSPOSITION            200      68.2      68.2      0.44      0.00
 400 GSTATS                                      12215       0.0       0.0      0.00      0.00
 401 GSTATS HOOK                                 11311       0.0       0.0      0.00      0.00
TOTAL MEASURED IMBALANCE =       0.0 SECONDS,  0.0 PERCENT
TOTAL WALLCLOCK TIME     3108.5 CPU TIME    2924.6 VECTOR TIME     2924.6


===-=== END   OF TIMING STATISTICS ===-===
FORECAST DAYS PER DAY      556.6
  *** END CNT0 *** 
------------------------------------------------
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMGGh7zg+001920
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMSHh7zg+001920
Moving to projects directory: /mydisks/b/boinc-client/slots/23/ICMUAh7zg+001920
Adding to the zip: /mydisks/b/boinc-client/slots/23/NODE.001_01
Adding to the zip: /mydisks/b/boinc-client/slots/23/ifs.stat
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMGGh7zg+001824
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMSHh7zg+001824
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMUAh7zg+001824
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMGGh7zg+001836
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMSHh7zg+001836
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMUAh7zg+001836
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMGGh7zg+001848
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMSHh7zg+001848
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMUAh7zg+001848
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMGGh7zg+001860
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMSHh7zg+001860
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMUAh7zg+001860
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMGGh7zg+001872
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMSHh7zg+001872
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMUAh7zg+001872
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMGGh7zg+001884
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMSHh7zg+001884
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMUAh7zg+001884
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMGGh7zg+001896
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMSHh7zg+001896
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMUAh7zg+001896
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMGGh7zg+001908
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMSHh7zg+001908
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMUAh7zg+001908
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMGGh7zg+001920
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMSHh7zg+001920
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292854/ICMUAh7zg+001920
Zipping up the final file: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a3hr_2016092300_20_1018_12292854_0_r265557904_19.zip
Uploading the final file: upload_file_19.zip
Uploading trickle at timestep: 1727100
03:19:07 (467513): called boinc_finish(0)

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
15 Jun 2024 11:01:50 1517679 22446464 oifs_43r3_bl_a3hr_2016092300_20_1018_12292854_0 1,727,100 59,061 0.0342


©2024 climateprediction.net