climateprediction.net home page
Task 22282905

Task 22282905

Name oifs_43r3_ps_0416_2001050100_123_970_12187060_1
Workunit 12187060
Created 24 Dec 2022, 1:52:16 UTC
Sent 6 Jan 2023, 21:24:33 UTC
Report deadline 5 Feb 2023, 21:24:33 UTC
Received 7 Jan 2023, 22:30:20 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 1 (0x00000001) Unknown error code
Computer ID 1537854
Run time 17 hours 35 min 41 sec
CPU time 16 hours 40 min 47 sec
Validate state Invalid
Credit 5,685.12
Device peak FLOPS 3.60 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,419.02 MB
Peak swap size 4,977.96 MB
Peak disk usage 2,668.75 MB
Stderr
<core_client_version>7.20.5</core_client_version>
<![CDATA[
<message>
process exited with code 1 (0x1, -255)</message>
<stderr_txt>
PU= 44.412
  14:30:07 STEP  557 H= 557:00 +CPU= 43.174
  14:31:19 STEP  558 H= 558:00 +CPU= 69.221
  14:32:04 STEP  559 H= 559:00 +CPU= 43.132
  14:32:51 STEP  560 H= 560:00 +CPU= 44.345
  14:34:02 STEP  561 H= 561:00 +CPU= 67.589
  14:34:49 STEP  562 H= 562:00 +CPU= 44.345
  14:35:34 STEP  563 H= 563:00 +CPU= 43.106
  14:36:53 STEP  564 H= 564:00 +CPU= 75.259
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+000564
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+000564
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+000564
  14:37:38 STEP  565 H= 565:00 +CPU= 43.259
  14:38:25 STEP  566 H= 566:00 +CPU= 44.214
  14:39:35 STEP  567 H= 567:00 +CPU= 67.787
  14:40:22 STEP  568 H= 568:00 +CPU= 44.069
  14:41:07 STEP  569 H= 569:00 +CPU= 43.122
  14:42:20 STEP  570 H= 570:00 +CPU= 69.266
  14:43:05 STEP  571 H= 571:00 +CPU= 43.040
  14:43:51 STEP  572 H= 572:00 +CPU= 44.353
  14:45:02 STEP  573 H= 573:00 +CPU= 67.791
  14:45:48 STEP  574 H= 574:00 +CPU= 44.234
  14:46:35 STEP  575 H= 575:00 +CPU= 44.347
  14:48:00 STEP  576 H= 576:00 +CPU= 79.788
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+000576
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+000576
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+000576
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMGGhq0f+000552
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMSHhq0f+000552
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMUAhq0f+000552
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMGGhq0f+000564
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMSHhq0f+000564
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMUAhq0f+000564
Zipping up the intermediate file: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_0416_2001050100_123_970_12187060_1_r1621304528_23.zip
Uploading the intermediate file: upload_file_23.zip
  14:48:45 STEP  577 H= 577:00 +CPU= 43.087
  14:49:32 STEP  578 H= 578:00 +CPU= 44.440
  14:50:42 STEP  579 H= 579:00 +CPU= 67.658
  14:51:29 STEP  580 H= 580:00 +CPU= 44.373
  14:52:14 STEP  581 H= 581:00 +CPU= 43.077
  14:53:26 STEP  582 H= 582:00 +CPU= 69.400
  14:54:12 STEP  583 H= 583:00 +CPU= 42.954
  14:54:59 STEP  584 H= 584:00 +CPU= 44.525
  14:56:10 STEP  585 H= 585:00 +CPU= 67.685
  14:56:56 STEP  586 H= 586:00 +CPU= 44.508
  14:57:42 STEP  587 H= 587:00 +CPU= 43.219
  14:59:01 STEP  588 H= 588:00 +CPU= 75.616
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+000588
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+000588
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+000588
  14:59:46 STEP  589 H= 589:00 +CPU= 42.968
  15:00:32 STEP  590 H= 590:00 +CPU= 44.147
  15:01:43 STEP  591 H= 591:00 +CPU= 67.922
  15:02:30 STEP  592 H= 592:00 +CPU= 44.383
  15:03:15 STEP  593 H= 593:00 +CPU= 43.128
  15:04:28 STEP  594 H= 594:00 +CPU= 69.232
  15:05:13 STEP  595 H= 595:00 +CPU= 43.109
  15:05:59 STEP  596 H= 596:00 +CPU= 44.286
  15:07:10 STEP  597 H= 597:00 +CPU= 67.732
  15:07:56 STEP  598 H= 598:00 +CPU= 44.294
  15:08:43 STEP  599 H= 599:00 +CPU= 44.462
  15:10:07 STEP  600 H= 600:00 +CPU= 79.655
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+000600
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+000600
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+000600
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMGGhq0f+000576
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMSHhq0f+000576
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMUAhq0f+000576
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMGGhq0f+000588
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMSHhq0f+000588
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMUAhq0f+000588
Zipping up the intermediate file: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_0416_2001050100_123_970_12187060_1_r1621304528_24.zip
Uploading the intermediate file: upload_file_24.zip
  15:10:53 STEP  601 H= 601:00 +CPU= 43.005
  15:11:39 STEP  602 H= 602:00 +CPU= 44.373
  15:12:50 STEP  603 H= 603:00 +CPU= 67.666
  15:13:36 STEP  604 H= 604:00 +CPU= 44.339
  15:14:21 STEP  605 H= 605:00 +CPU= 42.928
  15:15:34 STEP  606 H= 606:00 +CPU= 69.173
  15:16:19 STEP  607 H= 607:00 +CPU= 42.393
  15:17:05 STEP  608 H= 608:00 +CPU= 44.390
  15:18:16 STEP  609 H= 609:00 +CPU= 67.567
  15:19:03 STEP  610 H= 610:00 +CPU= 44.411
  15:19:48 STEP  611 H= 611:00 +CPU= 43.111
  15:21:07 STEP  612 H= 612:00 +CPU= 75.572
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+000612
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+000612
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+000612
  15:21:52 STEP  613 H= 613:00 +CPU= 43.237
  15:22:39 STEP  614 H= 614:00 +CPU= 44.328
  15:23:49 STEP  615 H= 615:00 +CPU= 67.773
  15:24:36 STEP  616 H= 616:00 +CPU= 44.398
  15:25:21 STEP  617 H= 617:00 +CPU= 43.045
  15:26:33 STEP  618 H= 618:00 +CPU= 69.277
  15:27:19 STEP  619 H= 619:00 +CPU= 43.216
  15:28:05 STEP  620 H= 620:00 +CPU= 44.318
  15:29:16 STEP  621 H= 621:00 +CPU= 67.667
  15:30:02 STEP  622 H= 622:00 +CPU= 44.356
  15:30:48 STEP  623 H= 623:00 +CPU= 44.286
  15:32:14 STEP  624 H= 624:00 +CPU= 79.728
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+000624
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+000624
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+000624
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMGGhq0f+000600
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMSHhq0f+000600
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMUAhq0f+000600
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMGGhq0f+000612
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMSHhq0f+000612
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMUAhq0f+000612
Zipping up the intermediate file: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_0416_2001050100_123_970_12187060_1_r1621304528_25.zip
Uploading the intermediate file: upload_file_25.zip
  15:32:59 STEP  625 H= 625:00 +CPU= 42.962
  15:33:45 STEP  626 H= 626:00 +CPU= 44.258
  15:34:55 STEP  627 H= 627:00 +CPU= 67.299
  15:35:42 STEP  628 H= 628:00 +CPU= 44.546
  15:36:28 STEP  629 H= 629:00 +CPU= 42.499
  15:37:40 STEP  630 H= 630:00 +CPU= 69.307
  15:38:26 STEP  631 H= 631:00 +CPU= 42.915
  15:39:12 STEP  632 H= 632:00 +CPU= 44.499
  15:40:23 STEP  633 H= 633:00 +CPU= 67.660
  15:41:10 STEP  634 H= 634:00 +CPU= 44.434
  15:41:55 STEP  635 H= 635:00 +CPU= 43.196
  15:43:14 STEP  636 H= 636:00 +CPU= 75.718
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+000636
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+000636
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+000636
  15:43:59 STEP  637 H= 637:00 +CPU= 43.102
  15:44:46 STEP  638 H= 638:00 +CPU= 44.453
  15:45:57 STEP  639 H= 639:00 +CPU= 67.989
  15:46:43 STEP  640 H= 640:00 +CPU= 44.431
  15:47:29 STEP  641 H= 641:00 +CPU= 43.317
  15:48:41 STEP  642 H= 642:00 +CPU= 69.341
  15:49:26 STEP  643 H= 643:00 +CPU= 42.981
  15:50:13 STEP  644 H= 644:00 +CPU= 44.386
  15:51:24 STEP  645 H= 645:00 +CPU= 67.812
  15:52:10 STEP  646 H= 646:00 +CPU= 44.229
  15:52:56 STEP  647 H= 647:00 +CPU= 44.377
  15:54:21 STEP  648 H= 648:00 +CPU= 79.372
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+000648
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+000648
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+000648
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMGGhq0f+000624
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMSHhq0f+000624
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMUAhq0f+000624
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMGGhq0f+000636
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMSHhq0f+000636
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMUAhq0f+000636
Zipping up the intermediate file: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_0416_2001050100_123_970_12187060_1_r1621304528_26.zip
Uploading the intermediate file: upload_file_26.zip
  15:55:06 STEP  649 H= 649:00 +CPU= 43.038
  15:55:52 STEP  650 H= 650:00 +CPU= 44.148
  15:57:03 STEP  651 H= 651:00 +CPU= 67.492
  15:57:49 STEP  652 H= 652:00 +CPU= 44.213
  15:58:35 STEP  653 H= 653:00 +CPU= 42.897
  15:59:47 STEP  654 H= 654:00 +CPU= 68.993
  16:00:32 STEP  655 H= 655:00 +CPU= 43.066
  16:01:18 STEP  656 H= 656:00 +CPU= 44.357
  16:02:29 STEP  657 H= 657:00 +CPU= 67.782
  16:03:16 STEP  658 H= 658:00 +CPU= 44.657
  16:04:01 STEP  659 H= 659:00 +CPU= 43.187
  16:05:20 STEP  660 H= 660:00 +CPU= 75.912
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+000660
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+000660
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+000660
  16:06:05 STEP  661 H= 661:00 +CPU= 43.066
  16:06:52 STEP  662 H= 662:00 +CPU= 44.396
  16:08:03 STEP  663 H= 663:00 +CPU= 67.956
  16:08:49 STEP  664 H= 664:00 +CPU= 44.353
  16:09:34 STEP  665 H= 665:00 +CPU= 43.027
  16:10:47 STEP  666 H= 666:00 +CPU= 68.962
  16:11:32 STEP  667 H= 667:00 +CPU= 43.090
  16:12:18 STEP  668 H= 668:00 +CPU= 44.281
  16:13:29 STEP  669 H= 669:00 +CPU= 67.770
  16:14:15 STEP  670 H= 670:00 +CPU= 44.030
  16:15:02 STEP  671 H= 671:00 +CPU= 44.479
  16:16:27 STEP  672 H= 672:00 +CPU= 79.684
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+000672
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+000672
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+000672
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMGGhq0f+000648
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMSHhq0f+000648
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMUAhq0f+000648
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMGGhq0f+000660
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMSHhq0f+000660
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMUAhq0f+000660
Zipping up the intermediate file: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_0416_2001050100_123_970_12187060_1_r1621304528_27.zip
Uploading the intermediate file: upload_file_27.zip
  16:17:12 STEP  673 H= 673:00 +CPU= 43.157
  16:17:59 STEP  674 H= 674:00 +CPU= 44.252
  16:19:10 STEP  675 H= 675:00 +CPU= 67.960
  16:19:56 STEP  676 H= 676:00 +CPU= 44.264
  16:20:42 STEP  677 H= 677:00 +CPU= 43.158
  16:21:54 STEP  678 H= 678:00 +CPU= 69.403
  16:22:40 STEP  679 H= 679:00 +CPU= 43.156
  16:23:26 STEP  680 H= 680:00 +CPU= 44.437
  16:24:37 STEP  681 H= 681:00 +CPU= 67.853
  16:25:23 STEP  682 H= 682:00 +CPU= 44.413
  16:26:09 STEP  683 H= 683:00 +CPU= 43.239
  16:27:28 STEP  684 H= 684:00 +CPU= 75.979
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+000684
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+000684
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+000684
  16:28:13 STEP  685 H= 685:00 +CPU= 43.123
  16:29:00 STEP  686 H= 686:00 +CPU= 44.377
  16:30:11 STEP  687 H= 687:00 +CPU= 67.918
  16:30:57 STEP  688 H= 688:00 +CPU= 44.347
  16:31:42 STEP  689 H= 689:00 +CPU= 43.106
  16:32:55 STEP  690 H= 690:00 +CPU= 69.156
  16:33:40 STEP  691 H= 691:00 +CPU= 43.047
  16:34:27 STEP  692 H= 692:00 +CPU= 44.257
  16:35:38 STEP  693 H= 693:00 +CPU= 67.896
  16:36:25 STEP  694 H= 694:00 +CPU= 44.350
  16:37:12 STEP  695 H= 695:00 +CPU= 44.608
  16:38:37 STEP  696 H= 696:00 +CPU= 79.883
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+000696
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+000696
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+000696
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMGGhq0f+000672
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMSHhq0f+000672
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMUAhq0f+000672
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMGGhq0f+000684
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMSHhq0f+000684
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMUAhq0f+000684
Zipping up the intermediate file: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_0416_2001050100_123_970_12187060_1_r1621304528_28.zip
Uploading the intermediate file: upload_file_28.zip
  16:39:22 STEP  697 H= 697:00 +CPU= 42.967
  16:40:08 STEP  698 H= 698:00 +CPU= 44.108
  16:41:19 STEP  699 H= 699:00 +CPU= 67.917
  16:42:05 STEP  700 H= 700:00 +CPU= 44.332
  16:42:51 STEP  701 H= 701:00 +CPU= 43.124
  16:44:03 STEP  702 H= 702:00 +CPU= 69.255
  16:44:48 STEP  703 H= 703:00 +CPU= 43.120
  16:45:35 STEP  704 H= 704:00 +CPU= 44.402
  16:46:46 STEP  705 H= 705:00 +CPU= 67.952
  16:47:32 STEP  706 H= 706:00 +CPU= 44.346
  16:48:17 STEP  707 H= 707:00 +CPU= 43.130
  16:49:36 STEP  708 H= 708:00 +CPU= 75.941
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+000708
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+000708
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+000708
  16:50:22 STEP  709 H= 709:00 +CPU= 43.354
  16:51:08 STEP  710 H= 710:00 +CPU= 44.425
  16:52:19 STEP  711 H= 711:00 +CPU= 67.883
  16:53:06 STEP  712 H= 712:00 +CPU= 44.352
  16:53:51 STEP  713 H= 713:00 +CPU= 43.134
  16:55:04 STEP  714 H= 714:00 +CPU= 69.261
  16:55:49 STEP  715 H= 715:00 +CPU= 42.997
  16:56:35 STEP  716 H= 716:00 +CPU= 44.190
  16:57:46 STEP  717 H= 717:00 +CPU= 67.827
  16:58:32 STEP  718 H= 718:00 +CPU= 44.223
  16:59:19 STEP  719 H= 719:00 +CPU= 44.575
  17:00:44 STEP  720 H= 720:00 +CPU= 79.829
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+000720
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+000720
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+000720
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMGGhq0f+000696
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMSHhq0f+000696
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMUAhq0f+000696
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMGGhq0f+000708
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMSHhq0f+000708
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMUAhq0f+000708
Zipping up the intermediate file: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_0416_2001050100_123_970_12187060_1_r1621304528_29.zip
Uploading the intermediate file: upload_file_29.zip
Uploading trickle at timestep: 2592000
  17:01:29 STEP  721 H= 721:00 +CPU= 43.026
  17:02:16 STEP  722 H= 722:00 +CPU= 44.310
  17:03:27 STEP  723 H= 723:00 +CPU= 67.936
  17:04:13 STEP  724 H= 724:00 +CPU= 44.339
  17:04:58 STEP  725 H= 725:00 +CPU= 43.175
  17:06:11 STEP  726 H= 726:00 +CPU= 69.558
  17:06:56 STEP  727 H= 727:00 +CPU= 43.203
  17:07:43 STEP  728 H= 728:00 +CPU= 44.370
  17:08:54 STEP  729 H= 729:00 +CPU= 68.006
  17:09:40 STEP  730 H= 730:00 +CPU= 44.314
  17:10:26 STEP  731 H= 731:00 +CPU= 43.221
  17:11:45 STEP  732 H= 732:00 +CPU= 75.915
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+000732
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+000732
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+000732
  17:12:30 STEP  733 H= 733:00 +CPU= 42.964
  17:13:17 STEP  734 H= 734:00 +CPU= 44.426
  17:14:28 STEP  735 H= 735:00 +CPU= 67.586
  17:15:14 STEP  736 H= 736:00 +CPU= 44.343
  17:15:59 STEP  737 H= 737:00 +CPU= 43.128
  17:17:12 STEP  738 H= 738:00 +CPU= 69.306
  17:17:57 STEP  739 H= 739:00 +CPU= 43.101
  17:18:44 STEP  740 H= 740:00 +CPU= 44.334
  17:19:54 STEP  741 H= 741:00 +CPU= 67.804
  17:20:41 STEP  742 H= 742:00 +CPU= 44.216
  17:21:28 STEP  743 H= 743:00 +CPU= 44.495
  17:22:53 STEP  744 H= 744:00 +CPU= 79.728
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+000744
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+000744
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+000744
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMGGhq0f+000720
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMSHhq0f+000720
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMUAhq0f+000720
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMGGhq0f+000732
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMSHhq0f+000732
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMUAhq0f+000732
Zipping up the intermediate file: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_0416_2001050100_123_970_12187060_1_r1621304528_30.zip
Uploading the intermediate file: upload_file_30.zip
  17:23:38 STEP  745 H= 745:00 +CPU= 42.908
  17:24:24 STEP  746 H= 746:00 +CPU= 44.250
  17:25:35 STEP  747 H= 747:00 +CPU= 67.819
  17:26:21 STEP  748 H= 748:00 +CPU= 44.283
  17:27:06 STEP  749 H= 749:00 +CPU= 43.159
  17:28:19 STEP  750 H= 750:00 +CPU= 69.425
  17:29:04 STEP  751 H= 751:00 +CPU= 43.092
  17:29:50 STEP  752 H= 752:00 +CPU= 44.234
  17:31:01 STEP  753 H= 753:00 +CPU= 67.715
  17:31:48 STEP  754 H= 754:00 +CPU= 44.294
  17:32:33 STEP  755 H= 755:00 +CPU= 42.998
  17:33:52 STEP  756 H= 756:00 +CPU= 75.763
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+000756
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+000756
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+000756
  17:34:37 STEP  757 H= 757:00 +CPU= 43.081
  17:35:24 STEP  758 H= 758:00 +CPU= 44.256
  17:36:35 STEP  759 H= 759:00 +CPU= 67.858
  17:37:21 STEP  760 H= 760:00 +CPU= 44.493
  17:38:07 STEP  761 H= 761:00 +CPU= 42.988
  17:39:19 STEP  762 H= 762:00 +CPU= 69.099
  17:40:04 STEP  763 H= 763:00 +CPU= 42.862
  17:40:50 STEP  764 H= 764:00 +CPU= 44.317
  17:42:01 STEP  765 H= 765:00 +CPU= 67.660
  17:42:47 STEP  766 H= 766:00 +CPU= 44.207
  17:43:34 STEP  767 H= 767:00 +CPU= 44.271
  17:44:58 STEP  768 H= 768:00 +CPU= 79.737
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+000768
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+000768
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+000768
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMGGhq0f+000744
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMSHhq0f+000744
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMUAhq0f+000744
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMGGhq0f+000756
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMSHhq0f+000756
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMUAhq0f+000756
Zipping up the intermediate file: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_0416_2001050100_123_970_12187060_1_r1621304528_31.zip
Uploading the intermediate file: upload_file_31.zip
  17:45:43 STEP  769 H= 769:00 +CPU= 43.035
  17:46:30 STEP  770 H= 770:00 +CPU= 44.339
  17:47:40 STEP  771 H= 771:00 +CPU= 67.616
  17:48:27 STEP  772 H= 772:00 +CPU= 44.223
  17:49:12 STEP  773 H= 773:00 +CPU= 43.067
  17:50:25 STEP  774 H= 774:00 +CPU= 69.187
  17:51:10 STEP  775 H= 775:00 +CPU= 43.066
  17:51:56 STEP  776 H= 776:00 +CPU= 44.439
  17:53:07 STEP  777 H= 777:00 +CPU= 67.997
  17:53:54 STEP  778 H= 778:00 +CPU= 44.219
  17:54:39 STEP  779 H= 779:00 +CPU= 42.914
  17:55:58 STEP  780 H= 780:00 +CPU= 75.724
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+000780
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+000780
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+000780
  17:56:43 STEP  781 H= 781:00 +CPU= 43.211
  17:57:30 STEP  782 H= 782:00 +CPU= 44.406
  17:58:41 STEP  783 H= 783:00 +CPU= 67.677
  17:59:27 STEP  784 H= 784:00 +CPU= 44.294
  18:00:12 STEP  785 H= 785:00 +CPU= 43.160
  18:01:25 STEP  786 H= 786:00 +CPU= 69.237
  18:02:10 STEP  787 H= 787:00 +CPU= 43.132
  18:02:57 STEP  788 H= 788:00 +CPU= 44.315
  18:04:07 STEP  789 H= 789:00 +CPU= 67.555
  18:04:54 STEP  790 H= 790:00 +CPU= 44.315
  18:05:40 STEP  791 H= 791:00 +CPU= 44.442
  18:07:05 STEP  792 H= 792:00 +CPU= 79.652
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+000792
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+000792
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+000792
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMGGhq0f+000768
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMSHhq0f+000768
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMUAhq0f+000768
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMGGhq0f+000780
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMSHhq0f+000780
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMUAhq0f+000780
Zipping up the intermediate file: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_0416_2001050100_123_970_12187060_1_r1621304528_32.zip
Uploading the intermediate file: upload_file_32.zip
  18:07:50 STEP  793 H= 793:00 +CPU= 42.998
  18:08:36 STEP  794 H= 794:00 +CPU= 44.264
  18:09:47 STEP  795 H= 795:00 +CPU= 67.995
  18:10:34 STEP  796 H= 796:00 +CPU= 44.466
  18:11:19 STEP  797 H= 797:00 +CPU= 43.040
  18:12:31 STEP  798 H= 798:00 +CPU= 69.169
  18:13:16 STEP  799 H= 799:00 +CPU= 43.237
  18:14:03 STEP  800 H= 800:00 +CPU= 44.463
  18:15:14 STEP  801 H= 801:00 +CPU= 67.845
  18:16:00 STEP  802 H= 802:00 +CPU= 44.263
  18:16:45 STEP  803 H= 803:00 +CPU= 43.173
  18:18:05 STEP  804 H= 804:00 +CPU= 75.723
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+000804
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+000804
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+000804
  18:18:50 STEP  805 H= 805:00 +CPU= 43.097
  18:19:36 STEP  806 H= 806:00 +CPU= 44.553
  18:20:47 STEP  807 H= 807:00 +CPU= 67.818
  18:21:34 STEP  808 H= 808:00 +CPU= 44.418
  18:22:19 STEP  809 H= 809:00 +CPU= 43.095
  18:23:31 STEP  810 H= 810:00 +CPU= 69.446
  18:24:16 STEP  811 H= 811:00 +CPU= 43.056
  18:25:03 STEP  812 H= 812:00 +CPU= 44.321
  18:26:14 STEP  813 H= 813:00 +CPU= 67.760
  18:27:00 STEP  814 H= 814:00 +CPU= 44.216
  18:27:46 STEP  815 H= 815:00 +CPU= 44.580
  18:29:11 STEP  816 H= 816:00 +CPU= 79.790
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+000816
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+000816
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+000816
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMGGhq0f+000792
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMSHhq0f+000792
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMUAhq0f+000792
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMGGhq0f+000804
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMSHhq0f+000804
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMUAhq0f+000804
Zipping up the intermediate file: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_0416_2001050100_123_970_12187060_1_r1621304528_33.zip
Uploading the intermediate file: upload_file_33.zip
  18:29:56 STEP  817 H= 817:00 +CPU= 42.953
  18:30:42 STEP  818 H= 818:00 +CPU= 44.326
  18:31:53 STEP  819 H= 819:00 +CPU= 67.692
  18:32:39 STEP  820 H= 820:00 +CPU= 44.312
  18:33:24 STEP  821 H= 821:00 +CPU= 43.181
  18:34:37 STEP  822 H= 822:00 +CPU= 69.207
  18:35:22 STEP  823 H= 823:00 +CPU= 43.189
  18:36:08 STEP  824 H= 824:00 +CPU= 44.325
  18:37:20 STEP  825 H= 825:00 +CPU= 68.043
  18:38:06 STEP  826 H= 826:00 +CPU= 44.375
  18:38:51 STEP  827 H= 827:00 +CPU= 43.077
  18:40:11 STEP  828 H= 828:00 +CPU= 75.818
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+000828
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+000828
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+000828
  18:40:56 STEP  829 H= 829:00 +CPU= 43.163
  18:41:43 STEP  830 H= 830:00 +CPU= 44.429
  18:42:54 STEP  831 H= 831:00 +CPU= 67.878
  18:43:40 STEP  832 H= 832:00 +CPU= 44.283
  18:44:25 STEP  833 H= 833:00 +CPU= 43.157
  18:45:38 STEP  834 H= 834:00 +CPU= 69.327
  18:46:23 STEP  835 H= 835:00 +CPU= 43.097
  18:47:09 STEP  836 H= 836:00 +CPU= 44.435
  18:48:20 STEP  837 H= 837:00 +CPU= 67.771
  18:49:06 STEP  838 H= 838:00 +CPU= 44.377
  18:49:53 STEP  839 H= 839:00 +CPU= 44.408
  18:51:17 STEP  840 H= 840:00 +CPU= 79.535
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+000840
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+000840
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+000840
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMGGhq0f+000816
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMSHhq0f+000816
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMUAhq0f+000816
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMGGhq0f+000828
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMSHhq0f+000828
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMUAhq0f+000828
Zipping up the intermediate file: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_0416_2001050100_123_970_12187060_1_r1621304528_34.zip
Uploading the intermediate file: upload_file_34.zip
  18:52:02 STEP  841 H= 841:00 +CPU= 43.013
  18:52:49 STEP  842 H= 842:00 +CPU= 44.413
  18:54:00 STEP  843 H= 843:00 +CPU= 67.682
  18:54:46 STEP  844 H= 844:00 +CPU= 44.479
  18:55:31 STEP  845 H= 845:00 +CPU= 43.088
  18:56:44 STEP  846 H= 846:00 +CPU= 69.203
  18:57:29 STEP  847 H= 847:00 +CPU= 43.120
  18:58:16 STEP  848 H= 848:00 +CPU= 44.341
  18:59:26 STEP  849 H= 849:00 +CPU= 67.723
  19:00:13 STEP  850 H= 850:00 +CPU= 44.361
  19:00:58 STEP  851 H= 851:00 +CPU= 43.101
  19:02:18 STEP  852 H= 852:00 +CPU= 75.909
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+000852
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+000852
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+000852
  19:03:03 STEP  853 H= 853:00 +CPU= 43.203
  19:03:49 STEP  854 H= 854:00 +CPU= 44.370
  19:05:00 STEP  855 H= 855:00 +CPU= 67.614
  19:05:46 STEP  856 H= 856:00 +CPU= 44.270
  19:06:31 STEP  857 H= 857:00 +CPU= 42.799
  19:07:43 STEP  858 H= 858:00 +CPU= 68.887
  19:08:29 STEP  859 H= 859:00 +CPU= 43.073
  19:09:15 STEP  860 H= 860:00 +CPU= 44.380
  19:10:26 STEP  861 H= 861:00 +CPU= 67.759
  19:11:12 STEP  862 H= 862:00 +CPU= 44.348
  19:11:59 STEP  863 H= 863:00 +CPU= 44.388
  19:13:24 STEP  864 H= 864:00 +CPU= 79.730
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+000864
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+000864
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+000864
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMGGhq0f+000840
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMSHhq0f+000840
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMUAhq0f+000840
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMGGhq0f+000852
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMSHhq0f+000852
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMUAhq0f+000852
Zipping up the intermediate file: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_0416_2001050100_123_970_12187060_1_r1621304528_35.zip
Uploading the intermediate file: upload_file_35.zip
  19:14:10 STEP  865 H= 865:00 +CPU= 43.066
  19:14:56 STEP  866 H= 866:00 +CPU= 44.380
  19:16:07 STEP  867 H= 867:00 +CPU= 67.526
  19:16:53 STEP  868 H= 868:00 +CPU= 44.383
  19:17:38 STEP  869 H= 869:00 +CPU= 43.169
  19:18:51 STEP  870 H= 870:00 +CPU= 69.107
  19:19:36 STEP  871 H= 871:00 +CPU= 43.057
  19:20:22 STEP  872 H= 872:00 +CPU= 44.357
  19:21:33 STEP  873 H= 873:00 +CPU= 67.777
  19:22:20 STEP  874 H= 874:00 +CPU= 44.357
  19:23:05 STEP  875 H= 875:00 +CPU= 43.214
  19:24:24 STEP  876 H= 876:00 +CPU= 75.769
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+000876
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+000876
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+000876
  19:25:09 STEP  877 H= 877:00 +CPU= 43.089
  19:25:56 STEP  878 H= 878:00 +CPU= 44.390
  19:27:07 STEP  879 H= 879:00 +CPU= 67.901
  19:27:53 STEP  880 H= 880:00 +CPU= 44.298
  19:28:38 STEP  881 H= 881:00 +CPU= 42.983
  19:29:50 STEP  882 H= 882:00 +CPU= 69.114
  19:30:35 STEP  883 H= 883:00 +CPU= 42.920
  19:31:22 STEP  884 H= 884:00 +CPU= 44.329
  19:32:32 STEP  885 H= 885:00 +CPU= 67.390
  19:33:19 STEP  886 H= 886:00 +CPU= 44.231
  19:34:05 STEP  887 H= 887:00 +CPU= 44.287
  19:35:30 STEP  888 H= 888:00 +CPU= 79.653
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+000888
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+000888
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+000888
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMGGhq0f+000864
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMSHhq0f+000864
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMUAhq0f+000864
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMGGhq0f+000876
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMSHhq0f+000876
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMUAhq0f+000876
Zipping up the intermediate file: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_0416_2001050100_123_970_12187060_1_r1621304528_36.zip
Uploading the intermediate file: upload_file_36.zip
  19:36:14 STEP  889 H= 889:00 +CPU= 42.929
  19:37:01 STEP  890 H= 890:00 +CPU= 44.339
  19:38:12 STEP  891 H= 891:00 +CPU= 67.729
  19:38:58 STEP  892 H= 892:00 +CPU= 44.345
  19:39:43 STEP  893 H= 893:00 +CPU= 43.006
  19:40:55 STEP  894 H= 894:00 +CPU= 69.154
  19:41:41 STEP  895 H= 895:00 +CPU= 43.242
  19:42:27 STEP  896 H= 896:00 +CPU= 44.263
  19:43:38 STEP  897 H= 897:00 +CPU= 67.877
  19:44:25 STEP  898 H= 898:00 +CPU= 44.463
  19:45:11 STEP  899 H= 899:00 +CPU= 43.521
  19:46:40 STEP  900 H= 900:00 +CPU= 78.232
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+000900
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+000900
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+000900
  19:47:33 STEP  901 H= 901:00 +CPU= 44.086
  19:48:36 STEP  902 H= 902:00 +CPU= 46.514
  19:50:00 STEP  903 H= 903:00 +CPU= 70.953
  19:50:56 STEP  904 H= 904:00 +CPU= 46.690
  19:51:51 STEP  905 H= 905:00 +CPU= 45.126
  19:53:16 STEP  906 H= 906:00 +CPU= 72.479
  19:54:08 STEP  907 H= 907:00 +CPU= 44.867
  19:55:02 STEP  908 H= 908:00 +CPU= 46.405
  19:56:23 STEP  909 H= 909:00 +CPU= 70.376
  19:57:16 STEP  910 H= 910:00 +CPU= 46.413
  19:58:12 STEP  911 H= 911:00 +CPU= 46.841
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2001050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0416
(argv4) batchid: 970
(argv5) wuid: 12187060
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /home/luuser/slots/0
Project directory is: /home/luuser/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060
..mkdir for temp folder for results failed
Copying: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /home/luuser/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /home/luuser/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_3f69f40f2ae1c3f44af3198801c24391 to: /home/luuser/slots/0/oifs_43r3_ps_0416_2001050100_123_970_12187060.zip
Unzipping the namelist zip file: /home/luuser/slots/0/oifs_43r3_ps_0416_2001050100_123_970_12187060.zip
ic_ancil_file: ic_ancil_12187060
ifsdata_file: ifsdata_12187060
climate_data_file: clim_data_12187060
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_f70c85ef1405c1a2da48f151f6d7ee8c to: /home/luuser/slots/0/ic_ancil_12187060.zip
Unzipping the IC ancils zip file: /home/luuser/slots/0/ic_ancil_12187060.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_7a72d89200114ad24676c3cd7f006a8e to: /home/luuser/slots/0/ifsdata/ifsdata_12187060.zip
Unzipping the ifsdata_zip file: /home/luuser/slots/0/ifsdata/ifsdata_12187060.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /home/luuser/slots/0/159l_2/clim_data_12187060.zip
Unzipping the climate data zip file: /home/luuser/slots/0/159l_2/clim_data_12187060.zip
Checking for progress XML file: /home/luuser/slots/0/progress_file_12187060.xml
Opened progress file ok : /home/luuser/slots/0/progress_file_12187060.xml
-- Model is restarting --
Adjusting last_iter, 912, to previous model restart step.
Creating progress file: /home/luuser/slots/0/progress_file_12187060.xml
last_cpu_time: 48222
upload_file_number: 37
last_iter: 911
last_upload: 3196800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0416_2001050100_123_970_12187060_1_r1621304528
The child process has been launched with process id: 10462
Executing the command: /home/luuser/slots/0/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:hp6005:1:1:10462:10462] [20230107:200023:1673114423:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:hp6005:1:1:10462:10462] [20230107:200023:1673114423:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:hp6005:1:1:10462:10462] [20230107:200023:1673114423:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:hp6005:1:1:10462:10462] [20230107:200023:1673114423:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:hp6005:1:1:10462:10462] [20230107:200023:1673114423:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:hp6005:1:1:10462:10462] [20230107:200023:1673114423:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:hp6005:1:1:10462:10462] [20230107:200023:1673114423:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:hp6005:1:1:10462:10462] [20230107:200023:1673114423:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:hp6005:1:1:10462:10462] [20230107:200023:1673114423:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:hp6005:1:1:10462:10462] [20230107:200023:1673114423:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:hp6005:1:1:10462:10462] [20230107:200023:1673114423:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:hp6005:1:1:10462:10462] [20230107:200023:1673114423:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:hp6005:1:1:10462:10462] [20230107:200023:1673114423:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:hp6005:1:1:10462:10462] [20230107:200023:1673114423:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:hp6005:1:1:10462:10462] [20230107:200023:1673114423:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:hp6005:1:1:10462:10462] [20230107:200023:1673114423:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:hp6005:1:1:10462:10462] [20230107:200023:1673114423:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:hp6005:1:1:10462:10462] [20230107:200023:1673114423:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:hp6005:1:1:10462:10462] [20230107:200023:1673114423:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:hp6005:1:1:10462:10462] [20230107:200023:1673114423:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:hp6005:1:1:10462:10462] [20230107:200023:1673114423:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:hp6005:1:1:10462:10462] [20230107:200023:1673114423:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:hp6005:1:1:10462:10462] [20230107:200023:1673114423:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:hp6005:1:1:10462:10462] [20230107:200023:1673114423:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:hp6005:1:1:10462:10462] [20230107:200023:1673114423:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:hp6005:1:1:10462:10462] [20230107:200023:1673114423:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:hp6005:1:1:10462:10462] [20230107:200023:1673114423:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:hp6005:1:1:10462:10462] [20230107:200023:1673114423:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:hp6005:1:1:10462:10462] [20230107:200023:1673114423:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:hp6005:1:1:10462:10462] [20230107:200023:1673114423:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  20:01:56 STEP  912 H= 912:00 +CPU= 80.084
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+000912
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+000912
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+000912
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMGGhq0f+000888
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMSHhq0f+000888
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMUAhq0f+000888
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMGGhq0f+000900
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMSHhq0f+000900
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMUAhq0f+000900
Zipping up the intermediate file: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_0416_2001050100_123_970_12187060_1_r1621304528_37.zip
Uploading the intermediate file: upload_file_37.zip
  20:02:43 STEP  913 H= 913:00 +CPU= 43.882
  20:03:31 STEP  914 H= 914:00 +CPU= 45.255
  20:04:45 STEP  915 H= 915:00 +CPU= 69.064
  20:05:33 STEP  916 H= 916:00 +CPU= 45.232
  20:06:20 STEP  917 H= 917:00 +CPU= 43.812
  20:07:35 STEP  918 H= 918:00 +CPU= 70.611
  20:08:22 STEP  919 H= 919:00 +CPU= 43.931
  20:09:10 STEP  920 H= 920:00 +CPU= 45.200
  20:10:24 STEP  921 H= 921:00 +CPU= 69.298
  20:11:13 STEP  922 H= 922:00 +CPU= 45.411
  20:12:00 STEP  923 H= 923:00 +CPU= 43.864
  20:13:22 STEP  924 H= 924:00 +CPU= 77.337
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+000924
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+000924
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+000924
  20:14:09 STEP  925 H= 925:00 +CPU= 44.052
  20:14:58 STEP  926 H= 926:00 +CPU= 45.308
  20:16:11 STEP  927 H= 927:00 +CPU= 69.191
  20:17:00 STEP  928 H= 928:00 +CPU= 45.179
  20:17:47 STEP  929 H= 929:00 +CPU= 43.993
  20:19:02 STEP  930 H= 930:00 +CPU= 70.591
  20:19:49 STEP  931 H= 931:00 +CPU= 43.982
  20:20:37 STEP  932 H= 932:00 +CPU= 45.244
  20:21:51 STEP  933 H= 933:00 +CPU= 69.279
  20:22:39 STEP  934 H= 934:00 +CPU= 45.084
  20:23:27 STEP  935 H= 935:00 +CPU= 45.343
  20:24:56 STEP  936 H= 936:00 +CPU= 81.279
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+000936
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+000936
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+000936
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMGGhq0f+000912
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMSHhq0f+000912
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMUAhq0f+000912
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMGGhq0f+000924
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMSHhq0f+000924
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMUAhq0f+000924
Zipping up the intermediate file: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_0416_2001050100_123_970_12187060_1_r1621304528_38.zip
Uploading the intermediate file: upload_file_38.zip
  20:25:42 STEP  937 H= 937:00 +CPU= 43.558
  20:26:30 STEP  938 H= 938:00 +CPU= 45.158
  20:27:43 STEP  939 H= 939:00 +CPU= 68.885
  20:28:31 STEP  940 H= 940:00 +CPU= 45.069
  20:29:17 STEP  941 H= 941:00 +CPU= 43.598
  20:30:31 STEP  942 H= 942:00 +CPU= 70.221
  20:31:17 STEP  943 H= 943:00 +CPU= 43.739
  20:32:05 STEP  944 H= 944:00 +CPU= 45.043
  20:33:18 STEP  945 H= 945:00 +CPU= 68.973
  20:34:06 STEP  946 H= 946:00 +CPU= 45.331
  20:34:52 STEP  947 H= 947:00 +CPU= 43.800
  20:36:14 STEP  948 H= 948:00 +CPU= 77.066
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+000948
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+000948
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+000948
  20:37:00 STEP  949 H= 949:00 +CPU= 43.739
  20:37:47 STEP  950 H= 950:00 +CPU= 44.937
  20:39:01 STEP  951 H= 951:00 +CPU= 69.182
  20:39:49 STEP  952 H= 952:00 +CPU= 45.272
  20:40:36 STEP  953 H= 953:00 +CPU= 43.878
  20:41:51 STEP  954 H= 954:00 +CPU= 70.820
  20:42:38 STEP  955 H= 955:00 +CPU= 43.864
  20:43:26 STEP  956 H= 956:00 +CPU= 45.257
  20:44:40 STEP  957 H= 957:00 +CPU= 69.057
 SMILAG TRAJECTORY UNDERGROUND      1 TIMES.
  POINT=           13  MAX ETADOT VERTICAL VEL.=   8.261413646111505E-008
  LON  =    168.750000000000       degrees
  LAT  =   -39.8128523737712       degrees
  20:45:28 STEP  958 H= 958:00 +CPU= 45.206
  20:46:17 STEP  959 H= 959:00 +CPU= 45.550
  20:47:47 STEP  960 H= 960:00 +CPU= 81.706
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+000960
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+000960
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+000960
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMGGhq0f+000936
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMSHhq0f+000936
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMUAhq0f+000936
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMGGhq0f+000948
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMSHhq0f+000948
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMUAhq0f+000948
Zipping up the intermediate file: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_0416_2001050100_123_970_12187060_1_r1621304528_39.zip
Uploading the intermediate file: upload_file_39.zip
 SMILAG TRAJECTORY UNDERGROUND      1 TIMES.
  POINT=           15  MAX ETADOT VERTICAL VEL.=   5.946130502045252E-008
  LON  =    166.250000000000       degrees
  LAT  =   -40.9343383242788       degrees
  20:48:34 STEP  961 H= 961:00 +CPU= 43.901
  20:49:23 STEP  962 H= 962:00 +CPU= 45.219
  20:50:37 STEP  963 H= 963:00 +CPU= 69.065
  20:51:25 STEP  964 H= 964:00 +CPU= 45.320
  20:52:12 STEP  965 H= 965:00 +CPU= 43.982
  20:53:27 STEP  966 H= 966:00 +CPU= 70.622
  20:54:14 STEP  967 H= 967:00 +CPU= 44.094
  20:55:03 STEP  968 H= 968:00 +CPU= 45.388
  20:56:16 STEP  969 H= 969:00 +CPU= 69.125
  20:57:05 STEP  970 H= 970:00 +CPU= 45.358
  20:57:52 STEP  971 H= 971:00 +CPU= 44.107
  20:59:14 STEP  972 H= 972:00 +CPU= 77.334
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+000972
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+000972
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+000972
  21:00:01 STEP  973 H= 973:00 +CPU= 44.154
  21:00:50 STEP  974 H= 974:00 +CPU= 45.297
  21:02:04 STEP  975 H= 975:00 +CPU= 69.211
  21:02:52 STEP  976 H= 976:00 +CPU= 45.252
  21:03:39 STEP  977 H= 977:00 +CPU= 44.189
  21:04:54 STEP  978 H= 978:00 +CPU= 70.457
  21:05:41 STEP  979 H= 979:00 +CPU= 44.210
  21:06:30 STEP  980 H= 980:00 +CPU= 45.155
  21:07:43 STEP  981 H= 981:00 +CPU= 69.091
  21:08:32 STEP  982 H= 982:00 +CPU= 45.188
  21:09:20 STEP  983 H= 983:00 +CPU= 45.329
  21:10:48 STEP  984 H= 984:00 +CPU= 81.407
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+000984
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+000984
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+000984
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMGGhq0f+000960
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMSHhq0f+000960
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMUAhq0f+000960
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMGGhq0f+000972
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMSHhq0f+000972
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMUAhq0f+000972
Zipping up the intermediate file: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_0416_2001050100_123_970_12187060_1_r1621304528_40.zip
Uploading the intermediate file: upload_file_40.zip
  21:11:35 STEP  985 H= 985:00 +CPU= 43.985
  21:12:23 STEP  986 H= 986:00 +CPU= 45.217
  21:13:37 STEP  987 H= 987:00 +CPU= 69.024
  21:14:25 STEP  988 H= 988:00 +CPU= 45.201
  21:15:12 STEP  989 H= 989:00 +CPU= 44.184
  21:16:27 STEP  990 H= 990:00 +CPU= 70.719
  21:17:15 STEP  991 H= 991:00 +CPU= 44.114
  21:18:03 STEP  992 H= 992:00 +CPU= 45.202
  21:19:17 STEP  993 H= 993:00 +CPU= 69.195
  21:20:05 STEP  994 H= 994:00 +CPU= 45.483
  21:20:52 STEP  995 H= 995:00 +CPU= 44.078
  21:22:15 STEP  996 H= 996:00 +CPU= 77.435
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+000996
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+000996
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+000996
  21:23:02 STEP  997 H= 997:00 +CPU= 44.187
  21:23:50 STEP  998 H= 998:00 +CPU= 45.415
  21:25:04 STEP  999 H= 999:00 +CPU= 69.005
  21:25:52 STEP 1000 H=1000:00 +CPU= 45.563
  21:26:39 STEP 1001 H=1001:00 +CPU= 43.927
  21:27:54 STEP 1002 H=1002:00 +CPU= 70.573
  21:28:41 STEP 1003 H=1003:00 +CPU= 43.996
  21:29:30 STEP 1004 H=1004:00 +CPU= 45.255
  21:30:43 STEP 1005 H=1005:00 +CPU= 69.227
  21:31:32 STEP 1006 H=1006:00 +CPU= 45.228
  21:32:20 STEP 1007 H=1007:00 +CPU= 45.301
  21:33:48 STEP 1008 H=1008:00 +CPU= 81.383
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+001008
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+001008
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+001008
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMGGhq0f+000984
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMSHhq0f+000984
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMUAhq0f+000984
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMGGhq0f+000996
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMSHhq0f+000996
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMUAhq0f+000996
Zipping up the intermediate file: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_0416_2001050100_123_970_12187060_1_r1621304528_41.zip
Uploading the intermediate file: upload_file_41.zip
  21:34:35 STEP 1009 H=1009:00 +CPU= 43.847
  21:35:24 STEP 1010 H=1010:00 +CPU= 45.389
  21:36:37 STEP 1011 H=1011:00 +CPU= 69.080
  21:37:26 STEP 1012 H=1012:00 +CPU= 45.528
  21:38:13 STEP 1013 H=1013:00 +CPU= 44.112
  21:39:29 STEP 1014 H=1014:00 +CPU= 70.871
  21:40:16 STEP 1015 H=1015:00 +CPU= 44.240
  21:41:05 STEP 1016 H=1016:00 +CPU= 45.413
  21:42:19 STEP 1017 H=1017:00 +CPU= 69.238
  21:43:07 STEP 1018 H=1018:00 +CPU= 45.444
  21:43:54 STEP 1019 H=1019:00 +CPU= 44.147
  21:45:17 STEP 1020 H=1020:00 +CPU= 77.595
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+001020
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+001020
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+001020
  21:46:04 STEP 1021 H=1021:00 +CPU= 43.929
  21:46:52 STEP 1022 H=1022:00 +CPU= 45.388
  21:48:06 STEP 1023 H=1023:00 +CPU= 69.281
  21:48:55 STEP 1024 H=1024:00 +CPU= 45.488
  21:49:42 STEP 1025 H=1025:00 +CPU= 44.178
  21:50:58 STEP 1026 H=1026:00 +CPU= 70.990
  21:51:45 STEP 1027 H=1027:00 +CPU= 43.997
  21:52:33 STEP 1028 H=1028:00 +CPU= 45.501
  21:53:47 STEP 1029 H=1029:00 +CPU= 69.354
  21:54:35 STEP 1030 H=1030:00 +CPU= 45.147
  21:55:24 STEP 1031 H=1031:00 +CPU= 45.532
  21:56:52 STEP 1032 H=1032:00 +CPU= 81.522
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+001032
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+001032
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+001032
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMGGhq0f+001008
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMSHhq0f+001008
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMUAhq0f+001008
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMGGhq0f+001020
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMSHhq0f+001020
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMUAhq0f+001020
Zipping up the intermediate file: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_0416_2001050100_123_970_12187060_1_r1621304528_42.zip
Uploading the intermediate file: upload_file_42.zip
  21:57:40 STEP 1033 H=1033:00 +CPU= 44.056
  21:58:28 STEP 1034 H=1034:00 +CPU= 45.387
  21:59:42 STEP 1035 H=1035:00 +CPU= 69.232
  22:00:30 STEP 1036 H=1036:00 +CPU= 45.439
  22:01:17 STEP 1037 H=1037:00 +CPU= 44.085
  22:02:33 STEP 1038 H=1038:00 +CPU= 70.715
  22:03:20 STEP 1039 H=1039:00 +CPU= 44.094
  22:04:08 STEP 1040 H=1040:00 +CPU= 45.385
  22:05:23 STEP 1041 H=1041:00 +CPU= 69.566
  22:06:11 STEP 1042 H=1042:00 +CPU= 45.394
  22:06:59 STEP 1043 H=1043:00 +CPU= 44.264
  22:08:21 STEP 1044 H=1044:00 +CPU= 77.656
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+001044
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+001044
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+001044
  22:09:09 STEP 1045 H=1045:00 +CPU= 44.198
  22:09:57 STEP 1046 H=1046:00 +CPU= 45.360
  22:11:11 STEP 1047 H=1047:00 +CPU= 69.246
  22:11:59 STEP 1048 H=1048:00 +CPU= 45.420
  22:12:47 STEP 1049 H=1049:00 +CPU= 44.243
  22:14:02 STEP 1050 H=1050:00 +CPU= 70.589
  22:14:49 STEP 1051 H=1051:00 +CPU= 44.085
  22:15:37 STEP 1052 H=1052:00 +CPU= 45.304
  22:16:51 STEP 1053 H=1053:00 +CPU= 69.070
  22:17:39 STEP 1054 H=1054:00 +CPU= 45.291
  22:18:28 STEP 1055 H=1055:00 +CPU= 45.295
  22:19:56 STEP 1056 H=1056:00 +CPU= 81.317
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+001056
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+001056
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+001056
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMGGhq0f+001032
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMSHhq0f+001032
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMUAhq0f+001032
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMGGhq0f+001044
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMSHhq0f+001044
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMUAhq0f+001044
Zipping up the intermediate file: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_0416_2001050100_123_970_12187060_1_r1621304528_43.zip
Uploading the intermediate file: upload_file_43.zip
  22:20:42 STEP 1057 H=1057:00 +CPU= 43.958
  22:21:31 STEP 1058 H=1058:00 +CPU= 45.469
  22:22:45 STEP 1059 H=1059:00 +CPU= 69.266
  22:23:33 STEP 1060 H=1060:00 +CPU= 45.411
  22:24:20 STEP 1061 H=1061:00 +CPU= 43.963
  22:25:36 STEP 1062 H=1062:00 +CPU= 70.671
  22:26:23 STEP 1063 H=1063:00 +CPU= 44.266
  22:27:12 STEP 1064 H=1064:00 +CPU= 45.427
  22:28:26 STEP 1065 H=1065:00 +CPU= 69.352
  22:29:14 STEP 1066 H=1066:00 +CPU= 45.493
  22:30:01 STEP 1067 H=1067:00 +CPU= 43.943
  22:31:24 STEP 1068 H=1068:00 +CPU= 77.703
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+001068
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+001068
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+001068
  22:32:11 STEP 1069 H=1069:00 +CPU= 44.082
  22:33:00 STEP 1070 H=1070:00 +CPU= 45.442
  22:34:14 STEP 1071 H=1071:00 +CPU= 69.390
  22:35:02 STEP 1072 H=1072:00 +CPU= 45.374
  22:35:49 STEP 1073 H=1073:00 +CPU= 43.959
  22:37:04 STEP 1074 H=1074:00 +CPU= 70.560
  22:37:51 STEP 1075 H=1075:00 +CPU= 44.133
  22:38:40 STEP 1076 H=1076:00 +CPU= 45.338
  22:39:54 STEP 1077 H=1077:00 +CPU= 69.168
  22:40:42 STEP 1078 H=1078:00 +CPU= 45.508
  22:41:31 STEP 1079 H=1079:00 +CPU= 45.553
  22:42:59 STEP 1080 H=1080:00 +CPU= 81.509
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+001080
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+001080
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+001080
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMGGhq0f+001056
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMSHhq0f+001056
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMUAhq0f+001056
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMGGhq0f+001068
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMSHhq0f+001068
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMUAhq0f+001068
Zipping up the intermediate file: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_0416_2001050100_123_970_12187060_1_r1621304528_44.zip
Uploading the intermediate file: upload_file_44.zip
  22:43:46 STEP 1081 H=1081:00 +CPU= 44.060
  22:44:35 STEP 1082 H=1082:00 +CPU= 45.380
  22:45:49 STEP 1083 H=1083:00 +CPU= 69.303
  22:46:37 STEP 1084 H=1084:00 +CPU= 45.346
  22:47:24 STEP 1085 H=1085:00 +CPU= 44.078
  22:48:40 STEP 1086 H=1086:00 +CPU= 70.853
  22:49:27 STEP 1087 H=1087:00 +CPU= 44.048
  22:50:15 STEP 1088 H=1088:00 +CPU= 45.403
  22:51:29 STEP 1089 H=1089:00 +CPU= 69.320
  22:52:18 STEP 1090 H=1090:00 +CPU= 45.335
  22:53:05 STEP 1091 H=1091:00 +CPU= 44.137
  22:54:27 STEP 1092 H=1092:00 +CPU= 77.231
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+001092
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+001092
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+001092
  22:55:14 STEP 1093 H=1093:00 +CPU= 44.185
  22:56:03 STEP 1094 H=1094:00 +CPU= 45.421
  22:57:17 STEP 1095 H=1095:00 +CPU= 69.239
  22:58:06 STEP 1096 H=1096:00 +CPU= 45.584
  22:58:53 STEP 1097 H=1097:00 +CPU= 44.283
  23:00:08 STEP 1098 H=1098:00 +CPU= 70.739
  23:00:55 STEP 1099 H=1099:00 +CPU= 43.976
  23:01:43 STEP 1100 H=1100:00 +CPU= 45.138
  23:02:57 STEP 1101 H=1101:00 +CPU= 69.365
  23:03:46 STEP 1102 H=1102:00 +CPU= 45.435
  23:04:34 STEP 1103 H=1103:00 +CPU= 45.465
  23:06:03 STEP 1104 H=1104:00 +CPU= 81.561
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+001104
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+001104
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+001104
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMGGhq0f+001080
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMSHhq0f+001080
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMUAhq0f+001080
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMGGhq0f+001092
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMSHhq0f+001092
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMUAhq0f+001092
Zipping up the intermediate file: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_0416_2001050100_123_970_12187060_1_r1621304528_45.zip
Uploading the intermediate file: upload_file_45.zip
  23:06:50 STEP 1105 H=1105:00 +CPU= 43.956
  23:07:38 STEP 1106 H=1106:00 +CPU= 45.511
  23:08:52 STEP 1107 H=1107:00 +CPU= 69.327
  23:09:41 STEP 1108 H=1108:00 +CPU= 45.319
  23:10:28 STEP 1109 H=1109:00 +CPU= 43.986
  23:11:43 STEP 1110 H=1110:00 +CPU= 71.048
  23:12:31 STEP 1111 H=1111:00 +CPU= 44.118
  23:13:19 STEP 1112 H=1112:00 +CPU= 45.438
  23:14:33 STEP 1113 H=1113:00 +CPU= 69.392
  23:15:22 STEP 1114 H=1114:00 +CPU= 45.506
  23:16:09 STEP 1115 H=1115:00 +CPU= 44.129
  23:17:32 STEP 1116 H=1116:00 +CPU= 77.629
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+001116
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+001116
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+001116
  23:18:19 STEP 1117 H=1117:00 +CPU= 44.333
  23:19:08 STEP 1118 H=1118:00 +CPU= 45.616
  23:20:22 STEP 1119 H=1119:00 +CPU= 69.614
  23:21:11 STEP 1120 H=1120:00 +CPU= 45.535
  23:21:58 STEP 1121 H=1121:00 +CPU= 44.139
  23:23:14 STEP 1122 H=1122:00 +CPU= 70.936
  23:24:01 STEP 1123 H=1123:00 +CPU= 44.211
  23:24:49 STEP 1124 H=1124:00 +CPU= 45.433
  23:26:03 STEP 1125 H=1125:00 +CPU= 69.337
  23:26:52 STEP 1126 H=1126:00 +CPU= 45.450
  23:27:40 STEP 1127 H=1127:00 +CPU= 45.511
  23:29:09 STEP 1128 H=1128:00 +CPU= 81.701
Moving to projects directory: /home/luuser/slots/0/ICMGGhq0f+001128
Moving to projects directory: /home/luuser/slots/0/ICMSHhq0f+001128
Moving to projects directory: /home/luuser/slots/0/ICMUAhq0f+001128
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMGGhq0f+001104
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMSHhq0f+001104
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMUAhq0f+001104
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMGGhq0f+001116
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMSHhq0f+001116
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12187060/ICMUAhq0f+001116
Zipping up the intermediate file: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_0416_2001050100_123_970_12187060_1_r1621304528_46.zip
Uploading the intermediate file: upload_file_46.zip
Uploading trickle at timestep: 4060800
  23:29:57 STEP 1129 H=1129:00 +CPU= 44.229
  23:30:51 STEP 1130 H=1130:00 +CPU= 45.267
The child process terminated with status: 0
..Failed, model did not complete successfully

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:12:06 1537854 22282905 oifs_43r3_ps_0416_2001050100_123_970_12187060_1 4,060,800 59,954 0.0148
09 May 2023 16:11:48 1537854 22282905 oifs_43r3_ps_0416_2001050100_123_970_12187060_1 2,592,000 38,111 0.0147
09 May 2023 16:11:38 1537854 22282905 oifs_43r3_ps_0416_2001050100_123_970_12187060_1 1,728,000 25,454 0.0147
09 May 2023 16:11:29 1537854 22282905 oifs_43r3_ps_0416_2001050100_123_970_12187060_1 864,000 12,805 0.0148


©2024 climateprediction.net