climateprediction.net home page
Task 22277093

Task 22277093

Name oifs_43r3_ps_0771_2005050100_123_974_12191415_0
Workunit 12191415
Created 23 Dec 2022, 11:26:26 UTC
Sent 11 Jan 2023, 12:44:55 UTC
Report deadline 10 Feb 2023, 12:44:55 UTC
Received 12 Jan 2023, 14:09:50 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1536520
Run time 1 days 0 hours 17 min 16 sec
CPU time 23 hours 50 min 28 sec
Validate state Invalid
Credit 6,531.84
Device peak FLOPS 1.00 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,419.20 MB
Peak swap size 4,978.14 MB
Peak disk usage 7,169.11 MB
Stderr
<core_client_version>7.20.2</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
_ps_12191415/ICMGGhq0f+000696
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMSHhq0f+000696
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMUAhq0f+000696
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMGGhq0f+000708
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMSHhq0f+000708
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMUAhq0f+000708
Zipping up the intermediate file: /home/steve/projects/climateprediction.net/oifs_43r3_ps_0771_2005050100_123_974_12191415_0_r671616468_29.zip
Uploading the intermediate file: upload_file_29.zip
  19:08:26 STEP  721 H= 721:00 +CPU= 56.622
  19:09:23 STEP  722 H= 722:00 +CPU= 55.245
  19:10:45 STEP  723 H= 723:00 +CPU= 80.957
  19:11:42 STEP  724 H= 724:00 +CPU= 56.246
  19:12:41 STEP  725 H= 725:00 +CPU= 58.005
  19:14:08 STEP  726 H= 726:00 +CPU= 86.016
  19:15:07 STEP  727 H= 727:00 +CPU= 58.202
  19:16:05 STEP  728 H= 728:00 +CPU= 56.454
  19:17:30 STEP  729 H= 729:00 +CPU= 83.518
  19:18:35 STEP  730 H= 730:00 +CPU= 63.786
  19:19:30 STEP  731 H= 731:00 +CPU= 54.407
  19:21:08 STEP  732 H= 732:00 +CPU= 96.135
Moving to projects directory: /home/steve/slots/18/ICMGGhq0f+000732
Moving to projects directory: /home/steve/slots/18/ICMSHhq0f+000732
Moving to projects directory: /home/steve/slots/18/ICMUAhq0f+000732
  19:22:02 STEP  733 H= 733:00 +CPU= 53.425
  19:23:02 STEP  734 H= 734:00 +CPU= 59.822
  19:24:21 STEP  735 H= 735:00 +CPU= 77.327
  19:25:17 STEP  736 H= 736:00 +CPU= 55.105
  19:26:10 STEP  737 H= 737:00 +CPU= 52.328
  19:27:38 STEP  738 H= 738:00 +CPU= 87.059
  19:28:36 STEP  739 H= 739:00 +CPU= 56.721
  19:29:33 STEP  740 H= 740:00 +CPU= 56.892
  19:30:58 STEP  741 H= 741:00 +CPU= 83.028
  19:31:57 STEP  742 H= 742:00 +CPU= 58.953
  19:32:56 STEP  743 H= 743:00 +CPU= 58.049
  19:34:40 STEP  744 H= 744:00 +CPU=102.456
Moving to projects directory: /home/steve/slots/18/ICMGGhq0f+000744
Moving to projects directory: /home/steve/slots/18/ICMSHhq0f+000744
Moving to projects directory: /home/steve/slots/18/ICMUAhq0f+000744
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMGGhq0f+000720
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMSHhq0f+000720
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMUAhq0f+000720
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMGGhq0f+000732
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMSHhq0f+000732
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMUAhq0f+000732
Zipping up the intermediate file: /home/steve/projects/climateprediction.net/oifs_43r3_ps_0771_2005050100_123_974_12191415_0_r671616468_30.zip
Uploading the intermediate file: upload_file_30.zip
  19:35:37 STEP  745 H= 745:00 +CPU= 55.675
  19:36:33 STEP  746 H= 746:00 +CPU= 55.111
  19:37:56 STEP  747 H= 747:00 +CPU= 81.301
  19:38:50 STEP  748 H= 748:00 +CPU= 52.930
  19:39:46 STEP  749 H= 749:00 +CPU= 55.043
  19:41:04 STEP  750 H= 750:00 +CPU= 77.446
  19:41:59 STEP  751 H= 751:00 +CPU= 54.283
  19:42:57 STEP  752 H= 752:00 +CPU= 55.280
  19:44:21 STEP  753 H= 753:00 +CPU= 80.227
  19:45:17 STEP  754 H= 754:00 +CPU= 54.731
  19:46:16 STEP  755 H= 755:00 +CPU= 56.287
  19:47:45 STEP  756 H= 756:00 +CPU= 86.624
Moving to projects directory: /home/steve/slots/18/ICMGGhq0f+000756
Moving to projects directory: /home/steve/slots/18/ICMSHhq0f+000756
Moving to projects directory: /home/steve/slots/18/ICMUAhq0f+000756
  19:48:39 STEP  757 H= 757:00 +CPU= 53.815
  19:49:36 STEP  758 H= 758:00 +CPU= 55.695
  19:50:59 STEP  759 H= 759:00 +CPU= 82.361
  19:51:55 STEP  760 H= 760:00 +CPU= 55.226
  19:52:49 STEP  761 H= 761:00 +CPU= 53.033
  19:54:08 STEP  762 H= 762:00 +CPU= 77.918
  19:55:01 STEP  763 H= 763:00 +CPU= 52.712
  19:55:57 STEP  764 H= 764:00 +CPU= 54.925
  19:57:21 STEP  765 H= 765:00 +CPU= 82.260
  19:58:18 STEP  766 H= 766:00 +CPU= 54.868
  19:59:19 STEP  767 H= 767:00 +CPU= 60.186
  20:00:58 STEP  768 H= 768:00 +CPU= 96.690
Moving to projects directory: /home/steve/slots/18/ICMGGhq0f+000768
Moving to projects directory: /home/steve/slots/18/ICMSHhq0f+000768
Moving to projects directory: /home/steve/slots/18/ICMUAhq0f+000768
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMGGhq0f+000744
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMSHhq0f+000744
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMUAhq0f+000744
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMGGhq0f+000756
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMSHhq0f+000756
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMUAhq0f+000756
Zipping up the intermediate file: /home/steve/projects/climateprediction.net/oifs_43r3_ps_0771_2005050100_123_974_12191415_0_r671616468_31.zip
Uploading the intermediate file: upload_file_31.zip
  20:01:53 STEP  769 H= 769:00 +CPU= 53.658
  20:02:52 STEP  770 H= 770:00 +CPU= 56.691
  20:04:19 STEP  771 H= 771:00 +CPU= 84.057
  20:05:16 STEP  772 H= 772:00 +CPU= 54.385
  20:06:14 STEP  773 H= 773:00 +CPU= 55.987
  20:07:40 STEP  774 H= 774:00 +CPU= 82.273
  20:08:40 STEP  775 H= 775:00 +CPU= 56.196
  20:09:40 STEP  776 H= 776:00 +CPU= 56.345
  20:11:05 STEP  777 H= 777:00 +CPU= 78.472
  20:12:07 STEP  778 H= 778:00 +CPU= 55.800
  20:13:07 STEP  779 H= 779:00 +CPU= 55.492
  20:14:43 STEP  780 H= 780:00 +CPU= 87.625
Moving to projects directory: /home/steve/slots/18/ICMGGhq0f+000780
Moving to projects directory: /home/steve/slots/18/ICMSHhq0f+000780
Moving to projects directory: /home/steve/slots/18/ICMUAhq0f+000780
  20:15:42 STEP  781 H= 781:00 +CPU= 55.581
  20:16:43 STEP  782 H= 782:00 +CPU= 56.647
  20:18:08 STEP  783 H= 783:00 +CPU= 80.359
  20:19:06 STEP  784 H= 784:00 +CPU= 55.452
  20:20:03 STEP  785 H= 785:00 +CPU= 56.328
  20:21:28 STEP  786 H= 786:00 +CPU= 82.466
  20:22:26 STEP  787 H= 787:00 +CPU= 56.141
  20:23:27 STEP  788 H= 788:00 +CPU= 58.712
  20:24:49 STEP  789 H= 789:00 +CPU= 79.755
  20:25:49 STEP  790 H= 790:00 +CPU= 57.968
  20:26:47 STEP  791 H= 791:00 +CPU= 56.958
  20:28:24 STEP  792 H= 792:00 +CPU= 93.498
Moving to projects directory: /home/steve/slots/18/ICMGGhq0f+000792
Moving to projects directory: /home/steve/slots/18/ICMSHhq0f+000792
Moving to projects directory: /home/steve/slots/18/ICMUAhq0f+000792
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMGGhq0f+000768
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMSHhq0f+000768
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMUAhq0f+000768
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMGGhq0f+000780
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMSHhq0f+000780
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMUAhq0f+000780
Zipping up the intermediate file: /home/steve/projects/climateprediction.net/oifs_43r3_ps_0771_2005050100_123_974_12191415_0_r671616468_32.zip
Uploading the intermediate file: upload_file_32.zip
  20:29:21 STEP  793 H= 793:00 +CPU= 54.836
  20:30:23 STEP  794 H= 794:00 +CPU= 58.255
  20:31:44 STEP  795 H= 795:00 +CPU= 80.018
  20:32:46 STEP  796 H= 796:00 +CPU= 60.866
  20:33:43 STEP  797 H= 797:00 +CPU= 55.981
  20:35:04 STEP  798 H= 798:00 +CPU= 80.860
  20:36:04 STEP  799 H= 799:00 +CPU= 58.978
  20:37:01 STEP  800 H= 800:00 +CPU= 55.742
  20:38:22 STEP  801 H= 801:00 +CPU= 80.512
  20:39:19 STEP  802 H= 802:00 +CPU= 56.305
  20:40:20 STEP  803 H= 803:00 +CPU= 58.095
  20:41:55 STEP  804 H= 804:00 +CPU= 88.262
Moving to projects directory: /home/steve/slots/18/ICMGGhq0f+000804
Moving to projects directory: /home/steve/slots/18/ICMSHhq0f+000804
Moving to projects directory: /home/steve/slots/18/ICMUAhq0f+000804
  20:42:55 STEP  805 H= 805:00 +CPU= 56.176
  20:43:58 STEP  806 H= 806:00 +CPU= 56.806
  20:45:26 STEP  807 H= 807:00 +CPU= 81.720
  20:46:24 STEP  808 H= 808:00 +CPU= 57.102
  20:47:18 STEP  809 H= 809:00 +CPU= 52.639
  20:48:41 STEP  810 H= 810:00 +CPU= 81.846
  20:49:41 STEP  811 H= 811:00 +CPU= 58.014
  20:50:33 STEP  812 H= 812:00 +CPU= 51.102
  20:51:50 STEP  813 H= 813:00 +CPU= 73.789
  20:52:44 STEP  814 H= 814:00 +CPU= 51.937
  20:53:39 STEP  815 H= 815:00 +CPU= 51.757
  20:55:12 STEP  816 H= 816:00 +CPU= 88.998
Moving to projects directory: /home/steve/slots/18/ICMGGhq0f+000816
Moving to projects directory: /home/steve/slots/18/ICMSHhq0f+000816
Moving to projects directory: /home/steve/slots/18/ICMUAhq0f+000816
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMGGhq0f+000792
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMSHhq0f+000792
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMUAhq0f+000792
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMGGhq0f+000804
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMSHhq0f+000804
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMUAhq0f+000804
Zipping up the intermediate file: /home/steve/projects/climateprediction.net/oifs_43r3_ps_0771_2005050100_123_974_12191415_0_r671616468_33.zip
Uploading the intermediate file: upload_file_33.zip
Uploading trickle at timestep: 2937600
  20:56:03 STEP  817 H= 817:00 +CPU= 49.518
  20:56:56 STEP  818 H= 818:00 +CPU= 52.310
  20:58:15 STEP  819 H= 819:00 +CPU= 77.130
  20:59:05 STEP  820 H= 820:00 +CPU= 49.190
  20:59:57 STEP  821 H= 821:00 +CPU= 51.813
  21:01:17 STEP  822 H= 822:00 +CPU= 78.738
  21:02:11 STEP  823 H= 823:00 +CPU= 53.205
  21:03:03 STEP  824 H= 824:00 +CPU= 51.364
  21:04:20 STEP  825 H= 825:00 +CPU= 74.564
  21:05:15 STEP  826 H= 826:00 +CPU= 52.719
  21:06:17 STEP  827 H= 827:00 +CPU= 48.212
  21:07:46 STEP  828 H= 828:00 +CPU= 84.560
Moving to projects directory: /home/steve/slots/18/ICMGGhq0f+000828
Moving to projects directory: /home/steve/slots/18/ICMSHhq0f+000828
Moving to projects directory: /home/steve/slots/18/ICMUAhq0f+000828
  21:08:41 STEP  829 H= 829:00 +CPU= 52.593
  21:09:38 STEP  830 H= 830:00 +CPU= 52.954
  21:10:54 STEP  831 H= 831:00 +CPU= 74.200
  21:11:49 STEP  832 H= 832:00 +CPU= 52.930
  21:12:42 STEP  833 H= 833:00 +CPU= 51.747
  21:13:59 STEP  834 H= 834:00 +CPU= 75.453
  21:14:51 STEP  835 H= 835:00 +CPU= 50.584
  21:15:44 STEP  836 H= 836:00 +CPU= 51.634
  21:17:04 STEP  837 H= 837:00 +CPU= 76.362
  21:17:58 STEP  838 H= 838:00 +CPU= 52.652
  21:18:56 STEP  839 H= 839:00 +CPU= 53.845
  21:20:33 STEP  840 H= 840:00 +CPU= 91.673
Moving to projects directory: /home/steve/slots/18/ICMGGhq0f+000840
Moving to projects directory: /home/steve/slots/18/ICMSHhq0f+000840
Moving to projects directory: /home/steve/slots/18/ICMUAhq0f+000840
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMGGhq0f+000816
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMSHhq0f+000816
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMUAhq0f+000816
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMGGhq0f+000828
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMSHhq0f+000828
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMUAhq0f+000828
Zipping up the intermediate file: /home/steve/projects/climateprediction.net/oifs_43r3_ps_0771_2005050100_123_974_12191415_0_r671616468_34.zip
Uploading the intermediate file: upload_file_34.zip
  21:21:26 STEP  841 H= 841:00 +CPU= 51.825
  21:22:20 STEP  842 H= 842:00 +CPU= 53.499
  21:23:41 STEP  843 H= 843:00 +CPU= 78.963
  21:24:31 STEP  844 H= 844:00 +CPU= 49.888
  21:25:25 STEP  845 H= 845:00 +CPU= 52.664
  21:26:44 STEP  846 H= 846:00 +CPU= 77.866
  21:27:38 STEP  847 H= 847:00 +CPU= 52.399
  21:28:33 STEP  848 H= 848:00 +CPU= 52.233
  21:29:51 STEP  849 H= 849:00 +CPU= 74.887
  21:30:44 STEP  850 H= 850:00 +CPU= 51.257
  21:31:35 STEP  851 H= 851:00 +CPU= 49.983
  21:33:02 STEP  852 H= 852:00 +CPU= 85.981
Moving to projects directory: /home/steve/slots/18/ICMGGhq0f+000852
Moving to projects directory: /home/steve/slots/18/ICMSHhq0f+000852
Moving to projects directory: /home/steve/slots/18/ICMUAhq0f+000852
  21:33:51 STEP  853 H= 853:00 +CPU= 48.401
  21:34:44 STEP  854 H= 854:00 +CPU= 52.617
  21:36:03 STEP  855 H= 855:00 +CPU= 77.510
  21:36:55 STEP  856 H= 856:00 +CPU= 51.738
  21:37:51 STEP  857 H= 857:00 +CPU= 51.966
  21:39:13 STEP  858 H= 858:00 +CPU= 76.585
  21:40:05 STEP  859 H= 859:00 +CPU= 50.128
  21:40:58 STEP  860 H= 860:00 +CPU= 51.998
  21:42:12 STEP  861 H= 861:00 +CPU= 73.364
  21:43:04 STEP  862 H= 862:00 +CPU= 50.348
  21:43:58 STEP  863 H= 863:00 +CPU= 52.562
  21:45:29 STEP  864 H= 864:00 +CPU= 89.208
Moving to projects directory: /home/steve/slots/18/ICMGGhq0f+000864
Moving to projects directory: /home/steve/slots/18/ICMSHhq0f+000864
Moving to projects directory: /home/steve/slots/18/ICMUAhq0f+000864
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMGGhq0f+000840
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMSHhq0f+000840
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMUAhq0f+000840
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMGGhq0f+000852
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMSHhq0f+000852
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMUAhq0f+000852
Zipping up the intermediate file: /home/steve/projects/climateprediction.net/oifs_43r3_ps_0771_2005050100_123_974_12191415_0_r671616468_35.zip
Uploading the intermediate file: upload_file_35.zip
  21:46:24 STEP  865 H= 865:00 +CPU= 54.541
  21:47:19 STEP  866 H= 866:00 +CPU= 54.554
  21:48:34 STEP  867 H= 867:00 +CPU= 74.124
  21:49:28 STEP  868 H= 868:00 +CPU= 52.374
  21:50:18 STEP  869 H= 869:00 +CPU= 49.695
  21:51:35 STEP  870 H= 870:00 +CPU= 76.074
  21:52:29 STEP  871 H= 871:00 +CPU= 53.624
  21:53:24 STEP  872 H= 872:00 +CPU= 53.399
  21:54:40 STEP  873 H= 873:00 +CPU= 75.280
  21:55:34 STEP  874 H= 874:00 +CPU= 53.275
  21:56:26 STEP  875 H= 875:00 +CPU= 51.042
  21:57:50 STEP  876 H= 876:00 +CPU= 82.617
Moving to projects directory: /home/steve/slots/18/ICMGGhq0f+000876
Moving to projects directory: /home/steve/slots/18/ICMSHhq0f+000876
Moving to projects directory: /home/steve/slots/18/ICMUAhq0f+000876
  21:58:47 STEP  877 H= 877:00 +CPU= 55.418
  21:59:42 STEP  878 H= 878:00 +CPU= 54.080
  22:00:59 STEP  879 H= 879:00 +CPU= 76.904
  22:01:53 STEP  880 H= 880:00 +CPU= 53.154
  22:02:47 STEP  881 H= 881:00 +CPU= 53.049
  22:04:05 STEP  882 H= 882:00 +CPU= 76.428
  22:04:58 STEP  883 H= 883:00 +CPU= 52.705
  22:05:54 STEP  884 H= 884:00 +CPU= 55.144
  22:07:12 STEP  885 H= 885:00 +CPU= 76.710
  22:08:04 STEP  886 H= 886:00 +CPU= 50.951
  22:08:56 STEP  887 H= 887:00 +CPU= 50.997
  22:10:20 STEP  888 H= 888:00 +CPU= 82.781
Moving to projects directory: /home/steve/slots/18/ICMGGhq0f+000888
Moving to projects directory: /home/steve/slots/18/ICMSHhq0f+000888
Moving to projects directory: /home/steve/slots/18/ICMUAhq0f+000888
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMGGhq0f+000864
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMSHhq0f+000864
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMUAhq0f+000864
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMGGhq0f+000876
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMSHhq0f+000876
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMUAhq0f+000876
Zipping up the intermediate file: /home/steve/projects/climateprediction.net/oifs_43r3_ps_0771_2005050100_123_974_12191415_0_r671616468_36.zip
Uploading the intermediate file: upload_file_36.zip
  22:11:09 STEP  889 H= 889:00 +CPU= 48.525
  22:12:03 STEP  890 H= 890:00 +CPU= 53.747
  22:13:20 STEP  891 H= 891:00 +CPU= 75.378
  22:14:10 STEP  892 H= 892:00 +CPU= 49.710
  22:15:02 STEP  893 H= 893:00 +CPU= 51.774
  22:16:23 STEP  894 H= 894:00 +CPU= 80.092
  22:17:16 STEP  895 H= 895:00 +CPU= 51.834
  22:18:10 STEP  896 H= 896:00 +CPU= 52.876
  22:19:28 STEP  897 H= 897:00 +CPU= 76.666
  22:20:22 STEP  898 H= 898:00 +CPU= 53.125
  22:21:17 STEP  899 H= 899:00 +CPU= 54.423
  22:22:40 STEP  900 H= 900:00 +CPU= 81.671
Moving to projects directory: /home/steve/slots/18/ICMGGhq0f+000900
Moving to projects directory: /home/steve/slots/18/ICMSHhq0f+000900
Moving to projects directory: /home/steve/slots/18/ICMUAhq0f+000900
  22:23:32 STEP  901 H= 901:00 +CPU= 50.772
  22:24:26 STEP  902 H= 902:00 +CPU= 53.860
  22:25:47 STEP  903 H= 903:00 +CPU= 78.943
  22:26:42 STEP  904 H= 904:00 +CPU= 54.822
  22:27:35 STEP  905 H= 905:00 +CPU= 52.414
  22:28:54 STEP  906 H= 906:00 +CPU= 77.704
  22:29:48 STEP  907 H= 907:00 +CPU= 53.431
  22:30:41 STEP  908 H= 908:00 +CPU= 51.581
  22:32:00 STEP  909 H= 909:00 +CPU= 77.601
  22:32:52 STEP  910 H= 910:00 +CPU= 51.506
  22:33:44 STEP  911 H= 911:00 +CPU= 51.709
  22:35:18 STEP  912 H= 912:00 +CPU= 92.489
Moving to projects directory: /home/steve/slots/18/ICMGGhq0f+000912
Moving to projects directory: /home/steve/slots/18/ICMSHhq0f+000912
Moving to projects directory: /home/steve/slots/18/ICMUAhq0f+000912
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMGGhq0f+000888
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMSHhq0f+000888
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMUAhq0f+000888
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMGGhq0f+000900
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMSHhq0f+000900
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMUAhq0f+000900
Zipping up the intermediate file: /home/steve/projects/climateprediction.net/oifs_43r3_ps_0771_2005050100_123_974_12191415_0_r671616468_37.zip
Uploading the intermediate file: upload_file_37.zip
  22:36:13 STEP  913 H= 913:00 +CPU= 53.883
  22:37:05 STEP  914 H= 914:00 +CPU= 51.523
  22:38:20 STEP  915 H= 915:00 +CPU= 73.485
  22:39:14 STEP  916 H= 916:00 +CPU= 53.007
  22:40:08 STEP  917 H= 917:00 +CPU= 53.444
  22:41:28 STEP  918 H= 918:00 +CPU= 79.039
  22:42:18 STEP  919 H= 919:00 +CPU= 49.531
  22:43:11 STEP  920 H= 920:00 +CPU= 51.796
  22:44:33 STEP  921 H= 921:00 +CPU= 81.017
  22:45:28 STEP  922 H= 922:00 +CPU= 53.637
  22:46:18 STEP  923 H= 923:00 +CPU= 50.065
  22:47:45 STEP  924 H= 924:00 +CPU= 85.501
Moving to projects directory: /home/steve/slots/18/ICMGGhq0f+000924
Moving to projects directory: /home/steve/slots/18/ICMSHhq0f+000924
Moving to projects directory: /home/steve/slots/18/ICMUAhq0f+000924
  22:48:36 STEP  925 H= 925:00 +CPU= 50.534
  22:49:31 STEP  926 H= 926:00 +CPU= 53.463
  22:50:54 STEP  927 H= 927:00 +CPU= 82.128
  22:51:45 STEP  928 H= 928:00 +CPU= 50.524
  22:52:37 STEP  929 H= 929:00 +CPU= 51.565
  22:53:54 STEP  930 H= 930:00 +CPU= 75.234
  22:54:46 STEP  931 H= 931:00 +CPU= 51.568
  22:55:39 STEP  932 H= 932:00 +CPU= 52.891
  22:56:57 STEP  933 H= 933:00 +CPU= 76.639
  22:57:47 STEP  934 H= 934:00 +CPU= 49.586
  22:58:42 STEP  935 H= 935:00 +CPU= 54.341
  23:00:15 STEP  936 H= 936:00 +CPU= 90.946
Moving to projects directory: /home/steve/slots/18/ICMGGhq0f+000936
Moving to projects directory: /home/steve/slots/18/ICMSHhq0f+000936
Moving to projects directory: /home/steve/slots/18/ICMUAhq0f+000936
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMGGhq0f+000912
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMSHhq0f+000912
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMUAhq0f+000912
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMGGhq0f+000924
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMSHhq0f+000924
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMUAhq0f+000924
Zipping up the intermediate file: /home/steve/projects/climateprediction.net/oifs_43r3_ps_0771_2005050100_123_974_12191415_0_r671616468_38.zip
Uploading the intermediate file: upload_file_38.zip
  23:01:04 STEP  937 H= 937:00 +CPU= 48.890
  23:01:58 STEP  938 H= 938:00 +CPU= 53.011
  23:03:15 STEP  939 H= 939:00 +CPU= 75.522
  23:04:06 STEP  940 H= 940:00 +CPU= 50.260
  23:05:00 STEP  941 H= 941:00 +CPU= 53.249
  23:06:18 STEP  942 H= 942:00 +CPU= 76.950
  23:07:07 STEP  943 H= 943:00 +CPU= 48.350
  23:08:07 STEP  944 H= 944:00 +CPU= 59.380
  23:09:25 STEP  945 H= 945:00 +CPU= 76.260
  23:10:18 STEP  946 H= 946:00 +CPU= 52.883
  23:11:11 STEP  947 H= 947:00 +CPU= 51.756
  23:12:37 STEP  948 H= 948:00 +CPU= 84.559
Moving to projects directory: /home/steve/slots/18/ICMGGhq0f+000948
Moving to projects directory: /home/steve/slots/18/ICMSHhq0f+000948
Moving to projects directory: /home/steve/slots/18/ICMUAhq0f+000948
  23:13:29 STEP  949 H= 949:00 +CPU= 50.777
  23:14:25 STEP  950 H= 950:00 +CPU= 55.659
  23:15:44 STEP  951 H= 951:00 +CPU= 77.446
  23:16:39 STEP  952 H= 952:00 +CPU= 54.533
  23:17:32 STEP  953 H= 953:00 +CPU= 51.729
  23:18:57 STEP  954 H= 954:00 +CPU= 83.244
  23:19:48 STEP  955 H= 955:00 +CPU= 50.824
  23:20:44 STEP  956 H= 956:00 +CPU= 54.439
  23:22:06 STEP  957 H= 957:00 +CPU= 80.718
  23:23:02 STEP  958 H= 958:00 +CPU= 55.523
  23:23:57 STEP  959 H= 959:00 +CPU= 54.004
  23:25:30 STEP  960 H= 960:00 +CPU= 92.527
Moving to projects directory: /home/steve/slots/18/ICMGGhq0f+000960
Moving to projects directory: /home/steve/slots/18/ICMSHhq0f+000960
Moving to projects directory: /home/steve/slots/18/ICMUAhq0f+000960
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMGGhq0f+000936
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMSHhq0f+000936
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMUAhq0f+000936
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMGGhq0f+000948
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMSHhq0f+000948
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMUAhq0f+000948
Zipping up the intermediate file: /home/steve/projects/climateprediction.net/oifs_43r3_ps_0771_2005050100_123_974_12191415_0_r671616468_39.zip
Uploading the intermediate file: upload_file_39.zip
  23:26:25 STEP  961 H= 961:00 +CPU= 53.097
  23:27:20 STEP  962 H= 962:00 +CPU= 54.512
  23:28:34 STEP  963 H= 963:00 +CPU= 73.313
  23:29:32 STEP  964 H= 964:00 +CPU= 57.266
  23:30:21 STEP  965 H= 965:00 +CPU= 48.737
  23:31:42 STEP  966 H= 966:00 +CPU= 79.401
  23:32:32 STEP  967 H= 967:00 +CPU= 49.769
  23:33:27 STEP  968 H= 968:00 +CPU= 54.422
  23:34:45 STEP  969 H= 969:00 +CPU= 76.422
  23:35:38 STEP  970 H= 970:00 +CPU= 52.084
  23:36:33 STEP  971 H= 971:00 +CPU= 54.860
  23:37:59 STEP  972 H= 972:00 +CPU= 84.310
Moving to projects directory: /home/steve/slots/18/ICMGGhq0f+000972
Moving to projects directory: /home/steve/slots/18/ICMSHhq0f+000972
Moving to projects directory: /home/steve/slots/18/ICMUAhq0f+000972
  23:38:53 STEP  973 H= 973:00 +CPU= 53.694
  23:39:46 STEP  974 H= 974:00 +CPU= 52.211
  23:41:03 STEP  975 H= 975:00 +CPU= 76.053
  23:41:59 STEP  976 H= 976:00 +CPU= 54.918
  23:42:47 STEP  977 H= 977:00 +CPU= 47.150
  23:44:05 STEP  978 H= 978:00 +CPU= 77.070
  23:44:55 STEP  979 H= 979:00 +CPU= 49.651
  23:45:47 STEP  980 H= 980:00 +CPU= 50.431
  23:46:58 STEP  981 H= 981:00 +CPU= 70.308
  23:47:56 STEP  982 H= 982:00 +CPU= 57.259
  23:48:48 STEP  983 H= 983:00 +CPU= 51.187
  23:50:15 STEP  984 H= 984:00 +CPU= 85.752
Moving to projects directory: /home/steve/slots/18/ICMGGhq0f+000984
Moving to projects directory: /home/steve/slots/18/ICMSHhq0f+000984
Moving to projects directory: /home/steve/slots/18/ICMUAhq0f+000984
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMGGhq0f+000960
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMSHhq0f+000960
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMUAhq0f+000960
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMGGhq0f+000972
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMSHhq0f+000972
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMUAhq0f+000972
Zipping up the intermediate file: /home/steve/projects/climateprediction.net/oifs_43r3_ps_0771_2005050100_123_974_12191415_0_r671616468_40.zip
Uploading the intermediate file: upload_file_40.zip
  23:51:07 STEP  985 H= 985:00 +CPU= 51.496
  23:52:03 STEP  986 H= 986:00 +CPU= 54.762
  23:53:24 STEP  987 H= 987:00 +CPU= 79.723
  23:54:18 STEP  988 H= 988:00 +CPU= 53.157
  23:55:10 STEP  989 H= 989:00 +CPU= 51.317
  23:56:29 STEP  990 H= 990:00 +CPU= 78.126
  23:57:20 STEP  991 H= 991:00 +CPU= 50.341
  23:58:10 STEP  992 H= 992:00 +CPU= 49.328
  23:59:27 STEP  993 H= 993:00 +CPU= 75.957
  00:00:22 STEP  994 H= 994:00 +CPU= 54.108
  00:01:14 STEP  995 H= 995:00 +CPU= 51.396
  00:02:40 STEP  996 H= 996:00 +CPU= 85.004
Moving to projects directory: /home/steve/slots/18/ICMGGhq0f+000996
Moving to projects directory: /home/steve/slots/18/ICMSHhq0f+000996
Moving to projects directory: /home/steve/slots/18/ICMUAhq0f+000996
  00:03:33 STEP  997 H= 997:00 +CPU= 51.859
  00:04:25 STEP  998 H= 998:00 +CPU= 51.236
  00:05:41 STEP  999 H= 999:00 +CPU= 74.831
  00:06:33 STEP 1000 H=1000:00 +CPU= 51.510
  00:07:23 STEP 1001 H=1001:00 +CPU= 49.134
  00:08:40 STEP 1002 H=1002:00 +CPU= 76.543
  00:09:31 STEP 1003 H=1003:00 +CPU= 50.088
  00:10:23 STEP 1004 H=1004:00 +CPU= 50.736
  00:11:38 STEP 1005 H=1005:00 +CPU= 73.959
  00:12:34 STEP 1006 H=1006:00 +CPU= 55.106
  00:13:29 STEP 1007 H=1007:00 +CPU= 54.563
  00:14:59 STEP 1008 H=1008:00 +CPU= 88.958
Moving to projects directory: /home/steve/slots/18/ICMGGhq0f+001008
Moving to projects directory: /home/steve/slots/18/ICMSHhq0f+001008
Moving to projects directory: /home/steve/slots/18/ICMUAhq0f+001008
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMGGhq0f+000984
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMSHhq0f+000984
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMUAhq0f+000984
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMGGhq0f+000996
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMSHhq0f+000996
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMUAhq0f+000996
Zipping up the intermediate file: /home/steve/projects/climateprediction.net/oifs_43r3_ps_0771_2005050100_123_974_12191415_0_r671616468_41.zip
Uploading the intermediate file: upload_file_41.zip
  00:15:51 STEP 1009 H=1009:00 +CPU= 50.509
  00:16:43 STEP 1010 H=1010:00 +CPU= 51.993
  00:17:58 STEP 1011 H=1011:00 +CPU= 73.206
  00:18:50 STEP 1012 H=1012:00 +CPU= 51.110
  00:19:43 STEP 1013 H=1013:00 +CPU= 52.625
  00:21:01 STEP 1014 H=1014:00 +CPU= 76.949
  00:21:51 STEP 1015 H=1015:00 +CPU= 49.169
  00:22:44 STEP 1016 H=1016:00 +CPU= 52.604
  00:24:02 STEP 1017 H=1017:00 +CPU= 76.994
  00:24:57 STEP 1018 H=1018:00 +CPU= 54.325
  00:25:50 STEP 1019 H=1019:00 +CPU= 52.086
  00:27:17 STEP 1020 H=1020:00 +CPU= 85.193
Moving to projects directory: /home/steve/slots/18/ICMGGhq0f+001020
Moving to projects directory: /home/steve/slots/18/ICMSHhq0f+001020
Moving to projects directory: /home/steve/slots/18/ICMUAhq0f+001020
  00:28:07 STEP 1021 H=1021:00 +CPU= 49.224
  00:29:00 STEP 1022 H=1022:00 +CPU= 52.866
  00:30:21 STEP 1023 H=1023:00 +CPU= 79.939
  00:31:13 STEP 1024 H=1024:00 +CPU= 51.377
  00:32:09 STEP 1025 H=1025:00 +CPU= 54.818
  00:33:28 STEP 1026 H=1026:00 +CPU= 78.330
  00:34:19 STEP 1027 H=1027:00 +CPU= 49.921
  00:35:16 STEP 1028 H=1028:00 +CPU= 56.081
  00:36:30 STEP 1029 H=1029:00 +CPU= 73.550
  00:37:23 STEP 1030 H=1030:00 +CPU= 51.826
  00:38:18 STEP 1031 H=1031:00 +CPU= 53.746
  00:39:45 STEP 1032 H=1032:00 +CPU= 86.203
Moving to projects directory: /home/steve/slots/18/ICMGGhq0f+001032
Moving to projects directory: /home/steve/slots/18/ICMSHhq0f+001032
Moving to projects directory: /home/steve/slots/18/ICMUAhq0f+001032
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMGGhq0f+001008
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMSHhq0f+001008
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMUAhq0f+001008
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMGGhq0f+001020
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMSHhq0f+001020
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMUAhq0f+001020
Zipping up the intermediate file: /home/steve/projects/climateprediction.net/oifs_43r3_ps_0771_2005050100_123_974_12191415_0_r671616468_42.zip
Uploading the intermediate file: upload_file_42.zip
  00:40:39 STEP 1033 H=1033:00 +CPU= 53.037
  00:41:31 STEP 1034 H=1034:00 +CPU= 51.268
  00:42:45 STEP 1035 H=1035:00 +CPU= 72.419
  00:43:42 STEP 1036 H=1036:00 +CPU= 56.713
  00:44:37 STEP 1037 H=1037:00 +CPU= 53.631
  00:45:55 STEP 1038 H=1038:00 +CPU= 77.497
  00:46:53 STEP 1039 H=1039:00 +CPU= 56.802
  00:47:44 STEP 1040 H=1040:00 +CPU= 50.057
  00:48:57 STEP 1041 H=1041:00 +CPU= 72.627
  00:49:50 STEP 1042 H=1042:00 +CPU= 52.174
  00:50:45 STEP 1043 H=1043:00 +CPU= 53.809
  00:52:07 STEP 1044 H=1044:00 +CPU= 80.920
Moving to projects directory: /home/steve/slots/18/ICMGGhq0f+001044
Moving to projects directory: /home/steve/slots/18/ICMSHhq0f+001044
Moving to projects directory: /home/steve/slots/18/ICMUAhq0f+001044
  00:52:59 STEP 1045 H=1045:00 +CPU= 52.077
  00:53:53 STEP 1046 H=1046:00 +CPU= 52.809
  00:55:12 STEP 1047 H=1047:00 +CPU= 78.603
  00:56:06 STEP 1048 H=1048:00 +CPU= 52.962
  00:56:58 STEP 1049 H=1049:00 +CPU= 51.135
  00:58:10 STEP 1050 H=1050:00 +CPU= 71.684
  00:59:02 STEP 1051 H=1051:00 +CPU= 50.752
  00:59:56 STEP 1052 H=1052:00 +CPU= 52.801
  01:01:12 STEP 1053 H=1053:00 +CPU= 74.811
  01:02:08 STEP 1054 H=1054:00 +CPU= 55.511
  01:03:00 STEP 1055 H=1055:00 +CPU= 51.123
  01:04:29 STEP 1056 H=1056:00 +CPU= 87.735
Moving to projects directory: /home/steve/slots/18/ICMGGhq0f+001056
Moving to projects directory: /home/steve/slots/18/ICMSHhq0f+001056
Moving to projects directory: /home/steve/slots/18/ICMUAhq0f+001056
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMGGhq0f+001032
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMSHhq0f+001032
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMUAhq0f+001032
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMGGhq0f+001044
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMSHhq0f+001044
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMUAhq0f+001044
Zipping up the intermediate file: /home/steve/projects/climateprediction.net/oifs_43r3_ps_0771_2005050100_123_974_12191415_0_r671616468_43.zip
Uploading the intermediate file: upload_file_43.zip
Uploading trickle at timestep: 3801600
  01:05:26 STEP 1057 H=1057:00 +CPU= 56.214
  01:06:19 STEP 1058 H=1058:00 +CPU= 52.219
  01:07:31 STEP 1059 H=1059:00 +CPU= 71.166
  01:08:25 STEP 1060 H=1060:00 +CPU= 52.971
  01:09:16 STEP 1061 H=1061:00 +CPU= 50.529
  01:10:31 STEP 1062 H=1062:00 +CPU= 74.161
  01:11:22 STEP 1063 H=1063:00 +CPU= 50.786
  01:12:15 STEP 1064 H=1064:00 +CPU= 52.359
  01:13:29 STEP 1065 H=1065:00 +CPU= 72.959
  01:14:21 STEP 1066 H=1066:00 +CPU= 50.759
  01:15:14 STEP 1067 H=1067:00 +CPU= 52.239
  01:16:39 STEP 1068 H=1068:00 +CPU= 84.646
Moving to projects directory: /home/steve/slots/18/ICMGGhq0f+001068
Moving to projects directory: /home/steve/slots/18/ICMSHhq0f+001068
Moving to projects directory: /home/steve/slots/18/ICMUAhq0f+001068
  01:17:31 STEP 1069 H=1069:00 +CPU= 50.912
  01:18:22 STEP 1070 H=1070:00 +CPU= 50.547
  01:19:42 STEP 1071 H=1071:00 +CPU= 78.787
  01:20:34 STEP 1072 H=1072:00 +CPU= 51.640
  01:21:27 STEP 1073 H=1073:00 +CPU= 52.309
  01:22:45 STEP 1074 H=1074:00 +CPU= 77.140
  01:23:39 STEP 1075 H=1075:00 +CPU= 53.308
  01:24:33 STEP 1076 H=1076:00 +CPU= 52.804
  01:25:49 STEP 1077 H=1077:00 +CPU= 74.798
  01:26:41 STEP 1078 H=1078:00 +CPU= 51.183
  01:27:34 STEP 1079 H=1079:00 +CPU= 52.144
  01:29:05 STEP 1080 H=1080:00 +CPU= 89.678
Moving to projects directory: /home/steve/slots/18/ICMGGhq0f+001080
Moving to projects directory: /home/steve/slots/18/ICMSHhq0f+001080
Moving to projects directory: /home/steve/slots/18/ICMUAhq0f+001080
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMGGhq0f+001056
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMSHhq0f+001056
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMUAhq0f+001056
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMGGhq0f+001068
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMSHhq0f+001068
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMUAhq0f+001068
Zipping up the intermediate file: /home/steve/projects/climateprediction.net/oifs_43r3_ps_0771_2005050100_123_974_12191415_0_r671616468_44.zip
Uploading the intermediate file: upload_file_44.zip
  01:29:56 STEP 1081 H=1081:00 +CPU= 50.518
  01:30:51 STEP 1082 H=1082:00 +CPU= 53.655
  01:32:07 STEP 1083 H=1083:00 +CPU= 75.670
  01:33:04 STEP 1084 H=1084:00 +CPU= 55.605
  01:33:54 STEP 1085 H=1085:00 +CPU= 49.947
  01:35:14 STEP 1086 H=1086:00 +CPU= 78.542
  01:36:06 STEP 1087 H=1087:00 +CPU= 50.876
  01:37:01 STEP 1088 H=1088:00 +CPU= 54.270
  01:38:17 STEP 1089 H=1089:00 +CPU= 75.183
  01:39:09 STEP 1090 H=1090:00 +CPU= 50.523
  01:40:02 STEP 1091 H=1091:00 +CPU= 52.010
  01:41:25 STEP 1092 H=1092:00 +CPU= 81.971
Moving to projects directory: /home/steve/slots/18/ICMGGhq0f+001092
Moving to projects directory: /home/steve/slots/18/ICMSHhq0f+001092
Moving to projects directory: /home/steve/slots/18/ICMUAhq0f+001092
  01:42:15 STEP 1093 H=1093:00 +CPU= 49.245
  01:43:08 STEP 1094 H=1094:00 +CPU= 52.811
  01:44:24 STEP 1095 H=1095:00 +CPU= 74.714
  01:45:14 STEP 1096 H=1096:00 +CPU= 49.089
  01:46:09 STEP 1097 H=1097:00 +CPU= 54.581
  01:47:24 STEP 1098 H=1098:00 +CPU= 73.580
  01:48:13 STEP 1099 H=1099:00 +CPU= 48.874
  01:49:07 STEP 1100 H=1100:00 +CPU= 53.236
  01:50:23 STEP 1101 H=1101:00 +CPU= 74.664
  01:51:13 STEP 1102 H=1102:00 +CPU= 48.993
  01:52:06 STEP 1103 H=1103:00 +CPU= 51.995
  01:53:40 STEP 1104 H=1104:00 +CPU= 92.843
Moving to projects directory: /home/steve/slots/18/ICMGGhq0f+001104
Moving to projects directory: /home/steve/slots/18/ICMSHhq0f+001104
Moving to projects directory: /home/steve/slots/18/ICMUAhq0f+001104
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMGGhq0f+001080
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMSHhq0f+001080
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMUAhq0f+001080
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMGGhq0f+001092
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMSHhq0f+001092
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMUAhq0f+001092
Zipping up the intermediate file: /home/steve/projects/climateprediction.net/oifs_43r3_ps_0771_2005050100_123_974_12191415_0_r671616468_45.zip
Uploading the intermediate file: upload_file_45.zip
  01:54:31 STEP 1105 H=1105:00 +CPU= 51.127
  01:55:24 STEP 1106 H=1106:00 +CPU= 52.362
  01:56:44 STEP 1107 H=1107:00 +CPU= 78.462
  01:57:35 STEP 1108 H=1108:00 +CPU= 49.737
  01:58:26 STEP 1109 H=1109:00 +CPU= 50.832
  01:59:48 STEP 1110 H=1110:00 +CPU= 81.228
  02:00:39 STEP 1111 H=1111:00 +CPU= 50.042
  02:01:33 STEP 1112 H=1112:00 +CPU= 53.396
  02:02:53 STEP 1113 H=1113:00 +CPU= 79.145
  02:03:43 STEP 1114 H=1114:00 +CPU= 49.551
  02:04:36 STEP 1115 H=1115:00 +CPU= 52.095
  02:06:00 STEP 1116 H=1116:00 +CPU= 82.818
Moving to projects directory: /home/steve/slots/18/ICMGGhq0f+001116
Moving to projects directory: /home/steve/slots/18/ICMSHhq0f+001116
Moving to projects directory: /home/steve/slots/18/ICMUAhq0f+001116
  02:06:53 STEP 1117 H=1117:00 +CPU= 51.878
  02:07:46 STEP 1118 H=1118:00 +CPU= 52.710
  02:09:03 STEP 1119 H=1119:00 +CPU= 76.151
  02:09:54 STEP 1120 H=1120:00 +CPU= 50.347
  02:10:47 STEP 1121 H=1121:00 +CPU= 51.475
  02:12:05 STEP 1122 H=1122:00 +CPU= 77.542
  02:12:58 STEP 1123 H=1123:00 +CPU= 52.268
  02:13:52 STEP 1124 H=1124:00 +CPU= 53.054
  02:15:06 STEP 1125 H=1125:00 +CPU= 72.789
  02:16:01 STEP 1126 H=1126:00 +CPU= 54.057
  02:16:55 STEP 1127 H=1127:00 +CPU= 53.721
  02:18:29 STEP 1128 H=1128:00 +CPU= 92.436
Moving to projects directory: /home/steve/slots/18/ICMGGhq0f+001128
Moving to projects directory: /home/steve/slots/18/ICMSHhq0f+001128
Moving to projects directory: /home/steve/slots/18/ICMUAhq0f+001128
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMGGhq0f+001104
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMSHhq0f+001104
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMUAhq0f+001104
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMGGhq0f+001116
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMSHhq0f+001116
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMUAhq0f+001116
Zipping up the intermediate file: /home/steve/projects/climateprediction.net/oifs_43r3_ps_0771_2005050100_123_974_12191415_0_r671616468_46.zip
Uploading the intermediate file: upload_file_46.zip
  02:19:21 STEP 1129 H=1129:00 +CPU= 51.159
  02:20:16 STEP 1130 H=1130:00 +CPU= 53.751
  02:21:31 STEP 1131 H=1131:00 +CPU= 74.271
  02:22:23 STEP 1132 H=1132:00 +CPU= 51.205
  02:23:13 STEP 1133 H=1133:00 +CPU= 49.264
  02:24:32 STEP 1134 H=1134:00 +CPU= 78.385
  02:25:23 STEP 1135 H=1135:00 +CPU= 49.936
  02:26:19 STEP 1136 H=1136:00 +CPU= 55.060
  02:27:34 STEP 1137 H=1137:00 +CPU= 74.699
  02:28:30 STEP 1138 H=1138:00 +CPU= 54.821
  02:29:20 STEP 1139 H=1139:00 +CPU= 48.994
  02:30:43 STEP 1140 H=1140:00 +CPU= 82.283
Moving to projects directory: /home/steve/slots/18/ICMGGhq0f+001140
Moving to projects directory: /home/steve/slots/18/ICMSHhq0f+001140
Moving to projects directory: /home/steve/slots/18/ICMUAhq0f+001140
  02:31:32 STEP 1141 H=1141:00 +CPU= 48.628
  02:32:25 STEP 1142 H=1142:00 +CPU= 51.996
  02:33:39 STEP 1143 H=1143:00 +CPU= 72.980
  02:34:32 STEP 1144 H=1144:00 +CPU= 52.275
  02:35:23 STEP 1145 H=1145:00 +CPU= 50.091
  02:36:37 STEP 1146 H=1146:00 +CPU= 73.086
  02:37:28 STEP 1147 H=1147:00 +CPU= 50.051
  02:38:22 STEP 1148 H=1148:00 +CPU= 53.259
  02:39:36 STEP 1149 H=1149:00 +CPU= 73.099
  02:40:28 STEP 1150 H=1150:00 +CPU= 51.608
  02:41:19 STEP 1151 H=1151:00 +CPU= 49.973
  02:42:48 STEP 1152 H=1152:00 +CPU= 88.473
Moving to projects directory: /home/steve/slots/18/ICMGGhq0f+001152
Moving to projects directory: /home/steve/slots/18/ICMSHhq0f+001152
Moving to projects directory: /home/steve/slots/18/ICMUAhq0f+001152
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMGGhq0f+001128
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMSHhq0f+001128
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMUAhq0f+001128
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMGGhq0f+001140
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMSHhq0f+001140
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMUAhq0f+001140
Zipping up the intermediate file: /home/steve/projects/climateprediction.net/oifs_43r3_ps_0771_2005050100_123_974_12191415_0_r671616468_47.zip
Uploading the intermediate file: upload_file_47.zip
  02:43:39 STEP 1153 H=1153:00 +CPU= 50.314
  02:44:31 STEP 1154 H=1154:00 +CPU= 51.884
  02:45:51 STEP 1155 H=1155:00 +CPU= 78.373
  02:46:43 STEP 1156 H=1156:00 +CPU= 51.163
  02:47:34 STEP 1157 H=1157:00 +CPU= 50.274
  02:48:53 STEP 1158 H=1158:00 +CPU= 77.532
  02:49:43 STEP 1159 H=1159:00 +CPU= 49.423
  02:50:39 STEP 1160 H=1160:00 +CPU= 55.595
  02:51:58 STEP 1161 H=1161:00 +CPU= 78.315
  02:52:53 STEP 1162 H=1162:00 +CPU= 53.623
  02:53:43 STEP 1163 H=1163:00 +CPU= 49.526
  02:55:11 STEP 1164 H=1164:00 +CPU= 86.725
Moving to projects directory: /home/steve/slots/18/ICMGGhq0f+001164
Moving to projects directory: /home/steve/slots/18/ICMSHhq0f+001164
Moving to projects directory: /home/steve/slots/18/ICMUAhq0f+001164
  02:56:06 STEP 1165 H=1165:00 +CPU= 54.127
  02:56:58 STEP 1166 H=1166:00 +CPU= 51.916
  02:58:17 STEP 1167 H=1167:00 +CPU= 78.267
  02:59:08 STEP 1168 H=1168:00 +CPU= 50.430
  02:59:58 STEP 1169 H=1169:00 +CPU= 49.478
  03:01:16 STEP 1170 H=1170:00 +CPU= 76.509
  03:02:06 STEP 1171 H=1171:00 +CPU= 49.794
  03:02:58 STEP 1172 H=1172:00 +CPU= 50.929
  03:04:14 STEP 1173 H=1173:00 +CPU= 74.329
  03:05:06 STEP 1174 H=1174:00 +CPU= 51.866
  03:06:02 STEP 1175 H=1175:00 +CPU= 55.509
  03:07:36 STEP 1176 H=1176:00 +CPU= 92.706
Moving to projects directory: /home/steve/slots/18/ICMGGhq0f+001176
Moving to projects directory: /home/steve/slots/18/ICMSHhq0f+001176
Moving to projects directory: /home/steve/slots/18/ICMUAhq0f+001176
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMGGhq0f+001152
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMSHhq0f+001152
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMUAhq0f+001152
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMGGhq0f+001164
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMSHhq0f+001164
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMUAhq0f+001164
Zipping up the intermediate file: /home/steve/projects/climateprediction.net/oifs_43r3_ps_0771_2005050100_123_974_12191415_0_r671616468_48.zip
Uploading the intermediate file: upload_file_48.zip
  03:08:29 STEP 1177 H=1177:00 +CPU= 51.353
  03:09:26 STEP 1178 H=1178:00 +CPU= 56.422
  03:10:43 STEP 1179 H=1179:00 +CPU= 75.900
  03:11:33 STEP 1180 H=1180:00 +CPU= 49.345
  03:12:26 STEP 1181 H=1181:00 +CPU= 51.960
  03:13:44 STEP 1182 H=1182:00 +CPU= 77.240
  03:14:32 STEP 1183 H=1183:00 +CPU= 46.978
  03:15:25 STEP 1184 H=1184:00 +CPU= 52.347
  03:16:40 STEP 1185 H=1185:00 +CPU= 73.999
  03:17:31 STEP 1186 H=1186:00 +CPU= 50.563
  03:18:24 STEP 1187 H=1187:00 +CPU= 51.490
  03:19:53 STEP 1188 H=1188:00 +CPU= 88.154
Moving to projects directory: /home/steve/slots/18/ICMGGhq0f+001188
Moving to projects directory: /home/steve/slots/18/ICMSHhq0f+001188
Moving to projects directory: /home/steve/slots/18/ICMUAhq0f+001188
  03:20:48 STEP 1189 H=1189:00 +CPU= 54.494
  03:21:46 STEP 1190 H=1190:00 +CPU= 56.277
  03:23:10 STEP 1191 H=1191:00 +CPU= 83.236
  03:24:07 STEP 1192 H=1192:00 +CPU= 55.999
  03:25:03 STEP 1193 H=1193:00 +CPU= 54.512
  03:26:24 STEP 1194 H=1194:00 +CPU= 80.613
  03:27:19 STEP 1195 H=1195:00 +CPU= 54.015
  03:28:17 STEP 1196 H=1196:00 +CPU= 57.194
  03:29:36 STEP 1197 H=1197:00 +CPU= 77.196
  03:30:36 STEP 1198 H=1198:00 +CPU= 59.235
  03:31:35 STEP 1199 H=1199:00 +CPU= 58.459
  03:33:11 STEP 1200 H=1200:00 +CPU= 94.019
Moving to projects directory: /home/steve/slots/18/ICMGGhq0f+001200
Moving to projects directory: /home/steve/slots/18/ICMSHhq0f+001200
Moving to projects directory: /home/steve/slots/18/ICMUAhq0f+001200
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMGGhq0f+001176
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMSHhq0f+001176
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMUAhq0f+001176
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMGGhq0f+001188
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMSHhq0f+001188
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMUAhq0f+001188
Zipping up the intermediate file: /home/steve/projects/climateprediction.net/oifs_43r3_ps_0771_2005050100_123_974_12191415_0_r671616468_49.zip
Uploading the intermediate file: upload_file_49.zip
  03:34:08 STEP 1201 H=1201:00 +CPU= 56.520
  03:35:07 STEP 1202 H=1202:00 +CPU= 57.869
  03:36:28 STEP 1203 H=1203:00 +CPU= 79.844
  03:37:27 STEP 1204 H=1204:00 +CPU= 58.179
  03:38:28 STEP 1205 H=1205:00 +CPU= 59.250
  03:39:53 STEP 1206 H=1206:00 +CPU= 84.210
  03:40:51 STEP 1207 H=1207:00 +CPU= 57.228
  03:41:50 STEP 1208 H=1208:00 +CPU= 57.859
  03:43:11 STEP 1209 H=1209:00 +CPU= 80.441
  03:44:09 STEP 1210 H=1210:00 +CPU= 56.853
  03:45:07 STEP 1211 H=1211:00 +CPU= 57.232
  03:46:33 STEP 1212 H=1212:00 +CPU= 84.780
Moving to projects directory: /home/steve/slots/18/ICMGGhq0f+001212
Moving to projects directory: /home/steve/slots/18/ICMSHhq0f+001212
Moving to projects directory: /home/steve/slots/18/ICMUAhq0f+001212
  03:47:30 STEP 1213 H=1213:00 +CPU= 56.092
  03:48:26 STEP 1214 H=1214:00 +CPU= 55.443
  03:49:42 STEP 1215 H=1215:00 +CPU= 74.799
  03:50:40 STEP 1216 H=1216:00 +CPU= 57.259
  03:51:36 STEP 1217 H=1217:00 +CPU= 54.890
  03:53:02 STEP 1218 H=1218:00 +CPU= 85.629
  03:53:58 STEP 1219 H=1219:00 +CPU= 54.488
  03:54:57 STEP 1220 H=1220:00 +CPU= 58.096
  03:56:16 STEP 1221 H=1221:00 +CPU= 78.648
  03:57:16 STEP 1222 H=1222:00 +CPU= 58.956
  03:58:16 STEP 1223 H=1223:00 +CPU= 59.286
  03:59:52 STEP 1224 H=1224:00 +CPU= 94.602
Moving to projects directory: /home/steve/slots/18/ICMGGhq0f+001224
Moving to projects directory: /home/steve/slots/18/ICMSHhq0f+001224
Moving to projects directory: /home/steve/slots/18/ICMUAhq0f+001224
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMGGhq0f+001200
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMSHhq0f+001200
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMUAhq0f+001200
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMGGhq0f+001212
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMSHhq0f+001212
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMUAhq0f+001212
Zipping up the intermediate file: /home/steve/projects/climateprediction.net/oifs_43r3_ps_0771_2005050100_123_974_12191415_0_r671616468_50.zip
Uploading the intermediate file: upload_file_50.zip
  04:00:49 STEP 1225 H=1225:00 +CPU= 56.044
  04:01:47 STEP 1226 H=1226:00 +CPU= 56.645
  04:03:09 STEP 1227 H=1227:00 +CPU= 80.592
  04:04:07 STEP 1228 H=1228:00 +CPU= 57.206
  04:05:04 STEP 1229 H=1229:00 +CPU= 56.279
  04:06:30 STEP 1230 H=1230:00 +CPU= 85.372
  04:07:28 STEP 1231 H=1231:00 +CPU= 57.311
  04:08:29 STEP 1232 H=1232:00 +CPU= 59.848
  04:09:45 STEP 1233 H=1233:00 +CPU= 75.699
  04:10:43 STEP 1234 H=1234:00 +CPU= 56.351
  04:11:40 STEP 1235 H=1235:00 +CPU= 56.438
  04:13:13 STEP 1236 H=1236:00 +CPU= 92.131
Moving to projects directory: /home/steve/slots/18/ICMGGhq0f+001236
Moving to projects directory: /home/steve/slots/18/ICMSHhq0f+001236
Moving to projects directory: /home/steve/slots/18/ICMUAhq0f+001236
  04:14:07 STEP 1237 H=1237:00 +CPU= 53.271
  04:15:08 STEP 1238 H=1238:00 +CPU= 59.363
  04:16:30 STEP 1239 H=1239:00 +CPU= 81.529
  04:17:28 STEP 1240 H=1240:00 +CPU= 56.519
  04:18:24 STEP 1241 H=1241:00 +CPU= 55.184
  04:19:47 STEP 1242 H=1242:00 +CPU= 81.576
  04:20:44 STEP 1243 H=1243:00 +CPU= 56.548
  04:21:39 STEP 1244 H=1244:00 +CPU= 53.680
  04:23:04 STEP 1245 H=1245:00 +CPU= 84.487
  04:24:01 STEP 1246 H=1246:00 +CPU= 55.994
  04:25:00 STEP 1247 H=1247:00 +CPU= 57.239
  04:26:34 STEP 1248 H=1248:00 +CPU= 92.655
Moving to projects directory: /home/steve/slots/18/ICMGGhq0f+001248
Moving to projects directory: /home/steve/slots/18/ICMSHhq0f+001248
Moving to projects directory: /home/steve/slots/18/ICMUAhq0f+001248
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMGGhq0f+001224
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMSHhq0f+001224
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMUAhq0f+001224
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMGGhq0f+001236
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMSHhq0f+001236
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMUAhq0f+001236
Zipping up the intermediate file: /home/steve/projects/climateprediction.net/oifs_43r3_ps_0771_2005050100_123_974_12191415_0_r671616468_51.zip
Uploading the intermediate file: upload_file_51.zip
  04:27:30 STEP 1249 H=1249:00 +CPU= 54.751
  04:28:29 STEP 1250 H=1250:00 +CPU= 58.662
  04:29:48 STEP 1251 H=1251:00 +CPU= 76.908
  04:30:46 STEP 1252 H=1252:00 +CPU= 57.741
  04:31:42 STEP 1253 H=1253:00 +CPU= 54.810
  04:33:04 STEP 1254 H=1254:00 +CPU= 81.158
  04:34:00 STEP 1255 H=1255:00 +CPU= 55.549
  04:34:55 STEP 1256 H=1256:00 +CPU= 53.486
  04:36:15 STEP 1257 H=1257:00 +CPU= 79.403
  04:37:12 STEP 1258 H=1258:00 +CPU= 55.742
  04:38:05 STEP 1259 H=1259:00 +CPU= 51.962
  04:39:30 STEP 1260 H=1260:00 +CPU= 84.049
Moving to projects directory: /home/steve/slots/18/ICMGGhq0f+001260
Moving to projects directory: /home/steve/slots/18/ICMSHhq0f+001260
Moving to projects directory: /home/steve/slots/18/ICMUAhq0f+001260
  04:40:28 STEP 1261 H=1261:00 +CPU= 56.722
  04:41:26 STEP 1262 H=1262:00 +CPU= 57.998
  04:42:47 STEP 1263 H=1263:00 +CPU= 79.603
  04:43:44 STEP 1264 H=1264:00 +CPU= 56.014
  04:44:38 STEP 1265 H=1265:00 +CPU= 52.982
  04:46:00 STEP 1266 H=1266:00 +CPU= 80.688
  04:47:00 STEP 1267 H=1267:00 +CPU= 58.808
  04:47:53 STEP 1268 H=1268:00 +CPU= 52.467
  04:49:15 STEP 1269 H=1269:00 +CPU= 80.257
  04:50:13 STEP 1270 H=1270:00 +CPU= 57.209
  04:51:09 STEP 1271 H=1271:00 +CPU= 55.680
  04:52:48 STEP 1272 H=1272:00 +CPU= 97.470
Moving to projects directory: /home/steve/slots/18/ICMGGhq0f+001272
Moving to projects directory: /home/steve/slots/18/ICMSHhq0f+001272
Moving to projects directory: /home/steve/slots/18/ICMUAhq0f+001272
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMGGhq0f+001248
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMSHhq0f+001248
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMUAhq0f+001248
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMGGhq0f+001260
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMSHhq0f+001260
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMUAhq0f+001260
Zipping up the intermediate file: /home/steve/projects/climateprediction.net/oifs_43r3_ps_0771_2005050100_123_974_12191415_0_r671616468_52.zip
Uploading the intermediate file: upload_file_52.zip
  04:53:46 STEP 1273 H=1273:00 +CPU= 56.714
  04:54:44 STEP 1274 H=1274:00 +CPU= 58.082
  04:56:11 STEP 1275 H=1275:00 +CPU= 85.198
  04:57:12 STEP 1276 H=1276:00 +CPU= 60.599
  04:58:11 STEP 1277 H=1277:00 +CPU= 58.248
  04:59:42 STEP 1278 H=1278:00 +CPU= 89.005
  05:00:45 STEP 1279 H=1279:00 +CPU= 63.288
  05:01:45 STEP 1280 H=1280:00 +CPU= 58.663
  05:03:12 STEP 1281 H=1281:00 +CPU= 85.890
  05:04:14 STEP 1282 H=1282:00 +CPU= 61.618
  05:05:17 STEP 1283 H=1283:00 +CPU= 62.095
  05:06:59 STEP 1284 H=1284:00 +CPU=100.880
Moving to projects directory: /home/steve/slots/18/ICMGGhq0f+001284
Moving to projects directory: /home/steve/slots/18/ICMSHhq0f+001284
Moving to projects directory: /home/steve/slots/18/ICMUAhq0f+001284
  05:07:57 STEP 1285 H=1285:00 +CPU= 57.047
  05:08:55 STEP 1286 H=1286:00 +CPU= 57.252
  05:10:17 STEP 1287 H=1287:00 +CPU= 81.039
  05:11:17 STEP 1288 H=1288:00 +CPU= 58.632
  05:12:21 STEP 1289 H=1289:00 +CPU= 63.596
  05:13:54 STEP 1290 H=1290:00 +CPU= 92.046
  05:14:54 STEP 1291 H=1291:00 +CPU= 59.441
  05:15:52 STEP 1292 H=1292:00 +CPU= 57.322
  05:17:20 STEP 1293 H=1293:00 +CPU= 86.263
  05:18:23 STEP 1294 H=1294:00 +CPU= 62.482
  05:19:24 STEP 1295 H=1295:00 +CPU= 59.655
  05:21:12 STEP 1296 H=1296:00 +CPU=106.300
Moving to projects directory: /home/steve/slots/18/ICMGGhq0f+001296
Moving to projects directory: /home/steve/slots/18/ICMSHhq0f+001296
Moving to projects directory: /home/steve/slots/18/ICMUAhq0f+001296
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMGGhq0f+001272
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMSHhq0f+001272
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMUAhq0f+001272
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMGGhq0f+001284
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMSHhq0f+001284
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMUAhq0f+001284
Zipping up the intermediate file: /home/steve/projects/climateprediction.net/oifs_43r3_ps_0771_2005050100_123_974_12191415_0_r671616468_53.zip
Uploading the intermediate file: upload_file_53.zip
Uploading trickle at timestep: 4665600
  05:22:06 STEP 1297 H=1297:00 +CPU= 53.354
  05:23:05 STEP 1298 H=1298:00 +CPU= 58.566
  05:24:29 STEP 1299 H=1299:00 +CPU= 83.063
  05:25:28 STEP 1300 H=1300:00 +CPU= 57.719
  05:26:29 STEP 1301 H=1301:00 +CPU= 60.884
  05:28:03 STEP 1302 H=1302:00 +CPU= 92.316
  05:29:05 STEP 1303 H=1303:00 +CPU= 61.792
  05:30:06 STEP 1304 H=1304:00 +CPU= 60.184
  05:31:34 STEP 1305 H=1305:00 +CPU= 86.790
  05:32:34 STEP 1306 H=1306:00 +CPU= 59.815
  05:33:34 STEP 1307 H=1307:00 +CPU= 59.125
  05:35:12 STEP 1308 H=1308:00 +CPU= 96.093
Moving to projects directory: /home/steve/slots/18/ICMGGhq0f+001308
Moving to projects directory: /home/steve/slots/18/ICMSHhq0f+001308
Moving to projects directory: /home/steve/slots/18/ICMUAhq0f+001308
  05:36:12 STEP 1309 H=1309:00 +CPU= 59.978
  05:37:11 STEP 1310 H=1310:00 +CPU= 57.555
  05:38:35 STEP 1311 H=1311:00 +CPU= 83.435
  05:39:39 STEP 1312 H=1312:00 +CPU= 63.300
  05:40:40 STEP 1313 H=1313:00 +CPU= 60.570
  05:42:10 STEP 1314 H=1314:00 +CPU= 88.830
  05:43:08 STEP 1315 H=1315:00 +CPU= 57.247
  05:44:16 STEP 1316 H=1316:00 +CPU= 66.557
  05:45:41 STEP 1317 H=1317:00 +CPU= 84.628
  05:46:41 STEP 1318 H=1318:00 +CPU= 58.862
  05:47:42 STEP 1319 H=1319:00 +CPU= 59.927
  05:49:31 STEP 1320 H=1320:00 +CPU=107.474
Moving to projects directory: /home/steve/slots/18/ICMGGhq0f+001320
Moving to projects directory: /home/steve/slots/18/ICMSHhq0f+001320
Moving to projects directory: /home/steve/slots/18/ICMUAhq0f+001320
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMGGhq0f+001296
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMSHhq0f+001296
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMUAhq0f+001296
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMGGhq0f+001308
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMSHhq0f+001308
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMUAhq0f+001308
Zipping up the intermediate file: /home/steve/projects/climateprediction.net/oifs_43r3_ps_0771_2005050100_123_974_12191415_0_r671616468_54.zip
Uploading the intermediate file: upload_file_54.zip
  05:50:33 STEP 1321 H=1321:00 +CPU= 60.748
  05:51:34 STEP 1322 H=1322:00 +CPU= 60.624
  05:53:00 STEP 1323 H=1323:00 +CPU= 84.624
  05:54:07 STEP 1324 H=1324:00 +CPU= 66.369
  05:55:07 STEP 1325 H=1325:00 +CPU= 59.697
  05:56:45 STEP 1326 H=1326:00 +CPU= 96.623
  05:57:49 STEP 1327 H=1327:00 +CPU= 63.225
  05:58:54 STEP 1328 H=1328:00 +CPU= 63.594
  06:00:20 STEP 1329 H=1329:00 +CPU= 85.333
  06:01:29 STEP 1330 H=1330:00 +CPU= 67.472
  06:02:37 STEP 1331 H=1331:00 +CPU= 63.558
  06:04:32 STEP 1332 H=1332:00 +CPU=108.424
Moving to projects directory: /home/steve/slots/18/ICMGGhq0f+001332
Moving to projects directory: /home/steve/slots/18/ICMSHhq0f+001332
Moving to projects directory: /home/steve/slots/18/ICMUAhq0f+001332
  06:05:41 STEP 1333 H=1333:00 +CPU= 68.408
  06:06:51 STEP 1334 H=1334:00 +CPU= 69.092
  06:08:21 STEP 1335 H=1335:00 +CPU= 88.826
  06:09:26 STEP 1336 H=1336:00 +CPU= 63.953
  06:10:35 STEP 1337 H=1337:00 +CPU= 67.091
  06:12:07 STEP 1338 H=1338:00 +CPU= 91.300
  06:13:20 STEP 1339 H=1339:00 +CPU= 71.917
  06:14:30 STEP 1340 H=1340:00 +CPU= 69.343
  06:16:10 STEP 1341 H=1341:00 +CPU= 98.504
  06:17:23 STEP 1342 H=1342:00 +CPU= 71.471
  06:18:33 STEP 1343 H=1343:00 +CPU= 69.306
  06:20:24 STEP 1344 H=1344:00 +CPU=108.765
Moving to projects directory: /home/steve/slots/18/ICMGGhq0f+001344
Moving to projects directory: /home/steve/slots/18/ICMSHhq0f+001344
Moving to projects directory: /home/steve/slots/18/ICMUAhq0f+001344
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMGGhq0f+001320
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMSHhq0f+001320
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMUAhq0f+001320
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMGGhq0f+001332
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMSHhq0f+001332
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMUAhq0f+001332
Zipping up the intermediate file: /home/steve/projects/climateprediction.net/oifs_43r3_ps_0771_2005050100_123_974_12191415_0_r671616468_55.zip
Uploading the intermediate file: upload_file_55.zip
  06:21:28 STEP 1345 H=1345:00 +CPU= 63.839
  06:22:31 STEP 1346 H=1346:00 +CPU= 61.729
  06:24:10 STEP 1347 H=1347:00 +CPU= 98.021
  06:25:16 STEP 1348 H=1348:00 +CPU= 64.320
  06:26:24 STEP 1349 H=1349:00 +CPU= 67.402
  06:28:01 STEP 1350 H=1350:00 +CPU= 95.540
  06:29:06 STEP 1351 H=1351:00 +CPU= 64.153
  06:30:14 STEP 1352 H=1352:00 +CPU= 66.603
  06:31:53 STEP 1353 H=1353:00 +CPU= 96.868
  06:32:56 STEP 1354 H=1354:00 +CPU= 62.172
  06:33:58 STEP 1355 H=1355:00 +CPU= 61.442
  06:35:52 STEP 1356 H=1356:00 +CPU=113.034
Moving to projects directory: /home/steve/slots/18/ICMGGhq0f+001356
Moving to projects directory: /home/steve/slots/18/ICMSHhq0f+001356
Moving to projects directory: /home/steve/slots/18/ICMUAhq0f+001356
  06:37:00 STEP 1357 H=1357:00 +CPU= 66.688
  06:38:12 STEP 1358 H=1358:00 +CPU= 70.861
  06:39:50 STEP 1359 H=1359:00 +CPU= 97.190
  06:40:57 STEP 1360 H=1360:00 +CPU= 65.688
  06:42:02 STEP 1361 H=1361:00 +CPU= 64.605
  06:43:36 STEP 1362 H=1362:00 +CPU= 92.534
  06:44:40 STEP 1363 H=1363:00 +CPU= 63.541
  06:45:48 STEP 1364 H=1364:00 +CPU= 67.274
  06:47:25 STEP 1365 H=1365:00 +CPU= 95.765
  06:48:30 STEP 1366 H=1366:00 +CPU= 64.229
  06:49:37 STEP 1367 H=1367:00 +CPU= 66.119
  06:51:35 STEP 1368 H=1368:00 +CPU=115.967
Moving to projects directory: /home/steve/slots/18/ICMGGhq0f+001368
Moving to projects directory: /home/steve/slots/18/ICMSHhq0f+001368
Moving to projects directory: /home/steve/slots/18/ICMUAhq0f+001368
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMGGhq0f+001344
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMSHhq0f+001344
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMUAhq0f+001344
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMGGhq0f+001356
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMSHhq0f+001356
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMUAhq0f+001356
Zipping up the intermediate file: /home/steve/projects/climateprediction.net/oifs_43r3_ps_0771_2005050100_123_974_12191415_0_r671616468_56.zip
Uploading the intermediate file: upload_file_56.zip
  06:52:42 STEP 1369 H=1369:00 +CPU= 66.125
  06:53:52 STEP 1370 H=1370:00 +CPU= 69.165
  06:55:32 STEP 1371 H=1371:00 +CPU= 98.532
  06:56:36 STEP 1372 H=1372:00 +CPU= 62.790
  06:57:40 STEP 1373 H=1373:00 +CPU= 63.472
  06:59:21 STEP 1374 H=1374:00 +CPU= 98.572
  07:00:19 STEP 1375 H=1375:00 +CPU= 57.038
  07:01:18 STEP 1376 H=1376:00 +CPU= 57.650
  07:02:41 STEP 1377 H=1377:00 +CPU= 81.510
  07:03:38 STEP 1378 H=1378:00 +CPU= 55.713
  07:04:35 STEP 1379 H=1379:00 +CPU= 55.859
  07:06:13 STEP 1380 H=1380:00 +CPU= 96.364
Moving to projects directory: /home/steve/slots/18/ICMGGhq0f+001380
Moving to projects directory: /home/steve/slots/18/ICMSHhq0f+001380
Moving to projects directory: /home/steve/slots/18/ICMUAhq0f+001380
  07:07:12 STEP 1381 H=1381:00 +CPU= 58.286
  07:08:09 STEP 1382 H=1382:00 +CPU= 56.909
  07:09:35 STEP 1383 H=1383:00 +CPU= 83.860
  07:10:34 STEP 1384 H=1384:00 +CPU= 58.188
  07:11:31 STEP 1385 H=1385:00 +CPU= 56.443
  07:13:02 STEP 1386 H=1386:00 +CPU= 90.578
  07:14:03 STEP 1387 H=1387:00 +CPU= 60.013
  07:15:01 STEP 1388 H=1388:00 +CPU= 57.052
  07:16:30 STEP 1389 H=1389:00 +CPU= 87.457
  07:17:35 STEP 1390 H=1390:00 +CPU= 64.545
  07:18:38 STEP 1391 H=1391:00 +CPU= 61.781
  07:20:27 STEP 1392 H=1392:00 +CPU=107.045
Moving to projects directory: /home/steve/slots/18/ICMGGhq0f+001392
Moving to projects directory: /home/steve/slots/18/ICMSHhq0f+001392
Moving to projects directory: /home/steve/slots/18/ICMUAhq0f+001392
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMGGhq0f+001368
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMSHhq0f+001368
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMUAhq0f+001368
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMGGhq0f+001380
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMSHhq0f+001380
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_ps_12191415/ICMUAhq0f+001380
Zipping up the intermediate file: /home/steve/projects/climateprediction.net/oifs_43r3_ps_0771_2005050100_123_974_12191415_0_r671616468_57.zip
Uploading the intermediate file: upload_file_57.zip
  07:21:28 STEP 1393 H=1393:00 +CPU= 60.363
  07:22:30 STEP 1394 H=1394:00 +CPU= 61.443
  07:24:03 STEP 1395 H=1395:00 +CPU= 91.160
Abort request received from BOINC client, ending the child process
..Failed, model did not complete successfully

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:16:40 1536520 22277093 oifs_43r3_ps_0771_2005050100_123_974_12191415_0 4,665,600 78,559 0.0168
09 May 2023 16:16:27 1536520 22277093 oifs_43r3_ps_0771_2005050100_123_974_12191415_0 3,801,600 63,379 0.0167
09 May 2023 16:16:13 1536520 22277093 oifs_43r3_ps_0771_2005050100_123_974_12191415_0 2,937,600 48,688 0.0166


©2024 climateprediction.net