climateprediction.net home page
Task 22440010

Task 22440010

Name oifs_43r3_bl_a1rt_2016092300_20_1017_12284124_1
Workunit 12284124
Created 7 Jun 2024, 20:27:44 UTC
Sent 7 Jun 2024, 20:28:48 UTC
Report deadline 6 Aug 2024, 20:28:48 UTC
Received 8 Jun 2024, 18:41:28 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 0 (0x00000000)
Computer ID 1536520
Run time 21 hours 27 min 47 sec
CPU time 20 hours 59 min 57 sec
Validate state Invalid
Credit 1,976.32
Device peak FLOPS 7.23 GFLOPS
Application version OpenIFS 43r3 Baroclinic Lifecycle v1.13
x86_64-pc-linux-gnu
Peak working set size 5,025.19 MB
Peak swap size 5,980.80 MB
Peak disk usage 1,425.47 MB
Stderr
<core_client_version>7.22.1</core_client_version>
<![CDATA[
<stderr_txt>
+000780
  02:15:15 STEP  781 H= 195:15 +CPU= 57.097
  02:16:14 STEP  782 H= 195:30 +CPU= 58.752
  02:17:12 STEP  783 H= 195:45 +CPU= 57.082
  02:18:10 STEP  784 H= 196:00 +CPU= 58.192
  02:19:11 STEP  785 H= 196:15 +CPU= 59.794
  02:20:11 STEP  786 H= 196:30 +CPU= 60.054
  02:21:12 STEP  787 H= 196:45 +CPU= 59.647
  02:22:12 STEP  788 H= 197:00 +CPU= 60.087
  02:23:10 STEP  789 H= 197:15 +CPU= 56.898
  02:24:07 STEP  790 H= 197:30 +CPU= 56.658
  02:25:06 STEP  791 H= 197:45 +CPU= 58.058
  02:26:16 STEP  792 H= 198:00 +CPU= 69.864
Moving to projects directory: /home/steve/slots/17/ICMGGh7zg+000792
Moving to projects directory: /home/steve/slots/17/ICMSHh7zg+000792
Moving to projects directory: /home/steve/slots/17/ICMUAh7zg+000792
  02:27:12 STEP  793 H= 198:15 +CPU= 55.024
  02:28:09 STEP  794 H= 198:30 +CPU= 56.710
  02:29:08 STEP  795 H= 198:45 +CPU= 58.093
  02:30:06 STEP  796 H= 199:00 +CPU= 57.642
  02:31:04 STEP  797 H= 199:15 +CPU= 57.548
  02:32:02 STEP  798 H= 199:30 +CPU= 57.075
  02:33:01 STEP  799 H= 199:45 +CPU= 59.069
  02:34:02 STEP  800 H= 200:00 +CPU= 59.942
  02:35:02 STEP  801 H= 200:15 +CPU= 59.114
  02:36:02 STEP  802 H= 200:30 +CPU= 59.701
  02:37:00 STEP  803 H= 200:45 +CPU= 57.363
  02:38:17 STEP  804 H= 201:00 +CPU= 76.571
Moving to projects directory: /home/steve/slots/17/ICMGGh7zg+000804
Moving to projects directory: /home/steve/slots/17/ICMSHh7zg+000804
Moving to projects directory: /home/steve/slots/17/ICMUAh7zg+000804
  02:39:15 STEP  805 H= 201:15 +CPU= 56.844
  02:40:15 STEP  806 H= 201:30 +CPU= 59.290
  02:41:14 STEP  807 H= 201:45 +CPU= 58.622
  02:42:13 STEP  808 H= 202:00 +CPU= 58.836
  02:43:11 STEP  809 H= 202:15 +CPU= 56.823
  02:44:11 STEP  810 H= 202:30 +CPU= 59.673
  02:45:11 STEP  811 H= 202:45 +CPU= 59.241
  02:46:08 STEP  812 H= 203:00 +CPU= 56.379
  02:47:04 STEP  813 H= 203:15 +CPU= 55.671
  02:48:01 STEP  814 H= 203:30 +CPU= 56.448
  02:48:59 STEP  815 H= 203:45 +CPU= 57.213
  02:50:19 STEP  816 H= 204:00 +CPU= 79.590
Moving to projects directory: /home/steve/slots/17/ICMGGh7zg+000816
Moving to projects directory: /home/steve/slots/17/ICMSHh7zg+000816
Moving to projects directory: /home/steve/slots/17/ICMUAh7zg+000816
  02:51:18 STEP  817 H= 204:15 +CPU= 58.460
  02:52:18 STEP  818 H= 204:30 +CPU= 58.678
  02:53:16 STEP  819 H= 204:45 +CPU= 57.797
  02:54:19 STEP  820 H= 205:00 +CPU= 62.261
  02:55:18 STEP  821 H= 205:15 +CPU= 58.510
  02:56:17 STEP  822 H= 205:30 +CPU= 58.320
  02:57:15 STEP  823 H= 205:45 +CPU= 57.760
  02:58:16 STEP  824 H= 206:00 +CPU= 60.019
  02:59:15 STEP  825 H= 206:15 +CPU= 58.257
  03:00:13 STEP  826 H= 206:30 +CPU= 58.033
  03:01:11 STEP  827 H= 206:45 +CPU= 57.509
  03:02:25 STEP  828 H= 207:00 +CPU= 73.327
Moving to projects directory: /home/steve/slots/17/ICMGGh7zg+000828
Moving to projects directory: /home/steve/slots/17/ICMSHh7zg+000828
Moving to projects directory: /home/steve/slots/17/ICMUAh7zg+000828
  03:03:23 STEP  829 H= 207:15 +CPU= 57.248
  03:04:22 STEP  830 H= 207:30 +CPU= 57.200
  03:05:19 STEP  831 H= 207:45 +CPU= 56.468
  03:06:16 STEP  832 H= 208:00 +CPU= 56.386
  03:07:13 STEP  833 H= 208:15 +CPU= 56.643
  03:08:10 STEP  834 H= 208:30 +CPU= 56.672
  03:09:10 STEP  835 H= 208:45 +CPU= 59.077
  03:10:08 STEP  836 H= 209:00 +CPU= 57.945
  03:11:08 STEP  837 H= 209:15 +CPU= 59.224
  03:12:08 STEP  838 H= 209:30 +CPU= 58.923
  03:13:07 STEP  839 H= 209:45 +CPU= 58.250
  03:14:19 STEP  840 H= 210:00 +CPU= 71.572
Moving to projects directory: /home/steve/slots/17/ICMGGh7zg+000840
Moving to projects directory: /home/steve/slots/17/ICMSHh7zg+000840
Moving to projects directory: /home/steve/slots/17/ICMUAh7zg+000840
  03:15:16 STEP  841 H= 210:15 +CPU= 56.357
  03:16:14 STEP  842 H= 210:30 +CPU= 57.979
  03:17:14 STEP  843 H= 210:45 +CPU= 59.212
  03:18:18 STEP  844 H= 211:00 +CPU= 62.766
  03:19:17 STEP  845 H= 211:15 +CPU= 58.834
  03:20:18 STEP  846 H= 211:30 +CPU= 60.038
  03:21:19 STEP  847 H= 211:45 +CPU= 60.404
  03:22:19 STEP  848 H= 212:00 +CPU= 60.168
  03:23:19 STEP  849 H= 212:15 +CPU= 58.963
  03:24:17 STEP  850 H= 212:30 +CPU= 55.786
  03:25:15 STEP  851 H= 212:45 +CPU= 56.867
  03:26:24 STEP  852 H= 213:00 +CPU= 68.979
Moving to projects directory: /home/steve/slots/17/ICMGGh7zg+000852
Moving to projects directory: /home/steve/slots/17/ICMSHh7zg+000852
Moving to projects directory: /home/steve/slots/17/ICMUAh7zg+000852
  03:27:23 STEP  853 H= 213:15 +CPU= 57.723
  03:28:22 STEP  854 H= 213:30 +CPU= 58.427
  03:29:22 STEP  855 H= 213:45 +CPU= 59.711
  03:30:22 STEP  856 H= 214:00 +CPU= 59.557
  03:31:21 STEP  857 H= 214:15 +CPU= 57.848
  03:32:23 STEP  858 H= 214:30 +CPU= 61.744
  03:33:25 STEP  859 H= 214:45 +CPU= 60.882
  03:34:24 STEP  860 H= 215:00 +CPU= 58.721
  03:35:22 STEP  861 H= 215:15 +CPU= 56.295
  03:36:20 STEP  862 H= 215:30 +CPU= 58.012
  03:37:20 STEP  863 H= 215:45 +CPU= 59.061
  03:38:44 STEP  864 H= 216:00 +CPU= 82.621
Moving to projects directory: /home/steve/slots/17/ICMGGh7zg+000864
Moving to projects directory: /home/steve/slots/17/ICMSHh7zg+000864
Moving to projects directory: /home/steve/slots/17/ICMUAh7zg+000864
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMGGh7zg+000768
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMSHh7zg+000768
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMUAh7zg+000768
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMGGh7zg+000780
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMSHh7zg+000780
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMUAh7zg+000780
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMGGh7zg+000792
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMSHh7zg+000792
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMUAh7zg+000792
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMGGh7zg+000804
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMSHh7zg+000804
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMUAh7zg+000804
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMGGh7zg+000816
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMSHh7zg+000816
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMUAh7zg+000816
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMGGh7zg+000828
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMSHh7zg+000828
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMUAh7zg+000828
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMGGh7zg+000840
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMSHh7zg+000840
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMUAh7zg+000840
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMGGh7zg+000852
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMSHh7zg+000852
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMUAh7zg+000852
Zipping up the intermediate file: /home/steve/projects/climateprediction.net/oifs_43r3_bl_a1rt_2016092300_20_1017_12284124_1_r1590682354_8.zip
Uploading the intermediate file: upload_file_8.zip
  03:39:43 STEP  865 H= 216:15 +CPU= 57.544
  03:40:43 STEP  866 H= 216:30 +CPU= 59.434
  03:41:43 STEP  867 H= 216:45 +CPU= 58.891
  03:42:42 STEP  868 H= 217:00 +CPU= 58.047
  03:43:43 STEP  869 H= 217:15 +CPU= 60.302
  03:44:38 STEP  870 H= 217:30 +CPU= 55.347
  03:45:35 STEP  871 H= 217:45 +CPU= 55.892
  03:46:32 STEP  872 H= 218:00 +CPU= 56.826
  03:47:31 STEP  873 H= 218:15 +CPU= 57.992
  03:48:29 STEP  874 H= 218:30 +CPU= 57.167
  03:49:29 STEP  875 H= 218:45 +CPU= 59.996
  03:50:41 STEP  876 H= 219:00 +CPU= 71.125
Moving to projects directory: /home/steve/slots/17/ICMGGh7zg+000876
Moving to projects directory: /home/steve/slots/17/ICMSHh7zg+000876
Moving to projects directory: /home/steve/slots/17/ICMUAh7zg+000876
  03:51:41 STEP  877 H= 219:15 +CPU= 58.730
  03:52:39 STEP  878 H= 219:30 +CPU= 57.881
  03:53:38 STEP  879 H= 219:45 +CPU= 58.934
  03:54:39 STEP  880 H= 220:00 +CPU= 59.386
  03:55:36 STEP  881 H= 220:15 +CPU= 56.782
  03:56:34 STEP  882 H= 220:30 +CPU= 57.752
  03:57:35 STEP  883 H= 220:45 +CPU= 60.330
  03:58:33 STEP  884 H= 221:00 +CPU= 57.439
  03:59:33 STEP  885 H= 221:15 +CPU= 58.870
  04:00:32 STEP  886 H= 221:30 +CPU= 58.999
  04:01:32 STEP  887 H= 221:45 +CPU= 59.328
  04:02:43 STEP  888 H= 222:00 +CPU= 70.721
Moving to projects directory: /home/steve/slots/17/ICMGGh7zg+000888
Moving to projects directory: /home/steve/slots/17/ICMSHh7zg+000888
Moving to projects directory: /home/steve/slots/17/ICMUAh7zg+000888
  04:03:41 STEP  889 H= 222:15 +CPU= 57.555
  04:04:37 STEP  890 H= 222:30 +CPU= 54.521
  04:05:32 STEP  891 H= 222:45 +CPU= 55.021
  04:06:29 STEP  892 H= 223:00 +CPU= 56.656
  04:07:25 STEP  893 H= 223:15 +CPU= 53.513
  04:08:21 STEP  894 H= 223:30 +CPU= 55.402
  04:09:17 STEP  895 H= 223:45 +CPU= 55.592
  04:10:13 STEP  896 H= 224:00 +CPU= 55.214
  04:11:07 STEP  897 H= 224:15 +CPU= 53.206
  04:12:03 STEP  898 H= 224:30 +CPU= 55.748
  04:13:00 STEP  899 H= 224:45 +CPU= 55.697
  04:14:08 STEP  900 H= 225:00 +CPU= 67.769
Moving to projects directory: /home/steve/slots/17/ICMGGh7zg+000900
Moving to projects directory: /home/steve/slots/17/ICMSHh7zg+000900
Moving to projects directory: /home/steve/slots/17/ICMUAh7zg+000900
  04:15:02 STEP  901 H= 225:15 +CPU= 54.092
  04:15:58 STEP  902 H= 225:30 +CPU= 54.932
  04:16:54 STEP  903 H= 225:45 +CPU= 55.892
  04:17:50 STEP  904 H= 226:00 +CPU= 55.370
  04:18:45 STEP  905 H= 226:15 +CPU= 53.869
  04:19:40 STEP  906 H= 226:30 +CPU= 54.919
  04:20:36 STEP  907 H= 226:45 +CPU= 55.157
  04:21:32 STEP  908 H= 227:00 +CPU= 55.800
  04:22:28 STEP  909 H= 227:15 +CPU= 55.157
  04:23:24 STEP  910 H= 227:30 +CPU= 55.317
  04:24:21 STEP  911 H= 227:45 +CPU= 57.280
  04:25:38 STEP  912 H= 228:00 +CPU= 75.423
Moving to projects directory: /home/steve/slots/17/ICMGGh7zg+000912
Moving to projects directory: /home/steve/slots/17/ICMSHh7zg+000912
Moving to projects directory: /home/steve/slots/17/ICMUAh7zg+000912
  04:26:35 STEP  913 H= 228:15 +CPU= 56.474
  04:27:30 STEP  914 H= 228:30 +CPU= 54.302
  04:28:24 STEP  915 H= 228:45 +CPU= 54.263
  04:29:21 STEP  916 H= 229:00 +CPU= 56.671
  04:30:15 STEP  917 H= 229:15 +CPU= 53.576
  04:31:11 STEP  918 H= 229:30 +CPU= 54.960
  04:32:07 STEP  919 H= 229:45 +CPU= 55.382
  04:33:03 STEP  920 H= 230:00 +CPU= 55.669
  04:33:58 STEP  921 H= 230:15 +CPU= 53.963
  04:34:54 STEP  922 H= 230:30 +CPU= 55.467
  04:35:50 STEP  923 H= 230:45 +CPU= 56.138
  04:37:00 STEP  924 H= 231:00 +CPU= 68.567
Moving to projects directory: /home/steve/slots/17/ICMGGh7zg+000924
Moving to projects directory: /home/steve/slots/17/ICMSHh7zg+000924
Moving to projects directory: /home/steve/slots/17/ICMUAh7zg+000924
  04:37:55 STEP  925 H= 231:15 +CPU= 54.954
  04:38:50 STEP  926 H= 231:30 +CPU= 53.998
  04:39:45 STEP  927 H= 231:45 +CPU= 54.622
  04:40:41 STEP  928 H= 232:00 +CPU= 55.317
  04:41:36 STEP  929 H= 232:15 +CPU= 54.238
  04:42:32 STEP  930 H= 232:30 +CPU= 55.479
  04:43:28 STEP  931 H= 232:45 +CPU= 55.752
  04:44:25 STEP  932 H= 233:00 +CPU= 55.922
  04:45:20 STEP  933 H= 233:15 +CPU= 54.799
  04:46:15 STEP  934 H= 233:30 +CPU= 54.465
  04:47:09 STEP  935 H= 233:45 +CPU= 54.154
  04:48:19 STEP  936 H= 234:00 +CPU= 68.509
Moving to projects directory: /home/steve/slots/17/ICMGGh7zg+000936
Moving to projects directory: /home/steve/slots/17/ICMSHh7zg+000936
Moving to projects directory: /home/steve/slots/17/ICMUAh7zg+000936
  04:49:14 STEP  937 H= 234:15 +CPU= 55.131
  04:50:09 STEP  938 H= 234:30 +CPU= 54.127
  04:51:04 STEP  939 H= 234:45 +CPU= 54.996
  04:52:01 STEP  940 H= 235:00 +CPU= 55.732
  04:52:55 STEP  941 H= 235:15 +CPU= 53.816
  04:53:50 STEP  942 H= 235:30 +CPU= 54.422
  04:54:45 STEP  943 H= 235:45 +CPU= 54.272
  04:55:40 STEP  944 H= 236:00 +CPU= 54.949
  04:56:36 STEP  945 H= 236:15 +CPU= 54.742
  04:57:32 STEP  946 H= 236:30 +CPU= 54.201
  04:58:27 STEP  947 H= 236:45 +CPU= 54.598
  04:59:34 STEP  948 H= 237:00 +CPU= 66.311
Moving to projects directory: /home/steve/slots/17/ICMGGh7zg+000948
Moving to projects directory: /home/steve/slots/17/ICMSHh7zg+000948
Moving to projects directory: /home/steve/slots/17/ICMUAh7zg+000948
  05:00:30 STEP  949 H= 237:15 +CPU= 55.328
  05:01:26 STEP  950 H= 237:30 +CPU= 55.888
  05:02:23 STEP  951 H= 237:45 +CPU= 55.523
  05:03:20 STEP  952 H= 238:00 +CPU= 57.131
  05:04:15 STEP  953 H= 238:15 +CPU= 54.555
  05:05:11 STEP  954 H= 238:30 +CPU= 55.284
  05:06:06 STEP  955 H= 238:45 +CPU= 54.201
  05:07:03 STEP  956 H= 239:00 +CPU= 56.048
  05:07:58 STEP  957 H= 239:15 +CPU= 53.546
  05:08:54 STEP  958 H= 239:30 +CPU= 56.070
  05:09:50 STEP  959 H= 239:45 +CPU= 55.041
  05:11:07 STEP  960 H= 240:00 +CPU= 75.948
Moving to projects directory: /home/steve/slots/17/ICMGGh7zg+000960
Moving to projects directory: /home/steve/slots/17/ICMSHh7zg+000960
Moving to projects directory: /home/steve/slots/17/ICMUAh7zg+000960
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMGGh7zg+000864
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMSHh7zg+000864
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMUAh7zg+000864
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMGGh7zg+000876
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMSHh7zg+000876
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMUAh7zg+000876
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMGGh7zg+000888
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMSHh7zg+000888
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMUAh7zg+000888
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMGGh7zg+000900
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMSHh7zg+000900
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMUAh7zg+000900
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMGGh7zg+000912
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMSHh7zg+000912
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMUAh7zg+000912
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMGGh7zg+000924
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMSHh7zg+000924
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMUAh7zg+000924
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMGGh7zg+000936
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMSHh7zg+000936
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMUAh7zg+000936
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMGGh7zg+000948
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMSHh7zg+000948
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMUAh7zg+000948
Zipping up the intermediate file: /home/steve/projects/climateprediction.net/oifs_43r3_bl_a1rt_2016092300_20_1017_12284124_1_r1590682354_9.zip
Uploading the intermediate file: upload_file_9.zip
Uploading trickle at timestep: 864000
  05:12:02 STEP  961 H= 240:15 +CPU= 53.631
  05:12:59 STEP  962 H= 240:30 +CPU= 55.452
  05:13:55 STEP  963 H= 240:45 +CPU= 55.537
  05:14:50 STEP  964 H= 241:00 +CPU= 54.746
  05:15:45 STEP  965 H= 241:15 +CPU= 54.245
  05:16:40 STEP  966 H= 241:30 +CPU= 54.845
  05:17:35 STEP  967 H= 241:45 +CPU= 54.661
  05:18:32 STEP  968 H= 242:00 +CPU= 56.446
  05:19:27 STEP  969 H= 242:15 +CPU= 54.020
  05:20:23 STEP  970 H= 242:30 +CPU= 55.922
  05:21:19 STEP  971 H= 242:45 +CPU= 55.463
  05:22:28 STEP  972 H= 243:00 +CPU= 67.775
Moving to projects directory: /home/steve/slots/17/ICMGGh7zg+000972
Moving to projects directory: /home/steve/slots/17/ICMSHh7zg+000972
Moving to projects directory: /home/steve/slots/17/ICMUAh7zg+000972
  05:23:21 STEP  973 H= 243:15 +CPU= 52.959
  05:24:16 STEP  974 H= 243:30 +CPU= 54.111
  05:25:11 STEP  975 H= 243:45 +CPU= 54.425
  05:26:09 STEP  976 H= 244:00 +CPU= 57.200
  05:27:03 STEP  977 H= 244:15 +CPU= 53.918
  05:28:00 STEP  978 H= 244:30 +CPU= 55.785
  05:28:55 STEP  979 H= 244:45 +CPU= 54.958
  05:29:52 STEP  980 H= 245:00 +CPU= 56.513
  05:30:46 STEP  981 H= 245:15 +CPU= 53.687
  05:31:44 STEP  982 H= 245:30 +CPU= 57.052
  05:32:41 STEP  983 H= 245:45 +CPU= 56.793
  05:33:52 STEP  984 H= 246:00 +CPU= 70.013
Moving to projects directory: /home/steve/slots/17/ICMGGh7zg+000984
Moving to projects directory: /home/steve/slots/17/ICMSHh7zg+000984
Moving to projects directory: /home/steve/slots/17/ICMUAh7zg+000984
  05:34:47 STEP  985 H= 246:15 +CPU= 54.406
  05:35:41 STEP  986 H= 246:30 +CPU= 54.049
  05:36:38 STEP  987 H= 246:45 +CPU= 55.788
  05:37:34 STEP  988 H= 247:00 +CPU= 55.872
  05:38:28 STEP  989 H= 247:15 +CPU= 53.508
  05:39:24 STEP  990 H= 247:30 +CPU= 55.790
  05:40:20 STEP  991 H= 247:45 +CPU= 55.299
  05:41:17 STEP  992 H= 248:00 +CPU= 56.036
  05:42:11 STEP  993 H= 248:15 +CPU= 53.457
  05:43:08 STEP  994 H= 248:30 +CPU= 56.821
  05:44:04 STEP  995 H= 248:45 +CPU= 55.649
  05:45:13 STEP  996 H= 249:00 +CPU= 68.438
Moving to projects directory: /home/steve/slots/17/ICMGGh7zg+000996
Moving to projects directory: /home/steve/slots/17/ICMSHh7zg+000996
Moving to projects directory: /home/steve/slots/17/ICMUAh7zg+000996
  05:46:08 STEP  997 H= 249:15 +CPU= 54.429
  05:47:05 STEP  998 H= 249:30 +CPU= 56.565
  05:48:01 STEP  999 H= 249:45 +CPU= 54.749
  05:48:57 STEP 1000 H= 250:00 +CPU= 55.547
  05:49:53 STEP 1001 H= 250:15 +CPU= 55.463
  05:50:49 STEP 1002 H= 250:30 +CPU= 55.401
  05:51:45 STEP 1003 H= 250:45 +CPU= 55.298
  05:52:41 STEP 1004 H= 251:00 +CPU= 56.323
  05:53:36 STEP 1005 H= 251:15 +CPU= 53.754
  05:54:32 STEP 1006 H= 251:30 +CPU= 55.468
  05:55:28 STEP 1007 H= 251:45 +CPU= 55.862
  05:56:43 STEP 1008 H= 252:00 +CPU= 73.890
Moving to projects directory: /home/steve/slots/17/ICMGGh7zg+001008
Moving to projects directory: /home/steve/slots/17/ICMSHh7zg+001008
Moving to projects directory: /home/steve/slots/17/ICMUAh7zg+001008
  05:57:39 STEP 1009 H= 252:15 +CPU= 55.532
  05:58:35 STEP 1010 H= 252:30 +CPU= 56.064
  05:59:31 STEP 1011 H= 252:45 +CPU= 55.252
  06:00:26 STEP 1012 H= 253:00 +CPU= 54.457
  06:01:21 STEP 1013 H= 253:15 +CPU= 54.562
  06:02:16 STEP 1014 H= 253:30 +CPU= 54.371
  06:03:11 STEP 1015 H= 253:45 +CPU= 54.599
  06:04:08 STEP 1016 H= 254:00 +CPU= 55.972
  06:05:04 STEP 1017 H= 254:15 +CPU= 56.243
  06:06:00 STEP 1018 H= 254:30 +CPU= 55.392
  06:06:56 STEP 1019 H= 254:45 +CPU= 55.073
  06:08:05 STEP 1020 H= 255:00 +CPU= 68.052
Moving to projects directory: /home/steve/slots/17/ICMGGh7zg+001020
Moving to projects directory: /home/steve/slots/17/ICMSHh7zg+001020
Moving to projects directory: /home/steve/slots/17/ICMUAh7zg+001020
  06:09:00 STEP 1021 H= 255:15 +CPU= 54.697
  06:09:55 STEP 1022 H= 255:30 +CPU= 53.594
  06:10:51 STEP 1023 H= 255:45 +CPU= 55.889
  06:11:47 STEP 1024 H= 256:00 +CPU= 54.803
  06:12:41 STEP 1025 H= 256:15 +CPU= 54.291
  06:13:36 STEP 1026 H= 256:30 +CPU= 54.012
  06:14:33 STEP 1027 H= 256:45 +CPU= 55.886
  06:15:30 STEP 1028 H= 257:00 +CPU= 55.175
  06:16:24 STEP 1029 H= 257:15 +CPU= 53.777
  06:17:21 STEP 1030 H= 257:30 +CPU= 55.608
  06:18:16 STEP 1031 H= 257:45 +CPU= 55.272
  06:19:24 STEP 1032 H= 258:00 +CPU= 67.364
Moving to projects directory: /home/steve/slots/17/ICMGGh7zg+001032
Moving to projects directory: /home/steve/slots/17/ICMSHh7zg+001032
Moving to projects directory: /home/steve/slots/17/ICMUAh7zg+001032
  06:20:19 STEP 1033 H= 258:15 +CPU= 54.139
  06:21:20 STEP 1034 H= 258:30 +CPU= 51.190
  06:22:20 STEP 1035 H= 258:45 +CPU= 53.945
  06:23:18 STEP 1036 H= 259:00 +CPU= 54.447
  06:24:16 STEP 1037 H= 259:15 +CPU= 55.134
  06:25:14 STEP 1038 H= 259:30 +CPU= 53.786
  06:26:09 STEP 1039 H= 259:45 +CPU= 53.296
  06:27:07 STEP 1040 H= 260:00 +CPU= 55.392
  06:28:07 STEP 1041 H= 260:15 +CPU= 54.906
  06:29:03 STEP 1042 H= 260:30 +CPU= 52.368
  06:29:59 STEP 1043 H= 260:45 +CPU= 54.669
  06:31:11 STEP 1044 H= 261:00 +CPU= 68.877
Moving to projects directory: /home/steve/slots/17/ICMGGh7zg+001044
Moving to projects directory: /home/steve/slots/17/ICMSHh7zg+001044
Moving to projects directory: /home/steve/slots/17/ICMUAh7zg+001044
  06:32:13 STEP 1045 H= 261:15 +CPU= 56.435
  06:33:12 STEP 1046 H= 261:30 +CPU= 55.828
  06:34:13 STEP 1047 H= 261:45 +CPU= 55.870
  06:35:11 STEP 1048 H= 262:00 +CPU= 55.199
  06:36:07 STEP 1049 H= 262:15 +CPU= 55.432
  06:37:02 STEP 1050 H= 262:30 +CPU= 54.772
  06:38:00 STEP 1051 H= 262:45 +CPU= 54.195
  06:38:57 STEP 1052 H= 263:00 +CPU= 55.299
  06:39:55 STEP 1053 H= 263:15 +CPU= 52.686
  06:40:54 STEP 1054 H= 263:30 +CPU= 55.881
  06:41:51 STEP 1055 H= 263:45 +CPU= 54.003
  06:43:10 STEP 1056 H= 264:00 +CPU= 74.978
Moving to projects directory: /home/steve/slots/17/ICMGGh7zg+001056
Moving to projects directory: /home/steve/slots/17/ICMSHh7zg+001056
Moving to projects directory: /home/steve/slots/17/ICMUAh7zg+001056
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMGGh7zg+000960
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMSHh7zg+000960
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMUAh7zg+000960
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMGGh7zg+000972
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMSHh7zg+000972
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMUAh7zg+000972
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMGGh7zg+000984
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMSHh7zg+000984
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMUAh7zg+000984
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMGGh7zg+000996
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMSHh7zg+000996
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMUAh7zg+000996
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMGGh7zg+001008
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMSHh7zg+001008
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMUAh7zg+001008
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMGGh7zg+001020
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMSHh7zg+001020
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMUAh7zg+001020
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMGGh7zg+001032
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMSHh7zg+001032
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMUAh7zg+001032
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMGGh7zg+001044
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMSHh7zg+001044
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMUAh7zg+001044
Zipping up the intermediate file: /home/steve/projects/climateprediction.net/oifs_43r3_bl_a1rt_2016092300_20_1017_12284124_1_r1590682354_10.zip
Uploading the intermediate file: upload_file_10.zip
  06:44:06 STEP 1057 H= 264:15 +CPU= 52.870
  06:45:03 STEP 1058 H= 264:30 +CPU= 54.553
  06:46:01 STEP 1059 H= 264:45 +CPU= 55.021
  06:46:58 STEP 1060 H= 265:00 +CPU= 55.620
  06:47:57 STEP 1061 H= 265:15 +CPU= 56.299
  06:48:55 STEP 1062 H= 265:30 +CPU= 53.598
  06:49:52 STEP 1063 H= 265:45 +CPU= 55.565
  06:50:47 STEP 1064 H= 266:00 +CPU= 54.522
  06:51:44 STEP 1065 H= 266:15 +CPU= 54.680
  06:52:42 STEP 1066 H= 266:30 +CPU= 55.188
  06:53:39 STEP 1067 H= 266:45 +CPU= 55.797
  06:54:47 STEP 1068 H= 267:00 +CPU= 66.650
Moving to projects directory: /home/steve/slots/17/ICMGGh7zg+001068
Moving to projects directory: /home/steve/slots/17/ICMSHh7zg+001068
Moving to projects directory: /home/steve/slots/17/ICMUAh7zg+001068
  06:55:42 STEP 1069 H= 267:15 +CPU= 53.638
  06:56:39 STEP 1070 H= 267:30 +CPU= 55.604
  06:57:35 STEP 1071 H= 267:45 +CPU= 54.691
  06:58:32 STEP 1072 H= 268:00 +CPU= 54.225
  06:59:30 STEP 1073 H= 268:15 +CPU= 55.564
  07:00:30 STEP 1074 H= 268:30 +CPU= 54.997
  07:01:28 STEP 1075 H= 268:45 +CPU= 55.258
  07:02:26 STEP 1076 H= 269:00 +CPU= 55.825
  07:03:24 STEP 1077 H= 269:15 +CPU= 55.819
  07:04:22 STEP 1078 H= 269:30 +CPU= 55.369
  07:05:19 STEP 1079 H= 269:45 +CPU= 56.082
  07:06:27 STEP 1080 H= 270:00 +CPU= 67.640
Moving to projects directory: /home/steve/slots/17/ICMGGh7zg+001080
Moving to projects directory: /home/steve/slots/17/ICMSHh7zg+001080
Moving to projects directory: /home/steve/slots/17/ICMUAh7zg+001080
  07:07:22 STEP 1081 H= 270:15 +CPU= 53.839
  07:08:18 STEP 1082 H= 270:30 +CPU= 55.435
  07:09:12 STEP 1083 H= 270:45 +CPU= 54.080
  07:10:10 STEP 1084 H= 271:00 +CPU= 57.026
  07:11:05 STEP 1085 H= 271:15 +CPU= 55.153
  07:12:01 STEP 1086 H= 271:30 +CPU= 54.737
  07:12:56 STEP 1087 H= 271:45 +CPU= 54.658
  07:13:51 STEP 1088 H= 272:00 +CPU= 54.980
  07:14:49 STEP 1089 H= 272:15 +CPU= 56.535
  07:15:45 STEP 1090 H= 272:30 +CPU= 55.247
  07:16:40 STEP 1091 H= 272:45 +CPU= 54.997
  07:17:47 STEP 1092 H= 273:00 +CPU= 66.054
Moving to projects directory: /home/steve/slots/17/ICMGGh7zg+001092
Moving to projects directory: /home/steve/slots/17/ICMSHh7zg+001092
Moving to projects directory: /home/steve/slots/17/ICMUAh7zg+001092
  07:18:42 STEP 1093 H= 273:15 +CPU= 54.324
  07:19:36 STEP 1094 H= 273:30 +CPU= 53.970
  07:20:33 STEP 1095 H= 273:45 +CPU= 56.454
  07:21:29 STEP 1096 H= 274:00 +CPU= 54.598
  07:22:24 STEP 1097 H= 274:15 +CPU= 55.195
  07:23:19 STEP 1098 H= 274:30 +CPU= 54.365
  07:24:14 STEP 1099 H= 274:45 +CPU= 54.119
  07:25:12 STEP 1100 H= 275:00 +CPU= 55.317
  07:26:11 STEP 1101 H= 275:15 +CPU= 55.116
  07:27:09 STEP 1102 H= 275:30 +CPU= 55.012
  07:28:05 STEP 1103 H= 275:45 +CPU= 54.405
  07:29:24 STEP 1104 H= 276:00 +CPU= 74.811
Moving to projects directory: /home/steve/slots/17/ICMGGh7zg+001104
Moving to projects directory: /home/steve/slots/17/ICMSHh7zg+001104
Moving to projects directory: /home/steve/slots/17/ICMUAh7zg+001104
  07:30:23 STEP 1105 H= 276:15 +CPU= 56.006
  07:31:20 STEP 1106 H= 276:30 +CPU= 55.625
  07:32:19 STEP 1107 H= 276:45 +CPU= 56.195
  07:33:18 STEP 1108 H= 277:00 +CPU= 56.695
  07:34:16 STEP 1109 H= 277:15 +CPU= 55.085
  07:35:14 STEP 1110 H= 277:30 +CPU= 54.218
  07:36:11 STEP 1111 H= 277:45 +CPU= 54.683
  07:37:09 STEP 1112 H= 278:00 +CPU= 55.026
  07:38:05 STEP 1113 H= 278:15 +CPU= 53.868
  07:39:02 STEP 1114 H= 278:30 +CPU= 54.991
  07:39:58 STEP 1115 H= 278:45 +CPU= 54.737
  07:41:08 STEP 1116 H= 279:00 +CPU= 67.444
Moving to projects directory: /home/steve/slots/17/ICMGGh7zg+001116
Moving to projects directory: /home/steve/slots/17/ICMSHh7zg+001116
Moving to projects directory: /home/steve/slots/17/ICMUAh7zg+001116
  07:42:04 STEP 1117 H= 279:15 +CPU= 53.916
  07:43:00 STEP 1118 H= 279:30 +CPU= 54.117
  07:43:56 STEP 1119 H= 279:45 +CPU= 55.220
  07:44:53 STEP 1120 H= 280:00 +CPU= 56.024
  07:45:56 STEP 1121 H= 280:15 +CPU= 54.475
  07:46:57 STEP 1122 H= 280:30 +CPU= 54.199
  07:47:58 STEP 1123 H= 280:45 +CPU= 54.246
  07:48:59 STEP 1124 H= 281:00 +CPU= 55.053
  07:49:55 STEP 1125 H= 281:15 +CPU= 53.827
  07:50:52 STEP 1126 H= 281:30 +CPU= 55.281
  07:51:49 STEP 1127 H= 281:45 +CPU= 53.980
  07:52:58 STEP 1128 H= 282:00 +CPU= 66.806
Moving to projects directory: /home/steve/slots/17/ICMGGh7zg+001128
Moving to projects directory: /home/steve/slots/17/ICMSHh7zg+001128
Moving to projects directory: /home/steve/slots/17/ICMUAh7zg+001128
  07:53:56 STEP 1129 H= 282:15 +CPU= 54.061
  07:54:53 STEP 1130 H= 282:30 +CPU= 54.549
  07:55:50 STEP 1131 H= 282:45 +CPU= 54.860
  07:56:48 STEP 1132 H= 283:00 +CPU= 55.124
  07:57:47 STEP 1133 H= 283:15 +CPU= 54.815
  07:58:46 STEP 1134 H= 283:30 +CPU= 56.307
  07:59:40 STEP 1135 H= 283:45 +CPU= 53.234
  08:00:37 STEP 1136 H= 284:00 +CPU= 56.348
  08:01:33 STEP 1137 H= 284:15 +CPU= 55.413
  08:02:28 STEP 1138 H= 284:30 +CPU= 53.778
  08:03:24 STEP 1139 H= 284:45 +CPU= 54.601
  08:04:33 STEP 1140 H= 285:00 +CPU= 67.358
Moving to projects directory: /home/steve/slots/17/ICMGGh7zg+001140
Moving to projects directory: /home/steve/slots/17/ICMSHh7zg+001140
Moving to projects directory: /home/steve/slots/17/ICMUAh7zg+001140
  08:05:28 STEP 1141 H= 285:15 +CPU= 54.977
  08:06:24 STEP 1142 H= 285:30 +CPU= 55.028
  08:07:19 STEP 1143 H= 285:45 +CPU= 54.541
  08:08:15 STEP 1144 H= 286:00 +CPU= 55.239
  08:09:10 STEP 1145 H= 286:15 +CPU= 55.133
  08:10:07 STEP 1146 H= 286:30 +CPU= 56.217
  08:11:02 STEP 1147 H= 286:45 +CPU= 54.501
  08:11:59 STEP 1148 H= 287:00 +CPU= 55.902
  08:12:55 STEP 1149 H= 287:15 +CPU= 54.793
  08:13:50 STEP 1150 H= 287:30 +CPU= 54.926
  08:14:46 STEP 1151 H= 287:45 +CPU= 54.891
  08:16:02 STEP 1152 H= 288:00 +CPU= 74.837
Moving to projects directory: /home/steve/slots/17/ICMGGh7zg+001152
Moving to projects directory: /home/steve/slots/17/ICMSHh7zg+001152
Moving to projects directory: /home/steve/slots/17/ICMUAh7zg+001152
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMGGh7zg+001056
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMSHh7zg+001056
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMUAh7zg+001056
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMGGh7zg+001068
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMSHh7zg+001068
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMUAh7zg+001068
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMGGh7zg+001080
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMSHh7zg+001080
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMUAh7zg+001080
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMGGh7zg+001092
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMSHh7zg+001092
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMUAh7zg+001092
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMGGh7zg+001104
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMSHh7zg+001104
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMUAh7zg+001104
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMGGh7zg+001116
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMSHh7zg+001116
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMUAh7zg+001116
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMGGh7zg+001128
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMSHh7zg+001128
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMUAh7zg+001128
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMGGh7zg+001140
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMSHh7zg+001140
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMUAh7zg+001140
Zipping up the intermediate file: /home/steve/projects/climateprediction.net/oifs_43r3_bl_a1rt_2016092300_20_1017_12284124_1_r1590682354_11.zip
Uploading the intermediate file: upload_file_11.zip
  08:16:58 STEP 1153 H= 288:15 +CPU= 53.446
  08:17:53 STEP 1154 H= 288:30 +CPU= 54.753
  08:18:50 STEP 1155 H= 288:45 +CPU= 55.472
  08:19:46 STEP 1156 H= 289:00 +CPU= 56.047
  08:20:42 STEP 1157 H= 289:15 +CPU= 55.659
  08:21:38 STEP 1158 H= 289:30 +CPU= 54.976
  08:22:35 STEP 1159 H= 289:45 +CPU= 56.477
  08:23:29 STEP 1160 H= 290:00 +CPU= 53.906
  08:24:24 STEP 1161 H= 290:15 +CPU= 54.931
  08:25:20 STEP 1162 H= 290:30 +CPU= 55.345
  08:26:16 STEP 1163 H= 290:45 +CPU= 55.223
  08:27:24 STEP 1164 H= 291:00 +CPU= 67.636
Moving to projects directory: /home/steve/slots/17/ICMGGh7zg+001164
Moving to projects directory: /home/steve/slots/17/ICMSHh7zg+001164
Moving to projects directory: /home/steve/slots/17/ICMUAh7zg+001164
  08:28:21 STEP 1165 H= 291:15 +CPU= 55.554
  08:29:16 STEP 1166 H= 291:30 +CPU= 55.176
  08:30:12 STEP 1167 H= 291:45 +CPU= 54.693
  08:31:08 STEP 1168 H= 292:00 +CPU= 55.729
  08:32:02 STEP 1169 H= 292:15 +CPU= 54.008
  08:32:58 STEP 1170 H= 292:30 +CPU= 54.859
  08:33:54 STEP 1171 H= 292:45 +CPU= 55.613
  08:34:51 STEP 1172 H= 293:00 +CPU= 56.585
  08:35:47 STEP 1173 H= 293:15 +CPU= 55.728
  08:36:43 STEP 1174 H= 293:30 +CPU= 54.887
  08:37:38 STEP 1175 H= 293:45 +CPU= 54.933
  08:38:48 STEP 1176 H= 294:00 +CPU= 68.865
Moving to projects directory: /home/steve/slots/17/ICMGGh7zg+001176
Moving to projects directory: /home/steve/slots/17/ICMSHh7zg+001176
Moving to projects directory: /home/steve/slots/17/ICMUAh7zg+001176
  08:39:42 STEP 1177 H= 294:15 +CPU= 54.137
  08:40:38 STEP 1178 H= 294:30 +CPU= 54.671
  08:41:34 STEP 1179 H= 294:45 +CPU= 56.309
  08:42:32 STEP 1180 H= 295:00 +CPU= 56.901
  08:43:27 STEP 1181 H= 295:15 +CPU= 55.005
  08:44:23 STEP 1182 H= 295:30 +CPU= 55.102
  08:45:20 STEP 1183 H= 295:45 +CPU= 56.698
  08:46:15 STEP 1184 H= 296:00 +CPU= 54.503
  08:47:10 STEP 1185 H= 296:15 +CPU= 53.781
  08:48:06 STEP 1186 H= 296:30 +CPU= 55.229
  08:49:01 STEP 1187 H= 296:45 +CPU= 54.950
  08:50:12 STEP 1188 H= 297:00 +CPU= 70.219
Moving to projects directory: /home/steve/slots/17/ICMGGh7zg+001188
Moving to projects directory: /home/steve/slots/17/ICMSHh7zg+001188
Moving to projects directory: /home/steve/slots/17/ICMUAh7zg+001188
  08:51:07 STEP 1189 H= 297:15 +CPU= 54.501
  08:52:03 STEP 1190 H= 297:30 +CPU= 55.444
  08:52:58 STEP 1191 H= 297:45 +CPU= 54.373
  08:53:54 STEP 1192 H= 298:00 +CPU= 56.277
  08:54:51 STEP 1193 H= 298:15 +CPU= 56.078
  08:55:48 STEP 1194 H= 298:30 +CPU= 56.410
  08:56:45 STEP 1195 H= 298:45 +CPU= 56.106
  08:57:42 STEP 1196 H= 299:00 +CPU= 56.302
  08:58:38 STEP 1197 H= 299:15 +CPU= 55.564
  08:59:33 STEP 1198 H= 299:30 +CPU= 55.110
  09:00:30 STEP 1199 H= 299:45 +CPU= 56.208
  09:01:45 STEP 1200 H= 300:00 +CPU= 74.254
Moving to projects directory: /home/steve/slots/17/ICMGGh7zg+001200
Moving to projects directory: /home/steve/slots/17/ICMSHh7zg+001200
Moving to projects directory: /home/steve/slots/17/ICMUAh7zg+001200
  09:02:40 STEP 1201 H= 300:15 +CPU= 53.708
  09:03:35 STEP 1202 H= 300:30 +CPU= 55.033
  09:04:29 STEP 1203 H= 300:45 +CPU= 53.722
  09:05:25 STEP 1204 H= 301:00 +CPU= 55.642
  09:06:20 STEP 1205 H= 301:15 +CPU= 54.302
  09:07:16 STEP 1206 H= 301:30 +CPU= 55.027
  09:08:11 STEP 1207 H= 301:45 +CPU= 55.129
  09:09:08 STEP 1208 H= 302:00 +CPU= 56.334
  09:10:06 STEP 1209 H= 302:15 +CPU= 54.675
  09:11:04 STEP 1210 H= 302:30 +CPU= 55.529
  09:12:04 STEP 1211 H= 302:45 +CPU= 53.805
  09:13:16 STEP 1212 H= 303:00 +CPU= 66.878
Moving to projects directory: /home/steve/slots/17/ICMGGh7zg+001212
Moving to projects directory: /home/steve/slots/17/ICMSHh7zg+001212
Moving to projects directory: /home/steve/slots/17/ICMUAh7zg+001212
  09:14:14 STEP 1213 H= 303:15 +CPU= 54.661
  09:15:12 STEP 1214 H= 303:30 +CPU= 54.378
  09:16:09 STEP 1215 H= 303:45 +CPU= 53.510
  09:17:06 STEP 1216 H= 304:00 +CPU= 55.947
  09:18:01 STEP 1217 H= 304:15 +CPU= 54.236
  09:18:57 STEP 1218 H= 304:30 +CPU= 55.568
  09:19:52 STEP 1219 H= 304:45 +CPU= 54.373
  09:20:49 STEP 1220 H= 305:00 +CPU= 56.887
  09:21:45 STEP 1221 H= 305:15 +CPU= 55.066
  09:22:41 STEP 1222 H= 305:30 +CPU= 56.169
  09:23:37 STEP 1223 H= 305:45 +CPU= 55.256
  09:24:45 STEP 1224 H= 306:00 +CPU= 67.541
Moving to projects directory: /home/steve/slots/17/ICMGGh7zg+001224
Moving to projects directory: /home/steve/slots/17/ICMSHh7zg+001224
Moving to projects directory: /home/steve/slots/17/ICMUAh7zg+001224
  09:25:41 STEP 1225 H= 306:15 +CPU= 55.020
  09:26:34 STEP 1226 H= 306:30 +CPU= 53.229
  09:27:29 STEP 1227 H= 306:45 +CPU= 53.769
  09:28:26 STEP 1228 H= 307:00 +CPU= 56.929
  09:29:22 STEP 1229 H= 307:15 +CPU= 55.040
  09:30:17 STEP 1230 H= 307:30 +CPU= 54.733
  09:31:12 STEP 1231 H= 307:45 +CPU= 54.523
  09:32:09 STEP 1232 H= 308:00 +CPU= 56.529
  09:33:04 STEP 1233 H= 308:15 +CPU= 54.604
  09:34:00 STEP 1234 H= 308:30 +CPU= 54.347
  09:34:57 STEP 1235 H= 308:45 +CPU= 55.523
  09:36:04 STEP 1236 H= 309:00 +CPU= 66.209
Moving to projects directory: /home/steve/slots/17/ICMGGh7zg+001236
Moving to projects directory: /home/steve/slots/17/ICMSHh7zg+001236
Moving to projects directory: /home/steve/slots/17/ICMUAh7zg+001236
  09:36:59 STEP 1237 H= 309:15 +CPU= 54.224
  09:37:56 STEP 1238 H= 309:30 +CPU= 55.672
  09:38:51 STEP 1239 H= 309:45 +CPU= 54.523
  09:39:47 STEP 1240 H= 310:00 +CPU= 55.927
  09:40:42 STEP 1241 H= 310:15 +CPU= 54.767
  09:41:37 STEP 1242 H= 310:30 +CPU= 54.391
  09:42:32 STEP 1243 H= 310:45 +CPU= 54.057
  09:43:27 STEP 1244 H= 311:00 +CPU= 54.838
  09:44:22 STEP 1245 H= 311:15 +CPU= 53.351
  09:45:17 STEP 1246 H= 311:30 +CPU= 54.365
  09:46:13 STEP 1247 H= 311:45 +CPU= 55.910
  09:47:31 STEP 1248 H= 312:00 +CPU= 76.195
Moving to projects directory: /home/steve/slots/17/ICMGGh7zg+001248
Moving to projects directory: /home/steve/slots/17/ICMSHh7zg+001248
Moving to projects directory: /home/steve/slots/17/ICMUAh7zg+001248
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMGGh7zg+001152
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMSHh7zg+001152
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMUAh7zg+001152
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMGGh7zg+001164
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMSHh7zg+001164
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMUAh7zg+001164
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMGGh7zg+001176
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMSHh7zg+001176
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMUAh7zg+001176
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMGGh7zg+001188
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMSHh7zg+001188
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMUAh7zg+001188
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMGGh7zg+001200
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMSHh7zg+001200
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMUAh7zg+001200
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMGGh7zg+001212
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMSHh7zg+001212
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMUAh7zg+001212
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMGGh7zg+001224
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMSHh7zg+001224
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMUAh7zg+001224
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMGGh7zg+001236
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMSHh7zg+001236
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMUAh7zg+001236
Zipping up the intermediate file: /home/steve/projects/climateprediction.net/oifs_43r3_bl_a1rt_2016092300_20_1017_12284124_1_r1590682354_12.zip
Uploading the intermediate file: upload_file_12.zip
  09:48:26 STEP 1249 H= 312:15 +CPU= 53.502
  09:49:22 STEP 1250 H= 312:30 +CPU= 55.058
  09:50:17 STEP 1251 H= 312:45 +CPU= 54.593
  09:51:14 STEP 1252 H= 313:00 +CPU= 56.058
  09:52:08 STEP 1253 H= 313:15 +CPU= 54.295
  09:53:04 STEP 1254 H= 313:30 +CPU= 54.806
  09:53:58 STEP 1255 H= 313:45 +CPU= 53.711
  09:54:55 STEP 1256 H= 314:00 +CPU= 56.999
  09:55:50 STEP 1257 H= 314:15 +CPU= 53.902
  09:56:47 STEP 1258 H= 314:30 +CPU= 56.096
  09:57:42 STEP 1259 H= 314:45 +CPU= 54.730
  09:58:50 STEP 1260 H= 315:00 +CPU= 67.445
Moving to projects directory: /home/steve/slots/17/ICMGGh7zg+001260
Moving to projects directory: /home/steve/slots/17/ICMSHh7zg+001260
Moving to projects directory: /home/steve/slots/17/ICMUAh7zg+001260
  09:59:46 STEP 1261 H= 315:15 +CPU= 55.143
  10:00:42 STEP 1262 H= 315:30 +CPU= 55.374
  10:01:38 STEP 1263 H= 315:45 +CPU= 56.280
  10:02:36 STEP 1264 H= 316:00 +CPU= 56.988
  10:03:32 STEP 1265 H= 316:15 +CPU= 55.491
  10:04:28 STEP 1266 H= 316:30 +CPU= 55.249
  10:05:24 STEP 1267 H= 316:45 +CPU= 55.524
  10:06:21 STEP 1268 H= 317:00 +CPU= 56.163
  10:07:16 STEP 1269 H= 317:15 +CPU= 55.209
  10:08:13 STEP 1270 H= 317:30 +CPU= 55.978
  10:09:09 STEP 1271 H= 317:45 +CPU= 55.498
  10:10:17 STEP 1272 H= 318:00 +CPU= 67.919
Moving to projects directory: /home/steve/slots/17/ICMGGh7zg+001272
Moving to projects directory: /home/steve/slots/17/ICMSHh7zg+001272
Moving to projects directory: /home/steve/slots/17/ICMUAh7zg+001272
  10:11:13 STEP 1273 H= 318:15 +CPU= 54.696
  10:12:10 STEP 1274 H= 318:30 +CPU= 56.816
  10:13:04 STEP 1275 H= 318:45 +CPU= 53.558
  10:13:59 STEP 1276 H= 319:00 +CPU= 54.240
  10:14:54 STEP 1277 H= 319:15 +CPU= 54.834
  10:15:50 STEP 1278 H= 319:30 +CPU= 55.609
  10:16:45 STEP 1279 H= 319:45 +CPU= 53.579
  10:17:40 STEP 1280 H= 320:00 +CPU= 55.105
  10:18:36 STEP 1281 H= 320:15 +CPU= 54.849
  10:19:31 STEP 1282 H= 320:30 +CPU= 54.876
  10:20:26 STEP 1283 H= 320:45 +CPU= 54.586
  10:21:35 STEP 1284 H= 321:00 +CPU= 68.343
Moving to projects directory: /home/steve/slots/17/ICMGGh7zg+001284
Moving to projects directory: /home/steve/slots/17/ICMSHh7zg+001284
Moving to projects directory: /home/steve/slots/17/ICMUAh7zg+001284
  10:22:30 STEP 1285 H= 321:15 +CPU= 54.008
  10:23:25 STEP 1286 H= 321:30 +CPU= 55.022
  10:24:21 STEP 1287 H= 321:45 +CPU= 55.534
  10:25:17 STEP 1288 H= 322:00 +CPU= 55.505
  10:26:13 STEP 1289 H= 322:15 +CPU= 55.387
  10:27:10 STEP 1290 H= 322:30 +CPU= 56.463
  10:28:06 STEP 1291 H= 322:45 +CPU= 55.109
  10:29:02 STEP 1292 H= 323:00 +CPU= 55.204
  10:29:57 STEP 1293 H= 323:15 +CPU= 54.857
  10:30:54 STEP 1294 H= 323:30 +CPU= 56.265
  10:31:50 STEP 1295 H= 323:45 +CPU= 55.298
  10:33:06 STEP 1296 H= 324:00 +CPU= 75.312
Moving to projects directory: /home/steve/slots/17/ICMGGh7zg+001296
Moving to projects directory: /home/steve/slots/17/ICMSHh7zg+001296
Moving to projects directory: /home/steve/slots/17/ICMUAh7zg+001296
  10:34:00 STEP 1297 H= 324:15 +CPU= 53.510
  10:34:56 STEP 1298 H= 324:30 +CPU= 55.193
  10:35:52 STEP 1299 H= 324:45 +CPU= 55.740
  10:36:48 STEP 1300 H= 325:00 +CPU= 55.329
  10:37:43 STEP 1301 H= 325:15 +CPU= 54.687
  10:38:39 STEP 1302 H= 325:30 +CPU= 55.562
  10:39:35 STEP 1303 H= 325:45 +CPU= 54.874
  10:40:31 STEP 1304 H= 326:00 +CPU= 56.178
  10:41:25 STEP 1305 H= 326:15 +CPU= 53.043
  10:42:20 STEP 1306 H= 326:30 +CPU= 54.812
  10:43:16 STEP 1307 H= 326:45 +CPU= 55.099
  10:44:24 STEP 1308 H= 327:00 +CPU= 67.867
Moving to projects directory: /home/steve/slots/17/ICMGGh7zg+001308
Moving to projects directory: /home/steve/slots/17/ICMSHh7zg+001308
Moving to projects directory: /home/steve/slots/17/ICMUAh7zg+001308
  10:45:20 STEP 1309 H= 327:15 +CPU= 54.693
  10:46:15 STEP 1310 H= 327:30 +CPU= 53.681
  10:47:11 STEP 1311 H= 327:45 +CPU= 54.845
  10:48:07 STEP 1312 H= 328:00 +CPU= 55.746
  10:49:02 STEP 1313 H= 328:15 +CPU= 54.469
  10:49:59 STEP 1314 H= 328:30 +CPU= 56.145
  10:50:54 STEP 1315 H= 328:45 +CPU= 54.806
  10:51:50 STEP 1316 H= 329:00 +CPU= 55.765
  10:52:46 STEP 1317 H= 329:15 +CPU= 54.933
  10:53:40 STEP 1318 H= 329:30 +CPU= 54.243
  10:54:37 STEP 1319 H= 329:45 +CPU= 56.483
  10:55:48 STEP 1320 H= 330:00 +CPU= 68.381
Moving to projects directory: /home/steve/slots/17/ICMGGh7zg+001320
Moving to projects directory: /home/steve/slots/17/ICMSHh7zg+001320
Moving to projects directory: /home/steve/slots/17/ICMUAh7zg+001320
  10:56:46 STEP 1321 H= 330:15 +CPU= 54.325
  10:57:42 STEP 1322 H= 330:30 +CPU= 54.647
  10:58:40 STEP 1323 H= 330:45 +CPU= 54.918
  10:59:36 STEP 1324 H= 331:00 +CPU= 54.844
  11:00:34 STEP 1325 H= 331:15 +CPU= 55.328
  11:01:31 STEP 1326 H= 331:30 +CPU= 53.978
  11:02:31 STEP 1327 H= 331:45 +CPU= 53.069
  11:03:34 STEP 1328 H= 332:00 +CPU= 54.058
  11:04:36 STEP 1329 H= 332:15 +CPU= 54.368
  11:05:38 STEP 1330 H= 332:30 +CPU= 54.562
  11:06:41 STEP 1331 H= 332:45 +CPU= 55.808
  11:07:56 STEP 1332 H= 333:00 +CPU= 68.295
Moving to projects directory: /home/steve/slots/17/ICMGGh7zg+001332
Moving to projects directory: /home/steve/slots/17/ICMSHh7zg+001332
Moving to projects directory: /home/steve/slots/17/ICMUAh7zg+001332
  11:08:58 STEP 1333 H= 333:15 +CPU= 55.318
  11:10:00 STEP 1334 H= 333:30 +CPU= 56.015
  11:11:01 STEP 1335 H= 333:45 +CPU= 53.674
  11:12:07 STEP 1336 H= 334:00 +CPU= 56.653
  11:13:12 STEP 1337 H= 334:15 +CPU= 55.982
  11:14:16 STEP 1338 H= 334:30 +CPU= 57.013
  11:15:20 STEP 1339 H= 334:45 +CPU= 55.812
  11:16:23 STEP 1340 H= 335:00 +CPU= 54.799
  11:17:24 STEP 1341 H= 335:15 +CPU= 55.191
  11:18:24 STEP 1342 H= 335:30 +CPU= 55.571
  11:19:24 STEP 1343 H= 335:45 +CPU= 56.408
  11:20:47 STEP 1344 H= 336:00 +CPU= 77.043
Moving to projects directory: /home/steve/slots/17/ICMGGh7zg+001344
Moving to projects directory: /home/steve/slots/17/ICMSHh7zg+001344
Moving to projects directory: /home/steve/slots/17/ICMUAh7zg+001344
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMGGh7zg+001248
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMSHh7zg+001248
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMUAh7zg+001248
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMGGh7zg+001260
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMSHh7zg+001260
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMUAh7zg+001260
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMGGh7zg+001272
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMSHh7zg+001272
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMUAh7zg+001272
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMGGh7zg+001284
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMSHh7zg+001284
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMUAh7zg+001284
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMGGh7zg+001296
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMSHh7zg+001296
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMUAh7zg+001296
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMGGh7zg+001308
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMSHh7zg+001308
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMUAh7zg+001308
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMGGh7zg+001320
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMSHh7zg+001320
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMUAh7zg+001320
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMGGh7zg+001332
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMSHh7zg+001332
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMUAh7zg+001332
Zipping up the intermediate file: /home/steve/projects/climateprediction.net/oifs_43r3_bl_a1rt_2016092300_20_1017_12284124_1_r1590682354_13.zip
Uploading the intermediate file: upload_file_13.zip
  11:21:46 STEP 1345 H= 336:15 +CPU= 53.707
  11:22:47 STEP 1346 H= 336:30 +CPU= 55.435
  11:23:49 STEP 1347 H= 336:45 +CPU= 56.128
  11:24:50 STEP 1348 H= 337:00 +CPU= 55.890
  11:25:51 STEP 1349 H= 337:15 +CPU= 54.872
  11:26:55 STEP 1350 H= 337:30 +CPU= 55.548
  11:28:00 STEP 1351 H= 337:45 +CPU= 58.594
  11:29:04 STEP 1352 H= 338:00 +CPU= 57.965
  11:30:06 STEP 1353 H= 338:15 +CPU= 56.404
  11:31:11 STEP 1354 H= 338:30 +CPU= 56.585
  11:32:15 STEP 1355 H= 338:45 +CPU= 57.700
  11:33:35 STEP 1356 H= 339:00 +CPU= 70.139
Moving to projects directory: /home/steve/slots/17/ICMGGh7zg+001356
Moving to projects directory: /home/steve/slots/17/ICMSHh7zg+001356
Moving to projects directory: /home/steve/slots/17/ICMUAh7zg+001356
  11:34:39 STEP 1357 H= 339:15 +CPU= 56.226
  11:35:42 STEP 1358 H= 339:30 +CPU= 55.861
  11:36:46 STEP 1359 H= 339:45 +CPU= 56.804
  11:37:49 STEP 1360 H= 340:00 +CPU= 57.004
  11:38:52 STEP 1361 H= 340:15 +CPU= 56.800
  11:39:55 STEP 1362 H= 340:30 +CPU= 56.298
  11:40:59 STEP 1363 H= 340:45 +CPU= 55.901
  11:42:01 STEP 1364 H= 341:00 +CPU= 55.593
  11:43:01 STEP 1365 H= 341:15 +CPU= 56.417
  11:44:02 STEP 1366 H= 341:30 +CPU= 55.621
  11:45:02 STEP 1367 H= 341:45 +CPU= 55.090
  11:46:17 STEP 1368 H= 342:00 +CPU= 68.378
Moving to projects directory: /home/steve/slots/17/ICMGGh7zg+001368
Moving to projects directory: /home/steve/slots/17/ICMSHh7zg+001368
Moving to projects directory: /home/steve/slots/17/ICMUAh7zg+001368
  11:47:12 STEP 1369 H= 342:15 +CPU= 55.025
  11:48:06 STEP 1370 H= 342:30 +CPU= 53.531
  11:49:00 STEP 1371 H= 342:45 +CPU= 53.431
  11:49:56 STEP 1372 H= 343:00 +CPU= 55.112
  11:50:50 STEP 1373 H= 343:15 +CPU= 53.935
  11:51:47 STEP 1374 H= 343:30 +CPU= 56.293
  11:52:44 STEP 1375 H= 343:45 +CPU= 55.844
  11:53:40 STEP 1376 H= 344:00 +CPU= 55.834
  11:54:34 STEP 1377 H= 344:15 +CPU= 53.307
  11:55:30 STEP 1378 H= 344:30 +CPU= 55.102
  11:56:26 STEP 1379 H= 344:45 +CPU= 55.913
  11:57:36 STEP 1380 H= 345:00 +CPU= 69.126
Moving to projects directory: /home/steve/slots/17/ICMGGh7zg+001380
Moving to projects directory: /home/steve/slots/17/ICMSHh7zg+001380
Moving to projects directory: /home/steve/slots/17/ICMUAh7zg+001380
  11:58:30 STEP 1381 H= 345:15 +CPU= 53.387
  11:59:24 STEP 1382 H= 345:30 +CPU= 54.273
  12:00:20 STEP 1383 H= 345:45 +CPU= 54.606
  12:01:15 STEP 1384 H= 346:00 +CPU= 55.109
  12:02:10 STEP 1385 H= 346:15 +CPU= 54.034
  12:03:06 STEP 1386 H= 346:30 +CPU= 55.780
  12:04:01 STEP 1387 H= 346:45 +CPU= 54.629
  12:04:57 STEP 1388 H= 347:00 +CPU= 55.183
  12:05:53 STEP 1389 H= 347:15 +CPU= 55.140
  12:06:49 STEP 1390 H= 347:30 +CPU= 55.574
  12:07:45 STEP 1391 H= 347:45 +CPU= 56.163
  12:09:00 STEP 1392 H= 348:00 +CPU= 73.922
Moving to projects directory: /home/steve/slots/17/ICMGGh7zg+001392
Moving to projects directory: /home/steve/slots/17/ICMSHh7zg+001392
Moving to projects directory: /home/steve/slots/17/ICMUAh7zg+001392
  12:09:56 STEP 1393 H= 348:15 +CPU= 54.817
  12:10:50 STEP 1394 H= 348:30 +CPU= 54.204
  12:11:46 STEP 1395 H= 348:45 +CPU= 54.905
  12:12:40 STEP 1396 H= 349:00 +CPU= 54.051
  12:13:34 STEP 1397 H= 349:15 +CPU= 53.524
  12:14:30 STEP 1398 H= 349:30 +CPU= 54.587
  12:15:24 STEP 1399 H= 349:45 +CPU= 54.006
  12:16:21 STEP 1400 H= 350:00 +CPU= 55.921
  12:17:16 STEP 1401 H= 350:15 +CPU= 54.390
  12:18:10 STEP 1402 H= 350:30 +CPU= 54.063
  12:19:05 STEP 1403 H= 350:45 +CPU= 54.397
  12:20:16 STEP 1404 H= 351:00 +CPU= 69.945
Moving to projects directory: /home/steve/slots/17/ICMGGh7zg+001404
Moving to projects directory: /home/steve/slots/17/ICMSHh7zg+001404
Moving to projects directory: /home/steve/slots/17/ICMUAh7zg+001404
  12:21:10 STEP 1405 H= 351:15 +CPU= 53.558
  12:22:04 STEP 1406 H= 351:30 +CPU= 53.149
  12:23:00 STEP 1407 H= 351:45 +CPU= 54.850
  12:23:55 STEP 1408 H= 352:00 +CPU= 55.001
  12:24:51 STEP 1409 H= 352:15 +CPU= 54.655
  12:25:47 STEP 1410 H= 352:30 +CPU= 56.203
  12:26:43 STEP 1411 H= 352:45 +CPU= 54.802
  12:27:38 STEP 1412 H= 353:00 +CPU= 55.448
  12:28:34 STEP 1413 H= 353:15 +CPU= 54.876
  12:29:31 STEP 1414 H= 353:30 +CPU= 56.264
  12:30:26 STEP 1415 H= 353:45 +CPU= 54.444
  12:31:33 STEP 1416 H= 354:00 +CPU= 67.129
Moving to projects directory: /home/steve/slots/17/ICMGGh7zg+001416
Moving to projects directory: /home/steve/slots/17/ICMSHh7zg+001416
Moving to projects directory: /home/steve/slots/17/ICMUAh7zg+001416
  12:32:27 STEP 1417 H= 354:15 +CPU= 53.297
  12:33:22 STEP 1418 H= 354:30 +CPU= 53.986
  12:34:16 STEP 1419 H= 354:45 +CPU= 53.927
  12:35:10 STEP 1420 H= 355:00 +CPU= 53.897
  12:36:05 STEP 1421 H= 355:15 +CPU= 54.439
  12:37:01 STEP 1422 H= 355:30 +CPU= 55.367
  12:37:56 STEP 1423 H= 355:45 +CPU= 53.640
  12:38:52 STEP 1424 H= 356:00 +CPU= 56.258
  12:39:48 STEP 1425 H= 356:15 +CPU= 55.393
  12:40:44 STEP 1426 H= 356:30 +CPU= 54.664
  12:41:39 STEP 1427 H= 356:45 +CPU= 54.367
  12:42:46 STEP 1428 H= 357:00 +CPU= 66.712
Moving to projects directory: /home/steve/slots/17/ICMGGh7zg+001428
Moving to projects directory: /home/steve/slots/17/ICMSHh7zg+001428
Moving to projects directory: /home/steve/slots/17/ICMUAh7zg+001428
  12:43:42 STEP 1429 H= 357:15 +CPU= 54.856
  12:44:37 STEP 1430 H= 357:30 +CPU= 54.599
  12:45:32 STEP 1431 H= 357:45 +CPU= 54.530
  12:46:28 STEP 1432 H= 358:00 +CPU= 55.312
  12:47:24 STEP 1433 H= 358:15 +CPU= 54.628
  12:48:20 STEP 1434 H= 358:30 +CPU= 55.453
  12:49:13 STEP 1435 H= 358:45 +CPU= 52.852
  12:50:10 STEP 1436 H= 359:00 +CPU= 55.057
  12:51:01 STEP 1437 H= 359:15 +CPU= 50.420
  12:51:50 STEP 1438 H= 359:30 +CPU= 48.664
  12:52:39 STEP 1439 H= 359:45 +CPU= 48.697
  12:53:41 STEP 1440 H= 360:00 +CPU= 61.188
..The child process terminated with status: 0
>>> Printing last 70 lines from file: NODE.001_01

I   VARIABLE   I      Initial value         I        Current value        I
I-------------------------------------------------------------------------I
I  OMEGA (P/S) I     -0.7844337962E-21      I       0.5670751022E-07      I
ICloud fractionI      0.0000000000E+00      I       0.1158187120E-06      I
I  Relat. Hum. I      0.2394004451E+00      I       0.2496157520E+00      I
I  PASS 01     I      0.0000000000E+00      I       0.1228167819E+01      I
DDH------------------------------------------------------------------------
  
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1440 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001440 MODE=a
 IO-STREAM CLOSED - ICMGGh7zg+001440
 MPI-TASK:   1 -    306427716 BYTES IN      7 RECORDS TRANSFERRED IN    0.0008 SECONDS383034.6 Mbytes/s, TOTAL TIME=    0.0369( 2.2%)
 12:53:41 STEP 1440 H= 360:00 +CPU= 61.188
 END CNT3
 NSTEP =  1440 CNT0   000000000  43.242  43.242  43.515   0.994               0               0               0               0               0
IO-STREAM STATISTICS, TOTAL NO OF REC/BYTES READ            0/           0 WRITTEN        43246/************
 -TASK-OPENED-OPEN-RECS IN -KBYTE IN    -RECS OUT-KBYTE OUT-WALL   -WALL IN-WALL
  OU-TOT IN -TOT OUT
   1    758    0        0            0    43246 ************   126.8     0.0   126.8     0.0   284.0
===-=== START OF TIMING STATISTICS ===-===
 
STATS FOR ALL TASKS
 NUM ROUTINE                                     CALLS  MEAN(ms)   MAX(ms)   FRAC(%)  UNBAL(%)
   0 CNT0     - COMPLETE EXECUTION                   1 ********* *********    100.00      0.00
   1 CNT4     - FORWARD INTEGRATION                  1 ********* *********     99.99      0.00
   8 SCAN2M - GRID-POINT DYNAMICS                 1562    9998.6    9998.6     20.27      0.00
   9 SPCM     - SPECTRAL COMP.                    1440    1987.1    1987.1      3.71      0.00
  10 SCAN2M - PHYSICS                             1441   23436.2   23436.2     43.83      0.00
  11 IOPACK   - OUTPUT P.P. RESULTS                121    1696.3    1696.3      0.27      0.00
  12 SPNORM   - SPECTRAL NORM COMP.                 63     119.0     119.0      0.01      0.00
  14 SUINIF                                          1    7045.3    7045.3      0.01      0.00
  17 GRIDFPOS IN CNT4                              121      59.2      59.2      0.01      0.00
  18 SUSPECG                                         1    3672.4    3672.4      0.00      0.00
  19 SUSPEC                                          1    3685.2    3685.2      0.00      0.00
  24 SUGRIDU                                         1    2820.8    2820.8      0.00      0.00
  25 SPECRT                                          1     389.8     389.8      0.00      0.00
  26 SUGRIDF                                         1     149.5     149.5      0.00      0.00
  27 RESTART FILES - WRITING                        30    6841.6    6841.6      0.27      0.00
  28 RESTART FILES - READING                         1       0.0       0.0      0.00      0.00
  29 SU4FPOS IN CNT4                               121       0.1       0.1      0.00      0.00
  30 DYNFPOS IN CNT4                               121   11244.2   11244.2      1.77      0.00
  31 POSDDH IN STEPO                                 7       1.0       1.0      0.00      0.00
  37 CPGLAG   - SL COMPUTATIONS                   1441   10185.6   10185.6     19.05      0.00
  39 SU0YOMB                                         1     556.4     556.4      0.00      0.00
  51 SCAN2M   - SL COMM. PART 1                   1441      41.2      41.2      0.08      0.00
  54 SPCM     - M TO S/S TO M TRANSP.             1440    1087.0    1087.0      2.03      0.00
  55 SPCIMPF  - S TO M/M TO S TRANSP.             1440     274.3     274.3      0.51      0.00
  56 SPNORM   - SPECTRAL NORM COMM.                 63       0.8       0.8      0.00      0.00
 102 LTINV_CTL   - INVERSE LEGENDRE TRANSFORM     3125     775.8     775.8      3.15      0.00
 103 LTDIR_CTL   - DIRECT LEGENDRE TRANSFORM      3006     447.7     447.7      1.75      0.00
 106 FTDIR_CTL   - DIRECT FOURIER TRANSFORM       3006     159.5     159.5      0.62      0.00
 107 FTINV_CTL   - INVERSE FOURIER TRANSFORM      3125     247.7     247.7      1.00      0.00
 140 SULEG       - COMP. OF LEGENDRE POL.            1      50.9      50.9      0.00      0.00
 152 LTINV_CTL   - M TO L TRANSPOSITION           3125     134.7     134.7      0.55      0.00
 153 LTDIR_CTL   - L TO M TRANSPOSITION           3006      81.4      81.4      0.32      0.00
 157 FTINV_CTL   - L TO G TRANSPOSITION           3125     194.1     194.1      0.79      0.00
 158 FTDIR_CTL   - G TO L TRANSPOSITION           3006      83.6      83.6      0.33      0.00
 400 GSTATS                                     167734       0.0       0.0      0.00      0.00
 401 GSTATS HOOK                                156066       0.0       0.0      0.00      0.00
TOTAL MEASURED IMBALANCE =       0.0 SECONDS,  0.0 PERCENT
TOTAL WALLCLOCK TIME    77055.2 CPU TIME   75597.3 VECTOR TIME    75597.3


===-=== END   OF TIMING STATISTICS ===-===
FORECAST DAYS PER DAY       16.8
  *** END CNT0 *** 
------------------------------------------------
Moving to projects directory: /home/steve/slots/17/ICMGGh7zg+001440
Moving to projects directory: /home/steve/slots/17/ICMSHh7zg+001440
Moving to projects directory: /home/steve/slots/17/ICMUAh7zg+001440
Adding to the zip: /home/steve/slots/17/NODE.001_01
Adding to the zip: /home/steve/slots/17/ifs.stat
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMGGh7zg+001344
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMGGh7zg+001428
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMUAh7zg+001392
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMGGh7zg+001368
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMGGh7zg+001392
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMGGh7zg+001356
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMSHh7zg+001404
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMUAh7zg+001416
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMGGh7zg+001380
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMUAh7zg+001440
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMSHh7zg+001344
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMGGh7zg+001440
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMSHh7zg+001368
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMGGh7zg+001416
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMUAh7zg+001404
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMSHh7zg+001380
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMSHh7zg+001392
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMUAh7zg+001380
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMUAh7zg+001344
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMUAh7zg+001428
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMUAh7zg+001368
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMSHh7zg+001416
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMSHh7zg+001440
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMSHh7zg+001428
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMUAh7zg+001356
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMSHh7zg+001356
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284124/ICMGGh7zg+001404
Zipping up the final file: /home/steve/projects/climateprediction.net/oifs_43r3_bl_a1rt_2016092300_20_1017_12284124_1_r1590682354_14.zip
Uploading the final file: upload_file_14.zip
Uploading trickle at timestep: 1295100
12:57:08 (920): called boinc_finish(0)

</stderr_txt>
<message>
upload failure: <file_xfer_error>
  <file_name>oifs_43r3_bl_a1rt_2016092300_20_1017_12284124_1_r1590682354_15.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a1rt_2016092300_20_1017_12284124_1_r1590682354_16.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a1rt_2016092300_20_1017_12284124_1_r1590682354_17.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a1rt_2016092300_20_1017_12284124_1_r1590682354_18.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a1rt_2016092300_20_1017_12284124_1_r1590682354_19.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
</message>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
08 Jun 2024 18:42:10 1536520 22440010 oifs_43r3_bl_a1rt_2016092300_20_1017_12284124_1 1,295,100 75,597 0.0584
08 Jun 2024 11:08:43 1536520 22440010 oifs_43r3_bl_a1rt_2016092300_20_1017_12284124_1 864,000 48,618 0.0563


©2024 climateprediction.net