climateprediction.net home page
Task 22436491

Task 22436491

Name oifs_43r3_bl_a2dd_2016092300_20_1017_12284900_0
Workunit 12284900
Created 7 Jun 2024, 15:17:20 UTC
Sent 7 Jun 2024, 20:28:48 UTC
Report deadline 6 Aug 2024, 20:28:48 UTC
Received 8 Jun 2024, 18:41:28 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 0 (0x00000000)
Computer ID 1536520
Run time 21 hours 27 min 45 sec
CPU time 20 hours 59 min 47 sec
Validate state Invalid
Credit 1,976.32
Device peak FLOPS 7.23 GFLOPS
Application version OpenIFS 43r3 Baroclinic Lifecycle v1.13
x86_64-pc-linux-gnu
Peak working set size 5,024.93 MB
Peak swap size 5,980.80 MB
Peak disk usage 1,357.57 MB
Stderr
<core_client_version>7.22.1</core_client_version>
<![CDATA[
<stderr_txt>
s/19/ICMUAh7zg+000780
  02:15:08 STEP  781 H= 195:15 +CPU= 57.107
  02:16:07 STEP  782 H= 195:30 +CPU= 58.873
  02:17:05 STEP  783 H= 195:45 +CPU= 57.318
  02:18:04 STEP  784 H= 196:00 +CPU= 57.821
  02:19:04 STEP  785 H= 196:15 +CPU= 59.914
  02:20:03 STEP  786 H= 196:30 +CPU= 58.333
  02:21:02 STEP  787 H= 196:45 +CPU= 58.064
  02:22:04 STEP  788 H= 197:00 +CPU= 61.490
  02:23:04 STEP  789 H= 197:15 +CPU= 59.185
  02:24:04 STEP  790 H= 197:30 +CPU= 59.385
  02:25:03 STEP  791 H= 197:45 +CPU= 58.934
  02:26:13 STEP  792 H= 198:00 +CPU= 68.967
Moving to projects directory: /home/steve/slots/19/ICMGGh7zg+000792
Moving to projects directory: /home/steve/slots/19/ICMSHh7zg+000792
Moving to projects directory: /home/steve/slots/19/ICMUAh7zg+000792
  02:27:10 STEP  793 H= 198:15 +CPU= 56.277
  02:28:09 STEP  794 H= 198:30 +CPU= 57.850
  02:29:10 STEP  795 H= 198:45 +CPU= 60.543
  02:30:08 STEP  796 H= 199:00 +CPU= 58.141
  02:31:07 STEP  797 H= 199:15 +CPU= 58.024
  02:32:07 STEP  798 H= 199:30 +CPU= 59.169
  02:33:04 STEP  799 H= 199:45 +CPU= 56.535
  02:34:05 STEP  800 H= 200:00 +CPU= 60.248
  02:35:04 STEP  801 H= 200:15 +CPU= 58.261
  02:36:06 STEP  802 H= 200:30 +CPU= 61.588
  02:37:04 STEP  803 H= 200:45 +CPU= 57.306
  02:38:16 STEP  804 H= 201:00 +CPU= 71.237
Moving to projects directory: /home/steve/slots/19/ICMGGh7zg+000804
Moving to projects directory: /home/steve/slots/19/ICMSHh7zg+000804
Moving to projects directory: /home/steve/slots/19/ICMUAh7zg+000804
  02:39:13 STEP  805 H= 201:15 +CPU= 55.933
  02:40:10 STEP  806 H= 201:30 +CPU= 56.529
  02:41:10 STEP  807 H= 201:45 +CPU= 59.488
  02:42:08 STEP  808 H= 202:00 +CPU= 57.490
  02:43:05 STEP  809 H= 202:15 +CPU= 56.291
  02:44:04 STEP  810 H= 202:30 +CPU= 58.781
  02:45:02 STEP  811 H= 202:45 +CPU= 57.461
  02:46:01 STEP  812 H= 203:00 +CPU= 57.729
  02:46:57 STEP  813 H= 203:15 +CPU= 55.209
  02:47:58 STEP  814 H= 203:30 +CPU= 60.512
  02:48:58 STEP  815 H= 203:45 +CPU= 60.208
  02:50:19 STEP  816 H= 204:00 +CPU= 80.144
Moving to projects directory: /home/steve/slots/19/ICMGGh7zg+000816
Moving to projects directory: /home/steve/slots/19/ICMSHh7zg+000816
Moving to projects directory: /home/steve/slots/19/ICMUAh7zg+000816
  02:51:18 STEP  817 H= 204:15 +CPU= 57.743
  02:52:16 STEP  818 H= 204:30 +CPU= 57.913
  02:53:17 STEP  819 H= 204:45 +CPU= 60.306
  02:54:19 STEP  820 H= 205:00 +CPU= 60.887
  02:55:20 STEP  821 H= 205:15 +CPU= 60.397
  02:56:20 STEP  822 H= 205:30 +CPU= 59.943
  02:57:20 STEP  823 H= 205:45 +CPU= 59.365
  02:58:19 STEP  824 H= 206:00 +CPU= 58.910
  02:59:19 STEP  825 H= 206:15 +CPU= 59.084
  03:00:19 STEP  826 H= 206:30 +CPU= 58.838
  03:01:19 STEP  827 H= 206:45 +CPU= 59.464
  03:02:32 STEP  828 H= 207:00 +CPU= 72.721
Moving to projects directory: /home/steve/slots/19/ICMGGh7zg+000828
Moving to projects directory: /home/steve/slots/19/ICMSHh7zg+000828
Moving to projects directory: /home/steve/slots/19/ICMUAh7zg+000828
  03:03:30 STEP  829 H= 207:15 +CPU= 56.914
  03:04:30 STEP  830 H= 207:30 +CPU= 58.266
  03:05:26 STEP  831 H= 207:45 +CPU= 56.266
  03:06:25 STEP  832 H= 208:00 +CPU= 57.632
  03:07:21 STEP  833 H= 208:15 +CPU= 55.613
  03:08:19 STEP  834 H= 208:30 +CPU= 57.451
  03:09:21 STEP  835 H= 208:45 +CPU= 61.319
  03:10:21 STEP  836 H= 209:00 +CPU= 59.306
  03:11:19 STEP  837 H= 209:15 +CPU= 57.924
  03:12:17 STEP  838 H= 209:30 +CPU= 56.751
  03:13:15 STEP  839 H= 209:45 +CPU= 57.918
  03:14:29 STEP  840 H= 210:00 +CPU= 73.523
Moving to projects directory: /home/steve/slots/19/ICMGGh7zg+000840
Moving to projects directory: /home/steve/slots/19/ICMSHh7zg+000840
Moving to projects directory: /home/steve/slots/19/ICMUAh7zg+000840
  03:15:28 STEP  841 H= 210:15 +CPU= 58.107
  03:16:28 STEP  842 H= 210:30 +CPU= 59.323
  03:17:28 STEP  843 H= 210:45 +CPU= 59.222
  03:18:27 STEP  844 H= 211:00 +CPU= 58.840
  03:19:27 STEP  845 H= 211:15 +CPU= 59.585
  03:20:27 STEP  846 H= 211:30 +CPU= 58.907
  03:21:28 STEP  847 H= 211:45 +CPU= 60.632
  03:22:29 STEP  848 H= 212:00 +CPU= 60.890
  03:23:28 STEP  849 H= 212:15 +CPU= 57.922
  03:24:27 STEP  850 H= 212:30 +CPU= 56.897
  03:25:24 STEP  851 H= 212:45 +CPU= 56.510
  03:26:34 STEP  852 H= 213:00 +CPU= 68.998
Moving to projects directory: /home/steve/slots/19/ICMGGh7zg+000852
Moving to projects directory: /home/steve/slots/19/ICMSHh7zg+000852
Moving to projects directory: /home/steve/slots/19/ICMUAh7zg+000852
  03:27:31 STEP  853 H= 213:15 +CPU= 56.138
  03:28:31 STEP  854 H= 213:30 +CPU= 59.439
  03:29:33 STEP  855 H= 213:45 +CPU= 61.123
  03:30:38 STEP  856 H= 214:00 +CPU= 64.502
  03:31:35 STEP  857 H= 214:15 +CPU= 56.429
  03:32:34 STEP  858 H= 214:30 +CPU= 58.497
  03:33:30 STEP  859 H= 214:45 +CPU= 55.278
  03:34:29 STEP  860 H= 215:00 +CPU= 58.551
  03:35:26 STEP  861 H= 215:15 +CPU= 56.025
  03:36:26 STEP  862 H= 215:30 +CPU= 59.248
  03:37:27 STEP  863 H= 215:45 +CPU= 60.312
  03:38:50 STEP  864 H= 216:00 +CPU= 81.990
Moving to projects directory: /home/steve/slots/19/ICMGGh7zg+000864
Moving to projects directory: /home/steve/slots/19/ICMSHh7zg+000864
Moving to projects directory: /home/steve/slots/19/ICMUAh7zg+000864
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMGGh7zg+000768
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMSHh7zg+000768
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMUAh7zg+000768
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMGGh7zg+000780
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMSHh7zg+000780
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMUAh7zg+000780
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMGGh7zg+000792
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMSHh7zg+000792
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMUAh7zg+000792
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMGGh7zg+000804
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMSHh7zg+000804
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMUAh7zg+000804
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMGGh7zg+000816
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMSHh7zg+000816
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMUAh7zg+000816
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMGGh7zg+000828
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMSHh7zg+000828
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMUAh7zg+000828
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMGGh7zg+000840
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMSHh7zg+000840
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMUAh7zg+000840
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMGGh7zg+000852
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMSHh7zg+000852
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMUAh7zg+000852
Zipping up the intermediate file: /home/steve/projects/climateprediction.net/oifs_43r3_bl_a2dd_2016092300_20_1017_12284900_0_r64603195_8.zip
Uploading the intermediate file: upload_file_8.zip
  03:39:50 STEP  865 H= 216:15 +CPU= 57.813
  03:40:48 STEP  866 H= 216:30 +CPU= 58.003
  03:41:48 STEP  867 H= 216:45 +CPU= 59.636
  03:42:47 STEP  868 H= 217:00 +CPU= 58.082
  03:43:44 STEP  869 H= 217:15 +CPU= 56.429
  03:44:44 STEP  870 H= 217:30 +CPU= 58.910
  03:45:40 STEP  871 H= 217:45 +CPU= 55.766
  03:46:37 STEP  872 H= 218:00 +CPU= 56.571
  03:47:34 STEP  873 H= 218:15 +CPU= 56.446
  03:48:32 STEP  874 H= 218:30 +CPU= 57.355
  03:49:32 STEP  875 H= 218:45 +CPU= 59.027
  03:50:43 STEP  876 H= 219:00 +CPU= 70.796
Moving to projects directory: /home/steve/slots/19/ICMGGh7zg+000876
Moving to projects directory: /home/steve/slots/19/ICMSHh7zg+000876
Moving to projects directory: /home/steve/slots/19/ICMUAh7zg+000876
  03:51:41 STEP  877 H= 219:15 +CPU= 56.679
  03:52:42 STEP  878 H= 219:30 +CPU= 60.658
  03:53:39 STEP  879 H= 219:45 +CPU= 56.913
  03:54:37 STEP  880 H= 220:00 +CPU= 57.014
  03:55:35 STEP  881 H= 220:15 +CPU= 57.250
  03:56:35 STEP  882 H= 220:30 +CPU= 59.450
  03:57:32 STEP  883 H= 220:45 +CPU= 56.743
  03:58:32 STEP  884 H= 221:00 +CPU= 59.815
  03:59:29 STEP  885 H= 221:15 +CPU= 55.941
  04:00:32 STEP  886 H= 221:30 +CPU= 62.603
  04:01:32 STEP  887 H= 221:45 +CPU= 58.725
  04:02:45 STEP  888 H= 222:00 +CPU= 73.008
Moving to projects directory: /home/steve/slots/19/ICMGGh7zg+000888
Moving to projects directory: /home/steve/slots/19/ICMSHh7zg+000888
Moving to projects directory: /home/steve/slots/19/ICMUAh7zg+000888
  04:03:44 STEP  889 H= 222:15 +CPU= 57.867
  04:04:39 STEP  890 H= 222:30 +CPU= 54.853
  04:05:36 STEP  891 H= 222:45 +CPU= 56.353
  04:06:32 STEP  892 H= 223:00 +CPU= 55.495
  04:07:27 STEP  893 H= 223:15 +CPU= 53.039
  04:08:24 STEP  894 H= 223:30 +CPU= 55.471
  04:09:19 STEP  895 H= 223:45 +CPU= 55.025
  04:10:17 STEP  896 H= 224:00 +CPU= 57.596
  04:11:13 STEP  897 H= 224:15 +CPU= 54.939
  04:12:09 STEP  898 H= 224:30 +CPU= 55.998
  04:13:05 STEP  899 H= 224:45 +CPU= 54.890
  04:14:14 STEP  900 H= 225:00 +CPU= 68.767
Moving to projects directory: /home/steve/slots/19/ICMGGh7zg+000900
Moving to projects directory: /home/steve/slots/19/ICMSHh7zg+000900
Moving to projects directory: /home/steve/slots/19/ICMUAh7zg+000900
  04:15:10 STEP  901 H= 225:15 +CPU= 54.894
  04:16:04 STEP  902 H= 225:30 +CPU= 54.221
  04:17:00 STEP  903 H= 225:45 +CPU= 54.804
  04:17:56 STEP  904 H= 226:00 +CPU= 55.438
  04:18:51 STEP  905 H= 226:15 +CPU= 54.487
  04:19:47 STEP  906 H= 226:30 +CPU= 55.744
  04:20:43 STEP  907 H= 226:45 +CPU= 55.435
  04:21:41 STEP  908 H= 227:00 +CPU= 57.077
  04:22:36 STEP  909 H= 227:15 +CPU= 55.338
  04:23:31 STEP  910 H= 227:30 +CPU= 54.501
  04:24:26 STEP  911 H= 227:45 +CPU= 53.830
  04:25:42 STEP  912 H= 228:00 +CPU= 74.821
Moving to projects directory: /home/steve/slots/19/ICMGGh7zg+000912
Moving to projects directory: /home/steve/slots/19/ICMSHh7zg+000912
Moving to projects directory: /home/steve/slots/19/ICMUAh7zg+000912
  04:26:36 STEP  913 H= 228:15 +CPU= 53.360
  04:27:31 STEP  914 H= 228:30 +CPU= 54.578
  04:28:26 STEP  915 H= 228:45 +CPU= 54.319
  04:29:23 STEP  916 H= 229:00 +CPU= 56.696
  04:30:17 STEP  917 H= 229:15 +CPU= 53.910
  04:31:12 STEP  918 H= 229:30 +CPU= 54.166
  04:32:07 STEP  919 H= 229:45 +CPU= 54.979
  04:33:03 STEP  920 H= 230:00 +CPU= 55.147
  04:33:56 STEP  921 H= 230:15 +CPU= 52.424
  04:34:52 STEP  922 H= 230:30 +CPU= 55.802
  04:35:48 STEP  923 H= 230:45 +CPU= 55.348
  04:36:56 STEP  924 H= 231:00 +CPU= 67.064
Moving to projects directory: /home/steve/slots/19/ICMGGh7zg+000924
Moving to projects directory: /home/steve/slots/19/ICMSHh7zg+000924
Moving to projects directory: /home/steve/slots/19/ICMUAh7zg+000924
  04:37:50 STEP  925 H= 231:15 +CPU= 53.449
  04:38:45 STEP  926 H= 231:30 +CPU= 54.117
  04:39:41 STEP  927 H= 231:45 +CPU= 55.317
  04:40:39 STEP  928 H= 232:00 +CPU= 56.802
  04:41:35 STEP  929 H= 232:15 +CPU= 55.960
  04:42:31 STEP  930 H= 232:30 +CPU= 55.297
  04:43:24 STEP  931 H= 232:45 +CPU= 52.658
  04:44:22 STEP  932 H= 233:00 +CPU= 57.442
  04:45:17 STEP  933 H= 233:15 +CPU= 54.616
  04:46:12 STEP  934 H= 233:30 +CPU= 54.646
  04:47:08 STEP  935 H= 233:45 +CPU= 54.978
  04:48:15 STEP  936 H= 234:00 +CPU= 66.696
Moving to projects directory: /home/steve/slots/19/ICMGGh7zg+000936
Moving to projects directory: /home/steve/slots/19/ICMSHh7zg+000936
Moving to projects directory: /home/steve/slots/19/ICMUAh7zg+000936
  04:49:10 STEP  937 H= 234:15 +CPU= 54.273
  04:50:05 STEP  938 H= 234:30 +CPU= 54.397
  04:50:59 STEP  939 H= 234:45 +CPU= 53.873
  04:51:55 STEP  940 H= 235:00 +CPU= 55.408
  04:52:51 STEP  941 H= 235:15 +CPU= 54.827
  04:53:49 STEP  942 H= 235:30 +CPU= 57.515
  04:54:43 STEP  943 H= 235:45 +CPU= 54.361
  04:55:39 STEP  944 H= 236:00 +CPU= 55.156
  04:56:34 STEP  945 H= 236:15 +CPU= 54.390
  04:57:28 STEP  946 H= 236:30 +CPU= 52.347
  04:58:25 STEP  947 H= 236:45 +CPU= 56.176
  04:59:35 STEP  948 H= 237:00 +CPU= 68.715
Moving to projects directory: /home/steve/slots/19/ICMGGh7zg+000948
Moving to projects directory: /home/steve/slots/19/ICMSHh7zg+000948
Moving to projects directory: /home/steve/slots/19/ICMUAh7zg+000948
  05:00:29 STEP  949 H= 237:15 +CPU= 53.710
  05:01:25 STEP  950 H= 237:30 +CPU= 55.226
  05:02:22 STEP  951 H= 237:45 +CPU= 56.694
  05:03:18 STEP  952 H= 238:00 +CPU= 55.888
  05:04:13 STEP  953 H= 238:15 +CPU= 54.286
  05:05:09 STEP  954 H= 238:30 +CPU= 55.160
  05:06:06 STEP  955 H= 238:45 +CPU= 56.611
  05:07:01 STEP  956 H= 239:00 +CPU= 54.950
  05:07:56 STEP  957 H= 239:15 +CPU= 53.615
  05:08:52 STEP  958 H= 239:30 +CPU= 55.112
  05:09:47 STEP  959 H= 239:45 +CPU= 54.500
  05:11:04 STEP  960 H= 240:00 +CPU= 77.021
Moving to projects directory: /home/steve/slots/19/ICMGGh7zg+000960
Moving to projects directory: /home/steve/slots/19/ICMSHh7zg+000960
Moving to projects directory: /home/steve/slots/19/ICMUAh7zg+000960
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMGGh7zg+000864
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMSHh7zg+000864
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMUAh7zg+000864
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMGGh7zg+000876
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMSHh7zg+000876
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMUAh7zg+000876
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMGGh7zg+000888
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMSHh7zg+000888
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMUAh7zg+000888
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMGGh7zg+000900
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMSHh7zg+000900
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMUAh7zg+000900
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMGGh7zg+000912
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMSHh7zg+000912
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMUAh7zg+000912
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMGGh7zg+000924
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMSHh7zg+000924
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMUAh7zg+000924
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMGGh7zg+000936
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMSHh7zg+000936
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMUAh7zg+000936
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMGGh7zg+000948
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMSHh7zg+000948
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMUAh7zg+000948
Zipping up the intermediate file: /home/steve/projects/climateprediction.net/oifs_43r3_bl_a2dd_2016092300_20_1017_12284900_0_r64603195_9.zip
Uploading the intermediate file: upload_file_9.zip
Uploading trickle at timestep: 864000
  05:12:01 STEP  961 H= 240:15 +CPU= 53.792
  05:12:56 STEP  962 H= 240:30 +CPU= 54.705
  05:13:50 STEP  963 H= 240:45 +CPU= 53.961
  05:14:47 STEP  964 H= 241:00 +CPU= 55.642
  05:15:44 STEP  965 H= 241:15 +CPU= 56.537
  05:16:39 STEP  966 H= 241:30 +CPU= 54.673
  05:17:34 STEP  967 H= 241:45 +CPU= 54.191
  05:18:31 STEP  968 H= 242:00 +CPU= 57.108
  05:19:27 STEP  969 H= 242:15 +CPU= 54.902
  05:20:23 STEP  970 H= 242:30 +CPU= 55.793
  05:21:20 STEP  971 H= 242:45 +CPU= 57.013
  05:22:30 STEP  972 H= 243:00 +CPU= 68.375
Moving to projects directory: /home/steve/slots/19/ICMGGh7zg+000972
Moving to projects directory: /home/steve/slots/19/ICMSHh7zg+000972
Moving to projects directory: /home/steve/slots/19/ICMUAh7zg+000972
  05:23:25 STEP  973 H= 243:15 +CPU= 54.410
  05:24:21 STEP  974 H= 243:30 +CPU= 55.451
  05:25:17 STEP  975 H= 243:45 +CPU= 55.989
  05:26:13 STEP  976 H= 244:00 +CPU= 55.158
  05:27:08 STEP  977 H= 244:15 +CPU= 54.481
  05:28:04 STEP  978 H= 244:30 +CPU= 56.242
  05:29:00 STEP  979 H= 244:45 +CPU= 55.550
  05:29:54 STEP  980 H= 245:00 +CPU= 53.338
  05:30:48 STEP  981 H= 245:15 +CPU= 53.293
  05:31:44 STEP  982 H= 245:30 +CPU= 55.380
  05:32:40 STEP  983 H= 245:45 +CPU= 55.588
  05:33:50 STEP  984 H= 246:00 +CPU= 69.261
Moving to projects directory: /home/steve/slots/19/ICMGGh7zg+000984
Moving to projects directory: /home/steve/slots/19/ICMSHh7zg+000984
Moving to projects directory: /home/steve/slots/19/ICMUAh7zg+000984
  05:34:47 STEP  985 H= 246:15 +CPU= 56.054
  05:35:44 STEP  986 H= 246:30 +CPU= 56.457
  05:36:39 STEP  987 H= 246:45 +CPU= 54.867
  05:37:36 STEP  988 H= 247:00 +CPU= 56.877
  05:38:32 STEP  989 H= 247:15 +CPU= 55.163
  05:39:28 STEP  990 H= 247:30 +CPU= 55.266
  05:40:23 STEP  991 H= 247:45 +CPU= 54.236
  05:41:19 STEP  992 H= 248:00 +CPU= 55.651
  05:42:15 STEP  993 H= 248:15 +CPU= 55.122
  05:43:09 STEP  994 H= 248:30 +CPU= 53.999
  05:44:05 STEP  995 H= 248:45 +CPU= 55.542
  05:45:14 STEP  996 H= 249:00 +CPU= 67.999
Moving to projects directory: /home/steve/slots/19/ICMGGh7zg+000996
Moving to projects directory: /home/steve/slots/19/ICMSHh7zg+000996
Moving to projects directory: /home/steve/slots/19/ICMUAh7zg+000996
  05:46:08 STEP  997 H= 249:15 +CPU= 53.697
  05:47:05 STEP  998 H= 249:30 +CPU= 55.869
  05:48:01 STEP  999 H= 249:45 +CPU= 55.494
  05:48:56 STEP 1000 H= 250:00 +CPU= 55.307
  05:49:53 STEP 1001 H= 250:15 +CPU= 55.974
  05:50:48 STEP 1002 H= 250:30 +CPU= 54.937
  05:51:43 STEP 1003 H= 250:45 +CPU= 54.619
  05:52:39 STEP 1004 H= 251:00 +CPU= 55.320
  05:53:33 STEP 1005 H= 251:15 +CPU= 53.391
  05:54:29 STEP 1006 H= 251:30 +CPU= 55.307
  05:55:24 STEP 1007 H= 251:45 +CPU= 54.628
  05:56:42 STEP 1008 H= 252:00 +CPU= 76.682
Moving to projects directory: /home/steve/slots/19/ICMGGh7zg+001008
Moving to projects directory: /home/steve/slots/19/ICMSHh7zg+001008
Moving to projects directory: /home/steve/slots/19/ICMUAh7zg+001008
  05:57:38 STEP 1009 H= 252:15 +CPU= 55.318
  05:58:32 STEP 1010 H= 252:30 +CPU= 54.314
  05:59:28 STEP 1011 H= 252:45 +CPU= 54.564
  06:00:23 STEP 1012 H= 253:00 +CPU= 54.834
  06:01:16 STEP 1013 H= 253:15 +CPU= 53.127
  06:02:11 STEP 1014 H= 253:30 +CPU= 54.458
  06:03:07 STEP 1015 H= 253:45 +CPU= 54.958
  06:04:02 STEP 1016 H= 254:00 +CPU= 54.945
  06:04:57 STEP 1017 H= 254:15 +CPU= 53.783
  06:05:52 STEP 1018 H= 254:30 +CPU= 55.097
  06:06:48 STEP 1019 H= 254:45 +CPU= 55.672
  06:07:57 STEP 1020 H= 255:00 +CPU= 67.865
Moving to projects directory: /home/steve/slots/19/ICMGGh7zg+001020
Moving to projects directory: /home/steve/slots/19/ICMSHh7zg+001020
Moving to projects directory: /home/steve/slots/19/ICMUAh7zg+001020
  06:08:52 STEP 1021 H= 255:15 +CPU= 54.671
  06:09:49 STEP 1022 H= 255:30 +CPU= 55.133
  06:10:45 STEP 1023 H= 255:45 +CPU= 55.108
  06:11:42 STEP 1024 H= 256:00 +CPU= 56.626
  06:12:37 STEP 1025 H= 256:15 +CPU= 54.826
  06:13:33 STEP 1026 H= 256:30 +CPU= 54.994
  06:14:29 STEP 1027 H= 256:45 +CPU= 55.247
  06:15:26 STEP 1028 H= 257:00 +CPU= 55.316
  06:16:21 STEP 1029 H= 257:15 +CPU= 54.115
  06:17:17 STEP 1030 H= 257:30 +CPU= 55.421
  06:18:11 STEP 1031 H= 257:45 +CPU= 53.879
  06:19:19 STEP 1032 H= 258:00 +CPU= 67.692
Moving to projects directory: /home/steve/slots/19/ICMGGh7zg+001032
Moving to projects directory: /home/steve/slots/19/ICMSHh7zg+001032
Moving to projects directory: /home/steve/slots/19/ICMUAh7zg+001032
  06:20:14 STEP 1033 H= 258:15 +CPU= 53.967
  06:21:14 STEP 1034 H= 258:30 +CPU= 51.565
  06:22:12 STEP 1035 H= 258:45 +CPU= 53.194
  06:23:11 STEP 1036 H= 259:00 +CPU= 55.022
  06:24:06 STEP 1037 H= 259:15 +CPU= 54.096
  06:25:05 STEP 1038 H= 259:30 +CPU= 53.275
  06:26:01 STEP 1039 H= 259:45 +CPU= 54.773
  06:26:58 STEP 1040 H= 260:00 +CPU= 54.930
  06:27:59 STEP 1041 H= 260:15 +CPU= 55.290
  06:28:56 STEP 1042 H= 260:30 +CPU= 52.992
  06:29:53 STEP 1043 H= 260:45 +CPU= 54.695
  06:31:04 STEP 1044 H= 261:00 +CPU= 67.749
Moving to projects directory: /home/steve/slots/19/ICMGGh7zg+001044
Moving to projects directory: /home/steve/slots/19/ICMSHh7zg+001044
Moving to projects directory: /home/steve/slots/19/ICMUAh7zg+001044
  06:32:04 STEP 1045 H= 261:15 +CPU= 56.178
  06:33:05 STEP 1046 H= 261:30 +CPU= 55.911
  06:34:06 STEP 1047 H= 261:45 +CPU= 56.507
  06:35:05 STEP 1048 H= 262:00 +CPU= 55.624
  06:36:00 STEP 1049 H= 262:15 +CPU= 54.476
  06:36:55 STEP 1050 H= 262:30 +CPU= 54.809
  06:37:54 STEP 1051 H= 262:45 +CPU= 56.039
  06:38:52 STEP 1052 H= 263:00 +CPU= 54.656
  06:39:51 STEP 1053 H= 263:15 +CPU= 53.564
  06:40:49 STEP 1054 H= 263:30 +CPU= 55.630
  06:41:46 STEP 1055 H= 263:45 +CPU= 53.605
  06:43:05 STEP 1056 H= 264:00 +CPU= 74.169
Moving to projects directory: /home/steve/slots/19/ICMGGh7zg+001056
Moving to projects directory: /home/steve/slots/19/ICMSHh7zg+001056
Moving to projects directory: /home/steve/slots/19/ICMUAh7zg+001056
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMGGh7zg+000960
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMSHh7zg+000960
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMUAh7zg+000960
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMGGh7zg+000972
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMSHh7zg+000972
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMUAh7zg+000972
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMGGh7zg+000984
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMSHh7zg+000984
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMUAh7zg+000984
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMGGh7zg+000996
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMSHh7zg+000996
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMUAh7zg+000996
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMGGh7zg+001008
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMSHh7zg+001008
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMUAh7zg+001008
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMGGh7zg+001020
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMSHh7zg+001020
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMUAh7zg+001020
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMGGh7zg+001032
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMSHh7zg+001032
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMUAh7zg+001032
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMGGh7zg+001044
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMSHh7zg+001044
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMUAh7zg+001044
Zipping up the intermediate file: /home/steve/projects/climateprediction.net/oifs_43r3_bl_a2dd_2016092300_20_1017_12284900_0_r64603195_10.zip
Uploading the intermediate file: upload_file_10.zip
  06:43:59 STEP 1057 H= 264:15 +CPU= 51.700
  06:44:57 STEP 1058 H= 264:30 +CPU= 55.011
  06:45:55 STEP 1059 H= 264:45 +CPU= 55.592
  06:46:52 STEP 1060 H= 265:00 +CPU= 55.271
  06:47:49 STEP 1061 H= 265:15 +CPU= 54.927
  06:48:48 STEP 1062 H= 265:30 +CPU= 54.967
  06:49:45 STEP 1063 H= 265:45 +CPU= 55.030
  06:50:42 STEP 1064 H= 266:00 +CPU= 56.424
  06:51:37 STEP 1065 H= 266:15 +CPU= 53.911
  06:52:34 STEP 1066 H= 266:30 +CPU= 54.321
  06:53:30 STEP 1067 H= 266:45 +CPU= 54.242
  06:54:40 STEP 1068 H= 267:00 +CPU= 69.264
Moving to projects directory: /home/steve/slots/19/ICMGGh7zg+001068
Moving to projects directory: /home/steve/slots/19/ICMSHh7zg+001068
Moving to projects directory: /home/steve/slots/19/ICMUAh7zg+001068
  06:55:38 STEP 1069 H= 267:15 +CPU= 55.612
  06:56:34 STEP 1070 H= 267:30 +CPU= 54.813
  06:57:31 STEP 1071 H= 267:45 +CPU= 54.849
  06:58:29 STEP 1072 H= 268:00 +CPU= 54.777
  06:59:26 STEP 1073 H= 268:15 +CPU= 55.674
  07:00:27 STEP 1074 H= 268:30 +CPU= 55.313
  07:01:24 STEP 1075 H= 268:45 +CPU= 54.988
  07:02:23 STEP 1076 H= 269:00 +CPU= 56.362
  07:03:21 STEP 1077 H= 269:15 +CPU= 55.140
  07:04:18 STEP 1078 H= 269:30 +CPU= 55.205
  07:05:15 STEP 1079 H= 269:45 +CPU= 55.370
  07:06:23 STEP 1080 H= 270:00 +CPU= 68.147
Moving to projects directory: /home/steve/slots/19/ICMGGh7zg+001080
Moving to projects directory: /home/steve/slots/19/ICMSHh7zg+001080
Moving to projects directory: /home/steve/slots/19/ICMUAh7zg+001080
  07:07:20 STEP 1081 H= 270:15 +CPU= 56.146
  07:08:15 STEP 1082 H= 270:30 +CPU= 54.606
  07:09:11 STEP 1083 H= 270:45 +CPU= 54.751
  07:10:06 STEP 1084 H= 271:00 +CPU= 54.932
  07:11:02 STEP 1085 H= 271:15 +CPU= 55.173
  07:11:58 STEP 1086 H= 271:30 +CPU= 55.345
  07:12:53 STEP 1087 H= 271:45 +CPU= 54.532
  07:13:49 STEP 1088 H= 272:00 +CPU= 55.372
  07:14:44 STEP 1089 H= 272:15 +CPU= 54.987
  07:15:40 STEP 1090 H= 272:30 +CPU= 54.723
  07:16:36 STEP 1091 H= 272:45 +CPU= 55.431
  07:17:42 STEP 1092 H= 273:00 +CPU= 66.028
Moving to projects directory: /home/steve/slots/19/ICMGGh7zg+001092
Moving to projects directory: /home/steve/slots/19/ICMSHh7zg+001092
Moving to projects directory: /home/steve/slots/19/ICMUAh7zg+001092
  07:18:37 STEP 1093 H= 273:15 +CPU= 54.154
  07:19:33 STEP 1094 H= 273:30 +CPU= 55.790
  07:20:29 STEP 1095 H= 273:45 +CPU= 55.161
  07:21:27 STEP 1096 H= 274:00 +CPU= 57.521
  07:22:22 STEP 1097 H= 274:15 +CPU= 54.407
  07:23:16 STEP 1098 H= 274:30 +CPU= 53.491
  07:24:12 STEP 1099 H= 274:45 +CPU= 54.924
  07:25:10 STEP 1100 H= 275:00 +CPU= 55.575
  07:26:08 STEP 1101 H= 275:15 +CPU= 54.462
  07:27:05 STEP 1102 H= 275:30 +CPU= 53.346
  07:28:02 STEP 1103 H= 275:45 +CPU= 56.055
  07:29:22 STEP 1104 H= 276:00 +CPU= 75.031
Moving to projects directory: /home/steve/slots/19/ICMGGh7zg+001104
Moving to projects directory: /home/steve/slots/19/ICMSHh7zg+001104
Moving to projects directory: /home/steve/slots/19/ICMUAh7zg+001104
  07:30:20 STEP 1105 H= 276:15 +CPU= 54.691
  07:31:17 STEP 1106 H= 276:30 +CPU= 55.526
  07:32:16 STEP 1107 H= 276:45 +CPU= 55.563
  07:33:14 STEP 1108 H= 277:00 +CPU= 55.959
  07:34:13 STEP 1109 H= 277:15 +CPU= 56.031
  07:35:12 STEP 1110 H= 277:30 +CPU= 54.152
  07:36:08 STEP 1111 H= 277:45 +CPU= 54.372
  07:37:07 STEP 1112 H= 278:00 +CPU= 55.240
  07:38:04 STEP 1113 H= 278:15 +CPU= 55.869
  07:39:04 STEP 1114 H= 278:30 +CPU= 56.728
  07:39:59 STEP 1115 H= 278:45 +CPU= 53.898
  07:41:10 STEP 1116 H= 279:00 +CPU= 68.144
Moving to projects directory: /home/steve/slots/19/ICMGGh7zg+001116
Moving to projects directory: /home/steve/slots/19/ICMSHh7zg+001116
Moving to projects directory: /home/steve/slots/19/ICMUAh7zg+001116
  07:42:07 STEP 1117 H= 279:15 +CPU= 55.357
  07:43:05 STEP 1118 H= 279:30 +CPU= 54.833
  07:44:01 STEP 1119 H= 279:45 +CPU= 55.220
  07:44:58 STEP 1120 H= 280:00 +CPU= 55.841
  07:46:02 STEP 1121 H= 280:15 +CPU= 54.548
  07:47:04 STEP 1122 H= 280:30 +CPU= 54.311
  07:48:04 STEP 1123 H= 280:45 +CPU= 53.723
  07:49:04 STEP 1124 H= 281:00 +CPU= 55.057
  07:50:01 STEP 1125 H= 281:15 +CPU= 54.447
  07:50:57 STEP 1126 H= 281:30 +CPU= 54.542
  07:51:55 STEP 1127 H= 281:45 +CPU= 55.058
  07:53:07 STEP 1128 H= 282:00 +CPU= 68.660
Moving to projects directory: /home/steve/slots/19/ICMGGh7zg+001128
Moving to projects directory: /home/steve/slots/19/ICMSHh7zg+001128
Moving to projects directory: /home/steve/slots/19/ICMUAh7zg+001128
  07:54:03 STEP 1129 H= 282:15 +CPU= 53.769
  07:55:01 STEP 1130 H= 282:30 +CPU= 54.918
  07:55:58 STEP 1131 H= 282:45 +CPU= 54.628
  07:56:58 STEP 1132 H= 283:00 +CPU= 56.855
  07:57:56 STEP 1133 H= 283:15 +CPU= 54.540
  07:58:55 STEP 1134 H= 283:30 +CPU= 56.009
  07:59:51 STEP 1135 H= 283:45 +CPU= 55.101
  08:00:48 STEP 1136 H= 284:00 +CPU= 56.539
  08:01:45 STEP 1137 H= 284:15 +CPU= 56.238
  08:02:41 STEP 1138 H= 284:30 +CPU= 54.653
  08:03:37 STEP 1139 H= 284:45 +CPU= 54.500
  08:04:45 STEP 1140 H= 285:00 +CPU= 67.427
Moving to projects directory: /home/steve/slots/19/ICMGGh7zg+001140
Moving to projects directory: /home/steve/slots/19/ICMSHh7zg+001140
Moving to projects directory: /home/steve/slots/19/ICMUAh7zg+001140
  08:05:40 STEP 1141 H= 285:15 +CPU= 54.375
  08:06:36 STEP 1142 H= 285:30 +CPU= 55.267
  08:07:32 STEP 1143 H= 285:45 +CPU= 55.468
  08:08:27 STEP 1144 H= 286:00 +CPU= 55.193
  08:09:22 STEP 1145 H= 286:15 +CPU= 54.387
  08:10:18 STEP 1146 H= 286:30 +CPU= 55.454
  08:11:13 STEP 1147 H= 286:45 +CPU= 54.595
  08:12:09 STEP 1148 H= 287:00 +CPU= 54.885
  08:13:04 STEP 1149 H= 287:15 +CPU= 54.110
  08:13:59 STEP 1150 H= 287:30 +CPU= 54.417
  08:14:54 STEP 1151 H= 287:45 +CPU= 54.895
  08:16:13 STEP 1152 H= 288:00 +CPU= 77.361
Moving to projects directory: /home/steve/slots/19/ICMGGh7zg+001152
Moving to projects directory: /home/steve/slots/19/ICMSHh7zg+001152
Moving to projects directory: /home/steve/slots/19/ICMUAh7zg+001152
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMGGh7zg+001056
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMSHh7zg+001056
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMUAh7zg+001056
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMGGh7zg+001068
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMSHh7zg+001068
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMUAh7zg+001068
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMGGh7zg+001080
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMSHh7zg+001080
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMUAh7zg+001080
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMGGh7zg+001092
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMSHh7zg+001092
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMUAh7zg+001092
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMGGh7zg+001104
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMSHh7zg+001104
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMUAh7zg+001104
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMGGh7zg+001116
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMSHh7zg+001116
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMUAh7zg+001116
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMGGh7zg+001128
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMSHh7zg+001128
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMUAh7zg+001128
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMGGh7zg+001140
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMSHh7zg+001140
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMUAh7zg+001140
Zipping up the intermediate file: /home/steve/projects/climateprediction.net/oifs_43r3_bl_a2dd_2016092300_20_1017_12284900_0_r64603195_11.zip
Uploading the intermediate file: upload_file_11.zip
  08:17:07 STEP 1153 H= 288:15 +CPU= 51.855
  08:18:02 STEP 1154 H= 288:30 +CPU= 54.444
  08:18:58 STEP 1155 H= 288:45 +CPU= 55.764
  08:19:56 STEP 1156 H= 289:00 +CPU= 56.702
  08:20:50 STEP 1157 H= 289:15 +CPU= 53.805
  08:21:44 STEP 1158 H= 289:30 +CPU= 53.346
  08:22:40 STEP 1159 H= 289:45 +CPU= 55.603
  08:23:37 STEP 1160 H= 290:00 +CPU= 56.736
  08:24:32 STEP 1161 H= 290:15 +CPU= 54.058
  08:25:28 STEP 1162 H= 290:30 +CPU= 55.146
  08:26:24 STEP 1163 H= 290:45 +CPU= 55.750
  08:27:33 STEP 1164 H= 291:00 +CPU= 67.970
Moving to projects directory: /home/steve/slots/19/ICMGGh7zg+001164
Moving to projects directory: /home/steve/slots/19/ICMSHh7zg+001164
Moving to projects directory: /home/steve/slots/19/ICMUAh7zg+001164
  08:28:27 STEP 1165 H= 291:15 +CPU= 53.928
  08:29:23 STEP 1166 H= 291:30 +CPU= 55.663
  08:30:20 STEP 1167 H= 291:45 +CPU= 56.237
  08:31:17 STEP 1168 H= 292:00 +CPU= 55.991
  08:32:12 STEP 1169 H= 292:15 +CPU= 54.261
  08:33:06 STEP 1170 H= 292:30 +CPU= 54.075
  08:34:02 STEP 1171 H= 292:45 +CPU= 55.531
  08:34:58 STEP 1172 H= 293:00 +CPU= 55.484
  08:35:54 STEP 1173 H= 293:15 +CPU= 55.234
  08:36:51 STEP 1174 H= 293:30 +CPU= 56.019
  08:37:47 STEP 1175 H= 293:45 +CPU= 56.167
  08:38:55 STEP 1176 H= 294:00 +CPU= 66.476
Moving to projects directory: /home/steve/slots/19/ICMGGh7zg+001176
Moving to projects directory: /home/steve/slots/19/ICMSHh7zg+001176
Moving to projects directory: /home/steve/slots/19/ICMUAh7zg+001176
  08:39:49 STEP 1177 H= 294:15 +CPU= 54.037
  08:40:44 STEP 1178 H= 294:30 +CPU= 54.491
  08:41:38 STEP 1179 H= 294:45 +CPU= 53.274
  08:42:33 STEP 1180 H= 295:00 +CPU= 54.136
  08:43:28 STEP 1181 H= 295:15 +CPU= 55.083
  08:44:24 STEP 1182 H= 295:30 +CPU= 54.751
  08:45:20 STEP 1183 H= 295:45 +CPU= 55.687
  08:46:16 STEP 1184 H= 296:00 +CPU= 55.775
  08:47:11 STEP 1185 H= 296:15 +CPU= 54.956
  08:48:07 STEP 1186 H= 296:30 +CPU= 55.051
  08:49:03 STEP 1187 H= 296:45 +CPU= 55.156
  08:50:13 STEP 1188 H= 297:00 +CPU= 69.521
Moving to projects directory: /home/steve/slots/19/ICMGGh7zg+001188
Moving to projects directory: /home/steve/slots/19/ICMSHh7zg+001188
Moving to projects directory: /home/steve/slots/19/ICMUAh7zg+001188
  08:51:08 STEP 1189 H= 297:15 +CPU= 54.979
  08:52:03 STEP 1190 H= 297:30 +CPU= 54.518
  08:52:58 STEP 1191 H= 297:45 +CPU= 54.307
  08:53:57 STEP 1192 H= 298:00 +CPU= 57.718
  08:54:53 STEP 1193 H= 298:15 +CPU= 55.773
  08:55:49 STEP 1194 H= 298:30 +CPU= 55.692
  08:56:46 STEP 1195 H= 298:45 +CPU= 56.419
  08:57:42 STEP 1196 H= 299:00 +CPU= 55.077
  08:58:35 STEP 1197 H= 299:15 +CPU= 53.285
  08:59:31 STEP 1198 H= 299:30 +CPU= 54.897
  09:00:28 STEP 1199 H= 299:45 +CPU= 56.289
  09:01:44 STEP 1200 H= 300:00 +CPU= 75.543
Moving to projects directory: /home/steve/slots/19/ICMGGh7zg+001200
Moving to projects directory: /home/steve/slots/19/ICMSHh7zg+001200
Moving to projects directory: /home/steve/slots/19/ICMUAh7zg+001200
  09:02:39 STEP 1201 H= 300:15 +CPU= 53.872
  09:03:33 STEP 1202 H= 300:30 +CPU= 54.345
  09:04:29 STEP 1203 H= 300:45 +CPU= 55.220
  09:05:25 STEP 1204 H= 301:00 +CPU= 55.746
  09:06:21 STEP 1205 H= 301:15 +CPU= 55.268
  09:07:17 STEP 1206 H= 301:30 +CPU= 55.463
  09:08:13 STEP 1207 H= 301:45 +CPU= 55.213
  09:09:11 STEP 1208 H= 302:00 +CPU= 57.490
  09:10:08 STEP 1209 H= 302:15 +CPU= 53.907
  09:11:05 STEP 1210 H= 302:30 +CPU= 55.129
  09:12:04 STEP 1211 H= 302:45 +CPU= 53.918
  09:13:16 STEP 1212 H= 303:00 +CPU= 66.932
Moving to projects directory: /home/steve/slots/19/ICMGGh7zg+001212
Moving to projects directory: /home/steve/slots/19/ICMSHh7zg+001212
Moving to projects directory: /home/steve/slots/19/ICMUAh7zg+001212
  09:14:14 STEP 1213 H= 303:15 +CPU= 54.074
  09:15:11 STEP 1214 H= 303:30 +CPU= 53.977
  09:16:09 STEP 1215 H= 303:45 +CPU= 53.996
  09:17:06 STEP 1216 H= 304:00 +CPU= 56.740
  09:18:01 STEP 1217 H= 304:15 +CPU= 54.725
  09:18:57 STEP 1218 H= 304:30 +CPU= 54.828
  09:19:53 STEP 1219 H= 304:45 +CPU= 55.612
  09:20:49 STEP 1220 H= 305:00 +CPU= 55.907
  09:21:44 STEP 1221 H= 305:15 +CPU= 54.075
  09:22:39 STEP 1222 H= 305:30 +CPU= 54.804
  09:23:35 STEP 1223 H= 305:45 +CPU= 54.886
  09:24:44 STEP 1224 H= 306:00 +CPU= 68.870
Moving to projects directory: /home/steve/slots/19/ICMGGh7zg+001224
Moving to projects directory: /home/steve/slots/19/ICMSHh7zg+001224
Moving to projects directory: /home/steve/slots/19/ICMUAh7zg+001224
  09:25:39 STEP 1225 H= 306:15 +CPU= 54.422
  09:26:35 STEP 1226 H= 306:30 +CPU= 55.089
  09:27:31 STEP 1227 H= 306:45 +CPU= 55.294
  09:28:26 STEP 1228 H= 307:00 +CPU= 54.697
  09:29:23 STEP 1229 H= 307:15 +CPU= 56.791
  09:30:17 STEP 1230 H= 307:30 +CPU= 53.700
  09:31:14 STEP 1231 H= 307:45 +CPU= 55.801
  09:32:11 STEP 1232 H= 308:00 +CPU= 56.497
  09:33:05 STEP 1233 H= 308:15 +CPU= 53.614
  09:34:02 STEP 1234 H= 308:30 +CPU= 55.909
  09:34:58 STEP 1235 H= 308:45 +CPU= 54.369
  09:36:05 STEP 1236 H= 309:00 +CPU= 65.458
Moving to projects directory: /home/steve/slots/19/ICMGGh7zg+001236
Moving to projects directory: /home/steve/slots/19/ICMSHh7zg+001236
Moving to projects directory: /home/steve/slots/19/ICMUAh7zg+001236
  09:37:01 STEP 1237 H= 309:15 +CPU= 55.413
  09:37:57 STEP 1238 H= 309:30 +CPU= 55.334
  09:38:53 STEP 1239 H= 309:45 +CPU= 56.040
  09:39:48 STEP 1240 H= 310:00 +CPU= 54.708
  09:40:43 STEP 1241 H= 310:15 +CPU= 54.504
  09:41:39 STEP 1242 H= 310:30 +CPU= 54.695
  09:42:34 STEP 1243 H= 310:45 +CPU= 54.483
  09:43:30 STEP 1244 H= 311:00 +CPU= 55.326
  09:44:25 STEP 1245 H= 311:15 +CPU= 54.520
  09:45:20 STEP 1246 H= 311:30 +CPU= 54.053
  09:46:16 STEP 1247 H= 311:45 +CPU= 55.403
  09:47:31 STEP 1248 H= 312:00 +CPU= 73.675
Moving to projects directory: /home/steve/slots/19/ICMGGh7zg+001248
Moving to projects directory: /home/steve/slots/19/ICMSHh7zg+001248
Moving to projects directory: /home/steve/slots/19/ICMUAh7zg+001248
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMGGh7zg+001152
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMSHh7zg+001152
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMUAh7zg+001152
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMGGh7zg+001164
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMSHh7zg+001164
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMUAh7zg+001164
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMGGh7zg+001176
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMSHh7zg+001176
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMUAh7zg+001176
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMGGh7zg+001188
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMSHh7zg+001188
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMUAh7zg+001188
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMGGh7zg+001200
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMSHh7zg+001200
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMUAh7zg+001200
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMGGh7zg+001212
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMSHh7zg+001212
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMUAh7zg+001212
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMGGh7zg+001224
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMSHh7zg+001224
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMUAh7zg+001224
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMGGh7zg+001236
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMSHh7zg+001236
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMUAh7zg+001236
Zipping up the intermediate file: /home/steve/projects/climateprediction.net/oifs_43r3_bl_a2dd_2016092300_20_1017_12284900_0_r64603195_12.zip
Uploading the intermediate file: upload_file_12.zip
  09:48:25 STEP 1249 H= 312:15 +CPU= 52.140
  09:49:21 STEP 1250 H= 312:30 +CPU= 54.329
  09:50:17 STEP 1251 H= 312:45 +CPU= 55.442
  09:51:14 STEP 1252 H= 313:00 +CPU= 56.463
  09:52:07 STEP 1253 H= 313:15 +CPU= 53.288
  09:53:04 STEP 1254 H= 313:30 +CPU= 56.410
  09:53:59 STEP 1255 H= 313:45 +CPU= 54.586
  09:54:55 STEP 1256 H= 314:00 +CPU= 54.725
  09:55:51 STEP 1257 H= 314:15 +CPU= 55.695
  09:56:48 STEP 1258 H= 314:30 +CPU= 56.488
  09:57:42 STEP 1259 H= 314:45 +CPU= 53.869
  09:58:51 STEP 1260 H= 315:00 +CPU= 68.436
Moving to projects directory: /home/steve/slots/19/ICMGGh7zg+001260
Moving to projects directory: /home/steve/slots/19/ICMSHh7zg+001260
Moving to projects directory: /home/steve/slots/19/ICMUAh7zg+001260
  09:59:46 STEP 1261 H= 315:15 +CPU= 53.906
  10:00:43 STEP 1262 H= 315:30 +CPU= 56.703
  10:01:38 STEP 1263 H= 315:45 +CPU= 54.575
  10:02:34 STEP 1264 H= 316:00 +CPU= 55.430
  10:03:31 STEP 1265 H= 316:15 +CPU= 55.993
  10:04:27 STEP 1266 H= 316:30 +CPU= 55.304
  10:05:23 STEP 1267 H= 316:45 +CPU= 55.646
  10:06:21 STEP 1268 H= 317:00 +CPU= 57.331
  10:07:18 STEP 1269 H= 317:15 +CPU= 56.754
  10:08:14 STEP 1270 H= 317:30 +CPU= 55.219
  10:09:10 STEP 1271 H= 317:45 +CPU= 55.973
  10:10:18 STEP 1272 H= 318:00 +CPU= 67.154
Moving to projects directory: /home/steve/slots/19/ICMGGh7zg+001272
Moving to projects directory: /home/steve/slots/19/ICMSHh7zg+001272
Moving to projects directory: /home/steve/slots/19/ICMUAh7zg+001272
  10:11:13 STEP 1273 H= 318:15 +CPU= 54.409
  10:12:09 STEP 1274 H= 318:30 +CPU= 55.754
  10:13:06 STEP 1275 H= 318:45 +CPU= 55.990
  10:14:03 STEP 1276 H= 319:00 +CPU= 56.162
  10:14:58 STEP 1277 H= 319:15 +CPU= 54.584
  10:15:54 STEP 1278 H= 319:30 +CPU= 55.776
  10:16:49 STEP 1279 H= 319:45 +CPU= 54.746
  10:17:47 STEP 1280 H= 320:00 +CPU= 56.827
  10:18:41 STEP 1281 H= 320:15 +CPU= 54.068
  10:19:37 STEP 1282 H= 320:30 +CPU= 55.351
  10:20:34 STEP 1283 H= 320:45 +CPU= 55.892
  10:21:41 STEP 1284 H= 321:00 +CPU= 66.549
Moving to projects directory: /home/steve/slots/19/ICMGGh7zg+001284
Moving to projects directory: /home/steve/slots/19/ICMSHh7zg+001284
Moving to projects directory: /home/steve/slots/19/ICMUAh7zg+001284
  10:22:36 STEP 1285 H= 321:15 +CPU= 54.883
  10:23:31 STEP 1286 H= 321:30 +CPU= 54.148
  10:24:26 STEP 1287 H= 321:45 +CPU= 54.842
  10:25:21 STEP 1288 H= 322:00 +CPU= 54.068
  10:26:15 STEP 1289 H= 322:15 +CPU= 54.026
  10:27:11 STEP 1290 H= 322:30 +CPU= 55.729
  10:28:06 STEP 1291 H= 322:45 +CPU= 53.832
  10:29:02 STEP 1292 H= 323:00 +CPU= 55.435
  10:29:56 STEP 1293 H= 323:15 +CPU= 54.241
  10:30:52 STEP 1294 H= 323:30 +CPU= 55.356
  10:31:48 STEP 1295 H= 323:45 +CPU= 55.095
  10:33:03 STEP 1296 H= 324:00 +CPU= 74.844
Moving to projects directory: /home/steve/slots/19/ICMGGh7zg+001296
Moving to projects directory: /home/steve/slots/19/ICMSHh7zg+001296
Moving to projects directory: /home/steve/slots/19/ICMUAh7zg+001296
  10:33:58 STEP 1297 H= 324:15 +CPU= 54.444
  10:34:54 STEP 1298 H= 324:30 +CPU= 55.114
  10:35:50 STEP 1299 H= 324:45 +CPU= 55.459
  10:36:47 STEP 1300 H= 325:00 +CPU= 56.036
  10:37:43 STEP 1301 H= 325:15 +CPU= 56.077
  10:38:39 STEP 1302 H= 325:30 +CPU= 54.735
  10:39:35 STEP 1303 H= 325:45 +CPU= 55.495
  10:40:31 STEP 1304 H= 326:00 +CPU= 55.753
  10:41:27 STEP 1305 H= 326:15 +CPU= 56.062
  10:42:25 STEP 1306 H= 326:30 +CPU= 56.689
  10:43:20 STEP 1307 H= 326:45 +CPU= 55.015
  10:44:30 STEP 1308 H= 327:00 +CPU= 68.656
Moving to projects directory: /home/steve/slots/19/ICMGGh7zg+001308
Moving to projects directory: /home/steve/slots/19/ICMSHh7zg+001308
Moving to projects directory: /home/steve/slots/19/ICMUAh7zg+001308
  10:45:25 STEP 1309 H= 327:15 +CPU= 54.824
  10:46:20 STEP 1310 H= 327:30 +CPU= 53.224
  10:47:15 STEP 1311 H= 327:45 +CPU= 54.613
  10:48:10 STEP 1312 H= 328:00 +CPU= 54.428
  10:49:04 STEP 1313 H= 328:15 +CPU= 53.965
  10:49:59 STEP 1314 H= 328:30 +CPU= 54.564
  10:50:56 STEP 1315 H= 328:45 +CPU= 55.640
  10:51:52 STEP 1316 H= 329:00 +CPU= 55.321
  10:52:46 STEP 1317 H= 329:15 +CPU= 54.111
  10:53:42 STEP 1318 H= 329:30 +CPU= 54.865
  10:54:37 STEP 1319 H= 329:45 +CPU= 55.083
  10:55:48 STEP 1320 H= 330:00 +CPU= 68.502
Moving to projects directory: /home/steve/slots/19/ICMGGh7zg+001320
Moving to projects directory: /home/steve/slots/19/ICMSHh7zg+001320
Moving to projects directory: /home/steve/slots/19/ICMUAh7zg+001320
  10:56:45 STEP 1321 H= 330:15 +CPU= 53.386
  10:57:41 STEP 1322 H= 330:30 +CPU= 54.040
  10:58:40 STEP 1323 H= 330:45 +CPU= 55.880
  10:59:36 STEP 1324 H= 331:00 +CPU= 54.829
  11:00:34 STEP 1325 H= 331:15 +CPU= 55.267
  11:01:30 STEP 1326 H= 331:30 +CPU= 54.019
  11:02:32 STEP 1327 H= 331:45 +CPU= 53.741
  11:03:36 STEP 1328 H= 332:00 +CPU= 53.991
  11:04:38 STEP 1329 H= 332:15 +CPU= 53.818
  11:05:39 STEP 1330 H= 332:30 +CPU= 54.283
  11:06:42 STEP 1331 H= 332:45 +CPU= 55.445
  11:07:58 STEP 1332 H= 333:00 +CPU= 69.056
Moving to projects directory: /home/steve/slots/19/ICMGGh7zg+001332
Moving to projects directory: /home/steve/slots/19/ICMSHh7zg+001332
Moving to projects directory: /home/steve/slots/19/ICMUAh7zg+001332
  11:08:59 STEP 1333 H= 333:15 +CPU= 55.034
  11:10:01 STEP 1334 H= 333:30 +CPU= 56.265
  11:11:04 STEP 1335 H= 333:45 +CPU= 53.921
  11:12:10 STEP 1336 H= 334:00 +CPU= 56.371
  11:13:14 STEP 1337 H= 334:15 +CPU= 55.951
  11:14:18 STEP 1338 H= 334:30 +CPU= 57.203
  11:15:22 STEP 1339 H= 334:45 +CPU= 56.388
  11:16:25 STEP 1340 H= 335:00 +CPU= 54.624
  11:17:26 STEP 1341 H= 335:15 +CPU= 54.945
  11:18:26 STEP 1342 H= 335:30 +CPU= 55.674
  11:19:27 STEP 1343 H= 335:45 +CPU= 56.494
  11:20:49 STEP 1344 H= 336:00 +CPU= 76.195
Moving to projects directory: /home/steve/slots/19/ICMGGh7zg+001344
Moving to projects directory: /home/steve/slots/19/ICMSHh7zg+001344
Moving to projects directory: /home/steve/slots/19/ICMUAh7zg+001344
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMGGh7zg+001248
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMSHh7zg+001248
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMUAh7zg+001248
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMGGh7zg+001260
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMSHh7zg+001260
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMUAh7zg+001260
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMGGh7zg+001272
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMSHh7zg+001272
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMUAh7zg+001272
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMGGh7zg+001284
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMSHh7zg+001284
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMUAh7zg+001284
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMGGh7zg+001296
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMSHh7zg+001296
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMUAh7zg+001296
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMGGh7zg+001308
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMSHh7zg+001308
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMUAh7zg+001308
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMGGh7zg+001320
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMSHh7zg+001320
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMUAh7zg+001320
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMGGh7zg+001332
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMSHh7zg+001332
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMUAh7zg+001332
Zipping up the intermediate file: /home/steve/projects/climateprediction.net/oifs_43r3_bl_a2dd_2016092300_20_1017_12284900_0_r64603195_13.zip
Uploading the intermediate file: upload_file_13.zip
  11:21:48 STEP 1345 H= 336:15 +CPU= 53.813
  11:22:49 STEP 1346 H= 336:30 +CPU= 55.035
  11:23:50 STEP 1347 H= 336:45 +CPU= 55.707
  11:24:50 STEP 1348 H= 337:00 +CPU= 55.989
  11:25:52 STEP 1349 H= 337:15 +CPU= 55.140
  11:26:56 STEP 1350 H= 337:30 +CPU= 56.180
  11:28:02 STEP 1351 H= 337:45 +CPU= 58.407
  11:29:06 STEP 1352 H= 338:00 +CPU= 58.066
  11:30:09 STEP 1353 H= 338:15 +CPU= 56.547
  11:31:14 STEP 1354 H= 338:30 +CPU= 56.802
  11:32:18 STEP 1355 H= 338:45 +CPU= 56.415
  11:33:38 STEP 1356 H= 339:00 +CPU= 69.668
Moving to projects directory: /home/steve/slots/19/ICMGGh7zg+001356
Moving to projects directory: /home/steve/slots/19/ICMSHh7zg+001356
Moving to projects directory: /home/steve/slots/19/ICMUAh7zg+001356
  11:34:41 STEP 1357 H= 339:15 +CPU= 55.458
  11:35:45 STEP 1358 H= 339:30 +CPU= 56.729
  11:36:49 STEP 1359 H= 339:45 +CPU= 56.411
  11:37:53 STEP 1360 H= 340:00 +CPU= 57.634
  11:38:55 STEP 1361 H= 340:15 +CPU= 56.781
  11:39:57 STEP 1362 H= 340:30 +CPU= 56.004
  11:41:01 STEP 1363 H= 340:45 +CPU= 55.461
  11:42:03 STEP 1364 H= 341:00 +CPU= 55.111
  11:43:02 STEP 1365 H= 341:15 +CPU= 55.516
  11:44:03 STEP 1366 H= 341:30 +CPU= 55.931
  11:45:04 STEP 1367 H= 341:45 +CPU= 55.452
  11:46:18 STEP 1368 H= 342:00 +CPU= 68.605
Moving to projects directory: /home/steve/slots/19/ICMGGh7zg+001368
Moving to projects directory: /home/steve/slots/19/ICMSHh7zg+001368
Moving to projects directory: /home/steve/slots/19/ICMUAh7zg+001368
  11:47:14 STEP 1369 H= 342:15 +CPU= 56.228
  11:48:09 STEP 1370 H= 342:30 +CPU= 54.446
  11:49:05 STEP 1371 H= 342:45 +CPU= 55.120
  11:50:01 STEP 1372 H= 343:00 +CPU= 55.377
  11:50:55 STEP 1373 H= 343:15 +CPU= 53.758
  11:51:50 STEP 1374 H= 343:30 +CPU= 54.861
  11:52:46 STEP 1375 H= 343:45 +CPU= 54.618
  11:53:40 STEP 1376 H= 344:00 +CPU= 53.710
  11:54:36 STEP 1377 H= 344:15 +CPU= 55.210
  11:55:31 STEP 1378 H= 344:30 +CPU= 54.703
  11:56:27 STEP 1379 H= 344:45 +CPU= 56.039
  11:57:35 STEP 1380 H= 345:00 +CPU= 66.898
Moving to projects directory: /home/steve/slots/19/ICMGGh7zg+001380
Moving to projects directory: /home/steve/slots/19/ICMSHh7zg+001380
Moving to projects directory: /home/steve/slots/19/ICMUAh7zg+001380
  11:58:30 STEP 1381 H= 345:15 +CPU= 55.125
  11:59:24 STEP 1382 H= 345:30 +CPU= 53.554
  12:00:21 STEP 1383 H= 345:45 +CPU= 55.595
  12:01:15 STEP 1384 H= 346:00 +CPU= 53.629
  12:02:12 STEP 1385 H= 346:15 +CPU= 56.406
  12:03:06 STEP 1386 H= 346:30 +CPU= 54.147
  12:04:01 STEP 1387 H= 346:45 +CPU= 54.561
  12:04:57 STEP 1388 H= 347:00 +CPU= 54.718
  12:05:53 STEP 1389 H= 347:15 +CPU= 55.485
  12:06:48 STEP 1390 H= 347:30 +CPU= 55.024
  12:07:43 STEP 1391 H= 347:45 +CPU= 54.273
  12:08:58 STEP 1392 H= 348:00 +CPU= 73.927
Moving to projects directory: /home/steve/slots/19/ICMGGh7zg+001392
Moving to projects directory: /home/steve/slots/19/ICMSHh7zg+001392
Moving to projects directory: /home/steve/slots/19/ICMUAh7zg+001392
  12:09:52 STEP 1393 H= 348:15 +CPU= 53.519
  12:10:49 STEP 1394 H= 348:30 +CPU= 56.085
  12:11:44 STEP 1395 H= 348:45 +CPU= 55.162
  12:12:39 STEP 1396 H= 349:00 +CPU= 54.597
  12:13:34 STEP 1397 H= 349:15 +CPU= 53.889
  12:14:28 STEP 1398 H= 349:30 +CPU= 54.154
  12:15:25 STEP 1399 H= 349:45 +CPU= 56.379
  12:16:22 STEP 1400 H= 350:00 +CPU= 56.245
  12:17:17 STEP 1401 H= 350:15 +CPU= 54.806
  12:18:11 STEP 1402 H= 350:30 +CPU= 53.375
  12:19:04 STEP 1403 H= 350:45 +CPU= 52.718
  12:20:15 STEP 1404 H= 351:00 +CPU= 69.645
Moving to projects directory: /home/steve/slots/19/ICMGGh7zg+001404
Moving to projects directory: /home/steve/slots/19/ICMSHh7zg+001404
Moving to projects directory: /home/steve/slots/19/ICMUAh7zg+001404
  12:21:11 STEP 1405 H= 351:15 +CPU= 55.226
  12:22:06 STEP 1406 H= 351:30 +CPU= 53.816
  12:23:01 STEP 1407 H= 351:45 +CPU= 54.714
  12:23:58 STEP 1408 H= 352:00 +CPU= 55.699
  12:24:51 STEP 1409 H= 352:15 +CPU= 52.804
  12:25:45 STEP 1410 H= 352:30 +CPU= 54.186
  12:26:42 STEP 1411 H= 352:45 +CPU= 56.247
  12:27:38 STEP 1412 H= 353:00 +CPU= 55.007
  12:28:33 STEP 1413 H= 353:15 +CPU= 54.836
  12:29:28 STEP 1414 H= 353:30 +CPU= 54.250
  12:30:22 STEP 1415 H= 353:45 +CPU= 53.924
  12:31:30 STEP 1416 H= 354:00 +CPU= 66.937
Moving to projects directory: /home/steve/slots/19/ICMGGh7zg+001416
Moving to projects directory: /home/steve/slots/19/ICMSHh7zg+001416
Moving to projects directory: /home/steve/slots/19/ICMUAh7zg+001416
  12:32:25 STEP 1417 H= 354:15 +CPU= 54.956
  12:33:20 STEP 1418 H= 354:30 +CPU= 54.247
  12:34:16 STEP 1419 H= 354:45 +CPU= 55.353
  12:35:14 STEP 1420 H= 355:00 +CPU= 57.612
  12:36:08 STEP 1421 H= 355:15 +CPU= 53.705
  12:37:02 STEP 1422 H= 355:30 +CPU= 53.208
  12:37:58 STEP 1423 H= 355:45 +CPU= 55.210
  12:38:55 STEP 1424 H= 356:00 +CPU= 56.704
  12:39:50 STEP 1425 H= 356:15 +CPU= 54.313
  12:40:44 STEP 1426 H= 356:30 +CPU= 53.248
  12:41:41 STEP 1427 H= 356:45 +CPU= 55.957
  12:42:49 STEP 1428 H= 357:00 +CPU= 67.338
Moving to projects directory: /home/steve/slots/19/ICMGGh7zg+001428
Moving to projects directory: /home/steve/slots/19/ICMSHh7zg+001428
Moving to projects directory: /home/steve/slots/19/ICMUAh7zg+001428
  12:43:44 STEP 1429 H= 357:15 +CPU= 53.957
  12:44:38 STEP 1430 H= 357:30 +CPU= 53.956
  12:45:35 STEP 1431 H= 357:45 +CPU= 56.606
  12:46:31 STEP 1432 H= 358:00 +CPU= 54.901
  12:47:26 STEP 1433 H= 358:15 +CPU= 54.167
  12:48:22 STEP 1434 H= 358:30 +CPU= 55.025
  12:49:17 STEP 1435 H= 358:45 +CPU= 54.399
  12:50:14 STEP 1436 H= 359:00 +CPU= 55.894
  12:51:04 STEP 1437 H= 359:15 +CPU= 50.316
  12:51:54 STEP 1438 H= 359:30 +CPU= 49.235
  12:52:44 STEP 1439 H= 359:45 +CPU= 49.425
  12:53:44 STEP 1440 H= 360:00 +CPU= 59.909
..The child process terminated with status: 0
>>> Printing last 70 lines from file: NODE.001_01

I   VARIABLE   I      Initial value         I        Current value        I
I-------------------------------------------------------------------------I
I  OMEGA (P/S) I     -0.1283110424E-20      I       0.2916230851E-05      I
ICloud fractionI      0.0000000000E+00      I       0.3594365374E-01      I
I  Relat. Hum. I      0.3758118400E+00      I       0.4156987711E+00      I
I  PASS 01     I      0.0000000000E+00      I       0.3669269705E+03      I
DDH------------------------------------------------------------------------
  
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1440 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001440 MODE=a
 IO-STREAM CLOSED - ICMGGh7zg+001440
 MPI-TASK:   1 -    314971972 BYTES IN      7 RECORDS TRANSFERRED IN    0.0008 SECONDS393715.0 Mbytes/s, TOTAL TIME=    0.0190( 4.2%)
 12:53:44 STEP 1440 H= 360:00 +CPU= 59.909
 END CNT3
 NSTEP =  1440 CNT0   000000000  42.811  42.811  43.080   0.994               0               0               0               0               0
IO-STREAM STATISTICS, TOTAL NO OF REC/BYTES READ            0/           0 WRITTEN        43246/************
 -TASK-OPENED-OPEN-RECS IN -KBYTE IN    -RECS OUT-KBYTE OUT-WALL   -WALL IN-WALL
  OU-TOT IN -TOT OUT
   1    758    0        0            0    43246 ************   128.9     0.0   128.9     0.0   293.6
===-=== START OF TIMING STATISTICS ===-===
 
STATS FOR ALL TASKS
 NUM ROUTINE                                     CALLS  MEAN(ms)   MAX(ms)   FRAC(%)  UNBAL(%)
   0 CNT0     - COMPLETE EXECUTION                   1 ********* *********    100.00      0.00
   1 CNT4     - FORWARD INTEGRATION                  1 ********* *********     99.99      0.00
   8 SCAN2M - GRID-POINT DYNAMICS                 1562   10259.5   10259.5     20.80      0.00
   9 SPCM     - SPECTRAL COMP.                    1440    2013.2    2013.2      3.76      0.00
  10 SCAN2M - PHYSICS                             1441   22801.6   22801.6     42.64      0.00
  11 IOPACK   - OUTPUT P.P. RESULTS                121    1715.7    1715.7      0.27      0.00
  12 SPNORM   - SPECTRAL NORM COMP.                 63     116.7     116.7      0.01      0.00
  14 SUINIF                                          1    7610.6    7610.6      0.01      0.00
  17 GRIDFPOS IN CNT4                              121      61.3      61.3      0.01      0.00
  18 SUSPECG                                         1    3929.9    3929.9      0.01      0.00
  19 SUSPEC                                          1    3937.6    3937.6      0.01      0.00
  24 SUGRIDU                                         1    3132.0    3132.0      0.00      0.00
  25 SPECRT                                          1     399.8     399.8      0.00      0.00
  26 SUGRIDF                                         1     141.2     141.2      0.00      0.00
  27 RESTART FILES - WRITING                        30    6919.7    6919.7      0.27      0.00
  28 RESTART FILES - READING                         1       0.1       0.1      0.00      0.00
  29 SU4FPOS IN CNT4                               121       0.1       0.1      0.00      0.00
  30 DYNFPOS IN CNT4                               121   11279.3   11279.3      1.77      0.00
  31 POSDDH IN STEPO                                 7       1.2       1.2      0.00      0.00
  37 CPGLAG   - SL COMPUTATIONS                   1441   10434.2   10434.2     19.51      0.00
  39 SU0YOMB                                         1     492.5     492.5      0.00      0.00
  51 SCAN2M   - SL COMM. PART 1                   1441      43.0      43.0      0.08      0.00
  54 SPCM     - M TO S/S TO M TRANSP.             1440    1077.6    1077.6      2.01      0.00
  55 SPCIMPF  - S TO M/M TO S TRANSP.             1440     277.3     277.3      0.52      0.00
  56 SPNORM   - SPECTRAL NORM COMM.                 63       0.7       0.7      0.00      0.00
 102 LTINV_CTL   - INVERSE LEGENDRE TRANSFORM     3125     799.8     799.8      3.24      0.00
 103 LTDIR_CTL   - DIRECT LEGENDRE TRANSFORM      3006     445.1     445.1      1.74      0.00
 106 FTDIR_CTL   - DIRECT FOURIER TRANSFORM       3006     159.8     159.8      0.62      0.00
 107 FTINV_CTL   - INVERSE FOURIER TRANSFORM      3125     249.6     249.6      1.01      0.00
 140 SULEG       - COMP. OF LEGENDRE POL.            1      57.4      57.4      0.00      0.00
 152 LTINV_CTL   - M TO L TRANSPOSITION           3125     141.3     141.3      0.57      0.00
 153 LTDIR_CTL   - L TO M TRANSPOSITION           3006      82.1      82.1      0.32      0.00
 157 FTINV_CTL   - L TO G TRANSPOSITION           3125     197.3     197.3      0.80      0.00
 158 FTDIR_CTL   - G TO L TRANSPOSITION           3006      82.1      82.1      0.32      0.00
 400 GSTATS                                     167734       0.0       0.0      0.00      0.00
 401 GSTATS HOOK                                156066       0.0       0.0      0.00      0.00
TOTAL MEASURED IMBALANCE =       0.0 SECONDS,  0.0 PERCENT
TOTAL WALLCLOCK TIME    77050.8 CPU TIME   75587.0 VECTOR TIME    75587.0


===-=== END   OF TIMING STATISTICS ===-===
FORECAST DAYS PER DAY       16.8
  *** END CNT0 *** 
------------------------------------------------
Moving to projects directory: /home/steve/slots/19/ICMGGh7zg+001440
Moving to projects directory: /home/steve/slots/19/ICMSHh7zg+001440
Moving to projects directory: /home/steve/slots/19/ICMUAh7zg+001440
Adding to the zip: /home/steve/slots/19/NODE.001_01
Adding to the zip: /home/steve/slots/19/ifs.stat
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMGGh7zg+001344
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMGGh7zg+001428
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMUAh7zg+001392
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMGGh7zg+001368
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMGGh7zg+001392
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMGGh7zg+001356
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMSHh7zg+001404
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMUAh7zg+001416
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMGGh7zg+001380
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMUAh7zg+001440
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMSHh7zg+001344
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMGGh7zg+001440
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMSHh7zg+001368
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMGGh7zg+001416
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMUAh7zg+001404
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMSHh7zg+001380
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMSHh7zg+001392
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMUAh7zg+001380
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMUAh7zg+001344
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMUAh7zg+001428
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMUAh7zg+001368
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMSHh7zg+001416
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMSHh7zg+001440
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMSHh7zg+001428
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMUAh7zg+001356
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMSHh7zg+001356
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284900/ICMGGh7zg+001404
Zipping up the final file: /home/steve/projects/climateprediction.net/oifs_43r3_bl_a2dd_2016092300_20_1017_12284900_0_r64603195_14.zip
Uploading the final file: upload_file_14.zip
Uploading trickle at timestep: 1295100
12:57:14 (931): called boinc_finish(0)

</stderr_txt>
<message>
upload failure: <file_xfer_error>
  <file_name>oifs_43r3_bl_a2dd_2016092300_20_1017_12284900_0_r64603195_15.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a2dd_2016092300_20_1017_12284900_0_r64603195_16.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a2dd_2016092300_20_1017_12284900_0_r64603195_17.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a2dd_2016092300_20_1017_12284900_0_r64603195_18.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a2dd_2016092300_20_1017_12284900_0_r64603195_19.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
</message>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
08 Jun 2024 18:42:10 1536520 22436491 oifs_43r3_bl_a2dd_2016092300_20_1017_12284900_0 1,295,100 75,587 0.0584
08 Jun 2024 11:08:43 1536520 22436491 oifs_43r3_bl_a2dd_2016092300_20_1017_12284900_0 864,000 48,604 0.0563


©2024 climateprediction.net