climateprediction.net home page
Task 22434501

Task 22434501

Name oifs_43r3_bl_a0uo_2016092300_20_1017_12282931_0
Workunit 12282931
Created 7 Jun 2024, 12:26:37 UTC
Sent 7 Jun 2024, 20:28:48 UTC
Report deadline 6 Aug 2024, 20:28:48 UTC
Received 8 Jun 2024, 18:41:28 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 0 (0x00000000)
Computer ID 1536520
Run time 21 hours 27 min 43 sec
CPU time 20 hours 59 min 48 sec
Validate state Invalid
Credit 1,976.32
Device peak FLOPS 7.23 GFLOPS
Application version OpenIFS 43r3 Baroclinic Lifecycle v1.13
x86_64-pc-linux-gnu
Peak working set size 5,025.45 MB
Peak swap size 5,980.80 MB
Peak disk usage 1,347.79 MB
Stderr
<core_client_version>7.22.1</core_client_version>
<![CDATA[
<stderr_txt>
+000780
  02:15:16 STEP  781 H= 195:15 +CPU= 62.372
  02:16:14 STEP  782 H= 195:30 +CPU= 58.321
  02:17:16 STEP  783 H= 195:45 +CPU= 61.045
  02:18:16 STEP  784 H= 196:00 +CPU= 59.775
  02:19:15 STEP  785 H= 196:15 +CPU= 57.499
  02:20:14 STEP  786 H= 196:30 +CPU= 58.726
  02:21:13 STEP  787 H= 196:45 +CPU= 58.257
  02:22:11 STEP  788 H= 197:00 +CPU= 58.219
  02:23:11 STEP  789 H= 197:15 +CPU= 58.588
  02:24:09 STEP  790 H= 197:30 +CPU= 58.148
  02:25:08 STEP  791 H= 197:45 +CPU= 58.397
  02:26:18 STEP  792 H= 198:00 +CPU= 68.986
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+000792
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+000792
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+000792
  02:27:15 STEP  793 H= 198:15 +CPU= 56.282
  02:28:12 STEP  794 H= 198:30 +CPU= 56.530
  02:29:11 STEP  795 H= 198:45 +CPU= 58.172
  02:30:11 STEP  796 H= 199:00 +CPU= 59.327
  02:31:10 STEP  797 H= 199:15 +CPU= 58.322
  02:32:09 STEP  798 H= 199:30 +CPU= 58.767
  02:33:07 STEP  799 H= 199:45 +CPU= 57.228
  02:34:06 STEP  800 H= 200:00 +CPU= 59.081
  02:35:05 STEP  801 H= 200:15 +CPU= 57.923
  02:36:01 STEP  802 H= 200:30 +CPU= 55.733
  02:37:04 STEP  803 H= 200:45 +CPU= 62.490
  02:38:18 STEP  804 H= 201:00 +CPU= 73.251
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+000804
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+000804
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+000804
  02:39:18 STEP  805 H= 201:15 +CPU= 59.732
  02:40:18 STEP  806 H= 201:30 +CPU= 59.197
  02:41:18 STEP  807 H= 201:45 +CPU= 58.745
  02:42:18 STEP  808 H= 202:00 +CPU= 59.460
  02:43:19 STEP  809 H= 202:15 +CPU= 60.498
  02:44:16 STEP  810 H= 202:30 +CPU= 56.263
  02:45:12 STEP  811 H= 202:45 +CPU= 55.491
  02:46:08 STEP  812 H= 203:00 +CPU= 55.653
  02:47:04 STEP  813 H= 203:15 +CPU= 55.643
  02:48:01 STEP  814 H= 203:30 +CPU= 56.226
  02:48:59 STEP  815 H= 203:45 +CPU= 57.598
  02:50:21 STEP  816 H= 204:00 +CPU= 81.202
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+000816
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+000816
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+000816
  02:51:19 STEP  817 H= 204:15 +CPU= 56.762
  02:52:18 STEP  818 H= 204:30 +CPU= 58.675
  02:53:18 STEP  819 H= 204:45 +CPU= 59.222
  02:54:20 STEP  820 H= 205:00 +CPU= 61.348
  02:55:19 STEP  821 H= 205:15 +CPU= 58.536
  02:56:16 STEP  822 H= 205:30 +CPU= 56.697
  02:57:15 STEP  823 H= 205:45 +CPU= 58.862
  02:58:15 STEP  824 H= 206:00 +CPU= 59.197
  02:59:13 STEP  825 H= 206:15 +CPU= 56.924
  03:00:15 STEP  826 H= 206:30 +CPU= 61.532
  03:01:15 STEP  827 H= 206:45 +CPU= 59.725
  03:02:26 STEP  828 H= 207:00 +CPU= 70.281
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+000828
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+000828
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+000828
  03:03:25 STEP  829 H= 207:15 +CPU= 58.685
  03:04:26 STEP  830 H= 207:30 +CPU= 58.770
  03:05:23 STEP  831 H= 207:45 +CPU= 56.453
  03:06:21 STEP  832 H= 208:00 +CPU= 57.267
  03:07:17 STEP  833 H= 208:15 +CPU= 55.560
  03:08:16 STEP  834 H= 208:30 +CPU= 58.256
  03:09:14 STEP  835 H= 208:45 +CPU= 57.711
  03:10:14 STEP  836 H= 209:00 +CPU= 59.453
  03:11:13 STEP  837 H= 209:15 +CPU= 58.987
  03:12:12 STEP  838 H= 209:30 +CPU= 58.453
  03:13:12 STEP  839 H= 209:45 +CPU= 59.435
  03:14:25 STEP  840 H= 210:00 +CPU= 72.223
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+000840
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+000840
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+000840
  03:15:26 STEP  841 H= 210:15 +CPU= 60.162
  03:16:25 STEP  842 H= 210:30 +CPU= 57.948
  03:17:26 STEP  843 H= 210:45 +CPU= 60.678
  03:18:25 STEP  844 H= 211:00 +CPU= 58.421
  03:19:26 STEP  845 H= 211:15 +CPU= 59.673
  03:20:24 STEP  846 H= 211:30 +CPU= 57.524
  03:21:23 STEP  847 H= 211:45 +CPU= 58.529
  03:22:23 STEP  848 H= 212:00 +CPU= 59.680
  03:23:22 STEP  849 H= 212:15 +CPU= 57.910
  03:24:19 STEP  850 H= 212:30 +CPU= 55.244
  03:25:17 STEP  851 H= 212:45 +CPU= 57.758
  03:26:27 STEP  852 H= 213:00 +CPU= 69.119
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+000852
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+000852
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+000852
  03:27:26 STEP  853 H= 213:15 +CPU= 58.386
  03:28:25 STEP  854 H= 213:30 +CPU= 58.795
  03:29:26 STEP  855 H= 213:45 +CPU= 59.597
  03:30:24 STEP  856 H= 214:00 +CPU= 57.743
  03:31:23 STEP  857 H= 214:15 +CPU= 58.323
  03:32:22 STEP  858 H= 214:30 +CPU= 59.097
  03:33:24 STEP  859 H= 214:45 +CPU= 60.788
  03:34:25 STEP  860 H= 215:00 +CPU= 60.020
  03:35:22 STEP  861 H= 215:15 +CPU= 56.100
  03:36:22 STEP  862 H= 215:30 +CPU= 59.045
  03:37:21 STEP  863 H= 215:45 +CPU= 58.477
  03:38:43 STEP  864 H= 216:00 +CPU= 80.428
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+000864
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+000864
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+000864
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMGGh7zg+000768
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMSHh7zg+000768
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMUAh7zg+000768
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMGGh7zg+000780
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMSHh7zg+000780
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMUAh7zg+000780
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMGGh7zg+000792
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMSHh7zg+000792
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMUAh7zg+000792
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMGGh7zg+000804
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMSHh7zg+000804
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMUAh7zg+000804
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMGGh7zg+000816
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMSHh7zg+000816
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMUAh7zg+000816
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMGGh7zg+000828
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMSHh7zg+000828
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMUAh7zg+000828
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMGGh7zg+000840
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMSHh7zg+000840
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMUAh7zg+000840
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMGGh7zg+000852
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMSHh7zg+000852
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMUAh7zg+000852
Zipping up the intermediate file: /home/steve/projects/climateprediction.net/oifs_43r3_bl_a0uo_2016092300_20_1017_12282931_0_r1970859892_8.zip
Uploading the intermediate file: upload_file_8.zip
  03:39:42 STEP  865 H= 216:15 +CPU= 58.031
  03:40:41 STEP  866 H= 216:30 +CPU= 58.125
  03:41:41 STEP  867 H= 216:45 +CPU= 59.064
  03:42:42 STEP  868 H= 217:00 +CPU= 59.867
  03:43:42 STEP  869 H= 217:15 +CPU= 59.296
  03:44:38 STEP  870 H= 217:30 +CPU= 56.291
  03:45:35 STEP  871 H= 217:45 +CPU= 55.857
  03:46:32 STEP  872 H= 218:00 +CPU= 56.305
  03:47:30 STEP  873 H= 218:15 +CPU= 57.497
  03:48:29 STEP  874 H= 218:30 +CPU= 58.194
  03:49:28 STEP  875 H= 218:45 +CPU= 58.889
  03:50:40 STEP  876 H= 219:00 +CPU= 71.454
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+000876
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+000876
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+000876
  03:51:41 STEP  877 H= 219:15 +CPU= 59.862
  03:52:43 STEP  878 H= 219:30 +CPU= 61.107
  03:53:45 STEP  879 H= 219:45 +CPU= 61.683
  03:54:45 STEP  880 H= 220:00 +CPU= 59.131
  03:55:44 STEP  881 H= 220:15 +CPU= 58.630
  03:56:43 STEP  882 H= 220:30 +CPU= 58.336
  03:57:40 STEP  883 H= 220:45 +CPU= 56.154
  03:58:39 STEP  884 H= 221:00 +CPU= 58.920
  03:59:38 STEP  885 H= 221:15 +CPU= 58.516
  04:00:36 STEP  886 H= 221:30 +CPU= 56.895
  04:01:36 STEP  887 H= 221:45 +CPU= 59.495
  04:02:52 STEP  888 H= 222:00 +CPU= 75.823
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+000888
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+000888
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+000888
  04:03:49 STEP  889 H= 222:15 +CPU= 56.376
  04:04:44 STEP  890 H= 222:30 +CPU= 54.683
  04:05:40 STEP  891 H= 222:45 +CPU= 54.966
  04:06:36 STEP  892 H= 223:00 +CPU= 55.735
  04:07:32 STEP  893 H= 223:15 +CPU= 52.858
  04:08:28 STEP  894 H= 223:30 +CPU= 55.165
  04:09:23 STEP  895 H= 223:45 +CPU= 54.971
  04:10:19 STEP  896 H= 224:00 +CPU= 55.170
  04:11:14 STEP  897 H= 224:15 +CPU= 54.361
  04:12:10 STEP  898 H= 224:30 +CPU= 55.912
  04:13:07 STEP  899 H= 224:45 +CPU= 56.439
  04:14:16 STEP  900 H= 225:00 +CPU= 67.936
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+000900
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+000900
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+000900
  04:15:10 STEP  901 H= 225:15 +CPU= 53.889
  04:16:08 STEP  902 H= 225:30 +CPU= 57.109
  04:17:04 STEP  903 H= 225:45 +CPU= 55.655
  04:18:00 STEP  904 H= 226:00 +CPU= 55.200
  04:18:54 STEP  905 H= 226:15 +CPU= 53.829
  04:19:51 STEP  906 H= 226:30 +CPU= 56.568
  04:20:46 STEP  907 H= 226:45 +CPU= 54.461
  04:21:42 STEP  908 H= 227:00 +CPU= 55.502
  04:22:37 STEP  909 H= 227:15 +CPU= 54.544
  04:23:33 STEP  910 H= 227:30 +CPU= 55.437
  04:24:30 STEP  911 H= 227:45 +CPU= 55.580
  04:25:45 STEP  912 H= 228:00 +CPU= 74.155
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+000912
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+000912
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+000912
  04:26:42 STEP  913 H= 228:15 +CPU= 56.789
  04:27:36 STEP  914 H= 228:30 +CPU= 53.704
  04:28:32 STEP  915 H= 228:45 +CPU= 54.714
  04:29:26 STEP  916 H= 229:00 +CPU= 54.333
  04:30:21 STEP  917 H= 229:15 +CPU= 53.713
  04:31:17 STEP  918 H= 229:30 +CPU= 55.823
  04:32:14 STEP  919 H= 229:45 +CPU= 56.025
  04:33:09 STEP  920 H= 230:00 +CPU= 55.269
  04:34:04 STEP  921 H= 230:15 +CPU= 53.763
  04:35:00 STEP  922 H= 230:30 +CPU= 55.550
  04:35:57 STEP  923 H= 230:45 +CPU= 56.709
  04:37:05 STEP  924 H= 231:00 +CPU= 67.142
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+000924
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+000924
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+000924
  04:38:00 STEP  925 H= 231:15 +CPU= 54.519
  04:38:56 STEP  926 H= 231:30 +CPU= 54.305
  04:39:51 STEP  927 H= 231:45 +CPU= 54.301
  04:40:49 STEP  928 H= 232:00 +CPU= 57.574
  04:41:44 STEP  929 H= 232:15 +CPU= 54.305
  04:42:41 STEP  930 H= 232:30 +CPU= 56.684
  04:43:38 STEP  931 H= 232:45 +CPU= 56.153
  04:44:33 STEP  932 H= 233:00 +CPU= 55.012
  04:45:28 STEP  933 H= 233:15 +CPU= 54.658
  04:46:24 STEP  934 H= 233:30 +CPU= 55.036
  04:47:21 STEP  935 H= 233:45 +CPU= 56.937
  04:48:28 STEP  936 H= 234:00 +CPU= 66.515
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+000936
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+000936
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+000936
  04:49:22 STEP  937 H= 234:15 +CPU= 53.069
  04:50:16 STEP  938 H= 234:30 +CPU= 53.251
  04:51:11 STEP  939 H= 234:45 +CPU= 54.498
  04:52:07 STEP  940 H= 235:00 +CPU= 55.595
  04:53:00 STEP  941 H= 235:15 +CPU= 53.199
  04:53:55 STEP  942 H= 235:30 +CPU= 53.956
  04:54:50 STEP  943 H= 235:45 +CPU= 55.014
  04:55:47 STEP  944 H= 236:00 +CPU= 55.945
  04:56:41 STEP  945 H= 236:15 +CPU= 53.591
  04:57:39 STEP  946 H= 236:30 +CPU= 56.102
  04:58:35 STEP  947 H= 236:45 +CPU= 54.677
  04:59:44 STEP  948 H= 237:00 +CPU= 68.381
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+000948
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+000948
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+000948
  05:00:37 STEP  949 H= 237:15 +CPU= 52.305
  05:01:32 STEP  950 H= 237:30 +CPU= 54.782
  05:02:27 STEP  951 H= 237:45 +CPU= 54.707
  05:03:22 STEP  952 H= 238:00 +CPU= 54.837
  05:04:19 STEP  953 H= 238:15 +CPU= 55.833
  05:05:14 STEP  954 H= 238:30 +CPU= 54.243
  05:06:11 STEP  955 H= 238:45 +CPU= 56.649
  05:07:06 STEP  956 H= 239:00 +CPU= 54.893
  05:08:01 STEP  957 H= 239:15 +CPU= 53.188
  05:08:58 STEP  958 H= 239:30 +CPU= 55.918
  05:09:54 STEP  959 H= 239:45 +CPU= 55.418
  05:11:08 STEP  960 H= 240:00 +CPU= 73.263
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+000960
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+000960
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+000960
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMGGh7zg+000864
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMSHh7zg+000864
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMUAh7zg+000864
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMGGh7zg+000876
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMSHh7zg+000876
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMUAh7zg+000876
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMGGh7zg+000888
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMSHh7zg+000888
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMUAh7zg+000888
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMGGh7zg+000900
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMSHh7zg+000900
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMUAh7zg+000900
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMGGh7zg+000912
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMSHh7zg+000912
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMUAh7zg+000912
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMGGh7zg+000924
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMSHh7zg+000924
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMUAh7zg+000924
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMGGh7zg+000936
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMSHh7zg+000936
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMUAh7zg+000936
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMGGh7zg+000948
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMSHh7zg+000948
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMUAh7zg+000948
Zipping up the intermediate file: /home/steve/projects/climateprediction.net/oifs_43r3_bl_a0uo_2016092300_20_1017_12282931_0_r1970859892_9.zip
Uploading the intermediate file: upload_file_9.zip
Uploading trickle at timestep: 864000
  05:12:03 STEP  961 H= 240:15 +CPU= 53.225
  05:12:59 STEP  962 H= 240:30 +CPU= 55.291
  05:13:56 STEP  963 H= 240:45 +CPU= 56.467
  05:14:52 STEP  964 H= 241:00 +CPU= 56.199
  05:15:48 STEP  965 H= 241:15 +CPU= 54.803
  05:16:43 STEP  966 H= 241:30 +CPU= 54.698
  05:17:40 STEP  967 H= 241:45 +CPU= 56.405
  05:18:37 STEP  968 H= 242:00 +CPU= 56.263
  05:19:31 STEP  969 H= 242:15 +CPU= 53.353
  05:20:26 STEP  970 H= 242:30 +CPU= 54.997
  05:21:21 STEP  971 H= 242:45 +CPU= 54.579
  05:22:29 STEP  972 H= 243:00 +CPU= 67.388
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+000972
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+000972
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+000972
  05:23:25 STEP  973 H= 243:15 +CPU= 55.038
  05:24:21 STEP  974 H= 243:30 +CPU= 55.175
  05:25:16 STEP  975 H= 243:45 +CPU= 54.855
  05:26:12 STEP  976 H= 244:00 +CPU= 55.949
  05:27:07 STEP  977 H= 244:15 +CPU= 54.240
  05:28:02 STEP  978 H= 244:30 +CPU= 54.141
  05:28:57 STEP  979 H= 244:45 +CPU= 54.467
  05:29:53 STEP  980 H= 245:00 +CPU= 55.732
  05:30:49 STEP  981 H= 245:15 +CPU= 54.941
  05:31:44 STEP  982 H= 245:30 +CPU= 54.658
  05:32:40 STEP  983 H= 245:45 +CPU= 55.934
  05:33:48 STEP  984 H= 246:00 +CPU= 66.637
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+000984
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+000984
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+000984
  05:34:43 STEP  985 H= 246:15 +CPU= 54.705
  05:35:39 STEP  986 H= 246:30 +CPU= 55.745
  05:36:35 STEP  987 H= 246:45 +CPU= 55.605
  05:37:31 STEP  988 H= 247:00 +CPU= 55.618
  05:38:27 STEP  989 H= 247:15 +CPU= 54.640
  05:39:24 STEP  990 H= 247:30 +CPU= 56.659
  05:40:19 STEP  991 H= 247:45 +CPU= 54.698
  05:41:17 STEP  992 H= 248:00 +CPU= 57.839
  05:42:12 STEP  993 H= 248:15 +CPU= 53.873
  05:43:09 STEP  994 H= 248:30 +CPU= 56.586
  05:44:04 STEP  995 H= 248:45 +CPU= 54.730
  05:45:12 STEP  996 H= 249:00 +CPU= 67.308
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+000996
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+000996
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+000996
  05:46:07 STEP  997 H= 249:15 +CPU= 53.790
  05:47:04 STEP  998 H= 249:30 +CPU= 56.424
  05:47:59 STEP  999 H= 249:45 +CPU= 54.566
  05:48:55 STEP 1000 H= 250:00 +CPU= 56.043
  05:49:50 STEP 1001 H= 250:15 +CPU= 54.192
  05:50:45 STEP 1002 H= 250:30 +CPU= 53.987
  05:51:40 STEP 1003 H= 250:45 +CPU= 55.315
  05:52:38 STEP 1004 H= 251:00 +CPU= 56.789
  05:53:32 STEP 1005 H= 251:15 +CPU= 54.143
  05:54:26 STEP 1006 H= 251:30 +CPU= 53.457
  05:55:22 STEP 1007 H= 251:45 +CPU= 54.959
  05:56:38 STEP 1008 H= 252:00 +CPU= 75.699
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+001008
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+001008
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+001008
  05:57:33 STEP 1009 H= 252:15 +CPU= 54.292
  05:58:28 STEP 1010 H= 252:30 +CPU= 54.271
  05:59:24 STEP 1011 H= 252:45 +CPU= 55.237
  06:00:18 STEP 1012 H= 253:00 +CPU= 54.163
  06:01:14 STEP 1013 H= 253:15 +CPU= 55.342
  06:02:10 STEP 1014 H= 253:30 +CPU= 55.627
  06:03:07 STEP 1015 H= 253:45 +CPU= 55.542
  06:04:02 STEP 1016 H= 254:00 +CPU= 55.211
  06:04:56 STEP 1017 H= 254:15 +CPU= 53.530
  06:05:53 STEP 1018 H= 254:30 +CPU= 56.117
  06:06:49 STEP 1019 H= 254:45 +CPU= 55.104
  06:07:56 STEP 1020 H= 255:00 +CPU= 67.171
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+001020
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+001020
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+001020
  06:08:52 STEP 1021 H= 255:15 +CPU= 55.259
  06:09:48 STEP 1022 H= 255:30 +CPU= 54.136
  06:10:43 STEP 1023 H= 255:45 +CPU= 54.763
  06:11:39 STEP 1024 H= 256:00 +CPU= 55.599
  06:12:36 STEP 1025 H= 256:15 +CPU= 55.908
  06:13:32 STEP 1026 H= 256:30 +CPU= 55.638
  06:14:27 STEP 1027 H= 256:45 +CPU= 54.491
  06:15:25 STEP 1028 H= 257:00 +CPU= 55.385
  06:16:20 STEP 1029 H= 257:15 +CPU= 54.190
  06:17:15 STEP 1030 H= 257:30 +CPU= 54.612
  06:18:10 STEP 1031 H= 257:45 +CPU= 54.775
  06:19:18 STEP 1032 H= 258:00 +CPU= 67.485
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+001032
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+001032
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+001032
  06:20:15 STEP 1033 H= 258:15 +CPU= 55.242
  06:21:15 STEP 1034 H= 258:30 +CPU= 51.477
  06:22:14 STEP 1035 H= 258:45 +CPU= 54.459
  06:23:13 STEP 1036 H= 259:00 +CPU= 55.047
  06:24:11 STEP 1037 H= 259:15 +CPU= 55.965
  06:25:10 STEP 1038 H= 259:30 +CPU= 54.152
  06:26:07 STEP 1039 H= 259:45 +CPU= 54.949
  06:27:05 STEP 1040 H= 260:00 +CPU= 56.780
  06:28:06 STEP 1041 H= 260:15 +CPU= 55.738
  06:29:04 STEP 1042 H= 260:30 +CPU= 53.145
  06:30:02 STEP 1043 H= 260:45 +CPU= 55.737
  06:31:13 STEP 1044 H= 261:00 +CPU= 67.924
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+001044
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+001044
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+001044
  06:32:13 STEP 1045 H= 261:15 +CPU= 56.236
  06:33:13 STEP 1046 H= 261:30 +CPU= 55.943
  06:34:13 STEP 1047 H= 261:45 +CPU= 55.232
  06:35:11 STEP 1048 H= 262:00 +CPU= 55.999
  06:36:07 STEP 1049 H= 262:15 +CPU= 55.192
  06:37:04 STEP 1050 H= 262:30 +CPU= 55.333
  06:38:02 STEP 1051 H= 262:45 +CPU= 54.593
  06:39:00 STEP 1052 H= 263:00 +CPU= 55.704
  06:39:58 STEP 1053 H= 263:15 +CPU= 53.045
  06:40:56 STEP 1054 H= 263:30 +CPU= 55.329
  06:41:53 STEP 1055 H= 263:45 +CPU= 54.219
  06:43:11 STEP 1056 H= 264:00 +CPU= 73.793
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+001056
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+001056
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+001056
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMGGh7zg+000960
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMSHh7zg+000960
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMUAh7zg+000960
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMGGh7zg+000972
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMSHh7zg+000972
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMUAh7zg+000972
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMGGh7zg+000984
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMSHh7zg+000984
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMUAh7zg+000984
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMGGh7zg+000996
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMSHh7zg+000996
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMUAh7zg+000996
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMGGh7zg+001008
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMSHh7zg+001008
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMUAh7zg+001008
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMGGh7zg+001020
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMSHh7zg+001020
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMUAh7zg+001020
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMGGh7zg+001032
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMSHh7zg+001032
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMUAh7zg+001032
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMGGh7zg+001044
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMSHh7zg+001044
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMUAh7zg+001044
Zipping up the intermediate file: /home/steve/projects/climateprediction.net/oifs_43r3_bl_a0uo_2016092300_20_1017_12282931_0_r1970859892_10.zip
Uploading the intermediate file: upload_file_10.zip
  06:44:08 STEP 1057 H= 264:15 +CPU= 53.259
  06:45:04 STEP 1058 H= 264:30 +CPU= 54.591
  06:46:01 STEP 1059 H= 264:45 +CPU= 54.515
  06:46:58 STEP 1060 H= 265:00 +CPU= 54.932
  06:47:56 STEP 1061 H= 265:15 +CPU= 55.675
  06:48:56 STEP 1062 H= 265:30 +CPU= 55.231
  06:49:53 STEP 1063 H= 265:45 +CPU= 55.551
  06:50:50 STEP 1064 H= 266:00 +CPU= 56.191
  06:51:47 STEP 1065 H= 266:15 +CPU= 55.691
  06:52:45 STEP 1066 H= 266:30 +CPU= 55.390
  06:53:43 STEP 1067 H= 266:45 +CPU= 55.787
  06:54:52 STEP 1068 H= 267:00 +CPU= 68.036
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+001068
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+001068
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+001068
  06:55:48 STEP 1069 H= 267:15 +CPU= 55.147
  06:56:47 STEP 1070 H= 267:30 +CPU= 57.153
  06:57:44 STEP 1071 H= 267:45 +CPU= 54.219
  06:58:41 STEP 1072 H= 268:00 +CPU= 55.100
  06:59:39 STEP 1073 H= 268:15 +CPU= 55.483
  07:00:37 STEP 1074 H= 268:30 +CPU= 54.312
  07:01:35 STEP 1075 H= 268:45 +CPU= 55.027
  07:02:33 STEP 1076 H= 269:00 +CPU= 55.290
  07:03:32 STEP 1077 H= 269:15 +CPU= 56.150
  07:04:29 STEP 1078 H= 269:30 +CPU= 55.460
  07:05:26 STEP 1079 H= 269:45 +CPU= 56.543
  07:06:34 STEP 1080 H= 270:00 +CPU= 67.218
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+001080
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+001080
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+001080
  07:07:28 STEP 1081 H= 270:15 +CPU= 53.654
  07:08:23 STEP 1082 H= 270:30 +CPU= 54.548
  07:09:18 STEP 1083 H= 270:45 +CPU= 54.538
  07:10:16 STEP 1084 H= 271:00 +CPU= 56.665
  07:11:12 STEP 1085 H= 271:15 +CPU= 56.045
  07:12:07 STEP 1086 H= 271:30 +CPU= 54.026
  07:13:02 STEP 1087 H= 271:45 +CPU= 55.139
  07:13:58 STEP 1088 H= 272:00 +CPU= 54.764
  07:14:52 STEP 1089 H= 272:15 +CPU= 53.632
  07:15:46 STEP 1090 H= 272:30 +CPU= 53.799
  07:16:41 STEP 1091 H= 272:45 +CPU= 53.873
  07:17:49 STEP 1092 H= 273:00 +CPU= 67.639
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+001092
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+001092
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+001092
  07:18:44 STEP 1093 H= 273:15 +CPU= 54.512
  07:19:39 STEP 1094 H= 273:30 +CPU= 54.835
  07:20:35 STEP 1095 H= 273:45 +CPU= 55.547
  07:21:33 STEP 1096 H= 274:00 +CPU= 56.800
  07:22:28 STEP 1097 H= 274:15 +CPU= 54.622
  07:23:24 STEP 1098 H= 274:30 +CPU= 55.958
  07:24:20 STEP 1099 H= 274:45 +CPU= 55.046
  07:25:19 STEP 1100 H= 275:00 +CPU= 56.210
  07:26:17 STEP 1101 H= 275:15 +CPU= 53.623
  07:27:14 STEP 1102 H= 275:30 +CPU= 55.527
  07:28:10 STEP 1103 H= 275:45 +CPU= 55.104
  07:29:31 STEP 1104 H= 276:00 +CPU= 75.357
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+001104
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+001104
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+001104
  07:30:29 STEP 1105 H= 276:15 +CPU= 55.588
  07:31:27 STEP 1106 H= 276:30 +CPU= 55.592
  07:32:25 STEP 1107 H= 276:45 +CPU= 55.513
  07:33:23 STEP 1108 H= 277:00 +CPU= 55.175
  07:34:20 STEP 1109 H= 277:15 +CPU= 54.764
  07:35:19 STEP 1110 H= 277:30 +CPU= 54.346
  07:36:15 STEP 1111 H= 277:45 +CPU= 54.350
  07:37:12 STEP 1112 H= 278:00 +CPU= 54.442
  07:38:10 STEP 1113 H= 278:15 +CPU= 55.568
  07:39:08 STEP 1114 H= 278:30 +CPU= 55.812
  07:40:04 STEP 1115 H= 278:45 +CPU= 54.731
  07:41:14 STEP 1116 H= 279:00 +CPU= 66.564
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+001116
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+001116
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+001116
  07:42:10 STEP 1117 H= 279:15 +CPU= 53.751
  07:43:06 STEP 1118 H= 279:30 +CPU= 53.911
  07:44:02 STEP 1119 H= 279:45 +CPU= 54.921
  07:45:00 STEP 1120 H= 280:00 +CPU= 56.816
  07:46:04 STEP 1121 H= 280:15 +CPU= 54.450
  07:47:06 STEP 1122 H= 280:30 +CPU= 54.403
  07:48:06 STEP 1123 H= 280:45 +CPU= 54.065
  07:49:07 STEP 1124 H= 281:00 +CPU= 55.649
  07:50:04 STEP 1125 H= 281:15 +CPU= 54.656
  07:51:01 STEP 1126 H= 281:30 +CPU= 54.786
  07:51:58 STEP 1127 H= 281:45 +CPU= 54.039
  07:53:08 STEP 1128 H= 282:00 +CPU= 68.255
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+001128
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+001128
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+001128
  07:54:06 STEP 1129 H= 282:15 +CPU= 53.714
  07:55:02 STEP 1130 H= 282:30 +CPU= 54.711
  07:56:00 STEP 1131 H= 282:45 +CPU= 54.450
  07:56:58 STEP 1132 H= 283:00 +CPU= 55.245
  07:57:57 STEP 1133 H= 283:15 +CPU= 55.274
  07:58:54 STEP 1134 H= 283:30 +CPU= 55.076
  07:59:50 STEP 1135 H= 283:45 +CPU= 54.851
  08:00:45 STEP 1136 H= 284:00 +CPU= 54.036
  08:01:41 STEP 1137 H= 284:15 +CPU= 55.502
  08:02:36 STEP 1138 H= 284:30 +CPU= 53.898
  08:03:31 STEP 1139 H= 284:45 +CPU= 53.549
  08:04:41 STEP 1140 H= 285:00 +CPU= 69.725
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+001140
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+001140
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+001140
  08:05:37 STEP 1141 H= 285:15 +CPU= 54.298
  08:06:33 STEP 1142 H= 285:30 +CPU= 56.270
  08:07:30 STEP 1143 H= 285:45 +CPU= 55.969
  08:08:26 STEP 1144 H= 286:00 +CPU= 55.384
  08:09:21 STEP 1145 H= 286:15 +CPU= 54.497
  08:10:16 STEP 1146 H= 286:30 +CPU= 54.921
  08:11:12 STEP 1147 H= 286:45 +CPU= 54.883
  08:12:09 STEP 1148 H= 287:00 +CPU= 56.328
  08:13:04 STEP 1149 H= 287:15 +CPU= 54.256
  08:14:01 STEP 1150 H= 287:30 +CPU= 55.577
  08:14:56 STEP 1151 H= 287:45 +CPU= 54.685
  08:16:12 STEP 1152 H= 288:00 +CPU= 74.858
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+001152
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+001152
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+001152
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMGGh7zg+001056
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMSHh7zg+001056
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMUAh7zg+001056
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMGGh7zg+001068
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMSHh7zg+001068
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMUAh7zg+001068
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMGGh7zg+001080
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMSHh7zg+001080
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMUAh7zg+001080
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMGGh7zg+001092
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMSHh7zg+001092
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMUAh7zg+001092
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMGGh7zg+001104
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMSHh7zg+001104
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMUAh7zg+001104
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMGGh7zg+001116
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMSHh7zg+001116
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMUAh7zg+001116
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMGGh7zg+001128
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMSHh7zg+001128
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMUAh7zg+001128
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMGGh7zg+001140
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMSHh7zg+001140
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMUAh7zg+001140
Zipping up the intermediate file: /home/steve/projects/climateprediction.net/oifs_43r3_bl_a0uo_2016092300_20_1017_12282931_0_r1970859892_11.zip
Uploading the intermediate file: upload_file_11.zip
  08:17:09 STEP 1153 H= 288:15 +CPU= 54.178
  08:18:05 STEP 1154 H= 288:30 +CPU= 54.976
  08:19:01 STEP 1155 H= 288:45 +CPU= 55.044
  08:19:57 STEP 1156 H= 289:00 +CPU= 55.470
  08:20:52 STEP 1157 H= 289:15 +CPU= 54.503
  08:21:47 STEP 1158 H= 289:30 +CPU= 54.341
  08:22:42 STEP 1159 H= 289:45 +CPU= 54.672
  08:23:39 STEP 1160 H= 290:00 +CPU= 55.742
  08:24:33 STEP 1161 H= 290:15 +CPU= 54.006
  08:25:28 STEP 1162 H= 290:30 +CPU= 54.656
  08:26:23 STEP 1163 H= 290:45 +CPU= 54.632
  08:27:31 STEP 1164 H= 291:00 +CPU= 67.263
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+001164
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+001164
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+001164
  08:28:26 STEP 1165 H= 291:15 +CPU= 54.439
  08:29:22 STEP 1166 H= 291:30 +CPU= 55.540
  08:30:17 STEP 1167 H= 291:45 +CPU= 53.923
  08:31:12 STEP 1168 H= 292:00 +CPU= 54.659
  08:32:08 STEP 1169 H= 292:15 +CPU= 55.419
  08:33:03 STEP 1170 H= 292:30 +CPU= 54.122
  08:33:59 STEP 1171 H= 292:45 +CPU= 56.143
  08:34:55 STEP 1172 H= 293:00 +CPU= 54.751
  08:35:51 STEP 1173 H= 293:15 +CPU= 55.205
  08:36:46 STEP 1174 H= 293:30 +CPU= 55.359
  08:37:42 STEP 1175 H= 293:45 +CPU= 54.610
  08:38:51 STEP 1176 H= 294:00 +CPU= 68.204
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+001176
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+001176
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+001176
  08:39:45 STEP 1177 H= 294:15 +CPU= 53.859
  08:40:40 STEP 1178 H= 294:30 +CPU= 54.797
  08:41:37 STEP 1179 H= 294:45 +CPU= 55.931
  08:42:35 STEP 1180 H= 295:00 +CPU= 57.490
  08:43:30 STEP 1181 H= 295:15 +CPU= 54.674
  08:44:25 STEP 1182 H= 295:30 +CPU= 54.314
  08:45:22 STEP 1183 H= 295:45 +CPU= 56.537
  08:46:19 STEP 1184 H= 296:00 +CPU= 56.419
  08:47:16 STEP 1185 H= 296:15 +CPU= 55.806
  08:48:11 STEP 1186 H= 296:30 +CPU= 55.000
  08:49:06 STEP 1187 H= 296:45 +CPU= 54.315
  08:50:14 STEP 1188 H= 297:00 +CPU= 67.356
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+001188
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+001188
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+001188
  08:51:08 STEP 1189 H= 297:15 +CPU= 53.794
  08:52:04 STEP 1190 H= 297:30 +CPU= 54.876
  08:52:59 STEP 1191 H= 297:45 +CPU= 54.988
  08:53:56 STEP 1192 H= 298:00 +CPU= 55.693
  08:54:52 STEP 1193 H= 298:15 +CPU= 55.385
  08:55:47 STEP 1194 H= 298:30 +CPU= 54.806
  08:56:41 STEP 1195 H= 298:45 +CPU= 53.767
  08:57:39 STEP 1196 H= 299:00 +CPU= 57.288
  08:58:34 STEP 1197 H= 299:15 +CPU= 54.845
  08:59:31 STEP 1198 H= 299:30 +CPU= 55.831
  09:00:26 STEP 1199 H= 299:45 +CPU= 54.345
  09:01:42 STEP 1200 H= 300:00 +CPU= 75.315
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+001200
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+001200
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+001200
  09:02:37 STEP 1201 H= 300:15 +CPU= 54.784
  09:03:33 STEP 1202 H= 300:30 +CPU= 55.204
  09:04:29 STEP 1203 H= 300:45 +CPU= 55.051
  09:05:25 STEP 1204 H= 301:00 +CPU= 55.495
  09:06:19 STEP 1205 H= 301:15 +CPU= 53.884
  09:07:16 STEP 1206 H= 301:30 +CPU= 56.805
  09:08:12 STEP 1207 H= 301:45 +CPU= 55.389
  09:09:09 STEP 1208 H= 302:00 +CPU= 55.969
  09:10:07 STEP 1209 H= 302:15 +CPU= 55.071
  09:11:05 STEP 1210 H= 302:30 +CPU= 55.339
  09:12:05 STEP 1211 H= 302:45 +CPU= 53.992
  09:13:18 STEP 1212 H= 303:00 +CPU= 66.623
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+001212
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+001212
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+001212
  09:14:16 STEP 1213 H= 303:15 +CPU= 54.488
  09:15:14 STEP 1214 H= 303:30 +CPU= 54.147
  09:16:13 STEP 1215 H= 303:45 +CPU= 54.946
  09:17:09 STEP 1216 H= 304:00 +CPU= 55.775
  09:18:04 STEP 1217 H= 304:15 +CPU= 54.645
  09:19:00 STEP 1218 H= 304:30 +CPU= 55.283
  09:19:56 STEP 1219 H= 304:45 +CPU= 55.732
  09:20:54 STEP 1220 H= 305:00 +CPU= 56.952
  09:21:50 STEP 1221 H= 305:15 +CPU= 55.993
  09:22:46 STEP 1222 H= 305:30 +CPU= 55.236
  09:23:42 STEP 1223 H= 305:45 +CPU= 55.137
  09:24:49 STEP 1224 H= 306:00 +CPU= 66.322
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+001224
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+001224
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+001224
  09:25:44 STEP 1225 H= 306:15 +CPU= 53.920
  09:26:40 STEP 1226 H= 306:30 +CPU= 55.408
  09:27:36 STEP 1227 H= 306:45 +CPU= 56.255
  09:28:33 STEP 1228 H= 307:00 +CPU= 56.351
  09:29:28 STEP 1229 H= 307:15 +CPU= 54.456
  09:30:22 STEP 1230 H= 307:30 +CPU= 53.620
  09:31:18 STEP 1231 H= 307:45 +CPU= 55.145
  09:32:14 STEP 1232 H= 308:00 +CPU= 55.367
  09:33:10 STEP 1233 H= 308:15 +CPU= 55.485
  09:34:06 STEP 1234 H= 308:30 +CPU= 54.763
  09:35:02 STEP 1235 H= 308:45 +CPU= 55.064
  09:36:09 STEP 1236 H= 309:00 +CPU= 65.276
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+001236
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+001236
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+001236
  09:37:04 STEP 1237 H= 309:15 +CPU= 54.341
  09:37:59 STEP 1238 H= 309:30 +CPU= 54.237
  09:38:54 STEP 1239 H= 309:45 +CPU= 54.682
  09:39:52 STEP 1240 H= 310:00 +CPU= 57.142
  09:40:47 STEP 1241 H= 310:15 +CPU= 55.002
  09:41:43 STEP 1242 H= 310:30 +CPU= 54.829
  09:42:36 STEP 1243 H= 310:45 +CPU= 53.359
  09:43:33 STEP 1244 H= 311:00 +CPU= 55.890
  09:44:29 STEP 1245 H= 311:15 +CPU= 54.573
  09:45:24 STEP 1246 H= 311:30 +CPU= 54.551
  09:46:19 STEP 1247 H= 311:45 +CPU= 54.651
  09:47:34 STEP 1248 H= 312:00 +CPU= 74.011
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+001248
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+001248
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+001248
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMGGh7zg+001152
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMSHh7zg+001152
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMUAh7zg+001152
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMGGh7zg+001164
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMSHh7zg+001164
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMUAh7zg+001164
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMGGh7zg+001176
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMSHh7zg+001176
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMUAh7zg+001176
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMGGh7zg+001188
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMSHh7zg+001188
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMUAh7zg+001188
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMGGh7zg+001200
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMSHh7zg+001200
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMUAh7zg+001200
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMGGh7zg+001212
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMSHh7zg+001212
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMUAh7zg+001212
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMGGh7zg+001224
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMSHh7zg+001224
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMUAh7zg+001224
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMGGh7zg+001236
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMSHh7zg+001236
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMUAh7zg+001236
Zipping up the intermediate file: /home/steve/projects/climateprediction.net/oifs_43r3_bl_a0uo_2016092300_20_1017_12282931_0_r1970859892_12.zip
Uploading the intermediate file: upload_file_12.zip
  09:48:28 STEP 1249 H= 312:15 +CPU= 52.221
  09:49:24 STEP 1250 H= 312:30 +CPU= 54.875
  09:50:20 STEP 1251 H= 312:45 +CPU= 55.327
  09:51:16 STEP 1252 H= 313:00 +CPU= 54.747
  09:52:12 STEP 1253 H= 313:15 +CPU= 55.747
  09:53:06 STEP 1254 H= 313:30 +CPU= 53.774
  09:54:03 STEP 1255 H= 313:45 +CPU= 56.401
  09:54:58 STEP 1256 H= 314:00 +CPU= 54.354
  09:55:54 STEP 1257 H= 314:15 +CPU= 55.403
  09:56:51 STEP 1258 H= 314:30 +CPU= 56.671
  09:57:46 STEP 1259 H= 314:45 +CPU= 54.218
  09:58:54 STEP 1260 H= 315:00 +CPU= 68.181
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+001260
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+001260
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+001260
  09:59:50 STEP 1261 H= 315:15 +CPU= 54.428
  10:00:45 STEP 1262 H= 315:30 +CPU= 54.916
  10:01:40 STEP 1263 H= 315:45 +CPU= 55.039
  10:02:36 STEP 1264 H= 316:00 +CPU= 54.780
  10:03:31 STEP 1265 H= 316:15 +CPU= 54.572
  10:04:28 STEP 1266 H= 316:30 +CPU= 56.350
  10:05:25 STEP 1267 H= 316:45 +CPU= 56.634
  10:06:21 STEP 1268 H= 317:00 +CPU= 55.210
  10:07:16 STEP 1269 H= 317:15 +CPU= 54.660
  10:08:12 STEP 1270 H= 317:30 +CPU= 55.292
  10:09:08 STEP 1271 H= 317:45 +CPU= 55.624
  10:10:15 STEP 1272 H= 318:00 +CPU= 66.731
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+001272
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+001272
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+001272
  10:11:11 STEP 1273 H= 318:15 +CPU= 55.346
  10:12:06 STEP 1274 H= 318:30 +CPU= 54.433
  10:13:00 STEP 1275 H= 318:45 +CPU= 53.673
  10:13:57 STEP 1276 H= 319:00 +CPU= 56.403
  10:14:53 STEP 1277 H= 319:15 +CPU= 55.571
  10:15:48 STEP 1278 H= 319:30 +CPU= 54.123
  10:16:44 STEP 1279 H= 319:45 +CPU= 55.166
  10:17:40 STEP 1280 H= 320:00 +CPU= 55.971
  10:18:37 STEP 1281 H= 320:15 +CPU= 56.020
  10:19:31 STEP 1282 H= 320:30 +CPU= 53.849
  10:20:26 STEP 1283 H= 320:45 +CPU= 54.356
  10:21:34 STEP 1284 H= 321:00 +CPU= 67.338
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+001284
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+001284
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+001284
  10:22:27 STEP 1285 H= 321:15 +CPU= 52.873
  10:23:23 STEP 1286 H= 321:30 +CPU= 55.077
  10:24:18 STEP 1287 H= 321:45 +CPU= 54.453
  10:25:15 STEP 1288 H= 322:00 +CPU= 56.311
  10:26:10 STEP 1289 H= 322:15 +CPU= 55.107
  10:27:06 STEP 1290 H= 322:30 +CPU= 54.720
  10:28:02 STEP 1291 H= 322:45 +CPU= 55.971
  10:28:59 STEP 1292 H= 323:00 +CPU= 56.529
  10:29:57 STEP 1293 H= 323:15 +CPU= 56.839
  10:30:54 STEP 1294 H= 323:30 +CPU= 56.847
  10:31:50 STEP 1295 H= 323:45 +CPU= 55.311
  10:33:04 STEP 1296 H= 324:00 +CPU= 73.318
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+001296
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+001296
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+001296
  10:34:00 STEP 1297 H= 324:15 +CPU= 55.216
  10:34:55 STEP 1298 H= 324:30 +CPU= 54.932
  10:35:51 STEP 1299 H= 324:45 +CPU= 55.263
  10:36:48 STEP 1300 H= 325:00 +CPU= 56.686
  10:37:44 STEP 1301 H= 325:15 +CPU= 54.746
  10:38:38 STEP 1302 H= 325:30 +CPU= 54.102
  10:39:33 STEP 1303 H= 325:45 +CPU= 54.740
  10:40:29 STEP 1304 H= 326:00 +CPU= 54.579
  10:41:23 STEP 1305 H= 326:15 +CPU= 54.233
  10:42:19 STEP 1306 H= 326:30 +CPU= 55.097
  10:43:15 STEP 1307 H= 326:45 +CPU= 55.325
  10:44:21 STEP 1308 H= 327:00 +CPU= 65.400
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+001308
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+001308
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+001308
  10:45:15 STEP 1309 H= 327:15 +CPU= 53.987
  10:46:11 STEP 1310 H= 327:30 +CPU= 54.722
  10:47:08 STEP 1311 H= 327:45 +CPU= 55.977
  10:48:04 STEP 1312 H= 328:00 +CPU= 54.867
  10:48:59 STEP 1313 H= 328:15 +CPU= 54.893
  10:49:55 STEP 1314 H= 328:30 +CPU= 55.420
  10:50:51 STEP 1315 H= 328:45 +CPU= 55.107
  10:51:47 STEP 1316 H= 329:00 +CPU= 55.389
  10:52:41 STEP 1317 H= 329:15 +CPU= 53.850
  10:53:37 STEP 1318 H= 329:30 +CPU= 55.762
  10:54:32 STEP 1319 H= 329:45 +CPU= 54.109
  10:55:40 STEP 1320 H= 330:00 +CPU= 66.810
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+001320
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+001320
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+001320
  10:56:38 STEP 1321 H= 330:15 +CPU= 53.413
  10:57:35 STEP 1322 H= 330:30 +CPU= 54.524
  10:58:33 STEP 1323 H= 330:45 +CPU= 54.747
  10:59:29 STEP 1324 H= 331:00 +CPU= 55.376
  11:00:27 STEP 1325 H= 331:15 +CPU= 55.903
  11:01:26 STEP 1326 H= 331:30 +CPU= 55.989
  11:02:27 STEP 1327 H= 331:45 +CPU= 53.496
  11:03:30 STEP 1328 H= 332:00 +CPU= 53.870
  11:04:32 STEP 1329 H= 332:15 +CPU= 54.068
  11:05:33 STEP 1330 H= 332:30 +CPU= 54.317
  11:06:36 STEP 1331 H= 332:45 +CPU= 55.514
  11:07:51 STEP 1332 H= 333:00 +CPU= 68.214
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+001332
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+001332
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+001332
  11:08:53 STEP 1333 H= 333:15 +CPU= 56.355
  11:09:55 STEP 1334 H= 333:30 +CPU= 56.401
  11:10:59 STEP 1335 H= 333:45 +CPU= 54.000
  11:12:04 STEP 1336 H= 334:00 +CPU= 55.704
  11:13:08 STEP 1337 H= 334:15 +CPU= 55.763
  11:14:13 STEP 1338 H= 334:30 +CPU= 57.511
  11:15:17 STEP 1339 H= 334:45 +CPU= 56.536
  11:16:21 STEP 1340 H= 335:00 +CPU= 54.956
  11:17:22 STEP 1341 H= 335:15 +CPU= 55.164
  11:18:23 STEP 1342 H= 335:30 +CPU= 56.456
  11:19:23 STEP 1343 H= 335:45 +CPU= 56.478
  11:20:45 STEP 1344 H= 336:00 +CPU= 76.456
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+001344
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+001344
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+001344
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMGGh7zg+001248
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMSHh7zg+001248
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMUAh7zg+001248
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMGGh7zg+001260
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMSHh7zg+001260
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMUAh7zg+001260
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMGGh7zg+001272
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMSHh7zg+001272
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMUAh7zg+001272
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMGGh7zg+001284
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMSHh7zg+001284
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMUAh7zg+001284
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMGGh7zg+001296
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMSHh7zg+001296
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMUAh7zg+001296
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMGGh7zg+001308
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMSHh7zg+001308
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMUAh7zg+001308
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMGGh7zg+001320
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMSHh7zg+001320
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMUAh7zg+001320
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMGGh7zg+001332
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMSHh7zg+001332
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMUAh7zg+001332
Zipping up the intermediate file: /home/steve/projects/climateprediction.net/oifs_43r3_bl_a0uo_2016092300_20_1017_12282931_0_r1970859892_13.zip
Uploading the intermediate file: upload_file_13.zip
  11:21:44 STEP 1345 H= 336:15 +CPU= 53.539
  11:22:46 STEP 1346 H= 336:30 +CPU= 55.830
  11:23:47 STEP 1347 H= 336:45 +CPU= 56.309
  11:24:49 STEP 1348 H= 337:00 +CPU= 56.746
  11:25:50 STEP 1349 H= 337:15 +CPU= 54.715
  11:26:54 STEP 1350 H= 337:30 +CPU= 55.417
  11:27:59 STEP 1351 H= 337:45 +CPU= 57.907
  11:29:04 STEP 1352 H= 338:00 +CPU= 58.346
  11:30:07 STEP 1353 H= 338:15 +CPU= 56.667
  11:31:11 STEP 1354 H= 338:30 +CPU= 56.104
  11:32:15 STEP 1355 H= 338:45 +CPU= 56.943
  11:33:34 STEP 1356 H= 339:00 +CPU= 69.456
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+001356
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+001356
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+001356
  11:34:37 STEP 1357 H= 339:15 +CPU= 55.508
  11:35:41 STEP 1358 H= 339:30 +CPU= 56.480
  11:36:47 STEP 1359 H= 339:45 +CPU= 57.439
  11:37:52 STEP 1360 H= 340:00 +CPU= 58.321
  11:38:54 STEP 1361 H= 340:15 +CPU= 56.677
  11:39:57 STEP 1362 H= 340:30 +CPU= 56.478
  11:41:01 STEP 1363 H= 340:45 +CPU= 55.986
  11:42:03 STEP 1364 H= 341:00 +CPU= 55.770
  11:43:03 STEP 1365 H= 341:15 +CPU= 55.756
  11:44:04 STEP 1366 H= 341:30 +CPU= 56.529
  11:45:04 STEP 1367 H= 341:45 +CPU= 55.398
  11:46:17 STEP 1368 H= 342:00 +CPU= 67.968
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+001368
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+001368
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+001368
  11:47:11 STEP 1369 H= 342:15 +CPU= 53.043
  11:48:05 STEP 1370 H= 342:30 +CPU= 53.757
  11:49:00 STEP 1371 H= 342:45 +CPU= 54.014
  11:49:55 STEP 1372 H= 343:00 +CPU= 55.258
  11:50:51 STEP 1373 H= 343:15 +CPU= 54.754
  11:51:45 STEP 1374 H= 343:30 +CPU= 53.458
  11:52:41 STEP 1375 H= 343:45 +CPU= 56.112
  11:53:38 STEP 1376 H= 344:00 +CPU= 55.645
  11:54:34 STEP 1377 H= 344:15 +CPU= 55.591
  11:55:29 STEP 1378 H= 344:30 +CPU= 54.451
  11:56:22 STEP 1379 H= 344:45 +CPU= 53.351
  11:57:30 STEP 1380 H= 345:00 +CPU= 67.237
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+001380
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+001380
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+001380
  11:58:25 STEP 1381 H= 345:15 +CPU= 54.192
  11:59:20 STEP 1382 H= 345:30 +CPU= 54.993
  12:00:16 STEP 1383 H= 345:45 +CPU= 54.840
  12:01:11 STEP 1384 H= 346:00 +CPU= 54.447
  12:02:07 STEP 1385 H= 346:15 +CPU= 55.370
  12:03:03 STEP 1386 H= 346:30 +CPU= 55.998
  12:03:57 STEP 1387 H= 346:45 +CPU= 53.505
  12:04:54 STEP 1388 H= 347:00 +CPU= 56.267
  12:05:50 STEP 1389 H= 347:15 +CPU= 55.361
  12:06:45 STEP 1390 H= 347:30 +CPU= 54.451
  12:07:39 STEP 1391 H= 347:45 +CPU= 53.097
  12:08:54 STEP 1392 H= 348:00 +CPU= 74.597
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+001392
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+001392
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+001392
  12:09:49 STEP 1393 H= 348:15 +CPU= 54.015
  12:10:44 STEP 1394 H= 348:30 +CPU= 54.936
  12:11:39 STEP 1395 H= 348:45 +CPU= 54.673
  12:12:37 STEP 1396 H= 349:00 +CPU= 57.390
  12:13:34 STEP 1397 H= 349:15 +CPU= 56.676
  12:14:29 STEP 1398 H= 349:30 +CPU= 53.954
  12:15:24 STEP 1399 H= 349:45 +CPU= 54.316
  12:16:19 STEP 1400 H= 350:00 +CPU= 54.826
  12:17:14 STEP 1401 H= 350:15 +CPU= 54.235
  12:18:10 STEP 1402 H= 350:30 +CPU= 55.253
  12:19:05 STEP 1403 H= 350:45 +CPU= 55.040
  12:20:14 STEP 1404 H= 351:00 +CPU= 68.467
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+001404
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+001404
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+001404
  12:21:09 STEP 1405 H= 351:15 +CPU= 53.936
  12:22:03 STEP 1406 H= 351:30 +CPU= 52.905
  12:22:59 STEP 1407 H= 351:45 +CPU= 54.529
  12:23:54 STEP 1408 H= 352:00 +CPU= 54.501
  12:24:50 STEP 1409 H= 352:15 +CPU= 55.305
  12:25:43 STEP 1410 H= 352:30 +CPU= 53.166
  12:26:38 STEP 1411 H= 352:45 +CPU= 54.421
  12:27:34 STEP 1412 H= 353:00 +CPU= 54.788
  12:28:28 STEP 1413 H= 353:15 +CPU= 54.235
  12:29:23 STEP 1414 H= 353:30 +CPU= 54.052
  12:30:19 STEP 1415 H= 353:45 +CPU= 55.213
  12:31:25 STEP 1416 H= 354:00 +CPU= 65.744
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+001416
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+001416
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+001416
  12:32:19 STEP 1417 H= 354:15 +CPU= 53.599
  12:33:12 STEP 1418 H= 354:30 +CPU= 51.826
  12:34:06 STEP 1419 H= 354:45 +CPU= 53.893
  12:35:03 STEP 1420 H= 355:00 +CPU= 56.595
  12:35:58 STEP 1421 H= 355:15 +CPU= 54.720
  12:36:54 STEP 1422 H= 355:30 +CPU= 55.062
  12:37:49 STEP 1423 H= 355:45 +CPU= 54.790
  12:38:46 STEP 1424 H= 356:00 +CPU= 56.134
  12:39:41 STEP 1425 H= 356:15 +CPU= 54.334
  12:40:35 STEP 1426 H= 356:30 +CPU= 53.043
  12:41:29 STEP 1427 H= 356:45 +CPU= 53.833
  12:42:37 STEP 1428 H= 357:00 +CPU= 66.981
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+001428
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+001428
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+001428
  12:43:31 STEP 1429 H= 357:15 +CPU= 53.778
  12:44:27 STEP 1430 H= 357:30 +CPU= 55.133
  12:45:23 STEP 1431 H= 357:45 +CPU= 55.205
  12:46:21 STEP 1432 H= 358:00 +CPU= 57.452
  12:47:17 STEP 1433 H= 358:15 +CPU= 54.440
  12:48:13 STEP 1434 H= 358:30 +CPU= 55.581
  12:49:09 STEP 1435 H= 358:45 +CPU= 55.241
  12:50:04 STEP 1436 H= 359:00 +CPU= 54.281
  12:50:56 STEP 1437 H= 359:15 +CPU= 50.657
  12:51:45 STEP 1438 H= 359:30 +CPU= 48.985
  12:52:35 STEP 1439 H= 359:45 +CPU= 49.687
  12:53:39 STEP 1440 H= 360:00 +CPU= 63.286
..The child process terminated with status: 0
>>> Printing last 70 lines from file: NODE.001_01

I   VARIABLE   I      Initial value         I        Current value        I
I-------------------------------------------------------------------------I
I  OMEGA (P/S) I     -0.1523837590E-21      I       0.1021563885E-08      I
ICloud fractionI      0.0000000000E+00      I       0.3532205847E-05      I
I  Relat. Hum. I      0.1422062810E+00      I       0.1614931392E+00      I
I  PASS 01     I      0.0000000000E+00      I       0.5917343663E+00      I
DDH------------------------------------------------------------------------
  
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1440 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001440 MODE=a
 IO-STREAM CLOSED - ICMGGh7zg+001440
 MPI-TASK:   1 -    324396868 BYTES IN      7 RECORDS TRANSFERRED IN    0.0009 SECONDS360441.0 Mbytes/s, TOTAL TIME=    0.0171( 5.3%)
 12:53:39 STEP 1440 H= 360:00 +CPU= 63.286
 END CNT3
 NSTEP =  1440 CNT0   000000000  45.212  45.212  45.503   0.994               0               0               0               0               0
IO-STREAM STATISTICS, TOTAL NO OF REC/BYTES READ            0/           0 WRITTEN        43246/************
 -TASK-OPENED-OPEN-RECS IN -KBYTE IN    -RECS OUT-KBYTE OUT-WALL   -WALL IN-WALL
  OU-TOT IN -TOT OUT
   1    758    0        0            0    43246 ************   127.0     0.0   127.0     0.0   286.5
===-=== START OF TIMING STATISTICS ===-===
 
STATS FOR ALL TASKS
 NUM ROUTINE                                     CALLS  MEAN(ms)   MAX(ms)   FRAC(%)  UNBAL(%)
   0 CNT0     - COMPLETE EXECUTION                   1 ********* *********    100.00      0.00
   1 CNT4     - FORWARD INTEGRATION                  1 ********* *********     99.99      0.00
   8 SCAN2M - GRID-POINT DYNAMICS                 1562    9996.2    9996.2     20.27      0.00
   9 SPCM     - SPECTRAL COMP.                    1440    2020.9    2020.9      3.78      0.00
  10 SCAN2M - PHYSICS                             1441   23474.6   23474.6     43.90      0.00
  11 IOPACK   - OUTPUT P.P. RESULTS                121    1742.6    1742.6      0.27      0.00
  12 SPNORM   - SPECTRAL NORM COMP.                 63     118.1     118.1      0.01      0.00
  14 SUINIF                                          1    7997.0    7997.0      0.01      0.00
  17 GRIDFPOS IN CNT4                              121      58.4      58.4      0.01      0.00
  18 SUSPECG                                         1    4029.7    4029.7      0.01      0.00
  19 SUSPEC                                          1    4041.5    4041.5      0.01      0.00
  24 SUGRIDU                                         1    3137.4    3137.4      0.00      0.00
  25 SPECRT                                          1     385.1     385.1      0.00      0.00
  26 SUGRIDF                                         1     433.0     433.0      0.00      0.00
  27 RESTART FILES - WRITING                        30    7028.3    7028.3      0.27      0.00
  28 RESTART FILES - READING                         1       0.0       0.0      0.00      0.00
  29 SU4FPOS IN CNT4                               121       0.1       0.1      0.00      0.00
  30 DYNFPOS IN CNT4                               121   11088.6   11088.6      1.74      0.00
  31 POSDDH IN STEPO                                 7       1.1       1.1      0.00      0.00
  37 CPGLAG   - SL COMPUTATIONS                   1441   10077.5   10077.5     18.85      0.00
  39 SU0YOMB                                         1     472.4     472.4      0.00      0.00
  51 SCAN2M   - SL COMM. PART 1                   1441      41.2      41.2      0.08      0.00
  54 SPCM     - M TO S/S TO M TRANSP.             1440    1097.4    1097.4      2.05      0.00
  55 SPCIMPF  - S TO M/M TO S TRANSP.             1440     275.1     275.1      0.51      0.00
  56 SPNORM   - SPECTRAL NORM COMM.                 63       0.8       0.8      0.00      0.00
 102 LTINV_CTL   - INVERSE LEGENDRE TRANSFORM     3125     788.6     788.6      3.20      0.00
 103 LTDIR_CTL   - DIRECT LEGENDRE TRANSFORM      3006     442.3     442.3      1.73      0.00
 106 FTDIR_CTL   - DIRECT FOURIER TRANSFORM       3006     160.3     160.3      0.63      0.00
 107 FTINV_CTL   - INVERSE FOURIER TRANSFORM      3125     249.4     249.4      1.01      0.00
 140 SULEG       - COMP. OF LEGENDRE POL.            1      49.0      49.0      0.00      0.00
 152 LTINV_CTL   - M TO L TRANSPOSITION           3125     137.9     137.9      0.56      0.00
 153 LTDIR_CTL   - L TO M TRANSPOSITION           3006      81.8      81.8      0.32      0.00
 157 FTINV_CTL   - L TO G TRANSPOSITION           3125     194.9     194.9      0.79      0.00
 158 FTDIR_CTL   - G TO L TRANSPOSITION           3006      81.9      81.9      0.32      0.00
 400 GSTATS                                     167734       0.0       0.0      0.00      0.00
 401 GSTATS HOOK                                156066       0.0       0.0      0.00      0.00
TOTAL MEASURED IMBALANCE =       0.0 SECONDS,  0.0 PERCENT
TOTAL WALLCLOCK TIME    77048.6 CPU TIME   75588.1 VECTOR TIME    75588.1


===-=== END   OF TIMING STATISTICS ===-===
FORECAST DAYS PER DAY       16.8
  *** END CNT0 *** 
------------------------------------------------
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+001440
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+001440
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+001440
Adding to the zip: /home/steve/slots/18/NODE.001_01
Adding to the zip: /home/steve/slots/18/ifs.stat
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMGGh7zg+001344
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMGGh7zg+001428
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMUAh7zg+001392
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMGGh7zg+001368
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMGGh7zg+001392
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMGGh7zg+001356
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMSHh7zg+001404
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMUAh7zg+001416
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMGGh7zg+001380
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMUAh7zg+001440
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMSHh7zg+001344
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMGGh7zg+001440
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMSHh7zg+001368
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMGGh7zg+001416
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMUAh7zg+001404
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMSHh7zg+001380
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMSHh7zg+001392
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMUAh7zg+001380
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMUAh7zg+001344
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMUAh7zg+001428
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMUAh7zg+001368
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMSHh7zg+001416
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMSHh7zg+001440
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMSHh7zg+001428
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMUAh7zg+001356
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMSHh7zg+001356
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12282931/ICMGGh7zg+001404
Zipping up the final file: /home/steve/projects/climateprediction.net/oifs_43r3_bl_a0uo_2016092300_20_1017_12282931_0_r1970859892_14.zip
Uploading the final file: upload_file_14.zip
Uploading trickle at timestep: 1295100
12:57:08 (927): called boinc_finish(0)

</stderr_txt>
<message>
upload failure: <file_xfer_error>
  <file_name>oifs_43r3_bl_a0uo_2016092300_20_1017_12282931_0_r1970859892_15.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a0uo_2016092300_20_1017_12282931_0_r1970859892_16.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a0uo_2016092300_20_1017_12282931_0_r1970859892_17.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a0uo_2016092300_20_1017_12282931_0_r1970859892_18.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a0uo_2016092300_20_1017_12282931_0_r1970859892_19.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
</message>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
08 Jun 2024 18:42:10 1536520 22434501 oifs_43r3_bl_a0uo_2016092300_20_1017_12282931_0 1,295,100 75,588 0.0584
08 Jun 2024 11:08:43 1536520 22434501 oifs_43r3_bl_a0uo_2016092300_20_1017_12282931_0 864,000 48,624 0.0563


©2024 climateprediction.net