climateprediction.net home page
Task 22309882

Task 22309882

Name oifs_43r3_bl_a28f_2016092300_15_991_12209682_0
Workunit 12209682
Created 13 Feb 2023, 17:20:20 UTC
Sent 13 Feb 2023, 17:52:23 UTC
Report deadline 14 Apr 2023, 17:52:23 UTC
Received 18 Feb 2023, 7:34:19 UTC
Server state Over
Outcome Success
Client state Done
Exit status 0 (0x00000000)
Computer ID 1537793
Run time 18 hours 45 min 10 sec
CPU time 18 hours 38 min 30 sec
Validate state Valid
Credit 1,815.66
Device peak FLOPS 3.90 GFLOPS
Application version OpenIFS 43r3 Baroclinic Lifecycle v1.11
x86_64-pc-linux-gnu
Peak working set size 5,041.66 MB
Peak swap size 5,930.79 MB
Peak disk usage 1,361.10 MB
Stderr
<core_client_version>7.20.5</core_client_version>
<![CDATA[
<stderr_txt>
nc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMSHh7zg+000912
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMUAh7zg+000912
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMGGh7zg+000924
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMSHh7zg+000924
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMUAh7zg+000924
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMGGh7zg+000936
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMSHh7zg+000936
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMUAh7zg+000936
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMGGh7zg+000948
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMSHh7zg+000948
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMUAh7zg+000948
Zipping up the intermediate file: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_a28f_2016092300_15_991_12209682_0_r1681846707_9.zip
Uploading the intermediate file: upload_file_9.zip
Uploading trickle at timestep: 864000
  19:47:50 STEP  961 H= 240:15 +CPU= 43.407
  19:48:34 STEP  962 H= 240:30 +CPU= 43.818
  19:49:20 STEP  963 H= 240:45 +CPU= 46.002
  19:50:04 STEP  964 H= 241:00 +CPU= 43.949
  19:50:49 STEP  965 H= 241:15 +CPU= 44.262
  19:51:33 STEP  966 H= 241:30 +CPU= 43.757
  19:52:16 STEP  967 H= 241:45 +CPU= 43.238
  19:53:02 STEP  968 H= 242:00 +CPU= 46.369
  19:53:48 STEP  969 H= 242:15 +CPU= 45.568
  19:54:33 STEP  970 H= 242:30 +CPU= 44.711
  19:55:16 STEP  971 H= 242:45 +CPU= 43.354
  19:56:09 STEP  972 H= 243:00 +CPU= 53.031
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMGGh7zg+000972
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMSHh7zg+000972
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMUAh7zg+000972
  19:56:53 STEP  973 H= 243:15 +CPU= 43.999
  19:57:37 STEP  974 H= 243:30 +CPU= 43.539
  19:58:20 STEP  975 H= 243:45 +CPU= 43.077
  19:59:05 STEP  976 H= 244:00 +CPU= 43.988
  19:59:47 STEP  977 H= 244:15 +CPU= 42.353
  20:00:29 STEP  978 H= 244:30 +CPU= 41.774
  20:01:11 STEP  979 H= 244:45 +CPU= 41.685
  20:01:55 STEP  980 H= 245:00 +CPU= 44.271
  20:02:39 STEP  981 H= 245:15 +CPU= 43.701
  20:03:26 STEP  982 H= 245:30 +CPU= 46.899
  20:04:12 STEP  983 H= 245:45 +CPU= 45.568
  20:05:05 STEP  984 H= 246:00 +CPU= 53.031
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMGGh7zg+000984
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMSHh7zg+000984
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMUAh7zg+000984
  20:05:47 STEP  985 H= 246:15 +CPU= 42.419
  20:06:30 STEP  986 H= 246:30 +CPU= 42.457
  20:07:13 STEP  987 H= 246:45 +CPU= 42.925
  20:07:58 STEP  988 H= 247:00 +CPU= 44.904
  20:08:43 STEP  989 H= 247:15 +CPU= 45.095
  20:09:26 STEP  990 H= 247:30 +CPU= 43.166
  20:10:09 STEP  991 H= 247:45 +CPU= 43.036
  20:10:53 STEP  992 H= 248:00 +CPU= 43.502
  20:11:38 STEP  993 H= 248:15 +CPU= 45.144
  20:12:22 STEP  994 H= 248:30 +CPU= 43.973
  20:13:07 STEP  995 H= 248:45 +CPU= 44.253
  20:14:01 STEP  996 H= 249:00 +CPU= 54.853
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMGGh7zg+000996
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMSHh7zg+000996
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMUAh7zg+000996
  20:14:45 STEP  997 H= 249:15 +CPU= 43.914
  20:15:30 STEP  998 H= 249:30 +CPU= 44.087
  20:16:14 STEP  999 H= 249:45 +CPU= 44.475
  20:17:01 STEP 1000 H= 250:00 +CPU= 46.551
  20:17:46 STEP 1001 H= 250:15 +CPU= 44.598
  20:18:29 STEP 1002 H= 250:30 +CPU= 43.300
  20:19:15 STEP 1003 H= 250:45 +CPU= 45.341
  20:20:01 STEP 1004 H= 251:00 +CPU= 45.969
  20:20:46 STEP 1005 H= 251:15 +CPU= 45.294
  20:21:34 STEP 1006 H= 251:30 +CPU= 47.177
  20:22:20 STEP 1007 H= 251:45 +CPU= 45.900
  20:23:23 STEP 1008 H= 252:00 +CPU= 63.027
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMGGh7zg+001008
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMSHh7zg+001008
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMUAh7zg+001008
  20:24:07 STEP 1009 H= 252:15 +CPU= 44.324
  20:24:53 STEP 1010 H= 252:30 +CPU= 45.626
  20:25:39 STEP 1011 H= 252:45 +CPU= 46.239
  20:26:25 STEP 1012 H= 253:00 +CPU= 45.969
  20:27:11 STEP 1013 H= 253:15 +CPU= 45.547
  20:27:56 STEP 1014 H= 253:30 +CPU= 44.468
  20:28:41 STEP 1015 H= 253:45 +CPU= 44.891
  20:29:27 STEP 1016 H= 254:00 +CPU= 45.722
  20:30:12 STEP 1017 H= 254:15 +CPU= 45.088
  20:30:57 STEP 1018 H= 254:30 +CPU= 44.871
  20:31:45 STEP 1019 H= 254:45 +CPU= 47.581
  20:32:41 STEP 1020 H= 255:00 +CPU= 56.226
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMGGh7zg+001020
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMSHh7zg+001020
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMUAh7zg+001020
  20:33:25 STEP 1021 H= 255:15 +CPU= 44.287
  20:34:11 STEP 1022 H= 255:30 +CPU= 45.950
  20:34:56 STEP 1023 H= 255:45 +CPU= 44.220
  20:35:41 STEP 1024 H= 256:00 +CPU= 44.941
  20:36:26 STEP 1025 H= 256:15 +CPU= 45.656
  20:37:12 STEP 1026 H= 256:30 +CPU= 45.381
  20:37:58 STEP 1027 H= 256:45 +CPU= 45.834
  20:38:45 STEP 1028 H= 257:00 +CPU= 47.056
  20:39:32 STEP 1029 H= 257:15 +CPU= 46.525
  20:40:18 STEP 1030 H= 257:30 +CPU= 46.674
  20:41:04 STEP 1031 H= 257:45 +CPU= 45.764
  20:42:01 STEP 1032 H= 258:00 +CPU= 56.969
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMGGh7zg+001032
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMSHh7zg+001032
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMUAh7zg+001032
  20:42:46 STEP 1033 H= 258:15 +CPU= 44.730
  20:43:32 STEP 1034 H= 258:30 +CPU= 45.415
  20:44:18 STEP 1035 H= 258:45 +CPU= 46.007
  20:45:05 STEP 1036 H= 259:00 +CPU= 46.925
  20:45:51 STEP 1037 H= 259:15 +CPU= 46.307
  20:46:37 STEP 1038 H= 259:30 +CPU= 45.192
  20:47:22 STEP 1039 H= 259:45 +CPU= 45.711
  20:48:09 STEP 1040 H= 260:00 +CPU= 46.463
  20:48:54 STEP 1041 H= 260:15 +CPU= 44.332
  20:49:38 STEP 1042 H= 260:30 +CPU= 44.151
  20:50:23 STEP 1043 H= 260:45 +CPU= 45.310
  20:51:21 STEP 1044 H= 261:00 +CPU= 57.890
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMGGh7zg+001044
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMSHh7zg+001044
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMUAh7zg+001044
  20:52:06 STEP 1045 H= 261:15 +CPU= 44.801
  20:52:51 STEP 1046 H= 261:30 +CPU= 44.222
  20:53:35 STEP 1047 H= 261:45 +CPU= 44.284
  20:54:22 STEP 1048 H= 262:00 +CPU= 46.976
  20:55:06 STEP 1049 H= 262:15 +CPU= 43.992
  20:55:53 STEP 1050 H= 262:30 +CPU= 46.774
  20:56:41 STEP 1051 H= 262:45 +CPU= 47.261
  20:57:29 STEP 1052 H= 263:00 +CPU= 47.952
  20:58:17 STEP 1053 H= 263:15 +CPU= 46.823
  20:59:01 STEP 1054 H= 263:30 +CPU= 44.445
  20:59:45 STEP 1055 H= 263:45 +CPU= 43.969
  21:00:47 STEP 1056 H= 264:00 +CPU= 61.425
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMGGh7zg+001056
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMSHh7zg+001056
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMUAh7zg+001056
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMGGh7zg+000960
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMSHh7zg+000960
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMUAh7zg+000960
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMGGh7zg+000972
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMSHh7zg+000972
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMUAh7zg+000972
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMGGh7zg+000984
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMSHh7zg+000984
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMUAh7zg+000984
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMGGh7zg+000996
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMSHh7zg+000996
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMUAh7zg+000996
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMGGh7zg+001008
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMSHh7zg+001008
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMUAh7zg+001008
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMGGh7zg+001020
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMSHh7zg+001020
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMUAh7zg+001020
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMGGh7zg+001032
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMSHh7zg+001032
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMUAh7zg+001032
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMGGh7zg+001044
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMSHh7zg+001044
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMUAh7zg+001044
Zipping up the intermediate file: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_a28f_2016092300_15_991_12209682_0_r1681846707_10.zip
Uploading the intermediate file: upload_file_10.zip
  21:01:32 STEP 1057 H= 264:15 +CPU= 44.193
  21:02:19 STEP 1058 H= 264:30 +CPU= 46.864
  21:03:05 STEP 1059 H= 264:45 +CPU= 46.124
  21:03:56 STEP 1060 H= 265:00 +CPU= 50.243
  21:04:45 STEP 1061 H= 265:15 +CPU= 49.423
  21:05:30 STEP 1062 H= 265:30 +CPU= 44.427
  21:06:14 STEP 1063 H= 265:45 +CPU= 44.182
  21:07:00 STEP 1064 H= 266:00 +CPU= 45.778
  21:07:47 STEP 1065 H= 266:15 +CPU= 46.937
  21:08:35 STEP 1066 H= 266:30 +CPU= 47.438
  21:09:19 STEP 1067 H= 266:45 +CPU= 44.502
  21:10:16 STEP 1068 H= 267:00 +CPU= 56.775
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMGGh7zg+001068
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMSHh7zg+001068
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMUAh7zg+001068
  21:11:02 STEP 1069 H= 267:15 +CPU= 45.650
  21:11:47 STEP 1070 H= 267:30 +CPU= 44.889
  21:12:32 STEP 1071 H= 267:45 +CPU= 45.427
  21:13:27 STEP 1072 H= 268:00 +CPU= 54.581
  21:14:14 STEP 1073 H= 268:15 +CPU= 46.155
  21:14:59 STEP 1074 H= 268:30 +CPU= 45.843
  21:15:45 STEP 1075 H= 268:45 +CPU= 46.009
  21:16:33 STEP 1076 H= 269:00 +CPU= 47.274
  21:17:20 STEP 1077 H= 269:15 +CPU= 47.086
  21:18:08 STEP 1078 H= 269:30 +CPU= 47.622
  21:18:53 STEP 1079 H= 269:45 +CPU= 44.988
  21:19:49 STEP 1080 H= 270:00 +CPU= 56.645
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMGGh7zg+001080
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMSHh7zg+001080
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMUAh7zg+001080
  21:20:34 STEP 1081 H= 270:15 +CPU= 44.071
  21:21:21 STEP 1082 H= 270:30 +CPU= 47.058
  21:22:08 STEP 1083 H= 270:45 +CPU= 46.752
  21:22:55 STEP 1084 H= 271:00 +CPU= 47.495
  21:23:41 STEP 1085 H= 271:15 +CPU= 45.307
  21:24:24 STEP 1086 H= 271:30 +CPU= 43.442
  21:25:07 STEP 1087 H= 271:45 +CPU= 42.937
  21:25:52 STEP 1088 H= 272:00 +CPU= 44.478
  21:26:36 STEP 1089 H= 272:15 +CPU= 44.553
  21:27:19 STEP 1090 H= 272:30 +CPU= 42.896
  21:28:03 STEP 1091 H= 272:45 +CPU= 43.501
  21:28:57 STEP 1092 H= 273:00 +CPU= 54.205
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMGGh7zg+001092
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMSHh7zg+001092
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMUAh7zg+001092
  21:29:41 STEP 1093 H= 273:15 +CPU= 43.919
  21:30:28 STEP 1094 H= 273:30 +CPU= 46.011
  21:31:11 STEP 1095 H= 273:45 +CPU= 43.517
  21:31:56 STEP 1096 H= 274:00 +CPU= 44.145
  21:32:39 STEP 1097 H= 274:15 +CPU= 43.035
  21:33:22 STEP 1098 H= 274:30 +CPU= 43.243
  21:34:09 STEP 1099 H= 274:45 +CPU= 47.043
  21:34:57 STEP 1100 H= 275:00 +CPU= 48.250
  21:35:42 STEP 1101 H= 275:15 +CPU= 44.822
  21:36:26 STEP 1102 H= 275:30 +CPU= 43.913
  21:37:11 STEP 1103 H= 275:45 +CPU= 45.193
  21:38:13 STEP 1104 H= 276:00 +CPU= 61.509
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMGGh7zg+001104
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMSHh7zg+001104
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMUAh7zg+001104
  21:38:57 STEP 1105 H= 276:15 +CPU= 43.573
  21:39:40 STEP 1106 H= 276:30 +CPU= 43.623
  21:40:26 STEP 1107 H= 276:45 +CPU= 45.732
  21:41:12 STEP 1108 H= 277:00 +CPU= 46.122
  21:41:57 STEP 1109 H= 277:15 +CPU= 44.906
  21:42:42 STEP 1110 H= 277:30 +CPU= 44.408
  21:43:27 STEP 1111 H= 277:45 +CPU= 45.584
  21:44:15 STEP 1112 H= 278:00 +CPU= 47.978
  21:45:01 STEP 1113 H= 278:15 +CPU= 45.167
  21:45:46 STEP 1114 H= 278:30 +CPU= 45.496
  21:46:30 STEP 1115 H= 278:45 +CPU= 43.433
  21:47:25 STEP 1116 H= 279:00 +CPU= 55.703
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMGGh7zg+001116
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMSHh7zg+001116
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMUAh7zg+001116
  21:48:09 STEP 1117 H= 279:15 +CPU= 43.498
  21:48:53 STEP 1118 H= 279:30 +CPU= 43.892
  21:49:37 STEP 1119 H= 279:45 +CPU= 43.856
  21:50:22 STEP 1120 H= 280:00 +CPU= 44.816
  21:51:08 STEP 1121 H= 280:15 +CPU= 46.285
  21:51:55 STEP 1122 H= 280:30 +CPU= 46.918
  21:52:41 STEP 1123 H= 280:45 +CPU= 45.432
  21:53:25 STEP 1124 H= 281:00 +CPU= 43.970
  21:54:11 STEP 1125 H= 281:15 +CPU= 45.961
  21:54:55 STEP 1126 H= 281:30 +CPU= 43.866
  21:55:40 STEP 1127 H= 281:45 +CPU= 45.296
  21:56:35 STEP 1128 H= 282:00 +CPU= 55.414
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMGGh7zg+001128
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMSHh7zg+001128
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMUAh7zg+001128
  21:57:20 STEP 1129 H= 282:15 +CPU= 44.328
  21:58:05 STEP 1130 H= 282:30 +CPU= 45.280
  21:58:51 STEP 1131 H= 282:45 +CPU= 45.893
  21:59:38 STEP 1132 H= 283:00 +CPU= 46.506
  22:00:23 STEP 1133 H= 283:15 +CPU= 45.416
  22:01:09 STEP 1134 H= 283:30 +CPU= 46.394
  22:01:56 STEP 1135 H= 283:45 +CPU= 46.551
  22:02:41 STEP 1136 H= 284:00 +CPU= 44.808
  22:03:28 STEP 1137 H= 284:15 +CPU= 47.029
  22:04:16 STEP 1138 H= 284:30 +CPU= 47.747
  22:05:04 STEP 1139 H= 284:45 +CPU= 47.969
  22:05:59 STEP 1140 H= 285:00 +CPU= 54.886
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMGGh7zg+001140
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMSHh7zg+001140
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMUAh7zg+001140
  22:06:43 STEP 1141 H= 285:15 +CPU= 44.059
  22:07:27 STEP 1142 H= 285:30 +CPU= 43.850
  22:08:11 STEP 1143 H= 285:45 +CPU= 44.109
  22:08:55 STEP 1144 H= 286:00 +CPU= 44.358
  22:09:39 STEP 1145 H= 286:15 +CPU= 43.264
  22:10:23 STEP 1146 H= 286:30 +CPU= 43.932
  22:11:10 STEP 1147 H= 286:45 +CPU= 47.061
  22:11:56 STEP 1148 H= 287:00 +CPU= 46.302
  22:12:43 STEP 1149 H= 287:15 +CPU= 46.138
  22:13:26 STEP 1150 H= 287:30 +CPU= 43.537
  22:14:09 STEP 1151 H= 287:45 +CPU= 43.229
  22:15:10 STEP 1152 H= 288:00 +CPU= 60.138
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMGGh7zg+001152
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMSHh7zg+001152
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMUAh7zg+001152
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMGGh7zg+001056
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMSHh7zg+001056
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMUAh7zg+001056
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMGGh7zg+001068
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMSHh7zg+001068
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMUAh7zg+001068
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMGGh7zg+001080
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMSHh7zg+001080
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMUAh7zg+001080
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMGGh7zg+001092
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMSHh7zg+001092
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMUAh7zg+001092
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMGGh7zg+001104
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMSHh7zg+001104
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMUAh7zg+001104
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMGGh7zg+001116
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMSHh7zg+001116
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMUAh7zg+001116
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMGGh7zg+001128
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMSHh7zg+001128
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMUAh7zg+001128
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMGGh7zg+001140
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMSHh7zg+001140
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMUAh7zg+001140
Zipping up the intermediate file: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_a28f_2016092300_15_991_12209682_0_r1681846707_11.zip
Uploading the intermediate file: upload_file_11.zip
  22:15:57 STEP 1153 H= 288:15 +CPU= 47.378
  22:16:44 STEP 1154 H= 288:30 +CPU= 47.226
  22:17:32 STEP 1155 H= 288:45 +CPU= 47.897
  22:18:19 STEP 1156 H= 289:00 +CPU= 46.264
  22:19:03 STEP 1157 H= 289:15 +CPU= 44.403
  22:19:48 STEP 1158 H= 289:30 +CPU= 45.043
  22:20:34 STEP 1159 H= 289:45 +CPU= 45.395
  22:21:20 STEP 1160 H= 290:00 +CPU= 45.666
  22:22:05 STEP 1161 H= 290:15 +CPU= 45.660
  22:22:50 STEP 1162 H= 290:30 +CPU= 44.792
  22:23:37 STEP 1163 H= 290:45 +CPU= 47.010
  22:24:38 STEP 1164 H= 291:00 +CPU= 60.637
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMGGh7zg+001164
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMSHh7zg+001164
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMUAh7zg+001164
  22:25:25 STEP 1165 H= 291:15 +CPU= 47.106
  22:26:12 STEP 1166 H= 291:30 +CPU= 47.513
  22:26:59 STEP 1167 H= 291:45 +CPU= 46.907
  22:27:49 STEP 1168 H= 292:00 +CPU= 49.551
  22:28:34 STEP 1169 H= 292:15 +CPU= 45.344
  22:29:21 STEP 1170 H= 292:30 +CPU= 46.073
  22:30:06 STEP 1171 H= 292:45 +CPU= 44.973
  22:30:53 STEP 1172 H= 293:00 +CPU= 47.117
  22:31:39 STEP 1173 H= 293:15 +CPU= 46.187
  22:32:26 STEP 1174 H= 293:30 +CPU= 46.461
  22:33:12 STEP 1175 H= 293:45 +CPU= 45.860
  22:34:10 STEP 1176 H= 294:00 +CPU= 58.446
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMGGh7zg+001176
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMSHh7zg+001176
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMUAh7zg+001176
  22:34:56 STEP 1177 H= 294:15 +CPU= 45.648
  22:35:42 STEP 1178 H= 294:30 +CPU= 46.001
  22:36:27 STEP 1179 H= 294:45 +CPU= 44.747
  22:37:16 STEP 1180 H= 295:00 +CPU= 48.577
  22:38:02 STEP 1181 H= 295:15 +CPU= 46.252
  22:38:50 STEP 1182 H= 295:30 +CPU= 47.873
  22:39:36 STEP 1183 H= 295:45 +CPU= 45.731
  22:40:24 STEP 1184 H= 296:00 +CPU= 47.886
  22:41:10 STEP 1185 H= 296:15 +CPU= 46.458
  22:41:57 STEP 1186 H= 296:30 +CPU= 46.834
  22:42:47 STEP 1187 H= 296:45 +CPU= 50.010
  22:43:45 STEP 1188 H= 297:00 +CPU= 57.653
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMGGh7zg+001188
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMSHh7zg+001188
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMUAh7zg+001188
  22:44:35 STEP 1189 H= 297:15 +CPU= 50.009
  22:45:24 STEP 1190 H= 297:30 +CPU= 48.311
  22:46:14 STEP 1191 H= 297:45 +CPU= 50.422
  22:47:04 STEP 1192 H= 298:00 +CPU= 49.264
  22:47:49 STEP 1193 H= 298:15 +CPU= 45.282
  22:48:35 STEP 1194 H= 298:30 +CPU= 46.077
  22:49:24 STEP 1195 H= 298:45 +CPU= 48.483
  22:50:15 STEP 1196 H= 299:00 +CPU= 51.183
  22:51:00 STEP 1197 H= 299:15 +CPU= 45.272
  22:51:46 STEP 1198 H= 299:30 +CPU= 45.496
  22:52:31 STEP 1199 H= 299:45 +CPU= 44.632
  22:53:35 STEP 1200 H= 300:00 +CPU= 64.767
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMGGh7zg+001200
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMSHh7zg+001200
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMUAh7zg+001200
  22:54:21 STEP 1201 H= 300:15 +CPU= 45.786
  22:55:09 STEP 1202 H= 300:30 +CPU= 48.065
  22:55:58 STEP 1203 H= 300:45 +CPU= 48.288
  22:56:45 STEP 1204 H= 301:00 +CPU= 46.939
  22:57:30 STEP 1205 H= 301:15 +CPU= 45.359
  22:58:16 STEP 1206 H= 301:30 +CPU= 45.151
  22:59:02 STEP 1207 H= 301:45 +CPU= 46.816
  22:59:50 STEP 1208 H= 302:00 +CPU= 47.947
  23:00:36 STEP 1209 H= 302:15 +CPU= 45.195
  23:01:23 STEP 1210 H= 302:30 +CPU= 47.336
  23:02:07 STEP 1211 H= 302:45 +CPU= 44.322
  23:03:04 STEP 1212 H= 303:00 +CPU= 56.446
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMGGh7zg+001212
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMSHh7zg+001212
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMUAh7zg+001212
  23:03:49 STEP 1213 H= 303:15 +CPU= 45.351
  23:04:36 STEP 1214 H= 303:30 +CPU= 46.935
  23:05:20 STEP 1215 H= 303:45 +CPU= 44.163
  23:06:05 STEP 1216 H= 304:00 +CPU= 44.495
  23:06:49 STEP 1217 H= 304:15 +CPU= 43.834
  23:07:37 STEP 1218 H= 304:30 +CPU= 47.694
  23:08:22 STEP 1219 H= 304:45 +CPU= 45.675
  23:09:09 STEP 1220 H= 305:00 +CPU= 46.850
  23:09:53 STEP 1221 H= 305:15 +CPU= 43.929
  23:10:39 STEP 1222 H= 305:30 +CPU= 45.879
  23:11:26 STEP 1223 H= 305:45 +CPU= 47.438
  23:12:23 STEP 1224 H= 306:00 +CPU= 56.648
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMGGh7zg+001224
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMSHh7zg+001224
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMUAh7zg+001224
  23:13:08 STEP 1225 H= 306:15 +CPU= 45.139
  23:13:52 STEP 1226 H= 306:30 +CPU= 43.359
  23:14:36 STEP 1227 H= 306:45 +CPU= 44.482
  23:15:20 STEP 1228 H= 307:00 +CPU= 44.330
  23:16:05 STEP 1229 H= 307:15 +CPU= 44.154
  23:16:49 STEP 1230 H= 307:30 +CPU= 43.979
  23:17:33 STEP 1231 H= 307:45 +CPU= 44.111
  23:18:18 STEP 1232 H= 308:00 +CPU= 45.633
  23:19:04 STEP 1233 H= 308:15 +CPU= 45.341
  23:19:50 STEP 1234 H= 308:30 +CPU= 45.188
  23:20:36 STEP 1235 H= 308:45 +CPU= 46.753
  23:21:33 STEP 1236 H= 309:00 +CPU= 56.370
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMGGh7zg+001236
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMSHh7zg+001236
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMUAh7zg+001236
  23:22:16 STEP 1237 H= 309:15 +CPU= 43.183
  23:23:00 STEP 1238 H= 309:30 +CPU= 43.884
  23:23:44 STEP 1239 H= 309:45 +CPU= 43.995
  23:24:30 STEP 1240 H= 310:00 +CPU= 45.448
  23:25:15 STEP 1241 H= 310:15 +CPU= 44.784
  23:25:59 STEP 1242 H= 310:30 +CPU= 43.845
  23:26:43 STEP 1243 H= 310:45 +CPU= 44.395
  23:27:29 STEP 1244 H= 311:00 +CPU= 45.387
  23:28:13 STEP 1245 H= 311:15 +CPU= 44.291
  23:28:58 STEP 1246 H= 311:30 +CPU= 44.376
  23:29:41 STEP 1247 H= 311:45 +CPU= 43.187
  23:30:45 STEP 1248 H= 312:00 +CPU= 63.890
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMGGh7zg+001248
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMSHh7zg+001248
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMUAh7zg+001248
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMGGh7zg+001152
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMSHh7zg+001152
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMUAh7zg+001152
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMGGh7zg+001164
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMSHh7zg+001164
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMUAh7zg+001164
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMGGh7zg+001176
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMSHh7zg+001176
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMUAh7zg+001176
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMGGh7zg+001188
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMSHh7zg+001188
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMUAh7zg+001188
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMGGh7zg+001200
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMSHh7zg+001200
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMUAh7zg+001200
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMGGh7zg+001212
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMSHh7zg+001212
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMUAh7zg+001212
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMGGh7zg+001224
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMSHh7zg+001224
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMUAh7zg+001224
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMGGh7zg+001236
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMSHh7zg+001236
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMUAh7zg+001236
Zipping up the intermediate file: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_a28f_2016092300_15_991_12209682_0_r1681846707_12.zip
Uploading the intermediate file: upload_file_12.zip
  23:31:30 STEP 1249 H= 312:15 +CPU= 45.039
  23:32:14 STEP 1250 H= 312:30 +CPU= 44.093
  23:32:59 STEP 1251 H= 312:45 +CPU= 45.069
  23:33:46 STEP 1252 H= 313:00 +CPU= 47.026
  23:34:32 STEP 1253 H= 313:15 +CPU= 45.412
  23:35:16 STEP 1254 H= 313:30 +CPU= 43.955
  23:36:01 STEP 1255 H= 313:45 +CPU= 45.547
  23:36:46 STEP 1256 H= 314:00 +CPU= 44.758
  23:37:30 STEP 1257 H= 314:15 +CPU= 43.624
  23:38:15 STEP 1258 H= 314:30 +CPU= 44.812
  23:38:59 STEP 1259 H= 314:45 +CPU= 44.454
  23:39:54 STEP 1260 H= 315:00 +CPU= 54.279
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMGGh7zg+001260
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMSHh7zg+001260
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMUAh7zg+001260
  23:40:37 STEP 1261 H= 315:15 +CPU= 43.440
  23:41:22 STEP 1262 H= 315:30 +CPU= 45.022
  23:42:07 STEP 1263 H= 315:45 +CPU= 44.215
  23:42:52 STEP 1264 H= 316:00 +CPU= 44.796
  23:43:36 STEP 1265 H= 316:15 +CPU= 44.392
  23:44:20 STEP 1266 H= 316:30 +CPU= 43.843
  23:45:07 STEP 1267 H= 316:45 +CPU= 47.048
  23:45:55 STEP 1268 H= 317:00 +CPU= 48.491
  23:46:42 STEP 1269 H= 317:15 +CPU= 46.306
  23:47:26 STEP 1270 H= 317:30 +CPU= 44.095
  23:48:10 STEP 1271 H= 317:45 +CPU= 43.567
  23:49:04 STEP 1272 H= 318:00 +CPU= 54.664
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMGGh7zg+001272
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMSHh7zg+001272
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMUAh7zg+001272
  23:49:48 STEP 1273 H= 318:15 +CPU= 44.135
  23:50:32 STEP 1274 H= 318:30 +CPU= 43.743
  23:51:18 STEP 1275 H= 318:45 +CPU= 45.227
  23:52:07 STEP 1276 H= 319:00 +CPU= 49.337
  23:52:53 STEP 1277 H= 319:15 +CPU= 46.036
  23:53:40 STEP 1278 H= 319:30 +CPU= 46.762
  23:54:25 STEP 1279 H= 319:45 +CPU= 44.561
  23:55:12 STEP 1280 H= 320:00 +CPU= 47.029
  23:55:57 STEP 1281 H= 320:15 +CPU= 44.836
  23:56:42 STEP 1282 H= 320:30 +CPU= 44.960
  23:57:27 STEP 1283 H= 320:45 +CPU= 44.937
  23:58:21 STEP 1284 H= 321:00 +CPU= 54.659
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMGGh7zg+001284
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMSHh7zg+001284
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMUAh7zg+001284
  23:59:08 STEP 1285 H= 321:15 +CPU= 46.160
  23:59:53 STEP 1286 H= 321:30 +CPU= 45.190
  00:00:41 STEP 1287 H= 321:45 +CPU= 47.692
  00:01:26 STEP 1288 H= 322:00 +CPU= 45.697
  00:02:11 STEP 1289 H= 322:15 +CPU= 44.171
  00:02:56 STEP 1290 H= 322:30 +CPU= 45.553
  00:03:44 STEP 1291 H= 322:45 +CPU= 47.838
  00:04:33 STEP 1292 H= 323:00 +CPU= 48.392
  00:05:21 STEP 1293 H= 323:15 +CPU= 48.231
  00:06:07 STEP 1294 H= 323:30 +CPU= 46.210
  00:06:55 STEP 1295 H= 323:45 +CPU= 48.089
  00:07:58 STEP 1296 H= 324:00 +CPU= 62.829
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMGGh7zg+001296
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMSHh7zg+001296
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMUAh7zg+001296
  00:08:44 STEP 1297 H= 324:15 +CPU= 45.679
  00:09:31 STEP 1298 H= 324:30 +CPU= 47.514
  00:10:18 STEP 1299 H= 324:45 +CPU= 46.820
  00:11:06 STEP 1300 H= 325:00 +CPU= 47.110
  00:11:54 STEP 1301 H= 325:15 +CPU= 47.764
  00:12:42 STEP 1302 H= 325:30 +CPU= 47.951
  00:13:30 STEP 1303 H= 325:45 +CPU= 47.812
  00:14:18 STEP 1304 H= 326:00 +CPU= 47.812
  00:15:04 STEP 1305 H= 326:15 +CPU= 46.386
  00:15:52 STEP 1306 H= 326:30 +CPU= 47.774
  00:16:38 STEP 1307 H= 326:45 +CPU= 45.407
  00:17:35 STEP 1308 H= 327:00 +CPU= 56.959
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMGGh7zg+001308
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMSHh7zg+001308
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMUAh7zg+001308
  00:18:19 STEP 1309 H= 327:15 +CPU= 44.015
  00:19:03 STEP 1310 H= 327:30 +CPU= 44.348
  00:19:48 STEP 1311 H= 327:45 +CPU= 44.536
  00:20:34 STEP 1312 H= 328:00 +CPU= 45.820
  00:21:19 STEP 1313 H= 328:15 +CPU= 45.197
  00:22:04 STEP 1314 H= 328:30 +CPU= 44.657
  00:22:48 STEP 1315 H= 328:45 +CPU= 44.045
  00:23:34 STEP 1316 H= 329:00 +CPU= 46.517
  00:24:20 STEP 1317 H= 329:15 +CPU= 45.496
  00:25:07 STEP 1318 H= 329:30 +CPU= 47.061
  00:25:51 STEP 1319 H= 329:45 +CPU= 43.924
  00:26:48 STEP 1320 H= 330:00 +CPU= 56.208
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMGGh7zg+001320
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMSHh7zg+001320
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMUAh7zg+001320
  00:27:33 STEP 1321 H= 330:15 +CPU= 45.259
  00:28:18 STEP 1322 H= 330:30 +CPU= 44.915
  00:29:03 STEP 1323 H= 330:45 +CPU= 44.741
  00:29:48 STEP 1324 H= 331:00 +CPU= 45.469
  00:30:34 STEP 1325 H= 331:15 +CPU= 46.173
  00:31:22 STEP 1326 H= 331:30 +CPU= 47.268
  00:32:07 STEP 1327 H= 331:45 +CPU= 45.300
  00:32:54 STEP 1328 H= 332:00 +CPU= 46.922
  00:33:40 STEP 1329 H= 332:15 +CPU= 45.771
  00:34:26 STEP 1330 H= 332:30 +CPU= 46.283
  00:35:15 STEP 1331 H= 332:45 +CPU= 48.388
  00:36:13 STEP 1332 H= 333:00 +CPU= 57.226
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMGGh7zg+001332
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMSHh7zg+001332
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMUAh7zg+001332
  00:37:00 STEP 1333 H= 333:15 +CPU= 47.583
  00:37:49 STEP 1334 H= 333:30 +CPU= 49.048
  00:38:39 STEP 1335 H= 333:45 +CPU= 49.534
  00:39:28 STEP 1336 H= 334:00 +CPU= 49.200
  00:40:13 STEP 1337 H= 334:15 +CPU= 45.386
  00:40:59 STEP 1338 H= 334:30 +CPU= 44.850
  00:41:43 STEP 1339 H= 334:45 +CPU= 44.175
  00:42:30 STEP 1340 H= 335:00 +CPU= 46.476
  00:43:18 STEP 1341 H= 335:15 +CPU= 48.427
  00:44:07 STEP 1342 H= 335:30 +CPU= 49.006
  00:44:55 STEP 1343 H= 335:45 +CPU= 47.878
  00:45:58 STEP 1344 H= 336:00 +CPU= 62.373
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMGGh7zg+001344
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMSHh7zg+001344
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMUAh7zg+001344
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMGGh7zg+001248
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMSHh7zg+001248
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMUAh7zg+001248
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMGGh7zg+001260
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMSHh7zg+001260
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMUAh7zg+001260
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMGGh7zg+001272
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMSHh7zg+001272
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMUAh7zg+001272
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMGGh7zg+001284
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMSHh7zg+001284
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMUAh7zg+001284
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMGGh7zg+001296
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMSHh7zg+001296
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMUAh7zg+001296
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMGGh7zg+001308
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMSHh7zg+001308
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMUAh7zg+001308
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMGGh7zg+001320
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMSHh7zg+001320
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMUAh7zg+001320
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMGGh7zg+001332
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMSHh7zg+001332
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMUAh7zg+001332
Zipping up the intermediate file: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_a28f_2016092300_15_991_12209682_0_r1681846707_13.zip
Uploading the intermediate file: upload_file_13.zip
  00:46:44 STEP 1345 H= 336:15 +CPU= 45.743
  00:47:32 STEP 1346 H= 336:30 +CPU= 48.297
  00:48:22 STEP 1347 H= 336:45 +CPU= 49.483
  00:49:08 STEP 1348 H= 337:00 +CPU= 46.542
  00:49:57 STEP 1349 H= 337:15 +CPU= 47.861
  00:50:42 STEP 1350 H= 337:30 +CPU= 45.428
  00:51:28 STEP 1351 H= 337:45 +CPU= 45.500
  00:52:14 STEP 1352 H= 338:00 +CPU= 46.417
  00:53:02 STEP 1353 H= 338:15 +CPU= 47.493
  00:53:52 STEP 1354 H= 338:30 +CPU= 49.487
  00:54:36 STEP 1355 H= 338:45 +CPU= 44.774
  00:55:31 STEP 1356 H= 339:00 +CPU= 54.770
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMGGh7zg+001356
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMSHh7zg+001356
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMUAh7zg+001356
  00:56:16 STEP 1357 H= 339:15 +CPU= 45.033
  00:57:00 STEP 1358 H= 339:30 +CPU= 43.800
  00:57:44 STEP 1359 H= 339:45 +CPU= 43.925
  00:58:29 STEP 1360 H= 340:00 +CPU= 44.550
  00:59:13 STEP 1361 H= 340:15 +CPU= 43.997
  01:00:00 STEP 1362 H= 340:30 +CPU= 46.721
  01:00:45 STEP 1363 H= 340:45 +CPU= 45.252
  01:01:33 STEP 1364 H= 341:00 +CPU= 47.688
  01:02:21 STEP 1365 H= 341:15 +CPU= 47.259
  01:03:11 STEP 1366 H= 341:30 +CPU= 50.265
  01:04:03 STEP 1367 H= 341:45 +CPU= 51.865
  01:05:03 STEP 1368 H= 342:00 +CPU= 60.471
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMGGh7zg+001368
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMSHh7zg+001368
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMUAh7zg+001368
  01:05:51 STEP 1369 H= 342:15 +CPU= 47.473
  01:06:36 STEP 1370 H= 342:30 +CPU= 44.470
  01:07:21 STEP 1371 H= 342:45 +CPU= 45.514
  01:08:08 STEP 1372 H= 343:00 +CPU= 46.816
  01:08:52 STEP 1373 H= 343:15 +CPU= 44.055
  01:09:37 STEP 1374 H= 343:30 +CPU= 44.697
  01:10:21 STEP 1375 H= 343:45 +CPU= 43.982
  01:11:08 STEP 1376 H= 344:00 +CPU= 46.454
  01:11:54 STEP 1377 H= 344:15 +CPU= 46.002
  01:12:40 STEP 1378 H= 344:30 +CPU= 45.861
  01:13:25 STEP 1379 H= 344:45 +CPU= 45.423
  01:14:20 STEP 1380 H= 345:00 +CPU= 55.073
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMGGh7zg+001380
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMSHh7zg+001380
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMUAh7zg+001380
  01:15:07 STEP 1381 H= 345:15 +CPU= 46.706
  01:15:51 STEP 1382 H= 345:30 +CPU= 44.201
  01:16:38 STEP 1383 H= 345:45 +CPU= 46.309
  01:17:25 STEP 1384 H= 346:00 +CPU= 47.266
  01:18:10 STEP 1385 H= 346:15 +CPU= 44.330
  01:18:54 STEP 1386 H= 346:30 +CPU= 44.421
  01:19:41 STEP 1387 H= 346:45 +CPU= 46.310
  01:20:27 STEP 1388 H= 347:00 +CPU= 46.081
  01:21:12 STEP 1389 H= 347:15 +CPU= 45.287
  01:21:59 STEP 1390 H= 347:30 +CPU= 46.417
  01:22:45 STEP 1391 H= 347:45 +CPU= 46.140
  01:23:50 STEP 1392 H= 348:00 +CPU= 64.757
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMGGh7zg+001392
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMSHh7zg+001392
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMUAh7zg+001392
  01:24:35 STEP 1393 H= 348:15 +CPU= 45.282
  01:25:19 STEP 1394 H= 348:30 +CPU= 44.063
  01:26:04 STEP 1395 H= 348:45 +CPU= 44.461
  01:26:51 STEP 1396 H= 349:00 +CPU= 46.835
  01:27:36 STEP 1397 H= 349:15 +CPU= 44.577
  01:28:20 STEP 1398 H= 349:30 +CPU= 44.343
  01:29:05 STEP 1399 H= 349:45 +CPU= 44.869
  01:29:52 STEP 1400 H= 350:00 +CPU= 46.932
  01:30:38 STEP 1401 H= 350:15 +CPU= 46.066
  01:31:25 STEP 1402 H= 350:30 +CPU= 46.490
  01:32:11 STEP 1403 H= 350:45 +CPU= 45.539
  01:33:07 STEP 1404 H= 351:00 +CPU= 56.231
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMGGh7zg+001404
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMSHh7zg+001404
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMUAh7zg+001404
  01:33:51 STEP 1405 H= 351:15 +CPU= 44.169
  01:34:36 STEP 1406 H= 351:30 +CPU= 44.650
  01:35:20 STEP 1407 H= 351:45 +CPU= 43.742
  01:36:07 STEP 1408 H= 352:00 +CPU= 47.017
  01:36:52 STEP 1409 H= 352:15 +CPU= 44.297
  01:37:37 STEP 1410 H= 352:30 +CPU= 45.060
  01:38:21 STEP 1411 H= 352:45 +CPU= 44.039
  01:39:06 STEP 1412 H= 353:00 +CPU= 45.011
  01:39:51 STEP 1413 H= 353:15 +CPU= 44.100
  01:40:35 STEP 1414 H= 353:30 +CPU= 44.073
  01:41:19 STEP 1415 H= 353:45 +CPU= 44.141
  01:42:15 STEP 1416 H= 354:00 +CPU= 55.729
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMGGh7zg+001416
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMSHh7zg+001416
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMUAh7zg+001416
  01:43:02 STEP 1417 H= 354:15 +CPU= 47.400
  01:43:49 STEP 1418 H= 354:30 +CPU= 46.624
  01:44:33 STEP 1419 H= 354:45 +CPU= 44.456
  01:45:18 STEP 1420 H= 355:00 +CPU= 44.932
  01:46:05 STEP 1421 H= 355:15 +CPU= 46.449
  01:46:50 STEP 1422 H= 355:30 +CPU= 45.251
  01:47:36 STEP 1423 H= 355:45 +CPU= 46.240
  01:48:23 STEP 1424 H= 356:00 +CPU= 47.029
  01:49:07 STEP 1425 H= 356:15 +CPU= 43.859
  01:49:52 STEP 1426 H= 356:30 +CPU= 44.272
  01:50:38 STEP 1427 H= 356:45 +CPU= 46.086
  01:51:32 STEP 1428 H= 357:00 +CPU= 53.639
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMGGh7zg+001428
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMSHh7zg+001428
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMUAh7zg+001428
  01:52:14 STEP 1429 H= 357:15 +CPU= 42.420
  01:52:58 STEP 1430 H= 357:30 +CPU= 43.886
  01:53:42 STEP 1431 H= 357:45 +CPU= 43.600
  01:54:27 STEP 1432 H= 358:00 +CPU= 45.664
  01:55:11 STEP 1433 H= 358:15 +CPU= 43.433
  01:55:54 STEP 1434 H= 358:30 +CPU= 43.407
  01:56:37 STEP 1435 H= 358:45 +CPU= 42.190
  01:57:20 STEP 1436 H= 359:00 +CPU= 43.880
  01:58:03 STEP 1437 H= 359:15 +CPU= 42.656
  01:58:47 STEP 1438 H= 359:30 +CPU= 43.556
  01:59:30 STEP 1439 H= 359:45 +CPU= 43.503
  02:00:26 STEP 1440 H= 360:00 +CPU= 55.752
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.11_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a28f
(argv4) batchid: 991
(argv5) wuid: 12209682
(argv6) fclen: 15
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client50002/slots/1
Project directory is: /var/lib/boinc-client50002/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.11
Location of temp folder: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client50002/slots/1/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client50002/slots/1/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_41af80a075e85a8f1458c05f530d2fdb to: /var/lib/boinc-client50002/slots/1/oifs_43r3_bl_a28f_2016092300_15_991_12209682.zip
Unzipping the namelist zip file: /var/lib/boinc-client50002/slots/1/oifs_43r3_bl_a28f_2016092300_15_991_12209682.zip
ic_ancil_file: ic_ancil_12209682
ifsdata_file: ifsdata_12209682
climate_data_file: clim_data_12209682
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /var/lib/boinc-client50002/slots/1/ic_ancil_12209682.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client50002/slots/1/ic_ancil_12209682.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_012603835590dda067b0c12d4edf853f to: /var/lib/boinc-client50002/slots/1/ifsdata/ifsdata_12209682.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client50002/slots/1/ifsdata/ifsdata_12209682.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client50002/slots/1/159l_2/clim_data_12209682.zip
Unzipping the climate data zip file: /var/lib/boinc-client50002/slots/1/159l_2/clim_data_12209682.zip
Checking for progress XML file: /var/lib/boinc-client50002/slots/1/progress_file_12209682.xml
Opened progress file ok : /var/lib/boinc-client50002/slots/1/progress_file_12209682.xml
-- Model is restarting --
Adjusting last_iter, 1440, to previous model restart step.
Creating progress file: /var/lib/boinc-client50002/slots/1/progress_file_12209682.xml
last_cpu_time: 67066
upload_file_number: 14
last_iter: 1441
last_upload: 1209600
model_completed: 0
total_length_of_simulation: 1296000
result_base_name: oifs_43r3_bl_a28f_2016092300_15_991_12209682_0_r1681846707
The child process has been launched with process id: 878376
Executing the command: /var/lib/boinc-client50002/slots/1/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:proliant-ml350-g6:1:1:878376:878376] [20230218:072556:1676705156:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2091] fp = 0x30e4a00
[EC_DRHOOK:proliant-ml350-g6:1:1:878376:878376] [20230218:072556:1676705156:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:proliant-ml350-g6:1:1:878376:878376] [20230218:072556:1676705156:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:proliant-ml350-g6:1:1:878376:878376] [20230218:072556:1676705156:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:proliant-ml350-g6:1:1:878376:878376] [20230218:072556:1676705156:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:proliant-ml350-g6:1:1:878376:878376] [20230218:072556:1676705156:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:proliant-ml350-g6:1:1:878376:878376] [20230218:072556:1676705156:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:proliant-ml350-g6:1:1:878376:878376] [20230218:072556:1676705156:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:proliant-ml350-g6:1:1:878376:878376] [20230218:072556:1676705156:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:proliant-ml350-g6:1:1:878376:878376] [20230218:072556:1676705156:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:proliant-ml350-g6:1:1:878376:878376] [20230218:072556:1676705156:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:proliant-ml350-g6:1:1:878376:878376] [20230218:072556:1676705156:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:proliant-ml350-g6:1:1:878376:878376] [20230218:072556:1676705156:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:proliant-ml350-g6:1:1:878376:878376] [20230218:072556:1676705156:0.000] [ignore_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:proliant-ml350-g6:1:1:878376:878376] [20230218:072556:1676705156:0.000] [restore_default_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:proliant-ml350-g6:1:1:878376:878376] [20230218:072556:1676705156:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:proliant-ml350-g6:1:1:878376:878376] [20230218:072556:1676705156:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f520 (old at (nil))
[EC_DRHOOK:proliant-ml350-g6:1:1:878376:878376] [20230218:072556:1676705156:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:proliant-ml350-g6:1:1:878376:878376] [20230218:072556:1676705156:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f520 (old at (nil))
[EC_DRHOOK:proliant-ml350-g6:1:1:878376:878376] [20230218:072556:1676705156:0.000] [trapfpe_treatment@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:proliant-ml350-g6:1:1:878376:878376] [20230218:072556:1676705156:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:proliant-ml350-g6:1:1:878376:878376] [20230218:072556:1676705156:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:proliant-ml350-g6:1:1:878376:878376] [20230218:072556:1676705156:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f520 (old at (nil))
[EC_DRHOOK:proliant-ml350-g6:1:1:878376:878376] [20230218:072556:1676705156:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:proliant-ml350-g6:1:1:878376:878376] [20230218:072556:1676705156:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:proliant-ml350-g6:1:1:878376:878376] [20230218:072556:1676705156:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:proliant-ml350-g6:1:1:878376:878376] [20230218:072556:1676705156:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f520 (old at (nil))
[EC_DRHOOK:proliant-ml350-g6:1:1:878376:878376] [20230218:072556:1676705156:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f520 (old at (nil))
[EC_DRHOOK:proliant-ml350-g6:1:1:878376:878376] [20230218:072556:1676705156:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f520 (old at (nil))
[EC_DRHOOK:proliant-ml350-g6:1:1:878376:878376] [20230218:072556:1676705156:0.001] [catch_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  07:26:40 STEP 1440 H= 360:00 +CPU= 39.756
..The child process terminated with status: 0
>>> Printing last 70 lines from file: NODE.001_01

I      I Hor. Flux of PEII      0.00000 I              I              I              I               I
I      I Vert.Flux of PEII      0.00000 I              I              I              I               I
I      I Work Pres.G.F  II              I      0.04080 I              I              I               I
I      I Conv. Ad.-Int. II     -0.04080 I              I              I              I               I
DDH--------------------------------------------------------------------------------------------------I
I DYN- Horizont. Flux   II      0.01141 I      0.00187 I      0.12597 I      0.00146 I     -0.00014  I
I    I Vertical Flux    II      0.00000 I      0.00000 I      0.00000 I      0.00000 I      0.00000  I
DDH--------------------------------------------------------------------------------------------------I
  
DDH----------------------------------------------------------------------------------------------------------------------------------
  I   - Mass balance    (10**(-6)  KG/M**2) 
  
  I INITIAL      I HORIZ. FLUX  I DYN. FLUX    I DYN. FLUX    I PHYS. FLUX   I PHYS. FLUX   I CURRENT      I I I RESIDUAL     I
  I MASS         I              I TOA          I SURFACE      I TOA          I SURFACE      I MASS         I I I              I
  ---------------------------------------------------------------------------------------------------------- I ----------------
  -              -              -              -              -              -              -              - I -              -
  - 10206523039. I           0. .           0. .           0. .           0. .           0. I 10206580454. - I -       57414. -
  -              -              -              -              -              -              -              - I -              -
  ---------------------------------------------------------------------------------------------------------- I ----------------
 
DDH------------------------------------------------------------------------
I                                                                         I
I TIME =  360.000H DDH * Diagnostic Var.,     AREA                GLOBAL  I
I                                                                         I
I-------------------------------------------------------------------------I
I   VARIABLE   I      Initial value         I        Current value        I
I-------------------------------------------------------------------------I
I  OMEGA (P/S) I      0.7674077574E-21      I       0.1582592817E-05      I
ICloud fractionI      0.0000000000E+00      I       0.8267973707E-02      I
I  Relat. Hum. I      0.4456967856E+00      I       0.4593759016E+00      I
I  PASS 01     I      0.0000000000E+00      I       0.1382215916E+03      I
DDH------------------------------------------------------------------------
  
 07:26:40 STEP 1440 H= 360:00 +CPU= 39.756
 END CNT3
 NSTEP =  1440 CNT0   000000000  39.016  39.016  39.022   1.000               0               0               0               0               0
IO-STREAM STATISTICS, TOTAL NO OF REC/BYTES READ           34/  1069125708 WRITTEN            0/           0
 -TASK-OPENED-OPEN-RECS IN -KBYTE IN    -RECS OUT-KBYTE OUT-WALL   -WALL IN-WALL
  OU-TOT IN -TOT OUT
   1      3    0       34      1044068        0            0     1.8     1.8     0.0     2.7     0.0
===-=== START OF TIMING STATISTICS ===-===
 
STATS FOR ALL TASKS
 NUM ROUTINE                                     CALLS  MEAN(ms)   MAX(ms)   FRAC(%)  UNBAL(%)
   0 CNT0     - COMPLETE EXECUTION                   1   44410.6   44410.6    100.00      0.00
   1 CNT4     - FORWARD INTEGRATION                  1   39762.6   39762.6     89.53      0.00
   8 SCAN2M - GRID-POINT DYNAMICS                    1    6994.6    6994.6     15.75      0.00
  10 SCAN2M - PHYSICS                                1   20287.2   20287.2     45.68      0.00
  12 SPNORM   - SPECTRAL NORM COMP.                  1     105.5     105.5      0.24      0.00
  28 RESTART FILES - READING                         1    3892.3    3892.3      8.76      0.00
  31 POSDDH IN STEPO                                 1       1.6       1.6      0.00      0.00
  37 CPGLAG   - SL COMPUTATIONS                      1    8919.7    8919.7     20.08      0.00
  39 SU0YOMB                                         1     489.4     489.4      1.10      0.00
  51 SCAN2M   - SL COMM. PART 1                      1      31.2      31.2      0.07      0.00
  56 SPNORM   - SPECTRAL NORM COMM.                  1       0.4       0.4      0.00      0.00
 102 LTINV_CTL   - INVERSE LEGENDRE TRANSFORM        4     350.1     350.1      3.15      0.00
 107 FTINV_CTL   - INVERSE FOURIER TRANSFORM         4     182.0     182.0      1.64      0.00
 140 SULEG       - COMP. OF LEGENDRE POL.            1      60.4      60.4      0.14      0.00
 152 LTINV_CTL   - M TO L TRANSPOSITION              4      31.5      31.5      0.28      0.00
 157 FTINV_CTL   - L TO G TRANSPOSITION              4      94.2      94.2      0.85      0.00
 400 GSTATS                                       1083       0.0       0.0      0.00      0.00
 401 GSTATS HOOK                                   963       0.0       0.0      0.00      0.00
TOTAL MEASURED IMBALANCE =       0.0 SECONDS,  0.0 PERCENT
TOTAL WALLCLOCK TIME       44.4 CPU TIME      44.4 VECTOR TIME       44.4


===-=== END   OF TIMING STATISTICS ===-===
FORECAST DAYS PER DAY    32593.4
  *** END CNT0 *** 
------------------------------------------------
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMGGh7zg+001440
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMSHh7zg+001440
Moving to projects directory: /var/lib/boinc-client50002/slots/1/ICMUAh7zg+001440
Adding to the zip: /var/lib/boinc-client50002/slots/1/NODE.001_01
Adding to the zip: /var/lib/boinc-client50002/slots/1/ifs.stat
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMUAh7zg+001416
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMSHh7zg+001404
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMSHh7zg+001380
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMUAh7zg+001344
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMGGh7zg+001416
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMUAh7zg+001380
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMUAh7zg+001428
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMUAh7zg+001440
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMGGh7zg+001404
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMUAh7zg+001356
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMGGh7zg+001356
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMGGh7zg+001440
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMGGh7zg+001380
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMSHh7zg+001392
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMSHh7zg+001440
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMUAh7zg+001404
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMSHh7zg+001356
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMSHh7zg+001368
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMSHh7zg+001428
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMUAh7zg+001392
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMGGh7zg+001392
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMUAh7zg+001368
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMGGh7zg+001368
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMSHh7zg+001416
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMGGh7zg+001428
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMGGh7zg+001344
Adding to the zip: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_12209682/ICMSHh7zg+001344
Zipping up the final file: /var/lib/boinc-client50002/projects/climateprediction.net/oifs_43r3_bl_a28f_2016092300_15_991_12209682_0_r1681846707_14.zip
Uploading the final file: upload_file_14.zip
Uploading trickle at timestep: 1296900
07:30:14 (878372): called boinc_finish(0)

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:52:10 1537793 22309882 oifs_43r3_bl_a28f_2016092300_15_991_12209682_0 1,296,900 67,110 0.0517
09 May 2023 16:51:24 1537793 22309882 oifs_43r3_bl_a28f_2016092300_15_991_12209682_0 864,000 44,717 0.0518


©2024 climateprediction.net