climateprediction.net home page
Task 22311118

Task 22311118

Name oifs_43r3_bl_a2n0_2016092300_15_991_12210207_1
Workunit 12210207
Created 13 Feb 2023, 19:33:21 UTC
Sent 13 Feb 2023, 20:19:24 UTC
Report deadline 14 Apr 2023, 20:19:24 UTC
Received 14 Feb 2023, 16:58:36 UTC
Server state Over
Outcome Success
Client state Done
Exit status 0 (0x00000000)
Computer ID 1496882
Run time 17 hours 46 min 20 sec
CPU time 10 hours 47 min 27 sec
Validate state Valid
Credit 1,813.14
Device peak FLOPS 4.12 GFLOPS
Application version OpenIFS 43r3 Baroclinic Lifecycle v1.11
x86_64-pc-linux-gnu
Peak working set size 5,152.68 MB
Peak swap size 5,930.78 MB
Peak disk usage 2,131.86 MB
Stderr
<core_client_version>7.20.2</core_client_version>
<![CDATA[
<stderr_txt>
  940 H= 235:00 +CPU= 25.350
  00:46:02 STEP  941 H= 235:15 +CPU= 25.626
  00:46:45 STEP  942 H= 235:30 +CPU= 26.112
  00:47:33 STEP  943 H= 235:45 +CPU= 27.232
  00:48:14 STEP  944 H= 236:00 +CPU= 25.819
  00:48:48 STEP  945 H= 236:15 +CPU= 24.595
  00:49:28 STEP  946 H= 236:30 +CPU= 25.895
  00:50:14 STEP  947 H= 236:45 +CPU= 26.357
  00:51:02 STEP  948 H= 237:00 +CPU= 32.952
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+000948
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+000948
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+000948
  00:51:40 STEP  949 H= 237:15 +CPU= 25.986
  00:52:23 STEP  950 H= 237:30 +CPU= 27.579
  00:53:03 STEP  951 H= 237:45 +CPU= 26.151
  00:53:46 STEP  952 H= 238:00 +CPU= 27.834
  00:54:31 STEP  953 H= 238:15 +CPU= 27.013
  00:55:13 STEP  954 H= 238:30 +CPU= 26.391
  00:55:48 STEP  955 H= 238:45 +CPU= 29.852
  00:56:21 STEP  956 H= 239:00 +CPU= 30.108
  00:56:55 STEP  957 H= 239:15 +CPU= 30.497
  00:57:25 STEP  958 H= 239:30 +CPU= 28.192
  00:58:13 STEP  959 H= 239:45 +CPU= 27.167
  00:59:22 STEP  960 H= 240:00 +CPU= 35.719
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+000960
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+000960
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+000960
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMGGh7zg+000864
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMSHh7zg+000864
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMUAh7zg+000864
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMGGh7zg+000876
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMSHh7zg+000876
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMUAh7zg+000876
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMGGh7zg+000888
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMSHh7zg+000888
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMUAh7zg+000888
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMGGh7zg+000900
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMSHh7zg+000900
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMUAh7zg+000900
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMGGh7zg+000912
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMSHh7zg+000912
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMUAh7zg+000912
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMGGh7zg+000924
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMSHh7zg+000924
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMUAh7zg+000924
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMGGh7zg+000936
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMSHh7zg+000936
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMUAh7zg+000936
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMGGh7zg+000948
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMSHh7zg+000948
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMUAh7zg+000948
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_a2n0_2016092300_15_991_12210207_1_r1066345085_9.zip
Uploading the intermediate file: upload_file_9.zip
  00:59:58 STEP  961 H= 240:15 +CPU= 26.463
  01:00:39 STEP  962 H= 240:30 +CPU= 26.870
  01:01:31 STEP  963 H= 240:45 +CPU= 27.792
  01:02:17 STEP  964 H= 241:00 +CPU= 27.407
  01:03:01 STEP  965 H= 241:15 +CPU= 28.244
  01:03:33 STEP  966 H= 241:30 +CPU= 29.348
  01:04:04 STEP  967 H= 241:45 +CPU= 29.320
  01:04:38 STEP  968 H= 242:00 +CPU= 29.680
  01:05:08 STEP  969 H= 242:15 +CPU= 27.551
  01:05:49 STEP  970 H= 242:30 +CPU= 26.475
  01:06:25 STEP  971 H= 242:45 +CPU= 25.674
  01:07:38 STEP  972 H= 243:00 +CPU= 34.347
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+000972
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+000972
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+000972
  01:08:43 STEP  973 H= 243:15 +CPU= 27.809
  01:09:45 STEP  974 H= 243:30 +CPU= 28.111
  01:10:45 STEP  975 H= 243:45 +CPU= 30.596
  01:11:18 STEP  976 H= 244:00 +CPU= 30.356
  01:11:49 STEP  977 H= 244:15 +CPU= 29.324
  01:12:41 STEP  978 H= 244:30 +CPU= 28.450
  01:13:37 STEP  979 H= 244:45 +CPU= 26.972
  01:14:20 STEP  980 H= 245:00 +CPU= 25.944
  01:14:52 STEP  981 H= 245:15 +CPU= 25.333
  01:15:47 STEP  982 H= 245:30 +CPU= 26.992
  01:16:56 STEP  983 H= 245:45 +CPU= 28.693
  01:17:55 STEP  984 H= 246:00 +CPU= 34.991
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+000984
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+000984
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+000984
  01:18:27 STEP  985 H= 246:15 +CPU= 29.075
  01:18:58 STEP  986 H= 246:30 +CPU= 26.183
  01:19:38 STEP  987 H= 246:45 +CPU= 26.342
  01:20:49 STEP  988 H= 247:00 +CPU= 27.997
  01:21:30 STEP  989 H= 247:15 +CPU= 26.385
  01:22:06 STEP  990 H= 247:30 +CPU= 25.966
  01:22:44 STEP  991 H= 247:45 +CPU= 25.897
  01:23:27 STEP  992 H= 248:00 +CPU= 26.799
  01:24:04 STEP  993 H= 248:15 +CPU= 26.724
  01:24:36 STEP  994 H= 248:30 +CPU= 28.091
  01:25:10 STEP  995 H= 248:45 +CPU= 28.206
  01:25:53 STEP  996 H= 249:00 +CPU= 36.851
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+000996
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+000996
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+000996
  01:26:29 STEP  997 H= 249:15 +CPU= 26.148
  01:27:39 STEP  998 H= 249:30 +CPU= 27.625
  01:28:18 STEP  999 H= 249:45 +CPU= 25.845
  01:28:58 STEP 1000 H= 250:00 +CPU= 26.884
  01:29:33 STEP 1001 H= 250:15 +CPU= 25.850
  01:30:04 STEP 1002 H= 250:30 +CPU= 25.183
  01:30:40 STEP 1003 H= 250:45 +CPU= 26.270
  01:31:52 STEP 1004 H= 251:00 +CPU= 30.190
  01:32:49 STEP 1005 H= 251:15 +CPU= 30.309
  01:33:20 STEP 1006 H= 251:30 +CPU= 28.045
  01:34:00 STEP 1007 H= 251:45 +CPU= 26.117
  01:35:14 STEP 1008 H= 252:00 +CPU= 35.416
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+001008
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+001008
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+001008
  01:35:40 STEP 1009 H= 252:15 +CPU= 24.570
  01:36:30 STEP 1010 H= 252:30 +CPU= 26.027
  01:37:05 STEP 1011 H= 252:45 +CPU= 24.656
  01:37:44 STEP 1012 H= 253:00 +CPU= 26.869
  01:38:21 STEP 1013 H= 253:15 +CPU= 25.354
  01:38:59 STEP 1014 H= 253:30 +CPU= 27.020
  01:39:31 STEP 1015 H= 253:45 +CPU= 29.003
  01:40:02 STEP 1016 H= 254:00 +CPU= 29.020
  01:40:33 STEP 1017 H= 254:15 +CPU= 28.274
  01:41:10 STEP 1018 H= 254:30 +CPU= 27.103
  01:41:49 STEP 1019 H= 254:45 +CPU= 25.832
  01:42:33 STEP 1020 H= 255:00 +CPU= 31.237
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+001020
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+001020
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+001020
  01:43:34 STEP 1021 H= 255:15 +CPU= 27.576
  01:44:37 STEP 1022 H= 255:30 +CPU= 27.461
  01:45:30 STEP 1023 H= 255:45 +CPU= 26.907
  01:45:59 STEP 1024 H= 256:00 +CPU= 25.554
  01:46:37 STEP 1025 H= 256:15 +CPU= 27.726
  01:47:33 STEP 1026 H= 256:30 +CPU= 30.060
  01:48:16 STEP 1027 H= 256:45 +CPU= 29.355
  01:49:21 STEP 1028 H= 257:00 +CPU= 28.457
  01:50:14 STEP 1029 H= 257:15 +CPU= 26.657
  01:50:51 STEP 1030 H= 257:30 +CPU= 25.399
  01:51:32 STEP 1031 H= 257:45 +CPU= 26.303
  01:52:46 STEP 1032 H= 258:00 +CPU= 34.965
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+001032
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+001032
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+001032
  01:53:30 STEP 1033 H= 258:15 +CPU= 27.932
  01:54:02 STEP 1034 H= 258:30 +CPU= 29.075
  01:54:33 STEP 1035 H= 258:45 +CPU= 28.905
  01:55:14 STEP 1036 H= 259:00 +CPU= 27.093
  01:56:08 STEP 1037 H= 259:15 +CPU= 27.346
  01:56:49 STEP 1038 H= 259:30 +CPU= 26.368
  01:57:24 STEP 1039 H= 259:45 +CPU= 25.796
  01:58:04 STEP 1040 H= 260:00 +CPU= 26.440
  01:58:48 STEP 1041 H= 260:15 +CPU= 27.194
  01:59:24 STEP 1042 H= 260:30 +CPU= 25.555
  02:00:01 STEP 1043 H= 260:45 +CPU= 26.779
  02:00:57 STEP 1044 H= 261:00 +CPU= 38.028
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+001044
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+001044
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+001044
  02:01:48 STEP 1045 H= 261:15 +CPU= 30.678
  02:02:17 STEP 1046 H= 261:30 +CPU= 25.204
  02:02:43 STEP 1047 H= 261:45 +CPU= 24.270
  02:03:09 STEP 1048 H= 262:00 +CPU= 24.910
  02:03:42 STEP 1049 H= 262:15 +CPU= 24.671
  02:04:09 STEP 1050 H= 262:30 +CPU= 24.899
  02:04:38 STEP 1051 H= 262:45 +CPU= 25.779
  02:05:07 STEP 1052 H= 263:00 +CPU= 25.692
  02:05:32 STEP 1053 H= 263:15 +CPU= 24.544
  02:05:57 STEP 1054 H= 263:30 +CPU= 24.231
  02:06:36 STEP 1055 H= 263:45 +CPU= 25.640
  02:07:24 STEP 1056 H= 264:00 +CPU= 32.882
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+001056
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+001056
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+001056
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMGGh7zg+000960
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMSHh7zg+000960
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMUAh7zg+000960
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMGGh7zg+000972
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMSHh7zg+000972
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMUAh7zg+000972
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMGGh7zg+000984
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMSHh7zg+000984
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMUAh7zg+000984
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMGGh7zg+000996
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMSHh7zg+000996
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMUAh7zg+000996
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMGGh7zg+001008
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMSHh7zg+001008
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMUAh7zg+001008
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMGGh7zg+001020
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMSHh7zg+001020
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMUAh7zg+001020
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMGGh7zg+001032
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMSHh7zg+001032
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMUAh7zg+001032
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMGGh7zg+001044
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMSHh7zg+001044
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMUAh7zg+001044
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_a2n0_2016092300_15_991_12210207_1_r1066345085_10.zip
Uploading the intermediate file: upload_file_10.zip
  02:07:52 STEP 1057 H= 264:15 +CPU= 25.066
Uploading trickle at timestep: 950400
  02:08:25 STEP 1058 H= 264:30 +CPU= 25.228
  02:09:19 STEP 1059 H= 264:45 +CPU= 26.242
  02:10:10 STEP 1060 H= 265:00 +CPU= 27.904
  02:10:54 STEP 1061 H= 265:15 +CPU= 25.714
  02:11:20 STEP 1062 H= 265:30 +CPU= 24.195
  02:11:47 STEP 1063 H= 265:45 +CPU= 25.069
  02:12:13 STEP 1064 H= 266:00 +CPU= 24.992
  02:12:40 STEP 1065 H= 266:15 +CPU= 24.969
  02:13:16 STEP 1066 H= 266:30 +CPU= 25.678
  02:13:44 STEP 1067 H= 266:45 +CPU= 24.150
  02:14:28 STEP 1068 H= 267:00 +CPU= 31.418
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+001068
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+001068
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+001068
  02:14:54 STEP 1069 H= 267:15 +CPU= 24.335
  02:15:20 STEP 1070 H= 267:30 +CPU= 24.135
  02:15:46 STEP 1071 H= 267:45 +CPU= 24.552
  02:16:12 STEP 1072 H= 268:00 +CPU= 24.776
  02:16:37 STEP 1073 H= 268:15 +CPU= 24.171
  02:17:03 STEP 1074 H= 268:30 +CPU= 25.252
  02:17:30 STEP 1075 H= 268:45 +CPU= 24.512
  02:18:05 STEP 1076 H= 269:00 +CPU= 25.795
  02:18:32 STEP 1077 H= 269:15 +CPU= 24.744
  02:18:59 STEP 1078 H= 269:30 +CPU= 24.924
  02:19:26 STEP 1079 H= 269:45 +CPU= 24.701
  02:20:03 STEP 1080 H= 270:00 +CPU= 30.841
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+001080
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+001080
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+001080
  02:20:30 STEP 1081 H= 270:15 +CPU= 24.510
  02:20:57 STEP 1082 H= 270:30 +CPU= 24.578
  02:21:22 STEP 1083 H= 270:45 +CPU= 24.289
  02:21:50 STEP 1084 H= 271:00 +CPU= 25.514
  02:22:17 STEP 1085 H= 271:15 +CPU= 25.068
  02:22:42 STEP 1086 H= 271:30 +CPU= 23.745
  02:23:08 STEP 1087 H= 271:45 +CPU= 24.824
  02:23:35 STEP 1088 H= 272:00 +CPU= 25.169
  02:24:24 STEP 1089 H= 272:15 +CPU= 26.881
  02:25:12 STEP 1090 H= 272:30 +CPU= 25.808
  02:26:06 STEP 1091 H= 272:45 +CPU= 27.520
  02:26:45 STEP 1092 H= 273:00 +CPU= 32.241
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+001092
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+001092
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+001092
  02:27:10 STEP 1093 H= 273:15 +CPU= 24.308
  02:27:37 STEP 1094 H= 273:30 +CPU= 24.963
  02:28:05 STEP 1095 H= 273:45 +CPU= 24.548
  02:28:40 STEP 1096 H= 274:00 +CPU= 25.318
  02:29:05 STEP 1097 H= 274:15 +CPU= 24.088
  02:29:31 STEP 1098 H= 274:30 +CPU= 24.672
  02:29:59 STEP 1099 H= 274:45 +CPU= 25.154
  02:30:26 STEP 1100 H= 275:00 +CPU= 25.262
  02:30:53 STEP 1101 H= 275:15 +CPU= 24.235
  02:31:24 STEP 1102 H= 275:30 +CPU= 24.950
  02:31:50 STEP 1103 H= 275:45 +CPU= 24.288
  02:32:34 STEP 1104 H= 276:00 +CPU= 33.747
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+001104
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+001104
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+001104
  02:33:04 STEP 1105 H= 276:15 +CPU= 24.834
  02:33:31 STEP 1106 H= 276:30 +CPU= 25.124
  02:33:57 STEP 1107 H= 276:45 +CPU= 24.378
  02:34:49 STEP 1108 H= 277:00 +CPU= 27.028
  02:35:27 STEP 1109 H= 277:15 +CPU= 25.839
  02:35:54 STEP 1110 H= 277:30 +CPU= 24.494
  02:36:19 STEP 1111 H= 277:45 +CPU= 23.998
  02:36:47 STEP 1112 H= 278:00 +CPU= 25.181
  02:37:12 STEP 1113 H= 278:15 +CPU= 24.213
  02:37:38 STEP 1114 H= 278:30 +CPU= 24.792
  02:38:04 STEP 1115 H= 278:45 +CPU= 24.875
  02:38:37 STEP 1116 H= 279:00 +CPU= 30.802
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+001116
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+001116
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+001116
  02:39:04 STEP 1117 H= 279:15 +CPU= 24.906
  02:39:30 STEP 1118 H= 279:30 +CPU= 25.164
  02:39:59 STEP 1119 H= 279:45 +CPU= 25.291
  02:40:54 STEP 1120 H= 280:00 +CPU= 26.550
  02:41:37 STEP 1121 H= 280:15 +CPU= 25.900
  02:42:03 STEP 1122 H= 280:30 +CPU= 24.705
  02:42:29 STEP 1123 H= 280:45 +CPU= 24.550
  02:42:57 STEP 1124 H= 281:00 +CPU= 25.440
  02:43:23 STEP 1125 H= 281:15 +CPU= 24.637
  02:43:49 STEP 1126 H= 281:30 +CPU= 24.081
  02:44:14 STEP 1127 H= 281:45 +CPU= 24.221
  02:45:09 STEP 1128 H= 282:00 +CPU= 31.765
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+001128
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+001128
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+001128
  02:45:38 STEP 1129 H= 282:15 +CPU= 24.736
  02:46:03 STEP 1130 H= 282:30 +CPU= 24.270
  02:46:31 STEP 1131 H= 282:45 +CPU= 24.790
  02:47:00 STEP 1132 H= 283:00 +CPU= 25.920
  02:47:27 STEP 1133 H= 283:15 +CPU= 24.920
  02:47:53 STEP 1134 H= 283:30 +CPU= 24.986
  02:48:21 STEP 1135 H= 283:45 +CPU= 25.240
  02:48:48 STEP 1136 H= 284:00 +CPU= 25.487
  02:49:28 STEP 1137 H= 284:15 +CPU= 25.235
  02:49:59 STEP 1138 H= 284:30 +CPU= 25.391
  02:50:28 STEP 1139 H= 284:45 +CPU= 25.469
  02:51:15 STEP 1140 H= 285:00 +CPU= 32.963
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+001140
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+001140
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+001140
  02:51:43 STEP 1141 H= 285:15 +CPU= 25.124
  02:52:11 STEP 1142 H= 285:30 +CPU= 25.177
  02:52:39 STEP 1143 H= 285:45 +CPU= 25.260
  02:53:06 STEP 1144 H= 286:00 +CPU= 25.118
  02:53:32 STEP 1145 H= 286:15 +CPU= 25.156
  02:53:59 STEP 1146 H= 286:30 +CPU= 25.341
  02:54:25 STEP 1147 H= 286:45 +CPU= 24.608
  02:54:55 STEP 1148 H= 287:00 +CPU= 25.907
  02:55:22 STEP 1149 H= 287:15 +CPU= 24.766
  02:55:52 STEP 1150 H= 287:30 +CPU= 25.306
  02:56:36 STEP 1151 H= 287:45 +CPU= 26.450
  02:57:25 STEP 1152 H= 288:00 +CPU= 32.806
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+001152
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+001152
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+001152
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMGGh7zg+001056
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMSHh7zg+001056
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMUAh7zg+001056
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMGGh7zg+001068
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMSHh7zg+001068
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMUAh7zg+001068
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMGGh7zg+001080
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMSHh7zg+001080
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMUAh7zg+001080
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMGGh7zg+001092
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMSHh7zg+001092
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMUAh7zg+001092
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMGGh7zg+001104
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMSHh7zg+001104
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMUAh7zg+001104
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMGGh7zg+001116
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMSHh7zg+001116
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMUAh7zg+001116
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMGGh7zg+001128
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMSHh7zg+001128
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMUAh7zg+001128
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMGGh7zg+001140
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMSHh7zg+001140
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMUAh7zg+001140
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_a2n0_2016092300_15_991_12210207_1_r1066345085_11.zip
Uploading the intermediate file: upload_file_11.zip
  02:57:57 STEP 1153 H= 288:15 +CPU= 25.736
  02:58:27 STEP 1154 H= 288:30 +CPU= 24.245
  02:58:55 STEP 1155 H= 288:45 +CPU= 25.216
  02:59:22 STEP 1156 H= 289:00 +CPU= 25.812
  02:59:47 STEP 1157 H= 289:15 +CPU= 24.033
  03:00:16 STEP 1158 H= 289:30 +CPU= 25.544
  03:01:08 STEP 1159 H= 289:45 +CPU= 26.887
  03:01:46 STEP 1160 H= 290:00 +CPU= 26.222
  03:02:14 STEP 1161 H= 290:15 +CPU= 25.193
  03:03:01 STEP 1162 H= 290:30 +CPU= 26.417
  03:03:49 STEP 1163 H= 290:45 +CPU= 27.120
  03:04:43 STEP 1164 H= 291:00 +CPU= 32.395
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+001164
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+001164
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+001164
  03:05:29 STEP 1165 H= 291:15 +CPU= 25.685
  03:06:02 STEP 1166 H= 291:30 +CPU= 24.699
  03:06:47 STEP 1167 H= 291:45 +CPU= 26.714
  03:07:51 STEP 1168 H= 292:00 +CPU= 27.844
  03:08:41 STEP 1169 H= 292:15 +CPU= 26.407
  03:09:30 STEP 1170 H= 292:30 +CPU= 26.489
  03:10:21 STEP 1171 H= 292:45 +CPU= 26.735
  03:11:31 STEP 1172 H= 293:00 +CPU= 28.527
  03:12:43 STEP 1173 H= 293:15 +CPU= 28.212
  03:13:52 STEP 1174 H= 293:30 +CPU= 27.769
  03:15:00 STEP 1175 H= 293:45 +CPU= 27.366
  03:16:09 STEP 1176 H= 294:00 +CPU= 34.315
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+001176
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+001176
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+001176
  03:16:42 STEP 1177 H= 294:15 +CPU= 25.042
  03:17:39 STEP 1178 H= 294:30 +CPU= 27.424
  03:18:52 STEP 1179 H= 294:45 +CPU= 28.084
  03:19:33 STEP 1180 H= 295:00 +CPU= 26.487
  03:20:10 STEP 1181 H= 295:15 +CPU= 25.044
  03:20:57 STEP 1182 H= 295:30 +CPU= 26.153
  03:21:50 STEP 1183 H= 295:45 +CPU= 26.795
  03:23:35 STEP 1184 H= 296:00 +CPU= 28.584
  03:24:13 STEP 1185 H= 296:15 +CPU= 25.715
  03:24:50 STEP 1186 H= 296:30 +CPU= 25.522
  03:25:37 STEP 1187 H= 296:45 +CPU= 26.241
  03:26:30 STEP 1188 H= 297:00 +CPU= 32.549
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+001188
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+001188
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+001188
  03:27:26 STEP 1189 H= 297:15 +CPU= 27.094
  03:29:38 STEP 1190 H= 297:30 +CPU= 29.985
  03:30:26 STEP 1191 H= 297:45 +CPU= 27.503
  03:31:15 STEP 1192 H= 298:00 +CPU= 27.518
  03:32:00 STEP 1193 H= 298:15 +CPU= 26.794
  03:32:44 STEP 1194 H= 298:30 +CPU= 26.319
  03:33:22 STEP 1195 H= 298:45 +CPU= 25.823
  03:33:58 STEP 1196 H= 299:00 +CPU= 26.294
  03:34:41 STEP 1197 H= 299:15 +CPU= 26.295
  03:35:32 STEP 1198 H= 299:30 +CPU= 26.857
  03:36:18 STEP 1199 H= 299:45 +CPU= 26.535
  03:37:18 STEP 1200 H= 300:00 +CPU= 34.288
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+001200
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+001200
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+001200
  03:38:03 STEP 1201 H= 300:15 +CPU= 26.343
  03:38:53 STEP 1202 H= 300:30 +CPU= 26.632
  03:39:51 STEP 1203 H= 300:45 +CPU= 26.784
  03:40:57 STEP 1204 H= 301:00 +CPU= 28.274
  03:41:51 STEP 1205 H= 301:15 +CPU= 27.390
  03:42:39 STEP 1206 H= 301:30 +CPU= 26.498
  03:43:27 STEP 1207 H= 301:45 +CPU= 26.408
  03:44:33 STEP 1208 H= 302:00 +CPU= 28.821
  03:45:20 STEP 1209 H= 302:15 +CPU= 27.046
  03:45:59 STEP 1210 H= 302:30 +CPU= 25.237
  03:46:53 STEP 1211 H= 302:45 +CPU= 26.959
  03:47:33 STEP 1212 H= 303:00 +CPU= 30.130
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+001212
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+001212
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+001212
  03:48:08 STEP 1213 H= 303:15 +CPU= 25.431
  03:49:01 STEP 1214 H= 303:30 +CPU= 26.596
  03:49:36 STEP 1215 H= 303:45 +CPU= 24.739
  03:50:12 STEP 1216 H= 304:00 +CPU= 25.886
  03:50:59 STEP 1217 H= 304:15 +CPU= 25.845
  03:51:40 STEP 1218 H= 304:30 +CPU= 26.438
  03:52:22 STEP 1219 H= 304:45 +CPU= 26.048
  03:53:08 STEP 1220 H= 305:00 +CPU= 26.603
  03:54:15 STEP 1221 H= 305:15 +CPU= 28.016
  03:54:54 STEP 1222 H= 305:30 +CPU= 25.280
  03:55:42 STEP 1223 H= 305:45 +CPU= 26.159
  03:56:36 STEP 1224 H= 306:00 +CPU= 32.738
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+001224
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+001224
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+001224
  03:57:31 STEP 1225 H= 306:15 +CPU= 27.622
  03:58:13 STEP 1226 H= 306:30 +CPU= 26.699
  03:58:51 STEP 1227 H= 306:45 +CPU= 24.921
  03:59:31 STEP 1228 H= 307:00 +CPU= 26.609
  04:00:08 STEP 1229 H= 307:15 +CPU= 25.342
  04:00:44 STEP 1230 H= 307:30 +CPU= 25.556
  04:01:29 STEP 1231 H= 307:45 +CPU= 26.903
  04:02:19 STEP 1232 H= 308:00 +CPU= 26.763
  04:03:30 STEP 1233 H= 308:15 +CPU= 28.428
  04:04:22 STEP 1234 H= 308:30 +CPU= 27.568
  04:04:57 STEP 1235 H= 308:45 +CPU= 24.865
  04:05:59 STEP 1236 H= 309:00 +CPU= 34.122
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+001236
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+001236
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+001236
  04:06:39 STEP 1237 H= 309:15 +CPU= 26.106
  04:07:26 STEP 1238 H= 309:30 +CPU= 26.507
  04:08:10 STEP 1239 H= 309:45 +CPU= 25.822
  04:08:55 STEP 1240 H= 310:00 +CPU= 27.077
  04:09:36 STEP 1241 H= 310:15 +CPU= 26.189
  04:10:12 STEP 1242 H= 310:30 +CPU= 25.486
  04:10:54 STEP 1243 H= 310:45 +CPU= 26.340
  04:11:40 STEP 1244 H= 311:00 +CPU= 25.806
  04:12:09 STEP 1245 H= 311:15 +CPU= 24.712
  04:12:48 STEP 1246 H= 311:30 +CPU= 25.543
  04:13:41 STEP 1247 H= 311:45 +CPU= 26.981
  04:14:38 STEP 1248 H= 312:00 +CPU= 34.156
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+001248
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+001248
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+001248
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMGGh7zg+001152
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMSHh7zg+001152
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMUAh7zg+001152
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMGGh7zg+001164
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMSHh7zg+001164
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMUAh7zg+001164
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMGGh7zg+001176
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMSHh7zg+001176
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMUAh7zg+001176
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMGGh7zg+001188
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMSHh7zg+001188
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMUAh7zg+001188
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMGGh7zg+001200
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMSHh7zg+001200
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMUAh7zg+001200
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMGGh7zg+001212
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMSHh7zg+001212
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMUAh7zg+001212
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMGGh7zg+001224
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMSHh7zg+001224
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMUAh7zg+001224
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMGGh7zg+001236
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMSHh7zg+001236
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMUAh7zg+001236
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_a2n0_2016092300_15_991_12210207_1_r1066345085_12.zip
Uploading the intermediate file: upload_file_12.zip
  04:15:35 STEP 1249 H= 312:15 +CPU= 27.359
  04:16:23 STEP 1250 H= 312:30 +CPU= 26.652
  04:17:10 STEP 1251 H= 312:45 +CPU= 27.109
  04:17:41 STEP 1252 H= 313:00 +CPU= 24.924
  04:18:13 STEP 1253 H= 313:15 +CPU= 24.684
  04:18:57 STEP 1254 H= 313:30 +CPU= 26.414
  04:20:43 STEP 1255 H= 313:45 +CPU= 28.237
  04:21:52 STEP 1256 H= 314:00 +CPU= 28.762
  04:23:40 STEP 1257 H= 314:15 +CPU= 29.018
  04:24:34 STEP 1258 H= 314:30 +CPU= 26.831
  04:25:18 STEP 1259 H= 314:45 +CPU= 25.877
  04:26:07 STEP 1260 H= 315:00 +CPU= 31.826
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+001260
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+001260
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+001260
  04:26:56 STEP 1261 H= 315:15 +CPU= 27.537
  04:28:40 STEP 1262 H= 315:30 +CPU= 29.146
  04:29:34 STEP 1263 H= 315:45 +CPU= 28.024
  04:30:10 STEP 1264 H= 316:00 +CPU= 25.590
  04:30:59 STEP 1265 H= 316:15 +CPU= 26.851
  04:31:39 STEP 1266 H= 316:30 +CPU= 25.856
  04:32:15 STEP 1267 H= 316:45 +CPU= 25.886
  04:32:57 STEP 1268 H= 317:00 +CPU= 25.355
  04:33:33 STEP 1269 H= 317:15 +CPU= 25.655
  04:34:09 STEP 1270 H= 317:30 +CPU= 25.383
  04:35:10 STEP 1271 H= 317:45 +CPU= 27.949
  04:36:08 STEP 1272 H= 318:00 +CPU= 33.549
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+001272
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+001272
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+001272
  04:36:54 STEP 1273 H= 318:15 +CPU= 26.748
  04:37:38 STEP 1274 H= 318:30 +CPU= 26.403
  04:38:26 STEP 1275 H= 318:45 +CPU= 25.796
  04:40:17 STEP 1276 H= 319:00 +CPU= 29.503
  04:41:17 STEP 1277 H= 319:15 +CPU= 28.306
  04:41:58 STEP 1278 H= 319:30 +CPU= 26.394
  04:42:42 STEP 1279 H= 319:45 +CPU= 25.589
  04:43:28 STEP 1280 H= 320:00 +CPU= 27.267
  04:44:22 STEP 1281 H= 320:15 +CPU= 27.666
  04:45:04 STEP 1282 H= 320:30 +CPU= 26.373
  04:45:45 STEP 1283 H= 320:45 +CPU= 26.507
  04:46:46 STEP 1284 H= 321:00 +CPU= 32.868
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+001284
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+001284
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+001284
  04:47:33 STEP 1285 H= 321:15 +CPU= 25.929
  04:48:19 STEP 1286 H= 321:30 +CPU= 27.075
  04:49:06 STEP 1287 H= 321:45 +CPU= 26.539
  04:50:02 STEP 1288 H= 322:00 +CPU= 27.096
  04:50:46 STEP 1289 H= 322:15 +CPU= 26.478
  04:51:28 STEP 1290 H= 322:30 +CPU= 26.097
  04:52:16 STEP 1291 H= 322:45 +CPU= 26.266
  04:53:04 STEP 1292 H= 323:00 +CPU= 26.814
  04:53:51 STEP 1293 H= 323:15 +CPU= 26.695
  04:55:43 STEP 1294 H= 323:30 +CPU= 28.841
  04:56:45 STEP 1295 H= 323:45 +CPU= 27.811
  04:59:05 STEP 1296 H= 324:00 +CPU= 38.074
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+001296
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+001296
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+001296
  05:00:06 STEP 1297 H= 324:15 +CPU= 27.874
  05:01:57 STEP 1298 H= 324:30 +CPU= 28.700
  05:03:26 STEP 1299 H= 324:45 +CPU= 27.998
  05:04:11 STEP 1300 H= 325:00 +CPU= 26.142
  05:04:50 STEP 1301 H= 325:15 +CPU= 25.094
  05:06:02 STEP 1302 H= 325:30 +CPU= 27.157
  05:06:45 STEP 1303 H= 325:45 +CPU= 25.343
  05:07:22 STEP 1304 H= 326:00 +CPU= 25.334
  05:08:46 STEP 1305 H= 326:15 +CPU= 27.644
  05:10:38 STEP 1306 H= 326:30 +CPU= 29.097
  05:11:54 STEP 1307 H= 326:45 +CPU= 28.246
  05:12:54 STEP 1308 H= 327:00 +CPU= 33.340
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+001308
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+001308
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+001308
  05:13:54 STEP 1309 H= 327:15 +CPU= 29.191
  05:14:36 STEP 1310 H= 327:30 +CPU= 26.446
  05:15:33 STEP 1311 H= 327:45 +CPU= 26.889
  05:16:42 STEP 1312 H= 328:00 +CPU= 29.252
  05:17:47 STEP 1313 H= 328:15 +CPU= 28.795
  05:18:37 STEP 1314 H= 328:30 +CPU= 27.496
  05:19:18 STEP 1315 H= 328:45 +CPU= 25.924
  05:20:32 STEP 1316 H= 329:00 +CPU= 29.104
  05:21:41 STEP 1317 H= 329:15 +CPU= 28.032
  05:22:44 STEP 1318 H= 329:30 +CPU= 28.717
  05:24:29 STEP 1319 H= 329:45 +CPU= 29.924
  05:25:27 STEP 1320 H= 330:00 +CPU= 33.240
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+001320
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+001320
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+001320
  05:26:06 STEP 1321 H= 330:15 +CPU= 25.868
  05:26:53 STEP 1322 H= 330:30 +CPU= 26.718
  05:27:44 STEP 1323 H= 330:45 +CPU= 27.153
  05:28:24 STEP 1324 H= 331:00 +CPU= 26.921
  05:29:06 STEP 1325 H= 331:15 +CPU= 25.755
  05:29:49 STEP 1326 H= 331:30 +CPU= 26.481
  05:30:30 STEP 1327 H= 331:45 +CPU= 25.730
  05:31:20 STEP 1328 H= 332:00 +CPU= 28.108
  05:32:10 STEP 1329 H= 332:15 +CPU= 27.050
  05:34:10 STEP 1330 H= 332:30 +CPU= 29.428
  05:34:54 STEP 1331 H= 332:45 +CPU= 27.020
  05:35:52 STEP 1332 H= 333:00 +CPU= 33.497
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+001332
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+001332
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+001332
  05:36:40 STEP 1333 H= 333:15 +CPU= 26.848
  05:37:30 STEP 1334 H= 333:30 +CPU= 27.466
  05:38:18 STEP 1335 H= 333:45 +CPU= 27.294
  05:39:08 STEP 1336 H= 334:00 +CPU= 27.885
  05:40:00 STEP 1337 H= 334:15 +CPU= 26.764
  05:40:52 STEP 1338 H= 334:30 +CPU= 27.764
  05:41:43 STEP 1339 H= 334:45 +CPU= 27.098
  05:42:29 STEP 1340 H= 335:00 +CPU= 27.462
  05:43:23 STEP 1341 H= 335:15 +CPU= 26.804
  05:44:13 STEP 1342 H= 335:30 +CPU= 26.790
  05:45:18 STEP 1343 H= 335:45 +CPU= 26.842
  05:47:39 STEP 1344 H= 336:00 +CPU= 37.638
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+001344
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+001344
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+001344
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMGGh7zg+001248
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMSHh7zg+001248
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMUAh7zg+001248
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMGGh7zg+001260
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMSHh7zg+001260
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMUAh7zg+001260
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMGGh7zg+001272
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMSHh7zg+001272
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMUAh7zg+001272
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMGGh7zg+001284
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMSHh7zg+001284
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMUAh7zg+001284
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMGGh7zg+001296
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMSHh7zg+001296
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMUAh7zg+001296
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMGGh7zg+001308
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMSHh7zg+001308
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMUAh7zg+001308
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMGGh7zg+001320
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMSHh7zg+001320
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMUAh7zg+001320
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMGGh7zg+001332
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMSHh7zg+001332
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMUAh7zg+001332
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_a2n0_2016092300_15_991_12210207_1_r1066345085_13.zip
Uploading the intermediate file: upload_file_13.zip
  05:48:24 STEP 1345 H= 336:15 +CPU= 26.410
  05:49:26 STEP 1346 H= 336:30 +CPU= 26.266
  05:50:31 STEP 1347 H= 336:45 +CPU= 27.217
  05:51:09 STEP 1348 H= 337:00 +CPU= 26.868
  05:51:53 STEP 1349 H= 337:15 +CPU= 25.834
  05:53:11 STEP 1350 H= 337:30 +CPU= 27.660
  05:53:52 STEP 1351 H= 337:45 +CPU= 26.116
  05:54:27 STEP 1352 H= 338:00 +CPU= 25.725
  05:55:08 STEP 1353 H= 338:15 +CPU= 26.197
  05:56:18 STEP 1354 H= 338:30 +CPU= 28.599
  05:57:02 STEP 1355 H= 338:45 +CPU= 27.024
  05:58:03 STEP 1356 H= 339:00 +CPU= 33.432
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+001356
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+001356
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+001356
  05:58:42 STEP 1357 H= 339:15 +CPU= 25.452
  05:59:35 STEP 1358 H= 339:30 +CPU= 26.758
  06:00:29 STEP 1359 H= 339:45 +CPU= 27.134
  06:01:25 STEP 1360 H= 340:00 +CPU= 28.924
  06:02:05 STEP 1361 H= 340:15 +CPU= 26.381
  06:02:37 STEP 1362 H= 340:30 +CPU= 25.659
  06:03:27 STEP 1363 H= 340:45 +CPU= 27.091
  06:04:03 STEP 1364 H= 341:00 +CPU= 26.528
  06:04:37 STEP 1365 H= 341:15 +CPU= 25.480
  06:05:15 STEP 1366 H= 341:30 +CPU= 26.160
  06:06:05 STEP 1367 H= 341:45 +CPU= 27.877
  06:06:41 STEP 1368 H= 342:00 +CPU= 31.260
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+001368
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+001368
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+001368
  06:07:09 STEP 1369 H= 342:15 +CPU= 24.603
  06:07:43 STEP 1370 H= 342:30 +CPU= 24.293
  06:08:11 STEP 1371 H= 342:45 +CPU= 24.892
  06:08:40 STEP 1372 H= 343:00 +CPU= 25.636
  06:09:08 STEP 1373 H= 343:15 +CPU= 24.916
  06:09:35 STEP 1374 H= 343:30 +CPU= 25.018
  06:10:01 STEP 1375 H= 343:45 +CPU= 25.152
  06:10:28 STEP 1376 H= 344:00 +CPU= 24.851
  06:11:06 STEP 1377 H= 344:15 +CPU= 26.154
  06:11:57 STEP 1378 H= 344:30 +CPU= 26.607
  06:12:43 STEP 1379 H= 344:45 +CPU= 26.624
  06:13:17 STEP 1380 H= 345:00 +CPU= 30.815
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+001380
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+001380
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+001380
  06:13:43 STEP 1381 H= 345:15 +CPU= 24.735
  06:14:16 STEP 1382 H= 345:30 +CPU= 25.360
  06:14:42 STEP 1383 H= 345:45 +CPU= 24.574
  06:15:09 STEP 1384 H= 346:00 +CPU= 25.318
  06:15:34 STEP 1385 H= 346:15 +CPU= 24.540
  06:15:59 STEP 1386 H= 346:30 +CPU= 23.918
  06:16:28 STEP 1387 H= 346:45 +CPU= 25.639
  06:17:06 STEP 1388 H= 347:00 +CPU= 26.059
  06:17:46 STEP 1389 H= 347:15 +CPU= 25.109
  06:18:26 STEP 1390 H= 347:30 +CPU= 25.561
  06:19:01 STEP 1391 H= 347:45 +CPU= 25.459
  06:20:08 STEP 1392 H= 348:00 +CPU= 36.109
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+001392
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+001392
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+001392
  06:21:01 STEP 1393 H= 348:15 +CPU= 26.924
  06:21:44 STEP 1394 H= 348:30 +CPU= 26.416
  06:22:09 STEP 1395 H= 348:45 +CPU= 23.613
  06:23:23 STEP 1396 H= 349:00 +CPU= 27.142
  06:24:29 STEP 1397 H= 349:15 +CPU= 27.635
  06:25:23 STEP 1398 H= 349:30 +CPU= 27.086
  06:26:03 STEP 1399 H= 349:45 +CPU= 25.989
  06:26:49 STEP 1400 H= 350:00 +CPU= 26.710
  06:27:24 STEP 1401 H= 350:15 +CPU= 25.963
  06:28:04 STEP 1402 H= 350:30 +CPU= 26.020
  06:28:48 STEP 1403 H= 350:45 +CPU= 26.540
  06:30:01 STEP 1404 H= 351:00 +CPU= 34.234
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+001404
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+001404
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+001404
  06:30:53 STEP 1405 H= 351:15 +CPU= 27.272
  06:32:27 STEP 1406 H= 351:30 +CPU= 27.868
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.11_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a2n0
(argv4) batchid: 991
(argv5) wuid: 12210207
(argv6) fclen: 15
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/4
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.11
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/4/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/4/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_16993974079aaabec37a436229b46a41 to: /var/lib/boinc/slots/4/oifs_43r3_bl_a2n0_2016092300_15_991_12210207.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/4/oifs_43r3_bl_a2n0_2016092300_15_991_12210207.zip
ic_ancil_file: ic_ancil_12210207
ifsdata_file: ifsdata_12210207
climate_data_file: clim_data_12210207
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /var/lib/boinc/slots/4/ic_ancil_12210207.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/4/ic_ancil_12210207.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_ea3cbb653335e2688645464bfac38939 to: /var/lib/boinc/slots/4/ifsdata/ifsdata_12210207.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/4/ifsdata/ifsdata_12210207.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/4/159l_2/clim_data_12210207.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/4/159l_2/clim_data_12210207.zip
Checking for progress XML file: /var/lib/boinc/slots/4/progress_file_12210207.xml
Opened progress file ok : /var/lib/boinc/slots/4/progress_file_12210207.xml
-- Model is restarting --
Adjusting last_iter, 1407, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/4/progress_file_12210207.xml
last_cpu_time: 37536
upload_file_number: 14
last_iter: 1393
last_upload: 1209600
model_completed: 0
total_length_of_simulation: 1296000
result_base_name: oifs_43r3_bl_a2n0_2016092300_15_991_12210207_1_r1066345085
The child process has been launched with process id: 184570
Executing the command: /var/lib/boinc/slots/4/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:serv.home.saldino.net:1:1:184570:184570] [20230214:073328:1676388808:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2091] fp = 0x30e4a00
[EC_DRHOOK:serv.home.saldino.net:1:1:184570:184570] [20230214:073328:1676388808:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:serv.home.saldino.net:1:1:184570:184570] [20230214:073328:1676388808:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:serv.home.saldino.net:1:1:184570:184570] [20230214:073328:1676388808:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:serv.home.saldino.net:1:1:184570:184570] [20230214:073328:1676388808:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:serv.home.saldino.net:1:1:184570:184570] [20230214:073328:1676388808:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:serv.home.saldino.net:1:1:184570:184570] [20230214:073328:1676388808:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:serv.home.saldino.net:1:1:184570:184570] [20230214:073328:1676388808:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:serv.home.saldino.net:1:1:184570:184570] [20230214:073328:1676388808:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:serv.home.saldino.net:1:1:184570:184570] [20230214:073328:1676388808:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:serv.home.saldino.net:1:1:184570:184570] [20230214:073328:1676388808:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:serv.home.saldino.net:1:1:184570:184570] [20230214:073328:1676388808:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:serv.home.saldino.net:1:1:184570:184570] [20230214:073328:1676388808:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:serv.home.saldino.net:1:1:184570:184570] [20230214:073328:1676388808:0.000] [ignore_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:serv.home.saldino.net:1:1:184570:184570] [20230214:073328:1676388808:0.000] [restore_default_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:serv.home.saldino.net:1:1:184570:184570] [20230214:073328:1676388808:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:serv.home.saldino.net:1:1:184570:184570] [20230214:073328:1676388808:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f520 (old at (nil))
[EC_DRHOOK:serv.home.saldino.net:1:1:184570:184570] [20230214:073328:1676388808:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:serv.home.saldino.net:1:1:184570:184570] [20230214:073328:1676388808:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f520 (old at (nil))
[EC_DRHOOK:serv.home.saldino.net:1:1:184570:184570] [20230214:073328:1676388808:0.000] [trapfpe_treatment@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:serv.home.saldino.net:1:1:184570:184570] [20230214:073328:1676388808:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:serv.home.saldino.net:1:1:184570:184570] [20230214:073328:1676388808:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:serv.home.saldino.net:1:1:184570:184570] [20230214:073328:1676388808:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f520 (old at (nil))
[EC_DRHOOK:serv.home.saldino.net:1:1:184570:184570] [20230214:073328:1676388808:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:serv.home.saldino.net:1:1:184570:184570] [20230214:073328:1676388808:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:serv.home.saldino.net:1:1:184570:184570] [20230214:073328:1676388808:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:serv.home.saldino.net:1:1:184570:184570] [20230214:073328:1676388808:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f520 (old at (nil))
[EC_DRHOOK:serv.home.saldino.net:1:1:184570:184570] [20230214:073328:1676388808:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f520 (old at (nil))
[EC_DRHOOK:serv.home.saldino.net:1:1:184570:184570] [20230214:073328:1676388808:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f520 (old at (nil))
[EC_DRHOOK:serv.home.saldino.net:1:1:184570:184570] [20230214:073328:1676388808:0.000] [catch_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  07:34:51 STEP 1392 H= 348:00 +CPU= 28.760
  07:35:57 STEP 1393 H= 348:15 +CPU= 28.644
  07:36:40 STEP 1394 H= 348:30 +CPU= 26.859
  07:37:43 STEP 1395 H= 348:45 +CPU= 26.765
  07:38:25 STEP 1396 H= 349:00 +CPU= 26.591
  07:39:07 STEP 1397 H= 349:15 +CPU= 26.304
  07:39:36 STEP 1398 H= 349:30 +CPU= 24.474
  07:40:13 STEP 1399 H= 349:45 +CPU= 25.747
  07:40:56 STEP 1400 H= 350:00 +CPU= 26.415
  07:41:38 STEP 1401 H= 350:15 +CPU= 25.021
  07:42:08 STEP 1402 H= 350:30 +CPU= 24.621
  07:42:55 STEP 1403 H= 350:45 +CPU= 26.423
  07:43:44 STEP 1404 H= 351:00 +CPU= 30.460
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+001404
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+001404
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+001404
  07:44:08 STEP 1405 H= 351:15 +CPU= 23.819
  07:45:34 STEP 1406 H= 351:30 +CPU= 28.141
  07:46:11 STEP 1407 H= 351:45 +CPU= 24.464
  07:46:42 STEP 1408 H= 352:00 +CPU= 25.239
  07:47:53 STEP 1409 H= 352:15 +CPU= 26.912
  07:48:32 STEP 1410 H= 352:30 +CPU= 24.805
  07:49:02 STEP 1411 H= 352:45 +CPU= 24.246
  07:49:34 STEP 1412 H= 353:00 +CPU= 25.364
  07:50:11 STEP 1413 H= 353:15 +CPU= 25.818
  07:50:48 STEP 1414 H= 353:30 +CPU= 24.981
  07:51:24 STEP 1415 H= 353:45 +CPU= 25.243
  07:52:11 STEP 1416 H= 354:00 +CPU= 31.443
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+001416
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+001416
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+001416
  07:52:52 STEP 1417 H= 354:15 +CPU= 26.552
  07:53:19 STEP 1418 H= 354:30 +CPU= 23.632
  07:53:53 STEP 1419 H= 354:45 +CPU= 25.117
  07:54:40 STEP 1420 H= 355:00 +CPU= 27.018
  07:55:17 STEP 1421 H= 355:15 +CPU= 25.932
  07:55:49 STEP 1422 H= 355:30 +CPU= 24.644
  07:56:25 STEP 1423 H= 355:45 +CPU= 25.516
  07:57:26 STEP 1424 H= 356:00 +CPU= 27.222
  07:58:15 STEP 1425 H= 356:15 +CPU= 26.117
  07:59:22 STEP 1426 H= 356:30 +CPU= 27.740
  08:00:11 STEP 1427 H= 356:45 +CPU= 27.711
  08:01:04 STEP 1428 H= 357:00 +CPU= 32.235
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+001428
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+001428
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+001428
  08:02:04 STEP 1429 H= 357:15 +CPU= 28.154
  08:02:53 STEP 1430 H= 357:30 +CPU= 27.075
  08:03:47 STEP 1431 H= 357:45 +CPU= 27.100
  08:04:28 STEP 1432 H= 358:00 +CPU= 27.216
  08:05:17 STEP 1433 H= 358:15 +CPU= 26.629
  08:06:04 STEP 1434 H= 358:30 +CPU= 25.952
  08:06:50 STEP 1435 H= 358:45 +CPU= 27.096
  08:07:41 STEP 1436 H= 359:00 +CPU= 27.776
  08:08:48 STEP 1437 H= 359:15 +CPU= 27.942
  08:09:40 STEP 1438 H= 359:30 +CPU= 27.035
  08:10:18 STEP 1439 H= 359:45 +CPU= 25.701
  08:11:21 STEP 1440 H= 360:00 +CPU= 33.608
..The child process terminated with status: 0
>>> Printing last 70 lines from file: NODE.001_01

 
DDH------------------------------------------------------------------------
I                                                                         I
I TIME =  360.000H DDH * Diagnostic Var.,     AREA                GLOBAL  I
I                                                                         I
I-------------------------------------------------------------------------I
I   VARIABLE   I      Initial value         I        Current value        I
I-------------------------------------------------------------------------I
I  OMEGA (P/S) I     -0.7127425487E-21      I       0.2917441445E-06      I
ICloud fractionI      0.0000000000E+00      I       0.2080828170E-01      I
I  Relat. Hum. I      0.3357236957E+00      I       0.3745336585E+00      I
I  PASS 01     I      0.0000000000E+00      I       0.1331146334E+03      I
DDH------------------------------------------------------------------------
  
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1440 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001440 MODE=a
 IO-STREAM CLOSED - ICMGGh7zg+001440
 MPI-TASK:   1 -    320039921 BYTES IN      7 RECORDS TRANSFERRED IN    0.0007 SECONDS457199.9 Mbytes/s, TOTAL TIME=    0.1494( 0.5%)
 08:11:21 STEP 1440 H= 360:00 +CPU= 33.608
 END CNT3
 NSTEP =  1440 CNT0   000000000  25.734  25.734  44.450   0.579               0               0               0               0               0
IO-STREAM STATISTICS, TOTAL NO OF REC/BYTES READ           34/  1069125708 WRITTEN         1418/************
 -TASK-OPENED-OPEN-RECS IN -KBYTE IN    -RECS OUT-KBYTE OUT-WALL   -WALL IN-WALL
  OU-TOT IN -TOT OUT
   1     28    0       34      1044068     1418 ************     9.2     6.8     2.3     7.4    14.1
===-=== START OF TIMING STATISTICS ===-===
 
STATS FOR ALL TASKS
 NUM ROUTINE                                     CALLS  MEAN(ms)   MAX(ms)   FRAC(%)  UNBAL(%)
   0 CNT0     - COMPLETE EXECUTION                   1 2273002.6 2273002.6    100.00      0.00
   1 CNT4     - FORWARD INTEGRATION                  1 2264317.7 2264317.7     99.62      0.00
   8 SCAN2M - GRID-POINT DYNAMICS                   53    4607.7    4607.7     10.74      0.00
   9 SPCM     - SPECTRAL COMP.                      48     810.0     810.0      1.71      0.00
  10 SCAN2M - PHYSICS                               49   23328.8   23328.8     50.29      0.00
  11 IOPACK   - OUTPUT P.P. RESULTS                  4    2985.9    2985.9      0.53      0.00
  12 SPNORM   - SPECTRAL NORM COMP.                  3      42.9      42.9      0.01      0.00
  17 GRIDFPOS IN CNT4                                4     282.8     282.8      0.05      0.00
  27 RESTART FILES - WRITING                         1   11941.8   11941.8      0.53      0.00
  28 RESTART FILES - READING                         1    8126.2    8126.2      0.36      0.00
  29 SU4FPOS IN CNT4                                 4       8.7       8.7      0.00      0.00
  30 DYNFPOS IN CNT4                                 4    6018.9    6018.9      1.06      0.00
  31 POSDDH IN STEPO                                 1      28.4      28.4      0.00      0.00
  37 CPGLAG   - SL COMPUTATIONS                     49   12566.2   12566.2     27.09      0.00
  39 SU0YOMB                                         1     371.2     371.2      0.02      0.00
  51 SCAN2M   - SL COMM. PART 1                     49      47.0      47.0      0.10      0.00
  54 SPCM     - M TO S/S TO M TRANSP.               48     346.3     346.3      0.73      0.00
  55 SPCIMPF  - S TO M/M TO S TRANSP.               48      34.1      34.1      0.07      0.00
  56 SPNORM   - SPECTRAL NORM COMM.                  3       4.9       4.9      0.00      0.00
 102 LTINV_CTL   - INVERSE LEGENDRE TRANSFORM      108     329.8     329.8      1.57      0.00
 103 LTDIR_CTL   - DIRECT LEGENDRE TRANSFORM       100     397.9     397.9      1.75      0.00
 106 FTDIR_CTL   - DIRECT FOURIER TRANSFORM        100      96.7      96.7      0.43      0.00
 107 FTINV_CTL   - INVERSE FOURIER TRANSFORM       108     154.5     154.5      0.73      0.00
 140 SULEG       - COMP. OF LEGENDRE POL.            1      42.1      42.1      0.00      0.00
 152 LTINV_CTL   - M TO L TRANSPOSITION            108      35.2      35.2      0.17      0.00
 153 LTDIR_CTL   - L TO M TRANSPOSITION            100     200.4     200.4      0.88      0.00
 157 FTINV_CTL   - L TO G TRANSPOSITION            108     250.2     250.2      1.19      0.00
 158 FTDIR_CTL   - G TO L TRANSPOSITION            100     137.4     137.4      0.60      0.00
 400 GSTATS                                       6619       0.2       0.2      0.07      0.00
 401 GSTATS HOOK                                  6113       0.2       0.2      0.04      0.00
TOTAL MEASURED IMBALANCE =       0.0 SECONDS,  0.0 PERCENT
TOTAL WALLCLOCK TIME     2273.0 CPU TIME    1311.6 VECTOR TIME     1311.6


===-=== END   OF TIMING STATISTICS ===-===
FORECAST DAYS PER DAY      572.4
  *** END CNT0 *** 
------------------------------------------------
Moving to projects directory: /var/lib/boinc/slots/4/ICMGGh7zg+001440
Moving to projects directory: /var/lib/boinc/slots/4/ICMSHh7zg+001440
Moving to projects directory: /var/lib/boinc/slots/4/ICMUAh7zg+001440
Adding to the zip: /var/lib/boinc/slots/4/NODE.001_01
Adding to the zip: /var/lib/boinc/slots/4/ifs.stat
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMGGh7zg+001344
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMSHh7zg+001344
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMUAh7zg+001344
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMGGh7zg+001356
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMSHh7zg+001356
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMUAh7zg+001356
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMGGh7zg+001368
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMSHh7zg+001368
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMUAh7zg+001368
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMGGh7zg+001380
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMSHh7zg+001380
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMUAh7zg+001380
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMGGh7zg+001392
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMSHh7zg+001392
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMUAh7zg+001392
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMGGh7zg+001404
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMSHh7zg+001404
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMUAh7zg+001404
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMGGh7zg+001416
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMSHh7zg+001416
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMUAh7zg+001416
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMGGh7zg+001428
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMSHh7zg+001428
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMUAh7zg+001428
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMGGh7zg+001440
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMSHh7zg+001440
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12210207/ICMUAh7zg+001440
Zipping up the final file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_a2n0_2016092300_15_991_12210207_1_r1066345085_14.zip
Uploading the final file: upload_file_14.zip
Uploading trickle at timestep: 1295100
08:14:50 (184560): called boinc_finish(0)

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:51:19 1496882 22311118 oifs_43r3_bl_a2n0_2016092300_15_991_12210207_1 1,295,100 38,847 0.0300
09 May 2023 16:51:08 1496882 22311118 oifs_43r3_bl_a2n0_2016092300_15_991_12210207_1 950,400 28,161 0.0296


©2024 climateprediction.net