climateprediction.net home page
Task 22295889

Task 22295889

Name oifs_43r3_ps_0230_1995050100_123_964_12180874_1
Workunit 12180874
Created 5 Jan 2023, 21:31:05 UTC
Sent 5 Jan 2023, 22:01:04 UTC
Report deadline 4 Feb 2023, 22:01:04 UTC
Received 7 Jan 2023, 8:46:09 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1507457
Run time 8 hours 54 min 11 sec
CPU time 5 hours 20 min 51 sec
Validate state Invalid
Credit 6,652.80
Device peak FLOPS 5.98 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,746.47 MB
Peak swap size 4,978.08 MB
Peak disk usage 7,463.90 MB
Stderr
<core_client_version>7.16.16</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
:00 +CPU= 15.181
  18:50:50 STEP 1016 H=1016:00 +CPU= 15.779
  18:51:15 STEP 1017 H=1017:00 +CPU= 24.193
  18:51:31 STEP 1018 H=1018:00 +CPU= 15.658
  18:51:47 STEP 1019 H=1019:00 +CPU= 15.272
  18:52:15 STEP 1020 H=1020:00 +CPU= 26.397
Moving to projects directory: /media/nathan/512gb/boinc/slots/3/ICMGGhq0f+001020
Moving to projects directory: /media/nathan/512gb/boinc/slots/3/ICMSHhq0f+001020
Moving to projects directory: /media/nathan/512gb/boinc/slots/3/ICMUAhq0f+001020
  18:52:30 STEP 1021 H=1021:00 +CPU= 15.117
  18:52:46 STEP 1022 H=1022:00 +CPU= 15.605
  18:53:11 STEP 1023 H=1023:00 +CPU= 24.069
  18:53:27 STEP 1024 H=1024:00 +CPU= 15.643
  18:53:44 STEP 1025 H=1025:00 +CPU= 15.474
  18:54:11 STEP 1026 H=1026:00 +CPU= 25.069
  18:54:28 STEP 1027 H=1027:00 +CPU= 15.582
  18:54:44 STEP 1028 H=1028:00 +CPU= 15.939
  18:55:10 STEP 1029 H=1029:00 +CPU= 24.159
  18:55:26 STEP 1030 H=1030:00 +CPU= 15.543
  18:55:43 STEP 1031 H=1031:00 +CPU= 15.850
  18:56:13 STEP 1032 H=1032:00 +CPU= 28.187
Moving to projects directory: /media/nathan/512gb/boinc/slots/3/ICMGGhq0f+001032
Moving to projects directory: /media/nathan/512gb/boinc/slots/3/ICMSHhq0f+001032
Moving to projects directory: /media/nathan/512gb/boinc/slots/3/ICMUAhq0f+001032
Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12180874/ICMGGhq0f+001008
Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12180874/ICMSHhq0f+001008
Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12180874/ICMUAhq0f+001008
Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12180874/ICMGGhq0f+001020
Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12180874/ICMSHhq0f+001020
Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12180874/ICMUAhq0f+001020
Zipping up the intermediate file: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_0230_1995050100_123_964_12180874_1_r1714657139_28.zip
Uploading the intermediate file: upload_file_28.zip
  18:56:29 STEP 1033 H=1033:00 +CPU= 15.281
  18:56:46 STEP 1034 H=1034:00 +CPU= 15.849
  18:57:12 STEP 1035 H=1035:00 +CPU= 24.135
  18:57:29 STEP 1036 H=1036:00 +CPU= 15.755
  18:57:49 STEP 1037 H=1037:00 +CPU= 16.036
  18:58:16 STEP 1038 H=1038:00 +CPU= 24.827
  18:58:33 STEP 1039 H=1039:00 +CPU= 15.318
  18:58:49 STEP 1040 H=1040:00 +CPU= 15.708
  18:59:14 STEP 1041 H=1041:00 +CPU= 24.015
  18:59:31 STEP 1042 H=1042:00 +CPU= 15.786
  18:59:49 STEP 1043 H=1043:00 +CPU= 15.698
  19:00:16 STEP 1044 H=1044:00 +CPU= 26.518
Moving to projects directory: /media/nathan/512gb/boinc/slots/3/ICMGGhq0f+001044
Moving to projects directory: /media/nathan/512gb/boinc/slots/3/ICMSHhq0f+001044
Moving to projects directory: /media/nathan/512gb/boinc/slots/3/ICMUAhq0f+001044
  19:00:34 STEP 1045 H=1045:00 +CPU= 15.563
  19:00:52 STEP 1046 H=1046:00 +CPU= 16.182
  19:01:18 STEP 1047 H=1047:00 +CPU= 24.581
  19:01:35 STEP 1048 H=1048:00 +CPU= 15.947
  19:01:51 STEP 1049 H=1049:00 +CPU= 15.253
  19:02:17 STEP 1050 H=1050:00 +CPU= 24.616
  19:02:33 STEP 1051 H=1051:00 +CPU= 15.159
  19:02:49 STEP 1052 H=1052:00 +CPU= 15.796
  19:03:14 STEP 1053 H=1053:00 +CPU= 24.145
  19:03:32 STEP 1054 H=1054:00 +CPU= 15.970
  19:03:49 STEP 1055 H=1055:00 +CPU= 15.940
  19:04:20 STEP 1056 H=1056:00 +CPU= 28.211
Moving to projects directory: /media/nathan/512gb/boinc/slots/3/ICMGGhq0f+001056
Moving to projects directory: /media/nathan/512gb/boinc/slots/3/ICMSHhq0f+001056
Moving to projects directory: /media/nathan/512gb/boinc/slots/3/ICMUAhq0f+001056
Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12180874/ICMGGhq0f+001032
Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12180874/ICMSHhq0f+001032
Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12180874/ICMUAhq0f+001032
Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12180874/ICMGGhq0f+001044
Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12180874/ICMSHhq0f+001044
Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12180874/ICMUAhq0f+001044
Zipping up the intermediate file: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_0230_1995050100_123_964_12180874_1_r1714657139_29.zip
Uploading the intermediate file: upload_file_29.zip
  19:04:36 STEP 1057 H=1057:00 +CPU= 15.491
  19:04:53 STEP 1058 H=1058:00 +CPU= 15.735
  19:05:18 STEP 1059 H=1059:00 +CPU= 24.107
  19:05:35 STEP 1060 H=1060:00 +CPU= 15.876
  19:05:52 STEP 1061 H=1061:00 +CPU= 15.499
  19:06:18 STEP 1062 H=1062:00 +CPU= 24.757
  19:06:35 STEP 1063 H=1063:00 +CPU= 15.645
  19:06:53 STEP 1064 H=1064:00 +CPU= 16.072
  19:07:20 STEP 1065 H=1065:00 +CPU= 24.751
  19:07:38 STEP 1066 H=1066:00 +CPU= 15.934
  19:07:54 STEP 1067 H=1067:00 +CPU= 15.434
  19:08:22 STEP 1068 H=1068:00 +CPU= 26.578
Moving to projects directory: /media/nathan/512gb/boinc/slots/3/ICMGGhq0f+001068
Moving to projects directory: /media/nathan/512gb/boinc/slots/3/ICMSHhq0f+001068
Moving to projects directory: /media/nathan/512gb/boinc/slots/3/ICMUAhq0f+001068
  19:08:37 STEP 1069 H=1069:00 +CPU= 15.233
  19:08:54 STEP 1070 H=1070:00 +CPU= 15.747
  19:09:18 STEP 1071 H=1071:00 +CPU= 23.954
  19:09:35 STEP 1072 H=1072:00 +CPU= 15.886
  19:09:52 STEP 1073 H=1073:00 +CPU= 15.766
  19:10:18 STEP 1074 H=1074:00 +CPU= 24.630
  19:10:34 STEP 1075 H=1075:00 +CPU= 15.404
  19:10:51 STEP 1076 H=1076:00 +CPU= 15.899
  19:11:17 STEP 1077 H=1077:00 +CPU= 24.515
  19:11:35 STEP 1078 H=1078:00 +CPU= 16.186
  19:11:52 STEP 1079 H=1079:00 +CPU= 15.741
  19:12:21 STEP 1080 H=1080:00 +CPU= 27.472
Moving to projects directory: /media/nathan/512gb/boinc/slots/3/ICMGGhq0f+001080
Moving to projects directory: /media/nathan/512gb/boinc/slots/3/ICMSHhq0f+001080
Moving to projects directory: /media/nathan/512gb/boinc/slots/3/ICMUAhq0f+001080
Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12180874/ICMGGhq0f+001056
Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12180874/ICMSHhq0f+001056
Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12180874/ICMUAhq0f+001056
Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12180874/ICMGGhq0f+001068
Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12180874/ICMSHhq0f+001068
Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12180874/ICMUAhq0f+001068
Zipping up the intermediate file: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_0230_1995050100_123_964_12180874_1_r1714657139_30.zip
Uploading the intermediate file: upload_file_30.zip
  19:12:36 STEP 1081 H=1081:00 +CPU= 15.173
Uploading trickle at timestep: 3888000
  19:12:53 STEP 1082 H=1082:00 +CPU= 15.627
  19:13:17 STEP 1083 H=1083:00 +CPU= 23.838
  19:13:33 STEP 1084 H=1084:00 +CPU= 15.614
  19:13:49 STEP 1085 H=1085:00 +CPU= 15.213
  19:14:15 STEP 1086 H=1086:00 +CPU= 24.447
  19:14:31 STEP 1087 H=1087:00 +CPU= 15.270
  19:14:47 STEP 1088 H=1088:00 +CPU= 15.626
  19:15:14 STEP 1089 H=1089:00 +CPU= 24.337
  19:15:30 STEP 1090 H=1090:00 +CPU= 15.757
  19:15:47 STEP 1091 H=1091:00 +CPU= 15.436
  19:16:14 STEP 1092 H=1092:00 +CPU= 26.110
Moving to projects directory: /media/nathan/512gb/boinc/slots/3/ICMGGhq0f+001092
Moving to projects directory: /media/nathan/512gb/boinc/slots/3/ICMSHhq0f+001092
Moving to projects directory: /media/nathan/512gb/boinc/slots/3/ICMUAhq0f+001092
  19:16:29 STEP 1093 H=1093:00 +CPU= 15.040
  19:16:45 STEP 1094 H=1094:00 +CPU= 15.564
  19:17:11 STEP 1095 H=1095:00 +CPU= 24.146
  19:17:28 STEP 1096 H=1096:00 +CPU= 15.968
  19:17:46 STEP 1097 H=1097:00 +CPU= 15.832
  19:18:13 STEP 1098 H=1098:00 +CPU= 24.935
  19:18:30 STEP 1099 H=1099:00 +CPU= 15.693
  19:18:46 STEP 1100 H=1100:00 +CPU= 15.821
  19:19:11 STEP 1101 H=1101:00 +CPU= 24.102
  19:19:30 STEP 1102 H=1102:00 +CPU= 16.063
  19:19:47 STEP 1103 H=1103:00 +CPU= 15.927
  19:20:16 STEP 1104 H=1104:00 +CPU= 27.404
Moving to projects directory: /media/nathan/512gb/boinc/slots/3/ICMGGhq0f+001104
Moving to projects directory: /media/nathan/512gb/boinc/slots/3/ICMSHhq0f+001104
Moving to projects directory: /media/nathan/512gb/boinc/slots/3/ICMUAhq0f+001104
Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12180874/ICMGGhq0f+001080
Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12180874/ICMSHhq0f+001080
Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12180874/ICMUAhq0f+001080
Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12180874/ICMGGhq0f+001092
Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12180874/ICMSHhq0f+001092
Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12180874/ICMUAhq0f+001092
Zipping up the intermediate file: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_0230_1995050100_123_964_12180874_1_r1714657139_31.zip
Uploading the intermediate file: upload_file_31.zip
  19:20:32 STEP 1105 H=1105:00 +CPU= 15.316
  19:20:48 STEP 1106 H=1106:00 +CPU= 15.624
  19:21:13 STEP 1107 H=1107:00 +CPU= 23.924
  19:21:30 STEP 1108 H=1108:00 +CPU= 15.704
  19:21:46 STEP 1109 H=1109:00 +CPU= 15.185
  19:22:11 STEP 1110 H=1110:00 +CPU= 24.379
  19:22:27 STEP 1111 H=1111:00 +CPU= 15.143
  19:22:43 STEP 1112 H=1112:00 +CPU= 15.682
  19:23:07 STEP 1113 H=1113:00 +CPU= 23.726
  19:23:24 STEP 1114 H=1114:00 +CPU= 15.588
  19:23:39 STEP 1115 H=1115:00 +CPU= 15.131
  19:24:06 STEP 1116 H=1116:00 +CPU= 26.213
Moving to projects directory: /media/nathan/512gb/boinc/slots/3/ICMGGhq0f+001116
Moving to projects directory: /media/nathan/512gb/boinc/slots/3/ICMSHhq0f+001116
Moving to projects directory: /media/nathan/512gb/boinc/slots/3/ICMUAhq0f+001116
  19:24:22 STEP 1117 H=1117:00 +CPU= 15.200
  19:24:38 STEP 1118 H=1118:00 +CPU= 15.745
  19:25:03 STEP 1119 H=1119:00 +CPU= 23.909
  19:25:19 STEP 1120 H=1120:00 +CPU= 15.497
  19:25:36 STEP 1121 H=1121:00 +CPU= 15.335
  19:26:02 STEP 1122 H=1122:00 +CPU= 24.562
  19:26:17 STEP 1123 H=1123:00 +CPU= 15.035
  19:26:33 STEP 1124 H=1124:00 +CPU= 15.511
  19:26:57 STEP 1125 H=1125:00 +CPU= 23.859
  19:27:13 STEP 1126 H=1126:00 +CPU= 15.473
  19:27:29 STEP 1127 H=1127:00 +CPU= 15.471
  19:27:57 STEP 1128 H=1128:00 +CPU= 27.348
Moving to projects directory: /media/nathan/512gb/boinc/slots/3/ICMGGhq0f+001128
Moving to projects directory: /media/nathan/512gb/boinc/slots/3/ICMSHhq0f+001128
Moving to projects directory: /media/nathan/512gb/boinc/slots/3/ICMUAhq0f+001128
Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12180874/ICMGGhq0f+001104
Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12180874/ICMSHhq0f+001104
Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12180874/ICMUAhq0f+001104
Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12180874/ICMGGhq0f+001116
Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12180874/ICMSHhq0f+001116
Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12180874/ICMUAhq0f+001116
Zipping up the intermediate file: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_0230_1995050100_123_964_12180874_1_r1714657139_32.zip
Uploading the intermediate file: upload_file_32.zip
  19:28:12 STEP 1129 H=1129:00 +CPU= 15.033
  19:28:28 STEP 1130 H=1130:00 +CPU= 15.482
  19:28:52 STEP 1131 H=1131:00 +CPU= 23.765
  19:29:08 STEP 1132 H=1132:00 +CPU= 15.522
  19:29:23 STEP 1133 H=1133:00 +CPU= 15.052
  19:29:48 STEP 1134 H=1134:00 +CPU= 24.269
  19:30:03 STEP 1135 H=1135:00 +CPU= 15.062
  19:30:19 STEP 1136 H=1136:00 +CPU= 15.533
  19:30:44 STEP 1137 H=1137:00 +CPU= 23.763
  19:30:59 STEP 1138 H=1138:00 +CPU= 15.522
  19:31:15 STEP 1139 H=1139:00 +CPU= 15.069
  19:31:42 STEP 1140 H=1140:00 +CPU= 26.138
Moving to projects directory: /media/nathan/512gb/boinc/slots/3/ICMGGhq0f+001140
Moving to projects directory: /media/nathan/512gb/boinc/slots/3/ICMSHhq0f+001140
Moving to projects directory: /media/nathan/512gb/boinc/slots/3/ICMUAhq0f+001140
  19:31:57 STEP 1141 H=1141:00 +CPU= 14.989
  19:32:13 STEP 1142 H=1142:00 +CPU= 15.501
  19:32:37 STEP 1143 H=1143:00 +CPU= 23.702
  19:32:53 STEP 1144 H=1144:00 +CPU= 15.470
  19:33:09 STEP 1145 H=1145:00 +CPU= 15.209
  19:33:34 STEP 1146 H=1146:00 +CPU= 24.468
  19:33:50 STEP 1147 H=1147:00 +CPU= 15.184
  19:34:06 STEP 1148 H=1148:00 +CPU= 15.528
  19:34:30 STEP 1149 H=1149:00 +CPU= 23.577
  19:34:46 STEP 1150 H=1150:00 +CPU= 15.423
  19:35:02 STEP 1151 H=1151:00 +CPU= 15.436
  19:35:30 STEP 1152 H=1152:00 +CPU= 27.191
Moving to projects directory: /media/nathan/512gb/boinc/slots/3/ICMGGhq0f+001152
Moving to projects directory: /media/nathan/512gb/boinc/slots/3/ICMSHhq0f+001152
Moving to projects directory: /media/nathan/512gb/boinc/slots/3/ICMUAhq0f+001152
Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12180874/ICMGGhq0f+001128
Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12180874/ICMSHhq0f+001128
Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12180874/ICMUAhq0f+001128
Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12180874/ICMGGhq0f+001140
Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12180874/ICMSHhq0f+001140
Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12180874/ICMUAhq0f+001140
Zipping up the intermediate file: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_0230_1995050100_123_964_12180874_1_r1714657139_33.zip
Uploading the intermediate file: upload_file_33.zip
  19:35:45 STEP 1153 H=1153:00 +CPU= 14.981
  19:36:01 STEP 1154 H=1154:00 +CPU= 15.404
  19:36:25 STEP 1155 H=1155:00 +CPU= 23.566
  19:36:41 STEP 1156 H=1156:00 +CPU= 15.404
  19:36:57 STEP 1157 H=1157:00 +CPU= 14.953
  19:37:21 STEP 1158 H=1158:00 +CPU= 24.075
  19:37:37 STEP 1159 H=1159:00 +CPU= 14.986
  19:37:52 STEP 1160 H=1160:00 +CPU= 15.428
  19:38:17 STEP 1161 H=1161:00 +CPU= 23.581
  19:38:33 STEP 1162 H=1162:00 +CPU= 15.454
  19:38:48 STEP 1163 H=1163:00 +CPU= 15.013
  19:39:15 STEP 1164 H=1164:00 +CPU= 25.932
Moving to projects directory: /media/nathan/512gb/boinc/slots/3/ICMGGhq0f+001164
Moving to projects directory: /media/nathan/512gb/boinc/slots/3/ICMSHhq0f+001164
Moving to projects directory: /media/nathan/512gb/boinc/slots/3/ICMUAhq0f+001164
  19:39:30 STEP 1165 H=1165:00 +CPU= 14.981
  19:39:46 STEP 1166 H=1166:00 +CPU= 15.478
  19:40:10 STEP 1167 H=1167:00 +CPU= 23.568
  19:40:26 STEP 1168 H=1168:00 +CPU= 15.456
  19:40:41 STEP 1169 H=1169:00 +CPU= 15.012
  19:41:06 STEP 1170 H=1170:00 +CPU= 24.101
  19:41:21 STEP 1171 H=1171:00 +CPU= 14.963
  19:41:37 STEP 1172 H=1172:00 +CPU= 15.435
  19:42:02 STEP 1173 H=1173:00 +CPU= 23.627
  19:42:17 STEP 1174 H=1174:00 +CPU= 15.409
  19:42:33 STEP 1175 H=1175:00 +CPU= 15.384
  19:43:01 STEP 1176 H=1176:00 +CPU= 27.179
Moving to projects directory: /media/nathan/512gb/boinc/slots/3/ICMGGhq0f+001176
Moving to projects directory: /media/nathan/512gb/boinc/slots/3/ICMSHhq0f+001176
Moving to projects directory: /media/nathan/512gb/boinc/slots/3/ICMUAhq0f+001176
Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12180874/ICMGGhq0f+001152
Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12180874/ICMSHhq0f+001152
Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12180874/ICMUAhq0f+001152
Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12180874/ICMGGhq0f+001164
Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12180874/ICMSHhq0f+001164
Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12180874/ICMUAhq0f+001164
Zipping up the intermediate file: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_0230_1995050100_123_964_12180874_1_r1714657139_34.zip
Uploading the intermediate file: upload_file_34.zip
  19:43:16 STEP 1177 H=1177:00 +CPU= 14.923
  19:43:32 STEP 1178 H=1178:00 +CPU= 15.354
  19:43:56 STEP 1179 H=1179:00 +CPU= 23.540
  19:44:12 STEP 1180 H=1180:00 +CPU= 15.372
  19:44:27 STEP 1181 H=1181:00 +CPU= 14.941
  19:44:52 STEP 1182 H=1182:00 +CPU= 24.119
  19:45:07 STEP 1183 H=1183:00 +CPU= 14.968
  19:45:23 STEP 1184 H=1184:00 +CPU= 15.405
  19:45:47 STEP 1185 H=1185:00 +CPU= 23.579
  19:46:03 STEP 1186 H=1186:00 +CPU= 15.426
  19:46:19 STEP 1187 H=1187:00 +CPU= 14.988
  19:46:45 STEP 1188 H=1188:00 +CPU= 26.088
Moving to projects directory: /media/nathan/512gb/boinc/slots/3/ICMGGhq0f+001188
Moving to projects directory: /media/nathan/512gb/boinc/slots/3/ICMSHhq0f+001188
Moving to projects directory: /media/nathan/512gb/boinc/slots/3/ICMUAhq0f+001188
  19:47:01 STEP 1189 H=1189:00 +CPU= 14.972
  19:47:17 STEP 1190 H=1190:00 +CPU= 15.509
  19:47:41 STEP 1191 H=1191:00 +CPU= 23.519
  19:47:57 STEP 1192 H=1192:00 +CPU= 15.409
  19:48:12 STEP 1193 H=1193:00 +CPU= 14.936
  19:48:37 STEP 1194 H=1194:00 +CPU= 24.057
  19:48:52 STEP 1195 H=1195:00 +CPU= 14.938
  19:49:08 STEP 1196 H=1196:00 +CPU= 15.413
  19:49:32 STEP 1197 H=1197:00 +CPU= 23.453
  19:49:48 STEP 1198 H=1198:00 +CPU= 15.378
  19:50:03 STEP 1199 H=1199:00 +CPU= 15.335
  19:50:31 STEP 1200 H=1200:00 +CPU= 27.266
Moving to projects directory: /media/nathan/512gb/boinc/slots/3/ICMGGhq0f+001200
Moving to projects directory: /media/nathan/512gb/boinc/slots/3/ICMSHhq0f+001200
Moving to projects directory: /media/nathan/512gb/boinc/slots/3/ICMUAhq0f+001200
Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12180874/ICMGGhq0f+001176
Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12180874/ICMSHhq0f+001176
Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12180874/ICMUAhq0f+001176
Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12180874/ICMGGhq0f+001188
Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12180874/ICMSHhq0f+001188
Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12180874/ICMUAhq0f+001188
Zipping up the intermediate file: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_0230_1995050100_123_964_12180874_1_r1714657139_35.zip
Uploading the intermediate file: upload_file_35.zip
  19:50:47 STEP 1201 H=1201:00 +CPU= 14.919
  19:51:02 STEP 1202 H=1202:00 +CPU= 15.376
  19:51:26 STEP 1203 H=1203:00 +CPU= 23.487
  19:51:42 STEP 1204 H=1204:00 +CPU= 15.334
  19:51:57 STEP 1205 H=1205:00 +CPU= 14.917
  19:52:22 STEP 1206 H=1206:00 +CPU= 23.999
  19:52:37 STEP 1207 H=1207:00 +CPU= 14.921
  19:52:53 STEP 1208 H=1208:00 +CPU= 15.377
  19:53:17 STEP 1209 H=1209:00 +CPU= 23.521
  19:53:33 STEP 1210 H=1210:00 +CPU= 15.378
  19:53:48 STEP 1211 H=1211:00 +CPU= 14.941
  19:54:15 STEP 1212 H=1212:00 +CPU= 26.065
Moving to projects directory: /media/nathan/512gb/boinc/slots/3/ICMGGhq0f+001212
Moving to projects directory: /media/nathan/512gb/boinc/slots/3/ICMSHhq0f+001212
Moving to projects directory: /media/nathan/512gb/boinc/slots/3/ICMUAhq0f+001212
  19:54:30 STEP 1213 H=1213:00 +CPU= 14.994
  19:54:46 STEP 1214 H=1214:00 +CPU= 15.532
  19:55:10 STEP 1215 H=1215:00 +CPU= 23.613
  19:55:26 STEP 1216 H=1216:00 +CPU= 15.446
  19:55:42 STEP 1217 H=1217:00 +CPU= 14.989
  19:56:06 STEP 1218 H=1218:00 +CPU= 24.138
  19:56:22 STEP 1219 H=1219:00 +CPU= 14.970
  19:56:38 STEP 1220 H=1220:00 +CPU= 15.468
  19:57:02 STEP 1221 H=1221:00 +CPU= 23.587
  19:57:18 STEP 1222 H=1222:00 +CPU= 15.420
  19:57:33 STEP 1223 H=1223:00 +CPU= 15.392
  19:58:02 STEP 1224 H=1224:00 +CPU= 27.399
Moving to projects directory: /media/nathan/512gb/boinc/slots/3/ICMGGhq0f+001224
Moving to projects directory: /media/nathan/512gb/boinc/slots/3/ICMSHhq0f+001224
Moving to projects directory: /media/nathan/512gb/boinc/slots/3/ICMUAhq0f+001224
Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12180874/ICMGGhq0f+001200
Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12180874/ICMSHhq0f+001200
Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12180874/ICMUAhq0f+001200
Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12180874/ICMGGhq0f+001212
Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12180874/ICMSHhq0f+001212
Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12180874/ICMUAhq0f+001212
Zipping up the intermediate file: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_0230_1995050100_123_964_12180874_1_r1714657139_36.zip
Uploading the intermediate file: upload_file_36.zip
  19:58:17 STEP 1225 H=1225:00 +CPU= 14.964
  19:58:33 STEP 1226 H=1226:00 +CPU= 15.463
  19:58:58 STEP 1227 H=1227:00 +CPU= 23.685
  19:59:14 STEP 1228 H=1228:00 +CPU= 15.492
  19:59:29 STEP 1229 H=1229:00 +CPU= 14.949
  19:59:54 STEP 1230 H=1230:00 +CPU= 24.124
  20:00:09 STEP 1231 H=1231:00 +CPU= 14.993
  20:00:25 STEP 1232 H=1232:00 +CPU= 15.448
  20:00:50 STEP 1233 H=1233:00 +CPU= 23.679
  20:01:06 STEP 1234 H=1234:00 +CPU= 15.394
  20:01:21 STEP 1235 H=1235:00 +CPU= 15.042
  20:01:48 STEP 1236 H=1236:00 +CPU= 26.162
Moving to projects directory: /media/nathan/512gb/boinc/slots/3/ICMGGhq0f+001236
Moving to projects directory: /media/nathan/512gb/boinc/slots/3/ICMSHhq0f+001236
Moving to projects directory: /media/nathan/512gb/boinc/slots/3/ICMUAhq0f+001236
  20:02:04 STEP 1237 H=1237:00 +CPU= 15.036
  20:02:20 STEP 1238 H=1238:00 +CPU= 15.523
  20:02:44 STEP 1239 H=1239:00 +CPU= 23.575
  20:03:00 STEP 1240 H=1240:00 +CPU= 15.489
  20:03:15 STEP 1241 H=1241:00 +CPU= 14.981
  20:03:40 STEP 1242 H=1242:00 +CPU= 24.110
  20:03:55 STEP 1243 H=1243:00 +CPU= 14.948
  20:04:11 STEP 1244 H=1244:00 +CPU= 15.441
  20:04:35 STEP 1245 H=1245:00 +CPU= 23.540
  20:04:51 STEP 1246 H=1246:00 +CPU= 15.440
  20:05:07 STEP 1247 H=1247:00 +CPU= 15.392
  20:05:35 STEP 1248 H=1248:00 +CPU= 27.317
Moving to projects directory: /media/nathan/512gb/boinc/slots/3/ICMGGhq0f+001248
Moving to projects directory: /media/nathan/512gb/boinc/slots/3/ICMSHhq0f+001248
Moving to projects directory: /media/nathan/512gb/boinc/slots/3/ICMUAhq0f+001248
Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12180874/ICMGGhq0f+001224
Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12180874/ICMSHhq0f+001224
Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12180874/ICMUAhq0f+001224
Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12180874/ICMGGhq0f+001236
Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12180874/ICMSHhq0f+001236
Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12180874/ICMUAhq0f+001236
Zipping up the intermediate file: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_0230_1995050100_123_964_12180874_1_r1714657139_37.zip
Uploading the intermediate file: upload_file_37.zip
  20:05:50 STEP 1249 H=1249:00 +CPU= 14.901
  20:06:06 STEP 1250 H=1250:00 +CPU= 15.324
  20:06:30 STEP 1251 H=1251:00 +CPU= 23.446
  20:06:46 STEP 1252 H=1252:00 +CPU= 15.342
  20:07:01 STEP 1253 H=1253:00 +CPU= 14.898
  20:07:25 STEP 1254 H=1254:00 +CPU= 23.944
  20:07:41 STEP 1255 H=1255:00 +CPU= 14.919
  20:07:57 STEP 1256 H=1256:00 +CPU= 15.380
  20:08:21 STEP 1257 H=1257:00 +CPU= 23.402
  20:08:36 STEP 1258 H=1258:00 +CPU= 15.374
  20:08:52 STEP 1259 H=1259:00 +CPU= 14.908
  20:09:18 STEP 1260 H=1260:00 +CPU= 26.026
Moving to projects directory: /media/nathan/512gb/boinc/slots/3/ICMGGhq0f+001260
Moving to projects directory: /media/nathan/512gb/boinc/slots/3/ICMSHhq0f+001260
Moving to projects directory: /media/nathan/512gb/boinc/slots/3/ICMUAhq0f+001260
  20:09:34 STEP 1261 H=1261:00 +CPU= 15.023
  20:09:50 STEP 1262 H=1262:00 +CPU= 15.529
  20:10:14 STEP 1263 H=1263:00 +CPU= 23.640
  20:10:30 STEP 1264 H=1264:00 +CPU= 15.470
  20:10:45 STEP 1265 H=1265:00 +CPU= 15.014
  20:11:10 STEP 1266 H=1266:00 +CPU= 24.203
  20:11:25 STEP 1267 H=1267:00 +CPU= 14.979
  20:11:41 STEP 1268 H=1268:00 +CPU= 15.488
  20:12:05 STEP 1269 H=1269:00 +CPU= 23.565
  20:12:21 STEP 1270 H=1270:00 +CPU= 15.436
  20:12:37 STEP 1271 H=1271:00 +CPU= 15.405
  20:13:05 STEP 1272 H=1272:00 +CPU= 27.362
Moving to projects directory: /media/nathan/512gb/boinc/slots/3/ICMGGhq0f+001272
Moving to projects directory: /media/nathan/512gb/boinc/slots/3/ICMSHhq0f+001272
Moving to projects directory: /media/nathan/512gb/boinc/slots/3/ICMUAhq0f+001272
Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12180874/ICMGGhq0f+001248
Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12180874/ICMSHhq0f+001248
Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12180874/ICMUAhq0f+001248
Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12180874/ICMGGhq0f+001260
Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12180874/ICMSHhq0f+001260
Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12180874/ICMUAhq0f+001260
Zipping up the intermediate file: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_0230_1995050100_123_964_12180874_1_r1714657139_38.zip
Uploading the intermediate file: upload_file_38.zip
  20:13:20 STEP 1273 H=1273:00 +CPU= 14.938
  20:13:36 STEP 1274 H=1274:00 +CPU= 15.352
  20:14:00 STEP 1275 H=1275:00 +CPU= 23.514
  20:14:16 STEP 1276 H=1276:00 +CPU= 15.366
  20:14:31 STEP 1277 H=1277:00 +CPU= 14.937
  20:14:56 STEP 1278 H=1278:00 +CPU= 24.035
  20:15:11 STEP 1279 H=1279:00 +CPU= 14.963
  20:15:27 STEP 1280 H=1280:00 +CPU= 15.426
  20:15:51 STEP 1281 H=1281:00 +CPU= 23.545
  20:16:07 STEP 1282 H=1282:00 +CPU= 15.405
  20:16:22 STEP 1283 H=1283:00 +CPU= 14.973
  20:16:49 STEP 1284 H=1284:00 +CPU= 26.072
Moving to projects directory: /media/nathan/512gb/boinc/slots/3/ICMGGhq0f+001284
Moving to projects directory: /media/nathan/512gb/boinc/slots/3/ICMSHhq0f+001284
Moving to projects directory: /media/nathan/512gb/boinc/slots/3/ICMUAhq0f+001284
  20:17:05 STEP 1285 H=1285:00 +CPU= 15.031
  20:17:21 STEP 1286 H=1286:00 +CPU= 15.530
  20:17:45 STEP 1287 H=1287:00 +CPU= 23.596
  20:18:01 STEP 1288 H=1288:00 +CPU= 15.475
  20:18:16 STEP 1289 H=1289:00 +CPU= 14.961
  20:18:41 STEP 1290 H=1290:00 +CPU= 24.163
  20:18:56 STEP 1291 H=1291:00 +CPU= 14.976
  20:19:12 STEP 1292 H=1292:00 +CPU= 15.492
  20:19:36 STEP 1293 H=1293:00 +CPU= 23.577
  20:19:51 STEP 1294 H=1294:00 +CPU= 15.437
  20:20:07 STEP 1295 H=1295:00 +CPU= 15.387
  20:20:35 STEP 1296 H=1296:00 +CPU= 27.332
Moving to projects directory: /media/nathan/512gb/boinc/slots/3/ICMGGhq0f+001296
Moving to projects directory: /media/nathan/512gb/boinc/slots/3/ICMSHhq0f+001296
Moving to projects directory: /media/nathan/512gb/boinc/slots/3/ICMUAhq0f+001296
Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12180874/ICMGGhq0f+001272
Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12180874/ICMSHhq0f+001272
Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12180874/ICMUAhq0f+001272
Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12180874/ICMGGhq0f+001284
Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12180874/ICMSHhq0f+001284
Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12180874/ICMUAhq0f+001284
Zipping up the intermediate file: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_0230_1995050100_123_964_12180874_1_r1714657139_39.zip
Uploading the intermediate file: upload_file_39.zip
  20:20:51 STEP 1297 H=1297:00 +CPU= 14.931
  20:21:06 STEP 1298 H=1298:00 +CPU= 15.366
  20:21:31 STEP 1299 H=1299:00 +CPU= 23.500
  20:21:46 STEP 1300 H=1300:00 +CPU= 15.345
  20:22:02 STEP 1301 H=1301:00 +CPU= 14.917
  20:22:26 STEP 1302 H=1302:00 +CPU= 24.018
  20:22:42 STEP 1303 H=1303:00 +CPU= 14.934
  20:22:57 STEP 1304 H=1304:00 +CPU= 15.371
  20:23:21 STEP 1305 H=1305:00 +CPU= 23.559
  20:23:37 STEP 1306 H=1306:00 +CPU= 15.378
  20:23:53 STEP 1307 H=1307:00 +CPU= 14.946
  20:24:19 STEP 1308 H=1308:00 +CPU= 26.111
Moving to projects directory: /media/nathan/512gb/boinc/slots/3/ICMGGhq0f+001308
Moving to projects directory: /media/nathan/512gb/boinc/slots/3/ICMSHhq0f+001308
Moving to projects directory: /media/nathan/512gb/boinc/slots/3/ICMUAhq0f+001308
  20:24:35 STEP 1309 H=1309:00 +CPU= 15.027
  20:24:51 STEP 1310 H=1310:00 +CPU= 15.525
  20:25:15 STEP 1311 H=1311:00 +CPU= 23.634
  20:25:31 STEP 1312 H=1312:00 +CPU= 15.482
  20:25:46 STEP 1313 H=1313:00 +CPU= 15.000
  20:26:11 STEP 1314 H=1314:00 +CPU= 24.188
  20:26:26 STEP 1315 H=1315:00 +CPU= 14.956
  20:26:42 STEP 1316 H=1316:00 +CPU= 15.464
  20:27:06 STEP 1317 H=1317:00 +CPU= 23.564
  20:27:22 STEP 1318 H=1318:00 +CPU= 15.436
  20:27:38 STEP 1319 H=1319:00 +CPU= 15.463
  20:28:09 STEP 1320 H=1320:00 +CPU= 28.076
Moving to projects directory: /media/nathan/512gb/boinc/slots/3/ICMGGhq0f+001320
Moving to projects directory: /media/nathan/512gb/boinc/slots/3/ICMSHhq0f+001320
Moving to projects directory: /media/nathan/512gb/boinc/slots/3/ICMUAhq0f+001320
Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12180874/ICMGGhq0f+001296
Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12180874/ICMSHhq0f+001296
Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12180874/ICMUAhq0f+001296
Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12180874/ICMGGhq0f+001308
Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12180874/ICMSHhq0f+001308
Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12180874/ICMUAhq0f+001308
Zipping up the intermediate file: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_0230_1995050100_123_964_12180874_1_r1714657139_40.zip
Uploading the intermediate file: upload_file_40.zip
  20:28:27 STEP 1321 H=1321:00 +CPU= 15.764
Uploading trickle at timestep: 4752000
  20:28:47 STEP 1322 H=1322:00 +CPU= 16.024
  20:29:11 STEP 1323 H=1323:00 +CPU= 23.787
  20:29:27 STEP 1324 H=1324:00 +CPU= 15.525
  20:29:43 STEP 1325 H=1325:00 +CPU= 15.149
  20:30:10 STEP 1326 H=1326:00 +CPU= 24.625
  20:30:26 STEP 1327 H=1327:00 +CPU= 15.214
  20:30:47 STEP 1328 H=1328:00 +CPU= 16.007
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  20:31:42 STEP 1329 H=1329:00 +CPU= 25.504
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  20:32:18 STEP 1330 H=1330:00 +CPU= 16.222
  20:32:41 STEP 1331 H=1331:00 +CPU= 15.885
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  20:35:11 STEP 1332 H=1332:00 +CPU= 28.367
Moving to projects directory: /media/nathan/512gb/boinc/slots/3/ICMGGhq0f+001332
Moving to projects directory: /media/nathan/512gb/boinc/slots/3/ICMSHhq0f+001332
Moving to projects directory: /media/nathan/512gb/boinc/slots/3/ICMUAhq0f+001332
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  20:44:30 STEP 1333 H=1333:00 +CPU= 16.435
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  20:52:04 STEP 1334 H=1334:00 +CPU= 17.004
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:05:50 STEP 1335 H=1335:00 +CPU= 26.169
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:06:27 STEP 1336 H=1336:00 +CPU= 17.261
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:14:41 STEP 1337 H=1337:00 +CPU= 17.017
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:33:55 STEP 1338 H=1338:00 +CPU= 26.778
  21:34:14 STEP 1339 H=1339:00 +CPU= 16.145
  21:34:30 STEP 1340 H=1340:00 +CPU= 16.034
[EC_DRHOOK:debian-desktop:1:1:32773:32773] [20230106:213437:1673037277:22986.410] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 4035MB (heap), 4780MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:debian-desktop:1:1:32773:32773] [20230106:213437:1673037277:22986.534] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:debian-desktop:1:1:32773:32773] [20230106:213437:1673037277:22986.534] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f3c0 (old at (nil))
[EC_DRHOOK:debian-desktop:1:1:32773:32773] [20230106:213437:1673037277:22986.534] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1
[EC_DRHOOK:debian-desktop:1:1:32773:32773] [20230106:213437:1673037277:22986.534] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1
[EC_DRHOOK:debian-desktop:1:1:32773:32773] [20230106:213437:1673037277:22986.534] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 4035 MB (maxheap), 4780 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:debian-desktop:1:1:32773:32773] [20230106:213437:1673037277:22986.534] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:debian-desktop:1:1:32773:32773] [20230106:213437:1673037277:22986.569] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:debian-desktop:1:1:32773:32773] [20230106:213437:1673037277:22986.569] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:debian-desktop:1:1:32773:32773] [20230106:213437:1673037277:22986.569] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:debian-desktop:1:1:32773:32773] [20230106:213437:1673037277:22986.569] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:debian-desktop:1:1:32773:32773] [20230106:213437:1673037277:22986.569] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :       CNT4 
[EC_DRHOOK:debian-desktop:1:1:32773:32773] [20230106:213437:1673037277:22986.569] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :        STEPO 
[EC_DRHOOK:debian-desktop:1:1:32773:32773] [20230106:213437:1673037277:22986.569] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :         SCAN2M 
[EC_DRHOOK:debian-desktop:1:1:32773:32773] [20230106:213437:1673037277:22986.569] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :          GP_MODEL_HEAP 
[EC_DRHOOK:debian-desktop:1:1:32773:32773] [20230106:213437:1673037277:22986.569] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :           GP_MODEL 
[EC_DRHOOK:debian-desktop:1:1:32773:32773] [20230106:213437:1673037277:22986.569] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :            RADDRV 
[EC_DRHOOK:debian-desktop:1:1:32773:32773] [20230106:213437:1673037277:22986.569] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :             RADINTG 
[EC_DRHOOK:debian-desktop:1:1:32773:32773] [20230106:213437:1673037277:22986.569] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :              >OMP-RADINTG-RADLSW       (1210) 
[EC_DRHOOK:debian-desktop:1:1:32773:32773] [20230106:213437:1673037277:22986.569] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :               RADIATION_SCHEME 
[EC_DRHOOK:debian-desktop:1:1:32773:32773] [20230106:213437:1673037277:22986.569] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                radiation_interface:radiation 
[EC_DRHOOK:debian-desktop:1:1:32773:32773] [20230106:213437:1673037277:22986.569] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                 radiation_mcica_sw:solver_mcica_sw 
[EC_DRHOOK:debian-desktop:1:1:32773:32773] [20230106:213438:1673037278:22987.579] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1
[EC_DRHOOK:debian-desktop:1:1:32773:32773] [20230106:213438:1673037278:22987.579] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1ce8bb0 for signal#15, nsigs = 1
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source             
oifs_43r3_model.e  0000000001CE916B  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000081FFF1  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DC9090  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000196E9C8  radiation_mcica_s          28  radiation_delta_eddington.h
oifs_43r3_model.e  000000000196A3D5  radiation_mcica_s         250  radiation_mcica_sw.F90
oifs_43r3_model.e  0000000001862DC8  radiation_interfa         419  radiation_interface.F90
oifs_43r3_model.e  00000000012D30E5  radiation_scheme_         484  radiation_scheme.F90
oifs_43r3_model.e  000000000129A415  radintg_                 1455  radintg.F90
oifs_43r3_model.e  0000000001270124  raddrv_                   537  raddrv.F90
oifs_43r3_model.e  0000000000BFD839  gp_model_                 482  gp_model.F90
oifs_43r3_model.e  00000000012E9482  gp_model_heap_             74  gp_model_heap.F90
oifs_43r3_model.e  0000000000BBC1CE  scan2m_                   535  scan2m.F90
oifs_43r3_model.e  000000000057A664  stepo_                    327  stepo.F90
oifs_43r3_model.e  000000000055DEEC  cnt4_                    1133  cnt4.F90
oifs_43r3_model.e  00000000005471C9  cnt3_                     267  cnt3.F90
oifs_43r3_model.e  0000000000546412  cnt2_                      88  cnt2.F90
oifs_43r3_model.e  0000000000545E28  cnt1_                      92  cnt1.F90
oifs_43r3_model.e  000000000040708D  cnt0_                     146  cnt0.F90
oifs_43r3_model.e  000000000040220F  MAIN__                     96  master.F90
oifs_43r3_model.e  00000000004021A2  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCA390  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000040206E  Unknown               Unknown  Unknown
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1995050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0230
(argv4) batchid: 964
(argv5) wuid: 12180874
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /media/nathan/512gb/boinc/slots/3
Project directory is: /media/nathan/512gb/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12180874
..mkdir for temp folder for results failed
Copying: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /media/nathan/512gb/boinc/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /media/nathan/512gb/boinc/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_c1fe5c7944b44b9ffa51338ef2b1c704 to: /media/nathan/512gb/boinc/slots/3/oifs_43r3_ps_0230_1995050100_123_964_12180874.zip
Unzipping the namelist zip file: /media/nathan/512gb/boinc/slots/3/oifs_43r3_ps_0230_1995050100_123_964_12180874.zip
ic_ancil_file: ic_ancil_12180874
ifsdata_file: ifsdata_12180874
climate_data_file: clim_data_12180874
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_c001ae804a5d16c2869ee30c0f0c3f84 to: /media/nathan/512gb/boinc/slots/3/ic_ancil_12180874.zip
Unzipping the IC ancils zip file: /media/nathan/512gb/boinc/slots/3/ic_ancil_12180874.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_26fbb80c7d24b9dcd7971d71b500d37a to: /media/nathan/512gb/boinc/slots/3/ifsdata/ifsdata_12180874.zip
Unzipping the ifsdata_zip file: /media/nathan/512gb/boinc/slots/3/ifsdata/ifsdata_12180874.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /media/nathan/512gb/boinc/slots/3/159l_2/clim_data_12180874.zip
Unzipping the climate data zip file: /media/nathan/512gb/boinc/slots/3/159l_2/clim_data_12180874.zip
Checking for progress XML file: /media/nathan/512gb/boinc/slots/3/progress_file_12180874.xml
Opened progress file ok : /media/nathan/512gb/boinc/slots/3/progress_file_12180874.xml
-- Model is restarting --
Adjusting last_iter, 1341, to previous model restart step.
Creating progress file: /media/nathan/512gb/boinc/slots/3/progress_file_12180874.xml
last_cpu_time: 18449
upload_file_number: 41
last_iter: 1319
last_upload: 4752000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0230_1995050100_123_964_12180874_1_r1714657139
The child process has been launched with process id: 3258
Executing the command: /media/nathan/512gb/boinc/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:debian-desktop:1:1:3258:3258] [20220807:152605:1659878765:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:debian-desktop:1:1:3258:3258] [20220807:152605:1659878765:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:debian-desktop:1:1:3258:3258] [20220807:152605:1659878765:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:debian-desktop:1:1:3258:3258] [20220807:152605:1659878765:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:debian-desktop:1:1:3258:3258] [20220807:152605:1659878765:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:debian-desktop:1:1:3258:3258] [20220807:152605:1659878765:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:debian-desktop:1:1:3258:3258] [20220807:152605:1659878765:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:debian-desktop:1:1:3258:3258] [20220807:152605:1659878765:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:debian-desktop:1:1:3258:3258] [20220807:152605:1659878765:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:debian-desktop:1:1:3258:3258] [20220807:152605:1659878765:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:debian-desktop:1:1:3258:3258] [20220807:152605:1659878765:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:debian-desktop:1:1:3258:3258] [20220807:152605:1659878765:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:debian-desktop:1:1:3258:3258] [20220807:152605:1659878765:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:debian-desktop:1:1:3258:3258] [20220807:152605:1659878765:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:debian-desktop:1:1:3258:3258] [20220807:152605:1659878765:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:debian-desktop:1:1:3258:3258] [20220807:152605:1659878765:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:debian-desktop:1:1:3258:3258] [20220807:152605:1659878765:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:debian-desktop:1:1:3258:3258] [20220807:152605:1659878765:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:debian-desktop:1:1:3258:3258] [20220807:152605:1659878765:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:debian-desktop:1:1:3258:3258] [20220807:152605:1659878765:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:debian-desktop:1:1:3258:3258] [20220807:152605:1659878765:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:debian-desktop:1:1:3258:3258] [20220807:152605:1659878765:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:debian-desktop:1:1:3258:3258] [20220807:152605:1659878765:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:debian-desktop:1:1:3258:3258] [20220807:152605:1659878765:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:debian-desktop:1:1:3258:3258] [20220807:152605:1659878765:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:debian-desktop:1:1:3258:3258] [20220807:152605:1659878765:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:debian-desktop:1:1:3258:3258] [20220807:152605:1659878765:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:debian-desktop:1:1:3258:3258] [20220807:152605:1659878765:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:debian-desktop:1:1:3258:3258] [20220807:152605:1659878765:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:debian-desktop:1:1:3258:3258] [20220807:152605:1659878765:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  08:42:01 STEP 1320 H=1320:00 +CPU= 26.906
  08:42:17 STEP 1321 H=1321:00 +CPU= 15.186
  08:42:32 STEP 1322 H=1322:00 +CPU= 15.570
  08:42:57 STEP 1323 H=1323:00 +CPU= 23.961
  08:43:13 STEP 1324 H=1324:00 +CPU= 15.627
  08:43:28 STEP 1325 H=1325:00 +CPU= 14.934
  08:43:53 STEP 1326 H=1326:00 +CPU= 24.133
  08:44:08 STEP 1327 H=1327:00 +CPU= 15.164
  08:44:24 STEP 1328 H=1328:00 +CPU= 15.383
  08:44:48 STEP 1329 H=1329:00 +CPU= 23.589
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1995050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0230
(argv4) batchid: 964
(argv5) wuid: 12180874
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /media/nathan/512gb/boinc/slots/3
Project directory is: /media/nathan/512gb/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12180874
..mkdir for temp folder for results failed
Copying: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /media/nathan/512gb/boinc/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /media/nathan/512gb/boinc/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_c1fe5c7944b44b9ffa51338ef2b1c704 to: /media/nathan/512gb/boinc/slots/3/oifs_43r3_ps_0230_1995050100_123_964_12180874.zip
Unzipping the namelist zip file: /media/nathan/512gb/boinc/slots/3/oifs_43r3_ps_0230_1995050100_123_964_12180874.zip
ic_ancil_file: ic_ancil_12180874
ifsdata_file: ifsdata_12180874
climate_data_file: clim_data_12180874
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_c001ae804a5d16c2869ee30c0f0c3f84 to: /media/nathan/512gb/boinc/slots/3/ic_ancil_12180874.zip
Unzipping the IC ancils zip file: /media/nathan/512gb/boinc/slots/3/ic_ancil_12180874.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_26fbb80c7d24b9dcd7971d71b500d37a to: /media/nathan/512gb/boinc/slots/3/ifsdata/ifsdata_12180874.zip
Unzipping the ifsdata_zip file: /media/nathan/512gb/boinc/slots/3/ifsdata/ifsdata_12180874.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /media/nathan/512gb/boinc/slots/3/159l_2/clim_data_12180874.zip
Unzipping the climate data zip file: /media/nathan/512gb/boinc/slots/3/159l_2/clim_data_12180874.zip
Checking for progress XML file: /media/nathan/512gb/boinc/slots/3/progress_file_12180874.xml
Opened progress file ok : /media/nathan/512gb/boinc/slots/3/progress_file_12180874.xml
-- Model is restarting --
Adjusting last_iter, 1330, to previous model restart step.
Creating progress file: /media/nathan/512gb/boinc/slots/3/progress_file_12180874.xml
last_cpu_time: 18641
upload_file_number: 41
last_iter: 1319
last_upload: 4752000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0230_1995050100_123_964_12180874_1_r1714657139
The child process has been launched with process id: 4814
Executing the command: /media/nathan/512gb/boinc/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:debian-desktop:1:1:4814:4814] [20230107:084600:1673077560:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:debian-desktop:1:1:4814:4814] [20230107:084600:1673077560:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:debian-desktop:1:1:4814:4814] [20230107:084600:1673077560:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:debian-desktop:1:1:4814:4814] [20230107:084600:1673077560:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:debian-desktop:1:1:4814:4814] [20230107:084600:1673077560:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:debian-desktop:1:1:4814:4814] [20230107:084600:1673077560:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:debian-desktop:1:1:4814:4814] [20230107:084600:1673077560:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:debian-desktop:1:1:4814:4814] [20230107:084600:1673077560:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:debian-desktop:1:1:4814:4814] [20230107:084600:1673077560:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:debian-desktop:1:1:4814:4814] [20230107:084600:1673077560:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:debian-desktop:1:1:4814:4814] [20230107:084600:1673077560:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:debian-desktop:1:1:4814:4814] [20230107:084600:1673077560:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:debian-desktop:1:1:4814:4814] [20230107:084600:1673077560:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:debian-desktop:1:1:4814:4814] [20230107:084600:1673077560:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:debian-desktop:1:1:4814:4814] [20230107:084600:1673077560:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:debian-desktop:1:1:4814:4814] [20230107:084600:1673077560:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:debian-desktop:1:1:4814:4814] [20230107:084600:1673077560:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:debian-desktop:1:1:4814:4814] [20230107:084600:1673077560:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:debian-desktop:1:1:4814:4814] [20230107:084600:1673077560:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:debian-desktop:1:1:4814:4814] [20230107:084600:1673077560:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:debian-desktop:1:1:4814:4814] [20230107:084600:1673077560:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:debian-desktop:1:1:4814:4814] [20230107:084600:1673077560:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:debian-desktop:1:1:4814:4814] [20230107:084600:1673077560:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:debian-desktop:1:1:4814:4814] [20230107:084600:1673077560:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:debian-desktop:1:1:4814:4814] [20230107:084600:1673077560:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:debian-desktop:1:1:4814:4814] [20230107:084600:1673077560:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:debian-desktop:1:1:4814:4814] [20230107:084600:1673077560:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:debian-desktop:1:1:4814:4814] [20230107:084600:1673077560:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:debian-desktop:1:1:4814:4814] [20230107:084600:1673077560:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:debian-desktop:1:1:4814:4814] [20230107:084600:1673077560:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  08:46:31 STEP 1320 H=1320:00 +CPU= 25.891
  08:46:47 STEP 1321 H=1321:00 +CPU= 14.864
  08:47:02 STEP 1322 H=1322:00 +CPU= 15.443
  08:47:26 STEP 1323 H=1323:00 +CPU= 23.778
  08:47:43 STEP 1324 H=1324:00 +CPU= 15.856
  08:47:58 STEP 1325 H=1325:00 +CPU= 15.335
  08:48:22 STEP 1326 H=1326:00 +CPU= 23.489
  08:48:37 STEP 1327 H=1327:00 +CPU= 14.501
  08:48:52 STEP 1328 H=1328:00 +CPU= 14.875
  08:49:15 STEP 1329 H=1329:00 +CPU= 22.961
  08:49:30 STEP 1330 H=1330:00 +CPU= 15.136
  08:49:45 STEP 1331 H=1331:00 +CPU= 14.664
  08:50:11 STEP 1332 H=1332:00 +CPU= 25.475
Moving to projects directory: /media/nathan/512gb/boinc/slots/3/ICMGGhq0f+001332
Moving to projects directory: /media/nathan/512gb/boinc/slots/3/ICMSHhq0f+001332
Moving to projects directory: /media/nathan/512gb/boinc/slots/3/ICMUAhq0f+001332
  08:50:26 STEP 1333 H=1333:00 +CPU= 14.723
  08:50:41 STEP 1334 H=1334:00 +CPU= 15.156
  08:51:04 STEP 1335 H=1335:00 +CPU= 23.107
  08:51:20 STEP 1336 H=1336:00 +CPU= 15.146
  08:51:33 STEP 1337 H=1337:00 +CPU= 13.426
  08:51:55 STEP 1338 H=1338:00 +CPU= 21.836
  08:52:09 STEP 1339 H=1339:00 +CPU= 13.330
  08:52:23 STEP 1340 H=1340:00 +CPU= 13.756
  08:52:44 STEP 1341 H=1341:00 +CPU= 21.454
  08:52:58 STEP 1342 H=1342:00 +CPU= 13.747
  08:53:12 STEP 1343 H=1343:00 +CPU= 13.691
  08:53:37 STEP 1344 H=1344:00 +CPU= 24.742
Moving to projects directory: /media/nathan/512gb/boinc/slots/3/ICMGGhq0f+001344
Moving to projects directory: /media/nathan/512gb/boinc/slots/3/ICMSHhq0f+001344
Moving to projects directory: /media/nathan/512gb/boinc/slots/3/ICMUAhq0f+001344
Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12180874/ICMGGhq0f+001320
Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12180874/ICMSHhq0f+001320
Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12180874/ICMUAhq0f+001320
Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12180874/ICMGGhq0f+001332
Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12180874/ICMSHhq0f+001332
Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12180874/ICMUAhq0f+001332
Zipping up the intermediate file: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_0230_1995050100_123_964_12180874_1_r1714657139_41.zip
Uploading the intermediate file: upload_file_41.zip
  08:53:50 STEP 1345 H=1345:00 +CPU= 13.354
  08:54:04 STEP 1346 H=1346:00 +CPU= 13.755
  08:54:26 STEP 1347 H=1347:00 +CPU= 21.349
  08:54:40 STEP 1348 H=1348:00 +CPU= 13.765
  08:54:53 STEP 1349 H=1349:00 +CPU= 13.343
  08:55:15 STEP 1350 H=1350:00 +CPU= 21.870
  08:55:29 STEP 1351 H=1351:00 +CPU= 13.363
  08:55:43 STEP 1352 H=1352:00 +CPU= 13.792
  08:56:04 STEP 1353 H=1353:00 +CPU= 21.345
  08:56:18 STEP 1354 H=1354:00 +CPU= 13.792
Abort request received from BOINC client, ending the child process

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:10:52 1507457 22295889 oifs_43r3_ps_0230_1995050100_123_964_12180874_1 4,752,000 18,073 0.0038
09 May 2023 16:10:45 1507457 22295889 oifs_43r3_ps_0230_1995050100_123_964_12180874_1 3,888,000 13,657 0.0035
09 May 2023 16:10:42 1507457 22295889 oifs_43r3_ps_0230_1995050100_123_964_12180874_1 3,024,000 9,165 0.0030
09 May 2023 16:10:38 1507457 22295889 oifs_43r3_ps_0230_1995050100_123_964_12180874_1 2,160,000 4,620 0.0021


©2024 climateprediction.net