climateprediction.net home page
Task 22262867

Task 22262867

Name oifs_43r3_ps_0386_1992050100_123_961_12178030_0
Workunit 12178030
Created 22 Dec 2022, 0:18:38 UTC
Sent 4 Jan 2023, 23:21:18 UTC
Report deadline 3 Feb 2023, 23:21:18 UTC
Received 9 Jan 2023, 10:57:04 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1536968
Run time 1 days 12 hours 33 min 47 sec
CPU time 13 hours 29 min
Validate state Invalid
Credit 11,612.16
Device peak FLOPS 8.32 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,413.79 MB
Peak swap size 4,978.08 MB
Peak disk usage 7,594.18 MB
Stderr
<core_client_version>7.16.16</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
 signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:server0:1:1:274269:274269] [20230109:061452:1673216092:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:server0:1:1:274269:274269] [20230109:061452:1673216092:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:server0:1:1:274269:274269] [20230109:061452:1673216092:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:server0:1:1:274269:274269] [20230109:061452:1673216092:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  06:15:45 STEP 2064 H=2064:00 +CPU= 25.740
  06:16:07 STEP 2065 H=2065:00 +CPU= 13.929
  06:16:35 STEP 2066 H=2066:00 +CPU= 14.659
  06:17:16 STEP 2067 H=2067:00 +CPU= 23.080
  06:17:41 STEP 2068 H=2068:00 +CPU= 14.362
  06:18:04 STEP 2069 H=2069:00 +CPU= 14.311
  06:18:38 STEP 2070 H=2070:00 +CPU= 22.931
  06:19:01 STEP 2071 H=2071:00 +CPU= 13.889
  06:19:22 STEP 2072 H=2072:00 +CPU= 14.301
  06:19:57 STEP 2073 H=2073:00 +CPU= 22.371
  06:20:18 STEP 2074 H=2074:00 +CPU= 14.440
  06:20:41 STEP 2075 H=2075:00 +CPU= 13.894
  06:21:18 STEP 2076 H=2076:00 +CPU= 24.786
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002076
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002076
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002076
  06:21:40 STEP 2077 H=2077:00 +CPU= 13.984
  06:22:03 STEP 2078 H=2078:00 +CPU= 14.312
  06:22:35 STEP 2079 H=2079:00 +CPU= 22.321
  06:22:56 STEP 2080 H=2080:00 +CPU= 14.542
  06:23:23 STEP 2081 H=2081:00 +CPU= 13.975
  06:24:01 STEP 2082 H=2082:00 +CPU= 22.650
  06:24:28 STEP 2083 H=2083:00 +CPU= 14.371
  06:24:54 STEP 2084 H=2084:00 +CPU= 14.813
  06:25:28 STEP 2085 H=2085:00 +CPU= 22.305
  06:25:57 STEP 2086 H=2086:00 +CPU= 14.554
  06:26:22 STEP 2087 H=2087:00 +CPU= 15.196
  06:27:12 STEP 2088 H=2088:00 +CPU= 27.167
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002088
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002088
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002088
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMGGhq0f+002064
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMSHhq0f+002064
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMUAhq0f+002064
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMGGhq0f+002076
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMSHhq0f+002076
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMUAhq0f+002076
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0386_1992050100_123_961_12178030_0_r861713555_86.zip
Uploading the intermediate file: upload_file_86.zip
  06:27:47 STEP 2089 H=2089:00 +CPU= 14.385
  06:28:09 STEP 2090 H=2090:00 +CPU= 14.346
  06:28:49 STEP 2091 H=2091:00 +CPU= 22.944
  06:29:16 STEP 2092 H=2092:00 +CPU= 14.535
  06:29:41 STEP 2093 H=2093:00 +CPU= 14.874
  06:30:18 STEP 2094 H=2094:00 +CPU= 23.268
  06:30:41 STEP 2095 H=2095:00 +CPU= 14.514
  06:31:08 STEP 2096 H=2096:00 +CPU= 15.020
  06:31:41 STEP 2097 H=2097:00 +CPU= 22.374
  06:32:06 STEP 2098 H=2098:00 +CPU= 14.840
  06:32:38 STEP 2099 H=2099:00 +CPU= 14.620
  06:33:28 STEP 2100 H=2100:00 +CPU= 25.683
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002100
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002100
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002100
  06:34:03 STEP 2101 H=2101:00 +CPU= 14.623
  06:34:29 STEP 2102 H=2102:00 +CPU= 14.890
  06:35:23 STEP 2103 H=2103:00 +CPU= 24.499
  06:35:57 STEP 2104 H=2104:00 +CPU= 15.234
  06:36:30 STEP 2105 H=2105:00 +CPU= 15.031
  06:37:18 STEP 2106 H=2106:00 +CPU= 23.965
  06:37:46 STEP 2107 H=2107:00 +CPU= 13.956
  06:38:23 STEP 2108 H=2108:00 +CPU= 15.438
  06:39:02 STEP 2109 H=2109:00 +CPU= 22.419
  06:39:32 STEP 2110 H=2110:00 +CPU= 14.851
  06:39:59 STEP 2111 H=2111:00 +CPU= 15.833
  06:40:49 STEP 2112 H=2112:00 +CPU= 28.162
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002112
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002112
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002112
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMGGhq0f+002088
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMSHhq0f+002088
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMUAhq0f+002088
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMGGhq0f+002100
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMSHhq0f+002100
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMUAhq0f+002100
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0386_1992050100_123_961_12178030_0_r861713555_87.zip
Uploading the intermediate file: upload_file_87.zip
  06:41:15 STEP 2113 H=2113:00 +CPU= 14.291
  06:41:45 STEP 2114 H=2114:00 +CPU= 15.234
  06:42:26 STEP 2115 H=2115:00 +CPU= 23.227
  06:43:02 STEP 2116 H=2116:00 +CPU= 15.458
  06:43:30 STEP 2117 H=2117:00 +CPU= 14.325
  06:44:22 STEP 2118 H=2118:00 +CPU= 23.169
  06:44:53 STEP 2119 H=2119:00 +CPU= 14.704
  06:45:23 STEP 2120 H=2120:00 +CPU= 15.109
  06:46:00 STEP 2121 H=2121:00 +CPU= 21.929
  06:46:27 STEP 2122 H=2122:00 +CPU= 14.836
  06:46:54 STEP 2123 H=2123:00 +CPU= 14.127
  06:47:44 STEP 2124 H=2124:00 +CPU= 25.796
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002124
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002124
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002124
  06:48:05 STEP 2125 H=2125:00 +CPU= 14.142
  06:48:42 STEP 2126 H=2126:00 +CPU= 15.177
  06:49:21 STEP 2127 H=2127:00 +CPU= 22.662
  06:49:44 STEP 2128 H=2128:00 +CPU= 15.060
  06:50:18 STEP 2129 H=2129:00 +CPU= 15.613
  06:51:06 STEP 2130 H=2130:00 +CPU= 23.746
  06:51:37 STEP 2131 H=2131:00 +CPU= 15.530
  06:52:11 STEP 2132 H=2132:00 +CPU= 15.391
  06:53:01 STEP 2133 H=2133:00 +CPU= 23.755
  06:53:30 STEP 2134 H=2134:00 +CPU= 15.578
  06:54:06 STEP 2135 H=2135:00 +CPU= 15.815
  06:54:58 STEP 2136 H=2136:00 +CPU= 27.812
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002136
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002136
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002136
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMGGhq0f+002112
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMSHhq0f+002112
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMUAhq0f+002112
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMGGhq0f+002124
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMSHhq0f+002124
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMUAhq0f+002124
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0386_1992050100_123_961_12178030_0_r861713555_88.zip
Uploading the intermediate file: upload_file_88.zip
  06:55:33 STEP 2137 H=2137:00 +CPU= 14.874
  06:56:06 STEP 2138 H=2138:00 +CPU= 15.023
  06:56:45 STEP 2139 H=2139:00 +CPU= 23.186
  06:57:16 STEP 2140 H=2140:00 +CPU= 16.142
  06:57:56 STEP 2141 H=2141:00 +CPU= 15.101
  06:58:46 STEP 2142 H=2142:00 +CPU= 23.564
  06:59:15 STEP 2143 H=2143:00 +CPU= 14.432
  06:59:44 STEP 2144 H=2144:00 +CPU= 15.711
  07:00:30 STEP 2145 H=2145:00 +CPU= 22.687
  07:01:02 STEP 2146 H=2146:00 +CPU= 14.312
  07:01:24 STEP 2147 H=2147:00 +CPU= 14.858
  07:02:07 STEP 2148 H=2148:00 +CPU= 25.688
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002148
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002148
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002148
  07:02:37 STEP 2149 H=2149:00 +CPU= 14.221
  07:03:05 STEP 2150 H=2150:00 +CPU= 14.770
  07:03:51 STEP 2151 H=2151:00 +CPU= 23.091
  07:04:27 STEP 2152 H=2152:00 +CPU= 15.063
  07:04:50 STEP 2153 H=2153:00 +CPU= 13.808
  07:05:29 STEP 2154 H=2154:00 +CPU= 23.143
  07:06:14 STEP 2155 H=2155:00 +CPU= 16.478
  07:06:42 STEP 2156 H=2156:00 +CPU= 15.014
  07:07:28 STEP 2157 H=2157:00 +CPU= 23.694
  07:07:54 STEP 2158 H=2158:00 +CPU= 15.773
  07:08:24 STEP 2159 H=2159:00 +CPU= 16.299
  07:09:17 STEP 2160 H=2160:00 +CPU= 28.561
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002160
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002160
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002160
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMGGhq0f+002136
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMSHhq0f+002136
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMUAhq0f+002136
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMGGhq0f+002148
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMSHhq0f+002148
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMUAhq0f+002148
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0386_1992050100_123_961_12178030_0_r861713555_89.zip
Uploading the intermediate file: upload_file_89.zip
  07:09:46 STEP 2161 H=2161:00 +CPU= 15.461
  07:10:13 STEP 2162 H=2162:00 +CPU= 15.348
  07:10:43 STEP 2163 H=2163:00 +CPU= 22.600
  07:11:05 STEP 2164 H=2164:00 +CPU= 14.437
  07:11:25 STEP 2165 H=2165:00 +CPU= 13.566
  07:12:12 STEP 2166 H=2166:00 +CPU= 23.340
  07:12:39 STEP 2167 H=2167:00 +CPU= 14.577
  07:13:13 STEP 2168 H=2168:00 +CPU= 16.495
  07:14:06 STEP 2169 H=2169:00 +CPU= 24.452
  07:14:40 STEP 2170 H=2170:00 +CPU= 16.373
  07:15:12 STEP 2171 H=2171:00 +CPU= 15.100
  07:15:54 STEP 2172 H=2172:00 +CPU= 25.370
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002172
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002172
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002172
  07:16:12 STEP 2173 H=2173:00 +CPU= 13.425
  07:16:40 STEP 2174 H=2174:00 +CPU= 14.543
  07:17:14 STEP 2175 H=2175:00 +CPU= 22.117
  07:17:39 STEP 2176 H=2176:00 +CPU= 14.470
  07:17:58 STEP 2177 H=2177:00 +CPU= 14.003
  07:18:31 STEP 2178 H=2178:00 +CPU= 23.074
  07:18:52 STEP 2179 H=2179:00 +CPU= 14.096
  07:19:17 STEP 2180 H=2180:00 +CPU= 14.553
  07:19:46 STEP 2181 H=2181:00 +CPU= 22.666
  07:20:09 STEP 2182 H=2182:00 +CPU= 14.689
  07:20:29 STEP 2183 H=2183:00 +CPU= 14.401
  07:21:13 STEP 2184 H=2184:00 +CPU= 27.703
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002184
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002184
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002184
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMGGhq0f+002160
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMSHhq0f+002160
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMUAhq0f+002160
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMGGhq0f+002172
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMSHhq0f+002172
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMUAhq0f+002172
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0386_1992050100_123_961_12178030_0_r861713555_90.zip
Uploading the intermediate file: upload_file_90.zip
  07:21:33 STEP 2185 H=2185:00 +CPU= 13.844
  07:21:55 STEP 2186 H=2186:00 +CPU= 14.164
  07:22:30 STEP 2187 H=2187:00 +CPU= 22.521
  07:22:56 STEP 2188 H=2188:00 +CPU= 14.746
  07:23:18 STEP 2189 H=2189:00 +CPU= 13.754
  07:23:55 STEP 2190 H=2190:00 +CPU= 23.171
  07:24:15 STEP 2191 H=2191:00 +CPU= 14.312
  07:24:37 STEP 2192 H=2192:00 +CPU= 14.153
  07:25:12 STEP 2193 H=2193:00 +CPU= 22.061
  07:25:33 STEP 2194 H=2194:00 +CPU= 14.334
  07:25:55 STEP 2195 H=2195:00 +CPU= 14.308
  07:26:34 STEP 2196 H=2196:00 +CPU= 24.768
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002196
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002196
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002196
  07:26:57 STEP 2197 H=2197:00 +CPU= 14.113
  07:27:18 STEP 2198 H=2198:00 +CPU= 14.414
  07:27:48 STEP 2199 H=2199:00 +CPU= 22.084
  07:28:15 STEP 2200 H=2200:00 +CPU= 14.408
  07:28:39 STEP 2201 H=2201:00 +CPU= 14.377
  07:29:11 STEP 2202 H=2202:00 +CPU= 22.744
  07:29:30 STEP 2203 H=2203:00 +CPU= 13.894
  07:29:53 STEP 2204 H=2204:00 +CPU= 14.452
  07:30:29 STEP 2205 H=2205:00 +CPU= 22.107
  07:30:55 STEP 2206 H=2206:00 +CPU= 14.491
  07:31:19 STEP 2207 H=2207:00 +CPU= 14.781
  07:32:16 STEP 2208 H=2208:00 +CPU= 28.955
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002208
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002208
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002208
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMGGhq0f+002184
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMSHhq0f+002184
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMUAhq0f+002184
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMGGhq0f+002196
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMSHhq0f+002196
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMUAhq0f+002196
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0386_1992050100_123_961_12178030_0_r861713555_91.zip
Uploading the intermediate file: upload_file_91.zip
  07:32:43 STEP 2209 H=2209:00 +CPU= 14.206
  07:33:13 STEP 2210 H=2210:00 +CPU= 14.625
  07:33:44 STEP 2211 H=2211:00 +CPU= 22.688
  07:34:20 STEP 2212 H=2212:00 +CPU= 15.548
  07:34:45 STEP 2213 H=2213:00 +CPU= 14.455
  07:35:20 STEP 2214 H=2214:00 +CPU= 23.151
  07:35:53 STEP 2215 H=2215:00 +CPU= 15.472
  07:36:24 STEP 2216 H=2216:00 +CPU= 15.246
  07:37:20 STEP 2217 H=2217:00 +CPU= 24.891
  07:37:48 STEP 2218 H=2218:00 +CPU= 15.756
  07:38:26 STEP 2219 H=2219:00 +CPU= 14.538
  07:39:21 STEP 2220 H=2220:00 +CPU= 25.529
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002220
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002220
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002220
  07:39:45 STEP 2221 H=2221:00 +CPU= 15.036
  07:40:15 STEP 2222 H=2222:00 +CPU= 15.028
  07:40:55 STEP 2223 H=2223:00 +CPU= 23.174
  07:41:18 STEP 2224 H=2224:00 +CPU= 14.185
  07:41:50 STEP 2225 H=2225:00 +CPU= 16.424
  07:42:39 STEP 2226 H=2226:00 +CPU= 23.437
  07:43:06 STEP 2227 H=2227:00 +CPU= 14.899
  07:43:47 STEP 2228 H=2228:00 +CPU= 15.651
  07:44:40 STEP 2229 H=2229:00 +CPU= 24.346
  07:45:17 STEP 2230 H=2230:00 +CPU= 15.259
  07:45:45 STEP 2231 H=2231:00 +CPU= 14.807
  07:46:36 STEP 2232 H=2232:00 +CPU= 27.555
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002232
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002232
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002232
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMGGhq0f+002208
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMSHhq0f+002208
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMUAhq0f+002208
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMGGhq0f+002220
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMSHhq0f+002220
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMUAhq0f+002220
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0386_1992050100_123_961_12178030_0_r861713555_92.zip
Uploading the intermediate file: upload_file_92.zip
  07:47:01 STEP 2233 H=2233:00 +CPU= 14.264
  07:47:36 STEP 2234 H=2234:00 +CPU= 15.008
  07:48:23 STEP 2235 H=2235:00 +CPU= 25.187
  07:48:49 STEP 2236 H=2236:00 +CPU= 15.475
  07:49:26 STEP 2237 H=2237:00 +CPU= 15.617
  07:50:13 STEP 2238 H=2238:00 +CPU= 24.135
  07:50:45 STEP 2239 H=2239:00 +CPU= 15.428
  07:51:08 STEP 2240 H=2240:00 +CPU= 15.472
  07:52:02 STEP 2241 H=2241:00 +CPU= 23.664
  07:52:29 STEP 2242 H=2242:00 +CPU= 15.267
  07:52:54 STEP 2243 H=2243:00 +CPU= 14.391
  07:53:51 STEP 2244 H=2244:00 +CPU= 26.074
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002244
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002244
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002244
  07:54:17 STEP 2245 H=2245:00 +CPU= 14.224
  07:54:51 STEP 2246 H=2246:00 +CPU= 15.504
  07:55:34 STEP 2247 H=2247:00 +CPU= 23.674
  07:55:56 STEP 2248 H=2248:00 +CPU= 14.215
  07:56:24 STEP 2249 H=2249:00 +CPU= 14.451
  07:57:01 STEP 2250 H=2250:00 +CPU= 22.737
  07:57:33 STEP 2251 H=2251:00 +CPU= 14.906
  07:58:00 STEP 2252 H=2252:00 +CPU= 15.204
  07:58:46 STEP 2253 H=2253:00 +CPU= 23.735
  07:59:15 STEP 2254 H=2254:00 +CPU= 15.499
  07:59:38 STEP 2255 H=2255:00 +CPU= 14.611
  08:00:33 STEP 2256 H=2256:00 +CPU= 27.420
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002256
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002256
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002256
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMGGhq0f+002232
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMSHhq0f+002232
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMUAhq0f+002232
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMGGhq0f+002244
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMSHhq0f+002244
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMUAhq0f+002244
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0386_1992050100_123_961_12178030_0_r861713555_93.zip
Uploading the intermediate file: upload_file_93.zip
  08:00:56 STEP 2257 H=2257:00 +CPU= 14.410
  08:01:33 STEP 2258 H=2258:00 +CPU= 14.791
  08:02:17 STEP 2259 H=2259:00 +CPU= 22.774
  08:02:44 STEP 2260 H=2260:00 +CPU= 15.186
  08:03:15 STEP 2261 H=2261:00 +CPU= 14.740
  08:03:58 STEP 2262 H=2262:00 +CPU= 23.439
  08:04:27 STEP 2263 H=2263:00 +CPU= 14.004
  08:05:00 STEP 2264 H=2264:00 +CPU= 15.188
  08:05:42 STEP 2265 H=2265:00 +CPU= 22.581
  08:06:09 STEP 2266 H=2266:00 +CPU= 15.593
  08:06:51 STEP 2267 H=2267:00 +CPU= 14.851
  08:07:45 STEP 2268 H=2268:00 +CPU= 26.885
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002268
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002268
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002268
  08:08:11 STEP 2269 H=2269:00 +CPU= 14.134
  08:08:51 STEP 2270 H=2270:00 +CPU= 16.037
  08:09:29 STEP 2271 H=2271:00 +CPU= 22.610
  08:09:58 STEP 2272 H=2272:00 +CPU= 15.199
  08:10:23 STEP 2273 H=2273:00 +CPU= 14.477
  08:10:56 STEP 2274 H=2274:00 +CPU= 22.853
  08:11:14 STEP 2275 H=2275:00 +CPU= 13.764
  08:11:38 STEP 2276 H=2276:00 +CPU= 14.374
  08:12:20 STEP 2277 H=2277:00 +CPU= 23.545
  08:12:51 STEP 2278 H=2278:00 +CPU= 14.511
  08:13:17 STEP 2279 H=2279:00 +CPU= 15.460
  08:14:10 STEP 2280 H=2280:00 +CPU= 27.328
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002280
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002280
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMGGhq0f+002256
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMSHhq0f+002256
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMUAhq0f+002256
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMGGhq0f+002268
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMSHhq0f+002268
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMUAhq0f+002268
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0386_1992050100_123_961_12178030_0_r861713555_94.zip
Uploading the intermediate file: upload_file_94.zip
  08:14:38 STEP 2281 H=2281:00 +CPU= 15.532
  08:15:00 STEP 2282 H=2282:00 +CPU= 15.360
  08:15:48 STEP 2283 H=2283:00 +CPU= 23.044
  08:16:08 STEP 2284 H=2284:00 +CPU= 14.842
  08:16:29 STEP 2285 H=2285:00 +CPU= 14.191
  08:17:12 STEP 2286 H=2286:00 +CPU= 22.813
  08:17:31 STEP 2287 H=2287:00 +CPU= 13.574
  08:17:49 STEP 2288 H=2288:00 +CPU= 14.156
  08:18:26 STEP 2289 H=2289:00 +CPU= 22.336
  08:18:48 STEP 2290 H=2290:00 +CPU= 14.363
  08:19:08 STEP 2291 H=2291:00 +CPU= 13.911
  08:19:42 STEP 2292 H=2292:00 +CPU= 24.471
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002292
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002292
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002292
  08:20:06 STEP 2293 H=2293:00 +CPU= 14.201
  08:20:27 STEP 2294 H=2294:00 +CPU= 14.400
  08:21:05 STEP 2295 H=2295:00 +CPU= 22.101
  08:21:34 STEP 2296 H=2296:00 +CPU= 15.190
  08:21:56 STEP 2297 H=2297:00 +CPU= 13.894
  08:22:29 STEP 2298 H=2298:00 +CPU= 22.620
  08:22:53 STEP 2299 H=2299:00 +CPU= 14.589
  08:23:13 STEP 2300 H=2300:00 +CPU= 14.124
  08:23:45 STEP 2301 H=2301:00 +CPU= 22.133
  08:24:07 STEP 2302 H=2302:00 +CPU= 14.114
  08:24:31 STEP 2303 H=2303:00 +CPU= 14.558
  08:25:12 STEP 2304 H=2304:00 +CPU= 26.673
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002304
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002304
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002304
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMGGhq0f+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMSHhq0f+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMUAhq0f+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMGGhq0f+002292
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMSHhq0f+002292
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMUAhq0f+002292
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0386_1992050100_123_961_12178030_0_r861713555_95.zip
Uploading the intermediate file: upload_file_95.zip
  08:25:31 STEP 2305 H=2305:00 +CPU= 14.190
Uploading trickle at timestep: 8294400
  08:25:55 STEP 2306 H=2306:00 +CPU= 14.721
  08:26:29 STEP 2307 H=2307:00 +CPU= 22.938
  08:26:54 STEP 2308 H=2308:00 +CPU= 14.325
  08:27:16 STEP 2309 H=2309:00 +CPU= 14.193
  08:27:48 STEP 2310 H=2310:00 +CPU= 22.928
  08:28:09 STEP 2311 H=2311:00 +CPU= 14.792
  08:28:27 STEP 2312 H=2312:00 +CPU= 14.262
  08:29:02 STEP 2313 H=2313:00 +CPU= 22.181
  08:29:22 STEP 2314 H=2314:00 +CPU= 14.441
  08:29:49 STEP 2315 H=2315:00 +CPU= 14.377
  08:30:32 STEP 2316 H=2316:00 +CPU= 24.989
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002316
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002316
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002316
  08:30:54 STEP 2317 H=2317:00 +CPU= 13.948
  08:31:17 STEP 2318 H=2318:00 +CPU= 14.580
  08:31:59 STEP 2319 H=2319:00 +CPU= 22.940
  08:32:29 STEP 2320 H=2320:00 +CPU= 16.178
  08:32:51 STEP 2321 H=2321:00 +CPU= 13.869
  08:33:38 STEP 2322 H=2322:00 +CPU= 23.379
  08:34:04 STEP 2323 H=2323:00 +CPU= 15.827
  08:34:31 STEP 2324 H=2324:00 +CPU= 15.667
  08:35:13 STEP 2325 H=2325:00 +CPU= 22.621
  08:35:42 STEP 2326 H=2326:00 +CPU= 16.206
  08:36:08 STEP 2327 H=2327:00 +CPU= 15.339
  08:37:08 STEP 2328 H=2328:00 +CPU= 27.934
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002328
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002328
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002328
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMGGhq0f+002304
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMSHhq0f+002304
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMUAhq0f+002304
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMGGhq0f+002316
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMSHhq0f+002316
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMUAhq0f+002316
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0386_1992050100_123_961_12178030_0_r861713555_96.zip
Uploading the intermediate file: upload_file_96.zip
  08:37:35 STEP 2329 H=2329:00 +CPU= 13.795
  08:38:14 STEP 2330 H=2330:00 +CPU= 15.411
  08:38:53 STEP 2331 H=2331:00 +CPU= 22.229
  08:39:20 STEP 2332 H=2332:00 +CPU= 15.686
  08:39:57 STEP 2333 H=2333:00 +CPU= 14.758
  08:40:43 STEP 2334 H=2334:00 +CPU= 23.748
  08:41:18 STEP 2335 H=2335:00 +CPU= 14.820
  08:41:52 STEP 2336 H=2336:00 +CPU= 15.492
  08:42:32 STEP 2337 H=2337:00 +CPU= 23.081
  08:43:04 STEP 2338 H=2338:00 +CPU= 15.373
  08:43:37 STEP 2339 H=2339:00 +CPU= 14.814
  08:44:22 STEP 2340 H=2340:00 +CPU= 25.708
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002340
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002340
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002340
  08:45:00 STEP 2341 H=2341:00 +CPU= 15.030
  08:45:25 STEP 2342 H=2342:00 +CPU= 14.356
  08:46:20 STEP 2343 H=2343:00 +CPU= 23.146
  08:46:56 STEP 2344 H=2344:00 +CPU= 16.099
  08:47:23 STEP 2345 H=2345:00 +CPU= 14.682
  08:48:07 STEP 2346 H=2346:00 +CPU= 22.832
  08:48:38 STEP 2347 H=2347:00 +CPU= 14.645
  08:49:12 STEP 2348 H=2348:00 +CPU= 15.410
  08:49:57 STEP 2349 H=2349:00 +CPU= 22.052
  08:50:24 STEP 2350 H=2350:00 +CPU= 15.382
  08:50:53 STEP 2351 H=2351:00 +CPU= 15.327
  08:51:39 STEP 2352 H=2352:00 +CPU= 28.028
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002352
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002352
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002352
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMGGhq0f+002328
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMSHhq0f+002328
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMUAhq0f+002328
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMGGhq0f+002340
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMSHhq0f+002340
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMUAhq0f+002340
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0386_1992050100_123_961_12178030_0_r861713555_97.zip
Uploading the intermediate file: upload_file_97.zip
  08:52:08 STEP 2353 H=2353:00 +CPU= 14.355
  08:52:43 STEP 2354 H=2354:00 +CPU= 14.785
  08:53:43 STEP 2355 H=2355:00 +CPU= 23.697
  08:54:10 STEP 2356 H=2356:00 +CPU= 14.221
  08:54:40 STEP 2357 H=2357:00 +CPU= 14.711
  08:55:20 STEP 2358 H=2358:00 +CPU= 23.345
  08:55:51 STEP 2359 H=2359:00 +CPU= 14.409
  08:56:25 STEP 2360 H=2360:00 +CPU= 15.754
  08:57:15 STEP 2361 H=2361:00 +CPU= 23.313
  08:57:44 STEP 2362 H=2362:00 +CPU= 14.838
  08:58:23 STEP 2363 H=2363:00 +CPU= 16.911
  08:59:20 STEP 2364 H=2364:00 +CPU= 26.574
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002364
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002364
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002364
  08:59:41 STEP 2365 H=2365:00 +CPU= 14.248
  09:00:10 STEP 2366 H=2366:00 +CPU= 14.749
  09:01:01 STEP 2367 H=2367:00 +CPU= 23.784
  09:01:27 STEP 2368 H=2368:00 +CPU= 15.052
  09:01:54 STEP 2369 H=2369:00 +CPU= 14.695
  09:02:39 STEP 2370 H=2370:00 +CPU= 23.928
  09:03:06 STEP 2371 H=2371:00 +CPU= 14.844
  09:03:41 STEP 2372 H=2372:00 +CPU= 15.179
  09:04:26 STEP 2373 H=2373:00 +CPU= 24.063
  09:04:51 STEP 2374 H=2374:00 +CPU= 15.453
  09:05:28 STEP 2375 H=2375:00 +CPU= 15.348
  09:06:17 STEP 2376 H=2376:00 +CPU= 28.844
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002376
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002376
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002376
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMGGhq0f+002352
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMSHhq0f+002352
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMUAhq0f+002352
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMGGhq0f+002364
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMSHhq0f+002364
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMUAhq0f+002364
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0386_1992050100_123_961_12178030_0_r861713555_98.zip
Uploading the intermediate file: upload_file_98.zip
  09:06:51 STEP 2377 H=2377:00 +CPU= 15.380
  09:07:31 STEP 2378 H=2378:00 +CPU= 15.874
  09:08:06 STEP 2379 H=2379:00 +CPU= 22.460
  09:08:42 STEP 2380 H=2380:00 +CPU= 15.076
  09:09:11 STEP 2381 H=2381:00 +CPU= 14.997
  09:09:52 STEP 2382 H=2382:00 +CPU= 24.082
  09:10:25 STEP 2383 H=2383:00 +CPU= 14.429
  09:11:00 STEP 2384 H=2384:00 +CPU= 15.890
  09:11:50 STEP 2385 H=2385:00 +CPU= 23.686
  09:12:23 STEP 2386 H=2386:00 +CPU= 15.122
  09:13:00 STEP 2387 H=2387:00 +CPU= 15.451
  09:13:43 STEP 2388 H=2388:00 +CPU= 25.577
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002388
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002388
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002388
  09:14:14 STEP 2389 H=2389:00 +CPU= 14.198
  09:14:52 STEP 2390 H=2390:00 +CPU= 15.312
  09:15:33 STEP 2391 H=2391:00 +CPU= 22.964
  09:16:02 STEP 2392 H=2392:00 +CPU= 15.111
  09:16:24 STEP 2393 H=2393:00 +CPU= 13.899
  09:17:00 STEP 2394 H=2394:00 +CPU= 22.753
  09:17:21 STEP 2395 H=2395:00 +CPU= 14.252
  09:17:46 STEP 2396 H=2396:00 +CPU= 14.706
  09:18:20 STEP 2397 H=2397:00 +CPU= 22.256
  09:18:47 STEP 2398 H=2398:00 +CPU= 14.641
  09:19:06 STEP 2399 H=2399:00 +CPU= 14.231
  09:19:46 STEP 2400 H=2400:00 +CPU= 26.786
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002400
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002400
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002400
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMGGhq0f+002376
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMSHhq0f+002376
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMUAhq0f+002376
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMGGhq0f+002388
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMSHhq0f+002388
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMUAhq0f+002388
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0386_1992050100_123_961_12178030_0_r861713555_99.zip
Uploading the intermediate file: upload_file_99.zip
  09:20:16 STEP 2401 H=2401:00 +CPU= 14.431
  09:20:40 STEP 2402 H=2402:00 +CPU= 14.453
  09:21:16 STEP 2403 H=2403:00 +CPU= 22.898
  09:21:37 STEP 2404 H=2404:00 +CPU= 14.103
  09:22:00 STEP 2405 H=2405:00 +CPU= 13.951
  09:22:38 STEP 2406 H=2406:00 +CPU= 22.798
  09:22:59 STEP 2407 H=2407:00 +CPU= 14.080
  09:23:27 STEP 2408 H=2408:00 +CPU= 14.394
  09:24:10 STEP 2409 H=2409:00 +CPU= 23.420
  09:24:30 STEP 2410 H=2410:00 +CPU= 14.346
  09:24:53 STEP 2411 H=2411:00 +CPU= 14.063
  09:25:30 STEP 2412 H=2412:00 +CPU= 25.006
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002412
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002412
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002412
  09:25:48 STEP 2413 H=2413:00 +CPU= 13.550
  09:26:10 STEP 2414 H=2414:00 +CPU= 14.882
  09:26:46 STEP 2415 H=2415:00 +CPU= 22.335
  09:27:10 STEP 2416 H=2416:00 +CPU= 14.812
  09:27:31 STEP 2417 H=2417:00 +CPU= 13.728
  09:28:09 STEP 2418 H=2418:00 +CPU= 22.616
  09:28:35 STEP 2419 H=2419:00 +CPU= 14.282
  09:29:05 STEP 2420 H=2420:00 +CPU= 14.593
  09:29:51 STEP 2421 H=2421:00 +CPU= 22.743
  09:30:17 STEP 2422 H=2422:00 +CPU= 14.566
  09:30:42 STEP 2423 H=2423:00 +CPU= 14.752
  09:31:23 STEP 2424 H=2424:00 +CPU= 26.272
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002424
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002424
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002424
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMGGhq0f+002400
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMSHhq0f+002400
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMUAhq0f+002400
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMGGhq0f+002412
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMSHhq0f+002412
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMUAhq0f+002412
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0386_1992050100_123_961_12178030_0_r861713555_100.zip
Uploading the intermediate file: upload_file_100.zip
  09:31:43 STEP 2425 H=2425:00 +CPU= 13.759
  09:32:07 STEP 2426 H=2426:00 +CPU= 14.232
  09:32:45 STEP 2427 H=2427:00 +CPU= 22.709
  09:33:07 STEP 2428 H=2428:00 +CPU= 14.663
  09:33:31 STEP 2429 H=2429:00 +CPU= 14.430
  09:34:12 STEP 2430 H=2430:00 +CPU= 23.047
  09:34:30 STEP 2431 H=2431:00 +CPU= 14.440
  09:34:59 STEP 2432 H=2432:00 +CPU= 15.559
  09:35:41 STEP 2433 H=2433:00 +CPU= 22.311
  09:36:06 STEP 2434 H=2434:00 +CPU= 14.845
  09:36:31 STEP 2435 H=2435:00 +CPU= 14.356
  09:37:19 STEP 2436 H=2436:00 +CPU= 24.870
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002436
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002436
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002436
  09:37:45 STEP 2437 H=2437:00 +CPU= 14.737
  09:38:11 STEP 2438 H=2438:00 +CPU= 14.987
  09:39:07 STEP 2439 H=2439:00 +CPU= 23.997
  09:39:37 STEP 2440 H=2440:00 +CPU= 15.827
  09:40:09 STEP 2441 H=2441:00 +CPU= 14.699
  09:40:55 STEP 2442 H=2442:00 +CPU= 23.567
  09:41:23 STEP 2443 H=2443:00 +CPU= 14.417
  09:41:54 STEP 2444 H=2444:00 +CPU= 15.090
  09:42:34 STEP 2445 H=2445:00 +CPU= 23.538
  09:43:08 STEP 2446 H=2446:00 +CPU= 15.999
  09:43:46 STEP 2447 H=2447:00 +CPU= 15.568
  09:44:24 STEP 2448 H=2448:00 +CPU= 27.771
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002448
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002448
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002448
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMGGhq0f+002424
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMSHhq0f+002424
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMUAhq0f+002424
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMGGhq0f+002436
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMSHhq0f+002436
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMUAhq0f+002436
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0386_1992050100_123_961_12178030_0_r861713555_101.zip
Uploading the intermediate file: upload_file_101.zip
  09:45:05 STEP 2449 H=2449:00 +CPU= 14.950
  09:45:32 STEP 2450 H=2450:00 +CPU= 14.780
  09:46:13 STEP 2451 H=2451:00 +CPU= 22.524
  09:46:45 STEP 2452 H=2452:00 +CPU= 14.932
  09:47:20 STEP 2453 H=2453:00 +CPU= 15.226
  09:48:06 STEP 2454 H=2454:00 +CPU= 24.235
  09:48:31 STEP 2455 H=2455:00 +CPU= 13.988
  09:48:54 STEP 2456 H=2456:00 +CPU= 14.678
  09:49:52 STEP 2457 H=2457:00 +CPU= 23.724
  09:50:18 STEP 2458 H=2458:00 +CPU= 14.540
  09:50:48 STEP 2459 H=2459:00 +CPU= 14.854
  09:51:41 STEP 2460 H=2460:00 +CPU= 26.281
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002460
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002460
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002460
  09:52:15 STEP 2461 H=2461:00 +CPU= 15.474
  09:52:40 STEP 2462 H=2462:00 +CPU= 14.621
  09:53:19 STEP 2463 H=2463:00 +CPU= 22.991
  09:54:05 STEP 2464 H=2464:00 +CPU= 15.541
  09:54:31 STEP 2465 H=2465:00 +CPU= 14.982
  09:55:12 STEP 2466 H=2466:00 +CPU= 23.104
  09:55:40 STEP 2467 H=2467:00 +CPU= 15.064
  09:56:08 STEP 2468 H=2468:00 +CPU= 14.904
  09:56:59 STEP 2469 H=2469:00 +CPU= 22.913
  09:57:31 STEP 2470 H=2470:00 +CPU= 15.251
  09:57:56 STEP 2471 H=2471:00 +CPU= 14.957
  09:58:44 STEP 2472 H=2472:00 +CPU= 27.017
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002472
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002472
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMGGhq0f+002448
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMSHhq0f+002448
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMUAhq0f+002448
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMGGhq0f+002460
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMSHhq0f+002460
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMUAhq0f+002460
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0386_1992050100_123_961_12178030_0_r861713555_102.zip
Uploading the intermediate file: upload_file_102.zip
  09:59:15 STEP 2473 H=2473:00 +CPU= 14.731
  09:59:42 STEP 2474 H=2474:00 +CPU= 15.404
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1992050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0386
(argv4) batchid: 961
(argv5) wuid: 12178030
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/10
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/10/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/10/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_93d403dc5d2d8f303821dbdd8c286d37 to: /var/lib/boinc-client/slots/10/oifs_43r3_ps_0386_1992050100_123_961_12178030.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/10/oifs_43r3_ps_0386_1992050100_123_961_12178030.zip
ic_ancil_file: ic_ancil_12178030
ifsdata_file: ifsdata_12178030
climate_data_file: clim_data_12178030
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_23643553a3c6592278d62d455080551e to: /var/lib/boinc-client/slots/10/ic_ancil_12178030.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/10/ic_ancil_12178030.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_8ecf8badfbc93f33135afefc1d0c40ab to: /var/lib/boinc-client/slots/10/ifsdata/ifsdata_12178030.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/10/ifsdata/ifsdata_12178030.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/10/159l_2/clim_data_12178030.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/10/159l_2/clim_data_12178030.zip
Checking for progress XML file: /var/lib/boinc-client/slots/10/progress_file_12178030.xml
Opened progress file ok : /var/lib/boinc-client/slots/10/progress_file_12178030.xml
-- Model is restarting --
Adjusting last_iter, 2475, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/10/progress_file_12178030.xml
last_cpu_time: 47808
upload_file_number: 103
last_iter: 2471
last_upload: 8899200
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0386_1992050100_123_961_12178030_0_r861713555
The child process has been launched with process id: 318039
Executing the command: /var/lib/boinc-client/slots/10/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:server0:1:1:318039:318039] [20230109:183009:1673260209:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:server0:1:1:318039:318039] [20230109:183009:1673260209:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:server0:1:1:318039:318039] [20230109:183009:1673260209:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:server0:1:1:318039:318039] [20230109:183009:1673260209:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:server0:1:1:318039:318039] [20230109:183009:1673260209:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:server0:1:1:318039:318039] [20230109:183009:1673260209:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:server0:1:1:318039:318039] [20230109:183009:1673260209:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:server0:1:1:318039:318039] [20230109:183009:1673260209:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:server0:1:1:318039:318039] [20230109:183009:1673260209:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:server0:1:1:318039:318039] [20230109:183009:1673260209:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:server0:1:1:318039:318039] [20230109:183009:1673260209:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:server0:1:1:318039:318039] [20230109:183009:1673260209:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:server0:1:1:318039:318039] [20230109:183009:1673260209:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:server0:1:1:318039:318039] [20230109:183009:1673260209:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:server0:1:1:318039:318039] [20230109:183009:1673260209:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:server0:1:1:318039:318039] [20230109:183009:1673260209:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:server0:1:1:318039:318039] [20230109:183009:1673260209:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:server0:1:1:318039:318039] [20230109:183009:1673260209:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:server0:1:1:318039:318039] [20230109:183009:1673260209:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:server0:1:1:318039:318039] [20230109:183009:1673260209:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:server0:1:1:318039:318039] [20230109:183009:1673260209:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:server0:1:1:318039:318039] [20230109:183009:1673260209:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:server0:1:1:318039:318039] [20230109:183009:1673260209:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:server0:1:1:318039:318039] [20230109:183009:1673260209:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:server0:1:1:318039:318039] [20230109:183009:1673260209:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:server0:1:1:318039:318039] [20230109:183009:1673260209:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:server0:1:1:318039:318039] [20230109:183009:1673260209:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:server0:1:1:318039:318039] [20230109:183009:1673260209:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:server0:1:1:318039:318039] [20230109:183009:1673260209:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:server0:1:1:318039:318039] [20230109:183009:1673260209:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  18:30:56 STEP 2472 H=2472:00 +CPU= 25.528
  18:31:31 STEP 2473 H=2473:00 +CPU= 14.767
  18:31:57 STEP 2474 H=2474:00 +CPU= 14.440
  18:32:53 STEP 2475 H=2475:00 +CPU= 23.771
  18:33:21 STEP 2476 H=2476:00 +CPU= 14.968
  18:33:55 STEP 2477 H=2477:00 +CPU= 14.395
  18:34:43 STEP 2478 H=2478:00 +CPU= 24.236
  18:35:14 STEP 2479 H=2479:00 +CPU= 15.209
  18:35:40 STEP 2480 H=2480:00 +CPU= 14.457
  18:36:22 STEP 2481 H=2481:00 +CPU= 22.634
  18:36:49 STEP 2482 H=2482:00 +CPU= 15.769
  18:37:18 STEP 2483 H=2483:00 +CPU= 14.522
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1992050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0386
(argv4) batchid: 961
(argv5) wuid: 12178030
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/10
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/10/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/10/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_93d403dc5d2d8f303821dbdd8c286d37 to: /var/lib/boinc-client/slots/10/oifs_43r3_ps_0386_1992050100_123_961_12178030.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/10/oifs_43r3_ps_0386_1992050100_123_961_12178030.zip
ic_ancil_file: ic_ancil_12178030
ifsdata_file: ifsdata_12178030
climate_data_file: clim_data_12178030
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_23643553a3c6592278d62d455080551e to: /var/lib/boinc-client/slots/10/ic_ancil_12178030.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/10/ic_ancil_12178030.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_8ecf8badfbc93f33135afefc1d0c40ab to: /var/lib/boinc-client/slots/10/ifsdata/ifsdata_12178030.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/10/ifsdata/ifsdata_12178030.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/10/159l_2/clim_data_12178030.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/10/159l_2/clim_data_12178030.zip
Checking for progress XML file: /var/lib/boinc-client/slots/10/progress_file_12178030.xml
Opened progress file ok : /var/lib/boinc-client/slots/10/progress_file_12178030.xml
-- Model is restarting --
Adjusting last_iter, 2484, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/10/progress_file_12178030.xml
last_cpu_time: 48025
upload_file_number: 103
last_iter: 2471
last_upload: 8899200
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0386_1992050100_123_961_12178030_0_r861713555
The child process has been launched with process id: 319010
Executing the command: /var/lib/boinc-client/slots/10/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:server0:1:1:319010:319010] [20230109:183835:1673260715:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:server0:1:1:319010:319010] [20230109:183835:1673260715:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:server0:1:1:319010:319010] [20230109:183835:1673260715:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:server0:1:1:319010:319010] [20230109:183835:1673260715:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:server0:1:1:319010:319010] [20230109:183835:1673260715:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:server0:1:1:319010:319010] [20230109:183835:1673260715:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:server0:1:1:319010:319010] [20230109:183835:1673260715:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:server0:1:1:319010:319010] [20230109:183835:1673260715:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:server0:1:1:319010:319010] [20230109:183835:1673260715:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:server0:1:1:319010:319010] [20230109:183835:1673260715:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:server0:1:1:319010:319010] [20230109:183835:1673260715:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:server0:1:1:319010:319010] [20230109:183835:1673260715:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:server0:1:1:319010:319010] [20230109:183835:1673260715:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:server0:1:1:319010:319010] [20230109:183835:1673260715:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:server0:1:1:319010:319010] [20230109:183835:1673260715:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:server0:1:1:319010:319010] [20230109:183835:1673260715:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:server0:1:1:319010:319010] [20230109:183835:1673260715:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:server0:1:1:319010:319010] [20230109:183835:1673260715:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:server0:1:1:319010:319010] [20230109:183835:1673260715:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:server0:1:1:319010:319010] [20230109:183835:1673260715:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:server0:1:1:319010:319010] [20230109:183835:1673260715:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:server0:1:1:319010:319010] [20230109:183835:1673260715:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:server0:1:1:319010:319010] [20230109:183835:1673260715:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:server0:1:1:319010:319010] [20230109:183835:1673260715:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:server0:1:1:319010:319010] [20230109:183835:1673260715:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:server0:1:1:319010:319010] [20230109:183835:1673260715:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:server0:1:1:319010:319010] [20230109:183835:1673260715:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:server0:1:1:319010:319010] [20230109:183835:1673260715:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:server0:1:1:319010:319010] [20230109:183835:1673260715:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:server0:1:1:319010:319010] [20230109:183835:1673260715:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  18:39:40 STEP 2472 H=2472:00 +CPU= 26.490
  18:40:12 STEP 2473 H=2473:00 +CPU= 14.690
  18:40:38 STEP 2474 H=2474:00 +CPU= 14.154
  18:41:20 STEP 2475 H=2475:00 +CPU= 22.536
  18:41:49 STEP 2476 H=2476:00 +CPU= 14.629
  18:42:27 STEP 2477 H=2477:00 +CPU= 15.418
  18:43:17 STEP 2478 H=2478:00 +CPU= 24.927
  18:43:51 STEP 2479 H=2479:00 +CPU= 15.426
  18:44:29 STEP 2480 H=2480:00 +CPU= 16.109
  18:45:07 STEP 2481 H=2481:00 +CPU= 22.794
  18:45:42 STEP 2482 H=2482:00 +CPU= 15.491
  18:46:13 STEP 2483 H=2483:00 +CPU= 14.179
  18:47:02 STEP 2484 H=2484:00 +CPU= 26.462
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002484
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002484
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002484
  18:47:29 STEP 2485 H=2485:00 +CPU= 14.352
  18:47:55 STEP 2486 H=2486:00 +CPU= 14.296
  18:48:42 STEP 2487 H=2487:00 +CPU= 23.816
  18:49:14 STEP 2488 H=2488:00 +CPU= 14.870
  18:49:38 STEP 2489 H=2489:00 +CPU= 14.614
  18:50:26 STEP 2490 H=2490:00 +CPU= 23.653
  18:50:53 STEP 2491 H=2491:00 +CPU= 14.607
  18:51:24 STEP 2492 H=2492:00 +CPU= 14.964
  18:51:57 STEP 2493 H=2493:00 +CPU= 21.959
  18:52:28 STEP 2494 H=2494:00 +CPU= 15.246
  18:53:02 STEP 2495 H=2495:00 +CPU= 15.401
  18:53:51 STEP 2496 H=2496:00 +CPU= 27.385
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002496
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002496
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMGGhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMSHhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMUAhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMGGhq0f+002484
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMSHhq0f+002484
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178030/ICMUAhq0f+002484
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0386_1992050100_123_961_12178030_0_r861713555_103.zip
Uploading the intermediate file: upload_file_103.zip
  18:54:22 STEP 2497 H=2497:00 +CPU= 14.643
  18:54:45 STEP 2498 H=2498:00 +CPU= 14.352
  18:55:26 STEP 2499 H=2499:00 +CPU= 22.805
Abort request received from BOINC client, ending the child process

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:13:15 1536968 22262867 oifs_43r3_ps_0386_1992050100_123_961_12178030_0 8,294,400 44,775 0.0054
09 May 2023 16:13:06 1536968 22262867 oifs_43r3_ps_0386_1992050100_123_961_12178030_0 6,825,600 37,095 0.0054
09 May 2023 16:12:55 1536968 22262867 oifs_43r3_ps_0386_1992050100_123_961_12178030_0 5,270,400 29,341 0.0056
09 May 2023 16:12:45 1536968 22262867 oifs_43r3_ps_0386_1992050100_123_961_12178030_0 3,801,600 21,819 0.0057
09 May 2023 16:12:35 1536968 22262867 oifs_43r3_ps_0386_1992050100_123_961_12178030_0 2,332,800 14,192 0.0061
09 May 2023 16:12:25 1536968 22262867 oifs_43r3_ps_0386_1992050100_123_961_12178030_0 864,000 6,903 0.0080


©2024 climateprediction.net