climateprediction.net home page
Task 21911084

Task 21911084

Name wah2_sam50_n1ni_201312_28_859_011975977_1
Workunit 11975977
Created 2 Feb 2020, 9:34:31 UTC
Sent 2 Feb 2020, 9:34:45 UTC
Report deadline 14 Jan 2021, 14:54:45 UTC
Received 12 Jun 2020, 16:07:59 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 194 (0x000000C2) EXIT_ABORTED_BY_CLIENT
Computer ID 1484388
Run time 14 days 5 hours 9 min 52 sec
CPU time 2 sec
Validate state Invalid
Credit 13,698.58
Device peak FLOPS 2.46 GFLOPS
Application version Weather At Home 2 (wah2) v8.24
windows_intelx86
Peak working set size 225.52 MB
Peak swap size 189.58 MB
Peak disk usage 163.00 MB
Stderr
<core_client_version>7.6.33</core_client_version>
<![CDATA[
<message>
finish file present too long
</message>
<stderr_txt>
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=452, iMonCtr=2
Model crash detected, will try to restart...
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4744, iMonCtr=2
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3340, iMonCtr=2
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4744, iMonCtr=2
Model crash detected, will try to restart...
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2968, iMonCtr=2
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4244, iMonCtr=2
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5040, selfPID=1216, iMonCtr=1
Model crash detected, will try to restart...
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2636, iMonCtr=2
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4404, selfPID=3556, iMonCtr=1
Model crash detected, will try to restart...
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4948, iMonCtr=2
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3432, selfPID=1480, iMonCtr=1
Model crash detected, will try to restart...
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5560, iMonCtr=2
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3992, iMonCtr=2
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4152, selfPID=3840, iMonCtr=1
Model crash detected, will try to restart...
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5080, iMonCtr=2
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5352, selfPID=4052, iMonCtr=1
Model crash detected, will try to restart...
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5284, iMonCtr=2
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5144, iMonCtr=2
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2688, iMonCtr=2
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3832, iMonCtr=2
Model crash detected, will try to restart...
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4172, iMonCtr=2
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4256, selfPID=3372, iMonCtr=1
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4960, selfPID=4744, iMonCtr=1
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3636, iMonCtr=2
Model crash detected, will try to restart...
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4196, iMonCtr=2
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4288, selfPID=3908, iMonCtr=1
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3276, iMonCtr=2
Model crash detected, will try to restart...
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2628, iMonCtr=2
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4736, selfPID=4680, iMonCtr=1
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4484, selfPID=1208, iMonCtr=1
Model crash detected, will try to restart...
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4428, iMonCtr=2
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4288, iMonCtr=2
Model crash detected, will try to restart...
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5384, iMonCtr=2
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3384, iMonCtr=2
Model crash detected, will try to restart...
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4152, iMonCtr=2
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3456, iMonCtr=2
Model crash detected, will try to restart...
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5008, iMonCtr=2
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1104, iMonCtr=2
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3492, selfPID=4696, iMonCtr=1
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4336, selfPID=3328, iMonCtr=1
Model crash detected, will try to restart...
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5108, iMonCtr=2
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1044, iMonCtr=2
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3604, iMonCtr=2
Model crash detected, will try to restart...
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4120, iMonCtr=2
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4756, iMonCtr=2
Model crash detected, will try to restart...
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5312, iMonCtr=2
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4952, iMonCtr=2
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5036, selfPID=4496, iMonCtr=1
Model crash detected, will try to restart...
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5864, iMonCtr=2
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5996, selfPID=4636, iMonCtr=1
Model crash detected, will try to restart...
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4828, iMonCtr=2
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5096, iMonCtr=2
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1464, iMonCtr=2
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4904, iMonCtr=2
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5112, selfPID=1104, iMonCtr=1
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5896, iMonCtr=2
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6020, selfPID=5476, iMonCtr=1
Model crash detected, will try to restart...
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1476, iMonCtr=2
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1116, iMonCtr=2
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4456, selfPID=2612, iMonCtr=1
Model crash detected, will try to restart...
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3360, iMonCtr=2
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4216, iMonCtr=2
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4556, iMonCtr=2
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3352, iMonCtr=2
Model crash detected, will try to restart...
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4244, iMonCtr=2
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4316, iMonCtr=2
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3876, iMonCtr=2
Model crash detected, will try to restart...
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2760, iMonCtr=2
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4104, selfPID=3940, iMonCtr=1
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2704, iMonCtr=2
Model crash detected, will try to restart...
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5312, iMonCtr=2
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5224, iMonCtr=2
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5352, selfPID=3388, iMonCtr=1
Model crash detected, will try to restart...
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4368, iMonCtr=2
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3604, iMonCtr=2
Model crash detected, will try to restart...
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3028, iMonCtr=2
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3744, selfPID=3696, iMonCtr=1
Model crash detected, will try to restart...
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1448, iMonCtr=2
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4100, selfPID=3936, iMonCtr=1
Model crash detected, will try to restart...
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5084, iMonCtr=2
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3740, iMonCtr=2
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4132, selfPID=4712, iMonCtr=1
Model crash detected, will try to restart...
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4696, iMonCtr=2
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4448, selfPID=4188, iMonCtr=1
Model crash detected, will try to restart...
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4928, iMonCtr=2
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5012, selfPID=4488, iMonCtr=1
Model crash detected, will try to restart...
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4952, iMonCtr=2
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3616, iMonCtr=2
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5356, iMonCtr=2
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=572, iMonCtr=2
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4172, selfPID=3460, iMonCtr=1
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5312, selfPID=3804, iMonCtr=1
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6220, selfPID=6564, iMonCtr=1
Model crash detected, will try to restart...
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5360, iMonCtr=2
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4868, iMonCtr=2
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4992, selfPID=4224, iMonCtr=1
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1512, iMonCtr=2
Model crash detected, will try to restart...
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4956, iMonCtr=2
GCM : BUFFIN: C I/O Error feof - Unit 61 - Return code = 16
GCM : BUFFIN: C I/O Error feof - Unit 61 - Return code = 16

GCM : BUFFIN: C I/O Error feof - Unit 63 - Return code = 16
GCM : BUFFIN: C I/O Error feof - Unit 63 - Return code = 16

GCM : BUFFIN: C I/O Error feof - Unit 64 - Return code = 16
GCM : BUFFIN: C I/O Error feof - Unit 64 - Return code = 16

Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4688, iMonCtr=2
Model crash detected, will try to restart...
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5100, iMonCtr=2
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4636, iMonCtr=2
Controller:: CPDN process is not running, exiGlobal Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=509Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6036, iMonCtr=2
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4368, iMonCtr=2
Model crash detected, will try to restart...
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4528, iMonCtr=2
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4472, iMonCtr=2
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4688, selfPID=4160, iMonCtr=1
Model crash detected, will try to restart...
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4812, iMonCtr=2
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4928, selfPID=4248, iMonCtr=1
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4972, selfPID=4496, iMonCtr=1
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3292, selfPID=4404, iMonCtr=1
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=1500, selfPID=4448, iMonCtr=1
Model crash detected, will try to restart...
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4972, iMonCtr=2
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4912, iMonCtr=2
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4152, iMonCtr=2
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3084, selfPID=4424, iMonCtr=1
Model crash detected, will try to restart...
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4892, iMonCtr=2
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4372, iMonCtr=2
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4332, iMonCtr=2
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4516, iMonCtr=2
Model crash detected, will try to restart...
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4980, iMonCtr=2
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4308, iMonCtr=2
Model crash detected, will try to restart...
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4052, iMonCtr=2
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3792, selfPID=4420, iMonCtr=1
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2860, iMonCtr=2
Model crash detected, will try to restart...
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4548, iMonCtr=2
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4648, selfPID=3396, iMonCtr=1
Model crash detected, will try to restart...
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5088, iMonCtr=2
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4892, iMonCtr=2
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4208, iMonCtr=2
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2808, selfPID=3520, iMonCtr=1
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4132, iMonCtr=2
Model crash detected, will try to restart...
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4680, iMonCtr=2
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3572, selfPID=3596, iMonCtr=1
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4120, iMonCtr=2
Model crash detected, will try to restart...
Leaving CPDN_ain::Monitor...
17:16:15 (4120): called boinc_finish(0)
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=1140, selfPID=1260, iMonCtr=1
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=732, selfPID=4408, iMonCtr=1
Model crash detected, will try to restart...
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4352, iMonCtr=2
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4336, selfPID=4432, iMonCtr=1
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4596, selfPID=3988, iMonCtr=1
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5108, selfPID=4120, iMonCtr=1
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=1288, selfPID=4444, iMonCtr=1
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4180, selfPID=4708, iMonCtr=1
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4516, selfPID=3356, iMonCtr=1
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6596, selfPID=4436, iMonCtr=1
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2872, selfPID=2176, iMonCtr=1
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3152, selfPID=4520, iMonCtr=1
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5064, selfPID=2936, iMonCtr=1
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=1032, selfPID=4392, iMonCtr=1
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3800, selfPID=4436, iMonCtr=1
Model crash detected, will try to restart...
Leaving CPDN_ain::Monitor...
17:50:06 (4436): called boinc_finish(0)
CPDN Monitor - Abort request from BOINC...
18:04:43 (3912): called boinc_finish(10)

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
29 May 2020 12:28:47 1484388 21911084 wah2_sam50_n1ni_201312_28_859_011975977_1 207,659 1,088,917 5.2438
19 May 2020 13:37:11 1484388 21911084 wah2_sam50_n1ni_201312_28_859_011975977_1 196,139 1,040,675 5.3058
16 May 2020 17:13:28 1484388 21911084 wah2_sam50_n1ni_201312_28_859_011975977_1 184,619 992,335 5.3750
07 May 2020 17:17:53 1484388 21911084 wah2_sam50_n1ni_201312_28_859_011975977_1 173,099 942,345 5.4440
02 May 2020 16:05:58 1484388 21911084 wah2_sam50_n1ni_201312_28_859_011975977_1 161,579 891,890 5.5198
29 Apr 2020 14:32:40 1484388 21911084 wah2_sam50_n1ni_201312_28_859_011975977_1 150,059 840,621 5.6019
24 Apr 2020 14:49:29 1484388 21911084 wah2_sam50_n1ni_201312_28_859_011975977_1 138,539 789,034 5.6954
21 Apr 2020 11:31:49 1484388 21911084 wah2_sam50_n1ni_201312_28_859_011975977_1 127,019 739,410 5.8213
17 Apr 2020 10:30:01 1484388 21911084 wah2_sam50_n1ni_201312_28_859_011975977_1 115,499 691,681 5.9886
09 Apr 2020 17:13:04 1484388 21911084 wah2_sam50_n1ni_201312_28_859_011975977_1 103,979 636,086 6.1174
04 Apr 2020 09:42:35 1484388 21911084 wah2_sam50_n1ni_201312_28_859_011975977_1 92,459 577,404 6.2450
01 Apr 2020 09:21:36 1484388 21911084 wah2_sam50_n1ni_201312_28_859_011975977_1 80,939 518,782 6.4095
01 Apr 2020 09:19:56 1484388 21911084 wah2_sam50_n1ni_201312_28_859_011975977_1 69,419 458,929 6.6110
18 Mar 2020 17:09:38 1484388 21911084 wah2_sam50_n1ni_201312_28_859_011975977_1 57,899 392,461 6.7784
09 Mar 2020 17:02:56 1484388 21911084 wah2_sam50_n1ni_201312_28_859_011975977_1 46,379 322,037 6.9436
01 Mar 2020 09:40:16 1484388 21911084 wah2_sam50_n1ni_201312_28_859_011975977_1 34,859 241,697 6.9336
17 Feb 2020 15:40:36 1484388 21911084 wah2_sam50_n1ni_201312_28_859_011975977_1 23,339 161,617 6.9248
10 Feb 2020 12:40:30 1484388 21911084 wah2_sam50_n1ni_201312_28_859_011975977_1 11,819 82,054 6.9426


©2024 climateprediction.net