climateprediction.net home page
Task 11435089

Task 11435089

Name famous_u33h_1999_200_006636856_2
Workunit 6840228
Created 10 Jun 2010, 11:36:05 UTC
Sent 20 Jul 2010, 19:54:41 UTC
Report deadline 20 Oct 2010, 3:21:52 UTC
Received 8 Aug 2010, 12:42:47 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status -226 (0xFFFFFF1E) ERR_TOO_MANY_EXITS
Computer ID 590081
Run time 9 days 2 hours 21 min 34 sec
CPU time 5 days 23 hours 0 min 39 sec
Validate state Invalid
Credit 3,088.25
Device peak FLOPS 1.77 GFLOPS
Application version UK Met Office FAMOUS v6.11
windows_intelx86
Stderr
<core_client_version>6.10.56</core_client_version>
<![CDATA[
<message>
too many exit(0)s
</message>
<stderr_txt>
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6152, iMonCtr=1
Model crash detected, will try to restart...
19:31:16 (3892): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
19:32:37 (4516): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
20:13:47 (6124): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3268, iMonCtr=1
Model crash detected, will try to restart...
22:41:44 (5988): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
23:50:15 (3968): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
18:43:42 (3688): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
19:47:38 (4376): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
19:47:39 (4376): No heartbeat from core client for 30 sec - exiting
23:16:23 (6332): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 60 - Return code = 16

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 61 - Return code = 16

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 68 - Return code = 16

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 69 - Return code = 16
11:12:59 (4852): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 60 - Return code = 16

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 61 - Return code = 16

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 68 - Return code = 16

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 69 - Return code = 16
Suspended CPDN Monitor - Suspend request from BOINC...
23:20:22 (6196): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
23:20:23 (6196): No heartbeat from core client for 30 sec - exiting
02:23:28 (472): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
02:23:30 (472): No heartbeat from core client for 30 sec - exiting
11:27:30 (6352): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
11:27:32 (6352): No heartbeat from core client for 30 sec - exiting
20:43:54 (3824): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
20:43:55 (3824): No heartbeat from core client for 30 sec - exiting
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5328, iMonCtr=1
Model crash detected, will try to restart...
23:11:07 (2452): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 60 - Return code = 16

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 61 - Return code = 16

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 68 - Return code = 16

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 69 - Return code = 16
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3884, iMonCtr=1
Model crash detected, will try to restart...
19:30:26 (5956): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
20:30:27 (3780): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
18:48:50 (5124): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
18:48:52 (5124): No heartbeat from core client for 30 sec - exiting
18:50:49 (5772): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
18:53:25 (3232): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
21:55:41 (1932): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
21:55:43 (1932): No heartbeat from core client for 30 sec - exiting
21:55:44 (1932): No heartbeat from core client for 30 sec - exiting
22:24:37 (4752): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 60 - Return code = 16

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 61 - Return code = 16

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 68 - Return code = 16

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 69 - Return code = 16
22:25:34 (5184): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...

BUFFIN: Read Failed: Result too large
BUFFIN: C I/O Error feof - Unit 60 - Return code = 16

BUFFIN: Read Failed: Result too large
BUFFIN: C I/O Error feof - Unit 61 - Return code = 16

BUFFIN: Read Failed: Result too large
BUFFIN: C I/O Error feof - Unit 68 - Return code = 16

BUFFIN: Read Failed: Result too large
BUFFIN: C I/O Error feof - Unit 69 - Return code = 16
16:52:00 (4276): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
21:47:13 (2876): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
21:52:52 (2688): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
21:53:54 (7020): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
04:58:14 (6352): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
19:07:40 (7040): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
19:07:42 (7040): No heartbeat from core client for 30 sec - exiting
19:49:28 (6980): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 60 - Return code = 16

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 61 - Return code = 16

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 68 - Return code = 16

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 69 - Return code = 16
19:51:54 (1476): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...

BUFFIN: Read Failed: Result too large
BUFFIN: C I/O Error feof - Unit 60 - Return code = 16

BUFFIN: Read Failed: Result too large
BUFFIN: C I/O Error feof - Unit 61 - Return code = 16

BUFFIN: Read Failed: Result too large
BUFFIN: C I/O Error feof - Unit 68 - Return code = 16

BUFFIN: Read Failed: Result too large
BUFFIN: C I/O Error feof - Unit 69 - Return code = 16
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3920, iMonCtr=1
Model crash detected, will try to restart...
08:56:44 (2116): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
09:58:07 (4428): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
09:58:08 (4428): No heartbeat from core client for 30 sec - exiting
11:31:29 (3416): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
21:37:42 (4392): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
09:16:21 (4316): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 60 - Return code = 16

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 61 - Return code = 16

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 68 - Return code = 16

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 69 - Return code = 16
09:24:21 (1900): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
10:26:28 (5260): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
10:26:29 (5260): No heartbeat from core client for 30 sec - exiting
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
21:15:08 (1392): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
08:37:05 (6724): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
10:55:56 (2760): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
10:55:57 (2760): No heartbeat from core client for 30 sec - exiting
16:01:12 (6332): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
16:01:14 (6332): No heartbeat from core client for 30 sec - exiting
19:10:11 (6996): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
19:10:12 (6996): No heartbeat from core client for 30 sec - exiting
22:04:03 (6352): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
22:04:04 (6352): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - Quit request from BOINC...
05:42:10 (272): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
05:42:12 (272): No heartbeat from core client for 30 sec - exiting
09:56:12 (7028): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
09:56:14 (7028): No heartbeat from core client for 30 sec - exiting

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 60 - Return code = 16

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 61 - Return code = 16

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 68 - Return code = 16

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 69 - Return code = 16
10:18:35 (3272): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
10:18:36 (3272): No heartbeat from core client for 30 sec - exiting
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6384, iMonCtr=1
Model crash detected, will try to restart...
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3888, selfPID=3888, iMonCtr=1
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5512, selfPID=5512, iMonCtr=1
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5592, selfPID=5592, iMonCtr=1
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5668, selfPID=5668, iMonCtr=1
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5996, selfPID=5996, iMonCtr=1
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5496, selfPID=5496, iMonCtr=1
forrtl: severe (47): write to READONLY file, unit 6, file C:\ProgramData\BOINC\projects\climateprediction.net\famous_u33h_1999_200_006636856\dataout\stdout_um.txt

Image              PC        Routine            Line        Source             
famous_um_6.11_wi  008846D2  Unknown               Unknown  Unknown

Stack trace terminated abnormally.
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=860, iMonCtr=1
Model crash detected, will try to restart...
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6076, selfPID=6076, iMonCtr=1
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5980, selfPID=5980, iMonCtr=1
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5772, selfPID=5772, iMonCtr=1
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5788, selfPID=5788, iMonCtr=1
Could not launch model process. Last Error=5
forrtl: severe (24): end-of-file during read, unit 9, file C:\ProgramData\BOINC\projects\climateprediction.net\famous_u33h_1999_200_006636856\tmp\cp.namelists, line 1, position 0

Image              PC        Routine            Line        Source             
famous_um_6.11_wi  008846D2  Unknown               Unknown  Unknown

Stack trace terminated abnormally.
forrtl: severe (24): end-of-file during read, unit 9, file C:\ProgramData\BOINC\projects\climateprediction.net\famous_u33h_1999_200_006636856\tmp\cp.namelists, line 1, position 0

Image              PC        Routine            Line        Source             
famous_um_6.11_wi  008846D2  Unknown               Unknown  Unknown

Stack trace terminated abnormally.
forrtl: severe (24): end-of-file during read, unit 9, file C:\ProgramData\BOINC\projects\climateprediction.net\famous_u33h_1999_200_006636856\tmp\cp.namelists, line 1, position 0

Image              PC        Routine            Line        Source             
famous_um_6.11_wi  008846D2  Unknown               Unknown  Unknown

Stack trace terminated abnormally.
forrtl: severe (24): end-of-file during read, unit 9, file C:\ProgramData\BOINC\projects\climateprediction.net\famous_u33h_1999_200_006636856\tmp\cp.namelists, line 1, position 0

Image              PC        Routine            Line        Source             
famous_um_6.11_wi  008846D2  Unknown               Unknown  Unknown

Stack trace terminated abnormally.
forrtl: severe (24): end-of-file during read, unit 9, file C:\ProgramData\BOINC\projects\climateprediction.net\famous_u33h_1999_200_006636856\tmp\cp.namelists, line 1, position 0

Image              PC        Routine            Line        Source             
famous_um_6.11_wi  008846D2  Unknown               Unknown  Unknown

Stack trace terminated abnormally.
forrtl: severe (24): end-of-file during read, unit 9, file C:\ProgramData\BOINC\projects\climateprediction.net\famous_u33h_1999_200_006636856\tmp\cp.namelists, line 1, position 0

Image              PC        Routine            Line        Source             
famous_um_6.11_wi  008846D2  Unknown               Unknown  Unknown

Stack trace terminated abnormally.
forrtl: severe (24): end-of-file during read, unit 9, file C:\ProgramData\BOINC\projects\climateprediction.net\famous_u33h_1999_200_006636856\tmp\cp.namelists, line 1, position 0

Image              PC        Routine            Line        Source             
famous_um_6.11_wi  008846D2  Unknown               Unknown  Unknown

Stack trace terminated abnormally.
forrtl: severe (24): end-of-file during read, unit 9, file C:\ProgramData\BOINC\projects\climateprediction.net\famous_u33h_1999_200_006636856\tmp\cp.namelists, line 1, position 0

Image              PC        Routine            Line        Source             
famous_um_6.11_wi  008846D2  Unknown               Unknown  Unknown

Stack trace terminated abnormally.
forrtl: severe (24): end-of-file during read, unit 9, file C:\ProgramData\BOINC\projects\climateprediction.net\famous_u33h_1999_200_006636856\tmp\cp.namelists, line 1, position 0

Image              PC        Routine            Line        Source             
famous_um_6.11_wi  008846D2  Unknown               Unknown  Unknown

Stack trace terminated abnormally.
forrtl: severe (24): end-of-file during read, unit 9, file C:\ProgramData\BOINC\projects\climateprediction.net\famous_u33h_1999_200_006636856\tmp\cp.namelists, line 1, position 0

Image              PC        Routine            Line        Source             
famous_um_6.11_wi  008846D2  Unknown               Unknown  Unknown

Stack trace terminated abnormally.
forrtl: severe (24): end-of-file during read, unit 9, file C:\ProgramData\BOINC\projects\climateprediction.net\famous_u33h_1999_200_006636856\tmp\cp.namelists, line 1, position 0

Image              PC        Routine            Line        Source             
famous_um_6.11_wi  008846D2  Unknown               Unknown  Unknown

Stack trace terminated abnormally.
forrtl: severe (24): end-of-file during read, unit 9, file C:\ProgramData\BOINC\projects\climateprediction.net\famous_u33h_1999_200_006636856\tmp\cp.namelists, line 1, position 0

Image              PC        Routine            Line        Source             
famous_um_6.11_wi  008846D2  Unknown               Unknown  Unknown

Stack trace terminated abnormally.
forrtl: severe (24): end-of-file during read, unit 9, file C:\ProgramData\BOINC\projects\climateprediction.net\famous_u33h_1999_200_006636856\tmp\cp.namelists, line 1, position 0

Image              PC        Routine            Line        Source             
famous_um_6.11_wi  008846D2  Unknown               Unknown  Unknown

Stack trace terminated abnormally.
forrtl: severe (24): end-of-file during read, unit 9, file C:\ProgramData\BOINC\projects\climateprediction.net\famous_u33h_1999_200_006636856\tmp\cp.namelists, line 1, position 0

Image              PC        Routine            Line        Source             
famous_um_6.11_wi  008846D2  Unknown               Unknown  Unknown

Stack trace terminated abnormally.
forrtl: severe (24): end-of-file during read, unit 9, file C:\ProgramData\BOINC\projects\climateprediction.net\famous_u33h_1999_200_006636856\tmp\cp.namelists, line 1, position 0

Image              PC        Routine            Line        Source             
famous_um_6.11_wi  008846D2  Unknown               Unknown  Unknown

Stack trace terminated abnormally.
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5968, iMonCtr=1
Model crash detected, will try to restart...
forrtl: severe (24): end-of-file during read, unit 9, file C:\ProgramData\BOINC\projects\climateprediction.net\famous_u33h_1999_200_006636856\tmp\cp.namelists, line 1, position 0

Image              PC        Routine            Line        Source             
famous_um_6.11_wi  008846D2  Unknown               Unknown  Unknown

Stack trace terminated abnormally.
forrtl: severe (24): end-of-file during read, unit 9, file C:\ProgramData\BOINC\projects\climateprediction.net\famous_u33h_1999_200_006636856\tmp\cp.namelists, line 1, position 0

Image              PC        Routine            Line        Source             
famous_um_6.11_wi  008846D2  Unknown               Unknown  Unknown

Stack trace terminated abnormally.
forrtl: severe (24): end-of-file during read, unit 9, file C:\ProgramData\BOINC\projects\climateprediction.net\famous_u33h_1999_200_006636856\tmp\cp.namelists, line 1, position 0

Image              PC        Routine            Line        Source             
famous_um_6.11_wi  008846D2  Unknown               Unknown  Unknown

Stack trace terminated abnormally.
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5300, iMonCtr=1
Model crash detected, will try to restart...
forrtl: severe (24): end-of-file during read, unit 9, file C:\ProgramData\BOINC\projects\climateprediction.net\famous_u33h_1999_200_006636856\tmp\cp.namelists, line 1, position 0

Image              PC        Routine            Line        Source             
famous_um_6.11_wi  008846D2  Unknown               Unknown  Unknown

Stack trace terminated abnormally.
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6116, iMonCtr=1
Model crash detected, will try to restart...
forrtl: severe (24): end-of-file during read, unit 9, file C:\ProgramData\BOINC\projects\climateprediction.net\famous_u33h_1999_200_006636856\tmp\cp.namelists, line 1, position 0

Image              PC        Routine            Line        Source             
famous_um_6.11_wi  008846D2  Unknown               Unknown  Unknown

Stack trace terminated abnormally.
forrtl: severe (24): end-of-file during read, unit 9, file C:\ProgramData\BOINC\projects\climateprediction.net\famous_u33h_1999_200_006636856\tmp\cp.namelists, line 1, position 0

Image              PC        Routine            Line        Source             
famous_um_6.11_wi  008846D2  Unknown               Unknown  Unknown

Stack trace terminated abnormally.

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
08 Aug 2010 09:02:38 590081 11435089 famous_u33h_1999_200_006636856_2 936,026 513,389 0.5485
08 Aug 2010 06:50:51 590081 11435089 famous_u33h_1999_200_006636856_2 926,666 507,859 0.5480
08 Aug 2010 06:07:10 590081 11435089 famous_u33h_1999_200_006636856_2 917,306 502,370 0.5477
08 Aug 2010 03:33:58 590081 11435089 famous_u33h_1999_200_006636856_2 907,946 496,480 0.5468
07 Aug 2010 22:00:43 590081 11435089 famous_u33h_1999_200_006636856_2 898,586 490,486 0.5458
07 Aug 2010 18:59:00 590081 11435089 famous_u33h_1999_200_006636856_2 889,226 484,812 0.5452
07 Aug 2010 16:04:20 590081 11435089 famous_u33h_1999_200_006636856_2 879,866 479,059 0.5445
07 Aug 2010 13:35:41 590081 11435089 famous_u33h_1999_200_006636856_2 870,506 473,171 0.5436
07 Aug 2010 10:41:04 590081 11435089 famous_u33h_1999_200_006636856_2 861,146 467,239 0.5426
07 Aug 2010 06:51:50 590081 11435089 famous_u33h_1999_200_006636856_2 851,786 461,548 0.5419
07 Aug 2010 04:06:22 590081 11435089 famous_u33h_1999_200_006636856_2 842,426 455,915 0.5412
07 Aug 2010 01:54:46 590081 11435089 famous_u33h_1999_200_006636856_2 833,066 450,080 0.5403
06 Aug 2010 21:39:51 590081 11435089 famous_u33h_1999_200_006636856_2 823,706 444,114 0.5392
04 Aug 2010 05:48:52 590081 11435089 famous_u33h_1999_200_006636856_2 814,346 438,637 0.5386
04 Aug 2010 04:54:32 590081 11435089 famous_u33h_1999_200_006636856_2 804,986 433,738 0.5388
04 Aug 2010 04:54:32 590081 11435089 famous_u33h_1999_200_006636856_2 795,626 428,954 0.5391
04 Aug 2010 02:42:53 590081 11435089 famous_u33h_1999_200_006636856_2 786,266 424,104 0.5394
04 Aug 2010 02:26:57 590081 11435089 famous_u33h_1999_200_006636856_2 776,906 419,380 0.5398
03 Aug 2010 22:17:44 590081 11435089 famous_u33h_1999_200_006636856_2 767,546 414,566 0.5401
03 Aug 2010 20:51:44 590081 11435089 famous_u33h_1999_200_006636856_2 758,186 409,837 0.5405


©2024 climateprediction.net