climateprediction.net home page
Task 11555400

Task 11555400

Name famous_ulnm_1799_200_006660909_2
Workunit 6864281
Created 10 Jun 2010, 15:05:46 UTC
Sent 10 Jul 2010, 7:54:15 UTC
Report deadline 9 Oct 2010, 15:21:26 UTC
Received 1 Aug 2010, 18:22:33 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 22 (0x00000016) Unknown error code
Computer ID 992256
Run time 7 days 0 hours 21 min 52 sec
CPU time 6 days 13 hours 29 min 22 sec
Validate state Invalid
Credit 4,385.28
Device peak FLOPS 2.82 GFLOPS
Application version UK Met Office FAMOUS v6.11
windows_intelx86
Stderr
<core_client_version>6.10.18</core_client_version>
<![CDATA[
<message>
The device does not recognize the command. (0x16) - exit code 22 (0x16)
</message>
<stderr_txt>
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5540, selfPID=5540, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=364, selfPID=364, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6680, selfPID=6680, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6196, selfPID=6196, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3428, selfPID=3428, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6160, selfPID=6160, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6840, selfPID=6840, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7384, selfPID=7384, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6496, selfPID=6496, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7608, selfPID=7608, iMonCtr=1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 60 - Return code = 16

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 61 - Return code = 16

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 68 - Return code = 16

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 69 - Return code = 16
CPDN Monitor - Quit request from BOINC...
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=8028, selfPID=8028, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7644, selfPID=7644, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7124, selfPID=7124, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5156, selfPID=5156, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6096, selfPID=6096, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4176, selfPID=4176, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5908, selfPID=5908, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7488, selfPID=7488, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=1164, selfPID=1164, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7448, selfPID=7448, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6900, selfPID=6900, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6204, selfPID=6204, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5408, selfPID=5408, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6872, selfPID=6872, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=8208, selfPID=8208, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6568, selfPID=6568, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6488, selfPID=6488, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7004, selfPID=7004, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=8788, selfPID=8788, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=8676, selfPID=8676, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=1916, selfPID=1916, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6464, selfPID=6464, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6068, selfPID=6068, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6792, selfPID=6792, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7980, selfPID=7980, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6760, selfPID=6760, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=9144, selfPID=9144, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=9016, selfPID=9016, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=8960, selfPID=8960, iMonCtr=1
15:07:15 (8204): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
15:07:16 (8204): No heartbeat from core client for 30 sec - exiting
15:07:17 (8204): No heartbeat from core client for 30 sec - exiting
15:07:18 (8204): No heartbeat from core client for 30 sec - exiting
15:07:19 (8204): No heartbeat from core client for 30 sec - exiting
15:07:20 (8204): No heartbeat from core client for 30 sec - exiting
15:07:21 (8204): No heartbeat from core client for 30 sec - exiting
15:07:22 (8204): No heartbeat from core client for 30 sec - exiting
15:07:23 (8204): No heartbeat from core client for 30 sec - exiting
15:07:24 (8204): No heartbeat from core client for 30 sec - exiting
15:07:25 (8204): No heartbeat from core client for 30 sec - exiting
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6912, selfPID=6912, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=8156, selfPID=8156, iMonCtr=1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 60 - Return code = 16

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 61 - Return code = 16

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 68 - Return code = 16

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 69 - Return code = 16
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2384, selfPID=2384, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=8448, selfPID=8448, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7424, selfPID=7424, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7568, selfPID=7568, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=8216, selfPID=8216, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=824, selfPID=824, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=8992, selfPID=8992, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7212, selfPID=7212, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=8856, selfPID=8856, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6992, selfPID=6992, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5516, selfPID=5516, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=8424, selfPID=8424, iMonCtr=1
21:43:19 (4684): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
21:43:20 (4684): No heartbeat from core client for 30 sec - exiting
21:43:21 (4684): No heartbeat from core client for 30 sec - exiting
21:43:22 (4684): No heartbeat from core client for 30 sec - exiting
21:43:23 (4684): No heartbeat from core client for 30 sec - exiting
21:43:24 (4684): No heartbeat from core client for 30 sec - exiting
21:43:25 (4684): No heartbeat from core client for 30 sec - exiting
21:43:26 (4684): No heartbeat from core client for 30 sec - exiting
21:43:27 (4684): No heartbeat from core client for 30 sec - exiting
21:43:28 (4684): No heartbeat from core client for 30 sec - exiting
21:43:29 (4684): No heartbeat from core client for 30 sec - exiting
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2624, selfPID=2624, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5252, selfPID=5252, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4136, selfPID=4136, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5656, selfPID=5656, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5236, selfPID=5236, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5116, selfPID=5116, iMonCtr=1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 60 - Return code = 16

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 61 - Return code = 16

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 68 - Return code = 16

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 69 - Return code = 16
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5268, selfPID=5268, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5896, selfPID=5896, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5564, selfPID=5564, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6044, selfPID=6044, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4684, selfPID=4684, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2180, selfPID=2180, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3560, selfPID=3560, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6940, selfPID=6940, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7236, selfPID=7236, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4768, selfPID=4768, iMonCtr=1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 60 - Return code = 16

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 61 - Return code = 16

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 68 - Return code = 16

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 69 - Return code = 16
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3908, selfPID=3908, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6304, selfPID=6304, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7476, selfPID=7476, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7880, selfPID=7880, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=1468, selfPID=1468, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6680, selfPID=6680, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5948, selfPID=5948, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7644, selfPID=7644, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=8188, selfPID=8188, iMonCtr=1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 60 - Return code = 16

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 61 - Return code = 16

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 68 - Return code = 16

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 69 - Return code = 16
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=8052, selfPID=8052, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7380, selfPID=7380, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6852, selfPID=6852, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=8108, selfPID=8108, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6504, selfPID=6504, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5928, selfPID=5928, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
forrtl: severe (47): write to READONLY file, unit 6, file C:\ProgramData\BOINC\projects\climateprediction.net\famous_ulnm_1799_200_006660909\dataout\stdout_um.txt
Image              PC        Routine            Line        Source             
famous_um_6.11_wi  008846D2  Unknown               Unknown  Unknown

Stack trace terminated abnormally.
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7744, iMonCtr=1
Model crash detected, will try to restart...
forrtl: severe (47): write to READONLY file, unit 6, file C:\ProgramData\BOINC\projects\climateprediction.net\famous_ulnm_1799_200_006660909\dataout\stdout_um.txt
Image              PC        Routine            Line        Source             
famous_um_6.11_wi  008846D2  Unknown               Unknown  Unknown

Stack trace terminated abnormally.
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7744, iMonCtr=1
Model crash detected, will try to restart...
forrtl: severe (47): write to READONLY file, unit 6, file C:\ProgramData\BOINC\projects\climateprediction.net\famous_ulnm_1799_200_006660909\dataout\stdout_um.txt
Image              PC        Routine            Line        Source             
famous_um_6.11_wi  008846D2  Unknown               Unknown  Unknown

Stack trace terminated abnormally.
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7744, iMonCtr=1
Model crash detected, will try to restart...
forrtl: severe (47): write to READONLY file, unit 6, file C:\ProgramData\BOINC\projects\climateprediction.net\famous_ulnm_1799_200_006660909\dataout\stdout_um.txt
Image              PC        Routine            Line        Source             
famous_um_6.11_wi  008846D2  Unknown               Unknown  Unknown

Stack trace terminated abnormally.
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7744, iMonCtr=1
Model crash detected, will try to restart...
forrtl: severe (47): write to READONLY file, unit 6, file C:\ProgramData\BOINC\projects\climateprediction.net\famous_ulnm_1799_200_006660909\dataout\stdout_um.txt
Image              PC        Routine            Line        Source             
famous_um_6.11_wi  008846D2  Unknown               Unknown  Unknown

Stack trace terminated abnormally.
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7744, iMonCtr=1
Model crash detected, will try to restart...
forrtl: severe (47): write to READONLY file, unit 6, file C:\ProgramData\BOINC\projects\climateprediction.net\famous_ulnm_1799_200_006660909\dataout\stdout_um.txt
Image              PC        Routine            Line        Source             
famous_um_6.11_wi  008846D2  Unknown               Unknown  Unknown

Stack trace terminated abnormally.
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7744, iMonCtr=1
Model crash detected, will try to restart...
Sorry, too many model crashes! :-(
11:51:30 (7744): called boinc_finish
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7532, selfPID=7532, iMonCtr=1

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
01 Aug 2010 17:13:54 992256 11555400 famous_ulnm_1799_200_006660909_2 1,329,146 564,818 0.4249
01 Aug 2010 12:50:18 992256 11555400 famous_ulnm_1799_200_006660909_2 1,319,786 560,865 0.4250
01 Aug 2010 11:40:41 992256 11555400 famous_ulnm_1799_200_006660909_2 1,310,426 556,856 0.4249
01 Aug 2010 10:31:04 992256 11555400 famous_ulnm_1799_200_006660909_2 1,301,066 552,867 0.4249
01 Aug 2010 08:30:54 992256 11555400 famous_ulnm_1799_200_006660909_2 1,291,706 548,905 0.4249
01 Aug 2010 06:18:54 992256 11555400 famous_ulnm_1799_200_006660909_2 1,282,346 544,929 0.4249
01 Aug 2010 05:21:22 992256 11555400 famous_ulnm_1799_200_006660909_2 1,272,986 540,962 0.4250
01 Aug 2010 02:58:10 992256 11555400 famous_ulnm_1799_200_006660909_2 1,263,626 536,990 0.4250
01 Aug 2010 01:45:12 992256 11555400 famous_ulnm_1799_200_006660909_2 1,254,266 532,964 0.4249
01 Aug 2010 01:45:12 992256 11555400 famous_ulnm_1799_200_006660909_2 1,244,906 528,981 0.4249
31 Jul 2010 21:38:28 992256 11555400 famous_ulnm_1799_200_006660909_2 1,235,546 524,987 0.4249
31 Jul 2010 20:33:20 992256 11555400 famous_ulnm_1799_200_006660909_2 1,226,186 520,975 0.4249
31 Jul 2010 18:31:16 992256 11555400 famous_ulnm_1799_200_006660909_2 1,216,826 516,985 0.4249
31 Jul 2010 17:17:32 992256 11555400 famous_ulnm_1799_200_006660909_2 1,207,466 513,038 0.4249
31 Jul 2010 15:06:56 992256 11555400 famous_ulnm_1799_200_006660909_2 1,198,106 509,040 0.4249
31 Jul 2010 13:56:25 992256 11555400 famous_ulnm_1799_200_006660909_2 1,188,746 505,030 0.4248
31 Jul 2010 10:47:39 992256 11555400 famous_ulnm_1799_200_006660909_2 1,179,386 501,067 0.4249
31 Jul 2010 07:38:04 992256 11555400 famous_ulnm_1799_200_006660909_2 1,170,026 497,089 0.4249
31 Jul 2010 06:24:55 992256 11555400 famous_ulnm_1799_200_006660909_2 1,160,666 493,111 0.4249
31 Jul 2010 03:30:33 992256 11555400 famous_ulnm_1799_200_006660909_2 1,151,306 489,169 0.4249


©2024 climateprediction.net