climateprediction.net home page
Task 11726627

Task 11726627

Name famous_v5v9_1399_200_006693371_3
Workunit 6896624
Created 26 Aug 2010, 16:02:58 UTC
Sent 14 Dec 2010, 23:27:19 UTC
Report deadline 16 Mar 2011, 6:54:30 UTC
Received 11 Mar 2011, 14:55:56 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 22 (0x00000016) Unknown error code
Computer ID 1121181
Run time 6 days 22 hours 28 min 29 sec
CPU time 5 days 21 hours 31 min 50 sec
Validate state Invalid
Credit 1,822.10
Device peak FLOPS 1.60 GFLOPS
Application version UK Met Office FAMOUS v6.11
i686-pc-linux-gnu
Stderr
<core_client_version>6.10.58</core_client_version>
<![CDATA[
<message>
process exited with code 22 (0x16, -234)
</message>
<stderr_txt>
t from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
CPDN Monitor - Quit request from BOINC...

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
CPDN Monitor - Quit request from BOINC...

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 60 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 61 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 68 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 69 - Return code = 1

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Suspended CPDN Monitor - Suspend request from BOINC...

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
CPDN Monitor - Quit request from BOINC...

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
CPDN Monitor - Quit request from BOINC...
Signal 1 received, exiting...
 (4008): called boinc_finish
Signal 15 received, exiting...
 (4008): called boinc_finish
CPDN Monitor - Quit request from BOINC...

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
CPDN Monitor - Quit request from BOINC...

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
CPDN Monitor - Quit request from BOINC...

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
CPDN Monitor - Quit request from BOINC...

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 60 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 61 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 68 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 69 - Return code = 1

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
CPDN Monitor - Quit request from BOINC...
 (4061): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Signal 15 received, exiting...
 (6919): called boinc_finish

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
CPDN Monitor - Quit request from BOINC...

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 60 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 61 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 68 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 69 - Return code = 1

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

BUFFIN: Read Failed: Inappropriate ioctl for device
BUFFIN: C I/O Error feof - Unit 60 - Return code = 1

BUFFIN: Read Failed: Invalid argument
BUFFIN: C I/O Error feof - Unit 61 - Return code = 1

BUFFIN: Read Failed: Invalid argument
BUFFIN: C I/O Error feof - Unit 68 - Return code = 1

BUFFIN: Read Failed: Invalid argument
BUFFIN: C I/O Error feof - Unit 69 - Return code = 1

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
CPDN Monitor - Quit request from BOINC...

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
CPDN Monitor - Quit request from BOINC...

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
CPDN Monitor - Quit request from BOINC...

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
CPDN Monitor - Quit request from BOINC...

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
CPDN Monitor - Quit request from BOINC...

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
CPDN Monitor - Quit request from BOINC...

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
CPDN Monitor - Quit request from BOINC...

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
CPDN Monitor - Quit request from BOINC...
 (4199): Can't acquire lockfile (-154) - waiting 35s
 (4199): Can't acquire lockfile (-154) - exiting
 (4215): Can't acquire lockfile (-154) - waiting 35s
 (4215): Can't acquire lockfile (-154) - exiting
 (4292): Can't acquire lockfile (-154) - waiting 35s
 (4292): Can't acquire lockfile (-154) - exiting
 (4368): Can't acquire lockfile (-154) - waiting 35s
 (4368): Can't acquire lockfile (-154) - exiting
 (7632): Can't acquire lockfile (-154) - waiting 35s
 (7632): Can't acquire lockfile (-154) - exiting
 (7655): Can't acquire lockfile (-154) - waiting 35s
 (7655): Can't acquire lockfile (-154) - exiting
 (7657): Can't acquire lockfile (-154) - waiting 35s
 (7657): Can't acquire lockfile (-154) - exiting
 (7669): Can't acquire lockfile (-154) - waiting 35s
 (7669): Can't acquire lockfile (-154) - exiting
 (7709): Can't acquire lockfile (-154) - waiting 35s
 (7709): Can't acquire lockfile (-154) - exiting
 (7713): Can't acquire lockfile (-154) - waiting 35s
 (7713): Can't acquire lockfile (-154) - exiting
 (7738): Can't acquire lockfile (-154) - waiting 35s
 (7738): Can't acquire lockfile (-154) - exiting
 (7747): Can't acquire lockfile (-154) - waiting 35s
 (7747): Can't acquire lockfile (-154) - exiting
 (7760): Can't acquire lockfile (-154) - waiting 35s
 (7760): Can't acquire lockfile (-154) - exiting
 (9500): Can't acquire lockfile (-154) - waiting 35s
 (9500): Can't acquire lockfile (-154) - exiting
 (10122): Can't acquire lockfile (-154) - waiting 35s
 (10122): Can't acquire lockfile (-154) - exiting
 (10156): Can't acquire lockfile (-154) - waiting 35s
 (10156): Can't acquire lockfile (-154) - exiting
 (10172): Can't acquire lockfile (-154) - waiting 35s
 (10172): Can't acquire lockfile (-154) - exiting
 (10204): Can't acquire lockfile (-154) - waiting 35s
 (10204): Can't acquire lockfile (-154) - exiting
 (10207): Can't acquire lockfile (-154) - waiting 35s
 (10207): Can't acquire lockfile (-154) - exiting
 (10284): Can't acquire lockfile (-154) - waiting 35s
 (10284): Can't acquire lockfile (-154) - exiting
 (10316): Can't acquire lockfile (-154) - waiting 35s
 (10316): Can't acquire lockfile (-154) - exiting
 (10349): Can't acquire lockfile (-154) - waiting 35s
 (10349): Can't acquire lockfile (-154) - exiting
 (10373): Can't acquire lockfile (-154) - waiting 35s
 (10373): Can't acquire lockfile (-154) - exiting
 (10390): Can't acquire lockfile (-154) - waiting 35s
 (10390): Can't acquire lockfile (-154) - exiting
 (10438): Can't acquire lockfile (-154) - waiting 35s
 (10438): Can't acquire lockfile (-154) - exiting
 (10461): Can't acquire lockfile (-154) - waiting 35s
 (10461): Can't acquire lockfile (-154) - exiting
 (10471): Can't acquire lockfile (-154) - waiting 35s

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 60 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 61 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 68 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 69 - Return code = 1
 (10471): Can't acquire lockfile (-154) - exiting
 (10476): Can't acquire lockfile (-154) - waiting 35s
 (10476): Can't acquire lockfile (-154) - exiting
 (10485): Can't acquire lockfile (-154) - waiting 35s
 (10485): Can't acquire lockfile (-154) - exiting
 (10488): Can't acquire lockfile (-154) - waiting 35s
 (10488): Can't acquire lockfile (-154) - exiting
 (10492): Can't acquire lockfile (-154) - waiting 35s
 (10492): Can't acquire lockfile (-154) - exiting
 (10494): Can't acquire lockfile (-154) - waiting 35s
 (10494): Can't acquire lockfile (-154) - exiting
 (10520): Can't acquire lockfile (-154) - waiting 35s
 (10520): Can't acquire lockfile (-154) - exiting
 (10532): Can't acquire lockfile (-154) - waiting 35s
 (10532): Can't acquire lockfile (-154) - exiting
 (10565): Can't acquire lockfile (-154) - waiting 35s
 (10565): Can't acquire lockfile (-154) - exiting
 (10570): Can't acquire lockfile (-154) - waiting 35s
 (10570): Can't acquire lockfile (-154) - exiting
 (10617): Can't acquire lockfile (-154) - waiting 35s
 (10617): Can't acquire lockfile (-154) - exiting
 (10673): Can't acquire lockfile (-154) - waiting 35s

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

BUFFIN: Read Failed: Inappropriate ioctl for device
BUFFIN: C I/O Error feof - Unit 60 - Return code = 1

BUFFIN: Read Failed: Invalid argument
BUFFIN: C I/O Error feof - Unit 61 - Return code = 1

BUFFIN: Read Failed: Invalid argument
BUFFIN: C I/O Error feof - Unit 68 - Return code = 1

BUFFIN: Read Failed: Invalid argument
BUFFIN: C I/O Error feof - Unit 69 - Return code = 1
 (10673): Can't acquire lockfile (-154) - exiting
 (10682): Can't acquire lockfile (-154) - waiting 35s
 (10682): Can't acquire lockfile (-154) - exiting
 (10705): Can't acquire lockfile (-154) - waiting 35s
 (10705): Can't acquire lockfile (-154) - exiting
 (10727): Can't acquire lockfile (-154) - waiting 35s
 (10727): Can't acquire lockfile (-154) - exiting
 (10762): Can't acquire lockfile (-154) - waiting 35s
 (10762): Can't acquire lockfile (-154) - exiting
 (10765): Can't acquire lockfile (-154) - waiting 35s
 (10765): Can't acquire lockfile (-154) - exiting
 (10770): Can't acquire lockfile (-154) - waiting 35s
 (10770): Can't acquire lockfile (-154) - exiting
 (10774): Can't acquire lockfile (-154) - waiting 35s
 (10774): Can't acquire lockfile (-154) - exiting
 (10819): Can't acquire lockfile (-154) - waiting 35s
 (10819): Can't acquire lockfile (-154) - exiting
 (10874): Can't acquire lockfile (-154) - waiting 35s
 (10874): Can't acquire lockfile (-154) - exiting
 (10880): Can't acquire lockfile (-154) - waiting 35s
 (10880): Can't acquire lockfile (-154) - exiting
 (10890): Can't acquire lockfile (-154) - waiting 35s
 (10890): Can't acquire lockfile (-154) - exiting
 (10921): Can't acquire lockfile (-154) - waiting 35s
 (10921): Can't acquire lockfile (-154) - exiting
 (10970): Can't acquire lockfile (-154) - waiting 35s
 (10970): Can't acquire lockfile (-154) - exiting
 (10987): Can't acquire lockfile (-154) - waiting 35s
 (10987): Can't acquire lockfile (-154) - exiting
 (10989): Can't acquire lockfile (-154) - waiting 35s
 (10989): Can't acquire lockfile (-154) - exiting
 (11036): Can't acquire lockfile (-154) - waiting 35s
 (11036): Can't acquire lockfile (-154) - exiting
 (11053): Can't acquire lockfile (-154) - waiting 35s
 (11053): Can't acquire lockfile (-154) - exiting
 (11062): Can't acquire lockfile (-154) - waiting 35s
 (11062): Can't acquire lockfile (-154) - exiting
 (11065): Can't acquire lockfile (-154) - waiting 35s
 (11065): Can't acquire lockfile (-154) - exiting
 (11072): Can't acquire lockfile (-154) - waiting 35s
 (11072): Can't acquire lockfile (-154) - exiting
 (11086): Can't acquire lockfile (-154) - waiting 35s
 (11086): Can't acquire lockfile (-154) - exiting
 (11099): Can't acquire lockfile (-154) - waiting 35s
 (11099): Can't acquire lockfile (-154) - exiting
 (11581): Can't acquire lockfile (-154) - waiting 35s
 (11581): Can't acquire lockfile (-154) - exiting
 (11774): Can't acquire lockfile (-154) - waiting 35s
 (11774): Can't acquire lockfile (-154) - exiting
 (11776): Can't acquire lockfile (-154) - waiting 35s
 (11776): Can't acquire lockfile (-154) - exiting
 (11844): Can't acquire lockfile (-154) - waiting 35s
 (11844): Can't acquire lockfile (-154) - exiting
 (11860): Can't acquire lockfile (-154) - waiting 35s
 (11860): Can't acquire lockfile (-154) - exiting
 (11862): Can't acquire lockfile (-154) - waiting 35s
 (11862): Can't acquire lockfile (-154) - exiting
 (11864): Can't acquire lockfile (-154) - waiting 35s
 (11864): Can't acquire lockfile (-154) - exiting
 (11867): Can't acquire lockfile (-154) - waiting 35s
 (11867): Can't acquire lockfile (-154) - exiting
 (11869): Can't acquire lockfile (-154) - waiting 35s
 (11869): Can't acquire lockfile (-154) - exiting
 (11871): Can't acquire lockfile (-154) - waiting 35s
 (11871): Can't acquire lockfile (-154) - exiting
 (11873): Can't acquire lockfile (-154) - waiting 35s
 (11873): Can't acquire lockfile (-154) - exiting
 (11875): Can't acquire lockfile (-154) - waiting 35s
 (11875): Can't acquire lockfile (-154) - exiting
 (11877): Can't acquire lockfile (-154) - waiting 35s
 (11877): Can't acquire lockfile (-154) - exiting
 (11879): Can't acquire lockfile (-154) - waiting 35s
 (11879): Can't acquire lockfile (-154) - exiting
 (11881): Can't acquire lockfile (-154) - waiting 35s
 (11881): Can't acquire lockfile (-154) - exiting
 (11905): Can't acquire lockfile (-154) - waiting 35s
 (11905): Can't acquire lockfile (-154) - exiting
 (11907): Can't acquire lockfile (-154) - waiting 35s
 (11907): Can't acquire lockfile (-154) - exiting
 (11909): Can't acquire lockfile (-154) - waiting 35s
 (11909): Can't acquire lockfile (-154) - exiting
 (11914): Can't acquire lockfile (-154) - waiting 35s
 (11914): Can't acquire lockfile (-154) - exiting
 (11917): Can't acquire lockfile (-154) - waiting 35s
 (11917): Can't acquire lockfile (-154) - exiting
 (11919): Can't acquire lockfile (-154) - waiting 35s
 (11919): Can't acquire lockfile (-154) - exiting
 (11924): Can't acquire lockfile (-154) - waiting 35s
 (11924): Can't acquire lockfile (-154) - exiting
 (11926): Can't acquire lockfile (-154) - waiting 35s
 (11926): Can't acquire lockfile (-154) - exiting
 (11928): Can't acquire lockfile (-154) - waiting 35s
 (11928): Can't acquire lockfile (-154) - exiting
 (11930): Can't acquire lockfile (-154) - waiting 35s
 (11930): Can't acquire lockfile (-154) - exiting
 (11932): Can't acquire lockfile (-154) - waiting 35s
 (11932): Can't acquire lockfile (-154) - exiting
 (11934): Can't acquire lockfile (-154) - waiting 35s
 (11934): Can't acquire lockfile (-154) - exiting
 (11936): Can't acquire lockfile (-154) - waiting 35s
 (11936): Can't acquire lockfile (-154) - exiting
 (11940): Can't acquire lockfile (-154) - waiting 35s
 (11940): Can't acquire lockfile (-154) - exiting
 (11943): Can't acquire lockfile (-154) - waiting 35s
 (11943): Can't acquire lockfile (-154) - exiting
 (11952): Can't acquire lockfile (-154) - waiting 35s
 (11952): Can't acquire lockfile (-154) - exiting
 (11955): Can't acquire lockfile (-154) - waiting 35s
 (11955): Can't acquire lockfile (-154) - exiting
 (11961): Can't acquire lockfile (-154) - waiting 35s
 (11961): Can't acquire lockfile (-154) - exiting
 (11972): Can't acquire lockfile (-154) - waiting 35s
 (11972): Can't acquire lockfile (-154) - exiting
 (11978): Can't acquire lockfile (-154) - waiting 35s
 (11978): Can't acquire lockfile (-154) - exiting
 (11980): Can't acquire lockfile (-154) - waiting 35s
 (11980): Can't acquire lockfile (-154) - exiting
 (11988): Can't acquire lockfile (-154) - waiting 35s
 (11988): Can't acquire lockfile (-154) - exiting
 (11990): Can't acquire lockfile (-154) - waiting 35s
 (11990): Can't acquire lockfile (-154) - exiting
 (11998): Can't acquire lockfile (-154) - waiting 35s
 (11998): Can't acquire lockfile (-154) - exiting
 (12002): Can't acquire lockfile (-154) - waiting 35s
 (12002): Can't acquire lockfile (-154) - exiting
 (12008): Can't acquire lockfile (-154) - waiting 35s
 (12008): Can't acquire lockfile (-154) - exiting
 (12037): Can't acquire lockfile (-154) - waiting 35s
 (12037): Can't acquire lockfile (-154) - exiting
 (12039): Can't acquire lockfile (-154) - waiting 35s
 (12039): Can't acquire lockfile (-154) - exiting
 (12045): Can't acquire lockfile (-154) - waiting 35s
 (12045): Can't acquire lockfile (-154) - exiting
 (12048): Can't acquire lockfile (-154) - waiting 35s
 (12048): Can't acquire lockfile (-154) - exiting
 (12051): Can't acquire lockfile (-154) - waiting 35s
 (12051): Can't acquire lockfile (-154) - exiting
 (12053): Can't acquire lockfile (-154) - waiting 35s
 (12053): Can't acquire lockfile (-154) - exiting
 (12060): Can't acquire lockfile (-154) - waiting 35s
 (12060): Can't acquire lockfile (-154) - exiting
 (12063): Can't acquire lockfile (-154) - waiting 35s
 (12063): Can't acquire lockfile (-154) - exiting
 (12067): Can't acquire lockfile (-154) - waiting 35s
 (12067): Can't acquire lockfile (-154) - exiting
 (12077): Can't acquire lockfile (-154) - waiting 35s
 (12077): Can't acquire lockfile (-154) - exiting
 (12080): Can't acquire lockfile (-154) - waiting 35s
 (12080): Can't acquire lockfile (-154) - exiting
 (12084): Can't acquire lockfile (-154) - waiting 35s
 (12084): Can't acquire lockfile (-154) - exiting
 (12089): Can't acquire lockfile (-154) - waiting 35s
 (12089): Can't acquire lockfile (-154) - exiting
 (12091): Can't acquire lockfile (-154) - waiting 35s
 (12091): Can't acquire lockfile (-154) - exiting
 (12093): Can't acquire lockfile (-154) - waiting 35s
 (12093): Can't acquire lockfile (-154) - exiting
 (12100): Can't acquire lockfile (-154) - waiting 35s
 (12100): Can't acquire lockfile (-154) - exiting
 (12102): Can't acquire lockfile (-154) - waiting 35s
 (12102): Can't acquire lockfile (-154) - exiting
 (12105): Can't acquire lockfile (-154) - waiting 35s
 (12105): Can't acquire lockfile (-154) - exiting
 (12110): Can't acquire lockfile (-154) - waiting 35s
 (12110): Can't acquire lockfile (-154) - exiting
 (12114): Can't acquire lockfile (-154) - waiting 35s
 (12114): Can't acquire lockfile (-154) - exiting
 (12116): Can't acquire lockfile (-154) - waiting 35s
 (12116): Can't acquire lockfile (-154) - exiting
 (12118): Can't acquire lockfile (-154) - waiting 35s
 (12118): Can't acquire lockfile (-154) - exiting
 (12126): Can't acquire lockfile (-154) - waiting 35s
 (12126): Can't acquire lockfile (-154) - exiting
 (12129): Can't acquire lockfile (-154) - waiting 35s
 (12129): Can't acquire lockfile (-154) - exiting
 (12131): Can't acquire lockfile (-154) - waiting 35s
 (12131): Can't acquire lockfile (-154) - exiting
 (12156): Can't acquire lockfile (-154) - waiting 35s
 (12156): Can't acquire lockfile (-154) - exiting
 (12161): Can't acquire lockfile (-154) - waiting 35s
 (12161): Can't acquire lockfile (-154) - exiting
 (12163): Can't acquire lockfile (-154) - waiting 35s
 (12163): Can't acquire lockfile (-154) - exiting
 (12166): Can't acquire lockfile (-154) - waiting 35s
 (12166): Can't acquire lockfile (-154) - exiting
 (12174): Can't acquire lockfile (-154) - waiting 35s
 (12174): Can't acquire lockfile (-154) - exiting
 (12177): Can't acquire lockfile (-154) - waiting 35s
 (12177): Can't acquire lockfile (-154) - exiting
 (12181): Can't acquire lockfile (-154) - waiting 35s
 (12181): Can't acquire lockfile (-154) - exiting
 (12185): Can't acquire lockfile (-154) - waiting 35s
 (12185): Can't acquire lockfile (-154) - exiting
 (12187): Can't acquire lockfile (-154) - waiting 35s
 (12187): Can't acquire lockfile (-154) - exiting
 (12191): Can't acquire lockfile (-154) - waiting 35s
 (12191): Can't acquire lockfile (-154) - exiting
 (12193): Can't acquire lockfile (-154) - waiting 35s
 (12193): Can't acquire lockfile (-154) - exiting
 (12200): Can't acquire lockfile (-154) - waiting 35s
 (12200): Can't acquire lockfile (-154) - exiting
 (12202): Can't acquire lockfile (-154) - waiting 35s
 (12202): Can't acquire lockfile (-154) - exiting
 (12204): Can't acquire lockfile (-154) - waiting 35s
 (12204): Can't acquire lockfile (-154) - exiting
 (12209): Can't acquire lockfile (-154) - waiting 35s
 (12209): Can't acquire lockfile (-154) - exiting
 (12211): Can't acquire lockfile (-154) - waiting 35s
 (12211): Can't acquire lockfile (-154) - exiting
 (12213): Can't acquire lockfile (-154) - waiting 35s
 (12213): Can't acquire lockfile (-154) - exiting
 (12215): Can't acquire lockfile (-154) - waiting 35s
 (12215): Can't acquire lockfile (-154) - exiting
 (12223): Can't acquire lockfile (-154) - waiting 35s
 (12223): Can't acquire lockfile (-154) - exiting
 (12225): Can't acquire lockfile (-154) - waiting 35s
 (12225): Can't acquire lockfile (-154) - exiting
 (12229): Can't acquire lockfile (-154) - waiting 35s
 (12229): Can't acquire lockfile (-154) - exiting
 (12241): Can't acquire lockfile (-154) - waiting 35s
 (12241): Can't acquire lockfile (-154) - exiting
 (12243): Can't acquire lockfile (-154) - waiting 35s
 (12243): Can't acquire lockfile (-154) - exiting
 (12245): Can't acquire lockfile (-154) - waiting 35s
 (12245): Can't acquire lockfile (-154) - exiting
 (12256): Can't acquire lockfile (-154) - waiting 35s
 (12256): Can't acquire lockfile (-154) - exiting
 (12258): Can't acquire lockfile (-154) - waiting 35s
 (12258): Can't acquire lockfile (-154) - exiting
 (12282): Can't acquire lockfile (-154) - waiting 35s
 (12282): Can't acquire lockfile (-154) - exiting
 (12284): Can't acquire lockfile (-154) - waiting 35s
 (12284): Can't acquire lockfile (-154) - exiting
 (12290): Can't acquire lockfile (-154) - waiting 35s
 (12290): Can't acquire lockfile (-154) - exiting
 (12292): Can't acquire lockfile (-154) - waiting 35s
 (12292): Can't acquire lockfile (-154) - exiting
 (12294): Can't acquire lockfile (-154) - waiting 35s
 (12294): Can't acquire lockfile (-154) - exiting
 (12306): Can't acquire lockfile (-154) - waiting 35s
 (12306): Can't acquire lockfile (-154) - exiting
 (12308): Can't acquire lockfile (-154) - waiting 35s
 (12308): Can't acquire lockfile (-154) - exiting
 (12311): Can't acquire lockfile (-154) - waiting 35s
 (12311): Can't acquire lockfile (-154) - exiting
 (12313): Can't acquire lockfile (-154) - waiting 35s
 (12313): Can't acquire lockfile (-154) - exiting
 (12320): Can't acquire lockfile (-154) - waiting 35s
 (12320): Can't acquire lockfile (-154) - exiting
 (12324): Can't acquire lockfile (-154) - waiting 35s
 (12324): Can't acquire lockfile (-154) - exiting
 (12326): Can't acquire lockfile (-154) - waiting 35s
 (12326): Can't acquire lockfile (-154) - exiting
 (12335): Can't acquire lockfile (-154) - waiting 35s
 (12335): Can't acquire lockfile (-154) - exiting
 (12346): Can't acquire lockfile (-154) - waiting 35s
 (12346): Can't acquire lockfile (-154) - exiting
 (12348): Can't acquire lockfile (-154) - waiting 35s
 (12348): Can't acquire lockfile (-154) - exiting
 (12350): Can't acquire lockfile (-154) - waiting 35s
 (12350): Can't acquire lockfile (-154) - exiting
 (12360): Can't acquire lockfile (-154) - waiting 35s
 (12360): Can't acquire lockfile (-154) - exiting
 (12366): Can't acquire lockfile (-154) - waiting 35s
 (12366): Can't acquire lockfile (-154) - exiting
 (12368): Can't acquire lockfile (-154) - waiting 35s
 (12368): Can't acquire lockfile (-154) - exiting
 (12375): Can't acquire lockfile (-154) - waiting 35s
 (12375): Can't acquire lockfile (-154) - exiting
 (12377): Can't acquire lockfile (-154) - waiting 35s
 (12377): Can't acquire lockfile (-154) - exiting
 (12386): Can't acquire lockfile (-154) - waiting 35s
 (12386): Can't acquire lockfile (-154) - exiting
 (12388): Can't acquire lockfile (-154) - waiting 35s
 (12388): Can't acquire lockfile (-154) - exiting
 (12391): Can't acquire lockfile (-154) - waiting 35s
 (12391): Can't acquire lockfile (-154) - exiting
 (12396): Can't acquire lockfile (-154) - waiting 35s
 (12396): Can't acquire lockfile (-154) - exiting
 (12421): Can't acquire lockfile (-154) - waiting 35s
 (12421): Can't acquire lockfile (-154) - exiting
 (12425): Can't acquire lockfile (-154) - waiting 35s
 (12425): Can't acquire lockfile (-154) - exiting
 (12427): Can't acquire lockfile (-154) - waiting 35s
 (12427): Can't acquire lockfile (-154) - exiting
 (12433): Can't acquire lockfile (-154) - waiting 35s
 (12433): Can't acquire lockfile (-154) - exiting
 (12435): Can't acquire lockfile (-154) - waiting 35s
 (12435): Can't acquire lockfile (-154) - exiting
 (12439): Can't acquire lockfile (-154) - waiting 35s
 (12439): Can't acquire lockfile (-154) - exiting
 (12441): Can't acquire lockfile (-154) - waiting 35s
 (12441): Can't acquire lockfile (-154) - exiting
 (12444): Can't acquire lockfile (-154) - waiting 35s
 (12444): Can't acquire lockfile (-154) - exiting
 (12446): Can't acquire lockfile (-154) - waiting 35s
 (12446): Can't acquire lockfile (-154) - exiting
 (12448): Can't acquire lockfile (-154) - waiting 35s
 (12448): Can't acquire lockfile (-154) - exiting
 (12450): Can't acquire lockfile (-154) - waiting 35s
 (12450): Can't acquire lockfile (-154) - exiting
 (12452): Can't acquire lockfile (-154) - waiting 35s
 (12452): Can't acquire lockfile (-154) - exiting
 (12454): Can't acquire lockfile (-154) - waiting 35s
 (12454): Can't acquire lockfile (-154) - exiting
 (12456): Can't acquire lockfile (-154) - waiting 35s
 (12456): Can't acquire lockfile (-154) - exiting
 (12461): Can't acquire lockfile (-154) - waiting 35s
 (12461): Can't acquire lockfile (-154) - exiting
 (12463): Can't acquire lockfile (-154) - waiting 35s
 (12463): Can't acquire lockfile (-154) - exiting
 (12469): Can't acquire lockfile (-154) - waiting 35s
 (12469): Can't acquire lockfile (-154) - exiting
 (12472): Can't acquire lockfile (-154) - waiting 35s
 (12472): Can't acquire lockfile (-154) - exiting
 (12492): Can't acquire lockfile (-154) - waiting 35s
CPDN Monitor - Quit request from BOINC...

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Sorry, too many model crashes! :-(
 (6534): called boinc_finish

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
11 Mar 2011 14:57:25 1121181 11726627 famous_v5v9_1399_200_006693371_3 552,266 501,785 0.9086
11 Mar 2011 14:57:25 1121181 11726627 famous_v5v9_1399_200_006693371_3 542,906 493,302 0.9086
27 Feb 2011 19:28:03 1121181 11726627 famous_v5v9_1399_200_006693371_3 533,546 484,490 0.9081
27 Feb 2011 15:42:46 1121181 11726627 famous_v5v9_1399_200_006693371_3 524,186 475,707 0.9075
27 Feb 2011 15:42:46 1121181 11726627 famous_v5v9_1399_200_006693371_3 514,826 466,669 0.9065
26 Feb 2011 23:08:13 1121181 11726627 famous_v5v9_1399_200_006693371_3 505,466 458,496 0.9071
26 Feb 2011 20:22:32 1121181 11726627 famous_v5v9_1399_200_006693371_3 496,106 450,136 0.9073
20 Feb 2011 20:45:39 1121181 11726627 famous_v5v9_1399_200_006693371_3 486,746 441,529 0.9071
20 Feb 2011 19:29:03 1121181 11726627 famous_v5v9_1399_200_006693371_3 477,386 433,213 0.9075
20 Feb 2011 15:42:32 1121181 11726627 famous_v5v9_1399_200_006693371_3 468,026 425,042 0.9082
20 Feb 2011 12:04:51 1121181 11726627 famous_v5v9_1399_200_006693371_3 458,666 416,790 0.9087
12 Feb 2011 18:32:55 1121181 11726627 famous_v5v9_1399_200_006693371_3 449,306 408,145 0.9084
12 Feb 2011 13:52:52 1121181 11726627 famous_v5v9_1399_200_006693371_3 439,946 399,464 0.9080
11 Feb 2011 20:49:57 1121181 11726627 famous_v5v9_1399_200_006693371_3 430,586 390,440 0.9068
11 Feb 2011 19:08:40 1121181 11726627 famous_v5v9_1399_200_006693371_3 421,226 381,945 0.9067
11 Feb 2011 14:13:18 1121181 11726627 famous_v5v9_1399_200_006693371_3 411,866 373,064 0.9058
06 Feb 2011 13:07:04 1121181 11726627 famous_v5v9_1399_200_006693371_3 402,506 364,777 0.9063
05 Feb 2011 18:50:41 1121181 11726627 famous_v5v9_1399_200_006693371_3 393,146 356,289 0.9063
05 Feb 2011 15:05:42 1121181 11726627 famous_v5v9_1399_200_006693371_3 383,786 348,102 0.9070
05 Feb 2011 12:34:19 1121181 11726627 famous_v5v9_1399_200_006693371_3 374,426 339,753 0.9074


©2024 climateprediction.net