Task 12854773

Name	hadcm3n_p1cj_1900_40_007219331_2
Workunit	7417571
Created	4 May 2011, 0:49:11 UTC
Sent	4 May 2011, 1:00:49 UTC
Report deadline	3 Aug 2011, 8:28:00 UTC
Received	15 Jun 2011, 17:44:13 UTC
Server state	Over
Outcome	Success
Client state	Done
Exit status	0 (0x00000000)
Computer ID	1073754
Run time	24 days 19 hours 59 min 45 sec
CPU time	19 days 6 hours 9 min 57 sec
Validate state	Valid
Credit	12,441.60
Device peak FLOPS	1.66 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>6.10.17</core_client_version> <![CDATA[ <stderr_txt> Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=11996, iMonCtr=1 Model crash detected, will try to restart... BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 forrtl: No space left on device forrtl: severe (38): error during write, unit -1, file stdout Image PC Routine Line Source hadcm3n_um_6.07_i 0848EB7D Unknown Unknown Unknown hadcm3n_um_6.07_i 0848D975 Unknown Unknown Unknown hadcm3n_um_6.07_i 0845F3CF Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F90D Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F257 Unknown Unknown Unknown hadcm3n_um_6.07_i 08450B99 Unknown Unknown Unknown hadcm3n_um_6.07_i 0844E937 Unknown Unknown Unknown hadcm3n_um_6.07_i 083009BD Unknown Unknown Unknown hadcm3n_um_6.07_i 0838EDB1 Unknown Unknown Unknown hadcm3n_um_6.07_i 0839BDF8 Unknown Unknown Unknown libc.so.6 F75DABD6 Unknown Unknown Unknown hadcm3n_um_6.07_i 0804CB11 Unknown Unknown Unknown Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=11996, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 forrtl: No space left on device forrtl: severe (38): error during write, unit -1, file stdout Image PC Routine Line Source hadcm3n_um_6.07_i 0848EB7D Unknown Unknown Unknown hadcm3n_um_6.07_i 0848D975 Unknown Unknown Unknown hadcm3n_um_6.07_i 0845F3CF Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F90D Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F257 Unknown Unknown Unknown hadcm3n_um_6.07_i 08450B99 Unknown Unknown Unknown hadcm3n_um_6.07_i 0844E937 Unknown Unknown Unknown hadcm3n_um_6.07_i 083009BD Unknown Unknown Unknown hadcm3n_um_6.07_i 0838EDB1 Unknown Unknown Unknown hadcm3n_um_6.07_i 0839BDF8 Unknown Unknown Unknown libc.so.6 F7642BD6 Unknown Unknown Unknown hadcm3n_um_6.07_i 0804CB11 Unknown Unknown Unknown Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=11996, iMonCtr=1 Model crash detected, will try to restart... BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 SETPOS: Seek Failed: No space left on device SETPOS: Unit 22 to Word Address 186368 Failed with Error Code -1 Model crashed: SETPOS: Unit 22 to Word Address 186368 Failed with Error Code -1 forrtl: No space left on device forrtl: severe (38): error during write, unit 6, file /apps/BOINC/projects/climateprediction.net/hadcm3n_p1cj_1900_40_007219331/dataout/stdout_um.txt Image PC Routine Line Source hadcm3n_um_6.07_i 0848EB7D Unknown Unknown Unknown hadcm3n_um_6.07_i 0848D975 UnSuspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 forrtl: No space left on device forrtl: severe (38): error during write, unit -1, file stdout Image PC Routine Line Source hadcm3n_um_6.07_i 0848EB7D Unknown Unknown Unknown hadcm3n_um_6.07_i 0848D975 Unknown Unknown Unknown hadcm3n_um_6.07_i 0845F3CF Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F90D Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F257 Unknown Unknown Unknown hadcm3n_um_6.07_i 08450B99 Unknown Unknown Unknown hadcm3n_um_6.07_i 0844E937 Unknown Unknown Unknown hadcm3n_um_6.07_i 083009BD Unknown Unknown Unknown hadcm3n_um_6.07_i 0838EDB1 Unknown Unknown Unknown hadcm3n_um_6.07_i 0839BDF8 Unknown Unknown Unknown libc.so.6 F7638BD6 Unknown Unknown Unknown hadcm3n_um_6.07_i 0804CB11 Unknown Unknown Unknown Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=11996, iMonCtr=1 Model crash detected, will try to restart... BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 forrtl: No space lefController:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=11996, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
15 Jun 2011 17:35:17	1073754	12854773	hadcm3n_p1cj_1900_40_007219331_2	1,036,800	1,662,959	1.6039
15 Jun 2011 17:35:15	1073754	12854773	hadcm3n_p1cj_1900_40_007219331_2	1,010,880	1,610,259	1.5929
15 Jun 2011 17:35:14	1073754	12854773	hadcm3n_p1cj_1900_40_007219331_2	984,960	1,555,638	1.5794
15 Jun 2011 17:35:16	1073754	12854773	hadcm3n_p1cj_1900_40_007219331_2	959,040	1,502,273	1.5664
15 Jun 2011 17:35:14	1073754	12854773	hadcm3n_p1cj_1900_40_007219331_2	933,120	1,450,787	1.5548
15 Jun 2011 17:35:16	1073754	12854773	hadcm3n_p1cj_1900_40_007219331_2	907,200	1,399,323	1.5425
15 Jun 2011 17:35:14	1073754	12854773	hadcm3n_p1cj_1900_40_007219331_2	881,280	1,347,824	1.5294
15 Jun 2011 17:35:14	1073754	12854773	hadcm3n_p1cj_1900_40_007219331_2	855,360	1,296,356	1.5156
15 Jun 2011 17:35:14	1073754	12854773	hadcm3n_p1cj_1900_40_007219331_2	829,440	1,244,910	1.5009
15 Jun 2011 17:35:14	1073754	12854773	hadcm3n_p1cj_1900_40_007219331_2	803,520	1,193,418	1.4852
15 Jun 2011 17:35:16	1073754	12854773	hadcm3n_p1cj_1900_40_007219331_2	777,600	1,141,948	1.4686
15 Jun 2011 17:35:16	1073754	12854773	hadcm3n_p1cj_1900_40_007219331_2	751,680	1,090,462	1.4507
15 Jun 2011 17:35:15	1073754	12854773	hadcm3n_p1cj_1900_40_007219331_2	725,760	1,038,995	1.4316
15 Jun 2011 17:35:17	1073754	12854773	hadcm3n_p1cj_1900_40_007219331_2	699,840	987,465	1.4110
15 Jun 2011 17:35:15	1073754	12854773	hadcm3n_p1cj_1900_40_007219331_2	673,920	935,947	1.3888
15 Jun 2011 17:35:14	1073754	12854773	hadcm3n_p1cj_1900_40_007219331_2	648,000	884,420	1.3648
15 Jun 2011 17:35:14	1073754	12854773	hadcm3n_p1cj_1900_40_007219331_2	622,080	832,913	1.3389
15 Jun 2011 17:35:16	1073754	12854773	hadcm3n_p1cj_1900_40_007219331_2	596,160	781,299	1.3106
15 Jun 2011 17:35:14	1073754	12854773	hadcm3n_p1cj_1900_40_007219331_2	570,240	729,697	1.2796
15 Jun 2011 17:35:17	1073754	12854773	hadcm3n_p1cj_1900_40_007219331_2	544,320	678,146	1.2459