Task 15936678

Name	hadcm3n_706f_1980_40_008411488_0
Workunit	8562344
Created	23 Aug 2013, 10:18:13 UTC
Sent	24 Aug 2013, 8:45:24 UTC
Report deadline	23 Nov 2013, 16:12:35 UTC
Received	25 Sep 2013, 2:50:23 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1285313
Run time	15 days 17 hours 5 min 16 sec
CPU time	1 min 16 sec
Validate state	Invalid
Credit	5,287.68
Device peak FLOPS	1.93 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>7.0.27</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: STWORK : Error in PP_FILE tmp/pipe_dummy 2048 BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: STWORK : Error in PP_FILE tmp/pipe_dummy 2048 forrtl: No space left on device forrtl: severe (38): error during write, unit 6, file /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_706f_1980_40_008411488/dataout/stdout_um.txt Image PC Routine Line Source hadcm3n_um_6.07_i 0848EB7D Unknown Unknown Unknown hadcm3n_um_6.07_i 0848D975 Unknown Unknown Unknown hadcm3n_um_6.07_i 0845F3CF Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F90D Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F257 Unknown Unknown Unknown hadcm3n_um_6.07_i 084554BF Unknown Unknown Unknown hadcm3n_um_6.07_i 0832327C Unknown Unknown Unknown hadcm3n_um_6.07_i 08198894 Unknown Unknown Unknown hadcm3n_um_6.07_i 08391957 Unknown Unknown Unknown hadcm3n_um_6.07_i 0838F8B7 Unknown Unknown Unknown hadcm3n_um_6.07_i 0839BDF8 Unknown Unknown Unknown libc.so.6 B7564E46 Unknown Unknown Unknown hadcm3n_um_6.07_i 0804CB11 Unknown Unknown Unknown Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=13308, iMonCtr=1 Model crash detected, will try to restart... BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITHEAD: I/O error tmp/pipe_dummy 2048 BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O EController:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=13308, iMonCtr=1 Model crash detected, will try to restart... BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITHEAD: I/O error tmp/pipe_dummy 2048 BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITHEAD: I/O error tmp/pipe_dummy 2048 BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 SETPOS: Seek Failed: No space left on device SETPOS: Unit 22 to Word Address 169984 Failed with Error Code -1 Model crashed: SETPOS: Unit 22 to Word Address 169984 Failed with Error Code -1 BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=13308, iMonCtr=1 Model crash detected, will try to restart... forrtl: No space left on device forrtl: severe (38): error during write, unit 6, file /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_706f_1980_40_008411488/dataout/stdout_um.txt Image PC Routine Line Source hadcm3n_um_6.07_i 0848EB7D Unknown Unknown Unknown hadcm3n_um_6.07_i 0848D975 Unknown Unknown Unknown hadcm3n_um_6.07_i 0845F3CF Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F90D Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F257 Unknown Unknown Unknown hadcm3n_um_6.07_i 08450E2C Unknown Unknown Unknown hadcm3n_um_6.07_i 0844E937 Unknown Unknown Unknown hadcm3n_um_6.07_i 0822F5BB Unknown Unknown Unknown hadcm3n_um_6.07_i 0818A767 Unknown Unknown Unknown hadcm3n_um_6.07_i 0818D749 Unknown Unknown Unknown hadcm3n_um_6.07_i 08391957 Unknown Unknown Unknown hadcm3n_um_6.07_i 0838F8B7 Unknown Unknown Unknown hadcm3n_um_6.07_i 0839BDF8 Unknown Unknown Unknown libc.so.6 B75B4E46 Unknown Unknown Unknown hadcm3n_um_6.07_i 0804CB11 Unknown Unknown Unknown Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=13308, iMonCtr=1 Model crash detected, will try to restart... forrtl: No space left on device forrtl: severe (38): error during write, unit 6, file /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_706f_1980_40_008411488/dataout/stdout_um.txt Image PC Routine Line Source hadcm3n_um_6.07_i 0848EB7D Unknown Unknown Unknown hadcm3n_um_6.07_i 0848D975 Unknown Unknown Unknown hadcm3n_um_6.07_i 0845F3CF Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F90D Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F257 Unknown Unknown Unknown hadcm3n_um_6.07_i 08450E2C Unknown Unknown Unknown hadcm3n_um_6.07_i 0844E937 Unknown Unknown Unknown hadcm3n_um_6.07_i 0822F5BB Unknown Unknown Unknown hadcm3n_um_6.07_i 0818A767 Unknown Unknown Unknown hadcm3n_um_6.07_i 0818D749 Unknown Unknown Unknown hadcm3n_um_6.07_i 08391957 Unknown Unknown Unknown hadcm3n_um_6.07_i 0838F8B7 Unknown Unknown Unknown hadcm3n_um_6.07_i 0839BDF8 Unknown Unknown Unknown libc.so.6 B7648E46 Unknown Unknown Unknown hadcm3n_um_6.07_i 0804CB11 Unknown Unknown Unknown Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=13308, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
25 Sep 2013 09:24:45	1285313	15936678	hadcm3n_706f_1980_40_008411488_0	440,640	5,093	0.0116
24 Sep 2013 00:29:48	1285313	15936678	hadcm3n_706f_1980_40_008411488_0	414,720	1,079,812	2.6037
23 Sep 2013 09:53:08	1285313	15936678	hadcm3n_706f_1980_40_008411488_0	388,800	1,011,858	2.6025
23 Sep 2013 09:53:08	1285313	15936678	hadcm3n_706f_1980_40_008411488_0	362,880	944,142	2.6018
21 Sep 2013 05:49:37	1285313	15936678	hadcm3n_706f_1980_40_008411488_0	336,960	876,193	2.6003
20 Sep 2013 08:39:15	1285313	15936678	hadcm3n_706f_1980_40_008411488_0	311,040	808,828	2.6004
19 Sep 2013 10:57:51	1285313	15936678	hadcm3n_706f_1980_40_008411488_0	285,120	741,443	2.6005
18 Sep 2013 13:00:21	1285313	15936678	hadcm3n_706f_1980_40_008411488_0	259,200	673,962	2.6002
17 Sep 2013 15:43:10	1285313	15936678	hadcm3n_706f_1980_40_008411488_0	233,280	606,643	2.6005
16 Sep 2013 17:54:22	1285313	15936678	hadcm3n_706f_1980_40_008411488_0	207,360	539,251	2.6006
15 Sep 2013 19:11:59	1285313	15936678	hadcm3n_706f_1980_40_008411488_0	181,440	471,797	2.6003
14 Sep 2013 21:11:06	1285313	15936678	hadcm3n_706f_1980_40_008411488_0	155,520	404,718	2.6024
13 Sep 2013 19:58:34	1285313	15936678	hadcm3n_706f_1980_40_008411488_0	129,600	337,024	2.6005
12 Sep 2013 22:33:40	1285313	15936678	hadcm3n_706f_1980_40_008411488_0	103,680	269,545	2.5998
12 Sep 2013 00:51:18	1285313	15936678	hadcm3n_706f_1980_40_008411488_0	77,760	202,154	2.5997
11 Sep 2013 03:19:46	1285313	15936678	hadcm3n_706f_1980_40_008411488_0	51,840	134,868	2.6016
10 Sep 2013 05:38:15	1285313	15936678	hadcm3n_706f_1980_40_008411488_0	25,920	67,442	2.6019