Task 13073597

Name	hadcm3n_t576_1940_40_007313122_2
Workunit	7510552
Created	4 Jul 2011, 11:14:03 UTC
Sent	4 Jul 2011, 11:29:28 UTC
Report deadline	3 Oct 2011, 18:56:39 UTC
Received	2 Aug 2011, 14:32:36 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	429009
Run time	13 days 12 hours 34 min 22 sec
CPU time	12 days 12 hours 16 min 17 sec
Validate state	Invalid
Credit	4,665.60
Device peak FLOPS	1.96 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>6.12.33</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> 07:34:08 (9034): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:28:44 (31671): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:29:11 (31671): No heartbeat from core client for 30 sec - exiting 07:56:28 (5982): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:15:08 (7094): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:15:39 (7094): No heartbeat from core client for 30 sec - exiting 07:15:40 (7094): No heartbeat from core client for 30 sec - exiting 07:15:41 (7094): No heartbeat from core client for 30 sec - exiting 07:15:42 (7094): No heartbeat from core client for 30 sec - exiting 07:15:43 (7094): No heartbeat from core client for 30 sec - exiting 07:15:44 (7094): No heartbeat from core client for 30 sec - exiting 07:15:45 (7094): No heartbeat from core client for 30 sec - exiting 07:15:46 (7094): No heartbeat from core client for 30 sec - exiting 07:15:53 (7094): No heartbeat from core client for 30 sec - exiting 07:15:54 (7094): No heartbeat from core client for 30 sec - exiting 07:15:55 (7094): No heartbeat from core client for 30 sec - exiting 07:15:56 (7094): No heartbeat from core client for 30 sec - exiting 07:15:57 (7094): No heartbeat from core client for 30 sec - exiting 07:16:00 (7094): No heartbeat from core client for 30 sec - exiting 07:16:01 (7094): No heartbeat from core client for 30 sec - exiting 07:16:02 (7094): No heartbeat from core client for 30 sec - exiting 07:16:06 (7094): No heartbeat from core client for 30 sec - exiting 07:16:07 (7094): No heartbeat from core client for 30 sec - exiting 07:16:08 (7094): No heartbeat from core client for 30 sec - exiting 07:16:15 (7094): No heartbeat from core client for 30 sec - exiting 07:16:16 (7094): No heartbeat from core client for 30 sec - exiting 07:21:48 (4710): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:22:12 (4710): No heartbeat from core client for 30 sec - exiting 07:22:20 (4710): No heartbeat from core client for 30 sec - exiting 07:20:13 (15692): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:34:47 (19855): No heartbeat from core client for 30 sec - exiting 07:35:48 (19855): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:32:48 (29011): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:32:50 (29011): No heartbeat from core client for 30 sec - exiting 07:22:46 (16719): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:13:06 (16376): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:13:21 (16376): No heartbeat from core client for 30 sec - exiting 12:13:29 (16376): No heartbeat from core client for 30 sec - exiting 12:13:42 (16376): No heartbeat from core client for 30 sec - exiting 12:13:43 (16376): No heartbeat from core client for 30 sec - exiting 10:46:23 (28298): No heartbeat from core client for 30 sec - exiting 10:46:30 (28298): No heartbeat from core client for 30 sec - exiting 10:46:31 (28298): No heartbeat from core client for 30 sec - exiting 10:46:32 (28298): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:46:33 (28298): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 17:06:18 (22174): No heartbeat from core client for 30 sec - exiting 17:07:09 (22174): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:07:10 (22174): No heartbeat from core client for 30 sec - exiting 17:07:11 (22174): No heartbeat from core client for 30 sec - exiting 17:07:27 (22174): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 07:32:07 (4545): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:32:13 (4545): No heartbeat from core client for 30 sec - exiting 07:32:14 (4545): No heartbeat from core client for 30 sec - exiting 07:32:15 (4545): No heartbeat from core client for 30 sec - exiting 07:32:16 (4545): No heartbeat from core client for 30 sec - exiting 07:32:17 (4545): No heartbeat from core client for 30 sec - exiting 21:27:01 (25334): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITHEAD: I/O error tmp/pipe_dummy 2048 forrtl: No space left on device forrtl: severe (38): error during write, unit 6, file /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_t576_1940_40_007313122/dataout/stdout_um.txt Image PC Routine Line Source hadcm3n_um_6.07_i 0848EB7D Unknown Unknown Unknown hadcm3n_um_6.07_i 0848D975 Unknown Unknown Unknown hadcm3n_um_6.07_i 0845F3CF Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F90D Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F257 Unknown Unknown Unknown hadcm3n_um_6.07_i 08451069 Unknown Unknown Unknown hadcm3n_um_6.07_i 0844E937 Unknown Unknown Unknown hadcm3n_um_6.07_i 0836D10D Unknown Unknown Unknown hadcm3n_um_6.07_i 082EB086 Unknown Unknown Unknown hadcm3n_um_6.07_i 0838F66D Unknown Unknown Unknown hadcm3n_um_6.07_i 0839BDF8 Unknown Unknown Unknown libc.so.6 F7542E46 Unknown Unknown Unknown hadcm3n_um_6.07_i 0804CB11 Unknown Unknown Unknown Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=31349, iMonCtr=1 Model crash detected, will try to restart... forrtl: No space left on device forrtl: severe (38): error during write, unit 6, file /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_t576_1940_40_007313122/dataout/stdout_um.txt Image PC Routine Line Source hadcm3n_um_6.07_i 0848EB7D Unknown Unknown Unknown hadcm3n_um_6.07_i 0848D975 Unknown Unknown Unknown hadcm3n_um_6.07_i 0845F3CF Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F90D Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F257 Unknown Unknown Unknown hadcm3n_um_6.07_i 08451069 Unknown Unknown Unknown hadcm3n_um_6.07_i 0844E937 Unknown Unknown Unknown hadcm3n_um_6.07_i 0836D10D Unknown Unknown Unknown hadcm3n_um_6.07_i 082EB086 Unknown Unknown Unknown hadcm3n_um_6.07_i 0838F66D Unknown Unknown Unknown hadcm3n_um_6.07_i 0839BDF8 Unknown Unknown Unknown libc.so.6 F753AE46 Unknown Unknown Unknown hadcm3n_um_6.07_i 0804CB11 Unknown Unknown Unknown Controller:: CPDN process is not running, exiting, bRetVaCalled boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
01 Aug 2011 12:53:12	429009	13073597	hadcm3n_t576_1940_40_007313122_2	388,800	1,057,001	2.7186
30 Jul 2011 12:48:18	429009	13073597	hadcm3n_t576_1940_40_007313122_2	362,880	985,851	2.7167
29 Jul 2011 08:06:46	429009	13073597	hadcm3n_t576_1940_40_007313122_2	336,960	916,558	2.7201
27 Jul 2011 23:07:05	429009	13073597	hadcm3n_t576_1940_40_007313122_2	311,040	846,218	2.7206
26 Jul 2011 20:22:08	429009	13073597	hadcm3n_t576_1940_40_007313122_2	285,120	775,143	2.7187
25 Jul 2011 22:54:41	429009	13073597	hadcm3n_t576_1940_40_007313122_2	259,200	704,837	2.7193
25 Jul 2011 17:22:52	429009	13073597	hadcm3n_t576_1940_40_007313122_2	233,280	634,857	2.7214
25 Jul 2011 16:08:42	429009	13073597	hadcm3n_t576_1940_40_007313122_2	207,360	564,265	2.7212
25 Jul 2011 13:22:38	429009	13073597	hadcm3n_t576_1940_40_007313122_2	181,440	493,635	2.7207
25 Jul 2011 13:22:38	429009	13073597	hadcm3n_t576_1940_40_007313122_2	155,520	423,761	2.7248
25 Jul 2011 13:22:37	429009	13073597	hadcm3n_t576_1940_40_007313122_2	129,600	352,772	2.7220
25 Jul 2011 13:22:36	429009	13073597	hadcm3n_t576_1940_40_007313122_2	103,680	282,280	2.7226
08 Jul 2011 14:55:56	429009	13073597	hadcm3n_t576_1940_40_007313122_2	77,760	210,508	2.7072
07 Jul 2011 17:48:00	429009	13073597	hadcm3n_t576_1940_40_007313122_2	51,840	140,756	2.7152
05 Jul 2011 17:25:31	429009	13073597	hadcm3n_t576_1940_40_007313122_2	25,920	69,865	2.6954