Task 16144383

Name	hadcm3n_7y9m_1980_40_008455693_2
Workunit	8606549
Created	13 Dec 2013, 21:02:24 UTC
Sent	13 Dec 2013, 21:02:34 UTC
Report deadline	15 Mar 2014, 4:29:45 UTC
Received	20 Jan 2014, 21:24:22 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1143126
Run time	26 days 18 hours 46 min 5 sec
CPU time	11 days 20 hours 44 min 17 sec
Validate state	Invalid
Credit	7,153.92
Device peak FLOPS	3.22 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 Error: Input file: dataout/7y9mko.pji3c10 is not a valid UM file. Error converting file to netcdf: dataout/7y9mko.pji3c10 Error: Input file: dataout/7y9mko.pii3c10 is not a valid UM file. Error converting file to netcdf: dataout/7y9mko.pii3c10 Error: Input file: dataout/7y9mko.pfi3c10 is not a valid UM file. Error converting file to netcdf: dataout/7y9mko.pfi3c10 Error: Input file: dataout/7y9mka.phi3c10 is not a valid UM file. Error converting file to netcdf: dataout/7y9mka.phi3c10 Error: Input file: dataout/7y9mka.pgi3c10 is not a valid UM file. Error converting file to netcdf: dataout/7y9mka.pgi3c10 Error: Input file: dataout/7y9mka.pei3c10 is not a valid UM file. Error converting file to netcdf: dataout/7y9mka.pei3c10 Error: Input file: dataout/7y9mka.pdi3c10 is not a valid UM file. Error converting file to netcdf: dataout/7y9mka.pdi3c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - ReSuspended CPDN Monitor - Suspend request from BOINC... BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 23:27:54 (86920): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:29:12 (97199): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 forrtl: No space left on device forrtl: severe (38): error during write, unit 6, file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_7y9m_1980_40_008455693/dataout/stdout_um.txt Image PC Routine Line Source hadcm3n_um_6.07_i 0042E622 Unknown Unknown Unknown hadcm3n_um_6.07_i 0042D0EB Unknown Unknown Unknown hadcm3n_um_6.07_i 003EF4DD Unknown Unknown Unknown hadcm3n_um_6.07_i 003A66F5 Unknown Unknown Unknown hadcm3n_um_6.07_i 003A5E57 Unknown Unknown Unknown hadcm3n_um_6.07_i 003E1BF7 Unknown Unknown Unknown hadcm3n_um_6.07_i 003DF4CC Unknown Unknown Unknown hadcm3n_um_6.07_i 0020BEDF Unknown Unknown Unknown hadcm3n_um_6.07_i 00160E7A Unknown Unknown Unknown hadcm3n_um_6.07_i 00164453 Unknown Unknown Unknown hadcm3n_um_6.07_i 0038CE7D Unknown Unknown Unknown hadcm3n_um_6.07_i 0038A1ED Unknown Unknown Unknown hadcm3n_um_6.07_i 000048E1 Unknown Unknown Unknown hadcm3n_um_6.07_i 000027FB Unknown Unknown Unknown hadcm3n_um_6.07_i 00002729 Unknown Unknown Unknown Unknown 00000003 Unknown Unknown Unknown Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=20309, iMonCtr=1 Model crash detected, will try to restart... forrtl: No space left on device forrtl: severe (38): error during write, unit 6, file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_7y9m_1980_40_008455693/dataout/stdout_um.txt Image PC Routine Line Source hadcm3n_um_6.07_i 0042E622 Unknown Unknown Unknown hadcm3n_um_6.07_i 0042D0EB Unknown Unknown Unknown hadcm3n_um_6.07_i 003EF4DD Unknown Unknown Unknown hadcm3n_um_6.07_i 003A66F5 Unknown Unknown Unknown hadcm3n_um_6.07_i 003A5E57 Unknown Unknown Unknown hadcm3n_um_6.07_i 003E1BF7 Unknown Unknown Unknown hadcm3n_um_6.07_i 003DF4CC Unknown Unknown Unknown hadcm3n_um_6.07_i 0020BEDF Unknown Unknown Unknown hadcm3n_um_6.07_i 00160E7A Unknown Unknown Unknown hadcm3n_um_6.07_i 00164453 Unknown Unknown Unknown hadcm3n_um_6.07_i 0038CE7D Unknown Unknown Unknown hadcm3n_um_6.07_i 0038A1ED Unknown Unknown Unknown hadcm3n_um_6.07_i 000048E1 Unknown Unknown Unknown hadcm3n_um_6.07_i 000027FB Unknown Unknown Unknown hadcm3n_um_6.07_i 00002729 Unknown Unknown Unknown Unknown 00000003 Unknown Unknown Unknown Controller:: CPDNCalled boinc_finish CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 Error: Input file: dataout/7y9mko.pjj2c10 is not a valid UM file. Error converting file to netcdf: dataout/7y9mko.pjj2c10 Error: Input file: dataout/7y9mko.pij2c10 is not a valid UM file. Error converting file to netcdf: dataout/7y9mko.pij2c10 Error: Input file: dataout/7y9mko.pfj2c10 is not a valid UM file. Error converting file to netcdf: dataout/7y9mko.pfj2c10 Error: Input file: dataout/7y9mka.phj2c10 is not a valid UM file. Error converting file to netcdf: dataout/7y9mka.phj2c10 Error: Input file: dataout/7y9mka.pgj2c10 is not a valid UM file. Error converting file to netcdf: dataout/7y9mka.pgj2c10 Error: Input file: dataout/7y9mka.pej2c10 is not a valid UM file. Error converting file to netcdf: dataout/7y9mka.pej2c10 Error: Input file: dataout/7y9mka.pdj2c10 is not a valid UM file. Error converting file to netcdf: dataout/7y9mka.pdj2c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 forrtl: No space left on device forrtl: severe (38): error during write, unit 6, file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_7y9m_1980_40_008455693/dataout/stdout_um.txt Image PC Routine Line Source hadcm3n_um_6.07_i 0042E622 Unknown Unknown Unknown hadcm3n_um_6.07_i 0042D0EB Unknown Unknown Unknown hadcm3n_um_6.07_i 003EF4DD Unknown Unknown Unknown hadcm3n_um_6.07_i 003A66F5 Unknown Unknown Unknown hadcm3n_um_6.07_i 003A5E57 Unknown Unknown Unknown hadcm3n_um_6.07_i 003E1E32 Unknown Unknown Unknown hadcm3n_um_6.07_i 003DF4CC Unknown Unknown Unknown hadcm3n_um_6.07_i 003651C5 Unknown Unknown Unknown hadcm3n_um_6.07_i 002DE95F Unknown Unknown Unknown hadcm3n_um_6.07_i 00389F90 Unknown Unknown Unknown hadcm3n_um_6.07_i 000048E1 Unknown Unknown Unknown hadcm3n_um_6.07_i 000027FB Unknown Unknown Unknown hadcm3n_um_6.07_i 00002729 Unknown Unknown Unknown Unknown 00000003 Unknown Unknown Unknown Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=12775, iMonCtr=1 Model crash detected, will try to restart... forrtl: No space left on device forrtl: severe (38): error during write, unit 6, file /Library/Application Support/BOINC Data/projCalled boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
02 Jan 2014 21:58:04	1143126	16144383	hadcm3n_7y9m_1980_40_008455693_2	596,160	201,718	0.3384
02 Jan 2014 21:58:04	1143126	16144383	hadcm3n_7y9m_1980_40_008455693_2	570,240	164,805	0.2890
02 Jan 2014 21:58:04	1143126	16144383	hadcm3n_7y9m_1980_40_008455693_2	544,320	703,129	1.2918
02 Jan 2014 21:58:04	1143126	16144383	hadcm3n_7y9m_1980_40_008455693_2	518,400	668,311	1.2892
02 Jan 2014 21:58:04	1143126	16144383	hadcm3n_7y9m_1980_40_008455693_2	492,480	635,736	1.2909
02 Jan 2014 21:58:04	1143126	16144383	hadcm3n_7y9m_1980_40_008455693_2	466,560	603,203	1.2929
02 Jan 2014 21:58:04	1143126	16144383	hadcm3n_7y9m_1980_40_008455693_2	440,640	570,652	1.2951
02 Jan 2014 21:58:04	1143126	16144383	hadcm3n_7y9m_1980_40_008455693_2	414,720	538,095	1.2975
02 Jan 2014 21:58:04	1143126	16144383	hadcm3n_7y9m_1980_40_008455693_2	388,800	505,587	1.3004
02 Jan 2014 21:58:04	1143126	16144383	hadcm3n_7y9m_1980_40_008455693_2	362,880	473,082	1.3037
02 Jan 2014 21:58:04	1143126	16144383	hadcm3n_7y9m_1980_40_008455693_2	336,960	440,603	1.3076
02 Jan 2014 21:58:04	1143126	16144383	hadcm3n_7y9m_1980_40_008455693_2	311,040	408,183	1.3123
02 Jan 2014 21:58:04	1143126	16144383	hadcm3n_7y9m_1980_40_008455693_2	285,120	375,752	1.3179
02 Jan 2014 21:58:04	1143126	16144383	hadcm3n_7y9m_1980_40_008455693_2	259,200	343,372	1.3247
02 Jan 2014 21:58:04	1143126	16144383	hadcm3n_7y9m_1980_40_008455693_2	233,280	310,985	1.3331
02 Jan 2014 21:58:04	1143126	16144383	hadcm3n_7y9m_1980_40_008455693_2	207,360	278,756	1.3443
02 Jan 2014 21:58:04	1143126	16144383	hadcm3n_7y9m_1980_40_008455693_2	181,440	246,597	1.3591
02 Jan 2014 21:58:04	1143126	16144383	hadcm3n_7y9m_1980_40_008455693_2	155,520	214,412	1.3787
02 Jan 2014 21:58:04	1143126	16144383	hadcm3n_7y9m_1980_40_008455693_2	129,600	182,267	1.4064
18 Dec 2013 10:47:27	1143126	16144383	hadcm3n_7y9m_1980_40_008455693_2	103,680	147,295	1.4207