climateprediction.net home page
Task 18034277

Task 18034277

Name hadam3pm2_pfc9_1991_10_009530178_1
Workunit 9611921
Created 9 Mar 2015, 22:47:57 UTC
Sent 10 Mar 2015, 0:27:33 UTC
Report deadline 20 Feb 2016, 5:47:33 UTC
Received 16 Mar 2015, 18:15:29 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 22 (0x00000016) Unknown error code
Computer ID 1356434
Run time 6 days 8 hours 10 min 41 sec
CPU time 1 days 12 hours 39 min 39 sec
Validate state Invalid
Credit 7,320.04
Device peak FLOPS 3.97 GFLOPS
Application version UK Met Office HadAM3P (global only) with MOSES II landsurface scheme v7.03
i686-pc-linux-gnu
Stderr 
<core_client_version>7.2.42</core_client_version>
<![CDATA[
<message>
process exited with code 22 (0x16, -234)
</message>
<stderr_txt>
oa.pc|1dec.nc
oa.pc|2jan.nc
oa.pc|2feb.nc
oa.pc|2mar.nc
oa.pc|2apr.nc
oa.pc|2may.nc
oa.pc|2jun.nc
oa.pc|2jul.nc
oa.pc|2aug.nc
oa.pc|2sep.nc
oa.pc|2oct.nc
oa.pc|2nov.nc
oa.pe|1dec.nc
oa.pe|2jan.nc
oa.pe|2feb.nc
oa.pe|2mar.nc
oa.pe|2apr.nc
oa.pe|2may.nc
oa.pe|2jun.nc
oa.pe|2jul.nc
oa.pe|2aug.nc
oa.pe|2sep.nc
oa.pe|2oct.nc
oa.pe|2nov.nc
Suspended CPDN Monitor - Suspend request from BOINC...
oa.pc|2dec.nc
oa.pc|3jan.nc
oa.pc|3feb.nc
oa.pc|3mar.nc
oa.pc|3apr.nc
oa.pc|3may.nc
oa.pc|3jun.nc
oa.pc|3jul.nc
oa.pc|3aug.nc
oa.pc|3sep.nc
oa.pc|3oct.nc
oa.pc|3nov.nc
oa.pe|2dec.nc
oa.pe|3jan.nc
oa.pe|3feb.nc
oa.pe|3mar.nc
oa.pe|3apr.nc
oa.pe|3may.nc
oa.pe|3jun.nc
oa.pe|3jul.nc
oa.pe|3aug.nc
oa.pe|3sep.nc
oa.pe|3oct.nc
oa.pe|3nov.nc

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: 

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: 

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: 

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: 

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: 

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: 

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: 

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: 

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: 

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: 

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: 
forrtl: No space left on device
forrtl: severe (38): error during write, unit 6, file /var/lib/boinc-client/projects/climateprediction.net/hadam3pm2_pfc9_1991_10_009530178/dataout/stdout_um.txt
Image              PC        Routine            Line        Source             
hadam3pm2_um_7.03  0845AB9D  Unknown               Unknown  Unknown
hadam3pm2_um_7.03  08459995  Unknown               Unknown  Unknown
hadam3pm2_um_7.03  0842865F  Unknown               Unknown  Unknown
hadam3pm2_um_7.03  083E7E1D  Unknown               Unknown  Unknown
hadam3pm2_um_7.03  083E7767  Unknown               Unknown  Unknown
hadam3pm2_um_7.03  0841A2F9  Unknown               Unknown  Unknown
hadam3pm2_um_7.03  08417BC7  Unknown               Unknown  Unknown
hadam3pm2_um_7.03  08346323  Unknown               Unknown  Unknown
hadam3pm2_um_7.03  082D4171  Unknown               Unknown  Unknown
hadam3pm2_um_7.03  0832046D  Unknown               Unknown  Unknown
hadam3pm2_um_7.03  0834C82D  Unknown               Unknown  Unknown
libc.so.6          F0CD9A83  Unknown               Unknown  Unknown
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=16899, iMonCtr=1
Model crash detected, will try to restart...

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: 

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: go

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: 

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: go

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: 

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: 

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: go

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: 

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: 
oa.pc|3dec.nc
oa.pc|4jan.nc
oa.pc|4feb.nc
oa.pc|4mar.nc
oa.pc|4apr.nc
oa.pc|4may.nc
oa.pc|4jun.nc
oa.pc|4jul.nc
oa.pc|4aug.nc
oa.pc|4sep.nc
oa.pc|4oct.nc
oa.pc|4nov.nc
oa.pe|3dec.nc
oa.pe|4jan.nc
oa.pe|4feb.nc
oa.pe|4mar.nc
oa.pe|4apr.nc
oa.pe|4may.nc
oa.pe|4jun.nc
oa.pe|4jul.nc
oa.pe|4aug.nc
oa.pe|4sep.nc
oa.pe|4oct.nc
oa.pe|4nov.nc

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: 

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: 

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: 

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: 

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: 

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: 

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: 

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: 

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: 

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: 

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: 

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: 

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: 

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: 

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: 

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: 

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: 

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: 

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: 

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: 

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: 

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: 

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: 

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: 

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: 

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: 

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: 

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: 

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: 

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: 

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: 

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: 
forrtl: No space left on device
forrtl: severe (38): error during write, unit 6, file /var/lib/boinc-client/projects/climateprediction.net/hadam3pm2_pfc9_1991_10_009530178/dataout/stdout_um.txt
Image              PC        Routine            Line        Source             
hadam3pm2_um_7.03  0845AB9D  Unknown               Unknown  Unknown
hadam3pm2_um_7.03  08459995  Unknown               Unknown  Unknown
hadam3pm2_um_7.03  0842865F  Unknown               Unknown  Unknown
h
BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: go

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: go

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: 

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: go

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: go

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: 

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: go

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: go

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: go

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: 

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: 

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: 
oa.pc|4dec.nc
oa.pc|5jan.nc
oa.pc|5feb.nc
oa.pc|5mar.nc
oa.pc|5apr.nc
oa.pc|5may.nc
oa.pc|5jun.nc
oa.pc|5jul.nc
oa.pc|5aug.nc
oa.pc|5sep.nc
oa.pc|5oct.nc
oa.pc|5nov.nc
oa.pe|4dec.nc
oa.pe|5jan.nc
oa.pe|5feb.nc
oa.pe|5mar.nc
oa.pe|5apr.nc
oa.pe|5may.nc
oa.pe|5jun.nc
oa.pe|5jul.nc
oa.pe|5aug.nc
oa.pe|5sep.nc
oa.pe|5oct.nc
oa.pe|5nov.nc

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: 

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: 

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: 

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: 

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: 

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: 

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: 

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: 
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: 

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: 

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: 

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: 

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: 

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: 

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: 

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: 

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: 

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: 

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: 

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: go

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: 

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: 

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return codeoa.pc|7dec.nc
oa.pc|8jan.nc
oa.pc|8feb.nc
oa.pc|8mar.nc
oa.pc|8apr.nc
oa.pc|8may.nc
oa.pc|8jun.nc
oa.pc|8jul.nc
oa.pc|8aug.nc
oa.pc|8sep.nc
oa.pc|8oct.nc
oa.pc|8nov.nc
oa.pe|7dec.nc
oa.pe|8jan.nc
oa.pe|8feb.nc
oa.pe|8mar.nc
oa.pe|8apr.nc
oa.pe|8may.nc
oa.pe|8jun.nc
oa.pe|8jul.nc
oa.pe|8aug.nc
oa.pe|8sep.nc
oa.pe|8oct.nc
oa.pe|8nov.nc
Suspended CPDN Monitor - Suspend request from BOINC...

SETPOS: Seek Failed: Invalid argument
SETPOS: Unit 64 to Word Address -198 Failed with Error Code -1

Model crashed: SETPOS: Unit 64 to Word Address -198 Failed with Error Code -1

SETPOS: Seek Failed: Invalid argument
SETPOS: Unit 64 to Word Address -198 Failed with Error Code -1

Model crashed: SETPOS: Unit 64 to Word Address -198 Failed with Error Code -1

SETPOS: Seek Failed: Invalid argument
SETPOS: Unit 64 to Word Address -198 Failed with Error Code -1

Model crashed: SETPOS: Unit 64 to Word Address -198 Failed with Error Code -1

SETPOS: Seek Failed: Invalid argument
SETPOS: Unit 64 to Word Address -198 Failed with Error Code -1

Model crashed: SETPOS: Unit 64 to Word Address -198 Failed with Error Code -1

SETPOS: Seek Failed: Invalid argument
SETPOS: Unit 64 to Word Address -198 Failed with Error Code -1

Model crashed: SETPOS: Unit 64 to Word Address -198 Failed with Error Code -1

SETPOS: Seek Failed: Invalid argument
SETPOS: Unit 64 to Word Address -198 Failed with Error Code -1

Model crashed: SETPOS: Unit 64 to Word Address -198 Failed with Error Code -1
Sorry, too many model crashes! :-(
20:14:09 (2013): called boinc_finish

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
16 Mar 2015 18:15:51 1356434 18034277 hadam3pm2_pfc9_1991_10_009530178_1 253,508 131,027 0.5169
16 Mar 2015 17:05:34 1356434 18034277 hadam3pm2_pfc9_1991_10_009530178_1 250,628 126,905 0.5063
16 Mar 2015 16:05:20 1356434 18034277 hadam3pm2_pfc9_1991_10_009530178_1 247,748 122,497 0.4944
16 Mar 2015 14:04:47 1356434 18034277 hadam3pm2_pfc9_1991_10_009530178_1 244,868 117,891 0.4814
16 Mar 2015 11:54:12 1356434 18034277 hadam3pm2_pfc9_1991_10_009530178_1 239,108 109,401 0.4575
16 Mar 2015 10:53:54 1356434 18034277 hadam3pm2_pfc9_1991_10_009530178_1 236,228 105,276 0.4457
16 Mar 2015 09:53:36 1356434 18034277 hadam3pm2_pfc9_1991_10_009530178_1 233,348 101,100 0.4333
16 Mar 2015 08:53:17 1356434 18034277 hadam3pm2_pfc9_1991_10_009530178_1 230,468 97,027 0.4210
16 Mar 2015 07:47:58 1356434 18034277 hadam3pm2_pfc9_1991_10_009530178_1 227,588 92,962 0.4085
16 Mar 2015 06:47:32 1356434 18034277 hadam3pm2_pfc9_1991_10_009530178_1 224,708 88,831 0.3953
16 Mar 2015 04:05:02 1356434 18034277 hadam3pm2_pfc9_1991_10_009530178_1 221,828 88,165 0.3974
16 Mar 2015 01:59:24 1356434 18034277 hadam3pm2_pfc9_1991_10_009530178_1 218,948 112,070 0.5119
16 Mar 2015 00:58:49 1356434 18034277 hadam3pm2_pfc9_1991_10_009530178_1 216,068 107,566 0.4978
15 Mar 2015 23:53:33 1356434 18034277 hadam3pm2_pfc9_1991_10_009530178_1 213,188 102,915 0.4827
15 Mar 2015 21:53:02 1356434 18034277 hadam3pm2_pfc9_1991_10_009530178_1 210,308 98,080 0.4664
15 Mar 2015 10:24:38 1356434 18034277 hadam3pm2_pfc9_1991_10_009530178_1 201,668 83,276 0.4129
15 Mar 2015 08:23:59 1356434 18034277 hadam3pm2_pfc9_1991_10_009530178_1 198,788 78,056 0.3927
15 Mar 2015 07:23:37 1356434 18034277 hadam3pm2_pfc9_1991_10_009530178_1 195,908 73,032 0.3728
15 Mar 2015 05:21:46 1356434 18034277 hadam3pm2_pfc9_1991_10_009530178_1 193,028 67,841 0.3515
15 Mar 2015 04:25:39 1356434 18034277 hadam3pm2_pfc9_1991_10_009530178_1 190,148 62,769 0.3301


©2024 climateprediction.net