Task 19160974

Name	hadcm3n_sa9t_198012_480_010222856_2
Workunit	10222856
Created	15 Dec 2015, 4:04:26 UTC
Sent	15 Dec 2015, 4:04:43 UTC
Report deadline	26 Nov 2016, 9:24:43 UTC
Received	20 Dec 2015, 16:11:09 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1298787
Run time	4 days 1 hours 21 min 46 sec
CPU time	3 days 14 hours 48 min 29 sec
Validate state	Invalid
Credit	1,866.24
Device peak FLOPS	2.90 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>7.6.12</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> 06:36:42 (45243): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 09:03:05 (52332): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 09:31:18 (53098): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 00:53:43 (73006): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:00:57 (75738): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:00:59 (75738): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... forrtl: severe (17): syntax error in NAMELIST input, unit 5, file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_sa9t_198012_480_010222856/jobs/climate.cpdc, line 393, position 19 Image PC Routine Line Source hadcm3n_um_6.07_i 0042E622 Unknown Unknown Unknown hadcm3n_um_6.07_i 0042D0EB Unknown Unknown Unknown hadcm3n_um_6.07_i 003EF4DD Unknown Unknown Unknown hadcm3n_um_6.07_i 003A66F5 Unknown Unknown Unknown hadcm3n_um_6.07_i 003A5E57 Unknown Unknown Unknown hadcm3n_um_6.07_i 003CACB0 Unknown Unknown Unknown hadcm3n_um_6.07_i 00261D1E Unknown Unknown Unknown hadcm3n_um_6.07_i 0016269A Unknown Unknown Unknown hadcm3n_um_6.07_i 00164453 Unknown Unknown Unknown hadcm3n_um_6.07_i 0038CE7D Unknown Unknown Unknown hadcm3n_um_6.07_i 0038A1ED Unknown Unknown Unknown hadcm3n_um_6.07_i 000048E1 Unknown Unknown Unknown hadcm3n_um_6.07_i 000027FB Unknown Unknown Unknown hadcm3n_um_6.07_i 00002729 Unknown Unknown Unknown Unknown 00000003 Unknown Unknown Unknown Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1498, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 00:58:50 (15812): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:58:52 (15812): No heartbeat from core client for 30 sec - exiting 00:58:53 (15812): No heartbeat from core client for 30 sec - exiting BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 Error: Input file: dataout/sa9tko.pji2c10 is not a valid UM file. Error converting file to netcdf: dataout/sa9tko.pji2c10 Error: Input file: dataout/sa9tko.pii2c10 is not a valid UM file. Error converting file to netcdf: dataout/sa9tko.pii2c10 Error: Input file: dataout/sa9tko.pfi2c10 is not a valid UM file. Error converting file to netcdf: dataout/sa9tko.pfi2c10 Error: Input file: dataout/sa9tka.phi2c10 is not a valid UM file. Error converting file to netcdf: dataout/sa9tka.phi2c10 Error: Input file: dataout/sa9tka.pgi2c10 is not a valid UM file. Error converting file to netcdf: dataout/sa9tka.pgi2c10 Error: Input file: dataout/sa9tka.pei2c10 is not a valid UM file. Error converting file to netcdf: dataout/sa9tka.pei2c10 Error: Input file: dataout/sa9tka.pdi2c10 is not a valid UM file. Error converting file to netcdf: dataout/sa9tka.pdi2c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 11:42:40 (27997): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 18:39:32 (61777): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:40:17 (61801): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 03:20:04 (70190): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 08:59:40 (71350): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:59:42 (71350): No heartbeat from core client for 30 sec - exiting BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 Error: Input file: dataout/sa9tko.pji5c10 is not a valid UM file. Error converting file to netcdf: dataout/sa9tko.pji5c10 Error: Input file: dataout/sa9tko.pii5c10 is not a valid UM file. Error converting file to netcdf: dataout/sa9tko.pii5c10 Error: Input file: dataout/sa9tko.pfi5c10 is not a valid UM file. Error converting file to netcdf: dataout/sa9tko.pfi5c10 Error: Input file: dataout/sa9tka.phi5c10 is not a valid UM file. Error converting file to netcdf: dataout/sa9tka.phi5c10 Error: Input file: dataout/sa9tka.pgi5c10 is not a valid UM file. Error converting file to netcdf: dataout/sa9tka.pgi5c10 Error: Input file: dataout/sa9tka.pei5c10 is not a valid UM file. Error converting file to netcdf: dataout/sa9tka.pei5c10 Error: Input file: dataout/sa9tka.pdi5c10 is not a valid UM file. Error converting file to netcdf: dataout/sa9tka.pdi5c10 09:58:26 (76565): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 Error: Input file: dataout/sa9tko.pji5c10 is not a valid UM file. Error converting file to netcdf: dataout/sa9tko.pji5c10 Error: Input file: dataout/sa9tko.pii5c10 is not a valid UM file. Error converting file to netcdf: dataout/sa9tko.pii5c10 Error: Input file: dataout/sa9tko.pfi5c10 is not a valid UM file. Error converting file to netcdf: dataout/sa9tko.pfi5c10 Error: Input file: dataout/sa9tka.phi5c10 is not a valid UM file. Error converting file to netcdf: dataout/sa9tka.phi5c10 Error: Input file: dataout/sa9tka.pgi5c10 is not a valid UM file. Error converting file to netcdf: dataout/sa9tka.pgi5c10 Error: Input file: dataout/sa9tka.pei5c10 is not a valid UM file. Error converting file to netcdf: dataout/sa9tka.pei5c10 Error: Input file: dataout/sa9tka.pdi5c10 is not a valid UM file. Error converting file to netcdf: dataout/sa9tka.pdi5c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 11:02:06 (2248): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:02:58 (2272): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 11:16:58 (2508): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 11:20:03 (2588): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:21:11 (2619): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:23:37 (2677): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... forrtl: severe (17): syntax error in NAMELIST input, unit 5, file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_sa9t_198012_480_010222856/jobs/climate.cpdc, line 393, position 19 Image PC Routine Line Source hadcm3n_um_6.07_i 0042E622 Unknown Unknown Unknown hadcm3n_um_6.07_i 0042D0EB Unknown Unknown Unknown hadcm3n_um_6.07_i 003EF4DD Unknown Unknown Unknown hadcm3n_um_6.07_i 003A66F5 Unknown Unknown Unknown hadcm3n_um_6.07_i 003A5E57 Unknown Unknown Unknown hadcm3n_um_6.07_i 003CACB0 Unknown Unknown Unknown hadcm3n_um_6.07_i 00261D1E Unknown Unknown Unknown hadcm3n_um_6.07_i 0016269A Unknown Unknown Unknown hadcm3n_um_6.07_i 00164453 Unknown Unknown Unknown hadcm3n_um_6.07_i 0038CE7D Unknown Unknown Unknown hadcm3n_um_6.07_i 0038A1ED Unknown Unknown Unknown hadcm3n_um_6.07_i 000048E1 Unknown Unknown Unknown hadcm3n_um_6.07_i 000027FB Unknown Unknown Unknown hadcm3n_um_6.07_i 00002729 Unknown Unknown Unknown Unknown 00000003 Unknown Unknown Unknown Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2713, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... 11:35:52 (3110): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 14:32:24 (6550): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 14:58:50 (6947): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:04:38 (6983): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:09:39 (7171): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... SIGABRT: abort called Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.11.2 build 15C50 Sun Dec 20 15:10:31 2015 Thread 0 Crashed: Thread 1: Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffcc68 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x90093000 - 0x90093fff /usr/lib/system/libunc.dylib 0x9009d000 - 0x900a6fff /usr/lib/system/libcopyfile.dylib 0x909a0000 - 0x909b1fff /usr/lib/system/libsystem_trace.dylib 0x909e9000 - 0x909edfff /usr/lib/system/libcache.dylib 0x91c96000 - 0x91caffff /usr/lib/system/libsystem_asl.dylib 0x94a69000 - 0x94a93fff /usr/lib/system/libdispatch.dylib 0x94a94000 - 0x94a95fff /usr/lib/system/libremovefile.dylib 0x94fc0000 - 0x94fd7fff /usr/lib/system/libsystem_coretls.dylib 0x95afb000 - 0x95b2efff /usr/lib/system/libsystem_m.dylib 0x95b2f000 - 0x95b2ffff /usr/lib/system/libsystem_blocks.dylib 0x95b30000 - 0x95b30fff /usr/lib/system/liblaunch.dylib 0x95b31000 - 0x95b37fff /usr/lib/system/libunwind.dylib 0x95c59000 - 0x95c64fff /usr/lib/system/libcommonCrypto.dylib 0x96023000 - 0x9604dfff /usr/lib/system/libsystem_info.dylib 0x9604e000 - 0x96054fff /usr/lib/system/libsystem_platform.dylib 0x96055000 - 0x96057fff /usr/lib/system/libsystem_secinit.dylib 0x96a00000 - 0x96a69fff /usr/lib/system/libsystem_network.dylib 0x978da000 - 0x978ddfff /usr/lib/system/libsystem_sandbox.dylib 0x978de000 - 0x978e6fff /usr/lib/system/libsystem_pthread.dylib 0x9795b000 - 0x97983fff /usr/lib/system/libxpc.dylib 0x97ad4000 - 0x97adefff /usr/lib/system/libsystem_notify.dylib 0x985c0000 - 0x985e0fff /usr/lib/system/libsystem_kernel.dylib 0x98646000 - 0x98646fff /usr/lib/system/libkeymgr.dylib 0x987c2000 - 0x987cbfff /usr/lib/system/libsystem_networkextension.dylib 0x98eea000 - 0x98eecfff /usr/lib/system/libsystem_configuration.dylib 0x99155000 - 0x99157fff /usr/lib/system/libquarantine.dylib 0x99173000 - 0x99178fff /usr/lib/system/libmacho.dylib 0x99762000 - 0x9976afff /usr/lib/system/libsystem_dnssd.dylib 0x9a1b8000 - 0x9a1bbfff /usr/lib/system/libdyld.dylib 0x9a1bc000 - 0x9a1cafff /usr/lib/libz.1.dylib 0x9a8a8000 - 0x9a8a9fff /usr/lib/libSystem.B.dylib 0x9a8aa000 - 0x9a8affff /usr/lib/system/libcompiler_rt.dylib 0x9a8b3000 - 0x9a8ccfff /usr/lib/system/libsystem_malloc.dylib 0x9cbe8000 - 0x9cbeafff /usr/lib/system/libsystem_coreservices.dylib 0x9d09b000 - 0x9d130fff /usr/lib/system/libsystem_c.dylib 0x9e432000 - 0x9e499fff /usr/lib/system/libcorecrypto.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7352, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.11.2 build 15C50 Sun Dec 20 15:10:35 2015 Thread 0 Crashed: Thread 1: Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffcc68 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x90093000 - 0x90093fff /usr/lib/system/libunc.dylib 0x9009d000 - 0x900a6fff /usr/lib/system/libcopyfile.dylib 0x909a0000 - 0x909b1fff /usr/lib/system/libsystem_trace.dylib 0x909e9000 - 0x909edfff /usr/lib/system/libcache.dylib 0x91c96000 - 0x91caffff /usr/lib/system/libsystem_asl.dylib 0x94a69000 - 0x94a93fff /usr/lib/system/libdispatch.dylib 0x94a94000 - 0x94a95fff /usr/lib/system/libremovefile.dylib 0x94fc0000 - 0x94fd7fff /usr/lib/system/libsystem_coretls.dylib 0x95afb000 - 0x95b2efff /usr/lib/system/libsystem_m.dylib 0x95b2f000 - 0x95b2ffff /usr/lib/system/libsystem_blocks.dylib 0x95b30000 - 0x95b30fff /usr/lib/system/liblaunch.dylib 0x95b31000 - 0x95b37fff /usr/lib/system/libunwind.dylib 0x95c59000 - 0x95c64fff /usr/lib/system/libcommonCrypto.dylib 0x96023000 - 0x9604dfff /usr/lib/system/libsystem_info.dylib 0x9604e000 - 0x96054fff /usr/lib/system/libsystem_platform.dylib 0x96055000 - 0x96057fff /usr/lib/system/libsystem_secinit.dylib 0x96a00000 - 0x96a69fff /usr/lib/system/libsystem_network.dylib 0x978da000 - 0x978ddfff /usr/lib/system/libsystem_sandbox.dylib 0x978de000 - 0x978e6fff /usr/lib/system/libsystem_pthread.dylib 0x9795b000 - 0x97983fff /usr/lib/system/libxpc.dylib 0x97ad4000 - 0x97adefff /usr/lib/system/libsystem_notify.dylib 0x985c0000 - 0x985e0fff /usr/lib/system/libsystem_kernel.dylib 0x98646000 - 0x98646fff /usr/lib/system/libkeymgr.dylib 0x987c2000 - 0x987cbfff /usr/lib/system/libsystem_networkextension.dylib 0x98eea000 - 0x98eecfff /usr/lib/system/libsystem_configuration.dylib 0x99155000 - 0x99157fff /usr/lib/system/libquarantine.dylib 0x99173000 - 0x99178fff /usr/lib/system/libmacho.dylib 0x99762000 - 0x9976afff /usr/lib/system/libsystem_dnssd.dylib 0x9a1b8000 - 0x9a1bbfff /usr/lib/system/libdyld.dylib 0x9a1bc000 - 0x9a1cafff /usr/lib/libz.1.dylib 0x9a8a8000 - 0x9a8a9fff /usr/lib/libSystem.B.dylib 0x9a8aa000 - 0x9a8affff /usr/lib/system/libcompiler_rt.dylib 0x9a8b3000 - 0x9a8ccfff /usr/lib/system/libsystem_malloc.dylib 0x9cbe8000 - 0x9cbeafff /usr/lib/system/libsystem_coreservices.dylib 0x9d09b000 - 0x9d130fff /usr/lib/system/libsystem_c.dylib 0x9e432000 - 0x9e499fff /usr/lib/system/libcorecrypto.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7352, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.11.2 build 15C50 Sun Dec 20 15:10:38 2015 Thread 0 Crashed: Thread 1: Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffcc68 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x90093000 - 0x90093fff /usr/lib/system/libunc.dylib 0x9009d000 - 0x900a6fff /usr/lib/system/libcopyfile.dylib 0x909a0000 - 0x909b1fff /usr/lib/system/libsystem_trace.dylib 0x909e9000 - 0x909edfff /usr/lib/system/libcache.dylib 0x91c96000 - 0x91caffff /usr/lib/system/libsystem_asl.dylib 0x94a69000 - 0x94a93fff /usr/lib/system/libdispatch.dylib 0x94a94000 - 0x94a95fff /usr/lib/system/libremovefile.dylib 0x94fc0000 - 0x94fd7fff /usr/lib/system/libsystem_coretls.dylib 0x95afb000 - 0x95b2efff /usr/lib/system/libsystem_m.dylib 0x95b2f000 - 0x95b2ffff /usr/lib/system/libsystem_blocks.dylib 0x95b30000 - 0x95b30fff /usr/lib/system/liblaunch.dylib 0x95b31000 - 0x95b37fff /usr/lib/system/libunwind.dylib 0x95c59000 - 0x95c64fff /usr/lib/system/libcommonCrypto.dylib 0x96023000 - 0x9604dfff /usr/lib/system/libsystem_info.dylib 0x9604e000 - 0x96054fff /usr/lib/system/libsystem_platform.dylib 0x96055000 - 0x96057fff /usr/lib/system/libsystem_secinit.dylib 0x96a00000 - 0x96a69fff /usr/lib/system/libsystem_network.dylib 0x978da000 - 0x978ddfff /usr/lib/system/libsystem_sandbox.dylib 0x978de000 - 0x978e6fff /usr/lib/system/libsystem_pthread.dylib 0x9795b000 - 0x97983fff /usr/lib/system/libxpc.dylib 0x97ad4000 - 0x97adefff /usr/lib/system/libsystem_notify.dylib 0x985c0000 - 0x985e0fff /usr/lib/system/libsystem_kernel.dylib 0x98646000 - 0x98646fff /usr/lib/system/libkeymgr.dylib 0x987c2000 - 0x987cbfff /usr/lib/system/libsystem_networkextension.dylib 0x98eea000 - 0x98eecfff /usr/lib/system/libsystem_configuration.dylib 0x99155000 - 0x99157fff /usr/lib/system/libquarantine.dylib 0x99173000 - 0x99178fff /usr/lib/system/libmacho.dylib 0x99762000 - 0x9976afff /usr/lib/system/libsystem_dnssd.dylib 0x9a1b8000 - 0x9a1bbfff /usr/lib/system/libdyld.dylib 0x9a1bc000 - 0x9a1cafff /usr/lib/libz.1.dylib 0x9a8a8000 - 0x9a8a9fff /usr/lib/libSystem.B.dylib 0x9a8aa000 - 0x9a8affff /usr/lib/system/libcompiler_rt.dylib 0x9a8b3000 - 0x9a8ccfff /usr/lib/system/libsystem_malloc.dylib 0x9cbe8000 - 0x9cbeafff /usr/lib/system/libsystem_coreservices.dylib 0x9d09b000 - 0x9d130fff /usr/lib/system/libsystem_c.dylib 0x9e432000 - 0x9e499fff /usr/lib/system/libcorecrypto.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7352, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.11.2 build 15C50 Sun Dec 20 15:10:41 2015 Thread 0 Crashed: Thread 1: Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffcc68 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x90093000 - 0x90093fff /usr/lib/system/libunc.dylib 0x9009d000 - 0x900a6fff /usr/lib/system/libcopyfile.dylib 0x909a0000 - 0x909b1fff /usr/lib/system/libsystem_trace.dylib 0x909e9000 - 0x909edfff /usr/lib/system/libcache.dylib 0x91c96000 - 0x91caffff /usr/lib/system/libsystem_asl.dylib 0x94a69000 - 0x94a93fff /usr/lib/system/libdispatch.dylib 0x94a94000 - 0x94a95fff /usr/lib/system/libremovefile.dylib 0x94fc0000 - 0x94fd7fff /usr/lib/system/libsystem_coretls.dylib 0x95afb000 - 0x95b2efff /usr/lib/system/libsystem_m.dylib 0x95b2f000 - 0x95b2ffff /usr/lib/system/libsystem_blocks.dylib 0x95b30000 - 0x95b30fff /usr/lib/system/liblaunch.dylib 0x95b31000 - 0x95b37fff /usr/lib/system/libunwind.dylib 0x95c59000 - 0x95c64fff /usr/lib/system/libcommonCrypto.dylib 0x96023000 - 0x9604dfff /usr/lib/system/libsystem_info.dylib 0x9604e000 - 0x96054fff /usr/lib/system/libsystem_platform.dylib 0x96055000 - 0x96057fff /usr/lib/system/libsystem_secinit.dylib 0x96a00000 - 0x96a69fff /usr/lib/system/libsystem_network.dylib 0x978da000 - 0x978ddfff /usr/lib/system/libsystem_sandbox.dylib 0x978de000 - 0x978e6fff /usr/lib/system/libsystem_pthread.dylib 0x9795b000 - 0x97983fff /usr/lib/system/libxpc.dylib 0x97ad4000 - 0x97adefff /usr/lib/system/libsystem_notify.dylib 0x985c0000 - 0x985e0fff /usr/lib/system/libsystem_kernel.dylib 0x98646000 - 0x98646fff /usr/lib/system/libkeymgr.dylib 0x987c2000 - 0x987cbfff /usr/lib/system/libsystem_networkextension.dylib 0x98eea000 - 0x98eecfff /usr/lib/system/libsystem_configuration.dylib 0x99155000 - 0x99157fff /usr/lib/system/libquarantine.dylib 0x99173000 - 0x99178fff /usr/lib/system/libmacho.dylib 0x99762000 - 0x9976afff /usr/lib/system/libsystem_dnssd.dylib 0x9a1b8000 - 0x9a1bbfff /usr/lib/system/libdyld.dylib 0x9a1bc000 - 0x9a1cafff /usr/lib/libz.1.dylib 0x9a8a8000 - 0x9a8a9fff /usr/lib/libSystem.B.dylib 0x9a8aa000 - 0x9a8affff /usr/lib/system/libcompiler_rt.dylib 0x9a8b3000 - 0x9a8ccfff /usr/lib/system/libsystem_malloc.dylib 0x9cbe8000 - 0x9cbeafff /usr/lib/system/libsystem_coreservices.dylib 0x9d09b000 - 0x9d130fff /usr/lib/system/libsystem_c.dylib 0x9e432000 - 0x9e499fff /usr/lib/system/libcorecrypto.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7352, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.11.2 build 15C50 Sun Dec 20 15:10:47 2015 Thread 0 Crashed: Thread 1: Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffcc68 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x90093000 - 0x90093fff /usr/lib/system/libunc.dylib 0x9009d000 - 0x900a6fff /usr/lib/system/libcopyfile.dylib 0x909a0000 - 0x909b1fff /usr/lib/system/libsystem_trace.dylib 0x909e9000 - 0x909edfff /usr/lib/system/libcache.dylib 0x91c96000 - 0x91caffff /usr/lib/system/libsystem_asl.dylib 0x94a69000 - 0x94a93fff /usr/lib/system/libdispatch.dylib 0x94a94000 - 0x94a95fff /usr/lib/system/libremovefile.dylib 0x94fc0000 - 0x94fd7fff /usr/lib/system/libsystem_coretls.dylib 0x95afb000 - 0x95b2efff /usr/lib/system/libsystem_m.dylib 0x95b2f000 - 0x95b2ffff /usr/lib/system/libsystem_blocks.dylib 0x95b30000 - 0x95b30fff /usr/lib/system/liblaunch.dylib 0x95b31000 - 0x95b37fff /usr/lib/system/libunwind.dylib 0x95c59000 - 0x95c64fff /usr/lib/system/libcommonCrypto.dylib 0x96023000 - 0x9604dfff /usr/lib/system/libsystem_info.dylib 0x9604e000 - 0x96054fff /usr/lib/system/libsystem_platform.dylib 0x96055000 - 0x96057fff /usr/lib/system/libsystem_secinit.dylib 0x96a00000 - 0x96a69fff /usr/lib/system/libsystem_network.dylib 0x978da000 - 0x978ddfff /usr/lib/system/libsystem_sandbox.dylib 0x978de000 - 0x978e6fff /usr/lib/system/libsystem_pthread.dylib 0x9795b000 - 0x97983fff /usr/lib/system/libxpc.dylib 0x97ad4000 - 0x97adefff /usr/lib/system/libsystem_notify.dylib 0x985c0000 - 0x985e0fff /usr/lib/system/libsystem_kernel.dylib 0x98646000 - 0x98646fff /usr/lib/system/libkeymgr.dylib 0x987c2000 - 0x987cbfff /usr/lib/system/libsystem_networkextension.dylib 0x98eea000 - 0x98eecfff /usr/lib/system/libsystem_configuration.dylib 0x99155000 - 0x99157fff /usr/lib/system/libquarantine.dylib 0x99173000 - 0x99178fff /usr/lib/system/libmacho.dylib 0x99762000 - 0x9976afff /usr/lib/system/libsystem_dnssd.dylib 0x9a1b8000 - 0x9a1bbfff /usr/lib/system/libdyld.dylib 0x9a1bc000 - 0x9a1cafff /usr/lib/libz.1.dylib 0x9a8a8000 - 0x9a8a9fff /usr/lib/libSystem.B.dylib 0x9a8aa000 - 0x9a8affff /usr/lib/system/libcompiler_rt.dylib 0x9a8b3000 - 0x9a8ccfff /usr/lib/system/libsystem_malloc.dylib 0x9cbe8000 - 0x9cbeafff /usr/lib/system/libsystem_coreservices.dylib 0x9d09b000 - 0x9d130fff /usr/lib/system/libsystem_c.dylib 0x9e432000 - 0x9e499fff /usr/lib/system/libcorecrypto.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7352, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.11.2 build 15C50 Sun Dec 20 15:10:50 2015 Thread 0 Crashed: Thread 1: Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffcc68 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x90093000 - 0x90093fff /usr/lib/system/libunc.dylib 0x9009d000 - 0x900a6fff /usr/lib/system/libcopyfile.dylib 0x909a0000 - 0x909b1fff /usr/lib/system/libsystem_trace.dylib 0x909e9000 - 0x909edfff /usr/lib/system/libcache.dylib 0x91c96000 - 0x91caffff /usr/lib/system/libsystem_asl.dylib 0x94a69000 - 0x94a93fff /usr/lib/system/libdispatch.dylib 0x94a94000 - 0x94a95fff /usr/lib/system/libremovefile.dylib 0x94fc0000 - 0x94fd7fff /usr/lib/system/libsystem_coretls.dylib 0x95afb000 - 0x95b2efff /usr/lib/system/libsystem_m.dylib 0x95b2f000 - 0x95b2ffff /usr/lib/system/libsystem_blocks.dylib 0x95b30000 - 0x95b30fff /usr/lib/system/liblaunch.dylib 0x95b31000 - 0x95b37fff /usr/lib/system/libunwind.dylib 0x95c59000 - 0x95c64fff /usr/lib/system/libcommonCrypto.dylib 0x96023000 - 0x9604dfff /usr/lib/system/libsystem_info.dylib 0x9604e000 - 0x96054fff /usr/lib/system/libsystem_platform.dylib 0x96055000 - 0x96057fff /usr/lib/system/libsystem_secinit.dylib 0x96a00000 - 0x96a69fff /usr/lib/system/libsystem_network.dylib 0x978da000 - 0x978ddfff /usr/lib/system/libsystem_sandbox.dylib 0x978de000 - 0x978e6fff /usr/lib/system/libsystem_pthread.dylib 0x9795b000 - 0x97983fff /usr/lib/system/libxpc.dylib 0x97ad4000 - 0x97adefff /usr/lib/system/libsystem_notify.dylib 0x985c0000 - 0x985e0fff /usr/lib/system/libsystem_kernel.dylib 0x98646000 - 0x98646fff /usr/lib/system/libkeymgr.dylib 0x987c2000 - 0x987cbfff /usr/lib/system/libsystem_networkextension.dylib 0x98eea000 - 0x98eecfff /usr/lib/system/libsystem_configuration.dylib 0x99155000 - 0x99157fff /usr/lib/system/libquarantine.dylib 0x99173000 - 0x99178fff /usr/lib/system/libmacho.dylib 0x99762000 - 0x9976afff /usr/lib/system/libsystem_dnssd.dylib 0x9a1b8000 - 0x9a1bbfff /usr/lib/system/libdyld.dylib 0x9a1bc000 - 0x9a1cafff /usr/lib/libz.1.dylib 0x9a8a8000 - 0x9a8a9fff /usr/lib/libSystem.B.dylib 0x9a8aa000 - 0x9a8affff /usr/lib/system/libcompiler_rt.dylib 0x9a8b3000 - 0x9a8ccfff /usr/lib/system/libsystem_malloc.dylib 0x9cbe8000 - 0x9cbeafff /usr/lib/system/libsystem_coreservices.dylib 0x9d09b000 - 0x9d130fff /usr/lib/system/libsystem_c.dylib 0x9e432000 - 0x9e499fff /usr/lib/system/libcorecrypto.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7352, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
20 Dec 2015 03:39:32	1298787	19160974	hadcm3n_sa9t_198012_480_010222856_2	155,520	292,209	1.8789
19 Dec 2015 10:07:17	1298787	19160974	hadcm3n_sa9t_198012_480_010222856_2	129,600	243,877	1.8818
18 Dec 2015 13:13:52	1298787	19160974	hadcm3n_sa9t_198012_480_010222856_2	103,680	194,926	1.8801
17 Dec 2015 19:57:41	1298787	19160974	hadcm3n_sa9t_198012_480_010222856_2	77,760	146,240	1.8807
17 Dec 2015 01:03:23	1298787	19160974	hadcm3n_sa9t_198012_480_010222856_2	51,840	97,571	1.8822
16 Dec 2015 04:26:19	1298787	19160974	hadcm3n_sa9t_198012_480_010222856_2	25,920	49,084	1.8937