Task 17833660

Name	hadcm3n_savy_1940_40_009109580_1
Workunit	9239916
Created	26 Jan 2015, 0:36:56 UTC
Sent	26 Jan 2015, 0:37:28 UTC
Report deadline	27 Apr 2015, 8:04:39 UTC
Received	24 Mar 2015, 12:54:00 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1235736
Run time	21 days 21 hours 53 min 25 sec
CPU time	19 days 5 hours 22 min 59 sec
Validate state	Invalid
Credit	10,886.40
Device peak FLOPS	3.38 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>7.4.36</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 11:52:13 (11186): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:52:20 (11186): No heartbeat from core client for 30 sec - exiting 11:52:21 (11186): No heartbeat from core client for 30 sec - exiting 11:52:22 (11186): No heartbeat from core client for 30 sec - exiting SIGSEGV: segmentation violation 11:55:15 (11238): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:55:24 (11238): No heartbeat from core client for 30 sec - exiting 11:55:25 (11238): No heartbeat from core client for 30 sec - exiting 11:55:26 (11238): No heartbeat from core client for 30 sec - exiting 11:55:27 (11238): No heartbeat from core client for 30 sec - exiting SIGSEGV: segmentation violation CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 23:00:54 (24096): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 10:52:43 (87614): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 05:04:44 (26866): No heartbeat from core client for 30 sec - exiting 05:04:45 (26866): No heartbeat from core client for 30 sec - exiting 05:04:46 (26866): No heartbeat from core client for 30 sec - exiting 05:04:47 (26866): No heartbeat from core client for 30 sec - exiting 05:04:48 (26866): No heartbeat from core client for 30 sec - exiting 05:04:49 (26866): No heartbeat from core client for 30 sec - exiting 05:04:50 (26866): No heartbeat from core client for 30 sec - exiting 05:04:52 (26866): No heartbeat from core client for 30 sec - exiting 05:04:53 (26866): No heartbeat from core client for 30 sec - exiting 05:04:54 (26866): No heartbeat from core client for 30 sec - exiting 05:04:55 (26866): No heartbeat from core client for 30 sec - exiting 05:04:56 (26866): No heartbeat from core client for 30 sec - exiting 05:04:57 (26866): No heartbeat from core client for 30 sec - exiting 05:04:58 (26866): No heartbeat from core client for 30 sec - exiting 05:04:59 (26866): No heartbeat from core client for 30 sec - exiting 05:05:00 (26866): No heartbeat from core client for 30 sec - exiting 05:05:01 (26866): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 00:11:52 (37532): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 14:26:55 (41983): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... SIGSEGV: segmentation violation CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 00:20:20 (59466): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:20:25 (59466): No heartbeat from core client for 30 sec - exiting 00:20:26 (59466): No heartbeat from core client for 30 sec - exiting 00:20:27 (59466): No heartbeat from core client for 30 sec - exiting 00:20:28 (59466): No heartbeat from core client for 30 sec - exiting 00:20:29 (59466): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 23:10:42 (12828): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 18:45:55 (31714): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:46:05 (31714): No heartbeat from core client for 30 sec - exiting 18:46:06 (31714): No heartbeat from core client for 30 sec - exiting 18:46:07 (31714): No heartbeat from core client for 30 sec - exiting 18:49:13 (31794): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 15:14:08 (50142): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 23:03:53 (73071): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 03:23:47 (73135): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 04:29:43 (74917): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 11:08:59 (45183): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... SIGSEGV: segmentation violation CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... forrtl: severe (17): syntax error in NAMELIST input, unit 5, file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_savy_1940_40_009109580/jobs/climate.cpdc, line 393, position 19 Image PC Routine Line Source hadcm3n_um_6.07_i 0042E622 Unknown Unknown Unknown hadcm3n_um_6.07_i 0042D0EB Unknown Unknown Unknown hadcm3n_um_6.07_i 003EF4DD Unknown Unknown Unknown hadcm3n_um_6.07_i 003A66F5 Unknown Unknown Unknown hadcm3n_um_6.07_i 003A5E57 Unknown Unknown Unknown hadcm3n_um_6.07_i 003CACB0 Unknown Unknown Unknown hadcm3n_um_6.07_i 00261D1E Unknown Unknown Unknown hadcm3n_um_6.07_i 0016269A Unknown Unknown Unknown hadcm3n_um_6.07_i 00164453 Unknown Unknown Unknown hadcm3n_um_6.07_i 0038CE7D Unknown Unknown Unknown hadcm3n_um_6.07_i 0038A1ED Unknown Unknown Unknown hadcm3n_um_6.07_i 000048E1 Unknown Unknown Unknown hadcm3n_um_6.07_i 000027FB Unknown Unknown Unknown hadcm3n_um_6.07_i 00002729 Unknown Unknown Unknown Unknown 00000003 Unknown Unknown Unknown Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5291, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 23:01:03 (10350): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 23:00:44 (15821): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:01:08 (15868): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:01:11 (15868): No heartbeat from core client for 30 sec - exiting 00:01:12 (15868): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 23:00:49 (25891): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:00:58 (25937): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... SIGSEGV: segmentation violation CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 08:27:15 (52688): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:27:21 (52688): No heartbeat from core client for 30 sec - exiting 08:27:22 (52688): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 08:47:49 (62918): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:47:54 (62918): No heartbeat from core client for 30 sec - exiting 08:47:55 (62918): No heartbeat from core client for 30 sec - exiting 08:47:56 (62918): No heartbeat from core client for 30 sec - exiting 08:47:57 (62918): No heartbeat from core client for 30 sec - exiting 08:47:58 (62918): No heartbeat from core client for 30 sec - exiting 08:47:59 (62918): No heartbeat from core client for 30 sec - exiting 08:48:00 (62918): No heartbeat from core client for 30 sec - exiting 08:48:01 (62918): No heartbeat from core client for 30 sec - exiting 08:48:02 (62918): No heartbeat from core client for 30 sec - exiting 08:48:03 (62918): No heartbeat from core client for 30 sec - exiting 08:48:04 (62918): No heartbeat from core client for 30 sec - exiting 08:48:05 (62918): No heartbeat from core client for 30 sec - exiting 08:48:06 (62918): No heartbeat from core client for 30 sec - exiting 08:48:07 (62918): No heartbeat from core client for 30 sec - exiting 08:58:54 (62984): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:58:57 (62984): No heartbeat from core client for 30 sec - exiting 08:58:58 (62984): No heartbeat from core client for 30 sec - exiting 08:58:59 (62984): No heartbeat from core client for 30 sec - exiting 08:59:00 (62984): No heartbeat from core client for 30 sec - exiting 08:59:01 (62984): No heartbeat from core client for 30 sec - exiting 08:59:03 (62984): No heartbeat from core client for 30 sec - exiting 08:59:04 (62984): No heartbeat from core client for 30 sec - exiting 08:59:05 (62984): No heartbeat from core client for 30 sec - exiting 08:59:06 (62984): No heartbeat from core client for 30 sec - exiting 08:59:07 (62984): No heartbeat from core client for 30 sec - exiting 08:59:08 (62984): No heartbeat from core client for 30 sec - exiting 08:59:09 (62984): No heartbeat from core client for 30 sec - exiting 08:59:10 (62984): No heartbeat from core client for 30 sec - exiting SIGSEGV: segmentation violation CPDN Monitor - Quit request from BOINC... 23:15:23 (68853): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:15:25 (68853): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
23 Mar 2015 02:47:09	1235736	17833660	hadcm3n_savy_1940_40_009109580_1	907,200	1,645,113	1.8134
21 Mar 2015 11:11:37	1235736	17833660	hadcm3n_savy_1940_40_009109580_1	881,280	1,598,384	1.8137
20 Mar 2015 04:33:34	1235736	17833660	hadcm3n_savy_1940_40_009109580_1	855,360	1,550,438	1.8126
18 Mar 2015 05:55:55	1235736	17833660	hadcm3n_savy_1940_40_009109580_1	829,440	1,502,894	1.8119
17 Mar 2015 04:17:58	1235736	17833660	hadcm3n_savy_1940_40_009109580_1	803,520	1,455,446	1.8113
15 Mar 2015 12:55:24	1235736	17833660	hadcm3n_savy_1940_40_009109580_1	777,600	1,408,094	1.8108
12 Mar 2015 04:59:29	1235736	17833660	hadcm3n_savy_1940_40_009109580_1	751,680	1,360,667	1.8102
11 Mar 2015 14:33:32	1235736	17833660	hadcm3n_savy_1940_40_009109580_1	725,760	1,313,175	1.8094
10 Mar 2015 16:33:43	1235736	17833660	hadcm3n_savy_1940_40_009109580_1	699,840	1,265,678	1.8085
10 Mar 2015 16:31:31	1235736	17833660	hadcm3n_savy_1940_40_009109580_1	673,920	1,223,516	1.8155
10 Mar 2015 16:28:48	1235736	17833660	hadcm3n_savy_1940_40_009109580_1	648,000	1,180,302	1.8215
10 Mar 2015 16:25:02	1235736	17833660	hadcm3n_savy_1940_40_009109580_1	622,080	1,133,809	1.8226
10 Mar 2015 16:07:53	1235736	17833660	hadcm3n_savy_1940_40_009109580_1	596,160	1,085,644	1.8211
04 Mar 2015 23:43:54	1235736	17833660	hadcm3n_savy_1940_40_009109580_1	570,240	1,038,043	1.8204
03 Mar 2015 01:00:19	1235736	17833660	hadcm3n_savy_1940_40_009109580_1	544,320	990,443	1.8196
01 Mar 2015 06:27:46	1235736	17833660	hadcm3n_savy_1940_40_009109580_1	518,400	943,195	1.8194
28 Feb 2015 03:57:39	1235736	17833660	hadcm3n_savy_1940_40_009109580_1	492,480	895,617	1.8186
27 Feb 2015 04:36:55	1235736	17833660	hadcm3n_savy_1940_40_009109580_1	466,560	847,968	1.8175
25 Feb 2015 21:40:00	1235736	17833660	hadcm3n_savy_1940_40_009109580_1	440,640	800,486	1.8166
24 Feb 2015 02:10:11	1235736	17833660	hadcm3n_savy_1940_40_009109580_1	414,720	752,949	1.8156