Task 19750471

Name	hadcm3n_xdms_198012_480_374_010413468_1
Workunit	10413468
Created	16 Jun 2016, 17:15:54 UTC
Sent	16 Jun 2016, 17:16:13 UTC
Report deadline	29 May 2017, 22:36:13 UTC
Received	19 Jun 2016, 22:56:31 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1398426
Run time	6 hours 55 min 6 sec
CPU time
Validate state	Invalid
Credit	3,732.48
Device peak FLOPS	3.72 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.2.47</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> 13:23:56 (14796): start_timer_thread(): CreateThread() failed, errno 0 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 13:53:40 (17288): start_timer_thread(): CreateThread() failed, errno 0 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 14:35:00 (16924): start_timer_thread(): CreateThread() failed, errno 0 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 14:43:57 (15172): start_timer_thread(): CreateThread() failed, errno 0 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 15:01:28 (14428): start_timer_thread(): CreateThread() failed, errno 0 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 15:14:35 (16392): start_timer_thread(): CreateThread() failed, errno 0 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 15:34:19 (9144): start_timer_thread(): CreateThread() failed, errno 0 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 15:52:34 (16964): start_timer_thread(): CreateThread() failed, errno 0 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 17:17:21 (16996): start_timer_thread(): CreateThread() failed, errno 0 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 17:28:54 (14536): start_timer_thread(): CreateThread() failed, errno 0 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 diagnostics_init_unhandled_exception_monitor(): Creating hExceptionMonitorThread failed, errno 12 WARNING: BOINC Windows Runtime Debugger has been disabled. 17:56:39 (18188): start_timer_thread(): CreateThread() failed, errno 0 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 diagnostics_init_unhandled_exception_monitor(): Creating hExceptionMonitorThread failed, errno 12 WARNING: BOINC Windows Runtime Debugger has been disabled. 18:05:31 (17828): start_timer_thread(): CreateThread() failed, errno 0 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 diagnostics_init_unhandled_exception_monitor(): Creating hExceptionMonitorThread failed, errno 12 WARNING: BOINC Windows Runtime Debugger has been disabled. 18:55:40 (17536): start_timer_thread(): CreateThread() failed, errno 0 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 19:05:22 (13556): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 20:05:08 (18640): start_timer_thread(): CreateThread() failed, errno 0 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 20:22:58 (19072): start_timer_thread(): CreateThread() failed, errno 0 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 21:33:22 (15460): start_timer_thread(): CreateThread() failed, errno 0 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 22:11:31 (19340): start_timer_thread(): CreateThread() failed, errno 0 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 22:26:53 (19432): start_timer_thread(): CreateThread() failed, errno 0 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 22:28:19 (19004): start_timer_thread(): CreateThread() failed, errno 0 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 00:58:31 (18888): start_timer_thread(): CreateThread() failed, errno 0 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 01:06:14 (16448): start_timer_thread(): CreateThread() failed, errno 0 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 01:14:05 (17752): start_timer_thread(): CreateThread() failed, errno 0 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 01:27:18 (19168): start_timer_thread(): CreateThread() failed, errno 0 Model crashed: P_TH_ADJ : NEGATIVE PRESSURE VALUE CREATED. tmp/pipe_dummy 2048 01:29:28 (18336): start_timer_thread(): CreateThread() failed, errno 0 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 diagnostics_init_unhandled_exception_monitor(): Creating hExceptionMonitorThread failed, errno 12 WARNING: BOINC Windows Runtime Debugger has been disabled. 01:45:27 (17928): start_timer_thread(): CreateThread() failed, errno 0 17:53:06 (9624): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 18:55:21 (6108): start_timer_thread(): CreateThread() failed, errno 0 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 18:57:54 (5976): start_timer_thread(): CreateThread() failed, errno 0 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 18:58:43 (7120): start_timer_thread(): CreateThread() failed, errno 0 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 18:59:40 (9424): start_timer_thread(): CreateThread() failed, errno 0 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 19:06:19 (7668): start_timer_thread(): CreateThread() failed, errno 0 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 19:07:13 (10216): start_timer_thread(): CreateThread() failed, errno 0 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 19:13:53 (7320): start_timer_thread(): CreateThread() failed, errno 0 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 19:33:00 (9764): start_timer_thread(): CreateThread() failed, errno 0 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 19:46:15 (1452): start_timer_thread(): CreateThread() failed, errno 0 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 20:11:46 (9564): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 21:20:51 (4692): start_timer_thread(): CreateThread() failed, errno 0 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 23:07:32 (7120): start_timer_thread(): CreateThread() failed, errno 0 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 23:34:48 (1112): start_timer_thread(): CreateThread() failed, errno 0 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 00:49:26 (8544): start_timer_thread(): CreateThread() failed, errno 0 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 02:05:59 (8608): start_timer_thread(): CreateThread() failed, errno 0 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 03:18:11 (816): start_timer_thread(): CreateThread() failed, errno 0 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 06:26:54 (3100): start_timer_thread(): CreateThread() failed, errno 0 Model crashed: P_TH_ADJ : NEGATIVE PRESSURE VALUE CREATED. tmp/pipe_dummy 2048 08:28:38 (10212): start_timer_thread(): CreateThread() failed, errno 0 09:40:27 (7988): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 09:49:04 (5652): start_timer_thread(): CreateThread() failed, errno 0 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 diagnostics_init_unhandled_exception_monitor(): Creating hExceptionMonitorThread failed, errno 12 WARNING: BOINC Windows Runtime Debugger has been disabled. 09:52:33 (6444): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 diagnostics_init_unhandled_exception_monitor(): Creating hExceptionMonitorThread failed, errno 12 WARNING: BOINC Windows Runtime Debugger has been disabled. 12:36:19 (6652): start_timer_thread(): CreateThread() failed, errno 0 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 diagnostics_init_unhandled_exception_monitor(): Creating hExceptionMonitorThread failed, errno 12 WARNING: BOINC Windows Runtime Debugger has been disabled. 13:20:25 (5328): start_timer_thread(): CreateThread() failed, errno 0 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 13:53:06 (5452): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 16:47:16 (9044): start_timer_thread(): CreateThread() failed, errno 0 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 19:28:42 (5000): start_timer_thread(): CreateThread() failed, errno 0 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 19:38:36 (7824): start_timer_thread(): CreateThread() failed, errno 0 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 20:38:51 (3084): start_timer_thread(): CreateThread() failed, errno 0 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 20:57:23 (1440): start_timer_thread(): CreateThread() failed, errno 0 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 21:17:35 (5852): start_timer_thread(): CreateThread() failed, errno 0 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 22:12:59 (1008): start_timer_thread(): CreateThread() failed, errno 0 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 00:56:18 (8796): start_timer_thread(): CreateThread() failed, errno 0 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 03:39:49 (7480): start_timer_thread(): CreateThread() failed, errno 0 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 05:57:22 (2996): start_timer_thread(): CreateThread() failed, errno 0 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 06:35:25 (4652): start_timer_thread(): CreateThread() failed, errno 0 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 06:38:16 (8216): start_timer_thread(): CreateThread() failed, errno 0 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 06:41:29 (7816): start_timer_thread(): CreateThread() failed, errno 0 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 06:54:03 (6156): start_timer_thread(): CreateThread() failed, errno 0 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 09:15:13 (8768): start_timer_thread(): CreateThread() failed, errno 0 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 09:30:57 (7856): start_timer_thread(): CreateThread() failed, errno 0 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 09:31:15 (2616): start_timer_thread(): CreateThread() failed, errno 0 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 09:37:42 (4508): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 10:31:31 (8628): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 10:55:18 (4176): start_timer_thread(): CreateThread() failed, errno 0 Model crashed: INITDUMP: Wrong no of ocean prognostic fields tmp/pipe_dummy 2048 10:55:20 (5948): start_timer_thread(): CreateThread() failed, errno 0 Model crashed: INITDUMP: Wrong no of ocean prognostic fields tmp/pipe_dummy 2048 10:55:23 (9332): start_timer_thread(): CreateThread() failed, errno 0 Model crashed: INITDUMP: Wrong no of ocean prognostic fields tmp/pipe_dummy 2048 10:55:26 (9564): start_timer_thread(): CreateThread() failed, errno 0 Model crashed: INITDUMP: Wrong no of ocean prognostic fields tmp/pipe_dummy 2048 10:55:29 (9676): start_timer_thread(): CreateThread() failed, errno 0 Model crashed: INITDUMP: Wrong no of ocean prognostic fields tmp/pipe_dummy 2048 10:55:32 (9876): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=9152, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
19 Jun 2016 12:34:30	1398426	19750471	hadcm3n_xdms_198012_480_374_010413468_1	311,040	5,911	0.0190
19 Jun 2016 08:16:23	1398426	19750471	hadcm3n_xdms_198012_480_374_010413468_1	285,120	2,111	0.0074
19 Jun 2016 03:58:25	1398426	19750471	hadcm3n_xdms_198012_480_374_010413468_1	259,200	6,241	0.0241
18 Jun 2016 23:40:38	1398426	19750471	hadcm3n_xdms_198012_480_374_010413468_1	233,280	502	0.0022
18 Jun 2016 19:22:21	1398426	19750471	hadcm3n_xdms_198012_480_374_010413468_1	207,360	5,175	0.0250
18 Jun 2016 14:33:50	1398426	19750471	hadcm3n_xdms_198012_480_374_010413468_1	181,440	2,405	0.0133
18 Jun 2016 09:49:45	1398426	19750471	hadcm3n_xdms_198012_480_374_010413468_1	155,520	8,872	0.0570
18 Jun 2016 06:42:06	1398426	19750471	hadcm3n_xdms_198012_480_374_010413468_1	129,600	3,048	0.0235
18 Jun 2016 01:33:53	1398426	19750471	hadcm3n_xdms_198012_480_374_010413468_1	103,680	658	0.0063
17 Jun 2016 06:00:02	1398426	19750471	hadcm3n_xdms_198012_480_374_010413468_1	77,760	674	0.0087
17 Jun 2016 01:47:27	1398426	19750471	hadcm3n_xdms_198012_480_374_010413468_1	51,840	731	0.0141
16 Jun 2016 21:39:27	1398426	19750471	hadcm3n_xdms_198012_480_374_010413468_1	25,920	403	0.0155