Task 12784642

Name	famous_x51n_1699_200_007068370_1
Workunit	7271670
Created	7 Apr 2011, 22:44:46 UTC
Sent	7 Apr 2011, 22:55:54 UTC
Report deadline	8 Jul 2011, 6:23:05 UTC
Received	8 Apr 2011, 20:36:37 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	922557
Run time
CPU time	10 hours 31 min 23 sec
Validate state	Invalid
Credit	123.61
Device peak FLOPS	1.68 GFLOPS
Application version	UK Met Office FAMOUS v6.11 i686-pc-linux-gnu
Stderr	<core_client_version>5.10.45</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> _1699_200_007068370 - PH 1 TS 0005461 A - 01/07/1700 01:00 - H:M:S=0001:46:07 AVG= 1.17 DLT= 1.36 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model restart required... Preparing for restart attempt # 1... Starting model ID famous_x51n_1699_200_007068370 Phase 1 Getting pthread attributes - retval=0 Setting pthread size (62914560 bytes) - retval=0 Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 133480 /home/jsc/BOINC/slots/0 Program launched with process id # 31782 Climate model starting - use graphics to monitor progress. Or visit the website to see the graphs for this run. famous_x51n_1699_200_007068370 - PH 1 TS 0005851 A - 16/07/1700 01:00 - H:M:S=0001:52:01 AVG= 1.15 DLT= 1.39 famous_x51n_1699_200_007068370 - PH 1 TS 0006241 A - 01/08/1700 01:00 - H:M:S=0001:57:06 AVG= 1.13 DLT= 1.33 famous_x51n_1699_200_007068370 - PH 1 TS 0006631 A - 16/08/1700 01:00 - H:M:S=0002:02:12 AVG= 1.11 DLT= 1.50 famous_x51n_1699_200_007068370 - PH 1 TS 0007021 A - 01/09/1700 01:00 - H:M:S=0002:07:18 AVG= 1.09 DLT= 1.52 famous_x51n_1699_200_007068370 - PH 1 TS 0007411 A - 16/09/1700 01:00 - H:M:S=0002:12:26 AVG= 1.07 DLT= 1.14 famous_x51n_1699_200_007068370 - PH 1 TS 0007801 A - 01/10/1700 01:00 - H:M:S=0002:17:32 AVG= 1.06 DLT= 1.34 famous_x51n_1699_200_007068370 - PH 1 TS 0008191 A - 16/10/1700 01:00 - H:M:S=0002:22:39 AVG= 1.04 DLT= 1.50 famous_x51n_1699_200_007068370 - PH 1 TS 0008581 A - 01/11/1700 01:00 - H:M:S=0002:27:46 AVG= 1.03 DLT= 1.30 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model restart required... Preparing for restart attempt # 1... Starting model ID famous_x51n_1699_200_007068370 Phase 1 Getting pthread attributes - retval=0 Setting pthread size (62914560 bytes) - retval=0 Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 133480 /home/jsc/BOINC/slots/0 Program launched with process id # 729 Climate model starting - use graphics to monitor progress. Or visit the website to see the graphs for this run. famous_x51n_1699_200_007068370 - PH 1 TS 0008971 A - 16/11/1700 01:00 - H:M:S=0002:36:43 AVG= 1.05 DLT= 1.25 Trickling yearly means for 1700 Building file list for 1700 in dataout -- 4 entries Creating trickle file trickle_famous_x51n_1699_200_007068370_1_1700.zip for upload... famous_x51n_1699_200_007068370 - PH 1 TS 0009361 A - 01/12/1700 01:00 - H:M:S=0002:41:44 AVG= 1.04 DLT= 1.72 famous_x51n_1699_200_007068370 - PH 1 TS 0009751 A - 16/12/1700 01:00 - H:M:S=0002:46:44 AVG= 1.03 DLT= 1.60 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model restart required... Preparing for restart attempt # 1... Starting model ID famous_x51n_1699_200_007068370 Phase 1 Getting pthread attributes - retval=0 Setting pthread size (62914560 bytes) - retval=0 Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 133480 /home/jsc/BOINC/slots/0 Program launched with process id # 1326 Climate model starting - use graphics to monitor progress. Or visit the website to see the graphs for this run. famous_x51n_1699_200_007068370 - PH 1 TS 0010141 A - 01/01/1701 01:00 - H:M:S=0002:54:00 AVG= 1.03 DLT= 1.22 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model restart required... Preparing for restart attempt # 1... Starting model ID famous_x51n_1699_200_007068370 Phase 1 Getting pthread attributes - retval=0 Setting pthread size (62914560 bytes) - retval=0 Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 133480 /home/jsc/BOINC/slots/0 Program launched with process id # 1695 Climate model starting - use graphics to monitor progress. Or visit the website to see the graphs for this run. famous_x51n_1699_200_007068370 - PH 1 TS 0010531 A - 16/01/1701 01:00 - H:M:S=0003:01:31 AVG= 1.03 DLT= 1.50 famous_x51n_1699_200_007068370 - PH 1 TS 0010921 A - 01/02/1701 01:00 - H:M:S=0003:06:32 AVG= 1.02 DLT= 1.63 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model restart required... Preparing for restart attempt # 1... Starting model ID famous_x51n_1699_200_007068370 Phase 1 Getting pthread attributes - retval=0 Setting pthread size (62914560 bytes) - retval=0 Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 133480 /home/jsc/BOINC/slots/0 Program launched with process id # 2336 Climate model starting - use graphics to monitor progress. Or visit the website to see the graphs for this run. Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model restart required... Preparing for restart attempt # 2... Starting model ID famous_x51n_1699_200_007068370 Phase 1 Getting pthread attributes - retval=0 Setting pthread size (62914560 bytes) - retval=0 Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 133480 /home/jsc/BOINC/slots/0 Program launched with process id # 2410 Climate model starting - use graphics to monitor progress. Or visit the website to see the graphs for this run. Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model restart required... Preparing for restart attempt # 3... Starting model ID famous_x51n_1699_200_007068370 Phase 1 Getting pthread attributes - retval=0 Setting pthread size (62914560 bytes) - retval=0 Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 133480 /home/jsc/BOINC/slots/0 Program launched with process id # 2495 Climate model starting - use graphics to monitor progress. Or visit the website to see the graphs for this run. Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model restart required... Preparing for restart attempt # 4... Starting model ID famous_x51n_1699_200_007068370 Phase 1 Getting pthread attributes - retval=0 Setting pthread size (62914560 bytes) - retval=0 Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 133480 /home/jsc/BOINC/slots/0 Program launched with process id # 2565 Climate model starting - use graphics to monitor progress. Or visit the website to see the graphs for this run. Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model restart required... Preparing for restart attempt # 5... Starting model ID famous_x51n_1699_200_007068370 Phase 1 Getting pthread attributes - retval=0 Setting pthread size (62914560 bytes) - retval=0 Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 133480 /home/jsc/BOINC/slots/0 Program launched with process id # 2670 Climate model starting - use graphics to monitor progress. Or visit the website to see the graphs for this run. famous_x51n_1699_200_007068370 - PH 1 TS 0011311 A - 16/02/1701 01:00 - H:M:S=0003:22:27 AVG= 1.07 DLT= 1.22 famous_x51n_1699_200_007068370 - PH 1 TS 0011701 A - 01/03/1701 01:00 - H:M:S=0003:27:30 AVG= 1.06 DLT= 1.18 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model restart required... Preparing for restart attempt # 1... Starting model ID famous_x51n_1699_200_007068370 Phase 1 Getting pthread attributes - retval=0 Setting pthread size (62914560 bytes) - retval=0 Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 133480 /home/jsc/BOINC/slots/0 Program launched with process id # 3132 Climate model starting - use graphics to monitor progress. Or visit the website to see the graphs for this run. Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model restart required... Preparing for restart attempt # 2... Starting model ID famous_x51n_1699_200_007068370 Phase 1 Getting pthread attributes - retval=0 Setting pthread size (62914560 bytes) - retval=0 Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 133480 /home/jsc/BOINC/slots/0 Program launched with process id # 3208 Climate model starting - use graphics to monitor progress. Or visit the website to see the graphs for this run. Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model restart required... Preparing for restart attempt # 3... Starting model ID famous_x51n_1699_200_007068370 Phase 1 Getting pthread attributes - retval=0 Setting pthread size (62914560 bytes) - retval=0 Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 133480 /home/jsc/BOINC/slots/0 Program launched with process id # 3227 Climate model starting - use graphics to monitor progress. Or visit the website to see the graphs for this run. famous_x51n_1699_200_007068370 - PH 1 TS 0012091 A - 16/03/1701 01:00 - H:M:S=0003:36:41 AVG= 1.08 DLT= 1.17 famous_x51n_1699_200_007068370 - PH 1 TS 0012481 A - 01/04/1701 01:00 - H:M:S=0003:41:43 AVG= 1.07 DLT= 1.71 famous_x51n_1699_200_007068370 - PH 1 TS 0012871 A - 16/04/1701 01:00 - H:M:S=0003:46:45 AVG= 1.06 DLT= 1.20 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model restart required... Preparing for restart attempt # 1... Starting model ID famous_x51n_1699_200_007068370 Phase 1 Getting pthread attributes - retval=0 Setting pthread size (62914560 bytes) - retval=0 Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 133480 /home/jsc/BOINC/slots/0 Program launched with process id # 3969 Climate model starting - use graphics to monitor progress. Or visit the website to see the graphs for this run. famous_x51n_1699_200_007068370 - PH 1 TS 0013261 A - 01/05/1701 01:00 - H:M:S=0003:56:48 AVG= 1.07 DLT= 1.20 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model restart required... Preparing for restart attempt # 1... Starting model ID famous_x51n_1699_200_007068370 Phase 1 Getting pthread attributes - retval=0 Setting pthread size (62914560 bytes) - retval=0 Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 133480 /home/jsc/BOINC/slots/0 Program launched with process id # 4197 Climate model starting - use graphics to monitor progress. Or visit the website to see the graphs for this run. famous_x51n_1699_200_007068370 - PH 1 TS 0013651 A - 16/05/1701 01:00 - H:M:S=0004:03:11 AVG= 1.07 DLT= 1.53 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model restart required... Preparing for restart attempt # 1... Starting model ID famous_x51n_1699_200_007068370 Phase 1 Getting pthread attributes - retval=0 Setting pthread size (62914560 bytes) - retval=0 Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 133480 /home/jsc/BOINC/slots/0 Program launched with process id # 4470 Climate model starting - use graphics to monitor progress. Or visit the website to see the graphs for this run. Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model restart required... Preparing for restart attempt # 2... Starting model ID famous_x51n_1699_200_007068370 Phase 1 Getting pthread attributes - retval=0 Setting pthread size (62914560 bytes) - retval=0 Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 133480 /home/jsc/BOINC/slots/0 Program launched with process id # 4472 Climate model starting - use graphics to monitor progress. Or visit the website to see the graphs for this run. famous_x51n_1699_200_007068370 - PH 1 TS 0014041 A - 01/06/1701 01:00 - H:M:S=0004:10:31 AVG= 1.07 DLT= 1.33 famous_x51n_1699_200_007068370 - PH 1 TS 0014431 A - 16/06/1701 01:00 - H:M:S=0004:15:31 AVG= 1.06 DLT= 1.48 famous_x51n_1699_200_007068370 - PH 1 TS 0014821 A - 01/07/1701 01:00 - H:M:S=0004:20:32 AVG= 1.05 DLT= 1.48 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model restart required... Preparing for restart attempt # 1... Starting model ID famous_x51n_1699_200_007068370 Phase 1 Getting pthread attributes - retval=0 Setting pthread size (62914560 bytes) - retval=0 Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 133480 /home/jsc/BOINC/slots/0 Program launched with process id # 5102 Climate model starting - use graphics to monitor progress. Or visit the website to see the graphs for this run. famous_x51n_1699_200_007068370 - PH 1 TS 0015211 A - 16/07/1701 01:00 - H:M:S=0004:28:19 AVG= 1.06 DLT= 1.34 famous_x51n_1699_200_007068370 - PH 1 TS 0015601 A - 01/08/1701 01:00 - H:M:S=0004:33:23 AVG= 1.05 DLT= 1.25 famous_x51n_1699_200_007068370 - PH 1 TS 0015991 A - 16/08/1701 01:00 - H:M:S=0004:38:29 AVG= 1.04 DLT= 1.24 famous_x51n_1699_200_007068370 - PH 1 TS 0016381 A - 01/09/1701 01:00 - H:M:S=0004:43:35 AVG= 1.04 DLT= 1.17 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model restart required... Preparing for restart attempt # 1... Starting model ID famous_x51n_1699_200_007068370 Phase 1 Getting pthread attributes - retval=0 Setting pthread size (62914560 bytes) - retval=0 Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 133480 /home/jsc/BOINC/slots/0 Program launched with process id # 5835 Climate model starting - use graphics to monitor progress. Or visit the website to see the graphs for this run. famous_x51n_1699_200_007068370 - PH 1 TS 0016771 A - 16/09/1701 01:00 - H:M:S=0004:49:53 AVG= 1.04 DLT= 1.79 famous_x51n_1699_200_007068370 - PH 1 TS 0017161 A - 01/10/1701 01:00 - H:M:S=0004:55:00 AVG= 1.03 DLT= 1.45 CPDN Monitor - Quit request from BOINC... Detaching shared memory... Starting work on result famous_x51n_1699_200_007068370_1... Starting model in /home/jsc/BOINC/projects/climateprediction.net... Trying to kill old process # 28830 Trying to kill old process # 5835 Created shared memory region key = 133480 of size 743572 bytes (version 608) Starting model ID famous_x51n_1699_200_007068370 Phase 1 Getting pthread attributes - retval=0 Setting pthread size (62914560 bytes) - retval=0 Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 133480 /home/jsc/BOINC/slots/0 Program launched with process id # 6356 Climate model starting - use graphics to monitor progress. Or visit the website to see the graphs for this run. famous_x51n_1699_200_007068370 - PH 1 TS 0017161 A - 01/10/1701 01:00 - H:M:S=0004:55:00 AVG= 1.03 DLT= 0.00 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model restart required... Preparing for restart attempt # 1... Starting model ID famous_x51n_1699_200_007068370 Phase 1 Program launched with process id # 6466 Climate model starting - use graphics to monitor progress. Or visit the website to see the graphs for this run. Getting pthread attributes - retval=0 Setting pthread size (62914560 bytes) - retval=0 Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 133480 /home/jsc/BOINC/slots/0 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model restart required... Preparing for restart attempt # 2... Starting model ID famous_x51n_1699_200_007068370 Phase 1 Getting pthread attributes - retval=0 Setting pthread size (62914560 bytes) - retval=0 Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 133480 /home/jsc/BOINC/slots/0 Program launched with process id # 6468 Climate model starting - use graphics to monitor progress. Or visit the website to see the graphs for this run. famous_x51n_1699_200_007068370 - PH 1 TS 0017551 A - 16/10/1701 01:00 - H:M:S=0005:03:29 AVG= 1.04 DLT= 1.57 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model restart required... Preparing for restart attempt # 1... Starting model ID famous_x51n_1699_200_007068370 Phase 1 Getting pthread attributes - retval=0 Setting pthread size (62914560 bytes) - retval=0 Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 133480 /home/jsc/BOINC/slots/0 Program launched with process id # 6729 Climate model starting - use graphics to monitor progress. Or visit the website to see the graphs for this run. famous_x51n_1699_200_007068370 - PH 1 TS 0017941 A - 01/11/1701 01:00 - H:M:S=0005:10:46 AVG= 1.04 DLT= 1.42 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model restart required... Preparing for restart attempt # 1... Starting model ID famous_x51n_1699_200_007068370 Phase 1 Getting pthread attributes - retval=0 Setting pthread size (62914560 bytes) - retval=0 Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 133480 /home/jsc/BOINC/slots/0 Program launched with process id # 7060 Climate model starting - use graphics to monitor progress. Or visit the website to see the graphs for this run. Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model restart required... Preparing for restart attempt # 2... Starting model ID famous_x51n_1699_200_007068370 Phase 1 Getting pthread attributes - retval=0 Setting pthread size (62914560 bytes) - retval=0 Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 133480 /home/jsc/BOINC/slots/0 Program launched with process id # 7193 Climate model starting - use graphics to monitor progress. Or visit the website to see the graphs for this run. Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model restart required... Preparing for restart attempt # 3... Starting model ID famous_x51n_1699_200_007068370 Phase 1 Getting pthread attributes - retval=0 Setting pthread size (62914560 bytes) - retval=0 Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 133480 /home/jsc/BOINC/slots/0 Program launched with process id # 7247 Climate model starting - use graphics to monitor progress. Or visit the website to see the graphs for this run. famous_x51n_1699_200_007068370 - PH 1 TS 0018331 A - 16/11/1701 01:00 - H:M:S=0005:26:32 AVG= 1.07 DLT= 1.17 Trickling yearly means for 1701 Building file list for 1701 in dataout -- 4 entries Creating trickle file trickle_famous_x51n_1699_200_007068370_1_1701.zip for upload... famous_x51n_1699_200_007068370 - PH 1 TS 0018721 A - 01/12/1701 01:00 - H:M:S=0005:31:36 AVG= 1.06 DLT= 1.39 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model restart required... Preparing for restart attempt # 1... Starting model ID famous_x51n_1699_200_007068370 Phase 1 Getting pthread attributes - retval=0 Setting pthread size (62914560 bytes) - retval=0 Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 133480 /home/jsc/BOINC/slots/0 Program launched with process id # 7647 Climate model starting - use graphics to monitor progress. Or visit the website to see the graphs for this run. BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 famous_x51n_1699_200_007068370 - PH 1 TS 0019111 A - 16/12/1701 01:00 - H:M:S=0005:38:37 AVG= 1.06 DLT= 1.76 famous_x51n_1699_200_007068370 - PH 1 TS 0019501 A - 01/01/1702 01:00 - H:M:S=0005:43:36 AVG= 1.06 DLT= 1.84 famous_x51n_1699_200_007068370 - PH 1 TS 0019891 A - 16/01/1702 01:00 - H:M:S=0005:48:36 AVG= 1.05 DLT= 1.64 famous_x51n_1699_200_007068370 - PH 1 TS 0020281 A - 01/02/1702 01:00 - H:M:S=0005:53:34 AVG= 1.05 DLT= 1.38 famous_x51n_1699_200_007068370 - PH 1 TS 0020671 A - 16/02/1702 01:00 - H:M:S=0005:58:33 AVG= 1.04 DLT= 1.57 famous_x51n_1699_200_007068370 - PH 1 TS 0021061 A - 01/03/1702 01:00 - H:M:S=0006:03:33 AVG= 1.04 DLT= 1.61 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model restart required... Preparing for restart attempt # 1... Starting model ID famous_x51n_1699_200_007068370 Phase 1 Getting pthread attributes - retval=0 Setting pthread size (62914560 bytes) - retval=0 Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 133480 /home/jsc/BOINC/slots/0 Program launched with process id # 8674 Climate model starting - use graphics to monitor progress. Or visit the website to see the graphs for this run. famous_x51n_1699_200_007068370 - PH 1 TS 0021451 A - 16/03/1702 01:00 - H:M:S=0006:09:02 AVG= 1.03 DLT= 1.44 famous_x51n_1699_200_007068370 - PH 1 TS 0021841 A - 01/04/1702 01:00 - H:M:S=0006:14:06 AVG= 1.03 DLT= 1.80 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model restart required... Preparing for restart attempt # 1... Starting model ID famous_x51n_1699_200_007068370 Phase 1 Getting pthread attributes - retval=0 Setting pthread size (62914560 bytes) - retval=0 Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 133480 /home/jsc/BOINC/slots/0 Program launched with process id # 9119 Climate model starting - use graphics to monitor progress. Or visit the website to see the graphs for this run. famous_x51n_1699_200_007068370 - PH 1 TS 0022231 A - 16/04/1702 01:00 - H:M:S=0006:23:17 AVG= 1.03 DLT= 1.19 famous_x51n_1699_200_007068370 - PH 1 TS 0022621 A - 01/05/1702 01:00 - H:M:S=0006:28:19 AVG= 1.03 DLT= 1.23 famous_x51n_1699_200_007068370 - PH 1 TS 0023011 A - 16/05/1702 01:00 - H:M:S=0006:33:20 AVG= 1.03 DLT= 1.67 famous_x51n_1699_200_007068370 - PH 1 TS 0023401 A - 01/06/1702 01:00 - H:M:S=0006:38:23 AVG= 1.02 DLT= 1.60 famous_x51n_1699_200_007068370 - PH 1 TS 0023791 A - 16/06/1702 01:00 - H:M:S=0006:43:22 AVG= 1.02 DLT= 1.46 famous_x51n_1699_200_007068370 - PH 1 TS 0024181 A - 01/07/1702 01:00 - H:M:S=0006:48:23 AVG= 1.01 DLT= 1.41 famous_x51n_1699_200_007068370 - PH 1 TS 0024571 A - 16/07/1702 01:00 - H:M:S=0006:53:24 AVG= 1.01 DLT= 1.41 famous_x51n_1699_200_007068370 - PH 1 TS 0024961 A - 01/08/1702 01:00 - H:M:S=0006:58:24 AVG= 1.01 DLT= 1.47 famous_x51n_1699_200_007068370 - PH 1 TS 0025351 A - 16/08/1702 01:00 - H:M:S=0007:03:27 AVG= 1.00 DLT= 1.41 famous_x51n_1699_200_007068370 - PH 1 TS 0025741 A - 01/09/1702 01:00 - H:M:S=0007:08:32 AVG= 1.00 DLT= 1.24 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model restart required... Preparing for restart attempt # 1... Starting model ID famous_x51n_1699_200_007068370 Phase 1 Getting pthread attributes - retval=0 Setting pthread size (62914560 bytes) - retval=0 Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 133480 /home/jsc/BOINC/slots/0 Program launched with process id # 10830 Climate model starting - use graphics to monitor progress. Or visit the website to see the graphs for this run. famous_x51n_1699_200_007068370 - PH 1 TS 0026131 A - 16/09/1702 01:00 - H:M:S=0007:14:13 AVG= 1.00 DLT= 1.39 famous_x51n_1699_200_007068370 - PH 1 TS 0026521 A - 01/10/1702 01:00 - H:M:S=0007:19:18 AVG= 0.99 DLT= 1.60 famous_x51n_1699_200_007068370 - PH 1 TS 0026911 A - 16/10/1702 01:00 - H:M:S=0007:24:24 AVG= 0.99 DLT= 1.72 famous_x51n_1699_200_007068370 - PH 1 TS 0027301 A - 01/11/1702 01:00 - H:M:S=0007:29:26 AVG= 0.99 DLT= 1.15 famous_x51n_1699_200_007068370 - PH 1 TS 0027691 A - 16/11/1702 01:00 - H:M:S=0007:34:27 AVG= 0.98 DLT= 1.21 Trickling yearly means for 1702 Building file list for 1702 in dataout -- 4 entries Creating trickle file trickle_famous_x51n_1699_200_007068370_1_1702.zip for upload... famous_x51n_1699_200_007068370 - PH 1 TS 0028081 A - 01/12/1702 01:00 - H:M:S=0007:39:27 AVG= 0.98 DLT= 1.61 famous_x51n_1699_200_007068370 - PH 1 TS 0028471 A - 16/12/1702 01:00 - H:M:S=0007:44:30 AVG= 0.98 DLT= 1.18 famous_x51n_1699_200_007068370 - PH 1 TS 0028861 A - 01/01/1703 01:00 - H:M:S=0007:49:28 AVG= 0.98 DLT= 1.64 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model restart required... Preparing for restart attempt # 1... Starting model ID famous_x51n_1699_200_007068370 Phase 1 Getting pthread attributes - retval=0 Setting pthread size (62914560 bytes) - retval=0 Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 133480 /home/jsc/BOINC/slots/0 Program launched with process id # 12166 Climate model starting - use graphics to monitor progress. Or visit the website to see the graphs for this run. famous_x51n_1699_200_007068370 - PH 1 TS 0028861 A - 01/01/1703 01:00 - H:M:S=0007:49:43 AVG= 0.98 DLT= 0.00 famous_x51n_1699_200_007068370 - PH 1 TS 0029251 A - 16/01/1703 01:00 - H:M:S=0007:54:44 AVG= 0.97 DLT= 1.23 famous_x51n_1699_200_007068370 - PH 1 TS 0029641 A - 01/02/1703 01:00 - H:M:S=0007:59:44 AVG= 0.97 DLT= 1.53 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model restart required... Preparing for restart attempt # 1... Starting model ID famous_x51n_1699_200_007068370 Phase 1 Getting pthread attributes - retval=0 Setting pthread size (62914560 bytes) - retval=0 Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 133480 /home/jsc/BOINC/slots/0 Program launched with process id # 12518 Climate model starting - use graphics to monitor progress. Or visit the website to see the graphs for this run. famous_x51n_1699_200_007068370 - PH 1 TS 0030031 A - 16/02/1703 01:00 - H:M:S=0008:05:17 AVG= 0.97 DLT= 1.27 famous_x51n_1699_200_007068370 - PH 1 TS 0030421 A - 01/03/1703 01:00 - H:M:S=0008:10:17 AVG= 0.97 DLT= 1.29 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model restart required... Preparing for restart attempt # 1... Starting model ID famous_x51n_1699_200_007068370 Phase 1 Getting pthread attributes - retval=0 Setting pthread size (62914560 bytes) - retval=0 Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 133480 /home/jsc/BOINC/slots/0 Program launched with process id # 12887 Climate model starting - use graphics to monitor progress. Or visit the website to see the graphs for this run. famous_x51n_1699_200_007068370 - PH 1 TS 0030811 A - 16/03/1703 01:00 - H:M:S=0008:15:59 AVG= 0.97 DLT= 1.52 famous_x51n_1699_200_007068370 - PH 1 TS 0031201 A - 01/04/1703 01:00 - H:M:S=0008:21:01 AVG= 0.96 DLT= 1.65 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model restart required... Preparing for restart attempt # 1... Starting model ID famous_x51n_1699_200_007068370 Phase 1 Getting pthread attributes - retval=0 Setting pthread size (62914560 bytes) - retval=0 Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 133480 /home/jsc/BOINC/slots/0 Program launched with process id # 13401 Climate model starting - use graphics to monitor progress. Or visit the website to see the graphs for this run. famous_x51n_1699_200_007068370 - PH 1 TS 0031591 A - 16/04/1703 01:00 - H:M:S=0008:30:16 AVG= 0.97 DLT= 1.25 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model restart required... Preparing for restart attempt # 1... Starting model ID famous_x51n_1699_200_007068370 Phase 1 Getting pthread attributes - retval=0 Setting pthread size (62914560 bytes) - retval=0 Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 133480 /home/jsc/BOINC/slots/0 Program launched with process id # 13646 Climate model starting - use graphics to monitor progress. Or visit the website to see the graphs for this run. famous_x51n_1699_200_007068370 - PH 1 TS 0031981 A - 01/05/1703 01:00 - H:M:S=0008:37:16 AVG= 0.97 DLT= 1.63 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model restart required... Preparing for restart attempt # 1... Starting model ID famous_x51n_1699_200_007068370 Phase 1 Getting pthread attributes - retval=0 Setting pthread size (62914560 bytes) - retval=0 Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 133480 /home/jsc/BOINC/slots/0 Program launched with process id # 13891 Climate model starting - use graphics to monitor progress. Or visit the website to see the graphs for this run. famous_x51n_1699_200_007068370 - PH 1 TS 0032371 A - 16/05/1703 01:00 - H:M:S=0008:44:28 AVG= 0.97 DLT= 1.40 famous_x51n_1699_200_007068370 - PH 1 TS 0032761 A - 01/06/1703 01:00 - H:M:S=0008:49:33 AVG= 0.97 DLT= 1.64 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model restart required... Preparing for restart attempt # 1... Starting model ID famous_x51n_1699_200_007068370 Phase 1 Program launched with process id # 14265 Climate model starting - use graphics to monitor progress. Or visit the website to see the graphs for this run. Getting pthread attributes - retval=0 Setting pthread size (62914560 bytes) - retval=0 Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 133480 /home/jsc/BOINC/slots/0 famous_x51n_1699_200_007068370 - PH 1 TS 0033151 A - 16/06/1703 01:00 - H:M:S=0008:55:00 AVG= 0.97 DLT= 1.89 famous_x51n_1699_200_007068370 - PH 1 TS 0033541 A - 01/07/1703 01:00 - H:M:S=0009:00:02 AVG= 0.97 DLT= 1.49 famous_x51n_1699_200_007068370 - PH 1 TS 0033931 A - 16/07/1703 01:00 - H:M:S=0009:05:04 AVG= 0.96 DLT= 1.48 famous_x51n_1699_200_007068370 - PH 1 TS 0034321 A - 01/08/1703 01:00 - H:M:S=0009:10:07 AVG= 0.96 DLT= 1.57 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model restart required... Preparing for restart attempt # 1... Starting model ID famous_x51n_1699_200_007068370 Phase 1 Program launched with process id # 15176 Climate model starting - use graphics to monitor progress. Or visit the website to see the graphs for this run. Getting pthread attributes - retval=0 Setting pthread size (62914560 bytes) - retval=0 Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 133480 /home/jsc/BOINC/slots/0 famous_x51n_1699_200_007068370 - PH 1 TS 0034711 A - 16/08/1703 01:00 - H:M:S=0009:16:01 AVG= 0.96 DLT= 1.58 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model restart required... Preparing for restart attempt # 1... Starting model ID famous_x51n_1699_200_007068370 Phase 1 Getting pthread attributes - retval=0 Setting pthread size (62914560 bytes) - retval=0 Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 133480 /home/jsc/BOINC/slots/0 Program launched with process id # 15704 Climate model starting - use graphics to monitor progress. Or visit the website to see the graphs for this run. famous_x51n_1699_200_007068370 - PH 1 TS 0035101 A - 01/09/1703 01:00 - H:M:S=0009:25:23 AVG= 0.97 DLT= 1.29 famous_x51n_1699_200_007068370 - PH 1 TS 0035491 A - 16/09/1703 01:00 - H:M:S=0009:30:29 AVG= 0.96 DLT= 1.51 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model restart required... Preparing for restart attempt # 1... Starting model ID famous_x51n_1699_200_007068370 Phase 1 Getting pthread attributes - retval=0 Setting pthread size (62914560 bytes) - retval=0 Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 133480 /home/jsc/BOINC/slots/0 Program launched with process id # 16098 Climate model starting - use graphics to monitor progress. Or visit the website to see the graphs for this run. Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model restart required... Preparing for restart attempt # 2... Starting model ID famous_x51n_1699_200_007068370 Phase 1 Program launched with process id # 16139 Climate model starting - use graphics to monitor progress. Or visit the website to see the graphs for this run. Getting pthread attributes - retval=0 Setting pthread size (62914560 bytes) - retval=0 Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 133480 /home/jsc/BOINC/slots/0 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model restart required... Preparing for restart attempt # 3... Starting model ID famous_x51n_1699_200_007068370 Phase 1 Getting pthread attributes - retval=0 Setting pthread size (62914560 bytes) - retval=0 Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 133480 /home/jsc/BOINC/slots/0 Program launched with process id # 16256 Climate model starting - use graphics to monitor progress. Or visit the website to see the graphs for this run. Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model restart required... Preparing for restart attempt # 4... Starting model ID famous_x51n_1699_200_007068370 Phase 1 Getting pthread attributes - retval=0 Setting pthread size (62914560 bytes) - retval=0 Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 133480 /home/jsc/BOINC/slots/0 Program launched with process id # 16335 Climate model starting - use graphics to monitor progress. Or visit the website to see the graphs for this run. Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model restart required... Preparing for restart attempt # 5... Starting model ID famous_x51n_1699_200_007068370 Phase 1 Program launched with process id # 16393 Climate model starting - use graphics to monitor progress. Or visit the website to see the graphs for this run. Getting pthread attributes - retval=0 Setting pthread size (62914560 bytes) - retval=0 Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 133480 /home/jsc/BOINC/slots/0 famous_x51n_1699_200_007068370 - PH 1 TS 0035881 A - 01/10/1703 01:00 - H:M:S=0009:44:46 AVG= 0.98 DLT= 1.12 famous_x51n_1699_200_007068370 - PH 1 TS 0036271 A - 16/10/1703 01:00 - H:M:S=0009:49:51 AVG= 0.98 DLT= 1.41 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model restart required... Preparing for restart attempt # 1... Starting model ID famous_x51n_1699_200_007068370 Phase 1 Getting pthread attributes - retval=0 Setting pthread size (62914560 bytes) - retval=0 Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 133480 /home/jsc/BOINC/slots/0 Program launched with process id # 16830 Climate model starting - use graphics to monitor progress. Or visit the website to see the graphs for this run. Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model restart required... Preparing for restart attempt # 2... Starting model ID famous_x51n_1699_200_007068370 Phase 1 Program launched with process id # 16832 Climate model starting - use graphics to monitor progress. Or visit the website to see the graphs for this run. Getting pthread attributes - retval=0 Setting pthread size (62914560 bytes) - retval=0 Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 133480 /home/jsc/BOINC/slots/0 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model restart required... Preparing for restart attempt # 3... Starting model ID famous_x51n_1699_200_007068370 Phase 1 Getting pthread attributes - retval=0 Setting pthread size (62914560 bytes) - retval=0 Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 133480 /home/jsc/BOINC/slots/0 Program launched with process id # 16941 Climate model starting - use graphics to monitor progress. Or visit the website to see the graphs for this run. famous_x51n_1699_200_007068370 - PH 1 TS 0036661 A - 01/11/1703 01:00 - H:M:S=0009:58:20 AVG= 0.98 DLT= 1.32 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model restart required... Preparing for restart attempt # 1... Starting model ID famous_x51n_1699_200_007068370 Phase 1 Program launched with process id # 17236 Climate model starting - use graphics to monitor progress. Or visit the website to see the graphs for this run. Getting pthread attributes - retval=0 Setting pthread size (62914560 bytes) - retval=0 Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 133480 /home/jsc/BOINC/slots/0 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model restart required... Preparing for restart attempt # 2... Starting model ID famous_x51n_1699_200_007068370 Phase 1 Getting pthread attributes - retval=0 Setting pthread size (62914560 bytes) - retval=0 Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 133480 /home/jsc/BOINC/slots/0 Program launched with process id # 17307 Climate model starting - use graphics to monitor progress. Or visit the website to see the graphs for this run. famous_x51n_1699_200_007068370 - PH 1 TS 0037051 A - 16/11/1703 01:00 - H:M:S=0010:06:26 AVG= 0.98 DLT= 1.25 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model restart required... Preparing for restart attempt # 1... Starting model ID famous_x51n_1699_200_007068370 Phase 1 Program launched with process id # 17752 Climate model starting - use graphics to monitor progress. Or visit the website to see the graphs for this run. Getting pthread attributes - retval=0 Setting pthread size (62914560 bytes) - retval=0 Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 133480 /home/jsc/BOINC/slots/0 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model restart required... Preparing for restart attempt # 2... Starting model ID famous_x51n_1699_200_007068370 Phase 1 Program launched with process id # 17831 Climate model starting - use graphics to monitor progress. Or visit the website to see the graphs for this run. Getting pthread attributes - retval=0 Setting pthread size (62914560 bytes) - retval=0 Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 133480 /home/jsc/BOINC/slots/0 Trickling yearly means for 1703 Building file list for 1703 in dataout -- 4 entries Creating trickle file trickle_famous_x51n_1699_200_007068370_1_1703.zip for upload... famous_x51n_1699_200_007068370 - PH 1 TS 0037441 A - 01/12/1703 01:00 - H:M:S=0010:17:58 AVG= 0.99 DLT= 1.38 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model restart required... Preparing for restart attempt # 1... Starting model ID famous_x51n_1699_200_007068370 Phase 1 Program launched with process id # 18266 Climate model starting - use graphics to monitor progress. Or visit the website to see the graphs for this run. Getting pthread attributes - retval=0 Setting pthread size (62914560 bytes) - retval=0 Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 133480 /home/jsc/BOINC/slots/0 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model restart required... Preparing for restart attempt # 2... Starting model ID famous_x51n_1699_200_007068370 Phase 1 Getting pthread attributes - retval=0 Setting pthread size (62914560 bytes) - retval=0 Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 133480 /home/jsc/BOINC/slots/0 Program launched with process id # 18416 Climate model starting - use graphics to monitor progress. Or visit the website to see the graphs for this run. BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model restart required... Preparing for restart attempt # 3... Starting model ID famous_x51n_1699_200_007068370 Phase 1 Getting pthread attributes - retval=0 Setting pthread size (62914560 bytes) - retval=0 Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 133480 /home/jsc/BOINC/slots/0 Program launched with process id # 18600 Climate model starting - use graphics to monitor progress. Or visit the website to see the graphs for this run. BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model restart required... Preparing for restart attempt # 4... Starting model ID famous_x51n_1699_200_007068370 Phase 1 Getting pthread attributes - retval=0 Setting pthread size (62914560 bytes) - retval=0 Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 133480 /home/jsc/BOINC/slots/0 Program launched with process id # 18602 Climate model starting - use graphics to monitor progress. Or visit the website to see the graphs for this run. BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model restart required... Preparing for restart attempt # 5... Starting model ID famous_x51n_1699_200_007068370 Phase 1 Getting pthread attributes - retval=0 Setting pthread size (62914560 bytes) - retval=0 Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 133480 /home/jsc/BOINC/slots/0 Program launched with process id # 18604 Climate model starting - use graphics to monitor progress. Or visit the website to see the graphs for this run. BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Sorry, too many model crashes! :-( Model restart required... Sorry, too many model crashes! :-( Cleaning up from the run... (6354): called boinc_finish Detaching shared memory... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
08 Apr 2011 20:10:17	922557	12784642	famous_x51n_1699_200_007068370_1	37,466	37,078	0.9896
08 Apr 2011 14:35:19	922557	12784642	famous_x51n_1699_200_007068370_1	28,106	27,567	0.9808
08 Apr 2011 10:18:30	922557	12784642	famous_x51n_1699_200_007068370_1	18,746	19,896	1.0613
08 Apr 2011 04:45:55	922557	12784642	famous_x51n_1699_200_007068370_1	9,386	9,704	1.0339