climateprediction.net home page
Task 19792216

Task 19792216

Name wah2_sas50_fmll_201412_13_370_010399847_1
Workunit 10399847
Created 4 Jul 2016, 11:31:53 UTC
Sent 10 Jul 2016, 13:43:17 UTC
Report deadline 22 Jun 2017, 19:03:17 UTC
Received 28 Jul 2016, 9:45:51 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 193 (0x000000C1) EXIT_SIGNAL
Computer ID 1394876
Run time 5 days 19 hours 43 min 28 sec
CPU time 5 days 15 hours 34 min 2 sec
Validate state Invalid
Credit 7,619.09
Device peak FLOPS 2.79 GFLOPS
Application version Weather At Home 2 (wah2) (region independent) v8.12
i686-pc-linux-gnu
Peak working set size 238.83 MB
Peak swap size 272.76 MB
Peak disk usage 97.17 MB
Stderr
<core_client_version>7.6.33</core_client_version>
<![CDATA[
<message>
process exited with code 193 (0xc1, -63)
</message>
<stderr_txt>
Suspended CPDN Monitor - Suspend request from BOINC...
Signal 15 received, exiting...
Signal 15 received, exiting...
01:01:15 (3989): called boinc_finish(193)
01:01:15 (3990): called boinc_finish(193)
Signal 15 received, exiting...
01:01:16 (3990): called boinc_finish(193)
Signal 15 received, exiting...
01:01:16 (3989): called boinc_finish(193)
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Signal 15 received, exiting...
20:00:36 (3056): called boinc_finish(193)
Signal 15 received, exiting...
20:00:36 (3055): called boinc_finish(193)
Signal 15 received, exiting...
Signal 15 received, exiting...
20:00:37 (3055): called boinc_finish(193)
20:00:37 (3056): called boinc_finish(193)
Signal 15 received, exiting...
SIGSEGV: segmentation violation
Stack trace (19 frames):
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80c9f4f]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf77c3d20]
/usr/lib32/libc.so.6(getenv+0x9a)[0xf73cb13a]
/usr/lib32/libc.so.6(+0xa1c7b)[0xf743ec7b]
/usr/lib32/libc.so.6(+0xa20e8)[0xf743f0e8]
/usr/lib32/libc.so.6(localtime_r+0x1e)[0xf743d5be]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x80c4d96]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x80c528c]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x80c5391]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf77c3d20]
linux-gate.so.1(__kernel_vsyscall+0x9)[0xf77c3d49]
/usr/lib32/libc.so.6(nanosleep+0x4a)[0xf744ccea]
/usr/lib32/libc.so.6(usleep+0x35)[0xf747cad5]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x80dda50]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x8051114]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x80523d4]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x8052674]
/usr/lib32/libc.so.6(__libc_start_main+0xf6)[0xf73b5536]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu(__gxx_personality_v0+0x1bd)[0x804d4d1]

Exiting...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Signal 15 received, exiting...
Signal 15 received, exiting...
SIGSEGV: segmentation violation
Stack trace (19 frames):
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80c9f4f]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf774bd20]
/usr/lib32/libc.so.6(getenv+0x9a)[0xf734b13a]
/usr/lib32/libc.so.6(+0xa1c7b)[0xf73bec7b]
/usr/lib32/libc.so.6(+0xa20e8)[0xf73bf0e8]
/usr/lib32/libc.so.6(localtime_r+0x1e)[0xf73bd5be]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x80c4d96]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x80c528c]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x80c5391]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf774bd20]
linux-gate.so.1(__kernel_vsyscall+0x9)[0xf774bd49]
/usr/lib32/libc.so.6(nanosleep+0x4a)[0xf73cccea]
/usr/lib32/libc.so.6(usleep+0x35)[0xf73fcad5]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x80dda50]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x8051114]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x80523d4]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x8052674]
/usr/lib32/libc.so.6(__libc_start_main+0xf6)[0xf7335536]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu(__gxx_personality_v0+0x1bd)[0x804d4d1]

Exiting...
Signal 15 received, exiting...
Signal 15 received, exiting...
19:58:52 (1603): called boinc_finish(193)
19:58:52 (1602): called boinc_finish(193)
Signal 15 received, exiting...
Signal 15 received, exiting...
19:58:52 (1603): called boinc_finish(193)
19:58:52 (1602): called boinc_finish(193)
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Signal 15 received, exiting...
Signal 15 received, exiting...
SIGSEGV: segmentation violation
Stack trace (19 frames):
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80c9f4f]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf7793d20]
/usr/lib32/libc.so.6(getenv+0x9a)[0xf739b13a]
/usr/lib32/libc.so.6(+0xa1c7b)[0xf740ec7b]
/usr/lib32/libc.so.6(+0xa20e8)[0xf740f0e8]
/usr/lib32/libc.so.6(localtime_r+0x1e)[0xf740d5be]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x80c4d96]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x80c528c]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x80c5391]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf7793d20]
linux-gate.so.1(__kernel_vsyscall+0x9)[0xf7793d49]
/usr/lib32/libc.so.6(nanosleep+0x4a)[0xf741ccea]
/usr/lib32/libc.so.6(usleep+0x35)[0xf744cad5]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x80dda50]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x8051114]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x80523d4]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x8052674]
/usr/lib32/libc.so.6(__libc_start_main+0xf6)[0xf7385536]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu(__gxx_personality_v0+0x1bd)[0x804d4d1]

Exiting...
Signal 15 received, exiting...
Signal 15 received, exiting...
Signal 15 received, exiting...
SIGSEGV: segmentation violation
Stack trace (19 frames):
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80c9f4f]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf77bed20]
/usr/lib32/libc.so.6(getenv+0x9a)[0xf73c613a]
/usr/lib32/libc.so.6(+0xa1c7b)[0xf7439c7b]
/usr/lib32/libc.so.6(+0xa20e8)[0xf743a0e8]
/usr/lib32/libc.so.6(localtime_r+0x1e)[0xf74385be]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x80c4d96]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x80c528c]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x80c5391]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf77bed20]
linux-gate.so.1(__kernel_vsyscall+0x9)[0xf77bed49]
/usr/lib32/libc.so.6(nanosleep+0x4a)[0xf7447cea]
/usr/lib32/libc.so.6(usleep+0x35)[0xf7477ad5]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x80dda50]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x8051114]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x80523d4]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x8052674]
/usr/lib32/libc.so.6(__libc_start_main+0xf6)[0xf73b0536]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu(__gxx_personality_v0+0x1bd)[0x804d4d1]

Exiting...
Suspended CPDN Monitor - Suspend request from BOINC...
Signal 15 received, exiting...
Signal 15 received, exiting...
23:24:39 (5451): called boinc_finish(193)
23:24:39 (5450): called boinc_finish(193)
Signal 15 received, exiting...
23:24:40 (5451): called boinc_finish(193)
Signal 15 received, exiting...
23:24:40 (5450): called boinc_finish(193)
Suspended CPDN Monitor - Suspend request from BOINC...
Signal 15 received, exiting...
Signal 15 received, exiting...
00:42:19 (1927): called boinc_finish(193)
00:42:19 (1928): called boinc_finish(193)
Signal 15 received, exiting...
00:42:20 (1927): called boinc_finish(193)
Signal 15 received, exiting...
00:42:20 (1928): called boinc_finish(193)
Suspended CPDN Monitor - Suspend request from BOINC...
Signal 15 received, exiting...
Signal 15 received, exiting...
Signal 15 received, exiting...
SIGSEGV: segmentation violation
Stack trace (19 frames):
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80c9f4f]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf77e9d20]
/usr/lib32/libc.so.6(getenv+0x9a)[0xf73c713a]
/usr/lib32/libc.so.6(+0xa1c7b)[0xf743ac7b]
/usr/lib32/libc.so.6(+0xa20e8)[0xf743b0e8]
/usr/lib32/libc.so.6(localtime_r+0x1e)[0xf74395be]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x80c4d96]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x80c528c]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x80c5391]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf77e9d20]
linux-gate.so.1(__kernel_vsyscall+0x9)[0xf77e9d49]
/usr/lib32/libc.so.6(nanosleep+0x4a)[0xf7448cea]
/usr/lib32/libc.so.6(usleep+0x35)[0xf7478ad5]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x80dda50]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x8051114]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x80523d4]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x8052674]
/usr/lib32/libc.so.6(__libc_start_main+0xf6)[0xf73b1536]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu(__gxx_personality_v0+0x1bd)[0x804d4d1]

Exiting...
Suspended CPDN Monitor - Suspend request from BOINC...
Signal 15 received, exiting...
Signal 15 received, exiting...
SIGSEGV: segmentation violation
Stack trace (19 frames):
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80c9f4f]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf7727d20]
/usr/lib32/libc.so.6(getenv+0x9a)[0xf732f13a]
/usr/lib32/libc.so.6(+0xa1c7b)[0xf73a2c7b]
/usr/lib32/libc.so.6(+0xa20e8)[0xf73a30e8]
/usr/lib32/libc.so.6(localtime_r+0x1e)[0xf73a15be]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x80c4d96]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x80c528c]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x80c5391]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf7727d20]
linux-gate.so.1(__kernel_vsyscall+0x9)[0xf7727d49]
/usr/lib32/libc.so.6(nanosleep+0x4a)[0xf73b0cea]
/usr/lib32/libc.so.6(usleep+0x35)[0xf73e0ad5]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x80dda50]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x8051114]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x80523d4]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x8052674]
/usr/lib32/libc.so.6(__libc_start_main+0xf6)[0xf7319536]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu(__gxx_personality_v0+0x1bd)[0x804d4d1]

Exiting...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Signal 15 received, exiting...
Signal 15 received, exiting...
21:23:51 (1603): called boinc_finish(193)
21:23:51 (1604): called boinc_finish(193)
Signal 15 received, exiting...
Signal 15 received, exiting...
21:23:52 (1603): called boinc_finish(193)
21:23:52 (1604): called boinc_finish(193)
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Signal 15 received, exiting...
Signal 15 received, exiting...
20:54:20 (1615): called boinc_finish(193)
20:54:20 (1614): called boinc_finish(193)
Signal 15 received, exiting...
Signal 15 received, exiting...
20:54:20 (1614): called boinc_finish(193)
20:54:20 (1615): called boinc_finish(193)
Signal 15 received, exiting...
Signal 15 received, exiting...
SIGSEGV: segmentation violation
Stack trace (19 frames):
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80c9f4f]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf77eed20]
/usr/lib32/libc.so.6(getenv+0x9a)[0xf73cc13a]
/usr/lib32/libc.so.6(+0xa1c7b)[0xf743fc7b]
/usr/lib32/libc.so.6(+0xa20e8)[0xf74400e8]
/usr/lib32/libc.so.6(localtime_r+0x1e)[0xf743e5be]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x80c4d96]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x80c528c]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x80c5391]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf77eed20]
linux-gate.so.1(__kernel_vsyscall+0x9)[0xf77eed49]
/usr/lib32/libc.so.6(nanosleep+0x4a)[0xf744dcea]
/usr/lib32/libc.so.6(usleep+0x35)[0xf747dad5]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x80dda50]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x8051114]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x80523d4]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x8052674]
/usr/lib32/libc.so.6(__libc_start_main+0xf6)[0xf73b6536]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu(__gxx_personality_v0+0x1bd)[0x804d4d1]

Exiting...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Signal 15 received, exiting...
Signal 15 received, exiting...
SIGSEGV: segmentation violation
Stack trace (19 frames):
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80c9f4f]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf7714d20]
/usr/lib32/libc.so.6(getenv+0x9a)[0xf731c13a]
/usr/lib32/libc.so.6(+0xa1c7b)[0xf738fc7b]
/usr/lib32/libc.so.6(+0xa20e8)[0xf73900e8]
/usr/lib32/libc.so.6(localtime_r+0x1e)[0xf738e5be]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x80c4d96]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x80c528c]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x80c5391]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf7714d20]
linux-gate.so.1(__kernel_vsyscall+0x9)[0xf7714d49]
/usr/lib32/libc.so.6(nanosleep+0x4a)[0xf739dcea]
/usr/lib32/libc.so.6(usleep+0x35)[0xf73cdad5]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x80dda50]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x8051114]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x80523d4]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x8052674]
/usr/lib32/libc.so.6(__libc_start_main+0xf6)[0xf7306536]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu(__gxx_personality_v0+0x1bd)[0x804d4d1]

Exiting...
Signal 15 received, exiting...
11:44:32 (1342): called boinc_finish(193)
Signal 15 received, exiting...
11:44:32 (1343): called boinc_finish(193)
Signal 15 received, exiting...
SIGSEGV: segmentation violation
Stack trace (21 frames):
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80c9f4f]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf7774d20]
/usr/lib32/libc.so.6(getenv+0x9a)[0xf737c13a]
/usr/lib32/libc.so.6(+0xa1c7b)[0xf73efc7b]
/usr/lib32/libc.so.6(+0xa20e8)[0xf73f00e8]
/usr/lib32/libc.so.6(localtime_r+0x1e)[0xf73ee5be]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x80c4d96]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x80c528c]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x80c5391]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf7774d20]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x80c5130]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf7774d20]
linux-gate.so.1(__kernel_vsyscall+0x9)[0xf7774d49]
/usr/lib32/libc.so.6(nanosleep+0x4a)[0xf73fdcea]
/usr/lib32/libc.so.6(usleep+0x35)[0xf742dad5]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x80dda50]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x8051114]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x80523d4]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x8052674]
/usr/lib32/libc.so.6(__libc_start_main+0xf6)[0xf7366536]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu(__gxx_personality_v0+0x1bd)[0x804d4d1]

Exiting...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Signal 15 received, exiting...
17:28:41 (1340): called boinc_finish(193)
Signal 15 received, exiting...
17:28:41 (1341): called boinc_finish(193)
Signal 15 received, exiting...
17:28:41 (1340): called boinc_finish(193)
Signal 15 received, exiting...
17:28:41 (1341): called boinc_finish(193)
Signal 15 received, exiting...
Signal 15 received, exiting...
SIGSEGV: segmentation violation
Stack trace (19 frames):
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80c9f4f]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf76ded20]
/usr/lib32/libc.so.6(getenv+0x9a)[0xf72e613a]
/usr/lib32/libc.so.6(+0xa1c7b)[0xf7359c7b]
/usr/lib32/libc.so.6(+0xa20e8)[0xf735a0e8]
/usr/lib32/libc.so.6(localtime_r+0x1e)[0xf73585be]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x80c4d96]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x80c528c]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x80c5391]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf76ded20]
linux-gate.so.1(__kernel_vsyscall+0x9)[0xf76ded49]
/usr/lib32/libc.so.6(nanosleep+0x4a)[0xf7367cea]
/usr/lib32/libc.so.6(usleep+0x35)[0xf7397ad5]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x80dda50]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x8051114]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x80523d4]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x8052674]
/usr/lib32/libc.so.6(__libc_start_main+0xf6)[0xf72d0536]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu(__gxx_personality_v0+0x1bd)[0x804d4d1]

Exiting...
Suspended CPDN Monitor - Suspend request from BOINC...
Signal 15 received, exiting...
Signal 15 received, exiting...
SIGSEGV: segmentation violation
Stack trace (19 frames):
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80c9f4f]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf77a2d20]
/usr/lib32/libc.so.6(getenv+0x9a)[0xf737f13a]
/usr/lib32/libc.so.6(+0xa1c7b)[0xf73f2c7b]
/usr/lib32/libc.so.6(+0xa20e8)[0xf73f30e8]
/usr/lib32/libc.so.6(localtime_r+0x1e)[0xf73f15be]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x80c4d96]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x80c528c]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x80c5391]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf77a2d20]
linux-gate.so.1(__kernel_vsyscall+0x9)[0xf77a2d49]
/usr/lib32/libc.so.6(nanosleep+0x4a)[0xf7400cea]
/usr/lib32/libc.so.6(usleep+0x35)[0xf7430ad5]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x80dda50]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x8051114]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x80523d4]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x8052674]
/usr/lib32/libc.so.6(__libc_start_main+0xf6)[0xf7369536]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu(__gxx_personality_v0+0x1bd)[0x804d4d1]

Exiting...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Signal 15 received, exiting...
Signal 15 received, exiting...
23:00:19 (1344): called boinc_finish(193)
23:00:19 (1343): called boinc_finish(193)
Signal 15 received, exiting...
23:00:20 (1343): called boinc_finish(193)
Signal 15 received, exiting...
23:00:20 (1344): called boinc_finish(193)
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Signal 15 received, exiting...
18:16:00 (1345): called boinc_finish(193)
Signal 15 received, exiting...
SIGSEGV: segmentation violation
Stack trace (20 frames):
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80c9f4f]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf773ed20]
/usr/lib32/libc.so.6(getenv+0x9a)[0xf731c13a]
/usr/lib32/libc.so.6(+0xa1c7b)[0xf738fc7b]
/usr/lib32/libc.so.6(+0xa20e8)[0xf73900e8]
/usr/lib32/libc.so.6(localtime_r+0x1e)[0xf738e5be]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x80c4d96]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x80c528c]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x80c5391]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf773ed20]
linux-gate.so.1(__kernel_vsyscall+0x9)[0xf773ed49]
/usr/lib32/libc.so.6(nanosleep+0x4a)[0xf739dcea]
/usr/lib32/libc.so.6(usleep+0x35)[0xf73cdad5]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x80dda50]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x8050953]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x8051103]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x80523d4]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x8052674]
/usr/lib32/libc.so.6(__libc_start_main+0xf6)[0xf7306536]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu(__gxx_personality_v0+0x1bd)[0x804d4d1]

Exiting...
Suspended CPDN Monitor - Suspend request from BOINC...
Signal 15 received, exiting...
02:19:27 (2891): called boinc_finish(193)
Signal 15 received, exiting...
02:19:27 (2892): called boinc_finish(193)
Signal 15 received, exiting...
02:19:27 (2891): called boinc_finish(193)
Signal 15 received, exiting...
02:19:27 (2892): called boinc_finish(193)
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Signal 15 received, exiting...
13:52:25 (2672): called boinc_finish(193)
Signal 15 received, exiting...
13:52:25 (2673): called boinc_finish(193)
Signal 15 received, exiting...
SIGSEGV: segmentation violation
Stack trace (19 frames):
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80c9f4f]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf771dd20]
/usr/lib32/libc.so.6(getenv+0x9a)[0xf732513a]
/usr/lib32/libc.so.6(+0xa1c7b)[0xf7398c7b]
/usr/lib32/libc.so.6(+0xa20e8)[0xf73990e8]
/usr/lib32/libc.so.6(localtime_r+0x1e)[0xf73975be]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x80c4d96]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x80c528c]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x80c5391]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf771dd20]
linux-gate.so.1(__kernel_vsyscall+0x9)[0xf771dd49]
/usr/lib32/libc.so.6(nanosleep+0x4a)[0xf73a6cea]
/usr/lib32/libc.so.6(usleep+0x35)[0xf73d6ad5]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x80dda50]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x8051114]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x80523d4]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x8052674]
/usr/lib32/libc.so.6(__libc_start_main+0xf6)[0xf730f536]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu(__gxx_personality_v0+0x1bd)[0x804d4d1]

Exiting...
Signal 15 received, exiting...
15:40:56 (1615): called boinc_finish(193)
Signal 15 received, exiting...
SIGSEGV: segmentation violation
Stack trace (19 frames):
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80c9f4f]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf77b2d20]
/usr/lib32/libc.so.6(getenv+0x9a)[0xf73ba13a]
/usr/lib32/libc.so.6(+0xa1c7b)[0xf742dc7b]
/usr/lib32/libc.so.6(+0xa20e8)[0xf742e0e8]
/usr/lib32/libc.so.6(localtime_r+0x1e)[0xf742c5be]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x80c4d96]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x80c528c]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x80c5391]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf77b2d20]
linux-gate.so.1(__kernel_vsyscall+0x9)[0xf77b2d49]
/usr/lib32/libc.so.6(nanosleep+0x4a)[0xf743bcea]
/usr/lib32/libc.so.6(usleep+0x35)[0xf746bad5]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x80dda50]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x8051114]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x80523d4]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x8052674]
/usr/lib32/libc.so.6(__libc_start_main+0xf6)[0xf73a4536]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu(__gxx_personality_v0+0x1bd)[0x804d4d1]

Exiting...
Signal 15 received, exiting...
Signal 15 received, exiting...
17:28:09 (4436): called boinc_finish(193)
Signal 15 received, exiting...
SIGSEGV: segmentation violation
Stack trace (19 frames):
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80c9f4f]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf77cad20]
/usr/lib32/libc.so.6(getenv+0x9a)[0xf73d213a]
/usr/lib32/libc.so.6(+0xa1c7b)[0xf7445c7b]
/usr/lib32/libc.so.6(+0xa20e8)[0xf74460e8]
/usr/lib32/libc.so.6(localtime_r+0x1e)[0xf74445be]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x80c4d96]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x80c528c]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x80c5391]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf77cad20]
linux-gate.so.1(__kernel_vsyscall+0x9)[0xf77cad49]
/usr/lib32/libc.so.6(nanosleep+0x4a)[0xf7453cea]
/usr/lib32/libc.so.6(usleep+0x35)[0xf7483ad5]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x80dda50]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x8051114]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x80523d4]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x8052674]
/usr/lib32/libc.so.6(__libc_start_main+0xf6)[0xf73bc536]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu(__gxx_personality_v0+0x1bd)[0x804d4d1]

Exiting...
Suspended CPDN Monitor - Suspend request from BOINC...
Signal 15 received, exiting...
SIGSEGV: segmentation violation
Stack trace (19 frames):
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80c9f4f]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf776bd20]
/usr/lib32/libc.so.6(getenv+0x9a)[0xf734a13a]
/usr/lib32/libc.so.6(+0xa1c7b)[0xf73bdc7b]
/usr/lib32/libc.so.6(+0xa20e8)[0xf73be0e8]
/usr/lib32/libc.so.6(localtime_r+0x1e)[0xf73bc5be]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x80c4d96]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x80c528c]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x80c5391]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf776bd20]
linux-gate.so.1(__kernel_vsyscall+0x9)[0xf776bd49]
/usr/lib32/libc.so.6(nanosleep+0x4a)[0xf73cbcea]
/usr/lib32/libc.so.6(usleep+0x35)[0xf73fbad5]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x80dda50]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x8051114]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x80523d4]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x8052674]
/usr/lib32/libc.so.6(__libc_start_main+0xf6)[0xf7334536]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu(__gxx_personality_v0+0x1bd)[0x804d4d1]

Exiting...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Signal 15 received, exiting...
Signal 15 received, exiting...
SIGSEGV: segmentation violation
Stack trace (19 frames):
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80c9f4f]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf77c4d20]
/usr/lib32/libc.so.6(getenv+0x9a)[0xf73cc13a]
/usr/lib32/libc.so.6(+0xa1c7b)[0xf743fc7b]
/usr/lib32/libc.so.6(+0xa20e8)[0xf74400e8]
/usr/lib32/libc.so.6(localtime_r+0x1e)[0xf743e5be]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x80c4d96]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x80c528c]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x80c5391]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf77c4d20]
linux-gate.so.1(__kernel_vsyscall+0x9)[0xf77c4d49]
/usr/lib32/libc.so.6(nanosleep+0x4a)[0xf744dcea]
/usr/lib32/libc.so.6(usleep+0x35)[0xf747dad5]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x80dda50]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x8051114]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x80523d4]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x8052674]
/usr/lib32/libc.so.6(__libc_start_main+0xf6)[0xf73b6536]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu(__gxx_personality_v0+0x1bd)[0x804d4d1]

Exiting...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Signal 15 received, exiting...
18:32:07 (1328): called boinc_finish(193)
Signal 15 received, exiting...
18:32:07 (1327): called boinc_finish(193)
Signal 15 received, exiting...
Signal 15 received, exiting...
18:32:08 (1327): called boinc_finish(193)
18:32:08 (1328): called boinc_finish(193)
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Signal 15 received, exiting...
Signal 15 received, exiting...
SIGSEGV: segmentation violation
Stack trace (19 frames):
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80c9f4f]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf77c1d20]
/usr/lib32/libc.so.6(getenv+0x9a)[0xf73c913a]
/usr/lib32/libc.so.6(+0xa1c7b)[0xf743cc7b]
/usr/lib32/libc.so.6(+0xa20e8)[0xf743d0e8]
/usr/lib32/libc.so.6(localtime_r+0x1e)[0xf743b5be]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x80c4d96]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x80c528c]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x80c5391]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf77c1d20]
linux-gate.so.1(__kernel_vsyscall+0x9)[0xf77c1d49]
/usr/lib32/libc.so.6(nanosleep+0x4a)[0xf744acea]
/usr/lib32/libc.so.6(usleep+0x35)[0xf747aad5]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x80dda50]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x8051114]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x80523d4]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x8052674]
/usr/lib32/libc.so.6(__libc_start_main+0xf6)[0xf73b3536]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu(__gxx_personality_v0+0x1bd)[0x804d4d1]

Exiting...
Signal 3 received, exiting...
SIGSEGV: segmentation violation
10:45:12 (1325): called boinc_finish(193)
Stack trace (7 frames):
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80c9f4f]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf7799d20]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x804d85f]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x80514ed]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu[0x8052674]
/usr/lib32/libc.so.6(__libc_start_main+0xf6)[0xf738b536]
../../projects/climateprediction.net/wah2_8.12_i686-pc-linux-gnu(__gxx_personality_v0+0x1bd)[0x804d4d1]

Exiting...

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
26 Jul 2016 08:50:23 1394876 19792216 wah2_sas50_fmll_201412_13_370_010399847_1 115,499 447,499 3.8745
23 Jul 2016 22:43:07 1394876 19792216 wah2_sas50_fmll_201412_13_370_010399847_1 103,979 405,275 3.8977
22 Jul 2016 11:44:59 1394876 19792216 wah2_sas50_fmll_201412_13_370_010399847_1 92,459 362,327 3.9188
20 Jul 2016 10:32:39 1394876 19792216 wah2_sas50_fmll_201412_13_370_010399847_1 80,939 319,422 3.9465
18 Jul 2016 18:22:28 1394876 19792216 wah2_sas50_fmll_201412_13_370_010399847_1 69,419 277,042 3.9909
17 Jul 2016 12:18:44 1394876 19792216 wah2_sas50_fmll_201412_13_370_010399847_1 57,899 231,731 4.0023
15 Jul 2016 14:42:40 1394876 19792216 wah2_sas50_fmll_201412_13_370_010399847_1 46,379 183,586 3.9584
14 Jul 2016 14:32:20 1394876 19792216 wah2_sas50_fmll_201412_13_370_010399847_1 34,859 135,622 3.8906
13 Jul 2016 12:10:51 1394876 19792216 wah2_sas50_fmll_201412_13_370_010399847_1 23,339 88,270 3.7821
12 Jul 2016 15:02:52 1394876 19792216 wah2_sas50_fmll_201412_13_370_010399847_1 11,819 45,039 3.8107


©2024 climateprediction.net