climateprediction.net home page
Task 22298059

Task 22298059

Name oifs_43r3_ps_0817_1996050100_123_965_12182461_1
Workunit 12182461
Created 15 Jan 2023, 4:34:29 UTC
Sent 15 Jan 2023, 4:51:15 UTC
Report deadline 14 Feb 2023, 4:51:15 UTC
Received 15 Jan 2023, 7:20:14 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 1 (0x00000001) Unknown error code
Computer ID 1538137
Run time 46 min 50 sec
CPU time 43 min 48 sec
Validate state Invalid
Credit 0.00
Device peak FLOPS 1.00 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,271.09 MB
Peak swap size 4,978.08 MB
Peak disk usage 2,848.16 MB
Stderr 
<core_client_version>7.16.6</core_client_version>
<![CDATA[
<message>
process exited with code 1 (0x1, -255)</message>
<stderr_txt>
ptions@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:moi-desktop:1:1:2132:2132] [20230114:230805:1673759285:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:moi-desktop:1:1:2132:2132] [20230114:230805:1673759285:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:moi-desktop:1:1:2132:2132] [20230114:230805:1673759285:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:moi-desktop:1:1:2132:2132] [20230114:230805:1673759285:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:moi-desktop:1:1:2132:2132] [20230114:230805:1673759285:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:moi-desktop:1:1:2132:2132] [20230114:230805:1673759285:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:moi-desktop:1:1:2132:2132] [20230114:230805:1673759285:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:moi-desktop:1:1:2132:2132] [20230114:230805:1673759285:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:moi-desktop:1:1:2132:2132] [20230114:230805:1673759285:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:moi-desktop:1:1:2132:2132] [20230114:230805:1673759285:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:moi-desktop:1:1:2132:2132] [20230114:230805:1673759285:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:moi-desktop:1:1:2132:2132] [20230114:230805:1673759285:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:moi-desktop:1:1:2132:2132] [20230114:230805:1673759285:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:moi-desktop:1:1:2132:2132] [20230114:230805:1673759285:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:moi-desktop:1:1:2132:2132] [20230114:230805:1673759285:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:moi-desktop:1:1:2132:2132] [20230114:230805:1673759285:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:moi-desktop:1:1:2132:2132] [20230114:230805:1673759285:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:moi-desktop:1:1:2132:2132] [20230114:230805:1673759285:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:moi-desktop:1:1:2132:2132] [20230114:230805:1673759285:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:moi-desktop:1:1:2132:2132] [20230114:230805:1673759285:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:moi-desktop:1:1:2132:2132] [20230114:230805:1673759285:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:moi-desktop:1:1:2132:2132] [20230114:230805:1673759285:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:moi-desktop:1:1:2132:2132] [20230114:230805:1673759285:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:moi-desktop:1:1:2132:2132] [20230114:230805:1673759285:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:moi-desktop:1:1:2132:2132] [20230114:230805:1673759285:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  23:08:27 STEP    0 H=   0:00 +CPU= 12.573
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+000000
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+000000
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+000000
  23:08:42 STEP    1 H=   1:00 +CPU=  8.740
  23:08:53 STEP    2 H=   2:00 +CPU=  6.941
  23:09:04 STEP    3 H=   3:00 +CPU= 10.882
  23:09:11 STEP    4 H=   4:00 +CPU=  7.313
  23:09:25 STEP    5 H=   5:00 +CPU=  6.829
  23:09:36 STEP    6 H=   6:00 +CPU= 10.845
  23:09:46 STEP    7 H=   7:00 +CPU=  7.305
  23:09:54 STEP    8 H=   8:00 +CPU=  7.290
  23:10:04 STEP    9 H=   9:00 +CPU= 10.447
  23:10:13 STEP   10 H=  10:00 +CPU=  7.169
  23:10:19 STEP   11 H=  11:00 +CPU=  6.699
  23:10:32 STEP   12 H=  12:00 +CPU= 11.900
  23:10:38 STEP   13 H=  13:00 +CPU=  6.585
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+000012
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+000012
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+000012
  23:10:45 STEP   14 H=  14:00 +CPU=  6.796
  23:10:57 STEP   15 H=  15:00 +CPU= 10.754
  23:11:04 STEP   16 H=  16:00 +CPU=  6.740
  23:11:10 STEP   17 H=  17:00 +CPU=  6.785
  23:11:21 STEP   18 H=  18:00 +CPU= 10.779
  23:11:28 STEP   19 H=  19:00 +CPU=  6.552
  23:11:35 STEP   20 H=  20:00 +CPU=  6.930
  23:11:46 STEP   21 H=  21:00 +CPU= 10.512
  23:11:52 STEP   22 H=  22:00 +CPU=  6.865
  23:11:59 STEP   23 H=  23:00 +CPU=  6.768
  23:12:12 STEP   24 H=  24:00 +CPU= 12.683
  23:12:19 STEP   25 H=  25:00 +CPU=  6.502
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+000024
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+000024
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+000024
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182461/ICMGGhq0f+000000
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182461/ICMSHhq0f+000000
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182461/ICMUAhq0f+000000
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182461/ICMGGhq0f+000012
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182461/ICMSHhq0f+000012
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182461/ICMUAhq0f+000012
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0817_1996050100_123_965_12182461_1_r1488495194_0.zip
Uploading the intermediate file: upload_file_0.zip
  23:12:26 STEP   26 H=  26:00 +CPU=  6.849
  23:12:37 STEP   27 H=  27:00 +CPU= 10.970
  23:12:43 STEP   28 H=  28:00 +CPU=  6.734
  23:12:50 STEP   29 H=  29:00 +CPU=  6.900
  23:13:01 STEP   30 H=  30:00 +CPU= 11.094
  23:13:08 STEP   31 H=  31:00 +CPU=  6.919
  23:13:15 STEP   32 H=  32:00 +CPU=  6.803
  23:13:26 STEP   33 H=  33:00 +CPU= 10.771
  23:13:33 STEP   34 H=  34:00 +CPU=  6.730
  23:13:39 STEP   35 H=  35:00 +CPU=  6.456
  23:13:51 STEP   36 H=  36:00 +CPU= 11.876
  23:13:58 STEP   37 H=  37:00 +CPU=  6.371
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+000036
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+000036
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+000036
  23:14:05 STEP   38 H=  38:00 +CPU=  7.069
  23:14:15 STEP   39 H=  39:00 +CPU= 10.805
  23:14:22 STEP   40 H=  40:00 +CPU=  6.583
  23:14:29 STEP   41 H=  41:00 +CPU=  6.752
  23:14:40 STEP   42 H=  42:00 +CPU= 10.837
  23:14:46 STEP   43 H=  43:00 +CPU=  6.603
  23:14:53 STEP   44 H=  44:00 +CPU=  6.755
  23:15:03 STEP   45 H=  45:00 +CPU= 10.389
  23:15:10 STEP   46 H=  46:00 +CPU=  6.910
  23:15:17 STEP   47 H=  47:00 +CPU=  6.912
  23:15:31 STEP   48 H=  48:00 +CPU= 13.074
  23:15:38 STEP   49 H=  49:00 +CPU=  6.519
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+000048
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+000048
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+000048
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182461/ICMGGhq0f+000024
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182461/ICMSHhq0f+000024
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182461/ICMUAhq0f+000024
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182461/ICMGGhq0f+000036
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182461/ICMSHhq0f+000036
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182461/ICMUAhq0f+000036
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0817_1996050100_123_965_12182461_1_r1488495194_1.zip
Uploading the intermediate file: upload_file_1.zip
  23:15:45 STEP   50 H=  50:00 +CPU=  7.168
  23:15:55 STEP   51 H=  51:00 +CPU= 10.524
  23:16:02 STEP   52 H=  52:00 +CPU=  6.656
  23:16:09 STEP   53 H=  53:00 +CPU=  6.991
  23:16:20 STEP   54 H=  54:00 +CPU= 10.739
  23:16:27 STEP   55 H=  55:00 +CPU=  6.650
  23:16:34 STEP   56 H=  56:00 +CPU=  6.844
  23:16:45 STEP   57 H=  57:00 +CPU= 10.870
  23:16:51 STEP   58 H=  58:00 +CPU=  6.720
  23:17:00 STEP   59 H=  59:00 +CPU=  7.193
  23:17:14 STEP   60 H=  60:00 +CPU= 12.097
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+000060
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+000060
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+000060
  23:17:21 STEP   61 H=  61:00 +CPU=  6.584
  23:17:28 STEP   62 H=  62:00 +CPU=  6.733
  23:17:41 STEP   63 H=  63:00 +CPU= 11.202
  23:17:48 STEP   64 H=  64:00 +CPU=  7.452
  23:17:55 STEP   65 H=  65:00 +CPU=  6.653
  23:18:07 STEP   66 H=  66:00 +CPU= 11.070
  23:18:14 STEP   67 H=  67:00 +CPU=  6.729
  23:18:21 STEP   68 H=  68:00 +CPU=  6.989
  23:18:34 STEP   69 H=  69:00 +CPU= 10.887
  23:18:41 STEP   70 H=  70:00 +CPU=  7.141
  23:18:48 STEP   71 H=  71:00 +CPU=  6.807
  23:19:03 STEP   72 H=  72:00 +CPU= 12.683
  23:19:09 STEP   73 H=  73:00 +CPU=  6.488
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+000072
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+000072
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+000072
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182461/ICMGGhq0f+000048
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182461/ICMSHhq0f+000048
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182461/ICMUAhq0f+000048
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182461/ICMGGhq0f+000060
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182461/ICMSHhq0f+000060
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182461/ICMUAhq0f+000060
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0817_1996050100_123_965_12182461_1_r1488495194_2.zip
Uploading the intermediate file: upload_file_2.zip
  23:19:16 STEP   74 H=  74:00 +CPU=  7.018
  23:19:27 STEP   75 H=  75:00 +CPU= 10.606
  23:19:34 STEP   76 H=  76:00 +CPU=  6.871
  23:19:41 STEP   77 H=  77:00 +CPU=  6.798
  23:19:52 STEP   78 H=  78:00 +CPU= 11.115
  23:19:59 STEP   79 H=  79:00 +CPU=  6.586
  23:20:05 STEP   80 H=  80:00 +CPU=  6.945
  23:20:16 STEP   81 H=  81:00 +CPU= 10.585
  23:20:23 STEP   82 H=  82:00 +CPU=  6.928
  23:20:30 STEP   83 H=  83:00 +CPU=  6.523
  23:20:41 STEP   84 H=  84:00 +CPU= 11.872
  23:20:48 STEP   85 H=  85:00 +CPU=  6.796
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+000084
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+000084
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+000084
  23:20:55 STEP   86 H=  86:00 +CPU=  6.887
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1996050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0817
(argv4) batchid: 965
(argv5) wuid: 12182461
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/2
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182461
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_4257c35c35e9796a12c76f1818c88d39 to: /var/lib/boinc-client/slots/2/oifs_43r3_ps_0817_1996050100_123_965_12182461.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/2/oifs_43r3_ps_0817_1996050100_123_965_12182461.zip
ic_ancil_file: ic_ancil_12182461
ifsdata_file: ifsdata_12182461
climate_data_file: clim_data_12182461
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_23d6caed0f55b5cee1a78d96e58dd71b to: /var/lib/boinc-client/slots/2/ic_ancil_12182461.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/2/ic_ancil_12182461.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_a4352f687f6b13a07716a74e394a3078 to: /var/lib/boinc-client/slots/2/ifsdata/ifsdata_12182461.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/2/ifsdata/ifsdata_12182461.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/2/159l_2/clim_data_12182461.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/2/159l_2/clim_data_12182461.zip
Checking for progress XML file: /var/lib/boinc-client/slots/2/progress_file_12182461.xml
Opened progress file ok : /var/lib/boinc-client/slots/2/progress_file_12182461.xml
-- Model is restarting --
Adjusting last_iter, 87, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/2/progress_file_12182461.xml
last_cpu_time: 728
upload_file_number: 3
last_iter: 71
last_upload: 259200
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0817_1996050100_123_965_12182461_1_r1488495194
The child process has been launched with process id: 2152
Executing the command: /var/lib/boinc-client/slots/2/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:moi-desktop:1:1:2152:2152] [20230114:232204:1673760124:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:moi-desktop:1:1:2152:2152] [20230114:232204:1673760124:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:moi-desktop:1:1:2152:2152] [20230114:232204:1673760124:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:moi-desktop:1:1:2152:2152] [20230114:232204:1673760124:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:moi-desktop:1:1:2152:2152] [20230114:232204:1673760124:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:moi-desktop:1:1:2152:2152] [20230114:232204:1673760124:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:moi-desktop:1:1:2152:2152] [20230114:232204:1673760124:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:moi-desktop:1:1:2152:2152] [20230114:232204:1673760124:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:moi-desktop:1:1:2152:2152] [20230114:232204:1673760124:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:moi-desktop:1:1:2152:2152] [20230114:232204:1673760124:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:moi-desktop:1:1:2152:2152] [20230114:232204:1673760124:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:moi-desktop:1:1:2152:2152] [20230114:232204:1673760124:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:moi-desktop:1:1:2152:2152] [20230114:232204:1673760124:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:moi-desktop:1:1:2152:2152] [20230114:232204:1673760124:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:moi-desktop:1:1:2152:2152] [20230114:232204:1673760124:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:moi-desktop:1:1:2152:2152] [20230114:232204:1673760124:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:moi-desktop:1:1:2152:2152] [20230114:232204:1673760124:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:moi-desktop:1:1:2152:2152] [20230114:232204:1673760124:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:moi-desktop:1:1:2152:2152] [20230114:232204:1673760124:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:moi-desktop:1:1:2152:2152] [20230114:232204:1673760124:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:moi-desktop:1:1:2152:2152] [20230114:232204:1673760124:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:moi-desktop:1:1:2152:2152] [20230114:232204:1673760124:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:moi-desktop:1:1:2152:2152] [20230114:232204:1673760124:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:moi-desktop:1:1:2152:2152] [20230114:232204:1673760124:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:moi-desktop:1:1:2152:2152] [20230114:232204:1673760124:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:moi-desktop:1:1:2152:2152] [20230114:232204:1673760124:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:moi-desktop:1:1:2152:2152] [20230114:232204:1673760124:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:moi-desktop:1:1:2152:2152] [20230114:232204:1673760124:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:moi-desktop:1:1:2152:2152] [20230114:232204:1673760124:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:moi-desktop:1:1:2152:2152] [20230114:232204:1673760124:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  23:22:17 STEP   72 H=  72:00 +CPU= 11.860
  23:22:24 STEP   73 H=  73:00 +CPU=  7.006
  23:22:31 STEP   74 H=  74:00 +CPU=  6.722
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1996050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0817
(argv4) batchid: 965
(argv5) wuid: 12182461
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/2
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182461
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_4257c35c35e9796a12c76f1818c88d39 to: /var/lib/boinc-client/slots/2/oifs_43r3_ps_0817_1996050100_123_965_12182461.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/2/oifs_43r3_ps_0817_1996050100_123_965_12182461.zip
ic_ancil_file: ic_ancil_12182461
ifsdata_file: ifsdata_12182461
climate_data_file: clim_data_12182461
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_23d6caed0f55b5cee1a78d96e58dd71b to: /var/lib/boinc-client/slots/2/ic_ancil_12182461.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/2/ic_ancil_12182461.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_a4352f687f6b13a07716a74e394a3078 to: /var/lib/boinc-client/slots/2/ifsdata/ifsdata_12182461.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/2/ifsdata/ifsdata_12182461.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/2/159l_2/clim_data_12182461.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/2/159l_2/clim_data_12182461.zip
Checking for progress XML file: /var/lib/boinc-client/slots/2/progress_file_12182461.xml
Opened progress file ok : /var/lib/boinc-client/slots/2/progress_file_12182461.xml
-- Model is restarting --
Adjusting last_iter, 75, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/2/progress_file_12182461.xml
last_cpu_time: 755
upload_file_number: 3
last_iter: 71
last_upload: 259200
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0817_1996050100_123_965_12182461_1_r1488495194
The child process has been launched with process id: 2157
Executing the command: /var/lib/boinc-client/slots/2/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:moi-desktop:1:1:2157:2157] [20230114:232344:1673760224:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:moi-desktop:1:1:2157:2157] [20230114:232344:1673760224:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:moi-desktop:1:1:2157:2157] [20230114:232344:1673760224:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:moi-desktop:1:1:2157:2157] [20230114:232344:1673760224:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:moi-desktop:1:1:2157:2157] [20230114:232344:1673760224:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:moi-desktop:1:1:2157:2157] [20230114:232344:1673760224:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:moi-desktop:1:1:2157:2157] [20230114:232344:1673760224:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:moi-desktop:1:1:2157:2157] [20230114:232344:1673760224:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:moi-desktop:1:1:2157:2157] [20230114:232344:1673760224:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:moi-desktop:1:1:2157:2157] [20230114:232344:1673760224:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:moi-desktop:1:1:2157:2157] [20230114:232344:1673760224:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:moi-desktop:1:1:2157:2157] [20230114:232344:1673760224:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:moi-desktop:1:1:2157:2157] [20230114:232344:1673760224:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:moi-desktop:1:1:2157:2157] [20230114:232344:1673760224:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:moi-desktop:1:1:2157:2157] [20230114:232344:1673760224:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:moi-desktop:1:1:2157:2157] [20230114:232344:1673760224:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:moi-desktop:1:1:2157:2157] [20230114:232344:1673760224:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:moi-desktop:1:1:2157:2157] [20230114:232344:1673760224:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:moi-desktop:1:1:2157:2157] [20230114:232344:1673760224:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:moi-desktop:1:1:2157:2157] [20230114:232344:1673760224:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:moi-desktop:1:1:2157:2157] [20230114:232344:1673760224:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:moi-desktop:1:1:2157:2157] [20230114:232344:1673760224:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:moi-desktop:1:1:2157:2157] [20230114:232344:1673760224:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:moi-desktop:1:1:2157:2157] [20230114:232344:1673760224:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:moi-desktop:1:1:2157:2157] [20230114:232344:1673760224:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:moi-desktop:1:1:2157:2157] [20230114:232344:1673760224:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:moi-desktop:1:1:2157:2157] [20230114:232344:1673760224:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:moi-desktop:1:1:2157:2157] [20230114:232344:1673760224:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:moi-desktop:1:1:2157:2157] [20230114:232344:1673760224:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:moi-desktop:1:1:2157:2157] [20230114:232344:1673760224:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  23:23:57 STEP   72 H=  72:00 +CPU= 11.888
  23:24:05 STEP   73 H=  73:00 +CPU=  7.067
  23:24:13 STEP   74 H=  74:00 +CPU=  7.284
  23:24:23 STEP   75 H=  75:00 +CPU= 10.580
  23:24:31 STEP   76 H=  76:00 +CPU=  7.385
  23:24:38 STEP   77 H=  77:00 +CPU=  6.814
  23:24:49 STEP   78 H=  78:00 +CPU= 10.622
  23:24:56 STEP   79 H=  79:00 +CPU=  6.638
  23:25:02 STEP   80 H=  80:00 +CPU=  6.890
  23:25:13 STEP   81 H=  81:00 +CPU= 10.449
  23:25:20 STEP   82 H=  82:00 +CPU=  6.883
  23:25:26 STEP   83 H=  83:00 +CPU=  6.635
  23:25:38 STEP   84 H=  84:00 +CPU= 11.733
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+000084
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+000084
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+000084
  23:25:45 STEP   85 H=  85:00 +CPU=  6.692
  23:25:52 STEP   86 H=  86:00 +CPU=  6.816
  23:26:02 STEP   87 H=  87:00 +CPU= 10.473
  23:26:09 STEP   88 H=  88:00 +CPU=  6.677
  23:26:16 STEP   89 H=  89:00 +CPU=  6.776
  23:26:26 STEP   90 H=  90:00 +CPU= 10.681
  23:26:33 STEP   91 H=  91:00 +CPU=  6.768
  23:26:40 STEP   92 H=  92:00 +CPU=  6.703
  23:26:51 STEP   93 H=  93:00 +CPU= 10.740
  23:26:57 STEP   94 H=  94:00 +CPU=  6.717
  23:27:04 STEP   95 H=  95:00 +CPU=  6.545
  23:27:18 STEP   96 H=  96:00 +CPU= 12.709
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+000096
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+000096
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+000096
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182461/ICMGGhq0f+000072
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182461/ICMSHhq0f+000072
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182461/ICMUAhq0f+000072
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182461/ICMGGhq0f+000084
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182461/ICMSHhq0f+000084
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182461/ICMUAhq0f+000084
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0817_1996050100_123_965_12182461_1_r1488495194_3.zip
Uploading the intermediate file: upload_file_3.zip
  23:27:25 STEP   97 H=  97:00 +CPU=  6.390
  23:27:32 STEP   98 H=  98:00 +CPU=  6.824
  23:27:42 STEP   99 H=  99:00 +CPU= 10.307
  23:27:49 STEP  100 H= 100:00 +CPU=  6.895
  23:27:55 STEP  101 H= 101:00 +CPU=  6.485
  23:28:06 STEP  102 H= 102:00 +CPU= 10.619
  23:28:12 STEP  103 H= 103:00 +CPU=  6.427
  23:28:19 STEP  104 H= 104:00 +CPU=  6.649
  23:28:29 STEP  105 H= 105:00 +CPU= 10.553
  23:28:36 STEP  106 H= 106:00 +CPU=  6.606
  23:28:43 STEP  107 H= 107:00 +CPU=  6.730
  23:28:54 STEP  108 H= 108:00 +CPU= 11.573
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+000108
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+000108
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+000108
  23:29:01 STEP  109 H= 109:00 +CPU=  6.647
  23:29:08 STEP  110 H= 110:00 +CPU=  6.728
  23:29:19 STEP  111 H= 111:00 +CPU= 10.691
  23:29:25 STEP  112 H= 112:00 +CPU=  6.638
  23:29:33 STEP  113 H= 113:00 +CPU=  6.849
  23:29:43 STEP  114 H= 114:00 +CPU= 10.534
  23:29:50 STEP  115 H= 115:00 +CPU=  6.798
  23:29:57 STEP  116 H= 116:00 +CPU=  6.499
  23:30:07 STEP  117 H= 117:00 +CPU= 10.573
  23:30:14 STEP  118 H= 118:00 +CPU=  6.492
  23:30:21 STEP  119 H= 119:00 +CPU=  7.018
  23:30:34 STEP  120 H= 120:00 +CPU= 12.300
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+000120
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+000120
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+000120
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182461/ICMGGhq0f+000096
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182461/ICMSHhq0f+000096
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182461/ICMUAhq0f+000096
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182461/ICMGGhq0f+000108
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182461/ICMSHhq0f+000108
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182461/ICMUAhq0f+000108
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0817_1996050100_123_965_12182461_1_r1488495194_4.zip
Uploading the intermediate file: upload_file_4.zip
  23:30:41 STEP  121 H= 121:00 +CPU=  6.275
  23:30:47 STEP  122 H= 122:00 +CPU=  6.810
  23:30:58 STEP  123 H= 123:00 +CPU= 10.118
  23:31:04 STEP  124 H= 124:00 +CPU=  6.812
  23:31:11 STEP  125 H= 125:00 +CPU=  6.516
  23:31:22 STEP  126 H= 126:00 +CPU= 10.589
  23:31:28 STEP  127 H= 127:00 +CPU=  6.382
  23:31:34 STEP  128 H= 128:00 +CPU=  6.442
  23:31:45 STEP  129 H= 129:00 +CPU= 10.538
  23:31:51 STEP  130 H= 130:00 +CPU=  6.535
  23:31:58 STEP  131 H= 131:00 +CPU=  6.559
  23:32:09 STEP  132 H= 132:00 +CPU= 11.293
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+000132
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+000132
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+000132
  23:32:16 STEP  133 H= 133:00 +CPU=  6.691
  23:32:23 STEP  134 H= 134:00 +CPU=  6.724
  23:32:33 STEP  135 H= 135:00 +CPU= 10.727
  23:32:40 STEP  136 H= 136:00 +CPU=  6.814
  23:32:47 STEP  137 H= 137:00 +CPU=  6.544
  23:32:57 STEP  138 H= 138:00 +CPU= 10.425
  23:33:04 STEP  139 H= 139:00 +CPU=  6.363
  23:33:10 STEP  140 H= 140:00 +CPU=  6.865
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1996050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0817
(argv4) batchid: 965
(argv5) wuid: 12182461
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/2
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182461
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_4257c35c35e9796a12c76f1818c88d39 to: /var/lib/boinc-client/slots/2/oifs_43r3_ps_0817_1996050100_123_965_12182461.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/2/oifs_43r3_ps_0817_1996050100_123_965_12182461.zip
ic_ancil_file: ic_ancil_12182461
ifsdata_file: ifsdata_12182461
climate_data_file: clim_data_12182461
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_23d6caed0f55b5cee1a78d96e58dd71b to: /var/lib/boinc-client/slots/2/ic_ancil_12182461.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/2/ic_ancil_12182461.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_a4352f687f6b13a07716a74e394a3078 to: /var/lib/boinc-client/slots/2/ifsdata/ifsdata_12182461.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/2/ifsdata/ifsdata_12182461.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/2/159l_2/clim_data_12182461.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/2/159l_2/clim_data_12182461.zip
Checking for progress XML file: /var/lib/boinc-client/slots/2/progress_file_12182461.xml
Opened progress file ok : /var/lib/boinc-client/slots/2/progress_file_12182461.xml
-- Model is restarting --
Adjusting last_iter, 141, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/2/progress_file_12182461.xml
last_cpu_time: 1318
upload_file_number: 5
last_iter: 119
last_upload: 432000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0817_1996050100_123_965_12182461_1_r1488495194
The child process has been launched with process id: 2169
Executing the command: /var/lib/boinc-client/slots/2/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:moi-desktop:1:1:2169:2169] [20230114:233419:1673760859:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:moi-desktop:1:1:2169:2169] [20230114:233419:1673760859:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:moi-desktop:1:1:2169:2169] [20230114:233419:1673760859:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:moi-desktop:1:1:2169:2169] [20230114:233419:1673760859:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:moi-desktop:1:1:2169:2169] [20230114:233419:1673760859:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:moi-desktop:1:1:2169:2169] [20230114:233419:1673760859:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:moi-desktop:1:1:2169:2169] [20230114:233419:1673760859:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:moi-desktop:1:1:2169:2169] [20230114:233419:1673760859:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:moi-desktop:1:1:2169:2169] [20230114:233419:1673760859:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:moi-desktop:1:1:2169:2169] [20230114:233419:1673760859:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:moi-desktop:1:1:2169:2169] [20230114:233419:1673760859:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:moi-desktop:1:1:2169:2169] [20230114:233419:1673760859:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:moi-desktop:1:1:2169:2169] [20230114:233419:1673760859:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:moi-desktop:1:1:2169:2169] [20230114:233419:1673760859:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:moi-desktop:1:1:2169:2169] [20230114:233419:1673760859:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:moi-desktop:1:1:2169:2169] [20230114:233419:1673760859:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:moi-desktop:1:1:2169:2169] [20230114:233419:1673760859:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:moi-desktop:1:1:2169:2169] [20230114:233419:1673760859:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:moi-desktop:1:1:2169:2169] [20230114:233419:1673760859:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:moi-desktop:1:1:2169:2169] [20230114:233419:1673760859:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:moi-desktop:1:1:2169:2169] [20230114:233419:1673760859:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:moi-desktop:1:1:2169:2169] [20230114:233419:1673760859:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:moi-desktop:1:1:2169:2169] [20230114:233419:1673760859:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:moi-desktop:1:1:2169:2169] [20230114:233419:1673760859:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:moi-desktop:1:1:2169:2169] [20230114:233419:1673760859:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:moi-desktop:1:1:2169:2169] [20230114:233419:1673760859:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:moi-desktop:1:1:2169:2169] [20230114:233419:1673760859:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:moi-desktop:1:1:2169:2169] [20230114:233419:1673760859:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:moi-desktop:1:1:2169:2169] [20230114:233419:1673760859:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:moi-desktop:1:1:2169:2169] [20230114:233419:1673760859:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  23:34:32 STEP  120 H= 120:00 +CPU= 11.443
  23:34:39 STEP  121 H= 121:00 +CPU=  6.937
  23:34:45 STEP  122 H= 122:00 +CPU=  6.714
  23:34:57 STEP  123 H= 123:00 +CPU= 10.920
  23:35:04 STEP  124 H= 124:00 +CPU=  6.599
  23:35:10 STEP  125 H= 125:00 +CPU=  6.595
  23:35:21 STEP  126 H= 126:00 +CPU= 10.785
  23:35:28 STEP  127 H= 127:00 +CPU=  6.582
  23:35:35 STEP  128 H= 128:00 +CPU=  6.968
  23:35:45 STEP  129 H= 129:00 +CPU= 10.635
  23:35:52 STEP  130 H= 130:00 +CPU=  6.686
  23:35:58 STEP  131 H= 131:00 +CPU=  6.405
  23:36:10 STEP  132 H= 132:00 +CPU= 11.944
  23:36:17 STEP  133 H= 133:00 +CPU=  6.605
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+000132
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+000132
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+000132
  23:36:24 STEP  134 H= 134:00 +CPU=  6.866
  23:36:34 STEP  135 H= 135:00 +CPU= 10.201
  23:36:41 STEP  136 H= 136:00 +CPU=  6.826
  23:36:47 STEP  137 H= 137:00 +CPU=  6.350
  23:36:58 STEP  138 H= 138:00 +CPU= 10.798
  23:37:05 STEP  139 H= 139:00 +CPU=  6.637
  23:37:11 STEP  140 H= 140:00 +CPU=  6.699
  23:37:21 STEP  141 H= 141:00 +CPU= 10.147
  23:37:29 STEP  142 H= 142:00 +CPU=  7.011
  23:37:35 STEP  143 H= 143:00 +CPU=  6.650
  23:37:50 STEP  144 H= 144:00 +CPU= 12.705
  23:37:57 STEP  145 H= 145:00 +CPU=  6.449
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+000144
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+000144
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+000144
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182461/ICMGGhq0f+000120
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182461/ICMSHhq0f+000120
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182461/ICMUAhq0f+000120
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182461/ICMGGhq0f+000132
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182461/ICMSHhq0f+000132
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182461/ICMUAhq0f+000132
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0817_1996050100_123_965_12182461_1_r1488495194_5.zip
Uploading the intermediate file: upload_file_5.zip
  23:38:03 STEP  146 H= 146:00 +CPU=  6.686
  23:38:14 STEP  147 H= 147:00 +CPU= 10.906
  23:38:21 STEP  148 H= 148:00 +CPU=  6.650
  23:38:27 STEP  149 H= 149:00 +CPU=  6.623
  23:38:38 STEP  150 H= 150:00 +CPU= 10.385
  23:38:45 STEP  151 H= 151:00 +CPU=  6.720
  23:38:51 STEP  152 H= 152:00 +CPU=  6.872
  23:39:02 STEP  153 H= 153:00 +CPU= 10.311
  23:39:09 STEP  154 H= 154:00 +CPU=  6.776
  23:39:15 STEP  155 H= 155:00 +CPU=  6.646
  23:39:27 STEP  156 H= 156:00 +CPU= 11.250
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+000156
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+000156
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+000156
  23:39:33 STEP  157 H= 157:00 +CPU=  6.586
  23:39:40 STEP  158 H= 158:00 +CPU=  6.546
  23:39:50 STEP  159 H= 159:00 +CPU= 10.603
  23:39:57 STEP  160 H= 160:00 +CPU=  6.652
  23:40:04 STEP  161 H= 161:00 +CPU=  6.877
  23:40:14 STEP  162 H= 162:00 +CPU= 10.522
  23:40:21 STEP  163 H= 163:00 +CPU=  6.556
  23:40:28 STEP  164 H= 164:00 +CPU=  6.651
  23:40:40 STEP  165 H= 165:00 +CPU= 10.793
  23:40:47 STEP  166 H= 166:00 +CPU=  6.774
  23:40:53 STEP  167 H= 167:00 +CPU=  6.748
  23:41:08 STEP  168 H= 168:00 +CPU= 12.820
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+000168
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+000168
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+000168
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182461/ICMGGhq0f+000144
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182461/ICMSHhq0f+000144
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182461/ICMUAhq0f+000144
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182461/ICMGGhq0f+000156
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182461/ICMSHhq0f+000156
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182461/ICMUAhq0f+000156
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0817_1996050100_123_965_12182461_1_r1488495194_6.zip
Uploading the intermediate file: upload_file_6.zip
  23:41:15 STEP  169 H= 169:00 +CPU=  6.857
  23:41:22 STEP  170 H= 170:00 +CPU=  7.074
  23:41:33 STEP  171 H= 171:00 +CPU= 10.557
  23:41:39 STEP  172 H= 172:00 +CPU=  6.649
  23:41:46 STEP  173 H= 173:00 +CPU=  6.773
  23:41:57 STEP  174 H= 174:00 +CPU= 11.031
  23:42:04 STEP  175 H= 175:00 +CPU=  6.506
  23:42:11 STEP  176 H= 176:00 +CPU=  6.978
  23:42:21 STEP  177 H= 177:00 +CPU= 10.478
  23:42:28 STEP  178 H= 178:00 +CPU=  6.676
  23:42:35 STEP  179 H= 179:00 +CPU=  6.757
  23:42:46 STEP  180 H= 180:00 +CPU= 11.586
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+000180
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+000180
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+000180
  23:42:53 STEP  181 H= 181:00 +CPU=  6.329
  23:42:59 STEP  182 H= 182:00 +CPU=  6.776
  23:43:10 STEP  183 H= 183:00 +CPU= 10.824
  23:43:17 STEP  184 H= 184:00 +CPU=  6.741
  23:43:24 STEP  185 H= 185:00 +CPU=  6.893
  23:43:35 STEP  186 H= 186:00 +CPU= 10.938
  23:43:42 STEP  187 H= 187:00 +CPU=  6.564
  23:43:48 STEP  188 H= 188:00 +CPU=  6.655
  23:43:59 STEP  189 H= 189:00 +CPU= 10.750
  23:44:06 STEP  190 H= 190:00 +CPU=  6.769
  23:44:12 STEP  191 H= 191:00 +CPU=  6.553
  23:44:27 STEP  192 H= 192:00 +CPU= 12.781
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+000192
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+000192
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+000192
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182461/ICMGGhq0f+000168
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182461/ICMSHhq0f+000168
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182461/ICMUAhq0f+000168
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182461/ICMGGhq0f+000180
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182461/ICMSHhq0f+000180
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182461/ICMUAhq0f+000180
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0817_1996050100_123_965_12182461_1_r1488495194_7.zip
Uploading the intermediate file: upload_file_7.zip
  23:44:34 STEP  193 H= 193:00 +CPU=  6.560
  23:44:41 STEP  194 H= 194:00 +CPU=  6.953
  23:44:52 STEP  195 H= 195:00 +CPU= 10.802
  23:44:58 STEP  196 H= 196:00 +CPU=  6.843
  23:45:05 STEP  197 H= 197:00 +CPU=  6.555
  23:45:16 STEP  198 H= 198:00 +CPU= 10.886
  23:45:22 STEP  199 H= 199:00 +CPU=  6.334
  23:45:29 STEP  200 H= 200:00 +CPU=  6.724
  23:45:40 STEP  201 H= 201:00 +CPU= 10.736
  23:45:46 STEP  202 H= 202:00 +CPU=  6.586
  23:45:53 STEP  203 H= 203:00 +CPU=  6.875
  23:46:05 STEP  204 H= 204:00 +CPU= 11.649
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+000204
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+000204
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+000204
  23:46:11 STEP  205 H= 205:00 +CPU=  6.271
  23:46:22 STEP  206 H= 206:00 +CPU=  7.026
  23:47:34 STEP  207 H= 207:00 +CPU= 12.348
  23:47:41 STEP  208 H= 208:00 +CPU=  6.623
  23:47:47 STEP  209 H= 209:00 +CPU=  6.684
  23:47:58 STEP  210 H= 210:00 +CPU= 10.547
  23:48:05 STEP  211 H= 211:00 +CPU=  6.631
  23:48:11 STEP  212 H= 212:00 +CPU=  6.828
  23:48:22 STEP  213 H= 213:00 +CPU= 10.786
  23:48:29 STEP  214 H= 214:00 +CPU=  6.879
  23:48:39 STEP  215 H= 215:00 +CPU=  7.116
  23:48:52 STEP  216 H= 216:00 +CPU= 12.551
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+000216
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+000216
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+000216
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182461/ICMGGhq0f+000192
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182461/ICMSHhq0f+000192
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182461/ICMUAhq0f+000192
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182461/ICMGGhq0f+000204
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182461/ICMSHhq0f+000204
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182461/ICMUAhq0f+000204
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0817_1996050100_123_965_12182461_1_r1488495194_8.zip
  23:48:59 STEP  217 H= 217:00 +CPU=  6.430
Uploading the intermediate file: upload_file_8.zip
  23:49:06 STEP  218 H= 218:00 +CPU=  6.974
  23:49:17 STEP  219 H= 219:00 +CPU= 11.088
  23:49:24 STEP  220 H= 220:00 +CPU=  6.947
  23:49:31 STEP  221 H= 221:00 +CPU=  6.871
  23:49:42 STEP  222 H= 222:00 +CPU= 10.937
  23:49:48 STEP  223 H= 223:00 +CPU=  6.939
  23:49:58 STEP  224 H= 224:00 +CPU=  7.910
  23:50:09 STEP  225 H= 225:00 +CPU= 10.851
  23:50:16 STEP  226 H= 226:00 +CPU=  6.639
  23:50:23 STEP  227 H= 227:00 +CPU=  6.553
  23:50:36 STEP  228 H= 228:00 +CPU= 11.798
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+000228
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+000228
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+000228
  23:50:42 STEP  229 H= 229:00 +CPU=  6.651
  23:50:49 STEP  230 H= 230:00 +CPU=  6.919
  23:51:00 STEP  231 H= 231:00 +CPU= 10.749
  23:51:07 STEP  232 H= 232:00 +CPU=  6.788
  23:51:13 STEP  233 H= 233:00 +CPU=  6.493
  23:51:24 STEP  234 H= 234:00 +CPU= 11.035
  23:51:31 STEP  235 H= 235:00 +CPU=  6.557
  23:51:38 STEP  236 H= 236:00 +CPU=  6.719
  23:51:49 STEP  237 H= 237:00 +CPU= 10.826
  23:51:55 STEP  238 H= 238:00 +CPU=  6.924
  23:52:02 STEP  239 H= 239:00 +CPU=  6.773
  23:52:16 STEP  240 H= 240:00 +CPU= 12.718
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+000240
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+000240
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+000240
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182461/ICMGGhq0f+000216
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182461/ICMSHhq0f+000216
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182461/ICMUAhq0f+000216
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182461/ICMGGhq0f+000228
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182461/ICMSHhq0f+000228
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182461/ICMUAhq0f+000228
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0817_1996050100_123_965_12182461_1_r1488495194_9.zip
Uploading the intermediate file: upload_file_9.zip
  23:52:23 STEP  241 H= 241:00 +CPU=  6.684
  23:52:30 STEP  242 H= 242:00 +CPU=  6.795
  23:52:41 STEP  243 H= 243:00 +CPU= 10.626
  23:52:47 STEP  244 H= 244:00 +CPU=  6.787
  23:52:54 STEP  245 H= 245:00 +CPU=  6.642
  23:53:05 STEP  246 H= 246:00 +CPU= 10.819
  23:53:11 STEP  247 H= 247:00 +CPU=  6.592
  23:53:19 STEP  248 H= 248:00 +CPU=  6.845
  23:53:30 STEP  249 H= 249:00 +CPU= 10.436
  23:53:36 STEP  250 H= 250:00 +CPU=  6.678
  23:53:43 STEP  251 H= 251:00 +CPU=  6.407
  23:53:55 STEP  252 H= 252:00 +CPU= 11.988
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+000252
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+000252
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+000252
  23:54:02 STEP  253 H= 253:00 +CPU=  6.835
  23:54:09 STEP  254 H= 254:00 +CPU=  6.939
  23:54:20 STEP  255 H= 255:00 +CPU= 10.695
  23:54:27 STEP  256 H= 256:00 +CPU=  7.400
  23:54:34 STEP  257 H= 257:00 +CPU=  6.382
  23:54:44 STEP  258 H= 258:00 +CPU= 10.749
  23:54:51 STEP  259 H= 259:00 +CPU=  6.453
  23:54:58 STEP  260 H= 260:00 +CPU=  6.823
  23:55:08 STEP  261 H= 261:00 +CPU= 10.490
  23:55:15 STEP  262 H= 262:00 +CPU=  6.761
  23:55:22 STEP  263 H= 263:00 +CPU=  6.507
  23:55:53 STEP  264 H= 264:00 +CPU= 14.048
  23:56:00 STEP  265 H= 265:00 +CPU=  6.488
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+000264
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+000264
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+000264
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182461/ICMGGhq0f+000240
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182461/ICMSHhq0f+000240
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182461/ICMUAhq0f+000240
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182461/ICMGGhq0f+000252
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182461/ICMSHhq0f+000252
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182461/ICMUAhq0f+000252
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0817_1996050100_123_965_12182461_1_r1488495194_10.zip
Uploading the intermediate file: upload_file_10.zip
  23:56:07 STEP  266 H= 266:00 +CPU=  6.863
  23:56:18 STEP  267 H= 267:00 +CPU= 10.674
  23:56:25 STEP  268 H= 268:00 +CPU=  6.975
  23:56:32 STEP  269 H= 269:00 +CPU=  6.700
  23:56:43 STEP  270 H= 270:00 +CPU= 10.700
  23:56:49 STEP  271 H= 271:00 +CPU=  6.426
  23:56:56 STEP  272 H= 272:00 +CPU=  6.773
  23:57:07 STEP  273 H= 273:00 +CPU= 10.838
  23:57:14 STEP  274 H= 274:00 +CPU=  6.729
  23:57:20 STEP  275 H= 275:00 +CPU=  6.461
  23:57:33 STEP  276 H= 276:00 +CPU= 12.102
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+000276
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+000276
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+000276
  23:57:39 STEP  277 H= 277:00 +CPU=  6.571
  23:57:46 STEP  278 H= 278:00 +CPU=  6.610
The child process terminated with status: 0
..Failed, model did not complete successfully

</stderr_txt>
]]>
No trickles!


©2024 climateprediction.net