climateprediction.net home page
Task 22264047

Task 22264047

Name oifs_43r3_ps_0502_1993050100_123_962_12179146_0
Workunit 12179146
Created 22 Dec 2022, 3:17:40 UTC
Sent 6 Jan 2023, 6:20:34 UTC
Report deadline 5 Feb 2023, 6:20:34 UTC
Received 7 Jan 2023, 6:31:28 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1528008
Run time 12 hours 51 min 57 sec
CPU time 11 hours 38 min 31 sec
Validate state Invalid
Credit 4,717.44
Device peak FLOPS 5.54 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,398.12 MB
Peak swap size 4,978.14 MB
Peak disk usage 7,685.50 MB
Stderr 
<core_client_version>7.16.6</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
rom: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/6/159l_2/clim_data_12179146.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/6/159l_2/clim_data_12179146.zip
Checking for progress XML file: /var/lib/boinc-client/slots/6/progress_file_12179146.xml
Opened progress file ok : /var/lib/boinc-client/slots/6/progress_file_12179146.xml
-- Model is restarting --
Adjusting last_iter, 1016, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/6/progress_file_12179146.xml
last_cpu_time: 34169
upload_file_number: 42
last_iter: 1007
last_upload: 3628800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0502_1993050100_123_962_12179146_0_r863754718
The child process has been launched with process id: 2355466
Executing the command: /var/lib/boinc-client/slots/6/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:pres-incite:1:1:2355466:2355466] [20230106:220957:1673060997:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:pres-incite:1:1:2355466:2355466] [20230106:220957:1673060997:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:pres-incite:1:1:2355466:2355466] [20230106:220957:1673060997:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:pres-incite:1:1:2355466:2355466] [20230106:220957:1673060997:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:pres-incite:1:1:2355466:2355466] [20230106:220957:1673060997:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:pres-incite:1:1:2355466:2355466] [20230106:220957:1673060997:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:pres-incite:1:1:2355466:2355466] [20230106:220957:1673060997:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:pres-incite:1:1:2355466:2355466] [20230106:220957:1673060997:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:pres-incite:1:1:2355466:2355466] [20230106:220957:1673060997:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:pres-incite:1:1:2355466:2355466] [20230106:220957:1673060997:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:pres-incite:1:1:2355466:2355466] [20230106:220957:1673060997:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:pres-incite:1:1:2355466:2355466] [20230106:220957:1673060997:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:pres-incite:1:1:2355466:2355466] [20230106:220957:1673060997:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:pres-incite:1:1:2355466:2355466] [20230106:220957:1673060997:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:pres-incite:1:1:2355466:2355466] [20230106:220957:1673060997:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:pres-incite:1:1:2355466:2355466] [20230106:220957:1673060997:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:pres-incite:1:1:2355466:2355466] [20230106:220957:1673060997:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:pres-incite:1:1:2355466:2355466] [20230106:220957:1673060997:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:pres-incite:1:1:2355466:2355466] [20230106:220957:1673060997:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:pres-incite:1:1:2355466:2355466] [20230106:220957:1673060997:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:pres-incite:1:1:2355466:2355466] [20230106:220957:1673060997:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:pres-incite:1:1:2355466:2355466] [20230106:220957:1673060997:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:pres-incite:1:1:2355466:2355466] [20230106:220957:1673060997:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:pres-incite:1:1:2355466:2355466] [20230106:220957:1673060997:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:pres-incite:1:1:2355466:2355466] [20230106:220957:1673060997:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:pres-incite:1:1:2355466:2355466] [20230106:220957:1673060997:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:pres-incite:1:1:2355466:2355466] [20230106:220957:1673060997:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:pres-incite:1:1:2355466:2355466] [20230106:220957:1673060997:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:pres-incite:1:1:2355466:2355466] [20230106:220957:1673060997:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:pres-incite:1:1:2355466:2355466] [20230106:220957:1673060997:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  22:10:33 STEP 1008 H=1008:00 +CPU= 30.807
  22:10:59 STEP 1009 H=1009:00 +CPU= 17.388
  22:11:16 STEP 1010 H=1010:00 +CPU= 16.718
  22:11:42 STEP 1011 H=1011:00 +CPU= 25.453
  22:11:59 STEP 1012 H=1012:00 +CPU= 16.643
  22:12:16 STEP 1013 H=1013:00 +CPU= 16.265
  22:12:43 STEP 1014 H=1014:00 +CPU= 26.432
  22:12:59 STEP 1015 H=1015:00 +CPU= 16.314
  22:13:16 STEP 1016 H=1016:00 +CPU= 16.817
  22:13:42 STEP 1017 H=1017:00 +CPU= 25.843
  22:13:59 STEP 1018 H=1018:00 +CPU= 16.747
  22:14:16 STEP 1019 H=1019:00 +CPU= 16.265
  22:14:45 STEP 1020 H=1020:00 +CPU= 28.558
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001020
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001020
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001020
  22:15:02 STEP 1021 H=1021:00 +CPU= 16.321
  22:15:19 STEP 1022 H=1022:00 +CPU= 16.779
  22:15:45 STEP 1023 H=1023:00 +CPU= 25.882
  22:16:02 STEP 1024 H=1024:00 +CPU= 16.785
  22:16:19 STEP 1025 H=1025:00 +CPU= 16.309
  22:16:46 STEP 1026 H=1026:00 +CPU= 26.410
  22:17:02 STEP 1027 H=1027:00 +CPU= 16.253
  22:17:19 STEP 1028 H=1028:00 +CPU= 16.755
  22:17:46 STEP 1029 H=1029:00 +CPU= 25.864
  22:18:02 STEP 1030 H=1030:00 +CPU= 16.738
  22:18:20 STEP 1031 H=1031:00 +CPU= 16.780
  22:18:51 STEP 1032 H=1032:00 +CPU= 30.249
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001032
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001032
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179146/ICMGGhq0f+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179146/ICMSHhq0f+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179146/ICMUAhq0f+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179146/ICMGGhq0f+001020
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179146/ICMSHhq0f+001020
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179146/ICMUAhq0f+001020
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0502_1993050100_123_962_12179146_0_r863754718_42.zip
Uploading the intermediate file: upload_file_42.zip
  22:19:07 STEP 1033 H=1033:00 +CPU= 16.258
  22:19:24 STEP 1034 H=1034:00 +CPU= 16.762
  22:19:51 STEP 1035 H=1035:00 +CPU= 25.884
  22:20:08 STEP 1036 H=1036:00 +CPU= 16.758
  22:20:24 STEP 1037 H=1037:00 +CPU= 16.266
  22:20:51 STEP 1038 H=1038:00 +CPU= 26.432
  22:21:07 STEP 1039 H=1039:00 +CPU= 16.283
  22:21:24 STEP 1040 H=1040:00 +CPU= 16.779
  22:21:51 STEP 1041 H=1041:00 +CPU= 25.859
  22:22:08 STEP 1042 H=1042:00 +CPU= 16.755
  22:22:24 STEP 1043 H=1043:00 +CPU= 16.275
  22:22:53 STEP 1044 H=1044:00 +CPU= 28.636
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001044
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001044
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001044
  22:23:10 STEP 1045 H=1045:00 +CPU= 16.298
  22:23:27 STEP 1046 H=1046:00 +CPU= 16.780
  22:23:53 STEP 1047 H=1047:00 +CPU= 25.888
  22:24:10 STEP 1048 H=1048:00 +CPU= 16.775
  22:24:27 STEP 1049 H=1049:00 +CPU= 16.261
  22:24:53 STEP 1050 H=1050:00 +CPU= 26.406
  22:25:10 STEP 1051 H=1051:00 +CPU= 16.285
  22:25:27 STEP 1052 H=1052:00 +CPU= 16.751
  22:25:53 STEP 1053 H=1053:00 +CPU= 25.857
  22:26:10 STEP 1054 H=1054:00 +CPU= 16.739
  22:26:27 STEP 1055 H=1055:00 +CPU= 16.745
  22:27:00 STEP 1056 H=1056:00 +CPU= 30.178
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001056
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001056
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179146/ICMGGhq0f+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179146/ICMSHhq0f+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179146/ICMUAhq0f+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179146/ICMGGhq0f+001044
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179146/ICMSHhq0f+001044
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179146/ICMUAhq0f+001044
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0502_1993050100_123_962_12179146_0_r863754718_43.zip
Uploading the intermediate file: upload_file_43.zip
  22:27:16 STEP 1057 H=1057:00 +CPU= 16.264
  22:27:33 STEP 1058 H=1058:00 +CPU= 16.739
  22:28:00 STEP 1059 H=1059:00 +CPU= 25.869
  22:28:17 STEP 1060 H=1060:00 +CPU= 16.775
  22:28:33 STEP 1061 H=1061:00 +CPU= 16.289
  22:29:00 STEP 1062 H=1062:00 +CPU= 26.556
  22:29:17 STEP 1063 H=1063:00 +CPU= 16.337
  22:29:34 STEP 1064 H=1064:00 +CPU= 16.843
  22:30:00 STEP 1065 H=1065:00 +CPU= 25.946
  22:30:17 STEP 1066 H=1066:00 +CPU= 16.848
  22:30:34 STEP 1067 H=1067:00 +CPU= 16.327
  22:31:03 STEP 1068 H=1068:00 +CPU= 28.686
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001068
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001068
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001068
  22:31:20 STEP 1069 H=1069:00 +CPU= 16.331
  22:31:37 STEP 1070 H=1070:00 +CPU= 16.805
  22:32:03 STEP 1071 H=1071:00 +CPU= 25.981
  22:32:20 STEP 1072 H=1072:00 +CPU= 16.864
  22:32:37 STEP 1073 H=1073:00 +CPU= 16.349
  22:33:03 STEP 1074 H=1074:00 +CPU= 26.470
  22:33:20 STEP 1075 H=1075:00 +CPU= 16.302
  22:33:37 STEP 1076 H=1076:00 +CPU= 16.811
  22:34:03 STEP 1077 H=1077:00 +CPU= 25.917
  22:34:20 STEP 1078 H=1078:00 +CPU= 16.801
  22:34:37 STEP 1079 H=1079:00 +CPU= 16.808
  22:35:08 STEP 1080 H=1080:00 +CPU= 30.159
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001080
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001080
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179146/ICMGGhq0f+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179146/ICMSHhq0f+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179146/ICMUAhq0f+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179146/ICMGGhq0f+001068
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179146/ICMSHhq0f+001068
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179146/ICMUAhq0f+001068
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0502_1993050100_123_962_12179146_0_r863754718_44.zip
Uploading the intermediate file: upload_file_44.zip
  22:35:25 STEP 1081 H=1081:00 +CPU= 16.274
  22:35:42 STEP 1082 H=1082:00 +CPU= 16.783
  22:36:08 STEP 1083 H=1083:00 +CPU= 25.940
  22:36:25 STEP 1084 H=1084:00 +CPU= 16.810
  22:36:42 STEP 1085 H=1085:00 +CPU= 16.322
  22:37:09 STEP 1086 H=1086:00 +CPU= 26.514
  22:37:25 STEP 1087 H=1087:00 +CPU= 16.347
  22:37:43 STEP 1088 H=1088:00 +CPU= 16.815
  22:38:09 STEP 1089 H=1089:00 +CPU= 25.946
  22:38:26 STEP 1090 H=1090:00 +CPU= 16.816
  22:38:43 STEP 1091 H=1091:00 +CPU= 16.314
  22:39:12 STEP 1092 H=1092:00 +CPU= 28.660
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001092
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001092
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001092
  22:39:28 STEP 1093 H=1093:00 +CPU= 16.333
  22:39:45 STEP 1094 H=1094:00 +CPU= 16.801
  22:40:12 STEP 1095 H=1095:00 +CPU= 25.991
  22:40:29 STEP 1096 H=1096:00 +CPU= 16.815
  22:40:45 STEP 1097 H=1097:00 +CPU= 16.312
  22:41:12 STEP 1098 H=1098:00 +CPU= 26.478
  22:41:29 STEP 1099 H=1099:00 +CPU= 16.312
  22:41:46 STEP 1100 H=1100:00 +CPU= 16.792
  22:42:12 STEP 1101 H=1101:00 +CPU= 25.906
  22:42:29 STEP 1102 H=1102:00 +CPU= 16.783
  22:42:46 STEP 1103 H=1103:00 +CPU= 16.869
  22:43:17 STEP 1104 H=1104:00 +CPU= 30.361
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001104
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001104
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179146/ICMGGhq0f+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179146/ICMSHhq0f+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179146/ICMUAhq0f+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179146/ICMGGhq0f+001092
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179146/ICMSHhq0f+001092
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179146/ICMUAhq0f+001092
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0502_1993050100_123_962_12179146_0_r863754718_45.zip
Uploading the intermediate file: upload_file_45.zip
  22:43:34 STEP 1105 H=1105:00 +CPU= 16.310
  22:43:51 STEP 1106 H=1106:00 +CPU= 16.832
  22:44:17 STEP 1107 H=1107:00 +CPU= 26.024
  22:44:34 STEP 1108 H=1108:00 +CPU= 16.826
  22:44:51 STEP 1109 H=1109:00 +CPU= 16.353
  22:45:18 STEP 1110 H=1110:00 +CPU= 26.610
  22:45:34 STEP 1111 H=1111:00 +CPU= 16.440
  22:45:52 STEP 1112 H=1112:00 +CPU= 16.876
  22:46:18 STEP 1113 H=1113:00 +CPU= 26.093
  22:46:35 STEP 1114 H=1114:00 +CPU= 16.948
  22:46:52 STEP 1115 H=1115:00 +CPU= 16.464
  22:47:21 STEP 1116 H=1116:00 +CPU= 28.716
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001116
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001116
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001116
  22:47:38 STEP 1117 H=1117:00 +CPU= 16.361
  22:47:55 STEP 1118 H=1118:00 +CPU= 16.875
  22:48:21 STEP 1119 H=1119:00 +CPU= 26.021
  22:48:38 STEP 1120 H=1120:00 +CPU= 16.859
  22:48:55 STEP 1121 H=1121:00 +CPU= 16.334
  22:49:22 STEP 1122 H=1122:00 +CPU= 26.494
  22:49:38 STEP 1123 H=1123:00 +CPU= 16.360
  22:49:55 STEP 1124 H=1124:00 +CPU= 16.829
  22:50:22 STEP 1125 H=1125:00 +CPU= 25.945
  22:50:39 STEP 1126 H=1126:00 +CPU= 16.809
  22:50:56 STEP 1127 H=1127:00 +CPU= 16.807
  22:51:27 STEP 1128 H=1128:00 +CPU= 30.405
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001128
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001128
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179146/ICMGGhq0f+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179146/ICMSHhq0f+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179146/ICMUAhq0f+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179146/ICMGGhq0f+001116
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179146/ICMSHhq0f+001116
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179146/ICMUAhq0f+001116
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0502_1993050100_123_962_12179146_0_r863754718_46.zip
Uploading the intermediate file: upload_file_46.zip
  22:51:43 STEP 1129 H=1129:00 +CPU= 16.297
  22:52:00 STEP 1130 H=1130:00 +CPU= 16.747
  22:52:26 STEP 1131 H=1131:00 +CPU= 25.967
  22:52:44 STEP 1132 H=1132:00 +CPU= 16.815
  22:53:00 STEP 1133 H=1133:00 +CPU= 16.319
  22:53:27 STEP 1134 H=1134:00 +CPU= 26.541
  22:53:44 STEP 1135 H=1135:00 +CPU= 16.319
  22:54:01 STEP 1136 H=1136:00 +CPU= 16.817
  22:54:29 STEP 1137 H=1137:00 +CPU= 27.128
  22:54:48 STEP 1138 H=1138:00 +CPU= 18.468
  22:55:06 STEP 1139 H=1139:00 +CPU= 17.538
  22:55:36 STEP 1140 H=1140:00 +CPU= 29.075
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001140
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001140
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001140
  22:55:52 STEP 1141 H=1141:00 +CPU= 16.055
  22:56:09 STEP 1142 H=1142:00 +CPU= 16.545
  22:56:36 STEP 1143 H=1143:00 +CPU= 26.560
  22:56:54 STEP 1144 H=1144:00 +CPU= 17.927
  22:57:20 STEP 1145 H=1145:00 +CPU= 17.677
  22:57:48 STEP 1146 H=1146:00 +CPU= 27.027
  22:58:04 STEP 1147 H=1147:00 +CPU= 16.032
  22:58:21 STEP 1148 H=1148:00 +CPU= 16.525
  22:58:47 STEP 1149 H=1149:00 +CPU= 25.897
  22:59:04 STEP 1150 H=1150:00 +CPU= 16.830
  22:59:21 STEP 1151 H=1151:00 +CPU= 16.786
  22:59:53 STEP 1152 H=1152:00 +CPU= 30.319
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001152
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001152
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179146/ICMGGhq0f+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179146/ICMSHhq0f+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179146/ICMUAhq0f+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179146/ICMGGhq0f+001140
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179146/ICMSHhq0f+001140
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179146/ICMUAhq0f+001140
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0502_1993050100_123_962_12179146_0_r863754718_47.zip
Uploading the intermediate file: upload_file_47.zip
  23:00:10 STEP 1153 H=1153:00 +CPU= 16.296
  23:00:27 STEP 1154 H=1154:00 +CPU= 16.786
  23:00:53 STEP 1155 H=1155:00 +CPU= 25.896
  23:01:10 STEP 1156 H=1156:00 +CPU= 16.785
  23:01:26 STEP 1157 H=1157:00 +CPU= 16.292
  23:01:53 STEP 1158 H=1158:00 +CPU= 26.492
  23:02:10 STEP 1159 H=1159:00 +CPU= 16.321
  23:02:27 STEP 1160 H=1160:00 +CPU= 16.802
  23:02:53 STEP 1161 H=1161:00 +CPU= 26.004
  23:03:11 STEP 1162 H=1162:00 +CPU= 16.818
  23:03:27 STEP 1163 H=1163:00 +CPU= 16.314
  23:03:56 STEP 1164 H=1164:00 +CPU= 28.745
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001164
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001164
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001164
  23:04:13 STEP 1165 H=1165:00 +CPU= 16.300
  23:04:31 STEP 1166 H=1166:00 +CPU= 17.691
  23:05:02 STEP 1167 H=1167:00 +CPU= 27.909
  23:05:19 STEP 1168 H=1168:00 +CPU= 16.545
  23:05:35 STEP 1169 H=1169:00 +CPU= 15.976
  23:06:01 STEP 1170 H=1170:00 +CPU= 25.978
  23:06:18 STEP 1171 H=1171:00 +CPU= 16.368
  23:06:35 STEP 1172 H=1172:00 +CPU= 16.863
  23:07:01 STEP 1173 H=1173:00 +CPU= 26.021
  23:07:18 STEP 1174 H=1174:00 +CPU= 16.819
  23:07:36 STEP 1175 H=1175:00 +CPU= 16.858
  23:08:14 STEP 1176 H=1176:00 +CPU= 30.380
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001176
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001176
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179146/ICMGGhq0f+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179146/ICMSHhq0f+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179146/ICMUAhq0f+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179146/ICMGGhq0f+001164
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179146/ICMSHhq0f+001164
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179146/ICMUAhq0f+001164
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0502_1993050100_123_962_12179146_0_r863754718_48.zip
Uploading the intermediate file: upload_file_48.zip
  23:08:31 STEP 1177 H=1177:00 +CPU= 16.344
  23:08:49 STEP 1178 H=1178:00 +CPU= 17.551
  23:09:18 STEP 1179 H=1179:00 +CPU= 27.997
  23:09:38 STEP 1180 H=1180:00 +CPU= 18.326
  23:09:56 STEP 1181 H=1181:00 +CPU= 17.782
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1993050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0502
(argv4) batchid: 962
(argv5) wuid: 12179146
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/6
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179146
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/6/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/6/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_a7986e1c12565b4d1cb39bb991df9d46 to: /var/lib/boinc-client/slots/6/oifs_43r3_ps_0502_1993050100_123_962_12179146.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/6/oifs_43r3_ps_0502_1993050100_123_962_12179146.zip
ic_ancil_file: ic_ancil_12179146
ifsdata_file: ifsdata_12179146
climate_data_file: clim_data_12179146
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_f302c00c721d8d64a5e5ad3becdc4ec4 to: /var/lib/boinc-client/slots/6/ic_ancil_12179146.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/6/ic_ancil_12179146.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_44c18ade429c2c2b65b91018a0933848 to: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12179146.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12179146.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/6/159l_2/clim_data_12179146.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/6/159l_2/clim_data_12179146.zip
Checking for progress XML file: /var/lib/boinc-client/slots/6/progress_file_12179146.xml
Opened progress file ok : /var/lib/boinc-client/slots/6/progress_file_12179146.xml
-- Model is restarting --
Adjusting last_iter, 1182, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/6/progress_file_12179146.xml
last_cpu_time: 37679
upload_file_number: 49
last_iter: 1175
last_upload: 4233600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0502_1993050100_123_962_12179146_0_r863754718
The child process has been launched with process id: 2359539
Executing the command: /var/lib/boinc-client/slots/6/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:pres-incite:1:1:2359539:2359539] [20230106:231104:1673064664:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:pres-incite:1:1:2359539:2359539] [20230106:231104:1673064664:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:pres-incite:1:1:2359539:2359539] [20230106:231104:1673064664:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:pres-incite:1:1:2359539:2359539] [20230106:231104:1673064664:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:pres-incite:1:1:2359539:2359539] [20230106:231104:1673064664:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:pres-incite:1:1:2359539:2359539] [20230106:231104:1673064664:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:pres-incite:1:1:2359539:2359539] [20230106:231104:1673064664:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:pres-incite:1:1:2359539:2359539] [20230106:231104:1673064664:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:pres-incite:1:1:2359539:2359539] [20230106:231104:1673064664:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:pres-incite:1:1:2359539:2359539] [20230106:231104:1673064664:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:pres-incite:1:1:2359539:2359539] [20230106:231104:1673064664:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:pres-incite:1:1:2359539:2359539] [20230106:231104:1673064664:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:pres-incite:1:1:2359539:2359539] [20230106:231104:1673064664:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:pres-incite:1:1:2359539:2359539] [20230106:231104:1673064664:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:pres-incite:1:1:2359539:2359539] [20230106:231104:1673064664:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:pres-incite:1:1:2359539:2359539] [20230106:231104:1673064664:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:pres-incite:1:1:2359539:2359539] [20230106:231104:1673064664:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:pres-incite:1:1:2359539:2359539] [20230106:231104:1673064664:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:pres-incite:1:1:2359539:2359539] [20230106:231104:1673064664:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:pres-incite:1:1:2359539:2359539] [20230106:231104:1673064664:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:pres-incite:1:1:2359539:2359539] [20230106:231104:1673064664:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:pres-incite:1:1:2359539:2359539] [20230106:231104:1673064664:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:pres-incite:1:1:2359539:2359539] [20230106:231104:1673064664:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:pres-incite:1:1:2359539:2359539] [20230106:231104:1673064664:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:pres-incite:1:1:2359539:2359539] [20230106:231104:1673064664:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:pres-incite:1:1:2359539:2359539] [20230106:231104:1673064664:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:pres-incite:1:1:2359539:2359539] [20230106:231104:1673064664:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:pres-incite:1:1:2359539:2359539] [20230106:231104:1673064664:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:pres-incite:1:1:2359539:2359539] [20230106:231104:1673064664:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:pres-incite:1:1:2359539:2359539] [20230106:231104:1673064664:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  23:11:42 STEP 1176 H=1176:00 +CPU= 31.355
  23:12:01 STEP 1177 H=1177:00 +CPU= 17.053
  23:12:18 STEP 1178 H=1178:00 +CPU= 16.499
  23:12:44 STEP 1179 H=1179:00 +CPU= 25.872
  23:13:01 STEP 1180 H=1180:00 +CPU= 16.682
  23:13:18 STEP 1181 H=1181:00 +CPU= 16.368
  23:13:45 STEP 1182 H=1182:00 +CPU= 26.653
  23:14:01 STEP 1183 H=1183:00 +CPU= 16.403
  23:14:19 STEP 1184 H=1184:00 +CPU= 16.830
  23:14:45 STEP 1185 H=1185:00 +CPU= 26.041
  23:15:02 STEP 1186 H=1186:00 +CPU= 16.840
  23:15:19 STEP 1187 H=1187:00 +CPU= 16.365
  23:15:48 STEP 1188 H=1188:00 +CPU= 28.773
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001188
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001188
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001188
  23:16:05 STEP 1189 H=1189:00 +CPU= 16.408
  23:16:22 STEP 1190 H=1190:00 +CPU= 16.880
  23:16:48 STEP 1191 H=1191:00 +CPU= 26.091
  23:17:05 STEP 1192 H=1192:00 +CPU= 16.899
  23:17:22 STEP 1193 H=1193:00 +CPU= 16.395
  23:17:49 STEP 1194 H=1194:00 +CPU= 26.635
  23:18:06 STEP 1195 H=1195:00 +CPU= 16.377
  23:18:23 STEP 1196 H=1196:00 +CPU= 16.862
  23:18:49 STEP 1197 H=1197:00 +CPU= 26.078
  23:19:06 STEP 1198 H=1198:00 +CPU= 16.876
  23:19:24 STEP 1199 H=1199:00 +CPU= 16.832
  23:19:55 STEP 1200 H=1200:00 +CPU= 30.044
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001200
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001200
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179146/ICMGGhq0f+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179146/ICMSHhq0f+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179146/ICMUAhq0f+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179146/ICMGGhq0f+001188
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179146/ICMSHhq0f+001188
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179146/ICMUAhq0f+001188
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0502_1993050100_123_962_12179146_0_r863754718_49.zip
Uploading the intermediate file: upload_file_49.zip
  23:20:11 STEP 1201 H=1201:00 +CPU= 16.332
  23:20:29 STEP 1202 H=1202:00 +CPU= 16.802
  23:20:55 STEP 1203 H=1203:00 +CPU= 26.035
  23:21:12 STEP 1204 H=1204:00 +CPU= 16.834
  23:21:29 STEP 1205 H=1205:00 +CPU= 16.345
  23:21:56 STEP 1206 H=1206:00 +CPU= 26.598
  23:22:12 STEP 1207 H=1207:00 +CPU= 16.371
  23:22:29 STEP 1208 H=1208:00 +CPU= 16.878
  23:22:56 STEP 1209 H=1209:00 +CPU= 26.113
  23:23:13 STEP 1210 H=1210:00 +CPU= 16.883
  23:23:30 STEP 1211 H=1211:00 +CPU= 16.392
  23:23:59 STEP 1212 H=1212:00 +CPU= 28.817
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001212
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001212
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001212
  23:24:16 STEP 1213 H=1213:00 +CPU= 16.412
  23:24:33 STEP 1214 H=1214:00 +CPU= 16.879
  23:24:59 STEP 1215 H=1215:00 +CPU= 26.113
  23:25:16 STEP 1216 H=1216:00 +CPU= 16.891
  23:25:33 STEP 1217 H=1217:00 +CPU= 16.380
  23:26:00 STEP 1218 H=1218:00 +CPU= 26.627
  23:26:17 STEP 1219 H=1219:00 +CPU= 16.382
  23:26:34 STEP 1220 H=1220:00 +CPU= 16.900
  23:27:00 STEP 1221 H=1221:00 +CPU= 26.076
  23:27:17 STEP 1222 H=1222:00 +CPU= 16.868
  23:27:35 STEP 1223 H=1223:00 +CPU= 16.864
  23:28:06 STEP 1224 H=1224:00 +CPU= 30.183
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001224
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001224
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179146/ICMGGhq0f+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179146/ICMSHhq0f+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179146/ICMUAhq0f+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179146/ICMGGhq0f+001212
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179146/ICMSHhq0f+001212
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179146/ICMUAhq0f+001212
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0502_1993050100_123_962_12179146_0_r863754718_50.zip
Uploading the intermediate file: upload_file_50.zip
  23:28:22 STEP 1225 H=1225:00 +CPU= 16.358
  23:28:39 STEP 1226 H=1226:00 +CPU= 16.787
  23:29:06 STEP 1227 H=1227:00 +CPU= 26.092
  23:29:23 STEP 1228 H=1228:00 +CPU= 16.873
  23:29:40 STEP 1229 H=1229:00 +CPU= 16.398
  23:30:07 STEP 1230 H=1230:00 +CPU= 26.695
  23:30:23 STEP 1231 H=1231:00 +CPU= 16.415
  23:30:41 STEP 1232 H=1232:00 +CPU= 16.926
  23:31:07 STEP 1233 H=1233:00 +CPU= 26.142
  23:31:24 STEP 1234 H=1234:00 +CPU= 16.946
  23:31:41 STEP 1235 H=1235:00 +CPU= 16.422
  23:32:10 STEP 1236 H=1236:00 +CPU= 28.856
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001236
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001236
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001236
  23:32:27 STEP 1237 H=1237:00 +CPU= 16.487
  23:32:44 STEP 1238 H=1238:00 +CPU= 17.007
  23:33:11 STEP 1239 H=1239:00 +CPU= 26.141
  23:33:28 STEP 1240 H=1240:00 +CPU= 16.897
  23:33:45 STEP 1241 H=1241:00 +CPU= 16.426
  23:34:12 STEP 1242 H=1242:00 +CPU= 26.680
  23:34:28 STEP 1243 H=1243:00 +CPU= 16.448
  23:34:46 STEP 1244 H=1244:00 +CPU= 16.946
  23:35:12 STEP 1245 H=1245:00 +CPU= 26.171
  23:35:29 STEP 1246 H=1246:00 +CPU= 16.913
  23:35:46 STEP 1247 H=1247:00 +CPU= 16.907
  23:36:17 STEP 1248 H=1248:00 +CPU= 30.572
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001248
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001248
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179146/ICMGGhq0f+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179146/ICMSHhq0f+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179146/ICMUAhq0f+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179146/ICMGGhq0f+001236
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179146/ICMSHhq0f+001236
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179146/ICMUAhq0f+001236
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0502_1993050100_123_962_12179146_0_r863754718_51.zip
Uploading the intermediate file: upload_file_51.zip
  23:36:34 STEP 1249 H=1249:00 +CPU= 16.391
  23:36:51 STEP 1250 H=1250:00 +CPU= 16.863
  23:37:18 STEP 1251 H=1251:00 +CPU= 26.136
  23:37:35 STEP 1252 H=1252:00 +CPU= 16.888
  23:37:51 STEP 1253 H=1253:00 +CPU= 16.405
  23:38:19 STEP 1254 H=1254:00 +CPU= 26.698
  23:38:35 STEP 1255 H=1255:00 +CPU= 16.435
  23:38:52 STEP 1256 H=1256:00 +CPU= 16.932
  23:39:19 STEP 1257 H=1257:00 +CPU= 26.132
  23:39:36 STEP 1258 H=1258:00 +CPU= 16.920
  23:39:52 STEP 1259 H=1259:00 +CPU= 16.395
  23:40:22 STEP 1260 H=1260:00 +CPU= 28.879
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001260
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001260
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001260
  23:40:38 STEP 1261 H=1261:00 +CPU= 16.458
  23:40:56 STEP 1262 H=1262:00 +CPU= 16.947
  23:41:22 STEP 1263 H=1263:00 +CPU= 26.162
  23:41:39 STEP 1264 H=1264:00 +CPU= 16.937
  23:41:56 STEP 1265 H=1265:00 +CPU= 16.446
  23:42:23 STEP 1266 H=1266:00 +CPU= 26.699
  23:42:40 STEP 1267 H=1267:00 +CPU= 16.423
  23:42:57 STEP 1268 H=1268:00 +CPU= 16.918
  23:43:23 STEP 1269 H=1269:00 +CPU= 26.179
  23:43:40 STEP 1270 H=1270:00 +CPU= 16.922
  23:43:57 STEP 1271 H=1271:00 +CPU= 16.907
  23:44:29 STEP 1272 H=1272:00 +CPU= 30.544
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001272
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001272
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179146/ICMGGhq0f+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179146/ICMSHhq0f+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179146/ICMUAhq0f+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179146/ICMGGhq0f+001260
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179146/ICMSHhq0f+001260
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179146/ICMUAhq0f+001260
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0502_1993050100_123_962_12179146_0_r863754718_52.zip
Uploading the intermediate file: upload_file_52.zip
  23:44:46 STEP 1273 H=1273:00 +CPU= 16.385
  23:45:03 STEP 1274 H=1274:00 +CPU= 16.865
  23:45:30 STEP 1275 H=1275:00 +CPU= 26.170
  23:45:47 STEP 1276 H=1276:00 +CPU= 16.909
  23:46:04 STEP 1277 H=1277:00 +CPU= 16.449
  23:46:31 STEP 1278 H=1278:00 +CPU= 26.723
  23:46:48 STEP 1279 H=1279:00 +CPU= 16.459
  23:47:05 STEP 1280 H=1280:00 +CPU= 16.978
  23:47:31 STEP 1281 H=1281:00 +CPU= 26.199
  23:47:49 STEP 1282 H=1282:00 +CPU= 16.952
  23:48:05 STEP 1283 H=1283:00 +CPU= 16.448
  23:48:35 STEP 1284 H=1284:00 +CPU= 28.917
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001284
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001284
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001284
  23:48:51 STEP 1285 H=1285:00 +CPU= 16.464
  23:49:08 STEP 1286 H=1286:00 +CPU= 16.962
  23:49:35 STEP 1287 H=1287:00 +CPU= 26.186
  23:49:52 STEP 1288 H=1288:00 +CPU= 16.931
  23:50:09 STEP 1289 H=1289:00 +CPU= 16.450
  23:50:36 STEP 1290 H=1290:00 +CPU= 26.721
  23:50:53 STEP 1291 H=1291:00 +CPU= 16.430
  23:51:10 STEP 1292 H=1292:00 +CPU= 16.925
  23:51:36 STEP 1293 H=1293:00 +CPU= 26.189
  23:51:54 STEP 1294 H=1294:00 +CPU= 16.918
  23:52:11 STEP 1295 H=1295:00 +CPU= 16.893
  23:52:42 STEP 1296 H=1296:00 +CPU= 30.634
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001296
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001296
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179146/ICMGGhq0f+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179146/ICMSHhq0f+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179146/ICMUAhq0f+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179146/ICMGGhq0f+001284
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179146/ICMSHhq0f+001284
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179146/ICMUAhq0f+001284
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0502_1993050100_123_962_12179146_0_r863754718_53.zip
Uploading the intermediate file: upload_file_53.zip
  23:52:59 STEP 1297 H=1297:00 +CPU= 16.406
  23:53:16 STEP 1298 H=1298:00 +CPU= 16.938
  23:53:42 STEP 1299 H=1299:00 +CPU= 26.197
  23:54:00 STEP 1300 H=1300:00 +CPU= 16.934
  23:54:16 STEP 1301 H=1301:00 +CPU= 16.469
  23:54:43 STEP 1302 H=1302:00 +CPU= 26.743
  23:55:00 STEP 1303 H=1303:00 +CPU= 16.470
  23:55:17 STEP 1304 H=1304:00 +CPU= 16.952
  23:55:45 STEP 1305 H=1305:00 +CPU= 26.577
  23:56:02 STEP 1306 H=1306:00 +CPU= 17.223
  23:56:19 STEP 1307 H=1307:00 +CPU= 16.708
  23:56:49 STEP 1308 H=1308:00 +CPU= 29.347
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001308
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001308
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001308
  23:57:06 STEP 1309 H=1309:00 +CPU= 16.575
  23:57:24 STEP 1310 H=1310:00 +CPU= 17.074
  23:57:50 STEP 1311 H=1311:00 +CPU= 26.172
  23:58:07 STEP 1312 H=1312:00 +CPU= 16.955
  23:58:24 STEP 1313 H=1313:00 +CPU= 16.469
  23:58:51 STEP 1314 H=1314:00 +CPU= 26.728
  23:59:08 STEP 1315 H=1315:00 +CPU= 16.447
  23:59:25 STEP 1316 H=1316:00 +CPU= 16.935
  23:59:52 STEP 1317 H=1317:00 +CPU= 26.163
  00:00:09 STEP 1318 H=1318:00 +CPU= 16.946
  00:00:27 STEP 1319 H=1319:00 +CPU= 17.281
  00:01:06 STEP 1320 H=1320:00 +CPU= 31.305
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001320
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001320
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179146/ICMGGhq0f+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179146/ICMSHhq0f+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179146/ICMUAhq0f+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179146/ICMGGhq0f+001308
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179146/ICMSHhq0f+001308
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179146/ICMUAhq0f+001308
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0502_1993050100_123_962_12179146_0_r863754718_54.zip
Uploading the intermediate file: upload_file_54.zip
  00:01:22 STEP 1321 H=1321:00 +CPU= 16.031
  00:01:38 STEP 1322 H=1322:00 +CPU= 16.498
  00:02:04 STEP 1323 H=1323:00 +CPU= 25.635
  00:02:22 STEP 1324 H=1324:00 +CPU= 17.871
  00:02:40 STEP 1325 H=1325:00 +CPU= 17.564
  00:03:16 STEP 1326 H=1326:00 +CPU= 26.726
  00:03:33 STEP 1327 H=1327:00 +CPU= 16.165
  00:03:50 STEP 1328 H=1328:00 +CPU= 16.556
  00:04:17 STEP 1329 H=1329:00 +CPU= 26.869
  00:04:36 STEP 1330 H=1330:00 +CPU= 18.165
  00:05:05 STEP 1331 H=1331:00 +CPU= 16.898
  00:05:34 STEP 1332 H=1332:00 +CPU= 28.332
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001332
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001332
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001332
  00:05:50 STEP 1333 H=1333:00 +CPU= 16.127
  00:06:08 STEP 1334 H=1334:00 +CPU= 17.827
  00:06:37 STEP 1335 H=1335:00 +CPU= 27.640
  00:06:58 STEP 1336 H=1336:00 +CPU= 18.278
  00:07:16 STEP 1337 H=1337:00 +CPU= 17.587
  00:07:43 STEP 1338 H=1338:00 +CPU= 26.343
  00:07:59 STEP 1339 H=1339:00 +CPU= 16.122
  00:08:16 STEP 1340 H=1340:00 +CPU= 16.582
  00:08:44 STEP 1341 H=1341:00 +CPU= 27.149
  00:09:06 STEP 1342 H=1342:00 +CPU= 18.006
  00:09:23 STEP 1343 H=1343:00 +CPU= 16.576
  00:10:05 STEP 1344 H=1344:00 +CPU= 29.838
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001344
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001344
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179146/ICMGGhq0f+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179146/ICMSHhq0f+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179146/ICMUAhq0f+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179146/ICMGGhq0f+001332
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179146/ICMSHhq0f+001332
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179146/ICMUAhq0f+001332
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0502_1993050100_123_962_12179146_0_r863754718_55.zip
Uploading the intermediate file: upload_file_55.zip
  00:10:23 STEP 1345 H=1345:00 +CPU= 17.071
  00:10:42 STEP 1346 H=1346:00 +CPU= 18.131
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1993050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0502
(argv4) batchid: 962
(argv5) wuid: 12179146
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/6
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179146
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/6/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/6/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_a7986e1c12565b4d1cb39bb991df9d46 to: /var/lib/boinc-client/slots/6/oifs_43r3_ps_0502_1993050100_123_962_12179146.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/6/oifs_43r3_ps_0502_1993050100_123_962_12179146.zip
ic_ancil_file: ic_ancil_12179146
ifsdata_file: ifsdata_12179146
climate_data_file: clim_data_12179146
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_f302c00c721d8d64a5e5ad3becdc4ec4 to: /var/lib/boinc-client/slots/6/ic_ancil_12179146.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/6/ic_ancil_12179146.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_44c18ade429c2c2b65b91018a0933848 to: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12179146.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12179146.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/6/159l_2/clim_data_12179146.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/6/159l_2/clim_data_12179146.zip
Checking for progress XML file: /var/lib/boinc-client/slots/6/progress_file_12179146.xml
Opened progress file ok : /var/lib/boinc-client/slots/6/progress_file_12179146.xml
-- Model is restarting --
Adjusting last_iter, 1347, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/6/progress_file_12179146.xml
last_cpu_time: 41165
upload_file_number: 56
last_iter: 1343
last_upload: 4838400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0502_1993050100_123_962_12179146_0_r863754718
The child process has been launched with process id: 2363983
Executing the command: /var/lib/boinc-client/slots/6/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:pres-incite:1:1:2363983:2363983] [20230107:001713:1673068633:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:pres-incite:1:1:2363983:2363983] [20230107:001713:1673068633:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:pres-incite:1:1:2363983:2363983] [20230107:001713:1673068633:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:pres-incite:1:1:2363983:2363983] [20230107:001713:1673068633:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:pres-incite:1:1:2363983:2363983] [20230107:001713:1673068633:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:pres-incite:1:1:2363983:2363983] [20230107:001713:1673068633:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:pres-incite:1:1:2363983:2363983] [20230107:001713:1673068633:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:pres-incite:1:1:2363983:2363983] [20230107:001713:1673068633:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:pres-incite:1:1:2363983:2363983] [20230107:001713:1673068633:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:pres-incite:1:1:2363983:2363983] [20230107:001713:1673068633:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:pres-incite:1:1:2363983:2363983] [20230107:001713:1673068633:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:pres-incite:1:1:2363983:2363983] [20230107:001713:1673068633:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:pres-incite:1:1:2363983:2363983] [20230107:001713:1673068633:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:pres-incite:1:1:2363983:2363983] [20230107:001713:1673068633:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:pres-incite:1:1:2363983:2363983] [20230107:001713:1673068633:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:pres-incite:1:1:2363983:2363983] [20230107:001713:1673068633:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:pres-incite:1:1:2363983:2363983] [20230107:001713:1673068633:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:pres-incite:1:1:2363983:2363983] [20230107:001713:1673068633:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:pres-incite:1:1:2363983:2363983] [20230107:001713:1673068633:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:pres-incite:1:1:2363983:2363983] [20230107:001713:1673068633:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:pres-incite:1:1:2363983:2363983] [20230107:001713:1673068633:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:pres-incite:1:1:2363983:2363983] [20230107:001713:1673068633:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:pres-incite:1:1:2363983:2363983] [20230107:001713:1673068633:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:pres-incite:1:1:2363983:2363983] [20230107:001713:1673068633:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:pres-incite:1:1:2363983:2363983] [20230107:001713:1673068633:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:pres-incite:1:1:2363983:2363983] [20230107:001713:1673068633:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:pres-incite:1:1:2363983:2363983] [20230107:001713:1673068633:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:pres-incite:1:1:2363983:2363983] [20230107:001713:1673068633:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:pres-incite:1:1:2363983:2363983] [20230107:001713:1673068633:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:pres-incite:1:1:2363983:2363983] [20230107:001713:1673068633:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  00:17:57 STEP 1344 H=1344:00 +CPU= 31.061
  00:18:13 STEP 1345 H=1345:00 +CPU= 16.286
  00:18:30 STEP 1346 H=1346:00 +CPU= 16.418
  00:18:55 STEP 1347 H=1347:00 +CPU= 25.297
  00:19:13 STEP 1348 H=1348:00 +CPU= 17.213
  00:19:36 STEP 1349 H=1349:00 +CPU= 17.317
  00:20:02 STEP 1350 H=1350:00 +CPU= 25.903
  00:20:18 STEP 1351 H=1351:00 +CPU= 15.968
  00:20:34 STEP 1352 H=1352:00 +CPU= 16.503
  00:21:02 STEP 1353 H=1353:00 +CPU= 26.829
  00:21:21 STEP 1354 H=1354:00 +CPU= 18.017
  00:21:37 STEP 1355 H=1355:00 +CPU= 16.256
  00:22:05 STEP 1356 H=1356:00 +CPU= 28.045
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001356
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001356
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001356
  00:22:21 STEP 1357 H=1357:00 +CPU= 16.022
  00:22:40 STEP 1358 H=1358:00 +CPU= 17.100
  00:23:09 STEP 1359 H=1359:00 +CPU= 27.179
  00:23:28 STEP 1360 H=1360:00 +CPU= 17.385
  00:23:45 STEP 1361 H=1361:00 +CPU= 15.993
  00:24:11 STEP 1362 H=1362:00 +CPU= 25.951
  00:24:28 STEP 1363 H=1363:00 +CPU= 16.716
  00:24:46 STEP 1364 H=1364:00 +CPU= 17.762
  00:25:15 STEP 1365 H=1365:00 +CPU= 27.771
  00:25:46 STEP 1366 H=1366:00 +CPU= 17.286
  00:26:03 STEP 1367 H=1367:00 +CPU= 16.464
  00:26:34 STEP 1368 H=1368:00 +CPU= 29.482
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001368
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001368
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179146/ICMGGhq0f+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179146/ICMSHhq0f+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179146/ICMUAhq0f+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179146/ICMGGhq0f+001356
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179146/ICMSHhq0f+001356
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179146/ICMUAhq0f+001356
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0502_1993050100_123_962_12179146_0_r863754718_56.zip
Uploading the intermediate file: upload_file_56.zip
  00:26:52 STEP 1369 H=1369:00 +CPU= 16.864
  00:27:10 STEP 1370 H=1370:00 +CPU= 17.879
  00:27:40 STEP 1371 H=1371:00 +CPU= 26.219
  00:27:56 STEP 1372 H=1372:00 +CPU= 16.460
  00:28:12 STEP 1373 H=1373:00 +CPU= 15.989
  00:28:39 STEP 1374 H=1374:00 +CPU= 26.795
  00:28:57 STEP 1375 H=1375:00 +CPU= 17.223
  00:29:15 STEP 1376 H=1376:00 +CPU= 17.990
  00:29:45 STEP 1377 H=1377:00 +CPU= 27.515
  00:30:02 STEP 1378 H=1378:00 +CPU= 16.546
  00:30:18 STEP 1379 H=1379:00 +CPU= 16.056
Abort request received from BOINC client, ending the child process
..Failed, model did not complete successfully

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:11:12 1528008 22264047 oifs_43r3_ps_0502_1993050100_123_962_12179146_0 3,369,600 32,590 0.0097


©2024 climateprediction.net