<core_client_version>7.20.2</core_client_version>
<![CDATA[
<message>
process exited with code 1 (0x1, -255)</message>
<stderr_txt>
:00 +CPU= 41.809
05:38:37 STEP 666 H= 666:00 +CPU= 67.556
05:39:19 STEP 667 H= 667:00 +CPU= 41.611
05:40:02 STEP 668 H= 668:00 +CPU= 42.954
05:41:08 STEP 669 H= 669:00 +CPU= 65.946
05:41:51 STEP 670 H= 670:00 +CPU= 42.669
05:42:35 STEP 671 H= 671:00 +CPU= 43.117
05:43:54 STEP 672 H= 672:00 +CPU= 77.917
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+000672
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+000672
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+000672
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174353/ICMGGhq0f+000648
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174353/ICMSHhq0f+000648
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174353/ICMUAhq0f+000648
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174353/ICMGGhq0f+000660
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174353/ICMSHhq0f+000660
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174353/ICMUAhq0f+000660
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0709_1988050100_123_957_12174353_0_r803898076_27.zip
Uploading the intermediate file: upload_file_27.zip
05:44:36 STEP 673 H= 673:00 +CPU= 42.167
05:45:19 STEP 674 H= 674:00 +CPU= 42.903
05:46:26 STEP 675 H= 675:00 +CPU= 66.122
05:47:09 STEP 676 H= 676:00 +CPU= 42.682
05:47:51 STEP 677 H= 677:00 +CPU= 41.630
05:48:59 STEP 678 H= 678:00 +CPU= 67.307
05:49:41 STEP 679 H= 679:00 +CPU= 41.715
05:50:24 STEP 680 H= 680:00 +CPU= 42.928
05:51:31 STEP 681 H= 681:00 +CPU= 66.376
05:52:14 STEP 682 H= 682:00 +CPU= 42.830
05:52:56 STEP 683 H= 683:00 +CPU= 41.558
05:54:10 STEP 684 H= 684:00 +CPU= 73.345
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+000684
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+000684
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+000684
05:54:52 STEP 685 H= 685:00 +CPU= 41.887
05:55:35 STEP 686 H= 686:00 +CPU= 42.825
05:56:42 STEP 687 H= 687:00 +CPU= 66.512
05:57:25 STEP 688 H= 688:00 +CPU= 43.090
05:58:08 STEP 689 H= 689:00 +CPU= 41.889
05:59:16 STEP 690 H= 690:00 +CPU= 67.565
05:59:58 STEP 691 H= 691:00 +CPU= 41.633
06:00:41 STEP 692 H= 692:00 +CPU= 42.943
06:01:48 STEP 693 H= 693:00 +CPU= 66.305
06:02:31 STEP 694 H= 694:00 +CPU= 42.894
06:03:14 STEP 695 H= 695:00 +CPU= 42.910
06:04:32 STEP 696 H= 696:00 +CPU= 77.822
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+000696
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+000696
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+000696
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174353/ICMGGhq0f+000672
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174353/ICMSHhq0f+000672
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174353/ICMUAhq0f+000672
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174353/ICMGGhq0f+000684
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174353/ICMSHhq0f+000684
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174353/ICMUAhq0f+000684
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0709_1988050100_123_957_12174353_0_r803898076_28.zip
Uploading the intermediate file: upload_file_28.zip
06:05:15 STEP 697 H= 697:00 +CPU= 41.999
06:05:58 STEP 698 H= 698:00 +CPU= 43.212
06:07:04 STEP 699 H= 699:00 +CPU= 65.868
06:07:48 STEP 700 H= 700:00 +CPU= 42.816
06:08:29 STEP 701 H= 701:00 +CPU= 41.548
06:09:37 STEP 702 H= 702:00 +CPU= 67.561
06:10:20 STEP 703 H= 703:00 +CPU= 41.718
06:11:03 STEP 704 H= 704:00 +CPU= 42.920
06:12:09 STEP 705 H= 705:00 +CPU= 65.925
06:12:53 STEP 706 H= 706:00 +CPU= 42.994
06:13:34 STEP 707 H= 707:00 +CPU= 41.592
06:14:48 STEP 708 H= 708:00 +CPU= 73.177
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+000708
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+000708
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+000708
06:15:30 STEP 709 H= 709:00 +CPU= 41.768
06:16:13 STEP 710 H= 710:00 +CPU= 42.943
06:17:20 STEP 711 H= 711:00 +CPU= 66.284
06:18:03 STEP 712 H= 712:00 +CPU= 42.755
06:18:45 STEP 713 H= 713:00 +CPU= 41.722
06:19:54 STEP 714 H= 714:00 +CPU= 68.025
06:20:36 STEP 715 H= 715:00 +CPU= 41.977
06:21:19 STEP 716 H= 716:00 +CPU= 42.645
06:22:25 STEP 717 H= 717:00 +CPU= 66.267
06:23:09 STEP 718 H= 718:00 +CPU= 42.782
06:23:52 STEP 719 H= 719:00 +CPU= 43.091
06:25:13 STEP 720 H= 720:00 +CPU= 79.290
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+000720
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+000720
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+000720
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174353/ICMGGhq0f+000696
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174353/ICMSHhq0f+000696
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174353/ICMUAhq0f+000696
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174353/ICMGGhq0f+000708
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174353/ICMSHhq0f+000708
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174353/ICMUAhq0f+000708
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0709_1988050100_123_957_12174353_0_r803898076_29.zip
Uploading the intermediate file: upload_file_29.zip
06:25:56 STEP 721 H= 721:00 +CPU= 42.249
06:26:39 STEP 722 H= 722:00 +CPU= 42.950
06:27:46 STEP 723 H= 723:00 +CPU= 66.424
06:28:29 STEP 724 H= 724:00 +CPU= 42.822
06:29:11 STEP 725 H= 725:00 +CPU= 41.751
06:30:19 STEP 726 H= 726:00 +CPU= 67.843
06:31:02 STEP 727 H= 727:00 +CPU= 41.802
06:31:45 STEP 728 H= 728:00 +CPU= 43.075
06:32:52 STEP 729 H= 729:00 +CPU= 66.590
06:33:35 STEP 730 H= 730:00 +CPU= 42.886
06:34:17 STEP 731 H= 731:00 +CPU= 41.744
06:35:32 STEP 732 H= 732:00 +CPU= 73.521
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+000732
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+000732
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+000732
06:36:14 STEP 733 H= 733:00 +CPU= 42.044
06:36:57 STEP 734 H= 734:00 +CPU= 42.927
06:38:04 STEP 735 H= 735:00 +CPU= 66.492
06:38:47 STEP 736 H= 736:00 +CPU= 42.879
06:39:29 STEP 737 H= 737:00 +CPU= 41.820
06:40:37 STEP 738 H= 738:00 +CPU= 67.465
06:41:19 STEP 739 H= 739:00 +CPU= 41.663
06:42:03 STEP 740 H= 740:00 +CPU= 42.871
06:43:09 STEP 741 H= 741:00 +CPU= 66.140
06:43:53 STEP 742 H= 742:00 +CPU= 42.741
06:44:36 STEP 743 H= 743:00 +CPU= 43.069
06:45:55 STEP 744 H= 744:00 +CPU= 77.960
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+000744
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+000744
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+000744
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174353/ICMGGhq0f+000720
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174353/ICMSHhq0f+000720
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174353/ICMUAhq0f+000720
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174353/ICMGGhq0f+000732
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174353/ICMSHhq0f+000732
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174353/ICMUAhq0f+000732
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0709_1988050100_123_957_12174353_0_r803898076_30.zip
Uploading the intermediate file: upload_file_30.zip
06:46:38 STEP 745 H= 745:00 +CPU= 42.524
06:47:21 STEP 746 H= 746:00 +CPU= 42.685
06:48:32 STEP 747 H= 747:00 +CPU= 69.929
06:49:16 STEP 748 H= 748:00 +CPU= 43.764
06:49:58 STEP 749 H= 749:00 +CPU= 41.768
06:51:08 STEP 750 H= 750:00 +CPU= 69.539
06:51:50 STEP 751 H= 751:00 +CPU= 41.509
06:52:33 STEP 752 H= 752:00 +CPU= 42.924
06:53:41 STEP 753 H= 753:00 +CPU= 66.671
06:54:24 STEP 754 H= 754:00 +CPU= 42.912
06:55:06 STEP 755 H= 755:00 +CPU= 41.781
06:56:21 STEP 756 H= 756:00 +CPU= 74.024
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+000756
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+000756
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+000756
06:57:02 STEP 757 H= 757:00 +CPU= 41.629
06:57:46 STEP 758 H= 758:00 +CPU= 43.489
06:58:53 STEP 759 H= 759:00 +CPU= 66.387
06:59:37 STEP 760 H= 760:00 +CPU= 42.909
07:00:19 STEP 761 H= 761:00 +CPU= 41.816
07:01:27 STEP 762 H= 762:00 +CPU= 68.159
07:02:09 STEP 763 H= 763:00 +CPU= 41.748
07:02:52 STEP 764 H= 764:00 +CPU= 42.697
07:04:00 STEP 765 H= 765:00 +CPU= 67.086
07:04:43 STEP 766 H= 766:00 +CPU= 42.975
07:05:27 STEP 767 H= 767:00 +CPU= 43.250
07:06:46 STEP 768 H= 768:00 +CPU= 78.378
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+000768
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+000768
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+000768
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174353/ICMGGhq0f+000744
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174353/ICMSHhq0f+000744
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174353/ICMUAhq0f+000744
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174353/ICMGGhq0f+000756
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174353/ICMSHhq0f+000756
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174353/ICMUAhq0f+000756
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0709_1988050100_123_957_12174353_0_r803898076_31.zip
Uploading the intermediate file: upload_file_31.zip
07:07:29 STEP 769 H= 769:00 +CPU= 42.496
07:08:13 STEP 770 H= 770:00 +CPU= 43.439
07:09:20 STEP 771 H= 771:00 +CPU= 66.746
07:10:03 STEP 772 H= 772:00 +CPU= 42.779
07:10:45 STEP 773 H= 773:00 +CPU= 41.713
07:11:54 STEP 774 H= 774:00 +CPU= 67.906
07:12:36 STEP 775 H= 775:00 +CPU= 41.789
07:13:19 STEP 776 H= 776:00 +CPU= 42.942
07:14:26 STEP 777 H= 777:00 +CPU= 66.839
07:15:09 STEP 778 H= 778:00 +CPU= 42.730
07:15:51 STEP 779 H= 779:00 +CPU= 41.650
07:17:06 STEP 780 H= 780:00 +CPU= 74.395
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+000780
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+000780
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+000780
07:17:48 STEP 781 H= 781:00 +CPU= 41.794
07:18:32 STEP 782 H= 782:00 +CPU= 42.969
07:19:39 STEP 783 H= 783:00 +CPU= 66.756
07:20:22 STEP 784 H= 784:00 +CPU= 42.844
07:21:04 STEP 785 H= 785:00 +CPU= 41.448
07:22:12 STEP 786 H= 786:00 +CPU= 67.949
07:22:54 STEP 787 H= 787:00 +CPU= 41.463
07:23:37 STEP 788 H= 788:00 +CPU= 42.911
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1988050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0709
(argv4) batchid: 957
(argv5) wuid: 12174353
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/1
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174353
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_2c4a5711b467be9b2f09c8cd48f183f8 to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0709_1988050100_123_957_12174353.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0709_1988050100_123_957_12174353.zip
ic_ancil_file: ic_ancil_12174353
ifsdata_file: ifsdata_12174353
climate_data_file: clim_data_12174353
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_b69ac54b4c383fc2d0e4cb362451deb9 to: /var/lib/boinc-client/slots/1/ic_ancil_12174353.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/1/ic_ancil_12174353.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_fe3bb6a416b9412672bba12a96220ffa to: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12174353.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12174353.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/1/159l_2/clim_data_12174353.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/1/159l_2/clim_data_12174353.zip
Checking for progress XML file: /var/lib/boinc-client/slots/1/progress_file_12174353.xml
Opened progress file ok : /var/lib/boinc-client/slots/1/progress_file_12174353.xml
-- Model is restarting --
Adjusting last_iter, 789, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/1/progress_file_12174353.xml
last_cpu_time: 45219
upload_file_number: 32
last_iter: 767
last_upload: 2764800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0709_1988050100_123_957_12174353_0_r803898076
The child process has been launched with process id: 3821
Executing the command: /var/lib/boinc-client/slots/1/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:Shaynovo:1:1:3821:3821] [20230203:211112:1675483872:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:Shaynovo:1:1:3821:3821] [20230203:211112:1675483872:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:Shaynovo:1:1:3821:3821] [20230203:211112:1675483872:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:Shaynovo:1:1:3821:3821] [20230203:211112:1675483872:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:Shaynovo:1:1:3821:3821] [20230203:211112:1675483872:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:Shaynovo:1:1:3821:3821] [20230203:211112:1675483872:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:Shaynovo:1:1:3821:3821] [20230203:211112:1675483872:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:Shaynovo:1:1:3821:3821] [20230203:211112:1675483872:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:Shaynovo:1:1:3821:3821] [20230203:211112:1675483872:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:Shaynovo:1:1:3821:3821] [20230203:211112:1675483872:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:Shaynovo:1:1:3821:3821] [20230203:211112:1675483872:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:Shaynovo:1:1:3821:3821] [20230203:211112:1675483872:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:Shaynovo:1:1:3821:3821] [20230203:211112:1675483872:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:Shaynovo:1:1:3821:3821] [20230203:211112:1675483872:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:Shaynovo:1:1:3821:3821] [20230203:211112:1675483872:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:Shaynovo:1:1:3821:3821] [20230203:211112:1675483872:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Shaynovo:1:1:3821:3821] [20230203:211112:1675483872:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Shaynovo:1:1:3821:3821] [20230203:211112:1675483872:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Shaynovo:1:1:3821:3821] [20230203:211112:1675483872:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:Shaynovo:1:1:3821:3821] [20230203:211112:1675483872:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:Shaynovo:1:1:3821:3821] [20230203:211112:1675483872:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Shaynovo:1:1:3821:3821] [20230203:211112:1675483872:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Shaynovo:1:1:3821:3821] [20230203:211112:1675483872:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:Shaynovo:1:1:3821:3821] [20230203:211112:1675483872:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Shaynovo:1:1:3821:3821] [20230203:211112:1675483872:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Shaynovo:1:1:3821:3821] [20230203:211112:1675483872:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Shaynovo:1:1:3821:3821] [20230203:211112:1675483872:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:Shaynovo:1:1:3821:3821] [20230203:211112:1675483872:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:Shaynovo:1:1:3821:3821] [20230203:211112:1675483872:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Shaynovo:1:1:3821:3821] [20230203:211112:1675483872:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
21:14:37 STEP 768 H= 768:00 +CPU= 88.368
21:15:28 STEP 769 H= 769:00 +CPU= 47.193
21:16:20 STEP 770 H= 770:00 +CPU= 49.599
21:17:49 STEP 771 H= 771:00 +CPU= 82.383
21:18:45 STEP 772 H= 772:00 +CPU= 53.045
21:19:41 STEP 773 H= 773:00 +CPU= 51.616
21:21:02 STEP 774 H= 774:00 +CPU= 77.692
21:21:51 STEP 775 H= 775:00 +CPU= 47.110
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1988050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0709
(argv4) batchid: 957
(argv5) wuid: 12174353
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/1
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174353
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_2c4a5711b467be9b2f09c8cd48f183f8 to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0709_1988050100_123_957_12174353.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0709_1988050100_123_957_12174353.zip
ic_ancil_file: ic_ancil_12174353
ifsdata_file: ifsdata_12174353
climate_data_file: clim_data_12174353
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_b69ac54b4c383fc2d0e4cb362451deb9 to: /var/lib/boinc-client/slots/1/ic_ancil_12174353.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/1/ic_ancil_12174353.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_fe3bb6a416b9412672bba12a96220ffa to: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12174353.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12174353.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/1/159l_2/clim_data_12174353.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/1/159l_2/clim_data_12174353.zip
Checking for progress XML file: /var/lib/boinc-client/slots/1/progress_file_12174353.xml
Opened progress file ok : /var/lib/boinc-client/slots/1/progress_file_12174353.xml
-- Model is restarting --
Adjusting last_iter, 775, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/1/progress_file_12174353.xml
last_cpu_time: 45716
upload_file_number: 32
last_iter: 767
last_upload: 2764800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0709_1988050100_123_957_12174353_0_r803898076
The child process has been launched with process id: 10080
Executing the command: /var/lib/boinc-client/slots/1/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:Shaynovo:1:1:10080:10080] [20230203:212807:1675484887:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:Shaynovo:1:1:10080:10080] [20230203:212807:1675484887:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:Shaynovo:1:1:10080:10080] [20230203:212807:1675484887:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:Shaynovo:1:1:10080:10080] [20230203:212807:1675484887:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:Shaynovo:1:1:10080:10080] [20230203:212807:1675484887:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:Shaynovo:1:1:10080:10080] [20230203:212807:1675484887:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:Shaynovo:1:1:10080:10080] [20230203:212807:1675484887:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:Shaynovo:1:1:10080:10080] [20230203:212807:1675484887:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:Shaynovo:1:1:10080:10080] [20230203:212807:1675484887:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:Shaynovo:1:1:10080:10080] [20230203:212807:1675484887:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:Shaynovo:1:1:10080:10080] [20230203:212807:1675484887:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:Shaynovo:1:1:10080:10080] [20230203:212807:1675484887:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:Shaynovo:1:1:10080:10080] [20230203:212807:1675484887:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:Shaynovo:1:1:10080:10080] [20230203:212807:1675484887:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:Shaynovo:1:1:10080:10080] [20230203:212807:1675484887:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:Shaynovo:1:1:10080:10080] [20230203:212807:1675484887:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Shaynovo:1:1:10080:10080] [20230203:212807:1675484887:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Shaynovo:1:1:10080:10080] [20230203:212807:1675484887:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Shaynovo:1:1:10080:10080] [20230203:212807:1675484887:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:Shaynovo:1:1:10080:10080] [20230203:212807:1675484887:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:Shaynovo:1:1:10080:10080] [20230203:212807:1675484887:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Shaynovo:1:1:10080:10080] [20230203:212807:1675484887:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Shaynovo:1:1:10080:10080] [20230203:212807:1675484887:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:Shaynovo:1:1:10080:10080] [20230203:212807:1675484887:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Shaynovo:1:1:10080:10080] [20230203:212807:1675484887:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Shaynovo:1:1:10080:10080] [20230203:212807:1675484887:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Shaynovo:1:1:10080:10080] [20230203:212807:1675484887:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:Shaynovo:1:1:10080:10080] [20230203:212807:1675484887:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:Shaynovo:1:1:10080:10080] [20230203:212807:1675484887:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Shaynovo:1:1:10080:10080] [20230203:212807:1675484887:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
21:33:48 STEP 768 H= 768:00 +CPU=101.445
21:34:44 STEP 769 H= 769:00 +CPU= 50.981
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1988050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0709
(argv4) batchid: 957
(argv5) wuid: 12174353
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/1
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174353
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_2c4a5711b467be9b2f09c8cd48f183f8 to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0709_1988050100_123_957_12174353.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0709_1988050100_123_957_12174353.zip
ic_ancil_file: ic_ancil_12174353
ifsdata_file: ifsdata_12174353
climate_data_file: clim_data_12174353
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_b69ac54b4c383fc2d0e4cb362451deb9 to: /var/lib/boinc-client/slots/1/ic_ancil_12174353.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/1/ic_ancil_12174353.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_fe3bb6a416b9412672bba12a96220ffa to: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12174353.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12174353.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/1/159l_2/clim_data_12174353.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/1/159l_2/clim_data_12174353.zip
Checking for progress XML file: /var/lib/boinc-client/slots/1/progress_file_12174353.xml
Opened progress file ok : /var/lib/boinc-client/slots/1/progress_file_12174353.xml
-- Model is restarting --
Adjusting last_iter, 770, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/1/progress_file_12174353.xml
last_cpu_time: 45899
upload_file_number: 32
last_iter: 767
last_upload: 2764800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0709_1988050100_123_957_12174353_0_r803898076
The child process has been launched with process id: 12652
Executing the command: /var/lib/boinc-client/slots/1/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:Shaynovo:1:1:12652:12652] [20230203:214017:1675485617:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:Shaynovo:1:1:12652:12652] [20230203:214017:1675485617:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:Shaynovo:1:1:12652:12652] [20230203:214017:1675485617:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:Shaynovo:1:1:12652:12652] [20230203:214017:1675485617:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:Shaynovo:1:1:12652:12652] [20230203:214017:1675485617:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:Shaynovo:1:1:12652:12652] [20230203:214017:1675485617:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:Shaynovo:1:1:12652:12652] [20230203:214017:1675485617:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:Shaynovo:1:1:12652:12652] [20230203:214017:1675485617:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:Shaynovo:1:1:12652:12652] [20230203:214017:1675485617:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:Shaynovo:1:1:12652:12652] [20230203:214017:1675485617:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:Shaynovo:1:1:12652:12652] [20230203:214017:1675485617:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:Shaynovo:1:1:12652:12652] [20230203:214017:1675485617:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:Shaynovo:1:1:12652:12652] [20230203:214017:1675485617:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:Shaynovo:1:1:12652:12652] [20230203:214017:1675485617:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:Shaynovo:1:1:12652:12652] [20230203:214017:1675485617:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:Shaynovo:1:1:12652:12652] [20230203:214017:1675485617:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Shaynovo:1:1:12652:12652] [20230203:214017:1675485617:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Shaynovo:1:1:12652:12652] [20230203:214017:1675485617:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Shaynovo:1:1:12652:12652] [20230203:214017:1675485617:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:Shaynovo:1:1:12652:12652] [20230203:214017:1675485617:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:Shaynovo:1:1:12652:12652] [20230203:214017:1675485617:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Shaynovo:1:1:12652:12652] [20230203:214017:1675485617:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Shaynovo:1:1:12652:12652] [20230203:214017:1675485617:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:Shaynovo:1:1:12652:12652] [20230203:214017:1675485617:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Shaynovo:1:1:12652:12652] [20230203:214017:1675485617:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Shaynovo:1:1:12652:12652] [20230203:214017:1675485617:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Shaynovo:1:1:12652:12652] [20230203:214017:1675485617:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:Shaynovo:1:1:12652:12652] [20230203:214017:1675485617:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:Shaynovo:1:1:12652:12652] [20230203:214017:1675485617:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Shaynovo:1:1:12652:12652] [20230203:214017:1675485617:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
21:46:33 STEP 768 H= 768:00 +CPU= 86.179
21:47:26 STEP 769 H= 769:00 +CPU= 50.444
21:48:30 STEP 770 H= 770:00 +CPU= 56.912
21:50:04 STEP 771 H= 771:00 +CPU= 85.648
21:51:01 STEP 772 H= 772:00 +CPU= 53.519
21:51:58 STEP 773 H= 773:00 +CPU= 52.139
21:53:38 STEP 774 H= 774:00 +CPU= 86.410
21:54:28 STEP 775 H= 775:00 +CPU= 47.321
21:55:20 STEP 776 H= 776:00 +CPU= 49.770
21:56:41 STEP 777 H= 777:00 +CPU= 78.374
21:57:35 STEP 778 H= 778:00 +CPU= 51.193
21:58:26 STEP 779 H= 779:00 +CPU= 49.206
21:59:56 STEP 780 H= 780:00 +CPU= 86.062
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+000780
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+000780
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+000780
22:01:00 STEP 781 H= 781:00 +CPU= 55.133
22:02:10 STEP 782 H= 782:00 +CPU= 58.902
22:04:09 STEP 783 H= 783:00 +CPU= 95.491
22:05:44 STEP 784 H= 784:00 +CPU= 69.703
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
22:07:31 STEP 785 H= 785:00 +CPU= 68.151
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
22:10:19 STEP 786 H= 786:00 +CPU=108.343
22:11:47 STEP 787 H= 787:00 +CPU= 66.332
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
22:14:00 STEP 788 H= 788:00 +CPU= 69.271
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
22:16:56 STEP 789 H= 789:00 +CPU=103.802
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
22:18:32 STEP 790 H= 790:00 +CPU= 65.144
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
22:21:08 STEP 791 H= 791:00 +CPU= 70.410
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1988050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0709
(argv4) batchid: 957
(argv5) wuid: 12174353
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/1
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174353
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_2c4a5711b467be9b2f09c8cd48f183f8 to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0709_1988050100_123_957_12174353.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0709_1988050100_123_957_12174353.zip
ic_ancil_file: ic_ancil_12174353
ifsdata_file: ifsdata_12174353
climate_data_file: clim_data_12174353
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_b69ac54b4c383fc2d0e4cb362451deb9 to: /var/lib/boinc-client/slots/1/ic_ancil_12174353.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/1/ic_ancil_12174353.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_fe3bb6a416b9412672bba12a96220ffa to: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12174353.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12174353.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/1/159l_2/clim_data_12174353.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/1/159l_2/clim_data_12174353.zip
Checking for progress XML file: /var/lib/boinc-client/slots/1/progress_file_12174353.xml
Opened progress file ok : /var/lib/boinc-client/slots/1/progress_file_12174353.xml
-- Model is restarting --
Adjusting last_iter, 792, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/1/progress_file_12174353.xml
last_cpu_time: 47568
upload_file_number: 32
last_iter: 791
last_upload: 2764800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0709_1988050100_123_957_12174353_0_r803898076
The child process has been launched with process id: 13899
Executing the command: /var/lib/boinc-client/slots/1/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:Shaynovo:1:1:13899:13899] [20230203:222415:1675488255:0.013] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:Shaynovo:1:1:13899:13899] [20230203:222415:1675488255:0.013] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:Shaynovo:1:1:13899:13899] [20230203:222415:1675488255:0.013] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:Shaynovo:1:1:13899:13899] [20230203:222415:1675488255:0.013] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:Shaynovo:1:1:13899:13899] [20230203:222415:1675488255:0.013] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:Shaynovo:1:1:13899:13899] [20230203:222415:1675488255:0.013] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:Shaynovo:1:1:13899:13899] [20230203:222415:1675488255:0.013] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:Shaynovo:1:1:13899:13899] [20230203:222415:1675488255:0.013] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:Shaynovo:1:1:13899:13899] [20230203:222415:1675488255:0.013] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:Shaynovo:1:1:13899:13899] [20230203:222415:1675488255:0.013] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:Shaynovo:1:1:13899:13899] [20230203:222415:1675488255:0.013] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:Shaynovo:1:1:13899:13899] [20230203:222415:1675488255:0.013] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:Shaynovo:1:1:13899:13899] [20230203:222415:1675488255:0.013] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:Shaynovo:1:1:13899:13899] [20230203:222415:1675488255:0.013] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:Shaynovo:1:1:13899:13899] [20230203:222415:1675488255:0.014] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:Shaynovo:1:1:13899:13899] [20230203:222415:1675488255:0.014] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Shaynovo:1:1:13899:13899] [20230203:222415:1675488255:0.014] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Shaynovo:1:1:13899:13899] [20230203:222415:1675488255:0.014] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Shaynovo:1:1:13899:13899] [20230203:222415:1675488255:0.014] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:Shaynovo:1:1:13899:13899] [20230203:222415:1675488255:0.014] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:Shaynovo:1:1:13899:13899] [20230203:222415:1675488255:0.014] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Shaynovo:1:1:13899:13899] [20230203:222415:1675488255:0.014] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Shaynovo:1:1:13899:13899] [20230203:222415:1675488255:0.014] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:Shaynovo:1:1:13899:13899] [20230203:222415:1675488255:0.014] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Shaynovo:1:1:13899:13899] [20230203:222415:1675488255:0.014] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Shaynovo:1:1:13899:13899] [20230203:222415:1675488255:0.014] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Shaynovo:1:1:13899:13899] [20230203:222415:1675488255:0.014] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:Shaynovo:1:1:13899:13899] [20230203:222415:1675488255:0.014] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:Shaynovo:1:1:13899:13899] [20230203:222415:1675488255:0.014] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Shaynovo:1:1:13899:13899] [20230203:222415:1675488255:0.014] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174353/ICMGGhq0f+000768
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174353/ICMSHhq0f+000768
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174353/ICMUAhq0f+000768
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174353/ICMGGhq0f+000780
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174353/ICMSHhq0f+000780
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174353/ICMUAhq0f+000780
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0709_1988050100_123_957_12174353_0_r803898076_32.zip
Uploading the intermediate file: upload_file_32.zip
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
ABORT! 1 !! *** WAVE MODEL HAS ABORTED
[EC_DRHOOK:Shaynovo:1:1:13899:13899] [20230203:223041:1675488641:385.953] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 1721 MB (maxheap), 3519 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:Shaynovo:1:1:13899:13899] [20230203:223041:1675488641:385.980] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : MASTER
[EC_DRHOOK:Shaynovo:1:1:13899:13899] [20230203:223041:1675488641:385.983] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT0
[EC_DRHOOK:Shaynovo:1:1:13899:13899] [20230203:223041:1675488641:385.983] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT1
[EC_DRHOOK:Shaynovo:1:1:13899:13899] [20230203:223041:1675488641:385.983] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT2
[EC_DRHOOK:Shaynovo:1:1:13899:13899] [20230203:223041:1675488641:385.983] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT3
[EC_DRHOOK:Shaynovo:1:1:13899:13899] [20230203:223041:1675488641:385.983] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT4
[EC_DRHOOK:Shaynovo:1:1:13899:13899] [20230203:223041:1675488641:385.983] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : WVCOUPLE
[EC_DRHOOK:Shaynovo:1:1:13899:13899] [20230203:223041:1675488641:385.983] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : WAVEMDL
[EC_DRHOOK:Shaynovo:1:1:13899:13899] [20230203:223041:1675488641:385.983] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : INITMDL
[EC_DRHOOK:Shaynovo:1:1:13899:13899] [20230203:223041:1675488641:385.983] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : USERIN
SDL_TRACEBACK: Calling INTEL_TRBK, THRD = 1
Process 1 thread 1 calling linux_trbk from intel_trbk()
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:240] Backtrace(s) for program '/var/lib/boinc-client/slots/1/oifs_43r3_model.exe' : sigcontextptr=0x7fff1deb87a0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:277] Backtrace (size = 19) with addr2line-cmd
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:302] /usr/bin/addr2line -fs -e '/var/lib/boinc-client/slots/1/oifs_43r3_model.exe' 0x839273 0x839879 0x8c2957 0x81d89e 0x15f8aca 0x15d51c8 0x166b07f 0x15d7ff0 0x8711e0 0x7e0165 0x55e4ad 0x5471c9 0x546412 0x545e28 0x40708d 0x40220f 0x4021a2 0x1dca390 0x40206e
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [0]: [0x839273] : ??() at ??:0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [1]: [0x839879] : ??() at ??:0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [2]: [0x8c2957] : ??() at ??:0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [3]: [0x81d89e] : ??() at ??:0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [4]: [0x15f8aca] : ??() at ??:0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [5]: [0x15d51c8] : ??() at ??:0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [6]: [0x166b07f] : ??() at ??:0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [7]: [0x15d7ff0] : ??() at ??:0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [8]: [0x8711e0] : ??() at ??:0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [9]: [0x7e0165] : ??() at ??:0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [10]: [0x55e4ad] : ??() at ??:0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [11]: [0x5471c9] : ??() at ??:0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [12]: [0x546412] : ??() at ??:0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [13]: [0x545e28] : ??() at ??:0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [14]: [0x40708d] : ??() at ??:0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [15]: [0x40220f] : ??() at ??:0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [16]: [0x4021a2] : ??() at ??:0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [17]: [0x1dca390] : ??() at ??:0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [18]: [0x40206e] : ??() at ??:0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:370] End of backtrace(s)
SDL_TRACEBACK: Done INTEL_TRBK, THRD = 1
[EC_DRHOOK:Shaynovo:1:1:13899:13899] [20230203:223044:1675488644:389.265] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1538] Received signal#6 (SIGABRT) :: 1721MB (heap), 3519MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:Shaynovo:1:1:13899:13899] [20230203:223044:1675488644:389.265] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:Shaynovo:1:1:13899:13899] [20230203:223044:1675488644:389.265] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f3c0 (old at (nil))
[EC_DRHOOK:Shaynovo:1:1:13899:13899] [20230203:223044:1675488644:389.265] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1617] Signal#6 was caused by unrecognized si_code [memaddr=0x800000364b], nsigs = 1
[EC_DRHOOK:Shaynovo:1:1:13899:13899] [20230203:223044:1675488644:389.265] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#6, nsigs = 1
[EC_DRHOOK:Shaynovo:1:1:13899:13899] [20230203:223045:1675488645:390.266] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#6, nsigs = 1
[EC_DRHOOK:Shaynovo:1:1:13899:13899] [20230203:223045:1675488645:390.266] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1ce8bb0 for signal#6, nsigs = 1
forrtl: error (76): Abort trap signal
Image PC Routine Line Source
oifs_43r3_model.e 0000000001CE916B Unknown Unknown Unknown
oifs_43r3_model.e 000000000081FFF1 Unknown Unknown Unknown
oifs_43r3_model.e 0000000001DC9090 Unknown Unknown Unknown
oifs_43r3_model.e 0000000001DC903B Unknown Unknown Unknown
oifs_43r3_model.e 00000000008C2880 sdl_mod_mp_sdl_sr 100 sdl_mod.F90
oifs_43r3_model.e 000000000081D8AF abor1_ 41 abor1.F90
oifs_43r3_model.e 00000000015F8ACA mpabort_ 50 mpabort.F
oifs_43r3_model.e 00000000015D51C8 abort1_ 44 abort1.F
oifs_43r3_model.e 000000000166B07F userin_ 553 userin.F
oifs_43r3_model.e 00000000015D7FF0 initmdl_ 452 initmdl.F
oifs_43r3_model.e 00000000008711E0 wavemdl_ 393 wavemdl.F
oifs_43r3_model.e 00000000007E0165 wvcouple_ 260 wvcouple.F90
oifs_43r3_model.e 000000000055E4AD cnt4_ 1283 cnt4.F90
oifs_43r3_model.e 00000000005471C9 cnt3_ 267 cnt3.F90
oifs_43r3_model.e 0000000000546412 cnt2_ 88 cnt2.F90
oifs_43r3_model.e 0000000000545E28 cnt1_ 92 cnt1.F90
oifs_43r3_model.e 000000000040708D cnt0_ 146 cnt0.F90
oifs_43r3_model.e 000000000040220F MAIN__ 96 master.F90
oifs_43r3_model.e 00000000004021A2 Unknown Unknown Unknown
oifs_43r3_model.e 0000000001DCA390 Unknown Unknown Unknown
oifs_43r3_model.e 000000000040206E Unknown Unknown Unknown
The child process terminated with status: 0
..Failed, model did not complete successfully
</stderr_txt>
]]>
