climateprediction.net home page
Task 22298577

Task 22298577

Name oifs_43r3_ps_0513_2017050100_123_986_12203157_2
Workunit 12203157
Created 16 Jan 2023, 17:24:42 UTC
Sent 16 Jan 2023, 17:43:37 UTC
Report deadline 17 Mar 2023, 17:43:37 UTC
Received 17 Jan 2023, 17:34:36 UTC
Server state Over
Outcome Success
Client state Done
Exit status 0 (0x00000000)
Computer ID 1532237
Run time 18 hours 47 min 14 sec
CPU time 15 hours 19 min 59 sec
Validate state Valid
Credit 14,873.04
Device peak FLOPS 7.29 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,415.29 MB
Peak swap size 4,978.08 MB
Peak disk usage 5,539.95 MB
Stderr
<core_client_version>7.16.6</core_client_version>
<![CDATA[
<stderr_txt>
1673966620:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:1742909:1742909] [20230117:074340:1673966620:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:1742909:1742909] [20230117:074340:1673966620:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:1742909:1742909] [20230117:074340:1673966620:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:1742909:1742909] [20230117:074340:1673966620:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:1742909:1742909] [20230117:074340:1673966620:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:1742909:1742909] [20230117:074340:1673966620:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:1742909:1742909] [20230117:074340:1673966620:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:1742909:1742909] [20230117:074340:1673966620:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:1742909:1742909] [20230117:074340:1673966620:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:1742909:1742909] [20230117:074340:1673966620:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:1742909:1742909] [20230117:074340:1673966620:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  07:44:15 STEP 2568 H=2568:00 +CPU= 32.661
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMGGhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMSHhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMUAhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMGGhq0f+002556
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMSHhq0f+002556
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMUAhq0f+002556
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0513_2017050100_123_986_12203157_2_r835230155_106.zip
Uploading the intermediate file: upload_file_106.zip
  07:44:35 STEP 2569 H=2569:00 +CPU= 19.622
  07:44:54 STEP 2570 H=2570:00 +CPU= 18.464
  07:45:21 STEP 2571 H=2571:00 +CPU= 26.749
  07:45:38 STEP 2572 H=2572:00 +CPU= 17.323
  07:45:55 STEP 2573 H=2573:00 +CPU= 16.447
  07:46:25 STEP 2574 H=2574:00 +CPU= 30.159
  07:46:44 STEP 2575 H=2575:00 +CPU= 18.508
  07:47:02 STEP 2576 H=2576:00 +CPU= 18.250
  07:47:29 STEP 2577 H=2577:00 +CPU= 26.852
  07:47:48 STEP 2578 H=2578:00 +CPU= 18.479
  07:48:06 STEP 2579 H=2579:00 +CPU= 17.971
  07:48:37 STEP 2580 H=2580:00 +CPU= 30.194
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002580
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002580
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002580
  07:48:54 STEP 2581 H=2581:00 +CPU= 17.353
  07:49:14 STEP 2582 H=2582:00 +CPU= 19.371
  07:49:41 STEP 2583 H=2583:00 +CPU= 26.108
  07:49:57 STEP 2584 H=2584:00 +CPU= 16.149
  07:50:15 STEP 2585 H=2585:00 +CPU= 17.634
  07:50:46 STEP 2586 H=2586:00 +CPU= 30.228
  07:51:04 STEP 2587 H=2587:00 +CPU= 18.334
  07:51:22 STEP 2588 H=2588:00 +CPU= 17.529
  07:51:49 STEP 2589 H=2589:00 +CPU= 26.864
  07:52:07 STEP 2590 H=2590:00 +CPU= 18.008
  07:52:27 STEP 2591 H=2591:00 +CPU= 19.510
  07:53:00 STEP 2592 H=2592:00 +CPU= 31.981
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002592
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002592
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMGGhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMGGhq0f+002580
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMSHhq0f+002580
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMUAhq0f+002580
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0513_2017050100_123_986_12203157_2_r835230155_107.zip
Uploading the intermediate file: upload_file_107.zip
  07:53:18 STEP 2593 H=2593:00 +CPU= 18.025
  07:53:38 STEP 2594 H=2594:00 +CPU= 20.070
  07:54:07 STEP 2595 H=2595:00 +CPU= 27.935
  07:54:26 STEP 2596 H=2596:00 +CPU= 18.829
  07:54:44 STEP 2597 H=2597:00 +CPU= 18.622
  07:55:13 STEP 2598 H=2598:00 +CPU= 28.695
  07:55:30 STEP 2599 H=2599:00 +CPU= 15.972
  07:55:49 STEP 2600 H=2600:00 +CPU= 19.422
  07:56:20 STEP 2601 H=2601:00 +CPU= 30.021
  07:56:39 STEP 2602 H=2602:00 +CPU= 18.999
  07:56:56 STEP 2603 H=2603:00 +CPU= 17.501
  07:57:26 STEP 2604 H=2604:00 +CPU= 29.555
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002604
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002604
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002604
  07:57:43 STEP 2605 H=2605:00 +CPU= 16.636
  07:58:01 STEP 2606 H=2606:00 +CPU= 17.173
  07:58:30 STEP 2607 H=2607:00 +CPU= 28.439
  07:58:49 STEP 2608 H=2608:00 +CPU= 19.272
  07:59:07 STEP 2609 H=2609:00 +CPU= 17.378
  07:59:33 STEP 2610 H=2610:00 +CPU= 25.434
  07:59:51 STEP 2611 H=2611:00 +CPU= 18.430
  08:00:12 STEP 2612 H=2612:00 +CPU= 20.838
  08:00:41 STEP 2613 H=2613:00 +CPU= 28.677
  08:00:59 STEP 2614 H=2614:00 +CPU= 17.155
  08:01:17 STEP 2615 H=2615:00 +CPU= 18.363
  08:01:50 STEP 2616 H=2616:00 +CPU= 32.389
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002616
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002616
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMGGhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMSHhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMUAhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMGGhq0f+002604
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMSHhq0f+002604
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMUAhq0f+002604
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0513_2017050100_123_986_12203157_2_r835230155_108.zip
Uploading the intermediate file: upload_file_108.zip
  08:02:08 STEP 2617 H=2617:00 +CPU= 17.828
  08:02:28 STEP 2618 H=2618:00 +CPU= 18.962
  08:02:57 STEP 2619 H=2619:00 +CPU= 28.578
  08:03:17 STEP 2620 H=2620:00 +CPU= 19.902
  08:03:34 STEP 2621 H=2621:00 +CPU= 17.376
  08:04:02 STEP 2622 H=2622:00 +CPU= 27.133
  08:04:21 STEP 2623 H=2623:00 +CPU= 19.166
  08:04:41 STEP 2624 H=2624:00 +CPU= 20.036
  08:05:10 STEP 2625 H=2625:00 +CPU= 28.828
  08:05:28 STEP 2626 H=2626:00 +CPU= 17.828
  08:05:46 STEP 2627 H=2627:00 +CPU= 17.820
  08:06:18 STEP 2628 H=2628:00 +CPU= 31.602
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002628
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002628
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002628
  08:06:38 STEP 2629 H=2629:00 +CPU= 19.552
  08:06:58 STEP 2630 H=2630:00 +CPU= 19.742
  08:07:28 STEP 2631 H=2631:00 +CPU= 28.916
  08:07:48 STEP 2632 H=2632:00 +CPU= 19.496
  08:08:06 STEP 2633 H=2633:00 +CPU= 18.572
  08:08:33 STEP 2634 H=2634:00 +CPU= 26.509
  08:08:52 STEP 2635 H=2635:00 +CPU= 18.495
  08:09:12 STEP 2636 H=2636:00 +CPU= 20.244
  08:09:42 STEP 2637 H=2637:00 +CPU= 29.478
  08:10:00 STEP 2638 H=2638:00 +CPU= 17.842
  08:10:20 STEP 2639 H=2639:00 +CPU= 19.555
  08:10:52 STEP 2640 H=2640:00 +CPU= 31.711
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002640
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002640
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMGGhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMSHhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMUAhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMGGhq0f+002628
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMSHhq0f+002628
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMUAhq0f+002628
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0513_2017050100_123_986_12203157_2_r835230155_109.zip
Uploading the intermediate file: upload_file_109.zip
  08:11:11 STEP 2641 H=2641:00 +CPU= 18.327
  08:11:31 STEP 2642 H=2642:00 +CPU= 19.644
  08:11:59 STEP 2643 H=2643:00 +CPU= 27.682
  08:12:17 STEP 2644 H=2644:00 +CPU= 17.626
  08:12:35 STEP 2645 H=2645:00 +CPU= 17.927
  08:13:03 STEP 2646 H=2646:00 +CPU= 27.919
  08:13:20 STEP 2647 H=2647:00 +CPU= 17.152
  08:13:39 STEP 2648 H=2648:00 +CPU= 18.652
  08:14:06 STEP 2649 H=2649:00 +CPU= 26.323
  08:14:24 STEP 2650 H=2650:00 +CPU= 17.743
  08:14:44 STEP 2651 H=2651:00 +CPU= 19.575
  08:15:16 STEP 2652 H=2652:00 +CPU= 32.161
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002652
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002652
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002652
  08:15:34 STEP 2653 H=2653:00 +CPU= 17.185
  08:15:51 STEP 2654 H=2654:00 +CPU= 17.308
  08:16:20 STEP 2655 H=2655:00 +CPU= 28.702
  08:16:40 STEP 2656 H=2656:00 +CPU= 19.086
  08:16:59 STEP 2657 H=2657:00 +CPU= 18.737
  08:17:26 STEP 2658 H=2658:00 +CPU= 27.244
  08:17:45 STEP 2659 H=2659:00 +CPU= 18.047
  08:18:04 STEP 2660 H=2660:00 +CPU= 19.485
  08:18:32 STEP 2661 H=2661:00 +CPU= 27.564
  08:18:50 STEP 2662 H=2662:00 +CPU= 17.153
  08:19:10 STEP 2663 H=2663:00 +CPU= 19.688
  08:19:44 STEP 2664 H=2664:00 +CPU= 33.231
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002664
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002664
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMGGhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMSHhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMUAhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMGGhq0f+002652
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMSHhq0f+002652
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMUAhq0f+002652
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0513_2017050100_123_986_12203157_2_r835230155_110.zip
Uploading the intermediate file: upload_file_110.zip
  08:20:00 STEP 2665 H=2665:00 +CPU= 16.195
  08:20:18 STEP 2666 H=2666:00 +CPU= 18.007
  08:20:47 STEP 2667 H=2667:00 +CPU= 28.379
  08:21:06 STEP 2668 H=2668:00 +CPU= 19.351
  08:21:26 STEP 2669 H=2669:00 +CPU= 19.302
  08:21:56 STEP 2670 H=2670:00 +CPU= 30.114
  08:22:15 STEP 2671 H=2671:00 +CPU= 18.674
  08:22:34 STEP 2672 H=2672:00 +CPU= 18.510
  08:23:00 STEP 2673 H=2673:00 +CPU= 25.637
  08:23:18 STEP 2674 H=2674:00 +CPU= 18.358
  08:23:39 STEP 2675 H=2675:00 +CPU= 20.072
  08:24:10 STEP 2676 H=2676:00 +CPU= 31.134
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002676
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002676
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002676
  08:24:26 STEP 2677 H=2677:00 +CPU= 15.493
  08:24:45 STEP 2678 H=2678:00 +CPU= 18.783
  08:25:17 STEP 2679 H=2679:00 +CPU= 30.499
  08:25:36 STEP 2680 H=2680:00 +CPU= 19.604
  08:25:54 STEP 2681 H=2681:00 +CPU= 17.956
  08:26:23 STEP 2682 H=2682:00 +CPU= 28.031
  08:26:41 STEP 2683 H=2683:00 +CPU= 18.037
  08:26:59 STEP 2684 H=2684:00 +CPU= 17.628
  08:27:29 STEP 2685 H=2685:00 +CPU= 29.574
  08:27:49 STEP 2686 H=2686:00 +CPU= 19.719
  08:28:10 STEP 2687 H=2687:00 +CPU= 20.725
  08:28:45 STEP 2688 H=2688:00 +CPU= 33.875
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002688
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002688
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMGGhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMSHhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMUAhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMGGhq0f+002676
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMSHhq0f+002676
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMUAhq0f+002676
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0513_2017050100_123_986_12203157_2_r835230155_111.zip
Uploading the intermediate file: upload_file_111.zip
  08:29:02 STEP 2689 H=2689:00 +CPU= 17.580
  08:29:21 STEP 2690 H=2690:00 +CPU= 18.635
  08:29:50 STEP 2691 H=2691:00 +CPU= 28.177
  08:30:10 STEP 2692 H=2692:00 +CPU= 19.595
  08:30:30 STEP 2693 H=2693:00 +CPU= 20.086
  08:31:03 STEP 2694 H=2694:00 +CPU= 32.115
  08:31:22 STEP 2695 H=2695:00 +CPU= 19.158
  08:31:43 STEP 2696 H=2696:00 +CPU= 20.544
  08:32:14 STEP 2697 H=2697:00 +CPU= 30.701
  08:32:35 STEP 2698 H=2698:00 +CPU= 20.136
  08:32:55 STEP 2699 H=2699:00 +CPU= 19.866
  08:33:28 STEP 2700 H=2700:00 +CPU= 33.324
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002700
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002700
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002700
  08:33:47 STEP 2701 H=2701:00 +CPU= 18.527
  08:34:08 STEP 2702 H=2702:00 +CPU= 20.575
  08:34:37 STEP 2703 H=2703:00 +CPU= 28.350
  08:34:55 STEP 2704 H=2704:00 +CPU= 17.800
  08:35:15 STEP 2705 H=2705:00 +CPU= 19.598
  08:35:43 STEP 2706 H=2706:00 +CPU= 27.672
  08:36:01 STEP 2707 H=2707:00 +CPU= 17.757
  08:36:20 STEP 2708 H=2708:00 +CPU= 18.710
  08:36:49 STEP 2709 H=2709:00 +CPU= 28.596
  08:37:09 STEP 2710 H=2710:00 +CPU= 20.037
  08:37:29 STEP 2711 H=2711:00 +CPU= 20.197
  08:38:07 STEP 2712 H=2712:00 +CPU= 36.379
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002712
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002712
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMGGhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMSHhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMUAhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMGGhq0f+002700
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMSHhq0f+002700
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMUAhq0f+002700
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0513_2017050100_123_986_12203157_2_r835230155_112.zip
Uploading the intermediate file: upload_file_112.zip
  08:38:26 STEP 2713 H=2713:00 +CPU= 18.840
  08:38:47 STEP 2714 H=2714:00 +CPU= 20.503
  08:39:14 STEP 2715 H=2715:00 +CPU= 27.223
  08:39:34 STEP 2716 H=2716:00 +CPU= 19.349
  08:39:52 STEP 2717 H=2717:00 +CPU= 17.330
  08:40:21 STEP 2718 H=2718:00 +CPU= 28.895
  08:40:40 STEP 2719 H=2719:00 +CPU= 19.013
  08:41:01 STEP 2720 H=2720:00 +CPU= 20.197
  08:41:29 STEP 2721 H=2721:00 +CPU= 27.924
  08:41:50 STEP 2722 H=2722:00 +CPU= 20.384
  08:42:10 STEP 2723 H=2723:00 +CPU= 19.945
  08:42:43 STEP 2724 H=2724:00 +CPU= 32.365
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002724
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002724
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002724
  08:43:01 STEP 2725 H=2725:00 +CPU= 17.888
  08:43:20 STEP 2726 H=2726:00 +CPU= 19.214
  08:43:47 STEP 2727 H=2727:00 +CPU= 25.841
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2017050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0513
(argv4) batchid: 986
(argv5) wuid: 12203157
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/0
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_4ea7a0c1bb6bc512a1f3d63b82622f1b to: /var/lib/boinc-client/slots/0/oifs_43r3_ps_0513_2017050100_123_986_12203157.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/0/oifs_43r3_ps_0513_2017050100_123_986_12203157.zip
ic_ancil_file: ic_ancil_12203157
ifsdata_file: ifsdata_12203157
climate_data_file: clim_data_12203157
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_7c6b7a5239e8fa5551203885b25978bb to: /var/lib/boinc-client/slots/0/ic_ancil_12203157.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/0/ic_ancil_12203157.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_195d37be56ec72e91929a4c22a23e7da to: /var/lib/boinc-client/slots/0/ifsdata/ifsdata_12203157.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/0/ifsdata/ifsdata_12203157.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/0/159l_2/clim_data_12203157.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/0/159l_2/clim_data_12203157.zip
Checking for progress XML file: /var/lib/boinc-client/slots/0/progress_file_12203157.xml
Opened progress file ok : /var/lib/boinc-client/slots/0/progress_file_12203157.xml
-- Model is restarting --
Adjusting last_iter, 2728, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/0/progress_file_12203157.xml
last_cpu_time: 49455
upload_file_number: 113
last_iter: 2711
last_upload: 9763200
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0513_2017050100_123_986_12203157_2_r835230155
The child process has been launched with process id: 1744844
Executing the command: /var/lib/boinc-client/slots/0/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:1744844:1744844] [20230117:084501:1673970301:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:1744844:1744844] [20230117:084501:1673970301:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:1744844:1744844] [20230117:084501:1673970301:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:1744844:1744844] [20230117:084501:1673970301:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:1744844:1744844] [20230117:084501:1673970301:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:1744844:1744844] [20230117:084501:1673970301:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:1744844:1744844] [20230117:084501:1673970301:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:1744844:1744844] [20230117:084501:1673970301:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:1744844:1744844] [20230117:084501:1673970301:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:1744844:1744844] [20230117:084501:1673970301:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:1744844:1744844] [20230117:084501:1673970301:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:1744844:1744844] [20230117:084501:1673970301:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:1744844:1744844] [20230117:084501:1673970301:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:1744844:1744844] [20230117:084501:1673970301:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:1744844:1744844] [20230117:084501:1673970301:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:1744844:1744844] [20230117:084501:1673970301:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:1744844:1744844] [20230117:084501:1673970301:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:1744844:1744844] [20230117:084501:1673970301:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:1744844:1744844] [20230117:084501:1673970301:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:1744844:1744844] [20230117:084501:1673970301:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:1744844:1744844] [20230117:084501:1673970301:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:1744844:1744844] [20230117:084501:1673970301:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:1744844:1744844] [20230117:084501:1673970301:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:1744844:1744844] [20230117:084501:1673970301:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:1744844:1744844] [20230117:084501:1673970301:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:1744844:1744844] [20230117:084501:1673970301:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:1744844:1744844] [20230117:084501:1673970301:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:1744844:1744844] [20230117:084501:1673970301:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:1744844:1744844] [20230117:084501:1673970301:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:1744844:1744844] [20230117:084501:1673970301:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  08:45:38 STEP 2712 H=2712:00 +CPU= 33.750
  08:45:59 STEP 2713 H=2713:00 +CPU= 20.959
  08:46:19 STEP 2714 H=2714:00 +CPU= 19.278
  08:46:48 STEP 2715 H=2715:00 +CPU= 29.436
  08:47:05 STEP 2716 H=2716:00 +CPU= 16.769
  08:47:23 STEP 2717 H=2717:00 +CPU= 17.092
  08:47:52 STEP 2718 H=2718:00 +CPU= 29.213
  08:48:11 STEP 2719 H=2719:00 +CPU= 18.801
  08:48:29 STEP 2720 H=2720:00 +CPU= 17.097
  08:48:58 STEP 2721 H=2721:00 +CPU= 28.794
  08:49:18 STEP 2722 H=2722:00 +CPU= 20.260
  08:49:35 STEP 2723 H=2723:00 +CPU= 16.847
  08:50:06 STEP 2724 H=2724:00 +CPU= 30.453
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002724
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002724
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002724
  08:50:26 STEP 2725 H=2725:00 +CPU= 19.458
  08:50:45 STEP 2726 H=2726:00 +CPU= 18.453
  08:51:13 STEP 2727 H=2727:00 +CPU= 27.694
  08:51:33 STEP 2728 H=2728:00 +CPU= 19.292
  08:51:52 STEP 2729 H=2729:00 +CPU= 18.677
  08:52:22 STEP 2730 H=2730:00 +CPU= 29.027
  08:52:39 STEP 2731 H=2731:00 +CPU= 17.273
  08:52:58 STEP 2732 H=2732:00 +CPU= 18.435
  08:53:29 STEP 2733 H=2733:00 +CPU= 30.916
  08:53:50 STEP 2734 H=2734:00 +CPU= 20.439
  08:54:09 STEP 2735 H=2735:00 +CPU= 18.531
  08:54:45 STEP 2736 H=2736:00 +CPU= 35.523
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002736
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002736
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMGGhq0f+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMSHhq0f+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMUAhq0f+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMGGhq0f+002724
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMSHhq0f+002724
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMUAhq0f+002724
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0513_2017050100_123_986_12203157_2_r835230155_113.zip
Uploading the intermediate file: upload_file_113.zip
  08:55:04 STEP 2737 H=2737:00 +CPU= 18.710
  08:55:23 STEP 2738 H=2738:00 +CPU= 18.436
  08:55:53 STEP 2739 H=2739:00 +CPU= 29.297
  08:56:12 STEP 2740 H=2740:00 +CPU= 19.250
  08:56:32 STEP 2741 H=2741:00 +CPU= 19.498
  08:57:00 STEP 2742 H=2742:00 +CPU= 27.812
  08:57:16 STEP 2743 H=2743:00 +CPU= 16.291
  08:57:35 STEP 2744 H=2744:00 +CPU= 18.404
  08:58:02 STEP 2745 H=2745:00 +CPU= 26.146
  08:58:23 STEP 2746 H=2746:00 +CPU= 21.091
  08:58:41 STEP 2747 H=2747:00 +CPU= 18.303
  08:59:11 STEP 2748 H=2748:00 +CPU= 29.202
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002748
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002748
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002748
  08:59:28 STEP 2749 H=2749:00 +CPU= 16.461
  08:59:48 STEP 2750 H=2750:00 +CPU= 19.570
  09:00:18 STEP 2751 H=2751:00 +CPU= 29.293
  09:00:37 STEP 2752 H=2752:00 +CPU= 18.514
  09:00:54 STEP 2753 H=2753:00 +CPU= 16.800
  09:01:24 STEP 2754 H=2754:00 +CPU= 30.054
  09:01:45 STEP 2755 H=2755:00 +CPU= 20.165
  09:02:05 STEP 2756 H=2756:00 +CPU= 19.915
  09:02:35 STEP 2757 H=2757:00 +CPU= 30.019
  09:02:56 STEP 2758 H=2758:00 +CPU= 20.202
  09:03:16 STEP 2759 H=2759:00 +CPU= 19.794
  09:03:49 STEP 2760 H=2760:00 +CPU= 32.228
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002760
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002760
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMGGhq0f+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMSHhq0f+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMUAhq0f+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMGGhq0f+002748
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMSHhq0f+002748
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMUAhq0f+002748
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0513_2017050100_123_986_12203157_2_r835230155_114.zip
Uploading the intermediate file: upload_file_114.zip
  09:04:08 STEP 2761 H=2761:00 +CPU= 19.134
  09:04:28 STEP 2762 H=2762:00 +CPU= 19.629
  09:04:56 STEP 2763 H=2763:00 +CPU= 27.663
  09:05:16 STEP 2764 H=2764:00 +CPU= 19.226
  09:05:36 STEP 2765 H=2765:00 +CPU= 19.883
  09:06:06 STEP 2766 H=2766:00 +CPU= 29.561
  09:06:25 STEP 2767 H=2767:00 +CPU= 19.240
  09:06:46 STEP 2768 H=2768:00 +CPU= 20.686
  09:07:15 STEP 2769 H=2769:00 +CPU= 28.728
  09:07:33 STEP 2770 H=2770:00 +CPU= 18.058
  09:07:52 STEP 2771 H=2771:00 +CPU= 18.321
  09:08:27 STEP 2772 H=2772:00 +CPU= 34.301
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002772
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002772
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002772
  09:08:45 STEP 2773 H=2773:00 +CPU= 17.309
  09:09:04 STEP 2774 H=2774:00 +CPU= 19.445
  09:09:34 STEP 2775 H=2775:00 +CPU= 29.253
  09:09:54 STEP 2776 H=2776:00 +CPU= 19.465
  09:10:13 STEP 2777 H=2777:00 +CPU= 18.671
  09:10:43 STEP 2778 H=2778:00 +CPU= 29.952
  09:11:02 STEP 2779 H=2779:00 +CPU= 17.988
  09:11:21 STEP 2780 H=2780:00 +CPU= 18.821
  09:11:50 STEP 2781 H=2781:00 +CPU= 28.479
  09:12:09 STEP 2782 H=2782:00 +CPU= 19.194
  09:12:30 STEP 2783 H=2783:00 +CPU= 20.150
  09:13:04 STEP 2784 H=2784:00 +CPU= 33.796
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002784
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002784
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMGGhq0f+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMSHhq0f+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMUAhq0f+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMGGhq0f+002772
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMSHhq0f+002772
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMUAhq0f+002772
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0513_2017050100_123_986_12203157_2_r835230155_115.zip
Uploading the intermediate file: upload_file_115.zip
  09:13:23 STEP 2785 H=2785:00 +CPU= 18.711
  09:13:43 STEP 2786 H=2786:00 +CPU= 19.515
  09:14:13 STEP 2787 H=2787:00 +CPU= 29.913
  09:14:32 STEP 2788 H=2788:00 +CPU= 18.098
  09:14:50 STEP 2789 H=2789:00 +CPU= 18.401
  09:15:21 STEP 2790 H=2790:00 +CPU= 30.160
  09:15:40 STEP 2791 H=2791:00 +CPU= 18.689
  09:15:59 STEP 2792 H=2792:00 +CPU= 18.638
  09:16:28 STEP 2793 H=2793:00 +CPU= 29.392
  09:16:47 STEP 2794 H=2794:00 +CPU= 18.504
  09:17:06 STEP 2795 H=2795:00 +CPU= 18.221
  09:17:38 STEP 2796 H=2796:00 +CPU= 32.293
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002796
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002796
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002796
  09:17:58 STEP 2797 H=2797:00 +CPU= 19.109
  09:18:18 STEP 2798 H=2798:00 +CPU= 19.468
  09:18:48 STEP 2799 H=2799:00 +CPU= 29.735
  09:19:08 STEP 2800 H=2800:00 +CPU= 19.627
  09:19:28 STEP 2801 H=2801:00 +CPU= 19.859
  09:19:59 STEP 2802 H=2802:00 +CPU= 30.823
  09:20:18 STEP 2803 H=2803:00 +CPU= 18.214
  09:20:37 STEP 2804 H=2804:00 +CPU= 18.495
  09:21:04 STEP 2805 H=2805:00 +CPU= 27.066
  09:21:23 STEP 2806 H=2806:00 +CPU= 18.669
  09:21:43 STEP 2807 H=2807:00 +CPU= 19.609
  09:22:16 STEP 2808 H=2808:00 +CPU= 31.831
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002808
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002808
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMGGhq0f+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMSHhq0f+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMUAhq0f+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMGGhq0f+002796
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMSHhq0f+002796
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMUAhq0f+002796
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0513_2017050100_123_986_12203157_2_r835230155_116.zip
Uploading the intermediate file: upload_file_116.zip
  09:22:35 STEP 2809 H=2809:00 +CPU= 18.848
  09:22:55 STEP 2810 H=2810:00 +CPU= 19.559
  09:23:25 STEP 2811 H=2811:00 +CPU= 30.111
  09:23:44 STEP 2812 H=2812:00 +CPU= 18.696
  09:24:03 STEP 2813 H=2813:00 +CPU= 18.612
  09:24:32 STEP 2814 H=2814:00 +CPU= 28.966
  09:24:50 STEP 2815 H=2815:00 +CPU= 17.411
  09:25:09 STEP 2816 H=2816:00 +CPU= 18.492
  09:25:38 STEP 2817 H=2817:00 +CPU= 28.884
  09:25:57 STEP 2818 H=2818:00 +CPU= 18.555
  09:26:16 STEP 2819 H=2819:00 +CPU= 19.018
  09:26:47 STEP 2820 H=2820:00 +CPU= 30.501
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002820
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002820
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002820
  09:27:04 STEP 2821 H=2821:00 +CPU= 17.059
  09:27:24 STEP 2822 H=2822:00 +CPU= 19.250
  09:27:54 STEP 2823 H=2823:00 +CPU= 29.277
  09:28:12 STEP 2824 H=2824:00 +CPU= 17.522
  09:28:31 STEP 2825 H=2825:00 +CPU= 18.961
  09:29:02 STEP 2826 H=2826:00 +CPU= 30.198
  09:29:21 STEP 2827 H=2827:00 +CPU= 19.278
  09:29:41 STEP 2828 H=2828:00 +CPU= 19.544
  09:30:12 STEP 2829 H=2829:00 +CPU= 30.278
  09:30:31 STEP 2830 H=2830:00 +CPU= 18.790
  09:30:48 STEP 2831 H=2831:00 +CPU= 17.061
  09:31:24 STEP 2832 H=2832:00 +CPU= 34.829
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002832
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002832
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMGGhq0f+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMSHhq0f+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMUAhq0f+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMGGhq0f+002820
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMSHhq0f+002820
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMUAhq0f+002820
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0513_2017050100_123_986_12203157_2_r835230155_117.zip
Uploading the intermediate file: upload_file_117.zip
  09:31:43 STEP 2833 H=2833:00 +CPU= 18.421
  09:32:03 STEP 2834 H=2834:00 +CPU= 20.078
  09:32:32 STEP 2835 H=2835:00 +CPU= 28.428
  09:32:49 STEP 2836 H=2836:00 +CPU= 17.298
  09:33:07 STEP 2837 H=2837:00 +CPU= 17.821
  09:33:36 STEP 2838 H=2838:00 +CPU= 28.493
  09:33:55 STEP 2839 H=2839:00 +CPU= 18.700
  09:34:14 STEP 2840 H=2840:00 +CPU= 18.363
  09:34:42 STEP 2841 H=2841:00 +CPU= 27.913
  09:35:01 STEP 2842 H=2842:00 +CPU= 18.748
  09:35:19 STEP 2843 H=2843:00 +CPU= 18.438
  09:35:51 STEP 2844 H=2844:00 +CPU= 31.304
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002844
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002844
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002844
  09:36:09 STEP 2845 H=2845:00 +CPU= 18.096
  09:36:27 STEP 2846 H=2846:00 +CPU= 18.066
  09:36:55 STEP 2847 H=2847:00 +CPU= 27.547
  09:37:14 STEP 2848 H=2848:00 +CPU= 17.995
  09:37:32 STEP 2849 H=2849:00 +CPU= 18.410
  09:38:02 STEP 2850 H=2850:00 +CPU= 29.080
  09:38:20 STEP 2851 H=2851:00 +CPU= 17.896
  09:38:40 STEP 2852 H=2852:00 +CPU= 19.267
  09:39:07 STEP 2853 H=2853:00 +CPU= 27.216
  09:39:27 STEP 2854 H=2854:00 +CPU= 19.026
  09:39:47 STEP 2855 H=2855:00 +CPU= 20.761
  09:40:21 STEP 2856 H=2856:00 +CPU= 32.834
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002856
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002856
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002856
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMGGhq0f+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMSHhq0f+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMUAhq0f+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMGGhq0f+002844
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMSHhq0f+002844
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMUAhq0f+002844
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0513_2017050100_123_986_12203157_2_r835230155_118.zip
Uploading the intermediate file: upload_file_118.zip
  09:40:39 STEP 2857 H=2857:00 +CPU= 18.133
  09:40:58 STEP 2858 H=2858:00 +CPU= 18.434
  09:41:26 STEP 2859 H=2859:00 +CPU= 27.299
  09:41:45 STEP 2860 H=2860:00 +CPU= 19.055
  09:42:02 STEP 2861 H=2861:00 +CPU= 16.830
  09:42:32 STEP 2862 H=2862:00 +CPU= 28.905
  09:42:52 STEP 2863 H=2863:00 +CPU= 19.810
  09:43:12 STEP 2864 H=2864:00 +CPU= 19.389
  09:43:38 STEP 2865 H=2865:00 +CPU= 26.630
  09:43:58 STEP 2866 H=2866:00 +CPU= 19.300
  09:44:19 STEP 2867 H=2867:00 +CPU= 20.763
  09:44:51 STEP 2868 H=2868:00 +CPU= 31.986
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002868
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002868
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002868
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2017050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0513
(argv4) batchid: 986
(argv5) wuid: 12203157
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/0
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_4ea7a0c1bb6bc512a1f3d63b82622f1b to: /var/lib/boinc-client/slots/0/oifs_43r3_ps_0513_2017050100_123_986_12203157.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/0/oifs_43r3_ps_0513_2017050100_123_986_12203157.zip
ic_ancil_file: ic_ancil_12203157
ifsdata_file: ifsdata_12203157
climate_data_file: clim_data_12203157
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_7c6b7a5239e8fa5551203885b25978bb to: /var/lib/boinc-client/slots/0/ic_ancil_12203157.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/0/ic_ancil_12203157.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_195d37be56ec72e91929a4c22a23e7da to: /var/lib/boinc-client/slots/0/ifsdata/ifsdata_12203157.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/0/ifsdata/ifsdata_12203157.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/0/159l_2/clim_data_12203157.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/0/159l_2/clim_data_12203157.zip
Checking for progress XML file: /var/lib/boinc-client/slots/0/progress_file_12203157.xml
Opened progress file ok : /var/lib/boinc-client/slots/0/progress_file_12203157.xml
-- Model is restarting --
Adjusting last_iter, 2869, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/0/progress_file_12203157.xml
last_cpu_time: 53001
upload_file_number: 119
last_iter: 2855
last_upload: 10281600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0513_2017050100_123_986_12203157_2_r835230155
The child process has been launched with process id: 1747006
Executing the command: /var/lib/boinc-client/slots/0/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:1747006:1747006] [20230117:094607:1673973967:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:1747006:1747006] [20230117:094607:1673973967:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:1747006:1747006] [20230117:094607:1673973967:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:1747006:1747006] [20230117:094607:1673973967:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:1747006:1747006] [20230117:094607:1673973967:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:1747006:1747006] [20230117:094607:1673973967:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:1747006:1747006] [20230117:094607:1673973967:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:1747006:1747006] [20230117:094607:1673973967:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:1747006:1747006] [20230117:094607:1673973967:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:1747006:1747006] [20230117:094607:1673973967:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:1747006:1747006] [20230117:094607:1673973967:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:1747006:1747006] [20230117:094607:1673973967:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:1747006:1747006] [20230117:094607:1673973967:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:1747006:1747006] [20230117:094607:1673973967:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:1747006:1747006] [20230117:094607:1673973967:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:1747006:1747006] [20230117:094607:1673973967:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:1747006:1747006] [20230117:094607:1673973967:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:1747006:1747006] [20230117:094607:1673973967:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:1747006:1747006] [20230117:094607:1673973967:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:1747006:1747006] [20230117:094607:1673973967:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:1747006:1747006] [20230117:094607:1673973967:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:1747006:1747006] [20230117:094607:1673973967:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:1747006:1747006] [20230117:094607:1673973967:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:1747006:1747006] [20230117:094607:1673973967:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:1747006:1747006] [20230117:094607:1673973967:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:1747006:1747006] [20230117:094607:1673973967:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:1747006:1747006] [20230117:094607:1673973967:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:1747006:1747006] [20230117:094607:1673973967:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:1747006:1747006] [20230117:094607:1673973967:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:1747006:1747006] [20230117:094607:1673973967:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  09:46:42 STEP 2856 H=2856:00 +CPU= 31.735
  09:47:01 STEP 2857 H=2857:00 +CPU= 18.544
  09:47:18 STEP 2858 H=2858:00 +CPU= 17.704
  09:47:49 STEP 2859 H=2859:00 +CPU= 30.119
  09:48:06 STEP 2860 H=2860:00 +CPU= 16.864
  09:48:24 STEP 2861 H=2861:00 +CPU= 17.648
  09:48:55 STEP 2862 H=2862:00 +CPU= 30.330
  09:49:14 STEP 2863 H=2863:00 +CPU= 18.678
  09:49:32 STEP 2864 H=2864:00 +CPU= 17.970
  09:50:01 STEP 2865 H=2865:00 +CPU= 28.517
  09:50:20 STEP 2866 H=2866:00 +CPU= 18.774
  09:50:39 STEP 2867 H=2867:00 +CPU= 19.170
  09:51:08 STEP 2868 H=2868:00 +CPU= 28.291
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002868
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002868
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002868
  09:51:27 STEP 2869 H=2869:00 +CPU= 19.110
  09:51:49 STEP 2870 H=2870:00 +CPU= 20.879
  09:52:18 STEP 2871 H=2871:00 +CPU= 28.855
  09:52:37 STEP 2872 H=2872:00 +CPU= 18.397
  09:52:56 STEP 2873 H=2873:00 +CPU= 19.198
  09:53:25 STEP 2874 H=2874:00 +CPU= 28.377
  09:53:44 STEP 2875 H=2875:00 +CPU= 18.594
  09:54:03 STEP 2876 H=2876:00 +CPU= 18.917
  09:54:32 STEP 2877 H=2877:00 +CPU= 28.611
  09:54:52 STEP 2878 H=2878:00 +CPU= 19.221
  09:55:12 STEP 2879 H=2879:00 +CPU= 19.587
  09:55:47 STEP 2880 H=2880:00 +CPU= 34.261
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002880
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002880
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002880
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMGGhq0f+002856
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMSHhq0f+002856
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMUAhq0f+002856
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMGGhq0f+002868
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMSHhq0f+002868
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMUAhq0f+002868
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0513_2017050100_123_986_12203157_2_r835230155_119.zip
Uploading the intermediate file: upload_file_119.zip
  09:56:03 STEP 2881 H=2881:00 +CPU= 15.631
  09:56:22 STEP 2882 H=2882:00 +CPU= 18.500
  09:56:52 STEP 2883 H=2883:00 +CPU= 29.854
  09:57:11 STEP 2884 H=2884:00 +CPU= 19.200
  09:57:30 STEP 2885 H=2885:00 +CPU= 18.802
  09:58:02 STEP 2886 H=2886:00 +CPU= 30.845
  09:58:20 STEP 2887 H=2887:00 +CPU= 18.390
  09:58:40 STEP 2888 H=2888:00 +CPU= 19.800
  09:59:08 STEP 2889 H=2889:00 +CPU= 27.425
  09:59:28 STEP 2890 H=2890:00 +CPU= 19.649
  09:59:48 STEP 2891 H=2891:00 +CPU= 19.462
  10:00:20 STEP 2892 H=2892:00 +CPU= 31.999
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002892
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002892
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002892
  10:00:38 STEP 2893 H=2893:00 +CPU= 17.229
  10:00:57 STEP 2894 H=2894:00 +CPU= 19.073
  10:01:26 STEP 2895 H=2895:00 +CPU= 28.326
  10:01:46 STEP 2896 H=2896:00 +CPU= 19.391
  10:02:05 STEP 2897 H=2897:00 +CPU= 19.097
  10:02:34 STEP 2898 H=2898:00 +CPU= 28.013
  10:02:52 STEP 2899 H=2899:00 +CPU= 18.361
  10:03:14 STEP 2900 H=2900:00 +CPU= 20.791
  10:03:44 STEP 2901 H=2901:00 +CPU= 30.088
  10:04:03 STEP 2902 H=2902:00 +CPU= 18.784
  10:04:24 STEP 2903 H=2903:00 +CPU= 20.378
  10:04:58 STEP 2904 H=2904:00 +CPU= 33.689
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002904
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002904
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002904
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMGGhq0f+002880
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMSHhq0f+002880
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMUAhq0f+002880
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMGGhq0f+002892
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMSHhq0f+002892
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMUAhq0f+002892
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0513_2017050100_123_986_12203157_2_r835230155_120.zip
Uploading the intermediate file: upload_file_120.zip
  10:05:17 STEP 2905 H=2905:00 +CPU= 18.504
  10:05:36 STEP 2906 H=2906:00 +CPU= 19.045
  10:06:06 STEP 2907 H=2907:00 +CPU= 28.892
  10:06:25 STEP 2908 H=2908:00 +CPU= 19.072
  10:06:43 STEP 2909 H=2909:00 +CPU= 17.527
  10:07:14 STEP 2910 H=2910:00 +CPU= 30.714
  10:07:32 STEP 2911 H=2911:00 +CPU= 18.241
  10:07:53 STEP 2912 H=2912:00 +CPU= 20.404
  10:08:21 STEP 2913 H=2913:00 +CPU= 28.027
  10:08:41 STEP 2914 H=2914:00 +CPU= 19.496
  10:09:00 STEP 2915 H=2915:00 +CPU= 18.277
  10:09:31 STEP 2916 H=2916:00 +CPU= 30.530
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002916
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002916
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002916
  10:09:50 STEP 2917 H=2917:00 +CPU= 19.164
  10:10:11 STEP 2918 H=2918:00 +CPU= 20.243
  10:10:41 STEP 2919 H=2919:00 +CPU= 29.680
  10:11:01 STEP 2920 H=2920:00 +CPU= 19.477
  10:11:20 STEP 2921 H=2921:00 +CPU= 19.038
  10:11:49 STEP 2922 H=2922:00 +CPU= 28.853
  10:12:07 STEP 2923 H=2923:00 +CPU= 17.836
  10:12:26 STEP 2924 H=2924:00 +CPU= 18.243
  10:12:57 STEP 2925 H=2925:00 +CPU= 30.543
  10:13:16 STEP 2926 H=2926:00 +CPU= 18.927
  10:13:35 STEP 2927 H=2927:00 +CPU= 19.159
  10:14:11 STEP 2928 H=2928:00 +CPU= 34.811
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002928
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002928
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002928
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMGGhq0f+002904
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMSHhq0f+002904
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMUAhq0f+002904
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMGGhq0f+002916
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMSHhq0f+002916
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMUAhq0f+002916
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0513_2017050100_123_986_12203157_2_r835230155_121.zip
Uploading the intermediate file: upload_file_121.zip
  10:14:29 STEP 2929 H=2929:00 +CPU= 18.058
  10:14:48 STEP 2930 H=2930:00 +CPU= 18.009
  10:15:16 STEP 2931 H=2931:00 +CPU= 28.207
  10:15:36 STEP 2932 H=2932:00 +CPU= 19.345
  10:15:55 STEP 2933 H=2933:00 +CPU= 18.976
  10:16:25 STEP 2934 H=2934:00 +CPU= 29.526
  10:16:44 STEP 2935 H=2935:00 +CPU= 18.309
  10:17:03 STEP 2936 H=2936:00 +CPU= 19.004
  10:17:31 STEP 2937 H=2937:00 +CPU= 27.792
  10:17:51 STEP 2938 H=2938:00 +CPU= 19.590
  10:18:10 STEP 2939 H=2939:00 +CPU= 18.979
  10:18:42 STEP 2940 H=2940:00 +CPU= 30.878
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002940
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002940
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002940
  10:19:00 STEP 2941 H=2941:00 +CPU= 18.062
  10:19:19 STEP 2942 H=2942:00 +CPU= 18.933
  10:19:48 STEP 2943 H=2943:00 +CPU= 27.980
  10:20:06 STEP 2944 H=2944:00 +CPU= 17.999
  10:20:26 STEP 2945 H=2945:00 +CPU= 19.010
  10:20:57 STEP 2946 H=2946:00 +CPU= 30.916
  10:21:15 STEP 2947 H=2947:00 +CPU= 18.025
  10:21:36 STEP 2948 H=2948:00 +CPU= 20.658
  10:22:06 STEP 2949 H=2949:00 +CPU= 29.907
  10:22:25 STEP 2950 H=2950:00 +CPU= 18.455
  10:22:45 STEP 2951 H=2951:00 +CPU= 19.745
  10:23:18 STEP 2952 H=2952:00 +CPU= 32.833
The child process terminated with status: 0
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002952
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002952
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002952
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMGGhq0f+002940
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMSHhq0f+002928
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMUAhq0f+002928
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMUAhq0f+002940
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMSHhq0f+002940
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMGGhq0f+002928
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMSHhq0f+002952
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMUAhq0f+002952
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203157/ICMGGhq0f+002952
Zipping up the final file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0513_2017050100_123_986_12203157_2_r835230155_122.zip
Uploading the final file: upload_file_122.zip
Uploading trickle at timestep: 10623600
10:26:41 (1746999): called boinc_finish(0)

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:24:36 1532237 22298577 oifs_43r3_ps_0513_2017050100_123_986_12203157_2 10,623,600 55,199 0.0052
09 May 2023 16:24:11 1532237 22298577 oifs_43r3_ps_0513_2017050100_123_986_12203157_2 7,776,000 36,908 0.0047
09 May 2023 16:24:06 1532237 22298577 oifs_43r3_ps_0513_2017050100_123_986_12203157_2 6,912,000 32,758 0.0047
09 May 2023 16:24:01 1532237 22298577 oifs_43r3_ps_0513_2017050100_123_986_12203157_2 6,048,000 28,710 0.0047
09 May 2023 16:23:50 1532237 22298577 oifs_43r3_ps_0513_2017050100_123_986_12203157_2 5,184,000 24,659 0.0048
09 May 2023 16:23:46 1532237 22298577 oifs_43r3_ps_0513_2017050100_123_986_12203157_2 4,320,000 20,814 0.0048
09 May 2023 16:23:41 1532237 22298577 oifs_43r3_ps_0513_2017050100_123_986_12203157_2 3,456,000 16,793 0.0049
09 May 2023 16:23:36 1532237 22298577 oifs_43r3_ps_0513_2017050100_123_986_12203157_2 2,592,000 12,873 0.0050
09 May 2023 16:23:32 1532237 22298577 oifs_43r3_ps_0513_2017050100_123_986_12203157_2 1,728,000 8,749 0.0051
09 May 2023 16:23:27 1532237 22298577 oifs_43r3_ps_0513_2017050100_123_986_12203157_2 864,000 4,556 0.0053


©2024 climateprediction.net