climateprediction.net home page
Task 22276178

Task 22276178

Name oifs_43r3_ps_0897_2004050100_123_973_12190541_0
Workunit 12190541
Created 23 Dec 2022, 9:09:03 UTC
Sent 16 Jan 2023, 8:15:14 UTC
Report deadline 15 Feb 2023, 8:15:14 UTC
Received 18 Jan 2023, 2:19:03 UTC
Server state Over
Outcome Success
Client state Done
Exit status 0 (0x00000000)
Computer ID 1513103
Run time 15 hours 36 min 56 sec
CPU time 14 hours 33 min 52 sec
Validate state Valid
Credit 14,757.12
Device peak FLOPS 7.83 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,419.27 MB
Peak swap size 4,978.08 MB
Peak disk usage 4,433.30 MB
Stderr
<core_client_version>7.18.1</core_client_version>
<![CDATA[
<stderr_txt>
RHOOK:johann:1:1:3374:3374] [20230118:000429:1673996669:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:johann:1:1:3374:3374] [20230118:000429:1673996669:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  00:05:34 STEP 2520 H=2520:00 +CPU= 27.107
  00:05:51 STEP 2521 H=2521:00 +CPU= 16.888
  00:06:08 STEP 2522 H=2522:00 +CPU= 16.997
  00:06:33 STEP 2523 H=2523:00 +CPU= 25.522
  00:06:50 STEP 2524 H=2524:00 +CPU= 16.790
  00:07:06 STEP 2525 H=2525:00 +CPU= 16.101
  00:07:31 STEP 2526 H=2526:00 +CPU= 25.358
  00:07:49 STEP 2527 H=2527:00 +CPU= 17.313
  00:08:07 STEP 2528 H=2528:00 +CPU= 18.624
  00:08:36 STEP 2529 H=2529:00 +CPU= 28.576
  00:08:54 STEP 2530 H=2530:00 +CPU= 18.339
  00:09:12 STEP 2531 H=2531:00 +CPU= 17.584
  00:09:44 STEP 2532 H=2532:00 +CPU= 32.141
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002532
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002532
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002532
  00:10:03 STEP 2533 H=2533:00 +CPU= 18.867
  00:10:23 STEP 2534 H=2534:00 +CPU= 20.268
  00:10:53 STEP 2535 H=2535:00 +CPU= 29.924
  00:11:12 STEP 2536 H=2536:00 +CPU= 19.185
  00:11:30 STEP 2537 H=2537:00 +CPU= 18.065
  00:11:59 STEP 2538 H=2538:00 +CPU= 28.992
  00:12:20 STEP 2539 H=2539:00 +CPU= 20.246
  00:12:39 STEP 2540 H=2540:00 +CPU= 19.307
  00:13:08 STEP 2541 H=2541:00 +CPU= 28.694
  00:13:26 STEP 2542 H=2542:00 +CPU= 18.626
  00:13:44 STEP 2543 H=2543:00 +CPU= 17.909
  00:14:16 STEP 2544 H=2544:00 +CPU= 31.420
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002544
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002544
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMGGhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMSHhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMUAhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMGGhq0f+002532
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMSHhq0f+002532
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMUAhq0f+002532
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0897_2004050100_123_973_12190541_0_r1810036735_105.zip
Uploading the intermediate file: upload_file_105.zip
  00:14:33 STEP 2545 H=2545:00 +CPU= 17.496
  00:14:50 STEP 2546 H=2546:00 +CPU= 17.024
  00:15:16 STEP 2547 H=2547:00 +CPU= 25.636
  00:15:32 STEP 2548 H=2548:00 +CPU= 15.987
  00:15:50 STEP 2549 H=2549:00 +CPU= 18.163
  00:16:15 STEP 2550 H=2550:00 +CPU= 24.546
  00:16:30 STEP 2551 H=2551:00 +CPU= 15.524
  00:16:47 STEP 2552 H=2552:00 +CPU= 16.194
  00:17:11 STEP 2553 H=2553:00 +CPU= 24.067
  00:17:27 STEP 2554 H=2554:00 +CPU= 16.373
  00:17:44 STEP 2555 H=2555:00 +CPU= 16.908
  00:18:11 STEP 2556 H=2556:00 +CPU= 26.953
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002556
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002556
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002556
  00:18:27 STEP 2557 H=2557:00 +CPU= 15.589
  00:18:42 STEP 2558 H=2558:00 +CPU= 15.872
  00:19:07 STEP 2559 H=2559:00 +CPU= 24.956
  00:19:24 STEP 2560 H=2560:00 +CPU= 16.435
  00:19:40 STEP 2561 H=2561:00 +CPU= 16.399
  00:20:04 STEP 2562 H=2562:00 +CPU= 24.288
  00:20:20 STEP 2563 H=2563:00 +CPU= 15.538
  00:20:37 STEP 2564 H=2564:00 +CPU= 16.636
  00:21:02 STEP 2565 H=2565:00 +CPU= 25.011
  00:21:18 STEP 2566 H=2566:00 +CPU= 15.981
  00:21:34 STEP 2567 H=2567:00 +CPU= 16.009
  00:22:03 STEP 2568 H=2568:00 +CPU= 28.642
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002568
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002568
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMGGhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMSHhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMUAhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMGGhq0f+002556
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMSHhq0f+002556
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMUAhq0f+002556
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0897_2004050100_123_973_12190541_0_r1810036735_106.zip
Uploading the intermediate file: upload_file_106.zip
  00:22:18 STEP 2569 H=2569:00 +CPU= 15.783
  00:22:35 STEP 2570 H=2570:00 +CPU= 16.967
  00:23:00 STEP 2571 H=2571:00 +CPU= 24.551
  00:23:15 STEP 2572 H=2572:00 +CPU= 15.561
  00:23:32 STEP 2573 H=2573:00 +CPU= 16.779
  00:23:58 STEP 2574 H=2574:00 +CPU= 26.030
  00:24:14 STEP 2575 H=2575:00 +CPU= 15.863
  00:24:30 STEP 2576 H=2576:00 +CPU= 15.998
  00:24:55 STEP 2577 H=2577:00 +CPU= 24.555
  00:25:12 STEP 2578 H=2578:00 +CPU= 17.000
  00:25:28 STEP 2579 H=2579:00 +CPU= 15.902
  00:25:54 STEP 2580 H=2580:00 +CPU= 26.252
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002580
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002580
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002580
  00:26:10 STEP 2581 H=2581:00 +CPU= 15.576
  00:26:26 STEP 2582 H=2582:00 +CPU= 16.621
  00:26:52 STEP 2583 H=2583:00 +CPU= 25.332
  00:27:08 STEP 2584 H=2584:00 +CPU= 15.792
  00:27:23 STEP 2585 H=2585:00 +CPU= 15.566
  00:27:49 STEP 2586 H=2586:00 +CPU= 25.583
  00:28:05 STEP 2587 H=2587:00 +CPU= 15.884
  00:28:21 STEP 2588 H=2588:00 +CPU= 16.673
  00:28:46 STEP 2589 H=2589:00 +CPU= 24.095
  00:29:02 STEP 2590 H=2590:00 +CPU= 16.040
  00:29:18 STEP 2591 H=2591:00 +CPU= 16.674
  00:29:47 STEP 2592 H=2592:00 +CPU= 28.354
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002592
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002592
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMGGhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMSHhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMUAhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMGGhq0f+002580
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMSHhq0f+002580
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMUAhq0f+002580
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0897_2004050100_123_973_12190541_0_r1810036735_107.zip
Uploading the intermediate file: upload_file_107.zip
  00:30:03 STEP 2593 H=2593:00 +CPU= 15.661
  00:30:19 STEP 2594 H=2594:00 +CPU= 15.837
  00:30:43 STEP 2595 H=2595:00 +CPU= 24.462
  00:30:59 STEP 2596 H=2596:00 +CPU= 15.994
  00:31:15 STEP 2597 H=2597:00 +CPU= 16.094
  00:31:40 STEP 2598 H=2598:00 +CPU= 24.155
  00:31:55 STEP 2599 H=2599:00 +CPU= 15.892
  00:32:12 STEP 2600 H=2600:00 +CPU= 16.548
  00:32:37 STEP 2601 H=2601:00 +CPU= 24.702
  00:32:53 STEP 2602 H=2602:00 +CPU= 15.918
  00:33:08 STEP 2603 H=2603:00 +CPU= 15.416
  00:33:35 STEP 2604 H=2604:00 +CPU= 27.444
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002604
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002604
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002604
  00:33:51 STEP 2605 H=2605:00 +CPU= 15.467
  00:34:07 STEP 2606 H=2606:00 +CPU= 15.837
  00:34:31 STEP 2607 H=2607:00 +CPU= 24.446
  00:34:49 STEP 2608 H=2608:00 +CPU= 17.523
  00:35:05 STEP 2609 H=2609:00 +CPU= 15.610
  00:35:29 STEP 2610 H=2610:00 +CPU= 23.923
  00:35:45 STEP 2611 H=2611:00 +CPU= 15.780
  00:36:00 STEP 2612 H=2612:00 +CPU= 15.662
  00:36:24 STEP 2613 H=2613:00 +CPU= 23.817
  00:36:40 STEP 2614 H=2614:00 +CPU= 16.005
  00:36:56 STEP 2615 H=2615:00 +CPU= 15.928
  00:37:28 STEP 2616 H=2616:00 +CPU= 30.724
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002616
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002616
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMGGhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMSHhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMUAhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMGGhq0f+002604
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMSHhq0f+002604
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMUAhq0f+002604
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0897_2004050100_123_973_12190541_0_r1810036735_108.zip
Uploading the intermediate file: upload_file_108.zip
  00:37:44 STEP 2617 H=2617:00 +CPU= 15.153
  00:38:00 STEP 2618 H=2618:00 +CPU= 15.875
  00:38:24 STEP 2619 H=2619:00 +CPU= 23.917
  00:38:46 STEP 2620 H=2620:00 +CPU= 17.806
  00:39:00 STEP 2621 H=2621:00 +CPU= 13.692
  00:39:25 STEP 2622 H=2622:00 +CPU= 23.733
  00:39:40 STEP 2623 H=2623:00 +CPU= 15.436
  00:39:56 STEP 2624 H=2624:00 +CPU= 15.484
  00:40:21 STEP 2625 H=2625:00 +CPU= 25.953
  00:40:39 STEP 2626 H=2626:00 +CPU= 17.149
  00:40:54 STEP 2627 H=2627:00 +CPU= 15.112
  00:41:20 STEP 2628 H=2628:00 +CPU= 25.770
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002628
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002628
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002628
  00:41:34 STEP 2629 H=2629:00 +CPU= 14.343
  00:41:50 STEP 2630 H=2630:00 +CPU= 16.197
  00:42:16 STEP 2631 H=2631:00 +CPU= 25.850
  00:42:32 STEP 2632 H=2632:00 +CPU= 16.162
  00:42:48 STEP 2633 H=2633:00 +CPU= 15.521
  00:43:14 STEP 2634 H=2634:00 +CPU= 24.355
  00:43:30 STEP 2635 H=2635:00 +CPU= 16.081
  00:43:46 STEP 2636 H=2636:00 +CPU= 15.877
  00:44:10 STEP 2637 H=2637:00 +CPU= 23.416
  00:44:26 STEP 2638 H=2638:00 +CPU= 16.252
  00:44:45 STEP 2639 H=2639:00 +CPU= 16.309
  00:45:16 STEP 2640 H=2640:00 +CPU= 28.492
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002640
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002640
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMGGhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMSHhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMUAhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMGGhq0f+002628
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMSHhq0f+002628
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMUAhq0f+002628
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0897_2004050100_123_973_12190541_0_r1810036735_109.zip
Uploading the intermediate file: upload_file_109.zip
  00:45:32 STEP 2641 H=2641:00 +CPU= 15.677
  00:45:48 STEP 2642 H=2642:00 +CPU= 15.953
  00:46:13 STEP 2643 H=2643:00 +CPU= 25.431
  00:46:30 STEP 2644 H=2644:00 +CPU= 16.765
  00:46:46 STEP 2645 H=2645:00 +CPU= 15.385
  00:47:10 STEP 2646 H=2646:00 +CPU= 24.354
  00:47:27 STEP 2647 H=2647:00 +CPU= 16.694
  00:47:45 STEP 2648 H=2648:00 +CPU= 16.780
  00:48:09 STEP 2649 H=2649:00 +CPU= 24.196
  00:48:25 STEP 2650 H=2650:00 +CPU= 15.899
  00:48:39 STEP 2651 H=2651:00 +CPU= 14.461
  00:49:06 STEP 2652 H=2652:00 +CPU= 26.446
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002652
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002652
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002652
  00:49:23 STEP 2653 H=2653:00 +CPU= 16.962
  00:49:40 STEP 2654 H=2654:00 +CPU= 17.151
  00:50:06 STEP 2655 H=2655:00 +CPU= 24.597
  00:50:22 STEP 2656 H=2656:00 +CPU= 15.900
  00:50:38 STEP 2657 H=2657:00 +CPU= 15.561
  00:51:04 STEP 2658 H=2658:00 +CPU= 25.091
  00:51:20 STEP 2659 H=2659:00 +CPU= 15.414
  00:51:36 STEP 2660 H=2660:00 +CPU= 15.546
  00:51:59 STEP 2661 H=2661:00 +CPU= 23.772
  00:52:17 STEP 2662 H=2662:00 +CPU= 17.671
  00:52:36 STEP 2663 H=2663:00 +CPU= 16.204
  00:53:05 STEP 2664 H=2664:00 +CPU= 27.679
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002664
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002664
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMGGhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMSHhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMUAhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMGGhq0f+002652
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMSHhq0f+002652
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMUAhq0f+002652
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0897_2004050100_123_973_12190541_0_r1810036735_110.zip
Uploading the intermediate file: upload_file_110.zip
  00:53:20 STEP 2665 H=2665:00 +CPU= 15.628
  00:53:36 STEP 2666 H=2666:00 +CPU= 16.126
  00:54:04 STEP 2667 H=2667:00 +CPU= 25.977
  00:54:20 STEP 2668 H=2668:00 +CPU= 16.154
  00:54:35 STEP 2669 H=2669:00 +CPU= 15.341
  00:54:59 STEP 2670 H=2670:00 +CPU= 24.299
  00:55:20 STEP 2671 H=2671:00 +CPU= 17.262
  00:55:36 STEP 2672 H=2672:00 +CPU= 15.709
  00:55:59 STEP 2673 H=2673:00 +CPU= 23.630
  00:56:15 STEP 2674 H=2674:00 +CPU= 16.200
  00:56:35 STEP 2675 H=2675:00 +CPU= 17.069
  00:57:04 STEP 2676 H=2676:00 +CPU= 28.372
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002676
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002676
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002676
  00:57:20 STEP 2677 H=2677:00 +CPU= 15.808
  00:57:35 STEP 2678 H=2678:00 +CPU= 15.806
  00:58:02 STEP 2679 H=2679:00 +CPU= 25.493
  00:58:20 STEP 2680 H=2680:00 +CPU= 18.292
  00:58:36 STEP 2681 H=2681:00 +CPU= 16.212
  00:59:01 STEP 2682 H=2682:00 +CPU= 24.382
  00:59:16 STEP 2683 H=2683:00 +CPU= 15.820
  00:59:32 STEP 2684 H=2684:00 +CPU= 15.767
  00:59:58 STEP 2685 H=2685:00 +CPU= 25.140
  01:00:15 STEP 2686 H=2686:00 +CPU= 16.944
  01:00:33 STEP 2687 H=2687:00 +CPU= 16.839
  01:01:06 STEP 2688 H=2688:00 +CPU= 28.510
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002688
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002688
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMGGhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMSHhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMUAhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMGGhq0f+002676
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMSHhq0f+002676
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMUAhq0f+002676
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0897_2004050100_123_973_12190541_0_r1810036735_111.zip
Uploading the intermediate file: upload_file_111.zip
  01:01:23 STEP 2689 H=2689:00 +CPU= 16.049
  01:01:39 STEP 2690 H=2690:00 +CPU= 15.630
  01:02:06 STEP 2691 H=2691:00 +CPU= 26.314
  01:02:22 STEP 2692 H=2692:00 +CPU= 16.031
  01:02:38 STEP 2693 H=2693:00 +CPU= 16.033
  01:03:03 STEP 2694 H=2694:00 +CPU= 24.488
  01:03:18 STEP 2695 H=2695:00 +CPU= 15.410
  01:03:34 STEP 2696 H=2696:00 +CPU= 15.643
  01:03:59 STEP 2697 H=2697:00 +CPU= 24.947
  01:04:22 STEP 2698 H=2698:00 +CPU= 18.068
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2004050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0897
(argv4) batchid: 973
(argv5) wuid: 12190541
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/3
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_23dd631530bf9d5e7bf7109e4bec962c to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0897_2004050100_123_973_12190541.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0897_2004050100_123_973_12190541.zip
ic_ancil_file: ic_ancil_12190541
ifsdata_file: ifsdata_12190541
climate_data_file: clim_data_12190541
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_d0a4111b8bc796e1a52d2b880fd02ff1 to: /var/lib/boinc-client/slots/3/ic_ancil_12190541.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/3/ic_ancil_12190541.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_06c87b58b83fb2035499d8cde1315358 to: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12190541.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12190541.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/3/159l_2/clim_data_12190541.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/3/159l_2/clim_data_12190541.zip
Checking for progress XML file: /var/lib/boinc-client/slots/3/progress_file_12190541.xml
Opened progress file ok : /var/lib/boinc-client/slots/3/progress_file_12190541.xml
-- Model is restarting --
Adjusting last_iter, 2699, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/3/progress_file_12190541.xml
last_cpu_time: 47137
upload_file_number: 112
last_iter: 2687
last_upload: 9676800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0897_2004050100_123_973_12190541_0_r1810036735
The child process has been launched with process id: 3837
Executing the command: /var/lib/boinc-client/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:johann:1:1:3837:3837] [20230118:010542:1674000342:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:johann:1:1:3837:3837] [20230118:010542:1674000342:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:johann:1:1:3837:3837] [20230118:010542:1674000342:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:johann:1:1:3837:3837] [20230118:010542:1674000342:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:johann:1:1:3837:3837] [20230118:010542:1674000342:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:johann:1:1:3837:3837] [20230118:010542:1674000342:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:johann:1:1:3837:3837] [20230118:010542:1674000342:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:johann:1:1:3837:3837] [20230118:010542:1674000342:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:johann:1:1:3837:3837] [20230118:010542:1674000342:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:johann:1:1:3837:3837] [20230118:010542:1674000342:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:johann:1:1:3837:3837] [20230118:010542:1674000342:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:johann:1:1:3837:3837] [20230118:010542:1674000342:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:johann:1:1:3837:3837] [20230118:010542:1674000342:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:johann:1:1:3837:3837] [20230118:010542:1674000342:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:johann:1:1:3837:3837] [20230118:010542:1674000342:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:johann:1:1:3837:3837] [20230118:010542:1674000342:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:johann:1:1:3837:3837] [20230118:010542:1674000342:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:johann:1:1:3837:3837] [20230118:010542:1674000342:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:johann:1:1:3837:3837] [20230118:010542:1674000342:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:johann:1:1:3837:3837] [20230118:010542:1674000342:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:johann:1:1:3837:3837] [20230118:010542:1674000342:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:johann:1:1:3837:3837] [20230118:010542:1674000342:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:johann:1:1:3837:3837] [20230118:010542:1674000342:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:johann:1:1:3837:3837] [20230118:010542:1674000342:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:johann:1:1:3837:3837] [20230118:010542:1674000342:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:johann:1:1:3837:3837] [20230118:010542:1674000342:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:johann:1:1:3837:3837] [20230118:010542:1674000342:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:johann:1:1:3837:3837] [20230118:010542:1674000342:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:johann:1:1:3837:3837] [20230118:010542:1674000342:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:johann:1:1:3837:3837] [20230118:010542:1674000342:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  01:06:32 STEP 2688 H=2688:00 +CPU= 30.347
  01:06:50 STEP 2689 H=2689:00 +CPU= 15.971
  01:07:06 STEP 2690 H=2690:00 +CPU= 15.864
  01:07:30 STEP 2691 H=2691:00 +CPU= 24.393
  01:07:47 STEP 2692 H=2692:00 +CPU= 16.321
  01:08:02 STEP 2693 H=2693:00 +CPU= 15.808
  01:08:27 STEP 2694 H=2694:00 +CPU= 24.343
  01:08:42 STEP 2695 H=2695:00 +CPU= 15.506
  01:08:58 STEP 2696 H=2696:00 +CPU= 15.650
  01:09:24 STEP 2697 H=2697:00 +CPU= 25.094
  01:09:42 STEP 2698 H=2698:00 +CPU= 16.700
  01:09:58 STEP 2699 H=2699:00 +CPU= 15.905
  01:10:25 STEP 2700 H=2700:00 +CPU= 27.294
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002700
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002700
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002700
  01:10:41 STEP 2701 H=2701:00 +CPU= 16.022
  01:10:58 STEP 2702 H=2702:00 +CPU= 17.059
  01:11:24 STEP 2703 H=2703:00 +CPU= 25.696
  01:11:40 STEP 2704 H=2704:00 +CPU= 16.518
  01:11:55 STEP 2705 H=2705:00 +CPU= 15.058
  01:12:20 STEP 2706 H=2706:00 +CPU= 24.722
  01:12:37 STEP 2707 H=2707:00 +CPU= 17.078
  01:12:55 STEP 2708 H=2708:00 +CPU= 17.477
  01:13:20 STEP 2709 H=2709:00 +CPU= 24.890
  01:13:36 STEP 2710 H=2710:00 +CPU= 16.002
  01:13:52 STEP 2711 H=2711:00 +CPU= 15.774
  01:14:23 STEP 2712 H=2712:00 +CPU= 30.131
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002712
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002712
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMGGhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMSHhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMUAhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMGGhq0f+002700
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMSHhq0f+002700
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMUAhq0f+002700
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0897_2004050100_123_973_12190541_0_r1810036735_112.zip
Uploading the intermediate file: upload_file_112.zip
  01:14:39 STEP 2713 H=2713:00 +CPU= 16.575
  01:14:55 STEP 2714 H=2714:00 +CPU= 15.681
  01:15:19 STEP 2715 H=2715:00 +CPU= 24.397
  01:15:37 STEP 2716 H=2716:00 +CPU= 17.558
  01:15:53 STEP 2717 H=2717:00 +CPU= 15.962
  01:16:18 STEP 2718 H=2718:00 +CPU= 25.193
  01:16:33 STEP 2719 H=2719:00 +CPU= 15.158
  01:16:49 STEP 2720 H=2720:00 +CPU= 15.875
  01:17:14 STEP 2721 H=2721:00 +CPU= 24.447
  01:17:31 STEP 2722 H=2722:00 +CPU= 17.286
  01:17:47 STEP 2723 H=2723:00 +CPU= 15.695
  01:18:14 STEP 2724 H=2724:00 +CPU= 26.854
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002724
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002724
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002724
  01:18:30 STEP 2725 H=2725:00 +CPU= 15.811
  01:18:47 STEP 2726 H=2726:00 +CPU= 17.081
  01:19:11 STEP 2727 H=2727:00 +CPU= 24.438
  01:19:27 STEP 2728 H=2728:00 +CPU= 16.129
  01:19:43 STEP 2729 H=2729:00 +CPU= 15.661
  01:20:08 STEP 2730 H=2730:00 +CPU= 25.313
  01:20:24 STEP 2731 H=2731:00 +CPU= 15.834
  01:20:40 STEP 2732 H=2732:00 +CPU= 15.868
  01:21:04 STEP 2733 H=2733:00 +CPU= 24.165
  01:21:20 STEP 2734 H=2734:00 +CPU= 15.993
  01:21:37 STEP 2735 H=2735:00 +CPU= 17.142
  01:22:06 STEP 2736 H=2736:00 +CPU= 28.513
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002736
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002736
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMGGhq0f+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMSHhq0f+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMUAhq0f+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMGGhq0f+002724
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMSHhq0f+002724
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMUAhq0f+002724
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0897_2004050100_123_973_12190541_0_r1810036735_113.zip
Uploading the intermediate file: upload_file_113.zip
  01:22:21 STEP 2737 H=2737:00 +CPU= 15.524
  01:22:37 STEP 2738 H=2738:00 +CPU= 15.937
  01:23:02 STEP 2739 H=2739:00 +CPU= 24.504
  01:23:18 STEP 2740 H=2740:00 +CPU= 16.079
  01:23:33 STEP 2741 H=2741:00 +CPU= 15.522
  01:23:58 STEP 2742 H=2742:00 +CPU= 24.548
  01:24:14 STEP 2743 H=2743:00 +CPU= 16.533
  01:24:31 STEP 2744 H=2744:00 +CPU= 16.108
  01:24:55 STEP 2745 H=2745:00 +CPU= 24.064
  01:25:11 STEP 2746 H=2746:00 +CPU= 16.011
  01:25:27 STEP 2747 H=2747:00 +CPU= 15.960
  01:25:55 STEP 2748 H=2748:00 +CPU= 28.608
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002748
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002748
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002748
  01:26:11 STEP 2749 H=2749:00 +CPU= 15.480
  01:26:27 STEP 2750 H=2750:00 +CPU= 16.116
  01:26:51 STEP 2751 H=2751:00 +CPU= 23.982
  01:27:08 STEP 2752 H=2752:00 +CPU= 17.273
  01:27:25 STEP 2753 H=2753:00 +CPU= 16.555
  01:27:49 STEP 2754 H=2754:00 +CPU= 24.203
  01:28:05 STEP 2755 H=2755:00 +CPU= 15.545
  01:28:21 STEP 2756 H=2756:00 +CPU= 16.237
  01:28:46 STEP 2757 H=2757:00 +CPU= 25.173
  01:29:01 STEP 2758 H=2758:00 +CPU= 15.406
  01:29:16 STEP 2759 H=2759:00 +CPU= 14.753
  01:29:42 STEP 2760 H=2760:00 +CPU= 26.097
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002760
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002760
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMGGhq0f+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMSHhq0f+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMUAhq0f+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMGGhq0f+002748
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMSHhq0f+002748
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMUAhq0f+002748
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0897_2004050100_123_973_12190541_0_r1810036735_114.zip
Uploading the intermediate file: upload_file_114.zip
  01:29:57 STEP 2761 H=2761:00 +CPU= 14.774
  01:30:13 STEP 2762 H=2762:00 +CPU= 16.151
  01:30:38 STEP 2763 H=2763:00 +CPU= 24.437
  01:30:57 STEP 2764 H=2764:00 +CPU= 16.943
  01:31:13 STEP 2765 H=2765:00 +CPU= 15.712
  01:31:37 STEP 2766 H=2766:00 +CPU= 24.473
  01:31:53 STEP 2767 H=2767:00 +CPU= 15.549
  01:32:10 STEP 2768 H=2768:00 +CPU= 16.969
  01:32:35 STEP 2769 H=2769:00 +CPU= 25.365
  01:32:51 STEP 2770 H=2770:00 +CPU= 15.992
  01:33:07 STEP 2771 H=2771:00 +CPU= 15.664
  01:33:35 STEP 2772 H=2772:00 +CPU= 27.522
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002772
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002772
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002772
  01:33:51 STEP 2773 H=2773:00 +CPU= 16.160
  01:34:07 STEP 2774 H=2774:00 +CPU= 16.318
  01:34:31 STEP 2775 H=2775:00 +CPU= 24.227
  01:34:46 STEP 2776 H=2776:00 +CPU= 15.082
  01:35:02 STEP 2777 H=2777:00 +CPU= 15.443
  01:35:26 STEP 2778 H=2778:00 +CPU= 24.191
  01:35:42 STEP 2779 H=2779:00 +CPU= 15.718
  01:35:58 STEP 2780 H=2780:00 +CPU= 16.258
  01:36:22 STEP 2781 H=2781:00 +CPU= 23.762
  01:36:38 STEP 2782 H=2782:00 +CPU= 15.761
  01:36:54 STEP 2783 H=2783:00 +CPU= 16.293
  01:37:23 STEP 2784 H=2784:00 +CPU= 28.711
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002784
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002784
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMGGhq0f+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMSHhq0f+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMUAhq0f+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMGGhq0f+002772
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMSHhq0f+002772
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMUAhq0f+002772
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0897_2004050100_123_973_12190541_0_r1810036735_115.zip
Uploading the intermediate file: upload_file_115.zip
  01:37:39 STEP 2785 H=2785:00 +CPU= 16.314
  01:37:55 STEP 2786 H=2786:00 +CPU= 16.076
  01:38:19 STEP 2787 H=2787:00 +CPU= 23.898
  01:38:35 STEP 2788 H=2788:00 +CPU= 15.924
  01:38:52 STEP 2789 H=2789:00 +CPU= 16.787
  01:39:17 STEP 2790 H=2790:00 +CPU= 24.918
  01:39:32 STEP 2791 H=2791:00 +CPU= 15.498
  01:39:49 STEP 2792 H=2792:00 +CPU= 16.075
  01:40:13 STEP 2793 H=2793:00 +CPU= 24.759
  01:40:30 STEP 2794 H=2794:00 +CPU= 16.325
  01:40:46 STEP 2795 H=2795:00 +CPU= 15.927
  01:41:12 STEP 2796 H=2796:00 +CPU= 26.823
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002796
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002796
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002796
  01:41:29 STEP 2797 H=2797:00 +CPU= 16.158
  01:41:46 STEP 2798 H=2798:00 +CPU= 17.335
  01:42:10 STEP 2799 H=2799:00 +CPU= 24.027
  01:42:26 STEP 2800 H=2800:00 +CPU= 16.036
  01:42:42 STEP 2801 H=2801:00 +CPU= 15.629
  01:43:08 STEP 2802 H=2802:00 +CPU= 25.914
  01:43:24 STEP 2803 H=2803:00 +CPU= 16.408
  01:43:40 STEP 2804 H=2804:00 +CPU= 16.140
  01:44:04 STEP 2805 H=2805:00 +CPU= 24.101
  01:44:20 STEP 2806 H=2806:00 +CPU= 15.827
  01:44:37 STEP 2807 H=2807:00 +CPU= 16.631
  01:45:05 STEP 2808 H=2808:00 +CPU= 27.848
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002808
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002808
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMGGhq0f+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMSHhq0f+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMUAhq0f+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMGGhq0f+002796
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMSHhq0f+002796
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMUAhq0f+002796
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0897_2004050100_123_973_12190541_0_r1810036735_116.zip
Uploading the intermediate file: upload_file_116.zip
  01:45:21 STEP 2809 H=2809:00 +CPU= 16.015
  01:45:37 STEP 2810 H=2810:00 +CPU= 15.967
  01:46:03 STEP 2811 H=2811:00 +CPU= 25.032
  01:46:20 STEP 2812 H=2812:00 +CPU= 16.236
  01:46:35 STEP 2813 H=2813:00 +CPU= 15.440
  01:46:59 STEP 2814 H=2814:00 +CPU= 24.140
  01:47:15 STEP 2815 H=2815:00 +CPU= 16.022
  01:47:31 STEP 2816 H=2816:00 +CPU= 15.886
  01:47:55 STEP 2817 H=2817:00 +CPU= 24.045
  01:48:11 STEP 2818 H=2818:00 +CPU= 16.195
  01:48:28 STEP 2819 H=2819:00 +CPU= 16.328
  01:48:56 STEP 2820 H=2820:00 +CPU= 27.396
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002820
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002820
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002820
  01:49:11 STEP 2821 H=2821:00 +CPU= 15.017
  01:49:25 STEP 2822 H=2822:00 +CPU= 14.737
  01:49:48 STEP 2823 H=2823:00 +CPU= 22.329
  01:50:04 STEP 2824 H=2824:00 +CPU= 15.885
  01:50:20 STEP 2825 H=2825:00 +CPU= 15.845
  01:50:45 STEP 2826 H=2826:00 +CPU= 25.218
  01:51:02 STEP 2827 H=2827:00 +CPU= 16.414
  01:51:18 STEP 2828 H=2828:00 +CPU= 15.648
  01:51:40 STEP 2829 H=2829:00 +CPU= 22.179
  01:51:55 STEP 2830 H=2830:00 +CPU= 14.829
  01:52:11 STEP 2831 H=2831:00 +CPU= 15.866
  01:52:39 STEP 2832 H=2832:00 +CPU= 27.730
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002832
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002832
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMGGhq0f+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMSHhq0f+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMUAhq0f+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMGGhq0f+002820
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMSHhq0f+002820
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMUAhq0f+002820
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0897_2004050100_123_973_12190541_0_r1810036735_117.zip
Uploading the intermediate file: upload_file_117.zip
  01:52:55 STEP 2833 H=2833:00 +CPU= 15.679
  01:53:11 STEP 2834 H=2834:00 +CPU= 15.955
  01:53:35 STEP 2835 H=2835:00 +CPU= 23.828
  01:53:51 STEP 2836 H=2836:00 +CPU= 16.139
  01:54:06 STEP 2837 H=2837:00 +CPU= 15.552
  01:54:31 STEP 2838 H=2838:00 +CPU= 24.301
  01:54:48 STEP 2839 H=2839:00 +CPU= 15.946
  01:55:06 STEP 2840 H=2840:00 +CPU= 15.841
  01:55:30 STEP 2841 H=2841:00 +CPU= 23.711
  01:55:47 STEP 2842 H=2842:00 +CPU= 16.337
  01:56:03 STEP 2843 H=2843:00 +CPU= 15.838
  01:56:29 STEP 2844 H=2844:00 +CPU= 26.142
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002844
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002844
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002844
  01:56:45 STEP 2845 H=2845:00 +CPU= 15.842
  01:57:01 STEP 2846 H=2846:00 +CPU= 16.089
  01:57:25 STEP 2847 H=2847:00 +CPU= 23.943
  01:57:41 STEP 2848 H=2848:00 +CPU= 16.259
  01:57:57 STEP 2849 H=2849:00 +CPU= 15.738
  01:58:22 STEP 2850 H=2850:00 +CPU= 24.350
  01:58:38 STEP 2851 H=2851:00 +CPU= 16.026
  01:58:54 STEP 2852 H=2852:00 +CPU= 15.900
  01:59:17 STEP 2853 H=2853:00 +CPU= 23.819
  01:59:34 STEP 2854 H=2854:00 +CPU= 16.289
  01:59:50 STEP 2855 H=2855:00 +CPU= 16.165
  02:00:18 STEP 2856 H=2856:00 +CPU= 27.661
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002856
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002856
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002856
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMGGhq0f+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMSHhq0f+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMUAhq0f+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMGGhq0f+002844
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMSHhq0f+002844
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMUAhq0f+002844
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0897_2004050100_123_973_12190541_0_r1810036735_118.zip
Uploading the intermediate file: upload_file_118.zip
  02:00:34 STEP 2857 H=2857:00 +CPU= 15.880
  02:00:50 STEP 2858 H=2858:00 +CPU= 16.024
  02:01:14 STEP 2859 H=2859:00 +CPU= 23.733
  02:01:30 STEP 2860 H=2860:00 +CPU= 16.319
  02:01:46 STEP 2861 H=2861:00 +CPU= 15.652
  02:02:10 STEP 2862 H=2862:00 +CPU= 23.989
  02:02:26 STEP 2863 H=2863:00 +CPU= 15.861
  02:02:42 STEP 2864 H=2864:00 +CPU= 15.874
  02:03:05 STEP 2865 H=2865:00 +CPU= 23.671
  02:03:22 STEP 2866 H=2866:00 +CPU= 16.304
  02:03:37 STEP 2867 H=2867:00 +CPU= 15.633
  02:04:03 STEP 2868 H=2868:00 +CPU= 26.000
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002868
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002868
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002868
  02:04:19 STEP 2869 H=2869:00 +CPU= 15.898
  02:04:35 STEP 2870 H=2870:00 +CPU= 16.096
  02:04:59 STEP 2871 H=2871:00 +CPU= 23.680
  02:05:15 STEP 2872 H=2872:00 +CPU= 16.320
  02:05:31 STEP 2873 H=2873:00 +CPU= 15.468
  02:05:55 STEP 2874 H=2874:00 +CPU= 24.239
  02:06:11 STEP 2875 H=2875:00 +CPU= 16.009
  02:06:27 STEP 2876 H=2876:00 +CPU= 16.069
  02:06:51 STEP 2877 H=2877:00 +CPU= 23.483
  02:07:07 STEP 2878 H=2878:00 +CPU= 16.170
  02:07:23 STEP 2879 H=2879:00 +CPU= 16.125
  02:07:53 STEP 2880 H=2880:00 +CPU= 27.936
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002880
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002880
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002880
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMGGhq0f+002856
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMSHhq0f+002856
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMUAhq0f+002856
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMGGhq0f+002868
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMSHhq0f+002868
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMUAhq0f+002868
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0897_2004050100_123_973_12190541_0_r1810036735_119.zip
Uploading the intermediate file: upload_file_119.zip
  02:08:08 STEP 2881 H=2881:00 +CPU= 15.745
  02:08:25 STEP 2882 H=2882:00 +CPU= 16.010
  02:08:49 STEP 2883 H=2883:00 +CPU= 23.934
  02:09:05 STEP 2884 H=2884:00 +CPU= 16.355
  02:09:21 STEP 2885 H=2885:00 +CPU= 15.571
  02:09:45 STEP 2886 H=2886:00 +CPU= 24.417
  02:10:01 STEP 2887 H=2887:00 +CPU= 15.881
  02:10:17 STEP 2888 H=2888:00 +CPU= 15.988
  02:10:41 STEP 2889 H=2889:00 +CPU= 23.931
  02:10:57 STEP 2890 H=2890:00 +CPU= 16.433
  02:11:13 STEP 2891 H=2891:00 +CPU= 15.738
  02:11:39 STEP 2892 H=2892:00 +CPU= 25.763
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002892
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002892
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002892
  02:11:54 STEP 2893 H=2893:00 +CPU= 14.712
  02:12:08 STEP 2894 H=2894:00 +CPU= 14.756
  02:12:30 STEP 2895 H=2895:00 +CPU= 22.082
  02:12:46 STEP 2896 H=2896:00 +CPU= 15.446
  02:13:01 STEP 2897 H=2897:00 +CPU= 15.472
  02:13:26 STEP 2898 H=2898:00 +CPU= 24.486
  02:13:42 STEP 2899 H=2899:00 +CPU= 15.845
  02:13:58 STEP 2900 H=2900:00 +CPU= 15.880
  02:14:21 STEP 2901 H=2901:00 +CPU= 23.547
  02:14:36 STEP 2902 H=2902:00 +CPU= 14.832
  02:14:51 STEP 2903 H=2903:00 +CPU= 14.651
  02:15:17 STEP 2904 H=2904:00 +CPU= 26.244
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002904
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002904
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002904
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMGGhq0f+002880
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMSHhq0f+002880
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMUAhq0f+002880
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMGGhq0f+002892
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMSHhq0f+002892
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMUAhq0f+002892
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0897_2004050100_123_973_12190541_0_r1810036735_120.zip
Uploading the intermediate file: upload_file_120.zip
  02:15:32 STEP 2905 H=2905:00 +CPU= 14.640
  02:15:47 STEP 2906 H=2906:00 +CPU= 14.752
  02:16:12 STEP 2907 H=2907:00 +CPU= 23.991
  02:16:29 STEP 2908 H=2908:00 +CPU= 16.407
  02:16:44 STEP 2909 H=2909:00 +CPU= 15.409
  02:17:09 STEP 2910 H=2910:00 +CPU= 24.548
  02:17:25 STEP 2911 H=2911:00 +CPU= 15.940
  02:17:40 STEP 2912 H=2912:00 +CPU= 15.680
  02:18:04 STEP 2913 H=2913:00 +CPU= 23.938
  02:18:21 STEP 2914 H=2914:00 +CPU= 16.391
  02:18:36 STEP 2915 H=2915:00 +CPU= 15.414
  02:19:03 STEP 2916 H=2916:00 +CPU= 26.324
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002916
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002916
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002916
  02:19:19 STEP 2917 H=2917:00 +CPU= 16.055
  02:19:35 STEP 2918 H=2918:00 +CPU= 15.729
  02:19:58 STEP 2919 H=2919:00 +CPU= 23.814
  02:20:15 STEP 2920 H=2920:00 +CPU= 16.423
  02:20:30 STEP 2921 H=2921:00 +CPU= 15.490
  02:20:57 STEP 2922 H=2922:00 +CPU= 23.967
  02:21:12 STEP 2923 H=2923:00 +CPU= 14.978
  02:21:27 STEP 2924 H=2924:00 +CPU= 14.668
  02:21:49 STEP 2925 H=2925:00 +CPU= 22.390
  02:22:05 STEP 2926 H=2926:00 +CPU= 15.681
  02:22:21 STEP 2927 H=2927:00 +CPU= 15.905
  02:22:50 STEP 2928 H=2928:00 +CPU= 27.869
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002928
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002928
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002928
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMGGhq0f+002904
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMSHhq0f+002904
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMUAhq0f+002904
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMGGhq0f+002916
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMSHhq0f+002916
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMUAhq0f+002916
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0897_2004050100_123_973_12190541_0_r1810036735_121.zip
Uploading the intermediate file: upload_file_121.zip
  02:23:06 STEP 2929 H=2929:00 +CPU= 15.882
Uploading trickle at timestep: 10540800
  02:23:22 STEP 2930 H=2930:00 +CPU= 15.716
  02:23:46 STEP 2931 H=2931:00 +CPU= 24.217
  02:24:02 STEP 2932 H=2932:00 +CPU= 16.327
  02:24:18 STEP 2933 H=2933:00 +CPU= 15.334
  02:24:42 STEP 2934 H=2934:00 +CPU= 24.605
  02:24:58 STEP 2935 H=2935:00 +CPU= 15.903
  02:25:14 STEP 2936 H=2936:00 +CPU= 15.836
  02:25:38 STEP 2937 H=2937:00 +CPU= 24.186
  02:25:55 STEP 2938 H=2938:00 +CPU= 16.214
  02:26:10 STEP 2939 H=2939:00 +CPU= 15.207
  02:26:37 STEP 2940 H=2940:00 +CPU= 26.876
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002940
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002940
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002940
  02:26:53 STEP 2941 H=2941:00 +CPU= 15.937
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2004050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0897
(argv4) batchid: 973
(argv5) wuid: 12190541
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/3
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_23dd631530bf9d5e7bf7109e4bec962c to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0897_2004050100_123_973_12190541.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0897_2004050100_123_973_12190541.zip
ic_ancil_file: ic_ancil_12190541
ifsdata_file: ifsdata_12190541
climate_data_file: clim_data_12190541
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_d0a4111b8bc796e1a52d2b880fd02ff1 to: /var/lib/boinc-client/slots/3/ic_ancil_12190541.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/3/ic_ancil_12190541.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_06c87b58b83fb2035499d8cde1315358 to: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12190541.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12190541.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/3/159l_2/clim_data_12190541.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/3/159l_2/clim_data_12190541.zip
Checking for progress XML file: /var/lib/boinc-client/slots/3/progress_file_12190541.xml
Opened progress file ok : /var/lib/boinc-client/slots/3/progress_file_12190541.xml
-- Model is restarting --
Adjusting last_iter, 2942, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/3/progress_file_12190541.xml
last_cpu_time: 51963
upload_file_number: 122
last_iter: 2927
last_upload: 10540800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0897_2004050100_123_973_12190541_0_r1810036735
The child process has been launched with process id: 4155
Executing the command: /var/lib/boinc-client/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:johann:1:1:4155:4155] [20230118:022821:1674005301:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:johann:1:1:4155:4155] [20230118:022821:1674005301:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:johann:1:1:4155:4155] [20230118:022821:1674005301:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:johann:1:1:4155:4155] [20230118:022821:1674005301:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:johann:1:1:4155:4155] [20230118:022821:1674005301:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:johann:1:1:4155:4155] [20230118:022821:1674005301:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:johann:1:1:4155:4155] [20230118:022821:1674005301:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:johann:1:1:4155:4155] [20230118:022821:1674005301:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:johann:1:1:4155:4155] [20230118:022821:1674005301:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:johann:1:1:4155:4155] [20230118:022821:1674005301:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:johann:1:1:4155:4155] [20230118:022821:1674005301:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:johann:1:1:4155:4155] [20230118:022821:1674005301:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:johann:1:1:4155:4155] [20230118:022821:1674005301:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:johann:1:1:4155:4155] [20230118:022821:1674005301:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:johann:1:1:4155:4155] [20230118:022821:1674005301:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:johann:1:1:4155:4155] [20230118:022821:1674005301:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:johann:1:1:4155:4155] [20230118:022821:1674005301:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:johann:1:1:4155:4155] [20230118:022821:1674005301:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:johann:1:1:4155:4155] [20230118:022821:1674005301:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:johann:1:1:4155:4155] [20230118:022821:1674005301:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:johann:1:1:4155:4155] [20230118:022821:1674005301:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:johann:1:1:4155:4155] [20230118:022821:1674005301:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:johann:1:1:4155:4155] [20230118:022821:1674005301:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:johann:1:1:4155:4155] [20230118:022821:1674005301:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:johann:1:1:4155:4155] [20230118:022821:1674005301:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:johann:1:1:4155:4155] [20230118:022821:1674005301:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:johann:1:1:4155:4155] [20230118:022821:1674005301:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:johann:1:1:4155:4155] [20230118:022821:1674005301:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:johann:1:1:4155:4155] [20230118:022821:1674005301:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:johann:1:1:4155:4155] [20230118:022821:1674005301:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  02:28:50 STEP 2928 H=2928:00 +CPU= 27.601
  02:29:06 STEP 2929 H=2929:00 +CPU= 15.793
  02:29:22 STEP 2930 H=2930:00 +CPU= 16.026
  02:29:46 STEP 2931 H=2931:00 +CPU= 23.799
  02:30:01 STEP 2932 H=2932:00 +CPU= 14.964
  02:30:15 STEP 2933 H=2933:00 +CPU= 14.415
  02:30:37 STEP 2934 H=2934:00 +CPU= 22.157
  02:30:52 STEP 2935 H=2935:00 +CPU= 14.680
  02:31:08 STEP 2936 H=2936:00 +CPU= 15.730
  02:31:32 STEP 2937 H=2937:00 +CPU= 24.184
  02:31:48 STEP 2938 H=2938:00 +CPU= 16.395
  02:32:04 STEP 2939 H=2939:00 +CPU= 15.495
  02:32:30 STEP 2940 H=2940:00 +CPU= 26.026
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002940
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002940
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002940
  02:32:45 STEP 2941 H=2941:00 +CPU= 14.769
  02:33:00 STEP 2942 H=2942:00 +CPU= 14.774
  02:33:22 STEP 2943 H=2943:00 +CPU= 22.167
  02:33:37 STEP 2944 H=2944:00 +CPU= 15.337
  02:33:52 STEP 2945 H=2945:00 +CPU= 15.248
  02:34:17 STEP 2946 H=2946:00 +CPU= 24.482
  02:34:33 STEP 2947 H=2947:00 +CPU= 16.126
  02:34:49 STEP 2948 H=2948:00 +CPU= 15.621
  02:35:12 STEP 2949 H=2949:00 +CPU= 23.860
  02:35:29 STEP 2950 H=2950:00 +CPU= 16.244
  02:35:44 STEP 2951 H=2951:00 +CPU= 15.460
  02:36:10 STEP 2952 H=2952:00 +CPU= 26.372
The child process terminated with status: 0
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002952
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002952
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002952
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMUAhq0f+002940
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMGGhq0f+002940
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMSHhq0f+002928
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMSHhq0f+002940
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMSHhq0f+002952
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMUAhq0f+002952
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMGGhq0f+002928
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMGGhq0f+002952
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190541/ICMUAhq0f+002928
Zipping up the final file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0897_2004050100_123_973_12190541_0_r1810036735_122.zip
Uploading the final file: upload_file_122.zip
Uploading trickle at timestep: 10623600
02:39:34 (4152): called boinc_finish(0)

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:25:21 1513103 22276178 oifs_43r3_ps_0897_2004050100_123_973_12190541_0 10,623,600 52,432 0.0049
09 May 2023 16:25:21 1513103 22276178 oifs_43r3_ps_0897_2004050100_123_973_12190541_0 10,540,800 51,720 0.0049
09 May 2023 16:24:44 1513103 22276178 oifs_43r3_ps_0897_2004050100_123_973_12190541_0 8,726,400 41,344 0.0047
09 May 2023 16:23:59 1513103 22276178 oifs_43r3_ps_0897_2004050100_123_973_12190541_0 7,862,400 36,272 0.0046
09 May 2023 16:23:53 1513103 22276178 oifs_43r3_ps_0897_2004050100_123_973_12190541_0 6,998,400 31,903 0.0046
09 May 2023 16:23:48 1513103 22276178 oifs_43r3_ps_0897_2004050100_123_973_12190541_0 6,134,400 28,013 0.0046
09 May 2023 16:23:44 1513103 22276178 oifs_43r3_ps_0897_2004050100_123_973_12190541_0 5,270,400 24,112 0.0046
09 May 2023 16:23:38 1513103 22276178 oifs_43r3_ps_0897_2004050100_123_973_12190541_0 4,406,400 20,215 0.0046
09 May 2023 16:23:34 1513103 22276178 oifs_43r3_ps_0897_2004050100_123_973_12190541_0 3,542,400 16,318 0.0046
09 May 2023 16:23:29 1513103 22276178 oifs_43r3_ps_0897_2004050100_123_973_12190541_0 2,678,400 12,431 0.0046
09 May 2023 16:23:24 1513103 22276178 oifs_43r3_ps_0897_2004050100_123_973_12190541_0 1,814,400 8,454 0.0047
09 May 2023 16:23:20 1513103 22276178 oifs_43r3_ps_0897_2004050100_123_973_12190541_0 950,400 4,243 0.0045


©2024 climateprediction.net