climateprediction.net home page
Task 22317065

Task 22317065

Name oifs_43r3_01gg_2019110100_123_993_12215326_2
Workunit 12215326
Created 25 Feb 2023, 17:20:05 UTC
Sent 25 Feb 2023, 17:20:42 UTC
Report deadline 26 Apr 2023, 17:20:42 UTC
Received 26 Feb 2023, 9:51:21 UTC
Server state Over
Outcome Success
Client state Done
Exit status 0 (0x00000000)
Computer ID 1537623
Run time 15 hours 52 min 32 sec
CPU time 15 hours 52 min 32 sec
Validate state Valid
Credit 14,873.04
Device peak FLOPS 7.68 GFLOPS
Application version OpenIFS 43r3 v1.21
x86_64-pc-linux-gnu
Peak working set size 4,406.41 MB
Peak swap size 4,974.79 MB
Peak disk usage 1,298.41 MB
Stderr
<core_client_version>7.18.1</core_client_version>
<![CDATA[
<stderr_txt>
TEP 2462 H=2462:00 +CPU= 16.014
  07:37:15 STEP 2463 H=2463:00 +CPU= 24.354
  07:37:31 STEP 2464 H=2464:00 +CPU= 15.616
  07:37:46 STEP 2465 H=2465:00 +CPU= 15.364
  07:38:11 STEP 2466 H=2466:00 +CPU= 24.906
  07:38:27 STEP 2467 H=2467:00 +CPU= 15.480
  07:38:43 STEP 2468 H=2468:00 +CPU= 15.837
  07:39:07 STEP 2469 H=2469:00 +CPU= 24.456
  07:39:23 STEP 2470 H=2470:00 +CPU= 15.971
  07:39:39 STEP 2471 H=2471:00 +CPU= 15.438
  07:40:06 STEP 2472 H=2472:00 +CPU= 26.709
Moving to projects directory: /home/galfridian/slots/13/ICMGGhnw4+002472
Moving to projects directory: /home/galfridian/slots/13/ICMSHhnw4+002472
Moving to projects directory: /home/galfridian/slots/13/ICMUAhnw4+002472
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMGGhnw4+002448
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMSHhnw4+002448
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMUAhnw4+002448
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMGGhnw4+002460
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMSHhnw4+002460
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMUAhnw4+002460
Zipping up the intermediate file: /home/galfridian/projects/climateprediction.net/oifs_43r3_01gg_2019110100_123_993_12215326_2_r248070650_102.zip
Uploading the intermediate file: upload_file_102.zip
  07:40:21 STEP 2473 H=2473:00 +CPU= 15.145
  07:40:37 STEP 2474 H=2474:00 +CPU= 15.861
  07:41:01 STEP 2475 H=2475:00 +CPU= 24.174
  07:41:16 STEP 2476 H=2476:00 +CPU= 15.379
  07:41:31 STEP 2477 H=2477:00 +CPU= 15.175
  07:41:55 STEP 2478 H=2478:00 +CPU= 23.302
  07:42:08 STEP 2479 H=2479:00 +CPU= 13.120
  07:42:24 STEP 2480 H=2480:00 +CPU= 15.797
  07:42:47 STEP 2481 H=2481:00 +CPU= 23.541
  07:43:03 STEP 2482 H=2482:00 +CPU= 15.147
  07:43:17 STEP 2483 H=2483:00 +CPU= 14.171
  07:43:44 STEP 2484 H=2484:00 +CPU= 26.784
Moving to projects directory: /home/galfridian/slots/13/ICMGGhnw4+002484
Moving to projects directory: /home/galfridian/slots/13/ICMSHhnw4+002484
Moving to projects directory: /home/galfridian/slots/13/ICMUAhnw4+002484
  07:43:59 STEP 2485 H=2485:00 +CPU= 15.542
  07:44:15 STEP 2486 H=2486:00 +CPU= 16.198
  07:44:40 STEP 2487 H=2487:00 +CPU= 24.647
  07:44:56 STEP 2488 H=2488:00 +CPU= 16.068
  07:45:12 STEP 2489 H=2489:00 +CPU= 15.737
  07:45:37 STEP 2490 H=2490:00 +CPU= 24.425
  07:45:52 STEP 2491 H=2491:00 +CPU= 15.524
  07:46:08 STEP 2492 H=2492:00 +CPU= 16.175
  07:46:33 STEP 2493 H=2493:00 +CPU= 24.345
  07:46:49 STEP 2494 H=2494:00 +CPU= 15.876
  07:47:04 STEP 2495 H=2495:00 +CPU= 15.737
  07:47:33 STEP 2496 H=2496:00 +CPU= 28.553
Moving to projects directory: /home/galfridian/slots/13/ICMGGhnw4+002496
Moving to projects directory: /home/galfridian/slots/13/ICMSHhnw4+002496
Moving to projects directory: /home/galfridian/slots/13/ICMUAhnw4+002496
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMGGhnw4+002472
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMSHhnw4+002472
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMUAhnw4+002472
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMGGhnw4+002484
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMSHhnw4+002484
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMUAhnw4+002484
Zipping up the intermediate file: /home/galfridian/projects/climateprediction.net/oifs_43r3_01gg_2019110100_123_993_12215326_2_r248070650_103.zip
Uploading the intermediate file: upload_file_103.zip
  07:47:48 STEP 2497 H=2497:00 +CPU= 15.327
  07:48:04 STEP 2498 H=2498:00 +CPU= 15.613
  07:48:28 STEP 2499 H=2499:00 +CPU= 24.216
  07:48:44 STEP 2500 H=2500:00 +CPU= 15.999
  07:49:00 STEP 2501 H=2501:00 +CPU= 15.436
  07:49:25 STEP 2502 H=2502:00 +CPU= 25.204
  07:49:40 STEP 2503 H=2503:00 +CPU= 15.282
  07:49:56 STEP 2504 H=2504:00 +CPU= 15.654
  07:50:20 STEP 2505 H=2505:00 +CPU= 24.203
  07:50:36 STEP 2506 H=2506:00 +CPU= 15.741
  07:50:52 STEP 2507 H=2507:00 +CPU= 15.436
  07:51:18 STEP 2508 H=2508:00 +CPU= 26.590
Moving to projects directory: /home/galfridian/slots/13/ICMGGhnw4+002508
Moving to projects directory: /home/galfridian/slots/13/ICMSHhnw4+002508
Moving to projects directory: /home/galfridian/slots/13/ICMUAhnw4+002508
  07:51:34 STEP 2509 H=2509:00 +CPU= 15.487
  07:51:50 STEP 2510 H=2510:00 +CPU= 15.835
  07:52:14 STEP 2511 H=2511:00 +CPU= 24.383
  07:52:30 STEP 2512 H=2512:00 +CPU= 15.831
  07:52:45 STEP 2513 H=2513:00 +CPU= 14.655
  07:53:09 STEP 2514 H=2514:00 +CPU= 24.422
  07:53:25 STEP 2515 H=2515:00 +CPU= 15.383
  07:53:40 STEP 2516 H=2516:00 +CPU= 15.311
  07:54:03 STEP 2517 H=2517:00 +CPU= 23.288
  07:54:19 STEP 2518 H=2518:00 +CPU= 15.341
  07:54:35 STEP 2519 H=2519:00 +CPU= 16.295
  07:55:04 STEP 2520 H=2520:00 +CPU= 28.822
Moving to projects directory: /home/galfridian/slots/13/ICMGGhnw4+002520
Moving to projects directory: /home/galfridian/slots/13/ICMSHhnw4+002520
Moving to projects directory: /home/galfridian/slots/13/ICMUAhnw4+002520
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMGGhnw4+002496
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMSHhnw4+002496
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMUAhnw4+002496
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMGGhnw4+002508
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMSHhnw4+002508
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMUAhnw4+002508
Zipping up the intermediate file: /home/galfridian/projects/climateprediction.net/oifs_43r3_01gg_2019110100_123_993_12215326_2_r248070650_104.zip
Uploading the intermediate file: upload_file_104.zip
  07:55:19 STEP 2521 H=2521:00 +CPU= 15.366
  07:55:35 STEP 2522 H=2522:00 +CPU= 15.922
  07:56:00 STEP 2523 H=2523:00 +CPU= 24.700
  07:56:16 STEP 2524 H=2524:00 +CPU= 15.889
  07:56:32 STEP 2525 H=2525:00 +CPU= 15.557
  07:56:56 STEP 2526 H=2526:00 +CPU= 24.766
  07:57:12 STEP 2527 H=2527:00 +CPU= 15.426
  07:57:28 STEP 2528 H=2528:00 +CPU= 15.921
  07:57:52 STEP 2529 H=2529:00 +CPU= 24.228
  07:58:07 STEP 2530 H=2530:00 +CPU= 15.045
  07:58:22 STEP 2531 H=2531:00 +CPU= 14.696
  07:58:47 STEP 2532 H=2532:00 +CPU= 25.046
Moving to projects directory: /home/galfridian/slots/13/ICMGGhnw4+002532
Moving to projects directory: /home/galfridian/slots/13/ICMSHhnw4+002532
Moving to projects directory: /home/galfridian/slots/13/ICMUAhnw4+002532
  07:59:03 STEP 2533 H=2533:00 +CPU= 15.734
  07:59:19 STEP 2534 H=2534:00 +CPU= 15.939
  07:59:43 STEP 2535 H=2535:00 +CPU= 24.330
  07:59:59 STEP 2536 H=2536:00 +CPU= 15.964
  08:00:14 STEP 2537 H=2537:00 +CPU= 15.427
  08:00:40 STEP 2538 H=2538:00 +CPU= 24.959
  08:00:55 STEP 2539 H=2539:00 +CPU= 15.449
  08:01:11 STEP 2540 H=2540:00 +CPU= 15.901
  08:01:35 STEP 2541 H=2541:00 +CPU= 24.267
  08:01:51 STEP 2542 H=2542:00 +CPU= 15.964
  08:02:07 STEP 2543 H=2543:00 +CPU= 16.090
  08:02:36 STEP 2544 H=2544:00 +CPU= 28.887
Moving to projects directory: /home/galfridian/slots/13/ICMGGhnw4+002544
Moving to projects directory: /home/galfridian/slots/13/ICMSHhnw4+002544
Moving to projects directory: /home/galfridian/slots/13/ICMUAhnw4+002544
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMGGhnw4+002520
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMSHhnw4+002520
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMUAhnw4+002520
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMGGhnw4+002532
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMSHhnw4+002532
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMUAhnw4+002532
Zipping up the intermediate file: /home/galfridian/projects/climateprediction.net/oifs_43r3_01gg_2019110100_123_993_12215326_2_r248070650_105.zip
Uploading the intermediate file: upload_file_105.zip
  08:02:52 STEP 2545 H=2545:00 +CPU= 15.660
Uploading trickle at timestep: 9158400
  08:03:08 STEP 2546 H=2546:00 +CPU= 16.108
  08:03:32 STEP 2547 H=2547:00 +CPU= 23.315
  08:03:48 STEP 2548 H=2548:00 +CPU= 16.092
  08:04:03 STEP 2549 H=2549:00 +CPU= 15.591
  08:04:27 STEP 2550 H=2550:00 +CPU= 23.774
  08:04:42 STEP 2551 H=2551:00 +CPU= 15.152
  08:04:57 STEP 2552 H=2552:00 +CPU= 15.087
  08:05:20 STEP 2553 H=2553:00 +CPU= 22.878
  08:05:36 STEP 2554 H=2554:00 +CPU= 15.792
  08:05:51 STEP 2555 H=2555:00 +CPU= 14.985
  08:06:17 STEP 2556 H=2556:00 +CPU= 25.561
Moving to projects directory: /home/galfridian/slots/13/ICMGGhnw4+002556
Moving to projects directory: /home/galfridian/slots/13/ICMSHhnw4+002556
Moving to projects directory: /home/galfridian/slots/13/ICMUAhnw4+002556
  08:06:32 STEP 2557 H=2557:00 +CPU= 14.887
  08:06:47 STEP 2558 H=2558:00 +CPU= 15.140
  08:07:11 STEP 2559 H=2559:00 +CPU= 23.680
  08:07:26 STEP 2560 H=2560:00 +CPU= 15.629
  08:07:42 STEP 2561 H=2561:00 +CPU= 15.213
  08:08:07 STEP 2562 H=2562:00 +CPU= 24.868
  08:08:22 STEP 2563 H=2563:00 +CPU= 15.742
  08:08:39 STEP 2564 H=2564:00 +CPU= 16.144
  08:09:03 STEP 2565 H=2565:00 +CPU= 24.385
  08:09:19 STEP 2566 H=2566:00 +CPU= 15.794
  08:09:35 STEP 2567 H=2567:00 +CPU= 15.893
  08:10:03 STEP 2568 H=2568:00 +CPU= 28.433
Moving to projects directory: /home/galfridian/slots/13/ICMGGhnw4+002568
Moving to projects directory: /home/galfridian/slots/13/ICMSHhnw4+002568
Moving to projects directory: /home/galfridian/slots/13/ICMUAhnw4+002568
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMGGhnw4+002544
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMSHhnw4+002544
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMUAhnw4+002544
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMGGhnw4+002556
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMSHhnw4+002556
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMUAhnw4+002556
Zipping up the intermediate file: /home/galfridian/projects/climateprediction.net/oifs_43r3_01gg_2019110100_123_993_12215326_2_r248070650_106.zip
Uploading the intermediate file: upload_file_106.zip
  08:10:18 STEP 2569 H=2569:00 +CPU= 14.785
  08:10:34 STEP 2570 H=2570:00 +CPU= 15.621
  08:10:58 STEP 2571 H=2571:00 +CPU= 24.309
  08:11:14 STEP 2572 H=2572:00 +CPU= 15.827
  08:11:28 STEP 2573 H=2573:00 +CPU= 14.330
  08:11:52 STEP 2574 H=2574:00 +CPU= 24.111
  08:12:08 STEP 2575 H=2575:00 +CPU= 15.385
  08:12:24 STEP 2576 H=2576:00 +CPU= 15.816
  08:12:46 STEP 2577 H=2577:00 +CPU= 22.428
  08:13:00 STEP 2578 H=2578:00 +CPU= 14.101
  08:13:15 STEP 2579 H=2579:00 +CPU= 14.901
  08:13:39 STEP 2580 H=2580:00 +CPU= 24.126
Moving to projects directory: /home/galfridian/slots/13/ICMGGhnw4+002580
Moving to projects directory: /home/galfridian/slots/13/ICMSHhnw4+002580
Moving to projects directory: /home/galfridian/slots/13/ICMUAhnw4+002580
  08:13:54 STEP 2581 H=2581:00 +CPU= 14.589
  08:14:10 STEP 2582 H=2582:00 +CPU= 15.373
  08:14:32 STEP 2583 H=2583:00 +CPU= 22.495
  08:14:48 STEP 2584 H=2584:00 +CPU= 15.481
  08:15:03 STEP 2585 H=2585:00 +CPU= 15.133
  08:15:27 STEP 2586 H=2586:00 +CPU= 24.641
  08:15:43 STEP 2587 H=2587:00 +CPU= 15.716
  08:15:59 STEP 2588 H=2588:00 +CPU= 16.014
  08:16:23 STEP 2589 H=2589:00 +CPU= 24.191
  08:16:38 STEP 2590 H=2590:00 +CPU= 14.104
  08:16:51 STEP 2591 H=2591:00 +CPU= 13.829
  08:17:20 STEP 2592 H=2592:00 +CPU= 28.139
Moving to projects directory: /home/galfridian/slots/13/ICMGGhnw4+002592
Moving to projects directory: /home/galfridian/slots/13/ICMSHhnw4+002592
Moving to projects directory: /home/galfridian/slots/13/ICMUAhnw4+002592
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMGGhnw4+002568
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMSHhnw4+002568
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMUAhnw4+002568
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMGGhnw4+002580
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMSHhnw4+002580
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMUAhnw4+002580
Zipping up the intermediate file: /home/galfridian/projects/climateprediction.net/oifs_43r3_01gg_2019110100_123_993_12215326_2_r248070650_107.zip
Uploading the intermediate file: upload_file_107.zip
  08:17:34 STEP 2593 H=2593:00 +CPU= 14.724
  08:17:49 STEP 2594 H=2594:00 +CPU= 14.130
  08:18:11 STEP 2595 H=2595:00 +CPU= 22.230
  08:18:26 STEP 2596 H=2596:00 +CPU= 15.444
  08:18:41 STEP 2597 H=2597:00 +CPU= 14.187
  08:19:05 STEP 2598 H=2598:00 +CPU= 24.649
  08:19:21 STEP 2599 H=2599:00 +CPU= 15.391
  08:19:36 STEP 2600 H=2600:00 +CPU= 15.682
  08:20:01 STEP 2601 H=2601:00 +CPU= 24.397
  08:20:17 STEP 2602 H=2602:00 +CPU= 15.853
  08:20:31 STEP 2603 H=2603:00 +CPU= 14.089
  08:20:56 STEP 2604 H=2604:00 +CPU= 25.290
Moving to projects directory: /home/galfridian/slots/13/ICMGGhnw4+002604
Moving to projects directory: /home/galfridian/slots/13/ICMSHhnw4+002604
Moving to projects directory: /home/galfridian/slots/13/ICMUAhnw4+002604
  08:21:10 STEP 2605 H=2605:00 +CPU= 14.090
  08:21:26 STEP 2606 H=2606:00 +CPU= 15.614
  08:21:49 STEP 2607 H=2607:00 +CPU= 22.937
  08:22:04 STEP 2608 H=2608:00 +CPU= 14.711
  08:22:19 STEP 2609 H=2609:00 +CPU= 14.982
  08:22:43 STEP 2610 H=2610:00 +CPU= 23.857
  08:22:58 STEP 2611 H=2611:00 +CPU= 15.149
  08:23:14 STEP 2612 H=2612:00 +CPU= 15.803
  08:23:37 STEP 2613 H=2613:00 +CPU= 23.448
  08:23:53 STEP 2614 H=2614:00 +CPU= 15.616
  08:24:07 STEP 2615 H=2615:00 +CPU= 14.172
  08:24:36 STEP 2616 H=2616:00 +CPU= 28.653
Moving to projects directory: /home/galfridian/slots/13/ICMGGhnw4+002616
Moving to projects directory: /home/galfridian/slots/13/ICMSHhnw4+002616
Moving to projects directory: /home/galfridian/slots/13/ICMUAhnw4+002616
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMGGhnw4+002592
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMSHhnw4+002592
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMUAhnw4+002592
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMGGhnw4+002604
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMSHhnw4+002604
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMUAhnw4+002604
Zipping up the intermediate file: /home/galfridian/projects/climateprediction.net/oifs_43r3_01gg_2019110100_123_993_12215326_2_r248070650_108.zip
Uploading the intermediate file: upload_file_108.zip
  08:24:51 STEP 2617 H=2617:00 +CPU= 15.403
  08:25:07 STEP 2618 H=2618:00 +CPU= 15.929
  08:25:31 STEP 2619 H=2619:00 +CPU= 24.072
  08:25:47 STEP 2620 H=2620:00 +CPU= 15.862
  08:26:03 STEP 2621 H=2621:00 +CPU= 15.662
  08:26:28 STEP 2622 H=2622:00 +CPU= 24.789
  08:26:43 STEP 2623 H=2623:00 +CPU= 15.315
  08:26:59 STEP 2624 H=2624:00 +CPU= 15.957
  08:27:23 STEP 2625 H=2625:00 +CPU= 23.609
  08:27:38 STEP 2626 H=2626:00 +CPU= 15.461
  08:27:54 STEP 2627 H=2627:00 +CPU= 15.427
  08:28:20 STEP 2628 H=2628:00 +CPU= 26.658
Moving to projects directory: /home/galfridian/slots/13/ICMGGhnw4+002628
Moving to projects directory: /home/galfridian/slots/13/ICMSHhnw4+002628
Moving to projects directory: /home/galfridian/slots/13/ICMUAhnw4+002628
  08:28:36 STEP 2629 H=2629:00 +CPU= 15.506
  08:28:52 STEP 2630 H=2630:00 +CPU= 16.063
  08:29:15 STEP 2631 H=2631:00 +CPU= 23.501
  08:29:32 STEP 2632 H=2632:00 +CPU= 16.103
  08:29:47 STEP 2633 H=2633:00 +CPU= 15.567
  08:30:12 STEP 2634 H=2634:00 +CPU= 24.704
  08:30:28 STEP 2635 H=2635:00 +CPU= 15.595
  08:30:42 STEP 2636 H=2636:00 +CPU= 14.676
  08:31:06 STEP 2637 H=2637:00 +CPU= 24.034
  08:31:22 STEP 2638 H=2638:00 +CPU= 15.983
  08:31:38 STEP 2639 H=2639:00 +CPU= 15.326
  08:32:06 STEP 2640 H=2640:00 +CPU= 28.005
Moving to projects directory: /home/galfridian/slots/13/ICMGGhnw4+002640
Moving to projects directory: /home/galfridian/slots/13/ICMSHhnw4+002640
Moving to projects directory: /home/galfridian/slots/13/ICMUAhnw4+002640
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMGGhnw4+002616
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMSHhnw4+002616
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMUAhnw4+002616
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMGGhnw4+002628
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMSHhnw4+002628
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMUAhnw4+002628
Zipping up the intermediate file: /home/galfridian/projects/climateprediction.net/oifs_43r3_01gg_2019110100_123_993_12215326_2_r248070650_109.zip
Uploading the intermediate file: upload_file_109.zip
  08:32:21 STEP 2641 H=2641:00 +CPU= 15.422
  08:32:37 STEP 2642 H=2642:00 +CPU= 15.800
  08:33:01 STEP 2643 H=2643:00 +CPU= 24.247
  08:33:18 STEP 2644 H=2644:00 +CPU= 16.097
  08:33:33 STEP 2645 H=2645:00 +CPU= 15.250
  08:33:58 STEP 2646 H=2646:00 +CPU= 24.916
  08:34:13 STEP 2647 H=2647:00 +CPU= 15.478
  08:34:30 STEP 2648 H=2648:00 +CPU= 16.249
  08:34:54 STEP 2649 H=2649:00 +CPU= 24.150
  08:35:09 STEP 2650 H=2650:00 +CPU= 14.802
  08:35:24 STEP 2651 H=2651:00 +CPU= 14.961
  08:35:49 STEP 2652 H=2652:00 +CPU= 25.478
Moving to projects directory: /home/galfridian/slots/13/ICMGGhnw4+002652
Moving to projects directory: /home/galfridian/slots/13/ICMSHhnw4+002652
Moving to projects directory: /home/galfridian/slots/13/ICMUAhnw4+002652
  08:36:04 STEP 2653 H=2653:00 +CPU= 14.836
  08:36:18 STEP 2654 H=2654:00 +CPU= 14.428
  08:36:42 STEP 2655 H=2655:00 +CPU= 23.885
  08:36:59 STEP 2656 H=2656:00 +CPU= 16.242
  08:37:14 STEP 2657 H=2657:00 +CPU= 15.661
  08:37:39 STEP 2658 H=2658:00 +CPU= 24.250
  08:37:53 STEP 2659 H=2659:00 +CPU= 14.420
  08:38:09 STEP 2660 H=2660:00 +CPU= 15.831
  08:38:33 STEP 2661 H=2661:00 +CPU= 23.844
  08:38:48 STEP 2662 H=2662:00 +CPU= 15.233
  08:39:04 STEP 2663 H=2663:00 +CPU= 15.711
  08:39:31 STEP 2664 H=2664:00 +CPU= 27.344
Moving to projects directory: /home/galfridian/slots/13/ICMGGhnw4+002664
Moving to projects directory: /home/galfridian/slots/13/ICMSHhnw4+002664
Moving to projects directory: /home/galfridian/slots/13/ICMUAhnw4+002664
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMGGhnw4+002640
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMSHhnw4+002640
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMUAhnw4+002640
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMGGhnw4+002652
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMSHhnw4+002652
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMUAhnw4+002652
Zipping up the intermediate file: /home/galfridian/projects/climateprediction.net/oifs_43r3_01gg_2019110100_123_993_12215326_2_r248070650_110.zip
Uploading the intermediate file: upload_file_110.zip
  08:39:47 STEP 2665 H=2665:00 +CPU= 15.625
  08:40:03 STEP 2666 H=2666:00 +CPU= 16.032
  08:40:26 STEP 2667 H=2667:00 +CPU= 23.254
  08:40:42 STEP 2668 H=2668:00 +CPU= 15.677
  08:40:58 STEP 2669 H=2669:00 +CPU= 15.559
  08:41:22 STEP 2670 H=2670:00 +CPU= 24.363
  08:41:38 STEP 2671 H=2671:00 +CPU= 15.511
  08:41:54 STEP 2672 H=2672:00 +CPU= 16.196
  08:42:18 STEP 2673 H=2673:00 +CPU= 23.704
  08:42:33 STEP 2674 H=2674:00 +CPU= 14.775
  08:42:48 STEP 2675 H=2675:00 +CPU= 15.128
  08:43:13 STEP 2676 H=2676:00 +CPU= 25.297
Moving to projects directory: /home/galfridian/slots/13/ICMGGhnw4+002676
Moving to projects directory: /home/galfridian/slots/13/ICMSHhnw4+002676
Moving to projects directory: /home/galfridian/slots/13/ICMUAhnw4+002676
  08:43:29 STEP 2677 H=2677:00 +CPU= 15.652
  08:43:44 STEP 2678 H=2678:00 +CPU= 14.719
  08:44:07 STEP 2679 H=2679:00 +CPU= 23.414
  08:44:23 STEP 2680 H=2680:00 +CPU= 15.597
  08:44:38 STEP 2681 H=2681:00 +CPU= 14.886
  08:45:01 STEP 2682 H=2682:00 +CPU= 23.107
 SMILAG TRAJECTORY UNDERGROUND      1 TIMES.
  POINT=            5  MAX ETADOT VERTICAL VEL.=   3.122091154157448E-008
  LON  =    141.250000000000       degrees
  LAT  =    35.3269057618723       degrees
  08:45:15 STEP 2683 H=2683:00 +CPU= 14.527
  08:45:31 STEP 2684 H=2684:00 +CPU= 15.265
  08:45:54 STEP 2685 H=2685:00 +CPU= 23.574
 SMILAG TRAJECTORY UNDERGROUND      1 TIMES.
  POINT=            6  MAX ETADOT VERTICAL VEL.=   1.419418394801749E-007
  LON  =    142.500000000000       degrees
  LAT  =    36.4483927977945       degrees
  08:46:10 STEP 2686 H=2686:00 +CPU= 15.521
  08:46:25 STEP 2687 H=2687:00 +CPU= 14.614
  08:46:51 STEP 2688 H=2688:00 +CPU= 26.483
Moving to projects directory: /home/galfridian/slots/13/ICMGGhnw4+002688
Moving to projects directory: /home/galfridian/slots/13/ICMSHhnw4+002688
Moving to projects directory: /home/galfridian/slots/13/ICMUAhnw4+002688
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMGGhnw4+002664
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMSHhnw4+002664
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMUAhnw4+002664
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMGGhnw4+002676
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMSHhnw4+002676
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMUAhnw4+002676
Zipping up the intermediate file: /home/galfridian/projects/climateprediction.net/oifs_43r3_01gg_2019110100_123_993_12215326_2_r248070650_111.zip
Uploading the intermediate file: upload_file_111.zip
  08:47:04 STEP 2689 H=2689:00 +CPU= 12.968
  08:47:18 STEP 2690 H=2690:00 +CPU= 13.719
  08:47:41 STEP 2691 H=2691:00 +CPU= 23.018
  08:47:56 STEP 2692 H=2692:00 +CPU= 15.215
  08:48:10 STEP 2693 H=2693:00 +CPU= 14.015
  08:48:32 STEP 2694 H=2694:00 +CPU= 21.533
  08:48:46 STEP 2695 H=2695:00 +CPU= 14.190
  08:49:01 STEP 2696 H=2696:00 +CPU= 14.365
  08:49:25 STEP 2697 H=2697:00 +CPU= 23.852
  08:49:40 STEP 2698 H=2698:00 +CPU= 15.562
  08:49:55 STEP 2699 H=2699:00 +CPU= 15.049
  08:50:19 STEP 2700 H=2700:00 +CPU= 23.463
Moving to projects directory: /home/galfridian/slots/13/ICMGGhnw4+002700
Moving to projects directory: /home/galfridian/slots/13/ICMSHhnw4+002700
Moving to projects directory: /home/galfridian/slots/13/ICMUAhnw4+002700
  08:50:32 STEP 2701 H=2701:00 +CPU= 13.425
  08:50:46 STEP 2702 H=2702:00 +CPU= 13.321
  08:51:08 STEP 2703 H=2703:00 +CPU= 21.955
  08:51:23 STEP 2704 H=2704:00 +CPU= 14.855
  08:51:37 STEP 2705 H=2705:00 +CPU= 14.032
  08:52:00 STEP 2706 H=2706:00 +CPU= 22.765
  08:52:14 STEP 2707 H=2707:00 +CPU= 13.982
  08:52:29 STEP 2708 H=2708:00 +CPU= 15.337
  08:52:51 STEP 2709 H=2709:00 +CPU= 21.890
  08:53:05 STEP 2710 H=2710:00 +CPU= 14.184
  08:53:20 STEP 2711 H=2711:00 +CPU= 14.825
  08:53:46 STEP 2712 H=2712:00 +CPU= 26.036
Moving to projects directory: /home/galfridian/slots/13/ICMGGhnw4+002712
Moving to projects directory: /home/galfridian/slots/13/ICMSHhnw4+002712
Moving to projects directory: /home/galfridian/slots/13/ICMUAhnw4+002712
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMGGhnw4+002688
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMSHhnw4+002688
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMUAhnw4+002688
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMGGhnw4+002700
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMSHhnw4+002700
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMUAhnw4+002700
Zipping up the intermediate file: /home/galfridian/projects/climateprediction.net/oifs_43r3_01gg_2019110100_123_993_12215326_2_r248070650_112.zip
Uploading the intermediate file: upload_file_112.zip
  08:54:01 STEP 2713 H=2713:00 +CPU= 14.423
  08:54:15 STEP 2714 H=2714:00 +CPU= 14.685
  08:54:39 STEP 2715 H=2715:00 +CPU= 23.609
  08:54:52 STEP 2716 H=2716:00 +CPU= 12.803
  08:55:06 STEP 2717 H=2717:00 +CPU= 13.844
  08:55:28 STEP 2718 H=2718:00 +CPU= 22.439
  08:55:42 STEP 2719 H=2719:00 +CPU= 13.762
  08:55:56 STEP 2720 H=2720:00 +CPU= 13.820
  08:56:18 STEP 2721 H=2721:00 +CPU= 22.384
  08:56:34 STEP 2722 H=2722:00 +CPU= 15.099
  08:56:49 STEP 2723 H=2723:00 +CPU= 14.935
  08:57:15 STEP 2724 H=2724:00 +CPU= 25.891
Moving to projects directory: /home/galfridian/slots/13/ICMGGhnw4+002724
Moving to projects directory: /home/galfridian/slots/13/ICMSHhnw4+002724
Moving to projects directory: /home/galfridian/slots/13/ICMUAhnw4+002724
  08:57:29 STEP 2725 H=2725:00 +CPU= 14.710
  08:57:44 STEP 2726 H=2726:00 +CPU= 14.373
  08:58:06 STEP 2727 H=2727:00 +CPU= 22.366
  08:58:21 STEP 2728 H=2728:00 +CPU= 15.337
  08:58:36 STEP 2729 H=2729:00 +CPU= 14.203
  08:58:58 STEP 2730 H=2730:00 +CPU= 22.041
  08:59:13 STEP 2731 H=2731:00 +CPU= 15.021
  08:59:28 STEP 2732 H=2732:00 +CPU= 15.007
  08:59:50 STEP 2733 H=2733:00 +CPU= 21.696
  09:00:04 STEP 2734 H=2734:00 +CPU= 14.392
  09:00:18 STEP 2735 H=2735:00 +CPU= 13.932
  09:00:45 STEP 2736 H=2736:00 +CPU= 26.386
Moving to projects directory: /home/galfridian/slots/13/ICMGGhnw4+002736
Moving to projects directory: /home/galfridian/slots/13/ICMSHhnw4+002736
Moving to projects directory: /home/galfridian/slots/13/ICMUAhnw4+002736
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMGGhnw4+002712
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMSHhnw4+002712
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMUAhnw4+002712
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMGGhnw4+002724
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMSHhnw4+002724
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMUAhnw4+002724
Zipping up the intermediate file: /home/galfridian/projects/climateprediction.net/oifs_43r3_01gg_2019110100_123_993_12215326_2_r248070650_113.zip
Uploading the intermediate file: upload_file_113.zip
  09:00:59 STEP 2737 H=2737:00 +CPU= 14.566
  09:01:14 STEP 2738 H=2738:00 +CPU= 15.138
  09:01:37 STEP 2739 H=2739:00 +CPU= 22.495
  09:01:52 STEP 2740 H=2740:00 +CPU= 15.085
  09:02:07 STEP 2741 H=2741:00 +CPU= 14.856
  09:02:31 STEP 2742 H=2742:00 +CPU= 24.106
  09:02:46 STEP 2743 H=2743:00 +CPU= 15.054
  09:03:01 STEP 2744 H=2744:00 +CPU= 15.137
  09:03:23 STEP 2745 H=2745:00 +CPU= 21.833
  09:03:39 STEP 2746 H=2746:00 +CPU= 15.525
  09:03:54 STEP 2747 H=2747:00 +CPU= 15.217
  09:04:20 STEP 2748 H=2748:00 +CPU= 25.947
Moving to projects directory: /home/galfridian/slots/13/ICMGGhnw4+002748
Moving to projects directory: /home/galfridian/slots/13/ICMSHhnw4+002748
Moving to projects directory: /home/galfridian/slots/13/ICMUAhnw4+002748
  09:04:34 STEP 2749 H=2749:00 +CPU= 14.162
  09:04:48 STEP 2750 H=2750:00 +CPU= 13.967
  09:05:11 STEP 2751 H=2751:00 +CPU= 22.905
  09:05:26 STEP 2752 H=2752:00 +CPU= 14.663
  09:05:41 STEP 2753 H=2753:00 +CPU= 15.132
  09:06:06 STEP 2754 H=2754:00 +CPU= 24.422
  09:06:21 STEP 2755 H=2755:00 +CPU= 15.048
  09:06:36 STEP 2756 H=2756:00 +CPU= 15.283
  09:07:00 STEP 2757 H=2757:00 +CPU= 23.886
  09:07:16 STEP 2758 H=2758:00 +CPU= 15.398
  09:07:30 STEP 2759 H=2759:00 +CPU= 14.618
  09:07:58 STEP 2760 H=2760:00 +CPU= 27.404
Moving to projects directory: /home/galfridian/slots/13/ICMGGhnw4+002760
Moving to projects directory: /home/galfridian/slots/13/ICMSHhnw4+002760
Moving to projects directory: /home/galfridian/slots/13/ICMUAhnw4+002760
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMGGhnw4+002736
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMSHhnw4+002736
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMUAhnw4+002736
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMGGhnw4+002748
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMSHhnw4+002748
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMUAhnw4+002748
Zipping up the intermediate file: /home/galfridian/projects/climateprediction.net/oifs_43r3_01gg_2019110100_123_993_12215326_2_r248070650_114.zip
Uploading the intermediate file: upload_file_114.zip
  09:08:13 STEP 2761 H=2761:00 +CPU= 15.049
  09:08:28 STEP 2762 H=2762:00 +CPU= 15.159
  09:08:52 STEP 2763 H=2763:00 +CPU= 23.961
  09:09:08 STEP 2764 H=2764:00 +CPU= 15.847
  09:09:24 STEP 2765 H=2765:00 +CPU= 15.595
  09:09:48 STEP 2766 H=2766:00 +CPU= 24.174
  09:10:01 STEP 2767 H=2767:00 +CPU= 13.533
  09:10:17 STEP 2768 H=2768:00 +CPU= 15.723
  09:10:40 STEP 2769 H=2769:00 +CPU= 23.211
  09:10:55 STEP 2770 H=2770:00 +CPU= 14.726
  09:11:09 STEP 2771 H=2771:00 +CPU= 14.107
  09:11:36 STEP 2772 H=2772:00 +CPU= 26.135
Moving to projects directory: /home/galfridian/slots/13/ICMGGhnw4+002772
Moving to projects directory: /home/galfridian/slots/13/ICMSHhnw4+002772
Moving to projects directory: /home/galfridian/slots/13/ICMUAhnw4+002772
  09:11:50 STEP 2773 H=2773:00 +CPU= 14.671
  09:12:06 STEP 2774 H=2774:00 +CPU= 15.448
  09:12:30 STEP 2775 H=2775:00 +CPU= 24.074
  09:12:46 STEP 2776 H=2776:00 +CPU= 15.613
  09:13:01 STEP 2777 H=2777:00 +CPU= 14.892
  09:13:24 STEP 2778 H=2778:00 +CPU= 23.834
  09:13:40 STEP 2779 H=2779:00 +CPU= 15.124
  09:13:55 STEP 2780 H=2780:00 +CPU= 15.170
  09:14:17 STEP 2781 H=2781:00 +CPU= 22.369
  09:14:33 STEP 2782 H=2782:00 +CPU= 15.482
  09:14:48 STEP 2783 H=2783:00 +CPU= 15.437
  09:15:14 STEP 2784 H=2784:00 +CPU= 26.121
Moving to projects directory: /home/galfridian/slots/13/ICMGGhnw4+002784
Moving to projects directory: /home/galfridian/slots/13/ICMSHhnw4+002784
Moving to projects directory: /home/galfridian/slots/13/ICMUAhnw4+002784
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMGGhnw4+002760
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMSHhnw4+002760
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMUAhnw4+002760
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMGGhnw4+002772
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMSHhnw4+002772
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMUAhnw4+002772
Zipping up the intermediate file: /home/galfridian/projects/climateprediction.net/oifs_43r3_01gg_2019110100_123_993_12215326_2_r248070650_115.zip
Uploading the intermediate file: upload_file_115.zip
  09:15:28 STEP 2785 H=2785:00 +CPU= 13.945
Uploading trickle at timestep: 10022400
  09:15:43 STEP 2786 H=2786:00 +CPU= 14.938
  09:16:07 STEP 2787 H=2787:00 +CPU= 23.661
  09:16:23 STEP 2788 H=2788:00 +CPU= 15.391
  09:16:38 STEP 2789 H=2789:00 +CPU= 15.057
  09:17:01 STEP 2790 H=2790:00 +CPU= 22.861
  09:17:15 STEP 2791 H=2791:00 +CPU= 14.056
  09:17:31 STEP 2792 H=2792:00 +CPU= 15.801
  09:17:54 STEP 2793 H=2793:00 +CPU= 23.703
  09:18:10 STEP 2794 H=2794:00 +CPU= 15.522
  09:18:25 STEP 2795 H=2795:00 +CPU= 15.225
  09:18:52 STEP 2796 H=2796:00 +CPU= 26.179
Moving to projects directory: /home/galfridian/slots/13/ICMGGhnw4+002796
Moving to projects directory: /home/galfridian/slots/13/ICMSHhnw4+002796
Moving to projects directory: /home/galfridian/slots/13/ICMUAhnw4+002796
  09:19:07 STEP 2797 H=2797:00 +CPU= 14.889
  09:19:21 STEP 2798 H=2798:00 +CPU= 14.681
  09:19:44 STEP 2799 H=2799:00 +CPU= 23.083
  09:20:00 STEP 2800 H=2800:00 +CPU= 15.618
  09:20:15 STEP 2801 H=2801:00 +CPU= 15.059
  09:20:40 STEP 2802 H=2802:00 +CPU= 24.223
  09:20:54 STEP 2803 H=2803:00 +CPU= 14.277
  09:21:10 STEP 2804 H=2804:00 +CPU= 15.553
  09:21:34 STEP 2805 H=2805:00 +CPU= 24.203
  09:21:50 STEP 2806 H=2806:00 +CPU= 15.528
  09:22:05 STEP 2807 H=2807:00 +CPU= 14.985
  09:22:33 STEP 2808 H=2808:00 +CPU= 28.048
Moving to projects directory: /home/galfridian/slots/13/ICMGGhnw4+002808
Moving to projects directory: /home/galfridian/slots/13/ICMSHhnw4+002808
Moving to projects directory: /home/galfridian/slots/13/ICMUAhnw4+002808
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMGGhnw4+002784
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMSHhnw4+002784
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMUAhnw4+002784
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMGGhnw4+002796
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMSHhnw4+002796
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMUAhnw4+002796
Zipping up the intermediate file: /home/galfridian/projects/climateprediction.net/oifs_43r3_01gg_2019110100_123_993_12215326_2_r248070650_116.zip
Uploading the intermediate file: upload_file_116.zip
  09:22:49 STEP 2809 H=2809:00 +CPU= 15.134
  09:23:04 STEP 2810 H=2810:00 +CPU= 14.621
  09:23:26 STEP 2811 H=2811:00 +CPU= 22.407
  09:23:42 STEP 2812 H=2812:00 +CPU= 15.760
  09:23:58 STEP 2813 H=2813:00 +CPU= 15.398
  09:24:23 STEP 2814 H=2814:00 +CPU= 24.774
  09:24:38 STEP 2815 H=2815:00 +CPU= 15.107
  09:24:54 STEP 2816 H=2816:00 +CPU= 15.541
  09:25:19 STEP 2817 H=2817:00 +CPU= 24.474
  09:25:35 STEP 2818 H=2818:00 +CPU= 16.009
  09:25:50 STEP 2819 H=2819:00 +CPU= 15.538
  09:26:18 STEP 2820 H=2820:00 +CPU= 26.937
Moving to projects directory: /home/galfridian/slots/13/ICMGGhnw4+002820
Moving to projects directory: /home/galfridian/slots/13/ICMSHhnw4+002820
Moving to projects directory: /home/galfridian/slots/13/ICMUAhnw4+002820
  09:26:33 STEP 2821 H=2821:00 +CPU= 15.433
  09:26:50 STEP 2822 H=2822:00 +CPU= 16.246
  09:27:15 STEP 2823 H=2823:00 +CPU= 24.502
  09:27:31 STEP 2824 H=2824:00 +CPU= 15.757
  09:27:46 STEP 2825 H=2825:00 +CPU= 15.310
  09:28:11 STEP 2826 H=2826:00 +CPU= 24.459
  09:28:25 STEP 2827 H=2827:00 +CPU= 14.490
  09:28:39 STEP 2828 H=2828:00 +CPU= 13.704
  09:29:02 STEP 2829 H=2829:00 +CPU= 22.623
  09:29:18 STEP 2830 H=2830:00 +CPU= 15.255
  09:29:33 STEP 2831 H=2831:00 +CPU= 15.451
  09:30:02 STEP 2832 H=2832:00 +CPU= 28.685
Moving to projects directory: /home/galfridian/slots/13/ICMGGhnw4+002832
Moving to projects directory: /home/galfridian/slots/13/ICMSHhnw4+002832
Moving to projects directory: /home/galfridian/slots/13/ICMUAhnw4+002832
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMGGhnw4+002808
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMSHhnw4+002808
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMUAhnw4+002808
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMGGhnw4+002820
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMSHhnw4+002820
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMUAhnw4+002820
Zipping up the intermediate file: /home/galfridian/projects/climateprediction.net/oifs_43r3_01gg_2019110100_123_993_12215326_2_r248070650_117.zip
Uploading the intermediate file: upload_file_117.zip
  09:30:18 STEP 2833 H=2833:00 +CPU= 15.307
  09:30:34 STEP 2834 H=2834:00 +CPU= 16.040
  09:30:58 STEP 2835 H=2835:00 +CPU= 23.651
  09:31:14 STEP 2836 H=2836:00 +CPU= 15.382
  09:31:29 STEP 2837 H=2837:00 +CPU= 15.075
  09:31:52 STEP 2838 H=2838:00 +CPU= 22.895
  09:32:05 STEP 2839 H=2839:00 +CPU= 12.959
  09:32:19 STEP 2840 H=2840:00 +CPU= 13.955
  09:32:42 STEP 2841 H=2841:00 +CPU= 22.961
  09:32:57 STEP 2842 H=2842:00 +CPU= 15.094
  09:33:10 STEP 2843 H=2843:00 +CPU= 13.348
  09:33:36 STEP 2844 H=2844:00 +CPU= 25.219
Moving to projects directory: /home/galfridian/slots/13/ICMGGhnw4+002844
Moving to projects directory: /home/galfridian/slots/13/ICMSHhnw4+002844
Moving to projects directory: /home/galfridian/slots/13/ICMUAhnw4+002844
  09:33:48 STEP 2845 H=2845:00 +CPU= 12.345
  09:34:03 STEP 2846 H=2846:00 +CPU= 14.938
  09:34:26 STEP 2847 H=2847:00 +CPU= 22.838
  09:34:42 STEP 2848 H=2848:00 +CPU= 15.602
  09:34:57 STEP 2849 H=2849:00 +CPU= 15.500
  09:35:22 STEP 2850 H=2850:00 +CPU= 24.271
  09:35:37 STEP 2851 H=2851:00 +CPU= 14.670
  09:35:53 STEP 2852 H=2852:00 +CPU= 15.824
  09:36:17 STEP 2853 H=2853:00 +CPU= 23.930
  09:36:32 STEP 2854 H=2854:00 +CPU= 15.149
  09:36:47 STEP 2855 H=2855:00 +CPU= 15.475
  09:37:15 STEP 2856 H=2856:00 +CPU= 28.015
Moving to projects directory: /home/galfridian/slots/13/ICMGGhnw4+002856
Moving to projects directory: /home/galfridian/slots/13/ICMSHhnw4+002856
Moving to projects directory: /home/galfridian/slots/13/ICMUAhnw4+002856
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMGGhnw4+002832
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMSHhnw4+002832
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMUAhnw4+002832
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMGGhnw4+002844
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMSHhnw4+002844
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMUAhnw4+002844
Zipping up the intermediate file: /home/galfridian/projects/climateprediction.net/oifs_43r3_01gg_2019110100_123_993_12215326_2_r248070650_118.zip
Uploading the intermediate file: upload_file_118.zip
  09:37:31 STEP 2857 H=2857:00 +CPU= 15.084
  09:37:46 STEP 2858 H=2858:00 +CPU= 15.718
  09:38:11 STEP 2859 H=2859:00 +CPU= 24.951
  09:38:28 STEP 2860 H=2860:00 +CPU= 16.722
  09:38:47 STEP 2861 H=2861:00 +CPU= 18.608
  09:39:21 STEP 2862 H=2862:00 +CPU= 33.442
  09:39:56 STEP 2863 H=2863:00 +CPU= 23.010
  09:40:20 STEP 2864 H=2864:00 +CPU= 24.499
  09:41:02 STEP 2865 H=2865:00 +CPU= 40.477
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_1.21_x86_64-pc-linux-gnu
(argv1) start_date: 2019110100
(argv2) exptid: hnw4
(argv3) unique_member_id: 01gg
(argv4) batchid: 993
(argv5) wuid: 12215326
(argv6) fclen: 123
(argv7) app_name: oifs_43r3
(argv8) nthreads: 1
Working directory is: /home/galfridian/slots/13
Project directory is: /home/galfridian/projects/climateprediction.net/
app name: oifs_43r3
version: 1.21
Location of temp folder: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326
..mkdir for temp folder for results failed
Copying: /home/galfridian/projects/climateprediction.net/oifs_43r3_app_1.21_x86_64-pc-linux-gnu.zip to: /home/galfridian/slots/13/oifs_43r3_app_1.21_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /home/galfridian/slots/13/oifs_43r3_app_1.21_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_1d222e0d7222d0c8bd3fbd4ff9b226e1 to: /home/galfridian/slots/13/oifs_43r3_01gg_2019110100_123_993_12215326.zip
Unzipping the namelist zip file: /home/galfridian/slots/13/oifs_43r3_01gg_2019110100_123_993_12215326.zip
ic_ancil_file: ic_ancil_12215326
ifsdata_file: ifsdata_12215326
climate_data_file: clim_data_12215326
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_ca8fa23abb06cfcb3d05dd4def5aad97 to: /home/galfridian/slots/13/ic_ancil_12215326.zip
Unzipping the IC ancils zip file: /home/galfridian/slots/13/ic_ancil_12215326.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_40289548475a1de9ef4b62a6c33b9c27 to: /home/galfridian/slots/13/ifsdata/ifsdata_12215326.zip
Unzipping the ifsdata_zip file: /home/galfridian/slots/13/ifsdata/ifsdata_12215326.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /home/galfridian/slots/13/159l_2/clim_data_12215326.zip
Unzipping the climate data zip file: /home/galfridian/slots/13/159l_2/clim_data_12215326.zip
Checking for progress XML file: /home/galfridian/slots/13/progress_file_12215326.xml
Opened progress file ok : /home/galfridian/slots/13/progress_file_12215326.xml
-- Model is restarting --
Adjusting last_iter, 2866, to previous model restart step.
Creating progress file: /home/galfridian/slots/13/progress_file_12215326.xml
last_cpu_time: 54047
upload_file_number: 119
last_iter: 2857
last_upload: 10281600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_01gg_2019110100_123_993_12215326_2_r248070650
The child process has been launched with process id: 13119
Executing the command: /home/galfridian/slots/13/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:galfridian-systemproductname:1:1:13119:13119] [20230226:094227:1677400947:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2091] fp = 0x3124020
[EC_DRHOOK:galfridian-systemproductname:1:1:13119:13119] [20230226:094227:1677400947:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:galfridian-systemproductname:1:1:13119:13119] [20230226:094227:1677400947:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:galfridian-systemproductname:1:1:13119:13119] [20230226:094227:1677400947:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:galfridian-systemproductname:1:1:13119:13119] [20230226:094227:1677400947:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:galfridian-systemproductname:1:1:13119:13119] [20230226:094227:1677400947:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:galfridian-systemproductname:1:1:13119:13119] [20230226:094227:1677400947:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:galfridian-systemproductname:1:1:13119:13119] [20230226:094227:1677400947:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:galfridian-systemproductname:1:1:13119:13119] [20230226:094227:1677400947:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:galfridian-systemproductname:1:1:13119:13119] [20230226:094227:1677400947:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:galfridian-systemproductname:1:1:13119:13119] [20230226:094227:1677400947:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:galfridian-systemproductname:1:1:13119:13119] [20230226:094227:1677400947:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:galfridian-systemproductname:1:1:13119:13119] [20230226:094227:1677400947:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:galfridian-systemproductname:1:1:13119:13119] [20230226:094227:1677400947:0.000] [ignore_signals@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:galfridian-systemproductname:1:1:13119:13119] [20230226:094227:1677400947:0.000] [restore_default_signals@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:galfridian-systemproductname:1:1:13119:13119] [20230226:094227:1677400947:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:galfridian-systemproductname:1:1:13119:13119] [20230226:094227:1677400947:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x810730 (old at (nil))
[EC_DRHOOK:galfridian-systemproductname:1:1:13119:13119] [20230226:094227:1677400947:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:galfridian-systemproductname:1:1:13119:13119] [20230226:094227:1677400947:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x810730 (old at (nil))
[EC_DRHOOK:galfridian-systemproductname:1:1:13119:13119] [20230226:094227:1677400947:0.000] [trapfpe_treatment@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:galfridian-systemproductname:1:1:13119:13119] [20230226:094227:1677400947:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:galfridian-systemproductname:1:1:13119:13119] [20230226:094227:1677400947:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:galfridian-systemproductname:1:1:13119:13119] [20230226:094227:1677400947:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x810730 (old at (nil))
[EC_DRHOOK:galfridian-systemproductname:1:1:13119:13119] [20230226:094227:1677400947:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:galfridian-systemproductname:1:1:13119:13119] [20230226:094227:1677400947:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:galfridian-systemproductname:1:1:13119:13119] [20230226:094227:1677400947:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:galfridian-systemproductname:1:1:13119:13119] [20230226:094227:1677400947:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x810730 (old at (nil))
[EC_DRHOOK:galfridian-systemproductname:1:1:13119:13119] [20230226:094227:1677400947:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x810730 (old at (nil))
[EC_DRHOOK:galfridian-systemproductname:1:1:13119:13119] [20230226:094227:1677400947:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x810730 (old at (nil))
[EC_DRHOOK:galfridian-systemproductname:1:1:13119:13119] [20230226:094227:1677400947:0.000] [catch_signals@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  09:43:28 STEP 2856 H=2856:00 +CPU= 47.107
  09:43:55 STEP 2857 H=2857:00 +CPU= 26.561
  09:44:19 STEP 2858 H=2858:00 +CPU= 23.769
  09:44:55 STEP 2859 H=2859:00 +CPU= 36.157
  09:45:19 STEP 2860 H=2860:00 +CPU= 23.116
  09:45:42 STEP 2861 H=2861:00 +CPU= 23.159
  09:46:17 STEP 2862 H=2862:00 +CPU= 35.135
  09:46:40 STEP 2863 H=2863:00 +CPU= 22.529
  09:47:04 STEP 2864 H=2864:00 +CPU= 23.476
  09:47:38 STEP 2865 H=2865:00 +CPU= 33.701
  09:48:01 STEP 2866 H=2866:00 +CPU= 23.428
  09:48:24 STEP 2867 H=2867:00 +CPU= 22.773
  09:49:03 STEP 2868 H=2868:00 +CPU= 38.654
Moving to projects directory: /home/galfridian/slots/13/ICMGGhnw4+002868
Moving to projects directory: /home/galfridian/slots/13/ICMSHhnw4+002868
Moving to projects directory: /home/galfridian/slots/13/ICMUAhnw4+002868
  09:49:25 STEP 2869 H=2869:00 +CPU= 21.078
  09:49:49 STEP 2870 H=2870:00 +CPU= 24.086
  09:50:25 STEP 2871 H=2871:00 +CPU= 36.011
  09:50:47 STEP 2872 H=2872:00 +CPU= 21.691
  09:51:10 STEP 2873 H=2873:00 +CPU= 22.108
  09:51:48 STEP 2874 H=2874:00 +CPU= 37.532
  09:52:12 STEP 2875 H=2875:00 +CPU= 23.622
  09:52:36 STEP 2876 H=2876:00 +CPU= 23.365
  09:53:15 STEP 2877 H=2877:00 +CPU= 37.807
  09:53:37 STEP 2878 H=2878:00 +CPU= 22.442
  09:54:00 STEP 2879 H=2879:00 +CPU= 22.580
  09:54:40 STEP 2880 H=2880:00 +CPU= 39.630
Moving to projects directory: /home/galfridian/slots/13/ICMGGhnw4+002880
Moving to projects directory: /home/galfridian/slots/13/ICMSHhnw4+002880
Moving to projects directory: /home/galfridian/slots/13/ICMUAhnw4+002880
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMGGhnw4+002856
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMSHhnw4+002856
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMUAhnw4+002856
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMGGhnw4+002868
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMSHhnw4+002868
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMUAhnw4+002868
Zipping up the intermediate file: /home/galfridian/projects/climateprediction.net/oifs_43r3_01gg_2019110100_123_993_12215326_2_r248070650_119.zip
Uploading the intermediate file: upload_file_119.zip
  09:55:02 STEP 2881 H=2881:00 +CPU= 21.429
  09:55:26 STEP 2882 H=2882:00 +CPU= 23.659
  09:56:01 STEP 2883 H=2883:00 +CPU= 35.444
  09:56:25 STEP 2884 H=2884:00 +CPU= 23.164
  09:56:47 STEP 2885 H=2885:00 +CPU= 21.829
  09:57:22 STEP 2886 H=2886:00 +CPU= 35.330
  09:57:45 STEP 2887 H=2887:00 +CPU= 23.078
  09:58:10 STEP 2888 H=2888:00 +CPU= 24.335
  10:01:04 STEP 2889 H=2889:00 +CPU= 37.332
  10:01:23 STEP 2890 H=2890:00 +CPU= 18.519
  10:01:42 STEP 2891 H=2891:00 +CPU= 18.782
  10:02:15 STEP 2892 H=2892:00 +CPU= 32.812
Moving to projects directory: /home/galfridian/slots/13/ICMGGhnw4+002892
Moving to projects directory: /home/galfridian/slots/13/ICMSHhnw4+002892
Moving to projects directory: /home/galfridian/slots/13/ICMUAhnw4+002892
  10:02:32 STEP 2893 H=2893:00 +CPU= 16.967
  10:02:50 STEP 2894 H=2894:00 +CPU= 18.150
  10:03:20 STEP 2895 H=2895:00 +CPU= 29.806
  10:03:39 STEP 2896 H=2896:00 +CPU= 18.887
  10:03:59 STEP 2897 H=2897:00 +CPU= 20.488
  10:04:29 STEP 2898 H=2898:00 +CPU= 28.845
  10:04:47 STEP 2899 H=2899:00 +CPU= 18.243
  10:05:07 STEP 2900 H=2900:00 +CPU= 20.170
  10:06:04 STEP 2901 H=2901:00 +CPU= 36.220
  10:06:27 STEP 2902 H=2902:00 +CPU= 23.083
  10:06:52 STEP 2903 H=2903:00 +CPU= 24.444
  10:07:41 STEP 2904 H=2904:00 +CPU= 46.362
Moving to projects directory: /home/galfridian/slots/13/ICMGGhnw4+002904
Moving to projects directory: /home/galfridian/slots/13/ICMSHhnw4+002904
Moving to projects directory: /home/galfridian/slots/13/ICMUAhnw4+002904
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMGGhnw4+002880
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMSHhnw4+002880
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMUAhnw4+002880
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMGGhnw4+002892
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMSHhnw4+002892
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMUAhnw4+002892
Zipping up the intermediate file: /home/galfridian/projects/climateprediction.net/oifs_43r3_01gg_2019110100_123_993_12215326_2_r248070650_120.zip
Uploading the intermediate file: upload_file_120.zip
  10:08:07 STEP 2905 H=2905:00 +CPU= 25.673
  10:08:38 STEP 2906 H=2906:00 +CPU= 29.203
  10:09:20 STEP 2907 H=2907:00 +CPU= 41.410
  10:09:51 STEP 2908 H=2908:00 +CPU= 31.327
  10:10:21 STEP 2909 H=2909:00 +CPU= 29.210
  10:11:06 STEP 2910 H=2910:00 +CPU= 45.497
  10:11:37 STEP 2911 H=2911:00 +CPU= 29.864
  10:12:09 STEP 2912 H=2912:00 +CPU= 32.242
  10:12:59 STEP 2913 H=2913:00 +CPU= 49.397
  10:13:28 STEP 2914 H=2914:00 +CPU= 28.892
  10:13:59 STEP 2915 H=2915:00 +CPU= 30.264
  10:14:49 STEP 2916 H=2916:00 +CPU= 49.620
Moving to projects directory: /home/galfridian/slots/13/ICMGGhnw4+002916
Moving to projects directory: /home/galfridian/slots/13/ICMSHhnw4+002916
Moving to projects directory: /home/galfridian/slots/13/ICMUAhnw4+002916
  10:15:22 STEP 2917 H=2917:00 +CPU= 32.612
  10:15:57 STEP 2918 H=2918:00 +CPU= 34.618
  10:16:46 STEP 2919 H=2919:00 +CPU= 47.885
  10:17:16 STEP 2920 H=2920:00 +CPU= 29.967
  10:17:49 STEP 2921 H=2921:00 +CPU= 32.907
  10:18:38 STEP 2922 H=2922:00 +CPU= 48.345
  10:19:08 STEP 2923 H=2923:00 +CPU= 29.171
  10:19:37 STEP 2924 H=2924:00 +CPU= 28.429
  10:20:23 STEP 2925 H=2925:00 +CPU= 45.915
  10:21:00 STEP 2926 H=2926:00 +CPU= 36.451
  10:21:34 STEP 2927 H=2927:00 +CPU= 33.243
  10:22:33 STEP 2928 H=2928:00 +CPU= 58.487
Moving to projects directory: /home/galfridian/slots/13/ICMGGhnw4+002928
Moving to projects directory: /home/galfridian/slots/13/ICMSHhnw4+002928
Moving to projects directory: /home/galfridian/slots/13/ICMUAhnw4+002928
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMGGhnw4+002904
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMSHhnw4+002904
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMUAhnw4+002904
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMGGhnw4+002916
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMSHhnw4+002916
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMUAhnw4+002916
Zipping up the intermediate file: /home/galfridian/projects/climateprediction.net/oifs_43r3_01gg_2019110100_123_993_12215326_2_r248070650_121.zip
Uploading the intermediate file: upload_file_121.zip
  10:22:59 STEP 2929 H=2929:00 +CPU= 25.935
  10:23:38 STEP 2930 H=2930:00 +CPU= 38.321
  10:24:33 STEP 2931 H=2931:00 +CPU= 53.960
  10:25:06 STEP 2932 H=2932:00 +CPU= 33.069
  10:25:39 STEP 2933 H=2933:00 +CPU= 31.711
  10:26:29 STEP 2934 H=2934:00 +CPU= 49.622
  10:27:00 STEP 2935 H=2935:00 +CPU= 30.970
  10:27:30 STEP 2936 H=2936:00 +CPU= 29.897
  10:28:17 STEP 2937 H=2937:00 +CPU= 46.054
  10:28:48 STEP 2938 H=2938:00 +CPU= 30.905
  10:29:22 STEP 2939 H=2939:00 +CPU= 33.683
  10:30:13 STEP 2940 H=2940:00 +CPU= 50.773
Moving to projects directory: /home/galfridian/slots/13/ICMGGhnw4+002940
Moving to projects directory: /home/galfridian/slots/13/ICMSHhnw4+002940
Moving to projects directory: /home/galfridian/slots/13/ICMUAhnw4+002940
  10:30:44 STEP 2941 H=2941:00 +CPU= 30.224
  10:31:11 STEP 2942 H=2942:00 +CPU= 27.116
  10:31:59 STEP 2943 H=2943:00 +CPU= 46.994
  10:32:28 STEP 2944 H=2944:00 +CPU= 28.414
  10:32:59 STEP 2945 H=2945:00 +CPU= 31.383
  10:33:52 STEP 2946 H=2946:00 +CPU= 51.552
  10:34:18 STEP 2947 H=2947:00 +CPU= 25.620
  10:34:51 STEP 2948 H=2948:00 +CPU= 32.615
  10:35:38 STEP 2949 H=2949:00 +CPU= 47.228
  10:36:10 STEP 2950 H=2950:00 +CPU= 31.490
  10:36:41 STEP 2951 H=2951:00 +CPU= 30.570
  10:37:33 STEP 2952 H=2952:00 +CPU= 51.606
..The child process terminated with status: 0
>>> Printing last 70 lines from file: NODE.001_01

 STEPO, YSPSDT_AR1     (  1,...), SP NORMS 2D: 4.865283731608E-01 1.865286143845E-01 6.362598939121E-02
 STEPO, YSPSDT_AR1     (SUM,...), SP NORMS 2D: 5.260318722649E-01
 STEPO, YGPSDT         (  1,...), Grid Point Norms
  FIELD LEV        AVERAGE               MINIMUM               MAXIMUM       
    101  60  3.48989191816039E-02 -1.72994898449055E+00  1.89761310057317E+00
 STEPO, YGPSDT[clipped](  1,...), Grid Point Norms
  FIELD LEV        AVERAGE               MINIMUM               MAXIMUM       
    101  60  3.19925486346338E-02 -1.00000000000000E+00  1.00000000000000E+00
  MAXGPFV : MAX. VALUE =    1.03970507181223     
  MAXGPFV : MAX. VALUE =   0.743368388584092     
 NSTEP =  2952 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGhnw4+002952 MODE=a
 IO-STREAM CLOSED - ICMGGhnw4+002952
 MPI-TASK:   1 -    355634170 BYTES IN      8 RECORDS TRANSFERRED IN    0.0004 SECONDS889085.4 Mbytes/s, TOTAL TIME=    0.0083( 4.8%)
   !!!!!!!!!!!!!! WAVE FIELDS INTEGRATED.  DATE IS: 20200302231500  !!!!!!!!!!!! 
   !!!!!!!!!!!!!! WAVE FIELDS INTEGRATED.  DATE IS: 20200302233000  !!!!!!!!!!!! 
   !!!!!!!!!!!!!! WAVE FIELDS INTEGRATED.  DATE IS: 20200302234500  !!!!!!!!!!!! 
   !!!!!!!!!!!!!! WAVE FIELDS INTEGRATED.  DATE IS: 20200303000000  !!!!!!!!!!!! 
 10:37:33 STEP 2952 H=2952:00 +CPU= 51.606
 END CNT3
 NSTEP =  2952 CNT0   000000000  43.689  43.689  44.209   0.988               0               0               0               0               0
IO-STREAM STATISTICS, TOTAL NO OF REC/BYTES READ           35/   804992476 WRITTEN         1492/************
 -TASK-OPENED-OPEN-RECS IN -KBYTE IN    -RECS OUT-KBYTE OUT-WALL   -WALL IN-WALL
  OU-TOT IN -TOT OUT
   1     39    0       35       786125     1492 ************    14.4     3.5    10.9     5.2    17.4
===-=== START OF TIMING STATISTICS ===-===
 
STATS FOR ALL TASKS
 NUM ROUTINE                                     CALLS  MEAN(ms)   MAX(ms)   FRAC(%)  UNBAL(%)
   0 CNT0     - COMPLETE EXECUTION                   1 3306362.3 3306362.3    100.00      0.00
   1 CNT4     - FORWARD INTEGRATION                  1 3299095.4 3299095.4     99.78      0.00
   8 SCAN2M - GRID-POINT DYNAMICS                  105    2828.3    2828.3      8.98      0.00
   9 SPCM     - SPECTRAL COMP.                      96     557.5     557.5      1.62      0.00
  10 SCAN2M - PHYSICS                               97   13036.2   13036.2     38.24      0.00
  11 IOPACK   - OUTPUT P.P. RESULTS                  8    2134.6    2134.6      0.52      0.00
  12 SPNORM   - SPECTRAL NORM COMP.                  5      32.0      32.0      0.00      0.00
  13 SCAN2M - RADIATION CALC.                       33   15395.5   15395.5     15.37      0.00
  17 GRIDFPOS IN CNT4                                8      61.0      61.0      0.01      0.00
  27 RESTART FILES - WRITING                         4    4265.4    4265.4      0.52      0.00
  28 RESTART FILES - READING                         1    6646.5    6646.5      0.20      0.00
  29 SU4FPOS IN CNT4                                 8       0.5       0.5      0.00      0.00
  30 DYNFPOS IN CNT4                                 8    2569.3    2569.3      0.62      0.00
  31 POSDDH IN STEPO                                 1       3.6       3.6      0.00      0.00
  37 CPGLAG   - SL COMPUTATIONS                     97    6157.5    6157.5     18.06      0.00
  38 WAM      - TOTAL COST OF WAVE MODEL            97    3318.7    3318.7      9.74      0.00
  39 SU0YOMB                                         1     360.4     360.4      0.01      0.00
  51 SCAN2M   - SL COMM. PART 1                     97      25.1      25.1      0.07      0.00
  54 SPCM     - M TO S/S TO M TRANSP.               96     342.5     342.5      0.99      0.00
  55 SPCIMPF  - S TO M/M TO S TRANSP.               96      48.4      48.4      0.14      0.00
  56 SPNORM   - SPECTRAL NORM COMM.                  5       0.5       0.5      0.00      0.00
 102 LTINV_CTL   - INVERSE LEGENDRE TRANSFORM      334     151.3     151.3      1.53      0.00
 103 LTDIR_CTL   - DIRECT LEGENDRE TRANSFORM       200     156.8     156.8      0.95      0.00
 106 FTDIR_CTL   - DIRECT FOURIER TRANSFORM        200      77.3      77.3      0.47      0.00
 107 FTINV_CTL   - INVERSE FOURIER TRANSFORM       334      59.2      59.2      0.60      0.00
 140 SULEG       - COMP. OF LEGENDRE POL.            2      26.5      26.5      0.00      0.00
 152 LTINV_CTL   - M TO L TRANSPOSITION            334      43.3      43.3      0.44      0.00
 153 LTDIR_CTL   - L TO M TRANSPOSITION            200      40.5      40.5      0.25      0.00
 157 FTINV_CTL   - L TO G TRANSPOSITION            334      64.7      64.7      0.65      0.00
 158 FTDIR_CTL   - G TO L TRANSPOSITION            200      43.3      43.3      0.26      0.00
 400 GSTATS                                      20345       0.8       0.8      0.49      0.00
 401 GSTATS HOOK                                 19418       0.1       0.1      0.04      0.00
TOTAL MEASURED IMBALANCE =       0.0 SECONDS,  0.0 PERCENT
TOTAL WALLCLOCK TIME     3306.4 CPU TIME    3105.4 VECTOR TIME     3105.4


===-=== END   OF TIMING STATISTICS ===-===
FORECAST DAYS PER DAY     3221.2
  *** END CNT0 *** 
------------------------------------------------
Moving to projects directory: /home/galfridian/slots/13/ICMGGhnw4+002952
Moving to projects directory: /home/galfridian/slots/13/ICMSHhnw4+002952
Moving to projects directory: /home/galfridian/slots/13/ICMUAhnw4+002952
Adding to the zip: /home/galfridian/slots/13/NODE.001_01
Adding to the zip: /home/galfridian/slots/13/ifs.stat
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMUAhnw4+002928
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMSHhnw4+002928
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMSHhnw4+002952
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMSHhnw4+002940
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMUAhnw4+002952
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMUAhnw4+002940
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMGGhnw4+002928
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMGGhnw4+002940
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_12215326/ICMGGhnw4+002952
Zipping up the final file: /home/galfridian/projects/climateprediction.net/oifs_43r3_01gg_2019110100_123_993_12215326_2_r248070650_122.zip
Uploading the final file: upload_file_122.zip
Uploading trickle at timestep: 10623600
10:40:56 (13115): called boinc_finish(0)

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:54:24 1537623 22317065 oifs_43r3_01gg_2019110100_123_993_12215326_2 10,623,600 57,152 0.0054
09 May 2023 16:54:22 1537623 22317065 oifs_43r3_01gg_2019110100_123_993_12215326_2 10,022,400 52,505 0.0052
09 May 2023 16:54:21 1537623 22317065 oifs_43r3_01gg_2019110100_123_993_12215326_2 9,158,400 48,163 0.0053
09 May 2023 16:54:19 1537623 22317065 oifs_43r3_01gg_2019110100_123_993_12215326_2 8,294,400 43,690 0.0053
09 May 2023 16:54:16 1537623 22317065 oifs_43r3_01gg_2019110100_123_993_12215326_2 7,430,400 39,227 0.0053
09 May 2023 16:54:14 1537623 22317065 oifs_43r3_01gg_2019110100_123_993_12215326_2 6,566,400 34,784 0.0053
09 May 2023 16:54:12 1537623 22317065 oifs_43r3_01gg_2019110100_123_993_12215326_2 5,702,400 30,321 0.0053
09 May 2023 16:54:10 1537623 22317065 oifs_43r3_01gg_2019110100_123_993_12215326_2 4,838,400 25,819 0.0053
09 May 2023 16:54:06 1537623 22317065 oifs_43r3_01gg_2019110100_123_993_12215326_2 3,974,400 21,336 0.0054
09 May 2023 16:54:04 1537623 22317065 oifs_43r3_01gg_2019110100_123_993_12215326_2 3,110,400 16,895 0.0054
09 May 2023 16:54:01 1537623 22317065 oifs_43r3_01gg_2019110100_123_993_12215326_2 2,246,400 12,383 0.0055
09 May 2023 16:53:56 1537623 22317065 oifs_43r3_01gg_2019110100_123_993_12215326_2 864,000 4,570 0.0053


©2024 climateprediction.net