climateprediction.net home page
Task 22294258

Task 22294258

Name oifs_43r3_ps_0061_2003050100_123_972_12188705_1
Workunit 12188705
Created 29 Dec 2022, 9:14:32 UTC
Sent 10 Jan 2023, 10:45:40 UTC
Report deadline 9 Feb 2023, 10:45:40 UTC
Received 15 Jan 2023, 0:59:10 UTC
Server state Over
Outcome Success
Client state Done
Exit status 0 (0x00000000)
Computer ID 1494261
Run time 14 hours 36 min 35 sec
CPU time 14 hours 18 min 59 sec
Validate state Valid
Credit 14,873.04
Device peak FLOPS 7.91 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,414.92 MB
Peak swap size 4,978.00 MB
Peak disk usage 3,765.35 MB
Stderr
<core_client_version>7.20.2</core_client_version>
<![CDATA[
<stderr_txt>
2457:00 +CPU= 23.278
  00:00:19 STEP 2458 H=2458:00 +CPU= 15.913
  00:00:35 STEP 2459 H=2459:00 +CPU= 15.385
  00:01:00 STEP 2460 H=2460:00 +CPU= 24.475
Moving to projects directory: /home/antonio/slots/7/ICMGGhq0f+002460
Moving to projects directory: /home/antonio/slots/7/ICMSHhq0f+002460
Moving to projects directory: /home/antonio/slots/7/ICMUAhq0f+002460
  00:01:15 STEP 2461 H=2461:00 +CPU= 15.071
  00:01:31 STEP 2462 H=2462:00 +CPU= 15.065
  00:01:54 STEP 2463 H=2463:00 +CPU= 22.159
  00:02:09 STEP 2464 H=2464:00 +CPU= 14.820
  00:02:24 STEP 2465 H=2465:00 +CPU= 14.815
  00:02:47 STEP 2466 H=2466:00 +CPU= 22.635
  00:03:03 STEP 2467 H=2467:00 +CPU= 15.268
  00:03:18 STEP 2468 H=2468:00 +CPU= 15.025
  00:03:40 STEP 2469 H=2469:00 +CPU= 21.572
  00:03:56 STEP 2470 H=2470:00 +CPU= 15.564
  00:04:11 STEP 2471 H=2471:00 +CPU= 15.072
  00:04:38 STEP 2472 H=2472:00 +CPU= 25.828
Moving to projects directory: /home/antonio/slots/7/ICMGGhq0f+002472
Moving to projects directory: /home/antonio/slots/7/ICMSHhq0f+002472
Moving to projects directory: /home/antonio/slots/7/ICMUAhq0f+002472
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMGGhq0f+002448
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMSHhq0f+002448
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMUAhq0f+002448
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMGGhq0f+002460
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMSHhq0f+002460
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMUAhq0f+002460
Zipping up the intermediate file: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_0061_2003050100_123_972_12188705_1_r846408588_102.zip
Uploading the intermediate file: upload_file_102.zip
  00:04:53 STEP 2473 H=2473:00 +CPU= 14.711
  00:05:08 STEP 2474 H=2474:00 +CPU= 15.105
  00:05:31 STEP 2475 H=2475:00 +CPU= 21.707
  00:05:46 STEP 2476 H=2476:00 +CPU= 15.295
  00:06:02 STEP 2477 H=2477:00 +CPU= 15.149
  00:06:25 STEP 2478 H=2478:00 +CPU= 22.546
  00:06:40 STEP 2479 H=2479:00 +CPU= 15.201
  00:06:56 STEP 2480 H=2480:00 +CPU= 15.445
  00:07:19 STEP 2481 H=2481:00 +CPU= 22.003
  00:07:35 STEP 2482 H=2482:00 +CPU= 15.445
  00:07:50 STEP 2483 H=2483:00 +CPU= 15.024
  00:08:15 STEP 2484 H=2484:00 +CPU= 24.389
Moving to projects directory: /home/antonio/slots/7/ICMGGhq0f+002484
Moving to projects directory: /home/antonio/slots/7/ICMSHhq0f+002484
Moving to projects directory: /home/antonio/slots/7/ICMUAhq0f+002484
  00:08:30 STEP 2485 H=2485:00 +CPU= 14.902
  00:08:46 STEP 2486 H=2486:00 +CPU= 15.202
  00:09:09 STEP 2487 H=2487:00 +CPU= 22.217
  00:09:24 STEP 2488 H=2488:00 +CPU= 15.327
  00:09:40 STEP 2489 H=2489:00 +CPU= 15.128
  00:10:03 STEP 2490 H=2490:00 +CPU= 22.861
  00:10:19 STEP 2491 H=2491:00 +CPU= 15.349
  00:10:35 STEP 2492 H=2492:00 +CPU= 15.252
  00:10:58 STEP 2493 H=2493:00 +CPU= 22.606
  00:11:14 STEP 2494 H=2494:00 +CPU= 15.396
  00:11:30 STEP 2495 H=2495:00 +CPU= 15.797
  00:11:57 STEP 2496 H=2496:00 +CPU= 26.677
Moving to projects directory: /home/antonio/slots/7/ICMGGhq0f+002496
Moving to projects directory: /home/antonio/slots/7/ICMSHhq0f+002496
Moving to projects directory: /home/antonio/slots/7/ICMUAhq0f+002496
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMGGhq0f+002472
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMSHhq0f+002472
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMUAhq0f+002472
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMGGhq0f+002484
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMSHhq0f+002484
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMUAhq0f+002484
Zipping up the intermediate file: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_0061_2003050100_123_972_12188705_1_r846408588_103.zip
Uploading the intermediate file: upload_file_103.zip
  00:12:13 STEP 2497 H=2497:00 +CPU= 15.122
  00:12:29 STEP 2498 H=2498:00 +CPU= 15.813
  00:12:51 STEP 2499 H=2499:00 +CPU= 22.025
  00:13:08 STEP 2500 H=2500:00 +CPU= 15.782
  00:13:23 STEP 2501 H=2501:00 +CPU= 15.158
  00:13:47 STEP 2502 H=2502:00 +CPU= 23.022
  00:14:03 STEP 2503 H=2503:00 +CPU= 15.955
  00:14:20 STEP 2504 H=2504:00 +CPU= 15.883
  00:14:43 STEP 2505 H=2505:00 +CPU= 22.522
  00:14:59 STEP 2506 H=2506:00 +CPU= 16.058
  00:15:16 STEP 2507 H=2507:00 +CPU= 16.182
  00:15:40 STEP 2508 H=2508:00 +CPU= 23.951
Moving to projects directory: /home/antonio/slots/7/ICMGGhq0f+002508
Moving to projects directory: /home/antonio/slots/7/ICMSHhq0f+002508
Moving to projects directory: /home/antonio/slots/7/ICMUAhq0f+002508
  00:15:55 STEP 2509 H=2509:00 +CPU= 14.091
  00:16:10 STEP 2510 H=2510:00 +CPU= 14.698
  00:16:30 STEP 2511 H=2511:00 +CPU= 20.608
  00:16:45 STEP 2512 H=2512:00 +CPU= 14.269
  00:16:59 STEP 2513 H=2513:00 +CPU= 14.467
  00:17:21 STEP 2514 H=2514:00 +CPU= 21.293
  00:17:35 STEP 2515 H=2515:00 +CPU= 13.844
  00:17:49 STEP 2516 H=2516:00 +CPU= 13.767
  00:18:10 STEP 2517 H=2517:00 +CPU= 21.074
  00:18:25 STEP 2518 H=2518:00 +CPU= 14.444
  00:18:39 STEP 2519 H=2519:00 +CPU= 14.171
  00:19:04 STEP 2520 H=2520:00 +CPU= 25.037
Moving to projects directory: /home/antonio/slots/7/ICMGGhq0f+002520
Moving to projects directory: /home/antonio/slots/7/ICMSHhq0f+002520
Moving to projects directory: /home/antonio/slots/7/ICMUAhq0f+002520
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMGGhq0f+002496
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMSHhq0f+002496
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMUAhq0f+002496
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMGGhq0f+002508
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMSHhq0f+002508
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMUAhq0f+002508
Zipping up the intermediate file: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_0061_2003050100_123_972_12188705_1_r846408588_104.zip
Uploading the intermediate file: upload_file_104.zip
  00:19:18 STEP 2521 H=2521:00 +CPU= 13.757
  00:19:33 STEP 2522 H=2522:00 +CPU= 14.523
  00:19:54 STEP 2523 H=2523:00 +CPU= 21.051
  00:20:09 STEP 2524 H=2524:00 +CPU= 14.862
  00:20:23 STEP 2525 H=2525:00 +CPU= 13.914
  00:20:45 STEP 2526 H=2526:00 +CPU= 21.234
  00:20:59 STEP 2527 H=2527:00 +CPU= 13.938
  00:21:14 STEP 2528 H=2528:00 +CPU= 14.773
  00:21:35 STEP 2529 H=2529:00 +CPU= 20.570
  00:21:49 STEP 2530 H=2530:00 +CPU= 14.189
  00:22:03 STEP 2531 H=2531:00 +CPU= 14.332
  00:22:27 STEP 2532 H=2532:00 +CPU= 23.179
Moving to projects directory: /home/antonio/slots/7/ICMGGhq0f+002532
Moving to projects directory: /home/antonio/slots/7/ICMSHhq0f+002532
Moving to projects directory: /home/antonio/slots/7/ICMUAhq0f+002532
  00:22:41 STEP 2533 H=2533:00 +CPU= 13.936
  00:22:56 STEP 2534 H=2534:00 +CPU= 14.462
  00:23:17 STEP 2535 H=2535:00 +CPU= 21.117
  00:23:32 STEP 2536 H=2536:00 +CPU= 15.016
  00:23:46 STEP 2537 H=2537:00 +CPU= 14.139
  00:24:08 STEP 2538 H=2538:00 +CPU= 21.275
  00:24:23 STEP 2539 H=2539:00 +CPU= 14.457
  00:24:38 STEP 2540 H=2540:00 +CPU= 14.978
  00:25:00 STEP 2541 H=2541:00 +CPU= 21.278
  00:25:15 STEP 2542 H=2542:00 +CPU= 15.198
  00:25:30 STEP 2543 H=2543:00 +CPU= 15.020
  00:25:56 STEP 2544 H=2544:00 +CPU= 25.434
Moving to projects directory: /home/antonio/slots/7/ICMGGhq0f+002544
Moving to projects directory: /home/antonio/slots/7/ICMSHhq0f+002544
Moving to projects directory: /home/antonio/slots/7/ICMUAhq0f+002544
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMGGhq0f+002520
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMSHhq0f+002520
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMUAhq0f+002520
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMGGhq0f+002532
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMSHhq0f+002532
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMUAhq0f+002532
Zipping up the intermediate file: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_0061_2003050100_123_972_12188705_1_r846408588_105.zip
Uploading the intermediate file: upload_file_105.zip
  00:26:11 STEP 2545 H=2545:00 +CPU= 14.502
  00:26:26 STEP 2546 H=2546:00 +CPU= 15.092
  00:26:48 STEP 2547 H=2547:00 +CPU= 21.345
  00:27:03 STEP 2548 H=2548:00 +CPU= 14.615
  00:27:17 STEP 2549 H=2549:00 +CPU= 14.333
  00:27:39 STEP 2550 H=2550:00 +CPU= 21.052
  00:27:53 STEP 2551 H=2551:00 +CPU= 13.969
  00:28:08 STEP 2552 H=2552:00 +CPU= 15.053
  00:28:29 STEP 2553 H=2553:00 +CPU= 20.562
  00:28:43 STEP 2554 H=2554:00 +CPU= 14.129
  00:28:57 STEP 2555 H=2555:00 +CPU= 14.003
  00:29:21 STEP 2556 H=2556:00 +CPU= 23.253
Moving to projects directory: /home/antonio/slots/7/ICMGGhq0f+002556
Moving to projects directory: /home/antonio/slots/7/ICMSHhq0f+002556
Moving to projects directory: /home/antonio/slots/7/ICMUAhq0f+002556
  00:29:35 STEP 2557 H=2557:00 +CPU= 13.731
  00:29:49 STEP 2558 H=2558:00 +CPU= 14.688
  00:30:11 STEP 2559 H=2559:00 +CPU= 21.329
  00:30:26 STEP 2560 H=2560:00 +CPU= 14.444
  00:30:40 STEP 2561 H=2561:00 +CPU= 13.951
  00:31:01 STEP 2562 H=2562:00 +CPU= 20.995
  00:31:16 STEP 2563 H=2563:00 +CPU= 14.647
  00:31:31 STEP 2564 H=2564:00 +CPU= 14.591
  00:31:51 STEP 2565 H=2565:00 +CPU= 20.608
  00:32:06 STEP 2566 H=2566:00 +CPU= 14.527
  00:32:21 STEP 2567 H=2567:00 +CPU= 15.026
  00:32:46 STEP 2568 H=2568:00 +CPU= 24.168
Moving to projects directory: /home/antonio/slots/7/ICMGGhq0f+002568
Moving to projects directory: /home/antonio/slots/7/ICMSHhq0f+002568
Moving to projects directory: /home/antonio/slots/7/ICMUAhq0f+002568
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMGGhq0f+002544
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMSHhq0f+002544
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMUAhq0f+002544
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMGGhq0f+002556
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMSHhq0f+002556
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMUAhq0f+002556
Zipping up the intermediate file: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_0061_2003050100_123_972_12188705_1_r846408588_106.zip
Uploading the intermediate file: upload_file_106.zip
  00:33:00 STEP 2569 H=2569:00 +CPU= 13.874
  00:33:15 STEP 2570 H=2570:00 +CPU= 14.494
  00:33:35 STEP 2571 H=2571:00 +CPU= 20.428
  00:33:50 STEP 2572 H=2572:00 +CPU= 14.342
  00:34:04 STEP 2573 H=2573:00 +CPU= 13.788
  00:34:25 STEP 2574 H=2574:00 +CPU= 21.536
  00:34:40 STEP 2575 H=2575:00 +CPU= 14.357
  00:34:54 STEP 2576 H=2576:00 +CPU= 14.466
  00:35:15 STEP 2577 H=2577:00 +CPU= 20.473
  00:35:30 STEP 2578 H=2578:00 +CPU= 14.799
  00:35:44 STEP 2579 H=2579:00 +CPU= 14.216
  00:36:08 STEP 2580 H=2580:00 +CPU= 23.087
Moving to projects directory: /home/antonio/slots/7/ICMGGhq0f+002580
Moving to projects directory: /home/antonio/slots/7/ICMSHhq0f+002580
Moving to projects directory: /home/antonio/slots/7/ICMUAhq0f+002580
  00:36:22 STEP 2581 H=2581:00 +CPU= 14.309
  00:36:37 STEP 2582 H=2582:00 +CPU= 14.248
  00:36:57 STEP 2583 H=2583:00 +CPU= 20.447
  00:37:12 STEP 2584 H=2584:00 +CPU= 14.786
  00:37:26 STEP 2585 H=2585:00 +CPU= 13.606
  00:37:49 STEP 2586 H=2586:00 +CPU= 22.321
  00:38:03 STEP 2587 H=2587:00 +CPU= 14.373
  00:38:18 STEP 2588 H=2588:00 +CPU= 14.861
  00:38:39 STEP 2589 H=2589:00 +CPU= 20.859
  00:38:54 STEP 2590 H=2590:00 +CPU= 14.736
  00:39:09 STEP 2591 H=2591:00 +CPU= 14.473
  00:39:33 STEP 2592 H=2592:00 +CPU= 24.338
Moving to projects directory: /home/antonio/slots/7/ICMGGhq0f+002592
Moving to projects directory: /home/antonio/slots/7/ICMSHhq0f+002592
Moving to projects directory: /home/antonio/slots/7/ICMUAhq0f+002592
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMGGhq0f+002568
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMSHhq0f+002568
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMUAhq0f+002568
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMGGhq0f+002580
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMSHhq0f+002580
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMUAhq0f+002580
Zipping up the intermediate file: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_0061_2003050100_123_972_12188705_1_r846408588_107.zip
Uploading the intermediate file: upload_file_107.zip
  00:39:48 STEP 2593 H=2593:00 +CPU= 14.069
  00:40:02 STEP 2594 H=2594:00 +CPU= 14.218
  00:40:23 STEP 2595 H=2595:00 +CPU= 21.091
  00:40:37 STEP 2596 H=2596:00 +CPU= 13.977
  00:40:51 STEP 2597 H=2597:00 +CPU= 13.912
  00:41:14 STEP 2598 H=2598:00 +CPU= 21.842
  00:41:28 STEP 2599 H=2599:00 +CPU= 13.936
  00:41:42 STEP 2600 H=2600:00 +CPU= 14.220
  00:42:03 STEP 2601 H=2601:00 +CPU= 20.828
  00:42:18 STEP 2602 H=2602:00 +CPU= 14.745
  00:42:32 STEP 2603 H=2603:00 +CPU= 14.014
  00:42:55 STEP 2604 H=2604:00 +CPU= 23.060
Moving to projects directory: /home/antonio/slots/7/ICMGGhq0f+002604
Moving to projects directory: /home/antonio/slots/7/ICMSHhq0f+002604
Moving to projects directory: /home/antonio/slots/7/ICMUAhq0f+002604
  00:43:10 STEP 2605 H=2605:00 +CPU= 14.024
  00:43:24 STEP 2606 H=2606:00 +CPU= 14.615
  00:43:45 STEP 2607 H=2607:00 +CPU= 20.844
  00:44:00 STEP 2608 H=2608:00 +CPU= 14.353
  00:44:14 STEP 2609 H=2609:00 +CPU= 14.079
  00:44:36 STEP 2610 H=2610:00 +CPU= 21.494
  00:44:50 STEP 2611 H=2611:00 +CPU= 14.222
  00:45:05 STEP 2612 H=2612:00 +CPU= 14.853
  00:45:27 STEP 2613 H=2613:00 +CPU= 21.665
  00:45:42 STEP 2614 H=2614:00 +CPU= 14.125
  00:45:56 STEP 2615 H=2615:00 +CPU= 14.689
  00:46:21 STEP 2616 H=2616:00 +CPU= 24.130
Moving to projects directory: /home/antonio/slots/7/ICMGGhq0f+002616
Moving to projects directory: /home/antonio/slots/7/ICMSHhq0f+002616
Moving to projects directory: /home/antonio/slots/7/ICMUAhq0f+002616
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMGGhq0f+002592
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMSHhq0f+002592
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMUAhq0f+002592
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMGGhq0f+002604
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMSHhq0f+002604
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMUAhq0f+002604
Zipping up the intermediate file: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_0061_2003050100_123_972_12188705_1_r846408588_108.zip
Uploading the intermediate file: upload_file_108.zip
  00:46:35 STEP 2617 H=2617:00 +CPU= 14.003
  00:46:50 STEP 2618 H=2618:00 +CPU= 14.406
  00:47:10 STEP 2619 H=2619:00 +CPU= 20.265
  00:47:25 STEP 2620 H=2620:00 +CPU= 14.328
  00:47:39 STEP 2621 H=2621:00 +CPU= 13.806
  00:48:00 STEP 2622 H=2622:00 +CPU= 20.998
  00:48:14 STEP 2623 H=2623:00 +CPU= 13.975
  00:48:29 STEP 2624 H=2624:00 +CPU= 14.498
  00:48:49 STEP 2625 H=2625:00 +CPU= 20.532
  00:49:03 STEP 2626 H=2626:00 +CPU= 13.934
  00:49:18 STEP 2627 H=2627:00 +CPU= 14.417
  00:49:41 STEP 2628 H=2628:00 +CPU= 22.898
Moving to projects directory: /home/antonio/slots/7/ICMGGhq0f+002628
Moving to projects directory: /home/antonio/slots/7/ICMSHhq0f+002628
Moving to projects directory: /home/antonio/slots/7/ICMUAhq0f+002628
  00:49:55 STEP 2629 H=2629:00 +CPU= 13.965
  00:50:10 STEP 2630 H=2630:00 +CPU= 14.277
  00:50:31 STEP 2631 H=2631:00 +CPU= 20.977
  00:50:45 STEP 2632 H=2632:00 +CPU= 14.187
  00:50:59 STEP 2633 H=2633:00 +CPU= 13.910
  00:51:21 STEP 2634 H=2634:00 +CPU= 21.396
  00:51:35 STEP 2635 H=2635:00 +CPU= 13.630
  00:51:49 STEP 2636 H=2636:00 +CPU= 14.374
  00:52:10 STEP 2637 H=2637:00 +CPU= 20.555
  00:52:25 STEP 2638 H=2638:00 +CPU= 14.490
  00:52:40 STEP 2639 H=2639:00 +CPU= 14.872
  00:53:04 STEP 2640 H=2640:00 +CPU= 24.065
Moving to projects directory: /home/antonio/slots/7/ICMGGhq0f+002640
Moving to projects directory: /home/antonio/slots/7/ICMSHhq0f+002640
Moving to projects directory: /home/antonio/slots/7/ICMUAhq0f+002640
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMGGhq0f+002616
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMSHhq0f+002616
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMUAhq0f+002616
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMGGhq0f+002628
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMSHhq0f+002628
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMUAhq0f+002628
Zipping up the intermediate file: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_0061_2003050100_123_972_12188705_1_r846408588_109.zip
Uploading the intermediate file: upload_file_109.zip
  00:53:18 STEP 2641 H=2641:00 +CPU= 13.862
Uploading trickle at timestep: 9504000
  00:53:33 STEP 2642 H=2642:00 +CPU= 14.409
  00:53:53 STEP 2643 H=2643:00 +CPU= 20.277
  00:54:07 STEP 2644 H=2644:00 +CPU= 13.705
  00:54:21 STEP 2645 H=2645:00 +CPU= 14.371
  00:54:43 STEP 2646 H=2646:00 +CPU= 21.296
  00:54:57 STEP 2647 H=2647:00 +CPU= 13.821
  00:55:12 STEP 2648 H=2648:00 +CPU= 14.616
  00:55:33 STEP 2649 H=2649:00 +CPU= 21.092
  00:55:47 STEP 2650 H=2650:00 +CPU= 14.045
  00:56:02 STEP 2651 H=2651:00 +CPU= 14.112
  00:56:25 STEP 2652 H=2652:00 +CPU= 23.021
Moving to projects directory: /home/antonio/slots/7/ICMGGhq0f+002652
Moving to projects directory: /home/antonio/slots/7/ICMSHhq0f+002652
Moving to projects directory: /home/antonio/slots/7/ICMUAhq0f+002652
  00:56:39 STEP 2653 H=2653:00 +CPU= 13.927
  00:56:53 STEP 2654 H=2654:00 +CPU= 14.252
  00:57:14 STEP 2655 H=2655:00 +CPU= 20.397
  00:57:29 STEP 2656 H=2656:00 +CPU= 14.937
  00:57:43 STEP 2657 H=2657:00 +CPU= 14.004
  00:58:04 STEP 2658 H=2658:00 +CPU= 21.085
  00:58:19 STEP 2659 H=2659:00 +CPU= 14.022
  00:58:33 STEP 2660 H=2660:00 +CPU= 14.640
  00:58:54 STEP 2661 H=2661:00 +CPU= 20.577
  00:59:08 STEP 2662 H=2662:00 +CPU= 14.128
  00:59:23 STEP 2663 H=2663:00 +CPU= 14.750
  00:59:48 STEP 2664 H=2664:00 +CPU= 24.589
Moving to projects directory: /home/antonio/slots/7/ICMGGhq0f+002664
Moving to projects directory: /home/antonio/slots/7/ICMSHhq0f+002664
Moving to projects directory: /home/antonio/slots/7/ICMUAhq0f+002664
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMGGhq0f+002640
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMSHhq0f+002640
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMUAhq0f+002640
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMGGhq0f+002652
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMSHhq0f+002652
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMUAhq0f+002652
Zipping up the intermediate file: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_0061_2003050100_123_972_12188705_1_r846408588_110.zip
Uploading the intermediate file: upload_file_110.zip
  01:00:02 STEP 2665 H=2665:00 +CPU= 13.815
  01:00:18 STEP 2666 H=2666:00 +CPU= 15.154
  01:00:39 STEP 2667 H=2667:00 +CPU= 20.985
  01:00:53 STEP 2668 H=2668:00 +CPU= 14.322
  01:01:07 STEP 2669 H=2669:00 +CPU= 13.898
  01:01:29 STEP 2670 H=2670:00 +CPU= 21.695
  01:01:44 STEP 2671 H=2671:00 +CPU= 14.265
  01:01:59 STEP 2672 H=2672:00 +CPU= 14.577
  01:02:20 STEP 2673 H=2673:00 +CPU= 21.133
  01:02:35 STEP 2674 H=2674:00 +CPU= 14.966
  01:02:49 STEP 2675 H=2675:00 +CPU= 14.124
  01:03:12 STEP 2676 H=2676:00 +CPU= 22.498
Moving to projects directory: /home/antonio/slots/7/ICMGGhq0f+002676
Moving to projects directory: /home/antonio/slots/7/ICMSHhq0f+002676
Moving to projects directory: /home/antonio/slots/7/ICMUAhq0f+002676
  01:03:27 STEP 2677 H=2677:00 +CPU= 14.538
  01:03:41 STEP 2678 H=2678:00 +CPU= 13.902
  01:04:02 STEP 2679 H=2679:00 +CPU= 21.020
  01:04:16 STEP 2680 H=2680:00 +CPU= 14.212
  01:04:31 STEP 2681 H=2681:00 +CPU= 14.168
  01:04:52 STEP 2682 H=2682:00 +CPU= 21.552
  01:05:07 STEP 2683 H=2683:00 +CPU= 14.391
  01:05:22 STEP 2684 H=2684:00 +CPU= 14.821
  01:05:43 STEP 2685 H=2685:00 +CPU= 20.963
  01:05:58 STEP 2686 H=2686:00 +CPU= 14.423
  01:06:12 STEP 2687 H=2687:00 +CPU= 14.201
  01:06:37 STEP 2688 H=2688:00 +CPU= 24.412
Moving to projects directory: /home/antonio/slots/7/ICMGGhq0f+002688
Moving to projects directory: /home/antonio/slots/7/ICMSHhq0f+002688
Moving to projects directory: /home/antonio/slots/7/ICMUAhq0f+002688
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMGGhq0f+002664
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMSHhq0f+002664
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMUAhq0f+002664
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMGGhq0f+002676
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMSHhq0f+002676
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMUAhq0f+002676
Zipping up the intermediate file: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_0061_2003050100_123_972_12188705_1_r846408588_111.zip
Uploading the intermediate file: upload_file_111.zip
  01:06:51 STEP 2689 H=2689:00 +CPU= 13.815
  01:07:05 STEP 2690 H=2690:00 +CPU= 14.235
  01:07:26 STEP 2691 H=2691:00 +CPU= 20.906
  01:07:41 STEP 2692 H=2692:00 +CPU= 14.922
  01:07:57 STEP 2693 H=2693:00 +CPU= 15.125
  01:08:19 STEP 2694 H=2694:00 +CPU= 21.965
  01:08:35 STEP 2695 H=2695:00 +CPU= 15.378
  01:08:51 STEP 2696 H=2696:00 +CPU= 15.767
  01:09:12 STEP 2697 H=2697:00 +CPU= 21.171
  01:09:27 STEP 2698 H=2698:00 +CPU= 15.058
  01:09:42 STEP 2699 H=2699:00 +CPU= 14.739
  01:10:06 STEP 2700 H=2700:00 +CPU= 23.475
Moving to projects directory: /home/antonio/slots/7/ICMGGhq0f+002700
Moving to projects directory: /home/antonio/slots/7/ICMSHhq0f+002700
Moving to projects directory: /home/antonio/slots/7/ICMUAhq0f+002700
  01:10:20 STEP 2701 H=2701:00 +CPU= 14.040
  01:10:36 STEP 2702 H=2702:00 +CPU= 14.991
  01:10:58 STEP 2703 H=2703:00 +CPU= 22.227
  01:11:12 STEP 2704 H=2704:00 +CPU= 13.591
  01:11:25 STEP 2705 H=2705:00 +CPU= 12.443
  01:11:44 STEP 2706 H=2706:00 +CPU= 19.598
  01:11:58 STEP 2707 H=2707:00 +CPU= 13.403
  01:12:15 STEP 2708 H=2708:00 +CPU= 16.555
  01:12:37 STEP 2709 H=2709:00 +CPU= 21.593
  01:12:52 STEP 2710 H=2710:00 +CPU= 14.551
  01:13:07 STEP 2711 H=2711:00 +CPU= 15.210
  01:13:33 STEP 2712 H=2712:00 +CPU= 25.235
Moving to projects directory: /home/antonio/slots/7/ICMGGhq0f+002712
Moving to projects directory: /home/antonio/slots/7/ICMSHhq0f+002712
Moving to projects directory: /home/antonio/slots/7/ICMUAhq0f+002712
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMGGhq0f+002688
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMSHhq0f+002688
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMUAhq0f+002688
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMGGhq0f+002700
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMSHhq0f+002700
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMUAhq0f+002700
Zipping up the intermediate file: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_0061_2003050100_123_972_12188705_1_r846408588_112.zip
Uploading the intermediate file: upload_file_112.zip
  01:13:47 STEP 2713 H=2713:00 +CPU= 13.675
  01:14:02 STEP 2714 H=2714:00 +CPU= 15.227
  01:14:24 STEP 2715 H=2715:00 +CPU= 21.491
  01:14:39 STEP 2716 H=2716:00 +CPU= 14.657
  01:14:53 STEP 2717 H=2717:00 +CPU= 14.261
  01:15:16 STEP 2718 H=2718:00 +CPU= 22.879
  01:15:33 STEP 2719 H=2719:00 +CPU= 16.290
  01:15:48 STEP 2720 H=2720:00 +CPU= 14.666
  01:16:11 STEP 2721 H=2721:00 +CPU= 22.814
  01:16:27 STEP 2722 H=2722:00 +CPU= 15.816
  01:16:42 STEP 2723 H=2723:00 +CPU= 14.436
  01:17:05 STEP 2724 H=2724:00 +CPU= 22.616
Moving to projects directory: /home/antonio/slots/7/ICMGGhq0f+002724
Moving to projects directory: /home/antonio/slots/7/ICMSHhq0f+002724
Moving to projects directory: /home/antonio/slots/7/ICMUAhq0f+002724
  01:17:21 STEP 2725 H=2725:00 +CPU= 15.589
  01:17:37 STEP 2726 H=2726:00 +CPU= 15.417
  01:17:58 STEP 2727 H=2727:00 +CPU= 20.604
  01:18:12 STEP 2728 H=2728:00 +CPU= 14.319
  01:18:26 STEP 2729 H=2729:00 +CPU= 14.110
  01:18:48 STEP 2730 H=2730:00 +CPU= 21.320
  01:19:02 STEP 2731 H=2731:00 +CPU= 14.038
  01:19:17 STEP 2732 H=2732:00 +CPU= 14.242
  01:19:38 STEP 2733 H=2733:00 +CPU= 20.496
  01:19:53 STEP 2734 H=2734:00 +CPU= 14.868
  01:20:07 STEP 2735 H=2735:00 +CPU= 14.290
  01:20:32 STEP 2736 H=2736:00 +CPU= 24.492
Moving to projects directory: /home/antonio/slots/7/ICMGGhq0f+002736
Moving to projects directory: /home/antonio/slots/7/ICMSHhq0f+002736
Moving to projects directory: /home/antonio/slots/7/ICMUAhq0f+002736
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMGGhq0f+002712
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMSHhq0f+002712
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMUAhq0f+002712
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMGGhq0f+002724
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMSHhq0f+002724
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMUAhq0f+002724
Zipping up the intermediate file: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_0061_2003050100_123_972_12188705_1_r846408588_113.zip
Uploading the intermediate file: upload_file_113.zip
  01:20:46 STEP 2737 H=2737:00 +CPU= 14.186
  01:21:00 STEP 2738 H=2738:00 +CPU= 13.809
  01:21:21 STEP 2739 H=2739:00 +CPU= 20.571
  01:21:36 STEP 2740 H=2740:00 +CPU= 14.763
  01:21:50 STEP 2741 H=2741:00 +CPU= 13.676
  01:22:11 STEP 2742 H=2742:00 +CPU= 21.107
  01:22:25 STEP 2743 H=2743:00 +CPU= 14.061
  01:22:40 STEP 2744 H=2744:00 +CPU= 14.616
  01:23:02 STEP 2745 H=2745:00 +CPU= 21.282
  01:23:17 STEP 2746 H=2746:00 +CPU= 14.317
  01:23:31 STEP 2747 H=2747:00 +CPU= 13.880
  01:23:54 STEP 2748 H=2748:00 +CPU= 23.170
Moving to projects directory: /home/antonio/slots/7/ICMGGhq0f+002748
Moving to projects directory: /home/antonio/slots/7/ICMSHhq0f+002748
Moving to projects directory: /home/antonio/slots/7/ICMUAhq0f+002748
  01:24:09 STEP 2749 H=2749:00 +CPU= 15.148
  01:24:24 STEP 2750 H=2750:00 +CPU= 14.226
  01:24:45 STEP 2751 H=2751:00 +CPU= 20.958
  01:25:00 STEP 2752 H=2752:00 +CPU= 14.410
  01:25:14 STEP 2753 H=2753:00 +CPU= 13.858
  01:25:35 STEP 2754 H=2754:00 +CPU= 21.366
  01:25:50 STEP 2755 H=2755:00 +CPU= 14.078
  01:26:04 STEP 2756 H=2756:00 +CPU= 14.019
  01:26:24 STEP 2757 H=2757:00 +CPU= 20.231
  01:26:39 STEP 2758 H=2758:00 +CPU= 14.738
  01:26:53 STEP 2759 H=2759:00 +CPU= 14.072
  01:27:19 STEP 2760 H=2760:00 +CPU= 24.952
Moving to projects directory: /home/antonio/slots/7/ICMGGhq0f+002760
Moving to projects directory: /home/antonio/slots/7/ICMSHhq0f+002760
Moving to projects directory: /home/antonio/slots/7/ICMUAhq0f+002760
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMGGhq0f+002736
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMSHhq0f+002736
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMUAhq0f+002736
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMGGhq0f+002748
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMSHhq0f+002748
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMUAhq0f+002748
Zipping up the intermediate file: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_0061_2003050100_123_972_12188705_1_r846408588_114.zip
Uploading the intermediate file: upload_file_114.zip
  01:27:34 STEP 2761 H=2761:00 +CPU= 14.417
  01:27:49 STEP 2762 H=2762:00 +CPU= 14.947
  01:28:10 STEP 2763 H=2763:00 +CPU= 21.008
  01:28:25 STEP 2764 H=2764:00 +CPU= 14.206
  01:28:40 STEP 2765 H=2765:00 +CPU= 14.684
  01:29:02 STEP 2766 H=2766:00 +CPU= 21.563
  01:29:16 STEP 2767 H=2767:00 +CPU= 14.310
  01:29:31 STEP 2768 H=2768:00 +CPU= 14.814
  01:29:53 STEP 2769 H=2769:00 +CPU= 21.750
  01:30:08 STEP 2770 H=2770:00 +CPU= 14.764
  01:30:24 STEP 2771 H=2771:00 +CPU= 14.957
  01:30:48 STEP 2772 H=2772:00 +CPU= 23.621
Moving to projects directory: /home/antonio/slots/7/ICMGGhq0f+002772
Moving to projects directory: /home/antonio/slots/7/ICMSHhq0f+002772
Moving to projects directory: /home/antonio/slots/7/ICMUAhq0f+002772
  01:31:02 STEP 2773 H=2773:00 +CPU= 14.170
  01:31:17 STEP 2774 H=2774:00 +CPU= 14.248
  01:31:38 STEP 2775 H=2775:00 +CPU= 21.287
  01:31:53 STEP 2776 H=2776:00 +CPU= 14.486
  01:32:07 STEP 2777 H=2777:00 +CPU= 13.917
  01:32:29 STEP 2778 H=2778:00 +CPU= 21.283
  01:32:44 STEP 2779 H=2779:00 +CPU= 14.504
  01:32:58 STEP 2780 H=2780:00 +CPU= 14.339
  01:33:19 STEP 2781 H=2781:00 +CPU= 20.709
  01:33:34 STEP 2782 H=2782:00 +CPU= 14.467
  01:33:49 STEP 2783 H=2783:00 +CPU= 14.842
  01:34:14 STEP 2784 H=2784:00 +CPU= 24.314
Moving to projects directory: /home/antonio/slots/7/ICMGGhq0f+002784
Moving to projects directory: /home/antonio/slots/7/ICMSHhq0f+002784
Moving to projects directory: /home/antonio/slots/7/ICMUAhq0f+002784
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMGGhq0f+002760
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMSHhq0f+002760
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMUAhq0f+002760
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMGGhq0f+002772
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMSHhq0f+002772
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMUAhq0f+002772
Zipping up the intermediate file: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_0061_2003050100_123_972_12188705_1_r846408588_115.zip
Uploading the intermediate file: upload_file_115.zip
  01:34:27 STEP 2785 H=2785:00 +CPU= 13.546
  01:34:42 STEP 2786 H=2786:00 +CPU= 14.913
  01:35:03 STEP 2787 H=2787:00 +CPU= 20.540
  01:35:17 STEP 2788 H=2788:00 +CPU= 13.948
  01:35:31 STEP 2789 H=2789:00 +CPU= 13.743
  01:35:54 STEP 2790 H=2790:00 +CPU= 21.952
  01:36:07 STEP 2791 H=2791:00 +CPU= 13.768
  01:36:22 STEP 2792 H=2792:00 +CPU= 14.616
  01:36:44 STEP 2793 H=2793:00 +CPU= 21.369
  01:36:59 STEP 2794 H=2794:00 +CPU= 15.029
  01:37:14 STEP 2795 H=2795:00 +CPU= 14.973
  01:37:37 STEP 2796 H=2796:00 +CPU= 22.761
Moving to projects directory: /home/antonio/slots/7/ICMGGhq0f+002796
Moving to projects directory: /home/antonio/slots/7/ICMSHhq0f+002796
Moving to projects directory: /home/antonio/slots/7/ICMUAhq0f+002796
  01:37:53 STEP 2797 H=2797:00 +CPU= 14.902
  01:38:07 STEP 2798 H=2798:00 +CPU= 14.555
  01:38:28 STEP 2799 H=2799:00 +CPU= 20.911
  01:38:43 STEP 2800 H=2800:00 +CPU= 14.186
  01:38:57 STEP 2801 H=2801:00 +CPU= 14.041
  01:39:18 STEP 2802 H=2802:00 +CPU= 21.221
  01:39:32 STEP 2803 H=2803:00 +CPU= 13.776
  01:39:48 STEP 2804 H=2804:00 +CPU= 14.958
  01:40:08 STEP 2805 H=2805:00 +CPU= 20.404
  01:40:23 STEP 2806 H=2806:00 +CPU= 14.317
  01:40:37 STEP 2807 H=2807:00 +CPU= 14.430
  01:41:03 STEP 2808 H=2808:00 +CPU= 24.836
Moving to projects directory: /home/antonio/slots/7/ICMGGhq0f+002808
Moving to projects directory: /home/antonio/slots/7/ICMSHhq0f+002808
Moving to projects directory: /home/antonio/slots/7/ICMUAhq0f+002808
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMGGhq0f+002784
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMSHhq0f+002784
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMUAhq0f+002784
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMGGhq0f+002796
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMSHhq0f+002796
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMUAhq0f+002796
Zipping up the intermediate file: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_0061_2003050100_123_972_12188705_1_r846408588_116.zip
Uploading the intermediate file: upload_file_116.zip
  01:41:17 STEP 2809 H=2809:00 +CPU= 13.923
  01:41:31 STEP 2810 H=2810:00 +CPU= 13.906
  01:41:54 STEP 2811 H=2811:00 +CPU= 22.626
  01:42:08 STEP 2812 H=2812:00 +CPU= 14.405
  01:42:22 STEP 2813 H=2813:00 +CPU= 13.607
  01:42:44 STEP 2814 H=2814:00 +CPU= 21.413
  01:42:58 STEP 2815 H=2815:00 +CPU= 14.214
  01:43:12 STEP 2816 H=2816:00 +CPU= 14.060
  01:43:33 STEP 2817 H=2817:00 +CPU= 20.193
  01:43:48 STEP 2818 H=2818:00 +CPU= 15.098
  01:44:02 STEP 2819 H=2819:00 +CPU= 13.783
  01:44:25 STEP 2820 H=2820:00 +CPU= 23.105
Moving to projects directory: /home/antonio/slots/7/ICMGGhq0f+002820
Moving to projects directory: /home/antonio/slots/7/ICMSHhq0f+002820
Moving to projects directory: /home/antonio/slots/7/ICMUAhq0f+002820
  01:44:40 STEP 2821 H=2821:00 +CPU= 14.139
  01:44:55 STEP 2822 H=2822:00 +CPU= 14.705
  01:45:16 STEP 2823 H=2823:00 +CPU= 21.083
  01:45:31 STEP 2824 H=2824:00 +CPU= 14.612
  01:45:45 STEP 2825 H=2825:00 +CPU= 14.582
  01:46:07 STEP 2826 H=2826:00 +CPU= 21.025
  01:46:21 STEP 2827 H=2827:00 +CPU= 13.990
  01:46:35 STEP 2828 H=2828:00 +CPU= 13.846
  01:46:56 STEP 2829 H=2829:00 +CPU= 20.894
  01:47:11 STEP 2830 H=2830:00 +CPU= 14.410
  01:47:25 STEP 2831 H=2831:00 +CPU= 14.058
  01:47:50 STEP 2832 H=2832:00 +CPU= 25.263
Moving to projects directory: /home/antonio/slots/7/ICMGGhq0f+002832
Moving to projects directory: /home/antonio/slots/7/ICMSHhq0f+002832
Moving to projects directory: /home/antonio/slots/7/ICMUAhq0f+002832
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMGGhq0f+002808
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMSHhq0f+002808
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMUAhq0f+002808
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMGGhq0f+002820
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMSHhq0f+002820
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMUAhq0f+002820
Zipping up the intermediate file: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_0061_2003050100_123_972_12188705_1_r846408588_117.zip
Uploading the intermediate file: upload_file_117.zip
  01:48:04 STEP 2833 H=2833:00 +CPU= 13.392
  01:48:19 STEP 2834 H=2834:00 +CPU= 14.442
  01:48:40 STEP 2835 H=2835:00 +CPU= 20.953
  01:48:55 STEP 2836 H=2836:00 +CPU= 14.676
  01:49:09 STEP 2837 H=2837:00 +CPU= 14.232
  01:49:30 STEP 2838 H=2838:00 +CPU= 21.058
  01:49:45 STEP 2839 H=2839:00 +CPU= 14.185
  01:50:00 STEP 2840 H=2840:00 +CPU= 14.706
  01:50:20 STEP 2841 H=2841:00 +CPU= 20.639
  01:50:35 STEP 2842 H=2842:00 +CPU= 14.346
  01:50:49 STEP 2843 H=2843:00 +CPU= 14.275
  01:51:13 STEP 2844 H=2844:00 +CPU= 23.573
Moving to projects directory: /home/antonio/slots/7/ICMGGhq0f+002844
Moving to projects directory: /home/antonio/slots/7/ICMSHhq0f+002844
Moving to projects directory: /home/antonio/slots/7/ICMUAhq0f+002844
  01:51:28 STEP 2845 H=2845:00 +CPU= 14.816
  01:51:44 STEP 2846 H=2846:00 +CPU= 14.910
  01:52:07 STEP 2847 H=2847:00 +CPU= 22.669
  01:52:21 STEP 2848 H=2848:00 +CPU= 14.568
  01:52:34 STEP 2849 H=2849:00 +CPU= 12.715
  01:52:54 STEP 2850 H=2850:00 +CPU= 19.281
  01:53:07 STEP 2851 H=2851:00 +CPU= 12.757
  01:53:20 STEP 2852 H=2852:00 +CPU= 13.187
  01:53:39 STEP 2853 H=2853:00 +CPU= 18.585
  01:53:52 STEP 2854 H=2854:00 +CPU= 13.488
  01:54:06 STEP 2855 H=2855:00 +CPU= 13.483
  01:54:30 STEP 2856 H=2856:00 +CPU= 24.305
Moving to projects directory: /home/antonio/slots/7/ICMGGhq0f+002856
Moving to projects directory: /home/antonio/slots/7/ICMSHhq0f+002856
Moving to projects directory: /home/antonio/slots/7/ICMUAhq0f+002856
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMGGhq0f+002832
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMSHhq0f+002832
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMUAhq0f+002832
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMGGhq0f+002844
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMSHhq0f+002844
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMUAhq0f+002844
Zipping up the intermediate file: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_0061_2003050100_123_972_12188705_1_r846408588_118.zip
Uploading the intermediate file: upload_file_118.zip
  01:54:44 STEP 2857 H=2857:00 +CPU= 13.641
  01:54:58 STEP 2858 H=2858:00 +CPU= 14.303
  01:55:19 STEP 2859 H=2859:00 +CPU= 20.155
  01:55:34 STEP 2860 H=2860:00 +CPU= 14.910
[EC_DRHOOK:Durin:1:1:23385:23385] [20230111:015537:1673398537:50976.512] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 4745MB (heap), 4415MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:Durin:1:1:23385:23385] [20230111:015537:1673398537:50976.512] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:Durin:1:1:23385:23385] [20230111:015537:1673398537:50976.512] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f3c0 (old at (nil))
[EC_DRHOOK:Durin:1:1:23385:23385] [20230111:015537:1673398537:50976.512] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x3e800001273], nsigs = 1
[EC_DRHOOK:Durin:1:1:23385:23385] [20230111:015537:1673398537:50976.512] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1
[EC_DRHOOK:Durin:1:1:23385:23385] [20230111:015537:1673398537:50976.512] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 4745 MB (maxheap), 4415 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:Durin:1:1:23385:23385] [20230111:015537:1673398537:50976.512] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:Durin:1:1:23385:23385] [20230111:015537:1673398537:50976.512] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:Durin:1:1:23385:23385] [20230111:015537:1673398537:50976.512] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:Durin:1:1:23385:23385] [20230111:015537:1673398537:50976.512] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:Durin:1:1:23385:23385] [20230111:015537:1673398537:50976.512] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:Durin:1:1:23385:23385] [20230111:015537:1673398537:50976.512] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :       CNT4 
[EC_DRHOOK:Durin:1:1:23385:23385] [20230111:015537:1673398537:50976.512] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :        STEPO 
[EC_DRHOOK:Durin:1:1:23385:23385] [20230111:015537:1673398537:50976.512] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :         SCAN2M 
[EC_DRHOOK:Durin:1:1:23385:23385] [20230111:015537:1673398537:50976.512] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :          GP_MODEL_HEAP 
[EC_DRHOOK:Durin:1:1:23385:23385] [20230111:015537:1673398537:50976.512] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :           GP_MODEL 
[EC_DRHOOK:Durin:1:1:23385:23385] [20230111:015537:1673398537:50976.513] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :            CALL_SL_HEAP 
[EC_DRHOOK:Durin:1:1:23385:23385] [20230111:015537:1673398537:50976.513] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :             CALL_SL 
[EC_DRHOOK:Durin:1:1:23385:23385] [20230111:015537:1673398537:50976.513] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :              >OMP-CALL_SL     1        (1004) 
[EC_DRHOOK:Durin:1:1:23385:23385] [20230111:015537:1673398537:50976.513] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :               LAPINEA 
[EC_DRHOOK:Durin:1:1:23385:23385] [20230111:015537:1673398537:50976.513] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                LARMES 
[EC_DRHOOK:Durin:1:1:23385:23385] [20230111:015537:1673398537:50976.513] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                 LARCINA 
[EC_DRHOOK:Durin:1:1:23385:23385] [20230111:015537:1673398537:50976.513] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                  LASCAW 
[EC_DRHOOK:Durin:1:1:23385:23385] [20230111:015538:1673398538:50977.513] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1
[EC_DRHOOK:Durin:1:1:23385:23385] [20230111:015538:1673398538:50977.513] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1ce8bb0 for signal#15, nsigs = 1
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source             
oifs_43r3_model.e  0000000001CE916B  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000081FFF1  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DC9090  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000000DF1288  lascaw_                   470  lascaw.F90
oifs_43r3_model.e  0000000000DED525  larcina_                  319  larcina.F90
oifs_43r3_model.e  0000000000DEABF8  larmes_                   603  larmes.F90
oifs_43r3_model.e  0000000000DE50FD  lapinea_                  270  lapinea.F90
oifs_43r3_model.e  0000000000D7A4EC  call_sl_                  300  call_sl.F90
oifs_43r3_model.e  0000000000D784FD  call_sl_heap_              94  call_sl_heap.F90
oifs_43r3_model.e  0000000000BF9958  gp_model_                 533  gp_model.F90
oifs_43r3_model.e  00000000012E9482  gp_model_heap_             74  gp_model_heap.F90
oifs_43r3_model.e  0000000000BBC1CE  scan2m_                   535  scan2m.F90
oifs_43r3_model.e  000000000057A664  stepo_                    327  stepo.F90
oifs_43r3_model.e  000000000055DEEC  cnt4_                    1133  cnt4.F90
oifs_43r3_model.e  00000000005471C9  cnt3_                     267  cnt3.F90
oifs_43r3_model.e  0000000000546412  cnt2_                      88  cnt2.F90
oifs_43r3_model.e  0000000000545E28  cnt1_                      92  cnt1.F90
oifs_43r3_model.e  000000000040708D  cnt0_                     146  cnt0.F90
oifs_43r3_model.e  000000000040220F  MAIN__                     96  master.F90
oifs_43r3_model.e  00000000004021A2  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCA390  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000040206E  Unknown               Unknown  Unknown
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2003050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0061
(argv4) batchid: 972
(argv5) wuid: 12188705
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /home/antonio/slots/7
Project directory is: /home/antonio/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705
..mkdir for temp folder for results failed
Copying: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /home/antonio/slots/7/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /home/antonio/slots/7/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_4007e283032348d72ceae53478eb1897 to: /home/antonio/slots/7/oifs_43r3_ps_0061_2003050100_123_972_12188705.zip
Unzipping the namelist zip file: /home/antonio/slots/7/oifs_43r3_ps_0061_2003050100_123_972_12188705.zip
ic_ancil_file: ic_ancil_12188705
ifsdata_file: ifsdata_12188705
climate_data_file: clim_data_12188705
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_0def3e18bd943f5dc8136033ed55dd1e to: /home/antonio/slots/7/ic_ancil_12188705.zip
Unzipping the IC ancils zip file: /home/antonio/slots/7/ic_ancil_12188705.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_657df7d1b8eac2df351c790a3cfbb469 to: /home/antonio/slots/7/ifsdata/ifsdata_12188705.zip
Unzipping the ifsdata_zip file: /home/antonio/slots/7/ifsdata/ifsdata_12188705.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /home/antonio/slots/7/159l_2/clim_data_12188705.zip
Unzipping the climate data zip file: /home/antonio/slots/7/159l_2/clim_data_12188705.zip
Checking for progress XML file: /home/antonio/slots/7/progress_file_12188705.xml
Opened progress file ok : /home/antonio/slots/7/progress_file_12188705.xml
-- Model is restarting --
Adjusting last_iter, 2861, to previous model restart step.
Creating progress file: /home/antonio/slots/7/progress_file_12188705.xml
last_cpu_time: 50148
upload_file_number: 119
last_iter: 2855
last_upload: 10281600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0061_2003050100_123_972_12188705_1_r846408588
The child process has been launched with process id: 7521
Executing the command: /home/antonio/slots/7/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:Durin:1:1:7521:7521] [20230111:120508:1673435108:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:Durin:1:1:7521:7521] [20230111:120508:1673435108:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:Durin:1:1:7521:7521] [20230111:120508:1673435108:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:Durin:1:1:7521:7521] [20230111:120508:1673435108:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:Durin:1:1:7521:7521] [20230111:120508:1673435108:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:Durin:1:1:7521:7521] [20230111:120508:1673435108:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:Durin:1:1:7521:7521] [20230111:120508:1673435108:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:Durin:1:1:7521:7521] [20230111:120508:1673435108:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:Durin:1:1:7521:7521] [20230111:120508:1673435108:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:Durin:1:1:7521:7521] [20230111:120508:1673435108:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:Durin:1:1:7521:7521] [20230111:120508:1673435108:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:Durin:1:1:7521:7521] [20230111:120508:1673435108:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:Durin:1:1:7521:7521] [20230111:120508:1673435108:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:Durin:1:1:7521:7521] [20230111:120508:1673435108:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:Durin:1:1:7521:7521] [20230111:120508:1673435108:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:Durin:1:1:7521:7521] [20230111:120508:1673435108:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Durin:1:1:7521:7521] [20230111:120508:1673435108:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Durin:1:1:7521:7521] [20230111:120508:1673435108:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Durin:1:1:7521:7521] [20230111:120508:1673435108:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:Durin:1:1:7521:7521] [20230111:120508:1673435108:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:Durin:1:1:7521:7521] [20230111:120508:1673435108:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Durin:1:1:7521:7521] [20230111:120508:1673435108:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Durin:1:1:7521:7521] [20230111:120508:1673435108:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:Durin:1:1:7521:7521] [20230111:120508:1673435108:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Durin:1:1:7521:7521] [20230111:120508:1673435108:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Durin:1:1:7521:7521] [20230111:120508:1673435108:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Durin:1:1:7521:7521] [20230111:120508:1673435108:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:Durin:1:1:7521:7521] [20230111:120508:1673435108:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:Durin:1:1:7521:7521] [20230111:120508:1673435108:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Durin:1:1:7521:7521] [20230111:120508:1673435108:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  12:05:37 STEP 2856 H=2856:00 +CPU= 19.769
  12:05:48 STEP 2857 H=2857:00 +CPU= 11.546
  12:06:00 STEP 2858 H=2858:00 +CPU= 11.638
  12:06:17 STEP 2859 H=2859:00 +CPU= 16.905
  12:06:28 STEP 2860 H=2860:00 +CPU= 11.320
  12:06:40 STEP 2861 H=2861:00 +CPU= 11.127
  12:06:57 STEP 2862 H=2862:00 +CPU= 17.382
  12:07:08 STEP 2863 H=2863:00 +CPU= 11.229
  12:07:20 STEP 2864 H=2864:00 +CPU= 11.722
  12:07:38 STEP 2865 H=2865:00 +CPU= 18.013
  12:07:49 STEP 2866 H=2866:00 +CPU= 11.390
  12:08:01 STEP 2867 H=2867:00 +CPU= 11.313
  12:08:20 STEP 2868 H=2868:00 +CPU= 19.603
Moving to projects directory: /home/antonio/slots/7/ICMGGhq0f+002868
Moving to projects directory: /home/antonio/slots/7/ICMSHhq0f+002868
Moving to projects directory: /home/antonio/slots/7/ICMUAhq0f+002868
  12:08:33 STEP 2869 H=2869:00 +CPU= 12.333
  12:08:46 STEP 2870 H=2870:00 +CPU= 13.349
  12:09:07 STEP 2871 H=2871:00 +CPU= 20.106
  12:09:20 STEP 2872 H=2872:00 +CPU= 13.243
  12:09:32 STEP 2873 H=2873:00 +CPU= 11.489
  12:09:50 STEP 2874 H=2874:00 +CPU= 17.819
  12:10:02 STEP 2875 H=2875:00 +CPU= 11.578
  12:10:14 STEP 2876 H=2876:00 +CPU= 12.210
  12:10:32 STEP 2877 H=2877:00 +CPU= 18.186
  12:10:45 STEP 2878 H=2878:00 +CPU= 12.272
  12:10:56 STEP 2879 H=2879:00 +CPU= 11.805
  12:11:17 STEP 2880 H=2880:00 +CPU= 20.122
Moving to projects directory: /home/antonio/slots/7/ICMGGhq0f+002880
Moving to projects directory: /home/antonio/slots/7/ICMSHhq0f+002880
Moving to projects directory: /home/antonio/slots/7/ICMUAhq0f+002880
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMGGhq0f+002856
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMSHhq0f+002856
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMUAhq0f+002856
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMGGhq0f+002868
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMSHhq0f+002868
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMUAhq0f+002868
Zipping up the intermediate file: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_0061_2003050100_123_972_12188705_1_r846408588_119.zip
Uploading the intermediate file: upload_file_119.zip
  12:11:28 STEP 2881 H=2881:00 +CPU= 11.409
  12:11:40 STEP 2882 H=2882:00 +CPU= 11.715
  12:11:57 STEP 2883 H=2883:00 +CPU= 17.444
  12:12:09 STEP 2884 H=2884:00 +CPU= 11.880
  12:12:21 STEP 2885 H=2885:00 +CPU= 11.491
  12:12:38 STEP 2886 H=2886:00 +CPU= 17.619
  12:12:50 STEP 2887 H=2887:00 +CPU= 11.565
  12:13:02 STEP 2888 H=2888:00 +CPU= 11.879
  12:13:19 STEP 2889 H=2889:00 +CPU= 17.444
  12:13:31 STEP 2890 H=2890:00 +CPU= 11.771
  12:13:43 STEP 2891 H=2891:00 +CPU= 11.451
  12:14:02 STEP 2892 H=2892:00 +CPU= 19.313
Moving to projects directory: /home/antonio/slots/7/ICMGGhq0f+002892
Moving to projects directory: /home/antonio/slots/7/ICMSHhq0f+002892
Moving to projects directory: /home/antonio/slots/7/ICMUAhq0f+002892
  12:14:14 STEP 2893 H=2893:00 +CPU= 11.731
  12:14:26 STEP 2894 H=2894:00 +CPU= 11.963
  12:14:44 STEP 2895 H=2895:00 +CPU= 17.966
  12:14:57 STEP 2896 H=2896:00 +CPU= 12.634
  12:15:09 STEP 2897 H=2897:00 +CPU= 11.999
  12:15:27 STEP 2898 H=2898:00 +CPU= 18.378
  12:15:39 STEP 2899 H=2899:00 +CPU= 11.776
  12:15:52 STEP 2900 H=2900:00 +CPU= 12.191
  12:16:09 STEP 2901 H=2901:00 +CPU= 17.792
  12:16:22 STEP 2902 H=2902:00 +CPU= 12.204
  12:16:34 STEP 2903 H=2903:00 +CPU= 12.069
  12:16:55 STEP 2904 H=2904:00 +CPU= 20.981
Moving to projects directory: /home/antonio/slots/7/ICMGGhq0f+002904
Moving to projects directory: /home/antonio/slots/7/ICMSHhq0f+002904
Moving to projects directory: /home/antonio/slots/7/ICMUAhq0f+002904
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMGGhq0f+002880
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMSHhq0f+002880
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMUAhq0f+002880
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMGGhq0f+002892
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMSHhq0f+002892
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMUAhq0f+002892
Zipping up the intermediate file: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_0061_2003050100_123_972_12188705_1_r846408588_120.zip
Uploading the intermediate file: upload_file_120.zip
  12:17:07 STEP 2905 H=2905:00 +CPU= 11.840
  12:17:19 STEP 2906 H=2906:00 +CPU= 12.055
  12:17:37 STEP 2907 H=2907:00 +CPU= 17.908
  12:17:49 STEP 2908 H=2908:00 +CPU= 12.031
  12:18:01 STEP 2909 H=2909:00 +CPU= 11.872
  12:18:20 STEP 2910 H=2910:00 +CPU= 18.702
  12:18:33 STEP 2911 H=2911:00 +CPU= 12.852
  12:18:46 STEP 2912 H=2912:00 +CPU= 12.280
  12:19:04 STEP 2913 H=2913:00 +CPU= 18.413
  12:19:17 STEP 2914 H=2914:00 +CPU= 12.615
  12:19:29 STEP 2915 H=2915:00 +CPU= 12.167
  12:19:50 STEP 2916 H=2916:00 +CPU= 20.224
Moving to projects directory: /home/antonio/slots/7/ICMGGhq0f+002916
Moving to projects directory: /home/antonio/slots/7/ICMSHhq0f+002916
Moving to projects directory: /home/antonio/slots/7/ICMUAhq0f+002916
  12:20:01 STEP 2917 H=2917:00 +CPU= 11.383
  12:20:13 STEP 2918 H=2918:00 +CPU= 11.682
  12:20:30 STEP 2919 H=2919:00 +CPU= 17.224
  12:20:42 STEP 2920 H=2920:00 +CPU= 11.568
  12:20:53 STEP 2921 H=2921:00 +CPU= 11.477
  12:21:11 STEP 2922 H=2922:00 +CPU= 17.869
  12:21:23 STEP 2923 H=2923:00 +CPU= 11.430
  12:21:34 STEP 2924 H=2924:00 +CPU= 11.633
  12:21:52 STEP 2925 H=2925:00 +CPU= 17.577
  12:22:04 STEP 2926 H=2926:00 +CPU= 12.438
  12:22:16 STEP 2927 H=2927:00 +CPU= 11.786
  12:22:37 STEP 2928 H=2928:00 +CPU= 20.763
Moving to projects directory: /home/antonio/slots/7/ICMGGhq0f+002928
Moving to projects directory: /home/antonio/slots/7/ICMSHhq0f+002928
Moving to projects directory: /home/antonio/slots/7/ICMUAhq0f+002928
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMGGhq0f+002904
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMSHhq0f+002904
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMUAhq0f+002904
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMGGhq0f+002916
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMSHhq0f+002916
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMUAhq0f+002916
Zipping up the intermediate file: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_0061_2003050100_123_972_12188705_1_r846408588_121.zip
Uploading the intermediate file: upload_file_121.zip
  12:22:50 STEP 2929 H=2929:00 +CPU= 12.663
  12:23:02 STEP 2930 H=2930:00 +CPU= 11.995
  12:23:22 STEP 2931 H=2931:00 +CPU= 19.709
  12:23:38 STEP 2932 H=2932:00 +CPU= 15.043
  12:23:51 STEP 2933 H=2933:00 +CPU= 12.889
  12:24:11 STEP 2934 H=2934:00 +CPU= 19.897
  12:24:24 STEP 2935 H=2935:00 +CPU= 13.125
  12:24:37 STEP 2936 H=2936:00 +CPU= 12.638
  12:24:57 STEP 2937 H=2937:00 +CPU= 19.384
  12:25:11 STEP 2938 H=2938:00 +CPU= 13.838
  12:25:22 STEP 2939 H=2939:00 +CPU= 11.645
  12:25:42 STEP 2940 H=2940:00 +CPU= 19.660
Moving to projects directory: /home/antonio/slots/7/ICMGGhq0f+002940
Moving to projects directory: /home/antonio/slots/7/ICMSHhq0f+002940
Moving to projects directory: /home/antonio/slots/7/ICMUAhq0f+002940
  12:25:54 STEP 2941 H=2941:00 +CPU= 11.552
  12:26:06 STEP 2942 H=2942:00 +CPU= 11.755
  12:26:25 STEP 2943 H=2943:00 +CPU= 18.751
  12:26:38 STEP 2944 H=2944:00 +CPU= 12.908
  12:26:51 STEP 2945 H=2945:00 +CPU= 12.899
  12:27:11 STEP 2946 H=2946:00 +CPU= 19.559
  12:27:24 STEP 2947 H=2947:00 +CPU= 12.313
  12:27:36 STEP 2948 H=2948:00 +CPU= 12.002
  12:27:54 STEP 2949 H=2949:00 +CPU= 17.816
  12:28:06 STEP 2950 H=2950:00 +CPU= 12.009
  12:28:17 STEP 2951 H=2951:00 +CPU= 11.764
  12:28:38 STEP 2952 H=2952:00 +CPU= 20.215
The child process terminated with status: 0
Moving to projects directory: /home/antonio/slots/7/ICMGGhq0f+002952
Moving to projects directory: /home/antonio/slots/7/ICMSHhq0f+002952
Moving to projects directory: /home/antonio/slots/7/ICMUAhq0f+002952
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMGGhq0f+002940
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMSHhq0f+002952
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMUAhq0f+002928
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMGGhq0f+002928
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMSHhq0f+002940
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMUAhq0f+002940
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMUAhq0f+002952
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMSHhq0f+002928
Adding to the zip: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_12188705/ICMGGhq0f+002952
Zipping up the final file: /home/antonio/projects/climateprediction.net/oifs_43r3_ps_0061_2003050100_123_972_12188705_1_r846408588_122.zip
Uploading the final file: upload_file_122.zip
Uploading trickle at timestep: 10623600
12:32:00 (7328): called boinc_finish(0)

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:15:31 1494261 22294258 oifs_43r3_ps_0061_2003050100_123_972_12188705_1 10,623,600 51,539 0.0049
09 May 2023 16:15:07 1494261 22294258 oifs_43r3_ps_0061_2003050100_123_972_12188705_1 9,504,000 46,447 0.0049
09 May 2023 16:15:05 1494261 22294258 oifs_43r3_ps_0061_2003050100_123_972_12188705_1 8,640,000 42,301 0.0049
09 May 2023 16:15:03 1494261 22294258 oifs_43r3_ps_0061_2003050100_123_972_12188705_1 7,776,000 37,742 0.0049
09 May 2023 16:14:58 1494261 22294258 oifs_43r3_ps_0061_2003050100_123_972_12188705_1 6,912,000 33,613 0.0049
09 May 2023 16:14:56 1494261 22294258 oifs_43r3_ps_0061_2003050100_123_972_12188705_1 6,048,000 29,357 0.0049
09 May 2023 16:14:53 1494261 22294258 oifs_43r3_ps_0061_2003050100_123_972_12188705_1 5,184,000 25,048 0.0048
09 May 2023 16:14:51 1494261 22294258 oifs_43r3_ps_0061_2003050100_123_972_12188705_1 4,320,000 20,942 0.0048
09 May 2023 16:14:49 1494261 22294258 oifs_43r3_ps_0061_2003050100_123_972_12188705_1 3,456,000 16,809 0.0049
09 May 2023 16:14:46 1494261 22294258 oifs_43r3_ps_0061_2003050100_123_972_12188705_1 2,592,000 12,723 0.0049
09 May 2023 16:14:44 1494261 22294258 oifs_43r3_ps_0061_2003050100_123_972_12188705_1 1,728,000 8,619 0.0050
09 May 2023 16:14:39 1494261 22294258 oifs_43r3_ps_0061_2003050100_123_972_12188705_1 864,000 4,422 0.0051


©2024 climateprediction.net