climateprediction.net home page
Task 22313863

Task 22313863

Name oifs_43r3_bl_a1ni_2016092300_15_991_12208929_2
Workunit 12208929
Created 14 Feb 2023, 23:21:59 UTC
Sent 14 Feb 2023, 23:22:04 UTC
Report deadline 15 Apr 2023, 23:22:04 UTC
Received 18 Feb 2023, 20:41:23 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 1 (0x00000001) Unknown error code
Computer ID 1526327
Run time 12 hours 58 min 54 sec
CPU time 12 hours 50 min 17 sec
Validate state Invalid
Credit 1,209.60
Device peak FLOPS 2.88 GFLOPS
Application version OpenIFS 43r3 Baroclinic Lifecycle v1.11
x86_64-pc-linux-gnu
Peak working set size 5,066.81 MB
Peak swap size 5,930.79 MB
Peak disk usage 1,294.86 MB
Stderr
<core_client_version>7.18.1</core_client_version>
<![CDATA[
<message>
process exited with code 1 (0x1, -255)</message>
<stderr_txt>
0 +CPU= 45.497
  09:22:14 STEP  555 H= 138:45 +CPU= 45.601
  09:23:01 STEP  556 H= 139:00 +CPU= 46.992
  09:23:46 STEP  557 H= 139:15 +CPU= 45.302
  09:24:33 STEP  558 H= 139:30 +CPU= 46.304
  09:25:19 STEP  559 H= 139:45 +CPU= 46.410
  09:26:08 STEP  560 H= 140:00 +CPU= 48.279
  09:26:54 STEP  561 H= 140:15 +CPU= 46.336
  09:27:40 STEP  562 H= 140:30 +CPU= 45.642
  09:28:26 STEP  563 H= 140:45 +CPU= 46.197
  09:29:22 STEP  564 H= 141:00 +CPU= 56.123
Moving to projects directory: /etc/boinc-client/slots/0/ICMGGh7zg+000564
Moving to projects directory: /etc/boinc-client/slots/0/ICMSHh7zg+000564
Moving to projects directory: /etc/boinc-client/slots/0/ICMUAh7zg+000564
  09:30:04 STEP  565 H= 141:15 +CPU= 41.781
  09:30:43 STEP  566 H= 141:30 +CPU= 38.273
  09:31:22 STEP  567 H= 141:45 +CPU= 39.448
  09:32:02 STEP  568 H= 142:00 +CPU= 39.853
  09:32:40 STEP  569 H= 142:15 +CPU= 38.040
  09:33:18 STEP  570 H= 142:30 +CPU= 38.198
  09:34:02 STEP  571 H= 142:45 +CPU= 43.799
  09:34:49 STEP  572 H= 143:00 +CPU= 46.215
  09:35:35 STEP  573 H= 143:15 +CPU= 45.735
  09:36:21 STEP  574 H= 143:30 +CPU= 45.441
  09:37:07 STEP  575 H= 143:45 +CPU= 46.167
  09:38:10 STEP  576 H= 144:00 +CPU= 62.428
Moving to projects directory: /etc/boinc-client/slots/0/ICMGGh7zg+000576
Moving to projects directory: /etc/boinc-client/slots/0/ICMSHh7zg+000576
Moving to projects directory: /etc/boinc-client/slots/0/ICMUAh7zg+000576
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMGGh7zg+000480
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMSHh7zg+000480
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMUAh7zg+000480
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMGGh7zg+000492
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMSHh7zg+000492
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMUAh7zg+000492
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMGGh7zg+000504
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMSHh7zg+000504
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMUAh7zg+000504
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMGGh7zg+000516
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMSHh7zg+000516
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMUAh7zg+000516
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMGGh7zg+000528
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMSHh7zg+000528
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMUAh7zg+000528
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMGGh7zg+000540
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMSHh7zg+000540
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMUAh7zg+000540
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMGGh7zg+000552
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMSHh7zg+000552
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMUAh7zg+000552
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMGGh7zg+000564
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMSHh7zg+000564
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMUAh7zg+000564
Zipping up the intermediate file: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a1ni_2016092300_15_991_12208929_2_r641768228_5.zip
Uploading the intermediate file: upload_file_5.zip
  09:38:56 STEP  577 H= 144:15 +CPU= 46.357
  09:39:43 STEP  578 H= 144:30 +CPU= 46.374
  09:40:28 STEP  579 H= 144:45 +CPU= 45.336
  09:41:15 STEP  580 H= 145:00 +CPU= 46.201
  09:42:01 STEP  581 H= 145:15 +CPU= 46.607
  09:42:48 STEP  582 H= 145:30 +CPU= 46.698
  09:43:34 STEP  583 H= 145:45 +CPU= 45.341
  09:44:20 STEP  584 H= 146:00 +CPU= 46.695
  09:45:06 STEP  585 H= 146:15 +CPU= 45.315
  09:45:52 STEP  586 H= 146:30 +CPU= 46.175
  09:46:38 STEP  587 H= 146:45 +CPU= 45.659
  09:47:34 STEP  588 H= 147:00 +CPU= 56.512
Moving to projects directory: /etc/boinc-client/slots/0/ICMGGh7zg+000588
Moving to projects directory: /etc/boinc-client/slots/0/ICMSHh7zg+000588
Moving to projects directory: /etc/boinc-client/slots/0/ICMUAh7zg+000588
  09:48:21 STEP  589 H= 147:15 +CPU= 46.267
  09:49:08 STEP  590 H= 147:30 +CPU= 47.045
  09:49:54 STEP  591 H= 147:45 +CPU= 46.097
  09:50:42 STEP  592 H= 148:00 +CPU= 47.063
  09:51:28 STEP  593 H= 148:15 +CPU= 46.050
  09:52:13 STEP  594 H= 148:30 +CPU= 45.622
  09:52:59 STEP  595 H= 148:45 +CPU= 45.243
  09:53:46 STEP  596 H= 149:00 +CPU= 46.976
  09:54:31 STEP  597 H= 149:15 +CPU= 45.282
  09:55:18 STEP  598 H= 149:30 +CPU= 46.371
  09:56:04 STEP  599 H= 149:45 +CPU= 45.887
  09:57:02 STEP  600 H= 150:00 +CPU= 58.399
Moving to projects directory: /etc/boinc-client/slots/0/ICMGGh7zg+000600
Moving to projects directory: /etc/boinc-client/slots/0/ICMSHh7zg+000600
Moving to projects directory: /etc/boinc-client/slots/0/ICMUAh7zg+000600
  09:57:49 STEP  601 H= 150:15 +CPU= 46.611
  09:58:35 STEP  602 H= 150:30 +CPU= 46.263
  09:59:22 STEP  603 H= 150:45 +CPU= 46.642
  10:00:09 STEP  604 H= 151:00 +CPU= 46.715
  10:00:56 STEP  605 H= 151:15 +CPU= 46.883
  10:01:41 STEP  606 H= 151:30 +CPU= 44.853
  10:02:27 STEP  607 H= 151:45 +CPU= 46.259
  10:03:14 STEP  608 H= 152:00 +CPU= 46.217
  10:04:00 STEP  609 H= 152:15 +CPU= 46.056
  10:04:46 STEP  610 H= 152:30 +CPU= 46.092
  10:05:30 STEP  611 H= 152:45 +CPU= 44.423
  10:06:27 STEP  612 H= 153:00 +CPU= 56.871
Moving to projects directory: /etc/boinc-client/slots/0/ICMGGh7zg+000612
Moving to projects directory: /etc/boinc-client/slots/0/ICMSHh7zg+000612
Moving to projects directory: /etc/boinc-client/slots/0/ICMUAh7zg+000612
  10:07:15 STEP  613 H= 153:15 +CPU= 47.290
  10:08:02 STEP  614 H= 153:30 +CPU= 46.412
  10:08:48 STEP  615 H= 153:45 +CPU= 46.049
  10:09:33 STEP  616 H= 154:00 +CPU= 45.249
  10:10:20 STEP  617 H= 154:15 +CPU= 46.767
  10:11:06 STEP  618 H= 154:30 +CPU= 45.422
  10:11:51 STEP  619 H= 154:45 +CPU= 45.184
  10:12:38 STEP  620 H= 155:00 +CPU= 46.726
  10:13:24 STEP  621 H= 155:15 +CPU= 45.611
  10:14:09 STEP  622 H= 155:30 +CPU= 45.761
  10:14:56 STEP  623 H= 155:45 +CPU= 46.191
  10:15:59 STEP  624 H= 156:00 +CPU= 63.109
Moving to projects directory: /etc/boinc-client/slots/0/ICMGGh7zg+000624
Moving to projects directory: /etc/boinc-client/slots/0/ICMSHh7zg+000624
Moving to projects directory: /etc/boinc-client/slots/0/ICMUAh7zg+000624
  10:16:45 STEP  625 H= 156:15 +CPU= 45.897
  10:17:31 STEP  626 H= 156:30 +CPU= 45.982
  10:18:17 STEP  627 H= 156:45 +CPU= 45.796
  10:19:05 STEP  628 H= 157:00 +CPU= 47.125
  10:19:51 STEP  629 H= 157:15 +CPU= 46.503
  10:20:37 STEP  630 H= 157:30 +CPU= 45.576
  10:21:22 STEP  631 H= 157:45 +CPU= 44.801
  10:22:08 STEP  632 H= 158:00 +CPU= 45.791
  10:22:53 STEP  633 H= 158:15 +CPU= 45.499
  10:23:39 STEP  634 H= 158:30 +CPU= 45.533
  10:24:25 STEP  635 H= 158:45 +CPU= 45.916
  10:25:23 STEP  636 H= 159:00 +CPU= 57.609
Moving to projects directory: /etc/boinc-client/slots/0/ICMGGh7zg+000636
Moving to projects directory: /etc/boinc-client/slots/0/ICMSHh7zg+000636
Moving to projects directory: /etc/boinc-client/slots/0/ICMUAh7zg+000636
  10:26:09 STEP  637 H= 159:15 +CPU= 46.353
  10:26:55 STEP  638 H= 159:30 +CPU= 45.758
  10:27:42 STEP  639 H= 159:45 +CPU= 46.426
  10:28:28 STEP  640 H= 160:00 +CPU= 46.744
  10:29:15 STEP  641 H= 160:15 +CPU= 46.706
  10:30:01 STEP  642 H= 160:30 +CPU= 45.276
  10:30:47 STEP  643 H= 160:45 +CPU= 45.999
  10:31:33 STEP  644 H= 161:00 +CPU= 46.055
  10:32:18 STEP  645 H= 161:15 +CPU= 45.390
  10:33:04 STEP  646 H= 161:30 +CPU= 45.665
  10:33:51 STEP  647 H= 161:45 +CPU= 46.403
  10:34:48 STEP  648 H= 162:00 +CPU= 57.110
Moving to projects directory: /etc/boinc-client/slots/0/ICMGGh7zg+000648
Moving to projects directory: /etc/boinc-client/slots/0/ICMSHh7zg+000648
Moving to projects directory: /etc/boinc-client/slots/0/ICMUAh7zg+000648
  10:35:33 STEP  649 H= 162:15 +CPU= 44.699
  10:36:19 STEP  650 H= 162:30 +CPU= 46.596
  10:37:06 STEP  651 H= 162:45 +CPU= 47.098
  10:37:54 STEP  652 H= 163:00 +CPU= 47.152
  10:38:39 STEP  653 H= 163:15 +CPU= 45.001
  10:39:26 STEP  654 H= 163:30 +CPU= 46.640
  10:40:13 STEP  655 H= 163:45 +CPU= 47.106
  10:41:00 STEP  656 H= 164:00 +CPU= 47.378
  10:41:46 STEP  657 H= 164:15 +CPU= 45.643
  10:42:32 STEP  658 H= 164:30 +CPU= 45.828
  10:43:19 STEP  659 H= 164:45 +CPU= 46.729
  10:44:17 STEP  660 H= 165:00 +CPU= 58.178
Moving to projects directory: /etc/boinc-client/slots/0/ICMGGh7zg+000660
Moving to projects directory: /etc/boinc-client/slots/0/ICMSHh7zg+000660
Moving to projects directory: /etc/boinc-client/slots/0/ICMUAh7zg+000660
  10:45:03 STEP  661 H= 165:15 +CPU= 45.471
  10:45:49 STEP  662 H= 165:30 +CPU= 45.942
  10:46:34 STEP  663 H= 165:45 +CPU= 45.090
  10:47:21 STEP  664 H= 166:00 +CPU= 46.438
  10:48:08 STEP  665 H= 166:15 +CPU= 47.145
  10:48:54 STEP  666 H= 166:30 +CPU= 46.378
  10:49:42 STEP  667 H= 166:45 +CPU= 47.329
  10:50:30 STEP  668 H= 167:00 +CPU= 48.269
  10:51:17 STEP  669 H= 167:15 +CPU= 46.620
  10:52:04 STEP  670 H= 167:30 +CPU= 46.829
  10:52:50 STEP  671 H= 167:45 +CPU= 46.317
  10:53:56 STEP  672 H= 168:00 +CPU= 63.762
Moving to projects directory: /etc/boinc-client/slots/0/ICMGGh7zg+000672
Moving to projects directory: /etc/boinc-client/slots/0/ICMSHh7zg+000672
Moving to projects directory: /etc/boinc-client/slots/0/ICMUAh7zg+000672
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMGGh7zg+000576
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMSHh7zg+000576
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMUAh7zg+000576
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMGGh7zg+000588
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMSHh7zg+000588
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMUAh7zg+000588
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMGGh7zg+000600
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMSHh7zg+000600
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMUAh7zg+000600
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMGGh7zg+000612
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMSHh7zg+000612
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMUAh7zg+000612
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMGGh7zg+000624
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMSHh7zg+000624
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMUAh7zg+000624
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMGGh7zg+000636
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMSHh7zg+000636
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMUAh7zg+000636
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMGGh7zg+000648
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMSHh7zg+000648
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMUAh7zg+000648
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMGGh7zg+000660
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMSHh7zg+000660
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMUAh7zg+000660
Zipping up the intermediate file: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a1ni_2016092300_15_991_12208929_2_r641768228_6.zip
Uploading the intermediate file: upload_file_6.zip
  10:54:45 STEP  673 H= 168:15 +CPU= 47.108
  10:55:32 STEP  674 H= 168:30 +CPU= 46.879
  10:56:19 STEP  675 H= 168:45 +CPU= 47.155
  10:57:07 STEP  676 H= 169:00 +CPU= 47.765
  10:57:54 STEP  677 H= 169:15 +CPU= 46.636
  10:58:41 STEP  678 H= 169:30 +CPU= 46.590
  10:59:27 STEP  679 H= 169:45 +CPU= 46.005
  11:00:14 STEP  680 H= 170:00 +CPU= 47.232
  11:01:01 STEP  681 H= 170:15 +CPU= 46.755
  11:01:48 STEP  682 H= 170:30 +CPU= 46.733
  11:02:34 STEP  683 H= 170:45 +CPU= 45.932
  11:03:32 STEP  684 H= 171:00 +CPU= 58.262
Moving to projects directory: /etc/boinc-client/slots/0/ICMGGh7zg+000684
Moving to projects directory: /etc/boinc-client/slots/0/ICMSHh7zg+000684
Moving to projects directory: /etc/boinc-client/slots/0/ICMUAh7zg+000684
  11:04:19 STEP  685 H= 171:15 +CPU= 46.783
  11:05:06 STEP  686 H= 171:30 +CPU= 46.291
  11:05:54 STEP  687 H= 171:45 +CPU= 48.094
  11:06:42 STEP  688 H= 172:00 +CPU= 47.714
  11:07:29 STEP  689 H= 172:15 +CPU= 47.106
  11:08:16 STEP  690 H= 172:30 +CPU= 47.036
  11:09:02 STEP  691 H= 172:45 +CPU= 45.507
  11:09:48 STEP  692 H= 173:00 +CPU= 46.498
  11:10:35 STEP  693 H= 173:15 +CPU= 46.301
  11:11:21 STEP  694 H= 173:30 +CPU= 46.651
  11:12:08 STEP  695 H= 173:45 +CPU= 46.424
  11:13:07 STEP  696 H= 174:00 +CPU= 59.340
Moving to projects directory: /etc/boinc-client/slots/0/ICMGGh7zg+000696
Moving to projects directory: /etc/boinc-client/slots/0/ICMSHh7zg+000696
Moving to projects directory: /etc/boinc-client/slots/0/ICMUAh7zg+000696
  11:13:55 STEP  697 H= 174:15 +CPU= 47.706
  11:14:41 STEP  698 H= 174:30 +CPU= 45.876
  11:15:27 STEP  699 H= 174:45 +CPU= 46.144
  11:16:14 STEP  700 H= 175:00 +CPU= 46.599
  11:17:00 STEP  701 H= 175:15 +CPU= 46.089
  11:17:48 STEP  702 H= 175:30 +CPU= 47.059
  11:18:33 STEP  703 H= 175:45 +CPU= 45.762
  11:19:19 STEP  704 H= 176:00 +CPU= 45.489
  11:20:05 STEP  705 H= 176:15 +CPU= 45.484
  11:20:50 STEP  706 H= 176:30 +CPU= 45.771
  11:21:37 STEP  707 H= 176:45 +CPU= 46.094
  11:22:37 STEP  708 H= 177:00 +CPU= 59.928
Moving to projects directory: /etc/boinc-client/slots/0/ICMGGh7zg+000708
Moving to projects directory: /etc/boinc-client/slots/0/ICMSHh7zg+000708
Moving to projects directory: /etc/boinc-client/slots/0/ICMUAh7zg+000708
  11:23:24 STEP  709 H= 177:15 +CPU= 46.919
  11:24:10 STEP  710 H= 177:30 +CPU= 46.306
  11:24:56 STEP  711 H= 177:45 +CPU= 45.879
  11:25:44 STEP  712 H= 178:00 +CPU= 47.523
  11:26:30 STEP  713 H= 178:15 +CPU= 45.867
  11:27:17 STEP  714 H= 178:30 +CPU= 46.968
  11:28:02 STEP  715 H= 178:45 +CPU= 45.290
  11:28:49 STEP  716 H= 179:00 +CPU= 46.606
  11:29:36 STEP  717 H= 179:15 +CPU= 46.740
  11:30:23 STEP  718 H= 179:30 +CPU= 47.394
  11:31:08 STEP  719 H= 179:45 +CPU= 44.922
  11:32:15 STEP  720 H= 180:00 +CPU= 64.101
Moving to projects directory: /etc/boinc-client/slots/0/ICMGGh7zg+000720
Moving to projects directory: /etc/boinc-client/slots/0/ICMSHh7zg+000720
Moving to projects directory: /etc/boinc-client/slots/0/ICMUAh7zg+000720
  11:33:02 STEP  721 H= 180:15 +CPU= 47.061
  11:33:49 STEP  722 H= 180:30 +CPU= 47.094
  11:34:35 STEP  723 H= 180:45 +CPU= 45.670
  11:35:22 STEP  724 H= 181:00 +CPU= 46.690
  11:36:09 STEP  725 H= 181:15 +CPU= 46.756
  11:36:55 STEP  726 H= 181:30 +CPU= 46.803
  11:37:42 STEP  727 H= 181:45 +CPU= 46.275
  11:38:30 STEP  728 H= 182:00 +CPU= 48.418
  11:39:17 STEP  729 H= 182:15 +CPU= 46.671
  11:40:04 STEP  730 H= 182:30 +CPU= 46.410
  11:40:51 STEP  731 H= 182:45 +CPU= 47.525
  11:41:50 STEP  732 H= 183:00 +CPU= 58.791
Moving to projects directory: /etc/boinc-client/slots/0/ICMGGh7zg+000732
Moving to projects directory: /etc/boinc-client/slots/0/ICMSHh7zg+000732
Moving to projects directory: /etc/boinc-client/slots/0/ICMUAh7zg+000732
  11:42:38 STEP  733 H= 183:15 +CPU= 47.794
  11:43:26 STEP  734 H= 183:30 +CPU= 47.942
  11:44:13 STEP  735 H= 183:45 +CPU= 46.964
  11:45:02 STEP  736 H= 184:00 +CPU= 48.146
  11:45:49 STEP  737 H= 184:15 +CPU= 47.463
  11:46:37 STEP  738 H= 184:30 +CPU= 47.660
  11:47:24 STEP  739 H= 184:45 +CPU= 47.117
  11:48:12 STEP  740 H= 185:00 +CPU= 47.577
  11:48:58 STEP  741 H= 185:15 +CPU= 45.949
  11:49:45 STEP  742 H= 185:30 +CPU= 46.897
  11:50:31 STEP  743 H= 185:45 +CPU= 46.330
  11:51:30 STEP  744 H= 186:00 +CPU= 58.444
Moving to projects directory: /etc/boinc-client/slots/0/ICMGGh7zg+000744
Moving to projects directory: /etc/boinc-client/slots/0/ICMSHh7zg+000744
Moving to projects directory: /etc/boinc-client/slots/0/ICMUAh7zg+000744
  11:52:17 STEP  745 H= 186:15 +CPU= 46.943
  11:53:04 STEP  746 H= 186:30 +CPU= 46.516
  11:53:51 STEP  747 H= 186:45 +CPU= 46.766
  11:54:37 STEP  748 H= 187:00 +CPU= 46.874
  11:55:25 STEP  749 H= 187:15 +CPU= 47.594
  11:56:12 STEP  750 H= 187:30 +CPU= 47.034
  11:57:01 STEP  751 H= 187:45 +CPU= 48.040
  11:57:49 STEP  752 H= 188:00 +CPU= 48.310
  11:58:36 STEP  753 H= 188:15 +CPU= 46.918
  11:59:24 STEP  754 H= 188:30 +CPU= 48.077
  12:00:12 STEP  755 H= 188:45 +CPU= 47.619
  12:01:11 STEP  756 H= 189:00 +CPU= 58.698
Moving to projects directory: /etc/boinc-client/slots/0/ICMGGh7zg+000756
Moving to projects directory: /etc/boinc-client/slots/0/ICMSHh7zg+000756
Moving to projects directory: /etc/boinc-client/slots/0/ICMUAh7zg+000756
  12:01:58 STEP  757 H= 189:15 +CPU= 46.908
  12:02:45 STEP  758 H= 189:30 +CPU= 47.048
  12:03:32 STEP  759 H= 189:45 +CPU= 46.806
  12:04:21 STEP  760 H= 190:00 +CPU= 48.450
  12:05:08 STEP  761 H= 190:15 +CPU= 47.616
  12:05:54 STEP  762 H= 190:30 +CPU= 45.546
  12:06:42 STEP  763 H= 190:45 +CPU= 47.430
  12:07:30 STEP  764 H= 191:00 +CPU= 48.376
  12:08:19 STEP  765 H= 191:15 +CPU= 48.248
  12:09:06 STEP  766 H= 191:30 +CPU= 47.338
  12:09:52 STEP  767 H= 191:45 +CPU= 46.193
  12:10:57 STEP  768 H= 192:00 +CPU= 64.172
Moving to projects directory: /etc/boinc-client/slots/0/ICMGGh7zg+000768
Moving to projects directory: /etc/boinc-client/slots/0/ICMSHh7zg+000768
Moving to projects directory: /etc/boinc-client/slots/0/ICMUAh7zg+000768
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMGGh7zg+000672
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMSHh7zg+000672
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMUAh7zg+000672
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMGGh7zg+000684
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMSHh7zg+000684
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMUAh7zg+000684
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMGGh7zg+000696
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMSHh7zg+000696
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMUAh7zg+000696
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMGGh7zg+000708
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMSHh7zg+000708
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMUAh7zg+000708
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMGGh7zg+000720
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMSHh7zg+000720
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMUAh7zg+000720
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMGGh7zg+000732
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMSHh7zg+000732
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMUAh7zg+000732
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMGGh7zg+000744
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMSHh7zg+000744
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMUAh7zg+000744
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMGGh7zg+000756
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMSHh7zg+000756
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMUAh7zg+000756
Zipping up the intermediate file: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a1ni_2016092300_15_991_12208929_2_r641768228_7.zip
Uploading the intermediate file: upload_file_7.zip
  12:11:44 STEP  769 H= 192:15 +CPU= 46.474
  12:12:30 STEP  770 H= 192:30 +CPU= 46.617
  12:13:17 STEP  771 H= 192:45 +CPU= 46.452
  12:14:04 STEP  772 H= 193:00 +CPU= 46.802
  12:14:50 STEP  773 H= 193:15 +CPU= 45.914
  12:15:38 STEP  774 H= 193:30 +CPU= 48.080
  12:16:25 STEP  775 H= 193:45 +CPU= 47.243
  12:17:12 STEP  776 H= 194:00 +CPU= 46.477
  12:17:59 STEP  777 H= 194:15 +CPU= 47.128
  12:18:46 STEP  778 H= 194:30 +CPU= 47.142
  12:19:34 STEP  779 H= 194:45 +CPU= 47.181
  12:20:33 STEP  780 H= 195:00 +CPU= 59.046
Moving to projects directory: /etc/boinc-client/slots/0/ICMGGh7zg+000780
Moving to projects directory: /etc/boinc-client/slots/0/ICMSHh7zg+000780
Moving to projects directory: /etc/boinc-client/slots/0/ICMUAh7zg+000780
  12:21:20 STEP  781 H= 195:15 +CPU= 47.148
  12:22:08 STEP  782 H= 195:30 +CPU= 47.471
  12:22:54 STEP  783 H= 195:45 +CPU= 46.413
  12:23:42 STEP  784 H= 196:00 +CPU= 47.184
  12:24:28 STEP  785 H= 196:15 +CPU= 46.613
  12:25:14 STEP  786 H= 196:30 +CPU= 46.037
  12:26:02 STEP  787 H= 196:45 +CPU= 47.579
  12:26:49 STEP  788 H= 197:00 +CPU= 46.909
  12:27:34 STEP  789 H= 197:15 +CPU= 44.659
  12:28:21 STEP  790 H= 197:30 +CPU= 46.591
  12:29:08 STEP  791 H= 197:45 +CPU= 47.735
  12:30:08 STEP  792 H= 198:00 +CPU= 58.958
Moving to projects directory: /etc/boinc-client/slots/0/ICMGGh7zg+000792
Moving to projects directory: /etc/boinc-client/slots/0/ICMSHh7zg+000792
Moving to projects directory: /etc/boinc-client/slots/0/ICMUAh7zg+000792
  12:30:54 STEP  793 H= 198:15 +CPU= 46.630
  12:31:40 STEP  794 H= 198:30 +CPU= 45.107
  12:32:26 STEP  795 H= 198:45 +CPU= 46.609
  12:33:13 STEP  796 H= 199:00 +CPU= 46.450
  12:33:59 STEP  797 H= 199:15 +CPU= 46.254
  12:34:46 STEP  798 H= 199:30 +CPU= 46.963
  12:35:35 STEP  799 H= 199:45 +CPU= 48.557
  12:36:21 STEP  800 H= 200:00 +CPU= 46.273
  12:37:08 STEP  801 H= 200:15 +CPU= 47.093
  12:37:56 STEP  802 H= 200:30 +CPU= 46.904
  12:38:43 STEP  803 H= 200:45 +CPU= 46.443
  12:39:40 STEP  804 H= 201:00 +CPU= 57.166
Moving to projects directory: /etc/boinc-client/slots/0/ICMGGh7zg+000804
Moving to projects directory: /etc/boinc-client/slots/0/ICMSHh7zg+000804
Moving to projects directory: /etc/boinc-client/slots/0/ICMUAh7zg+000804
  12:40:27 STEP  805 H= 201:15 +CPU= 47.316
  12:41:14 STEP  806 H= 201:30 +CPU= 46.497
  12:42:02 STEP  807 H= 201:45 +CPU= 47.535
  12:42:50 STEP  808 H= 202:00 +CPU= 48.281
  12:43:36 STEP  809 H= 202:15 +CPU= 46.343
  12:44:22 STEP  810 H= 202:30 +CPU= 45.685
  12:45:09 STEP  811 H= 202:45 +CPU= 46.845
  12:45:58 STEP  812 H= 203:00 +CPU= 48.148
  12:46:44 STEP  813 H= 203:15 +CPU= 45.839
  12:47:29 STEP  814 H= 203:30 +CPU= 45.052
  12:48:15 STEP  815 H= 203:45 +CPU= 45.942
  12:49:17 STEP  816 H= 204:00 +CPU= 62.081
Moving to projects directory: /etc/boinc-client/slots/0/ICMGGh7zg+000816
Moving to projects directory: /etc/boinc-client/slots/0/ICMSHh7zg+000816
Moving to projects directory: /etc/boinc-client/slots/0/ICMUAh7zg+000816
  12:50:04 STEP  817 H= 204:15 +CPU= 47.024
  12:50:50 STEP  818 H= 204:30 +CPU= 45.211
  12:51:36 STEP  819 H= 204:45 +CPU= 46.585
  12:52:22 STEP  820 H= 205:00 +CPU= 45.719
  12:53:09 STEP  821 H= 205:15 +CPU= 46.638
  12:53:56 STEP  822 H= 205:30 +CPU= 46.652
  12:54:41 STEP  823 H= 205:45 +CPU= 45.173
  12:55:28 STEP  824 H= 206:00 +CPU= 47.471
  12:56:15 STEP  825 H= 206:15 +CPU= 45.900
  12:57:00 STEP  826 H= 206:30 +CPU= 45.174
  12:57:47 STEP  827 H= 206:45 +CPU= 46.722
  12:58:43 STEP  828 H= 207:00 +CPU= 56.431
Moving to projects directory: /etc/boinc-client/slots/0/ICMGGh7zg+000828
Moving to projects directory: /etc/boinc-client/slots/0/ICMSHh7zg+000828
Moving to projects directory: /etc/boinc-client/slots/0/ICMUAh7zg+000828
  12:59:30 STEP  829 H= 207:15 +CPU= 46.140
  13:00:16 STEP  830 H= 207:30 +CPU= 46.282
  13:01:02 STEP  831 H= 207:45 +CPU= 46.178
  13:01:50 STEP  832 H= 208:00 +CPU= 47.327
  13:02:36 STEP  833 H= 208:15 +CPU= 46.069
  13:03:21 STEP  834 H= 208:30 +CPU= 45.045
  13:04:08 STEP  835 H= 208:45 +CPU= 47.176
  13:04:56 STEP  836 H= 209:00 +CPU= 48.223
  13:05:43 STEP  837 H= 209:15 +CPU= 46.789
  13:06:30 STEP  838 H= 209:30 +CPU= 46.861
  13:07:17 STEP  839 H= 209:45 +CPU= 46.536
  13:08:17 STEP  840 H= 210:00 +CPU= 59.741
Moving to projects directory: /etc/boinc-client/slots/0/ICMGGh7zg+000840
Moving to projects directory: /etc/boinc-client/slots/0/ICMSHh7zg+000840
Moving to projects directory: /etc/boinc-client/slots/0/ICMUAh7zg+000840
  13:09:04 STEP  841 H= 210:15 +CPU= 46.814
  13:09:50 STEP  842 H= 210:30 +CPU= 45.813
  13:10:37 STEP  843 H= 210:45 +CPU= 47.097
  13:11:25 STEP  844 H= 211:00 +CPU= 47.759
  13:12:12 STEP  845 H= 211:15 +CPU= 46.725
  13:12:58 STEP  846 H= 211:30 +CPU= 46.123
  13:13:44 STEP  847 H= 211:45 +CPU= 46.506
  13:14:31 STEP  848 H= 212:00 +CPU= 46.730
  13:15:19 STEP  849 H= 212:15 +CPU= 47.624
  13:16:06 STEP  850 H= 212:30 +CPU= 46.967
  13:16:53 STEP  851 H= 212:45 +CPU= 47.083
  13:17:53 STEP  852 H= 213:00 +CPU= 59.449
Moving to projects directory: /etc/boinc-client/slots/0/ICMGGh7zg+000852
Moving to projects directory: /etc/boinc-client/slots/0/ICMSHh7zg+000852
Moving to projects directory: /etc/boinc-client/slots/0/ICMUAh7zg+000852
  13:18:40 STEP  853 H= 213:15 +CPU= 47.095
  13:19:27 STEP  854 H= 213:30 +CPU= 46.961
  13:20:14 STEP  855 H= 213:45 +CPU= 46.917
  13:21:02 STEP  856 H= 214:00 +CPU= 47.548
  13:21:50 STEP  857 H= 214:15 +CPU= 47.765
  13:22:37 STEP  858 H= 214:30 +CPU= 46.923
  13:23:26 STEP  859 H= 214:45 +CPU= 48.453
  13:24:14 STEP  860 H= 215:00 +CPU= 47.895
  13:25:02 STEP  861 H= 215:15 +CPU= 48.368
  13:25:50 STEP  862 H= 215:30 +CPU= 47.460
  13:26:38 STEP  863 H= 215:45 +CPU= 47.506
  13:27:44 STEP  864 H= 216:00 +CPU= 65.124
Moving to projects directory: /etc/boinc-client/slots/0/ICMGGh7zg+000864
Moving to projects directory: /etc/boinc-client/slots/0/ICMSHh7zg+000864
Moving to projects directory: /etc/boinc-client/slots/0/ICMUAh7zg+000864
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMGGh7zg+000768
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMSHh7zg+000768
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMUAh7zg+000768
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMGGh7zg+000780
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMSHh7zg+000780
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMUAh7zg+000780
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMGGh7zg+000792
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMSHh7zg+000792
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMUAh7zg+000792
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMGGh7zg+000804
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMSHh7zg+000804
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMUAh7zg+000804
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMGGh7zg+000816
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMSHh7zg+000816
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMUAh7zg+000816
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMGGh7zg+000828
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMSHh7zg+000828
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMUAh7zg+000828
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMGGh7zg+000840
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMSHh7zg+000840
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMUAh7zg+000840
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMGGh7zg+000852
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMSHh7zg+000852
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMUAh7zg+000852
Zipping up the intermediate file: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a1ni_2016092300_15_991_12208929_2_r641768228_8.zip
Uploading the intermediate file: upload_file_8.zip
  13:28:32 STEP  865 H= 216:15 +CPU= 48.141
  13:29:21 STEP  866 H= 216:30 +CPU= 48.797
  13:30:08 STEP  867 H= 216:45 +CPU= 46.730
  13:30:55 STEP  868 H= 217:00 +CPU= 47.220
  13:31:42 STEP  869 H= 217:15 +CPU= 46.518
  13:32:29 STEP  870 H= 217:30 +CPU= 46.817
  13:33:15 STEP  871 H= 217:45 +CPU= 46.293
  13:34:03 STEP  872 H= 218:00 +CPU= 47.566
  13:34:51 STEP  873 H= 218:15 +CPU= 47.827
  13:35:39 STEP  874 H= 218:30 +CPU= 47.846
  13:36:27 STEP  875 H= 218:45 +CPU= 48.597
  13:37:27 STEP  876 H= 219:00 +CPU= 59.668
Moving to projects directory: /etc/boinc-client/slots/0/ICMGGh7zg+000876
Moving to projects directory: /etc/boinc-client/slots/0/ICMSHh7zg+000876
Moving to projects directory: /etc/boinc-client/slots/0/ICMUAh7zg+000876
  13:38:14 STEP  877 H= 219:15 +CPU= 46.653
  13:39:01 STEP  878 H= 219:30 +CPU= 47.062
  13:39:48 STEP  879 H= 219:45 +CPU= 46.625
  13:40:38 STEP  880 H= 220:00 +CPU= 49.693
  13:41:25 STEP  881 H= 220:15 +CPU= 46.843
  13:42:12 STEP  882 H= 220:30 +CPU= 47.396
  13:43:00 STEP  883 H= 220:45 +CPU= 47.770
  13:43:47 STEP  884 H= 221:00 +CPU= 46.868
  13:44:36 STEP  885 H= 221:15 +CPU= 48.811
  13:45:23 STEP  886 H= 221:30 +CPU= 47.404
  13:46:11 STEP  887 H= 221:45 +CPU= 47.827
  13:47:12 STEP  888 H= 222:00 +CPU= 60.125
Moving to projects directory: /etc/boinc-client/slots/0/ICMGGh7zg+000888
Moving to projects directory: /etc/boinc-client/slots/0/ICMSHh7zg+000888
Moving to projects directory: /etc/boinc-client/slots/0/ICMUAh7zg+000888
  13:47:58 STEP  889 H= 222:15 +CPU= 46.449
  13:48:47 STEP  890 H= 222:30 +CPU= 48.239
  13:49:35 STEP  891 H= 222:45 +CPU= 48.313
  13:50:24 STEP  892 H= 223:00 +CPU= 49.186
  13:51:12 STEP  893 H= 223:15 +CPU= 47.573
  13:51:59 STEP  894 H= 223:30 +CPU= 46.490
  13:52:46 STEP  895 H= 223:45 +CPU= 46.754
  13:53:35 STEP  896 H= 224:00 +CPU= 48.778
  13:54:23 STEP  897 H= 224:15 +CPU= 48.030
  13:55:12 STEP  898 H= 224:30 +CPU= 48.960
  13:56:01 STEP  899 H= 224:45 +CPU= 48.597
  13:57:00 STEP  900 H= 225:00 +CPU= 59.207
Moving to projects directory: /etc/boinc-client/slots/0/ICMGGh7zg+000900
Moving to projects directory: /etc/boinc-client/slots/0/ICMSHh7zg+000900
Moving to projects directory: /etc/boinc-client/slots/0/ICMUAh7zg+000900
  13:57:47 STEP  901 H= 225:15 +CPU= 47.032
  13:58:35 STEP  902 H= 225:30 +CPU= 47.944
  13:59:23 STEP  903 H= 225:45 +CPU= 47.113
  14:00:11 STEP  904 H= 226:00 +CPU= 48.186
  14:00:59 STEP  905 H= 226:15 +CPU= 47.706
  14:01:47 STEP  906 H= 226:30 +CPU= 47.983
  14:02:33 STEP  907 H= 226:45 +CPU= 46.456
  14:03:20 STEP  908 H= 227:00 +CPU= 46.571
  14:04:07 STEP  909 H= 227:15 +CPU= 46.827
  14:04:54 STEP  910 H= 227:30 +CPU= 46.429
  14:05:41 STEP  911 H= 227:45 +CPU= 47.207
  14:06:48 STEP  912 H= 228:00 +CPU= 65.102
Moving to projects directory: /etc/boinc-client/slots/0/ICMGGh7zg+000912
Moving to projects directory: /etc/boinc-client/slots/0/ICMSHh7zg+000912
Moving to projects directory: /etc/boinc-client/slots/0/ICMUAh7zg+000912
  14:07:36 STEP  913 H= 228:15 +CPU= 47.866
  14:08:24 STEP  914 H= 228:30 +CPU= 47.945
  14:09:12 STEP  915 H= 228:45 +CPU= 47.773
  14:10:01 STEP  916 H= 229:00 +CPU= 49.446
  14:10:48 STEP  917 H= 229:15 +CPU= 46.495
  14:11:34 STEP  918 H= 229:30 +CPU= 46.571
  14:12:22 STEP  919 H= 229:45 +CPU= 47.504
  14:13:10 STEP  920 H= 230:00 +CPU= 47.748
  14:13:58 STEP  921 H= 230:15 +CPU= 48.375
  14:14:45 STEP  922 H= 230:30 +CPU= 46.583
  14:15:32 STEP  923 H= 230:45 +CPU= 46.751
  14:16:32 STEP  924 H= 231:00 +CPU= 59.599
Moving to projects directory: /etc/boinc-client/slots/0/ICMGGh7zg+000924
Moving to projects directory: /etc/boinc-client/slots/0/ICMSHh7zg+000924
Moving to projects directory: /etc/boinc-client/slots/0/ICMUAh7zg+000924
  14:17:19 STEP  925 H= 231:15 +CPU= 46.757
  14:18:05 STEP  926 H= 231:30 +CPU= 46.251
  14:18:53 STEP  927 H= 231:45 +CPU= 48.067
  14:19:42 STEP  928 H= 232:00 +CPU= 48.783
  14:20:30 STEP  929 H= 232:15 +CPU= 47.475
  14:21:16 STEP  930 H= 232:30 +CPU= 46.352
  14:22:04 STEP  931 H= 232:45 +CPU= 47.668
  14:22:51 STEP  932 H= 233:00 +CPU= 47.285
  14:23:38 STEP  933 H= 233:15 +CPU= 46.711
  14:24:25 STEP  934 H= 233:30 +CPU= 46.627
  14:25:12 STEP  935 H= 233:45 +CPU= 47.401
  14:26:11 STEP  936 H= 234:00 +CPU= 58.767
Moving to projects directory: /etc/boinc-client/slots/0/ICMGGh7zg+000936
Moving to projects directory: /etc/boinc-client/slots/0/ICMSHh7zg+000936
Moving to projects directory: /etc/boinc-client/slots/0/ICMUAh7zg+000936
  14:26:59 STEP  937 H= 234:15 +CPU= 47.389
  14:27:46 STEP  938 H= 234:30 +CPU= 46.977
  14:28:33 STEP  939 H= 234:45 +CPU= 46.737
  14:29:21 STEP  940 H= 235:00 +CPU= 48.779
  14:30:09 STEP  941 H= 235:15 +CPU= 47.399
  14:30:57 STEP  942 H= 235:30 +CPU= 47.935
  14:31:44 STEP  943 H= 235:45 +CPU= 46.922
  14:32:31 STEP  944 H= 236:00 +CPU= 47.171
  14:33:18 STEP  945 H= 236:15 +CPU= 46.507
  14:34:05 STEP  946 H= 236:30 +CPU= 46.879
  14:34:52 STEP  947 H= 236:45 +CPU= 46.606
  14:35:50 STEP  948 H= 237:00 +CPU= 58.628
Moving to projects directory: /etc/boinc-client/slots/0/ICMGGh7zg+000948
Moving to projects directory: /etc/boinc-client/slots/0/ICMSHh7zg+000948
Moving to projects directory: /etc/boinc-client/slots/0/ICMUAh7zg+000948
  14:36:37 STEP  949 H= 237:15 +CPU= 46.591
  14:37:25 STEP  950 H= 237:30 +CPU= 47.567
  14:38:13 STEP  951 H= 237:45 +CPU= 47.185
  14:39:00 STEP  952 H= 238:00 +CPU= 47.412
  14:39:46 STEP  953 H= 238:15 +CPU= 45.962
  14:40:33 STEP  954 H= 238:30 +CPU= 46.676
  14:41:21 STEP  955 H= 238:45 +CPU= 47.371
  14:42:09 STEP  956 H= 239:00 +CPU= 47.946
  14:42:56 STEP  957 H= 239:15 +CPU= 47.085
  14:43:42 STEP  958 H= 239:30 +CPU= 46.417
  14:44:28 STEP  959 H= 239:45 +CPU= 45.990
  14:45:34 STEP  960 H= 240:00 +CPU= 63.199
Moving to projects directory: /etc/boinc-client/slots/0/ICMGGh7zg+000960
Moving to projects directory: /etc/boinc-client/slots/0/ICMSHh7zg+000960
Moving to projects directory: /etc/boinc-client/slots/0/ICMUAh7zg+000960
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMGGh7zg+000864
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMSHh7zg+000864
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMUAh7zg+000864
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMGGh7zg+000876
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMSHh7zg+000876
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMUAh7zg+000876
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMGGh7zg+000888
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMSHh7zg+000888
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMUAh7zg+000888
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMGGh7zg+000900
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMSHh7zg+000900
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMUAh7zg+000900
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMGGh7zg+000912
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMSHh7zg+000912
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMUAh7zg+000912
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMGGh7zg+000924
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMSHh7zg+000924
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMUAh7zg+000924
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMGGh7zg+000936
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMSHh7zg+000936
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMUAh7zg+000936
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMGGh7zg+000948
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMSHh7zg+000948
Adding to the zip: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208929/ICMUAh7zg+000948
Zipping up the intermediate file: /etc/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a1ni_2016092300_15_991_12208929_2_r641768228_9.zip
Uploading the intermediate file: upload_file_9.zip
Uploading trickle at timestep: 864000
  14:46:22 STEP  961 H= 240:15 +CPU= 47.634
  14:47:09 STEP  962 H= 240:30 +CPU= 47.676
  14:47:56 STEP  963 H= 240:45 +CPU= 46.485
  14:48:45 STEP  964 H= 241:00 +CPU= 48.520
  14:49:31 STEP  965 H= 241:15 +CPU= 46.572
  14:50:19 STEP  966 H= 241:30 +CPU= 47.571
  14:51:06 STEP  967 H= 241:45 +CPU= 46.788
  14:51:54 STEP  968 H= 242:00 +CPU= 47.694
  14:52:40 STEP  969 H= 242:15 +CPU= 45.997
  14:53:26 STEP  970 H= 242:30 +CPU= 46.061
  14:54:12 STEP  971 H= 242:45 +CPU= 46.195
  14:55:11 STEP  972 H= 243:00 +CPU= 58.695
Moving to projects directory: /etc/boinc-client/slots/0/ICMGGh7zg+000972
Moving to projects directory: /etc/boinc-client/slots/0/ICMSHh7zg+000972
Moving to projects directory: /etc/boinc-client/slots/0/ICMUAh7zg+000972
  14:55:58 STEP  973 H= 243:15 +CPU= 46.559
  14:56:45 STEP  974 H= 243:30 +CPU= 47.341
  14:57:32 STEP  975 H= 243:45 +CPU= 46.771
  14:58:20 STEP  976 H= 244:00 +CPU= 47.699
  14:59:08 STEP  977 H= 244:15 +CPU= 47.575
  14:59:54 STEP  978 H= 244:30 +CPU= 46.372
  15:00:41 STEP  979 H= 244:45 +CPU= 46.472
  15:01:28 STEP  980 H= 245:00 +CPU= 46.935
  15:02:14 STEP  981 H= 245:15 +CPU= 46.560
  15:03:01 STEP  982 H= 245:30 +CPU= 47.160
  MAX U WIND=   250.183569066263     
  15:03:48 STEP  983 H= 245:45 +CPU= 46.226
[EC_DRHOOK:tommcfarland-500-314:1:1:136153:136153] [20230218:150352:1676750632:46353.880] [signal_drhook@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1538] Received signal#8 (SIGFPE) :: 2686MB (heap), 5067MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:tommcfarland-500-314:1:1:136153:136153] [20230218:150352:1676750632:46353.924] [signal_drhook@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:tommcfarland-500-314:1:1:136153:136153] [20230218:150352:1676750632:46353.924] [signal_drhook@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f0c0 (old at (nil))
[EC_DRHOOK:tommcfarland-500-314:1:1:136153:136153] [20230218:150352:1676750632:46353.924] [signal_drhook@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1617] Signal#8 was caused by floating-point overflow [memaddr=0x1cc4a8f], nsigs = 1
[EC_DRHOOK:tommcfarland-500-314:1:1:136153:136153] [20230218:150352:1676750632:46353.924] [signal_drhook@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#8, nsigs = 1
[EC_DRHOOK:tommcfarland-500-314:1:1:136153:136153] [20230218:150352:1676750632:46353.924] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3843] 2686 MB (maxheap), 5067 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:tommcfarland-500-314:1:1:136153:136153] [20230218:150352:1676750632:46353.924] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:tommcfarland-500-314:1:1:136153:136153] [20230218:150352:1676750632:46353.952] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:tommcfarland-500-314:1:1:136153:136153] [20230218:150352:1676750632:46353.955] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:tommcfarland-500-314:1:1:136153:136153] [20230218:150352:1676750632:46353.955] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:tommcfarland-500-314:1:1:136153:136153] [20230218:150352:1676750632:46353.956] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:tommcfarland-500-314:1:1:136153:136153] [20230218:150352:1676750632:46353.956] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] :       CNT4 
[EC_DRHOOK:tommcfarland-500-314:1:1:136153:136153] [20230218:150352:1676750632:46353.956] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] :        DYNFPOS 
[EC_DRHOOK:tommcfarland-500-314:1:1:136153:136153] [20230218:150352:1676750632:46353.956] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] :         STEPO_FPOS 
[EC_DRHOOK:tommcfarland-500-314:1:1:136153:136153] [20230218:150352:1676750632:46353.956] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] :          SCAN2M_VPOS 
[EC_DRHOOK:tommcfarland-500-314:1:1:136153:136153] [20230218:150352:1676750632:46353.956] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] :           >OMP-SCAN2M VPOS          (1435) 
[EC_DRHOOK:tommcfarland-500-314:1:1:136153:136153] [20230218:150352:1676750632:46353.956] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] :            VPOS 
[EC_DRHOOK:tommcfarland-500-314:1:1:136153:136153] [20230218:150352:1676750632:46353.956] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] :             POS 
[EC_DRHOOK:tommcfarland-500-314:1:1:136153:136153] [20230218:150352:1676750632:46353.956] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] :              GPRH 
[EC_DRHOOK:tommcfarland-500-314:1:1:136153:136153] [20230218:150353:1676750633:46354.956] [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:240] Backtrace(s) for program '/etc/boinc-client/slots/0/oifs_43r3_model.exe' : sigcontextptr=0x7ffe1a88fc50
[EC_DRHOOK:tommcfarland-500-314:1:1:136153:136153] [20230218:150353:1676750633:46354.956] [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:277] Backtrace (size = 20) with addr2line-cmd
[EC_DRHOOK:tommcfarland-500-314:1:1:136153:136153] [20230218:150353:1676750633:46354.956] [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:302] /usr/bin/addr2line -fs -e '/etc/boinc-client/slots/0/oifs_43r3_model.exe' 0x838803 0x838e09 0x1d8bb20 0x1cc4a8f 0x144b066 0x599499 0x576413 0x652b1a 0x4a9238 0x4a27a3 0x4a0968 0x49dc29 0x49ce72 0x49cd68 0x4078ad 0x40276f 0x40270d 0x1d8287a 0x1d840d7 0x4025e5
[EC_DRHOOK:tommcfarland-500-314:1:1:136153:136153] [20230218:150353:1676750633:46354.956] [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:319] [0]: [0x838803] : LinuxTraceBack() at linuxtrbk.c:264
[EC_DRHOOK:tommcfarland-500-314:1:1:136153:136153] [20230218:150353:1676750633:46354.956] [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:319] [1]: [0x838e09] : LinuxTraceBack() at linuxtrbk.c:236
[EC_DRHOOK:tommcfarland-500-314:1:1:136153:136153] [20230218:150353:1676750633:46354.956] [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:319] [2]: [0x1d8bb20] : __restore_rt() at libc_sigaction.o:?
[EC_DRHOOK:tommcfarland-500-314:1:1:136153:136153] [20230218:150353:1676750633:46354.956] [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:319] [3]: [0x1cc4a8f] : __libm_exp_e7() at ??:?
[EC_DRHOOK:tommcfarland-500-314:1:1:136153:136153] [20230218:150353:1676750633:46354.956] [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:319] [4]: [0x144b066] : gprh_() at ??:?
[EC_DRHOOK:tommcfarland-500-314:1:1:136153:136153] [20230218:150353:1676750633:46354.956] [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:319] [5]: [0x599499] : pos_() at ??:?
[EC_DRHOOK:tommcfarland-500-314:1:1:136153:136153] [20230218:150353:1676750633:46354.956] [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:319] [6]: [0x576413] : vpos_() at ??:?
[EC_DRHOOK:tommcfarland-500-314:1:1:136153:136153] [20230218:150353:1676750633:46354.956] [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:319] [7]: [0x652b1a] : scan2m_vpos_() at ??:?
[EC_DRHOOK:tommcfarland-500-314:1:1:136153:136153] [20230218:150353:1676750633:46354.956] [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:319] [8]: [0x4a9238] : stepo_fpos_() at ??:?
[EC_DRHOOK:tommcfarland-500-314:1:1:136153:136153] [20230218:150353:1676750633:46354.956] [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:319] [9]: [0x4a27a3] : dynfpos_() at ??:?
[EC_DRHOOK:tommcfarland-500-314:1:1:136153:136153] [20230218:150353:1676750633:46354.956] [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:319] [10]: [0x4a0968] : cnt4_() at ??:?
[EC_DRHOOK:tommcfarland-500-314:1:1:136153:136153] [20230218:150353:1676750633:46354.956] [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:319] [11]: [0x49dc29] : cnt3_() at ??:?
[EC_DRHOOK:tommcfarland-500-314:1:1:136153:136153] [20230218:150353:1676750633:46354.956] [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:319] [12]: [0x49ce72] : cnt2_() at ??:?
[EC_DRHOOK:tommcfarland-500-314:1:1:136153:136153] [20230218:150353:1676750633:46354.956] [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:319] [13]: [0x49cd68] : cnt1_() at ??:?
[EC_DRHOOK:tommcfarland-500-314:1:1:136153:136153] [20230218:150353:1676750633:46354.956] [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:319] [14]: [0x4078ad] : cnt0_() at ??:?
[EC_DRHOOK:tommcfarland-500-314:1:1:136153:136153] [20230218:150353:1676750633:46354.956] [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:319] [15]: [0x40276f] : MAIN__() at ??:?
[EC_DRHOOK:tommcfarland-500-314:1:1:136153:136153] [20230218:150353:1676750633:46354.956] [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:319] [16]: [0x40270d] : main() at ??:?
[EC_DRHOOK:tommcfarland-500-314:1:1:136153:136153] [20230218:150353:1676750633:46354.956] [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:319] [17]: [0x1d8287a] : __libc_start_call_main() at libc-start.o:?
[EC_DRHOOK:tommcfarland-500-314:1:1:136153:136153] [20230218:150353:1676750633:46354.956] [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:319] [18]: [0x1d840d7] : __libc_start_main_impl() at ??:?
[EC_DRHOOK:tommcfarland-500-314:1:1:136153:136153] [20230218:150353:1676750633:46354.956] [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:319] [19]: [0x4025e5] : _start() at ??:?
[EC_DRHOOK:tommcfarland-500-314:1:1:136153:136153] [20230218:150353:1676750633:46354.956] [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:370] End of backtrace(s)
 Process            1  thread            1  calling linux_trbk from intel_trbk()
  [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:240] Backtrace(s) for program '/etc/boinc-client/slots/0/oifs_43r3_model.exe' : sigcontextptr=0x7ffe1a88fb60
  [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:277] Backtrace (size = 21) with addr2line-cmd
  [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:302] /usr/bin/addr2line -fs -e '/etc/boinc-client/slots/0/oifs_43r3_model.exe' 0x838803 0x838e09 0x820d70 0x1d8bb20 0x1cc4a8f 0x144b066 0x599499 0x576413 0x652b1a 0x4a9238 0x4a27a3 0x4a0968 0x49dc29 0x49ce72 0x49cd68 0x4078ad 0x40276f 0x40270d 0x1d8287a 0x1d840d7 0x4025e5
  [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:319] [0]: [0x838803] : LinuxTraceBack() at linuxtrbk.c:264
  [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:319] [1]: [0x838e09] : LinuxTraceBack() at linuxtrbk.c:236
  [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:319] [2]: [0x820d70] : signal_drhook() at drhook.c:1734
  [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:319] [3]: [0x1d8bb20] : __restore_rt() at libc_sigaction.o:?
  [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:319] [4]: [0x1cc4a8f] : __libm_exp_e7() at ??:?
  [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:319] [5]: [0x144b066] : gprh_() at ??:?
  [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:319] [6]: [0x599499] : pos_() at ??:?
  [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:319] [7]: [0x576413] : vpos_() at ??:?
  [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:319] [8]: [0x652b1a] : scan2m_vpos_() at ??:?
  [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:319] [9]: [0x4a9238] : stepo_fpos_() at ??:?
  [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:319] [10]: [0x4a27a3] : dynfpos_() at ??:?
  [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:319] [11]: [0x4a0968] : cnt4_() at ??:?
  [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:319] [12]: [0x49dc29] : cnt3_() at ??:?
  [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:319] [13]: [0x49ce72] : cnt2_() at ??:?
  [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:319] [14]: [0x49cd68] : cnt1_() at ??:?
  [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:319] [15]: [0x4078ad] : cnt0_() at ??:?
  [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:319] [16]: [0x40276f] : MAIN__() at ??:?
  [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:319] [17]: [0x40270d] : main() at ??:?
  [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:319] [18]: [0x1d8287a] : __libc_start_call_main() at libc-start.o:?
  [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:319] [19]: [0x1d840d7] : __libc_start_main_impl() at ??:?
  [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:319] [20]: [0x4025e5] : _start() at ??:?
  [LinuxTraceBack@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/utilities/linuxtrbk.c:370] End of backtrace(s)
[EC_DRHOOK:tommcfarland-500-314:1:1:136153:136153] [20230218:150354:1676750634:46355.596] [signal_drhook@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#8, nsigs = 1
[EC_DRHOOK:tommcfarland-500-314:1:1:136153:136153] [20230218:150354:1676750634:46355.596] [signal_drhook@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x401140 for signal#8, nsigs = 1
forrtl: error (72): floating overflow
Image              PC                Routine            Line        Source             
oifs_43r3_model.e  000000000081FCF1  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001D8BB20  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001CC4A8F  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000144B066  gprh_                     128  gprh.F90
oifs_43r3_model.e  0000000000599499  pos_                     1103  pos.F90
oifs_43r3_model.e  0000000000576413  vpos_                     227  vpos.F90
oifs_43r3_model.e  0000000000652B1A  scan2m_vpos_              117  scan2m_vpos.F90
oifs_43r3_model.e  00000000004A9238  stepo_fpos_               170  stepo_fpos.F90
oifs_43r3_model.e  00000000004A27A3  dynfpos_                  170  dynfpos.F90
oifs_43r3_model.e  00000000004A0968  cnt4_                     947  cnt4.F90
oifs_43r3_model.e  000000000049DC29  cnt3_                     267  cnt3.F90
oifs_43r3_model.e  000000000049CE72  cnt2_                      88  cnt2.F90
oifs_43r3_model.e  000000000049CD68  cnt1_                      92  cnt1.F90
oifs_43r3_model.e  00000000004078AD  cnt0_                     146  cnt0.F90
oifs_43r3_model.e  000000000040276F  MAIN__                     96  master.F90
oifs_43r3_model.e  000000000040270D  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001D8287A  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001D840D7  Unknown               Unknown  Unknown
oifs_43r3_model.e  00000000004025E5  Unknown               Unknown  Unknown
..The child process has been killed with signal: 6
>>> Printing last 70 lines from file: NODE.001_01

 14:57:32 STEP  975 H= 243:45 +CPU= 46.771
 NSTEP =   976 STEPO   0AAA00AAA
 14:58:20 STEP  976 H= 244:00 +CPU= 47.699
 NSTEP =   977 STEPO   0AAA00AAA
 14:59:08 STEP  977 H= 244:15 +CPU= 47.575
 NSTEP =   978 STEPO   0AAA00AAA
 14:59:54 STEP  978 H= 244:30 +CPU= 46.372
 NSTEP =   979 STEPO   0AAA00AAA
 15:00:41 STEP  979 H= 244:45 +CPU= 46.472
ISTEP=     980 RSOLINC= 1361.4482 RI0= 1365.0000 ZI0= 1361.4482 RIP0= 1361.4482
ISTEP=     980 ZI0= 1361.4482 ZSEASON= 1.000000000 REA=149597870000.00 RDEASO=149597870000.00
 NSTEP =   980 STEPO   0AAA00AAA
 15:01:28 STEP  980 H= 245:00 +CPU= 46.935
 NSTEP =   981 STEPO   0AAA00AAA
 15:02:14 STEP  981 H= 245:15 +CPU= 46.560
 NSTEP =   982 STEPO   0AAA00AAA
 15:03:01 STEP  982 H= 245:30 +CPU= 47.160
 NSTEP =   983 STEPO   0AAA00AAA
 15:03:48 STEP  983 H= 245:45 +CPU= 46.226
  DATE= 2016    9   23
ISTEP=     984: Reset RSOLINC= 1361.4482 RIP0= 1361.4482
 SU_GHGCLIM, RCARDI,RCH4,RN2O,RCFC11,RCFC12,RNO2:   6.175516699499999E-004
  1.087291259320000E-006  4.989444016862500E-007  3.778881179906250E-009
  2.099153872040000E-009  7.940283000000001E-011
 suecaec NAERMACC =  1
 UPDT RAESHDU =  2000.0
 SUECAEC: GLOBAL AVERAGE VOLCANIC SULPHATE FOR (KINDAT,KMINUT)     20160923
         450  9.999999999999998E-005
 NORMS AT NSTEP CNT4  984
 SPECTRAL NORMS -   LOG(PREHYDS)    0.115129543146842E+02
 LEV     VORTICITY             DIVERGENCE           TEMPERATURE           KINETIC ENERGY    
 AVE 0.411376827956828E-04 0.425802186732228E-04 0.215066756129561E+03 0.513969122204226E+03
 GPNORM  HUMIDITY            AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.508594084484021E-02 0.999999999689129E-08 0.628295842054525E-01
 GPNORM  LIQUID WATER        AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.353589250919219E-05 0.000000000000000E+00 0.192934643263531E-01
 GPNORM  ICE WATER           AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.434917364653224E-04 0.000000000000000E+00 0.212139357614712E-01
 GPNORM SNOW                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.890003328650843E-04 0.000000000000000E+00 0.288368897498959E-01
 GPNORM RAIN                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.584864597053764E-05 0.000000000000000E+00 0.102933471205971E-01
 GPNORM  CLOUD FRACTION      AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.224234670050514E+00 0.000000000000000E+00 0.100000000000000E+01
 GPNORM CLW                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.334388947166215E-05 0.000000000000000E+00 0.178183308720693E-01
 GPNORM CIW                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.417498658860563E-04 0.000000000000000E+00 0.229345889856057E-01
 GPNORM CC                   AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.223901788967756E+00 0.000000000000000E+00 0.100000000000000E+01
 GPNORM RFL                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.197734183270650E-04 0.000000000000000E+00 0.278356103069769E-01
 GPNORM SFL                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.177014486394232E-04 0.000000000000000E+00 0.778204679367807E-02
 GPNORM CRFL                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.408440521014605E-05 0.000000000000000E+00 0.192102694389275E-01
 GPNORM CSFL                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.157017586900140E-05 0.000000000000000E+00 0.114351522307752E-01
 GPNORM PFRC                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.200731997179789E+00 0.000000000000000E+00 0.100000000000000E+01
 NSTEP =   984 STEPO   A00000000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =   984 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+000984 MODE=w
 IO-STREAM CLOSED - ICMGGh7zg+000984
 MPI-TASK:   1 -    304132426 BYTES IN      6 RECORDS TRANSFERRED IN    0.0006 SECONDS506887.4 Mbytes/s, TOTAL TIME=    0.0135( 4.4%)
 NSTEP =   984 STEPO   0AA000000
------------------------------------------------
CNT0 not found; string returned was: 'DYNFPOS'
>>> Printing last 8 lines from file: ifs.stat

 15:00:41 0AAA00AAA STEPO      980    46.472   46.472   46.566    766:59    769:22 0.33988105010362E-04       2GB       0MB
 15:01:28 0AAA00AAA STEPO      981    46.935   46.935   47.012    767:46    770:09 0.33988105010362E-04       2GB       0MB
 15:02:14 0AAA00AAA STEPO      982    46.561   46.561   46.653    768:32    770:56 0.33988105010362E-04       2GB       0MB
 15:03:01 0AAA00AAA STEPO      983    47.160   47.160   47.249    769:20    771:43 0.33988105010362E-04       2GB       0MB
 15:03:48 A00000000 STEPO      984    46.495   46.495   46.582    770:06    772:29 0.42580218673223E-04       2GB       0MB
 15:03:48 0AA000000 STEPO      984     0.035    0.035    0.035    770:06    772:29 0.42580218673223E-04       2GB       0MB
 15:03:51 FULLPOS-B DYNFPOS    984     2.541    2.541    2.544    770:09    772:32 0.42580218673223E-04       2GB       0MB
------------------------------------------------
>>> Printing last 11 lines from file: rcf

 CSTEP   = "     960",
 CTIME   = "00100000      ",
 NSTEPLPP        =         201,         202,          49,      228226,      228227, 2*0,         948, 2*0,
 IPRGPNSRES      =           1,
 IPRGPEWRES      =           1,
 IPRTRWRES       =           1,
 IPRTRVRES       =           1,
 GMASS0  =  0.000000000000000E+000,
 GMASSI  =  0.000000000000000E+000
 /
------------------------------------------------
>>> Printing last 8 lines from file: /etc/boinc-client/slots/0/progress_file_12208929.xml

<running_values>
  <last_cpu_time>46217.220000</last_cpu_time>
  <upload_file_number>10</upload_file_number>
  <last_iter>984</last_iter>
  <last_upload>864000</last_upload>
  <model_completed>0</model_completed>
</running_values>
------------------------------------------------
..Failed, model did not complete successfully

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:52:14 1526327 22313863 oifs_43r3_bl_a1ni_2016092300_15_991_12208929_2 864,000 45,118 0.0522


©2024 climateprediction.net