climateprediction.net home page
Task 22446423

Task 22446423

Name oifs_43r3_bl_a3gn_2016092300_20_1018_12292814_0
Workunit 12292814
Created 13 Jun 2024, 2:31:06 UTC
Sent 14 Jun 2024, 8:57:56 UTC
Report deadline 13 Aug 2024, 8:57:56 UTC
Received 15 Jun 2024, 0:01:36 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 1 (0x00000001) Unknown error code
Computer ID 1551386
Run time 4 hours 16 min 28 sec
CPU time 4 hours 12 min 58 sec
Validate state Invalid
Credit 0.00
Device peak FLOPS 1.00 GFLOPS
Application version OpenIFS 43r3 Baroclinic Lifecycle v1.13
x86_64-pc-linux-gnu
Peak working set size 5,552.98 MB
Peak swap size 5,981.07 MB
Peak disk usage 1,285.02 MB
Stderr
<core_client_version>7.24.1</core_client_version>
<![CDATA[
<message>
process exited with code 1 (0x1, -255)</message>
<stderr_txt>
ving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+000360
  22:01:51 STEP  361 H=  90:15 +CPU= 19.319
  22:02:10 STEP  362 H=  90:30 +CPU= 19.245
  22:02:30 STEP  363 H=  90:45 +CPU= 19.430
  22:02:50 STEP  364 H=  91:00 +CPU= 20.065
  22:03:09 STEP  365 H=  91:15 +CPU= 19.086
  22:03:29 STEP  366 H=  91:30 +CPU= 19.262
  22:03:48 STEP  367 H=  91:45 +CPU= 19.435
  22:04:08 STEP  368 H=  92:00 +CPU= 19.975
  22:04:28 STEP  369 H=  92:15 +CPU= 19.924
  22:04:48 STEP  370 H=  92:30 +CPU= 19.760
  22:05:08 STEP  371 H=  92:45 +CPU= 19.392
  22:05:33 STEP  372 H=  93:00 +CPU= 24.664
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+000372
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+000372
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+000372
  22:05:52 STEP  373 H=  93:15 +CPU= 19.477
  22:06:12 STEP  374 H=  93:30 +CPU= 19.549
  22:06:32 STEP  375 H=  93:45 +CPU= 19.679
  22:06:53 STEP  376 H=  94:00 +CPU= 20.353
  22:07:13 STEP  377 H=  94:15 +CPU= 19.845
  22:07:32 STEP  378 H=  94:30 +CPU= 19.311
  22:07:52 STEP  379 H=  94:45 +CPU= 19.601
  22:08:12 STEP  380 H=  95:00 +CPU= 20.318
  22:08:32 STEP  381 H=  95:15 +CPU= 19.979
  22:08:52 STEP  382 H=  95:30 +CPU= 19.788
  22:09:12 STEP  383 H=  95:45 +CPU= 19.955
  22:09:41 STEP  384 H=  96:00 +CPU= 27.770
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+000384
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+000384
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+000384
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292814/ICMGGh7zg+000288
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292814/ICMSHh7zg+000288
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292814/ICMUAh7zg+000288
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292814/ICMGGh7zg+000300
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292814/ICMSHh7zg+000300
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292814/ICMUAh7zg+000300
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292814/ICMGGh7zg+000312
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292814/ICMSHh7zg+000312
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292814/ICMUAh7zg+000312
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292814/ICMGGh7zg+000324
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292814/ICMSHh7zg+000324
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292814/ICMUAh7zg+000324
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292814/ICMGGh7zg+000336
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292814/ICMSHh7zg+000336
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292814/ICMUAh7zg+000336
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292814/ICMGGh7zg+000348
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292814/ICMSHh7zg+000348
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292814/ICMUAh7zg+000348
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292814/ICMGGh7zg+000360
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292814/ICMSHh7zg+000360
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292814/ICMUAh7zg+000360
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292814/ICMGGh7zg+000372
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292814/ICMSHh7zg+000372
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292814/ICMUAh7zg+000372
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_a3gn_2016092300_20_1018_12292814_0_r814913900_3.zip
Uploading the intermediate file: upload_file_3.zip
  22:10:00 STEP  385 H=  96:15 +CPU= 19.380
  22:10:20 STEP  386 H=  96:30 +CPU= 19.474
  22:10:39 STEP  387 H=  96:45 +CPU= 19.357
  22:10:59 STEP  388 H=  97:00 +CPU= 19.675
  22:11:19 STEP  389 H=  97:15 +CPU= 19.314
  22:11:38 STEP  390 H=  97:30 +CPU= 18.971
  22:11:57 STEP  391 H=  97:45 +CPU= 19.127
  22:12:17 STEP  392 H=  98:00 +CPU= 19.942
  22:12:36 STEP  393 H=  98:15 +CPU= 18.942
  22:12:56 STEP  394 H=  98:30 +CPU= 19.373
  22:13:15 STEP  395 H=  98:45 +CPU= 18.798
  22:13:39 STEP  396 H=  99:00 +CPU= 24.219
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+000396
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+000396
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+000396
  22:13:59 STEP  397 H=  99:15 +CPU= 19.221
  22:14:18 STEP  398 H=  99:30 +CPU= 19.290
  22:14:37 STEP  399 H=  99:45 +CPU= 19.222
  22:14:57 STEP  400 H= 100:00 +CPU= 19.624
  22:15:16 STEP  401 H= 100:15 +CPU= 19.041
  22:15:36 STEP  402 H= 100:30 +CPU= 18.896
  22:15:55 STEP  403 H= 100:45 +CPU= 18.941
  22:16:15 STEP  404 H= 101:00 +CPU= 19.691
  22:16:34 STEP  405 H= 101:15 +CPU= 19.554
  22:16:54 STEP  406 H= 101:30 +CPU= 19.486
  22:17:14 STEP  407 H= 101:45 +CPU= 19.514
  22:17:39 STEP  408 H= 102:00 +CPU= 24.796
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+000408
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+000408
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+000408
  22:17:58 STEP  409 H= 102:15 +CPU= 19.535
  22:18:18 STEP  410 H= 102:30 +CPU= 19.207
  22:18:38 STEP  411 H= 102:45 +CPU= 19.604
  22:18:57 STEP  412 H= 103:00 +CPU= 19.759
  22:19:17 STEP  413 H= 103:15 +CPU= 19.653
  22:19:37 STEP  414 H= 103:30 +CPU= 19.684
  22:19:57 STEP  415 H= 103:45 +CPU= 19.807
  22:20:18 STEP  416 H= 104:00 +CPU= 20.263
  22:20:37 STEP  417 H= 104:15 +CPU= 19.703
  22:20:57 STEP  418 H= 104:30 +CPU= 19.202
  22:21:16 STEP  419 H= 104:45 +CPU= 19.453
  22:21:41 STEP  420 H= 105:00 +CPU= 23.973
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+000420
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+000420
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+000420
  22:22:00 STEP  421 H= 105:15 +CPU= 18.921
  22:22:20 STEP  422 H= 105:30 +CPU= 19.830
  22:22:39 STEP  423 H= 105:45 +CPU= 19.265
  22:22:59 STEP  424 H= 106:00 +CPU= 20.148
  22:23:19 STEP  425 H= 106:15 +CPU= 19.701
  22:23:39 STEP  426 H= 106:30 +CPU= 19.539
  22:23:59 STEP  427 H= 106:45 +CPU= 19.404
  22:24:19 STEP  428 H= 107:00 +CPU= 20.305
  22:24:39 STEP  429 H= 107:15 +CPU= 19.301
  22:24:58 STEP  430 H= 107:30 +CPU= 18.987
  22:25:18 STEP  431 H= 107:45 +CPU= 19.500
  22:25:45 STEP  432 H= 108:00 +CPU= 27.294
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+000432
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+000432
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+000432
  22:26:05 STEP  433 H= 108:15 +CPU= 19.354
  22:26:24 STEP  434 H= 108:30 +CPU= 19.178
  22:26:44 STEP  435 H= 108:45 +CPU= 20.070
  22:27:05 STEP  436 H= 109:00 +CPU= 20.299
  22:27:24 STEP  437 H= 109:15 +CPU= 19.290
  22:27:44 STEP  438 H= 109:30 +CPU= 19.603
  22:28:04 STEP  439 H= 109:45 +CPU= 19.688
  22:28:24 STEP  440 H= 110:00 +CPU= 20.359
  22:28:44 STEP  441 H= 110:15 +CPU= 19.778
  22:29:04 STEP  442 H= 110:30 +CPU= 19.702
  22:29:24 STEP  443 H= 110:45 +CPU= 19.886
  22:29:49 STEP  444 H= 111:00 +CPU= 25.002
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+000444
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+000444
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+000444
  22:30:09 STEP  445 H= 111:15 +CPU= 19.268
  22:30:29 STEP  446 H= 111:30 +CPU= 20.293
  22:30:49 STEP  447 H= 111:45 +CPU= 19.540
  22:31:09 STEP  448 H= 112:00 +CPU= 20.040
  22:31:29 STEP  449 H= 112:15 +CPU= 19.526
  22:31:49 STEP  450 H= 112:30 +CPU= 19.846
  22:32:08 STEP  451 H= 112:45 +CPU= 19.299
  22:32:29 STEP  452 H= 113:00 +CPU= 20.109
  22:32:48 STEP  453 H= 113:15 +CPU= 18.889
  22:33:07 STEP  454 H= 113:30 +CPU= 19.257
  22:33:26 STEP  455 H= 113:45 +CPU= 18.997
  22:33:51 STEP  456 H= 114:00 +CPU= 24.557
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+000456
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+000456
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+000456
  22:34:10 STEP  457 H= 114:15 +CPU= 19.198
  22:34:30 STEP  458 H= 114:30 +CPU= 19.520
  22:34:50 STEP  459 H= 114:45 +CPU= 19.296
  22:35:10 STEP  460 H= 115:00 +CPU= 19.794
  22:35:29 STEP  461 H= 115:15 +CPU= 19.362
  22:35:49 STEP  462 H= 115:30 +CPU= 19.537
  22:36:08 STEP  463 H= 115:45 +CPU= 19.311
  22:36:28 STEP  464 H= 116:00 +CPU= 19.393
  22:36:47 STEP  465 H= 116:15 +CPU= 18.970
  22:37:07 STEP  466 H= 116:30 +CPU= 19.365
  22:37:26 STEP  467 H= 116:45 +CPU= 18.989
  22:37:50 STEP  468 H= 117:00 +CPU= 23.841
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+000468
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+000468
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+000468
  22:38:09 STEP  469 H= 117:15 +CPU= 19.253
  22:38:29 STEP  470 H= 117:30 +CPU= 19.452
  22:38:48 STEP  471 H= 117:45 +CPU= 19.503
  22:39:09 STEP  472 H= 118:00 +CPU= 19.997
  22:39:28 STEP  473 H= 118:15 +CPU= 19.303
  22:39:48 STEP  474 H= 118:30 +CPU= 19.572
  22:40:07 STEP  475 H= 118:45 +CPU= 19.472
  22:40:27 STEP  476 H= 119:00 +CPU= 19.635
  22:40:46 STEP  477 H= 119:15 +CPU= 18.971
  22:41:06 STEP  478 H= 119:30 +CPU= 19.392
  22:41:26 STEP  479 H= 119:45 +CPU= 19.782
  22:41:53 STEP  480 H= 120:00 +CPU= 26.944
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+000480
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+000480
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+000480
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292814/ICMGGh7zg+000384
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292814/ICMSHh7zg+000384
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292814/ICMUAh7zg+000384
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292814/ICMGGh7zg+000396
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292814/ICMSHh7zg+000396
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292814/ICMUAh7zg+000396
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292814/ICMGGh7zg+000408
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292814/ICMSHh7zg+000408
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292814/ICMUAh7zg+000408
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292814/ICMGGh7zg+000420
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292814/ICMSHh7zg+000420
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292814/ICMUAh7zg+000420
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292814/ICMGGh7zg+000432
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292814/ICMSHh7zg+000432
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292814/ICMUAh7zg+000432
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292814/ICMGGh7zg+000444
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292814/ICMSHh7zg+000444
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292814/ICMUAh7zg+000444
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292814/ICMGGh7zg+000456
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292814/ICMSHh7zg+000456
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292814/ICMUAh7zg+000456
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292814/ICMGGh7zg+000468
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292814/ICMSHh7zg+000468
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292814/ICMUAh7zg+000468
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_a3gn_2016092300_20_1018_12292814_0_r814913900_4.zip
Uploading the intermediate file: upload_file_4.zip
  22:42:12 STEP  481 H= 120:15 +CPU= 19.154
  22:42:32 STEP  482 H= 120:30 +CPU= 19.672
  22:42:52 STEP  483 H= 120:45 +CPU= 19.344
  22:43:11 STEP  484 H= 121:00 +CPU= 19.508
  22:43:31 STEP  485 H= 121:15 +CPU= 19.126
  22:43:51 STEP  486 H= 121:30 +CPU= 19.720
  22:44:10 STEP  487 H= 121:45 +CPU= 19.470
  22:44:31 STEP  488 H= 122:00 +CPU= 20.244
  22:44:51 STEP  489 H= 122:15 +CPU= 19.586
  22:45:10 STEP  490 H= 122:30 +CPU= 19.514
  22:45:30 STEP  491 H= 122:45 +CPU= 19.154
  22:45:55 STEP  492 H= 123:00 +CPU= 24.737
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+000492
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+000492
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+000492
  22:46:14 STEP  493 H= 123:15 +CPU= 18.952
  22:46:33 STEP  494 H= 123:30 +CPU= 19.595
  22:46:53 STEP  495 H= 123:45 +CPU= 19.250
  22:47:13 STEP  496 H= 124:00 +CPU= 19.735
  22:47:32 STEP  497 H= 124:15 +CPU= 19.242
  22:47:52 STEP  498 H= 124:30 +CPU= 19.558
  22:48:12 STEP  499 H= 124:45 +CPU= 19.422
  22:48:32 STEP  500 H= 125:00 +CPU= 20.167
  22:48:52 STEP  501 H= 125:15 +CPU= 19.694
  22:49:12 STEP  502 H= 125:30 +CPU= 19.853
  22:49:32 STEP  503 H= 125:45 +CPU= 19.679
  22:49:56 STEP  504 H= 126:00 +CPU= 24.687
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+000504
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+000504
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+000504
  22:50:16 STEP  505 H= 126:15 +CPU= 19.515
  22:50:36 STEP  506 H= 126:30 +CPU= 19.566
  22:50:56 STEP  507 H= 126:45 +CPU= 19.570
  22:51:17 STEP  508 H= 127:00 +CPU= 20.755
  22:51:36 STEP  509 H= 127:15 +CPU= 19.697
  22:51:56 STEP  510 H= 127:30 +CPU= 19.663
  22:52:16 STEP  511 H= 127:45 +CPU= 19.858
  22:52:37 STEP  512 H= 128:00 +CPU= 20.777
  22:52:57 STEP  513 H= 128:15 +CPU= 19.714
  22:53:17 STEP  514 H= 128:30 +CPU= 20.142
  22:53:37 STEP  515 H= 128:45 +CPU= 19.782
  22:54:02 STEP  516 H= 129:00 +CPU= 24.271
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+000516
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+000516
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+000516
  22:54:21 STEP  517 H= 129:15 +CPU= 19.157
  22:54:41 STEP  518 H= 129:30 +CPU= 19.733
  22:55:01 STEP  519 H= 129:45 +CPU= 19.468
  22:55:21 STEP  520 H= 130:00 +CPU= 20.213
  22:55:40 STEP  521 H= 130:15 +CPU= 19.108
  22:56:00 STEP  522 H= 130:30 +CPU= 19.395
  22:56:19 STEP  523 H= 130:45 +CPU= 19.260
  22:56:40 STEP  524 H= 131:00 +CPU= 19.914
  22:56:59 STEP  525 H= 131:15 +CPU= 19.257
  22:57:18 STEP  526 H= 131:30 +CPU= 19.126
  22:57:38 STEP  527 H= 131:45 +CPU= 19.407
  22:58:05 STEP  528 H= 132:00 +CPU= 26.565
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+000528
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+000528
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+000528
  22:58:24 STEP  529 H= 132:15 +CPU= 19.561
  22:58:43 STEP  530 H= 132:30 +CPU= 18.912
  22:59:03 STEP  531 H= 132:45 +CPU= 19.425
  22:59:23 STEP  532 H= 133:00 +CPU= 20.099
  22:59:43 STEP  533 H= 133:15 +CPU= 19.279
  23:00:02 STEP  534 H= 133:30 +CPU= 19.483
  23:00:22 STEP  535 H= 133:45 +CPU= 19.700
  23:00:43 STEP  536 H= 134:00 +CPU= 20.106
  23:01:02 STEP  537 H= 134:15 +CPU= 19.454
  23:01:22 STEP  538 H= 134:30 +CPU= 19.352
  23:01:41 STEP  539 H= 134:45 +CPU= 19.299
  23:02:06 STEP  540 H= 135:00 +CPU= 24.581
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+000540
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+000540
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+000540
  23:02:26 STEP  541 H= 135:15 +CPU= 20.192
  23:02:46 STEP  542 H= 135:30 +CPU= 19.470
  23:03:06 STEP  543 H= 135:45 +CPU= 19.579
  23:03:26 STEP  544 H= 136:00 +CPU= 20.355
  23:03:46 STEP  545 H= 136:15 +CPU= 19.293
  23:04:06 STEP  546 H= 136:30 +CPU= 19.637
  23:04:24 STEP  547 H= 136:45 +CPU= 18.670
  23:04:44 STEP  548 H= 137:00 +CPU= 19.084
  23:05:02 STEP  549 H= 137:15 +CPU= 18.483
  23:05:21 STEP  550 H= 137:30 +CPU= 19.018
  23:05:41 STEP  551 H= 137:45 +CPU= 18.940
  23:06:04 STEP  552 H= 138:00 +CPU= 23.579
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+000552
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+000552
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+000552
  23:06:23 STEP  553 H= 138:15 +CPU= 18.737
  23:06:42 STEP  554 H= 138:30 +CPU= 18.575
  23:07:01 STEP  555 H= 138:45 +CPU= 18.538
  23:07:20 STEP  556 H= 139:00 +CPU= 19.419
  23:07:39 STEP  557 H= 139:15 +CPU= 18.333
  23:07:57 STEP  558 H= 139:30 +CPU= 18.200
  23:08:16 STEP  559 H= 139:45 +CPU= 18.373
  23:08:34 STEP  560 H= 140:00 +CPU= 18.634
  23:08:53 STEP  561 H= 140:15 +CPU= 18.467
  23:09:12 STEP  562 H= 140:30 +CPU= 19.138
  23:09:31 STEP  563 H= 140:45 +CPU= 18.457
  23:09:55 STEP  564 H= 141:00 +CPU= 23.363
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+000564
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+000564
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+000564
  23:10:14 STEP  565 H= 141:15 +CPU= 18.920
  23:10:33 STEP  566 H= 141:30 +CPU= 19.015
  23:10:51 STEP  567 H= 141:45 +CPU= 18.560
  23:11:11 STEP  568 H= 142:00 +CPU= 19.197
  23:11:30 STEP  569 H= 142:15 +CPU= 18.679
  23:11:49 STEP  570 H= 142:30 +CPU= 19.254
  23:12:08 STEP  571 H= 142:45 +CPU= 19.015
  23:12:28 STEP  572 H= 143:00 +CPU= 19.634
  23:12:47 STEP  573 H= 143:15 +CPU= 18.676
  23:13:06 STEP  574 H= 143:30 +CPU= 19.426
  23:13:25 STEP  575 H= 143:45 +CPU= 18.890
  23:13:52 STEP  576 H= 144:00 +CPU= 26.039
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+000576
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+000576
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+000576
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292814/ICMGGh7zg+000480
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292814/ICMSHh7zg+000480
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292814/ICMUAh7zg+000480
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292814/ICMGGh7zg+000492
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292814/ICMSHh7zg+000492
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292814/ICMUAh7zg+000492
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292814/ICMGGh7zg+000504
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292814/ICMSHh7zg+000504
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292814/ICMUAh7zg+000504
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292814/ICMGGh7zg+000516
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292814/ICMSHh7zg+000516
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292814/ICMUAh7zg+000516
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292814/ICMGGh7zg+000528
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292814/ICMSHh7zg+000528
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292814/ICMUAh7zg+000528
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292814/ICMGGh7zg+000540
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292814/ICMSHh7zg+000540
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292814/ICMUAh7zg+000540
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292814/ICMGGh7zg+000552
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292814/ICMSHh7zg+000552
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292814/ICMUAh7zg+000552
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292814/ICMGGh7zg+000564
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292814/ICMSHh7zg+000564
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292814/ICMUAh7zg+000564
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_a3gn_2016092300_20_1018_12292814_0_r814913900_5.zip
Uploading the intermediate file: upload_file_5.zip
  23:14:11 STEP  577 H= 144:15 +CPU= 18.746
  23:14:30 STEP  578 H= 144:30 +CPU= 18.863
  23:14:48 STEP  579 H= 144:45 +CPU= 18.396
  23:15:08 STEP  580 H= 145:00 +CPU= 19.249
  23:15:26 STEP  581 H= 145:15 +CPU= 18.526
  23:15:45 STEP  582 H= 145:30 +CPU= 18.408
  23:16:04 STEP  583 H= 145:45 +CPU= 18.920
  23:16:23 STEP  584 H= 146:00 +CPU= 18.957
  23:16:41 STEP  585 H= 146:15 +CPU= 18.211
  23:17:00 STEP  586 H= 146:30 +CPU= 18.614
  23:17:19 STEP  587 H= 146:45 +CPU= 18.780
  23:17:43 STEP  588 H= 147:00 +CPU= 23.559
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+000588
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+000588
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+000588
  23:18:01 STEP  589 H= 147:15 +CPU= 18.313
  23:18:20 STEP  590 H= 147:30 +CPU= 18.351
  23:18:39 STEP  591 H= 147:45 +CPU= 18.921
  23:18:58 STEP  592 H= 148:00 +CPU= 19.202
  23:19:18 STEP  593 H= 148:15 +CPU= 19.079
  23:19:36 STEP  594 H= 148:30 +CPU= 18.635
  23:19:55 STEP  595 H= 148:45 +CPU= 18.539
  23:20:14 STEP  596 H= 149:00 +CPU= 19.169
  23:20:33 STEP  597 H= 149:15 +CPU= 18.596
  23:20:52 STEP  598 H= 149:30 +CPU= 18.822
  23:21:11 STEP  599 H= 149:45 +CPU= 18.994
  23:21:35 STEP  600 H= 150:00 +CPU= 23.556
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+000600
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+000600
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+000600
  23:21:54 STEP  601 H= 150:15 +CPU= 18.483
  23:22:13 STEP  602 H= 150:30 +CPU= 19.082
  23:22:32 STEP  603 H= 150:45 +CPU= 18.405
  23:22:51 STEP  604 H= 151:00 +CPU= 19.187
  23:23:10 STEP  605 H= 151:15 +CPU= 18.604
  23:23:28 STEP  606 H= 151:30 +CPU= 18.544
  23:23:47 STEP  607 H= 151:45 +CPU= 18.374
  23:24:06 STEP  608 H= 152:00 +CPU= 19.261
  23:24:25 STEP  609 H= 152:15 +CPU= 18.896
  23:24:44 STEP  610 H= 152:30 +CPU= 18.866
  23:25:03 STEP  611 H= 152:45 +CPU= 18.456
  23:25:27 STEP  612 H= 153:00 +CPU= 23.600
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+000612
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+000612
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+000612
  23:25:46 STEP  613 H= 153:15 +CPU= 18.523
  23:26:04 STEP  614 H= 153:30 +CPU= 18.518
  23:26:23 STEP  615 H= 153:45 +CPU= 18.574
  23:26:43 STEP  616 H= 154:00 +CPU= 19.542
  23:27:01 STEP  617 H= 154:15 +CPU= 18.504
  23:27:20 STEP  618 H= 154:30 +CPU= 18.802
  23:27:39 STEP  619 H= 154:45 +CPU= 18.333
  23:27:59 STEP  620 H= 155:00 +CPU= 19.579
  23:28:17 STEP  621 H= 155:15 +CPU= 18.528
  23:28:36 STEP  622 H= 155:30 +CPU= 18.509
  23:28:54 STEP  623 H= 155:45 +CPU= 18.223
  23:29:20 STEP  624 H= 156:00 +CPU= 25.897
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+000624
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+000624
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+000624
  23:29:39 STEP  625 H= 156:15 +CPU= 18.831
  23:29:58 STEP  626 H= 156:30 +CPU= 18.511
  23:30:17 STEP  627 H= 156:45 +CPU= 18.469
  23:30:37 STEP  628 H= 157:00 +CPU= 19.967
  23:30:56 STEP  629 H= 157:15 +CPU= 18.733
  23:31:15 STEP  630 H= 157:30 +CPU= 18.757
  23:31:33 STEP  631 H= 157:45 +CPU= 18.559
  23:31:53 STEP  632 H= 158:00 +CPU= 19.679
  23:32:13 STEP  633 H= 158:15 +CPU= 19.227
  23:32:31 STEP  634 H= 158:30 +CPU= 18.725
  23:32:50 STEP  635 H= 158:45 +CPU= 18.730
  23:33:14 STEP  636 H= 159:00 +CPU= 23.691
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+000636
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+000636
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+000636
  23:33:33 STEP  637 H= 159:15 +CPU= 18.986
  23:33:53 STEP  638 H= 159:30 +CPU= 19.195
  23:34:12 STEP  639 H= 159:45 +CPU= 18.739
  23:34:31 STEP  640 H= 160:00 +CPU= 19.667
  23:34:50 STEP  641 H= 160:15 +CPU= 18.796
  23:35:09 STEP  642 H= 160:30 +CPU= 18.898
  23:35:28 STEP  643 H= 160:45 +CPU= 18.590
  23:35:48 STEP  644 H= 161:00 +CPU= 19.495
  23:36:07 STEP  645 H= 161:15 +CPU= 18.948
  23:36:26 STEP  646 H= 161:30 +CPU= 18.725
  23:36:44 STEP  647 H= 161:45 +CPU= 18.224
  23:37:08 STEP  648 H= 162:00 +CPU= 23.365
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+000648
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+000648
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+000648
  23:37:27 STEP  649 H= 162:15 +CPU= 19.031
  23:37:46 STEP  650 H= 162:30 +CPU= 18.669
  23:38:05 STEP  651 H= 162:45 +CPU= 19.098
  23:38:25 STEP  652 H= 163:00 +CPU= 19.265
  23:38:43 STEP  653 H= 163:15 +CPU= 18.496
  23:39:02 STEP  654 H= 163:30 +CPU= 18.552
  23:39:21 STEP  655 H= 163:45 +CPU= 18.430
  23:39:40 STEP  656 H= 164:00 +CPU= 19.272
  23:39:59 STEP  657 H= 164:15 +CPU= 18.755
  23:40:18 STEP  658 H= 164:30 +CPU= 18.610
  23:40:36 STEP  659 H= 164:45 +CPU= 18.584
  23:41:00 STEP  660 H= 165:00 +CPU= 23.487
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+000660
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+000660
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+000660
  23:41:19 STEP  661 H= 165:15 +CPU= 18.962
  23:41:38 STEP  662 H= 165:30 +CPU= 19.008
  23:41:57 STEP  663 H= 165:45 +CPU= 18.744
  23:42:17 STEP  664 H= 166:00 +CPU= 19.642
  23:42:36 STEP  665 H= 166:15 +CPU= 18.583
  23:42:54 STEP  666 H= 166:30 +CPU= 18.415
  23:43:13 STEP  667 H= 166:45 +CPU= 18.861
  23:43:33 STEP  668 H= 167:00 +CPU= 19.174
  23:43:52 STEP  669 H= 167:15 +CPU= 18.702
  23:44:11 STEP  670 H= 167:30 +CPU= 19.085
  23:44:30 STEP  671 H= 167:45 +CPU= 19.038
  23:44:56 STEP  672 H= 168:00 +CPU= 25.824
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+000672
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+000672
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+000672
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292814/ICMGGh7zg+000576
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292814/ICMSHh7zg+000576
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292814/ICMUAh7zg+000576
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292814/ICMGGh7zg+000588
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292814/ICMSHh7zg+000588
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292814/ICMUAh7zg+000588
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292814/ICMGGh7zg+000600
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292814/ICMSHh7zg+000600
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292814/ICMUAh7zg+000600
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292814/ICMGGh7zg+000612
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292814/ICMSHh7zg+000612
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292814/ICMUAh7zg+000612
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292814/ICMGGh7zg+000624
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292814/ICMSHh7zg+000624
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292814/ICMUAh7zg+000624
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292814/ICMGGh7zg+000636
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292814/ICMSHh7zg+000636
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292814/ICMUAh7zg+000636
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292814/ICMGGh7zg+000648
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292814/ICMSHh7zg+000648
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292814/ICMUAh7zg+000648
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292814/ICMGGh7zg+000660
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292814/ICMSHh7zg+000660
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292814/ICMUAh7zg+000660
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_a3gn_2016092300_20_1018_12292814_0_r814913900_6.zip
Uploading the intermediate file: upload_file_6.zip
  23:45:15 STEP  673 H= 168:15 +CPU= 18.424
  23:45:33 STEP  674 H= 168:30 +CPU= 18.308
  23:45:52 STEP  675 H= 168:45 +CPU= 18.531
  23:46:11 STEP  676 H= 169:00 +CPU= 18.679
  23:46:29 STEP  677 H= 169:15 +CPU= 18.428
  23:46:48 STEP  678 H= 169:30 +CPU= 18.948
  23:47:07 STEP  679 H= 169:45 +CPU= 18.900
  23:47:27 STEP  680 H= 170:00 +CPU= 19.377
  23:47:46 STEP  681 H= 170:15 +CPU= 18.535
  23:48:05 STEP  682 H= 170:30 +CPU= 19.023
  23:48:24 STEP  683 H= 170:45 +CPU= 18.549
  23:48:47 STEP  684 H= 171:00 +CPU= 23.317
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+000684
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+000684
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+000684
  23:49:06 STEP  685 H= 171:15 +CPU= 18.559
  23:49:25 STEP  686 H= 171:30 +CPU= 18.670
  23:49:43 STEP  687 H= 171:45 +CPU= 18.500
  23:50:03 STEP  688 H= 172:00 +CPU= 19.111
  23:50:21 STEP  689 H= 172:15 +CPU= 18.693
  23:50:40 STEP  690 H= 172:30 +CPU= 18.436
  23:50:59 STEP  691 H= 172:45 +CPU= 18.788
  23:51:19 STEP  692 H= 173:00 +CPU= 19.468
  23:51:38 STEP  693 H= 173:15 +CPU= 18.913
  23:51:57 STEP  694 H= 173:30 +CPU= 19.227
  23:52:16 STEP  695 H= 173:45 +CPU= 18.795
  23:52:40 STEP  696 H= 174:00 +CPU= 23.539
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+000696
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+000696
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+000696
  23:52:59 STEP  697 H= 174:15 +CPU= 18.785
  23:53:18 STEP  698 H= 174:30 +CPU= 19.213
  23:53:37 STEP  699 H= 174:45 +CPU= 18.939
  23:53:57 STEP  700 H= 175:00 +CPU= 19.898
  23:54:16 STEP  701 H= 175:15 +CPU= 19.223
  23:54:36 STEP  702 H= 175:30 +CPU= 19.247
  23:54:55 STEP  703 H= 175:45 +CPU= 19.212
  23:55:15 STEP  704 H= 176:00 +CPU= 19.529
  23:55:33 STEP  705 H= 176:15 +CPU= 18.400
  23:55:52 STEP  706 H= 176:30 +CPU= 18.751
  23:56:11 STEP  707 H= 176:45 +CPU= 18.497
  23:56:35 STEP  708 H= 177:00 +CPU= 23.604
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+000708
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+000708
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+000708
  23:56:54 STEP  709 H= 177:15 +CPU= 18.635
  23:57:12 STEP  710 H= 177:30 +CPU= 17.904
  23:57:31 STEP  711 H= 177:45 +CPU= 18.693
  23:57:51 STEP  712 H= 178:00 +CPU= 19.866
  23:58:10 STEP  713 H= 178:15 +CPU= 18.965
  23:58:29 STEP  714 H= 178:30 +CPU= 18.921
  23:58:47 STEP  715 H= 178:45 +CPU= 18.343
  23:59:06 STEP  716 H= 179:00 +CPU= 18.721
  23:59:25 STEP  717 H= 179:15 +CPU= 18.610
  23:59:43 STEP  718 H= 179:30 +CPU= 18.329
  00:00:02 STEP  719 H= 179:45 +CPU= 18.790
  00:00:29 STEP  720 H= 180:00 +CPU= 26.349
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+000720
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+000720
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+000720
  00:00:48 STEP  721 H= 180:15 +CPU= 18.655
  00:01:06 STEP  722 H= 180:30 +CPU= 18.304
  00:01:25 STEP  723 H= 180:45 +CPU= 18.296
  00:01:44 STEP  724 H= 181:00 +CPU= 19.244
  00:02:03 STEP  725 H= 181:15 +CPU= 18.600
  00:02:22 STEP  726 H= 181:30 +CPU= 18.670
  00:02:41 STEP  727 H= 181:45 +CPU= 19.123
  00:03:00 STEP  728 H= 182:00 +CPU= 19.307
  00:03:19 STEP  729 H= 182:15 +CPU= 18.848
  00:03:38 STEP  730 H= 182:30 +CPU= 18.585
  00:03:56 STEP  731 H= 182:45 +CPU= 18.139
  00:04:20 STEP  732 H= 183:00 +CPU= 23.748
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+000732
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+000732
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+000732
  00:04:39 STEP  733 H= 183:15 +CPU= 18.379
  00:04:58 STEP  734 H= 183:30 +CPU= 18.442
  00:05:17 STEP  735 H= 183:45 +CPU= 18.829
  00:05:36 STEP  736 H= 184:00 +CPU= 19.317
  00:05:55 STEP  737 H= 184:15 +CPU= 18.614
  00:06:14 STEP  738 H= 184:30 +CPU= 19.295
  00:06:33 STEP  739 H= 184:45 +CPU= 18.681
  00:06:53 STEP  740 H= 185:00 +CPU= 19.332
  00:07:12 STEP  741 H= 185:15 +CPU= 19.007
  00:07:31 STEP  742 H= 185:30 +CPU= 18.628
  00:07:50 STEP  743 H= 185:45 +CPU= 18.667
  00:08:14 STEP  744 H= 186:00 +CPU= 24.034
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+000744
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+000744
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+000744
  00:08:32 STEP  745 H= 186:15 +CPU= 18.497
  00:08:51 STEP  746 H= 186:30 +CPU= 18.587
  00:09:10 STEP  747 H= 186:45 +CPU= 18.404
  00:09:28 STEP  748 H= 187:00 +CPU= 18.625
  00:09:47 STEP  749 H= 187:15 +CPU= 18.390
  00:10:05 STEP  750 H= 187:30 +CPU= 18.157
  00:10:24 STEP  751 H= 187:45 +CPU= 18.102
  00:10:43 STEP  752 H= 188:00 +CPU= 18.851
  00:11:01 STEP  753 H= 188:15 +CPU= 18.707
  00:11:21 STEP  754 H= 188:30 +CPU= 19.077
  00:11:39 STEP  755 H= 188:45 +CPU= 18.541
  00:12:03 STEP  756 H= 189:00 +CPU= 23.460
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+000756
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+000756
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+000756
  00:12:22 STEP  757 H= 189:15 +CPU= 18.443
  00:12:41 STEP  758 H= 189:30 +CPU= 18.989
  00:12:59 STEP  759 H= 189:45 +CPU= 18.671
  00:13:19 STEP  760 H= 190:00 +CPU= 19.630
  00:13:38 STEP  761 H= 190:15 +CPU= 18.667
  00:13:57 STEP  762 H= 190:30 +CPU= 18.998
  00:14:16 STEP  763 H= 190:45 +CPU= 18.708
  00:14:36 STEP  764 H= 191:00 +CPU= 19.652
  00:14:55 STEP  765 H= 191:15 +CPU= 18.654
  00:15:13 STEP  766 H= 191:30 +CPU= 18.342
 SMILAG TRAJECTORY OUT OF ATM     44 TIMES.
  POINT=           24  MAX ETADOT VERTICAL VEL.=   0.152527618177218     
  LON  =    4.99999999999999       degrees
  LAT  =    39.8128523737712       degrees
  POINT=           29  MAX ETADOT VERTICAL VEL.=   0.272945424723024     
  LON  =    11.2500000000000       degrees
  LAT  =    39.8128523737712       degrees
  POINT=           31  MAX ETADOT VERTICAL VEL.=   4.319891288754273E-002
  LON  =    13.7500000000000       degrees
  LAT  =    39.8128523737712       degrees
 SMILAG TRAJECTORY UNDERGROUND     24 TIMES.
  POINT=           20  MAX ETADOT VERTICAL VEL.=   8.879532209161721E-010
  LON  =   0.000000000000000E+000  degrees
  LAT  =    39.8128523737712       degrees
  POINT=           24  MAX ETADOT VERTICAL VEL.=   0.152527618177218     
  LON  =    4.99999999999999       degrees
  LAT  =    39.8128523737712       degrees
  POINT=           25  MAX ETADOT VERTICAL VEL.=   1.573734696496053E-009
  LON  =    6.24999999999998       degrees
  LAT  =    39.8128523737712       degrees
  POINT=           26  MAX ETADOT VERTICAL VEL.=   1.599077942792305E-009
  LON  =    7.50000000000001       degrees
  LAT  =    39.8128523737712       degrees
  POINT=           27  MAX ETADOT VERTICAL VEL.=   1.584410746741475E-008
  LON  =    8.75000000000002       degrees
  LAT  =    39.8128523737712       degrees
  POINT=           30  MAX ETADOT VERTICAL VEL.=   1.376202719146914E-008
  LON  =    12.5000000000000       degrees
  LAT  =    39.8128523737712       degrees
 SMILAG TRAJECTORY OUT OF ATM     31 TIMES.
  POINT=           10  MAX ETADOT VERTICAL VEL.=    1.96879474236771     
  LON  =    27.5000000000000       degrees
  LAT  =    39.8128523737712       degrees
  POINT=           16  MAX ETADOT VERTICAL VEL.=    2269.88539374159     
  LON  =    35.0000000000000       degrees
  LAT  =    39.8128523737712       degrees
  POINT=           17  MAX ETADOT VERTICAL VEL.=    476.580792026396     
  LON  =    36.2500000000000       degrees
  LAT  =    39.8128523737712       degrees
  POINT=           19  MAX ETADOT VERTICAL VEL.=    112.286984914828     
  LON  =    38.7500000000000       degrees
  LAT  =    39.8128523737712       degrees
  POINT=           20  MAX ETADOT VERTICAL VEL.=    132.621411244105     
  LON  =    40.0000000000000       degrees
  LAT  =    39.8128523737712       degrees
  POINT=           22  MAX ETADOT VERTICAL VEL.=    57894.6681703503     
  LON  =    42.5000000000000       degrees
  LAT  =    39.8128523737712       degrees
  POINT=           23  MAX ETADOT VERTICAL VEL.=    1019.40422877198     
  LON  =    43.7500000000000       degrees
  LAT  =    39.8128523737712       degrees
 SMILAG TRAJECTORY UNDERGROUND    238 TIMES.
  POINT=            4  MAX ETADOT VERTICAL VEL.=   5.418027415959968E-010
  LON  =    20.0000000000000       degrees
  LAT  =    39.8128523737712       degrees
  POINT=            5  MAX ETADOT VERTICAL VEL.=   6.575742074700871E-009
  LON  =    21.2500000000000       degrees
  LAT  =    39.8128523737712       degrees
  POINT=            6  MAX ETADOT VERTICAL VEL.=   8.272219985096741E-010
  LON  =    22.5000000000000       degrees
  LAT  =    39.8128523737712       degrees
  POINT=           10  MAX ETADOT VERTICAL VEL.=    1.96879474236771     
  LON  =    27.5000000000000       degrees
  LAT  =    39.8128523737712       degrees
  POINT=           16  MAX ETADOT VERTICAL VEL.=    2269.88539374159     
  LON  =    35.0000000000000       degrees
  LAT  =    39.8128523737712       degrees
  POINT=           17  MAX ETADOT VERTICAL VEL.=    476.580792026396     
  LON  =    36.2500000000000       degrees
  LAT  =    39.8128523737712       degrees
  POINT=           18  MAX ETADOT VERTICAL VEL.=   3.951492250957827E-008
  LON  =    37.5000000000000       degrees
  LAT  =    39.8128523737712       degrees
  POINT=           19  MAX ETADOT VERTICAL VEL.=    112.286984914828     
  LON  =    38.7500000000000       degrees
  LAT  =    39.8128523737712       degrees
  POINT=           20  MAX ETADOT VERTICAL VEL.=    132.621411244105     
  LON  =    40.0000000000000       degrees
  LAT  =    39.8128523737712       degrees
  POINT=           21  MAX ETADOT VERTICAL VEL.=   1.858216027688666E-009
  LON  =    41.2500000000000       degrees
  LAT  =    39.8128523737712       degrees
  POINT=           22  MAX ETADOT VERTICAL VEL.=    57894.6681703503     
  LON  =    42.5000000000000       degrees
  LAT  =    39.8128523737712       degrees
  POINT=           23  MAX ETADOT VERTICAL VEL.=    1019.40422877198     
  LON  =    43.7500000000000       degrees
  LAT  =    39.8128523737712       degrees
  POINT=           24  MAX ETADOT VERTICAL VEL.=   1.394694989561040E-008
  LON  =    45.0000000000000       degrees
  LAT  =    39.8128523737712       degrees
  POINT=           29  MAX ETADOT VERTICAL VEL.=   8.072014815181166E-008
  LON  =    51.2500000000000       degrees
  LAT  =    39.8128523737712       degrees
  POINT=           30  MAX ETADOT VERTICAL VEL.=   1.032770895459295E-007
  LON  =    52.5000000000000       degrees
  LAT  =    39.8128523737712       degrees
  POINT=           31  MAX ETADOT VERTICAL VEL.=   7.357727083980280E-008
  LON  =    53.7500000000000       degrees
  LAT  =    39.8128523737712       degrees
  POINT=           32  MAX ETADOT VERTICAL VEL.=   6.598298109469448E-008
  LON  =    55.0000000000000       degrees
  LAT  =    39.8128523737712       degrees
[EC_DRHOOK:connie:1:1:1804856:1804856] [20240615:001518:1718406918:15312.736] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1538] Received signal#8 (SIGFPE) :: 5752MB (heap), 5558MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:connie:1:1:1804856:1804856] [20240615:001518:1718406918:15312.736] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:connie:1:1:1804856:1804856] [20240615:001518:1718406918:15312.736] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x820890 (old at (nil))
[EC_DRHOOK:connie:1:1:1804856:1804856] [20240615:001518:1718406918:15312.736] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1617] Signal#8 was caused by floating-point invalid operation [memaddr=0xd854f2], nsigs = 1
[EC_DRHOOK:connie:1:1:1804856:1804856] [20240615:001518:1718406918:15312.736] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#8, nsigs = 1
[EC_DRHOOK:connie:1:1:1804856:1804856] [20240615:001518:1718406918:15312.736] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3843] 5752 MB (maxheap), 5558 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:connie:1:1:1804856:1804856] [20240615:001518:1718406918:15312.736] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:connie:1:1:1804856:1804856] [20240615:001518:1718406918:15312.736] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:connie:1:1:1804856:1804856] [20240615:001518:1718406918:15312.736] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:connie:1:1:1804856:1804856] [20240615:001518:1718406918:15312.736] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:connie:1:1:1804856:1804856] [20240615:001518:1718406918:15312.736] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:connie:1:1:1804856:1804856] [20240615:001518:1718406918:15312.736] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :       CNT4 
[EC_DRHOOK:connie:1:1:1804856:1804856] [20240615:001518:1718406918:15312.736] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :        STEPO 
[EC_DRHOOK:connie:1:1:1804856:1804856] [20240615:001518:1718406918:15312.736] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :         SCAN2M 
[EC_DRHOOK:connie:1:1:1804856:1804856] [20240615:001518:1718406918:15312.736] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :          GP_MODEL_HEAP 
[EC_DRHOOK:connie:1:1:1804856:1804856] [20240615:001518:1718406918:15312.736] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :           GP_MODEL 
[EC_DRHOOK:connie:1:1:1804856:1804856] [20240615:001518:1718406918:15312.736] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :            CALL_SL_HEAP 
[EC_DRHOOK:connie:1:1:1804856:1804856] [20240615:001518:1718406918:15312.736] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :             CALL_SL 
[EC_DRHOOK:connie:1:1:1804856:1804856] [20240615:001518:1718406918:15312.736] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :              >OMP-CALL_SL     1        (1004) 
[EC_DRHOOK:connie:1:1:1804856:1804856] [20240615:001518:1718406918:15312.736] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :               LAPINEA 
[EC_DRHOOK:connie:1:1:1804856:1804856] [20240615:001518:1718406918:15312.736] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :                LARMES 
[EC_DRHOOK:connie:1:1:1804856:1804856] [20240615:001519:1718406919:15313.736] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:240] Backtrace(s) for program '/var/lib/boinc/slots/13/oifs_43r3_model.exe' : sigcontextptr=0x7ffefb77e2d0
[EC_DRHOOK:connie:1:1:1804856:1804856] [20240615:001519:1718406919:15313.736] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:277] Backtrace (size = 21) with addr2line-cmd
[EC_DRHOOK:connie:1:1:1804856:1804856] [20240615:001519:1718406919:15313.736] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:302] /usr/bin/addr2line -fs -e '/var/lib/boinc/slots/13/oifs_43r3_model.exe' 0x83a8f3 0x83aef9 0x1dd5490 0xd854f2 0xd5968d 0xd56b78 0xd54bbc 0xbd4fd8 0x12c5002 0xbacbae 0x688153 0x4cd1dc 0x4b5d85 0x4b4f52 0x49d638 0x406fbd 0x401e7f 0x401e12 0x1dcc34a 0x1dcdbe7 0x401cd5
[EC_DRHOOK:connie:1:1:1804856:1804856] [20240615:001519:1718406919:15313.736] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [0]: [0x83a8f3] : ??() at ??:0
[EC_DRHOOK:connie:1:1:1804856:1804856] [20240615:001519:1718406919:15313.736] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [1]: [0x83aef9] : ??() at ??:0
[EC_DRHOOK:connie:1:1:1804856:1804856] [20240615:001519:1718406919:15313.736] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [2]: [0x1dd5490] : ??() at ??:0
[EC_DRHOOK:connie:1:1:1804856:1804856] [20240615:001519:1718406919:15313.736] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [3]: [0xd854f2] : ??() at ??:0
[EC_DRHOOK:connie:1:1:1804856:1804856] [20240615:001519:1718406919:15313.736] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [4]: [0xd5968d] : ??() at ??:0
[EC_DRHOOK:connie:1:1:1804856:1804856] [20240615:001519:1718406919:15313.736] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [5]: [0xd56b78] : ??() at ??:0
[EC_DRHOOK:connie:1:1:1804856:1804856] [20240615:001519:1718406919:15313.736] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [6]: [0xd54bbc] : ??() at ??:0
[EC_DRHOOK:connie:1:1:1804856:1804856] [20240615:001519:1718406919:15313.736] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [7]: [0xbd4fd8] : ??() at ??:0
[EC_DRHOOK:connie:1:1:1804856:1804856] [20240615:001519:1718406919:15313.736] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [8]: [0x12c5002] : ??() at ??:0
[EC_DRHOOK:connie:1:1:1804856:1804856] [20240615:001519:1718406919:15313.736] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [9]: [0xbacbae] : ??() at ??:0
[EC_DRHOOK:connie:1:1:1804856:1804856] [20240615:001519:1718406919:15313.736] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [10]: [0x688153] : ??() at ??:0
[EC_DRHOOK:connie:1:1:1804856:1804856] [20240615:001519:1718406919:15313.736] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [11]: [0x4cd1dc] : ??() at ??:0
[EC_DRHOOK:connie:1:1:1804856:1804856] [20240615:001519:1718406919:15313.736] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [12]: [0x4b5d85] : ??() at ??:0
[EC_DRHOOK:connie:1:1:1804856:1804856] [20240615:001519:1718406919:15313.736] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [13]: [0x4b4f52] : ??() at ??:0
[EC_DRHOOK:connie:1:1:1804856:1804856] [20240615:001519:1718406919:15313.736] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [14]: [0x49d638] : ??() at ??:0
[EC_DRHOOK:connie:1:1:1804856:1804856] [20240615:001519:1718406919:15313.736] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [15]: [0x406fbd] : ??() at ??:0
[EC_DRHOOK:connie:1:1:1804856:1804856] [20240615:001519:1718406919:15313.736] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [16]: [0x401e7f] : ??() at ??:0
[EC_DRHOOK:connie:1:1:1804856:1804856] [20240615:001519:1718406919:15313.736] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [17]: [0x401e12] : ??() at ??:0
[EC_DRHOOK:connie:1:1:1804856:1804856] [20240615:001519:1718406919:15313.736] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [18]: [0x1dcc34a] : ??() at ??:0
[EC_DRHOOK:connie:1:1:1804856:1804856] [20240615:001519:1718406919:15313.736] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [19]: [0x1dcdbe7] : ??() at ??:0
[EC_DRHOOK:connie:1:1:1804856:1804856] [20240615:001519:1718406919:15313.736] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [20]: [0x401cd5] : ??() at ??:0
[EC_DRHOOK:connie:1:1:1804856:1804856] [20240615:001519:1718406919:15313.736] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:370] End of backtrace(s)
 Process            1  thread            1  calling linux_trbk from intel_trbk()
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:240] Backtrace(s) for program '/var/lib/boinc/slots/13/oifs_43r3_model.exe' : sigcontextptr=0x7ffefb77e1e0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:277] Backtrace (size = 22) with addr2line-cmd
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:302] /usr/bin/addr2line -fs -e '/var/lib/boinc/slots/13/oifs_43r3_model.exe' 0x83a8f3 0x83aef9 0x822540 0x1dd5490 0xd854f2 0xd5968d 0xd56b78 0xd54bbc 0xbd4fd8 0x12c5002 0xbacbae 0x688153 0x4cd1dc 0x4b5d85 0x4b4f52 0x49d638 0x406fbd 0x401e7f 0x401e12 0x1dcc34a 0x1dcdbe7 0x401cd5
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [0]: [0x83a8f3] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [1]: [0x83aef9] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [2]: [0x822540] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [3]: [0x1dd5490] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [4]: [0xd854f2] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [5]: [0xd5968d] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [6]: [0xd56b78] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [7]: [0xd54bbc] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [8]: [0xbd4fd8] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [9]: [0x12c5002] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [10]: [0xbacbae] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [11]: [0x688153] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [12]: [0x4cd1dc] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [13]: [0x4b5d85] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [14]: [0x4b4f52] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [15]: [0x49d638] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [16]: [0x406fbd] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [17]: [0x401e7f] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [18]: [0x401e12] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [19]: [0x1dcc34a] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [20]: [0x1dcdbe7] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [21]: [0x401cd5] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:370] End of backtrace(s)
[EC_DRHOOK:connie:1:1:1804856:1804856] [20240615:001519:1718406919:15313.766] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#8, nsigs = 1
[EC_DRHOOK:connie:1:1:1804856:1804856] [20240615:001519:1718406919:15313.766] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1cf8da0 for signal#8, nsigs = 1
forrtl: error (65): floating invalid
Image              PC                Routine            Line        Source             
oifs_43r3_model.e  0000000001CF935B  Unknown               Unknown  Unknown
oifs_43r3_model.e  00000000008214C1  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DD5490  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000000D854F2  larmes_                   543  larmes.F90
oifs_43r3_model.e  0000000000D5968D  lapinea_                  270  lapinea.F90
oifs_43r3_model.e  0000000000D56B78  call_sl_                  300  call_sl.F90
oifs_43r3_model.e  0000000000D54BBC  call_sl_heap_              94  call_sl_heap.F90
oifs_43r3_model.e  0000000000BD4FD8  gp_model_                 533  gp_model.F90
oifs_43r3_model.e  00000000012C5002  gp_model_heap_             74  gp_model_heap.F90
oifs_43r3_model.e  0000000000BACBAE  scan2m_                   535  scan2m.F90
oifs_43r3_model.e  0000000000688153  stepo_                    327  stepo.F90
oifs_43r3_model.e  00000000004CD1DC  cnt4_                    1133  cnt4.F90
oifs_43r3_model.e  00000000004B5D85  cnt3_                     267  cnt3.F90
oifs_43r3_model.e  00000000004B4F52  cnt2_                      88  cnt2.F90
oifs_43r3_model.e  000000000049D638  cnt1_                      92  cnt1.F90
oifs_43r3_model.e  0000000000406FBD  cnt0_                     146  cnt0.F90
oifs_43r3_model.e  0000000000401E7F  MAIN__                     96  master.F90
oifs_43r3_model.e  0000000000401E12  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCC34A  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCDBE7  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000000401CD5  Unknown               Unknown  Unknown
..The child process has been killed with signal: 6
>>> Printing last 70 lines from file: NODE.001_01

         AVE   0.386224533488793E-07 0.000000000000000E+00 0.311473168614740E-03
 GPNORM CRFL                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.586241023061298E-08 0.000000000000000E+00 0.220952014922058E-03
 GPNORM CSFL                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.901405654642030E-09 0.000000000000000E+00 0.154078142369952E-03
 GPNORM PFRC                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.247219562804760E-02 0.000000000000000E+00 0.100000000000000E+01
 NSTEP =   756 STEPO   A00000000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =   756 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+000756 MODE=w
 IO-STREAM CLOSED - ICMGGh7zg+000756
 MPI-TASK:   1 -    311206420 BYTES IN      6 RECORDS TRANSFERRED IN    0.0004 SECONDS778016.1 Mbytes/s, TOTAL TIME=    0.0083( 4.8%)
 NSTEP =   756 STEPO   0AA000000
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMSHh7zg+000756 MODE=w
 IO-STREAM CLOSED - ./ICMSHh7zg+000756
 MPI-TASK:   1 -    707399250 BYTES IN    202 RECORDS TRANSFERRED IN    0.0112 SECONDS 63160.6 Mbytes/s, TOTAL TIME=    0.2241( 5.0%)
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMUAh7zg+000756 MODE=w
 IO-STREAM CLOSED - ./ICMUAh7zg+000756
 MPI-TASK:   1 -   4979302720 BYTES IN    125 RECORDS TRANSFERRED IN    0.0055 SECONDS905327.8 Mbytes/s, TOTAL TIME=    0.1356( 4.1%)
 STEPO_FPOS WAS : B00000F00E
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMUAh7zg+000756 MODE=a
 IO-STREAM CLOSED - ./ICMUAh7zg+000756
 MPI-TASK:   1 -           10 BYTES IN      2 RECORDS TRANSFERRED IN    0.0002 SECONDS     0.1 Mbytes/s, TOTAL TIME=    0.0036( 5.6%)
 STEPO_FPOS WAS : V00000000Y
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMUAh7zg+000756 MODE=a
 IO-STREAM CLOSED - ./ICMUAh7zg+000756
 MPI-TASK:   1 -    302626691 BYTES IN      4 RECORDS TRANSFERRED IN    0.0003 SECONDS******** Mbytes/s, TOTAL TIME=    0.0046( 6.5%)
 STEPO_FPOS WAS : T00000000M
 NSTEP =   756 STEPO   0AAA00AAA
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =   756 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+000756 MODE=a
 IO-STREAM CLOSED - ICMGGh7zg+000756
 MPI-TASK:   1 -    311206420 BYTES IN      7 RECORDS TRANSFERRED IN    0.0009 SECONDS345784.9 Mbytes/s, TOTAL TIME=    0.0095( 9.5%)
 00:12:03 STEP  756 H= 189:00 +CPU= 23.460
 NSTEP =   757 STEPO   0AAA00AAA
 00:12:22 STEP  757 H= 189:15 +CPU= 18.443
 NSTEP =   758 STEPO   0AAA00AAA
 00:12:41 STEP  758 H= 189:30 +CPU= 18.989
 NSTEP =   759 STEPO   0AAA00AAA
 00:12:59 STEP  759 H= 189:45 +CPU= 18.671
ISTEP=     760 RSOLINC= 1361.4482 RI0= 1365.0000 ZI0= 1361.4482 RIP0= 1361.4482
ISTEP=     760 ZI0= 1361.4482 ZSEASON= 1.000000000 REA=149597870000.00 RDEASO=149597870000.00
 NSTEP =   760 STEPO   0AAA00AAA
 00:13:19 STEP  760 H= 190:00 +CPU= 19.630
 NSTEP =   761 STEPO   0AAA00AAA
 00:13:38 STEP  761 H= 190:15 +CPU= 18.667
 NSTEP =   762 STEPO   0AAA00AAA
 00:13:57 STEP  762 H= 190:30 +CPU= 18.998
 NSTEP =   763 STEPO   0AAA00AAA
 00:14:16 STEP  763 H= 190:45 +CPU= 18.708
 NSTEP =   764 STEPO   0AAA00AAA
 00:14:36 STEP  764 H= 191:00 +CPU= 19.652
ISTEP=     765 RSOLINC= 1361.4482 RI0= 1365.0000 ZI0= 1361.4482 RIP0= 1361.4482
ISTEP=     765 ZI0= 1361.4482 ZSEASON= 1.000000000 REA=149597870000.00 RDEASO=149597870000.00
 NSTEP =   765 STEPO   0AAA00AAA
 00:14:55 STEP  765 H= 191:15 +CPU= 18.654
 NSTEP =   766 STEPO   0AAA00AAA
 00:15:13 STEP  766 H= 191:30 +CPU= 18.342
 NSTEP =   767 STEPO   0AAA00AAA
------------------------------------------------
CNT0 not found; string returned was: 'STEPO'
>>> Printing last 8 lines from file: ifs.stat

 00:13:19 0AAA00AAA STEPO      761    19.631   19.631   19.775    251:01    253:14 0.62676023197742E-06       2GB       0MB
 00:13:38 0AAA00AAA STEPO      762    18.667   18.667   18.809    251:19    253:33 0.62676023197742E-06       2GB       0MB
 00:13:57 0AAA00AAA STEPO      763    18.999   18.999   19.138    251:38    253:52 0.62676023197742E-06       2GB       0MB
 00:14:16 0AAA00AAA STEPO      764    18.708   18.708   18.849    251:57    254:11 0.62676023197742E-06       2GB       0MB
 00:14:36 0AAA00AAA STEPO      765    19.653   19.653   19.798    252:17    254:30 0.62676023197742E-06       2GB       0MB
 00:14:55 0AAA00AAA STEPO      766    18.654   18.654   18.796    252:35    254:49 0.62676023197742E-06       2GB       0MB
 00:15:13 0AAA00AAA STEPO      767    18.342   18.342   18.496    252:54    255:08 0.62676023197742E-06       2GB       0MB
------------------------------------------------
>>> Printing last 11 lines from file: rcf

 CSTEP   = "     720",
 CTIME   = "00071200      ",
 NSTEPLPP        =         201,         202,          49,      228226,      228227, 2*0,         708, 2*0,
 IPRGPNSRES      =           1,
 IPRGPEWRES      =           1,
 IPRTRWRES       =           1,
 IPRTRVRES       =           1,
 GMASS0  =  0.000000000000000E+000,
 GMASSI  =  0.000000000000000E+000
 /
------------------------------------------------
>>> Printing last 8 lines from file: /var/lib/boinc/slots/13/progress_file_12292814.xml

<running_values>
  <last_cpu_time>15175.450000</last_cpu_time>
  <upload_file_number>7</upload_file_number>
  <last_iter>767</last_iter>
  <last_upload>604800</last_upload>
  <model_completed>0</model_completed>
</running_values>
------------------------------------------------
..Failed, model did not complete successfully

</stderr_txt>
]]>
No trickles!


©2024 climateprediction.net