|
Message boards :
Number crunching :
OpenIFS Discussion
Message board moderation
Previous · 1 . . . 23 · 24 · 25 · 26 · 27 · 28 · 29 . . . 32 · Next
Author | Message |
---|---|
Send message Joined: 29 Oct 17 Posts: 944 Credit: 15,193,000 RAC: 23,306 |
The idea with this batch of the OpenIFS BL tasks is to explore a wide range of parameters in the model that can be changed to look at storms in a future climate. When the results are back, they will be filtered to look for good candidates to run again but this time at a much higher model resolution. We're not sure yet whether boinc is a suitable framework for these higher resolution tasks (will need > 20Gb RAM, more output etc). |
Send message Joined: 5 Aug 04 Posts: 173 Credit: 15,391,535 RAC: 85,753 |
We're not sure yet whether boinc is a suitable framework for these higher resolution tasks (will need > 20Gb RAM, more output etc).If you would enable Application-Selection for the User you could make it a different Application and only Users that have Opted-In should get them Supporting BOINC, a great concept ! |
![]() Send message Joined: 15 May 09 Posts: 4444 Credit: 17,930,168 RAC: 29,872 |
I agree. Application selection for the user used to be available. I think it was stopped around the same time that model types were only available for single platforms but the way OIFS is developing to my mind at least makes its reintroduction a good idea.We're not sure yet whether boinc is a suitable framework for these higher resolution tasks (will need > 20Gb RAM, more output etc).If you would enable Application-Selection for the User you could make it a different Application and only Users that have Opted-In should get them |
Send message Joined: 29 Oct 17 Posts: 944 Credit: 15,193,000 RAC: 23,306 |
Yes, that's how we're doing it. Each high res configuration will be done as a separate app so it can be controlled differently on the server & user's app_config.xml, but also selected on/off on the Project Preferences page. I suspect by default we'll have these apps deselected so it's an 'opt-in'. It's an identical OpenIFS binary, the model gets its configuration from the input files. This will be setup on the dev site first.I agree. Application selection for the user used to be available. I think it was stopped around the same time that model types were only available for single platforms but the way OIFS is developing to my mind at least makes its reintroduction a good idea.We're not sure yet whether boinc is a suitable framework for these higher resolution tasks (will need > 20Gb RAM, more output etc).If you would enable Application-Selection for the User you could make it a different Application and only Users that have Opted-In should get them --- CPDN Visiting Scientist |
![]() Send message Joined: 15 May 09 Posts: 4444 Credit: 17,930,168 RAC: 29,872 |
I see this one of mine ( the _2 so third attempt) crashed with the overflow message at the same point on all three attempts so good to see the consistent result across all three computers. (All from the Ryzen stable. Work unit This one That I got for second attempt after first computer had problems with the upload at the end, finished normally. |
Send message Joined: 4 Oct 19 Posts: 14 Credit: 8,158,737 RAC: 39,311 |
this approach works well at other projects, and i heartily support this |
Send message Joined: 29 Oct 17 Posts: 944 Credit: 15,193,000 RAC: 23,306 |
Is anyone interested in running a standalone test of the higher resolution OpenIFS configurations we're planning on rolling out? It's the same executable that's gone out before but with a different set of input files. It would not run under the client, a short simple linux script is used to start the model to run 1 forecast day with a small amount of output data. There is NO CREDIT for this - it's just if anyone is interested to see what the model will do on their system. The current batches use: oifs_43r3: Global 125km grid resolution (max virt mem 6Gb / max resident : 5.2Gb) (memory use does vary between systems). The new configurations would be (single core): oifs_43r3_l255 : Global 75km grid resolution (max vmem 20Gb / max res 18Gb ) oifs_43r3_l319 : Global 60km grid resolution (max vmem 32Gb / max res 29Gb) Please note the runtime memory requirements, which is why it would be useful to get some feedback before we decide exactly what to use. Do not attempt to run on machines with less memory than stated above, it will cripple your system. Send Personal Message to me if interested rather than reply here. If there is sufficient interest, I'll share the files on dropbox. I'll post answers to PM'd questions here. --- CPDN Visiting Scientist |
Send message Joined: 12 Apr 21 Posts: 286 Credit: 13,557,282 RAC: 31,430 |
Yes, that's how we're doing it. Each high res configuration will be done as a separate app so it can be controlled differently on the server & user's app_config.xml, but also selected on/off on the Project Preferences page. I suspect by default we'll have these apps deselected so it's an 'opt-in'. It's an identical OpenIFS binary, the model gets its configuration from the input files. This will be setup on the dev site first.I agree. Application selection for the user used to be available. I think it was stopped around the same time that model types were only available for single platforms but the way OIFS is developing to my mind at least makes its reintroduction a good idea.We're not sure yet whether boinc is a suitable framework for these higher resolution tasks (will need > 20Gb RAM, more output etc).If you would enable Application-Selection for the User you could make it a different Application and only Users that have Opted-In should get them I also agree that this should be reinstated as well as be an 'opt-in'. I think the only models that should be automatically opted-in are Windows and current, lowest resolution OIFS. All others require or will require either special configurations (32-bit libraries) or hardware (RAM, older Mac). |
Send message Joined: 5 Aug 04 Posts: 173 Credit: 15,391,535 RAC: 85,753 |
I also agree that this should be reinstated as well as be an 'opt-in'. I think the only models that should be automatically opted-in are Windows and current, lowest resolution OIFS. All others require or will require either special configurations (32-bit libraries) or hardware (RAM, older Mac). Perhaps you can place the Infos about needed RAM / HD-Space / Libraries / ... there. Then everybody who wants to Opt-In will have seen these Details. A good example for this is Primegrid, the site of the project-Preferences is very informativ Supporting BOINC, a great concept ! |
Send message Joined: 9 Mar 22 Posts: 30 Credit: 1,065,239 RAC: 556 |
As far as a simple RAM monitoring shows oifs tasks seem to have a large but (most of the time) stable RAM requirement while a step is in progress. At the beginning of each step there's a small additional peak and at the end of a step lots of RAM is released for a short time until the next step starts. That short release might be responsible for BOINC client's intermediate miscalculation regarding the RAM estimation and finally lead to an overcommitment. To make BOINC more stable the idea would be to check within oifs whether the maximum RAM requirement can be estimated before the task allocates it from the heap. It might then be possible to reuse the same RAM until all steps are processed. |
Send message Joined: 5 Aug 04 Posts: 1106 Credit: 17,025,459 RAC: 5,652 |
To make BOINC more stable the idea would be to check within oifs whether the maximum RAM requirement can be estimated before the task allocates it from the heap. I do not know how the Oifs task allocates and frees its RAM, but if it uses the malloc, free, calloc, realloc suite of functions, this can be fairly easily arranged (at least, in principle). See the section 3 of the manual pages for how to do that. Basically, you allocate the amount of storage you think you need, with malloc(). If you need more, you can get it (if it is available) with ralloc(). This may have already been done in Oifs. Generally, it is advised to Avoid using brk() and sbrk(): the malloc(3) memory allocation package is the portable and comfortable way of allocating memory. ![]() |
Send message Joined: 5 Aug 04 Posts: 1106 Credit: 17,025,459 RAC: 5,652 |
Send Personal Message to me if interested rather than reply here. User Glenn Carver (ID: 1560856) is not accepting private messages from you ![]() |
Send message Joined: 4 Dec 15 Posts: 52 Credit: 2,265,583 RAC: 3,415 |
Usually there's a feature in the Boinc Server software to send beta-tasks to users via the client if those users have opted-in to receive those tasks. I would have done so btw. if the CPDN preferences had supported this. ;-) - - - - - - - - - - Greetings, Jens |
Send message Joined: 12 Apr 21 Posts: 286 Credit: 13,557,282 RAC: 31,430 |
Usually there's a feature in the Boinc Server software to send beta-tasks to users via the client if those users have opted-in to receive those tasks. That's right and a number of projects have it, you also get some credit for it. Since CPDN already has a development site, it's unlikely they're going to change their system when it comes to beta testing. |
![]() Send message Joined: 15 May 09 Posts: 4444 Credit: 17,930,168 RAC: 29,872 |
Credit for tasks on testing site for cpdn shows up as a separate project.Usually there's a feature in the Boinc Server software to send beta-tasks to users via the client if those users have opted-in to receive those tasks. |
![]() Send message Joined: 15 May 09 Posts: 4444 Credit: 17,930,168 RAC: 29,872 |
Credit for tasks on testing site for cpdn shows up as a separate project.Usually there's a feature in the Boinc Server software to send beta-tasks to users via the client if those users have opted-in to receive those tasks. However as Glen has said, the test he wants to run is via a script to run in a terminal and not through BOINC at all. |
Send message Joined: 1 Jan 07 Posts: 1014 Credit: 35,789,563 RAC: 25,282 |
First Africa task from batch 992 failed - Task 22314299 oifs_43r3_0016_2019110100_123_992_12213329_0 ABORT! 1 SUECOZC:ERROR OPENING FILE ECOZC forrtl: error (76): Abort trap signal ..The child process has been killed with signal: 6 oifs_get_stat: Error. ifs.stat file is not open I'll leave the rest to the experts. |
Send message Joined: 4 Oct 19 Posts: 14 Credit: 8,158,737 RAC: 39,311 |
all mine on 2 machines give much the same |
Send message Joined: 1 Jan 07 Posts: 1014 Credit: 35,789,563 RAC: 25,282 |
Distribution of this batch has been paused while the errors are investigated. |
Send message Joined: 5 Aug 04 Posts: 1106 Credit: 17,025,459 RAC: 5,652 |
We use the large ensemble of the OpenIFS@Home to identify the connections between the tropical Indian Ocean and the North Atlantic region. Is that the name of the tasks I use in app_config.xml? Like this? app_config> <project_max_concurrent>6</project_max_concurrent> <app> <name>oifs_43r3</name> <---<<< <max_concurrent>1</max_concurrent> </app> <app> <name>oifs_43r3_bl</name> <max_concurrent>5</max_concurrent> </app> <app> <name>oifs_43r3_ps</name> <max_concurrent>4</max_concurrent> </app> <app> <name>oifs_43r3_l255</name> <max_concurrent>1</max_concurrent> </app> <app> <name>oifs_43r3_l319</name> <max_concurrent>1</max_concurrent> </app> </app_config> I am not accepting new tasks at the moment. My additional 64 GBytes of RAM are due in tomorrow sometime. ![]() |
©2024 climateprediction.net