| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-07-02 | |||
| 12:09:06 | openstackgerrit | Martin Midolesov proposed openstack/nova master: Implementing graceful shutdown. https://review.opendev.org/666245 | |
| 12:10:04 | maxbahosh3 | hi there, i have problem with numa and my flavors. | |
| 12:10:06 | maxbahosh3 | i have 2 socket servers and 4 socket servers, and i want to set numa to user all socket for memory | |
| 12:10:12 | maxbahosh3 | i set hw:numa_nodes='2' for 2 socket server and hw:numa_nodes='4' for 4 socket | |
| 12:10:18 | maxbahosh3 | is it any way to configure one flavor to use for both type of servers? | |
| 12:16:30 | sean-k-mooney | maxbahosh3: no | |
| 12:16:52 | sean-k-mooney | you need two flavors or just use hw:numa_nodes=2 for all of them | |
| 12:17:15 | sean-k-mooney | maxbahosh3: well that or just dont set hw:numa_nodes at all | |
| 12:17:29 | sean-k-mooney | but that also means you cant use cpu pinning or hugepages | |
| 12:18:20 | maxbahosh3 | if i use numa nodes 2, just use 2 socket and and oom maybe occur | |
| 12:18:56 | maxbahosh3 | if not set, does it use all of socket? | |
| 12:19:22 | sean-k-mooney | if not set it uses all soceckt | |
| 12:19:42 | sean-k-mooney | what oom errors are you seeing | |
| 12:20:02 | sean-k-mooney | you shoudl not see the kernel killing instances | |
| 12:20:29 | sean-k-mooney | you may not be able to fit a vm but that would just be beacuse of the normal ttris problem | |
| 12:21:05 | maxbahosh3 | i see oom kiiler error cause of it use one instance, use one socket | |
| 12:21:25 | maxbahosh3 | so, i use numa node to use two socket | |
| 12:21:38 | sean-k-mooney | that means you have not correctly reserved memoyr per numa node | |
| 12:22:34 | maxbahosh3 | how can i reserved memory? | |
| 12:22:38 | maxbahosh3 | do you mean reserved_host_memory_mb ? | |
| 12:22:46 | sean-k-mooney | the oom killer runs per numa node and will be trigered whenever the kernel needs memroy on a specific numa node even if there is memory free on a different hode | |
| 12:22:50 | sean-k-mooney | *node | |
| 12:23:05 | sean-k-mooney | the best way to avoid this is to use hugepages | |
| 12:24:13 | sean-k-mooney | reserved_host_memory_mb is not enough. you also neeed to use cgroups to limit the memoy available to qemu/libvirt so that it cannot use all memory on each indivigual node | |
| 12:24:20 | maxbahosh3 | mmm, i didnt ahave any configuration for hugepages ... | |
| 12:24:40 | sean-k-mooney | if you use vm with hugepages it will both improve teh performace of the vms and prevent the OOM issues | |
| 12:25:41 | maxbahosh3 | do you have good tuning link for it? (i'm going to search on google) | |
| 12:31:49 | maxbahosh3 | sean-k-mooney | |
| 12:31:51 | maxbahosh3 | https://www.openstack.org/assets/presentation-media/performance-rhosp-summit-rmahroua2.pdf | |
| 12:32:18 | maxbahosh3 | i found this, i think it's a good point to start this tuning. thanks a lot | |
| 13:04:58 | sean-k-mooney | gibi alex_xu just responded to https://review.opendev.org/#/c/662264/5 ill respin in an hour or two and adress the issue you raised, let me know if the repsoces make sense or if you have other question and ill adress them all as part of the respin. thanks for reviewing :) | |
| 13:05:24 | gibi | sean-k-mooney: ack | |
| 13:08:20 | gibi | sean-k-mooney: your responses looks good to me | |
| 13:08:52 | sean-k-mooney | ok ill leave it as is for people to leave more feedback if the chose too and ill resping as i said in an hour or so | |
| 13:09:10 | gibi | sean-k-mooney: cool | |
| 13:09:23 | sean-k-mooney | in general are you ok with the feature? include the seperate RPs to make reshap simpler in the future? | |
| 13:10:26 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Stabilize unshelve notification sample tests https://review.opendev.org/668675 | |
| 13:10:52 | gibi | efried, mriedem: super simple test only change ^^ | |
| 13:12:10 | sean-k-mooney | i need to read the commit but how is this sablising the test? | |
| 13:13:03 | openstackgerrit | Kashyap Chamarthy proposed openstack/nova-specs master: Add "Secure Boot support for KVM & QEMU guests" spec https://review.opendev.org/506720 | |
| 13:13:43 | kashyap | lyarwood: Hey, when you get 10-ish minutes, want to have a gander at ⮭ | |
| 13:14:11 | kashyap | (You looked at it previously.) I've also included a short changelog of all the things I addressed, based on johnthetubaguy's feedforward. | |
| 13:16:00 | lyarwood | kashyap: ack will do | |
| 13:16:54 | efried | sean-k-mooney: Thanks for pointing out that MKTME spec. I was expecting it to show up, but didn't see it come in for some reason, possibly because it's proposed to backlog/ and I'm focused on train/. | |
| 13:17:21 | sean-k-mooney | oh i totally missed that | |
| 13:17:28 | sean-k-mooney | i thought it was for train | |
| 13:17:31 | efried | cdent: Unfortunately set_inventory_for_provider retry isn't "clean" because it goes through a dozen layers of the periodic update in the resource tracker | |
| 13:17:39 | efried | gibi: ack | |
| 13:18:10 | sean-k-mooney | in that case im less worried about the scope/alingment with the current sev spec | |
| 13:18:19 | mriedem | lyarwood: melwitt: i replied about the conflicts in this queens backport https://review.opendev.org/#/c/662471/ | |
| 13:18:24 | gibi | sean-k-mooney: yes, those test could fail if the timing is unlucky | |
| 13:18:54 | sean-k-mooney | ya i just figured out how this fixes it | |
| 13:18:56 | efried | cdent: oh, actually, snot so bad. set_inventory_for_provider called by update_from_provider_tree (both in report client) called by _update_to_placement (in resource tracker) | |
| 13:19:04 | efried | cdent: _update_to_placement has the retry decorator on it. | |
| 13:19:11 | sean-k-mooney | gibi: we are setting up the verifies but the test could end before the notificaion is sent | |
| 13:19:15 | efried | cdent: | |
| 13:19:15 | efried | @retrying.retry(stop_max_attempt_number=4, | |
| 13:19:15 | efried | retry_on_exception=lambda e: isinstance( | |
| 13:19:15 | efried | e, exception.ResourceProviderUpdateConflict)) | |
| 13:19:26 | mriedem | lyarwood: sean-k-mooney: is stephenfin around this week? | |
| 13:19:45 | sean-k-mooney | mriedem: he was on eairlier | |
| 13:20:09 | sean-k-mooney | mriedem: and he does not have PTO in our team calander so i think so | |
| 13:20:17 | gibi | sean-k-mooney: yes, to core of the issue is that instance.vm_state is set ACTIVE before the the end notification is emitted but the test only waits for the ACTIVE state | |
| 13:20:23 | sean-k-mooney | he might be at the gym at the moment | |
| 13:20:26 | mriedem | ok looking for answers on this queens numa live migration change https://review.opendev.org/#/c/629597/ | |
| 13:20:57 | sean-k-mooney | mriedem: ya i was planning to bug him about that chagne today too | |
| 13:21:38 | sean-k-mooney | mriedem: your trying to do a queens release soon right | |
| 13:21:54 | stephenfin | mriedem: I'm here, yeah | |
| 13:21:56 | sean-k-mooney | i would like to include that in the next queens release if we can | |
| 13:22:28 | stephenfin | mriedem: Saw comments on a queens backport in the scrollback. I'll get to that once I've finished with this mountain of email :'( | |
| 13:23:52 | efried | yaawang: If you like, I can update the cpu_models spec real quick | |
| 13:24:07 | mriedem | sean-k-mooney: yes i'm flushing for a queens release | |
| 13:24:20 | mriedem | nearly there https://review.opendev.org/#/q/status:open+project:openstack/nova+branch:stable/queens+label:Code-Review=2 | |
| 13:27:27 | openstackgerrit | ya.wang proposed openstack/nova-specs master: Re-propose the spec to allow specifying a list of CPU models https://review.opendev.org/642030 | |
| 13:31:25 | efried | alex_xu, stephenfin, gibi: I'm +2 on ^ if one of you wants to push it. | |
| 13:31:54 | efried | kashyap: Are you convinced enough of the use case to accept this being implemented? ^ | |
| 13:31:56 | stephenfin | efried: The spec? Has kashyap looked through that again? | |
| 13:31:59 | stephenfin | Jinx :) | |
| 13:32:30 | stephenfin | I'd like to look through that again myself. Can we hold off until tomorrow at least? If I don't get to it by then, carry on | |
| 13:32:33 | efried | kashyap: note that under the latest design proposal, existing confs will continue to work, so the load on the deployer isn't as great. | |
| 13:32:37 | sean-k-mooney | ill quickly re reivew it too now but i was ok with the usecase in the past i think | |
| 13:33:16 | kashyap | efried: Will look once I address gibi's comments in my own spec | |
| 13:33:22 | kashyap | stephenfin: ^ | |
| 13:33:22 | efried | stephenfin, gibi, sean-k-mooney: ack, no big hurry I guess. Thanks kashyap. | |
| 13:34:01 | sean-k-mooney | efried: its proably one of the spec we should be able to close out on today as part of the review day | |
| 13:34:09 | kashyap | gibi: On the Secure Boot thing, fixing your remarks. My eyes are glazing over from looking at my own text. Note that the "firmware descriptor files" from QEMU are nice-to-have, but doesn't block Train | |
| 13:34:15 | kashyap | I'll add notes there. | |
| 13:34:19 | efried | sean-k-mooney: yeah, that was my hope :) | |
| 13:34:28 | gibi | kashyap: ack | |
| 13:34:31 | sean-k-mooney | efried: by the way do you have the link handy to the dashboard you created | |
| 13:34:33 | kashyap | sean-k-mooney: I was not OK with the spec, and actually abandoned it | |
| 13:34:41 | efried | sean-k-mooney: https://etherpad.openstack.org/p/nova-spec-review-day | |
| 13:34:45 | sean-k-mooney | efried: thanks | |
| 13:34:47 | kashyap | But it was re-opened, and I want to see if I can convince myself again | |
| 13:34:56 | efried | sean-k-mooney: front matter has a couple of gerrit dashboard links. | |
| 13:35:00 | stephenfin | Is today a review day? | |
| 13:35:11 | stephenfin | *spec review day | |
| 13:35:11 | sean-k-mooney | stephenfin: yes | |
| 13:35:19 | sean-k-mooney | i think the final one for train | |
| 13:35:22 | stephenfin | Crap. Ugh. Too much to do | |