Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-09-20
15:06:48 stephenfin Unfortunately the impact of a fix will likely be significant, which is why we've punted it continuously /o\
15:07:22 mriedem do you know if it's documented as a known issue?
15:07:22 sean-k-mooney they can get numa aware ram reservation by setting hw:mem_page_size=small
15:07:35 sean-k-mooney if you are using cpu pinning but not hugepages you should set that
15:07:48 mriedem yeah the bug says "Many mitigation are "invented", but those mitigation all have some form of technical or operational "difficulties". One mitigation, for example, is to enable huge pages, and put VMs on huge pages."
15:08:03 aspiers mriedem: I'm sure it was honest ignorance of the policy rather than deliberately avoiding work. Your -1 should be sufficient to get it fixed.
15:08:30 sean-k-mooney mriedem: hugepages will prevent it but hw:mem_page_size=small will use 4k pages and still fix the issue
15:08:36 mriedem aspiers: i didn't say anyone was deliberately avoiding doing the work,
15:08:44 stephenfin mriedem: yeah, what sean-k-mooney said
15:08:44 mriedem but i've also seen it across several projects from multiple suse developers
15:08:47 mriedem so it's a pattern
15:08:58 mriedem hence my post to the ML
15:09:10 aspiers mriedem: OK, I'll forward it internally
15:09:19 mriedem aspiers: thanks
15:09:31 mriedem sean-k-mooney: stephenfin: ok it would be good if we had this documented as a limitation somewhere if we don't already
15:09:49 mriedem i'm not sure if that's best in the numa topo admin docs, flavor extra spec, reserved host ram config option, other?
15:09:54 stephenfin I think we should, but I'll check. Will be easy tack onto this existing doc
15:10:12 sean-k-mooney ya we should document the requirement to set hw:mem_page_size small if you set hw:cpu_policy=dedicated and dont you hugepages
15:10:55 mriedem ok so maybe it's best to just document alongside that extra spec https://docs.openstack.org/nova/latest/user/flavors.html
15:11:09 sean-k-mooney the reason this happens is when you enable pinning with out settin hw:mem_page_size we only look at available cpus in the numa toplogy filter not allso aviable memory on the numa nodes
15:11:35 sean-k-mooney i fyou enable hw:mem_page_size=small|large it validate the avaiable memory too
15:12:03 stephenfin sean-k-mooney: isn't there some issue with how we track mempages though
15:12:23 stephenfin none of this is obviously in placement yet
15:12:34 sean-k-mooney you cant mix vms that use hw:mem_page_size with vms that dont on the same host or it invalidate the tracking
15:12:46 sean-k-mooney but you should not mix numa and non numa vms on the same host anyway
15:13:19 mriedem "but you should not mix numa and non numa vms on the same host anyway" - is that documented?
15:13:28 mriedem we have a shit load of tribal knowledge about this stuff
15:13:34 mriedem and by we i mean you
15:14:02 mriedem all i see on https://docs.openstack.org/nova/latest/admin/cpu-topologies.html is "Host aggregates should be used to separate pinned instances from unpinned instances as the latter will not respect the resourcing requirements of the former."
15:14:21 sean-k-mooney i dont think its in the upstream docs or redhats it was in the intel tuning guide i helped writhe in like 2014
15:14:35 sean-k-mooney so yes i shoudl document the tribal knowlage around this
15:14:49 sean-k-mooney *my tribale knowlasge around this
15:15:28 sean-k-mooney the extention of that warning to numa instead of pinned is to because of the OOM behavior
15:16:02 sean-k-mooney i can try and write something for this on monday if you / stephen can correct the spelling after
15:16:17 kashyap sean-k-mooney: Happy to look as well :-)
15:17:01 mriedem ah here is the duplicate https://bugs.launchpad.net/nova/+bug/1792985
15:17:03 openstack Launchpad bug 1439247 in OpenStack Compute (nova) "duplicate for #1792985 Small pages memory are not take into account when not explicitly requested" [Medium,Confirmed]
15:17:08 mriedem which is a duplicate of another bug
15:17:12 aspiers mriedem: forwarded internally. If you see it happen again feel free to ping me privately
15:17:57 mriedem ack
15:19:34 sean-k-mooney mriedem: do you want to assing one of those bugs to me and i can update the documentaiton regarding the mixing of numa and non numa instances and the use of hw:mem_page_size=small|large to avoid OOM events for numa affined instances
15:19:57 mriedem artom: this isn't fixed with your numa live migration series right? https://bugs.launchpad.net/nova/+bug/1496135 - that's the thing you were half fixing but we had you remove it?
15:19:58 openstack Launchpad bug 1496135 in OpenStack Compute (nova) "live-migration will not honor destination vcpu_pin_set config" [Medium,Confirmed]
15:20:19 mriedem sean-k-mooney: i can only do so much assing in one day
15:20:54 sean-k-mooney see alrealy indicating my spelling mistakes :)
15:21:08 mriedem sean-k-mooney: i don't think we probably need to assign it to you, a docs patch would just be a related bug anyway
15:21:13 sean-k-mooney am i thnk that should be fixed by the numa live migration changes
15:21:18 artom mriedem, not clear - they talk about policy, which I assume means hw:cpu_policy, which my thing *did* fix
15:21:19 mriedem you can track a todo however you want though
15:21:44 sean-k-mooney oh
15:21:51 sean-k-mooney this is the edgecase we removed
15:21:58 artom But... then they also talk about vcpuset
15:22:03 sean-k-mooney yes
15:22:03 artom So, it's confusing
15:22:17 sean-k-mooney this is migration of a non numa instance between host with vcpu_pin_set defined
15:22:27 sean-k-mooney so the thing we added then removed
15:22:35 sean-k-mooney so this is not fixed yet
15:22:45 artom I guess? In any case, that's what it can become, because we need something to track that
15:23:28 mriedem ok i left a comment https://bugs.launchpad.net/nova/+bug/1496135/comments/10
15:23:29 openstack Launchpad bug 1496135 in OpenStack Compute (nova) "live-migration will not honor destination vcpu_pin_set config" [Medium,Confirmed]
15:24:19 mriedem artom: while you're here, are we going to try and get https://review.opendev.org/#/c/672595/ into train yet?
15:24:33 mriedem i'm not sure how much fun that would be to backport
15:24:35 artom mriedem, I'm always here man
15:24:38 mriedem though it's all test stuff
15:24:44 mriedem you weren't yesterday when i needed to bug you
15:24:57 artom PTO, apple picking with son's daycare
15:25:07 mriedem see, you're not *always* here
15:25:09 artom Bought some cider, good stuff
15:25:14 artom In my hear I am
15:25:18 artom *heart, even
15:25:19 mriedem heart
15:25:28 artom Anyways
15:25:28 mriedem meanwhile i'm just assing around
15:25:41 artom Yeah, the func test should land in Train
15:25:58 artom If you and dansmith are ready to dive back in, I can pick it up again
15:26:14 artom Although it conflicts with some of the PCPU stuff that's still in flight
15:26:43 mriedem i've added it to https://etherpad.openstack.org/p/nova-train-release-todo
15:26:48 artom https://review.opendev.org/#/c/681060/ specifically
15:26:49 mriedem i'm not going to review that today
15:27:00 artom ^^ needs to land first, anyways, then I rebase and continue
15:27:14 mriedem so likely rc2 at this point
15:28:19 artom I guess yeah
15:29:57 mriedem what is the end of the pcpu series? the reshaper patch? because i was thinking about rebasing https://review.opendev.org/#/c/683011/ on top of that so we can approve it to make sure it gets into rc1
15:30:12 sean-k-mooney if its a test only patch we could technically backport it right
15:30:17 openstackgerrit Matt Riedemann proposed openstack/nova master: Revert "Temporarily skip TestNovaMigrationsMySQL" https://review.opendev.org/683011
15:30:18 sean-k-mooney so does it need to land now
15:30:21 mriedem sean-k-mooney: correct
15:30:49 mriedem sean-k-mooney: what i worry about is saying "sure we'll get to it in ussuri" and then 4 months go by and dansmith and i have lost all context on that stuff and then we don't want to land it and it just never lands
15:31:13 mriedem strike while the iron is hot and all that
15:31:22 sean-k-mooney i think we should time box any of this stuff to ensure it before m1
15:31:28 sean-k-mooney ya
15:32:20 sean-k-mooney i was just suggesting that if did not make rc1/2 it could still end up in train
15:35:56 mriedem if anyone is looking for glance image properties docs to work on https://bugs.launchpad.net/nova/+bug/1763761
15:35:57 openstack Launchpad bug 1763761 in Glance "CPU topologies in nova - doesn't mention numa specific image properties" [Medium,Triaged]
15:39:45 mriedem looking at old numa bugs, it looks like https://review.opendev.org/#/c/458848/ just needed some test,
15:39:51 mriedem might be something for a core to pick up
15:41:02 mriedem artom: i wonder if this is the same numa topology claim race failure thing described in the ML https://bugs.launchpad.net/nova/+bug/1829349
15:41:03 openstack Launchpad bug 1829349 in OpenStack Compute (nova) "Resource Tracker failed to update usage in case numa topology conflict happen" [Undecided,In progress] - Assigned to leehom (feli5)
15:41:06 artom mriedem, not sure that's still applicable, stephenfin did a thing around that a long time ago
15:41:11 mriedem note they are enabling he workaround config

Earlier   Later