Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-09-20
15:09:10 aspiers mriedem: OK, I'll forward it internally
15:09:19 mriedem aspiers: thanks
15:09:31 mriedem sean-k-mooney: stephenfin: ok it would be good if we had this documented as a limitation somewhere if we don't already
15:09:49 mriedem i'm not sure if that's best in the numa topo admin docs, flavor extra spec, reserved host ram config option, other?
15:09:54 stephenfin I think we should, but I'll check. Will be easy tack onto this existing doc
15:10:12 sean-k-mooney ya we should document the requirement to set hw:mem_page_size small if you set hw:cpu_policy=dedicated and dont you hugepages
15:10:55 mriedem ok so maybe it's best to just document alongside that extra spec https://docs.openstack.org/nova/latest/user/flavors.html
15:11:09 sean-k-mooney the reason this happens is when you enable pinning with out settin hw:mem_page_size we only look at available cpus in the numa toplogy filter not allso aviable memory on the numa nodes
15:11:35 sean-k-mooney i fyou enable hw:mem_page_size=small|large it validate the avaiable memory too
15:12:03 stephenfin sean-k-mooney: isn't there some issue with how we track mempages though
15:12:23 stephenfin none of this is obviously in placement yet
15:12:34 sean-k-mooney you cant mix vms that use hw:mem_page_size with vms that dont on the same host or it invalidate the tracking
15:12:46 sean-k-mooney but you should not mix numa and non numa vms on the same host anyway
15:13:19 mriedem "but you should not mix numa and non numa vms on the same host anyway" - is that documented?
15:13:28 mriedem we have a shit load of tribal knowledge about this stuff
15:13:34 mriedem and by we i mean you
15:14:02 mriedem all i see on https://docs.openstack.org/nova/latest/admin/cpu-topologies.html is "Host aggregates should be used to separate pinned instances from unpinned instances as the latter will not respect the resourcing requirements of the former."
15:14:21 sean-k-mooney i dont think its in the upstream docs or redhats it was in the intel tuning guide i helped writhe in like 2014
15:14:35 sean-k-mooney so yes i shoudl document the tribal knowlage around this
15:14:49 sean-k-mooney *my tribale knowlasge around this
15:15:28 sean-k-mooney the extention of that warning to numa instead of pinned is to because of the OOM behavior
15:16:02 sean-k-mooney i can try and write something for this on monday if you / stephen can correct the spelling after
15:16:17 kashyap sean-k-mooney: Happy to look as well :-)
15:17:01 mriedem ah here is the duplicate https://bugs.launchpad.net/nova/+bug/1792985
15:17:03 openstack Launchpad bug 1439247 in OpenStack Compute (nova) "duplicate for #1792985 Small pages memory are not take into account when not explicitly requested" [Medium,Confirmed]
15:17:08 mriedem which is a duplicate of another bug
15:17:12 aspiers mriedem: forwarded internally. If you see it happen again feel free to ping me privately
15:17:57 mriedem ack
15:19:34 sean-k-mooney mriedem: do you want to assing one of those bugs to me and i can update the documentaiton regarding the mixing of numa and non numa instances and the use of hw:mem_page_size=small|large to avoid OOM events for numa affined instances
15:19:57 mriedem artom: this isn't fixed with your numa live migration series right? https://bugs.launchpad.net/nova/+bug/1496135 - that's the thing you were half fixing but we had you remove it?
15:19:58 openstack Launchpad bug 1496135 in OpenStack Compute (nova) "live-migration will not honor destination vcpu_pin_set config" [Medium,Confirmed]
15:20:19 mriedem sean-k-mooney: i can only do so much assing in one day
15:20:54 sean-k-mooney see alrealy indicating my spelling mistakes :)
15:21:08 mriedem sean-k-mooney: i don't think we probably need to assign it to you, a docs patch would just be a related bug anyway
15:21:13 sean-k-mooney am i thnk that should be fixed by the numa live migration changes
15:21:18 artom mriedem, not clear - they talk about policy, which I assume means hw:cpu_policy, which my thing *did* fix
15:21:19 mriedem you can track a todo however you want though
15:21:44 sean-k-mooney oh
15:21:51 sean-k-mooney this is the edgecase we removed
15:21:58 artom But... then they also talk about vcpuset
15:22:03 sean-k-mooney yes
15:22:03 artom So, it's confusing
15:22:17 sean-k-mooney this is migration of a non numa instance between host with vcpu_pin_set defined
15:22:27 sean-k-mooney so the thing we added then removed
15:22:35 sean-k-mooney so this is not fixed yet
15:22:45 artom I guess? In any case, that's what it can become, because we need something to track that
15:23:28 mriedem ok i left a comment https://bugs.launchpad.net/nova/+bug/1496135/comments/10
15:23:29 openstack Launchpad bug 1496135 in OpenStack Compute (nova) "live-migration will not honor destination vcpu_pin_set config" [Medium,Confirmed]
15:24:19 mriedem artom: while you're here, are we going to try and get https://review.opendev.org/#/c/672595/ into train yet?
15:24:33 mriedem i'm not sure how much fun that would be to backport
15:24:35 artom mriedem, I'm always here man
15:24:38 mriedem though it's all test stuff
15:24:44 mriedem you weren't yesterday when i needed to bug you
15:24:57 artom PTO, apple picking with son's daycare
15:25:07 mriedem see, you're not *always* here
15:25:09 artom Bought some cider, good stuff
15:25:14 artom In my hear I am
15:25:18 artom *heart, even
15:25:19 mriedem heart
15:25:28 artom Anyways
15:25:28 mriedem meanwhile i'm just assing around
15:25:41 artom Yeah, the func test should land in Train
15:25:58 artom If you and dansmith are ready to dive back in, I can pick it up again
15:26:14 artom Although it conflicts with some of the PCPU stuff that's still in flight
15:26:43 mriedem i've added it to https://etherpad.openstack.org/p/nova-train-release-todo
15:26:48 artom https://review.opendev.org/#/c/681060/ specifically
15:26:49 mriedem i'm not going to review that today
15:27:00 artom ^^ needs to land first, anyways, then I rebase and continue
15:27:14 mriedem so likely rc2 at this point
15:28:19 artom I guess yeah
15:29:57 mriedem what is the end of the pcpu series? the reshaper patch? because i was thinking about rebasing https://review.opendev.org/#/c/683011/ on top of that so we can approve it to make sure it gets into rc1
15:30:12 sean-k-mooney if its a test only patch we could technically backport it right
15:30:17 openstackgerrit Matt Riedemann proposed openstack/nova master: Revert "Temporarily skip TestNovaMigrationsMySQL" https://review.opendev.org/683011
15:30:18 sean-k-mooney so does it need to land now
15:30:21 mriedem sean-k-mooney: correct
15:30:49 mriedem sean-k-mooney: what i worry about is saying "sure we'll get to it in ussuri" and then 4 months go by and dansmith and i have lost all context on that stuff and then we don't want to land it and it just never lands
15:31:13 mriedem strike while the iron is hot and all that
15:31:22 sean-k-mooney i think we should time box any of this stuff to ensure it before m1
15:31:28 sean-k-mooney ya
15:32:20 sean-k-mooney i was just suggesting that if did not make rc1/2 it could still end up in train
15:35:56 mriedem if anyone is looking for glance image properties docs to work on https://bugs.launchpad.net/nova/+bug/1763761
15:35:57 openstack Launchpad bug 1763761 in Glance "CPU topologies in nova - doesn't mention numa specific image properties" [Medium,Triaged]
15:39:45 mriedem looking at old numa bugs, it looks like https://review.opendev.org/#/c/458848/ just needed some test,
15:39:51 mriedem might be something for a core to pick up
15:41:02 mriedem artom: i wonder if this is the same numa topology claim race failure thing described in the ML https://bugs.launchpad.net/nova/+bug/1829349
15:41:03 openstack Launchpad bug 1829349 in OpenStack Compute (nova) "Resource Tracker failed to update usage in case numa topology conflict happen" [Undecided,In progress] - Assigned to leehom (feli5)
15:41:06 artom mriedem, not sure that's still applicable, stephenfin did a thing around that a long time ago
15:41:11 mriedem note they are enabling he workaround config
15:42:40 artom mriedem, I don't think that's related - their bug is "if two instances have the same pins, RT blows up"
15:43:00 artom But with NUMA LM we're not supposed to end up in that situation in the first place
15:43:21 mriedem "supposed to"
15:43:33 mriedem but if we have a race with claims not reporting correctly or something, we could still race and blow up right?
15:44:00 artom Right
15:44:34 mriedem anyway there are some details from the reporter on what they think the problem is and where, so maybe would help
15:45:02 artom mriedem, well they proposed https://review.opendev.org/#/c/661208/
15:45:06 artom Which is... no
15:46:10 artom I need food
15:52:03 sean-k-mooney mriedem: if your looking for numa related bug we should fix sooner rather then later we should aim to merge and backport https://review.opendev.org/#/c/662522/ sonner rather then later
15:52:30 sean-k-mooney its alex_xu and stephenfin fix for https://bugs.launchpad.net/nova/+bug/1805767
15:52:32 openstack Launchpad bug 1805767 in OpenStack Compute (nova) "The new numa topology in the new flavor extra specs weren't parsed when resize" [Medium,In progress] - Assigned to Stephen Finucane (stephenfinucane)

Earlier   Later