| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-09-20 | |||
| 14:59:26 | mriedem | i don't think so | |
| 15:02:19 | gibi | whaaat? | |
| 15:02:30 | gibi | who will be in Shanghai then? | |
| 15:04:26 | mnaser | its friday and i dont want to deal with this >:( | |
| 15:04:37 | mnaser | nova_api.instance_mappings shows a mapping for an instance | |
| 15:04:39 | bauzas | gibi: \o | |
| 15:04:49 | gibi | bauzas: :) | |
| 15:04:53 | mnaser | nova.instances shows the instance there with deleted=0, vm_state=error though | |
| 15:05:04 | bauzas | at least we can ask a room for two :p | |
| 15:05:14 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Func test for migrate reschedule with pinned compute rpc https://review.opendev.org/683385 | |
| 15:05:14 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Handle legacy request spec dict in ComputeTaskManager._cold_migrate https://review.opendev.org/680762 | |
| 15:05:15 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Isolate request spec handling from _cold_migrate https://review.opendev.org/680763 | |
| 15:05:18 | mriedem | stephenfin: you might have thoughts on this https://bugs.launchpad.net/nova/+bug/1844721 | |
| 15:05:19 | openstack | Launchpad bug 1844721 in OpenStack Compute (nova) "Need NUMA aware RAM reservation to avoid OOM killing host processes" [Undecided,New] | |
| 15:05:31 | mnaser | api gives 404 when looking up the instance.. | |
| 15:05:39 | mnaser | but its there when doing a list | |
| 15:06:20 | stephenfin | mriedem: Yup, that's a long standing issue. There's definitely at least one existing report of that. Possibly many | |
| 15:06:48 | stephenfin | Unfortunately the impact of a fix will likely be significant, which is why we've punted it continuously /o\ | |
| 15:07:22 | mriedem | do you know if it's documented as a known issue? | |
| 15:07:22 | sean-k-mooney | they can get numa aware ram reservation by setting hw:mem_page_size=small | |
| 15:07:35 | sean-k-mooney | if you are using cpu pinning but not hugepages you should set that | |
| 15:07:48 | mriedem | yeah the bug says "Many mitigation are "invented", but those mitigation all have some form of technical or operational "difficulties". One mitigation, for example, is to enable huge pages, and put VMs on huge pages." | |
| 15:08:03 | aspiers | mriedem: I'm sure it was honest ignorance of the policy rather than deliberately avoiding work. Your -1 should be sufficient to get it fixed. | |
| 15:08:30 | sean-k-mooney | mriedem: hugepages will prevent it but hw:mem_page_size=small will use 4k pages and still fix the issue | |
| 15:08:36 | mriedem | aspiers: i didn't say anyone was deliberately avoiding doing the work, | |
| 15:08:44 | stephenfin | mriedem: yeah, what sean-k-mooney said | |
| 15:08:44 | mriedem | but i've also seen it across several projects from multiple suse developers | |
| 15:08:47 | mriedem | so it's a pattern | |
| 15:08:58 | mriedem | hence my post to the ML | |
| 15:09:10 | aspiers | mriedem: OK, I'll forward it internally | |
| 15:09:19 | mriedem | aspiers: thanks | |
| 15:09:31 | mriedem | sean-k-mooney: stephenfin: ok it would be good if we had this documented as a limitation somewhere if we don't already | |
| 15:09:49 | mriedem | i'm not sure if that's best in the numa topo admin docs, flavor extra spec, reserved host ram config option, other? | |
| 15:09:54 | stephenfin | I think we should, but I'll check. Will be easy tack onto this existing doc | |
| 15:10:12 | sean-k-mooney | ya we should document the requirement to set hw:mem_page_size small if you set hw:cpu_policy=dedicated and dont you hugepages | |
| 15:10:55 | mriedem | ok so maybe it's best to just document alongside that extra spec https://docs.openstack.org/nova/latest/user/flavors.html | |
| 15:11:09 | sean-k-mooney | the reason this happens is when you enable pinning with out settin hw:mem_page_size we only look at available cpus in the numa toplogy filter not allso aviable memory on the numa nodes | |
| 15:11:35 | sean-k-mooney | i fyou enable hw:mem_page_size=small|large it validate the avaiable memory too | |
| 15:12:03 | stephenfin | sean-k-mooney: isn't there some issue with how we track mempages though | |
| 15:12:23 | stephenfin | none of this is obviously in placement yet | |
| 15:12:34 | sean-k-mooney | you cant mix vms that use hw:mem_page_size with vms that dont on the same host or it invalidate the tracking | |
| 15:12:46 | sean-k-mooney | but you should not mix numa and non numa vms on the same host anyway | |
| 15:13:19 | mriedem | "but you should not mix numa and non numa vms on the same host anyway" - is that documented? | |
| 15:13:28 | mriedem | we have a shit load of tribal knowledge about this stuff | |
| 15:13:34 | mriedem | and by we i mean you | |
| 15:14:02 | mriedem | all i see on https://docs.openstack.org/nova/latest/admin/cpu-topologies.html is "Host aggregates should be used to separate pinned instances from unpinned instances as the latter will not respect the resourcing requirements of the former." | |
| 15:14:21 | sean-k-mooney | i dont think its in the upstream docs or redhats it was in the intel tuning guide i helped writhe in like 2014 | |
| 15:14:35 | sean-k-mooney | so yes i shoudl document the tribal knowlage around this | |
| 15:14:49 | sean-k-mooney | *my tribale knowlasge around this | |
| 15:15:28 | sean-k-mooney | the extention of that warning to numa instead of pinned is to because of the OOM behavior | |
| 15:16:02 | sean-k-mooney | i can try and write something for this on monday if you / stephen can correct the spelling after | |
| 15:16:17 | kashyap | sean-k-mooney: Happy to look as well :-) | |
| 15:17:01 | mriedem | ah here is the duplicate https://bugs.launchpad.net/nova/+bug/1792985 | |
| 15:17:03 | openstack | Launchpad bug 1439247 in OpenStack Compute (nova) "duplicate for #1792985 Small pages memory are not take into account when not explicitly requested" [Medium,Confirmed] | |
| 15:17:08 | mriedem | which is a duplicate of another bug | |
| 15:17:12 | aspiers | mriedem: forwarded internally. If you see it happen again feel free to ping me privately | |
| 15:17:57 | mriedem | ack | |
| 15:19:34 | sean-k-mooney | mriedem: do you want to assing one of those bugs to me and i can update the documentaiton regarding the mixing of numa and non numa instances and the use of hw:mem_page_size=small|large to avoid OOM events for numa affined instances | |
| 15:19:57 | mriedem | artom: this isn't fixed with your numa live migration series right? https://bugs.launchpad.net/nova/+bug/1496135 - that's the thing you were half fixing but we had you remove it? | |
| 15:19:58 | openstack | Launchpad bug 1496135 in OpenStack Compute (nova) "live-migration will not honor destination vcpu_pin_set config" [Medium,Confirmed] | |
| 15:20:19 | mriedem | sean-k-mooney: i can only do so much assing in one day | |
| 15:20:54 | sean-k-mooney | see alrealy indicating my spelling mistakes :) | |
| 15:21:08 | mriedem | sean-k-mooney: i don't think we probably need to assign it to you, a docs patch would just be a related bug anyway | |
| 15:21:13 | sean-k-mooney | am i thnk that should be fixed by the numa live migration changes | |
| 15:21:18 | artom | mriedem, not clear - they talk about policy, which I assume means hw:cpu_policy, which my thing *did* fix | |
| 15:21:19 | mriedem | you can track a todo however you want though | |
| 15:21:44 | sean-k-mooney | oh | |
| 15:21:51 | sean-k-mooney | this is the edgecase we removed | |
| 15:21:58 | artom | But... then they also talk about vcpuset | |
| 15:22:03 | sean-k-mooney | yes | |
| 15:22:03 | artom | So, it's confusing | |
| 15:22:17 | sean-k-mooney | this is migration of a non numa instance between host with vcpu_pin_set defined | |
| 15:22:27 | sean-k-mooney | so the thing we added then removed | |
| 15:22:35 | sean-k-mooney | so this is not fixed yet | |
| 15:22:45 | artom | I guess? In any case, that's what it can become, because we need something to track that | |
| 15:23:28 | mriedem | ok i left a comment https://bugs.launchpad.net/nova/+bug/1496135/comments/10 | |
| 15:23:29 | openstack | Launchpad bug 1496135 in OpenStack Compute (nova) "live-migration will not honor destination vcpu_pin_set config" [Medium,Confirmed] | |
| 15:24:19 | mriedem | artom: while you're here, are we going to try and get https://review.opendev.org/#/c/672595/ into train yet? | |
| 15:24:33 | mriedem | i'm not sure how much fun that would be to backport | |
| 15:24:35 | artom | mriedem, I'm always here man | |
| 15:24:38 | mriedem | though it's all test stuff | |
| 15:24:44 | mriedem | you weren't yesterday when i needed to bug you | |
| 15:24:57 | artom | PTO, apple picking with son's daycare | |
| 15:25:07 | mriedem | see, you're not *always* here | |
| 15:25:09 | artom | Bought some cider, good stuff | |
| 15:25:14 | artom | In my hear I am | |
| 15:25:18 | artom | *heart, even | |
| 15:25:19 | mriedem | heart | |
| 15:25:28 | artom | Anyways | |
| 15:25:28 | mriedem | meanwhile i'm just assing around | |
| 15:25:41 | artom | Yeah, the func test should land in Train | |
| 15:25:58 | artom | If you and dansmith are ready to dive back in, I can pick it up again | |
| 15:26:14 | artom | Although it conflicts with some of the PCPU stuff that's still in flight | |
| 15:26:43 | mriedem | i've added it to https://etherpad.openstack.org/p/nova-train-release-todo | |
| 15:26:48 | artom | https://review.opendev.org/#/c/681060/ specifically | |
| 15:26:49 | mriedem | i'm not going to review that today | |
| 15:27:00 | artom | ^^ needs to land first, anyways, then I rebase and continue | |
| 15:27:14 | mriedem | so likely rc2 at this point | |
| 15:28:19 | artom | I guess yeah | |
| 15:29:57 | mriedem | what is the end of the pcpu series? the reshaper patch? because i was thinking about rebasing https://review.opendev.org/#/c/683011/ on top of that so we can approve it to make sure it gets into rc1 | |