| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-09-20 | |||
| 14:46:35 | bauzas | mriedem: just to make it clear, you want me providing the links to https://releases.openstack.org/train/highlights.html#nova-compute-service in https://review.opendev.org/#/c/683327/1/releasenotes/notes/train-prelude-3db0f5f6a75cc57a.yaml ? | |
| 14:47:09 | mriedem | bauzas: in the commit message? | |
| 14:47:10 | mriedem | i would | |
| 14:47:27 | mriedem | wait no not the latter | |
| 14:47:30 | bauzas | mriedem: looks like you also want to have links in the release note | |
| 14:47:32 | mriedem | the etherpad and the published highlights doc | |
| 14:47:35 | mriedem | bauzas: no | |
| 14:47:49 | mriedem | i'm telling you, pull your content from the highlights, | |
| 14:48:02 | bauzas | oh, k | |
| 14:48:03 | mriedem | plus additional stuff not in the highlights that's in the etherpad, like nova-cells/consoleauth removal and xenapi driver deprecatoin | |
| 14:48:08 | bauzas | ok, understood | |
| 14:48:10 | mriedem | we intentionally didn't include ^ in highlights | |
| 14:48:14 | mriedem | b/c they aren't highlights | |
| 14:48:16 | bauzas | cool | |
| 14:48:21 | bauzas | will do it later | |
| 14:48:24 | mriedem | xenapi driver deprecatoin is a brown light | |
| 14:50:03 | hemna | ok I'll try that | |
| 14:50:21 | hemna | I'm not sure I'll be successfull with this conflict resolution on that patch | |
| 14:50:28 | hemna | since so much changed between pike and queens | |
| 14:51:26 | mriedem | i didn't do the tests for a reason | |
| 14:52:00 | mriedem | i could, but i'd need some incentive$$$ | |
| 14:52:56 | openstackgerrit | Walter A. Boring IV (hemna) proposed openstack/nova stable/pike: WIP: Avoid redundant initialize_connection on source post live migration https://review.opendev.org/683008 | |
| 14:53:02 | hemna | fixed the pep8 issues at least..... | |
| 14:53:04 | hemna | heh | |
| 14:54:20 | KeithMnemonic1 | mriedem will you be in Shanghai. I can incent you there | |
| 14:59:26 | mriedem | i don't think so | |
| 15:02:19 | gibi | whaaat? | |
| 15:02:30 | gibi | who will be in Shanghai then? | |
| 15:04:26 | mnaser | its friday and i dont want to deal with this >:( | |
| 15:04:37 | mnaser | nova_api.instance_mappings shows a mapping for an instance | |
| 15:04:39 | bauzas | gibi: \o | |
| 15:04:49 | gibi | bauzas: :) | |
| 15:04:53 | mnaser | nova.instances shows the instance there with deleted=0, vm_state=error though | |
| 15:05:04 | bauzas | at least we can ask a room for two :p | |
| 15:05:14 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Func test for migrate reschedule with pinned compute rpc https://review.opendev.org/683385 | |
| 15:05:14 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Handle legacy request spec dict in ComputeTaskManager._cold_migrate https://review.opendev.org/680762 | |
| 15:05:15 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Isolate request spec handling from _cold_migrate https://review.opendev.org/680763 | |
| 15:05:18 | mriedem | stephenfin: you might have thoughts on this https://bugs.launchpad.net/nova/+bug/1844721 | |
| 15:05:19 | openstack | Launchpad bug 1844721 in OpenStack Compute (nova) "Need NUMA aware RAM reservation to avoid OOM killing host processes" [Undecided,New] | |
| 15:05:31 | mnaser | api gives 404 when looking up the instance.. | |
| 15:05:39 | mnaser | but its there when doing a list | |
| 15:06:20 | stephenfin | mriedem: Yup, that's a long standing issue. There's definitely at least one existing report of that. Possibly many | |
| 15:06:48 | stephenfin | Unfortunately the impact of a fix will likely be significant, which is why we've punted it continuously /o\ | |
| 15:07:22 | mriedem | do you know if it's documented as a known issue? | |
| 15:07:22 | sean-k-mooney | they can get numa aware ram reservation by setting hw:mem_page_size=small | |
| 15:07:35 | sean-k-mooney | if you are using cpu pinning but not hugepages you should set that | |
| 15:07:48 | mriedem | yeah the bug says "Many mitigation are "invented", but those mitigation all have some form of technical or operational "difficulties". One mitigation, for example, is to enable huge pages, and put VMs on huge pages." | |
| 15:08:03 | aspiers | mriedem: I'm sure it was honest ignorance of the policy rather than deliberately avoiding work. Your -1 should be sufficient to get it fixed. | |
| 15:08:30 | sean-k-mooney | mriedem: hugepages will prevent it but hw:mem_page_size=small will use 4k pages and still fix the issue | |
| 15:08:36 | mriedem | aspiers: i didn't say anyone was deliberately avoiding doing the work, | |
| 15:08:44 | stephenfin | mriedem: yeah, what sean-k-mooney said | |
| 15:08:44 | mriedem | but i've also seen it across several projects from multiple suse developers | |
| 15:08:47 | mriedem | so it's a pattern | |
| 15:08:58 | mriedem | hence my post to the ML | |
| 15:09:10 | aspiers | mriedem: OK, I'll forward it internally | |
| 15:09:19 | mriedem | aspiers: thanks | |
| 15:09:31 | mriedem | sean-k-mooney: stephenfin: ok it would be good if we had this documented as a limitation somewhere if we don't already | |
| 15:09:49 | mriedem | i'm not sure if that's best in the numa topo admin docs, flavor extra spec, reserved host ram config option, other? | |
| 15:09:54 | stephenfin | I think we should, but I'll check. Will be easy tack onto this existing doc | |
| 15:10:12 | sean-k-mooney | ya we should document the requirement to set hw:mem_page_size small if you set hw:cpu_policy=dedicated and dont you hugepages | |
| 15:10:55 | mriedem | ok so maybe it's best to just document alongside that extra spec https://docs.openstack.org/nova/latest/user/flavors.html | |
| 15:11:09 | sean-k-mooney | the reason this happens is when you enable pinning with out settin hw:mem_page_size we only look at available cpus in the numa toplogy filter not allso aviable memory on the numa nodes | |
| 15:11:35 | sean-k-mooney | i fyou enable hw:mem_page_size=small|large it validate the avaiable memory too | |
| 15:12:03 | stephenfin | sean-k-mooney: isn't there some issue with how we track mempages though | |
| 15:12:23 | stephenfin | none of this is obviously in placement yet | |
| 15:12:34 | sean-k-mooney | you cant mix vms that use hw:mem_page_size with vms that dont on the same host or it invalidate the tracking | |
| 15:12:46 | sean-k-mooney | but you should not mix numa and non numa vms on the same host anyway | |
| 15:13:19 | mriedem | "but you should not mix numa and non numa vms on the same host anyway" - is that documented? | |
| 15:13:28 | mriedem | we have a shit load of tribal knowledge about this stuff | |
| 15:13:34 | mriedem | and by we i mean you | |
| 15:14:02 | mriedem | all i see on https://docs.openstack.org/nova/latest/admin/cpu-topologies.html is "Host aggregates should be used to separate pinned instances from unpinned instances as the latter will not respect the resourcing requirements of the former." | |
| 15:14:21 | sean-k-mooney | i dont think its in the upstream docs or redhats it was in the intel tuning guide i helped writhe in like 2014 | |
| 15:14:35 | sean-k-mooney | so yes i shoudl document the tribal knowlage around this | |
| 15:14:49 | sean-k-mooney | *my tribale knowlasge around this | |
| 15:15:28 | sean-k-mooney | the extention of that warning to numa instead of pinned is to because of the OOM behavior | |
| 15:16:02 | sean-k-mooney | i can try and write something for this on monday if you / stephen can correct the spelling after | |
| 15:16:17 | kashyap | sean-k-mooney: Happy to look as well :-) | |
| 15:17:01 | mriedem | ah here is the duplicate https://bugs.launchpad.net/nova/+bug/1792985 | |
| 15:17:03 | openstack | Launchpad bug 1439247 in OpenStack Compute (nova) "duplicate for #1792985 Small pages memory are not take into account when not explicitly requested" [Medium,Confirmed] | |
| 15:17:08 | mriedem | which is a duplicate of another bug | |
| 15:17:12 | aspiers | mriedem: forwarded internally. If you see it happen again feel free to ping me privately | |
| 15:17:57 | mriedem | ack | |
| 15:19:34 | sean-k-mooney | mriedem: do you want to assing one of those bugs to me and i can update the documentaiton regarding the mixing of numa and non numa instances and the use of hw:mem_page_size=small|large to avoid OOM events for numa affined instances | |
| 15:19:57 | mriedem | artom: this isn't fixed with your numa live migration series right? https://bugs.launchpad.net/nova/+bug/1496135 - that's the thing you were half fixing but we had you remove it? | |
| 15:19:58 | openstack | Launchpad bug 1496135 in OpenStack Compute (nova) "live-migration will not honor destination vcpu_pin_set config" [Medium,Confirmed] | |
| 15:20:19 | mriedem | sean-k-mooney: i can only do so much assing in one day | |
| 15:20:54 | sean-k-mooney | see alrealy indicating my spelling mistakes :) | |
| 15:21:08 | mriedem | sean-k-mooney: i don't think we probably need to assign it to you, a docs patch would just be a related bug anyway | |
| 15:21:13 | sean-k-mooney | am i thnk that should be fixed by the numa live migration changes | |
| 15:21:18 | artom | mriedem, not clear - they talk about policy, which I assume means hw:cpu_policy, which my thing *did* fix | |
| 15:21:19 | mriedem | you can track a todo however you want though | |
| 15:21:44 | sean-k-mooney | oh | |
| 15:21:51 | sean-k-mooney | this is the edgecase we removed | |
| 15:21:58 | artom | But... then they also talk about vcpuset | |
| 15:22:03 | sean-k-mooney | yes | |
| 15:22:03 | artom | So, it's confusing | |
| 15:22:17 | sean-k-mooney | this is migration of a non numa instance between host with vcpu_pin_set defined | |
| 15:22:27 | sean-k-mooney | so the thing we added then removed | |
| 15:22:35 | sean-k-mooney | so this is not fixed yet | |
| 15:22:45 | artom | I guess? In any case, that's what it can become, because we need something to track that | |