| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-09-20 | |||
| 14:12:10 | gibi | mriedem: thanks, elod ^^ | |
| 14:12:27 | mriedem | gibi: btw, where do you get your packages? some upstream distro or do you build your own? | |
| 14:12:35 | mriedem | b/c for pike you might have to patch your osc package | |
| 14:13:00 | gibi | mriedem: packages come from Mirantis | |
| 14:13:43 | gibi | mriedem: so if I fail to do a clean thing upstream I will dump the "downstream" work there | |
| 14:15:52 | dtroyer | gibi, mriedem: that is enough of a bug it would be back-portable. OSC follows the same rules of working backwards through releases on backports. Of course, standard disclaimer of just using a modern OSC goes here, I understand that packagers don't do that, I wish they would/could… | |
| 14:16:12 | gibi | dtroyer: thanks | |
| 14:17:48 | mriedem | the biggest barrier to backporting osc fixes is if they rely on something in novaclient that's not in whatever stable branch you're targeting | |
| 14:17:55 | mriedem | it looks like in this case you might get lucky | |
| 14:36:57 | hemna | hey guys, any idea why I might be getting theses failures against stable/pike: http://paste.openstack.org/show/778056/ | |
| 14:37:26 | hemna | tox -epy27 against stable/pike gives about 6300 of those errors | |
| 14:42:59 | mriedem | oh hemna | |
| 14:43:11 | mriedem | hemna: KeithMnemonic1 must have some dirt on you | |
| 14:43:40 | KeithMnemonic1 | lol | |
| 14:45:03 | mriedem | hemna: so you're trying to figure out why these are failing https://storage.bhs1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_6e4/683008/2/check/openstack-tox-py27/6e40a0c/testr_results.html.gz right? i'm not sure why you'd be hitting that weird db issue locally - should be using sqlite | |
| 14:45:33 | mriedem | hemna: i'd try cleaning pycs and rebuilding the venv: | |
| 14:45:42 | mriedem | find . -name *.pyc delete | |
| 14:45:45 | mriedem | tox -r -e py27 | |
| 14:46:35 | bauzas | mriedem: just to make it clear, you want me providing the links to https://releases.openstack.org/train/highlights.html#nova-compute-service in https://review.opendev.org/#/c/683327/1/releasenotes/notes/train-prelude-3db0f5f6a75cc57a.yaml ? | |
| 14:47:09 | mriedem | bauzas: in the commit message? | |
| 14:47:10 | mriedem | i would | |
| 14:47:27 | mriedem | wait no not the latter | |
| 14:47:30 | bauzas | mriedem: looks like you also want to have links in the release note | |
| 14:47:32 | mriedem | the etherpad and the published highlights doc | |
| 14:47:35 | mriedem | bauzas: no | |
| 14:47:49 | mriedem | i'm telling you, pull your content from the highlights, | |
| 14:48:02 | bauzas | oh, k | |
| 14:48:03 | mriedem | plus additional stuff not in the highlights that's in the etherpad, like nova-cells/consoleauth removal and xenapi driver deprecatoin | |
| 14:48:08 | bauzas | ok, understood | |
| 14:48:10 | mriedem | we intentionally didn't include ^ in highlights | |
| 14:48:14 | mriedem | b/c they aren't highlights | |
| 14:48:16 | bauzas | cool | |
| 14:48:21 | bauzas | will do it later | |
| 14:48:24 | mriedem | xenapi driver deprecatoin is a brown light | |
| 14:50:03 | hemna | ok I'll try that | |
| 14:50:21 | hemna | I'm not sure I'll be successfull with this conflict resolution on that patch | |
| 14:50:28 | hemna | since so much changed between pike and queens | |
| 14:51:26 | mriedem | i didn't do the tests for a reason | |
| 14:52:00 | mriedem | i could, but i'd need some incentive$$$ | |
| 14:52:56 | openstackgerrit | Walter A. Boring IV (hemna) proposed openstack/nova stable/pike: WIP: Avoid redundant initialize_connection on source post live migration https://review.opendev.org/683008 | |
| 14:53:02 | hemna | fixed the pep8 issues at least..... | |
| 14:53:04 | hemna | heh | |
| 14:54:20 | KeithMnemonic1 | mriedem will you be in Shanghai. I can incent you there | |
| 14:59:26 | mriedem | i don't think so | |
| 15:02:19 | gibi | whaaat? | |
| 15:02:30 | gibi | who will be in Shanghai then? | |
| 15:04:26 | mnaser | its friday and i dont want to deal with this >:( | |
| 15:04:37 | mnaser | nova_api.instance_mappings shows a mapping for an instance | |
| 15:04:39 | bauzas | gibi: \o | |
| 15:04:49 | gibi | bauzas: :) | |
| 15:04:53 | mnaser | nova.instances shows the instance there with deleted=0, vm_state=error though | |
| 15:05:04 | bauzas | at least we can ask a room for two :p | |
| 15:05:14 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Func test for migrate reschedule with pinned compute rpc https://review.opendev.org/683385 | |
| 15:05:14 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Handle legacy request spec dict in ComputeTaskManager._cold_migrate https://review.opendev.org/680762 | |
| 15:05:15 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Isolate request spec handling from _cold_migrate https://review.opendev.org/680763 | |
| 15:05:18 | mriedem | stephenfin: you might have thoughts on this https://bugs.launchpad.net/nova/+bug/1844721 | |
| 15:05:19 | openstack | Launchpad bug 1844721 in OpenStack Compute (nova) "Need NUMA aware RAM reservation to avoid OOM killing host processes" [Undecided,New] | |
| 15:05:31 | mnaser | api gives 404 when looking up the instance.. | |
| 15:05:39 | mnaser | but its there when doing a list | |
| 15:06:20 | stephenfin | mriedem: Yup, that's a long standing issue. There's definitely at least one existing report of that. Possibly many | |
| 15:06:48 | stephenfin | Unfortunately the impact of a fix will likely be significant, which is why we've punted it continuously /o\ | |
| 15:07:22 | mriedem | do you know if it's documented as a known issue? | |
| 15:07:22 | sean-k-mooney | they can get numa aware ram reservation by setting hw:mem_page_size=small | |
| 15:07:35 | sean-k-mooney | if you are using cpu pinning but not hugepages you should set that | |
| 15:07:48 | mriedem | yeah the bug says "Many mitigation are "invented", but those mitigation all have some form of technical or operational "difficulties". One mitigation, for example, is to enable huge pages, and put VMs on huge pages." | |
| 15:08:03 | aspiers | mriedem: I'm sure it was honest ignorance of the policy rather than deliberately avoiding work. Your -1 should be sufficient to get it fixed. | |
| 15:08:30 | sean-k-mooney | mriedem: hugepages will prevent it but hw:mem_page_size=small will use 4k pages and still fix the issue | |
| 15:08:36 | mriedem | aspiers: i didn't say anyone was deliberately avoiding doing the work, | |
| 15:08:44 | stephenfin | mriedem: yeah, what sean-k-mooney said | |
| 15:08:44 | mriedem | but i've also seen it across several projects from multiple suse developers | |
| 15:08:47 | mriedem | so it's a pattern | |
| 15:08:58 | mriedem | hence my post to the ML | |
| 15:09:10 | aspiers | mriedem: OK, I'll forward it internally | |
| 15:09:19 | mriedem | aspiers: thanks | |
| 15:09:31 | mriedem | sean-k-mooney: stephenfin: ok it would be good if we had this documented as a limitation somewhere if we don't already | |
| 15:09:49 | mriedem | i'm not sure if that's best in the numa topo admin docs, flavor extra spec, reserved host ram config option, other? | |
| 15:09:54 | stephenfin | I think we should, but I'll check. Will be easy tack onto this existing doc | |
| 15:10:12 | sean-k-mooney | ya we should document the requirement to set hw:mem_page_size small if you set hw:cpu_policy=dedicated and dont you hugepages | |
| 15:10:55 | mriedem | ok so maybe it's best to just document alongside that extra spec https://docs.openstack.org/nova/latest/user/flavors.html | |
| 15:11:09 | sean-k-mooney | the reason this happens is when you enable pinning with out settin hw:mem_page_size we only look at available cpus in the numa toplogy filter not allso aviable memory on the numa nodes | |
| 15:11:35 | sean-k-mooney | i fyou enable hw:mem_page_size=small|large it validate the avaiable memory too | |
| 15:12:03 | stephenfin | sean-k-mooney: isn't there some issue with how we track mempages though | |
| 15:12:23 | stephenfin | none of this is obviously in placement yet | |
| 15:12:34 | sean-k-mooney | you cant mix vms that use hw:mem_page_size with vms that dont on the same host or it invalidate the tracking | |
| 15:12:46 | sean-k-mooney | but you should not mix numa and non numa vms on the same host anyway | |
| 15:13:19 | mriedem | "but you should not mix numa and non numa vms on the same host anyway" - is that documented? | |
| 15:13:28 | mriedem | we have a shit load of tribal knowledge about this stuff | |
| 15:13:34 | mriedem | and by we i mean you | |
| 15:14:02 | mriedem | all i see on https://docs.openstack.org/nova/latest/admin/cpu-topologies.html is "Host aggregates should be used to separate pinned instances from unpinned instances as the latter will not respect the resourcing requirements of the former." | |
| 15:14:21 | sean-k-mooney | i dont think its in the upstream docs or redhats it was in the intel tuning guide i helped writhe in like 2014 | |
| 15:14:35 | sean-k-mooney | so yes i shoudl document the tribal knowlage around this | |
| 15:14:49 | sean-k-mooney | *my tribale knowlasge around this | |
| 15:15:28 | sean-k-mooney | the extention of that warning to numa instead of pinned is to because of the OOM behavior | |
| 15:16:02 | sean-k-mooney | i can try and write something for this on monday if you / stephen can correct the spelling after | |
| 15:16:17 | kashyap | sean-k-mooney: Happy to look as well :-) | |
| 15:17:01 | mriedem | ah here is the duplicate https://bugs.launchpad.net/nova/+bug/1792985 | |
| 15:17:03 | openstack | Launchpad bug 1439247 in OpenStack Compute (nova) "duplicate for #1792985 Small pages memory are not take into account when not explicitly requested" [Medium,Confirmed] | |
| 15:17:08 | mriedem | which is a duplicate of another bug | |
| 15:17:12 | aspiers | mriedem: forwarded internally. If you see it happen again feel free to ping me privately | |
| 15:17:57 | mriedem | ack | |