| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-09-20 | |||
| 13:56:09 | mriedem | in tests i mean | |
| 13:56:11 | gibi | mriedem: in the meantime I can tell you that your fix works according to the functional test | |
| 13:56:19 | mriedem | \o/ | |
| 13:56:45 | gibi | ... what a nice Friday it is | |
| 14:02:08 | mriedem | my kid is also staying home from school today claiming to be sick and i can't tell if she's playing opposum | |
| 14:02:17 | mriedem | i need to put her to work | |
| 14:05:20 | artom | How good is she with Python? Weren't we looking for interns for all the osc gaps? | |
| 14:05:28 | gibi | mriedem: Is there a change that we consider this https://review.opendev.org/#/c/672577/ as a bug and backport it to stable/pike? (yeah I have a bug day today) | |
| 14:05:58 | gibi | artom, mriedem: hm osc gaps, and osc bugfix backport ^^ | |
| 14:07:48 | mriedem | aspiers: you might want to talk to some suse developers downstream about their upstream "fix only on the branch i care about" practices https://review.opendev.org/#/c/676882/ | |
| 14:08:31 | mriedem | gibi: s/change/chance/ ? | |
| 14:08:37 | gibi | mriedem: yes, sorry | |
| 14:08:43 | mriedem | going to pike is probably going to be tough... | |
| 14:09:02 | mriedem | but, that fix should be compatible... | |
| 14:09:12 | gibi | mriedem: OK. I will look into that as well | |
| 14:09:14 | mriedem | it's clearly a bug if you're using >= 2.53 | |
| 14:09:25 | gibi | mriedem: or find somebody who has time | |
| 14:09:40 | mriedem | pike is in extended maintenance upstream so i'm not sure how much dtroyer is going to care about backporting it that far | |
| 14:09:46 | gibi | mriedem: yes, we have pike deployemnts out there using 2.53 and seeing openstack client fail | |
| 14:10:22 | mriedem | gibi: are they pinning the version to 2.latest or something? | |
| 14:10:50 | mriedem | or they are just pinning to 2.53 since that's the max in pike? | |
| 14:10:53 | gibi | mriedem: using 2.53 by default, but sure we can go back to olderin for this command as a workaround | |
| 14:11:06 | gibi | mriedem: 2.53 is used as that is max pike | |
| 14:11:42 | gibi | it seems I was able to hook in elod to do this client backports... | |
| 14:11:52 | mriedem | i started stein for you https://review.opendev.org/#/c/683394/1 | |
| 14:12:10 | gibi | mriedem: thanks, elod ^^ | |
| 14:12:27 | mriedem | gibi: btw, where do you get your packages? some upstream distro or do you build your own? | |
| 14:12:35 | mriedem | b/c for pike you might have to patch your osc package | |
| 14:13:00 | gibi | mriedem: packages come from Mirantis | |
| 14:13:43 | gibi | mriedem: so if I fail to do a clean thing upstream I will dump the "downstream" work there | |
| 14:15:52 | dtroyer | gibi, mriedem: that is enough of a bug it would be back-portable. OSC follows the same rules of working backwards through releases on backports. Of course, standard disclaimer of just using a modern OSC goes here, I understand that packagers don't do that, I wish they would/could… | |
| 14:16:12 | gibi | dtroyer: thanks | |
| 14:17:48 | mriedem | the biggest barrier to backporting osc fixes is if they rely on something in novaclient that's not in whatever stable branch you're targeting | |
| 14:17:55 | mriedem | it looks like in this case you might get lucky | |
| 14:36:57 | hemna | hey guys, any idea why I might be getting theses failures against stable/pike: http://paste.openstack.org/show/778056/ | |
| 14:37:26 | hemna | tox -epy27 against stable/pike gives about 6300 of those errors | |
| 14:42:59 | mriedem | oh hemna | |
| 14:43:11 | mriedem | hemna: KeithMnemonic1 must have some dirt on you | |
| 14:43:40 | KeithMnemonic1 | lol | |
| 14:45:03 | mriedem | hemna: so you're trying to figure out why these are failing https://storage.bhs1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_6e4/683008/2/check/openstack-tox-py27/6e40a0c/testr_results.html.gz right? i'm not sure why you'd be hitting that weird db issue locally - should be using sqlite | |
| 14:45:33 | mriedem | hemna: i'd try cleaning pycs and rebuilding the venv: | |
| 14:45:42 | mriedem | find . -name *.pyc delete | |
| 14:45:45 | mriedem | tox -r -e py27 | |
| 14:46:35 | bauzas | mriedem: just to make it clear, you want me providing the links to https://releases.openstack.org/train/highlights.html#nova-compute-service in https://review.opendev.org/#/c/683327/1/releasenotes/notes/train-prelude-3db0f5f6a75cc57a.yaml ? | |
| 14:47:09 | mriedem | bauzas: in the commit message? | |
| 14:47:10 | mriedem | i would | |
| 14:47:27 | mriedem | wait no not the latter | |
| 14:47:30 | bauzas | mriedem: looks like you also want to have links in the release note | |
| 14:47:32 | mriedem | the etherpad and the published highlights doc | |
| 14:47:35 | mriedem | bauzas: no | |
| 14:47:49 | mriedem | i'm telling you, pull your content from the highlights, | |
| 14:48:02 | bauzas | oh, k | |
| 14:48:03 | mriedem | plus additional stuff not in the highlights that's in the etherpad, like nova-cells/consoleauth removal and xenapi driver deprecatoin | |
| 14:48:08 | bauzas | ok, understood | |
| 14:48:10 | mriedem | we intentionally didn't include ^ in highlights | |
| 14:48:14 | mriedem | b/c they aren't highlights | |
| 14:48:16 | bauzas | cool | |
| 14:48:21 | bauzas | will do it later | |
| 14:48:24 | mriedem | xenapi driver deprecatoin is a brown light | |
| 14:50:03 | hemna | ok I'll try that | |
| 14:50:21 | hemna | I'm not sure I'll be successfull with this conflict resolution on that patch | |
| 14:50:28 | hemna | since so much changed between pike and queens | |
| 14:51:26 | mriedem | i didn't do the tests for a reason | |
| 14:52:00 | mriedem | i could, but i'd need some incentive$$$ | |
| 14:52:56 | openstackgerrit | Walter A. Boring IV (hemna) proposed openstack/nova stable/pike: WIP: Avoid redundant initialize_connection on source post live migration https://review.opendev.org/683008 | |
| 14:53:02 | hemna | fixed the pep8 issues at least..... | |
| 14:53:04 | hemna | heh | |
| 14:54:20 | KeithMnemonic1 | mriedem will you be in Shanghai. I can incent you there | |
| 14:59:26 | mriedem | i don't think so | |
| 15:02:19 | gibi | whaaat? | |
| 15:02:30 | gibi | who will be in Shanghai then? | |
| 15:04:26 | mnaser | its friday and i dont want to deal with this >:( | |
| 15:04:37 | mnaser | nova_api.instance_mappings shows a mapping for an instance | |
| 15:04:39 | bauzas | gibi: \o | |
| 15:04:49 | gibi | bauzas: :) | |
| 15:04:53 | mnaser | nova.instances shows the instance there with deleted=0, vm_state=error though | |
| 15:05:04 | bauzas | at least we can ask a room for two :p | |
| 15:05:14 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Handle legacy request spec dict in ComputeTaskManager._cold_migrate https://review.opendev.org/680762 | |
| 15:05:14 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Func test for migrate reschedule with pinned compute rpc https://review.opendev.org/683385 | |
| 15:05:15 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Isolate request spec handling from _cold_migrate https://review.opendev.org/680763 | |
| 15:05:18 | mriedem | stephenfin: you might have thoughts on this https://bugs.launchpad.net/nova/+bug/1844721 | |
| 15:05:19 | openstack | Launchpad bug 1844721 in OpenStack Compute (nova) "Need NUMA aware RAM reservation to avoid OOM killing host processes" [Undecided,New] | |
| 15:05:31 | mnaser | api gives 404 when looking up the instance.. | |
| 15:05:39 | mnaser | but its there when doing a list | |
| 15:06:20 | stephenfin | mriedem: Yup, that's a long standing issue. There's definitely at least one existing report of that. Possibly many | |
| 15:06:48 | stephenfin | Unfortunately the impact of a fix will likely be significant, which is why we've punted it continuously /o\ | |
| 15:07:22 | sean-k-mooney | they can get numa aware ram reservation by setting hw:mem_page_size=small | |
| 15:07:22 | mriedem | do you know if it's documented as a known issue? | |
| 15:07:35 | sean-k-mooney | if you are using cpu pinning but not hugepages you should set that | |
| 15:07:48 | mriedem | yeah the bug says "Many mitigation are "invented", but those mitigation all have some form of technical or operational "difficulties". One mitigation, for example, is to enable huge pages, and put VMs on huge pages." | |
| 15:08:03 | aspiers | mriedem: I'm sure it was honest ignorance of the policy rather than deliberately avoiding work. Your -1 should be sufficient to get it fixed. | |
| 15:08:30 | sean-k-mooney | mriedem: hugepages will prevent it but hw:mem_page_size=small will use 4k pages and still fix the issue | |
| 15:08:36 | mriedem | aspiers: i didn't say anyone was deliberately avoiding doing the work, | |
| 15:08:44 | mriedem | but i've also seen it across several projects from multiple suse developers | |
| 15:08:44 | stephenfin | mriedem: yeah, what sean-k-mooney said | |
| 15:08:47 | mriedem | so it's a pattern | |
| 15:08:58 | mriedem | hence my post to the ML | |
| 15:09:10 | aspiers | mriedem: OK, I'll forward it internally | |
| 15:09:19 | mriedem | aspiers: thanks | |
| 15:09:31 | mriedem | sean-k-mooney: stephenfin: ok it would be good if we had this documented as a limitation somewhere if we don't already | |