| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-09-06 | |||
| 13:16:11 | artom | mriedem, https://review.opendev.org/#/c/635229/58/nova/virt/libvirt/driver.py@7345 for when you're ready to look | |
| 13:16:31 | mriedem | yonglihe: also, as i said on that novaclient change, i'd like to see what it actually looks like in the output | |
| 13:16:48 | mriedem | if it's just a bunch of json blobs dumped to the console that's not very useful | |
| 13:18:03 | mriedem | artom: is that because i was asking about guest_cpu_tune and guest_numa_tune being exclusive? | |
| 13:18:14 | openstackgerrit | Dmitry Tantsur proposed openstack/nova master: Fix incorrect invocation of openstacksdk's baremetal.nodes() https://review.opendev.org/680684 | |
| 13:18:17 | dtantsur | efried, dustinc, mriedem, another issue ^^^ | |
| 13:18:30 | dtantsur | this one probably affects ironic CI as well | |
| 13:18:30 | artom | mriedem, yeah, that's what lead me down that path | |
| 13:18:35 | mriedem | \o/ | |
| 13:18:42 | artom | mriedem, indeed :) | |
| 13:18:43 | mriedem | mdbooth: i'll follow dansmith's lead on that one | |
| 13:20:13 | mriedem | dustinc: commented | |
| 13:23:00 | yonglihe | mriedem: do you mean put json to release notes? It is definitely json blob. | |
| 13:23:41 | yonglihe | https://review.opendev.org/#/c/621476/62/doc/api_samples/os-server-topology/v2.78/servers-topology-resp.json | |
| 13:23:57 | dtantsur | mriedem: I can try to reproduce my thoughts process in a bug, although it was a bit of guessing initially :) do you still use launchpad? | |
| 13:25:21 | yonglihe | mriedem: BTW, I'm afraid i don't have time to do it today, sorry. | |
| 13:26:21 | mdbooth | mriedem: Ack, thanks. | |
| 13:26:23 | mriedem | yonglihe: no i meant i'd like to see what the CLI output looks like | |
| 13:26:34 | mriedem | dtantsur: yeah | |
| 13:26:41 | dtantsur | so old school :) | |
| 13:26:45 | mriedem | best school | |
| 13:27:00 | mriedem | dtantsur: btw how is that we're not seeing issues in ironic CI? | |
| 13:27:10 | mriedem | or is it just because the job is non-voting in nova? | |
| 13:27:14 | dtantsur | mriedem: it has only one node at all. so picking a random node works. | |
| 13:27:27 | dtantsur | our multinode grenade uses several nodes, so it got hard broken on this. | |
| 13:27:38 | dtantsur | I'm writing a bug with explanation. | |
| 13:27:45 | mriedem | ok well just a link to a failed instance of one of those jobs with logs would be good in a bug report | |
| 13:27:47 | mriedem | thanks | |
| 13:28:14 | yonglihe | mriedem: then i need to running it after this weekend, sorry. | |
| 13:30:34 | mriedem | ok, maybe sean-k-mooney or stephenfin or artom have a system with servers that have numa topology and they can pull this down and see what it looks like | |
| 13:30:34 | dtantsur | mriedem: https://bugs.launchpad.net/nova/+bug/1843050 | |
| 13:30:35 | openstack | Launchpad bug 1843050 in OpenStack Compute (nova) "Regression in the ironic driver: wait_for_active checks a random node" [Undecided,New] | |
| 13:31:08 | yonglihe | mriedem: that's so nice. | |
| 13:31:12 | stephenfin | mriedem: This is the patch to expose NUMA topology API change in novaclient? | |
| 13:31:35 | openstackgerrit | Dmitry Tantsur proposed openstack/nova master: Fix incorrect invocation of openstacksdk's baremetal.nodes() https://review.opendev.org/680684 | |
| 13:31:45 | mriedem | stephenfin: yeah | |
| 13:31:50 | artom | mriedem, they could probably pile on to sean-k-mooney's new nfv job | |
| 13:31:58 | mriedem | artom: it's novaclient | |
| 13:32:06 | mriedem | i just want someone to show me a paste | |
| 13:32:18 | stephenfin | Sure, I can do that, though to be clear it doesn't need a NUMA-based system | |
| 13:32:27 | stephenfin | well, a two node+ system | |
| 13:32:34 | artom | mriedem, ah, my machine's under my desk at the office, and not responding to ping. Since daughter started school I haven't had a chance to get over there. | |
| 13:34:09 | mriedem | stephenfin: i could probably also just hack it up with the fake data fixtures in novaclient using the api-ref sample from the docs and trigger a failure in shell tests to see the output | |
| 13:34:40 | mriedem | my goal is to get that change merged so we can also get the 2.79 novaclient change merged, which is much simpler, but is held up on the 2.78 change | |
| 13:34:50 | mriedem | i can ffffart around with that today | |
| 13:35:12 | stephenfin | Whatever suits. I'll take it as an opportunity to redeploy my DevStack node so I'll have something but it'll be a few hours? | |
| 13:35:40 | mriedem | stephenfin: if you want, but it's friday and you've probably got other plates spinning | |
| 13:36:15 | mriedem | dtantsur: do we have an ironic change that depends-on the nova fix to see the multinode grenade job pass with the fix? knowing that even if we do, it could be several hours before it even runs | |
| 13:36:32 | stephenfin | many plates. much spinning :) | |
| 13:36:37 | dtantsur | mriedem: https://review.opendev.org/#/c/680643/ (depends on both fixes) | |
| 13:37:03 | mriedem | oh i didn't know there were 2 | |
| 13:37:42 | mriedem | ok will get those merged today | |
| 13:37:51 | dtantsur | thx! | |
| 13:41:45 | mriedem | dtantsur: is this the job we care about? ironic-grenade-dsvm-multinode-multitenant | |
| 13:41:55 | dtantsur | lemme double-check | |
| 13:42:24 | mriedem | that's the only multinode grenade job i see in zuul on that change | |
| 13:42:28 | dtantsur | mriedem: oh, it seems to be a normal (non-grenade) multinode job: ironic-tempest-ipa-wholedisk-direct-tinyipa-multinode | |
| 13:43:01 | mriedem | ack | |
| 13:43:01 | dtantsur | although I did see something similar on one or two grenade runs, but they don't seem consistently broken | |
| 13:43:13 | mriedem | maybe randomly picking the correct node | |
| 13:43:18 | dtantsur | yep | |
| 13:43:24 | dansmith | mriedem: mdbooth decided to talk about some details about that patch downstream, so let me fill you in | |
| 13:44:04 | dansmith | mriedem: there are (suddenly?) two customer cases he thinks are caused by this issue, yet we apparently have no confirmation from either see a problem solved by applying this | |
| 13:44:20 | dansmith | mriedem: one clearly isn't fixed by it, and we don't have logs to show why | |
| 13:45:02 | alex_xu | stephenfin: thanks, actually I already go through that patch, I didn't found problem, I'm just waiting for huaqiang to do some manual test, then I will go through all the left patches and +2 | |
| 13:45:03 | dansmith | mriedem: so while the rationale for making the change seems fairly legit to me, I don't know if (a) that one place was never wrapped for some reason or (b) whether changing that is going to cause other thread interaction or timing-related problems | |
| 13:46:44 | dansmith | mriedem: so I'm not sure what to say... we could just merge it and watch carefully and hope that any negative impact is not in two years when people actually roll to it, as it seems likely to be just an oversight | |
| 13:46:59 | yonglihe | mriedem: sorry this is a barrier, i did not bring back my laptop today, and stephenfin, thanks. | |
| 13:47:01 | dansmith | but it definitely makes me nervous not having any "yes this fixes my problem" confirmation for code that has been around for ten years | |
| 13:47:17 | mriedem | dansmith: an oversight that we don't know if it has caused problems until now, though that code hasn't changed in a long time | |
| 13:47:26 | stephenfin | alex_xu: Huzzah. Good to hear (y) | |
| 13:47:42 | mriedem | dansmith: any chance of some weird eventlet patches in these customer environments? | |
| 13:47:51 | mriedem | so it would be rhosp specific? | |
| 13:48:04 | mriedem | or python-libvirt patches, something external to nova? | |
| 13:48:15 | dansmith | mriedem: I doubt it, but if they did and it was related, then they'd have been fixed by this (if that's what you mean) | |
| 13:48:21 | mriedem | also, if we land this on master, will backports be proposed upstream? | |
| 13:48:31 | dansmith | mriedem: they're probably on newer *libvirt*, but I dunno how often python-libvirt changes | |
| 13:49:01 | mriedem | i think you have to rebuild python-libvirt from whatever version of libvirt binary you're using | |
| 13:49:05 | mriedem | otherwise things get weird | |
| 13:49:32 | dansmith | mriedem: I dunno, ask mdbooth. I would expect he would want to | |
| 13:49:49 | mriedem | i remember we had issues in the past with running libvirt x in the gate jobs but python-libvirt was built against y and reporting y to nova, so we'd try to use features that weren't actually there | |
| 13:49:50 | dansmith | mriedem: right but that's just because the bindings are auto-generated I think | |
| 13:50:10 | mriedem | ok, so i'm less comfortable with backporting a thing that we're not sure fixes the issue | |
| 13:50:25 | mriedem | it's one thing to land on master and let it simmer until people get to it, | |
| 13:50:32 | mriedem | than to backport it and force it on people with otherwise stable environments | |
| 13:50:40 | dansmith | although we are right before FF, so not as much runway | |
| 13:51:01 | mriedem | rh probably just cares that it's merged on master and can backport internally is less of an issue yeah? | |
| 13:51:19 | mriedem | merge conflict risk on stable branches and such i know | |
| 13:51:24 | dansmith | maybe we could get efried to comment on the risk, and maybe cdent on the actual eventlet bits? | |
| 13:51:32 | dansmith | I'm not sure what context cdent has on eventlet, but it seems like sometimes he has a lot | |
| 13:51:44 | cdent | hello what? | |
| 13:51:49 | cdent | which thing? | |
| 13:51:49 | mriedem | efried isn't a stable branch maintainer though, or you just mean risk for FF? | |
| 13:52:01 | dansmith | mriedem: I mean for merging to master right now or in general yeah | |
| 13:52:10 | mriedem | cdent: https://review.opendev.org/#/c/677736/ | |
| 13:52:20 | mriedem | cdent: tl;dr old ass code, | |
| 13:52:40 | mriedem | suddenly 2 rhosp customers are seeing weird issues which mdbooth thinks are related to libvirt + python + eventlet, | |
| 13:52:41 | dansmith | (I wish I was working on things that *have* to merge by FF right now) | |
| 13:52:52 | mriedem | but there isn't a clear indication that patch fixes those problems | |
| 13:52:54 | openstackgerrit | Adam Spiers proposed openstack/nova master: Enable booting of libvirt guests with AMD SEV memory encryption https://review.opendev.org/666616 | |