| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-04-19 | |||
| 19:31:19 | eandersson | http://paste.openstack.org/show/eXVi99OZVrjJ6V9C496P/ | |
| 19:40:07 | fried_rice | eandersson: Oh. Ahem. That min_version/max_version thing is just wrong. No wonder it's borked. | |
| 19:40:26 | fried_rice | Those are supposed to be major version numbers, not microversions. | |
| 19:40:36 | fried_rice | yeah, those should just be removed entirely. | |
| 19:40:48 | fried_rice | ye gods, who *wrote* this crap?? | |
| 19:41:19 | fried_rice | eandersson: Do you have a bug for this? | |
| 19:56:42 | fried_rice | mriedem: I thought I could get rid of, "There is no script for 63 version" by rebuilding venv, wth? | |
| 19:58:39 | mriedem | is that an api db version? | |
| 19:59:08 | mriedem | do you have stale pycs or something? | |
| 20:00:02 | eandersson | no bug yet fried_rice | |
| 20:00:15 | fried_rice | eandersson: Okay, we're going to need one | |
| 20:00:24 | fried_rice | eandersson: Because this fix will want to be backported | |
| 20:00:36 | fried_rice | mriedem: Must be. | |
| 20:00:49 | fried_rice | mriedem: seems to have cleared up now | |
| 20:01:25 | eandersson | I'll create one in a bit | |
| 20:01:59 | fried_rice | thanks | |
| 20:02:35 | fried_rice | mriedem: I was on the second of those patches, 'Soft delete virtual_interfaces when instance is destroyed' -- test passes when fix is reverted. | |
| 20:03:17 | mriedem | ok i'll look in a bit | |
| 20:12:11 | fried_rice | eandersson: So this https://github.com/openstack-dev/devstack/blob/master/lib/nova_plugins/hypervisor-ironic#L44 isn't setting api_endpoint. Which means it still ought to be going through the code path that does get_ksa_adapter with the bogus min_version and max_version kwargs. Which should still be returning None. Which should still be attempting to call get_client with endpoint=None. So I don't get how adding region_name to | |
| 20:13:01 | fried_rice | because region_name isn't in the kwargs sent to get_client | |
| 20:15:31 | fried_rice | get_endpoint with min/max but no region in the conf somehow magically works, but adding region_name in the conf makes it not work until you remove the min/max version?? /me confused. | |
| 20:28:05 | eandersson | Yea - not sure why it works without the region_name | |
| 20:30:17 | mriedem | fried_rice: test_delete_virtual_interfaces_on_instance_destroy fails for me locally if i remove the fix | |
| 20:42:12 | eandersson | I wrote a bug report but... launchpad error'd out :'( | |
| 20:45:21 | fried_rice | eandersson: Yeah, I won't be able to post the fix until all the things come back anyway. | |
| 20:45:30 | fried_rice | eandersson: One other data point that would be interesting: | |
| 20:46:13 | fried_rice | The failure setup, i.e. with region_name and min_version and max_version, but change min_version from IRONIC_API_VERSION to the tuple (1, 0) | |
| 20:49:41 | eandersson | Yea that seems to work | |
| 20:50:44 | fried_rice | eandersson: Okay, good, theory confirmed. Soon as the dark, dark night is over, we can have this fix pushed. | |
| 20:51:08 | fried_rice | though it's still a mystery why it doesn't break until you specify region_name. mordred, you around or chasing bunnies? | |
| 20:51:48 | mordred | chasing finishing this opendev rollout | |
| 20:52:01 | fried_rice | oh, you're involved in that? I'm sorry. | |
| 20:52:04 | fried_rice | Please do carry on. | |
| 20:52:15 | mordred | sorry - I saw the note from eandersson earlier -just havne't gotten to it. | |
| 20:52:19 | mordred | we're having all the fun | |
| 20:53:01 | fried_rice | yeah, I don't want to distract you from that for sure. | |
| 20:53:05 | eandersson | btw fried_rice would be nice if we could error out if endpoint is None | |
| 20:53:30 | fried_rice | eandersson: Well, if you look at the comment there, I deliberately didn't do that | |
| 20:53:32 | eandersson | because at the moment if this gets fixed, and nova is misconfigured it will look like it's working | |
| 20:53:52 | eandersson | Yea - saw your comment on that | |
| 20:53:59 | eandersson | At the very least I believe we should log it | |
| 20:54:04 | fried_rice | I was basically paranoid about changing the nature of legacy broken behavior. | |
| 20:54:09 | fried_rice | Yeah, I can get behind a log for sure. | |
| 20:54:11 | eandersson | because it can lead to some bad stuff | |
| 20:54:30 | eandersson | or maybe we can pass on region to the ironic client | |
| 20:55:03 | eandersson | because we are essentially telling the ironic client, hey if I fail, just make the best judgment you can.... but I will not give you all the info you need | |
| 20:55:06 | fried_rice | eandersson: Well, I would rather rip out ironicclient altogether and use the sdk. | |
| 20:55:29 | eandersson | I do like that idea :p | |
| 20:56:07 | fried_rice | eandersson: https://blueprints.launchpad.net/nova/+spec/openstacksdk-in-nova | |
| 20:56:19 | fried_rice | (I wasn't being entirely hypothetical) | |
| 20:56:33 | eandersson | Yep - already commented on the very initial implementation :P | |
| 20:56:59 | fried_rice | cool. I would have known that if I could have referenced the gerrit part of my extended brain. | |
| 21:00:07 | eandersson | https://github.com/openstack/python-ironicclient/blob/master/ironicclient/client.py#L111 | |
| 21:00:32 | eandersson | I still think we should pass on region_name to the client if we keep the current behavior with a possible endpoint = None | |
| 21:01:13 | eandersson | That way at the least the ironicclient can make an educated decision | |
| 21:02:50 | fried_rice | That can be done. I think I would prefer it to be a separate change though. It's clearly a different bug from the bogus min_version thing. | |
| 21:02:57 | eandersson | For sure | |
| 21:03:48 | fried_rice | ugh, this is such a mess | |
| 21:04:35 | fried_rice | ironicclient is doing all this preliminary stuff that's almost (but not quite, and incorrectly) duplicating a subset of the logic in ksa itself. | |
| 21:05:03 | eandersson | Yea - this took me hours to figure out | |
| 21:05:12 | eandersson | Because there was actually 3 separate bugs | |
| 21:05:17 | fried_rice | and for that matter, nova's client_wrapper is *also* doing more of that | |
| 21:05:24 | fried_rice | mainly for backward compatibility | |
| 21:07:55 | eandersson | https://github.com/openstack/python-ironicclient/commit/466be3b6568b643605d826e5aa26d9a344cc74ae | |
| 21:08:01 | eandersson | This bug added A LOT of confusion for me | |
| 21:08:06 | fried_rice | mriedem: Confirmed the test fails when the fix is removed. Something bizarre happening with cached .pyc files maybe, would explain a couple of the weird things that have befallen me today. So I'll be +A on that guy when the world comes back up. | |
| 21:08:44 | fried_rice | eandersson: Oh yeah, that one had a knock-on effect resulting in several other bugs. | |
| 21:09:13 | melwitt | mriedem: re: tripleo, I think it supports neither deploy nor upgrade with extracted placement based on the last I heard from lyarwood and EmelienM. this isn't going to work right now but the relevant patches are https://review.openstack.org/#/c/630644/ and https://review.openstack.org/#/q/topic:tripleo-placement-extraction+status:open and https://review.openstack.org/#/q/topic:tripleo-placement-upgrade-from-nova-placement+status:open | |
| 21:09:29 | fried_rice | eandersson: like https://github.com/openstack/python-ironicclient/commit/ae1743d2c194c690c4d4629e51e860b5f5b84252 | |
| 21:09:49 | melwitt | mriedem: I'm not 100% clear on whether deploy is supported in any capacity or if it's just that it can't deploy it by default yet | |
| 21:12:04 | mriedem | melwitt: ok | |
| 21:16:26 | melwitt | it's being worked on and supposed to ramp up soon. there's an additional person who's going to help who has been taken on the task recently | |
| 21:24:40 | imacdonn | who knows stuff about eventlet monkey-patching and WSGI? Re. the problem I reported on the ML, with nova-API not maintain the heartbeat on AMQP connections when running under uWSGI, I've discovered that the problem goes away when I remove the eventlet monkey-patching | |
| 21:28:50 | fried_rice | imacdonn: isn't mdbooth_ our resident monkey-patching expert, and cdent our resident wsgi expert? I kinda think you might have poor luck raising the former, and I know the latter is on vacation. | |
| 21:30:31 | melwitt | imacdonn: mdbooth_ landed a patch to change eventlet monkey-patching around in train, that you could try to see if it helps if you were up for an experiment, but I can't link it to you right now bc gerrit | |
| 21:31:03 | melwitt | I guess I could find it in github | |
| 21:31:14 | imacdonn | melwitt, Are you sure that didn't make Stein? i.e. this: https://github.com/openstack/nova/commit/3c5e2b0e9fac985294a949852bb8c83d4ed77e04 | |
| 21:31:25 | imacdonn | problem started before that, and continues with that, though | |
| 21:32:02 | melwitt | imacdonn: I'm sure. it's proposed for backport to stein but we wanted to let it bake longer and see if anyone else was having problems first. bc only RHOSP reported the issue | |
| 21:33:58 | imacdonn | melwitt: hmm. Not finding anything beyond the above (which was definitely about "moving it around") | |
| 21:34:33 | imacdonn | melwitt: maybe not merged to master yet? | |
| 21:34:41 | melwitt | imacdonn: no, that's the one, it's just it's not in stein | |
| 21:34:46 | fried_rice | definitely merged in master | |
| 21:34:54 | melwitt | it's in train | |
| 21:35:24 | melwitt | I was saying you could try applying that patch to see if it helps | |
| 21:36:54 | imacdonn | hmm, that's odd .. I seemed to have that change ... which suggests either the RDO Stein packages are not actually Stein, or I'm doing something stupid\ | |
| 21:37:47 | imacdonn | python2-nova-19.0.0-1.el7.noarch | |
| 21:37:47 | imacdonn | # rpm -qf nova/monkey_patch.py | |
| 21:38:52 | melwitt | ohhh | |
| 21:39:12 | melwitt | I bet we (redhat) backported it downstream only. but I wasn't sure we do that for RDO | |
| 21:39:32 | fried_rice | Wow. That seems kinda crazy. | |
| 21:39:49 | melwitt | I guess they did in this case at least bc otherwise it was totally broken for us | |
| 21:39:51 | fried_rice | But I guess this was a special case because RH was busted. | |
| 21:39:53 | fried_rice | yeah | |
| 21:40:42 | fried_rice | but imacdonn you said the problem existed before this commit as well? Or is that erroneous in light of revelations above? | |
| 21:40:44 | imacdonn | so, in any case, my problem is present both with and without that change | |
| 21:40:48 | fried_rice | okay. | |
| 21:41:08 | melwitt | ok, was worth a shot | |
| 21:41:25 | melwitt | I have to run now, I'm off today because it's Spring Holiday™. see yall next week | |
| 21:41:42 | imacdonn | k, have fun ;) | |