| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-06-20 | |||
| 12:52:32 | sean-k-mooney | noonedeadpunk: you update the db password? rabbit? | |
| 12:52:35 | noonedeadpunk | sean-k-mooney: in keystone. | |
| 12:52:51 | noonedeadpunk | so issued previously tokens are invalidated | |
| 12:53:03 | noonedeadpunk | despite password is kinda the same | |
| 12:53:07 | sean-k-mooney | oh that sound like a keystone issue then | |
| 12:53:28 | sean-k-mooney | password or what by the way | |
| 12:53:44 | noonedeadpunk | but it happens kinda only when asking neutron api by nova service, and only first request fails, and then it re-asks for the token I believe | |
| 12:53:51 | sean-k-mooney | the password of the nova user? or the enduser creating the vm | |
| 12:53:57 | noonedeadpunk | of the nova user | |
| 12:54:15 | noonedeadpunk | and system service user per say | |
| 12:54:28 | sean-k-mooney | right so if you udpate it in keystoen you need to update it in the nova.conf and then restart the nova service | |
| 12:54:53 | sean-k-mooney | unless this is marked as mutable a sighup wont be enough | |
| 12:54:59 | noonedeadpunk | yeah, but it's reseted to the same value | |
| 12:55:12 | noonedeadpunk | So you kinda don't need to update conf | |
| 12:55:32 | noonedeadpunk | And thus - don't restart service | |
| 12:55:42 | sean-k-mooney | regardless of if its updateed to the same or diffent value i woudl expect keystone to acppet existing non expired tokens | |
| 12:55:54 | noonedeadpunk | And it wasn't an issue in X and even in Y I believe | |
| 12:56:00 | sean-k-mooney | so that why i said it sounded like a keystone issue | |
| 12:56:35 | sean-k-mooney | it sounds like they have made updating the password invalidation all current tokens | |
| 12:56:46 | sean-k-mooney | which would be a major api brake | |
| 12:57:15 | sean-k-mooney | as in it should requrie a major api bump as that woudl completely change the meaning of the token experation | |
| 12:58:15 | noonedeadpunk | I'm not sure that's exactly the case, was more guessing what hapens | |
| 12:58:34 | noonedeadpunk | As I can hardly imagine any other reason for such behaviour | |
| 12:59:05 | noonedeadpunk | The interesting part also, that according to the report, only neutron connection breaks | |
| 12:59:28 | sean-k-mooney | well if the user and password is the same when we create the admin client it shoudl get a new token | |
| 12:59:42 | sean-k-mooney | this is presumable port binding? | |
| 13:00:01 | noonedeadpunk | Yeah, I believe so. | |
| 13:00:03 | sean-k-mooney | we generally dont use nova for anything else we use the end useers token | |
| 13:00:22 | sean-k-mooney | ya so portbidning is admin only so we use the admin clinet withthe user/password form teh config | |
| 13:00:42 | noonedeadpunk | And it fails with `nova.exception.NeutronAdminCredentialConfigurationInvalid` | |
| 13:01:10 | noonedeadpunk | (which is weird exception kinda) | |
| 13:02:35 | noonedeadpunk | One thing ofc, is that we will just stop resetting password to the same value. This was added quite some time ago to ensure that password is correct | |
| 13:08:03 | noonedeadpunk | but even though I'd say it shouldn't fail that way. So maybe just having some retry or try/except would be good to get new token when needed | |
| 13:08:35 | anskiy | I'm the reporter of the https://bugs.launchpad.net/openstack-ansible/+bug/2023370. I see this behaviour in Y too, except that in Z it does recover, whilst in Y one needs to restart Nova API service. I haven't tested this in X, but I can do it if that would help. | |
| 13:08:58 | noonedeadpunk | anskiy: I think latest Y should be quite the same? | |
| 13:09:12 | noonedeadpunk | As OSSA-2023-003 was backported to Y | |
| 13:09:16 | anskiy | noonedeadpunk: oh, yes, true. | |
| 13:09:35 | noonedeadpunk | no idea though | |
| 13:27:09 | bauzas | noonedeadpunk: sean-k-mooney: I don't know how Neutron ensures Keystone authd, but at least for Nova, all our meat is provided by keystonemiddleware | |
| 13:27:19 | bauzas | I assume the same for Neutron | |
| 13:27:38 | bauzas | so yeah, sounds a Keystone issue, probably in middleware | |
| 13:28:09 | bauzas | I don't see why we would need to recycle the nova-api services, but if middleware does some sort of caching, this may explain | |
| 13:29:25 | noonedeadpunk | yeah, ok, will try to bother keystone folks then | |
| 13:30:00 | bauzas | noonedeadpunk: https://docs.openstack.org/keystonemiddleware/latest/middlewarearchitecture.html | |
| 13:30:31 | bauzas | particularly https://docs.openstack.org/keystonemiddleware/latest/middlewarearchitecture.html#improving-response-time | |
| 13:31:14 | noonedeadpunk | yeah, I'm pretty sure memcached is being configured | |
| 13:32:12 | bauzas | https://blog.benhall.me.uk/wp-content/uploads/2015/01/cache-invalidation.png | |
| 13:32:34 | bauzas | pretty well known stance, I assume | |
| 13:32:50 | noonedeadpunk | the werid thing though, is that on subsequent request it jsut passes | |
| 13:33:06 | noonedeadpunk | which made me thinking if it's jsut some simple retry that can be added somewhere | |
| 13:34:17 | bauzas | well, sounds a design problem to me, which is never simple to solve :) | |
| 13:34:48 | bauzas | like, memcached has to be flushed against a user if that user password changed | |
| 13:36:44 | bauzas | unrelated, was the f... is going on with nova-live-migration jobs ? I got two CI failures on the same patch with the same error https://review.opendev.org/c/openstack/nova/+/886232 | |
| 13:40:06 | stephenfin | bauzas: melwitt commented about that last week | |
| 13:40:19 | stephenfin | ||
| 13:40:19 | stephenfin | <melwitt> we likely need to look into this to improve gate fail rate | |
| 13:40:21 | bauzas | stephenfin: ack and the story is ? | |
| 13:40:30 | bauzas | ok | |
| 13:41:40 | bauzas | https://zuul.openstack.org/builds?job_name=nova-live-migration&skip=0 sounds pretty bad if you want MHO | |
| 13:42:24 | bauzas | ralonsoh: lajoskatona: you'll be happy to hear we have some kind of neutron hard failure on nova-live-migration jobs | |
| 13:42:43 | ralonsoh | bauzas, please! | |
| 13:43:00 | bauzas | what a nice day, isn't it ? :) | |
| 13:43:08 | lajoskatona | bauzas: this one perhaps: https://bugs.launchpad.net/neutron/+bug/2024160 ? | |
| 13:43:10 | ralonsoh | do you know what is happening? | |
| 13:43:27 | bauzas | ralonsoh: one of the failing logs https://ffd910c1fba2c4e789e5-da0bf4b2b0817e3979adc64c27f459a5.ssl.cf5.rackcdn.com/880719/5/check/nova-live-migration/08a7455/testr_results.html | |
| 13:43:31 | ralonsoh | so this is related only with the trunk ports, right? | |
| 13:43:46 | bauzas | yeah | |
| 13:43:47 | ralonsoh | ok, we have the triage meeting in 20 mins | |
| 13:43:53 | bauzas | tempest.api.compute.admin.test_live_migration.LiveMigrationTest is failing | |
| 13:43:56 | bauzas | due to a failed assert | |
| 13:43:58 | ralonsoh | I'll raise it as urgent | |
| 13:44:10 | ralonsoh | can you disable this test temporarily? | |
| 13:44:13 | ralonsoh | hold on | |
| 13:44:16 | bauzas | lajoskatona: sounds related | |
| 13:44:25 | ralonsoh | is this CI in a gate/check queue? | |
| 13:44:39 | ralonsoh | I mean, is this blocking you from merging patches? | |
| 13:45:19 | lajoskatona | ralonsoh, bauzas: yeah by the logs seems bug/2024160 , and to the OVN version bump | |
| 13:46:30 | ralonsoh | bauzas, is this in a gate/check queue? is this blocking you from merging patches? | |
| 13:46:44 | ralonsoh | if so, I'll disable this test temporarily | |
| 13:48:10 | bauzas | ralonsoh: (sorry, was elsewhere) yeah this is a voting job | |
| 13:48:22 | bauzas | on both check and gate | |
| 13:48:30 | bauzas | Tempest seems impacted too | |
| 13:48:58 | ralonsoh | ok, I'll disable make this test unstable temporarily | |
| 13:49:08 | ralonsoh | but we'll start working on this right now | |
| 13:50:07 | bauzas | I can add a skip test in zuul | |
| 13:50:23 | bauzas | ideally, we should have a tempest bug report against it | |
| 13:50:32 | bauzas | want me to declare it ? | |
| 13:51:43 | ralonsoh | bauzas, isn't enough with the Neutron LP bug? | |
| 13:51:54 | ralonsoh | I'm sending a patch right now for tempest | |
| 13:52:15 | bauzas | ralonsoh: sorry I misread your last comment, I thought you said the opposite | |
| 13:52:30 | ralonsoh | broken phone, I think | |
| 13:52:37 | ralonsoh | sorry if I wasn't clear | |
| 13:56:30 | ralonsoh | --> https://review.opendev.org/c/openstack/tempest/+/886496 | |
| 13:56:39 | bauzas | ralonsoh++ | |
| 13:59:34 | lajoskatona | +1 | |
| 15:02:44 | bauzas | reminder : nova meeting in 1 hour-ish here | |
| 16:01:32 | bauzas | sorry | |
| 16:01:36 | bauzas | #startmeeting nova | |
| 16:01:36 | opendevmeet | Meeting started Tue Jun 20 16:01:36 2023 UTC and is due to finish in 60 minutes. The chair is bauzas. Information about MeetBot at http://wiki.debian.org/MeetBot. | |
| 16:01:36 | opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | |