| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-06-14 | |||
| 21:43:57 | opendevreview | Merged openstack/nova stable/xena: Remove mentions of removed scheduler filters https://review.opendev.org/c/openstack/nova/+/858024 | |
| 22:48:13 | opendevreview | Merged openstack/placement master: db: Replace use of deprecated API https://review.opendev.org/c/openstack/placement/+/880623 | |
| #openstack-nova - 2023-06-15 | |||
| 00:42:48 | opendevreview | Artom Lifshitz proposed openstack/nova master: Call Neutron immediately upon _post_live_migration() start https://review.opendev.org/c/openstack/nova/+/883682 | |
| 00:42:49 | opendevreview | Artom Lifshitz proposed openstack/nova master: Parallelize calls to Neutron and Cinder in post_live_migration https://review.opendev.org/c/openstack/nova/+/883678 | |
| 11:49:17 | opendevreview | Merged openstack/placement master: tests: Use base class for all functional tests https://review.opendev.org/c/openstack/placement/+/880624 | |
| 21:54:19 | opendevreview | Stephen Finucane proposed openstack/placement master: Add job to test with SQLAlchemy master (2.x) https://review.opendev.org/c/openstack/placement/+/886229 | |
| 21:58:33 | opendevreview | Stephen Finucane proposed openstack/placement master: Add job to test with SQLAlchemy master (2.x) https://review.opendev.org/c/openstack/placement/+/886229 | |
| 21:58:41 | opendevreview | Stephen Finucane proposed openstack/nova master: Add job to test with SQLAlchemy master (2.x) https://review.opendev.org/c/openstack/nova/+/886230 | |
| 23:45:36 | opendevreview | Merged openstack/placement master: tests: Warn on *any* SAWarning warning https://review.opendev.org/c/openstack/placement/+/880625 | |
| 23:49:50 | opendevreview | Sylvain Bauza proposed openstack/nova master: Add a new NumInstancesWeigher https://review.opendev.org/c/openstack/nova/+/886232 | |
| #openstack-nova - 2023-06-16 | |||
| 06:37:12 | opendevreview | melanie witt proposed openstack/nova master: testing: Fix and robustify archive_deleted_rows test https://review.opendev.org/c/openstack/nova/+/877055 | |
| 06:37:12 | opendevreview | melanie witt proposed openstack/nova master: database: Archive parent and child rows "trees" one at a time https://review.opendev.org/c/openstack/nova/+/877056 | |
| 19:08:02 | opendevreview | melanie witt proposed openstack/nova master: database: Archive parent and child rows "trees" one at a time https://review.opendev.org/c/openstack/nova/+/877056 | |
| 20:51:31 | melwitt | fyi just got two CI failures on the same patch in the nova-live-migration and nova-grenade-multinode jobs https://bugs.launchpad.net/neutron/+bug/1940425 and it reminded me I have been seeing this failure often lately | |
| 20:52:05 | melwitt | we likely need to look into this to improve gate fail rate | |
| #openstack-nova - 2023-06-17 | |||
| 02:41:47 | opendevreview | Stephen Finucane proposed openstack/nova master: Add job to test with SQLAlchemy master (2.x) https://review.opendev.org/c/openstack/nova/+/886230 | |
| #openstack-nova - 2023-06-19 | |||
| 13:03:08 | opendevreview | Danila Balagansky proposed openstack/nova master: Show free vCPUs in HostState __repr__ https://review.opendev.org/c/openstack/nova/+/886400 | |
| #openstack-nova - 2023-06-20 | |||
| 10:56:05 | noonedeadpunk | Hey folks! Completely forgot to ask that during PTG last week :( But wdyt about this bug report? For me it sounds more like a nova issue, that service fails and do not try to re-auth? https://bugs.launchpad.net/openstack-ansible/+bug/2023370 | |
| 10:57:04 | noonedeadpunk | So what's hapenning in OSA here, is that we reset a password (in 99.9% to exactly same one). While I kinda agree we should not do that at the first place, but this should not break service either? | |
| 11:26:17 | stephenfin | melwitt: Sorry for the delay, reviewed that DB archive series. I addressed your feedback on https://review.opendev.org/c/openstack/nova/+/880669/ also. Thanks for that | |
| 11:49:31 | opendevreview | Merged openstack/nova master: testing: Fix and robustify archive_deleted_rows test https://review.opendev.org/c/openstack/nova/+/877055 | |
| 12:52:17 | sean-k-mooney | noonedeadpunk: reset the password where | |
| 12:52:32 | sean-k-mooney | noonedeadpunk: you update the db password? rabbit? | |
| 12:52:35 | noonedeadpunk | sean-k-mooney: in keystone. | |
| 12:52:51 | noonedeadpunk | so issued previously tokens are invalidated | |
| 12:53:03 | noonedeadpunk | despite password is kinda the same | |
| 12:53:07 | sean-k-mooney | oh that sound like a keystone issue then | |
| 12:53:28 | sean-k-mooney | password or what by the way | |
| 12:53:44 | noonedeadpunk | but it happens kinda only when asking neutron api by nova service, and only first request fails, and then it re-asks for the token I believe | |
| 12:53:51 | sean-k-mooney | the password of the nova user? or the enduser creating the vm | |
| 12:53:57 | noonedeadpunk | of the nova user | |
| 12:54:15 | noonedeadpunk | and system service user per say | |
| 12:54:28 | sean-k-mooney | right so if you udpate it in keystoen you need to update it in the nova.conf and then restart the nova service | |
| 12:54:53 | sean-k-mooney | unless this is marked as mutable a sighup wont be enough | |
| 12:54:59 | noonedeadpunk | yeah, but it's reseted to the same value | |
| 12:55:12 | noonedeadpunk | So you kinda don't need to update conf | |
| 12:55:32 | noonedeadpunk | And thus - don't restart service | |
| 12:55:42 | sean-k-mooney | regardless of if its updateed to the same or diffent value i woudl expect keystone to acppet existing non expired tokens | |
| 12:55:54 | noonedeadpunk | And it wasn't an issue in X and even in Y I believe | |
| 12:56:00 | sean-k-mooney | so that why i said it sounded like a keystone issue | |
| 12:56:35 | sean-k-mooney | it sounds like they have made updating the password invalidation all current tokens | |
| 12:56:46 | sean-k-mooney | which would be a major api brake | |
| 12:57:15 | sean-k-mooney | as in it should requrie a major api bump as that woudl completely change the meaning of the token experation | |
| 12:58:15 | noonedeadpunk | I'm not sure that's exactly the case, was more guessing what hapens | |
| 12:58:34 | noonedeadpunk | As I can hardly imagine any other reason for such behaviour | |
| 12:59:05 | noonedeadpunk | The interesting part also, that according to the report, only neutron connection breaks | |
| 12:59:28 | sean-k-mooney | well if the user and password is the same when we create the admin client it shoudl get a new token | |
| 12:59:42 | sean-k-mooney | this is presumable port binding? | |
| 13:00:01 | noonedeadpunk | Yeah, I believe so. | |
| 13:00:03 | sean-k-mooney | we generally dont use nova for anything else we use the end useers token | |
| 13:00:22 | sean-k-mooney | ya so portbidning is admin only so we use the admin clinet withthe user/password form teh config | |
| 13:00:42 | noonedeadpunk | And it fails with `nova.exception.NeutronAdminCredentialConfigurationInvalid` | |
| 13:01:10 | noonedeadpunk | (which is weird exception kinda) | |
| 13:02:35 | noonedeadpunk | One thing ofc, is that we will just stop resetting password to the same value. This was added quite some time ago to ensure that password is correct | |
| 13:08:03 | noonedeadpunk | but even though I'd say it shouldn't fail that way. So maybe just having some retry or try/except would be good to get new token when needed | |
| 13:08:35 | anskiy | I'm the reporter of the https://bugs.launchpad.net/openstack-ansible/+bug/2023370. I see this behaviour in Y too, except that in Z it does recover, whilst in Y one needs to restart Nova API service. I haven't tested this in X, but I can do it if that would help. | |
| 13:08:58 | noonedeadpunk | anskiy: I think latest Y should be quite the same? | |
| 13:09:12 | noonedeadpunk | As OSSA-2023-003 was backported to Y | |
| 13:09:16 | anskiy | noonedeadpunk: oh, yes, true. | |
| 13:09:35 | noonedeadpunk | no idea though | |
| 13:27:09 | bauzas | noonedeadpunk: sean-k-mooney: I don't know how Neutron ensures Keystone authd, but at least for Nova, all our meat is provided by keystonemiddleware | |
| 13:27:19 | bauzas | I assume the same for Neutron | |
| 13:27:38 | bauzas | so yeah, sounds a Keystone issue, probably in middleware | |
| 13:28:09 | bauzas | I don't see why we would need to recycle the nova-api services, but if middleware does some sort of caching, this may explain | |
| 13:29:25 | noonedeadpunk | yeah, ok, will try to bother keystone folks then | |
| 13:30:00 | bauzas | noonedeadpunk: https://docs.openstack.org/keystonemiddleware/latest/middlewarearchitecture.html | |
| 13:30:31 | bauzas | particularly https://docs.openstack.org/keystonemiddleware/latest/middlewarearchitecture.html#improving-response-time | |
| 13:31:14 | noonedeadpunk | yeah, I'm pretty sure memcached is being configured | |
| 13:32:12 | bauzas | https://blog.benhall.me.uk/wp-content/uploads/2015/01/cache-invalidation.png | |
| 13:32:34 | bauzas | pretty well known stance, I assume | |
| 13:32:50 | noonedeadpunk | the werid thing though, is that on subsequent request it jsut passes | |
| 13:33:06 | noonedeadpunk | which made me thinking if it's jsut some simple retry that can be added somewhere | |
| 13:34:17 | bauzas | well, sounds a design problem to me, which is never simple to solve :) | |
| 13:34:48 | bauzas | like, memcached has to be flushed against a user if that user password changed | |
| 13:36:44 | bauzas | unrelated, was the f... is going on with nova-live-migration jobs ? I got two CI failures on the same patch with the same error https://review.opendev.org/c/openstack/nova/+/886232 | |
| 13:40:06 | stephenfin | bauzas: melwitt commented about that last week | |
| 13:40:19 | stephenfin | ||
| 13:40:19 | stephenfin | <melwitt> we likely need to look into this to improve gate fail rate | |
| 13:40:21 | bauzas | stephenfin: ack and the story is ? | |
| 13:40:30 | bauzas | ok | |
| 13:41:40 | bauzas | https://zuul.openstack.org/builds?job_name=nova-live-migration&skip=0 sounds pretty bad if you want MHO | |
| 13:42:24 | bauzas | ralonsoh: lajoskatona: you'll be happy to hear we have some kind of neutron hard failure on nova-live-migration jobs | |
| 13:42:43 | ralonsoh | bauzas, please! | |
| 13:43:00 | bauzas | what a nice day, isn't it ? :) | |
| 13:43:08 | lajoskatona | bauzas: this one perhaps: https://bugs.launchpad.net/neutron/+bug/2024160 ? | |
| 13:43:10 | ralonsoh | do you know what is happening? | |
| 13:43:27 | bauzas | ralonsoh: one of the failing logs https://ffd910c1fba2c4e789e5-da0bf4b2b0817e3979adc64c27f459a5.ssl.cf5.rackcdn.com/880719/5/check/nova-live-migration/08a7455/testr_results.html | |
| 13:43:31 | ralonsoh | so this is related only with the trunk ports, right? | |
| 13:43:46 | bauzas | yeah | |
| 13:43:47 | ralonsoh | ok, we have the triage meeting in 20 mins | |
| 13:43:53 | bauzas | tempest.api.compute.admin.test_live_migration.LiveMigrationTest is failing | |
| 13:43:56 | bauzas | due to a failed assert | |
| 13:43:58 | ralonsoh | I'll raise it as urgent | |
| 13:44:10 | ralonsoh | can you disable this test temporarily? | |
| 13:44:13 | ralonsoh | hold on | |
| 13:44:16 | bauzas | lajoskatona: sounds related | |
| 13:44:25 | ralonsoh | is this CI in a gate/check queue? | |
| 13:44:39 | ralonsoh | I mean, is this blocking you from merging patches? | |
| 13:45:19 | lajoskatona | ralonsoh, bauzas: yeah by the logs seems bug/2024160 , and to the OVN version bump | |
| 13:46:30 | ralonsoh | bauzas, is this in a gate/check queue? is this blocking you from merging patches? | |