| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-06-13 | |||
| 15:33:58 | manuvakery1 | live_migration_downtime_delay: 50, Still its not helping | |
| 15:33:58 | manuvakery1 | live_migration_downtime_steps: 10 | |
| 15:33:58 | manuvakery1 | live_migration_downtime: 500 | |
| 15:33:58 | manuvakery1 | Hi, Few of my VM live migration never completes, it always get stuck at the end of memory copy , going from 1% to 0 and then to 1%. I tried to set the following params | |
| 15:34:32 | manuvakery1 | I see its sets downtime only once to 50ms | |
| 15:35:04 | manuvakery1 | Any help on this | |
| 15:36:38 | manuvakery1 | 2023-06-13 15:01:20.336 3223605 INFO nova.virt.libvirt.migration [req-42a1d50d-1411-416e-8084-cf929cf892f9 - - - - -] [instance: 09658040-2af3-4fbf-884f-601a4d1f2ce2] Increasing downtime to 50 ms after 0 sec elapsed time | |
| 15:37:02 | frickler | manuvakery1: you may wnat to look into auto-completion or post-copy https://docs.openstack.org/nova/yoga/admin/configuring-migrations.html | |
| 15:38:18 | manuvakery1 | frickler: Sure i will .. but based on the configs I set it should increase the downtime to 100ms and then 150ms until it reaches 500ms | |
| 18:47:42 | stephenfin | sean-k-mooney[m]: melwitt: Would you folks be able to look at https://review.opendev.org/c/openstack/placement/+/880623/ | |
| 18:48:27 | stephenfin | (Context is we discussed bumping SQLAlchemy to 2.x at a TC forum session this morning and I realised this was proposed but never merged /o\) | |
| 18:52:07 | melwitt | stephenfin: sure, I can take a look later | |
| 18:52:30 | stephenfin | ty 🙏 | |
| 20:10:22 | opendevreview | Amit Uniyal proposed openstack/nova master: Allow swap resize from non-zero to zero https://review.opendev.org/c/openstack/nova/+/857339 | |
| 21:50:31 | opendevreview | Stephen Finucane proposed openstack/nova master: tests: Update test for DB special characters https://review.opendev.org/c/openstack/nova/+/880669 | |
| 22:44:57 | opendevreview | Stephen Finucane proposed openstack/nova master: db: Store unmasked password https://review.opendev.org/c/openstack/nova/+/880669 | |
| #openstack-nova - 2023-06-14 | |||
| 07:41:43 | opendevreview | Pierre Riteau proposed openstack/nova stable/victoria: Remove mentions of removed scheduler filters https://review.opendev.org/c/openstack/nova/+/858051 | |
| 08:25:39 | opendevreview | Kiran Pawar proposed openstack/nova-specs master: SR-IOV NIC device tracking in Placement https://review.opendev.org/c/openstack/nova-specs/+/884569 | |
| 08:53:07 | opendevreview | Vitalii Vrublevskyi proposed openstack/nova master: Napatech SmartNIC support https://review.opendev.org/c/openstack/nova/+/859577 | |
| 15:25:20 | opendevreview | Artom Lifshitz proposed openstack/nova master: Call Neutron immediately upon _post_live_migration() start https://review.opendev.org/c/openstack/nova/+/883682 | |
| 15:25:21 | opendevreview | Artom Lifshitz proposed openstack/nova master: Parallelize calls to Neutron and Cinder in post_live_migration https://review.opendev.org/c/openstack/nova/+/883678 | |
| 17:53:39 | opendevreview | Navid Pustchi proposed openstack/nova-specs master: dnm test https://review.opendev.org/c/openstack/nova-specs/+/886116 | |
| 17:55:41 | opendevreview | Navid Pustchi proposed openstack/nova-specs master: dnm test https://review.opendev.org/c/openstack/nova-specs/+/886116 | |
| 21:27:19 | melwitt | stephenfin, sean-k-mooney: I have a couple of patches related to database archiving that need review if yall get a chance. the second change looks big but it's because of the addition of a for loop (and the subsequent indentation of the code block under it). I added a comment on the review to explain the change in hopes of easing review https://review.opendev.org/q/topic:archive-trees-one-at-a-time | |
| 21:43:57 | opendevreview | Merged openstack/nova stable/xena: Remove mentions of removed scheduler filters https://review.opendev.org/c/openstack/nova/+/858024 | |
| 22:48:13 | opendevreview | Merged openstack/placement master: db: Replace use of deprecated API https://review.opendev.org/c/openstack/placement/+/880623 | |
| #openstack-nova - 2023-06-15 | |||
| 00:42:48 | opendevreview | Artom Lifshitz proposed openstack/nova master: Call Neutron immediately upon _post_live_migration() start https://review.opendev.org/c/openstack/nova/+/883682 | |
| 00:42:49 | opendevreview | Artom Lifshitz proposed openstack/nova master: Parallelize calls to Neutron and Cinder in post_live_migration https://review.opendev.org/c/openstack/nova/+/883678 | |
| 11:49:17 | opendevreview | Merged openstack/placement master: tests: Use base class for all functional tests https://review.opendev.org/c/openstack/placement/+/880624 | |
| 21:54:19 | opendevreview | Stephen Finucane proposed openstack/placement master: Add job to test with SQLAlchemy master (2.x) https://review.opendev.org/c/openstack/placement/+/886229 | |
| 21:58:33 | opendevreview | Stephen Finucane proposed openstack/placement master: Add job to test with SQLAlchemy master (2.x) https://review.opendev.org/c/openstack/placement/+/886229 | |
| 21:58:41 | opendevreview | Stephen Finucane proposed openstack/nova master: Add job to test with SQLAlchemy master (2.x) https://review.opendev.org/c/openstack/nova/+/886230 | |
| 23:45:36 | opendevreview | Merged openstack/placement master: tests: Warn on *any* SAWarning warning https://review.opendev.org/c/openstack/placement/+/880625 | |
| 23:49:50 | opendevreview | Sylvain Bauza proposed openstack/nova master: Add a new NumInstancesWeigher https://review.opendev.org/c/openstack/nova/+/886232 | |
| #openstack-nova - 2023-06-16 | |||
| 06:37:12 | opendevreview | melanie witt proposed openstack/nova master: database: Archive parent and child rows "trees" one at a time https://review.opendev.org/c/openstack/nova/+/877056 | |
| 06:37:12 | opendevreview | melanie witt proposed openstack/nova master: testing: Fix and robustify archive_deleted_rows test https://review.opendev.org/c/openstack/nova/+/877055 | |
| 19:08:02 | opendevreview | melanie witt proposed openstack/nova master: database: Archive parent and child rows "trees" one at a time https://review.opendev.org/c/openstack/nova/+/877056 | |
| 20:51:31 | melwitt | fyi just got two CI failures on the same patch in the nova-live-migration and nova-grenade-multinode jobs https://bugs.launchpad.net/neutron/+bug/1940425 and it reminded me I have been seeing this failure often lately | |
| 20:52:05 | melwitt | we likely need to look into this to improve gate fail rate | |
| #openstack-nova - 2023-06-17 | |||
| 02:41:47 | opendevreview | Stephen Finucane proposed openstack/nova master: Add job to test with SQLAlchemy master (2.x) https://review.opendev.org/c/openstack/nova/+/886230 | |
| #openstack-nova - 2023-06-19 | |||
| 13:03:08 | opendevreview | Danila Balagansky proposed openstack/nova master: Show free vCPUs in HostState __repr__ https://review.opendev.org/c/openstack/nova/+/886400 | |
| #openstack-nova - 2023-06-20 | |||
| 10:56:05 | noonedeadpunk | Hey folks! Completely forgot to ask that during PTG last week :( But wdyt about this bug report? For me it sounds more like a nova issue, that service fails and do not try to re-auth? https://bugs.launchpad.net/openstack-ansible/+bug/2023370 | |
| 10:57:04 | noonedeadpunk | So what's hapenning in OSA here, is that we reset a password (in 99.9% to exactly same one). While I kinda agree we should not do that at the first place, but this should not break service either? | |
| 11:26:17 | stephenfin | melwitt: Sorry for the delay, reviewed that DB archive series. I addressed your feedback on https://review.opendev.org/c/openstack/nova/+/880669/ also. Thanks for that | |
| 11:49:31 | opendevreview | Merged openstack/nova master: testing: Fix and robustify archive_deleted_rows test https://review.opendev.org/c/openstack/nova/+/877055 | |
| 12:52:17 | sean-k-mooney | noonedeadpunk: reset the password where | |
| 12:52:32 | sean-k-mooney | noonedeadpunk: you update the db password? rabbit? | |
| 12:52:35 | noonedeadpunk | sean-k-mooney: in keystone. | |
| 12:52:51 | noonedeadpunk | so issued previously tokens are invalidated | |
| 12:53:03 | noonedeadpunk | despite password is kinda the same | |
| 12:53:07 | sean-k-mooney | oh that sound like a keystone issue then | |
| 12:53:28 | sean-k-mooney | password or what by the way | |
| 12:53:44 | noonedeadpunk | but it happens kinda only when asking neutron api by nova service, and only first request fails, and then it re-asks for the token I believe | |
| 12:53:51 | sean-k-mooney | the password of the nova user? or the enduser creating the vm | |
| 12:53:57 | noonedeadpunk | of the nova user | |
| 12:54:15 | noonedeadpunk | and system service user per say | |
| 12:54:28 | sean-k-mooney | right so if you udpate it in keystoen you need to update it in the nova.conf and then restart the nova service | |
| 12:54:53 | sean-k-mooney | unless this is marked as mutable a sighup wont be enough | |
| 12:54:59 | noonedeadpunk | yeah, but it's reseted to the same value | |
| 12:55:12 | noonedeadpunk | So you kinda don't need to update conf | |
| 12:55:32 | noonedeadpunk | And thus - don't restart service | |
| 12:55:42 | sean-k-mooney | regardless of if its updateed to the same or diffent value i woudl expect keystone to acppet existing non expired tokens | |
| 12:55:54 | noonedeadpunk | And it wasn't an issue in X and even in Y I believe | |
| 12:56:00 | sean-k-mooney | so that why i said it sounded like a keystone issue | |
| 12:56:35 | sean-k-mooney | it sounds like they have made updating the password invalidation all current tokens | |
| 12:56:46 | sean-k-mooney | which would be a major api brake | |
| 12:57:15 | sean-k-mooney | as in it should requrie a major api bump as that woudl completely change the meaning of the token experation | |
| 12:58:15 | noonedeadpunk | I'm not sure that's exactly the case, was more guessing what hapens | |
| 12:58:34 | noonedeadpunk | As I can hardly imagine any other reason for such behaviour | |
| 12:59:05 | noonedeadpunk | The interesting part also, that according to the report, only neutron connection breaks | |
| 12:59:28 | sean-k-mooney | well if the user and password is the same when we create the admin client it shoudl get a new token | |
| 12:59:42 | sean-k-mooney | this is presumable port binding? | |
| 13:00:01 | noonedeadpunk | Yeah, I believe so. | |
| 13:00:03 | sean-k-mooney | we generally dont use nova for anything else we use the end useers token | |
| 13:00:22 | sean-k-mooney | ya so portbidning is admin only so we use the admin clinet withthe user/password form teh config | |
| 13:00:42 | noonedeadpunk | And it fails with `nova.exception.NeutronAdminCredentialConfigurationInvalid` | |
| 13:01:10 | noonedeadpunk | (which is weird exception kinda) | |
| 13:02:35 | noonedeadpunk | One thing ofc, is that we will just stop resetting password to the same value. This was added quite some time ago to ensure that password is correct | |
| 13:08:03 | noonedeadpunk | but even though I'd say it shouldn't fail that way. So maybe just having some retry or try/except would be good to get new token when needed | |
| 13:08:35 | anskiy | I'm the reporter of the https://bugs.launchpad.net/openstack-ansible/+bug/2023370. I see this behaviour in Y too, except that in Z it does recover, whilst in Y one needs to restart Nova API service. I haven't tested this in X, but I can do it if that would help. | |
| 13:08:58 | noonedeadpunk | anskiy: I think latest Y should be quite the same? | |
| 13:09:12 | noonedeadpunk | As OSSA-2023-003 was backported to Y | |
| 13:09:16 | anskiy | noonedeadpunk: oh, yes, true. | |
| 13:09:35 | noonedeadpunk | no idea though | |
| 13:27:09 | bauzas | noonedeadpunk: sean-k-mooney: I don't know how Neutron ensures Keystone authd, but at least for Nova, all our meat is provided by keystonemiddleware | |
| 13:27:19 | bauzas | I assume the same for Neutron | |
| 13:27:38 | bauzas | so yeah, sounds a Keystone issue, probably in middleware | |
| 13:28:09 | bauzas | I don't see why we would need to recycle the nova-api services, but if middleware does some sort of caching, this may explain | |
| 13:29:25 | noonedeadpunk | yeah, ok, will try to bother keystone folks then | |
| 13:30:00 | bauzas | noonedeadpunk: https://docs.openstack.org/keystonemiddleware/latest/middlewarearchitecture.html | |
| 13:30:31 | bauzas | particularly https://docs.openstack.org/keystonemiddleware/latest/middlewarearchitecture.html#improving-response-time | |
| 13:31:14 | noonedeadpunk | yeah, I'm pretty sure memcached is being configured | |
| 13:32:12 | bauzas | https://blog.benhall.me.uk/wp-content/uploads/2015/01/cache-invalidation.png | |
| 13:32:34 | bauzas | pretty well known stance, I assume | |
| 13:32:50 | noonedeadpunk | the werid thing though, is that on subsequent request it jsut passes | |
| 13:33:06 | noonedeadpunk | which made me thinking if it's jsut some simple retry that can be added somewhere | |
| 13:34:17 | bauzas | well, sounds a design problem to me, which is never simple to solve :) | |
| 13:34:48 | bauzas | like, memcached has to be flushed against a user if that user password changed | |
| 13:36:44 | bauzas | unrelated, was the f... is going on with nova-live-migration jobs ? I got two CI failures on the same patch with the same error https://review.opendev.org/c/openstack/nova/+/886232 | |
| 13:40:06 | stephenfin | bauzas: melwitt commented about that last week | |