| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-07-05 | |||
| 10:14:47 | lyarwood | test_volume_backed_live_migration is still failing | |
| 10:14:49 | lyarwood | argh | |
| 10:15:23 | lyarwood | actually that might be an issue with the change | |
| 10:15:52 | lyarwood | becuase https://zuul.opendev.org/t/openstack/builds?job_name=nova-live-migration&project=openstack%2Fnova&branch=master&pipeline=check looks pretty good | |
| 10:48:45 | lyarwood | http://paste.openstack.org/show/807166/ - made some progress with this, it isn't related to the patch AFAICT | |
| 11:01:07 | gibi | lyarwood: good finding | |
| 11:02:51 | kashyap | gibi: Heh, I once did a bit of it on list: | |
| 11:03:56 | kashyap | gibi: You need to follow the request/response "libvirt-$ID": http://lists.openstack.org/pipermail/openstack-dev/2016-October/105158.html | |
| 11:04:11 | kashyap | Each request and response will have the same libvirt-ID: libvirt-30 (or whatever number) | |
| 11:04:32 | kashyap | lyarwood: I saw the brief chat on #virt; it's the dreaded "Device or resource busy" problem | |
| 11:04:57 | kashyap | It is my list of top two "most notorious libvirt errors" | |
| 11:05:02 | lyarwood | kinda | |
| 11:05:12 | lyarwood | that's causing libvirtd to lockup | |
| 11:05:19 | lyarwood | the migration is in a different thread | |
| 11:05:27 | lyarwood | and the dest times out before the src | |
| 11:05:31 | lyarwood | but the src then continues | |
| 11:05:51 | lyarwood | that's a separate behavioural issue in libvirtd tbh | |
| 11:06:16 | kashyap | Me nods | |
| 11:06:34 | kashyap | There's also the SIGTERM / SIGKILL dance. Often times only killing the instance and restarting seems to be the "solution" | |
| 11:07:55 | kashyap | (Aside: if you're wondering what is the other error in my top-2 list, it's the "cannot acquire state change lock") | |
| 11:11:34 | gibi | kashyap: thank I will read that | |
| 12:05:29 | opendevreview | Merged openstack/nova-specs master: [template]suggest work item ordering https://review.opendev.org/c/openstack/nova-specs/+/793197 | |
| 12:06:28 | sean-k-mooney | bauzas: im +2 on https://review.opendev.org/c/openstack/nova-specs/+/792796 by the way but held off +w to see if you want to adress teh nits in a follow up patch or if you wanted to quickly respin | |
| 12:17:49 | bauzas | sean-k-mooney: ack, will quickly look (was on PTO on Friday) | |
| 14:46:19 | alexe9191 | Goodday everyone :) | |
| 14:46:27 | alexe9191 | I was wondering if anyone has information on this blueprint? https://blueprints.launchpad.net/nova/+spec/detach-boot-volume | |
| 14:46:53 | alexe9191 | I see that it is waiting for approval since train and I was wondering if there are any plans to release it anytime soon ? | |
| 14:46:54 | sean-k-mooney | i think the people proposing it did not complete it | |
| 14:47:38 | alexe9191 | too bad | |
| 14:48:10 | sean-k-mooney | https://review.opendev.org/q/topic:%22bp%252Fdetach-boot-volume%22+(status:open%20OR%20status:merged) | |
| 14:48:24 | gibi | yepp it seems it is simply stalled out https://review.opendev.org/c/openstack/nova/+/623981/ | |
| 14:48:31 | sean-k-mooney | https://review.opendev.org/q/topic:%2522bp/detach-boot-volume%2522+ | |
| 14:48:44 | sean-k-mooney | so the most recent code is abandoned | |
| 14:49:19 | sean-k-mooney | alexe9191: it could be picked back up | |
| 14:49:27 | sean-k-mooney | i dont think it stalled out because of design reasons | |
| 14:51:01 | sean-k-mooney | matt's -1 was because we did not need to bump the object verion and the is_multi_attach is not implemted correctly | |
| 14:51:07 | sean-k-mooney | https://review.opendev.org/c/openstack/nova/+/623981/24/nova/objects/block_device.py | |
| 14:52:40 | alexe9191 | Abandoned meaning no one is working on it, or it won't be implemented for other reasons? | |
| 14:53:37 | sean-k-mooney | alexe9191: no one is working on it | |
| 14:53:57 | sean-k-mooney | it can be repoposed and implemented if people still want it and have time to work on it | |
| 16:11:56 | opendevreview | Merged openstack/nova-specs master: Re-propose "CPU selection with hypervisor consideration" https://review.opendev.org/c/openstack/nova-specs/+/799096 | |
| 16:18:11 | opendevreview | Lee Yarwood proposed openstack/nova master: WIP/DNM - block_device: Use initialize APIs to refresh when reported as idempotent https://review.opendev.org/c/openstack/nova/+/720769 | |
| 16:26:11 | opendevreview | Stephen Finucane proposed openstack/nova master: db: Use module-level imports for sqlalchemy (for real) https://review.opendev.org/c/openstack/nova/+/796519 | |
| 16:26:12 | opendevreview | Stephen Finucane proposed openstack/nova master: db: Move main DB migrations https://review.opendev.org/c/openstack/nova/+/799519 | |
| 16:26:12 | opendevreview | Stephen Finucane proposed openstack/nova master: db: Move db.sqalchemy.migration to db.migration https://review.opendev.org/c/openstack/nova/+/799518 | |
| 16:26:13 | opendevreview | Stephen Finucane proposed openstack/nova master: db: Remove constant aliases from 'nova.db.api' https://review.opendev.org/c/openstack/nova/+/799521 | |
| 16:26:13 | opendevreview | Stephen Finucane proposed openstack/nova master: db: Move 'sqlalchemy.types' up a directory https://review.opendev.org/c/openstack/nova/+/799520 | |
| 16:26:14 | opendevreview | Stephen Finucane proposed openstack/nova master: db: Drop support for experimental concurrency https://review.opendev.org/c/openstack/nova/+/799522 | |
| 16:26:15 | opendevreview | Stephen Finucane proposed openstack/nova master: db: Register database config options ourselves https://review.opendev.org/c/openstack/nova/+/799523 | |
| 16:26:16 | opendevreview | Stephen Finucane proposed openstack/nova master: db: Move remaining 'nova.db.sqlalchemy' modules https://review.opendev.org/c/openstack/nova/+/799525 | |
| 16:26:16 | opendevreview | Stephen Finucane proposed openstack/nova master: db: Unify 'nova.db.api', 'nova.db.sqlalchemy.api' https://review.opendev.org/c/openstack/nova/+/799524 | |
| 16:26:17 | opendevreview | Stephen Finucane proposed openstack/nova master: db: Post reshuffle cleanup https://review.opendev.org/c/openstack/nova/+/799526 | |
| 16:26:18 | opendevreview | Stephen Finucane proposed openstack/nova master: db: Add initial alembic migration for main DB https://review.opendev.org/c/openstack/nova/+/799527 | |
| 16:26:20 | opendevreview | Stephen Finucane proposed openstack/nova master: db: Add initial alembic migration for API DB https://review.opendev.org/c/openstack/nova/+/799528 | |
| 16:26:21 | opendevreview | Stephen Finucane proposed openstack/nova master: WIP: db: Integrate alembic https://review.opendev.org/c/openstack/nova/+/799530 | |
| 16:26:21 | opendevreview | Stephen Finucane proposed openstack/nova master: db: Trivial style changes https://review.opendev.org/c/openstack/nova/+/799529 | |
| 16:28:30 | opendevreview | Ghanshyam proposed openstack/placement master: Fix oslo policy DeprecatedRule warnings https://review.opendev.org/c/openstack/placement/+/799418 | |
| 16:29:07 | stephenfin | dansmith: If you have a bit of spare time, I'd appreciate some eyes on the last one of those patches ^ I can't quite figure out why the 'CellDatabases' fixtures is failing but it is :-\ | |
| 17:12:17 | sean-k-mooney | gibi: how do you feel about https://review.opendev.org/c/openstack/nova-specs/+/791047/2/specs/xena/approved/pci-device-tracking-in-placement.rst#152 doing the pci inventory reservation via moving the instance claim to the conductor | |
| 17:12:34 | sean-k-mooney | stephenfin: ^ also of interest to you | |
| 17:13:33 | sean-k-mooney | we have talked about moving the instance claim to the conductor for years and it will simplfy pci in placment as well as resolve some other numa issues | |
| 17:24:23 | opendevreview | Elod Illes proposed openstack/nova stable/pike: Use subqueryload() instead of joinedload() for (system_)metadata https://review.opendev.org/c/openstack/nova/+/799533 | |
| 17:36:45 | noonedeadpunk | hey there! We just relesed W and switched back to tracking master branch and found our upgrade jobs failing on Nova, after upgrade from stable/wallaby to master | |
| 17:37:27 | noonedeadpunk | with smth like http://paste.openstack.org/show/807170/ | |
| 17:38:08 | noonedeadpunk | and looking at https://opendev.org/openstack/nova/src/branch/master/nova/objects/service.py#L207-L211 it makes me wonder wtf is going on here :) | |
| 18:03:25 | lyarwood | noonedeadpunk: https://github.com/openstack/nova/blob/66fb0ecb5a867c054ab266aadcc06a940967abd4/nova/objects/service.py#L33-L34 looks like it is 53 on stable/victoria at least | |
| 18:03:37 | lyarwood | that would make sense given your error | |
| 18:03:56 | noonedeadpunk | ah, ok, yes | |
| 18:04:03 | noonedeadpunk | now it does :) | |
| 18:04:51 | lyarwood | I don't get why the alias is 52 in master however | |
| 18:04:51 | noonedeadpunk | lyarwood: so, eventually now jumping through releases is not going to work, right? So if I'd love to do V->X upgrade, it's not gonna work (like T->V did nicely) | |
| 18:05:27 | lyarwood | noonedeadpunk: we have only ever supported N to N+1 | |
| 18:05:46 | noonedeadpunk | but toher then that never failed explicitly :D | |
| 18:05:57 | lyarwood | noonedeadpunk: and even with things like FFU downstream we take everything down at N and bring everything back up at N+3 | |
| 18:06:32 | lyarwood | noonedeadpunk: yeah that was more luck than anything, there was no test coverage of anything other than N to N+1 | |
| 18:06:43 | noonedeadpunk | I think I found some mistake and in CI we were upgrading from V to master, which made this part fail | |
| 18:06:59 | noonedeadpunk | yeah, we also test only N to N+1 | |
| 18:07:51 | noonedeadpunk | btw, maybe you have some idea, would computes in N+1 work with N API? Ie if you upgrade computes first? | |
| 18:22:53 | lyarwood | Yes that should work | |
| 18:23:13 | lyarwood | you just need to pin your computes to the older rpc versions | |
| 18:23:18 | lyarwood | but it isn't the way we test upgrades | |
| 18:24:25 | sean-k-mooney | noonedeadpunk: actully it wont work | |
| 18:24:35 | sean-k-mooney | you have to upgrade the contoler first | |
| 18:25:02 | sean-k-mooney | so you always have to upgrade the api, conductor and schduler together | |
| 18:25:19 | sean-k-mooney | and they must be upgraded to N+1 before you upgade any comnputes to n+1 | |
| 18:26:00 | sean-k-mooney | while you could technially pin the rpc version and you might be able to get them to work it would be unsuppoted upstream and downstream | |
| 18:26:25 | sean-k-mooney | the compute always assume that the contoler are at least teh same version | |
| #openstack-nova - 2021-07-06 | |||
| 05:21:24 | noonedeadpunk | gotcha, thanks folks. | |
| 05:23:05 | noonedeadpunk | we were just tend to do upgrades once per year, so we usually jump through releases and that used to work nicely. But feels we would need to change that approach | |
| 07:09:55 | gibi | sean-k-mooney: re: moving the instance claim to the conductor. The idea itself make sense. I don't know how big of a surgery it is needed in the conductor / compute impementation. Also I guess due to rolling upgrade we need some way to do claim in both places for a release. | |
| 07:19:41 | gibi | gmann: we have a bug report about policies https://bugs.launchpad.net/nova/+bug/1934478 If you have time could you please check it and triage it? | |
| 07:27:22 | aarents | Hi nova! | |
| 07:27:39 | aarents | thks gibi for quick triage of https://bugs.launchpad.net/nova/+bug/1934742 I'm on the way to propose something I plan to prod downstream. | |
| 07:28:23 | gibi | aarents: ack. your suggestion about synchronizing the port delete handling with the the interface attach/detach make sense to me | |
| 08:12:26 | opendevreview | Lee Yarwood proposed openstack/nova master: zuul: Add CentOS 8 stream integrated compute tempest job to gate https://review.opendev.org/c/openstack/nova/+/797616 | |
| 08:15:07 | opendevreview | wushiming proposed openstack/placement master: Changed minversion in tox to 3.18.0 https://review.opendev.org/c/openstack/placement/+/799601 | |
| 08:28:04 | bauzas | good spec review day, folks | |
| 08:30:01 | opendevreview | Alexandre arents proposed openstack/nova master: Synchronize network-vif-deleted event processing https://review.opendev.org/c/openstack/nova/+/799606 | |
| 08:55:04 | bauzas | gibi: sean-k-mooney: do you folks agree on fast-approving a new revision for https://review.opendev.org/c/openstack/nova-specs/+/792796/3/specs/xena/approved/generic-mdevs.rst if I just fix the nits ? | |
| 08:56:16 | sean-k-mooney[m] | ill be starting soon but yes that works for me | |