| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-07-05 | |||
| 16:26:17 | opendevreview | Stephen Finucane proposed openstack/nova master: db: Post reshuffle cleanup https://review.opendev.org/c/openstack/nova/+/799526 | |
| 16:26:18 | opendevreview | Stephen Finucane proposed openstack/nova master: db: Add initial alembic migration for main DB https://review.opendev.org/c/openstack/nova/+/799527 | |
| 16:26:20 | opendevreview | Stephen Finucane proposed openstack/nova master: db: Add initial alembic migration for API DB https://review.opendev.org/c/openstack/nova/+/799528 | |
| 16:26:21 | opendevreview | Stephen Finucane proposed openstack/nova master: WIP: db: Integrate alembic https://review.opendev.org/c/openstack/nova/+/799530 | |
| 16:26:21 | opendevreview | Stephen Finucane proposed openstack/nova master: db: Trivial style changes https://review.opendev.org/c/openstack/nova/+/799529 | |
| 16:28:30 | opendevreview | Ghanshyam proposed openstack/placement master: Fix oslo policy DeprecatedRule warnings https://review.opendev.org/c/openstack/placement/+/799418 | |
| 16:29:07 | stephenfin | dansmith: If you have a bit of spare time, I'd appreciate some eyes on the last one of those patches ^ I can't quite figure out why the 'CellDatabases' fixtures is failing but it is :-\ | |
| 17:12:17 | sean-k-mooney | gibi: how do you feel about https://review.opendev.org/c/openstack/nova-specs/+/791047/2/specs/xena/approved/pci-device-tracking-in-placement.rst#152 doing the pci inventory reservation via moving the instance claim to the conductor | |
| 17:12:34 | sean-k-mooney | stephenfin: ^ also of interest to you | |
| 17:13:33 | sean-k-mooney | we have talked about moving the instance claim to the conductor for years and it will simplfy pci in placment as well as resolve some other numa issues | |
| 17:24:23 | opendevreview | Elod Illes proposed openstack/nova stable/pike: Use subqueryload() instead of joinedload() for (system_)metadata https://review.opendev.org/c/openstack/nova/+/799533 | |
| 17:36:45 | noonedeadpunk | hey there! We just relesed W and switched back to tracking master branch and found our upgrade jobs failing on Nova, after upgrade from stable/wallaby to master | |
| 17:37:27 | noonedeadpunk | with smth like http://paste.openstack.org/show/807170/ | |
| 17:38:08 | noonedeadpunk | and looking at https://opendev.org/openstack/nova/src/branch/master/nova/objects/service.py#L207-L211 it makes me wonder wtf is going on here :) | |
| 18:03:25 | lyarwood | noonedeadpunk: https://github.com/openstack/nova/blob/66fb0ecb5a867c054ab266aadcc06a940967abd4/nova/objects/service.py#L33-L34 looks like it is 53 on stable/victoria at least | |
| 18:03:37 | lyarwood | that would make sense given your error | |
| 18:03:56 | noonedeadpunk | ah, ok, yes | |
| 18:04:03 | noonedeadpunk | now it does :) | |
| 18:04:51 | lyarwood | I don't get why the alias is 52 in master however | |
| 18:04:51 | noonedeadpunk | lyarwood: so, eventually now jumping through releases is not going to work, right? So if I'd love to do V->X upgrade, it's not gonna work (like T->V did nicely) | |
| 18:05:27 | lyarwood | noonedeadpunk: we have only ever supported N to N+1 | |
| 18:05:46 | noonedeadpunk | but toher then that never failed explicitly :D | |
| 18:05:57 | lyarwood | noonedeadpunk: and even with things like FFU downstream we take everything down at N and bring everything back up at N+3 | |
| 18:06:32 | lyarwood | noonedeadpunk: yeah that was more luck than anything, there was no test coverage of anything other than N to N+1 | |
| 18:06:43 | noonedeadpunk | I think I found some mistake and in CI we were upgrading from V to master, which made this part fail | |
| 18:06:59 | noonedeadpunk | yeah, we also test only N to N+1 | |
| 18:07:51 | noonedeadpunk | btw, maybe you have some idea, would computes in N+1 work with N API? Ie if you upgrade computes first? | |
| 18:22:53 | lyarwood | Yes that should work | |
| 18:23:13 | lyarwood | you just need to pin your computes to the older rpc versions | |
| 18:23:18 | lyarwood | but it isn't the way we test upgrades | |
| 18:24:25 | sean-k-mooney | noonedeadpunk: actully it wont work | |
| 18:24:35 | sean-k-mooney | you have to upgrade the contoler first | |
| 18:25:02 | sean-k-mooney | so you always have to upgrade the api, conductor and schduler together | |
| 18:25:19 | sean-k-mooney | and they must be upgraded to N+1 before you upgade any comnputes to n+1 | |
| 18:26:00 | sean-k-mooney | while you could technially pin the rpc version and you might be able to get them to work it would be unsuppoted upstream and downstream | |
| 18:26:25 | sean-k-mooney | the compute always assume that the contoler are at least teh same version | |
| #openstack-nova - 2021-07-06 | |||
| 05:21:24 | noonedeadpunk | gotcha, thanks folks. | |
| 05:23:05 | noonedeadpunk | we were just tend to do upgrades once per year, so we usually jump through releases and that used to work nicely. But feels we would need to change that approach | |
| 07:09:55 | gibi | sean-k-mooney: re: moving the instance claim to the conductor. The idea itself make sense. I don't know how big of a surgery it is needed in the conductor / compute impementation. Also I guess due to rolling upgrade we need some way to do claim in both places for a release. | |
| 07:19:41 | gibi | gmann: we have a bug report about policies https://bugs.launchpad.net/nova/+bug/1934478 If you have time could you please check it and triage it? | |
| 07:27:22 | aarents | Hi nova! | |
| 07:27:39 | aarents | thks gibi for quick triage of https://bugs.launchpad.net/nova/+bug/1934742 I'm on the way to propose something I plan to prod downstream. | |
| 07:28:23 | gibi | aarents: ack. your suggestion about synchronizing the port delete handling with the the interface attach/detach make sense to me | |
| 08:12:26 | opendevreview | Lee Yarwood proposed openstack/nova master: zuul: Add CentOS 8 stream integrated compute tempest job to gate https://review.opendev.org/c/openstack/nova/+/797616 | |
| 08:15:07 | opendevreview | wushiming proposed openstack/placement master: Changed minversion in tox to 3.18.0 https://review.opendev.org/c/openstack/placement/+/799601 | |
| 08:28:04 | bauzas | good spec review day, folks | |
| 08:30:01 | opendevreview | Alexandre arents proposed openstack/nova master: Synchronize network-vif-deleted event processing https://review.opendev.org/c/openstack/nova/+/799606 | |
| 08:55:04 | bauzas | gibi: sean-k-mooney: do you folks agree on fast-approving a new revision for https://review.opendev.org/c/openstack/nova-specs/+/792796/3/specs/xena/approved/generic-mdevs.rst if I just fix the nits ? | |
| 08:56:16 | sean-k-mooney[m] | ill be starting soon but yes that works for me | |
| 09:02:41 | sean-k-mooney[m] | gibi: my idea for the compute claim in the conductor was to add a new rpc for the instance claim that returns true or false indicating if the claim succeeded. the side effect of the claim would be to store the claim object in a dict on the compute node indexed by the instance uuid. then we wrap the instance claim in spawn with a lookup in the dict and only do the claim if none found. | |
| 09:04:29 | sean-k-mooney[m] | in the conductor we can check the compute service version to determin if the compute is upgraded | |
| 09:16:03 | opendevreview | Sylvain Bauza proposed openstack/nova-specs master: Add generic mdevs to Nova https://review.opendev.org/c/openstack/nova-specs/+/792796 | |
| 09:16:10 | bauzas | gibi: sean-k-mooney: nits fixed ^ | |
| 09:20:09 | sean-k-mooney | looks good +2 from me. ill +w if gibi does not get to it later today | |
| 09:48:18 | bauzas | humpf, looking at https://review.opendev.org/q/project:openstack%252Fnova-specs+status:open I can't see a lot of specs needing my thoughts :( | |
| 09:53:11 | stephenfin | bauzas: that's good new, no? | |
| 09:53:13 | stephenfin | *news | |
| 09:53:27 | bauzas | stephenfin: well, maybe :) | |
| 09:53:28 | lyarwood | I'm about to drop a few today | |
| 09:53:39 | lyarwood | trying to catch up after two pretty unproductive weeks | |
| 09:53:52 | bauzas | maybe I'll only look at one spec https://review.opendev.org/c/openstack/nova-specs/+/787458 | |
| 09:53:55 | bauzas | this one is large ^ | |
| 09:54:05 | lyarwood | stephenfin: that's going to include the nova-manage commands for hardware model updates btw | |
| 09:54:14 | stephenfin | okay, sweet | |
| 10:05:13 | opendevreview | Stephen Finucane proposed openstack/nova master: db: Move remaining 'nova.db.sqlalchemy' modules https://review.opendev.org/c/openstack/nova/+/799525 | |
| 10:05:13 | opendevreview | Stephen Finucane proposed openstack/nova master: db: Unify 'nova.db.api', 'nova.db.sqlalchemy.api' https://review.opendev.org/c/openstack/nova/+/799524 | |
| 10:05:14 | opendevreview | Stephen Finucane proposed openstack/nova master: db: Add initial alembic migration for main DB https://review.opendev.org/c/openstack/nova/+/799527 | |
| 10:05:14 | opendevreview | Stephen Finucane proposed openstack/nova master: db: Post reshuffle cleanup https://review.opendev.org/c/openstack/nova/+/799526 | |
| 10:05:15 | opendevreview | Stephen Finucane proposed openstack/nova master: db: Trivial style changes https://review.opendev.org/c/openstack/nova/+/799529 | |
| 10:05:15 | opendevreview | Stephen Finucane proposed openstack/nova master: db: Add initial alembic migration for API DB https://review.opendev.org/c/openstack/nova/+/799528 | |
| 10:05:16 | opendevreview | Stephen Finucane proposed openstack/nova master: WIP: db: Integrate alembic https://review.opendev.org/c/openstack/nova/+/799530 | |
| 10:12:23 | opendevreview | Lee Yarwood proposed openstack/nova-specs master: Add nova-manage commands to show and refresh connection_info https://review.opendev.org/c/openstack/nova-specs/+/799624 | |
| 10:14:21 | opendevreview | sean mooney proposed openstack/os-vif master: update os-vif ci to account for devstack default changes https://review.opendev.org/c/openstack/os-vif/+/798038 | |
| 10:14:22 | opendevreview | sean mooney proposed openstack/os-vif master: add configurable per port bridges https://review.opendev.org/c/openstack/os-vif/+/798055 | |
| 10:22:24 | opendevreview | Merged openstack/os-resource-classes master: [doc] Redirect people to #openstack-nova https://review.opendev.org/c/openstack/os-resource-classes/+/793833 | |
| 10:41:24 | sean-k-mooney | stephenfin: care to pass an eye over the pci in placement draft https://review.opendev.org/c/openstack/nova-specs/+/791047/2/specs/xena/approved/pci-device-tracking-in-placement.rst | |
| 10:41:33 | stephenfin | already looking :) | |
| 10:41:37 | sean-k-mooney | :) | |
| 10:42:48 | sean-k-mooney | ill be reworkign it to include the "do instance claim via conductor" part later today | |
| 10:50:17 | sean-k-mooney | ill be afk for 20 mins just an fyi | |
| 11:09:13 | gibi | bauzas: +Ad the mdev spec | |
| 11:11:58 | gibi | sean-k-mooney: re: claim in conductor: so you say that the claim still happens on the compute service via a dedicated rpc call? | |
| 11:15:54 | sean-k-mooney[m] | yes via a dedicated rpc call since the instance claim need to claim things in the resource tracker on that node. to do it entirely on the conductor we would need to make other change so that the resource tracker state is entirely reflected in the db which i don’t think it currently is | |
| 11:16:10 | sean-k-mooney[m] | the pci devices and numa info should be | |
| 11:16:41 | sean-k-mooney[m] | but mdevs are not and there may be other info missing to do it entirely in the conductor | |
| 11:17:48 | sean-k-mooney[m] | we could try to confirm what the gaps would be i have not looked closely enough at what the claim is doing to confirm | |
| 11:18:01 | gibi | but if the conductor calls the compute to claim, then how is that different from the currently existing rpc cast from the conductor to the compute to do the instance lifecycle operation? | |
| 11:19:21 | sean-k-mooney[m] | well its a call for one so its synchronous, it will also happen earlier before we call any other service | |
| 11:19:57 | sean-k-mooney[m] | the claims themselves are not currently stored in the db at least not directly | |
| 11:20:37 | sean-k-mooney[m] | pci devices are claimed in the db and the instance numa topology blob contains any cpu or huge page assignments | |
| 11:21:38 | sean-k-mooney[m] | but as far as i know there is no instance claim object stored in the db at any point which the compute node could retrieve in the periodic task when its doing its resource update | |
| 11:22:34 | sean-k-mooney[m] | i could be wrong? | |
| 11:23:28 | sean-k-mooney[m] | but even for migrations ect i think the move claims only exist in memory inside the migration context | |
| 11:26:40 | gibi | so in case of a new instance build the scheduling happens here https://github.com/openstack/nova/blob/e7a7fd51d12d045ff134d55a4a1749c1feee0386/nova/conductor/manager.py#L1518 and the rpc cast to the compute is at the bottom of the function https://github.com/openstack/nova/blob/e7a7fd51d12d045ff134d55a4a1749c1feee0386/nova/conductor/manager.py#L1692 | |
| 11:26:56 | gibi | inbetween I only see external calls to cyborg to bind arqs | |
| 11:27:02 | gibi | the rest is pure python code afaik | |
| 11:28:56 | gibi | anyhow my point is that I cannot really see the gain just by looking at it in theory. | |
| 11:30:16 | gibi | I think a small PoC would be good to show that i) moving the claim to a separate rpc call is possible without blowing up both service and ii) we can make a claim a lot earlier than today by that move | |
| 11:32:18 | opendevreview | Merged openstack/nova-specs master: Add generic mdevs to Nova https://review.opendev.org/c/openstack/nova-specs/+/792796 | |
| 11:34:46 | sean-k-mooney | gibi: ack a small poc is resonable i can see if we can whip one up quickly | |