| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-04-16 | |||
| 14:09:57 | mriedem | but that's super latent if so, and xen specific, | |
| 14:10:00 | mriedem | so i care little | |
| 14:10:20 | mriedem | Theo could open a bug for it but i wouldn't hold this up for it | |
| 14:10:23 | melwitt | haha, nice | |
| 14:10:25 | mnaser | is migrate_data stored in a db anywhere or is it thrown into the rpc request | |
| 14:10:34 | mriedem | mnaser: it's only rpc, not persisted | |
| 14:12:55 | melwitt | we deprecated the os-console service kind of recently https://review.openstack.org/610075 | |
| 14:12:57 | mriedem | stephenfin: the bottom of your cells v1 removal series needs to be fixed | |
| 14:13:07 | stephenfin | mriedem: I just saw :( | |
| 14:13:11 | melwitt | oh yeah, you already said that | |
| 14:13:13 | stephenfin | Looking at it now | |
| 14:13:15 | mriedem | melwitt: yeah i was looking at that - that deprecates the service, | |
| 14:13:16 | mriedem | but not the apis | |
| 14:13:23 | melwitt | ah | |
| 14:13:39 | mriedem | so if we were to remove the nova-console service the os-consoles api would be broken, | |
| 14:13:54 | mriedem | so we can't really deprecate the api with a microversion and expect it to still work on 2.1 w/o the service itself, | |
| 14:14:16 | mriedem | which would mean we either (1) leave the nova-console service forever or (2) obsolete the api and drop the service, like we're doing with nova-cells and nova-network | |
| 14:14:43 | mriedem | maybe ptg topic fodder - but it'd be nice to have xen people there to actually tell us if they need the thing anymore, BobBall seemed to suggest in the ML that we didn't | |
| 14:15:01 | mriedem | i don't know who works on or uses xenapi in nova anymore though | |
| 14:15:29 | NewBruce | hey mriedem seen the latest in our lovely migration issues ?! | |
| 14:15:34 | openstackgerrit | Stephen Finucane proposed openstack/nova-specs master: Standardize CPU resource tracking https://review.openstack.org/555081 | |
| 14:15:47 | mriedem | NewBruce: not really - can't get the neutron api extension to go away? | |
| 14:16:11 | mriedem | melwitt: i'll just add an item in the ptg etherpad | |
| 14:16:44 | NewBruce | we upgraded the entire site to rocky, and now RDO - RDO fails in addition to RDO - OSA! | |
| 14:16:54 | NewBruce | (updated the launchpad) | |
| 14:18:54 | mriedem | :/ | |
| 14:19:13 | mriedem | we definitely have multinode jobs running live migration in rocky that uses the new flow | |
| 14:19:56 | melwitt | yeah, I had thought BobBall gave the go ahead to remove it but agreed, I don't know of any people currently using xenapi. actually, I thought I've seen some patches proposed over there fairly recently. /me looks | |
| 14:21:13 | melwitt | oh, nvm, it was coreycb fixing an openssl handling thing https://review.openstack.org/635533 | |
| 14:22:57 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Stop handling cells v1 in '/os-hypervisors' API https://review.openstack.org/651292 | |
| 14:22:57 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Remove '/os-cells' REST APIs https://review.openstack.org/651291 | |
| 14:22:58 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Remove 'nova-manage cell' commands https://review.openstack.org/651294 | |
| 14:22:58 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Stop handling cells v1 in '/os-servers' API https://review.openstack.org/651293 | |
| 14:22:59 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Remove old-style cell v1 instance listing https://review.openstack.org/651296 | |
| 14:22:59 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Stop handling cells v1 for console authentication https://review.openstack.org/651295 | |
| 14:23:00 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Remove 'instance_fault_create_at_top' https://review.openstack.org/651298 | |
| 14:23:00 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Remove 'bdm_(update_or_create|destroy)_at_top' https://review.openstack.org/651297 | |
| 14:23:01 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Remove 'get_keypair_at_top' https://review.openstack.org/651300 | |
| 14:23:01 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Remove 'instance_info_cache_update_at_top' https://review.openstack.org/651299 | |
| 14:23:02 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Remove 'instance_update_from_api' https://review.openstack.org/651302 | |
| 14:23:02 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Remove 'instance_update_at_top', 'instance_destroy_at_top' https://review.openstack.org/651301 | |
| 14:23:03 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Stop handling cells v1 for instance naming https://review.openstack.org/651304 | |
| 14:23:03 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Stop handling 'update_cells' on 'BandwidthUsage.create' https://review.openstack.org/651303 | |
| 14:23:04 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Stop handling 'InstanceUnknownCell' exception https://review.openstack.org/651307 | |
| 14:23:04 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Remove cells code https://review.openstack.org/651306 | |
| 14:23:05 | openstackgerrit | Stephen Finucane proposed openstack/nova master: db: Remove cell APIs https://review.openstack.org/651309 | |
| 14:23:05 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Remove unnecessary wrapper https://review.openstack.org/651308 | |
| 14:25:21 | mriedem | NewBruce: and you've checked to make sure there aren't any lingering nova-compute services table records in the rdo site database with an older version causing issues? | |
| 14:25:25 | stephenfin | melwitt: This is something we could do later this cycle, right? https://review.openstack.org/#/q/topic:bp/remove-consoleauth | |
| 14:25:40 | mriedem | NewBruce: and unpinned rpc versions for [upgrade_levels]/compute ? | |
| 14:25:52 | NewBruce | mriedem posted the contents of the services table to the launchpad; we have 3 entries which are not 35 but all have deleted status > 0 | |
| 14:25:53 | stephenfin | melwitt: Also, this has the feel of a bugfix to me. Thoughts? https://review.openstack.org/#/c/652967/ | |
| 14:27:15 | mriedem | stephenfin: the consoleauth thing has been really confusing for people, in rocky at least, | |
| 14:27:38 | mriedem | if there is some migration that people need to do and we can check that in some automated way, to say "you shouldn't upgrade to train" that would be good before ripping the service out | |
| 14:28:08 | stephenfin | mriedem: You mean more than the nova-status check we already have? | |
| 14:28:18 | mriedem | stephenfin: also note that there is a rest api that relies on consoleauth https://developer.openstack.org/api-ref/compute/?expanded=create-remote-console-detail#create-remote-console | |
| 14:29:14 | stephenfin | Oh, I thought that was able to use the DB tokens too? I must have misread it | |
| 14:29:31 | mriedem | maybe i'm thinking of https://developer.openstack.org/api-ref/compute/?expanded=create-remote-console-detail#show-console-connection-information | |
| 14:30:01 | mriedem | ok remote-consoles is something else, nvm | |
| 14:30:03 | stephenfin | mriedem: I had a look at that and thought it was also able to use the DB tokens | |
| 14:30:18 | mriedem | i think you are https://review.openstack.org/#/c/652969/1/nova/api/openstack/compute/console_auth_tokens.py | |
| 14:31:12 | bauzas | err, I was about to update my thoughts on the cpu-resources spec :) | |
| 14:31:21 | bauzas | but then I saw a new PS :) | |
| 14:31:27 | bauzas | dammit | |
| 14:31:29 | stephenfin | bauzas: Update them away :) I'm apply them retrospectively | |
| 14:31:50 | stephenfin | It hasn't changed significantly outside of changing how we use Flavour.vcpus | |
| 14:31:52 | bauzas | stephenfin: I'm trying to wrap my head around the upgrade impact | |
| 14:32:06 | melwitt | stephenfin: hm, yeah, I guess that xvp thing was missed.. so it feels more like a bug fix | |
| 14:32:49 | stephenfin | melwitt: Cool. I can drag that out so. I imagine no one has spotted it because no one is using it (It's Xen-specific and BobBall said we could kill it) | |
| 14:33:17 | melwitt | yeah, that's what I'm thinking too as far as it not being noticed | |
| 14:33:40 | stephenfin | Alas, that was only deprecated last cycle so I guess we can't kill that too this cycle | |
| 14:34:45 | stephenfin | (Removing nova-cells, nova-network, nova-consoleauth, nova-xvpvncproxy and the placement code in one fell swoop/cycle sure would make for interesting release note reading) | |
| 14:34:46 | NewBruce | mriedem we have upgrade_levels = auto across the site, but since everything is service level 35 that shouldnt be an issue right? | |
| 14:38:39 | mriedem | NewBruce: on a call and i'd need to load all of this context back into my head | |
| 14:39:11 | mriedem | but you're talking about this check https://github.com/openstack/nova/blob/stable/rocky/nova/conductor/tasks/live_migrate.py#L51 | |
| 14:41:26 | mriedem | NewBruce: "we have 3 entries which are not 35 but all have deleted status > 0" yeah those shouldn't be included in the min version check | |
| 14:43:45 | NewBruce | mriedem ok - ive updated the launchpad, i might have also mailed you some logs at some point | |
| 14:44:35 | mnaser | oh hmm now that I think about it | |
| 14:44:43 | NewBruce | but ping me when your off the call… i will test cold migrate tonight so that we have that as an option; the fact that its failing on RDO - RDO after the rocky upgrade is at least encouraging that the problem seems to be in the rocky side | |
| 14:44:57 | mriedem | s/rocky/rdo/? | |
| 14:45:01 | mnaser | could it be possible that not all services have been restarted since everything is at rocky | |
| 14:45:14 | mriedem | osa -> osa live migration is fine right? | |
| 14:45:15 | mnaser | and the max rpc version is not rocky ? | |
| 14:45:16 | NewBruce | mnaser very possible | |
| 14:45:31 | NewBruce | havent tested OSA - OSA yet, on my todo | |
| 14:45:37 | mnaser | aren't you supposed to "SIGHUP" (which is broken right now) to get new versions of rpc stuff | |
| 14:45:42 | mriedem | yes | |
| 14:45:44 | mnaser | so really a restart | |
| 14:45:57 | mnaser | so is it possible the conductors haven't been restarted and are running older code? | |
| 14:45:59 | mriedem | does rdo sighup rather than full restart the services on upgrade? | |
| 14:46:11 | mnaser | OSA used to do sighup | |
| 14:46:14 | mnaser | till we found that bug | |
| 14:46:21 | mriedem | yeah so maybe rdo still does as well | |
| 14:47:43 | mriedem | i was never able to recreate one of the theories about the break either https://review.openstack.org/#/c/649464/ | |
| 14:48:33 | openstackgerrit | Merged openstack/nova stable/rocky: Temporarily mutate migration object in finish_revert_resize https://review.openstack.org/648691 | |
| 14:50:45 | mriedem | i.e. i'm not able to recreate the duplicate entry error in neutron when post_live_migration_at_destination updates the port's host binding | |
| 14:50:56 | mriedem | sean-k-mooney: ^ were you able to recreate that with a neutron functional test? | |
| 14:51:09 | efried | mriedem: Are you my "stable release liaison"? | |
| 14:51:15 | mriedem | sure | |
| 14:51:45 | efried | just added you to https://review.openstack.org/#/c/652868/ and https://review.openstack.org/#/c/652869/ | |