Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-04-16
14:25:25 stephenfin melwitt: This is something we could do later this cycle, right? https://review.openstack.org/#/q/topic:bp/remove-consoleauth
14:25:40 mriedem NewBruce: and unpinned rpc versions for [upgrade_levels]/compute ?
14:25:52 NewBruce mriedem posted the contents of the services table to the launchpad; we have 3 entries which are not 35 but all have deleted status > 0
14:25:53 stephenfin melwitt: Also, this has the feel of a bugfix to me. Thoughts? https://review.openstack.org/#/c/652967/
14:27:15 mriedem stephenfin: the consoleauth thing has been really confusing for people, in rocky at least,
14:27:38 mriedem if there is some migration that people need to do and we can check that in some automated way, to say "you shouldn't upgrade to train" that would be good before ripping the service out
14:28:08 stephenfin mriedem: You mean more than the nova-status check we already have?
14:28:18 mriedem stephenfin: also note that there is a rest api that relies on consoleauth https://developer.openstack.org/api-ref/compute/?expanded=create-remote-console-detail#create-remote-console
14:29:14 stephenfin Oh, I thought that was able to use the DB tokens too? I must have misread it
14:29:31 mriedem maybe i'm thinking of https://developer.openstack.org/api-ref/compute/?expanded=create-remote-console-detail#show-console-connection-information
14:30:01 mriedem ok remote-consoles is something else, nvm
14:30:03 stephenfin mriedem: I had a look at that and thought it was also able to use the DB tokens
14:30:18 mriedem i think you are https://review.openstack.org/#/c/652969/1/nova/api/openstack/compute/console_auth_tokens.py
14:31:12 bauzas err, I was about to update my thoughts on the cpu-resources spec :)
14:31:21 bauzas but then I saw a new PS :)
14:31:27 bauzas dammit
14:31:29 stephenfin bauzas: Update them away :) I'm apply them retrospectively
14:31:50 stephenfin It hasn't changed significantly outside of changing how we use Flavour.vcpus
14:31:52 bauzas stephenfin: I'm trying to wrap my head around the upgrade impact
14:32:06 melwitt stephenfin: hm, yeah, I guess that xvp thing was missed.. so it feels more like a bug fix
14:32:49 stephenfin melwitt: Cool. I can drag that out so. I imagine no one has spotted it because no one is using it (It's Xen-specific and BobBall said we could kill it)
14:33:17 melwitt yeah, that's what I'm thinking too as far as it not being noticed
14:33:40 stephenfin Alas, that was only deprecated last cycle so I guess we can't kill that too this cycle
14:34:45 stephenfin (Removing nova-cells, nova-network, nova-consoleauth, nova-xvpvncproxy and the placement code in one fell swoop/cycle sure would make for interesting release note reading)
14:34:46 NewBruce mriedem we have upgrade_levels = auto across the site, but since everything is service level 35 that shouldnt be an issue right?
14:38:39 mriedem NewBruce: on a call and i'd need to load all of this context back into my head
14:39:11 mriedem but you're talking about this check https://github.com/openstack/nova/blob/stable/rocky/nova/conductor/tasks/live_migrate.py#L51
14:41:26 mriedem NewBruce: "we have 3 entries which are not 35 but all have deleted status > 0" yeah those shouldn't be included in the min version check
14:43:45 NewBruce mriedem ok - ive updated the launchpad, i might have also mailed you some logs at some point
14:44:35 mnaser oh hmm now that I think about it
14:44:43 NewBruce but ping me when your off the call… i will test cold migrate tonight so that we have that as an option; the fact that its failing on RDO - RDO after the rocky upgrade is at least encouraging that the problem seems to be in the rocky side
14:44:57 mriedem s/rocky/rdo/?
14:45:01 mnaser could it be possible that not all services have been restarted since everything is at rocky
14:45:14 mriedem osa -> osa live migration is fine right?
14:45:15 mnaser and the max rpc version is not rocky ?
14:45:16 NewBruce mnaser very possible
14:45:31 NewBruce havent tested OSA - OSA yet, on my todo
14:45:37 mnaser aren't you supposed to "SIGHUP" (which is broken right now) to get new versions of rpc stuff
14:45:42 mriedem yes
14:45:44 mnaser so really a restart
14:45:57 mnaser so is it possible the conductors haven't been restarted and are running older code?
14:45:59 mriedem does rdo sighup rather than full restart the services on upgrade?
14:46:11 mnaser OSA used to do sighup
14:46:14 mnaser till we found that bug
14:46:21 mriedem yeah so maybe rdo still does as well
14:47:43 mriedem i was never able to recreate one of the theories about the break either https://review.openstack.org/#/c/649464/
14:48:33 openstackgerrit Merged openstack/nova stable/rocky: Temporarily mutate migration object in finish_revert_resize https://review.openstack.org/648691
14:50:45 mriedem i.e. i'm not able to recreate the duplicate entry error in neutron when post_live_migration_at_destination updates the port's host binding
14:50:56 mriedem sean-k-mooney: ^ were you able to recreate that with a neutron functional test?
14:51:09 efried mriedem: Are you my "stable release liaison"?
14:51:15 mriedem sure
14:51:45 efried just added you to https://review.openstack.org/#/c/652868/ and https://review.openstack.org/#/c/652869/
14:51:59 efried It is not clear to me how much of https://review.openstack.org/#/q/status:open+(project:openstack/os-vif+OR+project:openstack/python-novaclient+OR+project:openstack/nova)+branch:stable/pike we need to merge before those ^ are a go.
14:52:22 efried and/or if more things need to be flushed from stein->rocky->queens->pike first
14:53:11 sean-k-mooney mriedem: no although it kind of fell of my plate. i can try again however.
14:53:30 mriedem efried: i've -1ed with a comment
14:53:34 efried thank you sir
14:53:57 sean-k-mooney mriedem: the main issue was figuring out how to test that withing neutron exsiting test suite
14:54:15 mnaser NewBruce: how long has this system been up for?
14:54:24 mnaser esp the nova ctlplane processes
14:55:04 NewBruce we had a maintenance window maybe a month ago when everything was shutdown
14:55:28 NewBruce so that would have been the latest - certainly quite some time before the rest of the computes were upgrade
14:55:28 mnaser there isn't a way to get the current "highest" level of detected rpc version eh
14:58:05 mriedem mnaser: i've thought about exposing the service version in the api and/or a nova-manage command, but the problem is the services will cache the version so an api/cli could tell you you're at the highest but a service could be running with a lower version in its cache
14:58:37 NewBruce is it worth restarting everything to remove this as a possibility?
14:58:54 mriedem it looks like if you sighup nova-conductor the only thing it will do is reset that cache
14:59:16 mriedem https://github.com/openstack/nova/blob/stable/rocky/nova/conductor/manager.py#L188
15:00:59 mordred mriedem: you know everything ...
15:01:16 mordred mriedem: https://wiki.openstack.org/wiki/VirtDriverImageProperties - are those documented anywhere other than that wiki page?
15:01:28 mriedem https://docs.openstack.org/glance/latest/admin/useful-image-properties.html ?
15:01:45 mriedem stephenfin: the table format in ^ is bad now
15:01:47 mriedem is that a known issue?
15:02:00 mriedem that used to actually have grid lines like a .... table
15:02:02 mordred oh! thanks
15:02:32 stephenfin mriedem: kashyap spotted that a few days ago. It's a style thing done by openstackdocstheme
15:03:07 sean-k-mooney mordred: they are ment to be documened in teh glance metadef too but several are missing
15:03:12 kashyap stephenfin: What is "that"? The "if you reference the same ref twice you'll see funny rendering"?
15:03:26 mriedem mordred: there are some missing from that docs page too - like sean-k-mooney said for metadefs
15:03:28 stephenfin kashyap: The lack of borders on tables
15:03:34 sean-k-mooney mordred: stephenfin is working on a better way to defien and validate imave properties and flavor extra specs
15:03:41 mriedem i try to report glance bugs when i find missing properties in nova code
15:03:45 kashyap stephenfin: Ah, I missed to read the context earlier
15:03:52 mriedem https://bugs.launchpad.net/glance/+bug/1811897
15:03:53 openstack Launchpad bug 1811897 in Glance "Useful image properties in glance - hw_disk_bus is also used by the vmware driver" [Undecided,New]
15:03:57 mriedem https://bugs.launchpad.net/glance/+bug/1808868
15:03:58 openstack Launchpad bug 1808868 in Glance "Useful image properties in glance - hw_cdrom_bus is not documented" [Medium,Confirmed]
15:03:59 mordred sean-k-mooney: I would support anything that improves a better way to define and validate image properties :)
15:04:02 mriedem https://bugs.launchpad.net/glance/+bugs?search=Search&field.bug_reporter=mriedem&orderby=-datecreated&start=0 etc
15:04:28 sean-k-mooney mordred: in theroy they should all be defined here https://github.com/openstack/glance/tree/master/etc/metadefs
15:04:39 mordred trait:<trait_name> = required is a really special interfae
15:05:06 sean-k-mooney but it has not been maintained for all new specs and across all drivers
15:05:21 mordred sean-k-mooney: of course it hasn't :)
15:05:53 sean-k-mooney mordred: well we never added any testing to enforce it so it never will be.
15:06:03 mriedem i found out the other day that azure has a completely undocumented templating rest api and i was pretty surprised and somehow happy that even a giant closed source thing like azure has poor documentation
15:06:39 mriedem sean-k-mooney: core reviewers in nova can certainly say "i'm not going to approve your shiny nugget until i see the glance docs patch written"
15:06:48 mordred also - fwiw - in the docs, it says auto_disk_config should return true of falase
15:06:57 mordred and on rackspace it returned "disabled"
15:07:03 mordred so - you know - there's that
15:07:23 sean-k-mooney mriedem: true we have mentioned that for some of the recent windriver ones like vTPM
15:07:41 sean-k-mooney its a relitvly tirival change if you do it when you add the feature

Earlier   Later