Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-04-16
14:31:27 bauzas dammit
14:31:29 stephenfin bauzas: Update them away :) I'm apply them retrospectively
14:31:50 stephenfin It hasn't changed significantly outside of changing how we use Flavour.vcpus
14:31:52 bauzas stephenfin: I'm trying to wrap my head around the upgrade impact
14:32:06 melwitt stephenfin: hm, yeah, I guess that xvp thing was missed.. so it feels more like a bug fix
14:32:49 stephenfin melwitt: Cool. I can drag that out so. I imagine no one has spotted it because no one is using it (It's Xen-specific and BobBall said we could kill it)
14:33:17 melwitt yeah, that's what I'm thinking too as far as it not being noticed
14:33:40 stephenfin Alas, that was only deprecated last cycle so I guess we can't kill that too this cycle
14:34:45 stephenfin (Removing nova-cells, nova-network, nova-consoleauth, nova-xvpvncproxy and the placement code in one fell swoop/cycle sure would make for interesting release note reading)
14:34:46 NewBruce mriedem we have upgrade_levels = auto across the site, but since everything is service level 35 that shouldnt be an issue right?
14:38:39 mriedem NewBruce: on a call and i'd need to load all of this context back into my head
14:39:11 mriedem but you're talking about this check https://github.com/openstack/nova/blob/stable/rocky/nova/conductor/tasks/live_migrate.py#L51
14:41:26 mriedem NewBruce: "we have 3 entries which are not 35 but all have deleted status > 0" yeah those shouldn't be included in the min version check
14:43:45 NewBruce mriedem ok - ive updated the launchpad, i might have also mailed you some logs at some point
14:44:35 mnaser oh hmm now that I think about it
14:44:43 NewBruce but ping me when your off the call… i will test cold migrate tonight so that we have that as an option; the fact that its failing on RDO - RDO after the rocky upgrade is at least encouraging that the problem seems to be in the rocky side
14:44:57 mriedem s/rocky/rdo/?
14:45:01 mnaser could it be possible that not all services have been restarted since everything is at rocky
14:45:14 mriedem osa -> osa live migration is fine right?
14:45:15 mnaser and the max rpc version is not rocky ?
14:45:16 NewBruce mnaser very possible
14:45:31 NewBruce havent tested OSA - OSA yet, on my todo
14:45:37 mnaser aren't you supposed to "SIGHUP" (which is broken right now) to get new versions of rpc stuff
14:45:42 mriedem yes
14:45:44 mnaser so really a restart
14:45:57 mnaser so is it possible the conductors haven't been restarted and are running older code?
14:45:59 mriedem does rdo sighup rather than full restart the services on upgrade?
14:46:11 mnaser OSA used to do sighup
14:46:14 mnaser till we found that bug
14:46:21 mriedem yeah so maybe rdo still does as well
14:47:43 mriedem i was never able to recreate one of the theories about the break either https://review.openstack.org/#/c/649464/
14:48:33 openstackgerrit Merged openstack/nova stable/rocky: Temporarily mutate migration object in finish_revert_resize https://review.openstack.org/648691
14:50:45 mriedem i.e. i'm not able to recreate the duplicate entry error in neutron when post_live_migration_at_destination updates the port's host binding
14:50:56 mriedem sean-k-mooney: ^ were you able to recreate that with a neutron functional test?
14:51:09 efried mriedem: Are you my "stable release liaison"?
14:51:15 mriedem sure
14:51:45 efried just added you to https://review.openstack.org/#/c/652868/ and https://review.openstack.org/#/c/652869/
14:51:59 efried It is not clear to me how much of https://review.openstack.org/#/q/status:open+(project:openstack/os-vif+OR+project:openstack/python-novaclient+OR+project:openstack/nova)+branch:stable/pike we need to merge before those ^ are a go.
14:52:22 efried and/or if more things need to be flushed from stein->rocky->queens->pike first
14:53:11 sean-k-mooney mriedem: no although it kind of fell of my plate. i can try again however.
14:53:30 mriedem efried: i've -1ed with a comment
14:53:34 efried thank you sir
14:53:57 sean-k-mooney mriedem: the main issue was figuring out how to test that withing neutron exsiting test suite
14:54:15 mnaser NewBruce: how long has this system been up for?
14:54:24 mnaser esp the nova ctlplane processes
14:55:04 NewBruce we had a maintenance window maybe a month ago when everything was shutdown
14:55:28 mnaser there isn't a way to get the current "highest" level of detected rpc version eh
14:55:28 NewBruce so that would have been the latest - certainly quite some time before the rest of the computes were upgrade
14:58:05 mriedem mnaser: i've thought about exposing the service version in the api and/or a nova-manage command, but the problem is the services will cache the version so an api/cli could tell you you're at the highest but a service could be running with a lower version in its cache
14:58:37 NewBruce is it worth restarting everything to remove this as a possibility?
14:58:54 mriedem it looks like if you sighup nova-conductor the only thing it will do is reset that cache
14:59:16 mriedem https://github.com/openstack/nova/blob/stable/rocky/nova/conductor/manager.py#L188
15:00:59 mordred mriedem: you know everything ...
15:01:16 mordred mriedem: https://wiki.openstack.org/wiki/VirtDriverImageProperties - are those documented anywhere other than that wiki page?
15:01:28 mriedem https://docs.openstack.org/glance/latest/admin/useful-image-properties.html ?
15:01:45 mriedem stephenfin: the table format in ^ is bad now
15:01:47 mriedem is that a known issue?
15:02:00 mriedem that used to actually have grid lines like a .... table
15:02:02 mordred oh! thanks
15:02:32 stephenfin mriedem: kashyap spotted that a few days ago. It's a style thing done by openstackdocstheme
15:03:07 sean-k-mooney mordred: they are ment to be documened in teh glance metadef too but several are missing
15:03:12 kashyap stephenfin: What is "that"? The "if you reference the same ref twice you'll see funny rendering"?
15:03:26 mriedem mordred: there are some missing from that docs page too - like sean-k-mooney said for metadefs
15:03:28 stephenfin kashyap: The lack of borders on tables
15:03:34 sean-k-mooney mordred: stephenfin is working on a better way to defien and validate imave properties and flavor extra specs
15:03:41 mriedem i try to report glance bugs when i find missing properties in nova code
15:03:45 kashyap stephenfin: Ah, I missed to read the context earlier
15:03:52 mriedem https://bugs.launchpad.net/glance/+bug/1811897
15:03:53 openstack Launchpad bug 1811897 in Glance "Useful image properties in glance - hw_disk_bus is also used by the vmware driver" [Undecided,New]
15:03:57 mriedem https://bugs.launchpad.net/glance/+bug/1808868
15:03:58 openstack Launchpad bug 1808868 in Glance "Useful image properties in glance - hw_cdrom_bus is not documented" [Medium,Confirmed]
15:03:59 mordred sean-k-mooney: I would support anything that improves a better way to define and validate image properties :)
15:04:02 mriedem https://bugs.launchpad.net/glance/+bugs?search=Search&field.bug_reporter=mriedem&orderby=-datecreated&start=0 etc
15:04:28 sean-k-mooney mordred: in theroy they should all be defined here https://github.com/openstack/glance/tree/master/etc/metadefs
15:04:39 mordred trait:<trait_name> = required is a really special interfae
15:05:06 sean-k-mooney but it has not been maintained for all new specs and across all drivers
15:05:21 mordred sean-k-mooney: of course it hasn't :)
15:05:53 sean-k-mooney mordred: well we never added any testing to enforce it so it never will be.
15:06:03 mriedem i found out the other day that azure has a completely undocumented templating rest api and i was pretty surprised and somehow happy that even a giant closed source thing like azure has poor documentation
15:06:39 mriedem sean-k-mooney: core reviewers in nova can certainly say "i'm not going to approve your shiny nugget until i see the glance docs patch written"
15:06:48 mordred also - fwiw - in the docs, it says auto_disk_config should return true of falase
15:06:57 mordred and on rackspace it returned "disabled"
15:07:03 mordred so - you know - there's that
15:07:23 sean-k-mooney mriedem: true we have mentioned that for some of the recent windriver ones like vTPM
15:07:41 sean-k-mooney its a relitvly tirival change if you do it when you add the feature
15:07:42 mriedem mordred: auto_disk_config is a string field in the nova schema so it can be whatever as far as nova is concerned
15:07:56 mordred mriedem: awesome
15:08:22 mnaser mriedem: I'd probably avoid suggesting SIGHUP-ing things for now till we figure out the oslo.service stuff, but yeah, maybe worth restarting nova-conductor to reset its cache I guess
15:08:28 mriedem it does look like the xen driver tries to treat it as a bool though
15:08:40 sean-k-mooney mordred: you shoudl review https://review.openstack.org/#/c/638734/
15:08:49 mriedem like other booleans in the openstack rest api like 1/yes/true/True etc
15:09:22 mordred are _all_ of the extra properties liek that string fields?
15:09:27 mriedem mordred: btw, i think i kind of have a monopoly on depressing topics in this channel and i will fight you over territory
15:09:54 mordred mriedem: I will not fight back - I concede your supremacy in depressing topics in this channel
15:10:02 mriedem mordred: not all https://github.com/openstack/nova/blob/master/nova/objects/image_meta.py#L233
15:10:04 mordred I claim that monopoly in #openstack-sdks
15:10:35 mriedem everything in ^ should be documented in https://docs.openstack.org/glance/latest/admin/useful-image-properties.html and in glance metadefs
15:10:38 sean-k-mooney mordred: wehre there is a finite set we restict it but where there is not we dont
15:10:47 mriedem anything not in https://github.com/openstack/nova/blob/master/nova/objects/image_meta.py#L233 will fail in nova (unless you've forked nova)
15:11:11 sean-k-mooney mriedem: well if the key does not match anything in that we will allow it

Earlier   Later