| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-04-16 | |||
| 14:23:05 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Remove unnecessary wrapper https://review.openstack.org/651308 | |
| 14:25:21 | mriedem | NewBruce: and you've checked to make sure there aren't any lingering nova-compute services table records in the rdo site database with an older version causing issues? | |
| 14:25:25 | stephenfin | melwitt: This is something we could do later this cycle, right? https://review.openstack.org/#/q/topic:bp/remove-consoleauth | |
| 14:25:40 | mriedem | NewBruce: and unpinned rpc versions for [upgrade_levels]/compute ? | |
| 14:25:52 | NewBruce | mriedem posted the contents of the services table to the launchpad; we have 3 entries which are not 35 but all have deleted status > 0 | |
| 14:25:53 | stephenfin | melwitt: Also, this has the feel of a bugfix to me. Thoughts? https://review.openstack.org/#/c/652967/ | |
| 14:27:15 | mriedem | stephenfin: the consoleauth thing has been really confusing for people, in rocky at least, | |
| 14:27:38 | mriedem | if there is some migration that people need to do and we can check that in some automated way, to say "you shouldn't upgrade to train" that would be good before ripping the service out | |
| 14:28:08 | stephenfin | mriedem: You mean more than the nova-status check we already have? | |
| 14:28:18 | mriedem | stephenfin: also note that there is a rest api that relies on consoleauth https://developer.openstack.org/api-ref/compute/?expanded=create-remote-console-detail#create-remote-console | |
| 14:29:14 | stephenfin | Oh, I thought that was able to use the DB tokens too? I must have misread it | |
| 14:29:31 | mriedem | maybe i'm thinking of https://developer.openstack.org/api-ref/compute/?expanded=create-remote-console-detail#show-console-connection-information | |
| 14:30:01 | mriedem | ok remote-consoles is something else, nvm | |
| 14:30:03 | stephenfin | mriedem: I had a look at that and thought it was also able to use the DB tokens | |
| 14:30:18 | mriedem | i think you are https://review.openstack.org/#/c/652969/1/nova/api/openstack/compute/console_auth_tokens.py | |
| 14:31:12 | bauzas | err, I was about to update my thoughts on the cpu-resources spec :) | |
| 14:31:21 | bauzas | but then I saw a new PS :) | |
| 14:31:27 | bauzas | dammit | |
| 14:31:29 | stephenfin | bauzas: Update them away :) I'm apply them retrospectively | |
| 14:31:50 | stephenfin | It hasn't changed significantly outside of changing how we use Flavour.vcpus | |
| 14:31:52 | bauzas | stephenfin: I'm trying to wrap my head around the upgrade impact | |
| 14:32:06 | melwitt | stephenfin: hm, yeah, I guess that xvp thing was missed.. so it feels more like a bug fix | |
| 14:32:49 | stephenfin | melwitt: Cool. I can drag that out so. I imagine no one has spotted it because no one is using it (It's Xen-specific and BobBall said we could kill it) | |
| 14:33:17 | melwitt | yeah, that's what I'm thinking too as far as it not being noticed | |
| 14:33:40 | stephenfin | Alas, that was only deprecated last cycle so I guess we can't kill that too this cycle | |
| 14:34:45 | stephenfin | (Removing nova-cells, nova-network, nova-consoleauth, nova-xvpvncproxy and the placement code in one fell swoop/cycle sure would make for interesting release note reading) | |
| 14:34:46 | NewBruce | mriedem we have upgrade_levels = auto across the site, but since everything is service level 35 that shouldnt be an issue right? | |
| 14:38:39 | mriedem | NewBruce: on a call and i'd need to load all of this context back into my head | |
| 14:39:11 | mriedem | but you're talking about this check https://github.com/openstack/nova/blob/stable/rocky/nova/conductor/tasks/live_migrate.py#L51 | |
| 14:41:26 | mriedem | NewBruce: "we have 3 entries which are not 35 but all have deleted status > 0" yeah those shouldn't be included in the min version check | |
| 14:43:45 | NewBruce | mriedem ok - ive updated the launchpad, i might have also mailed you some logs at some point | |
| 14:44:35 | mnaser | oh hmm now that I think about it | |
| 14:44:43 | NewBruce | but ping me when your off the call… i will test cold migrate tonight so that we have that as an option; the fact that its failing on RDO - RDO after the rocky upgrade is at least encouraging that the problem seems to be in the rocky side | |
| 14:44:57 | mriedem | s/rocky/rdo/? | |
| 14:45:01 | mnaser | could it be possible that not all services have been restarted since everything is at rocky | |
| 14:45:14 | mriedem | osa -> osa live migration is fine right? | |
| 14:45:15 | mnaser | and the max rpc version is not rocky ? | |
| 14:45:16 | NewBruce | mnaser very possible | |
| 14:45:31 | NewBruce | havent tested OSA - OSA yet, on my todo | |
| 14:45:37 | mnaser | aren't you supposed to "SIGHUP" (which is broken right now) to get new versions of rpc stuff | |
| 14:45:42 | mriedem | yes | |
| 14:45:44 | mnaser | so really a restart | |
| 14:45:57 | mnaser | so is it possible the conductors haven't been restarted and are running older code? | |
| 14:45:59 | mriedem | does rdo sighup rather than full restart the services on upgrade? | |
| 14:46:11 | mnaser | OSA used to do sighup | |
| 14:46:14 | mnaser | till we found that bug | |
| 14:46:21 | mriedem | yeah so maybe rdo still does as well | |
| 14:47:43 | mriedem | i was never able to recreate one of the theories about the break either https://review.openstack.org/#/c/649464/ | |
| 14:48:33 | openstackgerrit | Merged openstack/nova stable/rocky: Temporarily mutate migration object in finish_revert_resize https://review.openstack.org/648691 | |
| 14:50:45 | mriedem | i.e. i'm not able to recreate the duplicate entry error in neutron when post_live_migration_at_destination updates the port's host binding | |
| 14:50:56 | mriedem | sean-k-mooney: ^ were you able to recreate that with a neutron functional test? | |
| 14:51:09 | efried | mriedem: Are you my "stable release liaison"? | |
| 14:51:15 | mriedem | sure | |
| 14:51:45 | efried | just added you to https://review.openstack.org/#/c/652868/ and https://review.openstack.org/#/c/652869/ | |
| 14:51:59 | efried | It is not clear to me how much of https://review.openstack.org/#/q/status:open+(project:openstack/os-vif+OR+project:openstack/python-novaclient+OR+project:openstack/nova)+branch:stable/pike we need to merge before those ^ are a go. | |
| 14:52:22 | efried | and/or if more things need to be flushed from stein->rocky->queens->pike first | |
| 14:53:11 | sean-k-mooney | mriedem: no although it kind of fell of my plate. i can try again however. | |
| 14:53:30 | mriedem | efried: i've -1ed with a comment | |
| 14:53:34 | efried | thank you sir | |
| 14:53:57 | sean-k-mooney | mriedem: the main issue was figuring out how to test that withing neutron exsiting test suite | |
| 14:54:15 | mnaser | NewBruce: how long has this system been up for? | |
| 14:54:24 | mnaser | esp the nova ctlplane processes | |
| 14:55:04 | NewBruce | we had a maintenance window maybe a month ago when everything was shutdown | |
| 14:55:28 | NewBruce | so that would have been the latest - certainly quite some time before the rest of the computes were upgrade | |
| 14:55:28 | mnaser | there isn't a way to get the current "highest" level of detected rpc version eh | |
| 14:58:05 | mriedem | mnaser: i've thought about exposing the service version in the api and/or a nova-manage command, but the problem is the services will cache the version so an api/cli could tell you you're at the highest but a service could be running with a lower version in its cache | |
| 14:58:37 | NewBruce | is it worth restarting everything to remove this as a possibility? | |
| 14:58:54 | mriedem | it looks like if you sighup nova-conductor the only thing it will do is reset that cache | |
| 14:59:16 | mriedem | https://github.com/openstack/nova/blob/stable/rocky/nova/conductor/manager.py#L188 | |
| 15:00:59 | mordred | mriedem: you know everything ... | |
| 15:01:16 | mordred | mriedem: https://wiki.openstack.org/wiki/VirtDriverImageProperties - are those documented anywhere other than that wiki page? | |
| 15:01:28 | mriedem | https://docs.openstack.org/glance/latest/admin/useful-image-properties.html ? | |
| 15:01:45 | mriedem | stephenfin: the table format in ^ is bad now | |
| 15:01:47 | mriedem | is that a known issue? | |
| 15:02:00 | mriedem | that used to actually have grid lines like a .... table | |
| 15:02:02 | mordred | oh! thanks | |
| 15:02:32 | stephenfin | mriedem: kashyap spotted that a few days ago. It's a style thing done by openstackdocstheme | |
| 15:03:07 | sean-k-mooney | mordred: they are ment to be documened in teh glance metadef too but several are missing | |
| 15:03:12 | kashyap | stephenfin: What is "that"? The "if you reference the same ref twice you'll see funny rendering"? | |
| 15:03:26 | mriedem | mordred: there are some missing from that docs page too - like sean-k-mooney said for metadefs | |
| 15:03:28 | stephenfin | kashyap: The lack of borders on tables | |
| 15:03:34 | sean-k-mooney | mordred: stephenfin is working on a better way to defien and validate imave properties and flavor extra specs | |
| 15:03:41 | mriedem | i try to report glance bugs when i find missing properties in nova code | |
| 15:03:45 | kashyap | stephenfin: Ah, I missed to read the context earlier | |
| 15:03:52 | mriedem | https://bugs.launchpad.net/glance/+bug/1811897 | |
| 15:03:53 | openstack | Launchpad bug 1811897 in Glance "Useful image properties in glance - hw_disk_bus is also used by the vmware driver" [Undecided,New] | |
| 15:03:57 | mriedem | https://bugs.launchpad.net/glance/+bug/1808868 | |
| 15:03:58 | openstack | Launchpad bug 1808868 in Glance "Useful image properties in glance - hw_cdrom_bus is not documented" [Medium,Confirmed] | |
| 15:03:59 | mordred | sean-k-mooney: I would support anything that improves a better way to define and validate image properties :) | |
| 15:04:02 | mriedem | https://bugs.launchpad.net/glance/+bugs?search=Search&field.bug_reporter=mriedem&orderby=-datecreated&start=0 etc | |
| 15:04:28 | sean-k-mooney | mordred: in theroy they should all be defined here https://github.com/openstack/glance/tree/master/etc/metadefs | |
| 15:04:39 | mordred | trait:<trait_name> = required is a really special interfae | |
| 15:05:06 | sean-k-mooney | but it has not been maintained for all new specs and across all drivers | |
| 15:05:21 | mordred | sean-k-mooney: of course it hasn't :) | |
| 15:05:53 | sean-k-mooney | mordred: well we never added any testing to enforce it so it never will be. | |
| 15:06:03 | mriedem | i found out the other day that azure has a completely undocumented templating rest api and i was pretty surprised and somehow happy that even a giant closed source thing like azure has poor documentation | |
| 15:06:39 | mriedem | sean-k-mooney: core reviewers in nova can certainly say "i'm not going to approve your shiny nugget until i see the glance docs patch written" | |
| 15:06:48 | mordred | also - fwiw - in the docs, it says auto_disk_config should return true of falase | |
| 15:06:57 | mordred | and on rackspace it returned "disabled" | |
| 15:07:03 | mordred | so - you know - there's that | |