Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-04-03
14:46:12 dansmith jaypipes: we certainly have people that think they want it, but I definitely agree they're misguided :)
14:46:26 mriedem i have also told them ^
14:46:32 jaypipes I'm a bit slap-happy today, guys, you'll have to forgive me.
14:48:12 mriedem i'd be happy to drop the newton-era keypair migrations, but i think it's more complicated than just deleting that code, since we normally have something that checks to see if you have completed your homework before we just drop the migration routine and all of the compat code
14:48:19 mriedem which requires some thinking
14:48:34 dansmith well,
14:49:11 dansmith if the migration is acutely painful we could drop it without dropping the compat code and then take our time with the latter I think
14:49:18 dansmith or fix the migration to be more efficient if that's really possible
14:49:36 dansmith ISTR that since they're one-to-many and across DBs it's harder than that
14:49:45 dansmith but it's been a long time
14:49:50 jaypipes mriedem: yeah, I was pondering what that check would be other than SELECT COUNT(*) FROM instance_extra WHERE keypairs IS NULL, though.
14:50:39 mriedem there is also https://review.openstack.org/#/c/517158/ which might mean some of the migration routine is dead now
14:50:44 mriedem i.e. https://github.com/openstack/nova/blob/master/nova/objects/keypair.py#L242
14:51:29 mriedem oh nvm that just means we wouldn't hit this https://github.com/openstack/nova/blob/master/nova/objects/keypair.py#L260
14:53:02 mriedem https://github.com/openstack/nova/commit/be8242cb5a0f8396f6b8c042813847db0571df14#diff-f5877540177ee26b63552ec5f56d74fb
14:53:45 mriedem so as of ocata, the keypairs table in the cell dbs should be empty
14:53:56 mriedem and the keypair information per instance should be in the instance_extra table yeah
14:53:56 mriedem ?
14:56:59 mriedem i'm confused, if you can't upgrade to ocata while there are keypairs in the cell db https://github.com/openstack/nova/commit/be8242cb5a0f8396f6b8c042813847db0571df14#diff-f5877540177ee26b63552ec5f56d74fb then how would the 'migrate to api db' still hit anything here? https://github.com/openstack/nova/blob/master/nova/objects/keypair.py#L265
14:57:39 dansmith mriedem: I think it's not hitting anything, it's just scanning the whole table which is the problem
14:57:58 mriedem "select([func.count()]).select_from(keypairs).where( keypairs.c.deleted == 0).scalar()" is the query jaypipes just said
14:58:08 mriedem oh nvm it's not
14:59:26 mriedem based on the commit message in https://review.openstack.org/#/c/517158/ i think we probably have reasonable justification to just kill that 'migrate to api db' data migration from newton that is still getting run
14:59:36 jaypipes mriedem: yeah, the problematic one is the scan on instance_extra
15:00:40 mriedem i'm not sure why i didn't go further on https://review.openstack.org/#/c/517158/2/nova/cmd/manage.py and remove keypair_obj.migrate_keypairs_to_api_db - i probably just ran out of time and was doing them in order, and wanted to handle the request spec one first, which i did in rocky/stein
15:00:57 mriedem so.....i think we can probably remove migrate_keypairs_to_api_db now?
15:01:02 jaypipes mriedem: maybe you were performing a schema downgrade in production.
15:01:11 mriedem i will cut you
15:01:55 mriedem if we think we can drop this, and we're doing an rc2 for stein we might want to get off this pot and include it in rc2 if it's really causing pain for upgrades
15:02:20 mriedem but i feel like i'm trying to convince myself this is ok
15:02:35 mriedem like everything i do in nova, which comes back to bite my ass
15:04:31 dansmith seems risky for an rc2
15:04:37 dansmith not from any real data I have
15:04:49 dansmith but rc2 should be "we can't release without this" and this doesn't seem to fit that, IMHO
15:05:29 dansmith we probably need to nuke all those "added in neutron" migrations
15:07:04 mriedem i've got a local change, sec
15:07:11 dansmith only one more now I guess
15:07:20 dansmith looking at your patch from earlier there were abunch
15:07:25 mriedem yeah https://review.openstack.org/#/q/topic:remove-newton-online-compat-code+(status:open+OR+status:merged)
15:07:29 mriedem i tried
15:07:30 mriedem got tired
15:22:39 cdent Is there a concise description on the rules about migrations between AZs somewhere? cold and live, force and not force?
15:23:15 mriedem likely not, at least off the top of my head, but i could probably explain it quick
15:23:26 mriedem then we could report a docs bug to fill that in later
15:24:22 cdent I'll take that
15:24:29 mriedem tl;dr if the user creates the server with a specific AZ or https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.default_schedule_zone is not None (meaning it goes into some default AZ), then the instance is restricted to that AZ for all move operations,
15:24:38 mriedem UNLESS it is forced like with live migrate or evacuate
15:25:17 mriedem which, by default, with OSC you get a forced live migration every time
15:25:25 mriedem so very easy to shoot yourself in the foot there as an admin
15:25:42 mriedem which is why there are at least 5 patches to osc to fix that, listed at L18 here https://etherpad.openstack.org/p/DEN-osc-compute-api-gaps
15:26:00 cdent \o/
15:26:56 cdent are there any restrictions that distinguish the default AZ as special with regard to other AZs?
15:27:05 mriedem docs on migrate caveats regarding AZs could probably live here https://docs.openstack.org/nova/latest/user/aggregates.html#availability-zones-azs if you want to report a bug and i can wordsmith it later
15:27:11 mriedem no
15:27:31 mriedem if DEFAULT.default_schedule_zone is not None and the user doesn't request an AZ explicitly, it's treated as if the user did request the default AZ
15:27:43 cdent I guess if you specify that a sever is in foo-AZ (non-default) then you can't cold migrate from there to any other `one?
15:28:06 mriedem that happens here https://github.com/openstack/nova/blob/357da989c194a8b59842629cb64b2809143a4eae/nova/api/openstack/compute/servers.py#L641
15:28:19 mriedem cdent: correct
15:28:40 cdent okay. thanks. I'll make a bug, summarize this stuff there.
15:28:48 mriedem note there is a spec proposed to allow pasing a new AZ on unshelve, which i think is reasonable
15:30:11 mriedem [cinder]/cross_az_attach=False also makes this all more complicated....
15:30:35 mriedem ala https://review.openstack.org/#/c/469675/ wee
15:31:10 mriedem tl;dr if the cloud is configured for [cinder]/cross_az_attach=False and you boot from volume where the volume is in a non-default zone, server create explodes immediately
15:31:27 mriedem *and you don't create the server in the same zone
15:31:36 mriedem sorrison suffers from ^
15:32:05 mriedem side effects include chronic vegimitis
15:32:12 dansmith gdi mriedem, save your depressing stuff for mondays will you? wednesday is supposed to be cresting the hill of depression and heading down to friday happiness
15:32:54 mriedem being constantly depressed about bugs from essex still being in our code is all i have to live for anymore
15:34:37 mriedem not to mention all of my software skills are stuck in 6 years ago and i'm not learning anything new as a developer...wahwah
15:35:59 artom You're learning new soft skills
15:36:08 artom Like... how to deal with depression ^_^
15:37:38 dansmith nice
15:40:48 mriedem that'll be good on my next interview
15:40:48 openstackgerrit Matt Riedemann proposed openstack/nova master: Drop migrate_keypairs_to_api_db data migration https://review.openstack.org/649648
15:40:58 mriedem dansmith: jaypipes: mnaser: ^ there you go
15:43:13 openstackgerrit Matt Riedemann proposed openstack/nova master: Drop migrate_keypairs_to_api_db data migration https://review.openstack.org/649648
15:45:42 mriedem cdent: heh sound familiar? https://bugs.launchpad.net/nova/+bug/1822986
15:45:43 openstack Launchpad bug 1822986 in OpenStack Compute (nova) "Not clear if www_authenticate_uri is really needed" [Undecided,New]
15:45:55 cdent oh hai
15:46:20 cdent hmm, yes, rather familiar
15:51:09 melwitt mriedem, efried: I thought I was coordinating RC2, but if someone else wants to, that's fine with me. I put the things I thought should go into it on https://etherpad.openstack.org/p/nova-stein-rc-potential last night
15:52:44 mriedem melwitt: i think those are the same two changes that are being put in, so it's aligned
15:53:14 melwitt ok, cool
16:00:01 jaypipes mriedem: ++
16:06:09 melwitt mriedem, efried: fyi I've proposed RC2 at https://review.openstack.org/649656
16:19:00 efried mriedem: was away for a bit there. Did you decide to do something about SIGHUP for RC2?
16:19:55 mriedem efried: no
16:20:05 mriedem i went on a online data migrations tangent
16:20:24 efried ight
16:21:23 mriedem kashyap: MIN_LIBVIRT_VERSION = (3, 0, 0) and MIN_LIBVIRT_POSTCOPY_VERSION = (1, 3, 3), do you have a change to remove MIN_LIBVIRT_POSTCOPY_VERSION?
16:24:24 mriedem doesn't look like it
16:34:43 mriedem NewBruce: what do you have set for live_migration_permit_post_copy in nova.conf on your compute services?
16:38:51 openstackgerrit Matt Riedemann proposed openstack/nova master: libvirt: drop MIN_LIBVIRT_POSTCOPY_VERSION https://review.openstack.org/649671
16:50:39 openstackgerrit Matt Riedemann proposed openstack/nova master: libvirt: remove conditional on VIR_DOMAIN_EVENT_SUSPENDED_POSTCOPY https://review.openstack.org/649674
16:52:07 mriedem sean-k-mooney: so on https://review.openstack.org/#/c/649464/ we're never getting a post-copy event during live migration so we don't activate the dest host port binding prior to post_live_migration_at_destination, so that kind of throws that theory out the window as being the issue NewBruce is hitting
16:52:13 openstackgerrit Eric Fried proposed openstack/nova stable/rocky: Fix bug preventing forbidden traits from working https://review.openstack.org/649603
16:58:49 sean-k-mooney mriedem: ya i noticed that this morning that it passed
17:00:07 sean-k-mooney i was wondering if we shoudl maybe try and force the compute level to be different on the contoler vs the compute and see if that triggers the issue
17:00:53 sean-k-mooney mriedem: were you going to resubmit with post copy enabled?
17:02:16 mriedem sean-k-mooney: yeah i was going to try enabling post-copy but these cirros guests are so tiny i'm not sure it will mean anything

Earlier   Later