Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-04-03
15:27:05 mriedem docs on migrate caveats regarding AZs could probably live here https://docs.openstack.org/nova/latest/user/aggregates.html#availability-zones-azs if you want to report a bug and i can wordsmith it later
15:27:11 mriedem no
15:27:31 mriedem if DEFAULT.default_schedule_zone is not None and the user doesn't request an AZ explicitly, it's treated as if the user did request the default AZ
15:27:43 cdent I guess if you specify that a sever is in foo-AZ (non-default) then you can't cold migrate from there to any other `one?
15:28:06 mriedem that happens here https://github.com/openstack/nova/blob/357da989c194a8b59842629cb64b2809143a4eae/nova/api/openstack/compute/servers.py#L641
15:28:19 mriedem cdent: correct
15:28:40 cdent okay. thanks. I'll make a bug, summarize this stuff there.
15:28:48 mriedem note there is a spec proposed to allow pasing a new AZ on unshelve, which i think is reasonable
15:30:11 mriedem [cinder]/cross_az_attach=False also makes this all more complicated....
15:30:35 mriedem ala https://review.openstack.org/#/c/469675/ wee
15:31:10 mriedem tl;dr if the cloud is configured for [cinder]/cross_az_attach=False and you boot from volume where the volume is in a non-default zone, server create explodes immediately
15:31:27 mriedem *and you don't create the server in the same zone
15:31:36 mriedem sorrison suffers from ^
15:32:05 mriedem side effects include chronic vegimitis
15:32:12 dansmith gdi mriedem, save your depressing stuff for mondays will you? wednesday is supposed to be cresting the hill of depression and heading down to friday happiness
15:32:54 mriedem being constantly depressed about bugs from essex still being in our code is all i have to live for anymore
15:34:37 mriedem not to mention all of my software skills are stuck in 6 years ago and i'm not learning anything new as a developer...wahwah
15:35:59 artom You're learning new soft skills
15:36:08 artom Like... how to deal with depression ^_^
15:37:38 dansmith nice
15:40:48 mriedem that'll be good on my next interview
15:40:48 openstackgerrit Matt Riedemann proposed openstack/nova master: Drop migrate_keypairs_to_api_db data migration https://review.openstack.org/649648
15:40:58 mriedem dansmith: jaypipes: mnaser: ^ there you go
15:43:13 openstackgerrit Matt Riedemann proposed openstack/nova master: Drop migrate_keypairs_to_api_db data migration https://review.openstack.org/649648
15:45:42 mriedem cdent: heh sound familiar? https://bugs.launchpad.net/nova/+bug/1822986
15:45:43 openstack Launchpad bug 1822986 in OpenStack Compute (nova) "Not clear if www_authenticate_uri is really needed" [Undecided,New]
15:45:55 cdent oh hai
15:46:20 cdent hmm, yes, rather familiar
15:51:09 melwitt mriedem, efried: I thought I was coordinating RC2, but if someone else wants to, that's fine with me. I put the things I thought should go into it on https://etherpad.openstack.org/p/nova-stein-rc-potential last night
15:52:44 mriedem melwitt: i think those are the same two changes that are being put in, so it's aligned
15:53:14 melwitt ok, cool
16:00:01 jaypipes mriedem: ++
16:06:09 melwitt mriedem, efried: fyi I've proposed RC2 at https://review.openstack.org/649656
16:19:00 efried mriedem: was away for a bit there. Did you decide to do something about SIGHUP for RC2?
16:19:55 mriedem efried: no
16:20:05 mriedem i went on a online data migrations tangent
16:20:24 efried ight
16:21:23 mriedem kashyap: MIN_LIBVIRT_VERSION = (3, 0, 0) and MIN_LIBVIRT_POSTCOPY_VERSION = (1, 3, 3), do you have a change to remove MIN_LIBVIRT_POSTCOPY_VERSION?
16:24:24 mriedem doesn't look like it
16:34:43 mriedem NewBruce: what do you have set for live_migration_permit_post_copy in nova.conf on your compute services?
16:38:51 openstackgerrit Matt Riedemann proposed openstack/nova master: libvirt: drop MIN_LIBVIRT_POSTCOPY_VERSION https://review.openstack.org/649671
16:50:39 openstackgerrit Matt Riedemann proposed openstack/nova master: libvirt: remove conditional on VIR_DOMAIN_EVENT_SUSPENDED_POSTCOPY https://review.openstack.org/649674
16:52:07 mriedem sean-k-mooney: so on https://review.openstack.org/#/c/649464/ we're never getting a post-copy event during live migration so we don't activate the dest host port binding prior to post_live_migration_at_destination, so that kind of throws that theory out the window as being the issue NewBruce is hitting
16:52:13 openstackgerrit Eric Fried proposed openstack/nova stable/rocky: Fix bug preventing forbidden traits from working https://review.openstack.org/649603
16:58:49 sean-k-mooney mriedem: ya i noticed that this morning that it passed
17:00:07 sean-k-mooney i was wondering if we shoudl maybe try and force the compute level to be different on the contoler vs the compute and see if that triggers the issue
17:00:53 sean-k-mooney mriedem: were you going to resubmit with post copy enabled?
17:02:16 mriedem sean-k-mooney: yeah i was going to try enabling post-copy but these cirros guests are so tiny i'm not sure it will mean anything
17:02:37 mriedem efried: dansmith: on that SIGHUP n-cpu issue, i tried that in a train devstack created this morning and i don't even get to the wait for network event part
17:02:38 mriedem http://paste.openstack.org/show/748820/
17:02:54 mriedem libvirt blows up in privsep
17:03:16 sean-k-mooney mriedem: i might try and create a neutron fullstack test to reproduce the nutron db error
17:03:21 mriedem this is how i hup'ed: sudo systemctl kill -s HUP devstack@n-cpu.service
17:03:27 dansmith mriedem: what happens if you try it again?
17:03:37 mriedem the server create? or the hup?
17:03:42 dansmith mriedem: the create
17:03:55 mriedem same thing
17:03:58 dansmith okay
17:04:06 dansmith I have not seen it manifest that way
17:04:26 dansmith obviously that seems like a much bigger deal
17:04:36 mriedem unplug on cleaning up from the failure also then blows up b/c privsep http://paste.openstack.org/show/748821/
17:04:48 dansmith which should be more evidence that the oslo behavior is completely wrong
17:06:10 mriedem did a full systemctl restart and created a server and it was fine as expected
17:06:24 dansmith I forget, but can people still choose to run privsep things via rootwrap so we can restart the daemon?
17:06:36 dansmith because if so, maybe that's why you see that and it's devstack default config or whatever
17:06:56 dansmith maybe I could get mikal to yell at us over twitter to answer that
17:07:25 mriedem idk, privsep config et al is something i haven't had to look at in years
17:07:35 mriedem but was always confusing to me
17:07:37 dansmith yeah
17:07:55 dansmith or maybe it's different with some of the privsep things we merged recently,
17:08:05 dansmith but people reporting it are missing more of the privsepification
17:08:44 dansmith did you create a server before the hup too?
17:12:13 mriedem yup
17:12:35 mriedem created test1, fine, sighup'ed, create test2 fails, create test3 fails, restart n-cpu, create test4 ok
17:14:33 mriedem commented 34 and 35 fwiw in bug https://bugs.launchpad.net/nova/+bug/1715374
17:14:34 openstack Launchpad bug 1715374 in OpenStack Compute (nova) "Reloading compute with SIGHUP prevents instances from booting" [High,In progress] - Assigned to Ralf Haferkamp (rhafer)
17:14:52 dansmith okay just trying to think of reasons you see it differently than reported, but probably because of recent changes I guess
17:18:21 mriedem the privsep-helper child processes are definitely gone after the SIGHUP http://paste.openstack.org/show/748822/
17:19:26 dansmith so we probably just didn't notice before or something as fewer things used it
17:19:37 dansmith but I don't think there's any reason that they're being killed now and not before
17:20:48 mriedem yeah, this is what i see in the n-cpu logs on the HUP http://paste.openstack.org/show/748823/
17:21:08 mriedem note the
17:21:09 mriedem Apr 03 17:15:28 train nova-compute[19990]: DEBUG oslo_privsep.comm [-] EOF on privsep read channel {{(pid=19990) _reader_main /usr/local/lib/python2.7/dist-packages/oslo
17:21:16 dansmith yep
17:21:29 dansmith and that we're calling the sighup handler, but then restarting anyway
17:21:32 dansmith soft restart
17:21:54 dansmith so you might be able to do something to the global privsep state to cause it to be respawned again after the restart, but I dunno how that works
17:22:02 dansmith it's just all pretty wrong
17:24:40 mriedem not much interesting in the unit either http://paste.openstack.org/show/748824/
17:29:32 mriedem yeah i'm not sure what does this http://logs.openstack.org/64/649464/1/check/tempest-full-py3/f849a52/controller/logs/screen-n-cpu.txt.gz#_Apr_02_23_01_32_822059
17:30:35 mriedem oh probably this https://github.com/openstack/nova/blob/master/nova/cmd/compute.py#L45
17:33:16 mriedem i'll throw that into ComputeManager.reset() and see what happens
17:33:44 dansmith I think that won't help, because it's before the restart
17:34:00 dansmith but worth a try I guess
17:36:48 mriedem yeah it didn't do anything
17:37:13 mriedem well i guess we can say SIGHUP of n-cpu is f'ed
17:37:35 mnaser very very f'd
17:38:05 mnaser I mean I caught this through the openstack-ansible CI which was relying on it to refresh RPCs versions after upgrades
17:38:29 mnaser but for some reason we had vif_plugging_is_fatal=false so it never broke
17:38:34 mnaser but once i got rid of that option, we resorted to restarting all agents which is not ideal really

Earlier   Later