| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-04-03 | |||
| 15:28:40 | cdent | okay. thanks. I'll make a bug, summarize this stuff there. | |
| 15:28:48 | mriedem | note there is a spec proposed to allow pasing a new AZ on unshelve, which i think is reasonable | |
| 15:30:11 | mriedem | [cinder]/cross_az_attach=False also makes this all more complicated.... | |
| 15:30:35 | mriedem | ala https://review.openstack.org/#/c/469675/ wee | |
| 15:31:10 | mriedem | tl;dr if the cloud is configured for [cinder]/cross_az_attach=False and you boot from volume where the volume is in a non-default zone, server create explodes immediately | |
| 15:31:27 | mriedem | *and you don't create the server in the same zone | |
| 15:31:36 | mriedem | sorrison suffers from ^ | |
| 15:32:05 | mriedem | side effects include chronic vegimitis | |
| 15:32:12 | dansmith | gdi mriedem, save your depressing stuff for mondays will you? wednesday is supposed to be cresting the hill of depression and heading down to friday happiness | |
| 15:32:54 | mriedem | being constantly depressed about bugs from essex still being in our code is all i have to live for anymore | |
| 15:34:37 | mriedem | not to mention all of my software skills are stuck in 6 years ago and i'm not learning anything new as a developer...wahwah | |
| 15:35:59 | artom | You're learning new soft skills | |
| 15:36:08 | artom | Like... how to deal with depression ^_^ | |
| 15:37:38 | dansmith | nice | |
| 15:40:48 | mriedem | that'll be good on my next interview | |
| 15:40:48 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Drop migrate_keypairs_to_api_db data migration https://review.openstack.org/649648 | |
| 15:40:58 | mriedem | dansmith: jaypipes: mnaser: ^ there you go | |
| 15:43:13 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Drop migrate_keypairs_to_api_db data migration https://review.openstack.org/649648 | |
| 15:45:42 | mriedem | cdent: heh sound familiar? https://bugs.launchpad.net/nova/+bug/1822986 | |
| 15:45:43 | openstack | Launchpad bug 1822986 in OpenStack Compute (nova) "Not clear if www_authenticate_uri is really needed" [Undecided,New] | |
| 15:45:55 | cdent | oh hai | |
| 15:46:20 | cdent | hmm, yes, rather familiar | |
| 15:51:09 | melwitt | mriedem, efried: I thought I was coordinating RC2, but if someone else wants to, that's fine with me. I put the things I thought should go into it on https://etherpad.openstack.org/p/nova-stein-rc-potential last night | |
| 15:52:44 | mriedem | melwitt: i think those are the same two changes that are being put in, so it's aligned | |
| 15:53:14 | melwitt | ok, cool | |
| 16:00:01 | jaypipes | mriedem: ++ | |
| 16:06:09 | melwitt | mriedem, efried: fyi I've proposed RC2 at https://review.openstack.org/649656 | |
| 16:19:00 | efried | mriedem: was away for a bit there. Did you decide to do something about SIGHUP for RC2? | |
| 16:19:55 | mriedem | efried: no | |
| 16:20:05 | mriedem | i went on a online data migrations tangent | |
| 16:20:24 | efried | ight | |
| 16:21:23 | mriedem | kashyap: MIN_LIBVIRT_VERSION = (3, 0, 0) and MIN_LIBVIRT_POSTCOPY_VERSION = (1, 3, 3), do you have a change to remove MIN_LIBVIRT_POSTCOPY_VERSION? | |
| 16:24:24 | mriedem | doesn't look like it | |
| 16:34:43 | mriedem | NewBruce: what do you have set for live_migration_permit_post_copy in nova.conf on your compute services? | |
| 16:38:51 | openstackgerrit | Matt Riedemann proposed openstack/nova master: libvirt: drop MIN_LIBVIRT_POSTCOPY_VERSION https://review.openstack.org/649671 | |
| 16:50:39 | openstackgerrit | Matt Riedemann proposed openstack/nova master: libvirt: remove conditional on VIR_DOMAIN_EVENT_SUSPENDED_POSTCOPY https://review.openstack.org/649674 | |
| 16:52:07 | mriedem | sean-k-mooney: so on https://review.openstack.org/#/c/649464/ we're never getting a post-copy event during live migration so we don't activate the dest host port binding prior to post_live_migration_at_destination, so that kind of throws that theory out the window as being the issue NewBruce is hitting | |
| 16:52:13 | openstackgerrit | Eric Fried proposed openstack/nova stable/rocky: Fix bug preventing forbidden traits from working https://review.openstack.org/649603 | |
| 16:58:49 | sean-k-mooney | mriedem: ya i noticed that this morning that it passed | |
| 17:00:07 | sean-k-mooney | i was wondering if we shoudl maybe try and force the compute level to be different on the contoler vs the compute and see if that triggers the issue | |
| 17:00:53 | sean-k-mooney | mriedem: were you going to resubmit with post copy enabled? | |
| 17:02:16 | mriedem | sean-k-mooney: yeah i was going to try enabling post-copy but these cirros guests are so tiny i'm not sure it will mean anything | |
| 17:02:37 | mriedem | efried: dansmith: on that SIGHUP n-cpu issue, i tried that in a train devstack created this morning and i don't even get to the wait for network event part | |
| 17:02:38 | mriedem | http://paste.openstack.org/show/748820/ | |
| 17:02:54 | mriedem | libvirt blows up in privsep | |
| 17:03:16 | sean-k-mooney | mriedem: i might try and create a neutron fullstack test to reproduce the nutron db error | |
| 17:03:21 | mriedem | this is how i hup'ed: sudo systemctl kill -s HUP devstack@n-cpu.service | |
| 17:03:27 | dansmith | mriedem: what happens if you try it again? | |
| 17:03:37 | mriedem | the server create? or the hup? | |
| 17:03:42 | dansmith | mriedem: the create | |
| 17:03:55 | mriedem | same thing | |
| 17:03:58 | dansmith | okay | |
| 17:04:06 | dansmith | I have not seen it manifest that way | |
| 17:04:26 | dansmith | obviously that seems like a much bigger deal | |
| 17:04:36 | mriedem | unplug on cleaning up from the failure also then blows up b/c privsep http://paste.openstack.org/show/748821/ | |
| 17:04:48 | dansmith | which should be more evidence that the oslo behavior is completely wrong | |
| 17:06:10 | mriedem | did a full systemctl restart and created a server and it was fine as expected | |
| 17:06:24 | dansmith | I forget, but can people still choose to run privsep things via rootwrap so we can restart the daemon? | |
| 17:06:36 | dansmith | because if so, maybe that's why you see that and it's devstack default config or whatever | |
| 17:06:56 | dansmith | maybe I could get mikal to yell at us over twitter to answer that | |
| 17:07:25 | mriedem | idk, privsep config et al is something i haven't had to look at in years | |
| 17:07:35 | mriedem | but was always confusing to me | |
| 17:07:37 | dansmith | yeah | |
| 17:07:55 | dansmith | or maybe it's different with some of the privsep things we merged recently, | |
| 17:08:05 | dansmith | but people reporting it are missing more of the privsepification | |
| 17:08:44 | dansmith | did you create a server before the hup too? | |
| 17:12:13 | mriedem | yup | |
| 17:12:35 | mriedem | created test1, fine, sighup'ed, create test2 fails, create test3 fails, restart n-cpu, create test4 ok | |
| 17:14:33 | mriedem | commented 34 and 35 fwiw in bug https://bugs.launchpad.net/nova/+bug/1715374 | |
| 17:14:34 | openstack | Launchpad bug 1715374 in OpenStack Compute (nova) "Reloading compute with SIGHUP prevents instances from booting" [High,In progress] - Assigned to Ralf Haferkamp (rhafer) | |
| 17:14:52 | dansmith | okay just trying to think of reasons you see it differently than reported, but probably because of recent changes I guess | |
| 17:18:21 | mriedem | the privsep-helper child processes are definitely gone after the SIGHUP http://paste.openstack.org/show/748822/ | |
| 17:19:26 | dansmith | so we probably just didn't notice before or something as fewer things used it | |
| 17:19:37 | dansmith | but I don't think there's any reason that they're being killed now and not before | |
| 17:20:48 | mriedem | yeah, this is what i see in the n-cpu logs on the HUP http://paste.openstack.org/show/748823/ | |
| 17:21:08 | mriedem | note the | |
| 17:21:09 | mriedem | Apr 03 17:15:28 train nova-compute[19990]: DEBUG oslo_privsep.comm [-] EOF on privsep read channel {{(pid=19990) _reader_main /usr/local/lib/python2.7/dist-packages/oslo | |
| 17:21:16 | dansmith | yep | |
| 17:21:29 | dansmith | and that we're calling the sighup handler, but then restarting anyway | |
| 17:21:32 | dansmith | soft restart | |
| 17:21:54 | dansmith | so you might be able to do something to the global privsep state to cause it to be respawned again after the restart, but I dunno how that works | |
| 17:22:02 | dansmith | it's just all pretty wrong | |
| 17:24:40 | mriedem | not much interesting in the unit either http://paste.openstack.org/show/748824/ | |
| 17:29:32 | mriedem | yeah i'm not sure what does this http://logs.openstack.org/64/649464/1/check/tempest-full-py3/f849a52/controller/logs/screen-n-cpu.txt.gz#_Apr_02_23_01_32_822059 | |
| 17:30:35 | mriedem | oh probably this https://github.com/openstack/nova/blob/master/nova/cmd/compute.py#L45 | |
| 17:33:16 | mriedem | i'll throw that into ComputeManager.reset() and see what happens | |
| 17:33:44 | dansmith | I think that won't help, because it's before the restart | |
| 17:34:00 | dansmith | but worth a try I guess | |
| 17:36:48 | mriedem | yeah it didn't do anything | |
| 17:37:13 | mriedem | well i guess we can say SIGHUP of n-cpu is f'ed | |
| 17:37:35 | mnaser | very very f'd | |
| 17:38:05 | mnaser | I mean I caught this through the openstack-ansible CI which was relying on it to refresh RPCs versions after upgrades | |
| 17:38:29 | mnaser | but for some reason we had vif_plugging_is_fatal=false so it never broke | |
| 17:38:34 | mnaser | but once i got rid of that option, we resorted to restarting all agents which is not ideal really | |
| 17:38:46 | mriedem | i'm not sure if i should report a new bug for the privsep wrinkle here, or if that is a regression in stein - to find out i'd have to spin up a stable/rocky devstack | |
| 17:38:50 | dansmith | but this is more broken than what you were seeing | |
| 17:39:29 | mnaser | :< | |
| 17:39:29 | mriedem | right, i could probably just recreate in our post-test hook in the nova-next job | |
| 17:39:42 | mriedem | but first i need some lunch | |
| 17:40:05 | eandersson | alex_xu, for sure, but nothing supports it yet, not even the openstackclient | |