| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-04-03 | |||
| 16:50:39 | openstackgerrit | Matt Riedemann proposed openstack/nova master: libvirt: remove conditional on VIR_DOMAIN_EVENT_SUSPENDED_POSTCOPY https://review.openstack.org/649674 | |
| 16:52:07 | mriedem | sean-k-mooney: so on https://review.openstack.org/#/c/649464/ we're never getting a post-copy event during live migration so we don't activate the dest host port binding prior to post_live_migration_at_destination, so that kind of throws that theory out the window as being the issue NewBruce is hitting | |
| 16:52:13 | openstackgerrit | Eric Fried proposed openstack/nova stable/rocky: Fix bug preventing forbidden traits from working https://review.openstack.org/649603 | |
| 16:58:49 | sean-k-mooney | mriedem: ya i noticed that this morning that it passed | |
| 17:00:07 | sean-k-mooney | i was wondering if we shoudl maybe try and force the compute level to be different on the contoler vs the compute and see if that triggers the issue | |
| 17:00:53 | sean-k-mooney | mriedem: were you going to resubmit with post copy enabled? | |
| 17:02:16 | mriedem | sean-k-mooney: yeah i was going to try enabling post-copy but these cirros guests are so tiny i'm not sure it will mean anything | |
| 17:02:37 | mriedem | efried: dansmith: on that SIGHUP n-cpu issue, i tried that in a train devstack created this morning and i don't even get to the wait for network event part | |
| 17:02:38 | mriedem | http://paste.openstack.org/show/748820/ | |
| 17:02:54 | mriedem | libvirt blows up in privsep | |
| 17:03:16 | sean-k-mooney | mriedem: i might try and create a neutron fullstack test to reproduce the nutron db error | |
| 17:03:21 | mriedem | this is how i hup'ed: sudo systemctl kill -s HUP devstack@n-cpu.service | |
| 17:03:27 | dansmith | mriedem: what happens if you try it again? | |
| 17:03:37 | mriedem | the server create? or the hup? | |
| 17:03:42 | dansmith | mriedem: the create | |
| 17:03:55 | mriedem | same thing | |
| 17:03:58 | dansmith | okay | |
| 17:04:06 | dansmith | I have not seen it manifest that way | |
| 17:04:26 | dansmith | obviously that seems like a much bigger deal | |
| 17:04:36 | mriedem | unplug on cleaning up from the failure also then blows up b/c privsep http://paste.openstack.org/show/748821/ | |
| 17:04:48 | dansmith | which should be more evidence that the oslo behavior is completely wrong | |
| 17:06:10 | mriedem | did a full systemctl restart and created a server and it was fine as expected | |
| 17:06:24 | dansmith | I forget, but can people still choose to run privsep things via rootwrap so we can restart the daemon? | |
| 17:06:36 | dansmith | because if so, maybe that's why you see that and it's devstack default config or whatever | |
| 17:06:56 | dansmith | maybe I could get mikal to yell at us over twitter to answer that | |
| 17:07:25 | mriedem | idk, privsep config et al is something i haven't had to look at in years | |
| 17:07:35 | mriedem | but was always confusing to me | |
| 17:07:37 | dansmith | yeah | |
| 17:07:55 | dansmith | or maybe it's different with some of the privsep things we merged recently, | |
| 17:08:05 | dansmith | but people reporting it are missing more of the privsepification | |
| 17:08:44 | dansmith | did you create a server before the hup too? | |
| 17:12:13 | mriedem | yup | |
| 17:12:35 | mriedem | created test1, fine, sighup'ed, create test2 fails, create test3 fails, restart n-cpu, create test4 ok | |
| 17:14:33 | mriedem | commented 34 and 35 fwiw in bug https://bugs.launchpad.net/nova/+bug/1715374 | |
| 17:14:34 | openstack | Launchpad bug 1715374 in OpenStack Compute (nova) "Reloading compute with SIGHUP prevents instances from booting" [High,In progress] - Assigned to Ralf Haferkamp (rhafer) | |
| 17:14:52 | dansmith | okay just trying to think of reasons you see it differently than reported, but probably because of recent changes I guess | |
| 17:18:21 | mriedem | the privsep-helper child processes are definitely gone after the SIGHUP http://paste.openstack.org/show/748822/ | |
| 17:19:26 | dansmith | so we probably just didn't notice before or something as fewer things used it | |
| 17:19:37 | dansmith | but I don't think there's any reason that they're being killed now and not before | |
| 17:20:48 | mriedem | yeah, this is what i see in the n-cpu logs on the HUP http://paste.openstack.org/show/748823/ | |
| 17:21:08 | mriedem | note the | |
| 17:21:09 | mriedem | Apr 03 17:15:28 train nova-compute[19990]: DEBUG oslo_privsep.comm [-] EOF on privsep read channel {{(pid=19990) _reader_main /usr/local/lib/python2.7/dist-packages/oslo | |
| 17:21:16 | dansmith | yep | |
| 17:21:29 | dansmith | and that we're calling the sighup handler, but then restarting anyway | |
| 17:21:32 | dansmith | soft restart | |
| 17:21:54 | dansmith | so you might be able to do something to the global privsep state to cause it to be respawned again after the restart, but I dunno how that works | |
| 17:22:02 | dansmith | it's just all pretty wrong | |
| 17:24:40 | mriedem | not much interesting in the unit either http://paste.openstack.org/show/748824/ | |
| 17:29:32 | mriedem | yeah i'm not sure what does this http://logs.openstack.org/64/649464/1/check/tempest-full-py3/f849a52/controller/logs/screen-n-cpu.txt.gz#_Apr_02_23_01_32_822059 | |
| 17:30:35 | mriedem | oh probably this https://github.com/openstack/nova/blob/master/nova/cmd/compute.py#L45 | |
| 17:33:16 | mriedem | i'll throw that into ComputeManager.reset() and see what happens | |
| 17:33:44 | dansmith | I think that won't help, because it's before the restart | |
| 17:34:00 | dansmith | but worth a try I guess | |
| 17:36:48 | mriedem | yeah it didn't do anything | |
| 17:37:13 | mriedem | well i guess we can say SIGHUP of n-cpu is f'ed | |
| 17:37:35 | mnaser | very very f'd | |
| 17:38:05 | mnaser | I mean I caught this through the openstack-ansible CI which was relying on it to refresh RPCs versions after upgrades | |
| 17:38:29 | mnaser | but for some reason we had vif_plugging_is_fatal=false so it never broke | |
| 17:38:34 | mnaser | but once i got rid of that option, we resorted to restarting all agents which is not ideal really | |
| 17:38:46 | mriedem | i'm not sure if i should report a new bug for the privsep wrinkle here, or if that is a regression in stein - to find out i'd have to spin up a stable/rocky devstack | |
| 17:38:50 | dansmith | but this is more broken than what you were seeing | |
| 17:39:29 | mriedem | right, i could probably just recreate in our post-test hook in the nova-next job | |
| 17:39:29 | mnaser | :< | |
| 17:39:42 | mriedem | but first i need some lunch | |
| 17:40:05 | eandersson | alex_xu, for sure, but nothing supports it yet, not even the openstackclient | |
| 17:52:18 | KH-Jared | I'm learning that simple changes aren't always simple. Apparently I'm making the tearDown of a test fail | |
| 17:56:36 | artom | KH-Jared, yep. Actual fix: 1 day's work. Tests: Methuselah. | |
| 18:26:30 | openstackgerrit | Matt Riedemann proposed openstack/nova master: DNM: Test theory about bug 1822884 https://review.openstack.org/649464 | |
| 18:26:31 | openstack | bug 1822884 in OpenStack Compute (nova) "live migration fails due to port binding duplicate key entry in post_live_migrate" [Undecided,New] https://launchpad.net/bugs/1822884 | |
| 18:27:42 | openstackgerrit | Matt Riedemann proposed openstack/nova master: libvirt: drop MIN_LIBVIRT_POSTCOPY_VERSION https://review.openstack.org/649671 | |
| 18:27:43 | openstackgerrit | Matt Riedemann proposed openstack/nova master: libvirt: remove conditional on VIR_DOMAIN_EVENT_SUSPENDED_POSTCOPY https://review.openstack.org/649674 | |
| 18:29:24 | mriedem | hmm wtf http://logs.openstack.org/53/641453/1/check/nova-grenade-live-migration/88cd6f4/logs/screen-n-cpu.txt.gz?level=TRACE#_Apr_03_15_47_39_469617 | |
| 18:41:56 | KH-Jared | I feel like I'm missing something, because what failed for 'IBM zVM CI' was a tempest test, but under Zuul, the same tempest test was successful | |
| 18:42:27 | mriedem | KH-Jared: you don't need to worry about the zvm ci, your rbd change is only for the libvirt driver which is not what zvm ci is using | |
| 18:42:57 | mriedem | the zuul jobs and specifically the non-voting ceph plugin job are what you'd care about | |
| 18:43:01 | KH-Jared | good to know. I figured, but I was still going to dig to make sure it wasn't my changes somehow | |
| 18:43:21 | KH-Jared | so that was some time spent that I probably should've just asked sooner | |
| 18:45:24 | efried | mriedem: If privsep is part of what doesn't start back up properly, and there's privsep stuff in the critical path that's merged since that bug was opened (https://review.openstack.org/#/q/status:merged+project:openstack/nova+branch:master+topic:my-own-personal-alternative-universe) then that'd do it. | |
| 18:45:26 | sean-k-mooney | KH-Jared: well it good that you are paying attention to the thrid party ci jobs but yes its always good to ask yourself is it resonable the my change could have broken it | |
| 18:45:42 | sean-k-mooney | and if you dont know always feel free to ask | |
| 18:45:45 | efried | mriedem: We don't know if privsep was borked before and just not being hit because those paths were still using rootwrap | |
| 18:46:43 | mriedem | efried: i'm going to try on rocky devstack to see if it is a very obvious regression in stein because if so we should have a known issue reno | |
| 18:46:59 | efried | ack | |
| 19:53:27 | kashyap | mriedem: Was AFK as I was a bit "under the weather". To your question, no, I don't yet have a patch to remove MIN_LIBVIRT_POSTCOPY_VERSION (and a couple of other constants that I noted in the main commit message) | |
| 19:53:39 | kashyap | I'll get to it this week. As I noted I will fix them | |
| 19:53:58 | kashyap | mriedem: Curious, what made you notice it? Or just regular code audit caught your eye? | |
| 19:59:14 | openstackgerrit | Merged openstack/nova stable/stein: Add doc on VGPU allocs and inventories for nrp https://review.openstack.org/649454 | |
| 19:59:32 | mriedem | kashyap: i already pushed up a change, and i just noticed b/c i was looking at that post-copy code | |
| 20:14:51 | mriedem | efried: dansmith: i recreated that sighup privsep issue on rocky devstack so it's not a stein regression | |
| 20:16:15 | dansmith | "nice" | |
| 20:48:03 | mriedem | heh, if an instance action fails, the message is always "Error" | |
| 20:48:04 | mriedem | that's it | |
| 20:48:28 | mriedem | the api says, "The related error message for when an action fails." - but that's really only 'Error' | |
| 20:48:31 | mriedem | very helpful | |
| 20:51:38 | openstackgerrit | Merged openstack/nova master: Libvirt: gracefully handle non-nic VFs https://review.openstack.org/649409 | |
| 21:05:27 | mriedem | so this is probably going to be filed under a big pile of don't care, but when you resize a server, you get at least 2 events on the action in conductor, one here (conductor_migrate_server): https://github.com/openstack/nova/blob/e7ae6c65cd24fb3e0776fac80fbab2ab16e9d9ed/nova/conductor/manager.py#L266 | |
| 21:05:38 | mriedem | and one here which is just called 'cold_migrate': https://github.com/openstack/nova/blob/e7ae6c65cd24fb3e0776fac80fbab2ab16e9d9ed/nova/conductor/manager.py#L288 | |
| 21:05:44 | mriedem | the latter is confusing if you're doing a resize and not a cold migration | |
| 21:05:50 | mriedem | and both together is redundant | |
| 21:05:59 | mriedem | the latter was here first though | |