| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-04-03 | |||
| 17:12:13 | mriedem | yup | |
| 17:12:35 | mriedem | created test1, fine, sighup'ed, create test2 fails, create test3 fails, restart n-cpu, create test4 ok | |
| 17:14:33 | mriedem | commented 34 and 35 fwiw in bug https://bugs.launchpad.net/nova/+bug/1715374 | |
| 17:14:34 | openstack | Launchpad bug 1715374 in OpenStack Compute (nova) "Reloading compute with SIGHUP prevents instances from booting" [High,In progress] - Assigned to Ralf Haferkamp (rhafer) | |
| 17:14:52 | dansmith | okay just trying to think of reasons you see it differently than reported, but probably because of recent changes I guess | |
| 17:18:21 | mriedem | the privsep-helper child processes are definitely gone after the SIGHUP http://paste.openstack.org/show/748822/ | |
| 17:19:26 | dansmith | so we probably just didn't notice before or something as fewer things used it | |
| 17:19:37 | dansmith | but I don't think there's any reason that they're being killed now and not before | |
| 17:20:48 | mriedem | yeah, this is what i see in the n-cpu logs on the HUP http://paste.openstack.org/show/748823/ | |
| 17:21:08 | mriedem | note the | |
| 17:21:09 | mriedem | Apr 03 17:15:28 train nova-compute[19990]: DEBUG oslo_privsep.comm [-] EOF on privsep read channel {{(pid=19990) _reader_main /usr/local/lib/python2.7/dist-packages/oslo | |
| 17:21:16 | dansmith | yep | |
| 17:21:29 | dansmith | and that we're calling the sighup handler, but then restarting anyway | |
| 17:21:32 | dansmith | soft restart | |
| 17:21:54 | dansmith | so you might be able to do something to the global privsep state to cause it to be respawned again after the restart, but I dunno how that works | |
| 17:22:02 | dansmith | it's just all pretty wrong | |
| 17:24:40 | mriedem | not much interesting in the unit either http://paste.openstack.org/show/748824/ | |
| 17:29:32 | mriedem | yeah i'm not sure what does this http://logs.openstack.org/64/649464/1/check/tempest-full-py3/f849a52/controller/logs/screen-n-cpu.txt.gz#_Apr_02_23_01_32_822059 | |
| 17:30:35 | mriedem | oh probably this https://github.com/openstack/nova/blob/master/nova/cmd/compute.py#L45 | |
| 17:33:16 | mriedem | i'll throw that into ComputeManager.reset() and see what happens | |
| 17:33:44 | dansmith | I think that won't help, because it's before the restart | |
| 17:34:00 | dansmith | but worth a try I guess | |
| 17:36:48 | mriedem | yeah it didn't do anything | |
| 17:37:13 | mriedem | well i guess we can say SIGHUP of n-cpu is f'ed | |
| 17:37:35 | mnaser | very very f'd | |
| 17:38:05 | mnaser | I mean I caught this through the openstack-ansible CI which was relying on it to refresh RPCs versions after upgrades | |
| 17:38:29 | mnaser | but for some reason we had vif_plugging_is_fatal=false so it never broke | |
| 17:38:34 | mnaser | but once i got rid of that option, we resorted to restarting all agents which is not ideal really | |
| 17:38:46 | mriedem | i'm not sure if i should report a new bug for the privsep wrinkle here, or if that is a regression in stein - to find out i'd have to spin up a stable/rocky devstack | |
| 17:38:50 | dansmith | but this is more broken than what you were seeing | |
| 17:39:29 | mnaser | :< | |
| 17:39:29 | mriedem | right, i could probably just recreate in our post-test hook in the nova-next job | |
| 17:39:42 | mriedem | but first i need some lunch | |
| 17:40:05 | eandersson | alex_xu, for sure, but nothing supports it yet, not even the openstackclient | |
| 17:52:18 | KH-Jared | I'm learning that simple changes aren't always simple. Apparently I'm making the tearDown of a test fail | |
| 17:56:36 | artom | KH-Jared, yep. Actual fix: 1 day's work. Tests: Methuselah. | |
| 18:26:30 | openstackgerrit | Matt Riedemann proposed openstack/nova master: DNM: Test theory about bug 1822884 https://review.openstack.org/649464 | |
| 18:26:31 | openstack | bug 1822884 in OpenStack Compute (nova) "live migration fails due to port binding duplicate key entry in post_live_migrate" [Undecided,New] https://launchpad.net/bugs/1822884 | |
| 18:27:42 | openstackgerrit | Matt Riedemann proposed openstack/nova master: libvirt: drop MIN_LIBVIRT_POSTCOPY_VERSION https://review.openstack.org/649671 | |
| 18:27:43 | openstackgerrit | Matt Riedemann proposed openstack/nova master: libvirt: remove conditional on VIR_DOMAIN_EVENT_SUSPENDED_POSTCOPY https://review.openstack.org/649674 | |
| 18:29:24 | mriedem | hmm wtf http://logs.openstack.org/53/641453/1/check/nova-grenade-live-migration/88cd6f4/logs/screen-n-cpu.txt.gz?level=TRACE#_Apr_03_15_47_39_469617 | |
| 18:41:56 | KH-Jared | I feel like I'm missing something, because what failed for 'IBM zVM CI' was a tempest test, but under Zuul, the same tempest test was successful | |
| 18:42:27 | mriedem | KH-Jared: you don't need to worry about the zvm ci, your rbd change is only for the libvirt driver which is not what zvm ci is using | |
| 18:42:57 | mriedem | the zuul jobs and specifically the non-voting ceph plugin job are what you'd care about | |
| 18:43:01 | KH-Jared | good to know. I figured, but I was still going to dig to make sure it wasn't my changes somehow | |
| 18:43:21 | KH-Jared | so that was some time spent that I probably should've just asked sooner | |
| 18:45:24 | efried | mriedem: If privsep is part of what doesn't start back up properly, and there's privsep stuff in the critical path that's merged since that bug was opened (https://review.openstack.org/#/q/status:merged+project:openstack/nova+branch:master+topic:my-own-personal-alternative-universe) then that'd do it. | |
| 18:45:26 | sean-k-mooney | KH-Jared: well it good that you are paying attention to the thrid party ci jobs but yes its always good to ask yourself is it resonable the my change could have broken it | |
| 18:45:42 | sean-k-mooney | and if you dont know always feel free to ask | |
| 18:45:45 | efried | mriedem: We don't know if privsep was borked before and just not being hit because those paths were still using rootwrap | |
| 18:46:43 | mriedem | efried: i'm going to try on rocky devstack to see if it is a very obvious regression in stein because if so we should have a known issue reno | |
| 18:46:59 | efried | ack | |
| 19:53:27 | kashyap | mriedem: Was AFK as I was a bit "under the weather". To your question, no, I don't yet have a patch to remove MIN_LIBVIRT_POSTCOPY_VERSION (and a couple of other constants that I noted in the main commit message) | |
| 19:53:39 | kashyap | I'll get to it this week. As I noted I will fix them | |
| 19:53:58 | kashyap | mriedem: Curious, what made you notice it? Or just regular code audit caught your eye? | |
| 19:59:14 | openstackgerrit | Merged openstack/nova stable/stein: Add doc on VGPU allocs and inventories for nrp https://review.openstack.org/649454 | |
| 19:59:32 | mriedem | kashyap: i already pushed up a change, and i just noticed b/c i was looking at that post-copy code | |
| 20:14:51 | mriedem | efried: dansmith: i recreated that sighup privsep issue on rocky devstack so it's not a stein regression | |
| 20:16:15 | dansmith | "nice" | |
| 20:48:03 | mriedem | heh, if an instance action fails, the message is always "Error" | |
| 20:48:04 | mriedem | that's it | |
| 20:48:28 | mriedem | the api says, "The related error message for when an action fails." - but that's really only 'Error' | |
| 20:48:31 | mriedem | very helpful | |
| 20:51:38 | openstackgerrit | Merged openstack/nova master: Libvirt: gracefully handle non-nic VFs https://review.openstack.org/649409 | |
| 21:05:27 | mriedem | so this is probably going to be filed under a big pile of don't care, but when you resize a server, you get at least 2 events on the action in conductor, one here (conductor_migrate_server): https://github.com/openstack/nova/blob/e7ae6c65cd24fb3e0776fac80fbab2ab16e9d9ed/nova/conductor/manager.py#L266 | |
| 21:05:38 | mriedem | and one here which is just called 'cold_migrate': https://github.com/openstack/nova/blob/e7ae6c65cd24fb3e0776fac80fbab2ab16e9d9ed/nova/conductor/manager.py#L288 | |
| 21:05:44 | mriedem | the latter is confusing if you're doing a resize and not a cold migration | |
| 21:05:50 | mriedem | and both together is redundant | |
| 21:05:59 | mriedem | the latter was here first though | |
| 21:06:04 | mriedem | former was added in newton | |
| 21:06:16 | mriedem | any validity in dropping the confusing 'cold_migrate' one? | |
| 21:06:33 | mriedem | note for a resize the action name is still 'resize' rather than (cold) 'migrate' | |
| 21:08:38 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/stein: Libvirt: gracefully handle non-nic VFs https://review.openstack.org/649630 | |
| 21:09:42 | mriedem | efried: melwitt: ^ approved | |
| 21:12:17 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Fix ProviderUsageBaseTestCase._run_periodics for multi-cell https://review.openstack.org/641179 | |
| 21:12:18 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Improve CinderFixtureNewAttachFlow https://review.openstack.org/639382 | |
| 21:12:18 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add functional recreate test for bug 1818914 https://review.openstack.org/641521 | |
| 21:12:19 | openstack | bug 1818914 in OpenStack Compute (nova) "Hypervisor resource usage on source still shows old flavor usage after resize confirm until update_available_resource periodic runs" [Low,In progress] https://launchpad.net/bugs/1818914 - Assigned to Matt Riedemann (mriedem) | |
| 21:12:19 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove unused context parameter from RT._get_instance_type https://review.openstack.org/641792 | |
| 21:12:19 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Update usage in RT.drop_move_claim during confirm resize https://review.openstack.org/641806 | |
| 21:12:20 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add Migration.cross_cell_move and get_by_uuid https://review.openstack.org/614012 | |
| 21:12:20 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add InstanceAction/Event create() method https://review.openstack.org/614036 | |
| 21:12:21 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add Instance.hidden field https://review.openstack.org/631123 | |
| 21:12:21 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add TargetDBSetupTask https://review.openstack.org/627892 | |
| 21:12:22 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add CrossCellMigrationTask https://review.openstack.org/631581 | |
| 21:12:22 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Execute TargetDBSetupTask https://review.openstack.org/633853 | |
| 21:12:23 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add can_connect_volume() compute driver method https://review.openstack.org/621313 | |
| 21:12:23 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add prep_snapshot_based_resize_at_dest compute method https://review.openstack.org/633293 | |
| 21:12:24 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add PrepResizeAtDestTask https://review.openstack.org/627890 | |
| 21:12:24 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add prep_snapshot_based_resize_at_source compute method https://review.openstack.org/634832 | |
| 21:12:25 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add nova.compute.utils.delete_image https://review.openstack.org/637605 | |
| 21:12:25 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add PrepResizeAtSourceTask https://review.openstack.org/627891 | |
| 21:12:26 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Refactor ComputeManager.remove_volume_connection https://review.openstack.org/642183 | |
| 21:12:37 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add revert_snapshot_based_resize conductor RPC method https://review.openstack.org/638047 | |
| 21:12:37 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Revert cross-cell resize from the API https://review.openstack.org/638048 | |
| 21:12:38 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Confirm cross-cell resize while deleting a server https://review.openstack.org/638268 | |
| 21:12:38 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add CrossCellWeigher https://review.openstack.org/614353 | |
| 21:12:39 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add cross-cell resize policy rule and enable in API https://review.openstack.org/638269 | |
| 21:14:00 | mriedem | dansmith: i added a test for this cross-cell resize issue i ran into yesterday: https://review.openstack.org/#/c/643451/5/nova/tests/functional/test_cross_cell_migrate.py@499 - tl;dr is i'm going to need to hard destroy the instance in the db, instance.destroy() won't cut it | |
| 21:14:38 | mriedem | for example, resize to target cell and then revert back to source, then try to resize again to target will fail (that's that test) because there is a (soft) deleted instance record in the target cell db still | |