| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-04-03 | |||
| 17:37:35 | mnaser | very very f'd | |
| 17:38:05 | mnaser | I mean I caught this through the openstack-ansible CI which was relying on it to refresh RPCs versions after upgrades | |
| 17:38:29 | mnaser | but for some reason we had vif_plugging_is_fatal=false so it never broke | |
| 17:38:34 | mnaser | but once i got rid of that option, we resorted to restarting all agents which is not ideal really | |
| 17:38:46 | mriedem | i'm not sure if i should report a new bug for the privsep wrinkle here, or if that is a regression in stein - to find out i'd have to spin up a stable/rocky devstack | |
| 17:38:50 | dansmith | but this is more broken than what you were seeing | |
| 17:39:29 | mnaser | :< | |
| 17:39:29 | mriedem | right, i could probably just recreate in our post-test hook in the nova-next job | |
| 17:39:42 | mriedem | but first i need some lunch | |
| 17:40:05 | eandersson | alex_xu, for sure, but nothing supports it yet, not even the openstackclient | |
| 17:52:18 | KH-Jared | I'm learning that simple changes aren't always simple. Apparently I'm making the tearDown of a test fail | |
| 17:56:36 | artom | KH-Jared, yep. Actual fix: 1 day's work. Tests: Methuselah. | |
| 18:26:30 | openstackgerrit | Matt Riedemann proposed openstack/nova master: DNM: Test theory about bug 1822884 https://review.openstack.org/649464 | |
| 18:26:31 | openstack | bug 1822884 in OpenStack Compute (nova) "live migration fails due to port binding duplicate key entry in post_live_migrate" [Undecided,New] https://launchpad.net/bugs/1822884 | |
| 18:27:42 | openstackgerrit | Matt Riedemann proposed openstack/nova master: libvirt: drop MIN_LIBVIRT_POSTCOPY_VERSION https://review.openstack.org/649671 | |
| 18:27:43 | openstackgerrit | Matt Riedemann proposed openstack/nova master: libvirt: remove conditional on VIR_DOMAIN_EVENT_SUSPENDED_POSTCOPY https://review.openstack.org/649674 | |
| 18:29:24 | mriedem | hmm wtf http://logs.openstack.org/53/641453/1/check/nova-grenade-live-migration/88cd6f4/logs/screen-n-cpu.txt.gz?level=TRACE#_Apr_03_15_47_39_469617 | |
| 18:41:56 | KH-Jared | I feel like I'm missing something, because what failed for 'IBM zVM CI' was a tempest test, but under Zuul, the same tempest test was successful | |
| 18:42:27 | mriedem | KH-Jared: you don't need to worry about the zvm ci, your rbd change is only for the libvirt driver which is not what zvm ci is using | |
| 18:42:57 | mriedem | the zuul jobs and specifically the non-voting ceph plugin job are what you'd care about | |
| 18:43:01 | KH-Jared | good to know. I figured, but I was still going to dig to make sure it wasn't my changes somehow | |
| 18:43:21 | KH-Jared | so that was some time spent that I probably should've just asked sooner | |
| 18:45:24 | efried | mriedem: If privsep is part of what doesn't start back up properly, and there's privsep stuff in the critical path that's merged since that bug was opened (https://review.openstack.org/#/q/status:merged+project:openstack/nova+branch:master+topic:my-own-personal-alternative-universe) then that'd do it. | |
| 18:45:26 | sean-k-mooney | KH-Jared: well it good that you are paying attention to the thrid party ci jobs but yes its always good to ask yourself is it resonable the my change could have broken it | |
| 18:45:42 | sean-k-mooney | and if you dont know always feel free to ask | |
| 18:45:45 | efried | mriedem: We don't know if privsep was borked before and just not being hit because those paths were still using rootwrap | |
| 18:46:43 | mriedem | efried: i'm going to try on rocky devstack to see if it is a very obvious regression in stein because if so we should have a known issue reno | |
| 18:46:59 | efried | ack | |
| 19:53:27 | kashyap | mriedem: Was AFK as I was a bit "under the weather". To your question, no, I don't yet have a patch to remove MIN_LIBVIRT_POSTCOPY_VERSION (and a couple of other constants that I noted in the main commit message) | |
| 19:53:39 | kashyap | I'll get to it this week. As I noted I will fix them | |
| 19:53:58 | kashyap | mriedem: Curious, what made you notice it? Or just regular code audit caught your eye? | |
| 19:59:14 | openstackgerrit | Merged openstack/nova stable/stein: Add doc on VGPU allocs and inventories for nrp https://review.openstack.org/649454 | |
| 19:59:32 | mriedem | kashyap: i already pushed up a change, and i just noticed b/c i was looking at that post-copy code | |
| 20:14:51 | mriedem | efried: dansmith: i recreated that sighup privsep issue on rocky devstack so it's not a stein regression | |
| 20:16:15 | dansmith | "nice" | |
| 20:48:03 | mriedem | heh, if an instance action fails, the message is always "Error" | |
| 20:48:04 | mriedem | that's it | |
| 20:48:28 | mriedem | the api says, "The related error message for when an action fails." - but that's really only 'Error' | |
| 20:48:31 | mriedem | very helpful | |
| 20:51:38 | openstackgerrit | Merged openstack/nova master: Libvirt: gracefully handle non-nic VFs https://review.openstack.org/649409 | |
| 21:05:27 | mriedem | so this is probably going to be filed under a big pile of don't care, but when you resize a server, you get at least 2 events on the action in conductor, one here (conductor_migrate_server): https://github.com/openstack/nova/blob/e7ae6c65cd24fb3e0776fac80fbab2ab16e9d9ed/nova/conductor/manager.py#L266 | |
| 21:05:38 | mriedem | and one here which is just called 'cold_migrate': https://github.com/openstack/nova/blob/e7ae6c65cd24fb3e0776fac80fbab2ab16e9d9ed/nova/conductor/manager.py#L288 | |
| 21:05:44 | mriedem | the latter is confusing if you're doing a resize and not a cold migration | |
| 21:05:50 | mriedem | and both together is redundant | |
| 21:05:59 | mriedem | the latter was here first though | |
| 21:06:04 | mriedem | former was added in newton | |
| 21:06:16 | mriedem | any validity in dropping the confusing 'cold_migrate' one? | |
| 21:06:33 | mriedem | note for a resize the action name is still 'resize' rather than (cold) 'migrate' | |
| 21:08:38 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/stein: Libvirt: gracefully handle non-nic VFs https://review.openstack.org/649630 | |
| 21:09:42 | mriedem | efried: melwitt: ^ approved | |
| 21:12:17 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Fix ProviderUsageBaseTestCase._run_periodics for multi-cell https://review.openstack.org/641179 | |
| 21:12:18 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Improve CinderFixtureNewAttachFlow https://review.openstack.org/639382 | |
| 21:12:18 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add functional recreate test for bug 1818914 https://review.openstack.org/641521 | |
| 21:12:19 | openstack | bug 1818914 in OpenStack Compute (nova) "Hypervisor resource usage on source still shows old flavor usage after resize confirm until update_available_resource periodic runs" [Low,In progress] https://launchpad.net/bugs/1818914 - Assigned to Matt Riedemann (mriedem) | |
| 21:12:19 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove unused context parameter from RT._get_instance_type https://review.openstack.org/641792 | |
| 21:12:19 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Update usage in RT.drop_move_claim during confirm resize https://review.openstack.org/641806 | |
| 21:12:20 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add Migration.cross_cell_move and get_by_uuid https://review.openstack.org/614012 | |
| 21:12:20 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add InstanceAction/Event create() method https://review.openstack.org/614036 | |
| 21:12:21 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add Instance.hidden field https://review.openstack.org/631123 | |
| 21:12:21 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add TargetDBSetupTask https://review.openstack.org/627892 | |
| 21:12:22 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add CrossCellMigrationTask https://review.openstack.org/631581 | |
| 21:12:22 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Execute TargetDBSetupTask https://review.openstack.org/633853 | |
| 21:12:23 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add can_connect_volume() compute driver method https://review.openstack.org/621313 | |
| 21:12:23 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add prep_snapshot_based_resize_at_dest compute method https://review.openstack.org/633293 | |
| 21:12:24 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add PrepResizeAtDestTask https://review.openstack.org/627890 | |
| 21:12:24 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add prep_snapshot_based_resize_at_source compute method https://review.openstack.org/634832 | |
| 21:12:25 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add nova.compute.utils.delete_image https://review.openstack.org/637605 | |
| 21:12:25 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add PrepResizeAtSourceTask https://review.openstack.org/627891 | |
| 21:12:26 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Refactor ComputeManager.remove_volume_connection https://review.openstack.org/642183 | |
| 21:12:37 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add revert_snapshot_based_resize conductor RPC method https://review.openstack.org/638047 | |
| 21:12:37 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Revert cross-cell resize from the API https://review.openstack.org/638048 | |
| 21:12:38 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Confirm cross-cell resize while deleting a server https://review.openstack.org/638268 | |
| 21:12:38 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add CrossCellWeigher https://review.openstack.org/614353 | |
| 21:12:39 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add cross-cell resize policy rule and enable in API https://review.openstack.org/638269 | |
| 21:14:00 | mriedem | dansmith: i added a test for this cross-cell resize issue i ran into yesterday: https://review.openstack.org/#/c/643451/5/nova/tests/functional/test_cross_cell_migrate.py@499 - tl;dr is i'm going to need to hard destroy the instance in the db, instance.destroy() won't cut it | |
| 21:14:38 | mriedem | for example, resize to target cell and then revert back to source, then try to resize again to target will fail (that's that test) because there is a (soft) deleted instance record in the target cell db still | |
| 21:15:02 | mriedem | and the uuid unique constraint will prevent us from creating the instance in the target cell db on the 2nd attempt | |
| 21:15:25 | mriedem | same for rollbacks on failure (hard destroy in target cell db) and confirm resize (hard destroy from source cell db) | |
| 21:15:50 | mriedem | i think it's probably ok, although it sounds kind of scary - but you'll always have a copy of the instance in one of the db when the operation is over, so we don't lose anything | |
| 21:16:39 | mriedem | a real shitty alternative limitation/workaround is that it's just a no-go for cross-cell resizing that instance until the problem db is purged of the deleted instance | |
| 21:30:31 | melwitt | mriedem: thx | |
| 21:33:21 | efried | mriedem: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Deadlock%20found%20when%20trying%20to%20get%20lock%3B%20try%20restarting%20transaction%5C%22%20AND%20message%3A%5C%22UPDATE%20migrations%20SET%20updated_at%3D%25(updated_at)s%2C%20status%3D%25(status)s%20WHERE%20migrations.id%20%3D%20%25(migrations_id)s%5C%22 | |
| 21:33:55 | efried | looks... new? | |
| 21:34:50 | mriedem | efried: nope | |
| 21:35:05 | mriedem | https://bugs.launchpad.net/nova/+bug/1642537 | |
| 21:35:07 | openstack | Launchpad bug 1642537 in OpenStack Compute (nova) stein "finish_resize fails with DBDeadlock on migrations table" [Medium,In progress] - Assigned to Matt Riedemann (mriedem) | |
| 21:36:25 | dansmith | mriedem: you can't undelete it if you need to restore? | |
| 21:36:31 | efried | mriedem: not in e-r I guess? | |
| 21:37:53 | tonyb | Is there a configurable 'timeout' for server build operations? | |
| 21:39:06 | mriedem | efried: used to be i think | |
| 21:39:14 | mriedem | dansmith: no....but restore how/where? | |
| 21:39:29 | mriedem | like i said, you've have a copy of the instance and related records in the other cell db | |
| 21:39:47 | dansmith | mriedem: I mean undelete if you need to revert | |
| 21:39:58 | dansmith | mriedem: look it up with deleted=yes, set deleted=0, save | |
| 21:40:11 | dansmith | you'll have to fix up things we don't soft-delete, but that seems easier | |
| 21:40:44 | dansmith | I think you probably do want to hard-delete it when you confirm though, so you don't have to worry about the was-it-in-this-cell-before case each time, | |
| 21:40:45 | mriedem | that's not really the issue, unless i'm misunderstanding | |
| 21:40:50 | dansmith | but I dunno, while we're waiting for confirm... | |
| 21:40:52 | dansmith | okay | |
| 21:41:07 | dansmith | oh, I see, | |