| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-04-03 | |||
| 18:46:43 | mriedem | efried: i'm going to try on rocky devstack to see if it is a very obvious regression in stein because if so we should have a known issue reno | |
| 18:46:59 | efried | ack | |
| 19:53:27 | kashyap | mriedem: Was AFK as I was a bit "under the weather". To your question, no, I don't yet have a patch to remove MIN_LIBVIRT_POSTCOPY_VERSION (and a couple of other constants that I noted in the main commit message) | |
| 19:53:39 | kashyap | I'll get to it this week. As I noted I will fix them | |
| 19:53:58 | kashyap | mriedem: Curious, what made you notice it? Or just regular code audit caught your eye? | |
| 19:59:14 | openstackgerrit | Merged openstack/nova stable/stein: Add doc on VGPU allocs and inventories for nrp https://review.openstack.org/649454 | |
| 19:59:32 | mriedem | kashyap: i already pushed up a change, and i just noticed b/c i was looking at that post-copy code | |
| 20:14:51 | mriedem | efried: dansmith: i recreated that sighup privsep issue on rocky devstack so it's not a stein regression | |
| 20:16:15 | dansmith | "nice" | |
| 20:48:03 | mriedem | heh, if an instance action fails, the message is always "Error" | |
| 20:48:04 | mriedem | that's it | |
| 20:48:28 | mriedem | the api says, "The related error message for when an action fails." - but that's really only 'Error' | |
| 20:48:31 | mriedem | very helpful | |
| 20:51:38 | openstackgerrit | Merged openstack/nova master: Libvirt: gracefully handle non-nic VFs https://review.openstack.org/649409 | |
| 21:05:27 | mriedem | so this is probably going to be filed under a big pile of don't care, but when you resize a server, you get at least 2 events on the action in conductor, one here (conductor_migrate_server): https://github.com/openstack/nova/blob/e7ae6c65cd24fb3e0776fac80fbab2ab16e9d9ed/nova/conductor/manager.py#L266 | |
| 21:05:38 | mriedem | and one here which is just called 'cold_migrate': https://github.com/openstack/nova/blob/e7ae6c65cd24fb3e0776fac80fbab2ab16e9d9ed/nova/conductor/manager.py#L288 | |
| 21:05:44 | mriedem | the latter is confusing if you're doing a resize and not a cold migration | |
| 21:05:50 | mriedem | and both together is redundant | |
| 21:05:59 | mriedem | the latter was here first though | |
| 21:06:04 | mriedem | former was added in newton | |
| 21:06:16 | mriedem | any validity in dropping the confusing 'cold_migrate' one? | |
| 21:06:33 | mriedem | note for a resize the action name is still 'resize' rather than (cold) 'migrate' | |
| 21:08:38 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/stein: Libvirt: gracefully handle non-nic VFs https://review.openstack.org/649630 | |
| 21:09:42 | mriedem | efried: melwitt: ^ approved | |
| 21:12:17 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Fix ProviderUsageBaseTestCase._run_periodics for multi-cell https://review.openstack.org/641179 | |
| 21:12:18 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Improve CinderFixtureNewAttachFlow https://review.openstack.org/639382 | |
| 21:12:18 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add functional recreate test for bug 1818914 https://review.openstack.org/641521 | |
| 21:12:19 | openstack | bug 1818914 in OpenStack Compute (nova) "Hypervisor resource usage on source still shows old flavor usage after resize confirm until update_available_resource periodic runs" [Low,In progress] https://launchpad.net/bugs/1818914 - Assigned to Matt Riedemann (mriedem) | |
| 21:12:19 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove unused context parameter from RT._get_instance_type https://review.openstack.org/641792 | |
| 21:12:19 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Update usage in RT.drop_move_claim during confirm resize https://review.openstack.org/641806 | |
| 21:12:20 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add Migration.cross_cell_move and get_by_uuid https://review.openstack.org/614012 | |
| 21:12:20 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add InstanceAction/Event create() method https://review.openstack.org/614036 | |
| 21:12:21 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add Instance.hidden field https://review.openstack.org/631123 | |
| 21:12:21 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add TargetDBSetupTask https://review.openstack.org/627892 | |
| 21:12:22 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add CrossCellMigrationTask https://review.openstack.org/631581 | |
| 21:12:22 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Execute TargetDBSetupTask https://review.openstack.org/633853 | |
| 21:12:23 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add can_connect_volume() compute driver method https://review.openstack.org/621313 | |
| 21:12:23 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add prep_snapshot_based_resize_at_dest compute method https://review.openstack.org/633293 | |
| 21:12:24 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add PrepResizeAtDestTask https://review.openstack.org/627890 | |
| 21:12:24 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add prep_snapshot_based_resize_at_source compute method https://review.openstack.org/634832 | |
| 21:12:25 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add nova.compute.utils.delete_image https://review.openstack.org/637605 | |
| 21:12:25 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add PrepResizeAtSourceTask https://review.openstack.org/627891 | |
| 21:12:26 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Refactor ComputeManager.remove_volume_connection https://review.openstack.org/642183 | |
| 21:12:37 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add revert_snapshot_based_resize conductor RPC method https://review.openstack.org/638047 | |
| 21:12:37 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Revert cross-cell resize from the API https://review.openstack.org/638048 | |
| 21:12:38 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Confirm cross-cell resize while deleting a server https://review.openstack.org/638268 | |
| 21:12:38 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add CrossCellWeigher https://review.openstack.org/614353 | |
| 21:12:39 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add cross-cell resize policy rule and enable in API https://review.openstack.org/638269 | |
| 21:14:00 | mriedem | dansmith: i added a test for this cross-cell resize issue i ran into yesterday: https://review.openstack.org/#/c/643451/5/nova/tests/functional/test_cross_cell_migrate.py@499 - tl;dr is i'm going to need to hard destroy the instance in the db, instance.destroy() won't cut it | |
| 21:14:38 | mriedem | for example, resize to target cell and then revert back to source, then try to resize again to target will fail (that's that test) because there is a (soft) deleted instance record in the target cell db still | |
| 21:15:02 | mriedem | and the uuid unique constraint will prevent us from creating the instance in the target cell db on the 2nd attempt | |
| 21:15:25 | mriedem | same for rollbacks on failure (hard destroy in target cell db) and confirm resize (hard destroy from source cell db) | |
| 21:15:50 | mriedem | i think it's probably ok, although it sounds kind of scary - but you'll always have a copy of the instance in one of the db when the operation is over, so we don't lose anything | |
| 21:16:39 | mriedem | a real shitty alternative limitation/workaround is that it's just a no-go for cross-cell resizing that instance until the problem db is purged of the deleted instance | |
| 21:30:31 | melwitt | mriedem: thx | |
| 21:33:21 | efried | mriedem: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Deadlock%20found%20when%20trying%20to%20get%20lock%3B%20try%20restarting%20transaction%5C%22%20AND%20message%3A%5C%22UPDATE%20migrations%20SET%20updated_at%3D%25(updated_at)s%2C%20status%3D%25(status)s%20WHERE%20migrations.id%20%3D%20%25(migrations_id)s%5C%22 | |
| 21:33:55 | efried | looks... new? | |
| 21:34:50 | mriedem | efried: nope | |
| 21:35:05 | mriedem | https://bugs.launchpad.net/nova/+bug/1642537 | |
| 21:35:07 | openstack | Launchpad bug 1642537 in OpenStack Compute (nova) stein "finish_resize fails with DBDeadlock on migrations table" [Medium,In progress] - Assigned to Matt Riedemann (mriedem) | |
| 21:36:25 | dansmith | mriedem: you can't undelete it if you need to restore? | |
| 21:36:31 | efried | mriedem: not in e-r I guess? | |
| 21:37:53 | tonyb | Is there a configurable 'timeout' for server build operations? | |
| 21:39:06 | mriedem | efried: used to be i think | |
| 21:39:14 | mriedem | dansmith: no....but restore how/where? | |
| 21:39:29 | mriedem | like i said, you've have a copy of the instance and related records in the other cell db | |
| 21:39:47 | dansmith | mriedem: I mean undelete if you need to revert | |
| 21:39:58 | dansmith | mriedem: look it up with deleted=yes, set deleted=0, save | |
| 21:40:11 | dansmith | you'll have to fix up things we don't soft-delete, but that seems easier | |
| 21:40:44 | dansmith | I think you probably do want to hard-delete it when you confirm though, so you don't have to worry about the was-it-in-this-cell-before case each time, | |
| 21:40:45 | mriedem | that's not really the issue, unless i'm misunderstanding | |
| 21:40:50 | dansmith | but I dunno, while we're waiting for confirm... | |
| 21:40:52 | dansmith | okay | |
| 21:41:07 | dansmith | oh, I see, | |
| 21:41:21 | mriedem | when you revert, the instance is in both dbs, | |
| 21:41:22 | melwitt | tonyb: I found this https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.instance_build_timeout | |
| 21:41:22 | dansmith | you're talking about the instance in the target cell that got reverted back to source, a week later when you try to migrate again? | |
| 21:41:32 | mriedem | yeah | |
| 21:41:46 | mriedem | we literally can't have the instance.deleted!=0 in the cell db we're moving to | |
| 21:42:39 | tonyb | melwitt: Thanks | |
| 21:42:39 | tonyb | (undercloud) [root@director config-data]# grep -Erin instance_build_timeout nova | |
| 21:42:43 | tonyb | nova/etc/nova/nova.conf:983:#instance_build_timeout=0 | |
| 21:42:43 | mriedem | heh i didn't know instance_build_timeout existed | |
| 21:42:48 | tonyb | so I guess that isn't it | |
| 21:42:49 | mriedem | it's a periodic | |
| 21:43:18 | mriedem | looks like a hack for just setting instance status to ERROR for things that were hung or we didn't properly set to ERROR state on failure | |
| 21:43:31 | dansmith | mriedem: okay, I guess that seems obvious to me so I'm not sure why you're bringing it up as if it's a problem or something | |
| 21:43:49 | mriedem | dansmith: we just haven't hard hard destroy on stuff in the cell db records, except tags i guess | |
| 21:43:54 | mriedem | so it seems new and scary | |
| 21:43:58 | melwitt | tonyb: oh yeah, hm | |
| 21:44:01 | dansmith | mmkay | |
| 21:44:34 | mriedem | luckily theodoros already has a patch to add that support for his rebuild from cell0 series https://review.openstack.org/#/c/570202/ | |
| 21:44:42 | mriedem | which it turned out he later didn't need, but i will | |
| 21:47:14 | mriedem | anyway, i'll update the re-proposed spec to note it | |
| 22:06:41 | openstackgerrit | Matt Riedemann proposed openstack/nova master: api-ref: document ordering for instance actions and events https://review.openstack.org/649748 | |
| 22:09:18 | openstackgerrit | Matt Riedemann proposed openstack/nova master: api-ref: fix description of os-server-external-events 'events' param https://review.openstack.org/649750 | |
| 22:32:42 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove MIN_COMPUTE_MULTIATTACH conditions in API https://review.openstack.org/649757 | |
| 22:51:47 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add nova-status upgrade check for minimum required cinder API version https://review.openstack.org/649759 | |
| 23:01:14 | efried | mriedem: where does grenade source live? | |
| 23:03:32 | mriedem | http://git.openstack.org/cgit/openstack-dev/grenade/ | |