| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-04-03 | |||
| 18:43:01 | KH-Jared | good to know. I figured, but I was still going to dig to make sure it wasn't my changes somehow | |
| 18:43:21 | KH-Jared | so that was some time spent that I probably should've just asked sooner | |
| 18:45:24 | efried | mriedem: If privsep is part of what doesn't start back up properly, and there's privsep stuff in the critical path that's merged since that bug was opened (https://review.openstack.org/#/q/status:merged+project:openstack/nova+branch:master+topic:my-own-personal-alternative-universe) then that'd do it. | |
| 18:45:26 | sean-k-mooney | KH-Jared: well it good that you are paying attention to the thrid party ci jobs but yes its always good to ask yourself is it resonable the my change could have broken it | |
| 18:45:42 | sean-k-mooney | and if you dont know always feel free to ask | |
| 18:45:45 | efried | mriedem: We don't know if privsep was borked before and just not being hit because those paths were still using rootwrap | |
| 18:46:43 | mriedem | efried: i'm going to try on rocky devstack to see if it is a very obvious regression in stein because if so we should have a known issue reno | |
| 18:46:59 | efried | ack | |
| 19:53:27 | kashyap | mriedem: Was AFK as I was a bit "under the weather". To your question, no, I don't yet have a patch to remove MIN_LIBVIRT_POSTCOPY_VERSION (and a couple of other constants that I noted in the main commit message) | |
| 19:53:39 | kashyap | I'll get to it this week. As I noted I will fix them | |
| 19:53:58 | kashyap | mriedem: Curious, what made you notice it? Or just regular code audit caught your eye? | |
| 19:59:14 | openstackgerrit | Merged openstack/nova stable/stein: Add doc on VGPU allocs and inventories for nrp https://review.openstack.org/649454 | |
| 19:59:32 | mriedem | kashyap: i already pushed up a change, and i just noticed b/c i was looking at that post-copy code | |
| 20:14:51 | mriedem | efried: dansmith: i recreated that sighup privsep issue on rocky devstack so it's not a stein regression | |
| 20:16:15 | dansmith | "nice" | |
| 20:48:03 | mriedem | heh, if an instance action fails, the message is always "Error" | |
| 20:48:04 | mriedem | that's it | |
| 20:48:28 | mriedem | the api says, "The related error message for when an action fails." - but that's really only 'Error' | |
| 20:48:31 | mriedem | very helpful | |
| 20:51:38 | openstackgerrit | Merged openstack/nova master: Libvirt: gracefully handle non-nic VFs https://review.openstack.org/649409 | |
| 21:05:27 | mriedem | so this is probably going to be filed under a big pile of don't care, but when you resize a server, you get at least 2 events on the action in conductor, one here (conductor_migrate_server): https://github.com/openstack/nova/blob/e7ae6c65cd24fb3e0776fac80fbab2ab16e9d9ed/nova/conductor/manager.py#L266 | |
| 21:05:38 | mriedem | and one here which is just called 'cold_migrate': https://github.com/openstack/nova/blob/e7ae6c65cd24fb3e0776fac80fbab2ab16e9d9ed/nova/conductor/manager.py#L288 | |
| 21:05:44 | mriedem | the latter is confusing if you're doing a resize and not a cold migration | |
| 21:05:50 | mriedem | and both together is redundant | |
| 21:05:59 | mriedem | the latter was here first though | |
| 21:06:04 | mriedem | former was added in newton | |
| 21:06:16 | mriedem | any validity in dropping the confusing 'cold_migrate' one? | |
| 21:06:33 | mriedem | note for a resize the action name is still 'resize' rather than (cold) 'migrate' | |
| 21:08:38 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/stein: Libvirt: gracefully handle non-nic VFs https://review.openstack.org/649630 | |
| 21:09:42 | mriedem | efried: melwitt: ^ approved | |
| 21:12:17 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Fix ProviderUsageBaseTestCase._run_periodics for multi-cell https://review.openstack.org/641179 | |
| 21:12:18 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add functional recreate test for bug 1818914 https://review.openstack.org/641521 | |
| 21:12:18 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Improve CinderFixtureNewAttachFlow https://review.openstack.org/639382 | |
| 21:12:19 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Update usage in RT.drop_move_claim during confirm resize https://review.openstack.org/641806 | |
| 21:12:19 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove unused context parameter from RT._get_instance_type https://review.openstack.org/641792 | |
| 21:12:19 | openstack | bug 1818914 in OpenStack Compute (nova) "Hypervisor resource usage on source still shows old flavor usage after resize confirm until update_available_resource periodic runs" [Low,In progress] https://launchpad.net/bugs/1818914 - Assigned to Matt Riedemann (mriedem) | |
| 21:12:20 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add InstanceAction/Event create() method https://review.openstack.org/614036 | |
| 21:12:20 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add Migration.cross_cell_move and get_by_uuid https://review.openstack.org/614012 | |
| 21:12:21 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add TargetDBSetupTask https://review.openstack.org/627892 | |
| 21:12:21 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add Instance.hidden field https://review.openstack.org/631123 | |
| 21:12:22 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Execute TargetDBSetupTask https://review.openstack.org/633853 | |
| 21:12:22 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add CrossCellMigrationTask https://review.openstack.org/631581 | |
| 21:12:23 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add prep_snapshot_based_resize_at_dest compute method https://review.openstack.org/633293 | |
| 21:12:23 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add can_connect_volume() compute driver method https://review.openstack.org/621313 | |
| 21:12:24 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add prep_snapshot_based_resize_at_source compute method https://review.openstack.org/634832 | |
| 21:12:24 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add PrepResizeAtDestTask https://review.openstack.org/627890 | |
| 21:12:25 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add PrepResizeAtSourceTask https://review.openstack.org/627891 | |
| 21:12:25 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add nova.compute.utils.delete_image https://review.openstack.org/637605 | |
| 21:12:26 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Refactor ComputeManager.remove_volume_connection https://review.openstack.org/642183 | |
| 21:12:37 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Revert cross-cell resize from the API https://review.openstack.org/638048 | |
| 21:12:37 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add revert_snapshot_based_resize conductor RPC method https://review.openstack.org/638047 | |
| 21:12:38 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add CrossCellWeigher https://review.openstack.org/614353 | |
| 21:12:38 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Confirm cross-cell resize while deleting a server https://review.openstack.org/638268 | |
| 21:12:39 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add cross-cell resize policy rule and enable in API https://review.openstack.org/638269 | |
| 21:14:00 | mriedem | dansmith: i added a test for this cross-cell resize issue i ran into yesterday: https://review.openstack.org/#/c/643451/5/nova/tests/functional/test_cross_cell_migrate.py@499 - tl;dr is i'm going to need to hard destroy the instance in the db, instance.destroy() won't cut it | |
| 21:14:38 | mriedem | for example, resize to target cell and then revert back to source, then try to resize again to target will fail (that's that test) because there is a (soft) deleted instance record in the target cell db still | |
| 21:15:02 | mriedem | and the uuid unique constraint will prevent us from creating the instance in the target cell db on the 2nd attempt | |
| 21:15:25 | mriedem | same for rollbacks on failure (hard destroy in target cell db) and confirm resize (hard destroy from source cell db) | |
| 21:15:50 | mriedem | i think it's probably ok, although it sounds kind of scary - but you'll always have a copy of the instance in one of the db when the operation is over, so we don't lose anything | |
| 21:16:39 | mriedem | a real shitty alternative limitation/workaround is that it's just a no-go for cross-cell resizing that instance until the problem db is purged of the deleted instance | |
| 21:30:31 | melwitt | mriedem: thx | |
| 21:33:21 | efried | mriedem: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Deadlock%20found%20when%20trying%20to%20get%20lock%3B%20try%20restarting%20transaction%5C%22%20AND%20message%3A%5C%22UPDATE%20migrations%20SET%20updated_at%3D%25(updated_at)s%2C%20status%3D%25(status)s%20WHERE%20migrations.id%20%3D%20%25(migrations_id)s%5C%22 | |
| 21:33:55 | efried | looks... new? | |
| 21:34:50 | mriedem | efried: nope | |
| 21:35:05 | mriedem | https://bugs.launchpad.net/nova/+bug/1642537 | |
| 21:35:07 | openstack | Launchpad bug 1642537 in OpenStack Compute (nova) stein "finish_resize fails with DBDeadlock on migrations table" [Medium,In progress] - Assigned to Matt Riedemann (mriedem) | |
| 21:36:25 | dansmith | mriedem: you can't undelete it if you need to restore? | |
| 21:36:31 | efried | mriedem: not in e-r I guess? | |
| 21:37:53 | tonyb | Is there a configurable 'timeout' for server build operations? | |
| 21:39:06 | mriedem | efried: used to be i think | |
| 21:39:14 | mriedem | dansmith: no....but restore how/where? | |
| 21:39:29 | mriedem | like i said, you've have a copy of the instance and related records in the other cell db | |
| 21:39:47 | dansmith | mriedem: I mean undelete if you need to revert | |
| 21:39:58 | dansmith | mriedem: look it up with deleted=yes, set deleted=0, save | |
| 21:40:11 | dansmith | you'll have to fix up things we don't soft-delete, but that seems easier | |
| 21:40:44 | dansmith | I think you probably do want to hard-delete it when you confirm though, so you don't have to worry about the was-it-in-this-cell-before case each time, | |
| 21:40:45 | mriedem | that's not really the issue, unless i'm misunderstanding | |
| 21:40:50 | dansmith | but I dunno, while we're waiting for confirm... | |
| 21:40:52 | dansmith | okay | |
| 21:41:07 | dansmith | oh, I see, | |
| 21:41:21 | mriedem | when you revert, the instance is in both dbs, | |
| 21:41:22 | dansmith | you're talking about the instance in the target cell that got reverted back to source, a week later when you try to migrate again? | |
| 21:41:22 | melwitt | tonyb: I found this https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.instance_build_timeout | |
| 21:41:32 | mriedem | yeah | |
| 21:41:46 | mriedem | we literally can't have the instance.deleted!=0 in the cell db we're moving to | |
| 21:42:39 | tonyb | (undercloud) [root@director config-data]# grep -Erin instance_build_timeout nova | |
| 21:42:39 | tonyb | melwitt: Thanks | |
| 21:42:43 | mriedem | heh i didn't know instance_build_timeout existed | |
| 21:42:43 | tonyb | nova/etc/nova/nova.conf:983:#instance_build_timeout=0 | |
| 21:42:48 | tonyb | so I guess that isn't it | |
| 21:42:49 | mriedem | it's a periodic | |
| 21:43:18 | mriedem | looks like a hack for just setting instance status to ERROR for things that were hung or we didn't properly set to ERROR state on failure | |
| 21:43:31 | dansmith | mriedem: okay, I guess that seems obvious to me so I'm not sure why you're bringing it up as if it's a problem or something | |
| 21:43:49 | mriedem | dansmith: we just haven't hard hard destroy on stuff in the cell db records, except tags i guess | |
| 21:43:54 | mriedem | so it seems new and scary | |
| 21:43:58 | melwitt | tonyb: oh yeah, hm | |
| 21:44:01 | dansmith | mmkay | |
| 21:44:34 | mriedem | luckily theodoros already has a patch to add that support for his rebuild from cell0 series https://review.openstack.org/#/c/570202/ | |
| 21:44:42 | mriedem | which it turned out he later didn't need, but i will | |
| 21:47:14 | mriedem | anyway, i'll update the re-proposed spec to note it | |