Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-04-03
19:59:14 openstackgerrit Merged openstack/nova stable/stein: Add doc on VGPU allocs and inventories for nrp https://review.openstack.org/649454
19:59:32 mriedem kashyap: i already pushed up a change, and i just noticed b/c i was looking at that post-copy code
20:14:51 mriedem efried: dansmith: i recreated that sighup privsep issue on rocky devstack so it's not a stein regression
20:16:15 dansmith "nice"
20:48:03 mriedem heh, if an instance action fails, the message is always "Error"
20:48:04 mriedem that's it
20:48:28 mriedem the api says, "The related error message for when an action fails." - but that's really only 'Error'
20:48:31 mriedem very helpful
20:51:38 openstackgerrit Merged openstack/nova master: Libvirt: gracefully handle non-nic VFs https://review.openstack.org/649409
21:05:27 mriedem so this is probably going to be filed under a big pile of don't care, but when you resize a server, you get at least 2 events on the action in conductor, one here (conductor_migrate_server): https://github.com/openstack/nova/blob/e7ae6c65cd24fb3e0776fac80fbab2ab16e9d9ed/nova/conductor/manager.py#L266
21:05:38 mriedem and one here which is just called 'cold_migrate': https://github.com/openstack/nova/blob/e7ae6c65cd24fb3e0776fac80fbab2ab16e9d9ed/nova/conductor/manager.py#L288
21:05:44 mriedem the latter is confusing if you're doing a resize and not a cold migration
21:05:50 mriedem and both together is redundant
21:05:59 mriedem the latter was here first though
21:06:04 mriedem former was added in newton
21:06:16 mriedem any validity in dropping the confusing 'cold_migrate' one?
21:06:33 mriedem note for a resize the action name is still 'resize' rather than (cold) 'migrate'
21:08:38 openstackgerrit Matt Riedemann proposed openstack/nova stable/stein: Libvirt: gracefully handle non-nic VFs https://review.openstack.org/649630
21:09:42 mriedem efried: melwitt: ^ approved
21:12:17 openstackgerrit Matt Riedemann proposed openstack/nova master: Fix ProviderUsageBaseTestCase._run_periodics for multi-cell https://review.openstack.org/641179
21:12:18 openstackgerrit Matt Riedemann proposed openstack/nova master: Improve CinderFixtureNewAttachFlow https://review.openstack.org/639382
21:12:18 openstackgerrit Matt Riedemann proposed openstack/nova master: Add functional recreate test for bug 1818914 https://review.openstack.org/641521
21:12:19 openstack bug 1818914 in OpenStack Compute (nova) "Hypervisor resource usage on source still shows old flavor usage after resize confirm until update_available_resource periodic runs" [Low,In progress] https://launchpad.net/bugs/1818914 - Assigned to Matt Riedemann (mriedem)
21:12:19 openstackgerrit Matt Riedemann proposed openstack/nova master: Remove unused context parameter from RT._get_instance_type https://review.openstack.org/641792
21:12:19 openstackgerrit Matt Riedemann proposed openstack/nova master: Update usage in RT.drop_move_claim during confirm resize https://review.openstack.org/641806
21:12:20 openstackgerrit Matt Riedemann proposed openstack/nova master: Add Migration.cross_cell_move and get_by_uuid https://review.openstack.org/614012
21:12:20 openstackgerrit Matt Riedemann proposed openstack/nova master: Add InstanceAction/Event create() method https://review.openstack.org/614036
21:12:21 openstackgerrit Matt Riedemann proposed openstack/nova master: Add Instance.hidden field https://review.openstack.org/631123
21:12:21 openstackgerrit Matt Riedemann proposed openstack/nova master: Add TargetDBSetupTask https://review.openstack.org/627892
21:12:22 openstackgerrit Matt Riedemann proposed openstack/nova master: Add CrossCellMigrationTask https://review.openstack.org/631581
21:12:22 openstackgerrit Matt Riedemann proposed openstack/nova master: Execute TargetDBSetupTask https://review.openstack.org/633853
21:12:23 openstackgerrit Matt Riedemann proposed openstack/nova master: Add can_connect_volume() compute driver method https://review.openstack.org/621313
21:12:23 openstackgerrit Matt Riedemann proposed openstack/nova master: Add prep_snapshot_based_resize_at_dest compute method https://review.openstack.org/633293
21:12:24 openstackgerrit Matt Riedemann proposed openstack/nova master: Add PrepResizeAtDestTask https://review.openstack.org/627890
21:12:24 openstackgerrit Matt Riedemann proposed openstack/nova master: Add prep_snapshot_based_resize_at_source compute method https://review.openstack.org/634832
21:12:25 openstackgerrit Matt Riedemann proposed openstack/nova master: Add nova.compute.utils.delete_image https://review.openstack.org/637605
21:12:25 openstackgerrit Matt Riedemann proposed openstack/nova master: Add PrepResizeAtSourceTask https://review.openstack.org/627891
21:12:26 openstackgerrit Matt Riedemann proposed openstack/nova master: Refactor ComputeManager.remove_volume_connection https://review.openstack.org/642183
21:12:37 openstackgerrit Matt Riedemann proposed openstack/nova master: Add revert_snapshot_based_resize conductor RPC method https://review.openstack.org/638047
21:12:37 openstackgerrit Matt Riedemann proposed openstack/nova master: Revert cross-cell resize from the API https://review.openstack.org/638048
21:12:38 openstackgerrit Matt Riedemann proposed openstack/nova master: Confirm cross-cell resize while deleting a server https://review.openstack.org/638268
21:12:38 openstackgerrit Matt Riedemann proposed openstack/nova master: Add CrossCellWeigher https://review.openstack.org/614353
21:12:39 openstackgerrit Matt Riedemann proposed openstack/nova master: Add cross-cell resize policy rule and enable in API https://review.openstack.org/638269
21:14:00 mriedem dansmith: i added a test for this cross-cell resize issue i ran into yesterday: https://review.openstack.org/#/c/643451/5/nova/tests/functional/test_cross_cell_migrate.py@499 - tl;dr is i'm going to need to hard destroy the instance in the db, instance.destroy() won't cut it
21:14:38 mriedem for example, resize to target cell and then revert back to source, then try to resize again to target will fail (that's that test) because there is a (soft) deleted instance record in the target cell db still
21:15:02 mriedem and the uuid unique constraint will prevent us from creating the instance in the target cell db on the 2nd attempt
21:15:25 mriedem same for rollbacks on failure (hard destroy in target cell db) and confirm resize (hard destroy from source cell db)
21:15:50 mriedem i think it's probably ok, although it sounds kind of scary - but you'll always have a copy of the instance in one of the db when the operation is over, so we don't lose anything
21:16:39 mriedem a real shitty alternative limitation/workaround is that it's just a no-go for cross-cell resizing that instance until the problem db is purged of the deleted instance
21:30:31 melwitt mriedem: thx
21:33:21 efried mriedem: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Deadlock%20found%20when%20trying%20to%20get%20lock%3B%20try%20restarting%20transaction%5C%22%20AND%20message%3A%5C%22UPDATE%20migrations%20SET%20updated_at%3D%25(updated_at)s%2C%20status%3D%25(status)s%20WHERE%20migrations.id%20%3D%20%25(migrations_id)s%5C%22
21:33:55 efried looks... new?
21:34:50 mriedem efried: nope
21:35:05 mriedem https://bugs.launchpad.net/nova/+bug/1642537
21:35:07 openstack Launchpad bug 1642537 in OpenStack Compute (nova) stein "finish_resize fails with DBDeadlock on migrations table" [Medium,In progress] - Assigned to Matt Riedemann (mriedem)
21:36:25 dansmith mriedem: you can't undelete it if you need to restore?
21:36:31 efried mriedem: not in e-r I guess?
21:37:53 tonyb Is there a configurable 'timeout' for server build operations?
21:39:06 mriedem efried: used to be i think
21:39:14 mriedem dansmith: no....but restore how/where?
21:39:29 mriedem like i said, you've have a copy of the instance and related records in the other cell db
21:39:47 dansmith mriedem: I mean undelete if you need to revert
21:39:58 dansmith mriedem: look it up with deleted=yes, set deleted=0, save
21:40:11 dansmith you'll have to fix up things we don't soft-delete, but that seems easier
21:40:44 dansmith I think you probably do want to hard-delete it when you confirm though, so you don't have to worry about the was-it-in-this-cell-before case each time,
21:40:45 mriedem that's not really the issue, unless i'm misunderstanding
21:40:50 dansmith but I dunno, while we're waiting for confirm...
21:40:52 dansmith okay
21:41:07 dansmith oh, I see,
21:41:21 mriedem when you revert, the instance is in both dbs,
21:41:22 melwitt tonyb: I found this https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.instance_build_timeout
21:41:22 dansmith you're talking about the instance in the target cell that got reverted back to source, a week later when you try to migrate again?
21:41:32 mriedem yeah
21:41:46 mriedem we literally can't have the instance.deleted!=0 in the cell db we're moving to
21:42:39 tonyb melwitt: Thanks
21:42:39 tonyb (undercloud) [root@director config-data]# grep -Erin instance_build_timeout nova
21:42:43 tonyb nova/etc/nova/nova.conf:983:#instance_build_timeout=0
21:42:43 mriedem heh i didn't know instance_build_timeout existed
21:42:48 tonyb so I guess that isn't it
21:42:49 mriedem it's a periodic
21:43:18 mriedem looks like a hack for just setting instance status to ERROR for things that were hung or we didn't properly set to ERROR state on failure
21:43:31 dansmith mriedem: okay, I guess that seems obvious to me so I'm not sure why you're bringing it up as if it's a problem or something
21:43:49 mriedem dansmith: we just haven't hard hard destroy on stuff in the cell db records, except tags i guess
21:43:54 mriedem so it seems new and scary
21:43:58 melwitt tonyb: oh yeah, hm
21:44:01 dansmith mmkay
21:44:34 mriedem luckily theodoros already has a patch to add that support for his rebuild from cell0 series https://review.openstack.org/#/c/570202/
21:44:42 mriedem which it turned out he later didn't need, but i will
21:47:14 mriedem anyway, i'll update the re-proposed spec to note it
22:06:41 openstackgerrit Matt Riedemann proposed openstack/nova master: api-ref: document ordering for instance actions and events https://review.openstack.org/649748
22:09:18 openstackgerrit Matt Riedemann proposed openstack/nova master: api-ref: fix description of os-server-external-events 'events' param https://review.openstack.org/649750
22:32:42 openstackgerrit Matt Riedemann proposed openstack/nova master: Remove MIN_COMPUTE_MULTIATTACH conditions in API https://review.openstack.org/649757
22:51:47 openstackgerrit Matt Riedemann proposed openstack/nova master: Add nova-status upgrade check for minimum required cinder API version https://review.openstack.org/649759
23:01:14 efried mriedem: where does grenade source live?
23:03:32 mriedem http://git.openstack.org/cgit/openstack-dev/grenade/
#openstack-nova - 2019-04-04
00:32:54 openstackgerrit Merged openstack/nova master: Don't warn on network-vif-unplugged event during live migration https://review.openstack.org/642877
00:37:24 mriedem this uh, seems not so necessary https://github.com/openstack/nova/commit/70c7ba7324cf47179d978b4073545e3837d4fd24#diff-12951ec32bc2b8cae6ea1933599a3dbcR334
00:41:32 mriedem and potentially a security issue
00:43:22 mriedem efried: melwitt: ^ is new in stein so if it's something we shouldn't be exposing in the notifications we should probably think about that for rc2
00:47:12 sean-k-mooney wait we send the db connection field in the cellmapping notifcation?

Earlier   Later