Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-09-01
15:41:07 openstackgerrit Matt Riedemann proposed openstack/nova master: add online_data_migrations to nova docs https://review.openstack.org/493442
15:41:37 stephenfin mriedem, sean-k-mooney: Where did we come down on https://bugs.launchpad.net/nova/+bug/1698010? Do we actually want to use a FQDN for hostname?
15:41:38 openstack Launchpad bug 1698010 in OpenStack Compute (nova) "neutron-based instances should not use the nova-network 'dhcp_domain' option" [High,In progress] - Assigned to Stephen Finucane (stephenfinucane)
15:43:04 openstackgerrit Stephen Finucane proposed openstack/nova master: add online_data_migrations to nova docs https://review.openstack.org/493442
15:47:13 mriedem dansmith: can you clarify a question i have in here? https://review.openstack.org/#/c/493442/6
15:47:15 openstackgerrit Rodolfo Alonso Hernandez proposed openstack/os-vif master: Set HostPluginInfo plugin_name as an Enum https://review.openstack.org/500111
15:49:03 dansmith mriedem: yeah, so really it should be only the controllers, but we had one incident where it wouldn't complete until all the computes have been upgraded
15:49:05 dansmith mriedem: the pci one
15:49:17 dansmith I was talking to someone recently about this, not sure if it was this person
15:49:41 dansmith I dunno what to say here, because in some cases it'd be one or the other, but almost always just controllers
15:51:01 mriedem in the case of the pci issue,
15:51:20 mriedem did running online_data_migrations break anything? or just not migrate things as expected?
15:51:35 dansmith nope it would just warn you that it couldn't finish because it needed the computes to be upgraded first
15:51:41 mriedem ah, ok
15:52:00 mriedem so, then i think we leave the ideal case in the man page, and if we have a known weird wrinkle, then we release note that
15:52:16 dansmith sounds good
15:59:57 mriedem evacuating local git
16:00:06 mriedem force=True
16:00:07 openstackgerrit Matt Riedemann proposed openstack/nova master: Remove dest node allocation if evacuate MoveClaim fails https://review.openstack.org/499878
16:00:08 openstackgerrit Matt Riedemann proposed openstack/nova master: Add recreate test for forced host evacuate not setting dest allocations https://review.openstack.org/499678
16:00:08 openstackgerrit Matt Riedemann proposed openstack/nova master: Add a test to make sure failed evacuate cleans up dest allocation https://review.openstack.org/499877
16:00:09 openstackgerrit Matt Riedemann proposed openstack/nova master: Add recreate test for evacuate claim failure https://review.openstack.org/499874
16:00:09 openstackgerrit Matt Riedemann proposed openstack/nova master: Create allocations against forced dest host during evacuate https://review.openstack.org/499399
16:00:10 openstackgerrit Matt Riedemann proposed openstack/nova master: Refactor out claim_resources_on_destination into a utility https://review.openstack.org/499718
16:00:10 openstackgerrit Matt Riedemann proposed openstack/nova master: Modernize set_vm_state_and_notify https://review.openstack.org/499799
16:00:18 stephenfin sdague: If you get a chance, would appreciate a sanity check on https://bugs.launchpad.net/nova/+bug/1704458
16:00:19 openstack Launchpad bug 1704458 in OpenStack Compute (nova) "The use_ipv6 flag not only influences nova networking" [High,Invalid] - Assigned to Stephen Finucane (stephenfinucane)
16:07:32 mriedem stephenfin: regarding that dhcp_domain thing for dns, i think it makes sense that if you're using neutron, you pull it from the neutron network resource as it appears your series is doing
16:07:48 mriedem stephenfin: i just have to get through all of your unrelated changes to get to the meat of the fix first... :
16:07:49 mriedem :)
16:09:12 stephenfin mriedem: Right, but the question is whether we should actually be using FQDN at all, rather than from neutron instead. sean-k-mooney seemed to suggest it was a bad idea, but my network/sysadmin foo is not good enough to say why not :)
16:09:41 stephenfin Also, I like refactoring. Sorry :D I can pull those out
16:10:11 mriedem run swift young stephen
16:10:18 mriedem don't look back....
16:18:37 openstackgerrit Matt Riedemann proposed openstack/nova master: doc: fix online_data_migrations option in upgrades doc https://review.openstack.org/500124
16:34:37 openstackgerrit Matt Riedemann proposed openstack/nova master: Set error state after failed evacuation https://review.openstack.org/499237
16:35:50 mriedem we're going to want to backport this fix ^ back through newton
16:35:58 mriedem so it'd be nice to start that, it's pretty trivial
16:36:32 mriedem prevents the source compute from deleting local instance if the evacuation fails in conductor to find a dest host
16:39:00 dansmith mriedem: so just thinking about this quickly,
16:39:31 dansmith mriedem: are you thinking that if we fail to schedule a new place and the source comes back up that there's value in having it still there?
16:40:03 dansmith because the way this was designed, once an evacuation starts we're *going* to delete it from the source compute to avoid a race between rebuilding it elsewhere and the compute coming back up and deciding to keep it or not
16:40:42 mriedem melwitt: commented on https://review.openstack.org/#/c/407346/ - cinder would also need changes if we do something like this in nova
16:42:25 mriedem dansmith: hmm
16:42:28 mriedem that's a good point
16:42:51 dansmith knowing the folks that use this a lot from a script,
16:42:58 melwitt mriedem: cool thanks. looks like it's not as straightforward was I initially thought
16:43:00 dansmith the last thing they want is vague behavior from this
16:43:14 mriedem i wish that would have been a comment in the _delete_evacuated_instances code because i spent a good chunk of time yesterday trying to figure out why we include 'accepted' in the filter
16:43:42 dansmith mriedem: was it in my robustify spec? I think the logic was sussed out there
16:43:44 dansmith anyway,
16:43:52 mriedem i didn't read the spec
16:44:16 dansmith I would think that what we want is to make sure you can call rebuild (or at least reset-state) on the instance and have it rebuilt somewhere else
16:44:55 dansmith because I think we've nulled out the instance.host here at this point right? meaning it won't be a simple restart on the source host
16:45:05 openstackgerrit Merged openstack/nova master: Functional test for regression bug #1713783 https://review.openstack.org/500057
16:45:07 openstack bug 1713783 in OpenStack Compute (nova) "After failed evacuation the recovered source compute tries to delete the instance" [High,In progress] https://launchpad.net/bugs/1713783 - Assigned to Matt Riedemann (mriedem)
16:45:26 mriedem dansmith: i don't think the instance.host is nulled out
16:45:30 mriedem which is part of the problem
16:45:39 mriedem reading the docstring for _destroy_evacuated_instances, it says,
16:45:48 mriedem "Check that the instances reported by the driver are still associated with this host. If they are not, destroy them"
16:45:48 openstackgerrit Merged openstack/nova master: add online_data_migrations to nova docs https://review.openstack.org/493442
16:45:57 mriedem but it doesn't actually compare the instances that it finds against self.host
16:46:13 dansmith mriedem: right, it used to, which is why it would delete everything if your hostname changed
16:46:14 mriedem it just says, oh you had an evacuation from this host, we assume it was cool, delete local
16:47:23 mriedem it also says, "with the exception of instances which are in the MIGRATING, RESIZE_MIGRATING, RESIZE_MIGRATED, RESIZE_FINISH task state or RESIZED vm state"
16:47:24 dansmith mriedem: so it might not be nulled out, but if we're in the middle of it starting on the new destination host, we don't know, so if we don't delete it, we'll end up with it in two places
16:47:29 mriedem but i don't see that filtering happening anywhere
16:48:20 dansmith mriedem: yeah that's not from me, that's from dd6fb1246ff2789bd78b772b45e1fcac21eda67a
16:48:29 dansmith which looks like it had a filter in it which is now gone
16:48:41 dansmith that makes no sense to me though,
16:48:47 dansmith since that code already only looks at evacuations
16:49:48 mriedem https://review.openstack.org/#/c/101803/ predates the robustify that used the migration record for filtering
16:50:12 dansmith ah yep
16:50:20 mriedem so we just need to cleanup the comments in here a bit,
16:50:22 dansmith so maybe my bad for not removing that then
16:50:27 mriedem so if we don't take this change,
16:50:41 mriedem and the compute is up and running, need to think about what the RT is going to do
16:51:23 mriedem so we'd remove the allocation on the source node when it starts up
16:51:29 dansmith which is correct
16:51:38 mriedem right b/c the guest was deleted,
16:51:45 mriedem the instance isn't actually gone, and it points at the source host still
16:51:51 dansmith and it looks to me like we would be able to run rebuild on it
16:52:25 dansmith you can run rebuild from error state and it seems like it'll just pick up the evacuation and try again,
16:52:30 dansmith but not if we error it out like this patch does
16:52:55 mriedem the update_available_resource task is going to query for instances on this host/node, and pull this in since it's not deleted
16:53:02 mriedem pass that to _update_usage_from_instances
16:53:27 mriedem which calls _update_usage_from_instance with has_ocata_computes=True
16:53:30 mriedem *False
16:53:38 mriedem assuming you're upgraded
16:54:03 mriedem and it won't recreate the allocation in placement for the source node, which again is correct since we don't have the guest anymore
16:54:16 mriedem if you have ocata computes, it would, but...
16:54:18 mriedem meh?
16:54:18 dansmith we'll create the new allocation during the rebuild
16:54:56 dansmith and when build finishes it'll replace the doubled (due to migration) allocation with the single one for the new host
16:54:59 mriedem the instance wouldn't necessarily be in ERROR state,
16:55:08 mriedem and conductor doesn't set it to ERROR state if it fails to find a host
16:55:08 dansmith it would from that novalidhost
16:55:12 mriedem nope
16:55:17 mriedem it just resets the task_state to None
16:55:18 dansmith it does _set_vm_state_and_notify()

Earlier   Later