Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-10-03
14:41:04 efried okay, so mriedem that would explain the source allocs magically reappearing?
14:42:59 nicolasbock mriedem: This is using Newton
14:43:09 nicolasbock I'll try to delete the allocation
14:43:15 nicolasbock And wait to see what happens :)
14:43:48 mriedem newton/ocata computes will recreate allocations yes
14:44:14 mriedem until you get everything upgraded to >= pike, the resource tracker periodic task in the compute service will try to manage allocations
14:44:18 nicolasbock The new allocation was deleted while we were chatting
14:44:53 nicolasbock Interesting mriedem
14:45:14 nicolasbock But where is the periodic task getting its information from?
14:46:32 mriedem the instances it thinks are running on that host,
14:46:36 mriedem and those instances flavors
14:46:56 nicolasbock Is there a way to update that?
14:46:58 mriedem so if compute host A thinks instance B is running on it with a flavor that uses x,y,z vcpu/ram/disk, it's going to report that
14:47:08 mriedem update what?
14:47:20 nicolasbock So I would have to convince the compute host that it's not running the instance?
14:47:24 efried nicolasbock: I kind of missed how we got into this situation. What makes you think the instance was successfully removed from the source host?
14:47:56 mriedem nicolasbock: is the instance.host in the db pointing at that host?
14:47:57 nicolasbock I am going by what `openstack server show` is telling me :)
14:48:20 mriedem server show should also tell you yeah
14:48:21 openstackgerrit Matthew Booth proposed openstack/nova master: Run evacuate tests with local/lvm and shared/rbd storage https://review.openstack.org/604400
14:49:16 nicolasbock So `server show` is reporting an incorrect hypervisor
14:49:18 mriedem this is where the RT gets the instances it thinks are running on it https://github.com/openstack/nova/blob/newton-eol/nova/compute/resource_tracker.py#L556
14:49:56 openstackgerrit Vlad Gusev proposed openstack/nova master: Not instance to ERROR if set_admin_password failed https://review.openstack.org/555160
14:50:01 sean-k-mooney mriedem: there is a live migration edgecase that mdbooth was looking at a few weeks ago where post migrate source failed and we would not update the host the vm was running on
14:50:23 sean-k-mooney but the vm has actully been moved correectly
14:52:37 mriedem nicolasbock: so did you live migrate this vm or something? why is nova reporting its on the wrong host?
14:54:07 mdbooth Ah, yes. I do recall a bug with that. If we get an error in cleanup on the source host, called *post* successful migration, we then rollback the migration and put the instance in an error state, but it's still running fine on the destination.
14:54:32 mdbooth So, e.g. if you get an error in terminate_connection or whatever, you get in this state
14:54:53 mdbooth And you can't clean it up, because instance.host is pointing to the source, but it's actually running on the dest
14:55:16 mriedem terminate_connection as in cleaning up source node volume attachments and such right?
14:55:20 mriedem same with ports i'm sure
14:55:27 mriedem post live migration cleaning up the source
14:55:31 mdbooth mriedem: Right. Any cleanup on the source
14:55:40 mriedem we should just catch and log cleanup failures
14:55:42 openstackgerrit Vlad Gusev proposed openstack/nova master: Not set instance to ERROR if set_admin_password failed https://review.openstack.org/555160
14:57:07 mdbooth mriedem: Right. The error in my view was that we put the instance in an error state, when the instance was fine. We should put the migration in an error state, but leave the instance alone.
14:57:40 mdbooth And also do as much cleanup as possible in the presence of errors.
14:57:46 nicolasbock mriedem: Yes I think that's what happened
15:07:11 openstackgerrit Jan Gutter proposed openstack/nova-specs master: Spec to implement vRouter HW offloads https://review.openstack.org/567148
15:07:12 openstackgerrit Jan Gutter proposed openstack/nova-specs master: Spec to implement generic HW offloads for os-vif https://review.openstack.org/607610
15:09:30 openstackgerrit Claudiu Belu proposed openstack/nova master: tests: autospecs all the mock.patch usages https://review.openstack.org/470775
15:10:45 melwitt .
15:12:08 mriedem s10: commented in that patch
15:12:34 mriedem nicolasbock: so live migration was successful but something failed in post like mdbooth is mentioning
15:12:48 nicolasbock Ok
15:12:49 mriedem nicolasbock: you'll likely need to manually update the instances.host value in the db then for that instance
15:13:06 mriedem otherwise nova-compute on the source host is going to continue thinking it owns the instance
15:14:02 s10 mriedem: thank you
15:14:24 nicolasbock Ok, other than that this sounds mildly scary, could you give me a pointer where I find that value mriedem ?
15:14:53 mriedem do you know where the guest is actively running now?
15:14:57 openstackgerrit Merged openstack/nova stable/ocata: [Stable Only] Add amd-ssbd and amd-no-ssb CPU flags https://review.openstack.org/607296
15:15:03 nicolasbock Yes
15:15:05 mriedem it should be in the last live-migration migration record for the instance
15:15:14 nicolasbock Ok
15:15:16 mriedem well then you just update the table record in the nova db
15:16:00 mriedem update instances set host=<host> where uuid=<instance uuid>;
15:16:01 nicolasbock Ok, sounds so straightforward when you put it like that ;)
15:16:02 nicolasbock I'll give it a try
15:16:08 nicolasbock Thanks!
15:19:04 openstackgerrit Vlad Gusev proposed openstack/nova master: Not set instance to ERROR if set_admin_password failed https://review.openstack.org/555160
15:19:04 sean-k-mooney mriedem: out of interest what would happen if you tried to do a hard reboot or other lifecycle action on a vm in this state with the wrong host set
15:19:26 sean-k-mooney woudl it repare the instance or try to start it on the wrong host?
15:21:06 mriedem it would try to start it on the wrong host
15:21:44 mriedem i do'nt know what would then happen - would you get the same instance running on two hosts? or a domain not found from the wrong host when trying to reboot it?
15:21:47 mriedem i'd hope the latter
15:21:51 sean-k-mooney right which if it was using share storage could lead to data curoption correct
15:22:10 mriedem well i'd hope reboot would fail if the guest isn't actually on the hypervisor
15:22:14 sean-k-mooney i would hope the latter too
15:22:23 openstackgerrit Elod Illes proposed openstack/nova stable/ocata: Don't delete neutron port when attach failed https://review.openstack.org/607614
15:23:41 mriedem doesn't look like it would fail though https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L849
15:23:47 mriedem we just handle the not found and assume the guest is already gone
15:24:53 mriedem so maybe there is something to be said for putting hte instance into ERROR state on failed post live migration
15:24:57 mriedem to force the admin to fix it
15:25:02 sean-k-mooney mriedem: right we need that in case where we are usign hard reboot to "fix" things
15:25:05 mriedem so the user doesn't try to reboot the thing and screw it up
15:26:15 sean-k-mooney mriedem: perhaps when i discussed this wit mdbooth previorsly i was suggesting always updating the host to the correct location fo the vm
15:26:53 sean-k-mooney then you could decide if it shoudl stay in error or active state seperately without haveing more bugs if you left it in active
15:27:39 openstackgerrit Claudiu Belu proposed openstack/nova master: hyper-v: autospec classes before they are instantiated https://review.openstack.org/342211
15:28:33 sean-k-mooney mriedem: or to put that another way i think there are two issues in the other case. 1 the host is not upstaed when the vm is moved in some cases and 2 what to do when cleanup fails post migration
15:28:55 mriedem if post live migration set the instance.host to the correct host on which it's running then yeah my concern about the user rebooting it and now having the same guest on different hosts is less of an issue
15:32:08 sean-k-mooney nicolasbock: if you are still around the clip notes version of that conversation is you might want to consider locking the instance untill you have reparied the db to prevent any lifecylce envents
15:33:07 mdbooth mriedem: IIRC my thought at the time was that we should completely update the instance record for the destination immediately after we switch it, then run source cleanup.
15:33:36 mdbooth So if we fail for whatever reason we've recorded that we're running on the dest.
15:37:46 mriedem s10: i think the unit test is going to fail in that patch, see comments for why
15:37:51 mdbooth Incidentally, to reiterate something I said earlier, when this patch landed late last week it made the test_parallel_evacuate_with_server_group about 20 times more likely to occur: https://review.openstack.org/#/c/604859/
15:38:08 mdbooth That test is now failing around 50% for me.
15:38:14 mriedem we can skip the test for now
15:38:20 mriedem while the fix is being reviewed
15:38:44 mdbooth mriedem: ack. Given ^^^ it seems that the test has never been good.
15:40:52 openstackgerrit Rodolfo Alonso Hernandez proposed openstack/os-vif master: Add native implementation OVSDB API https://review.openstack.org/482226
15:43:09 openstackgerrit Matt Riedemann proposed openstack/nova master: Skip test_parallel_evacuate_with_server_group until fixed https://review.openstack.org/607620
15:43:19 mriedem dansmith: efried: mdbooth: ^ gives time to be comfortable with the fix
15:44:21 mdbooth mriedem: ack.
15:44:22 efried mriedem: What will the criteria for comfort be?
15:45:05 efried mriedem: https://review.openstack.org/#/c/605436/ has three +1s and a +2
15:45:33 mriedem i guess that's up to whoever +Ws it
15:46:12 efried mriedem: Well, I'm comfortable that it fixes the problem. But don't feel confident enough in the actual code change to +W. I would think someone like.... mriedem would be able to have that confidence.
15:47:47 mriedem i'm not very confident in anything atm
15:48:28 dansmith I think that change needs a lot of inspection
15:48:35 dansmith which I can't do right this moment

Earlier   Later