| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-10-03 | |||
| 14:33:01 | sean-k-mooney | sounds like a nice low hangin fruit bug | |
| 14:33:46 | efried | nicolasbock: Okay, so I think you're going to want to build your command with: | |
| 14:33:46 | efried | --allocation rp=6cbb84b0-02f4-4ee3-9df2-151475b1effe,MEMORY_MB=8192 \ | |
| 14:33:46 | efried | --allocation rp=6cbb84b0-02f4-4ee3-9df2-151475b1effe,VCPU=4 \ | |
| 14:33:46 | efried | --allocation rp=6cbb84b0-02f4-4ee3-9df2-151475b1effe,DISK_GB=80 | |
| 14:33:52 | mriedem | ha | |
| 14:33:57 | mriedem | "InstancePasswordSetFailed: Failed to set admin password on | |
| 14:33:57 | mriedem | 9f9330c2-4ab4-45f1-a9f9-2770dd34cf30 because error setting admin password" | |
| 14:34:00 | nicolasbock | Ah ok | |
| 14:34:03 | mriedem | "we failed because we failed" | |
| 14:34:06 | efried | mriedem: duh | |
| 14:34:21 | nicolasbock | Let me try that | |
| 14:34:25 | mriedem | s10: i don't know why the instance is put into ERROR state there, i want to say i've seen a patch to remove that | |
| 14:34:58 | efried | nicolasbock: Note that's gotta be all in one command. Otherwise you'll end up with an instance with just disk :) | |
| 14:35:26 | s10 | mriedem: yes, I see, there is https://review.openstack.org/#/c/555160/ | |
| 14:35:58 | nicolasbock | Good point efried :) | |
| 14:36:02 | mriedem | efried: nicolasbock: might be sensible to have an "openstack resource provider allocation class set" similar to the inventory class set CLI https://docs.openstack.org/osc-placement/latest/cli/index.html#resource-provider-inventory-class-set | |
| 14:36:02 | nicolasbock | So the command worked | |
| 14:36:20 | mriedem | ^ allows you to set inventory on a provider for a specific class, not replace the entire set of inventory for the provider | |
| 14:36:35 | nicolasbock | But now I have https://pastebin.com/KrcWAXbF | |
| 14:36:46 | mriedem | that uses https://developer.openstack.org/api-ref/placement/#update-resource-provider-inventory | |
| 14:36:46 | sean-k-mooney | efried: that is proably another reason to have an edit command since this all needs to be done atomicly | |
| 14:37:00 | mriedem | we don't have an api like that for allocations, which is why there isn't a CLI for it | |
| 14:37:17 | mriedem | we just have https://developer.openstack.org/api-ref/placement/#update-allocations | |
| 14:37:24 | openstackgerrit | Merged openstack/nova stable/rocky: Ignore VirtDriverNotReady in _sync_power_states periodic task https://review.openstack.org/605533 | |
| 14:37:34 | mriedem | but we could easily write a command that just updates one of the resource classes within the existing allocations | |
| 14:37:38 | nicolasbock | Yes that sounds sensible mriedem | |
| 14:37:59 | efried | nicolasbock: Oh, interesting. That's... probably a bug. | |
| 14:38:10 | nicolasbock | :) | |
| 14:38:23 | mriedem | i very much doubt osc-placement handles consumer generations yet, so it could be racy for the CLI to orchestrate this | |
| 14:38:28 | nicolasbock | I should remove the old allocation, right? | |
| 14:38:30 | mriedem | but that's probably a low risk | |
| 14:38:59 | efried | nicolasbock: Yeah, except the only way to do that is openstack resource provider allocation delete $instance_uuid which (I sincerely hope) removes all of them. | |
| 14:39:26 | efried | nicolasbock: actually what may have happened is that the source host still thinks it has the instance, and it "healed" the allocations. | |
| 14:39:28 | nicolasbock | All of them? | |
| 14:39:34 | efried | That would be something to look in the logs for. | |
| 14:39:46 | nicolasbock | Ok | |
| 14:39:48 | mriedem | do you have ocata computes? | |
| 14:39:59 | nicolasbock | But if it removes all of them, wouldn't that be bad? | |
| 14:40:14 | efried | nicolasbock: Well, if you remove all of them, then you can run your 'set' command to restore the proper ones. | |
| 14:40:15 | efried | But | |
| 14:40:19 | mriedem | if you have ocata computes, the resource tracker is reporting the allocations it thinks exist to placement | |
| 14:40:34 | efried | if my suspicion is correct, once you delete all the allocations and wait a minute, the original (source) allocations will magically reappear. | |
| 14:41:04 | efried | okay, so mriedem that would explain the source allocs magically reappearing? | |
| 14:42:59 | nicolasbock | mriedem: This is using Newton | |
| 14:43:09 | nicolasbock | I'll try to delete the allocation | |
| 14:43:15 | nicolasbock | And wait to see what happens :) | |
| 14:43:48 | mriedem | newton/ocata computes will recreate allocations yes | |
| 14:44:14 | mriedem | until you get everything upgraded to >= pike, the resource tracker periodic task in the compute service will try to manage allocations | |
| 14:44:18 | nicolasbock | The new allocation was deleted while we were chatting | |
| 14:44:53 | nicolasbock | Interesting mriedem | |
| 14:45:14 | nicolasbock | But where is the periodic task getting its information from? | |
| 14:46:32 | mriedem | the instances it thinks are running on that host, | |
| 14:46:36 | mriedem | and those instances flavors | |
| 14:46:56 | nicolasbock | Is there a way to update that? | |
| 14:46:58 | mriedem | so if compute host A thinks instance B is running on it with a flavor that uses x,y,z vcpu/ram/disk, it's going to report that | |
| 14:47:08 | mriedem | update what? | |
| 14:47:20 | nicolasbock | So I would have to convince the compute host that it's not running the instance? | |
| 14:47:24 | efried | nicolasbock: I kind of missed how we got into this situation. What makes you think the instance was successfully removed from the source host? | |
| 14:47:56 | mriedem | nicolasbock: is the instance.host in the db pointing at that host? | |
| 14:47:57 | nicolasbock | I am going by what `openstack server show` is telling me :) | |
| 14:48:20 | mriedem | server show should also tell you yeah | |
| 14:48:21 | openstackgerrit | Matthew Booth proposed openstack/nova master: Run evacuate tests with local/lvm and shared/rbd storage https://review.openstack.org/604400 | |
| 14:49:16 | nicolasbock | So `server show` is reporting an incorrect hypervisor | |
| 14:49:18 | mriedem | this is where the RT gets the instances it thinks are running on it https://github.com/openstack/nova/blob/newton-eol/nova/compute/resource_tracker.py#L556 | |
| 14:49:56 | openstackgerrit | Vlad Gusev proposed openstack/nova master: Not instance to ERROR if set_admin_password failed https://review.openstack.org/555160 | |
| 14:50:01 | sean-k-mooney | mriedem: there is a live migration edgecase that mdbooth was looking at a few weeks ago where post migrate source failed and we would not update the host the vm was running on | |
| 14:50:23 | sean-k-mooney | but the vm has actully been moved correectly | |
| 14:52:37 | mriedem | nicolasbock: so did you live migrate this vm or something? why is nova reporting its on the wrong host? | |
| 14:54:07 | mdbooth | Ah, yes. I do recall a bug with that. If we get an error in cleanup on the source host, called *post* successful migration, we then rollback the migration and put the instance in an error state, but it's still running fine on the destination. | |
| 14:54:32 | mdbooth | So, e.g. if you get an error in terminate_connection or whatever, you get in this state | |
| 14:54:53 | mdbooth | And you can't clean it up, because instance.host is pointing to the source, but it's actually running on the dest | |
| 14:55:16 | mriedem | terminate_connection as in cleaning up source node volume attachments and such right? | |
| 14:55:20 | mriedem | same with ports i'm sure | |
| 14:55:27 | mriedem | post live migration cleaning up the source | |
| 14:55:31 | mdbooth | mriedem: Right. Any cleanup on the source | |
| 14:55:40 | mriedem | we should just catch and log cleanup failures | |
| 14:55:42 | openstackgerrit | Vlad Gusev proposed openstack/nova master: Not set instance to ERROR if set_admin_password failed https://review.openstack.org/555160 | |
| 14:57:07 | mdbooth | mriedem: Right. The error in my view was that we put the instance in an error state, when the instance was fine. We should put the migration in an error state, but leave the instance alone. | |
| 14:57:40 | mdbooth | And also do as much cleanup as possible in the presence of errors. | |
| 14:57:46 | nicolasbock | mriedem: Yes I think that's what happened | |
| 15:07:11 | openstackgerrit | Jan Gutter proposed openstack/nova-specs master: Spec to implement vRouter HW offloads https://review.openstack.org/567148 | |
| 15:07:12 | openstackgerrit | Jan Gutter proposed openstack/nova-specs master: Spec to implement generic HW offloads for os-vif https://review.openstack.org/607610 | |
| 15:09:30 | openstackgerrit | Claudiu Belu proposed openstack/nova master: tests: autospecs all the mock.patch usages https://review.openstack.org/470775 | |
| 15:10:45 | melwitt | . | |
| 15:12:08 | mriedem | s10: commented in that patch | |
| 15:12:34 | mriedem | nicolasbock: so live migration was successful but something failed in post like mdbooth is mentioning | |
| 15:12:48 | nicolasbock | Ok | |
| 15:12:49 | mriedem | nicolasbock: you'll likely need to manually update the instances.host value in the db then for that instance | |
| 15:13:06 | mriedem | otherwise nova-compute on the source host is going to continue thinking it owns the instance | |
| 15:14:02 | s10 | mriedem: thank you | |
| 15:14:24 | nicolasbock | Ok, other than that this sounds mildly scary, could you give me a pointer where I find that value mriedem ? | |
| 15:14:53 | mriedem | do you know where the guest is actively running now? | |
| 15:14:57 | openstackgerrit | Merged openstack/nova stable/ocata: [Stable Only] Add amd-ssbd and amd-no-ssb CPU flags https://review.openstack.org/607296 | |
| 15:15:03 | nicolasbock | Yes | |
| 15:15:05 | mriedem | it should be in the last live-migration migration record for the instance | |
| 15:15:14 | nicolasbock | Ok | |
| 15:15:16 | mriedem | well then you just update the table record in the nova db | |
| 15:16:00 | mriedem | update instances set host=<host> where uuid=<instance uuid>; | |
| 15:16:01 | nicolasbock | Ok, sounds so straightforward when you put it like that ;) | |