| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-10-03 | |||
| 14:20:57 | mnaser | if you use lvm on centos with nova, the volumes are created under user 'root' | |
| 14:21:04 | mnaser | so qemu process cant touch them and it cant boot | |
| 14:21:12 | mnaser | is this technically a nova bug? | |
| 14:21:41 | mnaser | (aka the devices on the system /dev/vg_foo/vmuuid_disk are root:root, qemu-kvm runs on qemu:qemu) | |
| 14:23:05 | nicolasbock | Hi efried I am still here | |
| 14:25:29 | efried | nicolasbock: So looking at the manual (https://docs.openstack.org/osc-placement/latest/cli/index.html#resource-provider-allocation-set) it appears as though you're going to want multiple --alocation params... | |
| 14:25:42 | s10 | mriedem: Can we remove line https://github.com/openstack/nova/blob/stable/rocky/nova/compute/manager.py#L3669 ? It causes bug https://bugs.launchpad.net/nova/+bug/1746972 . I don't believe that error in set-admin-password on running instance should put instance in error state and require cloud admin intervention to reset instances state or that user should abandon and remove this vm. | |
| 14:25:42 | openstack | Launchpad bug 1746972 in OpenStack Compute (nova) "After setting the password failed, the VM state is set to error" [Undecided,Confirmed] | |
| 14:25:54 | nicolasbock | Yes, that's my reading too efried | |
| 14:26:03 | nicolasbock | But I don't understand what the other parameters should look like | |
| 14:26:07 | efried | nicolasbock: I *think* each should look like --allocation rp=$rp_uuid,$rc=$amount | |
| 14:26:10 | dansmith | mnaser: what do you want nova to do? chown them? afaik, it doesn't know what user qemu will run as | |
| 14:26:23 | openstackgerrit | Merged openstack/nova stable/pike: nova-status - don't count deleted compute_nodes https://review.openstack.org/604788 | |
| 14:27:00 | nicolasbock | Ok. What I don't get is why I need a resource-class in there as well. I don't want anything to update in terms of resource classes | |
| 14:27:26 | efried | nicolasbock: Oh, but you do :) | |
| 14:27:34 | nicolasbock | I do? | |
| 14:27:47 | efried | nicolasbock: I guess it's obvious to me because I know what the REST payload looks like, but come to think of it, it makes sense how you're thinking about it. | |
| 14:28:12 | nicolasbock | Maybe I am not looking at resource classes correctly | |
| 14:28:20 | efried | nicolasbock: See, the allocations in the API are a hierarchical structure like resource provider => resource class => amount | |
| 14:28:27 | nicolasbock | But the way I am thinking about them is that they specify things like memory and CPU cores | |
| 14:28:35 | nicolasbock | Ok | |
| 14:28:55 | efried | And also the CLI (and the API it's using) is designed to fully *replace* allocations, not like edit pieces of them. | |
| 14:29:00 | efried | So given that... | |
| 14:29:10 | efried | nicolasbock: You should do `openstack resource provider allocation show $instance_uuid` | |
| 14:29:23 | efried | Which should give you allocations in three-ish resource classes | |
| 14:29:34 | efried | nicolasbock: can you pastebin me that output? | |
| 14:29:59 | dansmith | you know what would be awesome | |
| 14:30:15 | dansmith | openstack resource provider allocation edit <uuid> | |
| 14:30:17 | nicolasbock | https://pastebin.com/sFdzQuPE | |
| 14:30:20 | dansmith | like virsh edit | |
| 14:30:45 | openstackgerrit | Matt Riedemann proposed openstack/nova master: doc: fix and clarify --block-device usage in user docs https://review.openstack.org/607589 | |
| 14:31:03 | sean-k-mooney | dansmith: that could be done as a client only feature but yes that would be nice | |
| 14:31:12 | dansmith | sean-k-mooney: obviously client-only | |
| 14:31:58 | sean-k-mooney | well i was debating if you wuld want to put it in th openstack sdk or just the osc plugin | |
| 14:32:11 | sean-k-mooney | but ya i think just in the plugin | |
| 14:32:12 | dansmith | oh, I meant just in the plugin | |
| 14:32:14 | dansmith | yeah | |
| 14:32:30 | dansmith | and you could translate to/from yaml for the actual editing maybe | |
| 14:32:35 | dansmith | so people aren't having to hand-edit json | |
| 14:32:45 | dansmith | since you need to validate the schema before you send it back anyway | |
| 14:33:00 | mnaser | dansmith: yeah, that why i don't think it's a nova problem but maybe something that we should document.. or libvirt should | |
| 14:33:01 | sean-k-mooney | sounds like a nice low hangin fruit bug | |
| 14:33:46 | efried | nicolasbock: Okay, so I think you're going to want to build your command with: | |
| 14:33:46 | efried | --allocation rp=6cbb84b0-02f4-4ee3-9df2-151475b1effe,MEMORY_MB=8192 \ | |
| 14:33:46 | efried | --allocation rp=6cbb84b0-02f4-4ee3-9df2-151475b1effe,VCPU=4 \ | |
| 14:33:46 | efried | --allocation rp=6cbb84b0-02f4-4ee3-9df2-151475b1effe,DISK_GB=80 | |
| 14:33:52 | mriedem | ha | |
| 14:33:57 | mriedem | "InstancePasswordSetFailed: Failed to set admin password on | |
| 14:33:57 | mriedem | 9f9330c2-4ab4-45f1-a9f9-2770dd34cf30 because error setting admin password" | |
| 14:34:00 | nicolasbock | Ah ok | |
| 14:34:03 | mriedem | "we failed because we failed" | |
| 14:34:06 | efried | mriedem: duh | |
| 14:34:21 | nicolasbock | Let me try that | |
| 14:34:25 | mriedem | s10: i don't know why the instance is put into ERROR state there, i want to say i've seen a patch to remove that | |
| 14:34:58 | efried | nicolasbock: Note that's gotta be all in one command. Otherwise you'll end up with an instance with just disk :) | |
| 14:35:26 | s10 | mriedem: yes, I see, there is https://review.openstack.org/#/c/555160/ | |
| 14:35:58 | nicolasbock | Good point efried :) | |
| 14:36:02 | mriedem | efried: nicolasbock: might be sensible to have an "openstack resource provider allocation class set" similar to the inventory class set CLI https://docs.openstack.org/osc-placement/latest/cli/index.html#resource-provider-inventory-class-set | |
| 14:36:02 | nicolasbock | So the command worked | |
| 14:36:20 | mriedem | ^ allows you to set inventory on a provider for a specific class, not replace the entire set of inventory for the provider | |
| 14:36:35 | nicolasbock | But now I have https://pastebin.com/KrcWAXbF | |
| 14:36:46 | mriedem | that uses https://developer.openstack.org/api-ref/placement/#update-resource-provider-inventory | |
| 14:36:46 | sean-k-mooney | efried: that is proably another reason to have an edit command since this all needs to be done atomicly | |
| 14:37:00 | mriedem | we don't have an api like that for allocations, which is why there isn't a CLI for it | |
| 14:37:17 | mriedem | we just have https://developer.openstack.org/api-ref/placement/#update-allocations | |
| 14:37:24 | openstackgerrit | Merged openstack/nova stable/rocky: Ignore VirtDriverNotReady in _sync_power_states periodic task https://review.openstack.org/605533 | |
| 14:37:34 | mriedem | but we could easily write a command that just updates one of the resource classes within the existing allocations | |
| 14:37:38 | nicolasbock | Yes that sounds sensible mriedem | |
| 14:37:59 | efried | nicolasbock: Oh, interesting. That's... probably a bug. | |
| 14:38:10 | nicolasbock | :) | |
| 14:38:23 | mriedem | i very much doubt osc-placement handles consumer generations yet, so it could be racy for the CLI to orchestrate this | |
| 14:38:28 | nicolasbock | I should remove the old allocation, right? | |
| 14:38:30 | mriedem | but that's probably a low risk | |
| 14:38:59 | efried | nicolasbock: Yeah, except the only way to do that is openstack resource provider allocation delete $instance_uuid which (I sincerely hope) removes all of them. | |
| 14:39:26 | efried | nicolasbock: actually what may have happened is that the source host still thinks it has the instance, and it "healed" the allocations. | |
| 14:39:28 | nicolasbock | All of them? | |
| 14:39:34 | efried | That would be something to look in the logs for. | |
| 14:39:46 | nicolasbock | Ok | |
| 14:39:48 | mriedem | do you have ocata computes? | |
| 14:39:59 | nicolasbock | But if it removes all of them, wouldn't that be bad? | |
| 14:40:14 | efried | nicolasbock: Well, if you remove all of them, then you can run your 'set' command to restore the proper ones. | |
| 14:40:15 | efried | But | |
| 14:40:19 | mriedem | if you have ocata computes, the resource tracker is reporting the allocations it thinks exist to placement | |
| 14:40:34 | efried | if my suspicion is correct, once you delete all the allocations and wait a minute, the original (source) allocations will magically reappear. | |
| 14:41:04 | efried | okay, so mriedem that would explain the source allocs magically reappearing? | |
| 14:42:59 | nicolasbock | mriedem: This is using Newton | |
| 14:43:09 | nicolasbock | I'll try to delete the allocation | |
| 14:43:15 | nicolasbock | And wait to see what happens :) | |
| 14:43:48 | mriedem | newton/ocata computes will recreate allocations yes | |
| 14:44:14 | mriedem | until you get everything upgraded to >= pike, the resource tracker periodic task in the compute service will try to manage allocations | |
| 14:44:18 | nicolasbock | The new allocation was deleted while we were chatting | |
| 14:44:53 | nicolasbock | Interesting mriedem | |
| 14:45:14 | nicolasbock | But where is the periodic task getting its information from? | |
| 14:46:32 | mriedem | the instances it thinks are running on that host, | |
| 14:46:36 | mriedem | and those instances flavors | |
| 14:46:56 | nicolasbock | Is there a way to update that? | |
| 14:46:58 | mriedem | so if compute host A thinks instance B is running on it with a flavor that uses x,y,z vcpu/ram/disk, it's going to report that | |
| 14:47:08 | mriedem | update what? | |
| 14:47:20 | nicolasbock | So I would have to convince the compute host that it's not running the instance? | |
| 14:47:24 | efried | nicolasbock: I kind of missed how we got into this situation. What makes you think the instance was successfully removed from the source host? | |