| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-11-20 | |||
| 09:48:49 | swp20 | hi guys, i meet a question, when i use 'virsh attach-device gpu.xml domain --persistent --live' to attach device to the running guest, | |
| 09:49:58 | swp20 | and then detach the device with 'virsh detach-device domain gpu.xml --persistent --live' to detach device to the guest, | |
| 09:51:41 | swp20 | even through these actions are successful, and again attach-device to the guest, libvirt raise 'pci device in use by driver qemu'. | |
| 09:53:34 | swp20 | do you meet this problem, and any suggestion? | |
| 10:00:38 | kashyap | QianbiaoNG: I think it might be the https://freeipa.org project ("Identity, Policy, Audit"). I haven't clicked on your URL | |
| 10:01:31 | QianbiaoNG | kashyap ty, seems so. | |
| 10:01:47 | kashyap | QianbiaoNG: Actually, I'm sure it is talking about the FreeIPA server instances. | |
| 10:02:09 | QianbiaoNG | another question: is there an official project provided for implement a nova vendor data metadata service. | |
| 10:03:29 | kashyap | swp20: No answer top off my head, but if you're not using Nova, and directly `virsh`, you can get better help on #virt (OFTC network) | |
| 10:03:34 | kashyap | s/can/might/ | |
| 10:08:19 | swp20 | kashyap: thanks, i joined and no person there in virt(OFTC network). | |
| 10:09:02 | kashyap | swp20: I think you're still on Freenode. I see more than 100 people :) | |
| 10:10:11 | kashyap | swp20: You need to use the '/connect' command to join the OFTC server, and then '/join #virt' | |
| 11:33:20 | admin0 | clue of ceph2 .. so it only has ceph details of the first ceph1 cluster only .. | |
| 11:33:20 | admin0 | good morning .. i have nova, cinder and glance setup using ceph1 .. works good.. now I added ceph2 for only cinder ( so that cinder has 2 glance backends) one type ssd and one type hdd .. i am able to create the volumes in both hdd and sdd.. i can mount sdd fine .. but when I try to mount hdd, i get: libvirt.libvirtError: internal error: unable to execute QEMU command 'blockdev-add': error connecting: Permission denied" .. the hypervisor has no | |
| 11:33:53 | admin0 | so question is, what config changes i need to do or what needs to be done so that i can mount the volume from the 2nd ceph to the instance | |
| 11:45:56 | openstackgerrit | Lee Yarwood proposed openstack/nova-specs master: Image and flavor defined ephemeral storage encryption https://review.opendev.org/752284 | |
| 11:51:51 | sean-k-mooney | admin0: its likely an issue with your cephx keys and the pool permeions for the nova user on the ceph clusters | |
| 11:52:13 | openstackgerrit | Stephen Finucane proposed openstack/nova-specs master: Update modernize-os-hypervisors-api spec https://review.opendev.org/763043 | |
| 11:55:54 | stephenfin | bauzas: I'm happy with https://review.opendev.org/#/c/759828/ now, but I'm holding off on +2 to give you a chance to revisit it if you want | |
| 12:10:30 | admin0 | sean-k-mooney, so hypervisor does not need to have the ceph2 keys or settings ? | |
| 12:10:36 | admin0 | it offloads to cinder ? | |
| 12:11:09 | admin0 | sean-k-mooney, so this means ceph2 also needs a nova user ( which is used to mount ) | |
| 12:15:23 | sean-k-mooney | i think it has to have both but lyarwood might be better able to respond | |
| 12:15:33 | sean-k-mooney | its been a while since i deploy ceph | |
| 12:23:23 | lyarwood | admin0: I don't think we support multiple clusters in Nova | |
| 12:23:38 | sean-k-mooney | lyarwood: we dont for rbd | |
| 12:23:47 | sean-k-mooney | lyarwood: bute we should via cinder | |
| 12:23:59 | sean-k-mooney | *rbd images backend | |
| 12:24:20 | lyarwood | sean-k-mooney: the user is still configured locally on the computes for volumes | |
| 12:24:43 | sean-k-mooney | right dont you have to configre the same user on all ceph clusters | |
| 12:24:57 | sean-k-mooney | with the same cephx key | |
| 12:24:58 | lyarwood | sean-k-mooney: with the same key | |
| 12:25:04 | lyarwood | and I'm not sure you can do that can you? | |
| 12:25:15 | lyarwood | I thought the keys were unique to a cluster | |
| 12:25:24 | lyarwood | but that might be a lack of knownledge on my side | |
| 12:25:39 | sean-k-mooney | im not sure i have only done it with one ceph personally | |
| 12:26:00 | sean-k-mooney | but i know people do dpleoy multiple ceph clusters | |
| 12:26:11 | sean-k-mooney | but im not sure if that is 1 per AZ or something like that | |
| 12:26:38 | sean-k-mooney | https://ceph.io/geen-categorie/openstack-nova-configure-multiple-ceph-backends-on-one-hypervisor/ | |
| 12:26:42 | lyarwood | yeah, I think the normal way to do with admin0 has done would be to use different pools | |
| 12:27:33 | sean-k-mooney | how the hell does ^ work | |
| 12:27:38 | lyarwood | yeah that's single cluster multiple pools | |
| 12:27:53 | lyarwood | and aggregates to keep them apart | |
| 12:28:17 | sean-k-mooney | no i mesn once nova compute with 2 files witht he same keys and different values | |
| 12:28:25 | sean-k-mooney | nova-compute --config-file /etc/nova/nova-ceph1.conf --config-file /etc/nova/nova-ceph2.conf | |
| 12:29:03 | lyarwood | I think that's a mistake in the write up tbh | |
| 12:29:12 | lyarwood | given what they are trying to do | |
| 12:30:08 | sean-k-mooney | they are trying to run two nova computes on the same hyperviour with different host values but they same hypervior hostname | |
| 12:31:03 | lyarwood | ew | |
| 12:31:43 | sean-k-mooney | but 1 if that comand even works it really should not and 2 that si really going to screw up placemnt | |
| 12:32:17 | sean-k-mooney | admin0: so dont do what in that ceph guide. | |
| 12:34:13 | openstackgerrit | Lucian Petrut proposed openstack/nova master: hyper-v rbd volume support https://review.opendev.org/763550 | |
| 12:34:17 | sean-k-mooney | ah found an actully sane example | |
| 12:34:20 | sean-k-mooney | https://medium.com/walmartglobaltech/deploying-cinder-with-multiple-ceph-cluster-backends-2cd90d64b10 | |
| 12:34:26 | sean-k-mooney | lyarwood: admin0 ^ | |
| 12:34:36 | sean-k-mooney | so the user has to be different on each cluster | |
| 12:35:56 | sean-k-mooney | actully no that is still a hack | |
| 12:40:28 | lyarwood | sean-k-mooney: hmm let me check the connection_info returned by c-vol, I might be missing something here | |
| 12:41:08 | sean-k-mooney | if you used differnt users e.g. nova-<cluster-name> | |
| 12:41:17 | sean-k-mooney | then you could have different cephx keys | |
| 12:41:32 | sean-k-mooney | 1 per user and it should work i think | |
| 12:41:46 | sean-k-mooney | but i did not like there add a libvirt secret thing | |
| 12:41:57 | sean-k-mooney | that was the hack in my view | |
| 12:41:57 | lyarwood | https://github.com/openstack/cinder/blob/30501b9ecd6d4ed8dd6405be01795d8634013b24/cinder/volume/drivers/rbd.py#L1438-L1453 | |
| 12:42:02 | lyarwood | https://github.com/openstack/nova/blob/e16800cc0aebf1174c5c0b6c4b043b09622524e9/nova/virt/libvirt/volume/net.py#L37-L76 | |
| 12:42:09 | lyarwood | you can have multiple secrets | |
| 12:42:24 | lyarwood | as long as the UUIDs are copied on the computes | |
| 12:42:27 | admin0 | the cinder part is good .. cinder can create volumes in both ceph .. only part is nova works with its default one and has no idea/clue about 2nd one | |
| 12:42:53 | sean-k-mooney | ya so they were manually defineing the second secret on the compute nodes | |
| 12:43:20 | lyarwood | that's fine | |
| 12:43:31 | lyarwood | admin0: apologies so I forgot about https://github.com/openstack/nova/blob/e16800cc0aebf1174c5c0b6c4b043b09622524e9/nova/virt/libvirt/volume/net.py#L37-L76 | |
| 12:43:56 | lyarwood | admin0: so c-vol needs to return connection_info with {"auth_enabled": True} | |
| 12:45:29 | sean-k-mooney | is that enough or will you need to maulaly add the persistint key first. i think that would be required yes? | |
| 12:45:41 | sean-k-mooney | that a one time operation on all the compute nodes | |
| 12:46:18 | lyarwood | sean-k-mooney: correct you still need to populate the secret on the computes | |
| 12:46:28 | lyarwood | admin0: and thinking about it that's what you have likely missed here | |
| 12:46:47 | lyarwood | admin0: you need two ceph secrets in libvirt per compute with these two clusters | |
| 12:47:36 | sean-k-mooney | this fells like something that could be imporved with a specless blueprint to allow the config options to have multiple vlaues | |
| 12:48:26 | lyarwood | sean-k-mooney: I'd rather drop the use of the config here and just rely on cinder | |
| 12:48:29 | lyarwood | sean-k-mooney: for volumes | |
| 12:49:07 | sean-k-mooney | well the config options were for the images_backend really | |
| 12:49:17 | sean-k-mooney | but i ment for createing the secret in libvirt | |
| 12:49:39 | sean-k-mooney | i guess we coudl drop that and make it just a deployment step | |
| 12:49:42 | stephenfin | lyarwood: Could you create a bug report for https://review.opendev.org/#/c/761730/ if you're planning to backport it? If you're not, I'll approve now | |
| 12:51:22 | lyarwood | stephenfin: yup can do | |
| 12:51:51 | lyarwood | sean-k-mooney: we can't automate that so it's always going to be a deployment step | |
| 12:52:15 | lyarwood | brb | |
| 12:53:45 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Prevent nova-compute start with DB credentials https://review.opendev.org/762176 | |
| 12:57:48 | admin0 | This uuid must be different than any existing uuid’s used by cinder/libvirt -- so i just generate a new uuid on the fly ? | |
| 12:58:19 | lyarwood | yeah it just needs to be the same across both the computes and host where cinder is running | |
| 12:58:45 | lyarwood | just use uuidgen | |
| 13:02:50 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Prevent nova-compute start with DB credentials https://review.opendev.org/762176 | |
| 13:03:03 | admin0 | lyarwood, this is what i have in cinder.conf -- https://gist.github.com/a1git/e8edfc7c0f148085165d4b8589d3f5a7 | |
| 13:03:08 | admin0 | i don't see any uuid in there | |
| 13:05:13 | sean-k-mooney | admin0: i think its in your ceph.conf | |
| 13:05:24 | admin0 | its not there | |
| 13:05:36 | admin0 | the uuid is what matches virsh secret-list | |
| 13:05:54 | admin0 | but does not exist on ceph or cinder side | |
| 13:07:07 | sean-k-mooney | [ceph] | |