| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-11-20 | |||
| 12:24:58 | lyarwood | sean-k-mooney: with the same key | |
| 12:25:04 | lyarwood | and I'm not sure you can do that can you? | |
| 12:25:15 | lyarwood | I thought the keys were unique to a cluster | |
| 12:25:24 | lyarwood | but that might be a lack of knownledge on my side | |
| 12:25:39 | sean-k-mooney | im not sure i have only done it with one ceph personally | |
| 12:26:00 | sean-k-mooney | but i know people do dpleoy multiple ceph clusters | |
| 12:26:11 | sean-k-mooney | but im not sure if that is 1 per AZ or something like that | |
| 12:26:38 | sean-k-mooney | https://ceph.io/geen-categorie/openstack-nova-configure-multiple-ceph-backends-on-one-hypervisor/ | |
| 12:26:42 | lyarwood | yeah, I think the normal way to do with admin0 has done would be to use different pools | |
| 12:27:33 | sean-k-mooney | how the hell does ^ work | |
| 12:27:38 | lyarwood | yeah that's single cluster multiple pools | |
| 12:27:53 | lyarwood | and aggregates to keep them apart | |
| 12:28:17 | sean-k-mooney | no i mesn once nova compute with 2 files witht he same keys and different values | |
| 12:28:25 | sean-k-mooney | nova-compute --config-file /etc/nova/nova-ceph1.conf --config-file /etc/nova/nova-ceph2.conf | |
| 12:29:03 | lyarwood | I think that's a mistake in the write up tbh | |
| 12:29:12 | lyarwood | given what they are trying to do | |
| 12:30:08 | sean-k-mooney | they are trying to run two nova computes on the same hyperviour with different host values but they same hypervior hostname | |
| 12:31:03 | lyarwood | ew | |
| 12:31:43 | sean-k-mooney | but 1 if that comand even works it really should not and 2 that si really going to screw up placemnt | |
| 12:32:17 | sean-k-mooney | admin0: so dont do what in that ceph guide. | |
| 12:34:13 | openstackgerrit | Lucian Petrut proposed openstack/nova master: hyper-v rbd volume support https://review.opendev.org/763550 | |
| 12:34:17 | sean-k-mooney | ah found an actully sane example | |
| 12:34:20 | sean-k-mooney | https://medium.com/walmartglobaltech/deploying-cinder-with-multiple-ceph-cluster-backends-2cd90d64b10 | |
| 12:34:26 | sean-k-mooney | lyarwood: admin0 ^ | |
| 12:34:36 | sean-k-mooney | so the user has to be different on each cluster | |
| 12:35:56 | sean-k-mooney | actully no that is still a hack | |
| 12:40:28 | lyarwood | sean-k-mooney: hmm let me check the connection_info returned by c-vol, I might be missing something here | |
| 12:41:08 | sean-k-mooney | if you used differnt users e.g. nova-<cluster-name> | |
| 12:41:17 | sean-k-mooney | then you could have different cephx keys | |
| 12:41:32 | sean-k-mooney | 1 per user and it should work i think | |
| 12:41:46 | sean-k-mooney | but i did not like there add a libvirt secret thing | |
| 12:41:57 | lyarwood | https://github.com/openstack/cinder/blob/30501b9ecd6d4ed8dd6405be01795d8634013b24/cinder/volume/drivers/rbd.py#L1438-L1453 | |
| 12:41:57 | sean-k-mooney | that was the hack in my view | |
| 12:42:02 | lyarwood | https://github.com/openstack/nova/blob/e16800cc0aebf1174c5c0b6c4b043b09622524e9/nova/virt/libvirt/volume/net.py#L37-L76 | |
| 12:42:09 | lyarwood | you can have multiple secrets | |
| 12:42:24 | lyarwood | as long as the UUIDs are copied on the computes | |
| 12:42:27 | admin0 | the cinder part is good .. cinder can create volumes in both ceph .. only part is nova works with its default one and has no idea/clue about 2nd one | |
| 12:42:53 | sean-k-mooney | ya so they were manually defineing the second secret on the compute nodes | |
| 12:43:20 | lyarwood | that's fine | |
| 12:43:31 | lyarwood | admin0: apologies so I forgot about https://github.com/openstack/nova/blob/e16800cc0aebf1174c5c0b6c4b043b09622524e9/nova/virt/libvirt/volume/net.py#L37-L76 | |
| 12:43:56 | lyarwood | admin0: so c-vol needs to return connection_info with {"auth_enabled": True} | |
| 12:45:29 | sean-k-mooney | is that enough or will you need to maulaly add the persistint key first. i think that would be required yes? | |
| 12:45:41 | sean-k-mooney | that a one time operation on all the compute nodes | |
| 12:46:18 | lyarwood | sean-k-mooney: correct you still need to populate the secret on the computes | |
| 12:46:28 | lyarwood | admin0: and thinking about it that's what you have likely missed here | |
| 12:46:47 | lyarwood | admin0: you need two ceph secrets in libvirt per compute with these two clusters | |
| 12:47:36 | sean-k-mooney | this fells like something that could be imporved with a specless blueprint to allow the config options to have multiple vlaues | |
| 12:48:26 | lyarwood | sean-k-mooney: I'd rather drop the use of the config here and just rely on cinder | |
| 12:48:29 | lyarwood | sean-k-mooney: for volumes | |
| 12:49:07 | sean-k-mooney | well the config options were for the images_backend really | |
| 12:49:17 | sean-k-mooney | but i ment for createing the secret in libvirt | |
| 12:49:39 | sean-k-mooney | i guess we coudl drop that and make it just a deployment step | |
| 12:49:42 | stephenfin | lyarwood: Could you create a bug report for https://review.opendev.org/#/c/761730/ if you're planning to backport it? If you're not, I'll approve now | |
| 12:51:22 | lyarwood | stephenfin: yup can do | |
| 12:51:51 | lyarwood | sean-k-mooney: we can't automate that so it's always going to be a deployment step | |
| 12:52:15 | lyarwood | brb | |
| 12:53:45 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Prevent nova-compute start with DB credentials https://review.opendev.org/762176 | |
| 12:57:48 | admin0 | This uuid must be different than any existing uuid’s used by cinder/libvirt -- so i just generate a new uuid on the fly ? | |
| 12:58:19 | lyarwood | yeah it just needs to be the same across both the computes and host where cinder is running | |
| 12:58:45 | lyarwood | just use uuidgen | |
| 13:02:50 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Prevent nova-compute start with DB credentials https://review.opendev.org/762176 | |
| 13:03:03 | admin0 | lyarwood, this is what i have in cinder.conf -- https://gist.github.com/a1git/e8edfc7c0f148085165d4b8589d3f5a7 | |
| 13:03:08 | admin0 | i don't see any uuid in there | |
| 13:05:13 | sean-k-mooney | admin0: i think its in your ceph.conf | |
| 13:05:24 | admin0 | its not there | |
| 13:05:36 | admin0 | the uuid is what matches virsh secret-list | |
| 13:05:54 | admin0 | but does not exist on ceph or cinder side | |
| 13:07:07 | sean-k-mooney | [ceph] | |
| 13:07:09 | sean-k-mooney | ... | |
| 13:07:11 | sean-k-mooney | rbd_user = cinder | |
| 13:07:13 | sean-k-mooney | rbd_secret_uuid = 457eb676-33da-42ec-9a8c-9293d545c337 | |
| 13:07:49 | sean-k-mooney | so you need to add rbd_secret_uuid to you your hdd and ssd sections i think | |
| 13:07:57 | sean-k-mooney | in cinder | |
| 13:09:16 | sean-k-mooney | yes https://docs.openstack.org/cinder/rocky/configuration/block-storage/drivers/ceph-rbd-volume-driver.html | |
| 13:10:11 | admin0 | but how come its working for the 1st ceph without this value being in ceph or cinder conf anywhere | |
| 13:10:28 | sean-k-mooney | beacus its usign the value form nova which matches | |
| 13:10:45 | sean-k-mooney | the secret for the first cluster | |
| 13:24:11 | openstackgerrit | Lucian Petrut proposed openstack/nova master: hyper-v rbd volume support https://review.opendev.org/763550 | |
| 13:25:11 | sean-k-mooney | lpetrut: do you have a spec filed for that? | |
| 13:25:20 | sean-k-mooney | or at least a specless blueprint | |
| 13:25:57 | lpetrut | sean-k-mooney: good point, I'm filing blueprints right away | |
| 13:26:00 | sean-k-mooney | lpetrut: we need something to track addign an entire storage protocol to the hyperv driver. | |
| 13:26:15 | lpetrut | sean-k-mooney: definitely | |
| 13:31:21 | lpetrut | sean-k-mooney: I think a specless blueprint should be enough: https://blueprints.launchpad.net/nova/+spec/hyperv-rbd | |
| 13:31:27 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Make nova-compute hard fail on db config https://review.opendev.org/763559 | |
| 13:37:34 | sean-k-mooney | lpetrut: if the chagne was larger it would not be but in this case i might be its really up to gibi and the rest of the core team to decide | |
| 13:38:00 | sean-k-mooney | lpetrut: do you have a third party ci in the works to test this | |
| 13:38:20 | sean-k-mooney | lpetrut: i dont see an os-brick-rbd voting on either change | |
| 13:38:41 | sean-k-mooney | i expect we will want to see that pass before merging this | |
| 13:41:13 | lpetrut | atm we're only testing smb, iscsi and FC volumes but we'll add another os-brick job for RBD | |
| 13:42:42 | sean-k-mooney | and a nova job? ideally tempest full with rbd and hyperv? | |
| 13:44:29 | lpetrut | atm the nova job only covers lvm. if by any chance a nova patch breaks support for one of the volume backends (very unlikely considering the minimal nova code for that), we'll notice it when the os-brick CI starts failing | |
| 13:44:36 | sean-k-mooney | it could be the same job just triggering it on nova | |
| 13:45:27 | gibi | lpetrut: thanks for the bp. Could you join to an upcoming Nova weekly meeting to shortly dicuss your bp and approve it to Wallaby? | |
| 13:45:49 | lpetrut | gibi: definitely | |
| 13:46:06 | gibi | lpetrut: then please add your bp to the OpenDiscussion section of the meeting agenda https://wiki.openstack.org/wiki/Meetings/Nova | |
| 13:46:14 | gibi | so we won't forget it | |
| 13:46:30 | lpetrut | gibi: awesome, thanks for the tip | |
| 13:46:38 | gibi | thanks! | |
| 14:24:29 | bauzas | stephenfin: thanks, looking | |