| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-02-03 | |||
| 16:25:19 | openstack | Launchpad bug 1914296 in OpenStack Compute (nova) "tests involving encrypted volumes sometimes (rarely) fail with libvirtError: [...] Volume is not in LUKS format" [Undecided,New] | |
| 16:26:00 | kashyap | melwitt: Hmm, so I can't see a unique signature either ... as most of the messages are "usual", but they're on loop | |
| 16:26:28 | kashyap | Unless I'm missing something badly, looks like it's tricky to check for its frequency | |
| 16:26:33 | melwitt | kashyap: yeah exactly. the only place where I think we have a chance is in the job-output.txt but IIRC that text is also quite generic | |
| 16:26:57 | melwitt | I will take another look to see if anything jumps out | |
| 16:27:36 | kashyap | melwitt: Thank you. I hope this won't come back to bite us - as we can't (yet) see how often we're hitting it | |
| 16:28:46 | lyarwood | melwitt: can't say I have, *clicks* | |
| 16:29:32 | lyarwood | melwitt: if it's a block device n-cpu should format it with LUKS before using it | |
| 16:30:03 | melwitt | lyarwood: it seems weird bc the error message says both format=luks and then "Volume not in LUKS format", like how is that possible? | |
| 16:31:12 | lyarwood | melwitt: urgh that chrome tab just died, I'll pull down the log after the call if that's okay | |
| 16:31:32 | melwitt | sure np, just wanted to link you before I forgot | |
| 16:31:42 | melwitt | it's not urgent | |
| 17:13:58 | kashyap | melwitt: I have attached the QEMU <-> libvirt exchange from the dest-host in the bug. | |
| 17:14:22 | melwitt | kashyap: thanks! | |
| 17:15:12 | kashyap | NP. Thx, you too | |
| 17:44:48 | sean-k-mooney | lyarwood: can you reveiw https://review.opendev.org/c/openstack/nova/+/622336/29/nova/api/openstack/compute/remote_consoles.py#183 and confirm if you expect that to have teh same effect as the issue we discussed downstream | |
| 17:45:21 | sean-k-mooney | we do not want the password to end up in the image data or in config_dirver/metadata api | |
| 17:45:49 | sean-k-mooney | but here it may end up in both correct | |
| 17:49:38 | melwitt | sean-k-mooney: hm, I'd have thought that should be stored encrypted, same as we do for admin password | |
| 17:49:57 | sean-k-mooney | melwitt: ya i mentioned that | |
| 17:50:08 | sean-k-mooney | although i dont think it can be | |
| 17:50:16 | sean-k-mooney | the admin password is not in the libvirt xml | |
| 17:50:24 | sean-k-mooney | we use the ssh public key to encypt it yes | |
| 17:50:31 | melwitt | yes | |
| 17:50:43 | sean-k-mooney | in this case we wont have the private key to decyprti it ot stick it in the libvirt xml | |
| 17:50:55 | melwitt | oh :\ | |
| 17:50:57 | sean-k-mooney | so we can use the ssh public key here | |
| 17:51:07 | sean-k-mooney | but we coudl use soemthign else | |
| 17:51:11 | sean-k-mooney | e.g. a keystore | |
| 17:51:31 | sean-k-mooney | or static encyption with a fixed key seeded with the ssk public key or somehtihng | |
| 17:52:22 | melwitt | yeah.. I guess I'm slowly remembering this spec, and this aspect of it is "better than nothing" I guess (comparing with no novnc password at all) | |
| 17:52:45 | sean-k-mooney | is fixed key enabled by default | |
| 17:53:07 | sean-k-mooney | because if it is using castalain with fixed key to store it would be better | |
| 17:53:17 | sean-k-mooney | as that would allowy you to plug in barbican to do it proably too | |
| 17:53:28 | sean-k-mooney | that said as long as we call this out | |
| 17:53:39 | sean-k-mooney | and ensure we dont make the password visable in ways we did not intend | |
| 17:53:52 | sean-k-mooney | then i think its ok to improve it later | |
| 17:53:54 | melwitt | I dunno if it is enabled by default. I don't know how to tell | |
| 17:54:20 | sean-k-mooney | the nova keysotre cofnig? thats a thing i think | |
| 17:54:28 | prometheanfire | any progress on updating mock (nova still holds back the 3.0.5 -> 4.0.3 update)? | |
| 17:54:48 | sean-k-mooney | prometheanfire: we were tryihng to stop using it entirly at one point | |
| 17:54:50 | melwitt | yeah there's a fixed_key conf option but it doesn't have a default value, so that means it's not enabled by default? | |
| 17:55:21 | melwitt | prometheanfire: I have a change to the one test that is failing that I need to upload | |
| 17:55:44 | prometheanfire | sean-k-mooney: iirc it was given up on at some point (using the built in) | |
| 17:56:17 | prometheanfire | melwitt: kk, feel free to add me to the review (or ping me) :D | |
| 17:56:18 | sean-k-mooney | prometheanfire: i am not aware of a reason not tooo | |
| 17:56:34 | sean-k-mooney | prometheanfire: i suspect if it was it was just capasity | |
| 17:56:35 | melwitt | will do prometheanfire | |
| 17:56:59 | sean-k-mooney | melwitt: ya not sure the https://docs.openstack.org/nova/latest/configuration/config.html#key-manager config is what i was thinking | |
| 17:57:21 | sean-k-mooney | i was hoptin we had a default fixed key version that was always there if you dont use a real one | |
| 17:57:52 | prometheanfire | sean-k-mooney: well, using the built-in is always preferred | |
| 17:58:14 | melwitt | sean-k-mooney: oh I see | |
| 17:58:17 | sean-k-mooney | prometheanfire: yep we only used mock the lib for python 2 compatiablity | |
| 17:58:58 | sean-k-mooney | melwitt: anyway i dont think we have a basic intree keymanager that implemnt the castalin api that we can just assume is a thing | |
| 17:59:37 | sean-k-mooney | melwitt: well we have https://github.com/openstack/nova/blob/master/nova/keymgr/conf_key_mgr.py | |
| 18:00:02 | lyarwood | sean-k-mooney: sorry was afk, instance.metadata != instance.system_metadata so I'm not sure if it's going to have any impact | |
| 18:00:39 | sean-k-mooney | lyarwood: right but instace.metadata is the place we store metata we set with --property | |
| 18:01:03 | sean-k-mooney | and i though that was put in the the instace_system_metatadata table | |
| 18:01:12 | sean-k-mooney | although it could be in instance_extra | |
| 18:01:15 | lyarwood | sean-k-mooney: the system_metadata issue I was talking about downstream comes from https://github.com/openstack/nova/blob/fefd984fd1294f23677e3df4390e37c7664883a0/nova/utils.py | |
| 18:01:27 | melwitt | isn't instance.metadata public? | |
| 18:01:34 | sean-k-mooney | melwitt: yes | |
| 18:01:36 | sean-k-mooney | with a nova show | |
| 18:01:42 | sean-k-mooney | i think | |
| 18:01:43 | lyarwood | ops that didnt link correctly | |
| 18:01:50 | melwitt | so the novnc password shouldn't be put in there, right? | |
| 18:01:54 | openstackgerrit | Sylvain Bauza proposed openstack/nova master: Add requested_networks field to RequestSpec object https://review.opendev.org/c/openstack/nova/+/749977 | |
| 18:01:54 | openstackgerrit | Sylvain Bauza proposed openstack/nova master: WIP: Add a routed networks scheduler pre-filter https://review.opendev.org/c/openstack/nova/+/749068 | |
| 18:01:55 | openstackgerrit | Sylvain Bauza proposed openstack/nova master: Rename ensure_network_metadata to amend requested_networks https://review.opendev.org/c/openstack/nova/+/773955 | |
| 18:01:56 | openstackgerrit | Sylvain Bauza proposed openstack/nova master: Add network and utils methods for getting routed networks and segments https://review.opendev.org/c/openstack/nova/+/773976 | |
| 18:02:08 | sean-k-mooney | melwitt: correct | |
| 18:02:12 | lyarwood | https://github.com/openstack/nova/blob/fefd984fd1294f23677e3df4390e37c7664883a0/nova/utils.py#L752-L778 was what I wanted to link to | |
| 18:02:19 | sean-k-mooney | if its there i belive it will be avaiable via the metadata service | |
| 18:02:25 | melwitt | if it goes in system_metadata I guess that's ok, IIUC | |
| 18:02:25 | sean-k-mooney | allowing anyone in the project ot get it | |
| 18:02:45 | melwitt | but not metadata, that will be public to all in the project like you said | |
| 18:02:52 | lyarwood | melwitt: so the issue then is it would show up as an image metadata property | |
| 18:03:05 | sean-k-mooney | yep and the feature is ment to make it so only the user that created the instace can log in | |
| 18:03:14 | sean-k-mooney | via vnc with the password they set | |
| 18:03:15 | melwitt | oh system_meta gets reflected out of metadata? oh, I didn't know that :\ | |
| 18:03:24 | lyarwood | neither did I until a few hours ago | |
| 18:03:34 | sean-k-mooney | or i | |
| 18:03:35 | lyarwood | slightly weird behaviour tbh, I can't think why | |
| 18:03:36 | melwitt | hrm | |
| 18:03:46 | sean-k-mooney | i though it was only stuff with the image_ prefix or img_ | |
| 18:04:04 | sean-k-mooney | not everything becase we store stuff for tpm there too | |
| 18:04:11 | melwitt | yeah I had thought system_meta was like internal system info that wasn't put out there | |
| 18:04:28 | melwitt | ok so you're saying a subset of it is reflected out | |
| 18:04:55 | sean-k-mooney | no i thouhg only a subset shoudl be relected out via the image | |
| 18:05:04 | lyarwood | my reading of https://github.com/openstack/nova/blob/fefd984fd1294f23677e3df4390e37c7664883a0/nova/utils.py#L752-L778 and what I've seen when stashing hw_machine_type in there is that it ends up being reflected in the image metadata properties | |
| 18:05:10 | sean-k-mooney | but lyarwood seams to have found all of it might be in a specific case? | |
| 18:05:29 | melwitt | oh | |
| 18:05:46 | sean-k-mooney | so SM_IMAGE_PROP_PREFIX | |
| 18:05:56 | sean-k-mooney | was ment to be filtering to just the image ones right | |
| 18:06:20 | lyarwood | you would assume so | |
| 18:07:12 | sean-k-mooney | so key = key[len(SM_IMAGE_PROP_PREFIX):] | |
| 18:07:17 | sean-k-mooney | is striping the key | |
| 18:07:21 | sean-k-mooney | i think that is the issue | |
| 18:08:21 | sean-k-mooney | because you have img_hw_machine_type that matches hw_machine_type which you added | |
| 18:08:21 | lyarwood | so in my case I've not even set it in the image metadata | |