| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-03-16 | |||
| 14:41:19 | kashyap | And _uri parameter is deprecated in favor of the above | |
| 14:41:41 | Luzi | but when you don't set it, as i had not done it, the code uses the hardcoded tcp parameter | |
| 14:41:44 | fungi | looks like that config option was added by https://review.openstack.org/410817 which merged in early 2017 | |
| 14:42:08 | kashyap | fungi: Heya; yep. That timeframe is right ... /me clicks | |
| 14:42:35 | kashyap | Yep; patch is also correct. | |
| 14:42:43 | Luzi | live_migration_scheme = tls | |
| 14:42:45 | fungi | first appeared in nova 15.0.0, so ocata and later | |
| 14:42:48 | kashyap | Luzi: Also you don't tell what version of OpenStack you were using? | |
| 14:43:13 | kashyap | Yeah, what fungi says. | |
| 14:43:19 | Luzi | https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L1224 | |
| 14:43:23 | openstackgerrit | Stephen Finucane proposed openstack/nova master: pci: Add vDPA vnic to PCI request mapping and filtering https://review.opendev.org/c/openstack/nova/+/778350 | |
| 14:43:24 | openstackgerrit | Stephen Finucane proposed openstack/nova master: tests: Add functional test for vDPA device https://review.opendev.org/c/openstack/nova/+/780112 | |
| 14:43:24 | openstackgerrit | Stephen Finucane proposed openstack/nova master: api: Block unsupported actions with vDPA https://review.opendev.org/c/openstack/nova/+/780333 | |
| 14:43:25 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Add release note for vDPA https://review.opendev.org/c/openstack/nova/+/780866 | |
| 14:43:25 | openstackgerrit | Stephen Finucane proposed openstack/nova master: WIP: tests: Make mdev stubs work like vDPA https://review.opendev.org/c/openstack/nova/+/780234 | |
| 14:43:31 | stephenfin | gibi: lyarwood: ^ | |
| 14:43:36 | Luzi | i am working on train and looking to upgrade soon | |
| 14:44:12 | Luzi | i posted the line of code which seems to be still on master | |
| 14:45:06 | fungi | same change deprecated live_migration_uri | |
| 14:45:36 | kashyap | (Yep) | |
| 14:46:22 | Luzi | in the guide the only config option which is mentioned is live_migration_with_native_tls = true | |
| 14:46:36 | Luzi | which is definitely not enough to enable native tls | |
| 14:46:40 | kashyap | Luzi: So, indeed - it checks if the config option is set, then uses the _scheme, if not defaults to TLS | |
| 14:47:20 | fungi | so alternatively, the document could warn that setting the _scheme will prevent a natural fallback to tls | |
| 14:47:21 | Luzi | well i thought it worked, until i started tcpdump | |
| 14:47:37 | Luzi | tcp | |
| 14:47:51 | Luzi | the default in code is tcp | |
| 14:48:10 | kashyap | Luzi: Err, defaults to TCP, I mean, I'm sorry. | |
| 14:48:22 | Luzi | the guid should at least say it is needed to set the _scheme to tls | |
| 14:48:38 | fungi | ayup, i agree it defaults to 'tcp' there | |
| 14:49:10 | kashyap | Luzi: Right; I'll check the installer code to double-confirm; and then can update the doc | |
| 14:49:20 | fungi | okay, so the doc just needs updating to mention setting live_migration_scheme="tls"? | |
| 14:49:37 | Luzi | i'm just worried, that this may also concern some users which did not check their deployment with a tcpdump :D | |
| 14:49:51 | fungi | it got mentioned in the release notes (under features) but yeah that's fairly hidden | |
| 14:50:17 | Luzi | it should be enough, to update the guide, i think | |
| 14:50:37 | fungi | and even the release note doesn't come out and say it's needed for turning on tls | |
| 14:50:58 | Luzi | fungi, yeah thats the point | |
| 14:51:06 | Luzi | i needed a whole day to find out :D | |
| 14:51:22 | kashyap | fungi: Yep - on the doc needs updating. Luzi: Good catch! | |
| 14:51:46 | fungi | Luzi: if you're worried that we need to do some outreach, the typical process for that is an openstack security note: https://wiki.openstack.org/wiki/Security/Security_Note_Process | |
| 14:52:00 | Luzi | i need to go off now, thank you for replying so quickly kashyap and fungi :) | |
| 14:52:03 | kashyap | Luzi: So you need _both_: | |
| 14:52:06 | kashyap | - live_migration_scheme | |
| 14:52:14 | fungi | once published, we can send a copy to the openstack-announce and openstack-discuss mailing lists | |
| 14:52:16 | kashyap | - live_migration_with_native_tls | |
| 14:52:44 | Luzi | fungi, i will do that when i'm at home :) | |
| 14:52:47 | Luzi | kashyap, yes | |
| 14:54:57 | kashyap | (Ah, Luzi is gone before I wanted to mention a way to actually verify how to check native TLS is in effect for disks during migration w/ low-level logs.) | |
| 15:03:54 | lyarwood | stephenfin: https://review.opendev.org/c/openstack/nova/+/780333 - quick commit message nit in there but otherwise LGTM | |
| 15:07:19 | openstackgerrit | Stephen Finucane proposed openstack/nova master: tests: Add functional test for vDPA device https://review.opendev.org/c/openstack/nova/+/780112 | |
| 15:07:19 | openstackgerrit | Stephen Finucane proposed openstack/nova master: api: Block unsupported actions with vDPA https://review.opendev.org/c/openstack/nova/+/780333 | |
| 15:07:20 | openstackgerrit | Stephen Finucane proposed openstack/nova master: WIP: tests: Make mdev stubs work like vDPA https://review.opendev.org/c/openstack/nova/+/780234 | |
| 15:07:20 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Add release note for vDPA https://review.opendev.org/c/openstack/nova/+/780866 | |
| 15:08:10 | lyarwood | ta | |
| 15:17:37 | lyarwood | stephenfin: https://review.opendev.org/c/openstack/nova/+/780866 - same in the releasenote btw | |
| 15:17:49 | stephenfin | gdi :) | |
| 15:17:59 | stephenfin | will fix | |
| 15:18:36 | lyarwood | np my review ratio thanks you ;) | |
| 15:18:56 | lyarwood | joys of being a stable core | |
| 15:21:31 | gibi | stephenfin: ack, I have no problem with 409, and I will review the series soon | |
| 15:52:32 | openstackgerrit | Kashyap Chamarthy proposed openstack/nova master: libvirt: Deprecate `live_migration_tunnelled` https://review.opendev.org/c/openstack/nova/+/780908 | |
| 15:54:27 | kashyap | gibi: stephenfin: Would be nice to get it going in Wallaby --^. Will reduce some future "live migration config fatigue" | |
| 15:55:23 | gibi | kashyap: ack, I will check after the vdpa series | |
| 15:55:38 | kashyap | Sure; no prob. It's just a deprecation meta-work; that's all. | |
| 15:55:53 | kashyap | I'll add a code-reference in the change comment | |
| 15:58:41 | lyarwood | kashyap: random question, do we have coverage of live_migration_with_native_tls in the gate somewhere? | |
| 15:59:15 | lyarwood | kashyap: I can't find anything but that smells like something we should enable in nova-next at least | |
| 15:59:29 | kashyap | lyarwood: Good question, I do not know; and near as I know, no. It requires a full TLS env to be setup -- which is what TripleO does | |
| 15:59:44 | kashyap | See my prerequisite here: https://docs.openstack.org/nova/latest/admin/secure-live-migration-with-qemu-native-tls.html#prerequisites | |
| 16:00:33 | kashyap | lyarwood: I swear there is some automated test of this upstream (perhaps the upstream whitebox from OSP QE folks), can't find a link yet | |
| 16:01:14 | lyarwood | kashyap: yeah it's likely on the TripleO side and sorry I thought the virt tools would automate the CA creation etc, if not ignore me as it's going to be too much work to enable in devstack for nova-next etc | |
| 16:01:54 | kashyap | lyarwood: Yeah, it requires good old PKI setup. It _can_ be automated, using GnuTLS | |
| 16:02:10 | kashyap | lyarwood: If you don't mind holding your nose, something like ... https://kashyapc.fedorapeople.org/Auto-Setup-GnuTLS/Setup-GnuTLS-CA-and-ServerCert.bash | |
| 16:02:55 | lyarwood | that said we do have some novnc tls stuff in devstack so... | |
| 16:03:12 | kashyap | I see | |
| 16:03:36 | kashyap | But pretty sure I recall chatting w/ TripleO folks to have an automated test. Once I find a URL, I'll link it in the change or post here | |
| 16:30:13 | bauzas | gibi: others, folks, i'm taking an urgent PTO tomorrow, my wife is turning 40 as well | |
| 16:32:31 | gibi | bauzas: ack, have a nice celebration! | |
| 16:33:57 | gibi | bauzas: will you push a new rev from the rpc bump patch before you leave? | |
| 16:34:36 | bauzas | gibi: not sure :( | |
| 16:34:53 | bauzas | still fixing to not support 5.0 | |
| 16:36:48 | gibi | bauzas: no worries just preparing my review queue | |
| 16:39:34 | Luzi | kashyap, are you around? | |
| 16:39:45 | kashyap | Luzi: Hi, yes | |
| 16:39:57 | kashyap | Luzi: So, I wanted to mention one more to thing to you, before you left | |
| 16:40:25 | kashyap | Luzi: If you want to see native TLS is *actually* in effect, you can verify it by a slightly tedious method in logs | |
| 16:40:37 | kashyap | Luzi: Is your env. production? Or do you have a staging setup? | |
| 16:40:47 | Luzi | testing setup | |
| 16:41:24 | kashyap | Luzi: Perfect. So here we go: | |
| 16:41:54 | kashyap | Luzi: What OS do you have? Ubuntu or Fedora? Doesn't matter: install the "libvirt-admin" tool | |
| 16:41:59 | kashyap | s/tool/package/ | |
| 16:43:03 | kashyap | To get the 'virt-admin' tool. Note: Ubuntu might name it differently. So "grep" your package repository for the tool. In Fedora, I can do it like this: `dnf whatprovides *virt-admin` | |
| 16:43:25 | kashyap | Luzi: Then follow this to enable the dynamic libvirt log filters on your compute nodes: https://kashyapc.fedorapeople.org/virt/virt-admin.txt | |
| 16:45:32 | kashyap | Once you have that setup; you can migrate a guest, and then you can look for: `grep tls-creds-x509` on your source and destination libvirtd.log. | |
| 16:45:53 | kashyap | Luzi: Did you hav a different question for me? I began talking right after you pinged me :) | |
| 16:46:45 | Luzi_ | here again, needed to change to vpn | |
| 16:47:05 | Luzi_ | i missed the last 6 minutes or so - did you wrote something kashyap ? | |
| 16:47:32 | kashyap | Luzi_: Yes, I did post something. I was talking to myself. Let me post a pastebin | |
| 16:48:15 | kashyap | Luzi_: http://paste.openstack.org/show/803621/ | |
| 16:50:25 | kashyap | Luzi_: For comparison, you should see commands like these in your _destination_ libvirt log file: https://kashyapc.fedorapeople.org/Native-TLS/Test-Evidence/DEST-QMP-commands-TLS-over-NBD-guestHyp2.log | |