| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-06-21 | |||
| 13:51:06 | noonedeadpunk | But was also wondering if anybody heard anything about x-multiple-fds in qemu capabilities | |
| 13:54:52 | bauzas | I'm not really a SME of this matter | |
| 13:54:59 | bauzas | maybe kashyap may help | |
| 13:56:14 | bauzas | kashyap: thanks, tl;dr noonedeadpunk has some concerns by the migration TLS performance due to a QEMU single threading | |
| 13:56:21 | kashyap | noonedeadpunk: Before I say anything further, please note that the "x-" prefix in QMP commands (like "x-multiple-fds") means it's experimental | |
| 13:56:34 | kashyap | That _said_: the multiple-fd work did come out of experimental, IIRC. Lemme check | |
| 13:56:52 | kashyap | bauzas: I see; I missed the earlier context; so would be good to know what kind of performance concerns are these | |
| 13:57:41 | noonedeadpunk | so using TLS for libvirt gives migration speed like 1.2gbitps - 3 gbitps depending on CPU, TCP live migrations are 20gbitps | |
| 13:57:54 | noonedeadpunk | (basically full link) | |
| 14:01:54 | kashyap | noonedeadpunk: Hmm, I mean I'd definitely expect some loss w/ built-in TLS -- as it involves the additional encryption over head -- but I don't have numbers off-hand as to what's "expected" | |
| 14:02:17 | kashyap | noonedeadpunk: Can you come to #virt (on OFTC) and ask there too? More libvirt migration experts hang out there | |
| 14:02:21 | noonedeadpunk | so diffference is quite dramatical | |
| 14:02:40 | noonedeadpunk | yes, sure | |
| 14:02:50 | kashyap | Yeah; it's almost 10 fold diff | |
| 14:03:02 | bauzas | noonedeadpunk: ohoh, good to know | |
| 14:07:50 | noonedeadpunk | enabling auto-convergance kinda solves issue, at least migrations do not fail | |
| 14:08:33 | noonedeadpunk | but being able to utilize more then 1 cpu would be better ofc... | |
| 14:08:36 | kashyap | Ah, that's good to know that auto-convergence did its job at least | |
| 14:08:55 | kashyap | I agree on utilization; let's check w/ the libvirt-QEMU guys | |
| 14:09:08 | bauzas | gtk | |
| 14:14:35 | noonedeadpunk | But I guess if that's performance diff is kinda confirmed, it might be worth mentioning in nova docs at very least | |
| 14:16:38 | kashyap | noonedeadpunk: I think you're using an older QEMU by judging the "x-" prefix; so first would be good to upgrade to a newer one that removes the "x-" prefix (experimental) | |
| 14:17:10 | noonedeadpunk | I've taken that from qemu wiki page :) | |
| 14:17:34 | noonedeadpunk | I guess I was testing on qemu 8.0.0 or smth | |
| 14:17:35 | kashyap | Yeah, the wiki-page tends to not get updated. I'm pretty sure it's now out of experimental; otherwise libvirt won't enable it (as you saw on #virt) | |
| 14:17:42 | kashyap | Also, see this response from danpb: https://listman.redhat.com/archives/libvir-list/2020-January/msg00527.html | |
| 14:17:43 | noonedeadpunk | yup | |
| 14:17:50 | kashyap | (Scroll down a bit) | |
| 14:22:20 | noonedeadpunk | Yeah, fair | |
| 14:45:20 | noonedeadpunk | Soooo.... Should we say that it make sense to add `parallel` option support to nova? | |
| 14:54:18 | noonedeadpunk | I would try to check what needs to be done for that... Should be pretty much trivial to add | |
| 14:57:28 | bauzas | noonedeadpunk: sorry again, but what do you mean ? is it due to a new QEMU release ? | |
| 14:58:53 | noonedeadpunk | bauzas: there's a parallel flag that's for sure already present in 8.0.0: https://libvirt.org/manpages/virsh.html#migrate | |
| 14:59:26 | noonedeadpunk | let me check if it's available even before that | |
| 15:01:01 | noonedeadpunk | It's also available in 6.0.0, and nova min libvirt version is already beyond that | |
| 15:02:38 | noonedeadpunk | but we also should be careful with amount of parallel connections not to affect running workloads | |
| 15:04:06 | bauzas | noonedeadpunk: sounds to me a feature request | |
| 15:04:39 | bauzas | but yeah, turning on this know could be possible | |
| 15:23:31 | kashyap | Yeah, it's a separate request to enable the parallel connections | |
| 16:32:22 | opendevreview | Merged openstack/nova stable/wallaby: Remove mentions of removed scheduler filters https://review.opendev.org/c/openstack/nova/+/858050 | |
| 17:33:02 | melwitt | stephenfin, sean-k-mooney: just fyi I updated https://review.opendev.org/c/openstack/nova/+/877056 to add the bug and release note and return early+unindent code block. and I've also rechecked the tempest test skip for the unrelated CI failures https://review.opendev.org/c/openstack/tempest/+/886496 | |
| 21:59:11 | opendevreview | Jakub Skunda proposed openstack/nova master: Drop Fedora support https://review.opendev.org/c/openstack/nova/+/886596 | |
| #openstack-nova - 2023-06-22 | |||
| 04:12:14 | opendevreview | Amit Uniyal proposed openstack/os-vif stable/zed: set default qos policy https://review.opendev.org/c/openstack/os-vif/+/883016 | |
| 05:45:49 | opendevreview | Amit Uniyal proposed openstack/os-vif stable/yoga: Delete trunk bridges to avoid race with Neutron https://review.opendev.org/c/openstack/os-vif/+/886709 | |
| 05:45:50 | opendevreview | Amit Uniyal proposed openstack/os-vif stable/yoga: set default qos policy https://review.opendev.org/c/openstack/os-vif/+/886710 | |
| 06:15:07 | opendevreview | Amit Uniyal proposed openstack/os-vif stable/xena: Delete trunk bridges to avoid race with Neutron https://review.opendev.org/c/openstack/os-vif/+/886715 | |
| 06:15:08 | opendevreview | Amit Uniyal proposed openstack/os-vif stable/xena: set default qos policy https://review.opendev.org/c/openstack/os-vif/+/886716 | |
| 10:30:52 | opendevreview | Sylvain Bauza proposed openstack/nova master: Add a new NumInstancesWeigher https://review.opendev.org/c/openstack/nova/+/886232 | |
| 11:03:50 | dvo-plv | Hello, Nova folks | |
| 11:04:02 | dvo-plv | Maybe you will have a cnahce to check this nova patch | |
| 11:04:03 | dvo-plv | https://review.opendev.org/c/openstack/nova/+/876075 | |
| 11:04:15 | dvo-plv | This is a part of the virtio packed ring blueprint | |
| 11:22:12 | manuvakery1 | Hi .. Is it possible to create an instance on a disabled host by the admin using the ---host option? | |
| 11:39:04 | manuvakery1 | IIRC .. this was working in queens ..but in train throwing "No valid host was found" | |
| 11:54:12 | opendevreview | Elod Illes proposed openstack/nova stable/victoria: Remove mentions of removed scheduler filters https://review.opendev.org/c/openstack/nova/+/858051 | |
| 12:36:29 | bauzas | manuvakery1: by providing the host value, you ask the scheduler to verify it | |
| 12:36:47 | bauzas | manuvakery1: if the host is disabled, then the Compute filter will say no | |
| 13:20:27 | noonedeadpunk | bauzas: returning back to the discussion in Vancouver regarding cross-AZ scheduling - I see that topic was mentioned in the etherpad, but I can't recall why that's smth we should not do? | |
| 13:21:11 | bauzas | noonedeadpunk: hmmm, about what exactly ? | |
| 13:21:31 | noonedeadpunk | like having anti-AZ server groups | |
| 13:22:02 | noonedeadpunk | Right now we're trying to implement a cross-AZ scheduling of octavia amphoras by defining AZs in their config. And then mess up with OCtavia flows to ensure it will supply AZ to the scheduler | |
| 13:22:54 | noonedeadpunk | But now I've started thinking if that's correct path at all, and if would be easier to just enable anti-az option (alike to anti-affinity) and nova will handle that | |
| 13:23:19 | bauzas | noonedeadpunk: fortunately, you mean you would use soft-anti-affinity group policy ? | |
| 13:24:32 | noonedeadpunk | well, soft anti-affinity does ensure that VMs will spawn on different computes, but it doesn't care about different AZs, right? | |
| 13:25:24 | noonedeadpunk | So what I was thinking is smth like that, but specifically for cross-AZs spawning of VMs | |
| 13:27:43 | noonedeadpunk | but maybe we should just continue doing that on Octavia side | |
| 13:28:22 | noonedeadpunk | ah, sorry, I meant this etherpad https://etherpad.opendev.org/p/nova-vancouver2023-meet-and-greet | |
| 13:32:00 | bauzas | sorry, was afk for a second | |
| 13:32:27 | bauzas | noonedeadpunk: we also discussed this on the PTG, you can look at my ML email | |
| 13:34:39 | noonedeadpunk | ah, true, failure-domain-anti-affinity | |
| 13:34:46 | bauzas | right | |
| 13:35:50 | noonedeadpunk | but I guess "failure-domain" functionality does not exist yet? | |
| 13:36:34 | noonedeadpunk | another thing that can be "interesting" if we consider storage should be shared between failure domains? As I won't count on that I guess | |
| 13:39:25 | bauzas | noonedeadpunk: tbc, we let operators define their own value of "failure domain" | |
| 13:39:57 | bauzas | in general, this is something larger than a single rack | |
| 13:40:48 | noonedeadpunk | ideally it's smth 30km in between :) | |
| 13:41:20 | bauzas | yeah, or just a DC room | |
| 13:41:38 | bauzas | (as a reminder, I was an operator before joining OpenStack ;) ) | |
| 13:41:56 | bauzas | just because in general, the failures are with A/C or network | |
| 13:42:25 | noonedeadpunk | or fire in DC :D | |
| 13:42:26 | bauzas | so, if the domain is larger than a rack, then it's simple to move instances between hosts | |
| 13:42:45 | bauzas | noonedeadpunk: lollylol (sorry OVH friends ;) ) | |
| 13:43:32 | noonedeadpunk | so, basically, option is: set metadata to aggregate which defines "failure-domain", then for server group provide the metadata and policy failure-domain-anti-affinity (or failure-domain-affinity)? | |
| 13:45:58 | bauzas | noonedeadpunk: what we said during the PTG to Bloomberg is that we could have a new API microversion for an admin saying "I'd want to live-migrate this instance from this server group to another" | |
| 13:46:27 | noonedeadpunk | that is completely different thing I guess? | |
| 13:46:42 | bauzas | and maybe have a new policy too | |
| 13:46:48 | bauzas | noonedeadpunk: https://etherpad.opendev.org/p/vancouver-june2023-nova#L121 | |
| 13:47:26 | noonedeadpunk | ok, yes, I was also talking about that... | |
| 13:47:42 | noonedeadpunk | then I'm slightly confused with live-migration | |
| 13:49:41 | noonedeadpunk | going back to our usecase with Octavia. What could be done instead of having AZ-scheduler in octavia code - just making a call to create 2 VMs withing same server group, and nova would take care of ensuring, they will be spawned on different failure domains | |
| 13:49:51 | noonedeadpunk | (just theoretically) | |
| 13:51:06 | noonedeadpunk | And why I got confused about live migrations, as in our usecase we have completely independent backing storages | |
| 13:51:41 | bauzas | noonedeadpunk: sorry I was quickly explaining about what we discussed with Bloomberg | |
| 13:51:56 | noonedeadpunk | yeah, ok, that makes sense as well :) | |
| 13:52:21 | bauzas | like, given they were using both AZ and server groups, they also wanted to violate their hard policy that they were using for packing instances to the same rack | |
| 13:52:39 | bauzas | basically, because sometimes you want to deprecate a rack :p | |
| 13:52:53 | noonedeadpunk | yeah, yeah, that makes sense as well | |
| 13:53:10 | bauzas | that's also why I provided https://review.opendev.org/c/openstack/nova/+/886232 to help operators to pack instances to hosts :) | |
| 13:53:32 | bauzas | instead of having to create some hard policies, which I utterly dislike :D | |
| 13:53:51 | noonedeadpunk | oh, that is neat | |