| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-11-02 | |||
| 09:23:22 | frickler | I'm pretty confident by now though that tb-size is the trigger | |
| 09:37:46 | stephenfin | I need to test the 'GET /servers/{server_id}/migrations/{id}' API, meaning I need a way to slow down live migrations so I catch one in the act. Anyone have a suggestion for an easy throttle I can set to do that? | |
| 09:38:09 | stephenfin | (rather than relying on big or busy guests) | |
| 09:39:12 | gibi | stephenfin: limiting bandwidth ? | |
| 09:39:37 | stephenfin | I assume there isn't a nova or libvirt config option I can use for that though? | |
| 09:39:58 | gibi | there is something in libvirt | |
| 09:39:58 | stephenfin | This is a simple DevStack two-node deployment, so I don't have a separate management network :) | |
| 09:40:05 | gibi | as there is virsh migrate-setspeed command in virsh | |
| 09:40:13 | stephenfin | oh, I looked and didn't see anything obvious | |
| 09:41:24 | kashyap | frickler: Do mention the Neutron issue in the change, if / when you get a minute | |
| 09:41:38 | kashyap | frickler: And probably Cc some folks from Neutron who might be able to debug | |
| 09:41:51 | stephenfin | gibi: that's exactly what I wanted. Thanks! | |
| 09:42:04 | kashyap | stephenfin: Yes, migrate-setspeed lets you throttle indeed | |
| 09:42:15 | gibi | stephenfin: cool | |
| 09:51:07 | bauzas | mmmm | |
| 09:51:22 | bauzas | just saw a new "Your Turn" series in Gerrit default dashboard | |
| 09:51:40 | bauzas | what's the "attention:self" query ? | |
| 09:52:00 | bauzas | hah, nevermind, found https://gerrit-review.googlesource.com/Documentation/user-attention-set.html | |
| 09:52:56 | bauzas | interesting | |
| 09:59:25 | gibi | I'm still learning the rules described in ^^ | |
| 10:15:32 | opendevreview | Balazs Gibizer proposed openstack/nova master: Reno for qos-minimum-guaranteed-packet-rate https://review.opendev.org/c/openstack/nova/+/805046 | |
| 10:17:00 | gibi | bauzas: fyi, this is the final patch (the reno) https://review.opendev.org/c/openstack/nova/+/805046 for the https://blueprints.launchpad.net/openstack/?searchtext=qos-minimum-guaranteed-packet-rate blueprint. So we can close that bp soon \o/ | |
| 10:17:38 | bauzas | gibi: wow, this was fast. | |
| 10:18:03 | gibi | bauzas: we only missed the nova-manage part of that bp in xena | |
| 10:18:13 | bauzas | yup, I know | |
| 10:18:17 | bauzas | but still :) | |
| 10:18:30 | gibi | yeah, it is always nice to close out a bp even before M1 | |
| 10:18:34 | bauzas | we discussed this at the PTG, I wasn't expecting the nova-manage patch to land that soon :) | |
| 10:19:02 | gibi | it is thanks to stephenfin and melwitt | |
| 10:24:04 | opendevreview | Balazs Gibizer proposed openstack/nova master: Reno for qos-minimum-guaranteed-packet-rate https://review.opendev.org/c/openstack/nova/+/805046 | |
| 10:26:29 | gibi | bauzas: btw, there is a bug fix for the series (for those part we landed in xena) https://review.opendev.org/c/openstack/nova/+/811396 | |
| 10:28:37 | bauzas | +w | |
| 10:32:03 | opendevreview | Federico Ressi proposed openstack/nova master: Debug Nova APIs call failures https://review.opendev.org/c/openstack/nova/+/806683 | |
| 10:38:03 | lyarwood | frickler: just catching up after a few weeks out, excellent work with the QEMU tb-size issue! | |
| 10:42:29 | kashyap | lyarwood: Yeah, libvirt needs to wire it up now, though | |
| 10:43:21 | kashyap | I'll file a RHEL libvirt RFE - that might get on their triage queue quicker | |
| 10:44:38 | gibi | bauzas: awesome, thank you | |
| 10:50:25 | lyarwood | kashyap: yeah, shame we can't hackaround this in the meantime somehow | |
| 10:50:52 | lyarwood | kashyap: couldn't we pass QEMU args directly through libvirt from Nova in the meantime? | |
| 10:51:04 | kashyap | lyarwood: Definitely, there's QEMU command-line passthrough... | |
| 10:51:07 | kashyap | For libvirt XML | |
| 10:51:19 | lyarwood | second day back and I'm already writing another hackaround | |
| 10:51:22 | kashyap | lyarwood: But wait: | |
| 10:51:42 | kashyap | Nova doesn't have that XML modelling classes for command-line passthrough (for good reasons) :-( | |
| 10:52:01 | kashyap | lyarwood: The only current hack is to upload a custom QEMU build with that built in | |
| 10:52:09 | lyarwood | ewww | |
| 10:52:22 | lyarwood | I'd rather add the logic in Nova with a workaround option tbh | |
| 10:52:29 | lyarwood | than build our own custom QEMU | |
| 10:52:43 | kashyap | lyarwood: I agree, it's nasty to do the custom builds for medium-term | |
| 10:53:33 | kashyap | The logic in Nova would require to wire in these bits, BTW: https://libvirt.org/kbase/qemu-passthrough-security.html | |
| 10:53:43 | kashyap | (Including the namespace at the top) | |
| 10:53:56 | lyarwood | Yup that's easy enough | |
| 10:54:03 | kashyap | And still it requires more edits. I was testing last week | |
| 10:54:55 | kashyap | When using `-accel tcg,tb-size=256`, we should remove "accel=tcg" from `-machine q35,accel=tcg` | |
| 10:55:14 | kashyap | Otherwise QEMU fails to launch | |
| 10:55:37 | kashyap | (I think libvirt uses the latter syntax by default: "-machine ... accel=") | |
| 10:56:03 | kashyap | (Yep, it does. Just verified) | |
| 10:56:19 | ebbex | Is there a option/toggle to disable sending numa_topology from nova-compute? (We have some numascale hardware that submits "Data too long for column 'numa_topology') | |
| 10:57:20 | lyarwood | kashyap: oh fun | |
| 10:58:51 | kashyap | lyarwood: Yeah. For more on the nature of QEMU command-line, see my LWN article: https://lwn.net/SubscriberLink/872321/221e8d48eb609a38/) | |
| 10:59:25 | kashyap | (Especially the "Complexity on the QEMU command line" section) | |
| 11:01:23 | gibi | lyarwood: o/ we can revert the temp disable on the heal_allocation in nova-next the nova-manage support landed during the night. https://review.opendev.org/c/openstack/nova/+/816242 | |
| 11:01:41 | lyarwood | awesome checking | |
| 11:01:44 | gibi | thanks | |
| 11:01:59 | lyarwood | +W'd | |
| 11:02:02 | gibi | thanks | |
| 11:03:55 | lyarwood | kashyap: would you be able to test if we could overwrite the original `-machine q35,accel=tcg` part using <qemu:commandline> via libvirt? | |
| 11:04:17 | kashyap | lyarwood: Let me try | |
| 11:04:41 | kashyap | I think <qemu:commandline> _does_ take precedence | |
| 11:04:55 | lyarwood | would be ace as Nova could do that itself then | |
| 11:17:14 | kashyap | lyarwood: Afraid, I was wrong :-( | |
| 11:17:19 | kashyap | I tried this: | |
| 11:19:36 | kashyap | lyarwood: It doesn't overwrite, that was the XML (see line-1 and lines 102-105) https://paste.centos.org/view/1fcbc6a4 | |
| 11:19:52 | kashyap | With that, when I start the guest, it gives the familiar: | |
| 11:19:53 | kashyap | error: internal error: process exited while connecting to monitor: 2021-11-02T11:18:34.543504Z qemu-kvm: The -accel and "-machine accel=" options are incompatible | |
| 11:19:53 | kashyap | error: Failed to start domain 'cvm2' | |
| 11:19:53 | kashyap | $> virsh start cvm2 | |
| 11:21:30 | lyarwood | sorry was just on a call | |
| 11:21:39 | kashyap | No rush; I don't count on instant responses :-) | |
| 11:22:02 | lyarwood | kashyap: what if you also define -machine in the XML? | |
| 11:22:24 | kashyap | Hmm, lemme try | |
| 11:22:53 | kashyap | lyarwood: Wait, you mean setting -accel and -machine in qemu:commandline explicitly? | |
| 11:23:45 | lyarwood | kashyap: yes | |
| 11:24:00 | kashyap | (If so, that should fail the same way as above, but lemme double-confirm. libvirt uses "-machine accel" under the hood, by inference from <domain type='kvm') | |
| 11:24:32 | kashyap | Yep, it fails the same way. | |
| 11:25:11 | kashyap | lyarwood: Oh, wait. There might be another hack, based on my chat w/ Paolo last week: | |
| 11:25:24 | kashyap | 17:55 < kashyap> bonzini: Hm, how exactl does "-machine accel=kvm -machine accel=tcg" differ from "-accel kvm -accel tcg"? | |
| 11:25:27 | kashyap | 17:55 < bonzini> "-machine accel=tcg" overwrites "-machine accel=kvm" | |
| 11:25:41 | kashyap | lyarwood: So, I can specify by "-accel tcg -accel kvm" ... and see if that works :D | |
| 11:26:26 | kashyap | Gaah, no, ignore me. I misread the above complexity. | |
| 11:39:55 | kashyap | No, there isn't a current trick. | |
| 11:40:32 | kashyap | lyarwood: That said, based on last week chat w/ QEMU folks, libvirt itself should switch to "-accel" as that's recommended than "-machine accel" | |
| 11:41:02 | lyarwood | kashyap: argh kk, so there's no workaround until that happens | |
| 11:41:17 | kashyap | No, besides the ugly hack we both revulse at :D | |
| 11:41:21 | lyarwood | kashyap: blocking pretty much all upstream Openstack testing using qemu until then | |
| 11:41:30 | lyarwood | yeah without the custom build | |
| 11:41:32 | lyarwood | urgh | |
| 11:41:47 | lyarwood | tbh we need to make a big deal out of this | |
| 11:42:50 | kashyap | Yeah, QEMU changed it pretty much w/o considering the management tools :-( | |
| 11:47:05 | opendevreview | Lee Yarwood proposed openstack/nova master: nova-manage: Always get BDMs using get_by_volume_and_instance https://review.opendev.org/c/openstack/nova/+/811716 | |