| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-11-02 | |||
| 10:50:52 | lyarwood | kashyap: couldn't we pass QEMU args directly through libvirt from Nova in the meantime? | |
| 10:51:04 | kashyap | lyarwood: Definitely, there's QEMU command-line passthrough... | |
| 10:51:07 | kashyap | For libvirt XML | |
| 10:51:19 | lyarwood | second day back and I'm already writing another hackaround | |
| 10:51:22 | kashyap | lyarwood: But wait: | |
| 10:51:42 | kashyap | Nova doesn't have that XML modelling classes for command-line passthrough (for good reasons) :-( | |
| 10:52:01 | kashyap | lyarwood: The only current hack is to upload a custom QEMU build with that built in | |
| 10:52:09 | lyarwood | ewww | |
| 10:52:22 | lyarwood | I'd rather add the logic in Nova with a workaround option tbh | |
| 10:52:29 | lyarwood | than build our own custom QEMU | |
| 10:52:43 | kashyap | lyarwood: I agree, it's nasty to do the custom builds for medium-term | |
| 10:53:33 | kashyap | The logic in Nova would require to wire in these bits, BTW: https://libvirt.org/kbase/qemu-passthrough-security.html | |
| 10:53:43 | kashyap | (Including the namespace at the top) | |
| 10:53:56 | lyarwood | Yup that's easy enough | |
| 10:54:03 | kashyap | And still it requires more edits. I was testing last week | |
| 10:54:55 | kashyap | When using `-accel tcg,tb-size=256`, we should remove "accel=tcg" from `-machine q35,accel=tcg` | |
| 10:55:14 | kashyap | Otherwise QEMU fails to launch | |
| 10:55:37 | kashyap | (I think libvirt uses the latter syntax by default: "-machine ... accel=") | |
| 10:56:03 | kashyap | (Yep, it does. Just verified) | |
| 10:56:19 | ebbex | Is there a option/toggle to disable sending numa_topology from nova-compute? (We have some numascale hardware that submits "Data too long for column 'numa_topology') | |
| 10:57:20 | lyarwood | kashyap: oh fun | |
| 10:58:51 | kashyap | lyarwood: Yeah. For more on the nature of QEMU command-line, see my LWN article: https://lwn.net/SubscriberLink/872321/221e8d48eb609a38/) | |
| 10:59:25 | kashyap | (Especially the "Complexity on the QEMU command line" section) | |
| 11:01:23 | gibi | lyarwood: o/ we can revert the temp disable on the heal_allocation in nova-next the nova-manage support landed during the night. https://review.opendev.org/c/openstack/nova/+/816242 | |
| 11:01:41 | lyarwood | awesome checking | |
| 11:01:44 | gibi | thanks | |
| 11:01:59 | lyarwood | +W'd | |
| 11:02:02 | gibi | thanks | |
| 11:03:55 | lyarwood | kashyap: would you be able to test if we could overwrite the original `-machine q35,accel=tcg` part using <qemu:commandline> via libvirt? | |
| 11:04:17 | kashyap | lyarwood: Let me try | |
| 11:04:41 | kashyap | I think <qemu:commandline> _does_ take precedence | |
| 11:04:55 | lyarwood | would be ace as Nova could do that itself then | |
| 11:17:14 | kashyap | lyarwood: Afraid, I was wrong :-( | |
| 11:17:19 | kashyap | I tried this: | |
| 11:19:36 | kashyap | lyarwood: It doesn't overwrite, that was the XML (see line-1 and lines 102-105) https://paste.centos.org/view/1fcbc6a4 | |
| 11:19:52 | kashyap | With that, when I start the guest, it gives the familiar: | |
| 11:19:53 | kashyap | error: internal error: process exited while connecting to monitor: 2021-11-02T11:18:34.543504Z qemu-kvm: The -accel and "-machine accel=" options are incompatible | |
| 11:19:53 | kashyap | error: Failed to start domain 'cvm2' | |
| 11:19:53 | kashyap | $> virsh start cvm2 | |
| 11:21:30 | lyarwood | sorry was just on a call | |
| 11:21:39 | kashyap | No rush; I don't count on instant responses :-) | |
| 11:22:02 | lyarwood | kashyap: what if you also define -machine in the XML? | |
| 11:22:24 | kashyap | Hmm, lemme try | |
| 11:22:53 | kashyap | lyarwood: Wait, you mean setting -accel and -machine in qemu:commandline explicitly? | |
| 11:23:45 | lyarwood | kashyap: yes | |
| 11:24:00 | kashyap | (If so, that should fail the same way as above, but lemme double-confirm. libvirt uses "-machine accel" under the hood, by inference from <domain type='kvm') | |
| 11:24:32 | kashyap | Yep, it fails the same way. | |
| 11:25:11 | kashyap | lyarwood: Oh, wait. There might be another hack, based on my chat w/ Paolo last week: | |
| 11:25:24 | kashyap | 17:55 < kashyap> bonzini: Hm, how exactl does "-machine accel=kvm -machine accel=tcg" differ from "-accel kvm -accel tcg"? | |
| 11:25:27 | kashyap | 17:55 < bonzini> "-machine accel=tcg" overwrites "-machine accel=kvm" | |
| 11:25:41 | kashyap | lyarwood: So, I can specify by "-accel tcg -accel kvm" ... and see if that works :D | |
| 11:26:26 | kashyap | Gaah, no, ignore me. I misread the above complexity. | |
| 11:39:55 | kashyap | No, there isn't a current trick. | |
| 11:40:32 | kashyap | lyarwood: That said, based on last week chat w/ QEMU folks, libvirt itself should switch to "-accel" as that's recommended than "-machine accel" | |
| 11:41:02 | lyarwood | kashyap: argh kk, so there's no workaround until that happens | |
| 11:41:17 | kashyap | No, besides the ugly hack we both revulse at :D | |
| 11:41:21 | lyarwood | kashyap: blocking pretty much all upstream Openstack testing using qemu until then | |
| 11:41:30 | lyarwood | yeah without the custom build | |
| 11:41:32 | lyarwood | urgh | |
| 11:41:47 | lyarwood | tbh we need to make a big deal out of this | |
| 11:42:50 | kashyap | Yeah, QEMU changed it pretty much w/o considering the management tools :-( | |
| 11:47:05 | opendevreview | Lee Yarwood proposed openstack/nova master: nova-manage: Always get BDMs using get_by_volume_and_instance https://review.opendev.org/c/openstack/nova/+/811716 | |
| 11:47:41 | lyarwood | https://review.opendev.org/q/topic:%22bug%252F1943431%22+(status:open%20OR%20status:merged) & https://review.opendev.org/q/topic:%22bug%252F1937084%22+(status:open%20OR%20status:merged) should be ready for reviews if people have time btw, simple bugfixes | |
| 12:01:15 | EugenMayer | When trying to rebuild an instance and use --preserve-ephimeral is see `The current driver does not support preserving ephemeral partitions.` | |
| 12:01:54 | EugenMayer | Is this option only available when using a storage like nfs/ceph? But how is that different to volumes then? Currently i use the compute node local storage | |
| 12:03:21 | EugenMayer | I use LVM on my computes with ext4 - do i need zfs/btrfs for that to work? | |
| 12:04:10 | sean-k-mooney[m] | EugenMayer that is only supported on ironic | |
| 12:04:34 | sean-k-mooney[m] | its not supported with libvirt or any other vm or container based driver | |
| 12:05:15 | sean-k-mooney[m] | rebuild is intended to remove all data from ths instance by recrating the root disk any epmeral disks | |
| 12:05:40 | sean-k-mooney[m] | if you want to use rebuild and preserve data you should store your data in cinder volumes | |
| 12:06:32 | EugenMayer | i though of ironic as just a barebone provisioning (via biofrost?) then running libvirt - but that is wrong? | |
| 12:07:02 | EugenMayer | ironic means, that one does not use any hypervisor at all - the barebone is the actual instance. That is the point right? sean-k-mooney[m] | |
| 12:07:47 | sean-k-mooney[m] | ironic is openstack beremetal as a service project and it can be used with nova to provide instance that are phsyicl servers instead of vms | |
| 12:08:08 | EugenMayer | Understood - thank you for clarifing | |
| 12:08:12 | sean-k-mooney[m] | bifrost is an installer for ironic written in ansible | |
| 12:09:05 | sean-k-mooney[m] | bifrost installs ironic in standalone mode so it canbe used without the rest of openstack to manage your phsyical hardware | |
| 12:11:11 | EugenMayer | Thank you! | |
| 12:12:26 | sean-k-mooney[m] | lyarwood: given we do not allow the use of qemu arg passthough in nova i dont see anyway for us to adress this in nova | |
| 12:12:44 | EugenMayer | actually have local latency and 'sync' the data to the central storage? | |
| 12:12:44 | EugenMayer | One question to cinder - how do you deal with databases? I mean using NFS or alikes and storing (running) a database on such a storage with heavily impact performance - this was the main reason to use local disk (we yet avoided cinder in our setup idea). How do you deal with that? Are you using a specific cinder backend like ceph/gluster so you | |
| 12:14:51 | sean-k-mooney[m] | for database workload i think its more common to use a dedicated san and mount the data over iscsi nfs really is not up to that level of iops. ceph can handel database but generally you will need to use flash if you have high iops | |
| 12:15:06 | sean-k-mooney[m] | i know that many do use local for dbs | |
| 12:15:51 | sean-k-mooney[m] | e.g. the root disk or epmeral disk but then you just need to ensure that you do not use rebuild and make backups at the application level | |
| 12:16:15 | EugenMayer | ok so this is a common issue | |
| 12:17:01 | EugenMayer | sean-k-mooney[m] with flash you mean SSD/NVME drives, right? (we have those only, the latter) | |
| 12:18:34 | sean-k-mooney[m] | yes if you have nvme storage and a high speed 25G+ networking you can deploy high iops workloads on cpeh but your network will become the bottleneck | |
| 12:19:43 | EugenMayer | well our network is about 1GB | |
| 12:19:52 | EugenMayer | it's provider based | |
| 12:20:45 | EugenMayer | do i understand ceph correcly here, that it is actually local access when a 'backend sync' in the background, other then nfs which is a transparent access on the network mount with the performance pain | |
| 12:21:09 | sean-k-mooney[m] | the normal way to deploy databases with local storage is to deploy them in a 3 node ha cluster with local sotrage and backup to cinder volumes with update managed via yum/apt ectra inside the vms. | |
| 12:21:43 | sean-k-mooney[m] | no ceph is directly acessed over the network | |
| 12:22:04 | sean-k-mooney[m] | it use the rbd protocol rather then iscsi but its more similar to iscsi then nfs | |
| 12:23:43 | EugenMayer | this means that one rather runs central db clusters for each DB variant (5.5,5.6,8 or pg 9.6,10,10) and loses the encapsulation that every app travels with it's database like we rather used right now (self container docker-compose / k8s stacks) | |
| 12:24:02 | sean-k-mooney[m] | if you need to use rebuild because of a higher level orchestror then effectivly the best way to do that with only local sotrage i is to serials the rebuild and by removing 1 instace form the cluser, rebuilding it, rejoin it to the cluster then waiting for it to sync with the latest state. then repeat for the rest. | |
| 12:24:44 | sean-k-mooney[m] | well with k8s it change slightly | |
| 12:25:03 | sean-k-mooney[m] | in that it assumes that you have shared network based storage by default | |
| 12:25:21 | EugenMayer | it can be, also k8s can use ephemeral, which we plan to | |
| 12:25:37 | sean-k-mooney[m] | so it assume it cna just terminate the db contaienr and when its recreated after update it can reconnect to the same storeage on any host and get the data back | |
| 12:26:19 | sean-k-mooney[m] | you plan ot use the local provide to back your persitnet volume claims? | |
| 12:27:01 | EugenMayer | yeah, that is the optional/usual assumed mode in k8s - it can though also use ephemeral storage which then cannot be distributed to other nodes just like that | |