Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2021-11-02
11:21:39 kashyap No rush; I don't count on instant responses :-)
11:22:02 lyarwood kashyap: what if you also define -machine in the XML?
11:22:24 kashyap Hmm, lemme try
11:22:53 kashyap lyarwood: Wait, you mean setting -accel and -machine in qemu:commandline explicitly?
11:23:45 lyarwood kashyap: yes
11:24:00 kashyap (If so, that should fail the same way as above, but lemme double-confirm. libvirt uses "-machine accel" under the hood, by inference from <domain type='kvm')
11:24:32 kashyap Yep, it fails the same way.
11:25:11 kashyap lyarwood: Oh, wait. There might be another hack, based on my chat w/ Paolo last week:
11:25:24 kashyap 17:55 < kashyap> bonzini: Hm, how exactl does "-machine accel=kvm -machine accel=tcg" differ from "-accel kvm -accel tcg"?
11:25:27 kashyap 17:55 < bonzini> "-machine accel=tcg" overwrites "-machine accel=kvm"
11:25:41 kashyap lyarwood: So, I can specify by "-accel tcg -accel kvm" ... and see if that works :D
11:26:26 kashyap Gaah, no, ignore me. I misread the above complexity.
11:39:55 kashyap No, there isn't a current trick.
11:40:32 kashyap lyarwood: That said, based on last week chat w/ QEMU folks, libvirt itself should switch to "-accel" as that's recommended than "-machine accel"
11:41:02 lyarwood kashyap: argh kk, so there's no workaround until that happens
11:41:17 kashyap No, besides the ugly hack we both revulse at :D
11:41:21 lyarwood kashyap: blocking pretty much all upstream Openstack testing using qemu until then
11:41:30 lyarwood yeah without the custom build
11:41:32 lyarwood urgh
11:41:47 lyarwood tbh we need to make a big deal out of this
11:42:50 kashyap Yeah, QEMU changed it pretty much w/o considering the management tools :-(
11:47:05 opendevreview Lee Yarwood proposed openstack/nova master: nova-manage: Always get BDMs using get_by_volume_and_instance https://review.opendev.org/c/openstack/nova/+/811716
11:47:41 lyarwood https://review.opendev.org/q/topic:%22bug%252F1943431%22+(status:open%20OR%20status:merged) & https://review.opendev.org/q/topic:%22bug%252F1937084%22+(status:open%20OR%20status:merged) should be ready for reviews if people have time btw, simple bugfixes
12:01:15 EugenMayer When trying to rebuild an instance and use --preserve-ephimeral is see `The current driver does not support preserving ephemeral partitions.`
12:01:54 EugenMayer Is this option only available when using a storage like nfs/ceph? But how is that different to volumes then? Currently i use the compute node local storage
12:03:21 EugenMayer I use LVM on my computes with ext4 - do i need zfs/btrfs for that to work?
12:04:10 sean-k-mooney[m] EugenMayer that is only supported on ironic
12:04:34 sean-k-mooney[m] its not supported with libvirt or any other vm or container based driver
12:05:15 sean-k-mooney[m] rebuild is intended to remove all data from ths instance by recrating the root disk any epmeral disks
12:05:40 sean-k-mooney[m] if you want to use rebuild and preserve data you should store your data in cinder volumes
12:06:32 EugenMayer i though of ironic as just a barebone provisioning (via biofrost?) then running libvirt - but that is wrong?
12:07:02 EugenMayer ironic means, that one does not use any hypervisor at all - the barebone is the actual instance. That is the point right? sean-k-mooney[m]
12:07:47 sean-k-mooney[m] ironic is openstack beremetal as a service project and it can be used with nova to provide instance that are phsyicl servers instead of vms
12:08:08 EugenMayer Understood - thank you for clarifing
12:08:12 sean-k-mooney[m] bifrost is an installer for ironic written in ansible
12:09:05 sean-k-mooney[m] bifrost installs ironic in standalone mode so it canbe used without the rest of openstack to manage your phsyical hardware
12:11:11 EugenMayer Thank you!
12:12:26 sean-k-mooney[m] lyarwood: given we do not allow the use of qemu arg passthough in nova i dont see anyway for us to adress this in nova
12:12:44 EugenMayer One question to cinder - how do you deal with databases? I mean using NFS or alikes and storing (running) a database on such a storage with heavily impact performance - this was the main reason to use local disk (we yet avoided cinder in our setup idea). How do you deal with that? Are you using a specific cinder backend like ceph/gluster so you
12:12:44 EugenMayer actually have local latency and 'sync' the data to the central storage?
12:14:51 sean-k-mooney[m] for database workload i think its more common to use a dedicated san and mount the data over iscsi nfs really is not up to that level of iops. ceph can handel database but generally you will need to use flash if you have high iops
12:15:06 sean-k-mooney[m] i know that many do use local for dbs
12:15:51 sean-k-mooney[m] e.g. the root disk or epmeral disk but then you just need to ensure that you do not use rebuild and make backups at the application level
12:16:15 EugenMayer ok so this is a common issue
12:17:01 EugenMayer sean-k-mooney[m] with flash you mean SSD/NVME drives, right? (we have those only, the latter)
12:18:34 sean-k-mooney[m] yes if you have nvme storage and a high speed 25G+ networking you can deploy high iops workloads on cpeh but your network will become the bottleneck
12:19:43 EugenMayer well our network is about 1GB
12:19:52 EugenMayer it's provider based
12:20:45 EugenMayer do i understand ceph correcly here, that it is actually local access when a 'backend sync' in the background, other then nfs which is a transparent access on the network mount with the performance pain
12:21:09 sean-k-mooney[m] the normal way to deploy databases with local storage is to deploy them in a 3 node ha cluster with local sotrage and backup to cinder volumes with update managed via yum/apt ectra inside the vms.
12:21:43 sean-k-mooney[m] no ceph is directly acessed over the network
12:22:04 sean-k-mooney[m] it use the rbd protocol rather then iscsi but its more similar to iscsi then nfs
12:23:43 EugenMayer this means that one rather runs central db clusters for each DB variant (5.5,5.6,8 or pg 9.6,10,10) and loses the encapsulation that every app travels with it's database like we rather used right now (self container docker-compose / k8s stacks)
12:24:02 sean-k-mooney[m] if you need to use rebuild because of a higher level orchestror then effectivly the best way to do that with only local sotrage i is to serials the rebuild and by removing 1 instace form the cluser, rebuilding it, rejoin it to the cluster then waiting for it to sync with the latest state. then repeat for the rest.
12:24:44 sean-k-mooney[m] well with k8s it change slightly
12:25:03 sean-k-mooney[m] in that it assumes that you have shared network based storage by default
12:25:21 EugenMayer it can be, also k8s can use ephemeral, which we plan to
12:25:37 sean-k-mooney[m] so it assume it cna just terminate the db contaienr and when its recreated after update it can reconnect to the same storeage on any host and get the data back
12:26:19 sean-k-mooney[m] you plan ot use the local provide to back your persitnet volume claims?
12:27:01 EugenMayer yeah, that is the optional/usual assumed mode in k8s - it can though also use ephemeral storage which then cannot be distributed to other nodes just like that
12:27:31 sean-k-mooney[m] right the local provide is not normally intended for production use
12:27:36 EugenMayer you plan ot use the local provide to back your persitnet volume claims? <- not sure i can answer this question / understand it
12:27:52 EugenMayer local provide is what you define as 'local disk'
12:28:07 sean-k-mooney[m] it can be used of corse but it puts the burden of persitign the storage on the operator to configure the storage to be ha by some means
12:28:34 EugenMayer or no ha at all
12:28:46 EugenMayer depends on the needs, of course
12:29:36 EugenMayer sean-k-mooney[m] you really helped enormously. Thanks. I guess i have to rethink the ideas with all the input given.
12:29:38 sean-k-mooney[m] EugenMayer: am in k8s there are 2 ways to use local storage 1 dont use a k8s volume and just use the storage in the contaienr fs, 2 confire a local storage provider on each host that can be used to creat persitent volumen that are attached to pods vis persistent volume claims
12:30:32 EugenMayer sean-k-mooney[m] i think we planned the second
12:30:47 sean-k-mooney[m] but yes if you have no ha that works as long as you are careful with the data replication and or dont need the data to outlive the conatienr
12:33:00 sean-k-mooney[m] i have never done this mind you but i have always wondered how mergerfs would work in production. e.g. could you use it to merge your local storage with remote such that all data is sync to the remote sotrage in the background.
12:33:23 sean-k-mooney[m] anyway im not sure how much i helped but i have to run o/
12:42:19 opendevreview Merged openstack/nova master: Fix unit test for oslo.concurrency 4.5 https://review.opendev.org/c/openstack/nova/+/815940
12:42:41 opendevreview Merged openstack/nova master: Query ports with admin client to get resource_request https://review.opendev.org/c/openstack/nova/+/811396
12:42:51 opendevreview Merged openstack/nova master: Revert "Temp disable nova-manage placement heal_allocation testing" https://review.opendev.org/c/openstack/nova/+/816242
13:24:56 lyarwood sean-k-mooney: hey sorry missed your reply above, yeah we could easily add the config classes and only use them with a workarounds configurable but as kashyap highlighted even then thanks to some other QEMU bugs it isn't going to work
13:25:35 EugenMayer sean-k-mooney[m] you helped me BIG times.
13:29:47 EugenMayer sean-k-mooney[m] i ask myself if working with freezer could be an option for rebuilding, while sticking to local disks
14:31:07 gibi lyarwood: could I ask you a favor to look at the backports of https://review.opendev.org/q/topic:bug/1944759 it is a fairly easy patch and clean backport
14:32:18 lyarwood ack, just catching up with some downstream backports first then I'll try to get to these
14:32:43 lyarwood FWIW I'll be afk for the upstream meeting today, have to fetch my kid from nursery
14:35:14 gibi thanks lyarwood
14:37:44 bauzas gibi: hmpff, I need to ask you two things
14:37:51 gibi bauzas: sure, shoot
14:38:13 bauzas 1/ I'll need to go off at 5.30pm in our TZ, so I could chair the nova meeting for only 30 mins
14:38:22 bauzas 2/ I'll be off tomorrow
14:38:47 gibi 1/ I will be here to chair things after you left
14:38:55 gibi 2/ I will be off tomorrow too :)
14:40:10 bauzas ahah ok :)
14:40:12 bauzas thanks :)
14:40:23 gibi I'm not sure what was the request in 2/ :D
14:50:28 bauzas gibi: just to tell you I was off :)
14:50:48 gibi ahh OK
15:05:19 artom That (wait, what excactly?) reminds me - what do we need to do to get https://review.opendev.org/c/openstack/nova/+/796909 moving?
15:05:45 artom That's only the top change, there's 2 backport series below it, with some dependencies that are... borderline backportable? I mean to me they clearly are
15:05:53 artom But there's some policy controversy there :)
15:09:43 EugenMayer when having not swift / cinder active, and i do a snapshot on a compute, where is the snapshot located?
15:20:27 opendevreview Balazs Gibizer proposed openstack/nova master: Enable min pps tempest testing in nova-next https://review.opendev.org/c/openstack/nova/+/811748
15:32:27 bauzas warning European folks, our nova meeting will start in 28 mins !!!
15:32:46 bauzas daylight savings change

Earlier   Later