| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-11-02 | |||
| 12:01:54 | EugenMayer | Is this option only available when using a storage like nfs/ceph? But how is that different to volumes then? Currently i use the compute node local storage | |
| 12:03:21 | EugenMayer | I use LVM on my computes with ext4 - do i need zfs/btrfs for that to work? | |
| 12:04:10 | sean-k-mooney[m] | EugenMayer that is only supported on ironic | |
| 12:04:34 | sean-k-mooney[m] | its not supported with libvirt or any other vm or container based driver | |
| 12:05:15 | sean-k-mooney[m] | rebuild is intended to remove all data from ths instance by recrating the root disk any epmeral disks | |
| 12:05:40 | sean-k-mooney[m] | if you want to use rebuild and preserve data you should store your data in cinder volumes | |
| 12:06:32 | EugenMayer | i though of ironic as just a barebone provisioning (via biofrost?) then running libvirt - but that is wrong? | |
| 12:07:02 | EugenMayer | ironic means, that one does not use any hypervisor at all - the barebone is the actual instance. That is the point right? sean-k-mooney[m] | |
| 12:07:47 | sean-k-mooney[m] | ironic is openstack beremetal as a service project and it can be used with nova to provide instance that are phsyicl servers instead of vms | |
| 12:08:08 | EugenMayer | Understood - thank you for clarifing | |
| 12:08:12 | sean-k-mooney[m] | bifrost is an installer for ironic written in ansible | |
| 12:09:05 | sean-k-mooney[m] | bifrost installs ironic in standalone mode so it canbe used without the rest of openstack to manage your phsyical hardware | |
| 12:11:11 | EugenMayer | Thank you! | |
| 12:12:26 | sean-k-mooney[m] | lyarwood: given we do not allow the use of qemu arg passthough in nova i dont see anyway for us to adress this in nova | |
| 12:12:44 | EugenMayer | One question to cinder - how do you deal with databases? I mean using NFS or alikes and storing (running) a database on such a storage with heavily impact performance - this was the main reason to use local disk (we yet avoided cinder in our setup idea). How do you deal with that? Are you using a specific cinder backend like ceph/gluster so you | |
| 12:12:44 | EugenMayer | actually have local latency and 'sync' the data to the central storage? | |
| 12:14:51 | sean-k-mooney[m] | for database workload i think its more common to use a dedicated san and mount the data over iscsi nfs really is not up to that level of iops. ceph can handel database but generally you will need to use flash if you have high iops | |
| 12:15:06 | sean-k-mooney[m] | i know that many do use local for dbs | |
| 12:15:51 | sean-k-mooney[m] | e.g. the root disk or epmeral disk but then you just need to ensure that you do not use rebuild and make backups at the application level | |
| 12:16:15 | EugenMayer | ok so this is a common issue | |
| 12:17:01 | EugenMayer | sean-k-mooney[m] with flash you mean SSD/NVME drives, right? (we have those only, the latter) | |
| 12:18:34 | sean-k-mooney[m] | yes if you have nvme storage and a high speed 25G+ networking you can deploy high iops workloads on cpeh but your network will become the bottleneck | |
| 12:19:43 | EugenMayer | well our network is about 1GB | |
| 12:19:52 | EugenMayer | it's provider based | |
| 12:20:45 | EugenMayer | do i understand ceph correcly here, that it is actually local access when a 'backend sync' in the background, other then nfs which is a transparent access on the network mount with the performance pain | |
| 12:21:09 | sean-k-mooney[m] | the normal way to deploy databases with local storage is to deploy them in a 3 node ha cluster with local sotrage and backup to cinder volumes with update managed via yum/apt ectra inside the vms. | |
| 12:21:43 | sean-k-mooney[m] | no ceph is directly acessed over the network | |
| 12:22:04 | sean-k-mooney[m] | it use the rbd protocol rather then iscsi but its more similar to iscsi then nfs | |
| 12:23:43 | EugenMayer | this means that one rather runs central db clusters for each DB variant (5.5,5.6,8 or pg 9.6,10,10) and loses the encapsulation that every app travels with it's database like we rather used right now (self container docker-compose / k8s stacks) | |
| 12:24:02 | sean-k-mooney[m] | if you need to use rebuild because of a higher level orchestror then effectivly the best way to do that with only local sotrage i is to serials the rebuild and by removing 1 instace form the cluser, rebuilding it, rejoin it to the cluster then waiting for it to sync with the latest state. then repeat for the rest. | |
| 12:24:44 | sean-k-mooney[m] | well with k8s it change slightly | |
| 12:25:03 | sean-k-mooney[m] | in that it assumes that you have shared network based storage by default | |
| 12:25:21 | EugenMayer | it can be, also k8s can use ephemeral, which we plan to | |
| 12:25:37 | sean-k-mooney[m] | so it assume it cna just terminate the db contaienr and when its recreated after update it can reconnect to the same storeage on any host and get the data back | |
| 12:26:19 | sean-k-mooney[m] | you plan ot use the local provide to back your persitnet volume claims? | |
| 12:27:01 | EugenMayer | yeah, that is the optional/usual assumed mode in k8s - it can though also use ephemeral storage which then cannot be distributed to other nodes just like that | |
| 12:27:31 | sean-k-mooney[m] | right the local provide is not normally intended for production use | |
| 12:27:36 | EugenMayer | you plan ot use the local provide to back your persitnet volume claims? <- not sure i can answer this question / understand it | |
| 12:27:52 | EugenMayer | local provide is what you define as 'local disk' | |
| 12:28:07 | sean-k-mooney[m] | it can be used of corse but it puts the burden of persitign the storage on the operator to configure the storage to be ha by some means | |
| 12:28:34 | EugenMayer | or no ha at all | |
| 12:28:46 | EugenMayer | depends on the needs, of course | |
| 12:29:36 | EugenMayer | sean-k-mooney[m] you really helped enormously. Thanks. I guess i have to rethink the ideas with all the input given. | |
| 12:29:38 | sean-k-mooney[m] | EugenMayer: am in k8s there are 2 ways to use local storage 1 dont use a k8s volume and just use the storage in the contaienr fs, 2 confire a local storage provider on each host that can be used to creat persitent volumen that are attached to pods vis persistent volume claims | |
| 12:30:32 | EugenMayer | sean-k-mooney[m] i think we planned the second | |
| 12:30:47 | sean-k-mooney[m] | but yes if you have no ha that works as long as you are careful with the data replication and or dont need the data to outlive the conatienr | |
| 12:33:00 | sean-k-mooney[m] | i have never done this mind you but i have always wondered how mergerfs would work in production. e.g. could you use it to merge your local storage with remote such that all data is sync to the remote sotrage in the background. | |
| 12:33:23 | sean-k-mooney[m] | anyway im not sure how much i helped but i have to run o/ | |
| 12:42:19 | opendevreview | Merged openstack/nova master: Fix unit test for oslo.concurrency 4.5 https://review.opendev.org/c/openstack/nova/+/815940 | |
| 12:42:41 | opendevreview | Merged openstack/nova master: Query ports with admin client to get resource_request https://review.opendev.org/c/openstack/nova/+/811396 | |
| 12:42:51 | opendevreview | Merged openstack/nova master: Revert "Temp disable nova-manage placement heal_allocation testing" https://review.opendev.org/c/openstack/nova/+/816242 | |
| 13:24:56 | lyarwood | sean-k-mooney: hey sorry missed your reply above, yeah we could easily add the config classes and only use them with a workarounds configurable but as kashyap highlighted even then thanks to some other QEMU bugs it isn't going to work | |
| 13:25:35 | EugenMayer | sean-k-mooney[m] you helped me BIG times. | |
| 13:29:47 | EugenMayer | sean-k-mooney[m] i ask myself if working with freezer could be an option for rebuilding, while sticking to local disks | |
| 14:31:07 | gibi | lyarwood: could I ask you a favor to look at the backports of https://review.opendev.org/q/topic:bug/1944759 it is a fairly easy patch and clean backport | |
| 14:32:18 | lyarwood | ack, just catching up with some downstream backports first then I'll try to get to these | |
| 14:32:43 | lyarwood | FWIW I'll be afk for the upstream meeting today, have to fetch my kid from nursery | |
| 14:35:14 | gibi | thanks lyarwood | |
| 14:37:44 | bauzas | gibi: hmpff, I need to ask you two things | |
| 14:37:51 | gibi | bauzas: sure, shoot | |
| 14:38:13 | bauzas | 1/ I'll need to go off at 5.30pm in our TZ, so I could chair the nova meeting for only 30 mins | |
| 14:38:22 | bauzas | 2/ I'll be off tomorrow | |
| 14:38:47 | gibi | 1/ I will be here to chair things after you left | |
| 14:38:55 | gibi | 2/ I will be off tomorrow too :) | |
| 14:40:10 | bauzas | ahah ok :) | |
| 14:40:12 | bauzas | thanks :) | |
| 14:40:23 | gibi | I'm not sure what was the request in 2/ :D | |
| 14:50:28 | bauzas | gibi: just to tell you I was off :) | |
| 14:50:48 | gibi | ahh OK | |
| 15:05:19 | artom | That (wait, what excactly?) reminds me - what do we need to do to get https://review.opendev.org/c/openstack/nova/+/796909 moving? | |
| 15:05:45 | artom | That's only the top change, there's 2 backport series below it, with some dependencies that are... borderline backportable? I mean to me they clearly are | |
| 15:05:53 | artom | But there's some policy controversy there :) | |
| 15:09:43 | EugenMayer | when having not swift / cinder active, and i do a snapshot on a compute, where is the snapshot located? | |
| 15:20:27 | opendevreview | Balazs Gibizer proposed openstack/nova master: Enable min pps tempest testing in nova-next https://review.opendev.org/c/openstack/nova/+/811748 | |
| 15:32:27 | bauzas | warning European folks, our nova meeting will start in 28 mins !!! | |
| 15:32:46 | bauzas | daylight savings change | |
| 15:33:19 | bauzas | 1600UTC is now 5pm for CET and 4pm for BST | |
| 15:36:15 | opendevreview | Balazs Gibizer proposed openstack/nova stable/xena: Reproduce bug 1945310 https://review.opendev.org/c/openstack/nova/+/811405 | |
| 15:36:16 | opendevreview | Balazs Gibizer proposed openstack/nova stable/xena: Query ports with admin client to get resource_request https://review.opendev.org/c/openstack/nova/+/811407 | |
| 15:37:49 | EugenMayer | running kvm on a compute (empty) with 128GB ram, EE nvmes (very fast, raid1 mdadm) with 4 cores on a AMD Ryzen 9 5950X 16-Core Processor, the instance takes 3 minutes to install python3-pip, while it is not the download, but the processing. What could be the reason that it underperformance so massively? | |
| 15:38:07 | EugenMayer | doing the same on the compute host would take < 8s | |
| 15:39:32 | EugenMayer | wait the compute shows as QEMU in the hypervisor overview..how did that happen? | |
| 15:40:17 | sean-k-mooney[m] | EugenMayer: kvm is an accleration not a hypervior | |
| 15:40:25 | sean-k-mooney[m] | qemu is the hyperviors | |
| 15:40:47 | sean-k-mooney[m] | we show the same regardless of if you use qemu with the tcg backend or kvm | |
| 15:41:19 | EugenMayer | i see | |
| 15:41:21 | bauzas | kashyap: lyarwood: need help with some bug to triage upstream https://bugs.launchpad.net/nova/+bug/1949224 | |
| 15:41:39 | EugenMayer | i wondered, since the default of nova_compute_virt_type is kvm | |
| 15:42:05 | EugenMayer | sean-k-mooney[m] how to be sure KVM is used? | |
| 15:42:14 | kashyap | bauzas: Hmm, I have a vague recall of a similar issue from the past, but I need to reload some context. Will check tomm | |
| 15:42:31 | bauzas | kashyap: I just asked for qemu/libvirt versions | |
| 15:42:32 | sean-k-mooney[m] | on the compute host you can do a virsh list then a virsh dumpxml on the domian | |
| 15:42:42 | sean-k-mooney[m] | EugenMayer: ^ | |
| 15:42:54 | bauzas | kashyap: I'm tempted to say this is unrelated to Nova but rather a libvirt/virt bug | |
| 15:42:57 | kashyap | bauzas: That's a good catch. We don't test this area all that much | |
| 15:43:03 | kashyap | Let's see if it's a bug yet :) | |
| 15:43:36 | EugenMayer | sean-k-mooney[m]: <domain type='kvm' id='1'> | |
| 15:44:02 | sean-k-mooney[m] | yep that should be using kvm then | |
| 15:44:08 | EugenMayer | so what could be the actual reason for this massive slowdown? this host easily hosted heavy load with proxmox | |
| 15:45:02 | sean-k-mooney[m] | what do you have the cpu model set to? | |