Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2021-11-02
12:33:00 sean-k-mooney[m] i have never done this mind you but i have always wondered how mergerfs would work in production. e.g. could you use it to merge your local storage with remote such that all data is sync to the remote sotrage in the background.
12:33:23 sean-k-mooney[m] anyway im not sure how much i helped but i have to run o/
12:42:19 opendevreview Merged openstack/nova master: Fix unit test for oslo.concurrency 4.5 https://review.opendev.org/c/openstack/nova/+/815940
12:42:41 opendevreview Merged openstack/nova master: Query ports with admin client to get resource_request https://review.opendev.org/c/openstack/nova/+/811396
12:42:51 opendevreview Merged openstack/nova master: Revert "Temp disable nova-manage placement heal_allocation testing" https://review.opendev.org/c/openstack/nova/+/816242
13:24:56 lyarwood sean-k-mooney: hey sorry missed your reply above, yeah we could easily add the config classes and only use them with a workarounds configurable but as kashyap highlighted even then thanks to some other QEMU bugs it isn't going to work
13:25:35 EugenMayer sean-k-mooney[m] you helped me BIG times.
13:29:47 EugenMayer sean-k-mooney[m] i ask myself if working with freezer could be an option for rebuilding, while sticking to local disks
14:31:07 gibi lyarwood: could I ask you a favor to look at the backports of https://review.opendev.org/q/topic:bug/1944759 it is a fairly easy patch and clean backport
14:32:18 lyarwood ack, just catching up with some downstream backports first then I'll try to get to these
14:32:43 lyarwood FWIW I'll be afk for the upstream meeting today, have to fetch my kid from nursery
14:35:14 gibi thanks lyarwood
14:37:44 bauzas gibi: hmpff, I need to ask you two things
14:37:51 gibi bauzas: sure, shoot
14:38:13 bauzas 1/ I'll need to go off at 5.30pm in our TZ, so I could chair the nova meeting for only 30 mins
14:38:22 bauzas 2/ I'll be off tomorrow
14:38:47 gibi 1/ I will be here to chair things after you left
14:38:55 gibi 2/ I will be off tomorrow too :)
14:40:10 bauzas ahah ok :)
14:40:12 bauzas thanks :)
14:40:23 gibi I'm not sure what was the request in 2/ :D
14:50:28 bauzas gibi: just to tell you I was off :)
14:50:48 gibi ahh OK
15:05:19 artom That (wait, what excactly?) reminds me - what do we need to do to get https://review.opendev.org/c/openstack/nova/+/796909 moving?
15:05:45 artom That's only the top change, there's 2 backport series below it, with some dependencies that are... borderline backportable? I mean to me they clearly are
15:05:53 artom But there's some policy controversy there :)
15:09:43 EugenMayer when having not swift / cinder active, and i do a snapshot on a compute, where is the snapshot located?
15:20:27 opendevreview Balazs Gibizer proposed openstack/nova master: Enable min pps tempest testing in nova-next https://review.opendev.org/c/openstack/nova/+/811748
15:32:27 bauzas warning European folks, our nova meeting will start in 28 mins !!!
15:32:46 bauzas daylight savings change
15:33:19 bauzas 1600UTC is now 5pm for CET and 4pm for BST
15:36:15 opendevreview Balazs Gibizer proposed openstack/nova stable/xena: Reproduce bug 1945310 https://review.opendev.org/c/openstack/nova/+/811405
15:36:16 opendevreview Balazs Gibizer proposed openstack/nova stable/xena: Query ports with admin client to get resource_request https://review.opendev.org/c/openstack/nova/+/811407
15:37:49 EugenMayer running kvm on a compute (empty) with 128GB ram, EE nvmes (very fast, raid1 mdadm) with 4 cores on a AMD Ryzen 9 5950X 16-Core Processor, the instance takes 3 minutes to install python3-pip, while it is not the download, but the processing. What could be the reason that it underperformance so massively?
15:38:07 EugenMayer doing the same on the compute host would take < 8s
15:39:32 EugenMayer wait the compute shows as QEMU in the hypervisor overview..how did that happen?
15:40:17 sean-k-mooney[m] EugenMayer: kvm is an accleration not a hypervior
15:40:25 sean-k-mooney[m] qemu is the hyperviors
15:40:47 sean-k-mooney[m] we show the same regardless of if you use qemu with the tcg backend or kvm
15:41:19 EugenMayer i see
15:41:21 bauzas kashyap: lyarwood: need help with some bug to triage upstream https://bugs.launchpad.net/nova/+bug/1949224
15:41:39 EugenMayer i wondered, since the default of nova_compute_virt_type is kvm
15:42:05 EugenMayer sean-k-mooney[m] how to be sure KVM is used?
15:42:14 kashyap bauzas: Hmm, I have a vague recall of a similar issue from the past, but I need to reload some context. Will check tomm
15:42:31 bauzas kashyap: I just asked for qemu/libvirt versions
15:42:32 sean-k-mooney[m] on the compute host you can do a virsh list then a virsh dumpxml on the domian
15:42:42 sean-k-mooney[m] EugenMayer: ^
15:42:54 bauzas kashyap: I'm tempted to say this is unrelated to Nova but rather a libvirt/virt bug
15:42:57 kashyap bauzas: That's a good catch. We don't test this area all that much
15:43:03 kashyap Let's see if it's a bug yet :)
15:43:36 EugenMayer sean-k-mooney[m]: <domain type='kvm' id='1'>
15:44:02 sean-k-mooney[m] yep that should be using kvm then
15:44:08 EugenMayer so what could be the actual reason for this massive slowdown? this host easily hosted heavy load with proxmox
15:45:02 sean-k-mooney[m] what do you have the cpu model set to?
15:45:08 sean-k-mooney[m] and mode
15:45:28 sean-k-mooney[m] by default its unset which for the libvirt dirver will get treated as if you set host-model
15:45:36 EugenMayer sean-k-mooney[m] https://gist.github.com/EugenMayer/cacd5c44ae7dafffa31c9c8025bee3aa
15:46:15 sean-k-mooney[m] i know that there were slowdowns on older qemu versions with amd cpus because they did not have the correct cpu_models defiedn
15:46:49 sean-k-mooney[m] you should proably try setting [libvirt]/cpu_mode=host-passthrough
15:46:57 EugenMayer this is bullseye debian - same as for proxmox
15:47:26 EugenMayer i would assume that it would run the same way. Sureley pve uses a different kernel but still suprised
15:48:00 sean-k-mooney[m] if that improves the performance on the razen 5950x then it likely means you just need a newer qemu to get better perfromance out of the box without using host-passhtough
15:48:25 EugenMayer understood
15:48:42 EugenMayer trying to up the kernel from the backports and will check what qemu version would be on ubuntu
15:48:48 EugenMayer thank you for the hint!
15:48:59 EugenMayer how would i actually change the cpu_mode with nova?
15:49:26 sean-k-mooney[m] in the nova.conf set cpu_mode in the libvirt section
15:49:40 opendevreview Balazs Gibizer proposed openstack/nova stable/wallaby: Query ports with admin client to get resource_request https://review.opendev.org/c/openstack/nova/+/811416
15:50:17 EugenMayer sean-k-mooney[m] do i need to redo the instances or just stop/start?
15:50:59 bauzas last reminder: nova weekly meeting starts in 10 mins now here in this chan :)
15:51:14 EugenMayer bauzas i will shut up then, promised :)
15:51:31 bauzas (spoiler alert, you'll see new meetbot commands :p )
15:51:37 sean-k-mooney[m] just hard reboot after restarting the agent
15:51:41 sean-k-mooney[m] https://docs.openstack.org/nova/latest/configuration/config.html#libvirt.cpu_mode
15:52:14 EugenMayer sean-k-mooney[m] again, thanks!
15:52:21 bauzas (actually, not 'new', since I used them in 2014, but I haven't seen them used for a while now, so I'm all about using them back :D )
15:52:34 sean-k-mooney[m] EugenMayer: you likely dont need to update the kernel by the way but its worth a shot if that does not help
15:52:45 EugenMayer bauzas you jinxed it, so it might fail! :D
15:53:55 bauzas the more people joining us, the better it will be :)
15:54:05 opendevreview Balazs Gibizer proposed openstack/nova stable/wallaby: Reproduce bug 1945310 https://review.opendev.org/c/openstack/nova/+/811414
15:54:14 bauzas if I can make jokes to let people chime in, I can try
15:55:12 opendevreview Balazs Gibizer proposed openstack/nova stable/wallaby: Query ports with admin client to get resource_request https://review.opendev.org/c/openstack/nova/+/811416
15:55:20 opendevreview Balazs Gibizer proposed openstack/nova stable/wallaby: Query ports with admin client to get resource_request https://review.opendev.org/c/openstack/nova/+/811416
15:58:18 EugenMayer sean-k-mooney[m] massive speedup - 17 seconds
15:59:49 sean-k-mooney[m] cool so basically what happening qemu does not have a profile that matches your cpu. the simplet way to fix that is to update qemu to on that does. failing that host-passthough is a good choice if you dont need to live migrate the vms to different cpu models
16:00:04 sean-k-mooney[m] althernitvie you can pick one that is close and add flags
16:00:18 sean-k-mooney[m] we can tell you how to do that after the meeting but what you have should be fine for now
16:00:42 bauzas 3...
16:00:47 bauzas 2...
16:00:48 bauzas 1...
16:00:51 bauzas #startmeeting nova
16:00:51 opendevmeet Meeting started Tue Nov 2 16:00:51 2021 UTC and is due to finish in 60 minutes. The chair is bauzas. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:51 opendevmeet Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:00:51 opendevmeet The meeting name has been set to 'nova'
16:01:07 bauzas #link https://wiki.openstack.org/wiki/Meetings/Nova#Agenda_for_next_meeting
16:01:16 bauzas good day, everyone
16:01:19 dansmith o/
16:01:27 sean-k-mooney[m] o/
16:01:59 gibi \o
16:02:02 bauzas as discussed before, I will exceptionnally be only able to chair this meeting for 30 mins

Earlier   Later