| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-11-02 | |||
| 12:42:51 | opendevreview | Merged openstack/nova master: Revert "Temp disable nova-manage placement heal_allocation testing" https://review.opendev.org/c/openstack/nova/+/816242 | |
| 13:24:56 | lyarwood | sean-k-mooney: hey sorry missed your reply above, yeah we could easily add the config classes and only use them with a workarounds configurable but as kashyap highlighted even then thanks to some other QEMU bugs it isn't going to work | |
| 13:25:35 | EugenMayer | sean-k-mooney[m] you helped me BIG times. | |
| 13:29:47 | EugenMayer | sean-k-mooney[m] i ask myself if working with freezer could be an option for rebuilding, while sticking to local disks | |
| 14:31:07 | gibi | lyarwood: could I ask you a favor to look at the backports of https://review.opendev.org/q/topic:bug/1944759 it is a fairly easy patch and clean backport | |
| 14:32:18 | lyarwood | ack, just catching up with some downstream backports first then I'll try to get to these | |
| 14:32:43 | lyarwood | FWIW I'll be afk for the upstream meeting today, have to fetch my kid from nursery | |
| 14:35:14 | gibi | thanks lyarwood | |
| 14:37:44 | bauzas | gibi: hmpff, I need to ask you two things | |
| 14:37:51 | gibi | bauzas: sure, shoot | |
| 14:38:13 | bauzas | 1/ I'll need to go off at 5.30pm in our TZ, so I could chair the nova meeting for only 30 mins | |
| 14:38:22 | bauzas | 2/ I'll be off tomorrow | |
| 14:38:47 | gibi | 1/ I will be here to chair things after you left | |
| 14:38:55 | gibi | 2/ I will be off tomorrow too :) | |
| 14:40:10 | bauzas | ahah ok :) | |
| 14:40:12 | bauzas | thanks :) | |
| 14:40:23 | gibi | I'm not sure what was the request in 2/ :D | |
| 14:50:28 | bauzas | gibi: just to tell you I was off :) | |
| 14:50:48 | gibi | ahh OK | |
| 15:05:19 | artom | That (wait, what excactly?) reminds me - what do we need to do to get https://review.opendev.org/c/openstack/nova/+/796909 moving? | |
| 15:05:45 | artom | That's only the top change, there's 2 backport series below it, with some dependencies that are... borderline backportable? I mean to me they clearly are | |
| 15:05:53 | artom | But there's some policy controversy there :) | |
| 15:09:43 | EugenMayer | when having not swift / cinder active, and i do a snapshot on a compute, where is the snapshot located? | |
| 15:20:27 | opendevreview | Balazs Gibizer proposed openstack/nova master: Enable min pps tempest testing in nova-next https://review.opendev.org/c/openstack/nova/+/811748 | |
| 15:32:27 | bauzas | warning European folks, our nova meeting will start in 28 mins !!! | |
| 15:32:46 | bauzas | daylight savings change | |
| 15:33:19 | bauzas | 1600UTC is now 5pm for CET and 4pm for BST | |
| 15:36:15 | opendevreview | Balazs Gibizer proposed openstack/nova stable/xena: Reproduce bug 1945310 https://review.opendev.org/c/openstack/nova/+/811405 | |
| 15:36:16 | opendevreview | Balazs Gibizer proposed openstack/nova stable/xena: Query ports with admin client to get resource_request https://review.opendev.org/c/openstack/nova/+/811407 | |
| 15:37:49 | EugenMayer | running kvm on a compute (empty) with 128GB ram, EE nvmes (very fast, raid1 mdadm) with 4 cores on a AMD Ryzen 9 5950X 16-Core Processor, the instance takes 3 minutes to install python3-pip, while it is not the download, but the processing. What could be the reason that it underperformance so massively? | |
| 15:38:07 | EugenMayer | doing the same on the compute host would take < 8s | |
| 15:39:32 | EugenMayer | wait the compute shows as QEMU in the hypervisor overview..how did that happen? | |
| 15:40:17 | sean-k-mooney[m] | EugenMayer: kvm is an accleration not a hypervior | |
| 15:40:25 | sean-k-mooney[m] | qemu is the hyperviors | |
| 15:40:47 | sean-k-mooney[m] | we show the same regardless of if you use qemu with the tcg backend or kvm | |
| 15:41:19 | EugenMayer | i see | |
| 15:41:21 | bauzas | kashyap: lyarwood: need help with some bug to triage upstream https://bugs.launchpad.net/nova/+bug/1949224 | |
| 15:41:39 | EugenMayer | i wondered, since the default of nova_compute_virt_type is kvm | |
| 15:42:05 | EugenMayer | sean-k-mooney[m] how to be sure KVM is used? | |
| 15:42:14 | kashyap | bauzas: Hmm, I have a vague recall of a similar issue from the past, but I need to reload some context. Will check tomm | |
| 15:42:31 | bauzas | kashyap: I just asked for qemu/libvirt versions | |
| 15:42:32 | sean-k-mooney[m] | on the compute host you can do a virsh list then a virsh dumpxml on the domian | |
| 15:42:42 | sean-k-mooney[m] | EugenMayer: ^ | |
| 15:42:54 | bauzas | kashyap: I'm tempted to say this is unrelated to Nova but rather a libvirt/virt bug | |
| 15:42:57 | kashyap | bauzas: That's a good catch. We don't test this area all that much | |
| 15:43:03 | kashyap | Let's see if it's a bug yet :) | |
| 15:43:36 | EugenMayer | sean-k-mooney[m]: <domain type='kvm' id='1'> | |
| 15:44:02 | sean-k-mooney[m] | yep that should be using kvm then | |
| 15:44:08 | EugenMayer | so what could be the actual reason for this massive slowdown? this host easily hosted heavy load with proxmox | |
| 15:45:02 | sean-k-mooney[m] | what do you have the cpu model set to? | |
| 15:45:08 | sean-k-mooney[m] | and mode | |
| 15:45:28 | sean-k-mooney[m] | by default its unset which for the libvirt dirver will get treated as if you set host-model | |
| 15:45:36 | EugenMayer | sean-k-mooney[m] https://gist.github.com/EugenMayer/cacd5c44ae7dafffa31c9c8025bee3aa | |
| 15:46:15 | sean-k-mooney[m] | i know that there were slowdowns on older qemu versions with amd cpus because they did not have the correct cpu_models defiedn | |
| 15:46:49 | sean-k-mooney[m] | you should proably try setting [libvirt]/cpu_mode=host-passthrough | |
| 15:46:57 | EugenMayer | this is bullseye debian - same as for proxmox | |
| 15:47:26 | EugenMayer | i would assume that it would run the same way. Sureley pve uses a different kernel but still suprised | |
| 15:48:00 | sean-k-mooney[m] | if that improves the performance on the razen 5950x then it likely means you just need a newer qemu to get better perfromance out of the box without using host-passhtough | |
| 15:48:25 | EugenMayer | understood | |
| 15:48:42 | EugenMayer | trying to up the kernel from the backports and will check what qemu version would be on ubuntu | |
| 15:48:48 | EugenMayer | thank you for the hint! | |
| 15:48:59 | EugenMayer | how would i actually change the cpu_mode with nova? | |
| 15:49:26 | sean-k-mooney[m] | in the nova.conf set cpu_mode in the libvirt section | |
| 15:49:40 | opendevreview | Balazs Gibizer proposed openstack/nova stable/wallaby: Query ports with admin client to get resource_request https://review.opendev.org/c/openstack/nova/+/811416 | |
| 15:50:17 | EugenMayer | sean-k-mooney[m] do i need to redo the instances or just stop/start? | |
| 15:50:59 | bauzas | last reminder: nova weekly meeting starts in 10 mins now here in this chan :) | |
| 15:51:14 | EugenMayer | bauzas i will shut up then, promised :) | |
| 15:51:31 | bauzas | (spoiler alert, you'll see new meetbot commands :p ) | |
| 15:51:37 | sean-k-mooney[m] | just hard reboot after restarting the agent | |
| 15:51:41 | sean-k-mooney[m] | https://docs.openstack.org/nova/latest/configuration/config.html#libvirt.cpu_mode | |
| 15:52:14 | EugenMayer | sean-k-mooney[m] again, thanks! | |
| 15:52:21 | bauzas | (actually, not 'new', since I used them in 2014, but I haven't seen them used for a while now, so I'm all about using them back :D ) | |
| 15:52:34 | sean-k-mooney[m] | EugenMayer: you likely dont need to update the kernel by the way but its worth a shot if that does not help | |
| 15:52:45 | EugenMayer | bauzas you jinxed it, so it might fail! :D | |
| 15:53:55 | bauzas | the more people joining us, the better it will be :) | |
| 15:54:05 | opendevreview | Balazs Gibizer proposed openstack/nova stable/wallaby: Reproduce bug 1945310 https://review.opendev.org/c/openstack/nova/+/811414 | |
| 15:54:14 | bauzas | if I can make jokes to let people chime in, I can try | |
| 15:55:12 | opendevreview | Balazs Gibizer proposed openstack/nova stable/wallaby: Query ports with admin client to get resource_request https://review.opendev.org/c/openstack/nova/+/811416 | |
| 15:55:20 | opendevreview | Balazs Gibizer proposed openstack/nova stable/wallaby: Query ports with admin client to get resource_request https://review.opendev.org/c/openstack/nova/+/811416 | |
| 15:58:18 | EugenMayer | sean-k-mooney[m] massive speedup - 17 seconds | |
| 15:59:49 | sean-k-mooney[m] | cool so basically what happening qemu does not have a profile that matches your cpu. the simplet way to fix that is to update qemu to on that does. failing that host-passthough is a good choice if you dont need to live migrate the vms to different cpu models | |
| 16:00:04 | sean-k-mooney[m] | althernitvie you can pick one that is close and add flags | |
| 16:00:18 | sean-k-mooney[m] | we can tell you how to do that after the meeting but what you have should be fine for now | |
| 16:00:42 | bauzas | 3... | |
| 16:00:47 | bauzas | 2... | |
| 16:00:48 | bauzas | 1... | |
| 16:00:51 | opendevmeet | The meeting name has been set to 'nova' | |
| 16:00:51 | opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | |
| 16:00:51 | opendevmeet | Meeting started Tue Nov 2 16:00:51 2021 UTC and is due to finish in 60 minutes. The chair is bauzas. Information about MeetBot at http://wiki.debian.org/MeetBot. | |
| 16:00:51 | bauzas | #startmeeting nova | |
| 16:01:07 | bauzas | #link https://wiki.openstack.org/wiki/Meetings/Nova#Agenda_for_next_meeting | |
| 16:01:16 | bauzas | good day, everyone | |
| 16:01:19 | dansmith | o/ | |
| 16:01:27 | sean-k-mooney[m] | o/ | |
| 16:01:59 | gibi | \o | |
| 16:02:02 | bauzas | as discussed before, I will exceptionnally be only able to chair this meeting for 30 mins | |
| 16:02:12 | bauzas | so I'll let gibi co-chair | |
| 16:02:15 | opendevmeet | Current chairs: bauzas gibi | |
| 16:02:15 | bauzas | #chair gibi | |
| 16:02:39 | bauzas | let's start while people join | |