| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-11-02 | |||
| 15:36:16 | opendevreview | Balazs Gibizer proposed openstack/nova stable/xena: Query ports with admin client to get resource_request https://review.opendev.org/c/openstack/nova/+/811407 | |
| 15:37:49 | EugenMayer | running kvm on a compute (empty) with 128GB ram, EE nvmes (very fast, raid1 mdadm) with 4 cores on a AMD Ryzen 9 5950X 16-Core Processor, the instance takes 3 minutes to install python3-pip, while it is not the download, but the processing. What could be the reason that it underperformance so massively? | |
| 15:38:07 | EugenMayer | doing the same on the compute host would take < 8s | |
| 15:39:32 | EugenMayer | wait the compute shows as QEMU in the hypervisor overview..how did that happen? | |
| 15:40:17 | sean-k-mooney[m] | EugenMayer: kvm is an accleration not a hypervior | |
| 15:40:25 | sean-k-mooney[m] | qemu is the hyperviors | |
| 15:40:47 | sean-k-mooney[m] | we show the same regardless of if you use qemu with the tcg backend or kvm | |
| 15:41:19 | EugenMayer | i see | |
| 15:41:21 | bauzas | kashyap: lyarwood: need help with some bug to triage upstream https://bugs.launchpad.net/nova/+bug/1949224 | |
| 15:41:39 | EugenMayer | i wondered, since the default of nova_compute_virt_type is kvm | |
| 15:42:05 | EugenMayer | sean-k-mooney[m] how to be sure KVM is used? | |
| 15:42:14 | kashyap | bauzas: Hmm, I have a vague recall of a similar issue from the past, but I need to reload some context. Will check tomm | |
| 15:42:31 | bauzas | kashyap: I just asked for qemu/libvirt versions | |
| 15:42:32 | sean-k-mooney[m] | on the compute host you can do a virsh list then a virsh dumpxml on the domian | |
| 15:42:42 | sean-k-mooney[m] | EugenMayer: ^ | |
| 15:42:54 | bauzas | kashyap: I'm tempted to say this is unrelated to Nova but rather a libvirt/virt bug | |
| 15:42:57 | kashyap | bauzas: That's a good catch. We don't test this area all that much | |
| 15:43:03 | kashyap | Let's see if it's a bug yet :) | |
| 15:43:36 | EugenMayer | sean-k-mooney[m]: <domain type='kvm' id='1'> | |
| 15:44:02 | sean-k-mooney[m] | yep that should be using kvm then | |
| 15:44:08 | EugenMayer | so what could be the actual reason for this massive slowdown? this host easily hosted heavy load with proxmox | |
| 15:45:02 | sean-k-mooney[m] | what do you have the cpu model set to? | |
| 15:45:08 | sean-k-mooney[m] | and mode | |
| 15:45:28 | sean-k-mooney[m] | by default its unset which for the libvirt dirver will get treated as if you set host-model | |
| 15:45:36 | EugenMayer | sean-k-mooney[m] https://gist.github.com/EugenMayer/cacd5c44ae7dafffa31c9c8025bee3aa | |
| 15:46:15 | sean-k-mooney[m] | i know that there were slowdowns on older qemu versions with amd cpus because they did not have the correct cpu_models defiedn | |
| 15:46:49 | sean-k-mooney[m] | you should proably try setting [libvirt]/cpu_mode=host-passthrough | |
| 15:46:57 | EugenMayer | this is bullseye debian - same as for proxmox | |
| 15:47:26 | EugenMayer | i would assume that it would run the same way. Sureley pve uses a different kernel but still suprised | |
| 15:48:00 | sean-k-mooney[m] | if that improves the performance on the razen 5950x then it likely means you just need a newer qemu to get better perfromance out of the box without using host-passhtough | |
| 15:48:25 | EugenMayer | understood | |
| 15:48:42 | EugenMayer | trying to up the kernel from the backports and will check what qemu version would be on ubuntu | |
| 15:48:48 | EugenMayer | thank you for the hint! | |
| 15:48:59 | EugenMayer | how would i actually change the cpu_mode with nova? | |
| 15:49:26 | sean-k-mooney[m] | in the nova.conf set cpu_mode in the libvirt section | |
| 15:49:40 | opendevreview | Balazs Gibizer proposed openstack/nova stable/wallaby: Query ports with admin client to get resource_request https://review.opendev.org/c/openstack/nova/+/811416 | |
| 15:50:17 | EugenMayer | sean-k-mooney[m] do i need to redo the instances or just stop/start? | |
| 15:50:59 | bauzas | last reminder: nova weekly meeting starts in 10 mins now here in this chan :) | |
| 15:51:14 | EugenMayer | bauzas i will shut up then, promised :) | |
| 15:51:31 | bauzas | (spoiler alert, you'll see new meetbot commands :p ) | |
| 15:51:37 | sean-k-mooney[m] | just hard reboot after restarting the agent | |
| 15:51:41 | sean-k-mooney[m] | https://docs.openstack.org/nova/latest/configuration/config.html#libvirt.cpu_mode | |
| 15:52:14 | EugenMayer | sean-k-mooney[m] again, thanks! | |
| 15:52:21 | bauzas | (actually, not 'new', since I used them in 2014, but I haven't seen them used for a while now, so I'm all about using them back :D ) | |
| 15:52:34 | sean-k-mooney[m] | EugenMayer: you likely dont need to update the kernel by the way but its worth a shot if that does not help | |
| 15:52:45 | EugenMayer | bauzas you jinxed it, so it might fail! :D | |
| 15:53:55 | bauzas | the more people joining us, the better it will be :) | |
| 15:54:05 | opendevreview | Balazs Gibizer proposed openstack/nova stable/wallaby: Reproduce bug 1945310 https://review.opendev.org/c/openstack/nova/+/811414 | |
| 15:54:14 | bauzas | if I can make jokes to let people chime in, I can try | |
| 15:55:12 | opendevreview | Balazs Gibizer proposed openstack/nova stable/wallaby: Query ports with admin client to get resource_request https://review.opendev.org/c/openstack/nova/+/811416 | |
| 15:55:20 | opendevreview | Balazs Gibizer proposed openstack/nova stable/wallaby: Query ports with admin client to get resource_request https://review.opendev.org/c/openstack/nova/+/811416 | |
| 15:58:18 | EugenMayer | sean-k-mooney[m] massive speedup - 17 seconds | |
| 15:59:49 | sean-k-mooney[m] | cool so basically what happening qemu does not have a profile that matches your cpu. the simplet way to fix that is to update qemu to on that does. failing that host-passthough is a good choice if you dont need to live migrate the vms to different cpu models | |
| 16:00:04 | sean-k-mooney[m] | althernitvie you can pick one that is close and add flags | |
| 16:00:18 | sean-k-mooney[m] | we can tell you how to do that after the meeting but what you have should be fine for now | |
| 16:00:42 | bauzas | 3... | |
| 16:00:47 | bauzas | 2... | |
| 16:00:48 | bauzas | 1... | |
| 16:00:51 | bauzas | #startmeeting nova | |
| 16:00:51 | opendevmeet | Meeting started Tue Nov 2 16:00:51 2021 UTC and is due to finish in 60 minutes. The chair is bauzas. Information about MeetBot at http://wiki.debian.org/MeetBot. | |
| 16:00:51 | opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | |
| 16:00:51 | opendevmeet | The meeting name has been set to 'nova' | |
| 16:01:07 | bauzas | #link https://wiki.openstack.org/wiki/Meetings/Nova#Agenda_for_next_meeting | |
| 16:01:16 | bauzas | good day, everyone | |
| 16:01:19 | dansmith | o/ | |
| 16:01:27 | sean-k-mooney[m] | o/ | |
| 16:01:59 | gibi | \o | |
| 16:02:02 | bauzas | as discussed before, I will exceptionnally be only able to chair this meeting for 30 mins | |
| 16:02:12 | bauzas | so I'll let gibi co-chair | |
| 16:02:15 | bauzas | #chair gibi | |
| 16:02:15 | opendevmeet | Current chairs: bauzas gibi | |
| 16:02:39 | bauzas | let's start while people join | |
| 16:02:46 | bauzas | #topic Bugs (stuck/critical) | |
| 16:02:50 | bauzas | No Critical bug | |
| 16:03:13 | bauzas | #link 20 new untriaged bugs (+0 since the last meeting): #link https://bugs.launchpad.net/nova/+bugs?search=Search&field.status=New | |
| 16:03:31 | bauzas | thanks to anybody who triaged a few of them | |
| 16:04:11 | bauzas | #help any help is appreciated with bug triage and we have a how-to https://wiki.openstack.org/wiki/Nova/BugTriage#Tags | |
| 16:04:35 | bauzas | 32 open stories (+0 since the last meeting) in Storyboard for Placement #link https://storyboard.openstack.org/#!/project/openstack/placement | |
| 16:05:20 | bauzas | so, maybe we closed one or more stories in Storyboard, but I don't think so | |
| 16:06:00 | bauzas | yeah, last story was written on Oct 21st | |
| 16:06:19 | bauzas | any bug to discuss in particular ? | |
| 16:07:00 | bauzas | ok, I guess no, moving on | |
| 16:07:05 | bauzas | #topic Gate status | |
| 16:07:11 | bauzas | Nova gate bugs #link https://bugs.launchpad.net/nova/+bugs?field.tag=gate-failure | |
| 16:07:25 | bauzas | nothing new | |
| 16:07:33 | bauzas | Placement periodic job status #link https://zuul.openstack.org/builds?project=openstack%2Fplacement&pipeline=periodic-weekly | |
| 16:07:45 | bauzas | no issues so far ^ | |
| 16:08:01 | bauzas | just the usual reminder, | |
| 16:08:03 | bauzas | Please look at the gate failures, file a bug, and add an elastic-recheck signature in the opendev/elastic-recheck repo (example: #link https://review.opendev.org/#/c/759967) | |
| 16:08:13 | bauzas | that's it for gate status | |
| 16:08:15 | gibi | this is a gate fix that needs a second core https://review.opendev.org/c/openstack/nova/+/814036 | |
| 16:08:20 | bauzas | oh right | |
| 16:08:44 | bauzas | gibi: I'll look at it while we speak | |
| 16:09:12 | gibi | thanks | |
| 16:09:26 | bauzas | (already looked at it, but need one last glance) | |
| 16:09:56 | bauzas | moving on or any gate failure to mention besides the above one ? | |
| 16:10:54 | bauzas | #topic Release Planning | |
| 16:10:58 | bauzas | Yoga-1 is due Nova 18th #link https://releases.openstack.org/yoga/schedule.html#y-1 | |
| 16:11:10 | bauzas | (3 weeks from now) | |
| 16:11:20 | bauzas | err, 2 weeks | |