| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-12-08 | |||
| 11:16:26 | bauzas | Red Hat Enterprise Linux release 9.2 (Plow) | |
| 11:16:28 | bauzas | gibi ^ | |
| 11:17:09 | bauzas | I don't want Stream f... up with cpufreq but they broke ity | |
| 11:17:30 | bauzas | don't know* what* | |
| 11:18:01 | gibi | CentOS Stream release 9 | |
| 11:18:01 | gibi | [root@osp-dev-01 ~]# cat /etc/redhat-release | |
| 11:18:18 | gibi | [root@osp-dev-01 ~]# ll /sys/devices/system/cpu/cpu1/cpufreq/scaling_governor | |
| 11:18:21 | gibi | ls: cannot access '/sys/devices/system/cpu/cpu1/cpufreq/scaling_governor': No such file or directory | |
| 11:18:24 | gibi | [root@osp-dev-01 ~]# ll /sys/devices/system/cpu/cpu1/cpufreq/ | |
| 11:18:26 | gibi | ls: cannot access '/sys/devices/system/cpu/cpu1/cpufreq/': No such file or directory | |
| 11:19:45 | bauzas | that's crazy | |
| 11:20:00 | bauzas | I +2d your change anyway, but I'm very surprised | |
| 11:20:02 | gibi | but I checked onlineing / offlineing works | |
| 11:20:37 | bauzas | cpufreq is one very common kernel module | |
| 11:28:13 | opendevreview | Alex Welsh proposed openstack/nova master: Make map_cell0 command update existing mappings https://review.opendev.org/c/openstack/nova/+/903140 | |
| 11:28:56 | bauzas | gibi: out of curiosity, could you please issue on your c9s box : | |
| 11:29:01 | bauzas | find /lib/modules/$(uname -r)/kernel/ | grep cpufreq | |
| 11:46:56 | gibi | [root@osp-dev-01 ~]# find /lib/modules/$(uname -r)/kernel/ | grep cpufreq | |
| 11:46:59 | gibi | /lib/modules/5.14.0-386.el9.x86_64/kernel/drivers/cpufreq | |
| 11:47:02 | gibi | /lib/modules/5.14.0-386.el9.x86_64/kernel/drivers/cpufreq/acpi-cpufreq.ko.xz | |
| 11:47:05 | gibi | /lib/modules/5.14.0-386.el9.x86_64/kernel/drivers/cpufreq/amd-pstate-ut.ko.xz | |
| 11:47:08 | gibi | /lib/modules/5.14.0-386.el9.x86_64/kernel/drivers/cpufreq/amd_freq_sensitivity.ko.xz | |
| 11:47:11 | gibi | /lib/modules/5.14.0-386.el9.x86_64/kernel/drivers/cpufreq/p4-clockmod.ko.xz | |
| 11:47:14 | gibi | /lib/modules/5.14.0-386.el9.x86_64/kernel/drivers/cpufreq/powernow-k8.ko.xz | |
| 11:47:17 | gibi | /lib/modules/5.14.0-386.el9.x86_64/kernel/drivers/cpufreq/speedstep-lib.ko.xz | |
| 11:52:25 | bauzas | gibi: thanks so cpufreq is shipped | |
| 11:52:38 | bauzas | gibi: lsmod | grep cpufreq ? | |
| 11:52:57 | gibi | that is empty | |
| 11:53:10 | bauzas | I'll ask cfontain if he knows why cpufreq isn't there in c9s | |
| 11:53:17 | gibi | but it is also empty in my desktop where I have cpufreq | |
| 11:53:21 | bauzas | maybe because of tuned | |
| 11:54:47 | admin1 | hi all .. how to get a vm out of " Confirm or Revert Resize/Migrate " stage where both confirm or revert fails to work .. these vms are backed by ceph and came to this state where their hypervisor died and were in shutdown state | |
| 11:55:23 | bauzas | gibi: oh that's a boot kernel config | |
| 11:56:26 | bauzas | gibi: cat /boot/config-$(uname -r) | grep CONFIG_CPU_FREQ | |
| 11:57:28 | gibi | https://paste.opendev.org/show/bknmxAOZRtxwb0pNBaSH/ | |
| 12:01:25 | bauzas | I'm then out of ideas why the sysfs directory isn't popping up | |
| 12:01:47 | bauzas | anyway, the patch is good to me, it's good to be conservative here | |
| 12:02:34 | bauzas | I'll talk to Christophe in case (which is more than possible) he knows the reason :) | |
| 12:03:40 | bauzas | oh damn, he's on PTO :D | |
| 12:03:55 | gibi | bauzas: OK. thanks for looking into it | |
| 12:04:07 | bauzas | np, thanks for testing it :) | |
| 12:04:35 | bauzas | I could have been more cautious | |
| 12:04:56 | bauzas | and not assume every single box (from RPi to large mainframes) includes cpufreq | |
| 12:05:12 | gibi | it was a surprise for me too | |
| 12:05:38 | gibi | first I thought it is due to using a VM as compute. But then I checked centos on a real hw and I see the same | |
| 12:06:02 | bauzas | maybe the journal said the reason | |
| 12:06:12 | bauzas | maybe it was deactivate | |
| 12:06:17 | bauzas | deactivated* | |
| 12:06:19 | bauzas | anyway | |
| 12:06:22 | gibi | nothing in journalctl | grep cpufreq | |
| 12:06:44 | gibi | journalctl | grep cpufreq | |
| 12:06:46 | gibi | sorry | |
| 12:07:19 | gibi | btw, offlining cpus when a compute is a VM works (but only offlines the cpu from the VM perspective obviously) | |
| 12:07:39 | bauzas | makes sense | |
| 12:07:50 | gibi | so at some point we can have some end to end test for this in openstack-k8s-operators CI upstrea | |
| 12:08:15 | gibi | currently I only have https://github.com/openstack-k8s-operators/nova-operator/pull/597#issuecomment-1846870113 as we don't have dedicated CPUs in CI yet | |
| 12:11:10 | bauzas | https://docs.kernel.org/core-api/cpu_hotplug.html | |
| 12:11:20 | bauzas | that's what the offline does | |
| 12:12:01 | bauzas | I meant https://docs.kernel.org/core-api/cpu_hotplug.html#the-offline-case | |
| 12:15:43 | gibi | yeah I saw in dmesg that IRQ was migrated away | |
| 12:31:23 | opendevreview | Sylvain Bauza proposed openstack/nova-specs master: Proposes mdev live-migration support in libvirt https://review.opendev.org/c/openstack/nova-specs/+/900636 | |
| 12:38:59 | opendevreview | Sylvain Bauza proposed openstack/nova-specs master: Proposes mdev live-migration support in libvirt https://review.opendev.org/c/openstack/nova-specs/+/900636 | |
| 12:46:39 | sean-k-mooney[m] | gibi: bauzas it might not be there if hardware pstates is enabled i guess or if its didsbaled in the bios | |
| 12:47:50 | sean-k-mooney[m] | gibi: people have a habit of turning off speedstepp ot do a test and not renabling it when they are done | |
| 12:49:04 | sean-k-mooney[m] | c9s does have cpufreq avaible but perhapas not in the beaker image | |
| 13:07:02 | sean-k-mooney[m] | gibi: bauzas so im off today and about to head out but if ye feel like reivewing some simple patches it would be nice if my codespell seriese could get some eyes https://review.opendev.org/c/openstack/nova/+/897092 | |
| 13:11:07 | gibi | sean-k-mooney[m]: I approved up until https://review.opendev.org/c/openstack/nova/+/897096 in that melwitt has a question | |
| 13:15:29 | sean-k-mooney[m] | gibi: replied with a link to the docs https://github.com/codespell-project/codespell#ignoring-words | |
| 13:15:44 | sean-k-mooney[m] | tldr is the ignore list must be lowercase | |
| 13:17:25 | gibi | ack, thanks | |
| 13:20:34 | sean-k-mooney[m] | https://opendev.org/openstack/nova/src/commit/3fdc97ca5f8e65546840cd453e86cb08cb4869e6/doc/source/admin/cells.rst?display=source#L602 this is what it was detecting | |
| 13:23:43 | gibi | I'm +2 all over it :) | |
| 13:37:10 | bauzas | sean-k-mooney: yeah I was thinking about some BIOS change | |
| 13:37:26 | bauzas | pstates shouldn't be related to the cpu states IIRC | |
| 13:37:40 | bauzas | c-states as well | |
| 14:34:34 | zigo | Hi. We're currently testing our upgrades, and I'm being very confused by the "nova-manage db online_data_migrations". First question: under what release will we need the online_data_migrations (we're doing victoria->wallaby->xena->yoga->zed for this run...)? Second question: I'm always being displayed "Total Needed: 50" but nothing in "Completed", am I doing something wrong?!? | |
| 14:36:33 | zigo | 995 rows matched query fill_virtual_interface_list, 0 migrated <--- something wrong ? | |
| 14:36:33 | zigo | # nova-manage db online_data_migrations --max-count 1000 | |
| 14:47:37 | bauzas | zigo: do you already know nova-status upgrade check ? | |
| 14:47:39 | bauzas | https://docs.openstack.org/nova/latest/cli/nova-status.html#upgrade | |
| 14:49:00 | bauzas | zigo: online db migrations are sometimes needed indeed, before upgrading | |
| 14:49:07 | zigo | I do know nova-status upgrade check yes. | |
| 14:49:55 | bauzas | so it will doublecheck whether you need to call online data migrations | |
| 14:50:35 | zigo | So, if it returns zero, I'm good, right? | |
| 14:50:48 | opendevreview | Arnaud Morin proposed openstack/nova master: Return proper number of pci devices for virtio https://review.opendev.org/c/openstack/nova/+/903176 | |
| 14:50:57 | zigo | In such case, victoria to zed don't need any online_data_migrations... | |
| 14:52:27 | zigo | Though this still feels weird... | |
| 14:53:40 | zigo | I can restart it many times, it always says "Total Needed 5" and "Completed 0", which doesn't feel right ... | |
| 14:53:40 | zigo | bauzas: https://paste.opendev.org/show/bmkrJ3MF5bYFOZvbnUa0/ | |
| 14:53:53 | zigo | How come it's not completing any? | |
| 14:55:45 | bauzas | zigo: I unfortunately need to get my daughter from the school in a sec, so could we discuss that on monday ? | |
| 14:55:56 | zigo | No worries, sure ! :) | |
| 14:55:59 | zigo | Thanks. | |
| 14:58:04 | bauzas | https://docs.openstack.org/nova/latest/cli/nova-manage.html#db-online-data-migrations | |
| 14:58:10 | bauzas | it tells what you need to update | |
| 14:59:32 | bauzas | and for fill_interface_virtual_list, this is https://github.com/openstack/nova/blob/master/nova/cmd/manage.py#L186C17-L186C17 | |
| 15:00:00 | bauzas | zigo: that's what it's calling https://github.com/openstack/nova/blob/master/nova/objects/virtual_interface.py#L156 | |
| 16:07:10 | admin1 | i have a vm with status active power_state nostate on a hypervisor that failed .. vm is backed by ceph .. how do i actually migrate this to other host ? | |
| 16:07:22 | admin1 | due to the power state, normal migration fails | |
| 16:08:03 | admin1 | and when done evauate host, a few other vms are in confirm resize/migration phase, where they dont work with confirm or resize and just stay like that | |