| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-05-25 | |||
| 15:24:37 | ozzzo | ok I'll try that | |
| 15:28:33 | gibi | fyi, the ovs -> ovn default switch have been merged (again) | |
| 15:28:35 | gibi | https://review.opendev.org/c/openstack/devstack/+/791436 | |
| 15:49:46 | gibi | nova weekly meeting starts in 10 minutes on #openstack-meeting-3 | |
| 15:49:54 | gibi | yes it is on Tuesday now :) | |
| 15:52:28 | sean-k-mooney | ah yes it is | |
| 15:52:43 | sean-k-mooney | ill just open that tab now just incase | |
| 15:56:58 | gibi | :) | |
| 16:26:44 | openstackgerrit | Stephen Finucane proposed openstack/nova master: docs: Address TODO https://review.opendev.org/c/openstack/nova/+/792363 | |
| 16:50:32 | stephenfin | bauzas: I know it's the end of your day, but could you pick up https://review.opendev.org/q/topic:%2522compute_rpc_6.0%2522+status:open again tomorrow? | |
| 16:50:47 | stephenfin | (well past the end of your day, in fact) | |
| 16:50:50 | bauzas | stephenfin: I could | |
| 16:51:08 | bauzas | stephenfin: thanks for working on it, btw. | |
| 16:51:29 | stephenfin | nw | |
| 16:51:53 | sean-k-mooney | dansmith: gibi lyarwood i think i see the isseu with lyarwood gate bug | |
| 16:51:56 | sean-k-mooney | https://zuul.opendev.org/t/openstack/build/bb6fd21b5d8c471a89f4f6598aa84e5d/log/controller/logs/screen-n-cpu.txt#12243 | |
| 16:52:18 | sean-k-mooney | May 24 10:23:19.977224 ubuntu-focal-inap-mtl01-0024794054 nova-compute[107012]: DEBUG ovsdbapp.backend.ovs_idl.vlog [-] [POLLIN] on fd 27 {{(pid=107012) __log_wakeup /usr/lib/python3/dist-packages/ovs/poller.py:263}} | |
| 16:52:20 | sean-k-mooney | May 24 10:23:24.981354 ubuntu-focal-inap-mtl01-0024794054 nova-compute[107012]: DEBUG ovsdbapp.backend.ovs_idl.vlog [-] 4999-ms timeout {{(pid=107012) __log_wakeup /usr/lib/python3/dist-packages/ovs/poller.py:248}} | |
| 16:52:22 | sean-k-mooney | May 24 10:23:24.982747 ubuntu-focal-inap-mtl01-0024794054 nova-compute[107012]: DEBUG ovsdbapp.backend.ovs_idl.vlog [-] tcp:127.0.0.1:6640: idle 5001 ms, sending inactivity probe {{(pid=107012) run /usr/lib/python3/dist-packages/ovs/reconnect.py:103}} | |
| 16:52:43 | sean-k-mooney | the gap in logs lins up to those ovs timeouts | |
| 16:54:15 | sean-k-mooney | so it look like this is possible causeing the compute agent to lock up | |
| 16:54:41 | sean-k-mooney | we dont need to do that polling in os-vif for nova use | |
| 16:55:07 | sean-k-mooney | that functionality in ovsdbapp is there for neutron to monitor the ovs bridge for the addtion and removal of ports | |
| 16:55:19 | sean-k-mooney | so i wonder if we can just turn that off | |
| 16:56:13 | dansmith | ah good catch | |
| 16:56:50 | sean-k-mooney | ill take a look at ovsdbapp and see if we can change that if not maybe i can kick it into a differen thread | |
| 16:58:49 | sean-k-mooney | actully its in a thread https://github.com/openstack/ovsdbapp/blob/master/ovsdbapp/backend/ovs_idl/connection.py#L60-L91 but its not a pthread | |
| 16:58:58 | sean-k-mooney | since we monkey patch in nova | |
| 17:03:56 | openstackgerrit | Stephen Finucane proposed openstack/nova-specs master: Repropose configurable-instance-hostnames https://review.opendev.org/c/openstack/nova-specs/+/783827 | |
| 17:03:57 | openstackgerrit | Stephen Finucane proposed openstack/nova-specs master: Amend configurable-instance-hostnames to include response changes https://review.opendev.org/c/openstack/nova-specs/+/793010 | |
| 17:04:47 | stephenfin | gibi: gmann: bauzas: sean-k-mooney: proposed a follow-up on that spec so we'd have a better history of the changes ^ | |
| 17:04:59 | stephenfin | I think it's okay to send the bulk of it on its way now? | |
| 17:05:02 | stephenfin | i.e. +w | |
| 17:05:14 | stephenfin | I'll rework the patches tomorrow | |
| 17:05:21 | stephenfin | and review some specs, since I only got to one today :( | |
| 17:05:47 | gmann | stephenfin: ack. | |
| 17:06:12 | gmann | stephenfin: dansmith gibi sean-k-mooney spec for project admin to create server on host (what we discussed in PTG) - https://review.opendev.org/c/openstack/nova-specs/+/793011 | |
| 17:07:34 | sean-k-mooney | oh white space errors remind me again that a -3 right :) ill review it shortly it look pretty short and too the point | |
| 17:07:50 | sean-k-mooney | actully im being summoned to go for a walk so when i get back | |
| 17:09:18 | gmann | ok, I was searching whitespace in stephenfin spec :P | |
| 17:10:12 | sean-k-mooney | hehe no in yours https://review.opendev.org/c/openstack/nova-specs/+/793011/1/specs/xena/approved/allow-project-admin-list-hypervisors.rst#39 | |
| 17:10:42 | gmann | yeah :) | |
| 17:11:02 | gmann | stephenfin: sean-k-mooney gibi bauzas I have +W this https://review.opendev.org/c/openstack/nova-specs/+/783827/2 and +2 on follow up 793010 | |
| 17:12:26 | gibi | gmann: ack on both count | |
| 17:23:31 | openstackgerrit | Merged openstack/nova-specs master: Repropose configurable-instance-hostnames https://review.opendev.org/c/openstack/nova-specs/+/783827 | |
| 17:42:34 | ozzzo | sean-k-mooney: I set cpu_models = Cascadelake-Server-noTSX,Skylake-Server-IBRS in nova.conf and also: GRUB_CMDLINE_LINUX="crashkernel=auto tsx=on" in /etc/default/grub, and rebooted, and then redeployed the compute, but I still get "ERROR oslo_service.service nova.exception.InvalidCPUInfo: Configured CPU model: Skylake-Server-IBRS is not compatible with host CPU." in nova-compute.log | |
| 18:09:45 | ozzzo | looks like I might need to use grubby; reading man page | |
| 18:17:55 | ozzzo | sean-k-mooney: that fixed it, ty! | |
| 18:34:41 | ozzzo | sean-k-mooney: that fix it in the sense that I can now start the nova-compute service with the new settings, but VMs still refuse to migrate with the same " Host filter ignoring hosts:" error | |
| 18:59:16 | ozzzo | sean-k-mooney: when I check the hypervisor now, cpu_info has changed from "Cascadelake-Server-noTSX" to "Cascadelake-Server" | |
| 18:59:27 | ozzzo | so it still isn't matching "Skylake-Server-IBRS" | |
| 19:00:15 | ozzzo | Trying "cpu_models=Skylake-Server-IBRS" in "/etc/kolla/config/nova.conf" | |
| 19:07:48 | ozzzo | that didn't make a difference. | |
| 19:08:44 | ozzzo | I looked inside the container, and I see "cpu_models = Skylake-Server-IBRS" in /etc/nova/nova.conf | |
| 19:09:15 | ozzzo | but the hypervisor show still says "Cascadelake-Server" | |
| 19:09:19 | ozzzo | what am I missing? | |
| 19:32:59 | masterpe | I had the same | |
| 19:37:05 | masterpe | at my place the commands says Cascadelake-Server-noTSX. This happend when I upgrade to qemu(1:4.2-3ubuntu6.13) or libvirt(6.0.0-0ubuntu8.8) | |
| 19:38:19 | masterpe | To solve this I restarted the instances and accepted that I can not live migrate those instances. | |
| 19:41:39 | ozzzo | that won't work for me. When I upgrade production, we have to be able to migrate instances there | |
| 19:44:21 | masterpe | ozzzo: I can Imagine, I was able to live migrate from a machine that was no upgraded to a node that is upgraded. | |
| 19:44:57 | ozzzo | masterpe: did you try the grub trick with tsx=on? | |
| 19:45:12 | masterpe | ozzzo: no I did not. | |
| 19:45:56 | masterpe | But that instance further live migraten to a second machine was not possible. | |
| 19:45:56 | ozzzo | I did that, and that allowed me to start nova_compute with "cpu_models=Skylake-Server-IBRS" but it looks like I'm still missing something | |
| 19:48:54 | masterpe | but Skylake- is older then Cascadelake | |
| 19:49:57 | masterpe | say ? | |
| 19:49:57 | masterpe | What does /usr/bin/virsh -r capabilities|grep \<model\>|head -1|cut -d '>' -f2|cut -d '<' -f1 | |
| 19:50:48 | masterpe | and libvirt and qemu version are you and also what processor do you have? | |
| 19:51:00 | ozzzo | my computes don't seem to have virsh installed | |
| 19:52:10 | ozzzo | I'm running kolla | |
| 19:52:14 | ozzzo | everything is in containers | |
| 19:52:44 | ozzzo | Skylake-Server-IBRS is how centos 7 recognized my compute CPUs | |
| 19:53:09 | ozzzo | centos 8 recognizes as "Cascadelake-Server-noTSX" | |
| 19:53:33 | ozzzo | and after the grub trick, it recogizes as "Cascadelake-Server" | |
| 19:54:09 | masterpe | I use the "Intel(R) Xeon(R) Gold 6226 CPU @ 2.70GHz" as CPU. | |
| 19:55:09 | ozzzo | my computes have Intel(R) Xeon(R) Platinum 8268 CPU @ 2.90GHz | |
| 19:55:28 | masterpe | thanks | |
| 19:59:02 | masterpe | Is this helpfull? https://bugzilla.redhat.com/show_bug.cgi?id=1678381 ? | |
| 19:59:04 | openstack | bugzilla.redhat.com bug 1678381 in libvirt "[RFE] - libvirt support for cascade lake cpu - Fast Train" [Unspecified,Closed: errata] - Assigned to jdenemar | |
| 20:01:09 | masterpe | and or https://bugzilla.redhat.com/show_bug.cgi?id=1839926 ? | |
| 20:01:29 | ozzzo | masterpe: it looks like the first bug was 8.1 and I'm on 8.3 | |
| 20:01:32 | ozzzo | reading 2nd | |
| 20:25:18 | ozzzo | masterpe: I think the different CPU might be related to the 2nd one, but I think sean's grub trick should have worked around that | |
| 20:25:22 | ozzzo | I must be missing something | |
| 20:27:50 | masterpe | I saw on https://bugzilla.redhat.com/show_bug.cgi?id=1839926#c7 the hle and rtm set to disabled. | |
| 20:27:51 | openstack | bugzilla.redhat.com bug 1839926 in libvirt "5 more -noTSX x86 CPU models are newly introduced" [Unspecified,Closed: errata] - Assigned to jdenemar | |
| 20:36:12 | ozzzo | Do I need to set something in cpu_model_extra_flags? | |
| 20:36:35 | ozzzo | I see it mentioned here: https://docs.openstack.org/nova/train/admin/configuration/hypervisor-kvm.html | |
| 23:23:42 | openstackgerrit | Merged openstack/nova-specs master: Amend configurable-instance-hostnames to include response changes https://review.opendev.org/c/openstack/nova-specs/+/793010 | |
| #openstack-nova - 2021-05-26 | |||
| 08:12:47 | bauzas | good morning Nova | |
| 08:17:48 | lyarwood | Morning | |
| 08:18:41 | lyarwood | so what happens to this channel if I look in a mirror and say `Libera Chat` three times? | |
| 08:19:24 | lyarwood | https://twitter.com/fosdem/status/1397454352835653632 context | |
| 08:19:54 | lyarwood | gibi: stephenfin | |
| 08:20:03 | lyarwood | argh sorry can you hit https://review.opendev.org/c/openstack/nova/+/790660 again please | |
| 08:20:34 | lyarwood | * lyarwood is still getting use to nick auto-complete in the element matrix client | |
| 08:38:05 | gibi | sure' | |
| 09:13:00 | openstackgerrit | Vlad Gusev proposed openstack/nova stable/stein: libvirt: Ignore device already in the process of unplug errors https://review.opendev.org/c/openstack/nova/+/793044 | |
| 10:13:42 | sean-k-mooney[m] | lyarwood: i think i figured out the cause of https://bugs.launchpad.net/nova/+bug/1929446 by the way so hopefully we will be able to fix that relitivly quickly | |