Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2021-05-25
15:13:11 gmann gibi: yeah
15:13:22 gibi gmann: but if changing all the prefixes was rejected then can we still change prefixes one by one?
15:13:28 ozzzo sean-k-mooney: I think so, yes, but I don't think I changed anything in the config during the upgrade
15:13:54 ozzzo I just removed computes from the cluster, installed centos 8, and then deployed kolla-ansible from the same playbook
15:13:57 sean-k-mooney ozzzo: can you checnk what you have in the libvirt section
15:14:15 gmann gibi: I think yes, if we improve that API/filed somewhere then it is very valid improvement. let me find example, i think we did it in past
15:14:19 sean-k-mooney ozzzo: oh you are using kolla
15:14:32 sean-k-mooney ozzzo: can you show me your config override for nova
15:15:48 ozzzo this is my /etc/kolla/config/nova.conf:
15:15:49 ozzzo https://paste.ubuntu.com/p/p5rCWRZt7n/
15:16:02 gibi gmann: OK. then stephenfin would you be against removing the prefix from the hostname in the response to make the req/rsp symmetric?
15:16:27 sean-k-mooney ozzzo: ok then its using the default since cpu_mode and cpu_model is not set which is host-model
15:16:38 ozzzo this is /etc/kolla/config/nova/nova-compute.conf:
15:16:39 stephenfin gibi: no issues with it from my end
15:16:39 ozzzo https://paste.ubuntu.com/p/nTW2s45Xxk/
15:16:46 sean-k-mooney ozzzo: so the model that is used depend on the qemu version
15:16:54 gibi stephenfin: then lets go that direction. gmann convinced me
15:17:00 sean-k-mooney ozzzo: so to fix this you need to pin the cpu model in the nova.conf
15:17:00 gibi sorry for the noise
15:17:11 sean-k-mooney ozzzo: ill show you and exmple one sec
15:17:17 ozzzo ok ty
15:18:51 sean-k-mooney ozzzo: http://paste.openstack.org/show/805703/ assume all you servers are the same you can do that intially to allow migration to work and then later you can change it
15:19:22 ozzzo sean-k-mooney: I'll try that, ty!
15:19:25 sean-k-mooney ozzzo: there is also a replacemnt for cpu_model called cpu_models which is a preferentially ordered list.
15:19:45 sean-k-mooney so you could do cpu_models=Skylake-Server-IBRS,Cascadelake-Server-noTSX
15:21:56 gmann gibi: stephenfin if it was different attribute then I would have fine in either way but it is same param we are ending in different name
15:22:23 sean-k-mooney ozzzo: http://paste.openstack.org/show/805704/ the cpu model you had enabled before however had tsx enabled
15:22:43 sean-k-mooney ozzzo: so the reason its failing is likely because by default TSX is now disabled in the upstream kerenl
15:22:57 sean-k-mooney ozzzo: so you likely will need to re enabled it.
15:22:57 ozzzo righton
15:23:06 sean-k-mooney with TSX=on
15:23:16 ozzzo you mean, with "cpu_models" ?
15:23:28 sean-k-mooney no on the grub commandline
15:23:41 ozzzo oic ok
15:23:53 sean-k-mooney in which case i would actully use cpu_models=Cascadelake-Server-noTSX,Skylake-Server-IBRS
15:24:21 sean-k-mooney that way new vms will boot without using tsx but you shoudl still be able to migrate your exsting instnaces
15:24:37 ozzzo ok I'll try that
15:28:33 gibi fyi, the ovs -> ovn default switch have been merged (again)
15:28:35 gibi https://review.opendev.org/c/openstack/devstack/+/791436
15:49:46 gibi nova weekly meeting starts in 10 minutes on #openstack-meeting-3
15:49:54 gibi yes it is on Tuesday now :)
15:52:28 sean-k-mooney ah yes it is
15:52:43 sean-k-mooney ill just open that tab now just incase
15:56:58 gibi :)
16:26:44 openstackgerrit Stephen Finucane proposed openstack/nova master: docs: Address TODO https://review.opendev.org/c/openstack/nova/+/792363
16:50:32 stephenfin bauzas: I know it's the end of your day, but could you pick up https://review.opendev.org/q/topic:%2522compute_rpc_6.0%2522+status:open again tomorrow?
16:50:47 stephenfin (well past the end of your day, in fact)
16:50:50 bauzas stephenfin: I could
16:51:08 bauzas stephenfin: thanks for working on it, btw.
16:51:29 stephenfin nw
16:51:53 sean-k-mooney dansmith: gibi lyarwood i think i see the isseu with lyarwood gate bug
16:51:56 sean-k-mooney https://zuul.opendev.org/t/openstack/build/bb6fd21b5d8c471a89f4f6598aa84e5d/log/controller/logs/screen-n-cpu.txt#12243
16:52:18 sean-k-mooney May 24 10:23:19.977224 ubuntu-focal-inap-mtl01-0024794054 nova-compute[107012]: DEBUG ovsdbapp.backend.ovs_idl.vlog [-] [POLLIN] on fd 27 {{(pid=107012) __log_wakeup /usr/lib/python3/dist-packages/ovs/poller.py:263}}
16:52:20 sean-k-mooney May 24 10:23:24.981354 ubuntu-focal-inap-mtl01-0024794054 nova-compute[107012]: DEBUG ovsdbapp.backend.ovs_idl.vlog [-] 4999-ms timeout {{(pid=107012) __log_wakeup /usr/lib/python3/dist-packages/ovs/poller.py:248}}
16:52:22 sean-k-mooney May 24 10:23:24.982747 ubuntu-focal-inap-mtl01-0024794054 nova-compute[107012]: DEBUG ovsdbapp.backend.ovs_idl.vlog [-] tcp:127.0.0.1:6640: idle 5001 ms, sending inactivity probe {{(pid=107012) run /usr/lib/python3/dist-packages/ovs/reconnect.py:103}}
16:52:43 sean-k-mooney the gap in logs lins up to those ovs timeouts
16:54:15 sean-k-mooney so it look like this is possible causeing the compute agent to lock up
16:54:41 sean-k-mooney we dont need to do that polling in os-vif for nova use
16:55:07 sean-k-mooney that functionality in ovsdbapp is there for neutron to monitor the ovs bridge for the addtion and removal of ports
16:55:19 sean-k-mooney so i wonder if we can just turn that off
16:56:13 dansmith ah good catch
16:56:50 sean-k-mooney ill take a look at ovsdbapp and see if we can change that if not maybe i can kick it into a differen thread
16:58:49 sean-k-mooney actully its in a thread https://github.com/openstack/ovsdbapp/blob/master/ovsdbapp/backend/ovs_idl/connection.py#L60-L91 but its not a pthread
16:58:58 sean-k-mooney since we monkey patch in nova
17:03:56 openstackgerrit Stephen Finucane proposed openstack/nova-specs master: Repropose configurable-instance-hostnames https://review.opendev.org/c/openstack/nova-specs/+/783827
17:03:57 openstackgerrit Stephen Finucane proposed openstack/nova-specs master: Amend configurable-instance-hostnames to include response changes https://review.opendev.org/c/openstack/nova-specs/+/793010
17:04:47 stephenfin gibi: gmann: bauzas: sean-k-mooney: proposed a follow-up on that spec so we'd have a better history of the changes ^
17:04:59 stephenfin I think it's okay to send the bulk of it on its way now?
17:05:02 stephenfin i.e. +w
17:05:14 stephenfin I'll rework the patches tomorrow
17:05:21 stephenfin and review some specs, since I only got to one today :(
17:05:47 gmann stephenfin: ack.
17:06:12 gmann stephenfin: dansmith gibi sean-k-mooney spec for project admin to create server on host (what we discussed in PTG) - https://review.opendev.org/c/openstack/nova-specs/+/793011
17:07:34 sean-k-mooney oh white space errors remind me again that a -3 right :) ill review it shortly it look pretty short and too the point
17:07:50 sean-k-mooney actully im being summoned to go for a walk so when i get back
17:09:18 gmann ok, I was searching whitespace in stephenfin spec :P
17:10:12 sean-k-mooney hehe no in yours https://review.opendev.org/c/openstack/nova-specs/+/793011/1/specs/xena/approved/allow-project-admin-list-hypervisors.rst#39
17:10:42 gmann yeah :)
17:11:02 gmann stephenfin: sean-k-mooney gibi bauzas I have +W this https://review.opendev.org/c/openstack/nova-specs/+/783827/2 and +2 on follow up 793010
17:12:26 gibi gmann: ack on both count
17:23:31 openstackgerrit Merged openstack/nova-specs master: Repropose configurable-instance-hostnames https://review.opendev.org/c/openstack/nova-specs/+/783827
17:42:34 ozzzo sean-k-mooney: I set cpu_models = Cascadelake-Server-noTSX,Skylake-Server-IBRS in nova.conf and also: GRUB_CMDLINE_LINUX="crashkernel=auto tsx=on" in /etc/default/grub, and rebooted, and then redeployed the compute, but I still get "ERROR oslo_service.service nova.exception.InvalidCPUInfo: Configured CPU model: Skylake-Server-IBRS is not compatible with host CPU." in nova-compute.log
18:09:45 ozzzo looks like I might need to use grubby; reading man page
18:17:55 ozzzo sean-k-mooney: that fixed it, ty!
18:34:41 ozzzo sean-k-mooney: that fix it in the sense that I can now start the nova-compute service with the new settings, but VMs still refuse to migrate with the same " Host filter ignoring hosts:" error
18:59:16 ozzzo sean-k-mooney: when I check the hypervisor now, cpu_info has changed from "Cascadelake-Server-noTSX" to "Cascadelake-Server"
18:59:27 ozzzo so it still isn't matching "Skylake-Server-IBRS"
19:00:15 ozzzo Trying "cpu_models=Skylake-Server-IBRS" in "/etc/kolla/config/nova.conf"
19:07:48 ozzzo that didn't make a difference.
19:08:44 ozzzo I looked inside the container, and I see "cpu_models = Skylake-Server-IBRS" in /etc/nova/nova.conf
19:09:15 ozzzo but the hypervisor show still says "Cascadelake-Server"
19:09:19 ozzzo what am I missing?
19:32:59 masterpe I had the same
19:37:05 masterpe at my place the commands says Cascadelake-Server-noTSX. This happend when I upgrade to qemu(1:4.2-3ubuntu6.13) or libvirt(6.0.0-0ubuntu8.8)
19:38:19 masterpe To solve this I restarted the instances and accepted that I can not live migrate those instances.
19:41:39 ozzzo that won't work for me. When I upgrade production, we have to be able to migrate instances there
19:44:21 masterpe ozzzo: I can Imagine, I was able to live migrate from a machine that was no upgraded to a node that is upgraded.
19:44:57 ozzzo masterpe: did you try the grub trick with tsx=on?
19:45:12 masterpe ozzzo: no I did not.
19:45:56 ozzzo I did that, and that allowed me to start nova_compute with "cpu_models=Skylake-Server-IBRS" but it looks like I'm still missing something

Earlier   Later