| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-05-25 | |||
| 15:16:02 | gibi | gmann: OK. then stephenfin would you be against removing the prefix from the hostname in the response to make the req/rsp symmetric? | |
| 15:16:27 | sean-k-mooney | ozzzo: ok then its using the default since cpu_mode and cpu_model is not set which is host-model | |
| 15:16:38 | ozzzo | this is /etc/kolla/config/nova/nova-compute.conf: | |
| 15:16:39 | ozzzo | https://paste.ubuntu.com/p/nTW2s45Xxk/ | |
| 15:16:39 | stephenfin | gibi: no issues with it from my end | |
| 15:16:46 | sean-k-mooney | ozzzo: so the model that is used depend on the qemu version | |
| 15:16:54 | gibi | stephenfin: then lets go that direction. gmann convinced me | |
| 15:17:00 | gibi | sorry for the noise | |
| 15:17:00 | sean-k-mooney | ozzzo: so to fix this you need to pin the cpu model in the nova.conf | |
| 15:17:11 | sean-k-mooney | ozzzo: ill show you and exmple one sec | |
| 15:17:17 | ozzzo | ok ty | |
| 15:18:51 | sean-k-mooney | ozzzo: http://paste.openstack.org/show/805703/ assume all you servers are the same you can do that intially to allow migration to work and then later you can change it | |
| 15:19:22 | ozzzo | sean-k-mooney: I'll try that, ty! | |
| 15:19:25 | sean-k-mooney | ozzzo: there is also a replacemnt for cpu_model called cpu_models which is a preferentially ordered list. | |
| 15:19:45 | sean-k-mooney | so you could do cpu_models=Skylake-Server-IBRS,Cascadelake-Server-noTSX | |
| 15:21:56 | gmann | gibi: stephenfin if it was different attribute then I would have fine in either way but it is same param we are ending in different name | |
| 15:22:23 | sean-k-mooney | ozzzo: http://paste.openstack.org/show/805704/ the cpu model you had enabled before however had tsx enabled | |
| 15:22:43 | sean-k-mooney | ozzzo: so the reason its failing is likely because by default TSX is now disabled in the upstream kerenl | |
| 15:22:57 | ozzzo | righton | |
| 15:22:57 | sean-k-mooney | ozzzo: so you likely will need to re enabled it. | |
| 15:23:06 | sean-k-mooney | with TSX=on | |
| 15:23:16 | ozzzo | you mean, with "cpu_models" ? | |
| 15:23:28 | sean-k-mooney | no on the grub commandline | |
| 15:23:41 | ozzzo | oic ok | |
| 15:23:53 | sean-k-mooney | in which case i would actully use cpu_models=Cascadelake-Server-noTSX,Skylake-Server-IBRS | |
| 15:24:21 | sean-k-mooney | that way new vms will boot without using tsx but you shoudl still be able to migrate your exsting instnaces | |
| 15:24:37 | ozzzo | ok I'll try that | |
| 15:28:33 | gibi | fyi, the ovs -> ovn default switch have been merged (again) | |
| 15:28:35 | gibi | https://review.opendev.org/c/openstack/devstack/+/791436 | |
| 15:49:46 | gibi | nova weekly meeting starts in 10 minutes on #openstack-meeting-3 | |
| 15:49:54 | gibi | yes it is on Tuesday now :) | |
| 15:52:28 | sean-k-mooney | ah yes it is | |
| 15:52:43 | sean-k-mooney | ill just open that tab now just incase | |
| 15:56:58 | gibi | :) | |
| 16:26:44 | openstackgerrit | Stephen Finucane proposed openstack/nova master: docs: Address TODO https://review.opendev.org/c/openstack/nova/+/792363 | |
| 16:50:32 | stephenfin | bauzas: I know it's the end of your day, but could you pick up https://review.opendev.org/q/topic:%2522compute_rpc_6.0%2522+status:open again tomorrow? | |
| 16:50:47 | stephenfin | (well past the end of your day, in fact) | |
| 16:50:50 | bauzas | stephenfin: I could | |
| 16:51:08 | bauzas | stephenfin: thanks for working on it, btw. | |
| 16:51:29 | stephenfin | nw | |
| 16:51:53 | sean-k-mooney | dansmith: gibi lyarwood i think i see the isseu with lyarwood gate bug | |
| 16:51:56 | sean-k-mooney | https://zuul.opendev.org/t/openstack/build/bb6fd21b5d8c471a89f4f6598aa84e5d/log/controller/logs/screen-n-cpu.txt#12243 | |
| 16:52:18 | sean-k-mooney | May 24 10:23:19.977224 ubuntu-focal-inap-mtl01-0024794054 nova-compute[107012]: DEBUG ovsdbapp.backend.ovs_idl.vlog [-] [POLLIN] on fd 27 {{(pid=107012) __log_wakeup /usr/lib/python3/dist-packages/ovs/poller.py:263}} | |
| 16:52:20 | sean-k-mooney | May 24 10:23:24.981354 ubuntu-focal-inap-mtl01-0024794054 nova-compute[107012]: DEBUG ovsdbapp.backend.ovs_idl.vlog [-] 4999-ms timeout {{(pid=107012) __log_wakeup /usr/lib/python3/dist-packages/ovs/poller.py:248}} | |
| 16:52:22 | sean-k-mooney | May 24 10:23:24.982747 ubuntu-focal-inap-mtl01-0024794054 nova-compute[107012]: DEBUG ovsdbapp.backend.ovs_idl.vlog [-] tcp:127.0.0.1:6640: idle 5001 ms, sending inactivity probe {{(pid=107012) run /usr/lib/python3/dist-packages/ovs/reconnect.py:103}} | |
| 16:52:43 | sean-k-mooney | the gap in logs lins up to those ovs timeouts | |
| 16:54:15 | sean-k-mooney | so it look like this is possible causeing the compute agent to lock up | |
| 16:54:41 | sean-k-mooney | we dont need to do that polling in os-vif for nova use | |
| 16:55:07 | sean-k-mooney | that functionality in ovsdbapp is there for neutron to monitor the ovs bridge for the addtion and removal of ports | |
| 16:55:19 | sean-k-mooney | so i wonder if we can just turn that off | |
| 16:56:13 | dansmith | ah good catch | |
| 16:56:50 | sean-k-mooney | ill take a look at ovsdbapp and see if we can change that if not maybe i can kick it into a differen thread | |
| 16:58:49 | sean-k-mooney | actully its in a thread https://github.com/openstack/ovsdbapp/blob/master/ovsdbapp/backend/ovs_idl/connection.py#L60-L91 but its not a pthread | |
| 16:58:58 | sean-k-mooney | since we monkey patch in nova | |
| 17:03:56 | openstackgerrit | Stephen Finucane proposed openstack/nova-specs master: Repropose configurable-instance-hostnames https://review.opendev.org/c/openstack/nova-specs/+/783827 | |
| 17:03:57 | openstackgerrit | Stephen Finucane proposed openstack/nova-specs master: Amend configurable-instance-hostnames to include response changes https://review.opendev.org/c/openstack/nova-specs/+/793010 | |
| 17:04:47 | stephenfin | gibi: gmann: bauzas: sean-k-mooney: proposed a follow-up on that spec so we'd have a better history of the changes ^ | |
| 17:04:59 | stephenfin | I think it's okay to send the bulk of it on its way now? | |
| 17:05:02 | stephenfin | i.e. +w | |
| 17:05:14 | stephenfin | I'll rework the patches tomorrow | |
| 17:05:21 | stephenfin | and review some specs, since I only got to one today :( | |
| 17:05:47 | gmann | stephenfin: ack. | |
| 17:06:12 | gmann | stephenfin: dansmith gibi sean-k-mooney spec for project admin to create server on host (what we discussed in PTG) - https://review.opendev.org/c/openstack/nova-specs/+/793011 | |
| 17:07:34 | sean-k-mooney | oh white space errors remind me again that a -3 right :) ill review it shortly it look pretty short and too the point | |
| 17:07:50 | sean-k-mooney | actully im being summoned to go for a walk so when i get back | |
| 17:09:18 | gmann | ok, I was searching whitespace in stephenfin spec :P | |
| 17:10:12 | sean-k-mooney | hehe no in yours https://review.opendev.org/c/openstack/nova-specs/+/793011/1/specs/xena/approved/allow-project-admin-list-hypervisors.rst#39 | |
| 17:10:42 | gmann | yeah :) | |
| 17:11:02 | gmann | stephenfin: sean-k-mooney gibi bauzas I have +W this https://review.opendev.org/c/openstack/nova-specs/+/783827/2 and +2 on follow up 793010 | |
| 17:12:26 | gibi | gmann: ack on both count | |
| 17:23:31 | openstackgerrit | Merged openstack/nova-specs master: Repropose configurable-instance-hostnames https://review.opendev.org/c/openstack/nova-specs/+/783827 | |
| 17:42:34 | ozzzo | sean-k-mooney: I set cpu_models = Cascadelake-Server-noTSX,Skylake-Server-IBRS in nova.conf and also: GRUB_CMDLINE_LINUX="crashkernel=auto tsx=on" in /etc/default/grub, and rebooted, and then redeployed the compute, but I still get "ERROR oslo_service.service nova.exception.InvalidCPUInfo: Configured CPU model: Skylake-Server-IBRS is not compatible with host CPU." in nova-compute.log | |
| 18:09:45 | ozzzo | looks like I might need to use grubby; reading man page | |
| 18:17:55 | ozzzo | sean-k-mooney: that fixed it, ty! | |
| 18:34:41 | ozzzo | sean-k-mooney: that fix it in the sense that I can now start the nova-compute service with the new settings, but VMs still refuse to migrate with the same " Host filter ignoring hosts:" error | |
| 18:59:16 | ozzzo | sean-k-mooney: when I check the hypervisor now, cpu_info has changed from "Cascadelake-Server-noTSX" to "Cascadelake-Server" | |
| 18:59:27 | ozzzo | so it still isn't matching "Skylake-Server-IBRS" | |
| 19:00:15 | ozzzo | Trying "cpu_models=Skylake-Server-IBRS" in "/etc/kolla/config/nova.conf" | |
| 19:07:48 | ozzzo | that didn't make a difference. | |
| 19:08:44 | ozzzo | I looked inside the container, and I see "cpu_models = Skylake-Server-IBRS" in /etc/nova/nova.conf | |
| 19:09:15 | ozzzo | but the hypervisor show still says "Cascadelake-Server" | |
| 19:09:19 | ozzzo | what am I missing? | |
| 19:32:59 | masterpe | I had the same | |
| 19:37:05 | masterpe | at my place the commands says Cascadelake-Server-noTSX. This happend when I upgrade to qemu(1:4.2-3ubuntu6.13) or libvirt(6.0.0-0ubuntu8.8) | |
| 19:38:19 | masterpe | To solve this I restarted the instances and accepted that I can not live migrate those instances. | |
| 19:41:39 | ozzzo | that won't work for me. When I upgrade production, we have to be able to migrate instances there | |
| 19:44:21 | masterpe | ozzzo: I can Imagine, I was able to live migrate from a machine that was no upgraded to a node that is upgraded. | |
| 19:44:57 | ozzzo | masterpe: did you try the grub trick with tsx=on? | |
| 19:45:12 | masterpe | ozzzo: no I did not. | |
| 19:45:56 | masterpe | But that instance further live migraten to a second machine was not possible. | |
| 19:45:56 | ozzzo | I did that, and that allowed me to start nova_compute with "cpu_models=Skylake-Server-IBRS" but it looks like I'm still missing something | |
| 19:48:54 | masterpe | but Skylake- is older then Cascadelake | |
| 19:49:57 | masterpe | say ? | |
| 19:49:57 | masterpe | What does /usr/bin/virsh -r capabilities|grep \<model\>|head -1|cut -d '>' -f2|cut -d '<' -f1 | |
| 19:50:48 | masterpe | and libvirt and qemu version are you and also what processor do you have? | |
| 19:51:00 | ozzzo | my computes don't seem to have virsh installed | |
| 19:52:10 | ozzzo | I'm running kolla | |
| 19:52:14 | ozzzo | everything is in containers | |
| 19:52:44 | ozzzo | Skylake-Server-IBRS is how centos 7 recognized my compute CPUs | |
| 19:53:09 | ozzzo | centos 8 recognizes as "Cascadelake-Server-noTSX" | |