| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-05-25 | |||
| 14:57:14 | bauzas | sean-k-mooney: can't see your comments on https://review.opendev.org/c/openstack/nova-specs/+/792796/1/specs/xena/approved/generic-mdevs.rst | |
| 14:57:25 | bauzas | replying for the moment to gibi | |
| 14:57:52 | sean-k-mooney | you cant? | |
| 14:58:13 | sean-k-mooney | can you refersh? | |
| 14:58:50 | sean-k-mooney | that is strang i cant see you replies | |
| 14:59:55 | sean-k-mooney | i see i my comments on mulitipel browser even when im logged out so i assume you are looking at a cached copy? | |
| 15:00:12 | gibi | I see sean-k-mooney's comments too | |
| 15:00:40 | sean-k-mooney | if you go to the commit message and go back tothe spec you should see them too | |
| 15:08:04 | bauzas | sean-k-mooney: gibi: oh, now I see those | |
| 15:08:07 | bauzas | weirdo | |
| 15:08:12 | gmann | gibi: stephenfin sean-k-mooney on https://review.opendev.org/c/openstack/nova-specs/+/783827 | |
| 15:09:05 | ozzzo | after upgrading compute hosts from centos 7-8, I can't migrate VMs to them. This is the error in the scheduler log: | |
| 15:09:05 | ozzzo | https://paste.ubuntu.com/p/5vqpFnb4ZZ/ | |
| 15:09:13 | ozzzo | When I look at the hypervisors, I see different cpu_info on the upgraded ones | |
| 15:09:16 | sean-k-mooney | bauzas: i just left 1 or 30 :) most are minor nits/word choice issues though so dont be scared by the number | |
| 15:09:19 | gmann | gibi: stephenfin sean-k-mooney it seems it is related to decide on prefix as we will end up different name for single field in request and response. ``hostname`` in request and ``OS-EXT-SRV-ATTR:hostname`` in response | |
| 15:09:26 | ozzzo | the old ones have "| cpu_info | {"vendor": "Intel", "model": "Skylake-Server-IBRS", " | |
| 15:09:26 | ozzzo | the new ones have "| cpu_info | {"arch": "x86_64", "model": "Cascadelake-Server-noTSX", " | |
| 15:09:31 | ozzzo | is that why VMs won't migrate? How can I change it? | |
| 15:09:56 | gmann | if we want to leave the prefix in response now then OS-EXT-SRV-ATTR:hostname in request make it consistent | |
| 15:10:17 | sean-k-mooney | ozzzo: i assume you are not pinning to a specific cpu_model today | |
| 15:10:24 | ozzzo | I have this in the flavors: | properties | hw:cpu_policy='dedicated', hw:cpu_sockets='1', hw:numa_nodes='1' | | |
| 15:10:45 | sean-k-mooney | ozzzo: if the live migration was rejected it typically means the destination is missign a cpu flag that is present on the source node but not the dest | |
| 15:10:48 | gmann | and removing all prefix together has negative feedback from team users in train PTG so i am not 100% sure if that can happen - http://lists.openstack.org/pipermail/openstack-discuss/2019-May/005824.html | |
| 15:11:00 | sean-k-mooney | ozzzo: the cpu model is set in the nova.conf not the flavor | |
| 15:11:12 | sean-k-mooney | ozzzo: i would gess the issue is TSX | |
| 15:11:31 | ozzzo | I kept the same nova.conf | |
| 15:11:41 | ozzzo | Is it possible that centos 8 recognizes the CPU differently from 7? | |
| 15:12:15 | sean-k-mooney | did it have cascade lake cpus before the upgrade | |
| 15:12:43 | sean-k-mooney | if so then host-model would have selected the clost named model which could have been Skylake-Server-IBRS | |
| 15:12:57 | gibi | gmann: so you see an asymetry between the request and the response due the the prefix in the response. | |
| 15:13:01 | sean-k-mooney | but as part of the upgrade if you cahnge the cpu/libvirt verion then it could now have a new cascadelake model | |
| 15:13:02 | gibi | gmann: that is a good point | |
| 15:13:11 | gmann | gibi: yeah | |
| 15:13:22 | gibi | gmann: but if changing all the prefixes was rejected then can we still change prefixes one by one? | |
| 15:13:28 | ozzzo | sean-k-mooney: I think so, yes, but I don't think I changed anything in the config during the upgrade | |
| 15:13:54 | ozzzo | I just removed computes from the cluster, installed centos 8, and then deployed kolla-ansible from the same playbook | |
| 15:13:57 | sean-k-mooney | ozzzo: can you checnk what you have in the libvirt section | |
| 15:14:15 | gmann | gibi: I think yes, if we improve that API/filed somewhere then it is very valid improvement. let me find example, i think we did it in past | |
| 15:14:19 | sean-k-mooney | ozzzo: oh you are using kolla | |
| 15:14:32 | sean-k-mooney | ozzzo: can you show me your config override for nova | |
| 15:15:48 | ozzzo | this is my /etc/kolla/config/nova.conf: | |
| 15:15:49 | ozzzo | https://paste.ubuntu.com/p/p5rCWRZt7n/ | |
| 15:16:02 | gibi | gmann: OK. then stephenfin would you be against removing the prefix from the hostname in the response to make the req/rsp symmetric? | |
| 15:16:27 | sean-k-mooney | ozzzo: ok then its using the default since cpu_mode and cpu_model is not set which is host-model | |
| 15:16:38 | ozzzo | this is /etc/kolla/config/nova/nova-compute.conf: | |
| 15:16:39 | stephenfin | gibi: no issues with it from my end | |
| 15:16:39 | ozzzo | https://paste.ubuntu.com/p/nTW2s45Xxk/ | |
| 15:16:46 | sean-k-mooney | ozzzo: so the model that is used depend on the qemu version | |
| 15:16:54 | gibi | stephenfin: then lets go that direction. gmann convinced me | |
| 15:17:00 | sean-k-mooney | ozzzo: so to fix this you need to pin the cpu model in the nova.conf | |
| 15:17:00 | gibi | sorry for the noise | |
| 15:17:11 | sean-k-mooney | ozzzo: ill show you and exmple one sec | |
| 15:17:17 | ozzzo | ok ty | |
| 15:18:51 | sean-k-mooney | ozzzo: http://paste.openstack.org/show/805703/ assume all you servers are the same you can do that intially to allow migration to work and then later you can change it | |
| 15:19:22 | ozzzo | sean-k-mooney: I'll try that, ty! | |
| 15:19:25 | sean-k-mooney | ozzzo: there is also a replacemnt for cpu_model called cpu_models which is a preferentially ordered list. | |
| 15:19:45 | sean-k-mooney | so you could do cpu_models=Skylake-Server-IBRS,Cascadelake-Server-noTSX | |
| 15:21:56 | gmann | gibi: stephenfin if it was different attribute then I would have fine in either way but it is same param we are ending in different name | |
| 15:22:23 | sean-k-mooney | ozzzo: http://paste.openstack.org/show/805704/ the cpu model you had enabled before however had tsx enabled | |
| 15:22:43 | sean-k-mooney | ozzzo: so the reason its failing is likely because by default TSX is now disabled in the upstream kerenl | |
| 15:22:57 | sean-k-mooney | ozzzo: so you likely will need to re enabled it. | |
| 15:22:57 | ozzzo | righton | |
| 15:23:06 | sean-k-mooney | with TSX=on | |
| 15:23:16 | ozzzo | you mean, with "cpu_models" ? | |
| 15:23:28 | sean-k-mooney | no on the grub commandline | |
| 15:23:41 | ozzzo | oic ok | |
| 15:23:53 | sean-k-mooney | in which case i would actully use cpu_models=Cascadelake-Server-noTSX,Skylake-Server-IBRS | |
| 15:24:21 | sean-k-mooney | that way new vms will boot without using tsx but you shoudl still be able to migrate your exsting instnaces | |
| 15:24:37 | ozzzo | ok I'll try that | |
| 15:28:33 | gibi | fyi, the ovs -> ovn default switch have been merged (again) | |
| 15:28:35 | gibi | https://review.opendev.org/c/openstack/devstack/+/791436 | |
| 15:49:46 | gibi | nova weekly meeting starts in 10 minutes on #openstack-meeting-3 | |
| 15:49:54 | gibi | yes it is on Tuesday now :) | |
| 15:52:28 | sean-k-mooney | ah yes it is | |
| 15:52:43 | sean-k-mooney | ill just open that tab now just incase | |
| 15:56:58 | gibi | :) | |
| 16:26:44 | openstackgerrit | Stephen Finucane proposed openstack/nova master: docs: Address TODO https://review.opendev.org/c/openstack/nova/+/792363 | |
| 16:50:32 | stephenfin | bauzas: I know it's the end of your day, but could you pick up https://review.opendev.org/q/topic:%2522compute_rpc_6.0%2522+status:open again tomorrow? | |
| 16:50:47 | stephenfin | (well past the end of your day, in fact) | |
| 16:50:50 | bauzas | stephenfin: I could | |
| 16:51:08 | bauzas | stephenfin: thanks for working on it, btw. | |
| 16:51:29 | stephenfin | nw | |
| 16:51:53 | sean-k-mooney | dansmith: gibi lyarwood i think i see the isseu with lyarwood gate bug | |
| 16:51:56 | sean-k-mooney | https://zuul.opendev.org/t/openstack/build/bb6fd21b5d8c471a89f4f6598aa84e5d/log/controller/logs/screen-n-cpu.txt#12243 | |
| 16:52:18 | sean-k-mooney | May 24 10:23:19.977224 ubuntu-focal-inap-mtl01-0024794054 nova-compute[107012]: DEBUG ovsdbapp.backend.ovs_idl.vlog [-] [POLLIN] on fd 27 {{(pid=107012) __log_wakeup /usr/lib/python3/dist-packages/ovs/poller.py:263}} | |
| 16:52:20 | sean-k-mooney | May 24 10:23:24.981354 ubuntu-focal-inap-mtl01-0024794054 nova-compute[107012]: DEBUG ovsdbapp.backend.ovs_idl.vlog [-] 4999-ms timeout {{(pid=107012) __log_wakeup /usr/lib/python3/dist-packages/ovs/poller.py:248}} | |
| 16:52:22 | sean-k-mooney | May 24 10:23:24.982747 ubuntu-focal-inap-mtl01-0024794054 nova-compute[107012]: DEBUG ovsdbapp.backend.ovs_idl.vlog [-] tcp:127.0.0.1:6640: idle 5001 ms, sending inactivity probe {{(pid=107012) run /usr/lib/python3/dist-packages/ovs/reconnect.py:103}} | |
| 16:52:43 | sean-k-mooney | the gap in logs lins up to those ovs timeouts | |
| 16:54:15 | sean-k-mooney | so it look like this is possible causeing the compute agent to lock up | |
| 16:54:41 | sean-k-mooney | we dont need to do that polling in os-vif for nova use | |
| 16:55:07 | sean-k-mooney | that functionality in ovsdbapp is there for neutron to monitor the ovs bridge for the addtion and removal of ports | |
| 16:55:19 | sean-k-mooney | so i wonder if we can just turn that off | |
| 16:56:13 | dansmith | ah good catch | |
| 16:56:50 | sean-k-mooney | ill take a look at ovsdbapp and see if we can change that if not maybe i can kick it into a differen thread | |
| 16:58:49 | sean-k-mooney | actully its in a thread https://github.com/openstack/ovsdbapp/blob/master/ovsdbapp/backend/ovs_idl/connection.py#L60-L91 but its not a pthread | |
| 16:58:58 | sean-k-mooney | since we monkey patch in nova | |
| 17:03:56 | openstackgerrit | Stephen Finucane proposed openstack/nova-specs master: Repropose configurable-instance-hostnames https://review.opendev.org/c/openstack/nova-specs/+/783827 | |
| 17:03:57 | openstackgerrit | Stephen Finucane proposed openstack/nova-specs master: Amend configurable-instance-hostnames to include response changes https://review.opendev.org/c/openstack/nova-specs/+/793010 | |
| 17:04:47 | stephenfin | gibi: gmann: bauzas: sean-k-mooney: proposed a follow-up on that spec so we'd have a better history of the changes ^ | |