| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-05-25 | |||
| 14:46:13 | gibi | bottom line, I'm OK with stephenfin's spec as is, and we can have a separate discussion if and when we drop all the prefixes | |
| 14:46:39 | stephenfin | yes, I do think we should treat this separately. Microversions are (relatively) cheap | |
| 14:47:01 | sean-k-mooney | ok in that case ill upgrade my +1 to +w and refrence this irc log | |
| 14:47:16 | sean-k-mooney | unless gibi gmann you want me to wait? | |
| 14:47:30 | gibi | sean-k-mooney: go for it | |
| 14:47:30 | openstackgerrit | masterpe proposed openstack/nova master: WIP: Fixes missing indexes on column deleted_at at the different tables. https://review.opendev.org/c/openstack/nova/+/792989 | |
| 14:47:40 | gibi | but wait for gmann | |
| 14:47:47 | gibi | as he said he reads the spec | |
| 14:48:03 | gmann | gibi: stephenfin sean-k-mooney item#3 in https://etherpad.opendev.org/p/nova-api-cleanup | |
| 14:48:08 | sean-k-mooney | ok in that case ill leave teh +w to gmann | |
| 14:49:07 | sean-k-mooney | ah yes i tought that had come up before now | |
| 14:49:08 | gibi | gmann: ack. microversion make us safe against breaking clients. | |
| 14:49:21 | gibi | gmann: but I'm fine both ways about the prefixes | |
| 14:49:29 | gibi | gmann: are you OK with the rest of stephenfin's proposal? | |
| 14:50:07 | gmann | gibi: you mean stephenfin spec or proposal of removing all prefix with one microversion ? | |
| 14:50:12 | gibi | spec | |
| 14:50:22 | gibi | the prefix removal an be taken separately from the spec | |
| 14:51:14 | sean-k-mooney | gmann: i think gibi was refering to stephenfin current spec | |
| 14:52:05 | gibi | yepp, this one https://review.opendev.org/c/openstack/nova-specs/+/783827 | |
| 14:52:11 | gibi | I want to get it merged :) | |
| 14:52:54 | gibi | gmann: is it OK to you we we skip the renaming now/ | |
| 14:52:55 | gibi | ? | |
| 14:55:22 | gmann | gibi: stephenfin sean-k-mooney sorry, give me some time, stick in two meeting in parallel | |
| 14:56:03 | sean-k-mooney | gmann: no worries, we have time | |
| 14:56:25 | gibi | gmann: sure, no worries | |
| 14:57:14 | bauzas | sean-k-mooney: can't see your comments on https://review.opendev.org/c/openstack/nova-specs/+/792796/1/specs/xena/approved/generic-mdevs.rst | |
| 14:57:25 | bauzas | replying for the moment to gibi | |
| 14:57:52 | sean-k-mooney | you cant? | |
| 14:58:13 | sean-k-mooney | can you refersh? | |
| 14:58:50 | sean-k-mooney | that is strang i cant see you replies | |
| 14:59:55 | sean-k-mooney | i see i my comments on mulitipel browser even when im logged out so i assume you are looking at a cached copy? | |
| 15:00:12 | gibi | I see sean-k-mooney's comments too | |
| 15:00:40 | sean-k-mooney | if you go to the commit message and go back tothe spec you should see them too | |
| 15:08:04 | bauzas | sean-k-mooney: gibi: oh, now I see those | |
| 15:08:07 | bauzas | weirdo | |
| 15:08:12 | gmann | gibi: stephenfin sean-k-mooney on https://review.opendev.org/c/openstack/nova-specs/+/783827 | |
| 15:09:05 | ozzzo | https://paste.ubuntu.com/p/5vqpFnb4ZZ/ | |
| 15:09:05 | ozzzo | after upgrading compute hosts from centos 7-8, I can't migrate VMs to them. This is the error in the scheduler log: | |
| 15:09:13 | ozzzo | When I look at the hypervisors, I see different cpu_info on the upgraded ones | |
| 15:09:16 | sean-k-mooney | bauzas: i just left 1 or 30 :) most are minor nits/word choice issues though so dont be scared by the number | |
| 15:09:19 | gmann | gibi: stephenfin sean-k-mooney it seems it is related to decide on prefix as we will end up different name for single field in request and response. ``hostname`` in request and ``OS-EXT-SRV-ATTR:hostname`` in response | |
| 15:09:26 | ozzzo | the new ones have "| cpu_info | {"arch": "x86_64", "model": "Cascadelake-Server-noTSX", " | |
| 15:09:26 | ozzzo | the old ones have "| cpu_info | {"vendor": "Intel", "model": "Skylake-Server-IBRS", " | |
| 15:09:31 | ozzzo | is that why VMs won't migrate? How can I change it? | |
| 15:09:56 | gmann | if we want to leave the prefix in response now then OS-EXT-SRV-ATTR:hostname in request make it consistent | |
| 15:10:17 | sean-k-mooney | ozzzo: i assume you are not pinning to a specific cpu_model today | |
| 15:10:24 | ozzzo | I have this in the flavors: | properties | hw:cpu_policy='dedicated', hw:cpu_sockets='1', hw:numa_nodes='1' | | |
| 15:10:45 | sean-k-mooney | ozzzo: if the live migration was rejected it typically means the destination is missign a cpu flag that is present on the source node but not the dest | |
| 15:10:48 | gmann | and removing all prefix together has negative feedback from team users in train PTG so i am not 100% sure if that can happen - http://lists.openstack.org/pipermail/openstack-discuss/2019-May/005824.html | |
| 15:11:00 | sean-k-mooney | ozzzo: the cpu model is set in the nova.conf not the flavor | |
| 15:11:12 | sean-k-mooney | ozzzo: i would gess the issue is TSX | |
| 15:11:31 | ozzzo | I kept the same nova.conf | |
| 15:11:41 | ozzzo | Is it possible that centos 8 recognizes the CPU differently from 7? | |
| 15:12:15 | sean-k-mooney | did it have cascade lake cpus before the upgrade | |
| 15:12:43 | sean-k-mooney | if so then host-model would have selected the clost named model which could have been Skylake-Server-IBRS | |
| 15:12:57 | gibi | gmann: so you see an asymetry between the request and the response due the the prefix in the response. | |
| 15:13:01 | sean-k-mooney | but as part of the upgrade if you cahnge the cpu/libvirt verion then it could now have a new cascadelake model | |
| 15:13:02 | gibi | gmann: that is a good point | |
| 15:13:11 | gmann | gibi: yeah | |
| 15:13:22 | gibi | gmann: but if changing all the prefixes was rejected then can we still change prefixes one by one? | |
| 15:13:28 | ozzzo | sean-k-mooney: I think so, yes, but I don't think I changed anything in the config during the upgrade | |
| 15:13:54 | ozzzo | I just removed computes from the cluster, installed centos 8, and then deployed kolla-ansible from the same playbook | |
| 15:13:57 | sean-k-mooney | ozzzo: can you checnk what you have in the libvirt section | |
| 15:14:15 | gmann | gibi: I think yes, if we improve that API/filed somewhere then it is very valid improvement. let me find example, i think we did it in past | |
| 15:14:19 | sean-k-mooney | ozzzo: oh you are using kolla | |
| 15:14:32 | sean-k-mooney | ozzzo: can you show me your config override for nova | |
| 15:15:48 | ozzzo | this is my /etc/kolla/config/nova.conf: | |
| 15:15:49 | ozzzo | https://paste.ubuntu.com/p/p5rCWRZt7n/ | |
| 15:16:02 | gibi | gmann: OK. then stephenfin would you be against removing the prefix from the hostname in the response to make the req/rsp symmetric? | |
| 15:16:27 | sean-k-mooney | ozzzo: ok then its using the default since cpu_mode and cpu_model is not set which is host-model | |
| 15:16:38 | ozzzo | this is /etc/kolla/config/nova/nova-compute.conf: | |
| 15:16:39 | ozzzo | https://paste.ubuntu.com/p/nTW2s45Xxk/ | |
| 15:16:39 | stephenfin | gibi: no issues with it from my end | |
| 15:16:46 | sean-k-mooney | ozzzo: so the model that is used depend on the qemu version | |
| 15:16:54 | gibi | stephenfin: then lets go that direction. gmann convinced me | |
| 15:17:00 | gibi | sorry for the noise | |
| 15:17:00 | sean-k-mooney | ozzzo: so to fix this you need to pin the cpu model in the nova.conf | |
| 15:17:11 | sean-k-mooney | ozzzo: ill show you and exmple one sec | |
| 15:17:17 | ozzzo | ok ty | |
| 15:18:51 | sean-k-mooney | ozzzo: http://paste.openstack.org/show/805703/ assume all you servers are the same you can do that intially to allow migration to work and then later you can change it | |
| 15:19:22 | ozzzo | sean-k-mooney: I'll try that, ty! | |
| 15:19:25 | sean-k-mooney | ozzzo: there is also a replacemnt for cpu_model called cpu_models which is a preferentially ordered list. | |
| 15:19:45 | sean-k-mooney | so you could do cpu_models=Skylake-Server-IBRS,Cascadelake-Server-noTSX | |
| 15:21:56 | gmann | gibi: stephenfin if it was different attribute then I would have fine in either way but it is same param we are ending in different name | |
| 15:22:23 | sean-k-mooney | ozzzo: http://paste.openstack.org/show/805704/ the cpu model you had enabled before however had tsx enabled | |
| 15:22:43 | sean-k-mooney | ozzzo: so the reason its failing is likely because by default TSX is now disabled in the upstream kerenl | |
| 15:22:57 | ozzzo | righton | |
| 15:22:57 | sean-k-mooney | ozzzo: so you likely will need to re enabled it. | |
| 15:23:06 | sean-k-mooney | with TSX=on | |
| 15:23:16 | ozzzo | you mean, with "cpu_models" ? | |
| 15:23:28 | sean-k-mooney | no on the grub commandline | |
| 15:23:41 | ozzzo | oic ok | |
| 15:23:53 | sean-k-mooney | in which case i would actully use cpu_models=Cascadelake-Server-noTSX,Skylake-Server-IBRS | |
| 15:24:21 | sean-k-mooney | that way new vms will boot without using tsx but you shoudl still be able to migrate your exsting instnaces | |
| 15:24:37 | ozzzo | ok I'll try that | |
| 15:28:33 | gibi | fyi, the ovs -> ovn default switch have been merged (again) | |
| 15:28:35 | gibi | https://review.opendev.org/c/openstack/devstack/+/791436 | |
| 15:49:46 | gibi | nova weekly meeting starts in 10 minutes on #openstack-meeting-3 | |
| 15:49:54 | gibi | yes it is on Tuesday now :) | |
| 15:52:28 | sean-k-mooney | ah yes it is | |