| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-06-13 | |||
| 11:15:36 | rmart04 | We're in the process of upgrading to C8Stream (OS Train) and noticed our crippled CPU profile (Skylake-Server-IBRS) we have been using across a couple of processor generations no longer works, nova-compute won't start. We get "invalid CPUinfo profile is not compatible with CPU". I'm guessing a microkernel update has changed the flags on the move from C7->C8 for our CascadeLake servers. Is there an easy way around this these days? I'm guessing | |
| 11:17:33 | sean-k-mooney | rmart04: just back with coffee reading back | |
| 11:17:49 | opendevreview | Balazs Gibizer proposed openstack/nova master: Refactor the nested if-else forest https://review.opendev.org/c/openstack/nova/+/845581 | |
| 11:18:36 | sean-k-mooney | rmart04: this s almost certainly caused by the fact that tsx was disabled/removed by intel in a microcode | |
| 11:19:11 | gibi | Uggla: I made an attempt to transform out the nested if-else forest from the unshelve patch https://review.opendev.org/c/openstack/nova/+/845581 | |
| 11:19:16 | sean-k-mooney | i belive the Skylake-Server-IBRS has tsx enabeld but the cascadelake cpus woudl have it disabeld by the new microcode in c8s | |
| 11:20:50 | sean-k-mooney | rmart04: you are currently using Skylake-Server-IBRS i would guess Skylake-Server-noTSX-IBRS will work | |
| 11:21:53 | rmart04 | I was pointed in that same direction by JGarbutt too, but I couldn't see the TSX flag specified in the old SkyLake profile so thought it might be somthing else. Possibly its still in play even though not specifically specified. | |
| 11:22:04 | sean-k-mooney | vagrant@compute ~]$ diff /usr/share/libvirt/cpu_map/x86_Skylake-Server-IBRS.xml /usr/share/libvirt/cpu_map/x86_Skylake-Server-noTSX-IBRS.xml | |
| 11:22:06 | sean-k-mooney | 2,3c2,3 | |
| 11:22:08 | sean-k-mooney | < <model name='Skylake-Server-IBRS'> | |
| 11:22:10 | sean-k-mooney | < <decode host='on' guest='on'/> | |
| 11:22:12 | sean-k-mooney | --- | |
| 11:22:14 | sean-k-mooney | > <model name='Skylake-Server-noTSX-IBRS'> | |
| 11:22:16 | sean-k-mooney | > <decode host='on' guest='off'/> | |
| 11:22:18 | sean-k-mooney | 33d32 | |
| 11:22:20 | sean-k-mooney | < <feature name='hle'/> | |
| 11:22:22 | sean-k-mooney | 58d56 | |
| 11:22:24 | sean-k-mooney | < <feature name='rtm'/> | |
| 11:22:26 | sean-k-mooney | it sthe hle and rtm flags | |
| 11:22:31 | sean-k-mooney | that are used to provde the tsx functionality | |
| 11:22:54 | rmart04 | ah OK I see, I was looking for TSX specifically. | |
| 11:23:01 | rmart04 | OK that's really useful, thank you. | |
| 11:23:48 | rmart04 | I see there is a tsx=on flag in the kernel which might make this go away quietly but unfortunately that probably won't fly on this deployment. | |
| 11:24:05 | sean-k-mooney | this is basically what nova/libvirt is comptueing https://paste.opendev.org/show/bMJWSQkhwySkS1FlGRSG/ | |
| 11:24:23 | rmart04 | If I change the profile, its going to break lots of LM i guess? | |
| 11:24:35 | sean-k-mooney | rmart04: tsx=on wont work | |
| 11:25:13 | sean-k-mooney | if they try to use the tsx functionality it will fail | |
| 11:25:48 | sean-k-mooney | rmart04: the only way to change the cpu modle is via a hard reboot or cold migration | |
| 11:26:01 | sean-k-mooney | so to resolve this you will need guest downtime | |
| 11:26:10 | sean-k-mooney | there is no way to avoid that im affriad | |
| 11:28:38 | rmart04 | eek. OK well thanks for this very useful info | |
| 11:52:28 | jhartkopf | Hey, I am currently working on the implementation for the approved spec for updating user data (https://review.opendev.org/c/openstack/nova-specs/+/816542). Part of the spec is to regenerate the config drive on hard reboots (reboot implementations are driver-specific). My approach would be to do this on every hard reboot as there seems to be no trivial way to check whether user data has been actually changed. | |
| 11:52:45 | jhartkopf | Would this be something that every driver needs to support? Any ideas/opinions on this? | |
| 11:55:13 | sean-k-mooney | jhartkopf: no this should be done only when requested | |
| 11:55:52 | sean-k-mooney | jhartkopf: the spec i tought said the hard-reboot api woudl be extended with a new paramter for this | |
| 11:57:57 | sean-k-mooney | jhartkopf: hum https://review.opendev.org/c/openstack/nova-specs/+/816542/7/specs/zed/approved/update-userdata.rst#96 might actully be insfficent | |
| 11:58:19 | sean-k-mooney | so that provide a way to update the metadta on hard reboot by providing the data again | |
| 11:59:08 | sean-k-mooney | but we likely shoudl have a seccodn boolean parmater like regenerate_configdrive=True|false | |
| 12:00:10 | sean-k-mooney | gibi: ^ what do you think? | |
| 12:00:27 | sean-k-mooney | im not sure we want the perfomance hit of always regenerating the config drive on hard reboot | |
| 12:00:59 | sean-k-mooney | jhartkopf: it should be implmented for all drivers that support config drive ideally but if you started with just libvirt that proably would cover most usecases | |
| 12:02:03 | gibi | so the user_data parameter on reboot will be optional, isn't it? | |
| 12:02:09 | sean-k-mooney | jhartkopf: actully the other way to do this is to store a flag in the instance_system_metadata | |
| 12:02:22 | sean-k-mooney | gibi: yes but the edge case here is | |
| 12:02:40 | sean-k-mooney | how to regenerate teh config drive if i have use server update | |
| 12:02:51 | sean-k-mooney | to modify the user data on the next reboot | |
| 12:03:28 | sean-k-mooney | jhartkopf: gibi so what we could do is add a flag to the instance_system_metadata to track that the config drive is dirty | |
| 12:03:36 | sean-k-mooney | and then check that on reboot and clear it | |
| 12:03:59 | sean-k-mooney | if its dirty regenerate the config drive with the new data if not use the exisitng one | |
| 12:04:25 | gibi | ahh I see | |
| 12:04:49 | sean-k-mooney | the user data on hard reboot is for when you want to fully replace it on reboot | |
| 12:05:02 | sean-k-mooney | which is how i woudl expect this to be used most offten honestly | |
| 12:05:02 | gibi | I'm fine with a flag in system meta | |
| 12:05:17 | jhartkopf | sean-k-mooney: Yes, so with an additional variable, we could track if the config drive should be regenerated or not | |
| 12:06:00 | sean-k-mooney | jhartkopf: yep the instance_system_metadata table is internal to nova and jsut a key value pair with the instance as a primary key | |
| 12:06:31 | sean-k-mooney | jhartkopf: so you can jsut add a flag to that on server update if you update the user-data field | |
| 12:07:17 | sean-k-mooney | then reboot can check it and clear it. does ^ work for you | |
| 12:07:37 | sean-k-mooney | if so we can leave the spec as it is. | |
| 12:10:22 | jhartkopf | Seems to be the part that was missing, thanks! I will then try to implement this for libvirt initially. | |
| 12:15:39 | sean-k-mooney | jhartkopf: just an fyi the instance system metadata is a feild on the instance object https://github.com/openstack/nova/blob/master/nova/objects/instance.py#L194= | |
| 12:16:14 | sean-k-mooney | so you will have direct access to that in the driver and you will be able to update it on save as part of the reboot flow | |
| 12:17:24 | sean-k-mooney | just set the key config_drive_dirty flag to false and it will be save when we next call save i.e. when we update the task/vm state | |
| 12:18:51 | sean-k-mooney | jhartkopf: if you want you can expose this directly as a property on the instance object to keep it clean so that the virt driver dont need to check the system metadata directly. | |
| 12:19:06 | sean-k-mooney | just have the property update the system metadata | |
| 12:33:24 | jhartkopf | sean-k-mooney: good point, I will look into this | |
| 13:11:24 | bauzas | sean-k-mooney: gibi: fwiw, I'm currently reviewing the hard https://review.opendev.org/c/openstack/nova/+/829248/9/nova/compute/manager.py | |
| 13:11:46 | bauzas | (that and doing some expense report...) | |
| 13:12:22 | sean-k-mooney | i dont think i have used concour sicne we switch to it | |
| 13:12:42 | sean-k-mooney | but if its like the deployment of concour we had at intel its much nicer | |
| 13:12:54 | sean-k-mooney | then the oracle suite we used to use for expenses | |
| 13:12:56 | gibi | expense was easy but that single line in the manager seems easier :D | |
| 13:12:57 | bauzas | concur is good for preparing your report | |
| 13:13:25 | bauzas | but there are some bugs in the concur android app | |
| 13:13:36 | bauzas | I can't modify some expenses | |
| 13:13:49 | sean-k-mooney | ah ok i have only really used the web app | |
| 13:13:59 | sean-k-mooney | i dont like haveing compay stuff on my personal phone | |
| 13:14:35 | sean-k-mooney | so i jsut take pictures fo the recipts and email them to my company adress and fill it out after the fact | |
| 13:15:38 | gibi | bauzas: yepp I had issues with the app but I was able to fix it with the web based too | |
| 13:15:41 | gibi | l | |
| 13:16:27 | gibi | I guess you hit the issue with setting expense type to ~food and the app asking for some additional fields that was not editable | |
| 13:17:38 | bauzas | gibi: yup | |
| 13:18:15 | bauzas | gibi: yeah, I provided all my receipts to concur thanks to the android app when waiting for my flight :) | |
| 13:18:31 | bauzas | gibi: but eventually I got some problem with a meal and for my hotel expense | |
| 13:19:03 | sean-k-mooney | bauzas: not to add to your paper work but are you plannign to send a writeup or operator feedback to the list or similar | |
| 13:19:06 | bauzas | but eventually, today I used the webapp and I was able to modify the expenses correctly | |
| 13:19:43 | bauzas | sean-k-mooney: we'll be discussing this tomorrow during the meeting and after that, I'll provide a feedback to the list | |
| 13:19:49 | sean-k-mooney | ack | |
| 13:19:59 | bauzas | sean-k-mooney: but most of the operators told they don't want to have a lot of emails from us | |
| 13:20:11 | sean-k-mooney | while you were away there was a request to do a release of stable yoga and xena by the way | |
| 13:20:17 | sean-k-mooney | i was going to bring that up in the meeting | |
| 13:20:23 | bauzas | sean-k-mooney: oh ok | |
| 13:20:31 | sean-k-mooney | i have not looked at whats pending but we have not done a release of yoga yet | |
| 13:20:52 | sean-k-mooney | so there proably are some valid bugfixes that would be good to release | |
| 13:22:28 | sean-k-mooney | bauzas: i asked mainly for people outside of redhat i was going to ask you about your feedback in our downstream meeting but if were planing to send somethign to the list anyway i was happy enough to just read that | |
| 13:23:17 | bauzas | sean-k-mooney: yeah I understand it about the feedback | |
| 13:23:41 | bauzas | like I said, I'd first discuss with our folks tomorrow before replying | |
| 13:24:05 | sean-k-mooney | no worries. did you enjoy the conference | |
| 13:24:17 | bauzas | sean-k-mooney: we had a problem with the etherpad, it became chinese but eventually thanks to fungi it was fixed | |
| 13:24:37 | sean-k-mooney | ah did we forget to add teh disclaminer at the top | |