| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-06-13 | |||
| 10:15:06 | gibi | that would be much appreciated | |
| 10:15:29 | sean-k-mooney | im just looking at the nova status check https://review.opendev.org/c/openstack/nova/+/829248 | |
| 10:16:14 | sean-k-mooney | do we want to mention the workaround option for FFU case | |
| 10:23:44 | songwenping__ | sean-k-mooney: do you have the lastest local.conf for both one controller node and two compute nodes deployed by devstack? | |
| 10:24:36 | sean-k-mooney | songwenping: what do you mean by default your local.conf can basically be empty | |
| 10:24:36 | gibi | sean-k-mooney: you mean in the upgrade check descrtiption? | |
| 10:24:57 | sean-k-mooney | songwenping: are you refering to in a ci job | |
| 10:25:17 | sean-k-mooney | gibi: https://review.opendev.org/c/openstack/nova/+/845262/2/nova/cmd/status.py#322 | |
| 10:25:45 | sean-k-mooney | the status check shoudl take account of the workarounds config option to determin if its a warnign or error | |
| 10:26:05 | sean-k-mooney | and the message when it prints shoudl mention the workarounds config option too | |
| 10:26:47 | songwenping | sean-k-mooney: no, i'm plan to deploy develop env with one controller and two computes. | |
| 10:27:22 | sean-k-mooney | for FFU this might fail depending on when you run it | |
| 10:28:00 | sean-k-mooney | songwenping: oh you just want a refernce local.conf to use | |
| 10:28:08 | songwenping | yeah | |
| 10:28:30 | sean-k-mooney | i think i have one ya one sec | |
| 10:28:33 | songwenping | i donnot know the services to enable or disable | |
| 10:28:55 | gibi | sean-k-mooney: replied | |
| 10:33:31 | sean-k-mooney | songwenping: its really only on the compute that you need to overried the default services but ill paste the ones my ansible code generate | |
| 10:35:29 | sean-k-mooney | songwenping: https://paste.opendev.org/show/bBQ5w8E9neWknQ6fF3CO/ | |
| 10:35:58 | sean-k-mooney | songwenping: those have mroe thigns set then you need to set but you can look at the enabled/disabeld services | |
| 10:36:36 | sean-k-mooney | songwenping: the other thing thats important for the comptue nodes is to ensure that | |
| 10:36:39 | sean-k-mooney | RABBIT_HOST="192.168.121.82" | |
| 10:36:40 | sean-k-mooney | RABBIT_PASSWORD="password" | |
| 10:36:42 | sean-k-mooney | RECLONE="True" | |
| 10:36:44 | sean-k-mooney | REMOTE_CEPH="True" | |
| 10:36:46 | sean-k-mooney | SERVICE_HOST="192.168.121.82" | |
| 10:36:48 | sean-k-mooney | SERVICE_PASSWORD="password" | |
| 10:36:53 | sean-k-mooney | the ip or the rabbit host and service host are set to the contoller | |
| 10:37:21 | sean-k-mooney | you proably dont want to have RECLONE=True" set by the way | |
| 10:38:20 | sean-k-mooney | im using these with ansibel automation and i have simple epo caching with rsync so i need to force devstack to checkout the correct branch so i have reclone=True to force that | |
| 10:38:49 | songwenping | sean-k-mooney: many thanks, great useful. | |
| 10:41:57 | sean-k-mooney | songwenping: those are just based on the gate jobs by the way. so if you ever want to recreate what a job does you can look in the logs and find the local.conf it used | |
| 10:43:48 | songwenping | sean-k-mooney: in the stack.sh.log? | |
| 10:45:03 | sean-k-mooney | not its there directly https://zuul.opendev.org/t/openstack/build/6f4ec842455e4231a659c7ac12fc5f7c/log/controller/logs/local_conf.txt | |
| 10:45:59 | sean-k-mooney | if that was a multi node job then you woudl also see a compute/logs/local.conf.txt | |
| 10:46:43 | sean-k-mooney | the local.conf is one of the files we capature and preserve in the log output dir | |
| 10:49:20 | songwenping | ok, thanks. | |
| 10:59:14 | sean-k-mooney | gibi: implying we supprot ffu in any way conviced me we should ignore my previous comment and put it down to a lack of morning coffee :) | |
| 10:59:41 | gibi | sean-k-mooney: FFU is hard | |
| 10:59:49 | gibi | without coffee it is even harder :D | |
| 11:00:53 | gibi | but yeah, if we want to talk about FFU then we need to start with testing it first | |
| 11:01:10 | gibi | as I'm not soo convinced that it can be done :) | |
| 11:01:23 | gibi | I mean can be FFU successfully | |
| 11:01:57 | rmart04 | Hey @sean-k-mooney hope you are well. Wondering if you might be able to provide some advice for some CPU profile pain? | |
| 11:07:03 | rmart04 | not strictly development I know :D | |
| 11:15:18 | rmart04 | I'll post the question just incase it piques your interest :D | |
| 11:15:36 | rmart04 | We're in the process of upgrading to C8Stream (OS Train) and noticed our crippled CPU profile (Skylake-Server-IBRS) we have been using across a couple of processor generations no longer works, nova-compute won't start. We get "invalid CPUinfo profile is not compatible with CPU". I'm guessing a microkernel update has changed the flags on the move from C7->C8 for our CascadeLake servers. Is there an easy way around this these days? I'm guessing | |
| 11:15:36 | rmart04 | if I change the flags I'm going to break future live migrations? :/ | |
| 11:17:33 | sean-k-mooney | rmart04: just back with coffee reading back | |
| 11:17:49 | opendevreview | Balazs Gibizer proposed openstack/nova master: Refactor the nested if-else forest https://review.opendev.org/c/openstack/nova/+/845581 | |
| 11:18:36 | sean-k-mooney | rmart04: this s almost certainly caused by the fact that tsx was disabled/removed by intel in a microcode | |
| 11:19:11 | gibi | Uggla: I made an attempt to transform out the nested if-else forest from the unshelve patch https://review.opendev.org/c/openstack/nova/+/845581 | |
| 11:19:16 | sean-k-mooney | i belive the Skylake-Server-IBRS has tsx enabeld but the cascadelake cpus woudl have it disabeld by the new microcode in c8s | |
| 11:20:50 | sean-k-mooney | rmart04: you are currently using Skylake-Server-IBRS i would guess Skylake-Server-noTSX-IBRS will work | |
| 11:21:53 | rmart04 | I was pointed in that same direction by JGarbutt too, but I couldn't see the TSX flag specified in the old SkyLake profile so thought it might be somthing else. Possibly its still in play even though not specifically specified. | |
| 11:22:04 | sean-k-mooney | vagrant@compute ~]$ diff /usr/share/libvirt/cpu_map/x86_Skylake-Server-IBRS.xml /usr/share/libvirt/cpu_map/x86_Skylake-Server-noTSX-IBRS.xml | |
| 11:22:06 | sean-k-mooney | 2,3c2,3 | |
| 11:22:08 | sean-k-mooney | < <model name='Skylake-Server-IBRS'> | |
| 11:22:10 | sean-k-mooney | < <decode host='on' guest='on'/> | |
| 11:22:12 | sean-k-mooney | --- | |
| 11:22:14 | sean-k-mooney | > <model name='Skylake-Server-noTSX-IBRS'> | |
| 11:22:16 | sean-k-mooney | > <decode host='on' guest='off'/> | |
| 11:22:18 | sean-k-mooney | 33d32 | |
| 11:22:20 | sean-k-mooney | < <feature name='hle'/> | |
| 11:22:22 | sean-k-mooney | 58d56 | |
| 11:22:24 | sean-k-mooney | < <feature name='rtm'/> | |
| 11:22:26 | sean-k-mooney | it sthe hle and rtm flags | |
| 11:22:31 | sean-k-mooney | that are used to provde the tsx functionality | |
| 11:22:54 | rmart04 | ah OK I see, I was looking for TSX specifically. | |
| 11:23:01 | rmart04 | OK that's really useful, thank you. | |
| 11:23:48 | rmart04 | I see there is a tsx=on flag in the kernel which might make this go away quietly but unfortunately that probably won't fly on this deployment. | |
| 11:24:05 | sean-k-mooney | this is basically what nova/libvirt is comptueing https://paste.opendev.org/show/bMJWSQkhwySkS1FlGRSG/ | |
| 11:24:23 | rmart04 | If I change the profile, its going to break lots of LM i guess? | |
| 11:24:35 | sean-k-mooney | rmart04: tsx=on wont work | |
| 11:25:13 | sean-k-mooney | if they try to use the tsx functionality it will fail | |
| 11:25:48 | sean-k-mooney | rmart04: the only way to change the cpu modle is via a hard reboot or cold migration | |
| 11:26:01 | sean-k-mooney | so to resolve this you will need guest downtime | |
| 11:26:10 | sean-k-mooney | there is no way to avoid that im affriad | |
| 11:28:38 | rmart04 | eek. OK well thanks for this very useful info | |
| 11:52:28 | jhartkopf | Hey, I am currently working on the implementation for the approved spec for updating user data (https://review.opendev.org/c/openstack/nova-specs/+/816542). Part of the spec is to regenerate the config drive on hard reboots (reboot implementations are driver-specific). My approach would be to do this on every hard reboot as there seems to be no trivial way to check whether user data has been actually changed. | |
| 11:52:45 | jhartkopf | Would this be something that every driver needs to support? Any ideas/opinions on this? | |
| 11:55:13 | sean-k-mooney | jhartkopf: no this should be done only when requested | |
| 11:55:52 | sean-k-mooney | jhartkopf: the spec i tought said the hard-reboot api woudl be extended with a new paramter for this | |
| 11:57:57 | sean-k-mooney | jhartkopf: hum https://review.opendev.org/c/openstack/nova-specs/+/816542/7/specs/zed/approved/update-userdata.rst#96 might actully be insfficent | |
| 11:58:19 | sean-k-mooney | so that provide a way to update the metadta on hard reboot by providing the data again | |
| 11:59:08 | sean-k-mooney | but we likely shoudl have a seccodn boolean parmater like regenerate_configdrive=True|false | |
| 12:00:10 | sean-k-mooney | gibi: ^ what do you think? | |
| 12:00:27 | sean-k-mooney | im not sure we want the perfomance hit of always regenerating the config drive on hard reboot | |
| 12:00:59 | sean-k-mooney | jhartkopf: it should be implmented for all drivers that support config drive ideally but if you started with just libvirt that proably would cover most usecases | |
| 12:02:03 | gibi | so the user_data parameter on reboot will be optional, isn't it? | |
| 12:02:09 | sean-k-mooney | jhartkopf: actully the other way to do this is to store a flag in the instance_system_metadata | |
| 12:02:22 | sean-k-mooney | gibi: yes but the edge case here is | |
| 12:02:40 | sean-k-mooney | how to regenerate teh config drive if i have use server update | |
| 12:02:51 | sean-k-mooney | to modify the user data on the next reboot | |
| 12:03:28 | sean-k-mooney | jhartkopf: gibi so what we could do is add a flag to the instance_system_metadata to track that the config drive is dirty | |
| 12:03:36 | sean-k-mooney | and then check that on reboot and clear it | |
| 12:03:59 | sean-k-mooney | if its dirty regenerate the config drive with the new data if not use the exisitng one | |
| 12:04:25 | gibi | ahh I see | |
| 12:04:49 | sean-k-mooney | the user data on hard reboot is for when you want to fully replace it on reboot | |
| 12:05:02 | gibi | I'm fine with a flag in system meta | |