| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-02-08 | |||
| 17:03:39 | gibi | the check queue getting long in zuul (>2h to get an executor). It almost feels like FF week :) | |
| 17:10:17 | gmann | gibi: thanks | |
| 17:10:34 | gmann | yeah, gate is in bad situation | |
| 17:36:35 | opendevreview | Dan Smith proposed openstack/nova master: Add docs for stable-compute-uuid behaviors https://review.opendev.org/c/openstack/nova/+/872977 | |
| 18:18:57 | gibi | bauzas: you might not like it but I reviewed the power managemenet series | |
| 18:19:08 | gibi | bauzas: I'm close to -2 on the tar.gz part ;) | |
| 18:19:32 | sean-k-mooney | gibi: that is proably resolveable | |
| 18:19:45 | bauzas | gibi: thanks for the fish | |
| 18:19:53 | sean-k-mooney | we started with a large copy of the syfs filesystem but the funcitonl test can be writeen a differnt way | |
| 18:19:53 | bauzas | gibi: do you have time for discussing about it ? | |
| 18:20:02 | gibi | bauzas: I have a bit of time yes | |
| 18:20:18 | sean-k-mooney | the current approch was taken when thsi started as arbiterd | |
| 18:20:30 | sean-k-mooney | it had a much larger scope | |
| 18:20:33 | gibi | my main concern with the tar.gz is that I cannot review it or diff it in a normal way | |
| 18:20:40 | gibi | so it is hidden information | |
| 18:20:41 | bauzas | gibi: I do understand your point | |
| 18:20:42 | sean-k-mooney | for the limite scope in nova i htink we can create a fixture another way | |
| 18:20:57 | gibi | I would do couple of helper function that generates the fs on the fly | |
| 18:21:09 | bauzas | gibi: the problem is, as said by sean-k-mooney, is that the large number of files was creating a problem | |
| 18:21:11 | sean-k-mooney | gibi: ya that was the alternitive | |
| 18:21:23 | sean-k-mooney | bauzas: we dont need a large number of files | |
| 18:21:29 | bauzas | ok, I can try to generate those arbitrary files then | |
| 18:21:29 | gibi | bauzas: I assume we have two type of files | |
| 18:21:33 | sean-k-mooney | we actully only need a small subset | |
| 18:21:39 | bauzas | we only need a subset of the sysfs | |
| 18:21:47 | bauzas | yep, ok | |
| 18:21:48 | gibi | one for the governor and one for the online | |
| 18:21:49 | bauzas | sounds a plan then | |
| 18:21:57 | sean-k-mooney | gibi: ya more or less | |
| 18:22:01 | bauzas | yup | |
| 18:22:05 | sean-k-mooney | and then the parent directory structure | |
| 18:22:05 | gibi | huhh, this went better than I expected :) | |
| 18:22:07 | bauzas | and we can augment the fixture if needed | |
| 18:22:28 | sean-k-mooney | but thats just a call to "mkdir -p" | |
| 18:22:29 | bauzas | for other usages, I mean | |
| 18:22:38 | gibi | sean-k-mooney: +1 | |
| 18:22:41 | bauzas | yeah, I can do it quick | |
| 18:22:41 | sean-k-mooney | within the tempfs/tempdir | |
| 18:23:00 | gibi | I found one edge case that might be intersting | |
| 18:23:22 | sean-k-mooney | i didnt do that for arbiterd partly becasue it was a quick hack to get fucntional test and i expecet to do much more eventully | |
| 18:23:30 | gibi | if you do reconfiguration from cpu_state to governor strategy or back then the patch might not do the right thing now | |
| 18:23:49 | sean-k-mooney | oh that is intersting | |
| 18:24:03 | gibi | i.e. we set the governor but we never online a cpu that was off before | |
| 18:24:12 | sean-k-mooney | we coudl disallow that. but that should be resolveable | |
| 18:24:22 | gibi | yeah I think this is resolveable | |
| 18:24:32 | gibi | an fairly easily testable I hope | |
| 18:24:45 | bauzas | gibi: ah, good edge case | |
| 18:24:49 | sean-k-mooney | so the issue is when we start we cant know what it used to be | |
| 18:24:52 | gibi | just need a bit more logic on power_up and down | |
| 18:24:56 | bauzas | what would be your preference ? | |
| 18:25:12 | sean-k-mooney | i actully think i know how to detect it | |
| 18:25:24 | bauzas | to check the core status and online if a governor helper is called ? | |
| 18:25:30 | sean-k-mooney | when its set to govoner the libvirt chack that checks the cores should be online should catch this | |
| 18:25:32 | gibi | bauzas: yeah | |
| 18:25:47 | bauzas | ok, I can augment power_down and up | |
| 18:26:06 | bauzas | gibi: sean-k-mooney: fwiw, I'll be traveling tomorrow back-and-forth to Paris | |
| 18:26:13 | bauzas | so most of my day will be trains | |
| 18:26:19 | bauzas | but I'll be online | |
| 18:26:24 | bauzas | and on work status | |
| 18:26:39 | gibi | maybe the other direction is trickier. so you have a governor startegy and you set the cpu to low. Then you reconfigure to cpu_state. And you boot a VM that need the cpu so you online it, but it still has the low governor set | |
| 18:26:50 | sean-k-mooney | bauzas: you chould change https://review.opendev.org/c/openstack/nova/+/821228/5/nova/virt/libvirt/host.py#755 | |
| 18:27:08 | sean-k-mooney | we shoudl check if cpu power management is enabled and its set to cpu_state | |
| 18:27:30 | sean-k-mooney | if its set to govoner then we can detect that cores are offlien and online them | |
| 18:27:42 | sean-k-mooney | or raise an error and stop the compute | |
| 18:27:58 | bauzas | damn operators who like to play with config options | |
| 18:28:01 | sean-k-mooney | this is not really something you should change when there are instance on the host | |
| 18:28:42 | sean-k-mooney | gibi: the other direction wee should not really do anything (state->govoner) | |
| 18:28:45 | bauzas | sean-k-mooney: true, I wonder whether it would be rather preferable to detect it at startup and fail | |
| 18:29:04 | sean-k-mooney | sorry (govoner->state) | |
| 18:29:07 | gibi | I'm OK to reject the reconfiguration with instance on the host, BUT we power down CPUs at init_host without an instance | |
| 18:29:19 | sean-k-mooney | because you are allowed to manage the govoner today outside of nova | |
| 18:29:25 | gibi | so you don't need an instance to get a discrepancy | |
| 18:29:26 | sean-k-mooney | so you may have change the state | |
| 18:29:40 | bauzas | I think we're entering danger zone | |
| 18:29:53 | sean-k-mooney | with that we could say if you have ONF.libvirt.cpu_power_management=true | |
| 18:30:00 | bauzas | here, we're asking operators to think about what they gonna do | |
| 18:30:08 | sean-k-mooney | then all govenros in the cpu_dedicated_set must be the same | |
| 18:30:24 | sean-k-mooney | we could detect that in that case and catch the (govoner->state) change | |
| 18:30:40 | bauzas | I mean, if I'm an operator that already sets governor levels, then I should NOT let nova play with my cores | |
| 18:31:13 | sean-k-mooney | so (govoner->state) we can detect that there is a govoner differnce on the cpus, (govoner<-state) we can detect the offline cores | |
| 18:31:28 | bauzas | sean-k-mooney: I don't want to add logic while I can add documentation about supported features | |
| 18:31:56 | sean-k-mooney | i agree on documenting that you cant cahnge this with instances on the host | |
| 18:32:05 | sean-k-mooney | and shoudl reboot if you do change it or something like that | |
| 18:32:18 | sean-k-mooney | but we can detect and error or fix it fi we want too | |
| 18:32:45 | sean-k-mooney | if we have CONF.libvirt.cpu_power_management=true i think that shoudl mean you cannot manage the cpus outside of nova | |
| 18:32:49 | gibi | again, becase we apply the config to the cpus at init_host regardless of any instance on the host you don't need an instance on the host to cause problems | |
| 18:33:11 | sean-k-mooney | gibi: we dont your corect | |
| 18:33:46 | sean-k-mooney | but we can detect it and document that you should not do this without a host reboot | |
| 18:33:53 | bauzas | I'm confused | |
| 18:34:02 | gibi | sean-k-mooney: I agree to document | |
| 18:34:07 | gibi | sean-k-mooney: and detect if possible | |
| 18:34:35 | sean-k-mooney | detect is possibel but i think bauzas would prefer to not require that for the feature to merge and maybe to it later | |
| 18:35:06 | sean-k-mooney | (govoner->state) the concer is we leave some cpus in low perfomacne mode | |
| 18:35:13 | sean-k-mooney | (state-) | |
| 18:35:26 | sean-k-mooney | (state->govoner) the concern is we leave some cores offline | |
| 18:35:44 | sean-k-mooney | we can detect boot and make it a hard error from init_host | |
| 18:36:03 | sean-k-mooney | in either case a host reboot will resolve it or they operator can fix it | |
| 18:37:35 | gibi | yepp | |
| 18:38:47 | sean-k-mooney | bauzas: does ^ make sense | |
| 18:39:06 | sean-k-mooney | the offline core check is triival its just https://review.opendev.org/c/openstack/nova/+/821228/5/nova/virt/libvirt/host.py#755 | |