| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-01-04 | |||
| 11:33:14 | sean-k-mooney | so now i dont have access to any of my dev envionments... | |
| 11:33:57 | sean-k-mooney | also our gpu passtough works but nvida chave changed how they do the vm detachtion so the kvm hidden feature we supprot nolonger works for windows guests | |
| 11:35:16 | sean-k-mooney | this came up on irc a a few weeks ago acctully but i can now confirm that our current code is not sufficent, if nova users upgrade there gpu driver it there gueste will eventually stop working which is lovely | |
| 11:49:36 | lyarwood | sean-k-mooney:s~. | |
| 11:49:52 | lyarwood | urgh sorry | |
| 11:50:06 | lyarwood | stephenfin: sorry was afk, I'll take a look at that now | |
| 11:53:43 | gibi | another easy spec update needs anther core eyes https://review.opendev.org/c/openstack/nova-specs/+/768803 | |
| 12:16:19 | stephenfin | gibi: done | |
| 12:16:38 | gibi | thanks | |
| 12:29:09 | openstackgerrit | Merged openstack/nova-specs master: [Trivial] Clarify the deprecated apis in *Proposed change* https://review.opendev.org/c/openstack/nova-specs/+/768803 | |
| 12:41:46 | bauzas | gibi: stephenfin: sure, will look at https://review.opendev.org/c/openstack/nova-specs/+/765797/ | |
| 12:47:57 | gibi | bauzas: thanks | |
| 12:48:06 | bauzas | done, +Wd | |
| 12:48:26 | gibi | stephenfin: you are good to go for a respin of your hypervisor patch | |
| 12:49:06 | gibi | meanwhile I looked at the vncconsole password patch (another microversion bump) but that is not ready yet, I still have serious comments there (not published yet) | |
| 12:49:22 | gibi | so right now the hypervisor patch feels the closest for the 2.88 microversion | |
| 12:49:30 | gibi | I will look at the tenant_id patch series too | |
| 12:54:11 | openstackgerrit | Merged openstack/nova-specs master: Update modernize-os-hypervisors-api spec, pt. 3 https://review.opendev.org/c/openstack/nova-specs/+/765797 | |
| 13:45:00 | stephenfin | gibi: Yup, I'll do it now | |
| 13:45:06 | gibi | stephenfin: cool | |
| 14:12:55 | zoharm | hi, regarding proposed nvme healing agent feature: https://review.opendev.org/c/openstack/cinder-specs/+/766732 | |
| 14:13:12 | zoharm | we would like to ask, what is the right way to spawn it? we had it proposed to use python-daemon python package for maintaining an independent process. Is it ok to add that dependency? | |
| 14:13:43 | zoharm | if using python-daemon is not ok, geguileo suggested previously to look at privsep daemon and see how it is spawned. which way in there is better? https://opendev.org/openstack/oslo.privsep/src/branch/master/oslo_privsep/daemon.py line 296 or 336 - we would like to have only one agent process per host. any pointers / details would be useful here. Thank you! | |
| 14:26:55 | kashyap | stephenfin: sean-k-mooney: Remind me again, Nova supports CPU pinnning to host core and to host NUMA node, both, yes? | |
| 14:27:32 | stephenfin | yes | |
| 14:27:32 | sean-k-mooney | kashyap: kindo of it depend on what you mean by host numa nodes | |
| 14:27:32 | gibi | zoharm: the nova services (daemons) using oslo_services to spawn the daemon process https://docs.openstack.org/oslo.service/latest/ | |
| 14:27:52 | stephenfin | hw:cpu_policy=dedicated to pin to individual host cores | |
| 14:27:59 | sean-k-mooney | we support soft pinning so a vm will float over the cores of a singel numa node if that is what you mean | |
| 14:28:26 | sean-k-mooney | and we obvioulsy support direct 1:1 gust cpu to host cpu pinning too | |
| 14:28:39 | kashyap | I'm just asked if "CPU Pinning to host core" is supported or not | |
| 14:28:52 | sean-k-mooney | hw:numa_nodes=1 to pin the vm to just one host numa node | |
| 14:29:07 | sean-k-mooney | kashyap: you cant choose the core but yes | |
| 14:29:15 | kashyap | Okay; thx | |
| 14:29:35 | zoharm | gibi thank you, we want to do this from os-brick, is that ok? | |
| 14:29:43 | zoharm | also, any chance for a pointer at an example of how to use it? | |
| 14:29:51 | sean-k-mooney | zoharm: not really | |
| 14:30:12 | sean-k-mooney | so os-brick is shared between nova and cinder | |
| 14:30:20 | sean-k-mooney | both run as deams already and invoke the lib | |
| 14:30:39 | kashyap | stephenfin: sean-k-mooney: One more, the above of what I asked is supported from Queens, is it? | |
| 14:30:42 | sean-k-mooney | you proably should not be sapawning a second deamon out of the os-brick lib | |
| 14:30:51 | zoharm | right, but we need an entry point for the agent/daemon as per the spec, and the connector is the place decided on | |
| 14:31:03 | zoharm | do you have an alternative? | |
| 14:31:35 | bnemec | zoharm: If you want to know more about how privsep works, you can read about it in https://github.com/openstack/oslo.privsep/blob/master/oslo_privsep/daemon.py#L15 | |
| 14:31:40 | sean-k-mooney | zoharm: a better apprcoh would be to have a seperate console script entry point that is run seperatly as a deamon and no tspawned out of eithet the nova-compute process or the cinder-volume process | |
| 14:32:12 | zoharm | what is a console script? | |
| 14:32:41 | zoharm | we are aware of the option of having an external script, but the goal here is to get it into openstack | |
| 14:32:42 | sean-k-mooney | zoharm: its a setuptool entroy point that creates a indepenten binary that you can run in the termal | |
| 14:33:07 | zoharm | ok that sounds interesting, can you give me a reference / example? | |
| 14:33:19 | sean-k-mooney | sure on sec | |
| 14:33:45 | zoharm | and proposal is not to run it from terminal, but have openstack run it only when needed. | |
| 14:33:57 | gibi | if this agent is just executes a periodic task then I guess such periodic task can be added to either the cinder or the nova agent to call a function in the os-brick lib | |
| 14:33:57 | sean-k-mooney | the nova compute agent itslef is one https://github.com/openstack/nova/blob/master/setup.cfg#L73 | |
| 14:34:11 | sean-k-mooney | that creats a binary that runs https://github.com/openstack/nova/blob/master/nova/cmd/compute.py | |
| 14:34:40 | sean-k-mooney | which uses oslo service to run the nova compute applicatoin as a deamon | |
| 14:34:46 | zoharm | i see, but that would be a whole new service, which the operator would need to launch independently | |
| 14:34:53 | sean-k-mooney | yes | |
| 14:35:03 | sean-k-mooney | privsep works slightly differently | |
| 14:35:08 | zoharm | this is not the proposal, it is just as good as an external script | |
| 14:35:18 | sean-k-mooney | in that we launch a different process in one of 2 ways | |
| 14:35:33 | sean-k-mooney | either we fork a seperate process using the privsep helper and typicaly sudo | |
| 14:35:43 | zoharm | i think having completley new service to something that is tightly coupled to an os-brick connector is overkill both operationally and in code | |
| 14:35:57 | sean-k-mooney | or we run nova with elevetated prviages for the process and drop privlates on nova-compute | |
| 14:36:14 | zoharm | ok, can we do that for this agent too? | |
| 14:36:30 | zoharm | if we can't use python-daemon | |
| 14:36:32 | sean-k-mooney | possibly but it depned on what it does i have not read the spec | |
| 14:36:52 | zoharm | it is ran only when an NVMe volume is connected to a host | |
| 14:37:08 | zoharm | it monitors the connection, and if its a replicated volume, self heals it | |
| 14:37:10 | sean-k-mooney | but then it continues running | |
| 14:37:23 | sean-k-mooney | basicaly what are the lifetime sematince you want | |
| 14:37:26 | zoharm | yes, but we can add a mechanism to self kill when there are not more nvme connections | |
| 14:37:42 | sean-k-mooney | should it continue running if nova-compute is stopped but the vm is still running | |
| 14:37:44 | zoharm | keep running as long as there are managed nvme volumes connected to by connector | |
| 14:37:52 | zoharm | yes | |
| 14:38:06 | zoharm | if vm is connected to nvme volume via the nvme connector | |
| 14:38:22 | sean-k-mooney | i think right now privsep will exit in that case wehn the last clinet on the privsep socket disconnects | |
| 14:38:43 | sean-k-mooney | privsep usins a unix socket to comunicate between the two processes | |
| 14:38:54 | zoharm | right | |
| 14:39:10 | zoharm | that is a good point, that all process can exit but VMs can keep running and consuming connected volumes... | |
| 14:39:17 | zoharm | looks like another pain point | |
| 14:39:28 | sean-k-mooney | that is one that comes up during upgrades | |
| 14:39:56 | zoharm | we can argue that it is ok to have this agent disabled for short time during upgrade, as long as something turns it back on | |
| 14:40:26 | sean-k-mooney | well the thing that turns it back on would be initalisin os-brick | |
| 14:40:36 | sean-k-mooney | when nova-compute is started after the upgrade | |
| 14:41:18 | sean-k-mooney | if the lifetime of this seperate process could be made the same as the process the spawned ti then mimicing privsep works | |
| 14:41:19 | zoharm | ok, so in this case in connector init, scan for nvme volumes that are managed by this and if there are launch the agent | |
| 14:41:55 | sean-k-mooney | the trigger from the nova side would likely be init_host | |
| 14:41:56 | zoharm | what if one process spawned it, then another process also wants it spawned, and then first process was permanently stopped | |
| 14:42:14 | zoharm | init_host sounds good, that is in os-brick? | |
| 14:42:15 | sean-k-mooney | where we loop over all the vms and check that there networking and presumabley volume connection are set up | |
| 14:42:22 | sean-k-mooney | no in nova | |
| 14:42:29 | sean-k-mooney | but i think we call into os-brick | |
| 14:42:44 | sean-k-mooney | we call into os-vif to ensure the vifs are pluuged in init host | |
| 14:42:50 | sean-k-mooney | i expect we do the same for os-brick | |
| 14:43:14 | zoharm | ok, we want to keep this change out of nova for now since its specifically deals with only volumes | |
| 14:43:51 | zoharm | unless you think its better to add it to nova | |
| 14:44:13 | sean-k-mooney | dose os-brick have an inialise function? | |
| 14:44:22 | zoharm | for volume connection? | |
| 14:44:36 | zoharm | there is init for the connector which is called when its loaded up for first time | |
| 14:44:41 | sean-k-mooney | no for the lib in general | |