| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-01-04 | |||
| 14:30:39 | kashyap | stephenfin: sean-k-mooney: One more, the above of what I asked is supported from Queens, is it? | |
| 14:30:42 | sean-k-mooney | you proably should not be sapawning a second deamon out of the os-brick lib | |
| 14:30:51 | zoharm | right, but we need an entry point for the agent/daemon as per the spec, and the connector is the place decided on | |
| 14:31:03 | zoharm | do you have an alternative? | |
| 14:31:35 | bnemec | zoharm: If you want to know more about how privsep works, you can read about it in https://github.com/openstack/oslo.privsep/blob/master/oslo_privsep/daemon.py#L15 | |
| 14:31:40 | sean-k-mooney | zoharm: a better apprcoh would be to have a seperate console script entry point that is run seperatly as a deamon and no tspawned out of eithet the nova-compute process or the cinder-volume process | |
| 14:32:12 | zoharm | what is a console script? | |
| 14:32:41 | zoharm | we are aware of the option of having an external script, but the goal here is to get it into openstack | |
| 14:32:42 | sean-k-mooney | zoharm: its a setuptool entroy point that creates a indepenten binary that you can run in the termal | |
| 14:33:07 | zoharm | ok that sounds interesting, can you give me a reference / example? | |
| 14:33:19 | sean-k-mooney | sure on sec | |
| 14:33:45 | zoharm | and proposal is not to run it from terminal, but have openstack run it only when needed. | |
| 14:33:57 | gibi | if this agent is just executes a periodic task then I guess such periodic task can be added to either the cinder or the nova agent to call a function in the os-brick lib | |
| 14:33:57 | sean-k-mooney | the nova compute agent itslef is one https://github.com/openstack/nova/blob/master/setup.cfg#L73 | |
| 14:34:11 | sean-k-mooney | that creats a binary that runs https://github.com/openstack/nova/blob/master/nova/cmd/compute.py | |
| 14:34:40 | sean-k-mooney | which uses oslo service to run the nova compute applicatoin as a deamon | |
| 14:34:46 | zoharm | i see, but that would be a whole new service, which the operator would need to launch independently | |
| 14:34:53 | sean-k-mooney | yes | |
| 14:35:03 | sean-k-mooney | privsep works slightly differently | |
| 14:35:08 | zoharm | this is not the proposal, it is just as good as an external script | |
| 14:35:18 | sean-k-mooney | in that we launch a different process in one of 2 ways | |
| 14:35:33 | sean-k-mooney | either we fork a seperate process using the privsep helper and typicaly sudo | |
| 14:35:43 | zoharm | i think having completley new service to something that is tightly coupled to an os-brick connector is overkill both operationally and in code | |
| 14:35:57 | sean-k-mooney | or we run nova with elevetated prviages for the process and drop privlates on nova-compute | |
| 14:36:14 | zoharm | ok, can we do that for this agent too? | |
| 14:36:30 | zoharm | if we can't use python-daemon | |
| 14:36:32 | sean-k-mooney | possibly but it depned on what it does i have not read the spec | |
| 14:36:52 | zoharm | it is ran only when an NVMe volume is connected to a host | |
| 14:37:08 | zoharm | it monitors the connection, and if its a replicated volume, self heals it | |
| 14:37:10 | sean-k-mooney | but then it continues running | |
| 14:37:23 | sean-k-mooney | basicaly what are the lifetime sematince you want | |
| 14:37:26 | zoharm | yes, but we can add a mechanism to self kill when there are not more nvme connections | |
| 14:37:42 | sean-k-mooney | should it continue running if nova-compute is stopped but the vm is still running | |
| 14:37:44 | zoharm | keep running as long as there are managed nvme volumes connected to by connector | |
| 14:37:52 | zoharm | yes | |
| 14:38:06 | zoharm | if vm is connected to nvme volume via the nvme connector | |
| 14:38:22 | sean-k-mooney | i think right now privsep will exit in that case wehn the last clinet on the privsep socket disconnects | |
| 14:38:43 | sean-k-mooney | privsep usins a unix socket to comunicate between the two processes | |
| 14:38:54 | zoharm | right | |
| 14:39:10 | zoharm | that is a good point, that all process can exit but VMs can keep running and consuming connected volumes... | |
| 14:39:17 | zoharm | looks like another pain point | |
| 14:39:28 | sean-k-mooney | that is one that comes up during upgrades | |
| 14:39:56 | zoharm | we can argue that it is ok to have this agent disabled for short time during upgrade, as long as something turns it back on | |
| 14:40:26 | sean-k-mooney | well the thing that turns it back on would be initalisin os-brick | |
| 14:40:36 | sean-k-mooney | when nova-compute is started after the upgrade | |
| 14:41:18 | sean-k-mooney | if the lifetime of this seperate process could be made the same as the process the spawned ti then mimicing privsep works | |
| 14:41:19 | zoharm | ok, so in this case in connector init, scan for nvme volumes that are managed by this and if there are launch the agent | |
| 14:41:55 | sean-k-mooney | the trigger from the nova side would likely be init_host | |
| 14:41:56 | zoharm | what if one process spawned it, then another process also wants it spawned, and then first process was permanently stopped | |
| 14:42:14 | zoharm | init_host sounds good, that is in os-brick? | |
| 14:42:15 | sean-k-mooney | where we loop over all the vms and check that there networking and presumabley volume connection are set up | |
| 14:42:22 | sean-k-mooney | no in nova | |
| 14:42:29 | sean-k-mooney | but i think we call into os-brick | |
| 14:42:44 | sean-k-mooney | we call into os-vif to ensure the vifs are pluuged in init host | |
| 14:42:50 | sean-k-mooney | i expect we do the same for os-brick | |
| 14:43:14 | zoharm | ok, we want to keep this change out of nova for now since its specifically deals with only volumes | |
| 14:43:51 | zoharm | unless you think its better to add it to nova | |
| 14:44:13 | sean-k-mooney | dose os-brick have an inialise function? | |
| 14:44:22 | zoharm | for volume connection? | |
| 14:44:36 | zoharm | there is init for the connector which is called when its loaded up for first time | |
| 14:44:41 | sean-k-mooney | no for the lib in general | |
| 14:44:46 | sean-k-mooney | os-vif has https://github.com/openstack/os-vif/blob/master/os_vif/__init__.py#L24 | |
| 14:45:04 | sean-k-mooney | which is called when the compute agent start to tell os vif to load all its plugin dirvers | |
| 14:45:05 | zoharm | i dont know, but i think that is not best place to put this re-init agent entrypoint | |
| 14:45:19 | zoharm | do you know if nova ends up calling os-brick connect_volume during host_init? | |
| 14:45:45 | zoharm | if it does, then that solves everything for us because connect_volume is currently the entry point | |
| 14:46:31 | zoharm | we dont want to launch this agent when os-brick or connector loads (since all connectors / os-brick are loaded for every service and we dont want this agent running everywhere, only where nvme volumes are connected via the connector) | |
| 14:46:35 | sean-k-mooney | im just reading https://github.com/openstack/nova/blob/b0f241e5425c99866223bae4b404a4aa1abdfddf/nova/compute/manager.py#L956 now | |
| 14:46:54 | sean-k-mooney | for each instance we call _init_instance in init_host | |
| 14:47:37 | sean-k-mooney | this is where we ensure the network interfaces are set up https://github.com/openstack/nova/blob/b0f241e5425c99866223bae4b404a4aa1abdfddf/nova/compute/manager.py#L1137 | |
| 14:48:26 | zoharm | im looking through it too now, found _init_volume_connection but it is not called during init_host | |
| 14:48:58 | zoharm | back to python-daemon, do you think it is not ok to propose this new depedency? | |
| 14:51:35 | sean-k-mooney | you could but do you need too | |
| 14:51:40 | sean-k-mooney | what benifit will it provide | |
| 14:56:50 | zoharm | it will allow to run this agent as a single independent process that will keep running regardless of the different services that may spawn it | |
| 14:57:25 | sean-k-mooney | ok but then how to you interact with it and manage its lifetime | |
| 14:57:30 | zoharm | and we can have it terminate itself when no connections are left if needed | |
| 14:58:22 | zoharm | we dont need to interact with it much, the interaction is mostly spawning it, and then it will be calling volume backend API for its functionality | |
| 14:59:22 | sean-k-mooney | based on what? i will need to have some set of input and know when a nvme volume attachemt is made | |
| 14:59:24 | zoharm | once it runs it basically just monitors local nvme connections belonging to it (it can tell them apart) and calling volume backend if necessary | |
| 15:00:02 | zoharm | the nvme connector connect_volume is called when nvme volume attachment is made, and that is the entry point where we ensure the agent is running | |
| 15:01:07 | zoharm | certain agent implementation is vendor specific, but it can tell by reading nvme device related dev paths information about where it came from | |
| 15:01:19 | zoharm | and backend provides metadata | |
| 15:01:28 | openstackgerrit | Merged openstack/python-novaclient master: Fix a functional test for 'nova agent-list' https://review.opendev.org/c/openstack/python-novaclient/+/769066 | |
| 15:02:05 | zoharm | basically what the agent does is reconcile metadata from backend with physical connection state on host (this is mostly for mdraid replicated volumes) | |
| 15:02:20 | sean-k-mooney | im reading the spec now but i dont think this agent really fits in the project scope fo os-brick | |
| 15:03:03 | zoharm | i understand your concern, this is where we settled on putting it for now | |
| 15:03:05 | sean-k-mooney | it really does seam like an indepenet service similar to multipathd rhater then a capablity a libviary shoudl be providing | |
| 15:03:20 | zoharm | good point | |
| 15:03:55 | sean-k-mooney | well im concerend that this is a possibel ddos vector for the cinder api or storage backend depending on how it monitors the conenctiions and what api calls its makink | |
| 15:04:17 | sean-k-mooney | e.g. if i have a 1000 node deployment and i install this on all notes how will that work | |
| 15:05:03 | sean-k-mooney | this agent will spawn a monitoring task which will repeat | |
| 15:05:05 | sean-k-mooney | periodically. | |
| 15:05:23 | sean-k-mooney | does it need to be an agent or could it be a perodic task defined in os-brick | |
| 15:05:47 | sean-k-mooney | that is then elecitly run by the consumer of os-brick | |
| 15:06:36 | sean-k-mooney | "One key problem that would need to be addressed by this selection is a scenario | |
| 15:06:38 | sean-k-mooney | where compute service goes down, while the VMs continue operating (and their | |
| 15:06:40 | sean-k-mooney | volumes remain attached) - we don't want to lose this agent in this case. | |
| 15:06:42 | sean-k-mooney | " | |
| 15:06:56 | sean-k-mooney | that is the main motivator for a seperate deamon process right | |