Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-06-10
03:07:50 openstackgerrit melanie witt proposed openstack/nova stable/queens: libvirt: Don't delete disks on shared storage during evacuate https://review.opendev.org/732717
06:41:48 tony_su dansmith: A reviewer of ProviderConfigFile patch wanted to understand why additionProperties is set to true by default in the first place. https://review.opendev.org/#/c/673341/45/nova/compute/provider_config_schemas/provider_config_schema_v1.yaml might require your attention. Thanks.
07:09:45 bauzas good morning Nova
07:20:37 gibi bauzas: good morning
07:21:12 bauzas gibi: morning
07:21:29 bauzas fwiw, just created a blueprint for tracking reshape tooling work for Ussuri https://blueprints.launchpad.net/nova/+spec/offline-reshape-tool
07:22:46 gibi bauzas: cool. Do you agree that we had enough discussion about FFU on the PTG so that I can approve that bp without further discussion on the weekly meeting?
07:23:28 openstackgerrit sunhao proposed openstack/nova master: Add checks for volume status when rebuilding https://review.opendev.org/734734
07:31:04 bauzas gibi: well, I'm torn about it
07:31:28 bauzas gibi: we discussed at the PTG about how to call placement without having the service running
07:31:37 bauzas but i'm not sure we had a consensus
07:32:22 bauzas we have a way to do it that was accepted for the reshape spec in https://specs.openstack.org/openstack/nova-specs/specs/stein/approved/reshape-provider-tree.html#direct-interface-to-placement
07:35:28 gibi bauzas: OK, then let's double check this with the team on the weekly meeting.
07:35:36 bauzas thanks
07:35:49 gibi I will add it to the agenda
07:36:04 bauzas gibi: looking at the spec, looks like we agreed on using PlacementDirect for this
07:36:28 bauzas so if folks agree in the meeting, we can have a specless bp
07:36:36 gibi bauzas: yes, but on the PTG I was tried to mention that fact and get a pushback because that would require DB access from the compute nodes
07:36:58 bauzas gibi: PlacementDirect supports oslo.config
07:37:28 bauzas so you would run the nova-manage command that would call PlacementDirect with DB options
07:37:29 gibi true, but our deployment model does not require db access for compute hosts so far
07:37:53 gibi so the db server might not even accessible from the compute hosts
07:37:59 gibi e.g network separation
07:38:11 bauzas gibi: surely, but that would be something we would comment
07:38:16 bauzas like a caveat
07:38:28 bauzas and it's just for FFUs
07:38:50 gibi OK
07:46:53 gibi let's go with the original plan from the spe
07:46:54 gibi c
08:14:12 sean-k-mooney gibi: bauzas well ingeneral we whould assume the db server is not accesable from the compute yes
08:15:41 sean-k-mooney this is why we discussed the need to potentialy seperate the data collection from the updating of placment or the db
08:16:18 sean-k-mooney e.g. we might need to run something on the compute node to generate some output which is the canges to be done via ssh then do the update from the contoler instead
08:16:38 sean-k-mooney i think johnthetubaguy brought that up or dansmith in the ptg sesssion
08:18:16 sean-k-mooney gibi: and ya its not even a case of our deployment model does not require it it activly discurages direct db acess form the compute on security grounds
08:18:56 sean-k-mooney at least it has sicne we got rid fo the local conducor stuff like 2-3 years ago
08:20:54 bauzas sean-k-mooney: operators can pipe their needs for just a FFU upgrade if they want
08:21:25 sean-k-mooney "pipe there needs" ?
08:22:53 sean-k-mooney by the way look like some people from vmware are fixing the vmware nsx ci
08:24:36 sean-k-mooney http://207.189.188.190/logs/58/734658/1/check-vote/ext-nova-zuul/f8d2e76/tempest_results.html.gz
08:25:57 sean-k-mooney i think it has to be manually triggered via vmware-recheck-patch but i guess that is still an improvement
08:26:16 bauzas sean-k-mooney: I mean, for example, to do something like piping netcat
08:27:49 sean-k-mooney bauzas: they could but i dont think we sould require direct db access
08:28:24 sean-k-mooney making them proxy it via netcat or ssh is a hack IMO
08:28:27 bauzas it's a nova-manage command so we need it anyway :)
08:28:40 bauzas they need to provide DB creds
08:28:57 sean-k-mooney bauzas: so that means nova-manage is not suitable solution to this probelem
08:29:14 bauzas but if you need to run this command on a nova-compute (and then you don't have a specific DB cred for it), then you could just use netcat
08:30:06 bauzas sean-k-mooney: it would work for any operator that doesn't want to have compute nodes be in a specific network
08:30:11 sean-k-mooney i think we should be providing a way to pass in some data structro to the nova-manage command which it uses to then do the reshapes
08:30:33 bauzas sean-k-mooney: we could do it, that's one of the things we said
08:30:41 bauzas hence two commands
08:31:07 bauzas and piping between them (or passing a data structure)
08:31:31 sean-k-mooney yep once command that you run on the compute node that gets the info form the virt driver and produces the reshap yaml or whatever and then a second that you run on a node with db access
08:31:36 bauzas sean-k-mooney: but tbc, https://specs.openstack.org/openstack/nova-specs/specs/stein/approved/reshape-provider-tree.html#direct-interface-to-placement already thought about it
08:32:44 sean-k-mooney bauzas: that was not thinking about reshapes
08:32:49 sean-k-mooney it was thinking about db upgrades
08:32:55 sean-k-mooney those are very different things
08:33:27 sean-k-mooney well it mentions reshpaes but we did not have any at that point
08:34:45 sean-k-mooney actully i guess look ing at the spec in general this is the resahpe spec
08:35:17 sean-k-mooney i think it was making the assumtion however that reshaps did not need compute specific infomation
08:36:43 sean-k-mooney looking at the ironic_flavor_migration examle i dont think that is the same as the libvirt case
08:36:45 sean-k-mooney https://review.opendev.org/#/c/501025/12/nova/cmd/manage.py
08:36:58 sean-k-mooney because the ironic virt dirver talks to the ironic api
08:37:45 sean-k-mooney so the compute serivce virt driver does not need to be colocated on the same host as the hypervior
08:37:59 gibi the spec gives solution for one side of the problem. We don't have a running placement API service during FFU but we need to write into the placement db due to reshape
08:38:14 gibi placement direct allows that.
08:38:27 sean-k-mooney yes it will
08:38:28 gibi sean-k-mooney is right that collecting the data for the reshape is not solved in tha spec
08:39:21 gibi except that it mentions to start up a virt driver, but it does not handle the implication of that statemen
08:39:24 gibi t
08:39:41 sean-k-mooney gibi: https://review.opendev.org/#/c/501025/ is the implemeation for ironic
08:39:52 sean-k-mooney but that is very different then libvirt
08:41:13 sean-k-mooney the ironic virt dirver is used here https://review.opendev.org/#/c/501025/12/nova/cmd/manage.py@988
08:42:30 gibi interesting
08:42:31 sean-k-mooney we cant just call the reshape function in the libvirt dirver however.
08:42:53 sean-k-mooney ironic works because all the info it need is either in the nova db or in the ironic api
08:43:50 sean-k-mooney this is all that was doing https://github.com/openstack/nova/blob/b6f3d393aa5cb755f30fa70d4acc614c32216c99/nova/virt/ironic/driver.py#L570-L582
08:44:50 sean-k-mooney thats doing an online data migration but its not doing a reshape
08:46:00 gibi the analogous call would be nova.virt.libvirt.driver.LibvirtDriver.update_provider_tree
08:46:37 gibi but that needs a hypervisor connection as far as I see
08:46:46 sean-k-mooney well yes an no that would update the provider i guess but it would do more then just run the reshapes
08:47:28 sean-k-mooney gibi: yes it need the resouce track and a bunch of other things that reuiqre not just a connection to libvirt but file access too
08:47:49 sean-k-mooney im pretty sure we need to check /sys for some things
08:48:31 sean-k-mooney like sev support? we need the compute nodes config for allocation ratio stuff at the very least
08:48:40 gibi sean-k-mooney: agree that update_provider_tree does more than what is needed for reshape
08:49:22 gibi I guess we need to see what are the bare minimum information to run each reshape codepath and see how can we provide that
08:51:01 sean-k-mooney the pcpu reshape would need the compute node cpu_shared_set and cpu_dedicated_set config options techincally vcpu_pin_set too
08:51:35 sean-k-mooney i think the vgpu reshape to have seperate RP per pGPU needs libvirt and the mdev type config options
08:51:59 sean-k-mooney gibi: if we proceeed with numa in placment that is going to need the host numa toplogy info from libfirt
08:54:22 sean-k-mooney gibi: this is the vgpu reshape https://github.com/openstack/nova/blob/b6f3d393aa5cb755f30fa70d4acc614c32216c99/nova/virt/libvirt/driver.py#L7757
08:56:09 sean-k-mooney well actully _update_provider_tree_for_vgpu is the top level function we would be calling and _update_provider_tree_for_pcpu is the cpu one
08:58:02 gibi I see
08:58:32 sean-k-mooney _get_all_assigned_mediated_devices which is used in the vgpu reshape directly read the guest xml https://github.com/openstack/nova/blob/b6f3d393aa5cb755f30fa70d4acc614c32216c99/nova/virt/libvirt/driver.py#L6940-L6973
09:35:30 openstackgerrit Stephen Finucane proposed openstack/nova master: libvirt: Don't allow "reserving" file-backed memory https://review.opendev.org/734658
09:35:51 openstack bug 1550919 in OpenStack Compute (nova) ussuri "[Libvirt]Evacuate fail may cause disk image be deleted" [Medium,In progress] https://launchpad.net/bugs/1550919 - Assigned to Lee Yarwood (lyarwood)
09:35:51 openstackgerrit Merged openstack/nova stable/ussuri: Add functional test for bug 1550919 https://review.opendev.org/731247
09:36:15 stephenfin bauzas: Can you look at https://review.opendev.org/#/c/733668 ?
09:36:54 bauzas stephenfin: ack, will do
09:37:23 bauzas gibi: sean-k-mooney: sorry was on a meeting, saw your discussion
09:38:04 bauzas yup, from a vgpu reshape perspective, we need to lookup the guest XMLs hence libvirt

Earlier   Later