| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-04-24 | |||
| 14:54:52 | mriedem | bauzas: fwiw, i spent 6 hours yesterday in the hell zone that is swap volume | |
| 14:54:53 | bauzas | mriedem: jaypipes: AFAICT, we currently unclaim the resources (ie. we DELETE allocations) only by the compute service, right ? | |
| 14:55:20 | jaypipes | bauzas: yes, AFAIK. | |
| 14:55:26 | bauzas | mriedem: jaypipes: so, what happens in the case of a local delete where we gently instance.delete() ? | |
| 14:55:40 | bauzas | do we DELETE /allocations too ? None from what I can tell | |
| 14:55:42 | jaypipes | bauzas: only in the periodic audit interval task (update_available_resource) | |
| 14:56:08 | mriedem | bauzas: we don't delete allocations for local delete, | |
| 14:56:11 | mriedem | remember i opened that bug? | |
| 14:56:19 | bauzas | jaypipes: mriedem: okay, so then should we keep the same behaviour ? | |
| 14:56:21 | mriedem | bauzas: and yes, the only place we delete allocations today is the scheduler report client via the RT | |
| 14:56:39 | bauzas | I mean, should we really not be concerned by DELETE allocations in the scheduler claims spec ? | |
| 14:56:59 | bauzas | or we could do that earlier in the compute API worker | |
| 14:57:22 | bauzas | well, only if not soft-deletes, of course | |
| 14:57:31 | mriedem | the fact we don't cleanup allocations during local delete is a bug | |
| 14:57:36 | mriedem | regardless of claims in the scheduler | |
| 14:57:38 | bauzas | mriedem: I vaguely remembered one bug, yup | |
| 14:57:54 | openstack | Launchpad bug 1679750 in OpenStack Compute (nova) "Allocations are not cleaned up in placement for instance 'local delete' case" [Medium,Confirmed] | |
| 14:57:54 | mriedem | bauzas: https://bugs.launchpad.net/nova/+bug/1679750 | |
| 14:58:20 | bauzas | okay, so my question is | |
| 14:58:24 | mriedem | we talked about doing the cleanup in the api service, or leaving it to init_host on the compute, | |
| 14:58:42 | bauzas | should we modify the current behaviour of when we delete allocations, regardless of the above bug ? | |
| 14:58:45 | mriedem | i.e. if the db says we have deleted instances for this host when nova-compute starts up, then go through and attempt to delete their allocation records for that host | |
| 14:59:11 | mriedem | modify how? | |
| 14:59:16 | bauzas | or should we keep the compute service responsible for notifying placement ? | |
| 14:59:29 | bauzas | mriedem: call placement in the compute worker | |
| 14:59:40 | bauzas | mriedem: I mean the api worker | |
| 14:59:55 | bauzas | mriedem: before sending the RPC call to compute | |
| 14:59:56 | mriedem | i thought people wanted to do it in compute | |
| 15:00:06 | mriedem | hold up | |
| 15:00:10 | mriedem | you're conflating two things | |
| 15:00:23 | mriedem | 1. normal delete where we rpc cast to the compute to do the delete | |
| 15:00:34 | bauzas | right, whatever the delete type is | |
| 15:00:38 | mriedem | 2. local delete in the api where the compute service is down, or the instance is shelved offloaded | |
| 15:00:48 | bauzas | yup, I'm not conflating | |
| 15:01:04 | bauzas | I just wonder if we wanted to notify placement earlier than now | |
| 15:01:10 | mriedem | no | |
| 15:01:15 | mriedem | we don't want to tie up the api | |
| 15:01:25 | bauzas | okay, so I'll just wipe out the paragraph I wrote about deletes | |
| 15:01:35 | bauzas | because it's unrelated | |
| 15:01:37 | mriedem | we only do that in the api for local deletes, but there isn't clear agreement on deleting allocations in the api even in the local delete case | |
| 15:01:46 | mriedem | the options are in the bug report above | |
| 15:01:50 | bauzas | I'll just mention the bug as a necessary prereq | |
| 15:02:00 | mriedem | yeah the bug needs to be fixed regardless | |
| 15:02:03 | mriedem | and most likely backported | |
| 15:03:11 | bauzas | okay, thanks for cleaning up the mess in my brain | |
| 15:03:15 | bauzas | I'm all clear | |
| 15:05:19 | bauzas | jaypipes: so, you told me we don't DELETE allocations right now but only when we self-heal ? | |
| 15:05:23 | openstackgerrit | Stephen Finucane proposed openstack/os-vif master: HW offload support for openvswitch https://review.openstack.org/398277 | |
| 15:05:24 | openstackgerrit | Stephen Finucane proposed openstack/os-vif master: Add documentation for Linux Bridge plugin https://review.openstack.org/459291 | |
| 15:05:32 | bauzas | jaypipes: that's something that needs to change then | |
| 15:06:31 | jaypipes | bauzas: no, sorry, I was referring to instances that were terminated but the compute host was down at the time. | |
| 15:06:53 | jaypipes | bauzas: in a "normal" DELETE /servers call, the allocations are indeed deleted. | |
| 15:07:05 | bauzas | okay, looking at code | |
| 15:07:41 | sean-k-mooney | sfinucan: moshele: on the vnic type i dont think this should be documented seperatly per plugin as they are common across all plugins | |
| 15:08:09 | sfinucan | sean-k-mooney: Have you seen the patches themselves | |
| 15:08:31 | sfinucan | They're common, yes, but not all VIF types are supported across all plugins | |
| 15:08:41 | sfinucan | I didn't grasp that nuance until this morning | |
| 15:09:30 | bauzas | jaypipes: mmm, can't find where we call DELETE /allocations in the compute manager, but only in the RT | |
| 15:09:42 | sean-k-mooney | sfinucan: no i havent atleast not lately. correct that is why im working on https://review.openstack.org/#/c/441590/ | |
| 15:10:09 | sean-k-mooney | i am hoping to implement the nova neutron negociation this cycle if i can ever finish the supprot for ovs-dpdk in kolla | |
| 15:10:15 | jaypipes | bauzas: https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L716-L731 | |
| 15:10:38 | jaypipes | bauzas: that's called at the end of the manager's _delete_instance() method. | |
| 15:10:49 | bauzas | ok, seeing it noxw | |
| 15:10:54 | bauzas | we force a RT update | |
| 15:11:00 | bauzas | which will delete the allocation, gotcha | |
| 15:11:02 | jaypipes | bauzas: the _update_resource_tracker() call eventually calls the scheduler client. | |
| 15:11:06 | jaypipes | right. | |
| 15:11:14 | sean-k-mooney | sfinucan: the host_info object discribes what vif types are supported by each plugin and it will with that patch allso describe what port profiles are supporte db each vif_type per plugin | |
| 15:11:27 | bauzas | jaypipes: okay, no explicit allocation delete, RT handles that | |
| 15:11:45 | bauzas | jaypipes: thanks, so nothing needed from a scheduler claims spec perspective | |
| 15:11:51 | bauzas | jaypipes: except the local delete problem | |
| 15:12:01 | sean-k-mooney | sfinucan: this object will be serialsed into the neutron binding profile and read by the mechanisum drivers as part of the port binding call. | |
| 15:12:11 | jaypipes | bauzas: no, that's still called for soft deletes. | |
| 15:12:20 | sfinucan | sean-k-mooney: Nice. I wasn't aware of host_info either | |
| 15:12:23 | bauzas | jaypipes: local delete != soft delete | |
| 15:12:37 | jaypipes | bauzas: no? | |
| 15:12:38 | bauzas | jaypipes: if the RPC compute service is down, we local delete in the api worker | |
| 15:12:57 | bauzas | totally unrelated to soft deletes, which are just a specific VM state | |
| 15:12:59 | jaypipes | bauzas: right. the api worker sets instance to SOFT_DELETED | |
| 15:13:04 | bauzas | no no | |
| 15:13:07 | jaypipes | no? | |
| 15:13:07 | bauzas | sec | |
| 15:13:14 | bauzas | yet another fun | |
| 15:13:22 | sfinucan | sean-k-mooney: Do you think there's value in documenting the built-in plugins, in that case, as done here https://review.openstack.org/#/c/459258/2 | |
| 15:14:01 | bauzas | jaypipes: https://github.com/openstack/nova/blob/master/nova/compute/api.py#L2001 and https://github.com/openstack/nova/blob/master/nova/compute/api.py#L2027-L2032 | |
| 15:14:05 | sfinucan | i.e. the plugins themselves along with a little summary of how a given VIF type works in the given plugin/networking backend | |
| 15:14:08 | sean-k-mooney | sfinucan: so the other part of this is that in the future the mapping we currently do between teh existing vif types and the os-vif ones will go away with neutron specifying plugin,vif_type and port prifile in a single os-vif object as a responce to nova in the port binding call | |
| 15:14:15 | sean-k-mooney | sfinucan: ill take a look at it now | |
| 15:14:33 | mriedem | melwitt: dansmith: some easy dead code cleanup in conductor https://review.openstack.org/#/c/434615/ | |
| 15:14:35 | bauzas | jaypipes: mriedem mentioned a bug report because we do delete the instance but we don't delete the allocation | |
| 15:14:51 | bauzas | jaypipes: https://bugs.launchpad.net/nova/+bug/1679750 | |
| 15:14:52 | openstack | Launchpad bug 1679750 in OpenStack Compute (nova) "Allocations are not cleaned up in placement for instance 'local delete' case" [Medium,Confirmed] | |
| 15:15:04 | bauzas | FWIW, I'll assign myself to the bug since it's related to the claims spec | |
| 15:15:09 | mriedem | jaypipes: we don't soft delete in the api if the compute host is down | |
| 15:15:28 | mriedem | jaypipes: we send out notifications saying we're soft deleting, but really we don't | |
| 15:15:30 | mriedem | we hard delete that mofo | |
| 15:15:41 | mriedem | this has been keeping gibi awake at night | |
| 15:15:41 | sean-k-mooney | sfinucan: so i dont think we should have redefinition of the vif types per plugin but i would be ok with referencing the vif types that are definded globally | |
| 15:16:06 | sean-k-mooney | sfinucan: essentailly like you have done with the glossary | |
| 15:16:26 | sean-k-mooney | define each fo them as a term or link back to the singel definition. | |