| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-02-20 | |||
| 17:02:31 | mriedem | melwitt: if compute is passing use_cache=False to an out of tree driver that implements get_info(instance) but doesn't have that kwarg in the method signature, it's going to blow up | |
| 17:02:54 | melwitt | oh, right. urgh | |
| 17:03:34 | mriedem | http://paste.openstack.org/show/745484/ | |
| 17:04:05 | mriedem | so the worst that happens is out of tree drivers don't have the sync power states task working until they fix that | |
| 17:04:53 | melwitt | jroll ^ | |
| 17:05:49 | jroll | melwitt: yep, that's correct, I just didn't know what the policy was given out of tree drivers are unsupported | |
| 17:06:29 | jroll | I'm not sure there's a different approach we can take, am open to it though | |
| 17:06:54 | jroll | other than reverting the thing that made the sync use the cache, and spamming ironic for that task | |
| 17:07:06 | mriedem | i'm commenting | |
| 17:07:11 | mriedem | cool your jets and hold your horses please | |
| 17:07:14 | mriedem | oath can wait | |
| 17:07:31 | jroll | it's not even for oath, larsks reported this bug and I hacked it up ¯\_(ツ)_/¯ | |
| 17:07:42 | jroll | besides oath is dead, long live verizon media :P | |
| 17:07:47 | mriedem | oh right | |
| 17:07:58 | mriedem | vmg 4ever | |
| 17:08:14 | openstackgerrit | Stephen Finucane proposed openstack/os-vif master: WIP: Add API docs for various VIF types https://review.openstack.org/637009 | |
| 17:10:32 | mriedem | jroll: posted | |
| 17:10:50 | mriedem | my suggestion is handling a TypeError in this change if you're going to backport it, and in a follow up remove the TypeError handling | |
| 17:13:24 | jroll | mriedem: oh, that's a good idea, thanks. I'll update with that | |
| 17:16:49 | mdbooth | sean-k-mooney: That port binding error I mentioned this morning could be a more serious thing. I'm currently messing with LM and generating lots of LM failures, and in every case it's not possible to repeat the LM because: | |
| 17:16:51 | mdbooth | Migration pre-check error: Binding failed for port 48c1c3b8-be35-46a0-8df2-60495b4c6b9d, please check neutron logs for more information. (HTTP 400) (Request-ID: req-3217c16e-438a-4025-a462-8979fc97bd80) | |
| 17:16:51 | mdbooth | $ openstack server migrate --live devstack-2.marston --block-migration test1 | |
| 17:16:58 | mriedem | jroll: you may also want to take the opportunity to refresh the node_cache but that would be a follow up | |
| 17:17:37 | mriedem | mdbooth: is the vif_type on the port binding "binding_failed" or "unbound"? | |
| 17:17:52 | mriedem | if so, we don't know how to translate that to an os-vif object to plug the vif on the dest host | |
| 17:17:54 | mriedem | and it blows up | |
| 17:18:00 | jroll | mriedem: eh, that's a bigger timesink that I'm not sure I want to do in the middle of that loop... though would help in the case of a whole rack getting unplugged or something | |
| 17:18:27 | mriedem | mdbooth: semi related https://review.openstack.org/#/c/603844/ | |
| 17:18:53 | mdbooth | mriedem: Well it remains bound on the source, so it looks normal I guess? | |
| 17:18:54 | mriedem | mdbooth: if the vif_type in the port binding is busted, you might have to detach and re-attach the port to the server to fix it | |
| 17:19:09 | mriedem | mdbooth: check the port binding details on the port in the neutron api | |
| 17:19:16 | mriedem | you probably have to be admin to get that field | |
| 17:19:35 | mriedem | binding:vif_type is what you're looking for | |
| 17:19:42 | mdbooth | It's 'ovs' | |
| 17:19:53 | mriedem | oh huh | |
| 17:20:03 | mriedem | well actually, | |
| 17:20:12 | mriedem | is this rocky? | |
| 17:20:15 | mdbooth | mriedem: This is a dark art to me, btw :) | |
| 17:20:17 | mdbooth | devstack master | |
| 17:20:36 | mriedem | the port binding for the dest host will be on GET /ports/{port_id}/bindings/{dest_host} | |
| 17:20:47 | mriedem | the port binding details you're looking at on the port are likely the source host | |
| 17:20:53 | mriedem | which is the 'active' binding | |
| 17:20:58 | mriedem | the dest host binding is the inactive one | |
| 17:21:02 | mriedem | and it's what you're trying to plug | |
| 17:21:10 | mdbooth | Ah, ok. | |
| 17:21:19 | mdbooth | No cli for that? | |
| 17:21:25 | sean-k-mooney | mdbooth: it was in the spec but no | |
| 17:21:26 | mriedem | unfortunately there is no api ref for the port bindings api yet | |
| 17:22:01 | mdbooth | I'm assuming if I look at the LM code in nova we're failing to clean that up | |
| 17:23:01 | sean-k-mooney | mdbooth: to clean up a host binding? | |
| 17:23:51 | mriedem | _rollback_live_migration should delete inactive dest host port bindings | |
| 17:23:53 | sean-k-mooney | mdbooth: i had tought we had code for that | |
| 17:24:21 | mriedem | https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L6958 | |
| 17:24:39 | mriedem | https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L7010 | |
| 17:25:12 | mriedem | if we failed to cleanup the dest host port bindings where should be an error in the compute logs | |
| 17:27:41 | mdbooth | Hmm, nothing obvious | |
| 17:28:08 | mdbooth | Anyway, I'm going to have to shoot. I'll investigate properly in the morning, was just sharing in case sean-k-mooney had immediate insight. | |
| 17:28:25 | mdbooth | Thanks for the help. | |
| 17:30:55 | mriedem | dansmith: commented https://review.openstack.org/#/c/635147/ | |
| 17:38:04 | sean-k-mooney | mdbooth i know that i have seen case where cleaning up the volume on a roleback fails and we raise an excpetion and dont then rollback the networking | |
| 17:39:29 | sean-k-mooney | mdbooth: my guess is that is the most likely cause. for example if self.compute_rpcapi.remove_volume_connection rased an exception we would never call self.network_api.setup_networks_on_host | |
| 17:40:43 | sean-k-mooney | mdbooth: we probaly should split _rollback_live_migration into several smaller fucntion and call them with a set fo try finally statement to ensure that all resouces that can be rolled back are | |
| 17:42:25 | sean-k-mooney | mdbooth: ill be fixing stuff arount this code for something else next sprint so i might look at doing that change then. | |
| 17:43:30 | jaypipes | gibi: still around? | |
| 17:44:00 | jaypipes | gibi: on https://review.openstack.org/#/c/616239/ did we decide not to change anything for the group_policy_isolate=true? | |
| 18:11:07 | artom | Resizing to the same host is expected to work, right? | |
| 18:11:10 | openstackgerrit | Dan Smith proposed openstack/nova master: API microversion 2.69: Handles Down Cells Documentation https://review.openstack.org/635147 | |
| 18:11:34 | artom | I'm debugging an internal thing, and it looks like we're still asking Neutron to update the port bindings, even though nothing's changed. | |
| 18:12:11 | artom | Bleargh no, it's the revert. | |
| 18:12:14 | artom | Never mind. | |
| 18:12:21 | artom | Thanks for being my rubber duck. | |
| 18:13:32 | sean-k-mooney | same host resize needs to be enable in the nova config i think but yes it should work | |
| 18:15:16 | sean-k-mooney | artom: by the way we do need to rebind on the neutron side as the operator could have changed the security group driver for instance so how we attach the instance to the netowrk backend could change | |
| 18:15:51 | sean-k-mooney | artom: this can happen for example if you did and inplace upgrade and then resize the vm | |
| 18:16:09 | artom | sean-k-mooney, aha, ok | |
| 18:16:11 | sean-k-mooney | normally the changes would get picked up on the next hard reboot | |
| 18:16:56 | sean-k-mooney | artom: but ya were you actully hittin an issue? | |
| 18:17:10 | artom | sean-k-mooney, not 100% sure yet, but looks like it's Neutron | |
| 18:17:23 | artom | We ask for a thing, and timeout waiting for it | |
| 18:18:16 | artom | Grr, and I can't find the corresponding Neutron request | |
| 18:18:35 | sean-k-mooney | oh is this the virtual interface thing we were talking about downstream earlier | |
| 18:19:48 | artom | Yeah | |
| 18:20:29 | sean-k-mooney | artom: as i mentioned to mdbooth eariler if any of the storage cleanup code raises an exception in _rollback_live_migration its possible today that we will never call self.network_api.setup_network_on_host to rollback the network | |
| 18:20:51 | sean-k-mooney | so if there is an exption earilier in the function https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L6958 will not be called | |
| 18:21:01 | artom | sean-k-mooney, no, I think we're reaching that part | |
| 18:21:10 | artom | As I said, I see Nova making the Neutron request | |
| 18:21:39 | artom | But debug's not enabled, so I can't find that request arriving on the Neutron side | |
| 18:21:54 | artom | I mean, maybe it doesn't? | |
| 18:22:09 | sean-k-mooney | you shoudl see it in the neutron api log without debugging | |
| 18:22:10 | artom | Actually, debug *is* enabled | |
| 18:22:58 | sean-k-mooney | you may not see al of the intermediate steps but the post to the port binding endoint should be there | |
| 18:23:32 | artom | It'll log the HTTP method, right? | |
| 18:23:45 | artom | I'm casting a wide net and straight up grep -E '2019-02-14 18:50:5.*(PUT|POST)' -R `find . -wholename '*neutron/*.log'` | |
| 18:23:46 | sean-k-mooney | ya and it should have the respocne code | |
| 18:23:51 | artom | Nothing. | |
| 18:25:23 | artom | That's weird though, 'cuz Nova would log a timeout | |
| 18:25:44 | sean-k-mooney | so it timed out waiting for the network-vif-plugged event | |
| 18:26:12 | sean-k-mooney | whcich got raised as a VirtualInterfaceCreateExcetiopn. | |
| 18:27:00 | sean-k-mooney | the logs i have did not containt the neutron server logs | |
| 18:27:30 | artom | sean-k-mooney, that might be why | |