Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-02-20
16:34:19 mriedem i'll post that as a patch before yours in the series
16:34:38 openstackgerrit Balazs Gibizer proposed openstack/nova master: cache neutron ports in heal allocation https://review.openstack.org/638207
16:34:38 openstackgerrit Balazs Gibizer proposed openstack/nova master: nova-manage: heal port allocations https://review.openstack.org/637955
16:43:15 melwitt mriedem: we have an ironic driver fix proposed that adds a kwarg to the get_info() virt method. we wanted your opinion about whether it's backportable and if not, advice on another way to approach it https://review.openstack.org/636699
16:57:46 mriedem well that's a backward incompatible change to any out of tree drivers on stable
16:57:51 mriedem so it depends on if you care about that
16:57:57 mriedem efried_pto probably cares about it
16:58:50 openstackgerrit Matt Riedemann proposed openstack/python-novaclient master: API microversion 2.69: Handles Down Cells https://review.openstack.org/579563
16:58:51 openstackgerrit Matt Riedemann proposed openstack/python-novaclient master: Make Server.networks use a predictable sort order https://review.openstack.org/638214
16:59:12 melwitt a new kwarg with a default value? I didn't think that's backward incompat
17:01:03 tssurya mriedem: oh thanks a lot!
17:01:22 tssurya and as far as the host name being "host1" always its because the fake has that hardcoded
17:01:34 tssurya I can just change that if you want them to be unique
17:02:31 mriedem melwitt: if compute is passing use_cache=False to an out of tree driver that implements get_info(instance) but doesn't have that kwarg in the method signature, it's going to blow up
17:02:54 melwitt oh, right. urgh
17:03:34 mriedem http://paste.openstack.org/show/745484/
17:04:05 mriedem so the worst that happens is out of tree drivers don't have the sync power states task working until they fix that
17:04:53 melwitt jroll ^
17:05:49 jroll melwitt: yep, that's correct, I just didn't know what the policy was given out of tree drivers are unsupported
17:06:29 jroll I'm not sure there's a different approach we can take, am open to it though
17:06:54 jroll other than reverting the thing that made the sync use the cache, and spamming ironic for that task
17:07:06 mriedem i'm commenting
17:07:11 mriedem cool your jets and hold your horses please
17:07:14 mriedem oath can wait
17:07:31 jroll it's not even for oath, larsks reported this bug and I hacked it up ¯\_(ツ)_/¯
17:07:42 jroll besides oath is dead, long live verizon media :P
17:07:47 mriedem oh right
17:07:58 mriedem vmg 4ever
17:08:14 openstackgerrit Stephen Finucane proposed openstack/os-vif master: WIP: Add API docs for various VIF types https://review.openstack.org/637009
17:10:32 mriedem jroll: posted
17:10:50 mriedem my suggestion is handling a TypeError in this change if you're going to backport it, and in a follow up remove the TypeError handling
17:13:24 jroll mriedem: oh, that's a good idea, thanks. I'll update with that
17:16:49 mdbooth sean-k-mooney: That port binding error I mentioned this morning could be a more serious thing. I'm currently messing with LM and generating lots of LM failures, and in every case it's not possible to repeat the LM because:
17:16:51 mdbooth Migration pre-check error: Binding failed for port 48c1c3b8-be35-46a0-8df2-60495b4c6b9d, please check neutron logs for more information. (HTTP 400) (Request-ID: req-3217c16e-438a-4025-a462-8979fc97bd80)
17:16:51 mdbooth $ openstack server migrate --live devstack-2.marston --block-migration test1
17:16:58 mriedem jroll: you may also want to take the opportunity to refresh the node_cache but that would be a follow up
17:17:37 mriedem mdbooth: is the vif_type on the port binding "binding_failed" or "unbound"?
17:17:52 mriedem if so, we don't know how to translate that to an os-vif object to plug the vif on the dest host
17:17:54 mriedem and it blows up
17:18:00 jroll mriedem: eh, that's a bigger timesink that I'm not sure I want to do in the middle of that loop... though would help in the case of a whole rack getting unplugged or something
17:18:27 mriedem mdbooth: semi related https://review.openstack.org/#/c/603844/
17:18:53 mdbooth mriedem: Well it remains bound on the source, so it looks normal I guess?
17:18:54 mriedem mdbooth: if the vif_type in the port binding is busted, you might have to detach and re-attach the port to the server to fix it
17:19:09 mriedem mdbooth: check the port binding details on the port in the neutron api
17:19:16 mriedem you probably have to be admin to get that field
17:19:35 mriedem binding:vif_type is what you're looking for
17:19:42 mdbooth It's 'ovs'
17:19:53 mriedem oh huh
17:20:03 mriedem well actually,
17:20:12 mriedem is this rocky?
17:20:15 mdbooth mriedem: This is a dark art to me, btw :)
17:20:17 mdbooth devstack master
17:20:36 mriedem the port binding for the dest host will be on GET /ports/{port_id}/bindings/{dest_host}
17:20:47 mriedem the port binding details you're looking at on the port are likely the source host
17:20:53 mriedem which is the 'active' binding
17:20:58 mriedem the dest host binding is the inactive one
17:21:02 mriedem and it's what you're trying to plug
17:21:10 mdbooth Ah, ok.
17:21:19 mdbooth No cli for that?
17:21:25 sean-k-mooney mdbooth: it was in the spec but no
17:21:26 mriedem unfortunately there is no api ref for the port bindings api yet
17:22:01 mdbooth I'm assuming if I look at the LM code in nova we're failing to clean that up
17:23:01 sean-k-mooney mdbooth: to clean up a host binding?
17:23:51 mriedem _rollback_live_migration should delete inactive dest host port bindings
17:23:53 sean-k-mooney mdbooth: i had tought we had code for that
17:24:21 mriedem https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L6958
17:24:39 mriedem https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L7010
17:25:12 mriedem if we failed to cleanup the dest host port bindings where should be an error in the compute logs
17:27:41 mdbooth Hmm, nothing obvious
17:28:08 mdbooth Anyway, I'm going to have to shoot. I'll investigate properly in the morning, was just sharing in case sean-k-mooney had immediate insight.
17:28:25 mdbooth Thanks for the help.
17:30:55 mriedem dansmith: commented https://review.openstack.org/#/c/635147/
17:38:04 sean-k-mooney mdbooth i know that i have seen case where cleaning up the volume on a roleback fails and we raise an excpetion and dont then rollback the networking
17:39:29 sean-k-mooney mdbooth: my guess is that is the most likely cause. for example if self.compute_rpcapi.remove_volume_connection rased an exception we would never call self.network_api.setup_networks_on_host
17:40:43 sean-k-mooney mdbooth: we probaly should split _rollback_live_migration into several smaller fucntion and call them with a set fo try finally statement to ensure that all resouces that can be rolled back are
17:42:25 sean-k-mooney mdbooth: ill be fixing stuff arount this code for something else next sprint so i might look at doing that change then.
17:43:30 jaypipes gibi: still around?
17:44:00 jaypipes gibi: on https://review.openstack.org/#/c/616239/ did we decide not to change anything for the group_policy_isolate=true?
18:11:07 artom Resizing to the same host is expected to work, right?
18:11:10 openstackgerrit Dan Smith proposed openstack/nova master: API microversion 2.69: Handles Down Cells Documentation https://review.openstack.org/635147
18:11:34 artom I'm debugging an internal thing, and it looks like we're still asking Neutron to update the port bindings, even though nothing's changed.
18:12:11 artom Bleargh no, it's the revert.
18:12:14 artom Never mind.
18:12:21 artom Thanks for being my rubber duck.
18:13:32 sean-k-mooney same host resize needs to be enable in the nova config i think but yes it should work
18:15:16 sean-k-mooney artom: by the way we do need to rebind on the neutron side as the operator could have changed the security group driver for instance so how we attach the instance to the netowrk backend could change
18:15:51 sean-k-mooney artom: this can happen for example if you did and inplace upgrade and then resize the vm
18:16:09 artom sean-k-mooney, aha, ok
18:16:11 sean-k-mooney normally the changes would get picked up on the next hard reboot
18:16:56 sean-k-mooney artom: but ya were you actully hittin an issue?
18:17:10 artom sean-k-mooney, not 100% sure yet, but looks like it's Neutron
18:17:23 artom We ask for a thing, and timeout waiting for it
18:18:16 artom Grr, and I can't find the corresponding Neutron request
18:18:35 sean-k-mooney oh is this the virtual interface thing we were talking about downstream earlier
18:19:48 artom Yeah
18:20:29 sean-k-mooney artom: as i mentioned to mdbooth eariler if any of the storage cleanup code raises an exception in _rollback_live_migration its possible today that we will never call self.network_api.setup_network_on_host to rollback the network
18:20:51 sean-k-mooney so if there is an exption earilier in the function https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L6958 will not be called
18:21:01 artom sean-k-mooney, no, I think we're reaching that part
18:21:10 artom As I said, I see Nova making the Neutron request
18:21:39 artom But debug's not enabled, so I can't find that request arriving on the Neutron side

Earlier   Later