Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-02-20
15:42:44 tssurya mriedem: oh okay
15:42:48 tssurya dansmith: thanks
15:53:07 jaypipes jackding: k, will review after gibi's patch.
15:53:38 jackding jaypipes: thanks
16:06:37 tssurya mriedem: also here https://review.openstack.org/#/c/637217/ I am not entirely sure why we need to consider the non-zero standard classes in the extra_specs
16:07:42 mriedem i don't think we're going to
16:07:56 tssurya ok good
16:08:09 mriedem we want to check if the standard resource class key exists in extra specs and if it's already 0, we consider that instance migrated
16:08:29 mriedem if it's non-zero, then i guess we log a warning and still consider it migrated
16:08:35 tssurya yea that's what I throught too, I was just reading jroll's and efried_pto's comments on it
16:08:46 tssurya which implied otherwise
16:09:11 tssurya ah a warning! ok then
16:09:44 mriedem note that on stable branches (at least stable/rocky) that calculation probably needs to account for report_ironic_standard_resource_class_inventory
16:09:54 mriedem which makes it even more complicated
16:10:01 tssurya yea saw that part :(
16:12:20 gibi jaypipes: it is not me it is the problem that is the source of the hurt :)
16:13:05 openstackgerrit melanie witt proposed openstack/nova master: WIP Count instances from mappings and cores/ram from placement https://review.openstack.org/638073
16:22:04 openstackgerrit Dan Smith proposed openstack/nova master: API microversion 2.69: Handles Down Cells Documentation https://review.openstack.org/635147
16:22:06 dansmith mriedem: tssurya see what you think of that ^
16:22:57 dansmith I was tempted to rename it as well to just say "Handling partial infrastructure failures" or something
16:23:31 tssurya dansmith: I don't mind at all :) feel free to change anything in that patch
16:25:35 jaypipes gibi: :)
16:26:32 openstackgerrit Merged openstack/nova master: Address nits from I9e30a24a4c0640f282f507d0a96640d3cdefe43c https://review.openstack.org/636967
16:34:08 mriedem tssurya: i have the novaclient test sorting thing fixed
16:34:19 mriedem i'll post that as a patch before yours in the series
16:34:38 openstackgerrit Balazs Gibizer proposed openstack/nova master: nova-manage: heal port allocations https://review.openstack.org/637955
16:34:38 openstackgerrit Balazs Gibizer proposed openstack/nova master: cache neutron ports in heal allocation https://review.openstack.org/638207
16:43:15 melwitt mriedem: we have an ironic driver fix proposed that adds a kwarg to the get_info() virt method. we wanted your opinion about whether it's backportable and if not, advice on another way to approach it https://review.openstack.org/636699
16:57:46 mriedem well that's a backward incompatible change to any out of tree drivers on stable
16:57:51 mriedem so it depends on if you care about that
16:57:57 mriedem efried_pto probably cares about it
16:58:50 openstackgerrit Matt Riedemann proposed openstack/python-novaclient master: API microversion 2.69: Handles Down Cells https://review.openstack.org/579563
16:58:51 openstackgerrit Matt Riedemann proposed openstack/python-novaclient master: Make Server.networks use a predictable sort order https://review.openstack.org/638214
16:59:12 melwitt a new kwarg with a default value? I didn't think that's backward incompat
17:01:03 tssurya mriedem: oh thanks a lot!
17:01:22 tssurya and as far as the host name being "host1" always its because the fake has that hardcoded
17:01:34 tssurya I can just change that if you want them to be unique
17:02:31 mriedem melwitt: if compute is passing use_cache=False to an out of tree driver that implements get_info(instance) but doesn't have that kwarg in the method signature, it's going to blow up
17:02:54 melwitt oh, right. urgh
17:03:34 mriedem http://paste.openstack.org/show/745484/
17:04:05 mriedem so the worst that happens is out of tree drivers don't have the sync power states task working until they fix that
17:04:53 melwitt jroll ^
17:05:49 jroll melwitt: yep, that's correct, I just didn't know what the policy was given out of tree drivers are unsupported
17:06:29 jroll I'm not sure there's a different approach we can take, am open to it though
17:06:54 jroll other than reverting the thing that made the sync use the cache, and spamming ironic for that task
17:07:06 mriedem i'm commenting
17:07:11 mriedem cool your jets and hold your horses please
17:07:14 mriedem oath can wait
17:07:31 jroll it's not even for oath, larsks reported this bug and I hacked it up ¯\_(ツ)_/¯
17:07:42 jroll besides oath is dead, long live verizon media :P
17:07:47 mriedem oh right
17:07:58 mriedem vmg 4ever
17:08:14 openstackgerrit Stephen Finucane proposed openstack/os-vif master: WIP: Add API docs for various VIF types https://review.openstack.org/637009
17:10:32 mriedem jroll: posted
17:10:50 mriedem my suggestion is handling a TypeError in this change if you're going to backport it, and in a follow up remove the TypeError handling
17:13:24 jroll mriedem: oh, that's a good idea, thanks. I'll update with that
17:16:49 mdbooth sean-k-mooney: That port binding error I mentioned this morning could be a more serious thing. I'm currently messing with LM and generating lots of LM failures, and in every case it's not possible to repeat the LM because:
17:16:51 mdbooth $ openstack server migrate --live devstack-2.marston --block-migration test1
17:16:51 mdbooth Migration pre-check error: Binding failed for port 48c1c3b8-be35-46a0-8df2-60495b4c6b9d, please check neutron logs for more information. (HTTP 400) (Request-ID: req-3217c16e-438a-4025-a462-8979fc97bd80)
17:16:58 mriedem jroll: you may also want to take the opportunity to refresh the node_cache but that would be a follow up
17:17:37 mriedem mdbooth: is the vif_type on the port binding "binding_failed" or "unbound"?
17:17:52 mriedem if so, we don't know how to translate that to an os-vif object to plug the vif on the dest host
17:17:54 mriedem and it blows up
17:18:00 jroll mriedem: eh, that's a bigger timesink that I'm not sure I want to do in the middle of that loop... though would help in the case of a whole rack getting unplugged or something
17:18:27 mriedem mdbooth: semi related https://review.openstack.org/#/c/603844/
17:18:53 mdbooth mriedem: Well it remains bound on the source, so it looks normal I guess?
17:18:54 mriedem mdbooth: if the vif_type in the port binding is busted, you might have to detach and re-attach the port to the server to fix it
17:19:09 mriedem mdbooth: check the port binding details on the port in the neutron api
17:19:16 mriedem you probably have to be admin to get that field
17:19:35 mriedem binding:vif_type is what you're looking for
17:19:42 mdbooth It's 'ovs'
17:19:53 mriedem oh huh
17:20:03 mriedem well actually,
17:20:12 mriedem is this rocky?
17:20:15 mdbooth mriedem: This is a dark art to me, btw :)
17:20:17 mdbooth devstack master
17:20:36 mriedem the port binding for the dest host will be on GET /ports/{port_id}/bindings/{dest_host}
17:20:47 mriedem the port binding details you're looking at on the port are likely the source host
17:20:53 mriedem which is the 'active' binding
17:20:58 mriedem the dest host binding is the inactive one
17:21:02 mriedem and it's what you're trying to plug
17:21:10 mdbooth Ah, ok.
17:21:19 mdbooth No cli for that?
17:21:25 sean-k-mooney mdbooth: it was in the spec but no
17:21:26 mriedem unfortunately there is no api ref for the port bindings api yet
17:22:01 mdbooth I'm assuming if I look at the LM code in nova we're failing to clean that up
17:23:01 sean-k-mooney mdbooth: to clean up a host binding?
17:23:51 mriedem _rollback_live_migration should delete inactive dest host port bindings
17:23:53 sean-k-mooney mdbooth: i had tought we had code for that
17:24:21 mriedem https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L6958
17:24:39 mriedem https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L7010
17:25:12 mriedem if we failed to cleanup the dest host port bindings where should be an error in the compute logs
17:27:41 mdbooth Hmm, nothing obvious
17:28:08 mdbooth Anyway, I'm going to have to shoot. I'll investigate properly in the morning, was just sharing in case sean-k-mooney had immediate insight.
17:28:25 mdbooth Thanks for the help.
17:30:55 mriedem dansmith: commented https://review.openstack.org/#/c/635147/
17:38:04 sean-k-mooney mdbooth i know that i have seen case where cleaning up the volume on a roleback fails and we raise an excpetion and dont then rollback the networking
17:39:29 sean-k-mooney mdbooth: my guess is that is the most likely cause. for example if self.compute_rpcapi.remove_volume_connection rased an exception we would never call self.network_api.setup_networks_on_host
17:40:43 sean-k-mooney mdbooth: we probaly should split _rollback_live_migration into several smaller fucntion and call them with a set fo try finally statement to ensure that all resouces that can be rolled back are
17:42:25 sean-k-mooney mdbooth: ill be fixing stuff arount this code for something else next sprint so i might look at doing that change then.

Earlier   Later