Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-02-15
22:37:02 mriedem removing nodes is a problem i can see, there is no api for that really - we have delete service, but that doesn't cleanup the compute_nodes table entry
22:37:26 dansmith belmoreira: oh okay, I guess my point was .. maybe using conf management for this is not a good idea ;)
22:38:29 dansmith mriedem: removing nodes what? you can remove a node from an aggregate via the api
22:38:43 mriedem i assumed he was talking about dropping hosts
22:38:54 mriedem not add/remove aggregate members
22:39:09 dansmith hmm
22:39:35 belmoreira mriedem was talking about aggregate management using a conf management tool
22:39:38 mriedem which spurred the discussion the other day about delete_cell needing a --ignore-placement flag or whatever
22:39:53 mriedem ok, ignore me then, i'm talking about something else
22:39:59 mriedem 'decomissioning' nodes
22:41:13 belmoreira that is a one time operation. I'm not that worry if is not full automated
22:42:23 dansmith belmoreira: are you going to be in dublin or just tssurya?
22:42:50 belmoreira dansmith just tssurya
22:42:57 dansmith okay
22:42:59 belmoreira I will be Vancouver
22:43:06 dansmith okay
22:44:19 belmoreira my previous point was that if for a specific use case we need to create aggregates/add nodes it will be very difficult to manage this at scale
22:45:14 melwitt mriedem: replied on https://review.openstack.org/#/c/340614 about instance.host = None + vm_state = ERROR
22:46:13 belmoreira especially because aggregates and cells need to be in sync. Missing a node in this mapping could mean that it will not be used even if it's in the cell
22:47:05 openstackgerrit Arvind Nadendla proposed openstack/nova-specs master: Support traits in Glance https://review.openstack.org/541507
22:48:57 mriedem edmondsw: efried: done https://review.openstack.org/#/c/545111/
22:49:06 efried mriedem: Thanks!
22:53:32 mriedem melwitt: we expect that in a *very specific scenario*
22:53:59 mriedem the commit message makes it sound like the instance is ever only in error state because of a failed build, which is not the case
22:56:30 melwitt okay, so the words should be "we can expect" instead of "we expect"
22:56:54 mriedem let me get cochran on the horn
22:57:55 melwitt "on the horn" means phone? that's new to me
22:58:04 mriedem yes
22:59:04 mriedem did you get the cochran reference at least?
22:59:27 melwitt like johnny cochran?
22:59:30 mriedem yes
22:59:37 melwitt yeah
23:00:20 mriedem how about just saying, "If the instance is in ERROR because of a failed build"
23:00:49 mriedem my council informs me that would be satisfactory
23:01:28 melwitt thank you, council people
23:08:39 openstack Launchpad bug 1749838 in OpenStack Compute (nova) "Rescheduled instace with pre-existing port fails with PortInUse exception" [Undecided,New]
23:08:39 imacdonn mriedem: I reproduced the PortInUse thing on Pike, and also confirmed it only happens when using a pre-existing port - created https://bugs.launchpad.net/nova/+bug/1749838
23:10:48 mriedem ok
23:12:11 mriedem imacdonn: this is latest stable/pike?
23:12:44 imacdonn It's RDO - nova 16.0.3
23:12:55 mriedem ok so port.device_id is set
23:13:01 mriedem is what it's failing on
23:13:39 mriedem when we unbind the port before rescheduling, we should wipe that out
23:13:40 mriedem port_req_body = {'port': {'device_id': '', 'device_owner': ''}}
23:14:06 mriedem https://github.com/openstack/nova/blob/stable/pike/nova/network/neutronv2/api.py#L511
23:14:21 mriedem do you see "Unable to clear device ID" in the logs?
23:14:23 mriedem on the first host?
23:14:39 imacdonn will look .. I did attach debug lots from both nodes to the bug just now
23:14:43 imacdonn logs*
23:15:55 imacdonn I don't see that message (grep for "clear" only finds a couple of config options)
23:16:28 mriedem i feel like i had a patch that added debug logging in this code for when we tore down, but i probably abandoned it
23:17:15 mriedem imacdonn: oh i bet this is the fix you need https://review.openstack.org/#/c/520248/
23:18:01 mriedem in the case of nova creating a port,
23:18:12 mriedem it doesn't fail because nova orphans the port created from the first host, and creates a new port when going through the 2nd host
23:18:24 mriedem so you end up with 2 ports for the instance that nova created even though you're only using 1
23:18:42 mriedem in the case that you bring a port, nova doesn't unbind it before rescheduling, and that's why we fail to use it on the 2nd host
23:18:52 imacdonn seems plausible ... looks like a one-line change, so I can try to drop it in
23:18:55 mriedem imacdonn: you could try applying that patch and see if it resolves it
23:35:27 imacdonn mriedem: yup, that fixed it - thanks!
23:46:03 openstackgerrit Takashi NATSUME proposed openstack/nova master: List/show all server migration types (1/2) https://review.openstack.org/430608
23:46:37 openstackgerrit Takashi NATSUME proposed openstack/nova master: List/show all server migration types (2/2) https://review.openstack.org/459483
23:54:52 cfriesen does anyone know offhand what the rules are for using instance.info_cache.network_info vs calling self.network_api.get_instance_nw_info(context, instance) ?
#openstack-nova - 2018-02-16
00:03:12 tonyb mriedem: any chnace you can take a quick look at: https://review.openstack.org/#/c/543348 ?
00:09:21 openstackgerrit Nicolas Bock proposed openstack/nova stable/pike: Fix SUSE Install Guide: Placement port https://review.openstack.org/545167
00:11:42 melwitt mnaser, mriedem: I'm going through your comments on https://review.openstack.org/#/c/340614/18/nova/compute/api.py@2055 and I'm not seeing what the 'available' volume scenario is ...
00:13:00 melwitt that is, even back in liberty, if the instance failed on the host, _cleanup_volumes in the compute manager would have called volume_api.delete, so maybe that's why the detach would fail in the compute api local delete ... but that doesn't add up with having a volume left in 'available' state
00:13:58 mnaser melwitt: in that case you are right there wouldn’t be any possible scenario where you would end up in an available volume if it failed in compute
00:14:06 melwitt so I'm not really sure what case the new except block is handling
00:14:39 melwitt (I didn't add that part but back when this was going on in liberty, an operator we were working with added that saying they ran into it when they were testing the patch)
00:15:30 melwitt I'm trying to figure out whether to nix it from this patch or if there's some scenario I'm missing where we need to handle it like this
00:16:04 mnaser melwitt: I just got home so I’m on mobile but can you boot from volume with an existing volume that’s also delete on terminate?
00:16:32 mnaser That would be a scenario where you’d want us to delete it, assuming scheduling failed and we want to follow through and delete the volume as requested
00:16:47 mnaser Though that might be super confusing to the user and unexpected
00:20:03 cfriesen mnaser: in that case wouldn't the instance sit around in an ERROR state with the volume still attached?
00:20:26 cfriesen ie the volume should stay attached until the instance is deleted
00:21:34 melwitt mnaser: cfriesen it was that way until very recently https://review.openstack.org/#/c/528385/
00:23:04 mnaser Will the volume ever end up on attached state? I think it would be in attaching because it never reached the compute note to finish the attach
00:27:34 openstackgerrit Merged openstack/nova stable/queens: Refine waiting for vif plug events during _hard_reboot https://review.openstack.org/542738
00:27:40 openstackgerrit Merged openstack/nova stable/queens: Don't JSON encode instance_info.traits for ironic https://review.openstack.org/545037
00:27:48 openstackgerrit Merged openstack/nova stable/queens: Use correct arguments in task inits https://review.openstack.org/544109
00:28:00 openstackgerrit Merged openstack/nova stable/queens: Update UPPER_CONSTRAINTS_FILE for stable/queens https://review.openstack.org/542657
00:28:05 melwitt mnaser: yeah, you're right, in the scenario you're describing, BFV with already existing volume + delete_on_termination, if scheduling fails, the volume would never have been attached
00:28:58 melwitt so it would be available, so detach would fail, so we would want to delete the volume.
00:29:37 melwitt er, detach might just do a no-op if available, but I'm not sure on that
00:31:37 mnaser melwitt: wouldn’t it actually be in “attaching” state because the API layer reserves the volume
00:32:00 melwitt mnaser: oh, yeah good point. it will be whatever the reserve did
00:32:52 mnaser melwitt: yeah so it would have an attachment or reservation depending on flow, same issue we’re trying to resolve here
00:34:06 mnaser melwitt: I do think the extra try except added in this patch is useless if detach is noop
00:34:39 mnaser If detach is noop, nothing will happen. If it fails because the volume doesn’t exist, then we won’t try to delete it anyways
00:34:53 melwitt yeah. that's where I'm at too
00:35:22 mnaser Now put multiattach in that equation lol..
00:35:27 melwitt but at least I think now I understand why it was added long ago, if detach didn't used to do a no-op if not attached
00:36:11 mnaser Yeah it’s a bit of old code
01:51:36 openstackgerrit Merged openstack/nova stable/queens: Cleanup the manage-volumes admin doc https://review.openstack.org/545141
01:51:46 openstackgerrit Merged openstack/nova stable/queens: Add admin guide doc on volume multiattach support https://review.openstack.org/545142
02:29:50 mriedem1 cfriesen: the method on the instance is just a helper method
02:30:02 mriedem melwitt: mnaser: did you sort out the available volume thing?
02:30:51 mriedem tonyb: done
02:32:27 tonyb mriedem: Thanks.
02:34:23 mriedem imacdonn: if you can, leave a comment on the patch that fixed the issue for you to support it's greatness

Earlier   Later