Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-02-15
21:56:08 mriedem heh i figured
21:59:16 imacdonn mriedem: FYI, I have been able to reproduce the issue where a rescheduled instance fails with a PortInUse exception (https://pastebin.com/6HAL944i)... on Ocata .. I guess I'll try on Pike next (in spare time)
22:02:58 mriedem dansmith: while i do that, mayhap you'd be so kind as to review these simple docs changes https://review.openstack.org/#/c/544090/ (and the one below it) so i can backport those to queens as well
22:05:21 dansmith mriedem: mayhap done
22:05:30 mriedem great
22:06:00 mriedem do you feel sufficiently up to date on volume multiattach now?
22:08:59 dansmith oh totes
22:13:04 mriedem dansmith: how do you feel about this one for queens? https://review.openstack.org/#/c/543489/
22:13:06 mriedem RC2 i mean
22:13:39 dansmith idk, I'd like to see those backported, but they don't _have_ to be in rc2
22:14:01 mriedem if it's low impact/low risk i think it's ok
22:14:25 belmoreira dansmith thanks for https://review.openstack.org/#/c/544585 I added few comments to clarify how we use the cell-scheduler in cellsV1
22:16:24 dansmith belmoreira: I think you're probably going to have to be willing to tweak some of your current mappings to make things work,
22:16:34 dansmith but yes, we can have more filters with more behaviors,
22:17:02 dansmith and/or tweak the stuff I have proposed
22:17:15 dansmith belmoreira: https://review.openstack.org/#/c/545002/
22:17:56 dansmith belmoreira: I would like to understand more about why aggregates are more complicated at your scale, as they're pretty much intended to simplify stuff at higher scale
22:18:11 openstackgerrit Merged openstack/nova master: Cleanup the manage-volumes admin doc https://review.openstack.org/544066
22:18:14 dansmith maybe we can hear from you/surya in dublin about it
22:18:28 openstackgerrit Merged openstack/nova master: Add admin guide doc on volume multiattach support https://review.openstack.org/544090
22:18:50 mriedem dansmith: ok +2s on the stuff that i thought was fine; https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:stable/queens - mine that aren't -W are up for review also, since i can't +2 those; i also didn't do takashi's for the placement global req id since they are rather large, but could maybe convinced otherwise
22:20:52 openstackgerrit Matt Riedemann proposed openstack/nova stable/queens: Cleanup the manage-volumes admin doc https://review.openstack.org/545141
22:20:52 openstackgerrit Matt Riedemann proposed openstack/nova stable/queens: Add admin guide doc on volume multiattach support https://review.openstack.org/545142
22:20:53 mriedem and these ^
22:21:05 dansmith okay let me hit your +2s first
22:22:52 dansmith ah yeah all those +2s other than the UC ones are things I had reviewed on master
22:24:52 mnaser melwitt: mriedem so i guess just updating the commit message remains but i'll leave that for melwitt i think she'll do a better job than i will :p
22:27:15 belmoreira dansmith sure, I'm willing to tweak the current mappings to work with the new scheduler schema. I just wanted to describe how we currently do it for reference
22:27:31 dansmith belmoreira: okay, definitely appreciate that :)
22:27:40 mriedem belmoreira: hard-coding projects in config is a bit odd
22:27:47 mriedem https://gitlab.cern.ch/belmiro/cellsv1-filters/blob/master/target_cell_project.py#L29
22:29:59 belmoreira mriedem I agree, that was the initial approach. Now the cell mapping is a property of the project. Just didn't remove that old code just in case...
22:31:04 mriedem oh yeah i see now
22:31:05 mriedem cells_mapping = client_key.get_cells_mapping(instance_project_id)
22:31:13 mriedem if config, use those, else get the mappings from keystone
22:31:50 belmoreira mriedem it was kept for the transition phase, when not all the projects had the cell mapping as a property
22:32:29 belmoreira I have >3000 projects :)
22:34:25 mriedem belmoreira: i assume the complexity with host aggregates at that scale is that a lot of hosts are in a lot of different aggregates
22:34:54 mriedem how much of the aggregate management is automated?
22:35:13 belmoreira dansmith mriedem also because is very difficult to automate add/remove nodes
22:35:20 dansmith why
22:35:21 dansmith ?
22:35:31 dansmith (is it difficult)
22:35:53 belmoreira using a configuration management tool is not that safe/easy for this operations
22:36:00 dansmith sure, but this is done via the api
22:36:46 belmoreira that is one of the main reasons. The conf tool will need to have credentials for these operations
22:37:02 mriedem removing nodes is a problem i can see, there is no api for that really - we have delete service, but that doesn't cleanup the compute_nodes table entry
22:37:26 dansmith belmoreira: oh okay, I guess my point was .. maybe using conf management for this is not a good idea ;)
22:38:29 dansmith mriedem: removing nodes what? you can remove a node from an aggregate via the api
22:38:43 mriedem i assumed he was talking about dropping hosts
22:38:54 mriedem not add/remove aggregate members
22:39:09 dansmith hmm
22:39:35 belmoreira mriedem was talking about aggregate management using a conf management tool
22:39:38 mriedem which spurred the discussion the other day about delete_cell needing a --ignore-placement flag or whatever
22:39:53 mriedem ok, ignore me then, i'm talking about something else
22:39:59 mriedem 'decomissioning' nodes
22:41:13 belmoreira that is a one time operation. I'm not that worry if is not full automated
22:42:23 dansmith belmoreira: are you going to be in dublin or just tssurya?
22:42:50 belmoreira dansmith just tssurya
22:42:57 dansmith okay
22:42:59 belmoreira I will be Vancouver
22:43:06 dansmith okay
22:44:19 belmoreira my previous point was that if for a specific use case we need to create aggregates/add nodes it will be very difficult to manage this at scale
22:45:14 melwitt mriedem: replied on https://review.openstack.org/#/c/340614 about instance.host = None + vm_state = ERROR
22:46:13 belmoreira especially because aggregates and cells need to be in sync. Missing a node in this mapping could mean that it will not be used even if it's in the cell
22:47:05 openstackgerrit Arvind Nadendla proposed openstack/nova-specs master: Support traits in Glance https://review.openstack.org/541507
22:48:57 mriedem edmondsw: efried: done https://review.openstack.org/#/c/545111/
22:49:06 efried mriedem: Thanks!
22:53:32 mriedem melwitt: we expect that in a *very specific scenario*
22:53:59 mriedem the commit message makes it sound like the instance is ever only in error state because of a failed build, which is not the case
22:56:30 melwitt okay, so the words should be "we can expect" instead of "we expect"
22:56:54 mriedem let me get cochran on the horn
22:57:55 melwitt "on the horn" means phone? that's new to me
22:58:04 mriedem yes
22:59:04 mriedem did you get the cochran reference at least?
22:59:27 melwitt like johnny cochran?
22:59:30 mriedem yes
22:59:37 melwitt yeah
23:00:20 mriedem how about just saying, "If the instance is in ERROR because of a failed build"
23:00:49 mriedem my council informs me that would be satisfactory
23:01:28 melwitt thank you, council people
23:08:39 imacdonn mriedem: I reproduced the PortInUse thing on Pike, and also confirmed it only happens when using a pre-existing port - created https://bugs.launchpad.net/nova/+bug/1749838
23:08:39 openstack Launchpad bug 1749838 in OpenStack Compute (nova) "Rescheduled instace with pre-existing port fails with PortInUse exception" [Undecided,New]
23:10:48 mriedem ok
23:12:11 mriedem imacdonn: this is latest stable/pike?
23:12:44 imacdonn It's RDO - nova 16.0.3
23:12:55 mriedem ok so port.device_id is set
23:13:01 mriedem is what it's failing on
23:13:39 mriedem when we unbind the port before rescheduling, we should wipe that out
23:13:40 mriedem port_req_body = {'port': {'device_id': '', 'device_owner': ''}}
23:14:06 mriedem https://github.com/openstack/nova/blob/stable/pike/nova/network/neutronv2/api.py#L511
23:14:21 mriedem do you see "Unable to clear device ID" in the logs?
23:14:23 mriedem on the first host?
23:14:39 imacdonn will look .. I did attach debug lots from both nodes to the bug just now
23:14:43 imacdonn logs*
23:15:55 imacdonn I don't see that message (grep for "clear" only finds a couple of config options)
23:16:28 mriedem i feel like i had a patch that added debug logging in this code for when we tore down, but i probably abandoned it
23:17:15 mriedem imacdonn: oh i bet this is the fix you need https://review.openstack.org/#/c/520248/
23:18:01 mriedem in the case of nova creating a port,
23:18:12 mriedem it doesn't fail because nova orphans the port created from the first host, and creates a new port when going through the 2nd host

Earlier   Later