Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-08-29
15:54:28 mriedem gibi: i think i know what's going on in https://bugs.launchpad.net/nova/+bug/1781648
15:54:28 openstack Launchpad bug 1781648 in OpenStack Compute (nova) "heal_allocations test randomly failing with "ValueError: Field `compute_node_uuid' cannot be None"" [Medium,Confirmed]
15:54:40 mriedem we're racing with the caching scheduler starting up and initializing the cache of compute nodes
15:54:50 mriedem so by the time we hit the scheduler, the cache is empty
15:56:07 gibi mriedem: that sounds like a reasonable explanation. I guess I missunderstood the logs when I stated in the bug that the host creation overlaps with the scheduling
15:59:07 gibi mriedem: so I guess the fix is that we need to wait in the test a bit
16:00:43 openstackgerrit Matt Riedemann proposed openstack/nova master: Restart scheduler in TestNovaManagePlacementHealAllocations https://review.openstack.org/597571
16:00:45 mriedem sort of ^
16:02:09 mriedem i thought about also changing this https://github.com/openstack/nova/blob/master/nova/scheduler/caching_scheduler.py#L78 to "if not self.all_host_states:" to handle the empty list case, but that could still be a weird race failure if we have 1 host but not both in the cache
16:02:23 gibi mriedem: your proposed start / stop is way better than a simple sleep in the test
16:02:34 mriedem i learned it from watching you gibi
16:04:09 gibi :)
16:05:32 melwitt .
16:05:51 gibi mriedem: I'm wondering how many other functional tests uses CachingScheduler and therefore might be affected by the same race
16:06:10 mriedem gibi: some do, but they might start the scheduler after the computes, i was just starting to audit that
16:06:43 gibi mriedem: cool
16:14:37 mriedem mlavalle: smcginnis: on this cross-cell cold migration thing nova will definitely need some pre-validation of the selected host,
16:15:04 mriedem i'm wondering if simply trying to create a volume attachment/port binding on the selected target host *in another cell* would be enough to tell us if that's not going to work for storage/networking
16:15:20 mriedem or in the case of port bindings, will neutron only fuss when we try to activate the target host port binding?
16:16:26 jaypipes zigo, mriedem: we have a rough timeframe of when this began occurring (the allocation ratio 0 thing...)?
16:17:28 mriedem earliest i know of is when naichuans reported it in the ML
16:18:01 mlavalle mriedem: the port binding is the result of asking the mechanism managers if they can bind the port in the indicated host. so the binding process is what you call the pre-validation of the selected host
16:18:11 melwitt dansmith: +1 to meeting skippage
16:18:12 mlavalle mechanism drivers^^^^
16:19:38 mlavalle so when you say create_port_binding, you are asking the mechanism drivers whether any of them can bind the port in the designated host
16:20:02 mriedem mlavalle: and if the network that the port is in doesn't cover that dest host, the port binding creation should fail?
16:20:48 sean-k-mooney mriedem: it will fail of non of the ml2 driver can bind the port to the correct network
16:20:53 mlavalle mriedem: correct. the question that the drivers ask themselves is whether they can a segment of the network reachable in that host
16:21:06 mlavalle they have a segment^^^^
16:21:28 melwitt can anyone confirm whether PCI and SRIOV PCI devices are treated the same as far as the scheduler is concerned?
16:21:54 sean-k-mooney mlavalle: that segment is only relevent for provider networks with the multi segment network extention correct normmal it 1 segment per network
16:22:48 sean-k-mooney mlavalle: as in in the pci filter?
16:22:59 sean-k-mooney melwitt: ^
16:23:53 sean-k-mooney melwitt: they are more or less treated the same but they do have a different type in the pci manager (type-pci for passthough and type-PF or type-VF for sriov)
16:23:57 melwitt sean-k-mooney: maybe? just in general, for NUMA scheduling, if SRIOV PCI devices are treated differently at all or no
16:23:59 mlavalle sean-k-mooney: but regardless, today we can bind a port in multiple cells deployments, right?
16:24:14 melwitt sean-k-mooney: I see, thank you
16:25:00 sean-k-mooney melwitt: from a numa perspctive they are treated the same. we store the numa node info the same way in the db
16:25:20 sean-k-mooney mlavalle: yes i think so but never tried it
16:26:38 mlavalle sean-k-mooney: so that means that the mechanism drivers are being able to answer the question, can I bind a port here or not? in multiple port bindings, we are asking that same question, it's just that the port is also bound somewhere else
16:26:46 sean-k-mooney mlavalle: cells is a nova only ting. neutron has availablity zones. if you had a different neutron availableity zone per cell can a netowrk span neutron availablity zones
16:27:53 mriedem i would think,
16:28:02 sean-k-mooney mlavalle: well we are talking about cross cell cold migration so jsut like the live migration case we will have multiple port bindings so from a neutron point of view it should be identical
16:28:03 mriedem when we have an attached port, bound to the source host,
16:28:09 mriedem nova puts the az on the port binding information,
16:28:19 mriedem and neutron would be relying on that to know if we can bind to another host in another a
16:28:21 mriedem *az
16:28:24 mlavalle sean-k-mooney: that is what I say
16:28:24 jaypipes mriedem: I have a sneaking suspicion this patch is the cause: https://github.com/openstack/nova/commit/c9b74bcfa09d11c2046ce1bfb6dd8463b3a2f3b0
16:28:37 mriedem jaypipes: that's only in master
16:28:41 mriedem zigo is hitting this in rocky
16:28:49 sean-k-mooney mlavalle: yep i am agreeing :)
16:28:59 mlavalle sean-k-mooney: LOL
16:29:16 mriedem btw, is it strange we don't have a cross_az_attach=False thing for nova/neutron like we have for nova/cinder?
16:29:17 mlavalle sean-k-mooney: will you be in Denver?
16:29:21 sean-k-mooney mriedem: neutron does not use the nova AZ as part of port bininging just the hostname
16:29:40 sean-k-mooney mlavalle: yes i will
16:29:43 mriedem so why do we set the device_owner on the port?
16:29:46 mlavalle Great!
16:29:47 mriedem using the instance az?
16:30:42 mlavalle because you want to keep track of where it is, on the Nova side, but I am just speculating
16:30:43 sean-k-mooney mriedem: that is a good question to which i do not know the answer
16:30:54 mriedem it might be used by the neutron callback code
16:31:10 jaypipes mriedem: oh... I thought you said this only recently occurred..
16:31:26 mriedem jaypipes: it is only recently reported
16:31:35 mriedem but i also thought about that _update change but it's master only
16:31:39 mriedem and zigo said he's hitting it on rocky
16:31:55 sean-k-mooney mriedem: im pretty sure its not used by neutron at all. my geuess is this might be from nova networks and we just kept doing it
16:32:08 sean-k-mooney the AZ that is
16:32:16 jaypipes mriedem: ack
16:32:18 mriedem nova net doesn't have any kind of device_owner thing
16:32:41 mlavalle yeah, the device_owner stuff is a Neutron port concept
16:32:45 sean-k-mooney mriedem: oh i was going to specalte we might of needit to do the nova-net multhost thing
16:32:52 mriedem bingo http://git.openstack.org/cgit/openstack/neutron/tree/neutron/notifiers/nova.py#n77
16:32:57 mriedem it's what i said it was
16:33:44 mriedem among apparently a lot of other things
16:33:59 sean-k-mooney mriedem: i dont see why we need the az form that
16:34:05 mriedem we probably dont
16:34:18 mlavalle there we only use the compute prefix
16:34:46 mlavalle as we do in other places of the code
16:35:19 mriedem probably explains why https://launchpad.net/bugs/1759924 isn't that big a deal
16:35:19 openstack Launchpad bug 1759924 in OpenStack Compute (nova) "Port device owner isn't updated with new host availability zone during unshelve" [Medium,In progress] - Assigned to Matt Riedemann (mriedem)
16:35:22 mriedem except it causes confusiong
16:35:24 mriedem *confusion
16:35:38 sean-k-mooney mriedem: silvanb is still on pto but i was discussing this with him a few weeks ago about should we remove setting it or not
16:35:56 mriedem bauzas you mean?
16:36:05 sean-k-mooney mriedem: yes
16:36:26 mriedem if there is one thing sylvain loves to talk about more than cheese and skiing, it's AZs
16:36:48 mlavalle LOL
16:36:54 jaypipes mriedem: zigo's using libvirt, right?
16:36:58 mriedem yup
16:37:00 jaypipes k
16:37:01 sean-k-mooney there was concern over is allowing livemigation across availablity zones breaking the contract with a user.
16:37:23 mriedem well that reminds me of another bug fix https://review.openstack.org/#/c/567701/
16:37:44 sean-k-mooney there are some open bugs where instances with floating ips break if you do this.
16:38:08 mriedem with neutron dvr?
16:38:11 mriedem or just in general?
16:38:45 sean-k-mooney i think it was in general. i should find out i will check when i have my live migration setup running again
16:40:35 sean-k-mooney mriedem: mlavalle actully while ye are both here i added some talking points to the nova neutron cross project session since it was blank
16:40:47 sean-k-mooney https://etherpad.openstack.org/p/nova-ptg-stein line 147

Earlier   Later