Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-09-28
16:17:13 johnthetubaguy yeah, each is offering 1: CUSTOM_GOLD
16:17:26 efried Seems like that ought to happen regardless, otherwise the scheduler will think it's still there and may try to schedule stuff to it.
16:17:35 johnthetubaguy so for that bug
16:17:40 efried yeah, for that bug
16:17:46 johnthetubaguy the dead thing would normally be responsible for the delete of it
16:17:59 efried That... doesn't make sense. It's dead. It can't call the placement API.
16:18:10 johnthetubaguy right, that's the problem
16:18:25 efried At some level, the new guy must know he's taking over for the old guy.
16:18:39 efried So why can't he delete the old guy's RP?
16:18:49 johnthetubaguy not sure if it sees that edge
16:18:59 johnthetubaguy it just sees a new node right now
16:19:09 johnthetubaguy but not 100% sure on that
16:19:10 efried Well.
16:19:23 efried Is there a split-brain possibility here?
16:19:44 cfriesen johnthetubaguy: for virtual machines (like with a vmware backend) nova-compute runs on the "node", and the hypervisor runs on the "host", right? why did ironic reverse the naming?
16:21:20 dansmith cfriesen: no, the reverse
16:21:30 dansmith cfriesen: but vmware doesn't actually run like that
16:21:34 dansmith (anymore)
16:21:41 dansmith well,
16:22:04 dansmith it doesn't expose a 1:N host:node structure I mean
16:22:05 dansmith it still runs external I think, but...
16:23:14 dansmith cfriesen: nova-compute always runs on the "host" which is where the service record is (nova-compute being a service) and the "node" is the actual hypervisor
16:23:36 dansmith ironic runs one nova-compute for multiple "hypervisor" nodes, which can't by definition run the service on them
16:24:05 dansmith ironic is the _reason_ we have that (blasted) model so it's setting the standard, not doing something reverse of what the model was designed for
16:27:45 cfriesen dansmith: got it, thanks. wonder where I got flipped around.
16:28:12 dansmith cfriesen: it's unfortunately very convoluted for little gain :(
16:30:42 cfriesen so how does "compute_node = self._get_compute_info(context, self.host)" make any sense then, if there could be multiple compute nodes per host?
16:30:59 dansmith cfriesen: is that in the resource tracker?
16:31:44 cfriesen nova/compute/manager.py
16:31:56 dansmith cfriesen: oh, look at the function
16:31:58 dansmith cfriesen: it returns the first
16:32:06 dansmith "for old compat"
16:32:35 dansmith and since ironic doesn't support live migration...YET
16:38:14 openstackgerrit Jay Pipes proposed openstack/nova master: placement: set/check if inventory change in tree https://review.openstack.org/470575
16:38:15 openstackgerrit Jay Pipes proposed openstack/nova master: placement: integrate ProviderTree to report client https://review.openstack.org/415921
16:38:15 openstackgerrit Jay Pipes proposed openstack/nova master: placement: add nested resource providers https://review.openstack.org/377138
16:38:16 openstackgerrit Jay Pipes proposed openstack/nova master: placement: allow filter providers in tree https://review.openstack.org/377215
16:38:16 openstackgerrit Jay Pipes proposed openstack/nova master: placement: adds REST API for nested providers https://review.openstack.org/384807
16:38:17 openstackgerrit Jay Pipes proposed openstack/nova master: placement: update client to set parent provider https://review.openstack.org/385693
16:43:04 mriedem stvnoyes: the multinode live migration grenade job failed on your live migration new style attach patch, but it failed here http://logs.openstack.org/87/463987/20/check/gate-grenade-dsvm-neutron-multinode-live-migration-nv/ae8875f/logs/subnode-2/screen-n-cpu.txt.gz?level=TRACE#_Sep_26_14_28_11_637958
16:43:25 mriedem and looking at logstash, that happens on a ton of patches, so it's probably just a 100% failure in that job right now
16:43:42 dansmith jaypipes: I'm missing the reasoning for the complicated "compare two dicts" method in that first patch
16:44:38 jaypipes dansmith: yeah, it's a long conversation on previous revisions... I'm going to change it to only compare the total field value and ignore the other things like reserved/min_unit/allocation_ratio, etc
16:45:14 dansmith jaypipes: oh, sorry I can re-read.. but, is this not to be used for "know when I need to report new inventory" sort of thing?
16:45:18 jaypipes dansmith: the reason it's like that is because we still don't have any consistency and agreement on who (virt driver or resource tracker) owns various fields about the inventory like alloc ratio etc
16:45:28 efried dansmith The history is here: https://review.openstack.org/#/c/470575/2/nova/compute/provider_tree.py@98
16:46:08 dansmith jaypipes: ah, to avoid us reporting new just because the virt driver thinks the allocation_ratio should be different?
16:46:21 jaypipes yup
16:47:05 dansmith hmm
16:47:16 dansmith the virt driver needs to report total and reserved
16:47:42 dansmith but allocation_ratio is clearly the purview of the compute manager
16:47:56 dansmith min_unit is probably virt driver I guess
16:48:15 dansmith taking the scaleio example of 8GB minimum slice or whatever
16:51:06 mriedem stvnoyes: yeah the grenade live migration job is just totally busted http://tinyurl.com/y9ca6zox
16:51:53 mriedem something changed in august http://tinyurl.com/yby7m5p8
16:52:27 stvnoyes that's pretty bad
16:53:45 efried johnthetubaguy jaypipes Where does ironic (or ftm any virt driver) set up its resource providers today?
16:53:59 jaypipes efried: it doesn't. the RT does.
16:54:06 jaypipes efried: because there's only one RP.
16:54:32 mriedem stvnoyes: looking at http://logs.openstack.org/87/463987/20/check/gate-grenade-dsvm-neutron-multinode-live-migration-nv/ae8875f/console.html#_2017-09-26_14_28_03_894125
16:54:37 efried jaypipes In ironic, only one RP per compute host? Not one RP per ironic node?
16:54:41 mriedem tempest.api.compute.admin.test_live_migration.LiveAutoBlockMigrationV225Test.test_live_block_migration [22.911966s] ... ok
16:54:50 mriedem tempest.api.compute.admin.test_live_migration.LiveAutoBlockMigrationV225Test.test_live_block_migration [22.911966s] ... ok
16:54:55 mriedem oops
16:55:02 jaypipes efried: we want to get to where there is a sort of mediation/negotiation between the RT and the virt driver and the generic device manager as each of them "processes" the ProviderTree and sets inventory records and traits based on their own information.
16:55:03 mriedem tempest.api.compute.admin.test_live_migration.LiveMigrationTest.test_live_block_migration [10.052345s] ... FAILED
16:55:17 mriedem so it works with the 2.25 microversion, where we pass block_migration=auto
16:55:21 mriedem but fails before that
16:55:22 jaypipes efried: in ironic, each baremetal Ironic node is its own RP in placementy.
16:55:48 mriedem so i bet something with the auto changes broke how we do live block migration between n-1 hosts
16:56:25 stvnoyes what were the auto changes?
16:56:26 efried jaypipes Aha. Then bug 1714248 exists because... the compute host is creating its own separate RP for some reason?
16:56:28 openstack bug 1714248 in OpenStack Compute (nova) "Compute node HA for ironic doesn't work due to the name duplication of Resource Provider " [High,Confirmed] https://launchpad.net/bugs/1714248
16:56:53 jaypipes efried: not sure, lemme read
16:57:18 efried oh, or because the RT is trying to create a new RP for the node it's taking over...
16:57:29 jaypipes efried: note that nova-compute was *never* intended for "HA setups"...
16:57:29 mriedem stvnoyes: https://docs.openstack.org/nova/pike/reference/api-microversion-history.html#maximum-in-mitaka
16:57:39 mriedem ^ predates the spike in failures though, which started around 8/18
16:57:48 efried ...And that's out of the control of ironic-specific code...
16:58:28 efried which is why my comment #5 is a non-starter
16:58:42 efried until, as you say, there's some coordination between RT and virt to manage the RPs.
16:59:00 stvnoyes mriedem: brb- grabbing lunch
17:00:53 mriedem stvnoyes: ok. it also fails on the pike compute...
17:04:22 johnthetubaguy efried: I have to head out now and cook my dinner, but I think this maps nicely to how we setup traits, I am leaning towards ironic doing things in placement
17:05:02 efried johnthetubaguy Okay. I've got a passel of draft comments on your spec, which should be ready for your perusal next time you're on.
17:05:13 johnthetubaguy cool, thanks
17:06:12 dansmith jaypipes: in the db migration, why aren't we referencing the parent provider by id instead of uuid?
17:06:36 dansmith jaypipes: and, isn't recording the root just going to limit us later when we need to restructure a tree?
17:06:39 efried johnthetubaguy (Looking back over 'em, removing the ones we've talked about in here, the only things left are typos ):
17:07:39 jaypipes dansmith: a good question on the parent provider UUID thing. not entirely sure why I did that.
17:09:22 dansmith jaypipes: I can't tell you how elated I feel -1ing a db schema change of yours for performance reasons
17:09:34 jaypipes dansmith: recording the root is an optimization to avoid needing to do hierarchical queries. and we don't restructure the root, only potentially children within the tree (in other words, root_provider_id won't change.
17:09:41 jaypipes dansmith: :)
17:10:05 dansmith jaypipes: right, it won't for compute nodes, but it could for other types of resources
17:10:22 dansmith jaypipes: like you move a disk shelf from one NAS device to another
17:11:08 jaypipes dansmith: possibly, sure, but those kinds of moves are few and far between in comparison to the 10 or 100 times as many read requests for tree data
17:11:44 dansmith hmm
17:13:33 dansmith jaypipes: well, I commented for later
17:13:45 jaypipes okey dokey
17:14:24 dansmith jaypipes: I haven't gotten to how we tell placement about our parent, but presumably we're forbidden from trying to reparent a first-level provider?

Earlier   Later