Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-09-28
14:33:20 mdbooth Hehe. It would be slightly simpler to leave it alone, of course. I just wonder if it wouldn't be easier for users.
14:35:49 johnthetubaguy edleafe: great updates for the alternate hosts spec, just have a question on the follow up to gibi around things other than build
14:36:13 johnthetubaguy edleafe: there are quite a few users of select_destinations, its worth having at least a note about the approach there
14:36:42 openstackgerrit sahid proposed openstack/nova master: pci: update PciDevice object field 'address' to accept NULL https://review.openstack.org/508175
14:36:43 openstackgerrit sahid proposed openstack/nova master: pci: generalize object unit-tests for different framework https://review.openstack.org/508177
14:36:43 openstackgerrit sahid proposed openstack/nova master: pci: add for PciDevice object new field mdev https://review.openstack.org/508176
14:36:44 openstackgerrit sahid proposed openstack/nova master: pci: generalize stats unit-tests for different framework https://review.openstack.org/508179
14:36:44 openstackgerrit sahid proposed openstack/nova master: pci: add support for mdev device type request https://review.openstack.org/508178
14:36:45 openstackgerrit sahid proposed openstack/nova master: pci: add support for resource pool stats of mdev devices https://review.openstack.org/508181
14:36:45 openstackgerrit sahid proposed openstack/nova master: pci: add support for mdev devices type devspec https://review.openstack.org/508180
14:36:46 openstackgerrit sahid proposed openstack/nova master: libvirt: update PCI node device to report mdev devices https://review.openstack.org/508183
14:36:46 openstackgerrit sahid proposed openstack/nova master: pci: make manager to accept handling mdev devices https://review.openstack.org/508182
14:36:47 openstackgerrit sahid proposed openstack/nova master: libvirt: add support to start vm with using mdev (vGPU) https://review.openstack.org/508185
14:36:47 openstackgerrit sahid proposed openstack/nova master: libvirt: report mdev resources https://review.openstack.org/508184
14:36:48 openstackgerrit sahid proposed openstack/nova master: libvirt: resuse SRIOV funtional tests for MDEV devices https://review.openstack.org/508187
14:36:48 openstackgerrit sahid proposed openstack/nova master: functional: rework fakelibvirt host pci devices https://review.openstack.org/508186
14:38:40 openstackgerrit Matt Riedemann proposed openstack/nova master: What is the meaning of....recreate? https://review.openstack.org/508190
14:39:01 mriedem johnthetubaguy: edleafe: good point - note that the only other place we do reschedules is cold migrate/resize
14:39:17 mriedem evacuate, unshelve, live migration don't do the reschedule dance between compute and conductor,
14:39:25 mriedem live migration does a reschedule dance of it's own, but that happens in super conductor
14:39:35 edleafe johnthetubaguy: ok, but I'm not sure as to the depth you would like, beyond stating that the return from select_destinations will change
14:40:02 mriedem edleafe: it's probably worth calling out (1) where we rely on reschedules and (2) that those paths will need to be aware of this change
14:40:10 mriedem so build, resize/migrate, live migrate
14:40:32 mriedem and we should probably be sure to have functional tests for hitting those reschedule flows
14:40:59 edleafe mriedem: yeah - they'll need to adapt to handle the different return value, but they don't need to change what they do with it
14:41:09 mriedem the good news is i think we already do have functional tests for reschedules with those 3 flows now
14:41:37 edleafe they *can* change, but that's out of scope for this
14:41:40 mriedem cdent: i thought you might like the philosophical tone of https://review.openstack.org/#/c/508190/
14:41:55 cdent heh
14:42:46 cdent “The meaning is, don't ask.” is my new bumper sticker, t-shirt, tattoo
14:43:06 cdent this ^ is very exciting
14:47:11 johnthetubaguy edleafe: so saying its out of scope, and only minimal changes required to make the new interface work is OK I guess, but my concern is what mriedem said
14:47:15 openstackgerrit Chris Dent proposed openstack/nova master: [placement] Update the placement deployment instructions https://review.openstack.org/469048
14:49:49 mriedem i don't know what 'can change' and 'out of scope' means here
14:50:02 mriedem alternative hosts means, reschedules should work, yes?
14:50:24 johnthetubaguy mriedem: well that way my initial take, I guess you could do builds first and other stuff second
14:50:34 mriedem are we saying, we'll make build reschedules work, but resize reschedules won't?
14:51:08 mriedem live migration reschedules are ok since they happen in super conductor
14:51:10 johnthetubaguy well I was assuming that would be some follow up spec, but that does seem a bit silly to me
14:51:18 mriedem it's the compute<>conductor action that is the problem i think
14:51:24 johnthetubaguy yeah, live-migrate ones seem fine
14:52:06 edleafe mriedem: I was concerned with the boiling the ocean type of spec
14:52:40 edleafe mriedem: if you think it's wise, I can expand it to cover every use of select_destinations()
14:52:44 mriedem edleafe: ok, but we also got into a lot of hot water late in pike because the claims in the scheduler stuff didn't take into account move operations
14:53:03 mriedem so the main issue we're trying to solve is that the cell conductor can't reach the scheduler
14:53:08 mriedem there are only 2 times that happens,
14:53:09 mriedem build and resize
14:53:28 mriedem live migrate reschedules happen in super condcutor which can reach the scheduler, so that's fine - you could make a note of it as something we know about and don't need to change
14:53:30 edleafe mriedem: ok, I'll dig into resize, and add some stuff about that
14:53:39 johnthetubaguy evacuate or shelve? not sure if they retry at all?
14:53:44 mriedem johnthetubaguy: they don't
14:53:52 johnthetubaguy OK
14:54:05 mriedem prep_resize on the compute will call back to resize_instance in the conductor, which calls migrate_server, which calls the scheduler for a new destination
14:54:15 mriedem so that is the flow, besides build, that has to also be fixed
14:54:47 johnthetubaguy OK, so if its only two, we should do them together I think
14:54:51 mriedem after 5 years working in nova, i think i finally have these conductor flows memorized
14:55:04 johnthetubaguy they changed after I last did that
14:55:12 johnthetubaguy and I slept since then
14:55:21 cdent you sleep?
14:55:33 johnthetubaguy yeah, I know, old school
14:56:02 edleafe mriedem: I don't know conductor flows that clearly, but that sounds like a very different problem than retries
14:56:20 johnthetubaguy its the same compute -> wrong conductor right?
14:56:25 edleafe mriedem: that sounds like the entire flow needs to change, and alternate hosts won't address that
14:57:02 johnthetubaguy you replace a call to select_destinations to a claim the next candidate host right?
14:57:14 johnthetubaguy (wibble passing the data through)
14:57:38 edleafe johnthetubaguy: the call up happens before select_destinations, at least in mriedem's flow
14:57:42 mriedem edleafe: it's essentially the same as the build flow
14:57:49 mriedem just different methods
14:58:04 mriedem compute:build_and_run_instances calls up to conductor:build_and_run_instances
14:58:13 mriedem compute:prep_resize calls up to conductor:resize_instance
14:58:33 mriedem both of those methods in the cell conductor eventually ask the scheduler for a new dest
14:58:34 edleafe oh, you're talking about API-level compute, not cell compute
14:58:44 dansmith what is api-level compute?
14:58:46 mriedem edleafe: there is no such thing
14:59:08 edleafe mriedem: I didn't think so, but it sounded like you were
14:59:23 mriedem https://docs.openstack.org/nova/pike/user/cellsv2_layout.html#multiple-cells
14:59:34 edleafe the call up from compute isn't after select_destinations; it's before in a resize
14:59:38 mriedem in ^ the compute only has access to the cell conductor
14:59:54 mriedem i think we're talking about different things
15:00:07 edleafe yes, we are - that's what I've been trying to say
15:00:26 edleafe alternate hosts just removes the need for a retry to have to call up from the cell
15:00:37 mriedem resize flow is, summarized: api -> superconductor -> scheduler -> superconductor -> compute (reschedule) -> cell conductor -> compute (with alternate hosts)
15:00:50 edleafe in your resize flow, the problem is that the call from the cell already is happening, and needs to change
15:00:50 mriedem yes, in ^ we can't upcall from the cell conductor to the scheduler
15:00:55 mriedem hence the need to pass the alternate hosts through
15:00:58 mriedem for both build and resize
15:01:46 johnthetubaguy https://github.com/openstack/nova/blob/8a386b055c82df67092a1abc683e7225ef80671e/nova/compute/manager.py#L3847
15:01:49 openstackgerrit Dan Smith proposed openstack/nova master: Make allocation cleanup honor new by-migration rules https://review.openstack.org/498948
15:01:49 openstackgerrit Dan Smith proposed openstack/nova master: Move allocation manipulation out of drop_move_claim() https://review.openstack.org/498947
15:01:50 openstackgerrit Dan Smith proposed openstack/nova master: Revert allocations by migration uuid https://review.openstack.org/498949
15:01:50 openstackgerrit Dan Smith proposed openstack/nova master: Pre-create migration object https://review.openstack.org/498950
15:01:51 openstackgerrit Dan Smith proposed openstack/nova master: Make migration uuid hold allocations for migrating instances https://review.openstack.org/506420
15:01:51 openstackgerrit Dan Smith proposed openstack/nova master: Refactor resource tracker to account for migration allocations https://review.openstack.org/506419
15:01:52 openstackgerrit Dan Smith proposed openstack/nova master: Make live migration hold resources with a migration allocation https://review.openstack.org/507638
15:02:19 johnthetubaguy vs https://github.com/openstack/nova/blob/8a386b055c82df67092a1abc683e7225ef80671e/nova/compute/manager.py#L1880
15:02:22 johnthetubaguy seems the same flow
15:02:38 johnthetubaguy i.e. +1 mriedem
15:03:43 johnthetubaguy I guess its the second visit here that should not call the scheduler again: https://github.com/openstack/nova/blob/7cd9e3b8bb7fc0601786847f19cdf3f706ec079f/nova/conductor/tasks/migrate.py#L67
15:03:53 mriedem correct
15:04:33 mriedem just like this one https://github.com/openstack/nova/blob/7cd9e3b8bb7fc0601786847f19cdf3f706ec079f/nova/conductor/manager.py#L552

Earlier   Later