| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-07-22 | |||
| 12:49:24 | boxiang_ | https://review.opendev.org/#/c/649963/ and https://review.opendev.org/#/c/651969/ Can someone help review these two patches? thanks :) | |
| 13:00:06 | efried | alex_xu: I don't understand. Why would we not use the migration consumer to migrate? | |
| 13:08:57 | sean-k-mooney | gibi_off: not sure if your are on vaction/PTO this week but just an fyi that i should have adressed your feedback in https://review.opendev.org/671338 | |
| 13:14:11 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/stein: Restore RT.old_resources if ComputeNode.save() fails https://review.opendev.org/672038 | |
| 13:14:59 | sean-k-mooney | efried: i am also confused. isnt a migration consumer proposed for use when doing cold and live migrations to hold the allocation on the second node | |
| 13:15:23 | efried | sean-k-mooney: He's off for two weeks. It looks like the changes are minor; if so another core will probably proxy his +2. | |
| 13:15:30 | efried | sean-k-mooney: that's my confusion as well. | |
| 13:15:42 | efried | perhaps we only use the migration consumer for live migration? | |
| 13:15:44 | efried | I don't know. | |
| 13:15:52 | sean-k-mooney | efried: ah ok good to know. | |
| 13:15:54 | mriedem | they're used for both cold and live | |
| 13:16:13 | mriedem | just not evacuate | |
| 13:16:19 | sean-k-mooney | mriedem: o/ welcome back, have a good vaction? | |
| 13:16:25 | mriedem | thanks, yeah | |
| 13:16:47 | sean-k-mooney | we dont use them for evac because we dont use them in rebuild | |
| 13:17:17 | mriedem | um | |
| 13:17:23 | mriedem | that's not really the reason | |
| 13:17:48 | mriedem | it's because for evac the source compute service is down, so it's not really the same flow for how the migration-based allocations are handled at the end | |
| 13:18:06 | mriedem | we have migration records for evac | |
| 13:18:25 | alex_xu | efried: if we use migration as consumer for the dst, when we confirm the resize, there isn't a call to dest host to change migration consumer back to instance consumer | |
| 13:18:35 | sean-k-mooney | right however event with a down host we could still free the allocation on the source node in placment | |
| 13:18:52 | mriedem | sean-k-mooney: we do that when the source compute service comes back up | |
| 13:19:12 | alex_xu | efried: if we use migration as consumer for the src, we will call the the dest host to claim the device, so we have no way to change the src host's claim consumer to migration. | |
| 13:19:12 | efried | alex_xu: oh, you're talking about swapping which side gets which consumer? | |
| 13:19:14 | sean-k-mooney | yes that is true. | |
| 13:19:18 | alex_xu | efried: yes | |
| 13:19:34 | mriedem | we *could* use migration-based consumers for evac, i just don't think it was really a priority when dansmith implemented that blueprint | |
| 13:20:04 | mriedem | i'm obviously walking into something late here though | |
| 13:20:09 | mriedem | who is out for 2 weeks? | |
| 13:20:13 | efried | gibi_off: | |
| 13:20:31 | efried | mriedem: but that's a separate topic | |
| 13:20:44 | mriedem | ok, which series are you talking about? | |
| 13:21:06 | efried | mriedem: we're talking about alex_xu's work to make hypervisor-specific claiming (atm for VGPUs and VPMEMs) happen in the virt driver. | |
| 13:21:23 | alex_xu | mriedem: https://review.opendev.org/#/q/status:open+project:openstack/nova+branch:master+topic:claim_for_instance | |
| 13:22:09 | mriedem | ok and this is prep work for vpmem i guess | |
| 13:22:23 | mriedem | claims do happen in the rt today | |
| 13:22:31 | mriedem | the only hypervisor-specific thing though is the overhead stuff | |
| 13:22:57 | alex_xu | mriedem: vgpu allocation has race problem also, in libvirt, we allocate vgpu out of rt, that means no lock | |
| 13:23:45 | efried | also, it just makes sense for hypervisor-specific claiming to happen in the virt driver. | |
| 13:24:31 | mriedem | alex_xu: it happens during driver.spawn correct? | |
| 13:24:41 | efried | So tldr we're adding a hook from the various *_claimZ to call into a new ComputeDriver interface which interprets allocations etc to earmark the real devices corresponding to the allocations | |
| 13:24:42 | openstack | Launchpad bug 1836204 in OpenStack Compute (nova) "The allocation of VGPU has race problem" [High,Triaged] - Assigned to Alex Xu (xuhj) | |
| 13:24:42 | alex_xu | mriedem: yes, https://bugs.launchpad.net/nova/+bug/1836204 | |
| 13:25:43 | mriedem | i guess we're not modeling each specific device in placement right? just the total number of available in inventory, and then it's up to the virt driver to pick one, which could conflict. | |
| 13:26:04 | efried | exactly | |
| 13:26:21 | mriedem | weee, so much for ever getting rid of the rt claims code... | |
| 13:26:59 | efried | The RT claims *framework* will need to stay, but IMO eventually "all" of the guts should wind up moving into this ComputeDriver.claim_for_instance | |
| 13:27:11 | efried | especially for things like PCI devices | |
| 13:27:18 | efried | and NUMA | |
| 13:27:44 | efried | the line where it makes sense to cut things over is when we move to tracking a resource in placement. | |
| 13:28:02 | efried | the messy ones obviously being VCPU, MEMORY_MB, DISK_GB | |
| 13:28:53 | sean-k-mooney | efried: well placement just keeps a tally count. if we need to do device assignment we need to continue to do tracking the the RT | |
| 13:29:03 | efried | yup | |
| 13:29:25 | efried | where "device" will eventually encompass NUMA-split proc/mem | |
| 13:29:50 | sean-k-mooney | ram and cpus are just devices in disgiuse | |
| 13:30:32 | alex_xu | I have too much fun with same host resize today | |
| 13:30:58 | efried | bbiab | |
| 13:36:47 | stephenfin | mriedem, efried: I'm hitting the boundaries of my resource tracking know how and could do with some help | |
| 13:37:03 | stephenfin | I've got this patch to start tracking pcpus as a ComputeNode/HostState field https://review.opendev.org/#/c/671794/ | |
| 13:37:47 | stephenfin | It's not complete, and it's also breaking some functional/tempest tests because one of those objects is being spat out for the os-hypervisors API | |
| 13:38:42 | stephenfin | However, does any of the resource tracking claims stuff matter for vcpus, ram and disk anymore, given we're actually requesting things from placement? | |
| 13:39:25 | openstackgerrit | sahid proposed openstack/nova master: cellv2: make update_cell to support cell0 https://review.opendev.org/672045 | |
| 13:39:28 | stephenfin | Referring specifically to this https://github.com/openstack/nova/blob/master/nova/compute/claims.py#L97-L109 | |
| 13:40:01 | mriedem | they used to matter for scheduler drivers that didn't use placement, like the caching scheduler, but that's gone now, | |
| 13:40:09 | mriedem | the overhead stuff comes from the driver and that's still part of the claim | |
| 13:40:17 | mriedem | i know the libvirt driver claims overhead in certain case | |
| 13:40:18 | mriedem | *cases | |
| 13:40:46 | mriedem | https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L884 | |
| 13:40:47 | stephenfin | Yup, for emulator threads, though I do have a chance to stop doing that now with the PCPU stuff | |
| 13:41:08 | mriedem | there are other drivers that implement that method as well | |
| 13:41:10 | mriedem | hyperv i think is one | |
| 13:41:46 | mriedem | https://github.com/openstack/nova/blob/master/nova/virt/hyperv/vmops.py#L119 | |
| 13:42:43 | mriedem | besides the overhead stuff i'm not sure anything cares about vcpus/ram/disk claims in the rt anymore since that should be handled by the scheduler + placement now | |
| 13:43:00 | mriedem | dansmith at one time had a patch to rip that out | |
| 13:43:05 | stephenfin | Hmm, so it sounds like that alone prevents me from simply removing this stuff https://github.com/openstack/nova/blob/master/nova/compute/claims.py#L160-L162 | |
| 13:43:28 | mriedem | https://review.opendev.org/#/c/551026/ | |
| 13:43:29 | dansmith | https://review.opendev.org/#/c/551026/ | |
| 13:43:32 | dansmith | heh | |
| 13:44:04 | stephenfin | Okay, I might incorporate that into my series, if that makes sense? | |
| 13:44:33 | dansmith | we never removed cachingscheduler did we? but surely we can now yeah? | |
| 13:44:36 | mriedem | we did | |
| 13:44:41 | mriedem | i just left a comment on dan's patch | |
| 13:44:41 | openstackgerrit | sahid proposed openstack/nova master: cellv2: make update_cell to support cell0 https://review.opendev.org/672045 | |
| 13:44:43 | dansmith | oh sweet | |
| 13:45:28 | mriedem | in boston we talked about the overhead-from-driver problem for claims / placement and i think had mostly just shrugged it off saying as a workaround operators could bump up the reserved amount of inventory on hosts that would use vms that need the overhead space | |
| 13:46:46 | mriedem | caching scheduler was removed in stein fwiw | |
| 13:47:57 | stephenfin | Hmm, okay, I clearly need to think about this | |
| 13:48:28 | stephenfin | Starting with figuring out if anything else is using/cares about the ComputeNode.vcpus/vcpus_used field | |
| 13:48:39 | stephenfin | (and therefore whether I need a pcpu equivalent) | |
| 13:55:00 | stephenfin | mriedem: Can/should 'Aggregate(Ram|Disk|Core)Filter' be deprecated? | |
| 13:55:34 | dansmith | those are widely used I believe | |
| 13:55:46 | dansmith | and don't have placement-based alternatives because they work on aggregates | |
| 13:56:04 | dansmith | we broke the allocation ratio one and people revolted | |
| 13:56:41 | stephenfin | I'm misreading this release note so, I guess :( https://github.com/openstack/nova/blob/master/releasenotes/notes/agg-resource-filters-6e24c92a69afa85f.yaml | |
| 13:56:56 | stephenfin | That suggests to me that aggregate-based overcommit ratios aren't a thing anymore | |
| 13:57:13 | stephenfin | and the filters can be removed because they're useless | |
| 13:57:40 | dansmith | there are more functions in the aggregate filters than just allocation ratios right? | |
| 13:57:49 | dansmith | however, that breakage was what led to the revolt I think | |
| 13:57:55 | dansmith | i.e. that commit | |
| 13:58:51 | stephenfin | Oh, quite possibly :) Looking at the AggregateDiskFilter, I don't see much more happening https://github.com/openstack/nova/blob/master/nova/scheduler/filters/disk_filter.py | |