| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-09-19 | |||
| 15:12:38 | mriedem | bauzas: at the start of live migration, conductor moves the existing allocations from the instance to the migration record, and then the scheduler is going to allocate resources from the dest (tree) for the instance | |
| 15:12:44 | mriedem | so i'm not sure why you'd need to reshape at all | |
| 15:13:03 | bauzas | mriedem: I know about how we manage allocations during a migration | |
| 15:13:05 | dansmith | mriedem: assume he means a symbolic reshape of allocations right? | |
| 15:13:16 | dansmith | not an actual POST /reshaper operation | |
| 15:13:25 | bauzas | the case I'm concerned is a cold migrate (because live migrating vGPUs is YAGNI) | |
| 15:13:25 | mriedem | my point is, why? | |
| 15:13:29 | dansmith | but a self-healing given the opportunity provided by a move | |
| 15:13:36 | bauzas | right | |
| 15:13:46 | dansmith | I dunno, I don't know what he's trying to heal exactly | |
| 15:13:58 | bauzas | so, say cold migrate is a thing | |
| 15:13:58 | mriedem | you can't heal a broken heart | |
| 15:14:09 | bauzas | I mean, cold migrate a VGPU | |
| 15:14:32 | bauzas | (which is not something there yet, but a bugfix I have targeted for Stein) | |
| 15:14:44 | bauzas | then cold migrating the instance would mean 2 allocations for this | |
| 15:15:09 | bauzas | one having the consumer UUID being the migration UUID on the source host, the other one being the real allocation | |
| 15:15:12 | bauzas | right? | |
| 15:15:13 | mriedem | and during the cold migrate you want to move the existing vgpu allocation from the root node provider on the source host to the child vgpu provider on the dest node? | |
| 15:15:45 | mriedem | the scheduler should take care of the latter | |
| 15:15:53 | mriedem | and after we confirm the resize, we'll drop the former | |
| 15:15:56 | mriedem | so i'm not sure why it matters | |
| 15:15:59 | bauzas | mriedem: no, I'd then just move the allocation on the source host to be on the right vGPU provider on the same host | |
| 15:16:13 | dansmith | wait what/ | |
| 15:16:24 | dansmith | we will do the same migration-holds-source-allocation yeah? | |
| 15:16:43 | bauzas | that's what I think yeah | |
| 15:17:00 | mriedem | me too | |
| 15:17:03 | mriedem | i don't see the issue | |
| 15:17:19 | bauzas | I'm probably confused too by what means a reshape | |
| 15:17:36 | mriedem | it sounds like you're trying to auto-heal/reshape duringa cold migrate so we don't have to run the maybe more expensive reshape-on-compute-startup upgrade thing | |
| 15:17:37 | bauzas | during the migration, we have 2 allocations on two different hosts, right? | |
| 15:18:00 | bauzas | ah shit, that's what I fucked | |
| 15:18:07 | dansmith | um, what? :) | |
| 15:18:14 | dansmith | lol. | |
| 15:18:23 | mriedem | maybe you want to just block az renames while the az has instances? | |
| 15:18:26 | mriedem | that's probably easier | |
| 15:18:26 | bauzas | can we have a migration running, and both services to be restarted ? | |
| 15:18:43 | mriedem | sure | |
| 15:18:45 | bauzas | in that case, it would trigger a reshape non? | |
| 15:18:55 | mriedem | sure | |
| 15:18:56 | bauzas | for both ? | |
| 15:19:12 | bauzas | so in that interim period, we have two allocations, nope ? | |
| 15:19:26 | mriedem | if the allocations have moved from the instance to the migration record on the source host and the source host is restarted and a reshape happens, we should still move the allocations for the migration record | |
| 15:19:41 | mriedem | the reshape in this case happens on distinct provider trees | |
| 15:19:42 | bauzas | that's my point | |
| 15:20:00 | bauzas | so in that case, I have a migration UUID that is the consumer | |
| 15:20:03 | bauzas | on the source host | |
| 15:20:06 | mriedem | if the resize is confirmed, we drop the source node tree allocations for the migration record, | |
| 15:20:25 | mriedem | on revert we drop the target node tree allocations for the instance and move them back to the instance for the source tree | |
| 15:20:28 | mriedem | but the reshape should be ok | |
| 15:20:56 | bauzas | ok, but then I have to figure out the real instance UUID for the migration record | |
| 15:21:01 | bauzas | that's a special case | |
| 15:21:06 | mriedem | the migration consumer is just a fill in so someone doesn't quack quack seat back the source node resources | |
| 15:21:33 | mriedem | honestly i don't know what problem you're trying to solve | |
| 15:21:52 | mriedem | we should probably identify that a problem exists before discussing designs on how to fix it | |
| 15:21:57 | bauzas | probably | |
| 15:22:02 | bauzas | and I'm unclear | |
| 15:22:25 | bauzas | I'll just leave a comment in my patch and I move on | |
| 15:28:27 | dansmith | mnaser: how much of your two minutes is writing and reading that dumpfile on disk? would you prefer we just pipe it between the export and import (optionally?) and maybe tee it out for forensics? might be faster | |
| 15:30:58 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove deprecated hide_server_address_states option https://review.openstack.org/603831 | |
| 15:31:02 | mriedem | gmann: ^ | |
| 15:34:51 | mnaser | dansmith: like 3-5 seconds for the dump. The write took the longest | |
| 15:35:48 | dansmith | mnaser: 3-5 seconds overhead for writing the file? or 3-5 seconds to do the dump and the rest of the minute to write it out? | |
| 15:36:00 | dansmith | on my tiny database it's immeasurable of course | |
| 15:36:14 | dansmith | I get a constant 3.6s to do the dump and import regardless | |
| 15:36:35 | dansmith | mnaser: I have a diff that does it pipely if you want to try it | |
| 15:36:48 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove deprecated hide_server_address_states option https://review.openstack.org/603831 | |
| 15:37:32 | dansmith | mnaser: https://termbin.co/FXJg | |
| 15:38:05 | dansmith | I also put a "tee $tmpfile |" in the middle of that pipe (hence the comment) but removed it for your test | |
| 15:56:09 | openstackgerrit | Ben Nemec proposed openstack/nova master: WIP: Migrate upgrade checks to oslo.upgradecheck https://review.openstack.org/603499 | |
| 16:00:26 | dansmith | tssurya: mriedem shall we meet about cells? we just talked about things last week and melwitt is not around this week | |
| 16:01:44 | mnaser | dansmith: sorry i was in a meeting, the dump was really quick but the push was much slower because of (what i assume) replication | |
| 16:01:58 | dansmith | ah the load, I see | |
| 16:02:08 | dansmith | yeah I'm sure that's the heavy part | |
| 16:02:16 | dansmith | it's heavy on my toy devstack even | |
| 16:03:05 | dansmith | in that case, I'd just say we should leave separate the dump/load like it is now so that one can complete without the other, leaving a file you can just import if you want it | |
| 16:05:15 | jaypipes | stephenfin: so, unless I'm mis-reading your comments on the cpu-resource-tracking spec, you'd actually favor an "opt-in" approach to hyperthread usage. Is that correct? i.e. the virt driver defaults to *not* counting hyperthreads as CPU resources for guests unless some knob is turned on? | |
| 16:06:06 | stephenfin | jaypipes: For instances with dedicated CPUs, yes. However, we need to avoid breaking users | |
| 16:06:13 | stephenfin | So that knob would have to default to on | |
| 16:06:58 | stephenfin | The reason I want that is because a hyperthread != a core and I think the decision to model it as one was a mistake on day one | |
| 16:06:59 | jaypipes | stephenfin: ack. no disagreement from me on that | |
| 16:07:12 | jaypipes | stephenfin: agreed. | |
| 16:10:21 | openstackgerrit | sean mooney proposed openstack/nova-specs master: [WIP] generic device discovery policy https://review.openstack.org/603805 | |
| 16:20:37 | openstackgerrit | Merged openstack/nova stable/queens: Fix DB archiver AttributeError due to wrong table name attribute used https://review.openstack.org/599882 | |
| 16:20:44 | openstackgerrit | Merged openstack/nova stable/queens: VMware: fix TypeError while get console log https://review.openstack.org/591365 | |
| 16:20:51 | openstackgerrit | Merged openstack/nova stable/queens: Move conductor wait_until_ready() delay before manager init https://review.openstack.org/599200 | |
| 16:28:59 | mriedem | dansmith: i'm good to skip; plan on starting my poc for cross-cell resize today | |
| 16:29:11 | dansmith | oh boy | |
| 16:30:33 | openstackgerrit | Merged openstack/nova stable/ocata: Return 400 when compute host is not found https://review.openstack.org/590649 | |
| 16:31:38 | stephenfin | jaypipes: If you don't have one already, a placement cheatsheet (in PDF and therefore printable) would be super useful | |
| 16:31:58 | jaypipes | stephenfin: also, I agree with you about getting rid of emulator_threads_policy eventually, but is it 100% critical that I update the cpu-resource-tracking spec with information about that? | |
| 16:32:24 | jaypipes | stephenfin: you mean cheatsheet about the REST API? | |
| 16:32:29 | jaypipes | stephenfin: or something else? | |
| 16:32:41 | jaypipes | stephenfin: or you mean a glossary of placement terms? | |
| 16:33:26 | stephenfin | jaypipes: No, it can be be handled separately. However, I imagine you do need to take cpu overhead, which is incremented for each dedicated emulator thread core, into account | |
| 16:33:31 | openstackgerrit | Jack Ding proposed openstack/nova master: Correct instance port binding for rebuilds/reboots https://review.openstack.org/603844 | |
| 16:33:55 | stephenfin | jaypipes: Both. Glossary of terms followed by examples of either the REST API or using osc-placement | |
| 16:34:17 | jaypipes | stephenfin: ack | |
| 16:34:21 | stephenfin | jaypipes: Just an idea, obviously. I might even start drafting something myself some point this week | |
| 16:34:44 | jaypipes | stephenfin: cool. let me know when you get to the "what is a consumer?" part... | |
| 16:36:43 | stephenfin | jaypipes: FYI, the reason I brought up the whole "let's kill 'isolate' idea" was that that approach seemed easier than handling the CPU overhead thing *and* it reduced complexity for the user in the process, which feels like one of the goals | |