| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-09-19 | |||
| 15:03:31 | bauzas | for the moment, this is going to be a TODO since we don't support migrations for VGPU, but I need to make sure the code I'm writing for the reshape works | |
| 15:03:52 | mnaser | dansmith: ^ fyi for some numbers, it's relatively painless comparing to something like.. cells v2 stuff | |
| 15:03:54 | bauzas | mnaser: yup, but then the allocated PCI devices could differ | |
| 15:04:22 | mnaser | bauzas: ah yes. that's a qurik i didn't think of | |
| 15:04:31 | dansmith | mnaser: ack | |
| 15:04:51 | mnaser | It would be nice if it could pull in environment variables | |
| 15:04:56 | mnaser | But we could iterate on that later | |
| 15:05:04 | mnaser | So we don’t have to write credentials on disk when we don’t have to | |
| 15:05:55 | dansmith | mnaser: you should be able to do that as-is | |
| 15:06:13 | dansmith | write an empty config file and make those variables be in the environment already | |
| 15:08:48 | openstackgerrit | Merged openstack/nova master: Fix some typos in nova api ref doc https://review.openstack.org/603306 | |
| 15:10:16 | mriedem | bauzas: the answer depends on what you need to know, | |
| 15:10:31 | mriedem | if you're cleaning up something on the source during live migratoin, then look at the source xml, else look at the dest xml | |
| 15:10:52 | bauzas | mriedem: no, I just want to reshape the existing allocations onto the right physical device | |
| 15:11:10 | bauzas | mriedem: for that, I need to get the corresponding mediated devices | |
| 15:11:23 | mriedem | umm, | |
| 15:11:29 | mriedem | reshape *during* a live migration? | |
| 15:11:38 | dansmith | mnaser: I can make the db and host things not clobber environment too and I guess add a flag that will allow the file to be missing or something | |
| 15:12:23 | dansmith | credentials in environment is easy, but not more secure necessarily, so I guess I'm not sure why that's better, but alas :) | |
| 15:12:38 | mriedem | bauzas: at the start of live migration, conductor moves the existing allocations from the instance to the migration record, and then the scheduler is going to allocate resources from the dest (tree) for the instance | |
| 15:12:44 | mriedem | so i'm not sure why you'd need to reshape at all | |
| 15:13:03 | bauzas | mriedem: I know about how we manage allocations during a migration | |
| 15:13:05 | dansmith | mriedem: assume he means a symbolic reshape of allocations right? | |
| 15:13:16 | dansmith | not an actual POST /reshaper operation | |
| 15:13:25 | mriedem | my point is, why? | |
| 15:13:25 | bauzas | the case I'm concerned is a cold migrate (because live migrating vGPUs is YAGNI) | |
| 15:13:29 | dansmith | but a self-healing given the opportunity provided by a move | |
| 15:13:36 | bauzas | right | |
| 15:13:46 | dansmith | I dunno, I don't know what he's trying to heal exactly | |
| 15:13:58 | mriedem | you can't heal a broken heart | |
| 15:13:58 | bauzas | so, say cold migrate is a thing | |
| 15:14:09 | bauzas | I mean, cold migrate a VGPU | |
| 15:14:32 | bauzas | (which is not something there yet, but a bugfix I have targeted for Stein) | |
| 15:14:44 | bauzas | then cold migrating the instance would mean 2 allocations for this | |
| 15:15:09 | bauzas | one having the consumer UUID being the migration UUID on the source host, the other one being the real allocation | |
| 15:15:12 | bauzas | right? | |
| 15:15:13 | mriedem | and during the cold migrate you want to move the existing vgpu allocation from the root node provider on the source host to the child vgpu provider on the dest node? | |
| 15:15:45 | mriedem | the scheduler should take care of the latter | |
| 15:15:53 | mriedem | and after we confirm the resize, we'll drop the former | |
| 15:15:56 | mriedem | so i'm not sure why it matters | |
| 15:15:59 | bauzas | mriedem: no, I'd then just move the allocation on the source host to be on the right vGPU provider on the same host | |
| 15:16:13 | dansmith | wait what/ | |
| 15:16:24 | dansmith | we will do the same migration-holds-source-allocation yeah? | |
| 15:16:43 | bauzas | that's what I think yeah | |
| 15:17:00 | mriedem | me too | |
| 15:17:03 | mriedem | i don't see the issue | |
| 15:17:19 | bauzas | I'm probably confused too by what means a reshape | |
| 15:17:36 | mriedem | it sounds like you're trying to auto-heal/reshape duringa cold migrate so we don't have to run the maybe more expensive reshape-on-compute-startup upgrade thing | |
| 15:17:37 | bauzas | during the migration, we have 2 allocations on two different hosts, right? | |
| 15:18:00 | bauzas | ah shit, that's what I fucked | |
| 15:18:07 | dansmith | um, what? :) | |
| 15:18:14 | dansmith | lol. | |
| 15:18:23 | mriedem | maybe you want to just block az renames while the az has instances? | |
| 15:18:26 | bauzas | can we have a migration running, and both services to be restarted ? | |
| 15:18:26 | mriedem | that's probably easier | |
| 15:18:43 | mriedem | sure | |
| 15:18:45 | bauzas | in that case, it would trigger a reshape non? | |
| 15:18:55 | mriedem | sure | |
| 15:18:56 | bauzas | for both ? | |
| 15:19:12 | bauzas | so in that interim period, we have two allocations, nope ? | |
| 15:19:26 | mriedem | if the allocations have moved from the instance to the migration record on the source host and the source host is restarted and a reshape happens, we should still move the allocations for the migration record | |
| 15:19:41 | mriedem | the reshape in this case happens on distinct provider trees | |
| 15:19:42 | bauzas | that's my point | |
| 15:20:00 | bauzas | so in that case, I have a migration UUID that is the consumer | |
| 15:20:03 | bauzas | on the source host | |
| 15:20:06 | mriedem | if the resize is confirmed, we drop the source node tree allocations for the migration record, | |
| 15:20:25 | mriedem | on revert we drop the target node tree allocations for the instance and move them back to the instance for the source tree | |
| 15:20:28 | mriedem | but the reshape should be ok | |
| 15:20:56 | bauzas | ok, but then I have to figure out the real instance UUID for the migration record | |
| 15:21:01 | bauzas | that's a special case | |
| 15:21:06 | mriedem | the migration consumer is just a fill in so someone doesn't quack quack seat back the source node resources | |
| 15:21:33 | mriedem | honestly i don't know what problem you're trying to solve | |
| 15:21:52 | mriedem | we should probably identify that a problem exists before discussing designs on how to fix it | |
| 15:21:57 | bauzas | probably | |
| 15:22:02 | bauzas | and I'm unclear | |
| 15:22:25 | bauzas | I'll just leave a comment in my patch and I move on | |
| 15:28:27 | dansmith | mnaser: how much of your two minutes is writing and reading that dumpfile on disk? would you prefer we just pipe it between the export and import (optionally?) and maybe tee it out for forensics? might be faster | |
| 15:30:58 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove deprecated hide_server_address_states option https://review.openstack.org/603831 | |
| 15:31:02 | mriedem | gmann: ^ | |
| 15:34:51 | mnaser | dansmith: like 3-5 seconds for the dump. The write took the longest | |
| 15:35:48 | dansmith | mnaser: 3-5 seconds overhead for writing the file? or 3-5 seconds to do the dump and the rest of the minute to write it out? | |
| 15:36:00 | dansmith | on my tiny database it's immeasurable of course | |
| 15:36:14 | dansmith | I get a constant 3.6s to do the dump and import regardless | |
| 15:36:35 | dansmith | mnaser: I have a diff that does it pipely if you want to try it | |
| 15:36:48 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove deprecated hide_server_address_states option https://review.openstack.org/603831 | |
| 15:37:32 | dansmith | mnaser: https://termbin.co/FXJg | |
| 15:38:05 | dansmith | I also put a "tee $tmpfile |" in the middle of that pipe (hence the comment) but removed it for your test | |
| 15:56:09 | openstackgerrit | Ben Nemec proposed openstack/nova master: WIP: Migrate upgrade checks to oslo.upgradecheck https://review.openstack.org/603499 | |
| 16:00:26 | dansmith | tssurya: mriedem shall we meet about cells? we just talked about things last week and melwitt is not around this week | |
| 16:01:44 | mnaser | dansmith: sorry i was in a meeting, the dump was really quick but the push was much slower because of (what i assume) replication | |
| 16:01:58 | dansmith | ah the load, I see | |
| 16:02:08 | dansmith | yeah I'm sure that's the heavy part | |
| 16:02:16 | dansmith | it's heavy on my toy devstack even | |
| 16:03:05 | dansmith | in that case, I'd just say we should leave separate the dump/load like it is now so that one can complete without the other, leaving a file you can just import if you want it | |
| 16:05:15 | jaypipes | stephenfin: so, unless I'm mis-reading your comments on the cpu-resource-tracking spec, you'd actually favor an "opt-in" approach to hyperthread usage. Is that correct? i.e. the virt driver defaults to *not* counting hyperthreads as CPU resources for guests unless some knob is turned on? | |
| 16:06:06 | stephenfin | jaypipes: For instances with dedicated CPUs, yes. However, we need to avoid breaking users | |
| 16:06:13 | stephenfin | So that knob would have to default to on | |
| 16:06:58 | stephenfin | The reason I want that is because a hyperthread != a core and I think the decision to model it as one was a mistake on day one | |
| 16:06:59 | jaypipes | stephenfin: ack. no disagreement from me on that | |
| 16:07:12 | jaypipes | stephenfin: agreed. | |