| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-09-19 | |||
| 14:51:11 | cdent | mnaser: I think you mentioned you had a huge database that you could test a nova->placement migration script on, to get a sense of timing and the like? If that's true dansmith has made a thing: https://review.openstack.org/#/c/603234/ | |
| 14:51:23 | mnaser | yup, i can try that out | |
| 14:51:24 | mriedem | looks like it is, the only other extension with @wsgi.extends for servers is hide_server_addresses and i think that's deprecated? | |
| 14:51:54 | dansmith | mnaser: it just dumps/imports, so it should be safe to run against a live source deployment.. it doesn't delete anything from the source | |
| 14:52:10 | dansmith | but also, you could just measure the size of the tables and have a pretty good idea anyway | |
| 14:52:23 | lbragstad | gmann is https://review.openstack.org/#/c/547850/ still being pursued? | |
| 14:56:49 | mriedem | melwitt: btw, if you don't like geddy's voice, it's no crime to enjoy https://www.youtube.com/watch?v=iB4uwO1Dmf4 which is purely instrumental | |
| 14:59:30 | johnthetubaguy | gmann: good question, he did say he was planning on doing something about that | |
| 15:01:49 | mnaser | so looks like the biggest table is consumers | |
| 15:02:43 | mnaser | i tried it on a relatively large ~100ish deployment and it took 2 minutes with a triple replicated galera cluster with pretty beefy controllers | |
| 15:02:50 | bauzas | stupid question, but say I'm migrating some libvirt instance and would like to introspect its domain XML, should I lookup the source or the destination ? (in other words, shall I look at the migration allocation or the instance allocation) ? | |
| 15:03:14 | mnaser | bauzas: afaik the domain xml should be the same for live migrations at least afaik | |
| 15:03:31 | bauzas | for the moment, this is going to be a TODO since we don't support migrations for VGPU, but I need to make sure the code I'm writing for the reshape works | |
| 15:03:52 | mnaser | dansmith: ^ fyi for some numbers, it's relatively painless comparing to something like.. cells v2 stuff | |
| 15:03:54 | bauzas | mnaser: yup, but then the allocated PCI devices could differ | |
| 15:04:22 | mnaser | bauzas: ah yes. that's a qurik i didn't think of | |
| 15:04:31 | dansmith | mnaser: ack | |
| 15:04:51 | mnaser | It would be nice if it could pull in environment variables | |
| 15:04:56 | mnaser | But we could iterate on that later | |
| 15:05:04 | mnaser | So we don’t have to write credentials on disk when we don’t have to | |
| 15:05:55 | dansmith | mnaser: you should be able to do that as-is | |
| 15:06:13 | dansmith | write an empty config file and make those variables be in the environment already | |
| 15:08:48 | openstackgerrit | Merged openstack/nova master: Fix some typos in nova api ref doc https://review.openstack.org/603306 | |
| 15:10:16 | mriedem | bauzas: the answer depends on what you need to know, | |
| 15:10:31 | mriedem | if you're cleaning up something on the source during live migratoin, then look at the source xml, else look at the dest xml | |
| 15:10:52 | bauzas | mriedem: no, I just want to reshape the existing allocations onto the right physical device | |
| 15:11:10 | bauzas | mriedem: for that, I need to get the corresponding mediated devices | |
| 15:11:23 | mriedem | umm, | |
| 15:11:29 | mriedem | reshape *during* a live migration? | |
| 15:11:38 | dansmith | mnaser: I can make the db and host things not clobber environment too and I guess add a flag that will allow the file to be missing or something | |
| 15:12:23 | dansmith | credentials in environment is easy, but not more secure necessarily, so I guess I'm not sure why that's better, but alas :) | |
| 15:12:38 | mriedem | bauzas: at the start of live migration, conductor moves the existing allocations from the instance to the migration record, and then the scheduler is going to allocate resources from the dest (tree) for the instance | |
| 15:12:44 | mriedem | so i'm not sure why you'd need to reshape at all | |
| 15:13:03 | bauzas | mriedem: I know about how we manage allocations during a migration | |
| 15:13:05 | dansmith | mriedem: assume he means a symbolic reshape of allocations right? | |
| 15:13:16 | dansmith | not an actual POST /reshaper operation | |
| 15:13:25 | bauzas | the case I'm concerned is a cold migrate (because live migrating vGPUs is YAGNI) | |
| 15:13:25 | mriedem | my point is, why? | |
| 15:13:29 | dansmith | but a self-healing given the opportunity provided by a move | |
| 15:13:36 | bauzas | right | |
| 15:13:46 | dansmith | I dunno, I don't know what he's trying to heal exactly | |
| 15:13:58 | bauzas | so, say cold migrate is a thing | |
| 15:13:58 | mriedem | you can't heal a broken heart | |
| 15:14:09 | bauzas | I mean, cold migrate a VGPU | |
| 15:14:32 | bauzas | (which is not something there yet, but a bugfix I have targeted for Stein) | |
| 15:14:44 | bauzas | then cold migrating the instance would mean 2 allocations for this | |
| 15:15:09 | bauzas | one having the consumer UUID being the migration UUID on the source host, the other one being the real allocation | |
| 15:15:12 | bauzas | right? | |
| 15:15:13 | mriedem | and during the cold migrate you want to move the existing vgpu allocation from the root node provider on the source host to the child vgpu provider on the dest node? | |
| 15:15:45 | mriedem | the scheduler should take care of the latter | |
| 15:15:53 | mriedem | and after we confirm the resize, we'll drop the former | |
| 15:15:56 | mriedem | so i'm not sure why it matters | |
| 15:15:59 | bauzas | mriedem: no, I'd then just move the allocation on the source host to be on the right vGPU provider on the same host | |
| 15:16:13 | dansmith | wait what/ | |
| 15:16:24 | dansmith | we will do the same migration-holds-source-allocation yeah? | |
| 15:16:43 | bauzas | that's what I think yeah | |
| 15:17:00 | mriedem | me too | |
| 15:17:03 | mriedem | i don't see the issue | |
| 15:17:19 | bauzas | I'm probably confused too by what means a reshape | |
| 15:17:36 | mriedem | it sounds like you're trying to auto-heal/reshape duringa cold migrate so we don't have to run the maybe more expensive reshape-on-compute-startup upgrade thing | |
| 15:17:37 | bauzas | during the migration, we have 2 allocations on two different hosts, right? | |
| 15:18:00 | bauzas | ah shit, that's what I fucked | |
| 15:18:07 | dansmith | um, what? :) | |
| 15:18:14 | dansmith | lol. | |
| 15:18:23 | mriedem | maybe you want to just block az renames while the az has instances? | |
| 15:18:26 | mriedem | that's probably easier | |
| 15:18:26 | bauzas | can we have a migration running, and both services to be restarted ? | |
| 15:18:43 | mriedem | sure | |
| 15:18:45 | bauzas | in that case, it would trigger a reshape non? | |
| 15:18:55 | mriedem | sure | |
| 15:18:56 | bauzas | for both ? | |
| 15:19:12 | bauzas | so in that interim period, we have two allocations, nope ? | |
| 15:19:26 | mriedem | if the allocations have moved from the instance to the migration record on the source host and the source host is restarted and a reshape happens, we should still move the allocations for the migration record | |
| 15:19:41 | mriedem | the reshape in this case happens on distinct provider trees | |
| 15:19:42 | bauzas | that's my point | |
| 15:20:00 | bauzas | so in that case, I have a migration UUID that is the consumer | |
| 15:20:03 | bauzas | on the source host | |
| 15:20:06 | mriedem | if the resize is confirmed, we drop the source node tree allocations for the migration record, | |
| 15:20:25 | mriedem | on revert we drop the target node tree allocations for the instance and move them back to the instance for the source tree | |
| 15:20:28 | mriedem | but the reshape should be ok | |
| 15:20:56 | bauzas | ok, but then I have to figure out the real instance UUID for the migration record | |
| 15:21:01 | bauzas | that's a special case | |
| 15:21:06 | mriedem | the migration consumer is just a fill in so someone doesn't quack quack seat back the source node resources | |
| 15:21:33 | mriedem | honestly i don't know what problem you're trying to solve | |
| 15:21:52 | mriedem | we should probably identify that a problem exists before discussing designs on how to fix it | |
| 15:21:57 | bauzas | probably | |
| 15:22:02 | bauzas | and I'm unclear | |
| 15:22:25 | bauzas | I'll just leave a comment in my patch and I move on | |
| 15:28:27 | dansmith | mnaser: how much of your two minutes is writing and reading that dumpfile on disk? would you prefer we just pipe it between the export and import (optionally?) and maybe tee it out for forensics? might be faster | |
| 15:30:58 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove deprecated hide_server_address_states option https://review.openstack.org/603831 | |
| 15:31:02 | mriedem | gmann: ^ | |
| 15:34:51 | mnaser | dansmith: like 3-5 seconds for the dump. The write took the longest | |
| 15:35:48 | dansmith | mnaser: 3-5 seconds overhead for writing the file? or 3-5 seconds to do the dump and the rest of the minute to write it out? | |
| 15:36:00 | dansmith | on my tiny database it's immeasurable of course | |
| 15:36:14 | dansmith | I get a constant 3.6s to do the dump and import regardless | |
| 15:36:35 | dansmith | mnaser: I have a diff that does it pipely if you want to try it | |
| 15:36:48 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove deprecated hide_server_address_states option https://review.openstack.org/603831 | |
| 15:37:32 | dansmith | mnaser: https://termbin.co/FXJg | |
| 15:38:05 | dansmith | I also put a "tee $tmpfile |" in the middle of that pipe (hence the comment) but removed it for your test | |
| 15:56:09 | openstackgerrit | Ben Nemec proposed openstack/nova master: WIP: Migrate upgrade checks to oslo.upgradecheck https://review.openstack.org/603499 | |