| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-09-19 | |||
| 13:30:47 | BlackDex | claudiub: yes | |
| 13:30:54 | claudiub | heh, that sounds like live-resize | |
| 13:31:02 | BlackDex | yea kinda is | |
| 13:31:11 | BlackDex | but i haven't seen that feature yet ;) | |
| 13:31:25 | BlackDex | i need it only for the iotune parameters | |
| 13:31:41 | claudiub | it's because the proposal for that feature didn't merge: https://review.openstack.org/#/c/141219/ | |
| 13:31:57 | BlackDex | the iotune parameters only seem to be changed when using volume-types | |
| 13:32:14 | BlackDex | those seem te be dynamic when i change the min/max iops | |
| 13:32:40 | BlackDex | but not with ephemeral disks | |
| 13:34:10 | claudiub | hmm, i'm not sure those are set for ephemeral disks | |
| 13:34:25 | BlackDex | you can set that in the flavor | |
| 13:34:59 | BlackDex | quota:disk_read_iops_sec for instnace | |
| 13:35:16 | BlackDex | that works for an empemeral disk | |
| 13:35:36 | BlackDex | but, if you want to change that you need to resize/cold-migrate the instace for it to work | |
| 13:36:15 | BlackDex | while volumes using a volume-type which are live-migrated will get a new iops setting if the volume-type has a qos set | |
| 13:36:46 | claudiub | yeah, indeed, they are being set during resize. | |
| 13:36:49 | BlackDex | i tried to change the mysql entries of the instance_extras table | |
| 13:37:09 | BlackDex | the volume-type qos is also set during a live-migrate! | |
| 13:37:17 | BlackDex | if it previously wasn't | |
| 13:38:28 | BlackDex | so, i changed in the mysql everything of the instance metadata info, but it doesn't use those values during a live-migrate | |
| 13:39:04 | BlackDex | so i wondered where the parameters for qemu are comming from durning the live-migrate | |
| 13:39:30 | BlackDex | it seems they do not come from the mysql database, or the libvirt xml | |
| 13:39:49 | BlackDex | there are pulled from somewhere els | |
| 13:39:51 | BlackDex | e | |
| 13:53:40 | openstackgerrit | sean mooney proposed openstack/nova-specs master: [WIP] generic device discovery policy https://review.openstack.org/603805 | |
| 13:55:29 | openstackgerrit | Surya Seetharaman proposed openstack/nova master: Return a minimal construct for nova list when a cell is down https://review.openstack.org/567785 | |
| 13:55:30 | openstackgerrit | Surya Seetharaman proposed openstack/nova master: Add scatter-gather-single-cell utility https://review.openstack.org/594947 | |
| 13:57:15 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Resource retrieving: add changes-before filter https://review.openstack.org/599276 | |
| 14:04:00 | openstackgerrit | Jay Pipes proposed openstack/os-traits master: clean up CUDA traits https://review.openstack.org/597170 | |
| 14:23:09 | mriedem | dansmith: still humping approved stable changes through the gate, fyi | |
| 14:23:17 | dansmith | ack | |
| 14:39:02 | mriedem | jaypipes: i'm +2 on the 2.66 changes-before change now https://review.openstack.org/#/c/599276/ | |
| 14:39:29 | jaypipes | mriedem: yeah, I +W'd that a while ago :) | |
| 14:39:55 | jaypipes | mriedem: "a while ago" == whole minutes ago. | |
| 14:40:09 | mriedem | ah heh | |
| 14:40:10 | mriedem | cool | |
| 14:48:58 | mriedem | gmann: is https://review.openstack.org/#/c/596285/ the last change for https://blueprints.launchpad.net/nova/+spec/api-extensions-merge-stein ? | |
| 14:51:11 | cdent | mnaser: I think you mentioned you had a huge database that you could test a nova->placement migration script on, to get a sense of timing and the like? If that's true dansmith has made a thing: https://review.openstack.org/#/c/603234/ | |
| 14:51:23 | mnaser | yup, i can try that out | |
| 14:51:24 | mriedem | looks like it is, the only other extension with @wsgi.extends for servers is hide_server_addresses and i think that's deprecated? | |
| 14:51:54 | dansmith | mnaser: it just dumps/imports, so it should be safe to run against a live source deployment.. it doesn't delete anything from the source | |
| 14:52:10 | dansmith | but also, you could just measure the size of the tables and have a pretty good idea anyway | |
| 14:52:23 | lbragstad | gmann is https://review.openstack.org/#/c/547850/ still being pursued? | |
| 14:56:49 | mriedem | melwitt: btw, if you don't like geddy's voice, it's no crime to enjoy https://www.youtube.com/watch?v=iB4uwO1Dmf4 which is purely instrumental | |
| 14:59:30 | johnthetubaguy | gmann: good question, he did say he was planning on doing something about that | |
| 15:01:49 | mnaser | so looks like the biggest table is consumers | |
| 15:02:43 | mnaser | i tried it on a relatively large ~100ish deployment and it took 2 minutes with a triple replicated galera cluster with pretty beefy controllers | |
| 15:02:50 | bauzas | stupid question, but say I'm migrating some libvirt instance and would like to introspect its domain XML, should I lookup the source or the destination ? (in other words, shall I look at the migration allocation or the instance allocation) ? | |
| 15:03:14 | mnaser | bauzas: afaik the domain xml should be the same for live migrations at least afaik | |
| 15:03:31 | bauzas | for the moment, this is going to be a TODO since we don't support migrations for VGPU, but I need to make sure the code I'm writing for the reshape works | |
| 15:03:52 | mnaser | dansmith: ^ fyi for some numbers, it's relatively painless comparing to something like.. cells v2 stuff | |
| 15:03:54 | bauzas | mnaser: yup, but then the allocated PCI devices could differ | |
| 15:04:22 | mnaser | bauzas: ah yes. that's a qurik i didn't think of | |
| 15:04:31 | dansmith | mnaser: ack | |
| 15:04:51 | mnaser | It would be nice if it could pull in environment variables | |
| 15:04:56 | mnaser | But we could iterate on that later | |
| 15:05:04 | mnaser | So we don’t have to write credentials on disk when we don’t have to | |
| 15:05:55 | dansmith | mnaser: you should be able to do that as-is | |
| 15:06:13 | dansmith | write an empty config file and make those variables be in the environment already | |
| 15:08:48 | openstackgerrit | Merged openstack/nova master: Fix some typos in nova api ref doc https://review.openstack.org/603306 | |
| 15:10:16 | mriedem | bauzas: the answer depends on what you need to know, | |
| 15:10:31 | mriedem | if you're cleaning up something on the source during live migratoin, then look at the source xml, else look at the dest xml | |
| 15:10:52 | bauzas | mriedem: no, I just want to reshape the existing allocations onto the right physical device | |
| 15:11:10 | bauzas | mriedem: for that, I need to get the corresponding mediated devices | |
| 15:11:23 | mriedem | umm, | |
| 15:11:29 | mriedem | reshape *during* a live migration? | |
| 15:11:38 | dansmith | mnaser: I can make the db and host things not clobber environment too and I guess add a flag that will allow the file to be missing or something | |
| 15:12:23 | dansmith | credentials in environment is easy, but not more secure necessarily, so I guess I'm not sure why that's better, but alas :) | |
| 15:12:38 | mriedem | bauzas: at the start of live migration, conductor moves the existing allocations from the instance to the migration record, and then the scheduler is going to allocate resources from the dest (tree) for the instance | |
| 15:12:44 | mriedem | so i'm not sure why you'd need to reshape at all | |
| 15:13:03 | bauzas | mriedem: I know about how we manage allocations during a migration | |
| 15:13:05 | dansmith | mriedem: assume he means a symbolic reshape of allocations right? | |
| 15:13:16 | dansmith | not an actual POST /reshaper operation | |
| 15:13:25 | bauzas | the case I'm concerned is a cold migrate (because live migrating vGPUs is YAGNI) | |
| 15:13:25 | mriedem | my point is, why? | |
| 15:13:29 | dansmith | but a self-healing given the opportunity provided by a move | |
| 15:13:36 | bauzas | right | |
| 15:13:46 | dansmith | I dunno, I don't know what he's trying to heal exactly | |
| 15:13:58 | bauzas | so, say cold migrate is a thing | |
| 15:13:58 | mriedem | you can't heal a broken heart | |
| 15:14:09 | bauzas | I mean, cold migrate a VGPU | |
| 15:14:32 | bauzas | (which is not something there yet, but a bugfix I have targeted for Stein) | |
| 15:14:44 | bauzas | then cold migrating the instance would mean 2 allocations for this | |
| 15:15:09 | bauzas | one having the consumer UUID being the migration UUID on the source host, the other one being the real allocation | |
| 15:15:12 | bauzas | right? | |
| 15:15:13 | mriedem | and during the cold migrate you want to move the existing vgpu allocation from the root node provider on the source host to the child vgpu provider on the dest node? | |
| 15:15:45 | mriedem | the scheduler should take care of the latter | |
| 15:15:53 | mriedem | and after we confirm the resize, we'll drop the former | |
| 15:15:56 | mriedem | so i'm not sure why it matters | |
| 15:15:59 | bauzas | mriedem: no, I'd then just move the allocation on the source host to be on the right vGPU provider on the same host | |
| 15:16:13 | dansmith | wait what/ | |
| 15:16:24 | dansmith | we will do the same migration-holds-source-allocation yeah? | |
| 15:16:43 | bauzas | that's what I think yeah | |
| 15:17:00 | mriedem | me too | |
| 15:17:03 | mriedem | i don't see the issue | |
| 15:17:19 | bauzas | I'm probably confused too by what means a reshape | |
| 15:17:36 | mriedem | it sounds like you're trying to auto-heal/reshape duringa cold migrate so we don't have to run the maybe more expensive reshape-on-compute-startup upgrade thing | |
| 15:17:37 | bauzas | during the migration, we have 2 allocations on two different hosts, right? | |
| 15:18:00 | bauzas | ah shit, that's what I fucked | |
| 15:18:07 | dansmith | um, what? :) | |