| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-05-04 | |||
| 12:37:29 | finucannot | giblet: One thought I had on numa-aware-vswitches: how do we deal with changes in the networks attached to the guest when migrating? I think you touched on that here https://review.openstack.org/#/c/541290/8/specs/rocky/approved/numa-aware-vswitches.rst@243 | |
| 12:38:25 | giblet | finucannot: yes. even if the network itself is not changing the NUMA affinity of the devices providing access to the given network can be different on different hosts | |
| 12:39:01 | giblet | finucannot: and the actual physnet can change if we have multiprovider networks with mutliple segments having different physnets | |
| 12:39:23 | giblet | finucannot: but the later does not supported today by nova anyhow | |
| 12:39:28 | finucannot | OK, I don't think that's actually an issue. In that case, we'd be regenerating the guest NUMA topology for the new host and that would use the new host's NUMA-network mapping | |
| 12:39:41 | finucannot | Also, I think we decided multi-segment is out-of-scope | |
| 12:39:59 | giblet | finucannot: multi-segment is out of scope, I agree | |
| 12:40:23 | finucannot | I'm more concerned about the actual networks attached. We've said that a user has to allocate networks at instance creation time so we can do NUMA affinity | |
| 12:40:51 | finucannot | However, you can attach/detach networks to/from a running instance, right? | |
| 12:40:52 | giblet | finucannot: could you be bit more specific how the network changes in your scenario | |
| 12:41:00 | giblet | finucannot: ahh, yes | |
| 12:41:28 | finucannot | So I boot an instance attached to network 'foo' and then, once it's running, detach from 'foo' and attach to 'bar' | |
| 12:41:37 | giblet | finucannot: when you attach a new port / network that might affect the necessary affinity | |
| 12:42:09 | finucannot | Yeah, exactly. We can't do anything about that while the instance is on the same host, but what about if we migrate/rebuild? | |
| 12:42:47 | giblet | finucannot: you can actually check if the port being attached creates a contradiction with the existing affinity of the instance | |
| 12:43:01 | giblet | finucannot: and you might reject the attach | |
| 12:43:45 | giblet | finucannot: but I agree that when you migrate you have to take every port / network into account | |
| 12:43:58 | giblet | finucannot: including those that was attached after the boot | |
| 12:44:40 | finucannot | Yeah, we could do that. I guess that could/should be a configurable policy option down the line | |
| 12:45:35 | finucannot | But yeah, I'm thinking I should regenerate the NetworkRequestList object attached to the instance/request spec when migrating to reflect the network topology pre-migration | |
| 12:45:42 | finucannot | That might even happen already. I should check | |
| 12:46:24 | giblet | finucannot: I think regenerating the information from Neutron is the safe solution | |
| 12:47:15 | giblet | finucannot: I would even go that far that don't persist the NetworkRequestList but simply regenerate when it is needed | |
| 12:52:15 | finucannot | You need to though so that you can use it during claiming. Without that we have no way to figure out what's necessary https://review.openstack.org/#/c/541290/8/specs/rocky/approved/numa-aware-vswitches.rst@243 | |
| 12:52:21 | finucannot | whoops | |
| 12:52:28 | finucannot | https://review.openstack.org/#/c/564449/1/nova/compute/claims.py | |
| 12:53:34 | finucannot | It also needs to be stored in the RequestSpec object so that we don't need to query neutron from the filters (which we can't do because we don't have correct context) | |
| 12:55:14 | giblet | finucannot: does the RequestSpec loaded from the db in the scheduler or passed via rpc? | |
| 12:55:31 | finucannot | The former, to the best of my knowledge | |
| 12:56:00 | giblet | finucannot: then I agree that you have to store the NetworkREquestList to the db along with the RequestSpec | |
| 12:56:43 | finucannot | Yeah, I mapped the whole thing here http://paste.openstack.org/show/720365/ | |
| 12:59:58 | giblet | finucannot: I think the RequestSpec is passed to the scheduler via rpc https://github.com/openstack/nova/blob/5d97937c3c56a3e240a3350a7a9f0e3dcb954c52/nova/scheduler/rpcapi.py#L132 | |
| 13:00:34 | giblet | the spec_obj there is a RequestSpec obj | |
| 13:03:33 | giblet | finucannot: also the build_and_run_instance leading to the claim gets the instance object via rpc | |
| 13:04:46 | finucannot | Yup, so if I wanted to, for example, pass an additional 'network_requests' parameter to 'claim()', I guess I'd have to bump the RPC version | |
| 13:05:01 | finucannot | I'm pretty sure I looked at that though and it wasn't possible. Lemme look again | |
| 13:06:29 | giblet | finucannot: you can still add the network_requests as a field to the Instance ovo and to the RequestSpec ovo, but you not necessary to persist the content of that fields to the db, as the user of that field always get the object via rpc and the sender can regenerate the content of the network_request | |
| 13:07:44 | giblet | so in case of boot, the conductor generates the content of the network_request field and pass it down via the ReqestSpec of the Instance object | |
| 13:07:52 | giblet | similarly in case of a VM move operation | |
| 13:09:49 | giblet | persistence is itching my mind because this data is already persisted in neutron so as soon as nova also persist it we will have two possible divergent copies | |
| 13:09:49 | finucannot | Yeah, that shouldn't be an issue for RequestSpec as I don't think those are persisted. What about Instance though. Can you mark a field in a persistent object as non-persistent? | |
| 13:11:10 | giblet | finucannot: there is a request_specs table in the api db so your technical problem how to not persist a field is valid for both object | |
| 13:13:17 | giblet | finucannot: we should ask dansmith about this persistency issue but I have an idea | |
| 13:13:24 | giblet | finucannot: make the new field lazy_loaded | |
| 13:13:49 | giblet | finucannot: and try to generate the value of the field in obj_load_attr | |
| 13:14:41 | giblet | finucannot: or another idea. never set the field in _from_db_object but set it from the conductor when the data is available | |
| 13:15:48 | giblet | finucannot: also never pass this field to the db method in save() | |
| 13:16:57 | jmccarthy | @jaypipes You about ? | |
| 13:18:03 | finucannot | giblet: They're all good ideas. Let me explore them and see what I can do | |
| 13:18:16 | finucannot | Agreed on split brain issue too. | |
| 13:18:57 | giblet | finucannot: OK. I will have same issue with the bandwidth related resource requests coming from the neutron port and going to placement a_c query via nova-conductor and nova-scheduler | |
| 13:20:06 | giblet | finucannot: but you are further down the road as you already have code up | |
| 13:20:49 | finucannot | Yeah, I found I needed to write that so I could actually reason about stuff properly, otherwise I was guessing a lot of it :D | |
| 13:21:01 | finucannot | Turns out I didn't know the scheduler all that well, heh | |
| 13:24:05 | giblet | finucannot: yeah devil is in the details and in this cases writing code means seeing the details | |
| 13:26:10 | mriedem | bauwser: can you get https://review.openstack.org/#/c/566161/ ? | |
| 13:28:04 | pipesinpain | jmccarthy: yes | |
| 13:28:30 | pipesinpain | finucannot: cool, will review this morning. | |
| 13:28:49 | bauwser | mriedem: ack, looking | |
| 13:30:06 | jmccarthy | pipesinpain: Hiya ! Just wondering if you might have some insight into this bug here ? 1769131 It sounds a lot like 1666831 | |
| 13:31:21 | giblet | mriedem: both pipesinpain, fried_rice and mlavalle seems to be OK with the nova bandwidth spec. So if you are interested then this is a good time to look at it https://review.openstack.org/#/c/502306/ | |
| 13:31:50 | fried_rice | pipesinpain: elbow? | |
| 13:32:47 | pipesinpain | fried_rice: yeah :( | |
| 13:33:04 | pipesinpain | jmccarthy: will take a look after caffeinating. | |
| 13:33:07 | fried_rice | Did you golf or something? | |
| 13:33:17 | pipesinpain | fried_rice: nope. didn't do anything. | |
| 13:33:39 | fried_rice | boo. Your elbow and my knees. No idea what happened, just swelling and pain this morning. Hell to get old. | |
| 13:35:14 | giblet | pipesinpain, fried_rice: I hope both of you get better soon | |
| 13:35:21 | fried_rice | thanks giblet | |
| 13:35:56 | pipesinpain | giblet: thx man. | |
| 13:36:33 | jmccarthy | Erg vpn blipped | |
| 13:37:25 | mriedem | giblet: ok will do | |
| 13:37:25 | jmccarthy | I've been looking at driver.py but I may be misinterpreting some of the comments that are in there - this seems relevant https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L1164 | |
| 13:37:44 | giblet | mriedem: thanks | |
| 13:38:31 | pipesinpain | jmccarthy: you are using volume-backed instances, yes? | |
| 13:38:41 | pipesinpain | jmccarthy: i.e. boot-from-volume | |
| 13:39:44 | jmccarthy | I have to check what horizon says, spawning cirros with cinder, so volume backed yep | |
| 13:40:11 | pipesinpain | jmccarthy: right. so that is the behaviour by design. | |
| 13:40:38 | pipesinpain | jmccarthy: maciej had added that code to ensure that the volume wasn't removed when resizing. | |
| 13:40:57 | pipesinpain | jmccarthy: for boot-from-volume instances, you don't want to delete the original root volume, clearly :) | |
| 13:41:28 | pipesinpain | jmccarthy: I'm wondering if this is because of some thin/sparse copy-on-write stuff that Docker is doing maybe.. | |
| 13:42:05 | jmccarthy | pipesinpain: The volume itself is in cinder and seems safe enough in my case ? It's this disk.info that shows up on host that is throwing me off | |
| 13:42:22 | pipesinpain | jmccarthy: but in all honesty, I'm pretty much the worst person to ask on this :) a) I have little knowledge of the block device layer, b) I don't use (or support) boot-from-volume, and c) I don't use resize ;) | |
| 13:42:55 | pipesinpain | jmccarthy: good people to hit up are mriedem when he's online and maybe mdbooth | |
| 13:43:08 | jmccarthy | pipesinpain: Oh ok I thought you might know, since my disk.info issue sounds a lot like a bug you fixed up | |
| 13:43:43 | jmccarthy | pippesinpain: Ok I'll ask around - I'm just not sure why it would leave this file on the host afterwards, seems odd | |
| 13:43:44 | pipesinpain | jmccarthy: that was more melwitt (soon to be jgwentworth) that fixed that bug :) | |
| 13:43:48 | openstackgerrit | Kashyap Chamarthy proposed openstack/nova master: libvirt: Remove support for monitoring Intel CMT `perf` events https://review.openstack.org/565242 | |
| 13:43:57 | jmccarthy | pipesinpain: Ok ! :) Sorry for the noise | |
| 13:44:05 | pipesinpain | jmccarthy: no problem at all, man :) | |
| 13:44:38 | jmccarthy | Any of those people sitting quielty by ? ;) | |
| 13:46:14 | pipesinpain | jmccarthy: melwitt will be up in the next hour or so. mriedem will be by soon I imagine (PST and CST respectively) | |
| 13:46:36 | jmccarthy | In horizon, if it boots from image and you create a new volume at the same time in persistent storage for it, that not boot-from-volume strictly speaking, correct ? | |
| 13:46:44 | pipesinpain | jmccarthy: mdbooth is in GMT so should be around (he's likely seen "boot from volume" and run screaming for the hills) :P | |
| 13:47:04 | pipesinpain | jmccarthy: no, that's not boot-from-volume. | |
| 13:47:26 | jmccarthy | pipesinpain: That is a volume backed instance, but booted off image - does that have a special name ? | |
| 13:47:35 | pipesinpain | jmccarthy: boot-from-volume is when you supply a prebuilt bootable volume as your root disk (disk 0) for the instance | |
| 13:47:50 | pipesinpain | jmccarthy: that's a normal instance, then | |
| 13:47:59 | jmccarthy | pipesinpain: I am using volumes, but nope, not boot from vol, ok normal instances | |
| 13:48:04 | pipesinpain | jmccarthy: you mean you just added a non-bootable data volume to the instance, right? | |