| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-05-04 | |||
| 08:49:53 | bauwser | no worries, I think I have all that I want | |
| 08:49:58 | giblet | OK | |
| 08:50:06 | bauwser | we discover the versions with the shell | |
| 08:50:16 | bauwser | but we don't with the python bindings directly | |
| 09:51:44 | rabel_ | hi there. I just saw that add-floating-ip action is deprecated in compute api. but how is a floating ip associated to an instance then? | |
| 09:55:48 | rabel_ | found it. network api https://developer.openstack.org/api-ref/network/v2/#floating-ips-floatingips | |
| 10:48:42 | openstackgerrit | Stephen Finucane proposed openstack/nova-specs master: Add 'numa-aware-vswitches' spec https://review.openstack.org/541290 | |
| 10:49:20 | finucannot | jaypipes, giblet: OK, I think that spec is good to go now. I've clarified a lot of the physnet/provider net stuff and fixed the diagrams https://review.openstack.org/541290 | |
| 10:55:42 | giblet | finucannot: looking | |
| 11:01:59 | openstackgerrit | Surya Seetharaman proposed openstack/nova stable/queens: Make association_refresh configurable https://review.openstack.org/566288 | |
| 11:08:37 | giblet | finucannot: your spec looks good to me | |
| 11:50:03 | openstackgerrit | Balazs Gibizer proposed openstack/nova-specs master: Placement: any traits in allocation_candidate query https://review.openstack.org/565730 | |
| 11:50:31 | openstackgerrit | Balazs Gibizer proposed openstack/nova-specs master: Placement: support mixing required traits with any traits https://review.openstack.org/565741 | |
| 12:02:40 | openstackgerrit | Balazs Gibizer proposed openstack/nova-specs master: Placement: support mixing required traits with any traits https://review.openstack.org/565741 | |
| 12:06:22 | jmccarthy | Hmm I still have this issue where after a cold migration, a disk.info file shows up in instance dir and is left on source host | |
| 12:06:43 | jmccarthy | I'm using kolla stable/queens - does this log seem right ? | |
| 12:06:45 | jmccarthy | http://paste.openstack.org/show/720358/ | |
| 12:07:13 | jmccarthy | It's when the resize/migration is confirmed that it shows up | |
| 12:09:40 | jmccarthy | This part looks good CMD "rm -rf /var/lib/nova/instances/371e669b-0f15-49f2-9a84-bd1e89f34294_resize" returned: 0 | |
| 12:10:00 | jmccarthy | But then a lock is acquired on disk.info *after that ? and it's left there .. | |
| 12:21:22 | jmccarthy | After migrating an instance, is /var/lib/nova/instances/371e669b-0f15-49f2-9a84-bd1e89f34294/disk.info supposed to written out on the source host ? | |
| 12:21:46 | jmccarthy | s/After/After cold-/ | |
| 12:31:19 | openstackgerrit | Merged openstack/nova master: libvirt: Drop BAD_LIBVIRT_CPU_POLICY_VERSIONS https://review.openstack.org/564012 | |
| 12:31:25 | openstackgerrit | Merged openstack/nova master: libvirt: Drop MIN_QEMU_POSTCOPY_VERSION https://review.openstack.org/565724 | |
| 12:31:33 | openstackgerrit | Merged openstack/nova master: libvirt: Drop MIN_LIBVIRT_REALTIME_VERSION https://review.openstack.org/565707 | |
| 12:37:29 | finucannot | giblet: One thought I had on numa-aware-vswitches: how do we deal with changes in the networks attached to the guest when migrating? I think you touched on that here https://review.openstack.org/#/c/541290/8/specs/rocky/approved/numa-aware-vswitches.rst@243 | |
| 12:38:25 | giblet | finucannot: yes. even if the network itself is not changing the NUMA affinity of the devices providing access to the given network can be different on different hosts | |
| 12:39:01 | giblet | finucannot: and the actual physnet can change if we have multiprovider networks with mutliple segments having different physnets | |
| 12:39:23 | giblet | finucannot: but the later does not supported today by nova anyhow | |
| 12:39:28 | finucannot | OK, I don't think that's actually an issue. In that case, we'd be regenerating the guest NUMA topology for the new host and that would use the new host's NUMA-network mapping | |
| 12:39:41 | finucannot | Also, I think we decided multi-segment is out-of-scope | |
| 12:39:59 | giblet | finucannot: multi-segment is out of scope, I agree | |
| 12:40:23 | finucannot | I'm more concerned about the actual networks attached. We've said that a user has to allocate networks at instance creation time so we can do NUMA affinity | |
| 12:40:51 | finucannot | However, you can attach/detach networks to/from a running instance, right? | |
| 12:40:52 | giblet | finucannot: could you be bit more specific how the network changes in your scenario | |
| 12:41:00 | giblet | finucannot: ahh, yes | |
| 12:41:28 | finucannot | So I boot an instance attached to network 'foo' and then, once it's running, detach from 'foo' and attach to 'bar' | |
| 12:41:37 | giblet | finucannot: when you attach a new port / network that might affect the necessary affinity | |
| 12:42:09 | finucannot | Yeah, exactly. We can't do anything about that while the instance is on the same host, but what about if we migrate/rebuild? | |
| 12:42:47 | giblet | finucannot: you can actually check if the port being attached creates a contradiction with the existing affinity of the instance | |
| 12:43:01 | giblet | finucannot: and you might reject the attach | |
| 12:43:45 | giblet | finucannot: but I agree that when you migrate you have to take every port / network into account | |
| 12:43:58 | giblet | finucannot: including those that was attached after the boot | |
| 12:44:40 | finucannot | Yeah, we could do that. I guess that could/should be a configurable policy option down the line | |
| 12:45:35 | finucannot | But yeah, I'm thinking I should regenerate the NetworkRequestList object attached to the instance/request spec when migrating to reflect the network topology pre-migration | |
| 12:45:42 | finucannot | That might even happen already. I should check | |
| 12:46:24 | giblet | finucannot: I think regenerating the information from Neutron is the safe solution | |
| 12:47:15 | giblet | finucannot: I would even go that far that don't persist the NetworkRequestList but simply regenerate when it is needed | |
| 12:52:15 | finucannot | You need to though so that you can use it during claiming. Without that we have no way to figure out what's necessary https://review.openstack.org/#/c/541290/8/specs/rocky/approved/numa-aware-vswitches.rst@243 | |
| 12:52:21 | finucannot | whoops | |
| 12:52:28 | finucannot | https://review.openstack.org/#/c/564449/1/nova/compute/claims.py | |
| 12:53:34 | finucannot | It also needs to be stored in the RequestSpec object so that we don't need to query neutron from the filters (which we can't do because we don't have correct context) | |
| 12:55:14 | giblet | finucannot: does the RequestSpec loaded from the db in the scheduler or passed via rpc? | |
| 12:55:31 | finucannot | The former, to the best of my knowledge | |
| 12:56:00 | giblet | finucannot: then I agree that you have to store the NetworkREquestList to the db along with the RequestSpec | |
| 12:56:43 | finucannot | Yeah, I mapped the whole thing here http://paste.openstack.org/show/720365/ | |
| 12:59:58 | giblet | finucannot: I think the RequestSpec is passed to the scheduler via rpc https://github.com/openstack/nova/blob/5d97937c3c56a3e240a3350a7a9f0e3dcb954c52/nova/scheduler/rpcapi.py#L132 | |
| 13:00:34 | giblet | the spec_obj there is a RequestSpec obj | |
| 13:03:33 | giblet | finucannot: also the build_and_run_instance leading to the claim gets the instance object via rpc | |
| 13:04:46 | finucannot | Yup, so if I wanted to, for example, pass an additional 'network_requests' parameter to 'claim()', I guess I'd have to bump the RPC version | |
| 13:05:01 | finucannot | I'm pretty sure I looked at that though and it wasn't possible. Lemme look again | |
| 13:06:29 | giblet | finucannot: you can still add the network_requests as a field to the Instance ovo and to the RequestSpec ovo, but you not necessary to persist the content of that fields to the db, as the user of that field always get the object via rpc and the sender can regenerate the content of the network_request | |
| 13:07:44 | giblet | so in case of boot, the conductor generates the content of the network_request field and pass it down via the ReqestSpec of the Instance object | |
| 13:07:52 | giblet | similarly in case of a VM move operation | |
| 13:09:49 | giblet | persistence is itching my mind because this data is already persisted in neutron so as soon as nova also persist it we will have two possible divergent copies | |
| 13:09:49 | finucannot | Yeah, that shouldn't be an issue for RequestSpec as I don't think those are persisted. What about Instance though. Can you mark a field in a persistent object as non-persistent? | |
| 13:11:10 | giblet | finucannot: there is a request_specs table in the api db so your technical problem how to not persist a field is valid for both object | |
| 13:13:17 | giblet | finucannot: we should ask dansmith about this persistency issue but I have an idea | |
| 13:13:24 | giblet | finucannot: make the new field lazy_loaded | |
| 13:13:49 | giblet | finucannot: and try to generate the value of the field in obj_load_attr | |
| 13:14:41 | giblet | finucannot: or another idea. never set the field in _from_db_object but set it from the conductor when the data is available | |
| 13:15:48 | giblet | finucannot: also never pass this field to the db method in save() | |
| 13:16:57 | jmccarthy | @jaypipes You about ? | |
| 13:18:03 | finucannot | giblet: They're all good ideas. Let me explore them and see what I can do | |
| 13:18:16 | finucannot | Agreed on split brain issue too. | |
| 13:18:57 | giblet | finucannot: OK. I will have same issue with the bandwidth related resource requests coming from the neutron port and going to placement a_c query via nova-conductor and nova-scheduler | |
| 13:20:06 | giblet | finucannot: but you are further down the road as you already have code up | |
| 13:20:49 | finucannot | Yeah, I found I needed to write that so I could actually reason about stuff properly, otherwise I was guessing a lot of it :D | |
| 13:21:01 | finucannot | Turns out I didn't know the scheduler all that well, heh | |
| 13:24:05 | giblet | finucannot: yeah devil is in the details and in this cases writing code means seeing the details | |
| 13:26:10 | mriedem | bauwser: can you get https://review.openstack.org/#/c/566161/ ? | |
| 13:28:04 | pipesinpain | jmccarthy: yes | |
| 13:28:30 | pipesinpain | finucannot: cool, will review this morning. | |
| 13:28:49 | bauwser | mriedem: ack, looking | |
| 13:30:06 | jmccarthy | pipesinpain: Hiya ! Just wondering if you might have some insight into this bug here ? 1769131 It sounds a lot like 1666831 | |
| 13:31:21 | giblet | mriedem: both pipesinpain, fried_rice and mlavalle seems to be OK with the nova bandwidth spec. So if you are interested then this is a good time to look at it https://review.openstack.org/#/c/502306/ | |
| 13:31:50 | fried_rice | pipesinpain: elbow? | |
| 13:32:47 | pipesinpain | fried_rice: yeah :( | |
| 13:33:04 | pipesinpain | jmccarthy: will take a look after caffeinating. | |
| 13:33:07 | fried_rice | Did you golf or something? | |
| 13:33:17 | pipesinpain | fried_rice: nope. didn't do anything. | |
| 13:33:39 | fried_rice | boo. Your elbow and my knees. No idea what happened, just swelling and pain this morning. Hell to get old. | |
| 13:35:14 | giblet | pipesinpain, fried_rice: I hope both of you get better soon | |
| 13:35:21 | fried_rice | thanks giblet | |
| 13:35:56 | pipesinpain | giblet: thx man. | |
| 13:36:33 | jmccarthy | Erg vpn blipped | |
| 13:37:25 | mriedem | giblet: ok will do | |
| 13:37:25 | jmccarthy | I've been looking at driver.py but I may be misinterpreting some of the comments that are in there - this seems relevant https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L1164 | |
| 13:37:44 | giblet | mriedem: thanks | |
| 13:38:31 | pipesinpain | jmccarthy: you are using volume-backed instances, yes? | |