| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-01-04 | |||
| 21:48:22 | jaypipes | why *isn't*... | |
| 21:48:24 | jaypipes | sorry | |
| 21:48:53 | ab2434_ | i dont think thats what's happening today | |
| 21:49:22 | ab2434_ | as a side note there could be multiple switches | |
| 21:49:29 | jaypipes | ab2434_: but why should Nova twist and turn to satisfy the needs of one particular VNF? :) | |
| 21:50:09 | jaypipes | ab2434_: especially when said VNF isn't *really* a VM but instead just hardware masquerading as software ;) | |
| 21:50:18 | jaypipes | but I digress... | |
| 21:50:22 | ab2434_ | well. it boils down to the mapping, having nova provide a way to map the ports | |
| 21:50:55 | ab2434_ | for the sdn-f to configure the correct switch /port for the vm | |
| 21:52:04 | mriedem | do vif tags not work here? | |
| 21:52:06 | jaypipes | ab2434_: my point is this: if the Neutron port binding can be decorated with the switch group or port tag, then that information can be passed down to Nova (in the instance PCI request) and used to identify the physical function that should be selected for the VM. what is being proposed here is the opposite of that design. the proposal here is to essentially inventory the topology and hardware for the entire deployment ahead of time and | |
| 21:52:07 | jaypipes | pre-schedule/place VMs that consume specific PCI devices on specific hardware all at once. | |
| 21:52:17 | jaypipes | it's the opposite of cloud... the opposite of on-demand. | |
| 21:52:43 | ab2434_ | ok | |
| 21:55:12 | jaypipes | ab2434_: the world that we (Nova/OpenStack, whatever) is trying to get to is a world where the VNFs/applications *describe to Nova the resources and traits that it needs* and Nova goes and finds an appropriate place for that workload to land and devices to consume. What the world that NFV is trying to force on Nova is the opposite of that: a world where a VNF says to Nova "hey, put me on node X and PF Y and use VF foo. Oh, and also pin me to | |
| 21:55:12 | jaypipes | CPUs 1-12, 18-24 and NUMA cell 0." | |
| 21:56:07 | ab2434_ | yes i see your point | |
| 21:56:09 | jaypipes | ab2434_: it's this incongruence of worldviews that is at the root of the issue I think. | |
| 21:56:58 | jaypipes | ab2434_: and yes, I understand I work for Verizon (used to be AT&T) and that certain groups at Verizon think that OpenStack should just get on with the business of being an NFVI and nothing more ;) | |
| 21:58:43 | tonyb | kashyap, mriedem: we had the beginings of one but it bitrotted. We could look at reviving it after the PTG. | |
| 21:59:24 | mriedem | tonyb: i don't think that job is useful anymore now that we're using the pike UCA by default in devstack | |
| 22:00:04 | tonyb | mriedem: Yeah the UCA stuff was always s'posed to be a POC the real plan was to have a tandem repo where we built tagged snapshots and use that | |
| 22:03:04 | tonyb | mriedem: At times there is a reasonable Fedora$current image, if ianw_pto has that working we can use the std. virt repo for Fedora also (again that was part of the plan) | |
| 22:03:48 | tonyb | mriedem, kashyap: I'm really happy to help revive that work QA, Neuton and infra all want somethign like that I just can't really be the driver | |
| 22:05:54 | jaypipes | efried: still around? | |
| 22:06:59 | jaypipes | efried: so... this will fail a functional test: https://review.openstack.org/#/c/531243/ and I'm not entirely sure why. perhaps if you're around later you could pull that patch and have a looksie? it looks like the instance creation ain't actually working. | |
| 22:07:12 | mriedem | mdbooth: so, long-term we should probably store the multiattach value on the bdm record... | |
| 22:07:15 | mriedem | the more i think about it | |
| 22:07:19 | mriedem | just like a tag | |
| 22:07:20 | jaypipes | efried: in any case, meh, will hit it later and tomorrow but if you have any time, could use your eyeballs. | |
| 22:07:46 | jaypipes | efried: it will fail the assertion here: https://review.openstack.org/#/c/531243/1/nova/tests/functional/db/test_instance_group.py on line 351 | |
| 22:07:56 | mriedem | mdbooth: and that always tells us, the volume representing this bdm was attached and supported multiattach at that time, so treat it like that until it's detached and the bdm is deleted | |
| 22:22:08 | efried | jaypipes Sorry, I'm back now. Catching up... | |
| 22:23:46 | efried | jaypipes wtf is an instance group? | |
| 22:25:07 | efried | jaypipes And are these host aggregates (as opposed to placement aggregates)? | |
| 22:26:24 | hemna_ | mriedem, hey man, I added an update to bug https://bugs.launchpad.net/nova/+bug/1452641 | |
| 22:26:25 | openstack | Launchpad bug 1452641 in OpenStack Compute (nova) "Static Ceph mon IP addresses in connection_info can prevent VM startup" [Medium,Confirmed] | |
| 22:26:39 | hemna_ | mriedem, thought you might want to take a look and see what you thought. | |
| 22:30:34 | mriedem | yikes, super rbd specific code in there | |
| 22:31:37 | mriedem | hemna_: at the ptg in denver we said we'd just always refresh the connection_info when we needed it http://lists.openstack.org/pipermail/openstack-dev/2017-September/122170.html | |
| 22:32:04 | hemna_ | yah, that was just a customer's hacked patch to get it to work | |
| 22:32:22 | lyarwood | that's imagebackend rbd btw | |
| 22:32:29 | lyarwood | not connection_info volume rbd | |
| 22:32:30 | mriedem | sure. the forced refresh_connection_info=True thing would arguably be much simpler, if it actually works | |
| 22:32:53 | hemna_ | my customer tried to live migrate and it didn't get the updated info, hence his hack | |
| 22:33:18 | lyarwood | hemna_: vms/3b97914e-3f9b-410a-b3d9-6c1a83244136_disk isn't a volume however right? | |
| 22:33:47 | jaypipes | efried: and instance group is an abomination otherwise called a server group. never mind, though... I'm gonna sleep on it and tackle manana | |
| 22:33:58 | mriedem | hemnahttps://github.com/openstack/nova/blob/master/nova/compute/manager.py#L5957 | |
| 22:34:06 | hemna_ | I'm not sure, this is a dump direct from the customer's env. | |
| 22:34:09 | mriedem | hemna_: there is a refresh_conn_info kwarg to _get_instance_block_device_info | |
| 22:34:11 | efried | jaypipes Okay. I'll play with it a bit if I get some time here. | |
| 22:34:40 | mriedem | so the idea from the ptg was just always pass refresh_conn_info=True there when we were going to do something like this | |
| 22:34:47 | hemna_ | mriedem, is that an option at nova cmdln time ? | |
| 22:34:52 | mriedem | no | |
| 22:35:04 | mriedem | it was a way to do this w/o any api changes | |
| 22:35:17 | lyarwood | mriedem: the issue isn't with the rbd volume, but the imagebackend rbd images | |
| 22:35:34 | mriedem | i don't know what that means | |
| 22:35:51 | mriedem | i thought the issue was stale rbd information in the connection_info, which nova gets from cinder | |
| 22:35:53 | lyarwood | mriedem: volumes/volume-6d04520d-0029-499c-af81-516a7ba37a54 is the volume | |
| 22:35:56 | mriedem | and uses to populate the disk config | |
| 22:36:22 | lyarwood | mriedem: not for ephemerial rbd images, it's all hard coded from the local nova.conf iirc | |
| 22:36:37 | mriedem | lyarwood: yeah i'm not talking about ephemeral, | |
| 22:36:38 | mriedem | just volumes | |
| 22:37:13 | mriedem | this https://github.com/openstack/nova/blob/master/nova/virt/libvirt/volume/net.py#L56 | |
| 22:37:22 | lyarwood | mriedem: yeah that appears to be updated in the LM flow | |
| 22:37:32 | lyarwood | mriedem: <source protocol='rbd' name='volumes/volume-6d04520d-0029-499c-af81-516a7ba37a54'> <-- this one is changed, new ips | |
| 22:37:44 | mriedem | right, because the live migration flow calls _get_volume_config | |
| 22:37:50 | mriedem | using the bdm.connection_info | |
| 22:37:53 | mriedem | which is stale | |
| 22:38:00 | hemna_ | yup | |
| 22:38:01 | lyarwood | yup | |
| 22:38:06 | mriedem | what we said in denver, | |
| 22:38:18 | mriedem | was when we start live migration, and get the bdms, we call cinder to refresh the connection info | |
| 22:38:37 | mriedem | by pass refresh_conn_info=True to https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L1683 | |
| 22:38:52 | mriedem | that will make a new os-initialize_connection call to cinder and return the latest connection_info | |
| 22:39:07 | mriedem | then we save that on the bdm.connection_info | |
| 22:39:18 | mriedem | for reasons, 'ive just never written that patch, | |
| 22:39:26 | mriedem | but i also have concerns about the new attach flow making that no longer work | |
| 22:39:33 | hemna_ | :( | |
| 22:39:46 | mriedem | with the new flow, nova calls cinder to get the attachment record which has the connection_info stored in the cinder db, | |
| 22:39:58 | mriedem | so i don't think it actually creates a new connection (export?) so we wouldn't refresh | |
| 22:40:05 | hemna_ | which could be stale... | |
| 22:40:12 | mriedem | this https://github.com/openstack/nova/blob/master/nova/virt/block_device.py#L557 | |
| 22:40:20 | hemna_ | in this case we just moved the stale info from nova to cinder, and cinder is stale? | |
| 22:40:39 | mriedem | hemna_: right, if cinder doesn't update that, and just returns whta's in the db, it would be stale - and we'd have the same problem as using the stale nova db info | |
| 22:40:43 | mriedem | yes | |
| 22:40:45 | mriedem | exactly | |
| 22:40:51 | hemna_ | bleh | |
| 22:42:22 | mriedem | yeah idk, nova could still call os-initialize_connection if we wanted to, | |
| 22:42:31 | mriedem | i don't know if that would update the attachment record in cinder or not | |
| 22:43:09 | hemna_ | so I thought the purpose of storing the conn info in cinder's db is for force delete time when nova doesn't know anything | |
| 22:43:12 | mriedem | otherwise we'd likely need like a refresh=True query parameter to GET /attachments/{id} | |
| 22:43:37 | mriedem | hemna_: that's part of it yeah | |
| 22:43:38 | hemna_ | in which case, can't cinder always refresh that if nova asks for it? | |
| 22:43:45 | hemna_ | I dunno | |
| 22:43:55 | mriedem | but how does cinder know that we're asking for a refresh? | |
| 22:43:58 | mriedem | or just a simple read-only get | |
| 22:44:13 | hemna_ | true | |
| 22:44:13 | mriedem | this is where we got talking about API changes to force a refresh | |
| 22:44:52 | hemna_ | I guess this could affect other cinder backends too | |