| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-03-27 | |||
| 18:42:37 | cdent | efried: I know they _can_ be, but I thought we said that for the use case you are describing, making it work requires cn .... pf -> vf all as providers | |
| 18:42:38 | mriedem | sean-k-mooney[m]: it appears the sriov bond port stuff moved to neutron specs https://review.openstack.org/#/c/506066/ and talks about new configs in nova.conf for how that bonding actually gets configured on the compute host... | |
| 18:43:02 | sean-k-mooney[m] | cfriesen: yes. for a new instance that will be the only time the vif is pluged | |
| 18:43:37 | cfriesen | sean-k-mooney[m]: right, but that happens on a resize, or a poweron, so we're plugging the vif again unnecessarily | |
| 18:43:42 | sean-k-mooney[m] | mriedem: :( really that just makes me sad. we should not have any config options needed for bonding | |
| 18:43:45 | edleafe | efried: you get separate RPs if the requirements cannot be met by a single RP. But if a single RP can satify all groups, there is nothing preventing that from being selected | |
| 18:43:51 | efried | cdent: That helps a little, but still leaves us no way to model bandwidth. | |
| 18:43:59 | edleafe | That's what I'm trying to say | |
| 18:44:04 | efried | edleafe: Yes, you are correct. | |
| 18:44:47 | efried | edleafe: And in that case, in the VF example, if you add bandwidth - which is, how did cdent put it, a "contiguous" resource? - then we can no longer figure out how much bandwidth each VF should get just from the allocation_request. | |
| 18:45:58 | sean-k-mooney[m] | cfriesen: well not nessicaily. on power we should not need to call LibvirtDriver._create_domain_and_network() the domain should still exist. it should just be "virsh start" effectivly | |
| 18:46:44 | cfriesen | sean-k-mooney[m]: power_on calls hard_reboot which calls create_domain_and_network | |
| 18:46:47 | sean-k-mooney[m] | cfriesen: on resize we will be resizeing to a new host no? so we need to set up the vif there | |
| 18:47:03 | cfriesen | sean-k-mooney[m]: can resize to same host depending on config option | |
| 18:47:04 | cdent | efried: I think I'm too tapped out to really form any solid opinion on this today. Can we rejoin in progress another time, or perhaps do it in writing? | |
| 18:47:34 | efried | cdent: Yeah, I think I'll write it up as a delta to the granular spec, gods help me. | |
| 18:47:47 | cfriesen | sean-k-mooney[m]: _hard_reboot() undefines the domain | |
| 18:47:51 | sean-k-mooney[m] | cfriesen: uh why. im sure there is a reason but thats wrong | |
| 18:47:58 | efried | cdent: Maybe a ML post would be more expeditious. | |
| 18:48:12 | sean-k-mooney[m] | cfriesen: but for resize we go via the scheduler so we cant assume we will | |
| 18:48:18 | cdent | I think email would be an easier starting point perhaps? | |
| 18:48:39 | efried | ight | |
| 18:49:02 | sean-k-mooney[m] | cfriesen: yes it does but we should not be hard rebooting when we resize or power on a vm | |
| 18:49:04 | cfriesen | sean-k-mooney[m]: hmm, does the fact that we undefine and then define a new domain affect the need to plug the vifs again? | |
| 18:49:57 | sean-k-mooney[m] | cfriesen: actully so again for linux bridge we would need to plug again | |
| 18:50:12 | sean-k-mooney[m] | when qemu exits the tap would be removed form the kernel and the linux bridge | |
| 18:50:22 | cfriesen | sean-k-mooney[m]: from the code comments: "We use _hard_reboot here to ensure that all backing files, network, and block device connections, etc. are established and available before we attempt to start the instance." | |
| 18:50:44 | cfriesen | sean-k-mooney[m]: okay, that explains it then | |
| 18:50:44 | sean-k-mooney[m] | so when we power on we have to add it back | |
| 18:50:49 | sean-k-mooney[m] | for ovs this is persited in the db and not needed | |
| 18:51:13 | cfriesen | sean-k-mooney[m]: persisted over a compute node reboot? | |
| 18:51:31 | sean-k-mooney[m] | cfriesen: is this causeing a bug for you or jsut interested in why its working this way? | |
| 18:51:59 | cfriesen | sean-k-mooney[m]: we're having performance issues and races on compute node startup in a small edge node. | |
| 18:52:10 | cfriesen | sean-k-mooney[m]: trying to figure out what my options are | |
| 18:52:12 | sean-k-mooney[m] | cfriesen: ya the port is stored in the ovs db so when the tap shows up the revalidator treads in the ovs-vswitchd will detect it and add it | |
| 18:52:43 | openstackgerrit | Merged openstack/nova-specs master: Amend volume multi-attach spec https://review.openstack.org/544152 | |
| 18:53:06 | sean-k-mooney[m] | cfriesen: i assume you have configured nova to start all instance on server reboot | |
| 18:53:50 | cfriesen | sean-k-mooney[m]: nope, there's another component that's in charge of that. but nova will plug all the vifs on nova-compute startup even if the instances are powered off. | |
| 18:54:48 | cfriesen | that slows down neutron, which causes it to be delayed responding to something else, which times out and does exponential backoff...and the end result is that one or two instance take a long time to become pingable | |
| 18:55:26 | sean-k-mooney[m] | cfriesen: ah ok. i still think its a bug that nova plugs the vifs on compute agent start up. we should proably revisiti if its really needed. as you said we unconditionally plug the vifs when starting a vm so not sure what edge case still requires it | |
| 18:56:31 | sean-k-mooney[m] | the current bevior dates back from nova-networks with linuxbridge but we do several things differently now | |
| 18:57:59 | cfriesen | sean-k-mooney[m]: if we did get rid of the vif plugging at startup, would we have to wait for the network events in _create_domain_and_network() when powering on? | |
| 18:58:48 | sean-k-mooney[m] | cfriesen: we cant wait because only ovs with the neutron agents sends those events correctly | |
| 18:59:52 | sean-k-mooney[m] | cfriesen: linux bridge does not send them reliably and odl only send them when the port is first bound not when port are plugged | |
| 19:00:28 | cfriesen | sean-k-mooney[m]: we do wait on instance spawn though, is that different? | |
| 19:01:24 | sean-k-mooney[m] | cfriesen: we have no ideay what happens for all the rest of the backend but that is why we dont wiat currently | |
| 19:02:12 | cfriesen | sean-k-mooney[m]: kay, thanks for the info. | |
| 19:02:37 | sean-k-mooney[m] | on instance spawn you meen on first boot. and yes that would be different as all backend send a vif plugged event in that case | |
| 19:03:21 | sean-k-mooney[m] | cfriesen: by the way its oke for us to wait for os-vif to plug the vif we just cant wait for neutron to say its wired up. we would like to but all the ml2 dirvers do different things | |
| 19:03:25 | mlavalle | thanks gibi :-) | |
| 19:15:24 | openstackgerrit | Matt Riedemann proposed openstack/nova-specs master: Non-unique network names in Servers IPs API response https://review.openstack.org/521392 | |
| 19:16:14 | mriedem | sean-k-mooney[m]: that reminds me, you know how for the new port binding live migration spec we talked about adding a wait on the source host for the vifs to be plugged on the dest host during pre_live_migration? | |
| 19:16:25 | mriedem | i think that is likely not possible now because of ODL | |
| 19:16:39 | mriedem | since ODL won't send a vif plugged event until the port binding changes, right? | |
| 19:16:43 | mriedem | which currently happens in post live migration | |
| 19:17:19 | sean-k-mooney[m] | mriedem: yes didnt you already remove that in your code. we were talking about this last week i think? | |
| 19:17:22 | mriedem | i'm almost inclined to add a [workarounds] config option to disable that wait for vif plugged, but enable the wait by default | |
| 19:17:28 | mriedem | it's a todo in the code right now | |
| 19:17:40 | mriedem | so for ovs and LB we'd default to wait, | |
| 19:17:46 | mriedem | but if you're using ODL, you disable the wait | |
| 19:18:57 | sean-k-mooney[m] | ya odl wont send the event until there is a neutron port update which will only happen when we activate the new building as a result of the change to the host-id in the vif binding_details dicts | |
| 19:19:06 | sean-k-mooney[m] | mriedem: we cant tell if its odl or ovs from nova so we cant wait | |
| 19:19:21 | sean-k-mooney[m] | we can wait for the compute to tell use it has plugged the interface we just cant wait for neutron to say its wired it up | |
| 19:19:23 | mriedem | sean-k-mooney[m]: i know we can't | |
| 19:19:28 | mriedem | but the operator can | |
| 19:20:00 | mriedem | so for live migration, we can default to wait, assuming you're using a networking backend that doesn't suck | |
| 19:20:12 | mriedem | and if you are, then you need to configure nova-compute to not wait for vif plugged events during live migration | |
| 19:20:44 | mriedem | to other specs cores, i think https://review.openstack.org/#/c/521392/ is a no-brainer | |
| 19:20:58 | sean-k-mooney[m] | oh yes we could have a per compute config option or i guess a could wide one for the conductor | |
| 19:21:11 | mriedem | conductor doesn't wait | |
| 19:21:12 | mriedem | compute does | |
| 19:21:44 | sean-k-mooney[m] | mriedem: so in the prelive migrate would we have the dest read the config value and stick it in the migration data | |
| 19:22:01 | mriedem | umm | |
| 19:22:09 | mriedem | i figured the source would read the config since the source host is what's waiting | |
| 19:22:21 | mriedem | if the wait has to depend on the dest, then yeah maybe that has to go into the migrate data object | |
| 19:22:31 | mriedem | if the dest host is using ODL but the source is using OVS | |
| 19:23:26 | sean-k-mooney[m] | well since the dest might have a different network backend to the souce we should get the dest value no? | |
| 19:23:37 | mriedem | probably | |
| 19:24:54 | sean-k-mooney[m] | we proabbly also want to default to not waiting to not break upgrades as that is the default behavior today | |
| 19:24:59 | sean-k-mooney[m] | we can then change the default after 1 release | |
| 19:25:20 | mriedem | oh i suppose | |
| 19:25:44 | sean-k-mooney[m] | or put somthing in the neutorn port binding so neutron can tell use if we should wait or not in the future | |
| 19:25:46 | mriedem | older dest computes wouldn't send the value anyway | |
| 19:26:10 | mriedem | neutron telling us if waiting is safe would be great, but i'm not driving that on the neutron side | |
| 19:27:00 | sean-k-mooney[m] | well the sepc said we would only use the new flow if they both supported it but ya old dest would not send it so its safer to assume dont wait | |
| 19:28:01 | sean-k-mooney[m] | mriedem: honestly i dont think that would be too hard to enable in neutron. i dont know if i have time to look at it but it should be a minor enough api extention | |
| 19:28:20 | mriedem | i mention the old dest compute thing because we had also talked about doing this separately from the blueprint as a bug fix so we could backport it | |
| 19:28:28 | mriedem | but if it requires changes to the versioned object, we can't really backport it | |
| 19:28:45 | mriedem | that was also before we knew that ODL didn't send events | |
| 19:29:01 | sean-k-mooney[m] | basicaly i think we could have a singel key in the portbining dict. vif_plugged_event_policy that could be never,on_bind or on_plug then just hard code it for each of the ml2 drivers | |
| 19:30:03 | sean-k-mooney[m] | mriedem: ah ya that makes sense | |
| 19:32:24 | sean-k-mooney[m] | mriedem: do you want me to open an rfe but for the event stuff and we can see what the neutron team says? it would not be a hard dependecy but it would allow us to know in the future if we can wait | |
| 19:33:13 | cdent | edleafe: responded to your coments on forbidden traits in rp in db. I think you're missing a point or I've explained things very poorly | |
| 19:33:57 | openstackgerrit | Merged openstack/nova-specs master: Non-unique network names in Servers IPs API response https://review.openstack.org/521392 | |
| 19:34:10 | mriedem | sean-k-mooney[m]: sure | |
| 19:37:57 | efried | jaypipes, cdent, edleafe: Is there anything preventing two records in the allocations table having the same consumer+rp+rc? | |
| 19:39:29 | cdent | efried: in the table itself, no, but the interface on the object does a full wipe before it does any set (for a given consumer) | |
| 19:40:08 | efried | cdent: duly noted. | |
| 19:40:41 | mriedem | do we have a bp or anything anymore for supporting shared storage pools in nova? there was a bp but i can't find it now, maybe it was marked obsolete? | |
| 19:41:08 | edleafe | cdent: ok, but it seems like overkill. No one would be sending !TRAIT now, so I don't quite get the need to support it | |