| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-03-27 | |||
| 18:28:20 | cdent | s/nova$/neutron/ | |
| 18:28:41 | efried | No, if that's the part that's bothering you, let's remove neutron from the equation. | |
| 18:28:48 | efried | Just looking at nova talking to nova. | |
| 18:29:43 | efried | nova synthesizes resource+trait info from flavor, image, request spec, whatever sources. It comes up with a ResourceRequest (which is a set of RequestGroup) by the time it's ready to ask for allocation candidates. | |
| 18:29:47 | sean-k-mooney[m] | cdent: we have the same problem with numa in nova alone | |
| 18:30:16 | sean-k-mooney[m] | cdent: placement has 2 numa RP both with cpus. placement claims against one of the RPs how do i know which phyical numa node that alloaction is against | |
| 18:30:28 | cdent | so you have some state in the flavor, you use it to make a request and because you don't want to look back at that state in the flavor, you want to extend placement to transmit that state (again I'm not saying this is the worst thing ever, just trying to identify what's being done) | |
| 18:30:30 | efried | sean-k-mooney[m]: Not that, no. | |
| 18:30:53 | openstackgerrit | Balazs Gibizer proposed openstack/nova-specs master: Network bandwidth resource provider https://review.openstack.org/502306 | |
| 18:31:10 | efried | How is what's in the flavor "state"? It's a set of resource requests. | |
| 18:31:35 | efried | As currently conceived, we'd be summing up resource requests that happen to land in the same RP. | |
| 18:31:42 | cfriesen | does anyone know why we plug vifs at nova-compute startup, but then also plug them again (but don't wait for them) when powering on the instance? (for libvirt anyway) | |
| 18:31:43 | efried | That takes away information. | |
| 18:32:11 | sean-k-mooney[m] | cfriesen: linux bridge | |
| 18:32:13 | efried | If they happened to be assigned from different RPs, we would *have* that information. | |
| 18:32:28 | efried | But since they happened to be assigned from the same RP, they get rolled together, and we lose that information. | |
| 18:33:03 | sean-k-mooney[m] | cfriesen: the plug on startup is because on a reboot the tap wont be added to linux bridge because linux bridge does not preseits state. we then need to do it as part of boot a second time | |
| 18:33:05 | efried | Certainly when the virt driver is trying to grab real resources corresponding to an allocation, it needs to be able to correlate the RC+RP back to something real. | |
| 18:34:17 | gibi | mlavalle, jaypipes: I updated the bandwidth spec based on our discussion. | |
| 18:34:22 | cdent | I think may be using the term "state" far more generally than you are, but that's probably not all that germane | |
| 18:34:51 | jaypipes | gibi: ty gibi. | |
| 18:35:18 | gibi | efried: I don't have the brainpower any more to think through you mapping comments in the bandwidth spec today. I will get back to that tomorrow | |
| 18:35:19 | efried | cdent: looklook, here's another example. Today I have no way of requesting two different disks from placement. With granular, I could. I want a 1G disk and a 2G disk, so I would say: resources1=DISK_GB:1024&resources2=DISK_GB:2048. | |
| 18:35:35 | gibi | efried: thanks for pushing the discussion forward with cdent right now :) | |
| 18:35:38 | efried | cdent: In a scenario where I have sharing providers, or maybe multiple disk providers in my tree, or whatever, I may get back a candidate like { STOR_RP1: { DISK_GB: 1024 }, STOR_RP2: {DISK_GB: 2048 } } | |
| 18:36:03 | cdent | efried: I'm still trying to think this through, but it's slow going because I need to get past a fundamental issue for me: I really don't want to expose granularity in allocations... | |
| 18:36:19 | efried | cdent: And that would be fine, cause now my virt driver can tell it needs to get 1024 from STOR_RP1 and 2048 from STOR_RP2. | |
| 18:36:19 | edleafe | efried: in that scenario, there isn't anything preventing placement from getting both "disks" from the same disk RP | |
| 18:36:50 | efried | edleafe: Exactly. In which case, as currently conceived, we would report a candidate like: { STOR_RP1: { DISK_GB: 3072 } } | |
| 18:36:58 | cfriesen | sean-k-mooney[m]: I get the boot-time one...why do we need to do it again at instance boot if vifs_already_plugged=True ? | |
| 18:37:06 | cdent | didnt we have a discussion (or was it in my mind) recently about the difference between contiguous and non-contiguous resource providers? | |
| 18:37:20 | efried | cdent: Somehow virt needs to figure out that that's really a 1G disk and a 2G disk. How does he figure that out? Where does he get that information? | |
| 18:37:44 | efried | cdent: Yes, this is related to that discussion. It's why we couldn't e.g. split a VCPU:2 request across two numa nodes. | |
| 18:37:52 | edleafe | efried: what I'm saying is that there is no way to express "these must be two separate RPs" | |
| 18:38:06 | efried | edleafe: Yes, but in this case that's not something I need to express. | |
| 18:38:23 | edleafe | what granular gets you is "everything in this group must be on the same RP tree" | |
| 18:38:26 | efried | edleafe: What I *am* trying to express is that "these must be two separate *disks*". | |
| 18:38:38 | sean-k-mooney[m] | cfriesen: i dont think vifs_already_plugged will be true on the boot case. just on soft reboot | |
| 18:38:39 | efried | edleafe: ...which I expressed by putting them into separate granular groups. | |
| 18:38:39 | mriedem | are people ok with me just self approving this spec update to match the actual multiattach implementation? https://review.openstack.org/#/c/544152/ | |
| 18:38:47 | sean-k-mooney[m] | cfriesen: are you using hybrid plug? | |
| 18:39:02 | sean-k-mooney[m] | cfriesen: e.g. are you using the iptables firewall driver or conntrack | |
| 18:39:08 | efried | mriedem: Given the +1s that are on it, fine by me. | |
| 18:39:09 | cfriesen | sean-k-mooney[m]: for power-on vifs_already_plugged is true | |
| 18:39:24 | edleafe | efried: that's not what granular is for, is it? It's to ensure that all the requirements in a single group come from the same RP tree | |
| 18:39:39 | edleafe | not that every group must come from different trees | |
| 18:39:41 | mriedem | efried: done | |
| 18:39:42 | mriedem | thnaks | |
| 18:39:47 | mriedem | thanks even | |
| 18:40:07 | efried | edleafe: "what granular gets you is "everything in this group must be on the same RP tree"" -- no, "everything in this group must be on the same RP". But also it allows you to request different resources of the same class. | |
| 18:40:27 | openstackgerrit | Kashyap Chamarthy proposed openstack/nova master: libvirt: Allow to specify granular CPU feature flags https://review.openstack.org/534384 | |
| 18:40:36 | sean-k-mooney[m] | if vifs_already_plugged is true then we should not be calling plug at least in the os-vif case | |
| 18:40:50 | efried | edleafe: The canonical example being two VFs on different networks. resources1=VF:1&required1=PHYSNET_A&resources2=VF:1&required2=PHYSNET_B | |
| 18:40:59 | efried | edleafe: No way to do that in a single group. | |
| 18:41:01 | sean-k-mooney[m] | cfriesen: if libvirt is doing the plugging (kernel ovs with conntrack or noop security gorup driver) then its likely a side effect of libvirts ovs code | |
| 18:41:08 | edleafe | efried: of course | |
| 18:41:30 | efried | edleafe: And in that case, you would be assured that the resources came from different providers (because presumably one PF can't be on two physnets at once), so you would be fine. | |
| 18:41:31 | cdent | at one point I thought we had decided that if we wanted to do that the vf's had to be resource providers | |
| 18:41:46 | edleafe | efried: but if you have resources1=VF:1&required1=PHYSNET_A&resources2=VF:1&required2=PHYSNET_A you don't necessarily get different PFs | |
| 18:41:47 | efried | cdent: No, the PFs can still be providers. | |
| 18:41:56 | cfriesen | sean-k-mooney[m]: LibvirtDriver._create_domain_and_network() plugs the vifs | |
| 18:41:58 | efried | edleafe: Yes, correct. | |
| 18:42:12 | efried | edleafe: In the case of VF resources, where one VF is one resource, that's okay. | |
| 18:42:14 | edleafe | efried: but you're claiming that separate groups return separate RPs | |
| 18:42:18 | efried | edleafe: But now let's add bandwidth to the mix. | |
| 18:42:29 | efried | edleafe: No, only if traits make them split. | |
| 18:42:29 | cfriesen | sean-k-mooney[m]: unconditionally, then decides whether or not to wait for them based on vifs_already_plugged | |
| 18:42:37 | cdent | efried: I know they _can_ be, but I thought we said that for the use case you are describing, making it work requires cn .... pf -> vf all as providers | |
| 18:42:38 | mriedem | sean-k-mooney[m]: it appears the sriov bond port stuff moved to neutron specs https://review.openstack.org/#/c/506066/ and talks about new configs in nova.conf for how that bonding actually gets configured on the compute host... | |
| 18:43:02 | sean-k-mooney[m] | cfriesen: yes. for a new instance that will be the only time the vif is pluged | |
| 18:43:37 | cfriesen | sean-k-mooney[m]: right, but that happens on a resize, or a poweron, so we're plugging the vif again unnecessarily | |
| 18:43:42 | sean-k-mooney[m] | mriedem: :( really that just makes me sad. we should not have any config options needed for bonding | |
| 18:43:45 | edleafe | efried: you get separate RPs if the requirements cannot be met by a single RP. But if a single RP can satify all groups, there is nothing preventing that from being selected | |
| 18:43:51 | efried | cdent: That helps a little, but still leaves us no way to model bandwidth. | |
| 18:43:59 | edleafe | That's what I'm trying to say | |
| 18:44:04 | efried | edleafe: Yes, you are correct. | |
| 18:44:47 | efried | edleafe: And in that case, in the VF example, if you add bandwidth - which is, how did cdent put it, a "contiguous" resource? - then we can no longer figure out how much bandwidth each VF should get just from the allocation_request. | |
| 18:45:58 | sean-k-mooney[m] | cfriesen: well not nessicaily. on power we should not need to call LibvirtDriver._create_domain_and_network() the domain should still exist. it should just be "virsh start" effectivly | |
| 18:46:44 | cfriesen | sean-k-mooney[m]: power_on calls hard_reboot which calls create_domain_and_network | |
| 18:46:47 | sean-k-mooney[m] | cfriesen: on resize we will be resizeing to a new host no? so we need to set up the vif there | |
| 18:47:03 | cfriesen | sean-k-mooney[m]: can resize to same host depending on config option | |
| 18:47:04 | cdent | efried: I think I'm too tapped out to really form any solid opinion on this today. Can we rejoin in progress another time, or perhaps do it in writing? | |
| 18:47:34 | efried | cdent: Yeah, I think I'll write it up as a delta to the granular spec, gods help me. | |
| 18:47:47 | cfriesen | sean-k-mooney[m]: _hard_reboot() undefines the domain | |
| 18:47:51 | sean-k-mooney[m] | cfriesen: uh why. im sure there is a reason but thats wrong | |
| 18:47:58 | efried | cdent: Maybe a ML post would be more expeditious. | |
| 18:48:12 | sean-k-mooney[m] | cfriesen: but for resize we go via the scheduler so we cant assume we will | |
| 18:48:18 | cdent | I think email would be an easier starting point perhaps? | |
| 18:48:39 | efried | ight | |
| 18:49:02 | sean-k-mooney[m] | cfriesen: yes it does but we should not be hard rebooting when we resize or power on a vm | |
| 18:49:04 | cfriesen | sean-k-mooney[m]: hmm, does the fact that we undefine and then define a new domain affect the need to plug the vifs again? | |
| 18:49:57 | sean-k-mooney[m] | cfriesen: actully so again for linux bridge we would need to plug again | |
| 18:50:12 | sean-k-mooney[m] | when qemu exits the tap would be removed form the kernel and the linux bridge | |
| 18:50:22 | cfriesen | sean-k-mooney[m]: from the code comments: "We use _hard_reboot here to ensure that all backing files, network, and block device connections, etc. are established and available before we attempt to start the instance." | |
| 18:50:44 | sean-k-mooney[m] | so when we power on we have to add it back | |
| 18:50:44 | cfriesen | sean-k-mooney[m]: okay, that explains it then | |
| 18:50:49 | sean-k-mooney[m] | for ovs this is persited in the db and not needed | |
| 18:51:13 | cfriesen | sean-k-mooney[m]: persisted over a compute node reboot? | |
| 18:51:31 | sean-k-mooney[m] | cfriesen: is this causeing a bug for you or jsut interested in why its working this way? | |
| 18:51:59 | cfriesen | sean-k-mooney[m]: we're having performance issues and races on compute node startup in a small edge node. | |