| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-02-26 | |||
| 10:18:18 | gibi | rm_work: responded in https://review.opendev.org/#/c/709280 I'm happy to help with technicalities but I missing the what the feature is supposed to achive and how the existing impl in neutron works | |
| 10:19:51 | rm_work | gibi: just read it, thanks! | |
| 10:20:07 | rm_work | so, the key is that it's not really a feature for end users at all -- it's for service integration cases | |
| 10:20:37 | rm_work | the octavia use-case is here: https://review.opendev.org/#/c/706153/ | |
| 10:21:07 | rm_work | gibi: as it is, neutron has been creating aggregates for each segment_it in a routed network automatically, for like... 3 cycles already maybe? | |
| 10:24:12 | rm_work | so for example, if i take a look at my Staging environment's aggregate list, it looks like this: http://paste.openstack.org/show/7YyK5FtL2OiXgErAjMbM/ | |
| 10:24:41 | rm_work | that aggregate was automatically created by neutron because it was configured as a routed network with a segment and those HVs were assigned to that segment in neutron | |
| 10:25:56 | rm_work | gibi: i don't think there's really any further work to be done -- this filter *just works* already for the intended purpose -- allowing services to hint to nova which aggregate to schedule to, based on network segment | |
| 10:26:05 | rm_work | the real challenge is just getting it merged :) | |
| 10:26:51 | rm_work | I know of at least two companies running this filter on live clouds already, one of them for many years | |
| 10:27:57 | rm_work | the second use-case is live-migrate, and I already have the basic patch for it (again, has been running for quite a while in some clouds) but I'm not spending the effort to get it all prettified yet until I have some confidence that this filter might be accepted | |
| 10:28:35 | gibi | rm_work: reading back ... | |
| 10:29:13 | rm_work | this is it though: http://paste.openstack.org/show/Z8ZYksr4uisFUddFoyOu/ | |
| 10:29:37 | rm_work | basically all I could ask from you is a +1 instead of a -1 if you understand what I'm going for here | |
| 10:30:26 | rm_work | trying to unlock useful features that have been stuck downstream because people didn't have the energy to do what I'm doing and post them upstream with tests and docs, and convince people to review :D | |
| 10:31:57 | gibi | rm_work: so the new scheduler hint routed_segments is a list of segment ids | |
| 10:32:03 | rm_work | yes | |
| 10:32:26 | gibi | rm_work: and the filter try to find an aggregate that has a name that ends with such segment id | |
| 10:32:28 | rm_work | users can't actually see segment_ids, so it's not really useful for the end-user side -- it's designed for service interactions | |
| 10:32:48 | rm_work | yes, as that's the neutron spec to auto-create aggregates with segment_ids with that name format | |
| 10:33:04 | gibi | rm_work: do you have a link to that neutron spec? | |
| 10:33:09 | rm_work | let me look | |
| 10:33:46 | gibi | so how the 'service' that calls nova knows the segment id it needs to pass in the scheduler hint? | |
| 10:36:21 | rm_work | nova side: https://specs.openstack.org/openstack/nova-specs/specs/newton/implemented/neutron-routed-networks.html#proposed-change | |
| 10:36:43 | rm_work | neutron side: https://specs.openstack.org/openstack/neutron-specs/specs/newton/routed-networks.html | |
| 10:36:50 | gibi | thanks | |
| 10:37:17 | gibi | I have to read up | |
| 10:37:21 | rm_work | gibi: so for the octavia case, the issue is that we need both of the service-vms we build to be in the same segment as the VIP (an unbound neutron port) we create | |
| 10:37:37 | rm_work | so, we simply look at that port, then look at the subnet it's on, and get the segment_id, then boot using that | |
| 10:37:53 | gibi | rm_work: thanks ^^ | |
| 10:38:55 | rm_work | for the live-migrate case in nova, it's basically the same -- all the ports need to be pluggable to the new VM, so the new VM needs to have access to the same segments -- so it looks at all the ports on the old VM, gets the subnets->segments from them, and then does the migrate to a host that schedules with all of the relevant segments | |
| 10:39:38 | rm_work | you probably don't have to read the ENTIRETY of both specs to get the idea, and I don't know whether they ended up matching the implementation EXACTLY, but I can answer questions | |
| 10:40:47 | ileixe | rm_work: gibi: sorry to hijack, may I ask one more question? | |
| 10:40:54 | rm_work | o/ | |
| 10:41:05 | gibi | rm_work: thanks. This description helped me a lot | |
| 10:41:24 | ileixe | You said normal users were unaware of segments, does it mean normal user does not make VM using routed network? | |
| 10:41:41 | rm_work | when routed-networks are in use, the user is really not even aware of it | |
| 10:41:53 | rm_work | it's all handled inside nova/neutron | |
| 10:42:02 | rm_work | as far as the user is aware, they simply ask for a VM on a network, and it happens | |
| 10:42:03 | ileixe | Yes, so what happens to the normal user? | |
| 10:42:18 | ileixe | then they are not using routed network, right? | |
| 10:42:21 | gibi | rm_work: can we trade? :) I will support your way forward to either have the filter or the placement based sollution in ussuri, in reaturn you can help me reviewing the tempest patch? | |
| 10:42:34 | rm_work | so in the normal workflow, the user will do a nova boot, and it will go for scheduling, schedule to any HV, and then network plugging will be deferred to neutron instead of nova picking the IP | |
| 10:42:55 | rm_work | then neutron will look at the HV it was scheduled to, pick an appropriate segment, and do the IP assignment/plug on the neutron side | |
| 10:44:04 | rm_work | routed networks is kinda just... something that a cloud either needs (and thus uses) or doesn't. in a cloud that's routed-network enabled, there really aren't *non-routed* networks, usually (at least, not that i've seen) | |
| 10:44:05 | gibi | so for first boot neutron will select the segment, but for live migrate nova should select a host that supports the same segment | |
| 10:44:10 | rm_work | it's really a network implementation choice | |
| 10:44:17 | rm_work | gibi: correct | |
| 10:44:31 | rm_work | currently, live-migrate cannot work in a routed-network enabled cloud | |
| 10:44:44 | gibi | rm_work: wooot! Finally I undestood it :) | |
| 10:45:04 | rm_work | because the scheduling doesn't understand it, and will pick some random HV based on other scheduling criteria, and then that HV may not actually be in the correct aggregate for plugging the old port :) | |
| 10:45:57 | gibi | plugging the old port in live migrate case fails becuase the port is already assigned to segment? while in the normal boot case the port is in a deferred state so the plug wont fail | |
| 10:46:04 | rm_work | yes | |
| 10:46:16 | rm_work | live-migrate needs to keep the old address (port) | |
| 10:46:25 | rm_work | a segment is a subnet attribute | |
| 10:46:33 | rm_work | port has an address (which is part of a subnet) | |
| 10:46:54 | sean-k-mooney | gibi: pluging the port on live migration would work if we live migrated to the same netwrok segment | |
| 10:46:56 | rm_work | so one Network can have many Subnets, each which is a separate segment | |
| 10:47:03 | gibi | ok, so during cold migrate if the port needs to be plugged to another segment then it will change IP address but it si not a problem as the guest is rebooted anyhow | |
| 10:47:12 | ileixe | "neutron will look at the HV it was scheduled to, pick an appropriate segment, and do the IP assignment/plug on the neutron side" <- that's the point that I was unaware of | |
| 10:47:15 | rm_work | sean-k-mooney: right, so that's what we're talking about -- this simple filter allows for that | |
| 10:47:22 | ileixe | Thanks for explanation I will look at it | |
| 10:47:23 | sean-k-mooney | gibi: it should not change ip | |
| 10:47:57 | gibi | sean-k-mooney: but if it changes segment then it will change IP | |
| 10:48:04 | rm_work | I actually haven't even paid attention to cold-migrate, we don't do it either, or care :D it may have the same issue as live-migrate | |
| 10:48:14 | gibi | ahh I see | |
| 10:48:20 | rm_work | I am not sure | |
| 10:48:22 | sean-k-mooney | that my point its not vaild for the port to change its ip on a move operation so it cannot change segment | |
| 10:48:30 | rm_work | right | |
| 10:48:32 | sean-k-mooney | which is why it fails today | |
| 10:48:42 | rm_work | sean-k-mooney: https://review.opendev.org/#/c/709280/ | |
| 10:48:56 | gibi | rm_work: look, I understand you dont care but I as a nova maintainer needs to care. Hence my previous -1 as I did not understood the feature | |
| 10:49:00 | rm_work | this is a very very simple way to get live-migrate and other service-to-service use-cases unblocked | |
| 10:49:26 | sean-k-mooney | we use ip_allocation=defer to allow the ip to be allcoated when we bind the port but it cannoth change again after that point | |
| 10:49:36 | rm_work | gibi: right, sorry, I didn't mean that I don't care if all features work -- i mean i didn't care personally so hadn't checked whether it was the same issue -- this would solve it anyway if it is, and not affect it if it isn't | |
| 10:49:58 | rm_work | i'm not going to propose a feature that breaks other stuff :) | |
| 10:50:20 | gibi | sean-k-mooney: I rely on you about defining what is the expected behavior of a port during cold migrate in the current case so if you say it should not change IP then it accept that | |
| 10:50:42 | rm_work | sure, and in that case, this would allow to solve for cold-migrate as well | |
| 10:50:49 | rm_work | in the same way | |
| 10:51:18 | gibi | rm_work: no worries. I need to act as a guardian not you. I just explained why I was -1 | |
| 10:51:25 | sean-k-mooney | rm_work: just looking at it now but how is the segment id passed to nova? a schduler hint or somehting like that | |
| 10:51:34 | rm_work | sean-k-mooney: scheduler hint, yes | |
| 10:52:12 | sean-k-mooney | --hint routed_segments=96a9316a-54cb-4043-9ccc-b9cacd0d4d52 | |
| 10:52:25 | sean-k-mooney | so is that uuid a placement aggreate uuid | |
| 10:52:37 | sean-k-mooney | so we convert that to a member_of ? | |
| 10:52:41 | gibi | right now I like the simplicity of the filter, but I does not like matching the _name_ of the aggregate as that feels hackish | |
| 10:52:49 | rm_work | sean-k-mooney: this is a neutron segment_id | |
| 10:53:10 | rm_work | gibi: yeah I am not a huge fan of name-comparison as a matcher either, but per the spec it DOES work | |
| 10:53:19 | rm_work | so, this stuff is also in Placement as well | |
| 10:53:27 | rm_work | and in Placement it has a little tighter mapping | |
| 10:53:37 | sean-k-mooney | right so filters are not allowed to call other servcies rest apis so im wondering how nova knows if a host is connected to the segment | |
| 10:53:39 | gibi | rm_work: yeah I got it that this aggregate naming thing was how it is speced and implemented | |
| 10:53:46 | rm_work | sean-k-mooney: nova aggregates | |
| 10:53:55 | gibi | sean-k-mooney: ^^ | |
| 10:54:03 | sean-k-mooney | if we have nova host aggreates we dont need a new filter | |
| 10:54:13 | rm_work | sean-k-mooney: see for example http://paste.openstack.org/show/7YyK5FtL2OiXgErAjMbM/ | |
| 10:54:45 | rm_work | we don't know the correct nova aggregate | |
| 10:54:56 | rm_work | other services would know the segment_id | |
| 10:55:07 | rm_work | nove understands the host-aggregates internally and how they map to segments | |
| 10:55:43 | rm_work | sorry, when I say "we" in this case I mean other services | |