| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-10-25 | |||
| 16:19:57 | sean-k-mooney | mainly for ovs-dpdk | |
| 16:20:18 | sean-k-mooney | all ovs-dpdk deployment have numa toplogyes because of hugepages | |
| 16:20:22 | cfriesen | opposite of sean-k-mooney :) | |
| 16:20:27 | stephenfin | cfriesen: Yeah, that's what I was thinking | |
| 16:20:58 | stephenfin | sean-k-mooney: Relatively speaking though, I imagine users with NUMA topologies >>> users with OVS-DPDK | |
| 16:21:08 | sean-k-mooney | i know yahoo japan use livemigration in there production ovs-dpdk based deployment | |
| 16:21:28 | cfriesen | sean-k-mooney: wouldn't anyone using ovs-dpdk also use pinned CPUs? | |
| 16:21:36 | sean-k-mooney | cfriesen: no | |
| 16:21:44 | cfriesen | freaky | |
| 16:21:50 | sean-k-mooney | they may but that should not be assumed | |
| 16:22:22 | sean-k-mooney | cfriesen: if you entire cloud uses ovs-dpdk you still have instance that you want to over subscive so no pinning | |
| 16:22:50 | stephenfin | sean-k-mooney: I imagine it's still fairly common though | |
| 16:23:25 | sean-k-mooney | there are 2 main usecases for ovsdpdk hosting providres that want to run allot of instnace on a singel host where the aggreate throuput exceed what kernel ovs can do | |
| 16:23:46 | sean-k-mooney | or a small number of vnf that each neeed a lot of bandwith | |
| 16:23:58 | sean-k-mooney | the frist group dont use pinning the second do | |
| 16:24:21 | stephenfin | cfriesen: So, to be clear, workaround option is good enough, as opposed to allowing it for migrations with a requested_destination? | |
| 16:24:25 | stephenfin | For you, at least | |
| 16:25:29 | stephenfin | sean-k-mooney: The way I'm thinking is that this has a potentially devastating impact on workloads and their performance, so if operators are knowingly doing it, they will switch the toggle | |
| 16:25:53 | stephenfin | but in the meanwhile, we'll catch all the other folks who didn't know this wasn't working the way they thought | |
| 16:25:53 | sean-k-mooney | stephenfin: in the pinning case | |
| 16:26:18 | sean-k-mooney | in the hugepages case the livemigation will fail if it can allocat the right hugepages | |
| 16:26:27 | sean-k-mooney | and there is no performacne impact | |
| 16:26:46 | sean-k-mooney | modulo your numa aware vswithc stuff | |
| 16:26:47 | stephenfin | can or can't? | |
| 16:27:34 | sean-k-mooney | the migration will fail if it can not alloate hugepages | |
| 16:27:47 | cfriesen | stephenfin: yes, workaround option would be good enough for me. Let's keep it simple for now. | |
| 16:27:58 | sean-k-mooney | the dest node will emit a qemu error and we will abort the migration | |
| 16:28:09 | cfriesen | I'd rather see live migrations get fixed than get fancy with the workaround. | |
| 16:28:25 | sean-k-mooney | cfriesen: same | |
| 16:28:32 | stephenfin | cfriesen: Agreed. With any luck, this won't even survive the release | |
| 16:28:39 | stephenfin | But it'll be a useful backport, IMO | |
| 16:28:54 | stephenfin | sean-k-mooney: There's also the non-homogeneous hosts case to consider too | |
| 16:28:59 | sean-k-mooney | stephenfin: if you backport it it needs to be off by defualt | |
| 16:29:09 | sean-k-mooney | stephenfin: yes i know | |
| 16:29:10 | stephenfin | sean-k-mooney: Yup, in that case, agreed | |
| 16:29:57 | stephenfin | cfriesen: Cool. I'll wait to see if artom or bauzas want to pipe up. If not, I'll rework accordingly | |
| 16:30:09 | sean-k-mooney | basically want yahoo were doing is if the migration aported (that will happen in the non-homogeneous case too) then they did a manual migration specifying the host | |
| 16:30:33 | sean-k-mooney | e.g. if it did not jsut work the found a node that could fit it and forced the migration by setting it | |
| 16:31:31 | stephenfin | It wouldn't fail if the guest ended up split over multiple host NUMA nodes though | |
| 16:31:41 | sean-k-mooney | stephenfin: you can proably simply this a lot but doing the check at the api level and if the vm has a numa toplogy just reject the migration based on the config value | |
| 16:32:27 | stephenfin | I know that brings us back into performance issue territory but still, just shows how this is all sorts of broken rn | |
| 16:32:29 | sean-k-mooney | stephenfin: you are thinkg of pinning agin yes. and ture | |
| 16:33:17 | stephenfin | No, pinning isn't needed. We restrict an instance NUMA node to a range of host cores corresponding to host NUMA node | |
| 16:33:19 | cfriesen | sean-k-mooney: do you have any thoughts on whether we should support TIS vs CRB in the emulated TPM spec? and if so, should that be a separate thing from the TPM version? | |
| 16:33:24 | sean-k-mooney | stephenfin: yes it is but we know that you are technically in undefined behavior land if you migrate with a numa toplogy | |
| 16:33:52 | sean-k-mooney | cfriesen: i know thate they are a thing but not enough to comment on that | |
| 16:34:18 | stephenfin | Also, mriedem noted about doing it higher but it seemed no simpler. This doesn't affect anything but libvirt so that still seems like the place to do this. | |
| 16:34:21 | sean-k-mooney | cfriesen: prably use two traits one for the version and a second for the type | |
| 16:34:27 | cfriesen | sean-k-mooney: basically, with TPM 2.0 there are two different ways of talking to the TPM. | |
| 16:34:33 | cfriesen | and qemu supports both | |
| 16:34:44 | sean-k-mooney | ok | |
| 16:34:48 | cfriesen | so I was thinking it's not really a trait | |
| 16:34:56 | cfriesen | since it's not schedulable, at least for qemu | |
| 16:35:02 | sean-k-mooney | well then it s a flavor extra spec | |
| 16:35:09 | sean-k-mooney | and or image meta | |
| 16:36:00 | sean-k-mooney | so retuired trait for tpm_2.0 and hw:tpm_connection_mode=CRB | |
| 16:36:04 | cfriesen | if they specify CRB with TPM 1.2, should I ignore the CRB request or error out? | |
| 16:36:27 | sean-k-mooney | cfriesen: proabley error but you could leave that up to qemu | |
| 16:36:58 | sean-k-mooney | libvirt/qemu should be validteing there input and should reject it if invalide | |
| 16:37:35 | sean-k-mooney | you could optimes by checking in the driver but not sure that needs to be in the spec | |
| 16:37:49 | cfriesen | makes sense | |
| 16:39:20 | sean-k-mooney | stephenfin: if you have a config or provide a way to bypass the check im ok with it | |
| 16:40:17 | sean-k-mooney | stephenfin: if i cant turn it off you will break people and im not really ok with saying vpp ovs-dpdk, vrouter and snabb swtich deployments can nolonger live migrate | |
| 16:58:49 | kmalloc | cdent: out of curiosity, does Nova really support passing config data via paste-ini or is it mostly assumed you're passing through config via .conf files [even to middleware]? | |
| 16:59:11 | kmalloc | cdent: looking at how deep i need to support config passing into the new loader when allowing folks to drop paste. | |
| 16:59:28 | kmalloc | if you know. | |
| 16:59:43 | cdent | it's not so much the passing of config data (although that may be in there) but the idea that paste.ini can be edited to add other middlewares | |
| 17:00:06 | kmalloc | right. that is fine. | |
| 17:00:09 | kmalloc | i'm planning on supporting that | |
| 17:00:23 | cdent | and so it falls under the domain of stuff that we need to carry on dealing with incase people have their own changes | |
| 17:00:38 | kmalloc | the other question i have is: if i only allow middleware to be injected into a specific place in the pipeline, where would that be? | |
| 17:00:46 | kmalloc | or do we need it to really be very dynamic | |
| 17:00:47 | kmalloc | ? | |
| 17:00:56 | cdent | sean dague had more to say on the topic way back in the day, and probably new more about what was happening in the wild | |
| 17:01:23 | kmalloc | right. i can flip the coin either way, obviously i wont allow injecting things *after* nova's application | |
| 17:01:38 | cdent | I'm unable to answer that question with about "where" with regard to nova | |
| 17:01:56 | cdent | some of the nova-api oriented folks might be able to (gmann, alex_xu ) | |
| 17:02:04 | kmalloc | thnx :) | |
| 17:02:31 | kmalloc | i figure nova is the ideal target to aim for here, since much follows in it's lead | |
| 17:03:20 | cdent | yup | |
| 17:04:05 | kmalloc | keystone was the outlier but we drew the line and said "nope, no unwhitelisted middleware in our app space" but mostly for security/maintenance/behavior reasons | |
| 17:04:39 | kmalloc | it's super easy to add outer middleware to anything, but i don't want to block people who legitimately need things somewhere in the middle | |
| 17:04:47 | kmalloc | say post keystonemiddleware. | |
| 17:07:32 | melwitt | they're APAC, so you might consider the dev ML with [nova][api] | |
| 17:15:00 | kmalloc | melwitt: ah thnx. yeah Pacific has poor overlap with <insert most currently relevant timezones for OpenStack here> | |
| 17:15:18 | kmalloc | melwitt: yeah i'll get an email spun up as well. | |
| 17:16:01 | melwitt | sounds cool | |
| 17:32:04 | openstackgerrit | Chris Dent proposed openstack/nova master: WIP: Integration tests using gabbi-tempest https://review.openstack.org/613386 | |
| 17:46:28 | cfriesen | is there a current recommendation on whether to enable the "send_service_user_token" config option? It still defaults to "False", just wondering what the downsides would be...is there an additional keystone round-trip? | |
| 17:51:13 | artom | stephenfin, sorry, missed your ping, I think you settled on config option? I'm cool with that | |
| 17:51:35 | melwitt | cfriesen: found this in the spec, maybe that is the main downside https://specs.openstack.org/openstack/nova-specs/specs/ocata/implemented/use-service-tokens.html#performance-impact | |
| 18:26:02 | openstackgerrit | Merged openstack/nova master: api-ref: Add descriptions of error cases https://review.openstack.org/610429 | |
| 18:32:58 | Sundar | melwitt: Do specs need to merged this week (for Stein milestone 1) to be considered for implementation in Stein? | |
| 18:33:23 | melwitt | Sundar: no. our spec freeze is milestone 2, Jan 10 | |
| 18:35:01 | Sundar | melwitt: Thanks. Is there any way to accelerate the coverage for the spec https://review.openstack.org/#/c/603955/ ? | |
| 18:35:26 | Sundar | I understand there are lots of other stuff in the Nova plate | |
| 18:37:13 | melwitt | Sundar: looks like there was some review last week, and you've made some updates | |
| 18:39:13 | melwitt | I can mention it in the nova meeting today that it's been updated and ready for review again | |
| 18:39:36 | Sundar | Great. Thanks! | |
| 18:47:58 | melwitt | sean-k-mooney: could you please reply on this os-vif release review about whether we need a major version bump? https://review.openstack.org/612844 | |