Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-02-04
18:22:24 cdent big hack, but fun
18:23:20 sean-k-mooney the bigger the hack the more fun it becomes when it starts to work
18:23:50 sean-k-mooney cdent: are you using etcd to store any state
18:24:21 cdent etcd is used for basically two things:
18:24:35 cdent message the creation or destruction of a vm (via a watcher) to a target host
18:24:45 cdent store the IP of the created vm (while it exists)
18:24:51 openstackgerrit Matt Riedemann proposed openstack/nova master: Extend RequestGroup object for mapping https://review.openstack.org/619527
18:25:17 sean-k-mooney its architected similar to OVN in that respect
18:25:53 sean-k-mooney e.g. etcd is beign used to store teh desireds state and then distibuted workers are watching for changes and locally trying to achive teh desired state
18:26:19 sean-k-mooney so rather then an imperitive workflow its declaritve
18:26:22 cdent creation or destruction is based on the allocations: if they are there, create (with those resources), if not, destroy, in either case the same operation happens to placement: PUT /allocations/{uuid}
18:26:31 cdent yup
18:26:45 cdent it _feels_ really nice
18:27:05 sean-k-mooney it _feels_ a bit like k8s
18:27:25 sean-k-mooney perhaps with less yaml
18:27:39 cdent yes, less yaml
18:28:04 cdent my desire for using declarative messaging with vm creation predates my awareness of k8s
18:28:09 cdent but k8s provided the mechanism
18:28:16 cdent (awareness of the mechanism)
18:28:42 sean-k-mooney cdent: yes the mechanism was not invented by k8s but its one of the more promenet examples
18:29:17 cdent I was using similar mechanism in a tool called mcfeely back in the 90s
18:29:32 cdent mcfeely was your special delivery man
18:36:21 artom efried, looks like sean-k-mooney answered all your things?
18:36:44 sean-k-mooney artom: we went on a bit of a tangent
18:37:17 sean-k-mooney artom: but i think i answered his first question e.g. your stuff does not depend on placement and uses the ResocueTracker
18:37:33 sean-k-mooney *i.e. not e.g.
18:38:00 artom No, because let's be real, NUMA in placement is closer to a pipe dream than a real thing at this point (not dissing anyone, it's just a complicated thing to get working properly)
18:39:30 artom efried, and the resource tracker will always remain, because it covered the use case of tracking specific individual things (CPU 0, NUMA node 1), not just quantities of resources
18:39:38 sean-k-mooney artom: right and paraphasing efried main question. is nuam/cpu pinning in placments someting we can start working on/proposing specs for in train
18:39:53 artom Don't see why not, I mean the spec was up for Stein
18:40:07 artom It stalled, but there was no massive disagreement AFAICT
18:40:42 sean-k-mooney that was mainly because we had too many other things to do in stien
18:46:54 cdent hopefully when placement is more independent, it will be more clear that nova's inability to take advantage of placement is nova's problem, not placement's
18:47:05 cdent placement has been able to support a lot of this stuff for a _long_ time
18:47:22 cdent but working the changes through nova is complex
18:47:42 sean-k-mooney cdent: yes but it can support plociy that chage the resouces requried dependent on the host that is selected
18:47:43 krasmussen I made my first blueprint for a stopgap on subnet capacity aware scheduling via a filter that uses aggregates with keys containing subnets that are available to those hosts. Is this the correct place to ask for input on this?
18:47:49 sean-k-mooney *cant
18:48:08 sean-k-mooney and yes i know that by design :)
18:48:37 sean-k-mooney krasmussen: here or the nova meeting
18:48:46 krasmussen https://blueprints.launchpad.net/nova/+spec/ip-aware-scheduling-aggregate
18:49:09 sean-k-mooney krasmussen: the mailing list is also good however that is not ideal from a design point of view
18:49:31 krasmussen Figure if I'm going to take up time in a meeting I would prefer this at least have some eyes and I can improve the POC a bit more first.
18:49:45 sean-k-mooney krasmussen there is already a what of mapping host to reachable subnets
18:50:04 krasmussen Where is that accessible at?
18:51:34 sean-k-mooney krasmussen: are you aware of https://specs.openstack.org/openstack/nova-specs/specs/newton/implemented/neutron-routed-networks.html#neutron-routed-networks
18:52:35 efried artom: Save me looking for it, where's that stein spec for numa modeling in placement?
18:52:53 artom efried, https://review.openstack.org/#/c/552924/
18:53:04 efried artom: And yes, partly because too much to do, partly because there wasn't a super big motivator for it yet, but partly because we couldn't settle on how we were going to model subtree affinity.
18:53:13 sean-k-mooney krasmussen: currently neutron reporst network segments to placement and uses placemetn aggreates to assocaite them with hosts
18:53:27 efried There were several ideas, but none was great, and we stopped discussing before we approached consensus.
18:55:52 efried artom: Ah, vaguely remembering that we did settle on some kind of compromise in that spec, will reread more thoroughly...
18:59:45 krasmussen I'm reading through all that now and following the rabbithole a bit.
19:00:20 sean-k-mooney krasmussen: not to be deter you by the way but the blueprint freeze for stien was in january so this feature would need to be proposed for stien
19:00:27 sean-k-mooney *train
19:01:56 sean-k-mooney krasmussen: as a comunity we are trying to move away from using nova filters for filtering based on resocue available intead prefering ot model such constriants in placement
19:02:56 krasmussen Thats fine :) just trying to get in the upstream flow here opposed to just local patches.
19:03:26 sean-k-mooney krasmussen: nova also trys not to model network backend info internally as such extending the host_state with segment/ip reachablity would be counter to the direction we have been movign over the last few cycles
19:03:27 krasmussen I thought scheduler had to pick a node before getting sent to placement? Or do I have that backwards?
19:03:59 sean-k-mooney krasmussen: other way around
19:04:11 krasmussen So placement -> scheduler yeah?
19:04:29 sean-k-mooney placement -> schduler -> claim resouce in placement
19:05:00 sean-k-mooney so placement give the schduler a set of host that could work and then the shcudler chooes form that set
19:05:35 artom efried, it's been so long since I looked at it that I don't remember :/
19:05:36 krasmussen Ahhh gotcha I only had the "scheduler -> claim resource in placement" in my head. I'll look into what can be done on the placement side for this and will likely abandon the BP and associated POC and look at doing it in placement instead.
19:05:42 sean-k-mooney krasmussen: is your primary usecase modeling what hosts have available public ips?
19:05:55 artom But yeah, that and jaypipes's CPU resources spec should be proposed for Train
19:06:26 krasmussen We were seeing scheduler selecting hosts that only had access to certain subnets which were out of ips.
19:06:58 sean-k-mooney krasmussen: well i think the best first step would be to write up what your usecase is and then send an email to the mailing list
19:07:49 krasmussen Which is why I went down the scheduler route, we are kinda new to anything "placement" though as we moved from Liberty -> Pike for some of our newer clouds.
19:08:18 sean-k-mooney krasmussen: how is you ip addressign done in the cloud? are you using routed network or ovs or something else?
19:09:21 krasmussen routed network, we used ovs before in the old environment, so again another recent change for us.
19:10:06 sean-k-mooney krasmussen: right so in pike i belive the routed network work should already work.
19:10:19 sean-k-mooney krasmussen: but i also think you need to set ip_allocation=defer
19:10:34 sean-k-mooney krasmussen: the neuton folks would be able to confim
19:11:57 krasmussen This gives me some things to look into further, thanks a TON!
19:17:53 sean-k-mooney krasmussen: before you close your blueprit i woudl follow up on the mailing list. i will check the code when i gave time but its posibly that nova is not using the ip secments reported to placemetn by neutron in its placement query
19:47:12 openstackgerrit Merged openstack/nova master: Rip the report client out of SchedulerClient https://review.openstack.org/617042
19:47:20 openstackgerrit Merged openstack/nova master: Rip out the SchedulerClient https://review.openstack.org/617049
19:53:39 openstackgerrit Merged openstack/nova master: Use a static resource tracker in compute manager https://review.openstack.org/620711
20:02:59 openstackgerrit Merged openstack/nova master: FUPs: ReportClient traffic series https://review.openstack.org/633609
20:13:09 openstackgerrit Eric Fried proposed openstack/nova master: Use :oslo-config: role in hypervisor-kvm doc https://review.openstack.org/634798
20:17:21 openstackgerrit Artom Lifshitz proposed openstack/nova master: [WIP] Create claims when live migrating https://review.openstack.org/634606
20:17:21 openstackgerrit Artom Lifshitz proposed openstack/nova master: [WIP] Add migration param to check_can_live_migrate_destination https://review.openstack.org/634605
21:31:26 openstackgerrit Vlad Gusev proposed openstack/nova stable/rocky: Use X-Forwarded-Proto as origin protocol if present https://review.openstack.org/634814
22:07:16 artom sean-k-mooney, still around? Wanted to talk about https://review.openstack.org/#/c/599587/5/specs/stein/approved/numa-aware-live-migration.rst@247 - but maybe tomorrow via bjns?
22:09:22 artom (22:00, that's late even for you)
22:22:32 openstackgerrit Merged openstack/nova master: Per-instance serial number https://review.openstack.org/619953
22:22:40 openstackgerrit Merged openstack/nova master: doc: Switch header styles in support doc https://review.openstack.org/626930
22:22:48 openstackgerrit Merged openstack/nova master: doc: Add solution to live migration ssh issues https://review.openstack.org/626931
22:28:38 mriedem forever in bluejeans
22:32:02 artom mriedem, at least the abbreviation is "bjns" now
22:32:36 artom It used to be "Are you ready for a bj?" "Can we do a bj tomorrow?" and so on and so forth
22:34:19 mriedem so i've heard
22:46:55 openstackgerrit Artom Lifshitz proposed openstack/nova master: [WIP] Create claims when live migrating https://review.openstack.org/634606
22:46:56 openstackgerrit Artom Lifshitz proposed openstack/nova master: [Very WIP] Send dest NUMA info inside LiveMigrateData https://review.openstack.org/634828
22:46:56 openstackgerrit Artom Lifshitz proposed openstack/nova master: [Very WIP] New objects to transmit NUMA config from dest to source https://review.openstack.org/634827
22:48:21 artom Wow. So WIP. Very workflow. Much -1.
22:48:27 artom And on that, I'm done.
22:48:29 artom o/

Earlier   Later