Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-03-23
14:46:57 figleaf leakypipes: that's what I thought. When an a-c is chosen and allocated against, is there anything that constrains the building of the VM to the particular NUMA node that was allocated against?
14:47:32 bauwser figleaf: leakypipes: reviewed
14:47:33 openstackgerrit Merged openstack/nova-specs master: libvirt: add support for virtio-net rx/tx queue sizes https://review.openstack.org/539605
14:47:33 leakypipes figleaf: nothing besides the provider UUID, no.
14:48:14 bauwser figleaf: leakypipes: my main concern is that existing flavors that don't ask for NUMA specifics shouldn't be constrained on NUMA nodes
14:48:25 figleaf leakypipes: and there is no relation between the RP UUID in placement and a particular NUMA node in hardware, right?
14:48:45 leakypipes bauwser: they aren't "constrained". they just end up on a NUMA node is the host system is NUMA..
14:48:52 leakypipes s/is/if/
14:48:59 bauwser leakypipes: back up a second
14:49:12 bauwser leakypipes: I'm taking the example I had previously
14:49:20 bauwser say I ask for two VCPUs, period.
14:49:48 bauwser what if my compute only has only vCPU left on one NUMA node, but one other vCPU left on another NUMA node ?
14:49:59 bauwser at the moment, nova will satisfy that
14:50:18 leakypipes bauwser: if the host system is NUMA, then the VCPU inventories will be against child provider representing NUMA nodes. And the allocation candidates will contain allocation requests that consume against one of those NUMA node resource providers.
14:50:25 bauwser because the strategy is "spread" between all NUMA nodes, except if you clearly ask for another strategy
14:50:37 leakypipes bauwser: in that case, that compute node would not be returned.
14:50:46 leakypipes bauwser: since no single provider has capacity for 2 VCPU
14:50:51 bauwser leakypipes: that's the whole point I discussed previously
14:51:02 bauwser leakypipes: that's a behavioural change
14:51:36 bauwser from a placement perspective, what was successful before is no longer possible
14:52:19 leakypipes bauwser: then don't have the host configured to create NUMA node resource providers.
14:52:30 leakypipes bauwser: and just use a single provider for the comute node.
14:53:05 bauwser leakypipes: so only specific NUMA workloads would run on a NUMA-specified compute, and non-NUMA workloads would end up on the rest ?
14:53:24 bauwser if that's a config option, the default value would be "flat topology"
14:53:38 leakypipes bauwser: not following you... how is that different from today?
14:54:06 bauwser leakypipes: today you can end up having non-NUMA workloads that are running on a NUMA compute
14:54:32 bauwser actually
14:54:34 bauwser sec
14:54:44 bauwser doing huge pages and CPU pinning would be insane
14:54:54 bauwser if some workloads would just pick any CPU
14:55:03 leakypipes bauwser: yes, and there's nothing in the new system that would change that. the operator simply wouldn't set up the compute host to create NUMA child providers on that host (if this is really an issue that anyone is concerned about)(
14:55:47 bauwser leakypipes: okay, I see your point, how the operator would drive whether they want a non-NUMA inventory or not ?
14:55:53 bauwser thru a config option ?
14:56:09 leakypipes bauwser: that's up to your spec, no? :)
14:56:15 bauwser fair enough :)
14:56:44 bauwser okay, I think that's a reasonable trade-off
14:58:36 figleaf bauwser: that's what I meant here [t 3hcx]
14:58:36 purplerbot <figleaf> bauwser: "only in root" - that was one of the decisions we made back in Atlanta. A machine would be configured by the operator ahead of time. If a VCPU was assigned to a NUMA node, it could not be considered also to be part of the root compute node [2018-03-23 14:06:19.844755] [n 3hcx]
15:00:01 bauwser figleaf: yeah, what changed now is that I think that most operators already shard their cloud in between hosts that supports NUMA workloads and others (thru aggregates I guess)
15:00:19 bauwser because doing CPU pinning requires that your host is pretty dedicated for
15:00:48 bauwser I'm not saying you were wrong :)
15:01:13 figleaf bauwser: of course not! :)
15:02:45 mlavalle rybridges: hi, when you have a chance, plese look here: https://github.com/openstack/neutron/blob/stable/ocata/neutron/services/segments/plugin.py#L232. As you can see, after adding the hosts to the aggregate, it adds the IPv$ inventory to the RP. Was the inventory created?
15:09:02 sean-k-mooney[m] leakypipes: so just finished the v3 of the spec. i like this much more but some nits inline
15:09:14 leakypipes sean-k-mooney[m]: cool, ty sir
15:09:29 leakypipes sean-k-mooney[m], cfriesen: wanted to make sure I got all the definitions crystal clear...
15:09:45 leakypipes sean-k-mooney[m], cfriesen: did I properly describe vCPU threads, emulator threads and I/O threads?
15:11:41 ktibi Hi, I try to add a compute on existing platform and compute return errors : Another thread already created a resource provider with the UUID XXXXX. I think it's because I remove this compute previously and added it with same hostname.
15:11:52 rybridges mlavalle: The inventory is not being created. All the resource provider tied to the segment does get created, but the inventories, usages, and aggregates associated with the resource provider are all empty
15:12:17 sean-k-mooney[m] leakypipes: just read them again and yes i think so. we dont support configuring io threads in nova but the definition is correct
15:12:19 ktibi where do you think I need to purge or clean compute ?
15:13:00 ktibi full log: http://paste.openstack.org/show/710134/
15:13:41 rybridges mlavalle: In fact, I dont even think the code gets to the line and executes it at all because the aggregate never even gets associated. I added a very simple log statement before and after this line: https://github.com/openstack/neutron/blob/stable/ocata/neutron/services/segments/plugin.py#L225 and I see the one before being printed every time, but the one after never gets printed impying that after
15:13:43 rybridges we call associate_aggregates() the control never returns
15:14:06 rybridges I also put a log statement on this line and I never see it get printed: https://github.com/openstack/neutron/blob/stable/ocata/neutron/services/segments/placement_client.py#L163
15:14:08 rybridges very strange
15:14:51 sean-k-mooney[m] bauwser: do you have a minute to discuss the nic feature based scheduling spec
15:15:07 bauwser sean-k-mooney[m]: sorry I saw your pings earlier
15:15:15 bauwser sean-k-mooney[m]: I'm currently in a team meeting but shoot
15:15:31 leakypipes ktibi: can you re-start the nova-compoute with the DEBUG level logging and pastebin the log please?
15:15:52 ktibi leakypipes, y of course
15:16:00 leakypipes ktibi: it does indeed look like you may need to issue a couple manual cleanup calls to the placement API. but tough to tell without the debug logs
15:16:10 sean-k-mooney[m] bauwser: https://review.openstack.org/#/c/545951/, i dont know how to proceed with this. i am only ment to be spending 10% of my time upstream this cycle.
15:16:37 sean-k-mooney[m] bauwser: if i was to start this spec from scratch i would do it differently today but i wont have time to rewrite it this cycle
15:17:19 bauwser sean-k-mooney[m]: opening the spec
15:17:27 bauwser I need to reload context in my mind
15:18:04 bauwser aaaaaah I remember
15:18:28 ktibi leakypipes, http://paste.openstack.org/show/710154/
15:18:38 bauwser sean-k-mooney[m]: well, I don't remember the specific outcome we had during the PTG
15:19:18 bauwser sean-k-mooney[m]: could you please refresh my mind ?
15:19:18 leakypipes ktibi: cool. thanks. btw, is this queens?
15:19:37 bauwser sean-k-mooney[m]: IIRC, we discussed on the opportunity to move things to conductor, right?
15:19:41 ktibi leakypipes, pike with kolla
15:19:43 bauwser not nobody objected
15:19:50 leakypipes ktibi: if you could pastebin the placement-api logs, too, that would be great
15:20:25 sean-k-mooney[m] bauwser: ya so rofolfo work on this for 2 cycles. i was just ment to be rebasing it an resubmitting his work this cycle. if its beyond that however i have to escalate to my managment to see if i can work on it an rewrite it to track the nic features as tratis in placement.
15:20:40 bauwser I see
15:20:52 bauwser okay, I don't honestly want to throw you under the bus
15:21:06 sean-k-mooney[m] bauwser: today there is no reason we could not use nested RPs + traits to track the nic feartres for sriov device.
15:21:06 bauwser my only concern is that it creates more tech debt
15:21:20 bauwser but I see leakypipes +2ing that spec
15:21:26 sean-k-mooney[m] we have been storing that in the nova db since pike. the remainder of the spec is consuming it in the schduler.
15:21:30 bauwser so I guess it's more or less a trade-off
15:21:34 sean-k-mooney[m] bauwser: ya that is fair
15:22:02 bauwser well, okay, I have to disappear in a short few because of kids, but once I'm back, I'll reevaluate the spec
15:22:06 sean-k-mooney[m] well i can ask can i try to port this once it merges to placement
15:22:53 ktibi leakypipes, I checked placement log, i can see only info log with GET on resource_providers/705b7a45-ce71-4455-8349-6cee4c051c0e/inventories
15:23:12 leakypipes ktibi: question for you... it looks like there are no instances in the cell DB for that compute node. were there originally instances on it?
15:23:22 sean-k-mooney[m] ok cool im technically off today but let me know what you decide and ill be back monday
15:24:01 ktibi leakypipes, no when I remove this node, I disable service, evacuate all instances, remove service and shutdown the compute
15:24:32 leakypipes ktibi: k. I'm wondering maybe the placement service for some reason didn't get updated with the removal of those instances...
15:24:50 sean-k-mooney[m] bauwser: if i was able to also port to doing this in placement in rocky that would be the best outcome but im ment to work on someting else internally from april that 10%.
15:24:53 leakypipes ktibi: can you please execute the following SQL query against the nova api database?
15:25:55 leakypipes ktibi: SELECT COUNT(*) FROM allocations AS a JOIN inventories AS i ON a.resource_provider_id = i.resource_provider_id JOIN resource_providers AS rp ON i.resource_provider_id = rp.id WHERE rp.uuid = '91f9e750-351f-463e-aa00-c8843c37560e';
15:26:10 ktibi leakypipes, count : 0
15:26:14 sean-k-mooney[m] o/ ok ill stop working on my day off now... maybe...
15:26:15 leakypipes ktibi: ok, good.
15:26:47 leakypipes ktibi: what about this one? SELECT COUNT(*) FROM inventories AS i ON a.resource_provider_id = i.resource_provider_id JOIN resource_providers AS rp ON i.resource_provider_id = rp.id WHERE rp.uuid = '91f9e750-351f-463e-aa00-c8843c37560e';
15:26:56 leakypipes ktibi: oops, sorry
15:27:12 leakypipes ktibi: what about this one? SELECT COUNT(*) FROM inventories AS i JOIN resource_providers AS rp ON i.resource_provider_id = rp.id WHERE rp.uuid = '91f9e750-351f-463e-aa00-c8843c37560e';

Earlier   Later