| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-03-23 | |||
| 14:45:54 | leakypipes | figleaf: I can add a note to clarify that. | |
| 14:45:57 | kashyap | A new Ubuntu version? | |
| 14:46:01 | mriedem | solar is next openstack | |
| 14:46:12 | mriedem | (pending that name is approved by legal) | |
| 14:46:15 | kashyap | Aah, I didn't check the release naming for the next one | |
| 14:46:32 | leakypipes | I guess my suggestion of "Suck It, Trebek" didn't fly. | |
| 14:46:57 | figleaf | leakypipes: that's what I thought. When an a-c is chosen and allocated against, is there anything that constrains the building of the VM to the particular NUMA node that was allocated against? | |
| 14:47:32 | bauwser | figleaf: leakypipes: reviewed | |
| 14:47:33 | openstackgerrit | Merged openstack/nova-specs master: libvirt: add support for virtio-net rx/tx queue sizes https://review.openstack.org/539605 | |
| 14:47:33 | leakypipes | figleaf: nothing besides the provider UUID, no. | |
| 14:48:14 | bauwser | figleaf: leakypipes: my main concern is that existing flavors that don't ask for NUMA specifics shouldn't be constrained on NUMA nodes | |
| 14:48:25 | figleaf | leakypipes: and there is no relation between the RP UUID in placement and a particular NUMA node in hardware, right? | |
| 14:48:45 | leakypipes | bauwser: they aren't "constrained". they just end up on a NUMA node is the host system is NUMA.. | |
| 14:48:52 | leakypipes | s/is/if/ | |
| 14:48:59 | bauwser | leakypipes: back up a second | |
| 14:49:12 | bauwser | leakypipes: I'm taking the example I had previously | |
| 14:49:20 | bauwser | say I ask for two VCPUs, period. | |
| 14:49:48 | bauwser | what if my compute only has only vCPU left on one NUMA node, but one other vCPU left on another NUMA node ? | |
| 14:49:59 | bauwser | at the moment, nova will satisfy that | |
| 14:50:18 | leakypipes | bauwser: if the host system is NUMA, then the VCPU inventories will be against child provider representing NUMA nodes. And the allocation candidates will contain allocation requests that consume against one of those NUMA node resource providers. | |
| 14:50:25 | bauwser | because the strategy is "spread" between all NUMA nodes, except if you clearly ask for another strategy | |
| 14:50:37 | leakypipes | bauwser: in that case, that compute node would not be returned. | |
| 14:50:46 | leakypipes | bauwser: since no single provider has capacity for 2 VCPU | |
| 14:50:51 | bauwser | leakypipes: that's the whole point I discussed previously | |
| 14:51:02 | bauwser | leakypipes: that's a behavioural change | |
| 14:51:36 | bauwser | from a placement perspective, what was successful before is no longer possible | |
| 14:52:19 | leakypipes | bauwser: then don't have the host configured to create NUMA node resource providers. | |
| 14:52:30 | leakypipes | bauwser: and just use a single provider for the comute node. | |
| 14:53:05 | bauwser | leakypipes: so only specific NUMA workloads would run on a NUMA-specified compute, and non-NUMA workloads would end up on the rest ? | |
| 14:53:24 | bauwser | if that's a config option, the default value would be "flat topology" | |
| 14:53:38 | leakypipes | bauwser: not following you... how is that different from today? | |
| 14:54:06 | bauwser | leakypipes: today you can end up having non-NUMA workloads that are running on a NUMA compute | |
| 14:54:32 | bauwser | actually | |
| 14:54:34 | bauwser | sec | |
| 14:54:44 | bauwser | doing huge pages and CPU pinning would be insane | |
| 14:54:54 | bauwser | if some workloads would just pick any CPU | |
| 14:55:03 | leakypipes | bauwser: yes, and there's nothing in the new system that would change that. the operator simply wouldn't set up the compute host to create NUMA child providers on that host (if this is really an issue that anyone is concerned about)( | |
| 14:55:47 | bauwser | leakypipes: okay, I see your point, how the operator would drive whether they want a non-NUMA inventory or not ? | |
| 14:55:53 | bauwser | thru a config option ? | |
| 14:56:09 | leakypipes | bauwser: that's up to your spec, no? :) | |
| 14:56:15 | bauwser | fair enough :) | |
| 14:56:44 | bauwser | okay, I think that's a reasonable trade-off | |
| 14:58:36 | figleaf | bauwser: that's what I meant here [t 3hcx] | |
| 14:58:36 | purplerbot | <figleaf> bauwser: "only in root" - that was one of the decisions we made back in Atlanta. A machine would be configured by the operator ahead of time. If a VCPU was assigned to a NUMA node, it could not be considered also to be part of the root compute node [2018-03-23 14:06:19.844755] [n 3hcx] | |
| 15:00:01 | bauwser | figleaf: yeah, what changed now is that I think that most operators already shard their cloud in between hosts that supports NUMA workloads and others (thru aggregates I guess) | |
| 15:00:19 | bauwser | because doing CPU pinning requires that your host is pretty dedicated for | |
| 15:00:48 | bauwser | I'm not saying you were wrong :) | |
| 15:01:13 | figleaf | bauwser: of course not! :) | |
| 15:02:45 | mlavalle | rybridges: hi, when you have a chance, plese look here: https://github.com/openstack/neutron/blob/stable/ocata/neutron/services/segments/plugin.py#L232. As you can see, after adding the hosts to the aggregate, it adds the IPv$ inventory to the RP. Was the inventory created? | |
| 15:09:02 | sean-k-mooney[m] | leakypipes: so just finished the v3 of the spec. i like this much more but some nits inline | |
| 15:09:14 | leakypipes | sean-k-mooney[m]: cool, ty sir | |
| 15:09:29 | leakypipes | sean-k-mooney[m], cfriesen: wanted to make sure I got all the definitions crystal clear... | |
| 15:09:45 | leakypipes | sean-k-mooney[m], cfriesen: did I properly describe vCPU threads, emulator threads and I/O threads? | |
| 15:11:41 | ktibi | Hi, I try to add a compute on existing platform and compute return errors : Another thread already created a resource provider with the UUID XXXXX. I think it's because I remove this compute previously and added it with same hostname. | |
| 15:11:52 | rybridges | mlavalle: The inventory is not being created. All the resource provider tied to the segment does get created, but the inventories, usages, and aggregates associated with the resource provider are all empty | |
| 15:12:17 | sean-k-mooney[m] | leakypipes: just read them again and yes i think so. we dont support configuring io threads in nova but the definition is correct | |
| 15:12:19 | ktibi | where do you think I need to purge or clean compute ? | |
| 15:13:00 | ktibi | full log: http://paste.openstack.org/show/710134/ | |
| 15:13:41 | rybridges | mlavalle: In fact, I dont even think the code gets to the line and executes it at all because the aggregate never even gets associated. I added a very simple log statement before and after this line: https://github.com/openstack/neutron/blob/stable/ocata/neutron/services/segments/plugin.py#L225 and I see the one before being printed every time, but the one after never gets printed impying that after | |
| 15:13:43 | rybridges | we call associate_aggregates() the control never returns | |
| 15:14:06 | rybridges | I also put a log statement on this line and I never see it get printed: https://github.com/openstack/neutron/blob/stable/ocata/neutron/services/segments/placement_client.py#L163 | |
| 15:14:08 | rybridges | very strange | |
| 15:14:51 | sean-k-mooney[m] | bauwser: do you have a minute to discuss the nic feature based scheduling spec | |
| 15:15:07 | bauwser | sean-k-mooney[m]: sorry I saw your pings earlier | |
| 15:15:15 | bauwser | sean-k-mooney[m]: I'm currently in a team meeting but shoot | |
| 15:15:31 | leakypipes | ktibi: can you re-start the nova-compoute with the DEBUG level logging and pastebin the log please? | |
| 15:15:52 | ktibi | leakypipes, y of course | |
| 15:16:00 | leakypipes | ktibi: it does indeed look like you may need to issue a couple manual cleanup calls to the placement API. but tough to tell without the debug logs | |
| 15:16:10 | sean-k-mooney[m] | bauwser: https://review.openstack.org/#/c/545951/, i dont know how to proceed with this. i am only ment to be spending 10% of my time upstream this cycle. | |
| 15:16:37 | sean-k-mooney[m] | bauwser: if i was to start this spec from scratch i would do it differently today but i wont have time to rewrite it this cycle | |
| 15:17:19 | bauwser | sean-k-mooney[m]: opening the spec | |
| 15:17:27 | bauwser | I need to reload context in my mind | |
| 15:18:04 | bauwser | aaaaaah I remember | |
| 15:18:28 | ktibi | leakypipes, http://paste.openstack.org/show/710154/ | |
| 15:18:38 | bauwser | sean-k-mooney[m]: well, I don't remember the specific outcome we had during the PTG | |
| 15:19:18 | bauwser | sean-k-mooney[m]: could you please refresh my mind ? | |
| 15:19:18 | leakypipes | ktibi: cool. thanks. btw, is this queens? | |
| 15:19:37 | bauwser | sean-k-mooney[m]: IIRC, we discussed on the opportunity to move things to conductor, right? | |
| 15:19:41 | ktibi | leakypipes, pike with kolla | |
| 15:19:43 | bauwser | not nobody objected | |
| 15:19:50 | leakypipes | ktibi: if you could pastebin the placement-api logs, too, that would be great | |
| 15:20:25 | sean-k-mooney[m] | bauwser: ya so rofolfo work on this for 2 cycles. i was just ment to be rebasing it an resubmitting his work this cycle. if its beyond that however i have to escalate to my managment to see if i can work on it an rewrite it to track the nic features as tratis in placement. | |
| 15:20:40 | bauwser | I see | |
| 15:20:52 | bauwser | okay, I don't honestly want to throw you under the bus | |
| 15:21:06 | sean-k-mooney[m] | bauwser: today there is no reason we could not use nested RPs + traits to track the nic feartres for sriov device. | |
| 15:21:06 | bauwser | my only concern is that it creates more tech debt | |
| 15:21:20 | bauwser | but I see leakypipes +2ing that spec | |
| 15:21:26 | sean-k-mooney[m] | we have been storing that in the nova db since pike. the remainder of the spec is consuming it in the schduler. | |
| 15:21:30 | bauwser | so I guess it's more or less a trade-off | |
| 15:21:34 | sean-k-mooney[m] | bauwser: ya that is fair | |
| 15:22:02 | bauwser | well, okay, I have to disappear in a short few because of kids, but once I'm back, I'll reevaluate the spec | |
| 15:22:06 | sean-k-mooney[m] | well i can ask can i try to port this once it merges to placement | |
| 15:22:53 | ktibi | leakypipes, I checked placement log, i can see only info log with GET on resource_providers/705b7a45-ce71-4455-8349-6cee4c051c0e/inventories | |
| 15:23:12 | leakypipes | ktibi: question for you... it looks like there are no instances in the cell DB for that compute node. were there originally instances on it? | |
| 15:23:22 | sean-k-mooney[m] | ok cool im technically off today but let me know what you decide and ill be back monday | |
| 15:24:01 | ktibi | leakypipes, no when I remove this node, I disable service, evacuate all instances, remove service and shutdown the compute | |
| 15:24:32 | leakypipes | ktibi: k. I'm wondering maybe the placement service for some reason didn't get updated with the removal of those instances... | |
| 15:24:50 | sean-k-mooney[m] | bauwser: if i was able to also port to doing this in placement in rocky that would be the best outcome but im ment to work on someting else internally from april that 10%. | |
| 15:24:53 | leakypipes | ktibi: can you please execute the following SQL query against the nova api database? | |
| 15:25:55 | leakypipes | ktibi: SELECT COUNT(*) FROM allocations AS a JOIN inventories AS i ON a.resource_provider_id = i.resource_provider_id JOIN resource_providers AS rp ON i.resource_provider_id = rp.id WHERE rp.uuid = '91f9e750-351f-463e-aa00-c8843c37560e'; | |