Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-03-23
14:53:38 leakypipes bauwser: not following you... how is that different from today?
14:54:06 bauwser leakypipes: today you can end up having non-NUMA workloads that are running on a NUMA compute
14:54:32 bauwser actually
14:54:34 bauwser sec
14:54:44 bauwser doing huge pages and CPU pinning would be insane
14:54:54 bauwser if some workloads would just pick any CPU
14:55:03 leakypipes bauwser: yes, and there's nothing in the new system that would change that. the operator simply wouldn't set up the compute host to create NUMA child providers on that host (if this is really an issue that anyone is concerned about)(
14:55:47 bauwser leakypipes: okay, I see your point, how the operator would drive whether they want a non-NUMA inventory or not ?
14:55:53 bauwser thru a config option ?
14:56:09 leakypipes bauwser: that's up to your spec, no? :)
14:56:15 bauwser fair enough :)
14:56:44 bauwser okay, I think that's a reasonable trade-off
14:58:36 purplerbot <figleaf> bauwser: "only in root" - that was one of the decisions we made back in Atlanta. A machine would be configured by the operator ahead of time. If a VCPU was assigned to a NUMA node, it could not be considered also to be part of the root compute node [2018-03-23 14:06:19.844755] [n 3hcx]
14:58:36 figleaf bauwser: that's what I meant here [t 3hcx]
15:00:01 bauwser figleaf: yeah, what changed now is that I think that most operators already shard their cloud in between hosts that supports NUMA workloads and others (thru aggregates I guess)
15:00:19 bauwser because doing CPU pinning requires that your host is pretty dedicated for
15:00:48 bauwser I'm not saying you were wrong :)
15:01:13 figleaf bauwser: of course not! :)
15:02:45 mlavalle rybridges: hi, when you have a chance, plese look here: https://github.com/openstack/neutron/blob/stable/ocata/neutron/services/segments/plugin.py#L232. As you can see, after adding the hosts to the aggregate, it adds the IPv$ inventory to the RP. Was the inventory created?
15:09:02 sean-k-mooney[m] leakypipes: so just finished the v3 of the spec. i like this much more but some nits inline
15:09:14 leakypipes sean-k-mooney[m]: cool, ty sir
15:09:29 leakypipes sean-k-mooney[m], cfriesen: wanted to make sure I got all the definitions crystal clear...
15:09:45 leakypipes sean-k-mooney[m], cfriesen: did I properly describe vCPU threads, emulator threads and I/O threads?
15:11:41 ktibi Hi, I try to add a compute on existing platform and compute return errors : Another thread already created a resource provider with the UUID XXXXX. I think it's because I remove this compute previously and added it with same hostname.
15:11:52 rybridges mlavalle: The inventory is not being created. All the resource provider tied to the segment does get created, but the inventories, usages, and aggregates associated with the resource provider are all empty
15:12:17 sean-k-mooney[m] leakypipes: just read them again and yes i think so. we dont support configuring io threads in nova but the definition is correct
15:12:19 ktibi where do you think I need to purge or clean compute ?
15:13:00 ktibi full log: http://paste.openstack.org/show/710134/
15:13:41 rybridges mlavalle: In fact, I dont even think the code gets to the line and executes it at all because the aggregate never even gets associated. I added a very simple log statement before and after this line: https://github.com/openstack/neutron/blob/stable/ocata/neutron/services/segments/plugin.py#L225 and I see the one before being printed every time, but the one after never gets printed impying that after
15:13:43 rybridges we call associate_aggregates() the control never returns
15:14:06 rybridges I also put a log statement on this line and I never see it get printed: https://github.com/openstack/neutron/blob/stable/ocata/neutron/services/segments/placement_client.py#L163
15:14:08 rybridges very strange
15:14:51 sean-k-mooney[m] bauwser: do you have a minute to discuss the nic feature based scheduling spec
15:15:07 bauwser sean-k-mooney[m]: sorry I saw your pings earlier
15:15:15 bauwser sean-k-mooney[m]: I'm currently in a team meeting but shoot
15:15:31 leakypipes ktibi: can you re-start the nova-compoute with the DEBUG level logging and pastebin the log please?
15:15:52 ktibi leakypipes, y of course
15:16:00 leakypipes ktibi: it does indeed look like you may need to issue a couple manual cleanup calls to the placement API. but tough to tell without the debug logs
15:16:10 sean-k-mooney[m] bauwser: https://review.openstack.org/#/c/545951/, i dont know how to proceed with this. i am only ment to be spending 10% of my time upstream this cycle.
15:16:37 sean-k-mooney[m] bauwser: if i was to start this spec from scratch i would do it differently today but i wont have time to rewrite it this cycle
15:17:19 bauwser sean-k-mooney[m]: opening the spec
15:17:27 bauwser I need to reload context in my mind
15:18:04 bauwser aaaaaah I remember
15:18:28 ktibi leakypipes, http://paste.openstack.org/show/710154/
15:18:38 bauwser sean-k-mooney[m]: well, I don't remember the specific outcome we had during the PTG
15:19:18 leakypipes ktibi: cool. thanks. btw, is this queens?
15:19:18 bauwser sean-k-mooney[m]: could you please refresh my mind ?
15:19:37 bauwser sean-k-mooney[m]: IIRC, we discussed on the opportunity to move things to conductor, right?
15:19:41 ktibi leakypipes, pike with kolla
15:19:43 bauwser not nobody objected
15:19:50 leakypipes ktibi: if you could pastebin the placement-api logs, too, that would be great
15:20:25 sean-k-mooney[m] bauwser: ya so rofolfo work on this for 2 cycles. i was just ment to be rebasing it an resubmitting his work this cycle. if its beyond that however i have to escalate to my managment to see if i can work on it an rewrite it to track the nic features as tratis in placement.
15:20:40 bauwser I see
15:20:52 bauwser okay, I don't honestly want to throw you under the bus
15:21:06 bauwser my only concern is that it creates more tech debt
15:21:06 sean-k-mooney[m] bauwser: today there is no reason we could not use nested RPs + traits to track the nic feartres for sriov device.
15:21:20 bauwser but I see leakypipes +2ing that spec
15:21:26 sean-k-mooney[m] we have been storing that in the nova db since pike. the remainder of the spec is consuming it in the schduler.
15:21:30 bauwser so I guess it's more or less a trade-off
15:21:34 sean-k-mooney[m] bauwser: ya that is fair
15:22:02 bauwser well, okay, I have to disappear in a short few because of kids, but once I'm back, I'll reevaluate the spec
15:22:06 sean-k-mooney[m] well i can ask can i try to port this once it merges to placement
15:22:53 ktibi leakypipes, I checked placement log, i can see only info log with GET on resource_providers/705b7a45-ce71-4455-8349-6cee4c051c0e/inventories
15:23:12 leakypipes ktibi: question for you... it looks like there are no instances in the cell DB for that compute node. were there originally instances on it?
15:23:22 sean-k-mooney[m] ok cool im technically off today but let me know what you decide and ill be back monday
15:24:01 ktibi leakypipes, no when I remove this node, I disable service, evacuate all instances, remove service and shutdown the compute
15:24:32 leakypipes ktibi: k. I'm wondering maybe the placement service for some reason didn't get updated with the removal of those instances...
15:24:50 sean-k-mooney[m] bauwser: if i was able to also port to doing this in placement in rocky that would be the best outcome but im ment to work on someting else internally from april that 10%.
15:24:53 leakypipes ktibi: can you please execute the following SQL query against the nova api database?
15:25:55 leakypipes ktibi: SELECT COUNT(*) FROM allocations AS a JOIN inventories AS i ON a.resource_provider_id = i.resource_provider_id JOIN resource_providers AS rp ON i.resource_provider_id = rp.id WHERE rp.uuid = '91f9e750-351f-463e-aa00-c8843c37560e';
15:26:10 ktibi leakypipes, count : 0
15:26:14 sean-k-mooney[m] o/ ok ill stop working on my day off now... maybe...
15:26:15 leakypipes ktibi: ok, good.
15:26:47 leakypipes ktibi: what about this one? SELECT COUNT(*) FROM inventories AS i ON a.resource_provider_id = i.resource_provider_id JOIN resource_providers AS rp ON i.resource_provider_id = rp.id WHERE rp.uuid = '91f9e750-351f-463e-aa00-c8843c37560e';
15:26:56 leakypipes ktibi: oops, sorry
15:27:12 leakypipes ktibi: what about this one? SELECT COUNT(*) FROM inventories AS i JOIN resource_providers AS rp ON i.resource_provider_id = rp.id WHERE rp.uuid = '91f9e750-351f-463e-aa00-c8843c37560e';
15:27:47 ktibi leakypipes, maybe that can help you http://paste.openstack.org/show/710164/
15:28:13 ktibi leakypipes, count : 0 too
15:28:22 leakypipes k
15:28:47 leakypipes ktibi: got it...
15:28:56 cdent figleaf, fried_rice, leakypipes : just to confirm: I assume we would like the forbidden traits support to show up in the same microversion for /resource_providers and /allocation_candidates, yes?
15:28:59 leakypipes ktibi: your services table has two records for compute06
15:29:11 leakypipes ktibi: delete the one where deleted_at is not NULL.
15:29:15 ktibi yes I see :) I waiting your comfirmation to delete it ^^
15:29:19 fried_rice cdent: Sure, unless there's some reason to split 'em.
15:29:38 cdent fried_rice: only if we want to make the code more split into stages
15:29:52 fried_rice no, not necessary IMO.
15:30:33 fried_rice cdent: In fact, I find myself slightly irked that we can't include multiple features in a single microversion - for reasons of our development process, not for any technical reason.
15:30:50 fried_rice cdent: But in this case, it's the same feature, so we can totally get away with it.
15:31:00 cdent yes and yes
15:31:04 leakypipes ktibi: and then set disabled=0 on the other services table record with copmute06 host name.
15:32:41 leakypipes ktibi: finally, can you show me what the compute_nodes table looks like please? just do this: SELECT id, uuid, hypervisor_hostname FROM compute_nodes;
15:33:10 ktibi leakypipes, ho I can see two compute06
15:34:46 ktibi leakypipes, http://paste.openstack.org/show/710169/
15:35:09 mlavalle rybridges: in that log statement right above L225, could you log aggregate.uuid? The reason I am asking this is https://review.openstack.org/#/c/415031/, which fixes https://bugs.launchpad.net/nova/+bug/1652642
15:35:10 openstack Launchpad bug 1652642 in OpenStack Compute (nova) "Nova API doesn't return aggregate's uuid, which is needed when using the placement API" [High,Fix released] - Assigned to Matt Riedemann (mriedem)
15:35:55 leakypipes ktibi: DELETE FROM compute_nodes WHERE id = 15;
15:36:07 leakypipes ktibi: that *should* fix things up...
15:36:22 ktibi leakypipes, ok I restart nova compute on 06
15:36:44 openstackgerrit melanie witt proposed openstack/nova master: Move _make_instance_list call outside of DB transaction context https://review.openstack.org/555093

Earlier   Later