Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-03-23
15:11:52 rybridges mlavalle: The inventory is not being created. All the resource provider tied to the segment does get created, but the inventories, usages, and aggregates associated with the resource provider are all empty
15:12:17 sean-k-mooney[m] leakypipes: just read them again and yes i think so. we dont support configuring io threads in nova but the definition is correct
15:12:19 ktibi where do you think I need to purge or clean compute ?
15:13:00 ktibi full log: http://paste.openstack.org/show/710134/
15:13:41 rybridges mlavalle: In fact, I dont even think the code gets to the line and executes it at all because the aggregate never even gets associated. I added a very simple log statement before and after this line: https://github.com/openstack/neutron/blob/stable/ocata/neutron/services/segments/plugin.py#L225 and I see the one before being printed every time, but the one after never gets printed impying that after
15:13:43 rybridges we call associate_aggregates() the control never returns
15:14:06 rybridges I also put a log statement on this line and I never see it get printed: https://github.com/openstack/neutron/blob/stable/ocata/neutron/services/segments/placement_client.py#L163
15:14:08 rybridges very strange
15:14:51 sean-k-mooney[m] bauwser: do you have a minute to discuss the nic feature based scheduling spec
15:15:07 bauwser sean-k-mooney[m]: sorry I saw your pings earlier
15:15:15 bauwser sean-k-mooney[m]: I'm currently in a team meeting but shoot
15:15:31 leakypipes ktibi: can you re-start the nova-compoute with the DEBUG level logging and pastebin the log please?
15:15:52 ktibi leakypipes, y of course
15:16:00 leakypipes ktibi: it does indeed look like you may need to issue a couple manual cleanup calls to the placement API. but tough to tell without the debug logs
15:16:10 sean-k-mooney[m] bauwser: https://review.openstack.org/#/c/545951/, i dont know how to proceed with this. i am only ment to be spending 10% of my time upstream this cycle.
15:16:37 sean-k-mooney[m] bauwser: if i was to start this spec from scratch i would do it differently today but i wont have time to rewrite it this cycle
15:17:19 bauwser sean-k-mooney[m]: opening the spec
15:17:27 bauwser I need to reload context in my mind
15:18:04 bauwser aaaaaah I remember
15:18:28 ktibi leakypipes, http://paste.openstack.org/show/710154/
15:18:38 bauwser sean-k-mooney[m]: well, I don't remember the specific outcome we had during the PTG
15:19:18 bauwser sean-k-mooney[m]: could you please refresh my mind ?
15:19:18 leakypipes ktibi: cool. thanks. btw, is this queens?
15:19:37 bauwser sean-k-mooney[m]: IIRC, we discussed on the opportunity to move things to conductor, right?
15:19:41 ktibi leakypipes, pike with kolla
15:19:43 bauwser not nobody objected
15:19:50 leakypipes ktibi: if you could pastebin the placement-api logs, too, that would be great
15:20:25 sean-k-mooney[m] bauwser: ya so rofolfo work on this for 2 cycles. i was just ment to be rebasing it an resubmitting his work this cycle. if its beyond that however i have to escalate to my managment to see if i can work on it an rewrite it to track the nic features as tratis in placement.
15:20:40 bauwser I see
15:20:52 bauwser okay, I don't honestly want to throw you under the bus
15:21:06 sean-k-mooney[m] bauwser: today there is no reason we could not use nested RPs + traits to track the nic feartres for sriov device.
15:21:06 bauwser my only concern is that it creates more tech debt
15:21:20 bauwser but I see leakypipes +2ing that spec
15:21:26 sean-k-mooney[m] we have been storing that in the nova db since pike. the remainder of the spec is consuming it in the schduler.
15:21:30 bauwser so I guess it's more or less a trade-off
15:21:34 sean-k-mooney[m] bauwser: ya that is fair
15:22:02 bauwser well, okay, I have to disappear in a short few because of kids, but once I'm back, I'll reevaluate the spec
15:22:06 sean-k-mooney[m] well i can ask can i try to port this once it merges to placement
15:22:53 ktibi leakypipes, I checked placement log, i can see only info log with GET on resource_providers/705b7a45-ce71-4455-8349-6cee4c051c0e/inventories
15:23:12 leakypipes ktibi: question for you... it looks like there are no instances in the cell DB for that compute node. were there originally instances on it?
15:23:22 sean-k-mooney[m] ok cool im technically off today but let me know what you decide and ill be back monday
15:24:01 ktibi leakypipes, no when I remove this node, I disable service, evacuate all instances, remove service and shutdown the compute
15:24:32 leakypipes ktibi: k. I'm wondering maybe the placement service for some reason didn't get updated with the removal of those instances...
15:24:50 sean-k-mooney[m] bauwser: if i was able to also port to doing this in placement in rocky that would be the best outcome but im ment to work on someting else internally from april that 10%.
15:24:53 leakypipes ktibi: can you please execute the following SQL query against the nova api database?
15:25:55 leakypipes ktibi: SELECT COUNT(*) FROM allocations AS a JOIN inventories AS i ON a.resource_provider_id = i.resource_provider_id JOIN resource_providers AS rp ON i.resource_provider_id = rp.id WHERE rp.uuid = '91f9e750-351f-463e-aa00-c8843c37560e';
15:26:10 ktibi leakypipes, count : 0
15:26:14 sean-k-mooney[m] o/ ok ill stop working on my day off now... maybe...
15:26:15 leakypipes ktibi: ok, good.
15:26:47 leakypipes ktibi: what about this one? SELECT COUNT(*) FROM inventories AS i ON a.resource_provider_id = i.resource_provider_id JOIN resource_providers AS rp ON i.resource_provider_id = rp.id WHERE rp.uuid = '91f9e750-351f-463e-aa00-c8843c37560e';
15:26:56 leakypipes ktibi: oops, sorry
15:27:12 leakypipes ktibi: what about this one? SELECT COUNT(*) FROM inventories AS i JOIN resource_providers AS rp ON i.resource_provider_id = rp.id WHERE rp.uuid = '91f9e750-351f-463e-aa00-c8843c37560e';
15:27:47 ktibi leakypipes, maybe that can help you http://paste.openstack.org/show/710164/
15:28:13 ktibi leakypipes, count : 0 too
15:28:22 leakypipes k
15:28:47 leakypipes ktibi: got it...
15:28:56 cdent figleaf, fried_rice, leakypipes : just to confirm: I assume we would like the forbidden traits support to show up in the same microversion for /resource_providers and /allocation_candidates, yes?
15:28:59 leakypipes ktibi: your services table has two records for compute06
15:29:11 leakypipes ktibi: delete the one where deleted_at is not NULL.
15:29:15 ktibi yes I see :) I waiting your comfirmation to delete it ^^
15:29:19 fried_rice cdent: Sure, unless there's some reason to split 'em.
15:29:38 cdent fried_rice: only if we want to make the code more split into stages
15:29:52 fried_rice no, not necessary IMO.
15:30:33 fried_rice cdent: In fact, I find myself slightly irked that we can't include multiple features in a single microversion - for reasons of our development process, not for any technical reason.
15:30:50 fried_rice cdent: But in this case, it's the same feature, so we can totally get away with it.
15:31:00 cdent yes and yes
15:31:04 leakypipes ktibi: and then set disabled=0 on the other services table record with copmute06 host name.
15:32:41 leakypipes ktibi: finally, can you show me what the compute_nodes table looks like please? just do this: SELECT id, uuid, hypervisor_hostname FROM compute_nodes;
15:33:10 ktibi leakypipes, ho I can see two compute06
15:34:46 ktibi leakypipes, http://paste.openstack.org/show/710169/
15:35:09 mlavalle rybridges: in that log statement right above L225, could you log aggregate.uuid? The reason I am asking this is https://review.openstack.org/#/c/415031/, which fixes https://bugs.launchpad.net/nova/+bug/1652642
15:35:10 openstack Launchpad bug 1652642 in OpenStack Compute (nova) "Nova API doesn't return aggregate's uuid, which is needed when using the placement API" [High,Fix released] - Assigned to Matt Riedemann (mriedem)
15:35:55 leakypipes ktibi: DELETE FROM compute_nodes WHERE id = 15;
15:36:07 leakypipes ktibi: that *should* fix things up...
15:36:22 ktibi leakypipes, ok I restart nova compute on 06
15:36:44 openstackgerrit melanie witt proposed openstack/nova master: Move _make_instance_list call outside of DB transaction context https://review.openstack.org/555093
15:37:44 mlavalle rybridges: this is the Nova API version you require for aggregates to have a uuid: https://review.openstack.org/#/c/415031/9/nova/api/openstack/api_version_request.py@100
15:37:51 ktibi leakypipes, same error on compute06 : Another thread already created a resource provider with the UUID 91f9e750-351f-463e-aa00-c8843c37560e. Grabbing that record from the placement API.
15:37:59 ktibi maybe need to restart placement ?
15:38:29 leakypipes ktibi: well, that's not an error. :) the error is the "Cannot refresh inventory" thing.
15:38:36 openstackgerrit Matt Riedemann proposed openstack/nova master: Create volume attachment during boot from volume in compute https://review.openstack.org/541420
15:38:38 leakypipes ktibi: no, no need to restart placement.
15:38:57 leakypipes ktibi: what we want to see is inventory records being created in placement.
15:39:04 ktibi leakypipes, ok I'll test to migrate instance to this host
15:42:45 figleaf cdent: yes, one microversion for that makes sense
15:43:07 ktibi leakypipes, ok fail :/
15:43:44 ktibi leakypipes, exception thrown: No valid host was found. Unable to move instance f8e70257-244e-4fa7-b545-0a888b742b84 to host compute06. There is not enough capacity on the host for the instance.
15:44:04 giblet mriedem, leakypipes, mlavalle: is it OK for you If I publish a new version of the bandwidth spec soon, or you are just in the middle of reviewing the current PS?
15:44:22 mlavalle giblet: I'm good
15:45:07 mriedem giblet: i haven't gone back over the comments since i last reviewed it last week
15:45:29 mriedem i could do that quick i guess...
15:46:06 ktibi leakypipes, http://paste.openstack.org/show/710182/
15:46:17 giblet mriedem: I still need about an hour to do the update locally, if that is enough for you. Or you can go back after I published the new rev.
15:46:25 kashyap mriedem: So just checked w/ the folks on -infra, the Gate already does have the "Bionic" (upcoming Ubuntu LTS) images. Do we have any non-voting Nova jobs for it?
15:46:32 kashyap (Near as see, none.)
15:46:36 mriedem kashyap: no
15:46:51 mriedem kashyap: i'm just trying to get us to using the queens UCA right now
15:47:05 mriedem https://review.openstack.org/#/c/554317/
15:47:20 giblet mriedem: there will some new problems described in the update
15:47:39 giblet mriedem: so you might want to read the new rev anyhow

Earlier   Later