| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-03-23 | |||
| 15:15:15 | bauwser | sean-k-mooney[m]: I'm currently in a team meeting but shoot | |
| 15:15:31 | leakypipes | ktibi: can you re-start the nova-compoute with the DEBUG level logging and pastebin the log please? | |
| 15:15:52 | ktibi | leakypipes, y of course | |
| 15:16:00 | leakypipes | ktibi: it does indeed look like you may need to issue a couple manual cleanup calls to the placement API. but tough to tell without the debug logs | |
| 15:16:10 | sean-k-mooney[m] | bauwser: https://review.openstack.org/#/c/545951/, i dont know how to proceed with this. i am only ment to be spending 10% of my time upstream this cycle. | |
| 15:16:37 | sean-k-mooney[m] | bauwser: if i was to start this spec from scratch i would do it differently today but i wont have time to rewrite it this cycle | |
| 15:17:19 | bauwser | sean-k-mooney[m]: opening the spec | |
| 15:17:27 | bauwser | I need to reload context in my mind | |
| 15:18:04 | bauwser | aaaaaah I remember | |
| 15:18:28 | ktibi | leakypipes, http://paste.openstack.org/show/710154/ | |
| 15:18:38 | bauwser | sean-k-mooney[m]: well, I don't remember the specific outcome we had during the PTG | |
| 15:19:18 | leakypipes | ktibi: cool. thanks. btw, is this queens? | |
| 15:19:18 | bauwser | sean-k-mooney[m]: could you please refresh my mind ? | |
| 15:19:37 | bauwser | sean-k-mooney[m]: IIRC, we discussed on the opportunity to move things to conductor, right? | |
| 15:19:41 | ktibi | leakypipes, pike with kolla | |
| 15:19:43 | bauwser | not nobody objected | |
| 15:19:50 | leakypipes | ktibi: if you could pastebin the placement-api logs, too, that would be great | |
| 15:20:25 | sean-k-mooney[m] | bauwser: ya so rofolfo work on this for 2 cycles. i was just ment to be rebasing it an resubmitting his work this cycle. if its beyond that however i have to escalate to my managment to see if i can work on it an rewrite it to track the nic features as tratis in placement. | |
| 15:20:40 | bauwser | I see | |
| 15:20:52 | bauwser | okay, I don't honestly want to throw you under the bus | |
| 15:21:06 | bauwser | my only concern is that it creates more tech debt | |
| 15:21:06 | sean-k-mooney[m] | bauwser: today there is no reason we could not use nested RPs + traits to track the nic feartres for sriov device. | |
| 15:21:20 | bauwser | but I see leakypipes +2ing that spec | |
| 15:21:26 | sean-k-mooney[m] | we have been storing that in the nova db since pike. the remainder of the spec is consuming it in the schduler. | |
| 15:21:30 | bauwser | so I guess it's more or less a trade-off | |
| 15:21:34 | sean-k-mooney[m] | bauwser: ya that is fair | |
| 15:22:02 | bauwser | well, okay, I have to disappear in a short few because of kids, but once I'm back, I'll reevaluate the spec | |
| 15:22:06 | sean-k-mooney[m] | well i can ask can i try to port this once it merges to placement | |
| 15:22:53 | ktibi | leakypipes, I checked placement log, i can see only info log with GET on resource_providers/705b7a45-ce71-4455-8349-6cee4c051c0e/inventories | |
| 15:23:12 | leakypipes | ktibi: question for you... it looks like there are no instances in the cell DB for that compute node. were there originally instances on it? | |
| 15:23:22 | sean-k-mooney[m] | ok cool im technically off today but let me know what you decide and ill be back monday | |
| 15:24:01 | ktibi | leakypipes, no when I remove this node, I disable service, evacuate all instances, remove service and shutdown the compute | |
| 15:24:32 | leakypipes | ktibi: k. I'm wondering maybe the placement service for some reason didn't get updated with the removal of those instances... | |
| 15:24:50 | sean-k-mooney[m] | bauwser: if i was able to also port to doing this in placement in rocky that would be the best outcome but im ment to work on someting else internally from april that 10%. | |
| 15:24:53 | leakypipes | ktibi: can you please execute the following SQL query against the nova api database? | |
| 15:25:55 | leakypipes | ktibi: SELECT COUNT(*) FROM allocations AS a JOIN inventories AS i ON a.resource_provider_id = i.resource_provider_id JOIN resource_providers AS rp ON i.resource_provider_id = rp.id WHERE rp.uuid = '91f9e750-351f-463e-aa00-c8843c37560e'; | |
| 15:26:10 | ktibi | leakypipes, count : 0 | |
| 15:26:14 | sean-k-mooney[m] | o/ ok ill stop working on my day off now... maybe... | |
| 15:26:15 | leakypipes | ktibi: ok, good. | |
| 15:26:47 | leakypipes | ktibi: what about this one? SELECT COUNT(*) FROM inventories AS i ON a.resource_provider_id = i.resource_provider_id JOIN resource_providers AS rp ON i.resource_provider_id = rp.id WHERE rp.uuid = '91f9e750-351f-463e-aa00-c8843c37560e'; | |
| 15:26:56 | leakypipes | ktibi: oops, sorry | |
| 15:27:12 | leakypipes | ktibi: what about this one? SELECT COUNT(*) FROM inventories AS i JOIN resource_providers AS rp ON i.resource_provider_id = rp.id WHERE rp.uuid = '91f9e750-351f-463e-aa00-c8843c37560e'; | |
| 15:27:47 | ktibi | leakypipes, maybe that can help you http://paste.openstack.org/show/710164/ | |
| 15:28:13 | ktibi | leakypipes, count : 0 too | |
| 15:28:22 | leakypipes | k | |
| 15:28:47 | leakypipes | ktibi: got it... | |
| 15:28:56 | cdent | figleaf, fried_rice, leakypipes : just to confirm: I assume we would like the forbidden traits support to show up in the same microversion for /resource_providers and /allocation_candidates, yes? | |
| 15:28:59 | leakypipes | ktibi: your services table has two records for compute06 | |
| 15:29:11 | leakypipes | ktibi: delete the one where deleted_at is not NULL. | |
| 15:29:15 | ktibi | yes I see :) I waiting your comfirmation to delete it ^^ | |
| 15:29:19 | fried_rice | cdent: Sure, unless there's some reason to split 'em. | |
| 15:29:38 | cdent | fried_rice: only if we want to make the code more split into stages | |
| 15:29:52 | fried_rice | no, not necessary IMO. | |
| 15:30:33 | fried_rice | cdent: In fact, I find myself slightly irked that we can't include multiple features in a single microversion - for reasons of our development process, not for any technical reason. | |
| 15:30:50 | fried_rice | cdent: But in this case, it's the same feature, so we can totally get away with it. | |
| 15:31:00 | cdent | yes and yes | |
| 15:31:04 | leakypipes | ktibi: and then set disabled=0 on the other services table record with copmute06 host name. | |
| 15:32:41 | leakypipes | ktibi: finally, can you show me what the compute_nodes table looks like please? just do this: SELECT id, uuid, hypervisor_hostname FROM compute_nodes; | |
| 15:33:10 | ktibi | leakypipes, ho I can see two compute06 | |
| 15:34:46 | ktibi | leakypipes, http://paste.openstack.org/show/710169/ | |
| 15:35:09 | mlavalle | rybridges: in that log statement right above L225, could you log aggregate.uuid? The reason I am asking this is https://review.openstack.org/#/c/415031/, which fixes https://bugs.launchpad.net/nova/+bug/1652642 | |
| 15:35:10 | openstack | Launchpad bug 1652642 in OpenStack Compute (nova) "Nova API doesn't return aggregate's uuid, which is needed when using the placement API" [High,Fix released] - Assigned to Matt Riedemann (mriedem) | |
| 15:35:55 | leakypipes | ktibi: DELETE FROM compute_nodes WHERE id = 15; | |
| 15:36:07 | leakypipes | ktibi: that *should* fix things up... | |
| 15:36:22 | ktibi | leakypipes, ok I restart nova compute on 06 | |
| 15:36:44 | openstackgerrit | melanie witt proposed openstack/nova master: Move _make_instance_list call outside of DB transaction context https://review.openstack.org/555093 | |
| 15:37:44 | mlavalle | rybridges: this is the Nova API version you require for aggregates to have a uuid: https://review.openstack.org/#/c/415031/9/nova/api/openstack/api_version_request.py@100 | |
| 15:37:51 | ktibi | leakypipes, same error on compute06 : Another thread already created a resource provider with the UUID 91f9e750-351f-463e-aa00-c8843c37560e. Grabbing that record from the placement API. | |
| 15:37:59 | ktibi | maybe need to restart placement ? | |
| 15:38:29 | leakypipes | ktibi: well, that's not an error. :) the error is the "Cannot refresh inventory" thing. | |
| 15:38:36 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Create volume attachment during boot from volume in compute https://review.openstack.org/541420 | |
| 15:38:38 | leakypipes | ktibi: no, no need to restart placement. | |
| 15:38:57 | leakypipes | ktibi: what we want to see is inventory records being created in placement. | |
| 15:39:04 | ktibi | leakypipes, ok I'll test to migrate instance to this host | |
| 15:42:45 | figleaf | cdent: yes, one microversion for that makes sense | |
| 15:43:07 | ktibi | leakypipes, ok fail :/ | |
| 15:43:44 | ktibi | leakypipes, exception thrown: No valid host was found. Unable to move instance f8e70257-244e-4fa7-b545-0a888b742b84 to host compute06. There is not enough capacity on the host for the instance. | |
| 15:44:04 | giblet | mriedem, leakypipes, mlavalle: is it OK for you If I publish a new version of the bandwidth spec soon, or you are just in the middle of reviewing the current PS? | |
| 15:44:22 | mlavalle | giblet: I'm good | |
| 15:45:07 | mriedem | giblet: i haven't gone back over the comments since i last reviewed it last week | |
| 15:45:29 | mriedem | i could do that quick i guess... | |
| 15:46:06 | ktibi | leakypipes, http://paste.openstack.org/show/710182/ | |
| 15:46:17 | giblet | mriedem: I still need about an hour to do the update locally, if that is enough for you. Or you can go back after I published the new rev. | |
| 15:46:25 | kashyap | mriedem: So just checked w/ the folks on -infra, the Gate already does have the "Bionic" (upcoming Ubuntu LTS) images. Do we have any non-voting Nova jobs for it? | |
| 15:46:32 | kashyap | (Near as see, none.) | |
| 15:46:36 | mriedem | kashyap: no | |
| 15:46:51 | mriedem | kashyap: i'm just trying to get us to using the queens UCA right now | |
| 15:47:05 | mriedem | https://review.openstack.org/#/c/554317/ | |
| 15:47:20 | giblet | mriedem: there will some new problems described in the update | |
| 15:47:39 | giblet | mriedem: so you might want to read the new rev anyhow | |
| 15:48:11 | mriedem | giblet: i've got about 20 minutes so i'll go over the replies quick | |
| 15:48:38 | giblet | mriedem: OK, cool, thanks | |
| 15:52:49 | mriedem | giblet: yeah i can already see issues with https://review.openstack.org/#/c/502306/17/specs/rocky/approved/bandwidth-resource-provider.rst@100 | |
| 15:52:52 | mriedem | so commenting | |
| 15:54:51 | mriedem | and gerrit just shit the bed | |
| 15:56:06 | giblet | gerrit: you are not helpful at all! | |
| 15:59:15 | giblet | mriedem: gerrit is back | |
| 15:59:50 | mriedem | huzzah | |
| 15:59:54 | bauwser | sean-k-mooney[m]: +Wd | |
| 16:00:05 | openstackgerrit | Eric Berglund proposed openstack/nova master: PowerVM Driver: Snapshot https://review.openstack.org/543023 | |