Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-09-20
16:09:43 bauzas mriedem: cool
16:09:52 bauzas and I call it a day
16:10:18 mriedem stephenfin: man i wish there wasn't a big refactor in that patch
16:10:32 sean-k-mooney donnyd: thanks the first thing i hope to use it for is ovs-dpdk testing with https://review.opendev.org/#/c/656580/
16:10:42 stephenfin I literally just said I shouldn't have done that to sean-k-mooney /o\
16:10:55 stephenfin (who's working up in the RH office in Dublin with me for the day)
16:11:00 sean-k-mooney yep you did
16:11:01 stephenfin mriedem: Gimme five to separate it out
16:11:41 mriedem i know it's a compulsion
16:11:50 mriedem obligatory "P
16:11:51 mriedem :P
16:15:36 sean-k-mooney stephenfin: by the way once i get teh ovs-dpdk job working on networking-ovs-dpdk i assume you have no issue with me adding it as non voting to check in os-vif and then promote it to voting around m1
16:15:57 stephenfin not on the slightest
16:16:22 sean-k-mooney cool i think the vhost user path is the only one without tempest coverate in os-vif currently
16:16:45 sean-k-mooney i also need to port one of the jobs to zuul v3 which i hope to do next week
16:23:17 mriedem mnaser: does the instance mapping have a cell mapping?
16:23:23 mriedem does the instance have a stale build request?
16:23:54 mriedem in that weird GET /servers/{server_id} 404 case before i've handled the InstanceNotFound in the API controller code, logged the traceback and then re-raised
16:24:02 mriedem so you can see where we're coming from, e.g. build request or what
16:25:29 mriedem mnaser: log the traceback here https://github.com/openstack/nova/blob/stable/stein/nova/api/openstack/common.py#L471
16:26:43 mriedem i'm pretty sure you wrote a script to deal with the case that the instance is in a cell, the build request is gone, but the intsance mapping doesn't have a cell mapping
16:27:05 mriedem yeah which i used here https://review.opendev.org/#/c/655908/
16:27:48 openstackgerrit Stephen Finucane proposed openstack/nova master: docs: Clarify everything CPU pinning https://review.opendev.org/683437
16:27:48 openstackgerrit Stephen Finucane proposed openstack/nova master: docs: Update CPU topologies guide to reflect the new PCPU world https://review.opendev.org/683485
16:28:16 openstackgerrit OpenStack Release Bot proposed openstack/os-vif stable/train: Update .gitreview for stable/train https://review.opendev.org/683488
16:28:21 openstackgerrit OpenStack Release Bot proposed openstack/os-vif stable/train: Update TOX/UPPER_CONSTRAINTS_FILE for stable/train https://review.opendev.org/683489
16:28:25 openstackgerrit OpenStack Release Bot proposed openstack/os-vif master: Update master for stable/train https://review.opendev.org/683490
16:28:30 stephenfin mriedem: Split ^
16:29:22 stephenfin wait, that doesn't look right
16:29:26 stephenfin :(
16:31:21 mriedem i left some other comments/questions about upgrade and quota details, not sure if you saw those while splitting
16:32:11 stephenfin ack
16:34:50 sean-k-mooney lyarwood: can you take a look at the propasl bot patches for os-vif above on stable ? they look sane to me
16:35:32 sean-k-mooney lyarwood: also it not supper urgent so it can wait till next week but they are trivial
16:38:59 mriedem dansmith: finally got around to replying to our comments on this heal instance mappings command https://review.opendev.org/#/c/655908/ - since there was no vote i hadn't noticed
16:39:10 openstackgerrit Stephen Finucane proposed openstack/nova master: docs: Clarify everything CPU pinning https://review.opendev.org/683437
16:39:10 openstackgerrit Stephen Finucane proposed openstack/nova master: docs: Update CPU topologies guide to reflect the new PCPU world https://review.opendev.org/683485
16:45:18 dansmith mriedem: okay I'll have to re-read to build my context back up
16:51:40 mnaser mriedem: the instance does have a mapping, which is where this is different than every time
16:51:55 mnaser instance_mapping in nova_api exists, and instance in nova exists..
17:10:46 mnaser waaaaaaaaaaait
17:10:53 mnaser the cell mapping points at cell0
17:11:04 mnaser but the instance exists in error state.. in the actual cell (cell1 in this case)
17:12:18 mnaser and the cell0 one _is_ marked as deleted
17:12:36 mnaser given my cell knowledge so far, you cant start from cell1 and then get buried in cell0 right?
17:13:03 mnaser you can't end up in cell0 if you're in cell1?
17:13:55 dansmith correct
17:14:03 dansmith I mean.. that's the intent/rule
17:17:00 openstackgerrit Artom Lifshitz proposed openstack/nova master: Move pre-3.44 Cinder post live migration test to test_compute_mgr https://review.opendev.org/683597
17:22:17 mriedem artom: "The previous patch (Id0e8b1c32600d53382e5ac938e403258c80221a0) created" you're learning!
17:22:38 artom mriedem, next step: multicellular organisms
17:24:39 mriedem mnaser: that's really strange, there are only two places we bury in cell0,
17:24:42 mriedem 1. if scheduling fails https://github.com/openstack/nova/blob/stable/stein/nova/conductor/manager.py#L1359
17:24:58 mriedem 2. or scheduling picks a host that doesn't have a mapping https://github.com/openstack/nova/blob/stable/stein/nova/conductor/manager.py#L1386
17:25:17 mriedem i wonder if you had hosts incorrectly mapped to cell0?
17:25:21 mriedem https://github.com/openstack/nova/blob/stable/stein/nova/conductor/manager.py#L1397
17:25:42 mriedem there shouldn't ever be host mappings pointing at cell0 but we don't explicitly fail if you try to do that
17:26:04 mriedem e.g. if you started a compute service pointing at the wrong cell database and it was registered in cell0 and then got mapped there in the host mapping
17:27:57 mriedem dansmith: do you think it'd be worthwhile to have a sanity check around https://github.com/openstack/nova/blob/stable/stein/nova/conductor/manager.py#L1421 such that if the cell is actually cell0 in that case we blow up with some kind of "idk what you're doing but this is definitely wrong"
17:30:50 mriedem that still wouldn't explain how the instance in mnaser's case went from cell0 to cell1, unless he has a busted heal script
17:32:34 mriedem maybe a host mapping pointing at cell0 isn't possible after all https://github.com/openstack/nova/blob/stable/stein/nova/objects/host_mapping.py#L253
17:33:18 mnaser yeah so in this case, i have a db record for this instance both in cell0 and cell1
17:33:25 mnaser and i dont think the config changed anytime recently..
17:33:50 mriedem is the host value set for either instance record?
17:34:16 mnaser both null
17:34:27 mnaser but the cell0 one is deleted and cell1 is not (aka deleted=0)
17:34:38 mriedem which has the older created_at time?
17:34:47 mnaser oh good idea
17:35:11 mnaser the same exact time.
17:35:21 mriedem wtf
17:35:41 mriedem you don't have some janky homebrew scripts that try to heal and revive an instance in cell0 and put it into cell1?
17:36:22 mnaser not that i know of
17:36:25 mnaser and also in case im losing it
17:36:26 mnaser http://paste.openstack.org/show/778146/
17:37:34 mriedem some sort of weird mariadb cluster issue or something?
17:37:48 mriedem you wouldn't be clustering those together though unless you really screwed up
17:38:01 mnaser its just plain old galera, and those are two different databases too..
17:38:11 mnaser so i dont know how it would like "oh let me make a cell0 thing"
17:38:24 mriedem it's clearly a phantom, some kind of 9/11 phantom, and it showed up too early for halloween
17:38:44 mriedem no obvious weirdness for that instance uuid in the logs?
17:39:06 mnaser stuff are mostly rotated out but im looking at all the fields and see the diffs
17:39:42 mnaser oh hm
17:40:06 mnaser so different fields are
17:40:44 openstackgerrit Balazs Gibizer proposed openstack/nova master: Remove functional test specific nova code https://review.opendev.org/683609
17:40:57 mnaser updated_at (obviously cause deleted was changed), deleted_at (for the one that deleted), id (different ids from cell0 and cell1), availablity zone and launched_on (it looks like it was scheduled?!)
17:41:07 dansmith mriedem: hmm, that's probably not a bad idea
17:41:08 mnaser let me check if there's anything intersting on the host it was provisioned on..
17:41:52 mriedem it's very weird that launched_on would be set but host is not
17:42:03 openstackgerrit OpenStack Release Bot proposed openstack/python-novaclient stable/train: Update .gitreview for stable/train https://review.opendev.org/683625
17:42:06 dansmith mriedem: I agree that this sounds highly fishy
17:42:06 openstackgerrit OpenStack Release Bot proposed openstack/python-novaclient stable/train: Update TOX/UPPER_CONSTRAINTS_FILE for stable/train https://review.opendev.org/683626
17:42:08 openstackgerrit OpenStack Release Bot proposed openstack/python-novaclient master: Update master for stable/train https://review.opendev.org/683627
17:42:14 mriedem dansmith: not even sounds, smells!
17:42:35 mnaser ok, timed out waiting for network-vif-plugged
17:42:39 dansmith mriedem: reeks of fishy sounds
17:42:50 mriedem mnaser: this is what sets launched_on https://github.com/openstack/nova/blob/f4aaa9e229c98a97af085f31e43509189e2e4585/nova/compute/resource_tracker.py#L544
17:42:53 mriedem oh i bet i know why,
17:42:58 mnaser BuildAbortException: Build of instance 10d8a93a-bb6b-44ef-83f1-be6b21336651 aborted: Failed to allocate the network(s), not rescheduling.
17:43:05 mriedem we fail the build, set host/node to None but not launched_on or az
17:43:45 mriedem https://github.com/openstack/nova/blob/f4aaa9e229c98a97af085f31e43509189e2e4585/nova/compute/manager.py#L2069

Earlier   Later