Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-09-20
15:55:37 sean-k-mooney well its true
15:56:17 sean-k-mooney but also we have been without any real ci coverage for like 18 months so :(
15:56:24 mriedem you've done more to try and get numa tempest integration testing on nova changes than anyone so i'm not faulting you
15:56:52 sean-k-mooney oh i know i just whish i had more time to spend on it
15:57:03 mriedem it's just that my eyes glaze over when i hear "we have a bug with x that's related to numa"
15:57:11 sean-k-mooney although for the next week or two im going to continue looking at testing and ci
15:58:34 sean-k-mooney mriedem: i just pushed this by the way https://review.opendev.org/#/c/683431/
15:59:35 sean-k-mooney that will allow us to test cpu pinning hugepages and dpdk more or less reliably
16:00:32 sean-k-mooney im going to wait until after RC* to follow up with getting multi numa flavor on vexxhost and limestone
16:00:40 openstackgerrit Sylvain Bauza proposed openstack/nova master: Add a prelude for the Train release https://review.opendev.org/683327
16:04:11 mriedem stephenfin: i'm going to rebase https://review.opendev.org/#/c/682267/4 on top of https://review.opendev.org/#/c/674895/44 since it sounds like we're holding back on merging the former so as to not interrupte the latter
16:04:48 openstackgerrit Matt Riedemann proposed openstack/nova master: libvirt: Get the CPU model, not 'arch' from get_capabilities() https://review.opendev.org/682267
16:05:06 mriedem there wasn't even a merge conflict so i'm not sure what gerrit was complaining about
16:06:13 mriedem bauzas: do you want to re-approve https://review.opendev.org/#/c/681750/ since it was rebased?
16:06:23 mriedem the rest of that stack is stuck behind it
16:08:30 openstackgerrit Stephen Finucane proposed openstack/nova master: docs: Update CPU topologies guide to reflect the new PCPU world https://review.opendev.org/683437
16:08:37 donnyd sean-k-mooney: LGTM
16:08:42 stephenfin mriedem: Makes sense. Also, there the start of the doc ^
16:08:46 mriedem yu[p
16:08:47 donnyd I gave it my whole +1
16:09:43 bauzas mriedem: cool
16:09:52 bauzas and I call it a day
16:10:18 mriedem stephenfin: man i wish there wasn't a big refactor in that patch
16:10:32 sean-k-mooney donnyd: thanks the first thing i hope to use it for is ovs-dpdk testing with https://review.opendev.org/#/c/656580/
16:10:42 stephenfin I literally just said I shouldn't have done that to sean-k-mooney /o\
16:10:55 stephenfin (who's working up in the RH office in Dublin with me for the day)
16:11:00 sean-k-mooney yep you did
16:11:01 stephenfin mriedem: Gimme five to separate it out
16:11:41 mriedem i know it's a compulsion
16:11:50 mriedem obligatory "P
16:11:51 mriedem :P
16:15:36 sean-k-mooney stephenfin: by the way once i get teh ovs-dpdk job working on networking-ovs-dpdk i assume you have no issue with me adding it as non voting to check in os-vif and then promote it to voting around m1
16:15:57 stephenfin not on the slightest
16:16:22 sean-k-mooney cool i think the vhost user path is the only one without tempest coverate in os-vif currently
16:16:45 sean-k-mooney i also need to port one of the jobs to zuul v3 which i hope to do next week
16:23:17 mriedem mnaser: does the instance mapping have a cell mapping?
16:23:23 mriedem does the instance have a stale build request?
16:23:54 mriedem in that weird GET /servers/{server_id} 404 case before i've handled the InstanceNotFound in the API controller code, logged the traceback and then re-raised
16:24:02 mriedem so you can see where we're coming from, e.g. build request or what
16:25:29 mriedem mnaser: log the traceback here https://github.com/openstack/nova/blob/stable/stein/nova/api/openstack/common.py#L471
16:26:43 mriedem i'm pretty sure you wrote a script to deal with the case that the instance is in a cell, the build request is gone, but the intsance mapping doesn't have a cell mapping
16:27:05 mriedem yeah which i used here https://review.opendev.org/#/c/655908/
16:27:48 openstackgerrit Stephen Finucane proposed openstack/nova master: docs: Clarify everything CPU pinning https://review.opendev.org/683437
16:27:48 openstackgerrit Stephen Finucane proposed openstack/nova master: docs: Update CPU topologies guide to reflect the new PCPU world https://review.opendev.org/683485
16:28:16 openstackgerrit OpenStack Release Bot proposed openstack/os-vif stable/train: Update .gitreview for stable/train https://review.opendev.org/683488
16:28:21 openstackgerrit OpenStack Release Bot proposed openstack/os-vif stable/train: Update TOX/UPPER_CONSTRAINTS_FILE for stable/train https://review.opendev.org/683489
16:28:25 openstackgerrit OpenStack Release Bot proposed openstack/os-vif master: Update master for stable/train https://review.opendev.org/683490
16:28:30 stephenfin mriedem: Split ^
16:29:22 stephenfin wait, that doesn't look right
16:29:26 stephenfin :(
16:31:21 mriedem i left some other comments/questions about upgrade and quota details, not sure if you saw those while splitting
16:32:11 stephenfin ack
16:34:50 sean-k-mooney lyarwood: can you take a look at the propasl bot patches for os-vif above on stable ? they look sane to me
16:35:32 sean-k-mooney lyarwood: also it not supper urgent so it can wait till next week but they are trivial
16:38:59 mriedem dansmith: finally got around to replying to our comments on this heal instance mappings command https://review.opendev.org/#/c/655908/ - since there was no vote i hadn't noticed
16:39:10 openstackgerrit Stephen Finucane proposed openstack/nova master: docs: Clarify everything CPU pinning https://review.opendev.org/683437
16:39:10 openstackgerrit Stephen Finucane proposed openstack/nova master: docs: Update CPU topologies guide to reflect the new PCPU world https://review.opendev.org/683485
16:45:18 dansmith mriedem: okay I'll have to re-read to build my context back up
16:51:40 mnaser mriedem: the instance does have a mapping, which is where this is different than every time
16:51:55 mnaser instance_mapping in nova_api exists, and instance in nova exists..
17:10:46 mnaser waaaaaaaaaaait
17:10:53 mnaser the cell mapping points at cell0
17:11:04 mnaser but the instance exists in error state.. in the actual cell (cell1 in this case)
17:12:18 mnaser and the cell0 one _is_ marked as deleted
17:12:36 mnaser given my cell knowledge so far, you cant start from cell1 and then get buried in cell0 right?
17:13:03 mnaser you can't end up in cell0 if you're in cell1?
17:13:55 dansmith correct
17:14:03 dansmith I mean.. that's the intent/rule
17:17:00 openstackgerrit Artom Lifshitz proposed openstack/nova master: Move pre-3.44 Cinder post live migration test to test_compute_mgr https://review.opendev.org/683597
17:22:17 mriedem artom: "The previous patch (Id0e8b1c32600d53382e5ac938e403258c80221a0) created" you're learning!
17:22:38 artom mriedem, next step: multicellular organisms
17:24:39 mriedem mnaser: that's really strange, there are only two places we bury in cell0,
17:24:42 mriedem 1. if scheduling fails https://github.com/openstack/nova/blob/stable/stein/nova/conductor/manager.py#L1359
17:24:58 mriedem 2. or scheduling picks a host that doesn't have a mapping https://github.com/openstack/nova/blob/stable/stein/nova/conductor/manager.py#L1386
17:25:17 mriedem i wonder if you had hosts incorrectly mapped to cell0?
17:25:21 mriedem https://github.com/openstack/nova/blob/stable/stein/nova/conductor/manager.py#L1397
17:25:42 mriedem there shouldn't ever be host mappings pointing at cell0 but we don't explicitly fail if you try to do that
17:26:04 mriedem e.g. if you started a compute service pointing at the wrong cell database and it was registered in cell0 and then got mapped there in the host mapping
17:27:57 mriedem dansmith: do you think it'd be worthwhile to have a sanity check around https://github.com/openstack/nova/blob/stable/stein/nova/conductor/manager.py#L1421 such that if the cell is actually cell0 in that case we blow up with some kind of "idk what you're doing but this is definitely wrong"
17:30:50 mriedem that still wouldn't explain how the instance in mnaser's case went from cell0 to cell1, unless he has a busted heal script
17:32:34 mriedem maybe a host mapping pointing at cell0 isn't possible after all https://github.com/openstack/nova/blob/stable/stein/nova/objects/host_mapping.py#L253
17:33:18 mnaser yeah so in this case, i have a db record for this instance both in cell0 and cell1
17:33:25 mnaser and i dont think the config changed anytime recently..
17:33:50 mriedem is the host value set for either instance record?
17:34:16 mnaser both null
17:34:27 mnaser but the cell0 one is deleted and cell1 is not (aka deleted=0)
17:34:38 mriedem which has the older created_at time?
17:34:47 mnaser oh good idea
17:35:11 mnaser the same exact time.
17:35:21 mriedem wtf
17:35:41 mriedem you don't have some janky homebrew scripts that try to heal and revive an instance in cell0 and put it into cell1?
17:36:22 mnaser not that i know of
17:36:25 mnaser and also in case im losing it
17:36:26 mnaser http://paste.openstack.org/show/778146/
17:37:34 mriedem some sort of weird mariadb cluster issue or something?
17:37:48 mriedem you wouldn't be clustering those together though unless you really screwed up
17:38:01 mnaser its just plain old galera, and those are two different databases too..
17:38:11 mnaser so i dont know how it would like "oh let me make a cell0 thing"
17:38:24 mriedem it's clearly a phantom, some kind of 9/11 phantom, and it showed up too early for halloween
17:38:44 mriedem no obvious weirdness for that instance uuid in the logs?

Earlier   Later