Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-09-20
15:41:02 mriedem artom: i wonder if this is the same numa topology claim race failure thing described in the ML https://bugs.launchpad.net/nova/+bug/1829349
15:41:03 openstack Launchpad bug 1829349 in OpenStack Compute (nova) "Resource Tracker failed to update usage in case numa topology conflict happen" [Undecided,In progress] - Assigned to leehom (feli5)
15:41:06 artom mriedem, not sure that's still applicable, stephenfin did a thing around that a long time ago
15:41:11 mriedem note they are enabling he workaround config
15:42:40 artom mriedem, I don't think that's related - their bug is "if two instances have the same pins, RT blows up"
15:43:00 artom But with NUMA LM we're not supposed to end up in that situation in the first place
15:43:21 mriedem "supposed to"
15:43:33 mriedem but if we have a race with claims not reporting correctly or something, we could still race and blow up right?
15:44:00 artom Right
15:44:34 mriedem anyway there are some details from the reporter on what they think the problem is and where, so maybe would help
15:45:02 artom mriedem, well they proposed https://review.opendev.org/#/c/661208/
15:45:06 artom Which is... no
15:46:10 artom I need food
15:52:03 sean-k-mooney mriedem: if your looking for numa related bug we should fix sooner rather then later we should aim to merge and backport https://review.opendev.org/#/c/662522/ sonner rather then later
15:52:30 sean-k-mooney its alex_xu and stephenfin fix for https://bugs.launchpad.net/nova/+bug/1805767
15:52:32 openstack Launchpad bug 1805767 in OpenStack Compute (nova) "The new numa topology in the new flavor extra specs weren't parsed when resize" [Medium,In progress] - Assigned to Stephen Finucane (stephenfinucane)
15:53:42 sean-k-mooney i think of think all move operation bar shelve are broken in some way if you have a numa toplogy
15:54:10 sean-k-mooney there are speic things that work but it feels like there are more edgecase that dont then do somethimes
15:54:29 sean-k-mooney that said we have a tone of fixes for them pending in general
15:55:08 mriedem this is where i make a generic statement about lack of integration test coverage with tempest for numa,
15:55:16 mriedem and then you say, "i'm working on a ci job for that"
15:55:17 sean-k-mooney i still thinks there are enogh latent bugs to keep us going for U before im going to be happy we have hardended it enough
15:55:37 sean-k-mooney well its true
15:56:17 sean-k-mooney but also we have been without any real ci coverage for like 18 months so :(
15:56:24 mriedem you've done more to try and get numa tempest integration testing on nova changes than anyone so i'm not faulting you
15:56:52 sean-k-mooney oh i know i just whish i had more time to spend on it
15:57:03 mriedem it's just that my eyes glaze over when i hear "we have a bug with x that's related to numa"
15:57:11 sean-k-mooney although for the next week or two im going to continue looking at testing and ci
15:58:34 sean-k-mooney mriedem: i just pushed this by the way https://review.opendev.org/#/c/683431/
15:59:35 sean-k-mooney that will allow us to test cpu pinning hugepages and dpdk more or less reliably
16:00:32 sean-k-mooney im going to wait until after RC* to follow up with getting multi numa flavor on vexxhost and limestone
16:00:40 openstackgerrit Sylvain Bauza proposed openstack/nova master: Add a prelude for the Train release https://review.opendev.org/683327
16:04:11 mriedem stephenfin: i'm going to rebase https://review.opendev.org/#/c/682267/4 on top of https://review.opendev.org/#/c/674895/44 since it sounds like we're holding back on merging the former so as to not interrupte the latter
16:04:48 openstackgerrit Matt Riedemann proposed openstack/nova master: libvirt: Get the CPU model, not 'arch' from get_capabilities() https://review.opendev.org/682267
16:05:06 mriedem there wasn't even a merge conflict so i'm not sure what gerrit was complaining about
16:06:13 mriedem bauzas: do you want to re-approve https://review.opendev.org/#/c/681750/ since it was rebased?
16:06:23 mriedem the rest of that stack is stuck behind it
16:08:30 openstackgerrit Stephen Finucane proposed openstack/nova master: docs: Update CPU topologies guide to reflect the new PCPU world https://review.opendev.org/683437
16:08:37 donnyd sean-k-mooney: LGTM
16:08:42 stephenfin mriedem: Makes sense. Also, there the start of the doc ^
16:08:46 mriedem yu[p
16:08:47 donnyd I gave it my whole +1
16:09:43 bauzas mriedem: cool
16:09:52 bauzas and I call it a day
16:10:18 mriedem stephenfin: man i wish there wasn't a big refactor in that patch
16:10:32 sean-k-mooney donnyd: thanks the first thing i hope to use it for is ovs-dpdk testing with https://review.opendev.org/#/c/656580/
16:10:42 stephenfin I literally just said I shouldn't have done that to sean-k-mooney /o\
16:10:55 stephenfin (who's working up in the RH office in Dublin with me for the day)
16:11:00 sean-k-mooney yep you did
16:11:01 stephenfin mriedem: Gimme five to separate it out
16:11:41 mriedem i know it's a compulsion
16:11:50 mriedem obligatory "P
16:11:51 mriedem :P
16:15:36 sean-k-mooney stephenfin: by the way once i get teh ovs-dpdk job working on networking-ovs-dpdk i assume you have no issue with me adding it as non voting to check in os-vif and then promote it to voting around m1
16:15:57 stephenfin not on the slightest
16:16:22 sean-k-mooney cool i think the vhost user path is the only one without tempest coverate in os-vif currently
16:16:45 sean-k-mooney i also need to port one of the jobs to zuul v3 which i hope to do next week
16:23:17 mriedem mnaser: does the instance mapping have a cell mapping?
16:23:23 mriedem does the instance have a stale build request?
16:23:54 mriedem in that weird GET /servers/{server_id} 404 case before i've handled the InstanceNotFound in the API controller code, logged the traceback and then re-raised
16:24:02 mriedem so you can see where we're coming from, e.g. build request or what
16:25:29 mriedem mnaser: log the traceback here https://github.com/openstack/nova/blob/stable/stein/nova/api/openstack/common.py#L471
16:26:43 mriedem i'm pretty sure you wrote a script to deal with the case that the instance is in a cell, the build request is gone, but the intsance mapping doesn't have a cell mapping
16:27:05 mriedem yeah which i used here https://review.opendev.org/#/c/655908/
16:27:48 openstackgerrit Stephen Finucane proposed openstack/nova master: docs: Clarify everything CPU pinning https://review.opendev.org/683437
16:27:48 openstackgerrit Stephen Finucane proposed openstack/nova master: docs: Update CPU topologies guide to reflect the new PCPU world https://review.opendev.org/683485
16:28:16 openstackgerrit OpenStack Release Bot proposed openstack/os-vif stable/train: Update .gitreview for stable/train https://review.opendev.org/683488
16:28:21 openstackgerrit OpenStack Release Bot proposed openstack/os-vif stable/train: Update TOX/UPPER_CONSTRAINTS_FILE for stable/train https://review.opendev.org/683489
16:28:25 openstackgerrit OpenStack Release Bot proposed openstack/os-vif master: Update master for stable/train https://review.opendev.org/683490
16:28:30 stephenfin mriedem: Split ^
16:29:22 stephenfin wait, that doesn't look right
16:29:26 stephenfin :(
16:31:21 mriedem i left some other comments/questions about upgrade and quota details, not sure if you saw those while splitting
16:32:11 stephenfin ack
16:34:50 sean-k-mooney lyarwood: can you take a look at the propasl bot patches for os-vif above on stable ? they look sane to me
16:35:32 sean-k-mooney lyarwood: also it not supper urgent so it can wait till next week but they are trivial
16:38:59 mriedem dansmith: finally got around to replying to our comments on this heal instance mappings command https://review.opendev.org/#/c/655908/ - since there was no vote i hadn't noticed
16:39:10 openstackgerrit Stephen Finucane proposed openstack/nova master: docs: Clarify everything CPU pinning https://review.opendev.org/683437
16:39:10 openstackgerrit Stephen Finucane proposed openstack/nova master: docs: Update CPU topologies guide to reflect the new PCPU world https://review.opendev.org/683485
16:45:18 dansmith mriedem: okay I'll have to re-read to build my context back up
16:51:40 mnaser mriedem: the instance does have a mapping, which is where this is different than every time
16:51:55 mnaser instance_mapping in nova_api exists, and instance in nova exists..
17:10:46 mnaser waaaaaaaaaaait
17:10:53 mnaser the cell mapping points at cell0
17:11:04 mnaser but the instance exists in error state.. in the actual cell (cell1 in this case)
17:12:18 mnaser and the cell0 one _is_ marked as deleted
17:12:36 mnaser given my cell knowledge so far, you cant start from cell1 and then get buried in cell0 right?
17:13:03 mnaser you can't end up in cell0 if you're in cell1?
17:13:55 dansmith correct
17:14:03 dansmith I mean.. that's the intent/rule
17:17:00 openstackgerrit Artom Lifshitz proposed openstack/nova master: Move pre-3.44 Cinder post live migration test to test_compute_mgr https://review.opendev.org/683597
17:22:17 mriedem artom: "The previous patch (Id0e8b1c32600d53382e5ac938e403258c80221a0) created" you're learning!
17:22:38 artom mriedem, next step: multicellular organisms
17:24:39 mriedem mnaser: that's really strange, there are only two places we bury in cell0,
17:24:42 mriedem 1. if scheduling fails https://github.com/openstack/nova/blob/stable/stein/nova/conductor/manager.py#L1359
17:24:58 mriedem 2. or scheduling picks a host that doesn't have a mapping https://github.com/openstack/nova/blob/stable/stein/nova/conductor/manager.py#L1386
17:25:17 mriedem i wonder if you had hosts incorrectly mapped to cell0?
17:25:21 mriedem https://github.com/openstack/nova/blob/stable/stein/nova/conductor/manager.py#L1397
17:25:42 mriedem there shouldn't ever be host mappings pointing at cell0 but we don't explicitly fail if you try to do that
17:26:04 mriedem e.g. if you started a compute service pointing at the wrong cell database and it was registered in cell0 and then got mapped there in the host mapping

Earlier   Later