Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-06-19
15:25:25 bauzas I just think about the possible DB locks a "fix my cloud" button would make
15:25:40 mriedem bauzas: invariably we'd fuck up a "fix my cloud" thing anyway
15:26:12 bauzas ideally, this needs to work online
15:26:29 bauzas but this is the ideal
15:27:02 mriedem as i said in https://bugs.launchpad.net/nova/+bug/1793569/comments/5 i think one low hanging fruit is probably adding something to heal_allocations that scans for allocations held by migratoins which are not in progress and reports on those
15:27:02 openstack Launchpad bug 1793569 in OpenStack Compute (nova) "Add placement audit commands" [Wishlist,Confirmed]
15:27:06 bauzas belmoreira: you'd be okay with some tool that 'd fix allocations by a maintenance window ?
15:27:36 bauzas I mean, I guess the answer, it'd depend on the required time window :-)
15:28:38 bauzas mriedem: couldn't we just report allocations that are either for not-in-progress migrations or just not related to any instance or migration UUID ?
15:28:48 mriedem i'm torn about how much functionality to shove into heal_allocations b/c it's going to get more complicated https://review.opendev.org/#/q/topic:bug/1819923+(status:open+OR+status:merged)
15:29:35 mriedem bauzas: sure we *could* but that's likely a separate command i'd think
15:29:44 mriedem nova-manage placement audit_allocations or something
15:30:33 cdent which is part of how we ended up wanting consumer types
15:30:42 cdent for most clouds (so far) it won't matter because all allocations will come from nova
15:30:53 mriedem i thought about that, and we could fudge around consumer types for now by checking the resource classes involved to know it's coming from nova
15:31:12 bauzas cdent: heh touché
15:31:45 bauzas you know what ?
15:31:50 mriedem we could also use some troubleshooting docs about "my allocations seem all f'ed up, how can i tell for sure?" but i'm not sure how easy that is to write, but it's something that comes up in here almost every week i think
15:32:08 bauzas I've been given that escalation so I have free time to work on it
15:33:15 cdent having those docs and tools coming from the people who are experiencing and/or fixing the bugs would be ideal as they have the most idea of what's going on. in a perfect universe we'd have lots of people running around to do it, but we're long past that
15:33:22 bauzas mriedem: specless BP, there is ?
15:33:46 mriedem bauzas: i don't think providing tooling for fixing our messes is a feature
15:34:05 sean-k-mooney its not really a bug either
15:34:16 sean-k-mooney the thing that is cause the mess is a bug
15:34:26 sean-k-mooney i guess a tool could be a partial fix for that but
15:34:51 sean-k-mooney are you ok with tieing the tool to the bug rather then a seperate thing
15:34:57 mriedem https://bugs.launchpad.net/nova/+bug/1793569 is a bug
15:34:58 openstack Launchpad bug 1793569 in OpenStack Compute (nova) "Add placement audit commands" [Wishlist,Confirmed]
15:35:09 bauzas cool, I'll use this bug
15:35:51 sean-k-mooney am ok that feels more like an RFE then a bug but ok
15:36:03 sean-k-mooney no need to invent more paperwork when we dont have too
15:36:13 mriedem bauzas: i wouldn't mind you posting something to the ML with what you plan on adding before starting a bunch of work on it
15:36:43 bauzas ack, tagging ops too
15:36:56 belmoreira bauzas yes please
15:39:01 mriedem bauzas: you can reply to this thread i started last september http://lists.openstack.org/pipermail/openstack-discuss/2019-March/004223.html
15:39:30 bauzas mriedem: cool
15:40:30 mriedem i'll also say i expect functional tests for anything added b/c unit tests don't cut it with this kind of stuff that involves placement
15:42:04 mloza ohwhyosa: how did fix your issue?
15:43:40 bauzas belmoreira: do you know if [ops] tag in the email subject is enough for getting ops' eye on what I write ?
15:44:29 mriedem yes, it should be
15:44:47 belmoreira is the tag that we are using
15:44:58 mriedem it's what i used on http://lists.openstack.org/pipermail/openstack-discuss/2019-June/thread.html#7097
15:45:11 mriedem which is related to all of this also - nova mis-managing placement resources
15:46:46 artom sean-k-mooney, remind me again, what's the situation where we can have a single instance with bind-time and plug-time events?
15:46:53 mriedem bauzas: mayhap while you have been escalated to spend time upstream, you'd like to review https://review.opendev.org/#/q/topic:bug/1825537+(status:open+OR+status:merged) as well
15:47:11 artom OVS hybrid-plugged ports for the former, what's the latter?
15:47:23 artom 'cuz hybrid-plug is a neutron-wide setting, no?
15:47:28 mriedem artom: sriov direct-physical i thought
15:47:37 sean-k-mooney ovs + sriov or just tow different sriov ports
15:47:38 mriedem artom: different types of ports attached to the server
15:48:18 artom mriedem, that's what I have in my commit message currently (not pushed yet), but it means I got https://review.opendev.org/#/c/664431/ completely wrong
15:48:44 sean-k-mooney e.g. one port that is vnic_type=direct-physical + another port that is vnic_type=direct
15:48:50 artom I think I'l push, sean-k-mooney can shoot it down ;)
15:48:58 mriedem belmoreira: while you're around, here is another tool for ops for you https://review.opendev.org/#/c/655908/
15:49:35 sean-k-mooney artom: if i think its incorrect ills leave a comment or updated it for you when i review
15:49:45 artom sean-k-mooney, appreciated :)
15:50:17 sean-k-mooney but ya it basicaly only happens if you have two different network backend attached to the same vm which only happens if you are using sriov really
15:50:24 artom mriedem, I'm off until Tuesday, so I won't be the one harassing you for reviews. That honour goes to sean-k-mooney while I'm gone (because it's still on fire internally)
15:50:38 mriedem artom: as in you're off today?
15:50:48 artom mriedem, as of tonight, flying to Russia to see the folks
15:50:55 artom Well, this afternoon, really
15:51:13 mriedem ok so is someone going to be updating https://review.opendev.org/#/c/644881/ sometime soon because i'm getting really tired of having to re-load the context from that change into my head every other week
15:51:18 mriedem when dansmith asks me to
15:51:21 belmoreira mriedem thanks.
15:51:28 openstackgerrit Artom Lifshitz proposed openstack/nova master: Revert resize: wait for events according to hybrid plug https://review.opendev.org/644881
15:51:28 openstackgerrit Artom Lifshitz proposed openstack/nova master: [DNM] testing bug/1813789 revert resize events https://review.opendev.org/664442
15:51:33 artom mriedem, ^^ :)
15:51:44 artom And as I said, sean-k-mooney's carrying the torch
15:51:59 artom Unless it somehow merges today
15:54:02 sean-k-mooney mriedem: ill priorites any review feedback that you or dansmith have. artom ill also review this again shortly
15:57:22 jroll kashyap: so the concern on vTPM is around where keys "in the TPM" are stored. can you help me find docs on that? in other words, we don't want these keys sitting on a disk.
15:58:54 bauzas mriedem: ack, will review those
15:59:25 bauzas if working on customer escalations means more upstream time for me, then I sign off for moar :)
16:02:24 openstackgerrit Stephen Finucane proposed openstack/nova-specs master: Additional upgrade clarifications for cpu-resources https://review.opendev.org/666032
16:02:59 mriedem sean-k-mooney: i don't know what's up with https://review.opendev.org/#/c/647733/ but it's not queued in zuul and zuul hasn't reported on it
16:04:31 sean-k-mooney mriedem: patch set 7 didnt get queued either
16:04:47 sean-k-mooney mriedem: i realised like an hour ago that its still missing a test
16:04:54 sean-k-mooney so ill respin it later today
16:05:25 sean-k-mooney i still need to add the test for backleveling the object
16:06:06 sean-k-mooney :) you got to the - workflow before me
16:06:30 sean-k-mooney i think ther ewas a zuul restart yeasterday aroudn wehn i pushed it an i think that messed with it
16:06:45 mriedem you rechecked it about 3 hours ago though
16:06:59 mriedem *4
16:07:28 sean-k-mooney ya i dont know why that didnt go through.
16:07:54 sean-k-mooney anyway ill update it later today. sorry for the noise
16:27:26 sean-k-mooney alex_xu: just finished https://review.opendev.org/#/c/601596/14 im not sure we should default to copying pmem namespace for resize or cold migration
16:32:24 openstackgerrit jacky06 proposed openstack/os-traits master: Sync Sphinx requirement https://review.opendev.org/666386
16:34:30 openstackgerrit jacky06 proposed openstack/os-vif master: Sync Sphinx requirement https://review.opendev.org/666387
17:25:59 rouk so, update_available_resource in resource_tracker.py and _refresh_associations in report.py both take so long to complete that the main thread hangs long enough to miss rmq heartbeats for 2-3 minutes at a time on a specific node, causing nova service to be unusable and status to flap. does this sound familar to anyone?
17:27:10 rouk been debugging a specific compute node flapping/being broken for a while, narrowed it down this far, making these functions just return immediately has fixed the symptoms, but those functions are kinda needed for functioning.
17:27:14 sean-k-mooney rouk: no not really
17:27:31 sean-k-mooney what driver are you using
17:27:42 sean-k-mooney and how big is the host its happening on
17:28:28 rouk sean-k-mooney: libvirt, 256gb ram, 96 threads of cpu, epyc
17:28:56 sean-k-mooney and do you have a lot of running instnaces?
17:29:04 rouk other nodes at same spec and exact hw match are... fine, as far as i can tell, suddenly this unit became useless, fresh deploy of it still broken.
17:29:08 sean-k-mooney or pci deivces
17:29:35 rouk nope, totally evicted currently, its a fresh build, had to fail everyone out of it and manually recover, as nova wasnt working enough to complete any migrations.
17:30:35 rouk no crazy pcie devices, a 40gig card.
17:30:42 sean-k-mooney that seams strange. i have run nova on simlarly sized system in the past 88core intel system with 192GB of ram and seen no issues

Earlier   Later