Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-06-19
15:29:44 mriedem nova-manage placement audit_allocations or something
15:30:33 cdent which is part of how we ended up wanting consumer types
15:30:42 cdent for most clouds (so far) it won't matter because all allocations will come from nova
15:30:53 mriedem i thought about that, and we could fudge around consumer types for now by checking the resource classes involved to know it's coming from nova
15:31:12 bauzas cdent: heh touché
15:31:45 bauzas you know what ?
15:31:50 mriedem we could also use some troubleshooting docs about "my allocations seem all f'ed up, how can i tell for sure?" but i'm not sure how easy that is to write, but it's something that comes up in here almost every week i think
15:32:08 bauzas I've been given that escalation so I have free time to work on it
15:33:15 cdent having those docs and tools coming from the people who are experiencing and/or fixing the bugs would be ideal as they have the most idea of what's going on. in a perfect universe we'd have lots of people running around to do it, but we're long past that
15:33:22 bauzas mriedem: specless BP, there is ?
15:33:46 mriedem bauzas: i don't think providing tooling for fixing our messes is a feature
15:34:05 sean-k-mooney its not really a bug either
15:34:16 sean-k-mooney the thing that is cause the mess is a bug
15:34:26 sean-k-mooney i guess a tool could be a partial fix for that but
15:34:51 sean-k-mooney are you ok with tieing the tool to the bug rather then a seperate thing
15:34:57 mriedem https://bugs.launchpad.net/nova/+bug/1793569 is a bug
15:34:58 openstack Launchpad bug 1793569 in OpenStack Compute (nova) "Add placement audit commands" [Wishlist,Confirmed]
15:35:09 bauzas cool, I'll use this bug
15:35:51 sean-k-mooney am ok that feels more like an RFE then a bug but ok
15:36:03 sean-k-mooney no need to invent more paperwork when we dont have too
15:36:13 mriedem bauzas: i wouldn't mind you posting something to the ML with what you plan on adding before starting a bunch of work on it
15:36:43 bauzas ack, tagging ops too
15:36:56 belmoreira bauzas yes please
15:39:01 mriedem bauzas: you can reply to this thread i started last september http://lists.openstack.org/pipermail/openstack-discuss/2019-March/004223.html
15:39:30 bauzas mriedem: cool
15:40:30 mriedem i'll also say i expect functional tests for anything added b/c unit tests don't cut it with this kind of stuff that involves placement
15:42:04 mloza ohwhyosa: how did fix your issue?
15:43:40 bauzas belmoreira: do you know if [ops] tag in the email subject is enough for getting ops' eye on what I write ?
15:44:29 mriedem yes, it should be
15:44:47 belmoreira is the tag that we are using
15:44:58 mriedem it's what i used on http://lists.openstack.org/pipermail/openstack-discuss/2019-June/thread.html#7097
15:45:11 mriedem which is related to all of this also - nova mis-managing placement resources
15:46:46 artom sean-k-mooney, remind me again, what's the situation where we can have a single instance with bind-time and plug-time events?
15:46:53 mriedem bauzas: mayhap while you have been escalated to spend time upstream, you'd like to review https://review.opendev.org/#/q/topic:bug/1825537+(status:open+OR+status:merged) as well
15:47:11 artom OVS hybrid-plugged ports for the former, what's the latter?
15:47:23 artom 'cuz hybrid-plug is a neutron-wide setting, no?
15:47:28 mriedem artom: sriov direct-physical i thought
15:47:37 sean-k-mooney ovs + sriov or just tow different sriov ports
15:47:38 mriedem artom: different types of ports attached to the server
15:48:18 artom mriedem, that's what I have in my commit message currently (not pushed yet), but it means I got https://review.opendev.org/#/c/664431/ completely wrong
15:48:44 sean-k-mooney e.g. one port that is vnic_type=direct-physical + another port that is vnic_type=direct
15:48:50 artom I think I'l push, sean-k-mooney can shoot it down ;)
15:48:58 mriedem belmoreira: while you're around, here is another tool for ops for you https://review.opendev.org/#/c/655908/
15:49:35 sean-k-mooney artom: if i think its incorrect ills leave a comment or updated it for you when i review
15:49:45 artom sean-k-mooney, appreciated :)
15:50:17 sean-k-mooney but ya it basicaly only happens if you have two different network backend attached to the same vm which only happens if you are using sriov really
15:50:24 artom mriedem, I'm off until Tuesday, so I won't be the one harassing you for reviews. That honour goes to sean-k-mooney while I'm gone (because it's still on fire internally)
15:50:38 mriedem artom: as in you're off today?
15:50:48 artom mriedem, as of tonight, flying to Russia to see the folks
15:50:55 artom Well, this afternoon, really
15:51:13 mriedem ok so is someone going to be updating https://review.opendev.org/#/c/644881/ sometime soon because i'm getting really tired of having to re-load the context from that change into my head every other week
15:51:18 mriedem when dansmith asks me to
15:51:21 belmoreira mriedem thanks.
15:51:28 openstackgerrit Artom Lifshitz proposed openstack/nova master: Revert resize: wait for events according to hybrid plug https://review.opendev.org/644881
15:51:28 openstackgerrit Artom Lifshitz proposed openstack/nova master: [DNM] testing bug/1813789 revert resize events https://review.opendev.org/664442
15:51:33 artom mriedem, ^^ :)
15:51:44 artom And as I said, sean-k-mooney's carrying the torch
15:51:59 artom Unless it somehow merges today
15:54:02 sean-k-mooney mriedem: ill priorites any review feedback that you or dansmith have. artom ill also review this again shortly
15:57:22 jroll kashyap: so the concern on vTPM is around where keys "in the TPM" are stored. can you help me find docs on that? in other words, we don't want these keys sitting on a disk.
15:58:54 bauzas mriedem: ack, will review those
15:59:25 bauzas if working on customer escalations means more upstream time for me, then I sign off for moar :)
16:02:24 openstackgerrit Stephen Finucane proposed openstack/nova-specs master: Additional upgrade clarifications for cpu-resources https://review.opendev.org/666032
16:02:59 mriedem sean-k-mooney: i don't know what's up with https://review.opendev.org/#/c/647733/ but it's not queued in zuul and zuul hasn't reported on it
16:04:31 sean-k-mooney mriedem: patch set 7 didnt get queued either
16:04:47 sean-k-mooney mriedem: i realised like an hour ago that its still missing a test
16:04:54 sean-k-mooney so ill respin it later today
16:05:25 sean-k-mooney i still need to add the test for backleveling the object
16:06:06 sean-k-mooney :) you got to the - workflow before me
16:06:30 sean-k-mooney i think ther ewas a zuul restart yeasterday aroudn wehn i pushed it an i think that messed with it
16:06:45 mriedem you rechecked it about 3 hours ago though
16:06:59 mriedem *4
16:07:28 sean-k-mooney ya i dont know why that didnt go through.
16:07:54 sean-k-mooney anyway ill update it later today. sorry for the noise
16:27:26 sean-k-mooney alex_xu: just finished https://review.opendev.org/#/c/601596/14 im not sure we should default to copying pmem namespace for resize or cold migration
16:32:24 openstackgerrit jacky06 proposed openstack/os-traits master: Sync Sphinx requirement https://review.opendev.org/666386
16:34:30 openstackgerrit jacky06 proposed openstack/os-vif master: Sync Sphinx requirement https://review.opendev.org/666387
17:25:59 rouk so, update_available_resource in resource_tracker.py and _refresh_associations in report.py both take so long to complete that the main thread hangs long enough to miss rmq heartbeats for 2-3 minutes at a time on a specific node, causing nova service to be unusable and status to flap. does this sound familar to anyone?
17:27:10 rouk been debugging a specific compute node flapping/being broken for a while, narrowed it down this far, making these functions just return immediately has fixed the symptoms, but those functions are kinda needed for functioning.
17:27:14 sean-k-mooney rouk: no not really
17:27:31 sean-k-mooney what driver are you using
17:27:42 sean-k-mooney and how big is the host its happening on
17:28:28 rouk sean-k-mooney: libvirt, 256gb ram, 96 threads of cpu, epyc
17:28:56 sean-k-mooney and do you have a lot of running instnaces?
17:29:04 rouk other nodes at same spec and exact hw match are... fine, as far as i can tell, suddenly this unit became useless, fresh deploy of it still broken.
17:29:08 sean-k-mooney or pci deivces
17:29:35 rouk nope, totally evicted currently, its a fresh build, had to fail everyone out of it and manually recover, as nova wasnt working enough to complete any migrations.
17:30:35 rouk no crazy pcie devices, a 40gig card.
17:30:42 sean-k-mooney that seams strange. i have run nova on simlarly sized system in the past 88core intel system with 192GB of ram and seen no issues
17:31:35 sean-k-mooney if you do virsh capablities on the host system does it complete quickly or is it slow
17:31:40 rouk i have something like 60 other nodes at same spec without issue, this one just cant get through the libvirt calls fast enough even on a fresh deploy, fresh ip, fresh provider config, fresh hostname
17:31:45 sean-k-mooney im wondering if thise could be a libvirt issue
17:32:16 rouk it does seem to hang around in nova's libvirt.py for a while
17:32:34 sean-k-mooney what version of python are you running?
17:33:10 rouk the very first line in rocky's update_available_resource takes sometimes upwards of 60 seconds, the 2 functions i mention trade in terms of time spent, one cycle it will timeout on one, then the other
17:33:23 rouk 2.7.15
17:33:28 sean-k-mooney nova currently dose not supprot python 3.7 and i have seen the compute agent hang with 3.7
17:33:43 sean-k-mooney 2.7.15 on ubuntu bionic right?
17:34:11 sean-k-mooney mriedem: if you are around there was a gate issue with 2.7.15 recently right?
17:34:32 rouk yep

Earlier   Later