Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-06-19
15:45:11 mriedem which is related to all of this also - nova mis-managing placement resources
15:46:46 artom sean-k-mooney, remind me again, what's the situation where we can have a single instance with bind-time and plug-time events?
15:46:53 mriedem bauzas: mayhap while you have been escalated to spend time upstream, you'd like to review https://review.opendev.org/#/q/topic:bug/1825537+(status:open+OR+status:merged) as well
15:47:11 artom OVS hybrid-plugged ports for the former, what's the latter?
15:47:23 artom 'cuz hybrid-plug is a neutron-wide setting, no?
15:47:28 mriedem artom: sriov direct-physical i thought
15:47:37 sean-k-mooney ovs + sriov or just tow different sriov ports
15:47:38 mriedem artom: different types of ports attached to the server
15:48:18 artom mriedem, that's what I have in my commit message currently (not pushed yet), but it means I got https://review.opendev.org/#/c/664431/ completely wrong
15:48:44 sean-k-mooney e.g. one port that is vnic_type=direct-physical + another port that is vnic_type=direct
15:48:50 artom I think I'l push, sean-k-mooney can shoot it down ;)
15:48:58 mriedem belmoreira: while you're around, here is another tool for ops for you https://review.opendev.org/#/c/655908/
15:49:35 sean-k-mooney artom: if i think its incorrect ills leave a comment or updated it for you when i review
15:49:45 artom sean-k-mooney, appreciated :)
15:50:17 sean-k-mooney but ya it basicaly only happens if you have two different network backend attached to the same vm which only happens if you are using sriov really
15:50:24 artom mriedem, I'm off until Tuesday, so I won't be the one harassing you for reviews. That honour goes to sean-k-mooney while I'm gone (because it's still on fire internally)
15:50:38 mriedem artom: as in you're off today?
15:50:48 artom mriedem, as of tonight, flying to Russia to see the folks
15:50:55 artom Well, this afternoon, really
15:51:13 mriedem ok so is someone going to be updating https://review.opendev.org/#/c/644881/ sometime soon because i'm getting really tired of having to re-load the context from that change into my head every other week
15:51:18 mriedem when dansmith asks me to
15:51:21 belmoreira mriedem thanks.
15:51:28 openstackgerrit Artom Lifshitz proposed openstack/nova master: Revert resize: wait for events according to hybrid plug https://review.opendev.org/644881
15:51:28 openstackgerrit Artom Lifshitz proposed openstack/nova master: [DNM] testing bug/1813789 revert resize events https://review.opendev.org/664442
15:51:33 artom mriedem, ^^ :)
15:51:44 artom And as I said, sean-k-mooney's carrying the torch
15:51:59 artom Unless it somehow merges today
15:54:02 sean-k-mooney mriedem: ill priorites any review feedback that you or dansmith have. artom ill also review this again shortly
15:57:22 jroll kashyap: so the concern on vTPM is around where keys "in the TPM" are stored. can you help me find docs on that? in other words, we don't want these keys sitting on a disk.
15:58:54 bauzas mriedem: ack, will review those
15:59:25 bauzas if working on customer escalations means more upstream time for me, then I sign off for moar :)
16:02:24 openstackgerrit Stephen Finucane proposed openstack/nova-specs master: Additional upgrade clarifications for cpu-resources https://review.opendev.org/666032
16:02:59 mriedem sean-k-mooney: i don't know what's up with https://review.opendev.org/#/c/647733/ but it's not queued in zuul and zuul hasn't reported on it
16:04:31 sean-k-mooney mriedem: patch set 7 didnt get queued either
16:04:47 sean-k-mooney mriedem: i realised like an hour ago that its still missing a test
16:04:54 sean-k-mooney so ill respin it later today
16:05:25 sean-k-mooney i still need to add the test for backleveling the object
16:06:06 sean-k-mooney :) you got to the - workflow before me
16:06:30 sean-k-mooney i think ther ewas a zuul restart yeasterday aroudn wehn i pushed it an i think that messed with it
16:06:45 mriedem you rechecked it about 3 hours ago though
16:06:59 mriedem *4
16:07:28 sean-k-mooney ya i dont know why that didnt go through.
16:07:54 sean-k-mooney anyway ill update it later today. sorry for the noise
16:27:26 sean-k-mooney alex_xu: just finished https://review.opendev.org/#/c/601596/14 im not sure we should default to copying pmem namespace for resize or cold migration
16:32:24 openstackgerrit jacky06 proposed openstack/os-traits master: Sync Sphinx requirement https://review.opendev.org/666386
16:34:30 openstackgerrit jacky06 proposed openstack/os-vif master: Sync Sphinx requirement https://review.opendev.org/666387
17:25:59 rouk so, update_available_resource in resource_tracker.py and _refresh_associations in report.py both take so long to complete that the main thread hangs long enough to miss rmq heartbeats for 2-3 minutes at a time on a specific node, causing nova service to be unusable and status to flap. does this sound familar to anyone?
17:27:10 rouk been debugging a specific compute node flapping/being broken for a while, narrowed it down this far, making these functions just return immediately has fixed the symptoms, but those functions are kinda needed for functioning.
17:27:14 sean-k-mooney rouk: no not really
17:27:31 sean-k-mooney what driver are you using
17:27:42 sean-k-mooney and how big is the host its happening on
17:28:28 rouk sean-k-mooney: libvirt, 256gb ram, 96 threads of cpu, epyc
17:28:56 sean-k-mooney and do you have a lot of running instnaces?
17:29:04 rouk other nodes at same spec and exact hw match are... fine, as far as i can tell, suddenly this unit became useless, fresh deploy of it still broken.
17:29:08 sean-k-mooney or pci deivces
17:29:35 rouk nope, totally evicted currently, its a fresh build, had to fail everyone out of it and manually recover, as nova wasnt working enough to complete any migrations.
17:30:35 rouk no crazy pcie devices, a 40gig card.
17:30:42 sean-k-mooney that seams strange. i have run nova on simlarly sized system in the past 88core intel system with 192GB of ram and seen no issues
17:31:35 sean-k-mooney if you do virsh capablities on the host system does it complete quickly or is it slow
17:31:40 rouk i have something like 60 other nodes at same spec without issue, this one just cant get through the libvirt calls fast enough even on a fresh deploy, fresh ip, fresh provider config, fresh hostname
17:31:45 sean-k-mooney im wondering if thise could be a libvirt issue
17:32:16 rouk it does seem to hang around in nova's libvirt.py for a while
17:32:34 sean-k-mooney what version of python are you running?
17:33:10 rouk the very first line in rocky's update_available_resource takes sometimes upwards of 60 seconds, the 2 functions i mention trade in terms of time spent, one cycle it will timeout on one, then the other
17:33:23 rouk 2.7.15
17:33:28 sean-k-mooney nova currently dose not supprot python 3.7 and i have seen the compute agent hang with 3.7
17:33:43 sean-k-mooney 2.7.15 on ubuntu bionic right?
17:34:11 sean-k-mooney mriedem: if you are around there was a gate issue with 2.7.15 recently right?
17:34:32 rouk yep
17:35:21 rouk what command should i check for cap list?
17:35:50 sean-k-mooney well i was wondering if libvirt was taking a long time to repond to that command
17:36:07 sean-k-mooney e.g. was it haveing trouble iterspecing the plathform
17:36:31 rouk capabilities returns in under 100ms it seems
17:36:40 sean-k-mooney if you had a dodgey disk or a semi broken pci device it could cause that to be slow or toher command
17:36:50 sean-k-mooney *other
17:37:36 sean-k-mooney basicaly i was trying to figure ot is it slow because the compute agent is running slow or its waithing around for io form sysfs/libvirt
17:37:40 rouk network running at a stable 20gig with 0% loss to the rabbit nodes, disks are happy, 2GB/s to the root device, no other drives in the system...
17:38:11 sean-k-mooney 2GB/s to the root disk?
17:38:17 rouk yep
17:38:28 sean-k-mooney as in you currently can write at that speed or that is the activtiy you are seeing
17:38:44 rouk capability, stable, load is nothing right now
17:39:33 sean-k-mooney if you are seeing 2GB/s of writes and the system is idel somethign is wrong but if you ran dd and it worked a 2GB/s then your disk is fine
17:40:08 rouk 3GB/s to the docker mountpoint right now via DD from /dev/zero, no load otherwise
17:40:25 rouk 0.03 load across 96 threads
17:41:12 sean-k-mooney ya ok so this is likely not related to the system hardware then or to libvirt/sysfs
17:41:16 sean-k-mooney latency
17:41:51 rouk resources = self.driver.get_available_resource(nodename) is the line that takes the longest in the first freeze spot
17:42:03 rouk that one takes upwards of 60 seconds every 2nd cycle
17:43:26 rouk _refresh_associations is hanging within the if refresh_sharing:
17:43:51 rouk also every 2nd cycle, on the cycle which the other does not freeze, they trade, like some kind of shared lock
17:44:51 rouk again, nothing special about this compute node that i can see, same config, same deploy, completely fresh, fresh rmq queues, its never even had a vm on it.
17:45:12 rouk same containers as the others, central repo so versions always match.
17:48:33 rouk sean-k-mooney: any idea for next steps? or should i just trace all the code down till i find a specific thing blocking it?
17:49:43 sean-k-mooney am if you have time that would help. these fucntion are used to update the invetory of avialble resouce in placmente so we chan scheule properly
17:50:04 sean-k-mooney so without them you cant use that node but there is obviouly somethign wrong
17:50:23 sean-k-mooney do you use pci passthouhg on that node
17:50:49 sean-k-mooney if so you could try commenting out the whitelist to rule out the pci module
17:51:10 sean-k-mooney similarly if you are using vgpus you could disabel that for debuging in teh config
17:52:05 sean-k-mooney if the issue still exists with pci passthough and vgpus disabled in the conf(assuming you were using either) that narrows down the code that could be at falut
17:52:07 rouk nope, same config as all other nodes, just rbd storage, oslo memcached, lock_path = /var/lib/nova/tmp,

Earlier   Later