| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-06-19 | |||
| 15:57:22 | jroll | kashyap: so the concern on vTPM is around where keys "in the TPM" are stored. can you help me find docs on that? in other words, we don't want these keys sitting on a disk. | |
| 15:58:54 | bauzas | mriedem: ack, will review those | |
| 15:59:25 | bauzas | if working on customer escalations means more upstream time for me, then I sign off for moar :) | |
| 16:02:24 | openstackgerrit | Stephen Finucane proposed openstack/nova-specs master: Additional upgrade clarifications for cpu-resources https://review.opendev.org/666032 | |
| 16:02:59 | mriedem | sean-k-mooney: i don't know what's up with https://review.opendev.org/#/c/647733/ but it's not queued in zuul and zuul hasn't reported on it | |
| 16:04:31 | sean-k-mooney | mriedem: patch set 7 didnt get queued either | |
| 16:04:47 | sean-k-mooney | mriedem: i realised like an hour ago that its still missing a test | |
| 16:04:54 | sean-k-mooney | so ill respin it later today | |
| 16:05:25 | sean-k-mooney | i still need to add the test for backleveling the object | |
| 16:06:06 | sean-k-mooney | :) you got to the - workflow before me | |
| 16:06:30 | sean-k-mooney | i think ther ewas a zuul restart yeasterday aroudn wehn i pushed it an i think that messed with it | |
| 16:06:45 | mriedem | you rechecked it about 3 hours ago though | |
| 16:06:59 | mriedem | *4 | |
| 16:07:28 | sean-k-mooney | ya i dont know why that didnt go through. | |
| 16:07:54 | sean-k-mooney | anyway ill update it later today. sorry for the noise | |
| 16:27:26 | sean-k-mooney | alex_xu: just finished https://review.opendev.org/#/c/601596/14 im not sure we should default to copying pmem namespace for resize or cold migration | |
| 16:32:24 | openstackgerrit | jacky06 proposed openstack/os-traits master: Sync Sphinx requirement https://review.opendev.org/666386 | |
| 16:34:30 | openstackgerrit | jacky06 proposed openstack/os-vif master: Sync Sphinx requirement https://review.opendev.org/666387 | |
| 17:25:59 | rouk | so, update_available_resource in resource_tracker.py and _refresh_associations in report.py both take so long to complete that the main thread hangs long enough to miss rmq heartbeats for 2-3 minutes at a time on a specific node, causing nova service to be unusable and status to flap. does this sound familar to anyone? | |
| 17:27:10 | rouk | been debugging a specific compute node flapping/being broken for a while, narrowed it down this far, making these functions just return immediately has fixed the symptoms, but those functions are kinda needed for functioning. | |
| 17:27:14 | sean-k-mooney | rouk: no not really | |
| 17:27:31 | sean-k-mooney | what driver are you using | |
| 17:27:42 | sean-k-mooney | and how big is the host its happening on | |
| 17:28:28 | rouk | sean-k-mooney: libvirt, 256gb ram, 96 threads of cpu, epyc | |
| 17:28:56 | sean-k-mooney | and do you have a lot of running instnaces? | |
| 17:29:04 | rouk | other nodes at same spec and exact hw match are... fine, as far as i can tell, suddenly this unit became useless, fresh deploy of it still broken. | |
| 17:29:08 | sean-k-mooney | or pci deivces | |
| 17:29:35 | rouk | nope, totally evicted currently, its a fresh build, had to fail everyone out of it and manually recover, as nova wasnt working enough to complete any migrations. | |
| 17:30:35 | rouk | no crazy pcie devices, a 40gig card. | |
| 17:30:42 | sean-k-mooney | that seams strange. i have run nova on simlarly sized system in the past 88core intel system with 192GB of ram and seen no issues | |
| 17:31:35 | sean-k-mooney | if you do virsh capablities on the host system does it complete quickly or is it slow | |
| 17:31:40 | rouk | i have something like 60 other nodes at same spec without issue, this one just cant get through the libvirt calls fast enough even on a fresh deploy, fresh ip, fresh provider config, fresh hostname | |
| 17:31:45 | sean-k-mooney | im wondering if thise could be a libvirt issue | |
| 17:32:16 | rouk | it does seem to hang around in nova's libvirt.py for a while | |
| 17:32:34 | sean-k-mooney | what version of python are you running? | |
| 17:33:10 | rouk | the very first line in rocky's update_available_resource takes sometimes upwards of 60 seconds, the 2 functions i mention trade in terms of time spent, one cycle it will timeout on one, then the other | |
| 17:33:23 | rouk | 2.7.15 | |
| 17:33:28 | sean-k-mooney | nova currently dose not supprot python 3.7 and i have seen the compute agent hang with 3.7 | |
| 17:33:43 | sean-k-mooney | 2.7.15 on ubuntu bionic right? | |
| 17:34:11 | sean-k-mooney | mriedem: if you are around there was a gate issue with 2.7.15 recently right? | |
| 17:34:32 | rouk | yep | |
| 17:35:21 | rouk | what command should i check for cap list? | |
| 17:35:50 | sean-k-mooney | well i was wondering if libvirt was taking a long time to repond to that command | |
| 17:36:07 | sean-k-mooney | e.g. was it haveing trouble iterspecing the plathform | |
| 17:36:31 | rouk | capabilities returns in under 100ms it seems | |
| 17:36:40 | sean-k-mooney | if you had a dodgey disk or a semi broken pci device it could cause that to be slow or toher command | |
| 17:36:50 | sean-k-mooney | *other | |
| 17:37:36 | sean-k-mooney | basicaly i was trying to figure ot is it slow because the compute agent is running slow or its waithing around for io form sysfs/libvirt | |
| 17:37:40 | rouk | network running at a stable 20gig with 0% loss to the rabbit nodes, disks are happy, 2GB/s to the root device, no other drives in the system... | |
| 17:38:11 | sean-k-mooney | 2GB/s to the root disk? | |
| 17:38:17 | rouk | yep | |
| 17:38:28 | sean-k-mooney | as in you currently can write at that speed or that is the activtiy you are seeing | |
| 17:38:44 | rouk | capability, stable, load is nothing right now | |
| 17:39:33 | sean-k-mooney | if you are seeing 2GB/s of writes and the system is idel somethign is wrong but if you ran dd and it worked a 2GB/s then your disk is fine | |
| 17:40:08 | rouk | 3GB/s to the docker mountpoint right now via DD from /dev/zero, no load otherwise | |
| 17:40:25 | rouk | 0.03 load across 96 threads | |
| 17:41:12 | sean-k-mooney | ya ok so this is likely not related to the system hardware then or to libvirt/sysfs | |
| 17:41:16 | sean-k-mooney | latency | |
| 17:41:51 | rouk | resources = self.driver.get_available_resource(nodename) is the line that takes the longest in the first freeze spot | |
| 17:42:03 | rouk | that one takes upwards of 60 seconds every 2nd cycle | |
| 17:43:26 | rouk | _refresh_associations is hanging within the if refresh_sharing: | |
| 17:43:51 | rouk | also every 2nd cycle, on the cycle which the other does not freeze, they trade, like some kind of shared lock | |
| 17:44:51 | rouk | again, nothing special about this compute node that i can see, same config, same deploy, completely fresh, fresh rmq queues, its never even had a vm on it. | |
| 17:45:12 | rouk | same containers as the others, central repo so versions always match. | |
| 17:48:33 | rouk | sean-k-mooney: any idea for next steps? or should i just trace all the code down till i find a specific thing blocking it? | |
| 17:49:43 | sean-k-mooney | am if you have time that would help. these fucntion are used to update the invetory of avialble resouce in placmente so we chan scheule properly | |
| 17:50:04 | sean-k-mooney | so without them you cant use that node but there is obviouly somethign wrong | |
| 17:50:23 | sean-k-mooney | do you use pci passthouhg on that node | |
| 17:50:49 | sean-k-mooney | if so you could try commenting out the whitelist to rule out the pci module | |
| 17:51:10 | sean-k-mooney | similarly if you are using vgpus you could disabel that for debuging in teh config | |
| 17:52:05 | sean-k-mooney | if the issue still exists with pci passthough and vgpus disabled in the conf(assuming you were using either) that narrows down the code that could be at falut | |
| 17:52:07 | rouk | nope, same config as all other nodes, just rbd storage, oslo memcached, lock_path = /var/lib/nova/tmp, | |
| 17:52:27 | sean-k-mooney | ok so we can ignore those codepaths then | |
| 17:52:28 | rouk | no major customization other than standard kolla config for a compute node. | |
| 17:52:42 | sean-k-mooney | what relese did you say you were using | |
| 17:52:44 | rouk | can dump you configs if it would help | |
| 17:53:33 | rouk | rocky, nova==18.1.0, it was working on this node up until monday, no changes in version, got some stein deployments that are happy too. | |
| 17:54:11 | rouk | could move up to 18.2.1 i guess, if a fresh one would help maybe? | |
| 17:58:55 | openstackgerrit | Adam Spiers proposed openstack/nova master: Add extra spec parameter and image property for memory encryption https://review.opendev.org/664420 | |
| 17:59:09 | sean-k-mooney | am can you file a bug for this and inclode as much info as you can | |
| 17:59:52 | rouk | well if nobody has heard of it, and nobody has had similar, i would need to find the exact cause before a bugreport would be any use... | |
| 17:59:57 | sean-k-mooney | it sound like there is a deadlock or somethin that is preventing thinks form correctly yeilding | |
| 18:00:06 | openstackgerrit | Adam Spiers proposed openstack/nova master: Pass extra_specs to flavor in vif tests https://review.opendev.org/662556 | |
| 18:00:06 | openstackgerrit | Adam Spiers proposed openstack/nova master: Use fake flavor instead of empty dict in test https://review.opendev.org/662555 | |
| 18:00:07 | openstackgerrit | Adam Spiers proposed openstack/nova master: Add |
|
| 18:00:07 | openstackgerrit | Adam Spiers proposed openstack/nova master: Extract SEV-specific bits on host detection https://review.opendev.org/636334 | |
| 18:00:08 | openstackgerrit | Adam Spiers proposed openstack/nova master: Detect that SEV is required and enable iommu for devices https://review.opendev.org/644565 | |
| 18:00:08 | openstackgerrit | Adam Spiers proposed openstack/nova master: Allow guest devices to include |
|
| 18:00:09 | openstackgerrit | Adam Spiers proposed openstack/nova master: Enable memory locking if SEV is requested https://review.opendev.org/662558 | |
| 18:00:09 | openstackgerrit | Adam Spiers proposed openstack/nova master: Use |
|
| 18:00:33 | sean-k-mooney | rouk: well you dont need to fuly root cause it to open the bug you could update it as you go with your findings | |
| 18:00:56 | aspiers | sean-k-mooney, mriedem: this is the best I could come up with for the fake dict issue: https://review.opendev.org/#/c/664420/10/nova/virt/hardware.py@1142 | |
| 18:13:58 | mriedem | sean-k-mooney: the 2.7.15 python thing recently was that hacking check unit test failure | |
| 18:14:23 | mriedem | https://bugs.launchpad.net/nova/+bug/1832392 | |
| 18:14:24 | openstack | Launchpad bug 1804062 in nova (Ubuntu Eoan) "duplicate for #1832392 test_hacking fails for python 3.6.7 and newer" [High,Triaged] | |
| 18:16:04 | mriedem | aspiers: comment inline | |
| 18:25:24 | sean-k-mooney | mriedem: kolla-ansible also had some other issue but i think rouk issue is related to ceph network connectinvity | |
| 18:26:08 | aspiers | mriedem: thanks! | |
| 18:26:37 | sean-k-mooney | not 100% sure but when they tried to conenct to ceph form the node the ceph status commnd was filling speradically so that might be cause the agent to hang when it tries to get the capasity fo the rbd pool | |
| 18:31:21 | rouk | yeah, checking for ip conflicts that somehow got into the network managed only by kolla... | |