| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-06-19 | |||
| 17:37:36 | sean-k-mooney | basicaly i was trying to figure ot is it slow because the compute agent is running slow or its waithing around for io form sysfs/libvirt | |
| 17:37:40 | rouk | network running at a stable 20gig with 0% loss to the rabbit nodes, disks are happy, 2GB/s to the root device, no other drives in the system... | |
| 17:38:11 | sean-k-mooney | 2GB/s to the root disk? | |
| 17:38:17 | rouk | yep | |
| 17:38:28 | sean-k-mooney | as in you currently can write at that speed or that is the activtiy you are seeing | |
| 17:38:44 | rouk | capability, stable, load is nothing right now | |
| 17:39:33 | sean-k-mooney | if you are seeing 2GB/s of writes and the system is idel somethign is wrong but if you ran dd and it worked a 2GB/s then your disk is fine | |
| 17:40:08 | rouk | 3GB/s to the docker mountpoint right now via DD from /dev/zero, no load otherwise | |
| 17:40:25 | rouk | 0.03 load across 96 threads | |
| 17:41:12 | sean-k-mooney | ya ok so this is likely not related to the system hardware then or to libvirt/sysfs | |
| 17:41:16 | sean-k-mooney | latency | |
| 17:41:51 | rouk | resources = self.driver.get_available_resource(nodename) is the line that takes the longest in the first freeze spot | |
| 17:42:03 | rouk | that one takes upwards of 60 seconds every 2nd cycle | |
| 17:43:26 | rouk | _refresh_associations is hanging within the if refresh_sharing: | |
| 17:43:51 | rouk | also every 2nd cycle, on the cycle which the other does not freeze, they trade, like some kind of shared lock | |
| 17:44:51 | rouk | again, nothing special about this compute node that i can see, same config, same deploy, completely fresh, fresh rmq queues, its never even had a vm on it. | |
| 17:45:12 | rouk | same containers as the others, central repo so versions always match. | |
| 17:48:33 | rouk | sean-k-mooney: any idea for next steps? or should i just trace all the code down till i find a specific thing blocking it? | |
| 17:49:43 | sean-k-mooney | am if you have time that would help. these fucntion are used to update the invetory of avialble resouce in placmente so we chan scheule properly | |
| 17:50:04 | sean-k-mooney | so without them you cant use that node but there is obviouly somethign wrong | |
| 17:50:23 | sean-k-mooney | do you use pci passthouhg on that node | |
| 17:50:49 | sean-k-mooney | if so you could try commenting out the whitelist to rule out the pci module | |
| 17:51:10 | sean-k-mooney | similarly if you are using vgpus you could disabel that for debuging in teh config | |
| 17:52:05 | sean-k-mooney | if the issue still exists with pci passthough and vgpus disabled in the conf(assuming you were using either) that narrows down the code that could be at falut | |
| 17:52:07 | rouk | nope, same config as all other nodes, just rbd storage, oslo memcached, lock_path = /var/lib/nova/tmp, | |
| 17:52:27 | sean-k-mooney | ok so we can ignore those codepaths then | |
| 17:52:28 | rouk | no major customization other than standard kolla config for a compute node. | |
| 17:52:42 | sean-k-mooney | what relese did you say you were using | |
| 17:52:44 | rouk | can dump you configs if it would help | |
| 17:53:33 | rouk | rocky, nova==18.1.0, it was working on this node up until monday, no changes in version, got some stein deployments that are happy too. | |
| 17:54:11 | rouk | could move up to 18.2.1 i guess, if a fresh one would help maybe? | |
| 17:58:55 | openstackgerrit | Adam Spiers proposed openstack/nova master: Add extra spec parameter and image property for memory encryption https://review.opendev.org/664420 | |
| 17:59:09 | sean-k-mooney | am can you file a bug for this and inclode as much info as you can | |
| 17:59:52 | rouk | well if nobody has heard of it, and nobody has had similar, i would need to find the exact cause before a bugreport would be any use... | |
| 17:59:57 | sean-k-mooney | it sound like there is a deadlock or somethin that is preventing thinks form correctly yeilding | |
| 18:00:06 | openstackgerrit | Adam Spiers proposed openstack/nova master: Use fake flavor instead of empty dict in test https://review.opendev.org/662555 | |
| 18:00:06 | openstackgerrit | Adam Spiers proposed openstack/nova master: Pass extra_specs to flavor in vif tests https://review.opendev.org/662556 | |
| 18:00:07 | openstackgerrit | Adam Spiers proposed openstack/nova master: Extract SEV-specific bits on host detection https://review.opendev.org/636334 | |
| 18:00:07 | openstackgerrit | Adam Spiers proposed openstack/nova master: Add |
|
| 18:00:08 | openstackgerrit | Adam Spiers proposed openstack/nova master: Allow guest devices to include |
|
| 18:00:08 | openstackgerrit | Adam Spiers proposed openstack/nova master: Detect that SEV is required and enable iommu for devices https://review.opendev.org/644565 | |
| 18:00:09 | openstackgerrit | Adam Spiers proposed openstack/nova master: Use |
|
| 18:00:09 | openstackgerrit | Adam Spiers proposed openstack/nova master: Enable memory locking if SEV is requested https://review.opendev.org/662558 | |
| 18:00:33 | sean-k-mooney | rouk: well you dont need to fuly root cause it to open the bug you could update it as you go with your findings | |
| 18:00:56 | aspiers | sean-k-mooney, mriedem: this is the best I could come up with for the fake dict issue: https://review.opendev.org/#/c/664420/10/nova/virt/hardware.py@1142 | |
| 18:13:58 | mriedem | sean-k-mooney: the 2.7.15 python thing recently was that hacking check unit test failure | |
| 18:14:23 | mriedem | https://bugs.launchpad.net/nova/+bug/1832392 | |
| 18:14:24 | openstack | Launchpad bug 1804062 in nova (Ubuntu Eoan) "duplicate for #1832392 test_hacking fails for python 3.6.7 and newer" [High,Triaged] | |
| 18:16:04 | mriedem | aspiers: comment inline | |
| 18:25:24 | sean-k-mooney | mriedem: kolla-ansible also had some other issue but i think rouk issue is related to ceph network connectinvity | |
| 18:26:08 | aspiers | mriedem: thanks! | |
| 18:26:37 | sean-k-mooney | not 100% sure but when they tried to conenct to ceph form the node the ceph status commnd was filling speradically so that might be cause the agent to hang when it tries to get the capasity fo the rbd pool | |
| 18:31:21 | rouk | yeah, checking for ip conflicts that somehow got into the network managed only by kolla... | |
| 18:31:32 | rouk | see what else we can find, its flapping like an ip conflict. | |
| 18:32:10 | sean-k-mooney | you mentioned you changed the host name between redeploying right | |
| 18:32:25 | sean-k-mooney | did you use the kolla ansible bootstrap playbook | |
| 18:32:49 | sean-k-mooney | that templates out a /etc/hosts file with static assignments for all nodes | |
| 18:33:10 | sean-k-mooney | but if you are also using dhcp that could maybe cause issues | |
| 18:33:36 | sean-k-mooney | its been a while since i woked on kolla so they may have changed that | |
| 18:41:40 | mriedem | sean-k-mooney: mlavalle: comments on https://review.opendev.org/#/c/644881/ and question about the assertion that ovs hybrid plug vif types are "neutron wide" configuration - that seems surprising to me | |
| 18:41:48 | mriedem | i thought that conifguration was per neutron agent | |
| 18:48:15 | artom | mriedem, thanks for the review - we're in the final throws of packing, so sean-k-mooney will have to follow up. I've left a reply for 1 thing though. | |
| 19:40:47 | mriedem | https://review.opendev.org/#/c/571265/ adds a functional test for a scheduler filter that was otherwise not tested (i don't think when i wrote it anyway), open for over a year now, has a +2 if someone can look | |
| 19:50:27 | mriedem | is melwitt out this week? | |
| 19:51:32 | mriedem | dansmith: looking back on https://bugs.launchpad.net/nova/+bug/1773945 and https://bugs.launchpad.net/nova/+bug/1784074 - if we get here in conductor where the instance is deleted after the build request is created and we've done scheduling but before the instance is created in a cell, we delete the build request here and continue to the next instance we're building: https://github.com/openstack/nova/blob/74aebe0d4e5a978a4001 | |
| 19:51:35 | openstack | Launchpad bug 1773945 in OpenStack Compute (nova) "nova client servers.list crashes with bad marker" [Medium,In progress] - Assigned to Matt Riedemann (mriedem) | |
| 19:51:35 | mriedem | 0aee9e70e98246c4/nova/conductor/manager.py#L1350 | |
| 19:51:36 | openstack | Launchpad bug 1784074 in OpenStack Compute (nova) "Instances end up with no cell assigned in instance_mappings" [Medium,In progress] - Assigned to Matt Riedemann (mriedem) | |
| 19:51:46 | mriedem | we don't bury that instance in cell0 presumably because it's already been deleted via build request | |
| 19:52:05 | mriedem | however, since we don't bury it, and the build request is gone, and we didn't create the instance, we could have an orphaned instance mapping, yeah? | |
| 19:52:33 | jgwentworth | no, just didn't want to lose my other nick from not using it | |
| 19:54:31 | melwitt | concatenating not working | |
| 19:54:43 | mriedem | ok, was going to ping you on ^ as well | |
| 19:55:02 | melwitt | what you're saying makes sense though | |
| 19:55:44 | melwitt | interestingly recently I was trying to figure out how an instance could possibly be in both cell0 and cell1 in a single cell env, since that's happened to rdo cloud several times. but I found no way | |
| 19:56:05 | melwitt | so if you saw how that could be, lmk | |
| 19:59:02 | dansmith | mriedem: maybe? | |
| 19:59:19 | melwitt | so, we create instance mapping with cell_id=NULL in compute/api. so if we get to conductor and no build request found, we add a placeholder to the instance list | |
| 20:00:15 | melwitt | we don't create an instance record | |
| 20:01:02 | mriedem | right we'd skip those here https://github.com/openstack/nova/blob/74aebe0d4e5a978a40011e890aee9e70e98246c4/nova/conductor/manager.py#L1391 | |
| 20:01:29 | dansmith | yeah was just reading that loop | |
| 20:01:40 | dansmith | we assume that if instance is none that it's because we buried it already I think | |
| 20:01:42 | mriedem | while i'm looking at this code, we should probably make this fatal now :) https://github.com/openstack/nova/blob/74aebe0d4e5a978a40011e890aee9e70e98246c4/nova/conductor/manager.py#L1238 | |
| 20:01:58 | melwitt | we delete all the stuff if quota recheck fails. but if someone has requested a delete of the thing, wouldn't the compute/api delete the instance mapping? I can't remember where we do that for delete | |
| 20:02:03 | mriedem | dansmith: or it's already been deleted via the build request before we got that far | |
| 20:02:23 | mriedem | melwitt: no we don't delete the instance mapping when deleting the instance (or build request) in the api | |
| 20:02:41 | mriedem | that's why surya had to add it to the archive_deleted_rows cmd | |
| 20:02:51 | dansmith | mriedem: yeah, we get instance=None if the build request failed, do we get it for other reasons? | |
| 20:03:04 | melwitt | oh, ever? if we never do, then is it an issue? | |
| 20:03:06 | dansmith | ohh, | |
| 20:03:08 | dansmith | I see, | |
| 20:03:09 | openstackgerrit | Adam Spiers proposed openstack/nova master: Add extra spec parameter and image property for memory encryption https://review.opendev.org/664420 | |
| 20:03:18 | dansmith | we bury it right away if we get hostmappingnotfound | |
| 20:03:33 | dansmith | but if we get instance mapping not found, we don't bury, but instances.append(None) | |
| 20:03:40 | dansmith | the loop at the bottom assumes the former behavior | |
| 20:04:07 | dansmith | the latter looks like it was added for quota counting reasons | |
| 20:04:22 | melwitt | yeah, it was | |
| 20:04:26 | mriedem | dansmith: you mean build request not found | |
| 20:04:34 | melwitt | the append None thing | |
| 20:04:36 | mriedem | https://github.com/openstack/nova/blob/74aebe0d4e5a978a40011e890aee9e70e98246c4/nova/conductor/manager.py#L1350 | |