Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-07-24
14:03:44 sean-k-mooney what i think make sense is to just have 2 code paths. if its an sriov inteface find it by the pci adresss and remove it
14:03:51 sean-k-mooney if not find it by its mac and remove it
14:05:45 sean-k-mooney ill have to look at the code and see if that makes sense in partice however as im not sure if we have the vnic_type or vif type avaiable
14:06:32 gibi the code that searchs for the interface has access to the vif 124: enp129s16f6: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN group default qlen 1000
14:06:39 gibi bah
14:06:53 gibi the code that searchs for the interface has access to the vif 124: enp129s16f6: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN group default qlen 1000
14:07:03 gibi my copy paste buffer is brokn :/
14:07:19 sean-k-mooney ok cool
14:07:28 gibi https://github.com/openstack/nova/blob/4a925cf01ac6ca313ff10c3075a86d65095de299/nova/virt/libvirt/guest.py#L252-L257
14:07:56 gibi nah, this is the place where the matching between the current domain and the vif being detached happens https://github.com/openstack/nova/blob/4a925cf01ac6ca313ff10c3075a86d65095de299/nova/virt/libvirt/guest.py#L252-L257
14:09:06 sean-k-mooney same link :) i think you ment https://github.com/openstack/nova/blob/4a925cf01ac6ca313ff10c3075a86d65095de299/nova/virt/libvirt/driver.py#L2199
14:09:14 sean-k-mooney and yes it has the vif
14:10:03 stephenfin lyarwood: Are you the person I need to shout at for nova-ceph-multistore failing? :P
14:10:22 sean-k-mooney so we could add a get_interface_by_pci_address and call that instead for sriov devices.
14:10:25 gibi sean-k-mooney: yes both yours and mine points to the code that causes the failure
14:10:30 stephenfin jk, but heads up I'm seeing a lot of failures on that today. Haven't investigated yet though
14:12:10 lyarwood stephenfin: dansmith introduced it while I was out so no ;)
14:12:16 lyarwood stephenfin: what's up?
14:12:29 openstackgerrit Balazs Gibizer proposed openstack/nova master: [WIP] Support SRIOV interface attach and detach https://review.opendev.org/740995
14:12:34 dansmith stephenfin: link?
14:12:44 sean-k-mooney stephenfin: its a modifed verion of the previous ceph job
14:12:59 stephenfin dansmith: https://review.opendev.org/#/c/741286/
14:13:14 sean-k-mooney so the test are the same but the config is slighly different to enable multistore and the image import form copy feature
14:13:55 dansmith stephenfin: thanks will look through it in a sec
14:14:10 gibi sean-k-mooney: https://review.opendev.org/#/c/740995/5/nova/virt/libvirt/guest.py@240 this change fixes the macvtap detach issue in my env, but I agree that the condition might need a refactoring to have two condition one for pci and another for mac
14:15:28 sean-k-mooney gibi: ya so that will work for macvtap but we will still fail for direct-physical
14:15:42 gibi sean-k-mooney: yes, probably, haven't tried
14:15:57 sean-k-mooney interfaces = self.get_all_devices(
14:16:00 sean-k-mooney vconfig.LibvirtConfigGuestInterface)
14:16:07 sean-k-mooney that wont return the direct-physical interfaces
14:16:31 sean-k-mooney since they are not element <interface ...> and use <hostdev ...?
14:16:41 gibi ohh
14:16:45 gibi interesting
14:16:55 sean-k-mooney also they dont have a mac in the host develement
14:17:33 sean-k-mooney libvirt cant passthough a pf with the <interface type=hostdev> only VFs
14:18:12 sean-k-mooney and the hostdev element dose not have a mac either so interface.mac_addr == cfg.mac_addr would fail
14:18:21 sean-k-mooney proably with an attribute error if we got that far
14:20:21 dansmith stephenfin: did you look into those fails at all? looks to me like just novalidhost on at least one of the three failed tests, and it's a conflict from placement during scheduling:
14:20:36 dansmith https://zuul.opendev.org/t/openstack/build/13d8a055ff1b4be0b627205f4d51d50f/log/controller/logs/screen-n-sch.txt#3493
14:21:01 dansmith meaning, are you sure it's just that job failing more? because that fails way before the point where we get to any of the new (i.e. ceph or multistore) stuff
14:21:12 sean-k-mooney there are traces in the n-cpu log https://zuul.opendev.org/t/openstack/build/13d8a055ff1b4be0b627205f4d51d50f/log/controller/logs/screen-n-cpu.txt#14267-14323
14:21:30 sean-k-mooney nova.exception.ImageNotFound: Image a549f544-e4e3-4f66-962e-03c1514ee21f could not be found
14:21:36 stephenfin dansmith: Barely. I'm seeing image retrieval failures in n-cpu
14:21:39 stephenfin yeah, those ^
14:22:13 dansmith hmm, maybe the first test I picked was a rando failure then
14:22:15 stephenfin but there are a couple of patches in that series failing and I don't think they're related to the code
14:23:15 sean-k-mooney if those tests are uploading new images maybe they are not ready when the boot is started because the import/conversion takes longer or something
14:23:43 dansmith ah yeah, I see now
14:24:25 dansmith sean-k-mooney: yeah that could be
14:24:36 dansmith I think we should still be able to GET the image though
14:25:06 sean-k-mooney looks like that is not the case for rescure at least https://github.com/openstack/tempest/blob/257f3b009f7978723a8748f9f5b413aa8eb38e3a/tempest/api/compute/servers/test_server_rescue.py#L55-L67
14:26:12 dansmith sean-k-mooney: what is not the case for rescue?
14:26:15 sean-k-mooney ya it just does rescue without specifying an image so it will use the image the vm was booted with or the image specifid in the config. i wonder if it failed before that
14:26:31 sean-k-mooney dansmith: the rescue test is not uploading any images
14:26:32 dansmith are you looking at a different fail?
14:27:04 sean-k-mooney tempest.api.compute.servers.test_server_rescue.ServerRescueTestJSON.test_rescue_unrescue_instance
14:27:13 sean-k-mooney its the second failure in the test report
14:27:39 dansmith ack, the first thing you linked is to an ImagesTest not rescue right?
14:27:48 dansmith it's definitely doing a snapshot
14:29:15 sean-k-mooney actully looking at the server uuid its not in the ncpu log so the novalid host looks like it really could not fit
14:29:55 dansmith sean-k-mooney: right, that's what I was saying, I just picked poorly on the first test to look at :)
14:29:56 dansmith sean-k-mooney: hmm, I see a DELETE of the image just before the failed GET in the glance logs, for that snapshot one, which is odd
14:30:12 sean-k-mooney yep Got no allocation candidates from the Placement API.
14:31:05 sean-k-mooney oh downstream call
14:31:20 dansmith ah
14:31:22 dansmith so,
14:31:29 dansmith I think that stack trace from sean-k-mooney is a red herring
14:31:44 dansmith I think that's an images test that tries to delete the image whilst snapshotting or something
14:31:51 dansmith it's not even one of the tests that failed in the testr report :)
14:32:10 dansmith all three of those tests are novalidhost
14:32:32 dansmith so maybe we're actually reporting something different to placement and running out of disk or something?
14:32:56 sean-k-mooney ya maybe
14:33:12 sean-k-mooney we have 80G of disk in the ci vms but it may not all be avaible int /opt
14:33:21 sean-k-mooney so i dont know we might have ran out of space
14:34:10 dansmith well,
14:34:17 dansmith it might be a reporting thing or something and not actually out of space,
14:34:28 dansmith because we're not seeing problems, just placement is refusing to find space
14:36:59 dansmith Jul 24 12:44:22.575632 ubuntu-bionic-ovh-bhs1-0018770257 devstack@placement-api.service[50512]: DEBUG placement.wsgi_wrapper [req-eeb6d563-2483-4e4f-91e8-2dc3a694ade4 req-c57d5bd6-fc4e-469d-9784-cdfe1652d653 service placement] Placement API returning an error response: Unable to allocate inventory: Unable to create allocation for 'DISK_GB' on resource provider 'e786426a-5ae2-4732-8cf6-16325fd2bf2a'. The requested amount would exceed
14:37:00 dansmith the capacity. {{(pid=50513) call_func /opt/stack/placement/placement/wsgi_wrapper.py:31}}
14:37:16 dansmith Over capacity for DISK_GB on resource provider e786426a-5ae2-4732-8cf6-16325fd2bf2a. Needed: 1, Used: 10, Capacity: 10.0
14:37:22 dansmith 10G doesn't sound right
14:40:45 mriedem random drive by comment but https://review.opendev.org/#/c/586363/
14:41:06 mriedem anyway related to ceph ci jobs?
14:41:30 dansmith I'm trying to figure out, but we are running a ceph df right before we report inventory
14:41:40 openstackgerrit Alex Deiter proposed openstack/nova master: Detach is broken for multi-attached fs-based volumes https://review.opendev.org/741712
14:54:48 sean-k-mooney dansmith: by the way if the traceback is unrelated then we likely have another silent bug as we are not catching the excpetion in the missing image case
14:54:56 dansmith sean-k-mooney: yep
14:55:06 dansmith so we're calling ceph df to get the total size of the pool and reporting that
14:55:07 sean-k-mooney dansmith: i think you are right that its unrelated
14:55:16 dansmith as best I can tell, the ceph is backed by a 24G partition
14:55:27 dansmith so I dunno where the 10G is coming from
14:56:04 sean-k-mooney this is using the ceph image backend in nova so the local_GB should be the ceph pool size right
14:56:25 dansmith well, it should be yes
14:56:43 dansmith ceph has 24G, so I'm trying to find where our images pool would be limited to 10G but not seeing it
14:56:47 dansmith one thing that might explain this,
14:57:09 dansmith is that our normal ceph job was using qcow on rbd, which is not what you're supposed to do,
14:57:24 sean-k-mooney oh ya because we have to flatten it
14:57:31 sean-k-mooney it should be raw
14:57:36 sean-k-mooney to get the cow optimization
14:57:40 dansmith and so we convert the image to raw, which is 44M per image instead of 12 or something.. although we shouldn't really be using that much space, so... hmm

Earlier   Later