| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-07-24 | |||
| 13:48:22 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Add regression test for bug 1879787 https://review.opendev.org/741230 | |
| 13:49:02 | openstackgerrit | Merged openstack/nova stable/ussuri: libvirt: Handle VIR_ERR_DEVICE_MISSING when detaching devices https://review.opendev.org/742414 | |
| 13:54:45 | sean-k-mooney | gibi: hi o/ i was away for a funeral this week so just seeing your commnet on the attach/detach patch now. i can take a look at it more closely next week but still pretty burt out today. hopefully ill be less mentally exausted after the weekend | |
| 13:56:28 | sean-k-mooney | so ya i think your right there is a bug related to macvtap detach that is prexisting in the libvirt driver | |
| 13:57:33 | sean-k-mooney | well 2 one its not updating the domain correctly because its not finding the device properly and 2 its not relasing the vf claim because we just dont do that today for sriov detach | |
| 13:58:18 | sean-k-mooney | problem 1 is want prevents the vf mac from being reset and the macvtap being removed on detach | |
| 13:58:19 | gibi | sean-k-mooney: no worries. take your time to recover | |
| 13:58:53 | gibi | sean-k-mooney: I think I've just found the reason of 2 | |
| 13:58:57 | gibi | and I think I can fix it | |
| 13:59:06 | gibi | I will be away next week | |
| 13:59:34 | gibi | so feel free to touch my code or add patches to the series while I'm away and I will continue the weak after | |
| 13:59:48 | sean-k-mooney | what proably makes sense is to have 3 patches. 1 that block detach in the api, then your current one and a final patch for macvtap | |
| 14:00:20 | gibi | yes, make sense to have separate patches for the separate issues | |
| 14:00:58 | sean-k-mooney | we also need a ptach for direct-physical but it is basically the same issue the device lookup fails although it fails for a different reason | |
| 14:01:24 | sean-k-mooney | it filas because the mac is not present rather then the target_dev | |
| 14:01:58 | sean-k-mooney | but its still failing in the same if i belive https://github.com/openstack/nova/blob/4a925cf01ac6ca313ff10c3075a86d65095de299/nova/virt/libvirt/guest.py#L252-L257 | |
| 14:02:27 | gibi | sean-k-mooney: cool. I haven't had time to try direct-physical yet, | |
| 14:03:44 | sean-k-mooney | what i think make sense is to just have 2 code paths. if its an sriov inteface find it by the pci adresss and remove it | |
| 14:03:51 | sean-k-mooney | if not find it by its mac and remove it | |
| 14:05:45 | sean-k-mooney | ill have to look at the code and see if that makes sense in partice however as im not sure if we have the vnic_type or vif type avaiable | |
| 14:06:32 | gibi | the code that searchs for the interface has access to the vif 124: enp129s16f6: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN group default qlen 1000 | |
| 14:06:39 | gibi | bah | |
| 14:06:53 | gibi | the code that searchs for the interface has access to the vif 124: enp129s16f6: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN group default qlen 1000 | |
| 14:07:03 | gibi | my copy paste buffer is brokn :/ | |
| 14:07:19 | sean-k-mooney | ok cool | |
| 14:07:28 | gibi | https://github.com/openstack/nova/blob/4a925cf01ac6ca313ff10c3075a86d65095de299/nova/virt/libvirt/guest.py#L252-L257 | |
| 14:07:56 | gibi | nah, this is the place where the matching between the current domain and the vif being detached happens https://github.com/openstack/nova/blob/4a925cf01ac6ca313ff10c3075a86d65095de299/nova/virt/libvirt/guest.py#L252-L257 | |
| 14:09:06 | sean-k-mooney | same link :) i think you ment https://github.com/openstack/nova/blob/4a925cf01ac6ca313ff10c3075a86d65095de299/nova/virt/libvirt/driver.py#L2199 | |
| 14:09:14 | sean-k-mooney | and yes it has the vif | |
| 14:10:03 | stephenfin | lyarwood: Are you the person I need to shout at for nova-ceph-multistore failing? :P | |
| 14:10:22 | sean-k-mooney | so we could add a get_interface_by_pci_address and call that instead for sriov devices. | |
| 14:10:25 | gibi | sean-k-mooney: yes both yours and mine points to the code that causes the failure | |
| 14:10:30 | stephenfin | jk, but heads up I'm seeing a lot of failures on that today. Haven't investigated yet though | |
| 14:12:10 | lyarwood | stephenfin: dansmith introduced it while I was out so no ;) | |
| 14:12:16 | lyarwood | stephenfin: what's up? | |
| 14:12:29 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: [WIP] Support SRIOV interface attach and detach https://review.opendev.org/740995 | |
| 14:12:34 | dansmith | stephenfin: link? | |
| 14:12:44 | sean-k-mooney | stephenfin: its a modifed verion of the previous ceph job | |
| 14:12:59 | stephenfin | dansmith: https://review.opendev.org/#/c/741286/ | |
| 14:13:14 | sean-k-mooney | so the test are the same but the config is slighly different to enable multistore and the image import form copy feature | |
| 14:13:55 | dansmith | stephenfin: thanks will look through it in a sec | |
| 14:14:10 | gibi | sean-k-mooney: https://review.opendev.org/#/c/740995/5/nova/virt/libvirt/guest.py@240 this change fixes the macvtap detach issue in my env, but I agree that the condition might need a refactoring to have two condition one for pci and another for mac | |
| 14:15:28 | sean-k-mooney | gibi: ya so that will work for macvtap but we will still fail for direct-physical | |
| 14:15:42 | gibi | sean-k-mooney: yes, probably, haven't tried | |
| 14:15:57 | sean-k-mooney | interfaces = self.get_all_devices( | |
| 14:16:00 | sean-k-mooney | vconfig.LibvirtConfigGuestInterface) | |
| 14:16:07 | sean-k-mooney | that wont return the direct-physical interfaces | |
| 14:16:31 | sean-k-mooney | since they are not element <interface ...> and use <hostdev ...? | |
| 14:16:41 | gibi | ohh | |
| 14:16:45 | gibi | interesting | |
| 14:16:55 | sean-k-mooney | also they dont have a mac in the host develement | |
| 14:17:33 | sean-k-mooney | libvirt cant passthough a pf with the <interface type=hostdev> only VFs | |
| 14:18:12 | sean-k-mooney | and the hostdev element dose not have a mac either so interface.mac_addr == cfg.mac_addr would fail | |
| 14:18:21 | sean-k-mooney | proably with an attribute error if we got that far | |
| 14:20:21 | dansmith | stephenfin: did you look into those fails at all? looks to me like just novalidhost on at least one of the three failed tests, and it's a conflict from placement during scheduling: | |
| 14:20:36 | dansmith | https://zuul.opendev.org/t/openstack/build/13d8a055ff1b4be0b627205f4d51d50f/log/controller/logs/screen-n-sch.txt#3493 | |
| 14:21:01 | dansmith | meaning, are you sure it's just that job failing more? because that fails way before the point where we get to any of the new (i.e. ceph or multistore) stuff | |
| 14:21:12 | sean-k-mooney | there are traces in the n-cpu log https://zuul.opendev.org/t/openstack/build/13d8a055ff1b4be0b627205f4d51d50f/log/controller/logs/screen-n-cpu.txt#14267-14323 | |
| 14:21:30 | sean-k-mooney | nova.exception.ImageNotFound: Image a549f544-e4e3-4f66-962e-03c1514ee21f could not be found | |
| 14:21:36 | stephenfin | dansmith: Barely. I'm seeing image retrieval failures in n-cpu | |
| 14:21:39 | stephenfin | yeah, those ^ | |
| 14:22:13 | dansmith | hmm, maybe the first test I picked was a rando failure then | |
| 14:22:15 | stephenfin | but there are a couple of patches in that series failing and I don't think they're related to the code | |
| 14:23:15 | sean-k-mooney | if those tests are uploading new images maybe they are not ready when the boot is started because the import/conversion takes longer or something | |
| 14:23:43 | dansmith | ah yeah, I see now | |
| 14:24:25 | dansmith | sean-k-mooney: yeah that could be | |
| 14:24:36 | dansmith | I think we should still be able to GET the image though | |
| 14:25:06 | sean-k-mooney | looks like that is not the case for rescure at least https://github.com/openstack/tempest/blob/257f3b009f7978723a8748f9f5b413aa8eb38e3a/tempest/api/compute/servers/test_server_rescue.py#L55-L67 | |
| 14:26:12 | dansmith | sean-k-mooney: what is not the case for rescue? | |
| 14:26:15 | sean-k-mooney | ya it just does rescue without specifying an image so it will use the image the vm was booted with or the image specifid in the config. i wonder if it failed before that | |
| 14:26:31 | sean-k-mooney | dansmith: the rescue test is not uploading any images | |
| 14:26:32 | dansmith | are you looking at a different fail? | |
| 14:27:04 | sean-k-mooney | tempest.api.compute.servers.test_server_rescue.ServerRescueTestJSON.test_rescue_unrescue_instance | |
| 14:27:13 | sean-k-mooney | its the second failure in the test report | |
| 14:27:39 | dansmith | ack, the first thing you linked is to an ImagesTest not rescue right? | |
| 14:27:48 | dansmith | it's definitely doing a snapshot | |
| 14:29:15 | sean-k-mooney | actully looking at the server uuid its not in the ncpu log so the novalid host looks like it really could not fit | |
| 14:29:55 | dansmith | sean-k-mooney: right, that's what I was saying, I just picked poorly on the first test to look at :) | |
| 14:29:56 | dansmith | sean-k-mooney: hmm, I see a DELETE of the image just before the failed GET in the glance logs, for that snapshot one, which is odd | |
| 14:30:12 | sean-k-mooney | yep Got no allocation candidates from the Placement API. | |
| 14:31:05 | sean-k-mooney | oh downstream call | |
| 14:31:20 | dansmith | ah | |
| 14:31:22 | dansmith | so, | |
| 14:31:29 | dansmith | I think that stack trace from sean-k-mooney is a red herring | |
| 14:31:44 | dansmith | I think that's an images test that tries to delete the image whilst snapshotting or something | |
| 14:31:51 | dansmith | it's not even one of the tests that failed in the testr report :) | |
| 14:32:10 | dansmith | all three of those tests are novalidhost | |
| 14:32:32 | dansmith | so maybe we're actually reporting something different to placement and running out of disk or something? | |
| 14:32:56 | sean-k-mooney | ya maybe | |
| 14:33:12 | sean-k-mooney | we have 80G of disk in the ci vms but it may not all be avaible int /opt | |
| 14:33:21 | sean-k-mooney | so i dont know we might have ran out of space | |
| 14:34:10 | dansmith | well, | |
| 14:34:17 | dansmith | it might be a reporting thing or something and not actually out of space, | |
| 14:34:28 | dansmith | because we're not seeing problems, just placement is refusing to find space | |
| 14:36:59 | dansmith | Jul 24 12:44:22.575632 ubuntu-bionic-ovh-bhs1-0018770257 devstack@placement-api.service[50512]: DEBUG placement.wsgi_wrapper [req-eeb6d563-2483-4e4f-91e8-2dc3a694ade4 req-c57d5bd6-fc4e-469d-9784-cdfe1652d653 service placement] Placement API returning an error response: Unable to allocate inventory: Unable to create allocation for 'DISK_GB' on resource provider 'e786426a-5ae2-4732-8cf6-16325fd2bf2a'. The requested amount would exceed | |
| 14:37:00 | dansmith | the capacity. {{(pid=50513) call_func /opt/stack/placement/placement/wsgi_wrapper.py:31}} | |
| 14:37:16 | dansmith | Over capacity for DISK_GB on resource provider e786426a-5ae2-4732-8cf6-16325fd2bf2a. Needed: 1, Used: 10, Capacity: 10.0 | |
| 14:37:22 | dansmith | 10G doesn't sound right | |
| 14:40:45 | mriedem | random drive by comment but https://review.opendev.org/#/c/586363/ | |
| 14:41:06 | mriedem | anyway related to ceph ci jobs? | |