Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-04-02
15:48:15 cfriesen sean-k-mooney: I expect it's a config issue.
15:49:02 sean-k-mooney cfriesen: its posible but the pci_utils.get_ifname_by_pci_address call was intoduce for stien for band with based schudling so there could be a bug. i dont have qat devices to test with
15:49:37 sean-k-mooney well the fucntion existed before we jsut use it in more places now
15:50:48 efried sean-k-mooney: this look kosher to you: https://review.openstack.org/#/c/635533/
15:51:00 efried dansmith: ^ if you please?
15:51:35 sean-k-mooney xen is not really my thing but ill take a look
15:52:22 efried sean-k-mooney: it's not a xen thing, really an ssl thing.
15:54:07 dansmith it's not an ssl thing, it's a python/oslo thing AFAICT
15:55:21 sean-k-mooney ya its processutils exit code checking
15:55:34 sean-k-mooney but yes on second look it looks sane to me
15:56:32 sean-k-mooney efried: is there any change that the password could be loged to standard error if openssl did not like it
15:57:05 openstackgerrit Eric Fried proposed openstack/nova master: Add minimum value in max_concurrent_live_migrations https://review.openstack.org/648302
15:58:07 efried sean-k-mooney: I would seriously hope there is no ssl command in existence that will echo back a password to you :)
15:58:34 efried dansmith: only peripherally. They're just changing from "anything on stderr means we should fail" to "nonzero return code means we should fail".
15:58:48 dansmith ...right
16:00:25 sean-k-mooney efried: yes
16:00:27 dansmith efried: the patch asserts that before, writing anything to stderr would cause it to raise, even if it exited with zero, rght?
16:00:35 dansmith I don't see where in oslo that behavior happens
16:00:43 dansmith (it'd be pretty dumb, which is why I'm curious)
16:01:41 sean-k-mooney dansmith: https://github.com/openstack/oslo.concurrency/blob/master/oslo_concurrency/processutils.py#L414-L424
16:02:13 efried dansmith: It wasn't in oslo, they were triggering the exception here (in nova) based on stderr being nonempty.
16:02:16 dansmith sean-k-mooney: that's not what I'm asking about
16:02:40 dansmith efried: oh? I thought they were asserting that was oslo...
16:02:55 dansmith oh RuntImeError, I see
16:03:35 sean-k-mooney they are assertign that if openssl wrote anything to stderr then nova would raise the runtime error
16:04:15 dansmith efried: that makes it much more openssl-related than I thought, I just zeroed in on the process handling.. so I dunno, probably need someone much more familiar with openssl to validate those semantics
16:04:23 sean-k-mooney im guessing that if openssl ever decied to add a deperaction warning or somehting this caused issues for them
16:04:49 dansmith right, I definitely get that
16:05:37 sean-k-mooney i looks liek they jsut want to be a little more graceful and assume openssl follow the standard unix thing of if it returned and exit code of 0 it succeded
16:05:43 sean-k-mooney which i think is resonable
16:05:56 sean-k-mooney did the bug have an explcit example
16:06:25 dansmith sean-k-mooney: yes, that's all obvious :)
16:07:08 sean-k-mooney yes so the patch is sane yes to efried original quetion
16:07:24 sean-k-mooney the bug was reported as a deprecation warning as i gueesed
16:07:34 sean-k-mooney |RuntimeError: OpenSSL error: *** WARNING : deprecated key derivation used.
16:07:36 sean-k-mooney |Using -iter or -pbkdf2 would be better.
16:07:36 dansmith the original code didn't look at the return code for something security-related and so changing that behavior is potentially pretty impactful
16:08:40 sean-k-mooney that is possibly true yes in this specifc instance othe Runtime error that was raise i think its safe but you are concerend that there coudl be other case where it would not be
16:09:21 dansmith I'm saying they used to fail if anything was written to stderr and now they won't
16:09:26 sean-k-mooney i woudl hope openssl would not exit with code 0 for anything other then sucess but i dont know that for certine even if i belive it very likely
16:09:33 dansmith and depending on what is going on here, that could be, you know, a big deal
16:09:35 sean-k-mooney yes
16:09:42 dansmith depends on the command and what is going on
16:10:07 sean-k-mooney well we can see the command it fixed
16:10:30 dansmith I'm saying someone needs to go make that determination, IMHO and not just blindly approve this
16:10:38 sean-k-mooney we are encrypting input text with a shared key using ase-123-cbc
16:11:12 sean-k-mooney fair im still wondering why we are doing this via the shell in the first place
16:22:22 openstackgerrit Matt Riedemann proposed openstack/nova stable/pike: Add functional regression test for bug 1669054 https://review.openstack.org/649386
16:22:22 openstackgerrit Matt Riedemann proposed openstack/nova stable/pike: Fix functional tests for USE_NEUTRON https://review.openstack.org/649385
16:22:23 openstackgerrit Matt Riedemann proposed openstack/nova stable/pike: Do not persist RequestSpec.ignore_hosts https://review.openstack.org/649387
16:22:23 openstack bug 1669054 in OpenStack Compute (nova) stein "RequestSpec.ignore_hosts from resize is reused in subsequent evacuate" [Medium,In progress] https://launchpad.net/bugs/1669054 - Assigned to Matt Riedemann (mriedem)
16:23:41 openstackgerrit Merged openstack/nova master: Correct lower-constraints.txt and the related tox job https://review.openstack.org/622972
16:23:50 openstackgerrit Merged openstack/nova master: Adding tests to demonstrate bug #1821824 https://review.openstack.org/647957
16:23:51 openstack bug 1821824 in OpenStack Compute (nova) "Forbidden traits in flavor properties don't work" [Undecided,In progress] https://launchpad.net/bugs/1821824 - Assigned to Magnus Bergman (magnusbe)
16:24:02 dansmith looks like dimitri did a fairly decent analysis of the openssl code, which is good, but I also wonder if we couldn't just ignore lines that start with "WARNING" or something from the stderr and retain some of the original behavor
16:24:11 dansmith anyway, I don't really have time to dig into it super deep
16:25:42 openstackgerrit Merged openstack/nova master: Add placement as required project to functional py36 and 37 https://review.openstack.org/649068
16:34:22 openstackgerrit Merged openstack/nova master: libvirt: Use 'writeback' QEMU cache mode when 'none' is not viable https://review.openstack.org/641981
16:35:19 cfriesen sean-k-mooney: apparently nova-compute chokes on startup with this particular QAT hardware even with no whitelist/alias entries in nova.conf.
16:41:20 sean-k-mooney cfriesen: ok then this is likely related to gibi_off's change to auto lookup the netdev name for bandwith based scheduling
16:42:42 cfriesen sean-k-mooney: yeah, confirmed that we're dying in the code gibi_off added in Dec 2018.
16:44:34 cfriesen sean-k-mooney: we're hitting this with the standard embedded Intel QAT, so it's going to cause grief with standard hardware
16:45:13 sean-k-mooney its proably this bit https://github.com/openstack/nova/commit/c02e213d507c830427a86d6a4bb4f7a2f5158590#diff-f4019782d93a196a0d026479e6aa61b1R5938
16:45:58 cfriesen sean-k-mooney: the issue is that there is no "net" in the device path (i.e. /sys/bus/pci/devices/<pci_addr>/net)
16:46:23 sean-k-mooney ya
16:46:26 sean-k-mooney so https://github.com/openstack/nova/blob/c02e213d507c830427a86d6a4bb4f7a2f5158590/nova/virt/libvirt/driver.py#L5938-L5940
16:46:41 sean-k-mooney should only be executed for VF that are network devices
16:47:22 sean-k-mooney that shoudl be a simple fix
16:47:57 sean-k-mooney but we will need to land it in RC2 or backport to stien before thurday to include it in the GA release
16:49:39 sean-k-mooney cfriesen: we are expecting qat to hit the final return however
16:49:46 sean-k-mooney so there is something else going on
16:50:17 cfriesen sean-k-mooney: there are VFs for this device, so I was assuming we're enumerating the VFs
16:50:46 sean-k-mooney yay be we are only ment to report the it as type SRIOV_VF if its a nic
16:51:03 sean-k-mooney all non nic VF are ment to be TYPE_PCI
16:51:38 cfriesen sean-k-mooney: where is that code?
16:53:32 sean-k-mooney im looking for it now but its the only thing that prevented you geting a qat device instead fo a nic VF when you ahave a neturon prot of vnic_type direct in the past
17:01:09 cfriesen sean-k-mooney: it kind of looks like _get_pcidev_info() is calling self._host.device_lookup_by_name() to get the XML for the device. Is it possible libvirt is doing something different?
17:03:27 openstackgerrit Artem Vasilyev proposed openstack/nova master: systemd detection result caching nit fixes https://review.openstack.org/649229
17:03:48 sean-k-mooney is that where you are having the failure?
17:04:19 sean-k-mooney cfriesen: can you post a copy fo the error to paste.openstack.org
17:04:59 cfriesen sean-k-mooney: yeah, nova-compute startup. here's the starlingx bug, the nova stuff is partway down: https://bugs.launchpad.net/starlingx/+bug/1821938
17:05:00 openstack Launchpad bug 1821938 in StarlingX "No nova hypervisor can be enabled on workers with QAT devices" [High,Triaged]
17:05:19 sean-k-mooney cfriesen: thanks
17:05:39 cfriesen sean-k-mooney: extra info: http://paste.openstack.org/show/748734/
17:06:24 cfriesen we don't have the resources to fix this in the near future, got other stuff to deal with
17:06:24 sean-k-mooney ya so this is not failing because of https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L6067
17:06:43 sean-k-mooney its failing because of https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L6047
17:06:50 sean-k-mooney because it does not have a netdev
17:07:14 cfriesen sean-k-mooney: correct, but you were wondering why we were going down the VF path for things that werent nics
17:07:16 sean-k-mooney which is because of gibis change
17:07:34 sean-k-mooney ya but the other filtering im thinking of could be else where
17:07:48 sean-k-mooney i think we just need to put a guard around that call
17:08:00 cfriesen sean-k-mooney: looks like _get_device_capabilities() also assumes that SRIOV_VF is a NIC
17:09:12 sean-k-mooney yes it does
17:09:26 sean-k-mooney although it is reading form libvirt
17:09:34 sean-k-mooney instead of sysfs
17:09:38 sean-k-mooney so it proably fine
17:10:21 cfriesen sean-k-mooney: _get_pcinet_info calls get_net_name_by_vf_pci_address()
17:10:26 cfriesen so I think it'll choke
17:10:44 sean-k-mooney ill quickly hack something up one sec

Earlier   Later