Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-11-06
13:19:12 elod sean-k-mooney: sure! (at first glance the 'refactor' topic tricked me, but as far as i see it's a bugfix :))
13:30:23 sean-k-mooney ya kind of its pulling out some common code but its need for the follow up
13:57:00 mnasiadka good afternoon
13:57:19 mnasiadka does Nova support having ephemeral volumes from different backends? (e.g. one ceph, second from a local filesystem)
14:22:57 lyarwood mnasiadka: for the libvirt virt driver no, we only support a single images_type configurable
14:23:06 lyarwood mnasiadka: for multibackend support you should use cinder
14:23:27 mnasiadka lyarwood: that's what I considered as an alternative
14:23:30 mnasiadka lyarwood: thanks
14:23:32 lyarwood mnasiadka: np
14:25:53 sean-k-mooney stephenfin:.... https://github.com/openstack/nova/commit/b8695de6da56db42b83b9d9d4c330148766644be rewrote the pci generation that is why my fix is broken
14:26:49 sean-k-mooney i guess that needs to go back to queens too
14:44:18 bauzas gibi: others, some bugfix that could be backported https://review.opendev.org/#/c/758470/
14:44:25 bauzas easy peasy IMHO
14:45:58 gibi bauzas: will check soonish
14:46:04 bauzas no rush
14:46:06 bauzas but ta
14:47:15 gibi it is Friday I run out of rush alreay ;)
14:49:08 openstackgerrit sean mooney proposed openstack/nova master: Fix unplugging VIF when migrate/resize VM https://review.opendev.org/751642
14:49:08 openstackgerrit sean mooney proposed openstack/nova master: libvirt: delegate ovs plug to os-vif https://review.opendev.org/602432
14:52:36 lyarwood I'm not sure if anyone has been looking at this but I can't see an obvious issue with nova-ceph-multistore aside from instance spawns taking a long time overall
14:53:03 lyarwood I'll look at another example failure after a downstream call and file a bug to track these timeouts
14:54:13 gmann gibi: let me check. we did change one glance policy in nove ceph multistore job.
14:54:44 gmann gibi: ah this is same copy_image policy. checking why it is failing.
14:55:25 gibi gmann: thanks!
14:56:16 bauzas is it only me or you do you get a shit of exceptions when running functional and unittests locally with nova.storage.rbd_utils about the 'rados' and 'rbd' packages missing ?
14:56:24 bauzas or do*
14:56:27 openstackgerrit Ghanshyam Mann proposed openstack/nova stable/victoria: Modify glance's copy_image permission for nova-ceph-multistore https://review.opendev.org/761755
14:56:30 gibi bauzas: I see it too
14:56:42 gibi and I remember seeing a patch fixing it
14:56:45 bauzas I tested with both py38 and py39
14:56:54 bauzas but that's mostly annoying
14:56:55 gmann gibi: bauzas lyarwood backporting the nova-ceph-multistore fix https://review.opendev.org/#/c/761755/
14:57:13 gibi bauzas: https://review.opendev.org/#/c/746873/
14:57:15 gmann Tempest test merged before we backported this
14:58:08 gibi gmann: thanks!
14:58:13 bauzas gibi: meh, I should have it in my local repo
14:58:47 gibi bauzas: then maybe it is not the fix but it is the patch that generates the exception in the first place
14:58:58 lyarwood gmann: ah did that bork the job on stable?
14:59:21 gmann gibi: lyarwood nova-ceph-multistore job is from victoria so backporting until victoria is needed
14:59:29 gmann lyarwood: yeah - https://zuul.opendev.org/t/openstack/build/bf932888821345c1955cda68226b1a87/
14:59:59 gmann tempest test which started running in nova-ceph-multistore on master as well on stable/victoria started running and needed this change on nova side
15:00:16 bauzas gibi: hah, that's maybe because the exception is not the same
15:00:52 bauzas ah no
15:00:54 bauzas nevermind
15:00:58 bauzas we log the exception
15:01:02 lyarwood gmann: ack I was going to look at that failure so thanks :)
15:01:11 bauzas hence the stracktrace
15:01:18 bauzas but that's horribly annoying
15:01:45 bauzas lyarwood: any ways for us to not getting a shitty number of exception lines anytime we use tox ?
15:01:58 gmann lyarwood: we added tempest test to copy-image for private images also (https://review.opendev.org/#/c/742546/23) which needed this policy to be open for everyone
15:02:03 bauzas because of rados and rbd missing
15:02:29 bauzas lyarwood: context being https://review.opendev.org/#/c/746873/4/nova/storage/rbd_utils.py
15:02:59 lyarwood gmann: ack
15:03:37 lyarwood bauzas: we could add some logic to skip the check based on an ENV var or something in test envs?
15:04:29 lyarwood bauzas: or we add them as test requirements?
15:04:36 lyarwood bauzas: even though they are virt driver specific
15:05:06 bauzas lyarwood: context is http://paste.openstack.org/show/799789/
15:05:19 lyarwood bauzas: yeah I;ve seen it
15:05:32 bauzas lyarwood: IMHO we should lazy load the modules
15:05:42 bauzas for rdb and rados
15:06:03 bauzas or making them needed for all *nova*
15:06:11 bauzas but, definitely, the former
15:06:19 bauzas if they are optional
15:06:29 bauzas (and only for a specific driver)
15:06:39 lyarwood bauzas: we lazy loaded before silently and it was a PITA to debug issues when they failed to import
15:06:46 lyarwood bauzas: assuming that's what you mean
15:07:07 lyarwood bauzas: could we not just add them as test deps?
15:07:16 bauzas lyarwood: then, lazy load them by creating the libvirt object
15:07:26 bauzas when initting
15:07:43 bauzas gibi: ^ to be honestly, I'm against https://review.opendev.org/#/c/746873/4/nova/storage/rbd_utils.py
15:08:07 bauzas because we create more problems for folks that don't use ceph than we fix the needed ones
15:09:07 openstackgerrit Merged openstack/nova master: zuul: Merge nova-evacuate into nova-multinode-live-migration https://review.opendev.org/755309
15:09:18 dansmith bauzas: me too, that seems crazy
15:10:36 dansmith surely we could check images_type and decide whether or not to explode there?
15:10:37 gibi bauzas, lyarwood, dansmith: can we move these import into one of the classes in the rbd_utils?
15:11:03 gibi dansmith: it is not exploding the driver, it is logging the import error
15:11:10 dansmith gibi: I think the point here was to make it explode before those get loaded, but that's not really okay, IMHO
15:11:13 bauzas gibi: the problem is that when using import directly, you *need* rdb
15:11:22 bauzas by any module
15:11:34 bauzas hence either lazy-loading it
15:11:55 bauzas or having other ways to support it like we did by telling none
15:12:06 dansmith yeah, go back to the old way and make something else check to see if the rbd import is None during early init, if we're in a dependent code path
15:12:16 bauzas by providing an exception you say "sorry, but you need it"
15:12:17 dansmith check config, or put it in a class we only load if we're using ceph
15:13:10 bauzas that too
15:13:19 bauzas there are multiple ways to make sure it's optional
15:16:29 gibi filed a bug https://bugs.launchpad.net/nova/+bug/1903316
15:16:29 openstack Launchpad bug 1903316 in OpenStack Compute (nova) "excessive logging of rbd and rados import errors" [Undecided,New]
15:19:43 gibi feel free to push a fix against it ^^
15:20:00 gibi if nobody then I might get to it at some point next week
15:22:04 bauzas gibi: I can try to fix it
15:22:37 bauzas but ideally, I'd love to understand the dynamics behind us logging this as an exception and what we wanted to fix previously
15:22:38 gibi ack, thanks
15:23:00 gibi I hope lyarwood can provide such context ^^
15:23:04 bauzas because I don't see a problem with lazy-loading at init time
15:23:38 bauzas we could even make nova-compute refuse to start if config option says "you should have it"
15:24:20 dansmith right
15:24:25 lyarwood bauzas: the context there is that we previously silently failed without logging any context about why the import failed.
15:24:30 dansmith or .. log the warning if configured and missing
15:24:54 lyarwood bauzas: we had a few downstream reports where this failed even with the modules in the container image etc.

Earlier   Later