Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-07-25
11:38:22 sean-k-mooney so we call this https://github.com/openstack/nova/blob/master/nova/cache_utils.py#L47
11:40:24 sean-k-mooney there is no cache section defiend in the nova.conf http://logs.openstack.org/35/521035/8/check/tempest-full/031b0b9/controller/logs/etc/nova/nova_conf.txt.gz
11:41:30 sean-k-mooney meaning we fall back to olso_caches default with is an in memeory dogpile dict cache
11:42:48 sean-k-mooney slaweq: so looking at the metadata apis uwsgi config we are runnint 1 thread per process and 2 processes
11:43:08 sean-k-mooney so there are two workers each of which have seperate in memeory dict caches
11:43:28 sean-k-mooney slaweq: so this is like just because oslo cache is not configured to use memcache in that job
11:44:22 slaweq sean-k-mooney: so do You think that enabling memcache "globally" in all tempest based jobs would be good and could help to solve/workaround this issue?
11:44:44 sean-k-mooney i think it would not only solve this issue but signifcatntly speed up the gate
11:44:50 sean-k-mooney well maybe not
11:44:53 slaweq than always during /public-keys/ call we would already have cached data for instance
11:45:02 sean-k-mooney it is still doing an in memory dict cache
11:45:17 sean-k-mooney but it proably would resove this issue
11:45:28 sean-k-mooney yep
11:45:30 slaweq sean-k-mooney: ok, that's something
11:45:40 slaweq can You maybe write this in comment in launchpad?
11:45:51 slaweq I will try to propose patch to tempest repo probably
11:45:55 slaweq or devstack
11:47:01 sean-k-mooney sure. keysone is already configure to use memcache in that job
11:47:03 sean-k-mooney http://logs.openstack.org/35/521035/8/check/tempest-full/031b0b9/controller/logs/etc/keystone/keystone_conf.txt.gz
11:47:34 sean-k-mooney so it should be trival to copy the exact same caching logic to nova the config options are identical because they just come form oslo
11:47:39 slaweq sean-k-mooney: thx a lot
11:47:54 slaweq I will propose patch for this today
11:48:01 slaweq and we will see how it will work :)
11:48:23 slaweq but that is some idea which may solve this problem in gate and improve gate stability for all projects in fact :)
11:48:35 sean-k-mooney actully i think oslos cache's default is the null cache
11:48:49 sean-k-mooney so i would expect this to speed up the gate jobs
11:49:05 sean-k-mooney the null cache never caches anything
11:49:21 openstackgerrit Adam Spiers proposed openstack/nova master: libvirt: harden Host.get_domain_capabilities() https://review.opendev.org/670189
11:49:35 aspiers sean-k-mooney: ^^^ fixed a few minor typos/grammar issues in the comments but there is still one issue
11:50:38 sean-k-mooney aspiers: :) i think there always will be
11:50:48 sean-k-mooney aspiers: what is the latest one?
11:50:55 aspiers sean-k-mooney: just posted on the review
11:54:45 sean-k-mooney we dont have an easy way to check the excpetion unless we pars the error message text
11:54:54 aspiers yes that's what I was suggesting
11:55:03 aspiers it's not great but better than nothing
11:55:04 sean-k-mooney yes and i think that is a bad idea
11:55:12 sean-k-mooney it makes it rather fragile
11:55:26 aspiers well OK then the debug message should be more honest and not assume what it doesn't know
11:55:36 aspiers it's fine if it says it's guessing the issue
11:55:47 sean-k-mooney the debug message just states that we are skiping the arch because its in compatible with the virt type and machine type
11:55:56 sean-k-mooney we dont state which is the in compatibality
11:56:00 aspiers but it doesn't know that
11:56:06 aspiers it could fail due to libvirtd issues
11:56:08 sean-k-mooney it does
11:56:18 aspiers e.g. libvirtd crashes 1usec beforehand
11:56:29 aspiers and then you get a misleading debug message
11:56:38 sean-k-mooney so i originally was going to not have a debug message at all
11:56:47 aspiers no it's good to have one
11:56:47 sean-k-mooney so we can delete it if we want too
11:56:54 aspiers it should just not risk being wrong
11:57:06 aspiers it's fine if it says "this is *probably* what happened"
11:57:12 aspiers if that's the most likely thing
11:57:16 sean-k-mooney then we can simply state we are skiping the arch and not say way
11:57:23 sean-k-mooney *why
11:57:36 aspiers if that's the most likely thing it's better to expose the guess
11:57:47 aspiers since that's potentially more helpful to the operator/dev than not guessing
11:57:52 aspiers as long as it's not misleading
11:57:54 sean-k-mooney e.g. Skipping arch: becasue libvirt raised an error, check you libvirt logs for more info
11:57:58 aspiers think of it from the operator perspective
11:58:08 aspiers no the libvirt logs might not reveal anything
11:58:11 sean-k-mooney aspiers: this is never ment to be read by operators
11:58:13 aspiers if it was incompatible
11:58:22 sean-k-mooney that is why its at debug level
11:58:29 aspiers if you don't believe operators read DEBUG you live in a different universe ...
11:58:31 aspiers :)
11:58:42 sean-k-mooney they might but this is not intened for them
11:58:49 aspiers anyway it doesn't matter who is reading it
11:58:58 sean-k-mooney are you ok with the message i suggested above
11:59:00 aspiers the point is that the message needs to be a) not misleading b) as helpful as possible
11:59:30 aspiers OK I will paste a suggested message here, 1 sec
11:59:58 sean-k-mooney "Skipping arch: %s becasue libvirt raised an error, check you libvirt logs for more info."
12:00:03 aspiers nope
12:00:11 aspiers like I said libvirt logs might not help
12:00:32 aspiers and in this case we know we might be able to help by guessing the likely cause
12:00:49 sean-k-mooney you said you dont want it to be missleading
12:00:57 aspiers yes that is a)
12:01:01 aspiers but also b)
12:01:28 sean-k-mooney the current error message is our best guess at why the error was raised
12:01:41 aspiers Yes but it's not honest that it's a guess
12:01:48 aspiers This is better: "Failed to retrieve domain caps from libvirt for arch %s; maybe incompatible with virt_type %s / machine_type %s?"
12:02:04 sean-k-mooney it is honest it was a summary of the error message
12:02:15 sean-k-mooney we could just print the error message we get back from libvirt
12:02:31 aspiers Yes good idea
12:02:52 aspiers This is better: "Failed to retrieve domain caps from libvirt for arch %(arch)s (%(error)s); maybe incompatible with virt_type %(virt_type)s / machine_type %(mach_type)s?"
12:02:55 sean-k-mooney i wanted to avoid the stack trace but we should be able to just get the message
12:03:09 aspiers that last one includes the libvirt error message ^^^
12:03:34 sean-k-mooney i would not put the error in the middel
12:03:38 sean-k-mooney i would put it at the end
12:04:02 sean-k-mooney i guess its not that long
12:04:04 sean-k-mooney http://paste.openstack.org/show/754776/
12:04:14 sean-k-mooney its invalid argument: KVM is not supported by '/usr/bin/qemu-system-alpha' on this host
12:04:38 sean-k-mooney at least in the case where the virt type is the issue
12:04:48 aspiers OK good point
12:05:19 aspiers "Failed to retrieve domain caps from libvirt for arch %(arch)s; maybe incompatible with virt_type %(virt_type)s / machine_type %(mach_type)s? libvirt error was: %(error)s"
12:05:34 sean-k-mooney ya im fine with that
12:05:37 aspiers or actually
12:05:47 aspiers since the libvirt error is already helpful enough
12:06:04 aspiers "Failed to retrieve domain caps from libvirt for arch %(arch)s / virt_type %(virt_type)s / machine_type %(mach_type)s; libvirt error was: %(error)s"
12:06:26 sean-k-mooney sure we dont need to make it a question
12:06:38 sean-k-mooney since the lbvirt error states what the issue was

Earlier   Later