| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-07-25 | |||
| 11:43:08 | sean-k-mooney | so there are two workers each of which have seperate in memeory dict caches | |
| 11:43:28 | sean-k-mooney | slaweq: so this is like just because oslo cache is not configured to use memcache in that job | |
| 11:44:22 | slaweq | sean-k-mooney: so do You think that enabling memcache "globally" in all tempest based jobs would be good and could help to solve/workaround this issue? | |
| 11:44:44 | sean-k-mooney | i think it would not only solve this issue but signifcatntly speed up the gate | |
| 11:44:50 | sean-k-mooney | well maybe not | |
| 11:44:53 | slaweq | than always during /public-keys/ call we would already have cached data for instance | |
| 11:45:02 | sean-k-mooney | it is still doing an in memory dict cache | |
| 11:45:17 | sean-k-mooney | but it proably would resove this issue | |
| 11:45:28 | sean-k-mooney | yep | |
| 11:45:30 | slaweq | sean-k-mooney: ok, that's something | |
| 11:45:40 | slaweq | can You maybe write this in comment in launchpad? | |
| 11:45:51 | slaweq | I will try to propose patch to tempest repo probably | |
| 11:45:55 | slaweq | or devstack | |
| 11:47:01 | sean-k-mooney | sure. keysone is already configure to use memcache in that job | |
| 11:47:03 | sean-k-mooney | http://logs.openstack.org/35/521035/8/check/tempest-full/031b0b9/controller/logs/etc/keystone/keystone_conf.txt.gz | |
| 11:47:34 | sean-k-mooney | so it should be trival to copy the exact same caching logic to nova the config options are identical because they just come form oslo | |
| 11:47:39 | slaweq | sean-k-mooney: thx a lot | |
| 11:47:54 | slaweq | I will propose patch for this today | |
| 11:48:01 | slaweq | and we will see how it will work :) | |
| 11:48:23 | slaweq | but that is some idea which may solve this problem in gate and improve gate stability for all projects in fact :) | |
| 11:48:35 | sean-k-mooney | actully i think oslos cache's default is the null cache | |
| 11:48:49 | sean-k-mooney | so i would expect this to speed up the gate jobs | |
| 11:49:05 | sean-k-mooney | the null cache never caches anything | |
| 11:49:21 | openstackgerrit | Adam Spiers proposed openstack/nova master: libvirt: harden Host.get_domain_capabilities() https://review.opendev.org/670189 | |
| 11:49:35 | aspiers | sean-k-mooney: ^^^ fixed a few minor typos/grammar issues in the comments but there is still one issue | |
| 11:50:38 | sean-k-mooney | aspiers: :) i think there always will be | |
| 11:50:48 | sean-k-mooney | aspiers: what is the latest one? | |
| 11:50:55 | aspiers | sean-k-mooney: just posted on the review | |
| 11:54:45 | sean-k-mooney | we dont have an easy way to check the excpetion unless we pars the error message text | |
| 11:54:54 | aspiers | yes that's what I was suggesting | |
| 11:55:03 | aspiers | it's not great but better than nothing | |
| 11:55:04 | sean-k-mooney | yes and i think that is a bad idea | |
| 11:55:12 | sean-k-mooney | it makes it rather fragile | |
| 11:55:26 | aspiers | well OK then the debug message should be more honest and not assume what it doesn't know | |
| 11:55:36 | aspiers | it's fine if it says it's guessing the issue | |
| 11:55:47 | sean-k-mooney | the debug message just states that we are skiping the arch because its in compatible with the virt type and machine type | |
| 11:55:56 | sean-k-mooney | we dont state which is the in compatibality | |
| 11:56:00 | aspiers | but it doesn't know that | |
| 11:56:06 | aspiers | it could fail due to libvirtd issues | |
| 11:56:08 | sean-k-mooney | it does | |
| 11:56:18 | aspiers | e.g. libvirtd crashes 1usec beforehand | |
| 11:56:29 | aspiers | and then you get a misleading debug message | |
| 11:56:38 | sean-k-mooney | so i originally was going to not have a debug message at all | |
| 11:56:47 | sean-k-mooney | so we can delete it if we want too | |
| 11:56:47 | aspiers | no it's good to have one | |
| 11:56:54 | aspiers | it should just not risk being wrong | |
| 11:57:06 | aspiers | it's fine if it says "this is *probably* what happened" | |
| 11:57:12 | aspiers | if that's the most likely thing | |
| 11:57:16 | sean-k-mooney | then we can simply state we are skiping the arch and not say way | |
| 11:57:23 | sean-k-mooney | *why | |
| 11:57:36 | aspiers | if that's the most likely thing it's better to expose the guess | |
| 11:57:47 | aspiers | since that's potentially more helpful to the operator/dev than not guessing | |
| 11:57:52 | aspiers | as long as it's not misleading | |
| 11:57:54 | sean-k-mooney | e.g. Skipping arch: becasue libvirt raised an error, check you libvirt logs for more info | |
| 11:57:58 | aspiers | think of it from the operator perspective | |
| 11:58:08 | aspiers | no the libvirt logs might not reveal anything | |
| 11:58:11 | sean-k-mooney | aspiers: this is never ment to be read by operators | |
| 11:58:13 | aspiers | if it was incompatible | |
| 11:58:22 | sean-k-mooney | that is why its at debug level | |
| 11:58:29 | aspiers | if you don't believe operators read DEBUG you live in a different universe ... | |
| 11:58:31 | aspiers | :) | |
| 11:58:42 | sean-k-mooney | they might but this is not intened for them | |
| 11:58:49 | aspiers | anyway it doesn't matter who is reading it | |
| 11:58:58 | sean-k-mooney | are you ok with the message i suggested above | |
| 11:59:00 | aspiers | the point is that the message needs to be a) not misleading b) as helpful as possible | |
| 11:59:30 | aspiers | OK I will paste a suggested message here, 1 sec | |
| 11:59:58 | sean-k-mooney | "Skipping arch: %s becasue libvirt raised an error, check you libvirt logs for more info." | |
| 12:00:03 | aspiers | nope | |
| 12:00:11 | aspiers | like I said libvirt logs might not help | |
| 12:00:32 | aspiers | and in this case we know we might be able to help by guessing the likely cause | |
| 12:00:49 | sean-k-mooney | you said you dont want it to be missleading | |
| 12:00:57 | aspiers | yes that is a) | |
| 12:01:01 | aspiers | but also b) | |
| 12:01:28 | sean-k-mooney | the current error message is our best guess at why the error was raised | |
| 12:01:41 | aspiers | Yes but it's not honest that it's a guess | |
| 12:01:48 | aspiers | This is better: "Failed to retrieve domain caps from libvirt for arch %s; maybe incompatible with virt_type %s / machine_type %s?" | |
| 12:02:04 | sean-k-mooney | it is honest it was a summary of the error message | |
| 12:02:15 | sean-k-mooney | we could just print the error message we get back from libvirt | |
| 12:02:31 | aspiers | Yes good idea | |
| 12:02:52 | aspiers | This is better: "Failed to retrieve domain caps from libvirt for arch %(arch)s (%(error)s); maybe incompatible with virt_type %(virt_type)s / machine_type %(mach_type)s?" | |
| 12:02:55 | sean-k-mooney | i wanted to avoid the stack trace but we should be able to just get the message | |
| 12:03:09 | aspiers | that last one includes the libvirt error message ^^^ | |
| 12:03:34 | sean-k-mooney | i would not put the error in the middel | |
| 12:03:38 | sean-k-mooney | i would put it at the end | |
| 12:04:02 | sean-k-mooney | i guess its not that long | |
| 12:04:04 | sean-k-mooney | http://paste.openstack.org/show/754776/ | |
| 12:04:14 | sean-k-mooney | its invalid argument: KVM is not supported by '/usr/bin/qemu-system-alpha' on this host | |
| 12:04:38 | sean-k-mooney | at least in the case where the virt type is the issue | |
| 12:04:48 | aspiers | OK good point | |
| 12:05:19 | aspiers | "Failed to retrieve domain caps from libvirt for arch %(arch)s; maybe incompatible with virt_type %(virt_type)s / machine_type %(mach_type)s? libvirt error was: %(error)s" | |
| 12:05:34 | sean-k-mooney | ya im fine with that | |
| 12:05:37 | aspiers | or actually | |
| 12:05:47 | aspiers | since the libvirt error is already helpful enough | |
| 12:06:04 | aspiers | "Failed to retrieve domain caps from libvirt for arch %(arch)s / virt_type %(virt_type)s / machine_type %(mach_type)s; libvirt error was: %(error)s" | |
| 12:06:26 | sean-k-mooney | sure we dont need to make it a question | |
| 12:06:38 | sean-k-mooney | since the lbvirt error states what the issue was | |
| 12:07:11 | kashyap | aspiers: Can't we do a check to determine the arch to 'virt_type' compatibility? | |
| 12:07:22 | sean-k-mooney | no | |
| 12:07:24 | aspiers | sean-k-mooney: exactly, I removed the question mark | |
| 12:07:26 | sean-k-mooney | that is libvirt job | |