| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-07-25 | |||
| 10:37:46 | stephenfin | Otherwise how do we say "don't start converting these extra specs yet because I don't have enough hosts reporting PCPU inventory" ? | |
| 10:39:51 | bhagyashris | stephenfin: ohh ok | |
| 10:40:19 | stephenfin | I'm not crazy, right? We do need an option for that, yeah? | |
| 10:40:20 | bhagyashris | stephenfin: will wait for others opinion then | |
| 10:45:08 | bhagyashris | stephenfin: TO DO is 1. the aliasing of flavor extra spec 2. upgrade related stuff is remaining ... out of that for first to do will wait for other opinion and I will take the upgrade part on high priority | |
| 10:45:39 | bhagyashris | stephenfin: what's your opinion? | |
| 10:49:19 | openstackgerrit | Maksim Malchuk proposed openstack/nova stable/queens: fix cellv2 delete_host https://review.opendev.org/672690 | |
| 11:01:01 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Follow-up for I2936ce8cb293dc80e1a426094fdae6e675461470 https://review.opendev.org/672669 | |
| 11:01:02 | openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Start reporting PCPU inventory to placement https://review.opendev.org/671793 | |
| 11:01:02 | openstackgerrit | Stephen Finucane proposed openstack/nova master: trivial: Rename exception argument https://review.opendev.org/671795 | |
| 11:01:03 | openstackgerrit | Stephen Finucane proposed openstack/nova master: trivial: Remove unused function parameter https://review.opendev.org/671796 | |
| 11:01:03 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Remove 'hardware.get_host_numa_usage_from_instance' https://review.opendev.org/671797 | |
| 11:01:04 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Remove 'hardware.host_topology_and_format_from_host' https://review.opendev.org/671798 | |
| 11:01:04 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Remove 'hardware.instance_topology_from_instance' https://review.opendev.org/671799 | |
| 11:01:05 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Rework 'hardware.numa_usage_from_instances' https://review.opendev.org/672565 | |
| 11:01:05 | openstackgerrit | Stephen Finucane proposed openstack/nova master: tests: Split NUMA object tests https://review.opendev.org/672336 | |
| 11:01:06 | openstackgerrit | Stephen Finucane proposed openstack/nova master: WIP: hardware: Differentiate between shared and dedicated CPUs https://review.opendev.org/671800 | |
| 11:01:06 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Add support translating CPU policy extra specs, image meta https://review.opendev.org/671801 | |
| 11:01:07 | openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: '_get_(v|p)cpu_total' to '_get_(v|p)cpu_available' https://review.opendev.org/672693 | |
| 11:08:39 | openstackgerrit | Stephen Finucane proposed openstack/nova master: objects: Remove ConsoleAuthToken.to_dict https://review.opendev.org/652970 | |
| 11:08:39 | openstackgerrit | Stephen Finucane proposed openstack/nova master: WIP! docs: Rework nova console diagram https://review.opendev.org/660147 | |
| 11:08:39 | openstackgerrit | Stephen Finucane proposed openstack/nova master: docs: Integrate 'sphinx.ext.imgconverter' https://review.opendev.org/665693 | |
| 11:20:13 | slaweq | sean-k-mooney: hi again | |
| 11:20:22 | sean-k-mooney | o/ | |
| 11:20:50 | slaweq | sean-k-mooney: again about https://bugs.launchpad.net/neutron/+bug/1836642 - I replied to Your last comment | |
| 11:20:50 | openstack | Launchpad bug 1836642 in neutron "Metadata responses are very slow sometimes" [High,Confirmed] - Assigned to Slawek Kaplonski (slaweq) | |
| 11:20:54 | slaweq | can You check it? | |
| 11:21:11 | sean-k-mooney | ya i saw | |
| 11:22:52 | sean-k-mooney | what i think happened there was the first call took 13 seconds to complete a second call was recived while the first call was completeing and that took 2 secons since the first call hand not completed | |
| 11:23:10 | slaweq | sean-k-mooney: no, first call took 2 seconds | |
| 11:23:14 | slaweq | and second took 13 | |
| 11:23:30 | slaweq | first call is always for /instance-id | |
| 11:23:44 | sean-k-mooney | ok its proably a cache miss then | |
| 11:23:52 | slaweq | and that would be retried if it would took more than 10 seconds - it's how this ec2-metadata script from cirros works | |
| 11:24:17 | slaweq | and second call (after first was completed fine) was for public-keys and this took 13 seconds | |
| 11:24:36 | slaweq | and this isn't retried by ec2-metadata script so it failed | |
| 11:25:10 | sean-k-mooney | right ok | |
| 11:26:01 | slaweq | sean-k-mooney: so I think that it's like that because each of those requests are processed by different worker thus for each worker data isn't cached | |
| 11:26:24 | slaweq | sean-k-mooney: could it be like that? | |
| 11:26:57 | sean-k-mooney | i think we are using memcahce to chage it but if its in porcess yes | |
| 11:27:08 | sean-k-mooney | if its memcache it still takes a while to propagate | |
| 11:27:36 | sean-k-mooney | those request are ~500 ms apart | |
| 11:27:52 | slaweq | can You maybe check it to be sure? | |
| 11:28:03 | slaweq | 500ms when? usually when it's fine, right? | |
| 11:28:36 | sean-k-mooney | _Jul_09_17_23_47_872908 - _Jul_09_17_23_47_351149 ~ 500ms | |
| 11:29:24 | sean-k-mooney | actully i gues those are the log message for the completions? | |
| 11:29:30 | slaweq | yes | |
| 11:29:36 | slaweq | and there is time in each line | |
| 11:29:44 | slaweq | "time: 13.2546282" | |
| 11:29:59 | slaweq | this is how long this took to process this request and send response | |
| 11:30:43 | sean-k-mooney | anyway ill check | |
| 11:30:49 | slaweq | sean-k-mooney: thx | |
| 11:36:32 | openstackgerrit | Merged openstack/nova master: Remove deprecated CPU, RAM, disk claiming in resource tracker https://review.opendev.org/551026 | |
| 11:36:43 | openstackgerrit | Merged openstack/nova master: Pass extra_specs to flavor in vif tests https://review.opendev.org/662556 | |
| 11:36:50 | sean-k-mooney | so it checks the cache here https://opendev.org/openstack/nova/src/branch/master/nova/api/metadata/handler.py#L77 | |
| 11:37:24 | sean-k-mooney | which is initalised here https://opendev.org/openstack/nova/src/branch/master/nova/api/metadata/handler.py#L47 | |
| 11:37:36 | sean-k-mooney | i think this delegates to oslo cache | |
| 11:38:22 | sean-k-mooney | so we call this https://github.com/openstack/nova/blob/master/nova/cache_utils.py#L47 | |
| 11:40:24 | sean-k-mooney | there is no cache section defiend in the nova.conf http://logs.openstack.org/35/521035/8/check/tempest-full/031b0b9/controller/logs/etc/nova/nova_conf.txt.gz | |
| 11:41:30 | sean-k-mooney | meaning we fall back to olso_caches default with is an in memeory dogpile dict cache | |
| 11:42:48 | sean-k-mooney | slaweq: so looking at the metadata apis uwsgi config we are runnint 1 thread per process and 2 processes | |
| 11:43:08 | sean-k-mooney | so there are two workers each of which have seperate in memeory dict caches | |
| 11:43:28 | sean-k-mooney | slaweq: so this is like just because oslo cache is not configured to use memcache in that job | |
| 11:44:22 | slaweq | sean-k-mooney: so do You think that enabling memcache "globally" in all tempest based jobs would be good and could help to solve/workaround this issue? | |
| 11:44:44 | sean-k-mooney | i think it would not only solve this issue but signifcatntly speed up the gate | |
| 11:44:50 | sean-k-mooney | well maybe not | |
| 11:44:53 | slaweq | than always during /public-keys/ call we would already have cached data for instance | |
| 11:45:02 | sean-k-mooney | it is still doing an in memory dict cache | |
| 11:45:17 | sean-k-mooney | but it proably would resove this issue | |
| 11:45:28 | sean-k-mooney | yep | |
| 11:45:30 | slaweq | sean-k-mooney: ok, that's something | |
| 11:45:40 | slaweq | can You maybe write this in comment in launchpad? | |
| 11:45:51 | slaweq | I will try to propose patch to tempest repo probably | |
| 11:45:55 | slaweq | or devstack | |
| 11:47:01 | sean-k-mooney | sure. keysone is already configure to use memcache in that job | |
| 11:47:03 | sean-k-mooney | http://logs.openstack.org/35/521035/8/check/tempest-full/031b0b9/controller/logs/etc/keystone/keystone_conf.txt.gz | |
| 11:47:34 | sean-k-mooney | so it should be trival to copy the exact same caching logic to nova the config options are identical because they just come form oslo | |
| 11:47:39 | slaweq | sean-k-mooney: thx a lot | |
| 11:47:54 | slaweq | I will propose patch for this today | |
| 11:48:01 | slaweq | and we will see how it will work :) | |
| 11:48:23 | slaweq | but that is some idea which may solve this problem in gate and improve gate stability for all projects in fact :) | |
| 11:48:35 | sean-k-mooney | actully i think oslos cache's default is the null cache | |
| 11:48:49 | sean-k-mooney | so i would expect this to speed up the gate jobs | |
| 11:49:05 | sean-k-mooney | the null cache never caches anything | |
| 11:49:21 | openstackgerrit | Adam Spiers proposed openstack/nova master: libvirt: harden Host.get_domain_capabilities() https://review.opendev.org/670189 | |
| 11:49:35 | aspiers | sean-k-mooney: ^^^ fixed a few minor typos/grammar issues in the comments but there is still one issue | |
| 11:50:38 | sean-k-mooney | aspiers: :) i think there always will be | |
| 11:50:48 | sean-k-mooney | aspiers: what is the latest one? | |
| 11:50:55 | aspiers | sean-k-mooney: just posted on the review | |
| 11:54:45 | sean-k-mooney | we dont have an easy way to check the excpetion unless we pars the error message text | |
| 11:54:54 | aspiers | yes that's what I was suggesting | |
| 11:55:03 | aspiers | it's not great but better than nothing | |
| 11:55:04 | sean-k-mooney | yes and i think that is a bad idea | |
| 11:55:12 | sean-k-mooney | it makes it rather fragile | |
| 11:55:26 | aspiers | well OK then the debug message should be more honest and not assume what it doesn't know | |
| 11:55:36 | aspiers | it's fine if it says it's guessing the issue | |
| 11:55:47 | sean-k-mooney | the debug message just states that we are skiping the arch because its in compatible with the virt type and machine type | |
| 11:55:56 | sean-k-mooney | we dont state which is the in compatibality | |
| 11:56:00 | aspiers | but it doesn't know that | |
| 11:56:06 | aspiers | it could fail due to libvirtd issues | |