| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-07-25 | |||
| 10:22:33 | bhagyashris | stephenfin, sean-k-mooney: ok. | |
| 10:23:57 | stephenfin | bhagyashris: On the plus side, it should be very easy reuse it if/when that spec to allow mixed instances gets merged, so that's a win :) | |
| 10:24:50 | bhagyashris | stephenfin, sean-k-mooney: means for now we will consider the cpuset only | |
| 10:24:58 | bhagyashris | stephenfin: ok | |
| 10:26:00 | bhagyashris | stephenfin: and one more point we are going to allow the new syntax of flavor extra specs like "resources:PCPU=<no of cpus>" right? | |
| 10:26:45 | stephenfin | I think we decided we would have to, yes | |
| 10:26:53 | stephenfin | Though it wouldn't be the preferred option, of course | |
| 10:27:35 | bhagyashris | stephenfin: and I saw that this is not yet implemented in your series of patches ... and I have implemented that so may be we can use that code | |
| 10:28:04 | stephenfin | bhagyashris: I'm working on rebasing your stuff into my series (keeping authorship, of course) to do just that now :) | |
| 10:30:00 | bhagyashris | stephenfin: also the upgrade and reshape part is not in your series of patches and I have submitted the patche https://review.opendev.org/#/c/672224/1 to do so . I will address all your review comments on it and will upload the patch ( this is priority work for me) | |
| 10:32:00 | bhagyashris | stephenfin: now only the part remaining is which one will be the better option this one https://review.opendev.org/#/c/672223/1 or this one https://review.opendev.org/#/c/671801/7 | |
| 10:34:08 | stephenfin | Yeah, pretty much | |
| 10:34:19 | bhagyashris | stephenfin: I guess this one will be the better option ttps://review.opendev.org/#/c/672223/1 because it's simple and also as mentioned in spec that we will used scheduler profiler https://review.opendev.org/#/c/555081/28/specs/train/approved/cpu-resources.rst@451 | |
| 10:34:23 | stephenfin | I'm still on the fence, personally | |
| 10:34:42 | stephenfin | Yeah, definitely a lot less work there | |
| 10:35:11 | stephenfin | The only thing is that rewriting extra specs on the user feels a little wrong. I'm not sure why | |
| 10:35:18 | stephenfin | Assuming these are persisted somewhere | |
| 10:36:32 | bhagyashris | stephenfin: yeah, but it looks simple that and less work and there is no overhead of conf option as well | |
| 10:37:08 | stephenfin | I think we still need the config option, no? | |
| 10:37:46 | stephenfin | Otherwise how do we say "don't start converting these extra specs yet because I don't have enough hosts reporting PCPU inventory" ? | |
| 10:39:51 | bhagyashris | stephenfin: ohh ok | |
| 10:40:19 | stephenfin | I'm not crazy, right? We do need an option for that, yeah? | |
| 10:40:20 | bhagyashris | stephenfin: will wait for others opinion then | |
| 10:45:08 | bhagyashris | stephenfin: TO DO is 1. the aliasing of flavor extra spec 2. upgrade related stuff is remaining ... out of that for first to do will wait for other opinion and I will take the upgrade part on high priority | |
| 10:45:39 | bhagyashris | stephenfin: what's your opinion? | |
| 10:49:19 | openstackgerrit | Maksim Malchuk proposed openstack/nova stable/queens: fix cellv2 delete_host https://review.opendev.org/672690 | |
| 11:01:01 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Follow-up for I2936ce8cb293dc80e1a426094fdae6e675461470 https://review.opendev.org/672669 | |
| 11:01:02 | openstackgerrit | Stephen Finucane proposed openstack/nova master: trivial: Rename exception argument https://review.opendev.org/671795 | |
| 11:01:02 | openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Start reporting PCPU inventory to placement https://review.opendev.org/671793 | |
| 11:01:03 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Remove 'hardware.get_host_numa_usage_from_instance' https://review.opendev.org/671797 | |
| 11:01:03 | openstackgerrit | Stephen Finucane proposed openstack/nova master: trivial: Remove unused function parameter https://review.opendev.org/671796 | |
| 11:01:04 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Remove 'hardware.instance_topology_from_instance' https://review.opendev.org/671799 | |
| 11:01:04 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Remove 'hardware.host_topology_and_format_from_host' https://review.opendev.org/671798 | |
| 11:01:05 | openstackgerrit | Stephen Finucane proposed openstack/nova master: tests: Split NUMA object tests https://review.opendev.org/672336 | |
| 11:01:05 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Rework 'hardware.numa_usage_from_instances' https://review.opendev.org/672565 | |
| 11:01:06 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Add support translating CPU policy extra specs, image meta https://review.opendev.org/671801 | |
| 11:01:06 | openstackgerrit | Stephen Finucane proposed openstack/nova master: WIP: hardware: Differentiate between shared and dedicated CPUs https://review.opendev.org/671800 | |
| 11:01:07 | openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: '_get_(v|p)cpu_total' to '_get_(v|p)cpu_available' https://review.opendev.org/672693 | |
| 11:08:39 | openstackgerrit | Stephen Finucane proposed openstack/nova master: docs: Integrate 'sphinx.ext.imgconverter' https://review.opendev.org/665693 | |
| 11:08:39 | openstackgerrit | Stephen Finucane proposed openstack/nova master: WIP! docs: Rework nova console diagram https://review.opendev.org/660147 | |
| 11:08:39 | openstackgerrit | Stephen Finucane proposed openstack/nova master: objects: Remove ConsoleAuthToken.to_dict https://review.opendev.org/652970 | |
| 11:20:13 | slaweq | sean-k-mooney: hi again | |
| 11:20:22 | sean-k-mooney | o/ | |
| 11:20:50 | openstack | Launchpad bug 1836642 in neutron "Metadata responses are very slow sometimes" [High,Confirmed] - Assigned to Slawek Kaplonski (slaweq) | |
| 11:20:50 | slaweq | sean-k-mooney: again about https://bugs.launchpad.net/neutron/+bug/1836642 - I replied to Your last comment | |
| 11:20:54 | slaweq | can You check it? | |
| 11:21:11 | sean-k-mooney | ya i saw | |
| 11:22:52 | sean-k-mooney | what i think happened there was the first call took 13 seconds to complete a second call was recived while the first call was completeing and that took 2 secons since the first call hand not completed | |
| 11:23:10 | slaweq | sean-k-mooney: no, first call took 2 seconds | |
| 11:23:14 | slaweq | and second took 13 | |
| 11:23:30 | slaweq | first call is always for /instance-id | |
| 11:23:44 | sean-k-mooney | ok its proably a cache miss then | |
| 11:23:52 | slaweq | and that would be retried if it would took more than 10 seconds - it's how this ec2-metadata script from cirros works | |
| 11:24:17 | slaweq | and second call (after first was completed fine) was for public-keys and this took 13 seconds | |
| 11:24:36 | slaweq | and this isn't retried by ec2-metadata script so it failed | |
| 11:25:10 | sean-k-mooney | right ok | |
| 11:26:01 | slaweq | sean-k-mooney: so I think that it's like that because each of those requests are processed by different worker thus for each worker data isn't cached | |
| 11:26:24 | slaweq | sean-k-mooney: could it be like that? | |
| 11:26:57 | sean-k-mooney | i think we are using memcahce to chage it but if its in porcess yes | |
| 11:27:08 | sean-k-mooney | if its memcache it still takes a while to propagate | |
| 11:27:36 | sean-k-mooney | those request are ~500 ms apart | |
| 11:27:52 | slaweq | can You maybe check it to be sure? | |
| 11:28:03 | slaweq | 500ms when? usually when it's fine, right? | |
| 11:28:36 | sean-k-mooney | _Jul_09_17_23_47_872908 - _Jul_09_17_23_47_351149 ~ 500ms | |
| 11:29:24 | sean-k-mooney | actully i gues those are the log message for the completions? | |
| 11:29:30 | slaweq | yes | |
| 11:29:36 | slaweq | and there is time in each line | |
| 11:29:44 | slaweq | "time: 13.2546282" | |
| 11:29:59 | slaweq | this is how long this took to process this request and send response | |
| 11:30:43 | sean-k-mooney | anyway ill check | |
| 11:30:49 | slaweq | sean-k-mooney: thx | |
| 11:36:32 | openstackgerrit | Merged openstack/nova master: Remove deprecated CPU, RAM, disk claiming in resource tracker https://review.opendev.org/551026 | |
| 11:36:43 | openstackgerrit | Merged openstack/nova master: Pass extra_specs to flavor in vif tests https://review.opendev.org/662556 | |
| 11:36:50 | sean-k-mooney | so it checks the cache here https://opendev.org/openstack/nova/src/branch/master/nova/api/metadata/handler.py#L77 | |
| 11:37:24 | sean-k-mooney | which is initalised here https://opendev.org/openstack/nova/src/branch/master/nova/api/metadata/handler.py#L47 | |
| 11:37:36 | sean-k-mooney | i think this delegates to oslo cache | |
| 11:38:22 | sean-k-mooney | so we call this https://github.com/openstack/nova/blob/master/nova/cache_utils.py#L47 | |
| 11:40:24 | sean-k-mooney | there is no cache section defiend in the nova.conf http://logs.openstack.org/35/521035/8/check/tempest-full/031b0b9/controller/logs/etc/nova/nova_conf.txt.gz | |
| 11:41:30 | sean-k-mooney | meaning we fall back to olso_caches default with is an in memeory dogpile dict cache | |
| 11:42:48 | sean-k-mooney | slaweq: so looking at the metadata apis uwsgi config we are runnint 1 thread per process and 2 processes | |
| 11:43:08 | sean-k-mooney | so there are two workers each of which have seperate in memeory dict caches | |
| 11:43:28 | sean-k-mooney | slaweq: so this is like just because oslo cache is not configured to use memcache in that job | |
| 11:44:22 | slaweq | sean-k-mooney: so do You think that enabling memcache "globally" in all tempest based jobs would be good and could help to solve/workaround this issue? | |
| 11:44:44 | sean-k-mooney | i think it would not only solve this issue but signifcatntly speed up the gate | |
| 11:44:50 | sean-k-mooney | well maybe not | |
| 11:44:53 | slaweq | than always during /public-keys/ call we would already have cached data for instance | |
| 11:45:02 | sean-k-mooney | it is still doing an in memory dict cache | |
| 11:45:17 | sean-k-mooney | but it proably would resove this issue | |
| 11:45:28 | sean-k-mooney | yep | |
| 11:45:30 | slaweq | sean-k-mooney: ok, that's something | |
| 11:45:40 | slaweq | can You maybe write this in comment in launchpad? | |
| 11:45:51 | slaweq | I will try to propose patch to tempest repo probably | |
| 11:45:55 | slaweq | or devstack | |
| 11:47:01 | sean-k-mooney | sure. keysone is already configure to use memcache in that job | |
| 11:47:03 | sean-k-mooney | http://logs.openstack.org/35/521035/8/check/tempest-full/031b0b9/controller/logs/etc/keystone/keystone_conf.txt.gz | |
| 11:47:34 | sean-k-mooney | so it should be trival to copy the exact same caching logic to nova the config options are identical because they just come form oslo | |
| 11:47:39 | slaweq | sean-k-mooney: thx a lot | |
| 11:47:54 | slaweq | I will propose patch for this today | |
| 11:48:01 | slaweq | and we will see how it will work :) | |
| 11:48:23 | slaweq | but that is some idea which may solve this problem in gate and improve gate stability for all projects in fact :) | |