| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-10-30 | |||
| 22:12:28 | sean-k-mooney | ya that is much more hten i was expecting | |
| 22:13:23 | eandersson | I do wonder what kind of hardware people use for their larget clusters | |
| 22:13:28 | eandersson | for the "control plane" | |
| 22:13:48 | mriedem | are you sure it's using the cache properly? do you see "Using cached metadata for" in the metadata API debug logs? | |
| 22:14:11 | eandersson | I'll check in our lab setup sec | |
| 22:15:12 | sean-k-mooney | enabling memcache instead of the dict cache makes a huge difference so if its not useing it properly you could definetly reduce teh amount of workers as a resutl of fixing it | |
| 22:16:04 | mriedem | fwiw cern and huawei public cloud run a metadata api per cell with direct access to the cell db rather than a huge since metadata api that spans the entire deployment | |
| 22:16:16 | mriedem | s/since/single/ | |
| 22:16:42 | mriedem | https://docs.openstack.org/nova/latest/user/cellsv2-layout.html#nova-metadata-api-service | |
| 22:16:43 | sean-k-mooney | mriedem: ya that seams like a better way to approch scaling | |
| 22:16:56 | mriedem | blizzard is only 1 cell though right? | |
| 22:17:51 | sean-k-mooney | dont you have to modify neutorn to proxy to the local instance in that case too | |
| 22:17:56 | eandersson | Yea seeing | |
| 22:17:57 | eandersson | > Using cached metadata for X get_metadata_by_remote_address | |
| 22:18:06 | eandersson | Yea one cell many regions | |
| 22:19:12 | mriedem | sean-k-mooney: yes that's in that doc | |
| 22:20:05 | sean-k-mooney | woudl it make sense even in a singel cell deployment to configure different neutron metadata proxies to hit different apis directly rather then loadbalnce? | |
| 22:21:27 | sean-k-mooney | eandersson: out of interest do you knwo how long the resonce form the metadata api take when you dont get teh 502s | |
| 22:22:12 | eandersson | ~0.004 | |
| 22:23:06 | sean-k-mooney | ok that is normal. we have seen up to 10s of seconds in the gate randomly before we enabeld the memcache caching | |
| 22:23:44 | eandersson | I might have overtuned nginx tbh combined with the nature of metadata | |
| 22:24:25 | eandersson | I set worker count to auto in nginx which ends up creating processors based on the vcpu count | |
| 22:24:38 | eandersson | but uwsgi worker is set to 2/vcpu | |
| 22:25:13 | sean-k-mooney | so i honestly dont know how much load you have but i would try a smaller value like 8 | |
| 22:25:20 | sean-k-mooney | instead of 24 | |
| 22:25:32 | sean-k-mooney | i certely would not have more then 1 per hyper tread | |
| 22:25:48 | eandersson | Yea exactly got one per core now (48/2) | |
| 22:26:10 | eandersson | for nova-api at least | |
| 22:27:08 | sean-k-mooney | ya if your resonce time is in the order of 4ms i would bring that down as i siad to like 8-16 and see if that help rahter then go from 24 to 30 | |
| 22:27:23 | eandersson | We did a lot of benchmarking, but it's difficult to tell how the system performs in real enviornments | |
| 23:14:11 | sean-k-mooney | artom: i left a comment for you here https://review.opendev.org/#/c/692206/3/nova/api/openstack/compute/shelve.py | |
| 23:14:24 | sean-k-mooney | artom: i think that would also fix the issue more generally | |
| 23:14:39 | sean-k-mooney | but i have not tested it | |
| 23:15:25 | sean-k-mooney | anyway night all o/ | |
| 23:17:01 | artom | sean-k-mooney, nice idea | |
| 23:17:13 | artom | I'd kinda be afraid of any side effects | |
| 23:17:30 | artom | It'd a wide-reaching change | |
| 23:43:44 | openstackgerrit | Merged openstack/nova master: fixtures: Add support for security groups https://review.opendev.org/686802 | |
| 23:47:56 | openstackgerrit | Merged openstack/nova master: nova-net: Migrate 'test_floating_ips' functional tests https://review.opendev.org/684344 | |
| #openstack-nova - 2019-10-31 | |||
| 01:25:57 | openstackgerrit | Merged openstack/nova master: nova-net: Use deepcopy on value returned by NeutronFixture https://review.opendev.org/688139 | |
| 02:04:58 | openstackgerrit | melanie witt proposed openstack/nova master: Replace time.sleep(10) with service forced_down in tests https://review.opendev.org/692252 | |
| 02:07:43 | openstackgerrit | melanie witt proposed openstack/nova master: Replace time.sleep(10) with service forced_down in tests https://review.opendev.org/692252 | |
| 03:05:23 | openstackgerrit | Brin Zhang proposed openstack/nova-specs master: Support re-configure deleted_on_termination in server https://review.opendev.org/580336 | |
| 05:08:06 | openstackgerrit | Merged openstack/nova master: Update help for image_cache_manager_interval option https://review.opendev.org/678954 | |
| 05:08:12 | openstackgerrit | Merged openstack/nova master: Only allow one scheduler service in tests https://review.opendev.org/682486 | |
| 08:47:21 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Use API calls to force down compute in func test https://review.opendev.org/692291 | |
| 09:33:19 | openstackgerrit | Adit Sarfaty proposed openstack/nova master: Avoid fetching metadata when no subnets found https://review.opendev.org/679247 | |
| 09:53:52 | openstackgerrit | Adit Sarfaty proposed openstack/nova master: Avoid fetching metadata when no subnets found https://review.opendev.org/679247 | |
| 10:19:12 | openstackgerrit | Adit Sarfaty proposed openstack/nova master: Avoid fetching metadata when no subnets found https://review.opendev.org/679247 | |
| 10:29:04 | openstackgerrit | Adit Sarfaty proposed openstack/nova master: Avoid fetching metadata when no subnets found https://review.opendev.org/679247 | |
| 11:09:13 | openstackgerrit | Merged openstack/nova master: Add missing parameter https://review.opendev.org/682886 | |
| 11:20:58 | openstackgerrit | Brin Zhang proposed openstack/nova-specs master: Proposal for a safer noVNC console with password authentication https://review.opendev.org/623120 | |
| 11:22:55 | openstackgerrit | Brin Zhang proposed openstack/nova-specs master: Proposal for a safer noVNC console with password authentication https://review.opendev.org/623120 | |
| 13:09:03 | openstackgerrit | Merged openstack/nova stable/train: Switch to opensuse-15 nodeset https://review.opendev.org/692032 | |
| 13:13:52 | mriedem | boo | |
| 13:24:27 | mdbooth | sean-k-mooney: I was about to start reviewing https://review.opendev.org/#/c/687957/ . Any advice? | |
| 13:25:09 | sean-k-mooney | am i was going to avoid depending on your refactor to make backporting simpeler | |
| 13:25:41 | sean-k-mooney | i do think you functest refactor is good but im not sure it will be backported | |
| 13:26:16 | sean-k-mooney | as such i am wondering if i should avoid intoducing the new test class and use the old way in that patch | |
| 13:26:50 | mdbooth | sean-k-mooney: Sure. It's not strictly required either as I think I mentioned, as you can just inherit from the class I ripped everything out of. | |
| 13:27:03 | mdbooth | The additional cruft isn't actually harmful if you're not using it. | |
| 13:27:40 | sean-k-mooney | ya | |
| 13:28:54 | sean-k-mooney | anyway if you have input on how i should adress this let me know and i can adress it | |
| 13:30:31 | openstackgerrit | Fan Zhang proposed openstack/nova master: Rollback persistent domain xml after DeviceDetachFailed. https://review.opendev.org/692370 | |
| 13:39:19 | efried | mriedem: https://zuul.opendev.org/t/openstack/build/aeac2cbd2fc94751a4a5eb7e86b70981/log/logs/subnode-2/screen-n-cpu.txt.gz?severity=4 I don't see anything rbd in any of our requirements files. Does it have another name? | |
| 13:45:49 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add functional test for two-cell scheduler behaviors https://review.opendev.org/452006 | |
| 13:46:39 | mriedem | efried: librados | |
| 13:46:39 | mriedem | ? | |
| 13:47:05 | mriedem | it's not on pypi, | |
| 13:47:12 | efried | that neiter | |
| 13:47:13 | mriedem | it's a python lib built from the rbd binary | |
| 13:47:14 | efried | neither | |
| 13:47:14 | mriedem | i think | |
| 13:47:44 | efried | that error is only showing up on the "stop testing with py2" patch and the one above it, per logstash. But I don't see how that patch could have affected that job. | |
| 13:48:07 | mriedem | efried: we should do https://review.opendev.org/#/c/691765/ first then | |
| 13:48:10 | mriedem | in case it's related | |
| 13:48:38 | mriedem | though, | |
| 13:48:44 | mriedem | that's not the job that's failing in the drop py2 change right? | |
| 13:49:15 | mriedem | the nova-live-migration and nova-grenade-multinode jobs have a post-test hook script that sets up ceph | |
| 13:49:22 | efried | correct, but I'll approve that change anyway. | |
| 13:49:25 | mriedem | and those likely aren't setting USE_PYTHON3 | |
| 13:50:18 | mriedem | i can wip something up quick to try to move those to py3 | |
| 13:50:20 | efried | the other one is failing with a weird PyYAML dep | |
| 13:50:34 | efried | mriedem: stuff it in the pile underneath 'remove py2'? | |
| 13:50:38 | mriedem | yes | |
| 13:50:54 | mriedem | does this call for a "back to the pile" south park reference?! | |
| 13:51:14 | efried | It wasn't intentional. My South Park knowledge is slim | |
| 13:51:25 | mriedem | it's nsfw anyway | |
| 13:51:36 | efried | when have you ever cared about that? | |
| 13:51:48 | mriedem | ... | |
| 13:51:59 | mriedem | i've gotten better after being burned a few times over the years | |
| 13:52:10 | mriedem | despite artom still leaving traps for me | |
| 13:52:16 | efried | The PyYAML thing would be happening earlier than the rbd fail, so it's possible it would have hit the same thing later anyway. | |
| 13:52:57 | artom | mriedem, hey man, I've always had plausible deniability for everything, those traps are in your head | |
| 13:53:00 | efried | but again, I don't see how the 'stop testing py2' patch would have changed anything that would make the hook setup change. | |
| 13:53:18 | efried | the straight man in the comedy duo | |
| 13:53:59 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Convert legacy nova-live-migration and nova-multinode-grenade to py3 https://review.opendev.org/692374 | |
| 13:54:38 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Convert legacy nova-live-migration and nova-multinode-grenade to py3 https://review.opendev.org/692374 | |
| 13:54:52 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Stop testing Python 2 https://review.opendev.org/687954 | |
| 13:55:00 | mriedem | let's see how that works | |