Posted	Nick	Remark
#openstack-nova - 2019-10-30
22:16:42	mriedem	https://docs.openstack.org/nova/latest/user/cellsv2-layout.html#nova-metadata-api-service
22:16:43	sean-k-mooney	mriedem: ya that seams like a better way to approch scaling
22:16:56	mriedem	blizzard is only 1 cell though right?
22:17:51	sean-k-mooney	dont you have to modify neutorn to proxy to the local instance in that case too
22:17:56	eandersson	Yea seeing
22:17:57	eandersson	> Using cached metadata for X get_metadata_by_remote_address
22:18:06	eandersson	Yea one cell many regions
22:19:12	mriedem	sean-k-mooney: yes that's in that doc
22:20:05	sean-k-mooney	woudl it make sense even in a singel cell deployment to configure different neutron metadata proxies to hit different apis directly rather then loadbalnce?
22:21:27	sean-k-mooney	eandersson: out of interest do you knwo how long the resonce form the metadata api take when you dont get teh 502s
22:22:12	eandersson	~0.004
22:23:06	sean-k-mooney	ok that is normal. we have seen up to 10s of seconds in the gate randomly before we enabeld the memcache caching
22:23:44	eandersson	I might have overtuned nginx tbh combined with the nature of metadata
22:24:25	eandersson	I set worker count to auto in nginx which ends up creating processors based on the vcpu count
22:24:38	eandersson	but uwsgi worker is set to 2/vcpu
22:25:13	sean-k-mooney	so i honestly dont know how much load you have but i would try a smaller value like 8
22:25:20	sean-k-mooney	instead of 24
22:25:32	sean-k-mooney	i certely would not have more then 1 per hyper tread
22:25:48	eandersson	Yea exactly got one per core now (48/2)
22:26:10	eandersson	for nova-api at least
22:27:08	sean-k-mooney	ya if your resonce time is in the order of 4ms i would bring that down as i siad to like 8-16 and see if that help rahter then go from 24 to 30
22:27:23	eandersson	We did a lot of benchmarking, but it's difficult to tell how the system performs in real enviornments
23:14:11	sean-k-mooney	artom: i left a comment for you here https://review.opendev.org/#/c/692206/3/nova/api/openstack/compute/shelve.py
23:14:24	sean-k-mooney	artom: i think that would also fix the issue more generally
23:14:39	sean-k-mooney	but i have not tested it
23:15:25	sean-k-mooney	anyway night all o/
23:17:01	artom	sean-k-mooney, nice idea
23:17:13	artom	I'd kinda be afraid of any side effects
23:17:30	artom	It'd a wide-reaching change
23:43:44	openstackgerrit	Merged openstack/nova master: fixtures: Add support for security groups https://review.opendev.org/686802
23:47:56	openstackgerrit	Merged openstack/nova master: nova-net: Migrate 'test_floating_ips' functional tests https://review.opendev.org/684344
#openstack-nova - 2019-10-31
01:25:57	openstackgerrit	Merged openstack/nova master: nova-net: Use deepcopy on value returned by NeutronFixture https://review.opendev.org/688139
02:04:58	openstackgerrit	melanie witt proposed openstack/nova master: Replace time.sleep(10) with service forced_down in tests https://review.opendev.org/692252
02:07:43	openstackgerrit	melanie witt proposed openstack/nova master: Replace time.sleep(10) with service forced_down in tests https://review.opendev.org/692252
03:05:23	openstackgerrit	Brin Zhang proposed openstack/nova-specs master: Support re-configure deleted_on_termination in server https://review.opendev.org/580336
05:08:06	openstackgerrit	Merged openstack/nova master: Update help for image_cache_manager_interval option https://review.opendev.org/678954
05:08:12	openstackgerrit	Merged openstack/nova master: Only allow one scheduler service in tests https://review.opendev.org/682486
08:47:21	openstackgerrit	Balazs Gibizer proposed openstack/nova master: Use API calls to force down compute in func test https://review.opendev.org/692291
09:33:19	openstackgerrit	Adit Sarfaty proposed openstack/nova master: Avoid fetching metadata when no subnets found https://review.opendev.org/679247
09:53:52	openstackgerrit	Adit Sarfaty proposed openstack/nova master: Avoid fetching metadata when no subnets found https://review.opendev.org/679247
10:19:12	openstackgerrit	Adit Sarfaty proposed openstack/nova master: Avoid fetching metadata when no subnets found https://review.opendev.org/679247
10:29:04	openstackgerrit	Adit Sarfaty proposed openstack/nova master: Avoid fetching metadata when no subnets found https://review.opendev.org/679247
11:09:13	openstackgerrit	Merged openstack/nova master: Add missing parameter https://review.opendev.org/682886
11:20:58	openstackgerrit	Brin Zhang proposed openstack/nova-specs master: Proposal for a safer noVNC console with password authentication https://review.opendev.org/623120
11:22:55	openstackgerrit	Brin Zhang proposed openstack/nova-specs master: Proposal for a safer noVNC console with password authentication https://review.opendev.org/623120
13:09:03	openstackgerrit	Merged openstack/nova stable/train: Switch to opensuse-15 nodeset https://review.opendev.org/692032
13:13:52	mriedem	boo
13:24:27	mdbooth	sean-k-mooney: I was about to start reviewing https://review.opendev.org/#/c/687957/ . Any advice?
13:25:09	sean-k-mooney	am i was going to avoid depending on your refactor to make backporting simpeler
13:25:41	sean-k-mooney	i do think you functest refactor is good but im not sure it will be backported
13:26:16	sean-k-mooney	as such i am wondering if i should avoid intoducing the new test class and use the old way in that patch
13:26:50	mdbooth	sean-k-mooney: Sure. It's not strictly required either as I think I mentioned, as you can just inherit from the class I ripped everything out of.
13:27:03	mdbooth	The additional cruft isn't actually harmful if you're not using it.
13:27:40	sean-k-mooney	ya
13:28:54	sean-k-mooney	anyway if you have input on how i should adress this let me know and i can adress it
13:30:31	openstackgerrit	Fan Zhang proposed openstack/nova master: Rollback persistent domain xml after DeviceDetachFailed. https://review.opendev.org/692370
13:39:19	efried	mriedem: https://zuul.opendev.org/t/openstack/build/aeac2cbd2fc94751a4a5eb7e86b70981/log/logs/subnode-2/screen-n-cpu.txt.gz?severity=4 I don't see anything rbd in any of our requirements files. Does it have another name?
13:45:49	openstackgerrit	Matt Riedemann proposed openstack/nova master: Add functional test for two-cell scheduler behaviors https://review.opendev.org/452006
13:46:39	mriedem	?
13:46:39	mriedem	efried: librados
13:47:05	mriedem	it's not on pypi,
13:47:12	efried	that neiter
13:47:13	mriedem	it's a python lib built from the rbd binary
13:47:14	mriedem	i think
13:47:14	efried	neither
13:47:44	efried	that error is only showing up on the "stop testing with py2" patch and the one above it, per logstash. But I don't see how that patch could have affected that job.
13:48:07	mriedem	efried: we should do https://review.opendev.org/#/c/691765/ first then
13:48:10	mriedem	in case it's related
13:48:38	mriedem	though,
13:48:44	mriedem	that's not the job that's failing in the drop py2 change right?
13:49:15	mriedem	the nova-live-migration and nova-grenade-multinode jobs have a post-test hook script that sets up ceph
13:49:22	efried	correct, but I'll approve that change anyway.
13:49:25	mriedem	and those likely aren't setting USE_PYTHON3
13:50:18	mriedem	i can wip something up quick to try to move those to py3
13:50:20	efried	the other one is failing with a weird PyYAML dep
13:50:34	efried	mriedem: stuff it in the pile underneath 'remove py2'?
13:50:38	mriedem	yes
13:50:54	mriedem	does this call for a "back to the pile" south park reference?!
13:51:14	efried	It wasn't intentional. My South Park knowledge is slim
13:51:25	mriedem	it's nsfw anyway
13:51:36	efried	when have you ever cared about that?
13:51:48	mriedem	...
13:51:59	mriedem	i've gotten better after being burned a few times over the years
13:52:10	mriedem	despite artom still leaving traps for me
13:52:16	efried	The PyYAML thing would be happening earlier than the rbd fail, so it's possible it would have hit the same thing later anyway.
13:52:57	artom	mriedem, hey man, I've always had plausible deniability for everything, those traps are in your head
13:53:00	efried	but again, I don't see how the 'stop testing py2' patch would have changed anything that would make the hook setup change.
13:53:18	efried	the straight man in the comedy duo
13:53:59	openstackgerrit	Matt Riedemann proposed openstack/nova master: Convert legacy nova-live-migration and nova-multinode-grenade to py3 https://review.opendev.org/692374
13:54:38	openstackgerrit	Matt Riedemann proposed openstack/nova master: Convert legacy nova-live-migration and nova-multinode-grenade to py3 https://review.opendev.org/692374
13:54:52	openstackgerrit	Matt Riedemann proposed openstack/nova master: Stop testing Python 2 https://review.opendev.org/687954
13:55:00	mriedem	let's see how that works
13:55:11	mriedem	mels, then mine, then stephens
13:55:20	efried	zuul kicked it
13:55:27	efried	nm
13:55:32	efried	nova meeting in five minutes
13:57:31	sean-k-mooney	efried: that reminds me i need to recheck your os-vif job change.
13:57:40	sean-k-mooney	i think the devstack patch should now be merged
13:58:51	sean-k-mooney	actully no it faild to merge and is being rechecked so ill wait
14:00:17	efried	thanks for keeping track of that sean-k-mooney

Earlier   Later