Posted	Nick	Remark
#openstack-nova - 2023-01-18
16:37:35	dansmith	the images client in tempest already chunks the upload, it just does it from a fixed size buffer, so it just needs to be smarter
16:37:37	bauzas	sean-k-mooney: again, we'll know how much memory creates this test with my new CI job
16:38:03	sean-k-mooney	bauzas: just got back to this point in scrolback
16:38:07	sean-k-mooney	bauzas: ack
16:38:22	dansmith	the large image was specifically to flush out things like this, so I don't think going back to a small image gets us anything useful
16:38:26	bauzas	anyway, this is a guess
16:38:38	bauzas	nothing was changed in this module since Feb 22
16:38:39	dansmith	if anything, it makes me think we can make it larger as this might have been the OOM limit I was running into with 2G
16:38:40	sean-k-mooney	dansmith: i agree
16:39:03	bauzas	https://github.com/openstack/tempest/blob/master/tempest/api/compute/admin/test_volume.py
16:39:20	sean-k-mooney	dansmith: the trade of is we only have 80G of disk space in ci
16:39:33	sean-k-mooney	so back to 2G perhaps 20G proably not
16:39:34	bauzas	oh wait
16:39:36	dansmith	I understand, disk space is not the issue though
16:39:40	bauzas	maybe we change the image ref
16:40:19	opendevreview	Kashyap Chamarthy proposed openstack/nova master: libvirt: At start-up allow skiping compareCPU() with a workaround https://review.opendev.org/c/openstack/nova/+/870794
16:41:00	kashyap	Duh, forgot to commit 2 files
16:41:20	opendevreview	Kashyap Chamarthy proposed openstack/nova master: libvirt: At start-up allow skiping compareCPU() with a workaround https://review.opendev.org/c/openstack/nova/+/870794
16:45:41	bauzas	https://github.com/openstack/devstack/blob/master/lib/tempest#L213-L220
16:45:43	bauzas	hmmmm
16:46:14	gibi	propsed the skip for this test https://review.opendev.org/c/openstack/tempest/+/870974 I checked no other test using the _create_image_with_custom_property util function
16:46:29	bauzas	2023-01-18 11:54:33.763321 | controller | ++ lib/tempest:get_active_images:155 : '[' cirros-raw = cirros-0.5.2-x86_64-disk ']'
16:46:37	bauzas	https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_362/870924/2/check/nova-ceph-multistore/3626391/job-output.txt
16:46:46	bauzas	we only get the cirros image
16:46:53	bauzas	shouldn't be that large
16:48:22	dansmith	bauzas: you understand that the ceph job uses a 1G cirros image right?
16:49:37	bauzas	Jan 18 11:57:14.947333 np0032776548 glance-api[110229]: DEBUG glance.image_cache [None req-76dfbfc9-8d31-4a47-a529-e95d8077cfc0 tempest-AttachSCSIVolumeTestJSON-1393201534 tempest-AttachSCSIVolumeTestJSON-1393201534-project-admin] Tee'ing image '0bc12eec-2802-48e8-bedf-0931be582d19' into cache {{(pid=110229) get_caching_iter /opt/stack/glance/glance/image_cache/__init__.py:343}}
16:49:49	bauzas	dansmith: oh sorry no, wasn't knowing
16:49:58	gibi	950MB image is downloaded in my case
16:50:10	dansmith	bauzas: [08:31:27] <dansmith> bauzas: I recently increased the size of the image used on the ceph job from 16MB to 1G
16:50:17	bauzas	missed that line
16:50:23	dansmith	:)
16:50:53	bauzas	ok, so we know that we cache 1GB in memory
16:50:57	gibi	te test is nice as it get the image metadata first so in the log there is a "size": 996147200 before the data is downloaded
16:51:24	bauzas	gibi: I'm waiting for my test job to return but I guess we'll see a size of 1GB in memory for that variable
16:52:29	bauzas	dansmith: about your question (why do we trigger now the kill and not earlier), my guess is that we were just below the line
16:52:34	opendevreview	Balazs Gibizer proposed openstack/nova master: DNM: Test that OOM triggering test is skipped https://review.opendev.org/c/openstack/nova/+/870950
16:53:34	dansmith	bauzas: yeah, like I said, we're probably just swapping it all and never touching it again, so pressure is high and we're close to the edge :)
16:53:45	bauzas	one way to alleviate this issue would be to make sure we run that greedy test into a specific test runner worker
16:54:06	opendevreview	Aaron S proposed openstack/nova master: Add further workaround features for qemu_monitor_announce_self https://review.opendev.org/c/openstack/nova/+/867324
16:54:10	bauzas	https://stestr.readthedocs.io/en/latest/MANUAL.html#test-scheduling
16:54:29	gibi	bauzas: we just shouldn't load the whole image data in memory at once
16:54:29	bauzas	tempest exposes the worker configs from stestr
16:54:57	gibi	as in general image size can be way bigger than memory size
16:55:10	bauzas	gibi: that's true
16:55:20	bauzas	caching the metadata seems ok to me
16:55:34	bauzas	caching the data itself seems unnecessary
16:55:44	bauzas	unless you want to compare bytes per bytes
16:55:46	gibi	yeah meatadata is bound by glance API
16:55:59	gibi	image size is unbound
16:56:05	bauzas	but agreed you could and should compare streams and not objects
16:56:15	bauzas	for the dataz
16:56:28	dansmith	it's just a naive test not thinking about the world outside a 16MB image
16:56:30	bauzas	glance team is added on the bug report
16:56:38	dansmith	anyone using this for verification of a real cloud likely has an even larger image,
16:56:41	dansmith	so it's clearly not okay to do this
16:56:45	bauzas	agreed
16:56:55	bauzas	whoami-rajat: hey, happy new year :)
16:57:03	opendevreview	Aaron S proposed openstack/nova master: Add further workaround features for qemu_monitor_announce_self https://review.opendev.org/c/openstack/nova/+/867324
16:58:53	opendevreview	Aaron S proposed openstack/nova master: Add further workaround features for qemu_monitor_announce_self https://review.opendev.org/c/openstack/nova/+/867324
16:59:49	opendevreview	Merged openstack/nova master: Strictly follow placement allocation during PCI claim https://review.opendev.org/c/openstack/nova/+/855650
17:01:08	bauzas	hmmm, stackalytics.io is fone
17:01:10	bauzas	gone*
17:01:57	gibi	what do you mean by gone?
17:02:00	gibi	it loads for me with
17:02:01	gibi	Last updated on 18 Jan 2023 12:44:21 UTC
17:02:06	bauzas	I got a timeout
17:02:16	bauzas	oh this works now
17:04:20	gibi	gmann, dansmith: was there some policy change recently that can cause that the nova functional test locally gets 403 from placement?
17:04:33	gibi	{'errors': [{'status': 403, 'title': 'Forbidden', 'detail': 'Access was denied to this resource.\n\n placement:allocations:list ', 'request_id': 'req-c5c03029-ba79-4e4a-8ec8-03deadb24ded'}]}
17:04:43	dansmith	gibi: reliably?
17:04:46	gibi	yes
17:04:51	dansmith	oof, then probably
17:04:51	gibi	this is pure nova master functional test
17:04:53	gmann	gibi: yeah we changed the default and placement fixture needed fix whihc is merged i think
17:05:00	bauzas	gibi: yeah
17:05:09	gmann	gibi: can you rebase the placement repo?
17:05:12	bauzas	gibi: we switched to new RBAC policies
17:05:28	gibi	it is just my nova repo locally
17:05:32	gmann	this one https://review.opendev.org/c/openstack/placement/+/869525
17:05:58	gmann	gibi: because nova functional test use placement fixture from placement repo
17:06:17	bauzas	which we pull as a dependency, right?
17:06:31	gmann	yes
17:06:57	gibi	openstack-placement>=1.0.0
17:06:57	gibi	so nova's tox.ini has
17:07:06	gibi	that should pull in the latest openstack-placement
17:07:12	gibi	in the functional venv
17:07:35	bauzas	tox -r ?
17:07:42	gibi	I have openstack-placement==8.0.0
17:07:44	gibi	in the venv
17:07:56	gibi	I guess we merged the fix in placement but we haven't released it
17:08:04	gmann	i do not think we released placement with that
17:08:17	gmann	yeah not released yet, we should do
17:08:17	gibi	so nova's tox.ini pulls placement from pypi
17:08:25	gibi	^^ yepp
17:08:43	gibi	or change nova's tox.ini to pull placement from github
17:08:55	gmann	yeah for now this can be workaround
17:08:57	gmann	let me push it today unless bauzas you want to do?
17:10:15	gmann	I feel placement fixture import from placement in functional test should be changed like we do for cinder/glance fixture otherwise we need new placement release for any change in there
17:11:11	bauzas	gmann: do the push and I'll +1

Earlier   Later