Posted	Nick	Remark
#openstack-nova - 2019-02-18
18:00:13	zzzeek	jaypipes: I know, neutron is a PITA
18:00:26	zzzeek	jaypipes: i never work on it because it is nonsensical
18:00:35	zzzeek	jaypipes: but i figured out enough what this does :)
18:00:37	openstackgerrit	Elod Illes proposed openstack/nova stable/queens: Make host_manager use scatter-gather and ignore down cells https://review.openstack.org/637599
18:02:46	jaypipes	zzzeek: this patch definitely fixes a number of deadlocks you are seeing, yes?
18:03:14	jaypipes	zzzeek: where specifically are those deadlocks being hit? (in the neutron code, that is...)
18:04:06	zzzeek	jaypipes: it fixes SQLAlchemy version id misses, not "deadlocks" in the literal sense
18:04:16	zzzeek	jaypipes: SQLAlchemy StaleDataErrors
18:04:55	zzzeek	jaypipes: they have this test case where they create 300 security group rules
18:05:15	zzzeek	jaypipes: and that incurs a versino bump on many related objects because their API requiers that the new number is exposed
18:05:25	zzzeek	jaypipes: so the version_id here is used for two purposes
18:05:57	zzzeek	jaypipes: so when they make these 300 rules they get a ton of StaleDataErrors on all the related objects, for which they are changing nothing except bumping the version id
18:06:10	jaypipes	zzzeek: wait, are you saying that updates of dependent objects cause the version to be incremented more than once?
18:06:14	zzzeek	jaypipes: this also incurs that an ORM lookup is happening, a flush, etc. which is all overhead too
18:06:31	zzzeek	jaypipes: no, when they have multiple transactions on concurrent API requests
18:07:20	zzzeek	script I used to create 300 security group rules asynchronously http://paste.openstack.org/show/744728/"
18:07:20	zzzeek	jaypipes: look at feb 8 comments on the gerrit: "POST request response times for 300 security group rules creation http://paste.openstack.org/show/744727/
18:08:26	jaypipes	zzzeek: ok, creating 300 security group rules should increment the security group's version to 301, yes?
18:08:48	zzzeek	jaypipes: i dunno. the person testing is saying he is getting the correct results though
18:08:48	aspiers	Wow, plantuml is pretty cool. I was able to knock this up pretty quickly from a small text file: https://goo.gl/tPNAQK
18:09:04	zzzeek	yes 301
18:09:20	zzzeek	-----------------+ | 2151 | securitygroups | 2019-02-08 09:07:15 | 2019-02-08 09:09:05 | | 301 | +------+----------------+---------------------+---------------------+-------------+-----------------+
18:09:20	zzzeek	Even the revision_number for security group updated properly when creating 300 security group rules MariaDB [neutron]> select * from standardattributes where id="2151"; +------+----------------+---------------------+---------------------+-------------+-----------------+ | id | resource_type | created_at | updated_at | description | revision_number | +------+----------------+---------------------+---------------------+-------------+
18:42:42	mnaser	mriedem: we might want to ping the ML about the change we recently did with disallowing root_gb=0 flavors
18:43:33	mnaser	osa gates broke because i assume we created flavors which were using root_gb=0
18:43:48	mriedem	we?!?!
18:43:50	mnaser	i can shoot an email if you weren't planning to
18:43:55	mriedem	go nuts
18:44:04	mnaser	https://review.openstack.org/#/c/603910/
18:44:04	mriedem	i send enough "oh btw everything is going to be broken now" emails
18:44:08	mnaser	:D
18:44:09	mriedem	yeah i know
18:59:57	openstackgerrit	Surya Seetharaman proposed openstack/python-novaclient master: API microversion 2.69: Handles Down Cells https://review.openstack.org/579563
19:03:48	openstackgerrit	Matt Riedemann proposed openstack/nova master: Add TargetDBSetupTask https://review.openstack.org/627892
19:03:49	openstackgerrit	Matt Riedemann proposed openstack/nova master: Execute TargetDBSetupTask https://review.openstack.org/633853
19:03:49	openstackgerrit	Matt Riedemann proposed openstack/nova master: Add CrossCellMigrationTask https://review.openstack.org/631581
19:03:50	openstackgerrit	Matt Riedemann proposed openstack/nova master: WIP: Add prep_snapshot_based_resize_at_dest compute method https://review.openstack.org/633293
19:03:50	openstackgerrit	Matt Riedemann proposed openstack/nova master: Add can_connect_volume() compute driver method https://review.openstack.org/621313
19:03:51	openstackgerrit	Matt Riedemann proposed openstack/nova master: Add prep_snapshot_based_resize_at_source compute method https://review.openstack.org/634832
19:03:51	openstackgerrit	Matt Riedemann proposed openstack/nova master: WIP: Add PrepResizeAtDestTask https://review.openstack.org/627890
19:03:52	openstackgerrit	Matt Riedemann proposed openstack/nova master: WIP: Add finish_snapshot_based_resize_at_dest compute method https://review.openstack.org/635080
19:03:52	openstackgerrit	Matt Riedemann proposed openstack/nova master: Add PrepResizeAtSourceTask https://review.openstack.org/627891
19:03:53	openstackgerrit	Matt Riedemann proposed openstack/nova master: Execute CrossCellMigrationTask from MigrationTask https://review.openstack.org/635668
19:03:53	openstackgerrit	Matt Riedemann proposed openstack/nova master: Add FinishResizeAtDestTask https://review.openstack.org/635646
19:03:54	openstackgerrit	Matt Riedemann proposed openstack/nova master: WIP: Filter duplicates from compute API get_migrations_sorted() https://review.openstack.org/636224
19:03:54	openstackgerrit	Matt Riedemann proposed openstack/nova master: Plumb allow_cross_cell_resize into compute API resize() https://review.openstack.org/635684
19:03:55	openstackgerrit	Matt Riedemann proposed openstack/nova master: WIP: Add confirm_snapshot_based_resize_at_source https://review.openstack.org/637058
19:03:55	openstackgerrit	Matt Riedemann proposed openstack/nova master: Start functional testing for cross-cell resize https://review.openstack.org/636253
19:03:56	openstackgerrit	Matt Riedemann proposed openstack/nova master: WIP: Add confirm_snapshot_based_resize conductor RPC method https://review.openstack.org/637075
19:03:56	openstackgerrit	Matt Riedemann proposed openstack/nova master: WIP: Add ConfirmResizeTask https://review.openstack.org/637070
19:03:57	openstackgerrit	Matt Riedemann proposed openstack/nova master: Confirm cross-cell resize from the API https://review.openstack.org/637316
19:05:04	mriedem	that's only about half the series at this point
19:05:06	mriedem	swoon away
19:36:25	NewBruce	sean-k-mooney still here mate?
19:41:25	mordred	mriedem: did nova/devstack introduce a behavior change recentlyish? http://logs.openstack.org/63/637563/1/check/openstacksdk-functional-devstack-tips/f7e749c/testr_results.html.gz
19:41:45	mordred	mriedem:
19:41:47	mordred	openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://213.32.77.235/compute/v2.1/servers, Only volume-backed servers are allowed for flavors with zero disk.
19:41:54	dansmith	hah
19:42:02	dansmith	mordred: [10:42:42] <mnaser>mriedem: we might want to ping the ML about the change we recently did with disallowing root_gb=0 flavors
19:42:08	mordred	hahaha
19:42:23	dansmith	made it a whole hour
19:42:31	mnaser	mordred: http://lists.openstack.org/pipermail/openstack-discuss/2019-February/002842.html :D
19:43:36	mordred	mnaser: so - I might need to port that change to apply to non-tempest flavors I suppose?
19:44:01	mnaser	mordred: so whatever tests you were doing were trying to do a boot from volume using a flavor that has root_gb=0
19:44:14	mnaser	i assume openstacksdk maybe is trying to create its own flavor?
19:44:34	mordred	not to my knowledge, no
19:44:41	mordred	we just use whatever flavor is there
19:44:52	mordred	but we don't install/run tempest, so I doubt lib/tempest flavors exist
19:45:44	mnaser	{"href": "https://213.32.77.235/compute/flavors/c1", "rel": "bookmark"}], "ram": 256, "OS-FLV-DISABLED:disabled": false, "os-flavor-access:is_public": true, "rxtx_factor": 1.0, "disk": 0, "id": "c1", "name": "cirros256", "vcpus": 1, "swap": "", "OS-FLV-EXT-DATA:ephemeral": 0}
19:45:55	mordred	mnaser: http://paste.openstack.org/raw/745281/
19:46:25	mordred	mnaser: so I guess I just need to pick a different flavor
19:46:38	mnaser	we should probably fix what creates cirros256
19:47:15	mnaser	did taht fail in nodepool, mordred ?
19:47:32	mnaser	no openstacksdk, hmm weird
19:47:57	mordred	mnaser: we pick the smallest flavor sorted by ram
19:48:05	mnaser	*something* creates that flavor..
19:48:14	mordred	mnaser: so I think I need to add an 'if' for disk != 0
19:48:41	mnaser	yeah if you're not doing bfv and the flavor has disk == 0 then it should not use that flavor
19:49:21	mnaser	but http://git.openstack.org/cgit/openstack-dev/devstack/tree/lib/nova#n1165
19:49:22	cdent	devstack creates that flavor
19:49:24	cdent	jinx
19:49:56	mnaser	mordred: so i guess in openstacksdk world of things, it shouldn't pick a flavor that has disk_gb=0 when trying to bfv (as of train)
19:52:07	mordred	mnaser: https://review.openstack.org/637618 Make sure we pick flavors with disk <-- I think that should do it yeah?
19:53:23	mnaser	mordred: assuming python evaluates flavor.disk set to 0 as false then yes
19:53:31	mordred	mnaser: we'll find out! :)
19:53:35	mnaser	but i forget fundamental python rules
19:53:57	mnaser	we're supposed to be experts but half the tabs i open are "how the f does X language do switch cases"
20:36:36	openstackgerrit	Theodoros Tsioutsias proposed openstack/nova master: Introduce the PENDING instance state https://review.openstack.org/566473
20:36:37	openstackgerrit	Theodoros Tsioutsias proposed openstack/nova master: Allow rebuild for instances in PENDING state https://review.openstack.org/637585
20:37:55	openstackgerrit	Vlad Gusev proposed openstack/nova master: [WIP] libvirt: Add a default Virtio RNG device to guests https://review.openstack.org/616603
20:38:16	openstackgerrit	Vlad Gusev proposed openstack/nova master: libvirt: Add a default Virtio RNG device to guests https://review.openstack.org/616603
20:57:13	mriedem	weird that we send the resize.revert.(start|end) notifications from the source host during finish_revert_resize which is half of the actual revert
20:57:22	mriedem	the start notification should happen on the dest host in revert_resize()
20:57:25	mriedem	must not be a big deal
21:03:11	openstackgerrit	Matt Riedemann proposed openstack/nova master: WIP: Add revert_snapshot_based_resize_at_dest compute method https://review.openstack.org/637630
21:10:25	openstackgerrit	Artom Lifshitz proposed openstack/nova master: Introduce live_migration_claim() https://review.openstack.org/635669
21:10:26	openstackgerrit	Artom Lifshitz proposed openstack/nova master: LM: Make dest send NUMAMigrateData to the source https://review.openstack.org/634828
21:10:26	openstackgerrit	Artom Lifshitz proposed openstack/nova master: [WIP] Use live_migration_claim() to check dest resources https://review.openstack.org/634606
21:10:27	openstackgerrit	Artom Lifshitz proposed openstack/nova master: LM: update NUMA-related XML on the source https://review.openstack.org/635229
21:10:36	artom	^^ It works. It actually works.

Earlier   Later