| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-02-18 | |||
| 17:59:49 | jaypipes | zzzeek: hmm, neutron patch is a rabbithole for me... abstraction on top of abstraction. I'm afraid that I'm having a difficult time following what SQL statements would/will be produced at which events with this... | |
| 18:00:13 | zzzeek | jaypipes: I know, neutron is a PITA | |
| 18:00:26 | zzzeek | jaypipes: i never work on it because it is nonsensical | |
| 18:00:35 | zzzeek | jaypipes: but i figured out enough what this does :) | |
| 18:00:37 | openstackgerrit | Elod Illes proposed openstack/nova stable/queens: Make host_manager use scatter-gather and ignore down cells https://review.openstack.org/637599 | |
| 18:02:46 | jaypipes | zzzeek: this patch definitely fixes a number of deadlocks you are seeing, yes? | |
| 18:03:14 | jaypipes | zzzeek: where specifically are those deadlocks being hit? (in the neutron code, that is...) | |
| 18:04:06 | zzzeek | jaypipes: it fixes SQLAlchemy version id misses, not "deadlocks" in the literal sense | |
| 18:04:16 | zzzeek | jaypipes: SQLAlchemy StaleDataErrors | |
| 18:04:55 | zzzeek | jaypipes: they have this test case where they create 300 security group rules | |
| 18:05:15 | zzzeek | jaypipes: and that incurs a versino bump on many related objects because their API requiers that the new number is exposed | |
| 18:05:25 | zzzeek | jaypipes: so the version_id here is used for two purposes | |
| 18:05:57 | zzzeek | jaypipes: so when they make these 300 rules they get a ton of StaleDataErrors on all the related objects, for which they are changing nothing except bumping the version id | |
| 18:06:10 | jaypipes | zzzeek: wait, are you saying that updates of dependent objects cause the version to be incremented more than once? | |
| 18:06:14 | zzzeek | jaypipes: this also incurs that an ORM lookup is happening, a flush, etc. which is all overhead too | |
| 18:06:31 | zzzeek | jaypipes: no, when they have multiple transactions on concurrent API requests | |
| 18:07:20 | zzzeek | jaypipes: look at feb 8 comments on the gerrit: "POST request response times for 300 security group rules creation http://paste.openstack.org/show/744727/ | |
| 18:07:20 | zzzeek | script I used to create 300 security group rules asynchronously http://paste.openstack.org/show/744728/" | |
| 18:08:26 | jaypipes | zzzeek: ok, creating 300 security group rules should increment the security group's version to 301, yes? | |
| 18:08:48 | aspiers | Wow, plantuml is pretty cool. I was able to knock this up pretty quickly from a small text file: https://goo.gl/tPNAQK | |
| 18:08:48 | zzzeek | jaypipes: i dunno. the person testing is saying he is getting the correct results though | |
| 18:09:04 | zzzeek | yes 301 | |
| 18:09:20 | zzzeek | Even the revision_number for security group updated properly when creating 300 security group rules MariaDB [neutron]> select * from standardattributes where id="2151"; +------+----------------+---------------------+---------------------+-------------+-----------------+ | id | resource_type | created_at | updated_at | description | revision_number | +------+----------------+---------------------+---------------------+-------------+ | |
| 18:09:20 | zzzeek | -----------------+ | 2151 | securitygroups | 2019-02-08 09:07:15 | 2019-02-08 09:09:05 | | 301 | +------+----------------+---------------------+---------------------+-------------+-----------------+ | |
| 18:42:42 | mnaser | mriedem: we might want to ping the ML about the change we recently did with disallowing root_gb=0 flavors | |
| 18:43:33 | mnaser | osa gates broke because i assume we created flavors which were using root_gb=0 | |
| 18:43:48 | mriedem | we?!?! | |
| 18:43:50 | mnaser | i can shoot an email if you weren't planning to | |
| 18:43:55 | mriedem | go nuts | |
| 18:44:04 | mriedem | i send enough "oh btw everything is going to be broken now" emails | |
| 18:44:04 | mnaser | https://review.openstack.org/#/c/603910/ | |
| 18:44:08 | mnaser | :D | |
| 18:44:09 | mriedem | yeah i know | |
| 18:59:57 | openstackgerrit | Surya Seetharaman proposed openstack/python-novaclient master: API microversion 2.69: Handles Down Cells https://review.openstack.org/579563 | |
| 19:03:48 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add TargetDBSetupTask https://review.openstack.org/627892 | |
| 19:03:49 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add CrossCellMigrationTask https://review.openstack.org/631581 | |
| 19:03:49 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Execute TargetDBSetupTask https://review.openstack.org/633853 | |
| 19:03:50 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add can_connect_volume() compute driver method https://review.openstack.org/621313 | |
| 19:03:50 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Add prep_snapshot_based_resize_at_dest compute method https://review.openstack.org/633293 | |
| 19:03:51 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Add PrepResizeAtDestTask https://review.openstack.org/627890 | |
| 19:03:51 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add prep_snapshot_based_resize_at_source compute method https://review.openstack.org/634832 | |
| 19:03:52 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add PrepResizeAtSourceTask https://review.openstack.org/627891 | |
| 19:03:52 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Add finish_snapshot_based_resize_at_dest compute method https://review.openstack.org/635080 | |
| 19:03:53 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add FinishResizeAtDestTask https://review.openstack.org/635646 | |
| 19:03:53 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Execute CrossCellMigrationTask from MigrationTask https://review.openstack.org/635668 | |
| 19:03:54 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Plumb allow_cross_cell_resize into compute API resize() https://review.openstack.org/635684 | |
| 19:03:54 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Filter duplicates from compute API get_migrations_sorted() https://review.openstack.org/636224 | |
| 19:03:55 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Start functional testing for cross-cell resize https://review.openstack.org/636253 | |
| 19:03:55 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Add confirm_snapshot_based_resize_at_source https://review.openstack.org/637058 | |
| 19:03:56 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Add ConfirmResizeTask https://review.openstack.org/637070 | |
| 19:03:56 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Add confirm_snapshot_based_resize conductor RPC method https://review.openstack.org/637075 | |
| 19:03:57 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Confirm cross-cell resize from the API https://review.openstack.org/637316 | |
| 19:05:04 | mriedem | that's only about half the series at this point | |
| 19:05:06 | mriedem | swoon away | |
| 19:36:25 | NewBruce | sean-k-mooney still here mate? | |
| 19:41:25 | mordred | mriedem: did nova/devstack introduce a behavior change recentlyish? http://logs.openstack.org/63/637563/1/check/openstacksdk-functional-devstack-tips/f7e749c/testr_results.html.gz | |
| 19:41:45 | mordred | mriedem: | |
| 19:41:47 | mordred | openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://213.32.77.235/compute/v2.1/servers, Only volume-backed servers are allowed for flavors with zero disk. | |
| 19:41:54 | dansmith | hah | |
| 19:42:02 | dansmith | mordred: [10:42:42] <mnaser>mriedem: we might want to ping the ML about the change we recently did with disallowing root_gb=0 flavors | |
| 19:42:08 | mordred | hahaha | |
| 19:42:23 | dansmith | made it a whole hour | |
| 19:42:31 | mnaser | mordred: http://lists.openstack.org/pipermail/openstack-discuss/2019-February/002842.html :D | |
| 19:43:36 | mordred | mnaser: so - I might need to port that change to apply to non-tempest flavors I suppose? | |
| 19:44:01 | mnaser | mordred: so whatever tests you were doing were trying to do a boot from volume using a flavor that has root_gb=0 | |
| 19:44:14 | mnaser | i assume openstacksdk maybe is trying to create its own flavor? | |
| 19:44:34 | mordred | not to my knowledge, no | |
| 19:44:41 | mordred | we just use whatever flavor is there | |
| 19:44:52 | mordred | but we don't install/run tempest, so I doubt lib/tempest flavors exist | |
| 19:45:44 | mnaser | {"href": "https://213.32.77.235/compute/flavors/c1", "rel": "bookmark"}], "ram": 256, "OS-FLV-DISABLED:disabled": false, "os-flavor-access:is_public": true, "rxtx_factor": 1.0, "disk": 0, "id": "c1", "name": "cirros256", "vcpus": 1, "swap": "", "OS-FLV-EXT-DATA:ephemeral": 0} | |
| 19:45:55 | mordred | mnaser: http://paste.openstack.org/raw/745281/ | |
| 19:46:25 | mordred | mnaser: so I guess I just need to pick a different flavor | |
| 19:46:38 | mnaser | we should probably fix what creates cirros256 | |
| 19:47:15 | mnaser | did taht fail in nodepool, mordred ? | |
| 19:47:32 | mnaser | no openstacksdk, hmm weird | |
| 19:47:57 | mordred | mnaser: we pick the smallest flavor sorted by ram | |
| 19:48:05 | mnaser | *something* creates that flavor.. | |
| 19:48:14 | mordred | mnaser: so I think I need to add an 'if' for disk != 0 | |
| 19:48:41 | mnaser | yeah if you're not doing bfv and the flavor has disk == 0 then it should not use that flavor | |
| 19:49:21 | mnaser | but http://git.openstack.org/cgit/openstack-dev/devstack/tree/lib/nova#n1165 | |
| 19:49:22 | cdent | devstack creates that flavor | |
| 19:49:24 | cdent | jinx | |
| 19:49:56 | mnaser | mordred: so i guess in openstacksdk world of things, it shouldn't pick a flavor that has disk_gb=0 when trying to bfv (as of train) | |
| 19:52:07 | mordred | mnaser: https://review.openstack.org/637618 Make sure we pick flavors with disk <-- I think that should do it yeah? | |
| 19:53:23 | mnaser | mordred: assuming python evaluates flavor.disk set to 0 as false then yes | |
| 19:53:31 | mordred | mnaser: we'll find out! :) | |
| 19:53:35 | mnaser | but i forget fundamental python rules | |
| 19:53:57 | mnaser | we're supposed to be experts but half the tabs i open are "how the f does X language do switch cases" | |
| 20:36:36 | openstackgerrit | Theodoros Tsioutsias proposed openstack/nova master: Introduce the PENDING instance state https://review.openstack.org/566473 | |
| 20:36:37 | openstackgerrit | Theodoros Tsioutsias proposed openstack/nova master: Allow rebuild for instances in PENDING state https://review.openstack.org/637585 | |
| 20:37:55 | openstackgerrit | Vlad Gusev proposed openstack/nova master: [WIP] libvirt: Add a default Virtio RNG device to guests https://review.openstack.org/616603 | |
| 20:38:16 | openstackgerrit | Vlad Gusev proposed openstack/nova master: libvirt: Add a default Virtio RNG device to guests https://review.openstack.org/616603 | |
| 20:57:13 | mriedem | weird that we send the resize.revert.(start|end) notifications from the source host during finish_revert_resize which is half of the actual revert | |
| 20:57:22 | mriedem | the start notification should happen on the dest host in revert_resize() | |
| 20:57:25 | mriedem | must not be a big deal | |
| 21:03:11 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Add revert_snapshot_based_resize_at_dest compute method https://review.openstack.org/637630 | |
| 21:10:25 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Introduce live_migration_claim() https://review.openstack.org/635669 | |
| 21:10:26 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: [WIP] Use live_migration_claim() to check dest resources https://review.openstack.org/634606 | |
| 21:10:26 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: LM: Make dest send NUMAMigrateData to the source https://review.openstack.org/634828 | |
| 21:10:27 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: LM: update NUMA-related XML on the source https://review.openstack.org/635229 | |