| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-04-11 | |||
| 18:26:49 | mriedem | same method that changes the instance status to VERIFY_RESIZE | |
| 18:27:44 | melwitt | ah ok | |
| 18:28:34 | mriedem | maybe this is part of why we were talking about adding consumer types to placement so nova could ask for usage for 'instances' to filter out the usage tracked by the migration record holding the old flavor usage | |
| 18:30:22 | mriedem | anyway, it's a thing for cross-cell resize because while the instance is in VERIFY_RESIZE status we'll have the instance in both the source and target dbs and if we're counting from the dbs we don't want to double count the instance, so i need to filter out the hidden one, | |
| 18:30:30 | mriedem | but that got me thinking about how vcpus/ram will be counted | |
| 18:31:15 | melwitt | yeah. food for thought | |
| 18:31:30 | mriedem | i'm assuming no one wants to think about that or eat that food though | |
| 18:31:37 | melwitt | of course not | |
| 18:33:13 | openstackgerrit | Merged openstack/nova stable/rocky: doc: Fix openstack CLI command https://review.openstack.org/648425 | |
| 18:33:21 | openstackgerrit | Merged openstack/nova stable/rocky: doc: Capitalize keystone domain name https://review.openstack.org/650601 | |
| 18:33:40 | melwitt | this was brought up before in earlier discussions about counting from placement I think, and IIRC some (dansmith?) thought if placement is consuming 6 resources at that point in time, it makes sense for quota usage counting to reflect that as well | |
| 18:34:13 | dansmith | it depends on the resource and the direction | |
| 18:34:27 | melwitt | when do the old allocations go away from placement? VERIFY_RESIZE or after CONFIRM_RESIZE? | |
| 18:34:28 | dansmith | ideally we would only consume max(old_cpus, new_cpus) from quota | |
| 18:34:36 | dansmith | (and placement | |
| 18:34:41 | dansmith | because that's all we need to revert | |
| 18:34:51 | dansmith | but potentially we need to claim sum(old_disk, new_disk) depending | |
| 18:35:03 | melwitt | I see | |
| 18:35:04 | dansmith | and always both if we're not on the same node of course | |
| 18:35:49 | mriedem | right it's a mess and resize to same host doesn't help https://bugs.launchpad.net/nova/+bug/1790204 | |
| 18:35:51 | openstack | Launchpad bug 1790204 in OpenStack Compute (nova) "Allocations are "doubled up" on same host resize even though there is only 1 server on the host" [High,Triaged] | |
| 18:36:14 | mriedem | melwitt: the old allocations go away on confirm | |
| 18:36:24 | melwitt | ack | |
| 18:36:55 | mriedem | it's probably fair to say that our internal resource tracking and quota usage reporting during resize have just never aligned | |
| 18:37:13 | mriedem | the resource tracker would always report usage for the old flavor on the source node while resized to save room for a revert, | |
| 18:37:20 | mriedem | but the quota usage wouldn't reflect that | |
| 18:37:47 | mriedem | i don't know if the old pre-counting reservations stuff held some quota during a resize or not | |
| 18:38:17 | mriedem | i.e. did we hold a reservation on the target host and then /limits + reserved would report 6 in this scenario rather than 4, idk | |
| 18:38:22 | mriedem | no one would probably notice | |
| 18:44:23 | melwitt | mriedem: doesn't look like it. from this, you would fail quota check if you didn't have room to revert https://github.com/openstack/nova/blob/stable/ocata/nova/compute/api.py#L3201 | |
| 18:48:10 | mriedem | that code is all black magic to me, i'd have to setup an ocata devstack to see what happens in the API | |
| 18:48:30 | mriedem | definitely not a priority, was just thinking about it while i write a functional test for this for my cross-cell series | |
| 18:48:32 | melwitt | it does reserve for the new flavor here though at the beginning of the resize https://github.com/openstack/nova/blob/stable/ocata/nova/compute/api.py#L3351 | |
| 18:50:05 | melwitt | if it's an upsize | |
| 18:53:31 | melwitt | so from what I can tell, it will consume quota usage for the new flavor as soon as the resize starts | |
| 18:53:57 | melwitt | only for an upsize | |
| 18:54:21 | melwitt | otherwise it will consume for the old flavor still | |
| 18:55:22 | mriedem | ok so pre-pike the RT would report usage for both the old and new flavor and /limits API would show usage for the old and new flavor (assuming upsize), and starting in pike we'll only report usage for the new flavor | |
| 18:55:28 | mriedem | which could also be a downsize | |
| 18:55:44 | mriedem | maybe i drop vcpu but bump disk or something | |
| 19:00:37 | melwitt | no, I think the limits API would show the usage for only the new flavor because it would be upsize_delta + current usage (old flavor) | |
| 19:00:43 | melwitt | pre-pike | |
| 19:01:34 | melwitt | sorry I wasn't clear that it reserves the delta between old and new | |
| 19:21:11 | mriedem | incoming | |
| 19:21:47 | openstackgerrit | Merged openstack/nova stable/rocky: Add functional regression test for bug 1669054 https://review.openstack.org/649325 | |
| 19:21:48 | openstack | bug 1669054 in OpenStack Compute (nova) rocky "RequestSpec.ignore_hosts from resize is reused in subsequent evacuate" [Medium,In progress] https://launchpad.net/bugs/1669054 - Assigned to Matt Riedemann (mriedem) | |
| 19:22:04 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Fix ProviderUsageBaseTestCase._run_periodics for multi-cell https://review.openstack.org/641179 | |
| 19:22:04 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Improve CinderFixtureNewAttachFlow https://review.openstack.org/639382 | |
| 19:22:05 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add functional recreate test for bug 1818914 https://review.openstack.org/641521 | |
| 19:22:05 | openstack | bug 1818914 in OpenStack Compute (nova) "Hypervisor resource usage on source still shows old flavor usage after resize confirm until update_available_resource periodic runs" [Low,In progress] https://launchpad.net/bugs/1818914 - Assigned to Matt Riedemann (mriedem) | |
| 19:22:05 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove unused context parameter from RT._get_instance_type https://review.openstack.org/641792 | |
| 19:22:06 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Update usage in RT.drop_move_claim during confirm resize https://review.openstack.org/641806 | |
| 19:22:06 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add Migration.cross_cell_move and get_by_uuid https://review.openstack.org/614012 | |
| 19:22:07 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add InstanceAction/Event create() method https://review.openstack.org/614036 | |
| 19:22:07 | openstackgerrit | Matt Riedemann proposed openstack/nova master: DNM: Add instance hard delete https://review.openstack.org/650984 | |
| 19:22:08 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add Instance.hidden field https://review.openstack.org/631123 | |
| 19:22:08 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add TargetDBSetupTask https://review.openstack.org/627892 | |
| 19:22:09 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add CrossCellMigrationTask https://review.openstack.org/631581 | |
| 19:22:09 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Execute TargetDBSetupTask https://review.openstack.org/633853 | |
| 19:22:10 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add can_connect_volume() compute driver method https://review.openstack.org/621313 | |
| 19:22:10 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add prep_snapshot_based_resize_at_dest compute method https://review.openstack.org/633293 | |
| 19:22:11 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add PrepResizeAtDestTask https://review.openstack.org/627890 | |
| 19:22:11 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add prep_snapshot_based_resize_at_source compute method https://review.openstack.org/634832 | |
| 19:22:12 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add nova.compute.utils.delete_image https://review.openstack.org/637605 | |
| 19:22:12 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add PrepResizeAtSourceTask https://review.openstack.org/627891 | |
| 19:22:24 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add revert_snapshot_based_resize conductor RPC method https://review.openstack.org/638047 | |
| 19:22:24 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Revert cross-cell resize from the API https://review.openstack.org/638048 | |
| 19:22:25 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Confirm cross-cell resize while deleting a server https://review.openstack.org/638268 | |
| 19:22:25 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add archive_deleted_rows wrinkle to cross-cell functional test https://review.openstack.org/651650 | |
| 19:22:26 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add CrossCellWeigher https://review.openstack.org/614353 | |
| 19:22:26 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add cross-cell resize policy rule and enable in API https://review.openstack.org/638269 | |
| 19:22:35 | efried | sean-k-mooney: you still awake? | |
| 19:36:40 | melwitt | efried: fyi, patch to add a few post-release items to the ptl guide https://review.openstack.org/651009 | |
| 19:40:55 | efried | melwitt: ack, on the radar, thank you. | |
| 19:41:52 | efried | oh, that was easy. +2 | |
| 20:00:09 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: [DNM: extra logs] Revert resize: wait for external events in compute manager https://review.openstack.org/644881 | |
| 20:08:10 | mriedem | heh i didn't realize there was a ptl guide | |
| 20:14:28 | openstackgerrit | Merged openstack/python-novaclient master: Add test for console-log and docs for bug 1746534 https://review.openstack.org/651827 | |
| 20:14:29 | openstack | bug 1746534 in python-novaclient "encoding error when doing console-log" [High,Fix released] https://launchpad.net/bugs/1746534 - Assigned to Thomas Goirand (thomas-goirand) | |
| 20:22:15 | melwitt | mriedem: I added it recently. was just a copy-paste of a google doc I had made to help me as ptl | |
| 20:40:33 | melwitt | this came up in downstream bug triage, cells v2 discover_hosts can fail with DBDuplicateEntry if run in parallel per compute host deployed | |
| 20:41:01 | dansmith | melwitt: I'm sure | |
| 20:41:10 | melwitt | found related issues from OSA and chef: https://bugs.launchpad.net/openstack-ansible/+bug/1752540 https://github.com/bloomberg/chef-bcpc/issues/1378 | |
| 20:41:11 | openstack | Launchpad bug 1752540 in openstack-ansible "os_nova cell_v2 discover failure" [Undecided,In progress] - Assigned to git-harry (git-harry) | |
| 20:41:11 | dansmith | why would someone run it on each compute node | |
| 20:42:06 | dansmith | ah the OSA one seems to be that they're running it per conductor | |
| 20:42:18 | dansmith | which is also not really what should be happening, but less bad than compute I guess | |
| 20:42:18 | melwitt | maybe lack of clear documentation but probably just thinking deploy compute host => discover host and made it part of that task piece | |
| 20:42:45 | dansmith | except you have to give the compute node db credentials it doesn't otherwise need in order to do that :) | |
| 20:42:55 | dansmith | but yeah, misunderstanding of what should be done, I'm sure | |
| 20:43:42 | mriedem | i want to say mdbooth_ opened some bug like this | |
| 20:43:47 | mriedem | concurrency something or other | |
| 20:44:07 | mriedem | but it was like db sync concurrently i think | |
| 20:44:08 | dansmith | it's not really a concurrency thing in the way he normally looks for such issues, | |
| 20:44:28 | dansmith | yeah, that also is one of those "don't do that" things, IMHO | |
| 20:44:33 | melwitt | hm, I wasn't thinking that they give db credentials to compute node, just running discover_host per, centrally? I dunno, nevermind | |
| 20:44:45 | dansmith | unless we want to do our own locking in the DB to prevent it, which is kinda silly | |
| 20:44:58 | mriedem | it was this https://bugs.launchpad.net/nova/+bug/1804652 | |
| 20:44:59 | openstack | Launchpad bug 1804652 in OpenStack Compute (nova) "nova.db.sqlalchemy.migration.db_version is racy" [Low,In progress] - Assigned to Matthew Booth (mbooth-9) | |
| 20:45:13 | dansmith | melwitt: you have to have api db credentials to run discover hosts, which are normally not on the compute node (or shouldn't be) | |