| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-04-11 | |||
| 18:34:41 | dansmith | because that's all we need to revert | |
| 18:34:51 | dansmith | but potentially we need to claim sum(old_disk, new_disk) depending | |
| 18:35:03 | melwitt | I see | |
| 18:35:04 | dansmith | and always both if we're not on the same node of course | |
| 18:35:49 | mriedem | right it's a mess and resize to same host doesn't help https://bugs.launchpad.net/nova/+bug/1790204 | |
| 18:35:51 | openstack | Launchpad bug 1790204 in OpenStack Compute (nova) "Allocations are "doubled up" on same host resize even though there is only 1 server on the host" [High,Triaged] | |
| 18:36:14 | mriedem | melwitt: the old allocations go away on confirm | |
| 18:36:24 | melwitt | ack | |
| 18:36:55 | mriedem | it's probably fair to say that our internal resource tracking and quota usage reporting during resize have just never aligned | |
| 18:37:13 | mriedem | the resource tracker would always report usage for the old flavor on the source node while resized to save room for a revert, | |
| 18:37:20 | mriedem | but the quota usage wouldn't reflect that | |
| 18:37:47 | mriedem | i don't know if the old pre-counting reservations stuff held some quota during a resize or not | |
| 18:38:17 | mriedem | i.e. did we hold a reservation on the target host and then /limits + reserved would report 6 in this scenario rather than 4, idk | |
| 18:38:22 | mriedem | no one would probably notice | |
| 18:44:23 | melwitt | mriedem: doesn't look like it. from this, you would fail quota check if you didn't have room to revert https://github.com/openstack/nova/blob/stable/ocata/nova/compute/api.py#L3201 | |
| 18:48:10 | mriedem | that code is all black magic to me, i'd have to setup an ocata devstack to see what happens in the API | |
| 18:48:30 | mriedem | definitely not a priority, was just thinking about it while i write a functional test for this for my cross-cell series | |
| 18:48:32 | melwitt | it does reserve for the new flavor here though at the beginning of the resize https://github.com/openstack/nova/blob/stable/ocata/nova/compute/api.py#L3351 | |
| 18:50:05 | melwitt | if it's an upsize | |
| 18:53:31 | melwitt | so from what I can tell, it will consume quota usage for the new flavor as soon as the resize starts | |
| 18:53:57 | melwitt | only for an upsize | |
| 18:54:21 | melwitt | otherwise it will consume for the old flavor still | |
| 18:55:22 | mriedem | ok so pre-pike the RT would report usage for both the old and new flavor and /limits API would show usage for the old and new flavor (assuming upsize), and starting in pike we'll only report usage for the new flavor | |
| 18:55:28 | mriedem | which could also be a downsize | |
| 18:55:44 | mriedem | maybe i drop vcpu but bump disk or something | |
| 19:00:37 | melwitt | no, I think the limits API would show the usage for only the new flavor because it would be upsize_delta + current usage (old flavor) | |
| 19:00:43 | melwitt | pre-pike | |
| 19:01:34 | melwitt | sorry I wasn't clear that it reserves the delta between old and new | |
| 19:21:11 | mriedem | incoming | |
| 19:21:47 | openstackgerrit | Merged openstack/nova stable/rocky: Add functional regression test for bug 1669054 https://review.openstack.org/649325 | |
| 19:21:48 | openstack | bug 1669054 in OpenStack Compute (nova) rocky "RequestSpec.ignore_hosts from resize is reused in subsequent evacuate" [Medium,In progress] https://launchpad.net/bugs/1669054 - Assigned to Matt Riedemann (mriedem) | |
| 19:22:04 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Fix ProviderUsageBaseTestCase._run_periodics for multi-cell https://review.openstack.org/641179 | |
| 19:22:04 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Improve CinderFixtureNewAttachFlow https://review.openstack.org/639382 | |
| 19:22:05 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add functional recreate test for bug 1818914 https://review.openstack.org/641521 | |
| 19:22:05 | openstack | bug 1818914 in OpenStack Compute (nova) "Hypervisor resource usage on source still shows old flavor usage after resize confirm until update_available_resource periodic runs" [Low,In progress] https://launchpad.net/bugs/1818914 - Assigned to Matt Riedemann (mriedem) | |
| 19:22:05 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove unused context parameter from RT._get_instance_type https://review.openstack.org/641792 | |
| 19:22:06 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Update usage in RT.drop_move_claim during confirm resize https://review.openstack.org/641806 | |
| 19:22:06 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add Migration.cross_cell_move and get_by_uuid https://review.openstack.org/614012 | |
| 19:22:07 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add InstanceAction/Event create() method https://review.openstack.org/614036 | |
| 19:22:07 | openstackgerrit | Matt Riedemann proposed openstack/nova master: DNM: Add instance hard delete https://review.openstack.org/650984 | |
| 19:22:08 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add Instance.hidden field https://review.openstack.org/631123 | |
| 19:22:08 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add TargetDBSetupTask https://review.openstack.org/627892 | |
| 19:22:09 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add CrossCellMigrationTask https://review.openstack.org/631581 | |
| 19:22:09 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Execute TargetDBSetupTask https://review.openstack.org/633853 | |
| 19:22:10 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add can_connect_volume() compute driver method https://review.openstack.org/621313 | |
| 19:22:10 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add prep_snapshot_based_resize_at_dest compute method https://review.openstack.org/633293 | |
| 19:22:11 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add PrepResizeAtDestTask https://review.openstack.org/627890 | |
| 19:22:11 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add prep_snapshot_based_resize_at_source compute method https://review.openstack.org/634832 | |
| 19:22:12 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add nova.compute.utils.delete_image https://review.openstack.org/637605 | |
| 19:22:12 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add PrepResizeAtSourceTask https://review.openstack.org/627891 | |
| 19:22:24 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add revert_snapshot_based_resize conductor RPC method https://review.openstack.org/638047 | |
| 19:22:24 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Revert cross-cell resize from the API https://review.openstack.org/638048 | |
| 19:22:25 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Confirm cross-cell resize while deleting a server https://review.openstack.org/638268 | |
| 19:22:25 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add archive_deleted_rows wrinkle to cross-cell functional test https://review.openstack.org/651650 | |
| 19:22:26 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add CrossCellWeigher https://review.openstack.org/614353 | |
| 19:22:26 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add cross-cell resize policy rule and enable in API https://review.openstack.org/638269 | |
| 19:22:35 | efried | sean-k-mooney: you still awake? | |
| 19:36:40 | melwitt | efried: fyi, patch to add a few post-release items to the ptl guide https://review.openstack.org/651009 | |
| 19:40:55 | efried | melwitt: ack, on the radar, thank you. | |
| 19:41:52 | efried | oh, that was easy. +2 | |
| 20:00:09 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: [DNM: extra logs] Revert resize: wait for external events in compute manager https://review.openstack.org/644881 | |
| 20:08:10 | mriedem | heh i didn't realize there was a ptl guide | |
| 20:14:28 | openstackgerrit | Merged openstack/python-novaclient master: Add test for console-log and docs for bug 1746534 https://review.openstack.org/651827 | |
| 20:14:29 | openstack | bug 1746534 in python-novaclient "encoding error when doing console-log" [High,Fix released] https://launchpad.net/bugs/1746534 - Assigned to Thomas Goirand (thomas-goirand) | |
| 20:22:15 | melwitt | mriedem: I added it recently. was just a copy-paste of a google doc I had made to help me as ptl | |
| 20:40:33 | melwitt | this came up in downstream bug triage, cells v2 discover_hosts can fail with DBDuplicateEntry if run in parallel per compute host deployed | |
| 20:41:01 | dansmith | melwitt: I'm sure | |
| 20:41:10 | melwitt | found related issues from OSA and chef: https://bugs.launchpad.net/openstack-ansible/+bug/1752540 https://github.com/bloomberg/chef-bcpc/issues/1378 | |
| 20:41:11 | openstack | Launchpad bug 1752540 in openstack-ansible "os_nova cell_v2 discover failure" [Undecided,In progress] - Assigned to git-harry (git-harry) | |
| 20:41:11 | dansmith | why would someone run it on each compute node | |
| 20:42:06 | dansmith | ah the OSA one seems to be that they're running it per conductor | |
| 20:42:18 | dansmith | which is also not really what should be happening, but less bad than compute I guess | |
| 20:42:18 | melwitt | maybe lack of clear documentation but probably just thinking deploy compute host => discover host and made it part of that task piece | |
| 20:42:45 | dansmith | except you have to give the compute node db credentials it doesn't otherwise need in order to do that :) | |
| 20:42:55 | dansmith | but yeah, misunderstanding of what should be done, I'm sure | |
| 20:43:42 | mriedem | i want to say mdbooth_ opened some bug like this | |
| 20:43:47 | mriedem | concurrency something or other | |
| 20:44:07 | mriedem | but it was like db sync concurrently i think | |
| 20:44:08 | dansmith | it's not really a concurrency thing in the way he normally looks for such issues, | |
| 20:44:28 | dansmith | yeah, that also is one of those "don't do that" things, IMHO | |
| 20:44:33 | melwitt | hm, I wasn't thinking that they give db credentials to compute node, just running discover_host per, centrally? I dunno, nevermind | |
| 20:44:45 | dansmith | unless we want to do our own locking in the DB to prevent it, which is kinda silly | |
| 20:44:58 | mriedem | it was this https://bugs.launchpad.net/nova/+bug/1804652 | |
| 20:44:59 | openstack | Launchpad bug 1804652 in OpenStack Compute (nova) "nova.db.sqlalchemy.migration.db_version is racy" [Low,In progress] - Assigned to Matthew Booth (mbooth-9) | |
| 20:45:13 | dansmith | melwitt: you have to have api db credentials to run discover hosts, which are normally not on the compute node (or shouldn't be) | |
| 20:45:51 | dansmith | mriedem: yeah, would love to WONTFIX that | |
| 20:46:05 | mriedem | you could -1 the change | |
| 20:46:07 | mriedem | start a war | |
| 20:46:22 | mriedem | but this is now the month of positivity and motivation | |
| 20:46:38 | melwitt | dansmith: yeah... I was thinking maybe they're doing that in a central place that is deploying compute nodes in parallel but not necessarily running nova-manage _on_ the compute nodes? | |
| 20:47:26 | dansmith | melwitt: who are we talking about? I thought you said "per compute" so I thought you meant on a compute, but maybe you just mean spawning a bunch of nova-manage commands on one node one for each new host? | |
| 20:47:32 | melwitt | I'll reply on the bug with guidance to run discover_hosts once after deploying compute hosts. and then I think I'll have a docs patch in my future | |
| 20:47:33 | dansmith | that's even more.. crazy | |
| 20:47:49 | dansmith | should be using anisble triggers for that, which AFAIK, would mean a thing runs one | |
| 20:47:52 | dansmith | *once | |
| 20:48:08 | dansmith | like, you don't restart apache for every module you enable, you enable a bunch of modules and run restart at the end :) | |
| 20:48:25 | melwitt | dansmith: I don't actually know the details of how the deployment works, was just thinking it seems unlikely they're putting api db creds on compute hosts and running nova-manage on them | |
| 20:48:26 | dansmith | s/triggers/handlers/ | |
| 20:48:45 | dansmith | melwitt: ack, I just thought that was what you originally asserted | |
| 20:49:14 | openstackgerrit | Matt Riedemann proposed openstack/python-novaclient stable/stein: Add test for console-log and docs for bug 1746534 https://review.openstack.org/651925 | |