| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-09-20 | |||
| 16:12:23 | mnaser | holy crap | |
| 16:12:39 | mnaser | i just ran this on a much bigger deployment and its a terrifying amount of stuff that is misreported | |
| 16:13:10 | mnaser | deleted vms, rps missing, and incorrect assignments | |
| 16:13:58 | mriedem | so you have orphaned allocations and rps for deleted things in nova yes? | |
| 16:14:06 | mriedem | likely because of that fix referenced earlier | |
| 16:14:15 | mriedem | incorrect assignments is more troubling | |
| 16:14:30 | tobias-urdin | i've had the same after bombing clouds with testing | |
| 16:14:58 | mnaser | mriedem: im assuming its live migrations not cleaning up | |
| 16:15:08 | mriedem | failed live migrations? | |
| 16:15:22 | mnaser | i dont really know, this cloud is been around for a while so i wouldnt know | |
| 16:15:39 | mnaser | pike though | |
| 16:15:41 | mnaser | err no | |
| 16:15:41 | mnaser | queens | |
| 16:15:42 | mriedem | we definitely have functional tests for successful and failed live migrations where we assert the allocations are in the right place once we're done | |
| 16:16:06 | mnaser | 108 deleted vms that are in placement | |
| 16:16:08 | mriedem | at pike GA i think there were quite a few holes like that though for move operations | |
| 16:16:28 | mnaser | 8 missing RPs (but that is probably related to the bug we just spoke about and doesnt hurt that much) | |
| 16:16:43 | mnaser | 6 'incorrect' assignments | |
| 16:16:54 | mriedem | of those 108 deleted vms still in placement, | |
| 16:17:02 | mriedem | i'd be willing to bet a lot of those were local deletes in the API? | |
| 16:17:08 | mriedem | which was also fixed later | |
| 16:17:22 | mnaser | yeah, i mean i can do some checking if you want me to do that before deeleting | |
| 16:18:08 | mriedem | https://review.openstack.org/#/q/If507e23f0b7e5fa417041c3870d77786498f741d | |
| 16:18:23 | mriedem | https://review.openstack.org/#/c/580498/ isn't even released yet on pike | |
| 16:18:47 | dansmith | sheesh | |
| 16:19:06 | mriedem | merged july 9... | |
| 16:19:42 | mriedem | mnaser: so that 108 and 8 orphaned things are pretty safe to cleanup i think | |
| 16:20:08 | mriedem | the 6 incorrect will likely require a bit of investigation, but could just see where is the instance now and compare to what it's allocations are | |
| 16:24:30 | jroll | jaypipes-ooo: way late, but I just found out we also have a john snow in our org (cc stephenfin) | |
| 16:37:33 | openstackgerrit | Rodolfo Alonso Hernandez proposed openstack/os-vif master: Fix upper-constraints link in tox file https://review.openstack.org/604158 | |
| 16:39:07 | larsks | mnaser: that has been useful twice so far :) | |
| 16:39:21 | mnaser | larsks: meanwhile i just spent most of the past few hours doing the same thing | |
| 16:39:36 | mnaser | but in a different approach that does list all servers + all tenants because that would burn down some oof our deployments | |
| 16:47:58 | mriedem | dansmith: you know how we added a microversion to make live migration an async rpc cast from api to conductor? | |
| 16:48:07 | mriedem | i'm kind of wondering if we should do something similar for resize | |
| 16:48:13 | mnaser | mriedem: hacked some more, http://paste.openstack.org/show/730450/ generates http://paste.openstack.org/show/730460/ | |
| 16:48:25 | mnaser | ill test it out a bit more and try to see if i can add it to nova-manage | |
| 16:48:31 | mriedem | with resize we only finally return the api response once conductor casts to prep_resize on compute | |
| 16:48:35 | dansmith | mriedem: is resize not async already? | |
| 16:48:39 | mriedem | nope | |
| 16:48:40 | openstackgerrit | Merged openstack/nova master: Merge extended_volumes extension response into server view builder https://review.openstack.org/596285 | |
| 16:48:40 | dansmith | because it could take a long ass time | |
| 16:48:43 | mriedem | well, | |
| 16:48:47 | openstackgerrit | Merged openstack/nova stable/rocky: Update RequestSpec.flavor on resize_revert https://review.openstack.org/602403 | |
| 16:48:49 | mriedem | it's async once we pick a compute and cast to it | |
| 16:48:57 | dansmith | ah, okay | |
| 16:49:12 | mriedem | the live migration task before that async microversion would do the reschedules and such within conductor | |
| 16:49:16 | mriedem | which could timeout the api response | |
| 16:49:57 | mriedem | for cross-cell resize, i'm considering doing similar to the pre-migration checks *from* conductor and going back to the scheduler well for more hosts if the one selected doesn't pass the pre-check | |
| 16:50:06 | mriedem | *pre-live migration checks | |
| 16:50:34 | mriedem | likely a todo to throw on the pile for my patch at this point | |
| 16:50:39 | dansmith | ack | |
| 16:50:51 | mriedem | i have officially started losing sleep over this | |
| 16:51:16 | mriedem | need to contract out some hot alaski action | |
| 16:54:29 | dansmith | heh | |
| 16:56:59 | mnaser | i dont have time right now to get this audit code into nova-manage so i'll leave it for anyone to hack on later (till iget the time) | |
| 16:57:22 | mriedem | mnaser: can you report a bug and throw it in there so we don't lose track of it? | |
| 16:57:33 | mnaser | mriedem: fair enough | |
| 16:59:25 | mnaser | https://bugs.launchpad.net/nova/+bug/1793569 | |
| 16:59:25 | openstack | Launchpad bug 1793569 in OpenStack Compute (nova) "Add placement audit commands" [Undecided,New] | |
| 17:05:52 | mriedem | thanks | |
| 17:13:40 | openstackgerrit | Surya Seetharaman proposed openstack/nova master: API microversion bump for handling-down-cell https://review.openstack.org/591657 | |
| 17:19:36 | lucidguy | if I curl the placment API for an allocation (instance) its | |
| 17:19:36 | lucidguy | responding it exists on the wrong hypervisor ID. Openstack | |
| 17:19:39 | lucidguy | calls show it in the right place. How can I edit this, can't | |
| 17:19:42 | lucidguy | figure out the PUT equivalent syntax. Tempted to delete but | |
| 17:19:45 | lucidguy | that sounds very wrong | |
| 17:19:48 | lucidguy | Ouch, sorry about that | |
| 17:20:56 | lucidguy | if I curl the placment API for an allocation (instance) its responding it exists on the wrong hypervisor ID. Openstack calls show it in the right place. How can I edit this, can't figure out the PUT equivalent syntax. Tempted to delete but that sounds very wrong. | |
| 17:21:03 | lucidguy | That's better | |
| 17:25:04 | dansmith | lucidguy: you should use osc-placement to fix it | |
| 17:25:54 | lucidguy | osc-placement.. hmm, never heard of it. | |
| 17:27:09 | lucidguy | How does it work. | |
| 17:27:43 | dansmith | https://docs.openstack.org/osc-placement/latest/index.html | |
| 17:27:58 | dansmith | lucidguy: also check out https://github.com/larsks/os-placement-tools | |
| 17:28:18 | larsks | dansmith: hah, I just pasted that to him in #openstack :) | |
| 17:28:26 | lucidguy | Sounds like something new and time consuming to learn. | |
| 17:29:48 | lucidguy | I would pay for a quick solution.. :) | |
| 17:30:12 | larsks | lucidguy: I haven't been following the conversation, but that script on github may be a quick solution. | |
| 17:30:30 | dansmith | larsks: good, because that's the right place to discuss :) | |
| 17:30:37 | larsks | Yeah | |
| 17:32:24 | lucidguy | I don't think the orginization would like a random script hitting the production enviroment. | |
| 17:32:38 | lucidguy | I'm sure it's fine, but if things were to go wrong, I would be screwed | |
| 17:32:52 | dansmith | so wat | |
| 17:32:55 | dansmith | *wait | |
| 17:33:04 | dansmith | you wanted a quick solution instead of learning a tool, | |
| 17:33:12 | dansmith | and were just going to curl delete an instance's resource allocation, | |
| 17:33:33 | dansmith | but a "random script" that was written by people that know openstack and have tested it in production is not okay? | |
| 17:35:12 | lucidguy | Again, I'm sure it's safe. But my knowledge of the placment service is little to none. Ideally there was a tool provided that can assist me. | |
| 17:35:14 | cdent | lucidguy: what release are you running? | |
| 17:36:05 | lucidguy | Queens | |
| 17:36:51 | cdent | lucidguy: yeah, in that case osc-placement _is_ the tool provided to assist you | |
| 17:37:24 | dansmith | and check_placement is the automated use of that tool to fix your exact problem without you having to learn everything about the tool | |
| 17:38:00 | cdent | lucidguy: also see https://bugs.launchpad.net/nova/+bug/1793569 | |
| 17:38:00 | openstack | Launchpad bug 1793569 in OpenStack Compute (nova) "Add placement audit commands" [Wishlist,Confirmed] | |
| 17:39:06 | lucidguy | I can see https://paste.ubuntu.com/p/8F3JwHqSjz/ with the appropriate curl GET command, is it as simple as performing a PUT to change that hypervisor ID? | |
| 17:39:29 | bauzas | dansmith: mriedem: others: gentle notice that I'll be on PTO tomorrow | |
| 17:39:51 | bauzas | I'm actively working on libvirt reshape, still work to do on inventories | |
| 17:40:03 | bauzas | then I'll push a WIP for review | |
| 17:40:11 | dansmith | bauzas: ack | |