Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-09-20
15:58:48 mnaser woo
15:58:54 mnaser i got a script going and it helped figure out the issue too
15:59:49 mnaser it's really not cleaned up but does the job, here's a placement 'audit' script: http://paste.openstack.org/show/730450/
16:01:15 mriedem cool, throw that into "nova-manage placement audit" or something
16:02:07 mriedem should be able to use nova.scheduler.client.report.SchedulerReportClient for the HTTP calls
16:02:18 mnaser yeah, and probably can use nova objects to avoid http calls?
16:02:38 mriedem yeah maybe
16:03:20 mnaser oh boi
16:03:57 mnaser i think i'd say its a bug if a field is missing in nova api if it's empty/blank?
16:04:12 mriedem depends
16:04:13 mriedem which field
16:04:24 mnaser with_servers=True
16:04:31 mnaser the servers: [] field is omitted if no servers are there
16:04:40 mnaser so rather than servers: []. its just not there
16:04:46 mriedem welcome to the compute api
16:04:53 mnaser i.e.: {u'status': u'disabled', u'state': u'down', u'id': u'13b1a4b6-d242-44c5-828e-e6f0a94284da', u'hypervisor_hostname': u'foobar'}
16:05:51 mriedem i believe that's how https://developer.openstack.org/api-ref/compute/#list-hypervisor-servers worked
16:05:58 mnaser https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/hypervisors.py#L90-L92
16:06:05 mnaser if servers with servers = [] evaluates to false?
16:06:11 mriedem yes
16:06:50 mnaser i mean that could have been omitted but i dunno if thats an api bug or something we're expecting
16:07:14 mriedem that's just how that's always worked,
16:07:21 mriedem and this is probably not the only instance of it
16:07:30 mriedem where a field is omitted in the response if it doesn't have any value
16:07:52 mriedem in this case it was just an oversight when i rewrote os-hypervisors in 2.53
16:07:59 mriedem b/c it's using the same view builder code
16:08:07 mnaser its nice that it gave compute node uuids
16:08:08 mnaser it made this a lot easier
16:08:29 mriedem yup, that's part of why we had to do 2.53
16:08:53 mriedem https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#maximum-in-pike
16:08:59 mriedem it wasn't a fun microversion to write let me tell you
16:09:24 mriedem gmann does have a spec started for fixing random warts like this https://review.openstack.org/#/c/603969/
16:10:01 mriedem i'll throw this into the etherpad
16:11:54 openstackgerrit caoyuan proposed openstack/nova master: Option "scheduler_default_filters" is deprecated. https://review.openstack.org/604148
16:12:23 mnaser holy crap
16:12:39 mnaser i just ran this on a much bigger deployment and its a terrifying amount of stuff that is misreported
16:13:10 mnaser deleted vms, rps missing, and incorrect assignments
16:13:58 mriedem so you have orphaned allocations and rps for deleted things in nova yes?
16:14:06 mriedem likely because of that fix referenced earlier
16:14:15 mriedem incorrect assignments is more troubling
16:14:30 tobias-urdin i've had the same after bombing clouds with testing
16:14:58 mnaser mriedem: im assuming its live migrations not cleaning up
16:15:08 mriedem failed live migrations?
16:15:22 mnaser i dont really know, this cloud is been around for a while so i wouldnt know
16:15:39 mnaser pike though
16:15:41 mnaser err no
16:15:41 mnaser queens
16:15:42 mriedem we definitely have functional tests for successful and failed live migrations where we assert the allocations are in the right place once we're done
16:16:06 mnaser 108 deleted vms that are in placement
16:16:08 mriedem at pike GA i think there were quite a few holes like that though for move operations
16:16:28 mnaser 8 missing RPs (but that is probably related to the bug we just spoke about and doesnt hurt that much)
16:16:43 mnaser 6 'incorrect' assignments
16:16:54 mriedem of those 108 deleted vms still in placement,
16:17:02 mriedem i'd be willing to bet a lot of those were local deletes in the API?
16:17:08 mriedem which was also fixed later
16:17:22 mnaser yeah, i mean i can do some checking if you want me to do that before deeleting
16:18:08 mriedem https://review.openstack.org/#/q/If507e23f0b7e5fa417041c3870d77786498f741d
16:18:23 mriedem https://review.openstack.org/#/c/580498/ isn't even released yet on pike
16:18:47 dansmith sheesh
16:19:06 mriedem merged july 9...
16:19:42 mriedem mnaser: so that 108 and 8 orphaned things are pretty safe to cleanup i think
16:20:08 mriedem the 6 incorrect will likely require a bit of investigation, but could just see where is the instance now and compare to what it's allocations are
16:24:30 jroll jaypipes-ooo: way late, but I just found out we also have a john snow in our org (cc stephenfin)
16:37:33 openstackgerrit Rodolfo Alonso Hernandez proposed openstack/os-vif master: Fix upper-constraints link in tox file https://review.openstack.org/604158
16:39:07 larsks mnaser: that has been useful twice so far :)
16:39:21 mnaser larsks: meanwhile i just spent most of the past few hours doing the same thing
16:39:36 mnaser but in a different approach that does list all servers + all tenants because that would burn down some oof our deployments
16:47:58 mriedem dansmith: you know how we added a microversion to make live migration an async rpc cast from api to conductor?
16:48:07 mriedem i'm kind of wondering if we should do something similar for resize
16:48:13 mnaser mriedem: hacked some more, http://paste.openstack.org/show/730450/ generates http://paste.openstack.org/show/730460/
16:48:25 mnaser ill test it out a bit more and try to see if i can add it to nova-manage
16:48:31 mriedem with resize we only finally return the api response once conductor casts to prep_resize on compute
16:48:35 dansmith mriedem: is resize not async already?
16:48:39 mriedem nope
16:48:40 openstackgerrit Merged openstack/nova master: Merge extended_volumes extension response into server view builder https://review.openstack.org/596285
16:48:40 dansmith because it could take a long ass time
16:48:43 mriedem well,
16:48:47 openstackgerrit Merged openstack/nova stable/rocky: Update RequestSpec.flavor on resize_revert https://review.openstack.org/602403
16:48:49 mriedem it's async once we pick a compute and cast to it
16:48:57 dansmith ah, okay
16:49:12 mriedem the live migration task before that async microversion would do the reschedules and such within conductor
16:49:16 mriedem which could timeout the api response
16:49:57 mriedem for cross-cell resize, i'm considering doing similar to the pre-migration checks *from* conductor and going back to the scheduler well for more hosts if the one selected doesn't pass the pre-check
16:50:06 mriedem *pre-live migration checks
16:50:34 mriedem likely a todo to throw on the pile for my patch at this point
16:50:39 dansmith ack
16:50:51 mriedem i have officially started losing sleep over this
16:51:16 mriedem need to contract out some hot alaski action
16:54:29 dansmith heh
16:56:59 mnaser i dont have time right now to get this audit code into nova-manage so i'll leave it for anyone to hack on later (till iget the time)
16:57:22 mriedem mnaser: can you report a bug and throw it in there so we don't lose track of it?
16:57:33 mnaser mriedem: fair enough
16:59:25 mnaser https://bugs.launchpad.net/nova/+bug/1793569
16:59:25 openstack Launchpad bug 1793569 in OpenStack Compute (nova) "Add placement audit commands" [Undecided,New]
17:05:52 mriedem thanks
17:13:40 openstackgerrit Surya Seetharaman proposed openstack/nova master: API microversion bump for handling-down-cell https://review.openstack.org/591657
17:19:36 lucidguy if I curl the placment API for an allocation (instance) its
17:19:36 lucidguy responding it exists on the wrong hypervisor ID. Openstack
17:19:39 lucidguy calls show it in the right place. How can I edit this, can't
17:19:42 lucidguy figure out the PUT equivalent syntax. Tempted to delete but

Earlier   Later