Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-07-08
13:43:53 bauzas I mean, checking the DB
13:46:03 mriedem i'd think you'd avoid races by checking if an instance has a task_state != None, and if looking at migrations, is the migration in progress
13:48:09 mriedem mnaser's script doesn't loop over instances,
13:48:27 mriedem it gets hypervisors and the servers on those hypervisors and compares to the resource provider inventory/allocation per hypervisor/server
13:50:13 mriedem i'd think it would be easiest to first determine what kinds of problems the audit command is going to try and find
13:50:29 mriedem and then figure out the best way to implement that
13:51:08 mriedem i think we know the two main issues are missing allocations (which heal_allocations should fix) and leaked / orphaned allocations, so i'd start by focusing on the latter issue
13:55:17 mriedem i think we could pretty easily identify leaked allocations by starting from the provider allocations and working backward - looking for servers or migration records with the allocation consumer id (since we don't have the type recorded yet)
13:59:30 mriedem so i'd probably start by getting all hypervisors (nodes/resource providers) and for each hypervisor, get the allocations - similar to what mnaser's script does
14:00:00 mriedem getting the hypervisors + servers in the same call like his script would also tell you if allocations on that provider was a server or not - and if not, it's either leaked or it's a migration allocation
14:00:09 mriedem so no i don't think you need to loop over all instances
14:00:18 mriedem maybe i should just write this thing :)
14:06:37 bauzas mriedem: I'm pretty sure I can do it
14:16:29 openstackgerrit François Palin proposed openstack/nova master: WIP - call volume detach even if terminate_connection fails https://review.opendev.org/669674
14:19:01 kashyap mriedem: You're right ... indeed I didn't hear any responses on the list so far (http://lists.openstack.org/pipermail/openstack-discuss/2019-July/007521.html)
14:19:33 kashyap alex_xu: Hiya, when you're about, do you have a comment on the above thread? (And the change wich Matt Cced you on, which removes Intel CMT events)
14:20:10 efried kashyap: As predicted, my wife immediately stole the sleep book
14:20:20 efried and it is life-changing
14:20:41 kashyap efried: Hehe. Sharing is caring. I'm sure she will not hide her fascination with the book
14:20:56 efried I don't know why, but stuff I've been telling her for years for some reason is having more of an impact coming from a complete stranger
14:20:57 kashyap efried: Ah, she already finished it?
14:21:20 efried no, hasn't finished yet, but every chapter brings new insight.
14:21:26 efried she has bought a copy for her parents
14:21:31 efried and recommended it to a number of people.
14:21:36 kashyap :-)
14:21:49 efried I got a couple chapters in before she stole it, and I'm definitely sold
14:21:57 kashyap efried: I know, you must've become like a "broken record", and people zone out the moment broken reel start spinning :D
14:22:06 kashyap Sorry, just teasing.
14:22:35 efried though for me it's more fascination with the science than altering lifestyle, which was already pretty close to right sleep-wise.
14:23:07 kashyap Yeah, agreed.
14:23:46 kashyap efried: Also, I'd highly recommend to watch him give a tech talk. He's one of the most effective speakers I've ever seen.
14:24:01 efried noted
14:24:08 kashyap The way he pauses for the words to sink in, diction, is all worth observing. At least to me.
14:24:11 kashyap https://www.youtube.com/watch?v=aXflBZXAucQ
14:24:42 kashyap efried: If you click, look how quickly (within 5 seconds) he escalates the talk. :D
14:29:27 prometheanfire was pointed here for nova adding a cinder volume to a server without (the record/db at least) but not actually adding it to the running VM (client reports a 504 when trying to add the volume)
14:30:28 prometheanfire it shows up in the volumes_attached field, but can't remove any of them as they either do not exist or.....
14:30:32 prometheanfire Invalid volume: Invalid input received: Invalid volume: Unable to detach volume. Volume status must be 'in-use' and attach_status must be 'attached' to detach. (HTTP 400) (Request-ID: req-01d8a3c5-66f4-43a4-bac5-9f1104a292fe) (HTTP 400) (Request-ID: req-358132e4-38a1-493a-b87b-d135791bae2d)
15:02:46 efried kashyap: Would you mind taking a look at this please?
15:02:46 efried https://review.opendev.org/#/c/667975/
15:02:46 efried It seems straightforward enough, but I'm not sure if there could e.g. be security problems, or if there's another accepted way to do this, etc.
15:03:09 kashyap efried: In a meeting; will look
15:03:15 efried no hurry
15:03:22 efried thanks
15:09:28 stephenfin lyarwood: Could you send this on its merry way? https://review.opendev.org/#/c/667355/
15:11:04 lyarwood stephenfin: looking
15:11:48 lyarwood stephenfin: sorry I did mean to sort that out last week once the stein change landed
15:11:52 lyarwood stephenfin: done
15:13:07 mriedem gibi: comments in https://review.opendev.org/#/c/669188/ just so i'm sure i'm following it correctly (i know this patch was because i brought it up elsewhere)
15:13:57 gibi mriedem: looking..
15:26:43 gibi mriedem: replied. I only added some extra explanation but I think we agree.
15:27:22 gibi mriedem: I can re-spin the patch to fix the test redundancies
15:28:46 mriedem yeah just fix the test and i'm +2
15:28:57 gibi mriedem: OK, I will do it quickly
15:29:51 cdent mriedem, efried : I have a situation where under high load the _update_to_placement call made at the end of instance_claim in the compute manager can fail on ResourceProviderConflict all 4 retry attempts (because of nova-scheduler simultaneously making many allocations to the same resource provider). This fails the instance build even though the instance did build.
15:30:52 efried cdent: That is very interesting.
15:31:04 cdent I'm considering a hack to make instance_claim not call _update_to_placement and just letting the periodic job do that. The reason _update_to_placement is actually talking to placement is because inventory (DISK_GB max_unit) does change after many (most) instance creations
15:31:30 cdent ideally that max_unit thing wouldn't be in place but that's not really any immediate option
15:31:36 efried I'm guessing backing off the retry interval wouldn't be of help, because the problem is the timing between the GET and the PUT(s).
15:31:49 sean-k-mooney cdent: were we not looking at disabling the periodic job at one point
15:31:51 efried wait, what?
15:32:00 efried max_unit changes?
15:32:24 cdent to account for multiple different source of one disk_gb in a cluster
15:32:26 mriedem cdent: is the vcenter driver reporting different max_unit DISK_GB from update_provider_tree during the claim?
15:32:28 cdent being presented as one disk
15:32:47 sean-k-mooney cdent: you mean for things like ceph
15:32:59 openstackgerrit Balazs Gibizer proposed openstack/nova master: Remove assumption of http error if consumer not exists https://review.opendev.org/669188
15:33:00 mriedem this is likely not libvirt/ceph
15:33:10 mriedem it's vcenter with shared storage provider or something?
15:33:22 cdent vcenter _faking_ shared stored providers
15:33:22 sean-k-mooney cdent: normally it should not need to change and should stay at the total capasity right
15:33:28 cdent sean-k-mooney: yes
15:33:30 efried oh, this isn't in libvirt, okay. Yeah, in libvirt max_unit == total and doesn't seem to change.
15:33:54 cdent but if you have N datastores in one cluster, no single request can be more than the free space on one of those datastores
15:34:23 efried okay, but as you say, fixing that is unrelated to the 409 bounce
15:34:24 cdent (this is why I've been eager for shared providers, as it could fix this (except that storage "policies" break all that)
15:34:27 sean-k-mooney cdent: well if you have over allcoation set to somthing other then 1 it could
15:34:30 mriedem sure, but why would max_unit change between initially reporting the inventory and then during the claim?
15:34:34 sean-k-mooney but ya i get you point
15:34:59 mriedem when the service starts up, it runs upt which should report the max_unit from the Nth datastore with the most amount of total disk right?
15:34:59 efried mriedem: the claim itself reduces the amount of free space on one of those disks
15:35:04 sean-k-mooney we physically could not allocate more then the free space on the data store
15:35:40 cdent let's assume for the moment that the max_unit hack is immutable as that's not really germane to the point, it's just the proximate cause
15:35:58 mriedem efried: couldn't we make that same argument for the libvirt driver then?
15:36:04 kashyap melwitt: efried: On that 'initenv' change (https://review.opendev.org/#/c/667976/2), doesn't it seem like a broken detection by 'cloud-init'?
15:36:09 gibi mriedem: done https://review.opendev.org/669188
15:36:11 cdent the point is that a failed _update_to_placement during instance claim (which is about inventory, traits and aggregates, not allocations) can fail an already suceeded instance
15:36:26 sean-k-mooney mriedem: i dont think we should be adjusting max_unit in either case
15:36:32 efried mriedem: In libvirt it's not an issue because we're only looking at one disk, so the (total - usage) alread ylimits.
15:36:35 kashyap [Ignore me, I'll add a comment in the change.]
15:36:37 mriedem cdent: what do you mean by "already succeeded?
15:36:48 efried kashyap: Note the subsequent patch and the bug it points to.
15:36:54 mriedem we do the claim before the driver.spawn right?
15:36:57 sean-k-mooney it should remian at total size and really on palcment to do the free calulation + allcoation raitio
15:37:08 cdent mriedem: let me look at my notes
15:37:19 sean-k-mooney or total size of largest data store i gues for vcenter
15:37:28 mriedem yeah driver.spawn is way late in the build process, it's like the last thing
15:37:40 mriedem first we claim, then we build volumes and networking, then we call driver.spawn
15:37:55 efried anyway, I think the point here is whether we can maybe do without that _update_to_placement at the end of instance_claim? What else is it doing?
15:38:28 mriedem instance_claim calls _update which was historically for updating info on the compute node record

Earlier   Later