Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-07-08
14:00:18 mriedem maybe i should just write this thing :)
14:06:37 bauzas mriedem: I'm pretty sure I can do it
14:16:29 openstackgerrit François Palin proposed openstack/nova master: WIP - call volume detach even if terminate_connection fails https://review.opendev.org/669674
14:19:01 kashyap mriedem: You're right ... indeed I didn't hear any responses on the list so far (http://lists.openstack.org/pipermail/openstack-discuss/2019-July/007521.html)
14:19:33 kashyap alex_xu: Hiya, when you're about, do you have a comment on the above thread? (And the change wich Matt Cced you on, which removes Intel CMT events)
14:20:10 efried kashyap: As predicted, my wife immediately stole the sleep book
14:20:20 efried and it is life-changing
14:20:41 kashyap efried: Hehe. Sharing is caring. I'm sure she will not hide her fascination with the book
14:20:56 efried I don't know why, but stuff I've been telling her for years for some reason is having more of an impact coming from a complete stranger
14:20:57 kashyap efried: Ah, she already finished it?
14:21:20 efried no, hasn't finished yet, but every chapter brings new insight.
14:21:26 efried she has bought a copy for her parents
14:21:31 efried and recommended it to a number of people.
14:21:36 kashyap :-)
14:21:49 efried I got a couple chapters in before she stole it, and I'm definitely sold
14:21:57 kashyap efried: I know, you must've become like a "broken record", and people zone out the moment broken reel start spinning :D
14:22:06 kashyap Sorry, just teasing.
14:22:35 efried though for me it's more fascination with the science than altering lifestyle, which was already pretty close to right sleep-wise.
14:23:07 kashyap Yeah, agreed.
14:23:46 kashyap efried: Also, I'd highly recommend to watch him give a tech talk. He's one of the most effective speakers I've ever seen.
14:24:01 efried noted
14:24:08 kashyap The way he pauses for the words to sink in, diction, is all worth observing. At least to me.
14:24:11 kashyap https://www.youtube.com/watch?v=aXflBZXAucQ
14:24:42 kashyap efried: If you click, look how quickly (within 5 seconds) he escalates the talk. :D
14:29:27 prometheanfire was pointed here for nova adding a cinder volume to a server without (the record/db at least) but not actually adding it to the running VM (client reports a 504 when trying to add the volume)
14:30:28 prometheanfire it shows up in the volumes_attached field, but can't remove any of them as they either do not exist or.....
14:30:32 prometheanfire Invalid volume: Invalid input received: Invalid volume: Unable to detach volume. Volume status must be 'in-use' and attach_status must be 'attached' to detach. (HTTP 400) (Request-ID: req-01d8a3c5-66f4-43a4-bac5-9f1104a292fe) (HTTP 400) (Request-ID: req-358132e4-38a1-493a-b87b-d135791bae2d)
15:02:46 efried kashyap: Would you mind taking a look at this please?
15:02:46 efried https://review.opendev.org/#/c/667975/
15:02:46 efried It seems straightforward enough, but I'm not sure if there could e.g. be security problems, or if there's another accepted way to do this, etc.
15:03:09 kashyap efried: In a meeting; will look
15:03:15 efried no hurry
15:03:22 efried thanks
15:09:28 stephenfin lyarwood: Could you send this on its merry way? https://review.opendev.org/#/c/667355/
15:11:04 lyarwood stephenfin: looking
15:11:48 lyarwood stephenfin: sorry I did mean to sort that out last week once the stein change landed
15:11:52 lyarwood stephenfin: done
15:13:07 mriedem gibi: comments in https://review.opendev.org/#/c/669188/ just so i'm sure i'm following it correctly (i know this patch was because i brought it up elsewhere)
15:13:57 gibi mriedem: looking..
15:26:43 gibi mriedem: replied. I only added some extra explanation but I think we agree.
15:27:22 gibi mriedem: I can re-spin the patch to fix the test redundancies
15:28:46 mriedem yeah just fix the test and i'm +2
15:28:57 gibi mriedem: OK, I will do it quickly
15:29:51 cdent mriedem, efried : I have a situation where under high load the _update_to_placement call made at the end of instance_claim in the compute manager can fail on ResourceProviderConflict all 4 retry attempts (because of nova-scheduler simultaneously making many allocations to the same resource provider). This fails the instance build even though the instance did build.
15:30:52 efried cdent: That is very interesting.
15:31:04 cdent I'm considering a hack to make instance_claim not call _update_to_placement and just letting the periodic job do that. The reason _update_to_placement is actually talking to placement is because inventory (DISK_GB max_unit) does change after many (most) instance creations
15:31:30 cdent ideally that max_unit thing wouldn't be in place but that's not really any immediate option
15:31:36 efried I'm guessing backing off the retry interval wouldn't be of help, because the problem is the timing between the GET and the PUT(s).
15:31:49 sean-k-mooney cdent: were we not looking at disabling the periodic job at one point
15:31:51 efried wait, what?
15:32:00 efried max_unit changes?
15:32:24 cdent to account for multiple different source of one disk_gb in a cluster
15:32:26 mriedem cdent: is the vcenter driver reporting different max_unit DISK_GB from update_provider_tree during the claim?
15:32:28 cdent being presented as one disk
15:32:47 sean-k-mooney cdent: you mean for things like ceph
15:32:59 openstackgerrit Balazs Gibizer proposed openstack/nova master: Remove assumption of http error if consumer not exists https://review.opendev.org/669188
15:33:00 mriedem this is likely not libvirt/ceph
15:33:10 mriedem it's vcenter with shared storage provider or something?
15:33:22 cdent vcenter _faking_ shared stored providers
15:33:22 sean-k-mooney cdent: normally it should not need to change and should stay at the total capasity right
15:33:28 cdent sean-k-mooney: yes
15:33:30 efried oh, this isn't in libvirt, okay. Yeah, in libvirt max_unit == total and doesn't seem to change.
15:33:54 cdent but if you have N datastores in one cluster, no single request can be more than the free space on one of those datastores
15:34:23 efried okay, but as you say, fixing that is unrelated to the 409 bounce
15:34:24 cdent (this is why I've been eager for shared providers, as it could fix this (except that storage "policies" break all that)
15:34:27 sean-k-mooney cdent: well if you have over allcoation set to somthing other then 1 it could
15:34:30 mriedem sure, but why would max_unit change between initially reporting the inventory and then during the claim?
15:34:34 sean-k-mooney but ya i get you point
15:34:59 mriedem when the service starts up, it runs upt which should report the max_unit from the Nth datastore with the most amount of total disk right?
15:34:59 efried mriedem: the claim itself reduces the amount of free space on one of those disks
15:35:04 sean-k-mooney we physically could not allocate more then the free space on the data store
15:35:40 cdent let's assume for the moment that the max_unit hack is immutable as that's not really germane to the point, it's just the proximate cause
15:35:58 mriedem efried: couldn't we make that same argument for the libvirt driver then?
15:36:04 kashyap melwitt: efried: On that 'initenv' change (https://review.opendev.org/#/c/667976/2), doesn't it seem like a broken detection by 'cloud-init'?
15:36:09 gibi mriedem: done https://review.opendev.org/669188
15:36:11 cdent the point is that a failed _update_to_placement during instance claim (which is about inventory, traits and aggregates, not allocations) can fail an already suceeded instance
15:36:26 sean-k-mooney mriedem: i dont think we should be adjusting max_unit in either case
15:36:32 efried mriedem: In libvirt it's not an issue because we're only looking at one disk, so the (total - usage) alread ylimits.
15:36:35 kashyap [Ignore me, I'll add a comment in the change.]
15:36:37 mriedem cdent: what do you mean by "already succeeded?
15:36:48 efried kashyap: Note the subsequent patch and the bug it points to.
15:36:54 mriedem we do the claim before the driver.spawn right?
15:36:57 sean-k-mooney it should remian at total size and really on palcment to do the free calulation + allcoation raitio
15:37:08 cdent mriedem: let me look at my notes
15:37:19 sean-k-mooney or total size of largest data store i gues for vcenter
15:37:28 mriedem yeah driver.spawn is way late in the build process, it's like the last thing
15:37:40 mriedem first we claim, then we build volumes and networking, then we call driver.spawn
15:37:55 efried anyway, I think the point here is whether we can maybe do without that _update_to_placement at the end of instance_claim? What else is it doing?
15:38:28 mriedem instance_claim calls _update which was historically for updating info on the compute node record
15:38:30 cdent mriedem: yeah, sorry, not that it did work, but _would_ work
15:38:33 mriedem for things like free_vcpus and whatnot
15:38:54 mriedem yeah i was going to say... https://github.com/openstack/nova/blob/86524773b8cd3a52c98409c7ca183b4e1873e2b8/nova/compute/manager.py#L2223 :)
15:38:58 openstackgerrit Sundar Nadathur proposed openstack/nova master: ksa auth conf and client for cyborg access https://review.opendev.org/631242
15:38:59 openstackgerrit Sundar Nadathur proposed openstack/nova master: WIP: Add Cyborg device profile groups to request spec. https://review.opendev.org/631243
15:38:59 openstackgerrit Sundar Nadathur proposed openstack/nova master: WIP: Create and bind Cyborg ARQs. https://review.opendev.org/631244
15:39:00 openstackgerrit Sundar Nadathur proposed openstack/nova master: WIP: Get resolved Cyborg ARQs and add PCI BDFs to VM's domain XML. https://review.opendev.org/631245
15:39:50 mriedem so i think what's being asked is a kwarg on _update to say whether or not to call _update_to_placement
15:39:57 efried though the fact that it's doing something in this case is an argument in favor of keeping it. Kinda wondering what breaks if we don't have that. Presumably it would be possible for the disk allocation to be unsatisfiable late in the spawn process.
15:40:00 mriedem which in the startup and update_available_resource periodic would be yes,
15:40:06 mriedem but in instance_claim you're saying no

Earlier   Later