| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-07-08 | |||
| 13:43:05 | bauzas | but then, we could have some races | |
| 13:43:20 | bauzas | plus, it could be super huge to get all the instances for CERN eg. | |
| 13:43:36 | bauzas | so, I wonder whether I should call the API for every instance | |
| 13:43:49 | bauzas | erh | |
| 13:43:53 | bauzas | I mean, checking the DB | |
| 13:46:03 | mriedem | i'd think you'd avoid races by checking if an instance has a task_state != None, and if looking at migrations, is the migration in progress | |
| 13:48:09 | mriedem | mnaser's script doesn't loop over instances, | |
| 13:48:27 | mriedem | it gets hypervisors and the servers on those hypervisors and compares to the resource provider inventory/allocation per hypervisor/server | |
| 13:50:13 | mriedem | i'd think it would be easiest to first determine what kinds of problems the audit command is going to try and find | |
| 13:50:29 | mriedem | and then figure out the best way to implement that | |
| 13:51:08 | mriedem | i think we know the two main issues are missing allocations (which heal_allocations should fix) and leaked / orphaned allocations, so i'd start by focusing on the latter issue | |
| 13:55:17 | mriedem | i think we could pretty easily identify leaked allocations by starting from the provider allocations and working backward - looking for servers or migration records with the allocation consumer id (since we don't have the type recorded yet) | |
| 13:59:30 | mriedem | so i'd probably start by getting all hypervisors (nodes/resource providers) and for each hypervisor, get the allocations - similar to what mnaser's script does | |
| 14:00:00 | mriedem | getting the hypervisors + servers in the same call like his script would also tell you if allocations on that provider was a server or not - and if not, it's either leaked or it's a migration allocation | |
| 14:00:09 | mriedem | so no i don't think you need to loop over all instances | |
| 14:00:18 | mriedem | maybe i should just write this thing :) | |
| 14:06:37 | bauzas | mriedem: I'm pretty sure I can do it | |
| 14:16:29 | openstackgerrit | François Palin proposed openstack/nova master: WIP - call volume detach even if terminate_connection fails https://review.opendev.org/669674 | |
| 14:19:01 | kashyap | mriedem: You're right ... indeed I didn't hear any responses on the list so far (http://lists.openstack.org/pipermail/openstack-discuss/2019-July/007521.html) | |
| 14:19:33 | kashyap | alex_xu: Hiya, when you're about, do you have a comment on the above thread? (And the change wich Matt Cced you on, which removes Intel CMT events) | |
| 14:20:10 | efried | kashyap: As predicted, my wife immediately stole the sleep book | |
| 14:20:20 | efried | and it is life-changing | |
| 14:20:41 | kashyap | efried: Hehe. Sharing is caring. I'm sure she will not hide her fascination with the book | |
| 14:20:56 | efried | I don't know why, but stuff I've been telling her for years for some reason is having more of an impact coming from a complete stranger | |
| 14:20:57 | kashyap | efried: Ah, she already finished it? | |
| 14:21:20 | efried | no, hasn't finished yet, but every chapter brings new insight. | |
| 14:21:26 | efried | she has bought a copy for her parents | |
| 14:21:31 | efried | and recommended it to a number of people. | |
| 14:21:36 | kashyap | :-) | |
| 14:21:49 | efried | I got a couple chapters in before she stole it, and I'm definitely sold | |
| 14:21:57 | kashyap | efried: I know, you must've become like a "broken record", and people zone out the moment broken reel start spinning :D | |
| 14:22:06 | kashyap | Sorry, just teasing. | |
| 14:22:35 | efried | though for me it's more fascination with the science than altering lifestyle, which was already pretty close to right sleep-wise. | |
| 14:23:07 | kashyap | Yeah, agreed. | |
| 14:23:46 | kashyap | efried: Also, I'd highly recommend to watch him give a tech talk. He's one of the most effective speakers I've ever seen. | |
| 14:24:01 | efried | noted | |
| 14:24:08 | kashyap | The way he pauses for the words to sink in, diction, is all worth observing. At least to me. | |
| 14:24:11 | kashyap | https://www.youtube.com/watch?v=aXflBZXAucQ | |
| 14:24:42 | kashyap | efried: If you click, look how quickly (within 5 seconds) he escalates the talk. :D | |
| 14:29:27 | prometheanfire | was pointed here for nova adding a cinder volume to a server without (the record/db at least) but not actually adding it to the running VM (client reports a 504 when trying to add the volume) | |
| 14:30:28 | prometheanfire | it shows up in the volumes_attached field, but can't remove any of them as they either do not exist or..... | |
| 14:30:32 | prometheanfire | Invalid volume: Invalid input received: Invalid volume: Unable to detach volume. Volume status must be 'in-use' and attach_status must be 'attached' to detach. (HTTP 400) (Request-ID: req-01d8a3c5-66f4-43a4-bac5-9f1104a292fe) (HTTP 400) (Request-ID: req-358132e4-38a1-493a-b87b-d135791bae2d) | |
| 15:02:46 | efried | kashyap: Would you mind taking a look at this please? | |
| 15:02:46 | efried | https://review.opendev.org/#/c/667975/ | |
| 15:02:46 | efried | It seems straightforward enough, but I'm not sure if there could e.g. be security problems, or if there's another accepted way to do this, etc. | |
| 15:03:09 | kashyap | efried: In a meeting; will look | |
| 15:03:15 | efried | no hurry | |
| 15:03:22 | efried | thanks | |
| 15:09:28 | stephenfin | lyarwood: Could you send this on its merry way? https://review.opendev.org/#/c/667355/ | |
| 15:11:04 | lyarwood | stephenfin: looking | |
| 15:11:48 | lyarwood | stephenfin: sorry I did mean to sort that out last week once the stein change landed | |
| 15:11:52 | lyarwood | stephenfin: done | |
| 15:13:07 | mriedem | gibi: comments in https://review.opendev.org/#/c/669188/ just so i'm sure i'm following it correctly (i know this patch was because i brought it up elsewhere) | |
| 15:13:57 | gibi | mriedem: looking.. | |
| 15:26:43 | gibi | mriedem: replied. I only added some extra explanation but I think we agree. | |
| 15:27:22 | gibi | mriedem: I can re-spin the patch to fix the test redundancies | |
| 15:28:46 | mriedem | yeah just fix the test and i'm +2 | |
| 15:28:57 | gibi | mriedem: OK, I will do it quickly | |
| 15:29:51 | cdent | mriedem, efried : I have a situation where under high load the _update_to_placement call made at the end of instance_claim in the compute manager can fail on ResourceProviderConflict all 4 retry attempts (because of nova-scheduler simultaneously making many allocations to the same resource provider). This fails the instance build even though the instance did build. | |
| 15:30:52 | efried | cdent: That is very interesting. | |
| 15:31:04 | cdent | I'm considering a hack to make instance_claim not call _update_to_placement and just letting the periodic job do that. The reason _update_to_placement is actually talking to placement is because inventory (DISK_GB max_unit) does change after many (most) instance creations | |
| 15:31:30 | cdent | ideally that max_unit thing wouldn't be in place but that's not really any immediate option | |
| 15:31:36 | efried | I'm guessing backing off the retry interval wouldn't be of help, because the problem is the timing between the GET and the PUT(s). | |
| 15:31:49 | sean-k-mooney | cdent: were we not looking at disabling the periodic job at one point | |
| 15:31:51 | efried | wait, what? | |
| 15:32:00 | efried | max_unit changes? | |
| 15:32:24 | cdent | to account for multiple different source of one disk_gb in a cluster | |
| 15:32:26 | mriedem | cdent: is the vcenter driver reporting different max_unit DISK_GB from update_provider_tree during the claim? | |
| 15:32:28 | cdent | being presented as one disk | |
| 15:32:47 | sean-k-mooney | cdent: you mean for things like ceph | |
| 15:32:59 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Remove assumption of http error if consumer not exists https://review.opendev.org/669188 | |
| 15:33:00 | mriedem | this is likely not libvirt/ceph | |
| 15:33:10 | mriedem | it's vcenter with shared storage provider or something? | |
| 15:33:22 | cdent | vcenter _faking_ shared stored providers | |
| 15:33:22 | sean-k-mooney | cdent: normally it should not need to change and should stay at the total capasity right | |
| 15:33:28 | cdent | sean-k-mooney: yes | |
| 15:33:30 | efried | oh, this isn't in libvirt, okay. Yeah, in libvirt max_unit == total and doesn't seem to change. | |
| 15:33:54 | cdent | but if you have N datastores in one cluster, no single request can be more than the free space on one of those datastores | |
| 15:34:23 | efried | okay, but as you say, fixing that is unrelated to the 409 bounce | |
| 15:34:24 | cdent | (this is why I've been eager for shared providers, as it could fix this (except that storage "policies" break all that) | |
| 15:34:27 | sean-k-mooney | cdent: well if you have over allcoation set to somthing other then 1 it could | |
| 15:34:30 | mriedem | sure, but why would max_unit change between initially reporting the inventory and then during the claim? | |
| 15:34:34 | sean-k-mooney | but ya i get you point | |
| 15:34:59 | mriedem | when the service starts up, it runs upt which should report the max_unit from the Nth datastore with the most amount of total disk right? | |
| 15:34:59 | efried | mriedem: the claim itself reduces the amount of free space on one of those disks | |
| 15:35:04 | sean-k-mooney | we physically could not allocate more then the free space on the data store | |
| 15:35:40 | cdent | let's assume for the moment that the max_unit hack is immutable as that's not really germane to the point, it's just the proximate cause | |
| 15:35:58 | mriedem | efried: couldn't we make that same argument for the libvirt driver then? | |
| 15:36:04 | kashyap | melwitt: efried: On that 'initenv' change (https://review.opendev.org/#/c/667976/2), doesn't it seem like a broken detection by 'cloud-init'? | |
| 15:36:09 | gibi | mriedem: done https://review.opendev.org/669188 | |
| 15:36:11 | cdent | the point is that a failed _update_to_placement during instance claim (which is about inventory, traits and aggregates, not allocations) can fail an already suceeded instance | |
| 15:36:26 | sean-k-mooney | mriedem: i dont think we should be adjusting max_unit in either case | |
| 15:36:32 | efried | mriedem: In libvirt it's not an issue because we're only looking at one disk, so the (total - usage) alread ylimits. | |
| 15:36:35 | kashyap | [Ignore me, I'll add a comment in the change.] | |
| 15:36:37 | mriedem | cdent: what do you mean by "already succeeded? | |
| 15:36:48 | efried | kashyap: Note the subsequent patch and the bug it points to. | |
| 15:36:54 | mriedem | we do the claim before the driver.spawn right? | |
| 15:36:57 | sean-k-mooney | it should remian at total size and really on palcment to do the free calulation + allcoation raitio | |
| 15:37:08 | cdent | mriedem: let me look at my notes | |
| 15:37:19 | sean-k-mooney | or total size of largest data store i gues for vcenter | |