| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-12-08 | |||
| 20:37:34 | fried_rice | One would hope such changes to be rare to the point of almost-never, because virt doesn't manage allocations. | |
| 20:37:53 | fried_rice | So we're talking about if someone hotplugs storage or CoDs a bank of CPUs or something. | |
| 20:37:58 | fried_rice | anyway... | |
| 20:38:23 | cdent | (if virt could manage allocations, that would be cooler, for hybrid environments) | |
| 20:38:32 | fried_rice | As I was adding the code for that TODO at the bottom, I found that the traits were in fact not showing up for certain providers. | |
| 20:38:45 | fried_rice | Which turns out to be because the traits PUT is 400ing. | |
| 20:39:01 | cdent | but nobody can tell because of None? | |
| 20:39:05 | fried_rice | No | |
| 20:39:12 | fried_rice | Because we don't have a report client method for creating traits. | |
| 20:39:27 | fried_rice | or aggregates | |
| 20:39:36 | fried_rice | so I'm doing that directly in the test, and not checking the result. | |
| 20:39:58 | fried_rice | So that's one (or maybe two) still-needed patch(es). | |
| 20:40:47 | fried_rice | While we're talking about None, there's another patch (possibly more than one) needed, which could actually be done outside of this series, to make sure we're raising where we should. | |
| 20:41:13 | fried_rice | Per discussion yesterday, we only need to do it for methods where we don't always check the return | |
| 20:41:30 | fried_rice | I identified that at least the one that gets inventories qualifies. | |
| 20:41:34 | fried_rice | There may be others. | |
| 20:43:19 | fried_rice | Now, I think I identified a change that's needed to what's already in gerrit for the get_provider_tree method to make sure we're pulling the latest traits & aggs for all "associated" providers to populate the ProviderTree. But I won't know until I figure out & fix whatever that PUT /traits deal is. | |
| 20:43:42 | fried_rice | (not PUT /traits -- PUT /resource_provider/%u/traits or whatever - you get the idea) | |
| 20:45:46 | fried_rice | Okay, next to-do: The report client method that diffs what virt driver returns and pushes those changes back to placement. It'll go between the top two patches in the series, I think. Working title for that method is update_from_provider_tree, per https://review.openstack.org/#/c/520246/12/nova/compute/resource_tracker.py@839 | |
| 20:46:54 | fried_rice | The last major thing that affects the series is how we handle generations, which we started talking about some this morning. | |
| 20:48:35 | fried_rice | Honestly, I would need to relook at pretty much the whole report client with that in mind to figure out what we're gonna need to do about it. | |
| 20:49:49 | fried_rice | Absent the proposed get-everything-about-a-provider API (https://review.openstack.org/#/c/521875/) we would have to do stuff like: | |
| 20:49:51 | cdent | do you have a summary statement of the the “generations problem”? | |
| 20:50:44 | fried_rice | when we GET a provider's traits, that response payload contains the RP generation as known to the server. If that doesn't match what we have in the local ProviderTree, we should force-refresh everything we know about that provider. | |
| 20:51:36 | cdent | or more generally: any time we get a generation in a response, if it is not a match to us, refresh? | |
| 20:52:09 | fried_rice | yeah, I think that sums it up at least to the 80/20 point :) | |
| 20:52:53 | fried_rice | At the moment, my code that retrieves traits & aggs is ignoring the rp generation field in the response. | |
| 20:54:06 | fried_rice | In particular, what it's *not* doing is calling the set_*() method with the generation from that response, which would actually be the *wrong* thing to do I think. | |
| 20:54:44 | fried_rice | Because that would be updating the generation based on thing A that changed without taking into account that thing B (whose content we still have at the old generation level) might have also changed. | |
| 20:55:38 | cdent | quite | |
| 20:56:56 | fried_rice | Beyond all that, while restacking, there are some nice review comments to address from finucannot, giblet, figleaf, takashin... | |
| 20:57:44 | cdent | If you had to identify the scariest-here-be-dragons aspect, what would it be? | |
| 20:57:44 | fried_rice | And writing up an executive summary, as you suggested in your email today, which could largely comprise text from the foregoing. | |
| 20:58:23 | fried_rice | Scariest as in potential for largest amount of rework of stuff that's not already marked WIP? The generation thing. | |
| 20:59:43 | cdent | k | |
| 21:00:26 | cdent | Let me see if I can sort of translate some things to do: | |
| 21:01:06 | cdent | a) decode the last hour of the log to extract salient links, main points, action items | |
| 21:01:22 | cdent | b) attend to comments from reviews whilst restacking | |
| 21:01:57 | cdent | c) make the linked functional test, notably it’s ascii diagram “work” and make the code do that tree and allow that tree to change, and deal with generation changes | |
| 21:03:38 | cdent | eof | |
| 21:03:54 | openstackgerrit | Matt Riedemann proposed openstack/nova master: libvirt: mask InjectionInfo.admin_pass https://review.openstack.org/526772 | |
| 21:03:58 | cdent | (since c) ended up being sort of d) and e) as well) | |
| 21:04:03 | fried_rice | Pretty much. WRT (c), the only part of the diagram that the code doesn't "do" right is the traits on *some* providers. | |
| 21:04:25 | fried_rice | Which may be solved by resolving that 400, but I actually think there's more to it. I'm leaving some code comments with more info. | |
| 21:08:09 | fried_rice | Done: https://review.openstack.org/#/c/521098/10 | |
| 21:08:53 | cdent | cool | |
| 21:09:32 | fried_rice | Okay, I gotta run get the kids. Realistically, I'll be working a few scattered hours over the course of the next week, so drop me an email if you want to set up some more discussion time. | |
| 21:09:41 | cdent | I will endeavor to keep things moving. I suspect once I start breaking it it will make considerably more sense than now and happiness will abound and unicorns will leap and frolic etc | |
| 21:09:42 | figleaf | Any word from HR? | |
| 21:09:51 | figleaf | oops, wrong channel | |
| 21:09:56 | fried_rice | Yeah, don't tell HR I'm working. | |
| 21:10:26 | fried_rice | They're like, "you have HOW much vacation time left? Yeah, you gotta go." | |
| 21:10:29 | figleaf | Don't tell them I'm not :) | |
| 21:10:33 | leakypipes | figleaf: I've reported you to HR. | |
| 21:11:03 | fried_rice | leakypipes Dunno if you followed any/all of that, but would be interested to know whether you concur on the high points. | |
| 21:11:10 | figleaf | leakypipes: aw, c'mon - it was just a little hug | |
| 21:11:16 | leakypipes | figleaf: :) | |
| 21:11:23 | leakypipes | fried_rice: I was not paying attention, no... | |
| 21:11:53 | fried_rice | leakypipes And likewise, shoot me an email if we need to get together for an hour or whatever. | |
| 21:12:01 | cdent | leakypipes: the plan is to replace the scheduler, placement, the compute managers with etcd watches | |
| 21:12:16 | leakypipes | cdent: ah, cool. so k8s then. | |
| 21:12:18 | figleaf | leakypipes: running on Cassandra, of course | |
| 21:12:19 | cdent | before christmas | |
| 21:12:26 | figleaf | cya | |
| 21:12:33 | leakypipes | efried_cya_jan: cya! | |
| 21:12:35 | cdent | happy holidays or whathaveyou efried_cya_jan | |
| 21:12:46 | efried_cya_jan | Y'all too | |
| 21:13:21 | cdent | i assume it is glorious? | |
| 21:14:23 | cdent | actually, instead of etcd, we’ve got new technology that inverts cgroups to create global datacentre-wide share process spaces, and you can make global consensual RAM | |
| 21:14:26 | leakypipes | cdent: very similar to Percona/MySQL | |
| 21:14:31 | leakypipes | cdent: results-wise. | |
| 21:14:55 | leakypipes | cdent: gonna run the full bench tonight while nothing else is running on my puter. | |
| 21:15:43 | leakypipes | cdent: global consensual RAM sounds kinky. | |
| 21:16:11 | cdent | does a bit | |
| 21:16:19 | cdent | I will avoid a google search | |
| 21:19:34 | cdent | my eyes are done | |
| 21:19:42 | cdent | goodnight all, have good weekends | |
| 21:33:40 | leakypipes | and I'm signing off too... ciao folks. | |
| 21:59:31 | ihrachys | what could be the reason that I get this in placement-api (and compute service also spills something like that) | |
| 21:59:32 | ihrachys | Placement API returning an error response: Unable to update inventory for resource provider 2c677702-34b2-4e93-9124-a512bb97230f: Invalid inventory for 'MEMORY_MB' on resource provider '2c677702-34b2-4e93-9124-a512bb97230f'. The reserved value is greater than or equal to total. | |
| 22:00:05 | ihrachys | and the node can't boot instances, returning the usual No hosts found error. I suspect the placement api error is relevant. | |
| 22:00:54 | mriedem | ihrachys: check the reserved_host_memory_mb config option | |
| 22:01:39 | mriedem | the virt driver is saying there is x amount of total memory but the config is trying to reserve more than that for the host | |
| 22:01:58 | mriedem | there is probably something in the debug logs before that failure in nova-compute where it dumps what it's thinking the values are | |
| 22:04:02 | ihrachys | mriedem, reserved_host_memory_mb = 4096 | |
| 22:05:08 | ihrachys | oh so it tries to lock all this memory? | |
| 22:05:22 | ihrachys | I don't have it on the node, though I added some swap to make it running | |
| 22:07:08 | ihrachys | maybe I should try to reduce that | |
| 22:07:49 | melwitt | yeah it will make it unavailable https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.reserved_host_disk_mb | |
| 22:08:19 | melwitt | sorry that was disk. memory is https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.reserved_host_memory_mb | |
| 22:10:29 | mriedem | ihrachys: this is the method that sets up the inventory data to send to placement https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L83 | |
| 22:11:33 | openstackgerrit | melanie witt proposed openstack/nova master: Add API and nova-manage tests that use the NoopQuotaDriver https://review.openstack.org/526270 | |
| 22:11:34 | openstackgerrit | melanie witt proposed openstack/nova master: Follow up on removing old-style quotas code https://review.openstack.org/524234 | |
| 22:12:22 | ihrachys | mriedem, reduced to 512, restarted, and now it seems to have started the instance as expected. I guess the issue was because swap is not considered for the locking | |
| 22:12:32 | ihrachys | thanks mriedem for the pointer to the option! | |
| 22:14:24 | mriedem | yw | |
| 22:21:20 | openstackgerrit | melanie witt proposed openstack/nova master: DNM: Test websocketproxy with TLS https://review.openstack.org/513160 | |
| 22:34:09 | melwitt | mriedem: I had to rebase the noop quotas test coverage bc of your quota force bug fix shenanigans https://review.openstack.org/#/c/526270 | |
| 22:41:18 | mriedem | done | |
| 22:41:33 | melwitt | thanks \o/ | |