Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-12-08
20:43:19 fried_rice Now, I think I identified a change that's needed to what's already in gerrit for the get_provider_tree method to make sure we're pulling the latest traits & aggs for all "associated" providers to populate the ProviderTree. But I won't know until I figure out & fix whatever that PUT /traits deal is.
20:43:42 fried_rice (not PUT /traits -- PUT /resource_provider/%u/traits or whatever - you get the idea)
20:45:46 fried_rice Okay, next to-do: The report client method that diffs what virt driver returns and pushes those changes back to placement. It'll go between the top two patches in the series, I think. Working title for that method is update_from_provider_tree, per https://review.openstack.org/#/c/520246/12/nova/compute/resource_tracker.py@839
20:46:54 fried_rice The last major thing that affects the series is how we handle generations, which we started talking about some this morning.
20:48:35 fried_rice Honestly, I would need to relook at pretty much the whole report client with that in mind to figure out what we're gonna need to do about it.
20:49:49 fried_rice Absent the proposed get-everything-about-a-provider API (https://review.openstack.org/#/c/521875/) we would have to do stuff like:
20:49:51 cdent do you have a summary statement of the the “generations problem”?
20:50:44 fried_rice when we GET a provider's traits, that response payload contains the RP generation as known to the server. If that doesn't match what we have in the local ProviderTree, we should force-refresh everything we know about that provider.
20:51:36 cdent or more generally: any time we get a generation in a response, if it is not a match to us, refresh?
20:52:09 fried_rice yeah, I think that sums it up at least to the 80/20 point :)
20:52:53 fried_rice At the moment, my code that retrieves traits & aggs is ignoring the rp generation field in the response.
20:54:06 fried_rice In particular, what it's *not* doing is calling the set_*() method with the generation from that response, which would actually be the *wrong* thing to do I think.
20:54:44 fried_rice Because that would be updating the generation based on thing A that changed without taking into account that thing B (whose content we still have at the old generation level) might have also changed.
20:55:38 cdent quite
20:56:56 fried_rice Beyond all that, while restacking, there are some nice review comments to address from finucannot, giblet, figleaf, takashin...
20:57:44 fried_rice And writing up an executive summary, as you suggested in your email today, which could largely comprise text from the foregoing.
20:57:44 cdent If you had to identify the scariest-here-be-dragons aspect, what would it be?
20:58:23 fried_rice Scariest as in potential for largest amount of rework of stuff that's not already marked WIP? The generation thing.
20:59:43 cdent k
21:00:26 cdent Let me see if I can sort of translate some things to do:
21:01:06 cdent a) decode the last hour of the log to extract salient links, main points, action items
21:01:22 cdent b) attend to comments from reviews whilst restacking
21:01:57 cdent c) make the linked functional test, notably it’s ascii diagram “work” and make the code do that tree and allow that tree to change, and deal with generation changes
21:03:38 cdent eof
21:03:54 openstackgerrit Matt Riedemann proposed openstack/nova master: libvirt: mask InjectionInfo.admin_pass https://review.openstack.org/526772
21:03:58 cdent (since c) ended up being sort of d) and e) as well)
21:04:03 fried_rice Pretty much. WRT (c), the only part of the diagram that the code doesn't "do" right is the traits on *some* providers.
21:04:25 fried_rice Which may be solved by resolving that 400, but I actually think there's more to it. I'm leaving some code comments with more info.
21:08:09 fried_rice Done: https://review.openstack.org/#/c/521098/10
21:08:53 cdent cool
21:09:32 fried_rice Okay, I gotta run get the kids. Realistically, I'll be working a few scattered hours over the course of the next week, so drop me an email if you want to set up some more discussion time.
21:09:41 cdent I will endeavor to keep things moving. I suspect once I start breaking it it will make considerably more sense than now and happiness will abound and unicorns will leap and frolic etc
21:09:42 figleaf Any word from HR?
21:09:51 figleaf oops, wrong channel
21:09:56 fried_rice Yeah, don't tell HR I'm working.
21:10:26 fried_rice They're like, "you have HOW much vacation time left? Yeah, you gotta go."
21:10:29 figleaf Don't tell them I'm not :)
21:10:33 leakypipes figleaf: I've reported you to HR.
21:11:03 fried_rice leakypipes Dunno if you followed any/all of that, but would be interested to know whether you concur on the high points.
21:11:10 figleaf leakypipes: aw, c'mon - it was just a little hug
21:11:16 leakypipes figleaf: :)
21:11:23 leakypipes fried_rice: I was not paying attention, no...
21:11:53 fried_rice leakypipes And likewise, shoot me an email if we need to get together for an hour or whatever.
21:12:01 cdent leakypipes: the plan is to replace the scheduler, placement, the compute managers with etcd watches
21:12:16 leakypipes cdent: ah, cool. so k8s then.
21:12:18 figleaf leakypipes: running on Cassandra, of course
21:12:19 cdent before christmas
21:12:26 figleaf cya
21:12:33 leakypipes efried_cya_jan: cya!
21:12:35 cdent happy holidays or whathaveyou efried_cya_jan
21:12:46 efried_cya_jan Y'all too
21:13:21 cdent i assume it is glorious?
21:14:23 cdent actually, instead of etcd, we’ve got new technology that inverts cgroups to create global datacentre-wide share process spaces, and you can make global consensual RAM
21:14:26 leakypipes cdent: very similar to Percona/MySQL
21:14:31 leakypipes cdent: results-wise.
21:14:55 leakypipes cdent: gonna run the full bench tonight while nothing else is running on my puter.
21:15:43 leakypipes cdent: global consensual RAM sounds kinky.
21:16:11 cdent does a bit
21:16:19 cdent I will avoid a google search
21:19:34 cdent my eyes are done
21:19:42 cdent goodnight all, have good weekends
21:33:40 leakypipes and I'm signing off too... ciao folks.
21:59:31 ihrachys what could be the reason that I get this in placement-api (and compute service also spills something like that)
21:59:32 ihrachys Placement API returning an error response: Unable to update inventory for resource provider 2c677702-34b2-4e93-9124-a512bb97230f: Invalid inventory for 'MEMORY_MB' on resource provider '2c677702-34b2-4e93-9124-a512bb97230f'. The reserved value is greater than or equal to total.
22:00:05 ihrachys and the node can't boot instances, returning the usual No hosts found error. I suspect the placement api error is relevant.
22:00:54 mriedem ihrachys: check the reserved_host_memory_mb config option
22:01:39 mriedem the virt driver is saying there is x amount of total memory but the config is trying to reserve more than that for the host
22:01:58 mriedem there is probably something in the debug logs before that failure in nova-compute where it dumps what it's thinking the values are
22:04:02 ihrachys mriedem, reserved_host_memory_mb = 4096
22:05:08 ihrachys oh so it tries to lock all this memory?
22:05:22 ihrachys I don't have it on the node, though I added some swap to make it running
22:07:08 ihrachys maybe I should try to reduce that
22:07:49 melwitt yeah it will make it unavailable https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.reserved_host_disk_mb
22:08:19 melwitt sorry that was disk. memory is https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.reserved_host_memory_mb
22:10:29 mriedem ihrachys: this is the method that sets up the inventory data to send to placement https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L83
22:11:33 openstackgerrit melanie witt proposed openstack/nova master: Add API and nova-manage tests that use the NoopQuotaDriver https://review.openstack.org/526270
22:11:34 openstackgerrit melanie witt proposed openstack/nova master: Follow up on removing old-style quotas code https://review.openstack.org/524234
22:12:22 ihrachys mriedem, reduced to 512, restarted, and now it seems to have started the instance as expected. I guess the issue was because swap is not considered for the locking
22:12:32 ihrachys thanks mriedem for the pointer to the option!
22:14:24 mriedem yw
22:21:20 openstackgerrit melanie witt proposed openstack/nova master: DNM: Test websocketproxy with TLS https://review.openstack.org/513160
22:34:09 melwitt mriedem: I had to rebase the noop quotas test coverage bc of your quota force bug fix shenanigans https://review.openstack.org/#/c/526270
22:41:18 mriedem done
22:41:33 melwitt thanks \o/
22:49:04 mriedem superdan: were you going to update https://review.openstack.org/#/c/507638/ or save it for monday or are you ok with me updating it?
22:49:25 superdan mriedem: yeah I started locally but likely won't finish until monday
22:49:37 superdan that dependent one is in the gate atm too
22:50:24 mriedem alright
22:50:44 mriedem well i guess i'll just quit and spend quality time with my family
22:51:00 superdan good call
22:52:08 melwitt superdan: were you gonna remove one of the dupe tests in this "unmap compute nodes" one or should I do it? https://review.openstack.org/#/c/524641
22:52:38 superdan melwitt: yes I was
22:52:39 superdan so what you're both saying is, I suck, is that about right?
22:52:56 melwitt okay. no, of course not!
22:55:24 openstackgerrit Dan Smith proposed openstack/nova master: Unmap compute nodes when deleting host mapping https://review.openstack.org/524641
22:55:30 superdan melwitt: ^
22:56:21 melwitt saweet
22:57:12 melwitt mriedem: if you're still around ^
22:59:19 superdan I think you can slam it, he was +2 before right?
23:00:16 melwitt yeah he was. I'm conservative about doing that so thought I'd ask in case he's still there

Earlier   Later