Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-01-27
14:52:41 openstackgerrit Artom Lifshitz proposed openstack/nova: Pass APIVersionRequest to extensions https://review.openstack.org/425876
14:53:05 johnthetubaguy figleaf: cdent: I feel I am missing something big and obvious
14:53:10 jroll cdent: thanks
14:53:11 cdent jroll: in case you're curious: https://gist.github.com/cdent/a9590764fbc7402d450fa36df14f35e0
14:53:42 jroll cdent: I need to poll for resources in our devstack plugin, curl seems like the right thing for now
14:53:52 cdent jroll: for that, yeah
14:54:18 cdent (although, just because we're almost on the topic, gabbi can poll)
14:54:48 cdent johnthetubaguy: the cleanup or that the exception happens?
14:55:14 johnthetubaguy cdent: I am wondering if we need cleanup in other cases, like when an ironic node is there, but there is no allocation
14:55:26 johnthetubaguy cdent: maybe the API just does that for us already?
14:55:56 johnthetubaguy ah, I think it does
14:56:04 cdent I don't think I'm sufficiently contextualized to really follow what you're saying?
14:56:43 openstackgerrit Artom Lifshitz proposed openstack/nova-specs: Fix tag attribute disappearing https://review.openstack.org/426030
14:57:08 johnthetubaguy cdent: sorry, I was worried about us not always deleting the old style ironic inventories, but I didn't realise the API would do that for us when we put the inventories
14:57:39 mriedem superdan: cdent: jaypipes: bauzas: johnthetubaguy: melwitt: fyi i'm starting to make a list of things to do before rc1 https://etherpad.openstack.org/p/nova-ocata-rc1-todos
14:57:44 johnthetubaguy its just when we have allocations we need to do better
14:57:47 johnthetubaguy mriedem: perfect
14:57:51 mriedem jroll: https://github.com/miguellavalle/python-placementclient
14:58:39 jroll mriedem: cool, I'll wait til that's more "official" to depend on it in devstack, I think
14:59:43 cdent mriedem: do we want to be concerned with the traffic breadth from the resource tracker that I wrote about in: http://lists.openstack.org/pipermail/openstack-dev/2017-January/110953.html ? There are two bug fixes in progress related to that.
15:01:05 mriedem cdent: maybe? throw it into the etherpad, i haven't read that thread in detail yet
15:01:08 mriedem i know we're super chatty
15:01:19 cdent k, will do
15:01:19 openstackgerrit Jordan Pittier proposed openstack/nova: Fix unspecified bahavior on GET /servers/detail?tenant_id=X as admin https://review.openstack.org/426259
15:01:50 mriedem the RT is chatty i mean when using the report client
15:02:03 mriedem i think i pointed something out about that in the resource classes series that's still up
15:02:17 mriedem but now i can't remember what it was, i think something about updating inventory twice when we didn't need to
15:03:06 cdent mriedem: yeah
15:03:22 mriedem but thanks for doing an audit, it's easy to not care about performance when you're just trying to get something functional
15:03:25 cdent I think in order to maintain our self-healing we still need to report more than we want
15:03:34 cdent s/want/could/
15:03:49 cdent but at the moment it is very redundant, and some if it just because of old bugs
15:04:11 cdent mriedem: was worried about what would happen when this suddenly landed on a big deployment
15:04:17 superdan cdent: so one thing to consider, while we see how this goes,
15:04:33 superdan cdent: is are we likely to make more trouble by doing what we're doing than we have now..
15:04:35 superdan because currently,
15:04:54 superdan we do this, but over rpc and into the same database as everything else,
15:05:17 superdan with placement, it's over http, probably a little more compact, and to a potentially separate database
15:05:29 superdan and probably more efficient horizontally-scaled under apache
15:05:46 superdan so we could assume it's at least a wash for ocata and see how it goes with some early testers
15:05:51 cdent certainly the horizontal scale capabilities are more gooderer
15:05:57 mriedem and we're still doing the rpc to single db thing
15:06:07 superdan you didn't identify anything that is worse than today right?
15:06:10 mriedem will be nice in the future when we can drop that load from rpc traffic
15:06:15 cdent so yeah, we are continuing the existing, and adding more on top
15:06:16 superdan yeah
15:06:48 superdan yeah, I guess I forgot we're doing both, but, theoretically to different places
15:06:49 superdan but yeah
15:07:04 cdent the thing with object comparison is something we should probably just fix as it is clear and simple
15:07:18 cdent I'm not sure how much impact it has on the http side, but maybe some on the rpc side
15:07:18 mriedem if only we had a performance working group or something
15:07:33 cdent the thing with tracked_instances is just weird
15:07:34 superdan I haven't looked at the two fixes
15:07:47 cdent as the way it works now is that it doesn't at all do what it says it is going to do
15:07:48 superdan I'll try to do that and make a judgment
15:08:22 cdent thanks, that would be awesome, just doing that analysis felt like I had a significant leap forward on understanding the resource tracker, but then right at the end I felt like I fell way backwards :(
15:08:47 superdan well, I'm not a RT expert by any means
15:08:47 cdent in the "why does this even work" sense
15:08:49 superdan probably jay would be better, but..
15:09:08 johnthetubaguy figleaf: cdent: this ironic resource provider change, I could be missing something again, but I feel like we are missing something around updating new instance allocations? well possibly anyways.
15:09:59 openstackgerrit Stephen Finucane proposed openstack/nova: Combine obj_make_compatible tests https://review.openstack.org/426261
15:09:59 openstackgerrit Stephen Finucane proposed openstack/nova: Fix backwards compatibility for InstanceNUMACell https://review.openstack.org/396184
15:10:08 cdent johnthetubaguy: figleaf has gone to the docs, I think, and knows more about that than me, but I'll listen...
15:10:22 mriedem johnthetubaguy: replied in https://review.openstack.org/#/c/404472/ -
15:10:33 mriedem johnthetubaguy: none of that fancy new code is hit unless node.resource_class is set
15:10:38 mriedem which it's not today in our CI
15:10:41 mriedem which kind of blows
15:10:55 mriedem if you look at the logs for the ironic job on that patch it's still setting ram/cpu/disk
15:11:12 mriedem would be nice if we could have a one-off DNM patch to set node.resource_class and test that stuff
15:11:14 mriedem jroll: ^
15:11:39 mriedem johnthetubaguy: and i think the ironic-specific stuff is mostly contained in the RT
15:11:55 mriedem which is why in earlier patches i had asked that long-term we subclass the ResourceTracker to do the ironic-specific bits
15:12:13 mriedem because i don't like the "if ironic" spaghetti code
15:12:28 mriedem i'll review the latest on that patch when i get back in an hour or so
15:13:56 cdent I may be misremembering, but I thought part of the rt+resosurce class stack was to change the resource tracker so that there is only one of them (for everything it tracks), not one for each node it cares about?
15:14:08 cdent that might be somewhere else...
15:14:37 melwitt cdent: fwiw the tracked_instances clear every update_available_resource period is known and intentional. it surprised me too when I first realized it. I don't recall why it's that way, I guess to prevent stale things from hanging around potentially
15:14:55 cdent melwitt: I took it out and no tests failed...
15:15:01 cdent which made me sad and confused
15:15:10 mriedem cdent: that's the same
15:15:14 mriedem but not what i'm talking about
15:15:17 mriedem bbiab
15:16:10 melwitt yeah, I don't think tests would fail from it. I have mentioned it to jaypipes in the past and he knew about it
15:17:24 cdent melwitt: hmm. well, thanks for providing some background. I wonder if it is worth revisiting since now it is spinning off redudant http requests?
15:17:32 melwitt bauzas: do you know anything about why tracked_instances is cleared each update_available_resource period?
15:19:45 johnthetubaguy cdent: so the bit I am stuck with, is where we make the new style ironic resource_class allocations
15:20:22 melwitt cdent: I think in general it's worth considering since it seemed excessive even in its original pre placement api form. just don't know what's involved in doing that or what gotchas there are
15:21:37 jroll mriedem: I'm working on devstack support right now, do you want a throwaway hack in the meantime?
15:21:51 superdan melwitt: cdent it's just to try to always stab the latest info into the database in case it's stale, right?
15:22:15 melwitt that's what I'm guessing
15:22:50 cdent if the node is authoritative for itself, and it keeps a cache of what it successfully wrote last time, how does staleness happen?
15:23:03 cdent (where last time could be 60s or 60 days ago)
15:23:24 cdent (mostly thinking about inventory here, not allocations)
15:23:48 cdent johnthetubaguy: i'm rereading the code, to bring it back into my brain
15:26:09 cfriesen cdent: refreshing tracked_instances would allow it to recover from errors where it missed an event that would cause the instance to be added/removed from the list. But it should be possible to handle that in a different way.
15:26:25 johnthetubaguy cdent: you probably want to ignore any changes to the "service" field in the compute_node as well, as service will keep getting updated every DB service update interval, unsure if thats related (re https://review.openstack.org/#/c/424305)
15:27:47 johnthetubaguy cdent: ignore that
15:29:23 cdent cfriesen: is missing events of that nature a thing happens a) often, b) sometimes, c) rarely?
15:29:23 melwitt cdent: looking back at the code to refresh my memory ... I think without the periodic clear, the only thing that removes instances from tracked_instances is the vm_state DELETED or SHELVED_OFFLOADED and if for some reason (errors) an instance doesn't get its state saved that way, it would never be removed
15:32:29 cfriesen cdent: I would expect rarely

Earlier   Later