| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-01-27 | |||
| 14:51:35 | jroll | what's the best way to talk to the placement service in devstack? just grab a keystone token and use curl? | |
| 14:51:50 | mriedem | jroll: yeah, althouh mlavalle has an osc plugin somewhere | |
| 14:52:17 | jroll | mriedem: cool, thanks | |
| 14:52:19 | cdent | jroll: a) probably yes b) i'm not sure if mlavalle's thing is "done" c) I use gabbi yaml files, run throuh gabbi-run | |
| 14:52:37 | johnthetubaguy | figleaf: cdent: I am struggling to understand this line: https://review.openstack.org/#/c/404472/23/nova/compute/resource_tracker.py@548 | |
| 14:52:41 | openstackgerrit | Artom Lifshitz proposed openstack/nova: Fix tag attribute disappearing in 2.33 and 2.37 https://review.openstack.org/424759 | |
| 14:52:41 | openstackgerrit | Artom Lifshitz proposed openstack/nova: Pass APIVersionRequest to extensions https://review.openstack.org/425876 | |
| 14:53:05 | johnthetubaguy | figleaf: cdent: I feel I am missing something big and obvious | |
| 14:53:10 | jroll | cdent: thanks | |
| 14:53:11 | cdent | jroll: in case you're curious: https://gist.github.com/cdent/a9590764fbc7402d450fa36df14f35e0 | |
| 14:53:42 | jroll | cdent: I need to poll for resources in our devstack plugin, curl seems like the right thing for now | |
| 14:53:52 | cdent | jroll: for that, yeah | |
| 14:54:18 | cdent | (although, just because we're almost on the topic, gabbi can poll) | |
| 14:54:48 | cdent | johnthetubaguy: the cleanup or that the exception happens? | |
| 14:55:14 | johnthetubaguy | cdent: I am wondering if we need cleanup in other cases, like when an ironic node is there, but there is no allocation | |
| 14:55:26 | johnthetubaguy | cdent: maybe the API just does that for us already? | |
| 14:55:56 | johnthetubaguy | ah, I think it does | |
| 14:56:04 | cdent | I don't think I'm sufficiently contextualized to really follow what you're saying? | |
| 14:56:43 | openstackgerrit | Artom Lifshitz proposed openstack/nova-specs: Fix tag attribute disappearing https://review.openstack.org/426030 | |
| 14:57:08 | johnthetubaguy | cdent: sorry, I was worried about us not always deleting the old style ironic inventories, but I didn't realise the API would do that for us when we put the inventories | |
| 14:57:39 | mriedem | superdan: cdent: jaypipes: bauzas: johnthetubaguy: melwitt: fyi i'm starting to make a list of things to do before rc1 https://etherpad.openstack.org/p/nova-ocata-rc1-todos | |
| 14:57:44 | johnthetubaguy | its just when we have allocations we need to do better | |
| 14:57:47 | johnthetubaguy | mriedem: perfect | |
| 14:57:51 | mriedem | jroll: https://github.com/miguellavalle/python-placementclient | |
| 14:58:39 | jroll | mriedem: cool, I'll wait til that's more "official" to depend on it in devstack, I think | |
| 14:59:43 | cdent | mriedem: do we want to be concerned with the traffic breadth from the resource tracker that I wrote about in: http://lists.openstack.org/pipermail/openstack-dev/2017-January/110953.html ? There are two bug fixes in progress related to that. | |
| 15:01:05 | mriedem | cdent: maybe? throw it into the etherpad, i haven't read that thread in detail yet | |
| 15:01:08 | mriedem | i know we're super chatty | |
| 15:01:19 | cdent | k, will do | |
| 15:01:19 | openstackgerrit | Jordan Pittier proposed openstack/nova: Fix unspecified bahavior on GET /servers/detail?tenant_id=X as admin https://review.openstack.org/426259 | |
| 15:01:50 | mriedem | the RT is chatty i mean when using the report client | |
| 15:02:03 | mriedem | i think i pointed something out about that in the resource classes series that's still up | |
| 15:02:17 | mriedem | but now i can't remember what it was, i think something about updating inventory twice when we didn't need to | |
| 15:03:06 | cdent | mriedem: yeah | |
| 15:03:22 | mriedem | but thanks for doing an audit, it's easy to not care about performance when you're just trying to get something functional | |
| 15:03:25 | cdent | I think in order to maintain our self-healing we still need to report more than we want | |
| 15:03:34 | cdent | s/want/could/ | |
| 15:03:49 | cdent | but at the moment it is very redundant, and some if it just because of old bugs | |
| 15:04:11 | cdent | mriedem: was worried about what would happen when this suddenly landed on a big deployment | |
| 15:04:17 | superdan | cdent: so one thing to consider, while we see how this goes, | |
| 15:04:33 | superdan | cdent: is are we likely to make more trouble by doing what we're doing than we have now.. | |
| 15:04:35 | superdan | because currently, | |
| 15:04:54 | superdan | we do this, but over rpc and into the same database as everything else, | |
| 15:05:17 | superdan | with placement, it's over http, probably a little more compact, and to a potentially separate database | |
| 15:05:29 | superdan | and probably more efficient horizontally-scaled under apache | |
| 15:05:46 | superdan | so we could assume it's at least a wash for ocata and see how it goes with some early testers | |
| 15:05:51 | cdent | certainly the horizontal scale capabilities are more gooderer | |
| 15:05:57 | mriedem | and we're still doing the rpc to single db thing | |
| 15:06:07 | superdan | you didn't identify anything that is worse than today right? | |
| 15:06:10 | mriedem | will be nice in the future when we can drop that load from rpc traffic | |
| 15:06:15 | cdent | so yeah, we are continuing the existing, and adding more on top | |
| 15:06:16 | superdan | yeah | |
| 15:06:48 | superdan | yeah, I guess I forgot we're doing both, but, theoretically to different places | |
| 15:06:49 | superdan | but yeah | |
| 15:07:04 | cdent | the thing with object comparison is something we should probably just fix as it is clear and simple | |
| 15:07:18 | cdent | I'm not sure how much impact it has on the http side, but maybe some on the rpc side | |
| 15:07:18 | mriedem | if only we had a performance working group or something | |
| 15:07:33 | cdent | the thing with tracked_instances is just weird | |
| 15:07:34 | superdan | I haven't looked at the two fixes | |
| 15:07:47 | cdent | as the way it works now is that it doesn't at all do what it says it is going to do | |
| 15:07:48 | superdan | I'll try to do that and make a judgment | |
| 15:08:22 | cdent | thanks, that would be awesome, just doing that analysis felt like I had a significant leap forward on understanding the resource tracker, but then right at the end I felt like I fell way backwards :( | |
| 15:08:47 | superdan | well, I'm not a RT expert by any means | |
| 15:08:47 | cdent | in the "why does this even work" sense | |
| 15:08:49 | superdan | probably jay would be better, but.. | |
| 15:09:08 | johnthetubaguy | figleaf: cdent: this ironic resource provider change, I could be missing something again, but I feel like we are missing something around updating new instance allocations? well possibly anyways. | |
| 15:09:59 | openstackgerrit | Stephen Finucane proposed openstack/nova: Combine obj_make_compatible tests https://review.openstack.org/426261 | |
| 15:09:59 | openstackgerrit | Stephen Finucane proposed openstack/nova: Fix backwards compatibility for InstanceNUMACell https://review.openstack.org/396184 | |
| 15:10:08 | cdent | johnthetubaguy: figleaf has gone to the docs, I think, and knows more about that than me, but I'll listen... | |
| 15:10:22 | mriedem | johnthetubaguy: replied in https://review.openstack.org/#/c/404472/ - | |
| 15:10:33 | mriedem | johnthetubaguy: none of that fancy new code is hit unless node.resource_class is set | |
| 15:10:38 | mriedem | which it's not today in our CI | |
| 15:10:41 | mriedem | which kind of blows | |
| 15:10:55 | mriedem | if you look at the logs for the ironic job on that patch it's still setting ram/cpu/disk | |
| 15:11:12 | mriedem | would be nice if we could have a one-off DNM patch to set node.resource_class and test that stuff | |
| 15:11:14 | mriedem | jroll: ^ | |
| 15:11:39 | mriedem | johnthetubaguy: and i think the ironic-specific stuff is mostly contained in the RT | |
| 15:11:55 | mriedem | which is why in earlier patches i had asked that long-term we subclass the ResourceTracker to do the ironic-specific bits | |
| 15:12:13 | mriedem | because i don't like the "if ironic" spaghetti code | |
| 15:12:28 | mriedem | i'll review the latest on that patch when i get back in an hour or so | |
| 15:13:56 | cdent | I may be misremembering, but I thought part of the rt+resosurce class stack was to change the resource tracker so that there is only one of them (for everything it tracks), not one for each node it cares about? | |
| 15:14:08 | cdent | that might be somewhere else... | |
| 15:14:37 | melwitt | cdent: fwiw the tracked_instances clear every update_available_resource period is known and intentional. it surprised me too when I first realized it. I don't recall why it's that way, I guess to prevent stale things from hanging around potentially | |
| 15:14:55 | cdent | melwitt: I took it out and no tests failed... | |
| 15:15:01 | cdent | which made me sad and confused | |
| 15:15:10 | mriedem | cdent: that's the same | |
| 15:15:14 | mriedem | but not what i'm talking about | |
| 15:15:17 | mriedem | bbiab | |
| 15:16:10 | melwitt | yeah, I don't think tests would fail from it. I have mentioned it to jaypipes in the past and he knew about it | |
| 15:17:24 | cdent | melwitt: hmm. well, thanks for providing some background. I wonder if it is worth revisiting since now it is spinning off redudant http requests? | |
| 15:17:32 | melwitt | bauzas: do you know anything about why tracked_instances is cleared each update_available_resource period? | |
| 15:19:45 | johnthetubaguy | cdent: so the bit I am stuck with, is where we make the new style ironic resource_class allocations | |
| 15:20:22 | melwitt | cdent: I think in general it's worth considering since it seemed excessive even in its original pre placement api form. just don't know what's involved in doing that or what gotchas there are | |
| 15:21:37 | jroll | mriedem: I'm working on devstack support right now, do you want a throwaway hack in the meantime? | |
| 15:21:51 | superdan | melwitt: cdent it's just to try to always stab the latest info into the database in case it's stale, right? | |
| 15:22:15 | melwitt | that's what I'm guessing | |
| 15:22:50 | cdent | if the node is authoritative for itself, and it keeps a cache of what it successfully wrote last time, how does staleness happen? | |
| 15:23:03 | cdent | (where last time could be 60s or 60 days ago) | |
| 15:23:24 | cdent | (mostly thinking about inventory here, not allocations) | |
| 15:23:48 | cdent | johnthetubaguy: i'm rereading the code, to bring it back into my brain | |