| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-01-27 | |||
| 14:40:42 | BlackDex | i have configured memcached, but nothing is getting into memcached, i can connect to memcached from the nova-controle nodes | |
| 14:47:20 | superdan | that scheduler change failed on the old side due to volumes and neutron | |
| 14:47:21 | cdent | johnthetubaguy, dansmith : looks like the latest multinode grenade failed on timeouts this time | |
| 14:47:25 | cdent | jinx | |
| 14:47:27 | superdan | heh | |
| 14:48:16 | johnthetubaguy | is that a real something is broken timeout, or a you hit an unlucky node timeout? | |
| 14:49:18 | johnthetubaguy | cdent: did we get the correct min_version back now? | |
| 14:49:28 | cdent | no logs for the new side | |
| 14:49:53 | superdan | johnthetubaguy: we didn't get far enough to look | |
| 14:50:16 | johnthetubaguy | I was worried you would say that | |
| 14:50:36 | johnthetubaguy | heh | |
| 14:50:43 | mriedem | superdan: congrats https://review.openstack.org/319379 | |
| 14:51:17 | mriedem | now we just cleanup the release notes for cells v2 and maybe upgrade docs and we're good for ocata i think | |
| 14:51:35 | jroll | what's the best way to talk to the placement service in devstack? just grab a keystone token and use curl? | |
| 14:51:50 | mriedem | jroll: yeah, althouh mlavalle has an osc plugin somewhere | |
| 14:52:17 | jroll | mriedem: cool, thanks | |
| 14:52:19 | cdent | jroll: a) probably yes b) i'm not sure if mlavalle's thing is "done" c) I use gabbi yaml files, run throuh gabbi-run | |
| 14:52:37 | johnthetubaguy | figleaf: cdent: I am struggling to understand this line: https://review.openstack.org/#/c/404472/23/nova/compute/resource_tracker.py@548 | |
| 14:52:41 | openstackgerrit | Artom Lifshitz proposed openstack/nova: Fix tag attribute disappearing in 2.33 and 2.37 https://review.openstack.org/424759 | |
| 14:52:41 | openstackgerrit | Artom Lifshitz proposed openstack/nova: Pass APIVersionRequest to extensions https://review.openstack.org/425876 | |
| 14:53:05 | johnthetubaguy | figleaf: cdent: I feel I am missing something big and obvious | |
| 14:53:10 | jroll | cdent: thanks | |
| 14:53:11 | cdent | jroll: in case you're curious: https://gist.github.com/cdent/a9590764fbc7402d450fa36df14f35e0 | |
| 14:53:42 | jroll | cdent: I need to poll for resources in our devstack plugin, curl seems like the right thing for now | |
| 14:53:52 | cdent | jroll: for that, yeah | |
| 14:54:18 | cdent | (although, just because we're almost on the topic, gabbi can poll) | |
| 14:54:48 | cdent | johnthetubaguy: the cleanup or that the exception happens? | |
| 14:55:14 | johnthetubaguy | cdent: I am wondering if we need cleanup in other cases, like when an ironic node is there, but there is no allocation | |
| 14:55:26 | johnthetubaguy | cdent: maybe the API just does that for us already? | |
| 14:55:56 | johnthetubaguy | ah, I think it does | |
| 14:56:04 | cdent | I don't think I'm sufficiently contextualized to really follow what you're saying? | |
| 14:56:43 | openstackgerrit | Artom Lifshitz proposed openstack/nova-specs: Fix tag attribute disappearing https://review.openstack.org/426030 | |
| 14:57:08 | johnthetubaguy | cdent: sorry, I was worried about us not always deleting the old style ironic inventories, but I didn't realise the API would do that for us when we put the inventories | |
| 14:57:39 | mriedem | superdan: cdent: jaypipes: bauzas: johnthetubaguy: melwitt: fyi i'm starting to make a list of things to do before rc1 https://etherpad.openstack.org/p/nova-ocata-rc1-todos | |
| 14:57:44 | johnthetubaguy | its just when we have allocations we need to do better | |
| 14:57:47 | johnthetubaguy | mriedem: perfect | |
| 14:57:51 | mriedem | jroll: https://github.com/miguellavalle/python-placementclient | |
| 14:58:39 | jroll | mriedem: cool, I'll wait til that's more "official" to depend on it in devstack, I think | |
| 14:59:43 | cdent | mriedem: do we want to be concerned with the traffic breadth from the resource tracker that I wrote about in: http://lists.openstack.org/pipermail/openstack-dev/2017-January/110953.html ? There are two bug fixes in progress related to that. | |
| 15:01:05 | mriedem | cdent: maybe? throw it into the etherpad, i haven't read that thread in detail yet | |
| 15:01:08 | mriedem | i know we're super chatty | |
| 15:01:19 | cdent | k, will do | |
| 15:01:19 | openstackgerrit | Jordan Pittier proposed openstack/nova: Fix unspecified bahavior on GET /servers/detail?tenant_id=X as admin https://review.openstack.org/426259 | |
| 15:01:50 | mriedem | the RT is chatty i mean when using the report client | |
| 15:02:03 | mriedem | i think i pointed something out about that in the resource classes series that's still up | |
| 15:02:17 | mriedem | but now i can't remember what it was, i think something about updating inventory twice when we didn't need to | |
| 15:03:06 | cdent | mriedem: yeah | |
| 15:03:22 | mriedem | but thanks for doing an audit, it's easy to not care about performance when you're just trying to get something functional | |
| 15:03:25 | cdent | I think in order to maintain our self-healing we still need to report more than we want | |
| 15:03:34 | cdent | s/want/could/ | |
| 15:03:49 | cdent | but at the moment it is very redundant, and some if it just because of old bugs | |
| 15:04:11 | cdent | mriedem: was worried about what would happen when this suddenly landed on a big deployment | |
| 15:04:17 | superdan | cdent: so one thing to consider, while we see how this goes, | |
| 15:04:33 | superdan | cdent: is are we likely to make more trouble by doing what we're doing than we have now.. | |
| 15:04:35 | superdan | because currently, | |
| 15:04:54 | superdan | we do this, but over rpc and into the same database as everything else, | |
| 15:05:17 | superdan | with placement, it's over http, probably a little more compact, and to a potentially separate database | |
| 15:05:29 | superdan | and probably more efficient horizontally-scaled under apache | |
| 15:05:46 | superdan | so we could assume it's at least a wash for ocata and see how it goes with some early testers | |
| 15:05:51 | cdent | certainly the horizontal scale capabilities are more gooderer | |
| 15:05:57 | mriedem | and we're still doing the rpc to single db thing | |
| 15:06:07 | superdan | you didn't identify anything that is worse than today right? | |
| 15:06:10 | mriedem | will be nice in the future when we can drop that load from rpc traffic | |
| 15:06:15 | cdent | so yeah, we are continuing the existing, and adding more on top | |
| 15:06:16 | superdan | yeah | |
| 15:06:48 | superdan | yeah, I guess I forgot we're doing both, but, theoretically to different places | |
| 15:06:49 | superdan | but yeah | |
| 15:07:04 | cdent | the thing with object comparison is something we should probably just fix as it is clear and simple | |
| 15:07:18 | cdent | I'm not sure how much impact it has on the http side, but maybe some on the rpc side | |
| 15:07:18 | mriedem | if only we had a performance working group or something | |
| 15:07:33 | cdent | the thing with tracked_instances is just weird | |
| 15:07:34 | superdan | I haven't looked at the two fixes | |
| 15:07:47 | cdent | as the way it works now is that it doesn't at all do what it says it is going to do | |
| 15:07:48 | superdan | I'll try to do that and make a judgment | |
| 15:08:22 | cdent | thanks, that would be awesome, just doing that analysis felt like I had a significant leap forward on understanding the resource tracker, but then right at the end I felt like I fell way backwards :( | |
| 15:08:47 | superdan | well, I'm not a RT expert by any means | |
| 15:08:47 | cdent | in the "why does this even work" sense | |
| 15:08:49 | superdan | probably jay would be better, but.. | |
| 15:09:08 | johnthetubaguy | figleaf: cdent: this ironic resource provider change, I could be missing something again, but I feel like we are missing something around updating new instance allocations? well possibly anyways. | |
| 15:09:59 | openstackgerrit | Stephen Finucane proposed openstack/nova: Combine obj_make_compatible tests https://review.openstack.org/426261 | |
| 15:09:59 | openstackgerrit | Stephen Finucane proposed openstack/nova: Fix backwards compatibility for InstanceNUMACell https://review.openstack.org/396184 | |
| 15:10:08 | cdent | johnthetubaguy: figleaf has gone to the docs, I think, and knows more about that than me, but I'll listen... | |
| 15:10:22 | mriedem | johnthetubaguy: replied in https://review.openstack.org/#/c/404472/ - | |
| 15:10:33 | mriedem | johnthetubaguy: none of that fancy new code is hit unless node.resource_class is set | |
| 15:10:38 | mriedem | which it's not today in our CI | |
| 15:10:41 | mriedem | which kind of blows | |
| 15:10:55 | mriedem | if you look at the logs for the ironic job on that patch it's still setting ram/cpu/disk | |
| 15:11:12 | mriedem | would be nice if we could have a one-off DNM patch to set node.resource_class and test that stuff | |
| 15:11:14 | mriedem | jroll: ^ | |
| 15:11:39 | mriedem | johnthetubaguy: and i think the ironic-specific stuff is mostly contained in the RT | |
| 15:11:55 | mriedem | which is why in earlier patches i had asked that long-term we subclass the ResourceTracker to do the ironic-specific bits | |
| 15:12:13 | mriedem | because i don't like the "if ironic" spaghetti code | |
| 15:12:28 | mriedem | i'll review the latest on that patch when i get back in an hour or so | |
| 15:13:56 | cdent | I may be misremembering, but I thought part of the rt+resosurce class stack was to change the resource tracker so that there is only one of them (for everything it tracks), not one for each node it cares about? | |
| 15:14:08 | cdent | that might be somewhere else... | |
| 15:14:37 | melwitt | cdent: fwiw the tracked_instances clear every update_available_resource period is known and intentional. it surprised me too when I first realized it. I don't recall why it's that way, I guess to prevent stale things from hanging around potentially | |
| 15:14:55 | cdent | melwitt: I took it out and no tests failed... | |
| 15:15:01 | cdent | which made me sad and confused | |
| 15:15:10 | mriedem | cdent: that's the same | |
| 15:15:14 | mriedem | but not what i'm talking about | |