| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-06-01 | |||
| 18:18:35 | SpamapS | Yeah we just report on changes or something, yes? | |
| 18:18:47 | SpamapS | so could be the same for this score, no? | |
| 18:18:52 | superdan | well, we do via the old mechanism, but that's what we want to remove, and eventually hopefully the need to even run periodically | |
| 18:19:21 | SpamapS | can you point me at a description of those different ways? I want to understand. :) | |
| 18:19:24 | superdan | the thing that is responsible for checking resources has nothing to do with this either | |
| 18:19:48 | superdan | everything we've done with placement lately? | |
| 18:19:52 | superdan | I can't point you at one thing | |
| 18:20:05 | SpamapS | I don't want you to have to type it all into IRC.. if there's just a description, I can think more clearly about how to make a weighter based on the scheduling health, which I think might be a nice way to evolve this feature. | |
| 18:20:34 | superdan | it's not the weigher that's a problem, of course, | |
| 18:21:20 | superdan | it's just more complicated if you have a periodic, which has a decay interval (config) and runs either independently (we have so many) or glommed onto something else like resource audit, which it has nothing to do with | |
| 18:21:52 | superdan | and a successful boot either zeroing or aggressively decrementing the value is semantically closer to what we implemented initially, which people liked | |
| 18:22:12 | SpamapS | I think it makes sense to decay it based on time, but maybe there are better ways. I like the idea of just having a coeficient to pull the fail counter down faster than it rises. | |
| 18:23:32 | superdan | the less we change the behavior, the more likely we are to be able to maybe backport something too | |
| 18:23:59 | jgwentworth | SpamapS: coincidentally I happened upon this earlier today when trying to answer a different question, might be a good starting point for learning more if you're interested https://docs.openstack.org/nova/latest/reference/scheduling.html | |
| 18:24:10 | SpamapS | Yeah, I think we'd just have to tell people to turn it off if they're in a situation where they might get DoS'd. | |
| 18:24:30 | SpamapS | jgwentworth: thanks, was just reading that! :) | |
| 18:24:36 | jgwentworth | oh, heh | |
| 18:25:15 | SpamapS | like, 30s before you sent, so, we're on the same page. Literally. | |
| 18:25:24 | jgwentworth | haha | |
| 18:26:00 | SpamapS | The thing is, it would have to go down over time or nodes that got way off the rails might never see activity again. | |
| 18:26:21 | superdan | if you're packing, you're opting into empty nodes right? | |
| 18:26:57 | SpamapS | like if some aggregate got really full and a poorly weighted HV that is unhealthy gets scheduled and fails a lot for a while.. its score gets really high, then you add capacity somehow.. that node may never get any attempts unless you decrement or reset the counter. | |
| 18:27:03 | superdan | I'm just trying to think about how we can do this initially with minimal change (config, code) and minimal semantic difference | |
| 18:27:12 | superdan | for the purposes of applying to to existing stuff | |
| 18:27:31 | superdan | if we're not interested in backporting it (backporting a weigher would be a first I bet) then maybe it doesn't matter | |
| 18:27:56 | SpamapS | Yeah I'm not sure. Still thinking through what might be a more self-managing place for the feature is all. | |
| 18:29:16 | SpamapS | Good chat. I'll give it some thought, and see if we can also help the VMT determine if we should notify users about the potential for DoS. | |
| 18:35:00 | superdan | I just looked over all our periodics and I don't think it fits with any of the existing ones | |
| 18:35:26 | superdan | there are a couple that would be closeish, but would still look really random to just do this other thing in the middle, | |
| 18:35:58 | harlowja | people can if they want try http://paste.openstack.org/show/722481/ on some public openstack cloud, though i'd recommend u communicate with their operators before doing it at any scale | |
| 18:36:05 | superdan | plus you probably want to be able to control the interval of this, which means it needs its own knob | |
| 18:36:17 | harlowja | we ran that on one of our idle clouds and it was able to knock off 2 compute nodes in about 15 minutes | |
| 18:36:34 | harlowja | afaik because of https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L1804-L1810 | |
| 18:36:48 | harlowja | line 81 -> 85 are the 'triggers' | |
| 18:37:01 | harlowja | boot faster than nova can build | |
| 18:37:06 | jrollinhatin | probably best not to post a working exploit for a security bug in an irc channel this large :| | |
| 18:37:07 | harlowja | *boot and delete | |
| 18:37:15 | superdan | jrollinhatin: yeah nfs | |
| 18:39:00 | penick | harlowja dude | |
| 18:41:13 | harlowja | i'd have kept it on https://bugs.launchpad.net/nova/+bug/1774527 but that is already public as well | |
| 18:41:14 | openstack | Launchpad bug 1742102 in OpenStack Compute (nova) "duplicate for #1774527 Simple user can disable compute" [High,In progress] - Assigned to Matt Riedemann (mriedem) | |
| 18:41:53 | SpamapS | Cat's been out of the bag since January guys. That's not exactly rocket science. | |
| 18:43:24 | superdan | we've discussed it in public many times before then, | |
| 18:43:52 | superdan | but if ya'll are so concerned about disclosure.. seems weird to argue for that and posting exploits :) | |
| 18:47:29 | hans_lunch | jgwentworth: i think this bp is probably also done now https://review.openstack.org/#/q/topic:bp/overhead-pin-set+(status:open+OR+status:merged) | |
| 18:49:44 | jgwentworth | hans_lunch: awesome thanks. maybe I'll double check with sahid before closing it out | |
| 19:01:33 | mriedem | so it seems some fun times were had while i was eating bbq | |
| 19:02:19 | mriedem | mnaser: btw, https://review.openstack.org/#/c/510235/ was a brain dump on the "limit --min-count during server create" issue | |
| 19:02:29 | mriedem | it's really weird though | |
| 19:02:59 | mriedem | from what i remember (of me writing the thing) is proposing a per-request limit | |
| 19:03:03 | mriedem | configurable like quotas | |
| 19:03:37 | superdan | just like max_results, but inbound yeah? seems to make sense to me | |
| 19:03:59 | mriedem | right it's the exact same idea as metadata_items and injected_files quotas | |
| 19:04:03 | mriedem | those are purely rate limiting | |
| 19:04:28 | mriedem | https://review.openstack.org/#/c/510235/1/specs/queens/approved/instance-max-count-limit.rst@94 | |
| 19:04:43 | superdan | well, I wouldn't call them quotas since they're per-request and not per-tenant, but yeah | |
| 19:04:45 | superdan | being a loaded term and all | |
| 19:04:50 | mriedem | right | |
| 19:05:05 | mriedem | your rpc heartbeat thing solves part of this also | |
| 19:05:12 | mriedem | with the select_destinations retry thing | |
| 19:05:21 | mnaser | a per request limit is really what's needed because i can add some middleware to limit # of api requests easily | |
| 19:05:33 | mnaser | but min_servers=999999999999999 isn't something i can control much | |
| 19:05:43 | mriedem | right, well, it sounds like there is interest in this :) | |
| 19:05:51 | superdan | mnaser: yeah | |
| 19:06:07 | superdan | mriedem: if we enable it for select_destinations yeah | |
| 19:06:15 | mriedem | i currently seem to have too many spinning plates | |
| 19:06:37 | mriedem | i could probably get yikun to work on this though... | |
| 19:44:15 | openstackgerrit | Kashyap Chamarthy proposed openstack/nova master: libvirt: Remove usage of migrateToURI{2} APIs https://review.openstack.org/567258 | |
| 20:25:36 | cfriesen | kashyap: are the valid "param" keys for | |
| 20:25:37 | cfriesen | Personally I'd replace the first sentence with something like "Thanks for checking out the StarlingX codebase." I think we'd want to make inclusion in the community an explicit opt-in thing rather than making assumptions. Sounds like Matt's got the technical details covered though so maybe it's a moot point. | |
| 20:25:39 | cfriesen | I don't think there's any chance Kashyap would be interested in doing the actual work. He (and others) have already expressed skepticism at the idea that anyone outside of Intel/Wind River would ever contribute to StarlingX. | |
| 20:25:40 | cfriesen | Chris | |
| 20:26:00 | cfriesen | well that's embarassing | |
| 20:29:00 | cfriesen | what I was trying to say, was are the valid "param" keys for the migrateToURI3() function documented somewhere? I didn't see them on the Libvirt API docs. | |
| 20:56:33 | openstackgerrit | Merged openstack/nova master: Restrict CONF.quota.driver to DB and noop quota drivers https://review.openstack.org/410996 | |
| 20:56:39 | openstackgerrit | Merged openstack/nova master: Fix invalid raise in test_compute_mgr https://review.openstack.org/571610 | |
| 21:06:43 | openstackgerrit | Merged openstack/nova master: Implement granular policy rules for placement https://review.openstack.org/524425 | |
| 21:12:22 | openstackgerrit | Jay Pipes proposed openstack/nova master: Add a microversion for consumer generation support https://review.openstack.org/565604 | |
| 21:31:13 | openstackgerrit | Jay Pipes proposed openstack/nova master: placement: always create consumer records https://review.openstack.org/567678 | |
| 21:31:14 | openstackgerrit | Jay Pipes proposed openstack/nova master: add consumers generation field https://review.openstack.org/557958 | |
| 21:31:15 | openstackgerrit | Jay Pipes proposed openstack/nova master: placement: Allocation.consumer field https://review.openstack.org/565405 | |
| 21:31:16 | openstackgerrit | Jay Pipes proposed openstack/nova master: rework allocation handler _allocations_dict() https://review.openstack.org/565407 | |
| 21:31:17 | openstackgerrit | Jay Pipes proposed openstack/nova master: Add a microversion for consumer generation support https://review.openstack.org/565604 | |
| 21:41:45 | mriedem | woot 20 fake compute nodes in devstack single node, time for some fun | |
| 21:45:59 | openstackgerrit | Merged openstack/nova master: [placement] default to accept of application/json when */* https://review.openstack.org/568630 | |
| 21:53:08 | openstackgerrit | Chris Dent proposed openstack/nova master: Extract part of PlacementFixture to placement https://review.openstack.org/568359 | |
| 22:16:30 | openstackgerrit | karim proposed openstack/nova master: Handle rebuild of instances with image traits https://review.openstack.org/569498 | |
| 22:29:51 | openstackgerrit | Eric Fried proposed openstack/nova master: Test for multiple limit/group_policy qparams https://review.openstack.org/568713 | |
| 22:47:05 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add nova-manage placement heal_allocations CLI https://review.openstack.org/565886 | |
| 23:07:26 | openstackgerrit | Chris Dent proposed openstack/nova master: Optional separate database for placement API https://review.openstack.org/362766 | |
| 23:08:20 | openstackgerrit | Chris Dent proposed openstack/nova master: Isolate placement database config https://review.openstack.org/541435 | |
| 23:08:56 | openstackgerrit | Chris Dent proposed openstack/nova master: Ensure that os-traits sync is attempted only at start of process https://review.openstack.org/553857 | |
| 23:09:15 | openstackgerrit | Chris Dent proposed openstack/nova master: Add PLACEMENT_DB_ENABLED=True to the nova-next job https://review.openstack.org/564067 | |
| 23:10:42 | openstackgerrit | Chris Dent proposed openstack/nova master: Use nova.db.api directly https://review.openstack.org/543262 | |
| #openstack-nova - 2018-06-02 | |||
| 00:11:33 | sayalip | hi | |
| 00:33:58 | openstackgerrit | Tsuyoshi Nagata proposed openstack/nova master: nova improvement of maximum attach volumes more than 26 vols https://review.openstack.org/567472 | |
| 02:14:10 | openstackgerrit | Chris Dent proposed openstack/nova master: Use nova.db.api directly https://review.openstack.org/543262 | |
| 02:20:52 | openstackgerrit | Chris Dent proposed openstack/nova master: Ensure that os-traits sync is attempted only at start of process https://review.openstack.org/553857 | |
| 13:20:54 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Time how long it takes to get host instance info during scheduling https://review.openstack.org/571928 | |
| 13:23:29 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Time how long it takes to get host instance info during scheduling https://review.openstack.org/571928 | |