| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-10-18 | |||
| 17:45:38 | eandersson | I would assume so, but not 100% sure | |
| 17:46:28 | eandersson | If just ranzomizing the results a bit fixes it I probably prefer that route | |
| 17:46:39 | eandersson | but if that isn't enough I could look into doing that | |
| 17:47:48 | melwitt | fwiw I think you're already doing all you can to mitigate, by increasing the subset size and increasing retries. there's another option that could help, https://docs.openstack.org/nova/rocky/configuration/config.html#filter_scheduler.shuffle_best_same_weighed_hosts | |
| 17:48:07 | melwitt | (assuming you can't change anything about the numa) | |
| 17:48:35 | eandersson | interesting, didn't know about that setting | |
| 17:48:50 | melwitt | yeah, that's the only other one involved in all this that you haven't looked at already | |
| 17:49:19 | efried | eandersson: if the problem is numa as described, randomizing won't help at all. | |
| 17:49:35 | openstackgerrit | Dan Smith proposed openstack/python-novaclient master: Add images.GlanceManager.find_images() bulk query https://review.opendev.org/689537 | |
| 17:49:36 | openstackgerrit | Dan Smith proposed openstack/python-novaclient master: Add aggregate-cache-images command and client routines https://review.opendev.org/687141 | |
| 17:49:37 | eandersson | I see - I think what melwitt just linked might be the issue | |
| 17:49:46 | melwitt | eandersson: the randomize_allocation_candidates won't help you I don't think because it returns things in a deterministic order (so as not to necessarily break scheduling stacking) | |
| 17:50:05 | openstackgerrit | Dan Smith proposed openstack/python-novaclient master: Add images.GlanceManager.find_images() bulk query https://review.opendev.org/689537 | |
| 17:50:06 | openstackgerrit | Dan Smith proposed openstack/python-novaclient master: Add aggregate-cache-images command and client routines https://review.opendev.org/687141 | |
| 17:51:41 | efried | melwitt: "it returns things" <== "it" being the filter? | |
| 17:51:49 | melwitt | no, placement | |
| 17:52:09 | efried | with randomize_allocation_candidates placement does *not* return things in deterministic order. | |
| 17:52:36 | melwitt | oh, lol, sorry I read the help backwards | |
| 17:53:20 | efried | yeah, with it *off* the order is deterministic (but inscrutable, based on natural db order of some nonobvious kind) | |
| 17:53:45 | melwitt | right | |
| 17:54:49 | melwitt | eandersson: ^ ignore what I said earlier about randomize_allocation_candidates | |
| 17:54:57 | eandersson | randomize_allocation_candidates = good? | |
| 17:56:35 | efried | eandersson: if the problem is the numa thing, randomize_allocation_candidates = irrelevant. | |
| 17:57:20 | melwitt | eandersson: what efried said. it won't help you because your problem is the numa fitment. I was just correcting what I said about it being a deterministic order, it would be the opposite of that, a random order | |
| 17:57:43 | eandersson | I see - I assumed so, but wanted to make sure | |
| 17:57:47 | melwitt | I read the help backwards and... I don't have much of an excuse | |
| 17:58:14 | efried | The only way it would be at all likely to help you is if $number_of_concurrent_deploys < $number_of_possible_hosts *and* you use a ?limit < $number_of_possible_hosts | |
| 17:58:55 | efried | but like I said, that's a pretty hacky thing to try to figure out | |
| 17:59:19 | melwitt | eandersson: the shuffle_best_same_weighed_hosts might help you, I think. numa stuff might be why it was added in the first place | |
| 17:59:35 | eandersson | Yea - sounds super promising. | |
| 18:02:02 | melwitt | +1 efried. eandersson: the randomize_allocation_candidates use case is for very large deployments where the default limit for returned candidates is < the total number of compute hosts in the deployment. think CERN with multiple cells | |
| 18:02:19 | eandersson | Just fyi we have ~1k nodes in each deployment | |
| 18:03:42 | melwitt | ack | |
| 18:04:54 | eandersson | per cell | |
| 18:06:07 | melwitt | oh, ok that is a lot | |
| 18:06:08 | eandersson | We use regions instead of cells at the moment, but might want to change that in the future | |
| 18:08:15 | melwitt | here's the background on the option https://specs.openstack.org/openstack/nova-specs/specs/queens/implemented/allocation-candidates-limit.html | |
| 18:08:57 | eandersson | We limit ourself to 1k nodes today because we don't know if neutron can handle it | |
| 18:09:20 | eandersson | (used to be nova + neutron, but doubt nova will have a problem today) | |
| 18:09:49 | eandersson | when we upgraded to rocky neutron used up like 500GB memory lol | |
| 18:10:02 | eandersson | each neutron process peaked at 8.2GB memory used | |
| 18:10:31 | melwitt | gotcha. belmiro from CERN would be the one to chat with about neutron scaling | |
| 18:10:39 | eandersson | We work with them a lot | |
| 18:10:47 | melwitt | I don't remember if they are doing something special re: neutron | |
| 18:10:49 | eandersson | They use one big Layer 2 network | |
| 18:10:50 | melwitt | ok cool | |
| 18:11:19 | melwitt | and that's... a lot of memory O.o | |
| 18:11:32 | eandersson | Yea it's nuts. I tend to ping the neutron channel but no one ever replies | |
| 18:11:38 | eandersson | Nova channel is way better :p | |
| 18:12:07 | melwitt | ah yeah, I think they might be mostly EU timezone | |
| 18:12:17 | eandersson | Some awesome people here, melwitt efried sean-k-mooney mnaser mriedem to name a few ;) | |
| 18:12:35 | melwitt | :) | |
| 18:13:10 | mnaser | \o/ | |
| 18:13:14 | mnaser | eandersson: will you be at shanghai? | |
| 18:13:22 | eandersson | I will miss shanghai :'( | |
| 18:13:32 | eandersson | Will be attending the next one for sure | |
| 18:15:38 | eandersson | Which is a bummer, especially since Train has been the largest contribution from us yet :P | |
| 18:16:05 | eandersson | 90% of that was to Designate hehe | |
| 18:17:44 | openstackgerrit | Merged openstack/nova stable/queens: Drop source node allocations if finish_resize fails https://review.opendev.org/682722 | |
| 18:20:44 | mnaser | eandersson: awesome | |
| 18:25:38 | eandersson | btw some more shameless pr, but if you know anyone in South California or Austin Texas, we are hiring OpenStack people (systems and/or software) | |
| 18:25:48 | eandersson | *that is looking for work | |
| 18:50:23 | efried | eandersson: not that I'm looking, who's "we"? | |
| 18:51:09 | efried | I'm in Austin, but have a cushy wfh gig. Wouldn't consider anything with a commute longer than eleven seconds. | |
| 18:52:23 | dansmith | agree, although the traffic on my commute is getting ridiculous | |
| 18:52:35 | dansmith | earlier this week, I had to wait for the cat to pass before I made a left at the stairs | |
| 18:52:41 | dansmith | major road rage situation | |
| 19:02:56 | mordred | efried: I believe eandersson is at blizzard | |
| 19:03:15 | efried | that makes sense | |
| 19:04:05 | efried | Are game companies still as crazy as they were in the 90s? That's the last time I paid attention. | |
| 19:05:05 | efried | people living in their office for a month at a time, kind of thing | |
| 19:06:17 | mordred | I live in my office for a month at a time | |
| 19:09:21 | efried | aaand you're crazy, what's your point? | |
| 19:09:51 | efried | oh, that things other than game companies can be crazy, got it. | |
| 19:18:05 | openstackgerrit | Eric Fried proposed openstack/nova master: ItemsMatcher: mock call list arg in any order https://review.opendev.org/689487 | |
| 19:18:06 | openstackgerrit | Eric Fried proposed openstack/nova master: Always trait the compute node RP with COMPUTE_NODE https://review.opendev.org/688979 | |
| 19:23:52 | mordred | efried: yeah. I mostly meant, you know - like, my house, which is also my office | |
| 19:24:04 | mordred | efried: I'm maybe not as funny or cleve as I sometimes think | |
| 19:24:18 | efried | Oh, you are, but probably not for the *reasons* you think. | |
| 19:25:40 | efried | See, my funniness is inversely proportional to how much I'm trying. There's no surprise funny from me. | |
| 20:50:42 | eandersson | efried, yea unfortunately still all about that office | |
| 20:53:34 | eandersson | https://careers.blizzard.com/en-us/openings/o6vmbfwS (Austin) https://careers.blizzard.com/en-us/openings/ohdYafwm (Irvine) | |
| 21:38:07 | mriedem | gmann: remember how we thought host_status would be returned for GET /servers/detail with servers from down cells? i found that won't happen | |
| 21:38:35 | mriedem | because of this filtering https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/views/servers.py#L590 | |
| 21:39:08 | mriedem | efried: sorry about the late reply, i've been not around so much today - i took your comments and threw them into my patch so i can process them on monday | |
| 21:39:23 | efried | saw dat, get some rest and feel better. | |
| 21:53:57 | efried | (How) does nova (via oslo.config?) vet permissions of nova.conf? | |
| 21:54:17 | efried | or is that completely the responsibility of the admin? | |
| 22:00:13 | sean-k-mooney | efried: i think its up to the admin/installer | |
| 22:01:06 | sean-k-mooney | e.g. we dont check that the config is 600 or 660 as far as i am aware | |
| 22:01:22 | efried | so if /etc/nova and contents were 5777 and owned by $random we would truck happily along? | |
| 22:01:31 | mriedem | gmann: melwitt: a couple of questions in the host_status_unknown patch - mostly for gmann about policy rule naming and the docs on the policy rule since 2.75+ https://review.opendev.org/#/c/679181/ | |
| 22:01:48 | sean-k-mooney | efried: i think so | |
| 22:02:09 | efried | o...kay. | |
| 22:02:24 | sean-k-mooney | nova does not create the file and we never write to it | |
| 22:02:45 | mriedem | efried: likely a question for bnemec | |
| 22:02:48 | sean-k-mooney | so its really up to the install to make sure that files with password in them are not world readable | |
| 22:03:29 | mriedem | i thought there was some talk of some how vaultifying that stuff but i might be dreaming that up | |
| 22:03:30 | efried | sean-k-mooney: well, that and if they're writable in any way a malicious user could make nova do horrible things to the system | |
| 22:03:44 | efried | mriedem: I think sdk does something like that. | |