| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-06-01 | |||
| 16:26:29 | openstackgerrit | Sylvain Bauza proposed openstack/nova-specs master: Proposes Multiple GPU types https://review.openstack.org/557065 | |
| 16:26:30 | mnaser | while most other deployment tools that don't do ffu don't have to add too much wild logic | |
| 16:26:49 | PapaOurs | leakypipes: hope my new revision addressed most of your concerns https://review.openstack.org/557065 | |
| 16:26:51 | superdan | mnaser: yep | |
| 16:26:55 | PapaOurs | calling it a day now | |
| 16:26:58 | PapaOurs | bye, folks | |
| 16:27:17 | superdan | mnaser: I'm asserting that for larger clouds, the in-place upgrade case is more common, | |
| 16:27:36 | mnaser | superdan: i think i agree with you on that because usually they have more resources to keep them maintained | |
| 16:27:36 | superdan | and I think there is some perception that the common case is always to evacuate a compute node before upgrading nova-compute on it | |
| 16:27:46 | superdan | which works for 100 nodes, but not for 10,000 | |
| 16:28:05 | jgwentworth | mnaser: to be clear, doing online is the standard and the offline batch is an additional tool provided for those who prefer to leverage it or are doing FFU | |
| 16:28:17 | mnaser | jgwentworth: yep, i agree with that | |
| 16:28:23 | mnaser | so if this 'process' doesnt run, what are the consequences? | |
| 16:28:36 | mnaser | like will the scheduler be weird? instances wont spawn? | |
| 16:28:37 | superdan | mnaser: the NRP conversion? it can't not run | |
| 16:29:17 | superdan | it's a pivot of all the resource accounting we do | |
| 16:29:19 | mnaser | i'm just wondering what will happen with the inevitable cloud which will skip running them :p | |
| 16:30:27 | superdan | if they were manual only, | |
| 16:30:35 | superdan | then compute would have to refuse to start if the conversion hadn't been run | |
| 16:33:56 | mxevgenis | hi everyone! i have deployed an openstack cloud via ansible deployment and although i have 6 compute node in the nova availability zone all my instances are launched on the first 2 nodes. As a result the resources of the 4 nodes are not in use. Any idea what is going wrong? | |
| 16:34:41 | hansmoleman | scheduler defaults to pack first | |
| 16:34:44 | hansmoleman | rather than spread across all hosts | |
| 16:35:04 | jgwentworth | yep that | |
| 16:35:33 | hansmoleman | mxevgenis: see https://docs.openstack.org/nova/latest/configuration/config.html#filter_scheduler.host_subset_size | |
| 16:38:08 | mxevgenis | thanks a lot. However when i launched many instances in order to consume all the available resources and force it to use the rest node, i got the error no valid host. Which means there are no extra resources. Is it normal considering the problem you mentioned? | |
| 16:39:01 | hansmoleman | low-hanging-fruit for someone https://bugs.launchpad.net/nova/+bug/1774676 | |
| 16:39:02 | openstack | Launchpad bug 1774676 in OpenStack Compute (nova) "Confusing usage information in max_instances_per_host config option" [Medium,Triaged] | |
| 16:39:23 | hansmoleman | mxevgenis: then there is likely something wrong with those other hosts, not reporting inventory properly or something | |
| 16:39:45 | hansmoleman | you'd probably have to dig into what those are reporting for inventory, via the os-hypervisors API and checking in placement (using the osc-placement plugin) | |
| 16:39:56 | hansmoleman | https://developer.openstack.org/api-ref/compute/#hypervisors-os-hypervisors | |
| 16:40:03 | hansmoleman | https://docs.openstack.org/osc-placement/latest/index.html | |
| 16:40:44 | jgwentworth | mxevgenis: what version of nova are you using? do you mean that you launch N instances at the same time and you expect some to go to the rest of the nodes once the first nodes are full? by default, nova will try to reschedule to another host when one is full https://docs.openstack.org/nova/latest/configuration/config.html#scheduler.max_attempts | |
| 16:43:03 | mxevgenis | i deployed the ocata version of openstack using the openstack-ansible project. I launched many instances not simultaneously until all of my resources (of the first two nodes) where reserved. The n+1 instance which should be launched on one of the rest nodes failed | |
| 16:43:13 | mxevgenis | with no valid host error | |
| 16:43:51 | mxevgenis | The wired thing is that in the compute nodes tab the hosts are displayed as active | |
| 16:45:02 | jgwentworth | and any subsequent separate attempt to boot an instance is resulting in NoValidHost even though the rest of the nods are available? in that case, like hans_lunch mentioned, something else is wrong and you need to take a look at your nova-scheduler and nova-compute logs to see why it's being rejected | |
| 16:45:11 | jgwentworth | *nodes | |
| 16:45:37 | mxevgenis | thanks a lot. i appreciate your help!!! | |
| 16:46:00 | jgwentworth | prior to pike, it was possible for parallel requests to race in a way that even with reschedules, some requests could fail with NoValidHost as nodes filled up. but it sounds like that's not what you're hitting | |
| 16:46:43 | mxevgenis | i should take a look on the logs and see what happens | |
| 16:47:35 | jgwentworth | yeah, agreed. good luck | |
| 16:47:45 | mxevgenis | the wired thing is that the nodes which are working fine are different in terms of hardware to the rest servers | |
| 16:48:44 | mxevgenis | thanks a lot for your help again | |
| 16:49:37 | jgwentworth | yeah, there might be some host aggregate metadata mismatch going on (which if so, you will see in the logs) if you have things set differently depending on the hardware config of the servers. so yeah, have to check the logs to find out what's happening | |
| 17:02:00 | openstackgerrit | Dan Smith proposed openstack/nova master: Use oslo.messaging per-call monitoring https://review.openstack.org/566696 | |
| 17:25:13 | openstackgerrit | Merged openstack/nova master: libvirt: place emulator threads on CONF.compute.cpu_shared_set https://review.openstack.org/510897 | |
| 17:33:44 | cfriesen | mnaser: for what it's worth I've played with kingbird a bit and it's somewhat useful but lacking robustness and functionality. (It only supports specific quotas for example.) | |
| 17:43:53 | fried_rice | superdan, mnaser, SpamapS, jgwentworth: I think I'm providing a way to do it live without the operator/admin having to do anything or even know it's happening. | |
| 17:44:43 | superdan | fried_rice: yeah I think I acknowledged that you may be in my reply, but -EFRIDAY on processing it | |
| 17:45:17 | fried_rice | superdan: ack | |
| 17:45:53 | jgwentworth | kewl | |
| 17:49:54 | SpamapS | was hoping to discuss https://review.openstack.org/#/c/568953/ and https://bugs.launchpad.net/nova/+bug/1742102 today | |
| 17:49:55 | openstack | Launchpad bug 1742102 in OpenStack Compute (nova) "Simple user can disable compute" [High,In progress] - Assigned to Matt Riedemann (mriedem) | |
| 17:50:11 | SpamapS | but I don't see a mriedem so maybe will have to wait until next week | |
| 17:50:24 | SpamapS | I'm a little concerned that it may be very easy to DoS clouds that are set up to "pack" instead of "spread". | |
| 17:50:53 | superdan | SpamapS: so turn it off | |
| 17:51:29 | superdan | SpamapS: mriedem is hans_lunch today, btw | |
| 17:52:16 | SpamapS | superdan: yes I"m suggesting that telling people to turn it off should be a CVE | |
| 17:52:22 | jgwentworth | default is "pack", so this is a problem out-of-the-box | |
| 17:52:41 | SpamapS | since the default is to have it turned on | |
| 17:52:55 | superdan | calling it a CVE is way overblown, IMHO | |
| 17:53:11 | SpamapS | and one basically just has to get nova to try and send a bunch of broken image+flavor reqs to a single compute node to disable it... and then keep doing that until they're all disabled. | |
| 17:53:38 | SpamapS | If a regular user can disable your compute nodes, that rises to CVE IMO. | |
| 17:53:55 | jgwentworth | mnaser: I think you hit this too, right? ^ | |
| 17:54:05 | SpamapS | There are a lot of examples of advisory CVE's where certain configurations are vulnerable, and no code fix is available because it requires heavy refactoring. | |
| 17:54:20 | SpamapS | I don't really want to write an exploit for this | |
| 17:54:40 | SpamapS | but if you guys want to suggest it's not feasible.. we can go down that road, and maybe disprove it is feasible and forget about a CVE. | |
| 17:55:13 | SpamapS | But we already had our stage cloud get all of its compute nodes disabled because of a bad image. | |
| 17:55:26 | mnaser | yes i ran into this too | |
| 17:55:29 | jgwentworth | fwiw, I think it's feasible and I think it's a serious problem. I'm just not so experienced with CVEs | |
| 17:55:36 | mnaser | um one second | |
| 17:56:01 | superdan | SpamapS: that feature was specifically requested by a bunch of ops in Boston, you know | |
| 17:56:13 | SpamapS | Feature is great! Implementation, not so much. | |
| 17:56:29 | superdan | um | |
| 17:56:31 | mnaser | sometimes it would be like | |
| 17:56:47 | mnaser | nova-compute tries to create volume, user hit their quota, volume create fails, that gets labeled as a failed deploy | |
| 17:57:01 | mnaser | do enough of that and you'll start disabling everything, for us, we kinda just disabled that for now | |
| 17:57:03 | SpamapS | Though IMO it should have included a back-off re-enabler too since, presumably, a disabled compute node may recover on its own and be able to serve traffic again. It's worth it to retry nodes that were in bad shape before. | |
| 17:57:46 | superdan | mnaser: right, but that's just because volume create shouldn't be included in the list of disable-able things | |
| 17:57:56 | superdan | it was intended to only be things that were obviously fatally broken | |
| 17:58:05 | SpamapS | mnaser: yeah we're disabling the feature, and likely won't re-enable it until it also re-enables compute nodes automatically. But I figure there are likely Nova users out there that have it enabled, and are vulnerable to a malicious or even just poorly-configured user disabling all their compute nodes. | |
| 17:58:08 | superdan | SpamapS: well, we could do that, but when I brought it up in the room, nobody wanted it to re-enable | |
| 17:58:17 | mnaser | superdan: agreed, it includes things like ports, i think i worked partially on this but i forgot what progress i had :( | |
| 17:59:27 | superdan | for what it's worth, | |
| 17:59:36 | jgwentworth | mnaser: yeah. I think we saw that it would quickly become whack-a-mole, so we didn't have a straightforward way to solve it. trying to whitelist a bunch of things is a mess | |
| 17:59:48 | superdan | anything that accidentally falls into the disable bucket are also things that generate retries, | |
| 18:00:02 | superdan | so a user that can abuse that can also generate a ton of extra churn in the system, being DoSish on its own | |
| 18:00:11 | superdan | should we CVE for having max_attempts>1? | |
| 18:00:21 | SpamapS | Anyway, there are two things I'd like to see happen and I'm happy to drive either or both. (1) Fix it so that it only increments on *specific* faults that are permanent failures on the compute node, instead of just a whitelist for exceptions to ignore. And (2) inform the user community of the danger they may be in. | |
| 18:00:43 | superdan | SpamapS: it already only increments for specific things | |
| 18:00:44 | superdan | SpamapS: it's just that set needs tweaking | |
| 18:00:57 | SpamapS | did you see the list harlowja made? | |
| 18:01:01 | mnaser | superdan:, SpamapS: http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2018-04-12.log.html | |
| 18:01:03 | mnaser | brief discussion there | |
| 18:01:07 | SpamapS | Those don't seem well thought out. | |
| 18:01:12 | SpamapS | And aren't explicitly listed. | |
| 18:01:23 | SpamapS | mnaser: indeed, I found that yesterday. | |
| 18:01:39 | superdan | well, part of the problem is that we convert stupid exceptions to stupid build results | |
| 18:01:48 | superdan | and it operates on the latter | |
| 18:02:01 | SpamapS | https://bugs.launchpad.net/nova/+bug/1774527 | |