Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-06-01
16:16:19 superdan mnaser: yes
16:16:37 jgwentworth oh, right... depending on where the startup code is during the fast-forward. I see
16:16:55 hansmoleman which is why there was a migration CLI for the ironic flavor stuff as well
16:17:28 openstackgerrit Matt Riedemann proposed openstack/nova master: Trim the fat on HostState.instances https://review.openstack.org/569247
16:17:40 jgwentworth yeah. as long as there's a way we can, just worried about stuff like the PCI thing that required a pause in a fast forward
16:19:52 bpoulos jgwentworth: thank you for the heads up! I'll keep my eye on the cert validation patches
16:20:01 hansmoleman sob, oom with the fake virt driver in devstack after 26 computes out of 100
16:20:28 jgwentworth bpoulos: awesome, thanks
16:20:53 mnaser well it could be an upgrade note where "if you're doing ffu, run this nova-manage thing, if you're doing a normal upgrade, nova will fix things on start"
16:21:39 jgwentworth yeah, it could be that (and is how we've done previous things). provide a way to do an offline batch migration
16:22:39 superdan mnaser: we _have_ to provide that yeah
16:23:02 jgwentworth being able to do offline batch is useful in other ways too, for example the flavor migration from (was it kilo?) where instead of having flavors migrated on-the-fly while things are running, an operator could choose to do them in a batch via nova-manage during off peak time
16:23:09 superdan mnaser: what I want to avoid is you having to do N manual conversions for N compute nodes, when you're not FFUing and upgrading in place
16:23:34 superdan jgwentworth: yeah that was much easier because it didn't require knowledge of resource topologies
16:23:52 mnaser superdan: well, maybe i'm over simplifying things but if there is some function of fix_nested_resource_providers() and that same one can either be called from nova-manage or on start up
16:24:23 superdan mnaser: yeah, totes, I'm saying we should do it automatically if we can, fall back to manual if you are FFUing or want to do it while stuff is offline
16:25:53 mnaser the reason i like it being done online is that it simplifies life for those doing normal upgrades
16:26:03 mnaser and while i'm not implying lets make life hard for fast forward upgrades
16:26:18 mnaser but i assume they operate with the infrastructure needed to do this per-compute-node task etc
16:26:29 openstackgerrit Sylvain Bauza proposed openstack/nova-specs master: Proposes Multiple GPU types https://review.openstack.org/557065
16:26:30 mnaser while most other deployment tools that don't do ffu don't have to add too much wild logic
16:26:49 PapaOurs leakypipes: hope my new revision addressed most of your concerns https://review.openstack.org/557065
16:26:51 superdan mnaser: yep
16:26:55 PapaOurs calling it a day now
16:26:58 PapaOurs bye, folks
16:27:17 superdan mnaser: I'm asserting that for larger clouds, the in-place upgrade case is more common,
16:27:36 mnaser superdan: i think i agree with you on that because usually they have more resources to keep them maintained
16:27:36 superdan and I think there is some perception that the common case is always to evacuate a compute node before upgrading nova-compute on it
16:27:46 superdan which works for 100 nodes, but not for 10,000
16:28:05 jgwentworth mnaser: to be clear, doing online is the standard and the offline batch is an additional tool provided for those who prefer to leverage it or are doing FFU
16:28:17 mnaser jgwentworth: yep, i agree with that
16:28:23 mnaser so if this 'process' doesnt run, what are the consequences?
16:28:36 mnaser like will the scheduler be weird? instances wont spawn?
16:28:37 superdan mnaser: the NRP conversion? it can't not run
16:29:17 superdan it's a pivot of all the resource accounting we do
16:29:19 mnaser i'm just wondering what will happen with the inevitable cloud which will skip running them :p
16:30:27 superdan if they were manual only,
16:30:35 superdan then compute would have to refuse to start if the conversion hadn't been run
16:33:56 mxevgenis hi everyone! i have deployed an openstack cloud via ansible deployment and although i have 6 compute node in the nova availability zone all my instances are launched on the first 2 nodes. As a result the resources of the 4 nodes are not in use. Any idea what is going wrong?
16:34:41 hansmoleman scheduler defaults to pack first
16:34:44 hansmoleman rather than spread across all hosts
16:35:04 jgwentworth yep that
16:35:33 hansmoleman mxevgenis: see https://docs.openstack.org/nova/latest/configuration/config.html#filter_scheduler.host_subset_size
16:38:08 mxevgenis thanks a lot. However when i launched many instances in order to consume all the available resources and force it to use the rest node, i got the error no valid host. Which means there are no extra resources. Is it normal considering the problem you mentioned?
16:39:01 hansmoleman low-hanging-fruit for someone https://bugs.launchpad.net/nova/+bug/1774676
16:39:02 openstack Launchpad bug 1774676 in OpenStack Compute (nova) "Confusing usage information in max_instances_per_host config option" [Medium,Triaged]
16:39:23 hansmoleman mxevgenis: then there is likely something wrong with those other hosts, not reporting inventory properly or something
16:39:45 hansmoleman you'd probably have to dig into what those are reporting for inventory, via the os-hypervisors API and checking in placement (using the osc-placement plugin)
16:39:56 hansmoleman https://developer.openstack.org/api-ref/compute/#hypervisors-os-hypervisors
16:40:03 hansmoleman https://docs.openstack.org/osc-placement/latest/index.html
16:40:44 jgwentworth mxevgenis: what version of nova are you using? do you mean that you launch N instances at the same time and you expect some to go to the rest of the nodes once the first nodes are full? by default, nova will try to reschedule to another host when one is full https://docs.openstack.org/nova/latest/configuration/config.html#scheduler.max_attempts
16:43:03 mxevgenis i deployed the ocata version of openstack using the openstack-ansible project. I launched many instances not simultaneously until all of my resources (of the first two nodes) where reserved. The n+1 instance which should be launched on one of the rest nodes failed
16:43:13 mxevgenis with no valid host error
16:43:51 mxevgenis The wired thing is that in the compute nodes tab the hosts are displayed as active
16:45:02 jgwentworth and any subsequent separate attempt to boot an instance is resulting in NoValidHost even though the rest of the nods are available? in that case, like hans_lunch mentioned, something else is wrong and you need to take a look at your nova-scheduler and nova-compute logs to see why it's being rejected
16:45:11 jgwentworth *nodes
16:45:37 mxevgenis thanks a lot. i appreciate your help!!!
16:46:00 jgwentworth prior to pike, it was possible for parallel requests to race in a way that even with reschedules, some requests could fail with NoValidHost as nodes filled up. but it sounds like that's not what you're hitting
16:46:43 mxevgenis i should take a look on the logs and see what happens
16:47:35 jgwentworth yeah, agreed. good luck
16:47:45 mxevgenis the wired thing is that the nodes which are working fine are different in terms of hardware to the rest servers
16:48:44 mxevgenis thanks a lot for your help again
16:49:37 jgwentworth yeah, there might be some host aggregate metadata mismatch going on (which if so, you will see in the logs) if you have things set differently depending on the hardware config of the servers. so yeah, have to check the logs to find out what's happening
17:02:00 openstackgerrit Dan Smith proposed openstack/nova master: Use oslo.messaging per-call monitoring https://review.openstack.org/566696
17:25:13 openstackgerrit Merged openstack/nova master: libvirt: place emulator threads on CONF.compute.cpu_shared_set https://review.openstack.org/510897
17:33:44 cfriesen mnaser: for what it's worth I've played with kingbird a bit and it's somewhat useful but lacking robustness and functionality. (It only supports specific quotas for example.)
17:43:53 fried_rice superdan, mnaser, SpamapS, jgwentworth: I think I'm providing a way to do it live without the operator/admin having to do anything or even know it's happening.
17:44:43 superdan fried_rice: yeah I think I acknowledged that you may be in my reply, but -EFRIDAY on processing it
17:45:17 fried_rice superdan: ack
17:45:53 jgwentworth kewl
17:49:54 SpamapS was hoping to discuss https://review.openstack.org/#/c/568953/ and https://bugs.launchpad.net/nova/+bug/1742102 today
17:49:55 openstack Launchpad bug 1742102 in OpenStack Compute (nova) "Simple user can disable compute" [High,In progress] - Assigned to Matt Riedemann (mriedem)
17:50:11 SpamapS but I don't see a mriedem so maybe will have to wait until next week
17:50:24 SpamapS I'm a little concerned that it may be very easy to DoS clouds that are set up to "pack" instead of "spread".
17:50:53 superdan SpamapS: so turn it off
17:51:29 superdan SpamapS: mriedem is hans_lunch today, btw
17:52:16 SpamapS superdan: yes I"m suggesting that telling people to turn it off should be a CVE
17:52:22 jgwentworth default is "pack", so this is a problem out-of-the-box
17:52:41 SpamapS since the default is to have it turned on
17:52:55 superdan calling it a CVE is way overblown, IMHO
17:53:11 SpamapS and one basically just has to get nova to try and send a bunch of broken image+flavor reqs to a single compute node to disable it... and then keep doing that until they're all disabled.
17:53:38 SpamapS If a regular user can disable your compute nodes, that rises to CVE IMO.
17:53:55 jgwentworth mnaser: I think you hit this too, right? ^
17:54:05 SpamapS There are a lot of examples of advisory CVE's where certain configurations are vulnerable, and no code fix is available because it requires heavy refactoring.
17:54:20 SpamapS I don't really want to write an exploit for this
17:54:40 SpamapS but if you guys want to suggest it's not feasible.. we can go down that road, and maybe disprove it is feasible and forget about a CVE.
17:55:13 SpamapS But we already had our stage cloud get all of its compute nodes disabled because of a bad image.
17:55:26 mnaser yes i ran into this too
17:55:29 jgwentworth fwiw, I think it's feasible and I think it's a serious problem. I'm just not so experienced with CVEs
17:55:36 mnaser um one second
17:56:01 superdan SpamapS: that feature was specifically requested by a bunch of ops in Boston, you know
17:56:13 SpamapS Feature is great! Implementation, not so much.
17:56:29 superdan um
17:56:31 mnaser sometimes it would be like
17:56:47 mnaser nova-compute tries to create volume, user hit their quota, volume create fails, that gets labeled as a failed deploy
17:57:01 mnaser do enough of that and you'll start disabling everything, for us, we kinda just disabled that for now
17:57:03 SpamapS Though IMO it should have included a back-off re-enabler too since, presumably, a disabled compute node may recover on its own and be able to serve traffic again. It's worth it to retry nodes that were in bad shape before.
17:57:46 superdan mnaser: right, but that's just because volume create shouldn't be included in the list of disable-able things
17:57:56 superdan it was intended to only be things that were obviously fatally broken
17:58:05 SpamapS mnaser: yeah we're disabling the feature, and likely won't re-enable it until it also re-enables compute nodes automatically. But I figure there are likely Nova users out there that have it enabled, and are vulnerable to a malicious or even just poorly-configured user disabling all their compute nodes.

Earlier   Later