Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-06-01
16:13:22 superdan mnaser: right, knowing if you're affected by the NRP change, or how, is also something I don't want to mix into the upgrade decision for them either
16:13:22 SpamapS And when people are sad because their one pet VM was down for 10 minutes causing their service to be down, we definitely pat them on the back and say "there there". https://vignette.wikia.nocookie.net/glee/images/7/7a/Sheldon-leonard-there-there.gif/revision/latest?cb=20140908045609
16:13:34 superdan hansmoleman: that's what I'm saying we owe to the users
16:13:43 superdan SpamapS: ++
16:13:51 superdan oh shit
16:13:54 hansmoleman nova-manage placement heal_allocations --fix-nrp
16:13:55 superdan you woke the bear
16:14:12 hansmoleman --easy-button
16:14:21 superdan hansmoleman: unfortunately, we can't really do it completely outside with a tool I think, because we need info from the compute node
16:14:34 SpamapS superdan: the "lets me do it early" is interesting. So you're saying that if you hard reboot your instance, you don't get a downtime during the window?
16:14:53 hansmoleman superdan: maybe on restart of the compute service then? like we did for ironic instance flavors?
16:15:03 superdan SpamapS: I can opt to take the downtime early, which is just them (cold) migrating me to another node that is already fixed, but on my schedule instead of theirs, yeah
16:15:03 hansmoleman and we'll be doing for legacy bdm attachments
16:15:09 SpamapS AH
16:15:11 superdan hansmoleman: right, that's what I think we need to do
16:15:12 SpamapS that's neat
16:15:19 SpamapS I think I'm going to put that on our todo list.
16:15:33 superdan SpamapS: sometimes I opt fo that so I can check the health ofmy pet immediately, and sometimes I don't care, depending on which instance it is
16:15:41 jgwentworth superdan, hansmoleman: ++ cause that also works with FFU
16:15:57 hansmoleman well,
16:15:58 superdan jgwentworth: well, actually it doesn't
16:16:01 hansmoleman right
16:16:08 hansmoleman there was that big stink about pci stuff in one of the upgrades
16:16:08 mnaser ok so really silly could a rocky nova-compute check if it's running for the first time and do the migrations?
16:16:09 superdan jgwentworth: we have to provide them a way to do it outside too
16:16:12 superdan hansmoleman: right
16:16:19 superdan mnaser: yes
16:16:37 jgwentworth oh, right... depending on where the startup code is during the fast-forward. I see
16:16:55 hansmoleman which is why there was a migration CLI for the ironic flavor stuff as well
16:17:28 openstackgerrit Matt Riedemann proposed openstack/nova master: Trim the fat on HostState.instances https://review.openstack.org/569247
16:17:40 jgwentworth yeah. as long as there's a way we can, just worried about stuff like the PCI thing that required a pause in a fast forward
16:19:52 bpoulos jgwentworth: thank you for the heads up! I'll keep my eye on the cert validation patches
16:20:01 hansmoleman sob, oom with the fake virt driver in devstack after 26 computes out of 100
16:20:28 jgwentworth bpoulos: awesome, thanks
16:20:53 mnaser well it could be an upgrade note where "if you're doing ffu, run this nova-manage thing, if you're doing a normal upgrade, nova will fix things on start"
16:21:39 jgwentworth yeah, it could be that (and is how we've done previous things). provide a way to do an offline batch migration
16:22:39 superdan mnaser: we _have_ to provide that yeah
16:23:02 jgwentworth being able to do offline batch is useful in other ways too, for example the flavor migration from (was it kilo?) where instead of having flavors migrated on-the-fly while things are running, an operator could choose to do them in a batch via nova-manage during off peak time
16:23:09 superdan mnaser: what I want to avoid is you having to do N manual conversions for N compute nodes, when you're not FFUing and upgrading in place
16:23:34 superdan jgwentworth: yeah that was much easier because it didn't require knowledge of resource topologies
16:23:52 mnaser superdan: well, maybe i'm over simplifying things but if there is some function of fix_nested_resource_providers() and that same one can either be called from nova-manage or on start up
16:24:23 superdan mnaser: yeah, totes, I'm saying we should do it automatically if we can, fall back to manual if you are FFUing or want to do it while stuff is offline
16:25:53 mnaser the reason i like it being done online is that it simplifies life for those doing normal upgrades
16:26:03 mnaser and while i'm not implying lets make life hard for fast forward upgrades
16:26:18 mnaser but i assume they operate with the infrastructure needed to do this per-compute-node task etc
16:26:29 openstackgerrit Sylvain Bauza proposed openstack/nova-specs master: Proposes Multiple GPU types https://review.openstack.org/557065
16:26:30 mnaser while most other deployment tools that don't do ffu don't have to add too much wild logic
16:26:49 PapaOurs leakypipes: hope my new revision addressed most of your concerns https://review.openstack.org/557065
16:26:51 superdan mnaser: yep
16:26:55 PapaOurs calling it a day now
16:26:58 PapaOurs bye, folks
16:27:17 superdan mnaser: I'm asserting that for larger clouds, the in-place upgrade case is more common,
16:27:36 superdan and I think there is some perception that the common case is always to evacuate a compute node before upgrading nova-compute on it
16:27:36 mnaser superdan: i think i agree with you on that because usually they have more resources to keep them maintained
16:27:46 superdan which works for 100 nodes, but not for 10,000
16:28:05 jgwentworth mnaser: to be clear, doing online is the standard and the offline batch is an additional tool provided for those who prefer to leverage it or are doing FFU
16:28:17 mnaser jgwentworth: yep, i agree with that
16:28:23 mnaser so if this 'process' doesnt run, what are the consequences?
16:28:36 mnaser like will the scheduler be weird? instances wont spawn?
16:28:37 superdan mnaser: the NRP conversion? it can't not run
16:29:17 superdan it's a pivot of all the resource accounting we do
16:29:19 mnaser i'm just wondering what will happen with the inevitable cloud which will skip running them :p
16:30:27 superdan if they were manual only,
16:30:35 superdan then compute would have to refuse to start if the conversion hadn't been run
16:33:56 mxevgenis hi everyone! i have deployed an openstack cloud via ansible deployment and although i have 6 compute node in the nova availability zone all my instances are launched on the first 2 nodes. As a result the resources of the 4 nodes are not in use. Any idea what is going wrong?
16:34:41 hansmoleman scheduler defaults to pack first
16:34:44 hansmoleman rather than spread across all hosts
16:35:04 jgwentworth yep that
16:35:33 hansmoleman mxevgenis: see https://docs.openstack.org/nova/latest/configuration/config.html#filter_scheduler.host_subset_size
16:38:08 mxevgenis thanks a lot. However when i launched many instances in order to consume all the available resources and force it to use the rest node, i got the error no valid host. Which means there are no extra resources. Is it normal considering the problem you mentioned?
16:39:01 hansmoleman low-hanging-fruit for someone https://bugs.launchpad.net/nova/+bug/1774676
16:39:02 openstack Launchpad bug 1774676 in OpenStack Compute (nova) "Confusing usage information in max_instances_per_host config option" [Medium,Triaged]
16:39:23 hansmoleman mxevgenis: then there is likely something wrong with those other hosts, not reporting inventory properly or something
16:39:45 hansmoleman you'd probably have to dig into what those are reporting for inventory, via the os-hypervisors API and checking in placement (using the osc-placement plugin)
16:39:56 hansmoleman https://developer.openstack.org/api-ref/compute/#hypervisors-os-hypervisors
16:40:03 hansmoleman https://docs.openstack.org/osc-placement/latest/index.html
16:40:44 jgwentworth mxevgenis: what version of nova are you using? do you mean that you launch N instances at the same time and you expect some to go to the rest of the nodes once the first nodes are full? by default, nova will try to reschedule to another host when one is full https://docs.openstack.org/nova/latest/configuration/config.html#scheduler.max_attempts
16:43:03 mxevgenis i deployed the ocata version of openstack using the openstack-ansible project. I launched many instances not simultaneously until all of my resources (of the first two nodes) where reserved. The n+1 instance which should be launched on one of the rest nodes failed
16:43:13 mxevgenis with no valid host error
16:43:51 mxevgenis The wired thing is that in the compute nodes tab the hosts are displayed as active
16:45:02 jgwentworth and any subsequent separate attempt to boot an instance is resulting in NoValidHost even though the rest of the nods are available? in that case, like hans_lunch mentioned, something else is wrong and you need to take a look at your nova-scheduler and nova-compute logs to see why it's being rejected
16:45:11 jgwentworth *nodes
16:45:37 mxevgenis thanks a lot. i appreciate your help!!!
16:46:00 jgwentworth prior to pike, it was possible for parallel requests to race in a way that even with reschedules, some requests could fail with NoValidHost as nodes filled up. but it sounds like that's not what you're hitting
16:46:43 mxevgenis i should take a look on the logs and see what happens
16:47:35 jgwentworth yeah, agreed. good luck
16:47:45 mxevgenis the wired thing is that the nodes which are working fine are different in terms of hardware to the rest servers
16:48:44 mxevgenis thanks a lot for your help again
16:49:37 jgwentworth yeah, there might be some host aggregate metadata mismatch going on (which if so, you will see in the logs) if you have things set differently depending on the hardware config of the servers. so yeah, have to check the logs to find out what's happening
17:02:00 openstackgerrit Dan Smith proposed openstack/nova master: Use oslo.messaging per-call monitoring https://review.openstack.org/566696
17:25:13 openstackgerrit Merged openstack/nova master: libvirt: place emulator threads on CONF.compute.cpu_shared_set https://review.openstack.org/510897
17:33:44 cfriesen mnaser: for what it's worth I've played with kingbird a bit and it's somewhat useful but lacking robustness and functionality. (It only supports specific quotas for example.)
17:43:53 fried_rice superdan, mnaser, SpamapS, jgwentworth: I think I'm providing a way to do it live without the operator/admin having to do anything or even know it's happening.
17:44:43 superdan fried_rice: yeah I think I acknowledged that you may be in my reply, but -EFRIDAY on processing it
17:45:17 fried_rice superdan: ack
17:45:53 jgwentworth kewl
17:49:54 SpamapS was hoping to discuss https://review.openstack.org/#/c/568953/ and https://bugs.launchpad.net/nova/+bug/1742102 today
17:49:55 openstack Launchpad bug 1742102 in OpenStack Compute (nova) "Simple user can disable compute" [High,In progress] - Assigned to Matt Riedemann (mriedem)
17:50:11 SpamapS but I don't see a mriedem so maybe will have to wait until next week
17:50:24 SpamapS I'm a little concerned that it may be very easy to DoS clouds that are set up to "pack" instead of "spread".

Earlier   Later