Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-06-01
16:08:22 mnaser reasoning is: hypervisor and kernel touches something that affects the actual running vm, nova-compute does not
16:08:28 superdan mnaser: okay, you never schedule downtime per instance? my cloud provider does it that way
16:08:54 mnaser nope, since we have ceph everywhere, live migrations are easy and function extremely well
16:09:06 mnaser 10g network + ceph = no problems
16:09:19 superdan mnaser: yeah, so what if you didn't have that? I know it's a bit theoretical but,
16:09:35 superdan what if you had to transfer the entire disk and memory across the network for every migration?
16:09:48 hansmoleman we know that would kill NTT..
16:09:49 hansmoleman at least
16:09:50 superdan would that change the calculus?
16:10:02 mnaser yeah, big time, it would make the upgrade a huge pain
16:10:09 superdan ack, okay thanks
16:10:10 mnaser especially in a public cloud, you have to coordinate across so many customers..
16:10:37 hansmoleman is there something specifically up for review or being debated that would require us to move all instances to migrate to rocky?
16:10:57 superdan yeah, the NRP allocation conversion thread
16:11:10 hansmoleman ok i figured
16:11:18 hansmoleman i've been blissfully ignorant on that so far
16:11:25 mnaser ^^ me too
16:11:31 superdan I don't think forcing operators to migrate all instances to convert their allocations to nested is reasonable
16:11:34 mnaser because i don't particularly understand it that much :)
16:11:48 mnaser but maybe i should chime in, because i don't know all that much about the whole story
16:11:51 SpamapS We have 20gbit networking for migrations, and we're not ceph based. We aren't going to do a full migration for kernel/kvm updates, we'll schedule downtime. Reboot time is about 8 minutes.
16:12:12 SpamapS We do AZ's, and make sure to complete one AZ before doing the next.
16:12:14 superdan SpamapS: ah sweet, that's a data point I was looking for
16:12:21 superdan my cloud provider does the same
16:12:33 superdan basically tells me that my instances will reboot on X at Y hour, downtime is minutes
16:12:48 mnaser i think in subjects like this, it would be helpful if operators can get a much more simple question like the one superdan asked rather than having to understand whats going on with nested resource providers
16:12:50 jgwentworth my cloud provider does that too
16:12:50 superdan and lets me do it early if I want
16:13:15 hansmoleman can't we have an allocation transformer tool or something for NRP?
16:13:22 SpamapS And when people are sad because their one pet VM was down for 10 minutes causing their service to be down, we definitely pat them on the back and say "there there". https://vignette.wikia.nocookie.net/glee/images/7/7a/Sheldon-leonard-there-there.gif/revision/latest?cb=20140908045609
16:13:22 superdan mnaser: right, knowing if you're affected by the NRP change, or how, is also something I don't want to mix into the upgrade decision for them either
16:13:34 superdan hansmoleman: that's what I'm saying we owe to the users
16:13:43 superdan SpamapS: ++
16:13:51 superdan oh shit
16:13:54 hansmoleman nova-manage placement heal_allocations --fix-nrp
16:13:55 superdan you woke the bear
16:14:12 hansmoleman --easy-button
16:14:21 superdan hansmoleman: unfortunately, we can't really do it completely outside with a tool I think, because we need info from the compute node
16:14:34 SpamapS superdan: the "lets me do it early" is interesting. So you're saying that if you hard reboot your instance, you don't get a downtime during the window?
16:14:53 hansmoleman superdan: maybe on restart of the compute service then? like we did for ironic instance flavors?
16:15:03 hansmoleman and we'll be doing for legacy bdm attachments
16:15:03 superdan SpamapS: I can opt to take the downtime early, which is just them (cold) migrating me to another node that is already fixed, but on my schedule instead of theirs, yeah
16:15:09 SpamapS AH
16:15:11 superdan hansmoleman: right, that's what I think we need to do
16:15:12 SpamapS that's neat
16:15:19 SpamapS I think I'm going to put that on our todo list.
16:15:33 superdan SpamapS: sometimes I opt fo that so I can check the health ofmy pet immediately, and sometimes I don't care, depending on which instance it is
16:15:41 jgwentworth superdan, hansmoleman: ++ cause that also works with FFU
16:15:57 hansmoleman well,
16:15:58 superdan jgwentworth: well, actually it doesn't
16:16:01 hansmoleman right
16:16:08 mnaser ok so really silly could a rocky nova-compute check if it's running for the first time and do the migrations?
16:16:08 hansmoleman there was that big stink about pci stuff in one of the upgrades
16:16:09 superdan jgwentworth: we have to provide them a way to do it outside too
16:16:12 superdan hansmoleman: right
16:16:19 superdan mnaser: yes
16:16:37 jgwentworth oh, right... depending on where the startup code is during the fast-forward. I see
16:16:55 hansmoleman which is why there was a migration CLI for the ironic flavor stuff as well
16:17:28 openstackgerrit Matt Riedemann proposed openstack/nova master: Trim the fat on HostState.instances https://review.openstack.org/569247
16:17:40 jgwentworth yeah. as long as there's a way we can, just worried about stuff like the PCI thing that required a pause in a fast forward
16:19:52 bpoulos jgwentworth: thank you for the heads up! I'll keep my eye on the cert validation patches
16:20:01 hansmoleman sob, oom with the fake virt driver in devstack after 26 computes out of 100
16:20:28 jgwentworth bpoulos: awesome, thanks
16:20:53 mnaser well it could be an upgrade note where "if you're doing ffu, run this nova-manage thing, if you're doing a normal upgrade, nova will fix things on start"
16:21:39 jgwentworth yeah, it could be that (and is how we've done previous things). provide a way to do an offline batch migration
16:22:39 superdan mnaser: we _have_ to provide that yeah
16:23:02 jgwentworth being able to do offline batch is useful in other ways too, for example the flavor migration from (was it kilo?) where instead of having flavors migrated on-the-fly while things are running, an operator could choose to do them in a batch via nova-manage during off peak time
16:23:09 superdan mnaser: what I want to avoid is you having to do N manual conversions for N compute nodes, when you're not FFUing and upgrading in place
16:23:34 superdan jgwentworth: yeah that was much easier because it didn't require knowledge of resource topologies
16:23:52 mnaser superdan: well, maybe i'm over simplifying things but if there is some function of fix_nested_resource_providers() and that same one can either be called from nova-manage or on start up
16:24:23 superdan mnaser: yeah, totes, I'm saying we should do it automatically if we can, fall back to manual if you are FFUing or want to do it while stuff is offline
16:25:53 mnaser the reason i like it being done online is that it simplifies life for those doing normal upgrades
16:26:03 mnaser and while i'm not implying lets make life hard for fast forward upgrades
16:26:18 mnaser but i assume they operate with the infrastructure needed to do this per-compute-node task etc
16:26:29 openstackgerrit Sylvain Bauza proposed openstack/nova-specs master: Proposes Multiple GPU types https://review.openstack.org/557065
16:26:30 mnaser while most other deployment tools that don't do ffu don't have to add too much wild logic
16:26:49 PapaOurs leakypipes: hope my new revision addressed most of your concerns https://review.openstack.org/557065
16:26:51 superdan mnaser: yep
16:26:55 PapaOurs calling it a day now
16:26:58 PapaOurs bye, folks
16:27:17 superdan mnaser: I'm asserting that for larger clouds, the in-place upgrade case is more common,
16:27:36 mnaser superdan: i think i agree with you on that because usually they have more resources to keep them maintained
16:27:36 superdan and I think there is some perception that the common case is always to evacuate a compute node before upgrading nova-compute on it
16:27:46 superdan which works for 100 nodes, but not for 10,000
16:28:05 jgwentworth mnaser: to be clear, doing online is the standard and the offline batch is an additional tool provided for those who prefer to leverage it or are doing FFU
16:28:17 mnaser jgwentworth: yep, i agree with that
16:28:23 mnaser so if this 'process' doesnt run, what are the consequences?
16:28:36 mnaser like will the scheduler be weird? instances wont spawn?
16:28:37 superdan mnaser: the NRP conversion? it can't not run
16:29:17 superdan it's a pivot of all the resource accounting we do
16:29:19 mnaser i'm just wondering what will happen with the inevitable cloud which will skip running them :p
16:30:27 superdan if they were manual only,
16:30:35 superdan then compute would have to refuse to start if the conversion hadn't been run
16:33:56 mxevgenis hi everyone! i have deployed an openstack cloud via ansible deployment and although i have 6 compute node in the nova availability zone all my instances are launched on the first 2 nodes. As a result the resources of the 4 nodes are not in use. Any idea what is going wrong?
16:34:41 hansmoleman scheduler defaults to pack first
16:34:44 hansmoleman rather than spread across all hosts
16:35:04 jgwentworth yep that
16:35:33 hansmoleman mxevgenis: see https://docs.openstack.org/nova/latest/configuration/config.html#filter_scheduler.host_subset_size
16:38:08 mxevgenis thanks a lot. However when i launched many instances in order to consume all the available resources and force it to use the rest node, i got the error no valid host. Which means there are no extra resources. Is it normal considering the problem you mentioned?
16:39:01 hansmoleman low-hanging-fruit for someone https://bugs.launchpad.net/nova/+bug/1774676

Earlier   Later