Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-08-15
11:00:44 stephenfin compute nodes and controllers (well, scheduler hosts)
11:00:47 sean-k-mooney but it would have to be if you were to use
11:00:54 sean-k-mooney it to contol the reshape
11:01:09 stephenfin I'm tempted to drop the config option, to be honest
11:01:26 sean-k-mooney well the translation is on by default
11:01:43 stephenfin Yup. I'd rather we didn't support turning it off again
11:01:44 sean-k-mooney so im ok with saying that it should nver be set on a new host
11:02:15 stephenfin You just need to make sure you upgrade sufficient hosts at once or you'll see degraded scheduling performance due to lack of viable hosts
11:02:27 stephenfin *a sufficient number of compute nodes at once
11:02:29 sean-k-mooney e.g. you should only use it if and only if you want to first upgrade to train then when all node are on train enable this feature
11:03:17 stephenfin so you'd rather keep the config option and make it global?
11:03:21 sean-k-mooney we could remove it but it there for host with vms that cant be migrated
11:03:40 stephenfin why not? The hosts just have to be updated
11:03:44 sean-k-mooney it cant really be global unless you mean set in ever host
11:03:58 stephenfin I do. I think that's the only way this could work
11:04:06 sean-k-mooney ya
11:04:17 sean-k-mooney so if we keep it i think it needs to be set on all host
11:04:24 sean-k-mooney and your reshap code needs to check it
11:04:32 stephenfin It would need to toggle (a) flavor translation, (b) whether we report PCPUs (since that's a heuristic we rely on to detect if reshapes are needed) and (c) whether we reshape
11:04:44 stephenfin Is it worth the effort?
11:04:48 sean-k-mooney and we need to document that it must be set too the same value on all hosts
11:04:53 stephenfin Yup
11:05:10 sean-k-mooney can you support inplacem upgrade without live migration without it
11:05:23 stephenfin and turned off again on the controller first
11:06:30 stephenfin otherwise the compute nodes will have reshaped everything but the scheduler will still be making requests for VCPUs
11:06:51 sean-k-mooney the workflow you be. set workatound on all hosts-> upgrade->disbale workaound on all hosts. and yes it would have to toggel all those things
11:07:09 openstackgerrit Balazs Gibizer proposed openstack/nova master: Support reverting migration / resize with bandwidth https://review.opendev.org/676140
11:07:10 openstackgerrit Balazs Gibizer proposed openstack/nova master: Allow migrating server with port resource request https://review.opendev.org/671497
11:07:15 stephenfin "can you support inplacem upgrade without live migration without it" what do you mean?
11:07:29 sean-k-mooney in which case we shoudl renable it to disable_pcpus_in_placment or something
11:07:36 sean-k-mooney i mean for edge deployment
11:08:03 sean-k-mooney where we may not ever be able to cold or live migrate instances
11:08:32 sean-k-mooney for those deployments wee need to supprot inplace updates
11:08:47 sean-k-mooney and we need to support updating differen edge cells at differnet times
11:09:52 sean-k-mooney does that make sense?
11:12:10 openstackgerrit YumengBao proposed openstack/nova master: [WIP] add cyborg tempest job https://review.opendev.org/670999
11:12:31 stephenfin sean-k-mooney: Why would I want to live migrate an instance as part of the upgrade though?
11:14:00 stephenfin I pull in the latest packages, I restart the controllers followed by compute nodes, the controllers start requesting PCPUs instead of VCPUs and the compute nodes reshape their allocations so pinned instances are consuming PCPUs instead of VCPUs
11:14:08 sean-k-mooney you dont but we cant assume that you will upgrade part of the host first. allow them to reshape and then upgrade the rest
11:14:43 stephenfin Again, not sure how that's an issue
11:14:49 sean-k-mooney in edge deployment that is not a thing as we dont have spare hosts and we cant assume we can do a rolling upgrade of subset of hosts within an edge site
11:15:12 stephenfin what would they be doing instead of a rolling upgrade?
11:15:53 sean-k-mooney the upgrade all nodes within an edge site together and fail over traffic to a different edgesite
11:16:08 sean-k-mooney but that does not mean that all edge site will be upstad at the same time.
11:16:13 stephenfin Why would that be an issue?
11:16:18 stephenfin Oh, that on the other hand, yea
11:16:20 stephenfin *yeah
11:16:49 sean-k-mooney they do a rolling update at the site level rather then host level at least in some cases
11:17:13 sean-k-mooney so for an edge deployment you would have to update all edge sites to disabel the workaround
11:17:17 stephenfin Do those different sites share controllers or is there a controller per site?
11:17:50 sean-k-mooney i think that depends. i think each site is a different cell but they all share the same schduler
11:17:50 stephenfin *scheduler
11:18:06 stephenfin then we're kind of stuck whatever way we go
11:18:28 stephenfin if we have a config option, then we need to make sure every site has the config option toggled at the same time or things break
11:18:33 sean-k-mooney well the 3 step update with the config option will work
11:18:52 sean-k-mooney yes
11:19:10 stephenfin well, things don't break but you won't be able to schedule instances to hosts that still have the config option toggled
11:19:18 sean-k-mooney you could do it another way
11:19:18 stephenfin because they won't be reporting PCPU inventory
11:19:22 stephenfin shoot
11:19:51 sean-k-mooney we could use host_aggreate metadata to contol it instead
11:21:52 stephenfin Hmm, not sure how that helps
11:21:52 sean-k-mooney that would requrie the traslation to be moved to a prefilter proably
11:22:13 sean-k-mooney it would make the traslation non global.
11:22:38 sean-k-mooney its not ideal however
11:22:54 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Add new default roles in os-services API policies https://review.opendev.org/648480
11:23:11 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Add test coverage of existing admin_actions policies https://review.opendev.org/657698
11:23:32 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Ensure we pass a target in admin actions https://review.opendev.org/663095
11:23:45 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Introduce scope_types in Admin Actions https://review.opendev.org/657823
11:24:14 sean-k-mooney stephenfin: i would be good to get dansmith opipion on how much of an issue the edge deplopment case is likely to be
11:24:43 stephenfin Yeah, we need to have a discussion on this. I'll start a mailing list post about it
11:26:00 sean-k-mooney even in an edge deployment the n=>n+1 version compatiable still applies so we would expect edge sites to be updated at the same cadance as teh main datacenter
11:27:39 openstackgerrit Balazs Gibizer proposed openstack/nova stable/pike: Stabilize unshelve notification sample tests https://review.opendev.org/676677
11:27:42 sean-k-mooney belmoreira: you work at cern correct? or am i mixing you your irc nic?
11:28:16 belmoreira sean-k-mooney: yes I do
11:29:05 sean-k-mooney good :) am ye have a hevaily multi cell deployment. how do ye go about upgradeign cells
11:29:27 sean-k-mooney do ye do upgrades cell by cell or do ye do it some other way
11:30:18 sean-k-mooney im just wondering how long it takes ye to go form all host on version N to all on N+1 and if multiple cells alters that
11:31:35 sean-k-mooney i dont know if ye have done a major version update since ye moved to cellsv2 but it might be good input for stephenfin cpu pinning in plamcent issue
11:31:36 belmoreira I upgrade all the control plane! nova-api, nova-scheduler, nova-conductor "super", nova-conductor "cells", ... (This has downtime! ) later upgrade compute-nodes
11:32:12 sean-k-mooney ok but you do all the contol plane including the cell conductors all in one go
11:32:23 sean-k-mooney then upgrade the computes after
11:32:26 belmoreira yes
11:32:55 sean-k-mooney and would setting a config uption on each host before the upgrade be problematic for you in that case?
11:33:08 sean-k-mooney i guess it would be annoying but it should be managable?
11:33:26 belmoreira what do you mean?
11:33:53 sean-k-mooney for cpu pinning in placment there is currently a workaound config option to disable using the new resouce class
11:34:19 sean-k-mooney if we keep it then you would have to set that on all hosts. upgrade to train and only disable it after all host are on train
11:34:28 openstackgerrit Balazs Gibizer proposed openstack/nova stable/pike: Stabilize unshelve notification sample tests https://review.opendev.org/676677
11:35:27 belmoreira that looks reasonable. I don't have a big issue with that
11:36:15 sean-k-mooney roughly how long do your upgrades take end to end after you have done contol plane
11:36:30 belmoreira I'm still not familiar with the changes for the cpu pinning for train
11:36:32 sean-k-mooney would it be days/week or are we talking a few months
11:37:07 sean-k-mooney belmoreira: they allow pinned and unpinned guest on the same host and move counting pinned core to placement
11:37:31 openstackgerrit Chris Dent proposed openstack/nova master: add InstanceList.get_all_uuids_by_hosts() method https://review.opendev.org/623557
11:37:32 openstackgerrit Chris Dent proposed openstack/nova master: single pass instance info fetch in host manager https://review.opendev.org/623558
11:37:33 openstackgerrit Chris Dent proposed openstack/nova master: DNM: add profiling to scheduler https://review.opendev.org/674810
11:38:02 belmoreira in my case the control plane upgrade is less that ~1 hour, then all the compute nodes are upgraded during the next 24h.
11:38:45 sean-k-mooney belmoreira: to allow pinned and unpinned guest on the same host we need to have two resouce classes in placement instead of 1 vCPUs(shared) and pCPUs(dedicated) and as part of that we need too update exsiting allcoation to the new type
11:38:50 sean-k-mooney cool

Earlier   Later