Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-08-15
10:14:45 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Introduce scope_types in Admin Actions https://review.opendev.org/657823
10:19:49 openstackgerrit Guo Jingyu proposed openstack/nova master: define and catch InstanceGuestAgentNotRunning for set-password failure https://review.opendev.org/676667
10:39:11 openstackgerrit Guo Jingyu proposed openstack/nova master: define and catch InstanceAgentNotRunning for set-password failure https://review.opendev.org/676667
10:42:40 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Add policy deprecation fixture and Suppress warnings in tests https://review.opendev.org/676670
10:44:15 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Add new default roles in os-services API policies https://review.opendev.org/648480
10:44:27 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Add new default roles in os-services API policies https://review.opendev.org/648480
10:48:42 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Add new default roles in os-services API policies https://review.opendev.org/648480
10:53:45 openstackgerrit Chris Dent proposed openstack/nova master: Add a "Caveats" section to the eventlet profiling docs https://review.opendev.org/676672
10:56:39 sean-k-mooney what time does matt normally log on? about 1-2 hours form now?
10:57:22 stephenfin sean-k-mooney: We have a conflict with cpu-resources
10:57:23 sean-k-mooney i started early today so it feels weird to have been online for a while and not see the us folks about
10:57:34 sean-k-mooney we do?
10:57:44 sean-k-mooney with what
10:58:17 stephenfin There's a workaround option to disable VCPU -> PCPU flavor translation and we've decided to report both PCPUs and VCPUs for compute nodes without explicit CPU pinset configuration
10:58:33 sean-k-mooney yes
10:58:42 sean-k-mooney i was talking about this yesterday
10:58:47 stephenfin If I reshape, I move every allocation for pinned instances on that host from VCPUs to PCPUs
10:58:59 sean-k-mooney right
10:59:19 sean-k-mooney which is why i raised that workaround option as an issue
10:59:23 stephenfin But if you enable the workaround option (so your pinned instances continue to request VCPUs), more pinned instances will land on that host that use VCPUs instead of PCPUs
10:59:39 sean-k-mooney yes
10:59:48 stephenfin so those two things can't coexist
11:00:05 sean-k-mooney yes you cant reshap until you disabel the workaround
11:00:27 stephenfin yup, and the workaround needs to be set on every single host, and toggled on every host at the same time
11:00:41 sean-k-mooney well currently its only set on the schduler
11:00:44 stephenfin compute nodes and controllers (well, scheduler hosts)
11:00:47 sean-k-mooney but it would have to be if you were to use
11:00:54 sean-k-mooney it to contol the reshape
11:01:09 stephenfin I'm tempted to drop the config option, to be honest
11:01:26 sean-k-mooney well the translation is on by default
11:01:43 stephenfin Yup. I'd rather we didn't support turning it off again
11:01:44 sean-k-mooney so im ok with saying that it should nver be set on a new host
11:02:15 stephenfin You just need to make sure you upgrade sufficient hosts at once or you'll see degraded scheduling performance due to lack of viable hosts
11:02:27 stephenfin *a sufficient number of compute nodes at once
11:02:29 sean-k-mooney e.g. you should only use it if and only if you want to first upgrade to train then when all node are on train enable this feature
11:03:17 stephenfin so you'd rather keep the config option and make it global?
11:03:21 sean-k-mooney we could remove it but it there for host with vms that cant be migrated
11:03:40 stephenfin why not? The hosts just have to be updated
11:03:44 sean-k-mooney it cant really be global unless you mean set in ever host
11:03:58 stephenfin I do. I think that's the only way this could work
11:04:06 sean-k-mooney ya
11:04:17 sean-k-mooney so if we keep it i think it needs to be set on all host
11:04:24 sean-k-mooney and your reshap code needs to check it
11:04:32 stephenfin It would need to toggle (a) flavor translation, (b) whether we report PCPUs (since that's a heuristic we rely on to detect if reshapes are needed) and (c) whether we reshape
11:04:44 stephenfin Is it worth the effort?
11:04:48 sean-k-mooney and we need to document that it must be set too the same value on all hosts
11:04:53 stephenfin Yup
11:05:10 sean-k-mooney can you support inplacem upgrade without live migration without it
11:05:23 stephenfin and turned off again on the controller first
11:06:30 stephenfin otherwise the compute nodes will have reshaped everything but the scheduler will still be making requests for VCPUs
11:06:51 sean-k-mooney the workflow you be. set workatound on all hosts-> upgrade->disbale workaound on all hosts. and yes it would have to toggel all those things
11:07:09 openstackgerrit Balazs Gibizer proposed openstack/nova master: Support reverting migration / resize with bandwidth https://review.opendev.org/676140
11:07:10 openstackgerrit Balazs Gibizer proposed openstack/nova master: Allow migrating server with port resource request https://review.opendev.org/671497
11:07:15 stephenfin "can you support inplacem upgrade without live migration without it" what do you mean?
11:07:29 sean-k-mooney in which case we shoudl renable it to disable_pcpus_in_placment or something
11:07:36 sean-k-mooney i mean for edge deployment
11:08:03 sean-k-mooney where we may not ever be able to cold or live migrate instances
11:08:32 sean-k-mooney for those deployments wee need to supprot inplace updates
11:08:47 sean-k-mooney and we need to support updating differen edge cells at differnet times
11:09:52 sean-k-mooney does that make sense?
11:12:10 openstackgerrit YumengBao proposed openstack/nova master: [WIP] add cyborg tempest job https://review.opendev.org/670999
11:12:31 stephenfin sean-k-mooney: Why would I want to live migrate an instance as part of the upgrade though?
11:14:00 stephenfin I pull in the latest packages, I restart the controllers followed by compute nodes, the controllers start requesting PCPUs instead of VCPUs and the compute nodes reshape their allocations so pinned instances are consuming PCPUs instead of VCPUs
11:14:08 sean-k-mooney you dont but we cant assume that you will upgrade part of the host first. allow them to reshape and then upgrade the rest
11:14:43 stephenfin Again, not sure how that's an issue
11:14:49 sean-k-mooney in edge deployment that is not a thing as we dont have spare hosts and we cant assume we can do a rolling upgrade of subset of hosts within an edge site
11:15:12 stephenfin what would they be doing instead of a rolling upgrade?
11:15:53 sean-k-mooney the upgrade all nodes within an edge site together and fail over traffic to a different edgesite
11:16:08 sean-k-mooney but that does not mean that all edge site will be upstad at the same time.
11:16:13 stephenfin Why would that be an issue?
11:16:18 stephenfin Oh, that on the other hand, yea
11:16:20 stephenfin *yeah
11:16:49 sean-k-mooney they do a rolling update at the site level rather then host level at least in some cases
11:17:13 sean-k-mooney so for an edge deployment you would have to update all edge sites to disabel the workaround
11:17:17 stephenfin Do those different sites share controllers or is there a controller per site?
11:17:50 sean-k-mooney i think that depends. i think each site is a different cell but they all share the same schduler
11:17:50 stephenfin *scheduler
11:18:06 stephenfin then we're kind of stuck whatever way we go
11:18:28 stephenfin if we have a config option, then we need to make sure every site has the config option toggled at the same time or things break
11:18:33 sean-k-mooney well the 3 step update with the config option will work
11:18:52 sean-k-mooney yes
11:19:10 stephenfin well, things don't break but you won't be able to schedule instances to hosts that still have the config option toggled
11:19:18 sean-k-mooney you could do it another way
11:19:18 stephenfin because they won't be reporting PCPU inventory
11:19:22 stephenfin shoot
11:19:51 sean-k-mooney we could use host_aggreate metadata to contol it instead
11:21:52 stephenfin Hmm, not sure how that helps
11:21:52 sean-k-mooney that would requrie the traslation to be moved to a prefilter proably
11:22:13 sean-k-mooney it would make the traslation non global.
11:22:38 sean-k-mooney its not ideal however
11:22:54 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Add new default roles in os-services API policies https://review.opendev.org/648480
11:23:11 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Add test coverage of existing admin_actions policies https://review.opendev.org/657698
11:23:32 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Ensure we pass a target in admin actions https://review.opendev.org/663095
11:23:45 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Introduce scope_types in Admin Actions https://review.opendev.org/657823
11:24:14 sean-k-mooney stephenfin: i would be good to get dansmith opipion on how much of an issue the edge deplopment case is likely to be
11:24:43 stephenfin Yeah, we need to have a discussion on this. I'll start a mailing list post about it
11:26:00 sean-k-mooney even in an edge deployment the n=>n+1 version compatiable still applies so we would expect edge sites to be updated at the same cadance as teh main datacenter
11:27:39 openstackgerrit Balazs Gibizer proposed openstack/nova stable/pike: Stabilize unshelve notification sample tests https://review.opendev.org/676677
11:27:42 sean-k-mooney belmoreira: you work at cern correct? or am i mixing you your irc nic?
11:28:16 belmoreira sean-k-mooney: yes I do

Earlier   Later