| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-08-15 | |||
| 08:54:37 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Add test coverage of existing admin_actions policies https://review.opendev.org/657698 | |
| 08:54:55 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Ensure we pass a target in admin actions https://review.opendev.org/663095 | |
| 09:03:54 | gibi | sean-k-mooney: thanks | |
| 09:14:06 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Add test coverage of existing os-agents policies https://review.opendev.org/669578 | |
| 09:19:28 | gibi | gmann: hi! If you have time, could you give your oppinion in https://review.opendev.org/#/c/675371/ ? | |
| 09:23:10 | openstackgerrit | Merged openstack/nova master: Enhance SDK fixture for 0.34.0 https://review.opendev.org/676495 | |
| 09:39:14 | gmann | gibi: sure, i will check. | |
| 09:42:24 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Add new default roles and mapping in policy base class https://review.opendev.org/645452 | |
| 09:42:57 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Add new default roles in os-services API policies https://review.opendev.org/648480 | |
| 09:54:17 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Add test coverage of existing admin_actions policies https://review.opendev.org/657698 | |
| 09:54:38 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Ensure we pass a target in admin actions https://review.opendev.org/663095 | |
| 10:14:45 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Introduce scope_types in Admin Actions https://review.opendev.org/657823 | |
| 10:19:49 | openstackgerrit | Guo Jingyu proposed openstack/nova master: define and catch InstanceGuestAgentNotRunning for set-password failure https://review.opendev.org/676667 | |
| 10:39:11 | openstackgerrit | Guo Jingyu proposed openstack/nova master: define and catch InstanceAgentNotRunning for set-password failure https://review.opendev.org/676667 | |
| 10:42:40 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Add policy deprecation fixture and Suppress warnings in tests https://review.opendev.org/676670 | |
| 10:44:15 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Add new default roles in os-services API policies https://review.opendev.org/648480 | |
| 10:44:27 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Add new default roles in os-services API policies https://review.opendev.org/648480 | |
| 10:48:42 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Add new default roles in os-services API policies https://review.opendev.org/648480 | |
| 10:53:45 | openstackgerrit | Chris Dent proposed openstack/nova master: Add a "Caveats" section to the eventlet profiling docs https://review.opendev.org/676672 | |
| 10:56:39 | sean-k-mooney | what time does matt normally log on? about 1-2 hours form now? | |
| 10:57:22 | stephenfin | sean-k-mooney: We have a conflict with cpu-resources | |
| 10:57:23 | sean-k-mooney | i started early today so it feels weird to have been online for a while and not see the us folks about | |
| 10:57:34 | sean-k-mooney | we do? | |
| 10:57:44 | sean-k-mooney | with what | |
| 10:58:17 | stephenfin | There's a workaround option to disable VCPU -> PCPU flavor translation and we've decided to report both PCPUs and VCPUs for compute nodes without explicit CPU pinset configuration | |
| 10:58:33 | sean-k-mooney | yes | |
| 10:58:42 | sean-k-mooney | i was talking about this yesterday | |
| 10:58:47 | stephenfin | If I reshape, I move every allocation for pinned instances on that host from VCPUs to PCPUs | |
| 10:58:59 | sean-k-mooney | right | |
| 10:59:19 | sean-k-mooney | which is why i raised that workaround option as an issue | |
| 10:59:23 | stephenfin | But if you enable the workaround option (so your pinned instances continue to request VCPUs), more pinned instances will land on that host that use VCPUs instead of PCPUs | |
| 10:59:39 | sean-k-mooney | yes | |
| 10:59:48 | stephenfin | so those two things can't coexist | |
| 11:00:05 | sean-k-mooney | yes you cant reshap until you disabel the workaround | |
| 11:00:27 | stephenfin | yup, and the workaround needs to be set on every single host, and toggled on every host at the same time | |
| 11:00:41 | sean-k-mooney | well currently its only set on the schduler | |
| 11:00:44 | stephenfin | compute nodes and controllers (well, scheduler hosts) | |
| 11:00:47 | sean-k-mooney | but it would have to be if you were to use | |
| 11:00:54 | sean-k-mooney | it to contol the reshape | |
| 11:01:09 | stephenfin | I'm tempted to drop the config option, to be honest | |
| 11:01:26 | sean-k-mooney | well the translation is on by default | |
| 11:01:43 | stephenfin | Yup. I'd rather we didn't support turning it off again | |
| 11:01:44 | sean-k-mooney | so im ok with saying that it should nver be set on a new host | |
| 11:02:15 | stephenfin | You just need to make sure you upgrade sufficient hosts at once or you'll see degraded scheduling performance due to lack of viable hosts | |
| 11:02:27 | stephenfin | *a sufficient number of compute nodes at once | |
| 11:02:29 | sean-k-mooney | e.g. you should only use it if and only if you want to first upgrade to train then when all node are on train enable this feature | |
| 11:03:17 | stephenfin | so you'd rather keep the config option and make it global? | |
| 11:03:21 | sean-k-mooney | we could remove it but it there for host with vms that cant be migrated | |
| 11:03:40 | stephenfin | why not? The hosts just have to be updated | |
| 11:03:44 | sean-k-mooney | it cant really be global unless you mean set in ever host | |
| 11:03:58 | stephenfin | I do. I think that's the only way this could work | |
| 11:04:06 | sean-k-mooney | ya | |
| 11:04:17 | sean-k-mooney | so if we keep it i think it needs to be set on all host | |
| 11:04:24 | sean-k-mooney | and your reshap code needs to check it | |
| 11:04:32 | stephenfin | It would need to toggle (a) flavor translation, (b) whether we report PCPUs (since that's a heuristic we rely on to detect if reshapes are needed) and (c) whether we reshape | |
| 11:04:44 | stephenfin | Is it worth the effort? | |
| 11:04:48 | sean-k-mooney | and we need to document that it must be set too the same value on all hosts | |
| 11:04:53 | stephenfin | Yup | |
| 11:05:10 | sean-k-mooney | can you support inplacem upgrade without live migration without it | |
| 11:05:23 | stephenfin | and turned off again on the controller first | |
| 11:06:30 | stephenfin | otherwise the compute nodes will have reshaped everything but the scheduler will still be making requests for VCPUs | |
| 11:06:51 | sean-k-mooney | the workflow you be. set workatound on all hosts-> upgrade->disbale workaound on all hosts. and yes it would have to toggel all those things | |
| 11:07:09 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Support reverting migration / resize with bandwidth https://review.opendev.org/676140 | |
| 11:07:10 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Allow migrating server with port resource request https://review.opendev.org/671497 | |
| 11:07:15 | stephenfin | "can you support inplacem upgrade without live migration without it" what do you mean? | |
| 11:07:29 | sean-k-mooney | in which case we shoudl renable it to disable_pcpus_in_placment or something | |
| 11:07:36 | sean-k-mooney | i mean for edge deployment | |
| 11:08:03 | sean-k-mooney | where we may not ever be able to cold or live migrate instances | |
| 11:08:32 | sean-k-mooney | for those deployments wee need to supprot inplace updates | |
| 11:08:47 | sean-k-mooney | and we need to support updating differen edge cells at differnet times | |
| 11:09:52 | sean-k-mooney | does that make sense? | |
| 11:12:10 | openstackgerrit | YumengBao proposed openstack/nova master: [WIP] add cyborg tempest job https://review.opendev.org/670999 | |
| 11:12:31 | stephenfin | sean-k-mooney: Why would I want to live migrate an instance as part of the upgrade though? | |
| 11:14:00 | stephenfin | I pull in the latest packages, I restart the controllers followed by compute nodes, the controllers start requesting PCPUs instead of VCPUs and the compute nodes reshape their allocations so pinned instances are consuming PCPUs instead of VCPUs | |
| 11:14:08 | sean-k-mooney | you dont but we cant assume that you will upgrade part of the host first. allow them to reshape and then upgrade the rest | |
| 11:14:43 | stephenfin | Again, not sure how that's an issue | |
| 11:14:49 | sean-k-mooney | in edge deployment that is not a thing as we dont have spare hosts and we cant assume we can do a rolling upgrade of subset of hosts within an edge site | |
| 11:15:12 | stephenfin | what would they be doing instead of a rolling upgrade? | |
| 11:15:53 | sean-k-mooney | the upgrade all nodes within an edge site together and fail over traffic to a different edgesite | |
| 11:16:08 | sean-k-mooney | but that does not mean that all edge site will be upstad at the same time. | |
| 11:16:13 | stephenfin | Why would that be an issue? | |
| 11:16:18 | stephenfin | Oh, that on the other hand, yea | |
| 11:16:20 | stephenfin | *yeah | |
| 11:16:49 | sean-k-mooney | they do a rolling update at the site level rather then host level at least in some cases | |
| 11:17:13 | sean-k-mooney | so for an edge deployment you would have to update all edge sites to disabel the workaround | |
| 11:17:17 | stephenfin | Do those different sites share controllers or is there a controller per site? | |
| 11:17:50 | sean-k-mooney | i think that depends. i think each site is a different cell but they all share the same schduler | |
| 11:17:50 | stephenfin | *scheduler | |
| 11:18:06 | stephenfin | then we're kind of stuck whatever way we go | |
| 11:18:28 | stephenfin | if we have a config option, then we need to make sure every site has the config option toggled at the same time or things break | |
| 11:18:33 | sean-k-mooney | well the 3 step update with the config option will work | |
| 11:18:52 | sean-k-mooney | yes | |
| 11:19:10 | stephenfin | well, things don't break but you won't be able to schedule instances to hosts that still have the config option toggled | |
| 11:19:18 | sean-k-mooney | you could do it another way | |
| 11:19:18 | stephenfin | because they won't be reporting PCPU inventory | |
| 11:19:22 | stephenfin | shoot | |
| 11:19:51 | sean-k-mooney | we could use host_aggreate metadata to contol it instead | |
| 11:21:52 | stephenfin | Hmm, not sure how that helps | |
| 11:21:52 | sean-k-mooney | that would requrie the traslation to be moved to a prefilter proably | |
| 11:22:13 | sean-k-mooney | it would make the traslation non global. | |