Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-08-15
11:12:31 stephenfin sean-k-mooney: Why would I want to live migrate an instance as part of the upgrade though?
11:14:00 stephenfin I pull in the latest packages, I restart the controllers followed by compute nodes, the controllers start requesting PCPUs instead of VCPUs and the compute nodes reshape their allocations so pinned instances are consuming PCPUs instead of VCPUs
11:14:08 sean-k-mooney you dont but we cant assume that you will upgrade part of the host first. allow them to reshape and then upgrade the rest
11:14:43 stephenfin Again, not sure how that's an issue
11:14:49 sean-k-mooney in edge deployment that is not a thing as we dont have spare hosts and we cant assume we can do a rolling upgrade of subset of hosts within an edge site
11:15:12 stephenfin what would they be doing instead of a rolling upgrade?
11:15:53 sean-k-mooney the upgrade all nodes within an edge site together and fail over traffic to a different edgesite
11:16:08 sean-k-mooney but that does not mean that all edge site will be upstad at the same time.
11:16:13 stephenfin Why would that be an issue?
11:16:18 stephenfin Oh, that on the other hand, yea
11:16:20 stephenfin *yeah
11:16:49 sean-k-mooney they do a rolling update at the site level rather then host level at least in some cases
11:17:13 sean-k-mooney so for an edge deployment you would have to update all edge sites to disabel the workaround
11:17:17 stephenfin Do those different sites share controllers or is there a controller per site?
11:17:50 sean-k-mooney i think that depends. i think each site is a different cell but they all share the same schduler
11:17:50 stephenfin *scheduler
11:18:06 stephenfin then we're kind of stuck whatever way we go
11:18:28 stephenfin if we have a config option, then we need to make sure every site has the config option toggled at the same time or things break
11:18:33 sean-k-mooney well the 3 step update with the config option will work
11:18:52 sean-k-mooney yes
11:19:10 stephenfin well, things don't break but you won't be able to schedule instances to hosts that still have the config option toggled
11:19:18 sean-k-mooney you could do it another way
11:19:18 stephenfin because they won't be reporting PCPU inventory
11:19:22 stephenfin shoot
11:19:51 sean-k-mooney we could use host_aggreate metadata to contol it instead
11:21:52 stephenfin Hmm, not sure how that helps
11:21:52 sean-k-mooney that would requrie the traslation to be moved to a prefilter proably
11:22:13 sean-k-mooney it would make the traslation non global.
11:22:38 sean-k-mooney its not ideal however
11:22:54 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Add new default roles in os-services API policies https://review.opendev.org/648480
11:23:11 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Add test coverage of existing admin_actions policies https://review.opendev.org/657698
11:23:32 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Ensure we pass a target in admin actions https://review.opendev.org/663095
11:23:45 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Introduce scope_types in Admin Actions https://review.opendev.org/657823
11:24:14 sean-k-mooney stephenfin: i would be good to get dansmith opipion on how much of an issue the edge deplopment case is likely to be
11:24:43 stephenfin Yeah, we need to have a discussion on this. I'll start a mailing list post about it
11:26:00 sean-k-mooney even in an edge deployment the n=>n+1 version compatiable still applies so we would expect edge sites to be updated at the same cadance as teh main datacenter
11:27:39 openstackgerrit Balazs Gibizer proposed openstack/nova stable/pike: Stabilize unshelve notification sample tests https://review.opendev.org/676677
11:27:42 sean-k-mooney belmoreira: you work at cern correct? or am i mixing you your irc nic?
11:28:16 belmoreira sean-k-mooney: yes I do
11:29:05 sean-k-mooney good :) am ye have a hevaily multi cell deployment. how do ye go about upgradeign cells
11:29:27 sean-k-mooney do ye do upgrades cell by cell or do ye do it some other way
11:30:18 sean-k-mooney im just wondering how long it takes ye to go form all host on version N to all on N+1 and if multiple cells alters that
11:31:35 sean-k-mooney i dont know if ye have done a major version update since ye moved to cellsv2 but it might be good input for stephenfin cpu pinning in plamcent issue
11:31:36 belmoreira I upgrade all the control plane! nova-api, nova-scheduler, nova-conductor "super", nova-conductor "cells", ... (This has downtime! ) later upgrade compute-nodes
11:32:12 sean-k-mooney ok but you do all the contol plane including the cell conductors all in one go
11:32:23 sean-k-mooney then upgrade the computes after
11:32:26 belmoreira yes
11:32:55 sean-k-mooney and would setting a config uption on each host before the upgrade be problematic for you in that case?
11:33:08 sean-k-mooney i guess it would be annoying but it should be managable?
11:33:26 belmoreira what do you mean?
11:33:53 sean-k-mooney for cpu pinning in placment there is currently a workaound config option to disable using the new resouce class
11:34:19 sean-k-mooney if we keep it then you would have to set that on all hosts. upgrade to train and only disable it after all host are on train
11:34:28 openstackgerrit Balazs Gibizer proposed openstack/nova stable/pike: Stabilize unshelve notification sample tests https://review.opendev.org/676677
11:35:27 belmoreira that looks reasonable. I don't have a big issue with that
11:36:15 sean-k-mooney roughly how long do your upgrades take end to end after you have done contol plane
11:36:30 belmoreira I'm still not familiar with the changes for the cpu pinning for train
11:36:32 sean-k-mooney would it be days/week or are we talking a few months
11:37:07 sean-k-mooney belmoreira: they allow pinned and unpinned guest on the same host and move counting pinned core to placement
11:37:31 openstackgerrit Chris Dent proposed openstack/nova master: add InstanceList.get_all_uuids_by_hosts() method https://review.opendev.org/623557
11:37:32 openstackgerrit Chris Dent proposed openstack/nova master: single pass instance info fetch in host manager https://review.opendev.org/623558
11:37:33 openstackgerrit Chris Dent proposed openstack/nova master: DNM: add profiling to scheduler https://review.opendev.org/674810
11:38:02 belmoreira in my case the control plane upgrade is less that ~1 hour, then all the compute nodes are upgraded during the next 24h.
11:38:45 sean-k-mooney belmoreira: to allow pinned and unpinned guest on the same host we need to have two resouce classes in placement instead of 1 vCPUs(shared) and pCPUs(dedicated) and as part of that we need too update exsiting allcoation to the new type
11:38:50 sean-k-mooney cool
11:39:09 sean-k-mooney so ya that updating to the new type is what i was discussing with stephenfin
11:39:20 sean-k-mooney and what teh config option would be contolling
11:40:15 sean-k-mooney stephenfin: weel there is one data point but still good to bring this up on the list and bug dansmith about when he is online
11:40:26 belmoreira and what you are suggesting is to keep it disabled (config option) and then only enable when everything is upgraded
11:40:35 sean-k-mooney yes
11:41:35 belmoreira in that case I may suggest that the default of the config is "disabled" just in case the operator is not aware
11:45:51 sean-k-mooney you may. we were hoping to have the feature enable by defualt so that it just worked for new deployment but i guess we could handel that in installers
11:46:25 sean-k-mooney it would simplify things
11:46:34 sean-k-mooney if it was disabel by default
11:46:46 sean-k-mooney as you would just upgrade normally
11:47:01 sean-k-mooney the set teh config option if you wanted the feature.
11:47:34 sean-k-mooney i guess we could also cahge the default in U and remove the option in V
11:47:59 sean-k-mooney so it give people 3 cycle to switch instead of 1 which is what we were planning
11:48:09 sean-k-mooney stephenfin: ^ thoughts?
11:48:56 stephenfin Let me finish that email and then I'll think about this
11:49:41 sean-k-mooney tl;dr global config pcpus_in_placement=False by default in train change to true in U and remove in V. which completely disables the feature
11:50:00 sean-k-mooney once you set it to true you do it globally and cant set it back to false
11:50:47 sean-k-mooney in the V release we drop all support fo deploying without it enabled. in U its on by default and train its off by default
11:51:19 sean-k-mooney then we modify installer to enable it by defualt for new installs in train
11:58:13 mnaser if i am not wrong, it seems like we're installing `psycopg2` in test-requirements.txt for no reason?
11:58:31 mnaser i cant find any psycopg2 references except in test-requirements?
11:59:10 sean-k-mooney it used if you are using postgres via sqlalcambe for you db i think
11:59:34 stephenfin Do we still support that? There was talk of dropping it some time back, though cfriesen had objections
11:59:35 mnaser right but you'd have to install that yourself (i.e. it shouldn't be inside test-requirements.txt in openstack/nova ?)
11:59:52 mnaser oh i found this https://review.opendev.org/#/c/296604/
12:00:31 stephenfin What's the opportunistic testing he refers to?
12:00:51 mnaser this is 3 years ago, but i think there's a tc patch saying we dont support postgres.
12:01:03 sean-k-mooney we have a db mixing that will use mysql or postgress instead of sqlitght if it avioable i think
12:01:33 cdent mnaser: that patch never landed in full
12:01:34 sean-k-mooney mnaser: there was discussion or it yes
12:01:42 mnaser ah, ok.
12:01:48 cdent instead we decided that we wouldn't explicitly disallow it
12:02:02 mnaser i'm just annoyed by it because i just had tox fail on me cause i didnt have postgres development headers..
12:02:03 cdent instead projects could try to use sqlachemy in the way it was intended if they like
12:02:15 sean-k-mooney mnaser: are you just annoyed by the fact that requires use to install the postgres client and complie it
12:02:21 sean-k-mooney and that breaks on stable

Earlier   Later