Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-08-15
11:28:16 belmoreira sean-k-mooney: yes I do
11:29:05 sean-k-mooney good :) am ye have a hevaily multi cell deployment. how do ye go about upgradeign cells
11:29:27 sean-k-mooney do ye do upgrades cell by cell or do ye do it some other way
11:30:18 sean-k-mooney im just wondering how long it takes ye to go form all host on version N to all on N+1 and if multiple cells alters that
11:31:35 sean-k-mooney i dont know if ye have done a major version update since ye moved to cellsv2 but it might be good input for stephenfin cpu pinning in plamcent issue
11:31:36 belmoreira I upgrade all the control plane! nova-api, nova-scheduler, nova-conductor "super", nova-conductor "cells", ... (This has downtime! ) later upgrade compute-nodes
11:32:12 sean-k-mooney ok but you do all the contol plane including the cell conductors all in one go
11:32:23 sean-k-mooney then upgrade the computes after
11:32:26 belmoreira yes
11:32:55 sean-k-mooney and would setting a config uption on each host before the upgrade be problematic for you in that case?
11:33:08 sean-k-mooney i guess it would be annoying but it should be managable?
11:33:26 belmoreira what do you mean?
11:33:53 sean-k-mooney for cpu pinning in placment there is currently a workaound config option to disable using the new resouce class
11:34:19 sean-k-mooney if we keep it then you would have to set that on all hosts. upgrade to train and only disable it after all host are on train
11:34:28 openstackgerrit Balazs Gibizer proposed openstack/nova stable/pike: Stabilize unshelve notification sample tests https://review.opendev.org/676677
11:35:27 belmoreira that looks reasonable. I don't have a big issue with that
11:36:15 sean-k-mooney roughly how long do your upgrades take end to end after you have done contol plane
11:36:30 belmoreira I'm still not familiar with the changes for the cpu pinning for train
11:36:32 sean-k-mooney would it be days/week or are we talking a few months
11:37:07 sean-k-mooney belmoreira: they allow pinned and unpinned guest on the same host and move counting pinned core to placement
11:37:31 openstackgerrit Chris Dent proposed openstack/nova master: add InstanceList.get_all_uuids_by_hosts() method https://review.opendev.org/623557
11:37:32 openstackgerrit Chris Dent proposed openstack/nova master: single pass instance info fetch in host manager https://review.opendev.org/623558
11:37:33 openstackgerrit Chris Dent proposed openstack/nova master: DNM: add profiling to scheduler https://review.opendev.org/674810
11:38:02 belmoreira in my case the control plane upgrade is less that ~1 hour, then all the compute nodes are upgraded during the next 24h.
11:38:45 sean-k-mooney belmoreira: to allow pinned and unpinned guest on the same host we need to have two resouce classes in placement instead of 1 vCPUs(shared) and pCPUs(dedicated) and as part of that we need too update exsiting allcoation to the new type
11:38:50 sean-k-mooney cool
11:39:09 sean-k-mooney so ya that updating to the new type is what i was discussing with stephenfin
11:39:20 sean-k-mooney and what teh config option would be contolling
11:40:15 sean-k-mooney stephenfin: weel there is one data point but still good to bring this up on the list and bug dansmith about when he is online
11:40:26 belmoreira and what you are suggesting is to keep it disabled (config option) and then only enable when everything is upgraded
11:40:35 sean-k-mooney yes
11:41:35 belmoreira in that case I may suggest that the default of the config is "disabled" just in case the operator is not aware
11:45:51 sean-k-mooney you may. we were hoping to have the feature enable by defualt so that it just worked for new deployment but i guess we could handel that in installers
11:46:25 sean-k-mooney it would simplify things
11:46:34 sean-k-mooney if it was disabel by default
11:46:46 sean-k-mooney as you would just upgrade normally
11:47:01 sean-k-mooney the set teh config option if you wanted the feature.
11:47:34 sean-k-mooney i guess we could also cahge the default in U and remove the option in V
11:47:59 sean-k-mooney so it give people 3 cycle to switch instead of 1 which is what we were planning
11:48:09 sean-k-mooney stephenfin: ^ thoughts?
11:48:56 stephenfin Let me finish that email and then I'll think about this
11:49:41 sean-k-mooney tl;dr global config pcpus_in_placement=False by default in train change to true in U and remove in V. which completely disables the feature
11:50:00 sean-k-mooney once you set it to true you do it globally and cant set it back to false
11:50:47 sean-k-mooney in the V release we drop all support fo deploying without it enabled. in U its on by default and train its off by default
11:51:19 sean-k-mooney then we modify installer to enable it by defualt for new installs in train
11:58:13 mnaser if i am not wrong, it seems like we're installing `psycopg2` in test-requirements.txt for no reason?
11:58:31 mnaser i cant find any psycopg2 references except in test-requirements?
11:59:10 sean-k-mooney it used if you are using postgres via sqlalcambe for you db i think
11:59:34 stephenfin Do we still support that? There was talk of dropping it some time back, though cfriesen had objections
11:59:35 mnaser right but you'd have to install that yourself (i.e. it shouldn't be inside test-requirements.txt in openstack/nova ?)
11:59:52 mnaser oh i found this https://review.opendev.org/#/c/296604/
12:00:31 stephenfin What's the opportunistic testing he refers to?
12:00:51 mnaser this is 3 years ago, but i think there's a tc patch saying we dont support postgres.
12:01:03 sean-k-mooney we have a db mixing that will use mysql or postgress instead of sqlitght if it avioable i think
12:01:33 cdent mnaser: that patch never landed in full
12:01:34 sean-k-mooney mnaser: there was discussion or it yes
12:01:42 mnaser ah, ok.
12:01:48 cdent instead we decided that we wouldn't explicitly disallow it
12:02:02 mnaser i'm just annoyed by it because i just had tox fail on me cause i didnt have postgres development headers..
12:02:03 cdent instead projects could try to use sqlachemy in the way it was intended if they like
12:02:15 sean-k-mooney mnaser: are you just annoyed by the fact that requires use to install the postgres client and complie it
12:02:21 sean-k-mooney and that breaks on stable
12:02:28 sean-k-mooney ya
12:02:38 mnaser i have to get the postgres client to run tox locally (its inside test-requirements, not requirements)
12:02:39 sean-k-mooney we could replacy it with psycopg2-binary
12:03:06 mnaser oh thats a reasonable suggestion
12:03:32 sean-k-mooney "The binary package is a practical choice for development and testing but in production it is advised to use the package built from sources.£
12:03:44 cdent mnaser: yeah, in placement we did this: https://review.opendev.org/#/c/671249/ which may be a method for possible hacking around things
12:03:47 sean-k-mooney we are using it for testing so i think it makes sense
12:04:08 mnaser oh thats a nice workaround too
12:04:10 mnaser TIL about sys_platform
12:04:25 openstackgerrit Surya Seetharaman proposed openstack/nova master: API microversion 2.76: Add 'power-update' external event https://review.opendev.org/645611
12:04:48 sean-k-mooney cdent: so when you develop on macos you dont run all the db test and leave the gate check it
12:05:02 sean-k-mooney unfortunetly i dont think we will be allowed do the same on linux
12:05:20 sean-k-mooney maybe i should go back ot using windows
12:05:22 cdent sean-k-mooney: the db tests are opportunistic so the migrations are testing on sqlite but not msyql/postgresql
12:06:16 cdent i'm not suggesting that we do exactly what's done in that patch, rather that there might be _some_ strategy for making those requirements optional
12:06:32 sean-k-mooney ya but back to the binary package is there any reason not to jsut use that instead
12:06:53 sean-k-mooney cdent: well we could make it a tox siblig too
12:07:03 sean-k-mooney rather then have it in test requirements
12:12:28 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Add new default roles in Admin Action API policies https://review.opendev.org/676682
12:18:24 openstackgerrit Mohammed Naser proposed openstack/nova master: config: remove deprecated checksum options https://review.opendev.org/676539
12:18:25 openstackgerrit Mohammed Naser proposed openstack/nova master: Remove nova.image.download.modules extension point https://review.opendev.org/676540
12:18:25 openstackgerrit Mohammed Naser proposed openstack/nova master: requirements: move to using psycopg2-binary https://review.opendev.org/676684
12:25:31 stephenfin cdent: That's easy - 'extras'
12:25:50 stephenfin e.g. https://github.com/openstack/oslo.messaging/blob/master/setup.cfg#L22-L29
12:26:32 sean-k-mooney stephenfin ah ya that is what i was tihnking of when i said tox siblings
12:27:27 sean-k-mooney but we would have to be able to use tox -e py37[mysql,postgres] or whatever in the upstream gate
12:27:59 sean-k-mooney if we could not run the tox env with the extras installed then it wont help
12:28:38 stephenfin sean-k-mooney: Not necessarily
12:29:19 stephenfin sean-k-mooney: https://tox.readthedocs.io/en/latest/config.html#conf-extras
12:30:13 sean-k-mooney the goal is to not need to install postgres header locally to be able to run tox
12:30:34 sean-k-mooney but to still enable all the testing with the optional deps in the gate
12:31:47 sean-k-mooney so im not sure that actully helps
12:36:48 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Pass the target in os-services APIs policy https://review.opendev.org/676688
12:38:44 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Introduce scope_types in Admin Actions https://review.opendev.org/657823
12:39:10 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Add new default roles in Admin Action API policies https://review.opendev.org/676682
12:39:28 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Ensure we pass a target in admin actions https://review.opendev.org/663095
13:20:42 gmann melwitt: can you review this. This is followup fix for policy default refresh spec - https://review.opendev.org/#/c/669196/
13:21:44 gmann melwitt: johnthetubaguy also I have prepared the first set of changes and looking for feedback before I proceed on other policy changes.

Earlier   Later