Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-08-15
11:14:43 stephenfin Again, not sure how that's an issue
11:14:49 sean-k-mooney in edge deployment that is not a thing as we dont have spare hosts and we cant assume we can do a rolling upgrade of subset of hosts within an edge site
11:15:12 stephenfin what would they be doing instead of a rolling upgrade?
11:15:53 sean-k-mooney the upgrade all nodes within an edge site together and fail over traffic to a different edgesite
11:16:08 sean-k-mooney but that does not mean that all edge site will be upstad at the same time.
11:16:13 stephenfin Why would that be an issue?
11:16:18 stephenfin Oh, that on the other hand, yea
11:16:20 stephenfin *yeah
11:16:49 sean-k-mooney they do a rolling update at the site level rather then host level at least in some cases
11:17:13 sean-k-mooney so for an edge deployment you would have to update all edge sites to disabel the workaround
11:17:17 stephenfin Do those different sites share controllers or is there a controller per site?
11:17:50 sean-k-mooney i think that depends. i think each site is a different cell but they all share the same schduler
11:17:50 stephenfin *scheduler
11:18:06 stephenfin then we're kind of stuck whatever way we go
11:18:28 stephenfin if we have a config option, then we need to make sure every site has the config option toggled at the same time or things break
11:18:33 sean-k-mooney well the 3 step update with the config option will work
11:18:52 sean-k-mooney yes
11:19:10 stephenfin well, things don't break but you won't be able to schedule instances to hosts that still have the config option toggled
11:19:18 sean-k-mooney you could do it another way
11:19:18 stephenfin because they won't be reporting PCPU inventory
11:19:22 stephenfin shoot
11:19:51 sean-k-mooney we could use host_aggreate metadata to contol it instead
11:21:52 stephenfin Hmm, not sure how that helps
11:21:52 sean-k-mooney that would requrie the traslation to be moved to a prefilter proably
11:22:13 sean-k-mooney it would make the traslation non global.
11:22:38 sean-k-mooney its not ideal however
11:22:54 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Add new default roles in os-services API policies https://review.opendev.org/648480
11:23:11 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Add test coverage of existing admin_actions policies https://review.opendev.org/657698
11:23:32 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Ensure we pass a target in admin actions https://review.opendev.org/663095
11:23:45 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Introduce scope_types in Admin Actions https://review.opendev.org/657823
11:24:14 sean-k-mooney stephenfin: i would be good to get dansmith opipion on how much of an issue the edge deplopment case is likely to be
11:24:43 stephenfin Yeah, we need to have a discussion on this. I'll start a mailing list post about it
11:26:00 sean-k-mooney even in an edge deployment the n=>n+1 version compatiable still applies so we would expect edge sites to be updated at the same cadance as teh main datacenter
11:27:39 openstackgerrit Balazs Gibizer proposed openstack/nova stable/pike: Stabilize unshelve notification sample tests https://review.opendev.org/676677
11:27:42 sean-k-mooney belmoreira: you work at cern correct? or am i mixing you your irc nic?
11:28:16 belmoreira sean-k-mooney: yes I do
11:29:05 sean-k-mooney good :) am ye have a hevaily multi cell deployment. how do ye go about upgradeign cells
11:29:27 sean-k-mooney do ye do upgrades cell by cell or do ye do it some other way
11:30:18 sean-k-mooney im just wondering how long it takes ye to go form all host on version N to all on N+1 and if multiple cells alters that
11:31:35 sean-k-mooney i dont know if ye have done a major version update since ye moved to cellsv2 but it might be good input for stephenfin cpu pinning in plamcent issue
11:31:36 belmoreira I upgrade all the control plane! nova-api, nova-scheduler, nova-conductor "super", nova-conductor "cells", ... (This has downtime! ) later upgrade compute-nodes
11:32:12 sean-k-mooney ok but you do all the contol plane including the cell conductors all in one go
11:32:23 sean-k-mooney then upgrade the computes after
11:32:26 belmoreira yes
11:32:55 sean-k-mooney and would setting a config uption on each host before the upgrade be problematic for you in that case?
11:33:08 sean-k-mooney i guess it would be annoying but it should be managable?
11:33:26 belmoreira what do you mean?
11:33:53 sean-k-mooney for cpu pinning in placment there is currently a workaound config option to disable using the new resouce class
11:34:19 sean-k-mooney if we keep it then you would have to set that on all hosts. upgrade to train and only disable it after all host are on train
11:34:28 openstackgerrit Balazs Gibizer proposed openstack/nova stable/pike: Stabilize unshelve notification sample tests https://review.opendev.org/676677
11:35:27 belmoreira that looks reasonable. I don't have a big issue with that
11:36:15 sean-k-mooney roughly how long do your upgrades take end to end after you have done contol plane
11:36:30 belmoreira I'm still not familiar with the changes for the cpu pinning for train
11:36:32 sean-k-mooney would it be days/week or are we talking a few months
11:37:07 sean-k-mooney belmoreira: they allow pinned and unpinned guest on the same host and move counting pinned core to placement
11:37:31 openstackgerrit Chris Dent proposed openstack/nova master: add InstanceList.get_all_uuids_by_hosts() method https://review.opendev.org/623557
11:37:32 openstackgerrit Chris Dent proposed openstack/nova master: single pass instance info fetch in host manager https://review.opendev.org/623558
11:37:33 openstackgerrit Chris Dent proposed openstack/nova master: DNM: add profiling to scheduler https://review.opendev.org/674810
11:38:02 belmoreira in my case the control plane upgrade is less that ~1 hour, then all the compute nodes are upgraded during the next 24h.
11:38:45 sean-k-mooney belmoreira: to allow pinned and unpinned guest on the same host we need to have two resouce classes in placement instead of 1 vCPUs(shared) and pCPUs(dedicated) and as part of that we need too update exsiting allcoation to the new type
11:38:50 sean-k-mooney cool
11:39:09 sean-k-mooney so ya that updating to the new type is what i was discussing with stephenfin
11:39:20 sean-k-mooney and what teh config option would be contolling
11:40:15 sean-k-mooney stephenfin: weel there is one data point but still good to bring this up on the list and bug dansmith about when he is online
11:40:26 belmoreira and what you are suggesting is to keep it disabled (config option) and then only enable when everything is upgraded
11:40:35 sean-k-mooney yes
11:41:35 belmoreira in that case I may suggest that the default of the config is "disabled" just in case the operator is not aware
11:45:51 sean-k-mooney you may. we were hoping to have the feature enable by defualt so that it just worked for new deployment but i guess we could handel that in installers
11:46:25 sean-k-mooney it would simplify things
11:46:34 sean-k-mooney if it was disabel by default
11:46:46 sean-k-mooney as you would just upgrade normally
11:47:01 sean-k-mooney the set teh config option if you wanted the feature.
11:47:34 sean-k-mooney i guess we could also cahge the default in U and remove the option in V
11:47:59 sean-k-mooney so it give people 3 cycle to switch instead of 1 which is what we were planning
11:48:09 sean-k-mooney stephenfin: ^ thoughts?
11:48:56 stephenfin Let me finish that email and then I'll think about this
11:49:41 sean-k-mooney tl;dr global config pcpus_in_placement=False by default in train change to true in U and remove in V. which completely disables the feature
11:50:00 sean-k-mooney once you set it to true you do it globally and cant set it back to false
11:50:47 sean-k-mooney in the V release we drop all support fo deploying without it enabled. in U its on by default and train its off by default
11:51:19 sean-k-mooney then we modify installer to enable it by defualt for new installs in train
11:58:13 mnaser if i am not wrong, it seems like we're installing `psycopg2` in test-requirements.txt for no reason?
11:58:31 mnaser i cant find any psycopg2 references except in test-requirements?
11:59:10 sean-k-mooney it used if you are using postgres via sqlalcambe for you db i think
11:59:34 stephenfin Do we still support that? There was talk of dropping it some time back, though cfriesen had objections
11:59:35 mnaser right but you'd have to install that yourself (i.e. it shouldn't be inside test-requirements.txt in openstack/nova ?)
11:59:52 mnaser oh i found this https://review.opendev.org/#/c/296604/
12:00:31 stephenfin What's the opportunistic testing he refers to?
12:00:51 mnaser this is 3 years ago, but i think there's a tc patch saying we dont support postgres.
12:01:03 sean-k-mooney we have a db mixing that will use mysql or postgress instead of sqlitght if it avioable i think
12:01:33 cdent mnaser: that patch never landed in full
12:01:34 sean-k-mooney mnaser: there was discussion or it yes
12:01:42 mnaser ah, ok.
12:01:48 cdent instead we decided that we wouldn't explicitly disallow it
12:02:02 mnaser i'm just annoyed by it because i just had tox fail on me cause i didnt have postgres development headers..
12:02:03 cdent instead projects could try to use sqlachemy in the way it was intended if they like
12:02:15 sean-k-mooney mnaser: are you just annoyed by the fact that requires use to install the postgres client and complie it
12:02:21 sean-k-mooney and that breaks on stable
12:02:28 sean-k-mooney ya
12:02:38 mnaser i have to get the postgres client to run tox locally (its inside test-requirements, not requirements)
12:02:39 sean-k-mooney we could replacy it with psycopg2-binary

Earlier   Later