Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-11-29
23:02:28 mgagne I was a bit worried because I keep reading on the list that DNS split view is the (only) way to go. and everything else is "garbage" or a state of the art solution.
23:04:54 mriedem dansmith: melwitt: tssurya_: fyi https://etherpad.openstack.org/p/cellsv1-to-v2-migration
23:05:25 mgagne how don't know how to work that question: When should you move to cellsv2? Maybe based on multiple use cases: you current have no cells. you have only one cell, you have multiple cells. You need X features which is only available in release Y. etc.
23:05:42 mgagne and eventually: how
23:09:10 mriedem dansmith did a talk about that question in sydney
23:09:13 mriedem i'll link in the video
23:09:34 mriedem takashin: hey, i'm trying to fix https://review.openstack.org/#/c/408964/
23:09:46 mriedem takashin: i think the multi-cell functional test is intermittently failing
23:10:59 mriedem mgagne: https://www.openstack.org/videos/sydney-2017/adding-cellsv2-to-your-existing-nova-deployment
23:11:29 melwitt mriedem: if we have a nova change that depends on an os-brick fix, and we get a new os-brick release and it's in our requirements.txt, how would that work for a backport?
23:11:44 melwitt are we able to bump requirements.txt os-brick version on a stable branch or?
23:11:48 mriedem melwitt: this is your crazy newton hard reboot thing isn't it
23:11:53 mriedem melwitt: no
23:11:55 mriedem not on stable
23:12:19 mriedem melwitt: you can do whatever you want downstream of course
23:12:25 melwitt yes ... it's the compute host reboot thing
23:12:30 mriedem melwitt: but upstream it would likely mean somehow patching in the os-brick changes to nova stable
23:12:43 melwitt yeah, I know. just trying to think how we could fix this up for upstream ppl too
23:14:16 mgagne ok, now I have some talks to watch
23:14:41 melwitt I had been thinking, what if I backport the os-brick changes to their stable branches but I wasn't sure if that can map to our stable branches
23:20:44 mgagne mriedem: ok, so lets say you are running cellsv1 mitaka with multiple cells, your objective is to get to Pike with cellsv2 and with your cells intact. what's the plan? is that possible?
23:21:18 mgagne because that's my biggest concern and nowhere I can read about how to make it happen.
23:22:37 mriedem melwitt: you could backport the os-brick changes to stable, and release a patch .z release of os-brick on stable, you just can't bump the minimum required version of os-brick in g-r on stalbe
23:22:46 mriedem upper-constraints can be updated on stable though if needed
23:23:28 mriedem mgagne: i think that's what belmiro is working on as well, moving from mitaka with multiple child cells to v2
23:23:43 mriedem i'm not sure if his goal is pike, or just ocata
23:24:19 mgagne well, at this point, as long as I can get to Pike or a place where cells aren't a concern anymore, I'm fine.
23:24:43 mgagne it's just that it looks like I won't be able to skip releases like I did before due to online data migration
23:24:44 melwitt mriedem: yeah, okay. that's along the lines of what I was thinking. because basically, without the os-brick fix, the nova fix doesn't fully work. we'd have to handle the volume detach exit code in the libvirt driver to fix it on stable if there's no way to get it from a stable branch os-brick
23:25:29 mriedem melwitt: i'd just backport the brick stuff, release a .z on stable, bump upper-constraints and then if anyone needs the fix for libvirt in nova on stable, they have to pick up the updated os-brick
23:25:35 mriedem we just won't force it via g-r
23:26:09 mriedem mgagne: i don't think you've been able to skip the online data migrations since kilo
23:26:18 mriedem or shouldn't have
23:26:29 mgagne well, I upgraded from kilo to mitaka just fine =)
23:26:31 mriedem we've put in blocker schema migrations if you haven't completed the online data migrations
23:26:41 mriedem sure, maybe you just didn't need to migrate any of that stuff
23:27:20 mgagne well, there is a glitch with cellsv1 where service rpc version is checked in database and since you aren't running conductor or scheduler in API, migration fails. I just commented that part.
23:27:48 mriedem i might have been thinking of this https://github.com/openstack/nova/blob/stable/newton/nova/db/sqlalchemy/migrate_repo/versions/330_enforce_mitaka_online_migrations.py
23:27:58 mriedem newton schema migration requires that you have done the mitaka online data migrations
23:28:11 mgagne yes, I'm not there yet :P
23:29:08 mriedem "there is a glitch with cellsv1 where service rpc version is checked in database and since you aren't running conductor or scheduler in API, migration fails. I just commented that part."
23:29:11 mriedem where does that happen?
23:29:29 melwitt mriedem: just to clarify, if someone installs nova ocata, they will be prevented from installing the latest os-brick because of upper-constraints? or no? trying to understand the difference between them updating to a os-brick .z release vs just pulling the latest os-brick
23:29:46 mriedem that does remind me of something i think i've talked with dansmith about before which i don't think we're handling yet, maybe we are, but when the api checks the minimum compute service version, i don't know if it's iterating all cells
23:30:09 mriedem melwitt: totally depends on how nova is installed
23:30:20 mgagne will need to check
23:30:23 mriedem if you're pip installing into a tox venv from the nova git repo, then sure :)
23:30:28 mriedem but i doubt most people are doing that
23:30:40 melwitt mriedem: :\ okay. I was thinking whether going through all the .z release stuff helps much or not
23:30:49 mriedem fedora rpm specs null out the requirements.txt so that the rpms are just installed from the versions in the rpm spec fedora says is good
23:31:19 dansmith mriedem: it is iterating
23:31:41 melwitt I see
23:31:44 mriedem get_minimum_version_all_cells is
23:32:29 mriedem dansmith: looks like we only use that in one place, server create for device tagging,
23:32:45 mriedem but,
23:32:56 dansmith hmm, actually
23:33:00 mriedem anything else acting on an instance would be scoped to the cell that instance is in, so min service version checks scoped to that cell would be fine
23:33:01 dansmith now that you mention it,
23:33:18 mriedem i'm thinking about your conductor migration allocation min service versoin check
23:34:00 mriedem when we do that, the context should be targeted to the cell that the instance was pulled from
23:34:14 dansmith the problem is we cache it in places
23:34:19 mriedem https://github.com/openstack/nova/blob/master/nova/conductor/tasks/migrate.py#L148
23:35:01 mriedem oh hmm
23:35:06 dansmith https://github.com/openstack/nova/blob/master/nova/objects/service.py#L396
23:35:14 mriedem so min compute in cell1 might be 22 and 23 in cell2
23:35:21 mriedem and if we cached 23, we could f up cell1
23:35:25 mriedem ?
23:35:31 dansmith I was going to say we iterate it where we need it and otherwise it's scoped per instance, but....
23:36:02 mriedem yeah we cache in the api, compute, conductor and scheduler services
23:36:16 mriedem compute is probably fine right? it's scoped to a cell anyway
23:36:25 mriedem but api/scheduler/conductor could be a problem
23:36:46 dansmith right "inside the cell things" are fine
23:37:59 dansmith man I wish we had the CellMapping in the context so we could make the cache just use that to key
23:38:02 mgagne so I can't find the exact cause but I think it was looking for service entries in the API cell database and didn't exclude deleted rows.
23:38:39 dansmith mgagne: I have no doubt that none of this works the way it should for cellsv1
23:39:17 mgagne or could be with the online migration
23:39:17 dansmith mixed-version computes I mean
23:39:36 mriedem services entries are in the cell dbs
23:39:38 mriedem not the api db
23:40:11 mgagne here https://github.com/openstack/nova/blob/mitaka-eol/nova/objects/pci_device.py#L118-L127
23:40:24 mgagne if no conductor service is found, the value returned fails the check
23:40:53 mgagne well, maybe I shouldn't have run the online migration in the API cell? ¯\_(ツ)_/¯
23:41:44 mriedem hm
23:42:37 mriedem dansmith: we create the api/scheduler/conductor services in the cell0 db don't we?
23:42:47 dansmith yup
23:42:50 dansmith I mean, in devstack
23:42:54 mriedem right
23:43:26 mriedem yeah http://logs.openstack.org/87/523187/2/check/legacy-tempest-dsvm-neutron-full/6b78222/logs/etc/nova/nova.conf.txt.gz
23:43:29 mriedem [database] connection = mysql+pymysql://root:secretmysql@127.0.0.1/nova_cell0?charset=utf8
23:43:40 dansmith dude, passwords!
23:46:09 mriedem mgagne: i wonder if you had this fix before you upgraded https://review.openstack.org/#/q/Ic96a5eb3728f97a3c35d2c5121e6fdcd4fd1c70b
23:46:16 mriedem https://review.openstack.org/#/c/438632/
23:46:48 mgagne yes
23:47:07 mgagne but if no entry is found, I think it returns 0 ou None and it fails
23:47:59 mriedem yeah you're right https://github.com/openstack/nova/blob/master/nova/objects/service.py#L431-L434
23:48:03 mgagne but I don't know if I had to run the migration in api cell or not
23:48:54 mgagne if i shouldn't, well I think the migration script should have told me: hey, this is an api cell, you shouldn't do that. But I understand that cellsv1 isn't fully tested so yea, what can you do =)
23:49:26 mriedem https://github.com/openstack/nova/commit/50355c4595e08f293f610da32247e405b20c1c5b
23:49:44 mriedem yeah my guess is there was no consideration for cells v1 when that was written in mitaka
23:49:51 mriedem and we don't have grenade (upgrade) ci jobs for cellsv1

Earlier   Later