Posted	Nick	Remark
#openstack-nova - 2017-11-29
23:24:43	mgagne	it's just that it looks like I won't be able to skip releases like I did before due to online data migration
23:24:44	melwitt	mriedem: yeah, okay. that's along the lines of what I was thinking. because basically, without the os-brick fix, the nova fix doesn't fully work. we'd have to handle the volume detach exit code in the libvirt driver to fix it on stable if there's no way to get it from a stable branch os-brick
23:25:29	mriedem	melwitt: i'd just backport the brick stuff, release a .z on stable, bump upper-constraints and then if anyone needs the fix for libvirt in nova on stable, they have to pick up the updated os-brick
23:25:35	mriedem	we just won't force it via g-r
23:26:09	mriedem	mgagne: i don't think you've been able to skip the online data migrations since kilo
23:26:18	mriedem	or shouldn't have
23:26:29	mgagne	well, I upgraded from kilo to mitaka just fine =)
23:26:31	mriedem	we've put in blocker schema migrations if you haven't completed the online data migrations
23:26:41	mriedem	sure, maybe you just didn't need to migrate any of that stuff
23:27:20	mgagne	well, there is a glitch with cellsv1 where service rpc version is checked in database and since you aren't running conductor or scheduler in API, migration fails. I just commented that part.
23:27:48	mriedem	i might have been thinking of this https://github.com/openstack/nova/blob/stable/newton/nova/db/sqlalchemy/migrate_repo/versions/330_enforce_mitaka_online_migrations.py
23:27:58	mriedem	newton schema migration requires that you have done the mitaka online data migrations
23:28:11	mgagne	yes, I'm not there yet :P
23:29:08	mriedem	"there is a glitch with cellsv1 where service rpc version is checked in database and since you aren't running conductor or scheduler in API, migration fails. I just commented that part."
23:29:11	mriedem	where does that happen?
23:29:29	melwitt	mriedem: just to clarify, if someone installs nova ocata, they will be prevented from installing the latest os-brick because of upper-constraints? or no? trying to understand the difference between them updating to a os-brick .z release vs just pulling the latest os-brick
23:29:46	mriedem	that does remind me of something i think i've talked with dansmith about before which i don't think we're handling yet, maybe we are, but when the api checks the minimum compute service version, i don't know if it's iterating all cells
23:30:09	mriedem	melwitt: totally depends on how nova is installed
23:30:20	mgagne	will need to check
23:30:23	mriedem	if you're pip installing into a tox venv from the nova git repo, then sure :)
23:30:28	mriedem	but i doubt most people are doing that
23:30:40	melwitt	mriedem: :\ okay. I was thinking whether going through all the .z release stuff helps much or not
23:30:49	mriedem	fedora rpm specs null out the requirements.txt so that the rpms are just installed from the versions in the rpm spec fedora says is good
23:31:19	dansmith	mriedem: it is iterating
23:31:41	melwitt	I see
23:31:44	mriedem	get_minimum_version_all_cells is
23:32:29	mriedem	dansmith: looks like we only use that in one place, server create for device tagging,
23:32:45	mriedem	but,
23:32:56	dansmith	hmm, actually
23:33:00	mriedem	anything else acting on an instance would be scoped to the cell that instance is in, so min service version checks scoped to that cell would be fine
23:33:01	dansmith	now that you mention it,
23:33:18	mriedem	i'm thinking about your conductor migration allocation min service versoin check
23:34:00	mriedem	when we do that, the context should be targeted to the cell that the instance was pulled from
23:34:14	dansmith	the problem is we cache it in places
23:34:19	mriedem	https://github.com/openstack/nova/blob/master/nova/conductor/tasks/migrate.py#L148
23:35:01	mriedem	oh hmm
23:35:06	dansmith	https://github.com/openstack/nova/blob/master/nova/objects/service.py#L396
23:35:14	mriedem	so min compute in cell1 might be 22 and 23 in cell2
23:35:21	mriedem	and if we cached 23, we could f up cell1
23:35:25	mriedem	?
23:35:31	dansmith	I was going to say we iterate it where we need it and otherwise it's scoped per instance, but....
23:36:02	mriedem	yeah we cache in the api, compute, conductor and scheduler services
23:36:16	mriedem	compute is probably fine right? it's scoped to a cell anyway
23:36:25	mriedem	but api/scheduler/conductor could be a problem
23:36:46	dansmith	right "inside the cell things" are fine
23:37:59	dansmith	man I wish we had the CellMapping in the context so we could make the cache just use that to key
23:38:02	mgagne	so I can't find the exact cause but I think it was looking for service entries in the API cell database and didn't exclude deleted rows.
23:38:39	dansmith	mgagne: I have no doubt that none of this works the way it should for cellsv1
23:39:17	dansmith	mixed-version computes I mean
23:39:17	mgagne	or could be with the online migration
23:39:36	mriedem	services entries are in the cell dbs
23:39:38	mriedem	not the api db
23:40:11	mgagne	here https://github.com/openstack/nova/blob/mitaka-eol/nova/objects/pci_device.py#L118-L127
23:40:24	mgagne	if no conductor service is found, the value returned fails the check
23:40:53	mgagne	well, maybe I shouldn't have run the online migration in the API cell? ¯\_(ツ)_/¯
23:41:44	mriedem	hm
23:42:37	mriedem	dansmith: we create the api/scheduler/conductor services in the cell0 db don't we?
23:42:47	dansmith	yup
23:42:50	dansmith	I mean, in devstack
23:42:54	mriedem	right
23:43:26	mriedem	yeah http://logs.openstack.org/87/523187/2/check/legacy-tempest-dsvm-neutron-full/6b78222/logs/etc/nova/nova.conf.txt.gz
23:43:29	mriedem	[database] connection = mysql+pymysql://root:secretmysql@127.0.0.1/nova_cell0?charset=utf8
23:43:40	dansmith	dude, passwords!
23:46:09	mriedem	mgagne: i wonder if you had this fix before you upgraded https://review.openstack.org/#/q/Ic96a5eb3728f97a3c35d2c5121e6fdcd4fd1c70b
23:46:16	mriedem	https://review.openstack.org/#/c/438632/
23:46:48	mgagne	yes
23:47:07	mgagne	but if no entry is found, I think it returns 0 ou None and it fails
23:47:59	mriedem	yeah you're right https://github.com/openstack/nova/blob/master/nova/objects/service.py#L431-L434
23:48:03	mgagne	but I don't know if I had to run the migration in api cell or not
23:48:54	mgagne	if i shouldn't, well I think the migration script should have told me: hey, this is an api cell, you shouldn't do that. But I understand that cellsv1 isn't fully tested so yea, what can you do =)
23:49:26	mriedem	https://github.com/openstack/nova/commit/50355c4595e08f293f610da32247e405b20c1c5b
23:49:44	mriedem	yeah my guess is there was no consideration for cells v1 when that was written in mitaka
23:49:51	mriedem	and we don't have grenade (upgrade) ci jobs for cellsv1
23:50:18	mriedem	cellsv1 upgrade testing was usually literally alaski or johnthetubaguy saying something broke at rax
23:50:31	mgagne	ok, that's fine, mitaka migration is behind us. but I guess same migration will fail again with newton if I try to run it in api cell.
23:50:42	mriedem	that code was dropped in newton
23:50:50	mriedem	because we have that schema migration blocker in newton
23:50:56	mriedem	https://github.com/openstack/nova/blob/stable/newton/nova/db/sqlalchemy/migrate_repo/versions/330_enforce_mitaka_online_migrations.py
23:51:11	mriedem	but ^ assumes the child cell db
23:51:33	mriedem	so nova-manage db sync
23:51:38	mriedem	not nova-manage api_db sync
23:51:43	mgagne	that's a fun one: "until all records have been migrated". Ok, let's run that migration then! oh way, code is gone. what now? /sad panda
23:51:47	mriedem	so you should be fine
23:52:17	mgagne	but then, you will say there is an upgrade readiness check now you can run
23:52:27	mgagne	which I'm fine with. ;)
23:52:30	mriedem	nova-status was added in ocata
23:53:20	mriedem	and checked for things like making sure placement was deployed and cellsv2 mappings existed
23:53:22	mgagne	yea, I had a lot of "fun" when it complained about the online migration and old code was gone. had to find a copy in a different environment and rsync that thing.
23:54:16	mriedem	i thought people stood up a separate env to run the db sync on the new code before upgrading the old code that's actually running?
23:54:52	mgagne	I guess I'm not in that ideal world yet :P
23:55:33	mriedem	you can also run the online data migrations from the old mitaka code before upgrading to newton, and run them after upgrading to newton if yo uwant
23:55:36	mgagne	usually I stop all services, upgrade package, run db sync, start service.
23:56:19	mgagne	but if I forgot to run online migration and db sync fails, I'm screwed because I already upgraded the packages. but my bad for not checking if all online migration ran properly before the upgrade.
23:56:54	mgagne	sure but I wasn't prepared for that maneuver
23:58:02	mriedem	mgagne: even with that blocker migration script in newton, we didn't delete the cold that allows you to run the online data migrations from mitaka https://github.com/openstack/nova/blob/stable/newton/nova/cmd/manage.py#L787
23:58:08	mriedem	so, we didn't hose you there
23:58:18	mriedem	we just said, you can't continue until you do your homework from mitaka
23:58:21	mgagne	very much appreciated =)
23:59:31	mriedem	pretty sure that's standard operating procedure
23:59:41	mriedem	in queens we still have online data migration code from newton

Earlier   Later