Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-02-06
18:27:46 mriedem not dumb at all
18:28:07 mriedem if you have a local recreate, you could check the nova_api.instance_mappings table directly
18:28:17 mriedem or the cell db instances table
18:28:20 odyssey4me sure, can do that
18:28:37 mriedem looks like you have 2 cells, so you'd have to look in each
18:29:13 mriedem so you've got 4 dbs (nova_api, nova_cell0, cell1 and then cell 9461149a-52a9-495d-8021-d2cda1645d28)
18:29:21 odyssey4me right, so I have several DB's here: nova, nova_api, nova_cell0; nova_placement
18:29:35 mriedem nova is likely cell1
18:29:36 mriedem yes?
18:29:45 mriedem nova_placement isn't a thing...not sure what that is
18:31:25 odyssey4me ok, that's empty - I'll look into why that's there later
18:31:35 mriedem i probably know why
18:32:09 odyssey4me it might be some leftovers from previous work before things matured
18:33:24 mriedem odyssey4me: yeah https://review.openstack.org/#/q/I31293ac4689630e4113588ab2c6373cf572b8f38
18:34:08 odyssey4me haha, ok - thanks for the reference :)
18:34:18 mriedem odyssey4me: so looking at https://pastebin.com/mSwpZybQ there is something screwed up with the host mappings,
18:34:45 mriedem it looks like this compute is in two cells
18:34:46 mriedem Checking host mapping for compute host 'ocata-nova1': 6e6d71ab-0b18-416d-8a3a-ce713ac59637
18:35:02 odyssey4me if it'd make things simpler I can get your pub key on this host for you to poke around directly?
18:35:36 odyssey4me the host is a temp instance, so nothing special on it
18:35:36 mriedem i don't think our relationship has hit that level of maturity yet
18:35:44 odyssey4me hahaha, fair enough
18:36:37 odyssey4me otherwise, I'll need some guidance with db queries to get data out - I can gist the results as we go
18:36:54 mriedem well, it appears you have 2 nova_api.host_mappings entries for host "ocata-nova1"
18:36:56 mriedem which would be wrong
18:37:09 odyssey4me yeah, that seemed weird to me too
18:37:36 cfriesen is anyone aware of an issue where running "'wget http://169.254.169.254/latest/meta-data/instance-id" in the guest gives a result that is *not* the same as that instance's OS-EXT-SRV-ATTR:instance_name in "nova show"?
18:37:53 mriedem tssurya: dansmith: melwitt: any idea why we don't have a unique constraint across the cell_id and host colums in the host_mappings table?
18:38:33 dansmith mriedem: I thought there was some argument about that with overlapping hostnames (which won't work anyway) ?
18:38:36 mriedem odyssey4me: is this CI supposed to have 2 cells? because multi-cell wasn't supported in ocata
18:38:40 odyssey4me mriedem interestingly enough, I only see one entry in the DB for it
18:39:22 odyssey4me nah, it only has cell0 and cell1
18:39:23 mriedem odyssey4me: do you have a CI run with logs posted?
18:39:32 mriedem https://pastebin.com/mSwpZybQ is saying there are 3 cell mappings
18:39:36 mriedem "Found 3 cell mappings."
18:39:48 mriedem select uuid from nova_api.cell_mappings;
18:40:10 odyssey4me mriedem unfortunately our log collection is broken for ocata, so all we have is console output which only shows brokenness when tempest runs, so that's not very useful
18:40:34 odyssey4me I can fix up the CI to collect logs, but that'll take a few days to make its way through...
18:42:05 odyssey4me hmm, that is odd - three cells showing
18:43:04 odyssey4me it might be my bad, I re-ran 'nova-manage cell_v2 discover_hosts' and 'nova-manage cell_v2 map_instances --cell_uuid ...' a few times
18:43:44 odyssey4me also ran 'nova-manage cell_v2 simple_cell_setup' after the initial build
18:44:03 odyssey4me that gave me 'Cell0 is already setup', so I don't think that broke anything
18:45:40 mriedem discover_hosts is idempotent and doesn't create mappings
18:45:43 mriedem cell mappings i mean
18:45:48 mriedem map_instances should also be ok
18:45:57 mriedem my guess is something go f'ed up when running simple_cell_setup a few times
18:46:23 odyssey4me I only did it once :)
18:48:04 mriedem was create_cell ever called?
18:48:35 odyssey4me yep - this is the basic set of steps that would have executed before the discovery: https://github.com/openstack/openstack-ansible-os_nova/blob/stable/ocata/tasks/nova_db_setup.yml#L16-L55
18:49:03 odyssey4me so in this case, the initial api_db sync would have been skipped as this was an existing DB
18:49:23 odyssey4me then the cell0 map done: https://github.com/openstack/openstack-ansible-os_nova/blob/stable/ocata/tasks/nova_db_setup.yml#L31
18:49:37 odyssey4me then the creation of cell1: https://github.com/openstack/openstack-ansible-os_nova/blob/stable/ocata/tasks/nova_db_setup.yml#L37
18:49:54 odyssey4me then the api_db sync, and db sync
18:50:22 mriedem where does map_instances happen?
18:50:36 mriedem https://github.com/openstack/openstack-ansible-os_nova/blob/stable/ocata/tasks/nova_db_post_setup.yml#L38
18:50:38 mriedem found it
18:51:04 odyssey4me we wait for a compute instance to be ready in the handler: https://github.com/openstack/openstack-ansible-os_nova/blob/stable/ocata/handlers/main.yml#L67-L76
18:51:14 odyssey4me then yes, https://github.com/openstack/openstack-ansible-os_nova/blob/stable/ocata/tasks/nova_db_post_setup.yml gets run
18:53:15 odyssey4me you'll find those tasks, and their stdout/stderr in this console log: http://logs.openstack.org/29/540329/5/check/openstack-ansible-upgrade-ubuntu-xenial/30ae3e2/job-output.txt.gz
18:53:36 odyssey4me it's a bit verbose, apologies in advance
18:56:38 kashyap dansmith: When you get a moment, wrote this here: https://review.openstack.org/#/c/497457/18
18:56:51 kashyap Please correct / critique / answer as you see fit.
18:57:34 mriedem odyssey4me: hmm, for https://github.com/openstack/openstack-ansible-os_nova/blob/stable/ocata/tasks/nova_db_post_setup.yml#L39
18:57:39 mriedem --cell_uuid {{ cell1_uuid['stdout'].split()[3] }}"
18:57:44 mriedem are you sure that's hitting cell1 and not cell0?
18:58:07 mriedem oh i guess because of https://github.com/openstack/openstack-ansible-os_nova/blob/stable/ocata/tasks/nova_db_post_setup.yml#L32
18:58:13 mriedem so nvm
18:58:14 odyssey4me yep, exactly
18:58:33 odyssey4me if someone names their cell "cell12" then it might not be right - but we're using safe defaults here
18:58:50 kashyap dansmith: To finish up, for completeness' sake: The QEMU / libvirt 3.9.0 "pause-before-switchover" thing -- it won't be useful in Nova's case, as that will stop the guest CPUs, which will extend the guest down time.)
18:59:02 odyssey4me although the spacing in that grep should be reasonable to protect the boundaries
19:03:14 mriedem odyssey4me: just to verify, this is running on the newton code before upgrading to ocata yes?
19:03:30 mriedem because http://logs.openstack.org/29/540329/5/check/openstack-ansible-upgrade-ubuntu-xenial/30ae3e2/job-output.txt.gz#_2018-02-05_18_10_32_079153 returns version 21 for the nova_api db which was the version in newton
19:04:06 odyssey4me yep, the initial build from be from what was the head of stable/newton at the time the test ran
19:04:17 odyssey4me I'll push a patch up to make it use the EOL branch now :)
19:04:29 odyssey4me oh hang on, I think it mighth be sha pinned
19:04:45 mriedem there was only one nova_api db change in newton before eol https://github.com/openstack/nova/blob/newton-eol/nova/db/sqlalchemy/api_migrations/migrate_repo/versions/022_request_specs_spec_mediumtext.py
19:04:53 mriedem and shouldn't have anything to do with what you're seeing
19:05:04 odyssey4me heh, that makes it even weirder that this failed: https://github.com/openstack/openstack-ansible-tests/blob/stable/newton/test-vars.yml#L157
19:05:35 dansmith kashyap: yeah that's why I said it didn't seem like it would :)
19:06:58 mriedem odyssey4me: so everything in that console output looks ok to me,
19:07:10 mriedem where do instances actually get created in newton before the upgrade is started?
19:07:14 dansmith kashyap: I was also really looking for you to comment on the potential race with the setting of the bandwidth limit in two places.. did you look at that at all?
19:07:29 mriedem odyssey4me: i.e. how do you know that you've actually got instances from newton to map when map_instances runs?
19:07:36 kashyap dansmith: No, not yet. After I finish dinner
19:08:33 odyssey4me mriedem by 'instances' do you mean the compute host? The terminology is confusing to me here... because in this CI run no instance (by this I mean a cloud instance, not a nova hypervisor) is created until *after* the upgrade when tempest executes.
19:09:41 mriedem instances == vms
19:09:52 mriedem odyssey4me: ok then map_instances isn't going to map anything :)
19:10:02 mriedem map_instances means create nova_api.instance_mappings records in the db
19:10:07 mriedem based on VMs in the 'nova' db
19:10:11 mriedem the 'nova' db is the cell1 db
19:11:08 mriedem odyssey4me: so looking at https://pastebin.com/C9ji6vdS again, when/how did either of those instances get created?
19:11:13 mriedem the ones passed to verify_instance
19:11:13 odyssey4me ok, btw if we're doing something dumb here then please feel free to say so with any improvement suggestions :)
19:11:23 mriedem odyssey4me: the ansible tasks all look fine
19:11:39 mriedem the ansible is like a step for step copy of what's in the install guide here https://docs.openstack.org/nova/latest/user/cells.html
19:11:59 odyssey4me ok, given that I did those commands after the test failed, those instances would have been created by tempest after the upgrade completed
19:12:18 odyssey4me the one thing that may not have run here is the online migrations
19:12:26 dansmith kashyap: okay thanks
19:12:42 mriedem odyssey4me: do you see any errors in the nova-api logs?

Earlier   Later