Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-02-06
18:23:04 odyssey4me well, I got stuck there - given that nova-manage doesn't appear to have a way to list the instances :/
18:23:15 odyssey4me any tips for getting a listing out?
18:23:35 odyssey4me the closest I could find is "nova-manage cell_v2 discover_hosts --verbose"
18:24:49 mriedem that doesn't dump the instances
18:25:18 mriedem the api isn't up?
18:25:27 odyssey4me it gives me a set of uuid's which look rather like they belong to instances: https://pastebin.com/mSwpZybQ
18:25:43 mriedem Getting compute nodes from cell 'cell1': 16443e92-e33b-487e-8069-7c80e5bdbc33
18:25:46 mriedem that's a cell mapping uuid
18:25:54 mriedem Checking host mapping for compute host 'ocata-nova1': 6e6d71ab-0b18-416d-8a3a-ce713ac59637
18:25:56 mriedem that's a compute node uuid
18:26:08 mriedem instances aren't the only things that have uuids anymore
18:26:40 odyssey4me the API is up, so I can query things that way if it helps
18:27:18 mriedem worth a shot if you have a local recreate,
18:27:27 mriedem but my guess is the api won't find them either if nova-manage can't
18:27:29 odyssey4me my apologies for dumb questions - it's been a very long time since I actually worked with nova directly :)
18:27:37 mriedem if they don't have instance_mappings in nova_api i mean
18:27:42 mriedem totally fine
18:27:46 mriedem not dumb at all
18:28:07 mriedem if you have a local recreate, you could check the nova_api.instance_mappings table directly
18:28:17 mriedem or the cell db instances table
18:28:20 odyssey4me sure, can do that
18:28:37 mriedem looks like you have 2 cells, so you'd have to look in each
18:29:13 mriedem so you've got 4 dbs (nova_api, nova_cell0, cell1 and then cell 9461149a-52a9-495d-8021-d2cda1645d28)
18:29:21 odyssey4me right, so I have several DB's here: nova, nova_api, nova_cell0; nova_placement
18:29:35 mriedem nova is likely cell1
18:29:36 mriedem yes?
18:29:45 mriedem nova_placement isn't a thing...not sure what that is
18:31:25 odyssey4me ok, that's empty - I'll look into why that's there later
18:31:35 mriedem i probably know why
18:32:09 odyssey4me it might be some leftovers from previous work before things matured
18:33:24 mriedem odyssey4me: yeah https://review.openstack.org/#/q/I31293ac4689630e4113588ab2c6373cf572b8f38
18:34:08 odyssey4me haha, ok - thanks for the reference :)
18:34:18 mriedem odyssey4me: so looking at https://pastebin.com/mSwpZybQ there is something screwed up with the host mappings,
18:34:45 mriedem it looks like this compute is in two cells
18:34:46 mriedem Checking host mapping for compute host 'ocata-nova1': 6e6d71ab-0b18-416d-8a3a-ce713ac59637
18:35:02 odyssey4me if it'd make things simpler I can get your pub key on this host for you to poke around directly?
18:35:36 odyssey4me the host is a temp instance, so nothing special on it
18:35:36 mriedem i don't think our relationship has hit that level of maturity yet
18:35:44 odyssey4me hahaha, fair enough
18:36:37 odyssey4me otherwise, I'll need some guidance with db queries to get data out - I can gist the results as we go
18:36:54 mriedem well, it appears you have 2 nova_api.host_mappings entries for host "ocata-nova1"
18:36:56 mriedem which would be wrong
18:37:09 odyssey4me yeah, that seemed weird to me too
18:37:36 cfriesen is anyone aware of an issue where running "'wget http://169.254.169.254/latest/meta-data/instance-id" in the guest gives a result that is *not* the same as that instance's OS-EXT-SRV-ATTR:instance_name in "nova show"?
18:37:53 mriedem tssurya: dansmith: melwitt: any idea why we don't have a unique constraint across the cell_id and host colums in the host_mappings table?
18:38:33 dansmith mriedem: I thought there was some argument about that with overlapping hostnames (which won't work anyway) ?
18:38:36 mriedem odyssey4me: is this CI supposed to have 2 cells? because multi-cell wasn't supported in ocata
18:38:40 odyssey4me mriedem interestingly enough, I only see one entry in the DB for it
18:39:22 odyssey4me nah, it only has cell0 and cell1
18:39:23 mriedem odyssey4me: do you have a CI run with logs posted?
18:39:32 mriedem https://pastebin.com/mSwpZybQ is saying there are 3 cell mappings
18:39:36 mriedem "Found 3 cell mappings."
18:39:48 mriedem select uuid from nova_api.cell_mappings;
18:40:10 odyssey4me mriedem unfortunately our log collection is broken for ocata, so all we have is console output which only shows brokenness when tempest runs, so that's not very useful
18:40:34 odyssey4me I can fix up the CI to collect logs, but that'll take a few days to make its way through...
18:42:05 odyssey4me hmm, that is odd - three cells showing
18:43:04 odyssey4me it might be my bad, I re-ran 'nova-manage cell_v2 discover_hosts' and 'nova-manage cell_v2 map_instances --cell_uuid ...' a few times
18:43:44 odyssey4me also ran 'nova-manage cell_v2 simple_cell_setup' after the initial build
18:44:03 odyssey4me that gave me 'Cell0 is already setup', so I don't think that broke anything
18:45:40 mriedem discover_hosts is idempotent and doesn't create mappings
18:45:43 mriedem cell mappings i mean
18:45:48 mriedem map_instances should also be ok
18:45:57 mriedem my guess is something go f'ed up when running simple_cell_setup a few times
18:46:23 odyssey4me I only did it once :)
18:48:04 mriedem was create_cell ever called?
18:48:35 odyssey4me yep - this is the basic set of steps that would have executed before the discovery: https://github.com/openstack/openstack-ansible-os_nova/blob/stable/ocata/tasks/nova_db_setup.yml#L16-L55
18:49:03 odyssey4me so in this case, the initial api_db sync would have been skipped as this was an existing DB
18:49:23 odyssey4me then the cell0 map done: https://github.com/openstack/openstack-ansible-os_nova/blob/stable/ocata/tasks/nova_db_setup.yml#L31
18:49:37 odyssey4me then the creation of cell1: https://github.com/openstack/openstack-ansible-os_nova/blob/stable/ocata/tasks/nova_db_setup.yml#L37
18:49:54 odyssey4me then the api_db sync, and db sync
18:50:22 mriedem where does map_instances happen?
18:50:36 mriedem https://github.com/openstack/openstack-ansible-os_nova/blob/stable/ocata/tasks/nova_db_post_setup.yml#L38
18:50:38 mriedem found it
18:51:04 odyssey4me we wait for a compute instance to be ready in the handler: https://github.com/openstack/openstack-ansible-os_nova/blob/stable/ocata/handlers/main.yml#L67-L76
18:51:14 odyssey4me then yes, https://github.com/openstack/openstack-ansible-os_nova/blob/stable/ocata/tasks/nova_db_post_setup.yml gets run
18:53:15 odyssey4me you'll find those tasks, and their stdout/stderr in this console log: http://logs.openstack.org/29/540329/5/check/openstack-ansible-upgrade-ubuntu-xenial/30ae3e2/job-output.txt.gz
18:53:36 odyssey4me it's a bit verbose, apologies in advance
18:56:38 kashyap dansmith: When you get a moment, wrote this here: https://review.openstack.org/#/c/497457/18
18:56:51 kashyap Please correct / critique / answer as you see fit.
18:57:34 mriedem odyssey4me: hmm, for https://github.com/openstack/openstack-ansible-os_nova/blob/stable/ocata/tasks/nova_db_post_setup.yml#L39
18:57:39 mriedem --cell_uuid {{ cell1_uuid['stdout'].split()[3] }}"
18:57:44 mriedem are you sure that's hitting cell1 and not cell0?
18:58:07 mriedem oh i guess because of https://github.com/openstack/openstack-ansible-os_nova/blob/stable/ocata/tasks/nova_db_post_setup.yml#L32
18:58:13 mriedem so nvm
18:58:14 odyssey4me yep, exactly
18:58:33 odyssey4me if someone names their cell "cell12" then it might not be right - but we're using safe defaults here
18:58:50 kashyap dansmith: To finish up, for completeness' sake: The QEMU / libvirt 3.9.0 "pause-before-switchover" thing -- it won't be useful in Nova's case, as that will stop the guest CPUs, which will extend the guest down time.)
18:59:02 odyssey4me although the spacing in that grep should be reasonable to protect the boundaries
19:03:14 mriedem odyssey4me: just to verify, this is running on the newton code before upgrading to ocata yes?
19:03:30 mriedem because http://logs.openstack.org/29/540329/5/check/openstack-ansible-upgrade-ubuntu-xenial/30ae3e2/job-output.txt.gz#_2018-02-05_18_10_32_079153 returns version 21 for the nova_api db which was the version in newton
19:04:06 odyssey4me yep, the initial build from be from what was the head of stable/newton at the time the test ran
19:04:17 odyssey4me I'll push a patch up to make it use the EOL branch now :)
19:04:29 odyssey4me oh hang on, I think it mighth be sha pinned
19:04:45 mriedem there was only one nova_api db change in newton before eol https://github.com/openstack/nova/blob/newton-eol/nova/db/sqlalchemy/api_migrations/migrate_repo/versions/022_request_specs_spec_mediumtext.py
19:04:53 mriedem and shouldn't have anything to do with what you're seeing
19:05:04 odyssey4me heh, that makes it even weirder that this failed: https://github.com/openstack/openstack-ansible-tests/blob/stable/newton/test-vars.yml#L157
19:05:35 dansmith kashyap: yeah that's why I said it didn't seem like it would :)
19:06:58 mriedem odyssey4me: so everything in that console output looks ok to me,
19:07:10 mriedem where do instances actually get created in newton before the upgrade is started?
19:07:14 dansmith kashyap: I was also really looking for you to comment on the potential race with the setting of the bandwidth limit in two places.. did you look at that at all?

Earlier   Later