Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-02-06
18:17:04 odyssey4me Our last working test was the last patch we did which was in early December.
18:17:25 odyssey4me Any idea what may have changed to cause the previously working implementation to no longer work?
18:19:22 mriedem odyssey4me: i don't see anything related to that in https://github.com/openstack/nova/commits/stable/ocata going back to november
18:19:29 mriedem are you sure it wasn't something that changed on the OSA side?
18:19:47 odyssey4me mriedem yep, very sure - see https://review.openstack.org/#/q/branch:stable/ocata+project:openstack/openstack-ansible-os_nova
18:21:27 mriedem https://github.com/openstack/nova/commits/stable/ocata/nova/cmd/manage.py
18:21:37 mriedem oldest thing in stable there is from october
18:22:14 odyssey4me yeah, I'm wondering if something has perhaps changed in the database tables either in newton (the source), or in ocata (the destination)
18:22:40 mriedem have you verified if the instances you're verifying actually exist?
18:22:49 mriedem i.e. not just not mapped, but don't actually exist
18:23:04 odyssey4me well, I got stuck there - given that nova-manage doesn't appear to have a way to list the instances :/
18:23:15 odyssey4me any tips for getting a listing out?
18:23:35 odyssey4me the closest I could find is "nova-manage cell_v2 discover_hosts --verbose"
18:24:49 mriedem that doesn't dump the instances
18:25:18 mriedem the api isn't up?
18:25:27 odyssey4me it gives me a set of uuid's which look rather like they belong to instances: https://pastebin.com/mSwpZybQ
18:25:43 mriedem Getting compute nodes from cell 'cell1': 16443e92-e33b-487e-8069-7c80e5bdbc33
18:25:46 mriedem that's a cell mapping uuid
18:25:54 mriedem Checking host mapping for compute host 'ocata-nova1': 6e6d71ab-0b18-416d-8a3a-ce713ac59637
18:25:56 mriedem that's a compute node uuid
18:26:08 mriedem instances aren't the only things that have uuids anymore
18:26:40 odyssey4me the API is up, so I can query things that way if it helps
18:27:18 mriedem worth a shot if you have a local recreate,
18:27:27 mriedem but my guess is the api won't find them either if nova-manage can't
18:27:29 odyssey4me my apologies for dumb questions - it's been a very long time since I actually worked with nova directly :)
18:27:37 mriedem if they don't have instance_mappings in nova_api i mean
18:27:42 mriedem totally fine
18:27:46 mriedem not dumb at all
18:28:07 mriedem if you have a local recreate, you could check the nova_api.instance_mappings table directly
18:28:17 mriedem or the cell db instances table
18:28:20 odyssey4me sure, can do that
18:28:37 mriedem looks like you have 2 cells, so you'd have to look in each
18:29:13 mriedem so you've got 4 dbs (nova_api, nova_cell0, cell1 and then cell 9461149a-52a9-495d-8021-d2cda1645d28)
18:29:21 odyssey4me right, so I have several DB's here: nova, nova_api, nova_cell0; nova_placement
18:29:35 mriedem nova is likely cell1
18:29:36 mriedem yes?
18:29:45 mriedem nova_placement isn't a thing...not sure what that is
18:31:25 odyssey4me ok, that's empty - I'll look into why that's there later
18:31:35 mriedem i probably know why
18:32:09 odyssey4me it might be some leftovers from previous work before things matured
18:33:24 mriedem odyssey4me: yeah https://review.openstack.org/#/q/I31293ac4689630e4113588ab2c6373cf572b8f38
18:34:08 odyssey4me haha, ok - thanks for the reference :)
18:34:18 mriedem odyssey4me: so looking at https://pastebin.com/mSwpZybQ there is something screwed up with the host mappings,
18:34:45 mriedem it looks like this compute is in two cells
18:34:46 mriedem Checking host mapping for compute host 'ocata-nova1': 6e6d71ab-0b18-416d-8a3a-ce713ac59637
18:35:02 odyssey4me if it'd make things simpler I can get your pub key on this host for you to poke around directly?
18:35:36 mriedem i don't think our relationship has hit that level of maturity yet
18:35:36 odyssey4me the host is a temp instance, so nothing special on it
18:35:44 odyssey4me hahaha, fair enough
18:36:37 odyssey4me otherwise, I'll need some guidance with db queries to get data out - I can gist the results as we go
18:36:54 mriedem well, it appears you have 2 nova_api.host_mappings entries for host "ocata-nova1"
18:36:56 mriedem which would be wrong
18:37:09 odyssey4me yeah, that seemed weird to me too
18:37:36 cfriesen is anyone aware of an issue where running "'wget http://169.254.169.254/latest/meta-data/instance-id" in the guest gives a result that is *not* the same as that instance's OS-EXT-SRV-ATTR:instance_name in "nova show"?
18:37:53 mriedem tssurya: dansmith: melwitt: any idea why we don't have a unique constraint across the cell_id and host colums in the host_mappings table?
18:38:33 dansmith mriedem: I thought there was some argument about that with overlapping hostnames (which won't work anyway) ?
18:38:36 mriedem odyssey4me: is this CI supposed to have 2 cells? because multi-cell wasn't supported in ocata
18:38:40 odyssey4me mriedem interestingly enough, I only see one entry in the DB for it
18:39:22 odyssey4me nah, it only has cell0 and cell1
18:39:23 mriedem odyssey4me: do you have a CI run with logs posted?
18:39:32 mriedem https://pastebin.com/mSwpZybQ is saying there are 3 cell mappings
18:39:36 mriedem "Found 3 cell mappings."
18:39:48 mriedem select uuid from nova_api.cell_mappings;
18:40:10 odyssey4me mriedem unfortunately our log collection is broken for ocata, so all we have is console output which only shows brokenness when tempest runs, so that's not very useful
18:40:34 odyssey4me I can fix up the CI to collect logs, but that'll take a few days to make its way through...
18:42:05 odyssey4me hmm, that is odd - three cells showing
18:43:04 odyssey4me it might be my bad, I re-ran 'nova-manage cell_v2 discover_hosts' and 'nova-manage cell_v2 map_instances --cell_uuid ...' a few times
18:43:44 odyssey4me also ran 'nova-manage cell_v2 simple_cell_setup' after the initial build
18:44:03 odyssey4me that gave me 'Cell0 is already setup', so I don't think that broke anything
18:45:40 mriedem discover_hosts is idempotent and doesn't create mappings
18:45:43 mriedem cell mappings i mean
18:45:48 mriedem map_instances should also be ok
18:45:57 mriedem my guess is something go f'ed up when running simple_cell_setup a few times
18:46:23 odyssey4me I only did it once :)
18:48:04 mriedem was create_cell ever called?
18:48:35 odyssey4me yep - this is the basic set of steps that would have executed before the discovery: https://github.com/openstack/openstack-ansible-os_nova/blob/stable/ocata/tasks/nova_db_setup.yml#L16-L55
18:49:03 odyssey4me so in this case, the initial api_db sync would have been skipped as this was an existing DB
18:49:23 odyssey4me then the cell0 map done: https://github.com/openstack/openstack-ansible-os_nova/blob/stable/ocata/tasks/nova_db_setup.yml#L31
18:49:37 odyssey4me then the creation of cell1: https://github.com/openstack/openstack-ansible-os_nova/blob/stable/ocata/tasks/nova_db_setup.yml#L37
18:49:54 odyssey4me then the api_db sync, and db sync
18:50:22 mriedem where does map_instances happen?
18:50:36 mriedem https://github.com/openstack/openstack-ansible-os_nova/blob/stable/ocata/tasks/nova_db_post_setup.yml#L38
18:50:38 mriedem found it
18:51:04 odyssey4me we wait for a compute instance to be ready in the handler: https://github.com/openstack/openstack-ansible-os_nova/blob/stable/ocata/handlers/main.yml#L67-L76
18:51:14 odyssey4me then yes, https://github.com/openstack/openstack-ansible-os_nova/blob/stable/ocata/tasks/nova_db_post_setup.yml gets run
18:53:15 odyssey4me you'll find those tasks, and their stdout/stderr in this console log: http://logs.openstack.org/29/540329/5/check/openstack-ansible-upgrade-ubuntu-xenial/30ae3e2/job-output.txt.gz
18:53:36 odyssey4me it's a bit verbose, apologies in advance
18:56:38 kashyap dansmith: When you get a moment, wrote this here: https://review.openstack.org/#/c/497457/18
18:56:51 kashyap Please correct / critique / answer as you see fit.
18:57:34 mriedem odyssey4me: hmm, for https://github.com/openstack/openstack-ansible-os_nova/blob/stable/ocata/tasks/nova_db_post_setup.yml#L39
18:57:39 mriedem --cell_uuid {{ cell1_uuid['stdout'].split()[3] }}"
18:57:44 mriedem are you sure that's hitting cell1 and not cell0?
18:58:07 mriedem oh i guess because of https://github.com/openstack/openstack-ansible-os_nova/blob/stable/ocata/tasks/nova_db_post_setup.yml#L32
18:58:13 mriedem so nvm
18:58:14 odyssey4me yep, exactly
18:58:33 odyssey4me if someone names their cell "cell12" then it might not be right - but we're using safe defaults here
18:58:50 kashyap dansmith: To finish up, for completeness' sake: The QEMU / libvirt 3.9.0 "pause-before-switchover" thing -- it won't be useful in Nova's case, as that will stop the guest CPUs, which will extend the guest down time.)
18:59:02 odyssey4me although the spacing in that grep should be reasonable to protect the boundaries
19:03:14 mriedem odyssey4me: just to verify, this is running on the newton code before upgrading to ocata yes?
19:03:30 mriedem because http://logs.openstack.org/29/540329/5/check/openstack-ansible-upgrade-ubuntu-xenial/30ae3e2/job-output.txt.gz#_2018-02-05_18_10_32_079153 returns version 21 for the nova_api db which was the version in newton

Earlier   Later