Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-02-06
18:15:40 stvnoyes mriedem: good question. idk. I haven't played with CI stuff so all this is pretty new to me. I will see if there's someone around on the zfssa team who would know.
18:16:25 mriedem stvnoyes: https://wiki.openstack.org/wiki/ThirdPartySystems/Oracle_ZFSSA_CI
18:16:36 odyssey4me howdy all - we're having some trouble in the OSA deployment test for upgrades from newton (without cells v2) to ocata (with cells v2), it looks like there's some weird state which has been hit where the compute registration isn't working right any more. I'm seeing this: https://pastebin.com/C9ji6vdS
18:17:04 odyssey4me Our last working test was the last patch we did which was in early December.
18:17:25 odyssey4me Any idea what may have changed to cause the previously working implementation to no longer work?
18:19:22 mriedem odyssey4me: i don't see anything related to that in https://github.com/openstack/nova/commits/stable/ocata going back to november
18:19:29 mriedem are you sure it wasn't something that changed on the OSA side?
18:19:47 odyssey4me mriedem yep, very sure - see https://review.openstack.org/#/q/branch:stable/ocata+project:openstack/openstack-ansible-os_nova
18:21:27 mriedem https://github.com/openstack/nova/commits/stable/ocata/nova/cmd/manage.py
18:21:37 mriedem oldest thing in stable there is from october
18:22:14 odyssey4me yeah, I'm wondering if something has perhaps changed in the database tables either in newton (the source), or in ocata (the destination)
18:22:40 mriedem have you verified if the instances you're verifying actually exist?
18:22:49 mriedem i.e. not just not mapped, but don't actually exist
18:23:04 odyssey4me well, I got stuck there - given that nova-manage doesn't appear to have a way to list the instances :/
18:23:15 odyssey4me any tips for getting a listing out?
18:23:35 odyssey4me the closest I could find is "nova-manage cell_v2 discover_hosts --verbose"
18:24:49 mriedem that doesn't dump the instances
18:25:18 mriedem the api isn't up?
18:25:27 odyssey4me it gives me a set of uuid's which look rather like they belong to instances: https://pastebin.com/mSwpZybQ
18:25:43 mriedem Getting compute nodes from cell 'cell1': 16443e92-e33b-487e-8069-7c80e5bdbc33
18:25:46 mriedem that's a cell mapping uuid
18:25:54 mriedem Checking host mapping for compute host 'ocata-nova1': 6e6d71ab-0b18-416d-8a3a-ce713ac59637
18:25:56 mriedem that's a compute node uuid
18:26:08 mriedem instances aren't the only things that have uuids anymore
18:26:40 odyssey4me the API is up, so I can query things that way if it helps
18:27:18 mriedem worth a shot if you have a local recreate,
18:27:27 mriedem but my guess is the api won't find them either if nova-manage can't
18:27:29 odyssey4me my apologies for dumb questions - it's been a very long time since I actually worked with nova directly :)
18:27:37 mriedem if they don't have instance_mappings in nova_api i mean
18:27:42 mriedem totally fine
18:27:46 mriedem not dumb at all
18:28:07 mriedem if you have a local recreate, you could check the nova_api.instance_mappings table directly
18:28:17 mriedem or the cell db instances table
18:28:20 odyssey4me sure, can do that
18:28:37 mriedem looks like you have 2 cells, so you'd have to look in each
18:29:13 mriedem so you've got 4 dbs (nova_api, nova_cell0, cell1 and then cell 9461149a-52a9-495d-8021-d2cda1645d28)
18:29:21 odyssey4me right, so I have several DB's here: nova, nova_api, nova_cell0; nova_placement
18:29:35 mriedem nova is likely cell1
18:29:36 mriedem yes?
18:29:45 mriedem nova_placement isn't a thing...not sure what that is
18:31:25 odyssey4me ok, that's empty - I'll look into why that's there later
18:31:35 mriedem i probably know why
18:32:09 odyssey4me it might be some leftovers from previous work before things matured
18:33:24 mriedem odyssey4me: yeah https://review.openstack.org/#/q/I31293ac4689630e4113588ab2c6373cf572b8f38
18:34:08 odyssey4me haha, ok - thanks for the reference :)
18:34:18 mriedem odyssey4me: so looking at https://pastebin.com/mSwpZybQ there is something screwed up with the host mappings,
18:34:45 mriedem it looks like this compute is in two cells
18:34:46 mriedem Checking host mapping for compute host 'ocata-nova1': 6e6d71ab-0b18-416d-8a3a-ce713ac59637
18:35:02 odyssey4me if it'd make things simpler I can get your pub key on this host for you to poke around directly?
18:35:36 mriedem i don't think our relationship has hit that level of maturity yet
18:35:36 odyssey4me the host is a temp instance, so nothing special on it
18:35:44 odyssey4me hahaha, fair enough
18:36:37 odyssey4me otherwise, I'll need some guidance with db queries to get data out - I can gist the results as we go
18:36:54 mriedem well, it appears you have 2 nova_api.host_mappings entries for host "ocata-nova1"
18:36:56 mriedem which would be wrong
18:37:09 odyssey4me yeah, that seemed weird to me too
18:37:36 cfriesen is anyone aware of an issue where running "'wget http://169.254.169.254/latest/meta-data/instance-id" in the guest gives a result that is *not* the same as that instance's OS-EXT-SRV-ATTR:instance_name in "nova show"?
18:37:53 mriedem tssurya: dansmith: melwitt: any idea why we don't have a unique constraint across the cell_id and host colums in the host_mappings table?
18:38:33 dansmith mriedem: I thought there was some argument about that with overlapping hostnames (which won't work anyway) ?
18:38:36 mriedem odyssey4me: is this CI supposed to have 2 cells? because multi-cell wasn't supported in ocata
18:38:40 odyssey4me mriedem interestingly enough, I only see one entry in the DB for it
18:39:22 odyssey4me nah, it only has cell0 and cell1
18:39:23 mriedem odyssey4me: do you have a CI run with logs posted?
18:39:32 mriedem https://pastebin.com/mSwpZybQ is saying there are 3 cell mappings
18:39:36 mriedem "Found 3 cell mappings."
18:39:48 mriedem select uuid from nova_api.cell_mappings;
18:40:10 odyssey4me mriedem unfortunately our log collection is broken for ocata, so all we have is console output which only shows brokenness when tempest runs, so that's not very useful
18:40:34 odyssey4me I can fix up the CI to collect logs, but that'll take a few days to make its way through...
18:42:05 odyssey4me hmm, that is odd - three cells showing
18:43:04 odyssey4me it might be my bad, I re-ran 'nova-manage cell_v2 discover_hosts' and 'nova-manage cell_v2 map_instances --cell_uuid ...' a few times
18:43:44 odyssey4me also ran 'nova-manage cell_v2 simple_cell_setup' after the initial build
18:44:03 odyssey4me that gave me 'Cell0 is already setup', so I don't think that broke anything
18:45:40 mriedem discover_hosts is idempotent and doesn't create mappings
18:45:43 mriedem cell mappings i mean
18:45:48 mriedem map_instances should also be ok
18:45:57 mriedem my guess is something go f'ed up when running simple_cell_setup a few times
18:46:23 odyssey4me I only did it once :)
18:48:04 mriedem was create_cell ever called?
18:48:35 odyssey4me yep - this is the basic set of steps that would have executed before the discovery: https://github.com/openstack/openstack-ansible-os_nova/blob/stable/ocata/tasks/nova_db_setup.yml#L16-L55
18:49:03 odyssey4me so in this case, the initial api_db sync would have been skipped as this was an existing DB
18:49:23 odyssey4me then the cell0 map done: https://github.com/openstack/openstack-ansible-os_nova/blob/stable/ocata/tasks/nova_db_setup.yml#L31
18:49:37 odyssey4me then the creation of cell1: https://github.com/openstack/openstack-ansible-os_nova/blob/stable/ocata/tasks/nova_db_setup.yml#L37
18:49:54 odyssey4me then the api_db sync, and db sync
18:50:22 mriedem where does map_instances happen?
18:50:36 mriedem https://github.com/openstack/openstack-ansible-os_nova/blob/stable/ocata/tasks/nova_db_post_setup.yml#L38
18:50:38 mriedem found it
18:51:04 odyssey4me we wait for a compute instance to be ready in the handler: https://github.com/openstack/openstack-ansible-os_nova/blob/stable/ocata/handlers/main.yml#L67-L76
18:51:14 odyssey4me then yes, https://github.com/openstack/openstack-ansible-os_nova/blob/stable/ocata/tasks/nova_db_post_setup.yml gets run
18:53:15 odyssey4me you'll find those tasks, and their stdout/stderr in this console log: http://logs.openstack.org/29/540329/5/check/openstack-ansible-upgrade-ubuntu-xenial/30ae3e2/job-output.txt.gz
18:53:36 odyssey4me it's a bit verbose, apologies in advance
18:56:38 kashyap dansmith: When you get a moment, wrote this here: https://review.openstack.org/#/c/497457/18
18:56:51 kashyap Please correct / critique / answer as you see fit.
18:57:34 mriedem odyssey4me: hmm, for https://github.com/openstack/openstack-ansible-os_nova/blob/stable/ocata/tasks/nova_db_post_setup.yml#L39
18:57:39 mriedem --cell_uuid {{ cell1_uuid['stdout'].split()[3] }}"
18:57:44 mriedem are you sure that's hitting cell1 and not cell0?
18:58:07 mriedem oh i guess because of https://github.com/openstack/openstack-ansible-os_nova/blob/stable/ocata/tasks/nova_db_post_setup.yml#L32
18:58:13 mriedem so nvm
18:58:14 odyssey4me yep, exactly
18:58:33 odyssey4me if someone names their cell "cell12" then it might not be right - but we're using safe defaults here
18:58:50 kashyap dansmith: To finish up, for completeness' sake: The QEMU / libvirt 3.9.0 "pause-before-switchover" thing -- it won't be useful in Nova's case, as that will stop the guest CPUs, which will extend the guest down time.)

Earlier   Later