Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-02-06
18:35:44 odyssey4me hahaha, fair enough
18:36:37 odyssey4me otherwise, I'll need some guidance with db queries to get data out - I can gist the results as we go
18:36:54 mriedem well, it appears you have 2 nova_api.host_mappings entries for host "ocata-nova1"
18:36:56 mriedem which would be wrong
18:37:09 odyssey4me yeah, that seemed weird to me too
18:37:36 cfriesen is anyone aware of an issue where running "'wget http://169.254.169.254/latest/meta-data/instance-id" in the guest gives a result that is *not* the same as that instance's OS-EXT-SRV-ATTR:instance_name in "nova show"?
18:37:53 mriedem tssurya: dansmith: melwitt: any idea why we don't have a unique constraint across the cell_id and host colums in the host_mappings table?
18:38:33 dansmith mriedem: I thought there was some argument about that with overlapping hostnames (which won't work anyway) ?
18:38:36 mriedem odyssey4me: is this CI supposed to have 2 cells? because multi-cell wasn't supported in ocata
18:38:40 odyssey4me mriedem interestingly enough, I only see one entry in the DB for it
18:39:22 odyssey4me nah, it only has cell0 and cell1
18:39:23 mriedem odyssey4me: do you have a CI run with logs posted?
18:39:32 mriedem https://pastebin.com/mSwpZybQ is saying there are 3 cell mappings
18:39:36 mriedem "Found 3 cell mappings."
18:39:48 mriedem select uuid from nova_api.cell_mappings;
18:40:10 odyssey4me mriedem unfortunately our log collection is broken for ocata, so all we have is console output which only shows brokenness when tempest runs, so that's not very useful
18:40:34 odyssey4me I can fix up the CI to collect logs, but that'll take a few days to make its way through...
18:42:05 odyssey4me hmm, that is odd - three cells showing
18:43:04 odyssey4me it might be my bad, I re-ran 'nova-manage cell_v2 discover_hosts' and 'nova-manage cell_v2 map_instances --cell_uuid ...' a few times
18:43:44 odyssey4me also ran 'nova-manage cell_v2 simple_cell_setup' after the initial build
18:44:03 odyssey4me that gave me 'Cell0 is already setup', so I don't think that broke anything
18:45:40 mriedem discover_hosts is idempotent and doesn't create mappings
18:45:43 mriedem cell mappings i mean
18:45:48 mriedem map_instances should also be ok
18:45:57 mriedem my guess is something go f'ed up when running simple_cell_setup a few times
18:46:23 odyssey4me I only did it once :)
18:48:04 mriedem was create_cell ever called?
18:48:35 odyssey4me yep - this is the basic set of steps that would have executed before the discovery: https://github.com/openstack/openstack-ansible-os_nova/blob/stable/ocata/tasks/nova_db_setup.yml#L16-L55
18:49:03 odyssey4me so in this case, the initial api_db sync would have been skipped as this was an existing DB
18:49:23 odyssey4me then the cell0 map done: https://github.com/openstack/openstack-ansible-os_nova/blob/stable/ocata/tasks/nova_db_setup.yml#L31
18:49:37 odyssey4me then the creation of cell1: https://github.com/openstack/openstack-ansible-os_nova/blob/stable/ocata/tasks/nova_db_setup.yml#L37
18:49:54 odyssey4me then the api_db sync, and db sync
18:50:22 mriedem where does map_instances happen?
18:50:36 mriedem https://github.com/openstack/openstack-ansible-os_nova/blob/stable/ocata/tasks/nova_db_post_setup.yml#L38
18:50:38 mriedem found it
18:51:04 odyssey4me we wait for a compute instance to be ready in the handler: https://github.com/openstack/openstack-ansible-os_nova/blob/stable/ocata/handlers/main.yml#L67-L76
18:51:14 odyssey4me then yes, https://github.com/openstack/openstack-ansible-os_nova/blob/stable/ocata/tasks/nova_db_post_setup.yml gets run
18:53:15 odyssey4me you'll find those tasks, and their stdout/stderr in this console log: http://logs.openstack.org/29/540329/5/check/openstack-ansible-upgrade-ubuntu-xenial/30ae3e2/job-output.txt.gz
18:53:36 odyssey4me it's a bit verbose, apologies in advance
18:56:38 kashyap dansmith: When you get a moment, wrote this here: https://review.openstack.org/#/c/497457/18
18:56:51 kashyap Please correct / critique / answer as you see fit.
18:57:34 mriedem odyssey4me: hmm, for https://github.com/openstack/openstack-ansible-os_nova/blob/stable/ocata/tasks/nova_db_post_setup.yml#L39
18:57:39 mriedem --cell_uuid {{ cell1_uuid['stdout'].split()[3] }}"
18:57:44 mriedem are you sure that's hitting cell1 and not cell0?
18:58:07 mriedem oh i guess because of https://github.com/openstack/openstack-ansible-os_nova/blob/stable/ocata/tasks/nova_db_post_setup.yml#L32
18:58:13 mriedem so nvm
18:58:14 odyssey4me yep, exactly
18:58:33 odyssey4me if someone names their cell "cell12" then it might not be right - but we're using safe defaults here
18:58:50 kashyap dansmith: To finish up, for completeness' sake: The QEMU / libvirt 3.9.0 "pause-before-switchover" thing -- it won't be useful in Nova's case, as that will stop the guest CPUs, which will extend the guest down time.)
18:59:02 odyssey4me although the spacing in that grep should be reasonable to protect the boundaries
19:03:14 mriedem odyssey4me: just to verify, this is running on the newton code before upgrading to ocata yes?
19:03:30 mriedem because http://logs.openstack.org/29/540329/5/check/openstack-ansible-upgrade-ubuntu-xenial/30ae3e2/job-output.txt.gz#_2018-02-05_18_10_32_079153 returns version 21 for the nova_api db which was the version in newton
19:04:06 odyssey4me yep, the initial build from be from what was the head of stable/newton at the time the test ran
19:04:17 odyssey4me I'll push a patch up to make it use the EOL branch now :)
19:04:29 odyssey4me oh hang on, I think it mighth be sha pinned
19:04:45 mriedem there was only one nova_api db change in newton before eol https://github.com/openstack/nova/blob/newton-eol/nova/db/sqlalchemy/api_migrations/migrate_repo/versions/022_request_specs_spec_mediumtext.py
19:04:53 mriedem and shouldn't have anything to do with what you're seeing
19:05:04 odyssey4me heh, that makes it even weirder that this failed: https://github.com/openstack/openstack-ansible-tests/blob/stable/newton/test-vars.yml#L157
19:05:35 dansmith kashyap: yeah that's why I said it didn't seem like it would :)
19:06:58 mriedem odyssey4me: so everything in that console output looks ok to me,
19:07:10 mriedem where do instances actually get created in newton before the upgrade is started?
19:07:14 dansmith kashyap: I was also really looking for you to comment on the potential race with the setting of the bandwidth limit in two places.. did you look at that at all?
19:07:29 mriedem odyssey4me: i.e. how do you know that you've actually got instances from newton to map when map_instances runs?
19:07:36 kashyap dansmith: No, not yet. After I finish dinner
19:08:33 odyssey4me mriedem by 'instances' do you mean the compute host? The terminology is confusing to me here... because in this CI run no instance (by this I mean a cloud instance, not a nova hypervisor) is created until *after* the upgrade when tempest executes.
19:09:41 mriedem instances == vms
19:09:52 mriedem odyssey4me: ok then map_instances isn't going to map anything :)
19:10:02 mriedem map_instances means create nova_api.instance_mappings records in the db
19:10:07 mriedem based on VMs in the 'nova' db
19:10:11 mriedem the 'nova' db is the cell1 db
19:11:08 mriedem odyssey4me: so looking at https://pastebin.com/C9ji6vdS again, when/how did either of those instances get created?
19:11:13 mriedem the ones passed to verify_instance
19:11:13 odyssey4me ok, btw if we're doing something dumb here then please feel free to say so with any improvement suggestions :)
19:11:23 mriedem odyssey4me: the ansible tasks all look fine
19:11:39 mriedem the ansible is like a step for step copy of what's in the install guide here https://docs.openstack.org/nova/latest/user/cells.html
19:11:59 odyssey4me ok, given that I did those commands after the test failed, those instances would have been created by tempest after the upgrade completed
19:12:18 odyssey4me the one thing that may not have run here is the online migrations
19:12:26 dansmith kashyap: okay thanks
19:12:42 mriedem odyssey4me: do you see any errors in the nova-api logs?
19:12:46 mriedem or nova-conductor?
19:13:24 mriedem also, based on that git hash for nova, the CI isn't picking up any bug fixes since the newton GA
19:13:26 mriedem which seems less than ideal
19:16:24 odyssey4me nova-conductor: No host-to-cell mapping found for selected host ocata-nova1. Setup is incomplete.
19:16:42 odyssey4me Failed to compute_task_build_instances: Host 'ocata-nova1' is not mapped to any cell
19:17:48 hrw https://marcin.juszkiewicz.com.pl/2018/02/06/graphical-console-in-openstack-aarch64/ - please read ;)
19:24:15 tssurya odyssey4me: can you check the value of the "mapped" column inside compute_nodes table of the nova db (cell1 db), ? if you have this host there, then there should be a record
19:26:05 odyssey4me hmm, select mapped from compute_nodes; gives me an unknown column error
19:26:50 odyssey4me yup, none of those are working
19:27:02 odyssey4me none of the db's have that column
19:29:49 tssurya there should be a table called compute_nodes only in your nova db
19:29:52 tssurya not in the api
19:30:41 tssurya odyssey4me: that is in the cell1's db
19:31:02 odyssey4me yup, the nova db has that table - but the table has no 'mapped' column
19:32:26 odyssey4me tssurya here're the columns present: https://pastebin.com/grmGWEQe
19:33:11 odyssey4me tssurya in case you missed it, this is a newton build (with no cells) being upgraded to an ocata build (with cells being setup)
19:33:20 odyssey4me cells v2 to be clear
19:35:20 tssurya odyssey4me: oh okay, ocata..
19:35:49 odyssey4me tssurya yep :) ye olde crusty stable code ;)
19:37:02 odyssey4me mriedem any further thoughts or ideas? if not I'll work on fixing up the log capturing so that I can point you at a proper set of logs to peruse
19:37:54 mriedem odyssey4me: sorry was eating lunch,

Earlier   Later