| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-02-06 | |||
| 18:43:44 | odyssey4me | also ran 'nova-manage cell_v2 simple_cell_setup' after the initial build | |
| 18:44:03 | odyssey4me | that gave me 'Cell0 is already setup', so I don't think that broke anything | |
| 18:45:40 | mriedem | discover_hosts is idempotent and doesn't create mappings | |
| 18:45:43 | mriedem | cell mappings i mean | |
| 18:45:48 | mriedem | map_instances should also be ok | |
| 18:45:57 | mriedem | my guess is something go f'ed up when running simple_cell_setup a few times | |
| 18:46:23 | odyssey4me | I only did it once :) | |
| 18:48:04 | mriedem | was create_cell ever called? | |
| 18:48:35 | odyssey4me | yep - this is the basic set of steps that would have executed before the discovery: https://github.com/openstack/openstack-ansible-os_nova/blob/stable/ocata/tasks/nova_db_setup.yml#L16-L55 | |
| 18:49:03 | odyssey4me | so in this case, the initial api_db sync would have been skipped as this was an existing DB | |
| 18:49:23 | odyssey4me | then the cell0 map done: https://github.com/openstack/openstack-ansible-os_nova/blob/stable/ocata/tasks/nova_db_setup.yml#L31 | |
| 18:49:37 | odyssey4me | then the creation of cell1: https://github.com/openstack/openstack-ansible-os_nova/blob/stable/ocata/tasks/nova_db_setup.yml#L37 | |
| 18:49:54 | odyssey4me | then the api_db sync, and db sync | |
| 18:50:22 | mriedem | where does map_instances happen? | |
| 18:50:36 | mriedem | https://github.com/openstack/openstack-ansible-os_nova/blob/stable/ocata/tasks/nova_db_post_setup.yml#L38 | |
| 18:50:38 | mriedem | found it | |
| 18:51:04 | odyssey4me | we wait for a compute instance to be ready in the handler: https://github.com/openstack/openstack-ansible-os_nova/blob/stable/ocata/handlers/main.yml#L67-L76 | |
| 18:51:14 | odyssey4me | then yes, https://github.com/openstack/openstack-ansible-os_nova/blob/stable/ocata/tasks/nova_db_post_setup.yml gets run | |
| 18:53:15 | odyssey4me | you'll find those tasks, and their stdout/stderr in this console log: http://logs.openstack.org/29/540329/5/check/openstack-ansible-upgrade-ubuntu-xenial/30ae3e2/job-output.txt.gz | |
| 18:53:36 | odyssey4me | it's a bit verbose, apologies in advance | |
| 18:56:38 | kashyap | dansmith: When you get a moment, wrote this here: https://review.openstack.org/#/c/497457/18 | |
| 18:56:51 | kashyap | Please correct / critique / answer as you see fit. | |
| 18:57:34 | mriedem | odyssey4me: hmm, for https://github.com/openstack/openstack-ansible-os_nova/blob/stable/ocata/tasks/nova_db_post_setup.yml#L39 | |
| 18:57:39 | mriedem | --cell_uuid {{ cell1_uuid['stdout'].split()[3] }}" | |
| 18:57:44 | mriedem | are you sure that's hitting cell1 and not cell0? | |
| 18:58:07 | mriedem | oh i guess because of https://github.com/openstack/openstack-ansible-os_nova/blob/stable/ocata/tasks/nova_db_post_setup.yml#L32 | |
| 18:58:13 | mriedem | so nvm | |
| 18:58:14 | odyssey4me | yep, exactly | |
| 18:58:33 | odyssey4me | if someone names their cell "cell12" then it might not be right - but we're using safe defaults here | |
| 18:58:50 | kashyap | dansmith: To finish up, for completeness' sake: The QEMU / libvirt 3.9.0 "pause-before-switchover" thing -- it won't be useful in Nova's case, as that will stop the guest CPUs, which will extend the guest down time.) | |
| 18:59:02 | odyssey4me | although the spacing in that grep should be reasonable to protect the boundaries | |
| 19:03:14 | mriedem | odyssey4me: just to verify, this is running on the newton code before upgrading to ocata yes? | |
| 19:03:30 | mriedem | because http://logs.openstack.org/29/540329/5/check/openstack-ansible-upgrade-ubuntu-xenial/30ae3e2/job-output.txt.gz#_2018-02-05_18_10_32_079153 returns version 21 for the nova_api db which was the version in newton | |
| 19:04:06 | odyssey4me | yep, the initial build from be from what was the head of stable/newton at the time the test ran | |
| 19:04:17 | odyssey4me | I'll push a patch up to make it use the EOL branch now :) | |
| 19:04:29 | odyssey4me | oh hang on, I think it mighth be sha pinned | |
| 19:04:45 | mriedem | there was only one nova_api db change in newton before eol https://github.com/openstack/nova/blob/newton-eol/nova/db/sqlalchemy/api_migrations/migrate_repo/versions/022_request_specs_spec_mediumtext.py | |
| 19:04:53 | mriedem | and shouldn't have anything to do with what you're seeing | |
| 19:05:04 | odyssey4me | heh, that makes it even weirder that this failed: https://github.com/openstack/openstack-ansible-tests/blob/stable/newton/test-vars.yml#L157 | |
| 19:05:35 | dansmith | kashyap: yeah that's why I said it didn't seem like it would :) | |
| 19:06:58 | mriedem | odyssey4me: so everything in that console output looks ok to me, | |
| 19:07:10 | mriedem | where do instances actually get created in newton before the upgrade is started? | |
| 19:07:14 | dansmith | kashyap: I was also really looking for you to comment on the potential race with the setting of the bandwidth limit in two places.. did you look at that at all? | |
| 19:07:29 | mriedem | odyssey4me: i.e. how do you know that you've actually got instances from newton to map when map_instances runs? | |
| 19:07:36 | kashyap | dansmith: No, not yet. After I finish dinner | |
| 19:08:33 | odyssey4me | mriedem by 'instances' do you mean the compute host? The terminology is confusing to me here... because in this CI run no instance (by this I mean a cloud instance, not a nova hypervisor) is created until *after* the upgrade when tempest executes. | |
| 19:09:41 | mriedem | instances == vms | |
| 19:09:52 | mriedem | odyssey4me: ok then map_instances isn't going to map anything :) | |
| 19:10:02 | mriedem | map_instances means create nova_api.instance_mappings records in the db | |
| 19:10:07 | mriedem | based on VMs in the 'nova' db | |
| 19:10:11 | mriedem | the 'nova' db is the cell1 db | |
| 19:11:08 | mriedem | odyssey4me: so looking at https://pastebin.com/C9ji6vdS again, when/how did either of those instances get created? | |
| 19:11:13 | odyssey4me | ok, btw if we're doing something dumb here then please feel free to say so with any improvement suggestions :) | |
| 19:11:13 | mriedem | the ones passed to verify_instance | |
| 19:11:23 | mriedem | odyssey4me: the ansible tasks all look fine | |
| 19:11:39 | mriedem | the ansible is like a step for step copy of what's in the install guide here https://docs.openstack.org/nova/latest/user/cells.html | |
| 19:11:59 | odyssey4me | ok, given that I did those commands after the test failed, those instances would have been created by tempest after the upgrade completed | |
| 19:12:18 | odyssey4me | the one thing that may not have run here is the online migrations | |
| 19:12:26 | dansmith | kashyap: okay thanks | |
| 19:12:42 | mriedem | odyssey4me: do you see any errors in the nova-api logs? | |
| 19:12:46 | mriedem | or nova-conductor? | |
| 19:13:24 | mriedem | also, based on that git hash for nova, the CI isn't picking up any bug fixes since the newton GA | |
| 19:13:26 | mriedem | which seems less than ideal | |
| 19:16:24 | odyssey4me | nova-conductor: No host-to-cell mapping found for selected host ocata-nova1. Setup is incomplete. | |
| 19:16:42 | odyssey4me | Failed to compute_task_build_instances: Host 'ocata-nova1' is not mapped to any cell | |
| 19:17:48 | hrw | https://marcin.juszkiewicz.com.pl/2018/02/06/graphical-console-in-openstack-aarch64/ - please read ;) | |
| 19:24:15 | tssurya | odyssey4me: can you check the value of the "mapped" column inside compute_nodes table of the nova db (cell1 db), ? if you have this host there, then there should be a record | |
| 19:26:05 | odyssey4me | hmm, select mapped from compute_nodes; gives me an unknown column error | |
| 19:26:50 | odyssey4me | yup, none of those are working | |
| 19:27:02 | odyssey4me | none of the db's have that column | |
| 19:29:49 | tssurya | there should be a table called compute_nodes only in your nova db | |
| 19:29:52 | tssurya | not in the api | |
| 19:30:41 | tssurya | odyssey4me: that is in the cell1's db | |
| 19:31:02 | odyssey4me | yup, the nova db has that table - but the table has no 'mapped' column | |
| 19:32:26 | odyssey4me | tssurya here're the columns present: https://pastebin.com/grmGWEQe | |
| 19:33:11 | odyssey4me | tssurya in case you missed it, this is a newton build (with no cells) being upgraded to an ocata build (with cells being setup) | |
| 19:33:20 | odyssey4me | cells v2 to be clear | |
| 19:35:20 | tssurya | odyssey4me: oh okay, ocata.. | |
| 19:35:49 | odyssey4me | tssurya yep :) ye olde crusty stable code ;) | |
| 19:37:02 | odyssey4me | mriedem any further thoughts or ideas? if not I'll work on fixing up the log capturing so that I can point you at a proper set of logs to peruse | |
| 19:37:54 | mriedem | odyssey4me: sorry was eating lunch, | |
| 19:38:01 | mriedem | i think the compute_nodes.mapped column was added in ocata | |
| 19:38:31 | mriedem | the host 'ocata-nova1' is a problem i think, since an earlier paste showed it was discovered in 2 different cell mappings | |
| 19:38:47 | mriedem | https://pastebin.com/mSwpZybQ | |
| 19:39:13 | mriedem | at this point i'd probably need the logs from an untouched env | |
| 19:39:38 | odyssey4me | alright, thanks much for your time and input so far | |
| 19:39:41 | mriedem | yw | |
| 19:40:21 | odyssey4me | I'll work on getting the log collection fixed up so that we can debug better. | |
| 19:44:30 | efried | mriedem: I'm going to put up three pairs of patches to dig a bit deeper, see which part of ServiceTokenAuthWrapper is actually busted. | |
| 19:44:36 | efried | mriedem: Unless you have some other plan. | |
| 19:44:42 | mriedem | so 6 patches? | |
| 19:45:04 | mriedem | i don't, no :) | |
| 19:45:21 | mriedem | we might have to put this in the release notes as a known issue | |
| 19:45:46 | efried | Yeah, going to pass through each of the service_auth-y bits of ServiceTokenAuthWrapper - which I have to do in ksa - and then a blank Nova patch for each that Depends-On its ksa buddy. | |
| 19:45:52 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Create volume attachment during boot from volume in compute https://review.openstack.org/541420 | |
| 19:45:53 | mriedem | ildikov: ^ cleans up that bfv legacy attach stuff | |
| 19:47:10 | ildikov | mriedem: tnx, looking | |
| 19:47:36 | openstackgerrit | Matt Riedemann proposed openstack/nova master: DNM: debug nova-api service version check during boot from volume https://review.openstack.org/540992 | |
| 19:51:04 | mriedem | holy f 679 check queue length | |
| 19:52:21 | melwitt | I saw there was a status message about zuul from 02:30 having problems | |