| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-02-06 | |||
| 18:13:43 | mriedem | stvnoyes: that's b/c it's not set in tempest.conf | |
| 18:13:43 | mriedem | 2018-02-06 09:16:16.276 12703 DEBUG tempest [-] compute-feature-enabled.volume_multiattach = False log_opt_values /usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py:2898 | |
| 18:13:47 | mriedem | https://openstackci.oracle.com/zfssa_ci/refs-changes-68-541368-1-iscsi/tempest.log | |
| 18:13:56 | openstackgerrit | Dan Smith proposed openstack/nova master: Make service all-cells min version helper use scatter-gather https://review.openstack.org/539590 | |
| 18:13:56 | openstackgerrit | Dan Smith proposed openstack/nova master: Make host_manager use scatter-gather and ignore down cells https://review.openstack.org/539617 | |
| 18:13:57 | openstackgerrit | Patricia Domingues proposed openstack/nova master: load up the volume drivers by checking architecture https://review.openstack.org/541393 | |
| 18:14:07 | mriedem | stvnoyes: where is the code that controls the local.conf for the devstack setup in this CI? | |
| 18:14:13 | mriedem | is it internal to oracle? or on github? | |
| 18:15:40 | stvnoyes | mriedem: good question. idk. I haven't played with CI stuff so all this is pretty new to me. I will see if there's someone around on the zfssa team who would know. | |
| 18:16:25 | mriedem | stvnoyes: https://wiki.openstack.org/wiki/ThirdPartySystems/Oracle_ZFSSA_CI | |
| 18:16:36 | odyssey4me | howdy all - we're having some trouble in the OSA deployment test for upgrades from newton (without cells v2) to ocata (with cells v2), it looks like there's some weird state which has been hit where the compute registration isn't working right any more. I'm seeing this: https://pastebin.com/C9ji6vdS | |
| 18:17:04 | odyssey4me | Our last working test was the last patch we did which was in early December. | |
| 18:17:25 | odyssey4me | Any idea what may have changed to cause the previously working implementation to no longer work? | |
| 18:19:22 | mriedem | odyssey4me: i don't see anything related to that in https://github.com/openstack/nova/commits/stable/ocata going back to november | |
| 18:19:29 | mriedem | are you sure it wasn't something that changed on the OSA side? | |
| 18:19:47 | odyssey4me | mriedem yep, very sure - see https://review.openstack.org/#/q/branch:stable/ocata+project:openstack/openstack-ansible-os_nova | |
| 18:21:27 | mriedem | https://github.com/openstack/nova/commits/stable/ocata/nova/cmd/manage.py | |
| 18:21:37 | mriedem | oldest thing in stable there is from october | |
| 18:22:14 | odyssey4me | yeah, I'm wondering if something has perhaps changed in the database tables either in newton (the source), or in ocata (the destination) | |
| 18:22:40 | mriedem | have you verified if the instances you're verifying actually exist? | |
| 18:22:49 | mriedem | i.e. not just not mapped, but don't actually exist | |
| 18:23:04 | odyssey4me | well, I got stuck there - given that nova-manage doesn't appear to have a way to list the instances :/ | |
| 18:23:15 | odyssey4me | any tips for getting a listing out? | |
| 18:23:35 | odyssey4me | the closest I could find is "nova-manage cell_v2 discover_hosts --verbose" | |
| 18:24:49 | mriedem | that doesn't dump the instances | |
| 18:25:18 | mriedem | the api isn't up? | |
| 18:25:27 | odyssey4me | it gives me a set of uuid's which look rather like they belong to instances: https://pastebin.com/mSwpZybQ | |
| 18:25:43 | mriedem | Getting compute nodes from cell 'cell1': 16443e92-e33b-487e-8069-7c80e5bdbc33 | |
| 18:25:46 | mriedem | that's a cell mapping uuid | |
| 18:25:54 | mriedem | Checking host mapping for compute host 'ocata-nova1': 6e6d71ab-0b18-416d-8a3a-ce713ac59637 | |
| 18:25:56 | mriedem | that's a compute node uuid | |
| 18:26:08 | mriedem | instances aren't the only things that have uuids anymore | |
| 18:26:40 | odyssey4me | the API is up, so I can query things that way if it helps | |
| 18:27:18 | mriedem | worth a shot if you have a local recreate, | |
| 18:27:27 | mriedem | but my guess is the api won't find them either if nova-manage can't | |
| 18:27:29 | odyssey4me | my apologies for dumb questions - it's been a very long time since I actually worked with nova directly :) | |
| 18:27:37 | mriedem | if they don't have instance_mappings in nova_api i mean | |
| 18:27:42 | mriedem | totally fine | |
| 18:27:46 | mriedem | not dumb at all | |
| 18:28:07 | mriedem | if you have a local recreate, you could check the nova_api.instance_mappings table directly | |
| 18:28:17 | mriedem | or the cell db instances table | |
| 18:28:20 | odyssey4me | sure, can do that | |
| 18:28:37 | mriedem | looks like you have 2 cells, so you'd have to look in each | |
| 18:29:13 | mriedem | so you've got 4 dbs (nova_api, nova_cell0, cell1 and then cell 9461149a-52a9-495d-8021-d2cda1645d28) | |
| 18:29:21 | odyssey4me | right, so I have several DB's here: nova, nova_api, nova_cell0; nova_placement | |
| 18:29:35 | mriedem | nova is likely cell1 | |
| 18:29:36 | mriedem | yes? | |
| 18:29:45 | mriedem | nova_placement isn't a thing...not sure what that is | |
| 18:31:25 | odyssey4me | ok, that's empty - I'll look into why that's there later | |
| 18:31:35 | mriedem | i probably know why | |
| 18:32:09 | odyssey4me | it might be some leftovers from previous work before things matured | |
| 18:33:24 | mriedem | odyssey4me: yeah https://review.openstack.org/#/q/I31293ac4689630e4113588ab2c6373cf572b8f38 | |
| 18:34:08 | odyssey4me | haha, ok - thanks for the reference :) | |
| 18:34:18 | mriedem | odyssey4me: so looking at https://pastebin.com/mSwpZybQ there is something screwed up with the host mappings, | |
| 18:34:45 | mriedem | it looks like this compute is in two cells | |
| 18:34:46 | mriedem | Checking host mapping for compute host 'ocata-nova1': 6e6d71ab-0b18-416d-8a3a-ce713ac59637 | |
| 18:35:02 | odyssey4me | if it'd make things simpler I can get your pub key on this host for you to poke around directly? | |
| 18:35:36 | odyssey4me | the host is a temp instance, so nothing special on it | |
| 18:35:36 | mriedem | i don't think our relationship has hit that level of maturity yet | |
| 18:35:44 | odyssey4me | hahaha, fair enough | |
| 18:36:37 | odyssey4me | otherwise, I'll need some guidance with db queries to get data out - I can gist the results as we go | |
| 18:36:54 | mriedem | well, it appears you have 2 nova_api.host_mappings entries for host "ocata-nova1" | |
| 18:36:56 | mriedem | which would be wrong | |
| 18:37:09 | odyssey4me | yeah, that seemed weird to me too | |
| 18:37:36 | cfriesen | is anyone aware of an issue where running "'wget http://169.254.169.254/latest/meta-data/instance-id" in the guest gives a result that is *not* the same as that instance's OS-EXT-SRV-ATTR:instance_name in "nova show"? | |
| 18:37:53 | mriedem | tssurya: dansmith: melwitt: any idea why we don't have a unique constraint across the cell_id and host colums in the host_mappings table? | |
| 18:38:33 | dansmith | mriedem: I thought there was some argument about that with overlapping hostnames (which won't work anyway) ? | |
| 18:38:36 | mriedem | odyssey4me: is this CI supposed to have 2 cells? because multi-cell wasn't supported in ocata | |
| 18:38:40 | odyssey4me | mriedem interestingly enough, I only see one entry in the DB for it | |
| 18:39:22 | odyssey4me | nah, it only has cell0 and cell1 | |
| 18:39:23 | mriedem | odyssey4me: do you have a CI run with logs posted? | |
| 18:39:32 | mriedem | https://pastebin.com/mSwpZybQ is saying there are 3 cell mappings | |
| 18:39:36 | mriedem | "Found 3 cell mappings." | |
| 18:39:48 | mriedem | select uuid from nova_api.cell_mappings; | |
| 18:40:10 | odyssey4me | mriedem unfortunately our log collection is broken for ocata, so all we have is console output which only shows brokenness when tempest runs, so that's not very useful | |
| 18:40:34 | odyssey4me | I can fix up the CI to collect logs, but that'll take a few days to make its way through... | |
| 18:42:05 | odyssey4me | hmm, that is odd - three cells showing | |
| 18:43:04 | odyssey4me | it might be my bad, I re-ran 'nova-manage cell_v2 discover_hosts' and 'nova-manage cell_v2 map_instances --cell_uuid ...' a few times | |
| 18:43:44 | odyssey4me | also ran 'nova-manage cell_v2 simple_cell_setup' after the initial build | |
| 18:44:03 | odyssey4me | that gave me 'Cell0 is already setup', so I don't think that broke anything | |
| 18:45:40 | mriedem | discover_hosts is idempotent and doesn't create mappings | |
| 18:45:43 | mriedem | cell mappings i mean | |
| 18:45:48 | mriedem | map_instances should also be ok | |
| 18:45:57 | mriedem | my guess is something go f'ed up when running simple_cell_setup a few times | |
| 18:46:23 | odyssey4me | I only did it once :) | |
| 18:48:04 | mriedem | was create_cell ever called? | |
| 18:48:35 | odyssey4me | yep - this is the basic set of steps that would have executed before the discovery: https://github.com/openstack/openstack-ansible-os_nova/blob/stable/ocata/tasks/nova_db_setup.yml#L16-L55 | |
| 18:49:03 | odyssey4me | so in this case, the initial api_db sync would have been skipped as this was an existing DB | |
| 18:49:23 | odyssey4me | then the cell0 map done: https://github.com/openstack/openstack-ansible-os_nova/blob/stable/ocata/tasks/nova_db_setup.yml#L31 | |
| 18:49:37 | odyssey4me | then the creation of cell1: https://github.com/openstack/openstack-ansible-os_nova/blob/stable/ocata/tasks/nova_db_setup.yml#L37 | |
| 18:49:54 | odyssey4me | then the api_db sync, and db sync | |
| 18:50:22 | mriedem | where does map_instances happen? | |
| 18:50:36 | mriedem | https://github.com/openstack/openstack-ansible-os_nova/blob/stable/ocata/tasks/nova_db_post_setup.yml#L38 | |
| 18:50:38 | mriedem | found it | |
| 18:51:04 | odyssey4me | we wait for a compute instance to be ready in the handler: https://github.com/openstack/openstack-ansible-os_nova/blob/stable/ocata/handlers/main.yml#L67-L76 | |
| 18:51:14 | odyssey4me | then yes, https://github.com/openstack/openstack-ansible-os_nova/blob/stable/ocata/tasks/nova_db_post_setup.yml gets run | |
| 18:53:15 | odyssey4me | you'll find those tasks, and their stdout/stderr in this console log: http://logs.openstack.org/29/540329/5/check/openstack-ansible-upgrade-ubuntu-xenial/30ae3e2/job-output.txt.gz | |
| 18:53:36 | odyssey4me | it's a bit verbose, apologies in advance | |
| 18:56:38 | kashyap | dansmith: When you get a moment, wrote this here: https://review.openstack.org/#/c/497457/18 | |
| 18:56:51 | kashyap | Please correct / critique / answer as you see fit. | |