| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-02-06 | |||
| 18:02:53 | dansmith | it waits to exit the context until timeout or the event arrives | |
| 18:03:22 | kashyap | I see. | |
| 18:03:44 | kashyap | Is this what the Neutron events telling Nova: | |
| 18:04:00 | kashyap | "Hey, I've done with doing all the stuff I need to do, so I'm signalling that by sending out this event (from the above Neutron commit). Over to you now." | |
| 18:04:22 | kashyap | I.e. the destination Neutron has completed virtual interface plugging. | |
| 18:04:40 | dansmith | yes | |
| 18:04:54 | dansmith | I mean, that's what vif-plugged means | |
| 18:05:24 | kashyap | Noted. | |
| 18:05:31 | stvnoyes | mriedem: matt, I added a change to enable multiattach on the zfssa array, but when tempest runs during the zfssa ci, it's not running with multiattach enabled (in tempest.conf). Where would I make a change to have this enabled when running the Oracle ZFSSA CI? thanks... | |
| 18:05:43 | sean-k-mooney | dansmith: actully vif-plugged has different menanings | |
| 18:06:13 | sean-k-mooney | vif-plugged in the nova-compute agent log means os-vif finisshed plugging the interface | |
| 18:06:33 | sean-k-mooney | vif-plugged event form neutron means neutron finished wireing it up | |
| 18:06:54 | dansmith | sean-k-mooney: we're talking about when neutron does a POST of the event to nova's external event interface | |
| 18:07:27 | dansmith | and it better (still) mean that the wiring is done, else we have a problem :) | |
| 18:07:32 | mriedem | stvnoyes: do you have these set in devstack? https://github.com/openstack/nova/blob/master/playbooks/legacy/nova-multiattach/run.yaml#L31 | |
| 18:07:46 | sean-k-mooney | dansmith: ah then yes that is sent when neutron finishes wiring up the port that was plugged by nova/os-vif/libvirt into the network backend so your correct in what you said | |
| 18:08:17 | mriedem | stvnoyes: also, is tempest running against master nova so the 2.60 microversion is available? | |
| 18:08:37 | dansmith | sean-k-mooney: thanks :) | |
| 18:09:16 | kashyap | sean-k-mooney: Hey, I'm about to write some questions on this change: https://review.openstack.org/#/c/497457/18 | |
| 18:09:26 | kashyap | sean-k-mooney: Post some design discussion w/ QEMU & libvirt folks | |
| 18:09:39 | kashyap | Would appreciate if you also look at it | |
| 18:10:24 | sean-k-mooney | kashyap: am i am behind in some internal paper work but ill try and take a look tomorow | |
| 18:10:37 | kashyap | sean-k-mooney: Sure, tomm is fine; I'm already late for dinner by an hour | |
| 18:10:55 | kashyap | Just want to 'commit this to paper' before my "cache gets flushed" | |
| 18:11:12 | kashyap | As I've spent the last 2 hours in this rabbit hole :P | |
| 18:11:33 | mriedem | efried: on that service user / CONF.glance.api_servers bug https://bugs.launchpad.net/nova/+bug/1747511 - it's definitely related to using a service user; i removed that code and now the nova-next job passes | |
| 18:11:35 | openstack | Launchpad bug 1747511 in OpenStack Compute (nova) "legacy-tempest-dsvm-neutron-nova-next-full fails with "Can not find requested image" since 1/31" [Undecided,New] | |
| 18:12:04 | stvnoyes | mriedem: i didn't make any changes to the CI devstack (although I can run locally ok). I believe the CI is against nova master, but will check. The zfssa CI test is running for this review - https://review.openstack.org/#/c/541368/ | |
| 18:12:54 | mriedem | stvnoyes: so this job result right? https://openstackci.oracle.com/zfssa_ci/refs-changes-68-541368-1-iscsi/ | |
| 18:13:13 | stvnoyes | mriedem: yes. I'm seeing tempest.api.compute.volumes.test_attach_volume.AttachVolumeMultiAttachTest) ... SKIPPED: Volume multi-attach is not available. | |
| 18:13:43 | mriedem | 2018-02-06 09:16:16.276 12703 DEBUG tempest [-] compute-feature-enabled.volume_multiattach = False log_opt_values /usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py:2898 | |
| 18:13:43 | mriedem | stvnoyes: that's b/c it's not set in tempest.conf | |
| 18:13:47 | mriedem | https://openstackci.oracle.com/zfssa_ci/refs-changes-68-541368-1-iscsi/tempest.log | |
| 18:13:56 | openstackgerrit | Dan Smith proposed openstack/nova master: Make host_manager use scatter-gather and ignore down cells https://review.openstack.org/539617 | |
| 18:13:56 | openstackgerrit | Dan Smith proposed openstack/nova master: Make service all-cells min version helper use scatter-gather https://review.openstack.org/539590 | |
| 18:13:57 | openstackgerrit | Patricia Domingues proposed openstack/nova master: load up the volume drivers by checking architecture https://review.openstack.org/541393 | |
| 18:14:07 | mriedem | stvnoyes: where is the code that controls the local.conf for the devstack setup in this CI? | |
| 18:14:13 | mriedem | is it internal to oracle? or on github? | |
| 18:15:40 | stvnoyes | mriedem: good question. idk. I haven't played with CI stuff so all this is pretty new to me. I will see if there's someone around on the zfssa team who would know. | |
| 18:16:25 | mriedem | stvnoyes: https://wiki.openstack.org/wiki/ThirdPartySystems/Oracle_ZFSSA_CI | |
| 18:16:36 | odyssey4me | howdy all - we're having some trouble in the OSA deployment test for upgrades from newton (without cells v2) to ocata (with cells v2), it looks like there's some weird state which has been hit where the compute registration isn't working right any more. I'm seeing this: https://pastebin.com/C9ji6vdS | |
| 18:17:04 | odyssey4me | Our last working test was the last patch we did which was in early December. | |
| 18:17:25 | odyssey4me | Any idea what may have changed to cause the previously working implementation to no longer work? | |
| 18:19:22 | mriedem | odyssey4me: i don't see anything related to that in https://github.com/openstack/nova/commits/stable/ocata going back to november | |
| 18:19:29 | mriedem | are you sure it wasn't something that changed on the OSA side? | |
| 18:19:47 | odyssey4me | mriedem yep, very sure - see https://review.openstack.org/#/q/branch:stable/ocata+project:openstack/openstack-ansible-os_nova | |
| 18:21:27 | mriedem | https://github.com/openstack/nova/commits/stable/ocata/nova/cmd/manage.py | |
| 18:21:37 | mriedem | oldest thing in stable there is from october | |
| 18:22:14 | odyssey4me | yeah, I'm wondering if something has perhaps changed in the database tables either in newton (the source), or in ocata (the destination) | |
| 18:22:40 | mriedem | have you verified if the instances you're verifying actually exist? | |
| 18:22:49 | mriedem | i.e. not just not mapped, but don't actually exist | |
| 18:23:04 | odyssey4me | well, I got stuck there - given that nova-manage doesn't appear to have a way to list the instances :/ | |
| 18:23:15 | odyssey4me | any tips for getting a listing out? | |
| 18:23:35 | odyssey4me | the closest I could find is "nova-manage cell_v2 discover_hosts --verbose" | |
| 18:24:49 | mriedem | that doesn't dump the instances | |
| 18:25:18 | mriedem | the api isn't up? | |
| 18:25:27 | odyssey4me | it gives me a set of uuid's which look rather like they belong to instances: https://pastebin.com/mSwpZybQ | |
| 18:25:43 | mriedem | Getting compute nodes from cell 'cell1': 16443e92-e33b-487e-8069-7c80e5bdbc33 | |
| 18:25:46 | mriedem | that's a cell mapping uuid | |
| 18:25:54 | mriedem | Checking host mapping for compute host 'ocata-nova1': 6e6d71ab-0b18-416d-8a3a-ce713ac59637 | |
| 18:25:56 | mriedem | that's a compute node uuid | |
| 18:26:08 | mriedem | instances aren't the only things that have uuids anymore | |
| 18:26:40 | odyssey4me | the API is up, so I can query things that way if it helps | |
| 18:27:18 | mriedem | worth a shot if you have a local recreate, | |
| 18:27:27 | mriedem | but my guess is the api won't find them either if nova-manage can't | |
| 18:27:29 | odyssey4me | my apologies for dumb questions - it's been a very long time since I actually worked with nova directly :) | |
| 18:27:37 | mriedem | if they don't have instance_mappings in nova_api i mean | |
| 18:27:42 | mriedem | totally fine | |
| 18:27:46 | mriedem | not dumb at all | |
| 18:28:07 | mriedem | if you have a local recreate, you could check the nova_api.instance_mappings table directly | |
| 18:28:17 | mriedem | or the cell db instances table | |
| 18:28:20 | odyssey4me | sure, can do that | |
| 18:28:37 | mriedem | looks like you have 2 cells, so you'd have to look in each | |
| 18:29:13 | mriedem | so you've got 4 dbs (nova_api, nova_cell0, cell1 and then cell 9461149a-52a9-495d-8021-d2cda1645d28) | |
| 18:29:21 | odyssey4me | right, so I have several DB's here: nova, nova_api, nova_cell0; nova_placement | |
| 18:29:35 | mriedem | nova is likely cell1 | |
| 18:29:36 | mriedem | yes? | |
| 18:29:45 | mriedem | nova_placement isn't a thing...not sure what that is | |
| 18:31:25 | odyssey4me | ok, that's empty - I'll look into why that's there later | |
| 18:31:35 | mriedem | i probably know why | |
| 18:32:09 | odyssey4me | it might be some leftovers from previous work before things matured | |
| 18:33:24 | mriedem | odyssey4me: yeah https://review.openstack.org/#/q/I31293ac4689630e4113588ab2c6373cf572b8f38 | |
| 18:34:08 | odyssey4me | haha, ok - thanks for the reference :) | |
| 18:34:18 | mriedem | odyssey4me: so looking at https://pastebin.com/mSwpZybQ there is something screwed up with the host mappings, | |
| 18:34:45 | mriedem | it looks like this compute is in two cells | |
| 18:34:46 | mriedem | Checking host mapping for compute host 'ocata-nova1': 6e6d71ab-0b18-416d-8a3a-ce713ac59637 | |
| 18:35:02 | odyssey4me | if it'd make things simpler I can get your pub key on this host for you to poke around directly? | |
| 18:35:36 | mriedem | i don't think our relationship has hit that level of maturity yet | |
| 18:35:36 | odyssey4me | the host is a temp instance, so nothing special on it | |
| 18:35:44 | odyssey4me | hahaha, fair enough | |
| 18:36:37 | odyssey4me | otherwise, I'll need some guidance with db queries to get data out - I can gist the results as we go | |
| 18:36:54 | mriedem | well, it appears you have 2 nova_api.host_mappings entries for host "ocata-nova1" | |
| 18:36:56 | mriedem | which would be wrong | |
| 18:37:09 | odyssey4me | yeah, that seemed weird to me too | |
| 18:37:36 | cfriesen | is anyone aware of an issue where running "'wget http://169.254.169.254/latest/meta-data/instance-id" in the guest gives a result that is *not* the same as that instance's OS-EXT-SRV-ATTR:instance_name in "nova show"? | |
| 18:37:53 | mriedem | tssurya: dansmith: melwitt: any idea why we don't have a unique constraint across the cell_id and host colums in the host_mappings table? | |
| 18:38:33 | dansmith | mriedem: I thought there was some argument about that with overlapping hostnames (which won't work anyway) ? | |
| 18:38:36 | mriedem | odyssey4me: is this CI supposed to have 2 cells? because multi-cell wasn't supported in ocata | |
| 18:38:40 | odyssey4me | mriedem interestingly enough, I only see one entry in the DB for it | |
| 18:39:22 | odyssey4me | nah, it only has cell0 and cell1 | |