| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-02-06 | |||
| 17:55:39 | dansmith | https://review.openstack.org/#/c/539590/ | |
| 17:55:43 | mriedem | oh that's different | |
| 17:55:56 | mriedem | it's just tempest with devstack on py35 | |
| 17:56:15 | dansmith | tbachman: no | |
| 17:56:19 | dansmith | mriedem: hmm, wonder why it failed | |
| 17:56:27 | tbachman | dansmith: thx | |
| 17:56:29 | mriedem | dansmith: http://logs.openstack.org/90/539590/2/check/legacy-tempest-dsvm-py35/79137ee/job-output.txt.gz#_2018-01-31_19_33_29_779861 | |
| 17:56:40 | mriedem | likely something in the reserve_block_device_name rpc call? | |
| 17:56:49 | dansmith | mriedem: right, hence my "seemingly related reason" | |
| 17:57:00 | dansmith | no, it's because we check for service version before allowing tags | |
| 17:57:06 | dansmith | which is what that patch is doing | |
| 17:57:17 | dansmith | s/doing/changing/ | |
| 17:57:21 | dansmith | s/changing/touching/ | |
| 17:57:43 | bauzas | mriedem: mgagne: just added a thought on a possible workaround for https://review.openstack.org/#/c/541018/ | |
| 17:58:04 | bauzas | mriedem: mgagne: tl;dr just set allocation ratios on every compute to be 99999.0 | |
| 17:58:17 | sean-k-mooney | cfriesen: stephenfin yes numa support is off by default in dpdk so that libnuma can remain an optional dependecy not a required depency of dpdk | |
| 17:58:26 | mriedem | dansmith: maybe "if result is" is different in py35 | |
| 17:58:33 | mriedem | i don't see much "foo is bar" | |
| 17:58:50 | bauzas | mriedem: mgagne: that will allow Placement to accept all the hosts, and just keep the filtering logic in the appropriate AggregateRAMFilter and so on | |
| 18:00:25 | dansmith | mriedem: um, which thing are you talking about? | |
| 18:00:26 | kashyap | I'm Neutron-challenged, can anyone tell at _which_ of these points is Nova waiting (by pausing) on Neutron to let it complete things? http://git.openstack.org/cgit/openstack/neutron/commit/?h=master&id=b7c303e ("Notify nova with network-vif-plugged in case of live migration") | |
| 18:00:29 | dansmith | the sentinel check has to be "is" | |
| 18:01:21 | kashyap | "during live migration on pre migration" is terribly unclear | |
| 18:02:52 | dansmith | kashyap: nova waits when it uses the virtapi wait_for_events context manager | |
| 18:02:53 | dansmith | it waits to exit the context until timeout or the event arrives | |
| 18:03:22 | kashyap | I see. | |
| 18:03:44 | kashyap | Is this what the Neutron events telling Nova: | |
| 18:04:00 | kashyap | "Hey, I've done with doing all the stuff I need to do, so I'm signalling that by sending out this event (from the above Neutron commit). Over to you now." | |
| 18:04:22 | kashyap | I.e. the destination Neutron has completed virtual interface plugging. | |
| 18:04:40 | dansmith | yes | |
| 18:04:54 | dansmith | I mean, that's what vif-plugged means | |
| 18:05:24 | kashyap | Noted. | |
| 18:05:31 | stvnoyes | mriedem: matt, I added a change to enable multiattach on the zfssa array, but when tempest runs during the zfssa ci, it's not running with multiattach enabled (in tempest.conf). Where would I make a change to have this enabled when running the Oracle ZFSSA CI? thanks... | |
| 18:05:43 | sean-k-mooney | dansmith: actully vif-plugged has different menanings | |
| 18:06:13 | sean-k-mooney | vif-plugged in the nova-compute agent log means os-vif finisshed plugging the interface | |
| 18:06:33 | sean-k-mooney | vif-plugged event form neutron means neutron finished wireing it up | |
| 18:06:54 | dansmith | sean-k-mooney: we're talking about when neutron does a POST of the event to nova's external event interface | |
| 18:07:27 | dansmith | and it better (still) mean that the wiring is done, else we have a problem :) | |
| 18:07:32 | mriedem | stvnoyes: do you have these set in devstack? https://github.com/openstack/nova/blob/master/playbooks/legacy/nova-multiattach/run.yaml#L31 | |
| 18:07:46 | sean-k-mooney | dansmith: ah then yes that is sent when neutron finishes wiring up the port that was plugged by nova/os-vif/libvirt into the network backend so your correct in what you said | |
| 18:08:17 | mriedem | stvnoyes: also, is tempest running against master nova so the 2.60 microversion is available? | |
| 18:08:37 | dansmith | sean-k-mooney: thanks :) | |
| 18:09:16 | kashyap | sean-k-mooney: Hey, I'm about to write some questions on this change: https://review.openstack.org/#/c/497457/18 | |
| 18:09:26 | kashyap | sean-k-mooney: Post some design discussion w/ QEMU & libvirt folks | |
| 18:09:39 | kashyap | Would appreciate if you also look at it | |
| 18:10:24 | sean-k-mooney | kashyap: am i am behind in some internal paper work but ill try and take a look tomorow | |
| 18:10:37 | kashyap | sean-k-mooney: Sure, tomm is fine; I'm already late for dinner by an hour | |
| 18:10:55 | kashyap | Just want to 'commit this to paper' before my "cache gets flushed" | |
| 18:11:12 | kashyap | As I've spent the last 2 hours in this rabbit hole :P | |
| 18:11:33 | mriedem | efried: on that service user / CONF.glance.api_servers bug https://bugs.launchpad.net/nova/+bug/1747511 - it's definitely related to using a service user; i removed that code and now the nova-next job passes | |
| 18:11:35 | openstack | Launchpad bug 1747511 in OpenStack Compute (nova) "legacy-tempest-dsvm-neutron-nova-next-full fails with "Can not find requested image" since 1/31" [Undecided,New] | |
| 18:12:04 | stvnoyes | mriedem: i didn't make any changes to the CI devstack (although I can run locally ok). I believe the CI is against nova master, but will check. The zfssa CI test is running for this review - https://review.openstack.org/#/c/541368/ | |
| 18:12:54 | mriedem | stvnoyes: so this job result right? https://openstackci.oracle.com/zfssa_ci/refs-changes-68-541368-1-iscsi/ | |
| 18:13:13 | stvnoyes | mriedem: yes. I'm seeing tempest.api.compute.volumes.test_attach_volume.AttachVolumeMultiAttachTest) ... SKIPPED: Volume multi-attach is not available. | |
| 18:13:43 | mriedem | 2018-02-06 09:16:16.276 12703 DEBUG tempest [-] compute-feature-enabled.volume_multiattach = False log_opt_values /usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py:2898 | |
| 18:13:43 | mriedem | stvnoyes: that's b/c it's not set in tempest.conf | |
| 18:13:47 | mriedem | https://openstackci.oracle.com/zfssa_ci/refs-changes-68-541368-1-iscsi/tempest.log | |
| 18:13:56 | openstackgerrit | Dan Smith proposed openstack/nova master: Make host_manager use scatter-gather and ignore down cells https://review.openstack.org/539617 | |
| 18:13:56 | openstackgerrit | Dan Smith proposed openstack/nova master: Make service all-cells min version helper use scatter-gather https://review.openstack.org/539590 | |
| 18:13:57 | openstackgerrit | Patricia Domingues proposed openstack/nova master: load up the volume drivers by checking architecture https://review.openstack.org/541393 | |
| 18:14:07 | mriedem | stvnoyes: where is the code that controls the local.conf for the devstack setup in this CI? | |
| 18:14:13 | mriedem | is it internal to oracle? or on github? | |
| 18:15:40 | stvnoyes | mriedem: good question. idk. I haven't played with CI stuff so all this is pretty new to me. I will see if there's someone around on the zfssa team who would know. | |
| 18:16:25 | mriedem | stvnoyes: https://wiki.openstack.org/wiki/ThirdPartySystems/Oracle_ZFSSA_CI | |
| 18:16:36 | odyssey4me | howdy all - we're having some trouble in the OSA deployment test for upgrades from newton (without cells v2) to ocata (with cells v2), it looks like there's some weird state which has been hit where the compute registration isn't working right any more. I'm seeing this: https://pastebin.com/C9ji6vdS | |
| 18:17:04 | odyssey4me | Our last working test was the last patch we did which was in early December. | |
| 18:17:25 | odyssey4me | Any idea what may have changed to cause the previously working implementation to no longer work? | |
| 18:19:22 | mriedem | odyssey4me: i don't see anything related to that in https://github.com/openstack/nova/commits/stable/ocata going back to november | |
| 18:19:29 | mriedem | are you sure it wasn't something that changed on the OSA side? | |
| 18:19:47 | odyssey4me | mriedem yep, very sure - see https://review.openstack.org/#/q/branch:stable/ocata+project:openstack/openstack-ansible-os_nova | |
| 18:21:27 | mriedem | https://github.com/openstack/nova/commits/stable/ocata/nova/cmd/manage.py | |
| 18:21:37 | mriedem | oldest thing in stable there is from october | |
| 18:22:14 | odyssey4me | yeah, I'm wondering if something has perhaps changed in the database tables either in newton (the source), or in ocata (the destination) | |
| 18:22:40 | mriedem | have you verified if the instances you're verifying actually exist? | |
| 18:22:49 | mriedem | i.e. not just not mapped, but don't actually exist | |
| 18:23:04 | odyssey4me | well, I got stuck there - given that nova-manage doesn't appear to have a way to list the instances :/ | |
| 18:23:15 | odyssey4me | any tips for getting a listing out? | |
| 18:23:35 | odyssey4me | the closest I could find is "nova-manage cell_v2 discover_hosts --verbose" | |
| 18:24:49 | mriedem | that doesn't dump the instances | |
| 18:25:18 | mriedem | the api isn't up? | |
| 18:25:27 | odyssey4me | it gives me a set of uuid's which look rather like they belong to instances: https://pastebin.com/mSwpZybQ | |
| 18:25:43 | mriedem | Getting compute nodes from cell 'cell1': 16443e92-e33b-487e-8069-7c80e5bdbc33 | |
| 18:25:46 | mriedem | that's a cell mapping uuid | |
| 18:25:54 | mriedem | Checking host mapping for compute host 'ocata-nova1': 6e6d71ab-0b18-416d-8a3a-ce713ac59637 | |
| 18:25:56 | mriedem | that's a compute node uuid | |
| 18:26:08 | mriedem | instances aren't the only things that have uuids anymore | |
| 18:26:40 | odyssey4me | the API is up, so I can query things that way if it helps | |
| 18:27:18 | mriedem | worth a shot if you have a local recreate, | |
| 18:27:27 | mriedem | but my guess is the api won't find them either if nova-manage can't | |
| 18:27:29 | odyssey4me | my apologies for dumb questions - it's been a very long time since I actually worked with nova directly :) | |
| 18:27:37 | mriedem | if they don't have instance_mappings in nova_api i mean | |
| 18:27:42 | mriedem | totally fine | |
| 18:27:46 | mriedem | not dumb at all | |
| 18:28:07 | mriedem | if you have a local recreate, you could check the nova_api.instance_mappings table directly | |
| 18:28:17 | mriedem | or the cell db instances table | |
| 18:28:20 | odyssey4me | sure, can do that | |
| 18:28:37 | mriedem | looks like you have 2 cells, so you'd have to look in each | |
| 18:29:13 | mriedem | so you've got 4 dbs (nova_api, nova_cell0, cell1 and then cell 9461149a-52a9-495d-8021-d2cda1645d28) | |
| 18:29:21 | odyssey4me | right, so I have several DB's here: nova, nova_api, nova_cell0; nova_placement | |
| 18:29:35 | mriedem | nova is likely cell1 | |