Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-10-10
19:46:35 mriedem PUT https://10.210.129.28/volume/v3/2732673e8c4a46249b654cee4b963920/attachments/0797bbdd-ecd6-488b-bc85-b23b5de49643
19:46:42 mriedem -d '{"attachment": {"connector": {"platform": "x86_64", "host": "ubuntu-xenial-2-node-rax-ord-11308327-949541", "do_local_attach": false, "ip": "10.210.195.60", "os_type": "linux2", "multipath": false, "initiator": "iqn.1993-08.org.debian:01:cc478c1ad9"}}}'
19:46:48 mriedem it's updating the attachment with the host connector
19:47:24 sdague efried: nice
19:48:43 ildikov yeah, I saw that too on the Cinder side
19:48:51 ildikov I mean the new flow call
19:49:35 mriedem http://logs.openstack.org/85/330285/144/check/gate-tempest-dsvm-neutron-multinode-full-ubuntu-xenial-nv/89229f6/logs/subnode-2/screen-n-cpu.txt.gz#_Oct_10_13_01_49_423050
19:49:35 mriedem this is where the attachment is created during volume attach
19:51:51 mriedem http://logs.openstack.org/85/330285/144/check/gate-tempest-dsvm-neutron-multinode-full-ubuntu-xenial-nv/89229f6/logs/subnode-2/screen-n-cpu.txt.gz#_Oct_10_13_01_44_516548
19:51:51 mriedem and this is where we update the attachment with the host connector once it's connected during the initial attach
19:52:56 mriedem wth, those times don't line up
19:53:15 mriedem oh http://logs.openstack.org/85/330285/144/check/gate-tempest-dsvm-neutron-multinode-full-ubuntu-xenial-nv/89229f6/logs/subnode-2/screen-n-cpu.txt.gz#_Oct_10_13_01_49_423050 is the attachment_complete call
19:55:04 ildikov that update must be from refresh
19:55:07 ildikov what else?
19:55:20 mriedem that's what i'm thinking too, but don't see where
20:00:12 ildikov that refresh seems to be called in _get_instance_block_device_info in compute/manager.py which gets called in a zillion places
20:00:41 mriedem that's only if refresh_conn_info is True
20:00:43 mriedem it defaults to False
20:02:09 ildikov ah true
20:05:23 ildikov and it uses the default for reboot as it's supposed to
20:06:32 ildikov can anything in libvirt call that?
20:06:47 mriedem there are a couple of places but they are during snapshot
20:08:28 openstackgerrit Merged openstack/nova master: use already loaded BDM in instance.create https://review.openstack.org/483969
20:08:30 ildikov hmm, weird, as there's nothing around that update call, but brick doing its things...
20:09:53 openstackgerrit Merged openstack/nova master: Fix sending legacy instance.update notification https://review.openstack.org/510603
20:11:06 jgriffith_ mriedem ildikov http://logs.openstack.org/85/330285/144/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/7a079b9/logs/screen-c-api.txt.gz?#_Oct_10_12_53_01_672274
20:12:04 ildikov jgriffith_: I think that's 'begin_detaching'
20:12:11 jgriffith_ yes it is
20:12:18 ildikov jgriffith_: which we left untouched
20:12:28 jgriffith_ "we" ?
20:12:35 ildikov so that call is normal
20:12:53 ildikov jgriffith_: I kept you in the loop with that :)
20:13:37 ildikov jgriffith_: I mean to keep that in the flow as we don't want parallel detaches happening as we don't have anything to change the volume or whatever state to 'detaching'
20:15:08 openstackgerrit Eric Fried proposed openstack/nova master: DNM: CI check https://review.openstack.org/511006
20:15:54 mriedem i'm not really following what begin_detaching has to do with this
20:16:03 mriedem that's the symptom
20:16:06 openstackgerrit Jackie Truong proposed openstack/nova master: Implement certificate_utils https://review.openstack.org/479949
20:16:10 mriedem looking at the tempest logs, the volume is in-use here http://logs.openstack.org/85/330285/144/check/gate-tempest-dsvm-neutron-multinode-full-ubuntu-xenial-nv/89229f6/console.html#_2017-10-10_13_20_50_909920
20:16:34 mriedem then we reboot the server, and i see the phantom attachment_update call: http://logs.openstack.org/85/330285/144/check/gate-tempest-dsvm-neutron-multinode-full-ubuntu-xenial-nv/89229f6/logs/subnode-2/screen-n-cpu.txt.gz#_Oct_10_13_02_05_685303
20:17:03 ildikov jgriffith_: I know, we are not supposed to keep old flow stuff neither copy it to the new one, it seemed a reasonable idea to keep this for now back at the time
20:17:11 mriedem and the failed detach here http://logs.openstack.org/85/330285/144/check/gate-tempest-dsvm-neutron-multinode-full-ubuntu-xenial-nv/89229f6/console.html#_2017-10-10_13_20_50_908565
20:17:38 mriedem and http://logs.openstack.org/85/330285/144/check/gate-tempest-dsvm-neutron-multinode-full-ubuntu-xenial-nv/89229f6/console.html#_2017-10-10_13_20_50_909920 shows the volume went from in-use to attaching
20:18:33 ildikov mriedem: yeah, that last bit is the consequence of update
20:19:18 mriedem ildikov: if the new refresh_connection_info code thinks that calling attachment_update means we're attaching, then that's a bug
20:19:24 mriedem i thought we talked about this at the ptg
20:20:56 ildikov mriedem: about refresh?
20:21:05 mriedem yeah i'm looking up the etherpad
20:21:54 jgriffith_ ahh, I see; never did the attachment-complete on the reboot
20:22:14 mriedem there shouldn't be an attachment_update on the reboot
20:22:15 jgriffith_ mriedem and yes, you're correct
20:22:17 mriedem i don't know where it's coming from
20:24:10 ildikov mriedem: regardless of how refresh is behaving, there still doesn't seem to be a logical path to that attachment_update call :/
20:25:58 mriedem L269 https://etherpad.openstack.org/p/cinder-ptg-queens
20:26:02 mriedem apparently my notes were removed...
20:26:21 openstackgerrit OpenStack Proposal Bot proposed openstack/nova master: Updated from global requirements https://review.openstack.org/511021
20:28:00 ildikov mriedem: https://etherpad.openstack.org/p/cinder-ptg-queens-thursday-notes
20:28:09 ildikov L121
20:28:41 mriedem whew
20:28:43 mriedem thanks
20:29:04 mriedem "With the new style Cinder API, does nova.virt.block_device.DriverVolumeBlockDevice.refresh_connection_info change? Do we do an attachment_show or do we delete the old attachment and create a new one (or does that only happen on a HARD reboot - if we make that change)?"
20:29:36 mriedem "if bdm.attachment_id then we should fetch connection info from Cinder (on hard reboot) What I heard is this actually needs to be an attachment_delete/create swap thing because that's what gives us the connection_info from the storage backend. Although I wonder if attachment_update would refresh the connection_info on the cinder side...since attachment_update == os-initialize_connection."
20:33:53 ildikov not sure that's fully equal
20:34:23 ildikov I'm always open to be corrected by jgriffith_ :)
20:36:17 ildikov but with reboot we call the _get_instance_block_device_info with default on refresh, which is False
20:37:53 mriedem ildikov: what i'm going to do is just push a debug patch on top of john's patch which will dump a stacktrace if we call attachment_update while rebooting an instance
20:38:12 ildikov mriedem: sounds good, thank you
20:38:13 mriedem i've spent way too much time looking at logs today
20:38:23 ildikov mriedem: sorry :(
20:38:32 mriedem nah it's fine,
20:38:38 mriedem i just need to let the computer do some work
20:38:51 ildikov mriedem: I swear I just rebased the thing and bumped the service version :S
20:39:14 ildikov mriedem: +1 on the last comment :)
20:42:27 openstackgerrit OpenStack Proposal Bot proposed openstack/os-vif stable/pike: Updated from global requirements https://review.openstack.org/493146
20:48:35 openstackgerrit Brianna Poulos proposed openstack/nova master: Add trusted_image_certificates to REST API https://review.openstack.org/486204
20:50:01 openstackgerrit Matt Riedemann proposed openstack/nova master: DNM: Fail if we try to update volume connections during reboot https://review.openstack.org/511031
20:54:51 openstackgerrit OpenStack Proposal Bot proposed openstack/nova master: Updated from global requirements https://review.openstack.org/511021
20:56:56 openstackgerrit OpenStack Proposal Bot proposed openstack/os-vif master: Updated from global requirements https://review.openstack.org/511035
21:08:42 cdent edleafe: in the commit message on https://review.openstack.org/#/c/510625/4 there are some questions about possible todos, did you have an opinion on those?
21:09:34 edleafe cdent: you mean about including the project_id/user_id in the GET?
21:09:44 cdent yeah
21:09:58 edleafe I don't see why we shouldn't
21:10:38 openstackgerrit OpenStack Proposal Bot proposed openstack/nova master: Updated from global requirements https://review.openstack.org/511021
21:11:10 coreycb hi all, cells v2 question
21:11:38 cdent edleafe: does that mean I should put a -1 or -W or a something on there?
21:12:40 coreycb i'm hitting a window where after 'nova-manage cells_v2 list_cells' starts returning the cell i'm interested in, mapping hosts or instances for that cell fails during the window until the database is finished. is that expected?
21:12:44 openstackgerrit OpenStack Proposal Bot proposed openstack/os-vif master: Updated from global requirements https://review.openstack.org/511035
21:13:58 mriedem coreycb: until the database is finished with what?
21:16:29 openstackgerrit Chris Dent proposed openstack/nova master: Move project_id and user_id to Allocation object https://review.openstack.org/500410
21:16:30 openstackgerrit Chris Dent proposed openstack/nova master: [placement] Allow _set_allocations to delete allocations https://review.openstack.org/501051
21:16:31 openstackgerrit Chris Dent proposed openstack/nova master: [placement] Limit number of attempts to delete allocations https://review.openstack.org/507224
21:16:32 openstackgerrit Chris Dent proposed openstack/nova master: [placement] Enable PUT dict to /allocations/{consumer_uuid} https://review.openstack.org/510625
21:16:33 openstackgerrit Chris Dent proposed openstack/nova master: [placement] Document the changes to PUT /allocations/{consumer_uuid} https://review.openstack.org/510626
21:16:34 openstackgerrit Chris Dent proposed openstack/nova master: [placement] POST /allocations to set allocations for >1 consumers https://review.openstack.org/500073
21:17:37 edleafe cdent: You mean so they can be in the same microversion? Meh, they're cheap. You can do it in a follow-up patch if you like
21:18:24 coreycb mriedem: it seems to still be initializing the DB from prior nova-manage commands. here's some output of 'list_cells' looping and then running 'discover_hosts' once the cell starts showing up in list_cells output: https://launchpadlibrarian.net/339984742/printscreen
21:18:44 cdent edleafe: no, not worried about the microversion issue, just that somewhere it needs to be marked as we’re not done here. If it is to be done, I’d be inclined to do it in a prior patch.
21:18:53 cdent i’ll leave a comment-to-self
21:19:09 coreycb mriedem: search for 'cell1'
21:20:27 mriedem coreycb: Table 'nova.compute_nodes' doesn't exist
21:20:40 mriedem you can't run discover_hosts or map_instances unti you've synced the dbs on the cells

Earlier   Later