Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-10-10
19:38:32 mriedem refreshing the volume connection info would do it, but i don't see that called during reboot
19:39:13 mriedem this is a soft reboot too, so we're not rebuilding the guest at all
19:39:50 ildikov update is called only at three places
19:40:32 mriedem swap volume, volume attach, and refresh_connection_info
19:40:42 ildikov _init_volume_connection, _volume_attach, and _get_connection_updates which is called in refresh_connection_info
19:41:06 mriedem yeah i'm failing to see where reboot calls any of those, directly or otherwise
19:41:17 mriedem but the request id for the reboot operation is showing it happening
19:42:07 efried mriedem sdague FYI glance will only let you skip *all* conf after https://review.openstack.org/#/c/490057/ is in place. Which is to be proven by https://review.openstack.org/#/c/490031/ passing jenkins.
19:42:16 ildikov I couldn't get there to check that scenario test, I didn't think there's a reboot in it... :/
19:42:44 mriedem ildikov: yeah TestMinimumBasicScenario.test_minimum_basic_scenario attaches a volume and then soft reboots the guest
19:42:52 mriedem then ssh's into the guest to make sure the block device is still attached
19:43:20 mriedem at the end when the test is cleaning up it goes to detach the volume which fails because the volume status is 'attaching' rather than 'in-use'
19:43:30 mriedem and i think that's happening because of a wild attachment_update call during the reboot
19:43:36 mriedem which screws up the volume status
19:44:48 ildikov jgriffith_ said that it's calling the old flow
19:44:54 ildikov which seemed odd
19:45:32 ildikov and yeah, if we don;t have attachment_complete following attachment_update then the volume status remains "attaching"
19:46:06 mriedem it can't be the old flow because otherwise we wouldn't call attachment_update
19:46:22 mriedem which i see happen right here during reboot http://logs.openstack.org/85/330285/144/check/gate-tempest-dsvm-neutron-multinode-full-ubuntu-xenial-nv/89229f6/logs/subnode-2/screen-n-cpu.txt.gz#_Oct_10_13_02_05_685303
19:46:35 mriedem PUT https://10.210.129.28/volume/v3/2732673e8c4a46249b654cee4b963920/attachments/0797bbdd-ecd6-488b-bc85-b23b5de49643
19:46:42 mriedem -d '{"attachment": {"connector": {"platform": "x86_64", "host": "ubuntu-xenial-2-node-rax-ord-11308327-949541", "do_local_attach": false, "ip": "10.210.195.60", "os_type": "linux2", "multipath": false, "initiator": "iqn.1993-08.org.debian:01:cc478c1ad9"}}}'
19:46:48 mriedem it's updating the attachment with the host connector
19:47:24 sdague efried: nice
19:48:43 ildikov yeah, I saw that too on the Cinder side
19:48:51 ildikov I mean the new flow call
19:49:35 mriedem http://logs.openstack.org/85/330285/144/check/gate-tempest-dsvm-neutron-multinode-full-ubuntu-xenial-nv/89229f6/logs/subnode-2/screen-n-cpu.txt.gz#_Oct_10_13_01_49_423050
19:49:35 mriedem this is where the attachment is created during volume attach
19:51:51 mriedem http://logs.openstack.org/85/330285/144/check/gate-tempest-dsvm-neutron-multinode-full-ubuntu-xenial-nv/89229f6/logs/subnode-2/screen-n-cpu.txt.gz#_Oct_10_13_01_44_516548
19:51:51 mriedem and this is where we update the attachment with the host connector once it's connected during the initial attach
19:52:56 mriedem wth, those times don't line up
19:53:15 mriedem oh http://logs.openstack.org/85/330285/144/check/gate-tempest-dsvm-neutron-multinode-full-ubuntu-xenial-nv/89229f6/logs/subnode-2/screen-n-cpu.txt.gz#_Oct_10_13_01_49_423050 is the attachment_complete call
19:55:04 ildikov that update must be from refresh
19:55:07 ildikov what else?
19:55:20 mriedem that's what i'm thinking too, but don't see where
20:00:12 ildikov that refresh seems to be called in _get_instance_block_device_info in compute/manager.py which gets called in a zillion places
20:00:41 mriedem that's only if refresh_conn_info is True
20:00:43 mriedem it defaults to False
20:02:09 ildikov ah true
20:05:23 ildikov and it uses the default for reboot as it's supposed to
20:06:32 ildikov can anything in libvirt call that?
20:06:47 mriedem there are a couple of places but they are during snapshot
20:08:28 openstackgerrit Merged openstack/nova master: use already loaded BDM in instance.create https://review.openstack.org/483969
20:08:30 ildikov hmm, weird, as there's nothing around that update call, but brick doing its things...
20:09:53 openstackgerrit Merged openstack/nova master: Fix sending legacy instance.update notification https://review.openstack.org/510603
20:11:06 jgriffith_ mriedem ildikov http://logs.openstack.org/85/330285/144/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/7a079b9/logs/screen-c-api.txt.gz?#_Oct_10_12_53_01_672274
20:12:04 ildikov jgriffith_: I think that's 'begin_detaching'
20:12:11 jgriffith_ yes it is
20:12:18 ildikov jgriffith_: which we left untouched
20:12:28 jgriffith_ "we" ?
20:12:35 ildikov so that call is normal
20:12:53 ildikov jgriffith_: I kept you in the loop with that :)
20:13:37 ildikov jgriffith_: I mean to keep that in the flow as we don't want parallel detaches happening as we don't have anything to change the volume or whatever state to 'detaching'
20:15:08 openstackgerrit Eric Fried proposed openstack/nova master: DNM: CI check https://review.openstack.org/511006
20:15:54 mriedem i'm not really following what begin_detaching has to do with this
20:16:03 mriedem that's the symptom
20:16:06 openstackgerrit Jackie Truong proposed openstack/nova master: Implement certificate_utils https://review.openstack.org/479949
20:16:10 mriedem looking at the tempest logs, the volume is in-use here http://logs.openstack.org/85/330285/144/check/gate-tempest-dsvm-neutron-multinode-full-ubuntu-xenial-nv/89229f6/console.html#_2017-10-10_13_20_50_909920
20:16:34 mriedem then we reboot the server, and i see the phantom attachment_update call: http://logs.openstack.org/85/330285/144/check/gate-tempest-dsvm-neutron-multinode-full-ubuntu-xenial-nv/89229f6/logs/subnode-2/screen-n-cpu.txt.gz#_Oct_10_13_02_05_685303
20:17:03 ildikov jgriffith_: I know, we are not supposed to keep old flow stuff neither copy it to the new one, it seemed a reasonable idea to keep this for now back at the time
20:17:11 mriedem and the failed detach here http://logs.openstack.org/85/330285/144/check/gate-tempest-dsvm-neutron-multinode-full-ubuntu-xenial-nv/89229f6/console.html#_2017-10-10_13_20_50_908565
20:17:38 mriedem and http://logs.openstack.org/85/330285/144/check/gate-tempest-dsvm-neutron-multinode-full-ubuntu-xenial-nv/89229f6/console.html#_2017-10-10_13_20_50_909920 shows the volume went from in-use to attaching
20:18:33 ildikov mriedem: yeah, that last bit is the consequence of update
20:19:18 mriedem ildikov: if the new refresh_connection_info code thinks that calling attachment_update means we're attaching, then that's a bug
20:19:24 mriedem i thought we talked about this at the ptg
20:20:56 ildikov mriedem: about refresh?
20:21:05 mriedem yeah i'm looking up the etherpad
20:21:54 jgriffith_ ahh, I see; never did the attachment-complete on the reboot
20:22:14 mriedem there shouldn't be an attachment_update on the reboot
20:22:15 jgriffith_ mriedem and yes, you're correct
20:22:17 mriedem i don't know where it's coming from
20:24:10 ildikov mriedem: regardless of how refresh is behaving, there still doesn't seem to be a logical path to that attachment_update call :/
20:25:58 mriedem L269 https://etherpad.openstack.org/p/cinder-ptg-queens
20:26:02 mriedem apparently my notes were removed...
20:26:21 openstackgerrit OpenStack Proposal Bot proposed openstack/nova master: Updated from global requirements https://review.openstack.org/511021
20:28:00 ildikov mriedem: https://etherpad.openstack.org/p/cinder-ptg-queens-thursday-notes
20:28:09 ildikov L121
20:28:41 mriedem whew
20:28:43 mriedem thanks
20:29:04 mriedem "With the new style Cinder API, does nova.virt.block_device.DriverVolumeBlockDevice.refresh_connection_info change? Do we do an attachment_show or do we delete the old attachment and create a new one (or does that only happen on a HARD reboot - if we make that change)?"
20:29:36 mriedem "if bdm.attachment_id then we should fetch connection info from Cinder (on hard reboot) What I heard is this actually needs to be an attachment_delete/create swap thing because that's what gives us the connection_info from the storage backend. Although I wonder if attachment_update would refresh the connection_info on the cinder side...since attachment_update == os-initialize_connection."
20:33:53 ildikov not sure that's fully equal
20:34:23 ildikov I'm always open to be corrected by jgriffith_ :)
20:36:17 ildikov but with reboot we call the _get_instance_block_device_info with default on refresh, which is False
20:37:53 mriedem ildikov: what i'm going to do is just push a debug patch on top of john's patch which will dump a stacktrace if we call attachment_update while rebooting an instance
20:38:12 ildikov mriedem: sounds good, thank you
20:38:13 mriedem i've spent way too much time looking at logs today
20:38:23 ildikov mriedem: sorry :(
20:38:32 mriedem nah it's fine,
20:38:38 mriedem i just need to let the computer do some work
20:38:51 ildikov mriedem: I swear I just rebased the thing and bumped the service version :S
20:39:14 ildikov mriedem: +1 on the last comment :)
20:42:27 openstackgerrit OpenStack Proposal Bot proposed openstack/os-vif stable/pike: Updated from global requirements https://review.openstack.org/493146
20:48:35 openstackgerrit Brianna Poulos proposed openstack/nova master: Add trusted_image_certificates to REST API https://review.openstack.org/486204
20:50:01 openstackgerrit Matt Riedemann proposed openstack/nova master: DNM: Fail if we try to update volume connections during reboot https://review.openstack.org/511031
20:54:51 openstackgerrit OpenStack Proposal Bot proposed openstack/nova master: Updated from global requirements https://review.openstack.org/511021
20:56:56 openstackgerrit OpenStack Proposal Bot proposed openstack/os-vif master: Updated from global requirements https://review.openstack.org/511035
21:08:42 cdent edleafe: in the commit message on https://review.openstack.org/#/c/510625/4 there are some questions about possible todos, did you have an opinion on those?
21:09:34 edleafe cdent: you mean about including the project_id/user_id in the GET?
21:09:44 cdent yeah
21:09:58 edleafe I don't see why we shouldn't

Earlier   Later