| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-10-10 | |||
| 18:33:16 | sdague | maybe we stopped accepting image url somewhere | |
| 18:33:29 | mriedem | i remember gmann fixing something but can't remember what | |
| 18:36:15 | mriedem | unrelated, for some weird reason, i'm tracing a reboot request through the API, and i see the request ID in the n-api logs but it's not showing up in the n-cpu logs | |
| 18:39:44 | cfriesen | does anyone know why in nova.virt.libvirt.host.Host_init_events_pipe() we have a queue as well as a pipe? Why can't we just block on the queue rather than the pipe? | |
| 18:40:31 | mriedem | talk to mdbooth | |
| 18:41:40 | cfriesen | looks like it was dpb that added it originally | |
| 18:59:21 | openstackgerrit | Elod Illes proposed openstack/nova master: WIP: Transform scheduler.select_destinations notification https://review.openstack.org/508506 | |
| 19:14:57 | openstackgerrit | Elod Illes proposed openstack/nova master: WIP: Transform scheduler.select_destinations notification https://review.openstack.org/508506 | |
| 19:30:38 | mtreinish | mriedem: changing error codes on us | |
| 19:31:18 | mriedem | making bad requests | |
| 19:31:20 | mriedem | tsk tsk | |
| 19:32:00 | mriedem | jgriffith: ildikov: debugging a failed test on the new style attach patch, i can't figure out why but we're doing an attachment_update during a reboot | |
| 19:32:21 | mriedem | http://logs.openstack.org/85/330285/144/check/gate-tempest-dsvm-neutron-multinode-full-ubuntu-xenial-nv/89229f6/logs/subnode-2/screen-n-cpu.txt.gz#_Oct_10_13_02_05_685303 | |
| 19:32:39 | mriedem | i can't tell what's eventually calling that | |
| 19:33:56 | ildikov | mriedem: does that share code path with any other operation? | |
| 19:34:39 | ildikov | I don't recall a reboot scenario right now | |
| 19:34:54 | ildikov | maybe jgriffith_ does | |
| 19:35:26 | ildikov | mriedem: on that attachment_complete, I added the note on the higher level to clarify why that call is there as I kept re-investigating it | |
| 19:35:50 | mriedem | ildikov: yeah i saw the note, but the logic is basically: | |
| 19:36:01 | mriedem | 1. if do_driver_attach - call attachment_complete down in the bowels, else | |
| 19:36:03 | mriedem | 2. do it here | |
| 19:36:15 | mriedem | as noted, we should just always do it in the bowels as we did before | |
| 19:36:20 | ildikov | mriedem: I'm fine with just covering it in bdm.attach if we don't want to deal with the different flows for this | |
| 19:36:44 | mriedem | i definitely don't want to have to know in the compute manager about what's going on in the DriverVolumeBlockDevice.attach based on flags passe din | |
| 19:37:01 | mriedem | i.e. the logic above | |
| 19:37:21 | ildikov | mriedem: I agree, I will change that | |
| 19:38:07 | ildikov | however I don't think that has much to do with the current failures, except if something got introduced that would need that call | |
| 19:38:11 | mriedem | as for what's doing the attachment_update during reboot, it's not clear to me yet | |
| 19:38:32 | mriedem | refreshing the volume connection info would do it, but i don't see that called during reboot | |
| 19:39:13 | mriedem | this is a soft reboot too, so we're not rebuilding the guest at all | |
| 19:39:50 | ildikov | update is called only at three places | |
| 19:40:32 | mriedem | swap volume, volume attach, and refresh_connection_info | |
| 19:40:42 | ildikov | _init_volume_connection, _volume_attach, and _get_connection_updates which is called in refresh_connection_info | |
| 19:41:06 | mriedem | yeah i'm failing to see where reboot calls any of those, directly or otherwise | |
| 19:41:17 | mriedem | but the request id for the reboot operation is showing it happening | |
| 19:42:07 | efried | mriedem sdague FYI glance will only let you skip *all* conf after https://review.openstack.org/#/c/490057/ is in place. Which is to be proven by https://review.openstack.org/#/c/490031/ passing jenkins. | |
| 19:42:16 | ildikov | I couldn't get there to check that scenario test, I didn't think there's a reboot in it... :/ | |
| 19:42:44 | mriedem | ildikov: yeah TestMinimumBasicScenario.test_minimum_basic_scenario attaches a volume and then soft reboots the guest | |
| 19:42:52 | mriedem | then ssh's into the guest to make sure the block device is still attached | |
| 19:43:20 | mriedem | at the end when the test is cleaning up it goes to detach the volume which fails because the volume status is 'attaching' rather than 'in-use' | |
| 19:43:30 | mriedem | and i think that's happening because of a wild attachment_update call during the reboot | |
| 19:43:36 | mriedem | which screws up the volume status | |
| 19:44:48 | ildikov | jgriffith_ said that it's calling the old flow | |
| 19:44:54 | ildikov | which seemed odd | |
| 19:45:32 | ildikov | and yeah, if we don;t have attachment_complete following attachment_update then the volume status remains "attaching" | |
| 19:46:06 | mriedem | it can't be the old flow because otherwise we wouldn't call attachment_update | |
| 19:46:22 | mriedem | which i see happen right here during reboot http://logs.openstack.org/85/330285/144/check/gate-tempest-dsvm-neutron-multinode-full-ubuntu-xenial-nv/89229f6/logs/subnode-2/screen-n-cpu.txt.gz#_Oct_10_13_02_05_685303 | |
| 19:46:35 | mriedem | PUT https://10.210.129.28/volume/v3/2732673e8c4a46249b654cee4b963920/attachments/0797bbdd-ecd6-488b-bc85-b23b5de49643 | |
| 19:46:42 | mriedem | -d '{"attachment": {"connector": {"platform": "x86_64", "host": "ubuntu-xenial-2-node-rax-ord-11308327-949541", "do_local_attach": false, "ip": "10.210.195.60", "os_type": "linux2", "multipath": false, "initiator": "iqn.1993-08.org.debian:01:cc478c1ad9"}}}' | |
| 19:46:48 | mriedem | it's updating the attachment with the host connector | |
| 19:47:24 | sdague | efried: nice | |
| 19:48:43 | ildikov | yeah, I saw that too on the Cinder side | |
| 19:48:51 | ildikov | I mean the new flow call | |
| 19:49:35 | mriedem | this is where the attachment is created during volume attach | |
| 19:49:35 | mriedem | http://logs.openstack.org/85/330285/144/check/gate-tempest-dsvm-neutron-multinode-full-ubuntu-xenial-nv/89229f6/logs/subnode-2/screen-n-cpu.txt.gz#_Oct_10_13_01_49_423050 | |
| 19:51:51 | mriedem | and this is where we update the attachment with the host connector once it's connected during the initial attach | |
| 19:51:51 | mriedem | http://logs.openstack.org/85/330285/144/check/gate-tempest-dsvm-neutron-multinode-full-ubuntu-xenial-nv/89229f6/logs/subnode-2/screen-n-cpu.txt.gz#_Oct_10_13_01_44_516548 | |
| 19:52:56 | mriedem | wth, those times don't line up | |
| 19:53:15 | mriedem | oh http://logs.openstack.org/85/330285/144/check/gate-tempest-dsvm-neutron-multinode-full-ubuntu-xenial-nv/89229f6/logs/subnode-2/screen-n-cpu.txt.gz#_Oct_10_13_01_49_423050 is the attachment_complete call | |
| 19:55:04 | ildikov | that update must be from refresh | |
| 19:55:07 | ildikov | what else? | |
| 19:55:20 | mriedem | that's what i'm thinking too, but don't see where | |
| 20:00:12 | ildikov | that refresh seems to be called in _get_instance_block_device_info in compute/manager.py which gets called in a zillion places | |
| 20:00:41 | mriedem | that's only if refresh_conn_info is True | |
| 20:00:43 | mriedem | it defaults to False | |
| 20:02:09 | ildikov | ah true | |
| 20:05:23 | ildikov | and it uses the default for reboot as it's supposed to | |
| 20:06:32 | ildikov | can anything in libvirt call that? | |
| 20:06:47 | mriedem | there are a couple of places but they are during snapshot | |
| 20:08:28 | openstackgerrit | Merged openstack/nova master: use already loaded BDM in instance.create https://review.openstack.org/483969 | |
| 20:08:30 | ildikov | hmm, weird, as there's nothing around that update call, but brick doing its things... | |
| 20:09:53 | openstackgerrit | Merged openstack/nova master: Fix sending legacy instance.update notification https://review.openstack.org/510603 | |
| 20:11:06 | jgriffith_ | mriedem ildikov http://logs.openstack.org/85/330285/144/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/7a079b9/logs/screen-c-api.txt.gz?#_Oct_10_12_53_01_672274 | |
| 20:12:04 | ildikov | jgriffith_: I think that's 'begin_detaching' | |
| 20:12:11 | jgriffith_ | yes it is | |
| 20:12:18 | ildikov | jgriffith_: which we left untouched | |
| 20:12:28 | jgriffith_ | "we" ? | |
| 20:12:35 | ildikov | so that call is normal | |
| 20:12:53 | ildikov | jgriffith_: I kept you in the loop with that :) | |
| 20:13:37 | ildikov | jgriffith_: I mean to keep that in the flow as we don't want parallel detaches happening as we don't have anything to change the volume or whatever state to 'detaching' | |
| 20:15:08 | openstackgerrit | Eric Fried proposed openstack/nova master: DNM: CI check https://review.openstack.org/511006 | |
| 20:15:54 | mriedem | i'm not really following what begin_detaching has to do with this | |
| 20:16:03 | mriedem | that's the symptom | |
| 20:16:06 | openstackgerrit | Jackie Truong proposed openstack/nova master: Implement certificate_utils https://review.openstack.org/479949 | |
| 20:16:10 | mriedem | looking at the tempest logs, the volume is in-use here http://logs.openstack.org/85/330285/144/check/gate-tempest-dsvm-neutron-multinode-full-ubuntu-xenial-nv/89229f6/console.html#_2017-10-10_13_20_50_909920 | |
| 20:16:34 | mriedem | then we reboot the server, and i see the phantom attachment_update call: http://logs.openstack.org/85/330285/144/check/gate-tempest-dsvm-neutron-multinode-full-ubuntu-xenial-nv/89229f6/logs/subnode-2/screen-n-cpu.txt.gz#_Oct_10_13_02_05_685303 | |
| 20:17:03 | ildikov | jgriffith_: I know, we are not supposed to keep old flow stuff neither copy it to the new one, it seemed a reasonable idea to keep this for now back at the time | |
| 20:17:11 | mriedem | and the failed detach here http://logs.openstack.org/85/330285/144/check/gate-tempest-dsvm-neutron-multinode-full-ubuntu-xenial-nv/89229f6/console.html#_2017-10-10_13_20_50_908565 | |
| 20:17:38 | mriedem | and http://logs.openstack.org/85/330285/144/check/gate-tempest-dsvm-neutron-multinode-full-ubuntu-xenial-nv/89229f6/console.html#_2017-10-10_13_20_50_909920 shows the volume went from in-use to attaching | |
| 20:18:33 | ildikov | mriedem: yeah, that last bit is the consequence of update | |
| 20:19:18 | mriedem | ildikov: if the new refresh_connection_info code thinks that calling attachment_update means we're attaching, then that's a bug | |
| 20:19:24 | mriedem | i thought we talked about this at the ptg | |
| 20:20:56 | ildikov | mriedem: about refresh? | |
| 20:21:05 | mriedem | yeah i'm looking up the etherpad | |
| 20:21:54 | jgriffith_ | ahh, I see; never did the attachment-complete on the reboot | |
| 20:22:14 | mriedem | there shouldn't be an attachment_update on the reboot | |
| 20:22:15 | jgriffith_ | mriedem and yes, you're correct | |
| 20:22:17 | mriedem | i don't know where it's coming from | |
| 20:24:10 | ildikov | mriedem: regardless of how refresh is behaving, there still doesn't seem to be a logical path to that attachment_update call :/ | |
| 20:25:58 | mriedem | L269 https://etherpad.openstack.org/p/cinder-ptg-queens | |