| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-05-05 | |||
| 16:03:30 | mdbooth | johnthetubaguy: Got a second? I'm trying to work out where in the live migration flow nova tells cinder that a volume is going to be attached from a new host | |
| 16:03:43 | mdbooth | I can't find it, though. Is that because it doesn't exist, or because I'm blind? | |
| 16:04:01 | openstackgerrit | Alex Xu proposed openstack/nova master: Totally freeze the extension_info API https://review.openstack.org/462825 | |
| 16:05:44 | alex_xu | sfinucan: thanks for the review :) | |
| 16:06:16 | sfinucan | alex_xu: No problemo 👍 | |
| 16:07:46 | mriedem | mdbooth: pre_live_migration in the virt driver | |
| 16:07:56 | mriedem | mdbooth: that's run on the destination host | |
| 16:08:05 | mriedem | source compute rpc casts to the destination compute | |
| 16:08:40 | mdbooth | mriedem: So that calls vol_driver.connect_volume | |
| 16:08:46 | mriedem | mdbooth: sorry rpc call | |
| 16:09:02 | mriedem | mdbooth: yeah loops over the bdms and calls self._connect_volume(connection_info, disk_info) | |
| 16:09:12 | mdbooth | But I thought (and here I expect to find I'm wrong) that cinder didn't allow connections from unexpected hosts | |
| 16:09:26 | mdbooth | So I was expecting to see a cinder call in there | |
| 16:09:55 | mdbooth | That's just attempting to connect directly to the storage, and expects to be allowed already, right? | |
| 16:11:28 | mriedem | i'm not sure where it initializes the connection | |
| 16:11:34 | mriedem | btw, i think hemna has a flow chart for all of this | |
| 16:12:23 | mdbooth | I think I've got a good grasp of the nova flow, I just can't find this piece | |
| 16:12:34 | mdbooth | But that may be because it doesn't exist, or need to | |
| 16:12:45 | mdbooth | In which case I've misunderstood cinder | |
| 16:12:45 | johnthetubaguy | mdbooth: sorry stuck in a call right now | |
| 16:12:47 | hemna | yah I did | |
| 16:12:52 | hemna | let me see if I can find the url | |
| 16:12:53 | mriedem | mdbooth: looks like post_live_migration | |
| 16:13:30 | mdbooth | mriedem: Oh, interesting | |
| 16:13:59 | mdbooth | If the connection isn't initialised until post_live_migration, how can it expect to connect in pre_live_migration? | |
| 16:14:27 | mdbooth | Incidentally, I'm specifically looking for this because I've got logs where the connect_volume in pre_live_migration fails | |
| 16:14:55 | mdbooth | However, I'm starting from the assumption that this works, and I've just missed something | |
| 16:15:01 | hemna | https://drive.google.com/file/d/0B1Kp6K43HLHyRkFMZHVHWDJ3amM/view | |
| 16:15:07 | mriedem | oh post_live_migration is where it initializes the connection in cinder to get the connection_info, which it uses to disconnect the volume from the source host | |
| 16:15:08 | mdbooth | And the logs aren't due to anything architectural | |
| 16:15:43 | hemna | live migration flow: https://drive.google.com/file/d/0B1Kp6K43HLHydU1wWFVIN29tc2s/view | |
| 16:15:56 | mriedem | it seems odd we have to call cinder again for this since we store the connection_info in the bdm | |
| 16:16:06 | mdbooth | hemna: Wow, thanks | |
| 16:16:07 | mriedem | unless that's now the connection_info for the dest host? | |
| 16:16:19 | hemna | mdbooth, np. :) | |
| 16:16:38 | mdbooth | mriedem: I didn't mean to drag anybody else into my confusion :) Just wondering if anybody happened to know the answer already. | |
| 16:17:14 | mriedem | mdbooth: the answer to any question about the live migration flow is no one konws | |
| 16:17:24 | mdbooth | woooooooo, it's a mystery | |
| 16:24:15 | johnthetubaguy | mdbooth: sorry, looking back... which bit we focusing on? | |
| 16:24:40 | mdbooth | johnthetubaguy: I have log files where the connect_volume in pre_live_migration is failing | |
| 16:25:01 | mdbooth | I was trying to work my way back through the flow to the point where nova told cinder to allow connections from the dest compute | |
| 16:25:24 | johnthetubaguy | mdbooth: so today it kinda doesn't and breaks with some backends, I believe | |
| 16:25:49 | johnthetubaguy | future, we create new attachments, and connect to those | |
| 16:25:57 | mdbooth | Ah... | |
| 16:26:20 | mdbooth | Right, I wasn't entirely sure what form this would take using cinder v2 | |
| 16:26:27 | cfriesen | I just opened bug 1688599. I think this dates back to us allowing evac from an ERROR state, and we forgot to update _instance_in_resize_state(). I should be able to post a proposed fix shortly. | |
| 16:26:27 | openstack | bug 1688599 in OpenStack Compute (nova) "resource audit races against evacuating instance" [Undecided,New] https://launchpad.net/bugs/1688599 | |
| 16:26:42 | mdbooth | johnthetubaguy: So it's literally just trusting that the backend doesn't care where the connections come from? | |
| 16:27:00 | mdbooth | This could be a red herring, btw. | |
| 16:27:33 | openstack | Launchpad bug 1683972 in os-brick "Overlapping iSCSI volume detach/attach can leave behind broken SCSI devices and multipath maps." [Undecided,New] | |
| 16:27:55 | openstackgerrit | Sandeep Pawar proposed openstack/nova master: Libvirt volume driver for Veritas HyperScale https://review.openstack.org/443951 | |
| 16:36:31 | johnthetubaguy | mdbooth: yeah, maybe its just the connector we update in the BDM, rather than calling initalize on the new host, I don't remember now... | |
| 16:36:52 | johnthetubaguy | mdbooth: i remember some of this being why I wanted the connector in cinder and an attachment for each host | |
| 16:39:51 | johnthetubaguy | mdbooth: wow, yeah, its totally a blind connect right now: https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L6706 | |
| 16:42:01 | mdbooth | johnthetubaguy: I'm assuming this works most setups, though. | |
| 16:42:24 | johnthetubaguy | mdbooth: I think ISCSI works, if the password is volume specific not host specific | |
| 16:42:34 | johnthetubaguy | mdbooth: that probably means yes (shrugs) | |
| 16:42:50 | mdbooth | Hehe. That's good, anyway. This is iscsi. | |
| 16:43:16 | mdbooth | Although could be a funky proprietary iscsi target. | |
| 16:52:48 | cdent | needs to remember to pack before leaving | |
| 16:52:54 | cdent | anybody want anything from england? | |
| 17:03:43 | openstackgerrit | Chris Friesen proposed openstack/python-novaclient master: 2.46: Show flavor info in server details https://review.openstack.org/435141 | |
| 17:04:45 | cfriesen | cdent: beer. though I'm not sure how the head will hold up on the flight over. | |
| 17:10:09 | openstackgerrit | Dan Smith proposed openstack/nova master: WIP: Debug race in delete-while-booting test https://review.openstack.org/461513 | |
| 17:10:10 | openstackgerrit | Dan Smith proposed openstack/nova master: DNM: Canary test against multi-cell devstack https://review.openstack.org/458537 | |
| 17:10:19 | dansmith | melwitt: I fixed that vnc thing.. it was a thing I missed in my context usage conversion | |
| 17:10:38 | dansmith | melwitt: now hitting that weird delete case again, but hoping I can repro with debug and not have it dropped by journal this time | |
| 17:13:48 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Add keypairs field to InstanceCreatePayload https://review.openstack.org/419730 | |
| 17:13:48 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Add separate instance.create payload type https://review.openstack.org/463001 | |
| 17:13:49 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Add key_name field to InstancePayload https://review.openstack.org/463002 | |
| 17:43:32 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Add BDM to InstancePayload https://review.openstack.org/448779 | |
| 17:48:53 | melwitt | dansmith: ah, okay. cool | |
| 17:50:54 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Add BDM to InstancePayload https://review.openstack.org/448779 | |
| 17:50:55 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Add key_name field to InstancePayload https://review.openstack.org/463002 | |
| 17:50:55 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Add keypairs field to InstanceCreatePayload https://review.openstack.org/419730 | |
| 17:50:56 | melwitt | I had been looking at the code on master | |
| 17:50:56 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Add separate instance.create payload type https://review.openstack.org/463001 | |
| 17:52:52 | dansmith | yeah | |
| 17:56:04 | mriedem | dansmith: armax: didn't this already come up awhile back https://blueprints.launchpad.net/nova/+spec/sriov-bond? | |
| 17:56:07 | mriedem | for trunk ports? | |
| 17:56:38 | armax | I think so | |
| 17:56:42 | armax | that doesn’t sound new | |
| 17:59:05 | knikolla | hey guys. why does the 'deleted' column in block_device_mapping have a number which is not zero or one. which i would expect from a column showing state? we're having to do some database surgery and not sure what value to manually set it to. | |
| 18:01:03 | mriedem | knikolla: that's a legacy bdm v1 thing | |
| 18:01:06 | mriedem | just take the default | |
| 18:01:42 | knikolla | mriedem: so any nonzero number means deleted? | |
| 18:02:33 | knikolla | mriedem: anywhere else we need to clean up for a stuck volume in the nova side of things? | |
| 18:03:03 | mriedem | knikolla: actually sorry i was confusing that with the no_device field | |
| 18:03:15 | mriedem | knikolla: 'deleted' is whether or not it's "soft" deleted in the database, | |
| 18:03:18 | cfriesen | I'm hitting gate-tempest-dsvm failures in https://review.openstack.org/#/c/434514/. Is this a known thing at the moment? | |
| 18:03:20 | mriedem | so deleted=0 means it's not deleted | |
| 18:03:28 | mriedem | knikolla: if it's 'deleted', then deleted=id | |
| 18:03:40 | mriedem | where id is the block_device_mappings.id primary key | |
| 18:05:40 | knikolla | mriedem: i see, thanks a lot! | |
| 18:09:56 | openstackgerrit | Sandeep Pawar proposed openstack/nova master: Libvirt volume driver for Veritas HyperScale https://review.openstack.org/443951 | |
| 18:10:25 | mriedem | armax: yeah this is what i was thinking of https://review.openstack.org/#/c/213644/ | |
| 18:10:46 | armax | mriedem: no I have a diffeent one in mind | |
| 18:10:47 | armax | hang on | |
| 18:11:31 | armax | https://review.openstack.org/#/c/182242/ | |
| 18:11:52 | mriedem | yeah i pointed that one out in the bp | |