| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-03-03 | |||
| 11:35:36 | johnthetubaguy | ratailor: I have no reference, its just today we assume the IPs don't change, else the whole system falls over, which I guess is what the bugs are describing | |
| 11:36:25 | johnthetubaguy | ratailor: I think lyarwood and mdbooth are better looking at this than I am | |
| 11:36:53 | johnthetubaguy | ratailor: it sounds a bit like we need an os.brick connector refresh on nova-compute startup, but I have no idea how much that would mess up Cinder, probably a lot, for many drivers | |
| 11:37:36 | ratailor | johnthetubaguy, Thanks for your help! Where could I see if there is any progress on keeping hardcoded IPs that won't change ? | |
| 11:37:49 | mdbooth | lyarwood johnthetubaguy: Am I right in thinking that in the new world order we can just re-fetch connection_info from cinder? | |
| 11:38:19 | johnthetubaguy | mdbooth: basically, yes, although we are just getting back what we put there, I think | |
| 11:38:30 | mdbooth | So we could potentially have a retry loop of: try to connect, if it didn't work re-fetch (and re-stash) and try again. | |
| 11:38:50 | mdbooth | So if an iscsi endpoint changed, we'd eventually sort it out | |
| 11:39:02 | lyarwood | this is the ceph mon ip bug right? | |
| 11:39:07 | johnthetubaguy | ratailor: I can't say I heard about this problem before, so not sure if anyone is looking into it yet, looks like mdbooth might have some good ideas though | |
| 11:39:09 | johnthetubaguy | lyarwood: yeah | |
| 11:39:12 | mdbooth | lyarwood: Well this particular bug is about iscsi | |
| 11:39:18 | johnthetubaguy | well, both | |
| 11:39:29 | AJaeger | johnthetubaguy: yeah - and the priorities of the translators are nowadays horizon and its plugins, osc, docs. | |
| 11:39:29 | johnthetubaguy | they feel related | |
| 11:39:36 | mdbooth | The ceph thing is the same wrt volumes, but not wrt local disks | |
| 11:39:39 | johnthetubaguy | AJaeger: that makes total sense | |
| 11:39:42 | mdbooth | With local disks there's no connection_info | |
| 11:39:49 | lyarwood | yeah local disks look it up dynamically | |
| 11:40:02 | lyarwood | I was talking to melwitt about this at the ptg | |
| 11:40:04 | mdbooth | lyarwood: Nah, local disks it's hard-coded in nova.conf | |
| 11:40:12 | lyarwood | mdbooth: nope, not the mon ips | |
| 11:40:19 | mdbooth | lyarwood: Eh? | |
| 11:40:24 | mdbooth | Where does it look it up from? | |
| 11:40:39 | lyarwood | https://github.com/openstack/nova/blob/master/nova/virt/libvirt/storage/rbd_utils.py#L162 | |
| 11:40:50 | johnthetubaguy | well, nova.conf would still be better than in lots of different places in the DB | |
| 11:40:57 | lyarwood | https://github.com/openstack/nova/blob/master/nova/virt/libvirt/imagebackend.py#L813 | |
| 11:41:00 | lyarwood | mdbooth: ^ | |
| 11:41:05 | lyarwood | I was talking to melwitt about this at the ptg | |
| 11:41:13 | johnthetubaguy | it feels like we should just get this from os.brick every time? would that be possible / sensible? | |
| 11:41:19 | lyarwood | for volumes I think she verified a workaround of using shelve and unshelve | |
| 11:41:36 | johnthetubaguy | making hard reboot do it makes sense I guess | |
| 11:41:50 | johnthetubaguy | shelve you risk creating a snapshot | |
| 11:41:58 | ratailor | lyarwood, She also mentioned start/stop or hard reboot the instance. | |
| 11:41:58 | lyarwood | true | |
| 11:42:21 | lyarwood | ratailor: iirc start/stop doesn't update the connection_info | |
| 11:42:30 | mdbooth | lyarwood: Ok, it's hard-coded in ceph.conf | |
| 11:42:33 | johnthetubaguy | lyarwood: yeah, it doesn't today | |
| 11:42:38 | mdbooth | lyarwood: Same thing, really | |
| 11:42:39 | jaosorior | johnthetubaguy: that pretty much covers it, thanks! | |
| 11:42:52 | johnthetubaguy | jaosorior: cool, no problem | |
| 11:43:06 | mdbooth | lyarwood: Does ceph maintain a local db where it keeps up to date with changing mon ips? | |
| 11:43:21 | lyarwood | mdbooth: is that all the command is doing? | |
| 11:43:28 | lyarwood | mdbooth: I assumed this would get the latest | |
| 11:43:29 | mdbooth | I know it can do that after it comes up, but I thought it got the initial list from its config file | |
| 11:43:30 | ratailor | lyarwood, I think hard reboot only. | |
| 11:43:40 | mdbooth | lyarwood: Where would it get it from? | |
| 11:43:44 | mdbooth | It's a bootstrapping problem, right? | |
| 11:43:58 | mdbooth | It needs at least 1 valid mon ip in order to get the current full list | |
| 11:44:12 | lyarwood | mdbooth: no this is after that, once another is added | |
| 11:44:30 | mdbooth | lyarwood: So, when the host comes up, where does it get the list from? | |
| 11:44:33 | lyarwood | ratailor: yeah, I think there was an issue with hard reboot as well | |
| 11:45:09 | ratailor | lyarwood, could you provide any link of that ? as I'm not aware about that. | |
| 11:45:35 | lyarwood | ratailor: this was just from talking to melwitt at the ptg last week, let me try to find my notes on this | |
| 11:45:46 | ratailor | lyarwood, sure. | |
| 11:45:49 | lyarwood | mdbooth: right from the conf file, I'm talking about the nova bug | |
| 11:46:06 | mdbooth | The problem is that nova gets it from the same place | |
| 11:46:19 | mdbooth | So if you change *all* the mon ips, nova can't bootstrap any more | |
| 11:47:07 | lyarwood | I was just thinking of the simple addition use case, not the entire world has changed use case ;) | |
| 11:47:15 | lyarwood | anyway | |
| 11:47:19 | lyarwood | that's imagebackend | |
| 11:47:40 | mdbooth | Well, if we've got at least 1 at any point, ceph will eventually sort itself out with the correct full list | |
| 11:47:58 | mdbooth | Assuming the cluster remains quorate at all times | |
| 11:48:02 | johnthetubaguy | but if that 1 point is no longer valid? | |
| 11:48:23 | mdbooth | johnthetubaguy: You've got a list, and if any of them are valid you should be ok | |
| 11:48:53 | johnthetubaguy | mdbooth: I guess I was thinking about the list of one (which is dumb) and that is no longer valid | |
| 11:49:01 | mdbooth | If the world changes underneath you, though, I think you need to update ceph.conf | |
| 11:49:01 | johnthetubaguy | but I am probably over complicating it | |
| 11:49:13 | johnthetubaguy | the iSCSI one is maybe more problematic | |
| 11:49:48 | mdbooth | johnthetubaguy: I think this iscsi thing is simpler with cinderv3, right? We can just refetch the connection_info for this attachment and retry. | |
| 11:50:08 | lyarwood | so for both rbd and iscsi volumes we can do that in v3 | |
| 11:50:18 | mdbooth | Right, for volumes. | |
| 11:50:19 | lyarwood | we were talking about the rbd imagebackend above | |
| 11:50:27 | lyarwood | that doesn't have connection_info | |
| 11:50:33 | mdbooth | lyarwood: Right, which is why it's harder. | |
| 11:50:33 | lyarwood | just to make that clear | |
| 11:50:48 | mdbooth | I think the volume case is easy enough with cinderv3 | |
| 11:50:49 | lyarwood | mdbooth: agreed | |
| 11:51:13 | lyarwood | cinder-volume should give an updated connection-info dict | |
| 11:51:25 | johnthetubaguy | hmm, not sure thats how the cinder folks see that bit, but I could be wrong | |
| 11:51:34 | johnthetubaguy | that seems like how it should work I guess | |
| 11:51:46 | johnthetubaguy | unless it needs something from os.brick | |
| 11:51:49 | mdbooth | johnthetubaguy: Well if cinder doesn't know where its storage is, Nova should retire to the beach :) | |
| 11:52:13 | johnthetubaguy | mdbooth: true | |
| 11:52:16 | lyarwood | https://github.com/openstack/cinder/blob/master/cinder/volume/drivers/rbd.py#L1024 & https://github.com/openstack/cinder/blob/master/cinder/volume/drivers/rbd.py#L339 | |
| 11:52:22 | lyarwood | they do the same as we do within imagebackend | |
| 11:52:32 | lyarwood | so they would have the same issues if everything changes | |
| 11:53:10 | johnthetubaguy | yeah, they need some kind of refresh trigger from us, maybe? | |
| 11:54:01 | lyarwood | no, you would need to manually update the ceph.conf as mdbooth said above at that point on the c-vol node | |
| 12:00:15 | lyarwood | ratailor: looking at hard reboot again, we don't disconnect and then reconnect volumes from the host let alone grab any updated connection_info from cinder at the moment | |
| 12:03:25 | ratailor | lyarwood, does that mean that we won't be able to resolve this issue fully by only doing hard reboot ? | |
| 12:04:12 | lyarwood | ratailor: yes | |
| 12:05:56 | ratailor | lyarwood, so what could be proper direction to look into ? Could you provide any help ? | |
| 12:06:28 | lyarwood | https://review.openstack.org/#/c/400384/ disconnects and then reconnects volumes during a hard reboot | |
| 12:06:50 | lyarwood | we could also add in another call to initialize_connection somewhere but that's pretty horrid | |
| 12:10:22 | ratailor | lyarwood, once this patch got merged, we need to hard reboot only. But before that we might need to figure out where to call initialize_connection. right ? | |
| 12:12:02 | lyarwood | ratailor: no, you still need to call initialize_connection and save the updated connection_info to the bdm before calling connect_volume again | |
| 12:14:18 | ratailor | lyarwood, Would if be better to handle this issue on restart of nova-compute ? | |
| 12:22:15 | sdague | johnthetubaguy: so I started in on a clean page for the limits discussion wrap up here - https://review.openstack.org/#/c/440815/ | |
| 12:22:59 | johnthetubaguy | sdague: cool, I should take a peak at that soon | |