| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-03-03 | |||
| 11:31:18 | AJaeger | johnthetubaguy: you still need them so that the strings do not end in the main file. | |
| 11:31:32 | jbernard | ratailor: can we send hostnames instead of ip addresses? | |
| 11:31:43 | AJaeger | But translation of error and warnings is very very low prio for translators - and not sure it's usefull at all. | |
| 11:31:52 | johnthetubaguy | AJaeger: they would be the API responses then? or many CLI tool stuff? | |
| 11:32:14 | AJaeger | johnthetubaguy: _LW and _LE is what ends in your log files | |
| 11:32:15 | johnthetubaguy | AJaeger: honestly, it seems like totally a mistake we ever tried to translate the logs, but that might just be me | |
| 11:32:28 | ratailor | jbernard, I don't know, but the IPs are hardcoded in domain xml as well as BDM table. | |
| 11:32:39 | johnthetubaguy | AJaeger: yeah, I am just not sure what else we actually have in Nova that should be translated, other than logs | |
| 11:32:43 | AJaeger | johnthetubaguy: Me as well. So, goging forward you could not translate them at all - but discuss that with i18n team first | |
| 11:32:51 | johnthetubaguy | AJaeger: +1 | |
| 11:33:14 | ratailor | johnthetubaguy, https://bugs.launchpad.net/nova/+bug/1581367 | |
| 11:33:14 | openstack | Launchpad bug 1581367 in OpenStack Compute (nova) "Changed SAN iscsi IP addresses in connection_info can prevent VM startup" [Low,Confirmed] | |
| 11:33:35 | AJaeger | johnthetubaguy: for now, we just removed the log files from translation workflow completely... | |
| 11:33:43 | johnthetubaguy | ratailor: so they are soon going to be hardcoded into Cinder, in an immutable way | |
| 11:34:02 | ratailor | johnthetubaguy, I think, this is not only issue with ceph, its with how storage node are IPs are handled. | |
| 11:34:50 | ratailor | johnthetubaguy, could you please provide any reference for that ? | |
| 11:34:59 | johnthetubaguy | AJaeger: cool, I just think you are better removing all of Nova, I have a feeling the other stuff are just logs we haven't moved across, and API messages we never send back in anything but english right now (although I may have just missed that code that translates thouse) | |
| 11:35:36 | johnthetubaguy | ratailor: I have no reference, its just today we assume the IPs don't change, else the whole system falls over, which I guess is what the bugs are describing | |
| 11:36:25 | johnthetubaguy | ratailor: I think lyarwood and mdbooth are better looking at this than I am | |
| 11:36:53 | johnthetubaguy | ratailor: it sounds a bit like we need an os.brick connector refresh on nova-compute startup, but I have no idea how much that would mess up Cinder, probably a lot, for many drivers | |
| 11:37:36 | ratailor | johnthetubaguy, Thanks for your help! Where could I see if there is any progress on keeping hardcoded IPs that won't change ? | |
| 11:37:49 | mdbooth | lyarwood johnthetubaguy: Am I right in thinking that in the new world order we can just re-fetch connection_info from cinder? | |
| 11:38:19 | johnthetubaguy | mdbooth: basically, yes, although we are just getting back what we put there, I think | |
| 11:38:30 | mdbooth | So we could potentially have a retry loop of: try to connect, if it didn't work re-fetch (and re-stash) and try again. | |
| 11:38:50 | mdbooth | So if an iscsi endpoint changed, we'd eventually sort it out | |
| 11:39:02 | lyarwood | this is the ceph mon ip bug right? | |
| 11:39:07 | johnthetubaguy | ratailor: I can't say I heard about this problem before, so not sure if anyone is looking into it yet, looks like mdbooth might have some good ideas though | |
| 11:39:09 | johnthetubaguy | lyarwood: yeah | |
| 11:39:12 | mdbooth | lyarwood: Well this particular bug is about iscsi | |
| 11:39:18 | johnthetubaguy | well, both | |
| 11:39:29 | johnthetubaguy | they feel related | |
| 11:39:29 | AJaeger | johnthetubaguy: yeah - and the priorities of the translators are nowadays horizon and its plugins, osc, docs. | |
| 11:39:36 | mdbooth | The ceph thing is the same wrt volumes, but not wrt local disks | |
| 11:39:39 | johnthetubaguy | AJaeger: that makes total sense | |
| 11:39:42 | mdbooth | With local disks there's no connection_info | |
| 11:39:49 | lyarwood | yeah local disks look it up dynamically | |
| 11:40:02 | lyarwood | I was talking to melwitt about this at the ptg | |
| 11:40:04 | mdbooth | lyarwood: Nah, local disks it's hard-coded in nova.conf | |
| 11:40:12 | lyarwood | mdbooth: nope, not the mon ips | |
| 11:40:19 | mdbooth | lyarwood: Eh? | |
| 11:40:24 | mdbooth | Where does it look it up from? | |
| 11:40:39 | lyarwood | https://github.com/openstack/nova/blob/master/nova/virt/libvirt/storage/rbd_utils.py#L162 | |
| 11:40:50 | johnthetubaguy | well, nova.conf would still be better than in lots of different places in the DB | |
| 11:40:57 | lyarwood | https://github.com/openstack/nova/blob/master/nova/virt/libvirt/imagebackend.py#L813 | |
| 11:41:00 | lyarwood | mdbooth: ^ | |
| 11:41:05 | lyarwood | I was talking to melwitt about this at the ptg | |
| 11:41:13 | johnthetubaguy | it feels like we should just get this from os.brick every time? would that be possible / sensible? | |
| 11:41:19 | lyarwood | for volumes I think she verified a workaround of using shelve and unshelve | |
| 11:41:36 | johnthetubaguy | making hard reboot do it makes sense I guess | |
| 11:41:50 | johnthetubaguy | shelve you risk creating a snapshot | |
| 11:41:58 | lyarwood | true | |
| 11:41:58 | ratailor | lyarwood, She also mentioned start/stop or hard reboot the instance. | |
| 11:42:21 | lyarwood | ratailor: iirc start/stop doesn't update the connection_info | |
| 11:42:30 | mdbooth | lyarwood: Ok, it's hard-coded in ceph.conf | |
| 11:42:33 | johnthetubaguy | lyarwood: yeah, it doesn't today | |
| 11:42:38 | mdbooth | lyarwood: Same thing, really | |
| 11:42:39 | jaosorior | johnthetubaguy: that pretty much covers it, thanks! | |
| 11:42:52 | johnthetubaguy | jaosorior: cool, no problem | |
| 11:43:06 | mdbooth | lyarwood: Does ceph maintain a local db where it keeps up to date with changing mon ips? | |
| 11:43:21 | lyarwood | mdbooth: is that all the command is doing? | |
| 11:43:28 | lyarwood | mdbooth: I assumed this would get the latest | |
| 11:43:29 | mdbooth | I know it can do that after it comes up, but I thought it got the initial list from its config file | |
| 11:43:30 | ratailor | lyarwood, I think hard reboot only. | |
| 11:43:40 | mdbooth | lyarwood: Where would it get it from? | |
| 11:43:44 | mdbooth | It's a bootstrapping problem, right? | |
| 11:43:58 | mdbooth | It needs at least 1 valid mon ip in order to get the current full list | |
| 11:44:12 | lyarwood | mdbooth: no this is after that, once another is added | |
| 11:44:30 | mdbooth | lyarwood: So, when the host comes up, where does it get the list from? | |
| 11:44:33 | lyarwood | ratailor: yeah, I think there was an issue with hard reboot as well | |
| 11:45:09 | ratailor | lyarwood, could you provide any link of that ? as I'm not aware about that. | |
| 11:45:35 | lyarwood | ratailor: this was just from talking to melwitt at the ptg last week, let me try to find my notes on this | |
| 11:45:46 | ratailor | lyarwood, sure. | |
| 11:45:49 | lyarwood | mdbooth: right from the conf file, I'm talking about the nova bug | |
| 11:46:06 | mdbooth | The problem is that nova gets it from the same place | |
| 11:46:19 | mdbooth | So if you change *all* the mon ips, nova can't bootstrap any more | |
| 11:47:07 | lyarwood | I was just thinking of the simple addition use case, not the entire world has changed use case ;) | |
| 11:47:15 | lyarwood | anyway | |
| 11:47:19 | lyarwood | that's imagebackend | |
| 11:47:40 | mdbooth | Well, if we've got at least 1 at any point, ceph will eventually sort itself out with the correct full list | |
| 11:47:58 | mdbooth | Assuming the cluster remains quorate at all times | |
| 11:48:02 | johnthetubaguy | but if that 1 point is no longer valid? | |
| 11:48:23 | mdbooth | johnthetubaguy: You've got a list, and if any of them are valid you should be ok | |
| 11:48:53 | johnthetubaguy | mdbooth: I guess I was thinking about the list of one (which is dumb) and that is no longer valid | |
| 11:49:01 | johnthetubaguy | but I am probably over complicating it | |
| 11:49:01 | mdbooth | If the world changes underneath you, though, I think you need to update ceph.conf | |
| 11:49:13 | johnthetubaguy | the iSCSI one is maybe more problematic | |
| 11:49:48 | mdbooth | johnthetubaguy: I think this iscsi thing is simpler with cinderv3, right? We can just refetch the connection_info for this attachment and retry. | |
| 11:50:08 | lyarwood | so for both rbd and iscsi volumes we can do that in v3 | |
| 11:50:18 | mdbooth | Right, for volumes. | |
| 11:50:19 | lyarwood | we were talking about the rbd imagebackend above | |
| 11:50:27 | lyarwood | that doesn't have connection_info | |
| 11:50:33 | lyarwood | just to make that clear | |
| 11:50:33 | mdbooth | lyarwood: Right, which is why it's harder. | |
| 11:50:48 | mdbooth | I think the volume case is easy enough with cinderv3 | |
| 11:50:49 | lyarwood | mdbooth: agreed | |
| 11:51:13 | lyarwood | cinder-volume should give an updated connection-info dict | |
| 11:51:25 | johnthetubaguy | hmm, not sure thats how the cinder folks see that bit, but I could be wrong | |
| 11:51:34 | johnthetubaguy | that seems like how it should work I guess | |
| 11:51:46 | johnthetubaguy | unless it needs something from os.brick | |
| 11:51:49 | mdbooth | johnthetubaguy: Well if cinder doesn't know where its storage is, Nova should retire to the beach :) | |