| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-03-03 | |||
| 11:41:58 | lyarwood | true | |
| 11:41:58 | ratailor | lyarwood, She also mentioned start/stop or hard reboot the instance. | |
| 11:42:21 | lyarwood | ratailor: iirc start/stop doesn't update the connection_info | |
| 11:42:30 | mdbooth | lyarwood: Ok, it's hard-coded in ceph.conf | |
| 11:42:33 | johnthetubaguy | lyarwood: yeah, it doesn't today | |
| 11:42:38 | mdbooth | lyarwood: Same thing, really | |
| 11:42:39 | jaosorior | johnthetubaguy: that pretty much covers it, thanks! | |
| 11:42:52 | johnthetubaguy | jaosorior: cool, no problem | |
| 11:43:06 | mdbooth | lyarwood: Does ceph maintain a local db where it keeps up to date with changing mon ips? | |
| 11:43:21 | lyarwood | mdbooth: is that all the command is doing? | |
| 11:43:28 | lyarwood | mdbooth: I assumed this would get the latest | |
| 11:43:29 | mdbooth | I know it can do that after it comes up, but I thought it got the initial list from its config file | |
| 11:43:30 | ratailor | lyarwood, I think hard reboot only. | |
| 11:43:40 | mdbooth | lyarwood: Where would it get it from? | |
| 11:43:44 | mdbooth | It's a bootstrapping problem, right? | |
| 11:43:58 | mdbooth | It needs at least 1 valid mon ip in order to get the current full list | |
| 11:44:12 | lyarwood | mdbooth: no this is after that, once another is added | |
| 11:44:30 | mdbooth | lyarwood: So, when the host comes up, where does it get the list from? | |
| 11:44:33 | lyarwood | ratailor: yeah, I think there was an issue with hard reboot as well | |
| 11:45:09 | ratailor | lyarwood, could you provide any link of that ? as I'm not aware about that. | |
| 11:45:35 | lyarwood | ratailor: this was just from talking to melwitt at the ptg last week, let me try to find my notes on this | |
| 11:45:46 | ratailor | lyarwood, sure. | |
| 11:45:49 | lyarwood | mdbooth: right from the conf file, I'm talking about the nova bug | |
| 11:46:06 | mdbooth | The problem is that nova gets it from the same place | |
| 11:46:19 | mdbooth | So if you change *all* the mon ips, nova can't bootstrap any more | |
| 11:47:07 | lyarwood | I was just thinking of the simple addition use case, not the entire world has changed use case ;) | |
| 11:47:15 | lyarwood | anyway | |
| 11:47:19 | lyarwood | that's imagebackend | |
| 11:47:40 | mdbooth | Well, if we've got at least 1 at any point, ceph will eventually sort itself out with the correct full list | |
| 11:47:58 | mdbooth | Assuming the cluster remains quorate at all times | |
| 11:48:02 | johnthetubaguy | but if that 1 point is no longer valid? | |
| 11:48:23 | mdbooth | johnthetubaguy: You've got a list, and if any of them are valid you should be ok | |
| 11:48:53 | johnthetubaguy | mdbooth: I guess I was thinking about the list of one (which is dumb) and that is no longer valid | |
| 11:49:01 | johnthetubaguy | but I am probably over complicating it | |
| 11:49:01 | mdbooth | If the world changes underneath you, though, I think you need to update ceph.conf | |
| 11:49:13 | johnthetubaguy | the iSCSI one is maybe more problematic | |
| 11:49:48 | mdbooth | johnthetubaguy: I think this iscsi thing is simpler with cinderv3, right? We can just refetch the connection_info for this attachment and retry. | |
| 11:50:08 | lyarwood | so for both rbd and iscsi volumes we can do that in v3 | |
| 11:50:18 | mdbooth | Right, for volumes. | |
| 11:50:19 | lyarwood | we were talking about the rbd imagebackend above | |
| 11:50:27 | lyarwood | that doesn't have connection_info | |
| 11:50:33 | lyarwood | just to make that clear | |
| 11:50:33 | mdbooth | lyarwood: Right, which is why it's harder. | |
| 11:50:48 | mdbooth | I think the volume case is easy enough with cinderv3 | |
| 11:50:49 | lyarwood | mdbooth: agreed | |
| 11:51:13 | lyarwood | cinder-volume should give an updated connection-info dict | |
| 11:51:25 | johnthetubaguy | hmm, not sure thats how the cinder folks see that bit, but I could be wrong | |
| 11:51:34 | johnthetubaguy | that seems like how it should work I guess | |
| 11:51:46 | johnthetubaguy | unless it needs something from os.brick | |
| 11:51:49 | mdbooth | johnthetubaguy: Well if cinder doesn't know where its storage is, Nova should retire to the beach :) | |
| 11:52:13 | johnthetubaguy | mdbooth: true | |
| 11:52:16 | lyarwood | https://github.com/openstack/cinder/blob/master/cinder/volume/drivers/rbd.py#L1024 & https://github.com/openstack/cinder/blob/master/cinder/volume/drivers/rbd.py#L339 | |
| 11:52:22 | lyarwood | they do the same as we do within imagebackend | |
| 11:52:32 | lyarwood | so they would have the same issues if everything changes | |
| 11:53:10 | johnthetubaguy | yeah, they need some kind of refresh trigger from us, maybe? | |
| 11:54:01 | lyarwood | no, you would need to manually update the ceph.conf as mdbooth said above at that point on the c-vol node | |
| 12:00:15 | lyarwood | ratailor: looking at hard reboot again, we don't disconnect and then reconnect volumes from the host let alone grab any updated connection_info from cinder at the moment | |
| 12:03:25 | ratailor | lyarwood, does that mean that we won't be able to resolve this issue fully by only doing hard reboot ? | |
| 12:04:12 | lyarwood | ratailor: yes | |
| 12:05:56 | ratailor | lyarwood, so what could be proper direction to look into ? Could you provide any help ? | |
| 12:06:28 | lyarwood | https://review.openstack.org/#/c/400384/ disconnects and then reconnects volumes during a hard reboot | |
| 12:06:50 | lyarwood | we could also add in another call to initialize_connection somewhere but that's pretty horrid | |
| 12:10:22 | ratailor | lyarwood, once this patch got merged, we need to hard reboot only. But before that we might need to figure out where to call initialize_connection. right ? | |
| 12:12:02 | lyarwood | ratailor: no, you still need to call initialize_connection and save the updated connection_info to the bdm before calling connect_volume again | |
| 12:14:18 | ratailor | lyarwood, Would if be better to handle this issue on restart of nova-compute ? | |
| 12:22:15 | sdague | johnthetubaguy: so I started in on a clean page for the limits discussion wrap up here - https://review.openstack.org/#/c/440815/ | |
| 12:22:59 | johnthetubaguy | sdague: cool, I should take a peak at that soon | |
| 12:23:57 | sdague | johnthetubaguy: it's definitely shooting at being higher level. I'm building out a references section to it now. I would like to discuss with you next steps once you get back from lunch | |
| 12:50:41 | sdague | I just read through your spec, have a chunk of it in my head. Now would be the time to jump on that before I lose it all :) | |
| 13:01:42 | johnthetubaguy | sdague: jumping now | |
| 13:01:54 | johnthetubaguy | sdague: higher level sounds like a top plan | |
| 13:02:03 | sdague | ok, so I just pushed an iterating | |
| 13:02:05 | sdague | iteration | |
| 13:02:26 | sdague | so if you haven't yet looked, https://review.openstack.org/#/c/440815/ | |
| 13:03:06 | sdague | why don't I give you a minute to look at that, then we can chat about it | |
| 13:03:39 | sdague | actually, lets chat about it in #openstack-dev, as we've had tons of related conversations with keystone folks over there effectively the last couple of days | |
| 13:03:48 | johnthetubaguy | sdague: oh, good point | |
| 13:14:03 | openstackgerrit | Arnaud Morin proposed openstack/nova master: Add warning on setting secure_proxy_ssl_header https://review.openstack.org/423375 | |
| 13:37:42 | AJaeger | sdague: nova's api-ref fails with sphinx 1.5 - I fixed it with https://review.openstack.org/#/c/441053/ | |
| 13:38:47 | sdague | AJaeger: thank you | |
| 13:45:46 | edleafe | migi: just starting my day. What's up? | |
| 13:46:07 | edleafe | oops | |
| 13:47:35 | migi | figleaf: Morning, I've run into one issue while invoking tests which I think you wrote but looks like it's some dependency issue on my system. It's failing and still scratching my head why, but I need to spent more time to find out: https://github.com/openstack/nova/blob/master/nova/tests/functional/db/test_resource_provider.py#L1390 | |
| 13:48:01 | migi | figleaf: actually one on line 1372 | |
| 13:48:53 | migi | figleaf: so I may ask you again once I know more | |
| 13:49:01 | migi | for some guidance | |
| 13:49:26 | figleaf | migi: What sort of error are you seeing | |
| 13:49:44 | figleaf | migi: Can you show a traceback in a paste? | |
| 13:51:43 | migi | figleaf: http://paste.openstack.org/show/601322/ | |
| 13:52:00 | migi | figleaf: but don't spent time on that, just if you know from the top of your head what can be cause | |
| 13:54:17 | figleaf | Looks like another test may not have cleaned things up - you have another 'CUSTOM_TWO' class created | |
| 13:54:31 | figleaf | migi: let me look a little more | |
| 13:54:56 | migi | figleaf: well this is what I thought and then renamed this to WHATEVER_TWO | |
| 13:54:59 | migi | figleaf: same error | |
| 13:55:45 | figleaf | migi: heh, was just running the same | |
| 13:57:06 | figleaf | migi: Can you add a line before your trace? print(e.columns) | |
| 13:57:13 | migi | figleaf: just for you to know it's on centos if I install everything in venv on other system like fedora then it's ok, so maybe something is missing in the system | |
| 13:57:52 | figleaf | migi: I can't imagine what would be different in centos | |
| 13:58:02 | figleaf | migi: but that is a data point | |
| 13:58:27 | figleaf | :1372 | |