| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-03-03 | |||
| 11:43:21 | lyarwood | mdbooth: is that all the command is doing? | |
| 11:43:28 | lyarwood | mdbooth: I assumed this would get the latest | |
| 11:43:29 | mdbooth | I know it can do that after it comes up, but I thought it got the initial list from its config file | |
| 11:43:30 | ratailor | lyarwood, I think hard reboot only. | |
| 11:43:40 | mdbooth | lyarwood: Where would it get it from? | |
| 11:43:44 | mdbooth | It's a bootstrapping problem, right? | |
| 11:43:58 | mdbooth | It needs at least 1 valid mon ip in order to get the current full list | |
| 11:44:12 | lyarwood | mdbooth: no this is after that, once another is added | |
| 11:44:30 | mdbooth | lyarwood: So, when the host comes up, where does it get the list from? | |
| 11:44:33 | lyarwood | ratailor: yeah, I think there was an issue with hard reboot as well | |
| 11:45:09 | ratailor | lyarwood, could you provide any link of that ? as I'm not aware about that. | |
| 11:45:35 | lyarwood | ratailor: this was just from talking to melwitt at the ptg last week, let me try to find my notes on this | |
| 11:45:46 | ratailor | lyarwood, sure. | |
| 11:45:49 | lyarwood | mdbooth: right from the conf file, I'm talking about the nova bug | |
| 11:46:06 | mdbooth | The problem is that nova gets it from the same place | |
| 11:46:19 | mdbooth | So if you change *all* the mon ips, nova can't bootstrap any more | |
| 11:47:07 | lyarwood | I was just thinking of the simple addition use case, not the entire world has changed use case ;) | |
| 11:47:15 | lyarwood | anyway | |
| 11:47:19 | lyarwood | that's imagebackend | |
| 11:47:40 | mdbooth | Well, if we've got at least 1 at any point, ceph will eventually sort itself out with the correct full list | |
| 11:47:58 | mdbooth | Assuming the cluster remains quorate at all times | |
| 11:48:02 | johnthetubaguy | but if that 1 point is no longer valid? | |
| 11:48:23 | mdbooth | johnthetubaguy: You've got a list, and if any of them are valid you should be ok | |
| 11:48:53 | johnthetubaguy | mdbooth: I guess I was thinking about the list of one (which is dumb) and that is no longer valid | |
| 11:49:01 | mdbooth | If the world changes underneath you, though, I think you need to update ceph.conf | |
| 11:49:01 | johnthetubaguy | but I am probably over complicating it | |
| 11:49:13 | johnthetubaguy | the iSCSI one is maybe more problematic | |
| 11:49:48 | mdbooth | johnthetubaguy: I think this iscsi thing is simpler with cinderv3, right? We can just refetch the connection_info for this attachment and retry. | |
| 11:50:08 | lyarwood | so for both rbd and iscsi volumes we can do that in v3 | |
| 11:50:18 | mdbooth | Right, for volumes. | |
| 11:50:19 | lyarwood | we were talking about the rbd imagebackend above | |
| 11:50:27 | lyarwood | that doesn't have connection_info | |
| 11:50:33 | mdbooth | lyarwood: Right, which is why it's harder. | |
| 11:50:33 | lyarwood | just to make that clear | |
| 11:50:48 | mdbooth | I think the volume case is easy enough with cinderv3 | |
| 11:50:49 | lyarwood | mdbooth: agreed | |
| 11:51:13 | lyarwood | cinder-volume should give an updated connection-info dict | |
| 11:51:25 | johnthetubaguy | hmm, not sure thats how the cinder folks see that bit, but I could be wrong | |
| 11:51:34 | johnthetubaguy | that seems like how it should work I guess | |
| 11:51:46 | johnthetubaguy | unless it needs something from os.brick | |
| 11:51:49 | mdbooth | johnthetubaguy: Well if cinder doesn't know where its storage is, Nova should retire to the beach :) | |
| 11:52:13 | johnthetubaguy | mdbooth: true | |
| 11:52:16 | lyarwood | https://github.com/openstack/cinder/blob/master/cinder/volume/drivers/rbd.py#L1024 & https://github.com/openstack/cinder/blob/master/cinder/volume/drivers/rbd.py#L339 | |
| 11:52:22 | lyarwood | they do the same as we do within imagebackend | |
| 11:52:32 | lyarwood | so they would have the same issues if everything changes | |
| 11:53:10 | johnthetubaguy | yeah, they need some kind of refresh trigger from us, maybe? | |
| 11:54:01 | lyarwood | no, you would need to manually update the ceph.conf as mdbooth said above at that point on the c-vol node | |
| 12:00:15 | lyarwood | ratailor: looking at hard reboot again, we don't disconnect and then reconnect volumes from the host let alone grab any updated connection_info from cinder at the moment | |
| 12:03:25 | ratailor | lyarwood, does that mean that we won't be able to resolve this issue fully by only doing hard reboot ? | |
| 12:04:12 | lyarwood | ratailor: yes | |
| 12:05:56 | ratailor | lyarwood, so what could be proper direction to look into ? Could you provide any help ? | |
| 12:06:28 | lyarwood | https://review.openstack.org/#/c/400384/ disconnects and then reconnects volumes during a hard reboot | |
| 12:06:50 | lyarwood | we could also add in another call to initialize_connection somewhere but that's pretty horrid | |
| 12:10:22 | ratailor | lyarwood, once this patch got merged, we need to hard reboot only. But before that we might need to figure out where to call initialize_connection. right ? | |
| 12:12:02 | lyarwood | ratailor: no, you still need to call initialize_connection and save the updated connection_info to the bdm before calling connect_volume again | |
| 12:14:18 | ratailor | lyarwood, Would if be better to handle this issue on restart of nova-compute ? | |
| 12:22:15 | sdague | johnthetubaguy: so I started in on a clean page for the limits discussion wrap up here - https://review.openstack.org/#/c/440815/ | |
| 12:22:59 | johnthetubaguy | sdague: cool, I should take a peak at that soon | |
| 12:23:57 | sdague | johnthetubaguy: it's definitely shooting at being higher level. I'm building out a references section to it now. I would like to discuss with you next steps once you get back from lunch | |
| 12:50:41 | sdague | I just read through your spec, have a chunk of it in my head. Now would be the time to jump on that before I lose it all :) | |
| 13:01:42 | johnthetubaguy | sdague: jumping now | |
| 13:01:54 | johnthetubaguy | sdague: higher level sounds like a top plan | |
| 13:02:03 | sdague | ok, so I just pushed an iterating | |
| 13:02:05 | sdague | iteration | |
| 13:02:26 | sdague | so if you haven't yet looked, https://review.openstack.org/#/c/440815/ | |
| 13:03:06 | sdague | why don't I give you a minute to look at that, then we can chat about it | |
| 13:03:39 | sdague | actually, lets chat about it in #openstack-dev, as we've had tons of related conversations with keystone folks over there effectively the last couple of days | |
| 13:03:48 | johnthetubaguy | sdague: oh, good point | |
| 13:14:03 | openstackgerrit | Arnaud Morin proposed openstack/nova master: Add warning on setting secure_proxy_ssl_header https://review.openstack.org/423375 | |
| 13:37:42 | AJaeger | sdague: nova's api-ref fails with sphinx 1.5 - I fixed it with https://review.openstack.org/#/c/441053/ | |
| 13:38:47 | sdague | AJaeger: thank you | |
| 13:45:46 | edleafe | migi: just starting my day. What's up? | |
| 13:46:07 | edleafe | oops | |
| 13:47:35 | migi | figleaf: Morning, I've run into one issue while invoking tests which I think you wrote but looks like it's some dependency issue on my system. It's failing and still scratching my head why, but I need to spent more time to find out: https://github.com/openstack/nova/blob/master/nova/tests/functional/db/test_resource_provider.py#L1390 | |
| 13:48:01 | migi | figleaf: actually one on line 1372 | |
| 13:48:53 | migi | figleaf: so I may ask you again once I know more | |
| 13:49:01 | migi | for some guidance | |
| 13:49:26 | figleaf | migi: What sort of error are you seeing | |
| 13:49:44 | figleaf | migi: Can you show a traceback in a paste? | |
| 13:51:43 | migi | figleaf: http://paste.openstack.org/show/601322/ | |
| 13:52:00 | migi | figleaf: but don't spent time on that, just if you know from the top of your head what can be cause | |
| 13:54:17 | figleaf | Looks like another test may not have cleaned things up - you have another 'CUSTOM_TWO' class created | |
| 13:54:31 | figleaf | migi: let me look a little more | |
| 13:54:56 | migi | figleaf: well this is what I thought and then renamed this to WHATEVER_TWO | |
| 13:54:59 | migi | figleaf: same error | |
| 13:55:45 | figleaf | migi: heh, was just running the same | |
| 13:57:06 | figleaf | migi: Can you add a line before your trace? print(e.columns) | |
| 13:57:13 | migi | figleaf: just for you to know it's on centos if I install everything in venv on other system like fedora then it's ok, so maybe something is missing in the system | |
| 13:57:52 | figleaf | migi: I can't imagine what would be different in centos | |
| 13:58:02 | figleaf | migi: but that is a data point | |
| 13:58:27 | figleaf | :1372 | |
| 13:58:35 | figleaf | doh! | |
| 13:58:45 | sahid | johnthetubaguy any chance you re-approve https://review.openstack.org/#/c/427066/ ? | |
| 13:58:47 | migi | figleaf: I even installed almost identical pip dependencies same error and running in venv in centos reproduces error, so pretty wired, will do just need to spin this env, will get some results in 15min | |
| 13:58:55 | frickler | AJaeger: is that the same error as this one? http://logs.openstack.org/60/421760/12/check/gate-nova-api-ref/cfe9915/console.html#_2017-03-03_04_25_58_124928 | |
| 13:59:22 | johnthetubaguy | sahid: is that just the same words we had last time? | |
| 13:59:53 | figleaf | migi: are you running your tests under tox? If so, have you tried deleting the .tox/ directory? That is a good first step when tests go wonky | |
| 14:00:20 | openstackgerrit | Alex Xu proposed openstack/nova-specs master: The ResourceProviderTraits API - Manage traits with ResourceProviders https://review.openstack.org/345138 | |
| 14:00:27 | migi | figleaf: yep tried that and with pdb I'm running with testr | |
| 14:00:39 | johnthetubaguy | sahid: if its just the same words, add the git commit tag for that: https://github.com/openstack/nova-specs/blame/master/README.rst#L123 and we can get that re-approved right away | |