| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-03-03 | |||
| 11:46:19 | mdbooth | So if you change *all* the mon ips, nova can't bootstrap any more | |
| 11:47:07 | lyarwood | I was just thinking of the simple addition use case, not the entire world has changed use case ;) | |
| 11:47:15 | lyarwood | anyway | |
| 11:47:19 | lyarwood | that's imagebackend | |
| 11:47:40 | mdbooth | Well, if we've got at least 1 at any point, ceph will eventually sort itself out with the correct full list | |
| 11:47:58 | mdbooth | Assuming the cluster remains quorate at all times | |
| 11:48:02 | johnthetubaguy | but if that 1 point is no longer valid? | |
| 11:48:23 | mdbooth | johnthetubaguy: You've got a list, and if any of them are valid you should be ok | |
| 11:48:53 | johnthetubaguy | mdbooth: I guess I was thinking about the list of one (which is dumb) and that is no longer valid | |
| 11:49:01 | johnthetubaguy | but I am probably over complicating it | |
| 11:49:01 | mdbooth | If the world changes underneath you, though, I think you need to update ceph.conf | |
| 11:49:13 | johnthetubaguy | the iSCSI one is maybe more problematic | |
| 11:49:48 | mdbooth | johnthetubaguy: I think this iscsi thing is simpler with cinderv3, right? We can just refetch the connection_info for this attachment and retry. | |
| 11:50:08 | lyarwood | so for both rbd and iscsi volumes we can do that in v3 | |
| 11:50:18 | mdbooth | Right, for volumes. | |
| 11:50:19 | lyarwood | we were talking about the rbd imagebackend above | |
| 11:50:27 | lyarwood | that doesn't have connection_info | |
| 11:50:33 | lyarwood | just to make that clear | |
| 11:50:33 | mdbooth | lyarwood: Right, which is why it's harder. | |
| 11:50:48 | mdbooth | I think the volume case is easy enough with cinderv3 | |
| 11:50:49 | lyarwood | mdbooth: agreed | |
| 11:51:13 | lyarwood | cinder-volume should give an updated connection-info dict | |
| 11:51:25 | johnthetubaguy | hmm, not sure thats how the cinder folks see that bit, but I could be wrong | |
| 11:51:34 | johnthetubaguy | that seems like how it should work I guess | |
| 11:51:46 | johnthetubaguy | unless it needs something from os.brick | |
| 11:51:49 | mdbooth | johnthetubaguy: Well if cinder doesn't know where its storage is, Nova should retire to the beach :) | |
| 11:52:13 | johnthetubaguy | mdbooth: true | |
| 11:52:16 | lyarwood | https://github.com/openstack/cinder/blob/master/cinder/volume/drivers/rbd.py#L1024 & https://github.com/openstack/cinder/blob/master/cinder/volume/drivers/rbd.py#L339 | |
| 11:52:22 | lyarwood | they do the same as we do within imagebackend | |
| 11:52:32 | lyarwood | so they would have the same issues if everything changes | |
| 11:53:10 | johnthetubaguy | yeah, they need some kind of refresh trigger from us, maybe? | |
| 11:54:01 | lyarwood | no, you would need to manually update the ceph.conf as mdbooth said above at that point on the c-vol node | |
| 12:00:15 | lyarwood | ratailor: looking at hard reboot again, we don't disconnect and then reconnect volumes from the host let alone grab any updated connection_info from cinder at the moment | |
| 12:03:25 | ratailor | lyarwood, does that mean that we won't be able to resolve this issue fully by only doing hard reboot ? | |
| 12:04:12 | lyarwood | ratailor: yes | |
| 12:05:56 | ratailor | lyarwood, so what could be proper direction to look into ? Could you provide any help ? | |
| 12:06:28 | lyarwood | https://review.openstack.org/#/c/400384/ disconnects and then reconnects volumes during a hard reboot | |
| 12:06:50 | lyarwood | we could also add in another call to initialize_connection somewhere but that's pretty horrid | |
| 12:10:22 | ratailor | lyarwood, once this patch got merged, we need to hard reboot only. But before that we might need to figure out where to call initialize_connection. right ? | |
| 12:12:02 | lyarwood | ratailor: no, you still need to call initialize_connection and save the updated connection_info to the bdm before calling connect_volume again | |
| 12:14:18 | ratailor | lyarwood, Would if be better to handle this issue on restart of nova-compute ? | |
| 12:22:15 | sdague | johnthetubaguy: so I started in on a clean page for the limits discussion wrap up here - https://review.openstack.org/#/c/440815/ | |
| 12:22:59 | johnthetubaguy | sdague: cool, I should take a peak at that soon | |
| 12:23:57 | sdague | johnthetubaguy: it's definitely shooting at being higher level. I'm building out a references section to it now. I would like to discuss with you next steps once you get back from lunch | |
| 12:50:41 | sdague | I just read through your spec, have a chunk of it in my head. Now would be the time to jump on that before I lose it all :) | |
| 13:01:42 | johnthetubaguy | sdague: jumping now | |
| 13:01:54 | johnthetubaguy | sdague: higher level sounds like a top plan | |
| 13:02:03 | sdague | ok, so I just pushed an iterating | |
| 13:02:05 | sdague | iteration | |
| 13:02:26 | sdague | so if you haven't yet looked, https://review.openstack.org/#/c/440815/ | |
| 13:03:06 | sdague | why don't I give you a minute to look at that, then we can chat about it | |
| 13:03:39 | sdague | actually, lets chat about it in #openstack-dev, as we've had tons of related conversations with keystone folks over there effectively the last couple of days | |
| 13:03:48 | johnthetubaguy | sdague: oh, good point | |
| 13:14:03 | openstackgerrit | Arnaud Morin proposed openstack/nova master: Add warning on setting secure_proxy_ssl_header https://review.openstack.org/423375 | |
| 13:37:42 | AJaeger | sdague: nova's api-ref fails with sphinx 1.5 - I fixed it with https://review.openstack.org/#/c/441053/ | |
| 13:38:47 | sdague | AJaeger: thank you | |
| 13:45:46 | edleafe | migi: just starting my day. What's up? | |
| 13:46:07 | edleafe | oops | |
| 13:47:35 | migi | figleaf: Morning, I've run into one issue while invoking tests which I think you wrote but looks like it's some dependency issue on my system. It's failing and still scratching my head why, but I need to spent more time to find out: https://github.com/openstack/nova/blob/master/nova/tests/functional/db/test_resource_provider.py#L1390 | |
| 13:48:01 | migi | figleaf: actually one on line 1372 | |
| 13:48:53 | migi | figleaf: so I may ask you again once I know more | |
| 13:49:01 | migi | for some guidance | |
| 13:49:26 | figleaf | migi: What sort of error are you seeing | |
| 13:49:44 | figleaf | migi: Can you show a traceback in a paste? | |
| 13:51:43 | migi | figleaf: http://paste.openstack.org/show/601322/ | |
| 13:52:00 | migi | figleaf: but don't spent time on that, just if you know from the top of your head what can be cause | |
| 13:54:17 | figleaf | Looks like another test may not have cleaned things up - you have another 'CUSTOM_TWO' class created | |
| 13:54:31 | figleaf | migi: let me look a little more | |
| 13:54:56 | migi | figleaf: well this is what I thought and then renamed this to WHATEVER_TWO | |
| 13:54:59 | migi | figleaf: same error | |
| 13:55:45 | figleaf | migi: heh, was just running the same | |
| 13:57:06 | figleaf | migi: Can you add a line before your trace? print(e.columns) | |
| 13:57:13 | migi | figleaf: just for you to know it's on centos if I install everything in venv on other system like fedora then it's ok, so maybe something is missing in the system | |
| 13:57:52 | figleaf | migi: I can't imagine what would be different in centos | |
| 13:58:02 | figleaf | migi: but that is a data point | |
| 13:58:27 | figleaf | :1372 | |
| 13:58:35 | figleaf | doh! | |
| 13:58:45 | sahid | johnthetubaguy any chance you re-approve https://review.openstack.org/#/c/427066/ ? | |
| 13:58:47 | migi | figleaf: I even installed almost identical pip dependencies same error and running in venv in centos reproduces error, so pretty wired, will do just need to spin this env, will get some results in 15min | |
| 13:58:55 | frickler | AJaeger: is that the same error as this one? http://logs.openstack.org/60/421760/12/check/gate-nova-api-ref/cfe9915/console.html#_2017-03-03_04_25_58_124928 | |
| 13:59:22 | johnthetubaguy | sahid: is that just the same words we had last time? | |
| 13:59:53 | figleaf | migi: are you running your tests under tox? If so, have you tried deleting the .tox/ directory? That is a good first step when tests go wonky | |
| 14:00:20 | openstackgerrit | Alex Xu proposed openstack/nova-specs master: The ResourceProviderTraits API - Manage traits with ResourceProviders https://review.openstack.org/345138 | |
| 14:00:27 | migi | figleaf: yep tried that and with pdb I'm running with testr | |
| 14:00:39 | johnthetubaguy | sahid: if its just the same words, add the git commit tag for that: https://github.com/openstack/nova-specs/blame/master/README.rst#L123 and we can get that re-approved right away | |
| 14:02:54 | sahid | johnthetubaguy: ah ok will do | |
| 14:03:20 | AJaeger | frickler: yes | |
| 14:05:01 | AJaeger | frickler: unless that change introduces a new problem ;) | |
| 14:05:04 | openstackgerrit | sahid proposed openstack/nova-specs master: Libvirt driver emulator threads placement policy https://review.openstack.org/427066 | |
| 14:05:55 | sahid | johnthetubaguy: should be ok now, thanks | |
| 14:06:31 | johnthetubaguy | sahid: will try look at that soon, if its not swiped by someone else first | |
| 14:06:38 | johnthetubaguy | got my head in quotas right now | |
| 14:07:09 | sahid | ok understood | |
| 14:14:33 | mriedem | yikes gate is at nearly 18 hours | |
| 14:15:35 | frickler | mriedem: yup, would be nice if some infra-root would wake up ;) | |
| 14:15:52 | mriedem | frickler: i'm not sure what infra has to do with this? | |
| 14:16:30 | frickler | mriedem: the top gate check is failing to get some trusty-multinode job run. at least for the last 6 hours while I'm watching it | |
| 14:16:51 | mriedem | oh i see gate-grenade-dsvm-neutron-multinode-ubuntu-trusty | |
| 14:17:04 | mriedem | sdague: mordred: fungi: have you seen ^? | |
| 14:20:48 | mriedem | zuul jobs launched per hour has dropped way off, is zuul v3 being used now? | |