| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-06-04 | |||
| 15:44:47 | naichuans | efried: Np, thank you very much for the review :) | |
| 15:53:10 | openstackgerrit | MultipleCrashes proposed openstack/nova master: Retry decorator fix for autoscale delete https://review.openstack.org/570370 | |
| 17:39:52 | karimull | Hi ...need your reviews on https://review.openstack.org/#/c/569498/ | |
| 18:17:59 | mriedem | karimull: i suggest putting that into the runways review queue https://etherpad.openstack.org/p/nova-runways-rocky | |
| 18:43:12 | efried | karimull: Reviewed. | |
| 18:43:57 | efried | karimull: But I agree with mriedem - this definitely needs eyes from the likes of jaypipes, dansmith, mriedem. | |
| 18:51:42 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Fix typo in enable_certificate_validation config option help https://review.openstack.org/572185 | |
| 18:55:17 | openstackgerrit | Jay Pipes proposed openstack/nova-specs master: Standardize CPU resource tracking https://review.openstack.org/555081 | |
| 19:47:53 | openstackgerrit | Dan Smith proposed openstack/nova master: Change consecutive build failure limit to a weigher https://review.openstack.org/572195 | |
| 20:06:21 | openstackgerrit | Matt Riedemann proposed openstack/nova-specs master: Remove device from volume attach requests (spec) https://review.openstack.org/452546 | |
| 20:06:28 | mriedem | dansmith: finally got around to updating this old spec ^ | |
| 20:18:40 | openstackgerrit | Jay Pipes proposed openstack/nova-specs master: Standardize CPU resource tracking https://review.openstack.org/555081 | |
| 20:43:57 | lyarwood | efried: https://bugs.launchpad.net/nova/+bug/1775075 - afternoon, if you have a second could you take a look at this bug and the comment I left on the original change? https://review.openstack.org/#/c/510947/1/nova/context.py@121 | |
| 20:43:58 | openstack | Launchpad bug 1775075 in OpenStack Compute (nova) "EndpointNotFound raised by Pike n-cpu when running alongside Queens n-api" [Undecided,New] | |
| 20:45:32 | efried | lyarwood: looking... | |
| 20:45:47 | lyarwood | efried: thanks :) | |
| 20:48:21 | efried | lyarwood: Okay, I think this may come down to what version of ksa exists on pike, and whether os-service-types is available. | |
| 20:50:02 | lyarwood | efried: 3.1.0 iirc | |
| 20:50:40 | lyarwood | efried: adding volumev2 back in nova/context.py is enough in my local env FWIW | |
| 20:50:57 | lyarwood | efried: on the n-api host that is | |
| 20:51:19 | cdent | is tomorrow spec sprint again? | |
| 20:51:34 | efried | lyarwood: I'm fine with that as a resolution; but we should probably bring mordred in on this issue, since he was basically leading me through a lot of the service catalog work. | |
| 20:51:43 | melwitt | cdent: yes | |
| 20:51:43 | efried | cdent: yes, afaik | |
| 20:51:49 | cdent | cool, thanks | |
| 20:52:08 | melwitt | I'll send another email to remind to the dev ML late today | |
| 20:52:30 | lyarwood | efried: ack, I'll push something to stable/queens now | |
| 20:52:40 | lyarwood | efried: thanks :) | |
| 20:53:02 | efried | lyarwood: os-service-types is not in requirements in pike, and I think that's probably the source of the problem. | |
| 20:54:26 | lyarwood | efried: hmm okay, AFAICT the Queens n-api was just stripping the volumev2 endpoints from the request context it was sending to the Pike compute, I'm not sure how os-service-types could help there but I've never really touched this area before. | |
| 20:55:37 | efried | lyarwood: The endpoints themselves should be getting picked up correctly on the queens side. If we're sending actual endpoints across the wire, that should work fine. | |
| 20:56:05 | efried | lyarwood: Because the queens side will have os-service-types, which means it'll find the correct endpoints (whether they're called volumev2 or whatever) for block-storage. | |
| 20:57:05 | mordred | efried: aroo? | |
| 20:59:46 | efried | mordred: See https://bugs.launchpad.net/nova/+bug/1775075 and https://review.openstack.org/#/c/510947/1/nova/context.py@121 | |
| 20:59:47 | openstack | Launchpad bug 1775075 in OpenStack Compute (nova) "EndpointNotFound raised by Pike n-cpu when running alongside Queens n-api" [Undecided,New] | |
| 21:00:59 | efried | mordred: What lyarwood is finding is that when he adds 'volumev2' back in on the queens side, it fixes the problem. Which I don't quite understand why doing that on the *queens* side would fix. Because that guy has os-service-types and the right ksa... oh, unless this is one of those ksa bugs we've since fixed, where endpoint lookup isn't happening right. | |
| 21:02:10 | lyarwood | efried: so the Pike side is looking for the cinderv2 endpoint in the catalog stashed in the request context | |
| 21:02:28 | lyarwood | efried: the cinderv3 endpoint is there correctly, but without the cinderv2 endpoint it just fails | |
| 21:02:37 | mordred | efried: probably. when you say "queens side" and "pike side" - what does that mean here? | |
| 21:02:51 | lyarwood | mordred: queens n-api, pike n-cpu | |
| 21:02:56 | mordred | gotcha | |
| 21:03:49 | mordred | yah - I think you'll need the volumev2 in there until everything is on queens - I think efried is correct about there being missing fixes | |
| 21:05:11 | lyarwood | oh I forgot to say that our P n-cpu's have catalog_info=volumev2:cinderv2:internal in nova.conf forcing them to look for cinderv2 | |
| 21:06:19 | efried | ah | |
| 21:06:31 | efried | well then yeah | |
| 21:06:38 | lyarwood | yeah my bad, the default is cinderv3 right? | |
| 21:06:48 | efried | so is this a bug in the code, or is this something that should be fixed via conf on site? | |
| 21:07:21 | lyarwood | I'd say code still if possible | |
| 21:08:10 | efried | lyarwood: Is there a similar argument for restoring 'volume' ? | |
| 21:08:41 | lyarwood | efried: yeah assuming you could set catalog_info to look for it | |
| 21:11:12 | efried | lyarwood: tbc, this patch doesn't exist on pike, right? | |
| 21:11:19 | efried | https://review.openstack.org/#/c/510947/ that is | |
| 21:11:28 | lyarwood | efried: correct | |
| 21:11:31 | lyarwood | efried: just Queens | |
| 21:12:13 | efried | lyarwood: Okay, well, restoring 'volume' is not necessary because https://review.openstack.org/#/c/409904/ | |
| 21:12:45 | efried | lyarwood: But volumev2 was dropped in q (https://review.openstack.org/#/c/501874/) | |
| 21:13:09 | efried | lyarwood: So I guess unless queens n-api can talk to ocata n-cpu? Is that a thing? | |
| 21:13:40 | lyarwood | efried: nope, shouldn't be. | |
| 21:14:17 | efried | okay. Then I suppose restoring volumev2 on the premise that q can talk to p is a reasonable thing. Though I'm still confused as to why the problem would manifest if the endpoint lookup is being done on the q side. | |
| 21:14:52 | efried | lyarwood: Do you have the ability to check whether upgrading ksa (without reintroducing 'volumev2' in that RequestContext filter) also fixes the problem? | |
| 21:15:13 | lyarwood | efried: on the Q n-api host? | |
| 21:15:38 | efried | yeah - upgrade ksa and service-types-authoryt. | |
| 21:15:42 | efried | authority | |
| 21:15:58 | efried | lyarwood: That would give us the info as to whether the issue lies with missing ksa fixes, as mordred quasi-corroborated. | |
| 21:16:13 | lyarwood | efried: just to the latest stable versions I assume? | |
| 21:16:20 | lyarwood | efried: I can give that a try | |
| 21:16:26 | efried | lyarwood: yeah, that should be adequate. | |
| 21:20:51 | mordred | latest ksa SHOULD make this happier | |
| 21:27:02 | lyarwood | efried / mordred ; still fails with ksa 3.7.0 | |
| 21:27:42 | efried | lyarwood: Okay, that's an interesting data point. Are you sure the RequestContext itself is really going across the wire, complete with its service catalog? | |
| 21:28:28 | efried | lyarwood: If the cached service catalog is being stripped, but the remainder of the context is being trasmitted, then that would certainly cause the problem. Because the pike side doesn't have os-service-types so it doesn't know what to do with 'block-storage'. | |
| 21:29:10 | efried | or maybe we haven't looked up (the cinder endpoint in) the service catalog yet by the time it's being sent over. | |
| 21:29:39 | lyarwood | efried: yeah I'm sure, cinderv3 is there when it's going over the wire from Q | |
| 21:30:06 | efried | lyarwood: Ohh, because the queens side would be using the cinderv3 endpoint. | |
| 21:30:12 | efried | lyarwood: But the pike side wants the v2 endpoint. | |
| 21:30:23 | lyarwood | efried: yeah indeed | |
| 21:30:51 | efried | So if the q request context only looked up v3, even if it sent the cached service catalog over the wire, the p side wouldn't see the v2 endpoint in it. | |
| 21:31:51 | lyarwood | efried: it's actually finding both before we hit nova/context.py iirc | |
| 21:32:06 | efried | lyarwood: But would have no reason to load up the discovery info for v2 | |
| 21:32:08 | lyarwood | efried: on Q, but we then strip cinderv2 out causing the problem | |
| 21:32:20 | efried | oh, I see, yes. | |
| 21:32:42 | efried | lyarwood: IMO the bug sounds like we shouldn't be sending the RequestContext from Q to P. But we're not gonna fix that at this point. Adding volumev2 back in sounds like the less intrusive fix. | |
| 21:46:49 | openstackgerrit | Lee Yarwood proposed openstack/nova stable/queens: Allow cinderv2 endpoints within the request context catalog https://review.openstack.org/572213 | |
| 21:47:25 | lyarwood | efried: ^ I'll follow up with more tests in the morning. | |
| 21:47:30 | lyarwood | efried: thanks again :) | |
| 21:47:35 | efried | lyarwood: Thank you | |
| 22:08:02 | openstackgerrit | David Bingham proposed openstack/nova master: Adding apply_cells to nova-manage to enable automated cells_v2 configuration. https://review.openstack.org/572216 | |
| 22:28:25 | openstackgerrit | David Bingham proposed openstack/nova master: Adding apply_cells to nova-manage to enable automated cells_v2 configuration. https://review.openstack.org/568987 | |
| 22:36:22 | kb__ | I am running Mitaka with log leve in nova.conf set to INFO.. I want to capture the compute host info in any of the nova-*.logs . Setting up Verbose=True doesn't help, only setting debug=True in the nova.conf displays the compute host info. | |
| 22:36:34 | kb__ | Is there a way to display compute host info while having log level set to INFO. | |
| #openstack-nova - 2018-06-05 | |||
| 00:09:29 | openstackgerrit | Tsuyoshi Nagata proposed openstack/nova master: openstack server add volume fails over 26vols https://review.openstack.org/567472 | |
| 00:14:02 | openstackgerrit | Chris Dent proposed openstack/nova master: [placement] Do not import oslo_service for log_options https://review.openstack.org/572231 | |
| 01:31:14 | openstackgerrit | tianhui proposed openstack/nova master: Fix bug for hypervisors https://review.openstack.org/572063 | |
| 02:09:47 | pvc | hi | |
| 02:16:10 | openstackgerrit | Yikun Jiang (Kero) proposed openstack/nova master: Fix wrong commands in launch-from-volume doc https://review.openstack.org/572241 | |
| 02:19:46 | pvc | Filter PciPassthroughFilter returned 0 hosts | |
| 02:45:27 | pvc | qemu unexpectedly | |
| 02:45:31 | pvc | libvirtError: internal error: qemu unexpectedly closed the monitor: 2018-06-05T02:43:47.447602Z qemu-kvm: -device vfio-pci,host=01:00.0,id=hostdev0,bus=pci.0,addr=0x5: vfio error: 0000:01:00.0: group 1 is not viable | |
| 02:48:18 | openstackgerrit | Zhenyu Zheng proposed openstack/nova master: Use ThreadPoolExecutor for max_concurrent_live_migrations https://review.openstack.org/563505 | |
| 02:48:53 | pvc | gibi | |