| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-10-18 | |||
| 11:59:26 | janki | efried, all jenkins build logs are at https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-pike-upstream-stateful-carbon/36/ | |
| 12:01:29 | efried | rgerganov Responded, sorry I missed that the first time. I would have seen it eventually :) | |
| 12:02:15 | efried | rgerganov Let me know if you want to talk it though some more. There's a lot TBD about how drivers will model their RPs, and we especially haven't talked a lot about IP addresses as generic resources. | |
| 12:02:30 | efried | rgerganov At least *I* haven't. sean-k-mooney might know more on that front. | |
| 12:03:50 | sean-k-mooney | i heard my name :) | |
| 12:04:18 | sean-k-mooney | IP adress as a generic resource? | |
| 12:04:33 | sean-k-mooney | as in like routed network today or something different | |
| 12:04:46 | janki | efried, I dont find any api call before running this test. ANy specific file I should be looking into? | |
| 12:05:21 | efried | janki I was poking around for your n-api log, haven't found it yet. Was skimming the n-cpu log, but kinda doubt I'm going to find anything useful in there. | |
| 12:05:57 | efried | sean-k-mooney Greetings. rgerganov was asking about how https://review.openstack.org/511879 would play with IP addresses. | |
| 12:05:58 | rgerganov | efried, thanks for the response, it seems that our longstanding problem with VMW shared datastores won't get fixed any time soon :) | |
| 12:06:30 | efried | rgerganov Tell me more about the problem, if you have time. Perhaps there's a nice placement-y solution... | |
| 12:06:31 | sean-k-mooney | rgerganov: i think the intent is that if you need to boot a vm on a specifc nfs share you will associate the boot request with the nfs share by modeling the nfs share as a shared resource provider and using an aggregate to associate the hosts that can connect to that share | |
| 12:07:05 | efried | sean-k-mooney ++ | |
| 12:07:24 | efried | But note that shared RPs are not in scope for Queens. | |
| 12:07:41 | efried | Though there's some code already in place that's gonna have to be trodden carefully around. | |
| 12:08:32 | sean-k-mooney | ya thats the rub. rgerganov shared resouce providers is the feature you are looking for to correctly model vmw shared datastores | |
| 12:08:39 | rgerganov | even it is a shared RP, we still need to know how to connect to it in order to use it, right | |
| 12:09:03 | rgerganov | there should be some way for virt drivers to obtain connection info | |
| 12:09:04 | sean-k-mooney | rgerganov: yes that info can be sotred in the nova-compute agents config | |
| 12:09:30 | rgerganov | sean-k-mooney, is that some kind of static data? | |
| 12:10:16 | sean-k-mooney | rgerganov: well in the case of the the storage backend for the specific compute node yes its static | |
| 12:11:07 | sean-k-mooney | for libvirt for example by default we use the filesystemd under /var/lib/libvirt but i can configure it to use an nfs share or rbd to ceph by default instead | |
| 12:12:03 | rgerganov | sean-k-mooney, I don't think this will work well for the vmware driver | |
| 12:13:25 | rgerganov | it's hard to believe that after we added so much complexity with placement, RP, allocations, etc we can't solve a simple use case with shared datastores | |
| 12:13:46 | efried | janki https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-pike-upstream-stateful-carbon/36/control_1/n-api.log.2017-10-16-060444.gz Search for 'Floating IP' - confirms that the call is using microversion 2.1. There's a policy check failure a few lines earlier, not sure if that's related. | |
| 12:14:13 | sean-k-mooney | rgerganov: well we could but be dont have the shared resouce providers code merged yet | |
| 12:14:17 | janki | efried, ya. didnot find anything useful in cpu logs | |
| 12:14:37 | efried | rgerganov Is there some kind of API endpoint that can be queried to get the details on the NFS store? | |
| 12:14:41 | janki | efried, ya. policies is 1 of the reason for 404 as per the api doc | |
| 12:14:46 | efried | rgerganov That would be another way to do it. | |
| 12:14:48 | sean-k-mooney | if we did we coudl store info such as the ip adress as a trait on the resouce provider | |
| 12:14:51 | efried | janki Oh, then maybe that's the whole problem. | |
| 12:14:58 | efried | sean-k-mooney Just so. | |
| 12:15:30 | efried | sean-k-mooney Though I'm not sure that's a great idea. More likely you would want to maintain a mapping of RP UUIDs to configs. | |
| 12:15:52 | sean-k-mooney | efried: ya the down side to that is it is a public api... | |
| 12:16:07 | rgerganov | efried, yeah, this is what I am calling "connect info" for resource provider | |
| 12:16:10 | sean-k-mooney | at least i think it is placement is now admin only correct | |
| 12:16:17 | rgerganov | maybe the right term is just "config", idk | |
| 12:16:19 | efried | A trait like CUSTOM_NFS_SHARE_IP_192_168_0_55 is a pretty hideous thought. | |
| 12:17:01 | sean-k-mooney | efried: ya at least that was not an ipv6 address ... | |
| 12:17:12 | efried | Hah, totally | |
| 12:17:59 | efried | sean-k-mooney rgerganov As with PCI devices, we would rather maintain the specifics (like PCI address) external to placement, and have the driver responsible for mapping RP UUIDs to whatever entry in that external data store. | |
| 12:18:28 | janki | efried, POST /compute/v2.1/os-floating-ips => generated 73 bytes in 180 msecs (HTTP/1.1 404) 9 headers in 378 bytes (1 switches on core 0) - "generated" in this line means HTTP package is generated and not FIP is generated right | |
| 12:19:07 | efried | janki I don't know what FIP is. I think that's just saying the response payload was 73 bytes. | |
| 12:19:11 | rgerganov | efried, the problem with this is that it may go out of sync; why not storing this info in placement itself? | |
| 12:19:38 | efried | rgerganov How does it go out of sync? You would have to keep it up to date in placement somehow too, nah? | |
| 12:19:45 | janki | efried, FIP = Floating IP. sorry for short form | |
| 12:20:09 | efried | janki Oh, yeah, it's talking about the response payload. The API message knows nothing about the internals of what you're doing with the call. | |
| 12:22:16 | janki | efried, ack. "Policy check for os_compute_api:os-extended-server-attributes failed with credentials" doesnot mean that os-floating-ip has policy issues and couldnot find anthing in log which suggests that | |
| 12:22:36 | rgerganov | efried, ok maybe keeping them in sync is not a real problem; for me it just feels natural the config info to be associated with the RP and be available through the placement api | |
| 12:22:48 | sean-k-mooney | im sure we can come up with a clean way to store config info for a resouce provders such as adding a new field to the rp itself or reusing the description field | |
| 12:23:06 | rgerganov | sean-k-mooney, +1 | |
| 12:23:31 | sean-k-mooney | that config section though should be admin only | |
| 12:23:34 | janki | efried, but again there is no os-extended-server-attributes API | |
| 12:23:54 | sean-k-mooney | normal users never need to see it only openstack services like the virt diriver | |
| 12:24:40 | efried | janki I really don't understand the test case, but if the policy check failed, is it possible it didn't even get to the API you're concerned about? | |
| 12:24:51 | sean-k-mooney | janki: that api is being moved into the normal server respoce i belive | |
| 12:25:00 | efried | janki The API call in question, if it needed to use a specific microversion, would presumably be coded up to do that. | |
| 12:25:48 | janki | efried, I dont think it needed a specific microversion. I was just checking if that is the reason for failure | |
| 12:26:09 | efried | janki Okay, well, it sounds like the policy is the first thing to sort out. | |
| 12:27:19 | sean-k-mooney | janki: this might be of interest to you https://review.openstack.org/#/c/508101/5/specs/queens/approved/api-extensions-policy-removal.rst | |
| 12:28:02 | sean-k-mooney | you will see on line 75 the proposal is to add the extended attributes to the get server responce | |
| 12:29:57 | janki | sean-k-mooney, ya. looks like os-extended-server-attributes is a collection of REST APIs exposed by Nova. right? | |
| 12:30:33 | sean-k-mooney | janki: it was technically a buch of rest apis exposed by a nova api extention | |
| 12:31:04 | sean-k-mooney | janki: as of pike we nolonger support extending the api like that but the existing extions were more or less kept | |
| 12:31:59 | sean-k-mooney | janki: i think the useful ones will be adopted into the main api and the unsed ones will be drop. i would expect the extended server attribute to be adopted in to the server resouce | |
| 12:32:14 | janki | sean-k-mooney, ohk. so I am getting 404 on os-floating-ips and policy error for os-extended-server-attributes. could this be related? | |
| 12:32:51 | sean-k-mooney | janki: if you query with admin prviliges dose the same happen | |
| 12:33:16 | sean-k-mooney | janki: the reason for the clean up spec is that there are two set of policies applied to the extentions | |
| 12:33:36 | sean-k-mooney | the main api policies and the extention level policies that we are looking to remove | |
| 12:34:02 | janki | sean-k-mooney, I dont have acces to the setup. Its jenkins build. But I think the naswer is no. there is 1 more variable in policy.json "os_compute_api:os-floating-ips": "rule:admin_or_owner" so there are not related | |
| 12:36:45 | sean-k-mooney | dumb question im assumeing the setup is using neutron, im not sure how the extentions were enabled in the past but if it was using nova-network you would have no floating-ips | |
| 12:37:37 | alex_xu | janki: it looks like the floating pool isn't there | |
| 12:38:22 | janki | alex_xu, yes. because the API that creates floating ip is failing | |
| 12:39:24 | alex_xu | janki: ok, great, you already know that, I didn't read the full chat log yet :) | |
| 12:40:29 | janki | alex_xu, I am trying to figure out the reason for the API to fail. I checked, with help of efried, that microversion used is 2.1 (< 2.36). | |
| 12:42:11 | janki | alex_xu, also there is policy check failure for os-extended-server-attributes in log but no such failure for os-floating-ips | |
| 12:44:33 | alex_xu | janki: they are sounds unrelated | |
| 12:45:09 | janki | alex_xu, Ya. initially I thought they are but looking at sample policy file, I do think the same | |
| 12:48:22 | openstackgerrit | Jan Zerebecki proposed openstack/nova master: Only log not correcting allocation once per period https://review.openstack.org/508262 | |
| 12:50:53 | openstackgerrit | Jan Zerebecki proposed openstack/nova master: Only log not correcting allocation once per period https://review.openstack.org/508262 | |
| 12:55:22 | alex_xu | janki: looks like you needn't worry about that policy check failure, the log will be emitted each time when the user doesn't have the permission | |
| 12:55:33 | alex_xu | and that log looks like annoying | |
| 12:56:36 | janki | alex_xu, well then, the next step is look into tempest.conf | |
| 13:00:45 | alex_xu | nova api meeting is running at #openstack-meeting-4 | |
| 13:01:34 | mriedem | sdague: johnthetubaguy: can you take a look at this ocata-only fix? https://review.openstack.org/#/c/512406/ turns out we backported something awhile ago that requires some special handling based on the version of libvirt you're running | |
| 13:03:39 | gmann | janki: what is error actually, i can help from tempest.conf side | |
| 13:06:28 | janki | gmann, so I am running tempest for ODL + Pike setup and https://github.com/openstack/tempest/edit/master/tempest/scenario/test_server_basic_ops.py is failing with error https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-pike-upstream-stateful-carbon/36/tempest/tempest_results.html.gz | |
| 13:07:05 | stephenfin | mriedem: Any suggestions on who else I can ask to review https://review.openstack.org/#/c/361140/ in jaypipes' absence? | |
| 13:08:06 | janki | gmann, as per https://developer.openstack.org/api-ref/compute/#create-allocate-floating-ip-address, I verified that microverion is 2.1 and there are no logs about policy failure for os-floating-ips | |
| 13:08:28 | mriedem | stephenfin: i think that dansmith guy is pretty smart | |
| 13:09:00 | janki | gmann, next is to check tempest.conf for proper flag settings | |
| 13:10:32 | gmann | janki: its not policy related, what FIP pool you configured on nova> | |
| 13:11:29 | janki | gmann, I didnot manually configure anything. These logs are from jenkins build. Which file would those be in? I can dig that file up | |
| 13:11:54 | stephenfin | mriedem: I agree: dansmith is a super smart guy who'd surely love to review https://review.openstack.org/#/c/361140/ | |
| 13:11:57 | gmann | janki: sure in api meeting, ll ping/debug after that | |
| 13:12:18 | janki | gmann, ack. | |
| 13:13:08 | stephenfin | Then again, if jaypipes is back from the dead, he might also like to review https://review.openstack.org/#/c/361140/ ... | |
| 13:13:14 | stephenfin | No rest for the weary :) | |