| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-08-16 | |||
| 08:45:46 | Kevin_Zheng | Hmm, that might not work for us, as I mentioned before, we are unable to limit our users to cells | |
| 08:46:24 | tssurya | Kevin_Zheng: yea I remember your comment on that patch, but I guess we still need to find a way to make the data gathering part faster then | |
| 08:47:55 | mriedem | sean-k-mooney: yeah i'm aware of the spec | |
| 08:48:48 | mriedem | sean-k-mooney: i also added you to an sriov-bond spec that our product team is interested in | |
| 08:49:07 | mriedem | tssurya: Kevin_Zheng: that reminds me, | |
| 08:49:29 | mriedem | tssurya: the reason we can't restrict users to cells in our public cloud is because flavors are defined by cell (among other things) | |
| 08:49:41 | mriedem | so if a user wants to use different flavors they can get instances in different cells | |
| 08:49:47 | mriedem | tssurya: how is that handled at CERN? | |
| 08:50:27 | sapd1 | sean-k-mooney: I think nova should care about SR-IOV instead of octavia. | |
| 08:50:35 | tssurya | mriedem: we actually map projects to cells | |
| 08:51:28 | tssurya | the default flavors are available everywhere and then the special flavors are available in certain cells which are then restricted for the tenants as well | |
| 08:51:32 | sean-k-mooney | sapd1: well i think this is a nova issue rather then octavia. i cant unfortunetly make nova care about sriov but enough of our users do that we try to support it :) | |
| 08:51:40 | tssurya | so only certain tenats end up in certain cells | |
| 08:51:49 | sean-k-mooney | mriedem: oh i had not seen it but i will take a look | |
| 08:53:54 | sean-k-mooney | sapd1: i started looking into this again last night. i think i know a slightly hacky way to fix this in the libvirt virt diriver. i am look to see if we can adress the missing logic in the compute manager instead as most of the code should not depend on the driver | |
| 08:54:03 | mriedem | tssurya: yeah i know cern mapped projects to cells, | |
| 08:54:16 | mriedem | oh ok so basic flavors are global, | |
| 08:54:34 | mriedem | then per-cell flavors have tenant access for the tenants in those cells | |
| 08:54:55 | mriedem | do you have any idea how many projects exist in CERN? | |
| 08:54:56 | mdbooth | I'm trying to enable debug logs in a functional test. Thought it would be as simple as self.flags(debug=True), but apparently not. | |
| 08:55:56 | sean-k-mooney | tssurya: are you using dansmith's prefilter to do the tenat to cell mapping or something else? | |
| 08:56:14 | mriedem | mdbooth: OS_DEBUG=True | |
| 08:56:23 | lyarwood | OS_DEBUG=1 tox -e functional [..] | |
| 08:56:26 | lyarwood | yeah sorry | |
| 08:56:39 | mdbooth | mriedem lyarwood: Thanks! You saved me some rtfs :) | |
| 08:57:59 | mriedem | sean-k-mooney: wouldn't most of the work for sriov attach be doing the RT claim for a pci_request and if that fails, we fail the attach request | |
| 08:58:16 | mriedem | and then set the port host binding details as you mentioned | |
| 08:58:24 | mriedem | most of ^ is probably already done in the port binding code | |
| 08:59:32 | openstackgerrit | jichenjc proposed openstack/nova master: Add zvm admin intro and hypervisor information https://review.openstack.org/533125 | |
| 08:59:33 | openstackgerrit | jichenjc proposed openstack/nova master: Add zvm CI information https://review.openstack.org/533512 | |
| 09:01:17 | sean-k-mooney | mriedem: ya. which is why i think we can do all the work from the compute manager rather then the virt driver | |
| 09:01:26 | sapd1 | sean-k-mooney: Will you have plan to re-implement this feature? :D | |
| 09:03:30 | sean-k-mooney | mriedem: sapd1 these are my notes form last night https://etherpad.openstack.org/p/sriov-attach | |
| 09:04:11 | openstackgerrit | Merged openstack/nova master: VMware: add missing os types in vSphere sdk 6.5 https://review.openstack.org/591771 | |
| 09:05:00 | openstack | Launchpad bug 1708433 in OpenStack Compute (nova) "Attaching sriov nic VM fail with keyError pci_slot" [Medium,In progress] - Assigned to Matt Riedemann (mriedem) | |
| 09:05:00 | sean-k-mooney | sapd1: https://bugs.launchpad.net/nova/+bug/1708433 was opened by my old team as we had a customer request to enable sriov attach generically | |
| 09:05:38 | sean-k-mooney | sapd1: that was before my team was pulled off upstream work for internal projects. i have now move to redhat i might be able to work on this but not sure | |
| 09:05:52 | sean-k-mooney | i would have to ask | |
| 09:10:05 | sean-k-mooney | mriedem: there are a few more edge cases that we would have to handel. e.g. vm runs out of pci ports, updating instace request spec, numa affintiy and updating placement allocation would be the main ones. a spec is the best way to capture all the edgecases | |
| 09:11:14 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Remove the deprecated API extensions policies https://review.openstack.org/586872 | |
| 09:12:37 | sapd1 | sean-k-mooney: good point. Because I am not a developer, So I can't help you. Because nova souce code is too complicated | |
| 09:13:03 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Remove the deprecated API extensions policies https://review.openstack.org/586872 | |
| 09:13:09 | mriedem | sean-k-mooney: that's why there was a spec :) | |
| 09:13:21 | mriedem | but it's outdated by now probably | |
| 09:13:56 | mriedem | https://review.openstack.org/#/c/139910/ | |
| 09:14:54 | sean-k-mooney | yes i remember. i think we referenced that spec in the bug or it was pointed out at least | |
| 09:16:05 | sean-k-mooney | sapd1: well if this would have been your first topic in nova, it is proably in one of the most complicated parts. | |
| 09:16:13 | cdent | mriedem: I left a comment on https://review.openstack.org/#/c/592285/ trying to get more info about what is actually being fixed, because it's not clear what the actual problem is. Do you have more info? | |
| 09:17:04 | sean-k-mooney | sapd1: mriedem am i can write a spec for this and submit it if you like. but i will have to check internally if i can do more then that and impelmnte it assumeing it was approved. | |
| 09:17:44 | mriedem | cdent: yeah yikun was going to update the ML thread on cells perf testing, | |
| 09:17:55 | mriedem | but he needed to monkey_patch the time module | |
| 09:18:02 | sean-k-mooney | there are some other topics related to this i was planning on writeing proposals for so its somwhat alinged with what i was going to suggest working on anyway. | |
| 09:18:26 | sapd1 | sean-k-mooney: I will vote for you. :D | |
| 09:19:26 | openstackgerrit | jichenjc proposed openstack/nova master: Add zvm admin intro and hypervisor information https://review.openstack.org/533125 | |
| 09:19:27 | openstackgerrit | jichenjc proposed openstack/nova master: Add zvm CI information https://review.openstack.org/533512 | |
| 09:20:01 | mriedem | sean-k-mooney: i'm indifferent to sriov port attach capabilities in nova | |
| 09:20:34 | cdent | mriedem: would be interesting to see/know what ways nova api (as a wsgi app) needs eventlet at all | |
| 09:20:55 | cdent | or it just happens to be picking it up accidentally and getting into trouble | |
| 09:22:04 | sean-k-mooney | mriedem: for me originally it was parity thing. e.g. it was yet one more thing you had to be aware of did not work with sriov. | |
| 09:22:30 | sean-k-mooney | mriedem: it became more important with hardware offloaded ovs | |
| 09:22:33 | mriedem | cdent: i left a comment | |
| 09:22:38 | cdent | thanks | |
| 09:22:50 | mriedem | sean-k-mooney: sure i get that | |
| 09:22:59 | mriedem | much like volume-backed instances with 33% of the compute API | |
| 09:23:08 | mriedem | or 80% of the compute API with non-libvirt drivers :) | |
| 09:27:20 | sean-k-mooney | mriedem: :) the main thing i was thinking of working on in stien was a generic device management framework for nova. to unify handeling of all passthough devices via a form of flavor decompostion. that is a lot of work though and im not sure its a high enough proirty to try and do in stien. | |
| 09:28:02 | sean-k-mooney | basicaly the openstack server create --device stuff i mentioned in vancouver. | |
| 09:30:30 | mdbooth | Ah, ha! My test is non-deterministic because the fake image service returns images as a list from a dict, hence in non-deterministic order. If the first one returned defined an external kernel then the libvirt driver uses the Flat imagebackend explicitly, which I'm not mocking in the Rbd tests. | |
| 09:30:57 | mdbooth | That was satisfying :) | |
| 09:31:11 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Merge extended availability zone response into server controller https://review.openstack.org/502859 | |
| 09:31:23 | sean-k-mooney | mdbooth: if you ran it with python3 the dict has determistic ordering | |
| 09:31:34 | mdbooth | sean-k-mooney: Interestingly I was running py3 | |
| 09:31:43 | mdbooth | Although py35, which doesn't have deterministic dict order | |
| 09:31:52 | mdbooth | iirc that 36 or 37? | |
| 09:32:06 | sean-k-mooney | py36 i think | |
| 09:33:02 | sean-k-mooney | mdbooth: that said the iteration order is fixed not sure about calls to values or keys but would assume they return ordered lists also | |
| 09:33:16 | mdbooth | Yeah, I'd assume so | |
| 09:33:33 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Merge config drive extension response into server controller https://review.openstack.org/584223 | |
| 09:33:58 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Merge extended server attributes extension response https://review.openstack.org/584590 | |
| 09:35:20 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Merge extended server attributes extension response https://review.openstack.org/584590 | |
| 09:35:44 | mriedem | it's not sorted https://github.com/openstack/nova/blob/master/nova/tests/unit/image/fake.py#L164 | |
| 09:35:45 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Merge keypair extension response into server view builder https://review.openstack.org/584748 | |
| 09:35:55 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Merge server usage extension response into server view builder https://review.openstack.org/585262 | |
| 09:36:18 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Merge security groups extension response into server view builder https://review.openstack.org/585475 | |
| 09:36:20 | tssurya | mriedem: correct basic flavors (larger flavors users open special request) are available everywhere, special flavors for special projects in special cells where those tenants have access, by default projects are not mapped (when they are normal tenants/personal projects) and they land in a set of default cells. we have around ~4500 projects roughly which depending on where the resources are allocated, | |
| 09:36:22 | tssurya | get mapped to that cell | |
| 09:36:55 | tssurya | sean-k-mooney: we use the pre-filter from dansmith for scheduling based on the same criteria: projects mapped to cells/aggregates | |
| 09:37:14 | tssurya | so that placement returns only those nodes | |
| 09:38:02 | sean-k-mooney | tssurya: cool. i was wondering if ye were still using the oldway with the tenant affinity/isolation filter but that does not work right post palcement | |
| 09:38:46 | tssurya | right | |
| 09:39:56 | sean-k-mooney | tssurya: how have ye found the perfromce with the prefilter + placement vs cellsv1 and no placement. is it working well for ye? | |
| 09:41:50 | tssurya | sean-k-mooney: not really that good, we use the "limits" thing for limiting placement results for getting a decent performance | |
| 09:42:09 | tssurya | we are trying to improve the scheduling part performance wise, mriedem had some patches | |
| 09:42:43 | tssurya | but we have not seen much improvement in production after applying those | |
| 09:46:02 | sean-k-mooney | thats unfortunet. there is still alot of work going into makeing schdulign faster in general. hopefully the performacne is adaquit for now. | |
| 09:46:24 | mriedem | i imagine the big chunk of the per-request scheduling time is taken in the DB queries which iterate all selected compute nodes and then pull all instance UUIDs on each node | |
| 09:46:29 | mriedem | across all cells | |
| 09:46:42 | mriedem | that could be optimized into a single big join per cell probably | |
| 09:47:25 | mriedem | because CERN isn't using the track_instance_changes config option right? | |
| 09:47:34 | mriedem | where the computes report instance per host changes to the scheduler | |