Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-08-16
08:51:32 sean-k-mooney sapd1: well i think this is a nova issue rather then octavia. i cant unfortunetly make nova care about sriov but enough of our users do that we try to support it :)
08:51:40 tssurya so only certain tenats end up in certain cells
08:51:49 sean-k-mooney mriedem: oh i had not seen it but i will take a look
08:53:54 sean-k-mooney sapd1: i started looking into this again last night. i think i know a slightly hacky way to fix this in the libvirt virt diriver. i am look to see if we can adress the missing logic in the compute manager instead as most of the code should not depend on the driver
08:54:03 mriedem tssurya: yeah i know cern mapped projects to cells,
08:54:16 mriedem oh ok so basic flavors are global,
08:54:34 mriedem then per-cell flavors have tenant access for the tenants in those cells
08:54:55 mriedem do you have any idea how many projects exist in CERN?
08:54:56 mdbooth I'm trying to enable debug logs in a functional test. Thought it would be as simple as self.flags(debug=True), but apparently not.
08:55:56 sean-k-mooney tssurya: are you using dansmith's prefilter to do the tenat to cell mapping or something else?
08:56:14 mriedem mdbooth: OS_DEBUG=True
08:56:23 lyarwood OS_DEBUG=1 tox -e functional [..]
08:56:26 lyarwood yeah sorry
08:56:39 mdbooth mriedem lyarwood: Thanks! You saved me some rtfs :)
08:57:59 mriedem sean-k-mooney: wouldn't most of the work for sriov attach be doing the RT claim for a pci_request and if that fails, we fail the attach request
08:58:16 mriedem and then set the port host binding details as you mentioned
08:58:24 mriedem most of ^ is probably already done in the port binding code
08:59:32 openstackgerrit jichenjc proposed openstack/nova master: Add zvm admin intro and hypervisor information https://review.openstack.org/533125
08:59:33 openstackgerrit jichenjc proposed openstack/nova master: Add zvm CI information https://review.openstack.org/533512
09:01:17 sean-k-mooney mriedem: ya. which is why i think we can do all the work from the compute manager rather then the virt driver
09:01:26 sapd1 sean-k-mooney: Will you have plan to re-implement this feature? :D
09:03:30 sean-k-mooney mriedem: sapd1 these are my notes form last night https://etherpad.openstack.org/p/sriov-attach
09:04:11 openstackgerrit Merged openstack/nova master: VMware: add missing os types in vSphere sdk 6.5 https://review.openstack.org/591771
09:05:00 openstack Launchpad bug 1708433 in OpenStack Compute (nova) "Attaching sriov nic VM fail with keyError pci_slot" [Medium,In progress] - Assigned to Matt Riedemann (mriedem)
09:05:00 sean-k-mooney sapd1: https://bugs.launchpad.net/nova/+bug/1708433 was opened by my old team as we had a customer request to enable sriov attach generically
09:05:38 sean-k-mooney sapd1: that was before my team was pulled off upstream work for internal projects. i have now move to redhat i might be able to work on this but not sure
09:05:52 sean-k-mooney i would have to ask
09:10:05 sean-k-mooney mriedem: there are a few more edge cases that we would have to handel. e.g. vm runs out of pci ports, updating instace request spec, numa affintiy and updating placement allocation would be the main ones. a spec is the best way to capture all the edgecases
09:11:14 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Remove the deprecated API extensions policies https://review.openstack.org/586872
09:12:37 sapd1 sean-k-mooney: good point. Because I am not a developer, So I can't help you. Because nova souce code is too complicated
09:13:03 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Remove the deprecated API extensions policies https://review.openstack.org/586872
09:13:09 mriedem sean-k-mooney: that's why there was a spec :)
09:13:21 mriedem but it's outdated by now probably
09:13:56 mriedem https://review.openstack.org/#/c/139910/
09:14:54 sean-k-mooney yes i remember. i think we referenced that spec in the bug or it was pointed out at least
09:16:05 sean-k-mooney sapd1: well if this would have been your first topic in nova, it is proably in one of the most complicated parts.
09:16:13 cdent mriedem: I left a comment on https://review.openstack.org/#/c/592285/ trying to get more info about what is actually being fixed, because it's not clear what the actual problem is. Do you have more info?
09:17:04 sean-k-mooney sapd1: mriedem am i can write a spec for this and submit it if you like. but i will have to check internally if i can do more then that and impelmnte it assumeing it was approved.
09:17:44 mriedem cdent: yeah yikun was going to update the ML thread on cells perf testing,
09:17:55 mriedem but he needed to monkey_patch the time module
09:18:02 sean-k-mooney there are some other topics related to this i was planning on writeing proposals for so its somwhat alinged with what i was going to suggest working on anyway.
09:18:26 sapd1 sean-k-mooney: I will vote for you. :D
09:19:26 openstackgerrit jichenjc proposed openstack/nova master: Add zvm admin intro and hypervisor information https://review.openstack.org/533125
09:19:27 openstackgerrit jichenjc proposed openstack/nova master: Add zvm CI information https://review.openstack.org/533512
09:20:01 mriedem sean-k-mooney: i'm indifferent to sriov port attach capabilities in nova
09:20:34 cdent mriedem: would be interesting to see/know what ways nova api (as a wsgi app) needs eventlet at all
09:20:55 cdent or it just happens to be picking it up accidentally and getting into trouble
09:22:04 sean-k-mooney mriedem: for me originally it was parity thing. e.g. it was yet one more thing you had to be aware of did not work with sriov.
09:22:30 sean-k-mooney mriedem: it became more important with hardware offloaded ovs
09:22:33 mriedem cdent: i left a comment
09:22:38 cdent thanks
09:22:50 mriedem sean-k-mooney: sure i get that
09:22:59 mriedem much like volume-backed instances with 33% of the compute API
09:23:08 mriedem or 80% of the compute API with non-libvirt drivers :)
09:27:20 sean-k-mooney mriedem: :) the main thing i was thinking of working on in stien was a generic device management framework for nova. to unify handeling of all passthough devices via a form of flavor decompostion. that is a lot of work though and im not sure its a high enough proirty to try and do in stien.
09:28:02 sean-k-mooney basicaly the openstack server create --device stuff i mentioned in vancouver.
09:30:30 mdbooth Ah, ha! My test is non-deterministic because the fake image service returns images as a list from a dict, hence in non-deterministic order. If the first one returned defined an external kernel then the libvirt driver uses the Flat imagebackend explicitly, which I'm not mocking in the Rbd tests.
09:30:57 mdbooth That was satisfying :)
09:31:11 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Merge extended availability zone response into server controller https://review.openstack.org/502859
09:31:23 sean-k-mooney mdbooth: if you ran it with python3 the dict has determistic ordering
09:31:34 mdbooth sean-k-mooney: Interestingly I was running py3
09:31:43 mdbooth Although py35, which doesn't have deterministic dict order
09:31:52 mdbooth iirc that 36 or 37?
09:32:06 sean-k-mooney py36 i think
09:33:02 sean-k-mooney mdbooth: that said the iteration order is fixed not sure about calls to values or keys but would assume they return ordered lists also
09:33:16 mdbooth Yeah, I'd assume so
09:33:33 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Merge config drive extension response into server controller https://review.openstack.org/584223
09:33:58 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Merge extended server attributes extension response https://review.openstack.org/584590
09:35:20 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Merge extended server attributes extension response https://review.openstack.org/584590
09:35:44 mriedem it's not sorted https://github.com/openstack/nova/blob/master/nova/tests/unit/image/fake.py#L164
09:35:45 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Merge keypair extension response into server view builder https://review.openstack.org/584748
09:35:55 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Merge server usage extension response into server view builder https://review.openstack.org/585262
09:36:18 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Merge security groups extension response into server view builder https://review.openstack.org/585475
09:36:20 tssurya mriedem: correct basic flavors (larger flavors users open special request) are available everywhere, special flavors for special projects in special cells where those tenants have access, by default projects are not mapped (when they are normal tenants/personal projects) and they land in a set of default cells. we have around ~4500 projects roughly which depending on where the resources are allocated,
09:36:22 tssurya get mapped to that cell
09:36:55 tssurya sean-k-mooney: we use the pre-filter from dansmith for scheduling based on the same criteria: projects mapped to cells/aggregates
09:37:14 tssurya so that placement returns only those nodes
09:38:02 sean-k-mooney tssurya: cool. i was wondering if ye were still using the oldway with the tenant affinity/isolation filter but that does not work right post palcement
09:38:46 tssurya right
09:39:56 sean-k-mooney tssurya: how have ye found the perfromce with the prefilter + placement vs cellsv1 and no placement. is it working well for ye?
09:41:50 tssurya sean-k-mooney: not really that good, we use the "limits" thing for limiting placement results for getting a decent performance
09:42:09 tssurya we are trying to improve the scheduling part performance wise, mriedem had some patches
09:42:43 tssurya but we have not seen much improvement in production after applying those
09:46:02 sean-k-mooney thats unfortunet. there is still alot of work going into makeing schdulign faster in general. hopefully the performacne is adaquit for now.
09:46:24 mriedem i imagine the big chunk of the per-request scheduling time is taken in the DB queries which iterate all selected compute nodes and then pull all instance UUIDs on each node
09:46:29 mriedem across all cells
09:46:42 mriedem that could be optimized into a single big join per cell probably
09:47:25 mriedem because CERN isn't using the track_instance_changes config option right?
09:47:34 mriedem where the computes report instance per host changes to the scheduler
09:47:35 mriedem ?
09:48:34 mriedem this method https://github.com/openstack/nova/blob/master/nova/scheduler/host_manager.py#L710
09:48:46 mriedem specifically https://github.com/openstack/nova/blob/master/nova/scheduler/host_manager.py#L735
09:49:00 mriedem that queries the cell db per compute host to find the list of instance uuids on that host
09:50:29 sean-k-mooney mriedem: for the cern case you really only need to fine out what cell the project is mapped too right? so if we had placement aggreates per cell we would just need to add a member of to the placement query for the cells assocated with the tenant and let placement do all the work
09:51:13 mriedem sean-k-mooney: CERN already has that
09:51:19 tssurya mriedem: yea the time consumption is in the pulling of all instances info
09:51:22 mriedem they have tenants mapped to host aggregates which are mapped to resource provider aggregates
09:51:26 mriedem and that does the pre-filtering
09:51:46 sean-k-mooney mriedem: oh ok cool
09:52:07 mriedem they use this stuff https://docs.openstack.org/nova/latest/admin/configuration/schedulers.html#aggregates-in-placement

Earlier   Later