| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-03-21 | |||
| 17:41:43 | bauzas | let's go for a RECLONE=yes | |
| 17:45:01 | NobodyCam | fyi incase anyone ever runs in to this again: issue was bad rabbit ip in nova_api/cell_mappings.transport_url | |
| 17:52:18 | dansmith | efried: cdent: what's the status on this eventlet monkeypatch thing? I saw some discussion this morning, but when I looked the other day it had changed since my review for reasons I wasn't sure about | |
| 17:52:37 | efried | dansmith: tldr we think it's good, but wait for after rc1 | |
| 17:53:03 | dansmith | efried: ah right, because they said it was okay in train, cool | |
| 17:53:04 | efried | dansmith: was gonna bug you tomorrow (after rc1 cut) to re-up your +2 | |
| 17:53:18 | efried | well, they want it backported. We just don't want to drop it into rc1 | |
| 17:53:27 | dansmith | right | |
| 17:59:19 | melwitt | mriedem_grooming, efried: thanks for getting the cycle highlights updated | |
| 17:59:34 | efried | yw | |
| 18:09:38 | efried | our last rc1 patch just hit the 2h mark unstarted in the check queue :( | |
| 18:29:04 | openstackgerrit | Merged openstack/nova master: Move libvirt calculation of machine type to utils.py https://review.openstack.org/644554 | |
| 18:41:40 | openstackgerrit | melanie witt proposed openstack/nova master: Move creation of rpcapi.ComputeAPI object from __init__ https://review.openstack.org/644998 | |
| 18:44:16 | openstackgerrit | melanie witt proposed openstack/nova master: Move create of ComputeAPI object in websocketproxy https://review.openstack.org/644998 | |
| 18:45:36 | jroll | afaik there is no way to limit the project(s) which can boot an instance on a given set of compute nodes, right? surely someone has proposed this before, is it a hard no? | |
| 18:49:11 | dansmith | tenant isolation yo! | |
| 18:50:20 | dansmith | jroll: new placementy way to do it: https://specs.openstack.org/openstack/nova-specs/specs/rocky/implemented/placement-req-filter.html | |
| 18:50:36 | dansmith | jroll: also an old filter-based approach (google for filter_tenant_id) | |
| 18:50:36 | jroll | oooo | |
| 18:51:53 | jroll | dansmith: this should work perfectly, thanks! | |
| 18:52:13 | dansmith | finally, FINALLY, I do something to make jroll happy. | |
| 18:52:20 | jroll | it only took 5 years | |
| 18:52:32 | dansmith | better late than never | |
| 18:53:32 | sean-k-mooney | the tenant isolation filter has some edgecases | |
| 18:54:24 | sean-k-mooney | the main one being it limits the set of host a project can boot instnace on but does not prevent other project form booting instnaces on those same host | |
| 18:54:50 | dansmith | sean-k-mooney: you should read that spec | |
| 18:55:21 | dansmith | there's a provision (unlike the filter approach) that will reject if the tenant is not confined, so you can avoid that situation | |
| 18:56:05 | mriedem | melwitt: did you want to re-propose the counting quotas spec so we can line that up for landing early in train? | |
| 18:56:09 | sean-k-mooney | oh cool. well it was not really a edgecase in the old filter i guess as a desing choice | |
| 18:56:13 | mriedem | the code i mean, after the spec is re-approved | |
| 18:56:26 | sean-k-mooney | dansmith: nice to know the placement version accounts fot that | |
| 18:56:38 | melwitt | mriedem: oh, yes, I do. thanks for reminding me | |
| 18:57:08 | melwitt | let me do it now so I don't forget a third time | |
| 18:58:54 | jroll | dansmith: the use case mentions segregate users into cells, but it sounds like the solution only depends on aggregates, not cells, am I reading that right? | |
| 18:59:58 | mriedem | the scheduler does might be more clear than the spec https://docs.openstack.org/nova/latest/admin/configuration/schedulers.html#tenant-isolation-with-placement | |
| 19:00:01 | mriedem | *docs | |
| 19:00:32 | jroll | mriedem: thanks, that answers it | |
| 19:00:39 | dansmith | jroll: correct | |
| 19:01:40 | sean-k-mooney | jroll: the original usecase was a cellv1 -> cellv2 perfromcne regration which is why the spec focused on cells but ya it just needs aggreates | |
| 19:01:54 | jroll | sean-k-mooney: right, figured as much | |
| 19:01:57 | openstackgerrit | melanie witt proposed openstack/nova-specs master: Re-propose count quota usage from placement https://review.openstack.org/645302 | |
| 19:03:12 | amodi | dansmith: mriedem additional question related to this: im currently developing a test case on this, im getting a Scheduling error, No Hosts found. Is tempest generated project/tenant not able to execute this? cz i can run the same from admin project and it works | |
| 19:03:52 | amodi | sorry this error: "No valid host was found. Scheduling failed: No hosts available for tenant" | |
| 19:04:51 | dansmith | amodi: if the tenant is not allowed to any aggregate and the "require" flag is set, then that's expected yes | |
| 19:05:42 | dansmith | sean-k-mooney: not really performance regression, but replicating the inate behavior/feature of being able to schedule to specific cells, yes | |
| 19:05:52 | amodi | yes thats set placement_aggregate_required_for_tenants=True | |
| 19:06:47 | amodi | dansmith: i do set the filter_tenant_id metadata to the aggregate | |
| 19:07:12 | dansmith | amodi: and is the tempest tenant mentioned in there? | |
| 19:07:50 | dansmith | amodi: I'd have to see what all you have and the logs, but you can see from the functional tests how it's supposed to work (and that it does) so it must be config-related | |
| 19:07:51 | mriedem | amodi: why do we need a tempest test for that placement request filter? we have in-tree functional testing which should be sufficient | |
| 19:08:10 | sean-k-mooney | dansmith: well cern were using the tenant isolation filter to window a set of host down to a cell based on the teant then runing the rest of the filters on that reduced set or somthing like that right. so when the went the cells v2 + placement they had some scaling issue where by if the use the placement limit feature they could get no valid host so you impmented this feature to give the same | |
| 19:08:12 | sean-k-mooney | fucntionaliy the previously had in a placement native way. at lest that is more or less my memory of events | |
| 19:08:31 | amodi | dansmith: yeah the filter_tenant_id= $tempest_generated_tenant_id | |
| 19:09:45 | dansmith | sean-k-mooney: because they look for a small number of placement results it was just not working, not really scale-related, although this does/should perform better yes | |
| 19:10:23 | dansmith | amodi: yep, well, should be working then, if the host(s) are in that aggregate.. see the functional test for details :) | |
| 19:10:38 | amodi | mriedem: well im checking functional tests | |
| 19:10:41 | sean-k-mooney | dansmith: didnt the look at a small number of host because when they did not limit the got too much data back form placment and it was slowing down schduling | |
| 19:10:41 | amodi | dansmith: sure | |
| 19:11:44 | dansmith | sean-k-mooney: okay, yes, they limit placement to a very small number for performance reasons. And then that affects the correctness of the multi-tenant filter. | |
| 19:11:51 | sean-k-mooney | basically the "this tiny vm fits on all host so return all host proablem" | |
| 19:12:01 | sean-k-mooney | ya | |
| 19:12:16 | dansmith | I'm *so* glad we hashed that out. | |
| 19:13:01 | sean-k-mooney | sorry but this is likely to become more important in an edge cloud deployment too | |
| 19:13:19 | sean-k-mooney | or similar issues at anyrate | |
| 19:14:28 | dansmith | most edge cloud configs I know of are set up so tenants can boot things in geo-local areas, and thus limiting them to specific aggregates by project isn't as common | |
| 19:15:00 | dansmith | unless maybe you have really big edge sites with overlapping vertical AZs and horizontal non-AZ aggregates or something | |
| 19:16:30 | sean-k-mooney | i was more thinking of the schduler prefilters in general rather then this one implmenation. as the number of host in singel cloud grows with edge deployment the mechanium you intoduce to solve this problem will like become more useful going forward | |
| 19:18:25 | dansmith | indeed, and hence my desire to do the image type filtering in placement instead of after the fact | |
| 19:19:35 | cdent | prefilters++ | |
| 19:21:32 | mriedem | speaking of, i think i've got a hack for pre-filtering hosts that support multiattach volumes now | |
| 19:22:23 | sean-k-mooney | mriedem: im also working on a spec for prefilting host based on virtual device modesl they support | |
| 19:22:33 | mriedem | let us all work on pre-filters | |
| 19:23:08 | sean-k-mooney | i mean they are the new placement that will solve all proablems that and traits | |
| 19:24:47 | sean-k-mooney | this is what i have so far but ill have a first draft up on gerrit tommorow or monday | |
| 19:24:49 | sean-k-mooney | https://etherpad.openstack.org/p/image-metadata-prefilters | |
| 19:25:40 | dansmith | sean-k-mooney: and is this for places where you've passed a port in and it specifies some host model? | |
| 19:25:59 | dansmith | I would expect the normal "just hook me up" request to not need such filtering | |
| 19:27:11 | sean-k-mooney | no this is based on image metadata so if you select an image that says it ned hw_vif_type=e1000 then we should not schdule you to a hyperv host that does not support it | |
| 19:27:14 | dansmith | yeah | |
| 19:27:14 | dansmith | oh from | |
| 19:33:35 | sean-k-mooney | dansmith: i havent spell checked anything so before i submit it as a spec ill be running it true grammerly | |
| 19:34:00 | dansmith | sean-k-mooney: oh I don't think you need to spell check it | |
| 19:34:14 | dansmith | sean-k-mooney: =P | |
| 19:35:19 | sean-k-mooney | :) your right i just need to add stephenfin or jay to the review and wait for the wall of comments | |
| 19:36:03 | dansmith | hah, make gerrit OOM | |
| 19:40:48 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Pre-filter hosts based on multiattach volume support https://review.openstack.org/645316 | |
| 19:40:55 | mriedem | efried: cdent: dansmith: sean-k-mooney: ^ pretty hacky but it's also pretty easy building on top of the requested_resources stuff gibi added in stein | |
| 19:41:12 | dansmith | eeeeeverybody wants into the prefilter game | |
| 19:41:40 | cdent | some people can follow trends but only dansmith can make them possible | |
| 19:41:49 | mriedem | psh i was talking about this way back in https://review.openstack.org/#/c/538498/ | |
| 19:42:41 | cdent | now that efried is ptl do we have to roll to resolve disputes? | |
| 19:42:43 | mriedem | gonna go for a walk with the mrs while (1) it's nice out and (2) the neighbors and their annoying kids aren't outside mingling | |
| 19:47:31 | openstackgerrit | Merged openstack/python-novaclient master: Remove deprecated options https://review.openstack.org/642312 | |
| 19:52:36 | efried | I've long thought the center of the square of tables would make a good "cage". | |
| 19:55:27 | mnaser | should _local_delete() be called with instance.host == None ? | |
| 19:56:01 | mnaser | "instance's host None is down, deleting from database" with "Ignoring volume cleanup failure due to 'NoneType' object has no attribute 'get'" afterwards, i'm assuming that volume clean up has to do with the fact that host is None | |
| 19:57:06 | mnaser | https://github.com/openstack/nova/blob/5ca858eaa72acd0513e27a4c9518980b769f5d6e/nova/compute/api.py#L1967 | |
| 19:57:19 | mnaser | i can only imagine that not possibly being evaluated properly because its hitting the local delete path | |
| 19:57:24 | sean-k-mooney | mnaser: proably not | |
| 19:57:50 | sean-k-mooney | instance.host==None would only happen if the inscate was in cell0 right | |