| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-03-21 | |||
| 19:09:45 | dansmith | sean-k-mooney: because they look for a small number of placement results it was just not working, not really scale-related, although this does/should perform better yes | |
| 19:10:23 | dansmith | amodi: yep, well, should be working then, if the host(s) are in that aggregate.. see the functional test for details :) | |
| 19:10:38 | amodi | mriedem: well im checking functional tests | |
| 19:10:41 | sean-k-mooney | dansmith: didnt the look at a small number of host because when they did not limit the got too much data back form placment and it was slowing down schduling | |
| 19:10:41 | amodi | dansmith: sure | |
| 19:11:44 | dansmith | sean-k-mooney: okay, yes, they limit placement to a very small number for performance reasons. And then that affects the correctness of the multi-tenant filter. | |
| 19:11:51 | sean-k-mooney | basically the "this tiny vm fits on all host so return all host proablem" | |
| 19:12:01 | sean-k-mooney | ya | |
| 19:12:16 | dansmith | I'm *so* glad we hashed that out. | |
| 19:13:01 | sean-k-mooney | sorry but this is likely to become more important in an edge cloud deployment too | |
| 19:13:19 | sean-k-mooney | or similar issues at anyrate | |
| 19:14:28 | dansmith | most edge cloud configs I know of are set up so tenants can boot things in geo-local areas, and thus limiting them to specific aggregates by project isn't as common | |
| 19:15:00 | dansmith | unless maybe you have really big edge sites with overlapping vertical AZs and horizontal non-AZ aggregates or something | |
| 19:16:30 | sean-k-mooney | i was more thinking of the schduler prefilters in general rather then this one implmenation. as the number of host in singel cloud grows with edge deployment the mechanium you intoduce to solve this problem will like become more useful going forward | |
| 19:18:25 | dansmith | indeed, and hence my desire to do the image type filtering in placement instead of after the fact | |
| 19:19:35 | cdent | prefilters++ | |
| 19:21:32 | mriedem | speaking of, i think i've got a hack for pre-filtering hosts that support multiattach volumes now | |
| 19:22:23 | sean-k-mooney | mriedem: im also working on a spec for prefilting host based on virtual device modesl they support | |
| 19:22:33 | mriedem | let us all work on pre-filters | |
| 19:23:08 | sean-k-mooney | i mean they are the new placement that will solve all proablems that and traits | |
| 19:24:47 | sean-k-mooney | this is what i have so far but ill have a first draft up on gerrit tommorow or monday | |
| 19:24:49 | sean-k-mooney | https://etherpad.openstack.org/p/image-metadata-prefilters | |
| 19:25:40 | dansmith | sean-k-mooney: and is this for places where you've passed a port in and it specifies some host model? | |
| 19:25:59 | dansmith | I would expect the normal "just hook me up" request to not need such filtering | |
| 19:27:11 | sean-k-mooney | no this is based on image metadata so if you select an image that says it ned hw_vif_type=e1000 then we should not schdule you to a hyperv host that does not support it | |
| 19:27:14 | dansmith | yeah | |
| 19:27:14 | dansmith | oh from | |
| 19:33:35 | sean-k-mooney | dansmith: i havent spell checked anything so before i submit it as a spec ill be running it true grammerly | |
| 19:34:00 | dansmith | sean-k-mooney: oh I don't think you need to spell check it | |
| 19:34:14 | dansmith | sean-k-mooney: =P | |
| 19:35:19 | sean-k-mooney | :) your right i just need to add stephenfin or jay to the review and wait for the wall of comments | |
| 19:36:03 | dansmith | hah, make gerrit OOM | |
| 19:40:48 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Pre-filter hosts based on multiattach volume support https://review.openstack.org/645316 | |
| 19:40:55 | mriedem | efried: cdent: dansmith: sean-k-mooney: ^ pretty hacky but it's also pretty easy building on top of the requested_resources stuff gibi added in stein | |
| 19:41:12 | dansmith | eeeeeverybody wants into the prefilter game | |
| 19:41:40 | cdent | some people can follow trends but only dansmith can make them possible | |
| 19:41:49 | mriedem | psh i was talking about this way back in https://review.openstack.org/#/c/538498/ | |
| 19:42:41 | cdent | now that efried is ptl do we have to roll to resolve disputes? | |
| 19:42:43 | mriedem | gonna go for a walk with the mrs while (1) it's nice out and (2) the neighbors and their annoying kids aren't outside mingling | |
| 19:47:31 | openstackgerrit | Merged openstack/python-novaclient master: Remove deprecated options https://review.openstack.org/642312 | |
| 19:52:36 | efried | I've long thought the center of the square of tables would make a good "cage". | |
| 19:55:27 | mnaser | should _local_delete() be called with instance.host == None ? | |
| 19:56:01 | mnaser | "instance's host None is down, deleting from database" with "Ignoring volume cleanup failure due to 'NoneType' object has no attribute 'get'" afterwards, i'm assuming that volume clean up has to do with the fact that host is None | |
| 19:57:06 | mnaser | https://github.com/openstack/nova/blob/5ca858eaa72acd0513e27a4c9518980b769f5d6e/nova/compute/api.py#L1967 | |
| 19:57:19 | mnaser | i can only imagine that not possibly being evaluated properly because its hitting the local delete path | |
| 19:57:24 | sean-k-mooney | mnaser: proably not | |
| 19:57:50 | sean-k-mooney | instance.host==None would only happen if the inscate was in cell0 right | |
| 19:57:57 | mnaser | or not scheduled yet | |
| 19:58:32 | mnaser | if not instance.host and not may_have_ports_or_volumes <= the first part evaluates to True (as per the obvious warnings) | |
| 19:58:57 | mnaser | may_have_ports_or_volumes = compute_utils.may_have_ports_or_volumes(instance) is probably evaluating the true (say this is a bfv instance) | |
| 19:59:31 | sean-k-mooney | ya just read the code around it for context | |
| 20:00:14 | mnaser | so it may seem that _local_delete() assumes that a host exists | |
| 20:01:45 | sean-k-mooney | i might be blind but wher is _local_delete() called in that function | |
| 20:02:17 | mnaser | https://github.com/openstack/nova/blob/5ca858eaa72acd0513e27a4c9518980b769f5d6e/nova/compute/api.py#L2128-L2129 | |
| 20:02:37 | mnaser | the NoneType get happens inside here somewhere https://github.com/openstack/nova/blob/5ca858eaa72acd0513e27a4c9518980b769f5d6e/nova/compute/api.py#L2174-L2199 | |
| 20:02:39 | sean-k-mooney | ah that funciton is a lot longer then i tought it was | |
| 20:05:26 | sean-k-mooney | proably form compute_utils.get_stashed_volume_connector | |
| 20:06:22 | mnaser | i think so | |
| 20:06:32 | sean-k-mooney | yep https://github.com/openstack/nova/blob/a5e3054e1d6df248fc4c00b9abd7289dde160393/nova/compute/utils.py#L1262 | |
| 20:06:34 | mnaser | will LOG.error() print out the traceback or whats the best way to do it | |
| 20:06:59 | mnaser | sean-k-mooney: yeah but there is 'if connector:' before that | |
| 20:07:16 | mnaser | which if connector is None then it should evaluate to false right off the bat) | |
| 20:07:38 | sean-k-mooney | i think there is a log.exception() prints a nice traceback | |
| 20:11:12 | mnaser | connector = jsonutils.loads(bdm.connection_info).get('connector') | |
| 20:11:16 | mnaser | AttributeError: 'NoneType' object has no attribute 'get' | |
| 20:12:00 | mnaser | the interesting thing is there is an if statement `bdm.connection_info is not None` | |
| 20:12:01 | sean-k-mooney | ok so jsonutils.loads(bdm.connection_info) did not produce a dictionary | |
| 20:12:07 | mnaser | yeah, i will log it's value | |
| 20:12:21 | sean-k-mooney | it could be the empty sting | |
| 20:12:34 | sean-k-mooney | *string | |
| 20:13:48 | mnaser | LOG.debug(bdm.connection_info) => null ? | |
| 20:14:11 | sean-k-mooney | null | |
| 20:14:22 | sean-k-mooney | as in that is what it litally printed | |
| 20:14:23 | mnaser | and then json parsing null gives... None | |
| 20:14:36 | sean-k-mooney | ya that would make sense | |
| 20:14:36 | mnaser | so bdm.connection_info is the literal value 'null' | |
| 20:14:40 | mnaser | weird | |
| 20:14:56 | sean-k-mooney | well i dont know why bdm.connection_info is null | |
| 20:15:33 | mnaser | well i see it in the db equal to null too hmm | |
| 20:16:27 | sean-k-mooney | well the litral null is a valid json type | |
| 20:16:30 | sean-k-mooney | https://www.json.org/ | |
| 20:17:16 | mnaser | hmm | |
| 20:17:19 | sean-k-mooney | so id the db colume is null becuae no conection info exists for this incance becasue it has never been schduled then it makes sense that it woudl be null when we parse it there | |
| 20:17:35 | mnaser | Quota exceeded for 86bbbcfa8ad043109d2d7af530225c72, tried to create 80G volume (6080G of 6144G already consumed).: OverQuota: Quota exceeded for resources: ['gigabytes'] | |
| 20:18:05 | mnaser | so im wondering if when it tries to create bfv vol, it fails, instance tries to delete, poof | |
| 20:18:28 | sean-k-mooney | ya that sound plasible | |
| 20:18:36 | sean-k-mooney | we could jsut add a none check there | |
| 20:19:29 | sean-k-mooney | so store teh value of jsonutils.loads(bdm.connection_info) then do a "if connection_info is not None: ... | |
| 20:19:53 | mnaser | does nova get connection_info right away or does it poll till it gets it | |
| 20:19:55 | mnaser | im assuming it polls | |
| 20:20:26 | mnaser | looks like we do json loading often so it would be good for us to store {} instead of null i guess | |
| 20:20:30 | sean-k-mooney | i wont have connection info untill after it has schduled to a host | |
| 20:21:26 | sean-k-mooney | we proably dont store {} to avoid updating all old instances when we first load them | |
| 20:22:45 | sean-k-mooney | in anycase we proably should not be doing random json loads in the code like that | |
| 20:22:58 | mnaser | its happening a tons tho :X | |
| 20:23:34 | sean-k-mooney | sure but that does not mean its a good thing :) | |
| 20:24:01 | mnaser | ++ yeah i agree | |
| 20:24:08 | mnaser | https://github.com/openstack/nova/blob/6efa3861a5a829ba5883ff191e2552b063028bb0/nova/volume/cinder.py#L575-L605 | |
| 20:25:37 | edmondsw | I don't think openstack-tox-lower-constraints is working properly... I just had to put up a fix for nova-powervm that would appear to be needed for nova as well based on the lower-constraints.txt, but that check is passing... | |
| 20:25:38 | sean-k-mooney | if | |