Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-12-10
17:57:32 lbragstad it's related to https://review.openstack.org/#/c/619260/3/oslo_policy/policy.py
17:57:59 mriedem nova's RequestContext calls check_is_admin during init
17:58:02 mriedem which is for every request
17:58:34 lbragstad does that call enforce()?
17:59:23 mriedem calls return _ENFORCER.authorize('context_is_admin', target, credentials)
17:59:55 mriedem which calls enforce
17:59:59 lbragstad ah - it does
18:00:00 lbragstad right
18:00:04 lbragstad makes sense
18:00:41 lbragstad oh - weird, target is being overloaded as credentials?
18:00:59 lbragstad i suppose, is it too early in the pipeline for nova to know what the actual target is?
18:01:04 mriedem yeah
18:01:08 lbragstad huh
18:01:13 lbragstad interesting
18:01:28 mriedem remember this? https://review.openstack.org/#/c/564349/
18:03:12 lbragstad oh - yeah... kinda
18:04:32 lbragstad looks like that would still address the issue
18:05:00 mriedem this is where we get _DeprecatedPolicyValues https://github.com/openstack/oslo.context/blob/0daf01065d1d51694e06aaecb3dcf4dcc78710fe/oslo_context/context.py#L318
18:05:04 mriedem passing that in as the target
18:05:23 lbragstad right
18:05:38 lbragstad you should be able to pass context objects as credentials
18:05:59 lbragstad but we don't really link anything about target to context
18:06:00 lbragstad at least in oslo.policy
18:06:28 mriedem seems we should just pass target={}
18:06:30 mriedem since we don't know
18:07:06 lbragstad (we wanted to use context objects as a replacement for a dictionary called credentials since it makes things easier for services using oslo.policy - instead of just assuming service developers know what to put in credentials or how the internals of oslo.policy parses it)
18:07:37 lbragstad and it relays a bunch of useful information about the user making the request - which is what credentials was trying achieve anyway
18:08:22 lbragstad but yeah - if you don't know anything about the target, an empty dictionary would seem safer than overloading target as a set of context attributes
18:12:00 lbragstad for example - in keystone we protect APIs like this https://github.com/openstack/keystone/blob/master/keystone/api/users.py#L137
18:12:31 lbragstad we look at the context object associated with the request and generate "credentials" https://github.com/openstack/keystone/blob/master/keystone/common/rbac_enforcer/enforcer.py#L380
18:12:44 lbragstad and then pass that to oslo.policy https://github.com/openstack/keystone/blob/master/keystone/common/rbac_enforcer/enforcer.py#L81-L82
18:18:10 openstackgerrit Dan Smith proposed openstack/nova master: Make service.get_minimum_version_all_cells() cache the results https://review.openstack.org/623283
18:18:10 openstackgerrit Dan Smith proposed openstack/nova master: Make compute rpcapi version calculation check all cells https://review.openstack.org/623284
18:19:33 openstackgerrit Georg Hoesch proposed openstack/nova master: refactor get_console_output() for console logfiles https://review.openstack.org/575735
18:38:19 dtrainor Howdy. I'm trying to troubleshoot a failed deployment that ultimately results in "Message: No valid host was found. There are not enough hosts available., Code: 500". Digging deeper, it looks like RamFilter is returning 0 hosts because the hosts appears to return a value of 0 MB RAM http://paste.openstack.org/show/736921/
18:38:35 etp Do max_placement_results and host aggregates/AggregateInstanceExtraSpecsFilter work together or am I looking at configuration error? Say cloud has 100 hypervisors, max_placement_results = 10, two host aggregates 50 hypervisors each, flavor extra specs matching latter aggregate with hypervisors 51-100
18:38:59 dtrainor the nodes get properly introspected. the introspection data is good. there may be a bug in the Ravello hypervisor that this is being deployed on which ahs been known to return 0MB if the host BIOS returned a negative number (read something about an overcommit ratio sometimes causing this)
18:42:02 dtrainor I'm using all default values for this downstream queens deployment. The flavors show a minimum requirement of 4096MB, that's the only place I can find this number occurring.
18:50:07 dansmith etp: 10 is small enough that you're likely to not get back any hosts in the aggregate you're looking for yeah
18:50:27 dansmith etp: unless you're enabling a request filter in nova that helps to ensure that you do
18:50:38 dansmith which is currently az or tenant filtering
18:53:58 TheJulia Hopefully someone can help jog my memory... A long time ago I remember there was a discussion of how nova represents a baremetal node being torn down. I believe the state then and now is that the instance is generally "ACTIVE" until the node is actually deleted, and I think the motivation behind that was resource accounting wise. Does anyone remember this discussion (it would have been 2+ years ago I think)
18:57:42 dansmith TheJulia: the instance goes away immediately, AFAIK, as that's the way delete is supposed to work, but nova tries to make the node not schedule-able until the cleaning is done
18:58:06 mriedem dtrainor: you shouldn't need the RamFilter in queens
18:58:13 TheJulia I've got a QE person reporting that they are seeing it sit in "ACTIVE" until cleaning is over
18:58:17 dansmith the strategy we proposed for doing this betterly was to set the reserved amount of resource equal to the total for a node so that it appeared to not have any room until after it's cleaning
18:58:25 mriedem dtrainor: the placement service does the filtering on MEMORY_MB
18:58:30 TheJulia and I thought there was some reason for that, just struggling to decompress those memories
18:58:37 dansmith TheJulia: okay I could be wrong, but I don't think that's what is supposed to happen
18:59:33 etp dansmith: tnx, we have only one az atm and aggregate should be for all tenants so I think request filter doesn't help here, I'll revert config back to default
18:59:39 mriedem dtrainor: you're scheduling to baremetal nodes there?
19:00:10 mriedem dtrainor: if those are baremetal nodes, you should read up on https://docs.openstack.org/ironic/latest/install/configure-nova-flavors.html
19:01:43 mriedem dtrainor: i guess the Core/Ram/Disk filters weren't deprecated until stein but they should have been since pike i think https://docs.openstack.org/nova/latest/admin/configuration/schedulers.html#ramfilter
19:01:55 dansmith TheJulia: self.ironicclient.call("node.set_provision_state", node.uuid,
19:01:56 dansmith "deleted")
19:02:09 dansmith TheJulia: does that take a long time?
19:02:26 TheJulia dansmith: that can take hours upon hours depending on the hardware the operator configuration
19:02:40 dansmith TheJulia: yeah, looks like that has a poll loop after it
19:02:43 TheJulia dansmith: to be precise, that should return immediately
19:02:48 TheJulia but the background it runs for a long time
19:02:50 dtrainor sorry, i'm back. mriedem, I'm familiar with that doc, yep. I'm familiar with how it's scheduled and can even make this deployment work on different platforms, e.g. on top of different hardware. that's what leaves me puzzled.
19:03:46 dansmith TheJulia: yeah, so I guess it does have to go through a full cleaning before the instance appears to actually go away
19:03:53 dtrainor VCPU, MEMORY_MB, and DISK_GB are all set to 0 to disable scheduling based on properties for a bare metal flavor. i am using bare metal nodes with the intent of using the default flavor properties such as these
19:04:04 dansmith TheJulia: but the instance shouldn't be just ACTIVE, it should be ACTIVE with a task-state of "deleting" at least
19:04:08 TheJulia mriedem: I'm fairly sure somewhere there was a statement saying that it would no longer work and was deprecated, but that was a long time ago since we kept compatability for a while
19:04:43 TheJulia dansmith: that is what it reports now afaik
19:06:34 mriedem dtrainor: ok, well i can tell you the IronicHostManager, baremetal filters and core/ram/disk filters have all been deprecated and can cause issues when trying to schedule to baremetal nodes
19:06:38 mriedem so you shouldn't be using those
19:06:48 mriedem and should be scheduling based on custom resource classes
19:07:03 dtrainor right, that was my understanding
19:08:27 dtrainor all the nodes are created by default with a resource_class of 'baremetal', which my understanding is, this should match a flavor based on a flavor's properties of e.g. resources:CUSTOM_BAREMETAL='1'
19:08:37 dtrainor again, these all being defaults
19:10:18 dtrainor i have nova-compute and nova-scheduler log files, I don't want to ask for too much but maybe another set of eyes would help me find something that I'm not able to see?
19:10:43 TheJulia dansmith: do you know, off the top of your head, if nova only represents a deletion in progress in the task_status field?
19:10:53 TheJulia dansmith: I mean, in documentation
19:16:24 dansmith TheJulia: from the outside, that's the indication the user has yeah
19:16:32 dansmith TheJulia: normally it happens fairly quick of course
19:16:41 TheJulia yeah
19:18:53 mriedem dtrainor: paste your nova.conf for the scheduler
19:19:40 mriedem dtrainor: in addition, are the cpu/ram/disk values on the flavor overridden to report 0?
19:19:54 mriedem 1
19:20:01 mriedem see, "Another set of flavor properties must be used to disable scheduling based on standard properties for a bare metal flavor:" from https://docs.openstack.org/ironic/latest/install/configure-nova-flavors.html
19:24:41 dtrainor my scheduler section is pretty boring by default with only one change http://paste.openstack.org/show/736924/ reading...
19:25:15 dtrainor yes, the values are set to 0, as this is a default https://github.com/openstack/tripleo-heat-templates/blob/stable/queens/extraconfig/post_deploy/undercloud_post.sh#L124
19:27:05 cdent mriedem or yikun_ : https://github.com/openstack/nova/blob/master/nova/objects/compute_node.py#L213 has trouble when the compute node is 'deleted' and the context hasn't diddled read_deleted
19:27:36 mriedem where are we reading deleting compute_nodes?
19:27:40 mriedem *deleted
19:28:02 dtrainor so by default, a node's resource class is not uppercase, which sounds like it's required per https://docs.openstack.org/ironic/queens/install/configure-nova-flavors.html . I just changed the resource class of a node from 'baremetal' to 'BAREMETAL', I'll try a deployment with that.
19:28:03 cdent this is showing up in the status test that I'm moving for placement fixture: line 265 here https://review.openstack.org/#/c/617941/28/nova/tests/functional/test_nova_status.py
19:28:34 cdent cn2.create() is of a node that is deleted (so that it can be "not seen" during the test)
19:28:42 dtrainor I think the thing that bothers me the most is that this deployment succeeds on other sets of hardware, i'm starting to suspect a bug on the platform on which this is deployed but i can't prove it
19:28:46 cdent I can diddle the context, but wanted to be sure that was the right things to do
19:28:48 mriedem cdent: yeah you didn't copy part of that change from yikun
19:29:19 mriedem cdent: https://review.openstack.org/#/c/613499/16/nova/tests/unit/cmd/test_status.py
19:29:27 cdent thanks
19:29:51 mriedem dtrainor: hmm i'm not sure how you have the RamFilter enabled then
19:30:11 cdent latency is the everything killer
19:30:57 mriedem dtrainor: especially since we removed that from the default enabled filters list in pike https://review.openstack.org/#/c/491854/
19:31:04 mriedem although i see you're using the ironic_host_manager,
19:31:12 mriedem which is also deprecated and you shouldn't need

Earlier   Later