| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-12-10 | |||
| 18:00:04 | lbragstad | makes sense | |
| 18:00:41 | lbragstad | oh - weird, target is being overloaded as credentials? | |
| 18:00:59 | lbragstad | i suppose, is it too early in the pipeline for nova to know what the actual target is? | |
| 18:01:04 | mriedem | yeah | |
| 18:01:08 | lbragstad | huh | |
| 18:01:13 | lbragstad | interesting | |
| 18:01:28 | mriedem | remember this? https://review.openstack.org/#/c/564349/ | |
| 18:03:12 | lbragstad | oh - yeah... kinda | |
| 18:04:32 | lbragstad | looks like that would still address the issue | |
| 18:05:00 | mriedem | this is where we get _DeprecatedPolicyValues https://github.com/openstack/oslo.context/blob/0daf01065d1d51694e06aaecb3dcf4dcc78710fe/oslo_context/context.py#L318 | |
| 18:05:04 | mriedem | passing that in as the target | |
| 18:05:23 | lbragstad | right | |
| 18:05:38 | lbragstad | you should be able to pass context objects as credentials | |
| 18:05:59 | lbragstad | but we don't really link anything about target to context | |
| 18:06:00 | lbragstad | at least in oslo.policy | |
| 18:06:28 | mriedem | seems we should just pass target={} | |
| 18:06:30 | mriedem | since we don't know | |
| 18:07:06 | lbragstad | (we wanted to use context objects as a replacement for a dictionary called credentials since it makes things easier for services using oslo.policy - instead of just assuming service developers know what to put in credentials or how the internals of oslo.policy parses it) | |
| 18:07:37 | lbragstad | and it relays a bunch of useful information about the user making the request - which is what credentials was trying achieve anyway | |
| 18:08:22 | lbragstad | but yeah - if you don't know anything about the target, an empty dictionary would seem safer than overloading target as a set of context attributes | |
| 18:12:00 | lbragstad | for example - in keystone we protect APIs like this https://github.com/openstack/keystone/blob/master/keystone/api/users.py#L137 | |
| 18:12:31 | lbragstad | we look at the context object associated with the request and generate "credentials" https://github.com/openstack/keystone/blob/master/keystone/common/rbac_enforcer/enforcer.py#L380 | |
| 18:12:44 | lbragstad | and then pass that to oslo.policy https://github.com/openstack/keystone/blob/master/keystone/common/rbac_enforcer/enforcer.py#L81-L82 | |
| 18:18:10 | openstackgerrit | Dan Smith proposed openstack/nova master: Make compute rpcapi version calculation check all cells https://review.openstack.org/623284 | |
| 18:18:10 | openstackgerrit | Dan Smith proposed openstack/nova master: Make service.get_minimum_version_all_cells() cache the results https://review.openstack.org/623283 | |
| 18:19:33 | openstackgerrit | Georg Hoesch proposed openstack/nova master: refactor get_console_output() for console logfiles https://review.openstack.org/575735 | |
| 18:38:19 | dtrainor | Howdy. I'm trying to troubleshoot a failed deployment that ultimately results in "Message: No valid host was found. There are not enough hosts available., Code: 500". Digging deeper, it looks like RamFilter is returning 0 hosts because the hosts appears to return a value of 0 MB RAM http://paste.openstack.org/show/736921/ | |
| 18:38:35 | etp | Do max_placement_results and host aggregates/AggregateInstanceExtraSpecsFilter work together or am I looking at configuration error? Say cloud has 100 hypervisors, max_placement_results = 10, two host aggregates 50 hypervisors each, flavor extra specs matching latter aggregate with hypervisors 51-100 | |
| 18:38:59 | dtrainor | the nodes get properly introspected. the introspection data is good. there may be a bug in the Ravello hypervisor that this is being deployed on which ahs been known to return 0MB if the host BIOS returned a negative number (read something about an overcommit ratio sometimes causing this) | |
| 18:42:02 | dtrainor | I'm using all default values for this downstream queens deployment. The flavors show a minimum requirement of 4096MB, that's the only place I can find this number occurring. | |
| 18:50:07 | dansmith | etp: 10 is small enough that you're likely to not get back any hosts in the aggregate you're looking for yeah | |
| 18:50:27 | dansmith | etp: unless you're enabling a request filter in nova that helps to ensure that you do | |
| 18:50:38 | dansmith | which is currently az or tenant filtering | |
| 18:53:58 | TheJulia | Hopefully someone can help jog my memory... A long time ago I remember there was a discussion of how nova represents a baremetal node being torn down. I believe the state then and now is that the instance is generally "ACTIVE" until the node is actually deleted, and I think the motivation behind that was resource accounting wise. Does anyone remember this discussion (it would have been 2+ years ago I think) | |
| 18:57:42 | dansmith | TheJulia: the instance goes away immediately, AFAIK, as that's the way delete is supposed to work, but nova tries to make the node not schedule-able until the cleaning is done | |
| 18:58:06 | mriedem | dtrainor: you shouldn't need the RamFilter in queens | |
| 18:58:13 | TheJulia | I've got a QE person reporting that they are seeing it sit in "ACTIVE" until cleaning is over | |
| 18:58:17 | dansmith | the strategy we proposed for doing this betterly was to set the reserved amount of resource equal to the total for a node so that it appeared to not have any room until after it's cleaning | |
| 18:58:25 | mriedem | dtrainor: the placement service does the filtering on MEMORY_MB | |
| 18:58:30 | TheJulia | and I thought there was some reason for that, just struggling to decompress those memories | |
| 18:58:37 | dansmith | TheJulia: okay I could be wrong, but I don't think that's what is supposed to happen | |
| 18:59:33 | etp | dansmith: tnx, we have only one az atm and aggregate should be for all tenants so I think request filter doesn't help here, I'll revert config back to default | |
| 18:59:39 | mriedem | dtrainor: you're scheduling to baremetal nodes there? | |
| 19:00:10 | mriedem | dtrainor: if those are baremetal nodes, you should read up on https://docs.openstack.org/ironic/latest/install/configure-nova-flavors.html | |
| 19:01:43 | mriedem | dtrainor: i guess the Core/Ram/Disk filters weren't deprecated until stein but they should have been since pike i think https://docs.openstack.org/nova/latest/admin/configuration/schedulers.html#ramfilter | |
| 19:01:55 | dansmith | TheJulia: self.ironicclient.call("node.set_provision_state", node.uuid, | |
| 19:01:56 | dansmith | "deleted") | |
| 19:02:09 | dansmith | TheJulia: does that take a long time? | |
| 19:02:26 | TheJulia | dansmith: that can take hours upon hours depending on the hardware the operator configuration | |
| 19:02:40 | dansmith | TheJulia: yeah, looks like that has a poll loop after it | |
| 19:02:43 | TheJulia | dansmith: to be precise, that should return immediately | |
| 19:02:48 | TheJulia | but the background it runs for a long time | |
| 19:02:50 | dtrainor | sorry, i'm back. mriedem, I'm familiar with that doc, yep. I'm familiar with how it's scheduled and can even make this deployment work on different platforms, e.g. on top of different hardware. that's what leaves me puzzled. | |
| 19:03:46 | dansmith | TheJulia: yeah, so I guess it does have to go through a full cleaning before the instance appears to actually go away | |
| 19:03:53 | dtrainor | VCPU, MEMORY_MB, and DISK_GB are all set to 0 to disable scheduling based on properties for a bare metal flavor. i am using bare metal nodes with the intent of using the default flavor properties such as these | |
| 19:04:04 | dansmith | TheJulia: but the instance shouldn't be just ACTIVE, it should be ACTIVE with a task-state of "deleting" at least | |
| 19:04:08 | TheJulia | mriedem: I'm fairly sure somewhere there was a statement saying that it would no longer work and was deprecated, but that was a long time ago since we kept compatability for a while | |
| 19:04:43 | TheJulia | dansmith: that is what it reports now afaik | |
| 19:06:34 | mriedem | dtrainor: ok, well i can tell you the IronicHostManager, baremetal filters and core/ram/disk filters have all been deprecated and can cause issues when trying to schedule to baremetal nodes | |
| 19:06:38 | mriedem | so you shouldn't be using those | |
| 19:06:48 | mriedem | and should be scheduling based on custom resource classes | |
| 19:07:03 | dtrainor | right, that was my understanding | |
| 19:08:27 | dtrainor | all the nodes are created by default with a resource_class of 'baremetal', which my understanding is, this should match a flavor based on a flavor's properties of e.g. resources:CUSTOM_BAREMETAL='1' | |
| 19:08:37 | dtrainor | again, these all being defaults | |
| 19:10:18 | dtrainor | i have nova-compute and nova-scheduler log files, I don't want to ask for too much but maybe another set of eyes would help me find something that I'm not able to see? | |
| 19:10:43 | TheJulia | dansmith: do you know, off the top of your head, if nova only represents a deletion in progress in the task_status field? | |
| 19:10:53 | TheJulia | dansmith: I mean, in documentation | |
| 19:16:24 | dansmith | TheJulia: from the outside, that's the indication the user has yeah | |
| 19:16:32 | dansmith | TheJulia: normally it happens fairly quick of course | |
| 19:16:41 | TheJulia | yeah | |
| 19:18:53 | mriedem | dtrainor: paste your nova.conf for the scheduler | |
| 19:19:40 | mriedem | dtrainor: in addition, are the cpu/ram/disk values on the flavor overridden to report 0? | |
| 19:19:54 | mriedem | 1 | |
| 19:20:01 | mriedem | see, "Another set of flavor properties must be used to disable scheduling based on standard properties for a bare metal flavor:" from https://docs.openstack.org/ironic/latest/install/configure-nova-flavors.html | |
| 19:24:41 | dtrainor | my scheduler section is pretty boring by default with only one change http://paste.openstack.org/show/736924/ reading... | |
| 19:25:15 | dtrainor | yes, the values are set to 0, as this is a default https://github.com/openstack/tripleo-heat-templates/blob/stable/queens/extraconfig/post_deploy/undercloud_post.sh#L124 | |
| 19:27:05 | cdent | mriedem or yikun_ : https://github.com/openstack/nova/blob/master/nova/objects/compute_node.py#L213 has trouble when the compute node is 'deleted' and the context hasn't diddled read_deleted | |
| 19:27:36 | mriedem | where are we reading deleting compute_nodes? | |
| 19:27:40 | mriedem | *deleted | |
| 19:28:02 | dtrainor | so by default, a node's resource class is not uppercase, which sounds like it's required per https://docs.openstack.org/ironic/queens/install/configure-nova-flavors.html . I just changed the resource class of a node from 'baremetal' to 'BAREMETAL', I'll try a deployment with that. | |
| 19:28:03 | cdent | this is showing up in the status test that I'm moving for placement fixture: line 265 here https://review.openstack.org/#/c/617941/28/nova/tests/functional/test_nova_status.py | |
| 19:28:34 | cdent | cn2.create() is of a node that is deleted (so that it can be "not seen" during the test) | |
| 19:28:42 | dtrainor | I think the thing that bothers me the most is that this deployment succeeds on other sets of hardware, i'm starting to suspect a bug on the platform on which this is deployed but i can't prove it | |
| 19:28:46 | cdent | I can diddle the context, but wanted to be sure that was the right things to do | |
| 19:28:48 | mriedem | cdent: yeah you didn't copy part of that change from yikun | |
| 19:29:19 | mriedem | cdent: https://review.openstack.org/#/c/613499/16/nova/tests/unit/cmd/test_status.py | |
| 19:29:27 | cdent | thanks | |
| 19:29:51 | mriedem | dtrainor: hmm i'm not sure how you have the RamFilter enabled then | |
| 19:30:11 | cdent | latency is the everything killer | |
| 19:30:57 | mriedem | dtrainor: especially since we removed that from the default enabled filters list in pike https://review.openstack.org/#/c/491854/ | |
| 19:31:04 | mriedem | although i see you're using the ironic_host_manager, | |
| 19:31:12 | mriedem | which is also deprecated and you shouldn't need | |
| 19:31:36 | dtrainor | like i said earier, I'm using a downstream release of queens, so i think that may be adding to the confusion | |
| 19:31:42 | dtrainor | right | |
| 19:32:18 | dtrainor | feels like queens was a weird transition state to deprecate some scheduling bits in place of some more modern techniques, which may very well lead to me doing something Wrong(TM) | |
| 19:32:24 | mriedem | https://github.com/openstack/nova/blob/stable/queens/nova/conf/scheduler.py#L309 | |
| 19:32:35 | mriedem | https://github.com/openstack/nova/blob/stable/queens/nova/conf/scheduler.py#L341 | |
| 19:32:54 | dtrainor | very good. thanks for the links. | |
| 19:32:57 | mriedem | the default on use_baremetal_filters is False so you shouldn't be using those | |
| 19:33:48 | mriedem | anyway, i'd change host_manager=host_manager in nova.conf and see if that helps | |