Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-12-10
18:50:07 dansmith etp: 10 is small enough that you're likely to not get back any hosts in the aggregate you're looking for yeah
18:50:27 dansmith etp: unless you're enabling a request filter in nova that helps to ensure that you do
18:50:38 dansmith which is currently az or tenant filtering
18:53:58 TheJulia Hopefully someone can help jog my memory... A long time ago I remember there was a discussion of how nova represents a baremetal node being torn down. I believe the state then and now is that the instance is generally "ACTIVE" until the node is actually deleted, and I think the motivation behind that was resource accounting wise. Does anyone remember this discussion (it would have been 2+ years ago I think)
18:57:42 dansmith TheJulia: the instance goes away immediately, AFAIK, as that's the way delete is supposed to work, but nova tries to make the node not schedule-able until the cleaning is done
18:58:06 mriedem dtrainor: you shouldn't need the RamFilter in queens
18:58:13 TheJulia I've got a QE person reporting that they are seeing it sit in "ACTIVE" until cleaning is over
18:58:17 dansmith the strategy we proposed for doing this betterly was to set the reserved amount of resource equal to the total for a node so that it appeared to not have any room until after it's cleaning
18:58:25 mriedem dtrainor: the placement service does the filtering on MEMORY_MB
18:58:30 TheJulia and I thought there was some reason for that, just struggling to decompress those memories
18:58:37 dansmith TheJulia: okay I could be wrong, but I don't think that's what is supposed to happen
18:59:33 etp dansmith: tnx, we have only one az atm and aggregate should be for all tenants so I think request filter doesn't help here, I'll revert config back to default
18:59:39 mriedem dtrainor: you're scheduling to baremetal nodes there?
19:00:10 mriedem dtrainor: if those are baremetal nodes, you should read up on https://docs.openstack.org/ironic/latest/install/configure-nova-flavors.html
19:01:43 mriedem dtrainor: i guess the Core/Ram/Disk filters weren't deprecated until stein but they should have been since pike i think https://docs.openstack.org/nova/latest/admin/configuration/schedulers.html#ramfilter
19:01:55 dansmith TheJulia: self.ironicclient.call("node.set_provision_state", node.uuid,
19:01:56 dansmith "deleted")
19:02:09 dansmith TheJulia: does that take a long time?
19:02:26 TheJulia dansmith: that can take hours upon hours depending on the hardware the operator configuration
19:02:40 dansmith TheJulia: yeah, looks like that has a poll loop after it
19:02:43 TheJulia dansmith: to be precise, that should return immediately
19:02:48 TheJulia but the background it runs for a long time
19:02:50 dtrainor sorry, i'm back. mriedem, I'm familiar with that doc, yep. I'm familiar with how it's scheduled and can even make this deployment work on different platforms, e.g. on top of different hardware. that's what leaves me puzzled.
19:03:46 dansmith TheJulia: yeah, so I guess it does have to go through a full cleaning before the instance appears to actually go away
19:03:53 dtrainor VCPU, MEMORY_MB, and DISK_GB are all set to 0 to disable scheduling based on properties for a bare metal flavor. i am using bare metal nodes with the intent of using the default flavor properties such as these
19:04:04 dansmith TheJulia: but the instance shouldn't be just ACTIVE, it should be ACTIVE with a task-state of "deleting" at least
19:04:08 TheJulia mriedem: I'm fairly sure somewhere there was a statement saying that it would no longer work and was deprecated, but that was a long time ago since we kept compatability for a while
19:04:43 TheJulia dansmith: that is what it reports now afaik
19:06:34 mriedem dtrainor: ok, well i can tell you the IronicHostManager, baremetal filters and core/ram/disk filters have all been deprecated and can cause issues when trying to schedule to baremetal nodes
19:06:38 mriedem so you shouldn't be using those
19:06:48 mriedem and should be scheduling based on custom resource classes
19:07:03 dtrainor right, that was my understanding
19:08:27 dtrainor all the nodes are created by default with a resource_class of 'baremetal', which my understanding is, this should match a flavor based on a flavor's properties of e.g. resources:CUSTOM_BAREMETAL='1'
19:08:37 dtrainor again, these all being defaults
19:10:18 dtrainor i have nova-compute and nova-scheduler log files, I don't want to ask for too much but maybe another set of eyes would help me find something that I'm not able to see?
19:10:43 TheJulia dansmith: do you know, off the top of your head, if nova only represents a deletion in progress in the task_status field?
19:10:53 TheJulia dansmith: I mean, in documentation
19:16:24 dansmith TheJulia: from the outside, that's the indication the user has yeah
19:16:32 dansmith TheJulia: normally it happens fairly quick of course
19:16:41 TheJulia yeah
19:18:53 mriedem dtrainor: paste your nova.conf for the scheduler
19:19:40 mriedem dtrainor: in addition, are the cpu/ram/disk values on the flavor overridden to report 0?
19:19:54 mriedem 1
19:20:01 mriedem see, "Another set of flavor properties must be used to disable scheduling based on standard properties for a bare metal flavor:" from https://docs.openstack.org/ironic/latest/install/configure-nova-flavors.html
19:24:41 dtrainor my scheduler section is pretty boring by default with only one change http://paste.openstack.org/show/736924/ reading...
19:25:15 dtrainor yes, the values are set to 0, as this is a default https://github.com/openstack/tripleo-heat-templates/blob/stable/queens/extraconfig/post_deploy/undercloud_post.sh#L124
19:27:05 cdent mriedem or yikun_ : https://github.com/openstack/nova/blob/master/nova/objects/compute_node.py#L213 has trouble when the compute node is 'deleted' and the context hasn't diddled read_deleted
19:27:36 mriedem where are we reading deleting compute_nodes?
19:27:40 mriedem *deleted
19:28:02 dtrainor so by default, a node's resource class is not uppercase, which sounds like it's required per https://docs.openstack.org/ironic/queens/install/configure-nova-flavors.html . I just changed the resource class of a node from 'baremetal' to 'BAREMETAL', I'll try a deployment with that.
19:28:03 cdent this is showing up in the status test that I'm moving for placement fixture: line 265 here https://review.openstack.org/#/c/617941/28/nova/tests/functional/test_nova_status.py
19:28:34 cdent cn2.create() is of a node that is deleted (so that it can be "not seen" during the test)
19:28:42 dtrainor I think the thing that bothers me the most is that this deployment succeeds on other sets of hardware, i'm starting to suspect a bug on the platform on which this is deployed but i can't prove it
19:28:46 cdent I can diddle the context, but wanted to be sure that was the right things to do
19:28:48 mriedem cdent: yeah you didn't copy part of that change from yikun
19:29:19 mriedem cdent: https://review.openstack.org/#/c/613499/16/nova/tests/unit/cmd/test_status.py
19:29:27 cdent thanks
19:29:51 mriedem dtrainor: hmm i'm not sure how you have the RamFilter enabled then
19:30:11 cdent latency is the everything killer
19:30:57 mriedem dtrainor: especially since we removed that from the default enabled filters list in pike https://review.openstack.org/#/c/491854/
19:31:04 mriedem although i see you're using the ironic_host_manager,
19:31:12 mriedem which is also deprecated and you shouldn't need
19:31:36 dtrainor like i said earier, I'm using a downstream release of queens, so i think that may be adding to the confusion
19:31:42 dtrainor right
19:32:18 dtrainor feels like queens was a weird transition state to deprecate some scheduling bits in place of some more modern techniques, which may very well lead to me doing something Wrong(TM)
19:32:24 mriedem https://github.com/openstack/nova/blob/stable/queens/nova/conf/scheduler.py#L309
19:32:35 mriedem https://github.com/openstack/nova/blob/stable/queens/nova/conf/scheduler.py#L341
19:32:54 dtrainor very good. thanks for the links.
19:32:57 mriedem the default on use_baremetal_filters is False so you shouldn't be using those
19:33:48 mriedem anyway, i'd change host_manager=host_manager in nova.conf and see if that helps
19:34:17 mriedem as i said, the deprecations should have happened earlier, but we likely didn't catch this stuff until tripleo failed by using the old stuff
19:34:23 mriedem and we don't deprecate things in stable branches
19:34:50 dtrainor understood.
19:34:55 mriedem there is a mention of this in https://docs.openstack.org/releasenotes/nova/queens.html
19:34:56 openstackgerrit Chris Dent proposed openstack/nova master: Use external placement in functional tests https://review.openstack.org/617941
19:36:52 dtrainor interesting. again thanks for the links!
19:36:56 dtrainor i'll do some homework.
19:38:50 openstackgerrit Chris Dent proposed openstack/nova master: Use external placement in functional tests https://review.openstack.org/617941
19:38:50 openstackgerrit Chris Dent proposed openstack/nova master: Delete the placement code https://review.openstack.org/618215
19:43:58 openstackgerrit Matt Riedemann proposed openstack/nova master: DNM: add debug logging for bug 1807723 https://review.openstack.org/624181
19:43:58 openstack bug 1807723 in OpenStack Compute (nova) "swap multiattach volume intermittently fails when servers are on different hosts" [Medium,Confirmed] https://launchpad.net/bugs/1807723
20:04:04 openstackgerrit Matt Riedemann proposed openstack/nova master: Fix target used in nova.policy.check_is_admin https://review.openstack.org/624185
20:13:04 openstackgerrit Matt Riedemann proposed openstack/nova master: Fix target used in nova.policy.check_is_admin https://review.openstack.org/624185
20:15:33 mriedem gibi: there is a runway slot open https://etherpad.openstack.org/p/nova-runways-stein so i'll throw https://blueprints.launchpad.net/nova/+spec/bandwidth-resource-provider in there bug you've got two separate entries queued up so I assume you meant to keep those separate wrt runways slots
20:15:39 mriedem *but you've got
20:31:42 mriedem dansmith: melwitt: i think we can move forward with moving nova-cells-v1 to the experimental queue https://review.openstack.org/#/c/623538
20:32:14 dansmith mriedem: I figured you'd drop the +W when ready, but.. got it
20:32:32 mriedem don't want to +W my own change
20:33:47 sean-k-mooney mriedem: did you mention the intel nfv ci was broken recently
20:34:30 mriedem yes
20:34:32 mriedem well,
20:34:34 mriedem it just skips
20:34:42 mriedem so not sure why it's litsening on nova changes
20:34:45 mriedem *listening
20:34:45 dansmith mriedem: well, we left enough +2s on there, but ... as you wish
20:35:25 sean-k-mooney im currently setting up a replacement for it at home
20:36:56 sean-k-mooney im going to dedicate one of my dev servers too it. assumeing the noise stays low ill leave it that way for at least the time being. im going to see if i can use other resource i have acess too also.
20:38:20 sean-k-mooney my plan is to add the nova core team to the list of people that can leave a comment to have it run and try to have it run on all chagne to a subset of file.
20:38:42 sean-k-mooney e.g. the libvirt dirver and neutron code
20:40:38 dtrainor mriedem, setting host_manager=host_manager didn't seen to make a difference. The deployment still fails with the same errors.

Earlier   Later