Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-08-11
11:33:41 dtantsur (please pardon my ignorance)
11:34:39 dtantsur but yes, I suspect we cannot create resource classes for such nodes, because we don't report them back ever
11:35:19 mriedem ok so the node has the resource_class set after there is an instance associated and that's how we migrate the flavor extra spec,
11:35:35 mriedem but once the instance is associated, the node is in an ACTIVE state which means we don't report inventory for it?
11:35:40 dtantsur correct
11:35:43 mriedem and thus can't auto-create the newly added resource class
11:36:13 mriedem and that instance <> node is what's already consuming the existing vcpu/memory_mb/disk_gb inventory that we can't delete now
11:36:28 mriedem Aug 11 11:18:56 ubuntu nova-compute[7647]: WARNING nova.scheduler.client.report [None req-267b8c81-5ea5-4e13-9ea8-02354628c37f None None] [req-27dbace7-c527-4f8f-add8-d732047d7382] We c annot delete inventory 'VCPU, MEMORY_MB, DISK_GB' for resource provider 68d57495-2daa-46b3-8e2c-f2f0a19dbaf8 because the inventory is in use.
11:36:29 dtantsur also correct
11:37:58 mriedem heh, yeah, that warning shows up all the time in an ironic ci job run
11:38:00 mriedem http://logs.openstack.org/54/487954/12/check/gate-tempest-dsvm-ironic-ipa-wholedisk-bios-agent_ipmitool-tinyipa-ubuntu-xenial-nv/041c03a/logs/screen-n-cpu.txt.gz#_Aug_09_19_31_21_252127
11:38:35 mriedem i'll open a bug for this since i think it's something we haven't considered, obviously
11:38:43 dtantsur yes please
11:39:06 dtantsur mriedem: can we create resource classes in Placement from the ironic driver each time we encounter a new resource_class?
11:39:16 dtantsur or is it a crazy idea for some reason?
11:39:44 mriedem well,
11:39:56 mriedem i suppose the idea we return 0 inventory in this case is because there is an instance consuming the node,
11:39:59 cdent i seem to have been disconnected briefly so going back in the log, saw some discussion about trying to report 0 inventory for a ironic node with an instance on it
11:40:02 cdent this is bad
11:40:05 mriedem so we don't want to have the scheduler think there is a node available
11:40:12 mriedem for building a new instance
11:40:15 dtantsur right
11:40:18 mriedem as the node is wholly consumed
11:40:22 cdent the inventory should be whatever the capacity is, and then allocations to cover the instance
11:40:26 dtantsur right, and we cannot return VCPU=0
11:42:32 dtantsur cdent: I tend to agree, I'm not sure why we do it
11:43:00 cdent it’s not a question of tend to agree. if you’re doing that, it violates the principles of how placement is supposed to work...
11:43:21 cdent min_unit in the ironic case doesn’t make a lot of sense
11:43:43 cdent but for vms where you don’t want to allow people to slice up the host into lots of tiny pieces, it is meaningful
11:44:21 dtantsur cdent: I used "tend to agree" to designate that I do not know Nova well enough, not to question your findings
11:44:53 cdent I didn’t think you were questioning, I was just reinforcing the point: sounds like weird stuff afoot
11:45:20 dtantsur ok
11:45:36 dtantsur so
11:45:50 dtantsur should we just fix it to remove min_unit and always return the correct inventory?
11:45:58 dtantsur s/correct/complete/
11:46:23 dtantsur oh, I think I know why it was done
11:46:37 cdent one sec
11:47:08 vdrok so _refresh_cache is called in the driver.get_available_nodes from compute manager's update_available_resource periodic, and we migrate the flavor there. then we call the update_available_resource, which calls get_available_resource. we still include the resoruce_class in the return dict in https://github.com/openstack/nova/blob/master/nova/virt/ironic/driver.py#L335.
11:47:19 dtantsur cdent: this is mimicking the old behavior with bare metal nodes. if we always report the complete inventory, e.g. 2Gi of RAM. and there is an instance with 1Gi of RAM. the Placement will think that 1Gi of RAM is still free
11:47:30 vdrok dtantsur: so do you propose to call the placement to create resource class right during the update_available_resource?
11:47:47 dtantsur vdrok: something like that.. but let's figure out the inventory problem first
11:48:03 mriedem https://bugs.launchpad.net/nova/+bug/1710141
11:48:04 openstack Launchpad bug 1710141 in OpenStack Compute (nova) "Continual warnings in n-cpu logs about being unable to delete inventory for an ironic node with an instance on it" [High,Triaged]
11:48:09 mriedem cdent: dtantsur: vdrok: ^
11:48:14 dtantsur thanks mriedem
11:49:19 dtantsur anyway, cdent, mriedem, wdyt about returning the custom resource class in the inventory *always*, even for occupied nodes?
11:49:21 vdrok dtantsur: I think the reason of having min_unit=0 is because if the resource provider can provide 0 of some resource, it;s not really that resource's provider :)
11:49:30 vdrok err, min_unit=1
11:49:58 vdrok if that's what you're talking about
11:50:24 cdent dtantsur: are you talking about the get_inventory call in the virt driver?
11:50:40 dtantsur cdent: yes. maybe I should make a DNM patch showing it..
11:50:59 cdent if the physical hardware hasn’t changed, that should always return the same thing, without regard to presence of an instance
11:51:14 cdent so I think the answer to your question is "yes"
11:51:40 mriedem as cdent pointed out, it seems we should be reporting the inventory regardless of what's allocated on that provider, and let the allocation consume the inventory so the scheduler will ignore it
11:51:55 mriedem i.e. this node has 1 VCPU and that 1 VCPU is consumed, so it's not eligible for building another instance
11:52:00 boolman hi peeps, Can I disable local disk on hypervisors? eg: openstack server create --image xenial --security-group default --key-name emil --network backend --flavor smallish demo -- currently this creates the instance on local disk on the hypervisors
11:52:21 mriedem boolman: you'd have to use a volume
11:52:24 boolman since I'm using ceph rbd storage i want to force that
11:52:28 openstackgerrit Dmitry Tantsur proposed openstack/nova master: DNM PoC for fixing ironic with resource classes https://review.openstack.org/492964
11:52:29 dtantsur cdent, mriedem, something like ^^^
11:52:46 mriedem boolman: boot from volume that deletes on termination
11:52:49 openstackgerrit Chris Dent proposed openstack/nova master: Optional separate database for placement API https://review.openstack.org/362766
11:53:35 boolman mriedem: so I can't actually disable local disk? the user have to create a volume to use when creating the instance?
11:54:01 mriedem dtantsur: i'm not sure that will work, i'd expect the PUT /resource_providers/uuid/inventories to fail to remove the inventory for the vcpu/memory_mb/disk_gb because it's already being used
11:54:09 mriedem it would auto-create the custom resource class though,
11:54:12 mriedem if that's your aim
11:54:38 dtantsur mriedem: it already fails at removing the inventory. I'm trying to fix auto-creation for now
11:54:46 mriedem boolman: you could use the rbd imagebackend on the compute so your computes are shared a ceph pool of disk
11:54:51 mriedem if you don't want local being used
11:54:57 mriedem *sharing
11:55:31 mriedem dtantsur: yeah, this is just probably not the way to do this i don't think
11:55:42 boolman mriedem: you mean by modifying the pool in virsh?
11:55:48 mriedem it's super tightly coupled to knowing exactly how the inventory is used by the RT and the report client
11:56:02 mriedem boolman: see http://lists.openstack.org/pipermail/openstack-dev/2017-May/117012.html
11:56:47 dtantsur mriedem: but isn't it the correct thing to do? I mean, always return the inventory of this custom resource class?
11:57:03 dtantsur (given that we will remove the hacks around VCPU and friends in Queens)
11:57:10 mriedem sdague: docs migration annoyance of the day,
11:57:19 mriedem when i search for things in the nova docs now, they search all of docs.o.o
11:57:20 cdent dtantsur: the issue is that if there is already inventory in use that is based on vcpus, then you won’t be able to change the inventory
11:57:24 mriedem not just the nova docs
11:57:41 dtantsur cdent: why should I?
11:58:16 cdent as I read that code you are trying to replace existin inventory, but maybe I’m not understanding?
11:58:16 dtantsur okay, we're trying to solve two problems at the same time:
11:58:24 mriedem boolman: no i'm talking about this https://github.com/openstack/nova/blob/master/nova/conf/libvirt.py#L631
11:58:30 dtantsur 1. broken ironic scheduling with custom resource classes
11:58:33 vdrok dtantsur: you mean something like http://paste.openstack.org/show/618161/
11:58:34 vdrok ?
11:58:45 dtantsur 2. warning on trying to delete the inventory, because we stop reporting it correctly
11:58:54 dtantsur I'm trying to fix #1 first, as it's a hard blocker for this work
11:58:56 mriedem https://github.com/openstack/nova/blob/master/nova/virt/libvirt/imagebackend.py#L788
11:59:16 dtantsur vdrok: yes, see the DNM patch I posted above
11:59:42 cdent okay, let’s dismiss #2 entirely for a moment
11:59:45 boolman mriedem: ok thanks, will try that
11:59:54 vdrok dtantsur: ah, right, /me is slow :)
12:00:15 dtantsur cdent: so, I think we all agree that our get_inventory is incorrect. We cannot fix it at once, because of the nature of bare metal nodes. we can fix reporting the custom resource class, and just wait for VCPU handling to be removed in Queens completely. Does it make more sense?
12:00:26 cdent dtantsur: can’t you point me at some irc or test logs where the #1 problem is explained or demonstrated?
12:01:09 mriedem cdent: that's in https://bugs.launchpad.net/nova/+bug/1710141
12:01:10 openstack Launchpad bug 1710141 in OpenStack Compute (nova) "Continual warnings in n-cpu logs about being unable to delete inventory for an ironic node with an instance on it" [High,Triaged]
12:01:12 cdent dtantsur: your dnm code is create an entire new inventory, with just the resource class set
12:01:15 cdent thanks mriedem

Earlier   Later