Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-11-29
20:48:55 mriedem like, i get we might rebuild and the instance will end up on another host, but then the consumed resources on that HostState object are also wrong in that case
20:48:59 mriedem until they sync up
20:52:00 mriedem edleafe: yeah i'm surprised we don't call something like this during that "consume_from_request" call https://github.com/openstack/nova/blob/master/nova/scheduler/host_manager.py#L767
20:52:39 mriedem we "consume" the resources but we don't track that instance against that host until it syncs up from the compute
20:53:15 dansmith well, that's the classic scheduler race right?
20:53:17 mriedem which, in superconductor separated cells v2 world, we can't rely on b/c the computes can't cast to the scheduler
20:53:22 dansmith we consume in the num_instances loop,
20:53:34 openstackgerrit Eric Berglund proposed openstack/nova master: Add PowerVM to compute_driver options https://review.openstack.org/523978
20:53:35 dansmith but until it hits the compute it's still unclaimed
20:54:04 dansmith but placement claims should be sealing that off for everything now
20:54:35 mriedem sure, but the resource claim isn't the problem here, it's really the affinity stuff
20:54:39 mriedem which we know is still a gap
20:54:51 dansmith ah right
20:55:01 mriedem which now that i'm thinking about this,
20:55:18 mriedem i thought within a single request, we tracked which instances landed on which hosts so the affinity filters would work (within the same request)
20:55:25 mriedem but now i can't find where that happens
20:57:00 melwitt cells meeting in 4 minutes?
20:57:04 dansmith yep
20:57:25 melwitt cool
21:23:23 dansmith mriedem: so, yesterday I was thinking that this would be easier than what we do for instances because we don't have variable sort keys and such
21:23:26 dansmith and it is a little,
21:23:43 dansmith but the local/global marker thing still needs to happen, AFAIK, which makes it more similar I guess
21:24:28 dansmith it's quite unfortunate that we have to be able to do this tbh.. listing migrations per instance is a lot easier obviously
21:25:02 dansmith in general, I think that it's a good idea for us to try to avoid adding more operations that list multiple things that are not scoped to an instance, as that's kindof nova's primary key
21:25:19 openstackgerrit Brianna Poulos proposed openstack/nova master: Add trusted_image_certificates to REST API https://review.openstack.org/486204
21:25:19 openstackgerrit Brianna Poulos proposed openstack/nova master: Add trusted_certs object https://review.openstack.org/489408
21:26:53 mriedem well, listing things across cells is fine, like services and compute nodes
21:27:00 mriedem it's the paging that is killing this
21:27:24 openstackgerrit Brianna Poulos proposed openstack/nova master: Add trusted_image_certificates to REST API https://review.openstack.org/486204
21:27:34 dansmith right, but services and compute nodes scale with nodes,
21:27:37 dansmith migrations scale with instances
21:27:53 dansmith listing bdms would be similarly painful, for example
21:28:41 mriedem yeah, we should have far fewer migrations than instances or bdms, but yeah - but that's also why they have this spec, to page across a large number of migrations
21:29:04 mriedem anyway, maybe his older PS was more correct https://review.openstack.org/#/c/517273/21/nova/compute/migration_list.py
21:29:09 mriedem because it was doing local marker stuff
21:29:09 dansmith I dunno, over time you probably end up with more migrations than instances, no?
21:29:22 mriedem we delete migrations when we delete instances
21:29:25 openstackgerrit Brianna Poulos proposed openstack/nova master: Add trusted_image_certificates to REST API https://review.openstack.org/486204
21:29:41 dansmith sure, but if you've got like 40% pets, and they all move a couple times a year for maintenance,
21:29:47 dansmith you're over that limit
21:30:08 mriedem good poitn
21:30:10 mriedem *point
21:30:13 dansmith and if you have, gasp, 95% pets...
21:30:17 dansmith but surely nobody.... :D
21:30:30 mriedem i'd be willing to best most of huawei public cloud's customers are running pets
21:30:33 mriedem enterprise nfv pets
21:30:36 dansmith exactly
21:30:42 mriedem *bet
21:30:46 dansmith I expect most people that want this are running north of 90% :)
21:31:27 openstackgerrit Brianna Poulos proposed openstack/nova master: Add trusted_image_certificates to REST API https://review.openstack.org/486204
21:43:29 melwitt mriedem: did you want me to fix the thing I found in one of the volume update backports? https://review.openstack.org/#/c/523911
21:44:01 mriedem yeah
21:44:09 melwitt k, doing
21:47:47 oomichi_afk alex_xu: nice work, +2 for both
21:49:15 oomichi alex_xu: gmann_afk: happy to see your feedback on https://review.openstack.org/#/c/523194
21:54:00 mriedem melwitt: a couple of questions in your quota dead code removal patch https://review.openstack.org/#/c/511689/
21:54:13 melwitt mriedem: cool, thanks for looking
21:57:21 openstackgerrit Chris Dent proposed openstack/nova master: Optional separate database for placement API https://review.openstack.org/362766
21:57:35 openstackgerrit melanie witt proposed openstack/nova stable/ocata: Save updated libvirt domain XML after swapping volume https://review.openstack.org/523911
22:25:02 melwitt mriedem: replied. I can push a new rev if you don't find any other issues with it
22:30:37 mriedem melwitt: replied, +W, just deal with the other cleanup in a follow up
22:30:57 melwitt mriedem: k, thanks
22:33:49 openstackgerrit melanie witt proposed openstack/nova master: qemu-img do not use cache=none if no O_DIRECT support https://review.openstack.org/523554
22:34:31 melwitt I've had to fix that a billion times because that direct io test function tries to create a file to check something, but the file create fails in CI but not locally
22:38:44 mgagne I'm reading https://docs.openstack.org/nova/pike/user/placement.html#ocata-15-0-0
22:38:50 mgagne what does "Users of CellsV1 will need to deploy a placement per cell, matching the scope and cardinality of the regular nova-scheduler process." mean?
22:42:46 mriedem mgagne: means you'd have a placement service running per child cell, like the nova-scheduler process using the FilterScheduler (not the cell scheduler at the top api cell); that's what belmiro is doing at CERN, but I think you could also do a global placement (like keystone), which is what smorrison at NeCTAR is planning on doing.
22:43:46 mriedem i can't exactly remember why belmiro was going to do it per-cell to start, because he was also going to run into some service catalog entry issues with that, and maybe have to put the per-cell placement endpoints in per-cell regions in the catalog
22:43:54 mgagne ok, what I'm wondering is: isn't placement a HTTP api? if so, how do I declare it in the catalog if catalog is cell agnostic? I also couldn't find (when I checked) a way to override the placement API url in the config.
22:44:32 mriedem right, i think that's what belmiro ran into and was going to have to do some funny stuff with regions in the catalog
22:44:35 mgagne right... imo, the upgrade note suggests a solution which isn't really possible without thinking outside the box or something
22:44:39 mriedem there isn't an option to override the endpoint url for placement in nova.conf
22:44:49 mgagne there is not
22:44:55 mriedem mgagne: yeah, i think that was written before belmiro started looking at doing that
22:45:23 mriedem dansmith: tssurya_: do you remember why belmiro decided to do a per-cell placement service rather than a global one?
22:45:54 mriedem mgagne: would probaly be good if we could start an etherpad or get a ML thread started with notes from belmiro and smorrison about what they are doing
22:45:57 mriedem i could start that if you want
22:47:26 mgagne sure, I'm trying to understand what I need to do for newton and I keep reading stuff I don't fully understand or I don't know how it can technically be doable
22:48:42 mriedem yeah let me start an etherpad for notes
22:49:00 mgagne so if the recommended way to run placement api is globally, I don't understand why upgrade notes suggest something else
22:49:54 INeedaUUID dansmith: Ohhh. You're gonna hate what I did to sort changing the uuids.
22:50:12 INeedaUUID You're gonna tel me it's not a good idea. So happy for you to facepalm and tell me i'm stupid.
22:51:56 mriedem mgagne: i think that was written before we released the catalog issue
22:52:25 mgagne would there be a way to update the document?
22:53:08 mriedem yeah
22:53:56 openstackgerrit Merged openstack/nova master: Add 'all_tenants' for GET sec group api ref https://review.openstack.org/522910
22:53:57 mgagne also, is the placement api supposed to be available to the end user? if so, for what purpose?
22:55:26 mriedem no, it's admin-only
22:56:00 mriedem i think i remember why belmiro was doing per-cell, it's because then the per-child cell scheduler is only asking placement for hosts in that cell, so it's a perf thing in the scheduler
22:56:17 mriedem because cellA scheduler doesn't need to wittle out computes from cellB
22:56:22 mriedem https://etherpad.openstack.org/p/cellsv1-to-v2-migration
22:57:27 openstackgerrit Merged openstack/nova master: Remove unnecessary self.flags and ConfPatcher https://review.openstack.org/522993
22:58:16 mgagne "no way to override the placement endpoint URL in nova.conf per child cell before Queens" interesting. so it's now possible
22:58:50 mriedem mgagne: yeah, i forgot that with the keystoneauth adapter stuff that efried worked on, we now have an endpoint_override option for [placement] in nova.conf
22:58:51 mriedem in queens
22:59:54 mgagne good stuff, we heavily rely on endpoint override. and no, we don't care about DNS split view, not gonna happen for us. ;)
23:00:47 mriedem that's part of the point of https://specs.openstack.org/openstack/nova-specs/specs/queens/approved/use-ksa-adapter-for-endpoints.html
23:00:57 mriedem consistent configuration of all of the services that nova talks to
23:01:08 mgagne super good news!
23:02:28 mgagne I was a bit worried because I keep reading on the list that DNS split view is the (only) way to go. and everything else is "garbage" or a state of the art solution.

Earlier   Later