Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-11-29
20:43:17 mriedem which is...odd
20:43:32 mriedem i thought we always updated the host state after we've "claimed" an instance on that host
20:43:34 mriedem edleafe: ^
20:46:49 edleafe mriedem: https://github.com/openstack/nova/blob/master/nova/scheduler/filter_scheduler.py#L255
20:47:03 edleafe mriedem: is this running filter_scheduler?
20:47:08 mriedem i guess _locked_consume_from_request doesn't actually add the "consumed" instance to it's list of running instances on that host
20:47:10 mriedem edleafe: yes
20:47:30 mriedem it has to wait for the track_instance_changes stuff from the compute to send an rpc message to the scheduler that the instance actually landed there?
20:47:34 dansmith mriedem: how many times have you seen this?
20:47:38 mriedem dansmith: just now
20:47:40 dansmith I thought you said logstash isn't crying?
20:47:43 mriedem it's not
20:47:46 dansmith okay
20:47:51 mriedem so, probably totally latent
20:47:56 mriedem and just running on a slow node or something
20:48:33 mriedem it just seems odd that we update the HostState in memory to consume resources from the request spec, but we don't actually count that instance as tracked on the chosen host at the same time
20:48:55 mriedem like, i get we might rebuild and the instance will end up on another host, but then the consumed resources on that HostState object are also wrong in that case
20:48:59 mriedem until they sync up
20:52:00 mriedem edleafe: yeah i'm surprised we don't call something like this during that "consume_from_request" call https://github.com/openstack/nova/blob/master/nova/scheduler/host_manager.py#L767
20:52:39 mriedem we "consume" the resources but we don't track that instance against that host until it syncs up from the compute
20:53:15 dansmith well, that's the classic scheduler race right?
20:53:17 mriedem which, in superconductor separated cells v2 world, we can't rely on b/c the computes can't cast to the scheduler
20:53:22 dansmith we consume in the num_instances loop,
20:53:34 openstackgerrit Eric Berglund proposed openstack/nova master: Add PowerVM to compute_driver options https://review.openstack.org/523978
20:53:35 dansmith but until it hits the compute it's still unclaimed
20:54:04 dansmith but placement claims should be sealing that off for everything now
20:54:35 mriedem sure, but the resource claim isn't the problem here, it's really the affinity stuff
20:54:39 mriedem which we know is still a gap
20:54:51 dansmith ah right
20:55:01 mriedem which now that i'm thinking about this,
20:55:18 mriedem i thought within a single request, we tracked which instances landed on which hosts so the affinity filters would work (within the same request)
20:55:25 mriedem but now i can't find where that happens
20:57:00 melwitt cells meeting in 4 minutes?
20:57:04 dansmith yep
20:57:25 melwitt cool
21:23:23 dansmith mriedem: so, yesterday I was thinking that this would be easier than what we do for instances because we don't have variable sort keys and such
21:23:26 dansmith and it is a little,
21:23:43 dansmith but the local/global marker thing still needs to happen, AFAIK, which makes it more similar I guess
21:24:28 dansmith it's quite unfortunate that we have to be able to do this tbh.. listing migrations per instance is a lot easier obviously
21:25:02 dansmith in general, I think that it's a good idea for us to try to avoid adding more operations that list multiple things that are not scoped to an instance, as that's kindof nova's primary key
21:25:19 openstackgerrit Brianna Poulos proposed openstack/nova master: Add trusted_image_certificates to REST API https://review.openstack.org/486204
21:25:19 openstackgerrit Brianna Poulos proposed openstack/nova master: Add trusted_certs object https://review.openstack.org/489408
21:26:53 mriedem well, listing things across cells is fine, like services and compute nodes
21:27:00 mriedem it's the paging that is killing this
21:27:24 openstackgerrit Brianna Poulos proposed openstack/nova master: Add trusted_image_certificates to REST API https://review.openstack.org/486204
21:27:34 dansmith right, but services and compute nodes scale with nodes,
21:27:37 dansmith migrations scale with instances
21:27:53 dansmith listing bdms would be similarly painful, for example
21:28:41 mriedem yeah, we should have far fewer migrations than instances or bdms, but yeah - but that's also why they have this spec, to page across a large number of migrations
21:29:04 mriedem anyway, maybe his older PS was more correct https://review.openstack.org/#/c/517273/21/nova/compute/migration_list.py
21:29:09 mriedem because it was doing local marker stuff
21:29:09 dansmith I dunno, over time you probably end up with more migrations than instances, no?
21:29:22 mriedem we delete migrations when we delete instances
21:29:25 openstackgerrit Brianna Poulos proposed openstack/nova master: Add trusted_image_certificates to REST API https://review.openstack.org/486204
21:29:41 dansmith sure, but if you've got like 40% pets, and they all move a couple times a year for maintenance,
21:29:47 dansmith you're over that limit
21:30:08 mriedem good poitn
21:30:10 mriedem *point
21:30:13 dansmith and if you have, gasp, 95% pets...
21:30:17 dansmith but surely nobody.... :D
21:30:30 mriedem i'd be willing to best most of huawei public cloud's customers are running pets
21:30:33 mriedem enterprise nfv pets
21:30:36 dansmith exactly
21:30:42 mriedem *bet
21:30:46 dansmith I expect most people that want this are running north of 90% :)
21:31:27 openstackgerrit Brianna Poulos proposed openstack/nova master: Add trusted_image_certificates to REST API https://review.openstack.org/486204
21:43:29 melwitt mriedem: did you want me to fix the thing I found in one of the volume update backports? https://review.openstack.org/#/c/523911
21:44:01 mriedem yeah
21:44:09 melwitt k, doing
21:47:47 oomichi_afk alex_xu: nice work, +2 for both
21:49:15 oomichi alex_xu: gmann_afk: happy to see your feedback on https://review.openstack.org/#/c/523194
21:54:00 mriedem melwitt: a couple of questions in your quota dead code removal patch https://review.openstack.org/#/c/511689/
21:54:13 melwitt mriedem: cool, thanks for looking
21:57:21 openstackgerrit Chris Dent proposed openstack/nova master: Optional separate database for placement API https://review.openstack.org/362766
21:57:35 openstackgerrit melanie witt proposed openstack/nova stable/ocata: Save updated libvirt domain XML after swapping volume https://review.openstack.org/523911
22:25:02 melwitt mriedem: replied. I can push a new rev if you don't find any other issues with it
22:30:37 mriedem melwitt: replied, +W, just deal with the other cleanup in a follow up
22:30:57 melwitt mriedem: k, thanks
22:33:49 openstackgerrit melanie witt proposed openstack/nova master: qemu-img do not use cache=none if no O_DIRECT support https://review.openstack.org/523554
22:34:31 melwitt I've had to fix that a billion times because that direct io test function tries to create a file to check something, but the file create fails in CI but not locally
22:38:44 mgagne I'm reading https://docs.openstack.org/nova/pike/user/placement.html#ocata-15-0-0
22:38:50 mgagne what does "Users of CellsV1 will need to deploy a placement per cell, matching the scope and cardinality of the regular nova-scheduler process." mean?
22:42:46 mriedem mgagne: means you'd have a placement service running per child cell, like the nova-scheduler process using the FilterScheduler (not the cell scheduler at the top api cell); that's what belmiro is doing at CERN, but I think you could also do a global placement (like keystone), which is what smorrison at NeCTAR is planning on doing.
22:43:46 mriedem i can't exactly remember why belmiro was going to do it per-cell to start, because he was also going to run into some service catalog entry issues with that, and maybe have to put the per-cell placement endpoints in per-cell regions in the catalog
22:43:54 mgagne ok, what I'm wondering is: isn't placement a HTTP api? if so, how do I declare it in the catalog if catalog is cell agnostic? I also couldn't find (when I checked) a way to override the placement API url in the config.
22:44:32 mriedem right, i think that's what belmiro ran into and was going to have to do some funny stuff with regions in the catalog
22:44:35 mgagne right... imo, the upgrade note suggests a solution which isn't really possible without thinking outside the box or something
22:44:39 mriedem there isn't an option to override the endpoint url for placement in nova.conf
22:44:49 mgagne there is not
22:44:55 mriedem mgagne: yeah, i think that was written before belmiro started looking at doing that
22:45:23 mriedem dansmith: tssurya_: do you remember why belmiro decided to do a per-cell placement service rather than a global one?
22:45:54 mriedem mgagne: would probaly be good if we could start an etherpad or get a ML thread started with notes from belmiro and smorrison about what they are doing
22:45:57 mriedem i could start that if you want
22:47:26 mgagne sure, I'm trying to understand what I need to do for newton and I keep reading stuff I don't fully understand or I don't know how it can technically be doable
22:48:42 mriedem yeah let me start an etherpad for notes
22:49:00 mgagne so if the recommended way to run placement api is globally, I don't understand why upgrade notes suggest something else
22:49:54 INeedaUUID dansmith: Ohhh. You're gonna hate what I did to sort changing the uuids.
22:50:12 INeedaUUID You're gonna tel me it's not a good idea. So happy for you to facepalm and tell me i'm stupid.
22:51:56 mriedem mgagne: i think that was written before we released the catalog issue
22:52:25 mgagne would there be a way to update the document?

Earlier   Later