Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-11-29
20:37:41 mriedem it's enabled in devstack
20:38:04 dansmith oh
20:38:21 mriedem this isn't a new test and logstash is not showing me a bunch of failures, so maybe just very random
20:38:55 dansmith and the test isn't requesting samehost= ?
20:39:02 mriedem it is
20:39:13 mriedem and it's a single node job
20:39:19 mriedem separate requests though,
20:39:30 mriedem so the HostState might not have the previous host tracked in memory when the 2nd request comes in?
20:40:22 dansmith idk
20:40:26 dansmith I mean,
20:40:33 dansmith if it's requesting a samehost then that has to be why
20:40:40 dansmith it's pretty simple
20:40:46 efried cdent You got a handle on how API fixtures are working in the functional test suites?
20:41:01 mriedem it's doing same_host on 'dad917c1-11bf-4aa1-9c3b-c864cdbc97c6'
20:41:03 mriedem http://logs.openstack.org/10/523910/1/check/legacy-tempest-dsvm-cells/76a0551/logs/screen-n-sch.txt.gz#_Nov_29_16_05_42_182254
20:42:01 cdent efried: nova api fixtures? slightly. I made some adjustments to them recently. What’s up?
20:42:02 mriedem and when we request with the same_host, the HostState is tracking c9115b21-9355-4d1b-b797-62d2db7ca1ac and f5cb92ec-0474-4f4f-a1de-1d303e67e5f0
20:43:13 mriedem we don't update the HostState with that instance until later http://logs.openstack.org/10/523910/1/check/legacy-tempest-dsvm-cells/76a0551/logs/screen-n-sch.txt.gz#_Nov_29_16_06_17_769927
20:43:17 mriedem which is...odd
20:43:32 mriedem i thought we always updated the host state after we've "claimed" an instance on that host
20:43:34 mriedem edleafe: ^
20:46:49 edleafe mriedem: https://github.com/openstack/nova/blob/master/nova/scheduler/filter_scheduler.py#L255
20:47:03 edleafe mriedem: is this running filter_scheduler?
20:47:08 mriedem i guess _locked_consume_from_request doesn't actually add the "consumed" instance to it's list of running instances on that host
20:47:10 mriedem edleafe: yes
20:47:30 mriedem it has to wait for the track_instance_changes stuff from the compute to send an rpc message to the scheduler that the instance actually landed there?
20:47:34 dansmith mriedem: how many times have you seen this?
20:47:38 mriedem dansmith: just now
20:47:40 dansmith I thought you said logstash isn't crying?
20:47:43 mriedem it's not
20:47:46 dansmith okay
20:47:51 mriedem so, probably totally latent
20:47:56 mriedem and just running on a slow node or something
20:48:33 mriedem it just seems odd that we update the HostState in memory to consume resources from the request spec, but we don't actually count that instance as tracked on the chosen host at the same time
20:48:55 mriedem like, i get we might rebuild and the instance will end up on another host, but then the consumed resources on that HostState object are also wrong in that case
20:48:59 mriedem until they sync up
20:52:00 mriedem edleafe: yeah i'm surprised we don't call something like this during that "consume_from_request" call https://github.com/openstack/nova/blob/master/nova/scheduler/host_manager.py#L767
20:52:39 mriedem we "consume" the resources but we don't track that instance against that host until it syncs up from the compute
20:53:15 dansmith well, that's the classic scheduler race right?
20:53:17 mriedem which, in superconductor separated cells v2 world, we can't rely on b/c the computes can't cast to the scheduler
20:53:22 dansmith we consume in the num_instances loop,
20:53:34 openstackgerrit Eric Berglund proposed openstack/nova master: Add PowerVM to compute_driver options https://review.openstack.org/523978
20:53:35 dansmith but until it hits the compute it's still unclaimed
20:54:04 dansmith but placement claims should be sealing that off for everything now
20:54:35 mriedem sure, but the resource claim isn't the problem here, it's really the affinity stuff
20:54:39 mriedem which we know is still a gap
20:54:51 dansmith ah right
20:55:01 mriedem which now that i'm thinking about this,
20:55:18 mriedem i thought within a single request, we tracked which instances landed on which hosts so the affinity filters would work (within the same request)
20:55:25 mriedem but now i can't find where that happens
20:57:00 melwitt cells meeting in 4 minutes?
20:57:04 dansmith yep
20:57:25 melwitt cool
21:23:23 dansmith mriedem: so, yesterday I was thinking that this would be easier than what we do for instances because we don't have variable sort keys and such
21:23:26 dansmith and it is a little,
21:23:43 dansmith but the local/global marker thing still needs to happen, AFAIK, which makes it more similar I guess
21:24:28 dansmith it's quite unfortunate that we have to be able to do this tbh.. listing migrations per instance is a lot easier obviously
21:25:02 dansmith in general, I think that it's a good idea for us to try to avoid adding more operations that list multiple things that are not scoped to an instance, as that's kindof nova's primary key
21:25:19 openstackgerrit Brianna Poulos proposed openstack/nova master: Add trusted_certs object https://review.openstack.org/489408
21:25:19 openstackgerrit Brianna Poulos proposed openstack/nova master: Add trusted_image_certificates to REST API https://review.openstack.org/486204
21:26:53 mriedem well, listing things across cells is fine, like services and compute nodes
21:27:00 mriedem it's the paging that is killing this
21:27:24 openstackgerrit Brianna Poulos proposed openstack/nova master: Add trusted_image_certificates to REST API https://review.openstack.org/486204
21:27:34 dansmith right, but services and compute nodes scale with nodes,
21:27:37 dansmith migrations scale with instances
21:27:53 dansmith listing bdms would be similarly painful, for example
21:28:41 mriedem yeah, we should have far fewer migrations than instances or bdms, but yeah - but that's also why they have this spec, to page across a large number of migrations
21:29:04 mriedem anyway, maybe his older PS was more correct https://review.openstack.org/#/c/517273/21/nova/compute/migration_list.py
21:29:09 dansmith I dunno, over time you probably end up with more migrations than instances, no?
21:29:09 mriedem because it was doing local marker stuff
21:29:22 mriedem we delete migrations when we delete instances
21:29:25 openstackgerrit Brianna Poulos proposed openstack/nova master: Add trusted_image_certificates to REST API https://review.openstack.org/486204
21:29:41 dansmith sure, but if you've got like 40% pets, and they all move a couple times a year for maintenance,
21:29:47 dansmith you're over that limit
21:30:08 mriedem good poitn
21:30:10 mriedem *point
21:30:13 dansmith and if you have, gasp, 95% pets...
21:30:17 dansmith but surely nobody.... :D
21:30:30 mriedem i'd be willing to best most of huawei public cloud's customers are running pets
21:30:33 mriedem enterprise nfv pets
21:30:36 dansmith exactly
21:30:42 mriedem *bet
21:30:46 dansmith I expect most people that want this are running north of 90% :)
21:31:27 openstackgerrit Brianna Poulos proposed openstack/nova master: Add trusted_image_certificates to REST API https://review.openstack.org/486204
21:43:29 melwitt mriedem: did you want me to fix the thing I found in one of the volume update backports? https://review.openstack.org/#/c/523911
21:44:01 mriedem yeah
21:44:09 melwitt k, doing
21:47:47 oomichi_afk alex_xu: nice work, +2 for both
21:49:15 oomichi alex_xu: gmann_afk: happy to see your feedback on https://review.openstack.org/#/c/523194
21:54:00 mriedem melwitt: a couple of questions in your quota dead code removal patch https://review.openstack.org/#/c/511689/
21:54:13 melwitt mriedem: cool, thanks for looking
21:57:21 openstackgerrit Chris Dent proposed openstack/nova master: Optional separate database for placement API https://review.openstack.org/362766
21:57:35 openstackgerrit melanie witt proposed openstack/nova stable/ocata: Save updated libvirt domain XML after swapping volume https://review.openstack.org/523911
22:25:02 melwitt mriedem: replied. I can push a new rev if you don't find any other issues with it
22:30:37 mriedem melwitt: replied, +W, just deal with the other cleanup in a follow up
22:30:57 melwitt mriedem: k, thanks
22:33:49 openstackgerrit melanie witt proposed openstack/nova master: qemu-img do not use cache=none if no O_DIRECT support https://review.openstack.org/523554
22:34:31 melwitt I've had to fix that a billion times because that direct io test function tries to create a file to check something, but the file create fails in CI but not locally
22:38:44 mgagne I'm reading https://docs.openstack.org/nova/pike/user/placement.html#ocata-15-0-0
22:38:50 mgagne what does "Users of CellsV1 will need to deploy a placement per cell, matching the scope and cardinality of the regular nova-scheduler process." mean?

Earlier   Later