| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-11-29 | |||
| 19:06:56 | dansmith | why not just dump ass right there and stop? | |
| 19:07:30 | mriedem | we could, but what do we raise? it's a 500 if we raise i think | |
| 19:07:53 | dansmith | is that less right than logging an error and saying "this is just going to break in the next service down" ? | |
| 19:08:08 | mriedem | at this point it would run through the scheduler with the provided image | |
| 19:08:24 | mriedem | which is basically what this was trying to fix - not doing that if the image doesn't change | |
| 19:08:42 | mriedem | the final patch in that series will fail though | |
| 19:08:56 | mriedem | i'm cool with a 500 if there isn't a root_bdm | |
| 19:09:45 | dansmith | if we run the scheduler and instance.image_ref is falsey it'll be happy? | |
| 19:10:50 | mriedem | we'd run the scheduler on the new image provided to rebuild | |
| 19:10:57 | mriedem | well, new or the same as the original | |
| 19:11:08 | dansmith | oh, I see | |
| 19:11:11 | dansmith | still, I dunno | |
| 19:11:13 | dansmith | seems dumb to me | |
| 19:11:16 | mriedem | there is this no_device flag on the bdmv1 stuff, i don't know what that does | |
| 19:11:39 | dansmith | heh | |
| 19:11:40 | mriedem | like, i don't know if there is a way to tell the api not to create you any bdms but still give you a volume-backed instance | |
| 19:50:57 | openstackgerrit | Brianna Poulos proposed openstack/nova master: Add trusted_certs object https://review.openstack.org/489408 | |
| 19:55:41 | openstackgerrit | Merged openstack/nova master: placement: add nested resource providers https://review.openstack.org/377138 | |
| 19:55:54 | edleafe | ^^ woo hoo! | |
| 20:01:23 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Raise specific exception when swapping migration allocations fails https://review.openstack.org/517004 | |
| 20:05:27 | openstackgerrit | Merged openstack/nova master: Remove deprecated TrustedFilter https://review.openstack.org/506864 | |
| 20:05:40 | openstackgerrit | Merged openstack/nova master: libvirt: do unicode conversion for error messages. https://review.openstack.org/517555 | |
| 20:07:11 | efried_rollin | cdent jaypipes I'm not rebasing nrp at the moment. I just peeled out one of the changes. Since the first one merged, the bottom three should currently be clean and reviewable. I wouldn't go above that. | |
| 20:09:35 | openstackgerrit | melanie witt proposed openstack/nova master: qemu-img do not use cache=none if no O_DIRECT support https://review.openstack.org/523554 | |
| 20:18:11 | openstackgerrit | Eric Berglund proposed openstack/nova master: Add PowerVM to compute_driver options https://review.openstack.org/523978 | |
| 20:26:15 | cdent | efried: thanks for the info. will try to get some reviewing in before I stop today | |
| 20:35:27 | mriedem | anyone else noticing a spike in cellsv1 job failures? | |
| 20:35:28 | mriedem | http://logs.openstack.org/10/523910/1/check/legacy-tempest-dsvm-cells/76a0551/logs/screen-n-sch.txt.gz#_Nov_29_16_05_42_186316 | |
| 20:37:34 | dansmith | that filter isn't on by default, right? are we forcing it on? | |
| 20:37:41 | mriedem | it's enabled in devstack | |
| 20:38:04 | dansmith | oh | |
| 20:38:21 | mriedem | this isn't a new test and logstash is not showing me a bunch of failures, so maybe just very random | |
| 20:38:55 | dansmith | and the test isn't requesting samehost= ? | |
| 20:39:02 | mriedem | it is | |
| 20:39:13 | mriedem | and it's a single node job | |
| 20:39:19 | mriedem | separate requests though, | |
| 20:39:30 | mriedem | so the HostState might not have the previous host tracked in memory when the 2nd request comes in? | |
| 20:40:22 | dansmith | idk | |
| 20:40:26 | dansmith | I mean, | |
| 20:40:33 | dansmith | if it's requesting a samehost then that has to be why | |
| 20:40:40 | dansmith | it's pretty simple | |
| 20:40:46 | efried | cdent You got a handle on how API fixtures are working in the functional test suites? | |
| 20:41:01 | mriedem | it's doing same_host on 'dad917c1-11bf-4aa1-9c3b-c864cdbc97c6' | |
| 20:41:03 | mriedem | http://logs.openstack.org/10/523910/1/check/legacy-tempest-dsvm-cells/76a0551/logs/screen-n-sch.txt.gz#_Nov_29_16_05_42_182254 | |
| 20:42:01 | cdent | efried: nova api fixtures? slightly. I made some adjustments to them recently. What’s up? | |
| 20:42:02 | mriedem | and when we request with the same_host, the HostState is tracking c9115b21-9355-4d1b-b797-62d2db7ca1ac and f5cb92ec-0474-4f4f-a1de-1d303e67e5f0 | |
| 20:43:13 | mriedem | we don't update the HostState with that instance until later http://logs.openstack.org/10/523910/1/check/legacy-tempest-dsvm-cells/76a0551/logs/screen-n-sch.txt.gz#_Nov_29_16_06_17_769927 | |
| 20:43:17 | mriedem | which is...odd | |
| 20:43:32 | mriedem | i thought we always updated the host state after we've "claimed" an instance on that host | |
| 20:43:34 | mriedem | edleafe: ^ | |
| 20:46:49 | edleafe | mriedem: https://github.com/openstack/nova/blob/master/nova/scheduler/filter_scheduler.py#L255 | |
| 20:47:03 | edleafe | mriedem: is this running filter_scheduler? | |
| 20:47:08 | mriedem | i guess _locked_consume_from_request doesn't actually add the "consumed" instance to it's list of running instances on that host | |
| 20:47:10 | mriedem | edleafe: yes | |
| 20:47:30 | mriedem | it has to wait for the track_instance_changes stuff from the compute to send an rpc message to the scheduler that the instance actually landed there? | |
| 20:47:34 | dansmith | mriedem: how many times have you seen this? | |
| 20:47:38 | mriedem | dansmith: just now | |
| 20:47:40 | dansmith | I thought you said logstash isn't crying? | |
| 20:47:43 | mriedem | it's not | |
| 20:47:46 | dansmith | okay | |
| 20:47:51 | mriedem | so, probably totally latent | |
| 20:47:56 | mriedem | and just running on a slow node or something | |
| 20:48:33 | mriedem | it just seems odd that we update the HostState in memory to consume resources from the request spec, but we don't actually count that instance as tracked on the chosen host at the same time | |
| 20:48:55 | mriedem | like, i get we might rebuild and the instance will end up on another host, but then the consumed resources on that HostState object are also wrong in that case | |
| 20:48:59 | mriedem | until they sync up | |
| 20:52:00 | mriedem | edleafe: yeah i'm surprised we don't call something like this during that "consume_from_request" call https://github.com/openstack/nova/blob/master/nova/scheduler/host_manager.py#L767 | |
| 20:52:39 | mriedem | we "consume" the resources but we don't track that instance against that host until it syncs up from the compute | |
| 20:53:15 | dansmith | well, that's the classic scheduler race right? | |
| 20:53:17 | mriedem | which, in superconductor separated cells v2 world, we can't rely on b/c the computes can't cast to the scheduler | |
| 20:53:22 | dansmith | we consume in the num_instances loop, | |
| 20:53:34 | openstackgerrit | Eric Berglund proposed openstack/nova master: Add PowerVM to compute_driver options https://review.openstack.org/523978 | |
| 20:53:35 | dansmith | but until it hits the compute it's still unclaimed | |
| 20:54:04 | dansmith | but placement claims should be sealing that off for everything now | |
| 20:54:35 | mriedem | sure, but the resource claim isn't the problem here, it's really the affinity stuff | |
| 20:54:39 | mriedem | which we know is still a gap | |
| 20:54:51 | dansmith | ah right | |
| 20:55:01 | mriedem | which now that i'm thinking about this, | |
| 20:55:18 | mriedem | i thought within a single request, we tracked which instances landed on which hosts so the affinity filters would work (within the same request) | |
| 20:55:25 | mriedem | but now i can't find where that happens | |
| 20:57:00 | melwitt | cells meeting in 4 minutes? | |
| 20:57:04 | dansmith | yep | |
| 20:57:25 | melwitt | cool | |
| 21:23:23 | dansmith | mriedem: so, yesterday I was thinking that this would be easier than what we do for instances because we don't have variable sort keys and such | |
| 21:23:26 | dansmith | and it is a little, | |
| 21:23:43 | dansmith | but the local/global marker thing still needs to happen, AFAIK, which makes it more similar I guess | |
| 21:24:28 | dansmith | it's quite unfortunate that we have to be able to do this tbh.. listing migrations per instance is a lot easier obviously | |
| 21:25:02 | dansmith | in general, I think that it's a good idea for us to try to avoid adding more operations that list multiple things that are not scoped to an instance, as that's kindof nova's primary key | |
| 21:25:19 | openstackgerrit | Brianna Poulos proposed openstack/nova master: Add trusted_image_certificates to REST API https://review.openstack.org/486204 | |
| 21:25:19 | openstackgerrit | Brianna Poulos proposed openstack/nova master: Add trusted_certs object https://review.openstack.org/489408 | |
| 21:26:53 | mriedem | well, listing things across cells is fine, like services and compute nodes | |
| 21:27:00 | mriedem | it's the paging that is killing this | |
| 21:27:24 | openstackgerrit | Brianna Poulos proposed openstack/nova master: Add trusted_image_certificates to REST API https://review.openstack.org/486204 | |
| 21:27:34 | dansmith | right, but services and compute nodes scale with nodes, | |
| 21:27:37 | dansmith | migrations scale with instances | |
| 21:27:53 | dansmith | listing bdms would be similarly painful, for example | |
| 21:28:41 | mriedem | yeah, we should have far fewer migrations than instances or bdms, but yeah - but that's also why they have this spec, to page across a large number of migrations | |
| 21:29:04 | mriedem | anyway, maybe his older PS was more correct https://review.openstack.org/#/c/517273/21/nova/compute/migration_list.py | |
| 21:29:09 | mriedem | because it was doing local marker stuff | |
| 21:29:09 | dansmith | I dunno, over time you probably end up with more migrations than instances, no? | |
| 21:29:22 | mriedem | we delete migrations when we delete instances | |