| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-11-29 | |||
| 19:00:16 | mriedem | that reminds me, | |
| 19:00:33 | mriedem | jaypipes: did i tell you how 2 different groups of people approached me about their worries about deprecating the ability to do file injection? | |
| 19:00:34 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Get original image_id from volume for volume-backed instance rebuild https://review.openstack.org/521391 | |
| 19:00:34 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add regression test for rebuilding a volume-backed server https://review.openstack.org/521200 | |
| 19:00:35 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Fail fast if changing image on a volume-backed server rebuild https://review.openstack.org/520660 | |
| 19:00:39 | mriedem | at the summit | |
| 19:00:47 | mriedem | verizon and ericsson | |
| 19:00:50 | dansmith | only two? | |
| 19:00:56 | jaypipes | mriedem: yeah, only two? | |
| 19:01:02 | INeedaUUID | jaypipes: I could tell you. But then i'd have to murder you. | |
| 19:01:04 | dansmith | two seems like a mandate to me | |
| 19:01:06 | mriedem | well, most people only say things 3 years after the change is merged | |
| 19:01:44 | mriedem | mikal told the verizon guys that their vnf vendor was "shit" in his aussy words and that they should talk to him | |
| 19:01:48 | jaypipes | mriedem: also, it's probably all rooted in the same dependency on a specific particular vendor VNF or MANO system that requires file injection (and stuff like multi-attach boot-from-volume | |
| 19:02:36 | mriedem | multiattach bfv is the bees knees | |
| 19:02:49 | mriedem | i wonder how rebuild is going to handle that... | |
| 19:03:02 | dansmith | probably as well as it handles everything else | |
| 19:03:12 | dansmith | by opening security holes, leaking resource accounting, and destroying data | |
| 19:03:22 | jaypipes | luckily that shotgun I ordered through the mail should be here by the time multi-attach bfv rolls into Nova. | |
| 19:03:33 | dansmith | jaypipes: you're welcome, from america | |
| 19:03:39 | jaypipes | murka | |
| 19:03:50 | mriedem | dansmith: ok i did the deeds in https://review.openstack.org/#/c/521391/3/nova/compute/api.py | |
| 19:04:02 | mriedem | if i can't find a root bdm and image_ref is '', i just log an error and keep trucking | |
| 19:04:45 | mriedem | i don't know if that case is some scary pre-bdmv1 stuff or what, it predates me | |
| 19:05:45 | dansmith | I don't understand where that's ever going to happen | |
| 19:06:10 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Fail fast if changing image on a volume-backed server rebuild https://review.openstack.org/520660 | |
| 19:06:11 | mriedem | i don't either | |
| 19:06:18 | mriedem | which is why i just log an error and punt | |
| 19:06:34 | dansmith | but you don't punt, | |
| 19:06:50 | dansmith | you set it to None, which causes the need for that extra else and then kick the can to the next thing in line | |
| 19:06:56 | dansmith | why not just dump ass right there and stop? | |
| 19:07:30 | mriedem | we could, but what do we raise? it's a 500 if we raise i think | |
| 19:07:53 | dansmith | is that less right than logging an error and saying "this is just going to break in the next service down" ? | |
| 19:08:08 | mriedem | at this point it would run through the scheduler with the provided image | |
| 19:08:24 | mriedem | which is basically what this was trying to fix - not doing that if the image doesn't change | |
| 19:08:42 | mriedem | the final patch in that series will fail though | |
| 19:08:56 | mriedem | i'm cool with a 500 if there isn't a root_bdm | |
| 19:09:45 | dansmith | if we run the scheduler and instance.image_ref is falsey it'll be happy? | |
| 19:10:50 | mriedem | we'd run the scheduler on the new image provided to rebuild | |
| 19:10:57 | mriedem | well, new or the same as the original | |
| 19:11:08 | dansmith | oh, I see | |
| 19:11:11 | dansmith | still, I dunno | |
| 19:11:13 | dansmith | seems dumb to me | |
| 19:11:16 | mriedem | there is this no_device flag on the bdmv1 stuff, i don't know what that does | |
| 19:11:39 | dansmith | heh | |
| 19:11:40 | mriedem | like, i don't know if there is a way to tell the api not to create you any bdms but still give you a volume-backed instance | |
| 19:50:57 | openstackgerrit | Brianna Poulos proposed openstack/nova master: Add trusted_certs object https://review.openstack.org/489408 | |
| 19:55:41 | openstackgerrit | Merged openstack/nova master: placement: add nested resource providers https://review.openstack.org/377138 | |
| 19:55:54 | edleafe | ^^ woo hoo! | |
| 20:01:23 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Raise specific exception when swapping migration allocations fails https://review.openstack.org/517004 | |
| 20:05:27 | openstackgerrit | Merged openstack/nova master: Remove deprecated TrustedFilter https://review.openstack.org/506864 | |
| 20:05:40 | openstackgerrit | Merged openstack/nova master: libvirt: do unicode conversion for error messages. https://review.openstack.org/517555 | |
| 20:07:11 | efried_rollin | cdent jaypipes I'm not rebasing nrp at the moment. I just peeled out one of the changes. Since the first one merged, the bottom three should currently be clean and reviewable. I wouldn't go above that. | |
| 20:09:35 | openstackgerrit | melanie witt proposed openstack/nova master: qemu-img do not use cache=none if no O_DIRECT support https://review.openstack.org/523554 | |
| 20:18:11 | openstackgerrit | Eric Berglund proposed openstack/nova master: Add PowerVM to compute_driver options https://review.openstack.org/523978 | |
| 20:26:15 | cdent | efried: thanks for the info. will try to get some reviewing in before I stop today | |
| 20:35:27 | mriedem | anyone else noticing a spike in cellsv1 job failures? | |
| 20:35:28 | mriedem | http://logs.openstack.org/10/523910/1/check/legacy-tempest-dsvm-cells/76a0551/logs/screen-n-sch.txt.gz#_Nov_29_16_05_42_186316 | |
| 20:37:34 | dansmith | that filter isn't on by default, right? are we forcing it on? | |
| 20:37:41 | mriedem | it's enabled in devstack | |
| 20:38:04 | dansmith | oh | |
| 20:38:21 | mriedem | this isn't a new test and logstash is not showing me a bunch of failures, so maybe just very random | |
| 20:38:55 | dansmith | and the test isn't requesting samehost= ? | |
| 20:39:02 | mriedem | it is | |
| 20:39:13 | mriedem | and it's a single node job | |
| 20:39:19 | mriedem | separate requests though, | |
| 20:39:30 | mriedem | so the HostState might not have the previous host tracked in memory when the 2nd request comes in? | |
| 20:40:22 | dansmith | idk | |
| 20:40:26 | dansmith | I mean, | |
| 20:40:33 | dansmith | if it's requesting a samehost then that has to be why | |
| 20:40:40 | dansmith | it's pretty simple | |
| 20:40:46 | efried | cdent You got a handle on how API fixtures are working in the functional test suites? | |
| 20:41:01 | mriedem | it's doing same_host on 'dad917c1-11bf-4aa1-9c3b-c864cdbc97c6' | |
| 20:41:03 | mriedem | http://logs.openstack.org/10/523910/1/check/legacy-tempest-dsvm-cells/76a0551/logs/screen-n-sch.txt.gz#_Nov_29_16_05_42_182254 | |
| 20:42:01 | cdent | efried: nova api fixtures? slightly. I made some adjustments to them recently. What’s up? | |
| 20:42:02 | mriedem | and when we request with the same_host, the HostState is tracking c9115b21-9355-4d1b-b797-62d2db7ca1ac and f5cb92ec-0474-4f4f-a1de-1d303e67e5f0 | |
| 20:43:13 | mriedem | we don't update the HostState with that instance until later http://logs.openstack.org/10/523910/1/check/legacy-tempest-dsvm-cells/76a0551/logs/screen-n-sch.txt.gz#_Nov_29_16_06_17_769927 | |
| 20:43:17 | mriedem | which is...odd | |
| 20:43:32 | mriedem | i thought we always updated the host state after we've "claimed" an instance on that host | |
| 20:43:34 | mriedem | edleafe: ^ | |
| 20:46:49 | edleafe | mriedem: https://github.com/openstack/nova/blob/master/nova/scheduler/filter_scheduler.py#L255 | |
| 20:47:03 | edleafe | mriedem: is this running filter_scheduler? | |
| 20:47:08 | mriedem | i guess _locked_consume_from_request doesn't actually add the "consumed" instance to it's list of running instances on that host | |
| 20:47:10 | mriedem | edleafe: yes | |
| 20:47:30 | mriedem | it has to wait for the track_instance_changes stuff from the compute to send an rpc message to the scheduler that the instance actually landed there? | |
| 20:47:34 | dansmith | mriedem: how many times have you seen this? | |
| 20:47:38 | mriedem | dansmith: just now | |
| 20:47:40 | dansmith | I thought you said logstash isn't crying? | |
| 20:47:43 | mriedem | it's not | |
| 20:47:46 | dansmith | okay | |
| 20:47:51 | mriedem | so, probably totally latent | |
| 20:47:56 | mriedem | and just running on a slow node or something | |
| 20:48:33 | mriedem | it just seems odd that we update the HostState in memory to consume resources from the request spec, but we don't actually count that instance as tracked on the chosen host at the same time | |
| 20:48:55 | mriedem | like, i get we might rebuild and the instance will end up on another host, but then the consumed resources on that HostState object are also wrong in that case | |
| 20:48:59 | mriedem | until they sync up | |
| 20:52:00 | mriedem | edleafe: yeah i'm surprised we don't call something like this during that "consume_from_request" call https://github.com/openstack/nova/blob/master/nova/scheduler/host_manager.py#L767 | |
| 20:52:39 | mriedem | we "consume" the resources but we don't track that instance against that host until it syncs up from the compute | |
| 20:53:15 | dansmith | well, that's the classic scheduler race right? | |
| 20:53:17 | mriedem | which, in superconductor separated cells v2 world, we can't rely on b/c the computes can't cast to the scheduler | |
| 20:53:22 | dansmith | we consume in the num_instances loop, | |