| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-08-03 | |||
| 14:30:04 | dansmith | melwitt: L692 | |
| 14:30:46 | melwitt | right | |
| 14:31:16 | melwitt | okay | |
| 14:31:17 | dansmith | commented | |
| 14:32:31 | gnuoy | dansmith, I see context.target_cell getting called and adding the cells mq and db endpoints to cctxt | |
| 14:32:33 | melwitt | cool thanks | |
| 14:32:50 | gnuoy | but when the request executes those endpoints are ignored | |
| 14:33:23 | dansmith | gnuoy: did you open this bug? | |
| 14:33:26 | gnuoy | I did | |
| 14:33:45 | dansmith | gnuoy: please show some logs and config | |
| 14:34:15 | gnuoy | sure | |
| 14:36:40 | dansmith | gnuoy: are you running a standalone metadata server with cmd/api_metadata ? | |
| 14:38:24 | gnuoy | I am running a standalone metadata server. I don't follow the second part of the question | |
| 14:39:08 | dansmith | gnuoy: okay I think I see what's going on and why you want to make the change you're making | |
| 14:39:22 | gnuoy | ah, cool! | |
| 14:39:37 | gnuoy | dansmith, that change was a starter for 10, I'm happy to update it | |
| 14:39:46 | dansmith | gnuoy: so there are two ways to run metadata, either part of the regular api with enabled_apis= | |
| 14:40:06 | dansmith | and then with the standalone cmd/api_metadata thing which just starts up a standalone metadata server | |
| 14:40:19 | dansmith | the latter always forces the indirection api into place, | |
| 14:40:33 | dansmith | which was really intended for the case where you're running metadata on each compute node | |
| 14:40:34 | gnuoy | yep, I saw that | |
| 14:40:48 | gnuoy | ah | |
| 14:40:59 | dansmith | for the global case, you really should be running the regular api server, and just not enable the osapi api if you want to only run metadata | |
| 14:41:11 | dansmith | that will not install the indirection handler and do proper switching | |
| 14:41:14 | dansmith | and will perform better | |
| 14:41:49 | gnuoy | dansmith, ok, I will give that a try, thanks. | |
| 14:42:22 | dansmith | gnuoy: please confirm for us and assuming that shakes out, we should write a doc change to fix this bug | |
| 14:42:39 | gnuoy | dansmith, absolutely, thanks | |
| 14:43:48 | mriedem_afk | cfriesen: there is a config option to keep compute services disabled when they are first created | |
| 14:44:21 | gnuoy | dansmith, If the indirection api is not used will the metadata service expect to be able to talk directly to the cells individual dbs? | |
| 14:44:35 | dansmith | gnuoy: yeah | |
| 14:44:52 | gnuoy | oh, hmm, ok | |
| 14:45:11 | dansmith | but it's global, and thus the access pattern will look like all those other global ones | |
| 14:45:28 | gnuoy | right, I see | |
| 14:51:36 | mriedem | cfriesen: was a bug reported upstream for this? https://github.com/starlingx-staging/stx-nova/commit/71acfeae0d1c59fdc77704527d763bd85a276f9a#diff-516904cc81cade24a9122ecf96707bf0R3359 | |
| 14:54:51 | mriedem | seems like something that could be deal with during _init_instance on restart of the compute service | |
| 14:57:53 | mriedem | cfriesen: heh i can get behind this :) https://github.com/starlingx-staging/stx-nova/commit/71acfeae0d1c59fdc77704527d763bd85a276f9a#diff-516904cc81cade24a9122ecf96707bf0R4217 | |
| 14:59:47 | melwitt | can someone sanity check me on this? when configured to use qcow2 images with libvirt, the backing file on each compute host is expected to be raw format (not qcow2) being that each instance created on the compute will be a COW copy of it? | |
| 15:00:07 | openstack | Launchpad bug 1774730 in OpenStack Compute (nova) "Compute node convert qcow2 to raw even if force_raw_images=false" [Undecided,New] | |
| 15:00:07 | melwitt | I'm trying to triage this https://bugs.launchpad.net/nova/+bug/1774730 | |
| 15:01:07 | mriedem | cfriesen: i could have sworn we had a patch up for this upstream too https://github.com/starlingx-staging/stx-nova/commit/71acfeae0d1c59fdc77704527d763bd85a276f9a#diff-516904cc81cade24a9122ecf96707bf0R4226 | |
| 15:03:41 | melwitt | mdbooth: if you're around, question about force_raw_images=False and images_type=qcow2, the backing file is expected to be raw format right? since each instance is a COW copy of it? ^ | |
| 15:04:35 | mdbooth | melwitt: I *think* the backing file is allowed to be qcow2 in that case | |
| 15:04:45 | mdbooth | We done some sanity checking on it when we import it | |
| 15:04:53 | mdbooth | So it's not allowed to have a backing file iirc | |
| 15:05:30 | melwitt | mdbooth: oh, hm okay. then we might be a bug there. thank you for the info | |
| 15:05:38 | mdbooth | qcow2 can have a backing file of qcow2 | |
| 15:05:42 | mdbooth | melwitt: Got a link? | |
| 15:05:53 | openstack | Launchpad bug 1774730 in OpenStack Compute (nova) "Compute node convert qcow2 to raw even if force_raw_images=false" [Undecided,New] | |
| 15:05:53 | melwitt | mdbooth: yes https://bugs.launchpad.net/nova/+bug/1774730 | |
| 15:07:06 | mdbooth | melwitt: lemme have a quick dig | |
| 15:07:19 | cfriesen | mriedem: the first one you mentioned (15min ago) was flagged as upstreamable, checking if we ever actually tried | |
| 15:08:27 | leakypipes | cfriesen: if the exact same request to placement "works" (i.e. returns >0 results) in one moment, and then "fails" (returns 0 results) a short time after, that isn't a "failure of the service's SLA". | |
| 15:08:49 | leakypipes | cfriesen: it's not a failure to return 0 results. | |
| 15:09:26 | leakypipes | cfriesen: the capacity to meet that particular request may easily have been exceeded by the first "successful" request's claim of those resources. | |
| 15:10:21 | leakypipes | cfriesen: SLAs are for things like "mean time to recover" or "mean time to respond". not for everyday occurrences and normal business of a service. | |
| 15:11:33 | leakypipes | cfriesen: for instance, if WRS claimed to its customers that the placement service would always return a result within 50 milliseconds, and placement returned a result in 2 seconds, that would be a failure of the SLA. But if the placement service returns 0 results in 20 milliseconds, that's not a failure of the SLA. | |
| 15:12:49 | cfriesen | leakypipes: doesn't that depend what's in the SLA? If I'm Netflix, I could have an agreement with Amazon saying that I'll always be able to burst by X additional resources. | |
| 15:13:10 | leakypipes | cfriesen: that has nothing to do with the placement service, and you know it. :) | |
| 15:13:32 | fried_rice | oh, this conversation is happening. | |
| 15:13:45 | fried_rice | I just sent this in an email, prettymuch. | |
| 15:14:07 | cfriesen | leakypipes: true, but it's a failure to provide resources that are supposed to be available. it's not an exception in placement, but it's arguably exceptional for the provider. | |
| 15:14:30 | cfriesen | like I said in my email, I'm of two minds. | |
| 15:14:31 | fried_rice | ^ this exactly. | |
| 15:14:49 | leakypipes | cfriesen: who said the resources are "supposed to be available"? that's crazy talk, friend. | |
| 15:15:08 | sean-k-mooney | cfriesen: isnt that what blazer is for | |
| 15:15:08 | cfriesen | leakypipes: my hypothetical guarantee to netflix that they can always burst by X | |
| 15:15:15 | fried_rice | Nova certainly isn't in the business of enforcing, or even knowing about SLAs | |
| 15:15:23 | fried_rice | yeah, was gonna say, that's some service way above nova. | |
| 15:15:47 | leakypipes | fried_rice: or *multiple* services (running as a SaaS system or orchestrator of some sort or whatever) | |
| 15:15:54 | fried_rice | fosho | |
| 15:16:16 | leakypipes | cfriesen: nobody other than a service provider can or would make such a guarantee. | |
| 15:16:25 | leakypipes | cfriesen: we aren't a service provider. we're a placement service. | |
| 15:16:25 | mdbooth | melwitt: fwiw, I can't immediately see how that's possible. | |
| 15:16:26 | fried_rice | We're talking about FFDC where the second F is only a F from the perspective of something waaay above placement. | |
| 15:16:51 | cfriesen | fried_rice: okay, but now you have those services trying to figure out why the request couldn't be met, and I can see how it would be nice to have an exception object with nice logs in it rather than sending an operator digging through logs. | |
| 15:16:53 | melwitt | mdbooth: okay, that is odd. thank you for taking a look | |
| 15:17:01 | leakypipes | cfriesen, fried_rice: move this to #openstack-placement before we get told off... | |
| 15:17:24 | fried_rice | yuh, swhy I didn't notice the conversation until after I had sent my email. | |
| 15:17:35 | mdbooth | melwitt: I'd definitely want to see logs and config. Immediate suspect is conf error. | |
| 15:18:05 | melwitt | mdbooth: okay, that's helpful. I ask the reporter for more info | |
| 15:18:08 | melwitt | thanks | |
| 15:24:21 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Avoid live migrate to same host https://review.openstack.org/542689 | |
| 15:28:02 | openstackgerrit | Chris Dent proposed openstack/nova master: [placement] ensure_rc_cache only at start of process https://review.openstack.org/584086 | |
| 15:28:10 | openstack | Launchpad bug 1785270 in OpenStack Compute (nova) "allow confirmation of resize/migration for migrations in "confirming" status" [Undecided,New] | |
| 15:28:10 | cfriesen | mriedem: I don't see an upstream bug report. probably just got missed, so I opened one. https://bugs.launchpad.net/nova/+bug/1785270 | |
| 15:33:41 | mriedem | danke | |
| 15:34:39 | cfriesen | mriedem: I think we now have the ability to set RUN_ON_REBUILD to enforce validating the image on rebuild. I'm not aware of a similar thing to enforce always going through the scheduler for live migration, though I think I talked about it with dansmith. | |
| 15:36:45 | cfriesen | gotta step out for a bit....back later. | |
| 15:38:03 | mriedem | cfriesen_afk: i see you guys removed the force flag for live migrate so you can't do that, which means you'd go through the scheduler, but that *doesnt* apply to live migrations before the microversion that added the force flag because in those cases, simply specifying a host bypasses the scheduler | |
| 15:38:46 | mriedem | RUN_ON_REBUILD is only b/c we don't actually move hosts on rebuild | |
| 15:41:48 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Deprecate upgrade_levels options for deprecated/removed services https://review.openstack.org/588607 | |
| 15:41:52 | mriedem | dansmith: might want to get that into rocky ^ | |
| 15:41:54 | mriedem | to start the timer | |
| 15:41:59 | mriedem | melwitt: ^ you too given nova-consoleauth | |
| 15:43:51 | melwitt | ack | |
| 15:46:18 | dansmith | yar | |
| 15:49:55 | melwitt | mriedem: TYPO | |
| 15:50:03 | melwitt | in the reno | |
| 15:50:24 | dansmith | WUT? NO. | |