Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-08-03
14:32:31 gnuoy dansmith, I see context.target_cell getting called and adding the cells mq and db endpoints to cctxt
14:32:33 melwitt cool thanks
14:32:50 gnuoy but when the request executes those endpoints are ignored
14:33:23 dansmith gnuoy: did you open this bug?
14:33:26 gnuoy I did
14:33:45 dansmith gnuoy: please show some logs and config
14:34:15 gnuoy sure
14:36:40 dansmith gnuoy: are you running a standalone metadata server with cmd/api_metadata ?
14:38:24 gnuoy I am running a standalone metadata server. I don't follow the second part of the question
14:39:08 dansmith gnuoy: okay I think I see what's going on and why you want to make the change you're making
14:39:22 gnuoy ah, cool!
14:39:37 gnuoy dansmith, that change was a starter for 10, I'm happy to update it
14:39:46 dansmith gnuoy: so there are two ways to run metadata, either part of the regular api with enabled_apis=
14:40:06 dansmith and then with the standalone cmd/api_metadata thing which just starts up a standalone metadata server
14:40:19 dansmith the latter always forces the indirection api into place,
14:40:33 dansmith which was really intended for the case where you're running metadata on each compute node
14:40:34 gnuoy yep, I saw that
14:40:48 gnuoy ah
14:40:59 dansmith for the global case, you really should be running the regular api server, and just not enable the osapi api if you want to only run metadata
14:41:11 dansmith that will not install the indirection handler and do proper switching
14:41:14 dansmith and will perform better
14:41:49 gnuoy dansmith, ok, I will give that a try, thanks.
14:42:22 dansmith gnuoy: please confirm for us and assuming that shakes out, we should write a doc change to fix this bug
14:42:39 gnuoy dansmith, absolutely, thanks
14:43:48 mriedem_afk cfriesen: there is a config option to keep compute services disabled when they are first created
14:44:21 gnuoy dansmith, If the indirection api is not used will the metadata service expect to be able to talk directly to the cells individual dbs?
14:44:35 dansmith gnuoy: yeah
14:44:52 gnuoy oh, hmm, ok
14:45:11 dansmith but it's global, and thus the access pattern will look like all those other global ones
14:45:28 gnuoy right, I see
14:51:36 mriedem cfriesen: was a bug reported upstream for this? https://github.com/starlingx-staging/stx-nova/commit/71acfeae0d1c59fdc77704527d763bd85a276f9a#diff-516904cc81cade24a9122ecf96707bf0R3359
14:54:51 mriedem seems like something that could be deal with during _init_instance on restart of the compute service
14:57:53 mriedem cfriesen: heh i can get behind this :) https://github.com/starlingx-staging/stx-nova/commit/71acfeae0d1c59fdc77704527d763bd85a276f9a#diff-516904cc81cade24a9122ecf96707bf0R4217
14:59:47 melwitt can someone sanity check me on this? when configured to use qcow2 images with libvirt, the backing file on each compute host is expected to be raw format (not qcow2) being that each instance created on the compute will be a COW copy of it?
15:00:07 melwitt I'm trying to triage this https://bugs.launchpad.net/nova/+bug/1774730
15:00:07 openstack Launchpad bug 1774730 in OpenStack Compute (nova) "Compute node convert qcow2 to raw even if force_raw_images=false" [Undecided,New]
15:01:07 mriedem cfriesen: i could have sworn we had a patch up for this upstream too https://github.com/starlingx-staging/stx-nova/commit/71acfeae0d1c59fdc77704527d763bd85a276f9a#diff-516904cc81cade24a9122ecf96707bf0R4226
15:03:41 melwitt mdbooth: if you're around, question about force_raw_images=False and images_type=qcow2, the backing file is expected to be raw format right? since each instance is a COW copy of it? ^
15:04:35 mdbooth melwitt: I *think* the backing file is allowed to be qcow2 in that case
15:04:45 mdbooth We done some sanity checking on it when we import it
15:04:53 mdbooth So it's not allowed to have a backing file iirc
15:05:30 melwitt mdbooth: oh, hm okay. then we might be a bug there. thank you for the info
15:05:38 mdbooth qcow2 can have a backing file of qcow2
15:05:42 mdbooth melwitt: Got a link?
15:05:53 melwitt mdbooth: yes https://bugs.launchpad.net/nova/+bug/1774730
15:05:53 openstack Launchpad bug 1774730 in OpenStack Compute (nova) "Compute node convert qcow2 to raw even if force_raw_images=false" [Undecided,New]
15:07:06 mdbooth melwitt: lemme have a quick dig
15:07:19 cfriesen mriedem: the first one you mentioned (15min ago) was flagged as upstreamable, checking if we ever actually tried
15:08:27 leakypipes cfriesen: if the exact same request to placement "works" (i.e. returns >0 results) in one moment, and then "fails" (returns 0 results) a short time after, that isn't a "failure of the service's SLA".
15:08:49 leakypipes cfriesen: it's not a failure to return 0 results.
15:09:26 leakypipes cfriesen: the capacity to meet that particular request may easily have been exceeded by the first "successful" request's claim of those resources.
15:10:21 leakypipes cfriesen: SLAs are for things like "mean time to recover" or "mean time to respond". not for everyday occurrences and normal business of a service.
15:11:33 leakypipes cfriesen: for instance, if WRS claimed to its customers that the placement service would always return a result within 50 milliseconds, and placement returned a result in 2 seconds, that would be a failure of the SLA. But if the placement service returns 0 results in 20 milliseconds, that's not a failure of the SLA.
15:12:49 cfriesen leakypipes: doesn't that depend what's in the SLA? If I'm Netflix, I could have an agreement with Amazon saying that I'll always be able to burst by X additional resources.
15:13:10 leakypipes cfriesen: that has nothing to do with the placement service, and you know it. :)
15:13:32 fried_rice oh, this conversation is happening.
15:13:45 fried_rice I just sent this in an email, prettymuch.
15:14:07 cfriesen leakypipes: true, but it's a failure to provide resources that are supposed to be available. it's not an exception in placement, but it's arguably exceptional for the provider.
15:14:30 cfriesen like I said in my email, I'm of two minds.
15:14:31 fried_rice ^ this exactly.
15:14:49 leakypipes cfriesen: who said the resources are "supposed to be available"? that's crazy talk, friend.
15:15:08 cfriesen leakypipes: my hypothetical guarantee to netflix that they can always burst by X
15:15:08 sean-k-mooney cfriesen: isnt that what blazer is for
15:15:15 fried_rice Nova certainly isn't in the business of enforcing, or even knowing about SLAs
15:15:23 fried_rice yeah, was gonna say, that's some service way above nova.
15:15:47 leakypipes fried_rice: or *multiple* services (running as a SaaS system or orchestrator of some sort or whatever)
15:15:54 fried_rice fosho
15:16:16 leakypipes cfriesen: nobody other than a service provider can or would make such a guarantee.
15:16:25 mdbooth melwitt: fwiw, I can't immediately see how that's possible.
15:16:25 leakypipes cfriesen: we aren't a service provider. we're a placement service.
15:16:26 fried_rice We're talking about FFDC where the second F is only a F from the perspective of something waaay above placement.
15:16:51 cfriesen fried_rice: okay, but now you have those services trying to figure out why the request couldn't be met, and I can see how it would be nice to have an exception object with nice logs in it rather than sending an operator digging through logs.
15:16:53 melwitt mdbooth: okay, that is odd. thank you for taking a look
15:17:01 leakypipes cfriesen, fried_rice: move this to #openstack-placement before we get told off...
15:17:24 fried_rice yuh, swhy I didn't notice the conversation until after I had sent my email.
15:17:35 mdbooth melwitt: I'd definitely want to see logs and config. Immediate suspect is conf error.
15:18:05 melwitt mdbooth: okay, that's helpful. I ask the reporter for more info
15:18:08 melwitt thanks
15:24:21 openstackgerrit Matt Riedemann proposed openstack/nova master: Avoid live migrate to same host https://review.openstack.org/542689
15:28:02 openstackgerrit Chris Dent proposed openstack/nova master: [placement] ensure_rc_cache only at start of process https://review.openstack.org/584086
15:28:10 cfriesen mriedem: I don't see an upstream bug report. probably just got missed, so I opened one. https://bugs.launchpad.net/nova/+bug/1785270
15:28:10 openstack Launchpad bug 1785270 in OpenStack Compute (nova) "allow confirmation of resize/migration for migrations in "confirming" status" [Undecided,New]
15:33:41 mriedem danke
15:34:39 cfriesen mriedem: I think we now have the ability to set RUN_ON_REBUILD to enforce validating the image on rebuild. I'm not aware of a similar thing to enforce always going through the scheduler for live migration, though I think I talked about it with dansmith.
15:36:45 cfriesen gotta step out for a bit....back later.
15:38:03 mriedem cfriesen_afk: i see you guys removed the force flag for live migrate so you can't do that, which means you'd go through the scheduler, but that *doesnt* apply to live migrations before the microversion that added the force flag because in those cases, simply specifying a host bypasses the scheduler
15:38:46 mriedem RUN_ON_REBUILD is only b/c we don't actually move hosts on rebuild
15:41:48 openstackgerrit Matt Riedemann proposed openstack/nova master: Deprecate upgrade_levels options for deprecated/removed services https://review.openstack.org/588607
15:41:52 mriedem dansmith: might want to get that into rocky ^
15:41:54 mriedem to start the timer
15:41:59 mriedem melwitt: ^ you too given nova-consoleauth
15:43:51 melwitt ack
15:46:18 dansmith yar
15:49:55 melwitt mriedem: TYPO
15:50:03 melwitt in the reno
15:50:24 dansmith WUT? NO.
15:50:29 melwitt YUH HUH
15:50:31 dansmith zomg
15:50:43 dansmith let it be known on the third day of august, the year of our lord 2018...
15:53:13 openstackgerrit Matt Riedemann proposed openstack/nova master: Deprecate upgrade_levels options for deprecated/removed services https://review.openstack.org/588607

Earlier   Later