Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-08-03
13:42:23 openstackgerrit Stephen Finucane proposed openstack/nova stable/queens: Not use thread alloc policy for emulator thread https://review.openstack.org/588575
13:43:15 finucannot lyarwood: Fancy sticking those on your review queue? ^
13:44:00 lyarwood finucannot: sure thing
13:44:06 openstackgerrit Balazs Gibizer proposed openstack/nova master: Use placement 1.28 in scheduler report client https://review.openstack.org/583667
14:02:33 openstackgerrit Chris Dent proposed openstack/nova master: [placement] Move resource_class_cache into placement hierarchy https://review.openstack.org/584085
14:02:34 openstackgerrit Chris Dent proposed openstack/nova master: [placement] ensure_rc_cache only at start of process https://review.openstack.org/584086
14:26:12 melwitt dansmith: do you understand this bug? says guests can't retrieve metadata from the metadata API with multiple cells https://bugs.launchpad.net/nova/+bug/1785235 cc gnuoy
14:26:12 openstack Launchpad bug 1785235 in OpenStack Compute (nova) "metadata retrieval fails when using a global nova-api-metadata service" [Undecided,In progress] - Assigned to Liam Young (gnuoy)
14:27:26 dansmith well, I understand the words in the bug
14:27:30 melwitt I had thought guests retrieved metadata over http, not the MQ
14:27:31 cfriesen mriedem_afk: I think the idea was to allow a management layer to create a new compute node in the DB so that we can "disable" it, set up system-generated host aggregates, boot the node, do some health checks, then "enable" it once everything is ready.
14:27:46 dansmith I also understand that I'm going to -2 the code change
14:27:47 dansmith melwitt: of course they do,
14:27:49 melwitt okay. I didn't understand either
14:28:03 dansmith but they're saying that metadata service then doesn't hit the right db as a result
14:28:12 gnuoy yep
14:28:26 melwitt oh, I see
14:29:39 melwitt so, this method, or another one that we missed cell targeting in? https://github.com/openstack/nova/blob/master/nova/api/metadata/base.py#L677
14:30:04 dansmith melwitt: L692
14:30:46 melwitt right
14:31:16 melwitt okay
14:31:17 dansmith commented
14:32:31 gnuoy dansmith, I see context.target_cell getting called and adding the cells mq and db endpoints to cctxt
14:32:33 melwitt cool thanks
14:32:50 gnuoy but when the request executes those endpoints are ignored
14:33:23 dansmith gnuoy: did you open this bug?
14:33:26 gnuoy I did
14:33:45 dansmith gnuoy: please show some logs and config
14:34:15 gnuoy sure
14:36:40 dansmith gnuoy: are you running a standalone metadata server with cmd/api_metadata ?
14:38:24 gnuoy I am running a standalone metadata server. I don't follow the second part of the question
14:39:08 dansmith gnuoy: okay I think I see what's going on and why you want to make the change you're making
14:39:22 gnuoy ah, cool!
14:39:37 gnuoy dansmith, that change was a starter for 10, I'm happy to update it
14:39:46 dansmith gnuoy: so there are two ways to run metadata, either part of the regular api with enabled_apis=
14:40:06 dansmith and then with the standalone cmd/api_metadata thing which just starts up a standalone metadata server
14:40:19 dansmith the latter always forces the indirection api into place,
14:40:33 dansmith which was really intended for the case where you're running metadata on each compute node
14:40:34 gnuoy yep, I saw that
14:40:48 gnuoy ah
14:40:59 dansmith for the global case, you really should be running the regular api server, and just not enable the osapi api if you want to only run metadata
14:41:11 dansmith that will not install the indirection handler and do proper switching
14:41:14 dansmith and will perform better
14:41:49 gnuoy dansmith, ok, I will give that a try, thanks.
14:42:22 dansmith gnuoy: please confirm for us and assuming that shakes out, we should write a doc change to fix this bug
14:42:39 gnuoy dansmith, absolutely, thanks
14:43:48 mriedem_afk cfriesen: there is a config option to keep compute services disabled when they are first created
14:44:21 gnuoy dansmith, If the indirection api is not used will the metadata service expect to be able to talk directly to the cells individual dbs?
14:44:35 dansmith gnuoy: yeah
14:44:52 gnuoy oh, hmm, ok
14:45:11 dansmith but it's global, and thus the access pattern will look like all those other global ones
14:45:28 gnuoy right, I see
14:51:36 mriedem cfriesen: was a bug reported upstream for this? https://github.com/starlingx-staging/stx-nova/commit/71acfeae0d1c59fdc77704527d763bd85a276f9a#diff-516904cc81cade24a9122ecf96707bf0R3359
14:54:51 mriedem seems like something that could be deal with during _init_instance on restart of the compute service
14:57:53 mriedem cfriesen: heh i can get behind this :) https://github.com/starlingx-staging/stx-nova/commit/71acfeae0d1c59fdc77704527d763bd85a276f9a#diff-516904cc81cade24a9122ecf96707bf0R4217
14:59:47 melwitt can someone sanity check me on this? when configured to use qcow2 images with libvirt, the backing file on each compute host is expected to be raw format (not qcow2) being that each instance created on the compute will be a COW copy of it?
15:00:07 melwitt I'm trying to triage this https://bugs.launchpad.net/nova/+bug/1774730
15:00:07 openstack Launchpad bug 1774730 in OpenStack Compute (nova) "Compute node convert qcow2 to raw even if force_raw_images=false" [Undecided,New]
15:01:07 mriedem cfriesen: i could have sworn we had a patch up for this upstream too https://github.com/starlingx-staging/stx-nova/commit/71acfeae0d1c59fdc77704527d763bd85a276f9a#diff-516904cc81cade24a9122ecf96707bf0R4226
15:03:41 melwitt mdbooth: if you're around, question about force_raw_images=False and images_type=qcow2, the backing file is expected to be raw format right? since each instance is a COW copy of it? ^
15:04:35 mdbooth melwitt: I *think* the backing file is allowed to be qcow2 in that case
15:04:45 mdbooth We done some sanity checking on it when we import it
15:04:53 mdbooth So it's not allowed to have a backing file iirc
15:05:30 melwitt mdbooth: oh, hm okay. then we might be a bug there. thank you for the info
15:05:38 mdbooth qcow2 can have a backing file of qcow2
15:05:42 mdbooth melwitt: Got a link?
15:05:53 melwitt mdbooth: yes https://bugs.launchpad.net/nova/+bug/1774730
15:05:53 openstack Launchpad bug 1774730 in OpenStack Compute (nova) "Compute node convert qcow2 to raw even if force_raw_images=false" [Undecided,New]
15:07:06 mdbooth melwitt: lemme have a quick dig
15:07:19 cfriesen mriedem: the first one you mentioned (15min ago) was flagged as upstreamable, checking if we ever actually tried
15:08:27 leakypipes cfriesen: if the exact same request to placement "works" (i.e. returns >0 results) in one moment, and then "fails" (returns 0 results) a short time after, that isn't a "failure of the service's SLA".
15:08:49 leakypipes cfriesen: it's not a failure to return 0 results.
15:09:26 leakypipes cfriesen: the capacity to meet that particular request may easily have been exceeded by the first "successful" request's claim of those resources.
15:10:21 leakypipes cfriesen: SLAs are for things like "mean time to recover" or "mean time to respond". not for everyday occurrences and normal business of a service.
15:11:33 leakypipes cfriesen: for instance, if WRS claimed to its customers that the placement service would always return a result within 50 milliseconds, and placement returned a result in 2 seconds, that would be a failure of the SLA. But if the placement service returns 0 results in 20 milliseconds, that's not a failure of the SLA.
15:12:49 cfriesen leakypipes: doesn't that depend what's in the SLA? If I'm Netflix, I could have an agreement with Amazon saying that I'll always be able to burst by X additional resources.
15:13:10 leakypipes cfriesen: that has nothing to do with the placement service, and you know it. :)
15:13:32 fried_rice oh, this conversation is happening.
15:13:45 fried_rice I just sent this in an email, prettymuch.
15:14:07 cfriesen leakypipes: true, but it's a failure to provide resources that are supposed to be available. it's not an exception in placement, but it's arguably exceptional for the provider.
15:14:30 cfriesen like I said in my email, I'm of two minds.
15:14:31 fried_rice ^ this exactly.
15:14:49 leakypipes cfriesen: who said the resources are "supposed to be available"? that's crazy talk, friend.
15:15:08 cfriesen leakypipes: my hypothetical guarantee to netflix that they can always burst by X
15:15:08 sean-k-mooney cfriesen: isnt that what blazer is for
15:15:15 fried_rice Nova certainly isn't in the business of enforcing, or even knowing about SLAs
15:15:23 fried_rice yeah, was gonna say, that's some service way above nova.
15:15:47 leakypipes fried_rice: or *multiple* services (running as a SaaS system or orchestrator of some sort or whatever)
15:15:54 fried_rice fosho
15:16:16 leakypipes cfriesen: nobody other than a service provider can or would make such a guarantee.
15:16:25 mdbooth melwitt: fwiw, I can't immediately see how that's possible.
15:16:25 leakypipes cfriesen: we aren't a service provider. we're a placement service.
15:16:26 fried_rice We're talking about FFDC where the second F is only a F from the perspective of something waaay above placement.
15:16:51 cfriesen fried_rice: okay, but now you have those services trying to figure out why the request couldn't be met, and I can see how it would be nice to have an exception object with nice logs in it rather than sending an operator digging through logs.
15:16:53 melwitt mdbooth: okay, that is odd. thank you for taking a look
15:17:01 leakypipes cfriesen, fried_rice: move this to #openstack-placement before we get told off...
15:17:24 fried_rice yuh, swhy I didn't notice the conversation until after I had sent my email.
15:17:35 mdbooth melwitt: I'd definitely want to see logs and config. Immediate suspect is conf error.
15:18:05 melwitt mdbooth: okay, that's helpful. I ask the reporter for more info
15:18:08 melwitt thanks

Earlier   Later