Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-03-20
14:54:04 mriedem but it's enterprise ready
14:54:07 dansmith lol
15:05:09 mriedem how do we know which flavor was actually used in the server create request?
15:05:22 mriedem EmilienM: you pointed at http://logs.openstack.org/periodic/periodic-tripleo-ci-centos-7-ovb-nonha/12d034e/logs/undercloud/var/log/nova/nova-api.txt.gz#_2017-03-19_06_38_20_764 but there are more than that flavor
15:05:23 dansmith mriedem: you can look at the placement request
15:05:29 mriedem ok
15:05:46 mriedem we don't correlate the request ids between the nova and placement services yet do we?
15:05:46 dansmith mriedem: http://logs.openstack.org/periodic/periodic-tripleo-ci-centos-7-ovb-nonha/12d034e/logs/undercloud/var/log/nova/nova-placement-api.txt.gz#_2017-03-19_07_19_34_113
15:05:58 dansmith mriedem: bauzas was just telling me that we do
15:06:10 mriedem cdent: you were the last to look at that i thought
15:06:32 dansmith mriedem: looks like the request I linked matches the 40,4g,1 flavor
15:06:42 EmilienM jroll: https://review.openstack.org/#/q/topic:bug/1674375
15:06:43 bauzas dansmith: mriedem: enf of Ocata-3, we were not yet passing req-id
15:07:02 bauzas dansmith: mriedem: but I thought cdent threw a patch for that
15:07:03 mriedem i'm looking for req-702e96af-318d-4611-af19-d6a1f3321c6a
15:07:09 mriedem http://logs.openstack.org/periodic/periodic-tripleo-ci-centos-7-ovb-nonha/12d034e/logs/undercloud/var/log/nova/nova-api.txt.gz#_2017-03-19_07_19_33_201
15:07:13 mriedem that's the ceph storage instance
15:07:41 mriedem i know we log the placement request id from the response in the scheduler report client
15:07:54 mriedem but we don't pass and use a request id from n-sch to placement and log with that req id in the placement logs
15:07:57 mriedem which is what we need
15:08:16 jroll we don't do that between n-cpu and ironic either :(
15:08:17 dansmith mriedem: they all look the same to me
15:08:26 openstackgerrit Sean Dague proposed openstack/nova master: update i18n guide for nova https://review.openstack.org/447528
15:08:39 mriedem but yeah otherwise GET /placement/resource_providers?resources=DISK_GB%3A40%2CMEMORY_MB%3A4096%2CVCPU%3A1
15:09:41 bauzas mriedem: last time I debugged that, I was just looking at the time instead :(
15:10:04 mriedem in the n-cpu logs, i see the 3 nodes reporting capacity just before that
15:10:07 mriedem so those are all good
15:10:34 mriedem at least i n_report_hypervisor_resource_view
15:10:42 cdent bauzas, mriedem: not fully contextualized, but I improved some of the logging, but there was no super handy way to make the request id used on the nova side also because used on the placement id, it would require an additional header, or changing the middleware
15:10:53 mriedem i figured we could pass a header
15:11:05 mriedem and if the header is there, use it, else generate a new request id
15:11:31 cdent yeah, that would require changing the oslo middleware, so I stalled out
15:11:46 dansmith don't we already have a way to do that between services?
15:11:52 dansmith I thought we did between nova and glance, for example
15:12:01 cdent I think I left that as a comment on the bug, but it may have been lost to the sands of time
15:12:14 mriedem i think it's something that keeps coming up with operators
15:12:43 mriedem sdague might know
15:12:46 cdent it does seem like a thing worth doing
15:12:46 bauzas dansmith: mriedem: cdent: yeah I think we're using same req-id between projects, nope ?
15:12:49 EmilienM jroll: next review on puppet-nova and you're core
15:13:16 cdent if there's something we can borrow from nova glance interactions, great
15:13:23 jroll EmilienM: nice, I'll go find some typo fixes
15:13:34 mriedem i don't see anything between nova and glance
15:13:45 sdague mriedem: it does keep coming up, there is no generic solution
15:13:56 dansmith okay I thought that was sorted already
15:14:13 sdague there was going to be some logging of both at some point, but it remains unsolved
15:14:13 cdent is it safe for service to accent an incoming request id?
15:14:20 cdent wow, my typing
15:14:21 cdent sorry
15:14:23 cdent accept!
15:14:24 sdague cdent: that is the sticking point
15:14:31 sdague some people get really concerned about that
15:14:51 sdague it would all be super easy if we could get past that
15:14:55 mriedem what's the concern? the request id is read-only isn't it?
15:15:10 sdague mriedem: the concern is accepting request id from the user
15:15:12 jroll so weird to me, today it's always wrong, if we accept one then worst case it's wrong
15:15:18 dansmith mriedem: attackers confusing the ops' logs I imagine
15:15:37 jroll reject anything that isn't 'req-' + uuid_regex
15:15:44 dansmith accepting it but only using it if config says to seems reasonable
15:15:54 cdent jroll: then send all the same uuid to fudge up the logs
15:15:55 dansmith for ops that prefer correlation with some risk
15:16:06 mriedem can we tell if it's a service user request rather than a user user?
15:16:17 sdague yeh, so honestly the service user / service token bits might make this better
15:16:18 mriedem and if so, use the provided request id
15:16:32 sdague mriedem: we can now sometimes, that's really new though
15:16:32 cdent mriedem: then we get into strange issues with ordering of middleware
15:16:33 mriedem right, we're already doing the re-auth thing
15:16:44 jroll cdent: I guess, if I was an operator I'd just ignore support requests and bugs for that request id :P
15:17:11 sdague mriedem: I think there is probably a path forward that wouldn't freak everyone out based on service token
15:17:13 mriedem anywho, total tangent
15:17:15 sdague yep
15:17:45 mriedem sounds like something for lbragstad and the vm+baremetal+totally+not+exclusive+club+platform team
15:17:56 cdent i've always assumed we want the thing which logs begin and end of request to the first and last pieces of middleware that get touched, thus before any token analysis
15:18:48 mriedem so i'm trying to go back to what could possibly change with https://review.openstack.org/#/c/441544/
15:19:03 mriedem and how self.scheduler_client.update_compute_node(compute_node) works if get_inventory isn't there
15:19:18 dansmith mriedem: yeah, I don't know either, but if we're getting back zero from placement, I'm not sure what else to blame
15:19:20 mriedem if the node is disabled, they both do the same thing, they return {} inventory which tells the report client to delete inventory for that provider
15:19:28 dansmith mriedem: so the log result plus the revert test of that should be interesting
15:22:51 mriedem if inv_data == curr.get('inventories', {}):
15:22:51 mriedem so, does this seem weird to anyone else?
15:23:16 mriedem https://github.com/openstack/nova/blob/master/nova/scheduler/client/report.py#L495
15:23:52 dansmith if the thing we're going to report is equal to what we have, then don't report?
15:23:59 bauzas mriedem: that just means "heh look, we already have that"
15:24:07 dansmith and if what is being reported is nothing and nothing is already stored, then.. don't do anything either
15:24:18 bauzas dansmith: mriedem: FWIW, we already have that in place in the RT
15:24:31 mriedem is the dict compare always good in py27 even with random key order and whatnot?
15:24:33 bauzas old-world-ish of course
15:24:59 dansmith sure
15:25:18 lbragstad mriedem sdague just finished parsing the scroll back - kinda sounds similar to what jamielennox was doing with service users and validating expired tokens
15:25:19 mriedem bauzas: if you're referring to this https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L730 it's a bit different
15:25:22 jroll >>> {'a': 'b', 'c': 'd'} == {'c': 'd', 'a': 'b'}
15:25:24 jroll True
15:25:26 jroll mriedem: ^
15:25:27 bauzas mriedem: https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L730
15:25:33 bauzas yeah
15:26:17 openstackgerrit Sujitha proposed openstack/nova master: Add helper method to add additional data about policy rule. https://review.openstack.org/434842
15:26:22 mriedem ok nvm then, that just surprised me given how many times assertEqual on dicts has failed b/c of random hash seeds
15:27:17 jroll well, assertEqual isn't so smart iirc
15:27:22 siva_krishnan mriedem: can you remove -2 for this patch https://review.openstack.org/#/c/308198/ ?
15:27:57 dansmith I think it's more assertEqual where we've pulled things out of a dict in random order anyway, right?

Earlier   Later