Posted	Nick	Remark
#openstack-nova - 2017-03-20
14:49:32	openstackgerrit	Gábor Antal proposed openstack/nova master: Add json style checking for sample notifications https://review.openstack.org/443677
14:49:35	mriedem	but maybe the spaghetti makes sense someohw
14:49:37	dansmith	mriedem: right and we don't
14:50:02	dansmith	mriedem: https://github.com/openstack/nova/blob/master/nova/scheduler/filter_scheduler.py#L175-L176
14:50:03	openstack	Launchpad bug 1674375 in puppet-nova "compute_manager is deprecated, we should stop using it" [High,Triaged] - Assigned to Emilien Macchi (emilienm)
14:50:03	EmilienM	jroll: /me taking care of this cleanup https://bugs.launchpad.net/puppet-nova/+bug/1674375
14:50:18	mriedem	dansmith: i'm talking about this https://github.com/openstack/nova/blob/master/nova/scheduler/filter_scheduler.py#L108
14:50:21	jroll	EmilienM: nice, thanks! s/deprecated/removed/ btw :)
14:50:23	mriedem	hosts is [] as you just poined out
14:50:25	mriedem	*pointed
14:50:38	mriedem	but we still go through self.host_manager.get_filtered_hosts and all of it's checks on an empty list
14:50:43	mriedem	which seems like a waste of cycles
14:50:53	openstackgerrit	Silvan Kaiser proposed openstack/nova master: Removes getfattr from Quobyte Nova driver https://review.openstack.org/428646
14:50:54	dansmith	mriedem: ah, well, it's empty, but yeah I see what you mean
14:50:56	mriedem	unrelated to this problem
14:50:57	dansmith	not very impactful
14:50:59	dansmith	yeah
14:51:10	mriedem	but i thought we were always paranoid about scheduler perf
14:51:18	mriedem	like, no logging
14:51:40	dansmith	that will fail after the first filter, though,
14:51:43	dansmith	and the first one is retryfilter,
14:51:46	dansmith	which does basically nothing
14:51:50	dansmith	so it's really small I think
14:51:58	dansmith	still wasteful, but.. minor I beleive
14:51:59	mriedem	first default filter
14:52:01	mriedem	sure
14:52:08	dansmith	sure, first default
14:52:21	EmilienM	jroll: done, thx
14:52:37	mriedem	the first filter in my custom deployment actually calls out to my backend inventory datastore and gathers all of the latest goodies, just to filter them on an empty list :)
14:52:49	bauzas	dansmith: EmilienM: just seeing your discussions, any questions I could help you ?
14:53:18	dansmith	bauzas: we don't need any more questions.. we need ANSWERS!
14:53:27	dansmith	mriedem: that's quite unfortunate :)
14:53:40	bauzas	so, I could help for answers :p
14:53:41	dansmith	mriedem: especially for that to be before retryfilter :D
14:53:58	mriedem	it also ssh's into all of my computes and gathers data
14:54:00	mriedem	it's not very smart
14:54:04	mriedem	but it's enterprise ready
14:54:07	dansmith	lol
15:05:09	mriedem	how do we know which flavor was actually used in the server create request?
15:05:22	mriedem	EmilienM: you pointed at http://logs.openstack.org/periodic/periodic-tripleo-ci-centos-7-ovb-nonha/12d034e/logs/undercloud/var/log/nova/nova-api.txt.gz#_2017-03-19_06_38_20_764 but there are more than that flavor
15:05:23	dansmith	mriedem: you can look at the placement request
15:05:29	mriedem	ok
15:05:46	mriedem	we don't correlate the request ids between the nova and placement services yet do we?
15:05:46	dansmith	mriedem: http://logs.openstack.org/periodic/periodic-tripleo-ci-centos-7-ovb-nonha/12d034e/logs/undercloud/var/log/nova/nova-placement-api.txt.gz#_2017-03-19_07_19_34_113
15:05:58	dansmith	mriedem: bauzas was just telling me that we do
15:06:10	mriedem	cdent: you were the last to look at that i thought
15:06:32	dansmith	mriedem: looks like the request I linked matches the 40,4g,1 flavor
15:06:42	EmilienM	jroll: https://review.openstack.org/#/q/topic:bug/1674375
15:06:43	bauzas	dansmith: mriedem: enf of Ocata-3, we were not yet passing req-id
15:07:02	bauzas	dansmith: mriedem: but I thought cdent threw a patch for that
15:07:03	mriedem	i'm looking for req-702e96af-318d-4611-af19-d6a1f3321c6a
15:07:09	mriedem	http://logs.openstack.org/periodic/periodic-tripleo-ci-centos-7-ovb-nonha/12d034e/logs/undercloud/var/log/nova/nova-api.txt.gz#_2017-03-19_07_19_33_201
15:07:13	mriedem	that's the ceph storage instance
15:07:41	mriedem	i know we log the placement request id from the response in the scheduler report client
15:07:54	mriedem	but we don't pass and use a request id from n-sch to placement and log with that req id in the placement logs
15:07:57	mriedem	which is what we need
15:08:16	jroll	we don't do that between n-cpu and ironic either :(
15:08:17	dansmith	mriedem: they all look the same to me
15:08:26	openstackgerrit	Sean Dague proposed openstack/nova master: update i18n guide for nova https://review.openstack.org/447528
15:08:39	mriedem	but yeah otherwise GET /placement/resource_providers?resources=DISK_GB%3A40%2CMEMORY_MB%3A4096%2CVCPU%3A1
15:09:41	bauzas	mriedem: last time I debugged that, I was just looking at the time instead :(
15:10:04	mriedem	in the n-cpu logs, i see the 3 nodes reporting capacity just before that
15:10:07	mriedem	so those are all good
15:10:34	mriedem	at least i n_report_hypervisor_resource_view
15:10:42	cdent	bauzas, mriedem: not fully contextualized, but I improved some of the logging, but there was no super handy way to make the request id used on the nova side also because used on the placement id, it would require an additional header, or changing the middleware
15:10:53	mriedem	i figured we could pass a header
15:11:05	mriedem	and if the header is there, use it, else generate a new request id
15:11:31	cdent	yeah, that would require changing the oslo middleware, so I stalled out
15:11:46	dansmith	don't we already have a way to do that between services?
15:11:52	dansmith	I thought we did between nova and glance, for example
15:12:01	cdent	I think I left that as a comment on the bug, but it may have been lost to the sands of time
15:12:14	mriedem	i think it's something that keeps coming up with operators
15:12:43	mriedem	sdague might know
15:12:46	cdent	it does seem like a thing worth doing
15:12:46	bauzas	dansmith: mriedem: cdent: yeah I think we're using same req-id between projects, nope ?
15:12:49	EmilienM	jroll: next review on puppet-nova and you're core
15:13:16	cdent	if there's something we can borrow from nova glance interactions, great
15:13:23	jroll	EmilienM: nice, I'll go find some typo fixes
15:13:34	mriedem	i don't see anything between nova and glance
15:13:45	sdague	mriedem: it does keep coming up, there is no generic solution
15:13:56	dansmith	okay I thought that was sorted already
15:14:13	sdague	there was going to be some logging of both at some point, but it remains unsolved
15:14:13	cdent	is it safe for service to accent an incoming request id?
15:14:20	cdent	wow, my typing
15:14:21	cdent	sorry
15:14:23	cdent	accept!
15:14:24	sdague	cdent: that is the sticking point
15:14:31	sdague	some people get really concerned about that
15:14:51	sdague	it would all be super easy if we could get past that
15:14:55	mriedem	what's the concern? the request id is read-only isn't it?
15:15:10	sdague	mriedem: the concern is accepting request id from the user
15:15:12	jroll	so weird to me, today it's always wrong, if we accept one then worst case it's wrong
15:15:18	dansmith	mriedem: attackers confusing the ops' logs I imagine
15:15:37	jroll	reject anything that isn't 'req-' + uuid_regex
15:15:44	dansmith	accepting it but only using it if config says to seems reasonable
15:15:54	cdent	jroll: then send all the same uuid to fudge up the logs
15:15:55	dansmith	for ops that prefer correlation with some risk
15:16:06	mriedem	can we tell if it's a service user request rather than a user user?
15:16:17	sdague	yeh, so honestly the service user / service token bits might make this better
15:16:18	mriedem	and if so, use the provided request id

Earlier   Later