Posted	Nick	Remark
#openstack-nova - 2020-11-19
16:30:43	dansmith	but realized that I'll not be able to describe it fully without talking about changing the distro unit files, etc
16:31:07	gibi	dansmith: so either we start describing deployment scenarios (which feels bad) or start using separate config file name for each service
16:31:10	owalsh	dansmith: ack, and that's something puppet-nova could do
16:32:10	dansmith	gibi: well, I really think we should avoid getting into prescribing config file names for things, because again, that's very very distro and deployment specific,
16:32:22	dansmith	gibi: and I think that the suggestion on the ML to standardize those layers a bit actually got us into a more sticky situation
16:34:38	gibi	but then we need to iterate on at least AIO deployment scenario, Ironic, and a real cellv2 scenario
16:35:53	gibi	anyhow I agree that docing this properly is hard
16:36:24	owalsh	implementing it in code is no picnic either :-)
16:36:33	gibi	owalsh: good point :)
16:36:36	dansmith	at the very least, I think we should get rid of the 3x distro-based flavored docs for each
16:36:45	gibi	+1 ^^
16:37:28	dansmith	maybe I can just sprinkle some notes in around the configuration bits about service differences, and see if that helps clarity at all
16:37:46	gibi	yeah, that could be a good starting point
16:41:07	openstack	Launchpad bug 1903908 in puppet-nova "nova conf [api]/dhcp_domain is required on nova-compute" [Undecided,New] - Assigned to Oliver Walsh (owalsh)
16:41:07	gibi	dansmith: I don't know if you see this but there are also ~bugs uncovered by these config discussion https://bugs.launchpad.net/nova/+bug/1903908 (thank to owalsh for the report)
16:42:03	dansmith	I saw that, I'm not sure how I feel about it
16:42:15	dansmith	anything that the configdrive code uses will be shared with the metadata api
16:43:28	gibi	yeah, I don't like that metadata calls deep into the metadata code
16:44:11	dansmith	configdrive?
16:44:18	dansmith	well, that's pretty much the whole point of it :)
16:44:23	gibi	the gneeration of the config drive
16:44:45	gibi	but this result in the code use by two different services
16:45:56	gibi	I guess it is just the fact that I associate the the metadata code only to the metadata service and always forget that ther is a dependency to it from the nova-compute
16:46:22	gibi	I guess that lead to the fact that we moved the domain config to the [api] section
16:46:47	gibi	anyhow this is for another day as I have to go offline
16:46:53	gibi	o/
16:49:27	dansmith	ack, I'll put this up for discussion in a bit
16:50:56	owalsh	dansmith: I was going to respond to tkajinam on the ML but I don't really have a response, just a +1 really
16:52:12	dansmith	owalsh: okay I guess it's really the same thing anyway
16:52:49	owalsh	yes, the first point IIUC is if we have nova.conf and nova-cpu.conf, both containing the generated sample config, how do I know where to set option foo?
16:52:50	dansmith	if the packages want to use nova-compute.conf for the compute package, that'd solve it, AFAIK
16:55:01	owalsh	ack, only realistic solution I can see right now, but it's also a bit unpleasant
16:56:22	dansmith	well, nova/ironic has pretty much never fit into nova properly so .. unsurprising that we have this problem
16:56:35	dansmith	we could also just special case the check and not complain if we're using ironic,
16:56:48	dansmith	since we're not on a compromise-able situation
16:57:27	owalsh	I don't think so, the deb/rpm doesn't know what nova-compute will be used for
16:57:54	owalsh	... when it creates the service
16:59:11	dansmith	what I meant was, the compute package could keep using nova.conf, and if your tool then configures nova.conf for ironic-and-db-creds-because-single-node, nova-compute wouldn't explode when it starts
16:59:31	dansmith	it wouldn't solve the AIO case, but it would solve the ironic one
17:00:51	owalsh	rpm/deb would need a new service unit file e.g "nova-ironic" that uses nova.conf vs nova-compute.conf
17:01:04	dansmith	going back to the nova-db.conf suggestion.. if we allow the wsgi app to read that, then you can again just configure all the services to read from that, except for nova-compute, and then the only thing that doesn't work is AIO-multicell right/
17:02:20	owalsh	did seem like the most elegant solution, but also not had a lot of time to think of any gotchas
17:02:52	owalsh	and I think AIO-multicell could be considered not supported outside of devstack
17:03:19	owalsh	or if you really really want to do this for some of CI job then use containers
17:03:23	dansmith	well, I think we should fix wsgi to let you specify config files like everything else, and then let you guys decide how you want to template and split the config files to make the packages work (or not)
17:06:42	owalsh	ack, might also be worth looking at how we could parameterize the conf file path in the dev/rpm services
17:08:33	openstackgerrit	Artom Lifshitz proposed openstack/nova master: DNM: extra logging for unpin_cpus https://review.opendev.org/763409
17:08:59	dansmith	even just having /etc/nova.conf that everything reads, and then each service configured to load /etc/nova/nova-conductor.conf, nova-scheduler.conf, etc would maybe be useful... common config and per-service config.. the latter could be empty in a lot of cases
17:13:39	owalsh	ack, I think that was the direction this was going but the wsgi issue was a blocker
17:14:28	dansmith	yeah
17:14:31	openstackgerrit	Dan Smith proposed openstack/nova master: RFC: Update the install documents for cells and less distro specific https://review.opendev.org/763412
17:33:31	openstackgerrit	Lee Yarwood proposed openstack/nova-specs master: Image and flavor defined ephemeral storage encryption https://review.opendev.org/752284
18:51:25	sean-k-mooney	stephenfin: i dont know if you saw the discussion about the metadata serivce but we have some undocumented behavior realated to an old nova v2 api extention that got merged into the service when we remove the nova api v3 code and support for cell in newton
18:51:59	sean-k-mooney	stephenfin: specifically form within the guest without creds you can do a post to the password filed and update that filed
18:53:13	sean-k-mooney	stephenfin: so it looks like we need to update the documatnion for that and consider if we actully want to support that longterem i think we need to at least for now since cloudbase use it
18:54:27	sean-k-mooney	that said it was never actully accpeted into the v2.x api it was incorrectly included when we killed extention and drop the nova v3 api
18:59:07	sean-k-mooney	stephenfin: https://github.com/openstack/nova/commit/a2101c4e7017715af0a29675b89e14ee2884bd89 is the change that allowed the password to be updated once via the 169 adress
19:00:10	sean-k-mooney	which built on the server_password api extntion
19:00:45	sean-k-mooney	anyway none of our docs say ^ is a thing you can do.
23:10:26	NobodyCam	Good Afternoon Nova folks, I've started seeing hypervisor state flapping between up and down, The hypervisors are up, I've read that adjusting server_down_time and report_interval can help with this, are these the correct setting to adjust and is there a way to gauge what these values should be?
23:31:34	melwitt	NobodyCam: are you experiencing a problem with getting occasional NoValidHost because of this or seeing the hypervisor state going up and down or both?
23:31:52	NobodyCam	really both
23:33:34	melwitt	ok, for the scheduling thing, consider moving the ComputeFilter earlier in your list of configured filters if you have a large number of compute nodes. I have seen things where if there are a lot of computes (like 1000) the scheduling process is so slow that some nodes are considered "down" by the ComputeFilter because by the time that filter is reached, 60s have elapsed
23:34:10	melwitt	for the other settings, "service_down_time report_interval should be less than service_down_time. If service_down_time is less than report_interval, services will routinely be considered down, because they report in too rarely" from https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.report_interval is the main thing you need to make sure you have right
23:36:01	melwitt	here's another setting to make sure is set harmoniously with service_down_time https://docs.openstack.org/nova/latest/configuration/config.html#scheduler.periodic_task_interval
23:37:44	melwitt	I think those 3 settings are the only ones you need to adjust and they need to be set as recommended in that doc, that report_interval needs to be less than service_down_time and periodic_task_interval needs to also be less than service_down_time
23:44:26	melwitt	AFAIK the driving factor for choosing service_down_time will be how long scheduling is taking for you. if you move ComputeFilter first in the list, for example, and still get sporadic NoValidHost bc of compute node "down" when it's not really down, then you will need to increase service_down_time to accommodate the scheduling time. just make sure you don't set report_interval or periodic_task_interval to longer than service_down_time
23:54:17	NobodyCam	Thank you very much melwitt I will review our current configuration an let you know if it improves after any changes
23:54:55	melwitt	NobodyCam: np, good luck, will be interested to hear how it goes
#openstack-nova - 2020-11-20
00:09:15	sean-k-mooney	NobodyCam: you might also want to make sure that you have ntpd/chronny running and the clocks are in sync
00:09:49	sean-k-mooney	NobodyCam:if the clocks drift it can cause the servics to flap like that
00:15:23	NobodyCam	Oh good point, we should nap going, but I have not checked it
00:15:35	NobodyCam	s/nap/ntp
00:24:36	openstackgerrit	Brin Zhang proposed openstack/nova-specs master: Remove tenant_id https://review.opendev.org/737241
00:25:51	brinzhang_	gmann: addressed your comment in remove tenant_id specs, thanks
00:28:15	openstackgerrit	Brin Zhang proposed openstack/nova master: [Trivial] Rename host/node to hostname/nodename in conductor manager https://review.opendev.org/762499
00:51:24	openstackgerrit	Ghanshyam Mann proposed openstack/nova master: Reuse code from oslo lib for JSON policy migration https://review.opendev.org/763487
01:39:00	openstackgerrit	wu.shiming proposed openstack/os-vif master: Fix hacking min version to 3.0.1 https://review.opendev.org/755168
03:17:08	brinzhang	sean-k-mooney: I have a question in https://review.opendev.org/#/c/737241/5/specs/wallaby/approved/remove-tenant-id.rst@62, hope you can reply, thanks
07:16:47	openstackgerrit	zhufl proposed openstack/nova master: Fix invalid argument formatting in exception messages https://review.opendev.org/763511
08:12:43	bauzas	good morning Nova
08:31:35	gibi	good morning
08:35:22	bauzas	elod: gibi: I guess we can move to Fix Released for the Victoria series on https://bugs.launchpad.net/nova/+bug/1902925, right?
08:35:23	openstack	Launchpad bug 1902925 in OpenStack Compute (nova) victoria "Upgrades to compute RPC API 5.12 are broken" [Critical,Fix committed] - Assigned to Sylvain Bauza (sylvain-bauza)
08:35:30	bauzas	as we provided a new stable release
08:36:24	gibi	bauzas: yes it is released. Should this bug state transition automated somehow?
08:42:40	elod	yes, it's definitely released :) and usually it's automatically changed to released at launchpad :( but only "usually" :(
08:43:20	elod	I don't know where is the automation behind, though
08:44:27	elod	(now I've changed it manually)
08:57:48	bauzas	gibi: I don't think so for LP bugs that are targeted for a stable series
08:58:05	bauzas	it's done automatically for master changes of course
08:59:12	gibi	bauzas: so the at each stable release somebody has to go through an flip the released bugs manually
08:59:30	gibi	elod, lyarwood: did you know that and do that?
09:06:44	elod	as far as I knew it was automatically for stable branches, too. But maybe I was wrong. :) and I only moved the state in rare cases in the past.
09:07:01	elod	:/
09:09:07	lyarwood	yeah I also thought this was automatic
09:31:56	bauzas	we can ask reldel
09:40:40	QianbiaoNG	hello anyone knows what IPA means in this project https://opendev.org/x/novajoin
09:41:25	QianbiaoNG	And is there an official project for implement a nova vendor metadata service
09:47:21	swp20	bauzas, sean-k-mooney:

Earlier   Later