Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-11-19
16:11:38 sean-k-mooney can you point me to the test
16:11:43 artom sean-k-mooney, gmeet? :)
16:11:49 sean-k-mooney sure
16:13:03 artom sean-k-mooney, https://meet.google.com/eqm-jjcx-fjw
16:13:04 sean-k-mooney im going to make a cup of coffee but say 15 past
16:13:15 sean-k-mooney ill join when i get back
16:13:20 artom Take your time :)
16:16:24 sean-k-mooney just back :)
16:24:21 dansmith gibi: the distro-ness also affects our ability to describe what should go in each config file,
16:24:42 gibi dansmith: do you have an example?
16:25:04 dansmith gibi: because the problem on the ML is making all the distro packages work both across multiple nodes, as well as AIO, and currently there's no nova-conductor-cellN vs nova-superconductor service(s)
16:25:23 dansmith gibi: so I'm not quite sure how to write this to make sense for both
16:26:04 dansmith gibi: I'm going to try, but we'll probably need some input from people like owalsh .. maybe that means we need to use something other than the install doc to describe things, I dunno
16:26:14 dansmith writing docs is hard :(
16:27:13 gibi so the problem is that distros map packages to services but the nova-conductor package needs to be mapped to two type of services with different config?
16:27:52 dansmith yeah
16:27:58 dansmith and also,
16:28:02 owalsh hey
16:28:05 dansmith and this is what he was complaining about,
16:28:15 dansmith normally they'd all point to nova.conf, assuming they'd be on different hosts,
16:28:39 dansmith but if they're all on the same host, I'd have to describe changing/overriding unit files to get the right config to conductor vs. api, etc
16:29:22 owalsh tkajinam made an important point on the ML. It's not just AIO, it's the typical production deployment of ironic nova-compute
16:29:32 owalsh on the same host as nova-api etc...
16:30:02 dansmith owalsh: ack, yeah, that's the same problem, just another reason you might hit it
16:30:30 dansmith owalsh: anyway, what I'm describing above is just that I was going to start working on the config samples in this doc: https://docs.openstack.org/nova/latest/install/controller-install-ubuntu.html
16:30:43 dansmith but realized that I'll not be able to describe it fully without talking about changing the distro unit files, etc
16:31:07 gibi dansmith: so either we start describing deployment scenarios (which feels bad) or start using separate config file name for each service
16:31:10 owalsh dansmith: ack, and that's something puppet-nova could do
16:32:10 dansmith gibi: well, I really think we should avoid getting into prescribing config file names for things, because again, that's very very distro and deployment specific,
16:32:22 dansmith gibi: and I think that the suggestion on the ML to standardize those layers a bit actually got us into a more sticky situation
16:34:38 gibi but then we need to iterate on at least AIO deployment scenario, Ironic, and a real cellv2 scenario
16:35:53 gibi anyhow I agree that docing this properly is hard
16:36:24 owalsh implementing it in code is no picnic either :-)
16:36:33 gibi owalsh: good point :)
16:36:36 dansmith at the very least, I think we should get rid of the 3x distro-based flavored docs for each
16:36:45 gibi +1 ^^
16:37:28 dansmith maybe I can just sprinkle some notes in around the configuration bits about service differences, and see if that helps clarity at all
16:37:46 gibi yeah, that could be a good starting point
16:41:07 gibi dansmith: I don't know if you see this but there are also ~bugs uncovered by these config discussion https://bugs.launchpad.net/nova/+bug/1903908 (thank to owalsh for the report)
16:41:07 openstack Launchpad bug 1903908 in puppet-nova "nova conf [api]/dhcp_domain is required on nova-compute" [Undecided,New] - Assigned to Oliver Walsh (owalsh)
16:42:03 dansmith I saw that, I'm not sure how I feel about it
16:42:15 dansmith anything that the configdrive code uses will be shared with the metadata api
16:43:28 gibi yeah, I don't like that metadata calls deep into the metadata code
16:44:11 dansmith configdrive?
16:44:18 dansmith well, that's pretty much the whole point of it :)
16:44:23 gibi the gneeration of the config drive
16:44:45 gibi but this result in the code use by two different services
16:45:56 gibi I guess it is just the fact that I associate the the metadata code only to the metadata service and always forget that ther is a dependency to it from the nova-compute
16:46:22 gibi I guess that lead to the fact that we moved the domain config to the [api] section
16:46:47 gibi anyhow this is for another day as I have to go offline
16:46:53 gibi o/
16:49:27 dansmith ack, I'll put this up for discussion in a bit
16:50:56 owalsh dansmith: I was going to respond to tkajinam on the ML but I don't really have a response, just a +1 really
16:52:12 dansmith owalsh: okay I guess it's really the same thing anyway
16:52:49 owalsh yes, the first point IIUC is if we have nova.conf and nova-cpu.conf, both containing the generated sample config, how do I know where to set option foo?
16:52:50 dansmith if the packages want to use nova-compute.conf for the compute package, that'd solve it, AFAIK
16:55:01 owalsh ack, only realistic solution I can see right now, but it's also a bit unpleasant
16:56:22 dansmith well, nova/ironic has pretty much never fit into nova properly so .. unsurprising that we have this problem
16:56:35 dansmith we could also just special case the check and not complain if we're using ironic,
16:56:48 dansmith since we're not on a compromise-able situation
16:57:27 owalsh I don't think so, the deb/rpm doesn't know what nova-compute will be used for
16:57:54 owalsh ... when it creates the service
16:59:11 dansmith what I meant was, the compute package could keep using nova.conf, and if your tool then configures nova.conf for ironic-and-db-creds-because-single-node, nova-compute wouldn't explode when it starts
16:59:31 dansmith it wouldn't solve the AIO case, but it would solve the ironic one
17:00:51 owalsh rpm/deb would need a new service unit file e.g "nova-ironic" that uses nova.conf vs nova-compute.conf
17:01:04 dansmith going back to the nova-db.conf suggestion.. if we allow the wsgi app to read that, then you can again just configure all the services to read from that, except for nova-compute, and then the only thing that doesn't work is AIO-multicell right/
17:02:20 owalsh did seem like the most elegant solution, but also not had a lot of time to think of any gotchas
17:02:52 owalsh and I think AIO-multicell could be considered not supported outside of devstack
17:03:19 owalsh or if you really really want to do this for some of CI job then use containers
17:03:23 dansmith well, I think we should fix wsgi to let you specify config files like everything else, and then let you guys decide how you want to template and split the config files to make the packages work (or not)
17:06:42 owalsh ack, might also be worth looking at how we could parameterize the conf file path in the dev/rpm services
17:08:33 openstackgerrit Artom Lifshitz proposed openstack/nova master: DNM: extra logging for unpin_cpus https://review.opendev.org/763409
17:08:59 dansmith even just having /etc/nova.conf that everything reads, and then each service configured to load /etc/nova/nova-conductor.conf, nova-scheduler.conf, etc would maybe be useful... common config and per-service config.. the latter could be empty in a lot of cases
17:13:39 owalsh ack, I think that was the direction this was going but the wsgi issue was a blocker
17:14:28 dansmith yeah
17:14:31 openstackgerrit Dan Smith proposed openstack/nova master: RFC: Update the install documents for cells and less distro specific https://review.opendev.org/763412
17:33:31 openstackgerrit Lee Yarwood proposed openstack/nova-specs master: Image and flavor defined ephemeral storage encryption https://review.opendev.org/752284
18:51:25 sean-k-mooney stephenfin: i dont know if you saw the discussion about the metadata serivce but we have some undocumented behavior realated to an old nova v2 api extention that got merged into the service when we remove the nova api v3 code and support for cell in newton
18:51:59 sean-k-mooney stephenfin: specifically form within the guest without creds you can do a post to the password filed and update that filed
18:53:13 sean-k-mooney stephenfin: so it looks like we need to update the documatnion for that and consider if we actully want to support that longterem i think we need to at least for now since cloudbase use it
18:54:27 sean-k-mooney that said it was never actully accpeted into the v2.x api it was incorrectly included when we killed extention and drop the nova v3 api
18:59:07 sean-k-mooney stephenfin: https://github.com/openstack/nova/commit/a2101c4e7017715af0a29675b89e14ee2884bd89 is the change that allowed the password to be updated once via the 169 adress
19:00:10 sean-k-mooney which built on the server_password api extntion
19:00:45 sean-k-mooney anyway none of our docs say ^ is a thing you can do.
23:10:26 NobodyCam Good Afternoon Nova folks, I've started seeing hypervisor state flapping between up and down, The hypervisors are up, I've read that adjusting server_down_time and report_interval can help with this, are these the correct setting to adjust and is there a way to gauge what these values should be?
23:31:34 melwitt NobodyCam: are you experiencing a problem with getting occasional NoValidHost because of this or seeing the hypervisor state going up and down or both?
23:31:52 NobodyCam really both
23:33:34 melwitt ok, for the scheduling thing, consider moving the ComputeFilter earlier in your list of configured filters if you have a large number of compute nodes. I have seen things where if there are a lot of computes (like 1000) the scheduling process is so slow that some nodes are considered "down" by the ComputeFilter because by the time that filter is reached, 60s have elapsed
23:34:10 melwitt for the other settings, "service_down_time report_interval should be less than service_down_time. If service_down_time is less than report_interval, services will routinely be considered down, because they report in too rarely" from https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.report_interval is the main thing you need to make sure you have right
23:36:01 melwitt here's another setting to make sure is set harmoniously with service_down_time https://docs.openstack.org/nova/latest/configuration/config.html#scheduler.periodic_task_interval
23:37:44 melwitt I think those 3 settings are the only ones you need to adjust and they need to be set as recommended in that doc, that report_interval needs to be less than service_down_time and periodic_task_interval needs to also be less than service_down_time
23:44:26 melwitt AFAIK the driving factor for choosing service_down_time will be how long scheduling is taking for you. if you move ComputeFilter first in the list, for example, and still get sporadic NoValidHost bc of compute node "down" when it's not really down, then you will need to increase service_down_time to accommodate the scheduling time. just make sure you don't set report_interval or periodic_task_interval to longer than service_down_time
23:54:17 NobodyCam Thank you very much melwitt I will review our current configuration an let you know if it improves after any changes
23:54:55 melwitt NobodyCam: np, good luck, will be interested to hear how it goes
#openstack-nova - 2020-11-20
00:09:15 sean-k-mooney NobodyCam: you might also want to make sure that you have ntpd/chronny running and the clocks are in sync
00:09:49 sean-k-mooney NobodyCam:if the clocks drift it can cause the servics to flap like that
00:15:23 NobodyCam Oh good point, we should nap going, but I have not checked it
00:15:35 NobodyCam s/nap/ntp
00:24:36 openstackgerrit Brin Zhang proposed openstack/nova-specs master: Remove tenant_id https://review.opendev.org/737241
00:25:51 brinzhang_ gmann: addressed your comment in remove tenant_id specs, thanks

Earlier   Later