Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-11-19
16:07:46 artom AFAICT there's no race or anything, it's just booting the resized pinned instance on the host without cpu_dedicated_set or vcpu_pin_set configured
16:07:54 artom Pinning it to 2 random CPUs
16:07:59 artom Then complaining when it can't delete it
16:08:11 artom Which... does kinda smell like a minor bug...
16:08:11 sean-k-mooney well we need delete teh vms before we reconfigure
16:08:18 artom We *don't* reconfigure
16:08:31 artom This is all without any config changes or service restarts
16:08:46 sean-k-mooney no i mean im not sure we are waiting for all vms from the previous test to be deleted
16:09:08 artom AFAICT, yes
16:09:54 sean-k-mooney artom: we should never see nova.exception.CPUUnpinningInvalid: CPU set to unpin [2, 6] must be a subset of pinned CPU set []
16:10:12 sean-k-mooney if we do it means the config have chagned
16:10:31 sean-k-mooney or we are hitting the race which stephen fixed
16:10:40 artom But... what happens in the case of having booted an instance with pinned CPUs on a host without cpu_dedicated_set configured?
16:10:46 artom And then deleting that
16:10:52 artom Wouldn't we expect that error?
16:10:57 sean-k-mooney if its still not set its fine
16:11:07 artom Or does it use the database and not the config?
16:11:17 sean-k-mooney its not vallide to add cpu_dedicate_set to a host with vms
16:11:22 artom I know
16:11:26 artom That's not what we're doing here
16:11:38 sean-k-mooney can you point me to the test
16:11:43 artom sean-k-mooney, gmeet? :)
16:11:49 sean-k-mooney sure
16:13:03 artom sean-k-mooney, https://meet.google.com/eqm-jjcx-fjw
16:13:04 sean-k-mooney im going to make a cup of coffee but say 15 past
16:13:15 sean-k-mooney ill join when i get back
16:13:20 artom Take your time :)
16:16:24 sean-k-mooney just back :)
16:24:21 dansmith gibi: the distro-ness also affects our ability to describe what should go in each config file,
16:24:42 gibi dansmith: do you have an example?
16:25:04 dansmith gibi: because the problem on the ML is making all the distro packages work both across multiple nodes, as well as AIO, and currently there's no nova-conductor-cellN vs nova-superconductor service(s)
16:25:23 dansmith gibi: so I'm not quite sure how to write this to make sense for both
16:26:04 dansmith gibi: I'm going to try, but we'll probably need some input from people like owalsh .. maybe that means we need to use something other than the install doc to describe things, I dunno
16:26:14 dansmith writing docs is hard :(
16:27:13 gibi so the problem is that distros map packages to services but the nova-conductor package needs to be mapped to two type of services with different config?
16:27:52 dansmith yeah
16:27:58 dansmith and also,
16:28:02 owalsh hey
16:28:05 dansmith and this is what he was complaining about,
16:28:15 dansmith normally they'd all point to nova.conf, assuming they'd be on different hosts,
16:28:39 dansmith but if they're all on the same host, I'd have to describe changing/overriding unit files to get the right config to conductor vs. api, etc
16:29:22 owalsh tkajinam made an important point on the ML. It's not just AIO, it's the typical production deployment of ironic nova-compute
16:29:32 owalsh on the same host as nova-api etc...
16:30:02 dansmith owalsh: ack, yeah, that's the same problem, just another reason you might hit it
16:30:30 dansmith owalsh: anyway, what I'm describing above is just that I was going to start working on the config samples in this doc: https://docs.openstack.org/nova/latest/install/controller-install-ubuntu.html
16:30:43 dansmith but realized that I'll not be able to describe it fully without talking about changing the distro unit files, etc
16:31:07 gibi dansmith: so either we start describing deployment scenarios (which feels bad) or start using separate config file name for each service
16:31:10 owalsh dansmith: ack, and that's something puppet-nova could do
16:32:10 dansmith gibi: well, I really think we should avoid getting into prescribing config file names for things, because again, that's very very distro and deployment specific,
16:32:22 dansmith gibi: and I think that the suggestion on the ML to standardize those layers a bit actually got us into a more sticky situation
16:34:38 gibi but then we need to iterate on at least AIO deployment scenario, Ironic, and a real cellv2 scenario
16:35:53 gibi anyhow I agree that docing this properly is hard
16:36:24 owalsh implementing it in code is no picnic either :-)
16:36:33 gibi owalsh: good point :)
16:36:36 dansmith at the very least, I think we should get rid of the 3x distro-based flavored docs for each
16:36:45 gibi +1 ^^
16:37:28 dansmith maybe I can just sprinkle some notes in around the configuration bits about service differences, and see if that helps clarity at all
16:37:46 gibi yeah, that could be a good starting point
16:41:07 openstack Launchpad bug 1903908 in puppet-nova "nova conf [api]/dhcp_domain is required on nova-compute" [Undecided,New] - Assigned to Oliver Walsh (owalsh)
16:41:07 gibi dansmith: I don't know if you see this but there are also ~bugs uncovered by these config discussion https://bugs.launchpad.net/nova/+bug/1903908 (thank to owalsh for the report)
16:42:03 dansmith I saw that, I'm not sure how I feel about it
16:42:15 dansmith anything that the configdrive code uses will be shared with the metadata api
16:43:28 gibi yeah, I don't like that metadata calls deep into the metadata code
16:44:11 dansmith configdrive?
16:44:18 dansmith well, that's pretty much the whole point of it :)
16:44:23 gibi the gneeration of the config drive
16:44:45 gibi but this result in the code use by two different services
16:45:56 gibi I guess it is just the fact that I associate the the metadata code only to the metadata service and always forget that ther is a dependency to it from the nova-compute
16:46:22 gibi I guess that lead to the fact that we moved the domain config to the [api] section
16:46:47 gibi anyhow this is for another day as I have to go offline
16:46:53 gibi o/
16:49:27 dansmith ack, I'll put this up for discussion in a bit
16:50:56 owalsh dansmith: I was going to respond to tkajinam on the ML but I don't really have a response, just a +1 really
16:52:12 dansmith owalsh: okay I guess it's really the same thing anyway
16:52:49 owalsh yes, the first point IIUC is if we have nova.conf and nova-cpu.conf, both containing the generated sample config, how do I know where to set option foo?
16:52:50 dansmith if the packages want to use nova-compute.conf for the compute package, that'd solve it, AFAIK
16:55:01 owalsh ack, only realistic solution I can see right now, but it's also a bit unpleasant
16:56:22 dansmith well, nova/ironic has pretty much never fit into nova properly so .. unsurprising that we have this problem
16:56:35 dansmith we could also just special case the check and not complain if we're using ironic,
16:56:48 dansmith since we're not on a compromise-able situation
16:57:27 owalsh I don't think so, the deb/rpm doesn't know what nova-compute will be used for
16:57:54 owalsh ... when it creates the service
16:59:11 dansmith what I meant was, the compute package could keep using nova.conf, and if your tool then configures nova.conf for ironic-and-db-creds-because-single-node, nova-compute wouldn't explode when it starts
16:59:31 dansmith it wouldn't solve the AIO case, but it would solve the ironic one
17:00:51 owalsh rpm/deb would need a new service unit file e.g "nova-ironic" that uses nova.conf vs nova-compute.conf
17:01:04 dansmith going back to the nova-db.conf suggestion.. if we allow the wsgi app to read that, then you can again just configure all the services to read from that, except for nova-compute, and then the only thing that doesn't work is AIO-multicell right/
17:02:20 owalsh did seem like the most elegant solution, but also not had a lot of time to think of any gotchas
17:02:52 owalsh and I think AIO-multicell could be considered not supported outside of devstack
17:03:19 owalsh or if you really really want to do this for some of CI job then use containers
17:03:23 dansmith well, I think we should fix wsgi to let you specify config files like everything else, and then let you guys decide how you want to template and split the config files to make the packages work (or not)
17:06:42 owalsh ack, might also be worth looking at how we could parameterize the conf file path in the dev/rpm services
17:08:33 openstackgerrit Artom Lifshitz proposed openstack/nova master: DNM: extra logging for unpin_cpus https://review.opendev.org/763409
17:08:59 dansmith even just having /etc/nova.conf that everything reads, and then each service configured to load /etc/nova/nova-conductor.conf, nova-scheduler.conf, etc would maybe be useful... common config and per-service config.. the latter could be empty in a lot of cases
17:13:39 owalsh ack, I think that was the direction this was going but the wsgi issue was a blocker
17:14:28 dansmith yeah
17:14:31 openstackgerrit Dan Smith proposed openstack/nova master: RFC: Update the install documents for cells and less distro specific https://review.opendev.org/763412
17:33:31 openstackgerrit Lee Yarwood proposed openstack/nova-specs master: Image and flavor defined ephemeral storage encryption https://review.opendev.org/752284
18:51:25 sean-k-mooney stephenfin: i dont know if you saw the discussion about the metadata serivce but we have some undocumented behavior realated to an old nova v2 api extention that got merged into the service when we remove the nova api v3 code and support for cell in newton
18:51:59 sean-k-mooney stephenfin: specifically form within the guest without creds you can do a post to the password filed and update that filed
18:53:13 sean-k-mooney stephenfin: so it looks like we need to update the documatnion for that and consider if we actully want to support that longterem i think we need to at least for now since cloudbase use it

Earlier   Later