| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-11-19 | |||
| 16:05:34 | sean-k-mooney | it might be a missconfiguration or a bug in the white box tests | |
| 16:05:39 | artom | But it's happening because it's deleting the pinned instance when cpu_dedicated_set is None | |
| 16:05:45 | artom | So it complains about unpinning CPUs | |
| 16:05:48 | artom | Well... | |
| 16:05:56 | sean-k-mooney | anytime we define cpu_dedicated_set you should also disable the fallback | |
| 16:06:06 | sean-k-mooney | we likely are not doing the latter | |
| 16:06:09 | artom | We don't define it | |
| 16:06:18 | sean-k-mooney | its enabled by default | |
| 16:06:32 | artom | I mean cpu_dedicated_set | |
| 16:06:42 | artom | What I'm seeing is the test_resize_unpinned_server_to_pinned tearDown is failing | |
| 16:06:51 | artom | Because it can't delete the pinned instance | |
| 16:07:08 | artom | Because nova.exception.CPUUnpinningInvalid: CPU set to unpin [2, 6] must be a subset of pinned CPU set [] | |
| 16:07:46 | artom | AFAICT there's no race or anything, it's just booting the resized pinned instance on the host without cpu_dedicated_set or vcpu_pin_set configured | |
| 16:07:54 | artom | Pinning it to 2 random CPUs | |
| 16:07:59 | artom | Then complaining when it can't delete it | |
| 16:08:11 | sean-k-mooney | well we need delete teh vms before we reconfigure | |
| 16:08:11 | artom | Which... does kinda smell like a minor bug... | |
| 16:08:18 | artom | We *don't* reconfigure | |
| 16:08:31 | artom | This is all without any config changes or service restarts | |
| 16:08:46 | sean-k-mooney | no i mean im not sure we are waiting for all vms from the previous test to be deleted | |
| 16:09:08 | artom | AFAICT, yes | |
| 16:09:54 | sean-k-mooney | artom: we should never see nova.exception.CPUUnpinningInvalid: CPU set to unpin [2, 6] must be a subset of pinned CPU set [] | |
| 16:10:12 | sean-k-mooney | if we do it means the config have chagned | |
| 16:10:31 | sean-k-mooney | or we are hitting the race which stephen fixed | |
| 16:10:40 | artom | But... what happens in the case of having booted an instance with pinned CPUs on a host without cpu_dedicated_set configured? | |
| 16:10:46 | artom | And then deleting that | |
| 16:10:52 | artom | Wouldn't we expect that error? | |
| 16:10:57 | sean-k-mooney | if its still not set its fine | |
| 16:11:07 | artom | Or does it use the database and not the config? | |
| 16:11:17 | sean-k-mooney | its not vallide to add cpu_dedicate_set to a host with vms | |
| 16:11:22 | artom | I know | |
| 16:11:26 | artom | That's not what we're doing here | |
| 16:11:38 | sean-k-mooney | can you point me to the test | |
| 16:11:43 | artom | sean-k-mooney, gmeet? :) | |
| 16:11:49 | sean-k-mooney | sure | |
| 16:13:03 | artom | sean-k-mooney, https://meet.google.com/eqm-jjcx-fjw | |
| 16:13:04 | sean-k-mooney | im going to make a cup of coffee but say 15 past | |
| 16:13:15 | sean-k-mooney | ill join when i get back | |
| 16:13:20 | artom | Take your time :) | |
| 16:16:24 | sean-k-mooney | just back :) | |
| 16:24:21 | dansmith | gibi: the distro-ness also affects our ability to describe what should go in each config file, | |
| 16:24:42 | gibi | dansmith: do you have an example? | |
| 16:25:04 | dansmith | gibi: because the problem on the ML is making all the distro packages work both across multiple nodes, as well as AIO, and currently there's no nova-conductor-cellN vs nova-superconductor service(s) | |
| 16:25:23 | dansmith | gibi: so I'm not quite sure how to write this to make sense for both | |
| 16:26:04 | dansmith | gibi: I'm going to try, but we'll probably need some input from people like owalsh .. maybe that means we need to use something other than the install doc to describe things, I dunno | |
| 16:26:14 | dansmith | writing docs is hard :( | |
| 16:27:13 | gibi | so the problem is that distros map packages to services but the nova-conductor package needs to be mapped to two type of services with different config? | |
| 16:27:52 | dansmith | yeah | |
| 16:27:58 | dansmith | and also, | |
| 16:28:02 | owalsh | hey | |
| 16:28:05 | dansmith | and this is what he was complaining about, | |
| 16:28:15 | dansmith | normally they'd all point to nova.conf, assuming they'd be on different hosts, | |
| 16:28:39 | dansmith | but if they're all on the same host, I'd have to describe changing/overriding unit files to get the right config to conductor vs. api, etc | |
| 16:29:22 | owalsh | tkajinam made an important point on the ML. It's not just AIO, it's the typical production deployment of ironic nova-compute | |
| 16:29:32 | owalsh | on the same host as nova-api etc... | |
| 16:30:02 | dansmith | owalsh: ack, yeah, that's the same problem, just another reason you might hit it | |
| 16:30:30 | dansmith | owalsh: anyway, what I'm describing above is just that I was going to start working on the config samples in this doc: https://docs.openstack.org/nova/latest/install/controller-install-ubuntu.html | |
| 16:30:43 | dansmith | but realized that I'll not be able to describe it fully without talking about changing the distro unit files, etc | |
| 16:31:07 | gibi | dansmith: so either we start describing deployment scenarios (which feels bad) or start using separate config file name for each service | |
| 16:31:10 | owalsh | dansmith: ack, and that's something puppet-nova could do | |
| 16:32:10 | dansmith | gibi: well, I really think we should avoid getting into prescribing config file names for things, because again, that's very very distro and deployment specific, | |
| 16:32:22 | dansmith | gibi: and I think that the suggestion on the ML to standardize those layers a bit actually got us into a more sticky situation | |
| 16:34:38 | gibi | but then we need to iterate on at least AIO deployment scenario, Ironic, and a real cellv2 scenario | |
| 16:35:53 | gibi | anyhow I agree that docing this properly is hard | |
| 16:36:24 | owalsh | implementing it in code is no picnic either :-) | |
| 16:36:33 | gibi | owalsh: good point :) | |
| 16:36:36 | dansmith | at the very least, I think we should get rid of the 3x distro-based flavored docs for each | |
| 16:36:45 | gibi | +1 ^^ | |
| 16:37:28 | dansmith | maybe I can just sprinkle some notes in around the configuration bits about service differences, and see if that helps clarity at all | |
| 16:37:46 | gibi | yeah, that could be a good starting point | |
| 16:41:07 | gibi | dansmith: I don't know if you see this but there are also ~bugs uncovered by these config discussion https://bugs.launchpad.net/nova/+bug/1903908 (thank to owalsh for the report) | |
| 16:41:07 | openstack | Launchpad bug 1903908 in puppet-nova "nova conf [api]/dhcp_domain is required on nova-compute" [Undecided,New] - Assigned to Oliver Walsh (owalsh) | |
| 16:42:03 | dansmith | I saw that, I'm not sure how I feel about it | |
| 16:42:15 | dansmith | anything that the configdrive code uses will be shared with the metadata api | |
| 16:43:28 | gibi | yeah, I don't like that metadata calls deep into the metadata code | |
| 16:44:11 | dansmith | configdrive? | |
| 16:44:18 | dansmith | well, that's pretty much the whole point of it :) | |
| 16:44:23 | gibi | the gneeration of the config drive | |
| 16:44:45 | gibi | but this result in the code use by two different services | |
| 16:45:56 | gibi | I guess it is just the fact that I associate the the metadata code only to the metadata service and always forget that ther is a dependency to it from the nova-compute | |
| 16:46:22 | gibi | I guess that lead to the fact that we moved the domain config to the [api] section | |
| 16:46:47 | gibi | anyhow this is for another day as I have to go offline | |
| 16:46:53 | gibi | o/ | |
| 16:49:27 | dansmith | ack, I'll put this up for discussion in a bit | |
| 16:50:56 | owalsh | dansmith: I was going to respond to tkajinam on the ML but I don't really have a response, just a +1 really | |
| 16:52:12 | dansmith | owalsh: okay I guess it's really the same thing anyway | |
| 16:52:49 | owalsh | yes, the first point IIUC is if we have nova.conf and nova-cpu.conf, both containing the generated sample config, how do I know where to set option foo? | |
| 16:52:50 | dansmith | if the packages want to use nova-compute.conf for the compute package, that'd solve it, AFAIK | |
| 16:55:01 | owalsh | ack, only realistic solution I can see right now, but it's also a bit unpleasant | |
| 16:56:22 | dansmith | well, nova/ironic has pretty much never fit into nova properly so .. unsurprising that we have this problem | |
| 16:56:35 | dansmith | we could also just special case the check and not complain if we're using ironic, | |
| 16:56:48 | dansmith | since we're not on a compromise-able situation | |
| 16:57:27 | owalsh | I don't think so, the deb/rpm doesn't know what nova-compute will be used for | |
| 16:57:54 | owalsh | ... when it creates the service | |
| 16:59:11 | dansmith | what I meant was, the compute package could keep using nova.conf, and if your tool then configures nova.conf for ironic-and-db-creds-because-single-node, nova-compute wouldn't explode when it starts | |
| 16:59:31 | dansmith | it wouldn't solve the AIO case, but it would solve the ironic one | |
| 17:00:51 | owalsh | rpm/deb would need a new service unit file e.g "nova-ironic" that uses nova.conf vs nova-compute.conf | |
| 17:01:04 | dansmith | going back to the nova-db.conf suggestion.. if we allow the wsgi app to read that, then you can again just configure all the services to read from that, except for nova-compute, and then the only thing that doesn't work is AIO-multicell right/ | |
| 17:02:20 | owalsh | did seem like the most elegant solution, but also not had a lot of time to think of any gotchas | |
| 17:02:52 | owalsh | and I think AIO-multicell could be considered not supported outside of devstack | |