| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-08-07 | |||
| 13:30:55 | dansmith | mriedem: jes | |
| 13:31:41 | mriedem | thanks | |
| 13:42:44 | mnaser | can i get a second pair of eyes | |
| 13:42:50 | mnaser | i think there is a nova bug here that we uncovered in OSA | |
| 13:42:52 | mnaser | https://logs.opendev.org/33/660333/10/check/openstack-ansible-deploy-aio_metal-ubuntu-bionic/97d8bc3/logs/host/nova-compute.service.journal-23-20-40.log.txt.gz#_Aug_06_22_51_25 | |
| 13:42:58 | mnaser | nova-compute magically trying to access db? | |
| 13:43:07 | mnaser | (and raising an exception because of that) | |
| 13:45:44 | yonglihe | ||
| 13:47:28 | sean-k-mooney | am i can make time. i saw melwitt had a question when i looked last week about shutdown_unkown. was that resolved | |
| 13:47:37 | mnaser | im trying to do git blame on github and its timing out | |
| 13:47:37 | mnaser | boo | |
| 13:47:44 | mriedem | mnaser: hmm, yeah compute is calling compute.API.stop to stop the instance, which is hitting some code in the compute rpcapi client router to determine the minimum compute version across all cells which it shouldn't need to be doing, b/c you're in a cell | |
| 13:48:18 | mnaser | thats what i figured based on the traceback but i havent done my share of git blame yet to see when this might be started | |
| 13:48:25 | mnaser | https://github.com/openstack/nova/blame/master/nova/compute/manager.py crashes github | |
| 13:48:45 | mriedem | mnaser: https://review.opendev.org/#/c/649197/ | |
| 13:48:56 | sean-k-mooney | mnaser: ya that and the libvirt dirver are two of the files better blamed on the terminal | |
| 13:49:37 | mriedem | i want to say we should be hitting that when compute starts up though, or used to? | |
| 13:49:57 | mriedem | mnaser: do you have the api db configured in nova.conf that you're computes are using? | |
| 13:50:10 | mnaser | mriedem: in this case i believe its an aio so yes it will be there | |
| 13:50:23 | mnaser | in an actual deployment, osa doesnt do this | |
| 13:50:36 | mriedem | even if it's a single node you can run nova-compute with a different config file | |
| 13:50:38 | mriedem | w/o the db creds | |
| 13:50:42 | mriedem | that's what devstack does | |
| 13:50:42 | dansmith | mnaser: is the api db connection set? | |
| 13:50:55 | dansmith | I think that's how it determines if it should use the all-cells or this-cell method | |
| 13:51:14 | mnaser | yeah.. it would be a very 'CI' specific thing if we used 2 different configs, unless we always used 2 configs, but yeah | |
| 13:51:34 | mnaser | if we have to roll out nova-compute.conf and nova.conf -- that's fine, we'll fix that | |
| 13:51:38 | sean-k-mooney | mnaser: well always using two config makes sense in a way | |
| 13:51:40 | dansmith | mnaser: was that aimed at me? | |
| 13:52:01 | sean-k-mooney | it is nice to be able to seperate compute only things for contoler only things | |
| 13:52:05 | mnaser | dansmith: just in general, im wondering if we're doing things wrong (i.e. nova operates with the assumption that api db _should not_ be set in nova-compute) | |
| 13:52:23 | mriedem | i've got to run my kid somewhere, bbiab | |
| 13:52:38 | dansmith | mnaser: the assumption is that computes should not be talking to any db, and thus shouldn't have creds set at all | |
| 13:52:57 | dansmith | mnaser: like, the creds shouldn't even need to be on those systems | |
| 13:53:06 | sean-k-mooney | the compute talkign directly to a db was a legacy feature correect | |
| 13:53:13 | sean-k-mooney | its inteded to go via the conductor | |
| 13:53:18 | mnaser | right, in a 'production' deployment of OSA (where compute != control), the value is not set, but in our aio, the logic is compute is inside the controller group so it gets added | |
| 13:53:41 | mnaser | i agree with you that it should not be set | |
| 13:53:44 | dansmith | mnaser: okay, but it should have a different config which matches what a compute looks like non-aio | |
| 13:54:22 | mnaser | https://github.com/openstack/openstack-ansible-os_nova/blob/master/templates/nova.conf.j2#L196-L212 | |
| 13:54:44 | mnaser | so we only set these if we're running against a host that is not part of nova-compute group, but in this case in aio, every host is so it gets included | |
| 13:54:46 | dansmith | this is obviously failing because of a somewhat fragile assumption about what having api db creds set means, but.. this is why it's good to use realistic configs even in an aio situation | |
| 13:55:27 | sean-k-mooney | mnaser: do ye share config files between services? | |
| 13:55:43 | mnaser | sean-k-mooney: yes, but its templated to be different depending where it ends up getting deployed | |
| 13:55:46 | sean-k-mooney | e.g. i know kolla ansible generate a different nova.conf per nova service even on the same host | |
| 13:56:04 | mnaser | yes we do that too, but they all end up at /etc/nova/nova.conf | |
| 13:56:05 | sean-k-mooney | mnaser: right that is different then kolla ansible | |
| 13:56:17 | sean-k-mooney | right within the contianer | |
| 13:56:25 | sean-k-mooney | but different paths in the host right | |
| 13:56:42 | mnaser | nah its always at /etc/nova/nova.conf here | |
| 13:56:58 | sean-k-mooney | so its alway one file on the host? | |
| 13:57:04 | mnaser | yep | |
| 13:57:15 | mnaser | i mean for me its fine, if from a nova perspective, the assumption that api_db shouldnt be defined, i can fix it inside osa, but if you think that assumption isn't right, i can also fix it inside nova | |
| 13:57:24 | sean-k-mooney | ach well that explains why this does not work then | |
| 13:57:59 | sean-k-mooney | well the assumtion is no db will be defiend in the compute agent's nova.conf | |
| 13:58:23 | sean-k-mooney | all db acess form the comptue agent is expected to go via the cell conductor | |
| 13:58:48 | mnaser | yep, i agree with that | |
| 13:59:22 | dansmith | mnaser: I think you should change OSA so that AIO and non-AIO are effectively the same by using separate nova confs per service | |
| 14:00:08 | alex_xu | sean-k-mooney: can we have hw_cpu_policy as mixed, then have hw:cpu_dedicated_mask? | |
| 14:00:23 | mnaser | dansmith: makes sense, i'll have us roll out config to nova-$service.conf inside OSA and i think that should clear this up | |
| 14:00:31 | mnaser | thanks for the help and heads up :>? | |
| 14:00:31 | dansmith | mnaser: cool | |
| 14:01:22 | sean-k-mooney | for what its worth kolla-ansible still use one template but generates multiple files and this is how they guard the db section | |
| 14:01:24 | sean-k-mooney | https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/nova/templates/nova.conf.j2#L156-L166 | |
| 14:02:21 | mnaser | yeah im thinking that probably is in line of what we'll end u pdoing | |
| 14:03:02 | sean-k-mooney | it makes the condional simpler as most are based on the service name instaead of the group you are in or if a service is enabled | |
| 14:03:23 | mnaser | yeah that does sound easier than a bunch of complicated jinja2 | |
| 14:03:24 | sean-k-mooney | but it requires the template to be evaulated and generated seperatly per service | |
| 14:03:46 | mnaser | yeah that does increase the # of times we have to do it but eh | |
| 14:04:03 | mnaser | if its the right thing(tm) then i rather do that than a few less tasks | |
| 14:04:42 | sean-k-mooney | ya it synergies really nicely with kollas config overide mechanisum | |
| 14:05:30 | sean-k-mooney | perhaps it could be broken out at somepoint and shared between kolla-ansible, tripleo and osa | |
| 14:05:41 | yonglihe | sean-k-mooney: i reply melwitt but not sure is that what he concern. but thanks your review. | |
| 14:20:26 | dansmith | sean-k-mooney: you did not just say "it synergies" ... | |
| 14:21:41 | sean-k-mooney | for once im using it an example of its usage in a non marketing bs context | |
| 14:22:32 | dansmith | heh | |
| 14:24:43 | sean-k-mooney | the config file per service per host design works realy well with the config override system kolla-ansible has. i think it was the config override desigin goals that acttuly motivated that approch initally so i guess its not suprising they work well together | |
| 14:36:01 | noonedeadpunk | And does that make sense to create nova-conductor.conf and place db credentials only there? | |
| 14:36:49 | dansmith | noonedeadpunk: devstack does a nova-cell1.conf for all the non-compute services that need to talk to the database, but yeah | |
| 14:37:17 | dansmith | nova.conf has everything, and is used by the control services, api, etc. nova-cell1 is no api, just cell db creds, and nova-compute.conf is the bare minimum for compute | |
| 14:37:27 | stephenfin | efried, gibi: What's the expected behavior here? http://paste.openstack.org/show/755623/ | |
| 14:37:43 | stephenfin | Maybe dansmith and sean-k-mooney too, idk ^ | |
| 14:37:51 | noonedeadpunk | dansmith: thanks | |
| 14:38:14 | sean-k-mooney | stephenfin: for legacy reasons that shoudl be 2 | |
| 14:38:27 | sean-k-mooney | e.g. we have alway allowed that | |
| 14:38:39 | sean-k-mooney | but i dont like that you can change it | |
| 14:39:01 | sean-k-mooney | if we block it it could break existing flavors | |
| 14:39:13 | sean-k-mooney | so there would be an upgrade impact | |
| 14:39:39 | dansmith | sean-k-mooney: why for legacy reasons? | |
| 14:39:41 | mriedem | technically devstack has several conf files for nova because of the superconductor mode | |
| 14:40:07 | sean-k-mooney | dansmith: we decided to allow operators to specify resouce request in the flavor extra specs | |
| 14:40:16 | sean-k-mooney | that is the sole legacy use of this | |
| 14:40:28 | dansmith | sean-k-mooney: I'm not sure what you mean | |
| 14:40:38 | dansmith | resource requests take precedence over the base flavor values | |
| 14:40:40 | dansmith | that's new behavior | |
| 14:40:45 | dansmith | and critical for things like ironic | |
| 14:40:55 | sean-k-mooney | dansmith: no its not | |
| 14:41:05 | mriedem | yes...it is | |
| 14:41:08 | dansmith | where we "display" a value to the user via the flavor, but ask for something different from placement based on the overrides | |
| 14:41:10 | mriedem | as of pike | |