| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-08-07 | |||
| 12:51:56 | sean-k-mooney | actuuly xor the bits and check the result is 0 | |
| 12:52:30 | sean-k-mooney | actully no that is also wrong | |
| 12:53:05 | sean-k-mooney | but you get the idea we just need to validate each bit set in the realtime mask is set in the dedicated mask | |
| 12:53:19 | alex_xu | yea | |
| 12:53:23 | alex_xu | i got it | |
| 12:55:18 | sean-k-mooney | to dansmith point however i do belive we discusssed the idea of not exposing a mask in the past and instead doing what he say and just having tally counts of how many of each type you want | |
| 12:55:54 | sean-k-mooney | then evenly distubting them over the numa nodes and ensureing within a numa node the shared cpus are first followed by the pinned ones | |
| 12:56:16 | sean-k-mooney | that fits the paradime where core 0 is often reserved for the os | |
| 12:56:53 | sean-k-mooney | and in a multi numa works the first core on each numa node is often reserved for the os | |
| 12:57:17 | sean-k-mooney | but that does seam like magic that would easily break | |
| 13:00:39 | alex_xu | good to know | |
| 13:01:53 | sean-k-mooney | alex_xu: i saw the irc ping for the patch. what tempeted you to work on that | |
| 13:02:23 | sean-k-mooney | also what time zone are you in. i assume its eitehr quite late or very early for you | |
| 13:05:24 | alex_xu | sean-k-mooney: hah | |
| 13:19:53 | openstackgerrit | Shilpa Devharakar proposed openstack/nova master: Enable request filter isolate_aggregates https://review.opendev.org/667952 | |
| 13:23:58 | openstackgerrit | Matt Riedemann proposed openstack/python-novaclient master: WIP: Add --migration-type and --source-compute to migration-list https://review.opendev.org/675117 | |
| 13:23:58 | openstackgerrit | Matt Riedemann proposed openstack/python-novaclient master: docs: clarify nova migration-list --host option https://review.opendev.org/674950 | |
| 13:28:59 | mriedem | lyarwood: as you're probably aware once these land https://review.opendev.org/#/q/I5e0a43ec59341c9ac62f89105ddf82c4a014df81 we'll be doing stable branch releases | |
| 13:29:35 | sean-k-mooney | mriedem: those are the cve backports right | |
| 13:29:40 | mriedem | yes | |
| 13:29:46 | mriedem | dansmith: can you get this this stein backport? https://review.opendev.org/#/c/672723/ | |
| 13:30:09 | mriedem | melwitt: can you get these stein backports for a series you're familiar with https://review.opendev.org/#/q/topic:bug/1764556+(status:open+OR+status:merged)+branch:stable/stein | |
| 13:30:39 | lyarwood | mriedem: ack yup | |
| 13:30:55 | dansmith | mriedem: jes | |
| 13:31:41 | mriedem | thanks | |
| 13:42:44 | mnaser | can i get a second pair of eyes | |
| 13:42:50 | mnaser | i think there is a nova bug here that we uncovered in OSA | |
| 13:42:52 | mnaser | https://logs.opendev.org/33/660333/10/check/openstack-ansible-deploy-aio_metal-ubuntu-bionic/97d8bc3/logs/host/nova-compute.service.journal-23-20-40.log.txt.gz#_Aug_06_22_51_25 | |
| 13:42:58 | mnaser | nova-compute magically trying to access db? | |
| 13:43:07 | mnaser | (and raising an exception because of that) | |
| 13:45:44 | yonglihe | ||
| 13:47:28 | sean-k-mooney | am i can make time. i saw melwitt had a question when i looked last week about shutdown_unkown. was that resolved | |
| 13:47:37 | mnaser | boo | |
| 13:47:37 | mnaser | im trying to do git blame on github and its timing out | |
| 13:47:44 | mriedem | mnaser: hmm, yeah compute is calling compute.API.stop to stop the instance, which is hitting some code in the compute rpcapi client router to determine the minimum compute version across all cells which it shouldn't need to be doing, b/c you're in a cell | |
| 13:48:18 | mnaser | thats what i figured based on the traceback but i havent done my share of git blame yet to see when this might be started | |
| 13:48:25 | mnaser | https://github.com/openstack/nova/blame/master/nova/compute/manager.py crashes github | |
| 13:48:45 | mriedem | mnaser: https://review.opendev.org/#/c/649197/ | |
| 13:48:56 | sean-k-mooney | mnaser: ya that and the libvirt dirver are two of the files better blamed on the terminal | |
| 13:49:37 | mriedem | i want to say we should be hitting that when compute starts up though, or used to? | |
| 13:49:57 | mriedem | mnaser: do you have the api db configured in nova.conf that you're computes are using? | |
| 13:50:10 | mnaser | mriedem: in this case i believe its an aio so yes it will be there | |
| 13:50:23 | mnaser | in an actual deployment, osa doesnt do this | |
| 13:50:36 | mriedem | even if it's a single node you can run nova-compute with a different config file | |
| 13:50:38 | mriedem | w/o the db creds | |
| 13:50:42 | dansmith | mnaser: is the api db connection set? | |
| 13:50:42 | mriedem | that's what devstack does | |
| 13:50:55 | dansmith | I think that's how it determines if it should use the all-cells or this-cell method | |
| 13:51:14 | mnaser | yeah.. it would be a very 'CI' specific thing if we used 2 different configs, unless we always used 2 configs, but yeah | |
| 13:51:34 | mnaser | if we have to roll out nova-compute.conf and nova.conf -- that's fine, we'll fix that | |
| 13:51:38 | sean-k-mooney | mnaser: well always using two config makes sense in a way | |
| 13:51:40 | dansmith | mnaser: was that aimed at me? | |
| 13:52:01 | sean-k-mooney | it is nice to be able to seperate compute only things for contoler only things | |
| 13:52:05 | mnaser | dansmith: just in general, im wondering if we're doing things wrong (i.e. nova operates with the assumption that api db _should not_ be set in nova-compute) | |
| 13:52:23 | mriedem | i've got to run my kid somewhere, bbiab | |
| 13:52:38 | dansmith | mnaser: the assumption is that computes should not be talking to any db, and thus shouldn't have creds set at all | |
| 13:52:57 | dansmith | mnaser: like, the creds shouldn't even need to be on those systems | |
| 13:53:06 | sean-k-mooney | the compute talkign directly to a db was a legacy feature correect | |
| 13:53:13 | sean-k-mooney | its inteded to go via the conductor | |
| 13:53:18 | mnaser | right, in a 'production' deployment of OSA (where compute != control), the value is not set, but in our aio, the logic is compute is inside the controller group so it gets added | |
| 13:53:41 | mnaser | i agree with you that it should not be set | |
| 13:53:44 | dansmith | mnaser: okay, but it should have a different config which matches what a compute looks like non-aio | |
| 13:54:22 | mnaser | https://github.com/openstack/openstack-ansible-os_nova/blob/master/templates/nova.conf.j2#L196-L212 | |
| 13:54:44 | mnaser | so we only set these if we're running against a host that is not part of nova-compute group, but in this case in aio, every host is so it gets included | |
| 13:54:46 | dansmith | this is obviously failing because of a somewhat fragile assumption about what having api db creds set means, but.. this is why it's good to use realistic configs even in an aio situation | |
| 13:55:27 | sean-k-mooney | mnaser: do ye share config files between services? | |
| 13:55:43 | mnaser | sean-k-mooney: yes, but its templated to be different depending where it ends up getting deployed | |
| 13:55:46 | sean-k-mooney | e.g. i know kolla ansible generate a different nova.conf per nova service even on the same host | |
| 13:56:04 | mnaser | yes we do that too, but they all end up at /etc/nova/nova.conf | |
| 13:56:05 | sean-k-mooney | mnaser: right that is different then kolla ansible | |
| 13:56:17 | sean-k-mooney | right within the contianer | |
| 13:56:25 | sean-k-mooney | but different paths in the host right | |
| 13:56:42 | mnaser | nah its always at /etc/nova/nova.conf here | |
| 13:56:58 | sean-k-mooney | so its alway one file on the host? | |
| 13:57:04 | mnaser | yep | |
| 13:57:15 | mnaser | i mean for me its fine, if from a nova perspective, the assumption that api_db shouldnt be defined, i can fix it inside osa, but if you think that assumption isn't right, i can also fix it inside nova | |
| 13:57:24 | sean-k-mooney | ach well that explains why this does not work then | |
| 13:57:59 | sean-k-mooney | well the assumtion is no db will be defiend in the compute agent's nova.conf | |
| 13:58:23 | sean-k-mooney | all db acess form the comptue agent is expected to go via the cell conductor | |
| 13:58:48 | mnaser | yep, i agree with that | |
| 13:59:22 | dansmith | mnaser: I think you should change OSA so that AIO and non-AIO are effectively the same by using separate nova confs per service | |
| 14:00:08 | alex_xu | sean-k-mooney: can we have hw_cpu_policy as mixed, then have hw:cpu_dedicated_mask? | |
| 14:00:23 | mnaser | dansmith: makes sense, i'll have us roll out config to nova-$service.conf inside OSA and i think that should clear this up | |
| 14:00:31 | dansmith | mnaser: cool | |
| 14:00:31 | mnaser | thanks for the help and heads up :>? | |
| 14:01:22 | sean-k-mooney | for what its worth kolla-ansible still use one template but generates multiple files and this is how they guard the db section | |
| 14:01:24 | sean-k-mooney | https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/nova/templates/nova.conf.j2#L156-L166 | |
| 14:02:21 | mnaser | yeah im thinking that probably is in line of what we'll end u pdoing | |
| 14:03:02 | sean-k-mooney | it makes the condional simpler as most are based on the service name instaead of the group you are in or if a service is enabled | |
| 14:03:23 | mnaser | yeah that does sound easier than a bunch of complicated jinja2 | |
| 14:03:24 | sean-k-mooney | but it requires the template to be evaulated and generated seperatly per service | |
| 14:03:46 | mnaser | yeah that does increase the # of times we have to do it but eh | |
| 14:04:03 | mnaser | if its the right thing(tm) then i rather do that than a few less tasks | |
| 14:04:42 | sean-k-mooney | ya it synergies really nicely with kollas config overide mechanisum | |
| 14:05:30 | sean-k-mooney | perhaps it could be broken out at somepoint and shared between kolla-ansible, tripleo and osa | |
| 14:05:41 | yonglihe | sean-k-mooney: i reply melwitt but not sure is that what he concern. but thanks your review. | |
| 14:20:26 | dansmith | sean-k-mooney: you did not just say "it synergies" ... | |
| 14:21:41 | sean-k-mooney | for once im using it an example of its usage in a non marketing bs context | |
| 14:22:32 | dansmith | heh | |
| 14:24:43 | sean-k-mooney | the config file per service per host design works realy well with the config override system kolla-ansible has. i think it was the config override desigin goals that acttuly motivated that approch initally so i guess its not suprising they work well together | |