| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-11-19 | |||
| 18:26:30 | sean-k-mooney | ah ya | |
| 18:26:45 | dansmith | actually, | |
| 18:27:11 | dansmith | we might only need it on reqspec, since I think we have that the whole time, which means we could join it to BR if we need to | |
| 18:27:21 | dansmith | but anyway, just gotta do it, shouldn't be terrible | |
| 18:27:55 | sean-k-mooney | yep if we have it in at least one location in the api db and cell db for each instance we shoudl be ok | |
| 18:28:01 | dansmith | yeah | |
| 18:28:23 | sean-k-mooney | we would need the other serivce to have the same logic however so maybe a keysotne middelware change | |
| 18:28:43 | sean-k-mooney | regarding the "its a domain admin and the project nolonger exists" logic | |
| 18:29:25 | sean-k-mooney | so that when we call neutron and cinder it actully works | |
| 18:29:34 | dansmith | actually works would be good | |
| 18:29:46 | sean-k-mooney | i missed the list dicussion | |
| 18:30:00 | sean-k-mooney | is that server list --all-tenats | |
| 18:30:06 | dansmith | yeah | |
| 18:30:15 | dansmith | I gotta run do something, back later | |
| 18:30:29 | sean-k-mooney | ok to me the simpelt way to do that is again with domain tokens and scope it to the project in that domain | |
| 18:30:51 | sean-k-mooney | ok im going to finsih soon but ^ is how i asuemd that would work | |
| 18:31:14 | sean-k-mooney | if you really wanted all proejct then you would use a domain token on the root domain, assuming keystone exposes that | |
| 20:11:54 | opendevreview | Dmitrii Shcherbakov proposed openstack/nova master: [yoga] Add PCI VPD Capability Handling https://review.opendev.org/c/openstack/nova/+/808199 | |
| 20:11:55 | opendevreview | Dmitrii Shcherbakov proposed openstack/nova master: [yoga] Support remote-managed SmartNIC DPU ports https://review.opendev.org/c/openstack/nova/+/812111 | |
| #openstack-nova - 2021-11-21 | |||
| 09:55:16 | kevko | any help please ? https://bugs.launchpad.net/nova/+bug/1951720 ? have you seen this already ? | |
| #openstack-nova - 2021-11-22 | |||
| 08:50:21 | bauzas | good morning Nova | |
| 08:50:54 | gibi | good morning | |
| 08:51:50 | bauzas | you appreciate winter times when you have to light on | |
| 13:16:53 | gibi | bauzas: I think this spec is a quick +A https://review.opendev.org/c/openstack/nova-specs/+/810868 if you have minute : | |
| 13:16:56 | gibi | :) | |
| 13:17:16 | bauzas | gibi: I'll need to taxi my daughter in a few mins but I'll look at it after | |
| 13:19:21 | sean-k-mooney | gibi: i kind of agree | |
| 13:19:23 | gibi | bauzas: OK, sure | |
| 13:19:41 | sean-k-mooney | i can take a look but ill leave the +w to bauzas | |
| 13:26:29 | Henriqueof | Is there any articles/guides on how overcommiting CPU/RAM degrades performance? | |
| 13:34:38 | sean-k-mooney | Henriqueof: well over commiting ram is easy to understand in that when you actully start overcommiting the inuse ram it will start swapping to disk | |
| 13:35:42 | sean-k-mooney | using something like zram as first level swap and an actual swap partion as second level swap and have 1 swap partion per numa node can in some casess help but only if the storage is also numa alinged | |
| 13:36:06 | sean-k-mooney | Henriqueof: for cpus it is really just a matter of contntion and context switchign overhead | |
| 13:36:46 | sean-k-mooney | if you vms are mostly idel it will be fien to over commit them but once the host load starts to exceed the number of cpus then the performcne will degrade | |
| 13:37:18 | sean-k-mooney | you likely will hit memory bandwith, disk io or network io bottelneck too depending on your workloads | |
| 13:38:38 | sean-k-mooney | Henriqueof: my recommendation are never over commit cpus more then 4:1, always reserve at least 1 core per numa node for the host, if you are using hyper treading reserve the hypertread sibling of each host core too. | |
| 13:39:26 | sean-k-mooney | Henriqueof: hyperthreading only give you about a 1.4x increase in throughput by the way so dont expect to actully be able to service a load = nproc if you are using HT | |
| 13:40:07 | sean-k-mooney | Henriqueof: form memroy i normally recommend never over commiting and using hugepages but if you must over commit allocate swap equal to total memory*overcommit ratio | |
| 13:41:10 | sean-k-mooney | i would not really over commit more then about 2-4x your ram either. but as i said i recomend keeping memory over comiit at 1.0 so no over commit in most cases. | |
| 14:00:14 | Henriqueof | sean-k-mooney: You actually answered most of my questions, thank you! | |
| 14:02:16 | Henriqueof | I find odd the OpenStack docs says that they overcommit CPU and RAM by default, but kolla-ansible doesn't seens to do that. | |
| 14:03:25 | sean-k-mooney | we have our default set to overcommit cpu by 16:1 and ram by 1.5:1 | |
| 14:03:29 | kashyap | Henriqueof: Who are the users of 'kolla-ansible'? | |
| 14:03:41 | kashyap | (Do people use it manually, or do tools use it mostly?) | |
| 14:04:07 | sean-k-mooney | they are old defaults from when when openstack was used by nasa and rackspace mainly for webhosting/data storage | |
| 14:05:28 | sean-k-mooney | kashyap: its one of the more popular installers its often used via kayobe which is supported by stackhpc https://www.stackhpc.com/pages/kayobe.html | |
| 14:06:03 | sean-k-mooney | kashyap: the company johnthetubaguy[m] works at if he has not moved on. | |
| 14:06:22 | kashyap | sean-k-mooney: Right; I vaguely know the tool is an installer. Didn't know how much it is actually used in production | |
| 14:06:29 | kashyap | I see, noted. | |
| 14:06:40 | sean-k-mooney | kashyap: so most of the user are HPC or scientific user or goverment/university installation i belive | |
| 14:07:49 | sean-k-mooney | kashyap: the highest profile use is proably SKA the Square Kilometer Array telescope | |
| 14:08:05 | kashyap | Cool; good to know :) | |
| 14:08:44 | Henriqueof | sean-k-mooney: Really? Until now I thought kolla-ansible was one of the most popular deployment tool. | |
| 14:10:40 | sean-k-mooney | Henriqueof: it is yes | |
| 14:11:04 | sean-k-mooney | Henriqueof: im not sure how much market share it has vs tripleo,openstack charms and openstack ansible | |
| 14:11:16 | sean-k-mooney | but those are the big 4 deployment tools | |
| 14:14:06 | sean-k-mooney | looking at https://www.openstack.org/analytics | |
| 14:14:14 | sean-k-mooney | if you go to Deployment Decisions | |
| 14:14:27 | Henriqueof | Yeah, it is very straight forward and stable tool so I never felt the nned to experiment with the others. | |
| 14:14:34 | sean-k-mooney | 29% of respondence used kolla ansible | |
| 14:15:21 | sean-k-mooney | which is about the same as juju/triplo/OSA combined | |
| 14:15:48 | sean-k-mooney | that does not tell you how big the deployment are however | |
| 14:16:27 | sean-k-mooney | so ther might be more respondnce using kolla-ansible but that does not mean there are more servers managed by it but it at least give some indeicaionts of it popularity | |
| 14:21:15 | kashyap | stephenfin: Hey, have you ever used this? - sphinxcontrib-spelling | |
| 14:21:27 | kashyap | [https://sphinxcontrib-spelling.readthedocs.io/en/latest/] | |
| 14:23:28 | sean-k-mooney | kashyap: i would expect it would have issues with the terms we use like extra-spec | |
| 14:24:02 | kashyap | Right; but still I wonder is it overall a net win or not | |
| 14:24:03 | sean-k-mooney | we likely could include a dictionary with those but that might get tedious | |
| 14:24:36 | kashyap | sean-k-mooney: Yes, it can use project-specific dictionaries | |
| 15:25:46 | opendevreview | Merged openstack/nova-specs master: Repropose Add libvirt support for flavor and image defined ephemeral encryption https://review.opendev.org/c/openstack/nova-specs/+/810868 | |
| 15:38:08 | opendevreview | Dan Smith proposed openstack/nova master: Allow per-context rule in error messages https://review.opendev.org/c/openstack/nova/+/816865 | |
| 15:38:08 | opendevreview | Dan Smith proposed openstack/nova master: Revert project-specific APIs for servers https://review.opendev.org/c/openstack/nova/+/816206 | |
| 15:38:40 | dansmith | gmann: johnthetubaguy[m]: Removed the WIPs from these ^ as I'm assuming there are no more fundamental concerns | |
| 15:39:23 | gmann | dansmith: ack, I will check today. thanks | |
| 15:48:14 | dansmith | can we get this merged? https://review.opendev.org/c/openstack/nova/+/817030 | |
| 15:48:29 | dansmith | it's already being used to debug gate and real VIF plugging event failures | |
| 15:49:54 | gibi | dansmith: done | |
| 15:51:54 | dansmith | gibi: thanks | |
| 16:51:33 | kashyap | In CirrOS latest 0.5.2, where is this file? /etc/cirros-init/config? | |
| 16:51:46 | kashyap | Is it moved to somewhere else? /me didn't find it in a quick libguestfs inspection | |
| 16:57:49 | kashyap | Actually, ignore me. It's still there. | |
| 17:35:50 | opendevreview | Merged openstack/nova master: Log instance event wait times https://review.opendev.org/c/openstack/nova/+/817030 | |
| 18:39:05 | opendevreview | Merged openstack/nova master: nova-manage: Always get BDMs using get_by_volume_and_instance https://review.opendev.org/c/openstack/nova/+/811716 | |
| 18:56:29 | mnaser | hi y'all | |
| 18:56:57 | mnaser | has anyone ran into an issue where the api stops responding if the notification transport is failing? | |
| 18:57:36 | mnaser | i.e. oslo_messaging_notificaitons/transport_url = rabbit://foobar , where foobar goes down, and the DEFAULT/transport_url still is up, but i guess the threads all get blocked till it grinds down to a halt? | |
| 18:58:17 | mnaser | i've repro'd on a customer environment that is deployed by OSA but i'm trying to get a devstack up right now and get GMR to see how it hands | |
| 18:58:20 | mnaser | s/hands/hangs/ | |
| 18:59:40 | sean-k-mooney | it might be related to the heartbeat | |
| 18:59:51 | sean-k-mooney | or the wsgi server | |
| 19:00:18 | sean-k-mooney | if you are using mod_wsgi under apptach each worker will only ever service 1 api request at a time | |
| 19:01:04 | sean-k-mooney | we may monkey patch the api but that will never allow the apache process to service a second request in parallel as that is managed by apache | |
| 19:01:48 | sean-k-mooney | if all the api workers are trying to do somethign that needs rabit then it will stop responding until the request or rpc timeout fires and it retruns an error | |
| 19:02:30 | sean-k-mooney | i dont know if uwsgi is better in that regard | |
| 19:03:02 | mnaser | sean-k-mooney: OSA deploys with uwsgi | |
| 19:03:37 | mnaser | sean-k-mooney: i'm still doing my research, but also, i suspect this affects n-cond too | |
| 19:03:49 | mnaser | and anything rabbit related, it seems like the notification blocks the main process | |
| 19:04:50 | mnaser | or maybe when the queue of unsent messages gets so big, the whole process bogs down | |
| 19:05:01 | mnaser | or it has a limit of threads it will bubble up to and then the whole process stops responding | |
| 19:05:54 | sean-k-mooney | its possible that the eventlet thread pool will file up eventually | |