Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2022-05-24
14:34:16 sean-k-mooney that is not how we recommend deploying nova
14:34:31 ygk_12345 we have OSA Wallaby
14:34:41 sean-k-mooney the compute nodes espically shoudl have a differnt config then the contoller services
14:34:52 sean-k-mooney you can turn it off on the api too
14:35:10 ygk_12345 sean-k-mooney: ok so applying it for computes alone in this case would help. isn't it ?
14:35:13 sean-k-mooney i just will reslt in more rabbit heatbeat lost messages
14:35:30 sean-k-mooney yes applying it just to the comptue nodes will resolve most of your issues
14:35:49 sean-k-mooney but the oslo bug could also impact the conductor and schduler in the same way
14:36:02 ygk_12345 sean-k-mooney: also can you please reply in that bug post as to what are those eventpoll files in the lsof output ?
14:36:12 sean-k-mooney in general its safe to turn of the pthread for heartbeat on all nova services
14:36:21 sean-k-mooney no
14:36:33 sean-k-mooney i close this as a duplicate becasue this is not a nova bug
14:37:12 ygk_12345 sean-k-mooney: thats ok. you can close it,. I want to know for curiosity's sake what are those evenpolls ?
14:37:14 sean-k-mooney i dont know what they are exactly beut based on teh oslo mssaging bug it something related to the amqp lib we use to connect to rabbitmq
14:37:31 ygk_12345 sean-k-mooney: oh ok
14:37:36 sean-k-mooney i belive its eventlet polling the tcp connect to rabbit
14:38:16 ygk_12345 sean-k-mooney: but why there are so many in this case ?
14:38:17 sean-k-mooney but i have not acctully dug into the detail tobias-urdin knows more about it
14:38:25 ygk_12345 oh ok
14:39:01 sean-k-mooney ygk_12345: its leaking them due to an interaction between evently and py-amqp
14:39:06 sean-k-mooney *eventlet
14:39:43 sean-k-mooney ygk_12345: this https://github.com/celery/py-amqp/commit/f4fd4f952dded9ea1006e82642f2c15008bb68d3 apprently is the fix in py-amqp based on the other bug
14:40:16 ygk_12345 sean-k-mooney: ok
14:41:08 ygk_12345 sean-k-mooney: so in this case we are disabling the heartbeat through python green threading. isn't it ?
14:41:54 sean-k-mooney so the workaround is to fallback to runnign the heatbeat as a greentheread
14:42:11 ygk_12345 sean-k-mooney: if we disable pthread, how is the heartbeat maintained ?
14:42:14 sean-k-mooney instead of spwaning a fully unpatched pthread
14:42:25 kashyap A newbie question: on brand-new box, any reason why `git review` gets stuck at the ".git/hooks/pre-review" stage? - https://paste.opendev.org/show/bwdWn2H2H61ZYDUbqZqf/
14:42:46 sean-k-mooney ygk_12345: by a greenthread that is coperativly executed with all the others
14:43:08 sean-k-mooney ygk_12345: the pthread was only used to work around uwsgi killing the heatbeat greenthread
14:43:28 ygk_12345 sean-k-mooney: so there is a diff between pthread and a green thread ?
14:43:30 sean-k-mooney ygk_12345: it was used in teh api too escape the lifecycle manamgnet of the wsgi server
14:44:04 sean-k-mooney greenthreads are userland thread that the kernel does not know about created by greenlet/eventlet
14:44:16 sean-k-mooney pthread are real kernel/os threads
14:44:26 ygk_12345 sean-k-mooney: hmm ok. makes sense now
14:44:34 sean-k-mooney os there are many greenthread to one pthread normlay
14:45:06 sean-k-mooney the reason this was used by wsgi is after a time out appach or uwsgi would kill the interperer following a request
14:45:13 sean-k-mooney and that would kill the heatbeat
14:45:28 sean-k-mooney and that woudl cause a timeout error on rabbitmq side
14:45:41 sean-k-mooney it didnt break anything just create false errors
14:46:03 sean-k-mooney ygk_12345: using a pthread when runing under apache mod_wsgi or uwsgi avoids that
14:46:23 sean-k-mooney nova-compute does nto run under either so its not needed same for the conductor and schduler
14:47:18 sean-k-mooney kashyap: not sure did you install git-review from pypi. you should never use the disto version of git-review
14:47:34 kashyap Never? It works just fine normally :)
14:47:42 sean-k-mooney its very very old normally
14:47:49 sean-k-mooney even on fedora
14:47:54 kashyap Yes, it's distro-based; and it's not related to git-review. It's related to the +ssh-rsa thingie setup
14:48:12 sean-k-mooney oh you have not updated your keys
14:48:38 sean-k-mooney gerrit still does not negociate sha2 and fedora has disable all sha based auth
14:48:53 tobias-urdin ygk_12345: what sean-k-mooney says :D
14:50:21 sean-k-mooney kashyap: https://src.fedoraproject.org/rpms/git-review git review is currently 2.3.1
14:50:33 sean-k-mooney so even f37 is still behind
14:50:41 sean-k-mooney 2.3.1 came out in april
14:50:44 kashyap Yeah, saw that; I'm using 2.2.0.3 version
14:51:04 sean-k-mooney ya form last november
14:51:32 kashyap Right; but that's not the issue here for me. It's something else. I've aloso got the SSH setup:
14:51:35 kashyap Host review.openstack.org
14:51:37 kashyap HostName review.openstack.org
14:51:40 kashyap PubkeyAcceptedKeyTypes +ssh-rsa
14:51:40 kashyap (Required at least on Fedora)
14:51:51 sean-k-mooney yep unless you update your key
15:11:16 gibi I needed both
15:11:17 gibi HostkeyAlgorithms +ssh-rsa
15:11:17 gibi PubkeyAcceptedAlgorithms +ssh-rsa
15:11:30 gibi as new clients does not accept rsa as host key from old servers
15:12:18 sean-k-mooney gibi: ya it depens on the client and host i guess.
15:12:25 sean-k-mooney i dont have either
15:12:35 sean-k-mooney i just created a second key
15:13:05 bauzas reminder : nova meeting in 48 mins here
15:13:47 sean-k-mooney i have an id_ed25519 key and a less secure id_ecdsa as well as id_rsa now
15:14:30 gibi the ssh client stopped accepting rsa host keys since 8.8
15:14:38 gibi *openssh-client
15:15:00 gibi but old servers still send the host key as rsa
15:15:56 gibi https://www.openssh.com/txt/release-8.8
15:16:10 sean-k-mooney right but i think opendev have updated theirs
15:16:16 sean-k-mooney to have ec keys
15:17:51 kashyap Okay; /me uploads EC keys
15:17:59 kashyap gibi: Ah, good to know; lemme try
15:18:49 sean-k-mooney gibi: i think im using 8.8 but i guess i have not updated in a few days
15:19:24 kashyap gibi: W/o that HostkeyAlgorithms, I could SSH to `ssh kashyapc@review.opendev.org -p 29418` just fine
15:21:54 sean-k-mooney actully i think i have 8.7
15:22:27 sean-k-mooney hum OpenSSH_8.8p1, OpenSSL 1.1.1o 3 May 2022
15:22:50 kashyap gibi: No dice; for me `git review` just seem to be stuck in limbo trying to run the 'pre-review' hook (which in turn just runs `tox -e pep8`)
15:23:21 gibi ack
15:23:32 sean-k-mooney kashyap: i think they had to fix somethin gin git-review for this by the way
15:23:36 gibi I don't user git-review
15:24:29 kashyap Heh, I used to as well; but got used to git-review
15:24:35 sean-k-mooney gibi: via git push refs/for/chage?
15:24:38 gibi yepp
15:24:46 sean-k-mooney i havnt done that in about 5 years
15:24:54 kashyap gibi: Then you might like `git submit` -- I really like using it when submitting patches to libvirt/QEMU lists (or any list-based thing)
15:24:57 sean-k-mooney but for the first 2-3 of working on openstack that is what i did too
15:24:59 gibi it is a bit longer command but at least I'm in full control :)
15:25:11 kashyap https://github.com/stefanha/git-publish
15:26:29 kashyap gibi: See this: https://listman.redhat.com/archives/libguestfs/2022-May/028911.html
15:26:40 kashyap It's a wrapper around `git-format-patch` and `git-send-email`
15:26:59 kashyap But for us it doesn't make sense; as we don't use lists :D
15:27:02 gibi thanks, I will try to remember it when I have to send patches by mail
15:28:56 kashyap Okay, when I run 'tox' with -vv, it seems to be stuck here: https://paste.opendev.org/show/baNIMuM5YxlBA8nwxm10/
15:29:03 kashyap INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. See https://pip.pypa.io/warnings/backtracking for guidance. If you want to abort this run, press Ctrl + C. Using cached decorator-4.2.1-py2.py3-none-any.whl (9.3 kB)

Earlier   Later