| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-05-24 | |||
| 13:53:51 | sean-k-mooney | kashyap: well i knew exactly where too look if that was the issue and it was so it took like 20 seconds to find | |
| 13:54:07 | kashyap | sean-k-mooney: Sure, even that; you bothered to look :) | |
| 13:56:09 | kashyap | sean-k-mooney: gibi: Just to confirm; that's it - using py38 works. Thx! | |
| 13:58:33 | gmann | sean-k-mooney[m]: ussuri is EM, yes tempest and constraints both will be older and stable/constraints. that is I am working on this series and testing https://review.opendev.org/q/topic:ussuri-pin-tempest | |
| 13:58:39 | sean-k-mooney | we have the same issue on stable ussuri currently | |
| 13:59:12 | sean-k-mooney | gmann: right so currently its using master and failing because ussuri is using py36 and master dose not support it | |
| 13:59:47 | sean-k-mooney | so that need to be fixed in devstack to eitehr clamp ot stable/ussuri or at the latest yoga | |
| 13:59:59 | sean-k-mooney | since that is the last version to support py 36 | |
| 14:00:38 | gmann | yeah. i know | |
| 14:02:01 | opendevreview | Artom Lifshitz proposed openstack/nova stable/wallaby: DNM: Testing live migration with local attach https://review.opendev.org/c/openstack/nova/+/843146 | |
| 14:27:28 | ygk_12345 | can someone respond to this please https://bugs.launchpad.net/nova/+bug/1973887 | |
| 14:32:56 | sean-k-mooney | ygk_12345: you put [oslo_messaging_rabbit] | |
| 14:32:58 | sean-k-mooney | heartbeat_in_pthread = False | |
| 14:33:01 | sean-k-mooney | in nova.conf | |
| 14:33:22 | sean-k-mooney | in this case in the nova.conf used by nova-compute | |
| 14:33:49 | ygk_12345 | sean-k-mooney: you said to avoid it in other nova services except nova-api. how to do that since all nova services have a single nova.conf file | |
| 14:34:16 | sean-k-mooney | that is not how we recommend deploying nova | |
| 14:34:31 | ygk_12345 | we have OSA Wallaby | |
| 14:34:41 | sean-k-mooney | the compute nodes espically shoudl have a differnt config then the contoller services | |
| 14:34:52 | sean-k-mooney | you can turn it off on the api too | |
| 14:35:10 | ygk_12345 | sean-k-mooney: ok so applying it for computes alone in this case would help. isn't it ? | |
| 14:35:13 | sean-k-mooney | i just will reslt in more rabbit heatbeat lost messages | |
| 14:35:30 | sean-k-mooney | yes applying it just to the comptue nodes will resolve most of your issues | |
| 14:35:49 | sean-k-mooney | but the oslo bug could also impact the conductor and schduler in the same way | |
| 14:36:02 | ygk_12345 | sean-k-mooney: also can you please reply in that bug post as to what are those eventpoll files in the lsof output ? | |
| 14:36:12 | sean-k-mooney | in general its safe to turn of the pthread for heartbeat on all nova services | |
| 14:36:21 | sean-k-mooney | no | |
| 14:36:33 | sean-k-mooney | i close this as a duplicate becasue this is not a nova bug | |
| 14:37:12 | ygk_12345 | sean-k-mooney: thats ok. you can close it,. I want to know for curiosity's sake what are those evenpolls ? | |
| 14:37:14 | sean-k-mooney | i dont know what they are exactly beut based on teh oslo mssaging bug it something related to the amqp lib we use to connect to rabbitmq | |
| 14:37:31 | ygk_12345 | sean-k-mooney: oh ok | |
| 14:37:36 | sean-k-mooney | i belive its eventlet polling the tcp connect to rabbit | |
| 14:38:16 | ygk_12345 | sean-k-mooney: but why there are so many in this case ? | |
| 14:38:17 | sean-k-mooney | but i have not acctully dug into the detail tobias-urdin knows more about it | |
| 14:38:25 | ygk_12345 | oh ok | |
| 14:39:01 | sean-k-mooney | ygk_12345: its leaking them due to an interaction between evently and py-amqp | |
| 14:39:06 | sean-k-mooney | *eventlet | |
| 14:39:43 | sean-k-mooney | ygk_12345: this https://github.com/celery/py-amqp/commit/f4fd4f952dded9ea1006e82642f2c15008bb68d3 apprently is the fix in py-amqp based on the other bug | |
| 14:40:16 | ygk_12345 | sean-k-mooney: ok | |
| 14:41:08 | ygk_12345 | sean-k-mooney: so in this case we are disabling the heartbeat through python green threading. isn't it ? | |
| 14:41:54 | sean-k-mooney | so the workaround is to fallback to runnign the heatbeat as a greentheread | |
| 14:42:11 | ygk_12345 | sean-k-mooney: if we disable pthread, how is the heartbeat maintained ? | |
| 14:42:14 | sean-k-mooney | instead of spwaning a fully unpatched pthread | |
| 14:42:25 | kashyap | A newbie question: on brand-new box, any reason why `git review` gets stuck at the ".git/hooks/pre-review" stage? - https://paste.opendev.org/show/bwdWn2H2H61ZYDUbqZqf/ | |
| 14:42:46 | sean-k-mooney | ygk_12345: by a greenthread that is coperativly executed with all the others | |
| 14:43:08 | sean-k-mooney | ygk_12345: the pthread was only used to work around uwsgi killing the heatbeat greenthread | |
| 14:43:28 | ygk_12345 | sean-k-mooney: so there is a diff between pthread and a green thread ? | |
| 14:43:30 | sean-k-mooney | ygk_12345: it was used in teh api too escape the lifecycle manamgnet of the wsgi server | |
| 14:44:04 | sean-k-mooney | greenthreads are userland thread that the kernel does not know about created by greenlet/eventlet | |
| 14:44:16 | sean-k-mooney | pthread are real kernel/os threads | |
| 14:44:26 | ygk_12345 | sean-k-mooney: hmm ok. makes sense now | |
| 14:44:34 | sean-k-mooney | os there are many greenthread to one pthread normlay | |
| 14:45:06 | sean-k-mooney | the reason this was used by wsgi is after a time out appach or uwsgi would kill the interperer following a request | |
| 14:45:13 | sean-k-mooney | and that would kill the heatbeat | |
| 14:45:28 | sean-k-mooney | and that woudl cause a timeout error on rabbitmq side | |
| 14:45:41 | sean-k-mooney | it didnt break anything just create false errors | |
| 14:46:03 | sean-k-mooney | ygk_12345: using a pthread when runing under apache mod_wsgi or uwsgi avoids that | |
| 14:46:23 | sean-k-mooney | nova-compute does nto run under either so its not needed same for the conductor and schduler | |
| 14:47:18 | sean-k-mooney | kashyap: not sure did you install git-review from pypi. you should never use the disto version of git-review | |
| 14:47:34 | kashyap | Never? It works just fine normally :) | |
| 14:47:42 | sean-k-mooney | its very very old normally | |
| 14:47:49 | sean-k-mooney | even on fedora | |
| 14:47:54 | kashyap | Yes, it's distro-based; and it's not related to git-review. It's related to the +ssh-rsa thingie setup | |
| 14:48:12 | sean-k-mooney | oh you have not updated your keys | |
| 14:48:38 | sean-k-mooney | gerrit still does not negociate sha2 and fedora has disable all sha based auth | |
| 14:48:53 | tobias-urdin | ygk_12345: what sean-k-mooney says :D | |
| 14:50:21 | sean-k-mooney | kashyap: https://src.fedoraproject.org/rpms/git-review git review is currently 2.3.1 | |
| 14:50:33 | sean-k-mooney | so even f37 is still behind | |
| 14:50:41 | sean-k-mooney | 2.3.1 came out in april | |
| 14:50:44 | kashyap | Yeah, saw that; I'm using 2.2.0.3 version | |
| 14:51:04 | sean-k-mooney | ya form last november | |
| 14:51:32 | kashyap | Right; but that's not the issue here for me. It's something else. I've aloso got the SSH setup: | |
| 14:51:35 | kashyap | Host review.openstack.org | |
| 14:51:37 | kashyap | HostName review.openstack.org | |
| 14:51:40 | kashyap | (Required at least on Fedora) | |
| 14:51:40 | kashyap | PubkeyAcceptedKeyTypes +ssh-rsa | |
| 14:51:51 | sean-k-mooney | yep unless you update your key | |
| 15:11:16 | gibi | I needed both | |
| 15:11:17 | gibi | PubkeyAcceptedAlgorithms +ssh-rsa | |
| 15:11:17 | gibi | HostkeyAlgorithms +ssh-rsa | |
| 15:11:30 | gibi | as new clients does not accept rsa as host key from old servers | |
| 15:12:18 | sean-k-mooney | gibi: ya it depens on the client and host i guess. | |
| 15:12:25 | sean-k-mooney | i dont have either | |
| 15:12:35 | sean-k-mooney | i just created a second key | |
| 15:13:05 | bauzas | reminder : nova meeting in 48 mins here | |
| 15:13:47 | sean-k-mooney | i have an id_ed25519 key and a less secure id_ecdsa as well as id_rsa now | |
| 15:14:30 | gibi | the ssh client stopped accepting rsa host keys since 8.8 | |
| 15:14:38 | gibi | *openssh-client | |
| 15:15:00 | gibi | but old servers still send the host key as rsa | |
| 15:15:56 | gibi | https://www.openssh.com/txt/release-8.8 | |
| 15:16:10 | sean-k-mooney | right but i think opendev have updated theirs | |
| 15:16:16 | sean-k-mooney | to have ec keys | |
| 15:17:51 | kashyap | Okay; /me uploads EC keys | |
| 15:17:59 | kashyap | gibi: Ah, good to know; lemme try | |
| 15:18:49 | sean-k-mooney | gibi: i think im using 8.8 but i guess i have not updated in a few days | |
| 15:19:24 | kashyap | gibi: W/o that HostkeyAlgorithms, I could SSH to `ssh kashyapc@review.opendev.org -p 29418` just fine | |
| 15:21:54 | sean-k-mooney | actully i think i have 8.7 | |
| 15:22:27 | sean-k-mooney | hum OpenSSH_8.8p1, OpenSSL 1.1.1o 3 May 2022 | |
| 15:22:50 | kashyap | gibi: No dice; for me `git review` just seem to be stuck in limbo trying to run the 'pre-review' hook (which in turn just runs `tox -e pep8`) | |
| 15:23:21 | gibi | ack | |