| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-05-19 | |||
| 13:16:48 | opendevreview | Danylo Vodopianov proposed openstack/nova-specs master: Add support for Napatech LinkVirt SmartNICs https://review.opendev.org/c/openstack/nova-specs/+/859290 | |
| 13:41:12 | opendevreview | Danylo Vodopianov proposed openstack/nova-specs master: Add support for Napatech LinkVirt SmartNICs https://review.opendev.org/c/openstack/nova-specs/+/859290 | |
| 13:43:09 | opendevreview | Danylo Vodopianov proposed openstack/nova-specs master: Add support for Napatech LinkVirt SmartNICs https://review.opendev.org/c/openstack/nova-specs/+/859290 | |
| 13:44:29 | opendevreview | Amit Uniyal proposed openstack/nova master: [WIP] add healthcheck manager to manager base https://review.opendev.org/c/openstack/nova/+/827844 | |
| 13:44:29 | opendevreview | Amit Uniyal proposed openstack/nova master: [WIP] add initial healthcheck support https://review.opendev.org/c/openstack/nova/+/825015 | |
| 13:44:30 | opendevreview | Amit Uniyal proposed openstack/nova master: [WIP] add healthcheck utils and constants https://review.opendev.org/c/openstack/nova/+/829469 | |
| 13:44:30 | opendevreview | Amit Uniyal proposed openstack/nova master: [WIP] add healthcheck tracker to nova context https://review.opendev.org/c/openstack/nova/+/829468 | |
| 13:44:31 | opendevreview | Amit Uniyal proposed openstack/nova master: add healthcheck endpoint to proxy commands https://review.opendev.org/c/openstack/nova/+/830703 | |
| 15:01:01 | dansmith | sean-k-mooney: this is the binding error I've been seeing quite a bit latey: https://a6fc37e91c861c55cf2e-59e8bddca242bc843b9f9be8c2ce73c4.ssl.cf2.rackcdn.com/879905/5/check/nova-live-migration/e3a0c46/testr_results.html | |
| 15:01:21 | dansmith | it says to check neutron logs, so maybe it's just a neutron thing, but does that look familiar at all? | |
| 15:01:48 | dansmith | actually I though it used to say something about os-vif in the message now that I think about it so maybe it's something different | |
| 15:09:00 | opendevreview | yatin proposed openstack/nova master: Add config option to configure TB cache size https://review.opendev.org/c/openstack/nova/+/868419 | |
| 15:10:59 | opendevreview | Artom Lifshitz proposed openstack/nova master: POC: Re-order and parallelize calls to Neutron and Cinder in post_live_migration https://review.opendev.org/c/openstack/nova/+/883678 | |
| 15:17:57 | dansmith | artom: the GIL has nothing to do with your threading there | |
| 15:18:18 | dansmith | what you mean, I think, is "as concurrently as the activities are green(ed)" | |
| 15:20:09 | sean-k-mooney | dansmith: only because you linked it to me like two weeks ago | |
| 15:20:24 | sean-k-mooney | ill tak a look at it again i didnt get around to it then | |
| 15:20:33 | dansmith | sean-k-mooney: thanks | |
| 15:32:22 | sean-k-mooney | Refusing to bind port e3308a61-39ff-4064-abb2-76de0d2139dc to dead agent: <neutron.plugins.ml2.drivers.ovn.agent.neutron_agent.ControllerAgent object at 0x7f6a7a6d2950> | |
| 15:32:45 | dansmith | does that mean the agent on the compute died or something? | |
| 15:33:42 | sean-k-mooney | i think that meanst the ovn metadtaa agent is dead since ovn its slef is agent less | |
| 15:34:04 | sean-k-mooney | but yes neutorn think the agent is dead on the destination host | |
| 15:34:13 | sean-k-mooney | it bound fine orgianly on the other host | |
| 15:34:49 | artom | dansmith, right, I meant it in the sense of GIL only allowing one execution thread at a time, so we're counting to eventlet to do its yield thing | |
| 15:35:02 | artom | I'll just remove it from the commit message :P | |
| 15:35:14 | sean-k-mooney | ill see if i can figure out why | |
| 15:48:11 | sean-k-mooney | dansmith: so it soudn like its hitting the code for https://github.com/openstack/neutron/commit/8a55f091925fd5e6742fb92783c524450843f5a0 | |
| 15:50:38 | sean-k-mooney | hum so at the time of the port bidning | |
| 15:51:30 | sean-k-mooney | there are no errro in the metadta aganet log but there are gaps for 3-6 seconds at a tiem and its interacting with both ovs and privsep | |
| 16:12:18 | opendevreview | Artom Lifshitz proposed openstack/nova master: POC: Re-order and parallelize calls to Neutron and Cinder in post_live_migration https://review.opendev.org/c/openstack/nova/+/883678 | |
| 16:12:19 | opendevreview | Artom Lifshitz proposed openstack/nova master: POC: Call Neutron immediately upon _post_live_migration() start https://review.opendev.org/c/openstack/nova/+/883682 | |
| 16:15:42 | sean-k-mooney | dansmith: so my best guess is its related to thsi change https://github.com/openstack/neutron/commit/628442aed7400251f12809a45605bd717f494c4e | |
| 16:16:16 | sean-k-mooney | 7 mounts ago they started trying to spread the agent heatbeats | |
| 16:16:42 | sean-k-mooney | im seeing logs to the effect fo delaying update to the cachs table for 10 seconds | |
| 16:17:02 | sean-k-mooney | around when the agent prior to the agent being detected as dead | |
| 16:17:24 | sean-k-mooney | my guess is if the agent is doign somthign like writing to the ovs db | |
| 16:17:36 | sean-k-mooney | it can miss the heatbeat | |
| 16:18:04 | sean-k-mooney | Delaying updating chassis table for 23 seconds {{(pid=38857) run /opt/stack/neutron/neutron/agent/ovn/metadata/agent.py:243}} | |
| 16:18:23 | sean-k-mooney | im seeign quite a spread | |
| 16:20:08 | dansmith | artom: ack I figured, probably better to make it accurate though yeah :) | |
| 16:20:23 | dansmith | sean-k-mooney: ah, interesting | |
| 16:20:37 | dansmith | sean-k-mooney: so like under heavy load they're missing some heartbeats maybe | |
| 16:20:53 | sean-k-mooney | ya perhaps | |
| 16:21:21 | sean-k-mooney | im goign to put up a tiny patch to change that form cfg.CONF.agent_down_time // 2 to cfg.CONF.agent_down_time // 3 | |
| 16:21:33 | sean-k-mooney | that will make it heat beat a little more often | |
| 16:22:02 | dansmith | ack cool | |
| 16:22:06 | sean-k-mooney | that was recently done for rabbit 2 -> 3 for similar reasons | |
| 16:26:50 | sean-k-mooney | oh its not merged yet https://review.opendev.org/c/openstack/oslo.messaging/+/875615 | |
| 16:32:16 | sean-k-mooney | dansmith: i assume there isnt a bug currently | |
| 16:32:27 | dansmith | sean-k-mooney: not that I've opened | |
| 16:32:49 | sean-k-mooney | ok ill file one quickly with some of the errors i was seeing | |
| 16:33:00 | sean-k-mooney | the logs are not super helpful | |
| 16:40:59 | dansmith | sweet thanks | |
| 16:44:56 | sean-k-mooney | https://bugs.launchpad.net/neutron/+bug/2020215 | |
| 16:45:15 | sean-k-mooney | i will push a patch once i run the unit/functional tests and see what breaks | |
| 16:59:50 | opendevreview | Artom Lifshitz proposed openstack/nova master: POC: Parallelize calls to Neutron and Cinder in post_live_migration https://review.opendev.org/c/openstack/nova/+/883678 | |
| 17:11:11 | sean-k-mooney | dansmith: i think https://review.opendev.org/c/openstack/neutron/+/883687 will help but its hard to tell if not then https://bugs.launchpad.net/neutron/+bug/2020215 might give the neutron folks another idea | |
| 17:11:27 | dansmith | ack thanks for chasing that | |
| 17:11:45 | sean-k-mooney | im going to finish there for today o/ | |
| 17:11:55 | dansmith | thanks, enjoy the weekend | |
| 17:25:40 | opendevreview | Artom Lifshitz proposed openstack/nova master: POC: Parallelize calls to Neutron and Cinder in post_live_migration https://review.opendev.org/c/openstack/nova/+/883678 | |
| 20:06:25 | opendevreview | Artom Lifshitz proposed openstack/nova master: POC: Parallelize calls to Neutron and Cinder in post_live_migration https://review.opendev.org/c/openstack/nova/+/883678 | |
| 23:01:58 | opendevreview | Artom Lifshitz proposed openstack/nova master: POC: Parallelize calls to Neutron and Cinder in post_live_migration https://review.opendev.org/c/openstack/nova/+/883678 | |
| #openstack-nova - 2023-05-20 | |||
| 00:36:55 | opendevreview | David Hill proposed openstack/nova stable/train: Allow an operator to override the default proto type of a VF https://review.opendev.org/c/openstack/nova/+/883732 | |
| 00:38:49 | opendevreview | David Hill proposed openstack/nova stable/train: Allow an operator to override the default proto type of a VF https://review.opendev.org/c/openstack/nova/+/883732 | |
| 01:54:40 | opendevreview | Artom Lifshitz proposed openstack/nova master: Call Neutron immediately upon _post_live_migration() start https://review.opendev.org/c/openstack/nova/+/883682 | |
| 01:54:41 | opendevreview | Artom Lifshitz proposed openstack/nova master: Parallelize calls to Neutron and Cinder in post_live_migration https://review.opendev.org/c/openstack/nova/+/883678 | |
| 21:11:02 | opendevreview | David Hill proposed openstack/nova stable/train: Allow an operator to override the default proto type of a VF https://review.opendev.org/c/openstack/nova/+/883732 | |
| 21:11:47 | opendevreview | David Hill proposed openstack/nova master: Allow an operator to override the default proto type of a VF https://review.opendev.org/c/openstack/nova/+/883745 | |
| 21:16:13 | opendevreview | David Hill proposed openstack/nova master: Allow an operator to override the default proto type of a VF https://review.opendev.org/c/openstack/nova/+/883745 | |
| #openstack-nova - 2023-05-22 | |||
| 07:27:12 | bauzas | good morning | |
| 08:05:55 | gibi | o/ | |
| 08:06:00 | ykarel | sean-k-mooney[m], gibi can you please check https://bugs.launchpad.net/neutron/+bug/2015065 comment 7/8 | |
| 08:07:38 | ykarel | randomly one of nova-api worker just get's stuck when doing requests to neutron(not sure if same is seen with any other service yet) | |
| 08:42:37 | gibi | ykarel: quickly looked at the bug. Thanks for collecting all that data. When the nova-api stuck in calling neutronclient's show_security_group do you see that the actualy API request to neutron was sent but never received by neutron-server? Or nova-api is stuck on sending the message? | |
| 08:49:24 | ykarel | gibi, i don't see the request received on neutron side, not sure where to check if it's stuck on sending | |
| 08:51:27 | gibi | ykarel: ack | |
| 09:03:20 | gibi | ykarel: | |
| 09:03:47 | gibi | i feel like we are seeing an interesting interaction between multiple things | |
| 09:04:15 | gibi | I'm trying to follow the stack trace from the latest comment from the bug to see where the neutronclient got stuck | |
| 09:04:33 | gibi | /usr/local/lib/python3.10/dist-packages/urllib3/util/connection.py:28 in is_connection_dropped | |
| 09:04:33 | gibi | the firts interesting point is | |
| 09:04:58 | gibi | https://github.com/urllib3/urllib3/blob/a5b29ac1025f9bb30f2c9b756f3b171389c2c039/src/urllib3/connectionpool.py#L272 | |
| 09:05:32 | gibi | so urllib try to check if the existing client connection is still usable or got disconnected | |
| 09:05:49 | gibi | https://github.com/urllib3/urllib3/blob/a5b29ac1025f9bb30f2c9b756f3b171389c2c039/src/urllib3/util/connection.py#L28 | |
| 09:05:54 | gibi | wait_for_read(sock, timeout=0.0) | |
| 09:06:06 | gibi | os it checks if it can read from the socket with 0.0 timeout | |
| 09:06:40 | gibi | https://github.com/urllib3/urllib3/blob/a5b29ac1025f9bb30f2c9b756f3b171389c2c039/src/urllib3/util/wait.py#L84-L85 | |
| 09:06:58 | gibi | that 0.0 timeout is passed to python's select.select | |
| 09:07:07 | gibi | https://docs.python.org/3.10/library/select.html#select.select | |
| 09:07:23 | gibi | "The optional timeout argument specifies a time-out as a floating point number in seconds. When the timeout argument is omitted the function blocks until at least one file descriptor is ready. A time-out value of zero specifies a poll and never blocks." | |
| 09:07:47 | gibi | so that select.select called with 0.0 should never block | |
| 09:07:52 | gibi | BUT | |
| 09:08:22 | gibi | in our env the envtlet monkey patching is changing python's select.select | |
| 09:08:25 | gibi | /usr/local/lib/python3.10/dist-packages/eventlet/green/select.py:80 in select | |
| 09:08:46 | gibi | and redirects it to implement the envtlet switching mechanism | |
| 09:11:16 | gibi | https://github.com/eventlet/eventlet/blob/88ec603404b2ed25c610dead75d4693c7b3e8072/eventlet/green/select.py#L30-L80C32 | |
| 09:12:34 | gibi | looking at that code it seems enventlet sets a timer with the timeout value | |
| 09:12:45 | gibi | via hub.schedule_call_global | |
| 09:17:05 | gibi | here I'm getting lost in the eventlet code but I assume sheduling a timer with 0.0 timeout in eventlet can be racy | |