| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-09-12 | |||
| 12:08:31 | sean-k-mooney | noonedeadpunk: if your using ml2/ovs you should also be enabling https://github.com/openstack/os-vif/blob/master/vif_plug_ovs/ovs.py#L96-L99 | |
| 12:08:57 | sean-k-mooney | ignore per_port_bridge | |
| 12:08:59 | sean-k-mooney | i should remove that | |
| 12:09:56 | noonedeadpunk | til about `isolate_vif` | |
| 12:10:43 | noonedeadpunk | and yes, it your note does make sense | |
| 12:30:43 | noonedeadpunk | sean-k-mooney: `[os_vif_ovs]/ovsdb_interface = vsctl` seems not to solve the issue. | |
| 12:32:32 | noonedeadpunk | no ovsdbapp logs though | |
| 12:33:08 | sean-k-mooney | ok then its likely not beauce of the agent looking up | |
| 12:33:33 | sean-k-mooney | you mentioned you have disabled the pthread for heatbeat yes | |
| 12:34:06 | sean-k-mooney | do you you iptbale firewall or openvswtich | |
| 12:34:10 | noonedeadpunk | yeah, I did, for all neutron-ovs-agents, nova-compute/scheduler/conductor | |
| 12:34:40 | noonedeadpunk | huh. | |
| 12:34:53 | sean-k-mooney | iptables we add the linux bridge and veth pair ovs firewall we add the tap directly to ovs | |
| 12:35:06 | noonedeadpunk | I just realized I likely missed disabling for neutron-server | |
| 12:35:20 | noonedeadpunk | as I assumed it's running in uwsgi, but likely it's not in this deployment | |
| 12:35:28 | sean-k-mooney | well neutron-server | |
| 12:35:36 | sean-k-mooney | ah i was going to sayy its proably wsgi | |
| 12:36:03 | sean-k-mooney | ya worth checking | |
| 12:36:13 | sean-k-mooney | that has been reverted to off by default | |
| 12:36:22 | sean-k-mooney | not sure if its been released yet on stable branches | |
| 12:36:23 | noonedeadpunk | nah, I disabled heartbeat_in_pthread for neutron-server as well | |
| 12:37:21 | noonedeadpunk | firewall_driver = iptables_hybrid | |
| 12:37:46 | noonedeadpunk | yeah, I guess there's an lxb in place | |
| 12:49:30 | sean-k-mooney | yep | |
| 12:49:59 | sean-k-mooney | ok on sec | |
| 12:51:25 | sean-k-mooney | https://github.com/openstack/nova/commit/0b0f40d1b308b29da537859b72080488560c23d4 | |
| 12:51:33 | noonedeadpunk | Tbh I still kind of blame pthread, as actual error is `oslo_messaging.rpc.server eventlet.timeout.Timeout: 300 seconds` which is exactly what would happen because of pthreads iirc | |
| 12:51:43 | noonedeadpunk | and it's intermittent as well | |
| 12:52:07 | noonedeadpunk | but only revert is affected which is weird given it's pthread | |
| 12:52:26 | sean-k-mooney | https://bugs.launchpad.net/nova/+bug/1952003 | |
| 12:52:50 | sean-k-mooney | so we when back and foth with this due to a few inflight change at once | |
| 12:52:58 | sean-k-mooney | but you are probaly hitting that | |
| 12:53:32 | sean-k-mooney | we shoudl not be waiting for the network vif plugged event if you have a specic set of patches. | |
| 12:54:37 | sean-k-mooney | noonedeadpunk: do you have https://github.com/openstack/nova/commit/66c7f00e1d9d7c0eebe46eb4b24b2b21f7413789 | |
| 12:56:01 | gibi | zigo: ack, let me know if I can help somehow | |
| 12:56:23 | sean-k-mooney | when we adressed https://bugs.launchpad.net/nova/+bug/1895220 it intoduced https://bugs.launchpad.net/nova/+bug/1952003 which orginially fixed https://bugs.launchpad.net/nova/+bug/1832028 and https://bugs.launchpad.net/nova/+bug/1833902 | |
| 12:57:49 | sean-k-mooney | noonedeadpunk: https://github.com/openstack/nova/commit/0b0f40d1b308b29da537859b72080488560c23d4 is in yoga | |
| 12:58:07 | noonedeadpunk | sean-k-mooney: I do have https://github.com/openstack/nova/commit/66c7f00e1d9d7c0eebe46eb4b24b2b21f7413789 | |
| 12:58:41 | noonedeadpunk | I guess I don't have I3cb39a9ec2c260f422b3c48122b9db512cdd799b though, as it's Xena | |
| 12:58:42 | sean-k-mooney | noonedeadpunk: what about https://review.opendev.org/c/openstack/nova/+/828414 | |
| 12:58:53 | sean-k-mooney | we backproted it | |
| 12:59:03 | sean-k-mooney | but only 5 months ago | |
| 12:59:16 | noonedeadpunk | Nah, we did not do this minor upgrade | |
| 12:59:21 | noonedeadpunk | Let me check it out | |
| 12:59:55 | sean-k-mooney | im not sure if we have done a release since then | |
| 13:00:24 | noonedeadpunk | Gerrit says you did :p | |
| 13:00:41 | noonedeadpunk | But we run 24.0.1.dev10, and it's included in 24.1.1 | |
| 13:00:43 | sean-k-mooney | no it say we merge it | |
| 13:00:54 | sean-k-mooney | where did you see that in gerrit | |
| 13:01:14 | noonedeadpunk | three dots in upper right corner -> included in | |
| 13:01:44 | sean-k-mooney | oh wow didnt know that was a thing | |
| 13:02:26 | sean-k-mooney | https://github.com/openstack/releases/commit/ac4be06827ec7a450233244d8c5cae8834b95ffc | |
| 13:02:29 | noonedeadpunk | it was there even in gerrit 2 | |
| 13:02:37 | sean-k-mooney | but yes we did that on 21st jun | |
| 13:03:01 | sean-k-mooney | never used it i normally just check the release in github | |
| 13:04:47 | sean-k-mooney | noonedeadpunk: so ya soory i think its https://bugs.launchpad.net/nova/+bug/1952003 | |
| 13:05:17 | sean-k-mooney | on the pluse side if it is then you just need to do the minor update when you have time too | |
| 13:18:28 | noonedeadpunk | ah, bug report is indeed super familiar | |
| 13:29:17 | zigo | After a lot of head-scratching, I can't get how to fix it (note: I already cherry-picked https://github.com/eventlet/eventlet/pull/754/commits/cd2532168e33d892de625f9fc831bf0951f4e937 the collections.abc.Iterable one, and another about ssl_version=ssl.PROTOCOL_TLSv1_2). | |
| 13:29:17 | zigo | https://paste.opendev.org/show/bIinQaPTAy81Uac3ZPHS/ | |
| 13:29:17 | zigo | gibi: Building Eventlet, I still get this: | |
| 13:29:58 | zigo | The send_method object contains what, in fact? | |
| 13:30:11 | zigo | I see it's self.fd.send_method or something ... | |
| 13:43:33 | gibi | I looked at it but I have no ideas either on that | |
| 14:14:18 | noonedeadpunk | sean-k-mooney: seems that patch revert does work, thanks a lot! | |
| 14:16:17 | sean-k-mooney | we had 3 or 4 supper niche edgcases that we resolve and unfortetly it took use a while to relase that that was nolonger required after we fixed that previous bug | |
| 14:16:34 | sean-k-mooney | noonedeadpunk: im glad this is working for you | |
| 14:34:56 | zigo | gibi: In oslo.concurrency, I tried reverting "Fix fair internal lock used from eventlet.spawn_n" and it was still stuck, so now I'm trying to revert "Prove that spawn_n with fair lock is broken" ... | |
| 14:35:35 | sean-k-mooney | zigo: just an an fyi that fix is needed | |
| 14:35:49 | sean-k-mooney | zigo: without it none of the fair logs in nova actully work | |
| 14:36:07 | zigo | You mean the "Fix fair internal lock used from eventlet.spawn_n" ? | |
| 14:36:14 | sean-k-mooney | yes | |
| 14:36:24 | sean-k-mooney | that is required for correctness | |
| 14:36:26 | zigo | Right, though it doesn't seem to be the brokenness ... | |
| 14:36:31 | sean-k-mooney | ack | |
| 14:39:43 | zigo | It passed ... | |
| 14:40:27 | zigo | Removing https://review.opendev.org/c/openstack/oslo.concurrency/+/855713 fixed my issue. | |
| 14:42:49 | gibi | as I noted earlier you are probably running those test ^^ without monkey patching hence the they stuck | |
| 14:43:20 | gibi | you have no better option now but removing those tests | |
| 14:43:38 | gibi | but you still have to keep the fix form "Fix fair internal lock used from eventlet.spawn_n"as sean-k-mooney noted | |
| 14:44:30 | zigo | I did. | |
| 14:45:09 | zigo | gibi: Is it possible that I'm running into this problem because I'm not doing `TEST_EVENTLET=0 lockutils-wrapper` before stestr run ? | |
| 14:45:23 | gibi | zigo: yes, I think so | |
| 14:45:29 | zigo | Ok, will try. | |
| 14:46:16 | zigo | Thanks. | |
| 14:57:13 | zigo | Indeed, that looks like fixing the issue, thanks gibi! :) | |
| 14:58:08 | gibi | as a follow up we need a better handling of those test in oslo.concurrency. The eventlet specific test should be skipped if TEST_EVENTLET is not requested | |
| 15:11:42 | zigo | I'd prefer if the whole unit test suite was failing with a TEST_EVENTLET is not set ... | |
| 15:11:56 | zigo | This way, a guy like me would know what to do... :) | |
| 15:12:02 | zigo | (just my 2 cents of advice...) | |
| 15:27:15 | opendevreview | Dmitriy Rabotyagov proposed openstack/nova master: [doc] Add os_vif configuration options https://review.opendev.org/c/openstack/nova/+/857202 | |
| 15:27:27 | gibi | zigo: yeah it is not helpful if the test just stuck | |
| 22:10:17 | JayF | So heads up, it looks like there might be some persistent failure in stable/yoga: https://review.opendev.org/c/openstack/nova/+/854257 the openstacksdk functional job has failed almost every time on this change (and it's clearly unrelated) | |
| 22:10:49 | JayF | I didn't see it mentioned on the etherpad so I figured I'd mention it here. I'm not too attached to that patch specifically anymore (all the things we know need backporting from Ironic driver have been) -- but it works well as a test case to see the failures. | |
| #openstack-nova - 2022-09-13 | |||
| 08:24:11 | Uggla | sean-k-mooney[m], can you please recall me the link to your devstack installation tool using ansible + molecule ? | |
| 09:39:30 | whoami-rajat | hi sean-k-mooney , can you take a look at the comment here? basically stephenfin wants to rename the parameter to reimage-boot-volume and add a negative parameter along with it https://review.opendev.org/c/openstack/python-openstackclient/+/831014/7/openstackclient/compute/v2/server.py#3095 | |
| 10:32:23 | zigo | I'm reading Nova release notes for Zed, and saw the thingy about hw:locked_memory. I wonder: what's the point of this feature? In which case one would want the memory to be locked, and to achieve what? Is this a security feature? | |
| 10:36:43 | sean-k-mooney | zigo: o/ | |
| 10:37:01 | zigo | :) | |
| 10:37:06 | opendevreview | Amit Uniyal proposed openstack/nova master: Adds check if blk_dev_info has correct falvor.swap https://review.opendev.org/c/openstack/nova/+/857339 | |