Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2021-10-12
09:07:47 bauzas frickler: pslestang: about documenting it, we already discussed about the 2_to_3 issues we got and the pinned setuptools and we said "well, this isn't a nova issue, so why should we create some relnotes for this ?"
09:07:59 sean-k-mooney[m] ill try and submit a fix to oslo.vmware to use suds-community shortly
09:08:15 bauzas sean-k-mooney: locally, you mean ? ok
09:08:19 sean-k-mooney[m] this is not a devstack bug
09:08:23 sean-k-mooney[m] yes locally
09:08:29 sean-k-mooney[m] i just commented it out
09:09:01 sean-k-mooney[m] but the issue is in oslo.vmware it depens on suds-jaroko or something like that
09:09:11 sean-k-mooney[m] suds-community is python 3 compatible
09:09:31 sean-k-mooney[m] so once we fix there requirements file you can skip the local workaround
09:09:42 bauzas yes
09:15:29 frickler bauzas: yes, local workaround is to remove oslo.vmware from cinder requirements
09:15:37 bauzas kk
09:16:18 frickler sean-k-mooney[m]: there is already one, it needs to be added to reqs first, though: https://review.opendev.org/c/openstack/oslo.vmware/+/813377
09:17:14 frickler sean-k-mooney[m]: also, the most recent version of suds-community still hasn't fixed the 2to3 issue. but they provide wheels on pypi, which masks that issue
09:17:25 sean-k-mooney[m] ah ok good
09:17:48 frickler they need a new tag with this one https://github.com/suds-community/suds/pull/58
09:21:10 frickler ah, that'll be a major version bump, so the test release is suds-community-1.0.0b1
09:21:50 frickler which I can install locally from source just fine, so that looks like progress
09:27:02 mdbooth dansmith: Liking DEVSTACK_PARALLEL: Speedup: 1.2226 :)
09:27:34 sean-k-mooney[m] it works pretty well
09:28:28 mdbooth It had a smaller benefit in a GCE vm, but this one on PSI seems to like it
09:28:53 sean-k-mooney[m] are you using the ci flavors or normal ones
09:29:01 mdbooth normal
09:29:08 sean-k-mooney[m] the normal ones are all backed by ceph and have lower iops
09:29:20 sean-k-mooney[m] so ya it helps more there
09:29:33 sean-k-mooney[m] the ci ones have local nvme storage and it benifits less
09:30:34 sean-k-mooney[m] the more io overhead there is the more parrallel helps
10:15:52 opendevreview Balazs Gibizer proposed openstack/nova master: Add a WA flag waiting for vif-plugged event during reboot https://review.opendev.org/c/openstack/nova/+/813419
10:29:48 sean-k-mooney hum now that is interesting.
10:30:05 sean-k-mooney did people know that if you hit b in gerrit it loads git blam on the side
10:30:10 sean-k-mooney *blame
10:32:40 gibi that sounds usefull
13:29:07 dansmith mdbooth: sweet, I have a bunch of other parallel points to add, but I need to circle back and finish them
13:39:01 opendevreview norman shen proposed openstack/nova master: Recreate mdev devices according to placement https://review.opendev.org/c/openstack/nova/+/810220
14:21:38 bauzas but I'll back around 1515UTC (45 mins before the meeting)
14:23:56 gibi kashyap: I read the driver part of https://review.opendev.org/c/openstack/nova/+/762330 and left comments. (I still not read the test parts). I don't feel this commit is ready. If feel this is patched together in a rush.
14:25:17 opendevreview Alexey Stupnikov proposed openstack/nova master: Rollback problematic port bindings on source host only https://review.opendev.org/c/openstack/nova/+/813659
14:25:33 pslestang sean-k-mooney: FYI it seems like there is already someone patching oslo.vmware to use suds-community instead of suds-jurko https://review.opendev.org/c/openstack/oslo.vmware/+/813377
14:25:45 sean-k-mooney yes
14:26:18 sean-k-mooney pslestang: frickler mentioned that above
14:27:11 pslestang sean-k-mooney: ouup's I missed it
14:31:59 kashyap gibi: Hi, I'll look and investigate. The submitter told me they even tested it in a real deployment (I took their word)
14:32:39 gibi kashyap: I can accept that it works but for me it is really hard to follow. maybe other in the core team has more knowledge to figure out what happens.
14:32:40 kashyap gibi: Thanks for the review time! We definitely don't want this rushed in. And needs careful integration testing. As it impacts live migration
14:33:03 kashyap gibi: No, if it's hard to follow for you, that's a reason enough to clean it up. And also it needs code comments too
14:33:04 gibi so I'm more concerned about understandabilty now than correctness
14:33:15 gibi ahh, ok
14:33:30 kashyap Yeah; I agree this needs more explnaations
14:40:27 gibi lyarwood: I looked into https://bugs.launchpad.net/nova/+bug/1946339 in short I don't see what happens and I cannot reproduce it locally while it is happening on the gate frequently. I'm a bit stuck
14:46:07 sean-k-mooney gibi: lyarwood is hopfule preparing for an operation later today and will be recovering for the next ~2 weeks
14:47:05 gibi sean-k-mooney: ack, I know. I just wanted to get back to him. (I update the bug with my finding)
14:47:07 sean-k-mooney ah test_unshelve_offloaded_server_with_qos_port_pci_update_fails
14:50:12 sean-k-mooney gibi: presumable this is some interaction between artoms fix for updating the pci device slot in the port profile on unshleve and minium band with based scheduling
14:50:30 opendevreview Alexey Stupnikov proposed openstack/nova master: Rollback problematic port bindings on source host only https://review.opendev.org/c/openstack/nova/+/813659
14:50:32 sean-k-mooney gibi: have you tried ensuring that the pci device claimied as part of the unshelve is different
14:51:13 sean-k-mooney e.g. boot a vm, shelve it, boot another vm ensuring it claims the same pci device as the first then try unslevleing the first vm
14:52:22 sean-k-mooney oh this is in the func tests
14:52:29 sean-k-mooney not an end user bug
14:53:11 gibi nope
14:53:22 gibi and the first exception is part of the test
14:53:43 sean-k-mooney so it looks like somehow the db get torwn down too early?
14:53:50 gibi something like that
14:53:58 gibi but I don't get how can that be
14:54:18 sean-k-mooney that is presumable created by the fixture in setup
14:54:33 sean-k-mooney so ya that is hard to understand
14:56:07 sean-k-mooney gibi: this is not messing with any global state right https://github.com/openstack/nova/blob/a8d3ab2513c39aeac3393b2154988316dfa2db3a/nova/tests/functional/test_servers_resource_request.py#L2523-L2526
14:56:21 sean-k-mooney gibi: we have our own isntance of placement per test function
14:56:25 sean-k-mooney that has its own db
14:57:02 gibi I assume we have placement per test case per test executor
14:57:12 gibi otherwise everything would be unstable
14:57:23 sean-k-mooney ya that is my assumtion too
14:57:36 gibi btw the sriov placement tree is created in the test case setup so that is per test case for sure
14:57:45 gibi and the uuids shoudl be unique
15:01:09 sean-k-mooney could this be failing due to the update_avaiable_resouce_provier periodic task?
15:01:59 sean-k-mooney if that ran it would raise the same error internally but it might now catch it
15:02:53 gibi there is no proof in the logs that the period runs https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_b15/713498/27/check/nova-tox-functional-py38/b1582a8/job-output.txt
15:03:15 gibi we have test cases where we trigger the periodic manually, so I think we even turn of the periodic for the func test
15:10:19 gibi I cannot reproduce it even if I delay the compute side of the execution locally, the wait for server status timeouts first not the conductor
15:10:36 gibi so I don't get how the conductor can time out in the gate
15:12:37 gibi I will try pulling out the test execution order from a failed gate run and see if that reproduce it locally or not..
15:13:27 sean-k-mooney your referign to "oslo_messaging.exceptions.MessagingTimeout: No reply on topic conductor\n"
15:13:36 gibi yepp
15:14:03 sean-k-mooney this is also using the in memory driver so there is no networking issues that could be at play
15:14:35 sean-k-mooney with that said do you know what we set the time out too
15:14:54 sean-k-mooney we make spawn and other thing synconouse in the functional tests
15:15:17 sean-k-mooney i wonder if we only see this in a slow node
15:15:38 sean-k-mooney like is the time out 1 second of something tiny like that
15:15:43 sean-k-mooney when using the fake driver
15:15:51 gibi I don't know the timeout value in the funct test, but I put a sleep(10) in the compute side just before raising the expected exception, and it makes the test waiting for the server state time out instead of the fake driver
15:20:09 sean-k-mooney so i think this is where that messaging timeout gets raised https://github.com/openstack/oslo.messaging/blob/ca939fc0e4683efce87b567a9a074063a9c75b4f/oslo_messaging/_drivers/impl_fake.py#L207-L214
15:21:02 sean-k-mooney which happend due to a _queue.Empty from return self.greenlet.switch()
15:22:03 sean-k-mooney the resource provider error on the compute was boubled up all the way to the rpc server
15:25:05 sean-k-mooney that first traceback is form here https://github.com/openstack/oslo.messaging/blob/ca939fc0e4683efce87b567a9a074063a9c75b4f/oslo_messaging/rpc/server.py#L178-L180
15:26:04 gibi but that line is visible in case of a successful run too
15:27:07 sean-k-mooney well after that message is printed we sent the failure to the conductor/api here https://github.com/openstack/oslo.messaging/blob/ca939fc0e4683efce87b567a9a074063a9c75b4f/oslo_messaging/rpc/server.py#L186
15:29:16 sean-k-mooney would that be recied by https://github.com/openstack/nova/blob/master/nova/conductor/api.py#L140-L141 ?
15:29:43 gibi this is the log from a successful run https://paste.opendev.org/show/809928/
15:29:58 sean-k-mooney no it would be in the conductor maager https://github.com/openstack/nova/blob/fdfdba265833d237e22676f9a223ab8ca0fe1e03/nova/conductor/manager.py#L941
15:30:48 sean-k-mooney gibi: right so in that case we see more output in the python loggin capture
15:31:13 sean-k-mooney where as ehre it stops and we get stderr

Earlier   Later