| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-10-12 | |||
| 08:44:48 | frickler | the issue isn't seen in our CI because we use pre-built wheels | |
| 08:45:18 | frickler | if you do a local build without old pip cache, it will fail the same way | |
| 08:46:08 | pslestang | frickler: you're completely right | |
| 08:47:50 | bauzas | dependency* even | |
| 08:47:50 | bauzas | pslestang: yeah, eventually spotted the depency issue with devstack by reinstalling my testbed | |
| 08:51:17 | pslestang | I will try by pining the setuptools to 58.0.0 as proposed https://bugs.launchpad.net/devstack/+bug/1946340/comments/5 | |
| 08:57:19 | bauzas | ergh, network glitch here | |
| 09:06:42 | bauzas | frickler: fwiw, cinder installation on devstack failed due to this | |
| 09:07:00 | bauzas | I can't see any proper workaround | |
| 09:07:33 | sean-k-mooney[m] | for now you just remove oslo.vmware from cinders requirements file | |
| 09:07:47 | bauzas | frickler: pslestang: about documenting it, we already discussed about the 2_to_3 issues we got and the pinned setuptools and we said "well, this isn't a nova issue, so why should we create some relnotes for this ?" | |
| 09:07:59 | sean-k-mooney[m] | ill try and submit a fix to oslo.vmware to use suds-community shortly | |
| 09:08:15 | bauzas | sean-k-mooney: locally, you mean ? ok | |
| 09:08:19 | sean-k-mooney[m] | this is not a devstack bug | |
| 09:08:23 | sean-k-mooney[m] | yes locally | |
| 09:08:29 | sean-k-mooney[m] | i just commented it out | |
| 09:09:01 | sean-k-mooney[m] | but the issue is in oslo.vmware it depens on suds-jaroko or something like that | |
| 09:09:11 | sean-k-mooney[m] | suds-community is python 3 compatible | |
| 09:09:31 | sean-k-mooney[m] | so once we fix there requirements file you can skip the local workaround | |
| 09:09:42 | bauzas | yes | |
| 09:15:29 | frickler | bauzas: yes, local workaround is to remove oslo.vmware from cinder requirements | |
| 09:15:37 | bauzas | kk | |
| 09:16:18 | frickler | sean-k-mooney[m]: there is already one, it needs to be added to reqs first, though: https://review.opendev.org/c/openstack/oslo.vmware/+/813377 | |
| 09:17:14 | frickler | sean-k-mooney[m]: also, the most recent version of suds-community still hasn't fixed the 2to3 issue. but they provide wheels on pypi, which masks that issue | |
| 09:17:25 | sean-k-mooney[m] | ah ok good | |
| 09:17:48 | frickler | they need a new tag with this one https://github.com/suds-community/suds/pull/58 | |
| 09:21:10 | frickler | ah, that'll be a major version bump, so the test release is suds-community-1.0.0b1 | |
| 09:21:50 | frickler | which I can install locally from source just fine, so that looks like progress | |
| 09:27:02 | mdbooth | dansmith: Liking DEVSTACK_PARALLEL: Speedup: 1.2226 :) | |
| 09:27:34 | sean-k-mooney[m] | it works pretty well | |
| 09:28:28 | mdbooth | It had a smaller benefit in a GCE vm, but this one on PSI seems to like it | |
| 09:28:53 | sean-k-mooney[m] | are you using the ci flavors or normal ones | |
| 09:29:01 | mdbooth | normal | |
| 09:29:08 | sean-k-mooney[m] | the normal ones are all backed by ceph and have lower iops | |
| 09:29:20 | sean-k-mooney[m] | so ya it helps more there | |
| 09:29:33 | sean-k-mooney[m] | the ci ones have local nvme storage and it benifits less | |
| 09:30:34 | sean-k-mooney[m] | the more io overhead there is the more parrallel helps | |
| 10:15:52 | opendevreview | Balazs Gibizer proposed openstack/nova master: Add a WA flag waiting for vif-plugged event during reboot https://review.opendev.org/c/openstack/nova/+/813419 | |
| 10:29:48 | sean-k-mooney | hum now that is interesting. | |
| 10:30:05 | sean-k-mooney | did people know that if you hit b in gerrit it loads git blam on the side | |
| 10:30:10 | sean-k-mooney | *blame | |
| 10:32:40 | gibi | that sounds usefull | |
| 13:29:07 | dansmith | mdbooth: sweet, I have a bunch of other parallel points to add, but I need to circle back and finish them | |
| 13:39:01 | opendevreview | norman shen proposed openstack/nova master: Recreate mdev devices according to placement https://review.opendev.org/c/openstack/nova/+/810220 | |
| 14:21:38 | bauzas | but I'll back around 1515UTC (45 mins before the meeting) | |
| 14:23:56 | gibi | kashyap: I read the driver part of https://review.opendev.org/c/openstack/nova/+/762330 and left comments. (I still not read the test parts). I don't feel this commit is ready. If feel this is patched together in a rush. | |
| 14:25:17 | opendevreview | Alexey Stupnikov proposed openstack/nova master: Rollback problematic port bindings on source host only https://review.opendev.org/c/openstack/nova/+/813659 | |
| 14:25:33 | pslestang | sean-k-mooney: FYI it seems like there is already someone patching oslo.vmware to use suds-community instead of suds-jurko https://review.opendev.org/c/openstack/oslo.vmware/+/813377 | |
| 14:25:45 | sean-k-mooney | yes | |
| 14:26:18 | sean-k-mooney | pslestang: frickler mentioned that above | |
| 14:27:11 | pslestang | sean-k-mooney: ouup's I missed it | |
| 14:31:59 | kashyap | gibi: Hi, I'll look and investigate. The submitter told me they even tested it in a real deployment (I took their word) | |
| 14:32:39 | gibi | kashyap: I can accept that it works but for me it is really hard to follow. maybe other in the core team has more knowledge to figure out what happens. | |
| 14:32:40 | kashyap | gibi: Thanks for the review time! We definitely don't want this rushed in. And needs careful integration testing. As it impacts live migration | |
| 14:33:03 | kashyap | gibi: No, if it's hard to follow for you, that's a reason enough to clean it up. And also it needs code comments too | |
| 14:33:04 | gibi | so I'm more concerned about understandabilty now than correctness | |
| 14:33:15 | gibi | ahh, ok | |
| 14:33:30 | kashyap | Yeah; I agree this needs more explnaations | |
| 14:40:27 | gibi | lyarwood: I looked into https://bugs.launchpad.net/nova/+bug/1946339 in short I don't see what happens and I cannot reproduce it locally while it is happening on the gate frequently. I'm a bit stuck | |
| 14:46:07 | sean-k-mooney | gibi: lyarwood is hopfule preparing for an operation later today and will be recovering for the next ~2 weeks | |
| 14:47:05 | gibi | sean-k-mooney: ack, I know. I just wanted to get back to him. (I update the bug with my finding) | |
| 14:47:07 | sean-k-mooney | ah test_unshelve_offloaded_server_with_qos_port_pci_update_fails | |
| 14:50:12 | sean-k-mooney | gibi: presumable this is some interaction between artoms fix for updating the pci device slot in the port profile on unshleve and minium band with based scheduling | |
| 14:50:30 | opendevreview | Alexey Stupnikov proposed openstack/nova master: Rollback problematic port bindings on source host only https://review.opendev.org/c/openstack/nova/+/813659 | |
| 14:50:32 | sean-k-mooney | gibi: have you tried ensuring that the pci device claimied as part of the unshelve is different | |
| 14:51:13 | sean-k-mooney | e.g. boot a vm, shelve it, boot another vm ensuring it claims the same pci device as the first then try unslevleing the first vm | |
| 14:52:22 | sean-k-mooney | oh this is in the func tests | |
| 14:52:29 | sean-k-mooney | not an end user bug | |
| 14:53:11 | gibi | nope | |
| 14:53:22 | gibi | and the first exception is part of the test | |
| 14:53:43 | sean-k-mooney | so it looks like somehow the db get torwn down too early? | |
| 14:53:50 | gibi | something like that | |
| 14:53:58 | gibi | but I don't get how can that be | |
| 14:54:18 | sean-k-mooney | that is presumable created by the fixture in setup | |
| 14:54:33 | sean-k-mooney | so ya that is hard to understand | |
| 14:56:07 | sean-k-mooney | gibi: this is not messing with any global state right https://github.com/openstack/nova/blob/a8d3ab2513c39aeac3393b2154988316dfa2db3a/nova/tests/functional/test_servers_resource_request.py#L2523-L2526 | |
| 14:56:21 | sean-k-mooney | gibi: we have our own isntance of placement per test function | |
| 14:56:25 | sean-k-mooney | that has its own db | |
| 14:57:02 | gibi | I assume we have placement per test case per test executor | |
| 14:57:12 | gibi | otherwise everything would be unstable | |
| 14:57:23 | sean-k-mooney | ya that is my assumtion too | |
| 14:57:36 | gibi | btw the sriov placement tree is created in the test case setup so that is per test case for sure | |
| 14:57:45 | gibi | and the uuids shoudl be unique | |
| 15:01:09 | sean-k-mooney | could this be failing due to the update_avaiable_resouce_provier periodic task? | |
| 15:01:59 | sean-k-mooney | if that ran it would raise the same error internally but it might now catch it | |
| 15:02:53 | gibi | there is no proof in the logs that the period runs https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_b15/713498/27/check/nova-tox-functional-py38/b1582a8/job-output.txt | |
| 15:03:15 | gibi | we have test cases where we trigger the periodic manually, so I think we even turn of the periodic for the func test | |
| 15:10:19 | gibi | I cannot reproduce it even if I delay the compute side of the execution locally, the wait for server status timeouts first not the conductor | |
| 15:10:36 | gibi | so I don't get how the conductor can time out in the gate | |
| 15:12:37 | gibi | I will try pulling out the test execution order from a failed gate run and see if that reproduce it locally or not.. | |
| 15:13:27 | sean-k-mooney | your referign to "oslo_messaging.exceptions.MessagingTimeout: No reply on topic conductor\n" | |
| 15:13:36 | gibi | yepp | |
| 15:14:03 | sean-k-mooney | this is also using the in memory driver so there is no networking issues that could be at play | |
| 15:14:35 | sean-k-mooney | with that said do you know what we set the time out too | |
| 15:14:54 | sean-k-mooney | we make spawn and other thing synconouse in the functional tests | |
| 15:15:17 | sean-k-mooney | i wonder if we only see this in a slow node | |
| 15:15:38 | sean-k-mooney | like is the time out 1 second of something tiny like that | |
| 15:15:43 | sean-k-mooney | when using the fake driver | |
| 15:15:51 | gibi | I don't know the timeout value in the funct test, but I put a sleep(10) in the compute side just before raising the expected exception, and it makes the test waiting for the server state time out instead of the fake driver | |
| 15:20:09 | sean-k-mooney | so i think this is where that messaging timeout gets raised https://github.com/openstack/oslo.messaging/blob/ca939fc0e4683efce87b567a9a074063a9c75b4f/oslo_messaging/_drivers/impl_fake.py#L207-L214 | |