| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-03-07 | |||
| 11:16:16 | sean-k-mooney | ignaziocassano_: nova recommends v4.0 as a minium preferably 4.2 | |
| 11:17:17 | ignaziocassano_ | sean-k-mooney: I do not knkow why, but the controlles mount cinder with version 4.0 while the compute nodes are using nfs vers 3 | |
| 11:17:57 | sean-k-mooney | we know that v3 has some issue with lockign that might affect data integrety | |
| 11:18:11 | sean-k-mooney | im really not sure how mixing would affect things | |
| 11:18:33 | ignaziocassano_ | ok, so I must specify nfs_mount_options in nova.conf on compute nodes | |
| 11:18:40 | sean-k-mooney | it slikely not adviasble but i suspect that this is outside the scope of nova. | |
| 11:19:12 | ignaziocassano_ | I will try it | |
| 11:19:27 | kashyap | Oh, yeah - the NFS version also plays a role. Indeed we recommend a minimum of NFS > 4.2 | |
| 11:19:29 | sean-k-mooney | https://docs.openstack.org/nova/latest/configuration/config.html#libvirt.nfs_mount_options | |
| 11:19:38 | sean-k-mooney | likely can help | |
| 11:19:44 | sean-k-mooney | but i am not that familar with it | |
| 11:19:58 | sean-k-mooney | i know downstream we have some recommended optiosn for that related to selinux | |
| 11:20:12 | sean-k-mooney | beyond that i have never really looked at what we suggest setting there | |
| 11:24:42 | ignaziocassano_ | thanks for your help | |
| 14:49:59 | dmitriis | sean-k-mooney, gibi: zuul seems to be happy about https://review.opendev.org/c/openstack/nova/+/829974 | |
| 15:01:37 | sean-k-mooney | ah thanks for the reminder | |
| 15:10:26 | opendevreview | Alexey Stupnikov proposed openstack/nova master: Add functional tests to reproduce bug #1960412 https://review.opendev.org/c/openstack/nova/+/830010 | |
| 15:36:12 | sean-k-mooney | gibi: can you spot check my expectations, when we shelve an instance would you expect use to unbinding the ports form the host | |
| 15:36:29 | sean-k-mooney | or well when we shelve offload | |
| 15:39:31 | gibi | sean-k-mooney: I think when we offload we should unbind from the host, otherwise the physical resource (i.e. pci device) is not freed. Or can we free up a PCI device without unbinding the port? | |
| 15:39:52 | sean-k-mooney | ack that is what i woudl expect too but we dont | |
| 15:40:51 | sean-k-mooney | the port shoudl still be atached to the vm but it shoudl not be bound to a host or an ml2 driver once its offloaded | |
| 15:41:25 | gibi | so we shoudl keep device_id in the port. Is that enough to keep the port reserved in neutron? | |
| 15:41:51 | sean-k-mooney | yes the device_id is enough to track ownwership | |
| 15:42:05 | sean-k-mooney | but binding:host_id shoudl be set to '' | |
| 15:42:53 | sean-k-mooney | im trying to create functional test for vdpa move operations while if figure out if i can get a 2 node deployment to test/develop move ops | |
| 15:43:09 | sean-k-mooney | but when i was writign the shelve test i noticed it was not cleared | |
| 15:43:28 | sean-k-mooney | so i dont know of this cause any bug but its not what i was expecting | |
| 15:44:04 | sean-k-mooney | gibi: we are relying on driver.cleanup to tear down the networking on the host | |
| 15:44:24 | gibi | do we free the compute claim? | |
| 15:44:24 | sean-k-mooney | which it does and that also disconnects the volumes | |
| 15:44:35 | sean-k-mooney | but we dont actully unbind the ports | |
| 15:45:07 | sean-k-mooney | its a good question i would have to look but i think so | |
| 15:45:14 | sean-k-mooney | but im not certin now | |
| 15:46:18 | gibi | if we free the claim but does not unbind the port then I think we have no resource problems it is just ugly / misleading that we keep the binding:host_id in neutron | |
| 15:46:56 | sean-k-mooney | ya | |
| 15:47:05 | sean-k-mooney | that would happen in the compute manager i guess | |
| 15:47:31 | sean-k-mooney | we call self.rt.delete_allocation_for_shelve_offloaded_instance | |
| 15:47:46 | sean-k-mooney | i would assuem that would free them | |
| 15:48:59 | sean-k-mooney | no... that is just clearing the placment allocation | |
| 15:49:14 | sean-k-mooney | we do update the resouce tracker after that however | |
| 15:58:12 | sean-k-mooney | my func test say we do not free testtools.matchers._impl.MismatchError: 4 != 3 | |
| 15:58:39 | opendevreview | Alexey Stupnikov proposed openstack/nova master: Clean up when queued live migration aborted https://review.opendev.org/c/openstack/nova/+/828570 | |
| 16:05:35 | sean-k-mooney | gibi: i might try and reporduce this in devstack and see if the same is true in realitiy | |
| 16:06:13 | gibi | maybe the periodic update_available_resources task frees it | |
| 16:06:37 | sean-k-mooney | i ran that in the func test with run_perodics | |
| 16:06:42 | sean-k-mooney | i tought it would | |
| 16:06:52 | sean-k-mooney | but apperently not | |
| 16:08:33 | sean-k-mooney | its not inconciveable this is a func test issue but its worth figureing out | |
| 16:09:33 | dansmith | what is the nova-emulation job? | |
| 16:09:48 | sean-k-mooney | it test emulating arm vms | |
| 16:10:00 | sean-k-mooney | so x86 host bootign arm vms | |
| 16:10:03 | dansmith | is it supposed to be stable? | |
| 16:10:19 | dansmith | seems like a ton of rechecking going on these days, and nova jobs have gotten pretty fat | |
| 16:10:50 | sean-k-mooney | am i dont know if that is stable | |
| 16:10:56 | sean-k-mooney | we just enabled it a few days ago | |
| 16:11:20 | dansmith | it's voting, | |
| 16:11:21 | sean-k-mooney | i did not think it was failing but it can certenly be set non-voting or moved to periodic | |
| 16:11:26 | dansmith | and I just saw a kernel panic on it | |
| 16:11:33 | dansmith | I think it's a guest kernel | |
| 16:12:00 | sean-k-mooney | https://zuul.openstack.org/builds?job_name=nova-emulation&skip=0 | |
| 16:12:06 | sean-k-mooney | it looks kind of ok | |
| 16:12:14 | sean-k-mooney | i think that is the first failure since it was merged | |
| 16:12:16 | dansmith | https://zuul.opendev.org/t/openstack/build/cb1314bff0f34bfdbb3a4f1fd5547b72 | |
| 16:12:39 | dansmith | okay, well, regardless, nova jobs are looking pretty heavy | |
| 16:12:55 | dansmith | I dunno how widely-known it is, but we're losing 30% of our CI capacity at the end of the month | |
| 16:13:43 | dansmith | so we'll probably need to be making some cuts | |
| 16:13:51 | dansmith | what's the major benefit of testing arm-on-x86? | |
| 16:14:30 | sean-k-mooney | its a proxy for ensureing that the new emulation featur works in general | |
| 16:14:40 | sean-k-mooney | it could be a weekly job | |
| 16:14:51 | sean-k-mooney | or run only on libvirt changes | |
| 16:14:55 | dansmith | the thing that lets us choose the guest emulation mode you mean? | |
| 16:15:38 | dansmith | couldn't it be a single test? if we have an arm image available, couldn't we just boot one instance from it and make sure it's alive, instead of a whole other job? | |
| 16:16:04 | sean-k-mooney | well we ant to ensure resize ectra works | |
| 16:16:29 | sean-k-mooney | we could proably do it as a post action or something more light weight | |
| 16:16:35 | dansmith | sure, so one scenario test that boots, resize, snapshot, etc | |
| 16:16:43 | sean-k-mooney | ya we coudl do that | |
| 16:16:56 | dansmith | just saying, it seems pretty expensive for a minor verification | |
| 16:17:14 | sean-k-mooney | well the idea was to test all feature with emulation | |
| 16:17:49 | sean-k-mooney | but we can 1 move it to weakly and 2 make it a set of senario tests | |
| 16:17:57 | sean-k-mooney | chateaulav:^ | |
| 16:18:01 | dansmith | yeah, ideally we'd run every configuration on every patch, but.. | |
| 16:20:05 | chateaulav | sean-k-mooney: would the scenario tests need added to the tempest project? | |
| 16:20:15 | sean-k-mooney | ya | |
| 16:20:19 | sean-k-mooney | well or as a plugin | |
| 16:20:35 | sean-k-mooney | but upstream tempest i think would be ok | |
| 16:22:45 | chateaulav | yeah, i noticed it took some time for the ci itself to run. so then we want to pursue a new tempest scenario test that can be added into another ci? | |
| 16:23:05 | chateaulav | then pause the nova emulation, or run it not as frequently? | |
| 16:23:49 | dansmith | if there's a high likelihood of it being broken, then a tempest test to check that on each patch would be good | |
| 16:23:59 | dansmith | however, if it's not very likely, then a weekly periodic test would be better and easier | |
| 16:24:02 | dansmith | I suspect the latter | |
| 16:26:06 | chateaulav | yeah. i think long term, maybe next cycle add in the tempest scenario that we can leverage. I think the weekly periodic would be good for the interim though. | |
| 16:26:17 | chateaulav | your thoughts sean-k-mooney | |
| 16:27:11 | dansmith | was anything not working when we first tried to do this? | |
| 16:27:24 | opendevreview | Alexey Stupnikov proposed openstack/nova master: Clean up when queued live migration aborted https://review.opendev.org/c/openstack/nova/+/828570 | |
| 16:29:13 | chateaulav | what do you mena in regards to not working? | |
| 16:30:31 | dansmith | chateaulav: you added the ability to select the guest emulation mode right? when you added that, were other things broken that made that non-trivial? | |
| 16:30:52 | dansmith | or, how invasive was the change? it thought it was mostly just a flag | |
| 16:33:04 | chateaulav | yeah, so the main item is the meta property that lets you define the guest architecture | |
| 16:33:28 | chateaulav | everything else was mods to the various checks to account for reading that value along with the host arch | |