| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-02-07 | |||
| 11:09:41 | gibi | originally (in 2021) this way I was able to reproduce https://bugs.launchpad.net/nova/+bug/1946339 but I tried this couple weeks ago again and was not able to reproduce the current occasion after couple hour of --unit-failure run | |
| 11:10:47 | sean-k-mooney | dvo-plv: the destination object is constucted here https://github.com/openstack/nova/blob/1c46c4e9e5ba4b84816f5cadad0674f3a773e739/nova/conductor/tasks/live_migrate.py#L64 | |
| 11:11:19 | sean-k-mooney | dvo-plv: but as i said this more complex approch is only relevent if we wanted to automatically enabel this functionality | |
| 11:12:31 | sean-k-mooney | well technially its created here https://github.com/openstack/nova/blob/1c46c4e9e5ba4b84816f5cadad0674f3a773e739/nova/conductor/manager.py#L470 | |
| 11:13:36 | sean-k-mooney | this only matters for the live migration case as the feature can be renegociated on cold migration or other move operations | |
| 11:14:18 | bauzas | nova.tests.functional.libvirt.test_vtpm.VTPMServersTest.test_create_server | |
| 11:14:18 | bauzas | (functional-py38) [sbauza@sbauza nova]$ stestr load /tmp/zuul-logs.Edb6Rp/testrepository.subunit --subunit | subunit-filter -F | subunit-ls | |
| 11:14:30 | bauzas | gibi: I'm able to get the failing test | |
| 11:14:57 | bauzas | so given I'm looking at all the fetched logsearch subunits, I could extrapolate a list of usual suspects | |
| 11:15:38 | dvo-plv | So if you think that this way is very complex and can make code not so easy and familiar, maybe we should better use existing approach ( creates a new filter like accelerators_filter and check if the user requested packed option) what already exists and is easy to scale. I already check this approach, this approach also forbids migrating VM to the host without packed ring support and also start VM on the host without packed ring support | |
| 11:16:51 | sean-k-mooney | dvo-plv: yep that is the simpelest approch. we could in a future release enable it by default and add a migration mechaniums too if desired. | |
| 11:17:19 | sean-k-mooney | either by turning it on once we raise our min QEMU/Libvirt versions to one that means it will alwasy be aviable | |
| 11:17:39 | sean-k-mooney | or buy automatically adding the image property if not provided | |
| 11:18:09 | sean-k-mooney | so taking the explict approch now does not prevent use making it implict in the future | |
| 11:18:35 | sean-k-mooney | making it automatic now front loads a bunch of complexity | |
| 11:21:54 | gibi | bauzas: I tried that too. I fetched multiple failed worker test case list and intersected them it resulted in an empty list. probably we have multiple test cases that leaks | |
| 11:22:20 | gibi | bauzas: but you can be lucky | |
| 11:23:54 | bauzas | gibi: I have the uuids from logseearch but I don't have the subunit streams | |
| 11:24:11 | bauzas | gibi: any way to pull them with logsearhc ? maybe using the --file param ? | |
| 11:24:28 | gibi | I pull them manually | |
| 11:24:47 | gibi | I needed only about 5 to see that there is no one common test case in the list | |
| 11:25:29 | bauzas | I can hack download.sh | |
| 11:27:54 | gibi | bauzas: I added | |
| 11:28:00 | gibi | https://bugs.launchpad.net/tempest/+bug/1999893 | |
| 11:28:03 | gibi | to the etherpad | |
| 11:37:14 | bauzas | gibi: I'm downloading the subunit file from each of the 159 failing runs | |
| 11:37:44 | gibi | that is maybe overkill as I said 5 example was enough for me to end up in an empty intersect | |
| 11:37:59 | bauzas | we will see | |
| 11:38:17 | bauzas | and then I'll ask stestr run to run 10 times each of the failing tests | |
| 11:43:35 | sahid_ | o/ quick question, regqrding instance.props, we do copy image metadata to the instance right? I don't remember | |
| 12:10:53 | gibi | bauzas: added another bug to the etherpad https://bugs.launchpad.net/nova/+bug/2006467 | |
| 12:11:38 | bauzas | ack | |
| 12:11:59 | bauzas | fwiw, looping over 78 different libvirt funct tests | |
| 12:12:17 | bauzas | first pass was saying OK | |
| 12:18:47 | dvo-plv | Sorry, Sean, but I do not get your final think. Do you prefer to use. Honestly, I would like to implement it as I have suggested, all interfaces what I need already exists and I will not extend other methods and classes with one parameter that will not use often. It will be a delicate way to extend the existing bunch of filters with one more filter. | |
| 12:50:28 | gibi | bauzas: added another https://bugs.launchpad.net/glance/+bug/2006473 | |
| 12:57:06 | sean-k-mooney | dvo-plv: i was suggeting using the extra specs/image properties for now | |
| 12:57:51 | sean-k-mooney | dvo-plv: and when we raise our min libvirt/qemu version eventually we can enable it by defualt then | |
| 12:58:39 | sean-k-mooney | dvo-plv: that would be my prefence. its simple to add to nova, easy to document and understand and easy to test | |
| 12:59:26 | sean-k-mooney | dvo-plv: we can evenutally turn this on by default when we nolonger supprot qemu/libvirt version that dont have this and there is nolonger an upgade impact | |
| 13:01:23 | sean-k-mooney | dvo-plv: we did the same thing with the virtio random number generator in the past | |
| 13:01:53 | sean-k-mooney | dvo-plv: intially it was opt in and we enabled it by default after a few release after we raised our min libvirt/qemu version | |
| 13:14:48 | bauzas | gibi: after one hour, still none of the 79 tests were having an issue | |
| 13:18:17 | opendevreview | Jorge San Emeterio proposed openstack/nova master: Moving privsep profiles to nova/__init__.py https://review.opendev.org/c/openstack/nova/+/872010 | |
| 13:18:27 | dvo-plv | Yes, I would like to have some general pre-approval from you here, before starting to implement and verify this functionality and present it in the blueprint to be sure that it will work okay, and does not waste your time on the blueprint spec file review process 1) User will have the ability to enable/disable this feature via flavor/image. 2) User will have the ability to set trait COMPUTE_NET_VIRTIO_PACKED to the flavor | |
| 13:19:10 | dvo-plv | Sorry, I have interrupt, I will resend my question | |
| 13:19:30 | dvo-plv | Yes, I would like to have some general pre-approval from you here, before starting to implement and verify this functionality and present it in the blueprint to be sure that it will work okay, and does not waste your time on the blueprint spec file review process | |
| 13:19:53 | dvo-plv | 1) User will have the ability to enable/disable this feature via flavor/image. 2) User will have the ability to set trait COMPUTE_NET_VIRTIO_PACKED to the flavor to choose some specific servers. Compute node will set this trait to the resource provider here static_trait. | |
| 13:20:11 | dvo-plv | 3) Scheduler will handle migration and OpenStack cluster update process with automatically understanding which node has this function with extended ALL_REQUEST_FILTERS array with a new filter similarly how it was implemented for accelerators_filter ( get a packed request from flavor ). | |
| 13:20:15 | sean-k-mooney | dvo-plv: yep so requesting the feature via flavor/image shoudl automatically result in the trait request via a pre_fiter like the acclerator filter | |
| 13:20:19 | dvo-plv | 4) As far as Qemu from v4.2 can not be compiled without packed ring support and Libvirt from v6.3, we can get if the current compute node can use this functionality and if it is available for the user. | |
| 13:20:25 | dvo-plv | OR do we need just implement options 1, 2, and 4 without the automatic scheduler handling this feature exists on the compute target node? | |
| 13:20:33 | sean-k-mooney | so they can ask for COMPUTE_NET_VIRTIO_PACKED explictly but it should not be required | |
| 13:21:26 | sean-k-mooney | 2 you get for free we already support arbitry trait request in the flavor/image | |
| 13:21:58 | sean-k-mooney | as part of implementeing 1 you should add a schduler prefilter to request COMPUTE_NET_VIRTIO_PACKED if the extra_spec/image property is set | |
| 13:22:23 | sean-k-mooney | so 1 and 3 are what you need to enable this feature properly | |
| 13:23:20 | sean-k-mooney | dvo-plv: https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L212-L222 | |
| 13:23:38 | sean-k-mooney | our current min libvirt is 6.0 and Qemu is 4.2 | |
| 13:24:58 | sean-k-mooney | we have not bumped that in a few releases so we will likely go to libvirt 7.0 and qemu 5.2 in the B release | |
| 13:25:32 | sean-k-mooney | although we can technially do that in the A release | |
| 13:25:49 | dvo-plv | Okay, I see, but it in the future, for now Libvirt support packed from 6.3, Should I just update minimum libvirt version, or create my own define for my trait? | |
| 13:25:52 | sean-k-mooney | bauzas: kashyap any reason not to do that in A | |
| 13:26:38 | sean-k-mooney | dvo-plv: we have a speicifc procedure for updating it where we have to annowuch our new min version in advacne for at least 1 cycle | |
| 13:27:00 | sean-k-mooney | we declared 7.0 and 5.2 as our next verion in Wallaby | |
| 13:27:31 | sean-k-mooney | so we could have done that bump some time ago | |
| 13:27:46 | bauzas | sean-k-mooney: we can if you want | |
| 13:27:50 | sean-k-mooney | although we now have new upgrade requirement to test the previous LTS | |
| 13:27:54 | sean-k-mooney | bauzas: i just realsied we cant | |
| 13:28:03 | sean-k-mooney | we need to support focal for A for upgrade reasons | |
| 13:28:16 | sean-k-mooney | bauzas: so we should do this in early B | |
| 13:29:00 | sean-k-mooney | we need to not have 20.04 in our greade job to do this bump | |
| 13:29:12 | bauzas | hmmm ok | |
| 13:29:29 | sean-k-mooney | and the dedicated focal job to go away | |
| 13:29:43 | sean-k-mooney | for B we will be useing 22.04 | |
| 13:30:47 | sean-k-mooney | dvo-plv: so what that means for you is if your patch is after we have done the bump you will not need to do the version check | |
| 13:31:01 | gibi | bauzas: I'm not surpirsed, it seems both of us are missing some hidden ingredients to reproduce the same thing that happens on the gate | |
| 13:31:06 | sean-k-mooney | if its before we do the bump you will ahve to do the version check when reportin the trait | |
| 13:31:32 | bauzas | - Unexpected Success: 0 | |
| 13:31:32 | bauzas | - Expected Fail: 0 | |
| 13:31:32 | bauzas | - Skipped: 0 | |
| 13:31:32 | bauzas | - Passed: 4144 | |
| 13:31:32 | bauzas | Ran: 4144 tests in 4758.5975 sec. | |
| 13:31:34 | bauzas | - Failed: 0 | |
| 13:31:36 | bauzas | Sum of execute time for each test: 32443.8935 sec. | |
| 13:31:38 | bauzas | :) | |
| 13:31:51 | sean-k-mooney | what kind of potato is that running on | |
| 13:32:20 | sean-k-mooney | or were you just running those in a loop | |
| 13:32:46 | bauzas | sean-k-mooney: see what we discussed before you arrived | |
| 13:33:08 | bauzas | gibi: yah, maybe | |
| 13:33:17 | dvo-plv | Okay, If Libvirt version will be lower that 6.3, when I will present patch in the blueprint, I will create separate define with Libvirt version | |
| 13:33:23 | bauzas | gibi: I'm now looking at the code and trying to understand what we use | |
| 13:33:48 | opendevreview | Jorge San Emeterio proposed openstack/nova master: Moving privsep profiles to nova/__init__.py https://review.opendev.org/c/openstack/nova/+/872010 | |
| 13:33:51 | kashyap | sean-k-mooney: Hi, reading back. (Was buried elsewhere in an urgent thing) | |
| 13:34:13 | sean-k-mooney | kashyap: its fine it was just on our next libvirt/qemu version | |
| 13:34:14 | kashyap | sean-k-mooney: Yeah, bumping the min versions in 'A' is totally fine. | |
| 13:34:25 | sean-k-mooney | kashyap: actully it used to be its not anymore | |
| 13:34:40 | sean-k-mooney | kashyap: form a pure nova point of view it would be | |
| 13:34:53 | sean-k-mooney | kashyap: but we have PTI/governance requirements | |
| 13:34:59 | kashyap | Right, I'm talking from a Nova PoV | |
| 13:35:05 | sean-k-mooney | that reqire use to supprot 20.04 for A | |