| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-03-10 | |||
| 11:38:00 | dmitriis | gibi: o/ Apologies for an extra ping, just wanted to ask if you're good with https://review.opendev.org/c/openstack/nova/+/829974 since you've reviewed it before. | |
| 11:38:04 | dmitriis | Not sure if it's appropriate to land it at this point or not but that's more of a fix + test change rather than a new feature. | |
| 11:38:38 | noonedeadpunk | So to start nova-compute we indeed had to echo uuid to /sys | |
| 11:39:06 | sean-k-mooney[m] | ya that is the workaround for now | |
| 11:39:15 | noonedeadpunk | but it's hard as you need to get resource from placement to understand mapping of uuid to mdev pci device | |
| 11:39:19 | sean-k-mooney[m] | i prefer recommendign that as its safer then mdevctl | |
| 11:39:43 | sean-k-mooney[m] | yes as i siad i tought someone had wirtten that up | |
| 11:40:02 | noonedeadpunk | yeah, I see | |
| 11:40:14 | sean-k-mooney[m] | i know bauzas started at one point | |
| 11:41:18 | sean-k-mooney[m] | i remember discussing the algrothim with them and how they would have to use placement to find the parent pci device to know whic device to create the mdev on since that is not in the xml | |
| 11:42:42 | noonedeadpunk | well another weird thing is that mdev uuid is just random thing, while tbh I'd expect it to be resource uuid. Which would make things easier... | |
| 11:44:03 | noonedeadpunk | As we have resource uuid and then mdev uuid is just another thing that is stored _only_ in xml | |
| 11:44:22 | noonedeadpunk | (in case I'm not missing anything | |
| 11:44:56 | sean-k-mooney[m] | its randome because we do not track them in the db | |
| 11:45:08 | gibi | dmitriis, sean-k-mooney[m], bauzas: I'm generally OK with https://review.opendev.org/c/openstack/nova/+/829974 I do noted that this patch is now wide the gap we need to close later as part of https://bugs.launchpad.net/nova/+bug/1961587 as it adds more sysfs calls to our neutron code. | |
| 11:45:10 | sean-k-mooney[m] | i have been trying to change that since before we actully added the mdev feature | |
| 11:46:41 | sean-k-mooney[m] | noonedeadpunk: what i would like use to evolve to is have operators precreat the mdevs and we would track them in the resouces table in the db | |
| 11:46:52 | sean-k-mooney[m] | then claim and allocate them to instances | |
| 11:47:04 | sean-k-mooney[m] | or tack them in the pcidevice table | |
| 11:47:33 | sean-k-mooney[m] | part of the reason i want to do that is i want to get rid of the persitnt libvirt domain xml | |
| 11:47:59 | sean-k-mooney[m] | right now vgpus are the only thing that need to the persitent domain xml | |
| 11:48:40 | dmitriis | gibi: ack, I am going to look at changing this to use extra info instead in a follow-up | |
| 11:48:41 | noonedeadpunk | well, I wasn't having much fun with plain qemu without libvirt, as if you don't need config you can jsut operate qemu directly? | |
| 11:50:46 | noonedeadpunk | tbh for me from operator prespective is preferable that nova manage mdev creation. as otherwise there will be tons of nasty hooks which everybody do. This can be handled by deployment tools ofc, but considering devices are not persistant and drop on reboot... dunno. | |
| 11:51:28 | noonedeadpunk | and if uuid for mdev will be taken as placement resource id - wouldn't it solve issue with persistant libvirt config? | |
| 11:52:08 | noonedeadpunk | as at time mdev is created, I guess allocation should be already claimed and resource provided where to create it? | |
| 11:52:20 | gibi | dmitriis: ack, thanks | |
| 11:57:50 | sean-k-mooney[m] | noonedeadpunk: you can have multiple mdevs attached to a vm so we cant use placemnt ids for the uuid | |
| 11:58:00 | sean-k-mooney[m] | also they have to be gloally unique | |
| 11:58:37 | noonedeadpunk | ah, indeed, if it's not Nvidia Ampere that would be the case... | |
| 11:58:52 | noonedeadpunk | (as there you have resource per vGPU) | |
| 11:58:58 | sean-k-mooney[m] | removing the persitent domain is not the same as raw qemu | |
| 11:59:14 | sean-k-mooney[m] | libvirt has 2 domains for each insthace | |
| 11:59:24 | sean-k-mooney[m] | the running one in memory and an xml file on disk | |
| 11:59:44 | sean-k-mooney[m] | on ocation they get out of sync and cause issues in managing the vm | |
| 12:00:08 | sean-k-mooney[m] | so i would like to get rid of the file on disk and just use the in memory one which is the acurrate one | |
| 12:00:27 | noonedeadpunk | yeah, I do agree herem that makes sense | |
| 12:01:10 | noonedeadpunk | just sugested that it could be first step to raw qemu, as had to deal with such solution on some cloud platform | |
| 12:01:22 | sean-k-mooney[m] | if you have a patch or code to recreate the mdevs on boot using placement we would be happy to include it | |
| 12:02:00 | sean-k-mooney[m] | are you in favor of raw qemu or against? | |
| 12:02:08 | noonedeadpunk | I'm not at the moment as just found that out during incident, but will try to allocate some resources | |
| 12:02:14 | noonedeadpunk | I'm personally against :) | |
| 12:02:33 | sean-k-mooney[m] | ack just checkign libvirt does some useful stuff for us | |
| 12:02:42 | sean-k-mooney[m] | so we are not currently plannig to remove it | |
| 12:02:47 | noonedeadpunk | s/just suggested/just assumed/ | |
| 12:03:17 | sean-k-mooney[m] | we dont really want to have to manage cgroups directly for example | |
| 12:03:50 | noonedeadpunk | I guess it prings in selinux/apparmour rules as well | |
| 12:03:55 | noonedeadpunk | *brings | |
| 12:03:59 | sean-k-mooney[m] | yes | |
| 12:04:28 | sean-k-mooney[m] | it deals with selinux lables and a bunch of other things which we dont want to have to implement in nova | |
| 12:04:34 | noonedeadpunk | and some live migration features iirc | |
| 12:04:49 | sean-k-mooney[m] | yes libvirt does the cpu compatiblity check for us | |
| 12:05:13 | sean-k-mooney[m] | so removing the xml on disk simple. removign libvirt is basicaly a full rewrite of the dirver | |
| 12:05:26 | sean-k-mooney[m] | which is not something we have time for or really want to do | |
| 12:05:57 | noonedeadpunk | exeprience with raw qemu was quite painful | |
| 12:06:34 | sean-k-mooney[m] | the only real advantage to raw qemu would be if libvirt did not have support for feature x yet | |
| 12:06:55 | sean-k-mooney[m] | and at this point nova does not move fast enough for that to be an issue | |
| 12:07:10 | noonedeadpunk | just wanted to say the same:) | |
| 12:07:55 | noonedeadpunk | virtiofs great example of feature ppl love to get but nobody has capacity to help on with... | |
| 12:08:40 | sean-k-mooney[m] | actully that is on our radar | |
| 12:08:55 | sean-k-mooney[m] | it should hopefully get added in z | |
| 12:09:03 | sean-k-mooney[m] | for manilla shares | |
| 12:09:15 | sean-k-mooney[m] | but if you have other usecases for it let us know | |
| 12:10:03 | sean-k-mooney[m] | once we have the basic support in the libvirt driver we can extend it simply if cyborg or anything else wants to use it | |
| 12:10:30 | sean-k-mooney[m] | brb | |
| 12:12:41 | gibi | bauzas and others: I think we have a functional test instability https://paste.opendev.org/show/btHI7ErFfhKYFGdfoujl/ | |
| 12:12:50 | gibi | e.g.: https://6779a19d629122a2ad74-09e4be48fe62aca6e4b03d954e19defe.ssl.cf1.rackcdn.com/828570/8/check/nova-tox-functional-py38/99a1e4f/testr_results.html | |
| 12:13:33 | gibi | recently I saw this pretty frequently but It happened in the past too (I saw matches from mid february) | |
| 12:14:25 | gibi | Im filing a bug... | |
| 12:20:19 | gibi | https://bugs.launchpad.net/nova/+bug/1964472 | |
| 12:33:42 | gibi | ohh the failing testcase was added here https://review.opendev.org/c/openstack/nova/+/821840 | |
| 12:33:56 | gibi | this is one of the bug with rc1 tag, where the fix is still open | |
| 12:34:14 | gibi | so I guess the regression test added here is not stable and needs some tuning | |
| 12:35:24 | gibi | bauzas, sean-k-mooney[m] should we pull https://review.opendev.org/c/openstack/nova/+/815324/10 from the gate? | |
| 12:35:36 | gibi | or we expect a followup to stabilize the functional test in that? | |
| 12:41:22 | sean-k-mooney | is it unstable | |
| 12:41:44 | sean-k-mooney | oh the regerssion | |
| 12:42:24 | gibi | the test case is unstable | |
| 12:42:27 | sean-k-mooney | i see is it still unsable with the followup | |
| 12:43:07 | gibi | we only have data about it being unstable without the fix for the origina bug | |
| 12:43:14 | gibi | that bugfix is on the gate | |
| 12:43:32 | gibi | I tried to reproduce it locally but failed so far | |
| 12:43:40 | sean-k-mooney | In the last 7 days from 8 failed functional test run 6 was due to this in nova. But this happened as far back as 14th of February. | |
| 12:43:53 | sean-k-mooney | so the test has only merged yesterday | |
| 12:44:05 | sean-k-mooney | so those other failures would have been on the path itself | |
| 12:44:29 | sean-k-mooney | as in https://review.opendev.org/c/openstack/nova/+/821840/5 merged yesterday | |
| 12:44:43 | gibi | the test case merged on the 5th | |
| 12:44:56 | gibi | oh no | |
| 12:44:58 | gibi | it is 9th | |
| 12:45:01 | gibi | you are correct | |
| 12:45:02 | gibi | sorry | |
| 12:45:30 | gibi | and you are correct before yesterday it is only failed on 821840/5 | |
| 12:45:38 | gibi | after the merge it is failed on 3 other pathes | |
| 12:45:42 | gibi | see https://paste.opendev.org/show/btHI7ErFfhKYFGdfoujl/ | |
| 12:46:16 | sean-k-mooney | then ya we shoudl either revert or disable i guess | |
| 12:46:35 | opendevreview | Merged openstack/nova master: Add functional tests to reproduce bug #1960412 https://review.opendev.org/c/openstack/nova/+/830010 | |
| 12:47:39 | gibi | yeah, no point yanking the bugfix from the gate as that does not break anyithing, the patch that added the test case was already merged | |
| 12:48:07 | gibi | I can push a patch skipping the test case | |
| 12:52:32 | opendevreview | Balazs Gibizer proposed openstack/nova master: Skip TestRollbackWithHWOffloadedOVS.test_rollback_pre_live_migration https://review.opendev.org/c/openstack/nova/+/833079 | |
| 12:52:38 | gibi | bauzas, sean-k-mooney: ^^ | |