| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-05-11 | |||
| 18:30:00 | mnaser | i will start with: this is a terrible idea | |
| 18:30:41 | mnaser | but somehow, i've got a weird requirement where a vm needs have a mapping of the pci address on the host device <=> pci address on the vm | |
| 18:31:17 | mnaser | i know pci_devices in the db contains what is assigned to what, but i dont think it contains what is the guest addres.. and i think we're relying on libvirt to maintain that sort of thing, right? | |
| 18:32:56 | dansmith | I'm not sure, but I'm curious what kind of application has this requirement :) | |
| 18:34:12 | dansmith | and to be clear, you mean the guest needs to know what the host's address of a pci device is (since it clearly knows its own mapping)? | |
| 18:34:14 | mnaser | dansmith: accelerator cards that need to be setup in a specific topology, and it seems that the the vm needs to know which card is which | |
| 18:34:54 | mnaser | yep | |
| 18:35:03 | dansmith | meaning I got two gpus and I need to know which one is "first" on the host? | |
| 18:35:24 | mnaser | Yes, something like that | |
| 18:35:51 | dansmith | I think exposing the actual host address is a bad idea, but maybe the ordering wouldn't be so bad (although really odd) | |
| 18:36:16 | mnaser | I think it’s a bad idea too but it seems that the cards somehow communicate | |
| 18:36:29 | mnaser | So as part of their topology setup, they need to know where the other one is | |
| 18:36:42 | dansmith | ...which seems like asking for a hardware-assisted vulnerability, but whatevs | |
| 18:36:53 | mnaser | You’re not wrong :/ | |
| 18:37:48 | mnaser | dansmith: we’ll from what I understand they just want to know which card is attached from the system .. like is it card 1 or card 2 or card 5 | |
| 18:37:53 | dansmith | how is it that crazy composable hardware people want to make boxes that can re-assign dimms between logical systems, but accelerator people can't build things that don't require knowing the address of the other one? | |
| 18:37:57 | dansmith | nevermind, don't answer that | |
| 18:38:19 | mnaser | And since the pci address is the “stable” way of knowing which one is which | |
| 18:38:45 | dansmith | mnaser: right, so what I was saying is knowing "this is the 3rd card of type 8086:abcd in the host system" would be enough? | |
| 18:39:35 | mnaser | dansmith: I believe so yes | |
| 18:39:53 | dansmith | that'd be the only thing I could imagine reasonably supporting | |
| 18:40:06 | dansmith | and off by default because.. seriously. | |
| 18:40:22 | mnaser | oh yes I don’t even expect this to be in nova at all cause I think this is silly. | |
| 18:40:33 | dansmith | even better :) | |
| 18:41:05 | dansmith | sorting the devices we attach to the guest so that they're in the same logical order as on the host might not be too bad either I guess | |
| 18:41:09 | mnaser | I was mainly wondering if there’s somewhere I can scrape this info and plumb into vendor metadata | |
| 18:41:15 | dansmith | like "if you're going to get two of these, make sure they're in the same order" | |
| 18:41:30 | mnaser | Yeah it seems at random right now | |
| 18:41:46 | dansmith | well, once is the guest is running I expect you can see it in the libvirt xml | |
| 18:42:13 | mnaser | Yeah so the idea was that using vendor metadata 2 (with the proxy) and then doing terrible things to pull that info in | |
| 18:43:29 | dansmith | yeah I like that because the "terrible things" are yours and you get to own them :P | |
| 18:46:01 | melwitt | mnaser: this sounds like device tagging, will that not work for this? | |
| 18:46:37 | mnaser | dansmith: I recognize the pain of maintaining said terrible things | |
| 18:46:44 | mnaser | melwitt: lemme see | |
| 18:46:56 | mnaser | (Never heard/used) | |
| 18:47:04 | melwitt | trying to find a doc | |
| 18:47:28 | melwitt | https://specs.openstack.org/openstack/nova-specs/specs/mitaka/approved/virt-device-role-tagging.html | |
| 18:47:56 | dansmith | melwitt: device tagging lets you tag the thing you requested, but since you don't control which one you got of two identical types, I don't think it will | |
| 18:48:17 | dansmith | tagging might make it easier to line them up and expose them in metadata though | |
| 18:49:47 | mnaser | melwitt: yeah it looks like it doesn’t talk much about tagging pci pass through as well | |
| 18:50:18 | dansmith | I think you can tag pci now, but I still think it doesn't help | |
| 18:50:23 | melwitt | hm ok | |
| 18:51:20 | mnaser | i mean we already have to do a bunch of gross stuff to the libvirt definition right now when the system is being created | |
| 20:11:16 | sean-k-mooney | melwitt: hopefully the diagrams help https://review.opendev.org/c/openstack/nova-specs/+/791047/5/specs/zed/approved/pci-device-tracking-in-placement.rst#340= | |
| 20:11:20 | sean-k-mooney | https://paste.opendev.org/show/btZLEVa44TutlmzBIwax/ | |
| 20:11:39 | sean-k-mooney | o/ | |
| 20:20:39 | melwitt | sean-k-mooney: thanks! o/ | |
| 23:01:11 | opendevreview | Miguel Lavalle proposed openstack/os-vif master: Delete trunk bridges to avoid race with Neutron https://review.opendev.org/c/openstack/os-vif/+/841499 | |
| #openstack-nova - 2022-05-12 | |||
| 06:26:41 | EugenMayer | Today it happened the second time. A VM has been shutdown - by the audit logs by the user '-' - and i do not understand what that is and what that is causing this. Are there any ways to find out. is '-' a systemic users, or what could this stand for? | |
| 06:28:46 | EugenMayer | Found this https://serverfault.com/questions/783053/openstack-instances-power-off-by-itself - not sure that is related in some way | |
| 07:08:09 | SharathCk_ | Hi, I am trying to enable Keystone audit middleware in our openstack services, But I can see placement does not contain api paste init conf or any other configuration where middleware can be added. Hence wanted to confirm whether keystone audit middleware is supported in Placement service. | |
| 07:58:28 | bauzas | good morning Nova | |
| 08:50:32 | Uggla | bauzas, I was looking at "new" upstream bugs. Is it normal that https://bugs.launchpad.net/nova/+bug/1787846 was opened 2018-08-19 ? | |
| 08:50:56 | bauzas | Uggla: yes, if the bug was previously not "New" | |
| 08:51:31 | bauzas | https://bugs.launchpad.net/nova/+bug/1787846/+activity | |
| 08:51:42 | Uggla | yep but latest update 2020-09-18 is it still relevent ? | |
| 08:55:53 | gibi | Uggla: we then to go from the newest bugs backwards in time | |
| 08:56:32 | Uggla | bauzas, gibi, should we ask to close it ? As I guess no ones would like to dig into that one and that's probably not relevent anymore ? | |
| 08:59:24 | gibi | Uggla: one way to treat it is that pike is already EOL upstream. So ask the reported to reproduce the problem on newer branch | |
| 09:00:24 | gibi | and set the bug back to Incomplete | |
| 09:01:05 | gibi | you can order the bugs from newer to older with https://bugs.launchpad.net/nova/+bugs?search=Search&field.status=New&orderby=-date_last_updated&start=0 | |
| 09:01:14 | Uggla | gibi, ok sounds good, otherwise we will keep this bug until the end of time. :) | |
| 09:02:42 | Uggla | gibi, in fact I treated several recent yesterday. But I looked at that one and was surprised by the open date and the "new" state. :) | |
| 09:02:51 | gibi | ahh OK | |
| 09:02:55 | gibi | cool then | |
| 09:06:22 | Uggla | bauzas, did you had a chance to look at https://review.opendev.org/c/openstack/nova-specs/+/831506/4/specs/zed/approved/unshelve-to-host.rst just to know if it looks ok for you ? | |
| 09:12:28 | bauzas | Uggla: not yet, will do today | |
| 09:15:36 | Uggla | bauzas, no hurries so far. | |
| 10:03:57 | bauzas | gibi: sean-k-mooney: do you remember if we verify the usage about an inventory if we create a new allocation directly by calling the Placement API ? | |
| 10:04:11 | bauzas | or do we verify it by Nova ? | |
| 10:05:07 | sean-k-mooney | you mean prevent over allocation | |
| 10:05:27 | sean-k-mooney | we prevent over allocation in placment i belive | |
| 10:06:37 | sean-k-mooney | so if two boot requests both recive an allocation_candiate for the same host in parallel and then both try to allocate that candiate only one will succeed and that is enforce by placment if im not mistaken | |
| 10:07:28 | sean-k-mooney | gibi: ^ that is correct right? or have i been fundementaly misunderstanding placment for years | |
| 10:07:45 | bauzas | sean-k-mooney: gibi: let's discuss this on downstream IRC, I have a BZ | |
| 10:08:12 | sean-k-mooney | ok just getting coffee | |
| 10:08:18 | sean-k-mooney | we can have a call if it helps | |
| 10:08:38 | sean-k-mooney | you can get into over allocation situations if the allocation ratio is change by the way | |
| 10:08:44 | sean-k-mooney | brb | |
| 10:18:51 | bauzas | I need to prepare lunch for my kid, but let's discuss this when I'm back | |
| 11:12:13 | gibi | sorry I was having lunch | |
| 11:13:19 | gibi | but overall yes, placement prevent overallocation via allocation creation, placement does not prevent overallocation via inventory modification (i.e changing allocation ratio, reserved, or total field) | |
| 11:14:38 | sean-k-mooney | right and when you go into an overallocated situation via inventory modifcaiotn placemnt allows you to free allocaitons agaisnt that inventory but wont allow you to crate new ones until you have enough capsity | |
| 11:15:33 | sean-k-mooney | so we have seen that were people have updated the cpu_pin_set or altered the allocation raitos and over commited it can prevent things like evacuate form working | |
| 11:16:05 | sean-k-mooney | since evacuate currently uses one allcoation isntead of two | |
| 11:22:59 | gibi | yes | |
| 11:23:01 | gibi | for all tree | |
| 11:23:03 | gibi | three | |
| 11:23:22 | gibi | in an overallocation situation inventory can change but allocation cannot | |
| 11:23:28 | gibi | except removal | |
| 11:23:35 | gibi | so you can delete allocations | |
| 11:25:47 | gibi | and yes evac is special as it needs to modify the single allocation of the VM | |
| 11:27:09 | gibi | hm but non evacuation also needs to move the VM allocation from the VM uuid to the migration uuid so that would fail too in an overallocation | |
| 11:27:33 | gibi | this is suboptimal as it prevents cleaning up an overallocated host by moving VMs out | |
| 11:28:07 | gibi | so I would temporary remove the overallocation by bumping allocation ratio high up, move the VMs out, and then restore the allocation ratio | |
| 11:28:51 | sean-k-mooney | ya that is the workaround for now | |
| 11:29:13 | sean-k-mooney | ideally we woudl mvoe to useing the migration object to track either the source or dest allcoations | |
| 11:29:39 | sean-k-mooney | im not sure bauzas case was related to evac | |
| 11:29:44 | sean-k-mooney | we can see when they are back | |
| 11:30:25 | gibi | and we would also need to special case the allocation update that only change consumer uuid, in placement to be allowed even in overallocated situation | |