| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-02-14 | |||
| 15:26:16 | bauzas | for many reasons, one being at least that nvidia driver doesn't support more than one mdev attached to a guest per pGPU | |
| 15:26:33 | gibi | bauzas: I see | |
| 15:26:39 | gibi | bauzas: that hepls | |
| 15:26:42 | gibi | helps | |
| 15:26:49 | bauzas | ideally, the patch is ready | |
| 15:27:03 | gibi | bauzas: still I think we should fix the mapping now to avoid the pain of reshape in the future | |
| 15:27:07 | bauzas | but the problem is about the RP UUID > name transtive relationship | |
| 15:27:13 | bauzas | yeah | |
| 15:27:25 | bauzas | so, I need to find a quick and clean solution for this | |
| 15:27:45 | bauzas | I should probably ask for a hangout session if people agree | |
| 15:27:52 | bauzas | to get feedback | |
| 15:28:19 | gibi | bauzas: yeah that would make sense. Especially if we already have different solution proposals to choose from | |
| 15:28:35 | bauzas | gibi: the alternative would be asking operators to rebuild :p | |
| 15:28:57 | gibi | bauzas: hm, that could be option N+1 ;) | |
| 15:29:15 | bauzas | gibi: actually, lemme look at the driver code and see what I can do | |
| 15:29:22 | bauzas | we already have the providertree info | |
| 15:29:32 | bauzas | now the big concern I have is caching it | |
| 15:29:42 | bauzas | so that the method could use it | |
| 15:29:59 | bauzas | instead of passing straight the tree down to that method | |
| 15:30:03 | bauzas | which I dislike | |
| 15:30:16 | gibi | at least we have options | |
| 15:30:33 | gibi | In general I don't like caching as that is hard to make it right | |
| 15:41:55 | mriedem | stephenfin: replies in https://review.openstack.org/#/c/633620/ | |
| 15:42:08 | bauzas | gibi: yup me too | |
| 15:47:25 | stephenfin | mriedem: right back at ya (tl;dr: you do need to drop that line) | |
| 15:49:07 | mriedem | ok updating | |
| 15:52:38 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Make VolumeAttachmentsSampleV249 test other methods https://review.openstack.org/633621 | |
| 15:52:38 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Fix deps for api-samples tox env https://review.openstack.org/633620 | |
| 15:52:39 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add microversion to expose virtual device tags https://review.openstack.org/631948 | |
| 15:56:08 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove duplicate cleanup in functional tests https://review.openstack.org/636996 | |
| 16:11:21 | bauzas | efried: thanks for commenting https://review.openstack.org/#/c/636591/1/nova/virt/libvirt/driver.py | |
| 16:11:49 | efried | fo sho, look forward to seeing that patch. | |
| 16:11:52 | stephenfin | mriedem: Said it to lyarwood but I've backported that PYTHONDONTWRITEBYTECODE setting all the way back to queens. If you could take a look, I'd appreciate it https://review.openstack.org/#/q/topic:PYTHONDONTWRITEBYTECODE+status:open | |
| 16:12:21 | bauzas | efried: given we update the provider tree at compute startup, I just wonder if it's safe enough to assume that the libvirt in-memory cached attribute for the tree would be there anyway when an instance is spawned ? | |
| 16:12:29 | bauzas | mriedem: thoughts on that too ? | |
| 16:12:49 | bauzas | mriedem: context is https://review.openstack.org/#/c/636591/1/nova/virt/libvirt/driver.py@6347 | |
| 16:13:57 | efried | bauzas: We're pretty much *relying* on there being a match between upt and spawn | |
| 16:14:26 | bauzas | efried: there could be a race tho | |
| 16:14:28 | efried | bauzas: Also, pretty sure we run upt right before spawn, not just in the periodic, I could be wrong. So it should be pretty durn recent by then. | |
| 16:14:46 | bauzas | efried: where spawn() could run *before* upt | |
| 16:14:48 | bauzas | yeah | |
| 16:14:56 | efried | bauzas: But like I've said already (uh, possibly about something else): if there's a race, it's going to f up more than just this. | |
| 16:15:06 | mriedem | we run update_available_resource on compute start | |
| 16:15:08 | mriedem | which calls upt | |
| 16:15:11 | bauzas | i know | |
| 16:15:13 | mriedem | that's before the service is ready for spawn | |
| 16:15:15 | mriedem | so what's the issue | |
| 16:15:18 | bauzas | okay | |
| 16:15:33 | bauzas | I thought we were allowing instances to spawn, while compute was starting | |
| 16:15:41 | mriedem | f i hope not | |
| 16:15:46 | mriedem | https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L1259 | |
| 16:15:57 | mriedem | https://github.com/openstack/nova/blob/master/nova/service.py#L180 | |
| 16:16:08 | bauzas | oh good catch this is pre hook | |
| 16:16:12 | bauzas | perfect | |
| 16:16:35 | bauzas | mriedem: you're not worried by libvirt keeping a copy of the provider tree structure ? | |
| 16:16:47 | mriedem | i haven't been following along | |
| 16:17:07 | bauzas | mriedem: nah, it's just because we need to find the name of a RP by its uuid | |
| 16:17:08 | mriedem | sounds like more cache-tastic fun | |
| 16:17:24 | bauzas | mriedem: as we only get RP UUID from the allocations list we pass over spawn | |
| 16:17:28 | mriedem | which RP? | |
| 16:17:30 | efried | mriedem: TL;DR: the virt driver needs to be responsible for maintaining a mapping between RP and "real thing". The provider tree already has that information, so don't bother building something special, just save it off during upt. | |
| 16:17:47 | bauzas | mriedem: the RP which was allocated a VGPU | |
| 16:17:55 | mriedem | so the pgpu? | |
| 16:17:58 | mriedem | the child provider | |
| 16:18:01 | bauzas | yup | |
| 16:18:15 | bauzas | so that libvirt can use or create a mdev from *this* pgpu | |
| 16:18:25 | bauzas | http://paste.openstack.org/show/745109/ | |
| 16:18:44 | mriedem | what's the alternative? the driver holding it's own mapping of rp_uuid -> rp_name? which is something ProviderTree already does | |
| 16:18:49 | efried | yes ^ | |
| 16:18:56 | efried | (to both) | |
| 16:18:56 | bauzas | above is the fact that when we create a new instance, we use a random mdev which is unrelated to the pgpu which was allocated by placement | |
| 16:19:25 | mriedem | i don't have a strong feeling either way | |
| 16:19:46 | mriedem | ProviderTree is convenient albeit heavy weight and hopefully someone doesn't abuse it somehow, but doing our own mapping dict is redundant | |
| 16:19:49 | mriedem | so shrug | |
| 16:20:07 | bauzas | yeah me too, but I'm opinionated enough to make it the smoothiest | |
| 16:20:30 | bauzas | and caching the value in the libvirt driver object seems the smoothiest approach | |
| 16:20:45 | mriedem | so the provider tree would just get reset on every upt call? | |
| 16:20:47 | mriedem | if so, then sure | |
| 16:20:51 | bauzas | yeah | |
| 16:20:54 | mriedem | wfm | |
| 16:21:02 | bauzas | i'm done with complex cache invalidation mechanisms | |
| 16:21:13 | mriedem | are you SO done with them? | |
| 16:25:21 | bauzas | mriedem: /me tries to not think hard of AZs | |
| 16:25:34 | bauzas | or server groups | |
| 16:25:47 | mriedem | hey if you like azs i have some good news for you | |
| 16:25:48 | mriedem | https://review.openstack.org/#/c/509206/ | |
| 16:25:51 | mriedem | finally fixing that shitty old bug | |
| 16:26:35 | bauzas | \o/ | |
| 16:26:46 | bauzas | mriedem: straight adding it into my queue | |
| 16:27:39 | bauzas | gibi: you'll be happy to hear that functests work now, btw. :) | |
| 16:28:07 | gibi | bauzas: awesome | |
| 16:29:24 | gibi | every time I start kicking https://review.openstack.org/#/c/635859/ I got depressed after 15 minutes about I capabilities to understand what happening | |
| 16:29:38 | melwitt | o/ | |
| 16:30:14 | melwitt | gibi: me too. I tried for awhile and gave up | |
| 16:31:31 | gibi | melwitt: but I feel something deep is behind that so I cannot gave up :) | |
| 16:32:09 | melwitt | gibi: that's good, I think you will get to the bottom of it :) | |
| 16:32:20 | mriedem | well if you want to get really depressed http://logs.openstack.org/47/635147/11/check/nova-tox-functional-py35/5c3c2a9/job-output.txt.gz#_2019-02-13_13_33_41_103932 | |
| 16:32:34 | mriedem | that traceback wins the grand prize | |
| 16:32:41 | mriedem | and i don't know what to do about it | |
| 16:32:48 | mriedem | looks like the db dropped in the middle of the test | |