| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-04-09 | |||
| 15:03:18 | stephenfin | So it's possible there are nova.conf files in the wild with e.g. '[devices] enabled_vgpu_types = nvidia-35, nvidia-36' | |
| 15:03:21 | sean-k-mooney | bauzas: we could not start instance vm at that point right | |
| 15:03:45 | stephenfin | and if they upgrade to Ussuri, they suddenly won't be able to start nova | |
| 15:03:49 | stephenfin | *nova-compute | |
| 15:03:58 | bauzas | stephenfin: I considered this point (having set multiple types while we weren't supporting it) | |
| 15:04:22 | bauzas | stephenfin: I think the hardstop is a reasonable failure | |
| 15:04:53 | bauzas | stephenfin: for operators having set multiple types, they would see that it wasn't really supported before and now they have some action to do | |
| 15:05:08 | bauzas | stephenfin: instead of blindly thinking it was working | |
| 15:05:19 | stephenfin | right, but it was allowed and didn't break anything | |
| 15:05:28 | bauzas | stephenfin: we only refuse to start | |
| 15:05:38 | bauzas | stephenfin: people can just adapt their confs if so | |
| 15:05:52 | bauzas | again, we made it on purpose | |
| 15:06:11 | bauzas | I see this as "we won't impact you if you made things the right way" | |
| 15:06:26 | bauzas | 'but if you made things the wrong way, it's time to change your config to make it work correctly" | |
| 15:06:30 | stephenfin | idk, I thought it was a given that nova.conf from N should continue to work in N+1 | |
| 15:06:34 | stephenfin | with warnings, sure | |
| 15:06:38 | bauzas | stephenfin: it will | |
| 15:07:03 | bauzas | stephenfin: but I can honestly revert to PS10 | |
| 15:07:51 | bauzas | stephenfin: tbc, there is zero upgrade impact for single-typed configurations | |
| 15:08:13 | bauzas | it's debatable tho and I don't want to hold | |
| 15:08:18 | sean-k-mooney | bauzas: any chace you could add a nova-staus command for this? | |
| 15:08:32 | bauzas | sean-k-mooney: certainly not for Train :) | |
| 15:08:41 | bauzas | which is the case we're discussing about | |
| 15:08:46 | sean-k-mooney | ah ok hehe | |
| 15:09:05 | sean-k-mooney | im also not sure how useful it would be given you would have eto run it on the compute node | |
| 15:09:29 | stephenfin | Yeah, I think what you had before was better. "Your nova.conf is wrong and needs to be fixed. We're reverting to the old Train behaviour until you do that" | |
| 15:09:46 | bauzas | stephenfin: okay here is a trade-off, I can write an upgrade note saying that multiple-typed confs have to adapt their confs | |
| 15:10:06 | bauzas | stephenfin: okay, let's revert to PS10 then | |
| 15:10:14 | sean-k-mooney | that still does not adress https://review.opendev.org/#/c/715490/11/nova/virt/libvirt/driver.py@815 | |
| 15:10:32 | bauzas | sean-k-mooney: yup, it will be PS10 with recreate hard stopping | |
| 15:10:37 | bauzas | we all okay ? | |
| 15:10:49 | sean-k-mooney | in that case you have a direct conflict between an instance type used by a guest on the current host and and the config | |
| 15:10:52 | stephenfin | Yeah, change where we do the hard stop and I'm happy | |
| 15:10:55 | sean-k-mooney | bauzas: yep | |
| 15:10:57 | sean-k-mooney | that works for me | |
| 15:11:06 | bauzas | cool, on it | |
| 15:11:13 | stephenfin | that's a straight up misconfiguration and should be a hard stop | |
| 15:11:35 | stephenfin | Might even be worth backporting a variant of that to stable/train | |
| 15:11:54 | sean-k-mooney | the warning | |
| 15:11:59 | sean-k-mooney | maybe | |
| 15:21:33 | bauzas | stephenfin: you're talking of the recreate issue you raised ? | |
| 15:21:57 | bauzas | stephenfin: apologies btw. I fucked up with your point, I thought you were arguing about what I fixed in PS11 | |
| 15:22:11 | bauzas | looks like multitasking with kids raises bugs | |
| 15:22:27 | bauzas | -ETOOMANYTHINGS | |
| 15:23:53 | stephenfin | bauzas: Um, I am :) I'm saying I think you need to hard fail at https://review.opendev.org/#/c/715490/11/nova/virt/libvirt/driver.py@815 and *not* hard fail at https://review.opendev.org/#/c/715490/11/nova/virt/libvirt/driver.py@6531 | |
| 15:24:07 | bauzas | yup, I finally understood | |
| 15:27:57 | bauzas | stephenfin: worth saying, do you think we should return a InvalidLibvirtGPUConfig within https://review.opendev.org/#/c/715490/11/nova/virt/libvirt/driver.py@815 ? | |
| 15:28:02 | bauzas | or another exception ? | |
| 15:28:19 | stephenfin | It's invalid config, so that makes sense IMO | |
| 15:28:21 | bauzas | I honestly think the operator messed up their config if so | |
| 15:28:26 | bauzas | yup, ok | |
| 15:30:29 | bauzas | stephenfin: to answer your last comment on https://review.opendev.org/#/c/715490/11/nova/virt/libvirt/driver.py@815 , | |
| 15:31:05 | bauzas | if the operator messes up their config and self._get_vgpu_type_per_pgpu(parent) can't find the right vgpu type because $mess | |
| 15:31:25 | bauzas | then we get a None value and the conditional statement fails | |
| 15:32:31 | bauzas | stephenfin: but unless the operator did provided a section for each of the vGPU types and added devices, you're right, we fall back to only supporting one type, like we did previously | |
| 15:34:46 | stephenfin | cool. gtk I was reading that correctly | |
| 15:47:28 | openstackgerrit | Sylvain Bauza proposed openstack/nova master: Support different vGPU types per pGPU https://review.opendev.org/715490 | |
| 15:47:28 | openstackgerrit | Sylvain Bauza proposed openstack/nova master: Functional test with pGPUs https://review.opendev.org/717975 | |
| 15:47:55 | bauzas | stephenfin: sean-k-mooney: gibi ^ | |
| 15:49:23 | sean-k-mooney | i like how gerrit leaves the filtes ticked if they have not changed form the last revision | |
| 15:49:27 | sean-k-mooney | looking now | |
| 15:50:49 | gibi | bauzas: ack | |
| 15:50:57 | gibi | will look shortly | |
| 15:51:21 | gibi | nova meeting starts in 9 minutes on #openstack-meeting-3 | |
| 15:52:04 | bauzas | gibi: sean-k-mooney: shit, forgot to remove two things, will provide a FUP if you don't disagree | |
| 15:52:33 | stephenfin | bauzas: https://review.opendev.org/#/c/715490/12/nova/virt/libvirt/driver.py@817 | |
| 15:52:37 | stephenfin | ah, guess that's one of them | |
| 15:52:57 | stephenfin | just respin it now? not like it's even in the queue yet | |
| 15:52:58 | sean-k-mooney | the continue | |
| 15:52:59 | bauzas | stephenfin: yup | |
| 15:53:06 | bauzas | stephenfin: okay, just doing | |
| 15:53:08 | sean-k-mooney | i have a commnet i was going to make | |
| 15:53:20 | sean-k-mooney | its just dead code so it wont break anythying but ya | |
| 15:54:45 | bauzas | uploading a new rev as of I speak | |
| 15:55:01 | openstackgerrit | Sylvain Bauza proposed openstack/nova master: Support different vGPU types per pGPU https://review.opendev.org/715490 | |
| 15:55:01 | openstackgerrit | Sylvain Bauza proposed openstack/nova master: Functional test with pGPUs https://review.opendev.org/717975 | |
| 15:55:05 | bauzas | (live my life, DSL with 1Mbps up) | |
| 15:55:21 | bauzas | stephenfin: sean-k-mooney: gibi: sorry, last rev ^ | |
| 15:55:32 | sean-k-mooney | big claime :P | |
| 15:55:33 | gibi | last? are you sure? ;) (just kidding) | |
| 15:55:41 | bauzas | latest* | |
| 15:56:21 | sean-k-mooney | bauzas: so out of scope for this cycle but is there any reason in victora we could not auto report custom triats for the vgpu providres | |
| 15:57:07 | bauzas | sean-k-mooney: it's within the spec, said as "planned" | |
| 15:57:17 | sean-k-mooney | bauzas: so we can skip https://review.opendev.org/#/c/715490/13/doc/source/admin/virtual-gpu.rst@290 | |
| 15:57:21 | sean-k-mooney | ok cool | |
| 15:57:36 | sean-k-mooney | doing it manually for now is fine by the way i was just wondering | |
| 15:58:07 | gibi | bauzas: Is there proper 4G coverage where you live? that would be a lot more than 1Mbps | |
| 15:58:42 | sean-k-mooney | gibi: bauzas was ment to be getting fiber a few months ago but there were issues | |
| 15:59:24 | bauzas | sean-k-mooney: I did not implemented it on purpose since mdev types are passed directly from the kernel driver without any kind of abstractional outcome | |
| 15:59:49 | sean-k-mooney | bauzas: sure i dont really thing that is a proablem | |
| 15:59:56 | bauzas | from my position, it is | |
| 16:00:11 | sean-k-mooney | we dont really have an abstration for cpu flags | |
| 16:00:20 | sean-k-mooney | we do some normallisation but very little | |
| 16:00:30 | sean-k-mooney | its basically the same thing | |
| 16:00:37 | bauzas | if nvidia decides that nvidia-31 is no longer a thing and just uses a new typename, say nvidia-mygoo for the same headset etc. then nova would be impacted | |
| 16:00:43 | openstackgerrit | Merged openstack/nova master: Temporarily skip TestNovaMigrationsMySQL https://review.opendev.org/718629 | |
| 16:01:40 | sean-k-mooney | bauzas: sure, same is mostly true for cpu flags. we could provide a mapping layer if we wanted via config but anyway some other days problem | |
| 16:02:12 | bauzas | sean-k-mooney: yup, CPU flags are the exact same things | |
| 16:02:20 | bauzas | using traits for them is terrible | |
| 16:02:34 | sean-k-mooney | bauzas: traits exits basicaly because of them | |