| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-06-20 | |||
| 18:49:34 | sean-k-mooney | which we can replace with microversion_parse | |
| 18:51:20 | sean-k-mooney | frickler: the noble run is currently on tempet https://zuul.openstack.org/stream/0b61b3b596554e4093acd6e17f61b26c?logfile=console.log | |
| 18:51:37 | sean-k-mooney | so thats looking promising | |
| 18:51:43 | sean-k-mooney | im going to leave it there for today | |
| 22:48:13 | opendevreview | sean mooney proposed openstack/nova master: split monkey_patching form import https://review.opendev.org/c/openstack/nova/+/922425 | |
| #openstack-nova - 2024-06-21 | |||
| 07:55:36 | jopdorp_ | Hi | |
| 07:56:13 | jopdorp_ | I was wondering if anyone has experience with H100 HGX systems , or maybe DGX and GPU passthrough to KVM VMs in nova | |
| 07:56:28 | jopdorp_ | We're seeing an issue where the nvidia-smi command hangs | |
| 07:57:06 | jopdorp_ | we do isntall fabricmanager, and I've tried some different configs of it. | |
| 07:57:29 | jopdorp_ | We're trying to pass one gpu per vm, on a host that has 8xH100 SXM gpus | |
| 07:57:49 | jopdorp_ | nv_open_q takes 100% of a single cpu core when nvidia-smi is invoked and hangs | |
| 11:45:59 | sean-k-mooney | jopdorp_: even if our downstream customers ask that question we would have to direct them to nvida supprot | |
| 11:46:22 | sean-k-mooney | what nvidia-smi does internally is really a black box that only they understand | |
| 14:02:39 | opendevreview | sean mooney proposed openstack/nova master: refactor cmd dir for non eventlet console_scripts https://review.opendev.org/c/openstack/nova/+/904424 | |
| 16:47:15 | opendevreview | sean mooney proposed openstack/nova master: update nova-next to use ubuntu 24.04 https://review.opendev.org/c/openstack/nova/+/922492 | |
| 17:35:04 | opendevreview | sean mooney proposed openstack/nova master: Remove nova debugger funcitonality https://review.opendev.org/c/openstack/nova/+/922496 | |
| 17:35:05 | opendevreview | sean mooney proposed openstack/nova master: Add IO Thread Pool Executor https://review.opendev.org/c/openstack/nova/+/922497 | |
| 18:03:39 | opendevreview | sean mooney proposed openstack/nova master: remove eventlet.tpool https://review.opendev.org/c/openstack/nova/+/905287 | |
| #openstack-nova - 2024-06-22 | |||
| 03:10:45 | opendevreview | Michael Still proposed openstack/nova master: WIP: libvirt: Add config option to require secure SPICE. https://review.opendev.org/c/openstack/nova/+/922544 | |
| 03:10:45 | opendevreview | Michael Still proposed openstack/nova master: WIP: libvirt: allow direct SPICE connections to qemu https://review.opendev.org/c/openstack/nova/+/922545 | |
| 03:10:46 | opendevreview | Michael Still proposed openstack/nova master: WIP: libvirt: allow concurrent access to SPICE consoles. https://review.opendev.org/c/openstack/nova/+/922546 | |
| 03:10:46 | opendevreview | Michael Still proposed openstack/nova master: WIP: libvirt: Add guest devices to support SPICE USB. https://review.opendev.org/c/openstack/nova/+/922547 | |
| 03:10:47 | opendevreview | Michael Still proposed openstack/nova master: WIP: libvirt: Optionally enable SPICE debug logging. https://review.opendev.org/c/openstack/nova/+/922548 | |
| 03:10:47 | opendevreview | Michael Still proposed openstack/nova master: WIP: libvirt: Optionally support sound when using SPICE. https://review.opendev.org/c/openstack/nova/+/922549 | |
| 07:13:02 | opendevreview | Michael Still proposed openstack/nova-specs master: Propose a new API microversion for a spice-direct console. https://review.opendev.org/c/openstack/nova-specs/+/915190 | |
| #openstack-nova - 2024-06-23 | |||
| 05:53:58 | opendevreview | Michael Still proposed openstack/nova-specs master: Propose a new API microversion for a spice-direct console. https://review.opendev.org/c/openstack/nova-specs/+/915190 | |
| #openstack-nova - 2024-06-24 | |||
| 01:55:39 | opendevreview | Michael Still proposed openstack/nova-specs master: Propose a new API microversion for a spice-direct console. https://review.opendev.org/c/openstack/nova-specs/+/915190 | |
| 04:02:22 | opendevreview | Takashi Kajinami proposed openstack/nova master: libvirt: Launch instances with stateless firmware https://review.opendev.org/c/openstack/nova/+/908890 | |
| 04:06:02 | opendevreview | Takashi Kajinami proposed openstack/nova master: libvirt: Launch instances with stateless firmware https://review.opendev.org/c/openstack/nova/+/908890 | |
| 05:03:47 | opendevreview | Takashi Kajinami proposed openstack/nova master: libvirt: Launch instances with stateless firmware https://review.opendev.org/c/openstack/nova/+/908890 | |
| 07:30:58 | sahid | o/ morning all | |
| 07:31:10 | sahid | I this we are good with this one, CI is green now :-) | |
| 07:31:13 | sahid | https://review.opendev.org/c/openstack/nova/+/921665 | |
| 07:31:28 | sahid | sean-k-mooney, gibi if you have a moment | |
| 09:14:31 | mikal | Hey sean-k-mooney, sorry for the horrendous delay with that SPICE spec. I blame CentOS 7. I've replied to your various comments and done some updates. | |
| 09:14:44 | mikal | So whenever you're up for it I'd appreciate your thoughts. | |
| 09:33:21 | opendevreview | Takashi Kajinami proposed openstack/nova master: Migrate MEM_ENCRYPTION_CONTEXT from root provider https://review.opendev.org/c/openstack/nova/+/921814 | |
| 09:41:27 | opendevreview | Takashi Kajinami proposed openstack/nova master: Migrate MEM_ENCRYPTION_CONTEXT from root provider https://review.opendev.org/c/openstack/nova/+/921814 | |
| 10:22:38 | ralonsoh | sean-k-mooney, hello! maybe you'll like to check https://review.opendev.org/c/openstack/oslo-specs/+/922597 | |
| 10:23:11 | sean-k-mooney | ralonsoh: nice find ill take a look | |
| 10:23:43 | ralonsoh | sean-k-mooney, I talked to Daniel this morning, so I was warned | |
| 11:01:50 | opendevreview | Stephen Finucane proposed openstack/nova master: api: Add 'removed' decorator https://review.opendev.org/c/openstack/nova/+/915736 | |
| 11:01:50 | opendevreview | Stephen Finucane proposed openstack/nova master: api: Don't do version check if nothing required https://review.opendev.org/c/openstack/nova/+/915737 | |
| 11:01:51 | opendevreview | Stephen Finucane proposed openstack/nova master: api: Add remaining missing query parameter schema https://review.opendev.org/c/openstack/nova/+/915738 | |
| 11:01:51 | opendevreview | Stephen Finucane proposed openstack/nova master: tests: Ensure all APIs have a request query schema https://review.opendev.org/c/openstack/nova/+/915739 | |
| 11:01:52 | opendevreview | Stephen Finucane proposed openstack/nova master: conf: Add '[api] response_validation' option https://review.opendev.org/c/openstack/nova/+/915740 | |
| 11:01:52 | opendevreview | Stephen Finucane proposed openstack/nova master: api: Add response body validation helper https://review.opendev.org/c/openstack/nova/+/915741 | |
| 11:01:53 | opendevreview | Stephen Finucane proposed openstack/nova master: api: Add response body schemas for server action APIs https://review.opendev.org/c/openstack/nova/+/915742 | |
| 11:01:53 | opendevreview | Stephen Finucane proposed openstack/nova master: api: Add response body schemas for remaining server action APIs https://review.opendev.org/c/openstack/nova/+/915743 | |
| 12:15:54 | opendevreview | Takashi Kajinami proposed openstack/nova master: libvirt: Launch instances with stateless firmware https://review.opendev.org/c/openstack/nova/+/908890 | |
| 12:18:18 | sean-k-mooney | mikal: i did a quick review and responed to some of the comments | |
| 12:20:02 | sean-k-mooney | mikal: this is my main feedback https://review.opendev.org/c/openstack/nova-specs/+/915190/comment/2ba5ce88_5e260fbf/ | |
| 12:20:47 | LarsErikP | Hi. Anyone running 2023.1, and a server with SapphireRapids-noTSX ? I got this weird behavior: https://paste.opendev.org/show/b8BM3HtnTKlQEqtv6i4I/ | |
| 12:20:48 | sean-k-mooney | i would love if we coudl make it so we can just return '"url":"spice://${KERBSIDE_HOST}/?token=f9906a48-b71e-4f18-baca-c987da3ebdb3"' | |
| 12:21:08 | LarsErikP | not really sure what happens here. I did not act like that when it was running zed... | |
| 12:21:45 | sean-k-mooney | LarsErikP: its likely a combination of 2 things 1 the qemu version your using and 2 the way nova is using the older libvirt api to detect comaptiablity | |
| 12:22:35 | sean-k-mooney | we have started to use the newer api recommeneded by the libvirt folk around 2023.1 ish | |
| 12:23:04 | LarsErikP | right.. so maybe I need a newer qemu version? | |
| 12:23:04 | sean-k-mooney | but saphire rapid in particalar is causing issues with the places we still use the old api | |
| 12:23:38 | LarsErikP | qemu-system-x86 1:6.2+dfsg-2ubuntu6.21 | |
| 12:24:17 | sean-k-mooney | we have 2 workaroudn options that can be used to side step the issue at the expecse fo delegating all cpu compatiablity checkign to libvirt | |
| 12:24:28 | sean-k-mooney | https://docs.openstack.org/nova/latest/configuration/config.html#workarounds.skip_cpu_compare_on_dest and https://docs.openstack.org/nova/latest/configuration/config.html#workarounds.skip_cpu_compare_at_startup | |
| 12:25:25 | LarsErikP | ah. ok. I'll try those | |
| 12:26:06 | sean-k-mooney | setting skip_cpu_compare_on_dest=true means tha tif the dest cpu is not compatiable instead of failing in pre-live migrate we will fail when we call migrate which is much later in the live-migrate workflow | |
| 12:26:48 | sean-k-mooney | skip_cpu_compare_at_startup=true means that if you udpate your config to add/remvoe flag ectra that make the vms incompatibel with a host or provision new compute nodes with a common config | |
| 12:26:55 | sean-k-mooney | isntead of geting an error when the compute start | |
| 12:27:03 | sean-k-mooney | you wont get an error until you try to boot a vm | |
| 12:27:45 | sean-k-mooney | both are not idaly but if libvirt/qemu is capable of runing the vm on the host in etither case the operation will succeed | |
| 12:28:12 | sean-k-mooney | at the expence of disabling the safty net nova is trying to provide | |
| 12:28:17 | LarsErikP | I think I can live with this. Thank you so much | |
| 12:28:35 | sean-k-mooney | medium term we want to fix it properly in nova | |
| 12:28:42 | LarsErikP | exactly | |
| 12:28:53 | LarsErikP | so in a future release, I won't need these anymore | |
| 12:29:08 | sean-k-mooney | the delta between the api iss the old api compared the host cpus wihtout takign account fo the fact that qemu does not pass all flag to the guest adn can emulate some | |
| 12:29:21 | sean-k-mooney | the new api in libvirt is ment to take both of those facotrs into account | |
| 12:29:29 | LarsErikP | right | |
| 12:29:48 | sean-k-mooney | historically this has not been a problem but with the microcode chagne intel has been making for the vulnerablities it has started to be | |
| 12:30:35 | sean-k-mooney | we have a bug and attempeted fix to track this but its semi stalled mainly be cause of dev/review time | |
| 12:31:33 | sean-k-mooney | https://bugs.launchpad.net/nova/+bug/2039803 | |
| 12:31:59 | sean-k-mooney | and i think https://review.opendev.org/c/openstack/nova/+/899185 is the initall attempted fix but its not complete/correct | |
| 12:33:17 | sean-k-mooney | LarsErikP: our downsteam products are based on train and wallaby and we have had similar issue with saphire rappids in both cases. | |
| 12:33:50 | sean-k-mooney | in our case its compounted by the fact oru tain based product uses a kernel/qemu that predate teh reelase of saphire rapids silicon | |
| 12:34:13 | sean-k-mooney | so there is no SapphireRapids cpu model aviaable in our train based release | |
| 12:34:41 | sean-k-mooney | our wallaby baed reslese is new enough to actully know what SapphireRapids is but there we see the issue you hit | |
| 12:35:06 | LarsErikP | Understand. For me, this behaviour did not occur before I upgraded from Zed to 2023.1.. | |
| 12:35:39 | sean-k-mooney | ya w had a partical fix in 2023.1 for a related issue | |
| 12:36:01 | sean-k-mooney | ill see if i can fidn that quickly. its why we added the workarounds | |
| 12:36:50 | sean-k-mooney | https://review.opendev.org/c/openstack/nova/+/870794 and https://review.opendev.org/c/openstack/nova/+/869950 | |
| 12:38:23 | opendevreview | Takashi Kajinami proposed openstack/nova master: libvirt: Launch instances with stateless firmware https://review.opendev.org/c/openstack/nova/+/908890 | |
| 12:40:15 | sean-k-mooney | LarsErikP: it actully looks like you may not have https://review.opendev.org/c/openstack/nova/+/869950 | |
| 12:40:36 | sean-k-mooney | based on your stack trace | |
| 12:40:37 | sean-k-mooney | https://review.opendev.org/c/openstack/nova/+/869950/9/nova/virt/libvirt/driver.py | |
| 12:41:01 | sean-k-mooney | its calling File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 10015, in _compare_cpu | |
| 12:43:15 | sean-k-mooney | perhaps not | |
| 12:44:11 | LarsErik1 | sean-k-mooney: lol, my irc-vm kernel paniced :P | |
| 12:44:16 | LarsErik1 | sean-k-mooney: lol, my irc-vm kernel paniced :P | |
| 12:44:24 | LarsErik1 | might have lost a few messages there :P | |
| 12:44:53 | sean-k-mooney | no worries i was speculatting that perhaps eyou did not hae the 2 patches we emrged in 2023.1 | |
| 12:45:36 | LarsErik1 | if that's the case, then UCA does not include the patches... | |
| 12:46:40 | LarsErikP | I think they should be included. UCA ships 27.1.0 | |