| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-10-21 | |||
| 12:24:19 | gibi | sean-k-mooney: lyarwood and I looked at nova | |
| 12:24:34 | sean-k-mooney | ok so i assume we are good then | |
| 12:24:49 | sean-k-mooney | ill skip it if ye have already | |
| 12:24:50 | gibi | sean-k-mooney: http://lists.openstack.org/pipermail/openstack-discuss/2020-October/018154.html | |
| 12:25:09 | gibi | stephenfin: ack, +2 then | |
| 12:25:20 | sean-k-mooney | cool | |
| 12:30:52 | sean-k-mooney | is devstack in #openstack-qa | |
| 12:31:30 | lyarwood | yes | |
| 12:47:34 | gibi | stephenfin: are we using the same strategy to remove os-agents that we did for nova-network related APIs? https://review.opendev.org/#/c/749309/3/releasenotes/notes/remove-xenapi-driver-194756049f22dc9e.yaml@9 | |
| 12:48:06 | stephenfin | gibi: yes, that seems the most sensible approach to me | |
| 12:48:24 | stephenfin | they're dead APIs without virt driver support | |
| 12:54:00 | gibi | stephenfin: thanks I wanted to make sure that we are OK to have a breaking API change without a microverison. But here I agree that having a microvesion does not help much | |
| 12:54:37 | stephenfin | yeah, a microversion doesn't make sense here since requesting the older version won't magically restore the older behavior | |
| 12:55:05 | gibi | for xenapi yes, for non xenapi it could restore the empty result instead of 410 | |
| 12:56:12 | gibi | but I hope non xenapi user did not use this API :) | |
| 12:56:38 | stephenfin | True :) | |
| 13:01:09 | gibi | thanks, approved | |
| 13:06:09 | openstack | Launchpad bug 1900800 in OpenStack Compute (nova) "VGPUs is not recreated on host reboot" [Low,Confirmed] - Assigned to Sylvain Bauza (sylvain-bauza) | |
| 13:06:09 | bauzas | I feel stupid today... https://bugs.launchpad.net/nova/+bug/1900800 | |
| 13:06:28 | bauzas | the thing I wrote : if mdev_not_exists(): lookup_the_mdev() | |
| 13:06:34 | bauzas | bravo sylvain... | |
| 13:08:30 | gibi | bauzas: don't be hard on yourself. everybody make mistakes. you can look at the good side, somebody is using the feature your implemented! | |
| 13:08:58 | bauzas | well, this one was terrible tho | |
| 13:09:15 | bauzas | even my wife who's accountant can understand the bug | |
| 13:09:37 | bauzas | and no clear solutions except "well, don't reboot" are viable | |
| 13:12:15 | gibi | bauzas: Eigi has some suggestion in the bug about reassigning every gpu at host reboot | |
| 13:13:50 | bauzas | gibi: well, I'd prefer pursuing the idea that Nova shouldn't honestly try to support reboots | |
| 13:14:02 | bauzas | oVirt doesn't do it, neither KVM | |
| 13:14:31 | bauzas | we provided this as a gentle way but changing this would require substantial changes that are due to something external to nova | |
| 13:15:09 | bauzas | but I'll speak about a workaround, which is to use a 3rd-party tool https://github.com/mdevctl/mdevctl | |
| 13:15:12 | gibi | bauzas: do you mean nova should not persist end user VMs, or just not to try to recover them automatically at host reboot? | |
| 13:15:18 | bauzas | gibi: the latter | |
| 13:15:57 | gibi | bauzas: but then after a host reboot a VM hard-reboot from the API should work, isn't it? | |
| 13:16:09 | gibi | or expected to work | |
| 13:16:35 | bauzas | gibi: it would be the responsability of the operator to recreate the former mdevs that diseappeared at reboot before restarting n-cpu | |
| 13:16:51 | gibi | so no automatic recovery of the compute host at all | |
| 13:17:01 | gibi | interesting | |
| 13:17:43 | bauzas | gibi: well, correct me if I'm wrong but SR-IOV VFs aren't either created or persisted by neutron ? | |
| 13:17:48 | bauzas | or nova | |
| 13:18:05 | bauzas | we just lookup the resources that were allocated before, right? | |
| 13:18:31 | gibi | bauzas: yes, we expect that the admi of the comute host made sure that the VFs are re-created during host reboot | |
| 13:18:49 | bauzas | gibi: so, I guess we should make the same expectation for the mdevs | |
| 13:18:55 | gibi | agree | |
| 13:19:06 | bauzas | and there is a tool | |
| 13:19:40 | bauzas | that doesn't solve my customers, but that goes a Red Hat thing, not an upstream problme | |
| 13:21:17 | gibi | bauzas: so with mdevs you went a different way that how nova manages PCI devices. Nova never tries to create SRIOV VFs | |
| 13:21:38 | bauzas | indeed | |
| 13:22:00 | bauzas | when I started on it, I was somehow having the expectation that mdevs were persistent | |
| 13:22:03 | gibi | still VFs might dissapeare during host reboot and I guess that means VMs depending on such VF goes to error | |
| 13:22:45 | bauzas | so I provided a convenient workaround for recreating the mdev, but now we're smarter, this hack can't longer work | |
| 13:23:18 | bauzas | and one day if we go with allocating mdevs based on NUMA affinity, then we would absolutely need to reallocate the same mdev from the same pGPU | |
| 13:23:23 | bauzas | at reboot | |
| 13:23:31 | bauzas | because performance matters | |
| 13:24:12 | bauzas | also, you could have two different mdev types for each GPU | |
| 13:24:25 | bauzas | one for performance and one for inference, per say | |
| 13:24:41 | bauzas | we couldn't just pick any mdev at the best guess | |
| 13:25:00 | gibi | yeah, true | |
| 13:25:04 | bauzas | anyway, I think I'll reply | |
| 13:25:15 | bauzas | specs reviews I promised | |
| 13:38:36 | dansmith | johnthetubaguy: does that api make sense to anything other than nova/ironic? | |
| 13:38:56 | dansmith | johnthetubaguy: like, would you use it for nova/libvirt? I joined halfway through | |
| 13:39:26 | dansmith | 077933 | |
| 13:45:19 | gibi | stephenfin: what was the reason grenade barked when you removed the o.vo in https://review.opendev.org/#/c/749312/4/nova/objects/bandwidth_usage.py@19 ? I try to find the remaining piece that connect the BandwidthUsageList to the rest of the codebase without success | |
| 13:45:49 | stephenfin | gibi: It was something to do with o.vo namespaces. A broken test | |
| 13:46:08 | stephenfin | wait, no, different thing | |
| 13:47:10 | stephenfin | I think the older compute nodes are passing notifications back to the conductor, and the conductor can't understand those because we removed the definitions | |
| 13:47:25 | stephenfin | so in that change we've stopped passing them back, meaning we can remove the definitions in the next release | |
| 13:47:50 | stephenfin | here's we stopped passing them back, as you've seen https://review.opendev.org/#/c/749312/4/nova/compute/utils.py | |
| 13:48:22 | gibi | stephenfin: I don't see where we passing the o.vo via RPC | |
| 13:49:05 | stephenfin | Here, no? https://review.opendev.org/#/c/749312/4/nova/notifications/base.py@203 | |
| 13:50:20 | gibi | it just emits the notification o.vo not the nova bw ovo to the message bus as notification, not as RPC message to the conductor | |
| 13:51:02 | stephenfin | The notification o.vo contains the bandwidth payload o.vo though | |
| 13:51:23 | gibi | the bandwidth payloa o.vo yes, but not he bandwidth nova o.vo | |
| 13:51:30 | stephenfin | ohhhhh | |
| 13:51:47 | stephenfin | I understand the question now :) Sec, lemme check | |
| 13:53:38 | stephenfin | gibi: I guess I'm just going to have to push up a DNM patch to get new grenade logs for. Will do that shortly | |
| 13:53:51 | gibi | stephenfin: OK, let's do that | |
| 15:17:51 | markguz_ | Hi. is it possible to use alternate names for the nova databases? i have a second region i'm setting up, it's in the same physical location so i wanted to colocate the dbs on the existing galera cluster. | |
| 15:18:08 | markguz_ | nova-manage db sync does not seem to like it however | |
| 15:20:45 | gibi | markguz_: the [database]/connection config option defines which database nova uses. I think you are free to use any db name you wish | |
| 15:28:49 | bauzas | gibi: markguz_: indeed, you're free but you need to have distinct connection strings between the api connection and each of the cell connection URLs | |
| 15:29:01 | gibi | bauzas: true | |
| 15:29:27 | markguz_ | gibi: bauzas: i have defined those distinct connection strings in nova | |
| 15:31:03 | bauzas | markguz_: what kind of "weird" nova-manage exception do you get when you sync ? | |
| 15:31:13 | gibi | from the doc https://docs.openstack.org/nova/rocky/cli/nova-manage.html#nova-database "The local database connection is determined by [database]/connection in the configuration file passed to nova-manage." | |
| 15:31:34 | bauzas | gibi: for a single cellv2, yep | |
| 15:31:57 | bauzas | gibi: but if you wanna create a second cell v2, then you need to pass a second connection URL, right? | |
| 15:32:22 | gibi | bauzas: our you have to run nova-manage from the node of the second cell controller | |
| 15:32:28 | gibi | s/our/or/ | |
| 15:33:26 | gibi | anyhow if the correct config file is passed to nova-manage but db sync fails then it is an interesting bug to look at | |
| 15:33:44 | openstackgerrit | Lee Yarwood proposed openstack/nova stable/train: Add a workaround config toggle to refuse ceph image upload https://review.opendev.org/757177 | |
| 15:33:48 | bauzas | gibi: yup, indeed | |
| 15:34:00 | bauzas | devstack does this by passing the connection string URL on the fly | |
| 15:34:03 | bauzas | but meh | |
| 15:34:11 | bauzas | I guess this is maybe unrelated to it | |
| 15:34:47 | markguz_ | bauzas: gibi: so these are the connection strings as I defined them in /etc/nova/nova.conf https://pastebin.com/e1Sb9PjF | |
| 15:36:48 | gibi | markguz_: could you share the error message and any stack trace that nova-manage produced? | |
| 15:38:15 | markguz_ | gibi: i'm just going throught the manage steps to recreate. hold on | |
| 15:39:38 | openstackgerrit | Merged openstack/nova stable/ussuri: Follow up for cherry-pick check for merge patch https://review.opendev.org/758463 | |
| 15:45:07 | markguz_ | gibi: bauzas: https://pastebin.com/n9aXXKSK | |