| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-07-22 | |||
| 14:17:51 | mriedem | and the api right? | |
| 14:17:58 | stephenfin | and the API, correct | |
| 14:18:20 | mriedem | we have talked a few times about the os-hypervisors API, or part of it, just proxying to placement rather than rely on those values set by the RT | |
| 14:18:27 | stephenfin | Might want to put them behind "this field has been deprecated getters/setters, but that's a nice-to-have | |
| 14:19:05 | stephenfin | No one should have (Core|Disk|Ram)Filter enabled so I don't need to worry about those and could probably remove them (it's been long enough) | |
| 14:19:34 | stephenfin | Because of that, I probably don't need to distinguish between vcpus and pcpus in the ComputeNode object and can just lump them in together | |
| 14:20:15 | stephenfin | The os-hypervisors API is already lying about available vcpus (overcommit ratios aren't respected) so what's another lie | |
| 14:20:54 | stephenfin | As you say, could update that to proxy to placement but that's tangential to this and probably shouldn't be lumped into what is already a somewhat large series | |
| 14:21:45 | stephenfin | and I can probably remove the resource claiming stuff for CPU, RAM and disk like dansmith was doing but I need to think of a workaround for handling overhead | |
| 14:22:19 | dansmith | overhead is handled by setting reserved in the compute configs right? | |
| 14:22:37 | stephenfin | Afraid not. XenAPI seems to be adding some amount for each instance booted | |
| 14:23:07 | stephenfin | and that's what we use to account for the extra CPU consumed when you enable emulator thread offloading in Libvirt using the 'isolate' policy | |
| 14:23:12 | dansmith | right, we discussed that, | |
| 14:23:32 | dansmith | and I think we just settled on "meh, set enough reserved to cover enough of your instances" | |
| 14:23:43 | dansmith | in dublin, IIRC | |
| 14:24:06 | stephenfin | I'm happy with that but am I going to break people? | |
| 14:24:52 | stephenfin | i.e. nova will now allow you to boot N+1 instances on a XenAPI host because overcommit is being ignored and the operator forgot to update their reserved | |
| 14:24:58 | mriedem | dansmith: boston was the first time i remember talking about it, but yeah it was awhile ago and the agreed workaround was bumping reserved in config | |
| 14:25:10 | mriedem | jay said he'd write a doc about how to calculate all that but... | |
| 14:27:40 | mriedem | https://bugs.launchpad.net/nova/+bug/1683858 has the details | |
| 14:27:40 | openstack | Launchpad bug 1683858 in OpenStack Compute (nova) "Allocation records do not contain overhead information" [Medium,Won't fix] | |
| 14:28:53 | stephenfin | I'm already changing all the things, so what's another thing to that pile | |
| 14:35:51 | efried | (stephenfin, I'm glad Matt & Dan answered you; I probably wouldn't have been much help there) | |
| 14:37:05 | bauzas | I'm honestly not really paying attention to this channel, but I saw the above discussion | |
| 14:37:14 | bauzas | stephenfin: dansmith: others: any stuff I can help ? | |
| 14:39:40 | stephenfin | bauzas: I think I'm good for now, but keep your ears open :) | |
| 14:42:04 | bauzas | okidoki | |
| 14:43:28 | efried | bauzas: mriedem, or johnthetubaguy: could I please ask you to do the final on https://review.opendev.org/#/c/651681/ ? I was going to proxy sean-k-mooney's +1, but I think that would be three Intels. | |
| 14:43:45 | bauzas | efried: lemme look | |
| 14:43:53 | efried | (auto-converge/post-copy) | |
| 14:45:47 | artom | dansmith, could we get your stamp of (dis)approval on https://review.opendev.org/#/c/671471/ when you get a chance? tia :) | |
| 14:46:37 | bauzas | efried: oh this one ? | |
| 14:46:40 | bauzas | sure, I can +W | |
| 14:46:47 | efried | thanks | |
| 14:46:50 | bauzas | I already reviewed it once | |
| 14:48:23 | openstackgerrit | Bence Romsics proposed openstack/os-vif master: Insert osprofiler trace info as external_ids to the bridge table https://review.opendev.org/665715 | |
| 14:56:47 | mnaser | in nova, has there been some sort of 'standard' on what you do in code that is inherently race-y in the cases that you lose the race to another worker | |
| 14:57:12 | jaypipes | stephenfin: hyperv has a completely different calculation as well for overhead, IIRC. | |
| 14:57:27 | mnaser | in cinder, there's a cast operation that involves a race between workers, and the worker that loses seems to be raising an exception | |
| 14:57:40 | mnaser | which to me seems a little too much, raising an exception for a non-failing scenario | |
| 14:58:57 | efried | mnaser: That sounds really broad to me. I would think one would need to know more about the specific situation. | |
| 14:59:14 | efried | Sometimes you could retry. Sometimes you could log and continue. Sometimes you would have to fail the operation... | |
| 14:59:51 | mnaser | https://github.com/openstack/cinder/blob/master/cinder/objects/cleanable.py#L142-L155 | |
| 15:00:28 | mnaser | so the code seems to pretty much gives me the indication that the other service created the worker, and now it'll resume happily after and we'll stop | |
| 15:06:43 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Remove deprecated Core/Ram/DiskFilter https://review.opendev.org/672065 | |
| 15:06:59 | stephenfin | jaypipes: ack. This is going to be fun | |
| 15:07:09 | stephenfin | I might be kicking that can down the road... | |
| 15:10:12 | efried | mnaser: If the code is saying "make sure this thing gets cleaned" and the exception indicates "another thread cleaned it" then it seems like the right thing is to log info ("Another thread took care of this for us, carrying on") and proceed. | |
| 15:12:02 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add functional test to resize volume-backed server with zero root disk https://review.opendev.org/672067 | |
| 15:14:21 | efried | mriedem: Ima rebase & reapprove https://review.opendev.org/#/c/669523/1 unless you're already in the middle | |
| 15:15:14 | openstackgerrit | Eric Fried proposed openstack/nova master: Remove Newton-era min compute checks for server create with device tags https://review.opendev.org/669523 | |
| 15:16:08 | mriedem | efried: haven't noticed | |
| 15:16:17 | efried | done | |
| 15:17:52 | Alphazero_ | Hi All, I recently uninstalled and reinstalled keystone via Juju on an up and running cluster and have been receiving this message when trying to list existing instances: | |
| 15:18:00 | Alphazero_ | The server is currently unavailable. Please try again at a later time.<br /><br /> | |
| 15:18:00 | Alphazero_ | (HTTP 503) (Request-ID: req-22f009ce-d65c-4795-8dd5-ff7c2a351d6d) | |
| 15:18:35 | Alphazero_ | checked nova.conf and its being configured by JuJu | |
| 15:19:13 | Alphazero_ | Any help would be greatly app. thanks! | |
| 15:25:08 | stephenfin | Alphazero_: You probably want #openstack or, at a stretch, #openstack-keystone | |
| 15:25:47 | Alphazero_ | whoops sorry ^^ just noticed the message at the top - will shift over... | |
| 15:28:45 | artom | Hrmm, maybe wait for sean-k-mooney to weight, seems like he has doubts as to the completeness of the fix | |
| 15:29:27 | sean-k-mooney | i think you are still relying on the pf being bound to a networking driver and having a netdev | |
| 15:29:39 | sean-k-mooney | if the PF is bound to vfio-pci | |
| 15:29:44 | sean-k-mooney | it will not have a net dev | |
| 15:30:04 | sean-k-mooney | in which case im not sure pci_utils.get_mac_by_pci_address will work | |
| 15:30:05 | dansmith | artom: sorry meant to say "ack" above, but...ack. | |
| 15:30:20 | sean-k-mooney | im looking at that code now | |
| 15:30:23 | artom | dansmith, no worries, I saw the +W go through, and am thankful (in silence) | |
| 15:30:33 | dansmith | artom: ack | |
| 15:31:45 | sean-k-mooney | artom: so ya https://github.com/openstack/nova/blob/master/nova/pci/utils.py#L162-L180 relise on the PF being bound to a networking driver not vfio-pci | |
| 15:32:35 | sean-k-mooney | so in that case we will still skip exposing the metadata for that PF | |
| 15:32:53 | artom | sean-k-mooney, so, I did test this (unlike the original patch :( ), what would cause the PF to use networking driver vs vfio-pci? | |
| 15:33:08 | artom | Because in the env I had it was using the former, going by what you're saying | |
| 15:33:13 | sean-k-mooney | udev | |
| 15:33:22 | sean-k-mooney | or the kernel configuration in general | |
| 15:33:43 | sean-k-mooney | try manually binding the PF driver to vfio-pci | |
| 15:33:52 | artom | Env is gone :( | |
| 15:34:03 | sean-k-mooney | it will still be usable by kvm but the metadata will be empty | |
| 15:34:35 | sean-k-mooney | you change wont break anythin it just will end up in the except block and retrun none | |
| 15:34:49 | artom | So, strictly speaking, better than what we have now? ;) | |
| 15:34:50 | sean-k-mooney | so your fix is valid if the PF is boud to say i40e | |
| 15:34:56 | sean-k-mooney | yep | |
| 15:35:11 | sean-k-mooney | as i said i think its incomplte not wrong. | |
| 15:35:18 | artom | Some of the PFs our there will start working, the others will remain device-tag-less | |
| 15:35:19 | sean-k-mooney | so maybe partial-bug? | |
| 15:35:32 | artom | That'd wfm | |
| 15:35:46 | sean-k-mooney | instead of close-bug and i'd be +1 on it | |
| 15:35:58 | artom | Ack, leave a review and I'll update the commit message | |
| 15:36:24 | sean-k-mooney | sure thing. ill do that in a few minutes. just grabbing something to drink brb | |
| 15:36:41 | artom | And if I ever get an SRIOV env again, I'll come up with a patch to address vfio-pci-bound PFs, presumable with some new method to find the MAC address from the PCI address. | |
| 15:41:20 | artom | mriedem, thanks for looking, replied | |
| 15:41:41 | artom | And welcome back, btw. No way 1 week was enough, but we're still happy to see you :) | |
| 15:44:55 | jangutter | artom: I think if the PF is bound to vfio-pci then the kernel knows almost nothing about the device. Only way to get info is via some kind of backdoor (like another PCI device that happens to "manage" the PF). | |
| 15:45:47 | artom | jangutter, so in those cases we should just give up on getting its MAC address? | |
| 15:46:01 | jangutter | artom: the really ugly way is to rebind the PCI device to a kernel driver, let it probe, check what the MAC is, rebind it to vfio-pci. | |
| 15:46:21 | artom | jangutter, I feel like that's not Nova's job... | |
| 15:46:25 | jangutter | artom: I concur. | |
| 15:47:28 | artom | Would the device's PCI address be the same in the guest as on the host? If so it'd be enough to just expose tag + PCI address in the metadata | |
| 15:47:36 | artom | If not, exposing just a device tag makes no sense | |
| 15:47:48 | sean-k-mooney | jangutter: well for neutron sriov passthouhg of PF we are ment to be discovering the PF mac and setting the neutron port to it | |