| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-09-03 | |||
| 22:12:11 | sean-k-mooney | aspiers: i think it should not or it was a race | |
| 22:12:14 | aspiers | 2. why not do it on both tests? | |
| 22:12:44 | aspiers | AFAICS it can't be a race because we're only talking about a single thread here | |
| 22:12:53 | sean-k-mooney | the second test does not modify the capablities but it would be valid to do it for both | |
| 22:12:57 | openstackgerrit | Merged openstack/nova master: libvirt: Fold in argument to '_update_provider_tree_for_vgpu' https://review.opendev.org/676729 | |
| 22:13:25 | sean-k-mooney | aspiers: i ment between fucntional test and the order they execute in | |
| 22:13:34 | aspiers | oh ok the second test doesn't look up caps because it gives up as soon as it sees the kernel parameter file missing | |
| 22:13:42 | sean-k-mooney | yep | |
| 22:14:07 | aspiers | but the compute host isn't shared between the tests surely? | |
| 22:14:26 | sean-k-mooney | it depens on how your mocking. | |
| 22:14:39 | sean-k-mooney | it should not be but i have seen wierd issue | |
| 22:15:19 | aspiers | start_compute() is called by each test | |
| 22:15:37 | aspiers | and creates a fresh compute service | |
| 22:16:18 | sean-k-mooney | when i ran it under the debugger i found you were not geting to the updated domain caps because of https://github.com/openstack/nova/blob/master/nova/virt/libvirt/host.py#L759-L760 | |
| 22:16:37 | aspiers | right | |
| 22:17:12 | aspiers | er | |
| 22:17:48 | sean-k-mooney | i dont really know why it passed before but self.compute.driver._host._domain_caps = None seams to fix it | |
| 22:18:09 | aspiers | actually I think it's https://github.com/openstack/nova/blob/master/nova/virt/libvirt/host.py#L1191 | |
| 22:18:23 | aspiers | if the kernel param file is missing, it won't cache any caps | |
| 22:18:45 | aspiers | so when it appears later, it will call the fake getDomCaps API for the first time | |
| 22:18:47 | sean-k-mooney | no i steped throug it in the debugger | |
| 22:18:53 | aspiers | that's in the off_on case | |
| 22:19:26 | sean-k-mooney | it might be becasue of my traits code actully | |
| 22:19:32 | aspiers | in the on_off case, it wouldn't even care what caps were cached when it gets to off | |
| 22:19:53 | aspiers | but technically I think it's good to reset both caches in both tests | |
| 22:20:09 | aspiers | in case the order of tests ever got changed in the future | |
| 22:20:23 | sean-k-mooney | im calling this https://review.opendev.org/#/c/666915/16/nova/virt/libvirt/driver.py@6832 | |
| 22:20:38 | aspiers | sorry, I mean the order of these https://github.com/openstack/nova/blob/master/nova/virt/libvirt/host.py#L1188-L1193 | |
| 22:21:06 | sean-k-mooney | sure ill add that | |
| 22:21:28 | sean-k-mooney | it make it symetic and really we dont want the order of those check to break this | |
| 22:21:40 | aspiers | right | |
| 22:21:51 | aspiers | that static_traits looks fine to me | |
| 22:22:00 | aspiers | except for the blank line after the def ;-) | |
| 22:22:13 | sean-k-mooney | anyway the change looks sane to you? | |
| 22:22:17 | aspiers | yes | |
| 22:22:30 | aspiers | I didn't review the whole thing yet, but that file does | |
| 22:22:52 | aspiers | oh, according to Gerrit I did review the whole thing | |
| 22:22:54 | aspiers | I forgot :) | |
| 22:22:55 | sean-k-mooney | there is no other change to that form the previous versin | |
| 22:22:58 | sean-k-mooney | yes | |
| 22:23:05 | sean-k-mooney | a week or so ago | |
| 22:23:08 | aspiers | k | |
| 22:23:26 | sean-k-mooney | it been wait since and the only change was resolving the merge conflcit with the sev code | |
| 22:23:34 | aspiers | not since PS9 | |
| 22:23:58 | aspiers | I hate how this old Gerrit makes it so hard to compare patchsets after rebase | |
| 22:24:11 | aspiers | that's the #1 thing I want a new Gerrit for | |
| 22:27:34 | openstackgerrit | sean mooney proposed openstack/nova master: libvirt: use domain capabilities to get supported device models https://review.opendev.org/666915 | |
| 22:27:34 | openstackgerrit | sean mooney proposed openstack/nova master: Add transform_image_metadata request filter https://review.opendev.org/665775 | |
| #openstack-nova - 2019-09-04 | |||
| 00:27:03 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: NUMA live migration support https://review.opendev.org/634606 | |
| 00:27:04 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Deprecate CONF.workarounds.enable_numa_live_migration https://review.opendev.org/640021 | |
| 00:27:04 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Functional tests for NUMA live migration https://review.opendev.org/672595 | |
| 01:00:08 | openstackgerrit | Merged openstack/nova master: re-calculate provider mapping during migration https://review.opendev.org/655112 | |
| 02:39:55 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: NUMA live migration support https://review.opendev.org/634606 | |
| 02:39:56 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Deprecate CONF.workarounds.enable_numa_live_migration https://review.opendev.org/640021 | |
| 02:39:56 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Functional tests for NUMA live migration https://review.opendev.org/672595 | |
| 04:15:11 | openstackgerrit | Brin Zhang proposed openstack/nova master: Add delete_on_termination to volume-attach API https://review.opendev.org/673133 | |
| 05:10:26 | cervigni | Hello, I have a dual socket server. Each cpu with two numa nodes. I would like to have the host to have a cpu overallocation of 2.0. At the same time I would like each flavor to stay bounded to its own numa node | |
| 05:10:32 | cervigni | Is there a way to achieve that? This would mean 2x instances on each numa nodes | |
| 05:10:36 | cervigni | I did not find a way to do that | |
| 05:10:51 | cervigni | This is because each of the CPU is connected on the pci-e bus on a GPU | |
| 05:11:03 | cervigni | and i need to respect the numa nodes configuration | |
| 05:39:05 | openstackgerrit | Dustin Cowles proposed openstack/nova master: Provider Config File: YAML file loading and schema validation https://review.opendev.org/673341 | |
| 05:39:05 | openstackgerrit | Dustin Cowles proposed openstack/nova master: WIP: Provider Config File: Public method to retrieve custom resources https://review.opendev.org/676029 | |
| 05:39:06 | openstackgerrit | Dustin Cowles proposed openstack/nova master: WIP: Provider Config File: Update provider tree with new custom resources https://review.opendev.org/676522 | |
| 06:30:07 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove unused methods https://review.opendev.org/679918 | |
| 06:49:30 | openstackgerrit | Yongli He proposed openstack/python-novaclient master: Microversion 2.78 - show server topology https://review.opendev.org/670790 | |
| 06:58:46 | cervigni | guys, could you help me find out if this is a bug before I submit something stupid on Launchpad? | |
| 06:58:48 | cervigni | https://pastebin.com/SfD63LgJ | |
| 06:59:35 | cervigni | it seems that a flavour configured as in the logs, continusly allocate cores only on the first numa node, instead of checking the actual cpus already allocated in that node | |
| 07:02:28 | gibi | cervigni: do you want dedicated physical cpu cores for your servers? | |
| 07:03:04 | cervigni | I need to do an overallocation of 2x but that does not go out of its own numa node | |
| 07:03:56 | cervigni | the virsh vcpuinfo for first instance is correct, for the second one is correct, but from the third one should start allocating on another numa node, since there is not available RAM left on numa node 1 | |
| 07:04:14 | cervigni | I don't know how it can allocate further vcpus on that numa node since there is no ram available | |
| 07:05:13 | cervigni | can I give you any more logs? | |
| 07:05:33 | gibi | cervigni: so you don't have memory overallocation configured | |
| 07:05:38 | cervigni | no | |
| 07:05:46 | cervigni | memory allocation is 1.0 | |
| 07:07:14 | gibi | cervigni: and you have the NUMATopologyFilter configured for the scheduler | |
| 07:07:56 | cervigni | /etc/nova/nova.conf:enabled_filters=RetryFilter,AvailabilityZoneFilter,CoreFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,ServerGroupAntiAffinityFilter,ServerGroupAffinityFilter,DifferentHostFilter,NUMATopologyFilter,AggregateInstanceExtraSpecsFilter,PciPassthroughFilter,AggregateMultiTenancyIsolation | |
| 07:07:58 | cervigni | yes | |
| 07:08:10 | gibi | then this feels like a bug, yes. | |
| 07:08:10 | cervigni | I am using rocky | |
| 07:08:18 | cervigni | oh dear :( | |
| 07:08:36 | cervigni | should the scheduler fill a first round of the four numa nodes before overallocating? | |
| 07:09:58 | gibi | cervigni: I don't think so. If there is overallocation then the scheduler can overallocate before it fills the other numa nodes. The only rule is I know of that a server cannot overallocate against itself | |
| 07:10:24 | cervigni | ok | |
| 07:10:29 | cervigni | so I will report it as a bug | |
| 07:10:37 | cervigni | Thanks gibi | |
| 07:10:45 | cervigni | I don't know if it relates of the VGPU side | |
| 07:10:51 | cervigni | I hope not | |
| 07:11:10 | gibi | cervigni: could you reporduce the problem without the VGPU resource in the flavor? | |
| 07:11:21 | gibi | cervigni: thanks for reporting the bug | |
| 07:11:22 | cervigni | I'll do it now | |
| 07:13:38 | cervigni | gibi exactly same issue without VGPU | |
| 07:13:48 | cervigni | any idea if there is a workaround I can use while the problem is fixed? | |
| 07:22:20 | gibi | unfortunately you cannot control the numa allocation directly from the outside | |
| 07:22:54 | cervigni | https://bugs.launchpad.net/nova/+bug/1842616 | |
| 07:22:55 | openstack | Launchpad bug 1842616 in OpenStack Compute (nova) "NUMA vcpus not correctly allocated against numa regions" [Undecided,New] | |
| 07:23:02 | cervigni | I put you in the watchlist as well gibi | |
| 07:23:37 | cervigni | Ok, for the moment I will leave no NUMA affinity, but with vgpus will be very messy | |