| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-09-03 | |||
| 22:22:54 | aspiers | I forgot :) | |
| 22:22:55 | sean-k-mooney | there is no other change to that form the previous versin | |
| 22:22:58 | sean-k-mooney | yes | |
| 22:23:05 | sean-k-mooney | a week or so ago | |
| 22:23:08 | aspiers | k | |
| 22:23:26 | sean-k-mooney | it been wait since and the only change was resolving the merge conflcit with the sev code | |
| 22:23:34 | aspiers | not since PS9 | |
| 22:23:58 | aspiers | I hate how this old Gerrit makes it so hard to compare patchsets after rebase | |
| 22:24:11 | aspiers | that's the #1 thing I want a new Gerrit for | |
| 22:27:34 | openstackgerrit | sean mooney proposed openstack/nova master: Add transform_image_metadata request filter https://review.opendev.org/665775 | |
| 22:27:34 | openstackgerrit | sean mooney proposed openstack/nova master: libvirt: use domain capabilities to get supported device models https://review.opendev.org/666915 | |
| #openstack-nova - 2019-09-04 | |||
| 00:27:03 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: NUMA live migration support https://review.opendev.org/634606 | |
| 00:27:04 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Functional tests for NUMA live migration https://review.opendev.org/672595 | |
| 00:27:04 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Deprecate CONF.workarounds.enable_numa_live_migration https://review.opendev.org/640021 | |
| 01:00:08 | openstackgerrit | Merged openstack/nova master: re-calculate provider mapping during migration https://review.opendev.org/655112 | |
| 02:39:55 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: NUMA live migration support https://review.opendev.org/634606 | |
| 02:39:56 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Functional tests for NUMA live migration https://review.opendev.org/672595 | |
| 02:39:56 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Deprecate CONF.workarounds.enable_numa_live_migration https://review.opendev.org/640021 | |
| 04:15:11 | openstackgerrit | Brin Zhang proposed openstack/nova master: Add delete_on_termination to volume-attach API https://review.opendev.org/673133 | |
| 05:10:26 | cervigni | Hello, I have a dual socket server. Each cpu with two numa nodes. I would like to have the host to have a cpu overallocation of 2.0. At the same time I would like each flavor to stay bounded to its own numa node | |
| 05:10:32 | cervigni | Is there a way to achieve that? This would mean 2x instances on each numa nodes | |
| 05:10:36 | cervigni | I did not find a way to do that | |
| 05:10:51 | cervigni | This is because each of the CPU is connected on the pci-e bus on a GPU | |
| 05:11:03 | cervigni | and i need to respect the numa nodes configuration | |
| 05:39:05 | openstackgerrit | Dustin Cowles proposed openstack/nova master: WIP: Provider Config File: Public method to retrieve custom resources https://review.opendev.org/676029 | |
| 05:39:05 | openstackgerrit | Dustin Cowles proposed openstack/nova master: Provider Config File: YAML file loading and schema validation https://review.opendev.org/673341 | |
| 05:39:06 | openstackgerrit | Dustin Cowles proposed openstack/nova master: WIP: Provider Config File: Update provider tree with new custom resources https://review.opendev.org/676522 | |
| 06:30:07 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove unused methods https://review.opendev.org/679918 | |
| 06:49:30 | openstackgerrit | Yongli He proposed openstack/python-novaclient master: Microversion 2.78 - show server topology https://review.opendev.org/670790 | |
| 06:58:46 | cervigni | guys, could you help me find out if this is a bug before I submit something stupid on Launchpad? | |
| 06:58:48 | cervigni | https://pastebin.com/SfD63LgJ | |
| 06:59:35 | cervigni | it seems that a flavour configured as in the logs, continusly allocate cores only on the first numa node, instead of checking the actual cpus already allocated in that node | |
| 07:02:28 | gibi | cervigni: do you want dedicated physical cpu cores for your servers? | |
| 07:03:04 | cervigni | I need to do an overallocation of 2x but that does not go out of its own numa node | |
| 07:03:56 | cervigni | the virsh vcpuinfo for first instance is correct, for the second one is correct, but from the third one should start allocating on another numa node, since there is not available RAM left on numa node 1 | |
| 07:04:14 | cervigni | I don't know how it can allocate further vcpus on that numa node since there is no ram available | |
| 07:05:13 | cervigni | can I give you any more logs? | |
| 07:05:33 | gibi | cervigni: so you don't have memory overallocation configured | |
| 07:05:38 | cervigni | no | |
| 07:05:46 | cervigni | memory allocation is 1.0 | |
| 07:07:14 | gibi | cervigni: and you have the NUMATopologyFilter configured for the scheduler | |
| 07:07:56 | cervigni | /etc/nova/nova.conf:enabled_filters=RetryFilter,AvailabilityZoneFilter,CoreFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,ServerGroupAntiAffinityFilter,ServerGroupAffinityFilter,DifferentHostFilter,NUMATopologyFilter,AggregateInstanceExtraSpecsFilter,PciPassthroughFilter,AggregateMultiTenancyIsolation | |
| 07:07:58 | cervigni | yes | |
| 07:08:10 | cervigni | I am using rocky | |
| 07:08:10 | gibi | then this feels like a bug, yes. | |
| 07:08:18 | cervigni | oh dear :( | |
| 07:08:36 | cervigni | should the scheduler fill a first round of the four numa nodes before overallocating? | |
| 07:09:58 | gibi | cervigni: I don't think so. If there is overallocation then the scheduler can overallocate before it fills the other numa nodes. The only rule is I know of that a server cannot overallocate against itself | |
| 07:10:24 | cervigni | ok | |
| 07:10:29 | cervigni | so I will report it as a bug | |
| 07:10:37 | cervigni | Thanks gibi | |
| 07:10:45 | cervigni | I don't know if it relates of the VGPU side | |
| 07:10:51 | cervigni | I hope not | |
| 07:11:10 | gibi | cervigni: could you reporduce the problem without the VGPU resource in the flavor? | |
| 07:11:21 | gibi | cervigni: thanks for reporting the bug | |
| 07:11:22 | cervigni | I'll do it now | |
| 07:13:38 | cervigni | gibi exactly same issue without VGPU | |
| 07:13:48 | cervigni | any idea if there is a workaround I can use while the problem is fixed? | |
| 07:22:20 | gibi | unfortunately you cannot control the numa allocation directly from the outside | |
| 07:22:54 | cervigni | https://bugs.launchpad.net/nova/+bug/1842616 | |
| 07:22:55 | openstack | Launchpad bug 1842616 in OpenStack Compute (nova) "NUMA vcpus not correctly allocated against numa regions" [Undecided,New] | |
| 07:23:02 | cervigni | I put you in the watchlist as well gibi | |
| 07:23:37 | cervigni | Ok, for the moment I will leave no NUMA affinity, but with vgpus will be very messy | |
| 07:23:50 | cervigni | Guess Stein will have the same issue? | |
| 07:26:35 | gibi | unfortunately I cannot try to reproduce it as I don't have numa + VGPU capable machine. I might get hold of a 2 numa nodes machine ... | |
| 07:26:54 | gibi | anyhow thanks for the report | |
| 07:26:57 | cervigni | VGPU is irrelevant it seems | |
| 08:01:09 | kashyap | aspiers: Catching up with the relevant conversation w/ Eric in the scrollback. | |
| 08:04:08 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: update allocation in binding profile during migrate https://review.opendev.org/656422 | |
| 08:06:58 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: prepare func test env for moving servers with bandwidth https://review.opendev.org/655109 | |
| 08:09:44 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Func test for migrate server with ports having resource request https://review.opendev.org/655113 | |
| 08:12:28 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Make _rever_allocation nested allocation aware https://review.opendev.org/676138 | |
| 08:15:14 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Support reverting migration / resize with bandwidth https://review.opendev.org/676140 | |
| 08:18:00 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Func test for migrate re-schedule with bandwidth https://review.opendev.org/676972 | |
| 08:20:50 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Support migrating SRIOV port with bandwidth https://review.opendev.org/676980 | |
| 08:23:39 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Allow migrating server with port resource request https://review.opendev.org/671497 | |
| 08:26:26 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Do not query allocations twice in finish_revert_resize https://review.opendev.org/678827 | |
| 08:27:12 | Sundar | Hi gibi, hoaw are you doing? | |
| 08:29:15 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Allow resizing server with port resource request https://review.opendev.org/679019 | |
| 08:34:22 | openstackgerrit | Adrian Chiris proposed openstack/nova master: Revert "vif: Resolve a TODO and update another" https://review.opendev.org/675776 | |
| 08:40:32 | Sundar | Re. Cyborg notifications, I noticed that, once Nova kicks off the binding of ARQs, the CYborg event comes almost immediately if no programming is involved. The event gets lost because there are no waiters: I am doing the wait in the virt driver during spawn, The logs are here: http://paste.openstack.org/show/770470/ | |
| 08:42:51 | openstackgerrit | Liang Fang proposed openstack/nova master: [WIP] Leverage openCAS cache framework for VM disks https://review.opendev.org/663542 | |
| 08:43:07 | Sundar | gibi: ^ I am proposing to move the create/bind ARQs from build_and_run_instance (https://review.opendev.org/#/c/631244/37/nova/compute/manager.py@1940) to _build_resources (https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L2392). | |
| 08:44:06 | Sundar | gibi: This has 2 benefits: (a) The instance has th right host and hostname after resource claim, and (b) if VM spawn fails, there is a good place to delete the ARQs in flight. | |
| 08:47:34 | Sundar | The problem I am facing is this: can the logic be like http://paste.openstack.org/show/770481/? If non-ARQ-related stuff is not under the with, then the with statement is going to just kick off the binding and then wait till the resolution happens. Nothing else is going on during the wait. | |
| 09:04:29 | gibi | Sundar: on a call, I have to get back to you in 15 minutes | |
| 09:11:39 | openstackgerrit | Brin Zhang proposed openstack/nova master: Add delete_on_termination to volume-attach API https://review.opendev.org/673133 | |
| 09:18:46 | gibi | Sundar: there is already a wait_for_instance_events call in nova.virt.libvirt.driver.LibvirtDriver._create_domain_and_network (called from spawn()) | |
| 09:20:12 | gibi | Sundar: or you only want to wait for the cyborg specific event in _build_resources ? | |
| 09:27:41 | gibi | Sundar: but yeah, the goal is that you kick of the binding and while you are waiting for the response let other things progress for the build up until the point where you really need the ARQs to be bound | |
| 09:30:17 | gibi | Sundar: I might try to add the with wait_for_instance_events under with _build_resources call in https://github.com/openstack/nova/blob/cbaea3bd690c683e83a063acfa69919668eaa123/nova/compute/manager.py#L2233 | |
| 09:32:12 | gibi | Sundar: I'm not sure how the the two context manager would combine if you put the wait_for_instance call inside the _build_resources without trying | |
| 09:36:50 | kashyap | Huh, this commit added the CPU instruction: https://opendev.org/openstack/nova/commit/866ca1468dfa35f319f520d0f7b93b2d0fe32f62 | |
| 09:36:56 | kashyap | (trait, i.e.) | |
| 09:37:03 | kashyap | But it does _not_ exist in os-traits yet :-( | |
| 09:37:21 | kashyap | There's no such trait as HW_CPU_X86_AVX512VNNI in 'os-traits' | |
| 09:39:01 | kashyap | Hmm, never mind ... there _is_ the trait, and it is merged: https://review.opendev.org/#/c/672888/1/os_traits/hw/cpu/x86/__init__.py | |
| 09:45:01 | aspiers | sometimes I have to pip install the latest os-traits in my tox envs | |
| 09:45:11 | aspiers | it's quicker than a --recreate | |
| 09:46:23 | gibi | I guess tox did not pick up the os-traits >= 0.16.0 change in the requirements.txt | |