| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-09-04 | |||
| 05:10:32 | cervigni | Is there a way to achieve that? This would mean 2x instances on each numa nodes | |
| 05:10:36 | cervigni | I did not find a way to do that | |
| 05:10:51 | cervigni | This is because each of the CPU is connected on the pci-e bus on a GPU | |
| 05:11:03 | cervigni | and i need to respect the numa nodes configuration | |
| 05:39:05 | openstackgerrit | Dustin Cowles proposed openstack/nova master: Provider Config File: YAML file loading and schema validation https://review.opendev.org/673341 | |
| 05:39:05 | openstackgerrit | Dustin Cowles proposed openstack/nova master: WIP: Provider Config File: Public method to retrieve custom resources https://review.opendev.org/676029 | |
| 05:39:06 | openstackgerrit | Dustin Cowles proposed openstack/nova master: WIP: Provider Config File: Update provider tree with new custom resources https://review.opendev.org/676522 | |
| 06:30:07 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove unused methods https://review.opendev.org/679918 | |
| 06:49:30 | openstackgerrit | Yongli He proposed openstack/python-novaclient master: Microversion 2.78 - show server topology https://review.opendev.org/670790 | |
| 06:58:46 | cervigni | guys, could you help me find out if this is a bug before I submit something stupid on Launchpad? | |
| 06:58:48 | cervigni | https://pastebin.com/SfD63LgJ | |
| 06:59:35 | cervigni | it seems that a flavour configured as in the logs, continusly allocate cores only on the first numa node, instead of checking the actual cpus already allocated in that node | |
| 07:02:28 | gibi | cervigni: do you want dedicated physical cpu cores for your servers? | |
| 07:03:04 | cervigni | I need to do an overallocation of 2x but that does not go out of its own numa node | |
| 07:03:56 | cervigni | the virsh vcpuinfo for first instance is correct, for the second one is correct, but from the third one should start allocating on another numa node, since there is not available RAM left on numa node 1 | |
| 07:04:14 | cervigni | I don't know how it can allocate further vcpus on that numa node since there is no ram available | |
| 07:05:13 | cervigni | can I give you any more logs? | |
| 07:05:33 | gibi | cervigni: so you don't have memory overallocation configured | |
| 07:05:38 | cervigni | no | |
| 07:05:46 | cervigni | memory allocation is 1.0 | |
| 07:07:14 | gibi | cervigni: and you have the NUMATopologyFilter configured for the scheduler | |
| 07:07:56 | cervigni | /etc/nova/nova.conf:enabled_filters=RetryFilter,AvailabilityZoneFilter,CoreFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,ServerGroupAntiAffinityFilter,ServerGroupAffinityFilter,DifferentHostFilter,NUMATopologyFilter,AggregateInstanceExtraSpecsFilter,PciPassthroughFilter,AggregateMultiTenancyIsolation | |
| 07:07:58 | cervigni | yes | |
| 07:08:10 | gibi | then this feels like a bug, yes. | |
| 07:08:10 | cervigni | I am using rocky | |
| 07:08:18 | cervigni | oh dear :( | |
| 07:08:36 | cervigni | should the scheduler fill a first round of the four numa nodes before overallocating? | |
| 07:09:58 | gibi | cervigni: I don't think so. If there is overallocation then the scheduler can overallocate before it fills the other numa nodes. The only rule is I know of that a server cannot overallocate against itself | |
| 07:10:24 | cervigni | ok | |
| 07:10:29 | cervigni | so I will report it as a bug | |
| 07:10:37 | cervigni | Thanks gibi | |
| 07:10:45 | cervigni | I don't know if it relates of the VGPU side | |
| 07:10:51 | cervigni | I hope not | |
| 07:11:10 | gibi | cervigni: could you reporduce the problem without the VGPU resource in the flavor? | |
| 07:11:21 | gibi | cervigni: thanks for reporting the bug | |
| 07:11:22 | cervigni | I'll do it now | |
| 07:13:38 | cervigni | gibi exactly same issue without VGPU | |
| 07:13:48 | cervigni | any idea if there is a workaround I can use while the problem is fixed? | |
| 07:22:20 | gibi | unfortunately you cannot control the numa allocation directly from the outside | |
| 07:22:54 | cervigni | https://bugs.launchpad.net/nova/+bug/1842616 | |
| 07:22:55 | openstack | Launchpad bug 1842616 in OpenStack Compute (nova) "NUMA vcpus not correctly allocated against numa regions" [Undecided,New] | |
| 07:23:02 | cervigni | I put you in the watchlist as well gibi | |
| 07:23:37 | cervigni | Ok, for the moment I will leave no NUMA affinity, but with vgpus will be very messy | |
| 07:23:50 | cervigni | Guess Stein will have the same issue? | |
| 07:26:35 | gibi | unfortunately I cannot try to reproduce it as I don't have numa + VGPU capable machine. I might get hold of a 2 numa nodes machine ... | |
| 07:26:54 | gibi | anyhow thanks for the report | |
| 07:26:57 | cervigni | VGPU is irrelevant it seems | |
| 08:01:09 | kashyap | aspiers: Catching up with the relevant conversation w/ Eric in the scrollback. | |
| 08:04:08 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: update allocation in binding profile during migrate https://review.opendev.org/656422 | |
| 08:06:58 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: prepare func test env for moving servers with bandwidth https://review.opendev.org/655109 | |
| 08:09:44 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Func test for migrate server with ports having resource request https://review.opendev.org/655113 | |
| 08:12:28 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Make _rever_allocation nested allocation aware https://review.opendev.org/676138 | |
| 08:15:14 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Support reverting migration / resize with bandwidth https://review.opendev.org/676140 | |
| 08:18:00 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Func test for migrate re-schedule with bandwidth https://review.opendev.org/676972 | |
| 08:20:50 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Support migrating SRIOV port with bandwidth https://review.opendev.org/676980 | |
| 08:23:39 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Allow migrating server with port resource request https://review.opendev.org/671497 | |
| 08:26:26 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Do not query allocations twice in finish_revert_resize https://review.opendev.org/678827 | |
| 08:27:12 | Sundar | Hi gibi, hoaw are you doing? | |
| 08:29:15 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Allow resizing server with port resource request https://review.opendev.org/679019 | |
| 08:34:22 | openstackgerrit | Adrian Chiris proposed openstack/nova master: Revert "vif: Resolve a TODO and update another" https://review.opendev.org/675776 | |
| 08:40:32 | Sundar | Re. Cyborg notifications, I noticed that, once Nova kicks off the binding of ARQs, the CYborg event comes almost immediately if no programming is involved. The event gets lost because there are no waiters: I am doing the wait in the virt driver during spawn, The logs are here: http://paste.openstack.org/show/770470/ | |
| 08:42:51 | openstackgerrit | Liang Fang proposed openstack/nova master: [WIP] Leverage openCAS cache framework for VM disks https://review.opendev.org/663542 | |
| 08:43:07 | Sundar | gibi: ^ I am proposing to move the create/bind ARQs from build_and_run_instance (https://review.opendev.org/#/c/631244/37/nova/compute/manager.py@1940) to _build_resources (https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L2392). | |
| 08:44:06 | Sundar | gibi: This has 2 benefits: (a) The instance has th right host and hostname after resource claim, and (b) if VM spawn fails, there is a good place to delete the ARQs in flight. | |
| 08:47:34 | Sundar | The problem I am facing is this: can the logic be like http://paste.openstack.org/show/770481/? If non-ARQ-related stuff is not under the with, then the with statement is going to just kick off the binding and then wait till the resolution happens. Nothing else is going on during the wait. | |
| 09:04:29 | gibi | Sundar: on a call, I have to get back to you in 15 minutes | |
| 09:11:39 | openstackgerrit | Brin Zhang proposed openstack/nova master: Add delete_on_termination to volume-attach API https://review.opendev.org/673133 | |
| 09:18:46 | gibi | Sundar: there is already a wait_for_instance_events call in nova.virt.libvirt.driver.LibvirtDriver._create_domain_and_network (called from spawn()) | |
| 09:20:12 | gibi | Sundar: or you only want to wait for the cyborg specific event in _build_resources ? | |
| 09:27:41 | gibi | Sundar: but yeah, the goal is that you kick of the binding and while you are waiting for the response let other things progress for the build up until the point where you really need the ARQs to be bound | |
| 09:30:17 | gibi | Sundar: I might try to add the with wait_for_instance_events under with _build_resources call in https://github.com/openstack/nova/blob/cbaea3bd690c683e83a063acfa69919668eaa123/nova/compute/manager.py#L2233 | |
| 09:32:12 | gibi | Sundar: I'm not sure how the the two context manager would combine if you put the wait_for_instance call inside the _build_resources without trying | |
| 09:36:50 | kashyap | Huh, this commit added the CPU instruction: https://opendev.org/openstack/nova/commit/866ca1468dfa35f319f520d0f7b93b2d0fe32f62 | |
| 09:36:56 | kashyap | (trait, i.e.) | |
| 09:37:03 | kashyap | But it does _not_ exist in os-traits yet :-( | |
| 09:37:21 | kashyap | There's no such trait as HW_CPU_X86_AVX512VNNI in 'os-traits' | |
| 09:39:01 | kashyap | Hmm, never mind ... there _is_ the trait, and it is merged: https://review.opendev.org/#/c/672888/1/os_traits/hw/cpu/x86/__init__.py | |
| 09:45:01 | aspiers | sometimes I have to pip install the latest os-traits in my tox envs | |
| 09:45:11 | aspiers | it's quicker than a --recreate | |
| 09:46:23 | gibi | I guess tox did not pick up the os-traits >= 0.16.0 change in the requirements.txt | |
| 09:46:46 | gibi | the nova master has it, maybe you need to rebase your patch | |
| 09:47:17 | aspiers | tox doesn't seem good at noticing when to update anyway | |
| 09:47:24 | aspiers | IME at least | |
| 10:09:50 | openstackgerrit | Brin Zhang proposed openstack/python-novaclient master: Microversion 2.78: Add delete_on_termination to volume-attach API https://review.opendev.org/673485 | |
| 10:14:19 | kashyap | aspiers: Yeah, I see what you mean; what I found was that in my venv: | |
| 10:14:19 | kashyap | >>> import os_traits | |
| 10:14:19 | kashyap | >>> print os_traits.HW_CPU_X86_AVX512VNNI | |
| 10:14:19 | kashyap | HW_CPU_X86_AVX512VNNI | |
| 10:14:41 | kashyap | (I _think_ I manually 'pip'-installed os-traits in there, I guess. I already forget) | |
| 10:32:13 | brinzhang | While run functional test cases, either execute tox or python -m testtools.run command, it will be raise "AssertionError: There is no script for 401 version", details http://paste.openstack.org/show/770633/ | |
| 10:32:37 | brinzhang | How can I avoid this issue? | |
| 10:35:41 | Sundar | gibi, Yes, we want to maximize the overlap by kicking off the binding as early as possible and wait for the ARQs as late as possible, but that is not easy without major surgery. So, I'll be content with dong it inside or after _build_resources. However, if we do: "with .... : foo(); bar()", then only foo() and bar() are done concurrently while wai | |
| 10:35:42 | Sundar | ting for the event, right? Also, foo and bar must both be CYborg-related. | |
| 10:36:32 | sean-k-mooney | brinzhang: "tox -e functional-py36 -- nova.tests.functional.api_sample_tests.test_quota_sets.NoopQuotaSetsSampleJsonTests.test_show_quotas" this works for me | |
| 10:36:49 | sean-k-mooney | brinzhang: you might need to add -r to recreate the tox env. | |
| 10:36:55 | gibi | Sundar: yes, only foo and bar will be parallel because when the with wait context manager exits it waits for the events | |
| 10:36:59 | brinzhang | sean-k-mooney: I will have try, thanks | |
| 10:37:13 | artom | brinzhang, did you play with database migrate versions? Because it looks like you bumped the version but didn't provide a migration file | |
| 10:37:58 | brinzhang | artom: I am not update the db | |
| 10:39:34 | gibi | Sundar: I don't see why foo and bar needs to be Cyborg related | |