| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-09-04 | |||
| 07:08:10 | gibi | then this feels like a bug, yes. | |
| 07:08:10 | cervigni | I am using rocky | |
| 07:08:18 | cervigni | oh dear :( | |
| 07:08:36 | cervigni | should the scheduler fill a first round of the four numa nodes before overallocating? | |
| 07:09:58 | gibi | cervigni: I don't think so. If there is overallocation then the scheduler can overallocate before it fills the other numa nodes. The only rule is I know of that a server cannot overallocate against itself | |
| 07:10:24 | cervigni | ok | |
| 07:10:29 | cervigni | so I will report it as a bug | |
| 07:10:37 | cervigni | Thanks gibi | |
| 07:10:45 | cervigni | I don't know if it relates of the VGPU side | |
| 07:10:51 | cervigni | I hope not | |
| 07:11:10 | gibi | cervigni: could you reporduce the problem without the VGPU resource in the flavor? | |
| 07:11:21 | gibi | cervigni: thanks for reporting the bug | |
| 07:11:22 | cervigni | I'll do it now | |
| 07:13:38 | cervigni | gibi exactly same issue without VGPU | |
| 07:13:48 | cervigni | any idea if there is a workaround I can use while the problem is fixed? | |
| 07:22:20 | gibi | unfortunately you cannot control the numa allocation directly from the outside | |
| 07:22:54 | cervigni | https://bugs.launchpad.net/nova/+bug/1842616 | |
| 07:22:55 | openstack | Launchpad bug 1842616 in OpenStack Compute (nova) "NUMA vcpus not correctly allocated against numa regions" [Undecided,New] | |
| 07:23:02 | cervigni | I put you in the watchlist as well gibi | |
| 07:23:37 | cervigni | Ok, for the moment I will leave no NUMA affinity, but with vgpus will be very messy | |
| 07:23:50 | cervigni | Guess Stein will have the same issue? | |
| 07:26:35 | gibi | unfortunately I cannot try to reproduce it as I don't have numa + VGPU capable machine. I might get hold of a 2 numa nodes machine ... | |
| 07:26:54 | gibi | anyhow thanks for the report | |
| 07:26:57 | cervigni | VGPU is irrelevant it seems | |
| 08:01:09 | kashyap | aspiers: Catching up with the relevant conversation w/ Eric in the scrollback. | |
| 08:04:08 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: update allocation in binding profile during migrate https://review.opendev.org/656422 | |
| 08:06:58 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: prepare func test env for moving servers with bandwidth https://review.opendev.org/655109 | |
| 08:09:44 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Func test for migrate server with ports having resource request https://review.opendev.org/655113 | |
| 08:12:28 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Make _rever_allocation nested allocation aware https://review.opendev.org/676138 | |
| 08:15:14 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Support reverting migration / resize with bandwidth https://review.opendev.org/676140 | |
| 08:18:00 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Func test for migrate re-schedule with bandwidth https://review.opendev.org/676972 | |
| 08:20:50 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Support migrating SRIOV port with bandwidth https://review.opendev.org/676980 | |
| 08:23:39 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Allow migrating server with port resource request https://review.opendev.org/671497 | |
| 08:26:26 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Do not query allocations twice in finish_revert_resize https://review.opendev.org/678827 | |
| 08:27:12 | Sundar | Hi gibi, hoaw are you doing? | |
| 08:29:15 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Allow resizing server with port resource request https://review.opendev.org/679019 | |
| 08:34:22 | openstackgerrit | Adrian Chiris proposed openstack/nova master: Revert "vif: Resolve a TODO and update another" https://review.opendev.org/675776 | |
| 08:40:32 | Sundar | Re. Cyborg notifications, I noticed that, once Nova kicks off the binding of ARQs, the CYborg event comes almost immediately if no programming is involved. The event gets lost because there are no waiters: I am doing the wait in the virt driver during spawn, The logs are here: http://paste.openstack.org/show/770470/ | |
| 08:42:51 | openstackgerrit | Liang Fang proposed openstack/nova master: [WIP] Leverage openCAS cache framework for VM disks https://review.opendev.org/663542 | |
| 08:43:07 | Sundar | gibi: ^ I am proposing to move the create/bind ARQs from build_and_run_instance (https://review.opendev.org/#/c/631244/37/nova/compute/manager.py@1940) to _build_resources (https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L2392). | |
| 08:44:06 | Sundar | gibi: This has 2 benefits: (a) The instance has th right host and hostname after resource claim, and (b) if VM spawn fails, there is a good place to delete the ARQs in flight. | |
| 08:47:34 | Sundar | The problem I am facing is this: can the logic be like http://paste.openstack.org/show/770481/? If non-ARQ-related stuff is not under the with, then the with statement is going to just kick off the binding and then wait till the resolution happens. Nothing else is going on during the wait. | |
| 09:04:29 | gibi | Sundar: on a call, I have to get back to you in 15 minutes | |
| 09:11:39 | openstackgerrit | Brin Zhang proposed openstack/nova master: Add delete_on_termination to volume-attach API https://review.opendev.org/673133 | |
| 09:18:46 | gibi | Sundar: there is already a wait_for_instance_events call in nova.virt.libvirt.driver.LibvirtDriver._create_domain_and_network (called from spawn()) | |
| 09:20:12 | gibi | Sundar: or you only want to wait for the cyborg specific event in _build_resources ? | |
| 09:27:41 | gibi | Sundar: but yeah, the goal is that you kick of the binding and while you are waiting for the response let other things progress for the build up until the point where you really need the ARQs to be bound | |
| 09:30:17 | gibi | Sundar: I might try to add the with wait_for_instance_events under with _build_resources call in https://github.com/openstack/nova/blob/cbaea3bd690c683e83a063acfa69919668eaa123/nova/compute/manager.py#L2233 | |
| 09:32:12 | gibi | Sundar: I'm not sure how the the two context manager would combine if you put the wait_for_instance call inside the _build_resources without trying | |
| 09:36:50 | kashyap | Huh, this commit added the CPU instruction: https://opendev.org/openstack/nova/commit/866ca1468dfa35f319f520d0f7b93b2d0fe32f62 | |
| 09:36:56 | kashyap | (trait, i.e.) | |
| 09:37:03 | kashyap | But it does _not_ exist in os-traits yet :-( | |
| 09:37:21 | kashyap | There's no such trait as HW_CPU_X86_AVX512VNNI in 'os-traits' | |
| 09:39:01 | kashyap | Hmm, never mind ... there _is_ the trait, and it is merged: https://review.opendev.org/#/c/672888/1/os_traits/hw/cpu/x86/__init__.py | |
| 09:45:01 | aspiers | sometimes I have to pip install the latest os-traits in my tox envs | |
| 09:45:11 | aspiers | it's quicker than a --recreate | |
| 09:46:23 | gibi | I guess tox did not pick up the os-traits >= 0.16.0 change in the requirements.txt | |
| 09:46:46 | gibi | the nova master has it, maybe you need to rebase your patch | |
| 09:47:17 | aspiers | tox doesn't seem good at noticing when to update anyway | |
| 09:47:24 | aspiers | IME at least | |
| 10:09:50 | openstackgerrit | Brin Zhang proposed openstack/python-novaclient master: Microversion 2.78: Add delete_on_termination to volume-attach API https://review.opendev.org/673485 | |
| 10:14:19 | kashyap | aspiers: Yeah, I see what you mean; what I found was that in my venv: | |
| 10:14:19 | kashyap | >>> import os_traits | |
| 10:14:19 | kashyap | >>> print os_traits.HW_CPU_X86_AVX512VNNI | |
| 10:14:19 | kashyap | HW_CPU_X86_AVX512VNNI | |
| 10:14:41 | kashyap | (I _think_ I manually 'pip'-installed os-traits in there, I guess. I already forget) | |
| 10:32:13 | brinzhang | While run functional test cases, either execute tox or python -m testtools.run command, it will be raise "AssertionError: There is no script for 401 version", details http://paste.openstack.org/show/770633/ | |
| 10:32:37 | brinzhang | How can I avoid this issue? | |
| 10:35:41 | Sundar | gibi, Yes, we want to maximize the overlap by kicking off the binding as early as possible and wait for the ARQs as late as possible, but that is not easy without major surgery. So, I'll be content with dong it inside or after _build_resources. However, if we do: "with .... : foo(); bar()", then only foo() and bar() are done concurrently while wai | |
| 10:35:42 | Sundar | ting for the event, right? Also, foo and bar must both be CYborg-related. | |
| 10:36:32 | sean-k-mooney | brinzhang: "tox -e functional-py36 -- nova.tests.functional.api_sample_tests.test_quota_sets.NoopQuotaSetsSampleJsonTests.test_show_quotas" this works for me | |
| 10:36:49 | sean-k-mooney | brinzhang: you might need to add -r to recreate the tox env. | |
| 10:36:55 | gibi | Sundar: yes, only foo and bar will be parallel because when the with wait context manager exits it waits for the events | |
| 10:36:59 | brinzhang | sean-k-mooney: I will have try, thanks | |
| 10:37:13 | artom | brinzhang, did you play with database migrate versions? Because it looks like you bumped the version but didn't provide a migration file | |
| 10:37:58 | brinzhang | artom: I am not update the db | |
| 10:39:34 | gibi | Sundar: I don't see why foo and bar needs to be Cyborg related | |
| 10:39:41 | artom | brinzhang, well, that's where the problem seems to be based on the paste you linked | |
| 10:40:01 | artom | brinzhang, the stuff in nova/db/sqlalchemy/migrate_repo/versions/ | |
| 10:40:09 | brinzhang | artom: yeah, that why I am confusing | |
| 10:46:40 | brinzhang | sean-k-monney: http://paste.openstack.org/show/770638/ | |
| 10:47:32 | sean-k-mooney | brinzhang: do you have local changes? | |
| 10:47:46 | sean-k-mooney | i ran those tests against master | |
| 10:48:12 | brinzhang | Emm.. I am in https://review.opendev.org/#/c/673133/14 this patch | |
| 10:48:28 | brinzhang | I will test in master now | |
| 10:49:08 | sean-k-mooney | if you have a typo like incorrect indentaion or other syntax errors test discovery will fail | |
| 10:49:10 | brinzhang | http://paste.openstack.org/show/770638/ it's a wrong director | |
| 10:50:02 | sean-k-mooney | it will also fail if you if you have tempory filtes create by some edirtor like emacs where it adds a # to the filename | |
| 10:51:07 | Sundar | gibi, Yes, we want to maximize the overlap by kicking off the binding as early as possible and wait for the ARQs as late as possible, but that is not easy without major surgery. So, I'll be content with dong it inside or after _build_resources. However, if we do: "with .... : foo(); bar()", then only foo() and bar() are done concurrently while wait | |
| 10:51:08 | Sundar | ing for the event, right? Also, foo and bar must both be Cyborg-related. | |
| 10:51:29 | gibi | Sundar: yes, only foo and bar will be parallel because when the with wait context manager exits it waits for the events | |
| 10:51:33 | gibi | Sundar: I don't see why foo and bar needs to be Cyborg related | |
| 10:52:27 | brinzhang | sean-k-mooney: I will delete the edit patch https://review.opendev.org/#/c/673133/14, and then re-fetch it. | |
| 10:53:21 | Sundar | gibi: The timeout in the with will cover both foo and bar. Is it reasonable to impose that Cyborg ARQ bind timeout on non-Cyborg activities? it is 300 seconds by default. If it is ok, it will get us more concurrency. | |
| 11:00:13 | gibi | Sundar: the timeout is not for foo and bar it is for waiting for the events | |
| 11:01:17 | gibi | Sundar: also if the boot process cannot procede without such events then it is OK to wait for the events | |
| 11:03:50 | Sundar | gibi: What do you think of waiting for the events in _build_resources, but actually querying Cyborg for ARQs only in the virt driver's spawn? If we do the query when the event comes, we have to store the ARQs in the instance object (or elsewhere), which will require changes to objects and db. | |
| 11:05:16 | gibi | Sundar: I think it is better to wait earlier than potenitally loosing notifications as we start waiting too late | |
| 11:05:46 | gibi | Sundar: I think you should also ask others, like dansmith about this problem | |
| 11:07:27 | gibi | Sundar: if you start waiting in _build_resources then I would do the query in the virt driver instead of passing the ARQ down | |