Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-09-04
08:40:32 Sundar Re. Cyborg notifications, I noticed that, once Nova kicks off the binding of ARQs, the CYborg event comes almost immediately if no programming is involved. The event gets lost because there are no waiters: I am doing the wait in the virt driver during spawn, The logs are here: http://paste.openstack.org/show/770470/
08:42:51 openstackgerrit Liang Fang proposed openstack/nova master: [WIP] Leverage openCAS cache framework for VM disks https://review.opendev.org/663542
08:43:07 Sundar gibi: ^ I am proposing to move the create/bind ARQs from build_and_run_instance (https://review.opendev.org/#/c/631244/37/nova/compute/manager.py@1940) to _build_resources (https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L2392).
08:44:06 Sundar gibi: This has 2 benefits: (a) The instance has th right host and hostname after resource claim, and (b) if VM spawn fails, there is a good place to delete the ARQs in flight.
08:47:34 Sundar The problem I am facing is this: can the logic be like http://paste.openstack.org/show/770481/? If non-ARQ-related stuff is not under the with, then the with statement is going to just kick off the binding and then wait till the resolution happens. Nothing else is going on during the wait.
09:04:29 gibi Sundar: on a call, I have to get back to you in 15 minutes
09:11:39 openstackgerrit Brin Zhang proposed openstack/nova master: Add delete_on_termination to volume-attach API https://review.opendev.org/673133
09:18:46 gibi Sundar: there is already a wait_for_instance_events call in nova.virt.libvirt.driver.LibvirtDriver._create_domain_and_network (called from spawn())
09:20:12 gibi Sundar: or you only want to wait for the cyborg specific event in _build_resources ?
09:27:41 gibi Sundar: but yeah, the goal is that you kick of the binding and while you are waiting for the response let other things progress for the build up until the point where you really need the ARQs to be bound
09:30:17 gibi Sundar: I might try to add the with wait_for_instance_events under with _build_resources call in https://github.com/openstack/nova/blob/cbaea3bd690c683e83a063acfa69919668eaa123/nova/compute/manager.py#L2233
09:32:12 gibi Sundar: I'm not sure how the the two context manager would combine if you put the wait_for_instance call inside the _build_resources without trying
09:36:50 kashyap Huh, this commit added the CPU instruction: https://opendev.org/openstack/nova/commit/866ca1468dfa35f319f520d0f7b93b2d0fe32f62
09:36:56 kashyap (trait, i.e.)
09:37:03 kashyap But it does _not_ exist in os-traits yet :-(
09:37:21 kashyap There's no such trait as HW_CPU_X86_AVX512VNNI in 'os-traits'
09:39:01 kashyap Hmm, never mind ... there _is_ the trait, and it is merged: https://review.opendev.org/#/c/672888/1/os_traits/hw/cpu/x86/__init__.py
09:45:01 aspiers sometimes I have to pip install the latest os-traits in my tox envs
09:45:11 aspiers it's quicker than a --recreate
09:46:23 gibi I guess tox did not pick up the os-traits >= 0.16.0 change in the requirements.txt
09:46:46 gibi the nova master has it, maybe you need to rebase your patch
09:47:17 aspiers tox doesn't seem good at noticing when to update anyway
09:47:24 aspiers IME at least
10:09:50 openstackgerrit Brin Zhang proposed openstack/python-novaclient master: Microversion 2.78: Add delete_on_termination to volume-attach API https://review.opendev.org/673485
10:14:19 kashyap HW_CPU_X86_AVX512VNNI
10:14:19 kashyap >>> print os_traits.HW_CPU_X86_AVX512VNNI
10:14:19 kashyap >>> import os_traits
10:14:19 kashyap aspiers: Yeah, I see what you mean; what I found was that in my venv:
10:14:41 kashyap (I _think_ I manually 'pip'-installed os-traits in there, I guess. I already forget)
10:32:13 brinzhang While run functional test cases, either execute tox or python -m testtools.run command, it will be raise "AssertionError: There is no script for 401 version", details http://paste.openstack.org/show/770633/
10:32:37 brinzhang How can I avoid this issue?
10:35:41 Sundar gibi, Yes, we want to maximize the overlap by kicking off the binding as early as possible and wait for the ARQs as late as possible, but that is not easy without major surgery. So, I'll be content with dong it inside or after _build_resources. However, if we do: "with .... : foo(); bar()", then only foo() and bar() are done concurrently while wai
10:35:42 Sundar ting for the event, right? Also, foo and bar must both be CYborg-related.
10:36:32 sean-k-mooney brinzhang: "tox -e functional-py36 -- nova.tests.functional.api_sample_tests.test_quota_sets.NoopQuotaSetsSampleJsonTests.test_show_quotas" this works for me
10:36:49 sean-k-mooney brinzhang: you might need to add -r to recreate the tox env.
10:36:55 gibi Sundar: yes, only foo and bar will be parallel because when the with wait context manager exits it waits for the events
10:36:59 brinzhang sean-k-mooney: I will have try, thanks
10:37:13 artom brinzhang, did you play with database migrate versions? Because it looks like you bumped the version but didn't provide a migration file
10:37:58 brinzhang artom: I am not update the db
10:39:34 gibi Sundar: I don't see why foo and bar needs to be Cyborg related
10:39:41 artom brinzhang, well, that's where the problem seems to be based on the paste you linked
10:40:01 artom brinzhang, the stuff in nova/db/sqlalchemy/migrate_repo/versions/
10:40:09 brinzhang artom: yeah, that why I am confusing
10:46:40 brinzhang sean-k-monney: http://paste.openstack.org/show/770638/
10:47:32 sean-k-mooney brinzhang: do you have local changes?
10:47:46 sean-k-mooney i ran those tests against master
10:48:12 brinzhang Emm.. I am in https://review.opendev.org/#/c/673133/14 this patch
10:48:28 brinzhang I will test in master now
10:49:08 sean-k-mooney if you have a typo like incorrect indentaion or other syntax errors test discovery will fail
10:49:10 brinzhang http://paste.openstack.org/show/770638/ it's a wrong director
10:50:02 sean-k-mooney it will also fail if you if you have tempory filtes create by some edirtor like emacs where it adds a # to the filename
10:51:07 Sundar gibi, Yes, we want to maximize the overlap by kicking off the binding as early as possible and wait for the ARQs as late as possible, but that is not easy without major surgery. So, I'll be content with dong it inside or after _build_resources. However, if we do: "with .... : foo(); bar()", then only foo() and bar() are done concurrently while wait
10:51:08 Sundar ing for the event, right? Also, foo and bar must both be Cyborg-related.
10:51:29 gibi Sundar: yes, only foo and bar will be parallel because when the with wait context manager exits it waits for the events
10:51:33 gibi Sundar: I don't see why foo and bar needs to be Cyborg related
10:52:27 brinzhang sean-k-mooney: I will delete the edit patch https://review.opendev.org/#/c/673133/14, and then re-fetch it.
10:53:21 Sundar gibi: The timeout in the with will cover both foo and bar. Is it reasonable to impose that Cyborg ARQ bind timeout on non-Cyborg activities? it is 300 seconds by default. If it is ok, it will get us more concurrency.
11:00:13 gibi Sundar: the timeout is not for foo and bar it is for waiting for the events
11:01:17 gibi Sundar: also if the boot process cannot procede without such events then it is OK to wait for the events
11:03:50 Sundar gibi: What do you think of waiting for the events in _build_resources, but actually querying Cyborg for ARQs only in the virt driver's spawn? If we do the query when the event comes, we have to store the ARQs in the instance object (or elsewhere), which will require changes to objects and db.
11:05:16 gibi Sundar: I think it is better to wait earlier than potenitally loosing notifications as we start waiting too late
11:05:46 gibi Sundar: I think you should also ask others, like dansmith about this problem
11:07:27 gibi Sundar: if you start waiting in _build_resources then I would do the query in the virt driver instead of passing the ARQ down
11:07:38 gibi Sundar: to limit the impact
11:07:49 Sundar gibi: Great. That's what I had in mind. :)
11:08:26 Sundar Thank you very much, gibi.
11:08:37 Sundar Have a good day.
11:08:42 gibi Sundar: same to you
11:14:43 sean-k-mooney gibi: would you have time to review https://review.opendev.org/#/q/topic:bp/image-metadata-prefiltering+(status:open+OR+status:merged) today? im aiming to get that merged and free up a runway slot by the end of the week
11:15:05 sean-k-mooney the 3 remaining pataches are relitivly short
11:15:30 gibi sean-k-mooney: I will try...
11:16:48 sean-k-mooney gibi: thanks if you dont get to it its ok i have others to bug too like bauzas
11:17:50 sean-k-mooney bauzas: feel like re reviewing https://review.opendev.org/#/q/topic:bp/image-metadata-prefiltering+(status:open)
11:20:48 kashyap aspiers: gibi: Commented here: https://review.opendev.org/#/c/644565/49/nova/virt/libvirt/utils.py@546
11:21:25 sean-k-mooney artom: i see you currently dont have any ci jobs running using the multi numa job so im going to swap it over too the new lables for FN
11:21:42 artom sean-k-mooney, sure
11:21:45 sean-k-mooney im also going to reduce the concurance to 1
11:22:24 artom sean-k-mooney, unless I'm missing something, the ball's in the reviewers's court
11:22:39 sean-k-mooney assuming you are happy with them for testingin im going to then swap to testing the cpu work stephen has been doinging
11:22:55 gibi kashyap: thanks!
11:23:17 artom Though I need to play with the func test some more - the RPC pinning tests didn't pick up a thing they should have picked up
11:23:39 artom sean-k-mooney, yep, that's fair, again much thanks for all your help
11:24:04 sean-k-mooney thats the last patch in the seriese right e.g. it wont block other patches landing
11:24:16 artom Func tests? Don't think so
11:24:27 artom We can still land tests after FF, right?
11:24:34 sean-k-mooney yes
11:24:41 artom Even assuming the worst and don't get them sorted before then
11:24:48 sean-k-mooney although its better to do it before
11:24:57 artom That's the plan :)
11:32:38 brinzhang sean-k-mooney, artom: I remember the unit test or functional test can run any linux env, depend on the devstack or openstack env is not necessary, is that?
11:32:56 sean-k-mooney brinzhang: correct
11:33:06 sean-k-mooney you do not need openstack or devstack to run them
11:33:08 brinzhang sean-k-mooney, artom: I checked to the master and run the test, also report the error
11:33:27 sean-k-mooney what os are you running on?
11:33:34 brinzhang Centos7
11:33:48 sean-k-mooney brinzhang: you might want to clone a clean copy of nova and try it again
11:34:11 sean-k-mooney brinzhang: ok i tested on popos and it worked fine
11:34:19 sean-k-mooney i can check on rhel8
11:34:31 brinzhang On Monday, It runs ok, but today I git pull
11:35:00 sean-k-mooney i know some test fail in rhel8 but i think that is due to the version of python it uses

Earlier   Later