Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-01-08
14:53:11 stephenfin sean-k-mooney1: Not willingly - I dragged him into it slightly ;)
14:53:37 sean-k-mooney1 stephenfin: hugepages in placement is part of sylivans numa in placement spec i think
14:54:17 sean-k-mooney1 stephenfin: we have talked about it many times in the past and it was one of the main usecase we created nested resouce provirders for
14:55:03 sean-k-mooney1 stephenfin: infact it was higher on or prioity list for nested provider which is why i was surpised we were leading with vgpus in denver
14:55:12 stephenfin sean-k-mooney1: Possibly, I haven't checked
14:55:53 stephenfin sean-k-mooney1: Though I'd been thinking we'd report pagesizes on both the root provider and the numa cell providers, but that wouldn't work because they're the same thing viewed in different ways
14:56:14 stephenfin Doesn't matter though. Not anything we're going to be tackling in the immediate future
14:57:14 sean-k-mooney1 stephenfin: it was somthing i really want to fix last/this cycle but with the company move i didnt get to work on it
14:57:43 sean-k-mooney1 there were enough depencise however that it likely could nto have been done until this cycle anyway
14:58:09 openstackgerrit Jie Li proposed openstack/nova master: Support volume-backed server rebuild in compute https://review.openstack.org/625893
15:05:31 openstackgerrit Kashyap Chamarthy proposed openstack/nova master: libvirt: Support native TLS for migration and disks over NBD https://review.openstack.org/625216
15:08:20 openstackgerrit Rui Zang proposed openstack/nova-specs master: Virtual persistent memory libvirt driver implementation https://review.openstack.org/622893
15:13:03 jaypipes maciejjozefczyk: ok, reviewed https://review.openstack.org/#/c/591607/. If you can make that small change I can re-review quickly.
15:13:24 jaypipes cdent: oh, totes. just not something that's been on my priority list :)
15:14:04 openstackgerrit Rui Zang proposed openstack/nova-specs master: support virtual persistent memory https://review.openstack.org/601596
15:14:18 maciejjozefczyk jaypipes: checking
15:14:39 cdent jaypipes: sure. i was just pointing it out mostly in a sort of "hey, we got a thing for that now!" way
15:27:32 jaypipes cdent: ack, totes.
15:27:50 jaypipes cdent: that patch I linked to stephenfin was >1 year old, after all :)
15:28:11 cdent too much water under the bridge. I feel the abyss of time.
15:29:21 jaypipes cdent: ha! :)
15:33:17 openstackgerrit Rui Zang proposed openstack/nova-specs master: Virtual persistent memory libvirt driver implementation https://review.openstack.org/622893
15:49:42 openstackgerrit Maciej Jozefczyk proposed openstack/nova master: Force refresh instance info_cache during heal https://review.openstack.org/591607
15:50:00 maciejjozefczyk jaypipes: update done, about sorting: it's kind of chicken or egg problem, which came first, row in VirtualInterface or device_id set in neutron port_data
16:01:41 jaypipes maciejjozefczyk: +2 :)
16:02:04 maciejjozefczyk jaypipes: thanks
16:02:13 jaypipes maciejjozefczyk: no problem.
16:02:16 adrianc sean-k-mooney: Hey, hope you had a good holiday. do you reckon we can converge on libvirt-sriov-live-migration spec by end of M2 ?
16:02:22 jaypipes maciejjozefczyk: thx for your patience!
16:04:41 maciejjozefczyk jaypipes: I put the helmet on because im testing it on 40k instances restored prod db
16:06:20 sean-k-mooney adrianc: i would hope so
16:06:31 sean-k-mooney adrianc: and hi o/
16:07:43 gibi kashyap: I left a question inline in ttps://review.openstack.org/#/c/625216
16:07:54 sean-k-mooney adrianc: there are currently no outstanding comments that im aware of for https://review.openstack.org/#/c/605116/ so if people have review bandwith between now and the spec freeze i hope it ill get merged
16:08:05 jaypipes maciejjozefczyk: :)
16:08:16 kashyap gibi: Thanks, will look in a few.
16:08:39 adrianc sean-k-mooney: :) Great, i believe we dont have any more open issues according to the discussions. will it help to patition for a review in the nova weekly meeting on thursday ?
16:09:02 sean-k-mooney maciejjozefczyk: im really looking forword to your chages hopefully merging soon
16:09:45 sean-k-mooney maciejjozefczyk: im less loging forward to haveing to figure out how to backport it to newton downstream but we have customer hitting issue that your patches fix
16:10:22 maciejjozefczyk sean-k-mooney: I have backport to Newton
16:10:48 sean-k-mooney maciejjozefczyk: isnt newton eol upstream
16:11:05 maciejjozefczyk but for: https://review.openstack.org/#/c/614167 without functional tests
16:11:22 maciejjozefczyk sean-k-mooney: newton is the newest release I support, I could say ;]
16:11:39 maciejjozefczyk sean-k-mooney: Let me test a bit and I can send you patches if you want
16:12:03 maciejjozefczyk but for Newton I needed to cherrry-pick about 4-5 other patches along with one more to neutron
16:12:59 sean-k-mooney maciejjozefczyk: that woudl be useful. our donwsteam backport policy still say i have to wait for it to merge on at least master upstream before i can submit it for downsteam with getting an exception for out of tree hotfix so no rush
16:13:31 maciejjozefczyk ok
16:13:54 maciejjozefczyk sean-k-mooney: i'm not that lucky, sometimes we have patches that are in review in upstream, but only when we're confident
16:15:06 sean-k-mooney if we have ci/release blocking bugs we will sometimes make an excpetion but while this is an annoying issue its not that pressing
16:15:49 maciejjozefczyk for this particular one we have about 20-30 occurrences/day
16:15:56 maciejjozefczyk on all regions
16:16:53 sean-k-mooney ouch we have only had one customer report an issue and for them it happended after a power failuer broke quoram on there glara cluster
16:22:24 stephenfin I'm seeing 'sqlalchemy.exc.NoSuchTableError: migration_tmp' when running unit tests. Any idea why?
16:22:27 stephenfin *Anyone any
16:29:38 sean-k-mooney oh by the way i figured out why the intel nfv ci is still triggering but skipping all jobs. ie why intel has not disabled it totaly
16:29:52 sean-k-mooney it is still runing jobs properly on neutron
16:30:02 sean-k-mooney they have just disabel the nova jobs.
16:47:54 jaypipes stephenfin: from your ML post about hugepages... "It's perfectly fine to have NUMA without CPU pinning, though not the other way around." <-- is that true? Wouldn't a user want to request dedicated CPU resources regardless of whether those CPUs are associated with a particular NUMA node? I mean, if they don't have a device that has NUMA affinity requirements and they don't have a need for a specific guest NUMA layout/topology, why isn't that a
16:47:54 jaypipes legit request?
16:49:06 stephenfin jaypipes: Ah, I'm talking about the nova view of things. Nova doesn't currently let you have CPU pinning without a NUMA topology
16:49:33 jaypipes ah, yes, much to my chagrin.
16:49:39 sean-k-mooney jaypipes oversubscription is perfectly fine fore cpu pinning too
16:49:46 stephenfin Whether CPU pinning without taking NUMA affinity into account is a different issue entirely
16:49:59 stephenfin sean-k-mooney: but daft, given why people are using CPU pinning
16:50:01 sean-k-mooney stephenfin: oh you were refering to the cpu pinning bit
16:50:09 sean-k-mooney needing numa toplogy
16:50:10 jaypipes stephenfin: it's just the way you wrote that, made it seem that a request for dedicated CPU resources without any NUMA affinity/topology wasn't a valid request.
16:50:19 sean-k-mooney stephenfin: no its not
16:50:34 sean-k-mooney cpu pinning means i have a compute intensive workload
16:50:45 sean-k-mooney it dows nto mean that workload is memory sencitive
16:51:06 stephenfin jaypipes: Sorry, not what I was going for. If you are replying, to that email you might clarify
16:51:19 stephenfin sean-k-mooney: Think you're crossing wires here
16:51:36 jaypipes sean-k-mooney: pls see my comment on that topic on https://review.openstack.org/#/c/599957
16:51:36 stephenfin I'm saying oversubscription for pinned CPUs is daft, i.e. allowing two instances to share the same pCPU
16:52:03 jaypipes stephenfin: not a huge deal. if I reply, I'll mention it, but was more just making sure I was understanding you properly.
16:52:14 jaypipes stephenfin: see my comment on https://review.openstack.org/#/c/599957/ :)
16:52:15 stephenfin sean-k-mooney: I have no issues with unbinding CPU pinning from NUMA topologies
16:52:17 sean-k-mooney stephenfin: that i agree with but we were talking about memory oversubsription on the email tread
16:52:37 stephenfin jaypipes: Indeed :) I recall that discussion at the PTG
16:53:11 sean-k-mooney jaypipes: ya i stand by my comments on that spec i dont think we should support that
16:53:24 stephenfin sean-k-mooney: Right, you said "oversubscription is perfectly fine fore cpu pinning too" and I thought you mean CPU oversubscription
16:53:29 stephenfin *meant
16:53:33 sean-k-mooney jaypipes: that being https://review.openstack.org/#/c/599957/
16:53:57 stephenfin sean-k-mooney: Hence my "that is daft" comment :)
16:54:32 sean-k-mooney stephenfin: no dedicated means you get the whole cpus if we ever wanted to allow pinning and over subsrciption i woudl want to different cpu_policy to indicate that
16:55:09 openstackgerrit Merged openstack/nova stable/rocky: Make compute rpcapi version calculation check all cells https://review.openstack.org/624982
16:55:15 stephenfin sean-k-mooney: yup, I agree :) Just me misinterpreting what you said
16:55:21 sean-k-mooney we named it shared and dedicated to clearly callout that one allows over subscription or cpus and the other does not so https://review.openstack.org/#/c/599957/ is a definet no form me
16:56:20 sean-k-mooney stephenfin: i just resopned to that mail again pointing out the setting hw:numa_nodes=X does not prevent swaping the memory for 4k pages too incase you did not see
16:57:00 openstackgerrit Maciej Jozefczyk proposed openstack/nova master: Add fill_virtual_interface_list online_data_migration script https://review.openstack.org/614167
16:57:01 openstackgerrit Maciej Jozefczyk proposed openstack/nova master: Force refresh instance info_cache during heal https://review.openstack.org/591607
16:57:27 maciejjozefczyk jaypipes: I did a little update after your nits, will be silky smooth now
16:57:31 maciejjozefczyk needed to go, bb ;)
17:00:39 stephenfin sean-k-mooney, jaypipes, sahid: https://bugs.launchpad.net/nova/+bug/1810977
17:00:41 openstack Launchpad bug 1810977 in OpenStack Compute (nova) "Oversubscription broken for instances with NUMA topologies" [Undecided,New]
17:01:26 stephenfin I checked and reverting the patch gets us back to where we want to be. Have a better patch ready but currently trying to figure out why UTs won't run
17:01:45 jaypipes stephenfin: I would mark that bug as affecting me but... it doesn't. :)
17:02:37 sean-k-mooney im assuming oath does not over subscirbe ram or does not use the numa retlated features
17:03:28 jaypipes sean-k-mooney: that would be correct.
17:03:48 sean-k-mooney jaypipes: that certinly simplfies life.

Earlier   Later