Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-01-08
14:56:14 stephenfin Doesn't matter though. Not anything we're going to be tackling in the immediate future
14:57:14 sean-k-mooney1 stephenfin: it was somthing i really want to fix last/this cycle but with the company move i didnt get to work on it
14:57:43 sean-k-mooney1 there were enough depencise however that it likely could nto have been done until this cycle anyway
14:58:09 openstackgerrit Jie Li proposed openstack/nova master: Support volume-backed server rebuild in compute https://review.openstack.org/625893
15:05:31 openstackgerrit Kashyap Chamarthy proposed openstack/nova master: libvirt: Support native TLS for migration and disks over NBD https://review.openstack.org/625216
15:08:20 openstackgerrit Rui Zang proposed openstack/nova-specs master: Virtual persistent memory libvirt driver implementation https://review.openstack.org/622893
15:13:03 jaypipes maciejjozefczyk: ok, reviewed https://review.openstack.org/#/c/591607/. If you can make that small change I can re-review quickly.
15:13:24 jaypipes cdent: oh, totes. just not something that's been on my priority list :)
15:14:04 openstackgerrit Rui Zang proposed openstack/nova-specs master: support virtual persistent memory https://review.openstack.org/601596
15:14:18 maciejjozefczyk jaypipes: checking
15:14:39 cdent jaypipes: sure. i was just pointing it out mostly in a sort of "hey, we got a thing for that now!" way
15:27:32 jaypipes cdent: ack, totes.
15:27:50 jaypipes cdent: that patch I linked to stephenfin was >1 year old, after all :)
15:28:11 cdent too much water under the bridge. I feel the abyss of time.
15:29:21 jaypipes cdent: ha! :)
15:33:17 openstackgerrit Rui Zang proposed openstack/nova-specs master: Virtual persistent memory libvirt driver implementation https://review.openstack.org/622893
15:49:42 openstackgerrit Maciej Jozefczyk proposed openstack/nova master: Force refresh instance info_cache during heal https://review.openstack.org/591607
15:50:00 maciejjozefczyk jaypipes: update done, about sorting: it's kind of chicken or egg problem, which came first, row in VirtualInterface or device_id set in neutron port_data
16:01:41 jaypipes maciejjozefczyk: +2 :)
16:02:04 maciejjozefczyk jaypipes: thanks
16:02:13 jaypipes maciejjozefczyk: no problem.
16:02:16 adrianc sean-k-mooney: Hey, hope you had a good holiday. do you reckon we can converge on libvirt-sriov-live-migration spec by end of M2 ?
16:02:22 jaypipes maciejjozefczyk: thx for your patience!
16:04:41 maciejjozefczyk jaypipes: I put the helmet on because im testing it on 40k instances restored prod db
16:06:20 sean-k-mooney adrianc: i would hope so
16:06:31 sean-k-mooney adrianc: and hi o/
16:07:43 gibi kashyap: I left a question inline in ttps://review.openstack.org/#/c/625216
16:07:54 sean-k-mooney adrianc: there are currently no outstanding comments that im aware of for https://review.openstack.org/#/c/605116/ so if people have review bandwith between now and the spec freeze i hope it ill get merged
16:08:05 jaypipes maciejjozefczyk: :)
16:08:16 kashyap gibi: Thanks, will look in a few.
16:08:39 adrianc sean-k-mooney: :) Great, i believe we dont have any more open issues according to the discussions. will it help to patition for a review in the nova weekly meeting on thursday ?
16:09:02 sean-k-mooney maciejjozefczyk: im really looking forword to your chages hopefully merging soon
16:09:45 sean-k-mooney maciejjozefczyk: im less loging forward to haveing to figure out how to backport it to newton downstream but we have customer hitting issue that your patches fix
16:10:22 maciejjozefczyk sean-k-mooney: I have backport to Newton
16:10:48 sean-k-mooney maciejjozefczyk: isnt newton eol upstream
16:11:05 maciejjozefczyk but for: https://review.openstack.org/#/c/614167 without functional tests
16:11:22 maciejjozefczyk sean-k-mooney: newton is the newest release I support, I could say ;]
16:11:39 maciejjozefczyk sean-k-mooney: Let me test a bit and I can send you patches if you want
16:12:03 maciejjozefczyk but for Newton I needed to cherrry-pick about 4-5 other patches along with one more to neutron
16:12:59 sean-k-mooney maciejjozefczyk: that woudl be useful. our donwsteam backport policy still say i have to wait for it to merge on at least master upstream before i can submit it for downsteam with getting an exception for out of tree hotfix so no rush
16:13:31 maciejjozefczyk ok
16:13:54 maciejjozefczyk sean-k-mooney: i'm not that lucky, sometimes we have patches that are in review in upstream, but only when we're confident
16:15:06 sean-k-mooney if we have ci/release blocking bugs we will sometimes make an excpetion but while this is an annoying issue its not that pressing
16:15:49 maciejjozefczyk for this particular one we have about 20-30 occurrences/day
16:15:56 maciejjozefczyk on all regions
16:16:53 sean-k-mooney ouch we have only had one customer report an issue and for them it happended after a power failuer broke quoram on there glara cluster
16:22:24 stephenfin I'm seeing 'sqlalchemy.exc.NoSuchTableError: migration_tmp' when running unit tests. Any idea why?
16:22:27 stephenfin *Anyone any
16:29:38 sean-k-mooney oh by the way i figured out why the intel nfv ci is still triggering but skipping all jobs. ie why intel has not disabled it totaly
16:29:52 sean-k-mooney it is still runing jobs properly on neutron
16:30:02 sean-k-mooney they have just disabel the nova jobs.
16:47:54 jaypipes legit request?
16:47:54 jaypipes stephenfin: from your ML post about hugepages... "It's perfectly fine to have NUMA without CPU pinning, though not the other way around." <-- is that true? Wouldn't a user want to request dedicated CPU resources regardless of whether those CPUs are associated with a particular NUMA node? I mean, if they don't have a device that has NUMA affinity requirements and they don't have a need for a specific guest NUMA layout/topology, why isn't that a
16:49:06 stephenfin jaypipes: Ah, I'm talking about the nova view of things. Nova doesn't currently let you have CPU pinning without a NUMA topology
16:49:33 jaypipes ah, yes, much to my chagrin.
16:49:39 sean-k-mooney jaypipes oversubscription is perfectly fine fore cpu pinning too
16:49:46 stephenfin Whether CPU pinning without taking NUMA affinity into account is a different issue entirely
16:49:59 stephenfin sean-k-mooney: but daft, given why people are using CPU pinning
16:50:01 sean-k-mooney stephenfin: oh you were refering to the cpu pinning bit
16:50:09 sean-k-mooney needing numa toplogy
16:50:10 jaypipes stephenfin: it's just the way you wrote that, made it seem that a request for dedicated CPU resources without any NUMA affinity/topology wasn't a valid request.
16:50:19 sean-k-mooney stephenfin: no its not
16:50:34 sean-k-mooney cpu pinning means i have a compute intensive workload
16:50:45 sean-k-mooney it dows nto mean that workload is memory sencitive
16:51:06 stephenfin jaypipes: Sorry, not what I was going for. If you are replying, to that email you might clarify
16:51:19 stephenfin sean-k-mooney: Think you're crossing wires here
16:51:36 stephenfin I'm saying oversubscription for pinned CPUs is daft, i.e. allowing two instances to share the same pCPU
16:51:36 jaypipes sean-k-mooney: pls see my comment on that topic on https://review.openstack.org/#/c/599957
16:52:03 jaypipes stephenfin: not a huge deal. if I reply, I'll mention it, but was more just making sure I was understanding you properly.
16:52:14 jaypipes stephenfin: see my comment on https://review.openstack.org/#/c/599957/ :)
16:52:15 stephenfin sean-k-mooney: I have no issues with unbinding CPU pinning from NUMA topologies
16:52:17 sean-k-mooney stephenfin: that i agree with but we were talking about memory oversubsription on the email tread
16:52:37 stephenfin jaypipes: Indeed :) I recall that discussion at the PTG
16:53:11 sean-k-mooney jaypipes: ya i stand by my comments on that spec i dont think we should support that
16:53:24 stephenfin sean-k-mooney: Right, you said "oversubscription is perfectly fine fore cpu pinning too" and I thought you mean CPU oversubscription
16:53:29 stephenfin *meant
16:53:33 sean-k-mooney jaypipes: that being https://review.openstack.org/#/c/599957/
16:53:57 stephenfin sean-k-mooney: Hence my "that is daft" comment :)
16:54:32 sean-k-mooney stephenfin: no dedicated means you get the whole cpus if we ever wanted to allow pinning and over subsrciption i woudl want to different cpu_policy to indicate that
16:55:09 openstackgerrit Merged openstack/nova stable/rocky: Make compute rpcapi version calculation check all cells https://review.openstack.org/624982
16:55:15 stephenfin sean-k-mooney: yup, I agree :) Just me misinterpreting what you said
16:55:21 sean-k-mooney we named it shared and dedicated to clearly callout that one allows over subscription or cpus and the other does not so https://review.openstack.org/#/c/599957/ is a definet no form me
16:56:20 sean-k-mooney stephenfin: i just resopned to that mail again pointing out the setting hw:numa_nodes=X does not prevent swaping the memory for 4k pages too incase you did not see
16:57:00 openstackgerrit Maciej Jozefczyk proposed openstack/nova master: Add fill_virtual_interface_list online_data_migration script https://review.openstack.org/614167
16:57:01 openstackgerrit Maciej Jozefczyk proposed openstack/nova master: Force refresh instance info_cache during heal https://review.openstack.org/591607
16:57:27 maciejjozefczyk jaypipes: I did a little update after your nits, will be silky smooth now
16:57:31 maciejjozefczyk needed to go, bb ;)
17:00:39 stephenfin sean-k-mooney, jaypipes, sahid: https://bugs.launchpad.net/nova/+bug/1810977
17:00:41 openstack Launchpad bug 1810977 in OpenStack Compute (nova) "Oversubscription broken for instances with NUMA topologies" [Undecided,New]
17:01:26 stephenfin I checked and reverting the patch gets us back to where we want to be. Have a better patch ready but currently trying to figure out why UTs won't run
17:01:45 jaypipes stephenfin: I would mark that bug as affecting me but... it doesn't. :)
17:02:37 sean-k-mooney im assuming oath does not over subscirbe ram or does not use the numa retlated features
17:03:28 jaypipes sean-k-mooney: that would be correct.
17:03:48 sean-k-mooney jaypipes: that certinly simplfies life.
17:04:09 jaypipes sean-k-mooney: indeed.
17:04:43 jaypipes sean-k-mooney: for performance-sensitive things, we use Ironic and the tenant does whatever the f**k they want to.
17:05:26 sean-k-mooney ya when thats a option baremetal will always beat even the most tunned vm
17:05:39 jaypipes (which is why 90% of our infrastructure is Ironic unfortunately, because tenants assume they must have baremetal to achieve the performance they need, even though 99% of our tenants haven't actually quantified what "performance" they "need".
17:05:52 openstackgerrit Merged openstack/nova-specs master: Add PENDING vm state https://review.openstack.org/554212
17:07:01 sean-k-mooney jaypipes: ya, im so familar with the tuneing options at this point that i pick an chooes the ones that make sense when i need them

Earlier   Later