Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-10-22
12:38:47 jaypipes np! :)
12:39:01 jaypipes aspiers: sorry, was out at a $work thing all last week :(
12:39:10 aspiers np at all
12:39:17 aspiers I had enough other feedback to keep me busy ;-)
12:39:23 jaypipes rock on :)
12:40:05 sapd1 Hi everyone, I get this errror when live migrate instance. http://paste.openstack.org/show/732581/
12:40:18 sapd1 I upgraded from queens to rocky
12:40:46 sapd1 libvirt: 3.6 and qemu 2.11
13:08:45 openstackgerrit Takashi NATSUME proposed openstack/nova master: Transform compute_task notifications https://review.openstack.org/482629
13:29:23 openstackgerrit Takashi NATSUME proposed openstack/nova stable/rocky: Remove unnecessary redirect https://review.openstack.org/607400
13:29:56 openstackgerrit Takashi NATSUME proposed openstack/nova stable/queens: Remove unnecessary redirect https://review.openstack.org/607402
13:30:51 openstackgerrit Adam Spiers proposed openstack/nova-specs master: Add spec for libvirt driver launching AMD SEV-encrypted instances https://review.openstack.org/609779
13:31:00 aspiers jaypipes: ^^^
13:39:02 openstackgerrit Matthew Edmonds proposed openstack/nova master: Use tempfile for powervm config drive https://review.openstack.org/610174
13:40:05 jaypipes aspiers: rock on.
13:43:19 openstackgerrit Markus Hentsch proposed openstack/nova-specs master: Spec for the Nova part of Image Encryption https://review.openstack.org/608696
13:55:56 efried n-sch meeting in 5 minutes in #openstack-meeting-alt
14:03:06 mriedem alex_xu: you should probably take a look at this https://review.openstack.org/#/c/609591/
14:03:55 stephenfin mriedem: Could you take a look at this bug fix series this week? https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:master+topic:bug/1784353
14:04:10 alex_xu mriedem: got it, will take a look
14:04:44 mriedem stephenfin: yeah
14:04:49 stephenfin ta
14:05:44 sapd1 Does anyone get this problem when upgrade from queens to rocky?
14:10:42 s10 sapd1: looks like this issue is related to the upgrade to the qemu 2.11, not nova.
14:11:43 sapd1 s10: I have some compute nodes which are running qemu 2.11 and it gets same issue.
14:12:43 sapd1 Does nova-compute require higher qemu/libvirt version/
14:13:09 sapd1 sorry s/2.11/2.10/
14:13:52 mriedem johnthetubaguy: you might care about this https://review.openstack.org/#/c/609043/
14:26:46 mriedem stephenfin: these are pretty easy https://review.openstack.org/#/q/topic:bug/1763051+(status:open+OR+status:merged) - first is a refactor, second is a try/catch/log but don't blow up sort of thing
14:27:33 mriedem oh sorry, latter is try/catch/rollback
14:27:45 mriedem was mixing that up with a similar live migration error handling patch
14:27:58 stephenfin mriedem: Sounds good
14:28:15 stephenfin mriedem: Also, I took a look at those debug logs you reworked for the issue awaugama and I are seeing. +2'd the first but do you think the second one should be merged now? Comments are confusing/misleading
14:28:26 mriedem i believe your employer cares about those live migration ones
14:29:23 stephenfin This one (efried got to it too) https://review.openstack.org/#/c/597560/
14:29:33 mriedem what is confusing/misleading?
14:29:47 mriedem i also mostly restored/cleaned that up for awaugama to recreate the problem
14:29:59 mriedem if it's not helpful then i don't care if we don't use it
14:30:02 stephenfin "I don't really intend these overly verbose logs to get merged." doesn't apply anymore, right?
14:30:39 mriedem compare PS7 to PS1
14:30:46 mriedem it's a much slimmer version
14:31:29 stephenfin ack
14:31:38 efried mriedem: See comments, I still think it's going to be excessive.
14:32:04 efried but yeah, I didn't vote because I wasn't sure if we were now intending to actually merge it or if it was still just for people to debug with.
14:33:25 mriedem i just replied
14:33:40 mriedem if you want to wait for awaugama to recreate and say, yes this helped, then let's wait
14:34:20 awaugama owalsh: can you help with pulling in a patch from upstream into a build for extra logging?
14:39:31 openstackgerrit Ivaylo Mitev proposed openstack/nova master: VMware: OVA and StrOpt images as VM templates https://review.openstack.org/609736
15:04:40 mriedem stephenfin: lyarwood: https://review.openstack.org/#/c/587013/15/nova/tests/fixtures.py@1728
15:04:43 mriedem am i missing something?
15:23:43 melwitt
16:02:22 stephenfin mriedem: Yeah, but I've confused you in addressing your nit here https://review.openstack.org/#/c/587013/12/nova/tests/fixtures.py@1721
16:02:51 stephenfin mriedem: _attachment_id on that line is actually the two item tuple
16:03:13 stephenfin mriedem: See https://review.openstack.org/#/c/587013/15/nova/tests/fixtures.py@1711
16:04:17 openstackgerrit sean mooney proposed openstack/nova master: harden placement init under wsgi https://review.openstack.org/610034
16:04:31 stephenfin and two lines above that, where we explode it out into the two item tuple. I guess that should go back to be called 'attachment'
16:07:15 mriedem ok commented
16:07:26 mriedem i'd like that re-worked because the variable naming is really confusing
16:08:24 stephenfin Agreed. It was called 'attachment' before but I guess that isn't much better
16:20:54 openstackgerrit Merged openstack/nova master: Remove more code related to extensions and testing https://review.openstack.org/607088
16:21:01 openstackgerrit Merged openstack/nova master: Log the operation when updating generation in ProviderTree https://review.openstack.org/597553
16:21:19 openstackgerrit Stephen Finucane proposed openstack/nova master: fixtures: Track volume attachments within CinderFixtureNewAttachFlow https://review.openstack.org/587013
16:21:20 openstackgerrit Stephen Finucane proposed openstack/nova master: Add regression test for bug#1784353 https://review.openstack.org/587014
16:21:20 openstackgerrit Stephen Finucane proposed openstack/nova master: conductor: Recreate volume attachments during a reschedule https://review.openstack.org/587071
16:39:12 openstackgerrit sean mooney proposed openstack/nova master: harden placement init under wsgi https://review.openstack.org/610034
16:41:07 openstackgerrit Matthew Booth proposed openstack/nova master: Add regression test for bug 1550919 https://review.openstack.org/591733
16:41:07 openstack bug 1550919 in OpenStack Compute (nova) "[Libvirt]Evacuate fail may cause disk image be deleted" [Medium,In progress] https://launchpad.net/bugs/1550919 - Assigned to Matthew Booth (mbooth-9)
16:43:38 mnaser fyi -- if anyone from the nova team hasn't seen this (yet): http://lists.openstack.org/pipermail/openstack-dev/2018-October/135941.html
16:43:43 openstackgerrit Matt Riedemann proposed openstack/nova master: Fix jsonutils.to_primitive UserWarning https://review.openstack.org/612445
16:44:06 spatel sean-k-mooney: you there?
16:45:05 openstackgerrit Matt Riedemann proposed openstack/nova master: Fix jsonutils.to_primitive UserWarning https://review.openstack.org/612445
16:47:18 sean-k-mooney spatel: i was away for a minute but im back
16:47:49 spatel On 64GB compute node i have configure huge page "hugepagesz=2M hugepages=27000 transparent_hugepage=never"
16:48:05 spatel based on calculation around 55GB huge pages
16:48:34 spatel is that ok or i should squeeze more pages and keep 4GB for host only
16:49:38 sean-k-mooney i think that is ok you could squees it down to 6 or 4GB for the host but without data to support that it does not need more i would start with leaveing 8GB free
16:50:22 spatel okay! let me go with 8GB free for host
16:51:21 sean-k-mooney spatel: "wasting" 2-4GB by not allocating extra hugepages is still proably better then having the OOM killer trigger because you were too agressive
16:51:48 spatel totally valid point ++
16:52:07 sean-k-mooney as you start to operate the cluster over time and you mesure/log the ram usage you will have data that can let you make a more informed desision
16:52:52 spatel also if i run some agent for data collection or collected / snmp etc.. they will suck some memory too
16:53:02 sean-k-mooney yep
16:53:35 sean-k-mooney 4Gs might be fine without loging but adding collectd or someing else might bump it to 5Gs
16:53:59 spatel see its alway helpful to have second set of eye :)
16:54:47 spatel Thanks!
16:54:59 spatel i am going with 8G everywhere now!
16:55:52 sean-k-mooney leaving 8G free on a host with 32GB is a little painful but 8G on a host with 256GB of ram is perfectly resonable. finding a blance will depend on your workload so you will figure out what makes sense once you deploy and mesure your workload
16:56:54 spatel yup! i am slowly learning all equation and tuning compute nodes based on hardware specs
16:57:07 spatel Do you disable "specter" in kernel ?
16:57:19 spatel I heard it hit you CPU performance
16:57:32 sean-k-mooney disable the mitigations or disable the speculation
16:57:58 sean-k-mooney spatel: that really depends on your threat model.
16:58:06 spatel This is what i am doing on my compute nodes "spectre_v2=off nopti pti=off nospec_store_bypass_disable spec_store_bypass_disable=off l1tf=off"
16:59:12 sean-k-mooney spatel: that will give you more performance but also mean your system is vulnerbale to specter attacks
16:59:28 sean-k-mooney spatel: that said you have a private cloud correct
16:59:47 spatel yes totally private and we are not holding any PI data of any personal
16:59:53 sean-k-mooney spatel: so you controll all application that are running on the cloud e.g. there are no untrsed applications
16:59:59 spatel we are just streaming voip data
17:00:01 mriedem mnaser: replied
17:00:47 sean-k-mooney spatel: well if the lack of mitigations is fine based on your treat model and you know the implications of truning them off then that is your choice to make.

Earlier   Later