Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-02-03
14:37:15 lyarwood sean-k-mooney: I think that came up on the ML recently and I asked for logs
14:37:15 efried sean-k-mooney: Am I to understand that blockdiag accepted your fix?
14:37:23 lyarwood ignaziocassano1: can you write that up in a bug with logs?
14:37:44 sean-k-mooney efried: they merged a similar one that does the same thing but also updates the change log
14:37:48 sean-k-mooney so basically yes
14:38:12 efried sean-k-mooney: okay, cool, so we should blacklist 2.0.0 in u-c and accept, what, 2.0.1 once they cut it?
14:38:28 sean-k-mooney ya basically
14:38:49 sean-k-mooney we can blacklist it now if we want too but i dont know what the eta is on the 2.0.1 release is
14:39:15 efried Well, we know 2.0.0 is busted, but it also doesn't seem to be affecting anyone but us, so...
14:39:22 sean-k-mooney oh its already done
14:40:56 sean-k-mooney https://github.com/blockdiag/blockdiag/compare/2.0.0...2.0.1
14:40:58 ignaziocassano1 lyqrwood: I am using centos 7.....how can open a bug ? Which site must I use ?
14:41:20 sean-k-mooney efried: want me to submit the patch to the requiremetns repo
14:42:00 lyarwood ignaziocassano1: https://bugs.launchpad.net/os-brick/+filebug - if you could include DEBUG logs showing the volume attachment and attempt to detach that would be super useful
14:42:17 ignaziocassano1 thanks
14:44:26 openstackgerrit Stephen Finucane proposed openstack/nova master: nova-net: Remove unused nova-network objects https://review.opendev.org/697156
14:44:27 openstackgerrit Stephen Finucane proposed openstack/nova master: nova-net: Remove unused parameters https://review.opendev.org/703974
14:45:20 efried sean-k-mooney: sure
14:45:51 sean-k-mooney ok ill do it shortly once i finish my current review
14:47:09 sean-k-mooney efried: did we already merge stephens fix
14:47:19 efried yes
14:47:36 efried sean-k-mooney: https://review.opendev.org/#/c/705263/
14:48:06 sean-k-mooney cool im just going to blacklist it in gr we could raise the min version in nova if we want to removed the monkey patching seperatly
14:48:36 sean-k-mooney im not going to raise the lower constratin in the requirements repo.
14:50:00 ignaziocassano1 lyarwood: I opened a bug. Did you know if I will receive email when someone will work on it ?
14:50:33 lyarwood ignaziocassano1: can you share the bug URL here?
14:50:46 ignaziocassano1 https://bugs.launchpad.net/os-brick/+bug/1861691
14:50:46 openstack Launchpad bug 1861691 in os-brick "openstack queens emc iscsi errors migrating volume and or detaching volumes" [Undecided,New]
14:51:01 ignaziocassano1 I attached the nova-compute log
14:51:44 ignaziocassano1 mmm I must change the description
14:52:41 ignaziocassano1 It happens when I migrate a vm or detach a volume
14:53:25 lyarwood ignaziocassano1: do you have any logs showing the volume being attached?
14:54:36 ignaziocassano1 I included the nova-compute log ...the problem occurs non kvm node. I also have the volume log. Dow you want I attach it ?
14:57:17 ignaziocassano1 When I migrate a vm , os-brick try to use the command "multipath -f xxxxxxxx" to flusj the device. As reported in nova compute log, it finds "map in use"
14:58:21 sean-k-mooney efried: oh its already blacklisted
14:58:29 efried cool
14:58:56 sean-k-mooney im not sure we want the upper constratis file to cap it at 2.0.1 however
14:59:08 ignaziocassano1 So, often the vm migrates but remains in error state and horizon shows the vm on old node
15:00:52 lyarwood ignaziocassano1: this channel isn't really for support but yeah `map in use` errors suggest the multipath device is being used by multiple instances, normally due to c-vol or os-brick bugs
15:01:08 lyarwood ignaziocassano1: make sure both nova, os-brick and cinder are fully updated
15:01:43 ignaziocassano1 lyarwood: they are fully updated with last queens released on centos
15:02:00 dansmith efried: I know you have other stuff going on which is probably delaying your ability to run through the cyborg patches. If you could at least hit the profile groups thing, that would be helpful: https://review.opendev.org/#/c/631243
15:02:02 efried sean-k-mooney: certainly not until that's releaced
15:02:09 efried dansmith: ack
15:02:30 dansmith efried: knowing that latest version of that matches your expectation and what should be done is important
15:02:33 dansmith thanks
15:03:21 dansmith sean-k-mooney: last I heard you had deployed the cyborg set in devstack and had identified some real issue, but I don't know what that was, and it's been a while.. have you done it recently?
15:03:39 sean-k-mooney efried: it is alredy on pypi also its weired i have blockdiag in my local requiremetns but i dont see it on master. ill talk to the release folks and make sure we are good
15:03:59 sean-k-mooney dansmith: no but i can do it again tomorrow
15:04:06 ignaziocassano1 lyarwood: I am sure the device is not used by multiple instances. I tried substituing the command "multipath -f" with "mutipath -r" in /usr/lib/python2.7/site-packages/os_brick/initiator/linuxscsi.py and it works . But flush and reload are different operations :-(
15:04:22 sean-k-mooney dansmith: the issue was the patches needed to be rebased for root required and cyboge had a python 3 issue
15:04:34 dansmith sean-k-mooney: ah okay, good to know, thanks
15:04:38 sean-k-mooney dansmith: sundar has rebased the cybog patch chain so it should work
15:04:48 dansmith gotcha, I remember now
15:04:57 openstackgerrit Stephen Finucane proposed openstack/nova master: objects: Remove custom comparison methods https://review.opendev.org/472285
15:05:49 sean-k-mooney ill kick off devstack stacking in the backgound and if it stack ill let you know what i find.
15:05:58 dansmith thanks
15:06:08 ignaziocassano1 lyarwood: keep in minf all volumes I attach have lvm inside them. Is it possible lvm on kvm node lock them ?
15:06:50 Sundar sean-k-mooney: Yes. I have also updated the Nova patch series, with all expected functionality, including the RPCAPI version bump to pass ARQ UUIDs from the conductor to manager.
15:06:52 sean-k-mooney dansmith: you have ipv6 connectivity right? when its stacked i can proably just add your public key to the vm if you want to take alook later
15:07:07 dansmith I do
15:07:07 sean-k-mooney Sundar: cool
15:09:07 sean-k-mooney cool i can always port forward but i have my router handing out public ipv6 adress to my openstack vms by default so if i tweek the security group rule it shoudl work
15:13:53 lyarwood ignaziocassano1: if you're host is also using LVM and has a default filter then yes it could be picking them all up
15:14:00 lyarwood your*
15:16:15 ignaziocassano1 lyarwood: I added the filter filter = [ "a|^/dev/sda|", "r|.*|" ] because I use lvm on kvm node only on /dev/sda but it does not resolve
15:17:50 ignaziocassano1 lyarwood: has the above filter a correct syntax ?
15:21:47 lyarwood ignaziocassano1: sorry I don't have the bandwidth to debug this with you at the moment
15:22:54 ignaziocassano1 lyarwood: I wrote all information on the bug I opened. I hope someone can help.
15:34:24 sean-k-mooney dansmith: Sundar efried just as an fyi this is the local.conf im using to deploy the cyborg env. the only thing you need to do manually is install the correct version of the cyborg client
15:34:31 sean-k-mooney http://paste.openstack.org/show/789067/
15:34:44 artom stephenfin, reply about the 3/2 split here: https://review.opendev.org/#/c/672595/64
15:34:55 artom Let's get this sorted so we can both move on ;)
15:35:36 dansmith sean-k-mooney: why is that? shouldn't it come in via reqs?
15:36:10 sean-k-mooney the cyborg plugin does not install it
15:36:21 sean-k-mooney devstack plugin that is
15:36:31 sean-k-mooney its not needed for nova its just for osc
15:36:41 sean-k-mooney so i dont have to use curl
15:37:01 dansmith oh you mean you have to install the cyborg osc plugin?
15:37:08 sean-k-mooney ya
15:37:16 dansmith okay gotcha
15:37:31 stephenfin artom: Could just do a negative test?
15:37:41 stephenfin actually, nvm
15:37:49 sean-k-mooney thats why im also checking out the openstack sdk from a review branch
15:37:57 sean-k-mooney that isnt neededed by nova
15:38:09 sean-k-mooney its just for osc
15:38:18 stephenfin artom: I'm dumb and don't see the problem you're pointing out with that comment. Can you restate it for me? :)
15:38:45 artom stephenfin, image the ascii diagram on the first "line", but with VM1 on N2
15:38:58 mmidolesov Does anayone know why functional-py36 returns TIMED_OUT in nova when rechecking patch? This happens for quite some time now.
15:39:30 artom If we then live migrate it, and we end up with the ascii diagram on the second "line", we have no way of knowing whether that's because NUMA live migration worked and updated the XML, or just because that's the old behaviour anyways
15:39:42 sean-k-mooney mmidolesov: in the gate?
15:39:59 sean-k-mooney i have not seen that on any patches i have reviewd lately
15:40:13 stephenfin Oh, so you're saying if it ended up landing on node 2, we wouldn't really be testing anything? Fair point. Could you just force the instances onto separate hosts though?
15:40:22 stephenfin Using the 'host' parameter from...2.53?
15:40:32 sean-k-mooney mmidolesov: if you are seing it on patche you pushed it might mean the mocking is not done correctly
15:41:07 sean-k-mooney stephenfin: the anti affintiy filer might be better
15:41:18 stephenfin sean-k-mooney: any reason?
15:41:28 sean-k-mooney im not sure tempest suppofrt the host parmater currently
15:41:32 artom stephenfin, we can force on separate hosts, but... can we force on specific NUMA nodes? Or, can we count that they'll be assigned to instances deterministically?
15:41:40 sean-k-mooney and we dont know what the host would be called generically

Earlier   Later