| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-02-03 | |||
| 14:38:12 | efried | sean-k-mooney: okay, cool, so we should blacklist 2.0.0 in u-c and accept, what, 2.0.1 once they cut it? | |
| 14:38:28 | sean-k-mooney | ya basically | |
| 14:38:49 | sean-k-mooney | we can blacklist it now if we want too but i dont know what the eta is on the 2.0.1 release is | |
| 14:39:15 | efried | Well, we know 2.0.0 is busted, but it also doesn't seem to be affecting anyone but us, so... | |
| 14:39:22 | sean-k-mooney | oh its already done | |
| 14:40:56 | sean-k-mooney | https://github.com/blockdiag/blockdiag/compare/2.0.0...2.0.1 | |
| 14:40:58 | ignaziocassano1 | lyqrwood: I am using centos 7.....how can open a bug ? Which site must I use ? | |
| 14:41:20 | sean-k-mooney | efried: want me to submit the patch to the requiremetns repo | |
| 14:42:00 | lyarwood | ignaziocassano1: https://bugs.launchpad.net/os-brick/+filebug - if you could include DEBUG logs showing the volume attachment and attempt to detach that would be super useful | |
| 14:42:17 | ignaziocassano1 | thanks | |
| 14:44:26 | openstackgerrit | Stephen Finucane proposed openstack/nova master: nova-net: Remove unused nova-network objects https://review.opendev.org/697156 | |
| 14:44:27 | openstackgerrit | Stephen Finucane proposed openstack/nova master: nova-net: Remove unused parameters https://review.opendev.org/703974 | |
| 14:45:20 | efried | sean-k-mooney: sure | |
| 14:45:51 | sean-k-mooney | ok ill do it shortly once i finish my current review | |
| 14:47:09 | sean-k-mooney | efried: did we already merge stephens fix | |
| 14:47:19 | efried | yes | |
| 14:47:36 | efried | sean-k-mooney: https://review.opendev.org/#/c/705263/ | |
| 14:48:06 | sean-k-mooney | cool im just going to blacklist it in gr we could raise the min version in nova if we want to removed the monkey patching seperatly | |
| 14:48:36 | sean-k-mooney | im not going to raise the lower constratin in the requirements repo. | |
| 14:50:00 | ignaziocassano1 | lyarwood: I opened a bug. Did you know if I will receive email when someone will work on it ? | |
| 14:50:33 | lyarwood | ignaziocassano1: can you share the bug URL here? | |
| 14:50:46 | openstack | Launchpad bug 1861691 in os-brick "openstack queens emc iscsi errors migrating volume and or detaching volumes" [Undecided,New] | |
| 14:50:46 | ignaziocassano1 | https://bugs.launchpad.net/os-brick/+bug/1861691 | |
| 14:51:01 | ignaziocassano1 | I attached the nova-compute log | |
| 14:51:44 | ignaziocassano1 | mmm I must change the description | |
| 14:52:41 | ignaziocassano1 | It happens when I migrate a vm or detach a volume | |
| 14:53:25 | lyarwood | ignaziocassano1: do you have any logs showing the volume being attached? | |
| 14:54:36 | ignaziocassano1 | I included the nova-compute log ...the problem occurs non kvm node. I also have the volume log. Dow you want I attach it ? | |
| 14:57:17 | ignaziocassano1 | When I migrate a vm , os-brick try to use the command "multipath -f xxxxxxxx" to flusj the device. As reported in nova compute log, it finds "map in use" | |
| 14:58:21 | sean-k-mooney | efried: oh its already blacklisted | |
| 14:58:29 | efried | cool | |
| 14:58:56 | sean-k-mooney | im not sure we want the upper constratis file to cap it at 2.0.1 however | |
| 14:59:08 | ignaziocassano1 | So, often the vm migrates but remains in error state and horizon shows the vm on old node | |
| 15:00:52 | lyarwood | ignaziocassano1: this channel isn't really for support but yeah `map in use` errors suggest the multipath device is being used by multiple instances, normally due to c-vol or os-brick bugs | |
| 15:01:08 | lyarwood | ignaziocassano1: make sure both nova, os-brick and cinder are fully updated | |
| 15:01:43 | ignaziocassano1 | lyarwood: they are fully updated with last queens released on centos | |
| 15:02:00 | dansmith | efried: I know you have other stuff going on which is probably delaying your ability to run through the cyborg patches. If you could at least hit the profile groups thing, that would be helpful: https://review.opendev.org/#/c/631243 | |
| 15:02:02 | efried | sean-k-mooney: certainly not until that's releaced | |
| 15:02:09 | efried | dansmith: ack | |
| 15:02:30 | dansmith | efried: knowing that latest version of that matches your expectation and what should be done is important | |
| 15:02:33 | dansmith | thanks | |
| 15:03:21 | dansmith | sean-k-mooney: last I heard you had deployed the cyborg set in devstack and had identified some real issue, but I don't know what that was, and it's been a while.. have you done it recently? | |
| 15:03:39 | sean-k-mooney | efried: it is alredy on pypi also its weired i have blockdiag in my local requiremetns but i dont see it on master. ill talk to the release folks and make sure we are good | |
| 15:03:59 | sean-k-mooney | dansmith: no but i can do it again tomorrow | |
| 15:04:06 | ignaziocassano1 | lyarwood: I am sure the device is not used by multiple instances. I tried substituing the command "multipath -f" with "mutipath -r" in /usr/lib/python2.7/site-packages/os_brick/initiator/linuxscsi.py and it works . But flush and reload are different operations :-( | |
| 15:04:22 | sean-k-mooney | dansmith: the issue was the patches needed to be rebased for root required and cyboge had a python 3 issue | |
| 15:04:34 | dansmith | sean-k-mooney: ah okay, good to know, thanks | |
| 15:04:38 | sean-k-mooney | dansmith: sundar has rebased the cybog patch chain so it should work | |
| 15:04:48 | dansmith | gotcha, I remember now | |
| 15:04:57 | openstackgerrit | Stephen Finucane proposed openstack/nova master: objects: Remove custom comparison methods https://review.opendev.org/472285 | |
| 15:05:49 | sean-k-mooney | ill kick off devstack stacking in the backgound and if it stack ill let you know what i find. | |
| 15:05:58 | dansmith | thanks | |
| 15:06:08 | ignaziocassano1 | lyarwood: keep in minf all volumes I attach have lvm inside them. Is it possible lvm on kvm node lock them ? | |
| 15:06:50 | Sundar | sean-k-mooney: Yes. I have also updated the Nova patch series, with all expected functionality, including the RPCAPI version bump to pass ARQ UUIDs from the conductor to manager. | |
| 15:06:52 | sean-k-mooney | dansmith: you have ipv6 connectivity right? when its stacked i can proably just add your public key to the vm if you want to take alook later | |
| 15:07:07 | sean-k-mooney | Sundar: cool | |
| 15:07:07 | dansmith | I do | |
| 15:09:07 | sean-k-mooney | cool i can always port forward but i have my router handing out public ipv6 adress to my openstack vms by default so if i tweek the security group rule it shoudl work | |
| 15:13:53 | lyarwood | ignaziocassano1: if you're host is also using LVM and has a default filter then yes it could be picking them all up | |
| 15:14:00 | lyarwood | your* | |
| 15:16:15 | ignaziocassano1 | lyarwood: I added the filter filter = [ "a|^/dev/sda|", "r|.*|" ] because I use lvm on kvm node only on /dev/sda but it does not resolve | |
| 15:17:50 | ignaziocassano1 | lyarwood: has the above filter a correct syntax ? | |
| 15:21:47 | lyarwood | ignaziocassano1: sorry I don't have the bandwidth to debug this with you at the moment | |
| 15:22:54 | ignaziocassano1 | lyarwood: I wrote all information on the bug I opened. I hope someone can help. | |
| 15:34:24 | sean-k-mooney | dansmith: Sundar efried just as an fyi this is the local.conf im using to deploy the cyborg env. the only thing you need to do manually is install the correct version of the cyborg client | |
| 15:34:31 | sean-k-mooney | http://paste.openstack.org/show/789067/ | |
| 15:34:44 | artom | stephenfin, reply about the 3/2 split here: https://review.opendev.org/#/c/672595/64 | |
| 15:34:55 | artom | Let's get this sorted so we can both move on ;) | |
| 15:35:36 | dansmith | sean-k-mooney: why is that? shouldn't it come in via reqs? | |
| 15:36:10 | sean-k-mooney | the cyborg plugin does not install it | |
| 15:36:21 | sean-k-mooney | devstack plugin that is | |
| 15:36:31 | sean-k-mooney | its not needed for nova its just for osc | |
| 15:36:41 | sean-k-mooney | so i dont have to use curl | |
| 15:37:01 | dansmith | oh you mean you have to install the cyborg osc plugin? | |
| 15:37:08 | sean-k-mooney | ya | |
| 15:37:16 | dansmith | okay gotcha | |
| 15:37:31 | stephenfin | artom: Could just do a negative test? | |
| 15:37:41 | stephenfin | actually, nvm | |
| 15:37:49 | sean-k-mooney | thats why im also checking out the openstack sdk from a review branch | |
| 15:37:57 | sean-k-mooney | that isnt neededed by nova | |
| 15:38:09 | sean-k-mooney | its just for osc | |
| 15:38:18 | stephenfin | artom: I'm dumb and don't see the problem you're pointing out with that comment. Can you restate it for me? :) | |
| 15:38:45 | artom | stephenfin, image the ascii diagram on the first "line", but with VM1 on N2 | |
| 15:38:58 | mmidolesov | Does anayone know why functional-py36 returns TIMED_OUT in nova when rechecking patch? This happens for quite some time now. | |
| 15:39:30 | artom | If we then live migrate it, and we end up with the ascii diagram on the second "line", we have no way of knowing whether that's because NUMA live migration worked and updated the XML, or just because that's the old behaviour anyways | |
| 15:39:42 | sean-k-mooney | mmidolesov: in the gate? | |
| 15:39:59 | sean-k-mooney | i have not seen that on any patches i have reviewd lately | |
| 15:40:13 | stephenfin | Oh, so you're saying if it ended up landing on node 2, we wouldn't really be testing anything? Fair point. Could you just force the instances onto separate hosts though? | |
| 15:40:22 | stephenfin | Using the 'host' parameter from...2.53? | |
| 15:40:32 | sean-k-mooney | mmidolesov: if you are seing it on patche you pushed it might mean the mocking is not done correctly | |
| 15:41:07 | sean-k-mooney | stephenfin: the anti affintiy filer might be better | |
| 15:41:18 | stephenfin | sean-k-mooney: any reason? | |
| 15:41:28 | sean-k-mooney | im not sure tempest suppofrt the host parmater currently | |
| 15:41:32 | artom | stephenfin, we can force on separate hosts, but... can we force on specific NUMA nodes? Or, can we count that they'll be assigned to instances deterministically? | |
| 15:41:40 | sean-k-mooney | and we dont know what the host would be called generically | |
| 15:41:58 | sean-k-mooney | artom: no we cant force to a specif numa node | |
| 15:42:21 | artom | stephenfin, IOW, I want a situation where we *know for sure* that without the NUMA live migration feature, we'd have ended up with overlapping instances on the destination | |
| 15:42:41 | sean-k-mooney | we should not rely on the ordering | |
| 15:42:53 | sean-k-mooney | it tecnically is determisitic but only if we run serially | |
| 15:42:56 | artom | So 1. is that a fair thing to want? 2. can we achieve it with a less artificial NUMA topology | |