Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-04-13
14:34:43 mriedem clarkb: looks like we have a new set of live migration failures since the UCA change http://logs.openstack.org/83/455183/5/check/gate-tempest-dsvm-multinode-live-migration-ubuntu-xenial/38fe0cd/logs/screen-n-cpu.txt.gz?level=TRACE#_2017-04-13_11_58_36_986 - not sure if it's related to the other things we've already seen and UCA was meant to fix, i.e. the i/o error
14:34:58 bauzas mriedem: looks like the live-mig job is bonkers
14:35:01 mriedem sfinucan: that tab was already open :)
14:35:14 mriedem it is http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Live%20Migration%20failure%3A%20operation%20failed%3A%20Failed%20to%20connect%20to%20remote%20libvirt%20URI%5C%22%20AND%20message%3A%5C%22End%20of%20file%20while%20reading%20data%3A%20sh%3A%201%3A%20nc%3A%20not%20found%3A%20Input%2Foutput%20error%5C%22%20AND%20tags%3A%5C%22screen-n-cpu.txt%5C%22&from=7d
14:37:54 kashyap sfinucan: I'm obviously not a "core", but there's nothing inherently wrong with "two RH +2s"
14:38:02 kashyap Assuming they're acting in good faith
14:38:15 sfinucan kashyap: True, but it's just good practice
14:38:23 kashyap Having a blind rule (I'm not saying there is) like "We don't want two RH +2s" is just plain silly
14:38:30 sfinucan especially when the patch was RH authored
14:38:42 kashyap sfinucan: Ah, like that
14:38:58 kashyap sfinucan: Sure, it's a "nice to have" to keep biases in check
14:39:01 mriedem i'll review it when i'm off this call
14:39:14 kashyap But if they're are mature adults, then it's okay to assume they're acting in good faith :-)
14:39:16 openstackgerrit Sergey Nikitin proposed openstack/nova-specs master: PCI NUMA Policies https://review.openstack.org/361140
14:39:28 mriedem kashyap: they are also rushing because of tripleo pressure
14:39:30 mriedem so there is a conflict
14:39:36 mriedem just hold up 30 minutes please :)
14:39:38 sfinucan kashyap: z'actly :)
14:39:40 kashyap sfinucan: Case in point: Upstream QEMU block layer has lots of RH-authored patches reviewed by RHers
14:39:49 kashyap And the quality of those discussions is one of the highest I've ever seen.
14:40:00 kashyap And they disagree all the time with each other
14:40:03 jaypipes bauzas, sfinucan: will try my best. really not feeling well today :(
14:40:16 kashyap mriedem: Ah, I see - didn't know the TripleO context
14:40:17 bauzas jaypipes: no worries man
14:40:22 bauzas jaypipes: take rest
14:44:34 sfinucan jaypipes: Aye, take it easy in that case
14:45:21 jangutter Massive apologies again for the insane lateness of "Spec to enable Netronome SmartNIC support" https://review.openstack.org/#/c/456386/
14:46:15 jangutter At least I should be online for the next couple of hours if anyone is still willing to review it.
14:50:14 openstackgerrit Sylvain Bauza proposed openstack/nova master: Fix AZ related API docs https://review.openstack.org/456655
14:50:30 bauzas mriedem: johnthetubaguy: attempt to clarify our API docs wrt: AZs https://review.openstack.org/456655
14:55:37 mriedem live migration job is spiking at 75%+ fail, i'm going to make it non-voting
14:56:28 kashyap mriedem: Any specific bug (or a set of bugs) that's causing it?
14:56:39 openstack Launchpad bug 1682453 in OpenStack Compute (nova) "Live Migration failure: operation failed: Failed to connect to remote libvirt URI - End of file while reading data: sh: 1: nc: not found: Input/output error" [High,Confirmed]
14:56:39 mriedem https://bugs.launchpad.net/nova/+bug/1682453
14:56:44 mriedem libvirt craps it's pants
14:56:51 mriedem with newer libvirt/qemu
14:57:06 mriedem which we were told by some (cough cough) it would fix all of our problems :)
14:58:58 kashyap mriedem: I never make blanket promises, certainly to _you_ that it will "fix all the problems"
14:59:26 mriedem https://review.openstack.org/456656
14:59:28 kashyap s/to/not to/
14:59:39 mriedem kashyap: i'm just teasing
14:59:49 kashyap I know, I forgot to add a smiley
15:07:34 owalsh kashyap: that looks like a packaging issue, nc not installed
15:08:48 jangutter :O "nc not installed" conjures the same horror in me as "gcc not installed".
15:10:16 sdague mriedem: the fix is merged now, right?
15:10:53 mriedem clarks?
15:10:56 mriedem that merged last night
15:10:57 mriedem this is something else
15:11:14 sdague mriedem: https://review.openstack.org/#/c/456388/
15:11:21 sdague no, it didn't merge until this morning
15:11:50 sdague also, it seems totally weird to me that libvirt would shell out to nc
15:12:56 dansmith sdague: it has definitely used nc for things
15:13:26 dansmith sdague: used to be you couldn't connect a fedora virt-manager to an ubuntu host without installing netcat, or something like that
15:13:33 dansmith the reverse maybe, I forget
15:19:15 sdague so I think this is probably the problematic commit for us - https://libvirt.org/git/?p=libvirt.git;a=commitdiff;h=22eaee8e011645162e4b00d313bf2abdd4eb54e5
15:19:30 sdague probably good to get an ubuntu packaging bug openned up on that
15:20:17 sdague clarkb: did you do that yet?
15:21:52 owalsh sdague: that's for qemu+libssh, the qemu+ssh driver always requried netcat
15:25:28 sfinucan owalsh: Regarding https://review.openstack.org/456571, are you saying we need user and port options even if we don't deprecate 'live_migration_uri'
15:27:10 sdague it looks like the libvirt-daemon recommends netcat-openbsd, but does not require it
15:27:12 sdague Recommends: qemu-kvm | qemu (>= 0.9.1), libxml2-utils, netcat-openbsd
15:27:38 owalsh sfinucan: need to use live_migration_inbound_addr to set the IP network for live migration. We can't do this with live_migration_uri AFAIK
15:29:51 owalsh sfinucan: I assume that's one of the main reasons for deprecating live_migration_uri
15:30:44 owalsh sfinucan: but to use SSH we need to pass user, maybe port too, and almost certainly extraoptions, and there is no way to set this when using live_migration_inbound_addr
15:37:19 sfinucan owalsh: So for 'live_migration_uri', you could conceivably set it to something like 'xen+ssh://rjones@%s/?no_verify=1'. That has both a user and extra_options
15:37:46 sfinucan owalsh: Do we want the same functionality for whatever's using live_migration_inbound_addr?
15:39:49 owalsh sfinucan: yes, currently do set the user via live_migration_uri, but we can't control which network it users. %s is always replaced with the remote hostname
15:42:00 sfinucan owalsh: ...and live_migration_inbound_addr provides an alternative to that, but it doesn't allow you to a username or extra_specs currently, right?
15:43:59 sfinucan owalsh: If we allowed you to _not_ specify '%s' for 'live_migration_uri' and instead set a static destination/IP network, would that achieve what you want?
15:44:20 cdent edleafe: are your resource tracker related functional tests sort of on hold?
15:44:46 owalsh sfinucan: you have to specify %s - it's replaced with the hostname of the live miragation target
15:45:02 sfinucan owalsh: That's what I'm saying. What if I removed that requirement
15:45:26 sfinucan Or are the two complimentary somehow?
15:46:35 owalsh so live_migration_uri = foo://bar@%s/ will tranlate to foo://bar@nova1/ when migrating to nova1, and foo:/bar@nova2/ when migrating to nova2
15:48:07 owalsh when using the new approach when migrating to nova1 we get live_migration_inbound addr etc.. for nova1 to build the url to migrate to it
15:48:15 owalsh IIRC anyway
15:49:35 owalsh it been a while since I looked at it but I think there was consensus that the new options are required
15:49:47 openstackgerrit Artom Lifshitz proposed openstack/nova master: Plumbing for tagged nic attachment https://review.openstack.org/415622
15:49:48 openstackgerrit Artom Lifshitz proposed openstack/nova master: Delete nic metadata when detaching interface https://review.openstack.org/435387
15:49:49 openstackgerrit Artom Lifshitz proposed openstack/nova master: Libvirt support for tagged volume attachment https://review.openstack.org/418100
15:49:52 openstackgerrit Artom Lifshitz proposed openstack/nova master: Delete disk metadata when detaching volume https://review.openstack.org/435082
15:49:55 openstackgerrit Artom Lifshitz proposed openstack/nova master: Plumbing for tagged volume attachment https://review.openstack.org/391941
15:49:56 openstackgerrit Artom Lifshitz proposed openstack/nova master: API support for tagged device attachment https://review.openstack.org/376021
15:49:57 openstackgerrit Artom Lifshitz proposed openstack/nova master: Libvirt support for tagged nic attachment https://review.openstack.org/415520
15:57:30 edleafe cdent: yeah, more because I got distracted with other stuff while waiting for more feedback
15:58:18 cdent edleafe: yeah, just checking. I had a reminder to look at them but didn't have the current brain to say much in this round.
15:58:44 edleafe cdent: ok, remind me next week if you think of it
15:58:53 cdent ✔
15:59:22 mriedem dansmith: before you change things in https://review.openstack.org/#/c/456227/ i've left some comments
15:59:29 mriedem looks like some were at the same time you were replying to sfinucan
15:59:30 openstackgerrit Dan Smith proposed openstack/nova master: Fix hypervisors api missing HostMappingNotFound handlers https://review.openstack.org/456227
15:59:32 dansmith hah
15:59:33 mriedem doh
15:59:34 mriedem gdi
15:59:41 dansmith was literally just pushing when you said that
16:02:43 ildikov mriedem: johnthetubaguy: are you avaiable for the Cinder-Nova meeting?
16:02:54 mriedem yeah
16:18:16 openstackgerrit Daniel Pawlik proposed openstack/nova master: Changed network_info variable in test_create_configdrive https://review.openstack.org/436953
16:21:11 dtp good morning. can we get a ruling on https://blueprints.launchpad.net/nova/+spec/conf-network-metadata-always-static-ip, please?
16:23:38 kashyap mriedem: That live mig job failure is due to missing `nc` (Netcat), which seems to be resolved by clarkb's -- https://review.openstack.org/#/c/456388/ ("Install netcat for libvirt live migration")

Earlier   Later