Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-06-08
16:57:55 dansmith yeah, the rpc one he points to needs to stay single-cell, but I'll look to see if maybe we should do it higher
16:57:56 cfriesen sdague: I have no idea what's going on. I blew away ~/devstack and cloned it again, and blew away /opt/stack, and I'm still hitting version conflicts.
16:58:10 dansmith mriedem: we may also be able to drop lots of these
16:59:16 mriedem i'd like to drop paging from the hypervisors api
16:59:31 dansmith heh
17:01:52 clarkb cfriesen: sdague fwiw a trivial attempt at reproducing using a venv with 0.4.13 installed then an install of 0.4.15 using constraints works ok
17:02:05 mriedem btw i'm out tomorrow
17:03:04 clarkb so whatever it is seems more complicated than just pyroute2 and constraints
17:03:15 openstackgerrit Alex Szarka proposed openstack/nova master: Transform instance.exists notification https://review.openstack.org/403660
17:03:59 openstackgerrit Alex Szarka proposed openstack/nova master: Refactor instance-exists docstring and unit test https://review.openstack.org/462738
17:05:21 clarkb cfriesen: maybe try with pip -v and/or --log ?
17:05:50 cfriesen clarkb: I can upgrade this manually no problem....but I expect devstack to be doing the upgrading and I don't understand why it isn't
17:06:09 openstackgerrit Alex Szarka proposed openstack/nova master: Transform instance.resize.error notifications https://review.openstack.org/417882
17:06:35 clarkb cfriesen: so if you run the command yourself it works fine?
17:06:49 cfriesen clarkb: I'm hitting this with multiple packages. If I manually upgrade one package it continues on until it hits another one that needs upgrading.
17:06:58 mriedem dansmith: are you ok with this now after the cells v1 rpc version bump? https://review.openstack.org/#/c/461532/
17:07:03 cfriesen clarkb: something like this works fine: sudo pip install --upgrade ovsdbapp==0.4.0
17:07:52 dansmith mriedem: with that approach yeah, I need to look at the rest of the patch though
17:07:53 sdague cfriesen: I for installed pyroute2==0.4.13 then ran ./clean.sh && ./stack.sh and everthing works fine
17:07:57 clarkb perhaps something with the -H then and pip caching things? But also getting output from pip -v and/or --log would likely be helpful
17:08:27 sdague this is a 16.04 box
17:08:31 cfriesen sdague: I hadn't tried clean.sh, will give that a shot
17:09:20 cfriesen sdague: same here
17:11:19 openstackgerrit Merged openstack/nova master: code comments incorrectness https://review.openstack.org/449416
17:12:42 cfriesen mriedem: if you've got suggestions on how to properly mock the api_version in https://review.openstack.org/#/c/435141/ I'd be happy to make changes. I couldn't figure it out short of modifying each testcase using the "list" command.
17:16:56 cfriesen sdague: ./clean.sh wasn't enough. just hit ContextualVersionConflict with neutron-lib
17:18:07 cfriesen sdague: clarkb: http://paste.openstack.org/show/612034/
17:32:29 artom dansmith, huh, didn't notice the ping - so basically the other places where get_min_service_version is called will be fixed in different patches?
17:32:51 dansmith artom: yeah, I was just fixing that one place in device metadata because of mriedem's bug he opened
17:32:53 dansmith but I
17:32:58 dansmith will take a look at the rest in a few
17:33:18 artom dansmith, makes sense :)
17:33:26 artom Keep commits topical
17:36:04 mriedem i like my creams just like i like my commits
17:36:04 ihrachys seems like port detach issues still happens even with https://review.openstack.org/#/c/441204/ in
17:36:10 ihrachys mriedem, ^
17:36:20 ihrachys example: http://logs.openstack.org/63/471763/2/check/gate-tempest-dsvm-neutron-linuxbridge-ubuntu-xenial/dccf187/logs/screen-n-cpu.txt.gz?level=TRACE#_Jun_08_16_58_01_395989
17:37:17 mriedem hmm, that's not fun
17:38:11 mriedem ihrachys: it's always this same test right http://logs.openstack.org/63/471763/2/check/gate-tempest-dsvm-neutron-linuxbridge-ubuntu-xenial/dccf187/console.html#_2017-06-08_17_13_16_844927 ?
17:38:14 mriedem the hotpluc nic one?
17:38:19 mriedem and usually on the linuxbridge job?
17:38:25 artom mriedem, the word you're looking for is "ointments" - it's up there with "moist" as the most shudder-inducing word in the English language
17:38:32 mriedem yeah it's like 99% LB job
17:38:35 ihrachys I think yes
17:38:43 mriedem i wonder what's different about LB in thiscase
17:38:55 mriedem maybe the vif isn't unplugging how we'd expect before we detach from the guest?
17:39:13 mriedem artom: i think "topical cream"
17:39:16 mriedem not "topical ointment"
17:39:21 ihrachys afaik for linuxbridge case, neutron now manages the whole plugging cycle ?
17:39:28 ihrachys since late switch to tap device type
17:39:38 mriedem ihrachys: i don't know anything about that
17:39:47 ihrachys I mean https://review.openstack.org/#/c/447150/
17:39:55 artom mriedem, because of you my search history is weird now
17:40:08 mriedem artom: search on your phone, not your work system
17:40:25 mriedem nothing weird about grown man pain reliefe
17:40:27 mriedem *relief
17:40:37 mriedem just ask shaq
17:41:37 mriedem should be in the libvirtd logs wouldn't it?
17:43:22 mriedem i also don't understand why libvirt considers this a transient domain
17:43:52 artom I *think* it does both?
17:43:52 mriedem https://wiki.libvirt.org/page/VM_lifecycle#Transient_guest_domains_vs_Persistent_guest_domains
17:43:54 mriedem me neither
17:44:11 artom As in, a running domain has the persistent part, and the transient part
17:44:21 artom kashyap would help here, but I think he's gone
17:44:46 melwitt I think it's something like it creates a transient domain on-the-fly and uses that for the migration? something like that?
17:44:47 mriedem reading ^ makes me think all of our guests are transient
17:44:47 ihrachys oh it's just a in-ram domain?
17:44:57 mriedem this isn't a migration though
17:45:01 melwitt oh
17:45:15 mriedem http://git.openstack.org/cgit/openstack/tempest/tree/tempest/scenario/test_network_basic_ops.py#n475
17:46:34 cfriesen mriedem: I think we generally make the xml file and tell libvirt to make a guest from it, making them persistent
17:48:46 artom Wait, that's not a libvirt error
17:49:28 melwitt one thing to know is a detach from transient domain is optional, that is, the guest can ignore it
17:50:02 artom https://github.com/openstack/nova/blob/master/nova/virt/libvirt/guest.py#L442-L449
17:50:10 cfriesen melwitt: that sounds like a terrible idea.
17:50:51 mriedem artom: oh nice spot
17:50:56 mriedem we should be checking the persistent flag there
17:50:59 ihrachys ehm, why in the snippet do we raise unconditionally?
17:51:07 mriedem ihrachys: it's part of a retry loop
17:51:10 mriedem see the decorator
17:51:27 artom It retries until it stops raising?
17:51:27 mriedem it retries on that expected error until the device conf is gone or we timeout
17:51:31 ihrachys ah I see
17:51:41 cfriesen melwitt: mriedem: so under what circumstances would nova create a transient domain?
17:52:28 cfriesen looks like maybe while doing a snapshot
17:52:28 melwitt I don't think nova does it, I thought it was something with qemu but I can't remember what kashyap told me. maybe I'm totally off on that
17:53:23 mriedem we're not using os-vif http://logs.openstack.org/63/471763/2/check/gate-tempest-dsvm-neutron-linuxbridge-ubuntu-xenial/dccf187/logs/screen-n-cpu.txt.gz?level=DEBUG#_Jun_08_16_56_19_922445
17:53:31 mriedem but that probably doesn't matter
17:56:09 artom Hrmm, that could be a libvirt error though
17:56:25 artom If we've failed to detach and reach the max retry count
17:56:50 mriedem it's got to be something with linuxbridge and this tap vif type
17:56:54 mriedem not a problem for ovs
17:57:04 mriedem and a race, since it's not 100% fail
17:57:48 openstackgerrit Merged openstack/nova master: Use microversions for new style volume attachments https://review.openstack.org/469579
17:58:16 kfox1111 so... whats the easiest way to list out all vms in all tenants that use a particular flavor?
17:58:18 mriedem Successfully detached device <nova.virt.libvirt.config.LibvirtConfigGuestInterface object at 0x7f86a0051650> from guest. Persistent? 1. Live? True {{(pid=24466) _try_detach_device /opt/stack/new/nova/nova/virt/libvirt/guest.py:407}}
17:58:31 melwitt yeah, initially I was thinking we must have reached the max retry count because the guest wasn't responding, but I'm not sure how to tell from the log, if we can look at a timestamp delta at all to see how long it took before the error was raised
17:58:45 mriedem kfox1111: nova list --all-tenants --flavor x ?
17:59:01 ihrachys mriedem, you think we may try to revert the tap switch patch for the time being?
17:59:05 kfox1111 doh. there it was, right in front of me.

Earlier   Later