| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-06-08 | |||
| 16:57:37 | mriedem | the other is rpc, so it's not possibly (i think) | |
| 16:57:55 | dansmith | yeah, the rpc one he points to needs to stay single-cell, but I'll look to see if maybe we should do it higher | |
| 16:57:56 | cfriesen | sdague: I have no idea what's going on. I blew away ~/devstack and cloned it again, and blew away /opt/stack, and I'm still hitting version conflicts. | |
| 16:58:10 | dansmith | mriedem: we may also be able to drop lots of these | |
| 16:59:16 | mriedem | i'd like to drop paging from the hypervisors api | |
| 16:59:31 | dansmith | heh | |
| 17:01:52 | clarkb | cfriesen: sdague fwiw a trivial attempt at reproducing using a venv with 0.4.13 installed then an install of 0.4.15 using constraints works ok | |
| 17:02:05 | mriedem | btw i'm out tomorrow | |
| 17:03:04 | clarkb | so whatever it is seems more complicated than just pyroute2 and constraints | |
| 17:03:15 | openstackgerrit | Alex Szarka proposed openstack/nova master: Transform instance.exists notification https://review.openstack.org/403660 | |
| 17:03:59 | openstackgerrit | Alex Szarka proposed openstack/nova master: Refactor instance-exists docstring and unit test https://review.openstack.org/462738 | |
| 17:05:21 | clarkb | cfriesen: maybe try with pip -v and/or --log ? | |
| 17:05:50 | cfriesen | clarkb: I can upgrade this manually no problem....but I expect devstack to be doing the upgrading and I don't understand why it isn't | |
| 17:06:09 | openstackgerrit | Alex Szarka proposed openstack/nova master: Transform instance.resize.error notifications https://review.openstack.org/417882 | |
| 17:06:35 | clarkb | cfriesen: so if you run the command yourself it works fine? | |
| 17:06:49 | cfriesen | clarkb: I'm hitting this with multiple packages. If I manually upgrade one package it continues on until it hits another one that needs upgrading. | |
| 17:06:58 | mriedem | dansmith: are you ok with this now after the cells v1 rpc version bump? https://review.openstack.org/#/c/461532/ | |
| 17:07:03 | cfriesen | clarkb: something like this works fine: sudo pip install --upgrade ovsdbapp==0.4.0 | |
| 17:07:52 | dansmith | mriedem: with that approach yeah, I need to look at the rest of the patch though | |
| 17:07:53 | sdague | cfriesen: I for installed pyroute2==0.4.13 then ran ./clean.sh && ./stack.sh and everthing works fine | |
| 17:07:57 | clarkb | perhaps something with the -H then and pip caching things? But also getting output from pip -v and/or --log would likely be helpful | |
| 17:08:27 | sdague | this is a 16.04 box | |
| 17:08:31 | cfriesen | sdague: I hadn't tried clean.sh, will give that a shot | |
| 17:09:20 | cfriesen | sdague: same here | |
| 17:11:19 | openstackgerrit | Merged openstack/nova master: code comments incorrectness https://review.openstack.org/449416 | |
| 17:12:42 | cfriesen | mriedem: if you've got suggestions on how to properly mock the api_version in https://review.openstack.org/#/c/435141/ I'd be happy to make changes. I couldn't figure it out short of modifying each testcase using the "list" command. | |
| 17:16:56 | cfriesen | sdague: ./clean.sh wasn't enough. just hit ContextualVersionConflict with neutron-lib | |
| 17:18:07 | cfriesen | sdague: clarkb: http://paste.openstack.org/show/612034/ | |
| 17:32:29 | artom | dansmith, huh, didn't notice the ping - so basically the other places where get_min_service_version is called will be fixed in different patches? | |
| 17:32:51 | dansmith | artom: yeah, I was just fixing that one place in device metadata because of mriedem's bug he opened | |
| 17:32:53 | dansmith | but I | |
| 17:32:58 | dansmith | will take a look at the rest in a few | |
| 17:33:18 | artom | dansmith, makes sense :) | |
| 17:33:26 | artom | Keep commits topical | |
| 17:36:04 | mriedem | i like my creams just like i like my commits | |
| 17:36:04 | ihrachys | seems like port detach issues still happens even with https://review.openstack.org/#/c/441204/ in | |
| 17:36:10 | ihrachys | mriedem, ^ | |
| 17:36:20 | ihrachys | example: http://logs.openstack.org/63/471763/2/check/gate-tempest-dsvm-neutron-linuxbridge-ubuntu-xenial/dccf187/logs/screen-n-cpu.txt.gz?level=TRACE#_Jun_08_16_58_01_395989 | |
| 17:37:17 | mriedem | hmm, that's not fun | |
| 17:38:11 | mriedem | ihrachys: it's always this same test right http://logs.openstack.org/63/471763/2/check/gate-tempest-dsvm-neutron-linuxbridge-ubuntu-xenial/dccf187/console.html#_2017-06-08_17_13_16_844927 ? | |
| 17:38:14 | mriedem | the hotpluc nic one? | |
| 17:38:19 | mriedem | and usually on the linuxbridge job? | |
| 17:38:25 | artom | mriedem, the word you're looking for is "ointments" - it's up there with "moist" as the most shudder-inducing word in the English language | |
| 17:38:32 | mriedem | yeah it's like 99% LB job | |
| 17:38:35 | ihrachys | I think yes | |
| 17:38:43 | mriedem | i wonder what's different about LB in thiscase | |
| 17:38:55 | mriedem | maybe the vif isn't unplugging how we'd expect before we detach from the guest? | |
| 17:39:13 | mriedem | artom: i think "topical cream" | |
| 17:39:16 | mriedem | not "topical ointment" | |
| 17:39:21 | ihrachys | afaik for linuxbridge case, neutron now manages the whole plugging cycle ? | |
| 17:39:28 | ihrachys | since late switch to tap device type | |
| 17:39:38 | mriedem | ihrachys: i don't know anything about that | |
| 17:39:47 | ihrachys | I mean https://review.openstack.org/#/c/447150/ | |
| 17:39:55 | artom | mriedem, because of you my search history is weird now | |
| 17:40:08 | mriedem | artom: search on your phone, not your work system | |
| 17:40:25 | mriedem | nothing weird about grown man pain reliefe | |
| 17:40:27 | mriedem | *relief | |
| 17:40:37 | mriedem | just ask shaq | |
| 17:41:37 | mriedem | should be in the libvirtd logs wouldn't it? | |
| 17:43:22 | mriedem | i also don't understand why libvirt considers this a transient domain | |
| 17:43:52 | artom | I *think* it does both? | |
| 17:43:52 | mriedem | https://wiki.libvirt.org/page/VM_lifecycle#Transient_guest_domains_vs_Persistent_guest_domains | |
| 17:43:54 | mriedem | me neither | |
| 17:44:11 | artom | As in, a running domain has the persistent part, and the transient part | |
| 17:44:21 | artom | kashyap would help here, but I think he's gone | |
| 17:44:46 | melwitt | I think it's something like it creates a transient domain on-the-fly and uses that for the migration? something like that? | |
| 17:44:47 | mriedem | reading ^ makes me think all of our guests are transient | |
| 17:44:47 | ihrachys | oh it's just a in-ram domain? | |
| 17:44:57 | mriedem | this isn't a migration though | |
| 17:45:01 | melwitt | oh | |
| 17:45:15 | mriedem | http://git.openstack.org/cgit/openstack/tempest/tree/tempest/scenario/test_network_basic_ops.py#n475 | |
| 17:46:34 | cfriesen | mriedem: I think we generally make the xml file and tell libvirt to make a guest from it, making them persistent | |
| 17:48:46 | artom | Wait, that's not a libvirt error | |
| 17:49:28 | melwitt | one thing to know is a detach from transient domain is optional, that is, the guest can ignore it | |
| 17:50:02 | artom | https://github.com/openstack/nova/blob/master/nova/virt/libvirt/guest.py#L442-L449 | |
| 17:50:10 | cfriesen | melwitt: that sounds like a terrible idea. | |
| 17:50:51 | mriedem | artom: oh nice spot | |
| 17:50:56 | mriedem | we should be checking the persistent flag there | |
| 17:50:59 | ihrachys | ehm, why in the snippet do we raise unconditionally? | |
| 17:51:07 | mriedem | ihrachys: it's part of a retry loop | |
| 17:51:10 | mriedem | see the decorator | |
| 17:51:27 | artom | It retries until it stops raising? | |
| 17:51:27 | mriedem | it retries on that expected error until the device conf is gone or we timeout | |
| 17:51:31 | ihrachys | ah I see | |
| 17:51:41 | cfriesen | melwitt: mriedem: so under what circumstances would nova create a transient domain? | |
| 17:52:28 | cfriesen | looks like maybe while doing a snapshot | |
| 17:52:28 | melwitt | I don't think nova does it, I thought it was something with qemu but I can't remember what kashyap told me. maybe I'm totally off on that | |
| 17:53:23 | mriedem | we're not using os-vif http://logs.openstack.org/63/471763/2/check/gate-tempest-dsvm-neutron-linuxbridge-ubuntu-xenial/dccf187/logs/screen-n-cpu.txt.gz?level=DEBUG#_Jun_08_16_56_19_922445 | |
| 17:53:31 | mriedem | but that probably doesn't matter | |
| 17:56:09 | artom | Hrmm, that could be a libvirt error though | |
| 17:56:25 | artom | If we've failed to detach and reach the max retry count | |
| 17:56:50 | mriedem | it's got to be something with linuxbridge and this tap vif type | |
| 17:56:54 | mriedem | not a problem for ovs | |
| 17:57:04 | mriedem | and a race, since it's not 100% fail | |
| 17:57:48 | openstackgerrit | Merged openstack/nova master: Use microversions for new style volume attachments https://review.openstack.org/469579 | |
| 17:58:16 | kfox1111 | so... whats the easiest way to list out all vms in all tenants that use a particular flavor? | |
| 17:58:18 | mriedem | Successfully detached device <nova.virt.libvirt.config.LibvirtConfigGuestInterface object at 0x7f86a0051650> from guest. Persistent? 1. Live? True {{(pid=24466) _try_detach_device /opt/stack/new/nova/nova/virt/libvirt/guest.py:407}} | |
| 17:58:31 | melwitt | yeah, initially I was thinking we must have reached the max retry count because the guest wasn't responding, but I'm not sure how to tell from the log, if we can look at a timestamp delta at all to see how long it took before the error was raised | |
| 17:58:45 | mriedem | kfox1111: nova list --all-tenants --flavor x ? | |
| 17:59:01 | ihrachys | mriedem, you think we may try to revert the tap switch patch for the time being? | |