Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-03-16
14:45:57 aunnam lbragstad, just commented out the rule, that all rule, haven't changed the tests yet
14:46:03 mhenkel jaypipes: sure thing, ping me when you have a minute or two
14:46:30 openstackgerrit Sujitha proposed openstack/nova master: Change tags to default field in Instance object. https://review.openstack.org/415298
14:46:33 aunnam lbragstad, commented the rule in here https://github.com/openstack/oslo.policy/blob/master/oslo_policy/generator.py#L109
14:46:44 openstackgerrit Sujitha proposed openstack/nova master: Reduce calls to load_tags() to 0 https://review.openstack.org/435146
14:47:15 mriedem no that's not it, last time we bumped eventlet up was last may
14:48:16 lbragstad aunnam alright - this is what i have locally http://cdn.pasteraw.com/3u2pj6gigswl55mkeqdbojeifp6gzf4
14:48:30 lbragstad which is showing me 6 failures
14:48:39 lyarwood mriedem: do you have a logstash query for this already btw?
14:49:08 mriedem lyarwood: just this: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22AttachVolumeShelveTestJSON%5C%22%20AND%20message%3A%5C%22Failed%20to%20detach%20volume%5C%22%20AND%20message%3A%5C%22libvirtError%3A%20End%20of%20file%20while%20reading%20data%3A%20Input%2Foutput%20error%5C%22%20AND%20tags%3A%5C%22screen-n-cpu.txt%5C%22&from=7d
14:49:57 aunnam lbragstad, ya that's the same think that i have
14:49:59 mriedem lyarwood: did you create a bug? i think we need some more logging in detach_device_with_retry because i can't make out the stacktrace from that method given all of the context managers
14:50:05 lbragstad aunnam cool
14:50:59 lyarwood mriedem: thanks, not yet just waiting on devstack finishing locally before I try to reproduce
14:51:33 gibi sdague: responded on https://review.openstack.org/#/c/440580/5/specs/pike/approved/scheduler-hints-in-server-details.rst I can bargain on what hints the API will return
14:52:34 openstackgerrit Sujitha proposed openstack/nova master: Adding auto_disk_config field to InstancePayload https://review.openstack.org/419185
14:53:13 openstackgerrit Béla Vancsics proposed openstack/nova master: Reduce code complexity - libvirt/config.py https://review.openstack.org/359879
14:53:46 mriedem lyarwood: since we dont see "Original exception being dropped" i'm wondering if we're hitting _try_detach_device the first time, it's ok, and then when we try the 2nd time, it fails with that i/o error
14:54:07 mriedem that's why i want more debug logging in there to know if we're calling _try_detach_device before we get into the retry loop
14:54:27 mriedem because i'm guessing that the i/o error thing on an already detached device is newish in libvirt 1.3.1
14:54:47 mdbooth lyarwood mriedem: Just finished reading scrollback. Sounds fun.
14:55:21 mriedem mdbooth: definitely not what i planned on doing for my first 2 hours this morning :)
14:55:28 mriedem gate spelunking
14:57:54 dansmith gibi: if you're still around, it'd be good to get your ack on this too: https://review.openstack.org/#/c/446053/4
14:58:27 Jack_Iv Hey Folks! Since Nova 13 the hooks are DEPRECATED, but what can I use instead?
14:58:52 Jack_Iv I need to run some code, pre- and post- instance build
14:59:40 mriedem Jack_Iv: there is the dynamic vendordata metadata api and notifications
14:59:46 mriedem or upstream whatever your use case is
14:59:58 mriedem https://docs.openstack.org/developer/nova/vendordata.html
15:00:05 mriedem https://docs.openstack.org/developer/nova/notifications.html
15:00:35 mdbooth Jack_Iv: What is your use case, btw?
15:00:49 gibi dansmith: looking...
15:01:14 Jack_Iv I need to execute some code on compute nodes, after vm is UP and after termination
15:01:18 Jack_Iv mdbooth: ^
15:01:31 mdbooth Jack_Iv: Right, but what for?
15:01:40 Jack_Iv let's say, update some iptables rules
15:01:41 mdbooth What does the code do?
15:01:54 mdbooth On the host?
15:01:57 Jack_Iv right
15:02:00 kashyap Yeah, I'm curious, too. About the use case
15:02:13 mriedem lyarwood: https://bugs.launchpad.net/nova/+bug/1673483
15:02:13 openstack Launchpad bug 1673483 in OpenStack Compute (nova) "libvirt: test_attach_volume_shelved_or_offload_server times out waiting for device detach (which fails)" [Undecided,New]
15:02:14 openstackgerrit Moshe Levi proposed openstack/nova master: HW offload support for openvswitch https://review.openstack.org/398265
15:02:31 mdbooth Jack_Iv: Is this secret squirrel, or can you share the whole use case? I'm really curious as to why.
15:03:07 kashyap From the an old mailing list discussion on hooks, I see:
15:03:08 Jack_Iv I want to update some iptables rules after VM is up and delete those rules after termination
15:03:13 kashyap There's three core scenarios for hooks
15:03:13 kashyap 1. Modifying some aspect of the Nova operation
15:03:13 kashyap 2. Triggering an external action synchronously to some Nova operation
15:03:13 kashyap 3. Triggering an external action asynchronously to some Nova operation
15:03:20 kashyap [From danpb]
15:03:35 mriedem Jack_Iv: you can't do that with security group rules?
15:03:41 mdbooth Jack_Iv: Right. Just trying to understand why, and if there's some other way to achieve what you want.
15:03:47 mdbooth e.g. what mriedem said
15:04:11 mriedem Jack_Iv: i.e. listen for the notifications from nova, instance.create.start and instance.create.end, and when you get those notifications, adjust the secgroup rules in neutron
15:04:13 Jack_Iv No, because clients can just edit security group rules
15:04:40 mriedem Jack_Iv: so change the neutron api to make admin-only rules?
15:04:54 mriedem Jack_Iv: at the PTG we talked about a concept of service user locks on instances,
15:05:05 mriedem you could make a similar case for service/admin level "locks" on security group rules
15:05:25 dansmith I thought neutron had some base "provider rules" concept for this
15:05:36 mriedem dansmith: maybe they do
15:05:41 mriedem Jack_Iv: have you talked to the neutron people?
15:05:47 Jack_Iv not yet
15:05:47 mriedem Jack_Iv: which version of openstack are you on?
15:05:50 Jack_Iv newton
15:06:57 mriedem kevinbenton: does neutron have a concept of provider rules for security groups that the tenant user can't change?
15:07:58 mriedem Jack_Iv: i think that's probably the thread you need to pull on though
15:08:14 mriedem listen for notifications from nova for the instance build lifecycle, and then adjust rules via the networking API as needed
15:09:41 mriedem no vmware people around huh
15:09:51 mriedem their CI results come in, if at all, about a week late
15:15:50 openstackgerrit Lee Yarwood proposed openstack/nova master: compute: Move detach logic from manager into driver BDM https://review.openstack.org/439520
15:15:51 openstackgerrit Lee Yarwood proposed openstack/nova master: compute: Only destroy BDMs after successful detach call https://review.openstack.org/440693
15:16:13 gibi dansmith: I'm +1 on https://review.openstack.org/#/c/446053
15:16:35 dansmith gibi: awesome, thanks for looking at that
15:16:40 dansmith gibi: on the other patch,
15:16:44 lyarwood mdbooth: ^ updated removing some of the nits, just to confirm the LM rollback issue isn't new right?
15:17:02 mdbooth lyarwood: No, the LM rollback issue looks new
15:17:13 dansmith gibi: the change I made prevents projects from showing up on actual flavor create/delete, so I'm fixing that by just forcing projects to be loaded in the api before doing the notification
15:17:23 mdbooth The problem is you've combined 2 functions in your move that were previously separate
15:17:31 dansmith gibi: just fyi.. i'll push that up a bit later, it's not a huge rush
15:17:32 mdbooth And remove_volume_connection only called one of them
15:18:10 mdbooth The problem is the interaction of remove_volume_connection and the code from _detach_volume, which we previously didn't call
15:18:38 gibi dansmith: that sounds OK to me. I will keep an eye on that review
15:19:17 dansmith gibi: cool, thanks a bunch :)
15:19:59 mdbooth Incidentally, I really don't like the proliferation of do_all_the_things(but_not_this_thing=True, dont_not_do_that_thing=Maybe, do_this_extra_thing='True')
15:23:13 mdbooth lyarwood: So, I feel like the old _driver_detach_volume is what needs to move, with perhaps a couple of volume_api calls thrown in. Ideally, though, the code in block_device won't concern itself with CONF.host. It'll be called, or it won't be called.
15:25:03 mdbooth lyarwood: That's not 100% thought through, though. There may be practical reasons for dont_do_these_things=['foo', 'bar', 7, None, {}]
15:28:16 lyarwood mdbooth: sorry just went over the diff again, I see your point but just to confirm, instance.host is going to point to the source during a rollback right?
15:28:28 mriedem there doesn't seem to be much they're not willing to do in their company
15:28:29 mdbooth Not during rollback_at_destination
15:28:31 mriedem mdbooth: ^
15:28:48 mdbooth mriedem: Hehe.
15:30:42 mdbooth lyarwood: Sorry, yes. instance.host *will* be the source. I didn't read you right there.
15:30:56 mdbooth However, it's executing on the destination, which is the problem.
15:31:11 mdbooth self.compute_rpcapi.remove_volume_connection(
15:31:11 mdbooth context, instance, bdm.volume_id, dest)
15:31:20 mdbooth In _rollback_live_migration
15:31:25 lyarwood mdbooth: haha right, I just couldn't find the instance.host update
15:31:43 mdbooth So CONF.host != instance.host
15:31:52 mdbooth BUT, we still need to tear it down

Earlier   Later