| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-03-16 | |||
| 14:40:41 | mriedem | lyarwood: yeah that page doesn't help me out here, | |
| 14:40:57 | mriedem | because that implies nova is making a decision when creating the domain as to whether or not it's persistent, right? | |
| 14:41:27 | mriedem | maybe we are, but can you point out where we decide that? | |
| 14:41:50 | lyarwood | mriedem: just checking but for spawn() I think they are always persistent, live mighration is one of the few cases I know where we start a transient domain on the destination | |
| 14:41:53 | lbragstad | aunnam so what do you have locally? | |
| 14:42:21 | aunnam | lbragstad, so since I commented out the rule the parser is not getting the policy rules and its failing | |
| 14:42:48 | lbragstad | aunnam can you paste a `git diff`? | |
| 14:43:49 | openstackgerrit | Sujitha proposed openstack/nova master: Adding tags field to InstancePayload https://review.openstack.org/407228 | |
| 14:44:04 | aunnam | lbragstad, so was thinking to write to policy-sample.yaml file without calling _generate_sample https://github.com/openstack/oslo.policy/blob/master/oslo_policy/tests/test_generator.py#L252 | |
| 14:44:24 | openstackgerrit | Gábor Antal proposed openstack/nova master: Transform instance.reboot.error notification https://review.openstack.org/411791 | |
| 14:44:40 | aunnam | lbragstad, i am not sure if it is the right way so waiting on that | |
| 14:44:52 | mhenkel | hello jaypipes | |
| 14:44:54 | lyarwood | mriedem: https://github.com/openstack/nova/blob/master/nova/virt/libvirt/host.py#L833 - defineXML == creates a persistent domain | |
| 14:45:08 | mriedem | lyarwood: ok. another odd thing is in the stacktrace and error, if we were re-raising DeviceNotFound from _try_detach_device, we should see oslo.utils log the original libvirtError before reraising the new exception http://git.openstack.org/cgit/openstack/oslo.utils/tree/oslo_utils/excutils.py#n212 | |
| 14:45:12 | aunnam | lbragstad, thats what found when I look into code | |
| 14:45:12 | mriedem | lyarwood: but i don't see that happen | |
| 14:45:24 | mriedem | instead we get here http://git.openstack.org/cgit/openstack/oslo.utils/tree/oslo_utils/excutils.py#n220 | |
| 14:45:25 | lbragstad | aunnam i might have missed it, but what changes have you made locally? | |
| 14:45:30 | jaypipes | mhenkel: hi! currently on calls for another hour or so... gimme a bit? :) | |
| 14:45:37 | mriedem | i wonder if we're using a newer eventlet that's causing some context switching bugs | |
| 14:45:57 | aunnam | lbragstad, just commented out the rule, that all rule, haven't changed the tests yet | |
| 14:46:03 | mhenkel | jaypipes: sure thing, ping me when you have a minute or two | |
| 14:46:30 | openstackgerrit | Sujitha proposed openstack/nova master: Change tags to default field in Instance object. https://review.openstack.org/415298 | |
| 14:46:33 | aunnam | lbragstad, commented the rule in here https://github.com/openstack/oslo.policy/blob/master/oslo_policy/generator.py#L109 | |
| 14:46:44 | openstackgerrit | Sujitha proposed openstack/nova master: Reduce calls to load_tags() to 0 https://review.openstack.org/435146 | |
| 14:47:15 | mriedem | no that's not it, last time we bumped eventlet up was last may | |
| 14:48:16 | lbragstad | aunnam alright - this is what i have locally http://cdn.pasteraw.com/3u2pj6gigswl55mkeqdbojeifp6gzf4 | |
| 14:48:30 | lbragstad | which is showing me 6 failures | |
| 14:48:39 | lyarwood | mriedem: do you have a logstash query for this already btw? | |
| 14:49:08 | mriedem | lyarwood: just this: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22AttachVolumeShelveTestJSON%5C%22%20AND%20message%3A%5C%22Failed%20to%20detach%20volume%5C%22%20AND%20message%3A%5C%22libvirtError%3A%20End%20of%20file%20while%20reading%20data%3A%20Input%2Foutput%20error%5C%22%20AND%20tags%3A%5C%22screen-n-cpu.txt%5C%22&from=7d | |
| 14:49:57 | aunnam | lbragstad, ya that's the same think that i have | |
| 14:49:59 | mriedem | lyarwood: did you create a bug? i think we need some more logging in detach_device_with_retry because i can't make out the stacktrace from that method given all of the context managers | |
| 14:50:05 | lbragstad | aunnam cool | |
| 14:50:59 | lyarwood | mriedem: thanks, not yet just waiting on devstack finishing locally before I try to reproduce | |
| 14:51:33 | gibi | sdague: responded on https://review.openstack.org/#/c/440580/5/specs/pike/approved/scheduler-hints-in-server-details.rst I can bargain on what hints the API will return | |
| 14:52:34 | openstackgerrit | Sujitha proposed openstack/nova master: Adding auto_disk_config field to InstancePayload https://review.openstack.org/419185 | |
| 14:53:13 | openstackgerrit | Béla Vancsics proposed openstack/nova master: Reduce code complexity - libvirt/config.py https://review.openstack.org/359879 | |
| 14:53:46 | mriedem | lyarwood: since we dont see "Original exception being dropped" i'm wondering if we're hitting _try_detach_device the first time, it's ok, and then when we try the 2nd time, it fails with that i/o error | |
| 14:54:07 | mriedem | that's why i want more debug logging in there to know if we're calling _try_detach_device before we get into the retry loop | |
| 14:54:27 | mriedem | because i'm guessing that the i/o error thing on an already detached device is newish in libvirt 1.3.1 | |
| 14:54:47 | mdbooth | lyarwood mriedem: Just finished reading scrollback. Sounds fun. | |
| 14:55:21 | mriedem | mdbooth: definitely not what i planned on doing for my first 2 hours this morning :) | |
| 14:55:28 | mriedem | gate spelunking | |
| 14:57:54 | dansmith | gibi: if you're still around, it'd be good to get your ack on this too: https://review.openstack.org/#/c/446053/4 | |
| 14:58:27 | Jack_Iv | Hey Folks! Since Nova 13 the hooks are DEPRECATED, but what can I use instead? | |
| 14:58:52 | Jack_Iv | I need to run some code, pre- and post- instance build | |
| 14:59:40 | mriedem | Jack_Iv: there is the dynamic vendordata metadata api and notifications | |
| 14:59:46 | mriedem | or upstream whatever your use case is | |
| 14:59:58 | mriedem | https://docs.openstack.org/developer/nova/vendordata.html | |
| 15:00:05 | mriedem | https://docs.openstack.org/developer/nova/notifications.html | |
| 15:00:35 | mdbooth | Jack_Iv: What is your use case, btw? | |
| 15:00:49 | gibi | dansmith: looking... | |
| 15:01:14 | Jack_Iv | I need to execute some code on compute nodes, after vm is UP and after termination | |
| 15:01:18 | Jack_Iv | mdbooth: ^ | |
| 15:01:31 | mdbooth | Jack_Iv: Right, but what for? | |
| 15:01:40 | Jack_Iv | let's say, update some iptables rules | |
| 15:01:41 | mdbooth | What does the code do? | |
| 15:01:54 | mdbooth | On the host? | |
| 15:01:57 | Jack_Iv | right | |
| 15:02:00 | kashyap | Yeah, I'm curious, too. About the use case | |
| 15:02:13 | mriedem | lyarwood: https://bugs.launchpad.net/nova/+bug/1673483 | |
| 15:02:13 | openstack | Launchpad bug 1673483 in OpenStack Compute (nova) "libvirt: test_attach_volume_shelved_or_offload_server times out waiting for device detach (which fails)" [Undecided,New] | |
| 15:02:14 | openstackgerrit | Moshe Levi proposed openstack/nova master: HW offload support for openvswitch https://review.openstack.org/398265 | |
| 15:02:31 | mdbooth | Jack_Iv: Is this secret squirrel, or can you share the whole use case? I'm really curious as to why. | |
| 15:03:07 | kashyap | From the an old mailing list discussion on hooks, I see: | |
| 15:03:08 | Jack_Iv | I want to update some iptables rules after VM is up and delete those rules after termination | |
| 15:03:13 | kashyap | There's three core scenarios for hooks | |
| 15:03:13 | kashyap | 1. Modifying some aspect of the Nova operation | |
| 15:03:13 | kashyap | 2. Triggering an external action synchronously to some Nova operation | |
| 15:03:13 | kashyap | 3. Triggering an external action asynchronously to some Nova operation | |
| 15:03:20 | kashyap | [From danpb] | |
| 15:03:35 | mriedem | Jack_Iv: you can't do that with security group rules? | |
| 15:03:41 | mdbooth | Jack_Iv: Right. Just trying to understand why, and if there's some other way to achieve what you want. | |
| 15:03:47 | mdbooth | e.g. what mriedem said | |
| 15:04:11 | mriedem | Jack_Iv: i.e. listen for the notifications from nova, instance.create.start and instance.create.end, and when you get those notifications, adjust the secgroup rules in neutron | |
| 15:04:13 | Jack_Iv | No, because clients can just edit security group rules | |
| 15:04:40 | mriedem | Jack_Iv: so change the neutron api to make admin-only rules? | |
| 15:04:54 | mriedem | Jack_Iv: at the PTG we talked about a concept of service user locks on instances, | |
| 15:05:05 | mriedem | you could make a similar case for service/admin level "locks" on security group rules | |
| 15:05:25 | dansmith | I thought neutron had some base "provider rules" concept for this | |
| 15:05:36 | mriedem | dansmith: maybe they do | |
| 15:05:41 | mriedem | Jack_Iv: have you talked to the neutron people? | |
| 15:05:47 | Jack_Iv | not yet | |
| 15:05:47 | mriedem | Jack_Iv: which version of openstack are you on? | |
| 15:05:50 | Jack_Iv | newton | |
| 15:06:57 | mriedem | kevinbenton: does neutron have a concept of provider rules for security groups that the tenant user can't change? | |
| 15:07:58 | mriedem | Jack_Iv: i think that's probably the thread you need to pull on though | |
| 15:08:14 | mriedem | listen for notifications from nova for the instance build lifecycle, and then adjust rules via the networking API as needed | |
| 15:09:41 | mriedem | no vmware people around huh | |
| 15:09:51 | mriedem | their CI results come in, if at all, about a week late | |
| 15:15:50 | openstackgerrit | Lee Yarwood proposed openstack/nova master: compute: Move detach logic from manager into driver BDM https://review.openstack.org/439520 | |
| 15:15:51 | openstackgerrit | Lee Yarwood proposed openstack/nova master: compute: Only destroy BDMs after successful detach call https://review.openstack.org/440693 | |
| 15:16:13 | gibi | dansmith: I'm +1 on https://review.openstack.org/#/c/446053 | |
| 15:16:35 | dansmith | gibi: awesome, thanks for looking at that | |
| 15:16:40 | dansmith | gibi: on the other patch, | |
| 15:16:44 | lyarwood | mdbooth: ^ updated removing some of the nits, just to confirm the LM rollback issue isn't new right? | |
| 15:17:02 | mdbooth | lyarwood: No, the LM rollback issue looks new | |
| 15:17:13 | dansmith | gibi: the change I made prevents projects from showing up on actual flavor create/delete, so I'm fixing that by just forcing projects to be loaded in the api before doing the notification | |
| 15:17:23 | mdbooth | The problem is you've combined 2 functions in your move that were previously separate | |
| 15:17:31 | dansmith | gibi: just fyi.. i'll push that up a bit later, it's not a huge rush | |