| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-06-06 | |||
| 16:11:18 | armax | surely 6.3.0 might work prior to newton | |
| 16:11:26 | armax | server-side I mean | |
| 16:11:35 | sdague | mriedem: yeh, well, regardless, it's good enough for the nova patch to move forward | |
| 16:11:40 | armax | but if you ain’t test it you ain’t say it for soor | |
| 16:11:42 | armax | sure | |
| 16:11:55 | sdague | armax: this was mostly fighting against pecan magic? | |
| 16:12:00 | mriedem | sdague: i know this isn't nova's problem | |
| 16:12:15 | mriedem | i just feel bad for the puppies and kittens that now have to die | |
| 16:12:26 | mriedem | more bad for the kittens because, you know, dogs... | |
| 16:12:58 | armax | sdague: I don’t believe so, the issue seemed to be latent and just surface because of the client change that dropped the .json suffix on the API URIs | |
| 16:20:04 | efried | mriedem FYI PowerVM support matrix adds are ready: https://review.openstack.org/#/c/470999/ | |
| 16:20:21 | mriedem | oh i know | |
| 16:20:24 | mriedem | it's starred | |
| 16:20:44 | efried | mriedem On another topic, thinking it may be a good idea to blast the ML for the InstanceInfo change, since it would affect OOT drivers. | |
| 16:21:33 | mriedem | go head | |
| 16:21:35 | mriedem | *ahead | |
| 16:26:34 | openstackgerrit | Dan Smith proposed openstack/nova master: Fix and optimize external_events for multiple cells https://review.openstack.org/445142 | |
| 16:32:49 | openstackgerrit | Merged openstack/nova master: Remove DeviceIsBusy exception https://review.openstack.org/465723 | |
| 16:33:28 | openstackgerrit | Merged openstack/nova master: libvirt: Use config types to parse XML for root disk https://review.openstack.org/411941 | |
| 16:34:05 | openstackgerrit | Merged openstack/nova master: Remove oslo.config deprecated parameter enforce_type https://review.openstack.org/462364 | |
| 16:34:41 | openstackgerrit | Merged openstack/nova master: Modify incorrect comment on return_reservation_id https://review.openstack.org/471278 | |
| 16:34:58 | mhenkel_ | mriedem: would you mind to check https://blueprints.launchpad.net/openstack/?searchtext=opencontrail-nova-vif-plugin and approve? | |
| 16:35:21 | openstackgerrit | Merged openstack/nova master: Remove incorrect comments in multiple_create https://review.openstack.org/471273 | |
| 16:51:37 | openstackgerrit | Merged openstack/nova master: Support tag instances when boot(2/4) https://review.openstack.org/459569 | |
| 16:52:16 | openstackgerrit | Merged openstack/nova master: [placement] adjust resource provider links by microversion https://review.openstack.org/468923 | |
| 16:52:56 | openstackgerrit | Merged openstack/nova master: make route and controller in alpha sequence https://review.openstack.org/469380 | |
| 16:53:31 | openstackgerrit | Merged openstack/nova master: ComputeDriver.get_info not limited to inst name https://review.openstack.org/471106 | |
| 16:54:07 | openstackgerrit | Merged openstack/nova master: Refactor a test method including 7 test cases https://review.openstack.org/395436 | |
| 16:54:44 | openstackgerrit | Merged openstack/nova master: Remove usage of parameter enforce_type https://review.openstack.org/469745 | |
| 16:55:23 | openstackgerrit | Merged openstack/nova master: Pass a list of instance UUIDs to scheduler https://review.openstack.org/465171 | |
| 16:55:29 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Send out notifications when instance tags changed https://review.openstack.org/459923 | |
| 16:55:30 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Move notifications/objects/test_base.py https://review.openstack.org/471425 | |
| 17:11:59 | mlavalle | mriedem: so this https://bugs.launchpad.net/nova/+bug/1696125 | |
| 17:11:59 | openstack | Launchpad bug 1696125 in OpenStack Compute (nova) "Detach interface failed - Unable to detach from guest transient domain (pike)" [High,Confirmed] | |
| 17:12:12 | mlavalle | and https://bugs.launchpad.net/neutron/+bug/1696006 | |
| 17:12:12 | openstack | Launchpad bug 1696006 in neutron "Libvirt fails to detach network interface with Linux bridge" [Critical,New] | |
| 17:12:18 | mlavalle | are the same, right? | |
| 17:16:51 | mriedem | mlavalle: looks like it | |
| 17:17:07 | mriedem | mlavalle: the e-r query is written against bug 1696125 | |
| 17:17:07 | openstack | bug 1696125 in OpenStack Compute (nova) "Detach interface failed - Unable to detach from guest transient domain (pike)" [High,Confirmed] https://launchpad.net/bugs/1696125 | |
| 17:17:40 | mlavalle | mriedem: ok, cool. thanks for the follow up. will let you deal with it | |
| 17:23:11 | mriedem | dtp: remember the instance.exists periodic hammering thing mentioned at the summit? | |
| 17:23:17 | mriedem | did anyone ever report a bug on that with details? | |
| 17:23:27 | dtp | i'll check w/ kris | |
| 17:27:02 | efried | What's instance.exists? | |
| 17:27:51 | efried | Tell me there's a nova API where I can send a UUID and ask if the record exists, without it actually pulling the whole record and packaging it up and sending it over the wire... | |
| 17:28:03 | efried | I looked and couldn't find one. | |
| 17:28:09 | mriedem | https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L6121 | |
| 17:28:41 | dtp | mriedem - he never did | |
| 17:29:31 | efried | okay, yeah, that's not it at all :) | |
| 17:30:09 | mriedem | dtp: ok going to need some details on the issue | |
| 17:30:19 | mriedem | i see that by default the audit period is a month | |
| 17:30:25 | mriedem | so the task is going to run every minute by default, | |
| 17:30:40 | mriedem | and if configured to actually do an audit, it gets the instances that were active on that host within the last month | |
| 17:30:45 | mriedem | and generates this task log thing | |
| 17:31:00 | mriedem | and the instance.exists notification for each instance in that window | |
| 17:31:21 | mriedem | would need to know what your audit period is | |
| 17:31:26 | dtp | k sec | |
| 17:32:09 | harlowja | dtp whats up | |
| 17:32:27 | harlowja | kris is typically in #openstack-operators if u want to find him | |
| 17:32:32 | harlowja | under klindgren | |
| 17:32:40 | dtp | harlowja being a middleman, how you doin | |
| 17:32:57 | harlowja | okies, trying not to be middleman | |
| 17:32:57 | harlowja | lol | |
| 17:32:59 | harlowja | :-P | |
| 17:33:04 | harlowja | proxyharlow | |
| 17:33:04 | harlowja | lol | |
| 17:33:10 | openstackgerrit | Eric Fried proposed openstack/nova master: Trim the fat from InstanceInfo https://review.openstack.org/471146 | |
| 17:33:58 | gibi | mriedem: I feel that if there is already a task created for the given audit period then the periodic task just returns | |
| 17:34:09 | gibi | mriedem: here https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L6128 | |
| 17:35:33 | dtp | mriedem what's the config key for audit period? | |
| 17:35:45 | gibi | dtp: instance_usage_audit_period | |
| 17:35:48 | mriedem | instance_usage_audit_period | |
| 17:36:04 | dtp | ok, we do not set that value | |
| 17:37:07 | mriedem | ok so by default it's a month | |
| 17:37:28 | gibi | nova logs an info trace at every audit https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L6138 | |
| 17:37:47 | dtp | mriedem wait i was wrong - instance_usage_audit_period=hour | |
| 17:38:42 | mriedem | dtp: ok so do you guys see the instance.exists stuff run every minute, like is it sending notifications every minute, or every hour? | |
| 17:38:57 | mriedem | the periodic task will run by default every minute, | |
| 17:39:20 | mriedem | but as noted above, it should exit early if there is already a task log for the same audit period (the last hour) | |
| 17:39:28 | mriedem | but you would expect to get one every hour | |
| 17:39:30 | mriedem | at least | |
| 17:40:41 | gibi | yeah. If you happen to see 'Running instance usage audit for' log every minute then that is a bug. But if it is only every hour then that is the expected behavior | |
| 17:41:20 | dtp | the anecdote i've heard is we're seeing an instance.exist for every VM on the hour | |
| 17:43:14 | mriedem | thats the design | |
| 17:43:28 | mriedem | https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L6157 | |
| 17:44:12 | mriedem | dtp: if you configure all of your computes to do this every hour, | |
| 17:44:38 | mriedem | then every hour, it's going to get the list of instances that were/are active on that host within the last hour and send an instance.exists notification for each of them | |
| 17:44:50 | mriedem | for all computes | |
| 17:45:12 | mriedem | how many computes and how many live instances do you guys have? | |
| 17:45:44 | dtp | i think the issue was they're all sending them at the same time, clogging rabbit | |
| 17:46:15 | dtp | checking those #s for you | |
| 17:46:59 | gibi | there is a fuzzy delay config option 'periodic_fuzzy_delay' for periodic tasks to avoid starting them at the same time | |
| 17:47:09 | dtp | 1k computes, 13k instance in our largest control plane | |
| 17:48:02 | dtp | thanks gibi, passing that along | |
| 17:50:10 | gibi | I have to leave for today. I will read the back tomorrow. | |
| 17:50:54 | openstackgerrit | Dan Smith proposed openstack/nova master: Add get_minimum_version_all_cells() helper for service https://review.openstack.org/471431 | |
| 17:50:55 | openstackgerrit | Dan Smith proposed openstack/nova master: Fix device metadata service version check for multiple cells https://review.openstack.org/471432 | |
| 17:50:58 | dansmith | mriedem: artom ^ | |
| 17:51:25 | dansmith | mriedem: artom I was also thinking, can we remove that service version check now? or soon? I think that was ocata stuff, right? | |