Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-06-06
17:17:07 openstack bug 1696125 in OpenStack Compute (nova) "Detach interface failed - Unable to detach from guest transient domain (pike)" [High,Confirmed] https://launchpad.net/bugs/1696125
17:17:07 mriedem mlavalle: the e-r query is written against bug 1696125
17:17:40 mlavalle mriedem: ok, cool. thanks for the follow up. will let you deal with it
17:23:11 mriedem dtp: remember the instance.exists periodic hammering thing mentioned at the summit?
17:23:17 mriedem did anyone ever report a bug on that with details?
17:23:27 dtp i'll check w/ kris
17:27:02 efried What's instance.exists?
17:27:51 efried Tell me there's a nova API where I can send a UUID and ask if the record exists, without it actually pulling the whole record and packaging it up and sending it over the wire...
17:28:03 efried I looked and couldn't find one.
17:28:09 mriedem https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L6121
17:28:41 dtp mriedem - he never did
17:29:31 efried okay, yeah, that's not it at all :)
17:30:09 mriedem dtp: ok going to need some details on the issue
17:30:19 mriedem i see that by default the audit period is a month
17:30:25 mriedem so the task is going to run every minute by default,
17:30:40 mriedem and if configured to actually do an audit, it gets the instances that were active on that host within the last month
17:30:45 mriedem and generates this task log thing
17:31:00 mriedem and the instance.exists notification for each instance in that window
17:31:21 mriedem would need to know what your audit period is
17:31:26 dtp k sec
17:32:09 harlowja dtp whats up
17:32:27 harlowja kris is typically in #openstack-operators if u want to find him
17:32:32 harlowja under klindgren
17:32:40 dtp harlowja being a middleman, how you doin
17:32:57 harlowja lol
17:32:57 harlowja okies, trying not to be middleman
17:32:59 harlowja :-P
17:33:04 harlowja lol
17:33:04 harlowja proxyharlow
17:33:10 openstackgerrit Eric Fried proposed openstack/nova master: Trim the fat from InstanceInfo https://review.openstack.org/471146
17:33:58 gibi mriedem: I feel that if there is already a task created for the given audit period then the periodic task just returns
17:34:09 gibi mriedem: here https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L6128
17:35:33 dtp mriedem what's the config key for audit period?
17:35:45 gibi dtp: instance_usage_audit_period
17:35:48 mriedem instance_usage_audit_period
17:36:04 dtp ok, we do not set that value
17:37:07 mriedem ok so by default it's a month
17:37:28 gibi nova logs an info trace at every audit https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L6138
17:37:47 dtp mriedem wait i was wrong - instance_usage_audit_period=hour
17:38:42 mriedem dtp: ok so do you guys see the instance.exists stuff run every minute, like is it sending notifications every minute, or every hour?
17:38:57 mriedem the periodic task will run by default every minute,
17:39:20 mriedem but as noted above, it should exit early if there is already a task log for the same audit period (the last hour)
17:39:28 mriedem but you would expect to get one every hour
17:39:30 mriedem at least
17:40:41 gibi yeah. If you happen to see 'Running instance usage audit for' log every minute then that is a bug. But if it is only every hour then that is the expected behavior
17:41:20 dtp the anecdote i've heard is we're seeing an instance.exist for every VM on the hour
17:43:14 mriedem thats the design
17:43:28 mriedem https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L6157
17:44:12 mriedem dtp: if you configure all of your computes to do this every hour,
17:44:38 mriedem then every hour, it's going to get the list of instances that were/are active on that host within the last hour and send an instance.exists notification for each of them
17:44:50 mriedem for all computes
17:45:12 mriedem how many computes and how many live instances do you guys have?
17:45:44 dtp i think the issue was they're all sending them at the same time, clogging rabbit
17:46:15 dtp checking those #s for you
17:46:59 gibi there is a fuzzy delay config option 'periodic_fuzzy_delay' for periodic tasks to avoid starting them at the same time
17:47:09 dtp 1k computes, 13k instance in our largest control plane
17:48:02 dtp thanks gibi, passing that along
17:50:10 gibi I have to leave for today. I will read the back tomorrow.
17:50:54 openstackgerrit Dan Smith proposed openstack/nova master: Add get_minimum_version_all_cells() helper for service https://review.openstack.org/471431
17:50:55 openstackgerrit Dan Smith proposed openstack/nova master: Fix device metadata service version check for multiple cells https://review.openstack.org/471432
17:50:58 dansmith mriedem: artom ^
17:51:25 dansmith mriedem: artom I was also thinking, can we remove that service version check now? or soon? I think that was ocata stuff, right?
17:52:14 artom dansmith, newton
17:52:57 dansmith oh okay
17:53:14 artom Removing it would assume every deployment supports tagging, no?
17:53:20 artom That seems... imprudent
17:53:36 dansmith the check is to make sure that the compute nodes are upgraded to .. newton
17:54:06 dansmith in ocata, you needn't worry about mitaka computes, and definitely not in pike
17:55:59 artom Ah, I see - compute are at most N-1 behind controller
17:56:03 mriedem dtp: your wussy rabbit can't handle 13K notifications every hour :)
17:56:09 mriedem on the hour
17:56:38 mriedem dtp: well it at least sounds like you're not hitting it every minute or something
17:56:47 mriedem on the hour is kind of by design based on how you've configured things
17:57:05 dtp hehe
17:57:20 mriedem we could stagger those notifications to be 1 instance.exists per periodic (every minute) until the queue is drained, and then get a fresh set,
17:57:22 dansmith dtp: you gonna take that?
17:57:31 dansmith dtp: I think a dance off might be in order
17:57:34 mriedem however, waiting to dequeue might mess up the actual point of the audit period
17:58:00 mriedem like if the audit starts at 1:59 or something, and you got 10 instances, and it takes 10 minutes to dequeue them all
17:58:08 mriedem seems like numbers could get fudgy a bi
17:58:09 mriedem *bit
17:58:26 mriedem or,
17:58:30 mriedem >60 instances,
17:58:37 mriedem with audit period = hour
17:58:38 dtp dansmith twinkie eating contest?
17:58:47 dansmith nice
17:59:00 dansmith I will officiate
17:59:50 dtp ok. i've passed along that randomizer setting. we aren't even sure why we're running it every hour instead of the default, so checking on that too
18:00:24 dtp thanks for the infos
18:00:57 mriedem1 i'm not sure if this working outside in nature is a good idea
18:02:25 openstackgerrit Merged openstack/nova master: XenAPI: Deprecate nicira-iface-id for XenServer VIF https://review.openstack.org/463246
18:06:00 openstackgerrit Merged openstack/nova master: Remove mox from nova/tests/unit/virt/xenapi/test_xenapi.py[1] https://review.openstack.org/438825
18:06:06 oomichi gmann: I could not understand why we need different change on https://review.openstack.org/#/c/413876/39/nova/tests/unit/compute/test_compute_mgr.py in this patch
18:06:37 openstackgerrit Merged openstack/nova master: Update server create networks API reference description for tags https://review.openstack.org/470004
18:07:59 dtp gibi mriedem - we have some periodics we don't want to be too fuzzy. does the fuzzy setting define the range after the start time that it might run?
18:11:14 mriedem dtp: don't really know, it's not documented https://docs.openstack.org/developer/oslo.service/
18:19:03 dtp k. we're just gonna run it less frequently and see how that goes
18:33:01 mriedem dansmith: comments on your bottom change
18:33:08 mriedem for the service min version check
18:33:53 openstackgerrit Matt Riedemann proposed openstack/nova master: libvirt: Check if domain is persistent before detaching devices https://review.openstack.org/441204

Earlier   Later