| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-06-06 | |||
| 17:35:48 | mriedem | instance_usage_audit_period | |
| 17:36:04 | dtp | ok, we do not set that value | |
| 17:37:07 | mriedem | ok so by default it's a month | |
| 17:37:28 | gibi | nova logs an info trace at every audit https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L6138 | |
| 17:37:47 | dtp | mriedem wait i was wrong - instance_usage_audit_period=hour | |
| 17:38:42 | mriedem | dtp: ok so do you guys see the instance.exists stuff run every minute, like is it sending notifications every minute, or every hour? | |
| 17:38:57 | mriedem | the periodic task will run by default every minute, | |
| 17:39:20 | mriedem | but as noted above, it should exit early if there is already a task log for the same audit period (the last hour) | |
| 17:39:28 | mriedem | but you would expect to get one every hour | |
| 17:39:30 | mriedem | at least | |
| 17:40:41 | gibi | yeah. If you happen to see 'Running instance usage audit for' log every minute then that is a bug. But if it is only every hour then that is the expected behavior | |
| 17:41:20 | dtp | the anecdote i've heard is we're seeing an instance.exist for every VM on the hour | |
| 17:43:14 | mriedem | thats the design | |
| 17:43:28 | mriedem | https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L6157 | |
| 17:44:12 | mriedem | dtp: if you configure all of your computes to do this every hour, | |
| 17:44:38 | mriedem | then every hour, it's going to get the list of instances that were/are active on that host within the last hour and send an instance.exists notification for each of them | |
| 17:44:50 | mriedem | for all computes | |
| 17:45:12 | mriedem | how many computes and how many live instances do you guys have? | |
| 17:45:44 | dtp | i think the issue was they're all sending them at the same time, clogging rabbit | |
| 17:46:15 | dtp | checking those #s for you | |
| 17:46:59 | gibi | there is a fuzzy delay config option 'periodic_fuzzy_delay' for periodic tasks to avoid starting them at the same time | |
| 17:47:09 | dtp | 1k computes, 13k instance in our largest control plane | |
| 17:48:02 | dtp | thanks gibi, passing that along | |
| 17:50:10 | gibi | I have to leave for today. I will read the back tomorrow. | |
| 17:50:54 | openstackgerrit | Dan Smith proposed openstack/nova master: Add get_minimum_version_all_cells() helper for service https://review.openstack.org/471431 | |
| 17:50:55 | openstackgerrit | Dan Smith proposed openstack/nova master: Fix device metadata service version check for multiple cells https://review.openstack.org/471432 | |
| 17:50:58 | dansmith | mriedem: artom ^ | |
| 17:51:25 | dansmith | mriedem: artom I was also thinking, can we remove that service version check now? or soon? I think that was ocata stuff, right? | |
| 17:52:14 | artom | dansmith, newton | |
| 17:52:57 | dansmith | oh okay | |
| 17:53:14 | artom | Removing it would assume every deployment supports tagging, no? | |
| 17:53:20 | artom | That seems... imprudent | |
| 17:53:36 | dansmith | the check is to make sure that the compute nodes are upgraded to .. newton | |
| 17:54:06 | dansmith | in ocata, you needn't worry about mitaka computes, and definitely not in pike | |
| 17:55:59 | artom | Ah, I see - compute are at most N-1 behind controller | |
| 17:56:03 | mriedem | dtp: your wussy rabbit can't handle 13K notifications every hour :) | |
| 17:56:09 | mriedem | on the hour | |
| 17:56:38 | mriedem | dtp: well it at least sounds like you're not hitting it every minute or something | |
| 17:56:47 | mriedem | on the hour is kind of by design based on how you've configured things | |
| 17:57:05 | dtp | hehe | |
| 17:57:20 | mriedem | we could stagger those notifications to be 1 instance.exists per periodic (every minute) until the queue is drained, and then get a fresh set, | |
| 17:57:22 | dansmith | dtp: you gonna take that? | |
| 17:57:31 | dansmith | dtp: I think a dance off might be in order | |
| 17:57:34 | mriedem | however, waiting to dequeue might mess up the actual point of the audit period | |
| 17:58:00 | mriedem | like if the audit starts at 1:59 or something, and you got 10 instances, and it takes 10 minutes to dequeue them all | |
| 17:58:08 | mriedem | seems like numbers could get fudgy a bi | |
| 17:58:09 | mriedem | *bit | |
| 17:58:26 | mriedem | or, | |
| 17:58:30 | mriedem | >60 instances, | |
| 17:58:37 | mriedem | with audit period = hour | |
| 17:58:38 | dtp | dansmith twinkie eating contest? | |
| 17:58:47 | dansmith | nice | |
| 17:59:00 | dansmith | I will officiate | |
| 17:59:50 | dtp | ok. i've passed along that randomizer setting. we aren't even sure why we're running it every hour instead of the default, so checking on that too | |
| 18:00:24 | dtp | thanks for the infos | |
| 18:00:57 | mriedem1 | i'm not sure if this working outside in nature is a good idea | |
| 18:02:25 | openstackgerrit | Merged openstack/nova master: XenAPI: Deprecate nicira-iface-id for XenServer VIF https://review.openstack.org/463246 | |
| 18:06:00 | openstackgerrit | Merged openstack/nova master: Remove mox from nova/tests/unit/virt/xenapi/test_xenapi.py[1] https://review.openstack.org/438825 | |
| 18:06:06 | oomichi | gmann: I could not understand why we need different change on https://review.openstack.org/#/c/413876/39/nova/tests/unit/compute/test_compute_mgr.py in this patch | |
| 18:06:37 | openstackgerrit | Merged openstack/nova master: Update server create networks API reference description for tags https://review.openstack.org/470004 | |
| 18:07:59 | dtp | gibi mriedem - we have some periodics we don't want to be too fuzzy. does the fuzzy setting define the range after the start time that it might run? | |
| 18:11:14 | mriedem | dtp: don't really know, it's not documented https://docs.openstack.org/developer/oslo.service/ | |
| 18:19:03 | dtp | k. we're just gonna run it less frequently and see how that goes | |
| 18:33:01 | mriedem | dansmith: comments on your bottom change | |
| 18:33:08 | mriedem | for the service min version check | |
| 18:33:53 | openstackgerrit | Matt Riedemann proposed openstack/nova master: libvirt: Check if domain is persistent before detaching devices https://review.openstack.org/441204 | |
| 18:34:16 | dansmith | mriedem: sorry the sun out here is too bright to read your comments | |
| 18:37:11 | mriedem | the sound of large grading equipment can also be distracting | |
| 18:37:20 | sdague | mriedem: rebuild get a lot worse? | |
| 18:37:21 | sdague | https://review.openstack.org/#/c/471082/ | |
| 18:37:42 | mriedem | sdague: it failed in my novaclient change too | |
| 18:37:43 | mriedem | timed out | |
| 18:37:46 | mriedem | i don't know what's going on there | |
| 18:37:53 | sdague | yeh, that's double fail on a reno change | |
| 18:40:22 | mriedem | i don't see any recent changes to nova or novaclient related to rebuild | |
| 18:43:09 | mriedem | it also doesn't appear to be new | |
| 18:43:38 | mriedem | i think i have looked into that _poll_for_status method in novaclient before and it seems a bit wonky | |
| 18:45:04 | sdague | http://logs.openstack.org/82/471082/1/check/gate-novaclient-dsvm-functional-neutron-ubuntu-xenial/68a19f4/logs/screen-n-cpu.txt.gz#_Jun_06_17_35_44_082400 | |
| 18:45:26 | sdague | yeh, I don't know, shouldn't we get an instance started emit? | |
| 18:45:55 | mriedem | dansmith: there is a child wailing now - this outdoors thing isn't working | |
| 18:46:46 | dansmith | mriedem: don't they have guns in minnesota? | |
| 18:47:52 | mriedem | they do | |
| 18:48:04 | mriedem | and hotdish! | |
| 18:49:14 | mriedem | sdague: so it's req-5b3770c1-d332-4875-8933-97de8a9890b4 right? | |
| 18:50:46 | sdague | I actually think it's req-f0668b70-fac7-4dae-b3b3-afba6c36ec77 | |
| 18:51:09 | sdague | one sec, I just closed all my tabs | |
| 18:51:42 | sdague | yeh req-f0668b70-fac7-4dae-b3b3-afba6c36ec77 is the rebuild | |
| 18:51:49 | sdague | http://logs.openstack.org/82/471082/1/check/gate-novaclient-dsvm-functional-neutron-ubuntu-xenial/68a19f4/logs/screen-n-api.txt.gz#_Jun_06_17_35_38_992164 | |
| 18:52:01 | mriedem | sdague: http://logs.openstack.org/82/471082/1/gate/gate-novaclient-dsvm-functional-neutron-ubuntu-xenial/796acb7/logs/screen-n-cpu.txt.gz#_Jun_06_13_14_02_210879 | |
| 18:52:09 | mriedem | i'm looking at the gate queue job | |
| 18:52:11 | mriedem | you're looking at check | |
| 18:52:12 | mriedem | :) | |
| 18:53:01 | openstackgerrit | Merged openstack/nova master: Send request_id on glance calls https://review.openstack.org/467242 | |
| 18:53:10 | sdague | mriedem: ah, sure :) | |
| 18:55:04 | openstackgerrit | Merged openstack/nova master: Sync os-traits to Traits database table https://review.openstack.org/469578 | |
| 18:55:28 | mriedem | Jun 06 13:14:02.594292 ubuntu-xenial-infracloud-chocolate-9158824 nova-compute[20994]: DEBUG nova.virt.libvirt.driver [None req-5b3770c1-d332-4875-8933-97de8a9890b4 admin admin] [instance: 573258a4-9416-4e13-a765-7c90683f3526] Shutting down instance from state 1: NotImplementedError {{(pid=20994) _clean_shutdown /opt/stack/new/nova/nova/virt/libvirt/driver.py:2524}} | |
| 18:55:28 | mriedem | this is a weird message | |
| 18:55:40 | openstackgerrit | Merged openstack/nova master: Use VIR_DOMAIN_BLOCK_REBASE_COPY_DEV when rebasing https://review.openstack.org/465205 | |
| 18:57:39 | artom | mriedem, looks like something doesn't implement get_power_states | |
| 18:58:13 | mriedem | or it's something in a repr | |