| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-06-06 | |||
| 17:57:05 | dtp | hehe | |
| 17:57:20 | mriedem | we could stagger those notifications to be 1 instance.exists per periodic (every minute) until the queue is drained, and then get a fresh set, | |
| 17:57:22 | dansmith | dtp: you gonna take that? | |
| 17:57:31 | dansmith | dtp: I think a dance off might be in order | |
| 17:57:34 | mriedem | however, waiting to dequeue might mess up the actual point of the audit period | |
| 17:58:00 | mriedem | like if the audit starts at 1:59 or something, and you got 10 instances, and it takes 10 minutes to dequeue them all | |
| 17:58:08 | mriedem | seems like numbers could get fudgy a bi | |
| 17:58:09 | mriedem | *bit | |
| 17:58:26 | mriedem | or, | |
| 17:58:30 | mriedem | >60 instances, | |
| 17:58:37 | mriedem | with audit period = hour | |
| 17:58:38 | dtp | dansmith twinkie eating contest? | |
| 17:58:47 | dansmith | nice | |
| 17:59:00 | dansmith | I will officiate | |
| 17:59:50 | dtp | ok. i've passed along that randomizer setting. we aren't even sure why we're running it every hour instead of the default, so checking on that too | |
| 18:00:24 | dtp | thanks for the infos | |
| 18:00:57 | mriedem1 | i'm not sure if this working outside in nature is a good idea | |
| 18:02:25 | openstackgerrit | Merged openstack/nova master: XenAPI: Deprecate nicira-iface-id for XenServer VIF https://review.openstack.org/463246 | |
| 18:06:00 | openstackgerrit | Merged openstack/nova master: Remove mox from nova/tests/unit/virt/xenapi/test_xenapi.py[1] https://review.openstack.org/438825 | |
| 18:06:06 | oomichi | gmann: I could not understand why we need different change on https://review.openstack.org/#/c/413876/39/nova/tests/unit/compute/test_compute_mgr.py in this patch | |
| 18:06:37 | openstackgerrit | Merged openstack/nova master: Update server create networks API reference description for tags https://review.openstack.org/470004 | |
| 18:07:59 | dtp | gibi mriedem - we have some periodics we don't want to be too fuzzy. does the fuzzy setting define the range after the start time that it might run? | |
| 18:11:14 | mriedem | dtp: don't really know, it's not documented https://docs.openstack.org/developer/oslo.service/ | |
| 18:19:03 | dtp | k. we're just gonna run it less frequently and see how that goes | |
| 18:33:01 | mriedem | dansmith: comments on your bottom change | |
| 18:33:08 | mriedem | for the service min version check | |
| 18:33:53 | openstackgerrit | Matt Riedemann proposed openstack/nova master: libvirt: Check if domain is persistent before detaching devices https://review.openstack.org/441204 | |
| 18:34:16 | dansmith | mriedem: sorry the sun out here is too bright to read your comments | |
| 18:37:11 | mriedem | the sound of large grading equipment can also be distracting | |
| 18:37:20 | sdague | mriedem: rebuild get a lot worse? | |
| 18:37:21 | sdague | https://review.openstack.org/#/c/471082/ | |
| 18:37:42 | mriedem | sdague: it failed in my novaclient change too | |
| 18:37:43 | mriedem | timed out | |
| 18:37:46 | mriedem | i don't know what's going on there | |
| 18:37:53 | sdague | yeh, that's double fail on a reno change | |
| 18:40:22 | mriedem | i don't see any recent changes to nova or novaclient related to rebuild | |
| 18:43:09 | mriedem | it also doesn't appear to be new | |
| 18:43:38 | mriedem | i think i have looked into that _poll_for_status method in novaclient before and it seems a bit wonky | |
| 18:45:04 | sdague | http://logs.openstack.org/82/471082/1/check/gate-novaclient-dsvm-functional-neutron-ubuntu-xenial/68a19f4/logs/screen-n-cpu.txt.gz#_Jun_06_17_35_44_082400 | |
| 18:45:26 | sdague | yeh, I don't know, shouldn't we get an instance started emit? | |
| 18:45:55 | mriedem | dansmith: there is a child wailing now - this outdoors thing isn't working | |
| 18:46:46 | dansmith | mriedem: don't they have guns in minnesota? | |
| 18:47:52 | mriedem | they do | |
| 18:48:04 | mriedem | and hotdish! | |
| 18:49:14 | mriedem | sdague: so it's req-5b3770c1-d332-4875-8933-97de8a9890b4 right? | |
| 18:50:46 | sdague | I actually think it's req-f0668b70-fac7-4dae-b3b3-afba6c36ec77 | |
| 18:51:09 | sdague | one sec, I just closed all my tabs | |
| 18:51:42 | sdague | yeh req-f0668b70-fac7-4dae-b3b3-afba6c36ec77 is the rebuild | |
| 18:51:49 | sdague | http://logs.openstack.org/82/471082/1/check/gate-novaclient-dsvm-functional-neutron-ubuntu-xenial/68a19f4/logs/screen-n-api.txt.gz#_Jun_06_17_35_38_992164 | |
| 18:52:01 | mriedem | sdague: http://logs.openstack.org/82/471082/1/gate/gate-novaclient-dsvm-functional-neutron-ubuntu-xenial/796acb7/logs/screen-n-cpu.txt.gz#_Jun_06_13_14_02_210879 | |
| 18:52:09 | mriedem | i'm looking at the gate queue job | |
| 18:52:11 | mriedem | you're looking at check | |
| 18:52:12 | mriedem | :) | |
| 18:53:01 | openstackgerrit | Merged openstack/nova master: Send request_id on glance calls https://review.openstack.org/467242 | |
| 18:53:10 | sdague | mriedem: ah, sure :) | |
| 18:55:04 | openstackgerrit | Merged openstack/nova master: Sync os-traits to Traits database table https://review.openstack.org/469578 | |
| 18:55:28 | mriedem | Jun 06 13:14:02.594292 ubuntu-xenial-infracloud-chocolate-9158824 nova-compute[20994]: DEBUG nova.virt.libvirt.driver [None req-5b3770c1-d332-4875-8933-97de8a9890b4 admin admin] [instance: 573258a4-9416-4e13-a765-7c90683f3526] Shutting down instance from state 1: NotImplementedError {{(pid=20994) _clean_shutdown /opt/stack/new/nova/nova/virt/libvirt/driver.py:2524}} | |
| 18:55:28 | mriedem | this is a weird message | |
| 18:55:40 | openstackgerrit | Merged openstack/nova master: Use VIR_DOMAIN_BLOCK_REBASE_COPY_DEV when rebasing https://review.openstack.org/465205 | |
| 18:57:39 | artom | mriedem, looks like something doesn't implement get_power_states | |
| 18:58:13 | mriedem | or it's something in a repr | |
| 18:58:47 | artom | That... would make more sense | |
| 18:58:48 | mriedem | it should be a InstancePowerState object | |
| 19:00:48 | mriedem | weird, ": NotImplementedError" shows up a lot in the logs | |
| 19:00:55 | mriedem | http://logs.openstack.org/82/471082/1/gate/gate-novaclient-dsvm-functional-neutron-ubuntu-xenial/796acb7/logs/screen-n-cpu.txt.gz | |
| 19:01:13 | mriedem | maybe it's something in oslo.log? | |
| 19:01:48 | mriedem | sdague: wasn't there something done with oslo.log recently to put extra metadata in messages or something? | |
| 19:02:58 | mriedem | this error summary stuff | |
| 19:03:26 | mriedem | + error." | |
| 19:03:26 | mriedem | "If the log format string - does not include ``%(error_summary)s`` it will be appended to the end + does not include ``%(error_summary)s`` the error summary will be - of the line automatically, only if there is an error. + appended to the end of the line automatically, only if there is an | |
| 19:03:49 | sdague | yeh, the NotImplementedError stuff is bleeding through there, that's going to need to be a dhellman ask | |
| 19:04:03 | sdague | that was only supposed to happen if we were in an active exception block | |
| 19:04:40 | mriedem | i'm pinging him in -oslo | |
| 19:04:44 | sdague | I think that's irrelevant and just bleed over because that was the last exception thrown | |
| 19:04:47 | clarkb | not to interrupt but looks like the nova python35 job is failing a lot and reseting things in the gate? | |
| 19:05:04 | sdague | clarkb: not really | |
| 19:05:11 | clarkb | http://logs.openstack.org/71/465171/11/gate/gate-nova-python35/2fcc59a/console.html#_2017-06-06_14_57_13_022429 and http://logs.openstack.org/89/418489/10/gate/gate-nova-tox-functional-py35-ubuntu-xenial/51b5776/console.html#_2017-06-06_13_35_33_574055 appear to be two different classes of error | |
| 19:05:30 | sdague | clarkb: there was one bad patch, and because the gate was taking a long time and other things were running a lot, it managed to run and fail a lot in the gate | |
| 19:05:35 | clarkb | gotcha | |
| 19:05:51 | sdague | this is the params one? | |
| 19:06:02 | sdague | man, the log server seems slow today | |
| 19:06:19 | mriedem | i blame systemd | |
| 19:06:26 | mriedem | er journald :) | |
| 19:06:30 | mriedem | somethingd | |
| 19:06:32 | sdague | yeh, the wrong params to __init__ was just a patch with out of date tests | |
| 19:06:33 | clarkb | sdague: mriedem its less that the log server is slow and that its a 12MB html file | |
| 19:06:44 | clarkb | sdague: the second one is a different error fwiw | |
| 19:06:50 | mriedem | i kid | |
| 19:07:45 | sdague | clarkb: it's actually the same thing | |
| 19:08:00 | sdague | https://review.openstack.org/#/c/418489/ needs work | |
| 19:08:08 | sdague | stuff was probably behind it and failing | |
| 19:09:44 | mriedem | here is the oslo.log bug https://bugs.launchpad.net/oslo.log/+bug/1696213 | |
| 19:09:45 | openstack | Launchpad bug 1696213 in oslo.log "error summary formatter is putting ": NotImplementedError" in non-error log messages" [Undecided,New] | |
| 19:10:03 | mriedem | back to this rebuild failure | |
| 19:14:44 | mriedem | http://logs.openstack.org/82/471082/1/gate/gate-novaclient-dsvm-functional-neutron-ubuntu-xenial/796acb7/logs/screen-n-cpu.txt.gz#_Jun_06_13_19_17_300332 | |
| 19:14:44 | mriedem | sdague: i see the problem | |
| 19:14:49 | mriedem | vif plugging callback timeout | |
| 19:14:50 | mriedem | during rebuild | |
| 19:14:53 | mriedem | so the instance never goes active | |
| 19:15:26 | mriedem | which someone from heat reported a rebuild bug about vif plug timeouts recently | |