| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-11-15 | |||
| 14:29:48 | dansmith | artom: are you looking for help setting it up? If so, I'm sure the infra people are who you want to talk to | |
| 14:30:02 | artom | dansmith, yeah, that conversation is already happening - migi as well | |
| 14:30:12 | dansmith | ack | |
| 14:30:42 | mriedem | i wish the dell emc people would have asked b/c their's comments and always fails | |
| 14:32:01 | bauzas | gibi: I'm about to provide a new revision for the audit command, would it be possible for you to check it with the bandwidth-aware instances ? | |
| 14:32:15 | bauzas | (at least once I'm done with reno) | |
| 14:40:58 | gibi | bauzas: sure | |
| 14:41:28 | gibi | bauzas: If I time out on it today then I will do it next week | |
| 14:41:36 | bauzas | gibi: thanks | |
| 14:41:46 | bauzas | hopefully, I'll push it in 20 mins | |
| 14:47:31 | openstack | bug 1852610 in OpenStack Compute (nova) "API allows source compute service/node deletion while instances are pending a resize confirm/revert" [Medium,In progress] https://launchpad.net/bugs/1852610 - Assigned to Matt Riedemann (mriedem) | |
| 14:47:31 | openstackgerrit | Merged openstack/nova master: Add functional recreate revert resize test for bug 1852610 https://review.opendev.org/694364 | |
| 15:03:21 | openstackgerrit | Sylvain Bauza proposed openstack/nova master: Add a placement audit command https://review.opendev.org/670112 | |
| 15:12:16 | openstack | Launchpad bug 1852759 in OpenStack Compute (nova) rocky "false error log at compute restart during error out stuck instances" [Low,Triaged] - Assigned to Balazs Gibizer (balazs-gibizer) | |
| 15:12:16 | gibi | eandersson: fyi, I reported a bug about the false error log from the compute you found https://bugs.launchpad.net/nova/+bug/1852759 | |
| 15:14:06 | gibi | bauzas: building a devstack for your audit patch... | |
| 15:29:38 | slaweq | mriedem: hi | |
| 15:29:50 | slaweq | mriedem: recently I noticed in neutron job error like https://storage.gra1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_300/678438/22/check/neutron-tempest-dvr-ha-multinode-full/3008cc3/testr_results.html.gz | |
| 15:30:18 | slaweq | and I see in nova logs (src host) error while connecting to libvirt on dst node: https://storage.gra1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_300/678438/22/check/neutron-tempest-dvr-ha-multinode-full/3008cc3/compute2/logs/screen-n-cpu.txt.gz | |
| 15:30:33 | slaweq | do You know about such issue or maybe I should create new LP for this? | |
| 15:32:43 | artom | slaweq, mriedem, doesn't look like a Nova bug: https://zuul.opendev.org/t/openstack/build/3008cc3eeaea44369a2fa3db3a29ae67/log/compute2/logs/screen-n-cpu.txt.gz#3392 | |
| 15:32:52 | artom | Just unable to connect to the dest libvirt | |
| 15:32:58 | artom | Not sure why tho | |
| 15:33:00 | mriedem | you mean this libvirt.libvirtError: unable to connect to server at 'ubuntu-bionic-rax-dfw-0012676801:49152': Connection refused | |
| 15:33:15 | slaweq | artom: mriedem exactly | |
| 15:33:32 | mriedem | usually need to look at the guest log | |
| 15:33:39 | mriedem | but no this isn't a nova bug | |
| 15:33:54 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Remove 'os-security-group-default-rules' REST API https://review.opendev.org/686807 | |
| 15:33:54 | openstackgerrit | Stephen Finucane proposed openstack/nova master: nova-net: Add TODOs for remaining nova-network functional tests https://review.opendev.org/684345 | |
| 15:33:55 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Remove (most) '/os-networks' REST APIs https://review.opendev.org/686809 | |
| 15:33:55 | openstackgerrit | Stephen Finucane proposed openstack/nova master: nova-net: Remove unused '*_default_rules' security group DB APIs https://review.opendev.org/686808 | |
| 15:33:56 | openstackgerrit | Stephen Finucane proposed openstack/nova master: nova-net: Remove 'USE_NEUTRON' from functional tests https://review.opendev.org/686811 | |
| 15:33:56 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Remove '/os-tenant-networks' REST API https://review.opendev.org/686810 | |
| 15:33:57 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Remove nova-manage network, floating commands https://review.opendev.org/686813 | |
| 15:33:57 | openstackgerrit | Stephen Finucane proposed openstack/nova master: nova-net: Remove 'networks' quota https://review.opendev.org/686812 | |
| 15:33:58 | openstackgerrit | Stephen Finucane proposed openstack/nova master: nova-net: Remove 'nova-dhcpbridge' binary https://review.opendev.org/686815 | |
| 15:33:58 | openstackgerrit | Stephen Finucane proposed openstack/nova master: nova-net: Remove associate, disassociate network APIs https://review.opendev.org/686814 | |
| 15:33:59 | openstackgerrit | Stephen Finucane proposed openstack/nova master: docs: Blast most references to nova-network https://review.opendev.org/686817 | |
| 15:33:59 | openstackgerrit | Stephen Finucane proposed openstack/nova master: nova-net: Remove 'nova-network' binary https://review.opendev.org/686816 | |
| 15:34:00 | openstackgerrit | Stephen Finucane proposed openstack/nova master: WIP https://review.opendev.org/686818 | |
| 15:34:04 | slaweq | mriedem: artom ok, thx for looking into that | |
| 15:34:13 | slaweq | I will than just leave it alone for now :) | |
| 15:35:55 | artom | slaweq, uh, I think it's trying to live migrate to itself o_O | |
| 15:36:19 | artom | https://zuul.opendev.org/t/openstack/build/3008cc3eeaea44369a2fa3db3a29ae67/log/zuul-info/host-info.compute2.yaml#403 | |
| 15:36:51 | artom | Oh no, that's the controller | |
| 15:36:52 | mriedem | ubuntu-bionic-rax-dfw-0012676801 != ubuntu-bionic-rax-dfw-0012676804 | |
| 15:36:59 | artom | mriedem, yeah sorry, got confused | |
| 15:37:23 | artom | But anyways, something to look into - is the controller running the full nova-compute stack? | |
| 15:37:40 | mriedem | yes, this is a 3 node job | |
| 15:37:53 | mriedem | dvr-ha-multinode-full makes the lights dim when it runs | |
| 15:38:04 | artom | It dimmed my lights :( | |
| 15:38:14 | mriedem | you scamp | |
| 15:41:59 | mriedem | cpu usage was pretty high on the controller when it failed | |
| 15:42:27 | mriedem | load spiked up around then too | |
| 15:42:40 | openstackgerrit | Merged openstack/nova-specs master: Virtual instance rescue with stable disk devices https://review.opendev.org/693849 | |
| 15:42:53 | artom | slaweq, there's your failure: https://zuul.opendev.org/t/openstack/build/3008cc3eeaea44369a2fa3db3a29ae67/log/controller/logs/libvirt/qemu/instance-00000011_log.txt.gz#4 | |
| 15:43:05 | artom | mriedem too if you care ^^ | |
| 15:43:10 | mriedem | artom: that's not a faliure | |
| 15:43:15 | mriedem | that shows up in like every guest log in thegate | |
| 15:43:28 | artom | Oh | |
| 15:43:51 | mriedem | e.g. random guest https://storage.gra1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_300/678438/22/check/neutron-tempest-dvr-ha-multinode-full/3008cc3/controller/logs/libvirt/qemu/instance-00000023_log.txt.gz | |
| 15:44:14 | mriedem | my guess is the failure is due to, like many gate failures these days, overloaded nodes crapping out | |
| 15:44:22 | mriedem | though the rax nodes usually aren't one of them | |
| 15:46:37 | openstackgerrit | Balazs Gibizer proposed openstack/nova stable/rocky: Fix false ERROR message at compute restart https://review.opendev.org/694581 | |
| 15:50:02 | gibi | eandersson: ^^ | |
| 15:56:21 | openstackgerrit | Matt Riedemann proposed openstack/nova master: docs: Extract rescue from reboot https://review.opendev.org/694529 | |
| 16:00:48 | lyarwood | thanks for that mriedem | |
| 16:02:27 | openstackgerrit | Matt Riedemann proposed openstack/nova master: doc: mention that rescuing a volume-backed server is not supported https://review.opendev.org/694584 | |
| 16:02:39 | mriedem | np | |
| 16:03:03 | openstackgerrit | Matt Riedemann proposed openstack/nova master: doc: mention that rescuing a volume-backed server is not supported https://review.opendev.org/694584 | |
| 16:07:36 | gibi | bauzas: I hit a bug in https://review.opendev.org/#/c/670112/8/nova/cmd/manage.py@2867 | |
| 16:08:10 | mriedem | gibi: you were +2 on this before it was rebased a bit in earlier changes https://review.opendev.org/#/c/642591/ - can you hit that again? dansmith - that's also the one where you suggested re-doing the logic so it should be simple and familiar | |
| 16:08:21 | bauzas | gibi: graaah ok, thanks ! | |
| 16:08:44 | gibi | mriedem: on it. | |
| 16:08:46 | bauzas | gibi: just respinning, sec | |
| 16:09:02 | gibi | bauzas: sure, I will retry with the new ps | |
| 16:09:10 | mriedem | bauzas: why wouldn't your functional tests hit that? | |
| 16:09:33 | bauzas | mriedem: the CI is still in the weeds | |
| 16:09:37 | bauzas | no result yet | |
| 16:09:39 | bauzas | and I was lazy | |
| 16:09:44 | mriedem | bauzas: i meant locally... | |
| 16:09:45 | mriedem | yeah | |
| 16:10:01 | bauzas | but that's a good call, lemme check | |
| 16:10:08 | mriedem | this is why i said you don't really need unit testing on these types of commands, it should mostly, if not all, be functional test driven | |
| 16:10:30 | bauzas | that's what happens when you do three things at same time... | |
| 16:10:42 | bauzas | meeting, patching, discussing | |
| 16:11:14 | bauzas | gibi: I just run the functest now | |
| 16:11:28 | bauzas | hopefully it will get caught | |
| 16:12:00 | bauzas | oh, and FWIW, I was rushing on updating a new revision, I haven't used yet the COMPUTE_NODE trait or did the split | |
| 16:13:19 | openstackgerrit | Merged openstack/nova stable/train: Use admin neutron client to query ports for binding https://review.opendev.org/694013 | |
| 16:13:38 | bauzas | mriedem: I'm not opposed to have most of the logic being checked by functional tests, I just added a few unit tests for sanity :) | |
| 16:14:24 | gibi | bauzas: nova.tests.functional.test_nova_manage.TestNovaManagePlacementAudit.test_audit_orphaned_allocations_from_deleted_compute_evacuate failed for me locally now but with a different error than what I saw in the devstack | |
| 16:14:27 | mriedem | that's a slippery slope and bad habit to get into with commands like this | |
| 16:14:30 | mriedem | so just, don't do it | |
| 16:14:51 | gibi | File "nova/cmd/manage.py", line 2781, in _check_orphaned_allocations_for_provider | |
| 16:14:51 | gibi | ctxt, placement, output, provider, delete) | |
| 16:14:51 | gibi | bauzas: File "nova/cmd/manage.py", line 2927, in audit | |
| 16:14:55 | gibi | inst_uuids, mig_uuids = result | |
| 16:14:56 | bauzas | gibi: yeah, I just saw it | |
| 16:14:57 | gibi | TypeError: 'bool' object is not iterable | |