| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-11-15 | |||
| 14:23:15 | dansmith | it's for every project I think | |
| 14:23:41 | dansmith | we used to have problems with people setting up their own CI and it going haywire and spraying comments everywhere | |
| 14:23:42 | dansmith | (in nova) | |
| 14:23:51 | artom | dansmith, so context is http://post-office.corp.redhat.com/archives/rh-openstack-dev/2019-October/msg00060.html (sorry for the internal-only link) | |
| 14:25:48 | artom | dansmith, and http://post-office.corp.redhat.com/archives/rh-openstack-dev/2019-October/msg00209.html that's sort of a summary of where we ended up after initial discussions | |
| 14:26:14 | dansmith | you could probably just say "I want to set up a CI system" | |
| 14:27:10 | artom | I want to set up a CI system | |
| 14:27:13 | artom | :D | |
| 14:27:15 | dansmith | but anyway, yes, there are hoops to jump through | |
| 14:28:07 | artom | Yeah | |
| 14:28:12 | artom | Internally as well as here, *sigh* | |
| 14:28:22 | artom | Hopefully it'll be worth it | |
| 14:28:31 | artom | I also noticed Mellanox has their own SRIOV CI | |
| 14:28:40 | artom | But... presumably that only tests their hardware | |
| 14:28:47 | artom | And I have no idea what the status is | |
| 14:29:09 | artom | adrianc, ^^ ? | |
| 14:29:42 | artom | And the point of RHEx (Red Hat Exotic hardware CI) would be more than just SRIOV, SRIOV is just the initial MVP scope | |
| 14:29:47 | artom | GPUs come to mind | |
| 14:29:48 | dansmith | artom: are you looking for help setting it up? If so, I'm sure the infra people are who you want to talk to | |
| 14:30:02 | artom | dansmith, yeah, that conversation is already happening - migi as well | |
| 14:30:12 | dansmith | ack | |
| 14:30:42 | mriedem | i wish the dell emc people would have asked b/c their's comments and always fails | |
| 14:32:01 | bauzas | gibi: I'm about to provide a new revision for the audit command, would it be possible for you to check it with the bandwidth-aware instances ? | |
| 14:32:15 | bauzas | (at least once I'm done with reno) | |
| 14:40:58 | gibi | bauzas: sure | |
| 14:41:28 | gibi | bauzas: If I time out on it today then I will do it next week | |
| 14:41:36 | bauzas | gibi: thanks | |
| 14:41:46 | bauzas | hopefully, I'll push it in 20 mins | |
| 14:47:31 | openstackgerrit | Merged openstack/nova master: Add functional recreate revert resize test for bug 1852610 https://review.opendev.org/694364 | |
| 14:47:31 | openstack | bug 1852610 in OpenStack Compute (nova) "API allows source compute service/node deletion while instances are pending a resize confirm/revert" [Medium,In progress] https://launchpad.net/bugs/1852610 - Assigned to Matt Riedemann (mriedem) | |
| 15:03:21 | openstackgerrit | Sylvain Bauza proposed openstack/nova master: Add a placement audit command https://review.opendev.org/670112 | |
| 15:12:16 | gibi | eandersson: fyi, I reported a bug about the false error log from the compute you found https://bugs.launchpad.net/nova/+bug/1852759 | |
| 15:12:16 | openstack | Launchpad bug 1852759 in OpenStack Compute (nova) rocky "false error log at compute restart during error out stuck instances" [Low,Triaged] - Assigned to Balazs Gibizer (balazs-gibizer) | |
| 15:14:06 | gibi | bauzas: building a devstack for your audit patch... | |
| 15:29:38 | slaweq | mriedem: hi | |
| 15:29:50 | slaweq | mriedem: recently I noticed in neutron job error like https://storage.gra1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_300/678438/22/check/neutron-tempest-dvr-ha-multinode-full/3008cc3/testr_results.html.gz | |
| 15:30:18 | slaweq | and I see in nova logs (src host) error while connecting to libvirt on dst node: https://storage.gra1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_300/678438/22/check/neutron-tempest-dvr-ha-multinode-full/3008cc3/compute2/logs/screen-n-cpu.txt.gz | |
| 15:30:33 | slaweq | do You know about such issue or maybe I should create new LP for this? | |
| 15:32:43 | artom | slaweq, mriedem, doesn't look like a Nova bug: https://zuul.opendev.org/t/openstack/build/3008cc3eeaea44369a2fa3db3a29ae67/log/compute2/logs/screen-n-cpu.txt.gz#3392 | |
| 15:32:52 | artom | Just unable to connect to the dest libvirt | |
| 15:32:58 | artom | Not sure why tho | |
| 15:33:00 | mriedem | you mean this libvirt.libvirtError: unable to connect to server at 'ubuntu-bionic-rax-dfw-0012676801:49152': Connection refused | |
| 15:33:15 | slaweq | artom: mriedem exactly | |
| 15:33:32 | mriedem | usually need to look at the guest log | |
| 15:33:39 | mriedem | but no this isn't a nova bug | |
| 15:33:54 | openstackgerrit | Stephen Finucane proposed openstack/nova master: nova-net: Add TODOs for remaining nova-network functional tests https://review.opendev.org/684345 | |
| 15:33:54 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Remove 'os-security-group-default-rules' REST API https://review.opendev.org/686807 | |
| 15:33:55 | openstackgerrit | Stephen Finucane proposed openstack/nova master: nova-net: Remove unused '*_default_rules' security group DB APIs https://review.opendev.org/686808 | |
| 15:33:55 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Remove (most) '/os-networks' REST APIs https://review.opendev.org/686809 | |
| 15:33:56 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Remove '/os-tenant-networks' REST API https://review.opendev.org/686810 | |
| 15:33:56 | openstackgerrit | Stephen Finucane proposed openstack/nova master: nova-net: Remove 'USE_NEUTRON' from functional tests https://review.opendev.org/686811 | |
| 15:33:57 | openstackgerrit | Stephen Finucane proposed openstack/nova master: nova-net: Remove 'networks' quota https://review.opendev.org/686812 | |
| 15:33:57 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Remove nova-manage network, floating commands https://review.opendev.org/686813 | |
| 15:33:58 | openstackgerrit | Stephen Finucane proposed openstack/nova master: nova-net: Remove associate, disassociate network APIs https://review.opendev.org/686814 | |
| 15:33:58 | openstackgerrit | Stephen Finucane proposed openstack/nova master: nova-net: Remove 'nova-dhcpbridge' binary https://review.opendev.org/686815 | |
| 15:33:59 | openstackgerrit | Stephen Finucane proposed openstack/nova master: nova-net: Remove 'nova-network' binary https://review.opendev.org/686816 | |
| 15:33:59 | openstackgerrit | Stephen Finucane proposed openstack/nova master: docs: Blast most references to nova-network https://review.opendev.org/686817 | |
| 15:34:00 | openstackgerrit | Stephen Finucane proposed openstack/nova master: WIP https://review.opendev.org/686818 | |
| 15:34:04 | slaweq | mriedem: artom ok, thx for looking into that | |
| 15:34:13 | slaweq | I will than just leave it alone for now :) | |
| 15:35:55 | artom | slaweq, uh, I think it's trying to live migrate to itself o_O | |
| 15:36:19 | artom | https://zuul.opendev.org/t/openstack/build/3008cc3eeaea44369a2fa3db3a29ae67/log/zuul-info/host-info.compute2.yaml#403 | |
| 15:36:51 | artom | Oh no, that's the controller | |
| 15:36:52 | mriedem | ubuntu-bionic-rax-dfw-0012676801 != ubuntu-bionic-rax-dfw-0012676804 | |
| 15:36:59 | artom | mriedem, yeah sorry, got confused | |
| 15:37:23 | artom | But anyways, something to look into - is the controller running the full nova-compute stack? | |
| 15:37:40 | mriedem | yes, this is a 3 node job | |
| 15:37:53 | mriedem | dvr-ha-multinode-full makes the lights dim when it runs | |
| 15:38:04 | artom | It dimmed my lights :( | |
| 15:38:14 | mriedem | you scamp | |
| 15:41:59 | mriedem | cpu usage was pretty high on the controller when it failed | |
| 15:42:27 | mriedem | load spiked up around then too | |
| 15:42:40 | openstackgerrit | Merged openstack/nova-specs master: Virtual instance rescue with stable disk devices https://review.opendev.org/693849 | |
| 15:42:53 | artom | slaweq, there's your failure: https://zuul.opendev.org/t/openstack/build/3008cc3eeaea44369a2fa3db3a29ae67/log/controller/logs/libvirt/qemu/instance-00000011_log.txt.gz#4 | |
| 15:43:05 | artom | mriedem too if you care ^^ | |
| 15:43:10 | mriedem | artom: that's not a faliure | |
| 15:43:15 | mriedem | that shows up in like every guest log in thegate | |
| 15:43:28 | artom | Oh | |
| 15:43:51 | mriedem | e.g. random guest https://storage.gra1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_300/678438/22/check/neutron-tempest-dvr-ha-multinode-full/3008cc3/controller/logs/libvirt/qemu/instance-00000023_log.txt.gz | |
| 15:44:14 | mriedem | my guess is the failure is due to, like many gate failures these days, overloaded nodes crapping out | |
| 15:44:22 | mriedem | though the rax nodes usually aren't one of them | |
| 15:46:37 | openstackgerrit | Balazs Gibizer proposed openstack/nova stable/rocky: Fix false ERROR message at compute restart https://review.opendev.org/694581 | |
| 15:50:02 | gibi | eandersson: ^^ | |
| 15:56:21 | openstackgerrit | Matt Riedemann proposed openstack/nova master: docs: Extract rescue from reboot https://review.opendev.org/694529 | |
| 16:00:48 | lyarwood | thanks for that mriedem | |
| 16:02:27 | openstackgerrit | Matt Riedemann proposed openstack/nova master: doc: mention that rescuing a volume-backed server is not supported https://review.opendev.org/694584 | |
| 16:02:39 | mriedem | np | |
| 16:03:03 | openstackgerrit | Matt Riedemann proposed openstack/nova master: doc: mention that rescuing a volume-backed server is not supported https://review.opendev.org/694584 | |
| 16:07:36 | gibi | bauzas: I hit a bug in https://review.opendev.org/#/c/670112/8/nova/cmd/manage.py@2867 | |
| 16:08:10 | mriedem | gibi: you were +2 on this before it was rebased a bit in earlier changes https://review.opendev.org/#/c/642591/ - can you hit that again? dansmith - that's also the one where you suggested re-doing the logic so it should be simple and familiar | |
| 16:08:21 | bauzas | gibi: graaah ok, thanks ! | |
| 16:08:44 | gibi | mriedem: on it. | |
| 16:08:46 | bauzas | gibi: just respinning, sec | |
| 16:09:02 | gibi | bauzas: sure, I will retry with the new ps | |
| 16:09:10 | mriedem | bauzas: why wouldn't your functional tests hit that? | |
| 16:09:33 | bauzas | mriedem: the CI is still in the weeds | |
| 16:09:37 | bauzas | no result yet | |
| 16:09:39 | bauzas | and I was lazy | |
| 16:09:44 | mriedem | bauzas: i meant locally... | |
| 16:09:45 | mriedem | yeah | |