| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-08-05 | |||
| 05:56:59 | openstackgerrit | Merged openstack/nova master: Fix lower-constraints conflicts https://review.opendev.org/744506 | |
| 07:12:51 | gokhani | hi team , I have an OpenStack environment in pike version. we have been using it for 2 years. Our environment consists of 3 controller and 23 compute hosts. but for 1 week , we are getting warnings like "Function 'nova.servicegroup.drivers.db.DbDriver._report_state' run outlasted interval by 24.01 sec" in compute nodes. what can be the reason of this ? do I need to look at mariadb galera cluster or rabbitmq cluster ? logs are in http://paste.opens | |
| 07:12:51 | gokhani | tack.org/show/796586/ | |
| 07:28:24 | gokhani | * http://paste.openstack.org/show/796586/ | |
| 09:49:53 | openstackgerrit | Stephen Finucane proposed openstack/nova master: fakelibvirt: Remove nova-network remnants https://review.opendev.org/737329 | |
| 09:49:53 | openstackgerrit | Stephen Finucane proposed openstack/nova master: network: Add type hints https://review.opendev.org/744869 | |
| 09:49:54 | openstackgerrit | Stephen Finucane proposed openstack/nova master: network: Drop 'vpn' parameter from 'allocate_for_instance' https://review.opendev.org/744870 | |
| 09:49:54 | openstackgerrit | Stephen Finucane proposed openstack/nova master: network: Remove unused 'affect_auto_assigned' parameter https://review.opendev.org/744871 | |
| 09:49:55 | openstackgerrit | Stephen Finucane proposed openstack/nova master: network: Remove 'kwargs' from 'get_instance_nw_info' https://review.opendev.org/744872 | |
| 09:54:08 | openstackgerrit | Stephen Finucane proposed openstack/nova master: tests: Add helpers for suspend, resume and reboot of server https://review.opendev.org/741285 | |
| 09:54:08 | openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Pass context, instance to '_create_guest' https://review.opendev.org/741286 | |
| 09:54:09 | openstackgerrit | Stephen Finucane proposed openstack/nova master: api: Reject non-spawn operations for vTPM https://review.opendev.org/741500 | |
| 09:54:09 | openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Add emulated TPM support to Nova https://review.opendev.org/631363 | |
| 09:54:10 | openstackgerrit | Stephen Finucane proposed openstack/nova master: docs: Add docs for vTPM support https://review.opendev.org/739213 | |
| 09:54:10 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Add type hints to 'nova.compute.manager' https://review.opendev.org/742863 | |
| 09:54:11 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Don't unset Instance.old_flavor, new_flavor until necessary https://review.opendev.org/741995 | |
| 09:54:11 | openstackgerrit | Stephen Finucane proposed openstack/nova master: privsep: Add support for recursive chown, move_tree operations https://review.opendev.org/742864 | |
| 09:54:12 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Add type hints to 'nova.virt.libvirt.utils' https://review.opendev.org/742865 | |
| 09:54:12 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Add support for resize and cold migration of emulated TPM files https://review.opendev.org/639934 | |
| 09:57:45 | lyarwood | RIP zuul | |
| 10:11:28 | kashyap | stephenfin: In the default Nova configuration, VMs are allowed to overcommit and float freely across host CPUs -- still true? | |
| 10:11:57 | stephenfin | lyarwood: blame artom and his damned reviews | |
| 10:12:01 | stephenfin | kashyap: yes | |
| 10:12:19 | stephenfin | only caveat is they can't overcommit against themselves | |
| 10:12:32 | stephenfin | can't boot an 8 core instance on a 6 core host | |
| 10:12:43 | kashyap | stephenfin: Ah, okay; I was about to ask for an example :) | |
| 10:13:03 | stephenfin | (or host with only 6 cores enabled in nova, via 'vcpu_pin_set' (legacy) or combo of 'cpu_dedicated_set' and 'cpu_shared_set') | |
| 10:15:21 | kashyap | Thanks | |
| 10:18:30 | openstackgerrit | Tony Su proposed openstack/nova master: Provider Config File: Functions to merge provider configs to provider tree https://review.opendev.org/676522 | |
| 10:18:31 | openstackgerrit | Tony Su proposed openstack/nova master: Provider Config File: Enable loading and merging of provider configs https://review.opendev.org/693460 | |
| 10:19:00 | openstackgerrit | Lee Yarwood proposed openstack/nova master: zuul: Start to migrate nova-live-migration to zuulv3 https://review.opendev.org/711604 | |
| 10:21:10 | tony_su | stephenfin: gibi: the last 3 provider-config-file patches are ready to go. change log: 1) all comments reflected in code 2) add three more test case to cover negative coe logic 3) unified coding style. | |
| 10:27:24 | gibi | tony_su: thanks. I added that to the queue for runway slot | |
| 10:31:07 | tony_su | gibi: thanks gibi. I will continue working on these patches in the next two weeks... | |
| 10:43:26 | openstackgerrit | Lee Yarwood proposed openstack/nova master: WIP zuul: nova-multinode-evacuate https://review.opendev.org/744883 | |
| 10:44:00 | lyarwood | hmm I thought the grenade issues had been resolved? | |
| 11:01:14 | brinzhang | stephenfin: hi, do I need fixed your point comments in https://review.opendev.org/#/c/742863/2/nova/compute/manager.py@2615 | |
| 11:02:03 | stephenfin | brinzhang: Not at all. I was saying that your fix was correct and mine was not :) | |
| 11:02:04 | brinzhang | change the LOG.exception(exc.format_message()) to LOG.exception(exc), and use Exception instead of exception.NovaException | |
| 11:02:35 | brinzhang | an I will report a bug for this change | |
| 11:04:19 | brinzhang | stephenfin: oh sorry, your patch are working now, I will just fixed mine ^ | |
| 11:07:31 | openstackgerrit | Brin Zhang proposed openstack/nova master: [Trivial] Remove wrong format_message() conversion https://review.opendev.org/744280 | |
| 11:09:02 | brinzhang | stephenfin: updated, thanks | |
| 11:10:15 | brinzhang | stephenfin, sean-k-mooney, gibi: the cyborg evacuate support patch was updated, pls review again while you have time, thanks. | |
| 11:10:15 | brinzhang | https://review.opendev.org/#/c/715326/ | |
| 11:10:47 | brinzhang | and I was rebased on the optimize patch, pls see https://review.opendev.org/#/c/726564/6 | |
| 11:27:56 | artom | stephenfin, sorry :( it was a drive-by thing as well, I didn't actually look at anything else | |
| 11:28:17 | artom | Hence no vote | |
| 11:30:51 | lyarwood | artom: how dare you provide useful reviews | |
| 11:30:58 | lyarwood | I mean really | |
| 11:31:10 | lyarwood | you're making the rest of us look bad ^_^ | |
| 11:31:25 | lyarwood | or maybe I'm doing that on my own | |
| 11:31:59 | artom | lyarwood, "useful" would be a stretch :P | |
| 11:33:09 | gibi | sean-k-mooney: thanks for the repeated test of the sriov attach. I responded in the review. Unfortunately I cannot reproduce your failures in my env. | |
| 11:33:17 | artom | (Yes, I'm knowingly leaving your last sentence unanswered >;) | |
| 11:33:27 | gibi | brinzhang: ack | |
| 11:38:51 | openstackgerrit | Merged openstack/python-novaclient master: Remove unused code https://review.opendev.org/744136 | |
| 11:41:11 | sean-k-mooney | gibi: so in your env removal of direct physical or macvtap devices worked? | |
| 11:41:19 | gibi | sean-k-mooney: yes | |
| 11:41:37 | gibi | the only thing that I can see is the remaining MAC on the VF after macvtap removal | |
| 11:41:52 | gibi | the rest of your failure does not appear to me | |
| 11:42:08 | sean-k-mooney | i used a single vm to do all the testing | |
| 11:42:21 | sean-k-mooney | ill try it again with multiple vms and see it it makes any difference | |
| 11:42:53 | sean-k-mooney | what os and libvirt version are you using? | |
| 11:43:15 | gibi | ubuntu 18.04, libvirt 6.0.0 qemu 4.2 | |
| 11:43:39 | sean-k-mooney | the libvirt behavior may have changed? i was using centos 8.1 maybe 8.2 i might be using older libvirt | |
| 11:43:49 | sean-k-mooney | i need to boot up the server to check | |
| 11:44:07 | gibi | the silent failure of macvtap and direct physical removel feels like a problem with findind the device that needs to be removed. I can try to add extra LOGs around that logic to trace the matching in your nev | |
| 11:44:11 | gibi | env | |
| 11:45:23 | gibi | two weeks ago I upgraded the libvirt from 4.0.0 to 6.0.0 and qemu from 2.11 to 4.2 due to a different failure in the "simple" direct case | |
| 11:46:38 | sean-k-mooney | if we need libvirt 6.0.0 we could not that as a min verion i guess. | |
| 11:46:42 | sean-k-mooney | but yes | |
| 11:46:53 | sean-k-mooney | both feel like we just did not find it in the xml and remove it | |
| 11:47:53 | gibi | regarding the leaking MAC addess after macvtap removal, who should do the removal of the MAC from the VF? is it libvirt? | |
| 11:49:12 | sean-k-mooney | gibi: libvirt should altrhough we also have code in nova to clear it for old libvirts | |
| 11:49:25 | sean-k-mooney | its also not reseting the programed vlan on the vf | |
| 11:49:33 | sean-k-mooney | but that is likely the same issue | |
| 11:50:14 | sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/virt/libvirt/vif.py#L774-L784 | |
| 11:50:27 | sean-k-mooney | we seam not to be calling unplug | |
| 11:50:51 | sean-k-mooney | in the direct case it proably is not clearing the trused vf status | |
| 11:51:08 | sean-k-mooney | i didnt actuly test that | |
| 11:51:37 | sean-k-mooney | we should be calling https://github.com/openstack/nova/blob/master/nova/virt/libvirt/vif.py#L833 as part of detach | |
| 11:52:03 | gibi | sean-k-mooney: thanks I will trace this missing unplug in my env | |
| 11:52:06 | sean-k-mooney | we are apprently https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L2201 | |
| 11:53:14 | gibi | I even have a log in the dmesg about the MAC removal during macvtap | |
| 11:53:15 | gibi | [1911821.990047] ixgbe 0000:81:00.0: removing MAC on VF 2 | |
| 11:53:30 | gibi | but right after it | |
| 11:53:31 | gibi | [1911822.241163] ixgbe 0000:81:00.0: Setting VLAN 100, QOS 0x0 on VF 2 | |
| 11:53:42 | gibi | it seems the vlan is set back to it | |
| 11:53:57 | sean-k-mooney | oh god damit.... | |
| 11:54:18 | sean-k-mooney | the neutron sriov nic agent is proably racing with unplug | |
| 11:54:51 | sean-k-mooney | although i guess it could be libvirt? | |
| 11:55:14 | sean-k-mooney | we might want to do unplug after detach? | |
| 11:55:28 | gibi | good points | |
| 11:55:45 | gibi | I will gather logs from libvirt and the neutron agent to see if this is a race | |
| 11:57:02 | sean-k-mooney | [centos@sriov-1 ~]$ libvirtd --version | |
| 11:57:04 | sean-k-mooney | libvirtd (libvirt) 6.0.0 | |
| 11:57:09 | sean-k-mooney | so same libvirt version | |
| 11:57:46 | sean-k-mooney | im using differnet nics then you you have nicantic 10G nics and im using 1G e1000 nics but that should not matter | |
| 11:58:28 | sean-k-mooney | @chat:button1[centos@sriov-1 ~]$ /usr/libexec/qemu-kvm --version | |