| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-08-29 | |||
| 03:18:27 | opendevreview | melanie witt proposed openstack/nova master: Update quota_class APIs for db and api limits https://review.opendev.org/c/openstack/nova/+/712143 | |
| 03:18:27 | opendevreview | melanie witt proposed openstack/nova master: Enforce api and db limits https://review.opendev.org/c/openstack/nova/+/712142 | |
| 03:18:28 | opendevreview | melanie witt proposed openstack/nova master: Update quota sets APIs https://review.opendev.org/c/openstack/nova/+/712749 | |
| 03:18:28 | opendevreview | melanie witt proposed openstack/nova master: Update limit APIs https://review.opendev.org/c/openstack/nova/+/712707 | |
| 03:18:29 | opendevreview | melanie witt proposed openstack/nova master: Tell oslo.limit how to count nova resources https://review.opendev.org/c/openstack/nova/+/713301 | |
| 03:18:30 | opendevreview | melanie witt proposed openstack/nova master: Add legacy limits and usage to unified limits https://review.opendev.org/c/openstack/nova/+/713498 | |
| 03:18:30 | opendevreview | melanie witt proposed openstack/nova master: Enforce resource limits using oslo.limit https://review.opendev.org/c/openstack/nova/+/615180 | |
| 03:18:32 | opendevreview | melanie witt proposed openstack/nova master: Add reno for unified limits https://review.opendev.org/c/openstack/nova/+/715271 | |
| 03:18:32 | opendevreview | melanie witt proposed openstack/nova master: Update quota apis with keystone limits and usage https://review.opendev.org/c/openstack/nova/+/713499 | |
| 03:38:33 | melwitt | gibi, johnthetubaguy[m], dansmith: unified limits set has been updated to do all limits in keystone (no config option quota limits when the UnifiedLimitsDriver is being used) ^ apologies for the delay. I have summarized what I did in https://review.opendev.org/c/openstack/nova/+/712139/13#message-b41a9e4a8a96136ca7e4bff126b575effa2fe839 | |
| 04:28:17 | opendevreview | melanie witt proposed openstack/nova master: Update limit APIs https://review.opendev.org/c/openstack/nova/+/712707 | |
| 04:28:18 | opendevreview | melanie witt proposed openstack/nova master: Tell oslo.limit how to count nova resources https://review.opendev.org/c/openstack/nova/+/713301 | |
| 04:28:18 | opendevreview | melanie witt proposed openstack/nova master: Update quota sets APIs https://review.opendev.org/c/openstack/nova/+/712749 | |
| 04:28:19 | opendevreview | melanie witt proposed openstack/nova master: Add legacy limits and usage to unified limits https://review.opendev.org/c/openstack/nova/+/713498 | |
| 04:28:19 | opendevreview | melanie witt proposed openstack/nova master: Enforce resource limits using oslo.limit https://review.opendev.org/c/openstack/nova/+/615180 | |
| 04:28:20 | opendevreview | melanie witt proposed openstack/nova master: Add reno for unified limits https://review.opendev.org/c/openstack/nova/+/715271 | |
| 04:28:20 | opendevreview | melanie witt proposed openstack/nova master: Update quota apis with keystone limits and usage https://review.opendev.org/c/openstack/nova/+/713499 | |
| 12:12:55 | maxba | hi guys, I have problem about multicast and broadcast in public network. I have much broadcast traffic in the network from stupid application, eg. winbox, windows smb, .... how can I drop these traffic just for my public network? | |
| 12:13:57 | maxba | I use openvswitch for l2 networking | |
| 12:15:37 | maxba | i tested iptables, ebtables, in layer of compute and network namespace of public network in the compute, they didn't work | |
| 16:00:46 | opendevreview | melanie witt proposed openstack/nova master: Tell oslo.limit how to count nova resources https://review.opendev.org/c/openstack/nova/+/713301 | |
| 16:00:47 | opendevreview | melanie witt proposed openstack/nova master: Add legacy limits and usage to unified limits https://review.opendev.org/c/openstack/nova/+/713498 | |
| 16:00:47 | opendevreview | melanie witt proposed openstack/nova master: Enforce resource limits using oslo.limit https://review.opendev.org/c/openstack/nova/+/615180 | |
| 16:00:48 | opendevreview | melanie witt proposed openstack/nova master: Add reno for unified limits https://review.opendev.org/c/openstack/nova/+/715271 | |
| 16:00:48 | opendevreview | melanie witt proposed openstack/nova master: Update quota apis with keystone limits and usage https://review.opendev.org/c/openstack/nova/+/713499 | |
| #openstack-nova - 2021-08-30 | |||
| 06:11:31 | opendevreview | Balazs Gibizer proposed openstack/nova master: Add force kwarg to delete_allocation_for_instance https://review.opendev.org/c/openstack/nova/+/688802 | |
| 06:11:56 | gibi | melwitt: ^^ removed the co-authored line as you requested | |
| 06:34:31 | gibi | lyarwood, stephenfin, bauzas: we are still pretty much blocking the openstack gate without ^^ | |
| 06:45:41 | lyarwood | I'm out today, last public holiday of the year in the UK but I'll review from my phone now. | |
| 06:49:43 | lyarwood | Okay done, LGTM. | |
| 06:57:38 | gibi | lyarwood: thanks, enjoy your day off | |
| 08:39:45 | abhishekk | gibi, py38 post failure for https://review.opendev.org/c/openstack/nova/+/688802, could you please dd recheck ? | |
| 10:01:54 | elodilles | gibi: could you please have a quick look at this placement release patch for stable/ussuri? (it's a generated patch to avoid release rush around EM transition): https://review.opendev.org/c/openstack/releases/+/802110 | |
| 10:07:12 | gibi | elodilles: ack I will check | |
| 10:09:25 | gibi | abhishekk feel free to recheck next time | |
| 10:11:15 | gibi | elodilles: done and thanks | |
| 10:12:18 | elodilles | gibi: thanks \o/ | |
| 12:21:37 | gibi | the force kwargs patch https://review.opendev.org/c/openstack/nova/+/688802 bounced from the gate due to bug 1912310, I've requeued it | |
| 12:24:06 | sean-k-mooney | what causes https://bugs.launchpad.net/nova/+bug/1912310 | |
| 12:25:16 | gibi | I saw libvirt internal errors like | |
| 12:25:17 | gibi | 2021-07-30 08:56:25.528+0000: 57632: error : virProcessRunInFork:1159 : internal error: child reported (status=125): unable to open /dev/sda: No such device or address | |
| 12:26:10 | sean-k-mooney | ok so it looks like its actully libvirt that is having issue not nova connecting to it | |
| 12:26:16 | gibi | yepp | |
| 12:26:21 | gibi | as far as I understand | |
| 12:27:04 | gibi | there are also occasions with | |
| 12:27:05 | gibi | virKeepAliveTimerInternal:137 : internal error: connection closed due to keepalive timeout | |
| 12:27:14 | sean-k-mooney | we are not seeing any OOM events or anythying else strange on the node at the time are we | |
| 12:27:53 | gibi | I just like two occurence and found no such thing | |
| 12:27:58 | gibi | I just checked like | |
| 12:28:20 | gibi | the nova-live-migration job set to non-voting due to this | |
| 12:28:29 | gibi | but it seems we can hit the same in nova-next too | |
| 12:28:34 | gibi | but a lot less frequently | |
| 12:29:14 | sean-k-mooney | yep if failind in nova-net in this case | |
| 12:29:16 | sean-k-mooney | https://zuul.opendev.org/t/openstack/build/f888b58ca23f49fc8f9046e9c2ad18a0/log/controller/logs/screen-dstat.txt | |
| 12:29:37 | gibi | yes | |
| 12:29:48 | gibi | that is basically a first time I see it in nova-next | |
| 12:29:51 | sean-k-mooney | we got donw to 120MB a few times but i dont see any really evidence of memory issues so likely not the kernel randomly killing things | |
| 12:32:07 | gibi | around the time of the failure we were floating around 300MB free | |
| 12:33:52 | sean-k-mooney | ya its unlikely to be the cause but we have seen OOM issue break libvirt and other process in weird ways before. | |
| 12:34:20 | gibi | true, oom can cause weird thing | |
| 12:34:22 | gibi | s | |
| 12:35:09 | sean-k-mooney | ill quickly check the cloud archive | |
| 12:35:19 | sean-k-mooney | perhaps there is a newer libvirt avaiable we coudl use instead | |
| 12:36:07 | gibi | didn't we use the max available? | |
| 12:37:00 | sean-k-mooney | well im not sure we are using the xena cloud archive currently | |
| 12:37:14 | sean-k-mooney | but looking at it they are not shiping libvirt/qemu in the cloud archive currently | |
| 12:37:41 | sean-k-mooney | we tyically dont use the most recent cloud archive version | |
| 12:39:46 | sean-k-mooney | so ya looks like we are using 6.x for ubuntu "libvirt0:amd64 6.0.0-0ubuntu8.12" | |
| 12:40:15 | sean-k-mooney | on centos stream with the advance virt modulee we would be useing 7.x.y | |
| 12:43:18 | sean-k-mooney | its a long shot but we could enable this ppa as a test to see if that would resolve it. its the one i use when i need newer libvirt on ubutu but dont want to build from source | |
| 12:43:22 | sean-k-mooney | https://launchpad.net/~jacob/+archive/ubuntu/virtualisation | |
| 12:43:57 | sean-k-mooney | althopugh that still only provides 6.6.0-1ubuntu2~ppa0 | |
| 12:44:04 | sean-k-mooney | not 7.x | |
| 12:45:09 | gibi | I'm not sure how can we enable this in infra but feel free to go ahead. We can use the nova-live-migration job as canary as that is now non-voting but still run for almost all of our patches | |
| 12:45:49 | sean-k-mooney | ya i might porpose a DNM patch just to see if that works. if it does it means we need to talk to canonical about a missing backport | |
| 12:46:04 | sean-k-mooney | proablem is i have no idea what is missing | |
| 12:46:42 | gibi | cool, good ide | |
| 12:46:44 | gibi | idea | |
| 12:46:53 | sean-k-mooney | the other alternitive would be to move form ubuntu 20.04 to 21.04 or to centos 8 on the affected jobs | |
| 12:49:01 | sean-k-mooney | well there is another alternitive too which is complie libvirt/qemu form source which i have a devstack plugin to do but i would prefer to avoid that mainly due to extra job time. its not hard to do but if we can just use distro pacakages in this case its nicer | |
| 12:49:17 | gibi | as we declare our supported distros beforhand of the release I would go with trying to fix ubuntu 20.04 https://governance.openstack.org/tc/reference/runtimes/xena.html | |
| 12:50:26 | sean-k-mooney | yes although centos 8 stream is vaild too. but ya ill see if i can look into this a little later today. ill propose a couple of different patches for different options. | |
| 12:51:20 | sean-k-mooney | enableing "sudo add-apt-repository ppa:jacob/virtualisation" in a pre playbook is simple as is changing the base os to centos 8 stream | |
| 12:51:45 | sean-k-mooney | the other options are more complicated but doable | |
| 13:06:59 | opendevreview | Merged openstack/nova master: Functional tests removed direct post call https://review.opendev.org/c/openstack/nova/+/766068 | |
| 13:22:30 | sean-k-mooney | gibi: by the way we maintain a tempest plugin called whitebox that looks at some of the internals of how nova works and assert that it does the right thing. would you have any object to me enableing that for a subset of nova changes at least in a non voting capasity initaly? | |
| 13:22:48 | sean-k-mooney | gibi: i was thinking of making it run on change to the libvirt driver and hardware.py | |
| 13:23:20 | gibi | sean-k-mooney: I have no problem with it if it is actively maintained and won't take up much of the CI resources | |
| 13:25:11 | sean-k-mooney | yes its maintianed and runs downstream we also maintin the devstack support upstream | |
| 13:25:33 | sean-k-mooney | gibi: upstream many of the test are disabled because we dont have the hardware https://opendev.org/openstack/whitebox-tempest-plugin/src/branch/master/whitebox_tempest_plugin/api/compute | |
| 13:26:01 | sean-k-mooney | i.e. we can run the pmem, sriov or vgpu test in the gate | |
| 13:26:29 | sean-k-mooney | i know we were lucking to see if we coudl use this for third party ci but we still are having problems finding hardware internally to run it | |
| 13:27:46 | opendevreview | Balazs Gibizer proposed openstack/nova master: Add two new hacking rules https://review.opendev.org/c/openstack/nova/+/805668 | |
| 13:28:11 | gibi | just based on the test file names even without special hardware this plugin has useful coverage | |
| 13:28:45 | sean-k-mooney | yep it has all the test that were orginailly don by the intel thridpary nfv ci in it but updated | |
| 13:28:51 | sean-k-mooney | and some other test coverage | |
| 13:31:10 | gibi | then lets enable it | |
| 13:32:40 | sean-k-mooney | gibi: when the qe member of the comptue team downstream writes test automation that is not suitable for upstream tempest becasue it depens on speicic configuration of the serives this is where we try to add the test coverage. | |
| 13:33:07 | sean-k-mooney | like testing adding cpu flags which we can do https://opendev.org/openstack/whitebox-tempest-plugin/src/branch/master/.zuul.yaml#L53-L55 in the ci like this | |
| 13:34:22 | gibi | I agree to have that coverage in our upstream gate | |