| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-10-18 | |||
| 19:27:58 | stephenfin | (at least I think that's what his tox changes did - I'll have to revisit) | |
| 21:14:56 | opendevreview | melanie witt proposed openstack/nova master: Add logic to enforce local api and db limits https://review.opendev.org/c/openstack/nova/+/712139 | |
| 21:14:57 | opendevreview | melanie witt proposed openstack/nova master: Enforce api and db limits https://review.opendev.org/c/openstack/nova/+/712142 | |
| 21:14:57 | opendevreview | melanie witt proposed openstack/nova master: Update quota_class APIs for db and api limits https://review.opendev.org/c/openstack/nova/+/712143 | |
| 21:14:58 | opendevreview | melanie witt proposed openstack/nova master: Update limit APIs https://review.opendev.org/c/openstack/nova/+/712707 | |
| 21:14:58 | opendevreview | melanie witt proposed openstack/nova master: Update quota sets APIs https://review.opendev.org/c/openstack/nova/+/712749 | |
| 21:14:59 | opendevreview | melanie witt proposed openstack/nova master: Tell oslo.limit how to count nova resources https://review.opendev.org/c/openstack/nova/+/713301 | |
| 21:14:59 | opendevreview | melanie witt proposed openstack/nova master: Enforce resource limits using oslo.limit https://review.opendev.org/c/openstack/nova/+/615180 | |
| 21:15:00 | opendevreview | melanie witt proposed openstack/nova master: Add legacy limits and usage to placement unified limits https://review.opendev.org/c/openstack/nova/+/713498 | |
| 21:15:00 | opendevreview | melanie witt proposed openstack/nova master: Update quota apis with keystone limits and usage https://review.opendev.org/c/openstack/nova/+/713499 | |
| 21:15:02 | opendevreview | melanie witt proposed openstack/nova master: Add reno for unified limits https://review.opendev.org/c/openstack/nova/+/715271 | |
| 21:15:02 | opendevreview | melanie witt proposed openstack/nova master: WIP Enable unified limits in the nova-next job https://review.opendev.org/c/openstack/nova/+/789963 | |
| #openstack-nova - 2021-10-19 | |||
| 05:31:27 | hgy | 1111 | |
| 08:08:48 | bauzas | good morning Nova | |
| 08:16:23 | gibi | bauzas: good morning | |
| 08:16:38 | bauzas | gibi: hey ;) | |
| 08:36:05 | brinzhang | bauzas, gibi: we cannot open the ethercalc, it report Service Unavailable(503), about the nova-cyborg interaction vGPU topic you can talked and pasting the decision on the etherpad, and we will look it ASAP | |
| 08:36:28 | brinzhang | bauzas, gibi: thanks | |
| 08:43:38 | gibi | bauzas: fyi, reported a new gate failure bug https://bugs.launchpad.net/nova/+bug/1947687 | |
| 08:44:26 | gibi | brinzhang: we will have that session today from 13:00 UTC | |
| 08:47:49 | bauzas | brinzhang: the cyborg-nova meeting will happen at 1pm UTC | |
| 08:48:06 | bauzas | gibi: ack | |
| 08:48:27 | gibi | bauzas: I'm not jumping to investigative mode for tha bug now. | |
| 08:51:25 | brinzhang | ack,but I cannot open https://ethercalc.openstack.org/8tum5yl1bx43 this page | |
| 08:52:37 | brinzhang | report 503 now, in cyborg PTG meeting we were usd tencent meeting instead of zoom | |
| 08:58:31 | gibi | brinzhang: could you open https://ptg.opendev.org/ptg.html ? | |
| 08:59:06 | gibi | brinzhang: if yes, then try to click on the nova cell in the 13:00 UTC column | |
| 08:59:20 | gibi | brinzhang: that should bring you in the zoom call | |
| 08:59:35 | gibi | (probably nobody there at the momement) | |
| 09:00:44 | bauzas | brinzhang: what gibi said | |
| 09:00:54 | bauzas | they moved from ethercalc AFAIK | |
| 09:03:03 | brinzhang | ack | |
| 09:03:13 | opendevreview | Balazs Gibizer proposed openstack/nova-specs master: Re-propose QoS minimum guaranteed packet rate https://review.opendev.org/c/openstack/nova-specs/+/814531 | |
| 09:16:38 | stephenfin | gibi: bauzas: Have you noticed many failures on the 'nova.tests.unit.db.main.test_migrations.TestModelsSyncMySQL.test_models_sync' and 'TestModelsLegacySyncMySQL.test_models_sync' tests? | |
| 09:16:56 | gibi | stephenfin: I'm not | |
| 09:18:07 | stephenfin | Ah, okay, I'm seeing a lot of failures on the bp/remove-sqlalchemy-migrate series and trying to figure out if I've already broken master slightly or if its one of the unmerged patches that's breaking things, e.g. https://review.opendev.org/c/openstack/nova/+/812144 | |
| 09:24:38 | songwenping | gibi,bauzas: when i stop nova_compute service and evacuate the vm on the host, then i delete the vm when it is evacuating, the vm is local deleted by nova-api, but the libvirt process is not terminated, and the resources like GPU are still used by the vm process, how can we handle the vm process? | |
| 09:33:46 | gibi | songwenping: you delete the VM while it is being evacuated? | |
| 09:34:54 | songwenping | yes | |
| 09:35:18 | gibi | and then you start the source compute back up | |
| 09:35:48 | songwenping | yeah | |
| 09:36:01 | gibi | and the the source compute does not clean up the instance? | |
| 09:36:22 | songwenping | yeah | |
| 09:36:35 | gibi | I think the source compute should detect that there was a local delete while it was down and clean up the instance | |
| 09:36:38 | gibi | so it is probably a bug | |
| 09:37:15 | songwenping | i am looking into the init_host process. | |
| 10:14:04 | opendevreview | Stephen Finucane proposed openstack/nova master: db: Increase timeout for migration tests https://review.opendev.org/c/openstack/nova/+/814547 | |
| 10:14:51 | stephenfin | gibi: bauzas: I'm not sure how to test that, but I'm pretty sure we're going to need it to at least fix the migrations issues we've been seen in the ARM CI jobs ^ | |
| 10:15:00 | stephenfin | I guess I can mash recheck | |
| 10:16:43 | gibi | stephenfin: ack | |
| 10:16:54 | gibi | is it only affects ARM? | |
| 10:34:16 | stephenfin | I don't think so. I've seen failures on x86 also, but ARM is particularly badly affected | |
| 10:34:33 | stephenfin | example: https://zuul.opendev.org/t/openstack/build/00197f336d1249e4a55255769f24faf7 | |
| 10:34:42 | stephenfin | vs (aarch64) https://zuul.opendev.org/t/openstack/build/f693ac69815f4b92b75f7f8dc5ec3803 | |
| 10:34:49 | stephenfin | gibi: ^ | |
| 10:40:52 | gibi | stephenfin: when it passes it is also slow just not too slow to hit the timeout? | |
| 10:42:39 | gibi | I see this runs | |
| 10:42:40 | gibi | nova.tests.unit.db.main.test_migrations.TestModelsSyncMySQL.test_models_sync 11.154 | |
| 10:42:48 | gibi | where it is actully fast | |
| 10:43:00 | gibi | https://zuul.opendev.org/t/openstack/build/8c4f8a81770e4dd1a523e86690ce340b/log/job-output.txt | |
| 10:44:35 | gibi | anyhow I'm not against extending the timeout | |
| 10:44:51 | gibi | I just feel that we still not fully grok what can be the problems with this db tests | |
| 10:49:31 | opendevreview | Ilya Popov proposed openstack/nova master: Fix to use NUMA cell with more free memory first https://review.opendev.org/c/openstack/nova/+/805649 | |
| 11:09:02 | stephenfin | gibi: Yeah, I see about 7-8 seconds runtime on my localhost but it _seems_ very much dependent on the environment. Perhaps it's because some VMs are using spinning rust and others are SSD based? | |
| 11:09:30 | stephenfin | We can compare these two builds https://zuul.opendev.org/t/openstack/build/00197f336d1249e4a55255769f24faf7 (iWeb) and https://zuul.opendev.org/t/openstack/build/8c4f8a81770e4dd1a523e86690ce340b (OVH) | |
| 11:14:06 | stephenfin | gibi: Yeah, crazy differences in the MySQL tests https://paste.opendev.org/show/810073/ | |
| 11:15:11 | gibi | I remember I saw this craxy differences before alembic too, and I had not explanation for that | |
| 11:16:49 | songwenping | gibi: the evacuation status is failed, and this filter[1] cannot get the vm. [1]https://review.opendev.org/plugins/gitiles/openstack/nova/+/refs/heads/master/nova/compute/manager.py#703 | |
| 11:17:33 | songwenping | so the vm process is not destory when the compute service restore. | |
| 11:18:51 | gibi | songwenping: the problem with failed evacuation itself is that it does not imply the the VM is deleted (and hence the evac failed). If the evac fail for other reasons then keeping the VM on the source host after recovery is logical | |
| 11:19:32 | gibi | so I think you have to make some logic changes where the failed evacuations are also queried but only if the instance is already marked deleted | |
| 11:22:56 | songwenping | to report a bug? | |
| 11:24:35 | stephenfin | Looks like https://review.opendev.org/c/openstack/nova/+/814547/ is working at least, since the aarch64 jobs are passing. I left a comment for bauzas | |
| 11:39:03 | sean-k-mooney | stephenfin: i do have an arm laptop, my pinebook pro i can proably try and run it or power it up and let you ssh into it an use it although i proably will have to reinstall the os before its really useful | |
| 11:40:57 | stephenfin | sean-k-mooney: I don't think it's specific to the architecture. I think it's the host in general. Bad I/O performance or something similar | |
| 11:41:02 | stephenfin | but that's just a theory | |
| 11:41:13 | sean-k-mooney | ah ok | |
| 11:52:33 | opendevreview | Stephen Finucane proposed openstack/nova master: docs: Add notes about IoOpsWeigher, BuildFailureWeigher https://review.opendev.org/c/openstack/nova/+/814561 | |
| 11:52:34 | opendevreview | Stephen Finucane proposed openstack/nova master: docs: Add a new cells v2 document https://review.opendev.org/c/openstack/nova/+/814562 | |
| 11:52:34 | opendevreview | Stephen Finucane proposed openstack/nova master: docs: Add new architecture guide https://review.opendev.org/c/openstack/nova/+/814563 | |
| 11:54:48 | stephenfin | bauzas: you're probably interested in https://review.opendev.org/c/openstack/nova/+/814561 also when you're back | |
| 12:02:00 | gibi | songwenping: sure, a bug is needed too | |
| 12:03:43 | gibi | stephenfin: yep it seems to work. It runs the test longer than before | |
| 12:04:08 | gibi | nova.tests.unit.db.main.test_migrations.TestMigrationsWalkMySQL.test_walk_versions [372.936004s] | |
| 12:04:15 | gibi | that is a lot of time :) | |
| 12:06:59 | sean-k-mooney | gibi: stephenfin im proably goingt to add https://review.opendev.org/c/openstack/nova/+/805649 to the ptg adgenda at the end but i think directionally this is the correct approch to take and once the testing is built out and a release note added its close to a complete | |
| 12:07:43 | sean-k-mooney | if ye have time could ye take a look and or set review priority on it if you think it should have it | |
| 12:08:25 | gibi | sean-k-mooney: ack, does it have a bp attached? I'm not 100% sure it is a bug per se | |
| 12:08:34 | sean-k-mooney | tl;dr its adressign numa blancing and pci affintiy my making prefer actully prefer the numa node | |
| 12:08:46 | sean-k-mooney | gibi: right i think it a blueprint not a bug | |
| 12:09:16 | sean-k-mooney | so no it currently does not | |
| 12:10:26 | gibi | sean-k-mooney: also it worth discussing on the ptg. I think we need a consensus on what are the inputs we are sorting the NUMA cells on | |
| 12:10:41 | sean-k-mooney | https://bugs.launchpad.net/nova/+bug/1893121 was my orginal bug | |
| 12:11:16 | sean-k-mooney | which when we lass discussed it we said was a blueprint | |
| 12:11:23 | sean-k-mooney | gibi: yes that is why i was going to add it | |
| 12:11:39 | sean-k-mooney | to make sure we agreed on the algoritim | |
| 12:16:50 | gibi | cool | |
| 12:20:27 | gibi | bauzas: have you seen melwitt's comment on L83 on the ptg etherpad? | |
| 12:20:57 | gibi | bauzas: I guess we can re-visit oslo.limits later in the agenda | |
| 12:24:43 | sean-k-mooney | we are going to have a followup unifed limits session | |