Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2021-10-19
09:36:35 gibi I think the source compute should detect that there was a local delete while it was down and clean up the instance
09:36:38 gibi so it is probably a bug
09:37:15 songwenping i am looking into the init_host process.
10:14:04 opendevreview Stephen Finucane proposed openstack/nova master: db: Increase timeout for migration tests https://review.opendev.org/c/openstack/nova/+/814547
10:14:51 stephenfin gibi: bauzas: I'm not sure how to test that, but I'm pretty sure we're going to need it to at least fix the migrations issues we've been seen in the ARM CI jobs ^
10:15:00 stephenfin I guess I can mash recheck
10:16:43 gibi stephenfin: ack
10:16:54 gibi is it only affects ARM?
10:34:16 stephenfin I don't think so. I've seen failures on x86 also, but ARM is particularly badly affected
10:34:33 stephenfin example: https://zuul.opendev.org/t/openstack/build/00197f336d1249e4a55255769f24faf7
10:34:42 stephenfin vs (aarch64) https://zuul.opendev.org/t/openstack/build/f693ac69815f4b92b75f7f8dc5ec3803
10:34:49 stephenfin gibi: ^
10:40:52 gibi stephenfin: when it passes it is also slow just not too slow to hit the timeout?
10:42:39 gibi I see this runs
10:42:40 gibi nova.tests.unit.db.main.test_migrations.TestModelsSyncMySQL.test_models_sync 11.154
10:42:48 gibi where it is actully fast
10:43:00 gibi https://zuul.opendev.org/t/openstack/build/8c4f8a81770e4dd1a523e86690ce340b/log/job-output.txt
10:44:35 gibi anyhow I'm not against extending the timeout
10:44:51 gibi I just feel that we still not fully grok what can be the problems with this db tests
10:49:31 opendevreview Ilya Popov proposed openstack/nova master: Fix to use NUMA cell with more free memory first https://review.opendev.org/c/openstack/nova/+/805649
11:09:02 stephenfin gibi: Yeah, I see about 7-8 seconds runtime on my localhost but it _seems_ very much dependent on the environment. Perhaps it's because some VMs are using spinning rust and others are SSD based?
11:09:30 stephenfin We can compare these two builds https://zuul.opendev.org/t/openstack/build/00197f336d1249e4a55255769f24faf7 (iWeb) and https://zuul.opendev.org/t/openstack/build/8c4f8a81770e4dd1a523e86690ce340b (OVH)
11:14:06 stephenfin gibi: Yeah, crazy differences in the MySQL tests https://paste.opendev.org/show/810073/
11:15:11 gibi I remember I saw this craxy differences before alembic too, and I had not explanation for that
11:16:49 songwenping gibi: the evacuation status is failed, and this filter[1] cannot get the vm. [1]https://review.opendev.org/plugins/gitiles/openstack/nova/+/refs/heads/master/nova/compute/manager.py#703
11:17:33 songwenping so the vm process is not destory when the compute service restore.
11:18:51 gibi songwenping: the problem with failed evacuation itself is that it does not imply the the VM is deleted (and hence the evac failed). If the evac fail for other reasons then keeping the VM on the source host after recovery is logical
11:19:32 gibi so I think you have to make some logic changes where the failed evacuations are also queried but only if the instance is already marked deleted
11:22:56 songwenping to report a bug?
11:24:35 stephenfin Looks like https://review.opendev.org/c/openstack/nova/+/814547/ is working at least, since the aarch64 jobs are passing. I left a comment for bauzas
11:39:03 sean-k-mooney stephenfin: i do have an arm laptop, my pinebook pro i can proably try and run it or power it up and let you ssh into it an use it although i proably will have to reinstall the os before its really useful
11:40:57 stephenfin sean-k-mooney: I don't think it's specific to the architecture. I think it's the host in general. Bad I/O performance or something similar
11:41:02 stephenfin but that's just a theory
11:41:13 sean-k-mooney ah ok
11:52:33 opendevreview Stephen Finucane proposed openstack/nova master: docs: Add notes about IoOpsWeigher, BuildFailureWeigher https://review.opendev.org/c/openstack/nova/+/814561
11:52:34 opendevreview Stephen Finucane proposed openstack/nova master: docs: Add new architecture guide https://review.opendev.org/c/openstack/nova/+/814563
11:52:34 opendevreview Stephen Finucane proposed openstack/nova master: docs: Add a new cells v2 document https://review.opendev.org/c/openstack/nova/+/814562
11:54:48 stephenfin bauzas: you're probably interested in https://review.opendev.org/c/openstack/nova/+/814561 also when you're back
12:02:00 gibi songwenping: sure, a bug is needed too
12:03:43 gibi stephenfin: yep it seems to work. It runs the test longer than before
12:04:08 gibi nova.tests.unit.db.main.test_migrations.TestMigrationsWalkMySQL.test_walk_versions [372.936004s]
12:04:15 gibi that is a lot of time :)
12:06:59 sean-k-mooney gibi: stephenfin im proably goingt to add https://review.opendev.org/c/openstack/nova/+/805649 to the ptg adgenda at the end but i think directionally this is the correct approch to take and once the testing is built out and a release note added its close to a complete
12:07:43 sean-k-mooney if ye have time could ye take a look and or set review priority on it if you think it should have it
12:08:25 gibi sean-k-mooney: ack, does it have a bp attached? I'm not 100% sure it is a bug per se
12:08:34 sean-k-mooney tl;dr its adressign numa blancing and pci affintiy my making prefer actully prefer the numa node
12:08:46 sean-k-mooney gibi: right i think it a blueprint not a bug
12:09:16 sean-k-mooney so no it currently does not
12:10:26 gibi sean-k-mooney: also it worth discussing on the ptg. I think we need a consensus on what are the inputs we are sorting the NUMA cells on
12:10:41 sean-k-mooney https://bugs.launchpad.net/nova/+bug/1893121 was my orginal bug
12:11:16 sean-k-mooney which when we lass discussed it we said was a blueprint
12:11:23 sean-k-mooney gibi: yes that is why i was going to add it
12:11:39 sean-k-mooney to make sure we agreed on the algoritim
12:16:50 gibi cool
12:20:27 gibi bauzas: have you seen melwitt's comment on L83 on the ptg etherpad?
12:20:57 gibi bauzas: I guess we can re-visit oslo.limits later in the agenda
12:24:43 sean-k-mooney we are going to have a followup unifed limits session
12:24:57 sean-k-mooney so perhaps we can ask the oslo folk to join that
12:25:40 sean-k-mooney e.g. in the cross project seesion we can let them know we will be having a topic on unifed limits later and ask them who form the oslo.limits side would be good to ping to attend
12:29:54 gibi yeah that is a good idea
12:29:58 bauzas sorry folks, I was off during this lunch for the gym
12:30:15 bauzas gibi: looking at the etherpad
12:31:24 bauzas yeah, let's discuss with the oslo folks about this
12:31:33 sean-k-mooney ok so we are starting with cyborg then oslo then rbac correct
12:31:39 sean-k-mooney so all cross project sessions today
12:32:07 sean-k-mooney we wont be starting nova only adgenday until tommoro after the neutron session
12:33:32 sean-k-mooney i guess i better get a headset ready although i might just used my wired one i dont know if my wireless one is chaged since its in my backpack instead of plugged in on my desk
12:44:41 bauzas sean-k-mooney: if we have time, we can start a few
12:45:18 bauzas sean-k-mooney: given we will stop to discuss for the rbac at 4pm while we still have a slot until 5pm
12:45:32 bauzas (UTC of course)
12:56:37 bauzas starting the meetings in 3 mins
13:00:41 sean-k-mooney ah ya that makes sense
13:08:07 songwenping_ mdbooth: i have meet the problem last week, and have to reinstall my devstack
13:26:08 opendevreview Pierre-Samuel Le Stang proposed openstack/nova master: Reproducer unit test for bug 1934094 https://review.opendev.org/c/openstack/nova/+/812646
13:55:25 opendevreview Pierre-Samuel Le Stang proposed openstack/nova master: Reproducer unit test for bug 1934094 https://review.opendev.org/c/openstack/nova/+/812646
14:23:58 bauzas hopefully will be back at time !
14:26:48 gibi bauzas: belmoreira asked about when the vncproxy topic will be discussed so that he can join
14:28:13 bauzas gibi: we can start discussing the vncproxy topic at first after the break, wdyt ?
14:29:04 gibi as it is a cell issue I would rather wait for dansmith / melwitt to join for it
14:29:13 gibi at least I have not much to add there
14:29:26 dansmith have glance things, but planning to be in nova for the rbac stuff
14:30:14 gibi bauzas: and I think belmoreira is OK to have it tomorrow as well, so we don't need to rush, he just needs some timing so that he can join just for that discussion
14:31:05 bauzas gibi: fair point about the cell impact
14:32:37 bauzas gibi: let's target the vncproxy thing on tomorrow 15:30pm UTC
14:32:45 bauzas I'll amend the etherpad so
14:32:50 gibi belmoreira: is it OK for you ^^ ?
14:33:33 bauzas I'll put unified limits and vncproxy as a bulk for tomorrow late
14:34:21 gibi ack
14:35:28 opendevreview Pierre-Samuel Le Stang proposed openstack/nova master: Reproducer unit test for bug 1934094 https://review.opendev.org/c/openstack/nova/+/812646
14:35:54 belmoreira thanks gibi bauzas. I will be able to join between 13:00 and 15:00 utc. Just ping me here. I can also join in other timeslots but you will need to tell me in advance
14:41:53 bauzas belmoreira: the only problem is that we also want to wait for melwitt to be there
14:42:16 bauzas belmoreira: so that's why we'd prefer to discuss the vncproxy stuff at 15:00 UTC if you don't mind
14:42:32 bauzas 15:10pm if we break before
14:43:06 belmoreira thanks bauzas. I will join today at 15:10 utc
14:46:41 bauzas belmoreira: thanks, appreciated a lot.
14:47:06 bauzas belmoreira: wait, sorry, it's 3:40pm
14:47:09 bauzas my bad
14:47:18 bauzas we have a interop discussion before
14:47:45 belmoreira ok, thanks
14:53:11 dansmith rbac session coming up in nova yeah?

Earlier   Later