| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-03-05 | |||
| 13:55:53 | gibi | bauzas: thanks | |
| 13:56:30 | bauzas | gibi: IIRC, we wanted to have the placeholders *before* any other db migration for a cycle | |
| 13:56:37 | gibi | yes | |
| 13:56:48 | gibi | so if we merge it at the start of the cycle then we are goo | |
| 13:56:49 | gibi | d | |
| 13:56:57 | bauzas | like, you write a db migration for wallaby, you need to have the placeholders *before* | |
| 13:57:17 | bauzas | sure, but then we are lucky to not have any merged db migration during this cycle | |
| 13:57:37 | bauzas | my point is, we should backport this sooner than later and write another change for reserving the xena ones early in the cycle | |
| 13:57:53 | bauzas | don't we have this in the ptl guide or something like it ? | |
| 13:58:22 | bauzas | https://review.opendev.org/c/openstack/nova/+/686411 is the Train one | |
| 13:58:36 | gibi | bauzas: post release check list https://wiki.openstack.org/wiki/Nova/ReleaseChecklist | |
| 13:59:34 | gibi | linked in the https://docs.openstack.org/nova/latest/contributor/ptl-guide.html | |
| 13:59:38 | bauzas | the above change was merged on oct 4th | |
| 13:59:49 | bauzas | (2019) | |
| 13:59:56 | bauzas | and the train timeline was https://releases.openstack.org/train/schedule.html | |
| 14:00:08 | gibi | according to git it was included from ussuri | |
| 14:00:09 | sean-k-mooney | bauzas: so for xena i think stephenfin wants to move to alemnbic | |
| 14:00:22 | sean-k-mooney | so that might be different in the future | |
| 14:00:29 | bauzas | gibi: /me facepalms | |
| 14:00:38 | sean-k-mooney | not sure if we still need to do reservations the same way | |
| 14:00:40 | bauzas | so, yeah, this was merged early in the cycle, not late | |
| 14:00:56 | gibi | bauzas: hm, I think ti was merged after RC1 and therefore after we branched out stable/train | |
| 14:01:05 | bauzas | sean-k-mooney: we'll open the xena cycle before the PTG, we need the placeholders sooner than later | |
| 14:01:33 | bauzas | gibi: sure, but at this point in time, the ussuri placeholders change was merged when we did the master branch to ussuri | |
| 14:01:33 | sean-k-mooney | bauzas: sure just pointing out thye may not be used | |
| 14:01:41 | gibi | bauzas: correct | |
| 14:01:41 | sean-k-mooney | that said they dont hurt to have really | |
| 14:01:49 | gibi | bauzas: so it is only in ussuri but not in train | |
| 14:02:08 | bauzas | gibi: correct, that's a PEBKAC when I said Train | |
| 14:02:37 | bauzas | oh wai | |
| 14:02:40 | gibi | sean-k-mooney: it is like we don't know yet how the alembic migration will look like so I think we have to do the safe move and add reservation | |
| 14:02:43 | bauzas | man, this is confusing | |
| 14:02:47 | bauzas | https://review.opendev.org/c/openstack/nova/+/686411/2/nova/tests/functional/db/api/test_migrations.py | |
| 14:03:08 | bauzas | this was merged just after the branch but we name them train migrations | |
| 14:03:29 | bauzas | so, hence my confusion, placeholders go last | |
| 14:03:39 | bauzas | and just after we cut | |
| 14:03:40 | gibi | yeah it is like placeholder for migration backported from ussuri to train | |
| 14:03:47 | gibi | so both name can be correct :) | |
| 14:03:48 | bauzas | gibi: so you were right, we should hold until the branhc | |
| 14:03:58 | sean-k-mooney | gibi: yep we should for that reason and because whilte stephenfin want to do the move next cycle something might come up which means we cant finish it | |
| 14:04:10 | sean-k-mooney | so we need to prepare for backport in anycase | |
| 14:04:13 | gibi | sean-k-mooney: correct | |
| 14:04:40 | bauzas | sean-k-mooney: gibi: like I said to stephenfin, before discussing it at the PTG, I'd like to dig why we didn't like that before | |
| 14:07:08 | sean-k-mooney | alembic | |
| 14:07:17 | bauzas | yup | |
| 14:07:34 | sean-k-mooney | it was manly because we did not feel the effort of the move was worth it similar to removing mox | |
| 14:07:44 | bauzas | I recall this | |
| 14:07:51 | bauzas | but IIRC there were other concerns | |
| 14:08:02 | bauzas | that's what I'd like to dif | |
| 14:08:03 | sean-k-mooney | but like mox that ment the nova team was signing up to maintian sqlalchemy-migrate indefinetly | |
| 14:08:05 | bauzas | dig* | |
| 14:08:24 | bauzas | sure, but this prove to work ? | |
| 14:08:42 | bauzas | we would just pull another dependency we loose control | |
| 14:08:46 | bauzas | over their releases | |
| 14:08:56 | sean-k-mooney | bauzas: we are using a dep that neutron uses already | |
| 14:09:04 | bauzas | I know | |
| 14:09:23 | sean-k-mooney | bauzas: and the maintiner of sqlachmey asked us to move | |
| 14:09:25 | bauzas | but that just increases the potential flaw, it doesn't prove it works | |
| 14:09:40 | sean-k-mooney | it does prove it works they have used it for years | |
| 14:10:01 | sean-k-mooney | im pretty sure neutron moved beofre we created placment | |
| 14:15:48 | bauzas | oh yes, I remember | |
| 14:15:54 | sean-k-mooney | neutron dont seam to have placeholder migration anymore https://github.com/openstack/neutron/tree/b04dfcee192cf7ddcb69854e760739fdcfa7839e/neutron/db/migration/alembic_migrations/versions | |
| 14:16:03 | bauzas | I actually never remember neutron using sqalchemy-migrate | |
| 14:16:15 | bauzas | even when they were named quantum | |
| 14:16:16 | sean-k-mooney | and they have tooling to auto genrate the migrations | |
| 14:16:27 | sean-k-mooney | bauzas: i tought quantum did but maybe not | |
| 14:16:51 | bauzas | well, i played with quantum with the Folsom release, and I don't think they were using migrate | |
| 14:17:01 | bauzas | but that's been a while | |
| 14:17:11 | sean-k-mooney | actully hevan used it alredy | |
| 14:17:15 | sean-k-mooney | so i guess not | |
| 14:17:41 | bauzas | they were incubated during Essex, so I didn't use it in my prod | |
| 14:17:52 | bauzas | can't speak for Essex | |
| 14:18:19 | bauzas | oh gosh, remembrance times... | |
| 14:20:25 | sean-k-mooney | looking at the eol tags | |
| 14:20:36 | sean-k-mooney | up to fulsom they had not migration in repo | |
| 14:21:19 | sean-k-mooney | so the first migration they added was grizzly and that used alembic | |
| 14:21:35 | sean-k-mooney | so for as long as they had migghration yes they used alembic to handel it | |
| 14:21:58 | bauzas | glad to know my memory haven't vanished :) | |
| 14:22:06 | bauzas | hasn't* | |
| 14:22:09 | sean-k-mooney | if it woks well enough for neutron which allows exptions to also have addtional db tables i think it should be flexably enough ot use in nova | |
| 14:22:24 | bauzas | again, this is not news | |
| 14:22:35 | sean-k-mooney | the main issue for use will be learning how to write them correctly and review them | |
| 14:22:36 | bauzas | that's my point | |
| 14:23:19 | bauzas | neutron uses it for ages, but by the times we discussed this for nova, we nacked the idea a couple of times, so I just want to refresh my memory before jumping into conclusions | |
| 14:23:57 | sean-k-mooney | sure but for me continuing to use it is like continuting ot use mox or python2 | |
| 14:24:28 | sean-k-mooney | sure they work but it woudl be better to simply the dendencis fo openstack an converg on alembic | |
| 14:37:53 | openstackgerrit | sean mooney proposed openstack/nova master: support per port numa policies with sriov https://review.opendev.org/c/openstack/nova/+/773792 | |
| 15:09:15 | melwitt | lyarwood, elod: thank you for the reviews :) fyi the stable/ussuri changes for the same bug, I stacked them on top of this other bug (2 patches) fix to eliminate merge conflicts. I figured it was a useful fix for stable anyways https://review.opendev.org/c/openstack/nova/+/777216 and https://review.opendev.org/c/openstack/nova/+/777217 | |
| 15:40:50 | claudiub | Hello, so, I have been looking a bit into the NUMA node 0 overcommit issue I was having. As a reminder, I had ~400 instaHello, so, I have been looking a bit into the NUMA node 0 overcommit issue I was having. As a reminder, I had ~400 instances in NUMA node 0, while NUMA node 1 was mostly empty. | |
| 15:40:58 | claudiub | Turns out, there seems to be an issue when live-migrate / host-evacuate-live is called with a target destination. | |
| 15:41:07 | claudiub | It seems that the CPU allocation and the memory allocation ratio is ignored in this case, leading to the overcommit, and even OOM, apparently. | |
| 15:41:24 | claudiub | So, from what I saw, the live_migrate task will eventually call check_can_live_migrate_destination on the destination node (https://github.com/openstack/nova/blob/c8a6f8d2e502e139c2212fb20cc0f89de816aeb1/nova/compute/manager.py#L7931), | |
| 15:41:33 | claudiub | which will then call _live_migration_clain and then self.rt.live_migration_clain. In there, a new claims.MoveClaim is created (https://github.com/openstack/nova/blob/c8a6f8d2e502e139c2212fb20cc0f89de816aeb1/nova/compute/resource_tracker.py#L298), | |
| 15:41:34 | stephenfin | You mean e.g. 'openstack server migrate --live HOST SERVER' ? | |
| 15:41:51 | claudiub | hich will then call hardware.numa_fit_instance_to_host to get the new instance_topology (https://github.com/openstack/nova/blob/c8a6f8d2e502e139c2212fb20cc0f89de816aeb1/nova/compute/claims.py#L138), which will then set it in self.claimed_numa_topology. In the hardware.numa_fit_instance_to_host call, | |
| 15:42:02 | claudiub | because the given limits is None, there is no check that CPU / memory is not overcommited in numa cell 0, so it gets scheduled there, since it's first match. https://github.com/openstack/nova/blob/5b52db409b00e6257e83598efa0d8528ebe380e0/nova/virt/hardware.py#L1168 | |
| 15:42:09 | claudiub | And indeed, the limits are None, from what I've seen in the extra logging I've added. | |
| 15:42:24 | claudiub | stephenfin: yep. :) | |
| 15:42:48 | stephenfin | How about if you do 'openstack server migrate --live-migration --host HOST SERVER'? | |
| 15:43:09 | stephenfin | '--live HOST' bypasses the scheduler which means the NUMATopologyFilter never triggers to populate limits | |