| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-09-11 | |||
| 07:59:22 | sahid | I mean as admin | |
| 07:59:35 | sahid | that to valid the behavior of the host | |
| 08:34:00 | tobias-urdin | we usually create a instance then live migrate it to the host to test it out without enabling it for scheduling | |
| 10:03:32 | sahid | tobias-urdin: interesting workaround thank you | |
| 10:03:46 | sahid | btw it may be useful for an admin to have this priviledge | |
| 10:06:01 | tis_at_teuto | Hi, not shure if this is the right place to ask but currently we have a problem mit oom-kill pcpu-instances. They got oom-killed with `oom-kill:constraint=CONSTRAINT_MEMORY_POLICY`. We use pinned cpus and nova-compute sets the the libvirt numatune.memory and numatune.memnode statically to `mode=strict` https://github.com/openstack/nova/blob/master/nova/virt/libvirt/config.py#L2812 . Is it possible to get this configurable or | |
| 10:06:04 | tis_at_teuto | use preferred instead like mentioned in https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/6/html/virtualization_tuning_and_optimization_guide/sect-virtualization_tuning_optimization_guide-numa-allocation_policy#sect-Virtualization_Tuning_Optimization_Guide-NUMA-Allocation_Policy | |
| 11:40:04 | opendevreview | Merged openstack/osc-placement stable/2024.2: Update .gitreview for stable/2024.2 https://review.opendev.org/c/openstack/osc-placement/+/928358 | |
| 11:45:13 | sean-k-mooney | tobias-urdin: sahid if you have disabeld the compute you cannot live migrate to it by defualt. to do so you have to use the old microversin and force the migration | |
| 11:45:27 | sean-k-mooney | sahid: we intentionally do now allow this to work by defualt | |
| 11:45:48 | sean-k-mooney | when a comptue service is disabeld you shuld be able to move instance away form it but not to it | |
| 11:46:09 | opendevreview | Merged openstack/osc-placement stable/2024.2: Update TOX_CONSTRAINTS_FILE for stable/2024.2 https://review.opendev.org/c/openstack/osc-placement/+/928359 | |
| 11:46:40 | sean-k-mooney | tis_at_teuto: my initall gues is that you have not defied hw:mem_page_size in your pinned instance | |
| 11:47:16 | sean-k-mooney | tis_at_teuto: when using cpu pinnign either you need to request numa aware memory usign hw:mem_page_size or you need to configure the host for file backed memroy | |
| 11:47:48 | sean-k-mooney | if you dont do one of those two then we expect OOM issue for the vms, its an unsupproted configuraiton. | |
| 11:48:43 | sean-k-mooney | tis_at_teuto: we do not supprot settign the memroy policy to prefered or memory overcommit when using pinned cpus | |
| 11:49:17 | sean-k-mooney | there was a proposal for that many many years ago but it was never implemented | |
| 12:29:53 | opendevreview | Lajos Katona proposed openstack/nova master: docs: update libvirt support matrix for C and D https://review.opendev.org/c/openstack/nova/+/928903 | |
| 12:30:45 | opendevreview | Lajos Katona proposed openstack/nova master: docs: update libvirt support matrix for C and D https://review.opendev.org/c/openstack/nova/+/928903 | |
| 12:30:58 | tis_at_teuto | sean-k-mooney: Ok, thank you. I will have a look into `hw:mem_page_size`. In the meantime we mitigate the problem by configuring a swap-file. I think that is what you second approach is. | |
| 12:31:28 | sean-k-mooney | swapfiles will only help if you have multiple of them and also mark them as related to a given numa node | |
| 12:31:39 | sean-k-mooney | the OOM reaper runs on a per numa basis | |
| 12:31:58 | sean-k-mooney | so even if you have lots of free swap you can still have OOM issues | |
| 12:32:12 | sean-k-mooney | or free ram on the wrong numa node | |
| 12:33:48 | lajoskatona | bauzas: https://review.opendev.org/c/openstack/nova/+/928903 , pushed an update for qemu/libvirt support matrix, as I see no change in it in C and D, just to be clear for end-users | |
| 12:34:08 | sahid | sean-k-mooney: sean-k-mooney: do you remeber the microversion to use, even when i try to live-migrate on the specified host, I receive the error nova valid host was found | |
| 12:35:37 | sean-k-mooney | force was added in 2.30 and removed in 2.68 so 2.30-2.67 | |
| 12:36:02 | sean-k-mooney | sahid: note that force skips schdluer filters | |
| 12:36:28 | sean-k-mooney | it just checks the host exists, placement should still confirm that there is enouch suport for basic resouces | |
| 12:38:25 | sean-k-mooney | note that the compute status filter has been mandaory for many releases | |
| 12:38:28 | sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/scheduler/request_filter.py#L242-L255 | |
| 12:38:46 | sean-k-mooney | so where that is not configurabel there is no way to schdule to a disabeld host by design | |
| 12:40:11 | sean-k-mooney | ah i miss rememeberd we intoduced this as a mandatoy prefilter 5 years ago | |
| 12:40:13 | sean-k-mooney | https://github.com/openstack/nova/commit/168d34c8d1161dc4d62493e194819297e079bb51 | |
| 12:40:28 | sean-k-mooney | so since placment because mandatory there is not way by design to schduler to disabled computes | |
| 13:38:01 | bauzas | sean-k-mooney: thanks for your work on the "vfat is non-bootable so we can skip it" series, just +2d saying this is fine given oslo will support vfat checks shortly | |
| 13:38:23 | bauzas | if not, I'd be afraid of another attack vector | |
| 13:38:48 | sean-k-mooney | there is an oslo patch for it reference form the 3rd patch | |
| 13:38:57 | bauzas | yes I know, hence my +2 | |
| 13:39:22 | sean-k-mooney | im also considerign doing a quick poc of both makeing epmeral disk unformated and removing the backing files form them | |
| 13:39:53 | sean-k-mooney | with that said i dont really have much time to look at either but since im already looking at that code i was going to spend an hour or two and see if they would be easy to do for epoxy | |
| 13:40:49 | sean-k-mooney | the reason i choose bootable as the new flag is i think tis reasonabel to say in the qemu driver that non-bootable disk should not have backing files ever. | |
| 13:41:09 | sean-k-mooney | but i have not really tought about all the implications yet so i didnt want to include that in this inital work | |
| 13:43:06 | bauzas | no worries, given by 2025.1 we will be able to look at vfat, we could remove the skip for non-bootable disks | |
| 13:43:48 | sean-k-mooney | we could thats definetly a diesign disucsison we shoudl have early in 2025.1 | |
| 13:44:16 | sean-k-mooney | in my head im kind fo grouping this with the geneal "state of the image backends" topic | |
| 13:44:44 | sean-k-mooney | i think this is just one of the many workitems that go into improving that going forward | |
| 14:18:12 | opendevreview | Dmitriy Rabotyagov proposed openstack/nova master: Respect supplied arguments in novncproxy_base_url https://review.opendev.org/c/openstack/nova/+/928839 | |
| 14:32:02 | opendevreview | Kaloyan Kotlarski proposed openstack/nova master: libvirt: Fetch the available vCPUs from the respective cgroup https://review.opendev.org/c/openstack/nova/+/927474 | |
| 15:02:13 | sahid | alian | |
| 15:02:16 | sahid | alianz | |
| 15:23:27 | dansmith | sean-k-mooney: safe sounds okay to me | |
| 15:24:24 | sean-k-mooney | ok i can go change that how would you feel about using that for skiping the backing file too? | |
| 15:24:30 | sean-k-mooney | in a seperate patch obviously | |
| 15:24:53 | sean-k-mooney | i.e. not haveing swap an ephmeral have backing files with qcow backend | |
| 15:25:25 | sean-k-mooney | dansmith: ill fix the typoes ectra but were there any other concerns | |
| 15:25:42 | sean-k-mooney | ill repin the patch after the our internal calls end | |
| 15:27:12 | dansmith | sean-k-mooney: I have concerns about how many places you had to make that change :P | |
| 15:27:18 | dansmith | but that's imagebackend's fault not yours | |
| 15:33:44 | sean-k-mooney | ya at least 1 of those i was really not expecting | |
| 15:33:52 | sean-k-mooney | we have a lot of duplication | |
| 15:55:36 | opendevreview | Rajesh Tailor proposed openstack/nova master: Add support for showing finish_time https://review.opendev.org/c/openstack/nova/+/928933 | |
| 16:06:12 | opendevreview | Dan Smith proposed openstack/nova stable/2023.1: [stable-only]: Port qcow2v2 support from oslo https://review.opendev.org/c/openstack/nova/+/928937 | |
| 16:57:38 | bauzas | we need a second core for https://review.opendev.org/c/openstack/nova/+/928662 please :) | |
| 16:59:46 | sean-k-mooney | that looks correct to me | |
| 17:01:17 | bauzas | thanks | |
| 19:03:15 | opendevreview | sean mooney proposed openstack/nova master: [DNM] testing rdo experimental jobs https://review.opendev.org/c/openstack/nova/+/927762 | |
| 19:05:18 | opendevreview | sean mooney proposed openstack/nova master: Add functional repoducer for ephemeral disks https://review.opendev.org/c/openstack/nova/+/928310 | |
| 19:05:19 | opendevreview | sean mooney proposed openstack/nova master: only safety check bootable files created from glance https://review.opendev.org/c/openstack/nova/+/928829 | |
| 19:05:19 | opendevreview | sean mooney proposed openstack/nova master: adapt to vfat support in oslo.utils https://review.opendev.org/c/openstack/nova/+/928462 | |
| 21:12:57 | opendevreview | Matthew Heler proposed openstack/nova master: Fix regression with live migration on shared storage https://review.opendev.org/c/openstack/nova/+/928970 | |
| 21:18:15 | supamatt | Can I get reviews on this? https://review.opendev.org/c/openstack/nova/+/928970 It's a regression fix to how live migration should function on shared storage. | |
| #openstack-nova - 2024-09-12 | |||
| 01:20:53 | opendevreview | melanie witt proposed openstack/nova master: DNM try minimalistic imagebackend refactor https://review.opendev.org/c/openstack/nova/+/925635 | |
| 05:00:22 | opendevreview | melanie witt proposed openstack/nova master: DNM try minimalistic imagebackend refactor https://review.opendev.org/c/openstack/nova/+/925635 | |
| 06:42:40 | gibi | bauzas: supamatt's bug feels like an RC candidate https://review.opendev.org/c/openstack/nova/+/928970 | |
| 06:43:04 | gibi | bauzas: I've added it to the tracking etherpad and I will review it properly after a coffee | |
| 06:43:19 | gibi | supamatt: thanks for bringing this to our attention | |
| 07:15:40 | gibi | supamatt: I have doubts about the problem the code pointed out by the report and the fix does not effect the setting of the destroy_disks variable that is reponsible to signal deletion of files from the disk at cleanup | |
| 07:23:13 | gibi | it seems nobody is around so I will update the bug report with my doubt/questions | |
| 07:43:43 | gibi | done | |
| 07:47:21 | bauzas | gibi: ack, sorry, I had trouble with my home network | |
| 07:48:21 | gibi | bauzas: no worries | |
| 07:48:49 | bauzas | gibi: this is on 2024.1 | |
| 07:48:55 | bauzas | Caracal so | |
| 07:49:04 | gibi | bauzas: it was a backport from master | |
| 07:49:07 | gibi | merged this cycle | |
| 07:49:12 | bauzas | technically, we could deliver RC1 and then backport that one in a .z release | |
| 07:49:28 | gibi | https://review.opendev.org/c/openstack/nova/+/909806 | |
| 07:49:29 | bauzas | okay, lemme look at the commit SHA1 | |
| 07:49:40 | gibi | that is the offending patch according to the reporter | |
| 07:50:29 | gibi | bauzas: I lean towards that the reporter do see a numa live migration regression on shared storage but I'm not sure that the root cause is the power mgmt flag added. But more like that flag now trigger a driver cleanup call that wasn't triggered before | |
| 07:50:49 | gibi | and that cleanup call might have issues | |
| 07:50:58 | bauzas | agreed | |
| 07:51:02 | bauzas | that's the cleanup | |
| 07:53:01 | bauzas | gibi: oh, that's the conditional | |
| 07:53:16 | bauzas | we only cleanup if we don't have shared path | |
| 07:53:28 | bauzas | but here, the 'or' makes it now optional | |
| 07:53:47 | bauzas | we just need to change the conditional | |
| 07:56:04 | gibi | calling the cleanup with destroy_disks should be OK | |
| 07:56:17 | gibi | destroy_disk= false | |