| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-03-08 | |||
| 14:06:23 | gibi | yes I just read the fix today for that | |
| 14:06:26 | sean-k-mooney | melwitt: were you going to bring up the idea of using cacade delelets in the ptg | |
| 14:07:50 | sean-k-mooney | gibi: ya melwitt is fixing it in a way that potentally backportable. an alternitive is so use a cascading delete trigger and let the db do it atomically | |
| 14:08:01 | sean-k-mooney | but there were reasons we dont use those in nova in the past | |
| 14:08:18 | sean-k-mooney | neutron does use them but i dont think thye have shadow tables | |
| 14:08:21 | gibi | but we dont delete here we archive here | |
| 14:09:04 | sean-k-mooney | the issue downstream was beacuse we archive and purge at the same time | |
| 14:09:31 | sean-k-mooney | so when the purge is happening with a limit on the rows it can cause half deleted recored until the next time the purge happens | |
| 14:09:33 | gibi | we we replace archiving with direct cascad delete that could work | |
| 14:09:45 | gibi | s/we we/if we/ | |
| 14:10:34 | sean-k-mooney | ya there has been some work done by ovh to do archiving externally with https://github.com/ovh/osarchiver | |
| 14:11:04 | sean-k-mooney | they are going to contbute that to either os-opstools or to a new openstack/ repo | |
| 14:11:32 | sean-k-mooney | if that was adopted instead it could allow us to remove archiving form the project entirly | |
| 14:12:04 | sean-k-mooney | proably somethign we shoudl discuss at the ptg or on the maining list | |
| 14:13:33 | sean-k-mooney | they developed it orginally for manila which dose not have shadow tabels and then started using it for all projects in production to have common way to do it | |
| 14:15:56 | openstackgerrit | Lee Yarwood proposed openstack/nova master: WIP zuul: Replace grenade and nova-grenade-multinode with grenade-multinode https://review.opendev.org/c/openstack/nova/+/778885 | |
| 14:15:56 | openstackgerrit | Lee Yarwood proposed openstack/nova master: zuul: Remove nova-dsvm-multinode-base https://review.opendev.org/c/openstack/nova/+/778908 | |
| 14:15:57 | openstackgerrit | Lee Yarwood proposed openstack/nova master: zuul: Create and use nova-libvirt-irrelevant-files https://review.opendev.org/c/openstack/nova/+/778909 | |
| 14:20:30 | lemko | Hi, since I upgraded my openstack to Victoria (with upgrade of libvirt), my instances are randomly dying for no reason... "shutting down, reason=crashed" from qemu. Otherwise I can see this from nova-compute "Instance shutdown by itself. Calling the stop API. Current vm_state: active, current task_state: None, original DB power_state: 1, current VM | |
| 14:20:30 | lemko | power_state: 4". Any idea? | |
| 14:21:07 | sean-k-mooney | lemko: did you check the qemu instance log | |
| 14:21:12 | lemko | Yes. | |
| 14:21:24 | sean-k-mooney | lemko: /var/lib/libvirt/qemu/instace* | |
| 14:21:36 | sean-k-mooney | were there any errors in the log | |
| 14:21:41 | lemko | "shutting down, reason=crashed" is the only line interesting | |
| 14:22:02 | sean-k-mooney | ok that looks like a qemu or kvm issue then not openstack | |
| 14:22:28 | sean-k-mooney | did you have any OOM erros in dmesg/journalctl | |
| 14:22:48 | sean-k-mooney | its possible the kernel is killing them if the host ran out of memory | |
| 14:22:57 | sean-k-mooney | but more likely you are hitting a qemu bug | |
| 14:27:36 | lemko | I do not see anything related to OOM. | |
| 14:28:40 | sean-k-mooney | there are some more detailed logging option you can enabel for libvirt/qemu to help debug kashyap has a write up | |
| 14:28:52 | kashyap | We can't tell yet what bug it is, without more information on the error / crash | |
| 14:29:19 | sean-k-mooney | kashyap: do you have teh link to how to enable the debug logging for qemu instances | |
| 14:29:25 | kashyap | lemko: Try this, if you can use 'virt-admin' tool -- https://kashyapc.fedorapeople.org/virt/virt-admin.txt | |
| 14:29:28 | kashyap | sean-k-mooney: --^ | |
| 14:29:33 | sean-k-mooney | thanks :) | |
| 14:29:56 | kashyap | sean-k-mooney: lemko: Another option, the manual way (if you don't have 'virt-admin' tool): https://kashyapc.fedorapeople.org/virt/openstack/request-nova-libvirt-qemu-debug-logs.txt | |
| 14:31:14 | kashyap | lemko: Just to eliminate one line of debugging: you're not using nested KVM, are you? I.e. your compute node is baremetal, yes? | |
| 14:32:11 | kashyap | lemko: If you're the second URL I posted, just go straight to the bottom, this section: "To manually enable libvirt log filters" | |
| 14:38:53 | admin0 | sean-k-mooney, thank you .. it works as required | |
| 14:45:59 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Create qcow2 disks with the correct size without extending https://review.opendev.org/c/openstack/nova/+/779275 | |
| 15:12:13 | kashyap | The new Gerrit has been significantly slower for me; not sure if it's just me | |
| 15:13:36 | sean-k-mooney | its because of the db storage in git | |
| 15:13:50 | sean-k-mooney | it increase io on the server side | |
| 15:14:15 | sean-k-mooney | at least that is my understanding | |
| 15:21:42 | kashyap | I see | |
| 15:26:40 | artom | I need to clear my email filters and restart them from scratch | |
| 15:27:00 | artom | It's been incorrectly flagging actual people reviews as third party CIs, and marking them read | |
| 15:27:03 | artom | Leading to me missing reviews | |
| 15:38:31 | openstackgerrit | Claudiu Belu proposed openstack/nova master: live migration: Pass limits onto scheduler-chosen destination https://review.opendev.org/c/openstack/nova/+/779285 | |
| 15:40:32 | openstackgerrit | Claudiu Belu proposed openstack/nova master: live migration: Pass limits onto scheduler-chosen destination https://review.opendev.org/c/openstack/nova/+/779285 | |
| 15:42:37 | claudiub | ^ solved my NUMA node 0 overcommit due to live-migration issue. :) | |
| 15:44:05 | sean-k-mooney | im not sure how that would solve your issue | |
| 15:44:40 | sean-k-mooney | claudiub: its still not going to result in the vms decrementing resouces in the host numa toplogy blob | |
| 15:48:44 | claudiub | basically, just passing those limits we get from the scheduler to nova-compute. Those limits were None previously, so the CPU and memory allocation ratios were never checked, so NUMA node 0 was always a valid candidate for live-migration, even though it shouldn't if it's over the limit already. Now, when live-migrating, if the NUMA node 0 is full, they will be placed in NUMA node 1. | |
| 15:49:41 | sean-k-mooney | if you just use hw:numa_nodes=1 it will never be full | |
| 15:49:52 | sean-k-mooney | and you should have got that full behavior before | |
| 15:50:08 | claudiub | it wasn't only full, it was overscheduled. | |
| 15:50:29 | sean-k-mooney | no hw:numa_nodes does not claim/decrement that avaiable resouce on the host | |
| 15:50:43 | sean-k-mooney | without hw:cpu_policy=dedicated or hw:mem_page_size set | |
| 15:53:02 | sean-k-mooney | claudiub: at a minium you need to add func test for this here https://github.com/openstack/nova/blob/master/nova/tests/functional/libvirt/test_numa_servers.py | |
| 15:53:21 | claudiub | sure, will do. | |
| 15:53:24 | sean-k-mooney | and proably in https://github.com/openstack/nova/blob/master/nova/tests/functional/libvirt/test_numa_live_migration.py | |
| 15:54:55 | sean-k-mooney | basicaly you will need to boot 2 vms with just hw:numa_nodes=1 and show that the first one fills numa0 and the second get soft pinned to numa 1 | |
| 15:56:34 | openstackgerrit | Merged openstack/nova master: libvirt: Add firmware auto-configuration support https://review.opendev.org/c/openstack/nova/+/776678 | |
| 15:56:36 | claudiub | Anyways, that's what we've seen in our Train environment. For example, we had a node with 400+ instances on NUMA node 0 (max should have been ~320), and if we would do a live-migration and that node was chosen as the destination, the instance would be placed in NUMA node 0. After this fix, the instances were being placed in NUMA node 1 instead. | |
| 15:57:32 | sean-k-mooney | right but as i said as far as i am aware you fix should not actully fix it. it may be i doubt its a full fix for all edgecases | |
| 15:59:46 | claudiub | it's a simple fix for a simple usecase that can also be easily be backported. Definitely, I haven't fixed everything about NUMA in nova, but I don't think it detracts anything if we add it. | |
| 16:00:37 | sean-k-mooney | claudiub: we will see ill try an play with it after code feeze, are you ok waiting until after that. | |
| 16:01:19 | sean-k-mooney | if this is backporaable it can be backported after FF anyway | |
| 16:02:22 | claudiub | sure. As I've said, we tried this in Train, and we seem to be more stable now when it comes to live-migrations. | |
| 16:14:30 | melwitt | sean-k-mooney: no, not planning to talk about cascading deletes. that would require removing soft-deletion which from past attempts would be very involved and I don't have the bandwidth to get into it. it would also change behaviors like being able to view the instance action list of a deleted instance, there's also a literal soft delete instance API that we'd have to decide what to do with, and so on. I think it would likely need a | |
| 16:14:30 | melwitt | deprecation cycle(s) to get rid of because of the changes in behavior if we were to make soft deletion no longer possible | |
| 16:15:41 | sean-k-mooney | melwitt: ack cool | |
| 16:17:23 | lemko | kashyap, nope, I'm not using nested virtualizaiton | |
| 16:34:36 | melwitt | sean-k-mooney, gibi: the bug downstream will happen even if we don't purge at the same time as archive. the issue is only with archive. archive moves a record to the shadow table, so it deletes it from the main table and inserts it in the shadow table. we use the word "purge" to refer to deleting things from the shadow tables i.e. 'nova-manage db purge' | |
| 16:35:12 | gibi | melwitt: thanks | |
| 16:42:45 | openstackgerrit | Balazs Gibizer proposed openstack/nova stable/train: Warn when starting services with older than N-1 computes https://review.opendev.org/c/openstack/nova/+/779293 | |
| 16:42:46 | openstackgerrit | Balazs Gibizer proposed openstack/nova stable/train: Add upgrade check about old computes https://review.opendev.org/c/openstack/nova/+/779294 | |
| 16:58:51 | sean-k-mooney | bauzas: https://review.opendev.org/c/openstack/nova/+/773792 is the pci port numa policy patch. it nit a volume detach issue thats unrelated so just recheked it there a while ago | |
| 17:06:47 | openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Add tests for insecure UEFI configuration https://review.opendev.org/c/openstack/nova/+/777367 | |
| 17:06:48 | openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Stop passing around virt_type, caps https://review.opendev.org/c/openstack/nova/+/775689 | |
| 17:06:48 | openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Add missing type hints https://review.opendev.org/c/openstack/nova/+/775688 | |
| 17:06:49 | openstackgerrit | Stephen Finucane proposed openstack/nova master: trivial: Clarify purpose of 'Host.supports_*' properties https://review.opendev.org/c/openstack/nova/+/778739 | |
| 17:06:49 | openstackgerrit | Stephen Finucane proposed openstack/nova master: hardware: Start parsing 'os_secure_boot' https://review.opendev.org/c/openstack/nova/+/682628 | |
| 17:06:50 | openstackgerrit | Stephen Finucane proposed openstack/nova master: compute: Report COMPUTE_SECURITY_UEFI_SECURE_BOOT https://review.opendev.org/c/openstack/nova/+/776679 | |
| 17:06:50 | openstackgerrit | Stephen Finucane proposed openstack/nova master: scheduler: Translate secure boot requests to trait https://review.opendev.org/c/openstack/nova/+/776680 | |
| 17:06:51 | openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Wire up 'os_secure_boot' property https://review.opendev.org/c/openstack/nova/+/776681 | |
| 17:06:51 | openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Report secure boot support to scheduler https://review.opendev.org/c/openstack/nova/+/775690 | |
| 17:06:52 | openstackgerrit | Stephen Finucane proposed openstack/nova master: tests: Add functional tests for UEFI, secure boot https://review.opendev.org/c/openstack/nova/+/776682 | |
| 17:06:52 | openstackgerrit | Stephen Finucane proposed openstack/nova master: tests: Remove duplicated 'start_compute' helper https://review.opendev.org/c/openstack/nova/+/776683 | |
| 17:06:53 | openstackgerrit | Stephen Finucane proposed openstack/nova master: docs: Document UEFI secure boot feature https://review.opendev.org/c/openstack/nova/+/776684 | |
| 17:06:53 | openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Add parsing of firmware metadata files https://review.opendev.org/c/openstack/nova/+/779302 | |
| 17:06:54 | openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Move some host arch checks to guest checks https://review.opendev.org/c/openstack/nova/+/779303 | |
| 17:06:54 | openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Use firmware metadata files to configure instance https://review.opendev.org/c/openstack/nova/+/779304 | |
| 17:07:33 | stephenfin | Phew | |
| 17:13:20 | lyarwood | stephenfin: I'll hit that again in the morning | |
| 17:13:36 | lyarwood | or maybe later tonight if I get time | |
| 18:30:22 | openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Enable 'vmcoreinfo' feature by default https://review.opendev.org/c/openstack/nova/+/752912 | |
| 18:30:23 | openstackgerrit | Stephen Finucane proposed openstack/nova master: WIP: libvirt: Make vmcoreinfo dependent on feature flag https://review.opendev.org/c/openstack/nova/+/779342 | |