| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-03-08 | |||
| 13:31:53 | sean-k-mooney | we have the explaniaton in nova but it better to not need to cross reference | |
| 13:31:54 | gibi | requiered -> required | |
| 13:32:10 | sean-k-mooney | oh yes there is. | |
| 13:32:30 | gibi | also when artom's series lands with the socket value we need to document that for the port attribute too | |
| 13:32:30 | sean-k-mooney | only in the doc however | |
| 13:33:03 | sean-k-mooney | yep although that requires a neutron change to extend the db enum. its just a sting field but the model need to be extended | |
| 13:33:50 | gibi | I see | |
| 13:33:53 | gibi | good to know that | |
| 13:34:04 | gibi | I assumed it is an unrestricted string in neutron | |
| 13:34:12 | gibi | but the it is better | |
| 13:34:47 | sean-k-mooney | i think in terms fo the db coloum defiention yes but they are validating it in python | |
| 13:37:56 | sean-k-mooney | the only thing that is slightly unfortunete is they enabled each policy with a seperate flag in osc | |
| 13:38:21 | sean-k-mooney | rather then a vaule that takes a sting paramter but its not a huge issue | |
| 13:42:52 | admin0 | sean-k-mooney, i am using kvm .. first the flavor was created and the limit was set .. after that the instance was booted up | |
| 13:43:45 | sean-k-mooney | admin0: yes but you use a vaule that is only supproted for vmware | |
| 13:43:54 | sean-k-mooney | admin0: try the same with the one i pointed too | |
| 13:44:12 | admin0 | oh | |
| 13:44:15 | admin0 | i get it | |
| 13:44:16 | admin0 | thank you | |
| 13:44:18 | admin0 | i will retry | |
| 13:44:18 | sean-k-mooney | openstack flavor set g6 --property quota:disk_total_iops_sec=800 | |
| 13:45:29 | sean-k-mooney | that one and the others in the disk tuning section of https://docs.openstack.org/nova/latest/user/flavors.html work with libvirt | |
| 13:46:22 | openstackgerrit | Merged openstack/nova master: Replace md5 for fips https://review.opendev.org/c/openstack/nova/+/777686 | |
| 13:47:18 | openstackgerrit | Merged openstack/nova master: Centralize sqlite FK constraint enforcement https://review.opendev.org/c/openstack/nova/+/773833 | |
| 13:57:32 | openstackgerrit | sean mooney proposed openstack/nova master: support per port numa policies with sriov https://review.opendev.org/c/openstack/nova/+/773792 | |
| 13:58:20 | sean-k-mooney | alex_xu: stephenfin would either of ye have time to review ^ gibi made the doc update and fixed the pep8 issue. | |
| 13:59:39 | gibi | sean-k-mooney: thanks +2 | |
| 14:00:53 | sean-k-mooney | just lookign at your sqlite change that really is just refactoring it so that its cenratlised since we already had the dynamic logic. do we run with 3.7+ in the gate? | |
| 14:01:39 | sean-k-mooney | i assume we have no way to requrie a minium of 3.7 beyond moving to an os that provides it | |
| 14:01:42 | gibi | sean-k-mooney: I think you are referring to melwitt's change :) | |
| 14:01:52 | sean-k-mooney | https://review.opendev.org/c/openstack/nova/+/773833 | |
| 14:01:57 | sean-k-mooney | the one that just merged | |
| 14:02:03 | gibi | yepp | |
| 14:02:05 | gibi | that one | |
| 14:02:39 | sean-k-mooney | oh did melwitt write that oh she did sorry still find part of the new gerrit confusing | |
| 14:02:53 | gibi | we only have https://github.com/openstack/requirements/blob/master/bindep.txt#L18 in bindep about sqlite | |
| 14:03:16 | gibi | so I guess it depends on the host os | |
| 14:03:28 | sean-k-mooney | ya which is fine i guess | |
| 14:04:00 | sean-k-mooney | aslong as it compatible with our version of sqlalchmy then it should not impact use much since its mainly for testing | |
| 14:04:01 | gibi | yeah, it is just test dep | |
| 14:04:30 | sean-k-mooney | ah yes i rememebr the downstream bug melwitt is fixing | |
| 14:05:10 | sean-k-mooney | the partal archiving of db rows was causing odd behavior for the simple tenant usage api on our internal cloud | |
| 14:06:23 | gibi | yes I just read the fix today for that | |
| 14:06:26 | sean-k-mooney | melwitt: were you going to bring up the idea of using cacade delelets in the ptg | |
| 14:07:50 | sean-k-mooney | gibi: ya melwitt is fixing it in a way that potentally backportable. an alternitive is so use a cascading delete trigger and let the db do it atomically | |
| 14:08:01 | sean-k-mooney | but there were reasons we dont use those in nova in the past | |
| 14:08:18 | sean-k-mooney | neutron does use them but i dont think thye have shadow tables | |
| 14:08:21 | gibi | but we dont delete here we archive here | |
| 14:09:04 | sean-k-mooney | the issue downstream was beacuse we archive and purge at the same time | |
| 14:09:31 | sean-k-mooney | so when the purge is happening with a limit on the rows it can cause half deleted recored until the next time the purge happens | |
| 14:09:33 | gibi | we we replace archiving with direct cascad delete that could work | |
| 14:09:45 | gibi | s/we we/if we/ | |
| 14:10:34 | sean-k-mooney | ya there has been some work done by ovh to do archiving externally with https://github.com/ovh/osarchiver | |
| 14:11:04 | sean-k-mooney | they are going to contbute that to either os-opstools or to a new openstack/ repo | |
| 14:11:32 | sean-k-mooney | if that was adopted instead it could allow us to remove archiving form the project entirly | |
| 14:12:04 | sean-k-mooney | proably somethign we shoudl discuss at the ptg or on the maining list | |
| 14:13:33 | sean-k-mooney | they developed it orginally for manila which dose not have shadow tabels and then started using it for all projects in production to have common way to do it | |
| 14:15:56 | openstackgerrit | Lee Yarwood proposed openstack/nova master: zuul: Remove nova-dsvm-multinode-base https://review.opendev.org/c/openstack/nova/+/778908 | |
| 14:15:56 | openstackgerrit | Lee Yarwood proposed openstack/nova master: WIP zuul: Replace grenade and nova-grenade-multinode with grenade-multinode https://review.opendev.org/c/openstack/nova/+/778885 | |
| 14:15:57 | openstackgerrit | Lee Yarwood proposed openstack/nova master: zuul: Create and use nova-libvirt-irrelevant-files https://review.opendev.org/c/openstack/nova/+/778909 | |
| 14:20:30 | lemko | power_state: 4". Any idea? | |
| 14:20:30 | lemko | Hi, since I upgraded my openstack to Victoria (with upgrade of libvirt), my instances are randomly dying for no reason... "shutting down, reason=crashed" from qemu. Otherwise I can see this from nova-compute "Instance shutdown by itself. Calling the stop API. Current vm_state: active, current task_state: None, original DB power_state: 1, current VM | |
| 14:21:07 | sean-k-mooney | lemko: did you check the qemu instance log | |
| 14:21:12 | lemko | Yes. | |
| 14:21:24 | sean-k-mooney | lemko: /var/lib/libvirt/qemu/instace* | |
| 14:21:36 | sean-k-mooney | were there any errors in the log | |
| 14:21:41 | lemko | "shutting down, reason=crashed" is the only line interesting | |
| 14:22:02 | sean-k-mooney | ok that looks like a qemu or kvm issue then not openstack | |
| 14:22:28 | sean-k-mooney | did you have any OOM erros in dmesg/journalctl | |
| 14:22:48 | sean-k-mooney | its possible the kernel is killing them if the host ran out of memory | |
| 14:22:57 | sean-k-mooney | but more likely you are hitting a qemu bug | |
| 14:27:36 | lemko | I do not see anything related to OOM. | |
| 14:28:40 | sean-k-mooney | there are some more detailed logging option you can enabel for libvirt/qemu to help debug kashyap has a write up | |
| 14:28:52 | kashyap | We can't tell yet what bug it is, without more information on the error / crash | |
| 14:29:19 | sean-k-mooney | kashyap: do you have teh link to how to enable the debug logging for qemu instances | |
| 14:29:25 | kashyap | lemko: Try this, if you can use 'virt-admin' tool -- https://kashyapc.fedorapeople.org/virt/virt-admin.txt | |
| 14:29:28 | kashyap | sean-k-mooney: --^ | |
| 14:29:33 | sean-k-mooney | thanks :) | |
| 14:29:56 | kashyap | sean-k-mooney: lemko: Another option, the manual way (if you don't have 'virt-admin' tool): https://kashyapc.fedorapeople.org/virt/openstack/request-nova-libvirt-qemu-debug-logs.txt | |
| 14:31:14 | kashyap | lemko: Just to eliminate one line of debugging: you're not using nested KVM, are you? I.e. your compute node is baremetal, yes? | |
| 14:32:11 | kashyap | lemko: If you're the second URL I posted, just go straight to the bottom, this section: "To manually enable libvirt log filters" | |
| 14:38:53 | admin0 | sean-k-mooney, thank you .. it works as required | |
| 14:45:59 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Create qcow2 disks with the correct size without extending https://review.opendev.org/c/openstack/nova/+/779275 | |
| 15:12:13 | kashyap | The new Gerrit has been significantly slower for me; not sure if it's just me | |
| 15:13:36 | sean-k-mooney | its because of the db storage in git | |
| 15:13:50 | sean-k-mooney | it increase io on the server side | |
| 15:14:15 | sean-k-mooney | at least that is my understanding | |
| 15:21:42 | kashyap | I see | |
| 15:26:40 | artom | I need to clear my email filters and restart them from scratch | |
| 15:27:00 | artom | It's been incorrectly flagging actual people reviews as third party CIs, and marking them read | |
| 15:27:03 | artom | Leading to me missing reviews | |
| 15:38:31 | openstackgerrit | Claudiu Belu proposed openstack/nova master: live migration: Pass limits onto scheduler-chosen destination https://review.opendev.org/c/openstack/nova/+/779285 | |
| 15:40:32 | openstackgerrit | Claudiu Belu proposed openstack/nova master: live migration: Pass limits onto scheduler-chosen destination https://review.opendev.org/c/openstack/nova/+/779285 | |
| 15:42:37 | claudiub | ^ solved my NUMA node 0 overcommit due to live-migration issue. :) | |
| 15:44:05 | sean-k-mooney | im not sure how that would solve your issue | |
| 15:44:40 | sean-k-mooney | claudiub: its still not going to result in the vms decrementing resouces in the host numa toplogy blob | |
| 15:48:44 | claudiub | basically, just passing those limits we get from the scheduler to nova-compute. Those limits were None previously, so the CPU and memory allocation ratios were never checked, so NUMA node 0 was always a valid candidate for live-migration, even though it shouldn't if it's over the limit already. Now, when live-migrating, if the NUMA node 0 is full, they will be placed in NUMA node 1. | |
| 15:49:41 | sean-k-mooney | if you just use hw:numa_nodes=1 it will never be full | |
| 15:49:52 | sean-k-mooney | and you should have got that full behavior before | |
| 15:50:08 | claudiub | it wasn't only full, it was overscheduled. | |