| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-11-19 | |||
| 19:00:45 | sean-k-mooney | anyway none of our docs say ^ is a thing you can do. | |
| 23:10:26 | NobodyCam | Good Afternoon Nova folks, I've started seeing hypervisor state flapping between up and down, The hypervisors are up, I've read that adjusting server_down_time and report_interval can help with this, are these the correct setting to adjust and is there a way to gauge what these values should be? | |
| 23:31:34 | melwitt | NobodyCam: are you experiencing a problem with getting occasional NoValidHost because of this or seeing the hypervisor state going up and down or both? | |
| 23:31:52 | NobodyCam | really both | |
| 23:33:34 | melwitt | ok, for the scheduling thing, consider moving the ComputeFilter earlier in your list of configured filters if you have a large number of compute nodes. I have seen things where if there are a lot of computes (like 1000) the scheduling process is so slow that some nodes are considered "down" by the ComputeFilter because by the time that filter is reached, 60s have elapsed | |
| 23:34:10 | melwitt | for the other settings, "service_down_time report_interval should be less than service_down_time. If service_down_time is less than report_interval, services will routinely be considered down, because they report in too rarely" from https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.report_interval is the main thing you need to make sure you have right | |
| 23:36:01 | melwitt | here's another setting to make sure is set harmoniously with service_down_time https://docs.openstack.org/nova/latest/configuration/config.html#scheduler.periodic_task_interval | |
| 23:37:44 | melwitt | I think those 3 settings are the only ones you need to adjust and they need to be set as recommended in that doc, that report_interval needs to be less than service_down_time and periodic_task_interval needs to also be less than service_down_time | |
| 23:44:26 | melwitt | AFAIK the driving factor for choosing service_down_time will be how long scheduling is taking for you. if you move ComputeFilter first in the list, for example, and still get sporadic NoValidHost bc of compute node "down" when it's not really down, then you will need to increase service_down_time to accommodate the scheduling time. just make sure you don't set report_interval or periodic_task_interval to longer than service_down_time | |
| 23:54:17 | NobodyCam | Thank you very much melwitt I will review our current configuration an let you know if it improves after any changes | |
| 23:54:55 | melwitt | NobodyCam: np, good luck, will be interested to hear how it goes | |
| #openstack-nova - 2020-11-20 | |||
| 00:09:15 | sean-k-mooney | NobodyCam: you might also want to make sure that you have ntpd/chronny running and the clocks are in sync | |
| 00:09:49 | sean-k-mooney | NobodyCam:if the clocks drift it can cause the servics to flap like that | |
| 00:15:23 | NobodyCam | Oh good point, we should nap going, but I have not checked it | |
| 00:15:35 | NobodyCam | s/nap/ntp | |
| 00:24:36 | openstackgerrit | Brin Zhang proposed openstack/nova-specs master: Remove tenant_id https://review.opendev.org/737241 | |
| 00:25:51 | brinzhang_ | gmann: addressed your comment in remove tenant_id specs, thanks | |
| 00:28:15 | openstackgerrit | Brin Zhang proposed openstack/nova master: [Trivial] Rename host/node to hostname/nodename in conductor manager https://review.opendev.org/762499 | |
| 00:51:24 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Reuse code from oslo lib for JSON policy migration https://review.opendev.org/763487 | |
| 01:39:00 | openstackgerrit | wu.shiming proposed openstack/os-vif master: Fix hacking min version to 3.0.1 https://review.opendev.org/755168 | |
| 03:17:08 | brinzhang | sean-k-mooney: I have a question in https://review.opendev.org/#/c/737241/5/specs/wallaby/approved/remove-tenant-id.rst@62, hope you can reply, thanks | |
| 07:16:47 | openstackgerrit | zhufl proposed openstack/nova master: Fix invalid argument formatting in exception messages https://review.opendev.org/763511 | |
| 08:12:43 | bauzas | good morning Nova | |
| 08:31:35 | gibi | good morning | |
| 08:35:22 | bauzas | elod: gibi: I guess we can move to Fix Released for the Victoria series on https://bugs.launchpad.net/nova/+bug/1902925, right? | |
| 08:35:23 | openstack | Launchpad bug 1902925 in OpenStack Compute (nova) victoria "Upgrades to compute RPC API 5.12 are broken" [Critical,Fix committed] - Assigned to Sylvain Bauza (sylvain-bauza) | |
| 08:35:30 | bauzas | as we provided a new stable release | |
| 08:36:24 | gibi | bauzas: yes it is released. Should this bug state transition automated somehow? | |
| 08:42:40 | elod | yes, it's definitely released :) and usually it's automatically changed to released at launchpad :( but only "usually" :( | |
| 08:43:20 | elod | I don't know where is the automation behind, though | |
| 08:44:27 | elod | (now I've changed it manually) | |
| 08:57:48 | bauzas | gibi: I don't think so for LP bugs that are targeted for a stable series | |
| 08:58:05 | bauzas | it's done automatically for master changes of course | |
| 08:59:12 | gibi | bauzas: so the at each stable release somebody has to go through an flip the released bugs manually | |
| 08:59:30 | gibi | elod, lyarwood: did you know that and do that? | |
| 09:06:44 | elod | as far as I knew it was automatically for stable branches, too. But maybe I was wrong. :) and I only moved the state in rare cases in the past. | |
| 09:07:01 | elod | :/ | |
| 09:09:07 | lyarwood | yeah I also thought this was automatic | |
| 09:31:56 | bauzas | we can ask reldel | |
| 09:40:40 | QianbiaoNG | hello anyone knows what IPA means in this project https://opendev.org/x/novajoin | |
| 09:41:25 | QianbiaoNG | And is there an official project for implement a nova vendor metadata service | |
| 09:47:21 | swp20 | bauzas, sean-k-mooney: | |
| 09:48:49 | swp20 | hi guys, i meet a question, when i use 'virsh attach-device gpu.xml domain --persistent --live' to attach device to the running guest, | |
| 09:49:58 | swp20 | and then detach the device with 'virsh detach-device domain gpu.xml --persistent --live' to detach device to the guest, | |
| 09:51:41 | swp20 | even through these actions are successful, and again attach-device to the guest, libvirt raise 'pci device in use by driver qemu'. | |
| 09:53:34 | swp20 | do you meet this problem, and any suggestion? | |
| 10:00:38 | kashyap | QianbiaoNG: I think it might be the https://freeipa.org project ("Identity, Policy, Audit"). I haven't clicked on your URL | |
| 10:01:31 | QianbiaoNG | kashyap ty, seems so. | |
| 10:01:47 | kashyap | QianbiaoNG: Actually, I'm sure it is talking about the FreeIPA server instances. | |
| 10:02:09 | QianbiaoNG | another question: is there an official project provided for implement a nova vendor data metadata service. | |
| 10:03:29 | kashyap | swp20: No answer top off my head, but if you're not using Nova, and directly `virsh`, you can get better help on #virt (OFTC network) | |
| 10:03:34 | kashyap | s/can/might/ | |
| 10:08:19 | swp20 | kashyap: thanks, i joined and no person there in virt(OFTC network). | |
| 10:09:02 | kashyap | swp20: I think you're still on Freenode. I see more than 100 people :) | |
| 10:10:11 | kashyap | swp20: You need to use the '/connect' command to join the OFTC server, and then '/join #virt' | |
| 11:33:20 | admin0 | clue of ceph2 .. so it only has ceph details of the first ceph1 cluster only .. | |
| 11:33:20 | admin0 | good morning .. i have nova, cinder and glance setup using ceph1 .. works good.. now I added ceph2 for only cinder ( so that cinder has 2 glance backends) one type ssd and one type hdd .. i am able to create the volumes in both hdd and sdd.. i can mount sdd fine .. but when I try to mount hdd, i get: libvirt.libvirtError: internal error: unable to execute QEMU command 'blockdev-add': error connecting: Permission denied" .. the hypervisor has no | |
| 11:33:53 | admin0 | so question is, what config changes i need to do or what needs to be done so that i can mount the volume from the 2nd ceph to the instance | |
| 11:45:56 | openstackgerrit | Lee Yarwood proposed openstack/nova-specs master: Image and flavor defined ephemeral storage encryption https://review.opendev.org/752284 | |
| 11:51:51 | sean-k-mooney | admin0: its likely an issue with your cephx keys and the pool permeions for the nova user on the ceph clusters | |
| 11:52:13 | openstackgerrit | Stephen Finucane proposed openstack/nova-specs master: Update modernize-os-hypervisors-api spec https://review.opendev.org/763043 | |
| 11:55:54 | stephenfin | bauzas: I'm happy with https://review.opendev.org/#/c/759828/ now, but I'm holding off on +2 to give you a chance to revisit it if you want | |
| 12:10:30 | admin0 | sean-k-mooney, so hypervisor does not need to have the ceph2 keys or settings ? | |
| 12:10:36 | admin0 | it offloads to cinder ? | |
| 12:11:09 | admin0 | sean-k-mooney, so this means ceph2 also needs a nova user ( which is used to mount ) | |
| 12:15:23 | sean-k-mooney | i think it has to have both but lyarwood might be better able to respond | |
| 12:15:33 | sean-k-mooney | its been a while since i deploy ceph | |
| 12:23:23 | lyarwood | admin0: I don't think we support multiple clusters in Nova | |
| 12:23:38 | sean-k-mooney | lyarwood: we dont for rbd | |
| 12:23:47 | sean-k-mooney | lyarwood: bute we should via cinder | |
| 12:23:59 | sean-k-mooney | *rbd images backend | |
| 12:24:20 | lyarwood | sean-k-mooney: the user is still configured locally on the computes for volumes | |
| 12:24:43 | sean-k-mooney | right dont you have to configre the same user on all ceph clusters | |
| 12:24:57 | sean-k-mooney | with the same cephx key | |
| 12:24:58 | lyarwood | sean-k-mooney: with the same key | |
| 12:25:04 | lyarwood | and I'm not sure you can do that can you? | |
| 12:25:15 | lyarwood | I thought the keys were unique to a cluster | |
| 12:25:24 | lyarwood | but that might be a lack of knownledge on my side | |
| 12:25:39 | sean-k-mooney | im not sure i have only done it with one ceph personally | |
| 12:26:00 | sean-k-mooney | but i know people do dpleoy multiple ceph clusters | |
| 12:26:11 | sean-k-mooney | but im not sure if that is 1 per AZ or something like that | |
| 12:26:38 | sean-k-mooney | https://ceph.io/geen-categorie/openstack-nova-configure-multiple-ceph-backends-on-one-hypervisor/ | |
| 12:26:42 | lyarwood | yeah, I think the normal way to do with admin0 has done would be to use different pools | |
| 12:27:33 | sean-k-mooney | how the hell does ^ work | |
| 12:27:38 | lyarwood | yeah that's single cluster multiple pools | |
| 12:27:53 | lyarwood | and aggregates to keep them apart | |
| 12:28:17 | sean-k-mooney | no i mesn once nova compute with 2 files witht he same keys and different values | |
| 12:28:25 | sean-k-mooney | nova-compute --config-file /etc/nova/nova-ceph1.conf --config-file /etc/nova/nova-ceph2.conf | |
| 12:29:03 | lyarwood | I think that's a mistake in the write up tbh | |
| 12:29:12 | lyarwood | given what they are trying to do | |
| 12:30:08 | sean-k-mooney | they are trying to run two nova computes on the same hyperviour with different host values but they same hypervior hostname | |
| 12:31:03 | lyarwood | ew | |
| 12:31:43 | sean-k-mooney | but 1 if that comand even works it really should not and 2 that si really going to screw up placemnt | |
| 12:32:17 | sean-k-mooney | admin0: so dont do what in that ceph guide. | |
| 12:34:13 | openstackgerrit | Lucian Petrut proposed openstack/nova master: hyper-v rbd volume support https://review.opendev.org/763550 | |
| 12:34:17 | sean-k-mooney | ah found an actully sane example | |
| 12:34:20 | sean-k-mooney | https://medium.com/walmartglobaltech/deploying-cinder-with-multiple-ceph-cluster-backends-2cd90d64b10 | |
| 12:34:26 | sean-k-mooney | lyarwood: admin0 ^ | |
| 12:34:36 | sean-k-mooney | so the user has to be different on each cluster | |
| 12:35:56 | sean-k-mooney | actully no that is still a hack | |