| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-11-19 | |||
| 17:03:23 | dansmith | well, I think we should fix wsgi to let you specify config files like everything else, and then let you guys decide how you want to template and split the config files to make the packages work (or not) | |
| 17:06:42 | owalsh | ack, might also be worth looking at how we could parameterize the conf file path in the dev/rpm services | |
| 17:08:33 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: DNM: extra logging for unpin_cpus https://review.opendev.org/763409 | |
| 17:08:59 | dansmith | even just having /etc/nova.conf that everything reads, and then each service configured to load /etc/nova/nova-conductor.conf, nova-scheduler.conf, etc would maybe be useful... common config and per-service config.. the latter could be empty in a lot of cases | |
| 17:13:39 | owalsh | ack, I think that was the direction this was going but the wsgi issue was a blocker | |
| 17:14:28 | dansmith | yeah | |
| 17:14:31 | openstackgerrit | Dan Smith proposed openstack/nova master: RFC: Update the install documents for cells and less distro specific https://review.opendev.org/763412 | |
| 17:33:31 | openstackgerrit | Lee Yarwood proposed openstack/nova-specs master: Image and flavor defined ephemeral storage encryption https://review.opendev.org/752284 | |
| 18:51:25 | sean-k-mooney | stephenfin: i dont know if you saw the discussion about the metadata serivce but we have some undocumented behavior realated to an old nova v2 api extention that got merged into the service when we remove the nova api v3 code and support for cell in newton | |
| 18:51:59 | sean-k-mooney | stephenfin: specifically form within the guest without creds you can do a post to the password filed and update that filed | |
| 18:53:13 | sean-k-mooney | stephenfin: so it looks like we need to update the documatnion for that and consider if we actully want to support that longterem i think we need to at least for now since cloudbase use it | |
| 18:54:27 | sean-k-mooney | that said it was never actully accpeted into the v2.x api it was incorrectly included when we killed extention and drop the nova v3 api | |
| 18:59:07 | sean-k-mooney | stephenfin: https://github.com/openstack/nova/commit/a2101c4e7017715af0a29675b89e14ee2884bd89 is the change that allowed the password to be updated once via the 169 adress | |
| 19:00:10 | sean-k-mooney | which built on the server_password api extntion | |
| 19:00:45 | sean-k-mooney | anyway none of our docs say ^ is a thing you can do. | |
| 23:10:26 | NobodyCam | Good Afternoon Nova folks, I've started seeing hypervisor state flapping between up and down, The hypervisors are up, I've read that adjusting server_down_time and report_interval can help with this, are these the correct setting to adjust and is there a way to gauge what these values should be? | |
| 23:31:34 | melwitt | NobodyCam: are you experiencing a problem with getting occasional NoValidHost because of this or seeing the hypervisor state going up and down or both? | |
| 23:31:52 | NobodyCam | really both | |
| 23:33:34 | melwitt | ok, for the scheduling thing, consider moving the ComputeFilter earlier in your list of configured filters if you have a large number of compute nodes. I have seen things where if there are a lot of computes (like 1000) the scheduling process is so slow that some nodes are considered "down" by the ComputeFilter because by the time that filter is reached, 60s have elapsed | |
| 23:34:10 | melwitt | for the other settings, "service_down_time report_interval should be less than service_down_time. If service_down_time is less than report_interval, services will routinely be considered down, because they report in too rarely" from https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.report_interval is the main thing you need to make sure you have right | |
| 23:36:01 | melwitt | here's another setting to make sure is set harmoniously with service_down_time https://docs.openstack.org/nova/latest/configuration/config.html#scheduler.periodic_task_interval | |
| 23:37:44 | melwitt | I think those 3 settings are the only ones you need to adjust and they need to be set as recommended in that doc, that report_interval needs to be less than service_down_time and periodic_task_interval needs to also be less than service_down_time | |
| 23:44:26 | melwitt | AFAIK the driving factor for choosing service_down_time will be how long scheduling is taking for you. if you move ComputeFilter first in the list, for example, and still get sporadic NoValidHost bc of compute node "down" when it's not really down, then you will need to increase service_down_time to accommodate the scheduling time. just make sure you don't set report_interval or periodic_task_interval to longer than service_down_time | |
| 23:54:17 | NobodyCam | Thank you very much melwitt I will review our current configuration an let you know if it improves after any changes | |
| 23:54:55 | melwitt | NobodyCam: np, good luck, will be interested to hear how it goes | |
| #openstack-nova - 2020-11-20 | |||
| 00:09:15 | sean-k-mooney | NobodyCam: you might also want to make sure that you have ntpd/chronny running and the clocks are in sync | |
| 00:09:49 | sean-k-mooney | NobodyCam:if the clocks drift it can cause the servics to flap like that | |
| 00:15:23 | NobodyCam | Oh good point, we should nap going, but I have not checked it | |
| 00:15:35 | NobodyCam | s/nap/ntp | |
| 00:24:36 | openstackgerrit | Brin Zhang proposed openstack/nova-specs master: Remove tenant_id https://review.opendev.org/737241 | |
| 00:25:51 | brinzhang_ | gmann: addressed your comment in remove tenant_id specs, thanks | |
| 00:28:15 | openstackgerrit | Brin Zhang proposed openstack/nova master: [Trivial] Rename host/node to hostname/nodename in conductor manager https://review.opendev.org/762499 | |
| 00:51:24 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Reuse code from oslo lib for JSON policy migration https://review.opendev.org/763487 | |
| 01:39:00 | openstackgerrit | wu.shiming proposed openstack/os-vif master: Fix hacking min version to 3.0.1 https://review.opendev.org/755168 | |
| 03:17:08 | brinzhang | sean-k-mooney: I have a question in https://review.opendev.org/#/c/737241/5/specs/wallaby/approved/remove-tenant-id.rst@62, hope you can reply, thanks | |
| 07:16:47 | openstackgerrit | zhufl proposed openstack/nova master: Fix invalid argument formatting in exception messages https://review.opendev.org/763511 | |
| 08:12:43 | bauzas | good morning Nova | |
| 08:31:35 | gibi | good morning | |
| 08:35:22 | bauzas | elod: gibi: I guess we can move to Fix Released for the Victoria series on https://bugs.launchpad.net/nova/+bug/1902925, right? | |
| 08:35:23 | openstack | Launchpad bug 1902925 in OpenStack Compute (nova) victoria "Upgrades to compute RPC API 5.12 are broken" [Critical,Fix committed] - Assigned to Sylvain Bauza (sylvain-bauza) | |
| 08:35:30 | bauzas | as we provided a new stable release | |
| 08:36:24 | gibi | bauzas: yes it is released. Should this bug state transition automated somehow? | |
| 08:42:40 | elod | yes, it's definitely released :) and usually it's automatically changed to released at launchpad :( but only "usually" :( | |
| 08:43:20 | elod | I don't know where is the automation behind, though | |
| 08:44:27 | elod | (now I've changed it manually) | |
| 08:57:48 | bauzas | gibi: I don't think so for LP bugs that are targeted for a stable series | |
| 08:58:05 | bauzas | it's done automatically for master changes of course | |
| 08:59:12 | gibi | bauzas: so the at each stable release somebody has to go through an flip the released bugs manually | |
| 08:59:30 | gibi | elod, lyarwood: did you know that and do that? | |
| 09:06:44 | elod | as far as I knew it was automatically for stable branches, too. But maybe I was wrong. :) and I only moved the state in rare cases in the past. | |
| 09:07:01 | elod | :/ | |
| 09:09:07 | lyarwood | yeah I also thought this was automatic | |
| 09:31:56 | bauzas | we can ask reldel | |
| 09:40:40 | QianbiaoNG | hello anyone knows what IPA means in this project https://opendev.org/x/novajoin | |
| 09:41:25 | QianbiaoNG | And is there an official project for implement a nova vendor metadata service | |
| 09:47:21 | swp20 | bauzas, sean-k-mooney: | |
| 09:48:49 | swp20 | hi guys, i meet a question, when i use 'virsh attach-device gpu.xml domain --persistent --live' to attach device to the running guest, | |
| 09:49:58 | swp20 | and then detach the device with 'virsh detach-device domain gpu.xml --persistent --live' to detach device to the guest, | |
| 09:51:41 | swp20 | even through these actions are successful, and again attach-device to the guest, libvirt raise 'pci device in use by driver qemu'. | |
| 09:53:34 | swp20 | do you meet this problem, and any suggestion? | |
| 10:00:38 | kashyap | QianbiaoNG: I think it might be the https://freeipa.org project ("Identity, Policy, Audit"). I haven't clicked on your URL | |
| 10:01:31 | QianbiaoNG | kashyap ty, seems so. | |
| 10:01:47 | kashyap | QianbiaoNG: Actually, I'm sure it is talking about the FreeIPA server instances. | |
| 10:02:09 | QianbiaoNG | another question: is there an official project provided for implement a nova vendor data metadata service. | |
| 10:03:29 | kashyap | swp20: No answer top off my head, but if you're not using Nova, and directly `virsh`, you can get better help on #virt (OFTC network) | |
| 10:03:34 | kashyap | s/can/might/ | |
| 10:08:19 | swp20 | kashyap: thanks, i joined and no person there in virt(OFTC network). | |
| 10:09:02 | kashyap | swp20: I think you're still on Freenode. I see more than 100 people :) | |
| 10:10:11 | kashyap | swp20: You need to use the '/connect' command to join the OFTC server, and then '/join #virt' | |
| 11:33:20 | admin0 | clue of ceph2 .. so it only has ceph details of the first ceph1 cluster only .. | |
| 11:33:20 | admin0 | good morning .. i have nova, cinder and glance setup using ceph1 .. works good.. now I added ceph2 for only cinder ( so that cinder has 2 glance backends) one type ssd and one type hdd .. i am able to create the volumes in both hdd and sdd.. i can mount sdd fine .. but when I try to mount hdd, i get: libvirt.libvirtError: internal error: unable to execute QEMU command 'blockdev-add': error connecting: Permission denied" .. the hypervisor has no | |
| 11:33:53 | admin0 | so question is, what config changes i need to do or what needs to be done so that i can mount the volume from the 2nd ceph to the instance | |
| 11:45:56 | openstackgerrit | Lee Yarwood proposed openstack/nova-specs master: Image and flavor defined ephemeral storage encryption https://review.opendev.org/752284 | |
| 11:51:51 | sean-k-mooney | admin0: its likely an issue with your cephx keys and the pool permeions for the nova user on the ceph clusters | |
| 11:52:13 | openstackgerrit | Stephen Finucane proposed openstack/nova-specs master: Update modernize-os-hypervisors-api spec https://review.opendev.org/763043 | |
| 11:55:54 | stephenfin | bauzas: I'm happy with https://review.opendev.org/#/c/759828/ now, but I'm holding off on +2 to give you a chance to revisit it if you want | |
| 12:10:30 | admin0 | sean-k-mooney, so hypervisor does not need to have the ceph2 keys or settings ? | |
| 12:10:36 | admin0 | it offloads to cinder ? | |
| 12:11:09 | admin0 | sean-k-mooney, so this means ceph2 also needs a nova user ( which is used to mount ) | |
| 12:15:23 | sean-k-mooney | i think it has to have both but lyarwood might be better able to respond | |
| 12:15:33 | sean-k-mooney | its been a while since i deploy ceph | |
| 12:23:23 | lyarwood | admin0: I don't think we support multiple clusters in Nova | |
| 12:23:38 | sean-k-mooney | lyarwood: we dont for rbd | |
| 12:23:47 | sean-k-mooney | lyarwood: bute we should via cinder | |
| 12:23:59 | sean-k-mooney | *rbd images backend | |
| 12:24:20 | lyarwood | sean-k-mooney: the user is still configured locally on the computes for volumes | |
| 12:24:43 | sean-k-mooney | right dont you have to configre the same user on all ceph clusters | |
| 12:24:57 | sean-k-mooney | with the same cephx key | |
| 12:24:58 | lyarwood | sean-k-mooney: with the same key | |
| 12:25:04 | lyarwood | and I'm not sure you can do that can you? | |
| 12:25:15 | lyarwood | I thought the keys were unique to a cluster | |
| 12:25:24 | lyarwood | but that might be a lack of knownledge on my side | |
| 12:25:39 | sean-k-mooney | im not sure i have only done it with one ceph personally | |
| 12:26:00 | sean-k-mooney | but i know people do dpleoy multiple ceph clusters | |
| 12:26:11 | sean-k-mooney | but im not sure if that is 1 per AZ or something like that | |
| 12:26:38 | sean-k-mooney | https://ceph.io/geen-categorie/openstack-nova-configure-multiple-ceph-backends-on-one-hypervisor/ | |
| 12:26:42 | lyarwood | yeah, I think the normal way to do with admin0 has done would be to use different pools | |
| 12:27:33 | sean-k-mooney | how the hell does ^ work | |
| 12:27:38 | lyarwood | yeah that's single cluster multiple pools | |
| 12:27:53 | lyarwood | and aggregates to keep them apart | |