| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-12-03 | |||
| 23:05:09 | sean-k-mooney | shoudl we jsut retry? | |
| 23:06:30 | sean-k-mooney | i actully have no idea what the best path forward would be in that case but we kind of need to figure that out before we can fix the issue | |
| 23:06:48 | sean-k-mooney | we now have the feature that allows use to adress it however | |
| 23:11:24 | openstack | Launchpad bug 1661510 in oslo.messaging "topic_send may loss messages if the queue not exists" [Medium,In progress] - Assigned to Gabriele Santomaggio (gsantomaggio) | |
| 23:11:24 | sean-k-mooney | melwitt: did you ever have time to talk to the oslo folks on how we could use this https://blueprints.launchpad.net/oslo.messaging/+spec/transport-options and specificly the mandaroty flag so we can adress the nova aspect of https://bugs.launchpad.net/oslo.messaging/+bug/1661510 | |
| 23:12:05 | sean-k-mooney | i know i havent but it looks like all the patchs are in place on the oslo side | |
| 23:28:23 | melwitt | sean-k-mooney: I haven't talked to them bc like you said, their side is done. it's up to us (me) to figure out how to use it on the nova side and I haven't gotten a chance to dig in to it yet | |
| 23:28:54 | melwitt | I don't yet know where/how to pass it in the nova code | |
| 23:29:51 | sean-k-mooney | ya i just asked on the bug if they could provdie an example | |
| 23:30:51 | sean-k-mooney | i know its something we shoudl be setting when we do a topic send which is i guess a call to a compute node primarly | |
| 23:31:11 | sean-k-mooney | but i dont know what we should do if we are not able to deliver it to the message queue | |
| 23:40:40 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Add new default roles in os-services API policies https://review.opendev.org/648480 | |
| 23:44:56 | gmann | johnthetubaguy: this should be ready now. I was checking why project-admin and legacy admin can access the new system reader rule. which was correct because of old defaults(admin_api) are deprecated and still work. This make sure any old token will keep working for changed defaults also. | |
| 23:45:29 | gmann | johnthetubaguy: once you are ok with this then it will complete the first set of change - https://review.opendev.org/648480 | |
| 23:51:23 | openstackgerrit | sean mooney proposed openstack/nova master: Block rebuild when NUMA topology changed https://review.opendev.org/687957 | |
| 23:51:24 | openstackgerrit | sean mooney proposed openstack/nova master: Disable NUMATopologyFilter on rebuild https://review.opendev.org/689861 | |
| 23:56:08 | efried | https://bugs.launchpad.net/neutron/+bug/1855015 | |
| 23:56:09 | openstack | Launchpad bug 1855015 in OpenStack Compute (nova) "Intermittent fails since 11/23 with "Multiple possible networks found, use a Network ID to be more specific."" [Undecided,New] | |
| 23:56:57 | sean-k-mooney | has that reappared | |
| 23:57:37 | sean-k-mooney | that is a tempset bug and or we have not confgiured tempet correctly | |
| 23:58:32 | eandersson | If we can’t deliver to the queue I think we should just set to error or move to the next available compute | |
| 23:58:52 | sean-k-mooney | eandersson: set what to error? | |
| 23:58:57 | eandersson | The instance | |
| 23:59:12 | sean-k-mooney | that would not be correct in all cases | |
| 23:59:54 | sean-k-mooney | if the call was to say the instace diagnostics endpoint | |
| #openstack-nova - 2019-12-04 | |||
| 00:00:13 | sean-k-mooney | we shoudl not put the instance in error state because we were not able to get the diagnositcs right | |
| 00:00:17 | eandersson | sure this would only be for the scheduling event | |
| 00:00:29 | eandersson | since mandatory is per message not per queue | |
| 00:01:00 | sean-k-mooney | its per type of rpc call we do | |
| 00:01:15 | sean-k-mooney | e.g. you set it in your argmuent to send | |
| 00:02:59 | sean-k-mooney | if we pass mandaroy on all topic sends then we woudl ahve to deally with it constently for ever api action that needs to call the compute node | |
| 00:03:29 | sean-k-mooney | if we dont set it gloably for all topic sends we need to consdider how to handel the failure specific for each action | |
| 00:06:30 | sean-k-mooney | eandersson: in the specifc case of server create. if the downcall form the condutor to the compute failed to send you would like us to retry the next compute node correct. | |
| 00:07:04 | sean-k-mooney | if we set the instance to error at that point we would not retry | |
| 00:09:27 | sean-k-mooney | if we did retry i think we would still want to incremetn the failed boot counter for the compute node so the BuildFailureWeigher can avoid that host and log an error at a minium | |
| 00:09:59 | eandersson | Sorry stepping away again. Btw we got a ton of open positions if anyone wants to come and help me work on this stuff. :p | |
| 00:10:38 | sean-k-mooney | no worries im about to call it a night anyway | |
| 01:45:01 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Pass the target in os-services APIs policy https://review.opendev.org/676688 | |
| 01:45:45 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Add test coverage of existing admin_actions policies https://review.opendev.org/657698 | |
| 02:22:14 | openstackgerrit | Boxiang Zhu proposed openstack/nova master: Make evacuation respects anti-affinity rule https://review.opendev.org/649963 | |
| 02:27:12 | eandersson | Restart rmq and rebuilt it. Lets see if the issue goes away. | |
| 02:29:04 | openstackgerrit | Boxiang Zhu proposed openstack/nova master: Fix live migration break group policy simultaneously https://review.opendev.org/651969 | |
| 03:04:33 | openstackgerrit | Brin Zhang proposed openstack/nova master: PoC: Support re-configure the delete_on_termination in server https://review.opendev.org/693828 | |
| 03:24:17 | eandersson | sean-k-mooney yea I would prefer it to retry and write an error to the logs, but probably cleaner and easier to implement it to stop and put the instance into an ERROR'd state instead. | |
| 03:25:03 | eandersson | I am gonna see if I can set up something like Terraform to better handle instances stuck in BUILDING | |
| 03:25:28 | eandersson | I think out of the box it gets stuck in a weird state, but I am sure there is a way to have Terraform handle issues like this more gracefully. | |
| 03:59:10 | openstackgerrit | jichenjc proposed openstack/nova master: libvirt: avoid cpu check at s390x arch https://review.opendev.org/696228 | |
| 05:05:47 | openstackgerrit | jichenjc proposed openstack/nova master: downgrade when host does not support capabilities https://review.opendev.org/697228 | |
| 05:47:26 | openstackgerrit | Brin Zhang proposed openstack/nova-specs master: Support re-configure deleted_on_termination in server https://review.opendev.org/580336 | |
| 06:46:55 | openstackgerrit | OpenStack Proposal Bot proposed openstack/nova master: Imported Translations from Zanata https://review.opendev.org/694717 | |
| 07:37:35 | openstackgerrit | jichenjc proposed openstack/nova master: libvirt: avoid cpu check at s390x arch https://review.opendev.org/696228 | |
| 07:56:29 | openstackgerrit | Luyao Zhong proposed openstack/nova master: support live migration with vpmems https://review.opendev.org/687856 | |
| 09:08:52 | openstackgerrit | Sundar Nadathur proposed openstack/nova master: Add Cyborg device profile groups to request spec. https://review.opendev.org/631243 | |
| 09:08:52 | openstackgerrit | Sundar Nadathur proposed openstack/nova master: ksa auth conf and client for Cyborg access https://review.opendev.org/631242 | |
| 09:08:53 | openstackgerrit | Sundar Nadathur proposed openstack/nova master: Create and bind Cyborg ARQs. https://review.opendev.org/631244 | |
| 09:08:53 | openstackgerrit | Sundar Nadathur proposed openstack/nova master: Define Cyborg ARQ binding notification event. https://review.opendev.org/692707 | |
| 09:08:54 | openstackgerrit | Sundar Nadathur proposed openstack/nova master: Delete ARQs for an instance when the instance is deleted. https://review.opendev.org/673735 | |
| 09:08:54 | openstackgerrit | Sundar Nadathur proposed openstack/nova master: Compose accelerator PCI devices into VM's domain XML. https://review.opendev.org/631245 | |
| 09:08:55 | openstackgerrit | Sundar Nadathur proposed openstack/nova master: Add cyborg tempest job. https://review.opendev.org/670999 | |
| 09:57:38 | openstackgerrit | Stephen Finucane proposed openstack/nova master: nova-net: Remove use of legacy 'FloatingIP' object https://review.opendev.org/697153 | |
| 09:57:38 | openstackgerrit | Stephen Finucane proposed openstack/nova master: nova-net: Remove remaining nova-network quotas https://review.opendev.org/697152 | |
| 09:57:39 | openstackgerrit | Stephen Finucane proposed openstack/nova master: nova-net: Remove use of legacy 'SecurityGroup' object https://review.opendev.org/697155 | |
| 09:57:39 | openstackgerrit | Stephen Finucane proposed openstack/nova master: nova-net: Remove use of legacy 'Network' object https://review.opendev.org/697154 | |
| 09:57:40 | openstackgerrit | Stephen Finucane proposed openstack/nova master: nova-net: Follow-up for I12a96ea659ed402cc4d1bd52a50e2e16042b6372 https://review.opendev.org/697252 | |
| 09:57:40 | openstackgerrit | Stephen Finucane proposed openstack/nova master: nova-net: Remove unused nova-network objects https://review.opendev.org/697156 | |
| 09:59:10 | stephenfin | bauzas: Care to get some of that nova-net removal goodness in ya this morning? https://review.opendev.org/#/c/696503/ | |
| 09:59:21 | bauzas | stephenfin: I could | |
| 10:36:49 | openstackgerrit | Lee Yarwood proposed openstack/nova stable/rocky: block_device: Copy original volume_type when missing for snapshot based volumes https://review.opendev.org/697260 | |
| 10:37:12 | openstackgerrit | Lee Yarwood proposed openstack/nova stable/queens: block_device: Copy original volume_type when missing for snapshot based volumes https://review.opendev.org/697261 | |
| 11:34:58 | openstackgerrit | Thierry Carrez proposed openstack/nova master: Remove unused rootwrap filters https://review.opendev.org/697134 | |
| 12:34:38 | gibi | gmann: if you have time, this tempest fix would help with some of the tempest failures in CI https://review.opendev.org/#/c/682964 | |
| 12:42:59 | efried | o/ gibi stephenfin, lend me your ears for a moment? | |
| 12:43:38 | gibi | efried: I'm here but expect me to dissapeare randomly as I'm at the doctor's waiting room | |
| 12:44:25 | lyarwood | efried: I also think stephenfin is AFK at lunch at the moment btw | |
| 12:46:16 | efried | However, he has implied that he won't stand in the way if others can get behind it. | |
| 12:46:16 | efried | Dan helped me get it to this point, but doesn't like some of the remaining oddness around snapshot-related operations. | |
| 12:46:16 | efried | https://review.opendev.org/#/c/686804/ | |
| 12:46:16 | efried | I'm working on this virtual TPM thing, trying to get the spec approved. | |
| 12:46:16 | efried | okay, thanks, I'll just leave this here then: | |
| 12:46:17 | efried | So... gibi and stephenfin, can you get behind it? | |
| 12:47:12 | gibi | efried: opened the spec. Let's see how ugly is the snapshot part | |
| 12:55:57 | efried | The vTPM belongs to the instance, not the image. So you can't do things like createImage to clone the VM and expect the vTPM to be cloned. (IMO this is sensible. Don't copy my security stuff around, please.) | |
| 12:55:57 | efried | gibi: I can tldr it for you: | |
| 12:57:03 | gibi | yeah that seems reasonable. Do not leak secrete data between clones | |
| 12:57:38 | efried | Similarly, if you do a rebuild, you keep your vTPM; you don't get whatever came from the new image. | |
| 12:58:14 | efried | Whatever's in the image might be expecting a given vTPM state that won't exist, but that's up to you to get right. | |
| 12:58:25 | efried | In particular, we *do* save the vTPM state when you do createBackup | |
| 12:59:48 | efried | So if you rebuild from that backup, we'll restore the vTPM as it existed at the point when you did the backup, which ought to maximize the chance that it's in the state the VM is expecting. | |
| 13:01:15 | gibi | Can backups be used to clone a VM ? | |
| 13:01:56 | efried | theoretically, yes, they do almost the same thing as createImage. | |
| 13:02:22 | efried | Documentation would be the key here: If you're backing up, use createBackup; if you're cloning, use createImage. | |
| 13:02:35 | efried | Which is what ops should be doing anyway, but I'm sure those get abused frequently today. | |
| 13:03:44 | efried | I think createBackup tries to maintain one image at a time, deleting the previous one any time you create a new one. Otherwise it's not different than createImage. | |
| 13:03:55 | gibi | I see | |
| 13:05:33 | efried | - You have to do stuff as the instance owner. (IMO this is also good: my vTPM is secure stuff, I don't even want the admin to have access to it.) | |
| 13:05:33 | efried | - Evacuate doesn't restore your vTPM, sorry. | |
| 13:05:33 | efried | So the other two main quirks of concern are: | |
| 13:06:35 | gibi | doesn't shared disk + evacuate could keep the vTMP file? or the vTPM file is somewhere far from the instance disk? | |
| 13:06:43 | efried | the latter :( | |
| 13:07:03 | efried | the vTPM is maintained by libvirt | |