Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-12-03
23:58:32 eandersson If we can’t deliver to the queue I think we should just set to error or move to the next available compute
23:58:52 sean-k-mooney eandersson: set what to error?
23:58:57 eandersson The instance
23:59:12 sean-k-mooney that would not be correct in all cases
23:59:54 sean-k-mooney if the call was to say the instace diagnostics endpoint
#openstack-nova - 2019-12-04
00:00:13 sean-k-mooney we shoudl not put the instance in error state because we were not able to get the diagnositcs right
00:00:17 eandersson sure this would only be for the scheduling event
00:00:29 eandersson since mandatory is per message not per queue
00:01:00 sean-k-mooney its per type of rpc call we do
00:01:15 sean-k-mooney e.g. you set it in your argmuent to send
00:02:59 sean-k-mooney if we pass mandaroy on all topic sends then we woudl ahve to deally with it constently for ever api action that needs to call the compute node
00:03:29 sean-k-mooney if we dont set it gloably for all topic sends we need to consdider how to handel the failure specific for each action
00:06:30 sean-k-mooney eandersson: in the specifc case of server create. if the downcall form the condutor to the compute failed to send you would like us to retry the next compute node correct.
00:07:04 sean-k-mooney if we set the instance to error at that point we would not retry
00:09:27 sean-k-mooney if we did retry i think we would still want to incremetn the failed boot counter for the compute node so the BuildFailureWeigher can avoid that host and log an error at a minium
00:09:59 eandersson Sorry stepping away again. Btw we got a ton of open positions if anyone wants to come and help me work on this stuff. :p
00:10:38 sean-k-mooney no worries im about to call it a night anyway
01:45:01 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Pass the target in os-services APIs policy https://review.opendev.org/676688
01:45:45 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Add test coverage of existing admin_actions policies https://review.opendev.org/657698
02:22:14 openstackgerrit Boxiang Zhu proposed openstack/nova master: Make evacuation respects anti-affinity rule https://review.opendev.org/649963
02:27:12 eandersson Restart rmq and rebuilt it. Lets see if the issue goes away.
02:29:04 openstackgerrit Boxiang Zhu proposed openstack/nova master: Fix live migration break group policy simultaneously https://review.opendev.org/651969
03:04:33 openstackgerrit Brin Zhang proposed openstack/nova master: PoC: Support re-configure the delete_on_termination in server https://review.opendev.org/693828
03:24:17 eandersson sean-k-mooney yea I would prefer it to retry and write an error to the logs, but probably cleaner and easier to implement it to stop and put the instance into an ERROR'd state instead.
03:25:03 eandersson I am gonna see if I can set up something like Terraform to better handle instances stuck in BUILDING
03:25:28 eandersson I think out of the box it gets stuck in a weird state, but I am sure there is a way to have Terraform handle issues like this more gracefully.
03:59:10 openstackgerrit jichenjc proposed openstack/nova master: libvirt: avoid cpu check at s390x arch https://review.opendev.org/696228
05:05:47 openstackgerrit jichenjc proposed openstack/nova master: downgrade when host does not support capabilities https://review.opendev.org/697228
05:47:26 openstackgerrit Brin Zhang proposed openstack/nova-specs master: Support re-configure deleted_on_termination in server https://review.opendev.org/580336
06:46:55 openstackgerrit OpenStack Proposal Bot proposed openstack/nova master: Imported Translations from Zanata https://review.opendev.org/694717
07:37:35 openstackgerrit jichenjc proposed openstack/nova master: libvirt: avoid cpu check at s390x arch https://review.opendev.org/696228
07:56:29 openstackgerrit Luyao Zhong proposed openstack/nova master: support live migration with vpmems https://review.opendev.org/687856
09:08:52 openstackgerrit Sundar Nadathur proposed openstack/nova master: ksa auth conf and client for Cyborg access https://review.opendev.org/631242
09:08:52 openstackgerrit Sundar Nadathur proposed openstack/nova master: Add Cyborg device profile groups to request spec. https://review.opendev.org/631243
09:08:53 openstackgerrit Sundar Nadathur proposed openstack/nova master: Define Cyborg ARQ binding notification event. https://review.opendev.org/692707
09:08:53 openstackgerrit Sundar Nadathur proposed openstack/nova master: Create and bind Cyborg ARQs. https://review.opendev.org/631244
09:08:54 openstackgerrit Sundar Nadathur proposed openstack/nova master: Compose accelerator PCI devices into VM's domain XML. https://review.opendev.org/631245
09:08:54 openstackgerrit Sundar Nadathur proposed openstack/nova master: Delete ARQs for an instance when the instance is deleted. https://review.opendev.org/673735
09:08:55 openstackgerrit Sundar Nadathur proposed openstack/nova master: Add cyborg tempest job. https://review.opendev.org/670999
09:57:38 openstackgerrit Stephen Finucane proposed openstack/nova master: nova-net: Remove remaining nova-network quotas https://review.opendev.org/697152
09:57:38 openstackgerrit Stephen Finucane proposed openstack/nova master: nova-net: Remove use of legacy 'FloatingIP' object https://review.opendev.org/697153
09:57:39 openstackgerrit Stephen Finucane proposed openstack/nova master: nova-net: Remove use of legacy 'Network' object https://review.opendev.org/697154
09:57:39 openstackgerrit Stephen Finucane proposed openstack/nova master: nova-net: Remove use of legacy 'SecurityGroup' object https://review.opendev.org/697155
09:57:40 openstackgerrit Stephen Finucane proposed openstack/nova master: nova-net: Remove unused nova-network objects https://review.opendev.org/697156
09:57:40 openstackgerrit Stephen Finucane proposed openstack/nova master: nova-net: Follow-up for I12a96ea659ed402cc4d1bd52a50e2e16042b6372 https://review.opendev.org/697252
09:59:10 stephenfin bauzas: Care to get some of that nova-net removal goodness in ya this morning? https://review.opendev.org/#/c/696503/
09:59:21 bauzas stephenfin: I could
10:36:49 openstackgerrit Lee Yarwood proposed openstack/nova stable/rocky: block_device: Copy original volume_type when missing for snapshot based volumes https://review.opendev.org/697260
10:37:12 openstackgerrit Lee Yarwood proposed openstack/nova stable/queens: block_device: Copy original volume_type when missing for snapshot based volumes https://review.opendev.org/697261
11:34:58 openstackgerrit Thierry Carrez proposed openstack/nova master: Remove unused rootwrap filters https://review.opendev.org/697134
12:34:38 gibi gmann: if you have time, this tempest fix would help with some of the tempest failures in CI https://review.opendev.org/#/c/682964
12:42:59 efried o/ gibi stephenfin, lend me your ears for a moment?
12:43:38 gibi efried: I'm here but expect me to dissapeare randomly as I'm at the doctor's waiting room
12:44:25 lyarwood efried: I also think stephenfin is AFK at lunch at the moment btw
12:46:16 efried okay, thanks, I'll just leave this here then:
12:46:16 efried I'm working on this virtual TPM thing, trying to get the spec approved.
12:46:16 efried https://review.opendev.org/#/c/686804/
12:46:16 efried Dan helped me get it to this point, but doesn't like some of the remaining oddness around snapshot-related operations.
12:46:16 efried However, he has implied that he won't stand in the way if others can get behind it.
12:46:17 efried So... gibi and stephenfin, can you get behind it?
12:47:12 gibi efried: opened the spec. Let's see how ugly is the snapshot part
12:55:57 efried gibi: I can tldr it for you:
12:55:57 efried The vTPM belongs to the instance, not the image. So you can't do things like createImage to clone the VM and expect the vTPM to be cloned. (IMO this is sensible. Don't copy my security stuff around, please.)
12:57:03 gibi yeah that seems reasonable. Do not leak secrete data between clones
12:57:38 efried Similarly, if you do a rebuild, you keep your vTPM; you don't get whatever came from the new image.
12:58:14 efried Whatever's in the image might be expecting a given vTPM state that won't exist, but that's up to you to get right.
12:58:25 efried In particular, we *do* save the vTPM state when you do createBackup
12:59:48 efried So if you rebuild from that backup, we'll restore the vTPM as it existed at the point when you did the backup, which ought to maximize the chance that it's in the state the VM is expecting.
13:01:15 gibi Can backups be used to clone a VM ?
13:01:56 efried theoretically, yes, they do almost the same thing as createImage.
13:02:22 efried Documentation would be the key here: If you're backing up, use createBackup; if you're cloning, use createImage.
13:02:35 efried Which is what ops should be doing anyway, but I'm sure those get abused frequently today.
13:03:44 efried I think createBackup tries to maintain one image at a time, deleting the previous one any time you create a new one. Otherwise it's not different than createImage.
13:03:55 gibi I see
13:05:33 efried So the other two main quirks of concern are:
13:05:33 efried - Evacuate doesn't restore your vTPM, sorry.
13:05:33 efried - You have to do stuff as the instance owner. (IMO this is also good: my vTPM is secure stuff, I don't even want the admin to have access to it.)
13:06:35 gibi doesn't shared disk + evacuate could keep the vTMP file? or the vTPM file is somewhere far from the instance disk?
13:06:43 efried the latter :(
13:07:03 efried the vTPM is maintained by libvirt
13:07:10 efried not in the instance
13:07:13 gibi OK so it is some obscure location that the deployer won't share between compute hosts
13:07:17 efried well
13:07:36 gibi anyhow this could be a future improvement in libvirt + nova
13:07:39 efried the location is well known (though I'm not sure there's a contract that it won't change at some point)
13:07:51 efried theoretically it may be possible for that directory to be shared
13:07:55 gibi ack
13:08:13 efried but that would be up to the deployer to figure out, and I'm not going to do anything in nova to try to support (or prevent) it.
13:08:20 gibi sure
13:08:23 gibi make sense
13:08:53 gibi when you say "You have to do stuff as the instance owner" you mean nova needs to act in behalf of the instance owner to manage the vTPM or what is the stuff there?
13:09:59 efried correct
13:10:39 efried Nova needs to unlock the vTPM by defining a secret to libvirt when the guest is created
13:10:57 efried and, because security, the secret is owned by the instance owner; even the admin can't get at it.
13:11:45 gibi does it mean that the admin cannot delet the VM either?
13:11:51 efried Probably the most awkward of these areas is if you reboot the hypervisor, your VMs won't boot until the owner of the VM goes to boot them.
13:12:01 efried Hm, interesting question.
13:12:21 efried The admin won't be able to delete the secret in barbican, or the backup in swift
13:12:30 efried but we don't have to make those error conditions.
13:12:50 efried So the instance could still be destroyed, but those artifacts would be leaked in that case and the owner would have to go clean them up manually.

Earlier   Later