Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-07-08
15:48:21 cdent unicorn's are a PITA
15:48:28 cdent hmm unicorns too
15:48:48 mriedem always pooping out frosting on my yard and such
15:48:59 cdent Thanks mriedem, sean-k-mooney, efried this has been very useful.
15:49:07 cdent and such
15:49:15 kashyap efried: Yes, will review
15:49:43 cdent If it turns out I can extract a little bit of 'if this virtdriver magic' i'll make it so
15:50:41 openstackgerrit Sundar Nadathur proposed openstack/nova master: ksa auth conf and client for cyborg access https://review.opendev.org/631242
15:50:42 openstackgerrit Sundar Nadathur proposed openstack/nova master: WIP: Create and bind Cyborg ARQs. https://review.opendev.org/631244
15:50:42 openstackgerrit Sundar Nadathur proposed openstack/nova master: WIP: Add Cyborg device profile groups to request spec. https://review.opendev.org/631243
15:50:43 openstackgerrit Sundar Nadathur proposed openstack/nova master: WIP: Get resolved Cyborg ARQs and add PCI BDFs to VM's domain XML. https://review.opendev.org/631245
15:53:44 mriedem gibi: looks like https://review.opendev.org/#/c/637955/31 and https://review.opendev.org/#/c/669188/2 conflict, maybe you want to rebase the fix underneath the bigger change?
15:57:09 sean-k-mooney cdent: by the way i assume there are buisness reasons that vmware would prefer not to map each compute node to a specifc data store and use cinder with volume type to express storage policies for everything else
15:57:39 cdent sean-k-mooney: yeah, the basic root of all these things is "let the DRS continue to work"
15:57:47 sean-k-mooney i know vmware have lots of fancy feature built into the hypervior and storage solution so im guessign the current approch is trying to expose them
15:58:02 sean-k-mooney right ok makes sense
15:58:31 cdent it creates a huge impedance mismatch between the way nova thinks and the way vmware thinks, but that's the way it goes
15:59:48 sean-k-mooney ya hyperv also has some limited clustering of vms and storage pools but its not as advanced as what vcenter does so the impedence miss match is less of an issue for them i guess
16:00:43 cdent there's been discussion of tying to listen to events from the DRS via some agent that would then "correct" placement when DRS makes a change
16:01:14 cdent but that would require more effort than currently available resources
16:01:45 gibi mriedem: will do
16:01:55 sean-k-mooney that might require you to reshap allocation if it moved things right if you had a shareing provider per datastore
16:02:30 sean-k-mooney although if you had just one RP and hid the datastore i guess it woul just have to modify the max unit
16:02:35 sean-k-mooney as it does today
16:03:12 sean-k-mooney cdent: also i assume you are more or less the resouces that are available to work on it
16:04:42 cdent sean-k-mooney: I consider myself already fully booked
16:05:11 sean-k-mooney cdent: yep proably over booked
16:07:09 prometheanfire which db field is the 'volumes_attached' field sourced from?
16:07:36 prometheanfire smcginnis: you might know? ^
16:08:25 mriedem prometheanfire: block_device_mappings
16:08:42 smcginnis mriedem is the expert there. ;)
16:09:19 prometheanfire mriedem: thanks, I can't reproduce it, but cinder created a volume fine, but nova was 504 on the attach (it shows up in the db but no action taken on the compute host)
16:09:31 prometheanfire restart nova services with debug/verbose and it attaches fine
16:09:51 prometheanfire since it was only half added, the api can't delete it
16:10:28 mriedem prometheanfire: meaning nova-compute asked cinder to create the volume, right?
16:10:37 mriedem bdm source_type != 'volume'
16:11:21 mriedem "the api can't delete it" - you mean the cinder api can't delete the volume? or the compute api can't delete the server?
16:11:25 prometheanfire mriedem: no, volume created manually (worked), then nova was asked to attach volume (failed)
16:11:32 prometheanfire api can't remove it from the instance
16:11:41 prometheanfire the failed part half attached it to the instance
16:12:20 prometheanfire added to block_device_mapping table but no action taken on compute host
16:12:23 openstackgerrit Merged openstack/os-vif master: Add Python 3 Train unit tests https://review.opendev.org/669438
16:12:50 mriedem so there is a bdm, and the volume shows up for the server but when you try to detach it what happens? nova gets an error from cinder?
16:12:58 mriedem saying it's already detached (not attached) or something?
16:13:15 prometheanfire nova questions cinder if it's attached, cinder says no, nova does nothing
16:14:00 mriedem as in nova doesn't delete the bdm so it stops showing up as attached to the server
16:14:03 prometheanfire does nothing -> says it's not attached, why are you asking me to detach something that's not attached (returned to user)
16:14:15 mriedem which release?
16:14:19 prometheanfire stein
16:14:50 prometheanfire I haven't been able to reproduce this after restarting nova stuff infra side
16:15:01 prometheanfire so maybe some timeout or hanging connection?
16:15:20 mriedem my guess is cinder is failing on this call: https://github.com/openstack/nova/blob/86524773b8cd3a52c98409c7ca183b4e1873e2b8/nova/compute/api.py#L4175
16:15:43 prometheanfire maybe, for the detach part
16:16:14 mriedem well what error do you get from the compute API?
16:16:26 mriedem "Invalid volume: %(reason)s"
16:16:53 prometheanfire No volume with a name or ID of '3e92f9b6-1a3b-4a7e-8487-6ff253e888db' exists.
16:17:01 prometheanfire since I removed the volumes
16:17:36 prometheanfire Invalid volume: Invalid input received: Invalid volume: Unable to detach volume. Volume status must be 'in-use' and attach_status must be 'attached' to detach. (HTTP 400) (Request-ID: req-01d8a3c5-66f4-43a4-bac5-9f1104a292fe) (HTTP 400) (Request-ID: req-358132e4-38a1-493a-b87b-d135791bae2d)
16:17:40 mriedem msg = _("Unable to detach volume. Volume status must be 'in-use' "
16:17:41 mriedem yup
16:17:41 mriedem "and attach_status must be 'attached' to detach.")
16:17:48 mriedem that's the begin_detaching call failing
16:18:05 prometheanfire because it never actually attached :D
16:18:11 openstackgerrit sean mooney proposed openstack/os-vif master: Sync Sphinx requirement https://review.opendev.org/666387
16:18:14 mriedem right i get it
16:18:41 openstackgerrit Balazs Gibizer proposed openstack/nova master: nova-manage: heal port allocations https://review.opendev.org/637955
16:18:48 gibi mriedem: rebased ^^
16:19:43 mriedem prometheanfire: interesting you wouldn't hit this in the compute service when the attach fails and delete the bdm https://github.com/openstack/nova/blob/86524773b8cd3a52c98409c7ca183b4e1873e2b8/nova/compute/manager.py#L5722
16:20:41 prometheanfire haven't been able to reproduce after restart of services, but am leaving it in debug mode in case it happens again
16:21:06 openstackgerrit Balazs Gibizer proposed openstack/nova master: Move consts from neutronv2/api to constants module https://review.opendev.org/668945
16:21:07 mriedem were there errors in the compute log after the failed attach about not being able to delete the bdm?
16:21:14 openstackgerrit Balazs Gibizer proposed openstack/nova master: Use neutron contants in cmd/manage.py https://review.opendev.org/668946
16:21:26 openstackgerrit Balazs Gibizer proposed openstack/nova master: Add 'resource_request' to neutronv2/constants https://review.opendev.org/668947
16:22:04 prometheanfire compute didn't even attempt to attach from what I could see
16:22:12 prometheanfire didn't get that far
16:22:34 mriedem so...the rpc cast from nova api to nova compute failed?
16:22:39 mriedem you said you got a 504 somewhere
16:22:59 prometheanfire 504 from nova-api to client
16:23:05 prometheanfire openstackclient
16:23:34 mriedem ok i don't know why that would happen
16:24:12 prometheanfire I'm fine waiting til I can reproduce, I've left stuff in debug
16:26:37 sean-k-mooney stephenfin: jangutter so as i was saying on the placmeent channel ... i adressed my nit in https://review.opendev.org/#/c/666387/2 if ye want to take a look at that
16:28:08 openstackgerrit Balazs Gibizer proposed openstack/nova master: Translatable output strings in heal allocation https://review.opendev.org/668925
16:33:25 sean-k-mooney aspiers: so you know that domaincap api you added for sev ... its broken
16:34:45 sean-k-mooney we cant default to q35 here https://github.com/openstack/nova/blob/master/nova/virt/libvirt/host.py#L744-L745 the logic breaks if you have emulators installed that dont support it
16:35:43 sean-k-mooney currently we dont call this code in nova yet because your sev stuff that uses it has not merged yet but i tried to use it for my device model filter stuff and it expolodes beautifully
16:38:02 efried mriedem: I'm going to bump current runways by, what say, 2 days due to last week's holiday?
16:39:28 mriedem shrug
16:39:35 mriedem not sure how much it matters
16:40:50 efried aspiers: SEV series is in merge conflict - any chance of getting that rebased soon?
16:54:01 sean-k-mooney mriedem: so quick question. https://review.opendev.org/#/c/659703/8 change the behavior of force config drive to remove the config drive after first boot
16:55:15 sean-k-mooney to me that seams like a regression as it could lead to data lose in the form of injected files if the user does not copy the injected files form the config_drive or if the operator customerises the config dirve contentce with data scripts that run on each boot
16:57:17 sean-k-mooney if you are using the dynamic vendor data and do not have teh metadta service deployed is this not an issue?
16:58:36 sean-k-mooney the injected files case might be ok if that sets instance.config_drive automaticall but i think https://review.opendev.org/#/c/659703/8/nova/virt/configdrive.py@169 might break some usecauses
16:59:44 sean-k-mooney as such im not sure how safe it is to backport this
17:02:57 mriedem "change the behavior of force config drive to remove the config drive after first boot" - you mean if you boot the server with force_config_drive=True, then change to force_config_drive=False? the bug is actually the opposite - they created a server without a config drive, then changed to force_config_drive=True and after that they can't reboot the servers on the host w/o a config drive since the file doesn't exist
17:03:35 mriedem i said on lyarwood's stein backport, "I'd like to move a bit slowly with this one since it's a very latent issue and is a bit of a behavior change, though justified for the reboot issue. It should also be pretty rare (I don't imagine lots of people are changing the force_config_drive value on their computes once they are deployed)."
17:03:38 mriedem so there it's been sitting
17:04:08 sean-k-mooney mriedem: no what im concerned about is the comment sugges that on a reboot the config drive will be removed
17:04:12 mriedem sean-k-mooney: i'm not really following you on what specific use case you think this regressed

Earlier   Later