Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-09-19
14:54:12 efried sean-k-mooney: no, I'm saying there's no point writing a spec for something brand new.
14:54:21 sean-k-mooney https://review.opendev.org/#/c/674072/3 is the code for ^ by the way
14:54:41 sean-k-mooney because we have enough of a backlog already to finish
14:54:57 mriedem lyarwood: it is in pre_live_migration from the compute manager
14:55:09 mriedem eventually calls down into nova.virt.block_device - refresh_connection_info method
14:55:25 mriedem lyarwood: https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L6719
14:55:47 mriedem https://github.com/openstack/nova/blob/master/nova/virt/block_device.py#L677
14:55:56 efried sean-k-mooney: that, and because it's absolute hell for the "customer" relying on an upstream feature to meet a business commitment to have no better than a 60% chance of the feature actually landing, despite being "approved" (or even *prioritized*) for the release. I'm hoping by reducing scope we can increase that percentage.
14:56:10 lyarwood mriedem: thanks!
14:56:51 mriedem lyarwood: it's probably worth putting a comment by that call in the compute manager saying for legacy attached BDMs that will update the bdm connection_info using the dest host connector
14:57:17 lyarwood mriedem: ack, I'll do that shortly this isn't the first time I've missed this.
15:19:47 mriedem lyarwood: remember, anything involving block devices has to be 3 layers of abstraction removed from where you'd expect it to be happening
15:38:00 donnyd mriedem: 683166 in neutron is so the def route in ipv6 providers isn't purged when the public side of the network is created in the job
15:40:10 mriedem ok. do we have job failures as a result of that? if so, do you have a link to one so we can try to categorize it in elastic-recheck?
15:42:00 openstackgerrit Takashi Kajinami proposed openstack/nova master: conf: Add three new '[libvirt] live_migration_*' options https://review.opendev.org/456571
15:42:20 mriedem melwitt: on https://review.opendev.org/#/c/541420/ i'll try removing the weird dict/attribute stuff since that might be fixed now since i originally posted that patch,
15:42:28 mriedem melwitt: but i wasn't sure what you were saying with "But, I was wondering, should we cover the detach case in the DNM patch above this one too? (I left a comment there)."
15:42:41 mriedem oh nvm let me look at your comment first
15:42:54 openstackgerrit Takashi Kajinami proposed openstack/nova master: conf: Add three new '[libvirt] live_migration_*' options https://review.opendev.org/456571
15:43:22 donnyd mriedem: 2019-09-19 09:50:54.608322 | controller | +++ lib/neutron_plugins/services/l3:_neutron_create_public_subnet_v6:330 : oscwrap --os-cloud devstack-admin --os-region RegionOne subnet create --ip-version 6 --gateway 2001:db8::2 --network 0dd00e84-7a3a-41a5-bcf0-79097e5c1d1f --subnet-range 2001:db8::/64 --no-dhcp ipv6-public-subnet
15:43:22 donnyd https://storage.gra1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_650/682647/3/check/ironic-standalone-ipa-src/65091a8/job-output.txt
15:43:42 donnyd there are a couple more examples
15:44:27 donnyd https://www.irccloud.com/pastebin/7aez1E6y/
15:46:17 melwitt mriedem: ok cool. I never knew there was any issue with attr going to the object
15:46:29 mriedem donnyd: sorry i'm not sure what i'm looking for here that indicates the job fails because of that
15:47:02 donnyd The gist is on v6 providers the def route is dropped at this step and it may or may not come back up in time depending on RA / fw connections
15:48:28 mriedem donnyd: as indicated by "[Zuul] Log Stream did not terminate" ?
15:48:34 donnyd frickler: may have more insights
15:49:04 mriedem "[Zuul] Log Stream did not terminate" shows up in passing jobs so that's not related
15:49:36 mriedem anyway, i don't have +2 on devstack anyway, i was just doing a drive by review
15:50:59 donnyd mriedem: well I was hoping for at least 6 more +1's
15:52:50 melwitt 6 specifically? lol
15:52:58 donnyd yes
15:53:07 melwitt :)
15:53:29 donnyd but no more than 8... that is too many
15:53:50 mriedem back in the icehouse days i think 6 +1s from a single vendor equaled one +2
15:54:20 mriedem especially all within 1 hour of each other after their scrum call ended
15:54:31 melwitt lol
15:54:46 melwitt too real
15:55:06 openstackgerrit Matt Riedemann proposed openstack/nova master: DNM: Stop using volume_api.initialize/terminate_connection https://review.opendev.org/682508
15:55:09 donnyd mriedem: I am really just working on upping my level of irc sarcasm - ircasm
15:55:28 mriedem donnyd: i should probably use o-) or :P or something until you're used to it
15:55:45 donnyd :(
15:56:51 donnyd ok
15:57:56 mriedem somehow i've offended you and i'm not sure how i did
15:58:05 mriedem i meant, i'm 80% sarcasm
15:58:08 mriedem so don't take me literally
16:09:38 mriedem https://review.opendev.org/#/c/679640/ failed with something i haven't seen in the gate before
16:09:46 mriedem https://zuul.opendev.org/t/openstack/build/998bcd66b810467fae1664aaf280d53d/log/controller/logs/screen-n-cpu.txt.gz?severity=4#25797
16:09:52 mriedem os_brick.exception.VolumeDeviceNotFound: Volume device not found at /dev/disk/by-id.
16:10:57 mriedem lyarwood: you might care about that ^ since it's a luks encrypted volume test
16:14:48 lyarwood looking
16:15:29 mriedem this is the multi-cell job, so i wonder if the encrypted volume is created on one host and then the server is created on the other host - but i'd expect that to be a problem in any multi-host job
16:15:51 mriedem maybe we don't run that test in other multihost jobs
16:18:58 bauzas argh, I'm stuck with trying to find a way to delete a single allocation record that I found orphaned
16:19:23 bauzas because from the API perspective, I can only delete all allocs from a consumer UUID
16:20:04 sean-k-mooney really you can do it by RP?
16:20:07 bauzas mriedem_afk: in the case of an orphaned allocation from an evacuation, the audit will find it but we can't delete it or we gonna delete the whole instance resources (and then the user would have to heal)
16:20:19 bauzas sean-k-mooney: https://docs.openstack.org/api-ref/placement/?expanded=delete-allocations-detail#delete-allocations
16:20:39 bauzas I can list the allocations per RP, but I can't delete it
16:21:27 bauzas I was thinking of just checking the allocations and bail out, but in the case of a duplicate, I need to check every allocation per RP
16:21:47 sean-k-mooney ya i was wondering if there was a delete on /resource_providers/{uuid}/allocations
16:22:05 sean-k-mooney bauzas: you could do an update of the consumer allocation
16:22:17 sean-k-mooney and just remove teh allocation on the rp you want to delete
16:22:24 bauzas ah
16:23:14 sean-k-mooney so get the allcotion and do a put to /allocations/{consumer_uuid} - the allcoation you want to remove
16:23:33 bauzas that makes the code a bit less easy to read, but I can try
16:24:00 bauzas sean-k-mooney: I guess placement doesn't delete the allocation when I update by removing the last RP ?
16:24:25 sean-k-mooney there is one way to find out
16:24:30 sean-k-mooney but i dont know
16:24:59 bauzas writing code now
16:25:12 bauzas hopefully I should be able to push something tomorrow morning
16:29:30 bauzas sean-k-mooney: any idea on how I could remove a RP usage by https://docs.openstack.org/api-ref/placement/?&expanded=update-allocations-detail#update-allocations ?
16:29:53 bauzas sean-k-mooney: I guess I need to update with the existing RPs but the one I wanna remove
16:30:02 lyarwood mriedem_afk: this isn't specific to encrypted volumes, the call to connect_volume is failing because once we've connected to the target we can't find a block device locally that matches the WWN provided by c-api/c-vol.
16:30:12 lyarwood mriedem_afk: at this point the fact the volume is encrypted doesn't come into play
16:30:17 sean-k-mooney bauzas: isnt the usage caulated by subtracting the allcoation form the avilabe inventory
16:30:56 bauzas sean-k-mooney: sorry, nevermind the "usage" wording, it's confusing
16:30:58 sean-k-mooney i.e. once you remvoe teh allocation againt the RP the usage should be caulated correctly when next quired
16:31:05 bauzas sean-k-mooney: I meant, say https://docs.openstack.org/api-ref/placement/?&expanded=update-allocations-detail#id72
16:31:42 bauzas sean-k-mooney: if I'd like to remove allocations against 89873422-1373-46e5-b467-f0c5e6acf08f, then I would update the endpoint with a payload having 4e061c03-611e-4caa-bf26-999dcff4284e only ?
16:31:53 bauzas sean-k-mooney: and in this case, would it remove the DB records ?
16:31:54 sean-k-mooney yes
16:32:12 sean-k-mooney that is my understanding at least
16:32:43 bauzas ok, we can test with my functional test at least
16:33:01 sean-k-mooney or with osc-placment but sure
16:33:05 bauzas we miss cdent, tetsuro, efried_pto
16:33:06 sean-k-mooney functional test are good too
16:34:40 sean-k-mooney https://github.com/openstack/placement/blob/master/placement/tests/functional/gabbits/allocations-1.28.yaml#L174-L181
16:34:43 sean-k-mooney bauzas: ^
16:35:08 sean-k-mooney so based on there functionla test if you use 1.28+ it should work
16:36:50 sean-k-mooney well i should include the next test too so
16:36:52 sean-k-mooney https://github.com/openstack/placement/blob/master/placement/tests/functional/gabbits/allocations-1.28.yaml#L174-L187
16:37:50 sean-k-mooney bauzas: so a put of empty allcoation is accpeted implying a put of a partill set of allcoation shoudl also work
16:38:43 openstackgerrit Merged openstack/nova-specs master: Re-propose cross-cell-resize spec for Ussuri https://review.opendev.org/683002
16:52:10 lyarwood mriedem_afk: actually os-brick is doing some random lookups when encrypted volumes are used
17:01:12 bauzas sean-k-mooney: it works seamlessly
17:02:13 openstackgerrit Sylvain Bauza proposed openstack/nova master: WIP: Add a placement audit command https://review.opendev.org/670112
17:03:50 lyarwood mriedem_afk: so it looks like /dev/disk/by-id/ is just slow to populate on these CI nodes and we aren't retrying the lookup https://github.com/openstack/os-brick/blob/1c02d525bb77487d5de89c4bd58116b916d44207/os_brick/initiator/connectors/iscsi.py#L521-L540
17:04:30 lyarwood mriedem_afk: the fact we are using /dev/disk/by-id/ is due to the volume being encrypted https://github.com/openstack/os-brick/blob/1c02d525bb77487d5de89c4bd58116b916d44207/os_brick/initiator/connectors/iscsi.py#L545-L553

Earlier   Later