Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-09-19
15:44:27 donnyd https://www.irccloud.com/pastebin/7aez1E6y/
15:46:17 melwitt mriedem: ok cool. I never knew there was any issue with attr going to the object
15:46:29 mriedem donnyd: sorry i'm not sure what i'm looking for here that indicates the job fails because of that
15:47:02 donnyd The gist is on v6 providers the def route is dropped at this step and it may or may not come back up in time depending on RA / fw connections
15:48:28 mriedem donnyd: as indicated by "[Zuul] Log Stream did not terminate" ?
15:48:34 donnyd frickler: may have more insights
15:49:04 mriedem "[Zuul] Log Stream did not terminate" shows up in passing jobs so that's not related
15:49:36 mriedem anyway, i don't have +2 on devstack anyway, i was just doing a drive by review
15:50:59 donnyd mriedem: well I was hoping for at least 6 more +1's
15:52:50 melwitt 6 specifically? lol
15:52:58 donnyd yes
15:53:07 melwitt :)
15:53:29 donnyd but no more than 8... that is too many
15:53:50 mriedem back in the icehouse days i think 6 +1s from a single vendor equaled one +2
15:54:20 mriedem especially all within 1 hour of each other after their scrum call ended
15:54:31 melwitt lol
15:54:46 melwitt too real
15:55:06 openstackgerrit Matt Riedemann proposed openstack/nova master: DNM: Stop using volume_api.initialize/terminate_connection https://review.opendev.org/682508
15:55:09 donnyd mriedem: I am really just working on upping my level of irc sarcasm - ircasm
15:55:28 mriedem donnyd: i should probably use o-) or :P or something until you're used to it
15:55:45 donnyd :(
15:56:51 donnyd ok
15:57:56 mriedem somehow i've offended you and i'm not sure how i did
15:58:05 mriedem i meant, i'm 80% sarcasm
15:58:08 mriedem so don't take me literally
16:09:38 mriedem https://review.opendev.org/#/c/679640/ failed with something i haven't seen in the gate before
16:09:46 mriedem https://zuul.opendev.org/t/openstack/build/998bcd66b810467fae1664aaf280d53d/log/controller/logs/screen-n-cpu.txt.gz?severity=4#25797
16:09:52 mriedem os_brick.exception.VolumeDeviceNotFound: Volume device not found at /dev/disk/by-id.
16:10:57 mriedem lyarwood: you might care about that ^ since it's a luks encrypted volume test
16:14:48 lyarwood looking
16:15:29 mriedem this is the multi-cell job, so i wonder if the encrypted volume is created on one host and then the server is created on the other host - but i'd expect that to be a problem in any multi-host job
16:15:51 mriedem maybe we don't run that test in other multihost jobs
16:18:58 bauzas argh, I'm stuck with trying to find a way to delete a single allocation record that I found orphaned
16:19:23 bauzas because from the API perspective, I can only delete all allocs from a consumer UUID
16:20:04 sean-k-mooney really you can do it by RP?
16:20:07 bauzas mriedem_afk: in the case of an orphaned allocation from an evacuation, the audit will find it but we can't delete it or we gonna delete the whole instance resources (and then the user would have to heal)
16:20:19 bauzas sean-k-mooney: https://docs.openstack.org/api-ref/placement/?expanded=delete-allocations-detail#delete-allocations
16:20:39 bauzas I can list the allocations per RP, but I can't delete it
16:21:27 bauzas I was thinking of just checking the allocations and bail out, but in the case of a duplicate, I need to check every allocation per RP
16:21:47 sean-k-mooney ya i was wondering if there was a delete on /resource_providers/{uuid}/allocations
16:22:05 sean-k-mooney bauzas: you could do an update of the consumer allocation
16:22:17 sean-k-mooney and just remove teh allocation on the rp you want to delete
16:22:24 bauzas ah
16:23:14 sean-k-mooney so get the allcotion and do a put to /allocations/{consumer_uuid} - the allcoation you want to remove
16:23:33 bauzas that makes the code a bit less easy to read, but I can try
16:24:00 bauzas sean-k-mooney: I guess placement doesn't delete the allocation when I update by removing the last RP ?
16:24:25 sean-k-mooney there is one way to find out
16:24:30 sean-k-mooney but i dont know
16:24:59 bauzas writing code now
16:25:12 bauzas hopefully I should be able to push something tomorrow morning
16:29:30 bauzas sean-k-mooney: any idea on how I could remove a RP usage by https://docs.openstack.org/api-ref/placement/?&expanded=update-allocations-detail#update-allocations ?
16:29:53 bauzas sean-k-mooney: I guess I need to update with the existing RPs but the one I wanna remove
16:30:02 lyarwood mriedem_afk: this isn't specific to encrypted volumes, the call to connect_volume is failing because once we've connected to the target we can't find a block device locally that matches the WWN provided by c-api/c-vol.
16:30:12 lyarwood mriedem_afk: at this point the fact the volume is encrypted doesn't come into play
16:30:17 sean-k-mooney bauzas: isnt the usage caulated by subtracting the allcoation form the avilabe inventory
16:30:56 bauzas sean-k-mooney: sorry, nevermind the "usage" wording, it's confusing
16:30:58 sean-k-mooney i.e. once you remvoe teh allocation againt the RP the usage should be caulated correctly when next quired
16:31:05 bauzas sean-k-mooney: I meant, say https://docs.openstack.org/api-ref/placement/?&expanded=update-allocations-detail#id72
16:31:42 bauzas sean-k-mooney: if I'd like to remove allocations against 89873422-1373-46e5-b467-f0c5e6acf08f, then I would update the endpoint with a payload having 4e061c03-611e-4caa-bf26-999dcff4284e only ?
16:31:53 bauzas sean-k-mooney: and in this case, would it remove the DB records ?
16:31:54 sean-k-mooney yes
16:32:12 sean-k-mooney that is my understanding at least
16:32:43 bauzas ok, we can test with my functional test at least
16:33:01 sean-k-mooney or with osc-placment but sure
16:33:05 bauzas we miss cdent, tetsuro, efried_pto
16:33:06 sean-k-mooney functional test are good too
16:34:40 sean-k-mooney https://github.com/openstack/placement/blob/master/placement/tests/functional/gabbits/allocations-1.28.yaml#L174-L181
16:34:43 sean-k-mooney bauzas: ^
16:35:08 sean-k-mooney so based on there functionla test if you use 1.28+ it should work
16:36:50 sean-k-mooney well i should include the next test too so
16:36:52 sean-k-mooney https://github.com/openstack/placement/blob/master/placement/tests/functional/gabbits/allocations-1.28.yaml#L174-L187
16:37:50 sean-k-mooney bauzas: so a put of empty allcoation is accpeted implying a put of a partill set of allcoation shoudl also work
16:38:43 openstackgerrit Merged openstack/nova-specs master: Re-propose cross-cell-resize spec for Ussuri https://review.opendev.org/683002
16:52:10 lyarwood mriedem_afk: actually os-brick is doing some random lookups when encrypted volumes are used
17:01:12 bauzas sean-k-mooney: it works seamlessly
17:02:13 openstackgerrit Sylvain Bauza proposed openstack/nova master: WIP: Add a placement audit command https://review.opendev.org/670112
17:03:50 lyarwood mriedem_afk: so it looks like /dev/disk/by-id/ is just slow to populate on these CI nodes and we aren't retrying the lookup https://github.com/openstack/os-brick/blob/1c02d525bb77487d5de89c4bd58116b916d44207/os_brick/initiator/connectors/iscsi.py#L521-L540
17:04:30 lyarwood mriedem_afk: the fact we are using /dev/disk/by-id/ is due to the volume being encrypted https://github.com/openstack/os-brick/blob/1c02d525bb77487d5de89c4bd58116b916d44207/os_brick/initiator/connectors/iscsi.py#L545-L553
17:05:37 lyarwood mriedem_afk: and is due to the horrid design of the original encryptors that symlink things in a horrific way such that we can't use the actual block device path on the host otherwise we'd nuke it https://bugs.launchpad.net/os-brick/+bug/1703954
17:05:38 openstack Launchpad bug 1703954 in OpenStack Compute (nova) "Attach/Detach encrypted volume problems with real paths" [Undecided,Incomplete]
17:07:23 lyarwood mriedem_afk: I'll post something in the morning to retry _get_device_link but in the longer term I think I need to sort out the symlinking in the encryptors such that it's safe to provide them with a real block device path and they don't do anything stupid.
17:07:41 lyarwood mriedem_afk: anyway both tasks for the morning \o_
17:36:20 mriedem lyarwood: ack thanks for investigating, i'll open a bug (i found https://bugs.launchpad.net/os-brick/+bug/1820007 but looks old/different) and start tracking in e-r
17:36:21 openstack Launchpad bug 1820007 in os-brick "Failed to attach encrypted volumes after detach: volume device not found at /dev/disk/by-id" [Undecided,New]
17:36:29 mriedem it also only failed on ovh nodes so maybe those are slower here, idk
17:37:51 mriedem actually a lot of the details in that bug look the same so i'll just re-use that
18:10:15 openstackgerrit Walter A. Boring IV (hemna) proposed openstack/nova stable/pike: WIP: Avoid redundant initialize_connection on source post live migration https://review.opendev.org/683008
18:35:25 openstackgerrit Merged openstack/nova master: Trigger real BuildAbortException during migrate with bandwidth https://review.opendev.org/682876
20:15:17 openstackgerrit Merged openstack/nova master: Validate CPU config options against running instances https://review.opendev.org/680107
20:15:26 openstackgerrit Merged openstack/nova master: objects: Add 'NUMACell.pcpuset' field https://review.opendev.org/680108
20:15:36 openstackgerrit Merged openstack/nova master: hardware: Differentiate between shared and dedicated CPUs https://review.opendev.org/671800
21:56:57 openstackgerrit Matt Riedemann proposed openstack/nova master: Add evacuate vs rebuild contributor doc https://review.opendev.org/683245
21:57:03 mriedem efried_pto: you asked for it ^ dansmith your review on that would also be helpful due to your blog post
21:57:06 mriedem *the* blog post
22:00:59 dansmith mriedem: https://www.youtube.com/watch?v=gVpzrxjiAtc
22:01:29 mriedem coincidentally i'm wearing that same outfit
22:01:38 mriedem just went to the scarf store last night
22:01:47 dansmith I don't doubt it
22:01:58 dansmith coincidentally, I'm wearing nothing *but* that scarf
22:02:22 mriedem man, i can't imagine the amount of coke flowing through these guys in this video

Earlier   Later