Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-09-19
15:58:08 mriedem so don't take me literally
16:09:38 mriedem https://review.opendev.org/#/c/679640/ failed with something i haven't seen in the gate before
16:09:46 mriedem https://zuul.opendev.org/t/openstack/build/998bcd66b810467fae1664aaf280d53d/log/controller/logs/screen-n-cpu.txt.gz?severity=4#25797
16:09:52 mriedem os_brick.exception.VolumeDeviceNotFound: Volume device not found at /dev/disk/by-id.
16:10:57 mriedem lyarwood: you might care about that ^ since it's a luks encrypted volume test
16:14:48 lyarwood looking
16:15:29 mriedem this is the multi-cell job, so i wonder if the encrypted volume is created on one host and then the server is created on the other host - but i'd expect that to be a problem in any multi-host job
16:15:51 mriedem maybe we don't run that test in other multihost jobs
16:18:58 bauzas argh, I'm stuck with trying to find a way to delete a single allocation record that I found orphaned
16:19:23 bauzas because from the API perspective, I can only delete all allocs from a consumer UUID
16:20:04 sean-k-mooney really you can do it by RP?
16:20:07 bauzas mriedem_afk: in the case of an orphaned allocation from an evacuation, the audit will find it but we can't delete it or we gonna delete the whole instance resources (and then the user would have to heal)
16:20:19 bauzas sean-k-mooney: https://docs.openstack.org/api-ref/placement/?expanded=delete-allocations-detail#delete-allocations
16:20:39 bauzas I can list the allocations per RP, but I can't delete it
16:21:27 bauzas I was thinking of just checking the allocations and bail out, but in the case of a duplicate, I need to check every allocation per RP
16:21:47 sean-k-mooney ya i was wondering if there was a delete on /resource_providers/{uuid}/allocations
16:22:05 sean-k-mooney bauzas: you could do an update of the consumer allocation
16:22:17 sean-k-mooney and just remove teh allocation on the rp you want to delete
16:22:24 bauzas ah
16:23:14 sean-k-mooney so get the allcotion and do a put to /allocations/{consumer_uuid} - the allcoation you want to remove
16:23:33 bauzas that makes the code a bit less easy to read, but I can try
16:24:00 bauzas sean-k-mooney: I guess placement doesn't delete the allocation when I update by removing the last RP ?
16:24:25 sean-k-mooney there is one way to find out
16:24:30 sean-k-mooney but i dont know
16:24:59 bauzas writing code now
16:25:12 bauzas hopefully I should be able to push something tomorrow morning
16:29:30 bauzas sean-k-mooney: any idea on how I could remove a RP usage by https://docs.openstack.org/api-ref/placement/?&expanded=update-allocations-detail#update-allocations ?
16:29:53 bauzas sean-k-mooney: I guess I need to update with the existing RPs but the one I wanna remove
16:30:02 lyarwood mriedem_afk: this isn't specific to encrypted volumes, the call to connect_volume is failing because once we've connected to the target we can't find a block device locally that matches the WWN provided by c-api/c-vol.
16:30:12 lyarwood mriedem_afk: at this point the fact the volume is encrypted doesn't come into play
16:30:17 sean-k-mooney bauzas: isnt the usage caulated by subtracting the allcoation form the avilabe inventory
16:30:56 bauzas sean-k-mooney: sorry, nevermind the "usage" wording, it's confusing
16:30:58 sean-k-mooney i.e. once you remvoe teh allocation againt the RP the usage should be caulated correctly when next quired
16:31:05 bauzas sean-k-mooney: I meant, say https://docs.openstack.org/api-ref/placement/?&expanded=update-allocations-detail#id72
16:31:42 bauzas sean-k-mooney: if I'd like to remove allocations against 89873422-1373-46e5-b467-f0c5e6acf08f, then I would update the endpoint with a payload having 4e061c03-611e-4caa-bf26-999dcff4284e only ?
16:31:53 bauzas sean-k-mooney: and in this case, would it remove the DB records ?
16:31:54 sean-k-mooney yes
16:32:12 sean-k-mooney that is my understanding at least
16:32:43 bauzas ok, we can test with my functional test at least
16:33:01 sean-k-mooney or with osc-placment but sure
16:33:05 bauzas we miss cdent, tetsuro, efried_pto
16:33:06 sean-k-mooney functional test are good too
16:34:40 sean-k-mooney https://github.com/openstack/placement/blob/master/placement/tests/functional/gabbits/allocations-1.28.yaml#L174-L181
16:34:43 sean-k-mooney bauzas: ^
16:35:08 sean-k-mooney so based on there functionla test if you use 1.28+ it should work
16:36:50 sean-k-mooney well i should include the next test too so
16:36:52 sean-k-mooney https://github.com/openstack/placement/blob/master/placement/tests/functional/gabbits/allocations-1.28.yaml#L174-L187
16:37:50 sean-k-mooney bauzas: so a put of empty allcoation is accpeted implying a put of a partill set of allcoation shoudl also work
16:38:43 openstackgerrit Merged openstack/nova-specs master: Re-propose cross-cell-resize spec for Ussuri https://review.opendev.org/683002
16:52:10 lyarwood mriedem_afk: actually os-brick is doing some random lookups when encrypted volumes are used
17:01:12 bauzas sean-k-mooney: it works seamlessly
17:02:13 openstackgerrit Sylvain Bauza proposed openstack/nova master: WIP: Add a placement audit command https://review.opendev.org/670112
17:03:50 lyarwood mriedem_afk: so it looks like /dev/disk/by-id/ is just slow to populate on these CI nodes and we aren't retrying the lookup https://github.com/openstack/os-brick/blob/1c02d525bb77487d5de89c4bd58116b916d44207/os_brick/initiator/connectors/iscsi.py#L521-L540
17:04:30 lyarwood mriedem_afk: the fact we are using /dev/disk/by-id/ is due to the volume being encrypted https://github.com/openstack/os-brick/blob/1c02d525bb77487d5de89c4bd58116b916d44207/os_brick/initiator/connectors/iscsi.py#L545-L553
17:05:37 lyarwood mriedem_afk: and is due to the horrid design of the original encryptors that symlink things in a horrific way such that we can't use the actual block device path on the host otherwise we'd nuke it https://bugs.launchpad.net/os-brick/+bug/1703954
17:05:38 openstack Launchpad bug 1703954 in OpenStack Compute (nova) "Attach/Detach encrypted volume problems with real paths" [Undecided,Incomplete]
17:07:23 lyarwood mriedem_afk: I'll post something in the morning to retry _get_device_link but in the longer term I think I need to sort out the symlinking in the encryptors such that it's safe to provide them with a real block device path and they don't do anything stupid.
17:07:41 lyarwood mriedem_afk: anyway both tasks for the morning \o_
17:36:20 mriedem lyarwood: ack thanks for investigating, i'll open a bug (i found https://bugs.launchpad.net/os-brick/+bug/1820007 but looks old/different) and start tracking in e-r
17:36:21 openstack Launchpad bug 1820007 in os-brick "Failed to attach encrypted volumes after detach: volume device not found at /dev/disk/by-id" [Undecided,New]
17:36:29 mriedem it also only failed on ovh nodes so maybe those are slower here, idk
17:37:51 mriedem actually a lot of the details in that bug look the same so i'll just re-use that
18:10:15 openstackgerrit Walter A. Boring IV (hemna) proposed openstack/nova stable/pike: WIP: Avoid redundant initialize_connection on source post live migration https://review.opendev.org/683008
18:35:25 openstackgerrit Merged openstack/nova master: Trigger real BuildAbortException during migrate with bandwidth https://review.opendev.org/682876
20:15:17 openstackgerrit Merged openstack/nova master: Validate CPU config options against running instances https://review.opendev.org/680107
20:15:26 openstackgerrit Merged openstack/nova master: objects: Add 'NUMACell.pcpuset' field https://review.opendev.org/680108
20:15:36 openstackgerrit Merged openstack/nova master: hardware: Differentiate between shared and dedicated CPUs https://review.opendev.org/671800
21:56:57 openstackgerrit Matt Riedemann proposed openstack/nova master: Add evacuate vs rebuild contributor doc https://review.opendev.org/683245
21:57:03 mriedem efried_pto: you asked for it ^ dansmith your review on that would also be helpful due to your blog post
21:57:06 mriedem *the* blog post
22:00:59 dansmith mriedem: https://www.youtube.com/watch?v=gVpzrxjiAtc
22:01:29 mriedem coincidentally i'm wearing that same outfit
22:01:38 mriedem just went to the scarf store last night
22:01:47 dansmith I don't doubt it
22:01:58 dansmith coincidentally, I'm wearing nothing *but* that scarf
22:02:22 mriedem man, i can't imagine the amount of coke flowing through these guys in this video
22:03:15 dansmith yeah keith looks like he hasn't slept in two weeks
22:28:34 openstackgerrit Dustin Cowles proposed openstack/nova master: Provider Config File: YAML file loading and schema validation https://review.opendev.org/673341
22:28:35 openstackgerrit Dustin Cowles proposed openstack/nova master: Provider Config File: Function to further validate and retrieve configs https://review.opendev.org/676029
22:28:36 openstackgerrit Dustin Cowles proposed openstack/nova master: WIP: Provider Config File: Merge provider configs to provider tree https://review.opendev.org/676522
23:15:41 openstackgerrit Eric Fried proposed openstack/nova master: libvirt: Enable driver configuring PMEM namespaces https://review.opendev.org/679640
#openstack-nova - 2019-09-20
00:21:22 openstackgerrit sean mooney proposed openstack/nova-specs master: resubmit image metadata prefiltering spec for ussuri https://review.opendev.org/683258
08:59:52 luyao efried_pto, stephenfin: Hi, could you help review the vpmems doc if you get time, since you both are familiar with the vpmems series. :D I think I get the content close which is focusing on the current functionality. https://review.opendev.org/#/c/680300.
09:02:32 luyao And the patch 'objects: use all_things_equal from objects.base' is ready to merge , need +W on it, https://review.opendev.org/#/c/681397/13
09:07:35 openstackgerrit Sylvain Bauza proposed openstack/nova master: Add a prelude for the Train release https://review.opendev.org/683327
09:59:32 AdamMork Hello, friends! I have a openstack contains 2 controllers an 4 compute node's. I need to enable AES-NI for one or more vm. Openstack deployed via kolla and work on docker containers(nova container, neutron container etc. on controller). I read manual https://software.intel.com/en-us/articles/openstack-epa-feature-breakdown-and-analysis . On the
09:59:33 AdamMork compute node (in bios i enable AES-NI) . But in vm use command (cat /proc/cpuinfo | grep aes) i not found aes support. How to configure nova to enable aes instructions?
10:28:35 AdamMork ok! How i can modify Nova conf "The Nova* libvirt driver takes its configuration information from a section in the main Nova file /etc/nova/nova.conf." I have docker containers and how to find /etc/nova/nova.conf.? should i connect to nova container?
11:28:54 AdamMork ok. I see more manuals and understand : in globals.yml need uncomment :node_custom_config: "/etc/kolla/config" and overrides basic config for nova (As of now kolla only supports config overrides for ini based configs.) https://github.com/openstack/kolla-ansible/blob/master/doc/source/admin/advanced-configuration.rst THX
12:54:46 mriedem gibi: i saw your resize reschedule rpc pin bug, do you know if that's a new regression in train?
12:55:28 gibi mriedem: not yet. I'm about to push a reproduction patch, then I will look into when the fault was introduced
12:55:57 gibi it is clear that the problem is with the legacy request spec in prep_resize ending up in the conductor during a re-schedule with rpc pinned to 5.0
12:58:38 gibi mriedem: this is where it blows https://github.com/openstack/nova/blob/9b2e00e015f22b2d876cd3c239af8e139040c8c8/nova/conductor/manager.py#L327
12:58:51 mriedem i know that in compute rpc api 5.1 we send the RequestSpec to compute https://opendev.org/openstack/nova/src/branch/master/nova/compute/rpcapi.py#L833
12:58:58 mriedem but backlevel if to the dict form if we can't
12:59:13 mriedem gibi: ah yeah - that's my patch
12:59:15 mriedem remember?
12:59:34 mriedem https://review.opendev.org/#/c/680762/
13:00:03 mriedem so if you have a recreate we can lay that on top
13:00:43 gibi mriedem: ohh you have a fix for it. cool. Yeah I will push a functional repro soon and the you can rebase

Earlier   Later