Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-11-09
17:08:36 dansmith even if not backed by actual 8k
17:09:01 cfriesen as far as I know the block size can be different from the sector size
17:09:04 dansmith also, netapp I think uses some super odd sizes, even to the point of having weirdly low-level-formatted drives for them
17:09:38 sean-k-mooney i know some raid controls can be configured to exposed larger sector sizes but i dont know how common that is anymore
17:09:53 dansmith yep
17:10:41 sean-k-mooney mriedem: i think one of the things you suggested was just making a config option correct
17:11:34 sean-k-mooney something like directio_sector_sizes=512,4096
17:13:32 dansmith ah, but it's hidden to the LUN: https://kb.netapp.com/app/answers/answer_view/a_id/1001353/~/how-can-the-bytes%2Fsector-be-changed-in-a-luns-geometry%3F-
17:14:46 cfriesen the goal here is to figure out if the filesystem supports O_DIRECT. according to the man page, this should be set to the logical block size of the underlying storage, which can be determined using the ioctl() BLKSSZGET operation or by calling "blockdev --getss"
17:16:20 sean-k-mooney cfriesen: i think part of the issue is that on older kernel < 2.4 O_DIRECT required alinged access
17:16:42 sean-k-mooney but on bsd and newer linux kernel O_DIRECT did not reuqire alinged acess
17:17:13 sean-k-mooney cfriesen: im not actully sure of we need to do the alignment check we are doing anymore
17:17:30 openstackgerrit Merged openstack/nova stable/queens: fixtures: Track volume attachments within CinderFixtureNewAttachFlow https://review.openstack.org/612494
17:17:34 cfriesen size of the underlying storage (typically 512 bytes) suffices.
17:17:34 cfriesen of the filesystem. Since Linux 2.6.0, alignment to the logical block
17:17:34 cfriesen and the file offset must all be multiples of the logical block size
17:17:34 cfriesen the man page says: Under Linux 2.4, transfer sizes, and the alignment of the user buffer
17:17:40 cfriesen oh, ick. sorry
17:17:42 openstackgerrit Merged openstack/nova stable/queens: Add regression test for bug#1784353 https://review.openstack.org/612495
17:19:05 sean-k-mooney cfriesen: ok so we still need aligned acess but the function is ment to determin if directio is possibel what its currently doing is determing if direct 512B aligned acess is possible with is a different thing
17:19:34 cfriesen agreed. I think switching to 4K would cover 95% of the cases.
17:19:50 cfriesen doing it totally correctly woudl require querying the block size from the OS for that specific device
17:20:46 mriedem sean-k-mooney: i suggested a config option as an option because this sounds very hit or miss
17:21:31 sean-k-mooney well the backing store for libvirt instance is only going to be on one mountpoint
17:21:33 mriedem this is definitely not something i've got a lot of experience in though
17:22:18 sean-k-mooney presumable it will all have the same alignment/sector size so we could jsut have a single valus and defualt it to 512 and they could set it to 4k or 8k if they have something else
17:23:02 sean-k-mooney the other option is just super over align to like a 64K bondary
17:23:28 sean-k-mooney that said im sure someone will have a 128K lun now that i have said that
17:24:25 cfriesen so guaranteed setting it to 4K will work for both 4K and 512b disks, so I think 4K should be the default
17:24:58 sean-k-mooney cfriesen: its also the most common sector size on most new disks so ya that should work
17:25:10 dansmith for years now
17:25:25 cfriesen I'd be okay with a config option if someone has weird hardware
17:26:20 sean-k-mooney cfriesen: so your going to submit a patch :)_
17:26:47 cfriesen there's already a patch in progress
17:26:53 cfriesen by someone else
17:27:09 mriedem wee https://bugs.launchpad.net/nova/+bug/1798688
17:27:10 openstack Launchpad bug 1798688 in OpenStack Compute (nova) "AllocationUpdateFailed_Remote: Failed to update allocations for consumer. Error: another process changed the consumer after the report client read the consumer state during the claim" [Undecided,Triaged]
17:27:21 mriedem looks like our scheduler allocation claim races have shot up since nov 4
17:28:31 cfriesen mriedem: did you want to get the person to update the 4K patch to add a config option? or change the hardcoded number to 4K (which is still an improvement) and add the config option later if someone complains?
17:29:11 dansmith mriedem: what does that mean exactly? just the scheduler having to retry the allocation part?
17:29:29 mriedem dansmith: we already retry on PUT allocations in the scheduler
17:29:39 mriedem maybe the error message changed and we're not retrying properly now?
17:29:41 mriedem i haven't dug in yet
17:29:53 mriedem cfriesen: the patch already just changes 512 to 4k right?
17:30:06 mriedem cfriesen: i believe i just said we might want a 'fixes' release note for it
17:30:07 mriedem as a heads up
17:30:09 melwitt cfriesen: this is the method we have to checking for directio, if that's the same thing you mentioned earlier https://github.com/openstack/nova/blob/master/nova/privsep/utils.py#L34
17:30:10 dansmith mriedem: I know we do, I'm wondering if you mean it's just having to retry more lately or if it's failing
17:30:18 mriedem haven't dug into the logs yet
17:30:23 mriedem hopefully we log if we are retrying
17:30:45 sean-k-mooney mriedem: ya i was under teh impression we had planned at least to retry in this case which is why we have the generation on the resouce providers in the first place
17:31:01 dansmith sean-k-mooney: we do retry
17:31:06 mriedem the scheduler doesn't do anything with generations for this as far as i know
17:31:38 openstack Launchpad bug 1783338 in OpenStack Compute (nova) "Unexpected exception in API method: ValueError: year is out of range" [Medium,Confirmed] - Assigned to Ghanshyam Mann (ghanshyammann)
17:31:38 mriedem just duplicated another bug in triage to this if someone is looking for work https://bugs.launchpad.net/nova/+bug/1783338
17:31:48 mriedem something in the simple tenant usage code
17:32:01 cfriesen mriedem: in irc you were talking about a config option I thought. but yeah, I'd be cool with just a release note for now.
17:32:48 openstack Launchpad bug 1783338 in OpenStack Compute (nova) "Unexpected exception in API method: ValueError: year is out of range" [Medium,Confirmed] - Assigned to Ghanshyam Mann (ghanshyammann)
17:32:48 mriedem gmann: looks like https://bugs.launchpad.net/nova/+bug/1783338 was due to bad data in the db? you're probably traveling, but if you don't plan on handling this we should unassign you https://bugs.launchpad.net/nova/+bug/1783338
17:32:49 cfriesen melwitt: yes, that's the one. there's a patch in review to change the 512 to 4096 in there. which is good, but maybe not sufficient for exotic hardware
17:33:02 mriedem i'm mriedem
17:33:26 melwitt I said something to him earlier
17:34:06 melwitt cfriesen: ok, cool. *looks for the patch*
17:34:07 mriedem oh missed that
17:34:32 cfriesen melwitt: https://review.openstack.org/#/c/616580
17:34:45 melwitt probably because our nicks blend together. maybe I need to be jgwentworth all the time
17:35:44 mriedem dansmith: looks like, from the logs, that we're not retrying
17:36:30 dansmith mriedem: maybe something changed recently then?
17:39:26 melwitt cfriesen: ok, so looks like trying to decide which value to use for the check
17:40:36 mriedem dansmith: my guess would be https://review.openstack.org/#/c/583667/
17:40:54 mriedem Nov 06 19:48:36.969356 ubuntu-xenial-inap-mtl01-0000379614 nova-scheduler[12154]: DEBUG nova.scheduler.client.report [None req-f266a0ff-2840-413d-9877-4500e61512f5 tempest-ServersNegativeTestJSON-477704048 tempest-ServersNegativeTestJSON-477704048] Doubling-up allocation_request for move operation. {{(pid=13677) _move_operation_alloc_request /opt/stack/nova/nova/scheduler/client/report.py:203}}
17:40:54 mriedem because the scheduler logs are saying we're doing a double up allocation
17:41:09 mriedem but in this test, we're just unshelving a shelved offloaded server
17:41:15 mriedem so that shouldn't really double up any allocatoins
17:42:08 cfriesen melwitt: basically, yes. 4096 would work for the vast majority of systems
17:42:49 mriedem : 128}}}} {{(pid=13677) _move_operation_alloc_request /opt/stack/nova/nova/scheduler/client/report.py:234}}
17:42:49 mriedem Nov 06 19:48:36.969659 ubuntu-xenial-inap-mtl01-0000379614 nova-scheduler[12154]: DEBUG nova.scheduler.client.report [None req-f266a0ff-2840-413d-9877-4500e61512f5 tempest-ServersNegativeTestJSON-477704048 tempest-ServersNegativeTestJSON-477704048] New allocation_request containing both source and destination hosts in move operation: {'allocations': {u'3ceb7eab-549c-40ba-a70c-320822c310ab': {'resources': {u'VCPU': 2, u'MEMORY
17:42:50 dansmith mriedem: it also touches the code near where we raise retry...
17:42:52 mriedem ^ is definitely wrong
17:43:00 mriedem there is only one provider in that log
17:43:32 dansmith mriedem: it seems to specifically exclude the consumer generation conflict from the case where we retry
17:46:23 dansmith mriedem: do you see "another process changed the consumer" in the log?
17:46:43 mriedem yes
17:46:53 dansmith then it's hitting that consumer case and not retrying
17:46:55 mriedem that's why we don't retry
17:46:58 dansmith yeah
17:47:09 mriedem i also don't know why it thinks we're starting with existing allocations for a shelved offloaded server
17:47:44 dansmith and that's causing it to try to double?
17:48:10 mriedem well, it goes into _move_operation_alloc_request but doesn't actually double anything
17:48:14 mriedem : 128}}}} {{(pid=13677) _move_operation_alloc_request /opt/stack/nova/nova/scheduler/client/report.py:234}}
17:48:14 mriedem Nov 06 19:48:36.969659 ubuntu-xenial-inap-mtl01-0000379614 nova-scheduler[12154]: DEBUG nova.scheduler.client.report [None req-f266a0ff-2840-413d-9877-4500e61512f5 tempest-ServersNegativeTestJSON-477704048 tempest-ServersNegativeTestJSON-477704048] New allocation_request containing both source and destination hosts in move operation: {'allocations': {u'3ceb7eab-549c-40ba-a70c-320822c310ab': {'resources': {u'VCPU': 2, u'MEMORY
17:48:27 mriedem there is only one provider in that body
17:51:52 mriedem ment.concurrent_update", "detail": "There was a conflict when trying to complete your request.\n\n consumer generation conflict - expected null but got 1 ", "title": "Conflict"}]}
17:51:52 mriedem Nov 06 19:48:37.013780 ubuntu-xenial-inap-mtl01-0000379614 nova-scheduler[12154]: WARNING nova.scheduler.client.report [None req-f266a0ff-2840-413d-9877-4500e61512f5 tempest-ServersNegativeTestJSON-477704048 tempest-ServersNegativeTestJSON-477704048] Failed to save allocation for 6665f00a-dcf1-4286-b075-d7dcd7c37487. Got HTTP 409: {"errors": [{"status": 409, "request_id": "req-c9ba6cbd-3b6e-4e5d-b550-9588be8a49d2", "code": "p
17:51:52 mriedem this is the error from placement
17:58:38 mriedem idk wtf is going on, but i see 3 different PUT allocations in the placement logs for that consumer
17:58:52 mriedem first is probably for the initial scheduler, and then we offload and remove allocations
17:59:03 mriedem 2nd is for the unshelve
18:07:11 mriedem aha
18:07:16 mriedem the allocatoin delete on unshelve changed with this patch https://review.openstack.org/#/c/591597/

Earlier   Later