| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-01-19 | |||
| 18:33:21 | sean-k-mooney | so ya looks like a gluster issue | |
| 18:33:34 | spatel | but it deleted files inside that directory | |
| 18:33:48 | spatel | I am going to talk to someone who own that storage and see what is going on | |
| 18:33:55 | sean-k-mooney | ack | |
| 18:34:11 | spatel | Thanks for checking | |
| 18:34:12 | sean-k-mooney | im not sure how gluster tracks files vs direcoties | |
| 18:34:28 | sean-k-mooney | but it may handel directory inodes differently | |
| 18:34:53 | spatel | https://paste.opendev.org/show/812231/ | |
| 18:35:33 | spatel | something is wrong with gluster for sure.. i am able to delete files but not directory | |
| 18:35:46 | sean-k-mooney | ack | |
| 18:35:51 | sean-k-mooney | ya it could also be fuse | |
| 18:36:07 | sean-k-mooney | are you mounting glusterfs with fuse or using a kernel dirver | |
| 18:37:00 | spatel | 10.10.217.21:gluster_vol2/voyager /mnt/glusterfs glusterfs defaults,backup-volfile-servers=10.10.217.22:10.10.217.23:10.10.217.24:10.10.217.25:10.10.217.26 | |
| 18:37:22 | spatel | fuse.. hmm ? | |
| 18:37:47 | spatel | i just install glusterfs-client RPM package and use mount tool to mount it | |
| 18:37:49 | sean-k-mooney | i think there is a user space dirver and a kernel dirver for gluster like there is for ceph | |
| 18:38:02 | spatel | let me check.. that is good point | |
| 18:39:22 | sean-k-mooney | https://bugzilla.redhat.com/show_bug.cgi?id=1508999 | |
| 18:43:21 | spatel | Reading bug report, looks interesting | |
| 18:44:19 | sean-k-mooney | i have not looked at it closely but the title seamed relevent | |
| 18:44:58 | spatel | i am having same issue but trying to understand how they fix. i don't control gluster so not sure but i can explain someone | |
| 18:45:21 | sean-k-mooney | i would start with your simple repoducer | |
| 18:45:34 | sean-k-mooney | e.g. create teh dir and show you cant delete it | |
| 18:45:56 | sean-k-mooney | that really shoudl work | |
| 18:46:26 | sean-k-mooney | one thing to check is it any directoy or just ones at the root of the volume | |
| 18:47:34 | sean-k-mooney | i.e. does mkdir -p .../temp/mydata rm -rf .../temp/mydata work | |
| 18:51:24 | spatel | let me check that | |
| 18:52:36 | spatel | any directory in tree period | |
| 18:52:53 | sean-k-mooney | ack but it can delete files | |
| 18:53:01 | sean-k-mooney | that is very odd indeed | |
| 18:53:03 | spatel | i am able to delete files not matter where they located but not able to delete any dir | |
| 18:53:13 | spatel | Yes i can delete files.. | |
| 18:53:20 | spatel | not directory | |
| 18:53:44 | sean-k-mooney | ya i have never seen that before honestly unless the permission of the folder vs fiels are diffent | |
| 18:54:18 | sean-k-mooney | but the error implies it an internal gluster issue not a simple permissions one | |
| 18:54:40 | spatel | I asked someone to take a look | |
| 20:32:30 | spatel | sean-k-mooney i have ask someone to take a look so hope they find something | |
| 21:34:12 | opendevreview | Alexey Stupnikov proposed openstack/nova master: Support use_multipath for NVME driver https://review.opendev.org/c/openstack/nova/+/823941 | |
| 23:02:31 | clarkb | sean-k-mooney: fyi https://git.centos.org/rpms/systemd/c/3d3dc89fb25868e8038ecac8d5aef0603bdfaaa2?branch=c8s was recently committed. I don't know how/when/if that will become a package in the package repos but progress | |
| 23:39:15 | sean-k-mooney | ack i tihnk it should go through the koji automated build once th commit lands in dist git automatically | |
| 23:39:33 | sean-k-mooney | clarkb: so i woudl expect that to show up relitivly quickly once its commited | |
| 23:45:56 | sean-k-mooney | clarkb: https://koji.mbox.centos.org/koji/buildinfo?buildID=20898 there was the attmepted build | |
| 23:46:00 | sean-k-mooney | looks like it failed | |
| 23:49:33 | sean-k-mooney | 155/298 test-procfs-util FAIL 0.32s killed by signal 6 SIGABRT | |
| 23:50:01 | sean-k-mooney | ――――――――――――――――――――――――――――――――――――― ✀ ――――――――――――――――――――――――――――――――――――― | |
| 23:50:03 | sean-k-mooney | stderr: | |
| 23:50:05 | sean-k-mooney | Current system CPU time: 5month 4w 4h 23min 16.380000s | |
| 23:50:07 | sean-k-mooney | Current memory usage: 34.6G | |
| 23:50:17 | sean-k-mooney | Current number of tasks: 681 | |
| 23:50:17 | sean-k-mooney | kernel.pid_max: 40960 | |
| 23:50:17 | sean-k-mooney | kernel.threads-max: 1030309 | |
| 23:50:17 | sean-k-mooney | Limit of tasks: 40959 | |
| 23:50:18 | sean-k-mooney | Reducing limit by one to 40958… | |
| 23:50:18 | clarkb | Iguess a failed build is still prgress | |
| 23:50:19 | sean-k-mooney | procfs_tasks_set_limit: Permission denied | |
| 23:50:21 | sean-k-mooney | Assertion 'r >= 0 ? w == v - 1 : w == v' failed at ../src/test/test-procfs-util.c:59, function main(). Aborting. | |
| 23:50:23 | sean-k-mooney | ―――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――― | |
| 23:50:44 | sean-k-mooney | ya not really sure why that failed to be honest | |
| 23:50:59 | sean-k-mooney | there si one failrue out of aorund 300 build tests | |
| 23:51:09 | sean-k-mooney | so its morst fine | |
| 23:51:24 | sean-k-mooney | the build logs are here https://koji.mbox.centos.org/koji/taskinfo?taskID=334490 | |
| 23:51:29 | sean-k-mooney | in case your intersted | |
| 23:52:23 | sean-k-mooney | hopefully its just a buggy test and a rebuild wil fix it but in anycase hopefuly it will get adressed soon | |
| 23:53:35 | sean-k-mooney | im pretty sure i do not have an account that can retriger that on that koji instance so ill just have to wait and see but i can link the failed build on the bugzilla bug | |
| 23:54:14 | clarkb | ya I'm not really in a hurry myself more just trying to follow along since we get semi regular question about it. Though those have died down recently. I think people just know ping is broken now | |
| 23:58:55 | sean-k-mooney | ack https://bugzilla.redhat.com/show_bug.cgi?id=2037807#c10 comment on the bug. ill check it again tomorow but at least peopel are aware of the issue | |
| #openstack-nova - 2022-01-20 | |||
| 11:53:57 | gibi | anyone with placement insights do you agree my conclusion in the bug https://storyboard.openstack.org/#!/story/2009795 ? | |
| 11:58:47 | sean-k-mooney | gibi: ill read over it now | |
| 11:59:52 | gibi | sean-k-mooney: thanks | |
| 12:01:04 | gibi | I've just added one more link to the end of the ticket supporting that it is doc bug | |
| 12:01:06 | sean-k-mooney | oh this is the resouce less traits issue | |
| 12:01:56 | sean-k-mooney | we talked about this before with regrads to neted resource providres and numa i think | |
| 12:01:57 | gibi | yes, the nested magic spec class these traits as provider traits compared to resource traits that are tight to a specific resource class | |
| 12:02:24 | gibi | yes the nested magic spec captures some of that discussion I believe | |
| 12:03:19 | sean-k-mooney | we only have one type of traits in plamcent | |
| 12:03:28 | sean-k-mooney | we dont make a distinciton in the api today | |
| 12:03:31 | gibi | yes | |
| 12:03:35 | sean-k-mooney | af far as i know | |
| 12:03:45 | gibi | the distinction is just logical not implemented | |
| 12:04:06 | sean-k-mooney | right im assserting there is not distinciton and this is a bug not a docs bug | |
| 12:04:33 | sean-k-mooney | but ill re read teh nested magic spec now | |
| 12:07:05 | sean-k-mooney | im really not sure i agree with https://docs.openstack.org/placement/latest/specs/train/implemented/2005575-nested-magic-1.html#resource-versus-provider-traits | |
| 12:07:44 | sean-k-mooney | i do not belive we can assume all custom traits on a resouceless rp are provider traits | |
| 12:15:51 | sean-k-mooney | gibi: so https://docs.openstack.org/placement/latest/specs/train/implemented/2005575-nested-magic-1.html#why-enforce-resourceless-same-subtree does not apply to https://storyboard.openstack.org/#!/story/2009795 since its not a request for a resoucless request group | |
| 12:16:24 | sean-k-mooney | the unnamed group is speical and unlike all other groups it does not require the resouce in it to come form the same RP | |
| 12:16:38 | sean-k-mooney | every named request group does | |
| 12:21:07 | gibi | sean-k-mooney: so you are on the side that an required trait in an unnamed group can come from any RP of the tree? | |
| 12:21:19 | sean-k-mooney | yes stongly so | |
| 12:21:21 | gibi | the code today only considers RPs that are providing resources to the request | |
| 12:21:59 | sean-k-mooney | we do not use root_requires today | |
| 12:22:18 | gibi | I'm reading https://docs.openstack.org/placement/latest/user/provider-tree.html | |
| 12:22:21 | gibi | it seems to be relevant too | |
| 12:22:42 | sean-k-mooney | so if we moved all the resouces form the root rp to numa nodes but did not move the capablity traits they woudl stop working | |
| 12:24:22 | gibi | either we need root_required or we need a resource on the root rp | |
| 12:24:34 | gibi | * resource on the root rp that is requested | |
| 12:24:47 | sean-k-mooney | or the unnamed group is allowed to match it | |
| 12:25:02 | sean-k-mooney | but ya we would have to mvoe to root_reqiured which is an upgrade issue | |
| 12:25:16 | sean-k-mooney | as we would have to make 2 queires and combine them | |
| 12:25:22 | sean-k-mooney | to support rooling upgrades | |
| 12:26:23 | gibi | "Traits can be requested explicitly in the GET /allocation_candidates operation with the required query parameter, but traits on resource providers never span other resource providers. If a trait is requested, one of the resource providers that appears in the allocation candidate should have the trait regardless of sharing or nested providers." | |