| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-01-19 | |||
| 17:43:12 | sean-k-mooney | so in general we do not support it | |
| 17:43:19 | spatel | we have 800TB of glusterfs and it would be bad if i mount that on foo server and then mount on 50 compute nodes | |
| 17:44:07 | spatel | That is interesting... | |
| 17:44:12 | sean-k-mooney | in generall if you want ot use glusterfs with openstack the only supporte way to do it would via cinder | |
| 17:44:37 | sean-k-mooney | it can work the way you have it deplyed but its not a tested configuration and was never expiclty supported | |
| 17:46:06 | spatel | may be i am the first use case here.. :) i would still like to push it and try out because we already have large storage which i would like to use. | |
| 17:46:33 | spatel | if i see any issue (major issue) then i may switch to NFS | |
| 17:47:20 | spatel | If i go with cinder then its not going to be shared storage correct? does live migration will be supported with cinder base deployment? | |
| 17:48:07 | sean-k-mooney | spatel: well jsut a word of warning if we could remove support for this capablity entirly we would | |
| 17:48:25 | sean-k-mooney | we dont like support nfs backed /var/lib/nova | |
| 17:48:51 | sean-k-mooney | we only maintian supprot becasue we have some old large clouds that use it but this area fo the code is not activly maintained | |
| 17:49:03 | spatel | hmm! so in short no to shared storage except ceph. correct? | |
| 17:49:24 | sean-k-mooney | basically and even for ceph not by using cephfs | |
| 17:49:42 | spatel | ceph rbd | |
| 17:49:49 | sean-k-mooney | the probelm is that when using shared storate like this we are actully using a local storage driver in nova | |
| 17:50:06 | sean-k-mooney | e.g. image_type=qcow | |
| 17:50:07 | spatel | i know what you saying.. mounting nova on shared storage can have all kind of issue | |
| 17:50:31 | sean-k-mooney | and have sprincked some check in random parts of the code to do something else fi we detect its on shared storage | |
| 17:50:59 | sean-k-mooney | the main one come up aroudn move operations and evacuate | |
| 17:51:16 | sean-k-mooney | e.g. we need to make sure we dont overrie the image if we do a cold/live migration | |
| 17:51:36 | sean-k-mooney | and dont recreate it if we evacuate | |
| 17:51:42 | sean-k-mooney | to prevent loosing data | |
| 17:51:56 | spatel | hmm | |
| 17:52:22 | spatel | so what option i have in current design with glusterfs ? | |
| 17:52:36 | spatel | cinder boot volum? | |
| 17:52:43 | sean-k-mooney | if the sync time between writing a file on one host and reading it on another is long enough we might not detect its on shared storage whchi can be bad | |
| 17:52:58 | sean-k-mooney | cinder boot form vomue woudl work yes | |
| 17:53:19 | spatel | does cinder support glusterfs ? | |
| 17:53:23 | sean-k-mooney | what you are doign can work just be aware that there are dragons wehn you do this. | |
| 17:53:26 | sean-k-mooney | am it used too | |
| 17:54:08 | spatel | I understand this cluster not going to hold critical data. this is for crunch some data and give dynamic result | |
| 17:54:37 | sean-k-mooney | am its avaible for cinder backup | |
| 17:54:38 | sean-k-mooney | https://docs.openstack.org/cinder/latest/drivers.html#glusterfsbackupdriver | |
| 17:54:44 | sean-k-mooney | not sure about cidner in general | |
| 17:55:42 | spatel | let me do some research and see what i can do | |
| 17:56:21 | sean-k-mooney | in generall it looks like no | |
| 17:56:30 | spatel | oh boy :( | |
| 17:56:39 | sean-k-mooney | we do not have a volume driver in nova for gluster | |
| 17:56:41 | sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L176-L191 | |
| 17:56:51 | sean-k-mooney | so it can be used for backup bug not vms | |
| 17:57:25 | spatel | I think i should try NFS now because folks are using it.. | |
| 17:57:45 | spatel | i know you hate but i need something to move forward and deal with issue later. :) | |
| 17:58:22 | spatel | we have plan to buy dedicated ceph storage but that is not today but after few months | |
| 17:59:05 | sean-k-mooney | ack the way we detech shared storage https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L10893-L10912 | |
| 17:59:15 | sean-k-mooney | in principal shoudl work with gluster | |
| 17:59:29 | sean-k-mooney | but im not sure why it woudl not delete the instace directory | |
| 18:00:06 | sean-k-mooney | althogh i dont think that is the only way we check | |
| 18:00:54 | sean-k-mooney | spatel: we have things like this | |
| 18:00:56 | sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L1723-L1731 | |
| 18:01:38 | spatel | sean-k-mooney does nova has any time etc.. where it wait and delete? | |
| 18:01:55 | spatel | I am planning to turn on debug to see what is going on ? | |
| 18:02:01 | sean-k-mooney | well i think part of the delete wil be done by libvirt | |
| 18:02:38 | sean-k-mooney | this si where we do some o fthe lean up https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L1502 | |
| 18:02:54 | sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L11641-L11697 | |
| 18:02:55 | spatel | hmm.. is there any special config which i may missed | |
| 18:03:48 | sean-k-mooney | look for LOG.info('Deletion of %s failed', remaining_path, | |
| 18:03:50 | sean-k-mooney | instance=instance) | |
| 18:06:38 | sean-k-mooney | spatel: you could try https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L10619-L10647 | |
| 18:06:54 | sean-k-mooney | its contoled by instance_delete_interval | |
| 18:07:08 | sean-k-mooney | https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.instance_delete_interval | |
| 18:08:18 | sean-k-mooney | spatel: so nova will try to delete the instance every 5 minuts and then retry up to 5 times | |
| 18:08:26 | sean-k-mooney | https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.maximum_instance_delete_attempts | |
| 18:09:07 | spatel | let me check that setting and wait.. i can turn on debug also | |
| 18:09:31 | sean-k-mooney | spatel: check the compute agent log for that info message first | |
| 18:09:44 | sean-k-mooney | to confirm that nova failed to delete the instance files | |
| 18:11:56 | spatel | give me few minute to collect some logs | |
| 18:15:01 | sean-k-mooney | no worries | |
| 18:28:54 | spatel | sean-k-mooney as quick check i found this - https://paste.opendev.org/show/812229/ | |
| 18:29:18 | spatel | libvirt trying to delete but its getting failed.. very odd | |
| 18:29:34 | spatel | now going to turn on debug and see | |
| 18:30:05 | sean-k-mooney | ya that is the message i was expecting | |
| 18:31:38 | sean-k-mooney | so like nfs https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L11079 it look like deleteion on gluster is also unreliable | |
| 18:32:22 | spatel | https://paste.opendev.org/show/812230/ | |
| 18:33:08 | spatel | This is odd, when i tried to delete that by hand i got above error | |
| 18:33:21 | sean-k-mooney | so ya looks like a gluster issue | |
| 18:33:34 | spatel | but it deleted files inside that directory | |
| 18:33:48 | spatel | I am going to talk to someone who own that storage and see what is going on | |
| 18:33:55 | sean-k-mooney | ack | |
| 18:34:11 | spatel | Thanks for checking | |
| 18:34:12 | sean-k-mooney | im not sure how gluster tracks files vs direcoties | |
| 18:34:28 | sean-k-mooney | but it may handel directory inodes differently | |
| 18:34:53 | spatel | https://paste.opendev.org/show/812231/ | |
| 18:35:33 | spatel | something is wrong with gluster for sure.. i am able to delete files but not directory | |
| 18:35:46 | sean-k-mooney | ack | |
| 18:35:51 | sean-k-mooney | ya it could also be fuse | |
| 18:36:07 | sean-k-mooney | are you mounting glusterfs with fuse or using a kernel dirver | |
| 18:37:00 | spatel | 10.10.217.21:gluster_vol2/voyager /mnt/glusterfs glusterfs defaults,backup-volfile-servers=10.10.217.22:10.10.217.23:10.10.217.24:10.10.217.25:10.10.217.26 | |
| 18:37:22 | spatel | fuse.. hmm ? | |
| 18:37:47 | spatel | i just install glusterfs-client RPM package and use mount tool to mount it | |
| 18:37:49 | sean-k-mooney | i think there is a user space dirver and a kernel dirver for gluster like there is for ceph | |
| 18:38:02 | spatel | let me check.. that is good point | |
| 18:39:22 | sean-k-mooney | https://bugzilla.redhat.com/show_bug.cgi?id=1508999 | |
| 18:43:21 | spatel | Reading bug report, looks interesting | |
| 18:44:19 | sean-k-mooney | i have not looked at it closely but the title seamed relevent | |
| 18:44:58 | spatel | i am having same issue but trying to understand how they fix. i don't control gluster so not sure but i can explain someone | |
| 18:45:21 | sean-k-mooney | i would start with your simple repoducer | |
| 18:45:34 | sean-k-mooney | e.g. create teh dir and show you cant delete it | |
| 18:45:56 | sean-k-mooney | that really shoudl work | |
| 18:46:26 | sean-k-mooney | one thing to check is it any directoy or just ones at the root of the volume | |
| 18:47:34 | sean-k-mooney | i.e. does mkdir -p .../temp/mydata rm -rf .../temp/mydata work | |