Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2021-09-29
14:22:19 bauzas gmann: nah, no rush
14:34:38 lyarwood https://review.opendev.org/c/openstack/nova/+/811118/ is passing with just the master grenade fix that's in the gate FWIW
14:35:40 gibi \o/
14:52:19 bauzas lyarwood: https://media.giphy.com/media/HTjcWZwMtHpyhuCGKZ/giphy-downsized-large.gif?cid=ecf05e47cwkexzrwaxwlkbicd2ppy22bb1l52sg7yoa8sk9x&rid=giphy-downsized-large.gif&ct=g
14:52:49 bauzas but that's lasting 0.1sec and then I say "meh"
14:53:58 fungi if you used a matrix client with the matrix-oftc bridge to join this channel, you and other matrix users could share inline media while irc users would just see a url to it in-channel
15:04:32 bauzas fungi: what I said, "meh" :p
15:05:24 fungi heh
15:05:30 gmann lyarwood: perfect then I will say it was late night thing which thought that Xena fix is needed :)
15:07:17 gmann bauzas: for PTG oslo sessions, is it possible on Tuesday Oct 19th between 13-15 UTC
15:14:18 opendevreview melanie witt proposed openstack/nova stable/train: Reject open redirection in the console proxy https://review.opendev.org/c/openstack/nova/+/791807
15:14:19 opendevreview melanie witt proposed openstack/nova stable/train: address open redirect with 3 forward slashes https://review.opendev.org/c/openstack/nova/+/806629
15:19:44 gmann bauzas: and as per current topic I need only 30 min unless there is more things to discuss from anyone else
15:20:34 bauzas gmann: sorry, I'm on a meeting but I guess we could run a meeting after 1400UTC as we have a cyborg x-p session first
15:22:10 gmann bauzas: on tuesday right?
15:22:35 bauzas gmann: on the 19th of October, yes
15:22:56 gmann bauzas: perfect, sounds good. thanks
15:23:10 bauzas https://etherpad.opendev.org/p/nova-yoga-ptg L47 tells me the 14-15:00UTC slot is already taken
15:24:12 gmann bauzas: that is 'Cyborg-Nova: Tuesday (19th Oct) 13:00 UTC - 14: 00 UTC:'
15:24:24 bauzas my bad, yeah
15:24:44 bauzas again, wfm for a oslo x-p session on Oct-19 14:00UTC
15:24:53 gmann +1, thanks again
15:24:58 lyarwood gmann: ^_^ no issues thanks for working on it so late
15:25:04 bauzas gmann: all good
15:25:36 gmann lyarwood: np!, and for stable backport we can wait for neutron team opinion so I agree on 'no hurry for those' .
15:25:45 lyarwood awesome
16:12:53 bauzas artom: honestly, I'm torn with https://review.opendev.org/c/openstack/nova/+/808474
16:13:00 bauzas that's an behavioural change
16:13:10 opendevreview Balazs Gibizer proposed openstack/nova master: Enable min pps tempest testing in nova-next https://review.opendev.org/c/openstack/nova/+/811748
16:13:18 bauzas operators suppose a plugoff when delete
16:13:38 bauzas now, we'll first try to shutdown the guest for every instance
16:13:46 bauzas including other drivers but libvirt
16:14:20 melwitt that is my concern as well. maybe it could be conditional on bfv that is not "delete on termination"?
16:15:18 melwitt is that the only case where this would be desired?
16:15:22 melwitt or rather, is there any gain in doing it for non bfv non delete on termination?
16:15:58 dansmith generally you don't want that :)
16:16:07 melwitt maybe also shared storage is another case
16:16:08 dansmith the only real case for non-bfv volumes is for precious data
16:16:53 melwitt but precious data on a local disk that's going to be deleted anyway? I must be missing something
16:17:15 dansmith sorry, thought you were talking about delete-on-termination for non-bfv cinder volumes
16:17:23 artom melwitt, "is there any gain in doing it for non bfv non delete on termination?" None that I can see
16:17:33 artom Well, no, any volume, really
16:17:41 artom Doesn't have to be bfv
16:17:50 melwitt oh, yeah attached volumes. I wasn't thinking of that. yeah
16:18:02 dansmith that's my point, delete-on-termination should only be useful for bfv volumes we created with non-precious data from an image
16:18:05 artom It's still attached and mounted in the guest, and would ideally be flushed correctly if it's not delete_on_termination=True
16:18:38 artom I need to run an errand quickly, can this be carried over to the gerrit review?
16:18:44 artom And thanks for looking into it :)
16:19:22 artom And yeah, so bauzas's point, the compute manager/driver division of labour here is pretty muddy
16:19:32 melwitt my bad for not looking at the change yet. if it's targeted to only instance with volume(s) cases I think that makes a lot more sense
16:19:38 gibi is it really a gracefull shutdown via openstack server stop and then a openstack server delete?
16:19:56 dansmith gibi: stop + delete should be graceful
16:20:20 gibi so my point is this can already done with our APIs
16:20:23 melwitt just saying I don't think we should be doing it for everything, for things where the data is going to be blown away anyway
16:21:01 dansmith gibi: for sure. I assume the goal is to make nova do the graceful behavior if volumes are attached, but to do it properly really requires some higher-level orch, like a stop...timeout...destroy kind of thing
16:21:24 dansmith "do the graceful behavior *automatically*" I should have said
16:21:36 gibi OK I see
16:21:49 melwitt yeah that is my understanding as well
16:22:10 dansmith I'm a bit torn, because unless you're running with unsafe cache, I would think that fast destroy is fine.. might have a journal to replay when you use the volume later, but...
16:22:11 gibi it make sense for data consistency but it also makes delete slower so I think this should be opt in
16:23:16 melwitt dansmith: yeah it's weird, the user is experiencing volume gets corrupted and no longer usable when they delete without stopping first
16:23:23 melwitt we had thought just deleting should be fine but it's behaving in a way we didn't expect
16:23:34 melwitt not sure why
16:23:37 dansmith destroy of a running vm is the same as pulling the plug.. if you're using a precious volume, you wouldn't do that to a physical server, so...
16:24:27 artom dansmith, so the "real" problem is https://bugzilla.redhat.com/show_bug.cgi?id=1965081
16:24:29 melwitt yeah but re: "I would think that fast destroy is fine"?
16:24:39 artom Apparently in some cases stop+delete causes races
16:25:01 melwitt yeah I think you have to poll and wait for it to be stopped no?
16:25:32 dansmith melwitt: to do the graceful shutdown, you'd need some long-running task, yeah, what I said above
16:25:38 artom So we can either make delete safer, or require that any orchestration/automation on top of Nova does stop + delete, but then we would need to fix that race
16:25:41 dansmith artom: so this has nothing to do with volume safety?
16:26:10 artom dansmith, it does, because the reason for doing stop + delete (which can cause this deadlock) is volume safety
16:26:18 melwitt it does. it began with the racing problem and then we said "try a delete without the stop" and then their volumes got messed up
16:26:38 dansmith oh the dbdeadlock came from stop?
16:26:54 lyarwood doesn't delete take an instance.uuid lock on the compute?
16:26:57 artom dansmith, stop immediately followed by delete, apparently
16:27:18 melwitt it came from doing a delete right after a stop
16:27:20 melwitt without waiting for the stop to be stopped
16:27:24 melwitt lyarwood: good question
16:27:25 dansmith okay, so, fix that
16:27:42 dansmith don't engineer an orchestrated graceful delete, IMHO
16:27:59 dansmith I'm guessing task_state doesn't protect the delete from running,
16:28:12 artom I dunno, I don't necessarily think expecting an attached volume to not be corrupted after a delete is invalid
16:28:18 dansmith and the api might start the delete process while the stop is still running on the compute host or something
16:28:29 melwitt hm, stop and delete are both locked with instance.uuid
16:28:46 melwitt I just checked
16:28:46 dansmith artom: I agree, delete could leave a volume unhappy, but I'm thinking if you issued a stop and then delete you're assuming they're queued
16:28:47 lyarwood https://github.com/openstack/nova/blob/e07bb310b674fb471a92edf3258e564f05534595/nova/compute/manager.py#L3237-L3255 looks like soft delete doesnt
16:29:38 dansmith artom: in reality it's probably about like hitting shutdown on your server and then pulling the plug before it finishes, but...
16:29:42 melwitt dansmith: that's exactly what they want, the queuing
16:30:21 dansmith I think relying on the instance lock is probably too fragile here,
16:30:35 dansmith since it's just on the compute node, but I'd have to go look at the (many) delete path(s) we have
16:31:16 lyarwood I wouldn't say so for this case where the instance was running on a host
16:31:25 dansmith delete is and always has been pretty much "I want this to complete and stop charging me immediately", so... we're really not wrong here, IMHO
16:31:28 lyarwood that makes perfect sense
16:31:30 opendevreview Balazs Gibizer proposed openstack/nova master: Enable min pps tempest testing in nova-next https://review.opendev.org/c/openstack/nova/+/811748
16:32:26 dansmith lyarwood: wouldn't say what, that relying on the lock is unsafe? we can do stuff in the api to delete things (in the local case) which has nothing to do with the instance lock,
16:32:46 dansmith and I'm not sure that relying on the ordering of two calls is really safe either unless we have perfectly fair locks
16:32:50 bauzas sorry, I was afk
16:33:13 lyarwood dansmith: we can't force the delete in the API of an instance still associated with a host can we?

Earlier   Later