Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2021-09-29
14:16:46 fungi and the most we can do in those cases is dump an earlier text copy with the api and stick that into a new pad, but it loses all the attribution and history
14:17:40 fungi and formatting
14:18:55 bauzas fungi: thanks, gtk
14:19:23 bauzas gmann: i know you're busy with all those devstack and grenade stuff
14:19:50 bauzas gmann: but tell me when you think it would be a good opportunity for the x-p PTG session between oslo and nova re: policy
14:22:14 gmann bauzas: give me 10 min, currently internal meeting
14:22:19 bauzas gmann: nah, no rush
14:34:38 lyarwood https://review.opendev.org/c/openstack/nova/+/811118/ is passing with just the master grenade fix that's in the gate FWIW
14:35:40 gibi \o/
14:52:19 bauzas lyarwood: https://media.giphy.com/media/HTjcWZwMtHpyhuCGKZ/giphy-downsized-large.gif?cid=ecf05e47cwkexzrwaxwlkbicd2ppy22bb1l52sg7yoa8sk9x&rid=giphy-downsized-large.gif&ct=g
14:52:49 bauzas but that's lasting 0.1sec and then I say "meh"
14:53:58 fungi if you used a matrix client with the matrix-oftc bridge to join this channel, you and other matrix users could share inline media while irc users would just see a url to it in-channel
15:04:32 bauzas fungi: what I said, "meh" :p
15:05:24 fungi heh
15:05:30 gmann lyarwood: perfect then I will say it was late night thing which thought that Xena fix is needed :)
15:07:17 gmann bauzas: for PTG oslo sessions, is it possible on Tuesday Oct 19th between 13-15 UTC
15:14:18 opendevreview melanie witt proposed openstack/nova stable/train: Reject open redirection in the console proxy https://review.opendev.org/c/openstack/nova/+/791807
15:14:19 opendevreview melanie witt proposed openstack/nova stable/train: address open redirect with 3 forward slashes https://review.opendev.org/c/openstack/nova/+/806629
15:19:44 gmann bauzas: and as per current topic I need only 30 min unless there is more things to discuss from anyone else
15:20:34 bauzas gmann: sorry, I'm on a meeting but I guess we could run a meeting after 1400UTC as we have a cyborg x-p session first
15:22:10 gmann bauzas: on tuesday right?
15:22:35 bauzas gmann: on the 19th of October, yes
15:22:56 gmann bauzas: perfect, sounds good. thanks
15:23:10 bauzas https://etherpad.opendev.org/p/nova-yoga-ptg L47 tells me the 14-15:00UTC slot is already taken
15:24:12 gmann bauzas: that is 'Cyborg-Nova: Tuesday (19th Oct) 13:00 UTC - 14: 00 UTC:'
15:24:24 bauzas my bad, yeah
15:24:44 bauzas again, wfm for a oslo x-p session on Oct-19 14:00UTC
15:24:53 gmann +1, thanks again
15:24:58 lyarwood gmann: ^_^ no issues thanks for working on it so late
15:25:04 bauzas gmann: all good
15:25:36 gmann lyarwood: np!, and for stable backport we can wait for neutron team opinion so I agree on 'no hurry for those' .
15:25:45 lyarwood awesome
16:12:53 bauzas artom: honestly, I'm torn with https://review.opendev.org/c/openstack/nova/+/808474
16:13:00 bauzas that's an behavioural change
16:13:10 opendevreview Balazs Gibizer proposed openstack/nova master: Enable min pps tempest testing in nova-next https://review.opendev.org/c/openstack/nova/+/811748
16:13:18 bauzas operators suppose a plugoff when delete
16:13:38 bauzas now, we'll first try to shutdown the guest for every instance
16:13:46 bauzas including other drivers but libvirt
16:14:20 melwitt that is my concern as well. maybe it could be conditional on bfv that is not "delete on termination"?
16:15:18 melwitt is that the only case where this would be desired?
16:15:22 melwitt or rather, is there any gain in doing it for non bfv non delete on termination?
16:15:58 dansmith generally you don't want that :)
16:16:07 melwitt maybe also shared storage is another case
16:16:08 dansmith the only real case for non-bfv volumes is for precious data
16:16:53 melwitt but precious data on a local disk that's going to be deleted anyway? I must be missing something
16:17:15 dansmith sorry, thought you were talking about delete-on-termination for non-bfv cinder volumes
16:17:23 artom melwitt, "is there any gain in doing it for non bfv non delete on termination?" None that I can see
16:17:33 artom Well, no, any volume, really
16:17:41 artom Doesn't have to be bfv
16:17:50 melwitt oh, yeah attached volumes. I wasn't thinking of that. yeah
16:18:02 dansmith that's my point, delete-on-termination should only be useful for bfv volumes we created with non-precious data from an image
16:18:05 artom It's still attached and mounted in the guest, and would ideally be flushed correctly if it's not delete_on_termination=True
16:18:38 artom I need to run an errand quickly, can this be carried over to the gerrit review?
16:18:44 artom And thanks for looking into it :)
16:19:22 artom And yeah, so bauzas's point, the compute manager/driver division of labour here is pretty muddy
16:19:32 melwitt my bad for not looking at the change yet. if it's targeted to only instance with volume(s) cases I think that makes a lot more sense
16:19:38 gibi is it really a gracefull shutdown via openstack server stop and then a openstack server delete?
16:19:56 dansmith gibi: stop + delete should be graceful
16:20:20 gibi so my point is this can already done with our APIs
16:20:23 melwitt just saying I don't think we should be doing it for everything, for things where the data is going to be blown away anyway
16:21:01 dansmith gibi: for sure. I assume the goal is to make nova do the graceful behavior if volumes are attached, but to do it properly really requires some higher-level orch, like a stop...timeout...destroy kind of thing
16:21:24 dansmith "do the graceful behavior *automatically*" I should have said
16:21:36 gibi OK I see
16:21:49 melwitt yeah that is my understanding as well
16:22:10 dansmith I'm a bit torn, because unless you're running with unsafe cache, I would think that fast destroy is fine.. might have a journal to replay when you use the volume later, but...
16:22:11 gibi it make sense for data consistency but it also makes delete slower so I think this should be opt in
16:23:16 melwitt dansmith: yeah it's weird, the user is experiencing volume gets corrupted and no longer usable when they delete without stopping first
16:23:23 melwitt we had thought just deleting should be fine but it's behaving in a way we didn't expect
16:23:34 melwitt not sure why
16:23:37 dansmith destroy of a running vm is the same as pulling the plug.. if you're using a precious volume, you wouldn't do that to a physical server, so...
16:24:27 artom dansmith, so the "real" problem is https://bugzilla.redhat.com/show_bug.cgi?id=1965081
16:24:29 melwitt yeah but re: "I would think that fast destroy is fine"?
16:24:39 artom Apparently in some cases stop+delete causes races
16:25:01 melwitt yeah I think you have to poll and wait for it to be stopped no?
16:25:32 dansmith melwitt: to do the graceful shutdown, you'd need some long-running task, yeah, what I said above
16:25:38 artom So we can either make delete safer, or require that any orchestration/automation on top of Nova does stop + delete, but then we would need to fix that race
16:25:41 dansmith artom: so this has nothing to do with volume safety?
16:26:10 artom dansmith, it does, because the reason for doing stop + delete (which can cause this deadlock) is volume safety
16:26:18 melwitt it does. it began with the racing problem and then we said "try a delete without the stop" and then their volumes got messed up
16:26:38 dansmith oh the dbdeadlock came from stop?
16:26:54 lyarwood doesn't delete take an instance.uuid lock on the compute?
16:26:57 artom dansmith, stop immediately followed by delete, apparently
16:27:18 melwitt it came from doing a delete right after a stop
16:27:20 melwitt without waiting for the stop to be stopped
16:27:24 melwitt lyarwood: good question
16:27:25 dansmith okay, so, fix that
16:27:42 dansmith don't engineer an orchestrated graceful delete, IMHO
16:27:59 dansmith I'm guessing task_state doesn't protect the delete from running,
16:28:12 artom I dunno, I don't necessarily think expecting an attached volume to not be corrupted after a delete is invalid
16:28:18 dansmith and the api might start the delete process while the stop is still running on the compute host or something
16:28:29 melwitt hm, stop and delete are both locked with instance.uuid
16:28:46 dansmith artom: I agree, delete could leave a volume unhappy, but I'm thinking if you issued a stop and then delete you're assuming they're queued
16:28:46 melwitt I just checked
16:28:47 lyarwood https://github.com/openstack/nova/blob/e07bb310b674fb471a92edf3258e564f05534595/nova/compute/manager.py#L3237-L3255 looks like soft delete doesnt
16:29:38 dansmith artom: in reality it's probably about like hitting shutdown on your server and then pulling the plug before it finishes, but...
16:29:42 melwitt dansmith: that's exactly what they want, the queuing
16:30:21 dansmith I think relying on the instance lock is probably too fragile here,
16:30:35 dansmith since it's just on the compute node, but I'd have to go look at the (many) delete path(s) we have
16:31:16 lyarwood I wouldn't say so for this case where the instance was running on a host
16:31:25 dansmith delete is and always has been pretty much "I want this to complete and stop charging me immediately", so... we're really not wrong here, IMHO

Earlier   Later