Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2021-09-29
16:44:46 gibi can we add a flag to the delete api saying I-want-a-shutdown-first?
16:45:00 dansmith and you wouldn't do that while waiting for start->shutdown on a physical server, so they should wait for vm_state=STOPPED before delete
16:45:24 dansmith gibi: that makes it a little less obscure, but doesn't eliminate the need to make that orchestration bit work, of course
16:45:37 bauzas yeah
16:45:39 lyarwood artom: there main issue with stop was that it eventually destroys the instance right?
16:45:47 bauzas that doesn't solve the ordering problem
16:46:07 lyarwood artom: and in your change you've suggested that we call shutdown on the domain as an initial step to make this more graceful
16:46:35 dansmith lyarwood: can we call quiesce or something else constant-time before libvirt destroy?
16:46:43 dansmith instead of shutdown, which the guest can block or ignore?
16:47:31 bauzas the root problem is the I/O flushes, right?
16:47:40 dansmith novaclient has a --poll option for some things. "nova stop server --poll; nova delete server" would solve this pretty easy :)
16:49:01 melwitt they're using tripleo/heat but I'm pretty sure heat has dependency or waiting ability
16:49:09 dansmith surely
16:49:33 lyarwood dansmith: quiesce before $domain.shutdown() might help flush things if that's where the guestOS is getting hung up
16:49:52 melwitt their argument has been that stop + insta delete should not break the volume, IIUC
16:50:00 lyarwood waiting assumes we don't kill it before it's finished shutting down
16:50:14 dansmith lyarwood: I meant to make sure journal buffers are written before we nuke the guest
16:50:36 dansmith melwitt: that argument is fine as long as you wait between stop and delete :)
16:51:01 dansmith melwitt: because again, start->shutdown and then pulling the plug before it finishes yields a corrupted disk :)
16:51:21 melwitt dansmith: that was the first thing I said on the bug report but it's gone on for a long time now and gotten into the weeds
16:52:04 bauzas melwitt: I haven't seen any bug report against artom's patch
16:52:12 bauzas I guess you're talking internally
16:52:19 melwitt bauzas: yes internally
16:52:42 bauzas yeah, because that's what worried me originally
16:52:51 bauzas technically, destroy works like expected
16:52:52 dansmith we also really need to do a better job of making this a sanitized bug externally if we're going to claim this is a bug in nova
16:53:25 bauzas I'm not happy with claiming this as an upstream bug
16:53:37 dansmith me either, fwiw :)
16:53:40 bauzas a blueprint or a wishlist bug
16:54:08 dansmith we could add a feature as gibi said, but destroy is doing the right thing here
16:54:21 bauzas (18:52:51) bauzas: technically, destroy works like expected
16:54:28 melwitt +1 to all of that
16:54:30 dansmith an alternative to gibi's idea, would be a delete flag that says "assuming task_state=None" meaning "delete this if nothing else is going on"
16:54:30 bauzas strong agreement here
16:54:47 dansmith but it would require the client to retry, which they could currently do by just waiting for the stop to finish
16:54:50 dansmith so, meh
16:55:22 bauzas dansmith: yeah, that's why I was considering the vm state or the task state
16:55:38 bauzas if we really want to do *something*
16:55:39 bauzas but,
16:55:49 bauzas this can't be done with the current destroy API
16:55:52 dansmith it doesn't make it do what they want, and honestly it's kinda weird since they could just wait for the stop just as well, but it's less new orchestration stuff
16:56:09 dansmith [09:47:37] <dansmith> novaclient has a --poll option for some things. "nova stop server --poll; nova delete server" would solve this pretty easy :)
16:56:11 dansmith ^ :)
16:56:12 bauzas it would be a 'destroy++" API
16:56:34 bauzas heh
16:56:43 dansmith also, if melwitt is right and they're using heat, then FFS, get heat to wait or something
16:56:43 bauzas (18:44:41) bauzas: "ask your orchestration to be smarter ?"
16:56:47 dansmith lol
16:56:53 lyarwood we keep saying they could wait for the stop to finish but wasn't that part of the issue here? Even if they did wait the libvirt driver would kill the instance prematurely before it had finished shutting down?
16:57:01 dansmith if we're just re-quoting ourselves, are we done here? :P
16:57:19 gibi if currently stop + wait for STOPPED + delete works, then a new delete-with-gracefull-shutdown could also be orchestrated from the conductor but it is obviously an orchestration and adds complexity to the already complext delete codepaths
16:57:42 bauzas gibi: and again, this can't be the straight delete
16:57:53 dansmith gibi: we don't go through conductor directly for stop or delete right now, AFAIK, so it would really get confusing to add another whole path like that, IMHO
16:57:54 gibi bauzas: yepp this is delete++ :D
16:58:25 gibi dansmith: then we either need the make the stop RCP sync or do a polling from the nova-api
16:58:33 gibi RPC
16:58:35 melwitt lyarwood: I didn't think so?
16:58:53 bauzas can't we just have as dansmith suggested a "destroy++" API return some 40x when the task state is not None ?
16:59:15 lyarwood that's fine if they aren't waiting
16:59:18 melwitt if that's the case I missed it
16:59:21 dansmith gibi: we'd need a cast to conductor, a new method there, and a sync stop operation to compute from conductor followed by delete.. waiting from the api is not reasonable because stop can take a *long* time
16:59:35 gibi yeah
16:59:41 lyarwood melwitt: I was sure we had talked about them waiting and libvirt still killing the domain before things had shutdown sorry
16:59:45 lyarwood I'll look at the bug again
16:59:47 dansmith in addition to the other four ways we can delete things :)
16:59:48 melwitt lyarwood: my understanding (and I could be wrong) is that they have never tried waiting to stopped
17:00:00 lyarwood I was sure we suggested that early on
17:00:01 dansmith melwitt: right that seems like it to me
17:00:03 melwitt lyarwood: ok same, maybe I'm way off
17:00:07 bauzas I'd rather prefer this destroy++ API to stop synchronously at the API level if some conditions aren't met
17:00:36 bauzas rather than us pursuing the idea we could achieve some distributed locking mechanism
17:00:56 dansmith gibi: also delete returns "I will eventually do this" to the client, which it would no longer be able to guarantee
17:01:32 dansmith bauzas: fwiw, I don't think that is really going to do what this person wants, so we should make sure they think it would help before we do that work
17:01:37 gibi dansmith: ohh, that is correct and bad :/
17:02:05 gibi delete++ is just too complex
17:02:08 bauzas dansmith: welp, good point
17:07:07 bauzas either way, I think we need a proper tracking
17:07:41 bauzas artom: I guess you need to fill a blueprint and honestly given the brainstorm efforts we made over last hour, you need to write a bit of a spec
17:08:16 bauzas mostly because the current destroy action can't be used for this and we need to consider a new action parameter (or any change in our API)
17:08:41 bauzas artom: feel also free to write a PTG proposal for this one, we could continue the talk there
17:57:36 spatel Any idea how to do VM nic bonding using two VF with SRIOV ?
17:58:39 spatel i am looking for redundancy with sriov implementation and only solution is to do bonding inside vm
18:04:32 opendevreview Ivan Kolodyazhny proposed openstack/nova master: Add release note which descrube NVMe attach issue https://review.opendev.org/c/openstack/nova/+/811447
19:15:02 opendevreview melanie witt proposed openstack/nova stable/train: Reject open redirection in the console proxy https://review.opendev.org/c/openstack/nova/+/791807
19:15:03 opendevreview melanie witt proposed openstack/nova stable/train: address open redirect with 3 forward slashes https://review.opendev.org/c/openstack/nova/+/806629
19:15:04 opendevreview melanie witt proposed openstack/nova stable/train: [stable-only] Set lower-constraints job as non-voting https://review.opendev.org/c/openstack/nova/+/811762
21:59:20 opendevreview David Vallee Delisle proposed openstack/nova master: rephrasing config description for num_pcie_ports in libvirt https://review.opendev.org/c/openstack/nova/+/811173
22:19:17 opendevreview Goutham Pacha Ravi proposed openstack/nova stable/wallaby: DNM: Test wallaby with ceph pacific https://review.opendev.org/c/openstack/nova/+/811802
22:33:16 melwitt gouthamr: thanks for that! ^ I was going to do it and forgot
22:34:56 gouthamr o/ melwitt - i may have hit a circular dependency of sorts
22:35:45 melwitt oh hm.. (looking at it now)
22:36:54 melwitt gouthamr: I wonder if you need to use the url instead of the change-id? not sure how it would know which of the two (master or stable/wallaby) to use
22:37:02 melwitt for the depends-on
22:37:05 gouthamr +1 can try that
22:38:04 opendevreview Goutham Pacha Ravi proposed openstack/nova stable/wallaby: DNM: Test wallaby with ceph pacific https://review.opendev.org/c/openstack/nova/+/811802
22:38:22 clarkb the url is prefered now because you can depends on things in other code review systems
22:38:26 clarkb change id is gerrit specific
22:39:11 gmann gouthamr: i do not see circular deps, it should work fine. nova->plugin->devstack
22:39:19 melwitt good to know
22:40:32 gouthamr thanks, but zuul stull tells me "Unable to freeze job graph: 0"

Earlier   Later