Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-10-01
16:32:29 mriedem though stephen had comments
16:33:27 mriedem unrelated, while writing a resize reschedule test i realize that if we fail the resize claim we temporarily put the instance into ERROR state while rescheduling, and it will end up that way if rescheduling fails, which seems wrong b/c we didn't actually do anything with the guest, we just failed a claim
16:33:31 mriedem dansmith: agree? ^
16:34:15 dansmith agree with what? that leaving it in error state is wrong?
16:34:19 mriedem yeah
16:34:33 mriedem we should just fail the resize, mark the migration as error, inject a fault, etc
16:34:38 mriedem but the vm_state should still be active or stopped
16:35:50 dansmith I dunno, I guess I thought it was always a design goal that ERROR is your indication that something failed, and you start or stop the instance to get it back to normal state
16:36:26 mriedem that's a pretty big hammer for a user when resize fails b/c they lost a claim race
16:36:30 mriedem and the guest is otherwise untouched
16:36:38 mriedem the instance action will tell you as a user if the thing failed
16:36:42 dansmith isn't that why we can go from ACTIVE->ERROR->ACTIVE and the user can do the start operation from ERROR state?
16:36:57 gibi mriedem: ack, I will check back to the nova-next patch tomorrow
16:37:05 mriedem you can't start from error state
16:37:09 mriedem you can hard reboot
16:37:15 dansmith yeah, but you have to either poll it or something to know when to look right?
16:37:40 mriedem sure, poll until the task_state is reset to None
16:37:50 mriedem if the status is not what you expect, investigate the instance action
16:38:13 dansmith er, I really thought you could start to get out of error state
16:38:27 dansmith I had this long conversation with vish about it eons ago
16:38:36 dansmith although admittedly lots could have changed since then
16:38:42 mriedem hard reboot would do it but then you've hard to hard reboot your guest even though there is nothing wrong with it
16:38:48 mriedem *had to
16:38:58 dansmith no, I mean start or stop being a no-op from whatever real state it was in
16:39:06 efried mriedem: RC2 candidate, merged in master, wouldyouplease https://review.opendev.org/#/c/685742/
16:39:48 dansmith mriedem: well, stop works from error at least: https://github.com/openstack/nova/blob/master/nova/compute/api.py#L2401
16:40:00 efried and yes, on https://review.opendev.org/#/c/669879/ I'm waiting for stephenfin's questions to be answered.
16:40:44 mriedem dansmith: ok, but still, the user shouldn't have to stop and then start to get out of that, or hard reboot, when the thing that failed is a resize claim race
16:41:02 dansmith mriedem: so maybe it's just stop I'm thinking of.. anyway, I dunno.. it's very annoying as a user to do something, come back later and have it not obvious that the thing has happened, or failed or whatever
16:41:51 dansmith mriedem: if you're going to retry the operation for them, I agree. if you're not, then being super obvious about what has happened is best, IMHO
16:42:29 aspiers sean-k-mooney: got a response from AMD SEV guy
16:42:51 sean-k-mooney cool do we need to take acount of the qemu channels?
16:43:07 aspiers He didn't say so I'm following up explicitly on that
16:43:10 aspiers See my comment
16:43:25 aspiers https://review.opendev.org/#/c/684825/4/nova/virt/libvirt/designer.py@214
16:44:50 sean-k-mooney ok ya so we are missing the gpu supprot in the video model section
16:45:04 aspiers I'll add that now
16:45:38 sean-k-mooney and i think we do need the channels too since the channesl allow you to create virtio-serial deivces
16:46:15 sean-k-mooney a qemu channel has two parts a host backend and a guest front end
16:46:29 sean-k-mooney the guest front end is typeically a virtio-serial port
16:48:29 aspiers oops, the attribute for GuestVideo is 'type' not 'model'
16:48:59 sean-k-mooney we call it model in the image property
16:49:05 sean-k-mooney but maybe
16:49:39 sean-k-mooney oh its the type atribute in the model elemnt
16:49:41 aspiers yeah
16:50:00 sean-k-mooney that does not mean the class varible is called type
16:50:21 aspiers no but it is anyway
16:50:46 sean-k-mooney ya it is type https://github.com/openstack/nova/blob/master/nova/virt/libvirt/config.py#L1859
16:50:46 aspiers which IMHO is a bad choice because type is kind of reserved, but whatever
16:51:05 sean-k-mooney well its self.type
16:51:21 sean-k-mooney type as a variable would alisa the type fucntion which would be bad
16:51:25 aspiers sure
16:51:38 sean-k-mooney in class scope its kindo fo ok
16:51:43 aspiers emacs Python mode is too stupid to realise the difference
16:52:32 sean-k-mooney ya i can see that messing with alot of regex based syntax highlighting
16:53:12 sean-k-mooney so ya the guest agent is defulting to a virtio channel
16:53:14 sean-k-mooney https://github.com/openstack/nova/blob/master/nova/virt/libvirt/config.py#L2135-L2152
16:54:13 aspiers sean-k-mooney: vvvvvvv
16:54:20 openstackgerrit Adam Spiers proposed openstack/nova master: Also enable iommu for virtio controllers and video in libvirt https://review.opendev.org/684825
16:54:44 aspiers It may need another PS for virtio channels, waiting to see what Brijesh says
16:54:56 sean-k-mooney ya i would keep that seperate
16:55:09 aspiers OK
16:55:28 sean-k-mooney although you proably could jsut try enableing the qemu guest agent too and see if it works
16:55:37 aspiers how do I do that?
16:55:54 sean-k-mooney you need to install it in the guest image and set an image property let me get it
16:56:06 aspiers urgh
16:56:21 aspiers by install, do you mean at build-time not post launch?
16:56:45 sean-k-mooney you can do it post launch i think
16:57:01 sean-k-mooney you set hw_qemu_guest_agent=yes in the image metadata
16:57:23 sean-k-mooney that will be enough to triger the generation of the qemu channel
16:57:38 aspiers ok I'll try that thanks
16:57:47 sean-k-mooney then you should be able to ssh in and install it
16:57:48 aspiers I wonder what the package will be called
16:58:01 sean-k-mooney on suse i have no idea
16:58:08 aspiers what is it on RH?
16:58:09 sean-k-mooney proably qemu-somthing
16:58:13 aspiers haha no shit :)
16:58:18 sean-k-mooney i dont know
16:58:28 sean-k-mooney i have only ever installed it on ubunutu
16:58:54 sean-k-mooney yum install qemu-guest-agent
16:59:10 sean-k-mooney https://wiki.libvirt.org/page/Qemu_guest_agent
17:03:10 mnaser mriedem: did the cross-cell resizes end up finishing in train?
17:03:24 aspiers sean-k-mooney: it's the same on SUSE
17:03:27 mnaser looks like it just all still needs review and probably isnt landing in train, from what i see?
17:03:40 sean-k-mooney aspiers: its a pretty good name for it
17:03:54 aspiers sean-k-mooney: very catchy ;-)
17:04:25 sean-k-mooney aspiers: the main reason it exists is to provide a way to quiesce the diesk when doing an online snapshot
17:04:43 sean-k-mooney it has other uses too
17:04:58 aspiers sean-k-mooney: does it do more than just a sync? since stuff could still be writing
17:05:39 sean-k-mooney i think it does a flush then prevents any addtional wirtes untill the snapshot completes
17:05:55 aspiers ah ok
17:06:04 aspiers how does it prevent? some kernel voodoo?
17:06:22 sean-k-mooney aspiers: dansmith told me it was important to not currupt things and i took him at his word
17:07:31 sean-k-mooney i havent really looked in to how it does it but i assume its acting as a broker between qemu and the guest kernel block deice layer in some way
17:12:26 sean-k-mooney this covers part of it https://www.sebastien-han.fr/blog/2015/02/09/openstack-perform-consistent-snapshots-with-qemu-guest-agent/
17:13:38 openstackgerrit Merged openstack/nova master: api: Remove 'Debug' middleware https://review.opendev.org/662506
17:15:16 mriedem mnaser: not done in train no
17:15:26 sean-k-mooney aspiers: apparently you have to write your own freeze hooks
17:15:26 mriedem mnaser: do you have multiple cells yet?
17:30:21 mnaser mriedem: nope not yet

Earlier   Later