Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-02-06
17:28:24 cfriesen mriedem: what about a suspended guest?
17:29:37 mriedem cfriesen: i guess we do allow that for regular snapshots
17:30:26 mriedem i thought something was wonky with suspended instances in a volume-backed instance flow, but might be thinking of something else
17:33:54 dansmith mriedem: nice
17:37:00 mriedem cfriesen: ooo i actually had that backward
17:37:38 mriedem cfriesen: https://github.com/openstack/nova/blob/9ae5b2306b7a7cc9e77c77292256b13926920ead/nova/compute/api.py#L2805
17:37:58 mriedem we only allow active/stopped/suspended for volume-backed snapshot in the "normal" flow
17:38:03 mriedem so i'll change my patch to follow that
17:40:56 cfriesen mriedem: do we not support paused because we don't bother quiescing the disk first?
17:42:22 mriedem it does quiesce, https://github.com/openstack/nova/blob/9ae5b2306b7a7cc9e77c77292256b13926920ead/nova/virt/libvirt/driver.py#L1913
17:42:32 mriedem and if that fails, it snapshots w/o quiesce
17:42:37 mriedem so maybe paused would work too,
17:42:41 mriedem i just haven't tested it
17:43:05 mriedem i'd have to get a devstack + nfs plugin setup to test this out, which would take some time
17:43:24 dansmith someone should hit this: https://review.openstack.org/#/c/427981/
17:43:30 dansmith last doc change on the books for rc2
17:44:18 dansmith mriedem: this is on the list for rc2...should I spend time reviewing it or what? https://review.openstack.org/#/c/424428/
17:44:28 mriedem dansmith: yeah
17:44:53 cfriesen mriedem: I meant quiesce before pausing...since otherwise there could potentially be data loss in the snapshot....but maybe we'd get the same with suspend. I don't know enough about qemu behaviour in pause vs suspend
17:45:14 mriedem oh before pause,
17:45:36 mriedem https://github.com/openstack/nova/blob/9ae5b2306b7a7cc9e77c77292256b13926920ead/nova/virt/libvirt/guest.py#L566
17:46:16 openstackgerrit John Garbutt proposed openstack/nova master: Default live_migration_progress_timeout to off https://review.openstack.org/429798
17:47:06 praveen Hi mriedem , One help!
17:47:08 praveen There is an observation.. Lets suppose project quota for max number of vms is set as 5. 3 Vms are already up and running into that project. Now, when user triggers another 3 Vms at one go as multiple deployment then 2 vms gets deployed succesfully but another VM does not.. which is correct because of quota error... But there is no indication/error message which gets shown into logs or UI.....
17:47:09 praveen ...Is this the way multipe server create behaves?
17:47:11 praveen I found below lines in compute API reference wiki - Not sure if it says same thing ??
17:47:12 praveen "Error handling for multiple create is not as consistent as for single server create, and there is no guarantee that all the servers will be built. This call should generally be avoided in favor of clients doing direct individual server creates."
17:47:36 mriedem cfriesen: i always get confused between paused and suspended with qemu
17:47:48 mriedem since the code for pause in nova calls suspend on the domain
17:47:48 mriedem https://wiki.libvirt.org/page/VM_lifecycle#States_that_a_guest_domain_can_be_in
17:47:49 cfriesen mriedem: okay, so pause just immediately suspends the domain, while suspend does a managedSave on the domain.
17:48:12 johnthetubaguy mriedem: pkoniszewski: I wonder if we should look at getting the above change into RC2 https://review.openstack.org/429798 causes live-migration failures, some stress testing we were doing just hit this bug.
17:49:44 mriedem johnthetubaguy: when was live_migration_progress_timeout added?
17:50:12 johnthetubaguy mriedem: I will check, I thought it was a while ago
17:51:18 cfriesen mriedem: 2015, by dbp. git commit da33ab4f
17:51:48 openstackgerrit Matt Riedemann proposed openstack/nova master: Handle InstanceInvalidState for os-assisted-volume-snapshots https://review.openstack.org/429476
17:52:16 mriedem cfriesen: johnthetubaguy: yeah so since liberty
17:52:52 mriedem cfriesen: cburgess: are either of you using live_migration_progress_timeout ? are you at liberty or newer yet?
17:54:05 cfriesen mriedem: we're on Mitaka, using:
17:54:06 cfriesen live_migration_progress_timeout=150
17:54:07 cfriesen live_migration_completion_timeout=800
17:54:41 melwitt cfriesen, mriedem: for the regular snapshot path (not snapshot_volume_backed), there's code that saves the memory state of the domain if it's in PAUSED state. that's why PAUSED is okay for snapshot but not for snapshot_volume_backed, I think https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L1637-L1642
17:56:05 mriedem praveen: is min_count provided with the server create request?
17:56:29 mriedem praveen: this is the code that handles quota checks and reservations when creating instances https://github.com/openstack/nova/blob/9ae5b2306b7a7cc9e77c77292256b13926920ead/nova/compute/api.py#L299
17:56:49 praveen no i guess
17:56:50 johnthetubaguy mriedem: the bug gives interesting details, basically its broken and never did much other than spot false positives, it would seem
17:57:16 mriedem praveen: i don't know without recreating, but assume if you want 3 you need to specify min_count=3
17:57:34 mriedem johnthetubaguy: i'm hesitant to make that kind of change in rc2 for a thing that's been around since liberty
17:58:04 johnthetubaguy mriedem: this just turns off a thing that randomly makes live-migrate fail, all the old timeouts still trigger as before
17:59:11 praveen mriedem: Thanks. We are setting "max_count" to 3.
17:59:14 johnthetubaguy mriedem: most folks don't hit it, because they do shared storage, so their live-migrations take less that 150 seconds, or at least thats what its looking like
18:00:15 praveen mriedem: i will check with "min_count" .
18:00:57 mriedem johnthetubaguy: for ocata rc2, if you have updated wording for the config option with a known issue reno or something, i'm more comfortable with that
18:01:10 mriedem i.e. "if you're not using shared storage, this could cause issues, see bug"
18:01:23 cfriesen johnthetubaguy: if it's hitting that check doesn't that mean it hasn't transferred data for a substantial amount of time?
18:01:24 johnthetubaguy mriedem: yeah, that makes sense, I am cooking the reno now, with more context in the git commit message
18:01:38 mriedem dansmith: backports are up for those 2 merged docs changes
18:01:50 mriedem https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:stable/ocata
18:02:05 johnthetubaguy cfriesen: more details are in the bug, basically we are miss-interpreting the data we get back from QEMU, so it might be making progress and we still think it hasn't
18:02:24 dansmith mriedem: ack
18:02:25 johnthetubaguy cfriesen: that and during block_migration, we are busy, but will show no progress (apparently, anyways)
18:05:01 cfriesen johnthetubaguy: mriedem: it looks like locally we changed it to use info.data_processed rather than info.data_remaining
18:05:07 melwitt dansmith: are you working on the bfv patch today or do you mind if I hack on it? assuming it's still on the nice-to-have list for rc2
18:05:28 dansmith melwitt: I'm not, so go for it
18:05:32 mriedem cfriesen: you and your local hacks
18:05:38 melwitt dansmith: cool, thanks
18:06:47 johnthetubaguy cfriesen: oh, that might work better, although I think data_processed can be bigger that the total, which means you might never hit the timeout, but I am not sure.
18:06:58 johnthetubaguy cfriesen: that might support just turning it off quickly though, I guess
18:07:49 cfriesen johnthetubaguy: mriedem: we're basically just checking whether we've transferred some data since last time. I think it'd be better to check forward progress on the disk or the RAM since it doesn't help if you're transferring data but not keeping up with the rate of dirtying.
18:08:37 johnthetubaguy cfriesen: pkoniszewski added lots of details in the bug, but there are other issues in there too it seems
18:09:04 mriedem mdbooth: just fyi, you don't need this https://review.openstack.org/#/c/395709/
18:09:12 mriedem mdbooth: we have the nfs job in nova's experimental queue
18:09:21 mriedem to run it on your change, just leave a comment with "check experimental"
18:09:59 cfriesen johnthetubaguy: I had originally commented on https://bugs.launchpad.net/nova/+bug/1583145, but that was marked as a dupe of 1644248.
18:09:59 openstack Launchpad bug 1644248 in OpenStack Compute (nova) "duplicate for #1583145 Nova incorrectly tracks live migration progress" [High,Confirmed]
18:10:40 dansmith johnthetubaguy: questions in here, maybe you can explain them away? https://review.openstack.org/#/c/424428/8
18:13:36 cfriesen mriedem: johnthetubaguy: based on the commit message, the intent was to try to detect the case where the guest is dirtying faster than we can keep up and abort it rather than wait till the overall migration timeout expires.
18:13:39 johnthetubaguy dansmith: replied with the horrible truth on the nicera thing, but agreed with your test issue, not sure how that would work
18:15:52 cfriesen mriedem: johnthetubaguy: Ideally what we'd want is whether the full amount of data to transfer calculated by qemu at the beginning of each iteration is going down....not sure there's any way to get that though.
18:16:45 johnthetubaguy cfriesen: right, thats what pkoniszewski was looking into adding in libvirt and QEMU I think, but thats years away from distros I guess
18:17:12 mriedem johnthetubaguy: dansmith: fyi that i'm out of here in about a half hour,
18:17:47 mriedem fin
18:19:38 dansmith mriedem: for how long?
18:19:52 mriedem several hours
18:20:00 dansmith ack
18:20:02 dansmith approved
18:20:06 mriedem thank you
18:20:47 mriedem the alarming lack of fires today has made me not care about sitting in front of a computer
18:20:49 cfriesen johnthetubaguy: right, in that case it probably does make sense to remove this in Pike since it doesn't help.
18:28:53 openstackgerrit John Garbutt proposed openstack/nova master: Default live_migration_progress_timeout to off https://review.openstack.org/429798
18:29:23 johnthetubaguy cfriesen: does the above description make more sense now ^
18:34:15 mriedem melwitt: posted some comments/questions in https://review.openstack.org/#/c/427981/
18:35:02 mriedem owalsh-afk raises a good point about whether or not the hosts are up (compute_nodes entries created) before running discover_hosts
18:38:40 melwitt mriedem: thanks. it got approved a couple of minutes earlier. I can do a follow up to address your comments. or maybe roll them into my existing follow up that adds the cells v1 steps https://review.openstack.org/#/c/429075
18:39:22 mriedem i'd leave the cells v1 patch for just cells v1 and handle additional cleanups in a separate change
18:39:41 pkoniszewski cfriesen: so info.data_remaining is not useful at all, it is not correleted with whole live migration process, but only with a single iteration
18:40:09 pkoniszewski cfriesen: libvirt would need to expose something like low watermark for the whole process
18:50:52 openstackgerrit John Garbutt proposed openstack/nova master: Default live_migration_progress_timeout to off https://review.openstack.org/429798
19:08:53 openstackgerrit Jay Pipes proposed openstack/nova master: placement: RT now adds proper Ironic inventory https://review.openstack.org/404472
19:09:28 openstackgerrit Jay Pipes proposed openstack/nova master: placement: RT now adds proper Ironic inventory https://review.openstack.org/404472

Earlier   Later