| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-02-06 | |||
| 17:45:14 | mriedem | oh before pause, | |
| 17:45:36 | mriedem | https://github.com/openstack/nova/blob/9ae5b2306b7a7cc9e77c77292256b13926920ead/nova/virt/libvirt/guest.py#L566 | |
| 17:46:16 | openstackgerrit | John Garbutt proposed openstack/nova master: Default live_migration_progress_timeout to off https://review.openstack.org/429798 | |
| 17:47:06 | praveen | Hi mriedem , One help! | |
| 17:47:08 | praveen | There is an observation.. Lets suppose project quota for max number of vms is set as 5. 3 Vms are already up and running into that project. Now, when user triggers another 3 Vms at one go as multiple deployment then 2 vms gets deployed succesfully but another VM does not.. which is correct because of quota error... But there is no indication/error message which gets shown into logs or UI..... | |
| 17:47:09 | praveen | ...Is this the way multipe server create behaves? | |
| 17:47:11 | praveen | I found below lines in compute API reference wiki - Not sure if it says same thing ?? | |
| 17:47:12 | praveen | "Error handling for multiple create is not as consistent as for single server create, and there is no guarantee that all the servers will be built. This call should generally be avoided in favor of clients doing direct individual server creates." | |
| 17:47:36 | mriedem | cfriesen: i always get confused between paused and suspended with qemu | |
| 17:47:48 | mriedem | since the code for pause in nova calls suspend on the domain | |
| 17:47:48 | mriedem | https://wiki.libvirt.org/page/VM_lifecycle#States_that_a_guest_domain_can_be_in | |
| 17:47:49 | cfriesen | mriedem: okay, so pause just immediately suspends the domain, while suspend does a managedSave on the domain. | |
| 17:48:12 | johnthetubaguy | mriedem: pkoniszewski: I wonder if we should look at getting the above change into RC2 https://review.openstack.org/429798 causes live-migration failures, some stress testing we were doing just hit this bug. | |
| 17:49:44 | mriedem | johnthetubaguy: when was live_migration_progress_timeout added? | |
| 17:50:12 | johnthetubaguy | mriedem: I will check, I thought it was a while ago | |
| 17:51:18 | cfriesen | mriedem: 2015, by dbp. git commit da33ab4f | |
| 17:51:48 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Handle InstanceInvalidState for os-assisted-volume-snapshots https://review.openstack.org/429476 | |
| 17:52:16 | mriedem | cfriesen: johnthetubaguy: yeah so since liberty | |
| 17:52:52 | mriedem | cfriesen: cburgess: are either of you using live_migration_progress_timeout ? are you at liberty or newer yet? | |
| 17:54:05 | cfriesen | mriedem: we're on Mitaka, using: | |
| 17:54:06 | cfriesen | live_migration_progress_timeout=150 | |
| 17:54:07 | cfriesen | live_migration_completion_timeout=800 | |
| 17:54:41 | melwitt | cfriesen, mriedem: for the regular snapshot path (not snapshot_volume_backed), there's code that saves the memory state of the domain if it's in PAUSED state. that's why PAUSED is okay for snapshot but not for snapshot_volume_backed, I think https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L1637-L1642 | |
| 17:56:05 | mriedem | praveen: is min_count provided with the server create request? | |
| 17:56:29 | mriedem | praveen: this is the code that handles quota checks and reservations when creating instances https://github.com/openstack/nova/blob/9ae5b2306b7a7cc9e77c77292256b13926920ead/nova/compute/api.py#L299 | |
| 17:56:49 | praveen | no i guess | |
| 17:56:50 | johnthetubaguy | mriedem: the bug gives interesting details, basically its broken and never did much other than spot false positives, it would seem | |
| 17:57:16 | mriedem | praveen: i don't know without recreating, but assume if you want 3 you need to specify min_count=3 | |
| 17:57:34 | mriedem | johnthetubaguy: i'm hesitant to make that kind of change in rc2 for a thing that's been around since liberty | |
| 17:58:04 | johnthetubaguy | mriedem: this just turns off a thing that randomly makes live-migrate fail, all the old timeouts still trigger as before | |
| 17:59:11 | praveen | mriedem: Thanks. We are setting "max_count" to 3. | |
| 17:59:14 | johnthetubaguy | mriedem: most folks don't hit it, because they do shared storage, so their live-migrations take less that 150 seconds, or at least thats what its looking like | |
| 18:00:15 | praveen | mriedem: i will check with "min_count" . | |
| 18:00:57 | mriedem | johnthetubaguy: for ocata rc2, if you have updated wording for the config option with a known issue reno or something, i'm more comfortable with that | |
| 18:01:10 | mriedem | i.e. "if you're not using shared storage, this could cause issues, see bug" | |
| 18:01:23 | cfriesen | johnthetubaguy: if it's hitting that check doesn't that mean it hasn't transferred data for a substantial amount of time? | |
| 18:01:24 | johnthetubaguy | mriedem: yeah, that makes sense, I am cooking the reno now, with more context in the git commit message | |
| 18:01:38 | mriedem | dansmith: backports are up for those 2 merged docs changes | |
| 18:01:50 | mriedem | https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:stable/ocata | |
| 18:02:05 | johnthetubaguy | cfriesen: more details are in the bug, basically we are miss-interpreting the data we get back from QEMU, so it might be making progress and we still think it hasn't | |
| 18:02:24 | dansmith | mriedem: ack | |
| 18:02:25 | johnthetubaguy | cfriesen: that and during block_migration, we are busy, but will show no progress (apparently, anyways) | |
| 18:05:01 | cfriesen | johnthetubaguy: mriedem: it looks like locally we changed it to use info.data_processed rather than info.data_remaining | |
| 18:05:07 | melwitt | dansmith: are you working on the bfv patch today or do you mind if I hack on it? assuming it's still on the nice-to-have list for rc2 | |
| 18:05:28 | dansmith | melwitt: I'm not, so go for it | |
| 18:05:32 | mriedem | cfriesen: you and your local hacks | |
| 18:05:38 | melwitt | dansmith: cool, thanks | |
| 18:06:47 | johnthetubaguy | cfriesen: oh, that might work better, although I think data_processed can be bigger that the total, which means you might never hit the timeout, but I am not sure. | |
| 18:06:58 | johnthetubaguy | cfriesen: that might support just turning it off quickly though, I guess | |
| 18:07:49 | cfriesen | johnthetubaguy: mriedem: we're basically just checking whether we've transferred some data since last time. I think it'd be better to check forward progress on the disk or the RAM since it doesn't help if you're transferring data but not keeping up with the rate of dirtying. | |
| 18:08:37 | johnthetubaguy | cfriesen: pkoniszewski added lots of details in the bug, but there are other issues in there too it seems | |
| 18:09:04 | mriedem | mdbooth: just fyi, you don't need this https://review.openstack.org/#/c/395709/ | |
| 18:09:12 | mriedem | mdbooth: we have the nfs job in nova's experimental queue | |
| 18:09:21 | mriedem | to run it on your change, just leave a comment with "check experimental" | |
| 18:09:59 | cfriesen | johnthetubaguy: I had originally commented on https://bugs.launchpad.net/nova/+bug/1583145, but that was marked as a dupe of 1644248. | |
| 18:09:59 | openstack | Launchpad bug 1644248 in OpenStack Compute (nova) "duplicate for #1583145 Nova incorrectly tracks live migration progress" [High,Confirmed] | |
| 18:10:40 | dansmith | johnthetubaguy: questions in here, maybe you can explain them away? https://review.openstack.org/#/c/424428/8 | |
| 18:13:36 | cfriesen | mriedem: johnthetubaguy: based on the commit message, the intent was to try to detect the case where the guest is dirtying faster than we can keep up and abort it rather than wait till the overall migration timeout expires. | |
| 18:13:39 | johnthetubaguy | dansmith: replied with the horrible truth on the nicera thing, but agreed with your test issue, not sure how that would work | |
| 18:15:52 | cfriesen | mriedem: johnthetubaguy: Ideally what we'd want is whether the full amount of data to transfer calculated by qemu at the beginning of each iteration is going down....not sure there's any way to get that though. | |
| 18:16:45 | johnthetubaguy | cfriesen: right, thats what pkoniszewski was looking into adding in libvirt and QEMU I think, but thats years away from distros I guess | |
| 18:17:12 | mriedem | johnthetubaguy: dansmith: fyi that i'm out of here in about a half hour, | |
| 18:17:47 | mriedem | fin | |
| 18:19:38 | dansmith | mriedem: for how long? | |
| 18:19:52 | mriedem | several hours | |
| 18:20:00 | dansmith | ack | |
| 18:20:02 | dansmith | approved | |
| 18:20:06 | mriedem | thank you | |
| 18:20:47 | mriedem | the alarming lack of fires today has made me not care about sitting in front of a computer | |
| 18:20:49 | cfriesen | johnthetubaguy: right, in that case it probably does make sense to remove this in Pike since it doesn't help. | |
| 18:28:53 | openstackgerrit | John Garbutt proposed openstack/nova master: Default live_migration_progress_timeout to off https://review.openstack.org/429798 | |
| 18:29:23 | johnthetubaguy | cfriesen: does the above description make more sense now ^ | |
| 18:34:15 | mriedem | melwitt: posted some comments/questions in https://review.openstack.org/#/c/427981/ | |
| 18:35:02 | mriedem | owalsh-afk raises a good point about whether or not the hosts are up (compute_nodes entries created) before running discover_hosts | |
| 18:38:40 | melwitt | mriedem: thanks. it got approved a couple of minutes earlier. I can do a follow up to address your comments. or maybe roll them into my existing follow up that adds the cells v1 steps https://review.openstack.org/#/c/429075 | |
| 18:39:22 | mriedem | i'd leave the cells v1 patch for just cells v1 and handle additional cleanups in a separate change | |
| 18:39:41 | pkoniszewski | cfriesen: so info.data_remaining is not useful at all, it is not correleted with whole live migration process, but only with a single iteration | |
| 18:40:09 | pkoniszewski | cfriesen: libvirt would need to expose something like low watermark for the whole process | |
| 18:50:52 | openstackgerrit | John Garbutt proposed openstack/nova master: Default live_migration_progress_timeout to off https://review.openstack.org/429798 | |
| 19:08:53 | openstackgerrit | Jay Pipes proposed openstack/nova master: placement: RT now adds proper Ironic inventory https://review.openstack.org/404472 | |
| 19:09:28 | openstackgerrit | Jay Pipes proposed openstack/nova master: placement: RT now adds proper Ironic inventory https://review.openstack.org/404472 | |
| 19:12:43 | cfriesen | johnthetubaguy: pkoniszewski: agreed...reviewing latest now. where there earlier discussions about whether to do complete removal in Pike, or do we still need to decide that? | |
| 19:12:52 | cfriesen | s/where/were | |
| 19:43:46 | pkoniszewski | cfriesen: i see no point in keeping broken bits | |
| 19:44:12 | pkoniszewski | and those bits can cause headache to operators | |
| 19:48:07 | jaypipes | pkoniszewski: ++ | |
| 21:06:53 | openstackgerrit | Trevor McCasland proposed openstack/nova-specs master: Add spec for Accelerating SR-IOV with DPDK https://review.openstack.org/429867 | |
| 21:08:43 | cfriesen | pkoniszewski: are you planning on submitting the removal for Pike? If not, I could take it. | |
| 21:15:20 | openstackgerrit | Pushkar Umaranikar proposed openstack/nova master: [WIP] Placement api: Add json_error_formatter to defaults https://review.openstack.org/395194 | |
| 21:18:01 | openstackgerrit | Pushkar Umaranikar proposed openstack/nova master: [WIP] Placement api: Add json_error_formatter to defaults https://review.openstack.org/395194 | |
| 21:21:07 | openstackgerrit | Pushkar Umaranikar proposed openstack/nova master: Placement api: Add json_error_formatter to defaults https://review.openstack.org/395194 | |
| 21:26:05 | openstackgerrit | Pushkar Umaranikar proposed openstack/nova master: Placement api: Add json_error_formatter to defaults https://review.openstack.org/395194 | |
| 22:43:43 | openstackgerrit | Dan Smith proposed openstack/nova master: Allow pattern-based substitutions in CellMapping URLs https://review.openstack.org/429910 | |
| 22:49:07 | openstackgerrit | Sivasathurappan Radhakrishnan proposed openstack/nova-specs master: Live Migration of Rescued Instances https://review.openstack.org/429913 | |
| 22:58:03 | openstackgerrit | Dan Smith proposed openstack/nova master: Allow pattern-based substitutions in CellMapping URLs https://review.openstack.org/429910 | |
| 23:39:40 | oomichi | alex_xu: can you take a look at a critical patch: https://review.openstack.org/#/c/410006/ ? | |
| #openstack-nova - 2017-02-07 | |||
| 02:18:40 | openstackgerrit | ChangBo Guo(gcb) proposed openstack/nova master: Enable more virt.xenapi tests on Python 3 https://review.openstack.org/414421 | |
| 02:18:53 | openstackgerrit | ChangBo Guo(gcb) proposed openstack/nova master: Skip unit tests for SSL + py3 https://review.openstack.org/426505 | |
| 02:32:42 | openstackgerrit | Huan Xie proposed openstack/nova master: Fix live migrate with XenServer https://review.openstack.org/424428 | |
| 02:51:05 | alex_xu | oomichi: yea, will check that after meeting | |