Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-02-06
16:35:25 dansmith melwitt: cdent: FWIW, the reason I used a naked "placement" is because I'm referring to placement api, service, database, catalog entry, etc as a group
16:35:41 dansmith "placement service" would be fine too, but... that was my thought process
16:36:53 cdent dansmith: yeah, thus the "fine as is". I think "service" is a nice to have, because it's hard to be sure that "placement" on its own is a proper noun and not just some random thing, like layout of the knives and forks on the table.
16:37:12 melwitt dansmith: I see, thanks for clarifying. that makes it make a lot more sense
16:39:05 pumaranikar cdent, johnthetubaguy https://review.openstack.org/#/c/395194/ updated patchset for json error formatter. can you please quickly review and let me know that this is what we are expecting. ?
16:39:19 edleafe cdent: Capitalization maybe? E.g.: 'placement' (random thing) vs. 'Placement' (the resource thingy)
16:39:50 cdent edleafe: because it is new, I'm still not certain that's enough. All of us here think of it as this well known things, but it is not.
16:41:56 edleafe true
16:48:24 cdent pumaranikar: yeah, that looks like a good way to go assuming it works well. How do you feel about it?
16:49:37 pumaranikar cdent, thanks for quick review. Yes it will work. Only thing is we need to modify decorator everywhere
16:50:04 pumaranikar cdent, thats why I thought of middleware initially.
16:50:37 cdent that's okay, I think, as it is a pretty simple change and in any case, no matter how we add, we have to change a bunch of files to remove the keyword arg from all the raised exceptions
16:51:06 cdent I think to get a middleware version that actually worked properly would be considerably more complex (because of the need to parse headers and some response bodies)
16:51:51 pumaranikar cdent, yes. that makes sense.
16:52:10 pumaranikar cdent, i will try to update this today in all places.
16:52:41 cdent pumaranikar: you could consider doing a stack of changes where you change each file in handlers/*.py in a different patch
16:53:38 pumaranikar cdent, yes good idea. It would be easy for review.
16:55:39 pumaranikar cdent, are we ok with class name in wrapper file ? :)
16:56:18 cdent I think it is good enough. If people don't like it we can change it easily enough during the review process
16:57:37 pumaranikar cdent, ok. Will let you know once i make changes atleast in all allocations related actions
16:58:34 cdent cool, looking forward to it
16:58:42 edleafe cdent: First pass at functional code paths for placement testing: https://etherpad.openstack.org/p/nova-placement-functional
16:58:49 edleafe cdent: Now to disappear into meetings
16:58:59 cdent edleafe: cool, I'll look at that either this evening or tomorrow morning
16:59:34 edleafe cdent: thanks. Just wanted to get something down before I got sucked into other things
16:59:53 cdent ✔
17:03:45 mriedem dansmith: the revert on that manila change passed https://review.openstack.org/#/c/429729/ :)
17:03:52 mriedem so i guess it's already fixed...
17:16:58 openstackgerrit Matt Riedemann proposed openstack/nova master: Handle InstanceInvalidState for os-assisted-volume-snapshots https://review.openstack.org/429476
17:19:42 mriedem alex_xu: gmann: oomichi: i'm asserting that we don't need a microversion change for ^ but open to being correct
17:19:45 mriedem *corrected
17:28:24 cfriesen mriedem: what about a suspended guest?
17:29:37 mriedem cfriesen: i guess we do allow that for regular snapshots
17:30:26 mriedem i thought something was wonky with suspended instances in a volume-backed instance flow, but might be thinking of something else
17:33:54 dansmith mriedem: nice
17:37:00 mriedem cfriesen: ooo i actually had that backward
17:37:38 mriedem cfriesen: https://github.com/openstack/nova/blob/9ae5b2306b7a7cc9e77c77292256b13926920ead/nova/compute/api.py#L2805
17:37:58 mriedem we only allow active/stopped/suspended for volume-backed snapshot in the "normal" flow
17:38:03 mriedem so i'll change my patch to follow that
17:40:56 cfriesen mriedem: do we not support paused because we don't bother quiescing the disk first?
17:42:22 mriedem it does quiesce, https://github.com/openstack/nova/blob/9ae5b2306b7a7cc9e77c77292256b13926920ead/nova/virt/libvirt/driver.py#L1913
17:42:32 mriedem and if that fails, it snapshots w/o quiesce
17:42:37 mriedem so maybe paused would work too,
17:42:41 mriedem i just haven't tested it
17:43:05 mriedem i'd have to get a devstack + nfs plugin setup to test this out, which would take some time
17:43:24 dansmith someone should hit this: https://review.openstack.org/#/c/427981/
17:43:30 dansmith last doc change on the books for rc2
17:44:18 dansmith mriedem: this is on the list for rc2...should I spend time reviewing it or what? https://review.openstack.org/#/c/424428/
17:44:28 mriedem dansmith: yeah
17:44:53 cfriesen mriedem: I meant quiesce before pausing...since otherwise there could potentially be data loss in the snapshot....but maybe we'd get the same with suspend. I don't know enough about qemu behaviour in pause vs suspend
17:45:14 mriedem oh before pause,
17:45:36 mriedem https://github.com/openstack/nova/blob/9ae5b2306b7a7cc9e77c77292256b13926920ead/nova/virt/libvirt/guest.py#L566
17:46:16 openstackgerrit John Garbutt proposed openstack/nova master: Default live_migration_progress_timeout to off https://review.openstack.org/429798
17:47:06 praveen Hi mriedem , One help!
17:47:08 praveen There is an observation.. Lets suppose project quota for max number of vms is set as 5. 3 Vms are already up and running into that project. Now, when user triggers another 3 Vms at one go as multiple deployment then 2 vms gets deployed succesfully but another VM does not.. which is correct because of quota error... But there is no indication/error message which gets shown into logs or UI.....
17:47:09 praveen ...Is this the way multipe server create behaves?
17:47:11 praveen I found below lines in compute API reference wiki - Not sure if it says same thing ??
17:47:12 praveen "Error handling for multiple create is not as consistent as for single server create, and there is no guarantee that all the servers will be built. This call should generally be avoided in favor of clients doing direct individual server creates."
17:47:36 mriedem cfriesen: i always get confused between paused and suspended with qemu
17:47:48 mriedem since the code for pause in nova calls suspend on the domain
17:47:48 mriedem https://wiki.libvirt.org/page/VM_lifecycle#States_that_a_guest_domain_can_be_in
17:47:49 cfriesen mriedem: okay, so pause just immediately suspends the domain, while suspend does a managedSave on the domain.
17:48:12 johnthetubaguy mriedem: pkoniszewski: I wonder if we should look at getting the above change into RC2 https://review.openstack.org/429798 causes live-migration failures, some stress testing we were doing just hit this bug.
17:49:44 mriedem johnthetubaguy: when was live_migration_progress_timeout added?
17:50:12 johnthetubaguy mriedem: I will check, I thought it was a while ago
17:51:18 cfriesen mriedem: 2015, by dbp. git commit da33ab4f
17:51:48 openstackgerrit Matt Riedemann proposed openstack/nova master: Handle InstanceInvalidState for os-assisted-volume-snapshots https://review.openstack.org/429476
17:52:16 mriedem cfriesen: johnthetubaguy: yeah so since liberty
17:52:52 mriedem cfriesen: cburgess: are either of you using live_migration_progress_timeout ? are you at liberty or newer yet?
17:54:05 cfriesen mriedem: we're on Mitaka, using:
17:54:06 cfriesen live_migration_progress_timeout=150
17:54:07 cfriesen live_migration_completion_timeout=800
17:54:41 melwitt cfriesen, mriedem: for the regular snapshot path (not snapshot_volume_backed), there's code that saves the memory state of the domain if it's in PAUSED state. that's why PAUSED is okay for snapshot but not for snapshot_volume_backed, I think https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L1637-L1642
17:56:05 mriedem praveen: is min_count provided with the server create request?
17:56:29 mriedem praveen: this is the code that handles quota checks and reservations when creating instances https://github.com/openstack/nova/blob/9ae5b2306b7a7cc9e77c77292256b13926920ead/nova/compute/api.py#L299
17:56:49 praveen no i guess
17:56:50 johnthetubaguy mriedem: the bug gives interesting details, basically its broken and never did much other than spot false positives, it would seem
17:57:16 mriedem praveen: i don't know without recreating, but assume if you want 3 you need to specify min_count=3
17:57:34 mriedem johnthetubaguy: i'm hesitant to make that kind of change in rc2 for a thing that's been around since liberty
17:58:04 johnthetubaguy mriedem: this just turns off a thing that randomly makes live-migrate fail, all the old timeouts still trigger as before
17:59:11 praveen mriedem: Thanks. We are setting "max_count" to 3.
17:59:14 johnthetubaguy mriedem: most folks don't hit it, because they do shared storage, so their live-migrations take less that 150 seconds, or at least thats what its looking like
18:00:15 praveen mriedem: i will check with "min_count" .
18:00:57 mriedem johnthetubaguy: for ocata rc2, if you have updated wording for the config option with a known issue reno or something, i'm more comfortable with that
18:01:10 mriedem i.e. "if you're not using shared storage, this could cause issues, see bug"
18:01:23 cfriesen johnthetubaguy: if it's hitting that check doesn't that mean it hasn't transferred data for a substantial amount of time?
18:01:24 johnthetubaguy mriedem: yeah, that makes sense, I am cooking the reno now, with more context in the git commit message
18:01:38 mriedem dansmith: backports are up for those 2 merged docs changes
18:01:50 mriedem https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:stable/ocata
18:02:05 johnthetubaguy cfriesen: more details are in the bug, basically we are miss-interpreting the data we get back from QEMU, so it might be making progress and we still think it hasn't
18:02:24 dansmith mriedem: ack
18:02:25 johnthetubaguy cfriesen: that and during block_migration, we are busy, but will show no progress (apparently, anyways)
18:05:01 cfriesen johnthetubaguy: mriedem: it looks like locally we changed it to use info.data_processed rather than info.data_remaining
18:05:07 melwitt dansmith: are you working on the bfv patch today or do you mind if I hack on it? assuming it's still on the nice-to-have list for rc2
18:05:28 dansmith melwitt: I'm not, so go for it
18:05:32 mriedem cfriesen: you and your local hacks
18:05:38 melwitt dansmith: cool, thanks
18:06:47 johnthetubaguy cfriesen: oh, that might work better, although I think data_processed can be bigger that the total, which means you might never hit the timeout, but I am not sure.
18:06:58 johnthetubaguy cfriesen: that might support just turning it off quickly though, I guess
18:07:49 cfriesen johnthetubaguy: mriedem: we're basically just checking whether we've transferred some data since last time. I think it'd be better to check forward progress on the disk or the RAM since it doesn't help if you're transferring data but not keeping up with the rate of dirtying.

Earlier   Later