Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-08-01
10:29:31 bauzas honestly, looks like a doubling allocation seems difficult
10:29:54 bauzas when I provided my comment, I was just thinking of getting the original allocation and just passing it back in case of an exception
10:30:12 bauzas I didn't thought about all the other problems we could have :(
10:31:05 sdague cdent: yeh, I'm flushing the in progress bugs, it moves at least 25 out of in progress
10:31:30 bauzas sdague: I did that too
10:32:24 cdent bauzas: It’s good that we are talking this stuff out, because we don’t really have sufficient testing to find the bugs that we are creating (we need more gibi ) so applying brains is necessary. Each question, though annoying, is revealing something, and that’s better in the long run.
10:33:18 bauzas sdague: do we have jobs for resizing to a 2nd host ? I know that for live-migration, but I do wonder if we only verify resizes for the same host
10:39:04 johnthetubaguy bauzas: I think XenAPI already shrinks the disks when resizing down
10:40:10 bauzas johnthetubaguy: ack
10:40:30 bauzas johnthetubaguy: anyway, stopping to accept at the API level a resize down would probably need a microversion
10:40:37 johnthetubaguy cdent: when we chatted about this before, I remember we kinda liked the doubling up, are folks finding thats bad?
10:40:58 johnthetubaguy bauzas: I guess it should, although that's kinda removing a feature some folks use
10:41:15 cdent johnthetubaguy: we need the doubling up, the problem is that it is hard to manage the cleaning up afterwards
10:41:16 bauzas johnthetubaguy: exactly my point I wrote in the review
10:41:34 johnthetubaguy cdent: oh, I see what you mean now
10:41:56 johnthetubaguy cdent: would the migration uuid holding an allocation work, or mess things up totally for sync logic?
10:41:58 bauzas johnthetubaguy: either we say nova will stop supporting that for all drivers and then it requires a microversion, or we say it's per-driver and then we don't want to have the API verifying it
10:42:29 johnthetubaguy bauzas: I would rather the API new if the compute host could do it
10:42:40 johnthetubaguy knew
10:43:02 bauzas johnthetubaguy: yeah, agreed, a capability
10:43:05 cdent johnthetubaguy: if there were a migration uuid, that would help, but there’s not, and getting access to it in the confirm_resize is weird (not sure I have all the details of this right, was very tired while listening to dan and jay last night)
10:43:29 bauzas cdent: we have a migration object, couldn't that help ? (well, for all migrations except live-mig :D)
10:43:32 johnthetubaguy cdent: oh, damm, did we never add that
10:43:45 cdent johnthetubaguy: that’s what dan and jay said :)
10:43:51 cdent (the “oh damn”)
10:43:58 johnthetubaguy cdent: oh, thats correct, migration is completed before the confirm/revert phase I think
10:44:02 bauzas cdent: and FWIW we expose those migration objects to the API
10:44:14 bauzas s/FWIW/AFAIK
10:45:07 johnthetubaguy bauzas: in my head that was getting added as part of the cancel resize/migrate work
10:45:59 openstackgerrit Sean Dague proposed openstack/nova master: Show quota detail when inject file quota exceeds https://review.openstack.org/453040
10:46:18 bauzas johnthetubaguy: you mean os-migrations ?
10:46:30 johnthetubaguy bauzas: I think so
10:46:56 bauzas but whatever, we haven't yet a migration object for live migrations :)
10:47:21 johnthetubaguy bauzas: can't remember if the additions merged now, I thought it was shared for both now
10:47:31 bauzas maybe we could just split nikola's patch in two and just at least add the migration object at first, before trying to claim
10:47:34 johnthetubaguy anyways, not sure any of that helps
10:47:51 sdague bauzas: we run the resize tests in multihost I think
10:48:01 bauzas sdague: cool then
10:48:06 sdague bauzas: but I haven't looked that hard to verify
10:48:16 bauzas sdague: I can check
10:51:41 cdent sdague, bauzas: we talked about that last night, and dan confirmed there are some, but the issue with them is that they don’t run in a constrained environment nor concurrent placements and don’t validate the allocations, so the fact that there isn’t yet doubling of allocations isn’t an issue because there’s spare capacity
10:52:02 sdague cdent: yeh, that makes sense
10:52:31 sdague honestly, that's one of those things where doing the in tree functional testing with a couple of fake computes is probably the best way to flush it out
10:52:43 cdent sdague: that’s what gibi’s new test does
10:52:46 cdent so is very good to have
10:52:57 sdague cdent: does that still need review?
10:53:14 bauzas cdent: dan confirmed there are some what ?
10:53:39 cdent but I suspect that there are lots of edges that won’t get covered and in about 18 months we’ll see a lot of bugs from big deployments
10:53:46 cdent bauzas: there are some resize tests
10:54:01 cdent sdague: yes, it needs review, but I’m not sure how “done” it is: https://review.openstack.org/#/c/487958/
10:54:20 bauzas cdent: ah
10:54:32 bauzas cdent: yeah, I know that but I do wonder if we test those multihost
10:54:46 cdent yes, multihost
10:54:51 bauzas ok
10:55:36 bauzas cdent: I see your comment now, you mean those test exist but aren't really stress-testing at limits so we can't really verify capacity issues
10:55:42 bauzas tests*
10:56:09 bauzas so, not a big deal for functional testing, but not for placement-specific concerns
10:56:15 bauzas gotcha
10:58:23 gibi cdent, sdague: I'm still intended to understand and remove the last time.sleep(1) left in the setUp() of that tests as that is ugly and might not be necessary
11:05:32 openstackgerrit Merged openstack/nova master: fix test_rebuild_server_exc instability https://review.openstack.org/487382
11:08:54 gibi OK I think that sleep was only needed before the fake virt driver was updated to keep a local nodes copy and the periodic tasks was running every second
11:09:21 gibi that two things together caused a race on setting the nodes of the fake virt driver
11:09:25 sdague gibi: yeh, I just went through with a fine toothed comb there and provided some import
11:09:28 sdague input
11:09:57 gibi sdague: thanks. checking...
11:10:13 sdague gibi: mostly also bringing fresh eyes of not being familiar with some of this, so some code restructure for clarity that might help as well
11:11:46 gibi sdague: sure, fresh eyes helps a lot
11:12:39 openstackgerrit Merged openstack/nova master: Fix example in _serialize_allocations_for_consumer https://review.openstack.org/487614
11:16:22 openstackgerrit Merged openstack/nova master: rootwrap.d cleanup mislabeled files https://review.openstack.org/486831
11:30:38 sdague gibi / cdent - https://review.openstack.org/#/c/487327
11:30:47 sdague can we fix wsgi_intercept instead?
11:31:12 sdague because if we don't pass those vars, people can't build their venvs
11:34:51 openstackgerrit Gábor Antal proposed openstack/nova master: Transform instance.rebuild_scheduled notification https://review.openstack.org/473929
11:35:09 gibi sdague: I have an env that is behind proxy and I can build the env there without passing the PROXY wars
11:35:54 gibi sdague: But if this can be fixed in wsgi_intercept then I'm happy with that solution
11:36:08 openstackgerrit Merged openstack/nova master: Instance remains in migrating state forever https://review.openstack.org/483911
11:36:10 sdague gibi: not everyone can - https://review.openstack.org/#/c/189569/
11:36:29 sdague it was added because without it people were blocked from running tests
11:37:07 openstackgerrit Gábor Antal proposed openstack/nova master: Transform instance.rebuild_scheduled notification https://review.openstack.org/473929
11:37:09 sdague unless tox itself changed since then
11:37:32 jaypipes cdent: morning.
11:37:42 cdent sdague: I think tox has changed
11:38:10 cdent it’s difficult to fix in the wsgi_intercept because of the way urllib3 manages proxy variables very early in its handling
11:38:15 cdent (at least last time I checked)
11:38:18 cdent jaypipes: morning
11:38:20 openstack bug 1707071 in OpenStack Compute (nova) "Compute nodes will fight over allocations during migration" [Medium,In progress] https://launchpad.net/bugs/1707071 - Assigned to Jay Pipes (jaypipes)
11:38:20 jaypipes cdent: unfortunately, trying to fix bug #1707071 has been excruciating. Looks like I'm going to need to go back to the drawing board and rewrite my patch pretty much from scratch.
11:38:29 cdent jaypipes: oh noes!
11:38:30 sdague ok, so we need to figure out when it changed, because our minimum is 2.0
11:38:40 cdent sdague: i’ll look in the changelogs
11:38:59 sdague gibi: if you can confirm when tox changed, and bump the minimum at the same time, I'm +2
11:39:00 gibi cdent: thanks for taking that
11:39:10 gibi sdague: good point
11:39:14 sdague I just don't want to break folks
11:39:44 cdent jaypipes: is there a crucial bit, or is it many things combined?
11:41:30 cdent sdague, gibi: 2.1.0: http://tox.readthedocs.io/en/latest/changelog.html#id12
11:42:12 gibi cdent: thanks! I will update the patch with the mimimum bump soon
11:43:04 jaypipes cdent: the latter
11:43:18 jaypipes cdent: and the fact that we need to deal with Ocata computes migrating to Pike computes.
11:43:55 cdent jaypipes: do you have an idea/plan or still cogitating? anything I can do to help?

Earlier   Later