Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-10-03
10:23:51 bauzas well
10:23:55 sean-k-mooney you need to run it under wine however
10:24:02 bauzas just for what I want, a mysqldump is enough
10:36:33 sean-k-mooney the one thing that annoys me more then the fact that we use a 80 charter line lenght is that we configure pep8 on spec for 79 charters
10:43:06 openstackgerrit sean mooney proposed openstack/nova-specs master: Add spec for sriov live migration https://review.openstack.org/605116
10:57:24 stephenfin That moment of panic where you submit a review and spot a group of comments on older patchsets from the corner of your eye
10:57:48 stephenfin What *did* Stephen of June 2018 have to say about this...
11:08:59 mdbooth stephenfin: Fancy a bash at this one: https://review.openstack.org/#/c/605436/ I bitch about python3 in it ;)
11:09:24 openstackgerrit sean mooney proposed openstack/nova-specs master: Add spec for sriov live migration https://review.openstack.org/605116
11:39:48 sean-k-mooney mdbooth: one basic question regarding https://review.openstack.org/#/c/605436/5. the compaute manger runs on the compute agent which is singel treaded but uses eventlets. so there is no paralleism but there is concurancy. so the lock you are aquiring is the mockey patched greenthread lock. is the reason we need the lock in the first place the fact we are doing db io and eventlets is cause us to yeild
11:39:51 sean-k-mooney allow another invocation of the fuction to start concurrently which races
11:40:43 mdbooth sean-k-mooney: Without going into details of locking, I find it's safest to ignore eventlets entirely when considering locking.
11:41:01 mdbooth When you tie yourself in knots trying to second guess a scheduler, you make lots of mistakes.
11:41:11 sean-k-mooney mdbooth: if we did not have eventlets in this case we would not need to lock
11:41:41 mdbooth It has multiple threads of execution.
11:41:49 mdbooth I don't care how we achieve that.
11:42:08 sean-k-mooney the compute manager is exectued from the compute agent right which does not have workers so only one thread
11:42:22 mdbooth Eventlets or python's 'threading' all have the same issues.
11:43:06 sean-k-mooney mdbooth: yes but my point is eventlets intoduced the concurency so that therefor we need a lock to be correct
11:43:33 mdbooth sean-k-mooney: We have concurrency.
11:43:55 sean-k-mooney if we did not have eventlets the previous code would have been correct because it would have been singel threaded
11:44:02 sean-k-mooney mdbooth: yep i know
11:44:28 mdbooth sean-k-mooney: Sure. We do have concurrency, though. It uses eventlets.
11:44:30 sean-k-mooney just makeing sure i understand the subtelties of the patch. this is an example of why i hate eventlets it hides concurancy
11:44:50 mdbooth sean-k-mooney: It doesn't really by the time you get into the compute manager.
11:45:16 mdbooth You just ignore eventlets entirely and assume you have multiple concurrent threads. How they're implemented isn't all that important in that code.
11:45:32 mdbooth If we later switched to a multi-threaded model, it would still be fine.
11:46:07 sean-k-mooney mdbooth: well if you were new to nova or did not think about it at the time then you can write races because of eventlets easier then if it was expcitly threaded
11:46:47 mdbooth Honestly, I never consider eventlets. I assume it's explicitly threaded.
11:47:11 mdbooth It's not, but that doesn't have any bearing on writing safe code, except when there's bugs in eventlet.
11:47:39 sean-k-mooney mdbooth: most new people i have talked to that work on openstack dont think about threads at all because the say oh its python and that has a gil so i dont need to care
11:48:13 mdbooth sean-k-mooney: That is just one of many issues with python in the real world :(
11:48:59 mdbooth Also, the gil doesn't prevent overlapping threads of execution, it just stops them running at the same time. The problems are the same.
11:49:15 sean-k-mooney mdbooth: yes and no. there perception is normally correct. its eventlets that violates the paradime
11:49:42 mdbooth As an old curmudgeon, I think python has been extremely detrimental to software engineering, particular in education
11:50:32 mdbooth No, it would not be correct. If you have a multi-threaded python program, even though the gil prevents multiple threads running concurrently, you still need locks.
11:50:43 sean-k-mooney mdbooth: i self taught myself c++ as my first langage then java so ya i like to understand exactly what is going on
11:51:46 mdbooth Basically: eventlets or python-multithreading: I don't care. It shouldn't change how your write code.
11:51:49 sean-k-mooney learning c++ fist made me a better engineer then learing python would have.
11:52:04 mdbooth They both use fake threading.
11:53:03 openstackgerrit Matthew Booth proposed openstack/nova master: Run evacuate tests with local/lvm and shared/rbd storage https://review.openstack.org/604400
11:53:13 mdbooth I think ^^^ might fix that weird issue I was hitting
11:53:42 mdbooth There's a shortcut in service_is_up if the service is forced down, and we forced it down
11:54:29 sean-k-mooney mdbooth: yes i know if it was expcitly multi thraded it would be incorrect without the lock. anyway cool ill take a look at that too
11:55:10 mdbooth sean-k-mooney: Don't worry about ^^^ btw. Will just wait until zuul has voted.
11:55:22 mdbooth sean-k-mooney: I knew I was missing something simple there.
11:55:53 sean-k-mooney waith why was it forced down?
11:56:12 sean-k-mooney oh so you could evacuate
11:56:16 mdbooth The test_evacuate.sh script which ran before the second round...
11:56:18 mdbooth yeah
11:57:22 sean-k-mooney ha yet anthoer race this tiem between test :)
12:00:22 nicolasbock Morning
12:01:58 nicolasbock I have a run-away server, i.e. a VM that's running on a hypervisor different than what Nova thinks. So far I haven't quite been able to get the placement service to help me update Nova's view of reality...
12:02:22 nicolasbock I can see the server with 'openstack server show 2aa3a324-bf22-4e0c-912a-d7c52f59f1fd'
12:02:36 nicolasbock And it lists the wrong hypervisor
12:02:47 mdbooth sean-k-mooney: $ git grep test\.nested | wc -l
12:02:47 mdbooth 348
12:02:54 mdbooth I'm not moving that ;)
12:03:13 mdbooth sean-k-mooney: Feel free to write a follow-up patch
12:03:22 nicolasbock I can also check that with 'openstack resource provider allocation show 2aa3a324-bf22-4e0c-912a-d7c52f59f1fd'
12:03:57 sean-k-mooney mdbooth: ok i would have just done nested=common.nested in test.py
12:04:34 nicolasbock The VM is really running on 6cbb84b0-02f4-4ee3-9df2-151475b1effe
12:04:47 mdbooth sean-k-mooney: Sure. I don't want to mess with test.nested here, though.
12:04:52 nicolasbock But `openstack resource provider allocation set --allocation rp=6cbb84b0-02f4-4ee3-9df2-151475b1effe 2aa3a324-bf22-4e0c-912a-d7c52f59f1fd` is not working
12:05:31 sean-k-mooney mdbooth: ok ill submist a follow up patch. eventully... i have added it to my whiteboard
12:05:34 nicolasbock I suppose I am missing a `resource-class-name`
12:05:50 nicolasbock But what do I put there?
12:08:32 openstackgerrit Elod Illes proposed openstack/nova stable/ocata: Fix the help for the disk_weight_multiplier option https://review.openstack.org/607537
12:11:55 sean-k-mooney nicolasbock: was the vm migrated
12:12:09 nicolasbock Yes sean-k-mooney
12:12:38 sean-k-mooney mdbooth: was lookin at an edgecase recently where if cleanup on the source fail we dont update the vm host
12:14:29 sean-k-mooney nicolasbock: e.g. when you finish migrating the instace if the post migrate job on the source node failts to say unplug a vif we fail before we update the instace db record to refect that the vm is running on the new node
12:14:49 sean-k-mooney mdbooth: did you ever proposea a patch for ^
12:15:12 mdbooth sean-k-mooney: Probably.
12:17:27 sean-k-mooney nicolasbock: are the placement allocation currently associated with the vm correct for the host it is actully on
12:17:45 sean-k-mooney e.g. if you ignore the db host value and actully check the vms location
12:18:00 mdbooth sean-k-mooney: Trying to parse your comment here: https://review.openstack.org/#/c/605436/5/nova/compute/manager.py@1447
12:18:18 mdbooth Are you saying we can release the lock there?
12:18:25 mdbooth Or yield the context manager there?
12:18:35 mdbooth Or something else, because neither of ^^^ would be correct.
12:18:48 nicolasbock The VM is running on `6cbb84b0-02f4-4ee3-9df2-151475b1effe`
12:19:03 nicolasbock But placement says that it's on `57b0e4d5-3a3e-4cf3-ba8c-b88c8ce4679b`
12:19:09 sean-k-mooney mdbooth: im saying if we dont have a lock when we invoke that line the db request could cause use to yeild causeing a race
12:19:32 mdbooth Ah, *eventlet* yield
12:19:37 mdbooth Ok.
12:19:54 sean-k-mooney e.g. this is the thing that definetly needs to be in the critcal section of the lock
12:20:01 nicolasbock I ran `openstack server show 2aa3a324-bf22-4e0c-912a-d7c52f59f1fd`
12:20:15 nicolasbock and `openstack resource provider allocation show 2aa3a324-bf22-4e0c-912a-d7c52f59f1fd`
12:21:06 sean-k-mooney mdbooth: ya re reading that was not clear
12:21:53 sean-k-mooney nicolasbock: is 57b0e4d5-3a3e-4cf3-ba8c-b88c8ce4679b the source or destination of the migration
12:22:23 sean-k-mooney nicolasbock: im assume the destination correct?
12:22:36 sean-k-mooney sorry source
12:23:07 nicolasbock I don't know what happened, but I would guess that it is the source
12:23:27 nicolasbock Sorry, I am not sure I completely grasp the terminology of source and destination
12:23:43 sean-k-mooney ya so the resouce being used on 6cbb84b0-02f4-4ee3-9df2-151475b1effe are likely still owned by the migration object in placement
12:24:44 nicolasbock What's the migration object?
12:26:21 sean-k-mooney when you do a migration we create a migration record that we use to calim resouces on the destination host then when the vm is move we use a special atomic oepration in the placemnt api to change the allocation consumer form the migration recordds uuid to the vms uuid
12:27:23 nicolasbock So you are saying that that atomic operation wasn't executed?
12:27:30 sean-k-mooney yes
12:27:35 nicolasbock Ok

Earlier   Later