Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-10-03
11:45:16 mdbooth You just ignore eventlets entirely and assume you have multiple concurrent threads. How they're implemented isn't all that important in that code.
11:45:32 mdbooth If we later switched to a multi-threaded model, it would still be fine.
11:46:07 sean-k-mooney mdbooth: well if you were new to nova or did not think about it at the time then you can write races because of eventlets easier then if it was expcitly threaded
11:46:47 mdbooth Honestly, I never consider eventlets. I assume it's explicitly threaded.
11:47:11 mdbooth It's not, but that doesn't have any bearing on writing safe code, except when there's bugs in eventlet.
11:47:39 sean-k-mooney mdbooth: most new people i have talked to that work on openstack dont think about threads at all because the say oh its python and that has a gil so i dont need to care
11:48:13 mdbooth sean-k-mooney: That is just one of many issues with python in the real world :(
11:48:59 mdbooth Also, the gil doesn't prevent overlapping threads of execution, it just stops them running at the same time. The problems are the same.
11:49:15 sean-k-mooney mdbooth: yes and no. there perception is normally correct. its eventlets that violates the paradime
11:49:42 mdbooth As an old curmudgeon, I think python has been extremely detrimental to software engineering, particular in education
11:50:32 mdbooth No, it would not be correct. If you have a multi-threaded python program, even though the gil prevents multiple threads running concurrently, you still need locks.
11:50:43 sean-k-mooney mdbooth: i self taught myself c++ as my first langage then java so ya i like to understand exactly what is going on
11:51:46 mdbooth Basically: eventlets or python-multithreading: I don't care. It shouldn't change how your write code.
11:51:49 sean-k-mooney learning c++ fist made me a better engineer then learing python would have.
11:52:04 mdbooth They both use fake threading.
11:53:03 openstackgerrit Matthew Booth proposed openstack/nova master: Run evacuate tests with local/lvm and shared/rbd storage https://review.openstack.org/604400
11:53:13 mdbooth I think ^^^ might fix that weird issue I was hitting
11:53:42 mdbooth There's a shortcut in service_is_up if the service is forced down, and we forced it down
11:54:29 sean-k-mooney mdbooth: yes i know if it was expcitly multi thraded it would be incorrect without the lock. anyway cool ill take a look at that too
11:55:10 mdbooth sean-k-mooney: Don't worry about ^^^ btw. Will just wait until zuul has voted.
11:55:22 mdbooth sean-k-mooney: I knew I was missing something simple there.
11:55:53 sean-k-mooney waith why was it forced down?
11:56:12 sean-k-mooney oh so you could evacuate
11:56:16 mdbooth The test_evacuate.sh script which ran before the second round...
11:56:18 mdbooth yeah
11:57:22 sean-k-mooney ha yet anthoer race this tiem between test :)
12:00:22 nicolasbock Morning
12:01:58 nicolasbock I have a run-away server, i.e. a VM that's running on a hypervisor different than what Nova thinks. So far I haven't quite been able to get the placement service to help me update Nova's view of reality...
12:02:22 nicolasbock I can see the server with 'openstack server show 2aa3a324-bf22-4e0c-912a-d7c52f59f1fd'
12:02:36 nicolasbock And it lists the wrong hypervisor
12:02:47 mdbooth 348
12:02:47 mdbooth sean-k-mooney: $ git grep test\.nested | wc -l
12:02:54 mdbooth I'm not moving that ;)
12:03:13 mdbooth sean-k-mooney: Feel free to write a follow-up patch
12:03:22 nicolasbock I can also check that with 'openstack resource provider allocation show 2aa3a324-bf22-4e0c-912a-d7c52f59f1fd'
12:03:57 sean-k-mooney mdbooth: ok i would have just done nested=common.nested in test.py
12:04:34 nicolasbock The VM is really running on 6cbb84b0-02f4-4ee3-9df2-151475b1effe
12:04:47 mdbooth sean-k-mooney: Sure. I don't want to mess with test.nested here, though.
12:04:52 nicolasbock But `openstack resource provider allocation set --allocation rp=6cbb84b0-02f4-4ee3-9df2-151475b1effe 2aa3a324-bf22-4e0c-912a-d7c52f59f1fd` is not working
12:05:31 sean-k-mooney mdbooth: ok ill submist a follow up patch. eventully... i have added it to my whiteboard
12:05:34 nicolasbock I suppose I am missing a `resource-class-name`
12:05:50 nicolasbock But what do I put there?
12:08:32 openstackgerrit Elod Illes proposed openstack/nova stable/ocata: Fix the help for the disk_weight_multiplier option https://review.openstack.org/607537
12:11:55 sean-k-mooney nicolasbock: was the vm migrated
12:12:09 nicolasbock Yes sean-k-mooney
12:12:38 sean-k-mooney mdbooth: was lookin at an edgecase recently where if cleanup on the source fail we dont update the vm host
12:14:29 sean-k-mooney nicolasbock: e.g. when you finish migrating the instace if the post migrate job on the source node failts to say unplug a vif we fail before we update the instace db record to refect that the vm is running on the new node
12:14:49 sean-k-mooney mdbooth: did you ever proposea a patch for ^
12:15:12 mdbooth sean-k-mooney: Probably.
12:17:27 sean-k-mooney nicolasbock: are the placement allocation currently associated with the vm correct for the host it is actully on
12:17:45 sean-k-mooney e.g. if you ignore the db host value and actully check the vms location
12:18:00 mdbooth sean-k-mooney: Trying to parse your comment here: https://review.openstack.org/#/c/605436/5/nova/compute/manager.py@1447
12:18:18 mdbooth Are you saying we can release the lock there?
12:18:25 mdbooth Or yield the context manager there?
12:18:35 mdbooth Or something else, because neither of ^^^ would be correct.
12:18:48 nicolasbock The VM is running on `6cbb84b0-02f4-4ee3-9df2-151475b1effe`
12:19:03 nicolasbock But placement says that it's on `57b0e4d5-3a3e-4cf3-ba8c-b88c8ce4679b`
12:19:09 sean-k-mooney mdbooth: im saying if we dont have a lock when we invoke that line the db request could cause use to yeild causeing a race
12:19:32 mdbooth Ah, *eventlet* yield
12:19:37 mdbooth Ok.
12:19:54 sean-k-mooney e.g. this is the thing that definetly needs to be in the critcal section of the lock
12:20:01 nicolasbock I ran `openstack server show 2aa3a324-bf22-4e0c-912a-d7c52f59f1fd`
12:20:15 nicolasbock and `openstack resource provider allocation show 2aa3a324-bf22-4e0c-912a-d7c52f59f1fd`
12:21:06 sean-k-mooney mdbooth: ya re reading that was not clear
12:21:53 sean-k-mooney nicolasbock: is 57b0e4d5-3a3e-4cf3-ba8c-b88c8ce4679b the source or destination of the migration
12:22:23 sean-k-mooney nicolasbock: im assume the destination correct?
12:22:36 sean-k-mooney sorry source
12:23:07 nicolasbock I don't know what happened, but I would guess that it is the source
12:23:27 nicolasbock Sorry, I am not sure I completely grasp the terminology of source and destination
12:23:43 sean-k-mooney ya so the resouce being used on 6cbb84b0-02f4-4ee3-9df2-151475b1effe are likely still owned by the migration object in placement
12:24:44 nicolasbock What's the migration object?
12:26:21 sean-k-mooney when you do a migration we create a migration record that we use to calim resouces on the destination host then when the vm is move we use a special atomic oepration in the placemnt api to change the allocation consumer form the migration recordds uuid to the vms uuid
12:27:23 nicolasbock So you are saying that that atomic operation wasn't executed?
12:27:30 sean-k-mooney yes
12:27:35 nicolasbock Ok
12:27:55 nicolasbock Can I get it to execute?
12:28:01 sean-k-mooney so if you do nova server-migration-list 2aa3a324-bf22-4e0c-912a-d7c52f59f1fd does it have a migration object listed
12:28:53 nicolasbock No
12:30:10 sean-k-mooney oh hum strange. perhaps the migration has already been confirmed.
12:31:13 sean-k-mooney nicolasbock: efried might be able to help better the i if he is around
12:31:28 nicolasbock So I thought that `openstack resource provider allocation set` would allow me to update the DB
12:31:39 nicolasbock Thanks sean-k-mooney !
12:32:06 nicolasbock But I am not using that command correctly since it complains about an incorrect 'allocation string format'
12:36:24 openstackgerrit Vlad Gusev proposed openstack/nova stable/pike: libvirt: Use os.stat and os.path.getsize for RAW disk inspection https://review.openstack.org/607544
12:37:16 openstackgerrit Matthew Booth proposed openstack/nova master: DNM: Run against mriedem's evacuate test https://review.openstack.org/604423
12:44:15 mdbooth This is an interesting query: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22AssertionError%3A%20u'host3'%20%3D%3D%20u'host3'%5C%22
12:44:49 mdbooth I wonder why that started spiking only a few days ago: code change, or infra change?
13:03:06 stephenfin lyarwood: RE: https://review.openstack.org/588570 I'd been putting it off but will do that now. Will keep you posted
13:04:26 lyarwood stephenfin: cheers
13:06:26 openstack Launchpad bug 1209101 in OpenStack Compute (nova) "Non-public flavor cannot be used in created tenant" [High,Fix released] - Assigned to Sumanth Nagadavalli (sumanth-nagadavalli)
13:06:26 s10 How can I reopen bug https://bugs.launchpad.net/nova/+bug/1209101 ? It still exists.
13:07:02 sean-k-mooney s10: just change status and leave a comment with details
13:08:02 s10 sean-k-mooney: I've left comment, but I can't change status, all of them are grey.
13:08:38 sean-k-mooney s10: that said i dont think its nessisarly the same but. that was fixed in 2013
13:08:47 sean-k-mooney its more likely a regression
13:08:59 sean-k-mooney are you seeing this on master?
13:09:51 s10 sean-k-mooney: Yes, this regression have never been fixed. I will write a comment about how to reproduce it.
13:10:56 sean-k-mooney well it was fixed and then reverted so this likely needs to be treated as more then a bug fix but rather as a blueprint/spec
13:13:21 sean-k-mooney the original bug fix predates microversions so i think a mini spec + microversion but would be required to alter the api behavior
13:13:57 sean-k-mooney bauzas: is ^ correct

Earlier   Later