Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-03-07
10:53:31 openstackgerrit Akira KAMIO proposed openstack/nova master: Replace root volume during rebuild https://review.openstack.org/442295
11:07:13 openstackgerrit Evgeny Antyshev proposed openstack/nova master: get_model method missing for Ploop image https://review.openstack.org/441989
11:12:56 jvarlamova__ rpodolyaka: http://pastebin.com/0jY6Ax1C
11:20:13 rpodolyaka jvarlamova__: this exception is raised when nova-compute times out waiting for vif-plugged event - https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L5052-L5061
11:21:18 rpodolyaka jvarlamova__: you should check nova-api / neutron-server / neutron l2 agent logs for this device id (== instance uuid) to see if such event was delivered to nova at all
11:27:41 mdbooth alex_xu: Hey, just thinking about your response on https://review.openstack.org/#/c/441829/4 about the cache.clear(). This is all running with the placement service, right? Are we guaranteed to only have 1 of those globally?
11:27:52 mdbooth cdent: ^^^ ?
11:28:27 mdbooth alex_xu: I'm just thinking that if a Trait is created through 1 service, then accessed through another, the cache on the second won't have been refreshed.
11:28:47 mdbooth However, I don't know how this is intended to be deployed, so this may not make sense.
11:29:31 cdent mdbooth: there's only one placement database, but could be multiple placement api servers
11:29:44 mdbooth cdent: That's what I would have expected. Thanks.
11:30:15 cdent I'm a bit nonplussed about the traits and resource class caches
11:30:35 mdbooth cdent: Explain
11:30:56 cdent they are trying to optimize sql queries when some of the data is not in an rdbms
11:31:09 cdent and thus they are...weird
11:31:21 cdent I would have prefered the optimizations to come later
11:33:24 cdent mdbooth: bonus points for "Explain"
11:33:32 mdbooth :)
11:34:03 mdbooth cdent: They look like they're getting all their data from the db
11:34:08 mdbooth What did I miss?
11:34:23 mdbooth For context: I never looked at this before yesterday
11:34:37 cdent which "they" did you mean? the traits cache may be different (and I haven't looked closely at it yet)
11:35:00 mdbooth 'they' == resource class and traits caches
11:35:35 mdbooth cdent: Ah... Yes, some resource classes are hard-coded
11:35:45 mdbooth I've slept since I read that
11:36:03 cdent you sleep?
11:36:10 mdbooth Don't tell anyone
11:37:10 jvarlamova__ rpodolyaka: As I see from logs, neutron sent 'network-vif-plugged' event for that instance to nova, and there's message "Creating event network-vif -plugged ...for instance ..." in nova-api logs. Looks like nova received event.
11:41:49 alex_xu mdbooth: ah...i got you, good point
11:45:34 alex_xu mdbooth: cdent probably resource-class won't have such problem. the cache for resource-class is just for query single resource-class.
11:47:41 mdbooth alex_xu: The negative lookup is going to be tough for the startswith match, I think.
11:48:04 cdent alex_xu: is it possible to avoid having the cache through the initial implementation, and only add the cache at the end?
11:48:08 mdbooth alex_xu: Whereas the other 2 are definite lookups: it exists or it doesn't, and you normally expect that it exists
11:48:26 cdent building in cache from the outset almost always leads to harder to maintain code
11:48:29 mdbooth Returning no values or some values doesn't necessarily mean that you're not out of date
11:50:06 mdbooth cdent: How often will we hit this thing? Would an index optimised for a startswith query be appropriate?
11:50:47 cdent (thus my concerns about early optimizations and speculations before measuring, etc)
11:50:53 mdbooth From reading the context, the expectation seems to be a lot. Whose was that expectation?
11:51:02 alex_xu cdent: in the poc, actually I query the db directly. but I try to get the API "GET /traits?name=in:TRAIT1,TRAIT2" to use the cache, since that is the API used to validate whether the traits is valid or not
11:51:18 cdent mdbooth: it will be alot, eventually
11:51:31 cdent but that's based on abstractions, not measurement
11:52:03 mdbooth Are we carefully avoiding the db throughout this whole code path?
11:52:36 alex_xu wait, no, that is for "GET /traits?prefix=xxx", that can be from the db directly. "GET /traits?name=in:TRAIT1,TRAIT2" can get benefit from the single trait lookup
11:53:24 cdent but I haven't had a chance yet
11:53:29 mdbooth alex_xu: Yeah, I think the concrete lookup is ok to cache. Either you get a hit (but wait... what if it was deleted?), or you don't, in which case you can refresh.
11:53:30 cdent (been concentrating on the spec)
11:54:33 alex_xu mdbooth: yea, if user query a non-existed trait, that will trigger a db call
11:54:55 mdbooth alex_xu: Deleting a cached trait/resource class is also an interesting case.
11:55:38 mdbooth Does the user have an expectation that after a delete call completes, a subsequent lookup will fail?
11:55:43 mdbooth What if it doesn't?
11:55:59 mdbooth I suspect that probably doesn't matter as much as the other case.
11:56:48 alex_xu yes, I remember I point out that problem from the resource class patch, but I forget what answer I got now
11:57:07 cdent you guys are proving my point
11:57:10 cdent first make it right
11:57:12 cdent then add the cache
11:57:49 mdbooth alex_xu: Would cdent's suggestion fly?
11:57:52 jvarlamova__ rpodolyaka: by the way, some instances were created successfully while other got an exception. Is it a known issue of parallel instances creation?
11:58:10 alex_xu cdent: all the cache? even for single trait lookup?
11:59:03 cdent theoretically, yes: that's just good practice: make code that does the right thing
11:59:10 cdent only once it is doign the right thing consider adding cache
11:59:22 cdent the mix between custom traits and the os-traits library makes things a bit weird in all this
11:59:53 cdent but since the number of traits is, let's say, less than 10,000 python can happily work with lists of strings of that size
12:00:23 cdent but again: I haven't looked at the codet yet! :)
12:00:49 cdent I just hear a lot of early optimizating, and feel like that's worth flagging before we get too caught up in the details of that when we don't even have a fully working api yet
12:00:56 alex_xu cdent: yea, that sounds make sense. hope we can catch jay also. I feel he will say something when I remove the cache :)
12:01:14 cdent I'm sure he will, jay is a notorious early optimizer :)
12:01:39 alex_xu cdent: heh :)
12:01:47 mdbooth alex_xu: Anyway, I'm going to remind you again that I don't have nearly enough context here. I can only point out potential problems and ask if they're real problems.
12:02:02 mdbooth alex_xu: Don't mean to be a pita.
12:02:22 cdent if what we're really concerned about is performance of the api then we should, _later_, consider a namespaced memcached layer on top of the placement api that allows marking collections as dirty
12:02:35 alex_xu mdbooth: appreciate your review same, it is really helpful :)
12:02:54 cdent alex_xu, mdbooth: I think we should have lots more of this type of conversation, so I think this is good
12:03:10 alex_xu mdbooth: btw, the spec is at https://review.openstack.org/345138, in case you are interesting :)
12:03:29 mdbooth alex_xu: Thanks, I should probably have read that first.
12:04:12 cdent I think jay's concerns are at the database layer, in which case we can address that by simply having all the traits in the same table, and isn't that being done when we sync from os-traits?
12:04:56 alex_xu cdent: yes, all the traits in one table
12:05:07 cdent then why have a cache at all?
12:05:11 alex_xu by 'placement-manage os-traits sync'
12:06:16 alex_xu cdent: I guess jay think the trait/resource class won't update frequently, then to same some db call...
12:06:37 cdent :)
12:06:37 cdent -1
12:06:43 alex_xu :)
12:06:58 cdent with the resource class stuff the cache is because the classes are _not_ all int he same table
12:07:10 cdent with traits, if they are, then...
12:07:32 alex_xu ah...
12:07:40 alex_xu yea a abstraction layer
12:09:40 alex_xu that is why I put the traits sync in the cache before, now we pull that sync out..
12:09:47 openstackgerrit Takashi NATSUME proposed openstack/nova master: List/show all server migration types https://review.openstack.org/430608
12:10:44 cdent mdbooth: in a round about way you have done a good deed. if your quota is one a day, you're done, good job, time for a beer.
12:22:30 openstackgerrit Takashi NATSUME proposed openstack/nova-specs master: List/show all server migration types https://review.openstack.org/407237
12:24:25 openstackgerrit Danil Akhmetov proposed openstack/nova master: Use proper user and tenant in the owner section of libvirt.xml. https://review.openstack.org/399679
12:27:28 gibi bauzas: Hi! thanks for looking at the scheduler hint API spec. I suggested a way to handle missing hints in the review.
12:28:50 rfolco cdent, review_begging++ https://review.openstack.org/#/c/415885/
12:31:12 cdent rfolco: yup that's on my list for today
12:31:23 rfolco cdent, thx :)
12:32:56 lyarwood mdbooth: have you played around with lvm2 and sparse files yet?
12:33:14 lyarwood mdbooth: we are already creating thinp volumes AFAICT
12:34:25 lyarwood mdbooth: http://paste.openstack.org/show/601745/
12:34:39 openstackgerrit Steven Webster proposed openstack/nova master: Update nova network info when doing rebuild for evacuate operation https://review.openstack.org/382853
13:15:17 jsheeren hi, i'm having issues with nova-spiceproxy and spice-html5 using Windows instances. Everything is upside down, which makes it a little difficult to use the horizon console. Has anyone seen this as well?

Earlier   Later