Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-10-03
17:02:17 mriedem dude, you want to talk about fiber. i've been eating raisin bran for breakfast and leftover chili for lunch most of this week.
17:02:25 efried ye gods
17:02:34 mriedem i only put together the reason for the side effects last night
17:05:04 sean-k-mooney lol isnit awsome that everything we type is archive for ever :)
17:06:21 dansmith artom: by "double download niggle" you mean "what happens if an image is re-cached" and/or "what happens if two requests come in for the same image quickly" right?
17:06:58 sean-k-mooney dansmith: i assume we have some kind of lock in the image cache to prevent that already right?
17:07:15 dansmith sean-k-mooney: we do, which is what I was going to say when he replied
17:07:47 sean-k-mooney cool ya because if we didnt it would be an issue for booting two vms on the same host concurrenly
17:08:07 dansmith sean-k-mooney: which is what I said in the review when we were discussing this yesterday yes :)
17:08:47 sean-k-mooney i still have that open to review. i got as far as the commit message then got distracted this morining
17:09:11 sean-k-mooney form what i skimed it makes sense to me so far
17:10:28 SonPham after i write the blueprint , i must upload new spec to https://opendev.org/openstack/nova-specs/src/branch/master/specs ?
17:11:46 openstackgerrit Matt Riedemann proposed openstack/nova master: Fix exception translation when creating volume https://review.opendev.org/678991
17:11:51 mnaser is there a point where placement will be the ultimate authority in nova for resource usage
17:11:59 mnaser (like, in the actual reporting too)
17:12:15 mnaser (i understand if no one wrote the code for it yet, but wondering if its more of a "no we wont do that)
17:12:51 mriedem SonPham: yes https://specs.openstack.org/openstack/nova-specs/readme.html
17:13:45 mriedem mnaser: i think that's the utopian goal
17:13:53 mriedem you can already configure nova to use placement for counting cpu and ram usage
17:13:56 artom dansmith, yes
17:13:58 openstackgerrit Stephen Finucane proposed openstack/nova master: WIP: Group IPs by network UUID, not network names https://review.opendev.org/686444
17:14:05 artom dansmith, the second one, specifically
17:14:16 openstackgerrit Sylvain Bauza proposed openstack/nova master: Add a placement audit command https://review.opendev.org/670112
17:14:30 mnaser wait, really (about nova for counting cpu and ram usage)
17:14:34 mriedem artom: also note that the windriver krew added a semaphore on image downloads awhile back
17:14:46 mriedem i think by default it's unbounded but they use it to serialize at edge sites
17:15:07 mriedem https://docs.openstack.org/nova/latest/configuration/config.html#compute.max_concurrent_disk_ops
17:15:30 artom mriedem, if that does what I think it does, then that's the piece I was missing
17:15:43 bauzas mriedem: gibi: dansmith: I know you folks are pretty busy those days but reviews would be appreciated for https://review.opendev.org/#/c/670112
17:15:50 bauzas (placement audit command)
17:15:52 mriedem mnaser: https://docs.openstack.org/nova/latest/configuration/config.html#quota.count_usage_from_placement
17:16:03 mriedem artom: note that's relatively new,
17:16:09 artom Although if set to 2 or more, it doesn't strictly speaking solve the simultaneous requests for the same image problem
17:16:10 mriedem the image cache might already have a lock on the image id
17:16:23 mriedem if only the code were freely available and online...
17:16:23 bauzas mriedem: gibi: dansmith: it's now in a good shape with a few functests, so before I'm writing unittests, I'd appreciate comments
17:16:25 SonPham any way to make a snapshot keep vms state?
17:16:41 artom But... since booting an image had the same problem, we probably have something in place fo that
17:16:43 artom *for
17:16:48 dansmith artom: ack, then it's already handled by a lock.. I looked it up but didn't call it out in the spec because it's existing
17:16:54 dansmith artom: so just FYI
17:17:03 mnaser mriedem: oh neat, that's pretty cool
17:17:06 artom mriedem, aww, but it's way more fun to blindly rant and speculate
17:17:36 artom dansmith, yeah that's fair, thanks for taking the time to look it up
17:18:44 mriedem artom: https://github.com/openstack/nova/blob/master/nova/virt/libvirt/imagebackend.py#L257 i think?
17:19:48 artom mriedem, looks like it, yeah
17:20:06 artom That imagecache code though... *shudder*
17:20:14 openstackgerrit Sylvain Bauza proposed openstack/nova master: Add a placement audit command https://review.opendev.org/670112
17:20:27 mriedem dansmith: that reminds me, one thing we'll likely want documented with using your thing is making sure a service user token is configured to avoid the user token timing out while triyng to download an image across a bunch of computes
17:20:42 mriedem https://docs.openstack.org/nova/latest/admin/support-compute.html#user-token-times-out-during-long-running-operations
17:21:00 dansmith mriedem: ack
17:23:30 mriedem bauzas: easy and fast -1 from me
17:23:33 mriedem you're welcome
17:25:29 mnaser but i dont think a token timing out during download might an issue
17:25:38 mnaser cause the request will be done with the download is done anyways
17:25:47 mnaser i think image upload are an issue for things like submitting properties/etc when dnoe
17:26:41 dansmith mnaser: we're talking about image pre-caching, where the process may take an hour
17:26:59 dansmith mnaser: so by the time the last compute starts doing the work, the token may have expired and it can't pull the image
17:27:20 mnaser oh i see, this isnt necessarily the on-vm-launch thing
17:27:21 mriedem lyarwood: i'm not sure i got your question correct in https://review.opendev.org/#/c/541420/
17:27:32 dansmith mnaser: unrelated to vm launch entirely
17:27:39 mnaser okay, right :>
17:28:48 dansmith mnaser: is image pre-caching a thing you care about btw? I think your cloud is too general purpose to care maybe, but... ?
17:29:23 dansmith or you use ceph so not a thing anyway
17:29:25 mnaser dansmith: i think in our case most of our stuff is ceph based, and yeah, too general purpose to have images precached *but* it might be something as we are considering having VMs that are entirely local storage
17:29:38 dansmith okay
17:30:11 mnaser so maybe having a glance property that we'd use to maintain a bunch of 'hot' images that we expect users to launch often (then precache those images in some ci job)
17:31:19 sean-k-mooney mnaser: with ceph we do the whole cow clone in ceph to cache the image once its added to ceph the first time but precaching could still be useful in edge deployment if they had ceph too right?
17:31:46 dansmith mnaser: well, it won't be that automatic yet, but you could write a script to key it off of an image prop
17:31:51 sean-k-mooney it might not help vexhosts usecase but precaching with ceph chould still be useful
17:32:14 mnaser precache with ceph COULD solve that thing i talkeda bout a long time when multiple cells with their own ceph cluster each
17:32:15 dansmith sean-k-mooney: ceph at the edge doesn't work very well right now
17:32:31 sean-k-mooney dansmith: due to the 3 node requiremnet?
17:32:35 mnaser dansmith: yep, but at least id have the tool available in that case yeah
17:32:43 dansmith sean-k-mooney: because nova will do the pathologically terrible thing with ceph if it's not the same one as glance uses
17:32:50 dansmith mnaser: yep
17:33:21 sean-k-mooney download the image from glance and uplaod ti to ceph repeatedly?
17:33:42 stephenfin efried: I recall seeing a comment from you about doing this kind of thing differently https://github.com/openstack/nova/blob/master/nova/tests/functional/api_sample_tests/api_sample_base.py#L133
17:34:14 stephenfin I should know what that alternative way is but I'm having a senior moment. Care to remind me?
17:34:25 stephenfin (allows us avoid the 'fake_noop' arg)
17:34:35 dansmith sean-k-mooney: yes so you end up with no re-use, but silently
17:34:44 sean-k-mooney ya that sucks
17:35:04 sean-k-mooney and wastes space on a ceph cluster that is proably already constratied
17:35:45 dansmith sean-k-mooney: s'mah point. This is what I mentioned at the last PTG which I will fix when glance gets the backend management APIs
17:35:50 sean-k-mooney and bandwith over a potentially slow link. isnt the multipl glace locaiton feature ment to help with that
17:36:18 dansmith sean-k-mooney: it is, but nova only has one it can handle, so if it's not already stored there, it will silently do the bad thing
17:36:19 sean-k-mooney ya i recall you talking about this
17:37:12 sean-k-mooney do you intend to adress that as part of the precaching spec or declare it out of scope and adress that seperatly
17:37:18 sean-k-mooney i woudl think seperatly
17:37:42 dansmith completely unrelated
17:38:11 sean-k-mooney well they both adress image distubution but ya that is what i was hoping you would say
17:38:52 sean-k-mooney im just preempting the possiblity of scope creap
17:55:11 sean-k-mooney has anyone ever tried to do an FFU across more then 3 release?
17:55:47 sean-k-mooney e.g. form say newton to train
17:56:12 sean-k-mooney i feel like that would be kind of a pain
17:58:42 efried self.useFixture(fixtures.MockPatch(...))
17:58:42 efried stephenfin: I was probably just griping about stub_out being a thin wrapper around
17:58:48 efried you could replace it everywhere
17:58:59 mriedem stephenfin: are you thinking of efried always saying to use new= on a mock to avoid the method signature change?
17:59:45 efried in many stub_out cases where the replacement is actually a no-op, you can MockPatch('just.the.thing') and you get a default mock, i.e. you can do away with the replacement stub function

Earlier   Later