Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-07-17
15:31:08 sean-k-mooney alex_xu: could we optionally clean the unsed namespaces on agent start
15:31:09 dansmith yeah, I mean, that's a serious failure, and all kinds of things could go wrong there
15:31:23 dansmith sean-k-mooney: yeah was just going to say.. that's probably safest
15:31:39 alex_xu ok, but it should be slow
15:31:51 sean-k-mooney we could proably store a file somewhere of things we are currently programing and delete the file if we completed properly
15:32:03 alex_xu it is ok to cleanup, since the live-migration won't be continue again
15:32:10 dansmith because even if you were tracking them some other way, that failure scenario, followed by a db archive before the compute node starts up again would result in not knowing about the previous failure
15:32:34 dansmith sean-k-mooney: yeah, there may be residue just from the incoming disk images and stuff actually
15:32:38 dansmith also,
15:32:44 dansmith there should be an uncompleted migration record in the db
15:32:52 dansmith which we already look for in a couple situations to clean up
15:33:17 sean-k-mooney the migration record wont have the vpmem info unfortuately
15:33:20 dansmith so maybe if there are aborted migrations against us when we start, we do a full clean for safety or something?
15:33:24 dansmith no, I know
15:33:39 sean-k-mooney so we really need to write somethign to the compute node disk to keep track of it
15:34:38 sean-k-mooney e.g. we need to store it locally incase teh instance is delete and archive before we restart teh agaent on the failed node
15:34:41 dansmith ...or just clean all unused namespaces on startup if we see an aborted live migration
15:34:47 efried ++
15:34:51 sean-k-mooney ya that too
15:34:56 efried alex_xu: ^
15:35:11 alex_xu yea, I'm ok with that
15:35:20 dansmith why is it slow btw?
15:35:22 sean-k-mooney but alex_xu was concerned about the time. i guess if its done asyc outside of the agent then its fine
15:35:23 efried need some way to mark them unavailable for spawns
15:35:30 dansmith shouldn't this be a TRIM type operation on some silicon?
15:35:55 efried don't you have to write zeroes to the whole thing, or something?
15:35:57 dansmith efried: block compute startup until it's finished like all our other "recover from failure" type things
15:36:03 efried ah, nice
15:36:04 dansmith efried: shouldn't, that would be terrible
15:36:06 alex_xu no..just write zero to each bit...
15:36:15 dansmith that's ridiculous
15:36:29 dansmith isn't this effectively flash?
15:37:14 sean-k-mooney dansmith: well we would have to do a secure erase rather then trim but ya optane is like flash but different
15:37:35 alex_xu I guess there is no cache, since it is write by the DAX way
15:37:54 dansmith sean-k-mooney: either, trim or dump the key.. either should be fast and avoid wearing the flash to friggin write zeroes
15:38:12 dansmith sean-k-mooney: so does that mean every time we destroy an instance using one of these that we have to block destroy until we've written gigabytes to this device?
15:38:20 sean-k-mooney alex_xu: i assume we can delgate the erase to a cli too and just do it async or in parallel
15:38:33 dansmith whereas if we were giving them flash we could just run a single trim command and move on?
15:38:35 alex_xu we are using the daxio
15:38:49 sean-k-mooney dansmith: i mean technically yes
15:39:11 sean-k-mooney that would be similar to the lvm image backend erase
15:39:27 dansmith sean-k-mooney: right, which is ridiculous and based on 1980s notions of disk, so I give it a pass
15:39:58 dansmith and also not really enjoyed by operators everywhere :)
15:40:10 sean-k-mooney alex_xu: to dansmith's point if we are using daxio can we just tell the device to nuke everythin in the namespace
15:40:33 alex_xu no, I don't think we have that kind of TRIM
15:40:43 efried this seems like a separate issue though. Whatever "clear" we have to do, we have to do the same thing on destroy()
15:40:55 sean-k-mooney ya
15:40:59 sean-k-mooney that is fair
15:41:07 dansmith efried: sure, it's more just that if we're going to massively slow down destroy, it'd be good to know that
15:41:15 dansmith so we can mock this "feature" if nothing else
15:41:22 sean-k-mooney the question that raises is can we have the agent wait to start up or not
15:41:23 efried IIRC that was in the spec
15:41:23 dansmith jay would be rolling over in his grave
15:41:38 sean-k-mooney could we play with the reserved value in placment
15:41:40 dansmith sean-k-mooney: we can and should, like anything else we do on this
15:41:50 dansmith I'm less concerned about the delayed compute startup than I am with destroy
15:42:11 sean-k-mooney e.g. make the dirty namespces reserved until we clean it
15:42:26 dansmith if you're billing by the hour, and a bunch of people destroy instances all at once and they all have to wait in line to have zeroes written to this new whizbang memory thing...
15:42:53 efried sean-k-mooney: if we change inventory on destroy() we're going to run into cdent's generation conflict race all over the place.
15:42:54 sean-k-mooney for what its worth it can do that at several GB/S
15:42:54 dansmith sean-k-mooney: that's a lot more complex and more opportunity to be interrupted in the middle and needing to be properly resumed
15:43:23 sean-k-mooney ok then lets just keep it simple
15:43:43 cdent (my patch to not do that, controlled by config, got merged internally)
15:43:44 sean-k-mooney and block untill they are all cleaned if their was a faild migration as ye suggested
15:44:03 dansmith sean-k-mooney: ++
15:45:05 sean-k-mooney apparently the frist gen stuff will hit 8.9 GB/s in sequtial writes
15:45:29 sean-k-mooney although the cleaning operation would likely impact the performce of other isntances if you tried to max it out
15:45:54 dansmith which is why it's stupid to have to do this
15:46:01 dansmith I mean honestly, wtf
15:46:15 dansmith cheap sata SSDs have not required writing zeros to erase for ten years
15:46:58 sean-k-mooney its because its been mapped as memory rather then via a filesystem
15:47:12 dansmith which is also stupid
15:47:14 dansmith however,
15:47:30 sean-k-mooney if you did use it via the filesystem interface which is ssupported it proably would handel this interally
15:47:31 dansmith you have a cli tool to manage it, so there should be some thing to do it.. doesn't have to be _actual_ ATA TRIM
15:48:05 dansmith ...but intel *has* to have this seem memory like for optics, so.. :/
15:48:26 sean-k-mooney well it littaly replaces ram dims
15:48:39 dansmith so?
15:48:40 sean-k-mooney and is byte adressable
15:48:49 sean-k-mooney so its pretty ram like
15:48:54 dansmith it's a block device to the guest yeah?
15:48:58 sean-k-mooney no
15:49:05 alex_xu it is ram for the guest
15:49:08 sean-k-mooney its exposed as dims to the guest
15:49:26 dansmith but people are going to use it as semi-persistent storage right?
15:49:33 sean-k-mooney no
15:49:37 sean-k-mooney well indirectly
15:49:51 sean-k-mooney they will be mmaping it into there application adress space
15:49:56 dansmith right
15:50:16 sean-k-mooney and using it as an "in memeory" cache / scratch space for there workign set
15:50:19 dansmith which works fine for block-addressable storage, to look like byte-addressable memory, even for rotating rust :)
15:51:28 dansmith anyway, whatever, I know this ship has sailed
15:51:43 sean-k-mooney sure :) you know we live in a world of hardward defiend software right :P
15:51:51 dansmith all too well
15:55:34 alex_xu dansmith: efried sean-k-mooney anyway...thank you guys
15:55:47 openstackgerrit Dakshina Ilangovan proposed openstack/nova-specs master: Spec: Provider config YAML file https://review.opendev.org/612497
15:56:18 zzzeek is mriedem on vacation
15:56:38 dansmith zzzeek: yes
15:56:47 zzzeek dansmith: hokay
16:01:50 edleafe b

Earlier   Later