Cassandra 1.0, the cloud, and the future of big data
( -or- http://j.mp/qPyuOL )
-- Ed Leafe
_______________________________________________
Post Messages to: ProFox@leafe.com
Subscription Maintenance: http://leafe.com/mailman/listinfo/profox
OT-free version of this list: http://leafe.com/mailman/listinfo/profoxtech
Searchable Archive: http://leafe.com/archives/search/profox
This message: http://leafe.com/archives/byMID/profox/73BD4917-C23E-4886-BD09-E2156ADA4D71@leafe.com
** All postings, unless explicitly stated otherwise, are the opinions of the author, and do not constitute legal or medical advice. This statement is added to the messages for those lawyers who are too stupid to see the obvious.
On 09/22/11 09:03, Ed Leafe wrote:
> Cassandra 1.0, the cloud, and the future of big data
>
>
> ( -or- http://j.mp/qPyuOL )
>
> -- Ed Leafe
Wow! That's the most new terms I had to look up since the MS press
release for Win 8.
Kind of makes my head hurt, but I do see the problem.
Dan Covill
_______________________________________________
Post Messages to: ProFox@leafe.com
Subscription Maintenance: http://leafe.com/mailman/listinfo/profox
OT-free version of this list: http://leafe.com/mailman/listinfo/profoxtech
Searchable Archive: http://leafe.com/archives/search/profox
This message: http://leafe.com/archives/byMID/profox/4E7B6B5A.1040109@san.rr.com
** All postings, unless explicitly stated otherwise, are the opinions of the author, and do not constitute legal or medical advice. This statement is added to the messages for those lawyers who are too stupid to see the obvious.
On Sep 22, 2011, at 12:07 PM, Dan Covill wrote:
> Wow! That's the most new terms I had to look up since the MS press
> release for Win 8.
It took me several days of working with Cassandra until I could wrap my head around it. I have been working with data in relational sets for so long that the notion of working with a non-relational database seemed like reading Chinese. But I'm happy to say that I picked it up much faster than I would have become fluent in Chinese!
Non-relational data is not better or worse than relational - it's simply solving a different type of problem with a different approach.
-- Ed Leafe
_______________________________________________
Post Messages to: ProFox@leafe.com
Subscription Maintenance: http://leafe.com/mailman/listinfo/profox
OT-free version of this list: http://leafe.com/mailman/listinfo/profoxtech
Searchable Archive: http://leafe.com/archives/search/profox
This message: http://leafe.com/archives/byMID/profox/2CF328B0-0FEB-4C0A-A985-67D761EEE764@leafe.com
** All postings, unless explicitly stated otherwise, are the opinions of the author, and do not constitute legal or medical advice. This statement is added to the messages for those lawyers who are too stupid to see the obvious.
Author: <kamcginnis@gmail.com>
Posted: 2011-09-22 14:01:58 Link
It will be interesting to see how this progresses. I noticed in the link you sent that Facebook has 50TB and ebay has
2PB.
A long time ago there were some messages in this newsgroup about storing images in a field in the data table. The
consensus (I think) was that you should not do that. You should only store a pointer to a file on the disk with your
image or other very large file. The difference in this and NoSQL is that you can only find the file by whatever you used
in the data table to point to the file. NoSQL seems to be a way to search in the folder where you store the large files?
I would be interested to know how they do that. I don't see any specifics in those links.
Our company has had a EHR (electronic healthcare record) for years, but we recommend that the doctor store a simple note
like those in a paper chart. We definitely do not recommend trying to store scanned images (tiff or pdf, etc) or
pictures (jpg, etc). At this time we partnered with a company x-link.info to link our database to many 'certified' EHR
programs. To be 'certified' those EHR programs must store lab tests, x-rays, etc. For a small internal medicine group of
doctors (3-8 for example) that might be many terabytes for 7 years of data. How would you backup 7 TB data? If this new
healthcare bills survives the constitutional test, it might create a lot of work for companies that have 'server farms'
----- Original Message -----
From: "Ed Leafe" <ed@leafe.com>
To: "ProFox Email List" <profox@leafe.com>
Sent: Thursday, September 22, 2011 10:19 AM
Subject: Re: [NF] For those interested in Big Data
On Sep 22, 2011, at 12:07 PM, Dan Covill wrote:
> Wow! That's the most new terms I had to look up since the MS press
> release for Win 8.
It took me several days of working with Cassandra until I could wrap my head around it. I have been working with data in
relational sets for so long that the notion of working with a non-relational database seemed like reading Chinese. But
I'm happy to say that I picked it up much faster than I would have become fluent in Chinese!
Non-relational data is not better or worse than relational - it's simply solving a different type of problem with a
different approach.
-- Ed Leafe
[excessive quoting removed by server]
_______________________________________________
Post Messages to: ProFox@leafe.com
Subscription Maintenance: http://leafe.com/mailman/listinfo/profox
OT-free version of this list: http://leafe.com/mailman/listinfo/profoxtech
Searchable Archive: http://leafe.com/archives/search/profox
This message: http://leafe.com/archives/byMID/profox/033701cc7951$b93e1ff0$7a00a8c0@w2k3s02
** All postings, unless explicitly stated otherwise, are the opinions of the author, and do not constitute legal or medical advice. This statement is added to the messages for those lawyers who are too stupid to see the obvious.
On 9/22/11 11:01 AM, kamcginnis@gmail.com wrote:
> NoSQL seems to be a way to search in the folder where you store the large files?
> I would be interested to know how they do that. I don't see any specifics in those links.
I think most people just store the image data, video data, whatever, as a node in the
database. You want to limit the number of things to search for and pulling the
pointer from the database and then the file from the filesystem wouldn't be as fast
as just pulling the blob from the db to begin with.
Paul
_______________________________________________
Post Messages to: ProFox@leafe.com
Subscription Maintenance: http://leafe.com/mailman/listinfo/profox
OT-free version of this list: http://leafe.com/mailman/listinfo/profoxtech
Searchable Archive: http://leafe.com/archives/search/profox
This message: http://leafe.com/archives/byMID/profox/4E7B7F7F.5060009@ulmcnett.com
** All postings, unless explicitly stated otherwise, are the opinions of the author, and do not constitute legal or medical advice. This statement is added to the messages for those lawyers who are too stupid to see the obvious.
Paul,
> I think most people just store the image data, video data, whatever, as a node in the database. You want to limit the number of things to search for and pulling the pointer from the database and then the file from the filesystem wouldn't be as fast as just pulling the blob from the db to begin with.
I respectfully disagree with your position. There was a recent thread on
the SQLite mailing list that included some benchmarks that argued
against use of blobs for storing files. While much of the discussion was
SQLite specific, I think that many of the principles would also apply to
other databases.
Some generic reasons to store files outside of a database include the
ability to use lower cost/speed file systems for file storage, faster
database backups, and faster database maintenance activities when
schemas change. Also much less database fragmentation.
The only reason I can think to store files inside a database is for
convenience, eg. having a single container for all your data.
Curious to hear other opinions on this topic.
Malcolm
_______________________________________________
Post Messages to: ProFox@leafe.com
Subscription Maintenance: http://leafe.com/mailman/listinfo/profox
OT-free version of this list: http://leafe.com/mailman/listinfo/profoxtech
Searchable Archive: http://leafe.com/archives/search/profox
This message: http://leafe.com/archives/byMID/profox/1316870136.3146.140258147237085@webmail.messagingengine.com
** All postings, unless explicitly stated otherwise, are the opinions of the author, and do not constitute legal or medical advice. This statement is added to the messages for those lawyers who are too stupid to see the obvious.
On Sat, Sep 24, 2011 at 8:15 AM, Malcolm Greene <profox@bdurham.com> wrote:
> Paul,
>
>> I think most people just store the image data, video data, whatever, as a node in the database. You want to limit the number of things to search for and pulling the pointer from the database and then the file from the filesystem wouldn't be as fast as just pulling the blob from the db to begin with.
>
> I respectfully disagree with your position. There was a recent thread on
> the SQLite mailing list that included some benchmarks that argued
> against use of blobs for storing files. While much of the discussion was
> SQLite specific, I think that many of the principles would also apply to
> other databases.
>
> Some generic reasons to store files outside of a database include the
> ability to use lower cost/speed file systems for file storage, faster
> database backups, and faster database maintenance activities when
> schemas change. Also much less database fragmentation.
>
> The only reason I can think to store files inside a database is for
> convenience, eg. having a single container for all your data.
---------------
I think HIPAA rules state that you have to keep the image in a secure
fashion, more about the alteration and loss of images to a patient as
well as who did any of it. All that distills down to the storage of
images in the db and that one table is usually the PIG in the db in
size. When I was a db for a EMR company one client had an 800 gig
table for images alone.
That made restore the hard time to overcome.
When that big client left I think they took the storage to Racker county.
--
Stephen Russell
Unified Health Services
60 Germantown Court
Suite 220
Cordova, TN 38018
Telephone: 888.510.2667
901.246-0159 cell
_______________________________________________
Post Messages to: ProFox@leafe.com
Subscription Maintenance: http://leafe.com/mailman/listinfo/profox
OT-free version of this list: http://leafe.com/mailman/listinfo/profoxtech
Searchable Archive: http://leafe.com/archives/search/profox
This message: http://leafe.com/archives/byMID/profox/CAJidMY+OSQtHkCUy1_2xWi2TrzByGecDOOEtWfUkWvOCGCRUQQ@mail.gmail.com
** All postings, unless explicitly stated otherwise, are the opinions of the author, and do not constitute legal or medical advice. This statement is added to the messages for those lawyers who are too stupid to see the obvious.
On 09/24/11 06:15, Malcolm Greene wrote:
> I respectfully disagree with your position. There was a recent thread on
> the SQLite mailing list that included some benchmarks that argued
> against use of blobs for storing files. While much of the discussion was
> SQLite specific, I think that many of the principles would also apply to
> other databases.
Since SQLite is stored in a single file (like Access!), I'd expect
storing blobs in the DB to be inefficient. But that's SQLITE. Other DB
structures may have very different results.
> Some generic reasons to store files outside of a database include the
> ability to use lower cost/speed file systems for file storage, faster
> database backups, and faster database maintenance activities when
> schemas change. Also much less database fragmentation.
>
> The only reason I can think to store files inside a database is for
> convenience, eg. having a single container for all your data.
I submit that another very good reason to store files inside a database
is to be sure they're there when you want to read them! If your DB has
only a file name, you have to do something else to ensure that that file
name will continue to exist. In most cases that 'something' looks
pretty much like part of the database - i.e., same server, same main
folder, same Administrator, etc.
For VFP, I would agree it's usually better if they're outside the .DBF
itself, but that may well not be true for SQL Server or Oracle. And I
suspect it's definitely NOT true for BigTable or Cassandra.
Dan Covill
San Diego
_______________________________________________
Post Messages to: ProFox@leafe.com
Subscription Maintenance: http://leafe.com/mailman/listinfo/profox
OT-free version of this list: http://leafe.com/mailman/listinfo/profoxtech
Searchable Archive: http://leafe.com/archives/search/profox
This message: http://leafe.com/archives/byMID/profox/4E7E51B9.5060504@san.rr.com
** All postings, unless explicitly stated otherwise, are the opinions of the author, and do not constitute legal or medical advice. This statement is added to the messages for those lawyers who are too stupid to see the obvious.
> Since SQLite is stored in a single file (like Access!), I'd expect
> storing blobs in the DB to be inefficient. But that's SQLITE. Other DB
> structures may have very different results.
SQL Server has FileStream for the best of both the worlds
http://technet.microsoft.com/en-us/library/bb933993.aspx
_Ajit Abraham
_______________________________________________
Post Messages to: ProFox@leafe.com
Subscription Maintenance: http://leafe.com/mailman/listinfo/profox
OT-free version of this list: http://leafe.com/mailman/listinfo/profoxtech
Searchable Archive: http://leafe.com/archives/search/profox
This message: http://leafe.com/archives/byMID/profox/4E7EBB25.3090306@ua-group.com
** All postings, unless explicitly stated otherwise, are the opinions of the author, and do not constitute legal or medical advice. This statement is added to the messages for those lawyers who are too stupid to see the obvious.
Malcolm,
Been there, done that in terms of storing outside the database. Fine when it
is small amount of data but a nightmare from a security point of view with
large amounts of data.
We moved soma 600Gb of scanned & audio files from a VFP database (stored
only a link to the file) because of thise. Unfortunately if your folder
structure gets fouled up or slow to access say because you have more than
about 5,000 files in a folder then it totally screws you. Also people are
tempted to meddle with data in folders and a simple inadvertand rename of a
folder or file can render your data useless.
Now at least I sleep knowing the data is safe in SQL Server and is being
mirrored, snapshot and replicated without me doing anything. Also access is
of course quick and doesn't degrade. Currently the initial 600Gb has
increased to over 3Tb as the amount of paperwork has been reduced in the
company and it still runs as fast. Also, nobody can get at it from a
security point of view without knowing how to access SQL, so our company
security auditors are happy.
Dave
-----Original Message-----
From: profox-bounces@leafe.com [mailto:profox-bounces@leafe.com] On Behalf
Of Malcolm Greene
Sent: 24 September 2011 14:16
To: profox@leafe.com
Subject: Re: [NF] For those interested in Big Data
Paul,
> I think most people just store the image data, video data, whatever, as a
node in the database. You want to limit the number of things to search for
and pulling the pointer from the database and then the file from the
filesystem wouldn't be as fast as just pulling the blob from the db to begin
with.
I respectfully disagree with your position. There was a recent thread on
the SQLite mailing list that included some benchmarks that argued
against use of blobs for storing files. While much of the discussion was
SQLite specific, I think that many of the principles would also apply to
other databases.
Some generic reasons to store files outside of a database include the
ability to use lower cost/speed file systems for file storage, faster
database backups, and faster database maintenance activities when
schemas change. Also much less database fragmentation.
The only reason I can think to store files inside a database is for
convenience, eg. having a single container for all your data.
Curious to hear other opinions on this topic.
Malcolm
[excessive quoting removed by server]
_______________________________________________
Post Messages to: ProFox@leafe.com
Subscription Maintenance: http://leafe.com/mailman/listinfo/profox
OT-free version of this list: http://leafe.com/mailman/listinfo/profoxtech
Searchable Archive: http://leafe.com/archives/search/profox
This message: http://leafe.com/archives/byMID/profox/000301cc7bdc$adfca8e0$09f5faa0$@co.uk
** All postings, unless explicitly stated otherwise, are the opinions of the author, and do not constitute legal or medical advice. This statement is added to the messages for those lawyers who are too stupid to see the obvious.