Bumping into OS X's file limits
July 10, 2006 at 10:27 PM by Dr. Drang
I’ve been working on a database problem for the past few days, and because of the size of the problem I’ve learned a bit about the limits of the OS X file system.
The database organizes several thousand scanned pages, JPEGs and TIFFs. The database ties the pages together into a few thousand documents of various lengths and includes the OCR’d text of the scanned pages. I’m using FileMaker because I need to share the database with others that are not always going to be hooked up to a network. FileMaker is cross-platform and combines both a client and server.
My first thought was to import the scans directly into the database. This made the file about 13 gigabytes and I found I couldn’t copy it from one disk to another. Not in the Finder, not with the Unix cp command, not with rsync. In each case, I got an error message that said, in effect, “This file is too big to copy.”
I compressed the file down to just under 7 gigabytes with gzip and tried to copy it again—same error message.
Lesson 1: OS X cannot copy files of several gigabytes from disk to disk.
Back to the database. I cleared out the scanned images and reimported them as references. This kept the database down to a reasonable size of 30 megabytes or so. Which worked fine. Before doing this, though, I tried to copy all the JPEGs and TIFFs into a simpler directory structure than the one in which I received them. And this is where I ran into the second limitation: when you try to copy thousands of files with a single drag of the Finder or single cp command you get an error message that tells you the file list is too long (I didn’t try rsync; it wasn’t appropriate because I wasn’t trying to duplicate the existing directory structure).
Lesson 2: OS X cannot copy several thousand files in one go.
I worked around this problem by issuing several cps, each working on a subset of the scanned files. These were put into several different folders. None of the folders has more than about a thousand files in it, so I won’t run into the limitation if I need to move them again.
What are the exact limits? Dunno. But now I know enough to keep away from them.
