Hi all.

Up to now, we've managed our huge numbers of files and projects via custom scripts I had written, which created a mega Excel spreadsheet we could use. All this is on WinXP.

For any given shoot, we have a project folder that contains all the raw images, child TIFFs, JPGs, and more specific to that client. Each project is named by date (e.g. "YYYY-MM-DD, text description"). We never throw any file away.

We have long since outgrown that, but haven't had time to find/apply a working solution.

We have some 30 or more external hard disks spread across 6 different networked computers all on WinXP. Outside of what's currently active, most disks are turned off, but turned on when we need to access a given project. We also have many terabytes of projects that have been offloaded to tape (two different tape systems).

I need to get a very solid and robust cataloging application to survey this mess. Some of the individual projects can contain as many as 11,000 raw images (think covering a National AKC event, or a two week shoot through the Inside Passage to Alaska). There are easily close to 600 projects that are still on disks.

Did I mention we need some serious help here?

I am only looking for cataloging help (which would include keywording), but nothing for printing or image processing.

All projects have been duplicated for backup, but things are out of control and there may be 4 or 5 unwanted copies now for some projects.

Lightroom, iView (now MS Expression) - can anyone suggest the best fit application? I'm really seeking real-world experience, not "theoretical" knowledge, as I don't have time to experiment... (I vaguely recall hearing that LR had an upper limit to the number of files it could handle as a DB).

Again, this will involve at least six different computers, some with as many as 12 or more attached external drives. I can always get these drives shared to a master computer, but that will likely create issues as well (e.g. there are only many drive letter assignments I can use, so drive letters will have to be recycled for multiple shares).

Many thanks as always!

Lightroom does in fact have an upper limit as to how many images you can put in a library, but you can have more than one library.

However, frankly, if I were you, I would considering hiring a database programmer to create a custom database application for you based on a good platform with some history and some staying power. (For instance, a PHP/MySQL solution.) You are dealing with so many images that only a true database is going to be able to cope with this. There may be pro-level DAM programs that can handle this - I suspect there are. But you need serious horsepower, and you need a DAM strategy, because it sounds like your system, like Topsy, "just growed." I can tell you from bitter experience that systems like this tend to hit a catastrophic failure point with little or no warning. (E.G., one of the external hard drives experiences a head crash, and somehow, that was the ONE drive that didn't have full redundancy, etc.)

Also, you should be prepared to spend some money for a real server solution. Personal networking has come a long, long way since PhoneNET, but there comes a point where you need some iron that can manage the load intelligently. If you're not there now you will be soon. It doesn't have to cost a fortune - even Apple servers have gotten pretty reasonable. But you have to plan for the future. Even if your systems now can handle the load, I can't imagine they're exactly efficient at it, and with the sizes of sensors/scans increasing constantly and the number of image files growing steadily, you're going to hit a wall and all of a sudden it'll take hours to find one file.

M

Hey, StMarc -

Great reply, and great insights. I've come out of the software field (left that software developer engineer stuff nigh on 4 years ago), and certainly can see the "windshield" coming at me at 70 mph.

We're now at that point were it's hard sometimes, but not impossible, to find projects that are over 6 months old. And being generalists, we cover a lot of turf pretty quickly.

The notion of multiple LR libraries already is a big disqualifier - soon, you'd be having to guess which library to access, and we would have a lot I suspect.

Do you have any experience w/Extensis? Someone else had suggested that (last time I looked at it was back in version 2.x or something, it's now up to 8.5).

I'm also guessing that having zillions of big disks scattered across multiple machines (none of which is running as a true server) is likely to be messy. I'm not yet mentally ready to introduce a server into the system, and throttled by how much budget I can throw at this. On the other hand... there's that damned windshield.

And yes - our machines are tooled up for fast image production, but not server management - yet.

Thanks...

I have not used Extensis other than a brief trial review several years ago, sorry.

I hear you on budgetary concerns, but I think that you are just going to have to step up on this one... your requirements are insane, and a kludge system (no offense, I am the King of Kludge, I ran a network very similar to yours with an iMac for a server for years) is just not going to hack it much longer.

If nothing else, if I were you, I would budget either for a commercially-supported server version of Linux and the iron to run it, or a base-model XServe or MacPro with OSX Server, and try to get your network under control. (If you have a savvy person it's economically efficient to task on this, you can actually use an individual OSX system and some readily-available freeware to get essentially the same functionality.) Then you'd have a platform for later database services. It's just not a good idea - as you probably know - to have a workstation doing server duty, it's just asking for a good smiting from the IT Gods.

M

I am going thru a similar process and I have found that locating images thru keywords is much better that the title of each image. That being said you can't go thru and keyword all the legacy images so you are stuck with titles.


here is my post from s few weeks ago.
http://www.pdngallery.com/cgi-bin/ubb/ultimatebb.cgi?ub...t_topic;f=1;t=007097

So what I plan to do is select all the most important ones and re catalog them.

I still haven't found a good catalog software that doesn't want to create a huge cache of them all.

My first try with Iphoto worked pretty good and it did not include any dupes which was great and I was able to build a library of the images without moving them from their location.

There are "catalog" solutions and "browser" solutions and "what ever the hell bridge, light-room and aperture" are supposed to be.

Sooo I got Photo Mechanic and it's pretty fast but it doesn't build a library but you can go to the location and browse them pretty fast and keyword the relevant ones.

You can also keyword just one image from each job and then search the group of images within that key-worded image.

I don't know what your budjet is but putting them all in one place might be a good start.

Jerry,

I use Portfolio 8. Portfolio 8.5 server might be worth looking into. It would require a robust, dedicated server to run it, as portfolio can really bog down when the catalogs get huge. Might be worth doing some research in that direction.

Scotti

Hi Scotti -

I spoke w/the Portfolio folks at length this afternoon, and have at least an intermediate answer. Server would be huge overkill, even though my volume of images is large. Better to create multiple catalogs (the "slow to a crawl" factor is about a 4 gig catalog).

There are only my wife and myself, so support for a shared/server/multi-user environment isn't warranted. Additionally, I do all the IT work on our system, so the only other thing needed is a good search tool.

And... Portfolio Server's price tag: $6000. Not ready for that yet - I'd rather put that into a 2nd D3 or D3x when it comes out.

All our project top-level folders contain the shoot date along with a short text description. That's enough for us to find what we want pretty quickly, since most times we're looking for the project's location (usually know what images we're seeking).

We've been using Photomechanic and ThumbsPlus for many years, as browsers (even though TP sorta acts like a flimsy database).

The real advantage of Portfolio is that it can handle UNC paths (i.e. don't need a drive-letter assignment for a shared drive), and of course off-line thumbnails. Additionally it sounds like it can keep backup up project copies synch'd, something I already spend far too much time manually doing. (Please don't get me started on RAID).

StMarc (or is "Marc" acceptable): You're right - a more robust solution is needed. But this, in combination with a massive offloading to our 3TB tape system should help reduce some of the insane overhead we're dealing with.

BTW - Piktour - for PhotoMechanic - be sure to keep an eye on the cache it creates. Even with preferences set to clear the cache, it doesn't always do this, which can result in a r e a l l y s l o w session the next time you fire it up to look at a different project...

Again, thanks to all the great insights!!!
Jerry

Jerry,

I hit the "slow to a crawl" wall with Portfolio several years ago. Now, every client that I offer archiving services to gets their own catalog, and my own work is broken down into various categories. Portfolio works like a charm if you keep the catalog sizes down.

Scotti

Scotti,

Something you said jived with a fundamental question I've been trying to sort out.

As I mentioned earlier, we have many hundreds of projects, plus their backup copies.

The single biggest objective I have for the moment is simply to list where the projects all live, along with their backups.

In looking more at Portfolio, I don't know how to create a list of the projects (and where they live). It sounds like when I import projects, I will get all the contents of the project, but no hierarchical way to sort them out.

Unless - each project (master or backup) is its own catalog... And if so, will Portfolio help me keep masters and backup copies sync'd?

That sounds like your approach, but wanted to double check before I launch into this...

Thanks!

Cumulus from Canto is another good photo management program.

Here is the link: http://www.canto.com/

quote:

We're now at that point were it's hard sometimes, but not impossible, to find projects that are over 6 months old. And being generalists, we cover a lot of turf pretty quickly.

The notion of multiple LR libraries already is a big disqualifier - soon, you'd be having to guess which library to access, and we would have a lot I suspect.

quote:

I'm also guessing that having zillions of big disks scattered across multiple machines (none of which is running as a true server) is likely to be messy. I'm not yet mentally ready to introduce a server into the system, and throttled by how much budget I can throw at this. On the other hand... there's that damned windshield.

It sounds like your main problem, ironically, is a hardware problem. All those drives scattered across 6 machines...sounds like your first step should be to consolidate them onto one machine.

It doesn't have to be a server per se (as in Windows XP Server on server-class hardware), you should be able to use a good quality desktop running XP that everything attaches to and is managed from.

First of all, excluding backups (which your tape drive is fine for), how many terrabytes do you have right now? Knowing that would help. For example, let's say you have up to 15TB's spread across all those drives.

In which case you could use something like 3 StarTech InfoSafe's. Put 5 new 1TB drives into each one and you'll have 24/7 access to about 15TB from one machine and the LAN. Yes you'll need a host machine with 3 spare PCIx slots in it but you could build one up for less than a grand. It doesn't need to be super fast, just reliable.

The InfoSafe has what appears to be (if you read the manual) a very comprehensive management utility that will even let you configure all 5 drives as one volume...which I would avoid. Go with separate drive letters (JBOD) for each volume, 15 in your case, E through S.

Then put the host machine on your LAN and share each drive so the workstations can see them all. It's not a server but it's still a lot of networked storage that's relatively inexpensive to implement. And it should work fine with your Excel script once you've moved everything over, just have it re-catalog the drives until you find an ideal software solution. I think you're going to have to either have something developed ($$$) or re-consider using multiple Lightroom libraries in conjunction with your Excel scripts.

For copying the data over I recommend Beyond Compare because you can tell it to do a CRC or even a full, bit-level error check so that no files get corrupted. It also allows you to do things like copy a folder from the "server" to a workstation, edit images, then only copy the edited versions back to the server instead of the whole folder.

If you need more storage that doesn't need to be online all the time, just buy some more more of their drive trays and (gasp) label them with a pen. IOW, label your initial drives E1, F1, G1, etc., and use that as the volume name in Windows so it comes up like this in My Computer: "E1(E"

Now, with every shoot in a separate folder as you have them, you can use Lightroom to catalog and keyword everything in there. The trick is to create one Library file (.lrcat) for each shoot and store it in the same folder with a descriptive file name (of course), which LR will happily do.

BTW, the reason I'm suggesting Lightroom is that it plays well with Photoshop and can often replace it for many things for a one-stop solution.

That way your Excel script can catalog it, and the cache will be kept with the images. The Excel file wouldn't need any image filenames, just the path info and .lrcat files, which should simplify things by an order of magnitude. As long as you manually edit the Excel file after every shoot to further describe what's in each folder, I can't imagine this being too difficult to keep track of.

Now the fun part - adding metadata to the images. By "fun" I meant "not fun" but fortunately, any reasonably competent photo student or "image aware" person with software skills could be employed to do this for you.

Assistant goes through the list, navigates to each drive/folder/shoot via Lightroom, imports the shoot (using presets you've created to keep everything in the shoot's folder), then goes through all the images and adds whatever metadata you want, like keywords describing the content, meaningful captions, etc. This will be a lengthy process so make sure you keep your new friend happy and well fed! Copyright info and creator fields and some other things that don't need to be unique to an image can be automatically added when the Library is first created via a preset.

Then, using your now very much smaller Excel file, search it to find the shoot, navigate there via Windows, and double click on the .lrcat file to open the shoot in Lightroom. Then you can find a particular image via it's Find dialog, which will look in the metadata to retrieve your image.

If you do decide to go with a different application or have a custom database developed later on, the metadata from LR will already be there and it's only $400 vs. $6000 for Portfolio Server.

Whew. This is basically a scaled up version of a system I'm starting to experiment with for myself, minus the Excel part (don't really need it).

I realize this isn't exactly what you're looking for but it's a hell of a lot cheaper than the alternatives you're looking at, and should be reliable and effective if you stay on top of it. IOW, you may have outgrown your current Excel script solution but with some re-tooling and the addition of LR, you can make your solution catch up to your needs.

The main thing though IMO is to fix your hardware problem first...30 drives on 6 machines...scary!

I would sign up for the Controlled Vocabulary Yahoo! Group (scroll down) and ask this same question over there.

I believe IMatch can do what you need, and it seems quite scalable.

You'd be able to import your excel data into IMatch pretty easily via script as well. They have a pretty helpful user forum.

The single biggest objective I have for the moment is simply to list where the projects all live, along with their backups.


You create a catalog, and then drag a folder onto it. You can drag a whole volume on but that's when things begin to bog down. When you drag a folder in, you will be presented with a dialog box, Portfolio Cataloging Options. You can set it up any way you like, from assigning properties, assigning keywords, create custom fields, move and copy files during cataloging, extract metadata, etc. If you forget to set something while importing a folder you can do it later on from within the catalog.

In looking more at Portfolio, I don't know how to create a list of the projects (and where they live). It sounds like when I import projects, I will get all the contents of the project, but no hierarchical way to sort them out.

What Portfolio does is import the complete hierarchical path of your folder or volume so it is imperative to have a good hierarchical system for your volumes and folders before beginning. It does not import all the contents of a folder, only the path and thumbnails.

Unless - each project (master or backup) is its own catalog... And if so, will Portfolio help me keep masters and backup copies sync'd?

I think FolderSync will do that for you. I don't use it so you would have to ask the people at Extensis.

Don't forget that each catalog can be broken down into many Galleries.

Good luck.

Scotti

In this time of shrinking prices, is it logical to deal with the costs involved in managing huge catalogs? Is it profitable? I'd be selective over what is worth keeping more than 6 months as a first step.

Hey everyone!

Awesome info yet again.

Some partial replies:

We do maintain a well-defined and consistent hierarchical structure for each project, so that part's a done deal. FolderSync sounds like the trick (and hopefully better than the comparators I currently use).

Kevin - I'd already reached the conclusion that a hardware reduction was needed. Life would be a boatload easier if I only had a small handful of disks to manage instead of lots. The issue isn't so much how many projects, but how many disks are in use...

I am also aware of Controlled Vocabulary somewhat, didn't know there was a yahoo group, but will look at that as well, since keywording is a hefty exercise.

Great thread here - very appreciated (and hoping others are benefiting from this as well).