DigMyPics.com
May 8 Through May 19. Click here to go back to current posts.
On Monday, we contacted the major couriers and asked them to return any shipments bound for us to their sender. If you sent us something scheduled to arrive this week, expect it to be returned to you.
All of the local Phoenix stations carried the story. Here is the story as reported by Fox News 10 and here is the story on ABC15.com
Thursday, May 8 2:04 am
We've received an overwhelming amount of sympathy and encouragement from
friends, customers and the community. Most of the messages we're receiving
are quite incredible. Some have moved us to tears.
We continue to be impressed by the professionalism and care shown by the Gilbert
Fire Department. Wednesday morning we were introduced to a team of people they have
assembled to sort through any recovered material and collect and box up any
photos and videos found. We were encouraged by what they were finding in
the morning, but what we saw then was the bulk of what was to be recovered.
By the afternoon they had completed the entire production area and had recovered
only around 10 boxes full of photos and a box of 8mm reels. We may have no
way to know who the photos or videos belong to. What we plan to do is
digitize and restore everything and anything that we have and any without an
owner will be made available to everyone that had materials in the building.
Those people can review the material and download what's theirs. It might take a
few weeks to complete this process. I'll continue to write here as we
progress.
We've had a few people write interested in recovering their money. We're planning on refunding everything to anyone that had anything in the building but it may take us some time to sort through who that is. Please don't file a dispute with your credit card yet. That will complicate our task and increase our costs as the credit card company charges us a substantial fee for every dispute they receive.
We are receiving emails at
sales@digmypics.com but we aren't able to respond to every message. We
will in the coming days try to work through them.
Thursday May 8 2:45pm
A lot has happened in the last 12 hours since my previous post so I wanted to
take a minute to write an update.
The Gilbert Fire Dept (GFD) and their support team CERT has spared nothing to recover
our customer's memories. We stored each order in a plastic, sealable tote.
The fire melted many of these totes into tiny pods, encasing their contents in
the plastic. The CERT team has been cracking open these pods and finding
some stuff in pristine condition.
While Annette and I were at the site this morning, the fire chief, Wes Kemp, told us his team was attempting a rescue of some bins that we had kept up front near our servers for local customers awaiting pickup. The roof in that area remained attached to the wall but a section of it fell down at an angle and remained perched on top of our server server racks and these bins. A large A/C unit was also on that section of the roof. Wes's team has been determined to save as much as possible for our customers and they had their sites set on recovering these bins. They brought in some equipment to delicately lift the roof while firefighters crawled under debris and pulled them out. Everything was a little wet but in very good condition. One of these customers was Melinda, the customer who appeared in the Fox10 story.
We need to thank the fine people at American Family Insurance. They've been facilitating the efforts of the fire department by allowing them to release customer material. They've assembled a team of people that have come together to help us figure out what's what.
I hope people don't mind, but we continue to be touched by the outpouring of support and help. I'm going to post some of those messages here. Maybe they'll help others the way they're helping us.
See some of the messages from customers
Thursday May 8, 5:51pm
We just met with Fire investigators who have finished their investigation and
are turning the site over to the insurance company. They isolated a 10x10
area where the fire started. They suspect electrical wiring. An electrical
engineer from the insurance company will be working on that tomorrow.
Here are some photos of the fire and devastation if you're up for it.
Friday May 9, 4:05 - 9:32
Annette and I are exhausted. Except for the death of my grandmother 14
years ago, this has been the most difficult and
emotional week I've ever experienced. We haven't stopped since 2am
Monday morning. We've barely slept or eaten. Speaking of
grandmothers, I lost the photos of both of my grandmothers. They had been
sitting in my office waiting for a lull in the business that never happened.
Fortunately, my maternal grandmother had her 86th birthday in March and Annette
had pushed to put a video together for her so her photos had been recently
digitized. My other grandmother's photos are gone. I was the custodian of
all of her stuff.
The incredibly kind people at HomeMovieDepot.com reached out to us on Tuesday to offer their help. Here's the letter. We've worked with the company in the past and we've always considered them to be the best in the video and film transfer business for consumer level video. We strived to model the 8mm video transfer portion of our business after theirs. They are currently working with the video film and seven video tapes GFD recovered from the fire.
The Restoration Progress
We fear we've lost a lot but we're also surprised by what's been saved. Our team
is working through the boxes of photos that GFD rescued. The work is
difficult but I think we all feel that it's also rewarding and will help us heal
a little. We decided to run two 4 hour shifts 7 days a week so we can
rotate people out while we get through the recovery and digitization process of
the photos.
We've got a lot of prints and slides, very few negatives. It seems like stuff that was in hard plastic, like a carousel, fared the worse. Ziplock bags, seemed to do well. Some stuff we find completely encased in plastic and we can chisel it out almost like finding a nugget of gold in the side of a mountain.
Saturday May 10, 10:05pm
There's not a lot of news to report. We had a larger team working today
going through the photos and getting them cleaned. Several customers
emailed us some digital photos so we would know who to look for in their photos.
The large files exceeded our limits so some email started bouncing back. I
increased the limits so it shouldn't be a problem again. If you sent an email
but got a 'mailbox size limit exceeded' message, please resend it. Our
apologies and gratitude to our employees and their mothers for working on
Mothers Day to help get this done as soon as possible. We've been
scrambling to get our makeshift office up and running.
Sunday May 11, 9:25am
Some customers have begun sending us information to help us identify their
materials. We appreciate that and we're going to post the information in
the office where we're working. I encourage you to send us such
information but, try to make it as concise as possible. Two or three photos that
contain most of the people in your family, for instance, rather than a picture
for each person in your family. The smaller the breadth of information we
have, the easier it will be for everyone to remember as their going through the
material.
Tomorrow we're going to make an effort to retrieve our servers from the fire department. They told us the hard drives of those servers look ok but they haven't given us more than that. Computers in the production area don't look like they'll yield much. You can see a scanning station in one of the photos I posted.
Sunday May 11, 5:53 pm
Julie, Priscilla, Emily, and Will stuck it out and worked with Annette and me
all weekend long. A special BIG THANK YOU to them. Nick, Liz,
Kristen, Colby, Kyle and Hardev also helped out in a big way either Saturday or
Sunday. Thanks for the help. It's much appreciated.
Monday May 12, 4:05pm
 Our recovery effort underway |
We
continue to see an outpouring of help from many unexpected places. Today,
members of the LDS church heard about our story and wanted to help. They
actually brought us out some tables we could us to work on. Thanks
Bishop Terry. The CEO from Metrobridge, our internet service provider, personally called from Vancouver and promised to get our service relocated to our new place by tomorrow. We had a lot of people able to work today so we'll start seeing real results soon. The process is very labor intensive. Many customers have sent us a couple of pictures to help us identify them. We've put them on a board and posted it on the wall. Its pretty emotional to see that. We continue to get emails of encouragement from customers. Thanks for that. I'm going to post some more now on that page. |
Tuesday May 13, 7:15pm
Metrobridge was unable to install their service at our new location. We're
trying to get a measly DSL line from Qwest. A T1 takes almost 2 months to have
installed. We were in such a hurry to rent a place, and we found a place
right in town. We never considered getting internet service would be a
challenge.
We've been finding the photos of people who have written. There is a kind of air
of excitement when we put a family to the photos we have. We're going to start
contacting people as soon as we can. Being without Internet or phones at the
facility means we have to do it from home.
HomeMovieDepot has been working on the film we sent them. So far they've
invoiced us for 25,000 feet of film so we're encouraged that they've saved a
lot. Possibly 25% of the film we had at the time of the fire. It may not sound
like a large percentage, but substantially better than 0. Most of the film
was unidentifiable so once they return the film and digital files to us,
everyone will have to watch what we have and hope we have something, anything,
that belongs to them.
We've received a lot of email and we're being asked to reply. We're
getting your emails but please don't expect much in the way of correspondence
until we can catch our collective breath and get on top of the more critical
issues. Thanks for being patient with us. We have been contacting
those for whom we have complete orders. Specifically, the local people who
had their completed orders recovered by the GFD. We're also planning on
contacting a few people who we have recognized.
|
Not much news today. Everyone is just chugging through the photos.
We're battling with some technical issues on how to scan some of the more
damaged and warped prints. We're inventing some new techniques as we go
along. Fortunately we have some very smart and innovative people working
to figure out some of the new challenges. (That's a plug for Nick, Colby
and Tobin) I had designed and built a new print scanner and we had two
prototypes at our facility. They're gone now, but my software code was
still on my laptop so Tobin and I rebuilt the
hardware yesterday and today using some of the major parts I still had at home
for a third machine I wanted to build. We're going to try to get it working
tomorrow. What's unique about it is that it captures an image from above the
print and uses vacuum pressure to keep the print flat. It's also much
faster than a flatbed scanner and the quality is just a good as our Epson
flatbeds and in some cases better, since some objects can't scan well on a
flatbed.
The Arizona Republic published a couple stories about the fire and our recovery
efforts: http://www.azcentral.com/arizonarepublic/local/articles/0514evfire0514.html |
May 15, 9:15pm
No significant news to report today.
May 17, 8:45am
We continue to work through the photos we have. We have way more than we ever
thought. (That's a good thing, of course, but it's going to take longer
than we first calculated.) GFD still hasn't returned our servers to us,
but they did tell us that they are in good condition and likely usable.
They still haven't publicly released a cause for the fire but they have led us
to believe it may involve a product liability suit against a manufacturer.
I've been dealing this week with one customer who decided to attack Annette and
me using the text I'm writing here as one of his weapons. It's made me a
little guarded in what I would say but this morning I decided not to let this
one jerk interfere with the good work we're doing for our other customers.
He's not the only one, either. Another customer decided to have her lawyer send
us a terse and threatening letter. It's unfortunate that such people exist
but, fortunately, they're a small minority. Everyone else has been
incredibly kind and helpful. The two attacks have put me on the defensive so I'm
going to make some comments about what our legal responsibility is and isn't and
what we are and are not doing. I apologize if the following language comes
across as harsh or cold.
Here are the terms and conditions by which everyone
agreed when they submitted an order to us. Those terms limit our
responsibility to a maximum of $100 or the cost to replace the physical media
(like a roll of film or a VHS tape) whichever is less. The terms are
standard in the industry since no price can be placed on a photograph that could
never be reproduced.
ABOUT REFUNDS. PLEASE NOTE: If we were unable to return a finished
product to you, we will also refund any payments you had made. We can't begin
to do that however, until GFD returns our database to us.
The day of the fire our insurance adjuster asked us what liability we had for
the customer's photos we had in the building. I described our terms to him
and he was quite relieved knowing we could just pay a maximum of $100 to
everyone, bulldoze the place and move on. Annette and I didn't quite see it in
such financial terms, however. We immediately knew we wanted to recover
whatever could be recovered and return any thing, even if its just one photo, to
their owners. The insurance company really has no interest in recovering
the photos. They did allow us to remove them quickly from the building and
they haven't interfered but they think we're nuts for bothering and I'm not sure
how much financial support we'll be able to get them to contribute toward our
effort. Legal obligations aside, Annette and I felt a moral obligation to
undertake this task, possibly at great personal expense. Many people have
written to tell us how much they appreciate what we're doing.
Secondly, Let me set the expectation with regard to our recovery and restoration effort. We have no plans to try and restore or preserve the physical photos and videos in our possession. Most are in poor condition. We're simply working to clean and dry the materials so that we can digitize them. Any restoration work we perform will be on the digital photos and videos. Depending on the condition of the materials, we may or may not return the originals to our customers. We may not even have the originals to return. Some photos or videos exist solely on the servers. We may not even be able to match the owners up. We're doing what we can. In cases where we can match photos or videos to their owner, we will want to return those items if possible. We've had some success with that already.
Hopefully I can put this nasty part behind us and focus on helping the kind people that have been so gracious.
Saturday May 17, 6:29pm
| I've held back on showing any pictures of what we've
been given to work because its so hard to look at but maybe its time.
I took these photos about a week ago. The two photos shown below are not from the same batch. We've since completed our recovery work on the prints we had. They're now getting scanned. |
|
.JPG) |
|
| This is about as good as it gets. These chunks of charcoal have actually been salvageable though. We've been able to soak these bricks until they soften then we can peel each photo apart one by one. We let each photo dry, then trim each and every photo to remove the char. | |
.JPG) |
|
| After breaking the brick apart, each print is allowed to dry. Once these prints have dried, they will each be trimmed and scanned. | |
Sunday May 18, 9:00pm
Tomorrow morning marks 2 weeks since the fire. We're working hard and everyone is tired (and sometimes a little cranky. huh Priscilla?) but we are starting to feel like we're getting somewhere. We have matched a lot of photos to their owners. Annette's been trying to contact people to let them know and to ask questions if we're unsure. We have a system in place to help us track what came from where so when we digitize, we can go back and find a particular photo if we need to. It's exciting and relieving when we discover a group of photos and can identify them.
Our servers and our back up systems
Many people have asked us about or even shaken their fingers at our backup strategy wondering how we couldn't have offsite backups of all the photos. We had several backup strategies deployed to protect against a server failure or hard drive failures. We've suffered many hard drive failures in the past and those strategies served us well. But the truth is, I never envisioned such a devastating failure of every machine at once in such complete destruction. It just seemed like such a remote possibility.
Usually by the time a customer's order left our building, their photos would exist on 3 different computers for at least 14 days. (we always claimed 14 days, but we really tried to keep everything for a month). This gave us comfort that should anything happen during their return, we could always reproduce the disks and send out new sets.
No offsite backup of all those photos?
In our work, we created and edited several terabytes of data every single day using more than 100 different computers. Its an extremely large amount of data by any standard and moving that much data from that many computers over the internet to another location would have been extremely costly and impractical. The cost of the bandwidth we would need to move that much data during the 6.5 hours nobody was working (12:30 am to 7am) would have been in the tens of thousands of dollars a month and possibly more than our entire revenue. We did offer an offsite storage service we called Expo for a small additional charge, but we didn't move the photos from our facility to the off site location until a few days after we shipped the photos back. We did things this way because many people would ask us to rearrange and rename folders and it was better to do that before we copied the files to the offsite servers. Customers that had an Expo account still have their photos stored there. We had a very fast Internet connection of 10Mbs (Mega bits per second) dedicated up and down but it wasn't nearly fast enough to copy the large volume of data we produced, even if we maximized it's use 24/7. A T1, which is what most businesses our size would have, is 1.5Mbs up and down. Your cable connection is probably rated at 10Mbs down and 1Mbs up but its shared between you and your neighbors so you probably get much less than that.
We've been getting questions about whether we'll have the photos or videos on our servers that are to be returned to us by the GFD. The answer is, it depends. Prior to finalizing an order, we would process all of the photos and videos for posting on our website. During that processing, we would create copies of the photos in various resolutions ranging from roughly 100x100 pixels to 1200x1200 pixels. We stored those copies on the servers that are now in GFD's possession. This allowed our customers to quickly view their photos on our website.
What about our database? No backup of that?
We actually had several backups of our database, one was kept in real-time and another was made nightly. We had also kept our database offsite, but two weeks prior to the fire we had let an employee go. Before that employee left, he decided to damage his computer to make it inoperable. He was a technical guy and had access to our offsite servers and databases. We already had a plan in place to move all of our data processing in house but to avoid any further damage or possible theft of data by this employee I decided to accelerate the move. We were in the process of redesigning our backup database strategies when the fire happened. The timing was just bad. Fortunately, we've been told that those servers are fine.
I've worked at companies large and small. For a company our size, we had very good technology and strong data policies in place but nothing could have prepared us for what was to come. I suppose its easy now though to look back and say we should have done more.
Monday May 19, 6:00pm
Some of what we're finding has a note or a folder title or some other piece of
information to help us identify and match it to someone. Annette has been
making educated guesses on who she thinks it might belong to and then asking
those people for confirmation. Some items she's unsure of so she asked me to
post some of the tidbits of information here so that anyone looking can write in
and say its theirs. Click here to see the list.
Tomorrow afternoon we're meeting with the fire inspectors from GFD and American Family Insurance. They've invited the manufacturer of the suspected equipment to review their evidence and respond. It should be an interesting meeting. The site of the fire has been off limits and under 24 hour guard until this company has had time to review the site and the evidence. No one has been able to visit the site or touch anything without escort.
Server Status Update
We're still waiting....The servers are being used as evidence. We had 11
cameras in and around the building monitoring and recording onto two DVR's.
One of the DVRs recording 8 cameras survived, the other was found completely
melted. We also had 4 Internet cameras monitoring and alerting motion.
That data has been captured and studied by the forensics department of GFD,
mostly to show that no one had been in or around the building prior to the fire.
The servers also tracked entry to and exit from our building using a biometric
entry system that allowed entry, tracked entry in our database, and armed and
disarmed the alarm. We also had an extensive internet traffic router
logging system that we used to watch for and alert about hackers.
All that data was also stored and is being used to support the investigation. That's why its taking so long to get it all back but GFD expects to be able to release them this week. We're hoping. It will make our job much easier. Not only will we have a detailed inventory of who's orders we had, but we captured and stored one or more images in our database of the contents we received for each order. Those images will help us identify and match more people with their property.