In a comment on my Shaggy Dog story yesterday commenter Iztok gently chastised me for not having backups of my VM’s. Ironically I did. The problem is that my backup strategy is so silently effective that I completely forgot I had it ! Having been reminded, I thought it might be worth sharing.
The Good Old Days
Before I switched to Mac I went through a number of backup solutions for my various PC’s over the years. Usually some combination of optical archiving for long term backups and some form of replaceable media solution and backup software for day-to-day peace of mind.
None was 100% reliable and all were intrusive to some degree or another. Iomega featured a lot in this history… first Zip drives (cumbersome – all those discs), then Jaz (unreliable – the one time I needed it it failed to restore and ultimately died completely) and finally an Iomega HDD puck/caddy system the name of which escapes me (this worked well but was expensive and unfortunately didn’t last long).
Then I got my Mac and, based on my experiences of similarly supposedly effortless backup systems for PC’s, I was initially skeptical of Time Machine. Not any more.
Time Machine is completely reliable and entirely unobtrusive. Of all things Apple it perhaps best embodies the “It Just Works” aspect of the Apple approach.
Time Machine backs up everything – system HDD and external HDD’s. You have to explicitly exclude drives and/or folders to have them not included. It automatically and unobtrusively keeps hourly backups for the past 24 hours, daily backups for a month and weekly backups thereafter, deleting the oldest backups only to make room for new ones.
The only indication you generally have that Time Machine is doing it’s thing is that the icon in the Mac menu bar quietly spins while it is taking the hourly backups. VM snapshots are a bit more noticeable, but I’ll get to that in a minute.
On my Mac at work with a measly 1TB drive dedicated to Time Machine duty I have backups going back to January 2012 without ever having lifted a finger !
As I mentioned in the Shaggy Dog story, VM images can be significantly large files (actually with Parallels a VM is a “bundle” – which ultimately is a folder rather than a file, but the point remains). If Time Machine were to include changed virtual machine files on an hourly basis it would quickly fill even a 1TB volume.
Fortunately the VM solution I use – Parallels – has specific support for Time Machine.
Backup options can be set in different ways to work with Time Machine on a per VM basis. You can exclude a VM entirely if you manage a backup separately (or just don’t or need it backed up), or have it simply included as a regular file, or you can have Parallels take snapshots that will allow Time Machine to backup only those snapshots as needed.
When creating Snapshots (SmartGuard in Parallels parlance) you can define your own schedule or have it synchronized with Time Machine itself.
It should be mentioned that when a snapshot is taken the VM must be suspended, which occurs automatically. You can opt to be notified first in which case you can cancel the snapshot if you are in the middle of something that you’d rather not be interrupted. But snapshots take only a few seconds so I generally leave everything set to automatic.
That all takes care of my carefully configured VM environments.
The problem this week I think was that I was so engrossed in my problem – and Time Machine so unobtrusive – that when I needed to go back to a VM I had deleted I completely forgot that Time Machine had me covered!
Had I actually “lost” something entirely I would likely have remembered but it was just a question of inconvenience rather than catastrophe. And of course, as I mentioned in my reply to Iztok‘s comment, on this occasion my somewhat embarrassing Senior Moment led directly to the discovery of a bug and a resolution that might not otherwise have been uncovered. 🙂
The data that my VM’s exist to work with is all on an external HDD.
As mentioned, this is also covered by TimeMachine. The really important stuff is kept in various cloud-backed storage locations. Of course this is primarily for sharing between machines, not part of the backup strategy as such, but it does mean that if my system ever died completely I could quickly get to the current data from my other devices without having to rebuild/repair/restore anything.
But Backup is Only Half the Story !
Indeed, and the other half is restoring from those backups. And here again Time Machine is leaps ahead of any backup/restore solution I ever came across before.
To restore something from Time Machine you first simply “Enter Time Machine” itself. In this mode, Time Machine takes over your entire display, presenting a rather whimsical star field as a backdrop (I’d like to be able to pick a Marty McFly and/or Doc Brown moment myself, but I can’t have everything :)) to a simple file browser window and a timeline.
Obviously, if the particular folder you are viewing in the file browser is excluded from Time Machine then you lose the ability to navigate through time and can only see “Now”.
But otherwise you can navigate the file system in the file browser window and see exactly what your system looked like at the point of time currently selected in Time Machine. The initial time is “Now”. You can then step back or forward through time one snapshot at a time or you can use the timeline to jump straight to a specific moment in time.
However you choose to navigate through time, the file browser updates to reflect the chosen moment. Not the differences, just exactly how it looked, what files, folders etc were there, at that time.
You might be able to discern my interests of a year ago from the different contents in my Downloads folder at that time.
Once you have found the file(s) you want or need, click “Restore” and you will be asked where you want to put the restore file – either in it’s original location or somewhere else.
Now all I have to do is make sure to remember that I have this covering my a** in future! 🙂
There’s also a free app you can run that will run TimeMachine less frequently. I really get distracted by having it run every hour, and I don’t need that level of detail. It’s called “Time Machine Editor”. Ironically, it works when your TM is set to “Off”.
With Parallels I suppose you can just turn off the “Optimize for Time Machine” and set your own schedule.
But I think I misrepresented what the optimised setting does though. Looking at the way that turning this on affects the number and size of snapshots and the effect that different manual schedules have, I think the “Optimised” setting is a snapshot every 48 hours and only 3 snapshots retained (1 daily, 1 weekly, 1 monthly).
It says something about how unobtrusive I find the taking of snapshots that I had to investigate it. 🙂
I agree that it works really well but you haven’t yet encountered the dreaded “ime Machine completed a verification of your backups. To improve reliability, Time Machine must create a new backup for you.” message. This seems to occur more often on NAS than on a USB disk but I have had it occur on a normal USB disk as well. When it occurs the entire backup is rendered useless. Much as I have investigated the causes of this, I cannot see why it should occur if the system were properly designed. Surely you would ensure that the previous files were protected from whatever takes place afterwards? Surely no matter what error you encounter cannot render the entire backup useless? Yet if you read through the whole mass of articles on this subject you find that Apple has not determined the root cause of this problem and you cannot rely on TimeMachine to the extent that they would have you. I am glad that it is working for you reliably but from my experience I would be loath to ever trust really important things to it alone.
Every backup system has it’s problems and idiosyncrasies. In the context of everything I have ever used previously, all I can say is that so far (touch wood) Time Machine works the best for me in my circumstances.
I am not trying to say that it is a perfect or the most complete system.
Bear in mind that Time Machine effectively looks after those parts of my system that I could recover by other means – re-installing software etc. The truly valuable stuff, which can’t just be re-installed, is backed (not backed UP) by a cloud copy. Only of the latest version, of course.
Time Machine provides a convenient way to restore such data from any point in time (covered by the backups) and the cloud copy (of that data) is my back-stop should I lose my system and find myself with an unusable Time Machine for whatever reason.
Nice article. I don’t know what format the Time Machine uses, but I do not like applications that make a format only they can read. I only use solutions that use format I can then read with basic file explorer.
My current solution is simply a NAS with a RAID 5. All data is basically there I have very little stored locally. Thunderbird, Pidgin, Picasa, all personal data is on NAS. Then I have Genie Timeline which works sort of like time machine and makes backups in time on a backup system (for now that is 2TB external drive, but I just ordered a HP microserver with 4 HD SATA slots, so I will make a mirror of 2 3 TB driver 🙂 )
VMs are backed up with a script on a weekly basis (no need for more cause all data is on NAS as stated before)
Anyway, so far that works very well for me. I intend to blog about the whole dev enviroment soon.
Thanks. And thanks for providing the inspiration. 🙂
I’m not sure that Time Machine is as much a “closed box” as you might think. If I browse the Time Machine volume I can drill down into recognisable copies of my files.
e.g. /Volumes/DeLorean/Backups.backupdb/Deltics iMac/2013-08-28-161114/Data/band practice/5th Quarter/Song List.xls
Is the spreadsheet containing the list of songs that a covers band I used to be in used to perform. Obviously the 2013-08-28-161114 folder is the date/time of the “snapshot” of which that file forms a part. The immediate folder beneath that is the name of the volume that the file was original located on and everything after that just the normal path to it’s original location on that volume.
So even without the Time Machine UI I can just browse to those files and open them.
Time Machine presumably adds metadata somewhere to facilitate it’s own ability to present the snapshot contents in a more user friendly fashion (this metadata – if it exists – is hidden from Finder), but the backed-up files themselves are all, well, just there. 🙂
As I said I have no idea, how Time Machine stores the data. But from your description it seems that it is freely available and that is great. I think this is a must for a good backup solution. You have to be able to get to the data without additional software.
Anyway good debate here, it might actually help somebody think about their backup strategy. I have seen so many sad scenarios over the years due to people relying on their HD drives, as if they are immortal.
Indeed, it has made me think a bit more about how Time Machine works and whether I am as covered as I perhaps think.
A very useful topic to discuss, if only to get the old cogs turning. Thanks again. 🙂
You can go into the TimeMachine volume and access files directly through the Finder any time you want. It’s completely and totally accessible for read-only needs. You just cannot write anything to it manually.