18 Backing up your computer
As most of us use macOS laptop computers, this chapter will mostly be focused on that operating system. Though contributions for other operating systems are more than welcome.
For backing up your list of R packages you have installed, check the R setup: R packages subsection from a previous chapter.
The material described here was presented on the team meeting on 2024-02-28 as shown in the video available below.
18.1 macOS
As some of you know, I have a personal Dropbox account that I pay for myself. You don’t have to use Dropbox as we have access to OneDrive through LIBD (1 TB per account). The concepts below apply for any cloud back up service.
18.1.1 Time Machine
First of all, I highly encourage you to purchase a USB-C external hard drive with plenty of space (like around 2x your computer’s disk space) and use it for Time Machine backups. It overall works quite well, though as you increase the files in your computer it’ll take longer to run (can be several hours) 26.
A limitation of using Time Machine is how frequently you choose to update it. You could update it as frequently as every hour, which then makes each update fairly small and thus fairly fast. If you do it daily, it might take up to an hour to run. If you do it less frequently, then it’ll take up to a whole work day (8 hours) to run. You have to keep this in mind when using Time Machine, as it can be natural to avoid updating it frequently given how long it can take to run. But then if you have an update every month or every couple of months, that means that if you have some kind of failure on your computer, then you are very much not protected and will lose lots of valuable files.
Time Machine is great when you know that you are about to update your machine and want to be able to transfer files to the new one. It can also be quite cost effective as nowadays external storage devices with space for 2 to 5 TB are quite affordable (less than $100 USD), specially when you compare them against the cost of a yearly cloud backup service like Dropbox across multiple years. Of course, external storage devices have their own limitations, such as disk failures as well as the fact that there is only one copy of your device (unlike in the cloud) so if you lose it everything is lost.
18.1.2 Cloud
Given the limitations with external backup devices, I prefer to use a cloud storage service in addition to the external backup device. Dropbox has a nice website explaining why you’d want to backup your macOS on the cloud. Basically, it can backup your:
~/Desktop~/Documents~/Downloads~/Music
by copying them to your ~/Dropbox directory and then making links (aliases; soft links) on your home directory (~/) such that other macOS programs won’t even notice that the files are actually stored under your ~/Dropbox directory. Something similar is available from most cloud storage services.
That all sounds great, but then you are going to miss other files and information you are likely interested in. Broadly speaking this would be:
- Your installed software (
/Applications) - Preferences / configuration files for installed software (like your
RStudiopreferences) - Linux-style software configuration files (
~/.ssh,~/.Rprofile, etc) - Any Linux-style software that you installed (think about
Homebrew)
All of the above is actually backed by Time Machine. So that’s a big plus for that option. For a cloud-based option to be fully operational, you will have to do some extra work.
18.1.3 Complementing your cloud backup
On my ~/Dropbox directory I have a subdirectory called Mac-files (aka ~/Dropbox/Mac-files). There I have copied a few things that make it easier to restore a computer using a cloud storage service.
18.1.3.1 Applications
For my installed software (/Applications), I have taken screenshots of the contents of that directory and saved those screenshots on a Slack direct message to myself. Another option is to list contents of your Applications directory. Here’s my current list.
% ls -lh /Applications
total 0
drwxr-xr-x 3 leocollado staff 96 Feb 13 16:24 1Password.app
drwxrwxr-x 6 root wheel 192 Feb 27 18:59 'Adobe Acrobat DC'
drwxrwxr-x 5 root admin 160 Feb 27 18:54 'Adobe Creative Cloud'
drwxr-xr-x 11 root wheel 352 Feb 27 18:57 'Adobe Illustrator 2024'
drwxrwxr-x 5 root admin 160 Feb 27 19:02 'Adobe Media Encoder 2024'
drwxr-xr-x 9 root wheel 288 Feb 27 19:04 'Adobe Photoshop 2024'
drwxrwxr-x 5 root admin 160 Feb 27 19:00 'Adobe Premiere Pro 2024'
drwxrwxr-x 5 root admin 160 Feb 27 19:02 'Adobe Premiere Rush 2.0'
drwxr-xr-x 3 leocollado admin 96 Oct 25 05:13 'Alfred 5.app'
drwxr-xr-x 3 leocollado admin 96 Dec 20 03:53 'Bartender 5.app'
drwxr-xr-x 3 leocollado staff 96 Jan 11 04:45 Cyberduck.app
drwxr-xr-x 3 leocollado admin 96 Feb 27 16:42 Dropbox.app
drwxr-xr-x 3 leocollado admin 96 Feb 13 19:22 Firefox.app
drwxr-xr-x 3 leocollado staff 96 Feb 27 17:45 Flux.app
drwxr-xr-x 3 leocollado admin 96 Feb 26 18:13 'Google Chrome.app'
drwxr-xr-x 3 root wheel 96 Feb 27 19:46 'Google Docs.app'
drwxrwxr-x 3 root wheel 96 Feb 27 19:46 'Google Drive.app'
drwxr-xr-x 3 root wheel 96 Feb 27 19:46 'Google Sheets.app'
drwxr-xr-x 3 root wheel 96 Feb 27 19:46 'Google Slides.app'
drwxr-xr-x 3 root wheel 96 Feb 27 19:38 'Ivanti Secure Access.app'
drwxr-xr-x 3 root admin 96 Dec 26 13:42 LogiTune.app
drwxr-xr-x 3 leocollado admin 96 Oct 9 15:26 'Loupe Browser 7.app'
drwxr-xr-x 3 root wheel 96 Feb 5 14:07 'Microsoft Defender.app'
drwxrwxr-x 3 root wheel 96 Feb 27 14:00 'Microsoft Excel.app'
drwxrwxr-x 3 root wheel 96 Feb 27 14:00 'Microsoft OneNote.app'
drwxrwxr-x 3 root wheel 96 Feb 27 14:03 'Microsoft Outlook.app'
drwxrwxr-x 3 root wheel 96 Feb 27 14:00 'Microsoft PowerPoint.app'
drwxr-xr-x 3 root wheel 96 Feb 27 14:00 'Microsoft Teams classic.app'
drwxrwxr-x 3 root wheel 96 Feb 27 14:00 'Microsoft Word.app'
drwxr-xr-x 3 root wheel 96 Feb 27 14:00 OneDrive.app
drwxr-xr-x 3 root wheel 96 Feb 27 17:12 R.app
drwxr-xr-x 3 leocollado admin 96 Jan 29 16:57 RStudio.app
drwxr-xr-x 3 leocollado staff 96 May 25 2020 RSwitch.app
lrwxr-xr-x 1 root wheel 54 Feb 2 12:19 Safari.app -> ../System/Cryptexes/App/System/Applications/Safari.app
drwxr-xr-x 3 leocollado admin 96 Jan 24 17:20 Slack.app
drwxr-xr-x 3 leocollado staff 96 Feb 15 2021 TextMate.app
drwxr-xr-x 8 root wheel 256 Feb 27 18:55 Utilities
drwxr-xr-x 3 root wheel 96 Feb 27 14:03 'VIPRE Endpoint Security.app'
drwxr-xr-x 3 leocollado admin 96 Nov 1 09:48 VLC.app
drwxr-xr-x 3 root admin 96 Feb 27 14:07 'WatchGuard Mobile VPN.app'
drwxr-xr-x 3 leocollado admin 96 Feb 24 14:01 Zotero.app
drwxr-xr-x 3 root wheel 96 Dec 31 1969 connectwisecontrol-75da79a526aab2dd.app
drwxr-xr-x 3 leocollado staff 96 Jan 3 17:31 'iStat Menus.app'
drwxr-xr-x 3 leocollado staff 96 Dec 18 17:23 iTerm.app
drwxr-xr-x 4 root wheel 128 Feb 27 19:43 quarto
drwxr-xr-x 3 root wheel 96 Feb 27 15:12 zoom.us.appHaving this information (as a text file or as screenshots) is like having a recipe that you can then remember what are all the things you need to install. However, not all software can be re-installed so easily.
I have a few applications that I manually backed up by copying them from /Applications to ~/Dropbox/Mac-files. This won’t always work for all software, but it can in some cases.
18.1.3.2 Homebrew files
At ~/Dropbox/Mac-files I created a subdirectory called Root (aka ~/Dropbox/Mac-files/Root). There I have four shell scripts, the first one being update_brewfile.sh as shown below.
This is only applicable if you use Homebrew to install software on your computer. When you run it, it will create a file called Brewfile that contains the information you would need to have to re-install software managed by Homebrew. In my case, my Brewfile contains this:
tap "homebrew/bundle"
tap "homebrew/cask"
tap "homebrew/core"
brew "coreutils"
brew "git"
brew "git-lfs"
brew "go"
brew "hugo"
brew "libidn2"
brew "pandoc"
brew "wget"To restore your Homebrew-managed software, I have second shell script called restore_homebrew.sh.
#!/bin/bash
## Install homebrew from
## https://brew.sh/
## Restore homebrew config
## details: https://tomlankhorst.nl/brew-bundle-restore-backup/
brew bundleIt works pretty well, though note that I try not to install software with Homebrew in general, particularly when it comes to R and being able to compile packages. Check the R Setup section for more details on how to install all the binaries you need for compiling R packages.
18.1.3.3 Linux-style config files
Tough day, but you know what they say: “Don't cry because it's over. Cry because your laptop is about to be factory reset, and despite saving all the dotfiles and configs, you just know that it's gonna take you forever to get your dev environment right.” 😭
— Mara Averick (@dataandme) June 30, 2023
At ~/Dropbox/Mac-files/Root I have a third shell script, this one is called update_files.sh. In it, I basically keep track of what Linux-style configuration files I want to back up. I also use it to backup my RStudio preferences, TextMate bundles, and some other things (Music library, some Cyberduck files).
#!/bin/bash
# sh update_files.sh
## Created this file on 2020-01-14 in response to
## https://help.dropbox.com/installs-integrations/sync-uploads/symlinks
## and
## ~/Dropbox/Dropbox\ reports/Symlink\ report\ -\ Items\ that\ will\ stop\ syncing.csv
## More options:
## https://news.ycombinator.com/item?id=20844363
rsync -av ~/.Rhistory .
rsync -av ~/.Renviron .
rsync -av ~/.R .
rsync -av ~/.gists .
rsync -av ~/.Rprofile .
rsync -av ~/.gitignore_global .
rsync -av ~/.ssh .
rsync -av ~/.Rapp.history .
rsync -av ~/.rstudio-desktop .
rsync -av ~/.zshrc .
rsync -av ~/.zsh_history .
rsync -av ~/.zprofile .
rsync -av ~/.gitconfig .
rsync -av ~/.rtweet_token.rds .
## Older Mac config files: I no longer use these files but maybe you do
#rsync -av ~/.bashrc .
#rsync -av ~/.bash_profile .
#rsync -av ~/.inputrc .
#rsync -av ~/.bash_history .
mkdir -p Cyberduck
rsync -av ~/Library/Preferences/ch.sudo.cyberduck.plist Cyberduck/
mkdir -p TextMate
rsync -av ~/Library/Application\ Support/TextMate/Bundles TextMate/
## https://support.rstudio.com/hc/en-us/articles/200534577-Resetting-RStudio-Desktop-s-State
mkdir -p RStudio/share
rsync -av ~/.local/share/rstudio RStudio/share/
mkdir -p RStudio/config
rsync -av ~/.config/rstudio RStudio/config/
rm Brewfile
sh update_brewfile.shNote that it removes the Brewfile and re-runs update_brewfile.sh at the end. If you are not using Homebrew, feel free to remove those lines from the script.
A lot of work goes into creating several of these Linux-style configuration files such as your ~/.Rprofile file for configuring your R sessions 27, or your connections to other computers and clusters (saved in your ~/.ssh directory). To save myself the pain of having to recreate these files, I make sure to save them.
To restore files, the fourth and final shell script is restore_files.sh
#!/bin/bash
# sh restore_files.sh
## Created this file on 2024-02-27
## see more info at update_files.sh
rsync -av .Rhistory ~/
rsync -av .Renviron ~/
rsync -av .R ~/
rsync -av .gists ~/
rsync -av .Rprofile ~/
rsync -av .gitignore_global ~/
rsync -av .ssh ~/
## Fix ~/.ssh permissions
chmod -R 600 ~/.ssh
rsync -av .Rapp.history ~/
rsync -av .rstudio-desktop ~/
rsync -av .zshrc ~/
rsync -av .zsh_history ~/
rsync -av .zprofile ~/
rsync -av .gitconfig ~/
rsync -av .rtweet_token.rds ~/
rsync -av Cyberduck/ch.sudo.cyberduck.plist ~/Library/Preferences/ch.sudo.cyberduck.plist
mkdir -p ~/Library/Application\ Support/TextMate/Bundles
rsync -av TextMate/Bundles ~/Library/Application\ Support/TextMate/
osascript -e 'tell app "TextMate" to reload bundles'
## https://support.rstudio.com/hc/en-us/articles/200534577-Resetting-RStudio-Desktop-s-State
rsync -av RStudio/share/rstudio ~/.local/share/
rsync -av RStudio/config/rstudio ~/.config/
sh restore_brewfile.shYou have to have previously re-installed TextMate, Cyberduck, RStudio, and Homebrew manually before running restore_files.sh. Of course, you also have to have downloaded your files from your cloud storage service first too 28.
As a note on ~/.ssh permissions, they have to be restrictive both on your laptop as well as at JHPCE for things to work well. Here are mine:
- Laptop:
.ssh % ls -lah
total 64K
drwx------ 12 leocollado staff 384 Mar 5 09:18 .
drwxr-x---+ 40 leocollado staff 1.3K Mar 5 09:58 ..
-rw------- 1 leocollado staff 1.7K Oct 3 13:20 config
-rw------- 1 leocollado staff 1.7K Dec 21 2016 id_rsa
-rw------- 1 leocollado staff 433 Dec 21 2016 id_rsa.pub
-rw------- 1 leocollado staff 13K Mar 5 09:18 known_hosts- JHPCE:
.ssh $ ls -lah
total 443K
drwx------ 2 lcollado biostats 8 Jun 29 2023 .
drwx------ 51 lcollado biostats 87 Mar 5 09:35 ..
-rw------- 1 lcollado biostats 1.5K Dec 21 2016 authorized_keys
-rw-r--r-- 1 lcollado biostats 515 Mar 18 2022 config
-rw------- 1 lcollado lieber_jaffe 453K Jun 29 2023 known_hostsI removed a few files from the output above to just highlight the main files.
18.1.3.4 Misc
Similar to the screenshots of the contents of my /Applications directory, I have taken screenshots of my Slack Workspaces and Dock bar setup. Just to remember in what order I have things. I also keep around the license files for Cyberduck, Hyberdock, Bartender, AlfredApp, and other software that I have paid for before. For iTerm2, I have also exported my favorite profile to a JSON file. For Cyberduck, you can also drag to a Finder window your Cyberduck bookmarks.
Note that some software like Zotero + Zotfile, AlfredApp, and the OneTab Chrome extension have direct options for backing up your Zotero library (via a Zotero account), your AlfredApp preferences, or OneTab bookmarks. Check if this is the case for any other software you use and for which you’d like to save the preferences you customized.
Ultimately, when you restore a computer from a cloud storage service, you will miss some things you had in your previous computer. Doing so can be intentional if you don’t want to keep around software you are actually no longer using. But you have to be conscious about this choice and know well what you’ll miss: otherwise you might regret losing some hard to get software or configuration files.
18.2 winOS
Unlike macOS, I don’t really bother to have an external storage device backup solution for my winOS computer. That’s because I have my work related information on my macOS laptop. However, Linux-style configuration files can be a bit more tricky to locate. Use functions like usethis::edit_r_environ() to actually locate these files. Unlike macOS, I manually have backed up (aka copy and pasted) my Linux-style configuration files that I use on my winOS machine to my cloud storage solution (aka Dropbox/Win-files).
Note that
gitwill create many hidden files under the.gitdirectory for a given repository, so if you have localgit clones of many repositories, this will greatly increase the number of files in your computer. Backing up lots of small files can take longer than backing up a single large file, depending on disk input/output limits as well as your internet bandwidth download/upload limits.↩︎See the R config files section for more details.↩︎
I have over 300,000 files on Dropbox that I want to have on my computer, so it takes quite a few hours to download everything again.↩︎