Backup Procedures

Overview

Backup processes associated with each CEHD server. Process descriptions should include

  • What is being backed up
  • How often backups are occurring
  • Where the backups are located (server and directory)
  • Overview of the backup process used (e.g., rsync, scp, NFS to Utility)
  • Any special precautions or processes involved (e.g., database replication)

CEHDDC12

DHCP

  • Setup Backup of DHCP database

​Active Directory

  • Primary backup strategy is that we now have three 2008 R2 Domain Controllers in three different locations.  AD is replicated across all three to be able to handle down servers.
  • Is there a way to backup AD structure to be able to restore in the case of major AD problems?

updated 8/24/2012

HEATONDC

  • Domain Controller located in Heaton
  • DNS
  • DHCP - Setup Backup of DHCP database

updated 8/24/2012

RIVERSIDEDC

 

  • Domain Controller located at Riverside with TCALL
  • DNS

updated 8/24/2012

 

CEHDSERVER

Archives of removed and updated files are not kept.

  • Shares are mounted on HEATONFILEBACKUP and the files are synced using rsync.
  • rsync run nightly at 4:30am
  • This solution provides a mirror of the data from CEHDSERVER
  • This backup is for disaster recovery only

PRINTSERVERWIN2K8

Printers

APPSERVERWin2K8

This server contains no data, but the VM should be backed up.

  • CAC uses this server as a waypoint to Titanium.
  • NEED TO LINK TO TITANIUM SETUP INSTRUCTIONS

THERMOPYLAE

  • Exchange 2008
  • PST files of full mailboxes exported weekly
  • PST files of incremental changes exported daily
  • PST files are rsynced with HEATONFILEBACKUP daily
  • See THERMOPYLAE for more details.

PUBLICWEB2

  • Full Backups: Each user and vhost directory is tar-gzipped each Saturday
  • Incremental Backups: Any changed/new file since the previous Saturday's full backup is tar-gzipped Sun-Fri
  • The full and incremental backups are made to the utility server
  • The backups are rsynced to HEATONFILEBACKUP daily

PUBLICDB

  • Full Backups
    • 12:15am. Postgres DBs are backed up locally and copied to HFB
    • 1:15am. MySQL DBs are backed up locally and copied to HFB
  • Off Site Backups
    • HFB:/backup2/publicdb/[mysql] and [postgres]
  • Backups kept on a 1 week rotation
  • Reviewed 8/8/2012

Gateway

  • /etc/samba/smb.conf backup created on 4/25/2011 and synced to Heaton File Backup
  • /etc/ssh/sshd_config backup created on 4/25/2011 and synced to Heaton File Backup
  • /etc/vsftpd directory backup created on 4/25/2011 and synced to Heaton File Backup

Nagios/Utility

  • /usr/local/nagios directory tree backup created on 4/25/2011 and synced to HeatonFileBackup:/backup/nagios
  • add /etc/host.allow, /etc/exports
  • update backup at least annually or when significant changes

VPN

  • /etc/openvpn directory tree backup created on 4/25/2011 synced to Heaton File Backup
  • EASY-RSA directory tree (/root/openvpn-2.0.9/easy-rsa/2.0) backup created on 4/25/2011 synced to HeatonFileBackup:/backup/vpn
  • update backup at least annually or when significant changes

Drupal

  • education.tamu.edu; mycehd.tamu.edu; 3rd instance which will include many sites
  • Full Backups: /disks/www/drupal directory is tar-gzipped each Saturday
  • Incremental Backups: Any changed/new file since the previous Saturday's full backup is tar-gzipped Sun-Fri
  • The full and incremental backups are made to the utility server
  • The backups are rsynced to HEATONFILEBACKUP daily

CISVM

  • Full Backups of active sites under /disks/www/ tar-gzipped each Saturday
    • eahr, epsy, tlac, hlkn
    • /disks/backup/vhosts
  • Incremental Backups of active sites made each day other than Saturday tar-gzipped
    • eah, epsy, tlac, hlkn
    • /disks/backup/vhosts
  • 1 full backup + 6 incremental backups kept
  • Each backup is copied to HFB once created
    • HFB:/backup/cisvm/vhosts
  • mysqldump made daily of each active site
    • cw_eahrdev, cw_epsydev, cw_hlkndev, cw_tlacdev  (would be nice to rename these databases, but these are the correct ones)
    • backups made in /disks/backup/mysql
    • copied to HFB:/backup/cisvm/mysql
    • 1 weeks of backups are maintained
  • Backup of apache conf file made daily and copied to HFB:/backup/cisvm/conf

Data Portal Servers

DBHOST

All the databases are dumped and encrypted daily at 7pm. The encrypted dumps are shipped to a remote server (heatonfilebackup). Postgresql config files are also backuped up together.

On Heatonfilebackup, "thisweek" directory holds 7 day's worth of recent encrypted dumps.

Out of this, only the Saturday's backups are later archived into lastweek,lastmonth, ... directory. All other recent backups, like "last Tuesday's" will be overwritten away by a new "Tuesday's backup".

The backups are rotated on a 2 months cycle. At any time, 5 recent full backups are available.

  • this week (this past Friday), last week, the week before the last week.
  • this month (first Friday of this month), last month (first Friday of the last month)

For the rotation plan, see revolving_backup_scheme.

script: /root/bin/backup-dbhost .sh local backup directory: /disks/backup/pg84 remote backup directory: heatonfilebackup:/backup2/dbhost

DBWEB

Weekly Backups: Fri 8pm, by backup-weekly-hfb.sh Daily Backups: 8pm every day, by backup-daily-hfb.sh (will exit on Fridays) Location of Backups: heatonfilebackup:/backup2/dbweb Backup targets : web codes, files, perl lib

daily backup

On Heatonfilebackup, "daily-except-friday" directory holds recent 6 day's backups, excluding Friday's. The will be overwritten as new backups are made.

weekly backup

"thisweek" directory holds the last Friday's backup.

The files in the "thisweek" directory are archived into lastweek,lastmonth, ... directory as week passes.

The backups are rotated on a 2 months cycle. At any time, 5 recent full backups are available.

   this week (this past Friday), last week, the week before the last week.
   this month (first Friday of this month), last month (first Friday of the last month) 

For the rotation plan, see revolving_backup_scheme.

MYRECORD

Weekly Backups: Fri 8pm, by backup-weekly-hfb.sh Daily Backups: 8pm every day, by backup-daily-hfb.sh (will exit on Fridays) Location of Backups: heatonfilebackup:/backup2/dbweb Backup targets : web codes, files, perl lib, conf files

daily backup

On Heatonfilebackup, "daily-except-friday" directory holds recent 6 day's backups, excluding Friday's. The will be overwritten as new backups are made.

weekly backup

"thisweek" directory holds the last Friday's backup.

The files in the "thisweek" directory are archived into lastweek,lastmonth, ... directory as week passes.

The backups are rotated on a 2 months cycle. At any time, 5 recent full backups are available.

   this week (this past Friday), last week, the week before the last week.
   this month (first Friday of this month), last month (first Friday of the last month) 

For the rotation plan, see revolving_backup_scheme.

dbdev

dbdev is used as svn repository server for dbweb, myrecord, their perl libs, and courses11. The svn dump is created daily and stored locally, as well as sent to Heatonfilebackup daily.

Daily backup

Location: heatonfilebackup:/backup2/dbdev File names are in the following format, where $day is the day of the month. (For example, April 24's dump is ...24.txt.)

   courses11-svn-dump.$day.txt
   dbweb-svn-dump.$day.txt
   moodle20-svn-dump.$day.txt
   myrecord-svn-dump.$day.txt
   perllib-svn-dump.$day.txt

So, the dump is rotated after a month (can be 28 days ~ 31 days).

replication

For cehddb, faculty, and scheduler, we replicate the database from the dbhost (production) to dbdev server. The replication is almost realtime.

hourly backup

For cehddb, faculty, and scheduler, the replicated databases are dumped hourly by postgres user in /var/lib/pgsql, on dbdev server.

Moodle Servers

COURSES

Weekly Backups: Fri 10pm, by backup-weekly-hfb.sh (there can be exams if before 10pm.) Daily Backups: 3am every day, by backup-daily-hfb.sh (will not run on Fridays) Location of Backups: heatonfilebackup:/backup2/courses11 Backup targets : web codes, moodledata, conf files.

daily backup

On Heatonfilebackup, "daily" directory holds recent 7 day's moodledata backups. web codes or conf files are not backed up daily.

weekly backup

"thisweek" directory holds the last Friday's backup. This backup includes web files, conf, and moodledata.

The files in the "thisweek" directory are archived into lastweek,lastmonth, ... directory as week passes.

The backups are rotated on a 2 months cycle. At any time, 5 recent full backups are available.

   this week (this past Friday), last week, the week before the last week.
   this month (first Friday of this month), last month (first Friday of the last month) 

For the rotation plan, see revolving_backup_scheme.

Restoration Tests

  • December 5, 2013.  Restored an old backup for the purpose of finding a missing grade.  The restore worked correctly and we were able to load Moodle from a previous week.

TICONDEROGA

  • Ticonderoga is the MySQL backend for Moodle and some Drupal sites
  • A daily snapshot is taken of each producation database at 12:30am.
  • Database snapshot stored to /disks/backup on the local disk
  • SCP is then used to copy these to HeatonFileBackup's /backup2/ticonderoga
  • A month's worth of snapshots are kept.

MOODLE

  • Weekly Backups: Fri 8pm, by backup-weekly-hfb.sh
  • Daily Backups: Daily 1am by /root/bin/backup-daily.pl
  • Location of Backups: heatonfilebackup:/backup2/moodle11
  • Backup targets : web codes, moodledata, and conf files.

daily backup heatonfilebackup:/backup2/moodle11/daily

weekly backup

"thisweek" directory holds the last Friday's backup.
The files in the "thisweek" directory are archived into lastweek,lastmonth, ... directory as week passes.
The backups are rotated on a 2 months cycle. At any time, 5 recent full backups are available.

   this week (this past Friday), last week, the week before the last week.
   this month (first Friday of this month), last month (first Friday of the last month) 

For the rotation plan, see revolving_backup_scheme.

MOODLEDB

Since Fall 2011, we upgrade moodle (faculty) server in place, so we don't name directories and files with years, like moodle11, moodle12, any more. Moodle database is currently hosted in Ticonderoga and named "moodle".

We are doing full backups every day. There are no incremental backups. SQL dumps are kept for 31 days.

Daily Backups: 0:30 am every day, by dbbackup.sh Location of Backups: Backup is saved on nfs mount through nagios server.

 on ticonderoga: /backup 
   192.168.128.57:/backup/nfs_courses12db ... /backup
 on heatonfilebackup: /backup2/courses12db-nfs 
   165.91.232.14:/backup/nfs_courses12db ... /backup2/courses12db-nfs
 

Backup targets:   courses12 database and moodle database.

NSBRI Servers

All NSBRI server backups are under heatonfilebackup:/backup2/nsbri. In most cases, backup is done daily, and kept for 7 days, and then rotated away.

SVN

  • Backup script (/root/bin/svn-nims-backup.pl) runs daily and stored under /backups/
  • NSBRI contacts scripts, aliases file, and /etc/mail tar/gzipped --> /backups/nsbri-contacts.$dow.tgz
  • NIMS Wiki (Wiki) mysql database dumped to nimswiki.$dow.sql.gz
  • NIMS Help (Wiki) mysql database dumped to nimswiki.$dow.sql.gz
  • Apache files (/var/html) tar/gzipped to svn-www.$dow.tgz
  • NIMS Web files (waproot mounted from NIMSWeb.cehd.tamu.edu) backedup to nimsweb.$dow.tgz

NIMS (Web), nimshelp and nimswiki

 

NIMS, nimshelp and nimswiki Web codes are handled by subversion repository. The svn repository contains non-web codes, including nsbri-contacts solution in addition to is backed up every day.

List Example)

   conf.20110417.tgz
   nims-svn-dump.20110417.txt.gz
   nimshelp.20110417.sql.gz
   nimsweb.20110417.tgz
   nsbri_exchange-svn-dump.20110417.txt.gz
   nsbri-contacts.20110417.tgz
   nimswiki.20110417.sql.gz
   svn-www.20110417.tgz

NIMS (database)

The database is backed up every day. Then it is copied over to the HeatonFileBackup.

Ex) NIMS_backup_2011_04_17_200003_9488409.bak

The database backup scripts are stored under scripts subdirectory.

     Agent_job_NIMS_backup_daily.Data_backup.sql 
     Agent_job_Update_Project_Status.sql 
     nimsdb-backup.bat

NSBRI.org (Web)

The web code is static, and rarely changes. So we do not back it up every day.

This is the latest copy.

   MBR0303-NSBRI-web-20100911.zip

NSBRI (Database)

The database is backed up every day. It is copied over to the HeatonFileBackup, by the same script as NIMS (database) backup.

Ex) MBR0303-NSBRI_backup_2011_04_17_123502_8410647.bak

svn

svn.cehd.tamu.edu is used as the svn repository server for nsbri systems (nimsweb and accessory systems (mail server, etc.)).

nims code svn dump, nimshelp sql, nimswiki sql, nimsweb (actual nims web files), svn's www setup, and config files are backed up. Files are sent to Heaton File Backup, /backup2/nsbri. Backups older than 7 days are removed daily.

Backup Plan for db, myrecord, dbhost and moodle

A weekly backup is made and sent to the remote server (Heaton) into corresponding folders (dbweb, myrecord, dbhost, courses11, moodle11).

The weekly backup is first stored in "this week" folder. In the following week, this backup is moved to "Last week" folder and a new backup is put into the "this week" folder.

In the week after that, the backup in the "last week" is moved to "last week 2", and the backup in the "this week" is again moved to the "last week" folder. Again, a new backup is put into "this week" folder.

In the week thereafter, the existing backup in "last week 2" is overwritten with the backup from "last week" (so this backup is removed). The backup in "this week" is moved to "last week", and a new backup is put into "this week".

At the beginning of the month, the backup is copied to "this month" folder, as well as to "this week" folder. At the beginning of the next month, the backup in "this month" is moved to "last month", and a new backup is copied to "this month".

The following page illustrates the backup rotation.

http://cehdtech.tamu.edu/wiki/index.php/Revolving_backup_scheme

Taxonomy: