topic Re: Teamwork server selective backup in Teamwork & BIMcloud

Teamwork server selective backup

Anonymous — Wed, 07 Sep 2011 10:17:20 GMT

Hello

Just configured our BIMserver to backup everything daily. Seems to work nicely.

I just wonder is there any way to backup just those projects/libraries that have changed since last backup? Now when full backups are made daily, Time Machine of course thinks that they are all new. But for example libraries doesn't really change. This makes alot of extra backup data, which won't be needed. So is there any way to exclude libraries or projects with no changes with ServerBackupTooll?

Re: Teamwork server selective backup

schagemann — Sat, 10 Sep 2011 06:40:40 GMT

interesting question - however from what i gather you can not do this with the ServerBackupTool itself.

i suppose you could create your own shell script with rsync.

btw. an indication on what OS and AC version you are on, will greatly help your helpers.

Re: Teamwork server selective backup

Anonymous — Sat, 10 Sep 2011 12:40:40 GMT

Hello,

We are using SuperDuper on Mac to fully
clone the server to an external FireWire disk
nightly.
I think it only clones the changed information.
After a server disk failure or similar the server
is bootable from that external hard disk in matter
of minutes.

Regards, Juha

Re: Teamwork server selective backup

Anonymous — Fri, 28 Oct 2011 22:15:17 GMT

Backup is an interesting question so far as I'm concerned.
It is mandatory.... and it has to be totally automatic.. I'm just too undisciplined to push the button every night.. or whenever scheduled.

I work in a Mac environment and Automator is simply too complicated to sort out without days of research that I would use just the once, and then forget it until the next time it is needed.... at which time I have to research it all over again. Writing scripts ... I'm not an IT architect... you know?

So what's left?
The backup tool... I can just manage using Terminal... but I don't like having to figure out the paths and for some reason when I cd through the file path.. my mac gets stuck at the BIM folder saying folder not found.. when it is right there in front of me in the finder window. Nope.... that's for the birds... and then I would have to write a script to automate it? Seriously? You must be joking.

I put the BIM server in a RAID array... that's going to help.
Then superduper to firewire bootable disk. Firewire does clone and smart update the whole volume on which the BIM server resides.. so that's a start.... there's an automated backup process there without scripting.

It's not enough.... Off site storage... Now we're getting interesting. I sync the whole BIM server to a cloud host using Sugarsync. It seems to work seemlessly in the background with no noticeable degridation or timeslicing.

But that's not enough... incremental backup... There are ten backups of each project database within the BIM server itself... is that enough? I rely on Time Line for incremental project backups ~ these stored on a different volume on the firewire disk.

What is everyone else doing?

Re: Teamwork server selective backup

Anonymous — Thu, 03 Nov 2011 17:01:54 GMT

We are using the BimServerBackUpTool, which as far I am aware, is the only way of creating a usable backup. Simply copying the data the BimServer creates to an external drive is not sufficient. If you have a fire or theft you cannot transfer the data onto a new BimServer.

However, I recently tested our backup by restoring the data onto a spare computer running a BimServer. It failed. I contacted our tech support who are looking into the issue - they are also unable to restore backups the BimServerBackUpTool creates.

In the meantime we are having to create PLAs of teamwork jobs every day, otherwise we'd be screwed in the event of a total loss.

Re: Teamwork server selective backup

Anonymous — Thu, 03 Nov 2011 20:02:37 GMT

Hi,
I use Superduper to duplicate my entire mac server on a bootable firewire disk and it seems, I can boot from either firewire or the server hard-drive at liberty and in both instances the BIM server is up and running... no issues. I clone the drive every night... presently without stopping the BIM service.

I'm trying to avoid scripts.... that's because once I set them up, that knowledge goes right out of mind and when I need it next time I have to go through the whole learning process again.... that is, every time ther's an upgrade. That's just silly stuff and I have better things to think about.

I have up to the hour backups of each project using time machine... saved to a different partition on the Server Duplicate... these provide recoverable project files.. and once the server duplicate is up and running, can be recovered quickly... However, Time machine does not succesfully back up the BIM service.

I mirror the project files, project backups and all attachments off site using sugarsync just in case.... and then we have the client copies from which we can recover if we have to. Sugarsync does not provide a reliable BIM Server backup.

It would be rather nice, if the BIM server backup procedure was part and parcel of the BIM service Manager interface... wouldnt it. I've had no success at all using the Wiki scripts. They always return errors for me. Maybe that's just me... but your comment, a bit disturbing.... confirms my fears. I've been testing our backups to make sure the system works... and then I hope to let it work all by itself without any further consideration from lazy ol' me.

Gaz.

csparchitects wrote:
We are using the BimServerBackUpTool, which as far I am aware, is the only way of creating a usable backup. Simply copying the data the BimServer creates to an external drive is not sufficient. If you have a fire or theft you cannot transfer the data onto a new BimServer.

....
In the meantime we are having to create PLAs of teamwork jobs every day, otherwise we'd be screwed in the event of a total loss.

Re: Teamwork server selective backup

Anonymous — Mon, 07 Nov 2011 11:43:22 GMT

gazpacho40 wrote:
I use Superduper to duplicate my entire mac server on a bootable firewire disk and it seems, I can boot from either firewire or the server hard-drive at liberty and in both instances the BIM server is up and running... no issues. I clone the drive every night... presently without stopping the BIM service.

I assume that you can boot from your firewire disk on any MAC? We are on PC, do you (or does anyone else) know if we can do the same here?

gazpacho40 wrote:
It would be rather nice, if the BIM server backup procedure was part and parcel of the BIM service Manager interface... wouldnt it.

Quite! I have requested this on the ArchiCad Facebook page and Graphisoft have acknowledged the request.

Re: Teamwork server selective backup

Anonymous — Mon, 07 Nov 2011 22:08:34 GMT

I've quickly forgotten all my PC knowledge.... but in a PC environment is it possible to use HD Clone to clone your entire op system partition, BIM server and all to a bootable external drive? I suppose you would then enter your replacement computer's BIOS to select that disk to boot from if you find yourself in trouble.

On a Mac I've only ever tried to use the clone copy on a mac with similar specs. Apple hardware upgrades generally match system upgrades so you can't really use "any" mac... but you can use any mac of the similar vintage.

csparchitects wrote:

I assume that you can boot from your firewire disk on any MAC? We are on PC, do you (or does anyone else) know if we can do the same here?

........
Quite! I have requested this on the ArchiCad Facebook page and Graphisoft have acknowledged the request.

Re: Teamwork server selective backup

Anonymous — Tue, 08 Nov 2011 09:19:39 GMT

gazpacho40 wrote:
On a Mac I've only ever tried to use the clone copy on a mac with similar specs. Apple hardware upgrades generally match system upgrades so you can't really use "any" mac... but you can use any mac of the similar vintage.

OK, just curious - if you had a fire/theft you should be able to get going again from the ext hd?

Graphisoft are looking into the BimServerBackUpTool issue. We received a new script to run yesterday, unfortunately it's only backing up the Libraries!

Re: Teamwork server selective backup

Anonymous — Tue, 08 Nov 2011 10:49:53 GMT

You'll need off site storage to deal with fire/theft so simply cloning a disk every night will not help unless you store it off site.

That's why I also use Sugarsync. It syncs the entire bim server folder including auto backup files and project files- and therefore off site, and as it happens, now i do run a scripted nightly Bimserverbackup (I got a friend in to help me set it up yesterday). Mind you, I am in trouble again as soon as there is an upgrade of hardware or software. We shouldn't have to deal with scripts! The Bimserverbackup target folder is synced off site using Sugarsync so that it's automatic stored off site as well as locally.

Re: Teamwork server selective backup

Anonymous — Tue, 08 Nov 2011 11:28:04 GMT

I do the backup during the day and take the disks home for that reason.

We briefly looked at Online storage, but due to the amount of data we create I didn't think it would be suitable. We do not have a leased broadband line and our upload speed is only 318kbps!

I guess you don't have any issues?

Re: Teamwork server selective backup

Anonymous — Tue, 08 Nov 2011 12:52:42 GMT

Our upload speed test is 5000kbs and downloads on the local speed test server are 75000kbs. Online Storage is about $1 per Gb per year these days. About the only good thing going for cloud atm. I've started archiving on line. I have a bunch of floppy disks, tapes, zip drives, etc that cannot be read. Lol, soon the same will go for cd and DVD.... blue ray will never really take off if online storage becomes accepted.

Gaz

Re: Teamwork server selective backup

Anonymous — Tue, 08 Nov 2011 13:26:01 GMT

Not sure if have got my calculations correct, but to back up the data the BimServerBackupTool creates to 'the cloud' (currently 27Gig) would take 23.5hours @ 318kbps!

Re: Teamwork server selective backup

Anonymous — Tue, 08 Nov 2011 19:41:10 GMT

OFF SITE BACKUP
Yeah. It's important to remember that your off site storage is basically to recover from a fire or theft. You'll have no office or office equipment.... so you'll not be up and running in an instant.

Your upload speeds are not so good. Can you have a look at that first? I once archived 80+Gb of old project records online on sugarsync which took over a day.. but that was because Sugarsync is not exactly fast and neither is the download speed from their server. Not that download speed would worry you so much as you'd be busy putting out the fire and finding alternative office accommodation.

Do you really have 27Gb in your BIMSERVERBACKUP target folder? You would need to archive all your completed projects so you aren't uploading those every night. On top of the the actual project files the BIM server backup folder only consumes 70KB. My backup folder contains 1Gb of data which, at your upload speed would take about an hour.

Alternatively, Sugarsync is a "smart update" process. Once synced for the first time, only files that change are uploaded.
Perhaps you could sync off site:
/BIM Server/TeamworkMessagingServer/conf
/BIM Server/TeamworkServer/Config
/Graphisoft/BIM Server/TeamworkServer/Attachments
/Graphisoft/BIM Server/TeamworkServer/Projects

These components require a cold backup to be effectively reintroduced, which sugarsync does not provide. At worst, you could readily salvage the last working project backup file.

CLONE YOUR BIM SERVER HARD DRIVE
If your hardware fails and needs to go into the shop for repairs, you can boot another machine from your clone (an external drive) and keep working. Basically zero downtime.

INCREMENTAL BACKUPS
The Bim server does incremental backups. This might be enough for you. If you need to go back a few weeks and return to an old idea you'll need to have an archive of it. You can do that by generating a pln file at the time you decide to change things, or you can use an automatic incremental backup. In a MAC, time machine generates these automatically. I can go back a month or more, choose the relevant BIM server generated project backup file and restore it. On a PC, you would need to set up a son, Father, grandfather style backup routine and run it automatically. those backups need only be stored on site on an external drive.

Re: Teamwork server selective backup

Anonymous — Tue, 08 Nov 2011 20:03:47 GMT

Oh... archiving old projects... archive them to the cloud for 10 years. They're safe there and you can even bill your client an archiving fee just like a lawyer does.
Gaz

Re: Teamwork server selective backup

Anonymous — Wed, 09 Nov 2011 10:36:05 GMT

26.8GB to be exact! 24GB approx in current projects, so archiving won't make that much difference.

If I understand you correctly, I'm not sure backing up new and used files using Sugarsync will help, the BimServerBackupTool is the only way to restore BimData (providing Graphisoft get their finger out and actually make it work!). What would be better is if the BSBUT backed up just new and changed files.

I'm reasonably IT literate but I'm not 100% sure I could stick a cloned HD into a fresh machine and it will work, maybe someone could advise.

With regard BimServer auto backups, we have never been able to restore these, maybe because of the size of the projects? So I will not be relying on these.

I think Graphisoft have some work to do on this MAJOR ISSUE, what also worries me (I don't know why it should) is that other users are plodding on backing up Bim data, not aware that it's useless and those who are using the BSBUT don't realise that it is not a reliable way to restore data.

Re: Teamwork server selective backup

Anonymous — Wed, 09 Nov 2011 11:57:37 GMT

csparchitects wrote:
If I understand you correctly, I'm not sure backing up new and used files using Sugarsync will help, the BimServerBackupTool is the only way to restore BimData (providing Graphisoft get their finger out and actually make it work!). What would be better is if the BSBUT backed up just new and changed files.

You seem to have a lot of projects on one BIM server. I would have split that up on a team basis if possible, if only so as to reduce the risk of the whole office going down because of one PC with a cracked motherboard.

From Sugarsync project file backup, I set up a new BIM server on a different Mac and successfully attempted and restored 4 projects using the "Restore Project Database on a different server" method described here:
http://www.archicadwiki.com/Teamwork/HowToRecoverLostProjects?highlight=%28Teamwork%2FTeamworkMultiBimBackup%29
That process does not involve the BSBUT.

Those bim server automatic incremental backups also work for me and I don't know why they aren't working for you.

I agree there's improvements that have to be made. The trick is to get everything to backup simply and automatically to cover all your bases... The one reason I do not like the BSBUT is that I don't like scripting... But cloning the hard drive on which the server resides is simple and intuitive. You should give it a spin.... the software I mentioned for PC is free... and you can take the external drive home with you.... I assume you now have two of these and rotate them so that one is always off site? The external drives are simply plugged in.. you could use eSATA drives for instance.

Re: Teamwork server selective backup

Anonymous — Wed, 09 Nov 2011 12:27:00 GMT

I didn't know you could recover lost projects that way!! thanks Gaz, our support agent didn't tell me this! I'm going to try it later today.

If it works for me, this could solve lots of issues.

I am using SyncToy at the moment to back up our data, which seems to work nicely. I run the BSBUT on an evening, which is backed up the following day. I know we will lose a days worth of data in the event of a total loss, but I'd be happy enough at that.

All our backups are done late in the afternoon and I take the HD's home, our 'heafty' machine doesn't suffer performance issues when it's doing it. 24GB of RAM and a quad core helps with that!

Re: Teamwork server selective backup

Anonymous — Wed, 09 Nov 2011 14:56:52 GMT

Gaz, its works. You are a star, if you are in York sometime I'll buy you a pint.

Re: Teamwork server selective backup

Anonymous — Wed, 09 Nov 2011 21:40:34 GMT

Thanks. "I'll have half".... always wanted to say that

On the subject of selective backup, The BSBUT backs up your projects, users, roles, categories and BIM library.

If you dont use the tool, or a clone, you can reliably backup BIM server "users", "roles" and "categories" from within the BIM Server Manager... as frequently as you want to.

Ideally, the entire backup and recovery process should be encapsulated in the BIM Server Manager graphic interface.

Our BIM server works well enough across the internet, so the location of the BIM server could itself be located off site (in your home for example.. or a virtual server maintained by someone else) and for self assurance, you can mirror it locally... or vice versa. I think we're headed that way in the near future. It would be nice for Graphisoft to take the initiative.

Gaz