Another Database?? You're kidding Right??

[les_garten]

Hi,
This is one of my favorite forums. However, it is disheartening that we lose all the info from previous generations each time and have to start over. It is just a database migration. Think about it, what if your bank did this everytime it had to upgrade. Sorry, all your money's gone!! Make some more and we'll be happy to watch over it again for another year. The last time this happened I offered to help. I've got other things in my life to do, but I would have helped out to preserve the history of this rich forum. I hope for the sake of the history of the forum, that future migrations plan for migration of existing data. As someone mentioned, this is the fourth or so iteration of this forum and all that valuable data was lost. I'm not trying to rag on the forum moderators, it's just a shame to lose all the data each time. I have a SIGNIFICANT background in data engineering and am an email away. In a corporate environment I would only have lasted through one of these cycles and never been hired anywhere else. Just my .02

Les

 

[siddhartha]

Les,

I appreciate the comment, and the concern.

The issue was, and is database corruption. As it stands, we are unable to migrate the data from the old forum to this one without corrupting the database here, too. We have tried multiple times without success.

That said, it is not a forgotten issue, just not one that is solvable at this point. Trust me, I wish it was-far less work/frustration to migrate everyone's membership over here, instead of creating new ones here.

I think you can tell from the drastic increase in speed and stability here that we were definitely on to something. We currently back up everything here nightly, to avoid such a problem as we had in the past-so you know, the database corruption issue stemmed from the very first phpBB-based VLJ, as such, I was unable to get around it-it was there from the very beginning (as we were using a beta installation) and we were unable to correct that.

Chris

 

[les_garten]

Les,

I appreciate the comment, and the concern.

The issue was, and is database corruption. As it stands, we are unable to migrate the data from the old forum to this one without corrupting the database here, too. We have tried multiple times without success.

That said, it is not a forgotten issue, just not one that is solvable at this point. Trust me, I wish it was-far less work/frustration to migrate everyone's membership over here, instead of creating new ones here.

I think you can tell from the drastic increase in speed and stability here that we were definitely on to something. We currently back up everything here nightly, to avoid such a problem as we had in the past-so you know, the database corruption issue stemmed from the very first phpBB-based VLJ, as such, I was unable to get around it-it was there from the very beginning (as we were using a beta installation) and we were unable to correct that.

Chris

Hi Chris,
I understand you are a Physician. i recently built a Medical Records and billing System Database for a Internal Medicine practice. They will have perfectly up to date records even if their office burns down. There are 5 live synchronized copies of the database at any one time in different locations. I'm certainly not saying that "we" need this type of redundancy. But how about this, once a week or once a month, the database is stopped and made to do a backup on the server. That way, you have a maintenance window that everyone can be made aware of and your recovery is only as old as your last backup. I am assuming you are using mysql. It is trivial for it to do a backup. There is a reason why SUN Microsystems just purchased the rights to mysql, it is the pre-eminent DB software that Runs the Internet.

Also, if you can login to a DB and read the posts on the old forum, that DB can be recovered and used. The way to migrate would be to build the new server, migrate the users, migrate the DB, test the DB, then let the users know. Also new DB's are alays faster than old poorly designed DB's. A DB that contains no info will be quicker than a large non-optimized DB.

There is a reason why Physicians make the $$ they make an hour. There is a reason why Data Engineers make $150-200 an hour also.

I've been doing Data Engineering since '85. I've migrated from one OS to another. One DB language to another. Exported DB's, imported DB's. You name it. I've never lost a bit of data. You run a Mortgage company, Doctor's Office, or Bank's DB and lose their info, that's the ballgame, you don't ever get to play anymore.

Just my .04

Les

 

[bseal]

Somebody please be extra nice to Les, a new Goodwear perhaps, so we can get our old nuggets of info back.

 

[les_garten]

Somebody please be extra nice to Les, a new Goodwear perhaps, so we can get our old nuggets of info back.

Hi,
There in lies the snag. It's better to plan for catastrophe than respond to it. Now you are talking about merging two DB's that may not be setup the same. That's why when you are having problem, you build the new one, test, and migrate before the users have access to the new DB. Then make a hard cut from one to the other. I'm not saying it can't be done, it's just it's just tons less work than to have two operational Databases and try to merge them.

Les

 

[Andrew]

Just went over to the old site to drag across a post or two but it's currently down. Has it been switched off already?

 

[bfrench]

Hi,
This is one of my favorite forums. However, it is disheartening that we lose all the info from previous generations each time and have to start over. It is just a database migration. Think about it, what if your bank did this everytime it had to upgrade. Sorry, all your money's gone!! Make some more and we'll be happy to watch over it again for another year. The last time this happened I offered to help. I've got other things in my life to do, but I would have helped out to preserve the history of this rich forum. I hope for the sake of the history of the forum, that future migrations plan for migration of existing data. As someone mentioned, this is the fourth or so iteration of this forum and all that valuable data was lost. I'm not trying to rag on the forum moderators, it's just a shame to lose all the data each time. I have a SIGNIFICANT background in data engineering and am an email away. In a corporate environment I would only have lasted through one of these cycles and never been hired anywhere else. Just my .02

Les

Hi, Les,

Yes, you're right in being pissed off that the db is gone for a dump yet again.

But remember we're doing the backups regularly only to find that the backup doesn't work on another company's server - so we started fresh at my insistance.

A new installation of the forum program (phpbb3) and the creation of a new database with absolutely nothing else in those directories.

When Chris installed the backup along with phpbb on this server - we did dry runs - limited people (3) to try things - got errant multiple post replies to new posts made by me or one of the others.

These posts were in completely new topics and yet replies from members not registered in the new memberlist.

To me - this indicated something amiss either here on the new server or a carry over from the old server - and if I remember correctly we had the same foul up on the old server when you first offered help but indicating that at that time there wasn't much you'd be able to do after the fact.

So this time - with the move to the new host server - who specializes in these type forums along with their databases - I got insistant with Chris that we start all over with new everything.

I carry most of the work load registering 700 odd members and at times it does get busy but I felt it was in the best interest of the mods and members as a whole to begin anew.

There is nobody more disappointed than me about losing data but we were doing weekly backups and they didn't work - we had db copies on at least two different places - so where do we go from here?

We're doing weekly backups at this host - how do we know our data is not going to be corrupt?

These fora have very simple backup procedures - basically one file that brings all the data to one place. If it doesn't work on a RESTORE, then what?

And as far as migrating data - we should be able to use a db file for a same version forum on any host server - after all it's the same program using its own backup file.

We're not db experts - we rely on the host server to do that correctly and this is something that I feel that the original host server failed to do properly.

Maybe I'm all wet behind the ears in thinking this.

Bill French

 

[les_garten]

Hi Bill,
"We" appreciate everything you guys do to run the forum! Don't feel that I am ragging on you guys. It just is disconcerting to lose all that great information. Don't take this the wrong way, cause I mean it in the kindest possible way, but I have a sneaking suspicion that you guys are pretty ignorant here in the ways of L.A.M.P. Ignorant doesn't mean stupid, doesn't mean you are not doing the best job you know how. Just means you are not an expert, and this is not your primary vocation. Ask Priscilla Presley whether the guy doing your plastic Surgery needs to be a Board Certified Plastic Surgeon or just a guy who knows how to work a syringe?

If you know how to make LAMP work, you are worth $100-150K/year or more. You CAN make "bad" backups but most likely that was not your problem. If you were a customer and came to me and said here's our old DB backup, please build us a new Forum from it, I would estimate the charge to be $2-$3000 if that gives you any idea. It's a lot of work to make it seamless.

Anyhow, if you guys find yourselves in this position again.... Feel free to drop me an email and we'll see if we can figure it out. The main thing is to plan out your move and make sure you've got a good setup to go to before you make the jump. Also, it is not the job of the host to make this work for you. They are basically a storage facility, they give you a place to park your Data, and some tools to administer it. Host facilities actually make all this more difficult because they don't necessarily give you the proper access to administer your site as if it were your own server.

Let me say this again, THANX for the forum, THANX for the New URL for the site, THANX for your unselfish time commitments, and THANX for running a very classy forum for a Unique hobby and letting us operate in a "FREE SPEECH" environment.