In any game, breakable objects are tough enough. Only in extremely recent games like "The Force Unleashed" have they even been brought to a level higher in complexity than objects like tables that split into a few premodelled pieces when damaged enough. And even THAT takes considerable effort.
Even in 3D animation it's usually a simple illusion done with particles, and it's hard to pull off correctly because it's usually only on random faces that an object will break into chunks.
Now, you're talking about a BUILDING...That's different **** altogether. A building isn't a piece of wood that's solid and can break into a few jagged pieces and be done. If you want it to look realistic, you have to account for what's INSIDE the walls, what's inside the building, supports, drywall, what have you.
Making a realistic destruction in this instance is a project in itself. There are 3D plugins like ThinkingParticles and BlastCode (which looks kickass, but is for Maya only ;_; ) which have features/are designed to handle this sort of thing, but that stuff is extremely expensive.
Now, single-player, you can get a good deal of destruction out of say...the Source engine. In Garry's Mod, there are a few maps built around 100% destructible structure, like, a destructible house, drydock, Roman temple...
The first two of those no longer work because of something that hapenned between the last major update to GMod...But the house, for instance, was built up of custom models of drywall and plywood carefully arranged. It lagged like crazy when massive destruction was going on, and would kill any multiplay server, but essentially it could be chipped away piece by piece until it all collapsed into a very realistic looking wreck. It wasn't perfect mind you. Nearly all the custom objects had no proper gib and would just disappear when destroyed, or the house would stay standing at moments where structural fatigue would have no effect, because some of the supports where built to float apart from gravity.
Drydock was better, but only so much. Still no real gibbing, though all the breakable ropes in the ceiling helped make it look cool when it collapsed. Unfortunately it was a bit too big to completely collapse under any one thing lesser than a damned nuke.
Now, Temple was done very well. It was big, but not too big, used stone walls and realistic segments of Roman-esque pillars. It was simple, but looked realistic in construction. It didn't gib into models, but it did gib into sizeable concrete chunks that looked chaotic enough in a full blown demolition. This one was easily the most realistic in terms of how and when it would collapse. The inner walls could all be destroyed without affecting the pillars, if you were very careful, but normally this wasn't the case. take down two corners, and watchit collapse on itself. Then, later, take out a support or two and watch the back wall you left standing stretch and give out under it's own weight and fall into the mess. Adding an antlion guard and tricking it into ramming supports in this map is just plain fun.
But, I digress. Now, recently in HL2 Episode 2, the Source engine was equipped with a cinematic destruction feature, where a complex object like a bridge, wooden house or steel radio tower would be simulated in a destruction sequence, which would then be saved as cached vertex data, and easily called upon without lag in-game. Now, this destructible isn't really player manipulated, unless you count the triggers that cause them. They're pre-simulated and will always result in the same exact sequence, whatever the player does, so they're not really physics objects...just very very complex animated models.
In short, destructibles are extremely hard, an artform of their own, and expecting a relic like GldSource to handle it realistically is not so much a tall order as a request to part the Red Sea.