It's the fight by the bridge with the level 3 bandit and his two Half-Ogre buddies who despite only being level 3 are depleting my fatigue in a single round of fighting and then kicking my unconscious body to death. I have a 1 Dodge, 3 Melee, 24 Defense(like 24 AC and 25% DR) and even doing it in turn based mode I'm getting destroyed every time. Am I just unlucky? I'm level 3, the only piece of armor I haven't upgraded is my suit(because far as I can tell there's no damn LARGE armor in Shrouded Hills).

You should be at least level 5, maybe 6, when trying to leave Shrouded Hills.

What is your weapon? And you really should use turn-based combat; real-time is a forced, rushed addition just before game release. Well, there are two of them, for sure. You're trying to get them, do you? :]
In fact, Lukan's bodyguards' leather armours are unique and their AC value is lower than regular ones that you can buy or find elsewhere. Probably to make low-level character's life a bit easier, if one decides to fight them.

If you weren't lucky with random loot at the blimp crash site or in the mines, your only possibility in Shrouded Hills area is buying or stealing some armour from the blacksmith or Gypsy woman.

There are several ways to give yourself an advantage in this battle.

Remember the railroad spikes at the crash site? Drop a few of them in front of Lukan and each ogre, before you start talking to them. They'll pick them up, because the game's AI says that any weapon is better than no weapon; but in this case, the spikes do a lot less damage than their fists.

You can also stock up on Fatigue potions, including the fatigue restorer you found at the crash site. (Drag them into the hotkey bank — the empty row of squares near the bottom of the screen — and then click on them to use them.)

Did you help Doc Roberts thwart the bank robbers? He'll give you the weapon of your choice as a thank you. I personally prefer the axe, because it can also be used to smash open doors and barrels (like the one in the thieves' camp) without taking any damage itself.

You might also consider exploring the silver mine on the western side of town. You might get lucky there as well, and find a better weapon or some armor.

Other options include putting a point into Persuasion (so you can trick the thieves into thinking you're one as well) or actually doing their quest to destroy the other bridge (which will start you down the path to the dark side, muahaha!)

Good luc— oh wait, you said you were playing a half-ogre. I meant to say, GUD LUK