Is it possible to lose my castle?
No! That would be too mean. Of course, you receive less gold and experience points for a battle you lose, but even a defeat grants you at least a little experience.
In a multiplayer defeat, you lose crowns — the multiplayer coins — whenever you are looted or defeated in a duel. This could also lower your multiplayer ranking.