The film is based on the novel of the same name by A. J. Cronin. The year is 1880. On the outskirts of the fictional small Scottish town of Levenford there stands a strange building, half cottage, half castle, embraced with thick stone walls. The townsfolk nickname the fortress "Hatter's Castle", for James Brodie (Robert Newton), the man who built it. James Brodie is a hatter who keeps the members of the family in fear and submission; he is brutal, arrogant, selfish and cruel. His wife (Beatrice Varley), who has long been ailing, and his daughter Mary (Deborah Kerr), are in awe of him. His son Angus (Anthony Bateman), aged 15, alone dear to his heart, suffers under his love as the others suffer under his sternness. Brodie spends his evenings with Nancy (Enid Stamp-Taylor), the barmaid of the Winton Arms tavern. When Nancy's former lover, Dennis (Emlyn Williams), appears on the scene, she persuades Brodie to sack his faithful shop assistant, Perry, and give Dennis the job. Dennis, whom Brodie believes to be Nancy's step-brother, robs him secretly, and then aids Brodie's rivals, the Mungo Hat Co., in acquiring the shop next door. Meanwhile, Dennis schemes to marry Mary in order to gain a rich dowry. Dr. Renwick (James Mason) called in to assist the ailing Mrs. Brodie, loves Mary sincerely. Brodie forbids him the house. Mary, who has been taken advantage of by the unscrupulous and manipulative Dennis, sadly rejects the young doctor's proposal of marriage. Tragedy after tragedy ensues, all the direct results of Brodie's delusions of grandeur, selfishness and cruelty.