The Todoroki family is not a typical loving family, but a forced, abusive, unloving one from the start.
Enji Todoroki used his power and influence to force a marriage with Shoto's future mother. The purpose of this marriage was an attempt by Enji to conceive and raise a child with a powerful Quirk in order to surpass All Might. The marriage between the two was an unloving one and only done for Enji's personal desires.
The two had conceived two sons and one daughter before the hybrid Quirked Shoto was born. At the age of five, Enji trained Shoto to become a Hero through harsh and detestable means. Shoto had little interaction with his older brothers during his childhood due to his father's intervention. Shoto's mother tried to prevent Enji from making Shoto go through his intense training but Enji refused to listen and demanded she stayed out of it.
Shoto's mother was also a victim to Enji's abuse, to the point she started becoming mentally unstable. She snapped one day when she saw Shoto's left side which reminded her of Enji and threw boiling water over him. After this incident, Enji had her admitted to a mental hospital.
Over the course of the years, Shoto continued training while also hating his father before applying into U.A.
Aside from Shoto, it would appear that Enji treats his other children neutrally. He doesn't give them the harsh training that Shoto goes through nor does he seem to treat them properly as a traditional father would to his children, calling Shoto's brothers as 'things'. It is unknown as to what the relationship is between Shoto's brothers with Shoto and Enji. Nor is it confirmed what happened to them. However, it would appear Enji was tolerant with Shoto interacting with Fuyumi, as seen when they are older. This suggests she is treated differently from his sons.