Toshio's father married Takae and fathered Toshio. Toshio has a flashback about him at 7:20 into episode 6. He was a doctor too and Toshio was following in his footsteps. He died three years before the start of the show from pancreatic cancer. His death forced his son to return to the village.
Ozaki did not seem to have a good relation ship with his son and Toshio says that he hated him. Since he was young, Ozaki pressured Toshio into becoming a doctor, as he was the only Ozaki child.  Toshio also worries that he is becoming like his father.