Yoma 35b - Hillel mechayav es ha'ani'im

The gemara there says that if three classes of people (poor, rich, wicked) are asked, "Why weren't you involved in Torah?" and try to make excuses, there are three people who are counter-examples: Hillel, R' Elazar ben Charsom, and Yosef Hatzaddik. Then the gemara explains, bringing a story about Hillel's incredible mesiras nefesh for Torah, despite his great poverty, and R' Elazar's incredible mesiras nefesh for Torah, despite his great wealth. Then for Yosef it brings the story of his resisting Potiphar's wife, despite her charms and attempts to seduce him.
Unlike the other two, it's hard to see what that has to do the claim, "Why weren't you involved in Torah?" (It does show that the excuse is not very good.)

See also https://gemaraboards.forumotion.com/t71-yoma-35b-various-questions-on-yosef-and-r-elazar-ben-charsom#156