The misunderstanding lives in one slippage: treating non-attachment as not caring. Look closely at how the texts define it and the slippage falls apart.
Patanjali makes the structure explicit. In the Yoga Sutras (1.12) he says the restless mind is stilled by two things working together โ abhyasa, steady practice, and vairagya, dispassion. As Wikipedia notes, these are 'the key to restraint of the modifications of the mind.' Vairagya there is not withdrawal from the world; it is the inner half of a daily discipline, the loosening of the grip of craving so the mind can settle. It pairs with effort, not with escape.
Crucially, the word is built from vi (without) and raga (passion, craving) โ not from any word for 'leaving.' The reference works are emphatic that this is internal: as the Yoga literature puts it, true vairagya refers to a state of mind rather than to an external lifestyle, and can be practised just as well inside family life and a career as by a renunciate.
So the target is the clinging, the compulsive need for things to go a certain way โ not the love, the labour or the duty itself. You can pour yourself into raising a child, building work, tending a marriage, and still hold the outcomes with open hands. That open-handedness, not the empty hands of someone who walked out, is the real article.