Yeah, that way of viewing things does lack nuance.
You aren't forced to work for somebody else. You always have the option not to. No matter the consequences of that, there is no coercion involved so long as your prospective boss isn't the cause of said consequence. For example, you ate the last of your food and were unable to afford more, so you starved, but the cause of said running out of food wasn't another person. Being unable to afford more food likewise wasn't caused, as no one forcibly took away money you already had and would've used for this purpose (except the state when it taxes you, but I digress).
You might retort that food exists and your local supermarket's employees prevent you from just walking out with it, but that food was grown. It is the product of another's labors. Under a Georgist understanding you may be entitled to the natural fruits of the earth, but the pre-modern world didn't spontaneously generate vast quantities of wheat or magically see it ground into flour. The value added by humans belongs to them, not to you.
Then, you might add that owners of the means of production don't labor, meaning they don't add value to the earth's bounty. Rather, employees do. My answer is that owners do by proxy; their responsibility is to do the work, but they entered into consensual arrangements to sublet the responsibility to others for a fixed fraction of the end harvest. Because this didn't involve slavery, it wasn't immoral.
The final objection is to challenge their right to own the land to the exclusion of others. My answer is that arrangements involving private ownership of land and resources have proven most efficient in creating wealth for the masses, compared to all alternatives that both are known and have been tried. This is evident by all of the world's advanced economies having private property rights, and by no known anarchic society (e.g. countries in the middle of civil wars) being prosperous by modern standards.
So far as you lack the ability to change the way things are, it's because you lack the moral and legal right to do so. It's not oppression to be denied the ability to do what you mustn't do.
I will say that you are a "slave" to society so far as it informs your view of reality and traps your mind in a limited bubble. The best way to get around this is to study foreign or non-current perspectives. For example, pre-modern philosophies and diverse literature from around the world, and learn the history of the rise and fall of nations. This will broaden your horizons and give you a panoramic view of things transcending modern biases and blindspots.