Well, the thing is that nobody is saying it's your fault.
(All of the hypotheticals I present are for the sake of argument. Although I believe them to be true, if they are true can come later, I'd just like to establish this first.)
If it were true that nonwhites have a disadvantage because of the systems of oppression that once existed because of white people that didn't allow them to inherit the same wealth and power as whites and may still exist, then on average/as a group, white people benefit from the oppression their ancestors were the cause of because they inherited at least some of that wealth and power.
Moreover, if that is true then, on average/as a group, black people suffer from said oppression and reparations make sense, because white people would be in a worse position if that oppression had not occurred, and so it is not taking away from you, because you, in a just world, would be in the same place reparations ideally would leave you anyways. This is a hypothetical that leaves a lot of assumptions and questions (how do we make sure that every white person has benefited from oppression and they pay the amount their advantage has allowed them to have?), but if the premises are true, it seems, to me, self evident that reparations in a perfect world make sense.
If we can agree on that, then we only need to ask if we live in an equitable world where all people have the same ability to succeed, and, if they don't, if it's viable to do something to fix it through reparations or other means.