"me" would be correct, at least in the English translation everyone's gotten used to.  Obviously, both are true, where God commits us to our angel and our angel to us, but given the English phraseology, "me" makes more sense as the antecedent of "to whom".

Angel of God, my guardian dear, to whom His love commits me here ...

It would be very awkward to consider the "to whom" to refer to the me in "my".

Bit the Latin is the opposite ...

Angele Dei, qui custos es mei, me tibi commissum pietate superna; hodie (hac nocte) illumina, custodi, rege, et guberna. Amen.

Literally, "Angel of God, who are my guardian, illuminate, guard, rule, and govern me this day [or night], [who has been] committed to you by the piety from above".  Obviously, that literal translation is pretty crummy and completely loses the rhyme.

So, you can see how the Latin rhymes very nicely, but the English translator cleverly also tries to make it rhyme, but in order to do that, he has to invert the "commits", since the "to whom" only makes sense as having the guardian angel as its antecedent.

You could try your hand at a new translation if it bothers you that much, where it's more like "Angel of God, who art the guardian of me, to whom God's love has committed thee ..." (and continue)

Again, in the interests of keeping a nice rhyme in translation, the usual English translation necessities flipping the "me being committed to thee" to "thee being committed to me".  Same difference IMO, and I wouldn't bother retranslating it even if I could somehow do a little better, since the typical translation is so common that it's largely "traditional" in English speaking countries.

"Angel of God, who art guardian of me,
committed to you by divine piety ..."

You could do worse than to just reuse the last part of the original ...

Angel of God, who art guardian of me,
committed to you by divine piety,
ever this day [night] be at my side,
to light and guard, to rule and guide.
From stain of sin do keep me free,
at the hour of death my helper be.