Potential energy does not changeif you are just holding a body in a field(here magnetic) without changing the position of the body. So no, even if the magnet didn't lose its property over time, it would still not lead to an increase in PE if you didn't push the magnets around(i.e., changed their positions wrt to each other). So no, no infinite PE.