As I have previously mentioned, knowing whether the gerund (going), the infinitive (to go) or the bare infinitive (go) is supposed to come after the verb is often just a matter of memorizing which form follows which verbs. What makes this particularly difficult is the fact that sometimes both options are possible but they have different meanings.
Depending on the context, “remember” can be followed by either the gerund or the infinitive.
If you use the gerund, it means that it’s a memory you have (or don’t have). It’s something that happened in the past. Take the following for example:
I remember driving for the first time when I was 16. In particular, I remember it being much more difficult that I had expected.
Because this is something I remember and I can recall it from my memory, I use the gerund to talk about it.
If you use the gerund, however, it doesn’t refer to something you did in the past and therefore a memory of it exists in your mind. Instead, it is used in a context in which you need to remember (and not forget) to do something. Your roommate, for example, tells you to remember to buy milk on the way home because you never remember to do the things he asks of you.