This occurs when PuTTY tries to allocate more memory than the system can give it. This may happen for genuine reasons: if the computer really has run out of memory, or if you have configured an extremely large number of lines of scrollback in your terminal. PuTTY is not able to recover from running out of memory; it will terminate immediately after giving this error.
However, this error can also occur when memory is not running out at all, because PuTTY receives data in the wrong format. In SSH-2 and also in SFTP, the server sends the length of each message before the message itself; so PuTTY will receive the length, try to allocate space for the message, and then receive the rest of the message. If the length PuTTY receives is garbage, it will try to allocate a ridiculous amount of memory, and will terminate with an ‘Out of memory’ error.
This can happen in SSH-2, if PuTTY and the server have not enabled encryption in the same way (see question A.7.5 in the FAQ). Some versions of OpenSSH have a known problem with this: see question A.7.16.
This can also happen in PSCP or PSFTP, if your login scripts on the server generate output: the client program will be expecting an SFTP message starting with a length, and if it receives some text from your login scripts instead it will try to interpret them as a message length. See question A.7.6 for details of this.