Automatically close oldest buffers?

Let's solve the problem, not treat the symptoms. Vim shouldn't normally use large amounts of memory. It would be best isolate the issue. Some tips to help find the culprit:

• Disable plugins (use a binary search to make it faster)
• Reduce your ~/.vimrc down to see if anything in there is the problem
• Disable your ~/.vimrc completely via vim -u NONE

Also see How do I debug my vimrc file?

If you find a memory bug with a plugin then contact the plugin developer. If you find a memory bug with Vim then submit a bug report with steps to reproduce the error. See :h bugs

The following should answer your question.

function! s:SortTimeStamps(lhs, rhs)
  return a:lhs[1] > a:rhs[1] ?  1
      : a:lhs[1] < a:rhs[1] ? -1
      :                        0
endfunction

function! s:Close(nb_to_keep)
  let saved_buffers = filter(range(0, bufnr('$')), 'buflisted(v:val) && ! getbufvar(v:val, "&modified")')
  let times = map(copy(saved_buffers), '[(v:val), getftime(bufname(v:val))]')
  call filter(times, 'v:val[1] > 0')
  call sort(times, function('s:SortTimeStamps'))
  let nb_to_keep = min([a:nb_to_keep, len(times)])
  let buffers_to_strip = map(copy(times[0:(nb_to_keep-1)]), 'v:val[0]')
  exe 'bw '.join(buffers_to_strip, ' ') 
endfunction

" Two ways to use it
" - manually
command! -nargs=1 CloseOldBuffers call s:Close(<args>)
" - or automatically
augroup CloseOldBuffers
  au!
  au BufNew * call s:Close(g:nb_buffers_to_keep)
augroup END
" and don't forget to set the option in your .vimrc
let g:nb_buffers_to_keep = 42

This is to be dropped into a plugin or in your vimrc config. Then, you'll have to choose how to use it.

I am not sure how to get the oldest buffers with respect to editing time, but one could, instead, try to close the oldest unedited buffers. Something like:

function CloseLast ()
    python <<EOF
import vim
N = 10
listed_buffers = [b for b in vim.buffers if b.options['buflisted'] and not b.options['modified']]
for i in range (0, len (listed_buffers) - N):
    vim.command (':bd' + str (listed_buffers[i].number))
EOF
endfunction

autocmd BufNew * call CloseLast()

Notes:

• vim.buffers is a list of every buffer opened in the current session, so it also includes unlisted buffers. It is not the same as the list returned by :ls.
• Therefore, we must filter out the hidden or deleted buffers. This can be checked using options['buflisted'].
• Similarly, options['modified'] lets us check if the buffer is modified.
• N is the number of unmodified, listed buffers you want open.

Thanks to Luc Hermitte's answer from which I learnt how to get the timestamps, you could use the following instead, to get the oldest inactive kicked out first:

listed_buffers = (b for b in vim.buffers if b.options['buflisted'] and not b.options['modified'])
oldest_buffers = sorted (listed_buffers, key = lambda b: eval('getftime("' + b.name + '")'))
for i in range (0, len (oldest_buffers) - N):
    vim.command (':bd' + str (oldest_buffers[i].number))