The solution is provided by John Wilkinson

RubyMine always saves files on frame deactivation. You can disable this behaviour in File | Settings | General | Save files on frame deactivation.
But even if it is disabled RubyMine will save the project settings (stored under the .idea directory), so you need to exclude .idea from autotest monitoring.

To exclude the folder just put .autotest file into your home directory (ubuntu, not sure about osx and windows) with following content

Autotest.add_hook :initialize do |at|
  %w{.svn .hg .git vendor public log .idea}.each {|exception| at.add_exception(exception)}
end

See the original discussion