Error occurred during initialization of VM Could not reserve enough space for object heap Could not create the Java virtual machine

We get this error when java is running out of space and resulting in not functioning of Java virtual machine and it can be easily resolved by increasing the heap size.

=========================

[root@vps red5]# ./red5.sh

Running on  Linux

Starting Red5

Error occurred during initialization of VM

Could not reserve enough space for object heap

Could not create the Java virtual machine.

==========================

That means you need to increase the Heap-size.

SSH Server 

Run the command below

export CATALINA_OPTS="-Xms1024m -Xmx1024m"

*Note It will increase your heap size to 1024MB

If still red5 doesn't work increase more memory if still don't work 
Reboot server and then give it a try and If still doesn't work no way left increase RAM.