Documentum 6.5 SP1 and SP2 support windows 2008. However it requires an update to the java version.
The following steps describe the procedure
1. Stop the Documentum services 1. Docbase 2. Docbroker 3. Java method server
2. If we installed Developer edition, stop Master service
3. Rename the folders $Documentum\java\1.5.0_12 to something else
4. Rename the folder $\Documentum\jboss4.2.0\jdk to something else
5. Install JDK 1.5 Update 19 to $Documentum\java\
6. Copy the jdk folder to $Documentum\jboss4.2.0
7. Restart the services