There is a setting in Flash under Preferences > General > Workspace called "Open test movie in tabs". This works fine on both Windows and OSX when you test movie. However, it is broken on OSX when you open a swf using File > Open. It works as expected on Windows.

It is also broken when you use the new Flash CS4 Project panel and open a swf in the project panel which uses JSFL's command fl.openScript().

Regardless of how you open a swf on OSX, either using File > Open, the Project Panel, or fl.openScript, it still opens in a tab.

This is dead simple to replicate. Open Flash CS4 on OSX, make sure the "Open test movie in tabs" checkbox is unchecked, and use File > Open to open a swf. You will see it open in a tab. If you do the same thing on Windows, it opens in its own window just like it's supposed to. It's also worthwhile to point out that Flash CS3 on the Mac does not have this issue.

On a quick aside, in order to open swfs with JSFL, you have to use fl.openScript(). I find this interesting that this works because openScript is supposed to be for text files like .as, .jsfl, .txt, etc.

From the JSFL documentation, page 297:
fl.openScript()
A string, expressed as a file:/// URI, that specifies the path of the JSFL, AS, ASC, XML, TXT or other file that should be loaded into the Flash text editor.

Why would a swf open with that function, considering the description? At the very least, the docs need to be updated.