Have a quick look at the demo on the FTB home page. When you try out the InsertImage button on the demo you will notice that it does not display a dialog to allow you to browse to an image. So you are not doing anything wrong. This is just how the control works. With InsertImage you have to paste in the URL of the image you want to include into the Image Source textbox.
If you want a dialog box where you can browse for images, upload them to the server, and include them in your FTB, etc, you should really use the InsertImageFromGallery toolbar button instead.
Here are some additional instructions for setting it up. http://freetextbox.com/forums/thread/9050.aspx