Scott,
The Fein gives you good control of the sides of the groove, but like you questioned, how to control the depth? I called Teak Deck Systems and spoke to their tech guys a couple of times. As they explained it to me, the teak "ledge" is just to do two things: first, to minimize the volume of the groove, and second, to control the spacing of the deck planks. They said that if I was doing the deck in place I could just take the groove right down to the fiberglass underdeck. Not really any different than how planking was done a hundred years ago: spacing was maintained by shims and then the caulking was hammered into the goove. I could also dispense with the taping in the bottom of the groove, but the downside was in the use of more caulk. Instead of using roughly 40 tubes of SIS440 I used 70. Still, I'll have the advantage of being able to sand the decks again in the future without having to worry about taking so much off that I'll thin the caulking too much.
I was skeptical about the process, but TDS assured me that it would do no harm and if the deck becomes problematic in five or ten years I'll probably go with a complete deck replacement, or perhaps just revert to non-skid. As you cut through the teak you will feel the blade going through the bottom of the wood. You may scratch the gelcoat of the deck, but it's much harder than the teak and if you take your time you won't do any damage.
Just don't forget to sand the sides of the grooves to give the SIS a good clean surface to bond to. I used a scrap of lexan (plywood would be just as good) and wrapped 80 grit around it. It's going to take a couple of days to do each side properly, and you're going to hurt in the mornings, but wear knee pads and take a lot of breaks and you'll be satisfied with the results.
By the way, I did have to replace a couple of 5 and 8 foot long pieces of decking on the aft deck, but I did it before my regrooving. I had experimented with a variety of tools: both routers and saws. Control was impossible without elaborate guides, and I ended up with irregular grooves and the only remedy was to replace the planks. I had to plane the pieces to match the worn-down thickness of the rest of the deck, but I still had almost 3/4 inch thickness.
If someone could invent a tool or blade that would cut both the sides and bottom of the groove with total control they will be a hero (and probably make some money in the process)! Best of luck,
Tom