Keep it up, make it consistant. Pick a day to play and keep doing it. No matter the numbers. 3s,4s, 5s, 2s.
Theres games and things to play for any number of players. So dont just say ok, well lets go home if only 4 people show up, throw around, make up games, look up games from the web.
The closest tourneys to me have always been 2 hours minimum. Its definitely worth it.
Send emails to the Ultimate people close to you in that next town, tell them your story, and they will help you out.
Start picking up with them as much as you can. Try to meet halfway somewhere and play games. If they have a consistent following they should be willing to drive to you on occasion.
Tell your friends that a "team" from another town is coming to play a scrimmage and people will start to get excited about it.
But its true, show them some youtube videos or DVD first and get them pumped up.
My personal favorite is Above and Beyond
http://www.ultimatefrisbeevideo.com/Its older but the quality of presentation on that video is much more entertaining than any UltiVillage DVD I've seen,personally (havent checked out the new Club one yet).
It also has good introductions to Ultimate and how to play and such. Great highlight clips and great stories.
Good luck, yes where are you located?