i seem to have been building it for ages -2 hours and i'm only at 4% i have 2 sometimes four villagers building it. Why is it taking so long?

are u on a slower speed perhaps?

What speed are you running the game at? What are the levels skill of your builders? What is your food situation? If you have plenty of food they are more likely to wander off and do other things. Similarly, if you have too little food they will start 'worrying about food' instread of working steadily.

Even at 2X Speed, it does seem like a VERY long time especially when you're desperate to feed your villagers.

Make sure your builders are "on the job"; I tend to watch the trainee builders because they will eventually wander off and do their own thing until they become adept builders.

i am on the 2x speed, yes most of the builders are trainee i only have 236 food

Keep an eye on them, they must be running off, possibly due to your low food count. I just started building the dam two hours ago, with two builders, and I'm up to 12%. One builder is adept and the other is still just a trainee. 2x speed, normal difficulty, and with more then 500 in the bin. So I must assume your builders are not staying on task, due to the low food levels. I would suggest try keeping your food in the 350-450 range and see if that helps.

i only had one adept builder and little food also and there was no problem with him staying on the job or finishing the task quite quickly.
i guess your suggestions are helpful - just wanted to point out, there might be some other problem as well.

do any of your builder dislike work or like resting? this could also hint to some possible problems