Thanks for putting me in the right direction and for anyone else having the same problem this is what I had!
I put wordpress including the index.php into a folder called 'blog' within my main website. Then had a link from my main website as follows [external links are visible to admins only]
This link would lead to the wordpress blog, however when running the generator it would not pick up the files from within the folder because it was an 'external' link!
So I changed it simply to blog/index.php
Tested the link on the main website and it still goes to the wordpress blog.
More importantly when I run the generator it now picks up the files from within the folder because it follows the internal link (blog/index.php) and indexes the pages.
Hope my pains will help others who have a similar problem.