Recruiters (at least the better ones) love getting as much details possible to ensure the right role fit for you/client. Imagine a real estate agent who wants to know the buyer's preference on location, budget, size, etc etc etc. the more they know about you, the better they can match your expectation right?
- My ans to the head hunter is; if they can pay me then I'm sure not an issue to deliver (maybe I'm not at median pay at my "level")

On why they fail to get you a new job - the market is simply very competitive, you can be good but they're many better candidates too in the market. The only way is to skill up, gain better experiences, involve in interesting projects, furnish your CV, do interview preps, do mock interviews, up your game to compete. Only you can help yourself here 
- I'm in a VERY specific FI industry and the project I'm always working on is "limited" or would I say happen once or twice in a lifetime within the industry change

There are many jobs out there but all companies are competing for the top 20% of the available talent pool; likewise there are many job seekers out there but everyone is applying to the top 20% available jobs in the market.
- Yes, but the salary scale may not match back. Guess same old mentality to pay peanut and expect to get a "good" employee .. So far my 2 headhunter where I deal separately I asked for salary I want which I get ... Lol