No one likes us but we don't care, says Milner

It is hard to begrudge James Milner his Manchester City pay-day now that it has come. A long and tortuous domestic career has been the source of such upheaval that his first eight seasons in senior football never started and finished with the same manager and he certainly has no good reason to remember a fair few of them.

It was after relegation with Leeds in 2004 and a move to Newcastle that Graeme Souness claimed that – to paraphrase one headline – "Kids like Milner will get me the sack". There is also a painful story of how Glenn Roeder, Souness' successor, agreed to sell Milner to Aston Villa, then changed his mind. Milner arrived in the Midlands to receive Roeder's decision from Martin O'Neill and was apparently in tears when he turned his car around and returned north.

But this difficult journey means nothing to those who have a mind only for where Milner has alighted – the most wealthy club in the land – and who want nothing more than to see the 24-year-old and his club trip up, particularly at locations like Wigan where they play today. "I haven't been at the club too long but I would have thought it is ... jealousy with the money and opportunities the club has because of the owner," Milner reflected. "But to be honest, it doesn't affect us what people outside the club think."

City's stop-start opening to the League campaign – one win in four fixtures was not a part of the script before today's visit to a place where they have never won a Premier League game – has kept Roberto Mancini in that perennially unnerving state, a couple of games away from crisis. Milner believes the £130m summer spending spree may not elicit full rewards for a few games yet. "It will take a bit of time to hit our best because we've a lot of new players," he said. "But hopefully it will take games rather than months. We've been unlucky the last couple of games. We just want to get there as fast as we can. The more time we spend together, the sooner we will click."

The midweek 2-0 victory over Salzburg bore out that hope, with the paeans of praise for David Silva reflecting the first evidence of his stellar ability. But Milner also senses that the kind of dressing-room camaraderie – which saw a poster of Nigel de Jong's notorious World Cup tackle on Xabi Alonso pinned up on the day Silva arrived – is permeating the club, now the assembled band of new arrivals are merging. "There is always banter flying about and it's a really good atmosphere. One day people will have a go at someone's clothes, then their taste in music. It is great to be among that. We've got some overseas players in the dressing-room, but we've got that spirit."

Milner said that he felt Randy Lerner's ambitions for Villa had receded when he made his decision to leave the Midlands. "The club were just that tiny bit short of getting to the next level," he said. "We needed to bring players in, but it became clear from Randy Lerner that they might have to sell to buy."

Mancini may start him on the left side of a three-man attack today, though the same berth Fabio Capello has settled him into for England is not his preferred one. "I think it does help if you play in one position all the time, you get settled and into a rhythm and being comfortable," Milner said. "I think the middle [of midfield] is where I play my best football. It suits my attributes best, things I have in my game like my energy."

Sacrifices have always been a part of his sporting life. "I enjoyed playing cricket as a boy and played for Yorkshire at the ages of 10 and 11 as a wicketkeeper/batsman," Milner said. "It is something I enjoyed but obviously you get to a stage where you have to make a choice." Now central midfield, where he flourished most at Villa, might be sacrificed. "If it is an asset to the manager and the team that I can play in different positions, I'll accept playing where I am picked."

Though City's new captain Carlos Tevez has scored five goals in his last six League appearances against Wigan, Milner knows many neutrals will be hoping that Wigan's win at Tottenham has galvanised them. He has had far worse than that to contend with in his career. "If we can point to silverware, it won't matter if it makes people happy outside City fans, we know we will have done our jobs," he concluded. "That is the biggest thing. We don't care about anything else."