The German Chancellor Angela Merkel has said she wants to win back the voters who deserted her party for right-wing nationalist AfD.
Mrs Merkel is set for a fourth term in office, despite heavy losses for her party.
She now faces months of coalition talks with several parties in order to form a stable government.
After winning its first parliamentary seats, AfD vowed to fight "an invasion of foreigners" into the country.
"We want a different policy," co-leader Alexander Gauland said following the historic surge.
But splits have already emerged between AfD leaders on the party's direction. There were surprising scenes at the party's news conference on Monday morning when chairwoman Frauke Petry, the best known AfD politician, announced that she would not be part of the AfD bloc when she joins the Bundestag.
Mrs Merkel's conservative CDU/CSU alliance recorded its worst result in almost 70 years.
Mrs Merkel's conservative alliance has the largest number of seats and will form the next government - as the left-leaning parties together do not come remotely close to a parliamentary majority.
As working with AfD MPs would be unthinkable, and the second-placed SPD has apparently ruled itself out, the most likely scenario is a "Jamaica" coalition, so-called because of the colours of Jamaica's flag and the traditional colours of the German political parties.
Such a coalition - which has never been tried at the federal level - includes the black CDU/CSU, the yellow, business-friendly Free Democrats (FDP) - who are returning to parliament after a four-year hiatus with 80 seats - and the Greens (67 seats).
This combination could lead to prolonged horse-trading, as the Greens and FDP disagree on key policy issues.
Martin Schulz, who leads the Social Democrats (SPD), has vowed to prevent AfD from being the main opposition party and enjoying certain privileges as a result. He insists his party will not enter another coalition with the conservatives.
But the SPD's stance has drawn the ire of other party leaders.
The FDP's Christian Lindner said the Social Democrats' "rash and sudden decision to stay in the opposition in parliament is irresponsible".
The liberal FDP has been in government for more of Germany's post-war history than any other party.
After failing to win any seats in 2013, they have re-entered the Bundestag and will have 80 MPs.
The Greens, who increased their MPs to 67, said they were "deeply troubled" by the AfD being in parliament.
Regarding possible coalition talks, co-leader Katrin Göring-Eckardt said "negotiations between these four parties will be tough, it won't be easy"
On Monday the chancellor said she had wished for a better result in general.
"We have started to analyse the voters we lost, especially with regards to those who went on to vote for the AfD, we want to get them back by good politics and addressing some of the issues," Mrs Merkel said.
She said her party would speak to the liberal FDP and the Greens, "but also the SPD, as we governed together for a long time and it's always important to keep talking".
The party, which only formed four years ago, is expected to take 94 seats in the 709-seat federal parliament as the third-largest grouping.
AfD's campaign capitalised on a backlash over Mrs Merkel's decision to open Germany's borders to undocumented migrants and refugees in 2015, mainly from the Middle East.
Its success has shocked Germany's political establishment, and protests against the anti-Islam party have been held in several cities following the election result.
Alternative for Germany began as an anti-euro party but later turned its focus to immigration and Islam.
It called for a ban on minarets and declared Islam incompatible with German culture. Several of its candidates have been linked to far-right remarks.
Its election campaign posters carried messages such as "Stop Islamisation. Vote AFD" alongside images of women wearing burkas, and "Burkas? We stand for bikinis".
It performed particularly well in what was formerly East Germany, taking 21.5% of the vote as the second most popular party.
In Saxony, AfD narrowly beat the CDU to come out on top with 27% of the vote. According to analysis by broadcaster ARD, the AfD vote was highest among 35-44 year-olds, at 16%, and lowest among those above the age of 70, at 7%.
By BBC Monitoring
"Nightmare win for Merkel" was the headline in Germany's best-selling tabloid Bild, which published a picture of a tired-looking chancellor with her eyes shut.
Tanit Koch wrote in the paper that Merkel had "failed to ease concerns about crime and Islamisation". She said these fears had existed long before the refugee crisis of 2015 but politicians were "too genteel" to address them and AfD had "harvested the fruit".
Merkel's refugee policy was also seen as a crucial factor by Ulf Poschardt in Berlin's Die Welt broadsheet. He said that while it was "disastrous" that the "left opposition in the Bundestag" had not attacked the policy, the media were also to blame. "Broadcasts, articles and debates ignored many citizens," he added.
Berlin's left-wing Tageszeitung, meanwhile, focused on AfD leader Alexander Gauland with the headline "The rise of the scaremongers".
"The retros, the racists - now they're in," was the paper's warning. It said AfD would be provocative, as they would "spread anxiety and stoke aggression".
As to Mrs Merkel forming a government with the Greens and the Liberals, ARD TV's Thomas Baumann is sceptical, saying he cannot imagine what kind of "basic melody" such an alliance between ideologically different parties would produce.