The Oxford English Dictionary, as credited by Hugkiss.com, states the first"X-as-kiss" in literature occurred in 1901. (Where it appeared isn't mentioned.)
The site, as well as Wikipedia, describes the X symbol as originally referring to the Christian cross (and the first letter of the Greek word for Christ, Xristos). This dates back to a mostly pre-literate era when the marking of a cross, or "X," on a document functioned like a signature.
The X became associated with the kiss because signatories used to pucker up and plant one on their X mark, as one would do to a crucifix or bible. Trivia-Library.com cites a hypothesis from The People's Almanac, which theorizes that the X may have also functioned as a sort of early emoticon, representing a "stylized picture of two mouths touching each other."
As for that hug-of-an-O, it's something of a mystery. Hugkiss.com says it's a "North American custom" and cites Leo Rosten's "The Joys of Yiddish" as saying that Jews who were illiterate preferred using an "O" when signing documents to avoid making the cross mark. Others speculate the O represents two arms held in a circle around another person.
All well and good, but just know if you wander in somewhere flashing three neon Xs on a marquee, you're probably going to see a whole lot more than hugs or kisses.