The Inn at Little Washington, just over an hour away from D.C., is neutral ground for celebrities, foodies and politicians — regardless of party affiliation.