The X-linked inhibitor of apoptosis protein (XIAP) is a potent cellular inhibitor of apoptosis. Designing small-molecule inhibitors that target the BIR3 domain of XIAP, where Smac/DIABLO (second mitochondria-derived activator of caspase/direct IAP-binding protein with low pI) and caspase-9 bind, is a promising strategy for inhibiting the antiapoptotic activity of XIAP and for overcoming apoptosis resistance of cancer cells mediated by XIAP. Herein, we report the development of a homogeneous high-throughput assay based on fluorescence polarization for measuring the binding affinities of small-molecule inhibitors to the BIR3 domain of XIAP. Among four fluorescent probes tested, a mutated N-terminal Smac peptide (AbuRPFK-(5-Fam)-NH(2)) showed the highest affinity (Kd =17.92 nM) and a large dynamic range (deltamP = 231 +/- 0.9), and was selected as the most suitable probe for the binding assay. The binding conditions (DMSO tolerance and stability) have been investigated. Under optimized conditions, a Z' factor of 0.88 was achieved in a 96-well format for high-throughput screening. It was found that the popular Cheng-Prusoff equation is invalid for the calculation of the competitive inhibition constants (Ki values) for inhibitors in the FP-based competitive binding assay conditions, and accordingly, a new mathematical equation was developed, validated, and used to compute the Ki values. An associated Web-based computer program was also developed for this task. Several known Smac peptides with high and low affinities have been evaluated under the assay conditions and the results obtained indicated that the FP-based competitive binding assay performs correctly as designed: it can quantitatively and accurately determine the binding affinities of Smac-based peptide inhibitors with a wide range of affinities, and is suitable for high-throughput screening of inhibitors binding to the XIAP BIR3 domain.